# From General to Specific: Online Updating for Blind Super-Resolution

Shang Li<EMAIL_ADDRESS>Guixuan Zhang<EMAIL_ADDRESS>Zhengxiong Luo<EMAIL_ADDRESS>
Jie Liu<EMAIL_ADDRESS>Zhi Zeng<EMAIL_ADDRESS>Shuwu Zhang
<EMAIL_ADDRESS>Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences
No.95, Zhongguancun East Road, Beijing 100190, PR China

###### Abstract

Most deep learning-based super-resolution (SR) methods are not image-specific:
1) They are trained on samples synthesized by predefined degradations (_e.g_.
bicubic downsampling), regardless of the domain gap between training and
testing data. 2) During testing, they super-resolve all images by the same set
of model weights, ignoring the degradation variety. As a result, most previous
methods may suffer a performance drop when the degradations of test images are
unknown and various (_i.e_. the case of blind SR). To address these issues, we
propose an online SR (ONSR) method. It does not rely on predefined
degradations and allows the model weights to be updated according to the
degradation of the test image. Specifically, ONSR consists of two branches,
namely internal branch (IB) and external branch (EB). IB could learn the
specific degradation of the given test LR image, and EB could learn to super
resolve images degraded by the learned degradation. In this way, ONSR could
customize a specific model for each test image, and thus get more robust to
various degradations. Extensive experiments on both synthesized and real-world
images show that ONSR can generate more visually favorable SR results and
achieve state-of-the-art performance in blind SR.

###### keywords:

blind super-resolution , online updating , internal learning , external
learning

##  1 Introduction

Single image super-resolution (SISR) aims to reconstruct a plausible high-
resolution (HR) image from its low-resolution (LR) counterpart. As a
fundamental vision task, it has been widely applied in video enhancement,
medical imaging and surveillance imaging. Mathematically, the HR image
$\mathbf{x}$ and LR image $\mathbf{y}$ are related by a degradation model

$\small\mathbf{y}=(\mathbf{k}\otimes\mathbf{x})\downarrow_{s}+\mathbf{n},$ (1)

where $\otimes$ represents two-dimensional convolution of $\mathbf{x}$ with
blur kernel $\mathbf{k}$, $\downarrow_{s}$ denotes the $s$-fold downsampler,
and $\mathbf{n}$ is usually assumed to be additive, white Gaussian noise
(AWGN) [1]. The goal of SISR is to restore the corresponding HR image of the
given LR image, which is a classical ill-posed inverse problem.

Figure 1: The adaptation problem of the offline trained ESRGAN and RCAN. The
corresponding blur kernel is at the bottom right of each image.

Recently, super-resolution (SR) has been continuously advanced by various deep
learning-based methods. Although these methods have exhibited promising
performance, there is a common limitation: they are too ’general’ and not
image-specific. Firstly, they are exhaustively trained via LR-HR image pairs
synthesized by predefined degradations, ignoring the real degradations of test
images (_i.e_. non-blind SR). For example, most no-blind SR methods are
optimized by paired samples synthesized with Bicubic degradation [2, 3, 4].
And many blind SR methods also use Gaussian blur kernels to synthesize data to
optimize their SR modules, such as IKC [5] and DASR [6]. When the degradations
of test images are different from the predefined ones, they may suffer a
significant performance drop. Secondly, their model weights are fixed during
testing, and all images are super-resolved by the same set of weights.
However, test images are usually degraded by a wide range of degradations. If
the model performs well for certain degradations, it is likely to perform
badly for others. Thus, super-resolving all images with the same model weights
may lead to sub-optimal results. For example, as shown in Figure 1, ESRGAN
[3], and RCAN [2] are trained via bicubically synthesized LR-HR pairs. They
have excellent performance on bicubically downscaled images but may produce
undesirable results when the images are blurred by unseen kernels.

Towards these issues, we propose an online updating network namely ONSR, which
1) synthesizes the training data according to the degradation of the test
image, instead of by predefined degradations, and 2) updates the SR model
weights for each test image instead of keeping the model fixed. Specifically,
we design two branches, namely internal branch (IB) and external branch (EB).
For each given test image, IB tries to learn the internal degradation via
adversarial training [7]. With the learned degradation, EB degrades HR images
from external datasets to LR-HR pairs, which are then used to update the SR
module. Since the degradation is adaptively learned, the domain gap between
synthesized training data and the test image could be bridged. And as the SR
module can be updated for each test image, it may also get more robust to test
images with a large degradation variety.

In summary, our main contributions are as follows:

  * 1.

Towards the unknown and various degradations in blind SR, we propose an online
SR method. It could customize a specific model for each test LR image and thus
could have more robust performance in different cases.

  * 2.

We design two learning branches, IB and EB. They could learn the degradation
of the given test image and adaptively update the SR module according to
learned degradation.

  * 3.

Extensive experiments on both synthesized and real-world images show that ONSR
can generate more visually favorable SR results and achieve state-of-the-art
performance on blind SR.

##  2 Related Works

As shown in Table 1, nowadays SR methods can be roughly divided into two
categories: non-blind and blind. Blind SR methods assume that the degradation
of the test image is predefined (such as bicubic downsampling) or has been
estimated by degradation-prediction methods. While blind SR methods do not
need extra information about the degradation.

Table 1: Examples of non-blind and blind SR methods. Non-blind SR methods
assume that the degradation of the test image is predefined or has been
estimated by other methods. While blind SR methods assume that the degradation
is unavailable.

Non-blind SR methods | Blind SR methods
---|---
| SRCNN [8], RCAN [2], RRDB [3]
---
ZSSR [9], SRMD [10], USRNet [1]
| Ji _et al_. [11], Cornillere _et al_. [12], dSRVAE [13]
---
KernelGAN+ZSSR [14], IKC [1], DASR [6]

###  2.1 Non-Blind Super-Resolution

In non-blind SR the degradation is predefined or known beforehand, which is
easier to be studied. Thus most early SR methods are non-blind. These methods
are usually trained with paired LR-HR samples synthesized by predefined
degradation [15], such as bicubic downsampling. Since Dong  _et al_. propose
the first convolution neural network for SR (SRCNN) of bicubically downscaled
images and achieves remarkable performance. In the following years,
researchers focus on developing various network architectures. For example,
the residual dense network (RDN) [16] proposed by Zhang _et al_. and the
residual-in-residual dense block (RRDB) [3] proposed by Wang  _et al_. apply
skip connections [17] and dense connections [18] to make the SR network deeper
and hierarchical features more discriminative. The residual channel attention
network (RCAN) [2] proposed by Zhang _et al_., the channel attention and
spatial graph convolutional network (CASG) [4] proposed by Yang _et al_., and
the multi-attention augmented network (MAAN) [19] proposed by Chen _et al_.
all use attention mechanism to further enhance the representation capability.
Although these methods have largely advanced the SR performance for bicubic
downsampling, they usually perform poorly for other degradations. To alleviate
this problem, Zhang _et al_. propose two networks namely SRMD [10] and USRNet
[1]. They input the test image and its degradation simultaneously into the
network, in which way, the SR network can handle test images with various
degradations. However, these methods require extra accurate degradation-
estimation methods which are also left to be studied. Thus, in this paper, the
proposed method focuses on blind SR, which does not need degradation
prediction and is more applicable.

###  2.2 Blind Super-Resolution

Blind SR is much more challenging, as it is difficult for a single model to
generalize to different degradations. In [20], the final results are ensembled
from models that are capable of handling different cases. The SR modules in
IKC [5], DASR [6] and that proposed by Cornillere [12] are all pretrained by
synthesized data pairs containing a variety of degradations to be more robust
to different degradations. But there are countless degradations, and we cannot
train a model for each of them. Other methods try to utilize the internal
prior of the test image itself. In [21], the model is finetuned via similar
pairs searched from the test image. Irani _et al_. propose [22] and KernelGAN
[14] where the blur kernel is firstly estimated by maximizing the similarity
of recurring patches across multi-scale of the LR image, and then used to
synthesize LR-HR training samples. Ji _et al_. [11] also use a similar idea as
KernelGAN to estimate degradations. However, the number of internal patches is
limited, which heavily restricts the performance of these methods. Different
from these completely internal learning-based methods, our ONSR can apply the
degradation information via internal learning to external HR data, which helps
to better optimize the SR module. In this way, ONSR can simultaneously take
the benefits of internal and external priors in the LR and HR images
respectively.

###  2.3 Offline & Online Training in Super-Resolution

Most deep learning-based SR methods are offline optimized, _i.e_. their model
weights are only updated during training via the LR-HR pairs synthesized from
external data, while keep fixed during testing. Thus, the learned model
weights are completely determined by external data, without considering the
inherent information of the test image. Consequently, LR images degraded by
various degradations may get super-resolved by the same set of model weights.
It is likely that the model only performs well on certain types of images
while failing on others. Contrary to offline training, online training can get
the test LR image involved in model optimization. For example, ZSSR [9] is an
online trained SR method. It is optimized by the test LR image and its
downscaled version. Therefore, it can customize the network weights for each
test LR image, and could have more robust performance over different images.
However, the training samples of most online trained models are limited to
only one test image [22, 9, 14]. It will heavily restrict their performance.
Instead, in addition to the test LR image, our ONSR can also utilize the
external HR images during the online training phase. And in this way, it could
better incorporate general priors of the external data and the inherent
information of the test LR image.

Figure 2: (a) Offline training scheme. Training datasets are synthesized from
external HR image. The SR model are trained offline and only perform inference
online. (b) The online training scheme of ZSSR [9]. Only the test image is
used as the training data. The SR model is trained online.

##  3 Method

###  3.1 Motivation

As we have discussed above, previous non-blind SR methods are usually offline
trained (as shown in Figure 2(a)) [15], which means LR images with various
degradations are super-resolved with the same set of weights, regardless of
the specific degradation of the test image. Towards this problem, a
straightforward idea is to adopt an online training algorithm, _i.e_. adjust
the model weights for each test LR image with different degradations. A
similar idea namely “zero-shot” learning is used in ZSSR. As shown in Figure
2(b), ZSSR is trained with the test LR image and its downscaled version.
However, this pipeline has two in-born drawbacks: 1) with a limited number of
training samples, it only allows relatively simple network architectures to
avoid overfitting, thus adversely affecting the representation capability of
deep learning. 2) No HR images are involved. It is difficult for the model to
learn general priors of HR images, which is also essential for SR
reconstruction [23].

The drawbacks of ZSSR motivate us to think: a better online updating algorithm
should be able to utilize both the test LR image and external HR images. The
former provides inherent information about the degradation method, and the
latter enables the model to exploit better general priors. Therefore, a
“general” SR model can be adjusted to process the test LR image according to
its “specific” degradation, which we call: from “general” to “specific”.

Figure 3: The online updating scheme of ONSR. Top: internal branch. Bottom:
external branch. Images with solid borders are the input. Images with dotted
borders are the output of $G_{r}$ or $G_{d}$.

###  3.2 Formulation

Accoring to the framework of MAP (maximum a posterior) [24], the blind SR can
be formulated as:

$(\mathbf{k,x})=\mathop{\arg\max}_{\mathbf{k},\mathbf{x}}\|\mathbf{y}-(\mathbf{k}\otimes\mathbf{x})\downarrow_{s}\|^{2}+\mu\phi(\mathbf{x})+\nu\varphi(\mathbf{k})$
(2)

where $\|\mathbf{y}-(\mathbf{k}\otimes\mathbf{x})\downarrow_{s}\|^{2}$ is the
fidelity term. $\phi(\mathbf{x})$ and $\varphi(\mathbf{k})$ model the priors
of sharp image and blur kernel. $\mu$ and $\nu$ are trade-off regularization
parameters. Although many delicate handcrafted priors, such as the sparsity of
the dark channel [25], $L_{0}$-regularized intensity [26], and the recurrence
of the internal patch [27], have been suggested for $\phi(\mathbf{x})$ and
$\varphi(\mathbf{k})$, these heuristic priors could not cover more concrete
and essential characteristics of different LR images. To circumvent this
issue, we design two modules, _i.e_. the reconstruction module $G_{r}$ and the
degradation estimation module $G_{d}$, which can capture priors of
$\mathbf{x}$ and $\mathbf{k}$ in a learnable manner. We substitute
$\mathbf{x}$ by $G_{r}(\mathbf{y})$, and denote the degradation process as
$G_{d}(\cdot)$, then the problem becomes:

$\mathop{\arg\min}_{\theta_{G_{r}},\theta_{G_{d}}}\|\mathbf{y}-G_{d}(G_{r}(\mathbf{y};\theta_{G_{r}});\theta_{G_{d}})\|,$
(3)

The prior terms are removed because they could also be captured by the
generative networks $G_{r}(\cdot)$ and $G_{d}(\cdot)$ [23].

This problem involves the optimization of two neural networks, _i.e_. $G_{r}$
and $G_{d}$. Thus, we can adopt an alternating optimization strategy:

$\left\\{\begin{aligned}
&\theta_{G_{r}}^{i+1}=\mathop{\arg\min}_{\theta_{G_{r}}}\|\mathbf{y}-G_{d}(G_{r}(\mathbf{y};\theta_{G_{r}});\theta_{G_{d}}^{i})\|\\\
&\theta_{G_{d}}^{i+1}=\mathop{\arg\min}_{\theta_{G_{d}}}\|\mathbf{y}-G_{d}(G_{r}(\mathbf{y};\theta_{G_{r}}^{i});\theta_{G_{d}})\|.\\\
\end{aligned}\right.$ (4)

In the first step, we fix $G_{d}$ and optimize $G_{r}$, while in the second
step we fix $G_{r}$ and optimize $G_{d}$.

So far only the given LR image is involved in this optimization. However, as
we have discussed in Sec 3.1, the limited training sample may be not enough to
get $G_{r}$ sufficiently optimized, because there are usually too many
learnable parameters in $G_{r}$. Thus, we introduce the external HR images
$\mathbf{x}_{e}$ in the optimization of $G_{r}$. In the $i^{th}$ step, we
degrade the $\mathbf{x}_{e}$ by $G_{d}(\ \cdot\ ;\theta_{G_{d}}^{i})$ to
$\mathbf{y}_{e}$. Then $\mathbf{x}_{e}$ and $\mathbf{y}_{e}$ could form a
paired sample that could be used to optimize $G_{r}$. Thus, the alternating
optimization process becomes:

$\left\\{\begin{aligned}
&{\mathbf{y}_{e}}=G_{d}({\mathbf{x}_{e}};\theta_{G_{d}}^{i})\\\
&\theta_{G_{r}}^{i+1}=\mathop{\arg\min}_{\theta_{G_{r}}}\|{\mathbf{x}_{e}}-G_{r}({\mathbf{y}_{e}};\theta_{G_{r}})\|\\\
&\theta_{G_{d}}^{i+1}=\mathop{\arg\min}_{\theta_{G_{d}}}\|\mathbf{y}-G_{d}(G_{r}(\mathbf{y};\theta_{G_{r}}^{i});\theta_{G_{d}})\|,\\\
\end{aligned}\right.$ (5)

in which, $G_{r}$ is optimized by external datasets, while $G_{d}$ is
optimized by the given LR image only. At this point, we have derived the
proposed method from the perspective of alternating optimization. This may
help better understand OSNR.

###  3.3 Online Super-Resolution

As illustrated in Figure 3, our online SR (ONSR) consists of two branches,
_i.e_. IB and EB. IB and EB share the reconstruction module $G_{r}$ and the
degradation estimation module $G_{d}$. $G_{r}$ aims to map the given LR image
$\mathbf{y}$ from the LR domain $\mathcal{Y}\subset\mathbb{R}^{3\times H\times
W}$ to the HR domain $\mathcal{X}\subset\mathbb{R}^{3\times sH\times sW}$,
_i.e_. reconstructing an SR image $\mathbf{x}$. While $G_{d}$ aims to estimate
the specific degradation of the test LR image.

In IB, only the given LR image is involved. As shown in Figure 3, the input of
IB are patches randomly selected from the test LR image. The input LR patch
$\mathbf{y}\sim p_{\mathcal{Y}}$ is firstly super resolved by $G_{r}$ to an SR
patch. Then this SR patch is further degraded by $G_{d}$ to a fake LR patch.
To guarantee that the fake LR can be translated to the original LR domain, It
is supervised by the original LR patch via L1 loss. The paired SR and LR
patches could help $G_{d}$ to learn the specific degradation of the test
image. The optimization details will be further explained in Section 3.4.

In EB, only external HR images are involved. The input of EB are patches
randomly selected from different external HR images. Conversely, the external
patch $\mathbf{x}_{e}\sim p_{\mathcal{X}}$ is firstly degraded by $G_{d}$ to a
fake LR patch, . As the weights of $G_{d}$ are shared between IB and EB, the
external patches are actually degraded by the learned degradation. Thus, the
paired HR and fake LR patches could help $G_{r}$ learn to super resolve LR
images with specific degradations.

According to the above analysis, the loss functions of IB and EB can be
formulated as:

$\mathcal{L}_{IB}=\mathbb{E}_{\mathbf{y}\sim
p_{\mathcal{Y}}}\|\mathbf{y}-G_{d}(G_{r}(\mathbf{y};\theta_{G_{r}});\theta_{G_{d}})\|_{1},$
(6) $\mathcal{L}_{EB}=\mathbb{E}_{\mathbf{x}_{e}\sim
p_{\mathcal{X}}}\|\mathbf{x}_{e}-G_{r}(G_{d}(\mathbf{x}_{e};\theta_{G_{d}});\theta_{G_{r}})\|_{1}.$
(7)

Since information in the single test LR image is limited, to help $G_{d}$
better learn the specific degradation, we further adopt the adversarial
learning strategy. As shown in Figure 3, we introduce a discriminator $D_{l}$.
$D_{l}$ is used to discriminate the distribution characteristics of the LR
image. It could force $G_{d}$ to generate fake LR patches that are more
similar to the real ones. Thus more accurate degradations could be learned by
$G_{d}$. We use the GAN formulation as follows,

$\displaystyle\mathcal{L}_{GAN}=\mathbb{E}_{\mathbf{y}\sim
p_{\mathcal{Y}}}[logD_{l}(\mathbf{y};\theta_{D_{l}})]+\mathbb{E}_{\mathbf{x}_{e}\sim
p_{X}}[log(1-D_{l}(G_{d}(\mathbf{x}_{e};\theta_{G_{d}});\theta_{D_{l}}))].$
(8)

Adversarial training is not used for the intermediate output
$G_{r}(\mathbf{y})$, because it may lead $G_{r}(\mathbf{y})$ to generate
unrealistic textures [3].We also experimentally explain this problem in
Section 4.4.3.

###  3.4 Separate Optimization

Generally, most SR networks are optimized by the weighted sum of all
objectives. All modules in an SR network are treated indiscriminately. Unlike
this commonly used joint optimization method, we propose a separate
optimization strategy. Specifically, $G_{d}$ is optimized by the objectives
that are directly related to the test LR image, while $G_{r}$ is optimized by
objectives that are related to external HR images. The losses for these two
modules are as follows,

$\mathcal{L}_{G_{d}}=\mathcal{L}_{IB}+\lambda\mathcal{L}_{GAN}$ (9)
$\mathcal{L}_{G_{r}}=\mathcal{L}_{EB}$ (10)

where $\lambda$ controls the relative importance of the two losses.

Algorithm 1 Algorithm of ONSR

1:The LR image to be reconstructed: $I_{LR}$

2: The external HR image dataset: $\boldsymbol{S}_{HR}$

3: Maximum updating step: $T$

4: Online testing step interval: $t$

5:The best SR image: $\boldsymbol{I}_{SR}=G_{r}(I_{LR};\ g_{r})$

6:Load the pretrained model $G_{r}$

7:$i=0$

8:while $i\leq T$ do

9: $i\leftarrow{i+1}$

10: Sample $n$ LR image patches ${{\mathbf{\\{y}^{j}\\}}}_{j=1}^{n}$ from
$I_{LR}$

11: Sample $n$ HR image patches ${{\mathbf{\\{x}^{j}_{e}\\}}}_{j=1}^{n}$ from
$\boldsymbol{S}_{HR}$

12: // Online testing

13: if $i\%t==0$ then

14: $I_{SR}^{i}=G_{r}^{i}(I_{LR};\ g_{r}^{i})$

15: end if

16: // Online updating different modules

17: Update
$\theta_{G_{d}}\leftarrow\theta_{G_{d}}-(\nabla_{\theta_{G_{d}}}\mathcal{L}_{IB}+\lambda\nabla_{\theta_{G_{d}}}\mathcal{L}_{GAN})$

18: Update
$\theta_{G_{r}}\leftarrow\theta_{G_{r}}-\nabla_{\theta_{G_{r}}}\mathcal{L}_{EB}$

19: Update
$\Delta\theta_{D_{l}}\leftarrow\theta_{D_{l}}-\lambda\nabla_{\theta_{D_{l}}}\mathcal{L}_{GAN}$

20:end while

We adopt this separate optimization strategy for two reasons. Firstly, as the
analysis in Section 3.2 that $G_{d}$ and $G_{r}$ are alternately optimized in
ONSR, separate optimization may make these modules easier to converge [1].
Secondly, $G_{d}$ aims to learn the specific degradation of the test image,
while $G_{r}$ needs to learn the general priors from external HR images. Thus
it is more targeted for them to be separately optimized. We experimentally
prove the superiority of separate optimization in Sec 4.4.4. The overall
algorithm is shown in Algorithm 1.

Figure 4: The architecture and initialization of $G_{d}$.

###  3.5 Network Instantiation

Most existing SR structures can be used as $G_{r}$ and integrated into ONSR.
In this paper, we mainly use Residual-in-Residual Dense Block (RRDB) [3]. RRDB
combines the multi-level residual network and dense connections, which is easy
to be trained and has promising performance on SR. $G_{r}$ consists of 23
RRDBs and an upsampling module. It is initialized using the pre-trained
network parameters, which could render additional priors of external data, and
also provide a comparatively reasonable initial point to accelerate
optimization.

The architecture of the degradation module $G_{d}$ is shown in Figure 4. We
can see that blurring and downsampling are linear transforms in Eq. 1, so
$G_{d}$ is designed as a deep linear network. Theoretically, a single
convolutional layer should be able to represent all possible downsampling blur
methods in Eq. 1. However, linear networks have infinitely many equal global
minimums according to [28], which makes the gradient-based optimization faster
for deeper linear networks than shallower ones. Thus, we employ three
convolutional layers with no activations and a bicubic downsampling layer in
$G_{d}$. The bicubic downsampling layer could help obtain a reasonable initial
point, which is similar to that in [14] but simpler. Additionally, to
accelerate the convergence of $G_{d}$, we use isotropic Gaussian kernels with
a standard deviation of 1 to initialize all convolutional layers, as shown in
Figure 4. Considering that images with larger downsampling factor are usually
more seriously degraded, we set the size of the three convolutional layers to
$3\times 3$, $7\times 7$, $9\times 9$ for scale factor $\times 2$, and
$9\times 9$, $15\times 15$, $17\times 17$ for scale factor $\times 4$.

$D_{l}$ is a VGG-style network [29] to perform discrimination. The input size
of $D_{l}$ is $32\times 32$.

Figure 5: Visual comparison of ONSR and SotA SR methods for $2\times$ SR. The
model name is denoted above the corresponding patch and PSNR/SSIM is denoted
below.

Figure 6: Visual comparison of ONSR and SotA SR methods for $4\times$ SR. The
model name is denoted above the corresponding patch and PSNR/SSIM is denoted
below.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. We use 800 HR images from the training set of DIV2K [30] as the
external HR dataset and evaluate the SR performance on DIV2KRK [14]. LR images
in DIV2KRK are generated by blurring and subsampling each image from the
validation set (100 images) of DIV2K with randomly generated kernels. These
kernels are isotropic or anisotropic Gaussian kernels with random lengths
$\lambda_{1},\lambda_{2}\sim\mathcal{U}(0.6,5)$ independently distributed for
each axis, rotated by a random angle $\theta\sim\mathcal{U}[-\pi,\pi]$. To
deviate from a regular Gaussian kernel, uniform multiplicative noise (up to
$25\%$ of each pixel value of the kernel) is further applied.

Evaluation Metrics. To quantitatively compare the SR performance different
methods, we use PSNR, SSIM [31], Perceptual Index (PI) [32] and Learned
Perceptual Image Patch Similarity(LPIPS) [33]. Contrary to PSNR and SSIM,
lower PI and LPIPS indicate higher perceptual quality.

Training Details. We randomly sample 10 patches of $32\times 32$ from the LR
image and 10 patches of $32s\times 32s$ from different HR images for each
input minibatch, where $s$ denotes the scaling factor. ADAM [34] optimizer
with $\beta_{1}=0.9,\beta_{2}=0.999$ is used for optimization. We set the
online updating step to 500 for each image, and the LR image is tested every
10 steps. To accelerate the optimization, we initialize ONSR with the
bicubically pretrained model of RRDB, which is publicly available.

Table 2: Quantitative comparison on DIV2KRK dataset. $\times 2$ and $\times 4$
denote the scale factors. $\uparrow$ denotes the larger the better.
$\downarrow$ denotes the smaller the better. Best and second best results are
highlighted and underlined.

Type | Method |  $\times$2 |  |  $\times$4
---|---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | PI $\downarrow$ | LPIPS $\downarrow$ |  | PSNR $\uparrow$ | SSIM $\uparrow$ | PI $\downarrow$ | LPIPS $\downarrow$
Type 1: Non-BlindSR | Bicubic | 28.81 | 0.8090 | 6.7039 | 0.3609 |  | 25.46 | 0.6837 | 8.6414 | 0.5572
ZSSR [9] | 29.09 | 0.8215 | 6.2707 | 0.3252 |  | 25.61 | 0.6920 | 8.1941 | 0.5192
ESRGAN [3] | 29.18 | 0.8212 | 6.1826 | 0.3178 |  | 25.57 | 0.6906 | 8.3554 | 0.5266
RRDB [3] | 29.19 | 0.8224 | 6.4801 | 0.3376 |  | 25.66 | 0.6937 | 8.5510 | 0.5416
RCAN [2] | 27.94 | 0.7885 | 6.8855 | 0.3417 |  | 24.75 | 0.6337 | 8.4560 | 0.5830
Type 2: BlindSR | Cornillere _et al_. [12] | 29.42 | 0.8459 | 4.8343 | 0.1957 |  | - | - | - | -
dSRVAE [13] | - | - | - | - |  | 25.07 | 0.6553 | 5.7329 | 0.4664
Ji _et al_. [11] | - | - | - | - |  | 25.41 | 0.6890 | 8.2348 | 0.5219
KernelGAN+ZSSR [14] | 29.93 | 0.8548 | 5.2483 | 0.2430 |  | 26.76 | 0.7302 | 7.2357 | 0.4449
| ONSR (Ours) | 31.34 | 0.8866 | 4.7952 | 0.2207 |  | 27.66 | 0.7620 | 7.2298 | 0.4071

###  4.2 Super-Resolution on Synthetic Data

We compare ONSR with other state-of-the-art (SotA) methods on the synthetic
dataset DIV2KRK. We present two types of algorithms for analysis: 1) Type1
includes ESRGAN [3], RRDB [3], RCAN [2] and ZSSR [9]. They are non-blind SotA
SR methods trained on bicubically downsampled images. 2) Type 2 are blind SR
methods including KernelGAN+ZSSR [14], dSRVAE [13], Ji _et al_. [11] and
Cornillere _et al_. [12].

Quantitative Results. SotA non-bind SR methods such as ESRGAN and RRDB have
remarkable performance on bicubically downscaled images. However, as shown in
Table 2 Type 1, when tested on DIV2KRK, they perform only slightly better than
the naive bicubic interpolation. And RCAN is even worse. The performance drop
suggests that these methods fail to generalize to the test images with various
degradations in DIV2KRK. As a comparison, our ONSR outperforms these non-blind
SR methods by a large margin. Specifically, RRDB shares the same network
architectures with $G_{r}$ in ONSR, while ONSR outperforms it by 2.15 dB and 2
dB for scales $\times 2$ and $\times 4$ respectively. This comparison
demonstrates the effectiveness of the online updating strategy adopted by
ONSR.

As shown in Table 2 Type 2, ONSR also shows its superiority over other blind
SR methods. KernelGAN+ZSSR is a blind SR method that also adopts an online
updating strategy. However, as we have discussed in Sec 3.1, it does not
exploit information from the external dataset, which may restrict its
performance. As a result, ONSR outperforms it by 1.41 dB and 0.90 dB for
scales $\times 2$ and $\times 4$ respectively. This comparison indicates that
ONSR successfully integrates information from both internal and external
branches and achieves better SR performance.

Qualitative Results. In Figure 5 and 6, we present visual comparisons between
these methods on scales $\times 2$ and $\times 4$ respectively. As shown in
these figures, the images produced by SotA non-blind SR methods are not sharp
enough. RCAN even changes the colors of the original images. The results
produced by blind SR methods are relatively sharper, but they also tend to
contain undesirable artifacts and distortions, such as the window contours in
image 046. As a comparison, the SR images produced by ONSR are cleaner and
more visually favorable.

Figure 7: Visual comparison of model adaptation to the real image “Chip” and
real-world video frames (from YouTube) for $4\times$ SR.

###  4.3 Super-Resolution on Real-World Data

To further demonstrate the effectiveness of ONSR in real scenarios, we conduct
experiments on real images, which are more challenging due to the complicated
and unknown degradations. Since there are no ground-truth HR images for
calculating quantitative metrics, we only provide visual comparisons. As shown
in Figure 7, the letter “X” in the real image “Chip” restored by RRDB, RCAN,
Ji _et al_. and ZSSR are blurry or have unpleasant artifacts. As a comparison,
the super-resolved image of ONSR has more shaper edges and is more visually
pleasing. We also apply these methods to video frames from YouTube
111https://www.youtube.com. As shown in Figure 7, the generated $4\times$ SR
frames from most methods are seriously blurred or contain numerous mosaics.
While ONSR can produce visually promising images with clearer edges and fewer
artifacts. This comparison further demonstrates the robustness of ONSR against
various degradations in real scenarios.

Figure 8: PSNR and visual results of ONSR with $G_{r}$ is initialized (a)
randomly, (b) by model trained on data synthesized by bicubic downsampling,
and (c) by model trained on data synthesized by multiple Gaussian kernels.
Table 3: Quantitative comparison between ONSR with other SotA blind SR
methods. The $G_{r}$ in ONSR is initialized with RRDB-G, in which case our
method is denoted as ONSR-G. Results on DIV2KRK are reported. Best and second
best results are highlighted and underlined

Method | Scale | PSNR $\uparrow$ | SSIM $\uparrow$ | PI$\downarrow$ | LPIPS$\downarrow$
---|---|---|---|---|---
IKC [5] | $\times$2 | 31.20 | 0.8767 | 5.1511 | 0.2350
DASR [6] | 30.72 | 0.8606 | 5.3947 | 0.2501
RRDB-G [3] | 31.18 | 0.8763 | 4.8995 | 0.2213
ONSR-G (Ours) | 31.69 | 0.8907 | 4.6036 | 0.1947
KernelGAN [14] \+ USRNet [1] | $\times$4 | 24.32 | 0.6617 | 8.4425 | 0.5413
IKC [5] | 27.69 | 0.7657 | 6.9027 | 0.3863
DASR [6] | 27.48 | 0.7549 | 7.2024 | 0.4027
RRDB-G [3] | 27.73 | 0.7660 | 6.8767 | 0.3834
ONSR-G (Ours) | 28.05 | 0.7775 | 6.7716 | 0.3781

###  4.4 Ablation Study

####  4.4.1 Study on the initialization of $G_{r}$

As we have discussed above, the SR module $G_{r}$ in our ONSR is usually
initialized with pretrained model weights to reduce the updating steps. In
this section, we experimentally investigate the influence of different
initialization methods. Two initialization methods are firstly considered: a)
random initialization, and b) model pretrained on data synthesized by bicubic
downsampling. As shown in Figure 8 (a) and (b), ONSR can converge well under
both initialization, which demonstrates the good convergence of ONSR. In the
meanwhile, carefully initialized $G_{r}$ (case (b)) helps ONSR converge faster
and has a better optimum point. It indicates that more powerful initialized
$G_{r}$ may further improve the performance of ONSR.

Recently, some methods (such as IKC [5] and DASR [6]) are proposed to train
the SR model with data synthesized by multiple Gaussian kernels to improve its
robustness. This strategy has proved to be effective in training an accurate
model for blind SR [35]. To explore the performance of ONSR when $G_{r}$ is
initialized with such a powerful model, we also train an RRDB on data
synthesized by multiple isotropic Gaussian kernels, which is denoted as
RRDB-G. We initialize $G_{r}$ with RRDB-G, in which case our method is denoted
as ONSR-G. As shown in Figure 8 (c), ONSR-G can still further improve the
performance of the initialization model from $27.80$ dB to a better point of
$28.05$ dB. As shown in Table 3, ONSR-G outperforms IKC by about
$\mathbf{+0.4}$ dB on PSNR and $+\mathbf{0.012}$ on SSIM for scale factors
$\times 2$ and $\times 4$ respectively. The visual comparisons are shown in
Figure 9. As one can see, the results produced by ONSR-G are clearer and more
visually favorable. This comparison indicates that our method can easily enjoy
the merits of other powerful SR methods by taking them as the initialization
models.

Figure 9: Visual comparison of ONSR-G and other methods for $\times 4$ SR
(image 026 and image 099).  Table 4: The performance of ONSR with $G_{r}$ of
different architectures. ON-$Arch$ denotes ONSR with the $G_{r}$ of
architecture named $Arch$. Results are reported on DIV2KRK with scale factor
$\times 4$.

Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PI$\downarrow$ |  LPIPS$\downarrow$
---|---|---|---|---
RDN [16] |  25.66 |  0.6935 |  8.5341 |  0.5411
ON-RDN |  27.30 |  0.7498 |  7.4274 |  0.4377
RCAN [2] |  24.75 |  0.6337 |  8.4560 |  0.5830
ON-RCAN |  27.58 |  0.7612 |  7.1290 |  0.4020

####  4.4.2 Study on the architecture of $G_{r}$

In the experiments above, we only use RRDB as $G_{r}$ in our ONSR. In this
subsection, we experimentally prove that the proposed method works well for
$G_{r}$ of different architectures. Specifically, we perform experiments on
$G_{r}$ of two other architectures, _i.e_. RDN [16] and RCAN [2]. As shown in
Table 4, the original RDN and RCAN do not perform well on DIV2KRK. This is
because their original models are trained on data synthesized by bicubic
downsampling and can not well generalize to test images in DIV2KRK. As a
comparison, Our online updating strategy can largely improve the performance
of both models (denoted as ON-RDN and ON-RCAN respectively). For example, as
shown in 4, the online updating strategy improves the PSNR results of RDN and
RCAN by $\mathbf{+1.64}$ dB and $\mathbf{+2.73}$ dB respectively. It indicates
that the online updating strategy of ONSR works well for $G_{r}$ of different
architectures.

Figure 10: Details of different variants of ONSR.

####  4.4.3 Study on different modules in ONSR

To explain the roles of different modules (_i.e_. IB, EB and $D_{l}$) played
in ONSR, we study four different variants of ONSR, which are denoted as IBSR,
EBSR, IB-EBSR, and IB-EB-GSR. The details of these variants are shown in
Figure 10 and explained below:

IBSR. IBSR only has an internal branch to exploit the internal properties of
the test LR image for degradation estimation and SR reconstruction, which is
optimized online.

EBSR. Contrary to IBSR, EBSR only has an external branch to capture general
priors of external HR images, which is optimized offline. After offline
training, we use the fixed module $G_{r}$ to test LR images.

IB-EBSR. IB-EBSR has both internal branch and external branch but no GAN
modules.

IB-EB-GSR. IB-EB-GSR has both $D_{l}$ and $D_{h}$ to explore the underlying
distribution characteristics of the test LR and external HR images.

Table 5: Comparison of different variants of ONSR. Results are reported on
DIV2KRK.

Method |  Scale |  PSNR |  SSIM |  Scale |  PSNR |  SSIM
---|---|---|---|---|---|---
IBSR |  $\times$2 |  28.05 |  0.8277 |  $\times$4 |  25.51 |  0.6976
EBSR |  30.82 |  0.8806 |  26.56 |  0.7249
IB-EBSR |  31.10 |  0.8850 |  27.60 |  0.7609
IB-EB-GSR |  31.29 |  0.8859 |  27.34 |  0.7507
ONSR |  31.34 |  0.8866 |  27.66 |  0.7620

The quantitative comparisons on DIV2KRK are shown in Table 5. As one can see,
IB-EBSR outperforms both IBSR and EBSR by a large margin. It indicates that
both IB and EB are important for SR performance. The performance of IB-EBSR
could be further improved if $D_{l}$ is introduced. It suggests that
adversarial training can help $G_{r}$ to be better optimized. However, when
$D_{l}$ and $D_{h}$ are both added in IB-EB-GSR, the performance is inferior
to ONSR. In IB-EB-GSR, the initial SR results of $G_{r}(\mathbf{y})$ are
likely to have unpleasant artifacts or distortions. Besides, the external HR
image $\mathbf{x}_{e}$ can not provide directly pixelwise supervision to
$G_{r}(\mathbf{y})$. Therefore, the application of $D_{h}$ may affect the
optimization of IB-EB-GSR and make it inferior to ONSR.

####  4.4.4 Study on separate optimization

As we have mentioned in Sec 3.4, we adopt separate optimization instead of the
typically used joint optimization in our ONSR. In this section, we
experimentally compare the two optimization strategies. In separate
optimization, $G_{d}$ and $G_{r}$ are alternately optimized via the test LR
image and external HR images respectively. While in joint optimization, both
modules are optimized together. As shown in Table 6, separate optimization
surpasses the joint optimization in all metrics for scale factors $\times 2$
and $\times 4$. We also compare the convergence of these two optimization
strategies. We plot the PSNR and SSIM results of the two strategies every
$100$ steps. As shown in Figure 11, the results of separate optimization are
not only higher but also grow faster than that of joint optimization. It
suggests that separate optimization could not only help the network converge
faster, but also help it converge to a better optimum point. It may be because
that separate optimization divides the original highly complex problem into
two simpler subproblems, and both of them are easier to be solved. Thus
separate optimization could converge faster and reach a better optimum point.
This property of separate optimization allows us to make a trade-off between
SR effectiveness and efficiency by setting different training iterations.

Table 6: Comparisons between separate optimization and joint optimization.
Results are reported on DIV2KRK.

Method |  Scale |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  PI$\downarrow$ |  LPIPS$\downarrow$
---|---|---|---|---|---
Joint Optimization |  $\times$2 |  31.03 |  0.8827 |  4.8759 |  0.2212
Separate Optimization |  31.34 |  0.8860 |  4.7952 |  0.2207
Joint Optimization |  $\times$4 |  26.97 |  0.7399 |  7.5985 |  0.4445
Separate Optimization |  27.66 |  0.7620 |  7.2298 |  0.4071

Figure 11: The average PSNR (left) and SSIM (right) of Joint Optimization and
Separate Optimization for $4\times$ SR in differente training steps.

####  4.4.5 Study on the External data

In this section, we perform experiments to study the influence of the external
HR data. Firstly, we investigate how the number of external HR images will
influence SR performance. we randomly sample 200, 400, 600, 700, and all 800
HR images from the training set of DIV2K as the external data and perform
$4\times$ SR on DIV2KRK. As shown in Table 7, the performance of ONSR is
continuously improved as the number of HR images increases. It indicates that
more external HR images could help ONSR learn better general priors. Secondly,
we investigate the influence of different external HR datasets. We use another
popular dataset Flickr2K that consists of 2650 HR images as the external
dataset. As can be seen from Table 7, the SR performance achieved with DIV2K
is comparable to that with Flickr2K. Therefore, the SR performance tends to be
stable when the number of external HR images is large enough, and 800 HR
images from DIV2K could be sufficient for one test LR image.

Table 7: Performance of ONSR when different number of external HR images are
used. Results are reported on DIV2KRK.

# External HR | 200 | 400 | 600 | 700 | 800 (DIV2K) | 2650 (Flickr2K)
---|---|---|---|---|---|---
PSNR | 26.93 | 26.97 | 27.58 | 27.60 | 27.66 | 27.65
SSIM | 0.7390 | 0.7399 | 0.7606 | 0.7610 | 0.7620 | 0.7629

###  4.5 Non-Blind Setting

In this subsection, we explore the upper boundary of ONSR in a non-blind
setting. We replace $G_{d}$ in ONSR with the ground-truth blur kernel (denoted
as ONSR-NonBlind). Following the setting in [1], we evaluate our methods on
BSD68 [36], and 12 representative and diverse blur kernels are used to
synthesize test LR images, including 4 isotropic Gaussian kernels with
different widths, 4 anisotropic Gaussian kernels from [10], and 4 motion blur
kernels from [37, 38]. The quantitative results are shown in Table 8. As one
can see, under the non-blind setting, ONSR achieves the best performance among
the reference methods. We need to note that USRNet [1] is a SotA non-blind
method, while ONSR still outperforms it by a large margin on all 12 kernels.
For example, under the first blur kernel, ONSR outperforms USRNet by
$+\mathbf{1.11}$ dB and $+\mathbf{1.21}$ dB for scale factors $\times 2$ and
$\times 4$ respectively. It demonstrates that the online updating strategy of
ONSR also works well in a non-blind setting.

Table 8: Average PSNR results of non-blind setting for $4\times$ SR. Best and
second best results are highlighted and underlined.

Method | Scale |  |  |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---
EDSR [39] | $\times$2 | 25.54 | 27.82 | 20.59 | 21.34 | 27.66 | 27.28 | 26.90 | 26.07 | 27.14 | 26.96 | 19.72 | 19.86
RCAN [2] | 29.48 | 26.76 | 25.31 | 24.37 | 24.38 | 24.10 | 24.25 | 23.63 | 20.31 | 20.45 | 20.57 | 22.04
ZSSR [9] | 29.44 | 29.48 | 28.57 | 27.42 | 27.15 | 26.81 | 27.09 | 26.25 | 14.22 | 14.22 | 16.02 | 19.39
IRCNN [40] | 29.60 | 30.16 | 29.50 | 28.37 | 28.07 | 27.95 | 28.21 | 27.19 | 28.58 | 26.79 | 29.02 | 28.96
USRNet [1] | 30.55 | 30.96 | 30.56 | 29.49 | 29.13 | 29.12 | 29.28 | 28.28 | 30.90 | 30.65 | 30.60 | 30.75
ONSR-NonBlind | 31.66 | 31.98 | 31.40 | 30.17 | 29.76 | 29.63 | 29.86 | 28.87 | 30.93 | 30.78 | 30.80 | 31.12
EDSR [39] | $\times$4 | 21.45 | 22.73 | 21.60 | 20.62 | 23.16 | 23.66 | 23.16 | 23.00 | 24.00 | 23.78 | 19.79 | 19.67
RCAN [2] | 22.68 | 25.31 | 25.59 | 24.63 | 24.37 | 24.23 | 24.43 | 23.74 | 20.06 | 20.05 | 20.33 | 21.47
ZSSR [9] | 23.50 | 24.33 | 24.56 | 24.65 | 24.52 | 24.20 | 24.56 | 24.55 | 16.94 | 16.43 | 18.01 | 20.68
IRCNN [40] | 23.99 | 25.01 | 25.32 | 25.45 | 25.36 | 25.26 | 25.34 | 25.47 | 24.69 | 24.39 | 24.44 | 24.57
USRNet [1] | 25.30 | 25.96 | 26.18 | 26.29 | 26.20 | 26.15 | 26.17 | 26.30 | 25.91 | 25.57 | 25.76 | 25.70
ONSR-NonBlind | 26.51 | 27.24 | 27.50 | 27.57 | 27.43 | 27.30 | 27.36 | 27.51 | 26.17 | 26.17 | 26.21 | 26.30

###  4.6 Speed Comparison

We test the speed of different blind SR methods to compare their overall
performance in terms of effectiveness and efficiency. We evaluate their
average running time on DIV2KRK for $\times 2$ on the same machine with an
NVIDIA 2080Ti GPU. Since IKC [5] and DASR [6] are included in the referring
methods, we report the performance of ONSR-G for fair comparisons. We need to
note that the running time of ONSR-G is closely related to its online updating
steps. In previous experiments, the steps are set as 500 for best performance.
However, as we have discussed in Sec 4.4.4, due to the good convergence of
ONSR-G, we can set fewer steps to make a balance between accuracy and speed.
Thus, we also test the performance of ONSR-G when steps are 5, 10, and 100 for
comparisons. We use ONSR-G-$S$ to denote ONSR-G with updating steps as $S$. As
shown in Table 9, ONSR-G-5 outperforms KernelGAN + ZSSR [14] and Cornillere
_et al_. by $+\mathbf{4.39}$ dB and $+\mathbf{1.73}$ dB while with
$\mathbf{174}$ times and $21$ times faster speed respectively. When compared
with IKC, ONSR-G-10 also achieves a similar PSNR result with a comparable
speed. DASR is much faster than the other methods, but its PNSR result is
suboptimal. On the whole, ONSR-G can achieve competing performance with the
most recent blind SR methods. Moreover, the good convergence of ONSR allows us
to easily adjust its running time according to different scenarios.

Table 9: Average inference time comparison on DIV2KRK for $\times 2$ SR.
ONSR-G-$S$ denotes ONSR-G with updating steps as $S$.

Method |  Speed(s/image) |  PSNR(dB)
---|---|---
KernelGAN + ZSSR [14] |  1127.84 |  26.76
Cornillere _et al_. [12] |  135.51 |  29.42
IKC [5] |  7.14 |  31.20
DASR [6] |  0.24 |  30.72
ONSR-G-5 (Ours) |  6.47 |  31.15
ONSR-G-10 (Ours) |  8.00 |  31.21
ONSR-G-100 (Ours) |  75.29 |  31.61

##  5 Conclusion and Future Work

In this paper, we argue that most nowadays SR methods are not image-specific.
Towards the limitation, we propose an online SR (ONSR) method, which could
customize a specific model for each test image. In detail, we design two
branches, namely internal branch (IB) and external branch (EB). IB could learn
the specific degradation of the test image, and EB could learn to super
resolve images that are degraded by the learned degradation. IB involves only
the LR image, while EB uses external HR images. In this way, ONSR could
leverage the benefits of both inherent information of the test LR image and
general priors from external HR images. Extensive experiments on both
synthetic and real-world images prove the superiority of ONSR in the blind SR
problem. These results indicate that customizing a model for each test image
is more practical in real applications than training a general model for all
LR images. Moreover, the speed of ONSR may be further improved by designing
more lightweight modules for faster inference or elaborating the training
strategy to accelerate convergence. Faster speed can help it to be more
practical when processing large amounts of test images, such as videos of low
resolution, which is also the focus of our future work.

## References

  * [1] K. Zhang, L. V. Gool, R. Timofte, Deep unfolding network for image super-resolution, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3217–3226.
  * [2] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, Y. Fu, Image super-resolution using very deep residual channel attention networks, in: Proceedings of the European Conference on Computer Vision, 2018, pp. 286–301.
  * [3] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, C. Change Loy, Esrgan: Enhanced super-resolution generative adversarial networks, in: Proceedings of the European Conference on Computer Vision Workshops, 2018, pp. 0–0.
  * [4] Y. Yang, Y. Qi, Image super-resolution via channel attention and spatial graph convolutional network, Pattern Recognition 112 (2021) 107798.
  * [5] J. Gu, H. Lu, W. Zuo, C. Dong, Blind super-resolution with iterative kernel correction, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1604–1613.
  * [6] L. Wang, Y. Wang, X. Dong, Q. Xu, J. Yang, W. An, Y. Guo, Unsupervised degradation representation learning for blind super-resolution, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10581–10590.
  * [7] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, Advances in neural information processing systems 27.
  * [8] C. Dong, C. C. Loy, K. He, X. Tang, Learning a deep convolutional network for image super-resolution, in: European Conference on Computer Vision, Springer, 2014, pp. 184–199.
  * [9] A. Shocher, N. Cohen, M. Irani, “zero-shot” super-resolution using deep internal learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 3118–3126.
  * [10] K. Zhang, W. Zuo, L. Zhang, Learning a single convolutional super-resolution network for multiple degradations, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 3262–3271.
  * [11] X. Ji, Y. Cao, Y. Tai, C. Wang, J. Li, F. Huang, Real-world super-resolution via kernel estimation and noise injection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 466–467.
  * [12] V. Cornillere, A. Djelouah, W. Yifan, O. Sorkine-Hornung, C. Schroers, Blind image super-resolution with spatially variant degradations, ACM Transactions on Graphics (TOG) 38 (6) (2019) 1–13.
  * [13] Z.-S. Liu, W.-C. Siu, L.-W. Wang, C.-T. Li, M.-P. Cani, Unsupervised real image super-resolution via generative variational autoencoder, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 442–443.
  * [14] S. Bell-Kligler, A. Shocher, M. Irani, Blind super-resolution kernel estimation using an internal-gan, in: NeurIPS, 2019.
  * [15] N. Ahn, J. Yoo, K.-A. Sohn, Simusr: A simple but strong baseline for unsupervised image super-resolution, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 474–475.
  * [16] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, Y. Fu, Residual dense network for image super-resolution, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 2472–2481.
  * [17] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  * [18] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
  * [19] R. Chen, H. Zhang, J. Liu, Multi-attention augmented network for single image super-resolution, Pattern Recognition 122 (2022) 108349.
  * [20] Y. Wang, L. Wang, H. Wang, P. Li, H. Lu, Blind single image super-resolution with a mixture of deep networks, Pattern Recognition 102 (2020) 107169.
  * [21] Y. Liang, R. Timofte, J. Wang, S. Zhou, Y. Gong, N. Zheng, Single-image super-resolution-when model adaptation matters, Pattern Recognition (2021) 107931\.
  * [22] T. Michaeli, M. Irani, Nonparametric blind super-resolution, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2013, pp. 945–952.
  * [23] D. Ulyanov, A. Vedaldi, V. Lempitsky, Deep image prior, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 9446–9454.
  * [24] D. Ren, K. Zhang, Q. Wang, Q. Hu, W. Zuo, Neural blind deconvolution using deep priors, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3341–3350.
  * [25] J. Pan, D. Sun, H. Pfister, M.-H. Yang, Blind image deblurring using dark channel prior, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016, pp. 1628–1636.
  * [26] J. Pan, Z. Hu, Z. Su, M.-H. Yang, Deblurring text images via l0-regularized intensity and gradient prior, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2014, pp. 2901–2908.
  * [27] T. Michaeli, M. Irani, Blind deblurring using internal patch recurrence, in: European Conference on Computer Vision, Springer, 2014, pp. 783–798.
  * [28] S. Arora, N. Cohen, E. Hazan, On the optimization of deep networks: Implicit acceleration by overparameterization, in: International Conference on Machine Learning, PMLR, 2018, pp. 244–253.
  * [29] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, CoRR abs/1409.1556.
  * [30] E. Agustsson, R. Timofte, Ntire 2017 challenge on single image super-resolution: Dataset and study, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 126–135.
  * [31] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE transactions on image processing 13 (4) (2004) 600–612.
  * [32] Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, L. Zelnik-Manor, The 2018 pirm challenge on perceptual image super-resolution, in: Proceedings of the European Conference on Computer Vision Workshops, 2018, pp. 0–0.
  * [33] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang, The unreasonable effectiveness of deep features as a perceptual metric, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 586–595.
  * [34] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980.
  * [35] L. Xie, X. Wang, C. Dong, Z. Qi, Y. Shan, Finding discriminative filters for specific degradations in blind super-resolution, Advances in Neural Information Processing Systems 34.
  * [36] D. Martin, C. Fowlkes, D. Tal, J. Malik, A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE, 2001.
  * [37] G. Boracchi, A. Foi, Modeling the performance of image restoration from motion blur, IEEE Transactions on Image Processing 21 (8) (2012) 3502–3517.
  * [38] A. Levin, Y. Weiss, F. Durand, W. T. Freeman, Understanding and evaluating blind deconvolution algorithms, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2009, pp. 1964–1971.
  * [39] B. Lim, S. Son, H. Kim, S. Nah, K. Mu Lee, Enhanced deep residual networks for single image super-resolution, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 136–144.
  * [40] K. Zhang, W. Zuo, S. Gu, L. Zhang, Learning deep cnn denoiser prior for image restoration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017, pp. 3929–3938.