# Hitchhiker’s Guide to Super-Resolution: Introduction and Recent Advances

Brian Moser1,2, Federico Raue1, Stanislav Frolov1, Jörn Hees1, Sebastian
Palacio1 Andreas Dengel1,2
1 German Research Center for Artificial Intelligence (DFKI), Germany
2 TU Kaiserslautern, Germany
<EMAIL_ADDRESS>
###### Abstract

With the advent of Deep Learning (DL), Super-Resolution (SR) has also become a
thriving research area. However, despite promising results, the field still
faces challenges that require further research e.g., allowing flexible
upsampling, more effective loss functions, and better evaluation metrics. We
review the domain of SR in light of recent advances, and examine state-of-the-
art models such as diffusion (DDPM) and transformer-based SR models. We
present a critical discussion on contemporary strategies used in SR, and
identify promising yet unexplored research directions. We complement previous
surveys by incorporating the latest developments in the field such as
uncertainty-driven losses, wavelet networks, neural architecture search, novel
normalization methods, and the latests evaluation techniques. We also include
several visualizations for the models and methods throughout each chapter in
order to facilitate a global understanding of the trends in the field. This
review is ultimately aimed at helping researchers to push the boundaries of DL
applied to SR.

###### Index Terms:

Computer Science, Artificial Intelligence, Super-Resolution, Deep Learning,
Survey

##  1 Introduction

Super-Resolution (SR) is the process of enhancing Low-Resolution (LR) images
to High-Resolution (HR). The applications range from natural images [1, 2] to
highly advanced satellite [3], and medical imaging [4]. Despite its long
history [5], SR remains a challenging task in computer vision because it is
notoriously ill-posed: several HR images can be valid for any given LR image
due to many aspects like brightness, and coloring [6, 7]. The fundamental
uncertainties in the relation between LR and HR images pose a complex research
task. Thanks to rapid advances in Deep Learning (DL), SR has made significant
progress in recent years. Unfortunately, entry into this field is overwhelming
because of the abundance of publications. It is knotty work to get an overview
of advantages and disadvantages between publications. This work unwinds some
of the most representative advances in the crowded field of SR.

Existing surveys have mostly focus on classical methods [8, 9] whereas we, in
contrast, concentrate exclusively on DL methods given their proven
superiority. The most similar survey to this work is the one by Bashir et al.
[4], which itself is a logical continuation of Wang et al. [5]. However, our
work focuses on methods that have recently gained popularity: uncertainty-
driven loss, advances in image quality assessment methods, new datasets,
denoising diffusion probabilistic models, advances in normalization
techniques, and new architecture approaches. This work also reviews pioneering
work that combines neural architecture search with SR, which automatically
derives designs instead of relying solely on human dexterity [10]. Finally,
this work aims to give a broader overview of the field and highlight
challenges as well as future trends.

Section 2 lays out definitions and introduces known metrics used by most SR
publications. This section also introduces datasets and data types found in SR
research. Section 3 presents commonly used regression-based learning
objectives (pixel, uncertainty-driven, and content loss) as well as more
sophisticated objectives like adversarial loss and denoising diffusion
probabilistic models. Section 4 covers interpolation- and learning-based
upsampling. Section 5 goes over basic mechanisms of attention used in SR.
Section 6 explains additional learning strategies, covering a variety of
techniques that can be applied for performance improvement. It discusses
curriculum learning, enhanced predictions, network fusion, multi-task
learning, and normalization. Section 7 introduces SR models. It goes through
different architecture types, such as simple, residual, recurrent-based,
lightweight, and wavelet transform-based networks. Several graphical
visualizations accompany the chapter, highlighting the difference between the
proposed architectures. Section 8 and Section 9 introduce unsupervised SR and
neural architecture search combined with SR. Finally, this work summarizes and
points to future directions in Section 10 and Section 11.

##  2 Setting and Terminology

This section focuses on four questions: (1) What is SR? (2) How good is a
stated SR solution? (3) What SR datasets are available to test the solution?
(4) How are images represented in SR?

The first question introduces the fundamental definitions as well as the
terminology that is specific to SR. The second relates to evaluation metrics
that assess any proposed SR solution, such as Peak Signal-to-Noise Ratio
(PSNR) and Structural Similarity Index (SSIM). The third question is linked to
numerous datasets that provide various data types, such as 8K resolution
images or video sequences. Last but not least, the fourth question addresses
different image representations (i.e., color spaces) that SR models can use.

###  2.1 Problem Definition: Super-Resolution

Super-Resolution (SR) refers to methods that can develop High-Resolution (HR)
images from at least one Low-Resolution (LR) image [5]. The SR field divides
into Single Image Super-Resolution (SISR) and Multi-Image Super-Resolution
(MISR). In SISR, one LR image leads to one HR image, whereas MISR generates
many HR images from many LR images. Regarding popularity, most researchers
focus on SISR because its techniques are extendable to MISR.

####  2.1.1 Single Image Super-Resolution (SISR)

The goal of Single Image Super-Resolution (SISR) is to scale up a given Low-
Resolution (LR) image $\mathbf{x}\in\mathbb{R}^{\bar{w}\times\bar{h}\times c}$
to a High-Resolution (HR) image $\mathbf{y}\in\mathbb{R}^{w\times h\times c}$,
with $\bar{w}\leq w$ and $\bar{h}\leq h$. Throughout this work,
$N_{\mathbf{x}}=w\cdot h\cdot c$ defines the amount of pixels of an image
$\mathbf{x}\in\mathbb{R}^{w\times h\times c}$ and $\Omega_{\mathbf{x}}$ the
set of all valid positions in $\mathbf{x}$:

$\Omega_{\mathbf{x}}=\\{\left(i,j,k\right)\in\mathbb{N}_{1}^{3}|i\leq h,j\leq
w,k\leq c\\}$ (1)

Let $s\in\mathbb{N}_{1}$ be a scaling factor, it holds that $h=s\cdot\bar{h}$
and $w=s\cdot\bar{w}$. Furthermore, let $\mathcal{D}:\mathbb{R}^{w\times
h\times c}\rightarrow\mathbb{R}^{\bar{w}\times\bar{h}\times c}$ be a
degradation mapping that describes the inherent relationship between the two
entities LR ($\mathbf{x}$) and HR ($\mathbf{y}$):

$\mathbf{x}=\mathcal{D}\left(\mathbf{y};\delta\right),$ (2)

in which $\delta$ are parameters of $\mathcal{D}$ that contain, for example,
the scaling factor $s$ and other elements like blur type. In practice, the
degradation mapping is often unknown and therefore modeled, e.g., with bicubic
downsampling. The underlying challenge of SISR is to perform the inverse
mapping of $\mathcal{D}$. Unfortunately, this problem is ill-posed because one
LR image can lead to multiple nonidentical HR images. The goal is to find a SR
model $\mathcal{M}:\mathbb{R}^{\bar{w}\times\bar{h}\times
c}\rightarrow\mathbb{R}^{w\times h\times c}$, s.t. :

$\mathbf{\hat{y}}=\mathcal{M}\left(\mathbf{x};\theta\right),$ (3)

where $\mathbf{\hat{y}}$ is the predicted HR approximation of the LR image
$\mathbf{x}$ and $\theta$ the parameters of $\mathcal{M}$. For Deep Learning
(DL), this translates into an optimization objective that minimizes the
difference between the estimation $\mathbf{\hat{y}}$ and the ground-truth HR
image $\mathbf{y}$ under a given loss function $\mathcal{L}$:

$\hat{\theta}=\operatorname*{arg\,min}_{\theta}\mathcal{L}\left(\mathbf{\hat{y}},\mathbf{y}\right)$
(4)

The setting is illustrated in the supplementary material.

####  2.1.2 Multi-Image Super-Resolution (MISR)

Multi-Image Super-Resolution (MISR) is the task of yielding one or more HR
images from many LR images [11]. An example is satellite imagery [3], where
many LR examples direct to a single HR prediction, a so-called many-to-one
approach. An alternative is the many-to-many approach, where many LR images
lead to more than one HR image. It is usually employed for video sequence
enhancing [11]. The LR images are generally of the same scene e.g., multiple
satellite images of the same geographical location. Given a sequence
$\mathbf{x}=\left(\mathbf{x}_{1},...,\mathbf{x}_{T}\right)$ with
$T\in\mathbb{N}_{1}$ and
$\mathbf{x}_{t}\in\mathbb{R}^{\bar{w}\times\bar{h}\times c}$, $0<t\leq T$, the
task is to predict
$\mathbf{y}=\left(\mathbf{y}_{1},...,\mathbf{y}_{T^{\prime}}\right)$ with
$T^{\prime}\in\mathbb{N}_{1}$ and
$\mathbf{y}_{t^{\prime}}\in\mathbb{R}^{w\times h\times c}$, $0<t^{\prime}\leq
T^{\prime}$. The most frequent case is $T=T^{\prime}$, where $\mathbf{y}_{t}$
is supposed to be the HR image of $\mathbf{x}_{t}$. Generally, MISR is an
extension of the SISR setting [12].

###  2.2 Evaluation: Image Quality Assessment (IQA)

Many properties are associated to an excellent image quality, such as
sharpness, contrast or the absence of noise. Thus, fair evaluation of SR
models is challenging. This section shows different evaluation methods that
fall under the umbrella term Image Quality Assessment (IQA). Broadly speaking,
IQA refers to any metric that is based on a perceptual assessments of human
viewers i.e., how realistic the image appears after applying SR methods. IQA
can be subjective (e.g., human raters) or objective (e.g., formal metrics).

####  2.2.1 Mean Opinion Score (MOS)

Digital images are ultimately meant to be viewed by human beings. Thus, the
most appropriate way of assessing images is a subjective evaluation [13, 14].
One commonly used subjective IQA method is the Mean Opinion Score (MOS). Human
viewers rate images with quality scores, typically 1 (bad) to 5 (good). MOS is
the arithmetic mean of all ratings. Despite reliability, mobilizing human
resources is time-consuming and cumbersome, especially for large datasets.

####  2.2.2 Learning-based Perceptual Quality (LPQ)

Lately, researchers have tried to mitigate some weak points of MOS by using
DL, the so-called Learning-based Perceptual Quality (LPQ). In essence, LPQ
tries to approximate a variety of subjective ratings by applying DL methods.
One way is to use datasets that contain subjective scores, such as TID2013
[15], and a neural network to predict human rating scores, e.g., DeepQA [16]
or NIMA [17].

A significant drawback of LPQ is the limited availability of annotated
samples. One can augment a small-sized dataset by applying noise and ranking.
Adding minimal noise to an image should lead to poorer quality, making the
noisy image and the original counterpart pairwise discriminable. These pairs
are called Quality-Discriminable Image Pairs (DIP) [18]. The DIP Inferred
Quality (dipIQ) index uses RankNet [19], which is based on a pairwise
learning-to-rank algorithm [20]. The authors of dipIQ show higher accuracy and
robustness in variation-rich content than by training directly on IQA
databases like TID2013 [15]. Another example is RankIQA [21], in which a
Siamese Network [22] is used to rank image quality by using artificial
distortions.

####  2.2.3 Peak Signal-to-Noise Ratio (PSNR)

Objectively assessing image quality is of indisputable importance due to the
massive amount of images produced in recent years and the weaknesses of
subjective measurements. One of the most known objective quality measurements
is Peak Signal-to-Noise Ratio (PSNR). It is the ratio between the maximum
possible pixel-value $L$ (255 for 8-bit representations) and the Mean Squared
Error (MSE) of reference images. Given the approximation $\mathbf{\hat{y}}$
and the ground-truth $\mathbf{y}$, PSNR is a logarithmic quantity using the
decibel scale [dB]:

$\text{PSNR}\left(\mathbf{y},\mathbf{\hat{y}}\right)=10\cdot\log_{10}\left(\frac{L^{2}}{\frac{1}{N_{\mathbf{y}}}\sum_{p\in\Omega_{\mathbf{y}}}\left[\mathbf{y}_{p}-\mathbf{\hat{y}}_{p}\right]^{2}}\right)$
(5)

Although it is the most widely used evaluation criterion for SR models, it
often leads to mediocre reconstructions in real scenarios. It focuses on
pixel-level differences instead of mammalian visual perception, which is more
attracted to structures [23]. Subsequently, it does not correlate well with
subjectively perceived quality [24]. In other words, slight changes in pixels
(e.g., shifting) can lead to a significantly decreased PSNR while humans
barely recognize the difference [25]. Consequently, new metrics focus on more
structural features within an image.

####  2.2.4 Structural Similarity Index (SSIM)

The Structural Similarity Index (SSIM) depends on three relatively independent
entities: luminance, contrast, and structures [26]. It is widely known and
better meets the requirements of perceptual assessment [5]. SSIM estimates for
an image $\mathbf{y}$ the luminance $\mu_{\mathbf{y}}$ as the mean of the
intensity,

$\mu_{\mathbf{y}}=\frac{1}{N_{\mathbf{y}}}\sum_{p\in\Omega_{\mathbf{y}}}\mathbf{y}_{p},$
(6)

while it is estimating contrast $\sigma_{\mathbf{y}}$ as its standard
deviation,

$\sigma_{\mathbf{y}}=\frac{1}{N_{\mathbf{y}}-1}\sum_{p\in\Omega_{\mathbf{y}}}\left[\mathbf{y}_{p}-\mu_{\mathbf{y}}\right]^{2}$
(7)

In order to compare the entities, the authors of SSIM introduced a similarity
comparison function $S$:

$S\left(x,y,c\right)=\frac{2\cdot x\cdot y+c}{x^{2}+y^{2}+c}\,,$ (8)

where $x$ and $y$ are the compared scalar variables, and $c=\left(k\cdot
L\right)^{2}$, $0<k\ll 1$ is a constant to avoid instability. Given a ground-
truth image $\mathbf{y}$ and its approximation $\mathbf{\hat{y}}$, the
comparisons on luminance ($\mathcal{C}_{l}$) and contrast ($\mathcal{C}_{c}$)
are

$\mathcal{C}_{l}\left(\mathbf{y},\mathbf{\hat{y}}\right)=S\left(\mu_{\mathbf{y}},\mu_{\mathbf{\hat{y}}},c_{1}\right)\text{
and
}\mathcal{C}_{c}\left(\mathbf{y},\mathbf{\hat{y}}\right)=S\left(\sigma_{\mathbf{y}},\sigma_{\mathbf{\hat{y}}},c_{2}\right)$
(9)

where $c_{1},c_{2}>0$. The empirical co-variance

$\sigma_{\mathbf{y},\mathbf{\hat{y}}}=\frac{1}{N_{\mathbf{y}}-1}\sum_{p\in\Omega_{\mathbf{y}}}\left(\mathbf{y}_{p}-\mu_{\mathbf{y}}\right)\cdot\left(\mathbf{\hat{y}}_{p}-\mu_{\mathbf{\hat{y}}}\right),$
(10)

determines the structure comparison ($\mathcal{C}_{s}$), expressed as the
correlation coefficient between $\mathbf{y}$ and $\mathbf{\hat{y}}$:

$\mathcal{C}_{s}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\frac{\sigma_{\mathbf{y},\mathbf{\hat{y}}}+c_{3}}{\sigma_{\mathbf{y}}\cdot\sigma_{\mathbf{\hat{y}}}+c_{3}},$
(11)

where $c_{3}>0$. Finally, the SSIM is defined as:

$\text{SSIM}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\left[\mathcal{C}_{l}\left(\mathbf{y},\mathbf{\hat{y}}\right)\right]^{\alpha}\cdot\left[\mathcal{C}_{c}\left(\mathbf{y},\mathbf{\hat{y}}\right)\right]^{\beta}\cdot\left[\mathcal{C}_{s}\left(\mathbf{y},\mathbf{\hat{y}}\right)\right]^{\gamma}$
(12)

where $\alpha>0,\beta>0$ and $\gamma>0$ are adjustable control parameters for
weighting relative importance of all components.

####  2.2.5 Task-based Evaluation (TBE)

Alternatively to SSIM, one can focus on task-oriented features. For instance,
one can derive features from DL models that solve other Computer Vision (CV)
tasks like image classification. Other CV tasks can also benefit from
including SR as a pre-processing step. Measuring the performance of these
tasks, with and without SR, is yet another ingenious way to measure the
quality of an SR method. HR images provide more details, which are highly
desirable for CV tasks like object recognition [3]. Nevertheless, it requires
extra training steps, which can be avoided by using predefined features like
those presented in the following section.

####  2.2.6 Evaluation with defined Features

One example is the Gradient Magnitude Similarity Deviation (GMSD) [27], which
uses the pixel-wise Gradient Magnitude Similarity (GMS). Based on the
variation of local quality, which arises from the diversity of local image
structures, GMS calculates the gradient magnitudes with:

$\mathbf{m}_{\mathbf{x}}\left(p\right)=\sqrt{\left(\nabla_{h}\mathbf{x}\right)^{2}_{p}+\left(\nabla_{v}\mathbf{x}\right)^{2}_{p}},p\in\Omega_{\mathbf{x}}$
(13)

where $\nabla_{h}\mathbf{x}$ and $\nabla_{v}\mathbf{x}$ are the horizontal and
vertical gradient images of $\mathbf{x}$, respectively. The GMS map, similar
to Equation 9, is given by

$\text{GMS}\left(\mathbf{y},\mathbf{\hat{y}}\right)_{p}=S\left(\mathbf{m}_{\mathbf{y}}\left(p\right),\mathbf{m}_{\mathbf{\hat{y}}}\left(p\right),c\right)$
(14)

where $c$ is a positive constant. The final IQA score is given by the average
of the GMS map:

$\text{GMSD}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\frac{1}{N_{\mathbf{y}}}\sum_{p\in\Omega_{\mathbf{y}}}\text{GMS}\left(\mathbf{y},\mathbf{\hat{y}}\right)_{p}$
(15)

An alternative is the Feature Similarity (FSIM) Index. It also uses gradient
magnitudes, but combines them with Phase Congruency (PC), a local structure
measurement, as feature points [28]. The phase congruency model postulates a
biologically plausible model of how human visual systems detect and identify
image features. These features are perceived where the Fourier components are
maximal in phase. An extension to FSIM is the Haar wavelet-based Perceptual
Similarity Index (HaarPSI), which uses a Haar wavelet decomposition to assess
local similarities [29]. Its authors claim that it outperforms state-of-the-
art IQA methods like SSIM and FSIM in terms of execution time and higher
correlations with human opinion scores. They suggest that a multi-scale
complex-valued wavelet filterbank directly influences the computation of FSIM
in the computation of the Fourier components. The approach is similar to FSIM
but relies on 2D discrete Haar wavelet transform.

####  2.2.7 Multi-Scale Evaluation

In practice, SR models usually super-resolve to different scaling factors,
known as Multi-Scaling (MS). Thus, evaluating metrics should address this
scenario. The MS-Structural Similarity (MS-SSIM) index [30] is a direct
extension of SSIM that incorporates MS. Compared to SSIM, it can be more
robust to variations in viewing conditions. It adds parameters that weight the
relative importance of different scales. MS-SSIM applies a low-pass filter and
downsamples the filtered image by a factor of 2 for every scale level $1\leq
i\leq s_{max}$, where $s_{max}$ is the largest scale. By doing so, it
calculates the contrast and structure comparisons at each scale and the
luminance comparison for the largest scale $s_{max}$. MS-SSIM is formulated as

$\text{MS-SSIM}\left(\mathbf{y},\mathbf{\hat{y}}\right)\\\
=\mathcal{C}_{l}\left(\mathbf{y},\mathbf{\hat{y}}\right)^{\alpha}\cdot\prod_{i=1}^{s_{max}}\mathcal{C}_{c,i}\left(\mathbf{y},\mathbf{\hat{y}}\right)^{\beta_{i}}\cdot\mathcal{C}_{s,i}\left(\mathbf{y},\mathbf{\hat{y}}\right)^{\gamma_{i}},$
(16)

where $\alpha\negmedspace>\negmedspace 0$, $\beta_{i}\negmedspace>\negmedspace
0$ and $\gamma_{i}\negmedspace>\negmedspace 0$ are adjustable control
parameters. Similarly, the GMSD can be extended with

$\text{MS-
GMSD}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\sqrt{\sum^{s_{max}}_{i=0}\alpha_{i}\cdot\left(\text{GMSD}_{i}\left(\mathbf{y},\mathbf{\hat{y}}\right)\right)^{2}},$
(17)

where $\alpha_{i}$ are adjustable control parameters and $\text{GMSD}_{i}$ is
the GMSD score at $i_{th}$ scale [31].

###  2.3 Datasets and Challenges

Figure 1:  Example images from different SR datasets: Urban100 [32], T91 [33],
Set5 [34], Set14 [1], Manga109 [35], General100 [36], BSDS100 [2], and BSDS200
[2]. The ratio of the size differences is preserved.

For SR, more extensive datasets have been made available over the past years.
Various accessible datasets contain vast amounts of images of different
qualities and content, as shown in Figure 1. Some are created explicitly for
SR in a supervised manner with LR-HR pairs. Moreover, one can exploit datasets
that do not come with SR annotations by generating LR pairs and treating
original samples as the HR version. LR samples are computed by using bicubic
interpolation, and anti-aliasing [37]. The supplementary material list
typically used datasets. Some of them were published as part of a SR
challenge. Two of the most popular challenges are the New Trends in Image
Restoration and Enhancement (NTIRE) challenge [38], and the Perceptual Image
Restoration and Manipulation (PIRM) challenge [39].

###  2.4 Color Spaces

Data representation (i.e., color space) has played a crucial role for SR
methods before DL. Nowadays, most researchers use the RGB color space [40,
41]. However, researchers have tried to combine the advantages of other color-
spaces. The first DL-based SR model [42] used the first channel of the YCbCr
space. Throughout the literature, it is standard to capture only the Y channel
if YCbCr is employed [43] since focusing on structure instead of color is
preferred. Recent SR models use the RGB color space, also because using
something other than RGB color spaces is ill-defined when comparing state-of-
the-art results if PSNR is used as evaluation metric [44, 45]. Exploring other
color spaces for DL-based SR methods is nearly non-existent, which presents an
interesting research gap.

##  3 Learning Objectives

The ultimate goal of SR is to provide a model that maps an LR image to an HR
image. The question naturally arises: how to train a given SR model? The
answer to this question is given in the following sections.

###  3.1 Regression-based Objectives

Regression-based objectives attempt to model the relation between input and
the desired output explicitly. The parameters of the SR model are estimated
directly from the data, often by minimizing the $L1$ and $L2$ losses. While
both loss functions are easily applicable, the results tend to be blurry. We
also discuss one way to mitigate this shortcoming, namely by modeling
uncertainty into the loss function itself.

####  3.1.1 Pixel Loss

The pixel loss measures the pixel-wise difference, as illustrated in the
supplementary material.There are two well-known pixel loss functions in
literature. The first one is the Mean Absolute Error (MAE), or $L1$-loss:

$\mathcal{L}_{\text{L1}}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\frac{1}{N_{\mathbf{y}}}\sum_{p\in\Omega_{\mathbf{y}}}\left|\mathbf{y}_{p}-\mathbf{\hat{y}}_{p}\right|$
(18)

It takes the absolute differences between every pixel of both images and
returns the mean value. The second well-known pixel loss function is the Mean
Squared Error (MSE), or $L2$-loss. It weights high-value differences higher
than low-value differences due to an additional square operation:

$\mathcal{L}_{\text{L2}}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\frac{1}{N_{\mathbf{y}}}\sum_{p\in\Omega_{\mathbf{y}}}\left|\mathbf{y}_{p}-\mathbf{\hat{y}}_{p}\right|^{2}$
(19)

However, it is more common to see the $L1$-loss in literature because $L2$ is
very reactive to extraordinary values: too smooth for low values and too
variable for high values [46, 41]. There are variants in literature, depending
on the task and other specifications. A popular one is the Charbonnier-loss
[47], which is defined by

$\mathcal{L}_{\text{Charbonnier}}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\frac{1}{N_{\mathbf{y}}}\sum_{p\in\Omega_{\mathbf{y}}}\sqrt{\left|\mathbf{y}_{p}-\mathbf{\hat{y}}_{p}\right|^{2}+\epsilon^{2}},$
(20)

where $0<\epsilon\ll 1$ is a small constant such that the inner term is non-
zero. Equation 18 can be seen as a special case of Charbonnier with
$\epsilon=0$.

Pixel loss functions favor a high PSNR because both formulations use pixel
differences. Also, PSNR does not correlate well with subjectively perceived
quality (see Section 2.2.3), which makes pixel loss functions sub-optimal.
Another observation is that the resultant images tend to be blurry: sharp
edges are not modeled well. One way to combat this is by adding uncertainty.

####  3.1.2 Uncertainty-Driven Loss

Modeling uncertainty in DL improves the performance and robustness of deep
networks [48]. Therefore, Ning et al. proposed an adaptive weighted loss for
SISR [49], which aims at prioritizing texture and edge pixels that are
visually more significant than pixels in smooth regions. Thus, the adaptive
weighted loss treats every pixel unequally.

Inspired by insights from Variational Autoencoders (VAE) [50], they first
model an estimated uncertainty. Let $\mathcal{M}$ be the SR model with
parameters $\theta$, which learns two intermediate results:
$\mu_{\theta}\left(\mathbf{x}\right)$, the mean image, and
$\sigma_{\theta}\left(\mathbf{x}\right)$, the variance image (uncertainty).
Consequently, the approximated image $\mathbf{\hat{y}}$ is given by

$\mathbf{\hat{y}}=\mathcal{M}\left(\mathbf{x};\theta\right)=\underbrace{\mu_{\theta}\left(\mathbf{x}\right)}_{=\mathbf{\hat{y}}_{\mu}}+\epsilon\cdot\underbrace{\sigma_{\theta}\left(\mathbf{x}\right)}_{=\mathbf{\hat{y}}_{\sigma}},$
(21)

where $\epsilon\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}\right)$. While most
DL-based SISR methods estimate only the mean $\mathbf{\hat{y}}_{\mu}$. In
contrast, Ning et al. proposed estimating the uncertainty
$\mathbf{\hat{y}}_{\sigma}$ simultaneously [49].

Based on the observation that the uncertainty is sparse for SISR due to many
smooth areas, Ning et al. present an Estimating Sparse Uncertainty (ESU) loss:

$\mathcal{L}_{\text{ESU}}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\exp\left(-\ln\mathbf{\hat{y}}_{\sigma}\right)\cdot\left\lVert\mathbf{y}-\mathbf{\hat{y}}\right\rVert_{1}+2\cdot\ln\mathbf{\hat{y}}_{\sigma}$
(22)

However, they observed that $\mathcal{L}_{ESU}$ lowers the performance and
concluded that such a loss function is unsuitable for SISR. They also
concluded that prioritizing pixels with high uncertainty is necessary to
benefit from uncertainty estimation. As a result, they proposed an adaptive
weighted loss named Uncertainty-Driven Loss (UDL) and used a monotonically
increasing function instead of
$\exp\left(-\ln\mathbf{\hat{y}}_{\sigma}\right)$ in Equation 22:

$\mathcal{L}_{\text{UDL}}\left(\mathbf{y},\mathbf{\hat{y}}\right)=\left[\ln\mathbf{\hat{y}}_{\sigma}-\min\left(\ln\mathbf{\hat{y}}_{\sigma}\right)\right]\cdot\left\lVert\mathbf{y}-\mathbf{\hat{y}}_{\mu}\right\rVert_{1},$
(23)

where
$\left[\ln\mathbf{\hat{y}}_{\sigma}-\min\left(\ln\mathbf{\hat{y}}_{\sigma}\right)\right]$
is a non-negative linear scaling function. There are two variables that need
to be learned: $\mathbf{\hat{y}}_{\mu}$ and $\mathbf{\hat{y}}_{\sigma}$. Ning
et al. propose to use $\mathcal{L}_{ESU}$ to learn $\mathbf{\hat{y}}_{\sigma}$
and then to train a new network with $\mathcal{L}_{UDL}$ to learn
$\mathbf{\hat{y}}_{\mu}$, but with the pre-trained $\mathbf{\hat{y}}_{\sigma}$
fixed [49]. The idea of using two train processes is to prevent the variance
image from degenerating into zeros. With this setup, they achieve better
results than $L1$ and $L2$ for a wide range of network architectures [51, 52].
However, it requires a two-step training procedure, which adds extra training
time. To this day, we found no known techniques that circumvent such a two-
step training process. A visualization can be found in supplementary material.

####  3.1.3 Content Loss

Instead of using the difference between the approximated and the ground-truth
image, one can transform both entities further into a more discriminant
domain. Thus, the resultant loss function utilizes feature maps from an
external feature extractor. Such an approach is similar to TBE (see Section
2.2.5). In more detail, the feature extractor is pre-trained on another task,
i.e., image classification or segmentation. During the training of the actual
SR model on the difference of feature maps, the parameters of the feature
extractor remain fixed. Thus, the goal of the SR model is not to generate
pixel-perfect estimations. Instead, it produces images whose features are
close to the features of the target.

More formally, let $\varphi$ be a Convolutional Neural Network (CNN) that
extracts features like VGG [53] and let $\varphi^{l}$ be the $l$-th feature
map. The content loss describes the difference of feature maps generated from
the HR image $\varphi^{l}\left(\mathbf{y}\right)$ and the approximation of it,
$\varphi^{l}\left(\mathbf{\hat{y}}\right)$, and is defined as

$\mathcal{L}_{\text{Content}}\left(\mathbf{y},\mathbf{\hat{y}},l\right)=\left\lVert\varphi^{l}\left(\mathbf{\hat{y}}\right)-\varphi^{l}\left(\mathbf{y}\right)\right\rVert_{2}$
(24)

The supplementary material provides a visualization of this approach. It was
used by SRGAN [14] and many others [13]. The motivation is to incorporate
image features (content) instead of pixel-level details; a strategy that has
been frequently used for Generative Adversarial Networks (GANs).

###  3.2 Generative Adversarial Networks

Since the early days of GANs [54], they have had a variety of applications in
CV tasks e.g., in text-to-image synthesis [55]. The core idea is to use two
distinct networks: a generator $G$ and a discriminator $D$. The generator
network learns to produce samples close to a given dataset and to fool the
discriminator. In the case of SR, the generator is the SR model
($G=\mathcal{M}$). The discriminator tries to distinguish between samples
coming from the generator and samples from the actual dataset. The interaction
between the generator and discriminator corresponds to a minimax two-player
game and is optimized using the adversarial loss [14, 13]. Given the ground-
truth HR image $\mathbf{y}$ of the LR image $\mathbf{x}$ and the approximated
SR image $\mathbf{\hat{y}}=G_{\theta_{G}}\left(\mathbf{x}\right)$, the loss
functions based on Cross-Entropy (CE) are:

$\mathcal{L}^{\text{CE}}_{G}=-\mathbb{E}_{\mathbf{x}\sim\mathbb{P}_{\text{LR}}\left(\mathbf{x}\right)}\log
D_{\theta_{D}}\left[G_{\theta_{G}}\left(\mathbf{x}\right)\right]$ (25)
$\displaystyle\mathcal{L}^{\text{CE}}_{D}=$
$\displaystyle-\mathbb{E}_{\mathbf{y}\sim\mathbb{P}_{\text{HR}}\left(\mathbf{y}\right)}\left[\log
D_{\theta_{D}}\left(\mathbf{y}\right)\right]$ (26)
$\displaystyle-\mathbb{E}_{\mathbf{x}\sim\mathbb{P}_{\text{LR}}\left(\mathbf{x}\right)}\left[\log\left(1-D_{\theta_{D}}\left[G_{\theta_{G}}\left(\mathbf{x}\right)\right]\right)\right]$

An alternative is to use the Least Square (LS) [56, 57], which yields better
quality and training stability:

$\mathcal{L}^{\text{LS}}_{G}=\mathbb{E}_{\mathbf{x}\sim\mathbb{P}_{\text{LR}}\left(\mathbf{x}\right)}\left(D_{\theta_{D}}\left[G_{\theta_{G}}\left(\mathbf{x}\right)\right]-1\right)^{2}$
(27) $\displaystyle\mathcal{L}^{\text{LS}}_{D}=$
$\displaystyle\mathbb{E}_{\mathbf{x}\sim\mathbb{P}_{\text{LR}}\left(\mathbf{x}\right)}\left(D_{\theta_{D}}\left[G_{\theta_{G}}\left(\mathbf{x}\right)\right]\right)^{2}$
(28)
$\displaystyle+\mathbb{E}_{\mathbf{y}\sim\mathbb{P}_{\text{HR}}\left(\mathbf{y}\right)}\left(D_{\theta_{D}}\left(\mathbf{y}\right)-1\right)^{2}$

Famously known for the GAN research field, but less explored for SR [58], is
the Wasserstein loss with Gradient Penalty (WGAN-GP) [59], which is the
extended version of the Wasserstein loss (WGAN). Despite variations, Lucic et
al. have shown that most adversarial losses can reach comparable scores with
enough hyper-parameter tuning and random restarts [60]. However, this
statement remains unproven for SR and requires more rigorous confirmation in
the future. A recent review by Singla et al. [61] examines GANs for SR in more
detail. Generally, SR models perform better if an adversarial loss is
incorporated. The primary disadvantage is that sufficient training stability
is hard to reach due to GAN-specific issues like mode collapse [14]. One way
to improve the training stability is to introduce regularization terms.

####  3.2.1 Total Variation Loss

A conventional way to regularize GANs is the introduction of the Total
Variation (TV) loss [62]. The TV loss measures the difference of neighboring
pixels in the vertical and horizontal direction in one image. It is defined as

$\text{TV}\left(\mathbf{y}\right)=\frac{1}{N_{\mathbf{y}}}\sum_{i,j,k}\sqrt{\underbrace{\left(\mathbf{y}_{i+1,j,k}-\mathbf{y}_{i,j,k}\right)^{2}}_{\text{diff.
first
axis}}+\underbrace{\left(\mathbf{y}_{i,j+1,k}-\mathbf{y}_{i,j,k}\right)^{2}}_{\text{diff.
second axis}}}$ (29)

and minimizing it results in images with smooth instead of sharp edges. This
term helps in stabilizing the training of GANs [14, 56]. However, it can hurt
the overall image quality since sharp edges are essential.

####  3.2.2 Texture Loss

Gatys et al. [63] proposed a style transfer method that applies a global
texture (e.g., painting style) onto other images with a pre-trained network
$\varphi$ like VGG [53]. Inspired by this, EnhanceNet [13] used the texture
loss during training to enforce textural similarity. It uses the Gram Matrix
[64] to compute the correlation of different channels between feature maps:

$\mathcal{G}^{l}_{\left(c_{1},c_{2}\right)}\left(\mathbf{y}\right)=\sum_{\left(i,j,.\right)\in\Omega_{\mathbf{y}}}\varphi^{l}\left(\mathbf{y}\right)_{\left(i,j,c_{1}\right)}\cdot\varphi^{l}\left(\mathbf{y}\right)_{\left(i,j,c_{2}\right)}$
(30)

where $\varphi^{l}$ is the $l$-th feature map. The texture loss is

$\mathcal{L}_{\text{TEX}}\left(\mathbf{y},\mathbf{\hat{y}},l\right)=\left\lVert\mathcal{G}^{l}\left(\mathbf{\hat{y}}\right)-\mathcal{G}^{l}\left(\mathbf{y}\right)\right\rVert_{2}^{2}$
(31)

###  3.3 Denoising Diffusion Probabilistic Models

Interestingly, DL methods are well suited to remove denoising Gaussian noise.
Denoising Diffusion Probabilistic Models (DDPMs) exploit this insight by
formulating a Markov chain to gradually alter one image into a noise
distribution, and the other way around [65]. The idea of this approach is that
estimating small perturbations is more tractable for neural networks than
explicitly describing the whole distribution with a single, non-analytically-
normalizable function. First in SR was “Super-Resolution via Repeated
Refinement” (SR3) [66]. The DDPM adds noise to the LR image until
$\mathbf{y}_{T}\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}\right)$ [67, 68] and
generates a target HR image $\mathbf{y}_{0}$ iteratively in $T$ refinement
steps. While adding noise is straightforward, it uses a DL-based model that
transforms a standard normal distribution into an empirical data distribution
by reverting the noise-adding process as shown in Figure 2.

Figure 2:  Principle of DDPMs. The Gaussian diffusion process adds noise
iteratively. The iterative refinement process reverts the process. The task of
the SR model is to predict the noise added between two iterations. The
predicted noise is then used to revert one iteration.

The diffusion process $q$ adds Gaussian noise to a LR image
$\mathbf{y}=\mathbf{y}_{0}$ over $T$ iterations with

$q\left(\mathbf{y}_{t}|\mathbf{y}_{0}\right)=\mathcal{N}\left(\mathbf{y}_{t}|\sqrt{\gamma_{t}}\cdot\mathbf{y}_{0},\left(1-\gamma_{t}\right)\cdot\mathbf{I}\right),$
(32)

where $\gamma_{t}=\prod^{T}_{i=1}\alpha_{i}$ and $0<\alpha_{t}<1$ are hyper-
parameters to determine the variance of added noise per iteration.

Consequently, the noisy image $\widetilde{\mathbf{y}}$ can be expressed as

$\widetilde{\mathbf{y}}=\underbrace{\sqrt{\gamma}\cdot\mathbf{y}_{0}}_{\text{mean}}+\underbrace{\sqrt{1-\gamma}\cdot\epsilon}_{\text{variance}},$
(33)

where $\epsilon\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}\right)$. Given
$\gamma$ and $\widetilde{\mathbf{y}}$, one can derive $\mathbf{y}_{0}$ from
$\epsilon$ and vice versa by rearranging Equation 33. Thus, a denoising model
$\varphi_{\theta}$ with parameters $\theta$ has to either predict
$\mathbf{y}_{0}$ or $\epsilon$. SR3 applies the denoising model to predict the
noise $\epsilon$. The alternative regression target $\mathbf{y}_{0}$ remains
open for future research. The model
$\varphi_{\theta}\left(\mathbf{x},\widetilde{\mathbf{y}},\gamma\right)$ takes
as input the LR image $\mathbf{x}$, the variance of the noise $\gamma$, and
the noisy target image $\widetilde{\mathbf{y}}$ and is optimized by the
following loss function

$\mathcal{L}_{\text{SR3}}\left(\mathbf{x},\mathbf{y}_{0}\right)=\mathbb{E}_{\epsilon,\gamma}\left\lVert\varphi_{\theta}\left(\mathbf{x},\underbrace{\sqrt{\gamma}\cdot\mathbf{y}_{0}+\sqrt{1-\gamma}\cdot\epsilon}_{=\widetilde{\mathbf{y}}\text{
by \autoref{eq_sr3yt}}},\gamma\right)-\epsilon\right\rVert_{d}^{d},$ (34)

where $d\in\\{1,2\\}$.

The refinement process $p$ is the reverse direction of the diffusion process.
Given the prediction of $\epsilon_{t}$ by $\varphi_{\theta}$, Equation 33 can
be reformulated to approximate $\mathbf{y}_{0}$:

$\begin{split}&\mathbf{y}_{t}=\sqrt{\gamma_{t}}\cdot\mathbf{\hat{y}}_{0}+\sqrt{1-\gamma_{t}}\cdot\varphi_{\theta}\left(\mathbf{x},\mathbf{y}_{t},\gamma_{t}\right)\\\
\iff&\mathbf{\hat{y}}_{0}=\frac{1}{\sqrt{\gamma_{t}}}\cdot\left(\mathbf{y}_{t}-\sqrt{1-\gamma_{t}}\cdot\varphi_{\theta}\left(\mathbf{x},\mathbf{y}_{t},\gamma_{t}\right)\right)\end{split}$
(35)

Substituting $\mathbf{\hat{y}}_{0}$ into the posterior distribution to
parameterize the mean of
$p_{\theta}\left(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{x}\right)$ yields

$\mu_{\theta}\left(\mathbf{x},\mathbf{y}_{t},\gamma_{t}\right)=\frac{1}{\sqrt{\alpha_{t}}}\left[\mathbf{y}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\gamma_{t}}}\cdot\varphi_{\theta}\left(\mathbf{x},\mathbf{y}_{t},\gamma_{t}\right)\right]$
(36)

The authors of SR3 set the variance of
$p_{\theta}\left(\mathbf{y}_{t-1}|\mathbf{y}_{t},\mathbf{x}\right)$ to
$\left(1-\alpha_{t}\right)$ for the sake of simplicity. As a result, each
refinement step with
$\epsilon_{t}\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}\right)$ is realized as

$\mathbf{y}_{t-1}\leftarrow\frac{1}{\sqrt{\alpha_{t}}}\left[\mathbf{y}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\gamma_{t}}}\cdot\varphi_{\theta}\left(\mathbf{x},\mathbf{y}_{t},\gamma_{t}\right)\right]+\sqrt{1-\alpha_{t}}\cdot\epsilon_{t}$
(37)

And indeed, SR3 generates sharp images with more details than its regression-
based counterparts on natural and face images. The authors also strengthened
the results by positively testing the generated images with 50 human subjects.
Thus, the outcome of SR3 is a promising direction for future investigations.
Nevertheless, SR3 seems prone to bias, e.g., applied to a facial dataset; the
images result in too smooth skin texture by dropping moles, pimples, and
piercings. More extensive ablation studies could help in establishing the
practicality of this technique for real-world SR applications.

##  4 Upsampling

A critical aspect is how to increase the spatial size of a given feature map.
This section gives an overview of upsampling methods, either interpolation-
based (nearest-neighbor, bilinear, and bicubic interpolation) or learning-
based (transposed convolution, sub-pixel layer, and meta-upscale). Various
visualizations can be found in supplementary material.

###  4.1 Interpolation-based Upsampling

Many DL-based SR models use image interpolation methods because of their
simplicity. The most known methods are nearest-neighbor, bilinear, and bicubic
interpolation.

Nearest-neighbor interpolation is the most straightforward algorithm because
the interpolated value is based on its nearest pixel values. Nearest-neighbor
is swift since no calculations are needed, which is why it is favorable for SR
models [13]. However, there are no interpolated transitions, which results in
blocky artifacts.

In contrast, bilinear interpolation bypasses blocky artifacts by producing
smoother transitions with linear interpolation on both axes. It needs a
receptive field of $2\times 2$, making it relatively fast and easily
applicable [69].

However, SR methods typically do not use it because bicubic interpolation
delivers much smoother results. But it demands much more computation time due
to a receptive field size of $4\times 4$ and makes it the slowest approach
among all three methods. However, it is generally used by SR models if an
interpolation-based upsampling is applied [42, 70, 47] since the overall time
consumption is unessential for GPU-based models, which are common in SR
research.

###  4.2 Learning-based Upsampling

Learning-based upsampling introduces modules that upsample a given feature map
within a learnable setup. The most standard learning-based upsampling methods
are the transposed convolution and sub-pixel layer. Promising alternatives are
meta-upscaling, decomposed upsampling, attention-based upsampling, and
upsampling via Look-Up Tables, which are the last discussed methods in this
section.

####  4.2.1 Transposed Convolution

Transposed convolution expands the spatial size of a given feature map and
subsequently applies a convolution operation. In general, the expansion is
realized by adding zeros between given values. However, some approaches differ
by first applying nearest-neighbor interpolation and then applying zero-
padding, e.g., FSRCNN [36]. The receptive field of the transposed convolution
layer can be arbitrary (often set to $3\times 3$). The added values depend on
the kernel size of the subsequent convolution operation. This procedure is
widely known in literature and is also called deconvolution layer, although it
does not apply deconvolution [46, 42, 43]. In practice, transposed convolution
layers tend to produce crosshatch artifacts due to zero-padding. Also, the
upsampled feature values are fixed and redundant. The sub-pixel layer was
proposed to circumvent this problem.

####  4.2.2 Sub-Pixel Layer

Introduced with ESPCN [71], it uses a convolution layer to extract a deep
feature map and rearranges it to return an upsampled output. Thus, the
expansion is carried out in the channel dimension, which can be more efficient
for smaller kernel sizes than transposed convolution. However, assume zero
values are used instead of nearest-neighbor interpolation in the transposed
convolution. In that case, a transposed convolution can be simplified to a
sub-pixel layer [72]. Given a scaling factor $s$ and input channel size $c$,
it produces a feature map with $s^{2}\cdot c$ channels. Next, a convolution
operation with zero padding is applied, such that the spatial size of the
input and the resulting feature map remains the same. Finally, it reshapes the
feature map to produce a spatially upsampled output. The receptive field of
the convolutional layer can be arbitrary but is often $1\times 1$ or $3\times
3$. This procedure is widely known in the literature as pixel shuffle layer
[66, 57, 41]. In practice, the sub-pixel layer produces repeating artifacts
that can be difficult to unlearn for deeper layers.

####  4.2.3 Decomposed Upsampling

An extension to the above approaches is decomposed transposed convolution
[73]. Using 1D convolutions instead of 2D convolutions reduces the number of
operations and parameters for the component $k^{2}$ to $2\cdot k$. Note that
the applied decomposition is not exclusively bound to transposed convolution
and can also be used in sub-pixel layers.

####  4.2.4 Attention-based Upsampling

Another alternative to transposed convolution is attention-based upsampling
[74]. It follows the definition of attention-based convolution (or scaled dot
product attention) and replaces the 1x1 convolutions with upsampling methods.
In more detail, it replaces the convolution for the query matrix with bilinear
interpolation and the convolution for the key and value matrix with zero-
padding upsampling similar to transposed convolution. It needs fewer
parameters than transposed convolution but a slightly higher number of
operations. Also, it trains considerably slower: a shared issue across
attention mechanisms.

####  4.2.5 Upsampling with Look-Up Tables

A recently proposed alternative is to use a Look-Up Table (LUT) to upsample
[75]. Before generating the LUT, a small-scale SR model is trained to upscale
small patches of a LR image to target HR patches. Subsequently, the LUT is
created by saving the results of the trained SR model applied on a uniformly
distributed input space. It reduces the upsampling runtime to the time
necessary for memory access while achieving better quality than bicubic
interpolation. On the other hand, it requires additional training to create
the LUT. Also, the size of the small patches is crucial since the size of the
LUT increases exponentially. However, it is an exciting approach that is worth
further exploration.

####  4.2.6 Flexible Upsampling

Figure 3:  Visualization of meta-upscale, adapted from Hu et al. [44]. It can
upscale a feature map with arbitrary scaling factors. It generates a higher
resolution feature map iterative for each position in three steps: (1) It
locates the position in the original feature map via location projection. The
location in the HR feature map is divided component-wise by the scaling
factor. (2) It generates a set of filter weights. The prediction depends on
the location in the HR feature map and its corresponding projection in the LR
feature map. (3) It applies the weights derived by (2) to the LR feature map
and returns the HR feature map value.

Most existing SR methods only consider fixed, integer scale factors
$s\in\mathbb{N}$. However, most real-world scenarios require flexible zooming,
which demands SR methods that can be applied with arbitrary scale factors
$s\in\mathbb{R}^{+}$. It raises a significant problem for models that use
those learning-based upsampling methods: one has to train and save various
models for different scales, which limits the use of such methods for real-
world scenarios [71, 40]. In order to overcome this limitation, a meta-upscale
module was proposed [44]. It predicts a set of filters for each position in a
feature map that is later applied to a location in a lower-resolution feature
map. Figure 3 illustrates the aforementioned steps. This approach is exciting
for tasks that require arbitrary magnification levels, e.g., zooming.
Nevertheless, its downsides unfold if huge scaling factors are required
because it has to predict the filter weights $W\left(i,j\right)$ independently
for each position. There are only a few works [76] that involve this approach
in the literature. Besides existing SR methods, one can treat each image as a
continuous function and generate patches at a fixed pixel resolution around a
center like AnyRes-GAN (2022) [77], which could be an interesting path to
follow for future SR research.

##  5 Attention Mechanisms for SR

Attention revolutionized Natural Language Processing (NLP). It plays an
essential role for DL in several applications [78]. This section presents how
SR methods employ attention. There are two categories: Channel and spatial
attention, which can be applied simultaneously.

###  5.1 Channel-Attention

Figure 4:  Channel-attention mechanism [79]. It reduces a feature map in the
spatial dimensions and extracts weighting values by using several FC layers
that are element-wise multiplied to the initial feature map.  Figure 5:
Spatial-attention mechanism [80]. It extracts informations by inspecting the
relationship between two positions (first matrix multiplication) and returns
the importance of each position as a feature map (second matrix
multiplication). MM denotes the matrix multiplication.

Feature maps generated by CNNs are not equally important. Therefore, essential
channels should be weighted higher than counterpart channels, which is the
goal of channel attention. It focuses on “which” (channels) carry crucial
details. Hu et al. [79] introduced the channel-attention mechanism for CV as
an add-on module for any CNN architecture. Figure 4 illustrates the principle.
Upon this, the Residual Channel Attention Network (RCAN) [40] firstly applied
the channel-attention to SR. They used a second-order feature metric to reduce
the spatial size and applied two fully connected layers to extract the
importance weighting. Because of its simple add-on installation, it is still
in use for current research [81]. One interesting future direction of channel-
attention would be to combine it with pruning in order to not only amplify
quality but also to save computation time.

###  5.2 Spatial-Attention

In contrast to channel attention, spatial attention focuses on “where” the
input feature maps carry important details, which requires extracting global
information from the input. Therefore, Wang et al. [80] proposed non-local
attention, similar to self-attention for NLP, following the idea of Query-Key-
Value-paradigm known from transformers [78]. The principle is shown in Figure
5. Intuitively, the resulting feature map contains each position’s weighted
sum of features. It captures spatially separated information by a long range
in the feature map. In SR, the Residual Non-local Attention Network (RNAN)
[82] was the first architecture using non-local spatial attention to improve
performance.

A known architecture is SwinIR [83], which reaches state-of-the-art
performance and follows the idea of transformers (i.e., multi-head self-
attention [78]) more strictly by applying a sequence of multiple swin
transformer layers. SwinFIR [84] develops this idea further with fast fourier
convolutions. Since transformers originate from NLP, their natural application
is restricted to a sequence of patches, and therefore, it requires a patch-
wise formulation of an image. This is a significant drawback of transformers
applied to SR. SwinIR diminishes the problem by shifting between attention
blocks to transfer knowledge between patches.

Exploring its full potential is an exciting topic for the future [85, 86].
However, one major drawback is the difficulty of measuring the importance at a
global spatial scale since it often requires vector multiplications (along
rows and columns), which is slow for large-scale images. Bottleneck tricks and
patch-wise applications are used to circumvent this problem but are also sub-
optimal for importance weighting. Future research should further optimize
computation speed without forfeiting global information.

###  5.3 Mixed Attention

Since both attention types can be applied easily, merging them into one
framework is natural. Thus, the model focuses on “which” (channel) is
essential and “where” (spatially) to extract the most valuable features. This
combines the benefits of both approaches and introduces an exciting field of
research, especially in SR [87, 88]. One potential future direction would be
to introduce attention mechanisms incorporating both concerns in one module.

A well-known architecture for mixed attention is the Holistic Attention
Network (HAN) [87], which uses hierarchical features (from residual blocks)
and obtains dependencies between features of different depths while allocating
attention weights to them. Thus, it weighs the importance of depth layers in
the residual blocks instead of spatial locations within one feature map.
Finally, the network uses the layer attention weighted feature map and
combines it via addition with channel attention before the final prediction.

##  6 Additional Learning Strategies

Additional learning strategies are blueprints that can be used in addition to
regular training. This section presents the most common methods used in SR,
which can immensely impact an SR model’s overall performance.

###  6.1 Curriculum Learning

Curriculum learning follows the idea of training a model under easy conditions
and gradually involving more complexity [89], i.e., additional scaling sizes.
For instance, a model trains with a scale of two and gradually higher scaling
factors, e.g., ProSR [57]. ADRSR [90] concatenates all previously computed HR
outputs. CARN [91] updates the HR image in sequential order, where additional
layers extend the network for four times upsampling, which uses previously
learned layers, but ignores previously computed HR images [57]. Another way of
using curriculum learning is gradually increasing the noise in LR images,
e.g., in SRFBN [69]. Curriculum learning can shorten the training time by
reducing the difficulty of large scaling factors.

###  6.2 Enhanced Predictions

Instead of enhancing simple input-output pairs, one can use data augmentation
techniques like rotation and flipping for final prediction. More specifically,
create a set of images via data augmentation of one image, e.g., rotation.
Next, let the SR model reconstruct images of the set. Finally, inverse the
data augmentation via transformations and derive a final prediction, i.e., the
mean [43], or the median [92].

###  6.3 Learned Degradation

So far, this work assumed learning processes from LR to HR. The Content
Adaptive Resampler (CAR) [7] introduced a resampler for downscaling. It
predicts kernels to produce downscaled images according to its HR input. Next,
a SR model takes the LR image and predicts the SR image. Thus, it
simultaneously learns the degradation mapping and upsampling task. The authors
demonstrate that the CAR framework trained jointly with SR networks improves
state-of-the-art SR performance furthermore.

###  6.4 Network Fusion

Another way of improving performance is to apply multiple SR models. Network
fusion uses the output of all additional SR models and applies a fusion layer
to the outputs. Finally, it predicts the SR image used for the learning
objective. For instance, the Context-wise Network Fusion (CNF) [93] uses this
approach with SRCNN, a small SR model introduced in the next chapter. A
visualization can be found in supplementary material. This method adds a
considerable amount of computation and memory, depending on the complexity of
the SR models. However, it achieves a performance boost and is recommended if
computational resources are sufficiently available. A potential future
direction would be to use network fusion to determine critical computation
paths and to merge multiple networks into one by pruning or dropout
mechanisms.

###  6.5 Multi-Task Learning

Multi-task learning is an exciting research area for SR with the core idea to
train a given model to perform various tasks [90, 56]. E.g., one can assign a
label to each image and use multiple datasets for training. Next, a SR model
can learn to reconstruct the SR image and predict its category (e.g., natural
or manga image) [94]. Another idea is to use datasets that have access to
salient image boundaries or image segmentation and use features extracted from
the model to predict these [95]. In general, research on self-learning is an
interesting starting point to combine ideas with SR.

###  6.6 Normalization Techniques

A slight change in the input distribution is a cause of many issues because
layers need to continuously adapt to new distributions, which is known as
covariate shift and can be alleviated with BatchNorm [96]. While BatchNorm
helps in tasks like image classification, it is frequently used for SR [14,
56, 97]. However, BatchNorm does not apply well in SR because it removes
network range flexibility by normalizing features [41]. The performance of a
SR model can be increased substantially without BatchNorm [98].

Recently, the authors of Adaptive Deviation Modulator (AdaDM) [99] studied
this phenomenon and were able to show that the standard deviation of residual
features shrinks a lot after normalization layers (including BatchNorm, layer,
instance, and group normalization), which causes the performance degradation
in SR. The standard deviation resembles the amount of variation of pixel
values, which primarily affects edges in images. They proposed a module within
a feature extraction block to address this problem and amplify the pixel
deviation of normalized features. It calculates a modulated output
$\widetilde{\mathbf{y}}$ by

$\widetilde{\mathbf{y}}=\mathbf{y}\cdot\mathrm{e}^{\varphi\left[\log\left(\sigma\left[\mathbf{x}\right]\right)\right]},$
(38)

where $\mathbf{x}$ is the residually propagated input,
$\sigma\left[\mathbf{x}\right]$ its standard deviation along all three axes,
and $\varphi$ a learnable feature extraction module. The logarithmic scaling
guarantees stable training. As a result, it is possible to apply normalization
layers to state-of-the-art SR models and to achieve substantial performance
improvements. One limitation is the additional GPU memory during training,
roughly two times the size. Therefore, lightweight normalization techniques
similar to AdaDM, but with less GPU memory consumption are of high interest
for future research.

##  7 SR Models

This section presents the most technical part: How to construct an SR model?
In the beginning, we introduce different frameworks that all architectures
need to apply, which is the location of the upsampling itself. After that, it
chronologically examines different architecture types. Additional
visualizations can be found in the supplementary material.

###  7.1 Upsampling Location

Figure 6:  Visualization of different upsampling locations within a neural
network: (upper left) post-upsampling, (bottom left) pre-upsampling, (upper
right) progressive upsampling, and (bottom right) iterative up-and-down
upsampling.

Besides the choice of the upsampling method, one crucial decision in designing
SR models is where to place the upsampling in the architecture. There are four
variants explored in SR literature [4]: Pre-, post-, progressive, and
iterative up-and-down upsampling. They are shown in Figure 6 and discussed
next. An additional overview table can be found in the supplementary material.

Pre-Upsampling In a pre-upsampling framework, upsampling is applied in the
beginning. Hereafter, it uses a composition of convolution layers to extract
features. Since the feature maps match the spatial size of the HR image early,
the convolutional layers focus on refining the upsampled input. Thus, it eases
learning difficulty but introduces blur and noise amplification in the
upsampled image, which impacts the final result. Also, models trained with
pre-upsampling have less performance or application issues than other
frameworks since they use DL to refine independently of the scaling in the
beginning. However, it also comes with high memory and computational costs
compared to other frameworks like post-upsampling because of the larger input
for feature extraction.

Post-Upsampling Post-upsampling lowers memory and computation costs by
upsampling at the end, which results in feature extractions in lower
dimensional space. Due to this advantage, this framework decreases the
complexity of SR models, e.g., FSRCNN [36]. However, the computational burden
advances for multi-scaling, where approximations at different scales are
required. A step-wise upsampling during the feed-forward process was proposed
to alleviate this issue, the so-called progressive upsampling.

Progressive Upsampling Unlike pre- and post-upsampling, progressive upsampling
gradually increases the feature map size within the architecture. Cascaded
CNN-based modules typically enhance to a single scaling factor. Its output is
the input for the next module, typically built similarly. Thus, it segregates
the upscaling problem into small tasks, which is a perfect fit for multi-scale
SR tasks. Also, it reduces the learning difficulty for convolutional layers.
However, the benefit is limited to the highest set scale, and higher scaling
factors also require deeper architectures.

Iterative Up-and-Down Upsampling Learning the mapping from HR to LR helps to
understand the relation from LR to HR [100]. Iterative up-and-down upsampling
uses this insight by also downsampling within the architecture. It is found
mainly within recurrent-based network designs and demonstrated significant
improvement compared to the previous upsampling techniques. However, the
proper use requires further exploration. It can be applied with other
frameworks, an interesting avenue for future research.

###  7.2 Simple Networks

Simple networks are architectures that mainly apply a chain of convolutions.
They are easy to understand and typically use a bare minimum of computational
resources due to their size. Most of these architectures can be found in the
early days of DL-based SR because their performances are below state-of-the-
art. Also, the “the deeper, the better” paradigm of DL does not apply well for
simple networks because of vanishing/exploding gradients [101]. Figure 7 shows
the different simple network designs. The first CNN introduced for SR datasets
was SRCNN (2014) by Dong et al. [42]. It uses bicubic pre-upsampling to match
the ground-truth spatial size (see Figure 6). Subsequently, it consists of
three convolution layers, which followed a popular strategy in image
restoration [102]: patch extraction, non-linear mapping, and reconstruction.
The authors of SRCNN claimed that applying more layers hurts the performance,
which contradicts the DL paradigm “the deeper, the better” [103]. As seen in
the following, this observation was false and required more advanced building
blocks to work correctly, e.g., residual connections like in VDSR [101].

Figure 7:  Architecture designs of SRCNN [42], FSRCNN [36], ESPCN [71] and
LapSRN/MS-LapSRN [47]. They are grouped together in this work as simple
network designs, because they use only convolutional operations and upsampling
methods.

In their follow-up paper, the authors explored various ways to speed up SRCNN,
resulting in FSRCNN (2016) [36] that utilized three major tricks: First, they
reduced the kernel size of convolution layers. Second, they used a 1x1
convolution layer to enhance and reduce the channel dimension before and after
a feature processing with 3x3 convolutions. Third, they applied post-
upsampling with transposed convolution, which is the main reason for the speed
up (see Figure 6). Surprisingly, they outperformed SRCNN while obtaining
faster computation.

A year later, LapSRN (2017) [47] was proposed with a key contribution of a
Laplacian pyramid structure [104] that enables progressive upsampling (see
Figure 6). It takes coarse-resolution feature maps as input and predicts high-
frequency residuals that progressively refine the SR reconstruction at each
pyramid level. To this end, predicting multi-scale images in one feed-forward
pass is feasible, thereby facilitating resource-aware applications.

Simple network architecture designs are found primarily in the early days of
DL-based SR because of their limited capacity to learn complex structures due
to their size. Recently, researchers focused on networks with more depth,
either with residual networks or synthetic depth with recurrent-based
networks. The following sections introduce both possibilities.

Figure 8:  Architecture design of VDSR [101]. It applies a deep convolutional
network as feature extractor but also uses a residual from the input to the
final prediction. Therefore, the feature extractor can concentrate to add
high-frequency details onto the interpolated image.

###  7.3 Residual Networks

Residual networks use skip connections to jump over layers. The primary reason
behind adding skip connections is two-fold: To avoid vanishing gradients and
mitigate the accuracy saturation problem [103]. For SR, introducing skip
connections unlocked the world of deep-constructed models. The main advantage
is that deep architectures substitute convolutions with large receptive
fields, which are crucial for capturing important features. The authors of
SRCNN stated that the “the deeper, the better” paradigm does not hold to SR.
In contrast, Kim et al. refuted this statement with VDSR (2015) [101] and
showed that very deep networks could significantly improve SR, visualized in
Figure 8.

They used two insights from other DL approaches: First, they applied a famous
architecture VGG-19 [53] as a feature extraction block. Second, they used a
residual connection from the interpolation layer to the last layer. As a
result, the VGG-19 feature extraction block adds high-frequency details to the
interpolation, resulting in a target distribution that is normal and reduces
the learning difficulty immensely, as shown in Figure 9. Also, it diminishes
vanishing/exploding gradients due to the sparse representation of the high-
frequency added to the interpolation. This merit emitted a trend for following
residual networks that enhance the number of residuals used.

Figure 9:  Pixel value distribution of an example image from BSDS100 [2]. It
can be observed that the distribution of the LR and HR image are skewed. In
contrary, the distribution of the difference between LR and HR looks normally
distributed, which is easier learned by an SR model.

One example is RED-Net (2016) [105], which adapts the U-Net [106] architecture
to SR. It incorporates a downsampling encoder and an upsampling decoder
network, which downsamples the given output to extract features and then
upsamples the feature maps to a target spatial size. During this process, RED-
Net extends the upsample operation with residual information acquired
throughout the downsample procedure, which reduces vanishing gradients. As a
result, it outperforms SRCNN for several scaling factors.

Another example adapted ResNet [107], which consists of multiple residual
units, and DenseNet [108], which sends residual information to all later
appearing convolution operations, in the publication of SRGAN (2016) [14].
Moreover, the authors compared these architectures applied with pixel and
adversarial loss (see Section 3). SRResNet consists of multiple stacked
residual units that allow high-level feature extractions to access low-level
feature information through numerous summation operations. Therefore, it eases
optimization by providing an easy back-propagation path to early layers. Like
SRResNet, SRDenseNet [43] applies dense residual blocks, which utilize even
more residual connections to allow direct paths to earlier layers. In
comparison, SRResNet outperforms SRCNN, DRCN, and ESPCN by a large margin. An
extension to SRDenseNet is the Residual Dense Network [45], proposed in 2018,
which incorporates an additional residual connection upon the dense block.

The Densely Residual Laplacian Network (DRLN) [6] is an extension of
SRDenseNet and a post-upsampling, channel attention-based residual network and
achieves competitive results regarding state-of-the-art. Each dense block is
followed by a module based on Laplacian pyramid attention, which learns inter
and intra-level dependencies between feature maps. It weights sub-band
features progressively in each DRLM, similar to HAN (concatenating various
feature maps of different depths).

There are also variations of residual blocks found in SR, e.g., the
Information Distillation Network (IDN) [46]. It employs residual connections
to accumulate a portion of a feature map to later layers. Given six
convolution layers, it divides the feature map into two parts in the middle.
Then, one part is processed further by the last three layers and added to the
concatenation of the input and the other part. In brief, networks that utilize
residual connections are state-of-the-art. Their ability to efficiently
propagate information helps fight vanishing/exploding gradients, resulting in
excellent performances. Sometimes, residual block usage is combined with other
architectures, such as recurrent-based networks.

###  7.4 Recurrent-Based Networks

Figure 10:  Recurrent-based architecture designs of DRCN [109] (upper left),
DRRN [70] (upper right), and MemNet [97] (bottom). The DRCN applies the same
convolutional layer multiple times. The DRNN extends this idea to recursive
application of a residual block. The MemNet introduces a gate unit to distill
most important features from all intermediate convolution results in a
recursive manner.

Artificial depth can be accomplished with recurrence, where the receptive
field, crucial for capturing important information, is enlarged by repeating
the same operation. Also, recurrence reduces the number of parameters, which
helps to combat overfitting and memory consumption for small devices. It is
achieved by applying a convolution layer multiple times without introducing
new parameters.

The first recurrent-based network for SR was introduced with DRCN (2015) by
Kim et al. [109]. It uses the same convolution layer up to 16 times, and a
subsequent reconstruction layer considers all recursive outputs for final
estimation. However, they observed that their deeply-recursive network is hard
to train but eased it with skip connections and recursive supervision,
essentially auxiliary training. Figure 10 visualizes DRCN.

Combining the core ideas of DRCN [109] and VDSR [101], DRRN (2017) [70]
utilizes several stacked residual units in a recursive block structure. In
addition, it uses 6x and 14x fewer parameters than VDSR and DRCN,
respectively, while obtaining better results. Contrary to DRCN, DRRN shares
weight sets among residual units instead of one shared weight for all
recursively-applied convolution layers. DRNN trains more stable and with
deeper recursions than DRCN (52 in total) by emphasizing multi-path.

Inspired by DRCN, Tai et al. introduced MemNet (2017) [97]. The main
contribution is a memory block consisting of recursive and gate units to mine
persistent memory through adaptive learning. The recursive unit is applied
multiple times, similar to DRCN. The outputs are concatenated and sent to a
gate unit, which is a simple 1x1 convolution layer. The adaptive gate unit
controls the amount of prior information and the current state reserved.
Figure 10 visualizes MemNet. The effect of introducing gates was
groundbreaking for sequence-to-sequence tasks (e.g., GRU [110], or LSTM [111])
but deeply understanding its effect on SR tasks marks an open research
question.

Inspired by DRRN, the authors of DSRN (2018) [112] explored a dual-state
design with a multi-path network. It introduces two states, one operating in
HR and one in LR space, which jointly exploits both LR and HR signals. The
signals are exchanged recurrently between both spaces, LR-to-HR and HR-to-LR,
via delayed feedback [113]. From LR-to-HR, it uses a transposed convolution
layer to upsample. HR-to-LR is performed via strided convolutions. The final
approximation uses the average of all estimations done in the HR space. Thus,
it applies an extended formulation of iterative up-and-down upsampling (see
Figure 6). The two states use more parameters than DRRN, but less than DRCN.
However, exploiting the dual-state design appropriately requires more
exploration in the future.

The Super-Resolution Feedback Network (SRFBN, 2019) [69]is also using feedback
[114]. The essential contribution is the feedback block (FB) as an actual
recurrent cell. The FB uses multiple iterative up-and-downsamplings with dense
skip connections to produce high-level discriminant features. The SRFBN
generates a SR image for each iteration, and the FB block receives the
previous iteration’s output. It tries to generate the same SR image for single
degradation tasks in each iteration. For more complex cases, it trains to
return better and better quality images with each iteration via curriculum
learning (see Section 6.1). SRFBN has shown signiﬁcant improvement over the
other frameworks but requires more research in the future.

Liu et al. proposed NLRN (2018) [115] that provides a non-local module to
produce feature correlation for self-similarity. Each position in the image
measures the feature correlation of each location in its neighborhood. NRLN
utilizes adjacent recurrent stages between the feature correlation messages.
And indeed, NLRN achieved slightly better performances than DRCN, DRCN, and
MemNet.

Nevertheless, primary research on RNNs in SR is lately driven for MISR, such
as video SR [116] or meta-learning [117] related tasks. In general, recurrent-
based networks are interesting for saving parameters, but the major drawback
is their computational overhead by repeatedly applying the same operation.
Also, they are not parallelizable due to the time dependency. Alternatives are
lightweight architectures, which are introduced next.

Figure 11:  Architecture design of MDSR [41]. It utilizes multi-path learning
to select between multiple scaling factor paths (denoted by 2x, 3x, 4x on the
bottom right of the blue boxes). Also, all paths share an intermediate feature
extraction block to save parameters.

###  7.5 Lightweight Networks

So far, we have introduced models that increase the quality of SR images and a
few models that try to do the same, but with less computational effort. For
instance, FSRCNN was built to be faster than SRCNN. Another example is
recurrent-based networks that reduce redundant parameters. The downside of
those lean recurrent networks is that parameter reduction comes at the expense
of increased operations and inference time, an essential aspect of real-world
scenarios. E.g., SR on mobile devices is limited by battery capacity, which
depends on the computation power needed. Therefore, lightweight architectures
explicitly focus on both execution speed and memory usage.

The MDSR (2017) [41] uses the multi-path approach to learn multiple scaling
factors with shared parameters. It has three non-identical paths as pre-
processing steps and three paths for upsampling. For a given scaling factor
$s\in\\{2,3,4\\}$, MDSR is choosing deterministic between the three paths. The
paths for larger scales are built deeper than those for lower scaling factors.
Between the pre-processing and upsampling steps is a shared module consisting
of multiple residual blocks. This feature extraction block is trained and
commonly used for all scaling factors, visualized in Figure 11. The main
advantage is that one model is sufficient to train on multiple scales, which
saves parameters and memory. In contrast, other SR models must be trained
independently on different scales and saved independently for multi-scale
applications. Nevertheless, adding a new scaling factor requires training from
scratch.

Other lightweight architectures adapt this idea to enable parameter-efficient
multi-scale training, such as CARN/CARN-M (2018) [118].

Figure 12:  Design of cascading blocks [118]. It is a densely connected block,
which consists of Residual-E blocks [118] and 1x1 convolutions that take as
input the Residual-E block’s output and the residual connections from the
layers before. The Residual-E block itself is built like a residual unit but
performs group convolution, which is efficient in inference time, depending on
the group size.

Moreover, it implements a cascading mechanism upon the residual network [107].
CARN consists of multiple cascading blocks (see Figure 12) and 1x1
convolutions between them. The output of the cascading blocks is sent to all
subsequent 1x1 convolutions like in the cascading block itself. Thus, the
local cascading is almost identical to a global one. It allows multi-level
representations and stable training like for residual networks. Ultimately, it
chooses between three paths, which upsample the feature map to 2x, 3x, or 4x
scaling factors via efficient sub-pixel layers similar to MDSR. Inspired by
MobileNet [119], CARN also uses grouped convolution in each residual block
component. This allows configuration of the model’s efficiency since choosing
different group sizes and the resulting performances are in a trade-off
relationship. The residual blocks with group convolution can reduce the
computation up to 14 times, depending on the group size. They tested a variant
of CARN, which sets the group size so that the computation reduction is
maximized and called it CARN-Mobile (CARN-M). Moreover, they further reduced
CARN-M’s parameters by enabling weight-sharing of their residual blocks within
each cascading block (reduction by up to three times compared to non-shared).

Figure 13:  Visualization of the RFDB block [120]. It uses a multi-path
approach to distill features with 1x1 convolutions and introduces a shallow
representation of the residual block, which consists of only one 3x3
convolution. In the end, all intermediate results are combined with one 1x1
convolution layer and an enhanced spatial attention [121] layer.

Inspired by IDN and IMDB [122], the RFDN (2020) [120] rethinks the IMDB
architecture by using RFDB blocks as shown in Figure 13. RFDB blocks consist
of feature distillation connections, which cascade 1x1 convolutions towards a
final layer. Moreover, it uses shallow residual blocks (SRBs), consisting of
only one 3x3 convolution, to process a given input further. The final layer is
a 1x1 convolution layer that combines all intermediate results. In the end, it
applies the enhanced spatial attention [121], designed specifically for
lightweight models. The RFDN architecture comprises subsequent RFDB blocks and
uses a post-upsampling framework with a final sub-pixel layer.

A very hardware-aware and quantization-friendly network is XLSR (2021)
[123].It applies multi-paths to ease the burden of the convolution operations
and employs 1x1 convolution to combine them pixel-wise. Each convolution layer
has a small filter size (8, 16, 27). After the combination, it splits the
feature map and applies multi-paths again. A core aspect of XLSR is the end
activation layer, which exploits quantization benefits. Quantization is useful
since it can save parameters by using more miniature bit representations
[124]. Unfortunately, many mobile devices support 8-bit. Therefore, applying
uint8 quantization on SR models that performed well in float32 or float16 does
not work. Clipped ReLU (constrained to a max value of 1) as the last
activation layer instead of the typical ReLU [125] can dimish this issue.
Nevertheless, the authors recommend searching for other maximum values with
further experiments.

Generally, there are plenty of ideas to make SR models lightweight yet to be
discovered. They include simplifications, quantization, and pruning of
existing architectures. Also, more resource-constrained devices and
applications utilizing SR are a growing field of interest [126].

###  7.6 Wavelet Transform-based Networks

Different representations of images can bring benefits, such as computational
speed up. The wavelet theory gives a stable mathematical foundation to
represent and store multi-resolution images, depicting contextual and textural
information [127]. Discrete Wavelet Transform (DWT) decomposes an image into a
sequence of wavelet coefficients. The most frequent wavelet in SR is the Haar
wavelet, computed via 2D Fast Wavelet Transform. The wavelet coefficients are
calculated by repeating the decomposition to each output coefficient
iteratively. It captures image details in four sub-bands: average (LL),
vertical (HL), horizontal (LH), and diagonal (HH) information. One of the
first networks that worked with wavelet prediction was DWSR (2017) [128]. It
uses a simple network architecture to refine the differences between the LR
and HR image wavelet decompositions in a pre-upsampling framework. First, it
calculates the wavelet coefficients of the enlarged (with bicubic
interpolation) LR image. Then it processes the wavelet coefficients with
convolution layers. Next, it adds the initially calculated wavelet
coefficients, which are beared with a residual connection. Thus, the
convolution layers learn additional details of the coefficients. Finally, it
applies the reverse process of 2D-DWT to obtain the SR image, as depicted in
Figure 14.

Another approach was proposed with WIDN (2019) [129], which uses stationary
wavelet transform instead of DWT to perform better. A more sophisticated model
was proposed around the same time as DWSR with Wavelet-SRNet (2017) [130]. It
provides an embedding network, consisting of residual blocks, to generate
feature maps from the LR image. Next, it applies DWT several times and
utilizes multiple wavelet prediction networks. Finally, it applies the reverse
process and uses a transposed convolution for upsampling. The coefficients are
employed to a wavelet-based loss function, while the SR images are used for a
traditional texture and MSE loss function. As a result, their network applies
to different input resolutions with various magnifications and shows
robustness toward unknown Gaussian blur, poses, and occlusions for MS-SR. The
idea of multi-level wavelet CNNS can also be found in later publications,
i.e., MWCNN (2018) [131].

Following works apply a hybrid approach by mixing wavelet transform with other
well-known SR methods. I.e., Zhang et al. proposed a wavelet-based SRGAN
(2019) framework [132], which merged the advantages of SRGAN and wavelet
decomposition. The generator uses an embedding network to process the input
into feature maps, similar to Wavelet-SRNet. Next, it uses a wavelet
prediction network to refine the coefficients, similar to DWSR. In 2020, Xue
et al. [133] combined wavelets with residual attention blocks that contain
channel attention and spatial attention modules (mixed attention, see the
following Section 5) and called their network WRAN. Over the last several
years, the application of wavelets also found its way to Video SR [134].

In general, wavelet transformations enable an efficient representation of
images. As a result, SR models using this strategy often reduce the overall
model size and computational costs while reaching similar performances to
state-of-the-art architectures. However, this research area needs more
exploration.

Figure 14:  Visualization of DWSR [128]. It calculates wavelet coefficients
and uses a CNN to calculate the difference in the average (DA), vertical (DV),
horizontal (DH), and diagonal (DD) sub-bands. Next, it adds the initially
calculated coefficients via residual connection to the difference to derive
the coefficients of the predicted SR image (SA, SV, SH, and SD), which is
obtained by using the reverse 2D-DWT.

##  8 Unsupervised Super-Resolution

The astonishing performance of supervised SR is imputed to their ability to
learn natural image mainly from many LR-HR image pairs, mostly with known
degradation mapping, which is often unknown in practice. Thus, supervised
trained SR models are sometimes unreliable for practicable use-cases. In
addition, some specialized application areas lack LR-HR image pairs as the
training dataset. Therefore, there is a growing interest in unsupervised SR
methods.

###  8.1 Weakly-Supervised

Weakly-supervised methods use unpaired LR and HR images like WESPE (2018)
[135]. WESPE consists of two generators and two discriminators. The first
generator takes a LR image and super-resolves it. The output of the first
generator constitutes a SR image, but is also regularized with TV loss [62].
The second generator takes the prediction of the first generator and performs
the inverse mapping. The result of the second generator is optimized via
content loss [13] with the original input, the LR image. The two
discriminators take the SR image of the first generator and are trained to
distinguish between predictions and original HR images. The first
discriminator classifies inputs based on image colors into SR or HR images.
The second discriminator uses image textures [63] in order to classify.

A similar approach is a cycle-in-cycle SR framework called CinCGAN (2018)
[56], based on CycleGAN [136]. It uses a total of four generators and two
discriminators. The first generator takes a noisy LR image and maps it to the
clean version. The first discriminator is trained to distinguish between clean
LR images from the dataset and the predicted clean images. The second
generator trains the inverse function. Thus, it generates the noisy image from
the predicted clean version, which closes the first cycle of a CycleGan. The
third generator is of particular interest because it is the actual SR model
which upsamples the LR image to HR. The second discriminator is trained to
distinguish between the predicted and the dataset’s HR images. The last
generator maps the predicted HR image to the noisy LR image, which closes the
second cycle of a CycleGAN. Besides its promising results and similar
approaches [137], it requires further research to decrease learning difficulty
and computational cost.

Figure 15:  Zero-Shot Super-Resolution (ZSSR) [92] and Deep Image Prior (DIP)
[138]. ZSSR uses the LR image and downsamples it and the SR model learns to
reverse the downsampling. For the final prediction of the SR image of the LR
image, it is applied to the LR image directly. DIP uses fixed noise as input
and predicts the SR image, which is downsampled to optimize the difference
between the downsampled image and a given LR image. The final prediction uses
the SR model to predict the SR image but skips the degradation mapping.

###  8.2 Zero-Shot

Zero-shot or one-shot learning is associated with training on objects and
testing on entirely different objects from a different class that was never
observed. Ideally, a classifier trained on horses should recognize zebras if
the knowledge “zebras look like striped horses” is transferred [139]. The
first publication on Zero-Shot in SR is ZSSR (2017) [92]. The goal was to
train only on one image at hand, one of a kind. The degradation mapping for
ZSSR was chosen to be fixed, such as bicubic. ZSSR downsamples the LR image
and trains a CNN to reverse the degradation mapping. The trained CNN is then
finally used directly on the LR image. Surprisingly, this method reached
better results than SRCNN and was close to VDSR.

Upon this, a Degradation Simulation Network (DSN, 2020) [140] based on depth
information [141] was proposed to avoid a pre-defined degradation kernel. It
uses bi-cycle training to simultaneously learn the unknown degradation kernel
and the reconstruction of SR images. The MZSR [142] merged the ZSSR setting
with meta-learning and used an external dataset to learn different blur
kernels, which is called task distribution in the field of meta-learning. The
SR model is then trained on the downsampled image similar to ZSSR with the
blur kernel returned from the meta-test phase. The profit of this approach is
that it conducts the SR model to learn specific information faster and
performs better than pure ZSSR. Zero-shot learning for SR marks an exciting
area for further research because it is highly practical, especially for
applications where application-specific datasets are rare or non-existent.

###  8.3 Deep Image Prior

Ulyanov et al. [138] proposed Deep Image Prior (DIP), which contradicts the
conventional paradigm of training a CNN on large datasets. It uses a CNN to
predict the LR image when downsampled, given some random noise instead of an
actual image. Therefore, it follows the strategy of ZSSR by using only the LR
image. However, it fixes the input to random noise and applies a fixed
downsampling method to the prediction. Moreover, it optimizes the difference
between the downsampled prediction and the LR image. The CNN then produces the
SR image without using the fixed downsampling method. Thus, it generates an SR
image out of noise instead of transforming a raw image. The difference between
ZSSR and DIP is highlighted in Figure 15. Surprisingly, the results were close
to LapSRN [143]. Unfortunately, it is a theoretical publication about image
priors, and the approach is too slow to be useful for most practical
applications, as the authors stated themselves. However, it does not exclude
future ideas that could enhance the practicability of DIP concerning better
image reconstruction quality and especially runtime.

Figure 16:  Visualization of ESRN [144]. A controller derives an architecture
by selecting from a pre-set of various efficient residual blocks.

##  9 Neural Architecture Search

Recently, a new field called Neural Architecture Search (NAS) has gained
popularity, which aims to automatically derive designs instead of hand-
designed Neural Networks (NNs) created by human artistry [10]. Fortunately,
NAS can be applied to derive parts of SR models.

MoreMNAS (2019) [10] investigated NAS for SR and for resource-aware mobile
devices. The goal is to derive cell-based designs similar to the idea of
residual blocks. A SR model usually consists of feature extraction, non-linear
mapping, and reconstruction, like in SRCNN. Their search space is constrained
to the non-linear mapping. They modified the NAS algorithm NSGANet [145] to
SR, which is based on the evolution algorithm NSGA-II with multi-objective
evolution [146]. In the case of MoreMNAS, multi-objective fitness is defined
by high PSNR, low FLOPS, and a few parameters. Furthermore, they combine it
with reinforcement learning, in which an LSTM [111] controller returns design
choices based on NSGA-II, and the evaluator returns a MSE-based reward. The
evaluator trains and evaluates the design choices in a separate training
phase. As a result, MoreMNAS delivers rivaling models compared to methods
(like FSRCNN, VDSR, DRRN, CARN-M, and DRCN) with fewer FLOPS. FALSR (2019)
[147] is a similar approach, which differs by using a hybrid controller and a
cell-based elastic search space that enables both macro (the connections among
different cell blocks) and micro search (cell blocks). A faster method with
comparable performance to FALSR was proposed with ESRN (2019) [144]. They
search locations of pooling and upsampling operators and derive the
architecture with the guidance of block credits, which weighs the sampling
probability of mutation to favor admirable blocks. Moreover, they constrain
the search space to efficient residual dense blocks from known lightweight
architectures, such as RFDB from RFDN (see Section 7.5). The latter is shown
in Figure 16. A similar approach can be found in DeCoNAS (2021) [148], which
uses ENAS [149] and densely connected network blocks.

HNAS (2020) [150] introduces a hierarchical search space that consists of a
cell-level search space and a network-level search space, similar to FALSR.
The cell-level search space identifies a series of blocks that increase model
capacity. The network-level search space determines the position of upsampling
layers. For cell-level search space, HNAS uses two LSTM controllers: one
derives normal cells consisting of convolutions, and the other one the
upsampling cells (bilinear interpolation, sub-pixel layer, transposed
convolution, and more). On top, a network-level controller is used to select
the positions of the upsampling cells.

Figure 17:  Visualization of NAS-DIP [151]. A controller derives an
architecture. The architecture is then trained in a fashion described by DIP
[138]. It optimizes the loss function obtained by downsampling the SR output
and comparing it with the corresponding LR output. The SR output is then used
with the HR output to determine the reward.

NAS-DIP (2020) [151] combines NAS with DIP [138] (see Section 8.3 and Figure
17). It consists of two phases: In the first phase, they apply reinforcement
learning with a RNN controller (PSNR as a reward) to generate a network
structure. The second phase uses the network structure and optimizes the
mapping from random noise to a degenerated image (as in DIP [138]). Next, it
repeats the phases with the reward gained after the second phase. Their search
space is designed for two components. The first component is the upsampling
cell (e.g., bilinear, bicubic, nearest-neighbor). The second component is
feature extraction (e.g., convolution, depth-wise convolution, and others).
Moreover, they extend their search space to learn cross-level connection
patterns across different feature levels in an encoder-decoder network, which
is close to the U-Net architecture of DIP.

Another way of combining NAS with frameworks that worked well in SR was
proposed by Lee et al. (2020) [152]. They combined NAS with GANs and defined
search spaces for generators and discriminators. The GAN framework is then
used to train the generator in the manner of SRGAN. However, they faced
substantial stability problems during their work, which needs further
research.

One major drawback of reinforcement learning based NAS is that they need
additional training to evaluate design choices. It imposes a significant time
overhead on the search process. Another problem is that reinforcement learning
based NAS approaches rely on discrete choices performed and exploited by a
controller. One way to omit that is by using a gradient-based search like
DARTS [153].

Inspired by DARTS, HiNAS (2021) [154] adopts gradient-based search and builds
a flexible hierarchical search space. The hierarchical search space is similar
to HNAS, which refers to micro and macro search. During cell search (micro),
HiNAS considers all combinations of operations as a weighted sum. The highest
weights derive the final cell design. The macro search follows the same idea
as micro search and conducts several supercells with different settings, and
derives the final architecture gradient-based.

Wu et al. (2021) [155] proposed something similar. They adopt gradient-based
search but extend the search space into three levels. The first level
describes the network level, which defines all candidate network paths. The
second level determines possible candidate operations. Lastly, the kernel-
level is a subset of convolution kernel dimensions.

The main streams of NAS in SR are dominated by evolutionary algorithms paired
with reinforcement learning. They imply a significant search time, which is
inevitable due to the repetitive action and reward processes. Most recently,
gradient-based methods were applied to facilitate the search time. Moreover,
they enable continuous search by relaxing the search space, transforming
discrete choices into a weighted sum of possible paths. Moreover, the trend
points to various hierarchies within the search process, meaning that more
aspects of determining a network design for SR are incorporated into NAS.

##  10 Discussion and Future Directions

This section gives a short overview of the topics discussed in this paper and
shows potential future directions.

IQA Evaluating quality of generated images is difficult and still an open
problem. PSNR and SSIM are valuable metrics because of fast calculation, but
do not always match subjectively perceived quality. Developing a metric that
overcomes this problem is of high interest. Future research should focus on
such a metric since it would also be interesting besides SR, e.g., Text-to-
Image Synthesis [55].

Learning Objectives The selection of learning objectives strongly depends on
the data domain. A learning objective that fits all SR domains is not given
and remains open for research. Most SR papers use simple formulations, e.g.,
regression-based or combinations of them. Exploring various probable loss
functions is highly demanded. Recent developments in uncertainty-driven loss
functions can be a promising direction for new loss functions.

Upsampling One problem with the most commonly used upsampling layers is that
they produce artifacts. Also, the scale factor must be predefined. Therefore,
using them in an application like zooming is not feasible. An alternative is
meta-upscaling, which enables arbitrary scaling but comes with computational
overhead and stability issues. Thus, new lightweight layers for arbitrary
upscaling are highly interesting and should be focused on in the following
years.

Unsupervised Super-Resolution Data-rich datasets inherently dictate the
overall performance of any SR method. However, LR-HR image pairs are not
always given. In such a case, unsupervised SR methods are interesting. It
marks an exciting area for further research because it is highly practical,
especially for applications where datasets are rare or non-existent. Zero-shot
learning and DIP are good starting points, but the execution speed and
practicability is not enough for real-world applications. New ideas are
required to enhance practicability regarding better image reconstruction
quality and especially execution time.

Neural Network Architectures Concerning the statement of DIP, a well-designed
architecture is critical for success on SR tasks. Recently, a lot of
architectures were proposed to investigate certain aspects, such as SRResNet
[132] (handling vanishing/exploding gradients), DSRN [112] (exploring
recurrence for SR), XLSR [123] (hardware-aware architecture) and SRGAN [132]
(exploring GANS for SR). Future work has to investigate which modules (like
residual blocks or attention mechanisms) contribute the most regarding certain
aspects to formulate a guideline for engineers on constructing a suitable
architecture for a given problem: high quality vs. computational efficiency.

##  11 Conclusion

With the advent of DL, Super-Resolution (SR) has recently become a rapidly
moving research area. However, despite promising results, the field still
faces challenges that require further research, e.g., flexible upsampling. We
review the area of SR with recent advances and examine state-of-the-art
models, such as transformer-based SR, and other architecture designs proposed
lately (e.g., denoising diffusion probabilistic models). We critically examine
current strategies and identify new research areas. We complement previous
surveys by incorporating the latest developments and ideas, such as
uncertainty-driven loss functions, new normalization techniques, wavelet
networks, or neural architecture search. Also, we added various visualizations
of models and methods in the chapters and in the supplementary material to
make navigation through this domain more accessible. This review aims to help
researchers push further the boundaries of DL applied to SR.

## Acknowledgments

This work was supported by the BMBF projects ExplAINN, EDeL, and XAINES (Grant
01IS19074, 01IS19075, and 01IW20005).

## References

  * [1] R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in _International conference on curves and surfaces_. Springer, 2010, pp. 711–730.
  * [2] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , vol. 2. IEEE, 2001, pp. 416–423.
  * [3] D. Valsesia and E. Magli, “Permutation invariance and uncertainty in multitemporal image super-resolution,” _arXiv preprint arXiv:2105.12409_ , 2021.
  * [4] S. M. A. Bashir, Y. Wang, M. Khan, and Y. Niu, “A comprehensive review of deep learning-based single image super-resolution,” _PeerJ Computer Science_ , vol. 7, p. e621, 2021.
  * [5] Z. Wang, J. Chen, and S. C. H. Hoi, “Deep learning for image super-resolution: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 10, pp. 3365–3387, 2021.
  * [6] S. Anwar and N. Barnes, “Densely residual laplacian super-resolution,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [7] W. Sun and Z. Chen, “Learned image downscaling for upscaling using content adaptive resampler,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 4027–4040, 2020.
  * [8] C.-Y. Yang, C. Ma, and M.-H. Yang, “Single-image super-resolution: A benchmark,” in _ECCV_ , 2014.
  * [9] D. Thapa, K. Raahemifar, W. R. Bobier, and V. Lakshminarayanan, “A performance comparison among different super-resolution techniques,” _Comput. Electr. Eng._ , vol. 54, pp. 313–329, 2016.
  * [10] X. Chu, B. Zhang, and R. Xu, “Multi-objective reinforced evolution in mobile neural architecture search,” in _European Conference on Computer Vision_. Springer, 2020, pp. 99–113.
  * [11] A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, “Video super-resolution with convolutional neural networks,” _IEEE transactions on computational imaging_ , vol. 2, no. 2, pp. 109–122, 2016.
  * [12] M. Kawulok, P. Benecki, D. Kostrzewa, and L. Skonieczny, “Evolving imaging model for super-resolution reconstruction,” in _Proceedings of the Genetic and Evolutionary Computation Conference Companion_ , 2018, pp. 284–285.
  * [13] M. S. Sajjadi, B. Scholkopf, and M. Hirsch, “Enhancenet: Single image super-resolution through automated texture synthesis,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4491–4500.
  * [14] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4681–4690.
  * [15] N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti _et al._ , “Image database tid2013: Peculiarities, results and perspectives,” _Signal processing: Image communication_ , vol. 30, pp. 57–77, 2015.
  * [16] J. Kim and S. Lee, “Deep learning of human visual sensitivity in image quality assessment framework,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1676–1684.
  * [17] H. Talebi and P. Milanfar, “Nima: Neural image assessment,” _IEEE Transactions on Image Processing_ , vol. 27, no. 8, pp. 3998–4011, 2018.
  * [18] K. Ma, W. Liu, T. Liu, Z. Wang, and D. Tao, “dipiq: Blind image quality assessment by learning-to-rank discriminable image pairs,” _IEEE Transactions on Image Processing_ , vol. 26, no. 8, pp. 3951–3964, 2017.
  * [19] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, “Learning to rank using gradient descent,” in _Proceedings of the 22nd international conference on Machine learning_ , 2005, pp. 89–96.
  * [20] T.-Y. Liu, “Learning to rank for information retrieval,” 2011.
  * [21] X. Liu, J. Van De Weijer, and A. D. Bagdanov, “Rankiqa: Learning from rankings for no-reference image quality assessment,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1040–1049.
  * [22] G. Koch, R. Zemel, R. Salakhutdinov _et al._ , “Siamese neural networks for one-shot image recognition,” in _ICML deep learning workshop_ , vol. 2. Lille, 2015.
  * [23] Z. Wang, A. C. Bovik, and L. Lu, “Why is image quality assessment so difficult?” in _2002 IEEE International conference on acoustics, speech, and signal processing_ , vol. 4. IEEE, 2002, pp. IV–3313.
  * [24] M. P. Eckert and A. P. Bradley, “Perceptual quality metrics applied to still image compression,” _Signal processing_ , vol. 70, no. 3, pp. 177–200, 1998\.
  * [25] V. K. Ha, J. Ren, X. Xu, S. Zhao, G. Xie, and V. M. Vargas, “Deep learning based single image super-resolution: A survey,” in _International Conference on Brain Inspired Cognitive Systems_. Springer, 2018, pp. 106–119.
  * [26] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [27] W. Xue, L. Zhang, X. Mou, and A. C. Bovik, “Gradient magnitude similarity deviation: A highly efficient perceptual image quality index,” _IEEE Transactions on Image Processing_ , vol. 23, no. 2, pp. 684–695, 2013.
  * [28] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “Fsim: A feature similarity index for image quality assessment,” _IEEE transactions on Image Processing_ , vol. 20, no. 8, pp. 2378–2386, 2011.
  * [29] R. Reisenhofer, S. Bosse, G. Kutyniok, and T. Wiegand, “A haar wavelet-based perceptual similarity index for image quality assessment,” _Signal Processing: Image Communication_ , vol. 61, pp. 33–43, 2018.
  * [30] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, vol. 2. Ieee, 2003, pp. 1398–1402.
  * [31] B. Zhang, P. V. Sander, and A. Bermak, “Gradient magnitude similarity deviation on multiple scales for color image quality assessment,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 1253–1257.
  * [32] J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 5197–5206.
  * [33] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” _IEEE transactions on image processing_ , vol. 19, no. 11, pp. 2861–2873, 2010.
  * [34] M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” 2012\.
  * [35] Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa, “Sketch-based manga retrieval using manga109 dataset,” _Multimedia Tools and Applications_ , vol. 76, no. 20, pp. 21 811–21 838, 2017.
  * [36] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in _European conference on computer vision_. Springer, 2016, pp. 391–407.
  * [37] _MATLAB version 9.3.0.713579 (R2017b)_ , The Mathworks, Inc., Natick, Massachusetts, 2017.
  * [38] A. Lugmayr, M. Danelljan, and R. Timofte, “Ntire 2020 challenge on real-world image super-resolution: Methods and results,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 494–495.
  * [39] Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, and L. Zelnik-Manor, “The 2018 pirm challenge on perceptual image super-resolution,” in _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , 2018, pp. 0–0.
  * [40] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 286–301.
  * [41] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 136–144.
  * [42] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * [43] T. Tong, G. Li, X. Liu, and Q. Gao, “Image super-resolution using dense skip connections,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4799–4807.
  * [44] X. Hu, H. Mu, X. Zhang, Z. Wang, T. Tan, and J. Sun, “Meta-sr: A magnification-arbitrary network for super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1575–1584.
  * [45] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2472–2481.
  * [46] Z. Hui, X. Wang, and X. Gao, “Fast and accurate single image super-resolution via information distillation network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 723–731.
  * [47] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 624–632.
  * [48] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” _arXiv preprint arXiv:1703.04977_ , 2017.
  * [49] Q. Ning, W. Dong, X. Li, J. Wu, and G. Shi, “Uncertainty-driven loss for single image super-resolution,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [50] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [51] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep residual networks for single image super-resolution,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , July 2017.
  * [52] W. Dong, P. Wang, W. Yin, G. Shi, F. Wu, and X. Lu, “Denoising prior driven deep neural network for image restoration,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 10, pp. 2305–2318, 2018\.
  * [53] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [54] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [55] S. Frolov, T. Hinz, F. Raue, J. Hees, and A. Dengel, “Adversarial text-to-image synthesis: A review,” _arXiv preprint arXiv:2101.09983_ , 2021\.
  * [56] Y. Yuan, S. Liu, J. Zhang, Y. Zhang, C. Dong, and L. Lin, “Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 701–710.
  * [57] Y. Wang, F. Perazzi, B. McWilliams, A. Sorkine-Hornung, O. Sorkine-Hornung, and C. Schroers, “A fully progressive approach to single-image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2018, pp. 864–873.
  * [58] L. Yu, X. Long, and C. Tong, “Single image super-resolution based on improved wgan,” in _Proceedings of the 2018 International Conference on Advanced Control, Automation and Artificial Intelligence (ACAAI 2018), Shenzhen, China_ , 2018, pp. 21–22.
  * [59] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of wasserstein gans,” _arXiv preprint arXiv:1704.00028_ , 2017.
  * [60] M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet, “Are gans created equal? a large-scale study,” _arXiv preprint arXiv:1711.10337_ , 2017\.
  * [61] K. Singla, R. Pandey, and U. Ghanekar, “A review on single image super resolution techniques using generative adversarial network,” _Optik_ , p. 169607, 2022.
  * [62] M. Vella and J. F. Mota, “Single image super-resolution via cnn architectures and tv-tv minimization,” _arXiv preprint arXiv:1907.05380_ , 2019.
  * [63] L. Gatys, A. S. Ecker, and M. Bethge, “Texture synthesis using convolutional neural networks,” _Advances in neural information processing systems_ , vol. 28, pp. 262–270, 2015.
  * [64] O. Levy and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” _Advances in neural information processing systems_ , vol. 27, pp. 2177–2185, 2014.
  * [65] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in _International Conference on Machine Learning_. PMLR, 2015, pp. 2256–2265.
  * [66] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” _arXiv preprint arXiv:2104.07636_ , 2021.
  * [67] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _arXiv preprint arXiv:2006.11239_ , 2020.
  * [68] P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” _arXiv preprint arXiv:2105.05233_ , 2021.
  * [69] Z. Li, J. Yang, Z. Liu, X. Yang, G. Jeon, and W. Wu, “Feedback network for image super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3867–3876.
  * [70] Y. Tai, J. Yang, and X. Liu, “Image super-resolution via deep recursive residual network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3147–3155.
  * [71] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1874–1883.
  * [72] W. Shi, J. Caballero, L. Theis, F. Huszar, A. Aitken, C. Ledig, and Z. Wang, “Is the deconvolution layer the same as a convolutional layer?” _arXiv preprint arXiv:1609.07009_ , 2016.
  * [73] Z. Wojna, V. Ferrari, S. Guadarrama, N. Silberman, L.-C. Chen, A. Fathi, and J. Uijlings, “The devil is in the decoder: Classification, regression and gans,” _International Journal of Computer Vision_ , vol. 127, no. 11, pp. 1694–1706, 2019.
  * [74] S. Kundu, H. Mostafa, S. N. Sridhar, and S. Sundaresan, “Attention-based image upsampling,” _arXiv preprint arXiv:2012.09904_ , 2020.
  * [75] Y. Jo and S. J. Kim, “Practical single-image super-resolution using look-up table,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 691–700.
  * [76] C. Peng, W.-A. Lin, H. Liao, R. Chellappa, and S. K. Zhou, “Saint: spatially aware interpolation network for medical slice synthesis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7750–7759.
  * [77] L. Chai, M. Gharbi, E. Shechtman, P. Isola, and R. Zhang, “Any-resolution training for high-resolution image synthesis,” _arXiv preprint arXiv:2204.07156_ , 2022.
  * [78] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [79] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [80] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [81] Y. Yang and Y. Qi, “Image super-resolution via channel attention and spatial graph convolutional network,” _Pattern Recognition_ , vol. 112, p. 107798, 2021.
  * [82] Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual non-local attention networks for image restoration,” _arXiv preprint arXiv:1903.10082_ , 2019\.
  * [83] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1833–1844.
  * [84] D. Zhang, F. Huang, S. Liu, X. Wang, and Z. Jin, “Swinfir: Revisiting the swinir with fast fourier convolution and improved training for image super-resolution,” _arXiv preprint arXiv:2208.11247_ , 2022.
  * [85] T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 065–11 074.
  * [86] H. Zhao, X. Kong, J. He, Y. Qiao, and C. Dong, “Efficient image super-resolution using pixel attention,” in _European Conference on Computer Vision_. Springer, 2020, pp. 56–72.
  * [87] B. Niu, W. Wen, W. Ren, X. Zhang, L. Yang, S. Wang, K. Zhang, X. Cao, and H. Shen, “Single image super-resolution via a holistic attention network,” in _European Conference on Computer Vision_. Springer, 2020, pp. 191–207.
  * [88] F. Wang, H. Hu, and C. Shen, “Bam: A lightweight and efficient balanced attention mechanism for single image super resolution,” _arXiv preprint arXiv:2104.07566_ , 2021.
  * [89] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 41–48.
  * [90] Y. Bei, A. Damian, S. Hu, S. Menon, N. Ravi, and C. Rudin, “New techniques for preserving global structure and denoising with low information loss in single-image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 874–881.
  * [91] N. Ahn, B. Kang, and K.-A. Sohn, “Image super-resolution via progressive cascading residual network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 791–799.
  * [92] A. Shocher, N. Cohen, and M. Irani, ““zero-shot” super-resolution using deep internal learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3118–3126.
  * [93] H. Ren, M. El-Khamy, and J. Lee, “Image super resolution based on fusing multiple convolution neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 54–61.
  * [94] K. Urazoe, N. Kuroki, Y. Kato, S. Ohtani, T. Hirose, and M. Numa, “Multi-category image super-resolution with convolutional neural network and multi-task learning,” _IEICE TRANSACTIONS on Information and Systems_ , vol. 104, no. 1, pp. 183–193, 2021.
  * [95] Y. Shi, K. Wang, C. Chen, L. Xu, and L. Lin, “Structure-preserving image super-resolution via contextualized multitask learning,” _IEEE transactions on multimedia_ , vol. 19, no. 12, pp. 2804–2815, 2017.
  * [96] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [97] Y. Tai, J. Yang, X. Liu, and C. Xu, “Memnet: A persistent memory network for image restoration,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4539–4547.
  * [98] S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3883–3891.
  * [99] J. Liu, J. Tang, and G. Wu, “Adadm: Enabling normalization for image super-resolution,” _arXiv preprint arXiv:2111.13905_ , 2021.
  * [100] M. Irani and S. Peleg, “Improving resolution by image registration,” _CVGIP: Graphical models and image processing_ , vol. 53, no. 3, pp. 231–239, 1991.
  * [101] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1646–1654.
  * [102] M. Aharon, M. Elad, and A. Bruckstein, “K-svd: An algorithm for designing overcomplete dictionaries for sparse representation,” _IEEE Transactions on signal processing_ , vol. 54, no. 11, pp. 4311–4322, 2006.
  * [103] K. He and J. Sun, “Convolutional neural networks at constrained time cost,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 5353–5360.
  * [104] G. Ghiasi and C. C. Fowlkes, “Laplacian pyramid reconstruction and refinement for semantic segmentation,” in _European conference on computer vision_. Springer, 2016, pp. 519–534.
  * [105] X. Mao, C. Shen, and Y.-B. Yang, “Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections,” _Advances in neural information processing systems_ , vol. 29, pp. 2802–2810, 2016.
  * [106] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [107] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [108] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [109] J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional network for image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1637–1645.
  * [110] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [111] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [112] W. Han, S. Chang, D. Liu, M. Yu, M. Witbrock, and T. S. Huang, “Image super-resolution via dual-state recurrent networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1654–1663.
  * [113] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Gated feedback recurrent neural networks,” in _International conference on machine learning_. PMLR, 2015, pp. 2067–2075.
  * [114] A. R. Zamir, T.-L. Wu, L. Sun, W. B. Shen, B. E. Shi, J. Malik, and S. Savarese, “Feedback networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1308–1317.
  * [115] D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang, “Non-local recurrent network for image restoration,” _arXiv preprint arXiv:1806.02919_ , 2018\.
  * [116] T. Isobe, X. Jia, S. Gu, S. Li, S. Wang, and Q. Tian, “Video super-resolution with recurrent structure-detail network,” in _European Conference on Computer Vision_. Springer, 2020, pp. 645–660.
  * [117] S. Park, J. Yoo, D. Cho, J. Kim, and T. H. Kim, “Fast adaptation to super-resolution networks via meta-learning,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16_. Springer, 2020, pp. 754–769.
  * [118] N. Ahn, B. Kang, and K.-A. Sohn, “Fast, accurate, and lightweight super-resolution with cascading residual network,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 252–268.
  * [119] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [120] J. Liu, J. Tang, and G. Wu, “Residual feature distillation network for lightweight image super-resolution,” in _European Conference on Computer Vision_. Springer, 2020, pp. 41–55.
  * [121] J. Liu, W. Zhang, Y. Tang, J. Tang, and G. Wu, “Residual feature aggregation network for image super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2359–2368.
  * [122] Z. Hui, X. Gao, Y. Yang, and X. Wang, “Lightweight image super-resolution with information multi-distillation network,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , 2019, pp. 2024–2032.
  * [123] M. Ayazoglu, “Extremely lightweight quantization robust real-time single-image super resolution for mobile devices,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2472–2479.
  * [124] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with low precision weights and activations,” _The Journal of Machine Learning Research_ , vol. 18, no. 1, pp. 6869–6898, 2017.
  * [125] A. F. Agarap, “Deep learning using rectified linear units (relu),” _arXiv preprint arXiv:1803.08375_ , 2018.
  * [126] I. H. Sarker, “Deep learning: a comprehensive overview on techniques, taxonomy, applications and research directions,” _SN Computer Science_ , vol. 2, no. 6, pp. 1–20, 2021.
  * [127] M. Stephane, “A wavelet tour of signal processing,” 1999.
  * [128] T. Guo, H. Seyed Mousavi, T. Huu Vu, and V. Monga, “Deep wavelet prediction for image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 104–113.
  * [129] F. Sahito, P. Zhiwen, J. Ahmed, and R. A. Memon, “Wavelet-integrated deep networks for single image super-resolution,” _Electronics_ , vol. 8, no. 5, p. 553, 2019.
  * [130] H. Huang, R. He, Z. Sun, and T. Tan, “Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1689–1697.
  * [131] P. Liu, H. Zhang, K. Zhang, L. Lin, and W. Zuo, “Multi-level wavelet-cnn for image restoration,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2018, pp. 773–782.
  * [132] Q. Zhang, H. Wang, T. Du, S. Yang, Y. Wang, Z. Xing, W. Bai, and Y. Yi, “Super-resolution reconstruction algorithms based on fusion of deep learning mechanism and wavelet,” in _Proceedings of the 2nd International Conference on Artificial Intelligence and Pattern Recognition_ , 2019, pp. 102–107.
  * [133] S. Xue, W. Qiu, F. Liu, and X. Jin, “Wavelet-based residual attention network for image super-resolution,” _Neurocomputing_ , vol. 382, pp. 116–126, 2020\.
  * [134] X. Zhu, Z. Li, J. Lou, and Q. Shen, “Video super-resolution based on a spatio-temporal matching network,” _Pattern Recognition_ , vol. 110, p. 107619, 2021.
  * [135] A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, and L. Van Gool, “Wespe: weakly supervised photo enhancer for digital cameras,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 691–700.
  * [136] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [137] A. Bulat, J. Yang, and G. Tzimiropoulos, “To learn image super-resolution, use a gan to learn how to do image degradation first,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 185–200.
  * [138] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9446–9454.
  * [139] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 9, pp. 2251–2265, 2018.
  * [140] X. Cheng, Z. Fu, and J. Yang, “Zero-shot image super-resolution with depth guided internal degradation learning,” in _European Conference on Computer Vision_. Springer, 2020, pp. 265–280.
  * [141] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self-supervised monocular depth estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3828–3838.
  * [142] J. W. Soh, S. Cho, and N. I. Cho, “Meta-transfer learning for zero-shot super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3516–3525.
  * [143] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Fast and accurate image super-resolution with deep laplacian pyramid networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 11, pp. 2599–2613, 2018.
  * [144] D. Song, C. Xu, X. Jia, Y. Chen, C. Xu, and Y. Wang, “Efficient residual dense block search for image super-resolution,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 12 007–12 014.