# Revisiting $\ell_{1}$ Loss in Super-Resolution: A Probabilistic View and
Beyond

Xiangyu He
NLPR, CASIA
<EMAIL_ADDRESS>Jian Cheng
NLPR, CASIA
<EMAIL_ADDRESS>
###### Abstract

Super-resolution as an ill-posed problem has many high-resolution candidates
for a low-resolution input. However, the popular $\ell_{1}$ loss used to best
fit the given HR image fails to consider this fundamental property of non-
uniqueness in image restoration. In this work, we fix the missing piece in
$\ell_{1}$ loss by formulating super-resolution with neural networks as a
probabilistic model. It shows that $\ell_{1}$ loss is equivalent to a degraded
likelihood function that removes the randomness from the learning process. By
introducing a data-adaptive random variable, we present a new objective
function that aims at minimizing the expectation of the reconstruction error
over all plausible solutions. The experimental results show consistent
improvements on mainstream architectures, with no extra parameter or computing
cost at inference time.

##  1 Introduction

The behavior of single image super-resolution (SISR) networks is primarily
driven by the choice of the objective function. Two common examples in SISR
are the mean squared error (MSE) loss and the mean absolute error (MAE) loss.
Since the long-tested measure peak signal-to-noise ratio (PSNR) is essentially
defined on the MSE, pioneer works [20, 38, 21, 11, 66, 41] have largely
focused on minimizing the mean squared reconstruction error to achieve high
performances.

Due to the ill-posed nature of SISR problem, a low-resolution image
$\mathbf{x}$ can be well described as the downsampling result of many high-
resolution images $\mathbf{\hat{y}}_{1:k}$. This implies that, in the inverse
image reconstruction, the neural network trained by MSE loss, i.e.,
$\mathbb{E}_{(\mathbf{x},\mathbf{\hat{y}})}[\sum_{i}\|f(\mathbf{x})-\mathbf{\hat{y}}_{i}\|_{2}^{2}]$,
will learn to output the mean value of all HR targets, i.e.,
$f(\mathbf{x})\rightarrow\frac{1}{k}\sum_{i}\mathbf{\hat{y}}_{i}$, which best
fits the image pairs $\\{\mathbf{x},\mathbf{\hat{y}}_{1:k}\\}$ [56]. However,
the average of different HR images results in the absense of distinct edges
and the blurring effect, which suffers from low image fidelity despite
relatively high PSNR. Recently, $\ell_{1}$ loss has received much attention
from low-level vision researches, since it is in practice more robust to the
regression-to-the-mean problem in MSE loss [81, 47]. Despite its emprical
success in state-of-the-arts super-resolution methods [79, 17, 58, 55, 48,
82], $\ell_{1}$ loss itself has been rarely discussed in the context of image
processing.

Figure 1: Illustration of the learning process given an LR-HR image pair.
$\ell_{1}$ loss only considers the fitting error between model output
$f(\mathbf{x})$ and HR image $\mathbf{\hat{y}}$. We propose to learn an one-
to-many mapping by introducing $P(\mathbf{y}|\mathbf{x};\mathbf{W})$
explicitly: 1) first perform a random sampling from
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$ to obtain a underlying natural image
$\mathbf{y}$; 2) then maximize the likelihood of $\mathbf{y,\hat{y}}$ co-
occurrence. Since we focus on the expected value averaged across all
$\mathbf{y}$ instead of any specific sampling result, the learning process
serves as a multiple-valued mapping.

In probabilistic setting, it is well known that a median is the minimizer of
$\ell_{1}$ loss with respect to given targets [63]. In the case of training
neural networks over LR-HR image pairs, the situation is quite similar to MSE
loss where the network will learn to recover the median of all perceptually
convincing HR images $\mathbf{\hat{y}}_{1:k}$. However, both $\ell_{1}$ and
$\ell_{2}$ loss only capture the median/mean value of the whole solution
space, which can be inadequate for estimating any specific solution in high-
dimensional predictions, super-resolution in particular.

In this paper, we begin with a simple idea that it is intuitive to learn a
one-to-many mapping since there are plenty of perceptually convincing
solutions to an LR input (Figure 1). Ideally, we may approximate the local
manifold, corresponding to the ground-truth HR image, by the posterior
distribution $P(\mathbf{y}|\mathbf{x};\mathbf{W})$. Notice that, generative
methods are able to capture the more complex distribution of natural images,
which inspires us to revisit super-resolution with neural networks from a
probabilistic view. In this way, we have the following contributions:

  * •

We derive a loss function for solving SISR by introducing a posterior Gaussian
distribution of underlying natural image $\mathbf{y}$, i.e.,
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$, which minimizes the expected
reconstruction error across all plausible solutions in $\mathbf{y}$. The
formulated learning process can be easily extended to other
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$ with different distributions such as
Laplace.

  * •

The learned standard deviation of $P(\mathbf{y}|\mathbf{x};\mathbf{W})$
facilitates the HR image regression and, as a by-product, allows us to make
meaningful estimates of the model uncertainty in SISR.

  * •

The experimental results show consistent improvements over popular $\ell_{1}$
loss on mainstream networks, without the extra cost at runtime.

##  2 Related Work

Super-resolution Over the past decades, super-resolution is among the most
fundamental low-level vision problems. Early works often formulate the task as
an interpolation [3, 45, 78, 44]. Latter studies start to reconstruct high-
resolution images in a data-driven way, by estimating the natural image
statistics [64, 65] (including deep prior [70, 7]) and exploiting the neighbor
embedding [13, 27, 5]. Another group of methods utilizes the self-similarities
to learn the inverse mapping [24, 29, 71, 72]. Benefiting from the development
of sparse coding, [68, 74, 22] achieve reasonable results on benchmark
datasets.

Advances in deep learning have further enhanced the state-of-the-art
performance since [20, 21] first introduce a CNN-based SR method. Various CNN
architectures have been proposed such as residual networks [37, 38, 47],
densely connected networks [80, 69], ode-inspired design [30] and U-Net
encoder-decoder networks [50, 16, 49]. In particular, the attention mechanism
has become a popular tool for image super-resolution. Both channel attention
[79] and non-local technique [58, 17] achieve notable improvements over
baseline methods. Recently, self-correlation/self-attention that models the
long-distance relationship among similar patches have led to successful
results [82, 55, 54], as the large training dataset can be used as a means to
capture the global information [14, 46, 73]. Our approach can facilitate the
training of deep learning-based methods in a plug-in and play manner.

Loss functions Most learning objects in the literature fall into two main
categories: pixel-wise loss and perceptual loss. The former aims to optimize
the full reference metric such as PSNR and SSIM. MSE loss, in particular, is
equivalent to PSNR. However, [23, 36, 52] show that $\ell_{2}$ loss results in
overly-smooth SR images, and the following works tend to use $\ell_{1}$ during
training. In light of the success of knowledge distillation in image
recognition, [42, 26] use loss functions in the feature space to enhance the
performance of student networks. Our work also falls into this category. We
propose to learn from a posterior distribution instead of best-fitting ground-
truth images.

Perceptual loss tackles the problem by employing generative adversarial
networks [52, 18], especially in super-resolution with large upscaling factors
(e.g., $4\times$, $8\times$) [75, 41]. The ImageNet pre-trained networks are
commonly used as a discriminator loss [36, 23, 7], which leads to high-
frequency details and perceptually satisfying results in the sense. Our
approach can be compatible with those adversarial settings as the content loss
term in a perceptual loss.

The most related work is Noise2noise [43], which suffers from inaccurate
gradient directions (details in Section 3.3). We solve this problem by a well-
bounded gradient step and point out that [43] indeed serves as a special case
of our approach. Noise2Self [4] assumes that noise in each patch is
independent of other patches, which can not hold responsible when it comes to
the content of images instead of noise. Self2Self [60] presents dropout in
both image and feature space to reduce prediction variances, which is also
compatible with our loss functions.

Model uncertainty Despite their unprecedented power to solve the inverse
problem, deep learning-based SR methods are prone to make mistakes. As deep
models are widely used in computer-aided diagnosis [76, 15], it is crucial to
better understand the confidence of neural networks in their predictions. That
is, the SR network may provide uncertainty about the generated super-resolved
images: which part is “real” and when we should be cautious.

[28] first develops a framework for modeling uncertainty, which is hard to
describe the modern deep neural networks. Recently, Monte Carlo Dropout [25]
presents a practical solution to estimate uncertainty via the dropout layer.
[67] further shows that networks trained with Batch-Normalization [35] (BN) is
an approximate Bayesian model and allows an uncertainty estimation by sampling
the network’s BN parameters. Unfortunately, neither of them was used in recent
SR networks, which disables the variational inference in those approximate
Bayesian models. Instead, we model the posterior distribution
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$ explicitly, as a by-product, the
estimated standard deviation facilitates the uncertainty estimation for SISR.

##  3 Analysis

###  3.1 Probabilistic Modeling

From a statistical viewpoint, we are interested in the point estimation of the
model parameters $\mathbf{W}$, used in the following likelihood function:

$\displaystyle\max_{\mathbf{W}}\
\mathcal{L}(\mathbf{W}|\mathbf{\hat{y}})=P(\mathbf{\hat{y}}|\mathbf{x};\mathbf{W})$
(1)

where $\mathbf{W}$, $\mathbf{\hat{y}}$ and $\mathbf{x}$ indicate weights,
observed ground-truth labels and inputs respectively. Super-resolution using
convolutional neural networks also fall into this form, that learning
parameters $\mathbf{W}$ w.r.t. an inference function $f(\mathbf{x})$ from LR-
HR image pairs directly.

In contrast to finding a one-to-one mapping between corrupted images and
targets implied by $f(\mathbf{x})$, we begin with a subtle point: in reality,
the mapping from $\mathbf{\hat{y}}$ to $\mathbf{x}$ is surjective. For
example, a single LR image $\mathbf{x}$ may correspond to the downsampling
result of multiple natural images. Hence, it is intuitive to recover or
approximate the local image manifold instead of simply fitting the high-
resolution image $\mathbf{\hat{y}}$.

Objective function The idea is to explicitly estimate a posterior distribution
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$ where $\mathbf{y}$ refers to (multiple)
underlying natural images with the size of $H\times W$. Then, the observed HR
image $\mathbf{\hat{y}}$ should look like its counterpart $\mathbf{y}$ sampled
from $P(\mathbf{y}|\mathbf{x};\mathbf{W})$, with high probability. Formally,
our goal is to maximize the following conditional joint density function:

$P(\mathbf{\hat{y}},\mathbf{y}|\mathbf{x};\mathbf{W})=P(\mathbf{\hat{y}|y})P(\mathbf{y}|\mathbf{x};\mathbf{W})$
(2)

where $P(\mathbf{\hat{y}|y})$ is a measure of the probability that
$\mathbf{\hat{y}}$ being observed given $\mathbf{y}$. Note that, there are
plenty of $\mathbf{y}$ can serve as the perceptually convincing solution to
Eq.(2) then, as is common in machine learning, we take the expectation of that
formula to be optimized:

$\displaystyle\mathbb{E}_{\mathbf{y}\sim
P(\mathbf{y}|\mathbf{x};\mathbf{W})}[P(\mathbf{\hat{y}|y})].$ (3)

To solve Eq.(3), we will first need to deal with
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$, the distribution of underlying natural
images $\mathbf{y}$ consistent with the low-resolution $\mathbf{x}$.

Optimizing the objective Based on the finding of non-local means [8, 9] that
each pixel is obtained as a weighted average of pixels centered at regions
similar to the reference and the central limit theorem (CLT), we shall model
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$ as a multivariate normal
distribution111The distribution of $P_{\mathbf{y|x}}$ is not required to be
Gaussian: for instance, if $\mathbf{y}$ is in practice more-Laplace, then
$P_{\mathbf{y|x}}$ might be re-parameterized as
$\bm{\mu}-\bm{\sigma}*\mathrm{sgn}(\mathbf{z})*\mathrm{ln}(1-2|\mathbf{z}|)$,
$\mathbf{z}\sim\mathcal{N}(\bm{0},\bm{1})$.:

$P(\mathbf{y}|\mathbf{x};\mathbf{W})\sim\mathcal{N}(\bm{\mu}_{(\mathbf{x};\mathbf{W})};\bm{\Sigma}_{(\mathbf{x};\mathbf{W})})$
(4)

where mean $\bm{\mu}$ and covariance $\bm{\Sigma}$ are outputs of multi-layer
neural networks with parameters $\mathbf{W}$. $\bm{\sigma}^{2}$ corresponds to
the diagonal elements in $\bm{\Sigma}$. However, the expectation in Eq.(3)
still involves sampling $\mathbf{y}$ from
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$ which is a non-differentiable operation.
To allow the backprop through $\mathbf{y}$, we use one of the most common
trick in machine learning, the “reparameterization” [40], to move the sampling
to the random variable $\mathbf{z}\sim\mathcal{N}(\bm{0},\bm{1})$:

$\mathbb{E}_{\mathbf{y}\sim
P(\mathbf{y}|\mathbf{x};\mathbf{W})}[P(\mathbf{\hat{y}|y})]=\mathbb{E}_{\mathbf{z}\sim\mathcal{N}(\bm{0},\bm{1})}[P(\mathbf{\hat{y}|\bm{\mu}+\bm{\sigma}*z})]$

where $\mathbf{y}$ is reformulated as $\bm{\mu}+\bm{\sigma}*\mathbf{z}$ and
“$*$” indicates elementwise product. Then, we can conduct the standard
gradient descent and the gradient averaged over $\mathbf{z}$ should converge
to the actual gradient of Eq.(3).

Similarly, as [7], we consider the conditional model defined by the Gibbs
distribution (also known as Boltzmann distribution):

$P(\mathbf{\hat{y}|y})\varpropto\prod_{i}^{H\times
W}\mathrm{exp}(-\frac{|\mathbf{\hat{y}}_{i}-\mathbf{y}_{i}|}{kT})$ (5)

where the corresponding Gibbs energy is in the form of
$\|\mathbf{\hat{y}}-\mathbf{y}\|_{1}$ and $kT$ is the fixed partition
function. The model (5) assumes that $\mathbf{y}$, looks like the observed
ground-truth image $\mathbf{\hat{y}}$, is likely to be a strong evidence for
$\mathbf{\hat{y}}$ occurring. We can now get both
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$ and $P(\mathbf{\hat{y}|y})$ into Eq.(3)
then apply negative log-likelihood to the final objective for computational
convenience:

$\displaystyle\min\
\mathbb{E}_{\mathbf{z}}\Big{[}\frac{1}{kT}\sum_{i}^{H\times
W}|\mathbf{\hat{y}}_{i}-(\bm{\mu}_{i}+\bm{\sigma}_{i}*\mathbf{z})|\Big{]}.$
(6)

This equation implies a two-branch network ($\bm{\mu}$ and $\bm{\sigma}$) that
requires randomness during training. The expectation on the data-independent
random variable $\mathbf{z}$ allows us to move the gradient symbol into the
expectation to finish the standard stochastic gradient descent [19]. For
notational simplicity, we shall in the remaining text denote Eq.(6) as
$\mathbb{E}_{\mathbf{z}}\Big{[}\|\mathbf{\hat{y}}-(\bm{\mu}+\bm{\sigma}*\mathbf{z})\|_{1}\Big{]}$.

###  3.2 Revisiting $\ell_{1}$ Loss

According to Jensen’s inequality, for any measurable convex function
$f(\cdot)$, we have $\mathbb{E}[f(\mathbf{z})]\geq f(\mathbb{E}[\mathbf{z}])$.
Since every $p$-norm is convex and $\mathbf{z}\sim\mathcal{N}(\bm{0},\bm{1})$,
then we have

$\displaystyle\mathbb{E}_{\mathbf{z}}\Big{[}\|\mathbf{\hat{y}}-(\bm{\mu}+\bm{\sigma}*\mathbf{z})\|_{1}\Big{]}$
$\displaystyle\geq\|\mathbf{\hat{y}}-(\bm{\mu}+\bm{\sigma}*\mathbb{E}_{\mathbf{z}}[\mathbf{z}])\|_{1}$
$\displaystyle=\|\mathbf{\hat{y}}-\bm{\mu}\|_{1}.$

Notice that the proposed learning target (6) actually serves as an upper bound
of the popular $\ell_{1}$ loss function. In other words, $\ell_{1}$ loss also
suffers from the degradation that loses the randomness during training and is
confined to the given HR images $\mathbf{\hat{y}}$. Though the optimal
solution for the original function and for the upper bound are indeed
different, the probabilistic learning object can relax the deterministic
learning process at each step by focusing on the expectation of the loss
function.

(a) R channel

(b) G channel

(c) B channel

Figure 2: The degradation curves of $\bm{\sigma}_{i}$ during training. When
$\bm{\sigma}_{i}$ approaching zero, Eq.(6) is equivalent to $\ell_{1}$ loss.

Owing to the reparameterization trick [40], learning probabilistic model via
Eq.(6) seems straightforward. However, there exists a trivial solution to
easily bridge the Jensen gap (i.e.,
$\mathbb{E}[f(\mathbf{z})]=f(\mathbb{E}[\mathbf{z}])$) such that
$\bm{\sigma}\in\vec{\mathbf{0}}$. As shown in Figure 2, the output of
$\bm{\sigma}$ branch gradually approaches zero which ultimately eliminates the
randomness in our learning target. Similar degradation also occurs in
Variational Auto-Encoder (VAE). In a nutshell, a VAE is an auto-encoder when
latent variable $\mathbf{z}$ that determines encodings distribution lose
randomness222VAE regularises its random latent variable $\mathbf{z}$ towards
Normal distribution that enables generative process [19]. Otherwise, the
latent space will converge to a single point instead of a distribution. That
is, VAE degrades into an one-to-one mapping given input images, i.e., an
(deterministic) auto-encoder.. In light of this, we focus on avoiding the
degradation of $\bm{\sigma}$ in the following section.

###  3.3 Beyond $\ell_{1}$ Loss

Data-independent $\bm{\sigma}$ The simplest way is to remove the trainable
$\bm{\sigma}$ branch and let $\bm{\sigma}_{i}$ be a small constant $k$
invisible to the human eye. Then,
$P(\mathbf{y}|\mathbf{x};\mathbf{W})\sim\mathcal{N}(\bm{\mu};k^{2}\bm{I})$
where $\bm{I}$ refers to an identity matrix. Contracting this term into Eq.(6)
and rearranging yields:

$\mathbb{E}_{\mathbf{z}}\Big{[}\|\mathbf{\hat{y}}-(\bm{\mu}+k\cdot\mathbf{z})\|_{1}\Big{]}=\mathbb{E}_{\mathbf{z}}\Big{[}\|\underbrace{\mathbf{(\hat{y}}+\mathbf{z^{\prime}})}_{\mathrm{Noised\
HR}}-\bm{\mu}\|_{1}\Big{]}$ (7)

where $\mathbf{z^{\prime}}$ is an additive Gaussian noise $\mathcal{N}(0,k)$
and $k\neq 0$. The left hand side of this equation is something we hope to
optimize via backward propagation. The right hand side serves is the core of
Noise2Noise [43] that restores corrupted images without clean data, i.e.,
learning super-resolution by only looking at noised HR images.

Though the performance of training on corrupted observations may approach its
clean counterpart [43], the noise term can do harm to the optimization
process. To be specific, we expect $\bm{\mu}$ to approach $\mathbf{\hat{y}}$
via stochastic gradient descent. However, for some $\mathbf{z^{\prime}}$
satisfying
$(\mathbf{(\hat{y}}_{i}-\bm{\mu}_{i})+\mathbf{z^{\prime}})(\mathbf{\hat{y}}_{i}-\bm{\mu}_{i})<0$,
we have

$\displaystyle\mathrm{sgn}(\frac{\partial\|\mathbf{(\hat{y}}+\mathbf{z^{\prime}})-\bm{\mu}\|_{1}}{\partial\bm{\mu}_{i}})=-1\cdot\mathrm{sgn}(\frac{\partial\|\mathbf{\hat{y}}-\bm{\mu}\|_{1}}{\partial\bm{\mu}_{i}})$

where $\mathbf{z^{\prime}}$ results in gradient ascent instead of descent,
which ultimately hinders $\bm{\mu}$ from fitting $\mathbf{\hat{y}}$. To get a
good estimate of the proper gradient, it would require sampling massive noised
observations given a single clean image, which would be expensive. Hence, we
hope to find some $\bm{\sigma}$ that run correct backprop efficiently and
retain randomness.

Data-adaptive $\bm{\sigma}$ Fortunately, a delicate $\bm{\sigma}$ gives a
definite answer to both. The full equation we want to optimize is:

$\displaystyle\min_{\bm{\mu}}\
\mathbb{E}_{\mathbf{z}}\Big{[}\big{\|}\mathbf{\hat{y}}-(\bm{\mu}+|\mathbf{\hat{y}}-\bm{\mu}|*\mathbf{z})\big{\|}_{1}\Big{]}$
(8)

where $|\mathbf{\hat{y}}-\bm{\mu}|$ corresponds to the $\bm{\sigma}$ term and
$|\cdot|$ refers to element-wise absolute function. We shall denote Eq.(8) as
$\ell_{\mathbb{E}}$ in the remaining text for simplicity.

From the view of Eq.(7), $\ell_{\mathbb{E}}$ is also in the form of adding
some (data-adaptive) noise to HR images. However, for Eq.(8), we allow the
backprop through $|\mathbf{\hat{y}}-\bm{\mu}|$ instead of regarding it as a
scaling factor, which contributes to the following property: For any
$\mathbf{z}\sim\mathcal{N}(0,1)$, we have

$\displaystyle\mathrm{sgn}(\frac{\partial\ell_{\mathbb{E}}}{\partial\bm{\mu}_{i}})=\mathrm{sgn}(\frac{\partial\|\mathbf{\hat{y}}-\bm{\mu}\|_{1}}{\partial\bm{\mu}_{i}}),\quad\Big{|}\mathbb{E}_{\mathbf{z}}\big{[}\frac{\partial\ell_{\mathbb{E}}}{\partial\bm{\mu}_{i}}\big{]}\Big{|}\approx
1.$ (9)

Note that the gradient direction is always making $\bm{\mu}$ approach
$\mathbf{\hat{y}}$ and the step-size converges to nearly one. Therefore, we
can safely average the gradient of $\nabla_{\mathbf{W}}\ell_{\mathbb{E}}$ over
arbitrarily many samplings of $\mathbf{z}$ and training images $\mathbf{x}$,
without the risk of gradient ascent or the problem of vanishing/exploding
gradient. Due to the ill-posed nature of super-resolution,
$|\mathbf{\hat{y}}-\bm{\mu}|$ can never be a zero vector then the randomness
of (8) will be preserved.

Figure 3: An illustration of the proposed learning scheme and the sigma
branch. The newly added branch includes two operations: four Conv+PReLU blocks
serve as a mapping function and the upsampler (Pixelshuffle coupled with
$3\times 3$ convolution) adjusts the output size. “$\ell_{\mathbb{E}}$” and
“Auxiliary Loss” correspond to Eq.(8) and Eq.(10) respectively. We use
$\bm{\mu}$ as the super-resolved image at inference time.

It is shown that $|\mathbf{\hat{y}}-\bm{\mu}|$ plays a core role in learning
$\bm{\mu}$ and serves as $\bm{\sigma}$ in $\ell_{\mathbb{E}}$ loss, which
inspires us to re-fetch the $\bm{\sigma}$ branch to perform the following
multi-task learning except $\ell_{\mathbb{E}}$ loss:

$\displaystyle\min_{\bm{\sigma}}\
\beta\cdot\big{\|}|\mathbf{\hat{y}}-\bm{\mu}|-\bm{\sigma}\big{\|}_{1}$ (10)

where $\beta$ is a penalty factor and $\bm{\mu}$ is detached from backward
propagation. This equation also, as a by-product, allows us to make a rough
estimate of the model uncertainty in SISR via the prediction of $\bm{\sigma}$,
e.g., small $\bm{\sigma}_{i}$ means we can be certain of the super-resolved
image produced by neural networks, otherwise should be less confident (details
in Section 4.6). Finally, Figure 3 further illustrates the overall structure
of the proposed scheme.

##  4 Experiments

###  4.1 Dataset

For Bicubic image SR, following the common setting in [47, 30, 2, 82, 17, 58],
we conduct all experiments on the 1st-800th images from DIV2K [1] training set
then evaluate our models on five benchmarks: Set5 [6], Set14 [77], B100 [51],
Urban100 [32] and Manga109 [53] with scale factors $2\times$, $3\times$,
$4\times$. To make a fair comparison with early works trained on the tiny
dataset [68], we also cite the reproduced DIV2K results [42] in Table 4. For
real-world SR, we use the same dataset and training settings proposed in
RealSR benchmark [12]. We report PSNR and SSIM on the Y channel in YCbCr space
and ignore the same amount of pixels as scales from the border.

###  4.2 Training Details

We use low-resolution RGB image patches with the size of $48\times 48$ for
training. All HR-LR image pairs are randomly rotated by $90^{\circ}$,
$180^{\circ}$, $270^{\circ}$ and flipped horizontally [47]. We set the mini-
batch size to 16. Our models are trained by Adam optimizer [39] with
$\beta_{1}=0.9$, $\beta_{2}=0.999$, and $\epsilon=10^{-8}$. The initial
learning rate is set to $10^{-4}$ and decreases to half at every $2\times
10^{5}$ mini-batch updates. The total training cost is $10^{6}$ iterations
[47]. For RealSR, as described in [12], the training patch size is $192\times
192$ and learning rate is fixed at $10^{-4}$. Other details are the same as
classical SR. We use the PyTorch framework [59] to implement our methods with
an NVIDIA Tesla V100 GPU.

Table 1: EDSR-baseline [47] results on Set14 ($2\times$) for different
learning targets. We use the best setting in the following experiments.

$\ell_{1}$ | Eq.(8) | Eq.(10) | PSNR (dB) | SSIM | Training cost
---|---|---|---|---|---
✓ |  |  | 33.57 | 0.9175 | 2.00 ms/iter
| ✓ |  | 33.68 | 0.9181 | 2.06 ms/iter
|  | ✓ | 33.57 | 0.9176 | 2.16 ms/iter
✓ |  | ✓ | 33.64 | 0.9180 | 2.71 ms/iter
| ✓ | ✓ | 33.71 | 0.9185 | 2.77 ms/iter

Table 2: Ablation study on the penalty factor $\beta$. (EDSR-baseline [47] on
Set14 $2\times$)

$\beta$ | 0.001 | 0.005 | 0.01 | 0.05 | 0.1
---|---|---|---|---|---
PSNR (dB) | 33.605 | 33.641 | 33.707 | 33.663 | 33.647
SSIM | 0.9176 | 0.9181 | 0.9185 | 0.9184 | 0.9178

Figure 4: Trade-off between the model size and the average PSNR on Set5 [6]
(3$\times$). The marker size indictaes the number of parameters. The running
time is measured on an 720P SR image. (Best viewed in color.)

###  4.3 Ablation Studies

We first present an ablation analysis on each object function in our learning
process and report quantitative results in terms of average PSNR and SSIM on
Set14 [77] with the scale factor of 2. The baseline results in Table 1 (i.e.,
first row) are obtaind using EDSR pre-trained model
333https://cv.snu.ac.kr/research/EDSR/models/edsr_baseline_x2-1bc95232.pt. In
the case of training neural networks by Eq.(10) (i.e., third row), we run the
backpropagation though both $\bm{\mu}$ and $\bm{\sigma}$ to allow the
prediction of super-resolved images via $\bm{\mu}$ branch. The following rows
illustrate that both $\ell_{\mathbb{E}}$ loss (Eq.(8)) and the multi-task
learning (Eq.(10)) contribute to higher performances than $\ell_{1}$ baseline.
When applying both settings to the original network, we can further improve
the PSNR and SSIM results. In the following experiments, we use
“Eq.(8)+Eq.(10)” as the learning target.

Table 3: Quantitative comparison with lightweight models. Our approach
improves the performance of early baseline methods to the state-of-the-art. It
is shown that networks with simple structures such as EDSR [47] can be
competitive with recent elaborate-designed networks. When coupled with
efficient models (e.g., CARN [2], IDN [34]), the improvement is consistent as
shown in Table 4.

Scale | Methods | #Param. | MAC | Runtime |  Set5 [6] |  Set14 [77] |  B100 [51] |  Urban100 [32]
---|---|---|---|---|---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
2$\times$ |  VDSR [37] | 668K | 614.7G | 120.8ms | 37.53 | 0.9587 | 33.03 | 0.9124 | 31.90 | 0.8960 | 30.76 | 0.9140
VDSR+Ours | 37.83 | 0.9600 | 33.40 | 0.9158 | 32.05 | 0.8983 | 31.59 | 0.9226
EDSR-baseline [47] | 1370K | 316.2G | 56.5ms | 37.99 | 0.9604 | 33.57 | 0.9175 | 32.16 | 0.8994 | 31.98 | 0.9272
EDSR-baseline+Ours | 38.02 | 0.9606 | 33.71 | 0.9185 | 32.18 | 0.9001 | 32.19 | 0.9292
CARN [2] | 964K | 223.4G | 56.7ms | 37.76 | 0.9590 | 33.52 | 0.9166 | 32.09 | 0.8978 | 31.92 | 0.9256
IMDN [33] | 694K | 159.6G | 43.2ms | 38.00 | 0.9605 | 33.63 | 0.9177 | 32.19 | 0.8996 | 32.17 | 0.9283
OISR [30] | 1372K | 316.2G | 68.8ms | 38.02 | 0.9605 | 33.62 | 0.9178 | 32.20 | 0.9000 | 32.21 | 0.9290
3$\times$ |  VDSR [37] | 668K | 614.7G | 117.3ms | 33.66 | 0.9213 | 29.77 | 0.8314 | 28.82 | 0.7976 | 27.14 | 0.8279
VDSR+Ours | 34.18 | 0.9249 | 30.18 | 0.8395 | 28.99 | 0.8028 | 27.80 | 0.8436
EDSR-baseline [47] | 1555K | 160.1G | 30.4ms | 34.37 | 0.9270 | 30.28 | 0.8417 | 29.09 | 0.8052 | 28.15 | 0.8527
EDSR-baseline+Ours | 34.41 | 0.9271 | 30.38 | 0.8435 | 29.12 | 0.8062 | 28.22 | 0.8543
CARN [2] | 1149K | 118.9G | 30.7ms | 34.29 | 0.9255 | 30.29 | 0.8407 | 29.06 | 0.8034 | 28.06 | 0.8493
IMDN [33] | 703K | 71.7G | 22.4ms | 34.36 | 0.9270 | 30.32 | 0.8417 | 29.09 | 0.8046 | 28.17 | 0.8519
OISR [30] | 1557K | 160.1G | 36.2ms | 34.39 | 0.9272 | 30.35 | 0.8426 | 29.11 | 0.8058 | 28.24 | 0.8544
4$\times$ |  VDSR [37] | 668K | 614.7G | 121.4ms | 31.35 | 0.8838 | 28.01 | 0.7674 | 27.29 | 0.7251 | 25.18 | 0.7524
VDSR+Ours | 31.89 | 0.8900 | 28.45 | 0.7778 | 27.46 | 0.7327 | 25.76 | 0.7734
EDSR-baseline [47] | 1518K | 114.2G | 21.6ms | 32.09 | 0.8938 | 28.58 | 0.7813 | 27.57 | 0.7357 | 26.04 | 0.7849
EDSR-baseline+Ours | 32.15 | 0.8933 | 28.65 | 0.7831 | 27.59 | 0.7377 | 26.15 | 0.7885
CARN [2] | 1112K | 91.1G | 22.5ms | 32.13 | 0.8937 | 28.60 | 0.7806 | 27.58 | 0.7349 | 26.07 | 0.7837
IMDN [33] | 715K | 41.1G | 11.6ms | 32.21 | 0.8948 | 28.58 | 0.7811 | 27.56 | 0.7353 | 26.04 | 0.7838
OISR [30] | 1520K | 114.2G | 24.9ms | 32.14 | 0.8947 | 28.63 | 0.7819 | 27.60 | 0.7369 | 26.17 | 0.7888

Table 2 indicates that a proper $\beta$ matters in the balance between
reconstruction loss $\ell_{\mathbb{E}}$ and the auxiliary term. Fortunately,
$0.01$ seems to perform well in most cases. For simplicity, we set the channel
number of the sigma branch to 160 and introduce a sigmoid function to
normalize its output. In our experiments, we notice that nearly half of the
estimated $\bm{\sigma}$ approach zero, which correspond to the low-
frequency/smooth parts of the high-resolution images (as shown in Figure 7).
In light of this, we set $|\mathbf{\hat{y}}_{i}-\bm{\mu}_{i}|$ that is less
than the average to zero and focus on the hard samples. Due to the limited
computing resource, we did not conduct an ablation study on the structure of
the sigma branch, instead simply borrowing the idea of Conv+PReLU proposed in
[21], shown in Figure 3.

Table 4: PSNR comparison with recently proposed loss functions for SISR. PISR
[42] is a Knowledge Distillation framework where a “Teacher” network is
required during training. “DIV2K” means models reproduced by [42] using DIV2K
dataset with $\ell_{1}$ loss. “EDSR-baseline” refers to the baseline model in
[47].

Model | Loss Type |  | 2$\times$
---
Set5/B100
| 3$\times$
---
Set5/B100
| 4$\times$
---
Set5/B100
VDSR | $\ell_{2}$ | 37.53 / 31.90 | 33.67 / 28.82 | 31.35 / 27.29
DIV2K [1] | 37.64 / 31.96 | 33.80 / 28.83 | 31.37 / 27.25
Edge [61] | 37.70 / 31.98 | 33.82 / 28.88 | 31.45 / 27.35
Riemann [57] | 37.72 / 32.04 | 33.83 / 28.99 | 31.58 / 27.41
Teacher [42] | 37.77 / 32.00 | 33.85 / 28.86 | 31.51 / 27.29
Ours | 37.83 / 32.05 | 34.18 / 28.99 | 31.89 / 27.46
IDN | $\ell_{1}$ | 37.83 / 32.08 | 34.11 / 28.95 | 31.82 / 27.41
DIV2K [1] | 37.88 / 32.12 | 34.22 / 29.02 | 32.03 / 27.49
Teacher [42] | 37.93 / 32.14 | 34.31 / 29.03 | 32.01 / 27.51
Ours | 37.91 / 32.11 | 34.33 / 29.04 | 32.09 / 27.53
CARN | $\ell_{1}$ | 37.76 / 32.09 | 34.29 / 29.06 | 32.13 / 27.58
Teacher [42] | 37.82 / 32.08 | 34.10 / 28.95 | 31.83 / 27.45
Ours | 37.95 / 32.13 | 34.37 / 29.08 | 32.18 / 27.59
EDSR- baseline | $\ell_{1}$ | 37.99 / 32.16 | 34.37 / 29.09 | 32.09 / 27.57
Riemann [57] | 37.87 / 32.14 | 34.40 / 29.05 | 32.15 / 27.56
Ours | 38.02 / 32.20 | 34.43 / 29.11 | 32.15 / 27.59

Table 5: Average PSNR (dB) for different models on RealSR testing set. We
directly cite SRResNet and RCAN results reported in RealSR benchmark [12].
RCAN used in [12] is smaller than the original model. The running time is
measured on a $1200\times 2200$ SR image.

Model | #Param. | Runtime | PSNR(dB)
---|---|---|---
2$\times$ | 3$\times$ | 4$\times$
Bicubic | – | – | 32.61 | 29.34 | 27.99
DPS [12] | 1.26M | 122.8ms | 32.71 | 32.20 | 28.69
KPN [12] | 1.48M | 274.8ms | 33.86 | 30.39 | 28.90
LP-KPN [12] | 1.43M | 181.3ms | 33.90 | 30.42 | 28.92
SRResNet [41] | 1.31M | 143.1ms | 33.69 | 30.18 | 28.67
SRResNet+Ours | 33.87 | 30.49 | 28.84
RCAN [79] | 8.18M | 510.6ms | 33.87 | 30.40 | 28.88
RCAN+Ours | 34.06 | 30.63 | 29.00

###  4.4 Quantitative Results

Figure 4 shows the performance comparison of our approach coupled with various
SR methods to state-of-the-arts in terms of the number of operations and
parameters. We notably improve the Pareto frontier with no extra parameter or
computing cost at inference time. When compared with recent learning
strategies for image restoration, ours performs the best at most cases (Table
4), even surpasses the knowledge distillation method PISR [42]. Listed SR
methods generally benefit from the proposed scheme except for IDN [34] with
the scale factor of 2 on B100 [51].

(a) LR

(b) GT

(c) EDSR [47]

(d) [47]+Ours

(e) LR

(f) GT

(g) CARN [2]

(h) [2]+Ours

Figure 5: Visual comparison for $2\times$, $3\times$ and $4\times$ SR with BI
models on benchmark datasets. Table 6: Quantitative comparison with computing-
intensive models. We report the multiply-accumulate operations (MAC) and real
runtime cost, required to reconstruct a 1280$\times$720 HR image. RCAN [79]
coupled with our methods can be competitive with recent leading works.

Scale | Methods | #Param. | Runtime |  Set5 [6] |  Set14 [77] |  B100 [51] |  Urban100 [32] |  Manga109 [53]
---|---|---|---|---|---|---|---|---
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
2$\times$ |  EDSR [47] | 40.73M | 1003.6ms | 38.11 | 0.9602 | 33.92 | 0.9195 | 32.32 | 0.9013 | 32.93 | 0.9351 | 39.10 | 0.9773
EDSR+Ours | 38.24 | 0.9613 | 34.05 | 0.9209 | 32.36 | 0.9020 | 32.96 | 0.9360 | 39.19 | 0.9780
RCAN [79] | 15.44M | 667.8ms | 38.27 | 0.9614 | 34.12 | 0.9216 | 32.41 | 0.9027 | 33.34 | 0.9384 | 39.44 | 0.9786
RCAN+Ours | 38.25 | 0.9615 | 34.16 | 0.9231 | 32.40 | 0.9025 | 33.19 | 0.9381 | 39.41 | 0.9785
SAN [17] | 15.71M | 4817.6ms | 38.31 | 0.9620 | 34.07 | 0.9213 | 32.42 | 0.9028 | 33.10 | 0.9370 | 39.32 | 0.9792
HAN [58] | 15.92M | 743.3ms | 38.27 | 0.9614 | 34.16 | 0.9217 | 32.41 | 0.9027 | 33.35 | 0.9385 | 39.46 | 0.9785
NLSA [54] | 41.80M | 1915.8ms | 38.34 | 0.9618 | 34.08 | 0.9231 | 32.43 | 0.9027 | 33.42 | 0.9394 | 39.59 | 0.9789
3$\times$ |  EDSR [47] | 43.68M | 493.8ms | 34.65 | 0.9280 | 30.52 | 0.8462 | 29.25 | 0.8093 | 28.80 | 0.8653 | 34.17 | 0.9476
EDSR+Ours | 34.70 | 0.9295 | 30.58 | 0.8467 | 29.28 | 0.8100 | 28.87 | 0.8669 | 34.18 | 0.9487
RCAN [79] | 15.63M | 313.4ms | 34.74 | 0.9299 | 30.65 | 0.8482 | 29.32 | 0.8111 | 29.09 | 0.8702 | 34.44 | 0.9499
RCAN+Ours | 34.84 | 0.9304 | 30.70 | 0.8493 | 29.33 | 0.8112 | 29.14 | 0.8716 | 34.56 | 0.9504
SAN [17] | 15.90M | 1,587.9ms | 34.75 | 0.9300 | 30.59 | 0.8476 | 29.33 | 0.8112 | 28.93 | 0.8671 | 34.30 | 0.9494
HAN [58] | 16.11M | 343.7ms | 34.75 | 0.9299 | 30.67 | 0.8483 | 29.32 | 0.8110 | 29.10 | 0.8705 | 34.48 | 0.9500
NLSA [54] | 44.75M | 869.0ms | 34.84 | 0.9306 | 30.70 | 0.8485 | 29.34 | 0.8117 | 29.25 | 0.8726 | 34.57 | 0.9508
4$\times$ |  EDSR [47] | 43.10M | 322.6ms | 32.46 | 0.8968 | 28.80 | 0.7876 | 27.71 | 0.7420 | 26.64 | 0.8033 | 31.02 | 0.9148
EDSR+Ours | 32.48 | 0.8985 | 28.86 | 0.7883 | 27.74 | 0.7423 | 26.68 | 0.8045 | 31.12 | 0.9165
RCAN [79] | 15.59M | 181.1ms | 32.63 | 0.9002 | 28.87 | 0.7889 | 27.77 | 0.7436 | 26.82 | 0.8087 | 31.22 | 0.9173
RCAN+Ours | 32.69 | 0.9005 | 28.89 | 0.7892 | 27.78 | 0.7437 | 26.87 | 0.8094 | 31.40 | 0.9185
SAN [17] | 15.86M | 880.0ms | 32.64 | 0.9003 | 28.92 | 0.7888 | 27.78 | 0.7436 | 26.79 | 0.8068 | 31.18 | 0.9169
HAN [58] | 16.07M | 196.8ms | 32.64 | 0.9002 | 28.90 | 0.7890 | 27.80 | 0.7442 | 26.85 | 0.8094 | 31.42 | 0.9177
NLSA [54] | 44.15M | 523.6ms | 32.59 | 0.9000 | 28.87 | 0.7891 | 27.78 | 0.7444 | 26.96 | 0.8109 | 31.27 | 0.9184

(a) LR

(b) GT

(c) SRResNet[41]

(d) [41]+Ours

(e) LR

(f) GT

(g) RCAN[79]

(h) [79]+Ours

Figure 6: SR results on $3\times$ and $4\times$ RealSR testing set by
different methods.

We further detail the performance of our approach in contrast to recent
lightweight networks [2, 30, 33] in Table 3. To well describe different
methods, we report the number of parameters, operations required to
reconstruct a 720P image, and its average runtime speed measured on an NVIDIA
Titan RTX GPU. From Table 3, we observe that VDSR [37] trained by the proposed
learning process outperforms baselines by a large margin, consistently for all
scaling factors. Former state-of-the-art EDSR-baseline [47] with our approach
also shows competitive results with recent well-designed lightweight SR
methods. We further conduct experiments on the large BI degradation model EDSR
[47]. As shown in Table 6, the original model has achieved high PSNR/SSIM on
each dataset, while ours can further enhance the performance in most cases.

Table 7: The quality of model uncertainty measured on test datasets. PLL
(higher is better $\uparrow$) is the widely accepted uncertainty metric. Since
uncertainty should reflect underlying mistakes in super-resolved images, PSNR
($\uparrow$) between the predicted uncertainy $\bm{\sigma}$ and the actual
fitting error $|\mathbf{\hat{y}}-\bm{\mu}|$ is also reported (i.e., PSNR
between Fig. 7 and Fig. 7).

Model | Scale |  | S5 [6]
---
PLL/PSNR
| S14 [77]
---
PLL/PSNR
| B100 [51]
---
PLL/PSNR
| U100 [32]
---
PLL/PSNR
EDSR | 2$\times$ | 2.60 / 39.08 | 1.50 / 35.73 | 1.08 / 33.70 | 1.32 / 34.09
3$\times$ | 2.28 / 35.82 | 1.21 / 32.28 | 0.86 / 30.65 | 0.74 / 29.98
4$\times$ | 2.06 / 33.67 | 1.12 / 30.44 | 0.87 / 29.15 | 0.52 / 27.84
| 2$\times$ | 2.86 / 41.92 | 2.04 / 37.43 | 2.25 / 35.86 | 2.03 / 36.17
| 3$\times$ | 2.43 / 38.27 | 1.55 / 33.93 | 1.83 / 32.67 | 1.31 / 31.81
EDSR +Ours | 4$\times$ | 1.98 / 35.87 | 1.03 / 31.93 | 1.49 / 30.95 | 0.78 / 29.33
CARN | 2$\times$ | 2.58 / 38.56 | 1.54 / 35.28 | 1.19 / 33.47 | 1.14 / 32.96
3$\times$ | 2.25 / 35.24 | 1.28 / 31.94 | 0.99 / 30.45 | 0.70 / 29.28
4$\times$ | 2.02 / 33.23 | 1.13 / 30.17 | 0.91 / 29.01 | 0.44 / 27.35
| 2$\times$ | 2.80 / 41.67 | 1.98 / 36.99 | 2.29 / 35.70 | 2.07 / 35.35
| 3$\times$ | 2.28 / 37.92 | 1.40 / 33.62 | 1.80 / 32.46 | 1.34 / 31.19
CARN +Ours | 4$\times$ | 2.07 / 35.80 | 1.17 / 31.89 | 1.60 / 30.97 | 0.99 / 29.11

###  4.5 Qualitative Results

To further illustrate the analysis above, we show visual comparisons in Figure
5 and 6. Bicubic downsampling results in the loss of texture details and
structure information, which induces super-resolved images with blurring and
artifacts. Even worse, EDSR generates perceptually convincing results such as
“barbara” in Set14 (first row in Figure 5). However, the SR image contains
several lines with wrong directions. Instead, our models can recover them
being more faithful to the ground truth. CARN [2] with ours also obtains much
better results by recovering more informative details.

Despite the promising results, networks can make mistakes when exposed to
unseen data outside the trained distribution. In this case, we expect
information about model uncertainty in addition to the SR images.

###  4.6 Model Uncertainty

By introducing a posterior distribution $P(\mathbf{y}|\mathbf{x};\mathbf{W})$,
we can easily use the predicted $\bm{\sigma}$ as the estimated model
uncertainty (Figure 7). Following previous Bayesian models [31, 25, 67, 10],
we evaluate uncertainty quality based on the standard metric Predictive Log
Likelihood (PLL):

$\mathrm{PLL}(f(\mathbf{x}),(\mathbf{\hat{y}},\mathbf{x}))=\log
P(\mathbf{\hat{y}}|f(\mathbf{x}))$ (11)

where $P(\mathbf{\hat{y}}|f(\mathbf{x}))$ is the probability of target
$\mathbf{\hat{y}}$ generated by a probalistic model $f(\mathbf{x})$. In this
paper, $f(\mathbf{x})$ corresponds to the multivariate Gaussian
$\mathcal{N}(\bm{\mu},\bm{\sigma}^{2})$ with predicted mean $\bm{\mu}$ and
standard deviation $\bm{\sigma}$. $P(\mathbf{\hat{y}}|f(\mathbf{x}))$ is the
Gaussian probability density function (PDF) evaluated at $\mathbf{\hat{y}}$.

(a) LR

(b) HR: $\mathbf{\hat{y}}$

(c) SR: $\bm{\mu}$

(d) $|\mathbf{\hat{y}}-\bm{\mu}|$

(e) $\bm{\sigma}$

Figure 7: The last column (7) visualizes the estimated uncertainty
$\bm{\sigma}$ in $3\times$ SR. Dark color indicates large variance, i.e., high
uncertainty. Considering the visual differences between HR and SR images and
rael residual error (7), $\bm{\sigma}$ depicts a promising uncertainty
estimation that approximates the error with high fidelity.

We report the average PLL of our approach coupled with EDSR [47] and CARN [2]
in Table 7. The original EDSR/CARN networks fail to consider the model
uncertainty, hence we use their average fitting error
$\mathbb{E}_{\mathcal{D}}[|\mathbf{\hat{y}}-\bm{\mu}|]$ calculated on the
training dataset $\mathcal{D}$ to create a constant uncertainty estimation,
which corresponds to baseline results in Table 7. Any improvement over the
baseline reflects a sensible estimate of uncertainty. It is shown that PLL for
high upscaling factors are generally low, which indicates that reconstructed
$4\times$ image itself can be unreliable, i.e., networks fail to recognize the
hidden mistakes.

To improve the interpretability of the model uncertainty, we visualize the
predicted $\bm{\sigma}$ in Figure 7. It is shown that the estimated
uncertainty strongly correlates with the actual residual error
$|\mathbf{\hat{y}}-\bm{\mu}|$. We also report the PSNR results between them to
further evaluate the uncertainty quality fairly since PLL has been criticized
for being vulnerable to outliers [62].

##  5 Conclusions

In this paper, we present a novel learning process for single image super-
resolution with neural networks. Instead of best fitting the ground-truth HR
images, we introduce a posterior distribution
$P(\mathbf{y}|\mathbf{x};\mathbf{W})$ of underlying natural images
$\mathbf{y}$. By sampling from $P(\mathbf{y}|\mathbf{x};\mathbf{W})$, the
learning process serves as a multiple-valued mapping that better approximates
the local manifold corresponding to the observed HR image. The experiments
show that our method can produce SR images with minimal artifacts and the
predicted standard deviation of $P(\mathbf{y}|\mathbf{x};\mathbf{W})$, as a
by-product, allows us to make meaningful estimations on the model uncertainty.

## References

  * [1] Eirikur Agustsson and Radu Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  * [2] Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn.  Fast, accurate, and lightweight super-resolution with cascading residual network.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part X, volume 11214 of Lecture Notes in Computer Science, pages 256–272. Springer, 2018.
  * [3] Jan P. Allebach and Ping Wah Wong.  Edge-directed interpolation.  In Proceedings 1996 International Conference on Image Processing, Lausanne, Switzerland, September 16-19, 1996, pages 707–710. IEEE Computer Society, 1996.
  * [4] Joshua Batson and Loïc Royer.  Noise2self: Blind denoising by self-supervision.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 524–533. PMLR, 2019.
  * [5] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  In Richard Bowden, John P. Collomosse, and Krystian Mikolajczyk, editors, British Machine Vision Conference, BMVC 2012, Surrey, UK, September 3-7, 2012, pages 1–10. BMVA Press, 2012.
  * [6] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  In Richard Bowden, John P. Collomosse, and Krystian Mikolajczyk, editors, British Machine Vision Conference, BMVC 2012, Surrey, UK, September 3-7, 2012, pages 1–10. BMVA Press, 2012.
  * [7] Joan Bruna, Pablo Sprechmann, and Yann LeCun.  Super-resolution with deep convolutional sufficient statistics.  In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * [8] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel.  A non-local algorithm for image denoising.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego, CA, USA, pages 60–65. IEEE Computer Society, 2005.
  * [9] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel.  A review of image denoising algorithms, with a new one.  Multiscale Model. Simul., 4(2):490–530, 2005.
  * [10] Thang D. Bui, Daniel Hernández-Lobato, José Miguel Hernández-Lobato, Yingzhen Li, and Richard E. Turner.  Deep gaussian processes for regression using approximate expectation propagation.  In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1472–1481. JMLR.org, 2016.
  * [11] Jose Caballero, Christian Ledig, Andrew P. Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Real-time video super-resolution with spatio-temporal networks and motion compensation.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2848–2857. IEEE Computer Society, 2017.
  * [12] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang.  Toward real-world single image super-resolution: A new benchmark and a new model.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 3086–3095. IEEE, 2019.
  * [13] Hong Chang, Dit-Yan Yeung, and Yimin Xiong.  Super-resolution through neighbor embedding.  In 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2004), with CD-ROM, 27 June - 2 July 2004, Washington, DC, USA, pages 275–282. IEEE Computer Society, 2004.
  * [14] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.  Pre-trained image processing transformer.  CoRR, abs/2012.00364, 2020.
  * [15] Yuhua Chen, Feng Shi, Anthony G. Christodoulou, Yibin Xie, Zhengwei Zhou, and Debiao Li.  Efficient and accurate MRI super-resolution using a generative adversarial network and 3d multi-level densely connected network.  In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, and Gabor Fichtinger, editors, Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, volume 11070 of Lecture Notes in Computer Science, pages 91–99. Springer, 2018.
  * [16] Guoan Cheng, Ai Matsune, Qiuyu Li, Leilei Zhu, Huaijuan Zang, and Shu Zhan.  Encoder-decoder residual network for real super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, pages 2169–2178. Computer Vision Foundation / IEEE, 2019.
  * [17] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.  Second-order attention network for single image super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 11065–11074. Computer Vision Foundation / IEEE, 2019.
  * [18] Emily L. Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus.  Deep generative image models using a laplacian pyramid of adversarial networks.  In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1486–1494, 2015.
  * [19] Carl Doersch.  Tutorial on variational autoencoders.  CoRR, abs/1606.05908, 2016.
  * [20] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Learning a deep convolutional network for image super-resolution.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV, volume 8692 of Lecture Notes in Computer Science, pages 184–199. Springer, 2014.
  * [21] Chao Dong, Chen Change Loy, and Xiaoou Tang.  Accelerating the super-resolution convolutional neural network.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II, volume 9906 of Lecture Notes in Computer Science, pages 391–407. Springer, 2016.
  * [22] Weisheng Dong, Guangming Shi, Yi Ma, and Xin Li.  Image restoration via simultaneous sparse coding: Where structured sparsity meets gaussian scale mixture.  Int. J. Comput. Vis., 114(2-3):217–232, 2015.
  * [23] Alexey Dosovitskiy and Thomas Brox.  Generating images with perceptual similarity metrics based on deep networks.  In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 658–666, 2016.
  * [24] Gilad Freedman and Raanan Fattal.  Image and video upscaling from local self-examples.  ACM Trans. Graph., 30(2):12:1–12:11, 2011.
  * [25] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050–1059. JMLR.org, 2016.
  * [26] Qinquan Gao, Yan Zhao, Gen Li, and Tong Tong.  Image super-resolution using knowledge distillation.  In C. V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler, editors, Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part II, volume 11362 of Lecture Notes in Computer Science, pages 527–541. Springer, 2018.
  * [27] Xinbo Gao, Kaibing Zhang, Dacheng Tao, and Xuelong Li.  Image super-resolution with sparse neighbor embedding.  IEEE Trans. Image Process., 21(7):3194–3205, 2012.
  * [28] Zoubin Ghahramani.  Probabilistic machine learning and artificial intelligence.  Nat., 521(7553):452–459, 2015.
  * [29] Daniel Glasner, Shai Bagon, and Michal Irani.  Super-resolution from a single image.  In IEEE 12th International Conference on Computer Vision, ICCV 2009, Kyoto, Japan, September 27 - October 4, 2009, pages 349–356. IEEE Computer Society, 2009.
  * [30] Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan Yang, and Jian Cheng.  Ode-inspired network design for single image super-resolution.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1732–1741. Computer Vision Foundation / IEEE, 2019.
  * [31] José Miguel Hernández-Lobato and Ryan P. Adams.  Probabilistic backpropagation for scalable learning of bayesian neural networks.  In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1861–1869. JMLR.org, 2015.
  * [32] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.  Single image super-resolution from transformed self-exemplars.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 5197–5206. IEEE Computer Society, 2015.
  * [33] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang.  Lightweight image super-resolution with information multi-distillation network.  In Laurent Amsaleg, Benoit Huet, Martha A. Larson, Guillaume Gravier, Hayley Hung, Chong-Wah Ngo, and Wei Tsang Ooi, editors, Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019, pages 2024–2032. ACM, 2019.
  * [34] Zheng Hui, Xiumei Wang, and Xinbo Gao.  Fast and accurate single image super-resolution via information distillation network.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 723–731. IEEE Computer Society, 2018.
  * [35] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 448–456. JMLR.org, 2015.
  * [36] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II, volume 9906 of Lecture Notes in Computer Science, pages 694–711. Springer, 2016.
  * [37] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1646–1654. IEEE Computer Society, 2016.
  * [38] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Deeply-recursive convolutional network for image super-resolution.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1637–1645. IEEE Computer Society, 2016.
  * [39] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [40] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [41] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 105–114. IEEE Computer Society, 2017.
  * [42] Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham.  Learning with privileged information for efficient image super-resolution.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV, volume 12369 of Lecture Notes in Computer Science, pages 465–482. Springer, 2020.
  * [43] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila.  Noise2noise: Learning image restoration without clean data.  In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2971–2980. PMLR, 2018\.
  * [44] Min Li and Truong Q. Nguyen.  Markov random field model-based edge-directed image interpolation.  IEEE Trans. Image Process., 17(7):1121–1128, 2008.
  * [45] Xin Li and Michael T. Orchard.  New edge-directed interpolation.  IEEE Trans. Image Process., 10(10):1521–1527, 2001.
  * [46] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  CoRR, abs/2108.10257, 2021.
  * [47] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1132–1140. IEEE Computer Society, 2017.
  * [48] Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, and Gangshan Wu.  Residual feature aggregation network for image super-resolution.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2356–2365. IEEE, 2020.
  * [49] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo.  Multi-level wavelet-cnn for image restoration.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 773–782. IEEE Computer Society, 2018.
  * [50] Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang.  Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections.  In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2802–2810, 2016.
  * [51] David R. Martin, Charless C. Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings of the Eighth International Conference On Computer Vision (ICCV-01), Vancouver, British Columbia, Canada, July 7-14, 2001 - Volume 2, pages 416–425. IEEE Computer Society, 2001.
  * [52] Michaël Mathieu, Camille Couprie, and Yann LeCun.  Deep multi-scale video prediction beyond mean square error.  In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * [53] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Sketch-based manga retrieval using manga109 dataset.  Multim. Tools Appl., 76(20):21811–21838, 2017.
  * [54] Yiqun Mei, Yuchen Fan, and Yuqian Zhou.  Image super-resolution with non-local sparse attention.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3517–3526. Computer Vision Foundation / IEEE, 2021.
  * [55] Yiqun Mei, Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas S. Huang, and Honghui Shi.  Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 5689–5698. IEEE, 2020.
  * [56] John W. Miller, Rodney M. Goodman, and Padhraic Smyth.  On loss functions which minimize to conditional expected values and posterior proba- bilities.  IEEE Trans. Inf. Theory, 39(4):1404–1408, 1993.
  * [57] Jing Mu, Xinfeng Zhang, Shuyuan Zhu, and Ruiqin Xiong.  Riemannian loss for image restoration.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, pages 502–504. Computer Vision Foundation / IEEE, 2019.
  * [58] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen.  Single image super-resolution via a holistic attention network.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII, volume 12357 of Lecture Notes in Computer Science, pages 191–207. Springer, 2020\.
  * [59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024–8035, 2019.
  * [60] Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji.  Self2self with dropout: Learning self-supervised denoising from single image.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 1887–1895. Computer Vision Foundation / IEEE, 2020.
  * [61] George Seif and Dimitrios Androutsos.  Edge-based loss function for single image super-resolution.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, pages 1468–1472. IEEE, 2018.
  * [62] Reinhard Selten.  Axiomatic characterization of the quadratic scoring rule.  Experimental Economics, 1(1):43–61, 1998.
  * [63] Daniel W Stroock.  Probability theory: an analytic view.  Cambridge university press, 2010.
  * [64] Jian Sun, Zongben Xu, and Heung-Yeung Shum.  Image super-resolution using gradient profile prior.  In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2008), 24-26 June 2008, Anchorage, Alaska, USA. IEEE Computer Society, 2008.
  * [65] Yu-Wing Tai, Shuaicheng Liu, Michael S. Brown, and Stephen Lin.  Super resolution using edge prior and single image detail synthesis.  In The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pages 2400–2407. IEEE Computer Society, 2010.
  * [66] Ying Tai, Jian Yang, and Xiaoming Liu.  Image super-resolution via deep recursive residual network.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2790–2798. IEEE Computer Society, 2017.
  * [67] Mattias Teye, Hossein Azizpour, and Kevin Smith.  Bayesian uncertainty estimation for batch normalized deep networks.  In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4914–4923. PMLR, 2018\.
  * [68] Radu Timofte, Vincent De Smet, and Luc Van Gool.  A+: adjusted anchored neighborhood regression for fast super-resolution.  In Daniel Cremers, Ian D. Reid, Hideo Saito, and Ming-Hsuan Yang, editors, Computer Vision - ACCV 2014 - 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part IV, volume 9006 of Lecture Notes in Computer Science, pages 111–126. Springer, 2014.
  * [69] Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao.  Image super-resolution using dense skip connections.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 4809–4817. IEEE Computer Society, 2017.
  * [70] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Deep image prior.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 9446–9454. IEEE Computer Society, 2018.
  * [71] Zhangyang Wang, Yingzhen Yang, Zhaowen Wang, Shiyu Chang, Jianchao Yang, and Thomas S. Huang.  Learning super-resolution jointly from external and internal examples.  IEEE Trans. Image Process., 24(11):4359–4371, 2015.
  * [72] Chih-Yuan Yang and Ming-Hsuan Yang.  Fast direct super-resolution by simple functions.  In IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013, pages 561–568. IEEE Computer Society, 2013.
  * [73] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo.  Learning texture transformer network for image super-resolution.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 5790–5799. Computer Vision Foundation / IEEE, 2020.
  * [74] Jianchao Yang, John Wright, Thomas S. Huang, and Yi Ma.  Image super-resolution via sparse representation.  IEEE Trans. Image Process., 19(11):2861–2873, 2010.
  * [75] Xin Yu and Fatih Porikli.  Ultra-resolving face images by discriminative generative networks.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V, volume 9909 of Lecture Notes in Computer Science, pages 318–333. Springer, 2016.
  * [76] Linwei Yue, Huanfeng Shen, Jie Li, Qiangqiang Yuan, Hongyan Zhang, and Liangpei Zhang.  Image super-resolution: The techniques, applications, and future.  Signal Process., 128:389–408, 2016.
  * [77] Roman Zeyde, Michael Elad, and Matan Protter.  On single image scale-up using sparse-representations.  In Jean-Daniel Boissonnat, Patrick Chenin, Albert Cohen, Christian Gout, Tom Lyche, Marie-Laurence Mazure, and Larry L. Schumaker, editors, Curves and Surfaces - 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers, volume 6920 of Lecture Notes in Computer Science, pages 711–730. Springer, 2010.
  * [78] Lei Zhang and Xiaolin Wu.  An edge-guided image interpolation algorithm via directional filtering and data fusion.  IEEE Trans. Image Process., 15(8):2226–2238, 2006.
  * [79] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, volume 11211 of Lecture Notes in Computer Science, pages 294–310. Springer, 2018.
  * [80] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2472–2481. IEEE Computer Society, 2018.
  * [81] Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz.  Loss functions for image restoration with neural networks.  IEEE Trans. Computational Imaging, 3(1):47–57, 2017.
  * [82] Shangchen Zhou, Jiawei Zhang, Wangmeng Zuo, and Chen Change Loy.  Cross-scale internal graph neural network for image super-resolution.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.


Automorphism groups of rigid surfaces]Automorphism groups of rigid affine
surfaces: the identity component

A. Perepechko]Alexander Perepechko
Kharkevich Institute for Information Transmission Problems, 19 Bolshoy Karetny per., 127994 Moscow, Russia

HSE University,
20 Myasnitskaya ulitsa, Moscow 101000, Russia

M. Zaidenberg]Mikhail Zaidenberg
Institut Fourier, UMR 5582, Laboratoire de Mathématiques,Université Grenoble Alpes, CS 40700, 38058 Grenoble cedex 9, France

The research of A. Perepechko was carried out at the HSE University at the expense
of the Russian Science Foundation (project no. 22-41-02019).
Mathematics Subject Classification:
14J50, 14R20 (primary), 14L30, 05C60 (secondary).
Key words:  affine
surface, automorphism group, algebraic group, ind-group, group action, birational transformation, transformation of a graph.

It is known that the  identity component
of the automorphism group of a projective algebraic variety
is an algebraic group. This is not true in general for quasi-projective varieties.
In this note
we address the question: given an affine algebraic surface $Y$, as to when
the  identity component $\Aut^\0 (Y)$
of the automorphism group $\Aut (Y)$
is an algebraic group? We show that this happens if and only if $Y$ admits
no effective action of the additive group. In the latter case,
$\Aut^\0 (Y)$ is an algebraic torus of rank $\le 2$.

§ INTRODUCTION

Let $\K$ be an algebraically closed field of characteristic zero
and let $\Ga$ and $\G_m$ be the additive and the multiplicative groups of $\K$, respectively. If $Y$ is an affine variety over $\K$,
then the automorphism group $\Aut(Y)$ has a canonical structure of an ind-group endowed with its Zariski ind-topology,
see e.g. [18]. In this paper we concentrate on the identity component $\Aut^{\0} (Y)$ of $\Aut(Y)$ with respect to the Zariski topology.
Notice that, similarly to the case of a projective variety, the group of components $\Aut(Y)/\Aut^{\0} (Y)$ is at most countable, see <cit.>.
We address the following conjecture.
We say that $Y$ is rigid if  $Y$ admits
no effective $\G_{\rm a}$-action or, in other words, $\Aut^{\0} (Y)$ contains no $\G_{\rm a}$-subgroup, that is, no nontrivial unipotent element.

If $Y$ is a rigid affine algebraic variety over $\K$,
then the group $\Aut^{\0} (Y)$ is an algebraic torus of rank $\le\dim Y$.

If Conjecture <ref> is true,
then the following one is true as well[We thank Hanspeter Kraft who suggested the second conjecture.].

If an affine variety $Y$ admits no effective $\Ga$- and $\G_m$-actions, then $\Aut(Y)$ is a discrete group.

Some partial results on these conjectures can be found e.g. in <cit.>, <cit.>, [25, 26], <cit.>,
<cit.>, and [36, 37]. Conjecture <ref> holds, for instance, for toric affine varieties, see <cit.>.
In the present paper
we establish Conjecture <ref>
in the case $\dim Y=2$.

Let $Y$ be a normal affine surface over $\K$. Then the following hold.

$\Aut^{\0} (Y)$ is an algebraic group if and only
if $Y$ admits no effective $\Ga$-action,  if and only
if $\Aut^{\0} (Y)$ is an algebraic torus of rank $\le 2$.

Let $(X,D)$ be a minimal completion of $Y$ by a normal crossing divisor. Then
$\Aut^{\0} (Y)$ is an algebraic group if and only if
the dual graph $\Gamma(D)$ is birationally rigid.

See Definition <ref> for birationally rigidity of a weighted graph.
As a corollary, we establish the following effective combinatorial criterion of rigidity, cf. Corollary <ref>.

In the setup of Theorem <ref> the surface $Y$ is rigid if and only if $\Gamma(D)$
has no non-admissible extremal linear segment.

An extremal linear segment of a weighted graph $\Gamma$ is a maximal linear subgraph with no branching point of $\Gamma$ which contains a tip of $\Gamma$.
It is admissible if all its weights are $\le -2$, see Definitions <ref> and <ref>.

Using Corollary <ref> we show that the affine surface $Y=\PP^2\setminus \supp D$, where $D$ is a reduced effective divisor
on $\PP^2$ with only nodes as singularities, is rigid if and only if $\deg(D)\ge 3$, see Example <ref>.

The first part of Theorem <ref> was announced in <cit.>.
Our approach goes back to the work of Danilov
and Gizatullin [8].
Namely, the automorphism group $\Aut (Y)$ can be realized as a group of birational transformations
between the NC-completions of $Y$, i.e., the completions $(X,D)$
with a normal projective surface $X$ and a normal crossing boundary divisor $D$ contained in a smooth part of $X$.

Our main instrument is the birational transformations of weighted graphs.
Different aspects of this subject were developed e.g. in
[6, 7, 8, 9, 11, 13, 17, 20, 23, 34, 35, 39, 40], etc.
We consider, more generally, weighted graphs with a distinguished subset of marked vertices called rational.
In the case where our graph $\Gamma$ is the dual graph of a finite collection of curves on a surface,
the rational vertices of $\Gamma$ correspond to rational curves.

Section <ref> contains preliminaries on NC-completions, their dual graphs and
birational transformations of weighted graphs.
For the sake of completeness we provide thorough proofs of some known results.

In Section <ref> we discuss birationally rigid weighted graphs.
A weighted graph $\Gamma$ is minimal if it has no rational vertex of degree $\le 2$
and of weight $-1$, and birationally rigid if it is minimal and any birational transformation $\Gamma\dasharrow\Gamma'$ where $\Gamma'$ is a minimal weighted graph  is an isomorphism, see Definition <ref>.

The proof of Theorem <ref> is done in Section <ref>, see Theorem <ref>.
Let us indicate the main ingredients of the proof.

* It is well known that any birational transformation between smooth surfaces can be decomposed into a sequence of blowups and blowdowns.
The theory of birational transformations of weighted graphs is parallel to that of birational transformations of NC-pairs.

* Namely,  to every birational transformation of an NC-pair $(X,D)$ there corresponds a
birational transformation of the dual graph $\Gamma(D)$ and vice versa, see
Proposition <ref> and Lemma <ref>.

* The group $\Aut(Y)$ of automorphisms of a normal affine surface $Y$ contains a $\Ga$-subgroup if and only if $Y$ admits a minimal NC-completion $(X,D)$ whose dual graph $\Gamma(D)$ has an extremal linear segment with a tip of weight zero, see Proposition <ref> and its proof.

* A minimal linear weighted graph is non-admissible
if and only if it is birationally equivalent to a linear graph with a tip of weight zero, see e.g. <cit.>.

* A minimal weighted graph is birationally rigid if and only if it contains no non-admissible
extremal linear segment, see Proposition <ref>.

These facts imply that $Y$ is rigid if and only if the minimal graph $\Gamma(D)$ is birational rigid.
Now Corollary <ref> follows immediately.

* A blowup of an NC-pair $(X,D)$ is called inner if its center is a node of $D$.
A blowdown is inner if so is the inverse blowup.
A birational transformation is inner if it is composed of inner blowups, inner blowdowns and isomorphisms, see
Definitions <ref>–<ref>.

* Let $\Bir(X,D)$ stand for the subgroup of $\Bir(X)$ of birational transformations biregular on $Y=X\setminus D$.
Then the inner birational transformations of $(X,D)$ form a subgroup of $\Bir(X,D)$ denoted $\Inn(X,D)$.

* Similar definitions can be applied for birational transformations of weighted graphs, see Definitions <ref> and <ref>. To every inner birational transformation of an NC-pair $(X,D)$ there corresponds a unique inner
birational transformation of the dual graph $\Gamma(D)$ and vice versa, see
Proposition <ref>.

* If a minimal weighted graph $\Gamma$ is birationally rigid then any birational transformation of $\Gamma$ can be replaced by an inner birational transformation, see Proposition <ref>.

* Let  $(X,D)$ be a minimal  NC-completion of a rigid affine surface $Y$.
Then the ind-group $\Aut^\0 (Y)=:\Bir^0(X,D)$
coincides with its closed ind-subgroup $\Inn^\0(X,D)$ of
inner  automorphisms, see  Definitions <ref>, <ref>
and Theorem <ref>.

* In turn, the identity component $\Inn^\0 (X,D)$ coincides with
the affine algebraic
subgroup $\Aut^{\0} (X,D)$, see Proposition <ref>.

* The absence of nontrivial
unipotent subgroups forces $\Aut^{\0} (Y)=\Aut^{\0} (X,D)$ to be an algebraic torus.

These facts imply Theorem <ref>.
As a side result we  obtain
that for a rigid normal affine surface $Y$ the group $\Aut^{\0} (X,D)$ does not depend on the choice
of a minimal NC-completion $(X,D)$ of $Y$.

Acknowledgments. We are grateful to Ivan Arzhantsev, Hanspeter Kraft and Alvaro Liendo for some remarks and indications in the literature.

§ NC-PAIRS AND DUAL GRAPHS

We recall here the correspondence between compactifications of affine algebraic surfaces and weighted graphs following [11].
We use a slightly modified
version of the terminology in [11].

§.§ Birational transformations of NC-pairs

Let $X$ be a  normal projective surface  over $\K$ and $D$ be a  reduced effective divisor on $X$.
The pair $(X,D)$ is called an $NC$-pair if $D$ is a normal crossing divisor (i.e., the only singularities of $D$ are nodes) contained in the smooth locus of $X$.
An NC-pair $(X,D)$ is called an SNC-pair if $D$ is a simple normal crossing divisor, i.e., each component of $D$ is smooth and any two components of $D$ are disjoint or intersect at a single point.

Clearly, an NC-pair $(X,D)$ is not an SNC-pair if and only if $D$ has either  a nodal component, or a pair of components with more than one point in common. As an example, one can consider the pair $(\PP^2,C)$, where $C\subset\PP^2$ is a nodal cubic curve. Any NC-pair is dominated by an SNC-pair. For instance, in the above example such a domination is obtained after blowing up $\PP^2$ in the node of $C$ and one more blowing up in a node of the resulting curve, cf. Example <ref>.

Let $(X,D)$ and $(X',D')$ be NC-pairs. We denote by

* $\Bir((X,D),(X',D'))$ the  group of birational transformations
$X\dashrightarrow X'$ which restrict to biregular isomorphisms of the complements
$X\setminus \supp D\isom X'\setminus \supp D'$;

* $\Bir(X,D)$ the group $\Bir((X,D),(X,D))$;

* $\Aut(X,D)$ the group of  biregular automorphisms of $X$ that preserve $D$;

*  $\Aut^\natural(X,D)$ the subgroup of $\Aut(X,D)$ of automorphisms
preserving each component and each node of $D$;

* $\Aut^{\0}(X,D)$ the identity component of $\Aut(X,D)$.

The group $\Bir(X,D)$ is naturally isomorphic to the automorphism
group of the complement $Y=X\setminus \supp D$.
Indeed, for $g\in\Bir(X,D)$  the restriction $g|_Y$ is an automorphism of $Y$, and any
$f\in\Aut (Y)$
extends to a unique birational transformation of $X$.

Thus, the group $\Bir(X,D)\simeq\Aut (Y)$ does not depend on the completion $(X,D)$
of $Y=X\setminus \supp D$. However,
the group $\Aut (X,D)$ generally depends on the chosen completion. For instance,
for the completion of the affine plane $\AA^2\embed \PP^2$ we have
$\Aut (\PP^{2},\PP^{1})\simeq\Aff(\AA^2)$, whereas for the completion $\AA^2\embed
\PP^1\times\PP^1$ we have

\[\Aut(\PP^1\times\PP^1,\PP^{1}\times \{\infty\} \cup \{\infty\} \times \PP^{1})\simeq \ZZ_2\ltimes (\Aff(\AA^{1}))^2.\]

§.§ Inner transformations

A blowup of an NC-pair $(X,D)$ with center $x\in D$ gives again an NC-pair. Such a blowup is called inner
if $x$ is a node of $D$ and outer otherwise. A blowdown of a component of $D$ is called inner (resp. outer) if the inverse blowup is inner (resp. outer).

A birational transformation $(X,D)\dashrightarrow(X',D')$ is called inner
if it can be decomposed into a sequence of inner blowups, inner blowdowns and isomorphisms of NC-pairs.
The subset of inner birational transformations $\Inn(X,D)\subset\Bir (X,D)$ forms a subgroup.

We have
\[\Aut^{\0}(X,D)\subset\Aut^\natural(X,D)\subset\Aut(X,D)\subset\Inn(X,D)\subset\Bir(X,D).\]

This is immediate from our definitions.

$\Aut^{\0}(X,D)=(\Aut^\natural(X,D))^{\0}$ is a connected algebraic group.

It is well known that $\Aut^{\0}(X)$ is a connected algebraic group, see [31, 32, 38],
and so is its closed connected subgroup $\Aut^{\0}(X,D)$.

Every inner birational transformation $\phi\colon(X,D)\dashrightarrow(X',D')$ satisfies the relation

\[
\phi\circ\Aut^\natural (X,D) = \Aut^\natural (X',D')\circ\phi.
\]

In particular, $\Aut^\natural (X,D)$ is a normal subgroup of $\Inn(X,D)$.

It is enough to check the statement for an inner blowdown $\sigma\colon (X_1,D_1)\to (X_2,D_2)$. Indeed, $\sigma$ induces an isomorphism

\sigma_*\colon \Aut^\natural (X_1,D_1)\stackrel{\simeq}{\longrightarrow}\Aut^\natural (X_2,D_2)\,

$\pi_*\colon \alpha\mapsto \sigma\circ\alpha\circ\sigma^{-1}$.

The group $\Aut^\natural(X,D)$ is the kernel
of the  natural homomorphism
$\Aut(X,D)\to\Aut(\Gamma(D))$, where $\Gamma(D)$ is the dual graph of $D$, see Definition <ref>.

The group $\Inn(X,D)/\Aut^\natural (X,D)$ is countable.

Since $\Aut (X,D)/\Aut^\natural (X,D)$ is a finite group, it suffices to show that $\,$
$\Inn(X,D)/\Aut (X,D)$ is countable.
We claim that any $\phi\in\Inn(X,D)$ can be written as

\begin{equation}\label{eq:isomorphism} \phi=\alpha\sigma_n^{\pm 1}\cdots\sigma_1^{\pm 1}={\sigma'}_n^{\pm 1}\cdots{\sigma'}_1^{\pm 1}\beta\end{equation}

where $\alpha, \beta\in\Aut (X,D)$ and the $\sigma_i, \sigma_i'$ are inner blowdowns. Indeed, consider a commutative diagram

\[
\begin{tikzpicture}
\node (empty) at (-3,0) {};
\node  (INN) at (0,2) {$(X_1,D_1)$};
\node  (Inn) at (0,0)  {$(X'_1,D'_1)$};
\node (iNN)  at (4,2)  {$ (X_2,D_2)$};
\node  (inn) at (4,0)  {$(X'_2,D'_2)$};
\draw[->, dashed] (INN) to node[above] {$\sigma^{\pm 1}$}  (iNN);
\draw[->] (INN) to node[left] {$\alpha_1$} (Inn);
\draw[->] (INN) to node[right] {$\simeq$} (Inn);
\draw[->, dashed] (Inn) to node[above] {${\sigma'}^{\pm 1}$}  (inn);
\draw[->] (iNN) to node[right] {$\alpha_2$}   (inn);
\draw[->] (iNN) to node[left] {$\simeq$}   (inn);
\end{tikzpicture}
\]

where $\sigma$ and $\sigma'$ are inner blowdowns.
Given a pair $(\sigma',\alpha_1)$ there exists a unique pair $(\sigma,\alpha_2)$ fitting in the diagram, and vice versa.
Now the claim follows.

Therefore, any left coset in $\Inn(X,D)/\Aut (X,D)$ has the form $\psi\cdot\Aut (X,D)$ for some
$\psi={\sigma'}_n^{\pm 1}\cdots{\sigma'}_1^{\pm 1}\in\Inn(X,D)$. These cosets form a countable set.

§.§ Weighted graphs
The birational geometry of  NC-pairs has its
combinatorial counterpart, namely, the birational geometry of weighted graphs, see the references in the Introduction.

Let $\Gamma$ be a (nonempty)  finite weighted graph with the set of vertices $\Vert(\Gamma)$
where vertices are weighted by integer numbers, and let $\Ratio(\Gamma)\subseteq\Vert(\Gamma)$
be a distinguished subset of vertices  which we call rational vertices. A rational vertex of weight $-1$
(resp. of weight $0$) is called
($-1$)-vertex (resp. ($0$)-vertex). The graph $\Gamma$ may be disconnected and may contain loops,
cycles and multiple edges, that is, several edges joining the same pair of vertices.

* Given a weighted graph $\Gamma$, the degree $\deg_{\Gamma}(v)$ of a vertex $v\in\Vert(\Gamma)$ is the
number of incident edges of $\Gamma$ at $v$, where a loop $[v,v]$ is counted
twice. We simply write $\deg(v)$ when the graph $\Gamma$ has been fixed.

* A rational vertex $v$ of $\Gamma$ is called an
end vertex if $\deg(v)=1$, linear if $\deg(v)=2$ and
at most linear if $\deg(v)\le 2$ and no loop of $\Gamma$ is incident with $v$.
We let $\End(\Gamma)$ be the set of end vertices of $\Gamma$.

* A vertex $v$ is called branching
if either $v$ is non-rational or $\deg(v)\ge 3$.
We let $\Br(\Gamma)$ be the set of all branching vertices of $\Gamma$.
Thus, $\Vert(\Gamma)\setminus\Ratio(\Gamma)\subseteq\Br(\Gamma)$.

* Given a vertex $v\in \Gamma$, the connected components of
$\Gamma\setminus\{v\}$ linked to $v$ are the branches
of $\Gamma$ at $v$. A branch linked to $v$ via exactly one edge
is called simple.

Let $\Gamma$ be a weighted graph  with vertices $v_1,\ldots,v_n$.
The intersection form $I(\Gamma)$ is a bilinear form on $\RR^n$ given by the symmetric square matrix $A(\Gamma)=(a_{i,j})$
where $a_{i,i}$ is the weight of $v_i$ and for $j\neq i$, $a_{i,j}$ equals the number of edges between $v_i$ and $v_j$.
The discriminant of $\Gamma$ is the determinant $\det(-A(\Gamma))$.
We let $i_+(\Gamma), i_-(\Gamma)$ and $i_0(\Gamma)$ be the inertia indices of the quadratic form $I(\Gamma)$.

Let $[v_1,v_2]$ (where possibly $v_1=v_2$) be an edge of
The inner blowup in $[v_1,v_2]$ consists in adding a new ($-1$)-vertex $v$,
replacing the edge  $[v_1,v_2]$ by two new edges $[v_1,v]$ and $[v,v_2]$ and
decreasing the weights of $v_1$ and $v_2$  by $1$ if $v_1\neq v_2$ and by $4$ if $v_1=v_2$ (i.e. $[v_1,v_2]$ is a loop).

The outer blowup at
a vertex $v$ in $\Gamma$ consists in introducing a new ($-1$)-vertex
$v^\prime$ along with a new edge $[v,v^\prime]$ and decreasing
the weight of $v$ by $1$.

The modifications inverse to inner and outer blowups are called
inner and outer blowdowns, respectively.