# Multi–Grid Back–Projection Networks

Pablo Navarrete Michelini, Wenbin Chen, Hanwen Liu, Dan Zhu, Xingqun Jiang
Pablo Navarrete M., Wenbin Chen, Hanwen Liu, Dan Zhu, and Xingqun Jiang are
with the Department of Artificial Intelligence, BOE Technology Group Co.,
Ltd., Beijing, China.Manuscript received July $1^{st}$, 2020. First revision
November $17^{th}$, 2020. Second revision December $23^{rd}$, 2020. Accepted
December $28^{th}$, 2020.

###### Abstract

Multi–Grid Back–Projection (MGBP) is a fully–convolutional network
architecture that can learn to restore images and videos with upscaling
artifacts. Using the same strategy of multi–grid partial differential equation
(PDE) solvers this multiscale architecture scales computational complexity
efficiently with increasing output resolutions. The basic processing block is
inspired in the iterative back–projection (IBP) algorithm and constitutes a
type of cross–scale residual block with feedback from low resolution
references. The architecture performs in par with state–of–the-arts
alternatives for regression targets that aim to recover an exact copy of a
high resolution image or video from which only a downscale image is known. A
perceptual quality target aims to create more realistic outputs by introducing
artificial changes that can be different from a high resolution original
content as long as they are consistent with the low resolution input. For this
target we propose a strategy using noise inputs in different resolution scales
to control the amount of artificial details generated in the output. The noise
input controls the amount of innovation that the network uses to create
artificial realistic details. The effectiveness of this strategy is shown in
benchmarks and it is explained as a particular strategy to traverse the
perception–distortion plane.

###### Index Terms:

multigrid, iterative backprojection, super resolution, convolutional networks.

##  I Introduction

Image and video upscaling has been studied for decades and remains an active
topic of research because of constant technological advances in digital
imaging. One scenario where upscaling is now more demanding arises in digital
display technologies, where new standards like BT.2020 [1] are introduced. The
resolution of digital displays has experienced a tremendous growth over the
past few decades, as shown in Figure 1. The transition between different
formats leads to a challenging problem. On one hand, large amount of digital
content still exist in popular old standards such as standard–definition (SD).
On the other hand, the latest display technologies (e.g. 4K, 8K and above) are
expected to show this content with reasonable quality. Standard upscaling
technologies are clearly insufficient for this purpose. While a $2\times$
upscaler maps $1$ input pixel into $4$ output pixels, an $8\times$ upscaler
maps $1$ input pixel into $64$ output pixels, which already can contain a
small image. The problem demands advanced solutions that are capable of
understanding the content and filling in these large pieces of images with
visually appealing and consistent information. In particular, large upscaling
factors are needed to convert SD to ultra high–definition (UHD) resolutions.
Thus, large upscaling represents a real problem in current market and it is
expected to persist and become even more challenging with the rapid
development of new technologies.

Figure 1: The dramatic growth of standard resolutions demands the development
of Super–Resolution technologies to resize image with large upscaling factors.
Source: Wikimedia CC BY-SA.

In classical interpolation theory, upscaling images by integer factors is
explained as two sequential processes: upsample (insert zeros) and filter [2,
3]. Standard upscaler algorithms, such as Bicubic or Lanczos, find
high–resolution images with a narrow frequency content by using fixed low–pass
filters. Modern tensor processing frameworks (e.g. Pytorch, Tensorflow, etc.)
implement this process using a so–called _strided transposed convolutional
layer_. Similarly, the image acquisition process can be modeled as: filter and
downsample (drop samples). Many times we know the explicit model, e.g. bicubic
downscaler. Tensor processing frameworks implement this process using a
_strided convolutional layer_.

More advanced upscalers have followed geometric principles to improve image
quality. For example, _edge–directed interpolation_ uses adaptive filters to
improve edge smoothness [4, 5], or _bandlet_ methods use both adaptive
upsampling and filtering [6]. Later on, machine learning has been able to use
examples of pristine high–resolution images to learn a mapping from
low–resolution [7]. The rise of deep–learning and convolutional networks in
image classification tasks [8] quickly saw a series of important improvements
in image resizing with large upscaling factors, which is the process widely
know as image super–resolution (SR). Major progress in network architectures
for image classification often succeeded in image SR, as seen for example with
CNNs applied in SRCNN [9], ResNets [10] applied in EDSR [11], DenseNets [12]
applied in RDN [13], attention [14] applied in RCAN [15], and non–local
attention [16] applied in RNAN [17]. In all these examples, arguably the most
influential practice is the use of residual networks (ResNets).

For video content, single frame upscaling would loose the chance of improving
quality by learning temporal correlations and can result in visible flickering
artifacts at large upscaling factors. Popular approaches to capture temporal
information include: methods that align neighbor frames in addition to solve
single image artifacts [18, 19, 20], network architectures that use motion
compensation to align neighbor frames [21, 22, 23, 24], use transformer blocks
to learn the optical flow [25], learn dynamic upscaling filters [26], and more
recently use deformable convolutions for frame alignment [27]. To better
understand the mechanisms used to extend image to video SR we describe some of
these solution in more detail:

  * •

TOFlow [25] is a network designed with motion estimation and video processing
components. It combines three modules to: 1) estimate the motion fields
between input frames; 2) register all input frames based on estimated motion
fields; and 3) generate target output from registered frames. These three
modules are jointly trained to minimize the loss between output frames and
ground truth.

  * •

Dynamic Upsampling Filters (DUF) [26] is a network that generates dynamic
upsampling filters and a residual image, which are computed depending on the
local spatio–temporal neighborhood of each pixel to avoid explicit motion
compensation. A HR image is reconstructed directly from the input image using
the dynamic upsampling filters, and the fine details are added through the
computed residual.

  * •

Enhanced Deformable convolutions Video Restoration (EDVR) [27] is a network
that combines frame alignment and spatio–temporal fusion. To handle large
motions it uses a Pyramid, Cascading and Deformable (PCD) alignment module, in
which frame alignment is done at the feature level using deformable
convolutions in a coarse–to–fine manner. Then they use a Temporal and Spatial
Attention (TSA) fusion module, in which attention is applied both temporally
and spatially to emphasize important features for subsequent restoration.

  * •

The Video Enhancement and Super–Resolution network (VESR–Net) [28] uses both
PCD alignment and a Separate Non–Local attention (SNL) module to aggregate the
information among different frames. For reconstruction, they utilize stacked
channel–attention residual block (CARB) [29] followed by a feature decoder.
Finally, Efficient Video Enhancement and Super–Resolution Net (EVESR–Net) [30]
improved VESR–Net replacing the SNL module with an Efficient Point-Wise
Temporal Attention Block (EPAB) that aggregates the spatio–temporal
information with less operations and memory consumption.

The optimization problem of learning to map images/videos from low to high
resolution can have two distinctive targets:

  * •

High Fidelity, the aim is to obtain a solution capable to produce high
resolution results with the best fidelity (PSNR) to the ground truth. This is
also known as a _low distortion_ target where _distortion_ refers to a measure
of the difference between output content and ground truth.

  * •

Perceptual Quality, the aim is to obtain a solution capable to produce high
resolution results with the best perceptual quality similar to the ground
truth.

Historically, high fidelity was the major focus of research in image SR until
systems became more efficient and large upscaling factors where considered.
The empirical evidence then showed that high fidelity often leads to
cartoonish effects that are easily recognizable and looked far from real
photographs. Soon this reveal a fundamental phenomena that is now known as the
_Perception–Distortion_ trade–off established by Blau and Michaeli in [31].
That is, both perception and distortion targets cannot be achieved at the same
time, one must compromise perceptual quality to improve fidelity and vice
versa. The trade–off also motivated the first workshop and challenge focused
on perceptual super–resolution (PIRM 2018 [32]) that strongly confirmed and
gave further insight into this principle.

Here, we present our contributions to the development of new image and video
super–resolution technologies with major focus on: deep–learning
architectures, and training strategies for perceptual super–resolution. First,
we propose a deep learning architecture called Multi–Grid Back–Projection
(MGBP)[33, 34, 35] that has the following important characteristics:

  * •

It uses a _back–projection_ residual strategy that moves features from high to
low resolution to compare them with a low resolution reference and then comes
back to correct the high resolution feature. This is analogous to the
iterative back–projection (IBP) algorithm but its target is more far reaching
as a general learning module that aims to improve the high resolution output
in successive iterations.

  * •

It uses a _multi–grid_ recursive strategy using back–projection residual
blocks within back–projection residual blocks. Thus, it quickly moves features
to the lowest resolution stages where it concentrates most of the heavy
processing and leaves easier tasks for higher resolutions, efficiently
balancing the computational complexity across scales. This is analogous to
multigrid PDE solvers that are known to be optimal in the solution of large
systems of equations [36].

  * •

It allows a recursive configuration sharing parameters in all back–projection
blocks leading to good results and very small number of parameters, or
alternatively, a non–recursive configuration that leads to better performance
and an increased number of parameters.

  * •

It can be extended to use 3D–convolutions for video. This is due to the
property that back–projection blocks do not change the size of the input and
allows to learn a mapping from many input to many output frames. This is a
convenient property that we call _Cube–to–Cube_ and it leads to a fully
3D–convolutional network that we call MGBP–3D. It departs significantly from
other video solutions mentioned above as it relies completely on
3D–convolutions and do not include other non–linear blocks such as attention,
warping, dynamic filters or deformable convolutions.

Second, we designed a particular training strategy for perceptual
super–resolution with the following contributions:

  * •

We propose a strategy to control the perception–distortion trade-off as a
transition from a high–fidelity target to perceptual target through a so
called _innovation jump_.

  * •

We drive the innovation process of our MGBP architecture (generator) by
introducing random inputs at each resolution level. These inputs are
manipulated by the network to generate artificial details at different scales.

  * •

We propose a so–callled _variance–normalization and shift–correlator_ (VN+SC)
layer that provides meaningful features to a discriminator system based upon
previous research on the statistics of natural images.

  * •

We propose a _multiscale discriminator_ for adversarial training. It is a
configuration symmetric to the multi–scale upscaler, therefore it is more
effective for adversarial training. It can simultaneously evaluate several
upscaling factors, resembling a Progressive GAN[37] in the sense that the
optimizer can focus on smaller factors first and then work on larger factors.

  * •

We propose a novel _noise–adaptive training strategy_ that can avoid conflicts
between reconstruction and perceptual losses, combining loss functions with
different random inputs.

An early version of the MGBP architecture first appeared in the NTIRE 2018
challenge on image SR[38], together with Deep Back–Projection Networks
(DBPN)[39]. Both DBPN and MGBP used similar back–projection residual blocks
with DBPN achieving better results and winning this challenge with a two–level
configuration mixing up/down–backprojections. The architecture of MGBP was
later improved in [33] achieving state–of–the–arts results in high–fidelity SR
for lightweight networks with small number of parameters. The training
strategy for perceptual quality first appeared in PIRM 2018 achieving the
$\boldsymbol{2^{nd}}$ best perceptual quality in the Perceptual SR challenge
[32]. We refer to these early configurations using small number of parameters
as MGBP version 1 (MGBPv1). The architecture was later improved in [40] to
allow a non recursive structure that is more flexible and simple to configure,
allowing a larger amount of parameters while at the same time being able to
process very large images (e.g. 8K). This configuration was named MGBP version
2 (MGBPv2) and has won the $\boldsymbol{1^{st}}$ place for best perceptual
quality in the AIM 2019 challenge for Extreme Super–Resolution ($16\times$
factor) as well as $\boldsymbol{3^{rd}}$ best perceptual quality in NTIRE 2020
challenge on Real–World Image SR. Here, we include new experiments to show
that MGBPv2 can also achieve state–of–the–arts results in high–fidelity
targets for image SR. Finally, we show how to extend MGBPv2 to the fully
3D–convolutional network MGBP–3D. This new architecture was recently used in
the AIM 2020 Video Extreme Super–Resolution Challenge achieving the
$\boldsymbol{2^{nd}}$ best fidelity [30]. Overall, the MGBP architecture has
been particularly successful as a convenient configuration to balance
multi–scale computational complexity for large upscaling factors in both image
and video SR as well as achieving state–of–the–arts results in perceptual
super–resolution.

Figure 2: For a high–resolution image $X$ that looks real, distortion
optimization approaches an optimal solution $X^{*}$ that does not look real
because it lacks the unpredictable nature of natural images. We can still use
$X^{*}$ as a reference point to move through an _innovation jump_ into the set
of realistic images.  Figure 3: Diagram of the MultiGrid BackProjection
version 1 (MGBPv1) network [33] with $\mu=0$, $1$, and $2$.  Figure 4: Diagram
of the MultiGrid BackProjection network version 2 (MGBPv2) for images and
MGBP–3D for videos unfolded from Algorithm 1 with $\mu=2$.

##  II Perceptual and Distortion Targets

To better illustrate our target, we present a diagram of image sets in Figure
2. Here, $\mathcal{H}$ is the set of all high–resolution images,
$\mathcal{H}^{real}\subset\mathcal{H}$ is the subset of high–resolution images
that correspond to natural images, and $\mathcal{L}$ is the set of all
low–resolution images. Given an image $X\in\mathcal{H}^{real}$, we are
interested in the set of _aliased_ images:

$\mathcal{A}(X)=\left\\{Y\in\mathcal{H}\quad s.t.\quad
S_{f}(Y)=S_{f}(X)\right\\}\;,$ (1)

where $S_{f}:\mathcal{H}\rightarrow\mathcal{L}$ is a _downscale_ operator of
factor $f$. We are particularly interested in the set
$\mathcal{A}(X)\cap\mathcal{H}^{real}$ of alias images that correspond to real
content.

A _distortion_ function $\Delta(X,y)$ measures the dissimilarity between a
reconstructed image $y$ and the original image $X$. Popular and basic
distortion metrics such as L1, L2, PSNR, etc., are sensitive to changes (any
minor difference in pixel values would increase the amount of distortion) and
are known to have low correlation with human perception[41]. Several
distortion metrics have been proposed to approach perceptual quality by
emphasizing some differences more than others, either through normalization,
feature extraction or other approaches. These include metrics like SSIM[42],
VIF[43] and the VGG content loss[44]. By doing so, correlation with human
perception improves according to [41], but experiments in [31] show that these
metrics still focus more on distortion. More recently, the contextual loss has
been proposed to focus more on perceptual quality while maintaining a
reasonable level of distortion[45].

The solution of distortion optimization is obtained by:

$X^{*}=\text{argmin}_{y}\mathbb{E}\left[\Delta(X,y)\right]\;.$ (2)

The original image $X$ is fixed, and the expected value in (2) removes any
visible randomness in the search variable $y$. But, according to research on
the statistics of natural images, randomness plays an essential role in what
makes images look real[46]. This is well known for non–reference image quality
metrics such as NIQE[47] or BRISQUE[48], and led to a definition of perceptual
quality as a distance between probability distributions in [31]. It is also
known that distortion optimization solutions tend to look unreal, as seen in
state–of–the–art results from NTIRE–SR Challenges[49, 38]. Common distortion
metrics in these challenges (L1 and L2) make the image $X^{*}$ lose all
randomness. We argue that this removal of randomness in $X^{*}$ is what moves
it out of set $\mathcal{H}^{real}$, as we show in Figure 2.

We know that $X\neq X^{*}$ because $X\in\mathcal{H}^{real}$ and
$X^{*}\notin\mathcal{H}^{real}$ according to our previous discussion. However,
distortion optimization can still be useful to generate realistic images. By
approaching $X^{*}$ we are getting closer to $X$. As shown in Figure 2, both
$X$ and $X^{*}$ can be in $\mathcal{A}(X)$. Using a signal processing
terminology, the _innovation_[50] is the difference between $X$ and the
optimal forecast of that image based on prior information, $X^{*}$. Most SR
architectures take the randomness for the innovation process from the
low–resolution input image, which is a valid approach but loses the ability to
expose and control it.

In our proposed architecture we add randomness explicitly as noise inputs, so
that we can control the amount of innovation in the output. Independent and
identically distributed noise will enter the network architecture at different
scales, so that each of them can target artificial details of different sizes.
Generally speaking, our training strategy will be to approach $X^{*}$ with
zero input noise and any image in $\mathcal{A}(X)\cap\mathcal{H}^{real}$ with
unit input noise. By using noise to target perceptual quality, and remove it
for the distortion target, we teach the network to _jump_ from $X^{*}$ into
$\mathcal{H}^{real}$. With probability one the network cannot hit $X$, but the
perceptual target is any image in $\mathcal{A}(X)\cap\mathcal{H}^{real}$.

Algorithm 1 Multi–Grid Back–Projection version 2 (MGBPv2) MGBPv2$(X,W,\mu,L)$: |  $\boldsymbol{BP^{\mu}_{k}}(u\;|\;y_{1},\ldots,y_{k-1};\text{tag}_{1},\ldots,\text{tag}_{k-1})$:
---|---
0: Input image $X$.  0: Steps $\mu$, levels $L$ and noise amplitude $W$.  0: Output image $Y$.  1: $x=[X,\;W\cdot\mathcal{N}(0,1)]$ 2: for $k=1,\ldots,L$ do 3: $y_{k}=\text{Analysis}_{k}(x)$ 4: end for 5: for $k=1,\ldots,L$ do 6: $\text{tag}_{k}=0$ 7: end for 8: $y=BP^{\mu}_{L}(y_{L}\;|\;y_{1},\ldots,y_{L-1};\text{tag}_{1},\ldots,\text{tag}_{L-1})$ 9: $Y=\text{Synthesis}(y)$ |  0: Input image $u$, level index $k$, steps $\mu$.  0: Images $y_{1},\ldots,y_{k-1}$ and $\text{tag}_{1},\ldots,\text{tag}_{k-1}$ (for $k>1$).  0: Image $out$ 1: $out=u$ 2: if $k>1$ then 3: for $s=1,\ldots,\mu$ do 4: $\text{tag}_{k-1}=s$ 5: $LR=\text{Downscale}_{tag}(out)$ 6: $c=BP^{\mu}_{k-1}(LR\;|\;y_{1},\ldots,y_{k-2};\text{tag}_{1},\ldots,\text{tag}_{k-2})$ 7: $out=out+\text{Upscale}_{tag}([\;y_{k-1},c\;])$ 8: end for 9: end if

##  III Architecture

###  III-A Generator

Motivation. The general problem of image restoration is to recover an image
$X$ that has been degraded. This could mean that either some details have been
removed from the image (e.g. blurring) or some new information is interfering
and corrupting the image (e.g. noisy measurements). Here, we will focus on the
problem of image super–resolution, assuming a degradation model of the form:

$x=R(X)\;,$ (3)

where $X\in\mathbb{R}^{N}$ is the high–resolution source, $x\in\mathbb{R}^{n}$
is the low–resolution result, and $R:\mathbb{R}^{N}\rightarrow\mathbb{R}^{n}$
is a _restriction_ operator (downscaler). A classic linear model is
$R(X)=(X*g)\downarrow s$ where $g$ is a blurring kernel (e.g. bicubic) and
$\downarrow s$ is a downsampling by factor $s$ dropping pixels to reduce the
resolution.

The degradation model (3) represents a prior knowledge that we would like to
enforce on our system, aiming to recover the original image. This is the
motivation behind the classic Iterative Back–Projection (IBP) algorithm [51].
Given model (3) and an upscaled image $y_{0}$, the IBP algorithm iterates:

$\displaystyle e_{k}$ $\displaystyle=x-R(y_{k})\;,$ (4) $\displaystyle
y_{k+1}$ $\displaystyle=y_{k}+P(e_{k})\;.$ (5)

Here, $e(y_{k})$ is the mismatch error at low–resolution and
$P:\mathbb{R}^{n}\rightarrow\mathbb{R}^{N}$ is a projection operator
(upscaler). A classic linear projection is $P(x)=(x\uparrow s)*p$, where $p$
is an upscaling filter (a convolution) and $\uparrow s$ is increasing the
resolution by inserting zeros. For linear operators $P$ and $R$, it is simple
to see that $y_{k}$ will follow the degradation model (3) as
$k\rightarrow\infty$ when $||I-RP||<1$ [51].

IBP computes a residual $P(e_{k})$ to update the output in successive
iterations. It is thus natural to think of this process as a residual block.
The restriction operator $R$ can be implemented with a strided convolutional
layer and the projection operator $P$ with a strided transposed convolution.
In MGBP we use these residual steps and follow the design of residual blocks
in EDSR [11] by using a single rectified linear unit just before the
transposed convolution in the $P$ operator. We also make a small
generalization by changing the error equation (4) to $e_{k}=[x,R(y_{k})]$
where $[\cdot,\cdot]$ replaces the difference to concatenation of features (in
the channel dimension). The projector operator $P$ will then decide how to
compare $x$ and $R(y_{k})$ in the update equation (5).

The IBP iteration does work as a residual block but it is different to the
conventional blocks used in ResNets [10] in that IBP performs the correction
at a lower resolution. This is similar to the way a Full–Multigrid algorithm
solves linear equations [36]. That is: first, find an approximate solution in
a small grid; and second, interpolate the approximate solution to update the
solution in a large grid. Here, we borrow an essential idea of multigrid
methods, that is to repeat the process recursively with residual
backprojection blocks within residual backprojection blocks that quickly move
features the lowest resolutions. Hence, two step back–projection residuals in
high resolution ($\mu=2$) leads to $4$ residual steps in the next lower
resolution, $8$ in the next level, and so forth. The pattern unfolded is known
in multigrid literature as a W–cycle[36] as seen in Figure 3.

The MGBP algorithm has been updated from version 1 (MGPBv1) in [33] to version
2 (MGPBv2) in [35] but the core idea explained above has remained unchanged.
The latest Multi–Grid Back–Projection algorithm (MGBPv2) is shown in Figure 4
that follows from Algorithm 1.

Figure 5: a) A video clip is represented as a spatio-temporal cube. b) A
back–projection module using a convolution and transposed convolution with
same kernel size preserve the dimensions of the cube. Padding is no needed and
can reduce the memory footprint at low resolution levels.

Cube–to–Cube property. When we upscale sequences of images that come from
video streams it is well known that temporal information plays a crucial role.
Independently super–resolving frames looses important temporal correlations
and introduces temporal artifacts such as flickering. A simple model to add
this information is to interpret time as an extra dimension. In this
interpretation flickering artifacts are the temporal version of jaggies in
space. We can visualize the spatio–temporal information of a video clip in a
cubical arrangement of pixels as seen in Figure 5–a. The cross sections of the
cube along temporal dimension reveal a smooth continuity of pixel value
changes across time. A common solution for video super–resolution is to use 3D
convolutions to extract this information based on the fact that these cross
sections share strong similarities with 2D images (eg. sharp and smooth
edges).

Extending a network model from 2D to 3D convolutions is particularly simple
when using back–projections because of the property shown in Figure 5–b. As we
already know, a back–projection module uses a strided convolution to move down
in scale and strided transposed convolution to move back. Here, we assume that
both convolution and transposed convolutions use the same kernel sizes and
padding settings. Because the matrix representation of this transposed
convolution has the same dimensions of the transposed matrix for its
convolution counterpart, the residual update in the back–projection module has
the exact same resolution as the initial state. This remains true with or
without padding in the convolutional modules. We call this the _Cube–to–Cube_
property of back–projection modules. The property holds for any number of
dimension (e.g. using 1D, 2D or 3D convolutions) but it is particularly useful
for higher dimensions. This is because by removing padding we save a fix
amount of pixels in 1D, a fix amount of lines (rows or columns) in 2D, and a
fix amount of images in 3D, without reducing the receptive field of
convolutional layers. The memory saving becomes significant when processing
video as we move down in scale using back–projection blocks. For example, we
used $37$ frames to train a $1\times 16\times 16\times$ MGBP–3D (scaling in
space and not in time) that became $29$ frames at the lowest resolution of the
network when using kernel size $3$ in time and no padding, compared to $37$
frames with padding. The volume saving at the lowest resolution is $\sim 22\%$
which allows us to use a large number of features ($256$). This is a clear
advantage of using a fully 3D–convolutional network structure.

Figure 6: For video inference, using a fully 3D–convolutional architecture
such as MGBP–3D is more difficult than image since in practice we are unable
to input a 3D–cube with the entire video sequence. We solve this issue by
using overlapping 3D–patches that are averaged using a Hadamard window.

Video Inference. To upscale large video sequences we propose a patch based
approach in which we average the output of overlapping video patches taken
from the bicubic upscaled input. First, we divide input streams into
overlapping patches (of same size as training patches) as shown in Figure 6;
second, we multiply each output by weights set to a Hadamard window; and
third, we average the results. In our experiments we use overlapping patches
separated by $5$ video frames.

This strategy is necessary for video as we cannot input a spatio–temporal cube
with the complete input sequence. This limitation of fully 3D–convolutional
networks could be avoided if tensor processing frameworks implement _streaming
3D–convolutions_ where output frames come out as input frames arrive, using a
FIFO structure equivalent to the implementation of audio filters.

Generator Architecture. Algorithm 1 refers to the second version of MGBP [35].
One of the differences between the first and second versions is the
initialization step. Without loss of generality, in the second version we
tackled the image enhancement problem with an input resolution equal to the
output resolution. For the super–resolution task we input the low resolution
image upscaled using a bicubic method. This helps to make the system become
more general for applications and simplifies the process of generating pairs
of input/output patches during training. In contrast, the first version MGBPv1
shown in Figure 3 started at the lowest resolution and progressively upscaled
the input with a bicubic upscaler, entering the multigrid cycle at different
resolutions.

Figure 4 displays the diagram of the MGBPv2 algorithm unfolded for $\mu=2$ and
$L=5$. The _Analysis_ and _Synthesis_ modules convert an image into feature
space and vice–versa using single convolutional layers. The _Upscaler_ and
_Downscaler_ modules are composed of single strided (transposed and
conventional) convolutional layers. An important observation in Algorithm 1 is
the use of a tag label to differentiate each _Upscaler_ and _Downscaler_
module. This simple trick was introduced in MGBPv2 and makes every _Upscaler_
and _Downscaler_ module different in terms of parameters and hyper–parameters.
In particular, we can now set a different number of features in convolutional
layers from low to high resolution levels, allowing large images to be
processed at high resolutions by using small number of features and increase
it at lower resolutions. The system can be initialized by a _dry run_ of
Algorithm 1 where no computation is performed and modules are defined with
their correspondent tag labels. The MGBPv1 system did not include this feature
and it was force to use identical _Upscaler_ and _Downscaler_ in all instances
of these modules. In this case a single convolutional layer would not give
good results and $4$–layer DenseNets were used inside these modules [33].

The same diagram for MGBPv2 in Figure 4 represents the architecture of MGBP–3D
and the same Algorithm 1 is used for video SR. The difference between MGBPv2
and MGBP–3D is in the configuration of the _Analysis_ , _Synthesis_ ,
_Upscaler_ and _Downscaler_ modules that now used 3D–convolutions as shown in
Figure 4. This simple extension is due to the _Cube–to–Cube_ property
explained above.

Complexity. The W–cycle pattern shown in Figure 4 is unfolded when using
$\mu=2$, and it is also a popular configuration in multigrid methods [36]. Let
$f(n)$ be the computational complexity of the residual backprojection module
in a resolution of $n=H\times W$ pixels. For example,
$f(n)=\mathcal{O}(n^{1/2})$ if we use the same _Upscaler_ and _Downscaler_
modules all over the network. Then, the number of products $p(n)$ at
resolution $n$ using MGBP obeys the recurrence relation:

$p(n)=f(n)+\mu\cdot p(n/4)\;,$ (6)

assuming a downscaling by factor $2\times 2$ (horizontal and vertical). The
solutions of this recurrence are known as the master theorem for
divide–and–conquer [52]. In MGBPv1 we have $f(n)=\mathcal{O}(n^{1/2})$ and
$\mu=2$, that leads to a computational complexity $p(n)=\mathcal{O}(n\log n)$.
This is inconvenient in practice and it made it impossible, for example, to
apply MGBPv1 in the Extreme–SR AIM 2019 challenge with upscaling factor
$16\times$ and 8K output resolution. The complexity can be reduced to
$\mathcal{O}(\log n)$ if we manipulate the _Upscaler_ and _Downscaler_ modules
such that $f(n)=\mathcal{O}(n^{1/2-\epsilon})$ for some $\epsilon>0$. This is
easily achieved by both MGBPv2 and MGBP–3D by reducing the number of features
per level in convolutional layers as a function of $n$.

In terms of memory requirements the resolution of feature maps decreases by
factor $2\times 2=4$ at each scale (using stride $2$ in _Upscaler_ and
_Downscaler_ modules). All versions of MGBP need to keep one copy of the
feature map at each scale (similar to UNets) and the geometric sum of pixel
counts from different scales leads to $\mathcal{O}(n)$. Therefore, MGBP memory
footprint scales correctly by linearly increasing the memory requirements as
the number of pixels increases.

Figure 7: Discriminator system used for adversarial training. The high
resolution input image is downsized with standard bicubic downscalers to enter
the system at different scales. CNN modules do not share parameters. Only
image content is considered for adversarial training as the temporal evolution
of artificial details required for videos makes the problem significantly more
difficult and currently unresolved.  Figure 8: Our VN+SC layer considers the
luminance channel of a color image (using the BT.609 standard for conversion
from RGB), applies variance normalization (upper right image), and computes
the Hadamard product with its shift versions (bottom image showing only
outputs for the white box in variance normalization image). We use these $49$
channels as inputs of the Discriminator.

###  III-B Discriminator

The task of the discriminator is to measure how realistic is an image or video
frame. A straightforward approach is to input the color image to a sequential
convolutional network architecture. Then, we hope that the discriminator
learns from adversarial training using real and fake image examples. In
practice, we find that this approach works well to identify which areas of
upscale images need more textures but the artificial details look noisy and
have limited structure.

So what makes an image look natural? Extensive research has been carried to
address this question. Here, we follow the seminal work of Ruderman[46] who
found regular statistical properties in natural images that are modified by
distortions. In particular, Ruderman observed that applying the so–called
_variance normalization_ operation:

$\hat{I}_{i,j}=\frac{I_{i,j}-\mu_{i,j}(I)}{\sigma_{i,j}(I)+1}\;,$ (7)

has a decorrelating effect on natural images. Here, $I_{i,j}$ is the luminance
channel of an image with values in $[0,255]$ at pixel $(i,j)$, $\mu(I)$ is the
local mean of $I$ (e.g. output of a Gaussian filter), and
$\sigma(I)^{2}=\mu(I^{2})-\mu^{2}(I)$ is the local variance of $I$. Ruderman
also observed that these normalized values strongly tend towards a Gaussian
characteristic for natural images. These findings are used in the NIQE
perceptual quality metric considered for the PIRM–SR Challenge 2018[47]. NIQE
also models the statistical relationships between neighboring pixels by
considering horizontal and vertical neighbor products:
$\hat{I}_{i,j}\hat{I}_{i,j+1}$, $\hat{I}_{i,j}\hat{I}_{i+1,j}$,
$\hat{I}_{i,j}\hat{I}_{i,j-1}$ and $\hat{I}_{i,j}\hat{I}_{i-1,j}$.

Variance Normalization and Shift Correlator (VN+SC). Inspired by previous
research we define the Variance Normalization and Shift Correlator (VN+SC)
layer as follows:

$V^{7(p+3)+q+3}_{i,j}(I)=\hat{I}_{i,j}\cdot\hat{I}_{i+p,j+q}\;,\quad\begin{array}[]{ll}p=-3,\ldots,3,\\\
q=-3,\ldots,3\;.\end{array}$ (8)

Here, we transform a color image into a set of neighbor products (shift
correlator) $V^{k}_{i,j}$ with $k=0,\ldots,48$, using the variance normalized
image $\hat{I}$. The Gaussian filter used for local mean and variance is set
to have kernel size $7\times 7$ and $\sigma=1.17$ based on similar values used
in NIQE[47]. The number of neighbor products is an additional parameter that
we set to $7\times 7$ that is a number larger than $3\times 3$ used in NIQE
and BRISQUE and close to $64$ that is the number of features used in the
discriminator architecture. The luminance channel is obtained using the BT.609
color matrix following the implementation of NIQE[47].

Figure 8 shows the visual effect of the the VN+SC operation. A zoom on the
tail of a zebra shows a visible correlation along the tail and less so in the
stripes of the zebra. We use a VN+SC layer for each input of our
discriminator, as shown in Figure 7. This reduces the amount of information
received by the discriminator and, in particular, removes all color
information. This follows closely the empirical findings of Ruderman[46] to
determine if an image looks natural. Color information will be considered by
other terms of the loss function.

Discriminator Architecture. For the discriminator in adversarial training we
use the system shown in Figure 7. Here, we use $4$–layer sequential CNNs with
$3\times 3$ filters and stride $1$ except for the last layer that uses stride
$2$ to downscale the features. The discriminator used with MGBPv1 in [34] had
two minor differences: first, the CNNs shared parameters making the system
recursive (imitating MGBPv1); and second, the progressive outputs of MGBPv1
entered the discriminator at different stages. Adversarial training with this
discriminator resembles a Progressive GAN[37] because it can adjust parameters
to first solve the simpler problem of $2\times$ upscaling, and then follow
with larger factors. But, at the same time, it is significantly different
because a Progressive GAN system is neither multi–scale nor recursive.
Overall, the major properties of our discriminator architecture, both for
MGBPv1 and MGBPv2, are: input the RGB output of the generator in different
resolutions (high resolution enter first, lower resolutions enter later); and,
RGB images are converted into $49=7\times 7$ channels with the VN+SC modules.
A 3D–convolutional version of the discriminator was used for video content in
the AIM 2020 Video Extreme Super–Resolution Challenge [30] but did not succeed
to achieve the same realistic effect observed in images. The temporal
evolution of artificial details required for videos makes the problem
significantly more difficult and currently unresolved. Therefore, here we will
only consider the adversarial training strategy for image content.

##  IV Learning

###  IV-A Configurations

In the first version of MGBP (MGBPv1) the _Analysis_ , _Synthesis_ , _Upscale_
, _Downscale_ and _CNN_ –discriminator modules used $4$–layer dense
networks[12] with a correspondent strided convolutional or transposed
convolutional layer for downscale or upscale operations, respectively. We used
$48$ features and growth rate $16$ within dense networks. For classic upscaler
we configured a Bicubic and we set the upscaling filters as parameters to
learn as proposed in [53].

In the second version, MGBPv2, the _Analysis_ , _Synthesis_ , _Upscale_ ,
_Downscale_ were configured with single convolutional layers. Here, we used
different configurations depending on the upscaling factor. Table I–a shows
the number of levels, the parameter $\mu$ and the number of convolutional
layer features used for each upscaling factor. For small upscaling factors
($2$ and $3$) we chose a configuration almost equivalent to EDSR [11] with
only two levels and $32$ residual back–projection blocks.

For MGBP–3D the _Analysis_ , _Synthesis_ , _Upscale_ , _Downscale_ were
configured using single 3D–convolutional layers as shown in Figure 4. Similar
to MGBPv2, we used different configurations depending on the upscaling factors
$4\times$ and $16\times$. Table I–b shows the number of levels, the parameter
$\mu$ and the number of convolutional layer features used for each upscaling
factor.

TABLE I: MGBPv2 and MGBP–3D network configurations. High–fidelity performance
is shown in Tables II and IV.

a) MGBP–v2:  Factor $\mu$ Levels Channels per level 2 32 2 192–128 3 32 2
192–128 4 4 4 192–128–64–32 8 2 5 192–128–64–32–16 16 2 6 256–192–128–92–48–9

b) MGBP–3D:  Factor $\mu$ Levels Channels per level 4 6 4 192–128–64–32 16 2 6
256–192–128–92–48–9

For perceptual quality we concatenate a single noise channel,
$W\cdot\mathcal{N}(0,1)$, to the bicubic upscaled input in MGBPv2 and the
input at each resolution level in MGBPv1 as shown in Figures 3 and 4. The
amplitude of the noise, $W\in\mathbb{R}$, is set to zero for high–fidelity
targets and its purpose is to help in the adversarial training for perceptual
quality. The noise activates and deactivates the generation of artificial
details. In MGBPv1 [34], different i.i.d. noise channels are generated at each
resolution. In MGBPv2 we generate a single noise channel at the highest
resolution, that moves along with the input image to enter the network at
different scales by using _Analysis_ modules with different strides and
becomes more simple to manipulate.

We denote $Y_{W=0}$ and $Y_{W=1}$ the outputs of the generator architecture
using noise amplitudes $W=0$ and $W=1$, respectively. In all the experiments
and challenges that we present here, a high–resolution image has been resized
using a bicubic downscaler. Let $S_{f}$ represent a bicubic downscaler that
reduces the resolution by a factor $f\times f$ (horizontal and vertical). We
would like our output $Y$ to be such that $S_{f}(X)=S_{f}(Y)$ where $X$ is the
ground truth image. This is a prior knowledge that we will enforce in our loss
functions during optimization.

###  IV-B Learning High Fidelity

We use $\mathcal{L}^{L1}(x,y)=\mathbb{E}\left[|x-y|\right]$ at several
resolutions to define the total loss function:

$\displaystyle\mathcal{L}(Y,X;\theta)=\;$
$\displaystyle\mathcal{L}^{L1}(Y_{W=0},X)\;+$
$\displaystyle\sum_{k\in\\{2,4,\ldots,f\\}}\mathcal{L}^{L1}(S_{k}(Y_{W=0}),S_{k}(X))\;.$
(9)

The first term is our main target, this is, to recover the high–resolution
image. The sum over downscaling factors is meant to progressively give easier
targets and enforce the downscaling model. After every epoch we evaluate the
current model using $\mathcal{L}^{L2}(x,y)=\mathbb{E}\left[(x-y)^{2}\right]$
in the validation metric:

$\displaystyle\mathcal{V}(Y;\theta)=\mathcal{L}^{L2}(Y_{W=0},X)\;.$ (10)

We recorded the best models according to this metric (directly related to
PSNR) during the training process.

The input $X$ represents a 2D image or a 3D spatio–temporal cube for video
content. In both MGBP–v2 and MGBP–3D the input $X$ correspond to the bicubic
upscaled content with the same resolution as the output $Y$. For video content
this means that the network learns to map a number of input frames to the same
number of output frames. This is significantly different than the many–to–one
frame mapping approach commonly used for video SR training and it is a
consequence of the _Cube–to–Cube_ property of MGBP–3D.

Figure 9: Our loss function tries to: look real by moving into
$\mathcal{H}^{real}$ (GAN and CX loss), enforce a downscaling model by moving
into $\mathcal{A}(X)$, and be reachable by latent space interpolation from the
optimal distortion solution $X^{*}$ (distortion loss).

###  IV-C Learning Perceptual Quality

For perceptual training we use a generative adversarial approach, alternating
training steps for the generator and the discriminator, following the
Relativistic GAN from [54]. For total loss we use the following expression:

$\displaystyle\mathcal{L}(Y,X;\theta)=\;$ $\displaystyle
0.001\cdot{\color[rgb]{0.33203125,0.83203125,0}\mathcal{L}^{RSGAN}_{G}(Y_{W=1})}\;+$
$\displaystyle 10\cdot{\color[rgb]{1,0,0}\tfrac{1}{|C|}\sum_{k\in
C}\mathcal{L}^{L1}(S_{k}(Y_{W=1}),S_{k}(X))}\;+$ $\displaystyle
0.1\cdot{\color[rgb]{0.33203125,0.83203125,0}\mathcal{L}^{CX}(Y_{W=1},X)}\;+$
$\displaystyle 10\cdot{\color[rgb]{0,0,1}\mathcal{L}^{L1}(Y_{W=0},X)}\;+$
$\displaystyle 10\cdot{\color[rgb]{1,0,0}\tfrac{1}{|C|}\sum_{k\in
C}\mathcal{L}^{L1}(S_{k}(Y_{W=0}),S_{k}(X))}\;,$ (11)

where the color is associated with the targets displayed in Figure 9 and
$C=\\{2,4,\ldots,f\\}$ is the set of upscaling factors. This is, green losses
try to enter in the set of real images with $Y_{W=1}$; red losses try to enter
the set of aliased images with both $Y_{W=1}$ and $Y_{W=0}$; and, blue losses
try to recover the ground truth high–resolution image with $Y_{W=0}$.

The red losses in (11) enforcing the downscale model are also refer here as
_cycle_ losses. This is because the output images are brought back to the
input domain for comparison with the original input.

In particular, $\mathcal{L}^{CX}$ is the _contextual loss_ as defined in [45]
using features from _conv3–4_ of a VGG–19 network as suggested in [55].
Ablation tests performed in the next section show the effectiveness of this
loss function to improve perceptual quality while maintaining a reasonable
level of distortion. Next, the Relativistic GAN loss follows the definition in
[54], given by:

$\displaystyle\mathcal{L}_{D}^{RSGAN}=$
$\displaystyle-\mathbb{E}_{(R,F)}\left[\log(\text{sigmoid}(C(R)-C(F)))\right]\;,$
$\displaystyle\mathcal{L}_{G}^{RSGAN}=$
$\displaystyle-\mathbb{E}_{(R,F)}\left[\log(\text{sigmoid}(C(F)-C(R)))\right]\;.$
(12)

Here, $C$ is the output of the discriminator before the sigmoid function, as
shown in Figure 7. And $R$ and $F$ are the sets of real and fake inputs to the
discriminator, given by:

$\displaystyle F=$
$\displaystyle\left\\{Y_{W=1},S_{2}(Y_{W=1}),S_{4}(Y_{W=1}),\ldots\right\\}\;,$
$\displaystyle R=$
$\displaystyle\left\\{X,S_{2}(X),S_{4}(X),\ldots\right\\}\;.$ (13)

After every epoch we evaluated the current model according to the validation
metric based on the NIQE[47] index:

$\displaystyle\mathcal{V}(Y;\theta)=\mathbb{E}\Big{[}$ $\displaystyle
NIQE(Y_{W=1})\Big{]}\;.$ (14)

This metric works as a simple rule to help identify models that generate
realistic images in the full resolution.

TABLE II: Quantitative evaluation of MGBPv1 and MGBPv2 for image
super–resolution with high–fidelity target.

|  | Set14 | BSDS100 | Urban100 | Manga109
---|---|---|---|---|---
Algorithm |  | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Bicubic | $2\times$ | 30.34 | 0.870 | 29.56 | 0.844 | 26.88 | 0.841 | 30.84 | 0.935
A+ [56] | 32.40 | 0.906 | 31.22 | 0.887 | 29.23 | 0.894 | 35.33 | 0.967
FSRCNN [57] | 32.73 | 0.909 | 31.51 | 0.891 | 29.87 | 0.901 | 36.62 | 0.971
SRCNN [9] | 32.29 | 0.903 | 31.36 | 0.888 | 29.52 | 0.895 | 35.72 | 0.968
MSLapSRN [53] | 33.28 | 0.915 | 32.05 | 0.898 | 31.15 | 0.919 | 37.78 | 0.976
VDSR [58] | 32.97 | 0.913 | 31.90 | 0.896 | 30.77 | 0.914 | 37.16 | 0.974
LapSRN [59] | 33.08 | 0.913 | 31.80 | 0.895 | 30.41 | 0.910 | 37.27 | 0.974
DRCN [60] | 32.98 | 0.913 | 31.85 | 0.894 | 30.76 | 0.913 | 37.57 | 0.973
D-DBPN [39] | 33.85 | 0.919 | 32.27 | 0.900 | 32.70 | 0.931 | 39.10 | 0.978
EDSR [11] | 33.92 | 0.919 | 32.32 | 0.901 | 32.93 | 0.935 | 39.10 | 0.977
RDN [61] | 34.28 | 0.924 | 32.46 | 0.903 | 33.36 | 0.939 | 39.74 | 0.979
RCAN [15] | 34.12 | 0.921 | 32.41 | 0.903 | 33.34 | 0.938 | 39.44 | 0.979
MGBPv1 [33] | 33.27 | 0.915 | 31.99 | 0.897 | 31.37 | 0.920 | 37.92 | 0.976
MGBPv2 | 34.18 | 0.922 | 31.70 | 0.887 | 33.31 | 0.936 | 39.42 | 0.978
Bicubic | $3\times$ | 27.55 | 0.774 | 27.21 | 0.739 | 24.46 | 0.735 | 26.95 | 0.856
SRCNN [9] | 29.30 | 0.822 | 28.41 | 0.786 | 26.24 | 0.799 | 30.48 | 0.912
MSLapSRN [53] | 29.97 | 0.836 | 28.93 | 0.800 | 27.47 | 0.837 | 32.68 | 0.939
LapSRN [59] | 29.87 | 0.832 | 28.82 | 0.798 | 27.07 | 0.828 | 32.21 | 0.935
EDSR [11] | 30.52 | 0.846 | 29.25 | 0.809 | 28.80 | 0.865 | 34.17 | 0.948
RDN [61] | 30.74 | 0.850 | 29.38 | 0.812 | 29.18 | 0.872 | 34.81 | 0.951
MGBPv2 | 30.60 | 0.845 | 29.10 | 0.803 | 29.15 | 0.866 | 34.31 | 0.947
Bicubic | $4\times$ | 26.10 | 0.704 | 25.96 | 0.669 | 23.15 | 0.659 | 24.92 | 0.789
A+ [56] | 27.43 | 0.752 | 26.82 | 0.710 | 24.34 | 0.720 | 27.02 | 0.850
FSRCNN [57] | 27.70 | 0.756 | 26.97 | 0.714 | 24.61 | 0.727 | 27.89 | 0.859
SRCNN [9] | 27.61 | 0.754 | 26.91 | 0.712 | 24.53 | 0.724 | 27.66 | 0.858
MSLapSRN [53] | 28.26 | 0.774 | 27.43 | 0.731 | 25.51 | 0.768 | 29.54 | 0.897
VDSR [58] | 28.03 | 0.770 | 27.29 | 0.726 | 25.18 | 0.753 | 28.82 | 0.886
LapSRN [59] | 28.19 | 0.772 | 27.32 | 0.728 | 25.21 | 0.756 | 29.09 | 0.890
DRCN [60] | 28.04 | 0.770 | 27.24 | 0.724 | 25.14 | 0.752 | 28.97 | 0.886
D-DBPN [39] | 28.82 | 0.786 | 27.72 | 0.740 | 26.54 | 0.795 | 31.18 | 0.914
EDSR [11] | 28.80 | 0.788 | 27.71 | 0.742 | 26.64 | 0.803 | 31.02 | 0.915
RDN [61] | 29.01 | 0.791 | 27.85 | 0.745 | 27.01 | 0.812 | 31.74 | 0.921
RCAN [15] | 28.87 | 0.789 | 27.77 | 0.744 | 26.82 | 0.809 | 31.22 | 0.917
MGBPv1 [33] | 28.43 | 0.778 | 27.42 | 0.732 | 25.70 | 0.774 | 30.07 | 0.904
MGBPv2 | 29.00 | 0.790 | 27.87 | 0.745 | 27.08 | 0.820 | 31.45 | 0.917
Bicubic | $8\times$ | 23.19 | 0.568 | 23.67 | 0.547 | 20.74 | 0.516 | 21.47 | 0.647
A+ [56] | 23.98 | 0.597 | 24.20 | 0.568 | 21.37 | 0.545 | 22.39 | 0.680
FSRCNN [57] | 23.93 | 0.592 | 24.21 | 0.567 | 21.32 | 0.537 | 22.39 | 0.672
SRCNN [9] | 23.85 | 0.593 | 24.13 | 0.565 | 21.29 | 0.543 | 22.37 | 0.682
MSLapSRN [53] | 24.57 | 0.629 | 24.65 | 0.592 | 22.06 | 0.598 | 23.90 | 0.759
VDSR [58] | 24.21 | 0.609 | 24.37 | 0.576 | 21.54 | 0.560 | 22.83 | 0.707
LapSRN [59] | 24.44 | 0.623 | 24.54 | 0.586 | 21.81 | 0.582 | 23.39 | 0.735
D-DBPN [39] | 25.13 | 0.648 | 24.88 | 0.601 | 22.83 | 0.622 | 25.30 | 0.799
EDSR [11] | 24.94 | 0.640 | 24.80 | 0.596 | 22.47 | 0.620 | 24.58 | 0.778
RDN [61] | 25.38 | 0.654 | 25.01 | 0.606 | 23.04 | 0.644 | 25.48 | 0.806
RCAN [15] | 25.23 | 0.651 | 24.98 | 0.606 | 23.00 | 0.645 | 25.24 | 0.803
MGBPv1 [33] | 24.82 | 0.635 | 24.67 | 0.592 | 22.21 | 0.603 | 24.12 | 0.765
MGBPv2 | 25.37 | 0.652 | 25.08 | 0.606 | 22.99 | 0.640 | 25.07 | 0.795

##  V Performance

Figure 10: Qualitative evaluation of $16\times$ video SR results from the AIM 2020 Video Extreme Super–Resolution Challenge [30]. MGBP–3D frames can be downloaded from here. TABLE III: Quantitative comparison on REDS4 for $4\times$ video Super–Resolution. MGBP–3D frames can be downloaded from here. Method | Clip_000 | Clip_011 | Clip_015 | Clip_020 | Average
---|---|---|---|---|---
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Bicubic | 24.55 | 0.6489 | 26.06 | 0.7261 | 28.52 | 0.8034 | 25.41 | 0.7386 | 26.14 | 0.7292
RCAN [29] | 26.17 | 0.7371 | 29.34 | 0.8255 | 31.85 | 0.8881 | 27.74 | 0.8293 | 28.78 | 0.8200
TOFlow [25] | 26.52 | 0.7540 | 27.80 | 0.7858 | 30.67 | 0.8609 | 26.92 | 0.7953 | 27.98 | 0.7990
DUF [26] | 27.30 | 0.7937 | 28.38 | 0.8056 | 31.55 | 0.8846 | 27.30 | 0.8164 | 28.63 | 0.8251
EDVR [27] | 28.01 | 0.8250 | 32.17 | 0.8864 | 34.06 | 0.9206 | 30.09 | 0.8881 | 31.09 | 0.8800
MGBP–3D | 27.04 | 0.7793 | 29.11 | 0.8210 | 31.83 | 0.8855 | 27.77 | 0.8286 | 28.94 | 0.8286

###  V-A Fidelity Evaluation

In image SR we use DIV2K[62] and FLICKR–2K datasets for training and the
following datasets for test: Set–14[63], BSDS–100[64], Urban–100[65] and
Manga–109[66]. Impaired images were obtained by downscaling and then upscaling
ground truth images, using bicubic scaler, with scaling factors: $2\times$,
$3\times$, $4\times$ and $8\times$. For evaluation we measure PSNR and SSIM on
the Y–channel using the Matlab code from [67].

We follow the training settings from [11]. In each training batch, we randomly
take $16$ impaired patches from our training set ($800$ DIV2K plus $2,650$
FLICKR–2K images). We consider both MGBPv1 and MGBPv2 models for each
upscaling factor $f=2,4$ and $8$. We use patch size $48f\times 48f$, for
$f=2,3,4$ and $8$. We augment the patches by random horizontal/vertical
flipping and rotating $90^{\circ}$. We use Adam optimizer[68] with learning
rate initialized to $10^{-4}$ and decreased by half every $200,000$
back–propagation steps.

Table II shows the performance at different upscaling factors. We observe that
MGBPv2 performs very close to the best benchmarks, and performs better for
large upscale factors $4\times$ and $8\times$. MGBPv1 performance is lower
than MGBPv2, but it is only outperformed by systems with large number of
parameters like EDSR, DBPN, RDN, RCAN and MGBPv2. At small upscaling factors
$2\times$ and $3\times$ we used an MGBPv2 configuration very similar to EDSR,
with $32$ residual blocks in two levels, but MGBPv2 clearly outperforms EDSR.
This indicate the advantage of the residual back–projection block compared to
conventional residual blocks. The multigrid recursion is effective in
improving performance for large upscaling factors. It must be noted that the
major trend in CNN architectures is to do all processing at low resolution and
use PixelShuffle layers to quickly move to the highest resolution. MGBP
balances this process by doing most work on the lowest resolutions and
decrease the amount of work exponentially as it goes to higher resolutions.

In video SR we trained MGBP–3D using the REDS dataset [69] for $4\times$ and
Vid3oC [70] for $16\times$ upscaling. In particular, for $4\times$ we used the
REDS4 configuration as specified in [27], training MGBP–3D with $266$ video
sequences and testing with $4$ video sequences. Quantitative results are
included in Table III. Here, MGBP–3D outperforms the image SR network RCAN
[29] based on attention modules, the video SR network TOFlow [25] using
optical flow estimation, and the video SR network DUF [26] using dynamic
upscaling filters. But MGBP–3D is still far from the video SR network EDVR
[27] using deformable convolutions. In extreme video SR ($16\times$) we refer
to the results of MGBP–3D in the AIM 2020 Challenge on Video Extreme
Super–Resolution [30] with quantitative evaluations shown in Table IV and
qualitative evaluations shown in Figure 10. Team KirinUK won the competition
introducing EVESR–Net, using deformable convolutions for alignment and new
attention modules [30]. MGBP–3D shared the $2^{nd}$ place with Team–WVU that
also used an architecture based on deformable convolutions. Overall, the
result for both $4\times$ and $16\times$ indicate that MGBP–3D gains advantage
compared to other methods for large upscale factors. The arrangement of the
MGBP–3D architecture due to the multigrid strategy is particularly effective
considering that the architecture only uses 3D–convolutions and ReLU units.
This is in contrast to other methods using attention, deformable convolutions,
warping or other non–linear modules.

TABLE IV: Quantitative results for AIM 2020 Video Extreme Super–Resolution Challenge ($16\times$)[30]. Fidelity | Team | PSNR | SSIM | Runtime | Perceptual
---|---|---|---|---|---
Rank | Rank
1 | KirinUK | 22.83 | 0.6450 | 6.1s | 1
2 | MGBP–3D | 22.48 | 0.6304 | 4.83s | 4
2 | Team-WVU | 22.48 | 0.6378 | 4.90s | 2
3 | sr_xxx | 22.43 | 0.6353 | 4s | 5
4 | ZZX | 22.28 | 0.6321 | 4s | 3
5 | lyl | 22.08 | 0.6256 | 13s | 6
6 | TTI | 21.91 | 0.6165 | 0.249s | –
7 | CET_CVLab | 21.77 | 0.6112 | 0.04s | 7
| Bicubic | 20.69 | 0.5770 |  |

###  V-B Perceptual Evaluation

For image perceptual quality evaluations we refer to the results of MGBPv1 and
MGBPv2 in the first challenge on perceptual image SR, PIRM–SR 2018 [32], and
the first extreme SR challenge, AIM Extreme–SR 2019 [40], respectively.

Table V shows our best average scores and rankings in the PIRM–SR Challenge
2018[32] for Region 1 ($RMSE\leqslant 11.5$), Region 2 ($11.5<RMSE\leqslant
12.5$) and Region 3 ($12.5<RMSE\leqslant 16$). Figure 11 shows all submissions
in the perceptual–distortion plane, including the baseline methods: EDSR[11],
CX[45] and EnhanceNet[71].

Figure 11: Perception–distortion plane with average scores in the test set showing all submissions, from all teams in PIRM–SR 2018[32]. Our best scores are shown in green color together with the final ranking in PIRM–SR Challenge 2018. The perception–distortion trade–off is clearly visible with an empty area in the lower–left corner. TABLE V: PIRM 2018 Challenge results. MOS tests were performed only for top submissions [32]. Region 1 |  | Region 2 |  | Region 3
---|---|---|---|---
Rank | Team |  PI | RMSE | MOS |  | Rank | Team |  PI | RMSE | MOS |  | Rank | Team |  PI | RMSE | MOS
$1$ |  IPCV [72] | 2.709 | 11.48 | 2.22 |  | $1$ |  TTI [73] | 2.199 | 12.40 | 2.17 |  | $1$ |  SuperSR [74] | 1.978 | 15.30 | 2.64
$2$ |  MCML [75] | 2.750 | 11.44 | 1.87 |  | $2$ |  IPCV [72] | 2.275 | 12.47 | 2.43 |  | $2$ | MGBPv1 [34] | 2.019 | 14.24 | 2.61
$3$ |  SuperSR [74] | 2.933 | 11.50 | 2.19 |  | $2$ |  MCML [76] | 2.279 | 12.41 | 2.47 |  | $3$ |  IPCV [72] | 2.013 | 15.26 | 2.60
$3$ |  TTI [73] | 2.938 | 11.46 | 1.88 |  | $4$ |  SuperSR [74] | 2.424 | 12.50 | – |  | $4$ |  AIM [77] | 2.013 | 15.60 | –
$5$ |  AIM [77] | 3.321 | 11.37 | – |  | $5$ | MGBPv1 [34] | 2.484 | 12.50 | – |  | $5$ |  TTI [73] | 2.040 | 13.17 | –
$6$ | DSP–whu | 3.728 | 11.45 | – |  | $6$ |  AIM [77] | 2.600 | 12.42 | – |  | $6$ |  Haiyun [78] | 2.077 | 15.95 | –
$7$ | MGBPv1 [34] | 3.817 | 11.50 | – |  | $7$ |  REC–SR [79] | 2.635 | 12.37 | – |  | $7$ | gayNet | 2.104 | 15.88 | –
$7$ |  REC–SR [79] | 3.831 | 11.46 | – |  | $8$ | DSP–whu | 2.660 | 12.24 | – |  | $8$ | DSP–whu | 2.114 | 15.93 | –
$9$ |  Haiyun [78] | 4.440 | 11.19 | – |  | $9$ | XYN | 2.946 | 12.23 | – |  | $9$ | MCML | 2.136 | 13.44 | –
TABLE VI: AIM 2019 Extreme SR Challenge Track 2 Perceptual results and final rankings on the DIV8K test set [40]. Rank | Team/Method | PSNR | SSIM | Runtime
---|---|---|---|---
$1$ | MGBPv2 | 25.44 | 0.6551 | 47.11s
$2$ | TTI | 25.26 | 0.6523 | 80s
$3$ | SRSTAR | 26.72 | 0.7285 | 40s
$4$ | SSRR | 26.53 | 0.7246 | 40s

Compared to other submissions, we observe in Figure 11 that our system
performs better in Region 3. Here, we achieve the $2^{nd}$ place within very
small differences in perceptual scores but with significantly lower
distortion. This shows the advantage of our training strategy to optimize the
perception–distortion trade–off. In Regions 1 and 2 we were one among only two
teams that reached the exact distortion limit ($11.5$ in Region 1 and $12.5$
in Region 2). We were able to achieve this by controlling the noise amplitude,
without retraining the system. Our ranking lowers as the distortion target
gets more difficult. We believe that this is caused by the small size of our
system that becomes more important for low distortion targets, since we use
only $281k$ parameters compared to $43M$ of the EDSR baseline in Region 1.

Figure 12 shows comparisons of our results with the baselines, using images
from our validation set. We observe that in Region 3 we achieve better
perceptual scores even compared to the original HR images. While we
subjectively confirm this in some patches, we do not make the same conclusion
after observing the whole images. Somehow, we believe that our design for
adversarial training and validation strategy managed to overfit the perceptual
scores. Nevertheless, we observe clear advantages to the baselines, showing
better structure in textures and more consistent geometry in edges and shapes.

Figure 12: PIRM-SR 2018 comparisons of $4\times$ upscaling between our
solutions in R1, R2 and R3 (see Figure 11) and baseline methods in our
validation set. Perceptual and distortion scores of whole images are shown in
green and blue colors, respectively.  Figure 13: Example outputs and
performance metrics of our MGBPv2 $16\times$ model used for AIM Extreme–SR
2019 [40]. These images were selected from the DIV8K training set and we used
them for validation purposes.

In the Perceptual track of the Extreme–SR AIM 2019 challenge our results using
MGBPv2 obtained the $1^{st}$ place in terms of a subjective ranking based on
mean opinion scores (MOS) to measure perceptual quality. Our results obtained
an average PSNR of $25.44$ dB in the full output images of the test set. This
is, $0.18$ dB above the $2^{nd}$ place and $1.35$ dB below the best PSNR value
in the Fidelity track.

Figure 13 shows two examples of our best results for the Fidelity and
Perceptual tracks on images used for validation during our training process.
These images show the values of RMSE (measuring fidelity) as well as the
Perceptual index proposed in [32] to objectively measure perceptual quality.
Overall, the results are consistent with the perception/distortion trade–off
in [31]. For the image 622 we observe that our results for the Perceptual
track achieve a Perceptual index better than original image. We attribute this
result to a blurred background in the original image, that our system shows
more focused and with sharper features. For image 1466, the Perceptual index
is clearly below those of the original image. According to our subjective
evaluation, we observe clear differences in the fine level features like
textbook spines. From a far away look these details become less perceptible,
indicating that the Perceptual index correlates better with a close distance
observer. This is probably caused by the resolution of example images used to
adjust the Perceptual index, that are much smaller than 8K.

Figure 14: Evolution of perceptual and fidelity metrics when moving the input
noise amplitude from $W=0$ to $W=1$ in our MGBPv2 $16\times$ system used for
AIM Extreme–SR 2019 [40].

##  VI Analysis

###  VI-A Perception–Distortion Trajectory

An essential part of our generative SR architecture is the noise input. The
training strategy introduced in Section II teaches the system to optimize
distortion when the noise is set to zero, and maximize perceptual quality when
the noise is enabled. Thus, noise provides the randomness needed for natural
images and represents the _innovation jump_ according to Figure 2.

After training, we are free to control the noise inputs. In particular, we can
move the noise amplitude smoothly between $W=0$ and $W=1$ to inspect the path
to jump from distortion to perception optimization. Figure 14 shows an example
of this transition. Our training strategy does not optimize the trajectory in
the perception–distortion plane, but only the corner cases of best distortion
($W=0$) and best perception ($W=1$). The corner cases are clearly verified in
Figure 14. At this point, it is unknown which trajectory will the the network
take to move from one case to the other.

It is interesting to see in Figure 14 that the transition from best perception
to best distortion happens within a narrow margin of $\Delta W=0.2$ amplitude
values. We also observe that the parametric curve in the perception–distortion
plane looks like a monotonically non–increasing and convex function, similar
to the optimal solution studied in [31]. But, it is important to emphasize
that the curve in Figure 14 is not optimal as we are not enforcing optimality.
The results in Table V and Figure 11 clearly show that our
perception–distortion curve, despite being concave, gets farther from
optimality in low distortion regions.

Regarding image quality metrics, we see with no surprise that the _Perceptual_
index proposed for the PIRM–SR Challenge[32] improves as noise increases,
while the distortion measured by RMSE increases. We observed very similar
results for the perceptual metrics NIQE and Ma, as well as the L1 distortion
metric. More interesting is the transition observed in the _contextual
similarity_ index. First, it behaves as a perceptual score with the CX
similarity improving consistently as noise increases. Then, when the
_Perceptual_ score seems to stall, but RMSE keeps increasing, the CX
similarity changes to a distortion metric pattern, reducing as noise
increases. This is consistent with the design target of _CX similarity_ to
focus more on perceptual quality while maintaining a reasonable level of
distortion[45].

###  VI-B Ablation Tests

Figure 15: Ablation tests show the validation scores when training our network
for $100$ epochs. We consider removal of the loss terms: GAN, CX, L1 and Cycle
in (11), as well as VN+SC layers in the discriminator, and training the system
without noise inputs.

Our overall loss combines terms focused on different targets (e.g. low
distortion, perceptual quality). In Section IV-B we explained the purpose of
each term using the diagram in Figures 2 and 9. It remains to verify this
design and to quantify the relevance of each term. We also want to quantify
the contribution of our novel VN+SC layer. For this purpose we trained MGBPv1
network architecture for $100$ epochs using the loss function in section IV-B.
In Figure 15 we show our measurements of L1 (distortion) and NIQE (perceptual)
in a small validation set of $14$ images after each epoch. We display the
evolution through the number of epochs as well as the trajectories on the
perception–distortion plane.

Overall, we see that our strategy adding all the losses (in black color) gives
the best perception–distortion balance. In the extremes we see that removing
the L1 and GAN losses have catastrophic effects on distortion and perception,
respectively. Still, these cases do not diverge to infinity because of other
loss terms. Next, it is clear that the contextual loss helps improving the
perceptual quality, and regarding distortion the amount of improvement is not
conclusive. Then, the addition of the cycle loss shows a clear improvement
over distortion, with inconclusive improvements on perceptual quality. And
finally, we observe that the addition of the VN+SC layer in the discriminator
clearly improves perceptual quality, although not as much as CX and GAN
losses.

Figure 15 also shows a test in which we avoid the use of noise inputs by
setting $W=0$ in all losses. In this case we remove the L1 loss that would
otherwise interfere with the GAN loss, causing a catastrophic effect. In this
case distortion is controlled by the cycle loss, equivalent to how it is done
in [55]. In this configuration the network performs slightly worse in
perceptual quality and clearly worse on distortion, similar to only removing
the L1 loss. In this case, we believe that the network uses the randomness in
the input as innovation process, which cannot be controlled and limits the
diversity.

###  VI-C Interpretability

Figure 16 shows interpretability results obtained by using the Deep Filter
Visualization (DFV) method from [33]. To perform this complex analysis on a
large model such as MGBPv2, with more than $20$ million parameters, we use the
so–called _Linearscope_ method recently introduced in [80]. For a given pixel
in the input image (blue circles on the left side), Figure 16 displays the
impulse response for the network model with frozen activations (all ReLU’s
acting as if the input image did not change). This represents the equivalent
to an upscaling filter that adapts to the pixel location. In flat areas
(example at the bottom of Figure 16) the upscaling filter looks isotropic and
similar to a bicubic upscaler. In other locations, the filter strongly follows
edges in hair and fingers, with receptive fields that extend for several
hundred pixels. Overall, this confirms that the system has learned the
geometry of the content.

Figure 16: Deep filter visualization (DFV)[33, 80] experiments on our MGBPv2
$16\times$ model for the Fidelity track using a patch of size $767\times 767$.
The model shows good knowledge of the geometry and large receptive fields.

##  VII Conclusion

We have proposed the MultiGrid Back–Projection (MGBP) network architecture for
image super–resolution. The network combines: first, a novel type of
cross–scale residual block inspired in the IBP algorithm; and second, a
multigrid recursion that uses cross–scale residual blocks within cross–scale
residual blocks. The result is an architecture that balances computational
complexity in an efficient and effective way, focusing more on lower
resolutions and decreasing computations in higher resolutions. We also
introduce a particular training strategy to achieve good perceptual quality by
using noise inputs to traverse the perception–distortion plane. An
interpretation using the concept of innovation jumps, together with ablation
tests and results on international competitions prove the effectiveness of
this strategy.

## References

  * [1] M. Sugawara, C. S-Y, and D. Wood, “Ultra–high–definition television (Rec. ITU-R BT.2020): A generational leap in the evolution of television,” _Signal Processing Magazine, IEEE_ , vol. 31, no. 3, pp. 170–174, May 2014\.
  * [2] J. Proakis and D. Manolakis, _Digital Signal Processing_ , ser. Prentice Hall international editions. Pearson Prentice Hall, 2007.
  * [3] S. Mallat, _A Wavelet Tour of Signal Processing_. Academic Press, 1998.
  * [4] V. R. Algazi, G. E. Ford, and R. Potharlanka, “Directional interpolation of images based on visual properties and rank order filtering,” in _Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing_ , vol. 4. Toronto, ON: IEEE Signal Processing Society, May 1991, p. 3005–3008.
  * [5] X. Li and M. T. Orchard, “New edge–directed interpolation,” _IEEE Transactions on Image Processing_ , vol. 10, no. 10, pp. 1521–1527, October 2001\.
  * [6] S. Mallat and G. Peyre, “A review of bandlet methods for geometrical image representation,” _Numerical Algorithms_ , vol. 44, no. 3, pp. 205–234, March 2007.
  * [7] S. Park, M. Park, and M. Kang, “Super–resolution image reconstruction: a technical overview,” _Signal Processing Magazine, IEEE_ , vol. 20, no. 3, pp. 21–36, May 2003.
  * [8] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, no. 7553, pp. 436–444, May 2015.
  * [9] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super–resolution,” in _in Proceedings of European Conference on Computer Vision (ECCV)_ , 2014.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _computer vision and pattern recognition_ , pp. 770–778, 2016\.
  * [11] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, “Enhanced deep residual networks for single image super–resolution,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , July 2017.
  * [12] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [13] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in _CVPR_ , 2018.
  * [14] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” _computer vision and pattern recognition_ , pp. 7132–7141, 2018.
  * [15] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 286–301.
  * [16] X. Wang, R. B. Girshick, A. Gupta, and K. He, “Non-local neural networks,” _computer vision and pattern recognition_ , pp. 7794–7803, 2018.
  * [17] Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual non–local attention networks for image restoration,” _international conference on learning representations_ , 2019.
  * [18] S. Farsiu, M. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” _IEEE Transactions on Image Processing_ , vol. 13, no. 10, pp. 1327–1344, 2004.
  * [19] C. Liu and D. Sun, “On bayesian adaptive video super resolution,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 2, pp. 346–360, 2014.
  * [20] Z. Ma, R. Liao, X. Tao, L. Xu, J. Jia, and E. Wu, “Handling motion blur in multi-frame super-resolution,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 5224–5232.
  * [21] A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, “Video super-resolution with convolutional neural networks,” _IEEE Transactions on Computational Imaging_ , vol. 2, no. 2, pp. 109–122, 2016.
  * [22] R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia, “Video super-resolution via deep draft-ensemble learning,” in _2015 IEEE International Conference on Computer Vision (ICCV)_ , 2015, pp. 531–539.
  * [23] D. Liu, Z. Wang, Y. Fan, X. Liu, Z. Wang, S. Chang, and T. Huang, “Robust video super-resolution with learned temporal dynamics,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 2526–2534.
  * [24] X. Tao, H. Gao, R. Liao, J. Wang, and J. Jia, “Detail-revealing deep video super-resolution,” in _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2017, pp. 4482–4490.
  * [25] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” _International Journal of Computer Vision_ , vol. 127, no. 8, pp. 1106–1125, 2019.
  * [26] Y. Jo, S. W. Oh, J. Kang, and S. J. Kim, “Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3224–3232.
  * [27] X. Wang, K. C. Chan, K. Yu, C. Dong, and C. C. Loy, “Edvr: Video restoration with enhanced deformable convolutional networks,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2019, pp. 0–0.
  * [28] J. Chen, X. Tan, C. Shan, S. Liu, and Z. Chen, “VESR-Net: The winning solution to youku video enhancement and super-resolution challenge,” _arXiv preprint arXiv:2003.02115_ , 2020.
  * [29] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 294–310.
  * [30] D. Fuoli, Z. Huang, S. Gu, R. Timofte, A. Raventos, A. Esfandiari, S. Karout, X. Xu, X. Li, X. Xiong, J. Wang, P. Navarrete Michelini, W. Zhang, D. Zhang, H. Zhu, D. Xia, H. Chen, J. Gu, Z. Zhang, T. Zhao, S. Zhao, K. Akita, N. Ukita, H. P. S, D. Puthussery, and J. C, “AIM 2020 challenge on video extreme super-resolution: Methods and results,” _arXiv preprint arXiv:2009.06290_ , 2020.
  * [31] Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018\.
  * [32] Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, and L. Zelnik-Manor, “The 2018 pirm challenge on perceptual image super-resolution,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 0–0.
  * [33] P. Navarrete Michelini, H. Liu, and D. Zhu, “Multigrid backprojection super–resolution and deep filter visualization,” in _Proceedings of the Thirty–Third AAAI Conference on Artificial Intelligence (AAAI 2019)_. AAAI, 2019.
  * [34] ——, “Multi–scale recursive and perception–distortion controllable image super–resolution,” in _The European Conference on Computer Vision Workshops (ECCVW)_ , September 2018. [Online]. Available: http://arxiv.org/abs/1809.10711
  * [35] P. Navarrete Michelini, W. Chen, H. Liu, and D. Zhu, “MGBPv2: Scaling up multi–grid back–projection networks,” in _The IEEE International Conference on Computer Vision Workshops (ICCVW)_ , October 2019. [Online]. Available: https://arxiv.org/abs/1909.12983
  * [36] U. Trottenberg and A. Schuller, _Multigrid_. Orlando, FL, USA: Academic Press, Inc., 2001.
  * [37] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=Hk99zCeAb
  * [38] R. Timofte, S. Gu, J. Wu, L. Van Gool, L. Zhang, M.-H. Yang, and et al., “NTIRE 2018 challenge on single image super-resolution: Methods and results,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2018.
  * [39] M. Haris, G. Shakhnarovich, and N. Ukita, “Deep back–projection networks for super–resolution,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [40] A. Lugmayr, M. Danelljan, R. Timofte, M. Fritsche, S. Gu, K. Purohit, P. Kandula, M. Suin, A. Rajagopalan, N. H. Joon _et al._ , “AIM 2019 challenge on real-world image super-resolution: Methods and results,” _arXiv preprint arXiv:1911.07783_ , 2019.
  * [41] K. Seshadrinathan, R. Soundararajan, A. Bovik, and L. Cormack, “Study of subjective and objective quality assessment of video,” _IEEE Transactions on Image Processing_ , vol. 19, no. 6, pp. 1427–1441, June 2010.
  * [42] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” _IEEE TRANSACTIONS ON IMAGE PROCESSING_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [43] H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” _Trans. Img. Proc._ , vol. 15, no. 2, pp. 430–444, Feb. 2006. [Online]. Available: http://dx.doi.org/10.1109/TIP.2005.859378
  * [44] J. Johnson, A. Alahi, and F. Li, “Perceptual losses for real-time style transfer and super–resolution,” _CoRR_ , vol. abs/1603.08155, 2016. [Online]. Available: http://arxiv.org/abs/1603.08155
  * [45] R. Mechrez, I. Talmi, and L. Zelnik-Manor, “The contextual loss for image transformation with non-aligned data,” _arXiv preprint arXiv:1803.02077_ , 2018.
  * [46] D. L. Ruderman, “The statistics of natural images,” in _Network computation in neural systems_ , vol. 5, no. 4, 1994, pp. 517–548.
  * [47] A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “Completely Blind” Image Quality Analyzer,” _IEEE Signal Processing Letters_ , vol. 20, pp. 209–212, Mar. 2013.
  * [48] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No–reference image quality assessment in the spatial domain,” _IEEE Trans. Image Process_ , pp. 4695–4708, 2012.
  * [49] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang _et al._ , “NTIRE 2017 challenge on single image super-resolution: Methods and results,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , July 2017.
  * [50] S. K. Mitter, “Nonlinear filtering of diffusion processes a guided tour,” in _Advances in Filtering and Optimal Stochastic Control_ , W. H. Fleming and L. G. Gorostiza, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 1982, pp. 256–266.
  * [51] M. Irani and S. Peleg, “Improving resolution by image registration,” _CVGIP: Graph. Models Image Process._ , vol. 53, no. 3, pp. 231–239, Apr. 1991. [Online]. Available: http://dx.doi.org/10.1016/1049-9652(91)90045-L
  * [52] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, _Introduction to algorithms_. MIT press, 2009.
  * [53] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Fast and accurate image super–resolution with deep laplacian pyramid networks,” _arXiv:1710.01992_ , 2017.
  * [54] A. Jolicoeur-Martineau, “The relativistic discriminator: a key element missing from standard gan,” _arXiv preprint arXiv:1807.00734_ , 2018.
  * [55] R. Mechrez, I. Talmi, F. Shama, and L. Zelnik-Manor, “Learning to maintain natural image statistics, [arxiv](https://arxiv.org/abs/1803.04626),” _arXiv preprint arXiv:1803.04626_ , 2018.
  * [56] R. Timofte and V. D. Smet, “Gool, “A+: Adjusted anchored neighborhood regression for fast super–resolution,” in _in Proc. Asian Conf. Comput. Vis. (ACCV_ , 2014.
  * [57] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super–resolution convolutional neural network,” in _in Proceedings of European Conference on Computer Vision (ECCV)_ , 2016.
  * [58] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super–resolution using very deep convolutional networks,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR Oral)_ , June 2016.
  * [59] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super–resolution,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.
  * [60] J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional network for image super–resolution,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR Oral)_ , June 2016.
  * [61] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [62] E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image super-resolution: Dataset and study,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , July 2017.
  * [63] R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in _International conference on curves and surfaces_. Springer, 2010, pp. 711–730.
  * [64] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , vol. 2. IEEE, 2001, pp. 416–423.
  * [65] J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 5197–5206.
  * [66] Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa, “Sketch-based manga retrieval using manga109 dataset,” _Multimedia Tools and Applications_ , vol. 76, no. 20, pp. 21 811–21 838, 2017.
  * [67] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Evaluation code for residual dense networks,” https://github.com/yulunzhang/RDN/blob/master/RDN_TestCode/Evaluate_PSNR_SSIM.m, 2018, [Online; accessed 20-May-2019].
  * [68] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [69] S. Nah, S. Baik, S. Hong, G. Moon, S. Son, R. Timofte, and K. M. Lee, “NTIRE 2019 challenge on video deblurring and super–resolution: Dataset and study,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2019.
  * [70] S. Kim, G. Li, D. Fuoli, M. Danelljan, Z. Huang, S. Gu, and R. Timofte, “The Vid3oC and IntVid datasets for video super resolution and quality mapping,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_. IEEE, 2019, pp. 3609–3616.
  * [71] M. S. M. Sajjadi, B. Scholkopf, and M. Hirsch, “Enhancenet: Single image super-resolution through automated texture synthesis,” in _The IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [72] S. Vasu, N. Thekke Madam, and A. Rajagopalan, “Analyzing perception-distortion tradeoff using enhanced perceptual super-resolution network,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018\.
  * [73] M. Haris, G. Shakhnarovich, and N. Ukita, “Deep back-projection networks for single image super-resolution,” _arXiv preprint arXiv:1904.05677_ , 2019\.
  * [74] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 0–0.
  * [75] M. Cheon, J.-H. Kim, J.-H. Choi, and J.-S. Lee, “Generative adversarial network-based image super-resolution using perceptual content losses,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018\.
  * [76] J.-H. Choi, J.-H. Kim, M. Cheon, and J.-S. Lee, “Deep learning-based image super-resolution considering quantitative and perceptual quality,” _arXiv preprint arXiv:1809.04789_ , 2018.
  * [77] T. Vu, T. M. Luu, and C. D. Yoo, “Perception-enhanced image super-resolution via relativistic generative adversarial networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * [78] X. Luo, R. Chen, Y. Xie, Y. Qu, and C. Li, “Bi-gans-st for perceptual image super-resolution,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * [79] K. Purohit, S. Mandal, and A. Rajagopalan, “Scale-recurrent multi-residual dense network for image super-resolution,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * [80] P. Navarrete Michelini, H. Liu, Y. Lu, and X. Jiang, “A tour of convolutional networks guided by linear interpreters,” in _The IEEE International Conference on Computer Vision (ICCV)_. IEEE, Oct 2019. [Online]. Available: https://arxiv.org/abs/1908.05168

Pablo Navarrete Michelini was born in Santiago, Chile. He received the B.Sc.
in Physics (2001), B.Sc. in Electrical Engineering (2001) and the Electrical
Engineer Degree (2002), from Universidad de Chile at Santiago. He received the
Ph.D. degree in Electrical Engineering from Purdue University at West
Lafayette, in 2008. He worked as a research intern in CIMNE at Technical
University of Catalonia in 2006, and as a visitor student research
collaborator at Princeton University at Princeton, NJ, in 2006–2007. He was
Assistant Professor in the Department of Electrical Engineering at Universidad
de Chile in 2008–2011. He worked on video processing and compression at Yuvad
Technologies in 2011–2013. He joined BOE Technology Group Co., Ltd in 2014 and
he is currently a Research Scientist working on machine learning methods for
image and video processing applications.
---
Wenbin Chen was born in Kaifeng, China. He received a Bachelor degree in
Computer Science and Technology from Xidian University, in 2016, and a Master
degree in Information Technology from Melbourne University, in 2019. He
majored in Embedded System at bachelor level and then transferred to
Artificial Intelligence for his master degree. He joined BOE Technology Group
Co., Ltd in 2019 and he works on image and video processing applications.
---
Hanwen Liu was born in Shandong province, China. He received the Bachelor
degree in Automation (2010) from Central South University at Changsha, Hunan
province. He received the Master degree and Ph.D. (2016) in Armament Science
and Technology from Beijing Institute of Technology at Beijing. He joined BOE
Technology Group Co., Ltd after graduated in 2016 and he is currently a senior
researcher working on deep learning methods for image processing applications.
---
Dan Zhu received the B.E. in Communications Engineering (2014) and M.E. in
Communications Engineering (2017) from JiLin University in China. She joined
BOE Technology Group Co., Ltd in 2017 and is currently working on deep
learning methods for image and video processing applications.
---
Xingqun Jiang received his Ph.D. degree in Electrical and Computer Engineering
at Cornell University in 2009, specialized in chip technology and computer
graphics. In the past 10 years, Dr. Jiang has been working in the field of IOT
and artificial intelligence constantly. He joined BOE Technology Group Co.,
Ltd in 2015, and now serves as Chief Technology Officer in the smart system
business group, primarily responsible for the product and technology
innovation. In recent years, Dr. Jiang has led his team to achieve significant
progress in the research and development of artificial intelligence. In 2019,
they won three global championships in image super-resolution, object
detection and gesture recognition, respectively. More than 20 products and
solutions powered by artificial intelligence technology has been made and
delivered to customers in different markets. Dr. Jiang’s team has totally
contributed over 2200 patents to the corporate, making BOE ranked as 6th in
terms of AI patents among all the Chinese corporates. Currently, Dr. Jiang and
his team is focused on intelligent graphics and computer vision, continuing
the product and technology innovation in IOT and artificial intelligence.
---