# Enhancing Multi-Scale Implicit Learning in Image Super-Resolution with
Integrated Positional Encoding

Ying-Tian Liu Yuan-Chen Guo Song-Hai Zhang
BNRist, Department of Computer Science and Technology, Tsinghua University

###### Abstract

Is the center position fully capable of representing a pixel? There is nothing
wrong to represent pixels with their centers in a discrete image
representation, but it makes more sense to consider each pixel as the
aggregation of signals from a local area in an image super-resolution (SR)
context. Despite the great capability of coordinate-based implicit
representation in the field of arbitrary-scale image SR, this area’s nature of
pixels is not fully considered. To this end, we propose integrated positional
encoding (IPE), extending traditional positional encoding by aggregating
frequency information over the pixel area. We apply IPE to the state-of-the-
art arbitrary-scale image super-resolution method: local implicit image
function (LIIF), presenting IPE-LIIF. We show the effectiveness of IPE-LIIF by
quantitative and qualitative evaluations, and further demonstrate the
generalization ability of IPE to larger image scales and multiple implicit-
based methods. Code will be released.

##  1 Introduction

Single image super-resolution (SISR) aims to reconstruct a visually-natural
high-resolution (HR) image from its low-resolution (LR) counterpart. SISR is
an ill-posed problem since there exist many potential reasonable HR images for
a single LR input and the down-sampling filters for HR images are not
determined. Many methods have been proposed to bridge the gap in signal
frequency between LR and HR images by either regressing high-frequency
textures[52, 9, 24, 42, 13] or generating perceptually reasonable details [23,
43, 34, 16, 26]. Although super-resolution (SR) with predefined scale factor
has been fully explored with convolution-based architectures [9, 23, 22, 19,
24, 52], arbitrary-scale SISR is a more attractive and challenging task that
is still under exploration.

|
---|---
|
|  |

(a)

(b)

(c)

(d)

(e)

(f)

Figure 1: $\times 4$ SR results of methods using the integrated positional
encoding (IPE) compared with those without IPE.

Recently, implicit neural representations have emerged to represent various
kinds of signals. The key idea is to represent a signal as a function
parameterized with a multi-layer perceptron (MLP) that maps a coordinate input
to the signal value at that position. What makes the neural implicit
representation appealing is that a signal is often represented continuously or
in infinite resolution and contains sufficient detail. Different local signals
could form various kinds of entities on the whole, such as density and
radiance field for 3D scenes [30, 3], signed distance field for 3D shapes [27,
10, 7, 31, 8] and RGB values for images/videos [27, 36, 40].

Based on implicit neural representations, Meta-SR[13] achieved arbitrary-scale
image SR by generating variant weights of the up-scaling filters for a
position and scale input. Chen  _et al_. [6] proposed the local implicit image
function (LIIF), for continuously representing natural and complex images.
LIIF learns an implicit decoder across images to predict the color within the
continuous neighborhood of a feature vector in a 2D feature map. Therefore,
LIIF builds a bridge between the discrete and continuous representation for 2D
images. Both Meta-SR[13] and LIIF[6] exploit the convenience of implicit
representation.

However, MLPs in implicit representation have difficulty in learning high-
frequency functions, which is also known as the spectrum bias [32]. The bias
hinders the generation of detail-rich contents in some scenarios, such as
image SR and novel view synthesis. NeRF [30], a novel framework for novel view
synthesis, adopts positional encoding (PE) to address this issue. PE allows
MLPs to spatially distinguish inputs with small differences to recover high-
frequency details. But in NeRF’s rendering, each pixel emits only one ray to
predict its color, which may lead to observable aliasing artifacts. To conform
as much as possible to the physical aggregation of light, an approximate
integral of PE along the ray cone is used in [3] to make full use of multi-
scale training images and improve the rendering efficiency of anti-aliased
views.

In SISR, the same problem exists. Imagine that we have an HR image and a
corresponding LR image of half the size, where the pixels in the LR image are
not necessarily the same as the pixels at even positions in the HR image. To
address the problem, LIIF[6] directly inputs the pixel size to the implicit
decoder to distinguish them, called cell decoding. However, the direct input
of coordinates and size makes it difficult for the implicit decoder to fit
high-frequency details. Although UltraSR [46] has noticed the problem and
integrates PE to improve the SR performance, it still does not consider multi-
scale variations simultaneously, which is critical in tasks with the output of
different pixel sizes. We believe that each pixel in the SR task should be
treated as the aggregation of colors over a small area, instead of being
represented exactly by its center location. Both pixel size and position are
critical for high-quality arbitrary-scale SR. For this purpose, we propose
IPE, which is to compute the integral of PE over the pixel to be predicted. We
then present IPE-LIIF, where the IPE instead of the coordinate input is
concatenated with the local feature vector as the input of the implicit
decoder. Experiments reveal that IPE-LIIF achieves state-of-the-art
performance on several popular SR datasets, and multiple methods adopting IPE
also give consistently better results than the original version when the SR
scales varies over a wider range. The contributions of this paper are
threefold:

  * •

We propose integrated positional encoding (IPE), which can provide scale and
position-dependent encoding for implicitly represented pixels in image SR.

  * •

The arbitrary scale SR model incorporating IPE has demonstrated its optimal
competence in both qualitative and quantitative experiments.

  * •

Further experiments highlight that IPE can be applied to a variety of SR
models based on implicit representations, consistently improving their
capacity to learn multi-scale outputs.

##  2 Related Work

In this section, we will review the SISR task and the implicit representation
with a focus on positional encoding (PE).

Single image super-resolution SISR is one of the most popular tasks in the
visual community[41, 5, 9, 35, 51, 11, 19, 39, 42, 48]. Traditional
methods[41, 5] are mainly example-based. SRCNN[9] is the first end-to-end CNN-
based SR model. It implements the process of feature extraction, nonlinear
mapping, and image reconstruction with a linear three-layer structure.
Recursive strategies[20, 38] were proposed to improve the training stability
and detail fidelity. VDSR[19] and IRCNN[49] made the network deeper by
stacking convolutional layers with residual learning. For the first time,
ESPCNN[35] proposed to compute on low-resolution inputs before the up-sampling
layer, significantly reducing computational overhead. SRResNet[23] introduced
the residual connection[12] into image SR and generated perceptually sound
details via adversarial training. EDSR[24] improved the ResNet block by
removing the BatchNorm[17] layer. Moreover, it proposed to learn
simultaneously the backbone corresponding to various SR scales, forming the
fusion model MDSR[24], which allowed multi-scale SR. Similarly, LapSR[22]
proposed a progressive up-scaling architecture and a novel hierarchical
reconstruction loss to complete SR with scales for powers of 2. With the
observation that the dense connection[14] allows feature reuse from all the
preceding layers within each block, RDN[52] adopted stacked dense blocks to
make a full use of hierarchical features extracted from LR image, thereby
achieving relatively-high performance.

Arbitrary scale super-resolution Image SR on an arbitrary scale factor is
obviously more attractive and convenient than that on a pre-defined one. Hu
_et al_. [13] proposed to learn a meta-upscale module to dynamically predict
the weights of the up-scaling filters on different scales. Meta-SR[13] model
could combine the meta-upscale module with the feature learning module from
most previous methods[52, 24, 51], producing convincing SR results on
arbitrary scale. LIIF [6] replaced the meta-upscale module with an implicit
decoder, representing natural and complex images in a continuous manner. And
it is proved to be effective for image SR at out-of-distribution scales,
yielding visually smoother results than Meta-SR[13].

Coordinate-based neural representation Neural implicit representations can
faithfully reconstruct various signals and have therefore received extensive
attention and research[29, 31, 8, 30]. An object in implicit representation
can be typically modeled as a multi-layer perceptron (MLP), mapping a
coordinate input to the signal at that 2D/3D location. Such an idea has shown
effectiveness in a variety of scenarios, _e.g_. 3D shape modeling[8, 10, 7,
2], neural volume rendering[30, 3, 25], 3D scene modeling[18, 37] and
image/video representation[36, 6]. A recent work[32] has highlighted that
there exists a spectral bias in the MLP activated by ReLU, which leads to lack
of details in high-frequency signal representation. This problem can be solved
by either replacing ReLU with periodic functions[36] or adding an extra PE of
the coordinate input[30, 3, 25, 40, 46]. The use of proper PE can
significantly accelerate the training process and improve the fidelity of
signals at high-frequency positions. UltraSR[46] adds spatial encoding into
LIIF[6], reducing checkboard artifacts on the large scale SR. Inspired by the
pre-computed multi-scale anti-aliasing technique (_e.g_. mipmap[45]), Mip-
NeRF[3] accelerated the anti-alias novel view rendering in NeRF with an
approximate integral of PE within the interest region.

The method we propose, named IPE, provides position and shape information in a
uniform encoding for each pixel. IPE-LIIF applys IPE to LIIF, achieving the
state-of-the-art performance in the arbitrary-scale SR task. We verify the
effectiveness and generality of our design with extensive experiments and
convincing ablation studies.

s

Figure 2: Illustrations of different spatial encodings on this $3\times 3$
feature map: (a) no encoding, (b), cell decoding used in [6], (c) plain PE
used in[30, 46], (d) IPE (ours). $\mathbf{x}_{q}$ denotes the center of the
query pixel. The query pixel is marked with a red box in (b) and (d), with the
radius $(r_{W},r_{H})$. The differences in the expressions are marked in
green.

##  3 Preliminaries: local implicit image function

LIIF[6] performs the arbitrary-scale image SR by learning a continuous
representation for images. In continuous representation, each image $I$ is
represented as a 2D feature map $M\in\mathbb{R}^{H\times W\times C}$, where
the size is $H\times W$ and the channel is of length $C$. To predict the RGB
value at an arbitrary position, LIIF uses a learnable implicit function, which
takes the coordinate and the nearest feature as the input and predicts the RGB
value. The implicit function is parameterized by an MLP $f_{\theta}$(with
$\theta$ as its parameter) and has the following form:

$s=f_{\theta}(z,\mathbf{x})$ (1)

where $\mathbf{x}$ is a continuous 2D coordinate in the range
$\chi=[0,2W]\times[0,2H]$ and $z$ is a latent vector representing the local
feature. The output $s$ is the predicted color at position $\mathbf{x}$. With
a pre-defined $f_{\theta}$, $f_{\theta}(z,\cdot)$ becomes a local implicit
function representing the local image area.

In practice, the $H\times W$ features are evenly distributed in $\chi$. The
RGB value of the continuous image $I$ at a query coordinate $\mathbf{x}_{q}$
is

$I(\mathbf{x}_{q})=\hat{f}_{\theta}(z^{*},\mathbf{x}_{q}-p(z^{*}))$ (2)

where $z^{*}$ is the nearest feature vector to position $\mathbf{x}_{q}$ in
$M$, $p(z)$ is the position of $z$. Note that the implicit function
$\hat{f}_{\theta}$ here describes the location relatively, which is different
from $f_{\theta}$ defined in Eq. 1 with absolute coordinate input . In Eq. 2,
the relative coordinate $\delta\mathbf{x}=\mathbf{x}_{q}-p(z^{*})$ is in the
range of $[-1,1]^{2}$. Each latent code $z$ in $M$ is responsible for
predicting the color in such a local area around $p(z)$. With the shared
implicit decoder, LIIF bridges the gap between features at discrete positions
and a continuous image.

With this procedure for rendering the continuous image at the arbitrary
resolution, training LIIF is straightforward: sampling pixels from HR images
to build coordinate-color pairs, we minimize the difference between the
predicted RGB values and the ground truths with gradient descent. Since LIIF
works as a decoder that converts the feature map into an image, it is usually
jointly trained with a feature extraction module in CNN models[24, 52].

##  4 Method

In this section, we first discuss the shortcomings of LIIF[6], propose the
solution, and finally present the overall architecture of our model. As
described in Sec. 3, LIIF[6] samples only the center to render each pixel.
Essentially, LIIF ignores the fact that an image pixel is actually the
aggregation of the color in a small area, and considers the color value at the
central point as the color of that pixel. Imagine that we have an HR image and
a corresponding LR image down-sampled by 2, where pixels in the LR image are
not necessarily the same as those at even positions in HR. Therefore the
scaling factor or, in other words, the size of the target pixel is crucial in
this task. And to address the problem, the original LIIF[6] simply
concatenates the pixel size and the central position of pixel area as input to
the MLP, called cell decoding. But both our and LIIF’s experiments reveal that
this cell decoding strategy is not guaranteed to improve the quality of
results.

Figure 3: Comparison between the sine encoding of IPE (ours) and that of plain
PE with the frequency $\omega=8$.

###  4.1 Integrated Positional Encoding

We solve the problem by introducing IPE of each pixel, like [3], enabling the
implicit decoder to focus on the local region rather than a center point
without area. Moreover, with IPE, the implicit decoder could adapt to a
variety of SR scales, which are essentially related to the pixel size to
decode.

The commonly used PE[30, 40] transforms each position with periodic function
in different frequencies:

$\gamma(\mathbf{x})=[\sin(\mathbf{x}),\cos(\mathbf{x}),...,\sin(2^{L-1}\mathbf{x}),\cos(2^{L-1}\mathbf{x})]$
(3)

Similar with [3], we construct the featurized representation of the pixel with
the expected PE of all the coordinates that lie in it. The main differences
among several spatial encoding methods are illustrated in Fig. 2. In [3], the
integral of PE in a conical frustum is quite tricky and has no closed form
solution, so they use a multivariate Gaussian with the same mean and variance
to approximate the area. In the 2D case we are discussing here, the integral
is straightforward: within a pixel $l$ whose center is
$\boldsymbol{c}(c_{x},c_{y})$ and radius (half of the edge) is
$\boldsymbol{r}(r_{W},r_{H})$, the sine encoding with frequency $\omega$ has
an expectation as:

$\displaystyle\mathbb{E}_{\mathbf{x}\in l}[\sin(\omega\mathbf{x})]$
$\displaystyle=\frac{1}{4r_{W}r_{H}}\iint_{l}\sin(\omega\mathbf{x})dxdy$ (4)
$\displaystyle=\frac{\sin(\omega\boldsymbol{c})\sin(\omega\boldsymbol{r})}{\omega\boldsymbol{r}}$
$\displaystyle=\sin(\omega\boldsymbol{c})\operatorname{sinc}(\omega\boldsymbol{r})$

Similarly, the cosine encoding is as follows:

$\displaystyle\mathbb{E}_{\mathbf{x}\in l}[\cos(\omega\mathbf{x})]$
$\displaystyle=\frac{1}{4r_{W}r_{H}}\iint_{l}\cos(\omega\mathbf{x})dxdy$ (5)
$\displaystyle=\cos(\omega\boldsymbol{c})\operatorname{sinc}(\omega\boldsymbol{r})$

Hence, the pixel $l$ defined by its center $\boldsymbol{c}$ and radius
$\boldsymbol{r}$ has the following IPE:

$\displaystyle\hat{\gamma}(\boldsymbol{c},\boldsymbol{r})=\frac{1}{\boldsymbol{r}}[\sin(\boldsymbol{c})\sin(\boldsymbol{r}),\cos(\boldsymbol{c})\sin(\boldsymbol{r}),...,$
(6)
$\displaystyle\frac{\sin(2^{L-1}\boldsymbol{c})\sin(2^{L-1}\boldsymbol{r})}{2^{L-1}},\frac{\cos(2^{L-1}\boldsymbol{c})\sin(2^{L-1}\boldsymbol{r})}{2^{L-1}}]$

Intuitively, the IPE contains both location and size information, which are
modulated by sine/cosine and sinc function, respectively. When the SR scale is
small, the sinc part makes plain PE smoother, in which case the predicted
pixel can inherit enough information from the feature vector. As the SR scale
increases, the sinc part will move towards 1, making the IPE closer to plain
PE (see Fig. 3 for brief comparison). In that case, each feature vector is
responsible for a large number of pixels, which is roughly the square of the
scale. Therefore, the implicit decoder cannot reconstruct a detail-rich image
solely from the features. The IPE will provide high-frequency encoding of the
location to compensate for the lack of information here. The choice of hyper-
parameter $L$ will further be discussed in Sec. 5.3.

Figure 4: Local ensemble in IPE-LIIF. The query pixel along with its center is
marked in red. The RGB value of the query pixel is obtained by bilinear
interpolation of the results predicted by its surrounding feature vectors.

Method | In-distribution | Out-of-distribution
---|---|---
$\times 2$ | $\times 3$ | $\times 4$ | $\times 6$ | $\times 12$ | $\times 18$ | $\times 24$ | $\times 30$
Bicubic[24] | 31.01/0.9393 | 28.22/0.8906 | 26.66/0.8521 | 24.82/0.8014 | 22.27/0.7354 | 20.50/0.6986 | 19.79/0.6817 | 19.43/0.6711
EDSR-baseline[24] | 34.55/0.9671 | 30.90/0.9298 | 28.94/0.8962 | - | - | - | - | -
EDSR-baseline-MetaSR[13] | 34.64/0.9674 | 30.93/0.9301 | 28.92/0.8962 | 26.61/0.8442 | 23.55/0.7652 | 22.03/0.7285 | 21.06/0.7058 | 20.37/0.6903
EDSR-baseline-LIIF[6] | 34.67/0.9675 | 30.96/0.9306 | 29.00/0.8974 | 26.75/0.8477 | 23.71/0.7724 | 22.17/0.7369 | 21.18/0.7137 | 20.48/0.6977
EDSR-baseline-IPE-LIIF (ours) |  34.72/0.9678 |  31.01/0.9310 |  29.04/0.8979 |  26.79/0.8483 |  23.75/0.7731 |  22.21/0.7374 |  21.22/0.7143 |  20.51/0.6979
RDN-MetaSR[13] | 35.00/0.9692 | 31.27/0.9340 | 29.25/0.9016 | 26.88/0.8508 | 23.73/0.7721 | 22.18/0.7337 | 21.17/0.7104 | 20.47/0.6943
RDN-LIIF[6] | 34.99/0.9691 | 31.26/0.9339 | 29.27/0.9017 | 26.99/0.8528 | 23.89/0.7771 | 22.34/0.7412 | 21.31/0.7181 | 20.59/0.7013
RDN-IPE-LIIF (ours) |  35.04/0.9694 |  31.32/0.9343 |  29.32/0.9025 |  27.04/0.8537 |  23.93/0.7778 |  22.38/0.7415 |  21.34/0.7177 |  20.63/0.7011

Table 1: Quantitative comparison on DIV2K[1] validation set with PSNR (dB) /
SSIM on different scales. Bicubic, MetaSR[13], LIIF[6] uses a uniform
model/algorithm for all the scales. EDSR[24] uses different models for
different scales and cannot be evaluated on out-of-distribution scales. The
best results under different settings are bolded.

###  4.2 Architecture details

Given a pixel-based image $I\in\mathbb{R}^{H\times W\times 3}$, we extract the
2D feature map $M\in\mathbb{R}^{H\times W\times C}$ with a feature extraction
module $\mathcal{E}$. The module can inherit from EDSR-baseline[24] and RDN
[52]. Feature unfolding in [6] is preserved. Each latent vector in the new
feature map $\hat{M}\in\mathbb{R}^{H\times W\times 9C}$ is the concatenation
of the $3\times 3$ local neighboring vectors from zero-padded $M$. The feature
extraction process is formulated as:

$\displaystyle M$ $\displaystyle=\mathcal{E}(I)$ (7)
$\displaystyle\hat{M}_{ij}$
$\displaystyle=Concat(\\{M_{i+k,j+m},k,m\in\\{-1,0,1\\}\\})$

To make the predicted RGB value smoother, we apply the local ensemble as [6].
That is, given a SR scale $s$ and a query coordinate $\mathbf{x}_{q}$, we
extend Eq. 2 to

$I_{s}(\mathbf{x}_{q})=\sum_{t\in\\{00,01,10,11\\}}\frac{S_{t}}{S}\cdot\hat{f}_{\theta}(z_{t}^{*},\hat{\gamma}(\mathbf{x}_{q}-p(z_{t}^{*}),(\frac{1}{s},\frac{1}{s})))$
(8)

where $\\{z_{t}^{*}\\}$ are four feature vectors surrounding $\mathbf{x}_{q}$
in $\hat{M}$, and they are on the vertices of a square. $S$ is the area of
this square and $S_{t}$ is the area of the rectangle inside this square that
is opposite to $p(z_{t}^{*})$ (Fig. 4). The query pixel radius is
$(\frac{1}{s},\frac{1}{s})$ here if we specify the range of the $xy$-axis of
the per-pixel local coordinate system in the original input LR image to
$[-1,1]$.

The implicit decoder $\hat{f}_{\theta}$ is implemented as an MLP with skip
connections[30, 31] from the input to all hidden layers. The MLP has 4 hidden
layers, each with a width of 256. The bandwidth parameter $L$ of IPE is set to
10. To fuse the spatial encoding $\hat{\gamma}(\boldsymbol{c},\boldsymbol{r})$
and the feature vector $z^{*}$, we concatenate them as the input of
$\hat{f}_{\theta}$ (Fig. 2 (d)).

Dataset | Method | In-distribution | Out-of-distribution
---|---|---|---
$\times 2$ | $\times 3$ | $\times 4$ | $\times 6$ | $\times 8$
Set5[4] | RDN[52] |  38.24/0.9614 |  34.71/0.9296 | 32.47/0.8990 | - | -
RDN-MetaSR[13] | 38.10/0.9603 | 34.62/0.9279 | 32.37/0.8959 | 28.98/0.8262 | 26.95/0.7641
RDN-LIIF[6] | 38.07/0.9601 | 34.63/0.9280 | 32.47/0.8969 | 29.13/0.8327 | 27.12/0.7766
RDN-IPE-LIIF (ours) | 38.11/0.9604 | 34.68/0.9285 |  32.51/0.8975 |  29.25/0.8344 |  27.22/0.7797
Set14[47] | RDN[52] |  34.01/0.9212 |  30.57/0.8468 |  28.81/0.7871 | - | -
RDN-MetaSR[13] | 33.81/0.9194 | 30.43/0.8453 | 28.69/0.7858 | 26.43/0.6975 | 24.87/0.6370
RDN-LIIF[6] | 33.83/0.9196 | 30.41/0.8456 | 28.70/0.7857 | 26.54/0.7008 | 25.05/0.6446
RDN-IPE-LIIF (ours) | 33.94/0.9201 | 30.47/0.8456 | 28.75/0.7870 |  26.58/0.7021 |  25.09/0.6462
B100[28] | RDN[52] |  32.34/0.9017 | 29.26/0.8093 | 27.72/0.7419 | - | -
RDN-MetaSR[13] | 32.29/0.9008 | 29.24/0.8091 | 27.71/0.7414 | 25.89/0.6522 | 24.82/0.5977
RDN-LIIF[6] | 32.28/0.9004 | 29.25/0.8095 | 27.73/0.7420 | 25.97/0.6555 | 24.91/0.6033
RDN-IPE-LIIF (ours) | 32.31/0.9010 |  29.28/0.8098 |  27.76/0.7429 |  26.00/0.6563 |  24.93/0.6038
Urban100[15] | RDN[52] | 32.89/0.9353 | 28.80/0.8653 | 26.61/0.8028 | - | -
RDN-MetaSR[13] | 32.88/0.9349 | 28.85/0.8660 | 26.58/0.8020 | 24.00/0.6943 | 22.61/0.6205
RDN-LIIF[6] | 32.83/0.9344 | 28.80/0.8656 | 26.67/0.8037 | 24.20/0.7033 | 22.79/0.6348
RDN-IPE-LIIF (ours) |  32.97/0.9357 |  28.92/0.8675 |  26.76/0.8062 |  24.26/0.7061 |  22.87/0.6374

Table 2: Quantitative comparison on 4 benchmarks[4, 47, 28, 15] with PSNR (dB)
/ SSIM on different scales. LIIF[6], IPE-LIIF (ours) use a uniform model for
all the scales. RDN[52] uses different models for different scales and cannot
be evaluated on out-of-distribution scales. The best results on different
settings are bolded.

##  5 Experiments

In this section, we introduce the datasets and metrics in the experiments, the
implementation details, and compared methods. Then we compare IPE-LIIF with
some baseline alternatives both quantitatively and qualitatively. We further
design experiments to discuss the generalization ability of IPE and validate
some of the design choices via ablation studies.

Datasets and metrics We train all models using the DIV2K dataset [1] from
NTIRE 2017 Challenge for all the experiments. This dataset consists of 1,000
HR images in 2K resolution and their corresponding LR images of different
down-sampling scales ($\times 2-\times 4$). We follow the original split
setting, _i.e_. 800 images for training, 100 images for validation. Consistent
with previous works[6, 13, 52, 24], we also test our model on 4 popular
benchmarks, _i.e_. Set5[4], Set14[47], B100[28], and Urban100[15], and report
the results. We compare our method with baseline methods on several discrete
scales ($\times 2,\times 3,\times 4$). To demonstrate that our method can be
applied for arbitrary-scale SR, we also compare the SR results on a variety of
large scales ($\times 6-\times 30$) that do not appear in training time. We
use the Peak Signal-to-Noise Ratio (PSNR) and SSIM[44] as our evaluation
metrics. Consistent with most SR works, the metrics are computed on RGB
channels for DIV2K[1] and Y channel (luminance)for other benchmarks[4, 47, 28,
15].

Training and inference details During training, we feed patches of size
$48\times 48$ into the network. To enable the network to perform arbitrary-
scale SR on the input patches, we sample the SR scale $s$ from
$\mathcal{U}(1,4)$. We then randomly crop a patch of size $48s\times 48s$ from
the training HR image. The cropped HR patch is then down-sampled to $48\times
48$ with bi-cubic interpolation as the LR input. For the implicit decoder
training, we randomly select $48^{2}$ pixels from each HR patch. All the
modules are trained from scratch with L1 loss[24, 52] for better convergence.
We use Adam[21] optimizer with $\beta=(0.9,0.999)$ to train the whole network
for 1000 epochs. Each epoch contains 1000 iterations. In each iteration, we
feed a batch of 16 sampled patches. The learning rate is initialized to
${10}^{-4}$ and decays to half every 200 epochs. For evaluation on scales
within the distribution of training ($\times 2\sim\times 4)$, we use the LR
images provided in the DIV2K and different benchmark datasets. For out-of-
distribution validation ($\times 6\sim\times 30$), we generate the LR inputs
by resizing the HR images with bi-cubic interpolation.

Compared methods The methods involved in the experiments include: 1. models
with CNN architectures targeted for a specific scale: EDSR-baseline[24],
RDN[52], 2. the state-of-the-art arbitrary-scale SR methods at present: Meta-
SR[13], LIIF[6], 3. methods in group 2 that combines IPE: IPE-LIIF, IPE-
MetaSR. For the latter two groups, both the feature extraction modules in
EDSR-baseline and RDN could be adopted. In Sec. 5.1, we evaluate all the
models except IPE-MetaSR on benchmarks with various scales to verify the
effectiveness of IPE on improving performance for arbitrary-scale SR methods.
In Sec. 5.2, we evaluate LIIF, MetaSR and their IPE-version after training on
a wider range of SR scales to highlight the generality of IPE on multiple
methods and larger training scales.

###  5.1 Results and comparison

We compare IPE-LIIF with five baseline methods: bicubic interpolation, EDSR-
baseline[24], RDN[52], MetaSR[13] and LIIF[6]. Note that EDSR-baseline and RDN
train different models for different scales, thus cannot be applied to out-of-
distribution scales. The remaining parametric methods are designed for
arbitrary-scale image SR and their feature extraction modules are
exchangeable. The feature extraction module used by the three models is
indicated by the prefix of the name. All the parametric methods are trained on
the DIV2K[1] training set.

Quantitative comparison The quantitative results on the 5 benchmark
datasets[1, 5, 47, 28, 15] are listed in Tab. 1 and Tab. 2. We observe that
our method surpasses existing methods under almost all the test settings,
especially on large datasets[28, 15]. For SR on out-of-distribution scales, we
achieve consistently better results than existing methods on all the
benchmarks. In summary, the use of IPE enhances the existing SR model based on
implicit representation, allowing them to go beyond previous scale-specific
ones[52] to support multiple scale factors. We argue that the power of IPE-
LIIF comes from the combination of IPE and implicit representation, which
effectively fuse information from different scales and spatial locations.

Qualitative comparison In Fig. 1 and Fig. 5, we further show visual image
comparisons of the previous work and our method on different SR scales. For
areas containing scale variant stripes, the previous method[6] will produce
checkboard artifacts and could not faithfully recover the HR contents, while
IPE-LIIF can distinguish texture details by encoding the query pixel spatial
information. In these cases, IPE is able to better identify the position and
size of complex textures so that they can be clearly distinguished.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

Figure 5: Visual comparison of bicubic, LIIF, and IPE-LIIF on different SR
scales.

###  5.2 General enhancement for multi-scale implicit learning

Both Meta-SR[6] and LIIF[6] take the advantage of the implicit representation.
The former predicts an up-scaling filter for each output pixel and the latter
predicts the color directly with a coordinate input via implicit decoding. The
IPE strategy can also be easily applied to improve Meta-SR. That is, for a
given pixel $(i,j)$ on the SR image output up-scaled by $s$, we redefine the
original input to Meta-Upscale Module[13]:

$\mathbf{v}_{ij}=(\frac{i}{s}-\lfloor\frac{i}{s}\rfloor,\frac{j}{s}-\lfloor\frac{j}{s}\rfloor,\frac{1}{s})$
(9)

as

$\mathbf{v}_{ij}=\hat{\gamma}((\frac{i}{s}-\lfloor\frac{i}{s}\rfloor,\frac{j}{s}-\lfloor\frac{j}{s}\rfloor),(\frac{1}{s},\frac{1}{s}))$
(10)

and form IPE-MetaSR. To verify the effectiveness of IPE on a relatively large
range of scale variations, _e.g_. $\mathcal{U}(1,8),\mathcal{U}(1,16)$, we
train several candidate models and evaluate them on DIV2K[1] over the same
scale distribution. From Tab. 3, we can observe that with IPE, both models
obtain consistent improvements on different SR scales. By comparing the
results on large scales presented in Tab. 3 and Tab. 1, it can be seen that
LIIF fails to exceed the performance of our method without these training
samples even if the data on $\times 6-\times 12$ is provided. This experiment
illustrates the generality of IPE on multiple implicit models and a wider
range of SR scales.

Scale | Method | $\times 2$ | $\times 3$ | $\times 4$ | $\times 6$ | $\times 12$
---|---|---|---|---|---|---
$\mathcal{U}(1,8)$ | LIIF[6] | 34.60 | 30.94 | 29.01 | 26.79 | -
LIIF w/ PE | 34.47 | 30.87 | 29.01 | 26.81 | -
ipe-LIIF(ours) | 34.65 | 30.98 | 29.04 | 26.81 | -
MetaSR[13] | 34.57 | 30.91 | 28.97 | 26.72 | -
MetaSR w/ PE | 34.60 | 30.93 | 28.99 | 26.74 | -
ipe-MetaSR(ours) | 34.61 | 30.93 | 28.99 | 26.74 | -
$\mathcal{U}(1,16)$ | LIIF[6] | 34.50 | 30.89 | 28.97 | 26.76 | 23.75
LIIF w/ PE | 34.36 | 30.79 | 28.95 | 26.78 | 23.79
IPE-LIIF(ours) | 34.54 | 30.92 | 29.00 | 26.79 | 23.76
MetaSR[13] | 34.48 | 30.86 | 28.94 | 26.72 | 23.70
MetaSR w/ PE | 34.50 | 30.85 | 28.93 | 26.72 | 23.70
IPE-MetaSR(ours) | 34.51 | 30.86 | 28.94 | 26.74 | 23.71

Table 3: Validation results of LIIF[6] and Meta-SR[13] on DIV2K[1] combined
with EDSR-baseline and different coordinate encodings, including none, plain
positional encoding, and integrated positional encoding. The best results are
bolded.

###  5.3 Ablation studies

We discuss the effectiveness of some design choices of IPE-LIIF in this
section, including: 1\. the bandwidth $L$ of integrated positional encoding,
2\. whether to use the integrated version of PE, 3\. whether to use the
original cell decoding, 4\. whether to use skip connections in the implicit
decoder. EDSR-baseline is used as the feature extraction module for all the
ablation studies. The quantitative results are listed in Tab. 4 and green
indicates a decrease in results.

Bandwidth $L$ of integrated positional encoding In terms of bandwidth $L$, we
choose $L=10$ in IPE-LIIF and set $L=4,16$ for competitors. Comparing the two
variants $(L=4,16)$ with the original model, we observe that the effect of the
bandwidth is minor. Taking the results and computational overhead into
account, we choose a mild value $L=10$. The intuitive explanation of how a
large $L$ does not improve the SR quality is that the additional terms in IPE
(Eq. 6) have large denominators and remain almost constant near 0 as the query
pixel region changes. That is, a larger $L$ does not necessarily provide valid
spatial information, which is also highlighted in mip-NeRF[3].

Method | In-distribution | Out-of-distribution
---|---|---
$\times 2$ | $\times 3$ | $\times 4$ | $\times 6$ | $\times 18$ | $\times 30$
IPE-LIIF (ours) | 34.72 | 31.01 | 29.04 | 26.79 | 22.21 | 20.51
IPE-LIIF (L = 16) | 34.72 | 31.01 | 29.04 | 26.79 | 22.21 | 20.52
IPE-LIIF (L = 4) | 34.71 | 31.00 | 29.03 | 26.78 | 22.21 | 20.52
LIIF w/ PE (L = 10) | 34.59 | 30.92 | 29.00 | 26.76 | 22.21 | 20.52
IPE-LIIF + cell | 34.72 | 31.01 | 29.04 | 26.79 | 22.18 | 20.49
IPE-LIIF- skip | 34.67 | 30.97 | 29.01 | 26.76 | 22.19 | 20.50

Table 4: Ablation of design choices of IPE-LIIF on DIV2K[1] validation set.
EDSR-baseline is used as the feature extraction module for all the settings.
The results with degraded performance are marked in green.

Integrated positional encoding An obvious possible improvement on LIIF[6] is
to add plain PE to the coordinate input. To verify the effectiveness of IPE,
we perform plain PE on the input and form LIIF with PE. For a fair comparison,
we also set the bandwidth $L$ to 10. It can be seen that with PE, the network
focuses more on high-frequency information and performs poorly on relatively
small scale factors, sometimes even worse than the original LIIF.

Other design choices In Tab. 4, we show the ablation results on some other
design choices, including the use of cell decoding and skip connections in the
implicit decoder. We concatenate the cell size with the IPE input and form
IPE-LIIF + cell. It can be seen that cell decoding makes the results on large
scaling factors worse, which is because it is not able to generalize to
arbitrary scale factor. For the out-of-distribution scales, the cell code is
relatively small compared with those seen in training time. We also remove
skip connections and form IPE-LIIF- skip. The results verify that SR on almost
all scales benefits from the skip connections, as it provides a more flexible
network depth.

From Tab. 4, we find that IPE performs good aggregation of the spatial
information in the query region. It combines the advantages of PE and cell
decoding and provides a universal solution to implicit learning tasks with
multi-scale samples.

##  6 Limitations

Although we have improved the existing arbitrary-scale image SR methods with
IPE, regression-based methods still cannot fill in the missing details when
the scale factor is large. In contrast, generative methods[33, 23, 43] could
induct reasonable semantic fill-in and produce visually clear results with a
poor PSNR but high perceptual similarity[50]. In addition, in IPE-LIIF, the
wider layers in skip-MLP will increase the training time by 10% to 20%.

##  7 Conclusion

In this paper, we propose to use IPE for arbitrary-scale image SR, and apply
it to the state-of-the-art method LIIF. With the enhancement provided by IPE,
IPE-LIIF generates consistently high-fidelity SR results on various scales,
which is verified by quantitative and qualitative experiments. Further
experiments on the generalization ability suggest that IPE can be seamlessly
integrated to multiple architectures to increase the capacity of implicit
learning on multi-scale samples.

## References

  * [1] Eirikur Agustsson and Radu Timofte.  NTIRE 2017 challenge on single image super-resolution: Dataset and study.  In CVPR Workshops, pages 1122–1131. IEEE Computer Society, 2017\.
  * [2] Matan Atzmon and Yaron Lipman.  SAL: sign agnostic learning of shapes from raw data.  In CVPR, pages 2562–2571. Computer Vision Foundation / IEEE, 2020.
  * [3] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  ICCV, 2021.
  * [4] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  In BMVC, pages 1–10. BMVA Press, 2012.
  * [5] Hong Chang, Dit-Yan Yeung, and Yimin Xiong.  Super-resolution through neighbor embedding.  In CVPR (1), pages 275–282. IEEE Computer Society, 2004.
  * [6] Yinbo Chen, Sifei Liu, and Xiaolong Wang.  Learning continuous image representation with local implicit image function.  In CVPR, pages 8628–8638. Computer Vision Foundation / IEEE, 2021.
  * [7] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang.  Bsp-net: Generating compact meshes via binary space partitioning.  In CVPR, pages 42–51. Computer Vision Foundation / IEEE, 2020\.
  * [8] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In CVPR, pages 5939–5948. Computer Vision Foundation / IEEE, 2019.
  * [9] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  CoRR, abs/1501.00092, 2015.
  * [10] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas A. Funkhouser.  Learning shape templates with structured implicit functions.  In ICCV, pages 7153–7163. IEEE, 2019.
  * [11] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita.  Deep back-projection networks for super-resolution.  In CVPR, pages 1664–1673. Computer Vision Foundation / IEEE Computer Society, 2018.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778. IEEE Computer Society, 2016.
  * [13] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun.  Meta-sr: A magnification-arbitrary network for super-resolution.  In CVPR, pages 1575–1584. Computer Vision Foundation / IEEE, 2019.
  * [14] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In CVPR, pages 2261–2269. IEEE Computer Society, 2017.
  * [15] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.  Single image super-resolution from transformed self-exemplars.  In CVPR, pages 5197–5206. IEEE Computer Society, 2015.
  * [16] Zheng Hui, Jie Li, Xinbo Gao, and Xiumei Wang.  Progressive perception-oriented network for single image super-resolution.  Inf. Sci., 546:769–786, 2021.
  * [17] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 448–456. JMLR.org, 2015.
  * [18] Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas A. Funkhouser.  Local implicit grid representations for 3d scenes.  In CVPR, pages 6000–6009. Computer Vision Foundation / IEEE, 2020.
  * [19] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In CVPR, pages 1646–1654. IEEE Computer Society, 2016.
  * [20] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Deeply-recursive convolutional network for image super-resolution.  In CVPR, pages 1637–1645. IEEE Computer Society, 2016.
  * [21] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR (Poster), 2015.
  * [22] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.  Deep laplacian pyramid networks for fast and accurate super-resolution.  In CVPR, pages 5835–5843. IEEE Computer Society, 2017.
  * [23] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In CVPR, pages 105–114. IEEE Computer Society, 2017.
  * [24] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In CVPR Workshops, pages 1132–1140. IEEE Computer Society, 2017\.
  * [25] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  In NeurIPS, 2020.
  * [26] Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou.  Structure-preserving super resolution with gradient guidance.  In CVPR, pages 7766–7775. Computer Vision Foundation / IEEE, 2020.
  * [27] Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein.  Acorn: adaptive coordinate networks for neural scene representation.  ACM Trans. Graph., 40(4):58:1–58:13, 2021.
  * [28] David R. Martin, Charless C. Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In ICCV, pages 416–425. IEEE Computer Society, 2001.
  * [29] Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, pages 4460–4470. Computer Vision Foundation / IEEE, 2019.
  * [30] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV (1), volume 12346 of Lecture Notes in Computer Science, pages 405–421. Springer, 2020.
  * [31] Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, pages 165–174. Computer Vision Foundation / IEEE, 2019\.
  * [32] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron C. Courville.  On the spectral bias of neural networks.  In ICML, volume 97 of Proceedings of Machine Learning Research, pages 5301–5310. PMLR, 2019.
  * [33] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  CoRR, abs/2104.07636, 2021.
  * [34] Mehdi S. M. Sajjadi, Bernhard Schölkopf, and Michael Hirsch.  Enhancenet: Single image super-resolution through automated texture synthesis.  In ICCV, pages 4501–4510. IEEE Computer Society, 2017.
  * [35] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In CVPR, pages 1874–1883. IEEE Computer Society, 2016.
  * [36] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  In NeurIPS, 2020.
  * [37] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In NeurIPS, pages 1119–1130, 2019.
  * [38] Ying Tai, Jian Yang, and Xiaoming Liu.  Image super-resolution via deep recursive residual network.  In CVPR, pages 2790–2798. IEEE Computer Society, 2017.
  * [39] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu.  Memnet: A persistent memory network for image restoration.  In ICCV, pages 4549–4557. IEEE Computer Society, 2017.
  * [40] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  In NeurIPS, 2020.
  * [41] Radu Timofte, Vincent De Smet, and Luc Van Gool.  Anchored neighborhood regression for fast example-based super-resolution.  In ICCV, pages 1920–1927. IEEE Computer Society, 2013.
  * [42] Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao.  Image super-resolution using dense skip connections.  In ICCV, pages 4809–4817. IEEE Computer Society, 2017.
  * [43] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  ESRGAN: enhanced super-resolution generative adversarial networks.  In ECCV Workshops (5), volume 11133 of Lecture Notes in Computer Science, pages 63–79. Springer, 2018.
  * [44] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [45] Lance Williams.  Pyramidal parametrics.  In SIGGRAPH, pages 1–11. ACM, 1983.
  * [46] Xingqian Xu, Zhangyang Wang, and Humphrey Shi.  Ultrasr: Spatial encoding is a missing key for implicit image function-based arbitrary-scale super-resolution.  CoRR, abs/2103.12716, 2021.
  * [47] Roman Zeyde, Michael Elad, and Matan Protter.  On single image scale-up using sparse-representations.  In Curves and Surfaces, volume 6920 of Lecture Notes in Computer Science, pages 711–730. Springer, 2010.
  * [48] Kai Zhang, Luc Van Gool, and Radu Timofte.  Deep unfolding network for image super-resolution.  In CVPR, pages 3214–3223. Computer Vision Foundation / IEEE, 2020.
  * [49] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep CNN denoiser prior for image restoration.  In CVPR, pages 2808–2817. IEEE Computer Society, 2017.
  * [50] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, pages 586–595. Computer Vision Foundation / IEEE Computer Society, 2018.
  * [51] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In ECCV (7), volume 11211 of Lecture Notes in Computer Science, pages 294–310. Springer, 2018.
  * [52] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu.  Residual dense network for image super-resolution.  In CVPR, pages 2472–2481. Computer Vision Foundation / IEEE Computer Society, 2018.