Date of publication June 16, 2020, date of current version June 29, 2020.
10.1109/ACCESS.2020.3002921

This work was supported in part by the Samsung Research Funding Center of
Samsung Electronics under Project SRFC-IT1702-06, in part by the Institute of
Information and Communications Technology Planning and Evaluation (IITP)
funded by the Korea Government (MSIT) (Development of Acceleration SW Platform
Technology for On-device Intelligent Information Processing in Smart Devices)
under Grant 2017-0-00142, and in part by the Artificial Intelligence
Convergence Research Center (Inha University) under Grant 2020-0-01389.

Corresponding author: In Kyu Park (e-mail: pik@inha.ac.kr).

# Joint Spatial and Angular Super-Resolution from a Single Image

ANDRE IVAN1  WILLIEM2  and IN KYU PARK1  Department of Information and
Communication Engineering, Inha University, Incheon 22212, Korea (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS> Verihubs, Indonesia (e-mail:
<EMAIL_ADDRESS>
###### Abstract

Synthesizing a densely sampled light field from a single image is highly
beneficial for many applications. Moreover, jointly solving both angular and
spatial super-resolution problem also introduces new possibilities in light
field imaging. The conventional method relies on physical-based rendering and
a secondary network to solve the angular super-resolution problem. In
addition, pixel-based loss limits the network capability to infer scene
geometry globally. In this paper, we show that both super-resolution problems
can be solved jointly from a single image by proposing a single end-to-end
deep neural network that does not require a physical-based approach. Two novel
loss functions based on known light field domain knowledge are proposed to
enable the network to consider the relation between sub-aperture images.
Experimental results show that the proposed model successfully synthesizes
dense high resolution light field and it outperforms the state-of-the-art
method in both quantitative and qualitative criteria. The method can be
generalized to various scenes, rather than focusing on a particular subject.
The synthesized light field can be used as if it has been captured by a light
field camera, such as depth estimation and refocusing.

###### Index Terms:

Deep neural network, Light field, Machine learning, Super-resolution

=-15pt

##  I Introduction

Light fields have attracted considerable interest from computer vision and
graphic communities due to their capability to capture multiple light rays
from various directions. Recent studies utilized densely sampled light field
captured by off-the-shelf light field cameras. Many applications, such as
depth estimation [15, 16, 25, 26], refocusing [13], and 3D reconstruction [8,
21], exploit the rich information of a light field image.

(a)

(b)

(c)

Figure 1: Preview of the proposed method. (a) Patches of spatial SR results
(Input, 2$\times$, and 4$\times$). (b) Estimated appearance flow and depth
image. (c) Refocused image at foreground and background.

At present, a light field image is captured using either plenoptic (light
field) cameras [1] or camera arrays [24]. However, the absence of the only
available consumer light field camera, i.e. Lytro, has created a gap between
consumers and light field experiences. In addition, light field camera also
suffers from spatial and angular resolution trade-off due to the sensor
limitation. Most light field camera favor angular resolution which results in
a low spatial resolution light field image. We focus on filling this gap so
that end users can experience the advantages of light field imaging and
beyond. The idea is to jointly synthesize light field through angular and
spatial super-resolution (SR) from only a single image which is abundantly
available in the real world. In order to do that, the geometry information
from a single image should be inferred and used to synthesize the surrounding
angular images. Synthesizing a 4D light field is a severely ill-posed problem,
but the impact of such work is considerably significant. For example,
promoting a single image into a densely sampled light field can elevate
existing AR/VR immersion experiences.

In this context, light field synthesis has attracted considerable attention in
recent years [9, 12, 17, 19, 22, 28, 30]. Previous approaches can be grouped
into two categories based on the input type, i.e. single or multi-view inputs.
The multi-view input utilizes multiple images captured from specific
viewpoints to infer the geometric clue and use it to synthesize the light
field. However, only a few consumer cameras can simultaneously capture multi-
view images, which makes the approach impractical for general use.

Existing method involving a single input utilizes two-stage neural networks
and depth image-based rendering (DIBR) technique to synthesize the light field
[17]. [17] is inspired by previous view synthesis techniques using geometry
estimation [4, 5, 6, 29]. However, [17] is highly dependent on the estimated
depth quality and physical-based depth warping to synthesize angular images.
The depth-based approach also faces difficulty in reconstructing the occlusion
and homogeneous region. Typical learning based works rely on minimizing the
error between the synthesized view and the ground truth image
straightforwardly. This leads the network to rely on pixel intensity cue and
cannot be easily generalized to data with different and complex distribution.

In this paper, we develop a novel joint deep neural network for spatial and
angular light field SR that utilizes the appearance flow to synthesize novel
views. To the best of our knowledge, this is the only work that tackle the
joint super-resolution problem using only a single image. We also introduce a
spatio-angular consistent loss function based on known light field domain
knowledge that helps the network learn robustly and efficiently. Figure 1
shows the result and application of the proposed method.

The key contributions of this paper are summarized as follows.

  * •

End-to-end encoder-decoder style for joint spatial and angular light field SR
model.

  * •

Novel spatio-angular consistent (light field based) loss that imposes
geometric reasoning to the network.

  * •

Capability to be generalized to arbitrary scenes rather than a specific class
of object compared with the previous approach.

Figure 2: The structure of the proposed joint framework. Dashed green and blue
lines denote the angular decoder and spatial decoder, respectively. Red dashed
line denotes channel wise concatenation operation.

##  II Related Works

Learning based light field synthesis (angular SR) has been investigated by
many researchers in the past few years [9, 12, 19, 17, 22, 28, 30]. On the
basis of the number of input images, related approaches are categorized into
multi-view image [9, 22, 28, 30] and single image [17] based. Multi-view
image-based light field synthesis is impractical due to its specific input
pattern. Meanwhile a single image-based light field synthesis is severely ill-
posed although it is the most practical approach for light field synthesis.

#### Sparse-Input Light Field Synthesis

Wanner and Goldluecke [23] introduced a light field SR framework adopting the
estimated depth information and variational optimization to fill missing views
from a sparse light field image. Phase-based light field synthesis from a
micro-baseline stereo pair was proposed by Zhang et. al [31]. Those studies
were rooted on traditional approaches that use complex processing and various
optimization approaches. Meanwhile, learning-based view synthesis achieves
better results by using an end-to-end training strategy. Zhou et. al [32]
proposed a new geometric representation called appearance flow to synthesize
an image with a novel view. However, the proposed representation is not
generalized well to a complex scene with multiple object and non-homogeneous
background. Zhou et. al [33] presented a novel geometric representation called
multi-plane images (MPI) to synthesize a horizontal light field from a narrow
baseline camera. Srinivasan et. al [19] extended the MPI extrapolation
boundaries based on the Fourier domain analysis. Recent work by Mildenhall et.
al [12] exhibited state-of-the-art view synthesis performance with multi MPI
and a blending technique. However, these approaches require camera pose
information and/or multiple inputs to synthesize the novel view which is not
commonly available in the real world. In addition, MPI requires a significant
computing resource(increasing the layer count) to get an accurate result.

Kalantari et. al [9] introduced the first learning based light field synthesis
solution. They utilized four corner images to synthesize a 4D light field
using a depth estimation, warping, and color refinement approach. The inputs
to the depth estimation network were mean and variance images, as inspired by
the depth estimation work of [20]. Wu et. al [28] utilized an epipolar plane
image (EPI) obtained from sparse input images and synthesized an up-sampled
EPI through a specially designed blur kernel. The framework is then extended
further into several applications [27]. Wang et. al [22] employed a pseudo
4DCNN represented as 2D strided convolution and 3DCNN, where the light field
image was synthesized in a step-by-step manner. Yeung et. al [30] applied a
high dimensional convolutional kernel to infer spatial and angular information
from sparse input images. In summary, sparse input light field synthesis
focuses on synthesizing in-between views and could be regarded as solving an
interpolation problem. The specific input sampling pattern also hinders its
practical usage.

#### Single-Input Light Field Synthesis

Srinivasan et. al [17] introduced the first solution to solve light field
synthesis from a single image. They proposed a single image based depth
estimation to obtain the approximate geometry of a scene. Then, the estimated
depth was utilized to synthesize a novel view using the DIBR approach.
However, their method is constrained to a simple scene and highly dependent on
pixel intensity.

In this paper, we focus on solving the problem of single-image light field
angular synthesis. Contrary to [17], we propose to use an alternative
geometric representation (appearance flow) and present a light field based
loss function to avoid reliance on bright pixel color. Furthermore, we go
beyond the problem scope of [17] and solve the spatial resolution problem
simultaneously.

##  III Proposed Method

###  III-A Joint Light Field Super-Resolution

This paper aims to synthesize a high resolution 4D light field
${L}(\textbf{x},\textbf{u})$ given a single image that serves as the central
sub-aperture image (SAI) ${L}(\textbf{x},\textbf{0})$. We follow the two-plane
parametrization of light field ${L}(\textbf{x},\textbf{u})$, introduced by
[10], where x and u are the coordinates in spatial and angular planes,
respectively. In general, the light field synthesis problem is formulated as
DIBR problem, which is described as follows:

${L}(\textbf{x},\textbf{u})={L}(\textbf{x}+d(u),0),$ (1)

where $d(u)$ is the disparity in $x$ direction. Disparity depends on the depth
information of the central image and the novel angular coordinate $u$. We
address the light field synthesis problem by using an approximation function
$f(\cdot)$ represented as a deep convolutional neural network, as described in

${L}(\textbf{x},\textbf{u})=f({L}(\textbf{x},\textbf{0})).$ (2)

Function $f$ solves a highly ill-posed problem. To solve the problem jointly,
we design an encoder and multi decoder framework. We use the shared encoded
feature to solve both angular and spatial light field SR. The joint framework
is decomposed into two decoder branches. The top (angular) branch solves the
angular SR problem. While the bottom (spatial) branch solves the spatial SR
problem using additional information estimated by the first decoder branch.
The overall network structure is shown in Figure 2.

#### Angular Decoder

The angular decoder branch estimates appearance flows to extrapolate the
central view to each SAI in the 4D light field. Appearance flow [32]
represents 2D coordinate vectors specifying where pixels are mapped in the
reconstructed novel views. Appearance flow is accompanied with little blur,
preserves color identities, and removes the dependency on the physical-based
approach to synthesize novel views. Considering that the ground truth light
field appearance flow is difficult and expensive to obtain, the proposed
network is designed to estimate appearance flow in an unsupervised manner. The
network learns to estimate appearance flows by supervising the synthesized
light field (warped novel views) image.

The angular decoder is decomposed into three sub-problems, i.e. appearance
flow estimation for each viewpoint $u$, image shifting with respect to the
central view, and novel view extrapolation. Each sub-problem can be defined
mathematically as follows.

$\displaystyle{L}_{f}(\textbf{x},\textbf{u})$
$\displaystyle=\mathcal{F}({L}(\textbf{x},\textbf{0}))$ (3)
$\displaystyle{L}_{s}(\textbf{x},\textbf{u})$
$\displaystyle=\mathcal{S}({L}(\textbf{x},\textbf{0}),\nabla(u))$ (4)
$\displaystyle\hat{L}(\textbf{x},\textbf{u})$
$\displaystyle=\mathcal{W}({L}_{s}(\textbf{x},\textbf{u}),{L}_{f}(\textbf{x},\textbf{u})),$
(5)

where $\mathcal{F}$ estimates appearance flow ${L}_{f}$ for each novel angular
view. $\mathcal{S}$ performs angular shifting to the position $\nabla(u)$ of
novel views. Image shifting serves as a bias initialization for the network.
$\mathcal{W}$ is the warping function for shifted image
${L}_{s}(\textbf{x},\textbf{u})$ using its corresponding appearance flow
${L}_{f}(\textbf{x},\textbf{u})$. Image warping is performed using a bilinear
sampler module [7] to produce the light field image
$\hat{L}(\textbf{x},\textbf{u})$.

#### Spatial Decoder

We solve the angular SR first followed by spatial SR jointly. The spatial
decoder branch estimates a residual light field image to enhance the initially
estimated 4D light field (angular SR). The final HR light field is the result
of adding multi-stage residual images. The purpose of multi-stage residual
estimation is to solve missing high-frequency information and to post-process
the initial light field. They are further discussed in Section III-C. The
problem can be defined as follows.

${L}_{rf}(\textbf{x},\textbf{u}),{L}_{ri}(\textbf{x},\textbf{u})=\mathcal{R}({L}_{f}(\textbf{x},\textbf{u}),\hat{L}(\textbf{x},\textbf{u}))$
(6)
$\hat{L}^{H}(\textbf{x},\textbf{u})=\hat{L}(\textbf{x},\textbf{u})+{L}_{rf}(\textbf{x},\textbf{u})+{L}_{ri}(\textbf{x},\textbf{u})$
(7)

where $\mathcal{R}$ outputs two residual light field images, namely residual
flow ${L}_{rf}(\textbf{x},\textbf{u})$ and residual intensity
${L}_{ri}(\textbf{x},\textbf{u})$. The output of spatial decoder layer is
concatenated with appearance flow and initial light field image, indicated by
red dashed lines in Figure 2. Residual flow and residual intensity are named
based on the prior information concatenated to produce the residual light
field. The final high resolution light field is denoted by
$\hat{L}^{H}(\textbf{x},\textbf{u})$.

The proposed objective function is defined as

$\displaystyle\underset{\theta}{\text{min}}\sum[\lambda{{}_{g}}L_{g}(\theta)+\lambda{{}_{l}}L_{l}(\theta)+\lambda{{}_{tv}}\psi_{tv}(\theta)+\lambda{{}_{sr}}L_{sr}(\theta)],$
(8)

where $\theta$ denotes the deep neural network parameters. The problem
formulation and objective function enable the network to estimate an
appearance flow for each SAI at $u$ in an unsupervised manner through the
supervision of synthesized pixels. The common pixel-wise loss is not utilized
in the proposed method because it does not enforce the geometry constraint to
the network. Instead, we rely on the known light field domain knowledge and
design two geometrically constrained losses, i.e. global $L_{g}(\theta)$ and
local $L_{l}(\theta)$ light field losses. Both loss functions are useful in
preserving the spatio-angular consistency between light field SAIs. In
addition, we propose a regularization loss $\psi_{tv}(\theta)$ to the
estimated appearance flow to enforce angular consistency. $L_{sr}(\theta)$
denotes the spatial SR loss. Each loss is weighted with a corresponding
$\lambda$ to balance each loss contribution accordingly.

(a)

(b)

(c)

Figure 3: Estimated appearance flow. (a) Input image. (b) Appearance flow for
top left corner. (c) Appearance flow for bottom right corner. The color show
the direction of the flow with white color denotes zero flow.

###  III-B Angular Super-Resolution

Angular SR decoder relies on estimating appearance flow to warp the shifted
image into novel SAIs. To estimate an accurate and dense appearance flow, we
designed the encoder to extract all important representation in the image.
Sharing encoded feature with both decoders encourages the encoder to extract
important features in the input image. Angular decoder consists of convolution
layers with skip connections followed by leaky ReLU [11]. Additional details
about the network structure are available in the supplementary material.

In particular, the proposed flow estimation network produces coordinate
vectors $(x,y)$ to sample from ${L}_{s}(\textbf{x},\textbf{u})$ using the
bilinear sampler to synthesize the novel view
$\hat{L}(\textbf{x},\textbf{u})$. The appearance flow is visualized in Figure
3. It is observed that the estimated appearance flow is smooth, edge-aware,
and consistent in the inverse direction. In addition, it robustly estimates
the geometry of a scene with similar intensities. We learn $\mathcal{F}$ to
estimate the appearance flow by minimizing the loss between the extrapolated
novel views $\hat{L}(\textbf{x},\textbf{u})$ and ground truth light field
${L}(\textbf{{x}},\textbf{u})$. The angular decoder estimates appearance flow
for all novel SAIs in a single shot.

Image shifting is designed to guide the network by providing an initial bias.
This technique is inspired by the work of Xie et. al [29], in which the input
image is shifted to guide the network in synthesizing the corresponding stereo
image. The shifting operation can be written as follows.

${L}_{s}(\textbf{x},\textbf{u})={L}(\textbf{x}-\eta\Delta u,\textbf{0}),$ (9)

where $\eta$ is the constant angular shift value in horizontal and vertical
directions. $\Delta u$ is the angular distances between novel and central
views. Considering the redundancy in light field SAI, we can partially imitate
how pixels shift to each angular position and utilize this to provide better
initialization. The $\eta$ value is predetermined based on the disparity
between SAI in the target light field.

(a)

(b)

(c)

Figure 4: Residuals estimated from concatenating appearance flow and initial
light field image. (a) Input image. (b) Residual flow, ${L}_{rf}(x,u)$. (c)
Residual intensity, ${L}_{ri}(x,u)$.

###  III-C Spatial Super-Resolution

In spatial SR, we estimate residual images to recover the high frequency
information which is lost during initial light field upsampling. Moreover, we
try to refine the initial light field image. The spatial decoder is designed
to solve this problem by adding the initial light field image with residual
images. In particular, we incorporate information estimated by angular decoder
into the spatial decoder, such as appearance flow and the initial light field
image. We concatenate those information before the final convolution layer in
the decoder.

The estimated residual image confirms the effectiveness of the proposed multi-
stage residual framework, as shown in Figure 4. The first stage estimate the
high frequency information, such as edge region and textured (details) region.
While the second stage focus on a more sparse estimation of occlusion and
erroneous region. The second stage can be seen as post-processing or
refinement part of the framework.

The spatial decoder estimates residual image for every SAIs in the light field
image in a single shot. During inference, given a single input, the network
estimates high resolution light field in a single run. While in the training
stage, the spatial decoder is frozen for several iterations before both
decoders are trained together in an end-to-end fashion. These are discussed
further in Section IV.

###  III-D Light Field Loss

Although the proposed framework enables us to solve the super-resolution
problem jointly, a good objective function for learning the relation between
angular views is a mandatory. L1 or L2 loss, which are commonly used by
conventional approaches, cannot provide proper geometric reasoning to the
network. It encourages the network to look at dominant pixel color
individually instead of understanding the whole scene. We propose to use light
field angular information instead of pixel information individually. In
specific, we use mean and variance of a light field image which exists in any
light field image.

#### Global Light Field Loss

We propose a novel 4D light field loss, which is formulated as

$L_{g}(\theta)=|M(\hat{L}^{\theta})-M({L})|+|V(\hat{L}^{\theta})-V({L})|_{1}$
(10)

where

$\displaystyle M(L)=$
$\displaystyle\frac{1}{N}\displaystyle\sum_{s=1}^{N}L(\textbf{x},\textbf{u}_{s})$
(11) $\displaystyle V(L)=$
$\displaystyle{\frac{1}{N-1}\displaystyle\sum_{s=1}^{N}(L(\textbf{x},\textbf{u}_{s})-M(L))^{2}}.$
(12)

$M(L)$ and $V(L)$ denote the mean and variance while $s$ denotes the index of
SAI in the light field. Computing the light field mean is equivalent to
obtaining the refocus image at zero disparity. The refocused image correlates
to the depth of the light field image. We did not need labelled light field
depth because it is embedded in the light field itself through light field
mean. Therefore, the synthesized light field depth can be explicitly evaluated
in an efficient way and unsupervised manner. Meanwhile, variance captures the
difference between SAIs and helps the network learn the occlusion and edge
region. This is known from the light field depth estimation work of [20, 25,
26], which utilize the mean and variance to compute defocus and correspondence
responses, respectively. Kalantari et. al [9] also employed the mean and
variance images as the input to their depth estimation network. In this paper,
we show that mean and variance images can be used as loss function to help the
network learn the light field geometry efficiently.

Figure 5: Global and local light field losses. Blue and red dashed line denote
the global and local losses, respectively.

(a)

(b)

(c)

Figure 6: Denoising capability. (a) Noisy input. (b) Srinivasan et. al [17]
result. (c) Proposed result. Srinivasan’s result still contain some degree of
noise.

#### Local Light Field Loss

Although 4D global loss captures geometric information globally, the network
should learn the local geometric relation between SAIs in a refined manner.
The idea is to help the network explicitly understand angular relation in the
horizontal and vertical directions. We compute the mean and variance for SAIs
in each row and column in the 4D light field. The losses at each light field
row and column are accumulated to obtain the final local loss. The process can
be formulated as

$\displaystyle L_{e}(\theta)=$
$\displaystyle\sum_{m=1}^{U}\sum_{n=1}^{U}|M(\hat{L}_{m,n}^{\theta}(\textbf{x},\textbf{u}_{s}))-M({L_{m,n}}(\textbf{x},\textbf{u}_{s}))|_{1}+$
(13)
$\displaystyle|V(\hat{L}^{\theta}_{m,n}(\textbf{x},\textbf{u}_{s}))-V({L_{m,n}}(\textbf{x},\textbf{u}_{s}))|_{1},$

where $m,n$ denote the angular resolution of the light field image, and mean
and variance are computed for $s\in\\{1,\dotso,U\\}$. Without the loss of
generality, we assume the light field angular resolution is equal in both
horizontal and vertical directions. Figure 5 visualizes the losses by
computing the light field mean and variance.

#### Loss Regularization

An inconsistent appearance flow might appear and cause artifacts between SAIs.
This problem is expected because appearance flow is estimated from a single
image in an unsupervised manner. An alternative approach is to use the
conventional flow estimation method into the ground truth light field and
compare it with the estimated flow. However, this approach is tedious and
increases framework complexity. Thus, we present a strategy to remedy
inconsistent and incorrect flow by incorporating a regularization term into
the loss function.

To compress artifacts from inconsistent appearance flow, total variation
regularization is applied. Total variation is commonly used for noise removal
in image processing. The idea is to smooth inconsistent (noisy) appearance
flow while keeping important edge information. $L_{2}$ minimization is
performed on the gradient of the estimated appearance flow.

$\psi_{tv}(\theta)=||\nabla_{x}\hat{L}^{\theta}(\textbf{x},\textbf{u})||_{2}.$
(14)

Moreover, total variation denoising capability is also carried on to the
network, as shown in Figure 6.

#### Spatial Super-Resolution Loss

Spatial SR loss is straightforward. We minimize the error of upsampled initial
light field added by estimated residual images and the loss computation is
defined as follows.

$L_{sr}(\theta)=|\hat{L}^{H}(\textbf{x},\textbf{u})-{L}^{H}(\textbf{x},\textbf{u})|_{1},$
(15)

where ${L}^{H}(\textbf{x},\textbf{u})$ is the ground truth high resolution
light field image. We upsample the initial light field image using bilinear
interpolation.

[5pt]Reference image [5pt]Srininvasan et. al [17] [5pt]Proposed

Figure 7: Qualitative evaluation on Flower and Toys dataset. EPIs shown are
scaled and cropped for better visibility. The error map and sliced EPIs from
our method indicate geometrically better light field across all datasets. From
the top to bottom are Flower #5, Flower #15, Flower #94, Toys #29, and Toys
#35.

[5pt]Input image [5pt]Estimated depth [5pt]Background refocus [5pt]Foreground
refocus

Figure 8: Experiment on real single images and applications on the synthesized
light field.

##  IV Experimental Results

We evaluate the performance of the proposed framework both qualitatively and
quantitatively. We then compare its performance with the state-of-the-art
method of light field angular synthesis from a single image [17].

We utilize two datasets, i.e. Flower and Toys [17]. For the evaluation, all
networks are re-trained with the corresponding dataset. For evaluating
Srinivasan et. al [17], we use the author’s original code. Readers are
recommended to view the supplementary video for better understanding and
elaborated results.

Dataset | Srinivasan | Proposed LR | Proposed HR
---|---|---|---
Flower | 31.31 / 0.871 | 32.05 / 0.915 |  33.04 / 0.904
Toys | 32.21 / 0.798 | 35.91 / 0.901 |  36.92 / 0.898

TABLE I: Average PSNR (in dB) and SSIM from Flower (100 images) and Toys (100
images) test set.

###  IV-A Experiment on Available Light Field Dataset

Light field images in the Flower dataset mostly have a clear distinction
between the background and foreground. The flowers have a dominant color and
are located in the foreground. Unlike the Flower dataset, Toys dataset has
more variance in object shape, color, and location in the image. We utilize
Toys dataset to verify the performance of the proposed model in more general
scenes. Toys dataset is significantly more complex and closer to the real
world image. We evaluate both the initial light field (low resolution) and the
high resolution light field in PSNR and SSIM, which is shown in Table I. In
performing quantitative evaluation, we exclude the input image which is
directly used as the center of the synthesized light field. This is how
numbers for the quantitative evaluation is obtained. For each evaluation the
network is trained on the corresponding dataset. Note that we solve both
super-resolution problems simultaneously and therefore our network need to
solve more difficult problem than [17]. Nevertheless, the proposed method
outperforms the state-of-the-art method on both low resolution (LR) and high
resolution (HR) light field image. The proposed network outperforms [17],
which proves its capability for handling more complex and general scenes.

SSIM drop from LR to HR light field image stems from the second residual
addition which causes slight blur in the erroneous region. The spatial decoder
tries to smooth out the erroneous region in the initial light field. Figure 7
shows a qualitative comparison with the state-of-the-art work [17]. While [17]
achieves high PSNR, it suffers from unpleasing artifacts around the edge and
occluded region. In addition, [17] cannot handle the scenes with multiple
objects robustly. [17] assumes that any pixel with a similar intensity is
located in a close place. [17] fails on general scene due to its dependency on
finding a single object with a dominant color, which leads to inaccurate depth
estimation. As shown in the error map in Figure 7, pixels on the flower in the
backside have much error. On the other hand, the proposed method successfully
synthesizes a proper light field image due to the power of the proposed light
field based loss function. The error map and image patches show the proposed
method handles occlusion better and have less artifact.

Generally the result of [17] on Toys dataset shows incorrect EPI slope. The
slope direction is either reversed or flat indicating that the shifting
direction of the object is incorrect. [17] has difficulty in determining or
inferring objects position. On the contrary, the proposed method successfully
synthesizes a proper light field.

Metric | L1 | $L_{g}$ | $L_{l}$ | $L_{g}+L_{l}$ | $\mathcal{S}$
---|---|---|---|---|---
PSNR | 31.69 | 32.02 | 32.09 | 32.82 | 32.86
SSIM | 0.877 | 0.887 | 0.892 | 0.901 | 0.901

TABLE II: Quantitative evaluation of each loss function’s effect to the
network trained using Flower dataset. Each row represents the framework
trained using only the corresponding loss, except for $\mathcal{S}$ (full
framework trained without the image shifting). $L_{g}+L_{l}$ can also be
inferred as results without regularization.

Combination | First | Second | Metric (PSNR/SSIM)
---|---|---|---
1. | App. Flow | - | 32.55 / 0.891
2. | Intensity | - | 32.31 / 0.891
3. | App. Flow | App. Flow | 32.35 / 0.903
4. | Intensity | Intensity | 32.32 / 0.903
5. | Intensity | App. Flow | 33.01 / 0.904
6. | App. Flow | Intensity | 33.04 / 0.904

TABLE III: Quantitative evaluation of multi-stage (first and second) residual
estimation on Flower dataset. Appearance flow and intensity denotes
${L}_{rf}(\textbf{x},\textbf{u})$ and ${L}_{ri}(\textbf{x},\textbf{u})$,
respectively.

(a) Synthetic refocus on background and foreground, respectively.

(b) Estimated depth from the synthesized light field image.

(c) Estimated depth from the real light field image.

Figure 9: Synthetic defocus result and depth estimation results using CAE [26]
as a baseline.

(a)

(b)

(c)

Figure 10: Patches of spatial SR results. (a) LR input (256$\times$160). (b)
First inference (512$\times$320). (c) Second inference (1024$\times$640).

###  IV-B Ablation Study

To reveal the discriminate power of the proposed light field synthesis
network, we evaluate the performance of different loss functions, i.e. pixel-
wise L1 loss, global loss, local loss, and local–global loss. Table II shows
the quantitative evaluation of each loss function’s effect on the network. The
simple pixel-wise loss is outperformed by the proposed loss functions.
Meanwhile, local loss achieves satisfactory performance, and the combination
with global loss leads to the best result. Global and local losses complement
each other by forcing local and global consistency in synthesizing SAIs. Image
shifting and regularization are proven to improve network performance as shown
in $L_{g}+L_{l}$ and $\mathcal{S}$, respectively.

Additionally, we also show that the synthesized light field can be used for
common light field applications, such as depth estimation and refocusing. We
test our network on real single image captured using a smartphone or
downloaded from internet, as shown in Figure 8. This shows our network
capability to handle the images outside the training set distribution. In
addition, Figure 9 shows convincing depth estimation and synthetic refocus.
The estimated depth in both figures shows the network can estimate both steep
discontinuity or smooth discontinuity. It confirm that the synthesized light
field is geometrically correct and can be used in various applications of
light field processing.

To justify our multi-stage residual estimation design, we perform an extensive
experiment to identify the best combination. Table III shows that two-stage
residual estimation produces the best light field quantitatively. The order of
residual addition does not cause any significant difference. We hypothesize
that at the first stage the network recovers the lost information during
upsampling (anti-aliasing). Thus, we observe strong gradient in the estimated
residual and we believe this information strongly relates to the geometry
information (appearance flow). On the other hand, the second stage focuses on
the homogeneous region. Therefore, it relates to the pixel color. Note that we
try to increase the multi-stage residual into three stages, which does not
yield any meaningful boost to the performance.

We choose to perform angular resolution first as it saves the memory needed to
train the network. Upsampling the spatial resolution of the input image first
will lead to large memory consumption in estimating the appearance flow. The
process of reconstructing the light field is expensive during training stage.
Performing angular upsampling first gives us extra information to help the
spatial super resolution. In addition, using residual approach it acts as a
post-processing module while also solving the spatial SR problem.

Finally, we show the scalability of the proposed spatial SR module. Since
there is no available light field dataset in HR, we use similar technique by
[3] to predict image in HR. To obtain HR light field we run the network twice
sequentially. During the second inference, we only take the estimated residual
to enhance the previously estimated light field. Since no HR ground truth is
available, we show qualitatively in Figure 10 that 4x as well as 2x spatial SR
are performed successfully on the proposed framework.

##  V Proposed Network Structure

In this section, we provide details of the proposed network structure that
takes the luminance component of the input. The dimension is represented as
spatial (width and height) and channel or filter. Each convolution has the
bias initialized with 0 and the weight initialized using the He initializer
[34]. The image shifting operation is performed using tf.translate. Mean and
variance for computing loss are obtained using tf.nn.moments. The order of
light field stack is the column- or u based. The angular and spatial network
details are tabulated on Table IV and V, respectively. Every convolution is
followed by Leaky ReLU activation function. Convolution with stride 1,
convolution with stride 2, transpose convolution, and convolution without
activation function are denoted as ‘Conv’, ‘Conv*’, ‘Convt’, and ‘Convo’.
Concatenate operation on channel axis is denoted as ‘Concat’.

Network | Layer | Operation | Filter Size | Input Size | Output Size
---|---|---|---|---|---
| 1 | Conv + Conv | (3, 3, 16) | (128, 128, 1) | (128, 128, 16)
| 1 | Conv* | (3, 3, 16) | (128, 128, 16) | (64, 64, 16)
| 2 | Conv + Conv | (3, 3, 32) | (64, 64, 16) | (64, 64, 32)
| 2 | Conv* | (3, 3, 32) | (64, 64, 32) | (32, 32, 32)
Encoder | 3 | Conv + Conv | (3, 3, 64) | (32, 32, 32) | (32, 32, 64)
| 3 | Conv* | (3, 3, 64) | (32, 32, 64) | (16, 16, 64)
| 4 | Conv + Conv | (3, 3, 128) | (16, 16, 64) | (16, 16, 128)
| 4 | Conv* | (3, 3, 128) | (16, 16, 128) | (8, 8, 128)
| 5 | Conv + Conv | (3, 3, 256) | (8, 8, 128) | (8, 8, 256)
| 5 | Conv* | (3, 3, 256) | (8, 8, 256) | (4, 4, 256)
Bottleneck | 6 | Conv + Conv | (3, 3, 512) | (4, 4, 256) | (4, 4, 512)
| 7 | Convt \+ Concat | (3, 3, 256) | (4, 4, 512) | (8, 8, 256)
| 7 | Conv + Conv | (3, 3, 512) | (8, 8, 256) | (8, 8, 256)
| 8 | Convt \+ Concat | (3, 3, 128) | (8, 8, 256) | (16, 16, 128)
| 8 | Conv + Conv | (3, 3, 128) | (16, 16, 128) | (16, 16, 128)
| 9 | Convt \+ Concat | (3, 3, 64) | (16, 16, 128) | (32, 32, 64)
Angular | 9 | Conv + Conv | (3, 3, 64) | (32, 32, 64) | (32, 32, 64)
Decoder | 10 | Convt \+ Concat | (3, 3, 64) | (32, 32, 64) | (64, 64, 64)
| 10 | Conv + Conv | (3, 3, 64) | (64, 64, 64) | (64, 64, 64)
| 11 | Convt \+ Concat | (3, 3, 64) | (64, 64, 64) | (128, 128, 64)
| 11 | Conv + Conv | (3, 3, 64) | (128, 128, 64) | (128, 128, 64)
| 12 | Conv + Conv | (3, 3, 64) | (128, 128, 64) | (128, 128, 64)
| 12 | Convo | (3, 3, 128) | (128, 128, 64) | (128, 128, 128)

TABLE IV: The proposed angular decoder structure.

Network | Layer | Operation | Filter Size | Input Size | Output Size
---|---|---|---|---|---
| 7 | Convt | (3, 3, 256) | (4, 4, 512) | (8, 8, 256)
| 7 | Conv + Conv | (3, 3, 256) | (8, 8, 256) | (8, 8, 256)
| 8 | Convt | (3, 3, 128) | (8, 8, 256) | (16, 16, 128)
| 8 | Conv + Conv | (3, 3, 128) | (16, 16, 128) | (16, 16, 128)
| 9 | Convt | (3, 3, 64) | (16, 16, 128) | (32, 32, 64)
| 9 | Conv + Conv | (3, 3, 64) | (32, 32, 64) | (32, 32, 64)
| 10 | Convt | (3, 3, 64) | (32, 32, 64) | (64, 64, 64)
| 11 | Convt \+ Concat | (3, 3, 64) | (64, 64, 64) | (128, 128, 64)
Residual | 11 | Conv + Conv | (3, 3, 64) | (128, 128, 64) | (128, 128, 64)
Flow | 12 | Convt \+ Concat | (3, 3, 192) | (128, 128, 64) | (256, 256, 192)
| 12 | Conv + Conv | (3, 3, 192) | (256, 256, 192) | (256, 256, 192)
| 11* | Convt \+ Concat | (3, 3, 64) | (64, 64, 64) | (128, 128, 64)
Residual | 11* | Conv + Conv | (3, 3, 64) | (128, 128, 64) | (128, 128, 64)
Intensity | 12* | Convt \+ Concat | (3, 3, 192) | (128, 128, 64) | (256, 256, 192)
| 12* | Conv + Conv | (3, 3, 192) | (256, 256, 192) | (256, 256, 192)

TABLE V: The proposed spatial decoder structure. Layer 7 is the continuation
of bottleneck.

##  VI Training Details

The proposed network is implemented using TensorFlow [2]. We crop the light
field spatial resolution into random patches of
128$\times$128$\times$8$\times$8 for training. Full light field resolution is
used for inference. The proposed network is trained in an end-to-end fashion
for 280,000 iterations with a batch size of 1. The angular decoder is trained
alone for 100,000 iterations while the spatial decoder is freezed.
Hyperparameters $\eta$, $\lambda_{l}$, $\lambda_{g}$, $\lambda_{tv}$, and
$\lambda_{sr}$ are set to 0.8, 1.0, 10, $1e^{-4}$, and 10.0, respectively. We
augment the input with random gamma ranging from 0.4 to 1.0. The training
input is also randomly crop around the center region. The input augmentation
is designed to avoid overfitting. We use Adam optimizer [35] as our
optimization algorithm with the default parameters. Training is performed for
approximately 1 day on NVIDIA GTX 1080Ti GPU with 11GB of memory and Intel
i7-7700 @3.60 GHz CPU with 16GB of memory.

##  VII Conclusion

In this paper, we proposed an end-to-end deep model for joint angular and
spatial light field SR from a single image. Novel light field based loss
functions were introduced to preserve spatio-angular consistency and to remove
the dependency of pixel intensity. Joint end-to-end framework were presented
to solve both problem simultaneously. The experimental results showed that the
proposed network outperformed the state-of-the-art algorithm qualitatively and
quantitatively. In addition, the proposed method can also be generalized to
various scene, such as toys and internet images. Future work includes wide-
baseline light field synthesis, and higher degree of spatial SR.

## References

  * [1] Raytrix 3D light field camera.  https://raytrix.de/products/.  Accessed: 2018-09-08.
  * [2] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: a system for large-scale machine learning.  In OSDI, volume 16, pages 265–283, 2016.
  * [3] Michal Irani Assaf Shocher, Nadav Cohen.  ”zero-shot” super-resolution using deep internal learning.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [4] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely.  Deepstereo: Learning to predict new views from the world’s imagery.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 5515–5524, 2016.
  * [5] Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid.  Unsupervised cnn for single view depth estimation: Geometry to the rescue.  In Proc. of European Conference on Computer Vision, pages 740–756, 2016.
  * [6] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, volume 2, page 7, 2017.
  * [7] Max Jaderberg et al.  Spatial transformer networks.  In Prof. of Conference on Neural Information Processing Systems, 2015.
  * [8] Ole Johannsen, Antonin Sulc, Nico Marniok, and Bastian Goldluecke.  Layered scene reconstruction from multiple light field camera views.  In Proc. of Asian Conference on Computer Vision, pages 3–18, 2016\.
  * [9] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi.  Learning-based view synthesis for light field cameras.  ACM Trans. on Graphics, 35(6):193, 2016.
  * [10] Marc Levoy and Pat Hanrahan.  Light field rendering.  In ACM Trans. on Graphics, pages 31–42, 1996.
  * [11] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng.  Rectifier nonlinearities improve neural network acoustic models.  In Proc. of International Conference on Machine Learning, volume 30, page 3, 2013.
  * [12] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM Trans. on Graphics, 2019.
  * [13] Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, Pat Hanrahan, et al.  Light field photography with a hand-held plenoptic camera.  Computer Science Technical Report CSTR, 2(11):1–11, 2005.
  * [14] Jiayong Peng, Zhiwei Xiong, Dong Liu, and Xuejin Chen.  Unsupervised depth estimation from light field using a convolutional neural network.  In Proc. of IEEE International Conference on 3D Vision, pages 295–303, 2018.
  * [15] Hendrik Schilling, Maximilian Diebold, Carsten Rother, and Bernd Jähne.  Trust your model: Light field depth estimation with inline occlusion handling.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 4530–4538, 2018.
  * [16] Changha Shin, Hae-Gon Jeon, Youngjin Yoon, In So Kweon, and Seon Joo Kim.  Epinet: A fully-convolutional neural network using epipolar geometry for depth from light field images.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 4748–4757, 2018.
  * [17] Pratul P Srinivasan et al.  Learning to synthesize a 4D RGBD light field from a single image.  In Proc. of IEEE International Conference on Computer Vision, pages 2262–2270, 2017.
  * [18] Pratul P Srinivasan, Rahul Garg, Neal Wadhwa, Ren Ng, and Jonathan T Barron.  Aperture supervision for monocular depth estimation.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 6393–6401, 2018.
  * [19] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely.  Pushing the boundaries of view extrapolation with multiplane images.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 175–184, 2019.
  * [20] M. W. Tao, S. Hadap, J. Malik, and R. Ramamoorthi.  Depth from combining defocus and correspondence using light-field cameras.  In Proc. of IEEE International Conference on Computer Vision, pages 673–680, 2013.
  * [21] Alessandro Vianello, Jens Ackermann, Robert Bosch Campus, Maximilian Diebold, and Bernd Jähne.  Robust hough transform based 3D reconstruction from circular light fields.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 7327–7335, 2018.
  * [22] Yunlong Wang, Fei Liu, Zilei Wang, Guangqi Hou, Zhenan Sun, and Tieniu Tan.  End-to-end view synthesis for light field imaging with pseudo 4DCNN.  In Proc. of European Conference on Computer Vision, pages 340–355, 2018.
  * [23] Sven Wanner and Bastian Goldluecke.  Variational light field analysis for disparity estimation and super-resolution.  IEEE Trans. on Pattern Analysis and Machine Intelligence, 36(3):606–619, 2014.
  * [24] Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy.  High performance imaging using large camera arrays.  In ACM Trans. on Graphics, volume 24, pages 765–776, 2005.
  * [25] Williem and In Kyu Park.  Robust light field depth estimation for noisy scene with occlusion.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 4396–4404, 2016.
  * [26] Williem, In Kyu Park, and Kyoung Mu Lee.  Robust light field depth estimation using occlusion-noise aware data costs.  IEEE Trans. on Pattern Analysis and Machine Intelligence, (10):2484–2497, 2018.
  * [27] G. Wu, Y. Liu, L. Fang, Q. Dai, and T. Chai.  Light field reconstruction using convolutional network on EPI and extended applications.  IEEE Trans. on Pattern Analysis and Machine Intelligence, 41(7):1681–1694, 2019.
  * [28] Gaochang Wu, Mandan Zhao, Liangyong Wang, Qionghai Dai, Tianyou Chai, and Yebin Liu.  Light field reconstruction using deep convolutional network on EPI.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, volume 2017, page 2, 2017.
  * [29] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Deep3D: Fully automatic 2D-to-3D video conversion with deep convolutional neural networks.  In Proc. of European Conference on Computer Vision, pages 842–857, 2016.
  * [30] Henry Wing Fung Yeung, Junhui Hou, Jie Chen, Yuk Ying Chung, and Xiaoming Chen.  Fast light field reconstruction with deep coarse-to-fine modelling of spatial-angular clues.  In Proc. of European Conference on Computer Vision, 2018.
  * [31] Zhoutong Zhang, Yebin Liu, and Qionghai Dai.  Light field from micro-baseline image pair.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 3800–3809, 2015.
  * [32] Tinghui Zhou et al.  View synthesis by appearance flow.  In Proc. of European Conference on Computer Vision, 2016.
  * [33] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  ACM Trans. on Graphics, 37(4):65:1–65:12, 2018.
  * [34] He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jia.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.  In Proc. of IEEE International Conference on Computer Vision, 2015.
  * [35] Kingma, Diederik P and Ba, Jimmy.  Adam: A Method for Stochastic Optimization.  In Proc. of International Conference on Learning Representation, 2015.

|  Andre Ivan (S’18-M’19) received his B.CompSc degree in computer science
from Bina Nusantara University, Indonesia, in August 2017. He received his
M.S. degree from Inha University, Korea, in August 2019. He was a visiting
student in Center for Visual Computing, University of California, San Diego
(UCSD). He is currently working for Pretia Technology in Japan as a computer
vision researcher and development engineer. His research interests include
computer vision, computational photography, deep learning, and GPGPU. He is a
student member of the IEEE.
---|---
|  Williem (S’14-M’18) received his B.CompSc. degree in computer science from
Bina Nusantara University, Indonesia, in 2011, and the Ph.D. degree in
information and communication engineering from Inha University, Korea, in
August 2017. From September 2017, he has been in School of Computer Science,
Bina Nusantara University, Indonesia, as a faculty member. Currently, he is
the chief executive officer of Verihubs since June, 2019. His research focuses
on computer vision, computational photography, and image processing. Recent
research topics are depth estimation of various conditions and light field
applications. He is a member of the IEEE.
---|---
|  In Kyu Park (S’96-M’01-SM’14) received the B.S., M.S., and Ph.D. degrees
from Seoul National University (SNU) in 1995, 1997, and 2001, respectively,
all in electrical engineering and computer science. From September 2001 to
March 2004, he was a Member of Technical Staff at Samsung Advanced Institute
of Technology (SAIT). Since March 2004, he has been with the School of
Information and Communication Engineering, Inha University, where he is a full
professor. From January 2007 to February 2008, he was an exchange scholar at
Mitsubishi Electric Research Laboratories (MERL). From September 2014 to
August 2015, he was a visiting associate professor at MIT Media Lab. From July
2018 to June 2019, he was a visiting scholar at University of California, San
Diego. Dr. Park’s research interests include the joint area of computer vision
and graphics, including 3D shape reconstruction from multiple views, image-
based rendering, computational photography, deep learning, and GPGPU for image
processing and computer vision. He is a senior member of IEEE and a member of
ACM.
---|---