# Exploiting Raw Images for Real-Scene Super-Resolution

Xiangyu Xu, Yongrui Ma, Wenxiu Sun, Ming-Hsuan Yang  X. Xu is with Robotics
Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. E-mail:
<EMAIL_ADDRESS>Y. Ma is with SenseTime Research, Beijing, 100084,
China. E-mail<EMAIL_ADDRESS>W. Sun is with SenseTime Research,
HongKong, 999077. E-mail<EMAIL_ADDRESS>M.-H. Yang is with
School of Engineering, University of California, Merced, CA 95343, USA.
E-mail<EMAIL_ADDRESS>
###### Abstract

Super-resolution is a fundamental problem in computer vision which aims to
overcome the spatial limitation of camera sensors. While significant progress
has been made in single image super-resolution, most algorithms only perform
well on synthetic data, which limits their applications in real scenarios. In
this paper, we study the problem of real-scene single image super-resolution
to bridge the gap between synthetic data and real captured images. We focus on
two issues of existing super-resolution algorithms: lack of realistic training
data and insufficient utilization of visual information obtained from cameras.
To address the first issue, we propose a method to generate more realistic
training data by mimicking the imaging process of digital cameras. For the
second issue, we develop a two-branch convolutional neural network to exploit
the radiance information originally-recorded in raw images. In addition, we
propose a dense channel-attention block for better image restoration as well
as a learning-based guided filter network for effective color correction. Our
model is able to generalize to different cameras without deliberately training
on images from specific camera types. Extensive experiments demonstrate that
the proposed algorithm can recover fine details and clear structures, and
achieve high-quality results for single image super-resolution in real scenes.

###### Index Terms:

Super-resolution, raw image, training data generation, two-branch structure,
convolutional neural network (CNN).

##  1 Introduction

In digital photography, image resolution, _i.e_., the number of pixels
representing an object, is directly proportional to the square of the focal
length of a camera [1]. While one can use a long-focus lens to obtain a high-
resolution image, the range of the captured scene is usually limited by the
size of the sensor array at the imaging plane. Thus, it is often desirable to
capture the wide-range scene at a lower resolution with a short-focus camera
(_e.g_., a wide-angle lens), and then apply a single image super-resolution
algorithm to recover a high-resolution image from the corresponding low-
resolution frame.

Most state-of-the-art super-resolution methods [2, 3, 4, 5, 6, 7, 8, 9] are
based on data-driven models, and in particular, deep convolutional neural
networks (CNNs). While these methods are effective on synthetic data, they do
not perform well for images real captured by cameras or cellphones (see
examples in Figure 1(c)) mainly due to two important problems. First, existing
data generation pipelines are not able to synthesize realistic training data,
and models trained with such data do not perform well on most real images.
Second, the radiance information recorded by the cameras has not been fully
utilized. To address these problems for real-scene super-resolution, we
propose a pipeline for generating training data as well as a two-branch CNN
model for exploiting the raw information. We describe the above problems and
our solutions in more detail.

|  |  |  |
---|---|---|---|---
|  |  |  |
(a) Whole input image |  (b) Input |  (c) RDN with previous data |  (d) RDN with our data |  (e) Ours
Figure 1: Existing super-resolution method (_e.g_., RDN [8]) does not perform
well on real captured images as shown in (c). We can obtain sharper results
(d) by re-training the existing model [8] with more realistic training data
generated by our method. Furthermore, we recover more structures and details
(e) by exploiting the radiance information recorded in raw images. The two
input images in (a) are captured by Leica SL Typ-601 and iPhone 6s Plus
respectively, and both cameras are not seen by the models during training.
Best enlarge and view on screen.

Existing methods usually adopt an over-simplified generation model to
synthesize low-resolution images, including fixed downsampling blur kernels
(_e.g_., bicubic kernel) and homoscedastic Gaussian noise [8, 10]. However,
the blur kernel in practice may vary with zoom, focus, and camera shake during
the image formation process. As such, the fixed-kernel assumption usually does
not hold in real scenes. Furthermore, image noise usually obeys a
heteroscedastic Gaussian distribution [11, 12] in which variance depends on
the pixel intensity. More importantly, while the blur kernel and noise should
be applied to the linear raw data, existing approaches operate on the
nonlinear color images. To address these issues, we propose a data generation
pipeline by simulating the imaging process of digital cameras. We synthesize
the image degradations in the linear space and apply different downsampling
kernels as well as heteroscedastic Gaussian noise to approximate real
scenarios. As shown in Figure 1(d), training an existing model [8] using our
generated data leads to sharper results.

While modern cameras provide users with both the raw data and color image
processed by the image signal processing system (ISP) [13], most super-
resolution algorithms only take the color image as input without making full
use of the radiance information contained in the raw data. In contrast, we
directly use the raw data for restoring high-resolution clear images, which
conveys the following advantages. First, more information can be exploited in
raw pixels since they are typically 12 or 14 bits [13, 14], whereas the color
pixels produced by ISP are typically 8 bits [13]. In addition to the bit
depth, there is also information loss within a typical ISP (Figure 2) such as
noise reduction and compression [15]. Second, the raw data is linearly
proportional to scene radiance while the ISP system contains nonlinear
operations such as tone mapping. Thus, the linear degradations in the imaging
process, including blur and noise, are linear for raw data but nonlinear in
the processed RGB images. As this nonlinear property makes image restoration
more complicated, it is beneficial to address the problem on the raw data [16,
17]. Third, the demosaicing step in ISP is closely related to image super-
resolution because both problems are concerned with the spatial resolution
limitations of cameras [18]. Therefore, to solve the super-resolution problem
with processed images is sub-optimal and less effective than a single unified
model solving both problems simultaneously.

---
Figure 2: A typical ISP pipeline.
---
Figure 3: Overview of the proposed network. Our model consists of two modules,
where one exploits raw data $X_{raw}$ to restore high-resolution linear
measurements $\tilde{X}_{lin}$ for all color channels with clear structures
and fine details, and the other one estimates the transformation matrix to
recover the final color result $\tilde{X}$ using the low-resolution color
image $X_{ref}$ as reference.

Nevertheless, directly mapping a degraded raw image to a desired full color
output [19, 20] results in networks only suitable for specific cameras, as the
raw data does not contain the relevant information for color corrections
(_e.g_., color space conversion and tone mapping) conducted within the ISP. To
address this issue, we propose a dual CNN architecture (Figure 3) which takes
both the degraded raw and color images as input, such that our model can
generalize well to different cameras. The proposed model consists of two
branches, where one module restores clear structures and fine details from the
raw data, and the other recovers high-fidelity colors using the low-resolution
RGB image as a reference. For the image restoration module, we train a CNN to
reconstruct the high-resolution linear color measurements from the raw input.
To increase the representation capacity of the image restoration model, we
improve the densely-connected convolution layers [21, 22] and propose a dense
channel-attention network. For the color correction module, a straightforward
approach is to apply the guided filter [23] to transfer the color of the
reference image to the restored linear image. However, simply adopting this
method usually leads to artifacts and inaccurate color appearances due to
limitations of local constant assumptions of the guided filter [23]. Instead,
we present a learning-based guided filter which can better adapt to high-
frequency image structures and generate more appealing results. As shown in
Figure 1(e), the proposed algorithm significantly improves the super-
resolution results for real-scene images.

We make the following contributions in this work. First, we propose a data
generation pipeline which is able to synthesize realistic raw and color images
for learning real-scene image super-resolution models. Second, we develop a
two-branch network to exploit both raw data and color images. In the first
module, we design a dense channel-attention block to improve image
restoration. In the second module, we develop a learning-based guided filter
to overcome the limitations of the original guided filter, which helps recover
more appealing color appearances. Finally, we propose a real image dataset to
better evaluate different models. Extensive experiments demonstrate that the
proposed method can recover fine details as well as clear structures, and
generate high-quality results in real scenes.

##  2 Related Work

We discuss the state-of-the-art super-resolution methods as well as learning-
based raw image processing techniques, and put this work in proper context.

###  2.1 Super-resolution

Classical super-resolution methods can be broadly categorized into exemplar-
based [24, 25, 26, 27] and regression-based [28, 4]. One typical exemplar-
based method is developed based on the Markov model [24] to recover high-
resolution images from low-resolution inputs. As this algorithm directly
combines and stitches high-resolution patches from an external dataset, it
tends to introduce unrealistic details in the reconstructed results.
Regression-based approaches typically learn the priors from image patches with
either linear functions [28] or anchored neighborhood regressors [4]. However,
the reconstructed images are often over-smoothed due to the limited capacity
of the regression models.

With the rapid advances of deep learning, state-of-the-art super-resolution
methods based on deep CNNs have been developed to reconstruct high-resolution
images [5, 3, 7, 2, 8, 6, 29, 9]. Dong _et al_. [2] propose a three-layer CNN
for mapping the low-resolution representation to the high-resolution space,
but do not obtain better results with deeper networks [30]. To solve this
problem, Kim _et al_. [3] introduce residual connections to alleviate the
gradient-vanishing issue during the training process, which achieves
significantly better results. Similarly, recursive layers with skip
connections are introduced for more efficient network structures [31]. As the
dense connections [21] are shown to be effective for strengthening feature
propagation, they are also used for super-resolution to facilitate the
reconstruction process [7, 8]. However, these algorithms only perform well on
synthetic data and do not generate high-quality results for real-scene inputs.

Another line of work aims to increase the perceptual quality of the results
[32, 33, 34, 9, 35]. Johnson _et al_. [32] propose a perceptual loss based on
high-level features extracted from the VGG network [36]. Ledig _et al_. [33]
adopt the generative adversarial nets (GAN) [37] to learn the distributions of
real images. Sajjadi _et al_. [35] further improve the adversarial loss by
enforcing a Gram-matrix matching constraint. Xu _et al_. [9] analyze the GAN
loss from the maximum-a-posterior perspective and combine the pixel-wise loss,
perceptual loss and GAN loss in an adversarial framework, which super-resolves
text and face images well. However, the results hallucinated by GAN may
contain unpleasant artifacts and details not present in the ground truth.

While the aforementioned methods are effective in interpolating pixels and
hallucinating images, they are only based on processed color images and less
effective in generating sharp image structures and high-fidelity details for
real inputs. In contrast, we exploit both the raw and color images in a
unified framework for better super-resolution results. In addition, we propose
a pipeline for synthesizing realistic training data to improve the
performance.

Super-resolution with variable degradations. While most of the aforementioned
methods [5, 3, 7, 2, 8, 6, 29, 32, 33, 35] formulate the super-resolution
problem with a fixed degradation model, some recent appraches [9, 38, 39, 40]
also consider variable blur kernels and noise levels, which are closer to real
scenes. In particular, Zhang _et al_. [38] use the blur kernel and noise level
as additional channels of the input and attain favorable results under a
multiple-degradation setting. A deep unfolding network is introduced in [40]
to super-resolve blurry and noisy images. In this work, we also consider
variable degradations in the proposed data generation pipeline, which
facilitates training more robust models for real applications.

Joint super-resolution and demosaicing. Most existing methods for this problem
estimate a high-resolution color image with multiple low-resolution frames
[41, 42]. Closely related to our work is the method based on a deep residual
network [18] for single image super-resolution with mosaiced input. However,
this model is trained on gamma-corrected image pairs which may not perform
well on real-scene images. More importantly, these methods do not consider the
complex color correction steps in camera ISPs, and thus cannot recover high-
fidelity color appearances from the raw input. In contrast, the proposed
algorithm simultaneously addresses the problems of image restoration and color
correction to handle real-scene images.

###  2.2 Learning-based raw image processing

In recent years, numerous learning-based methods have been proposed for raw
image processing [43, 19, 20]. Jiang _et al_. [43] propose to learn a large
collection of local linear filters to approximate the complex nonlinear ISP
pipelines. In [20], Schwartz _et al_. use deep CNNs for learning the color
correction operations of specific digital cameras. In addition, Chen  _et al_.
[19] train a neural network with raw data as input for fast low-light imaging
and coloring. In this work, we learn color correction in the context of raw
image super-resolution. Instead of learning a color correction pipeline for
one specific camera, we use a low-resolution color image as reference for
handling images from more diverse ISPs.

---
Figure 4: Synthesizing linear color measurements from raw data. The sensels
surrounded by the black-dot curve is a Bayer pattern block and defined as a
new virtual sensel in Section 3.1.

##  3 Proposed Algorithm

For high-quality single image super-resolution, we propose a pipeline to
synthesize more realistic training images and a two-branch CNN model to
exploit the radiance information recorded in raw data.

###  3.1 Data generation pipeline

Most super-resolution methods [3, 2] generate training data with a fixed
downsampling kernel in the nonlinear color space. In addition, homoscedastic
Gaussian noise is often added to the images for modeling the degradation
process [8, 9]. However, as discussed in Section 1, the low-resolution images
generated in this manner do not resemble real captured images and are less
effective for training real-scene super-resolution models. Furthermore, we
need to generate low-resolution raw data for training the proposed neural
network, which is often realized by directly mosaicing the low-resolution
color images [44, 18]. This approach ignores the fact that the color images
have already been processed by the nonlinear operations of the ISP, and the
subsequent synthesized images suffer from information loss and do not
represent the linear response of the camera sensors. Instead, the raw data
should be linear measurements of the original pixels.

To address the above issues, we use high-quality raw images [14] and propose a
data generation pipeline by simulating the imaging process in real scenarios.
We first synthesize ground-truth linear color measurements by downsampling
high-quality raw images such that each pixel can have its ground-truth red,
green and blue values. For the Bayer pattern raw data (Figure 4), we define a
block of Bayer pattern sensels (_i.e_., sensor elements) as one new virtual
sensel, where all color measurements are available. As such, we can obtain the
desired images $X_{lin}\in\mathbb{R}^{H\times W\times 3}$ with linear
measurements of all three colors for each pixel, where $H$ and $W$ denote the
height and width of the ground-truth linear image. Similar to [45], we
compensate the color shift artifact by aligning the center of each color in
the new sensel.

---
Figure 5: The image restoration module uses the DCA blocks in an encoder-
decoder framework and reconstructs high-resolution linear color measurements
$\tilde{X}_{lin}$ from the degraded low-resolution raw input $X_{raw}$.

With the ground-truth linear measurements $X_{lin}$, we can easily generate
the ground-truth color images $X_{gt}\in\mathbb{R}^{H\times W\times 3}$ by
simulating the color correction steps of the ISP, such as color space
conversion and tone adjustment. In this work, we apply the widely-used Dcraw
[46] toolkit to process the raw image data.

To generate degraded low-resolution raw images
$X_{raw}\in\mathbb{R}^{\frac{H}{2}\times\frac{W}{2}}$, we apply the blurring,
downsampling, Bayer sampling operations to the linear color measurements:

$X_{raw}=f_{Bayer}(f_{down}(X_{lin}*k_{def}*k_{mot}))+n,$ (1)

where $f_{Bayer}$ is the Bayer sampling function which mosaics images in
accordance with the Bayer pattern (Figure 4), and $n$ is the noise term. In
addition, $f_{down}$ represents the downsampling function with a sampling
factor of two. Since the imaging process is likely to be affected by out-of-
focus effect and camera shake, we consider both defocus blur $k_{def}$ modeled
as disk kernels with variant sizes [9, 47], and modest motion blur $k_{mot}$
generated by random walk [48, 10]. In (1), $*$ denotes the convolution
operator. To synthesize more realistic training data, we add heteroscedastic
Gaussian noise [49, 11, 50] to the generated raw data:

$n(x)\sim\mathcal{N}(0,\sigma_{1}^{2}x+\sigma_{2}^{2}),$ (2)

where the variance of noise $n$ depends on the pixel intensity $x$, and
$\sigma_{1}$ and $\sigma_{2}$ are parameters of the noise. Finally, the raw
image $X_{raw}$ is demosaiced with the AHD [51] method, and processed by the
Dcraw toolkit to produce the low-resolution color image
$X_{ref}\in\mathbb{R}^{\frac{H}{2}\times\frac{W}{2}\times 3}$. We compress
$X_{ref}$ as 8-bit JPEG as normally done in digital cameras. Note that the
settings for the color correction steps in Dcraw are the same as those used in
generating $X_{gt}$, such that the reference colors correspond well to the
high-resolution ground-truth. The proposed pipeline is able to synthesize
realistic data for super-resolution, and thereby the models trained on
$X_{raw}$ and $X_{ref}$ can effectively generalize to real-degraded raw and
color images (Figure 1(e)).

---
Figure 6: Illustration of the proposed DCA block.

###  3.2 Network architecture

A straightforward approach to exploit raw data for super-resolution is to
directly learn a mapping function from raw inputs to high-resolution color
images with neural networks. Whereas the raw data is important for image
restoration, such a simple approach does not perform well in practice. The
reason is that the raw data does not have the relevant information for color
correction and tone enhancement within the ISP. A raw image could potentially
correspond to multiple ground-truth color images generated by different image
processing algorithms of various ISPs, which will complicate the network
training process and make it extremely difficult to train a model to
reconstruct high-resolution images with desired colors. In this work, we
propose a two-branch CNN (see Figure 3) where the first module exploits the
linear radiance information recorded in the raw data $X_{raw}$ to restore the
high-resolution linear measurements for all color channels with clear
structures and fine details. The second module estimates transformation
matrices to recover high-fidelity color appearances for the restored linear
measurements using the low-resolution color image $X_{ref}$ as a reference.
The reference image complements the limitations of raw data, and jointly
training two modules with both raw and color data helps reconstruct higher-
quality results.

####  3.2.1 Image restoration

The main components of the image restoration network are shown in Figure 5.
This network consists of four parts: packing raw data, low-level feature
extraction, deep nonlinear mapping, and high-resolution image reconstruction.
Similar to [19], we first convert an input image $X_{raw}$ into a simpler form
$X^{\prime}_{raw}\in\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times 4}$ by
packing the Bayer raw data into four color channels which correspond to the R,
G, B, G patterns in Figure 4, and reducing the spatial resolution by a factor
of two in each dimension. Similar to [7], we extract low-level features
$F_{low}$ from the packed four-channel input with a convolution layer with
kernel of $3\times 3$ pixels.

To learn more complex nonlinear functions with the features $F_{low}$, we
propose a dense channel-attention (DCA) block as shown in Figure 6 for better
image restoration performance. Details about the DCA block are elaborated
later in this section. We consecutively apply four DCA blocks after $F_{low}$.
Different from prior work [21, 8, 29] which repeat the blocks at the same
scale, we use the proposed blocks in an encoder-decoder framework (similar to
U-net [52]) to exploit multi-scale information. To downscale and upscale image
features, we use convolution and deconvolution layers with stride of 2,
respectively. As shown in Section 5.3, exploiting multi-scale features
effectively reduces blurry artifacts and improves the super-resolution
results.

Furthermore, we use skip connections to combine low-level and high-level
feature maps with the same spatial size to generate the decoded features
(_i.e_., $F_{s1}$ and $F_{s2}$ in Figure 5). We feed $F_{s2}$ into the final
image reconstruction module consisting of a convolution layer to generate $48$
feature maps, and a sub-pixel layer [53] to rearrange the produced features
into $\tilde{X}_{lin}\in\mathbb{R}^{H\times W\times 3}$. Here
$\tilde{X}_{lin}$ represents the high-resolution linear output of the image
restoration module which has clear structures and fine details, and is further
processed in the second module to recover desirable color appearances.

DCA block. The state-of-the-art neural networks [54, 21, 55] usually rely on
well-designed building blocks to achieve high-quality results. Since it has
recently been shown that the performance of convolutional networks can be
substantially improved by dense connections [21], we first apply $k$ densely-
connected layers in the proposed DCA block. As shown in Figure 6, for each
layer of the dense-connection module we can obtain the output by first
processing the input features with a 3$\times$3 convolution layer and then
concatenating the convolved feature maps with the input. The feature maps of
all preceding layers are thus used as input of the current layer, and its own
output features are used as inputs of all subsequent layers.

Note that the output of the dense module is a simple concatenation of features
from all the convolution layers in the dense module. Since these features are
usually of different significance for final image restoration, treating them
channel-wise equally is not flexible during the training process and will
restrict the representation strength of the network. To address this issue, we
utilize the channel-attention mechanism [55] which allows the network to
recalibrate inter-channel features, such that informative feature maps are
emphasized, and less useful ones are suppressed.

The channel attention module shown in Figure 6 consists of a global average
pooling which aggregates the input spatially to estimate channel-wise feature
statistic, and two fully-connected layers which exploit the statistic of all
channels to learn a gating function. We use the sigmoid activation to
normalize the gating function, and the output can adaptively recalibrate the
dense features with channel-wise multiplication. Similar to [54], we use a
residual connection between the block input and output for better information
flow. With the proposed DCA block, we can effectively improve the network
representation capacity and achieve better performance for image super-
resolution.

####  3.2.2 Color correction

After obtaining the linear measurements $\tilde{X}_{lin}$, we need to apply
color correction and reconstruct the final result
$\tilde{X}\in\mathbb{R}^{H\times W\times 3}$ using the reference color image
$X_{ref}$. This can be formulated as a joint image filtering problem and one
solution is to apply the guided filter [23]:

$\tilde{X}^{c}[\boldsymbol{x}]=\mathcal{A}_{g}(X^{c}_{ref},\tilde{X}^{c}_{lin})[\boldsymbol{x}]\tilde{X}^{c}_{lin}[\boldsymbol{x}]+\mathcal{B}_{g}(X^{c}_{ref},\tilde{X}^{c}_{lin})[\boldsymbol{x}],$
(3)

where $\mathcal{A}_{g}\in\mathbb{R}^{H\times W}$ and
$\mathcal{B}_{g}\in\mathbb{R}^{H\times W}$ are the transformation matrices of
the guided filter, which transfer the color appearances of $X_{ref}$ to
$\tilde{X}$ while preserving the structures of $\tilde{X}_{lin}$. In (3),
$X^{c}$ indicates the $c$-th channel of the image $X$, and $[\boldsymbol{x}]$
is the spatial indexing operation which fetches an element from the spatial
position $\boldsymbol{x}\in\mathbb{R}^{2}$.

However, both $\mathcal{A}_{g}$ and $\mathcal{B}_{g}$ are estimated based on
local constraints [23] that the transformation coefficients are constant in
local windows. Specifically, to solve $\mathcal{A}_{g}$ and $\mathcal{B}_{g}$
for a pixel $\mathbf{x}$, the guided filter takes a local window
$\mathcal{N}_{\mathbf{x}}$ where $\mathbf{x}\in\mathcal{N}_{\mathbf{x}}$, and
minimizes the following cost function:

$\sum_{\mathbf{y}\in\mathcal{N}_{\mathbf{x}}}(\mathcal{A}_{g}[\mathbf{y}]\tilde{X}_{lin}^{c}[\mathbf{y}]+\mathcal{B}_{g}[\mathbf{y}]-X_{ref}^{c}[\mathbf{y}])^{2},$
(4)

where the transformed $\tilde{X}_{lin}$ is used to approximate the reference
image $X_{ref}$. Since the local constraints assume $\mathcal{A}_{g}$ and
$\mathcal{B}_{g}$ are constant in $\mathcal{N}_{\mathbf{x}}$,
$\mathcal{A}_{g}[\mathbf{x}]$ and $\mathcal{B}_{g}[\mathbf{x}]$ can be
obtained by solving (4) with linear regression [23]. However, as the pixel
$\boldsymbol{x}$ is involved in all windows that contain $\boldsymbol{x}$, the
values of $\mathcal{A}_{g}[\mathbf{x}]$ and $\mathcal{B}_{g}[\mathbf{x}]$ are
not the same when they are computed in different windows. The final
coefficients $\mathcal{A}_{g}[\mathbf{x}]$ and $\mathcal{B}_{g}[\mathbf{x}]$
are estimated by averaging all possible values. Thus, the local constant
assumption leads to oversmoothed transformation matrix $\mathcal{A}_{g}$ and
$\mathcal{B}_{g}$, which cannot well handle the color correction around high-
frequency structures (Figure 15(d)). To address this issue, we propose a
learning-based guided filter for better color correction.

---
Figure 7: Network architecture of the color correction module. Our model predicts the pixel-wise transformations $\mathcal{A}$ and $\mathcal{B}$ with a reference color image.  |  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
(a) Whole input image |  (b) Input |  (c) SID |  (d) SRDenseNet |  (e) RDN |  (f) Ours |  (g) GT
Figure 8: Results from the proposed synthetic dataset. References for the baseline methods can be found in Table I. “GT” represents ground-truth.  |  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
(a) Whole input image |  (b) Input |  (c) SID |  (d) SRDenseNet |  (e) RDN |  (f) Ours |  (g) GT
Figure 9: Results of non-blind image super-resolution. References for the
baseline methods can be found in Table I. “GT” represents ground-truth.

Learning-based guided filtering. In contrast to the original guided filter
which estimates $\mathcal{A}_{g}$ and $\mathcal{B}_{g}$ with local
constraints, we propose to learn the color correction matrices with neural
networks. Similar to (3), the proposed filtering process can be formulated as:

$\tilde{X}[\boldsymbol{x}]=\mathcal{A}(X_{ref},\tilde{X}_{lin})[\boldsymbol{x}]\tilde{X}_{lin}[\boldsymbol{x}]+\mathcal{B}(X_{ref},\tilde{X}_{lin})[\boldsymbol{x}],$
(5)

where $\mathcal{A}\in\mathbb{R}^{H\times W\times 3\times 3}$ and
$\mathcal{B}\in\mathbb{R}^{H\times W\times 3}$ are the transformation matrices
predicted by a deep CNN (Figure 7). In (5),
$\mathcal{A}[\boldsymbol{x}]\in\mathbb{R}^{3\times 3}$ transforms the RGB
vector $\tilde{X}_{lin}[\boldsymbol{x}]\in\mathbb{R}^{3}$ to a new 3-channel
color space, and $\mathcal{B}[\boldsymbol{x}]\in\mathbb{R}^{3}$ serves as
biases of the transformation. Since the learning-based guided filter is not
restricted by the local constant assumption, the learned transformations can
better adapt to high-frequency color changes and achieve higher-quality color
correction results. In addition, the original guided filter processes each
image channel independently, whereas we learn pixel-wise 3$\times$3 matrices
which can exploit the inter-channel information for better color
transformation.

We use a light-weight CNN to predict the transformations $\mathcal{A}$ and
$\mathcal{B}$ as shown in Figure 7. The CNN begins with an encoder including
three convolution layers with 2$\times$2 average pooling to extract the color
features from $X_{ref}$. To estimate the spatially-variant transformations, we
use a decoder consisting of consecutive deconvolution and convolution layers,
which expands the encoded features to the desired resolution $H\times W$. We
concatenate the same-resolution feature maps in the encoder and decoder to
exploit hierarchical representations. Finally, the output layer of the decoder
generates 3$\times$3 weights and 3$\times$1 biases for each pixel, which form
the pixel-wise transformation matrices $\mathcal{A}$ and $\mathcal{B}$.

Note that in addition to the network input $X_{ref}$,
$\mathcal{A}(X_{ref},\tilde{X}_{lin})$ and
$\mathcal{B}(X_{ref},\tilde{X}_{lin})$ are estimated based on the output of
the first module $\tilde{X}_{lin}$ as well. Instead of directly using
$\tilde{X}_{lin}$ in the estimation network, we exploit richer features
extracted from the restoration module which have been used to generate
$\tilde{X}_{lin}$. Specifically, we fuse the decoded features $F_{s1}$ and
$F_{s2}$ into the color correction module via pixel-wise summation:

$G^{\prime}_{si}=G_{si}+\phi_{i}(F_{si}),~{}~{}i=1,2,\vspace{-1mm}$ (6)

where $G_{si}$ is the feature after the concatenation layer of the color
correction network as shown in Figure 7. As $G_{si}$ and $F_{si}$ usually have
different numbers of channels, we fuse them by introducing a 1$\times$1
convolution layer $\phi_{i}$ whose weights are learned to adaptively transform
$F_{si}$. After feature fusion, we put the updated features $G^{\prime}_{si}$
back to the color correction network, and the following layers can further
exploit the fused features for transformation matrix prediction. Note that we
adopt a multi-scale feature fusion strategy to improve our early model [22]
where only single scale features (_i.e_., $F_{s1}$ and $G_{s1}$) are used. As
the predicted transformations are applied to $\tilde{X}_{lin}$, it is
beneficial to make the matrix estimation network aware of the multi-scale
features in the image restoration module, such that $\mathcal{A}$ and
$\mathcal{B}$ can adapt more accurately to the structures of the restored
image.

Relationship to global color correction. Schwartz  _et al_. [20] estimate a
global transformation for color correction of the image. Similarly, we can
also apply this method to replace the proposed learning-based guided filter:

$\tilde{X}[\boldsymbol{x}]=\mathcal{A}_{s}(X_{ref},\tilde{X}_{lin})\tilde{X}_{lin}[\boldsymbol{x}]+\mathcal{B}_{s}(X_{ref},\tilde{X}_{lin}),$
(7)

where $\mathcal{A}_{s}\in\mathbb{R}^{3\times 3}$ and
$\mathcal{B}_{s}\in\mathbb{R}^{3}$ represent the global transformation
predicted by a CNN with global average pooling and fully-connected layers
[20]. However, this global approach does not perform well when the color
transformation of the ISP involves spatially-variant operations. In contrast,
we learn pixel-wise transformations $\mathcal{A}[\boldsymbol{x}]$ and
$\mathcal{B}[\boldsymbol{x}]$ which allow different color corrections for each
spatial location $\boldsymbol{x}$. Note that [20] uses a quadratic form of the
RGB vector $\tilde{X}_{lin}[\boldsymbol{x}]$ for (7), from which, we find no
obvious improvements empirically.

Relationship to end-to-end trainable guided filter. Another related method of
the color correction network is the end-to-end trainable guided filter (ETGF)
[56]:

$\displaystyle{\footnotesize\tilde{X}^{c}[\boldsymbol{x}]=\mathcal{A}_{g}(X^{c}_{ref},\varphi(\tilde{X}_{lin})^{c})[\boldsymbol{x}]\varphi(\tilde{X}_{lin})^{c}[\boldsymbol{x}]+\mathcal{B}_{g}(X^{c}_{ref},\varphi(\tilde{X}_{lin})^{c})[\boldsymbol{x}],}\vspace{-1mm}$
(8)

where $\varphi$ is a two-layer CNN to transform $\tilde{X}_{lin}$ into a new
3-channel representation for filtering, and $\varphi(\tilde{X}_{lin})^{c}$ is
the $c$-th channel of $\varphi(\tilde{X}_{lin})$. Since $\mathcal{A}_{g}$ and
$\mathcal{B}_{g}$ are computed with the same method as the original guided
filter (3), the ETGF is also restricted by the local constant assumption [23],
and thereby cannot adapt to high-frequency color changes as effectively as the
proposed learning-based guided filter (5).

##  4 Experimental Results

We first describe the implementation details as well as datasets, and then
evaluate the proposed algorithm on both synthetic and real images. We present
the main findings in this section and more results on the project website
https://sites.google.com/view/xiangyuxu/rawsr_pami.

TABLE I: Quantitative evaluations on the synthetic dataset. “Blind” represents the images with variable blur kernels, and “Non-blind” denotes fixed kernel.  Methods | Blind | Non-blind
---|---|---
PSNR | SSIM | PSNR | SSIM
SID [19] | 21.87 | 0.7325 | 21.89 | 0.7346
DeepISP [20] | 21.71 | 0.7323 | 21.84 | 0.7382
SRCNN [2] | 28.83 | 0.7556 | 29.19 | 0.7625
VDSR [3] | 29.32 | 0.7686 | 29.88 | 0.7803
SRDenseNet [7] | 29.46 | 0.7732 | 30.05 | 0.7844
RDN [8] | 29.93 | 0.7804 | 30.46 | 0.7897
SID∗ [19] | 29.55 | 0.7715 | 29.76 | 0.7749
RDN∗ [8] | 30.35 | 0.7954 | 31.19 | 0.8211
Ours | 31.06 | 0.8088 | 32.07 | 0.8314

###  4.1 Implementation details

To generate the training data, we use the MIT-Adobe 5K dataset [14] which
consists of $5000$ raw photographs with image size around $2000\times 3000$
pixels. After manually removing images with noticeable noise and blur, we
obtain $1300$ high-quality images for training and $150$ for testing which are
captured by $19$ types of Canon cameras. We use the method described in
Section 3.1 to synthesize the training and test datasets. The radius of the
defocus blur is randomly sampled from $[1,5]$ pixels and the maximum size of
the motion kernel is sampled from $[3,11]$ pixels. The noise parameters
$\sigma_{1},\sigma_{2}$ are respectively sampled from $[0,10^{-2}]$ and
$[0,10^{-3}]$.

We use the Xavier initializer [57] for the network weights and LeakyReLU [58]
with slope of 0.2 as the activation function. For the DCA block, we set the
growth rate as 16 and $k$ as $8$. We crop $256\times 256$ patches as input and
use a batch size of $6$ for training. During the testing phase, we divide an
input image into overlapping patches and process each patch separately. The
reconstructed high-resolution patches are placed back to the corresponding
locations and averaged in overlapping regions. This testing strategy is also
used in other image processing algorithms [48, 3, 2] to alleviate memory
issues.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
(a) Whole input image |  (b) Input |  (c) w/o color |  (d) w/o raw |  (e) Our full model |  (f) GT
Figure 10: Visual comparisons of different variants of the proposed network. “w/o color” indicates the model without the color correction module. “w/o raw” represents super-resolution without raw input by using the low-resolution color image as the input for both modules of our network.  TABLE II:  Ablation study of the proposed model on the synthetic dataset.  Methods | Blind | Non-blind
---|---|---
PSNR | SSIM | PSNR | SSIM
w/o color module | 22.04 | 0.7426 | 22.13 | 0.7599
w/o raw input | 30.21 | 0.7864 | 30.65 | 0.7948
Our full model | 31.06 | 0.8088 | 32.07 | 0.8314

Model training. We use the $L_{1}$ loss function to train the network and
adopt the Adam optimizer [59] with initial learning rate of $2\times 10^{-4}$.
Let $\\{X^{(i)}_{raw},X^{(i)}_{ref},X^{(i)}_{lin},X^{(i)}_{gt}\\}_{i=1}^{N}$
denotes our dataset which has $N$ training samples. We first pre-train the
image restoration module using
$\frac{1}{N}\sum_{i=1}^{N}\|\tilde{X}^{(i)}_{lin}-X^{(i)}_{lin}\|_{1}$ for
$4\times 10^{4}$ iterations with a fixed learning rate. Then we train the
whole network using
$\frac{1}{N}\sum_{i=1}^{N}\|\tilde{X}^{(i)}-X^{(i)}_{gt}\|_{1}$ for $8\times
10^{4}$ iterations, where we decrease the learning rate by a factor of $0.96$
every $2\times 10^{3}$ updates during the first $4\times 10^{4}$ iterations,
and then reduce it to a lower rate of $10^{-5}$ for the latter $4\times
10^{4}$ iterations.

Baseline methods. We evaluate our model with the state-of-the-art super-
resolution methods [2, 3, 7, 8] which take the degraded color image as input,
and the learning-based raw image processing algorithms [19, 20] that use low-
resolution raw data. As the raw image processing networks [19, 20] are not
designed for super-resolution, we add deconvolution layers to increase the
resolution of the output. All the baseline models are trained on our dataset
with the same settings.

###  4.2 Evaluation with the state-of-the-arts

We quantitatively evaluate our method using the test dataset described above.
As shown in Table I, the proposed algorithm performs favorably against the
baseline methods in terms of PSNR and structural similarity (SSIM) [60].
Figures 8 shows some restored results from our synthetic dataset. Since the
low-resolution raw input does not have the relevant information for color
conversion and tone adjustment carried out within the ISP, the raw image
processing methods [19, 20] can only approximate color corrections for one
specific camera, and thereby do not recover desirable color appearances for
diverse camera types in our test set (Figure 8(c)). In addition, the training
process of the raw processing models is negatively affected by the significant
color differences between the predictions and ground truth, and the network
cannot focus on restoring sharp structures due to the distraction of the color
errors. Thus, the restored images in Figures 8(c) are still blurry. Figure
8(d)-(e) show that the super-resolution methods with low-resolution color
image as input can generate clear results with correct colors. However, some
fine details are missing. In contrast, our method generates better results
with appealing color appearances as well as sharp structures and fine details,
as shown in Figure 8(f), by exploiting both the color input and raw radiance
information.  For a more comprehensive study, we improve the two
representative baselines SID [19] and RDN [8] by feeding both the raw and
color images into these models. As shown in Table I, this simple modification
leads to a significant performance boost for both the SID and RDN, which
demonstrates the effectiveness of exploiting the complementary information
from the two inputs. On the other hand, our network still outperforms the
improved baselines by a clear margin, which shows the effectiveness of the
proposed two-branch model.

Non-blind super-resolution. Since most super-resolution methods [2, 3, 8] are
non-blind with the assumption that the downsampling blur kernel is known and
fixed, we also evaluate our algorithm for this case by using fixed defocus
kernel with radius of 5 pixels to synthesize training and test datasets. As
shown in Table I and Figure 9, the proposed algorithm can also achieve better
results than the baseline methods under the non-blind settings.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
(a) Whole input image |  (b) Input |  (c) SID |  (d) SRDenseNet |  (e) RDN |  (f) Ours
Figure 11: Comparison with the state-of-the-arts on real-captured images.
Since the outputs are of ultra-high resolution, spanning from 6048$\times$8064
to 12416$\times$17472, we only show image patches cropped from the tiny green
boxes in (a). The input images from top to bottom are captured by Sony, Canon,
iPhone 6s Plus, and Nikon cameras, respectively.

###  4.3 Ablation study

To thoroughly evaluate the proposed algorithm, we experiment with two variants
of our model by removing different components. We show the quantitative
results on the synthetic datasets in Table II and provide qualitative
comparisons in Figure 10 with detailed explanations.

Color correction. As shown in Figure 10(c), without the color correction
module, the model directly uses the low-resolution raw input for super-
resolution, which cannot effectively recover high-fidelity colors for
different cameras. Furthermore, it cannot learn sharp structures during
training due to the distractions of the significant color differences similar
to the raw-processing methods (_e.g_., SID [19] and DeepISP [20]).

Raw input. To evaluate the importance of the raw information, we use the low-
resolution color image as the input for both modules of the proposed network,
where we accordingly change the packing-raw layer of the image restoration
module. Figure 10(d) shows that the model without raw input cannot generate
clear structures or fine details due to the information loss in the ISP. In
contrast, our full model effectively integrates different components, and
generates sharper results with more details and fewer artifacts in Figure
10(e) by exploiting the complementary information in the raw data and
processed color images.

TABLE III: Quantitative evaluations on the proposed real image dataset with blind image quality evaluation metrics [61, 62, 63]. Smaller values indicate better image qualities for DIIVINE [61] and BRISQUE [62], and worse image qualities for NIMA [63].  Method |  DIIVINE $\downarrow$ |  BRISQUE $\downarrow$ |  NIMA $\uparrow$
---|---|---|---
SID [19] | 55.03 | 48.75 | 4.452
SRDenseNet [7] | 52.94 | 48.13 | 4.72
RDN [8] | 54.80 | 48.39 | 4.91
Ours | 46.83 | 46.08 | 4.94
|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
(a) Whole input image |  (b) Input |  (c) RDN with pre. data |  (d) RDN with our data |  (e) Ours with pre. data |  (f) Ours with our data
Figure 12: Qualitative evaluations of the proposed data generation pipeline on
real images. The images from top to bottom are captured by Pantax, Canon,
Nikon, and iPad cameras, respectively.

###  4.4 Evaluation on real images

To evaluate our algorithm on real images, we propose a new test dataset (no
overlap with the training set) by collecting diverse real-captured raw data
paired with the corresponding color images from the Internet. There are 100
image pairs captured by different cameras including Nikon, Sony, Canon, Leica,
Pentax, Fuji, and Apple (iPad Pro and iPhone 6s Plus). The resolution of these
images spans from 3024$\times$4032 to 6208$\times$8736\. Several example
images from this dataset are shown in Figure 1(a), 11(a), and 12(a).

We first present qualitative comparisons between the baselines [19, 7, 8] and
our method. As shown in Figure 11(c)-(e), both the state-of-the-art raw-
processing and super-resolution approaches do not perform well on real inputs
and generate incorrect colors and oversmoothing artifacts. In contrast, the
proposed algorithm (Figure 11(f)) generates better results with sharper edges
and finer details. Although the operations within the ISPs for the real images
are unknown to our network, the proposed algorithm can effectively recover
high-fidelity color appearances with the color correction module.

We also quantitatively evaluate these algorithms on the real image dataset
using the blind image quality evaluation metrics: Distortion Identification-
based Image Verity and Integrity Evaluation (DIIVINE) [61],
Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [62], and Neural
Image Assessment (NIMA) [63]. As the images are too large (especially after
super-resolution), it is computationally infeasible to compute these blind
metrics on the whole outputs. Thus, we crop 5 patches of 512$\times$512 pixels
from the center and four corners of each input image, and use this subset of
500 patches for quantitative evaluation. Table III shows that our method
performs favorably against the baseline approaches on all the metrics. These
results are consistent with the visual comparisons in Figure 11 and further
demonstrate the effectiveness of our method.

TABLE IV: Effectiveness of the proposed data generation pipeline. The models are trained with different datasets and assessed using the blind image quality evaluation metrics [61, 62, 63]. “Our data” is generated by the proposed pipeline in Section 3.1, whereas “previous data” is generated by bicubic downsampling, homoscedastic Gaussian noise and nonlinear space mosaicing [2, 9, 44]. “Our data∗” represents a variant of our data generation pipeline by performing the realistic image degradations in the nonlinear color image space. Smaller values indicate better image qualities for DIIVINE [61] and BRISQUE [62], and worse image qualities for NIMA [63].  Method |  DIIVINE $\downarrow$ |  BRISQUE $\downarrow$ |  NIMA $\uparrow$
---|---|---|---
RDN w/ previous data | 59.11 | 50.26 | 4.74
RDN w/ our data | 54.80 | 48.39 | 4.91
Our model w/ previous data | 59.01 | 48.23 | 4.73
Our model w/ our data∗ | 47.51 | 47.09 | 4.91
Our model w/ our data | 46.83 | 46.08 | 4.94
---
Figure 13: User study of different super-resolution algorithms on real images. Each bar shows the mean and standard deviation of the normalized scores of each method. “RDN-p” and “Ours-p” represents the RDN [8] and our network trained on previous data, respectively. |  |  |  |  |
---|---|---|---|---|---
(a) Whole input image |  (b) Input |  (c) w/o ED |  (d) w/o CA |  (e) Ours |  (f) GT
Figure 14: Qualitative evaluations of the image restoration module. “ED” and “CA” represent the encoder-decoder structure and channel attention, respectively. Note the blurry shadows in the blank regions of (d).  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
(a) Whole input image |  (b) Input |  (c) Global |  (d) GF |  (e) ETGF |  (f) w/o FF |  (g) Ours |  (h) GT
Figure 15: Visual comparison of different methods for the color correction
module. (c) “Global” represents the global color correction [20]. (d) “GF”
indicates the guided filtering [23]. (f) “w/o FF” is our learning-based guided
filter without using feature fusion.

##  5 Analysis and Extension

In this section, we demonstrate the effectiveness of our data generation
pipeline, and the image restoration as well as color correction modules. We
then present the results of a user study for more comprehensive evaluation.
Further, we show that our model can be well extended to higher upsampling
factors. Finally, we show that the proposed network is able to generalize to
more complex color corrections.

###  5.1 Effectiveness of the data generation pipeline

We use the real image dataset to evaluate our data generation method. As shown
in Figure 1(d) and 12(d), the results of RDN become sharper after re-training
with the data generated by our pipeline. On the other hand, the proposed two-
branch CNN model cannot generate clear images (Figure 12(e)) when trained on
previous data synthesized by bicubic downsampling, homoscedastic Gaussian
noise and nonlinear space mosaicing [2, 9, 44]. In contrast, our method
generates images with sharper edges and finer details by using the training
data from our data generation pipeline as shown in Figure 1(e) and 12(f).

We present quantitative results in Table IV with the blind image quality
metrics [61, 62, 63]. Both RDN [8] and the proposed model obtain significant
improvements in terms of DIIVINE [61], BRISQUE [62], and NIMA [63] by training
the networks on our synthetic dataset, which shows the effectiveness of the
proposed data generation approach. We also present a variant of our data
generation pipeline (“our data∗”) by performing the realistic image
degradation operations in the nonlinear color image space, which leads to
inferior results than our full data generation pipeline and further
demonstrates the importance of the raw-space data synthesis scheme.

TABLE V:  Quantitative evaluation of the image restoration module network. “ED” and “CA” represent the encoder-decoder structure and channel attention, respectively.  Methods | Blind | Non-blind
---|---|---
PSNR | SSIM | PSNR | SSIM
w/o ED | 30.68 | 0.8016 | 31.31 | 0.8202
w/o CA | 30.90 | 0.8065 | 31.94 | 0.8294
Ours | 31.06 | 0.8088 | 32.07 | 0.8314
TABLE VI:  Quantitative evaluation of the proposed learning-based guided filter.  Methods | Blind | Non-blind
---|---|---
PSNR | SSIM | PSNR | SSIM
Guided filtering [23] | 29.86 | 0.7816 | 29.99 | 0.7864
Global color correction [20] | 30.18 | 0.8013 | 30.72 | 0.8161
ETGF [56] | 30.56 | 0.8026 | 31.65 | 0.8235
w/o feature fusion | 30.82 | 0.8049 | 31.85 | 0.8278
Ours | 31.06 | 0.8088 | 32.07 | 0.8314

###  5.2 User study

We conduct a user study for more comprehensive evaluation of the super-
resolution algorithms. This study uses 10 images randomly selected from the
real image dataset, and each low-resolution input is upsampled by 6 different
methods: SID [19], SRDenseNet [7], RDN with previous data [8], RDN with our
data, our model with previous data, and our model with our data. Twenty
subjects are asked to rank the reconstructed images by each method with the
input image as reference (1 for poor and 6 for excellent). We normalize the
rank values to $[0,1]$, and use the normalized scores for measuring image
quality in Figure 13, where we visualize the mean scores and the standard
deviation. The user study shows that the proposed method can generate results
with high perceptual image quality for real-captured inputs. In addition, our
data generation pipeline is effective in synthesizing realistic training data
and significantly improves the performance of both RDN and the proposed
network.

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
(a) Whole input image |  (b) Input |  (c) DehazeNet |  (d) GFN |  (e) GridDehazeNet |  (f) Ours |  (g) GT
Figure 16: Qualitative evaluation for image dehazing. Compared with the existing dehazing algorithms [64, 65, 66], our method can better remove haze from the input and generate fewer artifacts.  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |
(a) Guidance |  (b) Input | (c) GF |  (d) JBU |  (e) TGV |  (f) Park |  (g) DJF |  (h) Ours |  (i) GT
Figure 17: Qualitative evaluation for guided depth upsampling. We compare our
method against existing algorithms [23, 67, 68, 69, 70] for a scaling factor
of 8.

###  5.3 Effectiveness of the image restoration module

The proposed image restoration module differs from the existing networks [2,
3, 7, 8] in two aspects. First, while existing super-resolution algorithms
often learn the nonlinear mapping functions at the same scale [2, 3, 7, 8], we
use the deep convolutional blocks in an encoder-decoder manner similar to the
U-net [52]. As shown in Figure 14(c), simply applying the blocks at one scale
cannot effectively exploit multi-scale features and may lead to lower-quality
results. Second, we introduce the channel attention mechanism to improve the
dense blocks [21, 7] by recalibrating the naively concatenated features. The
proposed DCA blocks can increase the representational capacity of the image
restoration model and facilitate learning better nonlinear mapping functions.
As shown in Figure 14, using the DCA block effectively reduces artifacts of
the result produced by the model without channel attention (Figure 14(d)) and
generates a clearer image in Figure 14(e). In addition, we also present
quantitative evaluation results in Table V. Both the encoder-decoder and
channel attention strategies effectively improve the super-resolution
performance.

Image dehazing. The proposed image restoration module can also be applied to
other image processing tasks, such as image dehazing. As the input and output
of the dehazing task have the same image resolution, we remove the packing raw
and sub-pixel layers in Figure 5. Similar to the GridDehazeNet [66], we train
and test the proposed network using the RESIDE benchmark dataset [71]. Table
VII shows that our method performs favorably against the state-of-the-art
dehazing algorithms [72, 64, 73, 74, 65, 66]. We present some visual examples
from the RESIDE test set in Figure 16 for qualitative comparisons. Our image
restoration module can better remove thick haze from the input and produce
fewer artifacts as shown in Figure 16(f).

TABLE VII: Image dehazing results on the test sets of RESIDE [71]. Numbers in bold indicate the best performance, and underscored numbers indicate the second best.  Methods | Indoor | Outdoor
---|---|---
PSNR | SSIM | PSNR | SSIM
DCP [72] | 16.61 | 0.8546 | 19.14 | 0.8605
DehazeNet [64] | 19.82 | 0.8209 | 24.75 | 0.9269
MSCNN [73] | 19.84 | 0.8327 | 22.06 | 0.9078
AOD-Net [74] | 20.51 | 0.8162 | 24.14 | 0.9198
GFN [65] | 24.91 | 0.9186 | 28.29 | 0.9621
GridDehazeNet [66] | 32.16 | 0.9836 | 30.86 | 0.9819
Ours | 31.91 | 0.9778 | 33.05 | 0.9834
|  |  |  |  |  |
---|---|---|---|---|---|---
(a) Whole input image |  (b) Input |  (c) dense + sFF |  (d) DCA + sFF |  (e) dense + FF |  (f) DCA + FF |  (g) GT
Figure 18: Visual comparison against the model of [22]. “dense” denotes the dense block. “sFF” and “FF” respectively represent the single-scale and multi-scale feature fusion. The proposed DCA block can reduce artifacts better than the dense blocks and achieve clearer edges (yellow arrow). The multi-scale feature fusion can generate more vivid colors closer to the ground truth (green arrow). Note that (c) is the model of [22], and (f) is the full model of this work.  |  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
(a) Whole input image |  (b) Input |  (c) SID |  (d) SRDenseNet |  (e) RDN |  (f) Ours |  (g) GT
Figure 19: Qualitative evaluation of the proposed method for 4$\times$ image
super-resolution. The first and second examples are from the blind and non-
blind image datasets, respectively.

###  5.4 Effectiveness of the color correction module

As introduced in Section 3.2.2, we propose a learning-based guided filter for
the color correction module, which helps address the oversmoothing issue of
the guided filter [23]. There are two existing approaches closely related to
our work, _i.e_., global color correction [20] and ETGF [56], which we
evaluate in Figure 15 and Table VI and analyze as follows.

First, simply adopting the global color correction strategy from [20] can only
recover holistic color appearances, such as in the background region of Figure
15(c), but there are still significant distortions in local regions without
the pixel-wise transformations of the proposed method. Second, the ETGF [56]
method learns better guidance maps with CNNs (formulated in (8)) than the
guided filter, and generates high-fidelity colors as shown in Figure 15(e).
However, the ETGF method is still limited by the local constant assumption,
and thus cannot adapt to high-frequency color changes as well as our learning-
based guided filter. In contrast, the learning-based guided filter learns
pixel-wise transformation matrix for spatially-variant color corrections, and
recovers high-fidelity color appearances without the local constant constraint
(Figure 15(g)). In addition, our method achieves significant quantitative
improvement over the baseline models as shown in Table VI. Note that the
proposed learning-based guided filter relies on the information of the first
module for color transformation estimation. The network without feature fusion
cannot predict accurate transformation matrix and tends to bring color
artifacts around subtle structures in Figure 15(f).

TABLE VIII:  Evaluations of guided depth upsampling algorithms in terms of RMSE on the NYUv2 test set [75]. The depth values are measured in centimeter similar to [70]. Our method can achieve high-quality results for all the 4$\times$, 8$\times$, and 16$\times$ depth upsampling.  Methods |  4$\times$ |  8$\times$ |  16$\times$
---|---|---|---
GF [23] | 7.32 | 13.62 | 22.03
JBU [67] | 4.07 | 8.29 | 13.35
TGV [68] | 6.98 | 11.23 | 28.13
Park [69] | 5.21 | 9.56 | 18.10
Ham [76] | 5.27 | 12.31 | 19.24
DJF [70] | 3.38 | 5.86 | 10.11
Ours | 2.85 | 5.49 | 9.23

Guided depth upsampling. The proposed color correction network can also be
used for other joint image filtering tasks, such as guided depth upsampling.
We apply our network to the depth upsampling problem in which the input is a
low-resolution depth map paired with a corresponding high-resolution RGB
image, which respectively correspond to $X_{ref}$ and $\tilde{X}_{lin}$ in
(5). Since there is no image restoration network here, we do not use the
feature fusion strategy and instead directly feed the depth map and RGB image
into the color correction module to estimate the transformation matrix. The
predicted transformations are applied to the high-resolution RGB image to
generate the desired high-resolution depth map as in (5). Intuitively, our
method is in spirit similar to the guided filter [23] which uses the high-
resolution image to approximate the coarse depth such that the output is in
the same modality as the input depth while preserving the sharp structures of
the high-resolution image. More details about the network structure are
presented on the project web page:
https://sites.google.com/view/xiangyuxu/rawsr_pami. Similar to DJF [70], we
use the NYUv2 dataset [75] for training and testing. Table VIII shows that the
proposed method performs favorably against the state-of-the-art guided depth
upsampling approaches [23, 67, 68, 69, 76, 70]. In addition, we present visual
examples from the NYUv2 test set in Figure 17. Overall, the proposed method
can better exploit the structures of the RGB image by learning spatially-
variant transformations and generate higher-quality depth maps.

|
---|---
(a) Input processed by gamma correction |  (b) Input processed by [77]
|
(c) Super-resolution with (a) |  (d) Super-resolution with (b)
Figure 20: Generalization of the proposed method to different color correction operations. Our method can generate high-quality super-resolution results using the reference images produced by both gamma correction (a) and local Laplacian filter [77] (b).  |  |  |  |
---|---|---|---|---
Input 1 |  Input 2 |  Input 3 |  Input 4 |  Input 5
|  |  |  |
Output 1 |  Output 2 |  Output 3 |  Output 4 |  Output 5
Figure 21: Generalization of the proposed method to manually retouched color images from the MIT-Adobe 5K dataset [14]. The proposed method generates clear structures and high-fidelity colors.  TABLE IX:  Evaluations with our prior model [22] on the synthetic dataset. “dense” denotes the dense block. “sFF” and “FF” respectively represent the single-scale and multi-scale feature fusion. “dense + sFF” is the model of [22], and “DCA + FF” is the full model of this work.  Methods | Blind | Non-blind
---|---|---
PSNR | SSIM | PSNR | SSIM
dense + sFF | 30.79 | 0.8044 | 31.79 | 0.8272
DCA + sFF | 30.93 | 0.8068 | 31.97 | 0.8293
dense + FF | 30.90 | 0.8065 | 31.94 | 0.8294
DCA + FF | 31.06 | 0.8088 | 32.07 | 0.8314

###  5.5 Evaluation with our prior model

There are also two main differences between the models in this work and [22].
First, we propose the DCA block (see Section 5.3) which performs better than
the dense blocks in [22]. As shown in Figure 18(c) and (d) (the yellow arrow),
the DCA blocks can reduce artifacts and generate clearer edges for image
restoration. Second, we apply the feature fusion for the color correction
module in a multi-scale manner whereas [22] only fuses feature at one scale.
As shown in Figure 18(c) and (e) (the green arrow), the multi-scale feature
fusion can generate more vivid colors closer to the ground-truth (Figure
18(g)) than single-scale feature fusion. Finally, combining the above methods
can further improve the performance qualitatively and quantitatively as shown
in Figure 18(f) and Table IX.

In addition, we adapt the network structure for 4$\times$ super-resolution by
changing the factor of the sub-pixel layer in the first module and adding a
deconvolution layer to the second module. We also change the downsampling
factor of the proposed data generation pipeline in (1) accordingly. We train
the proposed model with the same settings as described in Section 4.1. As
shown in Table X and Figure 19, our method can also generate high-quality
results on 4$\times$ image upsampling both quantitatively and qualitatively.

TABLE X: Quantitative evaluations for 4$\times$ super-resolution on the synthetic dataset. “Blind” represents the images with variable blur kernels, and “Non-blind” denotes fixed kernel.  Methods | Blind | Non-blind
---|---|---
PSNR | SSIM | PSNR | SSIM
SID [19] | 21.34 | 0.7043 | 21.35 | 0.6993
DeepISP [20] | 21.06 | 0.7015 | 21.22 | 0.7015
SRCNN [2] | 27.60 | 0.7293 | 28.29 | 0.7413
VDSR [3] | 27.98 | 0.7367 | 28.55 | 0.7500
SRDenseNet [7] | 28.20 | 0.7418 | 28.69 | 0.7536
RDN [8] | 28.60 | 0.7481 | 29.08 | 0.7570
Ours | 29.74 | 0.7739 | 30.27 | 0.7871

###  5.6 Generalization to more complex color corrections

While the ISP for our synthetic data is simulated with the Dcraw [46] toolkit,
the proposed super-resolution algorithm can be generalized to more complex
color corrections. As one typical example, while the gamma correction is used
for tone adjustment in the Dcraw toolkit, modern camera systems often use more
complex algorithms [77] to deal with more challenging inputs, such as high
dynamic range (HDR) images. In Figure 20(a) and (b), we process the HDR image
by gamma correction and the local Laplacian filter [77] separately. We
evaluate the proposed network with the same raw input but different reference
images (see Figure 20(a) and (b)). Despite the drastic color change between
them, our super-resolution algorithm consistently generates clear results with
high-fidelity color appearances as shown in Figure 20(c) and (d) without
knowing the original ISP operations.

Human retouched images. In addition to the HDR images, we carry out
experiments on images manually retouched by photographers from the MIT-Adobe
5K dataset [14] to simulate more diverse ISPs. We evaluate our model on the
same raw input with different reference images retouched by different
photographers in Figure 21. The proposed algorithm restores clear structures
as well as high-fidelity colors, which shows that our method is able to
generalize to more diverse and complex ISPs.

TABLE XI:  Run-time of super-resolution algorithms on a 800$\times$600 image.  Method | DeepISP | VDSR | SRDenseNet | RDN | Ours
---|---|---|---|---|---
Time (s) | 0.2614 | 0.6892 | 0.9448 | 0.4987 | 0.647
---
Figure 22: Training the super-resolution model with GAN loss.
---
Figure 23: Failure case. The proposed method does not work well under severe
noise.

###  5.7 Adversarial training

As introduced in Section 4.1, we use the $L_{1}$ loss for training the
proposed network. A potential alternative is to use the adversarial training,
for which we employ a combination of the perceptual loss [32] and GAN loss
[37] following the SRGAN [33]. As shown in Figure 22, the generated image
based on this training loss appears sharper at first glance. However, upon
close inspection, the generated image is of low quality with unpleasant
artifacts and details not present in the ground truth. Therefore, we do not
use the GAN loss in our default settings.

###  5.8 Run-time

We evaluate run-time of different super-resolution methods on the same machine
with an Intel i7 CPU and a Nvidia GTX 1060 GPU. Although there are two modules
in the proposed model, our method has similar run-time as existing networks as
shown in Table XI. This is mainly due to the encoder-decoder formulation of
the first module which processes the image features at a smaller scale, and
the light-weight structure of the second module.

###  5.9 Practicality

The proposed algorithm requires both the raw data and the processed color
image as input. Most existing camera devices, such as SLRs and smartphones,
can generate the color image in a considerably efficient way, which makes the
proposed algorithm applicable for most users. For the cases where the color
input is not available, the users can process the raw data with their own ISP
tools, _e.g_., Dcraw [46] and Photoshop, before applying the proposed super-
resolution network. Further, it will be an interesting direction to explore
the combination of the ISP system and the super-resolution pipeline for saving
computation in future research.

###  5.10 Limitations

As introduced in Section 4.1, we use moderate noise parameters in our noise
model (2) to synthesize the training data. When the input image has severe
noise, the proposed method is likely to fail as shown in Figure 23. Our future
work will focus on developing a network that explicitly considers noise
reduction in the super-resolution process.

##  6 Conclusions

In this work, we propose a novel pipeline to generate realistic training data
by simulating the real imaging process of digital cameras. We develop a two-
branch CNN to exploit the radiance information recorded in raw data. The
proposed algorithm performs favorably against the state-of-the-arts on the
synthetic dataset and enables high-quality super-resolution results in real
scenarios. This work shows the effectiveness of learning with raw data, and we
expect more applications of it in other related image processing problems.

## References

  * [1] J. E. Greivenkamp, _Field guide to geometrical optics_. SPIE Press Bellingham, WA, 2004, vol. 1.
  * [2] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in _ECCV_ , 2014.
  * [3] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _CVPR_ , 2016.
  * [4] R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored neighborhood regression for fast super-resolution,” in _ACCV_ , 2014.
  * [5] Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, “Deep networks for image super-resolution with sparse prior,” in _ICCV_ , 2015.
  * [6] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate superresolution,” in _CVPR_ , 2017.
  * [7] T. Tong, G. Li, X. Liu, and Q. Gao, “Image super-resolution using dense skip connections,” in _ICCV_ , 2017.
  * [8] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in _CVPR_ , 2018.
  * [9] X. Xu, D. Sun, J. Pan, Y. Zhang, H. Pfister, and M.-H. Yang, “Learning to super-resolve blurry face and text images,” in _ICCV_ , 2017.
  * [10] X. Xu, J. Pan, Y.-J. Zhang, and M.-H. Yang, “Motion blur kernel estimation via deep learning,” _TIP_ , vol. 27, pp. 194–205, 2018.
  * [11] T. Plotz and S. Roth, “Benchmarking denoising algorithms with real photographs,” in _CVPR_ , 2017.
  * [12] T. Brooks, B. Mildenhall, T. Xue, J. Chen, D. Sharlet, and J. T. Barron, “Unprocessing images for learned raw denoising,” in _CVPR_ , 2019.
  * [13] S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” _ACM Trans. Graph. (SIGGRAPH Asia)_ , vol. 35, no. 6, 2016.
  * [14] V. Bychkovsky, S. Paris, E. Chan, and F. Durand, “Learning photographic global tonal adjustment with a database of input/output image pairs,” in _CVPR_ , 2011.
  * [15] W. B. Pennebaker and J. L. Mitchell, _JPEG: Still image data compression standard_. Springer Science & Business Media, 1992.
  * [16] R. M. Nguyen and M. S. Brown, “Raw image reconstruction using a self-contained srgb-jpeg image with only 64 kb overhead,” in _CVPR_ , 2016.
  * [17] Y.-W. Tai, X. Chen, S. Kim, S. J. Kim, F. Li, J. Yang, J. Yu, Y. Matsushita, and M. S. Brown, “Nonlinear camera response functions and image deblurring: Theoretical analysis and practice,” _TPAMI_ , vol. 35, no. 10, pp. 2498–2512, 2013.
  * [18] R. Zhou, R. Achanta, and S. Süsstrunk, “Deep residual network for joint demosaicing and super-resolution,” in _Color and Imaging Conference_ , 2018\.
  * [19] C. Chen, Q. Chen, J. Xu, and V. Koltun, “Learning to see in the dark,” in _CVPR_ , 2018.
  * [20] E. Schwartz, R. Giryes, and A. M. Bronstein, “Deepisp: Towards learning an end-to-end image processing pipeline,” _TIP_ , 2018.
  * [21] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks.” in _CVPR_ , 2017.
  * [22] X. Xu, Y. Ma, and W. Sun, “Towards real scene super-resolution with raw images,” in _CVPR_ , 2019.
  * [23] K. He, J. Sun, and X. Tang, “Guided image filtering,” _TPAMI_ , vol. 35, pp. 1397–1409, 2013.
  * [24] W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-based super-resolution,” _IEEE Computer graphics and Applications_ , vol. 22, 2002\.
  * [25] H. Chang, D.-Y. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” in _CVPR_ , 2004.
  * [26] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” _TIP_ , vol. 19, no. 11, pp. 2861–2873, 2010.
  * [27] S. Wang, L. Zhang, Y. Liang, and Q. Pan, “Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis,” in _CVPR_ , 2012.
  * [28] C.-Y. Yang and M.-H. Yang, “Fast direct super-resolution by simple functions,” in _CVPR_ , 2013.
  * [29] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in _ECCV_ , 2016.
  * [30] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _TPAMI_ , vol. 38, pp. 295–307, 2016.
  * [31] J. Kim, J. Kwon Lee, and K. Mu Lee, “Deeply-recursive convolutional network for image super-resolution,” in _CVPR_ , 2016.
  * [32] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _ECCV_ , 2016.
  * [33] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” in _CVPR_ , 2017.
  * [34] W. Zhang, Y. Liu, C. Dong, and Y. Qiao, “Ranksrgan: Generative adversarial networks with ranker for image super-resolution,” in _ICCV_ , 2019.
  * [35] M. S. Sajjadi, B. Scholkopf, and M. Hirsch, “Enhancenet: Single image super-resolution through automated texture synthesis,” in _ICCV_ , 2017.
  * [36] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _ICLR_ , 2015.
  * [37] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _NIPS_ , 2014\.
  * [38] K. Zhang, W. Zuo, and L. Zhang, “Learning a single convolutional super-resolution network for multiple degradations,” in _CVPR_ , 2018.
  * [39] ——, “Deep plug-and-play super-resolution for arbitrary blur kernels,” in _CVPR_ , 2019.
  * [40] K. Zhang, L. V. Gool, and R. Timofte, “Deep unfolding network for image super-resolution,” in _CVPR_ , 2020.
  * [41] S. Farsiu, M. Elad, and P. Milanfar, “Multiframe demosaicing and super-resolution of color images,” _TIP_ , vol. 15, pp. 141–159, 2006.
  * [42] P. Vandewalle, K. Krichane, D. Alleysson, and S. Süsstrunk, “Joint demosaicing and super-resolution imaging from a set of unregistered aliased images,” in _Digital Photography III_ , 2007.
  * [43] H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” _TIP_ , vol. 26, pp. 5032–5042, 2017.
  * [44] M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” _ACM Transactions on Graphics (TOG)_ , vol. 35, p. 191, 2016\.
  * [45] D. Khashabi, S. Nowozin, J. Jancsary, and A. W. Fitzgibbon, “Joint demosaicing and denoising via learned nonparametric random fields,” _TIP_ , vol. 23, pp. 4968–4981, 2014.
  * [46] D. Coffin, “Dcraw: Decoding raw digital photos in linux,” http://www.cybercom.net/ dcoffin/dcraw/.
  * [47] M. Hradiš, J. Kotera, P. Zemcík, and F. Šroubek, “Convolutional neural networks for direct text deblurring,” in _BMVC_ , 2015\.
  * [48] C. Schuler, H. Burger, S. Harmeling, and B. Scholkopf, “A machine learning approach for non-blind image deconvolution,” in _CVPR_ , 2013.
  * [49] B. Mildenhall, J. T. Barron, J. Chen, D. Sharlet, R. Ng, and R. Carroll, “Burst denoising with kernel prediction networks,” in _CVPR_ , 2018.
  * [50] G. E. Healey and R. Kondepudy, “Radiometric ccd camera calibration and noise estimation,” _TIP_ , vol. 16, pp. 267–276, 1994.
  * [51] K. Hirakawa and T. W. Parks, “Adaptive homogeneity-directed demosaicing algorithm,” _TIP_ , vol. 14, pp. 360–369, 2005.
  * [52] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_ , 2015.
  * [53] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in _CVPR_ , 2016\.
  * [54] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [55] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _CVPR_ , 2018.
  * [56] H. Wu, S. Zheng, J. Zhang, and K. Huang, “Fast end-to-end trainable guided filter,” in _CVPR_ , 2018.
  * [57] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _AISTATS_ , 2010.
  * [58] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in _ICML_ , 2013.
  * [59] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2014.
  * [60] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli _et al._ , “Image quality assessment: from error visibility to structural similarity,” _TIP_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [61] A. K. Moorthy and A. C. Bovik, “Blind image quality assessment: From natural scene statistics to perceptual quality,” _TIP_ , vol. 20, no. 12, pp. 3350–3364, 2011.
  * [62] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” _TIP_ , vol. 21, no. 12, pp. 4695–4708, 2012.
  * [63] H. Talebi and P. Milanfar, “Nima: Neural image assessment,” _TIP_ , vol. 27, no. 8, pp. 3998–4011, 2018.
  * [64] B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “Dehazenet: An end-to-end system for single image haze removal,” _TIP_ , vol. 25, no. 11, pp. 5187–5198, 2016\.
  * [65] W. Ren, L. Ma, J. Zhang, J. Pan, X. Cao, W. Liu, and M.-H. Yang, “Gated fusion network for single image dehazing,” in _CVPR_ , 2018.
  * [66] X. Liu, Y. Ma, Z. Shi, and J. Chen, “Griddehazenet: Attention-based multi-scale network for image dehazing,” in _ICCV_ , 2019.
  * [67] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” in _ACM Transactions on Graphics (TOG)_ , vol. 26, no. 3, 2007, p. 96.
  * [68] D. Ferstl, C. Reinbacher, R. Ranftl, M. Rüther, and H. Bischof, “Image guided depth upsampling using anisotropic total generalized variation,” in _ICCV_ , 2013.
  * [69] J. Park, H. Kim, Y.-W. Tai, M. S. Brown, and I. Kweon, “High quality depth map upsampling for 3d-tof cameras,” in _ICCV_ , 2011.
  * [70] Y. Li, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Joint image filtering with deep convolutional networks,” _TPAMI_ , vol. 41, no. 8, pp. 1909–1923, 2019.
  * [71] B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, “Benchmarking single-image dehazing and beyond,” _TIP_ , vol. 28, no. 1, pp. 492–505, 2019\.
  * [72] K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” _TPAMI_ , vol. 33, no. 12, pp. 2341–2353, 2010.
  * [73] W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in _ECCV_ , 2016\.
  * [74] B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, “Aod-net: All-in-one dehazing network,” in _ICCV_ , 2017.
  * [75] P. K. Nathan Silberman, Derek Hoiem and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _ECCV_ , 2012.
  * [76] B. Ham, M. Cho, and J. Ponce, “Robust image filtering using joint static and dynamic guidance,” in _CVPR_ , 2015.
  * [77] S. Paris, S. W. Hasinoff, and J. Kautz, “Local laplacian filters: Edge-aware image processing with a laplacian pyramid.” _ACM Trans. Graph._ , vol. 30, no. 4, p. 68, 2011.