# Learning Regularized Multi-Scale Feature Flow for
High Dynamic Range Imaging

Qian Ye1, Masanori Suganuma1,2, Jun Xiao3, Takayuki Okatani1,2
1 Graduate School of Information Sciences, Tohoku University
2 RIKEN Center for AIP
3 Department of Electronic and Information Engineering, The Hong Kong
Polytechnic University
{qian, suganuma<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Reconstructing ghosting-free high dynamic range (HDR) images of dynamic scenes
from a set of multi-exposure images is a challenging task, especially with
large object motion and occlusions, leading to visible artifacts using
existing methods. To address this problem, we propose a deep network that
tries to learn multi-scale feature flow guided by the regularized loss. It
first extracts multi-scale features and then aligns features from non-
reference images. After alignment, we use residual channel attention blocks to
merge the features from different images. Extensive qualitative and
quantitative comparisons show that our approach achieves state-of-the-art
performance and produces excellent results where color artifacts and geometric
distortions are significantly reduced.

##  1 Introduction

High dynamic range (HDR) imaging is a method to generate a larger dynamic
range of illumination than standard imaging systems. It has been applied to
movies [29] and computer rendering [1] to gain more information and better
visual experience. As cameras that can capture HDR images are generally
expensive, an alternative way to get HDR images is to reconstruct HDR images
from a series of low dynamic range (LDR) images captured by a standard camera
with different exposure settings. While they can reconstruct high-quality HDR
images for static scenes, the existing methods tend to yield images with many
ghosting artifacts for dynamic scenes, in which the imaging scenes are static
captured by a hand-held camera or there are some moving objects.

Increasing efforts have been invested in exploring how to remove ghosting
artifacts in the multi-exposure-based HDR reconstruction. There are several
methods that attempt to detect motion regions in the input LDR images and then
remove these regions in the step of merging the images [17, 11, 38]. However,
they tend to work well only when the motion in the input images is relatively
small. When there are large motions, a large number of image pixels need to be
removed, which results in incorrect reconstruction because the information
about these pixels is lost.

Figure 1: Examples of generated HDR images from the test set of [14]. The
zoomed regions of different methods are highlighted.

Another approach has also been studied, which is to align input LDR images to
a reference view, and then merge them altogether for HDR image reconstruction
[2, 12, 10]. Many recent methods employ convolutional neural network (CNN) to
improve reconstruction image quality. However, there is still room for
improvements in the area of ghosting artifacts. End-to-end learning-based
approaches such as [35, 37, 40] without implicit alignment directly feed LDR
images into a network to reconstruct HDR images, failing to deal with
scenarios with complex motion or large disparity. As shown in Fig. 1, these
end-to-end leaning-based methods fails to deal with the motion region. The
method in [14] performs optical flow-based image alignment followed by a
convolutional neural network at the merging process. Aligning images in the
pixel domain is often prone to noisy or saturated pixels-induced misalignment,
which leads to visible artifacts in the final synthesized presentation. In
addition, the classical optical flow methods and the optical flow models pre-
trained on other datasets can not deal with the occlusion region in which the
ghosting artifacts often occur. As suggested in [4], feature warping can
achieve better performance compared with warping the image. The method in [27]
performs alignment in feature domain by using deformable convolution layers
[45]. However, it has a limitation in finding long-distance correspondence; as
argued in [20], deformable convolution could also lead to an unstable training
process and limited generalization. Inspired by non-local structure [34], Choi
_et al_. [5] proposed to calculate the inter-similarity between LDR images for
every pixel, which are used to align non-reference features toward the
reference feature. However, this non-local structure-based operation is
computationally expensive and needs large memory when the size of input images
is large, while the images for HDR imaging often have a large size. According
to [36], Task-Oriented flow learns to handle occlusions well, though its
estimated motion field differs from the ground truth optical flow. Considered
the benefit from Task-Orient flow, Kalantari _et al_. [15] proposed a Task-
Oriented flow network which is specifically designed for HDR video
reconstruction and is only based on the loss for HDR video reconstruction.
This Task-Oriented flow network performs better than pre-trained or classical
optimization-based optical flow methods since it can deal with the occlusion,
which reduces the artifacts in occlusion regions. However, only trained on the
task-specific loss (_i.e_. HDR reconstruction loss), the Task-Oriented flow
will fail on the large saturated areas in which there are a few details. In
this case, this misalignment leads to artifacts on the over-exposed regions in
the reference image.

Inspired by the photometric loss [41] for self-supervised learning of optical
flow, we proposed the regularized loss to provide supervision for flow
learning to address the misalignment in Task-Orient flow in HDR imaging. We
directly reconstruct an HDR image based on the aligned features and compute
loss between this reconstructed image and the corresponding ground truth.

Differing from the previous methods [14, 15, 26] that use the existing optical
flow models like SPyNet [28] and PWC-Net [32], we design a simple but
effective network for learning the flow for feature alignment. We remove the
context encoder in the flow network and directly use the features for HDR
image reconstruction as the input for flow estimation. We argue that this flow
structure can achieve better alignment performance in the HDR imaging task
since there are large illumination changes and large saturated areas in image
space, while the extracted features for HDR reconstruction can provide rich
information to avoid misalignment. And we name this flow structure as feature
flow.

To this end, we propose a new network that enables end-to-end training,
including alignment. The proposed method consists of two networks: alignment
network and merging network in this order. The alignment network extracts
multi-scale features from the input LDR images and estimates optical flow. It
then aligns the non-reference LDR images to the reference LDR image in feature
space using the estimated flow. The merging network takes the aligned features
and multi-scale features as input and generates a final HDR image using a
residual attention mechanism. Experimental results show that our method can
achieve better quantitative and qualitative evaluation performance than the
existing state-of-the-art methods on the commonly used public test datasets.

##  2 Related Works

###  2.1 Motion Removal based Methods

These methods is firstly to detect the motion region and then remove these
pixels on the motion region in the merging processing. Khan _et al_. [17] use
a non-parametric model to compute weights iteratively and apply these computed
weights to pixels to fuse multiple LDR images to obtain final HDR images. Heo
_et al_. [11] detect motion regions based on the joint probability densities
and refine these regions by using energy minimization. Jacobs _et al_. [13]
propose a method to detect moving pixels based on the difference between the
LDR images. Zhang _et al_. [42] propose motion detection method based on the
image gradients between different images. Lee _et al_. [19] considered that
the noise, moving objects, and distortions as outliers, so they proposed a
low-rank model to reconstruct HDR images. Following their method, Yan _et al_.
[38] proposed a sparse model to detect motion regions. When the motion in LDR
images is small, motion-removal-based methods can achieve satisfactory
results. However, when the motion is large, a large number of pixels are
unavoidably removed in the merging stage, causing undesirable artifacts in the
generated HDR images.

###  2.2 Alignment based Methods

Most alignment based methods adopt optical flow and its variants to align LDR
images and then merge aligned LDR images to generate corresponding HDR images.
Bogoni [2] use optical flow to estimate motion filed between LDR images and
then warped and aligned these LDR images by using the computed motion field.
Instead of fusing LDR images in the spatial domain, Kang _et al_. [16] firstly
utilize the information of exposure time and converted LDR images into
luminance domain. In the fusion process, a method was proposed to eliminate
artifacts by using the optical flow. Sen _et al_. [31] propose a method based
on a patch-matching algorithm for HDR reconstruction. Hu _et al_. [12] propose
a displacement estimation method which converts images by the intensity
mapping function and then merging images in the transformed domain for HDR
image generation, which implicitly align LDR images by searching and
aggregating similar patches. Hafner _et al_. [10] propose a method to jointly
estimate the optical flow and reconstruct HDR image. However, since the
alignment process in the image domain is vulnerable to large motion and
excessively dark or bright regions, these methods tend to generate artifacts
in the aligned images.

###  2.3 CNN Based Methods

As with other computer vision tasks, CNNs have been applied to HDR imaging.
Eilertsen _et al_. [7] propose an encoder-decoder network to generate an HDR
image from a single LDR image. Endo _et al_. [8] synthesize multiple LDR
images with different exposures from a single LDR image by CNNs, and then
merge them to reconstruct an HDR image. These single-image-based methods are
unable to reconstruct the textures on saturated regions accurately.

To generating more accurate images, more attention is paid on obtaining HDR
images from a sequence of LDR images captured with different exposures.
Kalantari _et al_. [14] propose the first CNN-based method for HDR imaging,
where the input LDR images are first aligned by optical flow and then the
aligned LDR images are fed to CNNs to reconstruct an HDR image. In stead of
using explicit alignment, Wu _et al_. [35] directly concantenated the features
extracted from input LDR images and forwarded them to a deep model with the
U-net structure to reconstruct HDR images. Yan _et al_. [39] introduce a non-
local structure [34] into the U-net as for implicit alignment. Yan _et al_.
[37] propose an attention module to learn to identify misaligned elements
before merging the LDR images. Pu _et al_. applied the deformable convolution
[45] to multi-scale features, which aligned LDR images in a pyramidal manner,
and reconstructed the corresponding HDR images. To reduce the computational
cost by CNN-based methods, Prabhakar _et al_. [25] propose an efficient method
that performs all operations in low resolution and upscales the result to the
required full resolution. Similar to our work, a few studies consider using
optical flow for the alignment. But they either use a pre-trained estimator
[26] or optimize the estimator through the reconstruction loss [15], which may
lead to suboptimal results.

Figure 2: Three LDR images of the same scene captured with three different
exposures. $L_{1}$, $L_{2}$, and $L_{3}$ denote the images captured with the
low, medium, and high exposure, respectively. Figure 3: Overview of the
proposed network. It consists of two sub-networks: feature alignment and
merging networks. The alignment network warps the features of the non-
reference images onto those of the reference image using optical flow. The
merging network takes the warped features as input and reconstructs an HDR
image.

##  3 Proposed Method

Given a series of LDR images, $L_{1},L_{2},...,L_{k}$, captured with different
exposures, the goal of HDR imaging is to generate an HDR image $H$
corresponding to a selected reference image $L_{r}$.

There are two samples with different exposure settings shown in Fig. 2. The
sample in the first row shows that $L_{3}$ has little effect for image
restoration of $L_{2}$ since there are large areas of over-exposure region in
$L_{3}$. While the sample in the second row shows that $L_{3}$ can be helpful
for image restoration of $L_{2}$. In this case, the model can easily produce
high-quality HDR images duo to the efficient information. Without the input of
$L_{3}$, the model can also generate a high-quality HDR image though the model
needs to be more effective.

Unlike the previous methods taking three LDR images as input, we use two LDR
images, $L_{1}$ and $L_{2}$ as inputs, sorted in the order of exposures and
set $L_{2}$ as the reference image by considering the properties in $L_{3}$.
And two images for input can also reduce computational costs.

Following the settings of previous studies [14, 37], we first map the LDR
images into the HDR domain using gamma correction and then feed them into the
network. To be specific, we map $L_{i}$ to $H_{i}$ by

$H_{i}=L_{i}^{\gamma}/t_{i},\;\;i=1,2,$ (1)

where $\gamma$ denotes the gamma correction parameter and followed [24] we use
$\gamma=2.2$ in this paper. $t_{i}$ is the exposure time of $L_{i}$. As the
suggestion in [14], we concatenate $L_{i}$ and $H_{i}$ in the channel
dimension to obtain a six-channel tensor $X_{i}=[L_{i},H_{i}],i=1,2$, and
input $X_{1}$ and $X_{2}$ to the network.

Our network consists of two sub-networks, the alignment network and merging
network, as shown in Fig.3. We first describe the alignment network (Sec.3.1)
and then explain the merging network (Sec.3.2).

###  3.1 Feature Alignment Network

The feature alignment network first extracts multi-scale feature maps from the
input tensor $X_{i}$. Specifically, the feature extractor consists of a
convolution layer with stride $=1$ and the following two layers with stride
$=2$, which forms multi-scale feature maps with scale $=0,1,$ and $2$. We
represent the feature map at scale $s\in\\{0,1,2\\}$ as
$F_{i}^{s}\in\mathbb{R}^{H_{s}\times W_{s}\times C_{s}}$, where
$H_{s}=H/2^{s}$, $W_{s}=H/2^{s}$, and $C_{s}=C$. These feature maps will be
used in the subsequent modules. For clarity, we use the index $r(=2)$ to
indicate the reference LDR images; thus, $X_{r}=X_{2}$ and
$F_{r}^{s}=F_{2}^{s}$ in what follows.

Figure 4: Architecture of the multi-scale feature flow module (MS-Flow). It
follows the coarse-fine manner to generates multi-scale optical flows and the
multi-scale feature maps aligned to the reference image.

####  3.1.1 Multi-Scale Feature Flow Module (MS-Flow)

Following SPyNet [28] and PWC-Net [32], we estimate the optical flow in a
coarse-to-fine manner, as shown in Fig. 4. The estimated flows at the coarser
scales can capture the large motions. On the other hand, the flows at the
finer scales will be helpful to capture small motions.

We first concatenate the coarsest scale features $F_{i}^{s}$ with $F_{r}^{s}$
in the channel dimension and feed it to a flow estimator. The estimator
consists of five convolution layers with $7\times 7$ kernel size and generates
the $s$-th scale optical flow $\mathbf{f}_{1\rightarrow r}^{s}$. Then, we
upsample $\mathbf{f}_{1\rightarrow r}^{s}$ by factor $=2$ and use it to warp
the non-reference feature map $F_{1}^{s-1}$ onto the reference feature map
$F_{r}^{s-1}$. Specifically, we map each pixel $\mathbf{p}_{1}^{s-1}$ in
$F_{1}^{s-1}$ to its estimated correspondence in $F_{r}^{s-1}$ as

$\mathbf{p}_{r}^{s-1}=\mathbf{p}_{1}^{s-1}+\mathbf{\tilde{f}}_{1\rightarrow
r}^{s}(\mathbf{p}_{1}^{s-1}),$ (2)

where $\mathbf{\tilde{f}}_{1\rightarrow r}^{s}$ represents the upsampled flow
of the $s$-th scale flow $\mathbf{f}_{1\rightarrow r}^{s}$. We then
concatenate the warped and reference feature maps in the channel dimension and
feed it to the subsequent flow estimator. The output of the flow estimator is
then element-wise added to the upsampled flow, yielding the flow at scale
$s-1$. We iterate this procedure for $s=1$ and $0$, obtaining multi-scale
optical flows $\mathbf{f}_{1\rightarrow r}^{s}$ and the warped multi-scale
feature maps $F_{i\rightarrow r}^{s}$.

####  3.1.2 Multi-Scale Feature Fusion Module (MS-Fuse)

As shown in Fig. 6, the multi-scale feature fusion module takes the
concatenated feature maps at each scale $\bar{F}^{s}=[F_{1\rightarrow
r}^{s},F_{r}^{s}]$. We apply a convolution layer with the kernel size of
$3\times 3$ followed by ReLU to the finest feature map $\bar{F}^{0}$ to obtain
a feature map $O^{0}$. For the feature maps $\bar{F}^{1}$ and $\bar{F}^{2}$,
we first apply a convolution layer with the same kernel size and then upsample
the outputs with bilinear interpolation so that the resulting maps become the
same size as the finest one. Finally, all the outputs are concatenated as
$Z_{0}=[O^{0},O^{1},O^{2}]$ and then used as input for the following merging
network.

Figure 5: Multi-scale feature fusion.

Figure 6: Reconstruction of $H_{of}$.

####  3.1.3 Reconstruction using Warped Features

Unlike the previous HDR imaging studies using optical flow, we reconstruct an
HDR image $H_{of}$ using the feature maps $F_{1\rightarrow r}^{s}$, which are
the feature maps right after warping by the optical flow. Our intention behind
this reconstruction is to directly guide the network to generate accurate
optical flow and perform better alignment. As shown in Fig. 6, we first
upsample the feature maps $F_{1\rightarrow r}^{1},F_{1\rightarrow r}^{2}$ so
as to be the same size as the finest feature map $F_{1\rightarrow r}^{0}$. We
then concatenate them and feed them to a series of convolution layers with the
kernel size of $3\times 3$ followed by ReLU and five residual channel
attention blocks (RCAB) [43]; see Fig. 7 for the detail of the RCAB. We then
calculate $\ell_{1}$ loss between the reconstructed HDR image $H_{of}$ and its
ground truth HDR image, as will be explained later.

###  3.2 Merging Network

Following the previous methods [43, 37], we employ an attention mechanism to
merge the feature maps and generate an HDR image; in specific, we use the
RCAB. As shown in Fig. 3, the merging network takes the concatenated feature
maps $Z_{0}=[O^{0},O^{1},O^{2}]$. We apply a convolution layer and three RCABs
to $Z_{0}$ and then concatenate the outputs of each RCAB as
$Z_{5}=[Z_{2},Z_{3},Z_{4}]$. Applying three convolutions and a global skip
connection with $F_{r}^{0}$, the merging network outputs a final HDR image.

Figure 7: Architecture of a residual channel attention block (RCAB) [43].

###  3.3 Loss Function

Following [14], we consider the optimization in the domain of tonemapped HDR
images because the HDR images are usually displayed after tonemapping and
training the network on the domain is more effective than that on the original
domain of HDR images. Thus, we employ the $\mu$-law for tone mapping as
suggested in [14], which is formulated as

$T(H)=\frac{log(1+\mu H)}{log(1+\mu)},$ (3)

where $\mu$ is set to $5,000$ throughout our experiments. It is also reported
in [44, 37] that minimizing the $L_{1}$ norm between the predicted HDR image
$\hat{H}$ and its ground truth $H$ in the tone-mapped domain works better than
others. Following their studies, we use the following $\ell_{1}$ loss,

$L_{tm}=\lVert T(\hat{H})-T(H)\rVert_{1}.$ (4)

For the standard optical flow estimators such as SPyNet [28] and PWC-Net [32],
they are trained on the datasets with the standard exposure settings (e.g.
Sintel [3], KITTI [9], and Middlebury [30]). However, there is no dataset
containing ground truths of optical flow maps for the HDR imaging task.
Inspired by the photometric loss [41] for self-supervised learning of optical
flow, we use $\ell_{1}$ loss between the reconstruction $H_{of}$ and its
ground truth $H$ in the tone-mapped domain to provide supervision for the
optical flow learning,

$L_{reg}=\lVert T(H_{of})-T(H)\lVert_{1}.$ (5)

The reconstruction of $H_{of}$ is only based on the warped features from the
non-reference images. Then $L_{reg}$ computed on $H_{of}$ and the ground truth
imposes a heavy constraint on these warped features to provide more accurate
gradients to the warping field than the $L_{tm}$ which involves both the
warped features and the features from the reference image.

Our total loss is taken as the weighted sum of two losses

$L=L_{tm}+\lambda L_{reg},$ (6)

where we use $\lambda=2$ in this paper.

Table 1: Quantitative comparison on the Kalantari’s test sets [14]. The numbers in the table are the average values of the 15 test images. Methods |  PSNR-$\mu$ | PSNR-L |  SSIM-$\mu$ | SSIM-L | HDR-VDP-2
---|---|---|---|---|---
TMO [8] | 8.3120 | 8.8459 | 0.5029 | 0.0924 | 44.3345
HDRCNN [7] | 13.7054 | 13.8956 | 0.5924 | 0.3456 | 47.5690
Sen [31] | 40.9689 | 38.3425 | 0.9859 | 0.9764 | 60.3463
Kalantari [14] | 42.7177 | 41.2200 | 0.9889 | 0.9829 | 61.3139
Wu [35] | 41.9977 | 41.6593 | 0.9878 | 0.9860 | 61.7981
AHDR [37] | 43.7013 | 41.1782 | 0.9905 | 0.9857 | 62.0521
PANet [27] | 43.8487 | 41.6452 | 0.9906 | 0.9870 | 62.5495
PSFNet [40] | 44.0613 | 41.5736 | 0.9907 | 0.9867 | 63.1550
Ours | 44.3298 | 41.8936 | 0.9911 | 0.9878 | 63.1190

Testing data (08) in [14] Testing data (09) in [14]

Figure 8: Results from the test set of [14]. Upper row from left to right: the
two input LDR images, the HDR image produced by the proposed method, and
(zoomed-in) LDR image patches with two identical positions/sizes (in green and
red). Lower row: the same patches of the HDR images produced by different
existing methods.

Example of Sen _et al_. ’s dataset [31] Example of Tursun _et al_. ’s dataset
[33]

Figure 9: Visual comparisons on the datasets without ground truth. See Fig. 8
for the explanation of the panels.

##  4 Experiments

###  4.1 Experimental Settings

####  4.1.1 Training Data

To train our network, we adopt the HDR dataset [14] which consists of 74
samples for training and 15 samples for testing. We use the former for
training our model. Each sample includes a ground truth HDR image and three
LDR images with different exposure settings of $\\{-2,0,+2\\}$ or
$\\{-3,0,+3\\}$. All the images are resized to the resolution of $1000\times
1500$.

####  4.1.2 Testing Data

Following recent studies, we choose the following datasets for testing. We
evaluate our method on the 15 scenes of the dataset of [14], where we perform
a quantitative evaluation using the provided ground truths. We also test the
proposed method on the datasets of Sen _et al_. [31] and Tursun _et al_. [33].
Since these datasets do not contain ground truths of HDR images, we compare
the reconstructed HDR images by our method with those by state-of-the-art
methods for qualitative evaluation.

####  4.1.3 Evaluation Metrics

As used in the existing studies, we use PSNR-$\mu$ and SSIM-$\mu$ for primary
metrics, which are PSNR and SSIM values in the tone-mapped domain using
$\mu$-law. We show PSNR and SSIM values in the linear domain, which are
denoted by PSNR-L and SSIM-L for completeness. We also report HDR-VDP-2 [22],
which is designed to evaluate the quality of HDR images.

###  4.2 Implementation Details

For training, we first crop the training images into patches of $256\times
256$ pixel size with a stride of 128 pixels. We then apply random rotation and
flipping for data augmentation to avoid over-fitting. We use the Adam
optimizer [18] with $\beta_{1}=0.9$, $\beta_{2}=0.999$, initial learning rate
$1\times 10^{-4}$ and set the batch size to 8. We train our model for 210
epochs and employ the cosine annealing strategy [21] to steadily decrease the
learning rate from an initial value to $1\times 10^{-6}$. We implement our
model using PyTorch [23] on NVIDIA GeForce RTX 2080 GPUs.

###  4.3 Comparison with the State-of-Art Methods

We compare the proposed method with existing methods. Specifically, we compare
our model with two HDR imaging methods based on a single LDR image, TMO [8]
and HDRCNN [7], and five HDR imaging methods based on multi LDR images, the
patch-based method [31], the flow-based method with CNN merger [14], the U-net
structure without optical flow [35], the attention-guide method (AHDR) [37],
pyramidal alignment network (PANet) [27], and progressive and selective fusion
network (PSFNet) [40]. For all the methods, we used the authors’ code for
comparison, except for [27] since their code is not available as of the time
of writing this paper.

####  4.3.1 Evaluation on Kalantari _et al_. ’s Dataset

Figure 8 shows two examples on the test set of [14]; see the supplementary
material for more visualization results. The input LDR images contain
saturated background and foreground motions. We can observe from the results
of the single-image methods, TMO [8] and HDRCNN [7], that they cannot
sufficiently recover the detailed textures and generate artifacts in the over-
exposed regions; they also suffer from the color distortion. The patch-based
method of Sen _et al_. [31] generates some artifacts due to the failure of
finding patches correctly. Kalantari _et al_.’s method [14] cannot completely
eliminate the effects of the occlusion. The method of Wu _et al_. [35] cannot
deal with over-exposed regions and then produces artifacts on motion areas.
Non-aligned methods (_i.e_. AHDR [37] and PSFNet [40]) yield artifacts in the
saturated areas and also suffer from ghosting artifacts due to the large
motions. Compared with them, our proposed method produces less color
distortion and recovers the textures more accurately, leading to the best
qualitative results.

Table 1 shows the quantitative evaluation on the same dataset. In specific, we
report the averaged values over 15 test scenes. It can be seen that the
proposed method achieves better performance than the others in terms of
PSNR-$\mu$, SSIM-$\mu$, PSNR-L and SSIM-L. Also, our method achieves
comparable performance to the state-of-the-art method [40] in terms of the
HDR-VDP-2 metric.

####  4.3.2 Evaluation on Datasets w/o Ground Truth

We also provide comparisons using Sen’s [31] and Tursun’s [33] datasets. These
datasets do not have ground truths of HDR images and thus we qualitatively
compare the generated HDR images.

Some examples of the results are shown in Fig. 9. The single image methods,
TMO [8] and HDRCNN [7], generate serious noises and color distortions in the
under-exposed regions. The patch-based method (Sen et al [31]) also generates
severe artifacts. The method of Kalantari _et al_. [14] produces artifacts due
to the alignment error and also generates serious noises in the under-exposed
regions. These are arguable because of the misalignment by the estimated
optical flow and the limitation of the merging method. Wu _et al_. ’s method
[35] tends to yield over-smoothness and generate ghosting artifacts on the
large motion areas. AHDR [37] yields color distortions and also suffers from
ghosting artifacts due to large motions shown in Fig. 9 (b). PSFNet [40]
generates ghosting artifacts in the motion regions and generates the geometric
distortions shown in Fig. 9 (a). On the other hand, our method produces better
results with noticeably reduced geometric and color distortions compared with
others.

Table 2: Results of ablation tests on Kalantari’s test set. The upper row shows the effects of channel attention (CA), multi-scale feature flow module (MS-Flow), feature flow module (FF), and multi-scale feature fusion module (MS-Fuse). The lower row shows the effects of the choice of optical flow. Methods |  PSNR-$\mu$ | PSNR-L |  SSIM-$\mu$ | SSIM-L
---|---|---|---|---
MSFFNet w/o CA | 44.0377 | 41.8150 | 0.9909 | 0.9875
MSFFNet w/o MS | 44.1236 | 41.8216 | 0.9908 | 0.9869
MSFFNet w/o FF | 43.9466 | 41.3520 | 0.9909 | 0.9867
MSFFNet w/o MSFF | 43.6752 | 41.4698 | 0.9908 | 0.9868
MSFFNet w/o FF w/ SPyNet | 43.9717 | 41.3563 | 0.9905 | 0.9852
MSFFNet w/o FF w/ fixed-pre-trained SPyNet | 43.6611 | 41.6913 | 0.9896 | 0.9821
MSFFNet w/o FF w/ PWC-Net | 44.1436 | 42.0084 | 0.9911 | 0.9879
MSFFNet w/o FF w/ fixed-pre-trained PWC-Net | 43.3769 | 41.5546 | 0.9891 | 0.9808
MSFFNet | 44.3298 | 41.8936 | 0.9911 | 0.9878
Figure 10: Results obtained by ablated networks.

###  4.4 Ablation Study

We demonstrate the effectiveness of each component in the proposed method. We
use the same configurations as those used above unless otherwise noted.

####  4.4.1 Effects of Optical Flow Learning

First, we verify the effect of the proposed loss in Eq. 5 by changing the
value of the parameter $\lambda$. When $\lambda=0$, the network will be
trained only using the $\ell_{1}$ loss (_i.e_. Eq. 4) between the final
outputs and their ground truths. This is equivalent to the previous
methodology using optical flow [15]. It should be noted that although the work
[15] tackles the HDR video reconstruction, the approach also works on the HDR
imaging task. As shown in Fig. 11, the model without the proposed loss (_i.e_.
$\lambda=0$) generates sever color and geometry distortions. In contrast, the
models with the proposed training (_i.e_. $\lambda>0$) significantly improve
the reconstruction results. We also quantitatively evaluate them on the
Kalantari _et al_. ’s test set [14]. As shown in Table 3, our proposed
training with $\lambda=2$ achieves the best performance. Even though the gain
by the proposed training is not so large, the artifacts usually appear in a
small area of an image, and they have only small impacts on these metrics.

Table 3: Results obtained with different $\lambda$ values in Eq. 6. |  PSNR-$\mu$ | PSNR-L |  SSIM-$\mu$ | SSIM-L
---|---|---|---|---
$\lambda$=0 | 44.1229 | 41.8947 | 0.9911 | 0.9875
$\lambda$=1 | 44.0978 | 41.7622 | 0.9910 | 0.9868
$\lambda$=2 | 44.3298 | 41.8936 | 0.9911 | 0.9878
$\lambda$=3 | 44.1238 | 41.7043 | 0.9911 | 0.9871
Figure 11: Results obtained with different $\lambda$ values in Eq. 6. It can
be seen that our proposed training (_i.e_. $\lambda>0$) significantly improve
the reconstruction results.

####  4.4.2 Effects of Different Configurations

Since our method has several design choices, we conduct experiments to examine
which configuration shows the best. Specifically, we examine the effect of
channel attention (CA), multi-scale feature flow module (MS-Flow), feature
flow module (FF), and multi-scale feature (MS-Fuse). The results are shown in
Table 2. When we eliminate the MS-Flow, we use a single convolution layer to
extract feature maps and then concatenate them as $Z_{0}$. We also do not
reconstruct HDR images using the warped feature maps. When we eliminate the
FF, we also do not reconstruct HDR images and directly use $F_{1}^{s}$ as
$F_{1\rightarrow r}^{s}$ $(s=1,2,3)$ since there is no optical flow available
for the alignment. We can observe from Table 2 that CA and MS-Flow are
essential to achieve better performance. Figure 10 shows some examples of the
zoomed-in patches produced by the models with different configurations. It can
be seen that artifacts appear except the MSFFNet (with full components).

We compare the proposed multi-scale feature flow estimation module with other
optical flow methods (SPyNet [28] and PWC-Net [32]). We replace the FF with
the optical flow (SPyNet and PWC-Net) in the proposed network named MSFFNet
w/o FF w/ SPyNet and MSFFNet w/o FF w/ PWC-Net. As shown in Table 2, the
proposed method achieves better performance than other optical flow-based
methods even the PWC-Net has a more complicated network structure (e.g. with
correlation layer [6]) than ours. Since our feature flow network can not be
pre-trained on other datasets with optical flow ground truth, we also compare
the pre-trained optical flow network named MSFFNet w/o FF w/ fixed-pre-trained
SPyNet and MSFFNet w/o FF w/ fixed-pre-trained PWC-Net with the model trained
on the proposed regularized loss to demonstrate the effectiveness of the
proposed regularized loss. These two ablated networks are trained by only
using tone-mapped loss in Eq. 4. As shown in Table 2, the model trained on the
proposed regularized loss (MSFFNet w/o FF w/ SPyNet and MSFFNet w/o FF w/ PWC-
Net) achieve better performance than the pre-trained model (MSFFNet w/o FF w/
fixed-pre-trained SPyNet and MSFFNet w/o FF w/ fixed-pre-trained PWC-Net). As
shown in Figure 10, there is severe color and geometry distortion in the
images generated from the optical flow-based method.

##  5 Conclusions

In this paper, we propose a new method for generating an HDR image of a
dynamic scene from its LDR images along the alignment-before-merging
direction. The first step of feature alignment plays a central role in
generating high-quality HDR images. Trained by the regularized loss, the
multi-scale feature flow module can effectively learn the flow for alignment
even in occlusion regions and the large saturated areas, which greatly reduce
the artifacts in these regions. After the alignment by the estimated flow, the
features from the non-reference image will be fused with the features from the
reference image to reconstruct an HDR image. The experimental results have
validated the effectiveness of the proposed approach.

## References

  * [1] Eike Falk Anderson and Leigh McLoughlin.  Critters in the classroom: a 3d computer-game-like tool for teaching programming to computer animation students.  In ACM SIGGRAPH 2007 Educators Program, page 7. 2007.
  * [2] Luca Bogoni.  Extending dynamic range of monochrome and color images through fusion.  In Proceedings 15th International Conference on Pattern Recognition. ICPR-2000, volume 3, pages 7–12. IEEE, 2000.
  * [3] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black.  A naturalistic open source movie for optical flow evaluation.  In European conference on computer vision, pages 611–625. Springer, 2012.
  * [4] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Basicvsr: The search for essential components in video super-resolution and beyond.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4947–4956, 2021.
  * [5] Sungil Choi, Jaehoon Cho, Wonil Song, Jihwan Choe, Jisung Yoo, and Kwanghoon Sohn.  Pyramid inter-attention for high dynamic range imaging.  Sensors, 20(18):5102, 2020.
  * [6] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 2758–2766, 2015.
  * [7] Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K Mantiuk, and Jonas Unger.  Hdr image reconstruction from a single exposure using deep cnns.  ACM Transactions on Graphics (TOG), 36(6):1–15, 2017.
  * [8] Yuki Endo, Yoshihiro Kanamori, and Jun Mitani.  Deep reverse tone mapping.  ACM Transactions on Graphics, 36(6), 2017.
  * [9] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361. IEEE, 2012.
  * [10] David Hafner, Oliver Demetz, and Joachim Weickert.  Simultaneous hdr and optic flow computation.  In 2014 22nd International Conference on Pattern Recognition, pages 2065–2070. IEEE, 2014.
  * [11] Yong Seok Heo, Kyoung Mu Lee, Sang Uk Lee, Youngsu Moon, and Joonhyuk Cha.  Ghost-free high dynamic range imaging.  In Asian Conference on Computer Vision, pages 486–500. Springer, 2010.
  * [12] Jun Hu, Orazio Gallo, Kari Pulli, and Xiaobai Sun.  Hdr deghosting: How to deal with saturation?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1163–1170, 2013.
  * [13] Katrien Jacobs, Celine Loscos, and Greg Ward.  Automatic high-dynamic range image generation for dynamic scenes.  IEEE Computer Graphics and Applications, 28(2):84–93, 2008.
  * [14] Nima Khademi Kalantari and Ravi Ramamoorthi.  Deep high dynamic range imaging of dynamic scenes.  ACM Transactions on Graphics (TOG), 36(4):144:1–144:12, 2017.
  * [15] Nima Khademi Kalantari and Ravi Ramamoorthi.  Deep hdr video from sequences with alternating exposures.  In Computer graphics forum, volume 38, pages 193–205. Wiley Online Library, 2019.
  * [16] Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski.  High dynamic range video.  ACM Transactions on Graphics (TOG), 22(3):319–325, 2003.
  * [17] Erum Arif Khan, Ahmet Oguz Akyuz, and Erik Reinhard.  Ghost removal in high dynamic range images.  In 2006 International Conference on Image Processing, pages 2005–2008. IEEE, 2006.
  * [18] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  * [19] Chul Lee, Yuelong Li, and Vishal Monga.  Ghost-free high dynamic range imaging via rank minimization.  IEEE Signal Processing Letters, 21(9):1045–1049, 2014.
  * [20] Jiayi Lin, Yan Huang, and Liang Wang.  Fdan: Flow-guided deformable alignment network for video super-resolution.  arXiv preprint arXiv:2105.05640, 2021.
  * [21] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with warm restarts.  In 5th International Conference on Learning Representations, ICLR 2017, 2017.
  * [22] Rafał Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich.  Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions.  ACM Transactions on Graphics (TOG), 30(4):1–14, 2011.
  * [23] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [24] Charles Poynton.  Digital video and HD: Algorithms and Interfaces.  Elsevier, 2012.
  * [25] K Ram Prabhakar, Susmit Agrawal, Durgesh Kumar Singh, Balraj Ashwath, and R Venkatesh Babu.  Towards practical and efficient high-resolution hdr deghosting with cnn.  In European Conference on Computer Vision, pages 497–513. Springer, 2020.
  * [26] K Ram Prabhakar, Rajat Arora, Adhitya Swaminathan, Kunal Pratap Singh, and R Venkatesh Babu.  A fast, scalable, and reliable deghosting method for extreme exposure fusion.  In 2019 IEEE International Conference on Computational Photography (ICCP), pages 1–8. IEEE, 2019.
  * [27] Zhiyuan Pu, Peiyao Guo, M Salman Asif, and Zhan Ma.  Robust high dynamic range (hdr) imaging with complex motion and parallax.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [28] Anurag Ranjan and Michael J Black.  Optical flow estimation using a spatial pyramid network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4161–4170, 2017.
  * [29] Allan G Rempel, Wolfgang Heidrich, Hiroe Li, and Rafał Mantiuk.  Video viewing preferences for hdr displays under varying ambient illumination.  In Proceedings of the 6th Symposium on Applied Perception in Graphics and Visualization, pages 45–52, 2009.
  * [30] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling.  High-resolution stereo datasets with subpixel-accurate ground truth.  In German conference on pattern recognition, pages 31–42. Springer, 2014.
  * [31] Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B Goldman, and Eli Shechtman.  Robust patch-based hdr reconstruction of dynamic scenes.  ACM Transactions on Graphics (TOG), 31(6):203:1–203:12, 2012.
  * [32] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934–8943, 2018.
  * [33] Okan Tarhan Tursun, Ahmet Oğuz Akyüz, Aykut Erdem, and Erkut Erdem.  An objective deghosting quality metric for hdr images.  In Computer Graphics Forum, volume 35, pages 139–152. Wiley Online Library, 2016.
  * [34] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [35] Shangzhe Wu, Jiarui Xu, Yu-Wing Tai, and Chi-Keung Tang.  Deep high dynamic range imaging with large foreground motions.  In Proceedings of the European Conference on Computer Vision, pages 117–132, 2018.
  * [36] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  International Journal of Computer Vision, 127(8):1106–1125, 2019\.
  * [37] Qingsen Yan, Dong Gong, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian Reid, and Yanning Zhang.  Attention-guided network for ghost-free high dynamic range imaging.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1751–1760, 2019.
  * [38] Qingsen Yan, Jinqiu Sun, Haisen Li, Yu Zhu, and Yanning Zhang.  High dynamic range imaging by sparse representation.  Neurocomputing, 269:160–169, 2017.
  * [39] Qingsen Yan, Lei Zhang, Yu Liu, Yu Zhu, Jinqiu Sun, Qinfeng Shi, and Yanning Zhang.  Deep hdr imaging via a non-local network.  IEEE Transactions on Image Processing, 29:4308–4322, 2020.
  * [40] Qian Ye, Jun Xiao, Kin-man Lam, and Takayuki Okatani.  Progressive and selective fusion network for high dynamic range imaging.  In Proceedings of the ACM International Conference on Multimedia, 2021.
  * [41] Jason J Yu, Adam W Harley, and Konstantinos G Derpanis.  Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness.  In European Conference on Computer Vision, pages 3–10. Springer, 2016.
  * [42] Wei Zhang and Wai-Kuen Cham.  Gradient-directed multiexposure composition.  IEEE Transactions on Image Processing, 21(4):2318–2323, 2011.
  * [43] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Proceedings of the European conference on computer vision (ECCV), pages 286–301, 2018.
  * [44] Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz.  Loss functions for image restoration with neural networks.  IEEE Transactions on Computational Imaging, 3(1):47–57, 2016.
  * [45] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9308–9316, 2019.