# HDRVideo-GAN: Deep Generative HDR Video Reconstruction

Mrinal Anand  Indian Institute of TechnologyGandhinagarGujaratIndia
<EMAIL_ADDRESS>,  Nidhin Harilal  University of
ColoradoBoulderColoradoUSA<EMAIL_ADDRESS>,  Chandan Kumar
Indian Institute of TechnologyGandhinagarGujaratIndia
<EMAIL_ADDRESS>and  Shanmuganathan Raman  Indian Institute
of TechnologyGandhinagarGujaratIndia<EMAIL_ADDRESS>
###### Abstract.

High dynamic range (HDR) videos provide a more visually realistic experience
than the standard low dynamic range (LDR) videos. Despite having significant
progress in HDR imaging, it is still a challenging task to capture high-
quality HDR video with a conventional off-the-shelf camera. Existing
approaches rely entirely on using dense optical flow between the neighboring
LDR sequences to reconstruct an HDR frame. However, they lead to
inconsistencies in color and exposure over time when applied to alternating
exposures with noisy frames. In this paper, we propose an end-to-end GAN-based
framework for HDR video reconstruction from LDR sequences with alternating
exposures. We first extract clean LDR frames from noisy LDR video with
alternating exposures with a denoising network trained in a self-supervised
setting. Using optical flow, we then align the neighboring alternating-
exposure frames to a reference frame and then reconstruct high-quality HDR
frames in a complete adversarial setting. To further improve the robustness
and quality of generated frames, we incorporate temporal stability-based
regularization term along with content and style-based losses in the cost
function during the training procedure. Experimental results demonstrate that
our framework achieves state-of-the-art performance and generates superior
quality HDR frames of a video over the existing methods.

Computational Photography, HDR Video Reconstruction, GAN, Adversarial
Training, Self-Supervised Training

††copyright: none††ccs: Computing methodologies Computational
photography††ccs: Computing methodologies Image-based rendering

##  1\. Introduction

The dynamic range that the human visual system can experience in the real
world is vast. Unfortunately, most off-the-shelf digital cameras capture only
a limited range of illumination in the scene. This discrepancy has lead to a
great deal of research in reconstructing still HDR images from the
conventional LDR off-the-shelf camera images. Most of these works have used a
bracketed exposure imaging method (Mann and Picard, 1994; Debevec and Malik,
2008; Mann et al., 2002; Kalantari and Ramamoorthi, 2017) which involves
taking multiple images at different exposures and merging them to generate a
single HDR image.

Generating HDR images by taking multiple images with different exposures may
involve object/ camera movement. Therefore these methods end up producing
ghosting artifacts in dynamic scenes of the image. These artifacts can be
reduced through various methods like replacing/rejecting the pixels that move
across the images (Khan et al., 2006; Zhang et al., 2010; Zheng et al., 2013),
merging all different exposure images with a reference image (Ward, 2003;
Tomaszewska and Mantiuk, 2007; Gallo et al., 2015) or aligning and
reconstructing in a unified optimization system (Sen et al., 2012; Zheng et
al., 2013). Capturing HDR video directly involves expensive specialized
cameras that use complex optical systems (Tocci et al., [n.d.]), and sensors
(Zhao et al., 2015).

On the other hand, reconstructing HDR video from the LDR sequence obtained
from a standard off-the-shelf camera is a much more challenging task. Existing
methods that are focused on reconstructing HDR images have been observed to
generate temporally unstable results when applied to video sequences.

There exist few works addressing this problem (Kang et al., 2003; Mangiat and
Gibson, 2011; Kalantari et al., 2013; Li et al., 2016), which are typically
slow and have limitations in several scenarios. Recently, the first deep
learning-based approach was proposed by Kalantari  _et al._ (Kalantari and
Ramamoorthi, 2019) for HDR video reconstruction, which utilized dense frame-
to-frame motion information (optical flow) (Lucas and Kanade, 1981). Their
method first aligns the neighboring alternating exposure LDR frames to a
reference frame by computing the optical flow between them, and then they use
a convolutional neural network (CNN) based model to merge and reconstruct the
final HDR frame. Although they show a reduction in time while reconstructing
the HDR frames by a certain factor but as pointed out by its authors, their
approach still suffers from discoloration and flickering artifacts in the
reconstructed HDR video frames.

In this paper, we take inspiration from (Kalantari and Ramamoorthi, 2019) and
design a Generative Adversarial Network (GAN) based framework for
reconstructing HDR video frames from the LDR sequence with alternating
exposures. Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019) performed an
end-to-end training by minimizing the error between the reconstructed and
ground truth HDR video frames on a set of training scenes. We show that merely
reducing the pixel to pixel error between reconstructed and ground truth HDR
frames from noisy LDR is prone to content loss and undesirable artifacts in
the generated frames. We address this by proposing a framework comprising of
an LDR denoising network, a light-weight optical flow estimation network, and
a GAN based model for final HDR reconstruction. We modify the training
procedure of our GAN based model by incorporating a temporal-stability based
regularization term (Eilertsen et al., 2019) along with content and style-
based losses in the cost function while training the network. We use an
altered version of an existing optical flow estimation model, LiteFlowNet (Hui
et al., 2018), which is fine-tuned to estimate the dense optical flow between
LDR frames with varying exposures. The estimated optical flow is then used to
align the LDR frames with alternating exposures to the current frame. The
final HDR frame is generated by using a Generative Adversarial Network (GAN).
In addition to the regularization term, standard adversarial loss, and the HDR
reconstruction losses, we also incorporate perceptual loss (Johnson et al.,
2016), and style-aware content loss (Sanakoyeu et al., 2018) while training
the network for better performance. The proposed framework generates
temporally stable HDR video with high visual quality.

GAN based models require a lot of data for image synthesis tasks. For the HDR
reconstruction, we generate our training dataset synthetically by extracting
the input LDR frames from a set of open-sourced HDR video repositories
(Froehlich et al., 2014; Kronander et al., 2014). However, unlike these
synthetically generated LDR videos, frames captured from standard digital
cameras have varied noise in them. Therefore, for the framework to generalize
well, we fuse the LDR frames with the Gaussian noise of varied signal-to-noise
(SNR) ratios. The main contributions of our work are given below:

  * •

We propose the first GAN-based method for the HDR video reconstruction by
using LDR frames with alternating exposures. Our proposed framework consists
of a denoising network for extracting clean LDR frames, a light-weight optical
flow estimation network, and a GAN based model for final HDR reconstruction.

  * •

We incorporate perceptual as well as style-aware content losses to improve the
visual quality of HDR frames. Along with utilizing the optical flow, we also
incorporate a temporal-stability based regularization while training to
further reduce the temporal incoherence in the reconstructed HDR frames.

  * •

Our experimental results on the different HDR video datasets demonstrate that
the proposed framework outperforms the existing approaches and produces high-
quality HDR video.

Outline of the paper: The entire paper is organized as follows. Section 2
narrates the related work in the area of HDR imaging. Section 3 describes the
dataset used for the experimentation. Section 4 presents the model
architecture in detail. Section 5 describes the details about the used
hyperparameters. Section 6 reports the qualitative and quantitative
evaluations against the baseline. Finally, section 7 presents the ablation
study.

##  2\. Related Work

Notation |  Desciption
---|---
$L_{i}$ – |  original $i^{th}$ LDR frame (alternating expos.)
$\hat{L}_{i}$ – |  $i^{th}$ generated clean LDR (alternating expos.)
$\widetilde{L}_{i}$ – |  aligned $i^{th}$ clean LDR (alternating expos.)
$H_{i}$ – |  Original $i^{th}$ HDR frame
$\widetilde{H}_{i}$ – |  Generated $i^{th}$ HDR frame
$T_{i}$ – |  $i^{th}$ Tonemapped Frame of original HDR
$\widetilde{T}_{i}$ – |  $i^{th}$ Tonemapped Frame of generated HDR
Table 1. Description of notations frequently occurring in the paper

In the last few years with the onset on learning algorithm, the problem of HDR
imaging has also been extensively explored. However, a lot of work is centered
around the generation of still HDR images. One set of approaches uses a
sequence of different exposure images to generate HDR images (Debevec and
Malik, 1997; Sen et al., 2012; Hu et al., 2013; Oh et al., 2014; Ma et al.,
2017; Kalantari and Ramamoorthi, 2017), while the other approach uses burst
images to generate the HDR image (Liu et al., 2014; Hasinoff et al., 2016).
There are some more focused works in the last few years to generate HDR images
from a single image (Eilertsen et al., 2017; Marnerides et al., 2018). Almost
all these approaches are not suitable for generating HDR video because of the
lack of temporal consistency in still HDR imaging. For brevity, we will only
discuss the works related to the generation of HDR video.

The system that produced the most high-quality results to date has been the
specialized camera that directly captures HDR video. These cameras include
special sensors that can capture extensive dynamic range (Brajovic and Kanade,
1996; Seger et al., 1999; Nayar and Mitsunaga, 2000) or the camera which has
beam-splitters that deflects the light to many sensors such that ever sensors
measure the different amount of radiance concurrently. However, these
approaches are limited because they need specialized custom hardware that has
enormous costs and, therefore, less widespread (Tocci et al., 2011; Kronander
et al., 2013).

One way to generate HDR video is from the input sequence of frames having
alternate exposure of each frame. Kang  _et al._ (Kang et al., 2003) first
proposed the method of HDR video reconstruction using the alternating exposure
LDR frames. They used optical flow to align the neighboring frames to the
reference frame. After aligning the nearby frame to the reference frame, they
take a weighted sum to combine with the reference frame to avoid ghosting
artifacts. However, their approach leads to ghosting artifacts when the scene
has a significant amount of motion.

Mangiant and Gibson (Kang et al., 2003) improved the approach of Kang  _et
al._ (Kang et al., 2003) using a block-based motion estimation method which
was coupled with a refinement stage. In their successive work, they filtered
the region with a significant motion to minimize the blocking artifacts.
However, their approach still had the blocking artifacts when the scene has
substantial movement in it. In addition to that, their approach is limited to
working only on two exposure sequences. Kalantari  _et al._ (Kalantari et al.,
2013) propose a patch-based method to reconstruct the missing exposure at each
frame. After reconstruction, all the images produced were combined to obtain
the final HDR frame. Temporal coherency is improved by estimating the motion
between the neighboring/adjacent and the reference frame. However, the patch
search was constraint only to small window around the predicted movement,
where a greedy approach obtains the window size. This method produces the
result, which is significantly better than the previous approach. However, to
solve the complex patch-based optimization was a time consuming process and
produce a single HDR frame. A major drawback of this approach was that it
often was unable to constrain the patch search properly and underestimates the
search window size. Ghosting artifacts were observed in such cases. One
further work is (Gryaditskaya et al., 2015) that improves the method of
Kalantari  _et al._ (Kalantari et al., 2013) by adaptively adjusting the
exposure. In a recent work by Li  _et al._ (Li et al., 2016) proposes to
consider the HDR video reconstruction problem as a maximization of posteriori
estimate. Their method focuses on finding the foreground and background of
each HDR frame separately. They extract the background and foreground using
rank minimization and multiscale adaptive regression techniques, respectively.
The major drawback of this method is also the computational cost involved, it
takes around two hours to generate a single frame with $1280\times 720$
resolution. Additionally, many of the frames were accompanied by noise and
discoloration.

Recently, Kalantari and Ramamoorthi (Kalantari and Ramamoorthi, 2019) have
proposed an approach in which they have used two connected networks called
Flow network and Merge Network. Flow Network aligns the neighboring frame with
the current frame, while the Merge network is used to merge the aligned frames
with the reference frame. Their approach solves the problem in the best way so
far. This is also state of the art for HDR video generation. But there are
still ghosting artifacts in the challenging cases where the reference image is
overexposed or if there are notable parallax and occlusion.

##  3\. Dataset

Figure 1. Visual comparison of the over and under exposed LDR frames generated
using Equation 1 with the corresponding HDR frame. Note the loss of details in
the dark and bright regions of under and over exposed LDR frames.

We require a large dataset consisting of HDR video frames with corresponding
LDR frames having alternating exposures. We use two publicly available HDR
video datasets curated by Froehlich  _et al._ (Froehlich et al., 2014) (13
videos) and Kronander  _et al._ (Kronander et al., 2014) (8 videos). These
datasets were prepared using cameras with a specific optical design containing
external (Froehlich et al., 2014) and internal (Kronander et al., 2014) beam
filters. The dataset contains 41 HDR videos, out of which 38 were used for
training, and the remaining three were used as the hold-out test set
consistent with the current state-of-the-art method proposed by Kalantari  _et
al._ (Kalantari and Ramamoorthi, 2019) during experiments. We generate
synthetic LDR frames from ground truth HDR frames at different exposures using
Eq 1.

(1)
$\begin{split}L_{i}=g_{i}({H}_{i})=clip[({H}_{i}t_{i})^{1/\gamma}]\end{split}$

here $\gamma$ is 2.2, ${H_{i}}$ is HDR image in linear domain, $t_{i}$ is the
exposure time, and $clip$ function clips the output in the range $[0,1]$.
Figure 1 shows the comparison of an underexposed, and overexposed LDR frames,
along with an HDR frame. It can be clearly seen from Figure 1 (a) and (b)
clearly depicts the content loss in overexposed and underexposed LDR frames as
compared to the corresponding HDR.

##  4\. Proposed Architecture

In this section, we present a detailed discussion of our proposed framework.
Our proposed framework consists of three parts, i.e., an LDR denoising network
for extracting clean LDR frames from noisy LDR video, a lightweight optical
flow estimation network, and a GAN based model for the final reconstruction of
high-quality HDR frames.

Notations. Table 1 summarises all the variables used in this paper and their
corresponding definitions. Here $L$ denotes the LDR frame, $H$ denotes the HDR
frame, and $T$ denotes the tonemapped HDR. An important point to note here is
that $i$ represents the $i^{th}$ frame of the video. So $(i-1)^{th}$ and
$(i+1)^{th}$ represents previous frame and the next frame with respect to the
$i^{th}$ frame.

###  4.1. Self-Supervised Denoising Network

Figure 2. Visual comparison of noisy LDR frame $L^{\prime}_{t-1}$ generated by
adding gaussian noise using Equation 2 to the synthetically generated LDR
frames $L_{t-1}$ Figure 3. Architecture of our Denoising network

Real-world off-the-shelf cameras are prone to capturing noise while recording
LDR frames. This noise produces unwanted artifacts in both scenarios, i.e.,
first while aligning the neighboring frames by computing the optical flow
between the frames and secondly on final reconstruction from these aligned
frames. In order to reconstruct high-quality HDR frames, we require the
corresponding LDR frames to be less noisy. In our method, we incorporate a
denoising network that removes such imperfections from the noisy LDR frames.
We call our self-supervised denoising blocks as ELDR blocks.

Self-Supervised paradigm has shown promising results in learning feature
representation (Jenni and Favaro, 2018; Simonyan and Zisserman, 2014),
temporal coherency (Mobahi et al., 2009), image denoising (Xu et al., 2019),
and many other tasks (Zhang et al., 2016; Lai and Xie, 2019; Laine et al.,
2019). Inspired by this, we design a self-supervision based LDR frames
denoising network that learns to create clean LDR frames from the noisy LDR
video frames. For all our experimentation, we use Gaussian noise as the
perturbation function to generate noisy LDR frames from the synthetic LDR
frames as described in Equation 2, consistent with previous baselines
(Kalantari and Ramamoorthi, 2019). Figure 2 shows the example of an over-
exposed LDR and the corresponding frame after the Gaussian noise addition.

(2) $\begin{split}{L^{\prime}}_{i}=L_{i}+N(\mu,\sigma)\end{split}$

Our LDR frame denoise network consists of a series of convolution and
deconvolution operations along with a skip connection between them following a
U-Net (Ronneberger et al., 2015) like structure. Each 2-D convolution
operation is followed by a BatchNorm operation, ReLU activation, and a
$2\times 2$ Max-pooling layer to reach the bottleneck representation. Then the
bottleneck feature map is upsampled using deconvolution layers. Figure 3 shows
the architecture of our denoising network. In each iteration of the training
procedure, we take the LDR frames ($L_{i}$), and we add perturbation to it
according to Equation 2 and use these perturbed images as the input to the
network. The network is trained to extract out the original clean LDR frames
from all the added noise. Our proposed method consists of two such denoising
networks, each for a different exposure. All the parameters are identical
across both networks. The difference is that one network is trained on LDR
frames with low exposures and another for LDR frames with high exposures.
Formally, we train our denoising blocks using the $L1$ loss given below.

(3)
$\begin{split}L_{denoise}^{\prime}=||\widetilde{L}_{i}-L_{i}||_{1}\end{split}$

###  4.2. Flow Network

In video-to-video synthesis tasks, object movements across frames are known to
create temporal artifacts during the reconstruction. Visually incoherent
frames with poor temporal coherency is observed if existing image synthesis is
directly applied to videos without incorporating the temporal dynamics in the
model. We address this by aligning all the input LDR frames with alternating
exposure to a reference frame before using it for reconstructing the HDR
frames. In order to achieve such an alignment, we first estimate the optical
flow (Lucas and Kanade, 1981) between the consecutive LDR frames having
alternating exposures, which is then used to warp the previous frame to the
current frame.

Convolution neural network-based optical flow estimation was originally
proposed by Dosovitskiy  _et al._ (Fischer et al., 2015), which directly
generates a flow field from a pair of images. After this, many works have been
proposed on neural network-based optical flow estimation (Ranjan and Black,
2017; Ilg et al., 2017; Maurer and Bruhn, 2018). However, most of these
techniques are computationally expensive, and direct application of these
methods in our case, would not be scalable for real-time estimation of HDR
frames.

Therefore, we use a fine-tuned version of LiteFlowNet (Hui et al., 2018) in
our proposed pipeline for optical flow estimation, which outperforms most
other neural network-based flow estimation methods both in terms of speed and
accuracy. After obtaining the clean version of LDR frames with alternating
exposures, we compute the optical flow between these neighboring frames.
Originally, LiteFlowNet (Hui et al., 2018) was trained to generate flow-maps
between video-frames having similar exposures. Directly using a pre-trained
version of this LiteFlowNet (Hui et al., 2018) would result in inconsistencies
due to the difference in exposures. In order to utilize this across different
exposures, we fine-tune it by leaving it trainable during our end-to-end
training procedure after initializing LiteFlowNet (Hui et al., 2018) with the
pre-trained version.

###  4.3. GAN Based HDR Frame Generation

Figure 4. Our proposed method for HDR video generation consisting of two
Denoising networks, a LiteFlownet (Hui et al., 2018) model, and a final GAN
based reconstruction model. Layers in both generator and discriminator can be
identified by its color as described in the table on the right side. Each
label of the layer follows the convention of k-kernel size-n-number of
kernels-s-stride size.

Network Architecture. We adopt the GAN based architecture for our HDR frame
reconstruction proposed by (Thasarathan et al., 2019). The proposed generator
network is based on encoder-decoder architecture, where the encoder first
downsamples the image twice ($H\times W\to H/4\times W/4$). The feature map is
then passed through 8 Res-Blocks followed by two upsampling layers. Similar to
(Thasarathan et al., 2019), we use the instance norm layer. We warp a
convolution layer, an instance-norm layer, and a ReLU activation layer into
one basic unit. The Res-Block consists of 2 basic units stacked over each
other. The discriminator consists of 5 convolution layers followed by two
dense layers. To stabilize the training process, we use the spectral norm. By
restricting our discriminative function to 1-Lipschitz, we prevent the
gradient uninformativeness problem (Zhou et al., 2019). Figure 4 shows our
proposed architecture. Let G and D denotes the generator and discriminator
networks. Our generator network takes clean overexposed LDR
($\widetilde{L}_{i}$) and an underexposed LDR ($\widetilde{L}_{i-1}$) received
after flow correction from the denoising networks and generates the current
HDR frame ($H_{i}$).

(4)
$\begin{split}H_{i}=\textit{G}(\widetilde{L}_{i},\widetilde{L}_{i-1})\end{split}$

###  4.4. Objective Function

Tone Mapping. Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019) argued
that defining loss function in linear HDR domain underestimates the error in
darker regions. The solution that they suggested is to convert HDR from linear
domain to log domain (Kalantari and Ramamoorthi, 2017; Zhang and Lalonde,
2017). Consistent with the previous baselines (Kalantari and Ramamoorthi,
2019), we also use differentiable $\mu$-law function transformation denoted by
$T$. We compute our loss on tonemapped HDR frames.

(5) $\begin{split}T_{i}=\frac{log(1+\mu H_{i})}{log(1+\mu)}\end{split}$

$L_{1}$ Loss. Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019) used
$l_{1}$ loss computed between generated HDR frame and ground truth HDR frame
to train the model. The authors also argued that the use of $L_{1}$ loss
promotes sharpness in images as compared to $L_{2}$ loss. Again, consistent
with the previous baselines, we also use $L_{1}$ loss in our model.

(6) $\begin{split}L_{l_{1}}=||T_{i}-\widetilde{T}_{i}||_{1}\end{split}$

Adversarial Objective. Rather than discriminating with $i^{th}$ ground truth
HDR frame, Thasarathan and Nareri (Thasarathan et al., 2019) proposed use of
previous $(i-1)^{th}$ frame. They argued that using a previous frame for
discrimination generates temporally more coherent frames than frames generated
using conventional adversarial loss. This adversarial loss reduces flickering
artifacts in frames.

(7)
$\begin{split}L_{adv}&=\mathbb{E}_{(\widetilde{L}_{i},\widetilde{L}_{i-1})}log[\textit{D}(\widetilde{L}_{i},\widetilde{L}_{i-1})]\\\
&+\mathbb{E}_{\widetilde{L}_{i-1}}log[1-\textit{D}(T_{i},\widetilde{T}_{i-1})]\end{split}$

Content and Style Losses. Frame reconstruction is also accompanied by other
visual artifacts like blurriness and color mismatch. We incorporate content
and style loss (Sanakoyeu et al., 2018) to minimize visual artifacts. Let
$\phi_{i}$ represent the activated feature map of $j^{th}$ layer of pre-
trained VGG-19. For our experiments, we use feature maps of $1^{st}$ to
$5^{th}$ layers. We also use style loss to maintain spatial consistency in the
generated HDR frame. $\Delta^{\phi}_{j}$ represents Gram matrix of $j^{th}$
feature map $\phi$. Equation 8 represents content loss and Equation 9
represents style loss.

(8)
$\begin{split}L_{content}=\mathbb{E}_{j}\left[\frac{1}{N_{j}}||\phi_{j}(T_{i})-\phi_{j}(\widetilde{T}_{i})||_{1}\right]\end{split}$
(9)
$\begin{split}L_{style}=\mathbb{E}_{j}\left[||\Delta^{\phi}_{j}(T_{i})-\Delta^{\phi}_{j}(\widetilde{T}_{i})||_{1}\right]\end{split}$

For our experiments we use $\lambda_{adv}=5$, $\lambda_{content}=1$,
$\lambda_{style}=1000$ and $\lambda_{l_{1}}=30$, Equation 10 represents the
overall loss function.

(10)
$\begin{split}L_{rec}=\lambda_{adv}L_{adv}+\lambda_{content}L_{content}+\lambda_{style}L_{content}\\\
+\lambda_{style}L_{style}+\lambda_{l_{1}}L_{l_{1}}\end{split}$

Temporal Regularization. Finally, we incorporate explicit regularization for
additional temporal stability (Bonneel et al., 2015; Eilertsen et al., 2019)
between two consecutive frames, which further helps in reducing blurriness in
high motion frames. W represents warping function from our flow network and
$\alpha$ is our regularization parameter, we set $\alpha=0.3$.

(11) $\begin{split}L_{reg}=||T_{i}-W({T}_{i-1})||_{2}\end{split}$ (12)
$\begin{split}L_{total}=\alpha L_{rec}+(1-\alpha)L_{reg}\end{split}$

##  5\. Training Details

In this section, we discuss our training methodology and present the values of
hyper-parameters. For all of our experiments, we used $\mu=5000$ for
tonemapping from linear to a logarithmic scale. We train our self-supervised
denoising networks with $L_{l2}$ loss for 100 epochs. We train our GAN model
with $L_{rec}$ loss using a batch-size of 20 for 70 epochs, and then we fine-
tune our network with $L_{total}$ using a batch-size of 35 for 15 epochs. The
training was performed entirely on a machine with Intel Core i7, 64GB of
memory, and a GeForce RTX 2080-Ti GPU. It roughly takes six days to complete
the training procedure (both denoising and GAN combined). Note that we freeze
the weights for our denoising network before training our GAN model. We
optimize our loss objective function using Adam optimizer with a learning rate
of $10^{-4}$ for training self-supervised network and $10^{-4}$ for GANs with
a batch size of 12. We use Leaky ReLU activation for the discriminator network
(Thasarathan et al., 2019), and for the rest of the network, we used ReLU
activation. We used the spectral norm in the discriminator to stabilize our
training procedure.

##  6\. Results

We compare our approach against the method of Kalantari  _et al._ (Kalantari
et al., 2013), which uses a patch-based mechanism for high dynamic range video
generation and against the current state-of-the-art, Kalantari  _et al._
(Kalantari and Ramamoorthi, 2019) which is based on convolutional neural
networks (CNNs). We used the publicly available source code for the patch-
based method by Kalantari  _et al._ (Kalantari et al., 2013) and Li  _et al._
(Li et al., 2016). For CNN based approach by Kalantari  _et al._ (Kalantari
and Ramamoorthi, 2019), the authors provided their results on only three
scenes from the test set. Both the patch-based mechanism by Kalantari  _et
al._ (Kalantari et al., 2013) and Li  _et al._ (Li et al., 2016) takes roughly
1-2 hours for generating each of the frames with a resolution $1280\times
720$. Recent work on CNN based method by Kalantari  _et al._ (Kalantari and
Ramamoorthi, 2019) showed that both patch-based method (Kalantari et al.,
2013) and the method of Li  _et al._ (Li et al., 2016) produce poor results on
different scenes. Thus, visual comparison against patch-based method
(Kalantari et al., 2013) and the method of Li  _et al._ (Li et al., 2016) is
difficult and superfluous. Moreover, our proposed method has a training
mechanism that is consistent with that of the recent CNN based model by
Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019) as explained in Section
5. Therefore, we only compare the visual results against CNN based model of
Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019).

###  6.1. Evaluation Metrics

| Kalantari (Kalantari et al., 2013) | Kalantari (Kalantari and Ramamoorthi, 2019) | Ours
---|---|---|---
PSNR | 38.77 | 40.67 | 43.35
SSIM | - | 0.78 | 0.83
HDR-VDP-2 | 62.12 | 74.15 | 77.19
Table 2. Quantitative comparison of our method against the patch based method
of Kalantari  _et al._ (Kalantari et al., 2013) and CNN based method by
Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019).

To evaluate the performance of our end-to-end generative model we use PSNR
(Horé and Ziou, 2010), SSIM (Zhou Wang et al., 2004) and HDR-VDP-2 (Mantiuk et
al., 2011a). Given the ground truth image$(gt)$ and the predicted
image$(pred)$, PSNR$(gt,pred)$ is defined as in equation 13 -

(13) $PSNR(gt,pred)=10\log_{10}(255^{2}/MSE(gt,pred))$

where $MSE(gt,pred)$ is mean squared error between the ground truth image and
predicted image having a size of $M\times N$ as in equation 14 -

(14) $MSE(gt,pred)={1\over M\cdot
N}\sum_{i=1}^{M}\sum_{j=1}^{N}(gt_{ij}-pred_{ij})^{2}$

Higher the PSNR value better is the quality of the reconstructed image. The
SSIM metric is also a well-known metric for measuring the visual quality of
the reconstructed image. SSIM metric takes into account luminance, contrast
and structural similarity into account and hence it is highly correlated with
the human perception.

HDR-VDP-2 (Mantiuk et al., 2011a) is also a visual metric that compares the
visibility score, i.e., the difference between $gt$ and $pred$ with respect to
an average observer and the degradation quality of $pred$ with respect to $gt$
expressed as a mean opinion score.

###  6.2. Quantitative Comparison

We quantitatively compare our results against the methods of Kang  _et al._
(Khan et al., 2006), the patch-based method of Kalantari  _et al._ (Kalantari
et al., 2013), and the current state-of-the-art by Kalantari  _et al._
(Kalantari and Ramamoorthi, 2019). We select frames from the scenes of FISHING
LONGSHOT, CAROUSEL FIREWORKS, and POKER FULLSHOT, which all comprise of the
test set. We extract LDR frames with alternating exposures, as described in
previous sections. Each frame has a resolution of $1920\times 1080$, but has a
wide black border of 10 pixels around them, which we crop on-wards for
quantitative comparison.

We evaluate the results on PSNR (Peak signal-to-noise ratio) and SSIM
(Structural Similarity Index) in its tone-mapped domain as described in
Equation 5. To further evaluate the quality of the generated HDR frames, we
use HDR-VDP2 (Mantiuk et al., 2011b), which is designed specifically to
evaluate HDR images and videos. Table 2 shows all the values using these
metrics computed and averaged across all the frames on test data. It can be
seen from Table 2 that the proposed method outperforms the other existing
approaches with respect to all the considered metrics.

Figure 5. Visual Comparison of our generated HDR frames having high motion
from CAROUSEL FIREWORKS scene in test data against Kalantari  _et al._
(Kalantari and Ramamoorthi, 2019). Figure 6. Visual Comparison of the our
generated HDR frames from a scene of FISHING LONGSHOT in the test data against
Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019). Identical regions of
comparison are all grouped in the same color.

###  6.3. Visual Comparisons

Figure 7. Visual Comparison of our generated HDR frames having high motion
from FISHING LONGSHOT scene in test data against Kalantari  _et al._
(Kalantari and Ramamoorthi, 2019).

We compare the reconstructed HDR frames of our method, and the CNN based
method of Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019) on scenes in
the test set. Figure 6 shows a detailed comparison of an HDR frame from a test
video scene. The scene shows the FISHING LONGSHOT, which includes a bright
region exposed by the sun (marked in the blue box) and a dark region having
very low exposure (marked in red box). It can be clearly observed from the
regions bounded by the blue and green color that the proposed method produces
frames with much more dynamic range than that of Kalantari  _et al._
(Kalantari and Ramamoorthi, 2019). On a close observation near the region of
the sky (bounded in blue) in Figure 6, it can be seen that our method
reconstructs the details of the clouds, while the one produced by the method
of Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019) loses the content and
reconstructs an over-exposed frame. Overall, from Figure 6 it can be seen that
our method generates HDR frames with much more details in regions of both the
high and low exposure areas.

Figure 5 and Figure 7 compare our approach against the CNN based method by
Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019) on the scenes of
CAROUSEL FIREWORKS and FISHING LONGSHOT respectively, which were not part of
the training set. We selected those frames to form the video scene with
significant motion in-between the adjacent frames i.e. high motion frames. As
evident from the frames of CAROUSEL FIREWORKS in Figure 5, CNN based method by
Kalantari  _et al._ (Kalantari and Ramamoorthi, 2019) generates tearing
artifacts in moving parts of the person in the frame. It also produces blurred
frames with ghosting artifacts in the scene of FISHING LONGSHOT in Figure 7
marked by red arrows.

###  6.4. Denoising Network

Figure 8. Visual comparison of a reconstructed clean LDR frame using denoising
network from a noisy LDR frame of a scene used previously in Figure 2

We begin by showing the results of the denoising network (ELDR Blocks), which
is trained in a self-supervised manner on the same training set scenes. We
extract LDR frames with alternating exposures from the HDR videos, as
described in Section 3. For each LDR frame having alternating exposures, we
add Gaussian noise with varied signal-to-noise ratios (SNR), which we
described in Section 4.1. Figure 8 shows the results of a clean LDR frame
generated by our denoising network from a noisy LDR frame of a scene, which
was earlier described in Figure 2. In general, we observe that the generated
LDR frames have a coherent texture with sharp features as compared to the
noisy LDR frames. The use of $L_{1}$ loss function in the ELDR blocks can be
accounted for the above observation.

##  7\. Ablation Study

Importance of Separate LDR Denoising Blocks. To study the significance of
these ELDR blocks, we remove these blocks from our overall pipeline and re-
train the model. It is evident from Table 3 and Figure 9 that both visual
quality and metric-wise, removal of denoising network have a significant
effect on the performance of our method. We observe a notable drop in the
PSNR, SSIM, and HDR-VDP2 (Mantiuk et al., 2011b) values. We find HDR frames
generated directly from noise embedded LDRs to be blurry with less detailed
reconstruction as compared to our complete approach, which generates crisp
details as shown in Figure 9. Thus, creating a two-stage network where noise
removal and reconstruction of HDR videos are performed separately has shown to
perform better than a single network performing both the tasks. Moreover, the
generated intermediate clean LDR frames add more interpretability in terms of
noise removal to our model as compared to previous baselines (Kalantari and
Ramamoorthi, 2019, 2017).

| Without denosing net | Ours (Complete)
---|---|---
PSNR | 41.39 | 43.35
SSIM | 0.76 | 0.83
HDR-VDP-2 | 73.87 | 77.19
Table 3. Quantitative comparison of our complete approach to our method
without denoising network. Removing the denoising network clearly drops the
performace on all the mentioned metrics. Figure 9. Visual comparison of our
complete approach to our method without the denoising network. Removing the
denoising network leads to reconstruction of HDR frames having poor quality
with less details.

##  8\. Conclusion

In this paper, we proposed a temporally stable GAN-based HDR video
reconstruction network that reconstructs HDR videos from LDR sequences with
alternating exposures. Our method incorporates a separate LDR denoising
network for extracting clean LDR frames, and we showed that creating separate
denoising and reconstruction network outperforms a single network that
performs both the tasks. We first align the neighboring alternating exposure
frames using the LiteFlownet (Hui et al., 2018) to generate temporally
coherent frames. Training our model over a joint objective consisting of
$L_{1}$ loss, style-aware content losses (Sanakoyeu et al., 2018) and
augmented GAN loss (Thasarathan et al., 2019) helped in minimizing the visual
artifacts. Further, we fine-tune our model on a temporal stability based
regularization term to further reduce the tearing and ghosting artifacts due
to temporal incoherence. We perform all our experimentation consistent with
the previous baselines, and we demonstrate that our method outperforms the
previous baselines both visually and metric-wise.

We believe that there is a great scope for further improvement in terms of
better colors, more dynamic range, and in overall visual quality. In the
future, we would like to further test our proposed method on larger datasets
of HDR videos as and when they become available.

###### Acknowledgements.

We would like to thank Science and Engineering Research Board (SERB) Core
Research Grant for supporting our work.

## References

  * (1)
  * Bonneel et al. (2015) Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015\.  Blind Video Temporal Consistency.  _ACM Trans. Graph._ 34, 6, Article 196 (Oct. 2015), 9 pages.  https://doi.org/10.1145/2816795.2818107
  * Brajovic and Kanade (1996) Vladimir Brajovic and Takeo Kanade. 1996.  A sorting image sensor: An example of massively parallel intensity-to-time processing for low-latency computational sensors. In _Proceedings of IEEE International Conference on Robotics and Automation_ , Vol. 2. IEEE, 1638–1643.
  * Debevec and Malik (1997) PE Debevec and J Malik. 1997.  Recovering high dynamic range radiance maps from photographs: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques.  _Los Angeles, USA: SIGGRAPH_ (1997).
  * Debevec and Malik (2008) Paul E Debevec and Jitendra Malik. 2008.  Recovering high dynamic range radiance maps from photographs.  In _ACM SIGGRAPH 2008 classes_. 1–10.
  * Eilertsen et al. (2017) Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K Mantiuk, and Jonas Unger. 2017\.  HDR image reconstruction from a single exposure using deep CNNs.  _ACM Transactions on Graphics (TOG)_ 36, 6 (2017), 1–15.
  * Eilertsen et al. (2019) Gabriel Eilertsen, Rafal K. Mantiuk, and Jonas Unger. 2019\.  Single-Frame Regularization for Temporally Stable CNNs.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2019).  https://doi.org/10.1109/cvpr.2019.01143
  * Fischer et al. (2015) Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. 2015.  FlowNet: Learning Optical Flow with Convolutional Networks.  arXiv:1504.06852 [cs.CV]
  * Froehlich et al. (2014) Jan Froehlich, Stefan Grandinetti, Bernd Eberhardt, Simon Walter, Andreas Schilling, and Harald Brendel. 2014.  Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. In _Digital Photography X_ , Vol. 9023. International Society for Optics and Photonics, 90230X.
  * Gallo et al. (2015) Orazio Gallo, Alejandro Troccoli, Jun Hu, Kari Pulli, and Jan Kautz. 2015.  Locally non-rigid registration for mobile HDR photography. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_. 49–56.
  * Gryaditskaya et al. (2015) Yulia Gryaditskaya, Tania Pouli, Erik Reinhard, Karol Myszkowski, and Hans-Peter Seidel. 2015.  Motion aware exposure bracketing for HDR video. In _Computer Graphics Forum_ , Vol. 34. Wiley Online Library, 119–130.
  * Hasinoff et al. (2016) Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016\.  Burst photography for high dynamic range and low-light imaging on mobile cameras.  _ACM Transactions on Graphics (TOG)_ 35, 6 (2016), 1–12.
  * Horé and Ziou (2010) A. Horé and D. Ziou. 2010.  Image Quality Metrics: PSNR vs. SSIM. In _2010 20th International Conference on Pattern Recognition_. 2366–2369.  https://doi.org/10.1109/ICPR.2010.579
  * Hu et al. (2013) Jun Hu, Orazio Gallo, Kari Pulli, and Xiaobai Sun. 2013\.  HDR deghosting: How to deal with saturation?. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1163–1170.
  * Hui et al. (2018) Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy. 2018.  LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation. In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 8981–8989.
  * Ilg et al. (2017) E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. 2017\.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.  http://lmb.informatik.uni-freiburg.de//Publications/2017/IMKDB17
  * Jenni and Favaro (2018) Simon Jenni and Paolo Favaro. 2018.  Self-Supervised Feature Learning by Learning to Spot Artifacts.  _CoRR_ abs/1806.05024 (2018).  arXiv:1806.05024 http://arxiv.org/abs/1806.05024
  * Johnson et al. (2016) Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016.  Perceptual losses for real-time style transfer and super-resolution. In _European conference on computer vision_. Springer, 694–711.
  * Kalantari and Ramamoorthi (2017) Nima Khademi Kalantari and Ravi Ramamoorthi. 2017.  Deep high dynamic range imaging of dynamic scenes.  _ACM Trans. Graph._ 36, 4 (2017), 144–1.
  * Kalantari and Ramamoorthi (2019) Nima Khademi Kalantari and Ravi Ramamoorthi. 2019.  Deep HDR Video from Sequences with Alternating Exposures. In _Computer Graphics Forum_ , Vol. 38. Wiley Online Library, 193–205.
  * Kalantari et al. (2013) Nima Khademi Kalantari, Eli Shechtman, Connelly Barnes, Soheil Darabi, Dan B Goldman, and Pradeep Sen. 2013.  Patch-based high dynamic range video.  _ACM Trans. Graph._ 32, 6 (2013), 202–1.
  * Kang et al. (2003) Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2003.  High dynamic range video.  _ACM Transactions on Graphics (TOG)_ 22, 3 (2003), 319–325.
  * Khan et al. (2006) Erum Arif Khan, Ahmet Oguz Akyuz, and Erik Reinhard. 2006\.  Ghost removal in high dynamic range images. In _2006 International Conference on Image Processing_. IEEE, 2005–2008.
  * Kronander et al. (2013) Joel Kronander, Stefan Gustavson, Gerhard Bonnet, and Jonas Unger. 2013.  Unified HDR reconstruction from raw CFA data. In _IEEE international conference on computational photography (ICCP)_. IEEE, 1–9.
  * Kronander et al. (2014) Joel Kronander, Stefan Gustavson, Gerhard Bonnet, Anders Ynnerman, and Jonas Unger. 2014\.  A unified framework for multi-sensor HDR video reconstruction.  _Signal Processing: Image Communication_ 29, 2 (2014), 203–215.
  * Lai and Xie (2019) Zihang Lai and Weidi Xie. 2019.  Self-supervised Learning for Video Correspondence Flow.  _CoRR_ abs/1905.00875 (2019).  arXiv:1905.00875 http://arxiv.org/abs/1905.00875
  * Laine et al. (2019) Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. 2019.  High-Quality Self-Supervised Deep Image Denoising.  In _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 6970–6980.  http://papers.nips.cc/paper/8920-high-quality-self-supervised-deep-image-denoising.pdf
  * Li et al. (2016) Yuelong Li, Chul Lee, and Vishal Monga. 2016.  A maximum a posteriori estimation framework for robust high dynamic range video synthesis.  _IEEE Transactions on Image Processing_ 26, 3 (2016), 1143–1157.
  * Liu et al. (2014) Ziwei Liu, Lu Yuan, Xiaoou Tang, Matt Uyttendaele, and Jian Sun. 2014.  Fast burst images denoising.  _ACM Transactions on Graphics (TOG)_ 33, 6 (2014), 1–9.
  * Lucas and Kanade (1981) Bruce D. Lucas and Takeo Kanade. 1981.  An Iterative Image Registration Technique with an Application to Stereo Vision. In _Proceedings of the 7th International Joint Conference on Artificial Intelligence - Volume 2_ (Vancouver, BC, Canada) _(IJCAI’81)_. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 674–679.
  * Ma et al. (2017) Kede Ma, Hui Li, Hongwei Yong, Zhou Wang, Deyu Meng, and Lei Zhang. 2017\.  Robust multi-exposure image fusion: A structural patch decomposition approach.  _IEEE Transactions on Image Processing_ 26, 5 (2017), 2519–2532.
  * Mangiat and Gibson (2011) Stephen Mangiat and Jerry Gibson. 2011.  Spatially adaptive filtering for registration artifact removal in HDR video. In _2011 18th IEEE International Conference on Image Processing_. IEEE, 1317–1320.
  * Mann et al. (2002) Steve Mann, Corey Manders, and James Fung. 2002\.  Painting with looks: Photographic images from video using quantimetric processing. In _Proceedings of the tenth ACM international conference on Multimedia_. 117–126.
  * Mann and Picard (1994) S Mann and R Picard. 1994\.  Being undigital with digital cameras.  _MIT Media Lab Perceptual_ 1 (1994), 2.
  * Mantiuk et al. (2011a) Rafa Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011a.  HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions.  _ACM Trans. Graph._ 30, 4, Article 40 (July 2011), 14 pages.  https://doi.org/10.1145/2010324.1964935
  * Mantiuk et al. (2011b) Rafał Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011b.  HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions.  30, 4, Article 40 (July 2011), 14 pages.  https://doi.org/10.1145/2010324.1964935
  * Marnerides et al. (2018) Demetris Marnerides, Thomas Bashford-Rogers, Jonathan Hatchett, and Kurt Debattista. 2018\.  ExpandNet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content. In _Computer Graphics Forum_ , Vol. 37. Wiley Online Library, 37–49.
  * Maurer and Bruhn (2018) Daniel Maurer and Andrés Bruhn. 2018.  ProFlow: Learning to Predict Optical Flow.  arXiv:1806.00800 [cs.CV]
  * Mobahi et al. (2009) Hossein Mobahi, Ronan Collobert, and Jason Weston. 2009\.  Deep Learning from Temporal Coherence in Video. In _Proceedings of the 26th Annual International Conference on Machine Learning_ (Montreal, Quebec, Canada) _(ICML ’09)_. Association for Computing Machinery, New York, NY, USA, 737–744.  https://doi.org/10.1145/1553374.1553469
  * Nayar and Mitsunaga (2000) Shree K Nayar and Tomoo Mitsunaga. 2000.  High dynamic range imaging: Spatially varying pixel exposures. In _Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662)_ , Vol. 1. IEEE, 472–479.
  * Oh et al. (2014) Tae-Hyun Oh, Joon-Young Lee, Yu-Wing Tai, and In So Kweon. 2014\.  Robust high dynamic range imaging by rank minimization.  _IEEE transactions on pattern analysis and machine intelligence_ 37, 6 (2014), 1219–1232.
  * Ranjan and Black (2017) A. Ranjan and M. J. Black. 2017.  Optical Flow Estimation Using a Spatial Pyramid Network. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 2720–2729.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  arXiv:1505.04597 [cs.CV]
  * Sanakoyeu et al. (2018) Artsiom Sanakoyeu, Dmytro Kotovenko, Sabine Lang, and Bjorn Ommer. 2018.  A Style-Aware Content Loss for Real-time HD Style Transfer. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Seger et al. (1999) Ulrich Seger, Uwe Apel, and Bernd Höfflinger. 1999\.  HDRC-imagers for natural visual perception.  _Handbook of Computer Vision and Application_ 1 (1999), 223–235.
  * Sen et al. (2012) Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B Goldman, and Eli Shechtman. 2012.  Robust patch-based hdr reconstruction of dynamic scenes.  _ACM Trans. Graph._ 31, 6 (2012), 203–1.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Two-Stream Convolutional Networks for Action Recognition in Videos. In _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1_ (Montreal, Canada) _(NIPS’14)_. MIT Press, Cambridge, MA, USA, 568–576.
  * Thasarathan et al. (2019) Harrish Thasarathan, Kamyar Nazeri, and Mehran Ebrahimi. 2019\.  Automatic temporally coherent video colorization. In _2019 16th Conference on Computer and Robot Vision (CRV)_. IEEE, 189–194.
  * Tocci et al. ([n.d.]) MD Tocci, C Kiser, N Tocci, and P Sen. [n.d.].  A Versatile HDR Video Production System. ACM TOG 30 (4), 41: 1–41: 10 (Jul 2011).
  * Tocci et al. (2011) Michael D Tocci, Chris Kiser, Nora Tocci, and Pradeep Sen. 2011\.  A versatile HDR video production system.  _ACM Transactions on Graphics (TOG)_ 30, 4 (2011), 1–10.
  * Tomaszewska and Mantiuk (2007) Anna Tomaszewska and Radoslaw Mantiuk. 2007.  Image registration for multi-exposure high dynamic range image acquisition.  (2007).
  * Ward (2003) Greg Ward. 2003\.  Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures.  _Journal of graphics tools_ 8, 2 (2003), 17–30.
  * Xu et al. (2019) Jun Xu, Yuan Huang, Ming-Ming Cheng, Li Liu, Fan Zhu, Zhou Xu, and Ling Shao. 2019.  Noisy-As-Clean: Learning Self-supervised Denoising from the Corrupted Image.  arXiv:1906.06878 [cs.CV]
  * Zhang and Lalonde (2017) Jinsong Zhang and Jean-François Lalonde. 2017\.  Learning High Dynamic Range from Outdoor Panoramas. In _IEEE International Conference on Computer Vision_.
  * Zhang et al. (2010) Li Zhang, Alok Deshpande, and Xin Chen. 2010.  Denoising vs. deblurring: HDR imaging techniques using moving cameras. In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE, 522–529.
  * Zhang et al. (2016) Richard Zhang, Phillip Isola, and Alexei A. Efros. 2016\.  Colorful Image Colorization.  _CoRR_ abs/1603.08511 (2016).  arXiv:1603.08511 http://arxiv.org/abs/1603.08511
  * Zhao et al. (2015) Hang Zhao, Boxin Shi, Christy Fernandez-Cull, Sai-Kit Yeung, and Ramesh Raskar. 2015.  Unbounded high dynamic range photography using a modulo camera. In _2015 IEEE International Conference on Computational Photography (ICCP)_. IEEE, 1–10.
  * Zheng et al. (2013) Jinghong Zheng, Zhengguo Li, Zijian Zhu, Shiqian Wu, and Susanto Rahardja. 2013.  Hybrid patching for a sequence of differently exposed images with moving objects.  _IEEE Transactions on Image Processing_ 22, 12 (2013), 5190–5201.
  * Zhou et al. (2019) Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. 2019\.  Lipschitz Generative Adversarial Nets.  _CoRR_ abs/1902.05687 (2019).  arXiv:1902.05687 http://arxiv.org/abs/1902.05687
  * Zhou Wang et al. (2004) Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ 13, 4 (2004), 600–612.  https://doi.org/10.1109/TIP.2003.819861