11institutetext: University of Electronic Science and Technology of China
22institutetext: University of Würzburg 33institutetext: Shanghai Jiao Tong
University 44institutetext: ETH Zurich

# PASTA: Towards Flexible and Efficient HDR Imaging Via Progressively
Aggregated Spatio-Temporal Aligment

Xiaoning Liu 11 Ao Li 11 Zongwei Wu 22 Yapeng Du 11 Le Zhang 11
Yulun Zhang 33 Radu Timofte 2244 Ce Zhu 11

###### Abstract

Leveraging Transformer attention has led to great advancements in HDR
deghosting. However, the intricate nature of self-attention introduces
practical challenges, as existing state-of-the-art methods often demand high-
end GPUs or exhibit slow inference speeds, especially for high-resolution
images like 2K. Striking an optimal balance between performance and latency
remains a critical concern. In response, this work presents PASTA, a novel
Progressively Aggregated Spatio-Temporal Alignment framework for HDR
deghosting. Our approach achieves effectiveness and efficiency by harnessing
hierarchical representation during feature distanglement. Through the
utilization of diverse granularities within the hierarchical structure, our
method substantially boosts computational speed and optimizes the HDR imaging
workflow. In addition, we explore within-scale feature modeling with local and
global attention, gradually merging and refining them in a coarse-to-fine
fashion. Experimental results showcase PASTA’s superiority over current SOTA
methods in both visual quality and performance metrics, accompanied by a
substantial 3-fold ($\times 3$) increase in inference speed.

###### Keywords:

High Dynamic Range Imaging Hierarchical Representation Progressive Aggregation

##  1 Introduction

Imaging sensors’ limited dynamic range often compromises photo quality by
losing detail in highlights and shadows. Multi-frame HDR imaging, which
involves capturing a stack of low dynamic range (LDR) images at different
exposures and then combining them, addresses this by expanding the dynamic
range and enhancing visual quality. However, this technique faces challenges
in accurately aligning frames in the presence of movement and shake, as well
as effectively merging frames of varying exposures while dealing with
occlusion.

Recent advancements in HDR imaging have evolved from early methods using hand-
crafted features [18, 32, 21, 37, 17, 47] to contemporary learning-based
models. These models excel in cross-exposure mappings due to inductive bias
[20, 57, 60, 59, 35, 24], with Transformer models further enhancing HDR
imaging by introducing contextualized awareness [31, 48, 58, 28, 50].

Despite these advancements, a major hurdle is the rising computational
complexity with higher image resolutions. The self-attention mechanism in
Transformers requires significant computational resources and memory, limiting
their real-world applicability, especially with the common use of high-
resolution images. While methods like [49, 42] can process 2K images, this
often comes at the expense of performance.

\begin{overpic}[width=346.89731pt,tics=5]{Figs/Intro/fig1_compare.pdf}
\put(10.0,-2.0){\scriptsize{(a) Tursun \emph{et al}.\hbox{}'s dataset
\cite[cite]{[\@@bibref{}{tursun2015state}{}{}]}}}
\put(58.0,43.0){\scriptsize{(b) Kalantari~{}\emph{et
al}.\hbox{}'s~{}\cite[cite]{[\@@bibref{}{kalantari2017deep}{}{}]} dataset}}
\put(60.0,-2.0){\scriptsize{(c) Tel~{}\emph{et
al}.\hbox{}'s~{}\cite[cite]{[\@@bibref{}{tel2023alignment}{}{}]} dataset}}
\put(56.0,68.0){\tiny{PSNR-$\mu$}} \put(90.0,68.0){\tiny{PSNR-$\emph{l}$}}
\put(73.0,78.0){\tiny{H.V.-2}} \put(83.0,46.5){\tiny{SSIM-$\emph{l}$}}
\put(63.0,46.5){\tiny{SSIM-$\mu$}} \par\put(56.0,27.0){\tiny{PSNR-$\mu$}}
\put(90.0,27.0){\tiny{PSNR-$\emph{l}$}} \put(73.0,37.0){\tiny{H.V.-2}}
\put(83.0,5.0){\tiny{SSIM-$\emph{l}$}} \put(63.0,5.0){\tiny{SSIM-$\mu$}}
\par\put(6.0,45.5){\tiny{LDRs}} \put(0.3,1.0){\tiny{NHDRRnet}}
\put(17.0,1.0){\tiny{CA-ViT}} \put(30.0,1.0){\tiny{PASTA-I}}
\put(44.0,1.0){\tiny{PASTA-U}} \put(29.5,45.5){\tiny{LDRs Patches}}
\par\par\end{overpic} Figure 1: Visual and quantitative comparison. Our method
consistently outperforms in two benchmark HDR datasets, urpassing the state-
of-the-art CA-ViT [31] and NHDRRnet [60] (the hierarchical method) methods in
preserving content integrity, achieving ghost-free and high-fidelity results.
Zoom in to see more details. Table 1: Components in existing HDR deghosting
methods. Components are categorized by functionality: i) Alignment involves
the spatial transformation of misaligned images or features, including
homography transformation (HT), spatial attention (SA), pyramid cross
attention (PCA), patch aggregation (PC) and temporal attention (TA); ii)
Aggregation characterizes the architecture of feature aggregation; iii) Domain
specifies aggregation in either the spatial or frequency domain, and iv) High
Resolution (HR) refers to the capability to handle 2K images.

| CNN-based Methods | Transformer-based Methods
---|---|---
DeepHDR [57] | AHDRNet [59] | HDR-GAN [38] | FHDRNet [5] | CA-ViT [31] | Joint-HDRDN [28] | HyHDRNet [58] | SCTNet [50] | PASTA-I | PASTA-U
| (ECCV’18) | (CVPR’19) | (TIP’21) | (CVIU’24) | (ECCV’22) | (CVPR’23) | (CVPR’23) | (ICCV’23) | (Ours) | (Ours)
Alignment | Yes (HT) | Yes (SA) | No | No | Yes (SA) | Yes (PCA) | Yes (PC & SA) | No | Yes (TA) | No
Aggregation | Hierarchical | Plain | Hierarchical | Hierarchical | Plain | Plain | Plain | Plain | Hierarchical | Hierarchical
Domain | Spat. | Spat. | Spat. | Freq. | Spat. | Spat. | Spat. | Spat. | Spat. & Freq. | Spat. & Freq.
HR (2K) | Yes | No | Yes | Yes | No | No | No | No | Yes | Yes

This necessitates further reflection and reevaluation of HDR deghosting
methods, aiming to construct a simple, powerful and efficient framework. Our
summaries, detailed in Tab. 1, distill the methodology into four key
components: Alignment, Aggregation, Domain, and High Resolution, each critical
to the practical effectiveness of HDR deghosting approaches.

We discern that hierarchical representation presents a natural solution
capable of simultaneously addressing multiple challenges. This representation
inherently contains information from different levels, making it a robust
candidate for our objectives. Moreover, hierarchical representation coupled
with diverse granularities aligns well with computational efficiency, further
enhancing its appeal for our purposes. Motivated by these considerations, we
propose a novel framework named Progressively Aggregated Spatio-Temporal
Alignment (PASTA) designed to fully exploit the benefits of hierarchical
representation for HDR imaging. Inspired by the effectiveness of wavelet
statistical modeling [4], we utilize the discrete wavelet transform (DWT) [33]
to establish a hierarchical representation framework. Although DWT is commonly
employed in various low-level vision tasks [45, 64, 5], prior works
predominantly focus on utilizing it to compute frequency-domain features and
just perform image-frequency fusion, overlooking the statistical correlation
of wavelet coefficients. In contrast, we innovatively incorporate channel
attention and spatial self-attention to implicitly capture inter- and intra-
scale relationships among subbands. Our approach explores the potential of DWT
as a hierarchical representation, offering advantages such as inter-/cross-
scale interactions, flexibility, improved computational efficiency and
suitability, which is the first time addressed for high-resolution HDR
deghosting.

Still, an additional challenge emerges: how to efficiently extract the most
informative features and establish cross-hierarchy interactions for effective
fusion of all available information. To meet this challenge, we devise a
gradual fusion strategy. Initially, we undertake an in-depth analysis
incorporating global and local attention mechanisms. Subsequently, we
systematically progress to the fusion of information across different
hierarchies. This progressive approach aims to capture nuanced details and
global context, ensuring the comprehensive utilization of our hierarchical
representation.

As shown in Fig. 1, PASTA sets new SOTA records on HDR benchmarks, validating
its effectiveness and versatility. It also excels in efficiency, with fast
inference speeds, low GPU memory usage, and reduced latency. Remarkably, PASTA
outperforms existing SOTA methods in processing 1080P LDR images on a plain
GPU with just 12GB of RAM, achieving a substantial 3-fold ($\times 3$)
increase in inference speed. An ultra-lightweight version further enhances
this, boosting speed by nearly 9 times ($\times 9$) while still maintaining a
competitive level of performance.

##  2 Related Work

Rejection-based methods. These methods aim to identify and reject moving
regions inconsistent with a reference frame’s background during synthesis.
Various techniques, such as non-parametric estimation [22], local entropy
[18], threshold bitmap [40, 32], intensity histogram [36], and bi-directional
similarity [68, 25], detect moving pixels. However, they often lose fine
details in dynamic areas and struggle to adapt to different cameras and
exposure settings due to fixed threshold reliance [12].

Registration-based methods. The emphasis is very much on aligning exposure
images. In the case of only camera motion between exposure sources, _i.e_.,
the captured scene is rigid and on a plane, simple operations such as
translation [56], homography [51], correlation matching [1], and phase cross
correlation [62] can be used to crack this challenge. For dynamic scenes, most
methods seek non-grid dense correspondences (_e.g_., joint homography and SIFT
[14], joint homography and optical flow [21], and RANSAC [16, 11]) between
images. However, the warped images are prone to holes in the presence of large
displacements or severe occlusion.

Optimization-based methods. They jointly perform alignment and reconstruction
in the form of an energy function, or only use the framework to perform one of
them, for example, using Poisson equation [8, 10], graph cut [7], Markov
random field [19, 12], Bayesian [32], energy-based optic flow [69], PatchMatch
[47, 17], rank minimization [23, 39] and background estimation [13]. Despite
progress, they fall short of learning-based methods in deghosting results and
tend to be time-consuming.

Learning-based methods. They have evolved since [20], which uses optical flow
[27] for image alignment in HDR creation. Subsequent works directly cascaded
features [57, 38, 63] or utilize attention mechanisms [59, 35, 3, 24, 67] for
implicit motion alignment, bypassing traditional optical flow techniques.
Additionally, some methods integrate optical flow networks [2] or deformable
convolutions [30] for spatial feature alignment. Non-local attention [60] and
recurrent networks [41] have been used for large-scale motion modeling. Along
with distinctive merits of Transformer [53], _i.e_., long-range dependencies,
which is particularly suitable for capturing large object displacement, recent
deghosting approaches based on Transformers [31, 48, 58, 28, 61, 50] leverage
the traits for large object displacement, with CA-ViT [31] and SCTNet [50] as
new baselines using Swin Transformer block [29]. However, these methods face
challenges: 1) losing Swin Transformer’s significantly hierarchical advantage;
2) visible block artifacts in tone-mapped results—the resulting to partition
input images into blocks (_e.g_., 256$\times$256) for inference, as 2K images
cannot be processed on limited-memory GPUs. The latest GDP [9] employs a pre-
trained denoising diffusion probabilistic model (DDPM) [15] for image
restoration, including HDR, but it’s limited by less realistic hole filling in
occluded areas and slow inference speed.

##  3 Proposed Method

###  3.1 Preliminaries

Let $\left\\{{{I_{i}}}\right\\}_{i=1}^{n}$ be a set of LDR images with varying
exposure times $\left\\{{{t_{i}}}\right\\}_{i=1}^{n}$. If the input is not in
RAW format, they are transformed into the HDR domain via gamma correction,
defined by the power-law expansion:

${H_{i}}={{I_{i}^{\gamma}}\mathord{\left/{\vphantom{{{I_{i}}^{\gamma}}{{t_{i}}}}}\right.\kern-1.2pt}{{t_{i}}}},$
(1)

where $I_{i}$ is raised to a gamma value $\gamma>1$ and divided by time
${t_{\emph{i}}}$ to get the corresponding image ${H_{i}}$. In the common case
of $\gamma=2.2$, inputs and outputs are typically normalized to the interval
$\left[{0,1}\right]$. Similar to previous works [20, 57], we take
$\left\\{{{X_{i}}}\right\\}_{i=1}^{\emph{n}}$ as the input, where
${X_{i}}=\left[{{I_{i}},{H_{i}}}\right]$ concatenates $I_{i}$ and $H_{i}$
along the channel dimension. The acquisition of HDR imaging, _H_ , is to find
a mapping function $\boldsymbol{\varphi}$ with respect to the parameter
$\theta$, satisfying:

$H=\boldsymbol{\varphi}\left({{X_{1}},{X_{2}},\cdots,{X_{n}};\theta}\right).$
(2)

In this work, we also set $n=3$ and use ${X_{2}}$ with a medium exposure as
the reference frame, so that the motion content in the resulting HDR image _H_
is consistent with it.

Figure 2: Framework Overview. The proposed framework mainly consists of three
stages, _i.e_., shallow feature extraction, inter-frame temporal attention,
and hierarchical representation & progressive aggregation.

###  3.2 Overall Pipeline

Given a set of LDRs $\left\\{{{X_{i}}}\right\\}_{i=1}^{3}$, we utilize three
separate 3$\times$3 convolutions to extract corresponding shallow features
${F_{i}}\in\mathbb{R}^{H\times W\times C}$, where
${X_{i}}\in\mathbb{R}^{H\times W\times 6}$, $H\times W$ denotes the resolution
of the image, and _C_ denotes the number of channels. The architecture of
PASTA, as depicted in Fig. 2, comprises two key components, _i.e_., Inter-
Frame Temporal Attention and Hierarchical Representation & Progressive
Aggregation.

###  3.3 Inter-Frame Temporal Attention

The exposure stack captures the temporal information across frames. Previous
works [59, 31] utilize spatial attention between reference and non-reference
frames to suppress motion and saturation. Different from them, we replace
spatial attention with temporal attention [55] because 1) different regions
across frames are not equally informative due to exposure variations,
occlusions, and moving objects. 2) The spatial attention is implemented by
cascading the reference and non-reference frames instead of directly
calculating their similarity, and then performing convolution with sigmoid
activation to obtain the attention map. In fact, this computation way does not
easily discern the differences between frames when both motion and saturation
coexist [58]. 3) The computation of spatial attention can be regarded as a
local form of self-attention mechanism.

IFTA module is illustrated in Fig. 2(a) with the aim of computing attention
features between adjacent frames in the embedding space. For shallow feature
${F_{i}}\in{\mathbb{R}^{H\times W\times C}}$, $i\in\left\\{{1,2,3}\right\\}$,
the similarity matrix $S\in{\mathbb{R}^{H\times W}}$ is obtained by:

${S_{{F_{i}},{F_{2}}}}\left({x,y}\right)=\left\langle{{\phi_{i}}\left({x,y}\right),{\phi_{2}}\left({x,y}\right)}\right\rangle,$
(3)

where $x$ and $y$ are feature coordinates, ${\phi_{i}}$ and ${\phi_{2}}$ are
two embeddings of corresponding features ${F_{i}}$ and ${F_{2}}$, achieved via
a convolution operation, and $\left\langle{\cdot,\cdot}\right\rangle$ denotes
the element-wise inner product along the channel dimension. After sigmoid
function and channel expansion operations, we can obtain the attention map
${S^{\prime}_{{F_{i}},{F_{2}}}}\in{\mathbb{R}^{H\times W\times C}}$. The
contribution of the sigmoid function lies in: i) highlighting misaligned and
saturated feature regions; ii) By limiting the inner product output to
$[0,1]$, the function keeps gradients in check during backpropagation,
enhancing training stability; iii) improving convergence efficiency. The
similarity in the Eq. 3 belongs to an element-wise local attention mechanism.
It is noted that Eq. 3 not only includes cross-similarity but also self-
similarity for reference feature. This differs from spatial attention-based
methods [59, 31] which indiscriminately concatenate the reference feature
directly into the attention features. Then, temporal attention features
$\left\\{{{F_{i,t}}}\right\\}_{i=1}^{3}$ can be obtained by element-wise
multiplication of ${F_{i}}$ and ${S_{{F_{i}},{F_{2}}}}$, defined as
$F_{i,t}={F_{i}}\odot{S^{\prime}_{{F_{i}},{F_{2}}}}$.

Lastly, the temporal attention features mentioned above are concatenated by
${F_{t}}={\rm{Cat}}\left({{F_{1,t}},{F_{2,t}},{F_{3,t}}}\right)$.

(a) 1st level

(b) 2nd level

(c) 3rd level

(d) Inter-scale

(e) Intra-scale

Figure 3: (a)-(c) Feature Channels Correlation between subband coefficients of
three levels of wavelet decomposition measured by Pearson correlation
coefficient (PCCs). The channel dimension of each subband coefficient is 48.
In each subfigure, from the top left to the bottom right, the correlations
among the channels of the subband coefficients
$\left\\{{L{L_{k}},L{H_{k}},H{L_{k}},H{H_{k}}}\right\\}_{k=1}^{3}$ themselves
are shown. (d)-(e) Dependencies between wavelet coefficients across different
scales give rise to a quad-tree structure, where not only the parents but also
the children coefficients contain pertinent information.

###  3.4 Hierarchical Representation & Progressive Aggregation

To reduce the computational complexity of self-attention module, many works
utilize pooling [54], patch merging [29], and pixel-shuffle [65] for down-
sampling feature maps. These operations nevertheless are either irreversible
[54, 29], which potentially lead to a drop in the capability of feature
representation, especially for low-level pixel-wise visual tasks, or still
suffer from excessive memory in case of high-resolution images due to the
symmetric U-Net architecture [65]. Among of them, Swin [29] has demonstrated
that hierarchy is of vital importance in vision tasks. In addition to the
parallel computation achieved by shift window, the superiority of Swin over
ViT [6] is mainly attributed to this paramount trait. However, existing
Transformer-based deghosting works [31, 28, 58, 61], as well as SwinIR [26]
represented by the use of Swin blocks, overlook or do not fully inherit the
character of hierarchical representation.

The attractive properties of the wavelet transform enable it to well represent
and decouple features, including locality, multi-resolution, compression,
clustering and persistence [4, 46]. Figs. 3(d) and 3(e) show the magnitude
values of wavelet coefficients at three scales and indicate that subbands are
not statistically independent from visual inspection, Due to the non-
Gaussianity of wavelet marginals, previous works have primarily utilized
hidden Markov models [46, 4] to characterize the statistical dependencies
between those coefficients. In this work, our objective is to develop a module
to implicitly explore the relationships among subbands, which is rich and
flexible enough to represent coefficient characteristics yet concise,
tractable and efficient for high-resolution images.

####  3.4.1 Wavelet Representation.

DWT, as a classic technique for multi-resolution, decomposes a 2D image into
four subband components: one low-frequency band, LL, and three high-frequency
ones, _i.e_., LH, HL, and HH, with half the resolution of the source. The DWT
is reversible, and its inverse, denoted as IDWT. As shown in Fig. 2(b), let
$L{L_{0}}={F_{t}}$ and we recursively perform the DWT on the low-frequency
component $\left\\{{L{L_{k}}}\right\\}_{k=0}^{K-1}$ at each scale _k_ ,
obtaining a set of a set of high-frequency residual components
$\left\\{{L{H_{k}},\emph{H}{L_{k}},H{H_{k}}}\right\\}_{k=1}^{K}$ and the
coarset low-frequency approximated one $L{L_{K}}$. The low-frequency features
identify the motion between frames, illumination variations, and color
information, while the high-frequency ones capture contour information and
rapid changes in appearance. As the decomposition level _k_ increases, the
resolution of the corresponding high- and low-frequency components decreases
from $H\times W$ to $\frac{H}{{{2^{k}}}}\times\frac{W}{{{2^{k}}}}$ while
keeping the channel dimension unchanged. Compared to the U-Net architecture of
Restormer [65], our representation structure has no additional learnable
parameters except for the fixed ones of the wavelet filters in terms of
hierarchical representation. Furthermore, the channel dimension of Restormer
doubles as the decomposition levels increase.

####  3.4.2 Coarse-to-Fine Aggregation.

The wavelet representation not only expands the receptive field for the
aggregation stage, which will foster ghost removal for large object
displacement and large-disparity alignment, but also achieves feature
decoupling, specifically for low- and high-frequency components. Recalling the
remarkable properties of wavelet coefficients and the dependency on subband
coefficients stated in Sec. 3.4, this inspires us to implicitly explore and
utilize the statistical relationship. Figs. 3(a), 3(b) and 3(c) illustrates
the correlation coefficients between different frequency subband channels
measured by PCCs. Correlation and redundancy among wavelet subband channels
are evident. As the levels of DWT decomposition increase, the inter-channel
correlation gradually intensifies.

As shown in Fig. 2(b), after three levels of DWT, we obtain the final low-
frequency subband $L{L_{3}}$ and the corresponding detail ones
$\left\\{{L{H_{3}},H{L_{3}},H{H_{3}}}\right\\}$. Taking into account the
correlation between different frequency bands and the redundancy of subband
features, we process them by concatenating the high and low-frequency subbands
after wavelet decomposition. Another purpose of joint processing is to prevent
the loss of spatial coherence between low and high frequencies when they are
processed separately. It is common for the low frequencies to remove ghosting,
but the high frequencies fail to handle the boundaries or contours of moving
objects effectively, thereby resulting in ghosting. The developed progressive
aggregation strategy for large-scale dynamic scenes mainly includes four
stages. The core module is composed of a residual channel attention block
(RCAB) [66] and a residual Swin Transformer block (RSTB) [26], where RCAB is
used to explore inter-subband dependencies, followed by RSTB to build long-
range spatial dependencies. The first three stages involve aggregating within
the space of wavelet representation with spatial contextual information. From
bottom to top, _i.e_., from global to local, our progressive aggregation
framework iteratively upscales and refines the estimates from the previous
level by predicting the high- and low-frequency coefficients. The last stage
operates in the original feature space (without downsampling operations),
thereby preserving fine details in the final reconstructed HDR image.
Specifically, the reconstructed features $L{{L^{\prime}}_{0}}$ are
concatenated with the original features $F_{t}$. Then a 1$\times$1 convolution
is used to achieve channel dimension compression, followed by passing through
an RSTB module.

Skip Connection. To construct a ghost-free HDR image, we have developed a
model with a long skip connection extending from the IFTA feature $F_{t}$.
This connection refines spatio-temporal aggregated features by merging the
reconstructed wavelet coefficient features from the initial three stages with
the finest features of the last stage. We have named this model PASTA-I,
highlighting its capability for implicit alignment, which significantly
reduces motion blur and ghosting effects in case of shooting moving scenes.

####  3.4.3 w and w/o IFTA Module.

As the proposed model PASTA-I without IFTA strategy will simplify to cascading
shallow features directly, akin to [57, 38], this subsection emphasizes the
comparison of the developed variant with them. As we previously discussed, 1)
Both methods [57, 38] are vulnerable to the cases with occlusions and large
moving objects. 2) Self-attention inherently possesses alignment capabilities
to some extent. In contrast, applying the self-attention mechanism to [57,
38], _i.e_., our alignment-free variant PASTA-U, is competent to handle these
scenarios. To circumvent the generation of ghosting artifacts, PASTA-I’s long
skip ${F_{t}}$ is replaced in PASTA-U with ${{\mathop{\rm
Conv}\nolimits}_{3\times 3}}\left({{F_{2}}}\right)$, which considers the
motion in intermediate frames to direct network learning. The results from
PASTA-U show that: i) Self-attention can directly leverage unaligned feature
information, and ii) A hierarchical structure can facilitate the enhancement
of the self-attention mechanism’s ability to utilize unaligned feature
information. This is actually evidenced by the fact that Swin [29] outperforms
ViT [6].

###  3.5 Loss Function

We use $\mu$-law function to map the HDR image from the linear domain to the
tonemapped domain through $\mathcal{T}\left(x\right)=\frac{{\log\left({1+\mu
x}\right)}}{{\log\left({1+\mu}\right)}}$,

where $\mathcal{T}\left(\cdot\right)$ is the tone mappting function,
$\mu=5000$. In the tonemapped domain, in addition to the previous loss
functions [31, 58] that consist of $\mathcal{L}_{1}$ loss and perceptual loss
${\mathcal{L}_{p}}$, we further incorporate a edge loss ${\mathcal{L}_{e}}$ to
surpass the edge blurring possibly caused by the hierarchical representation,
which is defined as:

${\mathcal{L}_{\emph{e}}}=\sqrt{{{\left\|{\nabla\left({\mathcal{T}(H)}\right)-\nabla\left({\mathcal{T}(\hat{H})}\right)}\right\|}^{2}}+{\varepsilon^{2}}},$
(4)

where _$\hat{H}$_ is the estimated HDR image, and the constant $\varepsilon$
is empirically set to ${10^{-3}}$.

Overall, our loss function is
$\mathcal{L}={\mathcal{L}_{1}}+\alpha{\mathcal{L}_{p}}+\beta{\mathcal{L}_{e}}$,
where $\alpha$ and $\beta$ are set to 0.01 and 1, respectively.

##  4 Experiments

###  4.1 Experimental Settings

Datasets. We train the proposed model on Kalantari  _et al_. ’s [20] and Tel
_et al_. ’s [50] datasets, which consist of 74/15 training/testing scenes and
108/36 counterparts, respectively.

We first crop the images in the training set into image patches of size
128$\times$128 with a stride of 64, and then perform a clockwise rotation of
90∘, and horizaontal and vertical flips for data augmentation. To validate the
effectiveness of the proposed method, we also conduct testing on Sen _et
al_.’s [47] and Tursen _et al_.’s [52] datasets.

Metrics. We use five commonly used metrics, _i.e_., PSNR-_l_ , PSNR-$\mu$,
SSIM-_l_ , SSIM-$\mu$, and HDR-VDP-2 [34], to evaluate the performance of
different methods. Here, ‘-_l_ ’ and ‘-$\mu$’ refer to the calculation of the
respective metrics in the linear and tonemapped domain, respectively. HDR-
VDP-2 is a visual metric tailored for HDR and its two parameters are set to 25
and 0.5, respectively, for the Kalantari _et al_.’s dataset. To ensure a fair
comparison, we follow the same parameter settings [50] on Tel _et al_.’s
dataset.

Implementation Details. Our model is implemented based on PyTorch and uses the
Adam optimizer with an initial learning rate of 2$e$-4, where ${\beta_{1}}$,
${\beta_{2}}$, and $\varepsilon$ are set to 0.9, 0.999, and 1$e$-8,
respectively. The total number of iterations is 300K, and the learning rate is
halved every 50K ones. We set the batch size to 16 and trained on 8 NVIDIA
TITAN Xp GPUs. Tab. 2 lists the parameter configurations, highlighting the
tiny version’s key differences in channel dimension, Swin Transformer layers
(STL), and multi-head self-attention heads.

Table 2: Model architecture configuration. “ _C_ ”, “Reduction” and “Win. sz.”
refer to the channel dimension in shallow feature ${F_{i}}$, the channel
reduction factor for each RCAB, and the window size in the RSTB, respectively.

Model | _C_ | Num. of STL | Num. of heads | Reduction | MLP ratio | Win. sz. | #Param.
---|---|---|---|---|---|---|---
PASTA-I | 16 | [6, 6, 6, 6] | [4, 4, 4, 4] | [4, 4, 4] | 2 | 8 | 8.163M
PASTA-I-Tiny | 12 | [2, 2, 2, 2] | [3, 3, 3, 3] | [4, 4, 4] | 2 | 8 | 2.636M
PASTA-U | 16 | [6, 6, 6, 6] | [4, 4, 4, 4] | [4, 4, 4] | 2 | 8 | 8.165M
PASTA-U-Tiny | 12 | [2, 2, 2, 2] | [3, 3, 3, 3] | [4, 4, 4] | 2 | 8 | 2.637M

###  4.2 Comparison with SOTAs

We evaluate PASTA against 16 SOTA methods that include Sen _et al_. [47], Hu
_et al_. [17], Kalantari _et al_. [20], DeepHDR [57], AHDRNet [59], PAMNet
[44], NHDRRnet [60], PSFNet [63], FSHDR [43], HDRRNN [41], HDR-GAN [38], APNT
[3], FHDRNet [5], STHDR [48], CA-ViT [31], and SCTNet [50].

Table 3: Quantitative comparison with state-of-the-art methods on Kalantari
_et al_.’s [20] and Tel _et al_.’s [50] datasets. “-” indicates that the code
or pretrained weight is not available, and the parameters for calculating HDR-
VDP-2 (H.V.-2) are unspecified. The top two performers under each metric are
highlighted in red and blue, respectively.

Dataset | Method | Venue | H.V.-2 $\uparrow$ | Linear Imgs. | Tonemapped Imgs.
---|---|---|---|---|---
PSNR-_l_ $\uparrow$ |  SSIM-_l_ $\uparrow$ |  PSNR-$\mu$$\uparrow$ |  SSIM-$\mu$$\uparrow$
Kalantari |  Sen _et al_. [47] | TOG’12 | 59.38 | 38.11 | 0.9721 | 40.80 | 0.9808
Hu _et al_. [17] | CVPR’13 | - | 30.76 | 0.9503 | 35.79 | 0.9717
Kalantari _et al_. [20] | TOG’17 | 64.18 | 41.23 | 0.9846 | 42.67 | 0.9888
DeepHDR [57] | ECCV’18 | 64.90 | 40.88 | 0.9858 | 41.65 | 0.9860
AHDRNet [59] | CVPR’19 | 65.61 | 41.14 | 0.9702 | 43.63 | 0.9900
PAMNet [44] | ACCV’20 | - | 41.65 | 0.9870 | 43.85 | 0.9906
NHDRRnet [60] | TIP’20 | 64.08 | 41.08 | 0.9861 | 42.41 | 0.9887
PSFNet [63] | MM’21 | - | 41.57 | 0.9867 | 44.06 | 0.9907
HDRRNN [41] | TCI’21 | - | 41.68 | - | 42.07 | -
HDR-GAN [38] | TIP’21 | 65.54 | 41.57 | 0.9865 | 43.92 | 0.9905
APNT [3] | TIP’22 | - | 41.61 | 0.9879 | 43.94 | 0.9898
FHDRNet [5] | CVIU’24 | 65.71 | 41.47 | 0.9869 | 43.91 | 0.9908
STHDR [48] | ECCV’22 | - | 41.70 | 0.9872 | 44.10 | 0.9909
CA-ViT [31] | ECCV’22 | 65.66 | 42.18 | 0.9884 | 44.32 | 0.9916
SCTNet [50] | ICCV’23 | 64.48 | 42.29 | 0.9887 | 44.49 | 0.9924
PASTA-I | Ours | 65.92 | 42.50 | 0.9899 | 44.53 | 0.9918
| PASTA-U | Ours | 65.86 | 42.45 | 0.9895 | 44.49 | 0.9917
Tel |  Kalantari _et al_. [20] | TOG’17 | 67.09 | 43.37 | 0.9924 | 40.05 | 0.9794
NHDRRnet [60] | TIP’20 | 65.41 | 39.61 | 0.9853 | 36.68 | 0.9590
AHDRNet [59] | CVPR’19 | 68.80 | 45.30 | 0.9943 | 42.08 | 0.9837
FHDRNet [5] | CVIU’24 | 69.86 | 45.80 | 0.9948 | 42.41 | 0.9858
CA-ViT [31] | ECCV’22 | 69.23 | 46.35 | 0.9948 | 42.39 | 0.9844
SCTNet [50] | ICCV’23 | 70.66 | 47.51 | 0.9952 | 42.55 | 0.9850
PASTA-I | Ours | 71.30 | 48.24 | 0.9961 | 43.53 | 0.9883
PASTA-U | Ours | 71.20 | 48.16 | 0.9961 | 43.62 | 0.9884

Quantitative Comparison. Tab. 3 shows that PASTA-I and PASTA-U perform
competitively on two benchmarks, excelling particularly on the latest [50].
Notably, both models significantly outperform the latest baseline, CA-ViT, by
margins of 0.32dB/0.26dB and 0.21dB/0.17dB in PSNR-_l_ /PSNR-$\mu$ on
Kalantari  _et al_.’s dataset. Additionally, we have observed that HDR-VDP-2
metrics do not consistently exhibit proportionality to PSNR values. On Tel _et
al_.’s dataset, our methods surpass top contenders like SCTNet and CA-ViT
across all metrics. PASTA-U’s performance suggests that self-attention allows
for effective ghost-free HDR image reconstruction even with some large
misalignment. It should be noted that some of the values in Tab. 3 are derived
from [58, 31, 50].

Qualitative Comparison. Figs. 4 and 5 show two challenging scenes: large-scale
foreground motion and significant camera motion with large disparities. The
coexistence of occlusions and saturation is inevitable and intractable in the
presence of exposure and motion variations. In such case, non-attention or
non-transformer-based methods like the approaches of Sen _et al_. and
Kalantari _et al_., DeepHDR, NHDRRnet, HDRRNN and FSHDR struggle with ghosting
artifacts due to unreliable alignment (_e.g_., optical flow, homography) or
lack of feature alignment, often failing in fine detail preservation, such as
hair in Fig. 4. Additionally, the patch-based method of Sen _et al_. incurs
noticeable halos. In contrast, our method surpasses others like CA-ViT, SCT-
Net and AHDRNet in ghosting removal, saturation handling, detail retention,
and color consistency. In window scenes with significant disparities,
approaches like Kalantari _et al_. and NHDRRnet suffer content loss in
saturated areas, while CA-ViT and SCTNet struggle with fine detail recovery
and noise suppression, as shown in Fig. 5’s red close-up regions. By
comparison, our method achieves relatively noise-free results in these areas.
The proposed method, especially PASTA-I, also preserve edge sharpness and
intensity in smooth regions better than competitors like CA-ViT and SCTNet,
evident in the green close-ups in Fig. 5. For additional results, please refer
to the supplementary materials.

\begin{overpic}[width=390.25534pt,tics=2]{Figs/Experiment/VisualCompare/SIG17/Sig17.pdf} \end{overpic} Figure 4: Visual comparison of large-scale foreground and dense background motion with SOTA methods on Kalantari _et al_.’s dataset [20]. \begin{overpic}[width=411.93767pt,tics=2]{Figs/Experiment/VisualCompare/ICCV23/SCT23_new.pdf} \par\end{overpic} Figure 5: Visual comparison of large disparities with SOTA methods on Tel _et al_.’s dataset [50]. The bottom row displays 1D intensity shift of the green close-up regions labeled with the line. Zoom in for better viewing. Table 4: Ablation study including IFTA, RCAB, ${\mathcal{L}_{p}}$ and ${\mathcal{L}_{e}}$ on the Kalantari _et al_. ’ dataset [20] using PASTA-I (Ours). “SA vs. IFTA” refers to IFTA being replaced with spatial attention. Metrics |  SA vs. IFTA |  w/o IFTA |  w/o RCAB |  w/o ${\mathcal{L}_{p}}$ |  w/o ${\mathcal{L}_{e}}$ | PASTA-U | PASTA-I
---|---|---|---|---|---|---|---
PSNR-_l_ | 41.86 | 42.33 | 42.19 | 42.61 | 42.45 | 42.45 | 42.50
SSIM-_l_ | 0.9897 | 0.9896 | 0.9895 | 0.9897 | 0.9893 | 0.9895 | 0.9899
PSNR-$\mu$ | 44.40 | 44.45 | 43.96 | 44.46 | 44.40 | 44.49 | 44.53
SSIM-$\mu$ | 0.9918 | 0.9916 | 0.9917 | 0.9919 | 0.9917 | 0.9917 | 0.9918

###  4.3 Ablation Study

Table 5: Ablation studies of sampling methods and DWT scales. “PUS vs DWT”
indicates using PixelUnshuffle (PUS) [65] to replace DWT for downsampling, and
Pixelshuffle to replace IDWT for upsampling. “Conv. vs DWT” denotes the
substitution of the DWT with convolution, as used in [57, 60, 5]. “$K$” is the
scale of the hierarchical representation structure.

Settings | Variant |  Kalantari  _et al_.’s [20] |  1000$\times$1500 |  2560$\times$1440
---|---|---|---|---
#Imgs-_l_ |  #Imgs-$\mu$ |  G. Mem. | Time |  G. Mem. | Time
Sampling methods | PUS vs. DWT | 37.19/.9775 | 40.23/.9852 | 7.729 | 2.353 | OOM | -
Conv. vs. DWT | 42.07/.9880 | 44.52/.9916 | 7.725 | 2.516 | 18.399 | 5.871
1-8[1pt/1pt] DWT scales |  PASTA-I ($K$=1) | 42.20/.9887 | 44.46/.9917 | 7.298 | 2.382 | 17.464 | 5.520
PASTA-I ($K$=2) | 42.39/.9893 | 44.42/.9917 | 7.629 | 2.511 | 17.968 | 5.639
PASTA-I ($K$=4) | 42.31/.9895 | 44.35/.9916 | 7.869 | 2.550 | 19.584 | 5.942
1-8[1pt/1pt]  Our plain version with $K$=3 | PASTA-I | 42.50/.9899 | 44.53/.9918 | 7.850 | 2.543 | 18.743 | 5.891
PASTA-I-Tiny | 42.25/.9890 | 44.46/.9917 | 6.015 | 0.829 | 14.399 | 1.889

Table 6: Comparison of GPU memory (G) consumption and average inference time
(seconds) for three LDR images with different resolutions, _i.e_.,
1000$\times$1500 and 2560$\times$1440 (2K), implements on a RTX 3090 GPU
(24G). OOM and ‘-’ refer to the out-of-memory and being unavailable,
respectively.

Method |  Tel  _et al_.’s [50] |  Kalantari  _et al_.’s [20] |  1000$\times$1500 |  2560$\times$1440
---|---|---|---|---
#Imgs-_l_ |  #Imgs-$\mu$ |  #Imgs-_l_ |  #Imgs-$\mu$ |  G. Mem. | Time |  G. Mem. | Time
AHDRNet [59] (CVPR’19) | 45.30/.9943 | 42.08/.9837 | 41.14/.9702 | 43.63/.9900 | 9.069 | 0.348 | OOM | -
FHDRNet [5] (CVIU’24) | 45.80/.9948 | 42.41/.9858 | 41.47/.9869 | 43.91/.9908 | 5.643 | 0.329 | 13.449 | 0.985
1-9[1pt/1pt] CA-ViT [31] (ECCV’22) | 46.35/.9948 | 42.39/.9844 | 42.18/.9884 | 44.32/.9916 | 10.637 | 4.631 | OOM | -
SCTNet [50] (ICCV’23) | 47.51/.9952 | 42.55/.9850 | 42.29/.9887 | 44.49/.9924 | 9.732 | 6.621 | OOM | -
1-9[1pt/1pt] PASTA-I | 48.24/.9961 | 43.53/.9883 | 42.50/.9899 | 44.53/.9918 | 7.850 | 2.418 | 18.743 | 5.969
PASTA-I-Tiny | 48.08/.9957 | 43.45/.9878 | 42.25/.9890 | 44.46/.9917 | 6.015 | 0.867 | 14.399 | 2.071
1-9[1pt/1pt] PASTA-U | 48.16/.9961 | 43.62/.9884 | 42.45/.9895 | 44.49/.9917 | 7.850 | 2.242 | 18.743 | 5.765
PASTA-U-Tiny | 48.08/.9960 | 43.51/.9879 | 42.07/.9883 | 44.43/.9916 | 6.015 | 0.773 | 14.402 | 2.036

Here, taking PASTA-I as an example, we validate our framework’s efficacy via
IFTA, RCAB, edge loss, among others, as detailed in Tabs. 4 and 5.

Effectiveness of IFTA. To validate the effectiveness of IFTA, we replaced it
with spatial attention[59, 31], _i.e_., SA vs. IFTA, and directly removed it,
_i.e_., w/o IFTA, respectively. The performance of the variant based on
spatial attention is slightly lower than w/o IFTA and our full model,
especially in terms of PSNR-_l_ by a margin of 0.47dB and 0.64dB,
respectively. As we mentioned in Sec. 3.3, i) The indiscriminate concatenation
of reference frame features into attention features by spatial attention may
degrade the network performance in case of occlusions and saturation
coexisting in the reference frames; ii) The computation of spatial attention
can be seen as a localized form of self-attention mechanism, which also
indicates that self-attention has a certain alignment effect.

Effectiveness of RCAB. The RCAB module is integral to our framework, focusing
on the dependencies both within and across scales of wavelet subband
coefficients. When RCAB is removed, we observe a significant reduction in both
PSNR-_l_ and PSNR-$\mu$ by 0.31dB and 0.57dB. This indicates that the
superiority of our framework lies in its implicit statistical modeling of
wavelet coefficients.

Effectiveness of Edge Loss. The variant w/o ${\mathcal{L}_{p}}$, _i.e_., only
$\mathcal{L}_{1}$ and ${\mathcal{L}_{e}}$, achieved a higher PSNR-_l_ of
42.61dB, suggesting that this loss prevents edge blurring during the process
of progressive aggregation, thereby reducing or avoiding halo artifacts.
Additionally, its SSIM-$\mu$ slightly improved to 0.9919, indicating a
potential trade-off between the objective quality of exposure and the
perceived quality of texture and structure. The variant, w/o
${\mathcal{L}_{e}}$, falls slightly behind in all metrics compared to the
variant w/o ${\mathcal{L}_{p}}$. Our full model indicates that
${\mathcal{L}_{e}}$ plays a subtle yet tangible role in reconstructing high-
frequency detail, while edge loss in preserving edge sharpness.

Sampling methods and DWT scales. We also replace the DWT operation with
alternative sampling methods. The results, listed in the upper section of Tab.
5, affirm our efficacy. In comparison with Conv-based methods, our PASTA-I-
tiny achieved a +0.18dB improvement with only 77% memory utilization,
simultaneously accelerating inference time by a factor of $\times$3.

###  4.4 Efficiency Analysis

Wavelet Representation. Efficiency is achieved through the use of DWT for
feature extraction. DWT requires no additional learned parameters and
isotropic feature extraction avoids increasing channel capacity. It
efficiently captures multi-scale information.

Progressive Hierarchical Aggregation. Our progressive hierarchical aggregation
strategy speeds up the network by processing lower-resolution representations
and iteratively refining them. This approach efficiently retains essential
information while discarding redundant details.

Quantitative Results. Tab. 6 reports GPU memory consumption (G.
Mem.)111Computed by PyTorch torch.cuda.max_memory_allocated() function. and
average inference time of our methods and four baselines on three LDR inputs
of different resolutions, calculated over 100 iterations. Taking the
resolution of 1000$\times$1500 as an example, benefiting from the hierarchical
representation and progressive aggregation, the proposed method achieves GPU
costs of $74\%$ compared to CA-ViT and $80\%$ to SCTNet; the inference speed
is nearly 2.8 times faster than SCTNet, 2 times than CA-ViT. When inputs are
2K, aside from the FHDRNet, none of the aforementioned baselines can directly
perform inference. Tabs. 3 and 6 indicate that the proposed method achieves
the best balance in inference time, GPU memory usage and reconstruction
performance compared to SOTA methods, especially those [31, 50, 58] utilizing
Transformer.

\begin{overpic}[width=433.62pt]{Figs/Experiment/Discussion/occlusion_and_saturation_last.pdf}
\put(4.0,-1.0){\tiny{(a) Short}} \put(17.0,-1.0){\tiny{(b) Middle}}
\put(32.0,-1.0){\tiny{(c) Long}} \put(47.0,-1.0){\tiny{(d) GT}}
\put(57.0,-1.0){\tiny{ (e) CA-ViT
\cite[cite]{[\@@bibref{}{liu2022ghost}{}{}]}}} \put(72.5,-1.0){\tiny{(f)
PASTA-I}} \put(83.5,-1.0){\tiny{(g) PASTA-I (ref. a)}} \end{overpic} Figure 6:
Twp types of occlusion, _i.e_., compensatory and non-compensatory occlusion
(NCO). The former denotes the provision of beneficial information in the
corresponding regions between frames, _e.g_., the enlarged red region, while
the later the simultaneous presence of occlusion and saturation in the
corresponding regions between frames _e.g_., the first three close-up regions.
NCO can be effectively utilized by re-selecting the reference frame from the
perspective of maximizing information (_e.g_., entropy), rather than
attempting to hallucinate or fill in the occluded regions. Among them,
“PASTA-I (ref. a)” is generated using the proposed method, where “short” is
selected as the reference frame with entropy-maximization in the exposure
sequence. This strategy ensures a high-fidelity result without ghosting.

###  4.5 Discussion

Occlusion Coexist with Saturation. In both our method and CA-ViT [31], pixel
recovery encounters hurdles when occlusions coincide with saturation, as
depicted in Fig. 6, mirroring challenges observed in human perception. Despite
these obstacles, our model outperforms CA-ViT, generating more realistic
estimates. In fact, the effective generation of high-fidelity HDR images
without ghosting benefits from neighboring frames compensating for occlusions,
illustrated in the pink and red regions. Complications typically arise when
the corresponding region in neighboring frames is saturated, hindering
realistic content hallucination, as seen in the blue and green areas. While
techniques like diffusion models [15, 9] or inpainting may be applied, they
may not produce realistic results. For such case, leveraging occlusion rather
than attempting to hallucinate could be a practical approach. Opting for a
frame with maximum information content (_e.g_., entropy) as the reference from
the exposure sequence can be advantageous. Success with image entropy in
measuring pixel uncertainty [18] supports this strategy. When constructing
datasets or developing unsupervised models, prioritizing frames with maximum
information content as references may alleviate occlusion effects.

Potential Drawback to Benchmarks. Despite some SOTA methods achieving high
PSNR-_l_ values on benchmarks [20, 50, 28], significant deviations to around
30dB are observed in certain test cases. This discrepancy is attributed to
non-compensatory occlusions and dense subtle motion, challenging pixel-wise
evaluation metrics. Wind-induced subtle motion during outdoor shooting,
impacting scenes like tree branches, leaves, grass, clouds, and flowing water,
can have a substantial effect. This requires a further in-depth reflection on
the GT generation approach.

##  5 Conclusions

This work proposes a flexible and efficient framework, PASTA, to address
addresses HDR deghosting’s computational challenges, which optimizes the
trade-off between performance and computational efficiency, particularly at
high resolutions. By integrating hierarchical feature decoupling and
progressive aggregation, PASTA significantly boosts inference speed, achieving
a significant threefold improvement, and can even reach a ninefold increase
without sacrificing image quality. The comprehensive experiments validate that
PASTA not only rivals but surpasses the existing SOTA methods in performance
metrics, all while streamlining the use of computational resources.

## References

  * [1] Akyüz, A.O.: Photographically guided alignment for hdr images. In: Eurographics (Areas Papers). pp. 73–74 (2011)
  * [2] Catley-Chandar, S., Tanay, T., Vandroux, L., Leonardis, A., Slabaugh, G., Pérez-Pellitero, E.: Flexhdr: Modeling alignment and exposure uncertainties for flexible hdr imaging. IEEE Transactions on Image Processing 31, 5923–5935 (2022)
  * [3] Chen, J., Yang, Z., Chan, T.N., Li, H., Hou, J., Chau, L.P.: Attention-guided progressive neural texture fusion for high dynamic range image restoration. IEEE Transactions on Image Processing 31, 2661–2672 (2022)
  * [4] Crouse, M.S., Nowak, R.D., Baraniuk, R.G.: Wavelet-based statistical signal processing using hidden markov models. IEEE Transactions on signal processing 46(4), 886–902 (1998)
  * [5] Dai, T., Li, W., Cao, X., Liu, J., Jia, X., Leonardis, A., Yan, Y., Yuan, S.: Wavelet-based network for high dynamic range imaging. Computer Vision and Image Understanding 238, 103881 (2024)
  * [6] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. ICLR (2021)
  * [7] Eden, A., Uyttendaele, M., Szeliski, R.: Seamless image stitching of scenes with large motions and exposure differences. In: IEEE Conference on Computer Vision and Pattern Recognition. vol. 2, pp. 2498–2505. IEEE (2006)
  * [8] Fattal, R., Lischinski, D., Werman, M.: Gradient domain high dynamic range compression. In: Proceedings of the 29th annual conference on Computer graphics and interactive techniques. pp. 249–256 (2002)
  * [9] Fei, B., Lyu, Z., Pan, L., Zhang, J., Yang, W., Luo, T., Zhang, B., Dai, B.: Generative diffusion prior for unified image restoration and enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9935–9946 (2023)
  * [10] Gallo, O., Gelfandz, N., Chen, W.C., Tico, M., Pulli, K.: Artifact-free high dynamic range imaging. In: IEEE International conference on computational photography. pp. 1–7. IEEE (2009)
  * [11] Gallo, O., Troccoli, A., Hu, J., Pulli, K., Kautz, J.: Locally non-rigid registration for mobile hdr photography. In: Proceedings of the IEEE conference on computer vision and pattern recognition Workshops. pp. 49–56 (2015)
  * [12] Granados, M., Kim, K.I., Tompkin, J., Theobalt, C.: Automatic noise modeling for ghost-free hdr reconstruction. ACM Transactions on Graphics (TOG) 32(6), 1–10 (2013)
  * [13] Granados, M., Seidel, H.P., Lensch, H.P.: Background estimation from non-time sequence images. In: Proceedings of graphics interface 2008. pp. 33–40 (2008)
  * [14] Heo, Y.S., Lee, K.M., Lee, S.U., Moon, Y., Cha, J.: Ghost-free high dynamic range imaging. In: Asian Conference on Computer Vision. pp. 486–500. Springer (2010)
  * [15] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [16] Hu, J., Gallo, O., Pulli, K.: Exposure stacks of live scenes with hand-held cameras. In: European Conference on Computer Vision. pp. 499–512. Springer (2012)
  * [17] Hu, J., Gallo, O., Pulli, K., Sun, X.: Hdr deghosting: How to deal with saturation? In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition. pp. 1163–1170 (2013)
  * [18] Jacobs, K., Loscos, C., Ward, G.: Automatic high-dynamic range image generation for dynamic scenes. IEEE Computer Graphics and Applications 28(2), 84–93 (2008)
  * [19] Jinno, T., Okuda, M.: Motion blur free hdr image acquisition using multiple exposures. In: IEEE International Conference on Image Processing. pp. 1304–1307. IEEE (2008)
  * [20] Kalantari, N.K., Ramamoorthi, R., et al.: Deep high dynamic range imaging of dynamic scenes. ACM Trans. Graph. 36(4), 144–1 (2017)
  * [21] Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High dynamic range video. ACM Transactions on Graphics (TOG) 22(3), 319–325 (2003)
  * [22] Khan, E.A., Akyuz, A.O., Reinhard, E.: Ghost removal in high dynamic range images. In: 2006 International Conference on Image Processing. pp. 2005–2008. IEEE (2006)
  * [23] Lee, C., Li, Y., Monga, V.: Ghost-free high dynamic range imaging via rank minimization. IEEE signal processing letters 21(9), 1045–1049 (2014)
  * [24] Li, F., Gang, R., Li, C., Li, J., Ma, S., Liu, C., Cao, Y.: Gamma-enhanced spatial attention network for efficient high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1032–1040 (2022)
  * [25] Li, Z., Zheng, J., Zhu, Z., Wu, S.: Selectively detail-enhanced fusion of differently exposed images with moving objects. IEEE Transactions on Image Processing 23(10), 4372–4382 (2014)
  * [26] Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: Image restoration using swin transformer. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1833–1844 (2021)
  * [27] Liu, C., et al.: Beyond pixels: exploring new representations and applications for motion analysis. Ph.D. thesis, Massachusetts Institute of Technology (2009)
  * [28] Liu, S., Zhang, X., Sun, L., Liang, Z., Zeng, H., Zhang, L.: Joint hdr denoising and fusion: A real-world mobile hdr image dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13966–13975 (2023)
  * [29] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)
  * [30] Liu, Z., Lin, W., Li, X., Rao, Q., Jiang, T., Han, M., Fan, H., Sun, J., Liu, S.: Adnet: Attention-guided deformable convolutional network for high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 463–470 (2021)
  * [31] Liu, Z., Wang, Y., Zeng, B., Liu, S.: Ghost-free high dynamic range imaging with context-aware transformer. In: European Conference on Computer Vision. pp. 344–360. Springer (2022)
  * [32] Lu, P.Y., Huang, T.H., Wu, M.S., Cheng, Y.T., Chuang, Y.Y.: High dynamic range image reconstruction from hand-held cameras. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 509–516. IEEE (2009)
  * [33] Mallat, S.G.: A theory for multiresolution signal decomposition: the wavelet representation. IEEE transactions on pattern analysis and machine intelligence 11(7), 674–693 (1989)
  * [34] Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Transactions on graphics (TOG) 30(4), 1–14 (2011)
  * [35] Marín-Vega, J., Sloth, M., Schneider-Kamp, P., Röttger, R.: Drhdr: A dual branch residual network for multi-bracket high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 844–852 (2022)
  * [36] Min, T.H., Park, R.H., Chang, S.: Histogram based ghost removal in high dynamic range images. In: IEEE International Conference on Multimedia and Expo. pp. 530–533. IEEE (2009)
  * [37] Myszkowski, K., Mantiuk, R., Krawczyk, G.: High dynamic range video. Springer Nature (2022)
  * [38] Niu, Y., Wu, J., Liu, W., Guo, W., Lau, R.W.: Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions. IEEE Transactions on Image Processing 30, 3885–3896 (2021)
  * [39] Oh, T.H., Lee, J.Y., Tai, Y.W., Kweon, I.S.: Robust high dynamic range imaging by rank minimization. IEEE transactions on pattern analysis and machine intelligence 37(6), 1219–1232 (2014)
  * [40] Pece, F., Kautz, J.: Bitmap movement detection: Hdr for dynamic scenes. In: 2010 Conference on Visual Media Production. pp. 1–8. IEEE (2010)
  * [41] Prabhakar, K.R., Agrawal, S., Babu, R.V.: Self-gated memory recurrent network for efficient scalable hdr deghosting. IEEE Transactions on Computational Imaging 7, 1228–1239 (2021)
  * [42] Prabhakar, K.R., Agrawal, S., Singh, D.K., Ashwath, B., Babu, R.V.: Towards practical and efficient high-resolution hdr deghosting with cnn. In: European Conference on Computer Vision. pp. 497–513. Springer (2020)
  * [43] Prabhakar, K.R., Senthil, G., Agrawal, S., Babu, R.V., Gorthi, R.K.S.S.: Labeled from unlabeled: Exploiting unlabeled data for few-shot deep hdr deghosting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4875–4885 (2021)
  * [44] Pu, Z., Guo, P., Asif, M.S., Ma, Z.: Robust high dynamic range (hdr) imaging with complex motion and parallax. In: Proceedings of the Asian Conference on Computer Vision (2020)
  * [45] Ramamonjisoa, M., Firman, M., Watson, J., Lepetit, V., Turmukhambetov, D.: Single image depth prediction with wavelet decomposition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11089–11098 (2021)
  * [46] Romberg, J.K., Choi, H., Baraniuk, R.G.: Bayesian tree-structured image modeling using wavelet-domain hidden markov models. IEEE Transactions on image processing 10(7), 1056–1068 (2001)
  * [47] Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust patch-based hdr reconstruction of dynamic scenes. ACM Trans. Graph. 31(6), 203–1 (2012)
  * [48] Song, J.W., Park, Y.I., Kong, K., Kwak, J., Kang, S.J.: Selective transhdr: Transformer-based selective hdr imaging using ghost region mask. In: European Conference on Computer Vision. pp. 288–304. Springer (2022)
  * [49] Tel, S., Heyrman, B., Ginhac, D.: Cen-hdr: Computationally efficient neural network for real-time high dynamic range imaging. In: European Conference on Computer Vision. pp. 378–394. Springer (2022)
  * [50] Tel, S., Wu, Z., Zhang, Y., Heyrman, B., Demonceaux, C., Timofte, R., Ginhac, D.: Alignment-free hdr deghosting with semantics consistent transformer. arXiv preprint p. arXiv:2305.18135 (2023)
  * [51] Tomaszewska, A., Mantiuk, R.: Image registration for multi-exposure high dynamic range image acquisition. In: International Conference in Central Europe on Computer Graphics. pp. 49–56 (2007)
  * [52] Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: The state of the art in hdr deghosting: A survey and evaluation. Computer Graphics Forum 34(2), 683–707 (2015)
  * [53] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. vol. 30 (2017)
  * [54] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 568–578 (2021)
  * [55] Wang, X., Chan, K.C., Yu, K., Dong, C., Change Loy, C.: Edvr: Video restoration with enhanced deformable convolutional networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 0–0 (2019)
  * [56] Ward, G.: Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures. Journal of graphics tools 8(2), 17–30 (2003)
  * [57] Wu, S., Xu, J., Tai, Y.W., Tang, C.K.: Deep high dynamic range imaging with large foreground motions. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 117–132 (2018)
  * [58] Yan, Q., Chen, W., Zhang, S., Zhu, Y., Sun, J., Zhang, Y.: A unified hdr imaging method with pixel and patch level. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22211–22220 (2023)
  * [59] Yan, Q., Gong, D., Shi, Q., Hengel, A.v.d., Shen, C., Reid, I., Zhang, Y.: Attention-guided network for ghost-free high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1751–1760 (2019)
  * [60] Yan, Q., Zhang, L., Liu, Y., Zhu, Y., Sun, J., Shi, Q., Zhang, Y.: Deep hdr imaging via a non-local network. IEEE Transactions on Image Processing 29, 4308–4322 (2020)
  * [61] Yan, Q., Zhang, S., Chen, W., Tang, H., Zhu, Y., Sun, J., Van Gool, L., Zhang, Y.: Smae: Few-shot learning for hdr deghosting with saturation-aware masked autoencoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5775–5784 (2023)
  * [62] Yao, S.: Robust image registration for multiple exposure high dynamic range image synthesis. In: Image Processing: Algorithms and Systems IX. vol. 7870, pp. 216–224. SPIE (2011)
  * [63] Ye, Q., Xiao, J., Lam, K.m., Okatani, T.: Progressive and selective fusion network for high dynamic range imaging. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 5290–5297 (2021)
  * [64] Yu, Y., Zhan, F., Lu, S., Pan, J., Ma, F., Xie, X., Miao, C.: Wavefill: A wavelet-based generation network for image inpainting. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 14114–14123 (2021)
  * [65] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5728–5739 (2022)
  * [66] Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV). pp. 286–301 (2018)
  * [67] Zheng, B., Pan, X., Zhang, H., Zhou, X., Slabaugh, G., Yan, C., Yuan, S.: Domainplus: Cross transform domain learning towards high dynamic range imaging. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 1954–1963 (2022)
  * [68] Zheng, J., Li, Z., Zhu, Z., Wu, S., Rahardja, S.: Hybrid patching for a sequence of differently exposed images with moving objects. IEEE transactions on image processing 22(12), 5190–5201 (2013)
  * [69] Zimmer, H., Bruhn, A., Weickert, J.: Freehand hdr imaging of moving scenes with simultaneous resolution enhancement. Computer Graphics Forum 30(2), 405–414 (2011)