# The Long Filament of PSR J2030+4415

Martijn de Vries  Department of Physics/KIPAC, Stanford University, Stanford,
CA 94305-4060, USA  Roger W. Romani  Department of Physics/KIPAC, Stanford
University, Stanford, CA 94305-4060, USA

###### Abstract

New X-ray and optical observations shed light on the remarkable X-ray filament
of the Gamma-ray pulsar PSR J2030+4415. Images of the associated H$\alpha$ bow
shock’s evolution over the past decade compared with its velocity structure
provide an improved kinematic distance of $\sim$0.5kpc. These velocities also
imply that the pulsar spin axis lies $\sim 15^{\circ}$ from the proper motion
axis which is close to the plane of the sky. The multi-bubble shock structure
indicates that the bow shock stand-off was compressed to a small value $\sim
20-30$y ago when the pulsar broke through the bow shock to its present bubble.
This compression allowed multi-TeV pulsar $e^{\pm}$ to escape to the external
ISM, lighting up an external magnetic field structure as the ‘filament’. The
narrow filament indicates excellent initial confinement and the full
$15^{\prime}$ ($2.2$ pc=7 lt-y) projected length of the filament indicates
rapid $e^{\pm}$ propagation to its end. Spectral variation along the filament
suggests that the injected particle energy evolved during the break-through
event.

Pulsars — nebulae — gamma rays: stars — stars: individual (PSR J2030$+$4415)

††journal: ApJ††facilities: CXO, Gemini - GMOS

##  1 Introduction

In de Vries & Romani (2020) we found that PSR J2030+4415 (hereafter J2030), a
$P=308$ ms, $\tau=6\times 10^{5}$ yr (${\dot{E}}=2.2\times
10^{34}{\rm\,erg\,s^{-1}}$) radio-quiet $\gamma$-ray pulsar, has a long,
narrow X-ray filament. In the discovery image, we see a pulsar wind nebula
(PWN) trailed behind the pulsar and a filament at an angle $130^{\circ}$ to
this motion extending $5^{\prime}$ to the edge of the X-ray image. In this
paper we report on a measurement of the filament’s full extent as well as
supporting optical observations that help probe the nature of the filament’s
origin.

Only a handful of pulsar X-ray filaments are known. The most famous is
associated with the PSR B2224+65/Guitar nebula (Hui & Becker, 2007). Other
large angle long filaments are seen for PSR J1101$-$6101/Lighthouse (Pavan et
al., 2016) and PSR J1509$-$5830 Klingler et al. (2016) (other candidates have
short exposures or are ambiguous). Together with J2030, all four appear to be
fast moving pulsars and at least three have H$\alpha$ bow shock structures
(the H$\alpha$ image in Pavan et al. (2016) showed too much diffuse H$\alpha$
emission to detect, or exclude, a bow shock). The neutral hydrogen required
for the H$\alpha$ emission indicates a high ambient proton density $n_{0}$,
which together with a large pulsar velocity $v$ implies a bow shock with small
forward standoff distance $r_{0}=[{\dot{E}}/(4\pi\mu m_{p}n_{0}cv^{2})]^{1/2}$
for a mean particle mass in proton units $\mu\approx 1.38$. This association
with small $r_{0}$ tends to support the Bandiera (2008) picture for the origin
of the Guitar nebula filament: when the standoff distance $r_{0}$ approaches
the gyroradius $r_{g}=\gamma m_{e}c^{2}/eB$ of the relativistic $e^{\pm}$
generated by the pulsar and its wind termination shock, energetic charges can
escape to external magnetic field lines near the apex. Synchrotron emission
from particles propagating rapidly along such external fields give rise to the
X-ray filaments. These filaments are generally asymmetric, being much longer
on one side of the pulsar and its PWN trail. This asymmetry may be associated
with the magnetic field direction of the polar flow in the PWN, since
reconnection of the forward facing hemisphere tends to favor emission onto
field lines of the appropriate polarity. Simulations of particles in
magnetosphere termination shocks (Bucciantini, 2018; Barkov et al., 2019; Olmi
& Bucciantini, 2019) have illustrated such asymmetry.

Figure 1: The PSR J2030+4415 field in GMOS-N H$\alpha$ (red), $r$ (green) and
$g$ (blue). The smoothed green contours show 1-5keV CXO ACIS emission; some
arises from field stars and background sources, but the PWN and filament are
visible. The grey scale inset at upper right shows the combined ACIS data
following the filament’s 15′ length; the red square shows the optical image
coverage.

J2030’s filament provides a good opportunity to explore these ideas: it is
long, narrow, seen only on one side and arises from a complex PWN with
multiple H$\alpha$ bubbles. Here we describe a new imaging campaign which
helps us probe the filament geometry and origin.

##  2 Observations

Figure 2: The evolution of the apex bubble over a decade. For reference,
outlines in each frame mark the forward bow shock (solid red) and the apex
bubble (dashed red), as traced from the 2021 data. The panel cover
$13.4^{\prime\prime}\times 13.7^{\prime\prime}$ (only
$6.8^{\prime\prime}\times 6.5^{\prime\prime}$ for the panel 2 IFU image). The
pulsar location is marked with a cyan circle. In the rightmost frame the 2021
image is repeated with estimated bubble radii from the three epochs shown as
yellow, green and red circles. The common center is a red inner circle. Cyan
contours show the CXO flux; the filament is not visible at this fine scale.

###  2.1 A new H$\alpha$ bow shock Image

Gemini/GMOS-N observations of J2030 were taken on August 3 and 9, 2021 (MJD
59429, 59435) under good $0.5-0.6^{\prime\prime}$ imaging conditions (a few
usable, but sub-par exposures were also obtained on August 8). The data
include $6\times 600$s H$\alpha$ exposures, covering the PWN and the first
$5^{\prime}$ of the filament. Accompanying $3\times 30$s $gri$ images were
also obtained for continuum subtraction and characterization of the field
stars. Pipeline image reductions proved adequate and we assembled dithered
combined images of the field (Figure 1).

It is interesting to compare this latest H$\alpha$ exposure of the nebula apex
with the original 2011 WIYN discovery image and the velocity-integrated image
from GMOS-N IFU data taken in 2015 (Figure 2). While the bow shock hardly
shifts, the ‘bubble’ at the apex of the structure expands noticeably over the
past decade. The seeing-limited ground based images (esp. in 2011) and the
limb complexity preclude a direct fit, but matching to the apparent bubble
edge gives radii of $\approx 1.18^{\prime\prime}$, $1.45^{\prime\prime}$, and
$1.71^{\prime\prime}$ at the three epochs. This gives a mean expansion rate of
$\mu_{b}=56\pm 8$mas/y, assuming a common center, which is near the tip of the
bow shock structure (red circle in the right panel of Figure 2). If the
expansion were constant at the last decade’s rate it would have initiated from
this point in 1989$\pm 1$ (32y ago). Note that DR20 used archival and new CX0
imaging to measure the pulsar proper motion at $85\pm 16$mas/y. It is now
$\sim 1.0^{\prime\prime}$ North of the bubble center. Thus the pulsar passed
through this point $\sim 12\pm 2$y ago. This suggests that the bubble was
initiated by a break-through below this position, somewhat before the 2011
discovery epoch, and that rapid initial bubble expansion out-ran the pulsar,
but has now settled down to the present more modest rate. Note however that
the bubble expansion rate is only $1.2\sigma$ smaller than the pulsar velocity
and that, eventually, we expect that the bubble leading edge motion will
settle down to match the pulsar velocity, forming a new stationary bow shock
structure. While it seems that we are in transition from rapid bubble
expansion at break-out to bow shock formation, additional high resolution
imaging will be needed to resolve the bubble kinematics.

Some kinematic information also comes from the IFU velocity slices (Figure 2
of DR20). The bow shock structure behind the apex bubble is dominated by low
velocities. This flux is fairly symmetric over the range $\pm 70$ km/s,
indicating that the bow shock axis (and pulsar velocity) lie close to the
plane of the sky. Above this, we see the near-circular expanding apex bubble
shell, especially in the $\pm$94km/s channels. The bubble extreme velocity
peaks are offset N ($+v$, redshifted) and S ($-v$, blue shifted) from the
pulsar position. This indicates that the bubble is oblate and expands faster
along its major axis. Again S/N, resolution and complexity prevent a direct
fit, but matching an oblate spheroid expansion model to the position-velocity
cube indicates that the $+v$ major axis points $\approx 10^{\circ}$ into the
plane of the sky at PA$\approx-10^{\circ}$ (to the west); the model match
degrades appreciably for angles $\sim 10^{\circ}$ from these values. The
physical oblateness is $\approx 0.2$, while the velocity oblateness is
$\approx 0.5$. In DR20 we attempted to match a Wilkin (2000)-type bow shock
model with an equatorial ${\rm cos^{2}}\theta$ flow; this places the spin axis
close to the line-of-sight with the equatorial bulges forming the velocity
extrema. Modern pulsar models associate the gamma-ray emission with the outer
magnetosphere or the wind zone current sheet, both of which beam to the spin
equator. Thus the oblate spheroid model described here, with its equatorial
view of the pulsar, seems more satisfactory. We interpret the major axis as
marking the polar momentum flux of the pulsar wind. The larger velocity
oblateness suggests that the outflow was closer to spherical in the initial
outburst and that the pulsar wind latitudinal structure is increasingly
affecting the forward shock position. The best-match instantaneous equatorial
expansion velocity is $v_{e}\approx 120$km/s. Comparison with the 2011-2021
expansion proper motion then gives a kinematic pulsar distance of $d=500\,{\rm
pc}(\mu_{b}/56{\rm mas/y})/(v_{e}/120{\rm km/s})$, i.e. about 2/3 of the
distance from the heuristic $L_{\gamma}=(10^{34}{\rm
erg\,s^{-1}}{\dot{E}})^{1/2}$ pulsar luminosity estimate. This makes J2030 one
of the closest known energetic PWNe.

Figure 3: Surface brightness profiles across 9 segments of J2030’s filament.
The x-axis indicates the distance across the filament in arcsec, with the
negative direction being behind the filament (south/south-east), and the
positive direction ahead of the filament (north/north-west). To each profile,
we fit a Gaussian plus a constant background. The green dashed line indicates
the average width of the PSF (approximated as a Gaussian) at the location of
the segment. Labels on the top left of each panel indicate 1) the distance
range of the segment away from the pulsar, 2 and 3) the mean and the standard
deviation from the Gaussian fit, and 4) the corrected width, which is obtained
by subtracting the PSF width from the fitted width in quadrature and adding a
systematic error of $0.5^{\prime\prime}$ to the total uncertainty.

###  2.2 New CXO X-ray Measurements

In DR20, we reported on four CXO exposures from April 10-15, 2019 [ObsIDs
20298 (45 ks), 22171 (40 ks), 22172 (45 ks), 22173 (23 ks); $\sim 153$ ks
total] and compared with an archival exposure [ObsIDs 14827 (42 ks)] from
April 15, 2014. These placed the pulsar on the S3 chip, and the new filament
ran to the edge of the discovery image. We therefore augmented this study with
ACIS-I exposures on Feb 13 (44.6ks) and Nov. 8 (30 ks) 2021 [ObsIDs 23536,
24954 and 24236], with the pointing centered $\sim 5^{\prime}$ along the
filament, allowing us to trace the filament extent to much larger angle. All
exposures were in VF mode. The data were processed and analysed using CIAO
4.12 and CALDB 4.9.1 and the region of the mosaic covering the filament is
shown as the inset in Figure 1. Because the particle background level of the
back-illuminated ACIS S-3 chip is significantly higher than the front-
illuminated chips in the ACIS array, we created a particle background model
using the ACIS ‘stowed’ backgrounds, scaling these backgrounds to the science
images using the counts between $9.5-12\,$keV. At this energy range, the
effective area of CXO is negligible, and the vast majority of counts are
particle events. Subtracting this particle background model removes
approximately $80$ to $90\%$ of the background flux level and achieves a
roughly uniform background level across the field. After background
subtraction, we created a merged and exposure-corrected mosaic using the
merge_obs tool. For the exposure-corrected image, we used an energy range of
$1-5\,$keV to achieve maximum contrast between the filament and the
background. Finally, to avoid contamination from point sources in our
measurements of the filament surface brightness, we used the CIAO tool dmfilth
to mask out point sources around the filament and fill them in with Poisson-
noise background from a surrounding annulus.

The pulsar point source and PWN trail are unfortunately far enough off axis
that the new exposures do not contribute materially to mapping their complex
X-ray structure. A full study would require a very long CXO exposure (or
eventually LynX observations). However the new data do show that the filament
extends dramatically further than in the discovery image, fading out 15-16′
from the PWN. The filament shows appreciable curvature, with a particularly
pronounced southward bend 1′ from the pulsar.

In order to measure the properties of the filament, we extracted nine surface
brightness profiles across the filament, each perpendicular to a ridge line
drawn by eye on the exposure-corrected X-ray image. To each filament segment,
we fit a Gaussian plus a constant background, as shown in Figure 3. We used
the CIAO tool mkpsfmap to calculate the average PSF width (weighted by the
exposure time of each individual observation) at the location of the segment.

Although the filament ridge line was drawn by eye and is thus an imperfect
representation of its true location, the fitted Gaussian means are within 1′′
of the estimated ridge line in the inner 6 segments, and even the outer, most
diffuse segment does not deviate from the ridge more than 1.7′′. Despite this,
a slight systematic error can remain as the result of mismatch between the
drawn line and the filament. We therefore add a systematic error of 0.5′′ to
the filament width in each region.

Figure 4 shows the filament width and integrated flux per unit length in nine
bins along the filament. The corrected widths of each segment were calculated
by subtracting the PSF widths in quadrature from the fitted widths. Somewhat
surprisingly, we find that the first filament segment is significantly wider
than the subsequent two segments - deviating from the general trend of
increasing width with increasing distance. This might be due to unrecognized
filament curvature in its initial arcmin. Additionally, there could be
unaccounted for non-filamentary emission in a halo near the PWN. If we ignore
the first segment and fit a power law to the remaining segments, we find a
slope $\alpha=1.0\pm 0.2$. If we include the first point, we find
$\alpha=0.9\pm 0.2$.

Figure 4: Left panel: the filament Gaussian width $\sigma$ as a function of
distance along the filament. The blue points show the fitted width, while the
orange points show the width corrected for PSF size. The dashed black line
shows a power-law fit to the orange points, with fitted slope $1.0\pm 0.2$.
Right panel: the integrated 1 to 5 keV photon flux per unit length along the
filament, in units of $\rm photon\,cm^{-2}\,s^{-1}\,arcmin^{-1}$. The red
points indicate the fluxes per arcmin from the surface-brightness analysis,
using an exposure map with a $\Gamma=1.5$ energy weighting. The black points
indicate the fluxes from the spectral fits listed in Table 1, converted to
1$-$5 keV photon flux units.

X-ray Spectrum

With the modest surface brightness and low counts our X-ray spectral results
are limited. Nevertheless we can fit simple power law models to various
sections of the filament, assuming a constant absorption $N_{H}=6\times
10^{20}{\rm cm^{-2}}$, the value estimated by converting from the $A_{g}$ of
the Bayestar19 3-D dust maps (Green et al., 2019). We extracted spectra from 4
regions along the filament, and combined the extracted spectra from the 5
ACIS-S observations and the extracted spectra from the 3 ACIS-I observations
using the CIAO tool combine_spectra. As the background model, we used a local
background of spectra extracted from elliptical regions with centers $\approx
1^{\prime}$ north and south of the filament, which were then subtracted from
the filament spectra. The fit results, shown in Table 1, are consistent with
our previous findings in DR20. We detect a softening in the spectrum beyond
$3^{\prime}$, with a significance of around $2\sigma$. Of course if the true
absorption is larger, the $\Gamma$’s will increase (e.g. for $N_{H}=2\times
10^{21}{\rm cm^{-2}}$ all $\Gamma$ increase by $\approx 0.12$), however the
spectral index differences persist. The measured fluxes, when converted to
units of 1$-$5 keV photon flux agree with the fluxes estimated from the
exposure-corrected image (Figure 4), with the exception of the region nearest
the pulsar. The deviation between the two measurements suggests that perhaps
the width in the surface brightness fits is somewhat overestimated. Overall,
the $0.5-7\,$keV flux along the filament appears to increase very marginally
by about $1\sigma$. While it would be interesting to probe spectral variations
across the filament, we lack the resolution and sensitivity to do so.

Table 1: Spectral fit results for filament sections with an absorbed power law
modela

Region | Counts | $\Gamma$ |  $f_{-15}/$arcminb |  $\chi^{2}$/DoF
---|---|---|---|---
Fil (0-15’) | $905\pm 53$ | $1.50\pm 0.12$ | $8.0_{-0.6}^{+0.7}$ | 175/180
Fil (0-1’) | $122\pm 15$ | $1.28\pm 0.29$ | $7.1_{-1.0}^{+1.3}$ | 16/19
Fil (1-3’) | $219\pm 23$ | $1.09\pm 0.23$ | $7.5_{-1.0}^{+1.1}$ | 26/30
Fil (3-6’) | $294\pm 32$ | $1.50\pm 0.22$ | $8.5_{-1.1}^{+1.4}$ | 81/80
Fil (6-15’) | $270\pm 30$ | $1.76\pm 0.28$ | $8.1_{-0.9}^{+1.1}$ | 45/48

a $N_{H}$ fixed at $6\times 10^{20}{\rm cm^{-2}}$.

b $0.5-7\,$keV unabsorbed fluxes per arcmin filament length

in units of $10^{-15}{\rm erg\,cm^{-2}s^{-1}}\,{\rm arcmin}^{-1}$.

##  3 Filament Morphology and Conclusions

The best known filaments are those of the Guitar Nebula/PSR B2224+65 (Cordes
et al., 1993; Bandiera, 2008) the ‘Lighthouse’/PSR J1101$-$6101 (Pavan et al.,
2016) and PSR J1509$-$5058 (Klingler et al., 2016). J2030’s filament does have
some special features. It is purely one-sided, initially narrow, with an
approximate linear expansion, and its flux varies little with distance. These
properties are most similar to the filament of PSR J1509$-$5058.

We would like to connect these properties with the pulsar filament picture
outlined in Bandiera (2008) and elaborated by later authors (e.g. Olmi &
Bucciantini, 2019; Barkov et al., 2019). J2030’s extreme one-sidedness may be
connected with the unusually small spin-velocity angle inferred from the
H$\alpha$ bubble velocity structure $\theta_{{\vec{\Omega}}\cdot{\vec{v}}}\leq
15^{\circ}$; this ensures that one magnetic hemisphere is much closer to the
swept-up magnetic field at the apex, fostering reconnection to one sign of
${\vec{B}}$ (Figure 5).

Figure 5: Cartoon of magnetic geometry near the magnetopause at the contact
discontinuity. When the standoff distance $r_{0}$ is small, high $\gamma_{e}$
(large gyroradius) particles can escape to external ISM field lines. This is
enhanced by reconnection; when $\theta_{{\vec{\Omega}}\cdot{\vec{v}}}$ is
$<\pi/2$, reconnection to one side of the bow shock is preferred.

We note that a line following the brightest filament ridge out to $\sim
5^{\prime}$ passes $\approx 7^{\prime\prime}$ in front of the pulsar. Yet in
the nearest arcmin or so, the filament ridge appears to curve, such that it
meets the PSR/PWN axis several arcsec behind the pulsar (as can be seen in
Figure 1). This is close to the bow shock apex and the bubble expansion point.
A plausible interpretation is that filament follows a field line that
connected with the bow shock apex several decades ago, when $r_{0}$ was
minimal just before break-out to the new forward bubble structure. The pulsar-
trailing connection position might alternatively be associated with field line
‘draping’ as the magnetic lines are pushed forward by the bow shock as seen in
e.g. the lighthouse nebula (Klingler et al., 2016). However, draping would be
expected to produce field line curvature opposite that seen for J2030, so
perhaps the curvature is intrinsic to the external field.

The narrow initial filament width places some limits on the guiding magnetic
field, since a gyroradius $\rho=1.6\times 10^{17}\gamma_{8}/B_{\mu G}$ cm
subtends $10.7^{\prime\prime}\gamma_{8}/(B_{\mu G}d_{\rm kpc})$, for an
$e^{\pm}$ energy $10^{8}\gamma_{8}m_{e}c^{2}$ and magnetic field
$10^{-6}B_{\mu G}$ G at a distance $d_{\rm kpc}$kpc. As usual, using the
synchrotron peak energy $E_{\rm keV}$keV, we eliminate $\gamma_{8}=2.4(E_{\rm
keV}B_{\mu G})^{1/2}$ to write $\theta_{\rm gyro}=26^{\prime\prime}E_{\rm
keV}^{1/2}B_{\mu G}^{-3/2}d_{\rm kpc}^{-1}$. Thus using the minimum corrected
width and a mean photon energy of 2 keV, we infer that the filament has
$B>20\mu$G at our estimated $d_{\rm kpc}\approx 0.5$. Imperfect ridge line
estimates increases this lower bound. Of course we can also estimate the
filament field using our spectral measurements and the usual equipartition
synchrotron sums. The observed flux from a cylindrical emission region of
radius $\theta_{w}$ and length $\theta_{l}$ corresponds to a volume emissivity
$J=4\pi fd^{2}/V=1.9\times 10^{-22}f_{-15}/(\theta_{w}^{2}\theta_{l}d_{\rm
kpc})$ erg s-1 cm-3 (for $\theta_{w}$ in arcsec, $\theta_{l}$ in arcmin)
measured over $E_{1}-E_{2}$ from a full spectral range $E_{\rm min}-E_{\rm
max}$. Assuming a magnetization $\sigma=w_{B}/w_{e}$ and measuring $J_{-20}$
in units of $10^{-20}$ erg s-1 cm-3 we obtain

$B=46\left[J_{\rm-20}(E_{1},E_{2})\sigma\frac{C_{1.5-\Gamma}(E_{m},E_{M})}{C_{2-\Gamma}(E_{1},E_{2})}\right]^{2/7}\mu
G$ (1)

where $C_{q}(x_{1},x_{2})=\frac{x_{2}^{q}-x_{1}^{q}}{q}$, $\Gamma$ is the
spectral index and energies are in keV. Using $\Gamma\approx 1.5$ over an
assumed full range of 0.01-10 keV, $f_{-15}\sim 8$/arcmin from Table 1 and a
typical filament radius of $\theta_{w}=2^{\prime\prime}$, we obtain $B\approx
26(w_{B}/w_{e})^{2/7}\mu$G, in reasonable agreement with the minimum width
constraint.

In Table 1, the nearest two zones appear to be harder by $\Delta\Gamma\approx
0.5$ than the farther zones. Although the difference is marginally ($\sim
2\sigma$) significant, it is tempting to interpret this as a cooling break.
Similarly, in DR20 we found weak evidence for a $\tau=1.52\pm
0.59^{\prime\prime}$ exponential tail behind the relatively sharp leading
edge, again suggesting cooling. However, with an observed proper motion of 85
mas/y, this exponential tail would correspond to a cooling time of $\sim 18$
y, which for $\tau_{c}=8\times 10^{4}{\rm y}\,E_{\rm keV}^{-1/2}B_{\mu
G}^{-3/2}$ requires a field of $>200\mu$ G. Such a large field is not typical
of the ISM.

Moreover, in Table 1 and Figure 3 the filament does not fade along its length
as might be expected for cooling. Indeed, the flux/arcmin increases, if
anything, in the outer segment. The brightness profile also argues against
simple diffusion from on-going injection of pulsar particles filling the
filament since, again, one would expect a flux decrease with distance (age).
Instead we adopt the more typical $\sim 20-30\mu$G field from the gyroradius
and spectral estimates (still a factor of a few larger than general ISM
fields). We can then interpret the narrow initial filament width as indicating
a short $1^{\prime\prime}/85$ mas/y$\leq 12$y period of particle injection,
during which the pulsar traveled less than 1 arcsec. This injection is likely
associated with the bow shock break-through. Thus, the X-ray producing
filament particles have been resident for the 20-30y since the breakthrough
event, with adequate time to travel the $\approx 15^{\prime}\approx 7$ lt-y
(at $d_{\rm kpc}=0.5$) to the filament end at $v<c/3$. Little cooling has
occurred, since $\tau_{c}\approx 350$ y for $B=30\mu$G.

Still, the electrons near the pulsar should be somewhat younger than those at
the far end of the filament. In this picture the (tentative) evidence for
$\Gamma$ and flux increase along the filament might be ascribed to variation
in the injection process. Let us assume that the pulsar/wind termination shock
produces a spectrum of energetic $e^{\pm}$
$dN_{e}/d\gamma_{e}=C\gamma_{e}^{-p}$ up to some $\gamma_{e,{\rm max}}$. When
the standoff distance $r_{0}$ was compressed to its minimum value at the
breakthrough event (where the external density is maximal in the bow shock-
compressed ISM) a modest range $\eta$ of the largest energies
$>\gamma_{e,\,{\rm min}}=\gamma_{e,{\rm max}}/\eta$ would have sufficiently
large gyroradii to escape to the filament. As the apex bubble grew, with the
pulsar wind termination shock encroaching on the lower density general ISM,
$r_{0}$ increased and the escaping $\gamma_{e}$ range (and particle flux)
decreased until only $\gamma_{e,{\rm max}}$ could escape to the filament.
Continued increase of $r_{0}$ would shut off all particle injection. Thus the
last particles injected (nearest the pulsar) would have a nearly mono-
energetic $\gamma_{\rm max}$ spectrum, while more distant filament zones would
be dominated by a larger particle population with $\eta$ of a few. Figure 6
shows how this can produce a spectrum that softens with distance along the
filament. The higher number of particles injected at break-through can
compensate for diffusion spreading, keeping the distant zones of the filament
bright.

Figure 6: Variation in the X-ray spectral index near the synchrotron cutoff
(here $\gamma_{e,{\rm max}}$ is set so $E_{c}=4$ keV), with varying pulsar
injection. The PL fit to the near-pulsar filament 0.5-8 keV spectrum (green
shaded region) may be produced near the synchrotron peak of a mono-energetic
particle distribution. The softer emission from more distant (older) zones
(magenta shaded region) can be produced by a small range of particles
$\gamma_{e,{\rm max}}/\gamma_{e,{\rm min}}=\eta$ with power-law index $p$.
Relatively hard $p\approx 1-1.5$ are needed to produce the observed small
X-ray $\Gamma\approx 1.2-1.8$.

We can similarly understand the approximately linear increase of $\theta_{w}$
along the filament, if particles further along the filament are older,
reaching their position by flow and/or diffusion. One could attribute the
linear spreading of Figure 4 to cross-field diffusion, with the small
$\theta_{w}/\theta_{l}\approx
0.01\approx(D_{\perp}/D_{\parallel})^{1/2}\approx M_{A}^{2}$ attributed to a
small cross-field diffusion coefficient $D_{\perp}$. This suggests a
relatively small magnetic Alfven number $M_{A}\approx 0.1$. Again, stronger
injection at break-through might explain the lack of diffusion-induced
surface-brightness fall-off, although fine tuning might be needed to match the
profile of Figure 4.

Our revised picture of the J2030 bow shock and filament suggest rapid
evolution of these structures on a decade time scale and sub-arcsec angular
scale. This makes precision measurements of the shock structure and filament
connection very difficult with ground based seeing. High resolution H$\alpha$
images with HST could help, but since the break-through event was a few
decades in the past such data can primarily constrain the current shock
evolution. We still have a challenging task to connect the filament structure
with the dynamics of new bubble formation. One important clue could come from
a future high S/N X-ray filament measurement. At 85mas/y the pulsar
displacement is substantial on the decade timescale and if filament injection
is continuous rather than single-epoch, we would expect illumination of new
field lines ahead of the present edge. This is indeed what we see for the
Guitar nebula (e.g. Wang, 2021, de Vries et al., in prep) and is feasibly
constrained with a deep future CXO exposure. A lack of such shifted emission
would support the break-through injection picture proposed here. Even more
valuable would be a high S/N spatially resolved spectral study of the filament
to measure synchrotron spectral variations along and across its width. Figure
6 implies that, if we are observing near $E_{c}$, there should be substantial
spectral curvature across the soft X-ray band. Measurement of such curvature
requires substantially higher S/N; the long-field spectra and their variation
might be probed with XMM but probe of spectral variation across the filament
width will probably need LynX-era sensitivities.

Perhaps the most useful prediction of our study is that other H$\alpha$ pulsar
bow shocks with small $r_{0}$ should experience short periods of enhanced
injection, causing filaments to appear or brighten. Monitoring these rare bow
shocks, e.g. with H$\alpha$ imaging, could alert us to incipient events,
triggering extensive X-ray imaging campaigns to study the injected particles
and their evolution. If, in addition, we can use pulsar polarization, PWN jet
structures and H$\alpha$-revealed PWN momentum asymmetries (as for J2030) to
infer 3D geometries and spin-velocity angles, we will then have a powerful
tool for probing the conditions needed for efficient reconnection and multi-
TeV $e^{\pm}$ escape. This will be a useful guide for RMHD shock modeler and
may have important implications for the propagation of pulsar cosmic ray
positrons through the nearby ISM to Earth detectors.

We wish to thank the staffs of the observatories who helped in planning for
the exposures described in this paper, especially Jean Connelly at the CfA for
help with CXO scheduling and Siyi Xu for assistance with the Gemini
observations. This work was supported in part by NASA grants G08-19049X and
G01-22054X administered by the SAO.

## References

  * Bandiera (2008) Bandiera, R. 2008, A&A, 490, L3, doi: 10.1051/0004-6361:200810666
  * Barkov et al. (2019) Barkov, M. V., Lyutikov, M., Klingler, N., & Bordas, P. 2019, MNRAS, 485, 2041, doi: 10.1093/mnras/stz521
  * Bucciantini (2018) Bucciantini, N. 2018, MNRAS, 480, 5419, doi: 10.1093/mnras/sty2237
  * Cordes et al. (1993) Cordes, J. M., Romani, R. W., & Lundgren, S. C. 1993, Nature, 362, 133, doi: 10.1038/362133a0
  * de Vries & Romani (2020) de Vries, M., & Romani, R. W. 2020, ApJ, 896, L7, doi: 10.3847/2041-8213/ab9640
  * Green et al. (2019) Green, G. M., Schlafly, E., Zucker, C., Speagle, J. S., & Finkbeiner, D. 2019, ApJ, 887, 93, doi: 10.3847/1538-4357/ab5362
  * Hui & Becker (2007) Hui, C. Y., & Becker, W. 2007, A&A, 467, 1209, doi: 10.1051/0004-6361:20066562
  * Klingler et al. (2016) Klingler, N., Rangelov, B., Kargaltsev, O., et al. 2016, ApJ, 833, 253, doi: 10.3847/1538-4357/833/2/253
  * Olmi & Bucciantini (2019) Olmi, B., & Bucciantini, N. 2019, MNRAS, 490, 3608, doi: 10.1093/mnras/stz2819
  * Pavan et al. (2016) Pavan, L., Pühlhofer, G., Bordas, P., et al. 2016, A&A, 591, A91, doi: 10.1051/0004-6361/201527703
  * Wang (2021) Wang, Q. D. 2021, Research Notes of the American Astronomical Society, 5, 5, doi: 10.3847/2515-5172/abd854
  * Wilkin (2000) Wilkin, F. P. 2000, ApJ, 532, 400, doi: 10.1086/308576


# High Dynamic Range Imaging via Visual Attention Modules

Ali Reza Omrani  Institute of Information Science and Technologies (ISTI),
National Research Council of Italy, Pisa, Italy  Department of Engineering,
Università Campus Bio-Medico di Roma, Rome, Italy  Davide Moroni  Institute of
Information Science and Technologies (ISTI), National Research Council of
Italy, Pisa, Italy

##  1 Abstract

Thanks to High Dynamic Range (HDR) imaging methods, the scope of photography
has seen profound changes recently. To be more specific, such methods try to
reconstruct the lost luminosity of the real world caused by the limitation of
regular cameras from the Low Dynamic Range (LDR) images. Additionally,
although the State-Of-The-Art methods in this topic perform well, they mainly
concentrate on combining different exposures and have less attention to
extracting the informative parts of the images. Thus, this paper aims to
introduce a new model capable of incorporating information from the most
visible areas of each image extracted by a visual attention module which is a
result of a segmentation strategy. In particular, the model, based on a deep
learning architecture, utilizes the extracted areas to produce the final HDR
image. The results demonstrate that our method outperformed most of the State-
Of-The-Art algorithms.

Keywords: Deep Neural Network, High Dynamic Range imaging, Image Segmentation,
Multi-exposure Image, Visual Attention Module

##  2 Introduction

In the scope of photography, the real world consists of an unlimited range of
luminance. However, most devices are capable of capturing merely limited of
that light. Therefore, the taken images are not desirable and consist of
saturated regions, in which some parts of the images are too dark
(underexposed) or overly bright (overexposed). These types of pictures are
called LDR images.

Thus, in order to cope with this problem, highly advanced cameras [1, 2, 3, 4,
5, 6, 7] can be used, which have special sensors to capture more light.
However, such devices are mainly too expensive and overly heavy, which are not
suitable for daily life, and instead, are mostly used in industries.

A possible resolution for this drawback is developing software algorithms
called HDR imaging techniques. Moreover, HDR images can be implemented by a
single image [8, 9, 10, 11] or fusing a stack of images with different
exposures, which are called single- and multi-exposure methods, respectively.
In algorithms with a single LDR image, an HDR image can be produced from one
image. However, the generated picture might not be as informative as the HDR
image produced by several LDR images because the amount of detail in one
single picture is limited compared to several images with different exposures.
More precisely, [8] implemented an algorithm that only reconstructs the detail
of bright saturated areas. However, the model is not only not capable of
restoring the detail of dark regions but also does not perform well if the
amount of bright saturation is too much. Thus, [12] first combined several LDR
images and then fed the low-frequency response of the wavelet transform to the
network to produce more detail in a shorter time.

Luckily, multi-exposure methods are more effective and informative compared to
single-exposure techniques. Moreover, these methods perform well when the
images are static [13, 14], while when there are movements in the sequence of
pictures, the ghosting problem emerges, which is almost solved in [15, 16, 17,
18, 19, 20].

Deep learning has been a significant means of producing an HDR image for the
past decade. For instance, [8] produced an HDR picture in the logarithmic
domain with the help of a deep neural network. Additionally, [21], used a
neural network to reconstruct the detail of an image with different exposure
in each row in the irradiance domain. Moreover, unlike other multi-exposure
methods, [13, 14] used a neural network to produce synthetic LDR images with
different exposures from a single image. Furthermore, [16] proposed to first
align images with the help of the optical flow method, and then use a deep
neural network to combine them. Therewith, [15] instead of using optical flow
for alignment, proposed to use two different neural networks first to align
them and then combine the aligned images with the second neural network.
Finally, [22] used a neural network to learn the relative relation between the
inputs and the Ground Truth using input images in different scales.

In this article, we would like to exploit image segmentation with the help of
the Otsu method [23] in HDR imaging to extract the most visible areas of the
images and help the model produce pictures with more detail. Thus, to reach
this point, Visual Attention Modules will be proposed to obtain such regions.
Moreover, in this research, Spatial and Attention modules have been used from
a State-Of-The-Art method, and a new architecture for the Reconstruction stage
was designed and implemented, in which the visual attention and the reference
image were used in the decoder part. Finally, although VAMs helped in
producing pictures with more details and outperformed most of the State-Of-
The-Art methods, the results still illustrated a slight amount of noise that
was extracted from the input images.

In section 3, the State-Of-The-Art in HDR imaging and related image
segmentation is presented. In section 4, the proposed method is discussed in
detail. Section 5 demonstrates the experimental results and comparison with
the State-Of-The-Art methods. Moreover, section 6 concludes this article with
ideas for further works. Finally, the code will be available at the github
page.

##  3 Related work

In this section, we will discuss the State-Of-The-Art methods in the scope of
HDR imaging in the Multi-Exposure category (Section 3.1) and survey
unsupervised Image Segmentation methods for extracting regions (Section 3.2).

###  3.1 Multi-Exposure Methods

[24] proposed a two-stage algorithm, in which the first phase they extracted
features from the input images, and merged them to produce the HDR image in
the latter one. Additionally, to cope with the appeared noise from the gamma
correction operation on input images, i.e. the gamma-corrected Short-Exposure
image becoming similar to Medium-Exposure, they used a U-net to extract
noiseless features from it. Moreover, [25] implemented a model in which images
with lower scales were used to reduce the consuming sources. Additionally, a
novel loss function was defined to focus more on the motion. Furthermore, [26]
forwarded features with different scales to deformable and spatial attention
blocks to align images in the feature space and also extract the features of
the specific areas of the input images. Moreover, [27] proposed a model that
at first estimated the optical flow from the two input images in different
scales and then fused them to produce the final output. In [28], the features
are extracted from different scales and then are processed by sampling and
aggregation modules to align the pixels of the non-reference features.

The work [29] implemented a baseline that had lower computational resources
and acceptable results compared to the other State-Of-The-Art models. They
used a dual attention module, which includes both spatial and channel
attention modules, to cope with misalignment and to better learn the details
of the produced areas. In [30], the authors proposed a model that first
extracts features from input images by multi-scale encoding modules and then
produces an HDR image by progressively dilated U-shape blocks.

[31] demonstrated that the ghosting problem is mainly in short-frequency
signals, and therefore, they proposed a wavelet-based model to merge images in
the frequency domain and avoid any ghosting problems. [32] implemented an
algorithm that extracted dynamic areas of the images with the help of image
segmentation and applied two neural networks separately on the static and
dynamic scenes. Finally, they merged the information to produce an HDR image
without ghosting. In [33] a model based on bidirectional motion estimation was
proposed, in which, the amount of optical flow between LDR images was
estimated by motion estimation with cyclic cost volume and spatial attention
maps, and eventually, an HDR image was produced with the help of the extracted
local and global features. [34] implemented the first multi-bracket HDR
pipeline using event cameras, in which they merged the extracted features of
images and the events to produce an HDR image. [35] proposed a transformer-
based baseline, in which they used a context-aware vision transformer to
extract local and global features to model the movement of objects and the
diversity of intensity.

###  3.2 Image Segmentation

Image segmentation is a crucial task in computer vision, which tries to
partition images into segments to analyze the pictures more easily.
Additionally, image segmentation not only can be used for object recognition,
detection, and medical purposes but also can be applied for extracting regions
of pictures with more details. In [36] images were analyzed in HSV color space
to segment pixels based on Intensity or Hue value. Moreover, two image
segmentation methods were proposed based on luminance: histogram division [37]
and clustering based on Gaussian Mixture Model (GMM) of histogram [38].
Furthermore, [39] calculated an optimal valley point based on the slope
between the histogram value of each pixel and the neighboring points, and used
the computed valley point to segment regions. The literature on the topic is
endless, depending on applications and methodologies, from level set methods
[40] to graph cut [41] to recent deep learning-based frameworks [42].

##  4 Proposed Method

###  4.1 Overview

As cited in [43], it might be beneficial to first segment images based on
exposure information to extract the best and more detailed regions from the
Over- and Under-Exposure regions and exploit this knowledge in reconstructing
an HDR image. Following this idea, in this paper, a model is proposed in
which, with the help of image segmentation, regions with more detail are
segmented first in the preprocessing stage. Finally, they are fed to the model
along with the input images to produce an HDR image with the help of VAMs.

Generally, the model can be divided into several sections. Firstly, the input
images are fed into the feature extraction module, and afterward, the
extracted features enter the attention and spatial alignment modules to cope
with any possible misalignment. Moreover, the input images with their
corresponding masks go to the VAM simultaneously to extract the visible areas
of the LDR images. Next, the outputs of the three modules are fed to the
Reconstruction stage to produce the initial HDR image. Finally, the generated
outcomes with the features of the reference image enter the refinement section
to construct the final HDR image.

###  4.2 Preprocess

In this article, the inputs are three LDR images with different exposures, and
the image with Medium-Exposure is considered the reference image. Moreover,
before feeding the input images to the model, they are first mapped to the HDR
domain with the help of gamma correction. Finally, they are concatenated
channel-wise with their corresponding LDR images.

$\hat{I_{i}}=\frac{(I_{i})^{\gamma}}{t_{i}}\qquad\textrm{for}\,\,i=1,2,3$ (1)

Where $t_{i}$ is the exposure time of $I_{i}$. $\gamma$ is the gamma
correction parameter, which was 2.24, and $\hat{I_{i}}$ is the gamma-corrected
image.

####  4.2.1 Segmentation

Most of the present algorithms in HDR imaging focus more on the approach of
image production, but not many pay attention to how to extract the most
helpful features. Thus, in this research, the regions of the pictures with
more details are segmented and extracted as a preprocess and finally are fed
to the proposed model along with the LDR images as the inputs.

Different methods, such as the neural network and Otsu method were used for
the image segmentation stage; however, the neural network resulted in
overfitting. Thus, the Otsu method has been selected to segment the visible
areas of the pictures. Therefore, the images are converted into the YUV color
space to calculate a threshold based on the histograms of Short- and Long-
Exposure images.

$\mathrm{thresh}_{i}=G(Y_{i})\qquad\textrm{for}\,\,i=1,3$ (2)

In which $Y_{i}$ is the luminance channel of the LDR image, $G()$ is the Otsu
function, and $\mathrm{thresh}_{i}$ is the threshold value of image $i$.

In the Short-Exposure image, because most of the pixels are dark, and the
objective is to extract the regions with visible pixels, the values equal to
or more than the threshold are considered one, and the rest are zero for the
Short-Exposure mask.

$\begin{cases}1&p\geq\mathrm{thresh}_{1}\\\ 0&p<\mathrm{thresh}_{1}\\\
\end{cases}$ (3)

Where $\mathrm{thresh}_{1}$ is the threshold value of the Short-Exposure
image, and $p$ is the pixel.

On the other hand, because most of the pixels in the Long-Exposure image are
saturated, and the visible pixels have the lowest values, the values that are
less than the threshold were considered one, and the rest as zero in the Long-
Exposure mask.

$\begin{cases}0&p\geq\mathrm{thresh}_{3}\\\ 1&p<\mathrm{thresh}_{3}\\\
\end{cases}$ (4)

By doing so, the masks of the areas with more detail are extracted and can
help to produce an HDR image.

Generally, most of the pixels in Short- and Long-Exposure images are too dark
or bright, respectively. Therefore, the location of the areas with surplus
information is extracted and fed to the model. Doing so reduces the amount of
calculation and helps in producing an HDR image with more detail. Fig. 1
demonstrates the segmented and visible regions of both Short- and Long-
Exposure pictures.

Figure 1: Produced masks of Short- and Long-Exposure images.

Moreover, during experiments, three input images with different exposures were
used for image segmentation, in which, after obtaining the suitable areas of
Short- and Long-Exposure images, the remaining regions were extracted from the
Medium-Exposure image. However, the acquired areas of the Medium-Exposure were
not sensible, as most of them were only a few pixels. Thus, two reasons exist
for not using Medium-Exposure in the segmentation stage. First, it would be
challenging to calculate a range for the visibility of the pixels. Second,
Medium-Exposure is the reference image, and the picture will be used in the
neural network. Therefore, it is not necessary to use segmentation for it.

###  4.3 Proposed Method Structure

Figure 2: The total pipeline of the proposed.

As shown in Fig. 2, the proposed algorithm consists of six stages, which will
be discussed separately and in detail.

####  4.3.1 Feature Extraction

Figure 3: The structure of the Feature Extraction Block.

Fig. 3 illustrates the Feature Extraction block, in which a $\mathrm{SepConv}$
is applied to the image to extract 32 feature maps. Afterward, a Max Pool and
an Average Pool are used to not only smooth the features and focus on the
details but also pay more attention to the edges. Next, the outputs of
Poolings are concatenated, and another $\mathrm{Sep\mathrm{Conv+ReLU}}$ is
used to reduce the number of channels to 32. Finally, the extracted features
are Upsampled to make them the same size as the input image. The feature
extraction can be written as follows:

$C_{i}=\mathrm{concat}\left(M(\mathrm{SepConv}(I_{i})),A(\mathrm{SepConv}(I_{i}))\right)$
(5) $F_{i}=\mathrm{Upsample}(\mathrm{ReLU}(\mathrm{SepConv}(C_{i})))$ (6)

for $i=1,2,3$, where $A()$ and $M()$ functions are Max Pooling and Average
Pooling, respectively, and $C_{i}$ is the output of Concatenation. Finally,
$F_{i}$ is the output of the Feature Extraction Block.

####  4.3.2 Visual Attention Module

Figure 4: The structure of the Visual Attention Module (VAM).

As it was mentioned, in this article, Image Segmentation is used to help the
model to produce a better image. Therefore, as shown in Fig. 4, the input
images are multiplied element-wise by their corresponding masks first. By
doing so, the regions with more details are kept, and those that are overly
dark or too bright will be removed. Next, they are fed to the Feature
Extractor to extract Features. Finally, they are added together element-
wisely. The VAM can be formally defined as follows:

$\mathrm{features}_{L}=F(\mathrm{multiply}(\mathrm{mask}_{L},I_{L}))$ (7)
$\mathrm{features}_{H}=F(\mathrm{multiply}(\mathrm{mask}_{H},I_{H}))$ (8)
$V=\mathrm{add}(\mathrm{features}_{L},\mathrm{features}_{H})$ (9)

Where $F$ is a feature extractor function, and $V$ is the output feature of
the VAM.

####  4.3.3 Spatial Alignment Module

Figure 5: The structure of the Spatial Alignment Module.

Because the input LDR images are not aligned, the extracted features from the
LDR images without the gamma correction images are fed to an _ad hoc_ module
for aligning them. To this end, we used the same Feaure-alignment Module used
in [30]. As can be seen in Fig. 5, first a $\mathrm{Conv+ReLU}$ is applied to
the Reference Features, which can be called as $\mathrm{Ref}_{1}$. Next, a
$\mathrm{Conv+ReLU}$ is applied to $\mathrm{Ref}_{1}$ and is multiplied
element-wisely by the input LDR features, which can be called $M_{i}$ (for
$i=1,3$). Finally, another $\mathrm{Conv+ReLU}$ is applied to the
$\mathrm{Ref}_{1}$ and is added element-wisely with $M_{i}$. Formally, the
operation in the module can be written as follows:

$\mathrm{Ref}_{1}=\mathrm{ReLU}(\mathrm{Conv}(\mathrm{ref\,features}))$ (10)
$M_{i}=\mathrm{multiply}(\mathrm{ReLU}(\mathrm{Conv}(\mathrm{Ref}_{1})),\,\mathrm{inp\,features}_{i})$
(11)
$\mathrm{out}_{i}=\mathrm{add}(\mathrm{ReLU}(\mathrm{Conv}(\mathrm{Ref}_{1})),\,M_{i})$
(12)

####  4.3.4 Attention Module

Figure 6: The structure of the Attention Module.

The Attention Module is almost similar to [30], in which, as shown in Fig 6,
feature maps are produced for Short- and Long-Exposure images to merge them
with the reference image as guidance. After feeding the features of gamma-
corrected images with the reference image, they are concatenated. Afterward,
$\mathrm{Sep}\mathrm{Conv+ReLU}$ and $\mathrm{SepConv+Simgoid}$ operations are
applied to them. The module can be considered as follows:

$R_{i}=\mathrm{ReLU}(\mathrm{SepConv}(\mathrm{concat}(f_{i},f_{r}))\quad\textrm{for}\,\,i=1,3$
(13) $S_{i}=\mathrm{Sigmoid}(\mathrm{SepConv}(R_{i}))$ (14)

Where $f_{i}$ and $f_{r}$ are the features of gamma-corrected and reference
images, respectively.

####  4.3.5 Reconstruction

Figure 7: The total Scheme of the Reconstruction stage.

All the extracted features from the modules are concatenated and fed to the
reconstruction stage. As shown in Fig. 7, with the help of four encoder
blocks, the input is merged, and new features are produced. Next, each decoder
block receives features from the encoder along with features of the reference
image and VAM. Finally, a $\mathrm{Sep\mathrm{Conv+ReLU}}$ is used to produce
the output of the stage.

Figure 8: The structure of the blocks in the encoder (left) and the decoder
(right).

Each encoder block (Fig. 8, left) initially applies $\mathrm{SepConv}$,
$\mathrm{Batch\,Normalization}$, and $\mathrm{ReLU}$ layers to the inputs.
Afterward, similar to Feature Extraction Module, $\mathrm{Max}$ and
$\mathrm{AVG}$ $\mathrm{Poolings}$ are used. Finally, they are concatenated
and sent to the next block.

Moreover, each decoder block (Fig. 8, right) consists of three inputs, which
are features of the VAM, features of the reference image, and the output of
the previous block. First, $\mathrm{AVG\,pooling}$ is applied to the first two
inputs to make them the same size as the output of the previous block, and
then they are concatenated with each other. Finally,
$\mathrm{Sep\mathrm{Conv+ReLU}}$ and $\mathrm{Upsampling}$ are used,
respectively.

####  4.3.6 Refinement

Figure 9: The structure of the Refinement Stage.

Unfortunately, the output of the reconstruction stage may have blurry,
saturated, or dark areas; therefore, to cope with such possible issues with
the help of features of the reference image, a refinement section also has
been added.

As Fig. 9 illustrates, $\mathrm{Sep\mathrm{Conv+ReLU}}$ is applied to the
features of the reference image to reduce the number of feature maps.
Furthermore, after concatenating the inputs, $\mathrm{SepConv}$ and
$\mathrm{Sep\mathrm{Conv+ReLU}}$ are used, respectively. The process is
repeated two more times, and eventually, $Conv+Sigmoid$ is applied to produce
the final image in Sigmoid space. The process in Refinement can be represented
in pseudo-code as shown in Algorithm 1.

Algorithm 1 Pseudo-code of Refinement Stage.

$\hat{f}_{r}=\mathrm{ReLU}(\mathrm{SepConv}(f_{r}))$

$i\leftarrow 0$

while $i<3$ do

if $i==0$ then

$c\leftarrow concat(Reconstruction_{o},\hat{f}_{r})$

$x\leftarrow\mathrm{ReLU}(ConvSep(ConvSep(c)))$

else

$c\leftarrow concat(x,\hat{f}_{r})$

$x\leftarrow\mathrm{ReLU}(ConvSep(ConvSep(c)))$

end if

$i\leftarrow i+1$

end while

$out\leftarrow Sigmoid(Conv(x))$

Notice that, in this research, the Ground Truth images are mapped from HDR
Space into sigmoid space. Indeed, based on our experiments, transforming the
values into sigmoid helps the network converge more conveniently. The reason
for changing the space is that the values in HDR space are too large, and a
model with a low number of parameters is not able to learn to produce an HDR
image correctly; therefore, by mapping them to sigmoid space, the proposed
model outperforms the model in the HDR space.

##  5 Experiments and Results

###  5.1 Dataset

A new dataset was collected for HDR Imaging Challenge [44, 45]. In this
dataset, two types of pictures (Single-Exposure and Multi-Exposure images)
were provided; however, Multi-Exposure images only were used in this research.
More specifically, this dataset includes images from [46] that were generated
as follows. First, HDR images were produced natively by two Alexa Arri cameras
with a mirror rig; then, their corresponding LDR images were generated
synthetically with noise sources. There are approximately 1500 pairs of
HDR/LDR images in this dataset for the training set, 40 for the validation
set, and 200 pictures for the test set with a resolution of 1900x1060.
However, in this research, we randomly selected 200 images of the training set
as a test set and trained the model with around 1300 pairs.

###  5.2 Implementation Details

Dataset | NTIRE Challenge
---|---
Optimizer | Adam Optimizer
Initial LR | 0.001 with LR decay
|
Epoch | 100
Loss | MAE

Batch Size
---
Input Size
Augmentation

Train | Validation
---|---
16 | 2
256x256 | 1920x1088
True | False

Table 1:  Brief highlights regarding the training and validation settings for
the proposed method.

The highlights of the model are demonstrated in Table 1 briefly. Additionally,
the weights of the model were initialized randomly and no pre-trained weights
were used. Finally, the information regarding the proposed method will be
discussed in the following subsections.

####  5.2.1 Loss function

The Mean Absolute Error (MAE) loss function is used to train the model. The
difference is that the Ground Truth is first mapped to Sigmoid Domain, and
eventually, MAE is calculated in Sigmoid space between the Ground Truth and
the output of the model.

$GT_{n}=sigmoid(GT)$ (15) $L(\hat{y},GT_{n})=|GT_{n}-\hat{y}|$ (16)

Where $GT_{n}$ is the Ground Truth image in the new domain, and $L$ is the
loss between Ground Truth and the output.

Furthermore, after training the model in sigmoid space, inverse sigmoid is
used to re-map the output to HDR space. The inverse sigmoid can be written as
follows:

$HDR=\log(\frac{\hat{(y)}}{1-\hat{y}})$ (17)

Where $HDR$ is the output in HDR space and $\hat{y}$ is the image in the
sigmoid domain.

####  5.2.2 Training

Flipping the images vertically or horizontally is also used as an augmentation
method during training. Moreover, before feeding the images to the model, they
are resized into 256x256. The reason for doing so instead of producing patches
is that some generated patches from the masks may be totally black or
completely white, which causes the model to pay less attention to the images
with Short-Exposure.

Moreover, batch size and the number of epochs are set to 16 and 100,
respectively. In this article, Adam Optimizer with an initial learning of
0.001 is used, and it will be reduced by a factor of 0.1 if the validation
accuracy does not improve. Finally, the whole model is implemented in
Tensorflow (Keras) framework and is trained on a DGX-A100 GPU.

####  5.2.3 Validation

The images are first padded from 1900x1060 to 1920x1080 and then fed to the
model without any augmentation methods during validation.

###  5.3 Evaluation Metrics and Comparison

####  5.3.1 Quantitative Comparison

The results in this paper are compared with the State-Of-The-Art methods by
$PSNR$ in HDR and Tone-mapped domains. The $PSNR-\mu$ is the tone-mapped
version, where the images were tone-mapped in $\mu-law$. Moreover, the results
are compared with the State-Of-The-Art methods in $GMACs$ and the number of
parameters.

Methods | PSNR |  $\mu$-PSNR | GMACs | Param. × $10^{3}$
---|---|---|---|---
GSANet | 36.88 | 35.57 | $\underline{199.38}$ | $\boldsymbol{80}$
DRHDR | 38.5 | $\boldsymbol{36.91}$ | 1701.932 | 1190
Vien et al. | $\underline{39.44}$ | 35.39 | $\boldsymbol{198.819}$ | 1301
ours | $\boldsymbol{43.25}$ | $\underline{35.86}$ | 234.107 | $\underline{570}$
Table 2:  Comparison with the State-Of-The-Art methods. The bold numbers are
the best values, and the underline ones are the second best.

As mentioned in [45], the challenge focused on two tracks, which were Fidelity
and low complexity. In the first one, the methods were required to obtain the
highest $\mu-PSNR$ while the $GMACs$ value is less than 200. In the latter
track, it was asked to reduce the $GMACs$ value to less than the baseline
method while the $PSNR$ and $\mu-PSNR$ values are almost the same as the
baseline method. The proposed method has been compared with GSANet [24], DRHDR
[26], and Vein et al. [33] methods. As can be seen, Table 2 shows the proposed
method has the highest value in terms of $PSNR$, while having the second
highest value in $\mu-PSNR$. On the other hand, Vien et al. [33] had the
lowest GMACs value, and GSANet is ranked second lowest. Moreover, it is
visible that in terms of the number of parameters, GSANet has the lowest and
the proposed method is in the second place among the algorithms.

Methods | PSNR | Mu-PSNR
---|---|---
Ours (HDR Space) | 42.4 | 35.28
Ours (Sigmoid Space) | 43.25 | 35.86
Table 3: Comparison between the proposed method in HDR and Sigmoid Spaces.

Furthermore, for more study, the proposed method was trained and tested in HDR
and Sigmoid Spaces to check which space is superior for training the model.
Thus, as Table 3 demonstrates, the proposed method in Sigmoid Space
outperformed the algorithm in the HDR domain. Moreover, during training, the
model in Sigmoid space converged quicker than the model in the HDR domain.

Figure 10: Qualitative Comparison with the State-Of-The-Art. The first row of
each scene contains short, medium, and long exposure images, respectively. The
second row includes ours, DRHDR, An et al., and GSANet outcomes, respectively.

####  5.3.2 Qualitative Comparison

As can be seen in Fig. 10, the produced images by ours, worked better in terms
of image reconstruction compared to DRHDR and An et al. methods. More
specifically, Fig. 10 demonstrates the results of ours, DRHDR [26], An et al.
[33], and GSANet [24]. As can be seen, the output of An et al. in the first
scene has distortion in the bright areas, and it is visible that the algorithm
cannot restore the details from these areas correctly. Furthermore, there is
some degradation in the dark regions too. Moreover, although DRHDR worked
great and reconstructed both areas, this method was not able to acquire the
details in over-saturated areas. For instance, looking at the two red and
green boxes, the model did not reconstruct the details of the hands and the
shirt, while the proposed method produced more detail in these two regions.
Moreover, produced image from the GSANet method shows significant details and
is almost similar to ours. More precisely, although both methods could
reconstruct the shirt nicely, the details of the hand in the GSANet are more
than ours.

Additionally, in the second scene, the DRHDR and An et al. methods were not
able to reconstruct the branches that were only visible in the short exposure
image and restored only a part of them. In contrast, the proposed method and
the GSANet worked almost well in this regard. Finally, looking at the last
scene, it is visible that the proposed method outperformed the first two
algorithms and reconstructed more details in both dark and bright areas, and
the details of the sky show this point.

Furthermore, although the segmentation helped the model to produce better
results, the method might encounter two possible issues. Firstly, due to
plausible noise in input images, using segmentation for extracting visible
areas may also acquire the noise, and the produced image might become noisy.
Lastly, although spatial alignment and attention modules are used to avoid any
possible ghosting problems, if the input images have a severe amount of
movement, the output might also encounter a ghosting issue. Because the
segmentation is applied to the Short- and Long-Exposure images and extracts
their visible areas. Therefore, some parts of the images might not be aligned.
Moreover, for future research, we would like to investigate possible methods
to use segmentation and avoid any likely noise or misalignment.

##  6 Conclusion

In this article, we proposed a new method for HDR imaging with the help of
image segmentation. More specifically, we first applied the Otsu method on
Short- and Long-Exposure images to acquire the areas with more details.
Afterward, the input images along with the segmentation outputs were fed to
the model to produce the HDR image. The results show that the proposed method
outperformed the State-Of-The-Art and generated more details. However, the
proposed model is not free of issues, and in case of possible noise or
misalignment in input images, the output might have a slight amount of noise
or misalignment due to extracting areas of input images. Therefore, for future
research, we would like to focus on investigating these two problems.

## References

  * [1] S. Nayar and T. Mitsunaga, “High dynamic range imaging: spatially varying pixel exposures,” in Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662), vol. 1, pp. 472–479 vol.1, 2000.
  * [2] J. Tumblin, A. Agrawal, and R. Raskar, “Why i want a gradient camera,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 103–110 vol. 1, 2005.
  * [3] M. McGuire, W. Matusik, H. Pfister, B. Chen, J. F. Hughes, and S. K. Nayar, “Optical splitting trees for high-precision monocular imaging,” IEEE Computer Graphics and Applications, vol. 27, no. 2, pp. 32–42, 2007.
  * [4] M. D. Tocci, C. Kiser, N. Tocci, and P. Sen, “A versatile hdr video production system,” ACM Trans. Graph., vol. 30, jul 2011.
  * [5] S. Hajisharif, J. Kronander, and J. Unger, “Adaptive dualiso hdr reconstruction,” EURASIP Journal on Image and Video Processing, vol. 2015, p. 41, Dec 2015.
  * [6] H. Zhao, B. Shi, C. Fernandez-Cull, S.-K. Yeung, and R. Raskar, “Unbounded high dynamic range photography using a modulo camera,” in 2015 IEEE International Conference on Computational Photography (ICCP), pp. 1–10, 2015\.
  * [7] A. Serrano, F. Heide, D. Gutierrez, G. Wetzstein, and B. Masia, “Convolutional sparse coding for high dynamic range imaging,” in Proceedings of the 37th Annual Conference of the European Association for Computer Graphics, EG ’16, (Goslar, DEU), p. 153–163, Eurographics Association, 2016.
  * [8] G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger, “Hdr image reconstruction from a single exposure using deep cnns,” ACM Trans. Graph., vol. 36, nov 2017.
  * [9] L. She, M. Ye, S. Li, Y. Zhao, C. Zhu, and H. Wang, “Single-image hdr reconstruction by dual learning the camera imaging process,” Engineering Applications of Artificial Intelligence, vol. 120, p. 105947, 2023\.
  * [10] P.-H. Le, Q. Le, R. Nguyen, and B.-S. Hua, “Single-image hdr reconstruction by multi-exposure generation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2023.
  * [11] G. Cao, F. Zhou, K. Liu, A. Wang, and L. Fan, “A decoupled kernel prediction network guided by soft mask for single image hdr reconstruction,” ACM Trans. Multimedia Comput. Commun. Appl., vol. 19, feb 2023.
  * [12] A. Omrani, M. R. Soheili, and M. Kelarestaghi, “High dynamic range image reconstruction using multi-exposure wavelet hdrcnn,” in 2020 International Conference on Machine Vision and Image Processing (MVIP), pp. 1–4, 2020.
  * [13] S. Lee, G. H. An, and S.-J. Kang, “Deep chain hdri: Reconstructing a high dynamic range image from a single low dynamic range image,” IEEE Access, vol. 6, pp. 49913–49924, 2018.
  * [14] Y. Endo, Y. Kanamori, and J. Mitani, “Deep reverse tone mapping,” ACM Trans. Graph., vol. 36, nov 2017.
  * [15] G. R. K.S., A. Biswas, M. S. Patel, and B. H. P. Prasad, “Deep multi-stage learning for hdr with large object motions,” in 2019 IEEE International Conference on Image Processing (ICIP), pp. 4714–4718, 2019.
  * [16] N. K. Kalantari and R. Ramamoorthi, “Deep high dynamic range imaging of dynamic scenes,” ACM Trans. Graph., vol. 36, jul 2017.
  * [17] S. Wu, J. Xu, Y.-W. Tai, and C.-K. Tang, “Deep high dynamic range imaging with large foreground motions,” in Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [18] Q. Yan, D. Gong, Q. Shi, A. v. d. Hengel, C. Shen, I. Reid, and Y. Zhang, “Attention-guided network for ghost-free high dynamic range imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [19] K. R. Prabhakar, R. Arora, A. Swaminathan, K. P. Singh, and R. V. Babu, “A fast, scalable, and reliable deghosting method for extreme exposure fusion,” in 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1–8, 2019.
  * [20] K. R. Prabhakar, S. Agrawal, D. K. Singh, B. Ashwath, and R. V. Babu, “Towards practical and efficient high-resolution hdr deghosting with cnn,” in Computer Vision – ECCV 2020 (A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds.), (Cham), pp. 497–513, Springer International Publishing, 2020.
  * [21] V. G. An and C. Lee, “Single-shot high dynamic range imaging via deep convolutional neural network,” in 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1768–1772, 2017.
  * [22] Q. Yan, D. Gong, P. Zhang, Q. Shi, J. Sun, I. Reid, and Y. Zhang, “Multi-scale dense networks for deep high dynamic range imaging,” in 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 41–50, 2019\.
  * [23] N. Otsu, “A threshold selection method from gray-level histograms,” IEEE transactions on systems, man, and cybernetics, vol. 9, no. 1, pp. 62–66, 1979\.
  * [24] F. Li, R. Gang, C. Li, J. Li, S. Ma, C. Liu, and Y. Cao, “Gamma-enhanced spatial attention network for efficient high dynamic range imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1032–1040, June 2022.
  * [25] Y. Deng, Q. Liu, and T. Ikenaga, “Attention-guided network with inverse tone-mapping guided up-sampling for hdr imaging of dynamic scenes,” Multimedia Tools and Applications, vol. 81, pp. 12925–12944, Apr 2022.
  * [26] J. Mar’in-Vega, M. Sloth, P. Schneider-Kamp, and R. Rottger, “Drhdr: A dual branch residual network for multi-bracket high dynamic range imaging,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 843–851, 2022.
  * [27] Q. Ye, M. Suganuma, J. Xiao, and T. Okatani, “Learning regularized multi-scale feature flow for high dynamic range imaging,” 2022.
  * [28] J. Xiao, Q. Ye, T. Liu, C. Zhang, and K.-M. Lam, “Multi-scale sampling and aggregation network for high dynamic range imaging,” 2022.
  * [29] Q. Yan, S. Zhang, W. Chen, Y. Liu, Z. Zhang, Y. Zhang, J. Q. Shi, and D. Gong, “A lightweight network for high dynamic range imaging,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 823–831, 2022.
  * [30] G. Yu, J. Zhang, Z. Ma, and H. Wang, “Efficient progressive high dynamic range image restoration via attention and alignment network,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1123–1130, 2022.
  * [31] T. Dai, W. Li, X. Cao, J. Liu, X. Jia, A. Leonardis, Y. Yan, and S. Yuan, “Wavelet-based network for high dynamic range imaging,” 2022.
  * [32] K. R. Prabhakar, S. Agrawal, and R. V. Babu, “Segmentation guided deep hdr deghosting,” 2022.
  * [33] A. G. Vien, S. Park, T. T. N. Mai, G. Kim, and C. Lee, “Bidirectional motion estimation with cyclic cost volume for high dynamic range imaging,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1182–1189, 2022.
  * [34] N. Messikommer, S. Georgoulis, D. Gehrig, S. Tulyakov, J. Erbach, A. Bochicchio, Y. Li, and D. Scaramuzza, “Multi-bracket high dynamic range imaging with event cameras,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 547–557, June 2022.
  * [35] Z. Liu, Y. Wang, B. Zeng, and S. Liu, “Ghost-free high dynamic range imaging with context-aware transformer,” in Computer Vision – ECCV 2022 (S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, eds.), (Cham), pp. 344–360, Springer Nature Switzerland, 2022.
  * [36] A. Vadivel, M. Mohan, S. Sural, and A. K. Majumdar, “Segmentation using saturation thresholding and its application in content-based retrieval of images,” in Image Analysis and Recognition (A. Campilho and M. Kamel, eds.), (Berlin, Heidelberg), pp. 33–40, Springer Berlin Heidelberg, 2004.
  * [37] Y. Kinoshita and H. Kiya, “Scene segmentation-based luminance adjustment for multi-exposure image fusion,” IEEE Transactions on Image Processing, vol. 28, no. 8, pp. 4101–4116, 2019.
  * [38] Y. Kinoshita and H. Kiya, “Automatic exposure compensation using an image segmentation method for single-image-based multi-exposure fusion,” APSIPA Transactions on Signal and Information Processing, vol. 7, p. e22, 2018\.
  * [39] B. D. Lee and M. H. Sunwoo, “Hdr image reconstruction using segmented image learning,” IEEE Access, vol. 9, pp. 142729–142742, 2021.
  * [40] A. Mitiche and I. B. Ayed, Variational and level set methods in image segmentation, vol. 5.  Springer Science & Business Media, 2010.
  * [41] F. Yi and I. Moon, “Image segmentation: A survey of graph-cut methods,” in 2012 international conference on systems and informatics (ICSAI2012), pp. 1936–1941, IEEE, 2012.
  * [42] S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 7, pp. 3523–3542, 2021.
  * [43] L. Wang and K.-J. Yoon, “Deep learning for hdr imaging: State-of-the-art and future trends,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 8874–8895, 2022.
  * [44] E. Pérez-Pellitero, S. Catley-Chandar, A. Leonardis, R. Timofte, X. Wang, Y. Li, T. Wang, F. Song, Z. Liu, W. Lin, X. Li, Q. Rao, T. Jiang, M. Han, H. Fan, J. Sun, S. Liu, , et al., “Ntire 2021 challenge on high dynamic range imaging: Dataset, methods and results,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 691–700, 2021.
  * [45] E. Pérez-Pellitero, S. Catley-Chandar, R. Shaw, A. Leonardis, R. Timofte, Z. Zhang, C. Liu, Y. Peng, Y. Lin, G. Yu, et al., “Ntire 2022 challenge on high dynamic range imaging: Methods and results,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1009–1023, 2022.
  * [46] J. Froehlich, S. Grandinetti, B. Eberhardt, S. Walter, A. Schilling, and H. Brendel, “Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays,” in Digital Photography X (N. Sampat, R. Tezaur, S. Battiato, and B. A. Fowler, eds.), vol. 9023, p. 90230X, International Society for Optics and Photonics, SPIE, 2014.

  *[HDR]: High Dynamic Range
  *[LDR]: Low Dynamic Range
  *[VAMs]: Visual Attention Module
  *[GMM]: Gaussian Mixture Model
  *[VAM]: Visual Attention Module
  *[MAE]: Mean Absolute Error