# Deep Polarimetric HDR Reconstruction

Juiwen Ting, Moein Shakeri, Hong Zhang The authors are with Department of
Computing Science, University of Alberta, CanadaManuscript received April 19,
2021; revised August 16, 2021.

###### Abstract

This paper proposes a novel learning based high-dynamic-range (HDR)
reconstruction method using a polarization camera. We utilize a previous
observation that polarization filters with different orientations can
attenuate natural light differently, and we treat the multiple images acquired
by the polarization camera as a set acquired under different exposure times,
to introduce the development of solutions for the HDR reconstruction problem.
We propose a deep HDR reconstruction framework with a feature masking
mechanism that uses polarimetric cues available from the polarization camera,
called Deep Polarimetric HDR Reconstruction (DPHR). The proposed DPHR obtains
polarimetric information to propagate valid features through the network more
effectively to regress the missing pixels. We demonstrate through both
qualitative and quantitative evaluations that the proposed DPHR performs
favorably than state-of-the-art HDR reconstruction algorithms.

###### Index Terms:

High dynamic range imaging, polarization, convolutional neural network,
feature masking

##  I Introduction

In real-world scenes, the luminance varies over several orders of magnitude
and is presented as high-dynamic-range (HDR). Unfortunately, standard digital
cameras can only capture a limited fraction of this range due to sensor
constraints. As a result, the images captured by such cameras are low-dynamic-
range (LDR) with over- or under-exposed regions that do not reflect the human
visual system’s ability to simultaneously identify details in both bright and
dark regions of a scene. In LDR images, information is lost in the over- or
under-exposed regions, and thus presents challenges to computer vision related
robotic tasks [1, 2, 3]. In HDR reconstruction, the method seeks to restore
the lost information, and aims to output an HDR image that prevails a wider
range of illumination than is available directly from the camera sensor.

Classical HDR reconstruction methods propose to generate an HDR image from a
sequence of LDR images of the scene taken at different exposures [4, 5].
However, these approaches require multiple exposure images as input, which may
not always be available and practical. Single-shot HDR techniques resolves the
above issue by requiring only a single capture. As a result, they have
attracted considerable research efforts in recent years.

(a) Input LDRs (we show the other input $M_{1,i}$ in Fig. 2)

(b) DPHR (final result)

(c) ground truth

Figure 1: Sample HDR reconstruction result of the proposed DPHR method.

A plethora of single-shot HDR works have been proposed [6, 7, 8, 9, 10, 11].
However, almost all existing methods have some significant drawbacks including
ignoring reconstruction for under-exposed regions, neglecting dynamic range
expansion for properly exposed pixels, or generating artifacts (i.e., color
distortions, halo artifacts). An interesting recent development of imaging
technology is a polarization camera that uses multi-directional on-chip linear
polarizers to capture multi-polarization images in single-shot. Polarization
is a property of light, which can convey rich illumination information about
the environment to help with HDR reconstruction. Recent studies show that the
polarization camera can handle the aforementioned drawbacks due to the
acquisition of the different polarization filtered images. These polarization
filtered images can capture different intensity values, where a region that is
poorly exposed in one image may be properly exposed in another image. An
existing polarimetric method by Wu et al. [12] is a traditional model based
approach, and is vulnerable to deal with a) poorly exposed pixels in polarized
images, and b) pixels with low degree of polarization. Another existing
polarimetric method by Ting et al. [13] is a deep-learning-based approach,
whose input consists of the four images from the polarization camera. However,
where the degree of polarization is high, the method proposed by Ting et al.
[13] is unable to take advantage of the known physics of the polarization
images as compared to the method by Wu et al. [12]. Specifically, Ting et al.
[13] does not utilize degree of polarization to construct HDR images.

Motivated by these shortcomings, we present a deep-learning-based solution to
the problem of HDR reconstruction using a polarization camera, called Deep
Polarimetric HDR Reconstruction (DPHR). We design our approach based on two
main observations. First, pixels in an LDR image can be a) under-exposed, b)
properly exposed, or c) over-exposed. In HDR imaging, the goal is to not only
recover the lost information in poorly exposed regions, but to also increase
the bit-depth of the properly exposed pixels beyond 8-bits. Therefore, we use
the polarimetric cues to design a novel reconstruction process to a) identify
reconstruction regions based on our feature masking mechanism for the
convolutional neural network (CNN) to focus its learning upon, and b)
propagate the properly exposed pixels that contains the valid features through
the network. Second, the method by Wu et al. [12] is able to expand the
dynamic range in areas that have a strong measure of polarized light.
Therefore, we combine the HDR images generated by a traditional model based
approach [12] and our deep-learning-based DPHR model to obtain the final HDR
result. We evaluate the effectiveness of our method on outdoor scenes captured
by a polarization camera. Fig. 1 shows a sample result of the proposed DPHR
method. The main contributions of our work are as follows:

  * •

We propose a feature masking mechanism to solve the HDR reconstruction problem
using a CNN and a polarization camera. This masking approach is based on the
domain knowledge of polarization, and effectively identifies the invalid
content that requires HDR recovery. Thus, the quality of the final results
expands the dynamic range by synthesizing realistic textures and has reduced
artifacts.

  * •

We introduce an HDR reconstruction formulation that combines the complementary
results from the traditional model-based and deep-learning-based approaches to
generate the final output.

The remainder of this paper is organized as follows. Related works on HDR
reconstruction and the background physics of light polarization are summarized
in Section 2. Section 3 presents the details of our proposed DPHR method based
on the polarization camera. Experimental results and discussions are described
in Section 4, and concluding remarks in Section 5.

##  II Related Works

In this section, we first review the relevant works to the problem of single-
shot and multiple-shot HDR reconstruction, and then explain the background
physics of light polarization that is the basis of our proposed method.

###  II-A Multiple-shot HDR Reconstruction

Multiple-shot HDR reconstruction is a well-known area of research with a
series of works that focuses on removing ghosting artifacts caused by moving
objects or misalignment in the images shot from multiple acquisitions. Several
traditional works perform alignment and HDR reconstruction in a unified
optimization system. For example, Sen et al. [14] and Hu et al. [15] proposed
patch-based optimization systems to fill in the missing details due to the
saturated regions in the reference image using other images within the stack.
Other works by Oh et al. [16], Lee et al. [17] and Li et al. [18] use rank
minimization where image misalignments are considered as sparse outliers, to
align and reconstruct an HDR image. However, the above techniques are not
data-driven and produce unsatisfactory results in challenging cases where the
reference has pronounced saturated regions.

To alleviate the aforementioned issues, a group of works explores learning
based approaches to directly align and reconstruct pixels in all saturated
regions. Some works proposed a pre-processing step to align the multi-exposure
images prior to feeding them to a CNN. For example, pre-processed steps based
on optical flow and homography transformation are proposed, respectively by
Kalantari et al. [19] and Wu et al. [20]. Other works implemented an end-to-
end HDR framework. For example, Yan et al. [4], Liu et al. [21] and Pu et al.
[22] presented attention-merging networks to generate ghost-free HDR imaging.
Ma et al. [23] proposed an attention-merging network with guided image
filtering to optimize the fusion weight maps of the multi-exposure images. Niu
et al. [24] implemented a generative adversarial network (GAN) based model to
fuse the multi-exposure images into an HDR image. In a similar vein, Kalantari
et al. [25] uses two connected networks to fuse multiple LDR frames, and
generate an HDR video. All of these methods need multiple shots, though some
methods have accounted for motions between the input images [4, 21, 22],
suppressing severe motions remain a challenge thus is not practical in
applications with moving cameras. On the other hand, perfectly aligned
polarization images can be captured in just a single-shot, which is suitable
for applications with both stationary and moving cameras.

###  II-B Single-shot HDR Reconstruction

Single-shot HDR reconstruction is a well-studied area of research with a
plethora of solutions developed for various applications in computer vision
and robotics. A classical and important group of these solutions is expansion
operators, where different heuristic operators are implemented to transform a
single LDR image to an HDR image[26, 27, 28, 29, 30, 31, 32]. However, these
techniques make assumptions about the scene and use heuristics that result in
deviation from the true luminance value, and thus tend to induce pixel
artifacts.

To address these limitations, the second group of methods take advantage of
the power of CNNs and directly expands the dynamic range to recover pixels in
poorly exposed regions. Several novel architectures have been proposed to
utilize contextual information extracted by convolutional layers to improve
HDR reconstruction results [6, 7, 8, 9, 10, 11]. For example, Marnerides et
al. [6] proposed a three-branch CNN to extract global, semi-local, and local
features that exploit feature recovery at various scales, for reconstruction
in all poorly exposed regions. However, their method outputs overly bright and
smooth images as it over-enhances the extracted illumination features. Liu et
al. [11] presented an HDR framework that incorporates the domain knowledge of
the LDR image formation pipeline. They trained three specialized networks to
reverse the image formation steps of dynamic range clipping, non-linear
mapping from a CRF, and quantization to reconstruct an HDR image. Chen et al.
[33] proposed a U-Net like network to solve the joint tasks of HDR
reconstruction, denoising, and dequantization. Zhang et al. [34] designed a
three-stage cascade network to decompose the LDR image to learn the HDR
content. Eilersten et al. [7] proposed a U-Net like architecture to regress
details in over-exposed regions with a fixed mask, and later combined the
prediction with the input image for the final result. Santos et al. [8]
developed an autoencoder structure with a feature masking mechanism to also
regress details in over-exposed regions, and then reconstruct the HDR image
through combining with the input image for the end result. Namely, the feature
masking mechanism by Santos et al. [8] is based on a soft mask computed from
the pixel intensity of the input image. Thus, dark pixels in the soft mask
correspond to over-exposed pixels where the CNN focuses its learning upon, and
the soft mask is continuously updated by the feature weights of each
convolutional layer. However, both methods by Eilersten et al. [7] and Santos
et al. [8] neglect HDR reconstruction for under-exposed regions, and is unable
to improve HDR estimation for properly exposed regions, as it applies a mask
that focuses the reconstruction on over-exposed regions. Furthermore, the
methods by Eilersten et al. [7] and Santos et al. [8] has unreliable
reconstruction in all poorly exposed regions due to the lack of prior to
enforce consistency.

The third group of methods rely on specialized cameras or filters, such as
neuromorphic camera [35], saturation self-reset camera [36], modulo camera
[37], snapshot camera [38, 39, 40], polarization camera [12] or cross-screen
filter [41] to recover HDR images. Alternatively, some works presented end-to-
end joint learning of optics and HDR reconstruction. For example, Metzler et
al. [42] and Sun et al. [43] jointly trained an optical-based element and a
CNN to hallucinate the HDR content from a single LDR image. As another
example, Martel et al. [44], Alghamid et al. [45, 46], Fotiadou et al. [47]
and Serrano et al. [48] proposed joint designs to optimize a snapshot sensor
and a deep learning algorithm to reconstruct HDR images A snapshot camera is
an imaging device capable of acquiring multiple images in a single capture.
This is beneficial for HDR imaging as more information (i.e., luminance) about
the scene can be captured in a single-shot, to help recover the scene’s
dynamic range. For example, Nayar et al. [39] proposed a learning based
approach to handle the images captured by a spatially varying exposure camera
(i.e., multiple exposure images), and generate an HDR image. Recently,
traditional model based [12] and deep-learning-based [13] approaches using a
polarization camera have been proposed for HDR reconstruction, and showed
promising results. However, the method by Wu et al. [12] is vulnerable to
challenging cases with significant poorly exposed and low degree of
polarization pixels. Though the method by Ting et al. [13] can reconstruct
better details than Wu et al. [12], Ting et al. [13] still outputs limited
details. In contrast to the method by Ting et al. [13] that uses only the
polarization images to achieve HDR reconstruction, we propose an enhanced
deep-learning-based HDR reconstruction method called DPHR, that uses both the
polarization images and polarimetric cues captured by the polarization camera
to reconstruct HDR images as will be explained in Section 3.

Figure 2: Overview of the deep neural network used in constructing $H_{d}$ in
our proposed DPHR framework. The LDR input images $L_{i}$, $i$ = 1,2,3,4 are
propagated through the network, while updated by the corresponding mask
$M_{l,i}$ before going through the convolutional layers. The mask at each
layer is obtained by updating the mask from the previous layer.

###  II-C Background Physics of Light Polarization

The proposed DPHR method is based on a polarization camera, which implements
pixel-level polarizer filters that has made it possible to acquire real-time
and high resolution measurement of incident polarization information. Each
calculation unit has four pixels with four on-chip directional polarizer
filters, at 0∘, 45∘, 90∘, and 135∘, to capture four spatially aligned
polarization images. Each polarization image is a filtered version of the
image irradiance $I$ captured by the polarization camera, and thus the effect
of a polarizer on image irradiance can be written as [49]:

$\begin{gathered}I_{0},I_{90}=0.5\times I(1\pm\rho\cos 2\theta)\\\
I_{45},I_{135}=0.5\times I(1\pm\rho\sin 2\theta)\end{gathered}$ (1)

where $\rho\in$ [0,1] denotes the degree of linear polarization (DoLP), and
measures the portion of light that is polarized for a given pixel. $\theta\in$
[0∘, 180∘] denotes the angle of linear polarization (AoLP), and measures the
direction of light in which the polarized light oscillates at a given pixel.
The DoLP can be defined as follows:

$\rho=\frac{I_{max}-I_{min}}{I_{max}+I_{min}}$ (2)

where $I_{max}$ and $I_{min}$ are the maximum and the minimum measured
radiance at every pixel, respectively. These two parameters along with AoLP
$\theta$ can be determined by three or more polarization images with different
but known polarizer filter angles. As our camera can simultaneously capture
four images, the DoLP and AoLP can be readily computed, and their closed form
solutions can be found in the work by Huynh et al. [50].

In general, the relation between image irradiance $I_{i}$ and pixel value
$L_{i}$ at exposure time $t_{0}$ can be written as [51]:

$L_{i}=f(I_{i}t_{0})$ (3)

where $f$ is the camera response function. By substituting (1) into (3), and
applying the reciprocity relation in the work by Debevec and Malik [51], we
obtain:

$\begin{gathered}t_{1},t_{3}=0.5\times t_{0}(1\pm\rho\cos 2\theta)\\\
t_{2},t_{4}=0.5\times t_{0}(1\pm\rho\sin 2\theta)\end{gathered}$ (4)

The polarization image formation model in the case of HDR reconstruction is
provided by (4). The key observation is that when the incoming light is not
entirely unpolarized ($\rho\neq$ 0), the four polarizer orientation images
experience different exposure times, effectively creating the condition for
multiple exposures. In other words, each polarizer orientation image covers a
different luminance range, and thus can be combined to recover a wider dynamic
range by synthesizing realistic textures.

##  III Proposed Method

Fig. 2 shows an overview of the deep neural network used in constructing
$H_{d}$ in our proposed DPHR framework. The DPHR network architecture is
summarized in Table I. Our goal is to reconstruct HDR values for all properly
exposed and poorly exposed pixels. We achieve this using a CNN that takes as
input four LDR polarization images captured by the polarization camera, and
estimates the missing HDR information in all properly exposed and poorly
exposed regions with the help from the feature masking mechanism. We compute
an input mask image for the feature masking mechanism. Then the input mask
updates the features at each convolutional layer using the obtained
polarimetric information, in order to reduce the magnitude of the features
generated by the invalid areas, and thus better propagate the valid features.
Finally, we combine the predicted HDR image with the HDR image generated from
the traditional model based approach [12] to obtain the final HDR result.

TABLE I: Overview of DPHR network architecture Layer | Stage | # Filters | Filter size | Conv. stride | Activation
---|---|---|---|---|---
1 | conv+act | 64 | (7,7) | (2,2) | ReLU
2 | conv+bn+act | 128 | (5,5) | (2,2) | LeakyReLU
3 | conv+bn+act | 256 | (5,5) | (2,2) | LeakyReLU
4 | conv+bn+act | 512 | (3,3) | (2,2) | LeakyReLU
5 | conv+bn+act | 512 | (3,3) | (2,2) | LeakyReLU
6 | conv+bn+act | 512 | (3,3) | (2,2) | LeakyReLU
7 | conv+bn+act | 512 | (3,3) | (2,2) | LeakyReLU
8 | conv+bn+act | 512 | (3,3) | (2,2) | LeakyReLU
9 | conv+bn+act | 512 | (3,3) | (1,1) | LeakyReLU
10 | conv+bn+act | 512 | (3,3) | (1,1) | LeakyReLU
11 | conv+bn+act | 512 | (3,3) | (1,1) | LeakyReLU
12 | conv+bn+act | 256 | (3,3) | (1,1) | LeakyReLU
13 | conv+bn+act | 128 | (3,3) | (1,1) | LeakyReLU
14 | conv+bn+act | 64 | (3,3) | (1,1) | LeakyReLU
15 | conv+act | 3 | (3,3) | (1,1) | LeakyReLU

###  III-A Deep Polarimetric HDR Reconstruction

As discussed in Section II-C, the polarization state of light can be
quantified by DoLP, which determines the portion of light that is polarized
for a given pixel. Where DoLP is low, it indicates a low measure of
polarization that can be attributed to similar pixel values or pixel
saturation among the four polarization images. Such regions correspond to a
higher weight in the CNN, and thus are regions that the CNN will direct its
learning upon. In other words, the polarimetric information provided by DoLP
weighs the contribution of the features extracted at each convolutional layer,
and guides the CNN to estimate HDR for all properly exposed and poorly exposed
regions. On the other hand, where DoLP is high, it indicates large pixel
variations among the four polarization images, where existing rich and
reliable polarimetric cues are available for HDR recovery. Equipped with this
domain knowledge about polarization, it allows us to use this polarimetric cue
as a prior in the form of soft mask for our feature masking mechanism, to
reconstruct HDR using polarization images.

In the proposed DPHR method, the first step is to stack the four LDR
polarization images $L_{i}$, $i$ = 1,2,3,4 (512x512x12), to account for the
contributions from filters oriented at 0∘, 45∘, 90∘, and 135∘, respectively
for input to the CNN. The values in $L_{i}$ are in the range of [0,1]. Then we
use $L_{i}$ to compute the feature mask $M_{l,i}$ where the subscripts $l$ and
$i$ are the layer in the CNN and the index of each polarizer orientation
image, respectively. Specifically, we compute the feature mask $M_{1,i}$ for
input to the feature masking mechanism. The feature mask is constructed where
if the pixel value of $L_{i}$ is properly exposed and DoLP is low, then the
reconstruction is dominated by the predicted image $H_{d}$. However, if
$L_{i}$ is poorly exposed and DoLP is high, then the reconstruction is
dominated by the traditional model based method $H_{t}$. Then for $L_{i}$ and
DoLP values that lie somewhere in between, the reconstruction is shared by the
$H_{d}$ and $H_{t}$ images. The feature mask $M_{1,i}$ is defined as follows:

$M_{1,i}=\frac{\rho+K_{i}}{max(\rho+K_{i})}$ (5)

where the subscript $1$ indicates the first layer in the CNN ($l$ = 1). The
values in $K_{i}$ are in the range of [0,1], and indicate the proper
exposedness of each input pixel (of $L_{i}$) based on the pixel intensity, as
shown in Fig. 3. The value 0 in $K_{i}$ indicates the input pixel is
completely over-exposed. Therefore, $M_{1,i}$ computes the well exposedness of
each input pixel based on the combination of polarization and intensity
information. The values in $M_{1,i}$ are in the range of [0,1]. The value 0 in
$M_{1,i}$ indicates the feature is computed from low polarization and poorly
exposed pixels, and thus represents invalid content that require
reconstruction by the CNN. Then the feature mask is updated at each
convolutional layer $l$ to obtain $M_{l,i}$. As a result, the reconstruction
is achieved by using the mask $M_{l,i}$ to reduce the magnitude of the
features generated from the invalid content, by updating the feature maps
$X_{l,i}$ extracted at each convolutional layer as follows:

$X_{l+1,i}=X_{l,i}\cdot M_{l,i}$ (6)

where, with the abuse of notation, the multiplication sign $\cdot$ means the
pixel-wise multiplication where appropriate throughout the paper. In addition,
the mask at each layer is computed by applying the convolutional filter to the
mask at the previous layer. Since the masks are in the range of [0,1] and
weighs the contributions of the features, the magnitude of the filters are not
important. As a result, the filter weights are normalized before they are used
in convolution with the mask as follows:

$M_{l+1,i}=\bigg{(}\frac{|W_{l,i}|}{{\left\lVert
W_{l,i}\right\rVert}_{1}+\epsilon}\bigg{)}\ast M_{l,i}$ (7)

where ${\left\lVert\cdot\right\rVert}_{1}$ is the $l_{1}$ function, $|\cdot|$
is the absolute operator, $\ast$ is the convolution operator, and
$\epsilon=10^{-6}$ is a small constant added to avoid division by 0.

Figure 3: The defined function [8] measures how properly exposed a pixel is.
The value 1 indicates the pixel as properly exposed, while 0 indicates the
pixel as completely over-exposed. In our implementation, we set the threshold
at 0.95.

We define our loss function as follows:

$\mathcal{L}=\lambda_{1}\mathcal{L}_{r}+\lambda_{2}\mathcal{L}_{p}$ (8)

where $\mathcal{L}_{r}$ is an HDR reconstruction loss and $\mathcal{L}_{p}$ is
a perceptual loss. $\lambda_{1}=6.0$ and $\lambda_{2}=1.0$ in our
implementation.

The reconstruction loss computes the pixel-wise $l_{1}$ distance between the
ground truth image $H_{g}$ and predicted image $H_{d}$ for the invalid content
defined as follows:

$\mathcal{L}_{r}={{\left\lVert(1-M_{1})\cdot(H_{g}-H_{d})\right\rVert}_{1}}$
(9)

where $M_{1}$ is the mean of the four input feature masks, thereby accounting
for the weights from each polarization filter.

The perceptual loss has been demonstrated useful to improve visual quality
[43, 8], and its key idea is to enhance the similarity in feature space
between the ground truth and predicted images. Our perceptual loss is a
combination of the VGG loss $\mathcal{L}_{v}$ and style loss $\mathcal{L}_{s}$
as follows:

$\mathcal{L}_{p}=\lambda_{3}\mathcal{L}_{v}+\lambda_{4}\mathcal{L}_{s}$ (10)

where $\lambda_{3}=1.0$ and $\lambda_{4}=120.0$ in our implementation.

The VGG loss is defined as follows:

$\mathcal{L}_{v}=\sum_{l}{{\left\lVert\phi_{l}(H_{g})-\phi_{l}(H_{d})\right\rVert}_{1}}$
(11)

where $\phi_{l}$ is the feature map from the $l$-th layer of the pretrained
VGG-19. Then to recover more vivid textures, we add the style loss to (10),
and the style loss is defined as follows:

$\mathcal{L}_{s}=\sum_{l}{{\left\lVert
G_{l}(H_{g})-G_{l}(H_{d})\right\rVert}_{1}}$ (12)

where $G_{l}$ is the Gram matrix [52] applied on the feature map at the $l$-th
layer of the pretrained VGG-19. For the perceptual loss, we extract feature
maps from pool1, pool2 and pool3 layers of the VGG-19 network.

Figure 4: Visualization of the masks $M_{l,i=1}$ at different layers of the
network, where the dark pixels identify invalid regions with low polarization
and poorly exposed pixels that require CNN to direct its learning. We can
observe that the masks become blurrier and more uniform in the deeper layers,
as the receptive fields of the convolution filters become larger.

Fig. 4 shows the mask $M_{l,i=1}$ at different layers of the network. We can
observe from the input image that the building and the fence regions are over-
exposed. We can also observe from the input mask that these regions have a low
measure of polarization. The input mask informs the network where the invalid
features are for the CNN to direct its learning upon. Therefore, the building
and the fence regions consistently indicate a lower polarization and a poorer
exposed pixel (i.e., darker pixel) compared to other regions in the masks at
each convolutional layer. Also observed is that the masks become blurrier and
more uniform in the deeper layers, as the receptive fields of the convolution
filters become larger.

###  III-B HDR Formulation

With the CNN predicted image $H_{d}$, we formulate the final HDR image $H$
through HDR integration as follows:

$H=\alpha\cdot H_{t}+(1-\alpha)\cdot H_{d}$ (13)

where $\alpha$ is as follows:

$\alpha=\frac{\rho}{\rho+(1-M_{1})}$ (14)

In (13), $H_{t}$ is computed as follows [12]:

$H_{t}=\frac{\sum_{i=1}^{2}W\Big{(}L_{i}+L_{i+2}\Big{)}\Big{(}g\big{(}L_{i}\big{)}+g\big{(}L_{i+2}\big{)}\Big{)}}{\sum_{i=1}^{2}W\Big{(}L_{i}+L_{i+2}\Big{)}t_{0}}$
(15)

where $W$ is the Gaussian weighted function ($\sigma$ = 0.2 in our study), and
$g$ is the inverse camera response function.

The HDR formulation by (13) uses a combination of $H_{t}$ which is a
traditional model based method, and $H_{d}$ which is a deep-learning-based
method, to estimate HDR in all areas. $H_{t}$ can estimate HDR well for all
areas with high DoLP [12]. Then we let $H_{d}$ to estimate HDR for all areas
with low DoLP, using the predicted pixels from the CNN. Additionally,
normalization is performed where necessary.

##  IV Experimental Results and Discussion

In this section, we first present the experimental setups, and then evaluate
the experimental results of the proposed DPHR method on our collected outdoor
polarization images. Specifically, we evaluate the effectiveness of the
different components in our HDR formulation. To show the capability of the
proposed DPHR method, we compare the reconstructed results with state-of-the-
art methods.

Figure 5: Examples of input LDR images $L_{1}$ at $t_{0}$ = 0.769 ms from the
EdPolCommunity dataset.

###  IV-A Experiment Setups

Dataset. Since there is a lack of polarization images for training our network
for HDR reconstruction, we have built our own dataset. We collected an outdoor
dataset, under sunlight conditions, named EdPolCommunity. The dataset is
collected by the IMX250MYR polarization camera [53]. For each scene, four
high-resolution 1024x1224 colored polarization images is captured at 17
exposure times ($t_{0}$ = 0.03, 0.045, 0.068, 0.101, 0.152, 0.228, 0.342,
0.513, 0.769, 1.153, 1.73, 2.595, 3.592, 5.839, 8.758, 13.137, 19.705 ms).
Then four 512x512 patches are cropped from each image for improved
readability. In total, the EdPolCommunity dataset has 736 pairs of ground
truth HDR and LDR images available to train and test the network. We follow a
70/20/10 practice to split our dataset into train/val/test sets. The images
used for training are not used in the evaluation. Fig. 5 shows examples of
input LDR images $L_{1}$ at $t_{0}$ = 0.769 ms from the EdPolCommunity
dataset.

To gain further insights on the EdPolCommunityOutdoor dataset, a mixture model
and the expectation maximization algorithm [54] is used to fit the DoLP
distribution. The mixture model is given by:

$f=w\cdot\gamma+(1-w)\cdot U$ (16)

where $w$ is the weight of the distributions, $\gamma$ is the Gamma
distribution, and $U$ is the uniform distribution. For the
EdPolCommunityOutdoor dataset: $w$ = 0.934, $\gamma_{\alpha}$ = 6.264,
$\gamma_{\beta}$ = 0.023, $U_{start}$ = 0.0 and $U_{end}$ = 1.0. We note that
(16) is a general case that considers the presence of shot noise which is
prevalent in poorly-lit scenes. For the EdPolCommunityOutdoor dataset, $f$ in
(16) is predominantly weighted by the Gamma distribution since its weight $w$
is 0.934, and thus the data can be explained well with only the Gamma
distribution. The negative likelihood calculated for the mixture model is
-548.895. Additionally, the acquisition settings are summarized in Table II.

TABLE II: Polarized capture settings Aperture | 16 | FOV | 58∘x49∘x73∘
---|---|---|---
Focal length | 8 | Pixel format | Polarized 8
Gain | 0 | Pixel range | 0-255
Black level | 2.22 | ADC bit depth | 12
Sensor size | 2464x2056 | Frame rate | 73
Image size | 2448x2048 | # Exposures | 17

Implementation Details. The network adopts a U-Net architecture [55]. All the
encoder layers use the leaky ReLU activation ($\alpha$ = 0.2), while we use
ReLU in all the decoder layers, except for the last one, which has a linear
activation. Also the feature masking strategy is used in all the convolutional
layers. The framework is implemented using PyTorch and trained on NVIDIA
GeForce GTX 1080 Ti. We initialized the network using the Xavier approach.
Then trained using an Adam optimizer [56] with the default parameters
$\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.999, and mini-batch size of 4.

Evaluation Metrics. To evaluate the quality of the HDR images, we used the
popular HDR-VDP2 [57] metric, and adopted the means squared error (MSE) metric
as used in other works [11, 8]. We normalize the predicted and reference
ground truth HDR images [6]. Then to account for the human visual system’s
sensitivity to lighting at different luminance values, we apply perceptual
uniform (PU) encoding to calculate PU-PSNR and PU-SSIM metrics [58], to
further evaluate the accuracy of the HDR images.

(a) $L_{1}$

(b) $L_{2}$

(c) $L_{3}$

(d) $L_{4}$

(e) $M_{1,1}$

(f) $M_{1,2}$

(g) $M_{1,3}$

(h) $M_{1,4}$

(i) DPHR w/o $H_{t}$

(j) DPHR

(k) ground truth

Figure 6: Qualitative results on variants of DPHR. The DPHR model formulates
HDR using a combination of the traditional model based method $H_{t}$ and the
deep-learning-based method $H_{d}$, to estimate HDR in all areas. In contrast
to $L_{i}$ and DPHR w/o $H_{t}$, DPHR obtains results with better details that
are more closely matched to the ground truth. The input mask $M_{1,i}$
identifies dark pixels as invalid regions that require CNN to direct its
learning. The Photomatix Enhanced tone-mapping operator is used.

###  IV-B Evaluation of the HDR Formulation

In this experiment, we validate the model design choice for the HDR
formulation, which is performed by comparing the following variants of the HDR
formulation:

  1. (a)

DPHR (i.e., $H=\alpha\cdot H_{t}+(1-\alpha)\cdot H_{d}$). The full HDR
formulation, which is the combination of traditional model based and deep-
learning-based HDR reconstruction results.

  2. (b)

DPHR w/o $H_{t}$ (i.e., $H=H_{d}$). We remove the traditional model based
result in this variant, and obtain the final HDR image $H$ using our deep-
learning-based result to show the effect of polarimetric information in
comparison with the regular images in traditional deep-learning methods.

The HDR formulation with (a) is effective for estimating HDR values. As shown
in Fig. 6, compared with the input $L_{i}$, (b) can restore more details as
our CNN uses the polarimetric information as a prior. In particular, the CNN
itself can outperform the traditional deep-learning approach by Santos et al.
[8], where the (b) variant provides better quantitative results than [8], as
shown in Table V (seventh row). However, (b) suffers color distortions, and is
an overall darker image with limited recovery of details in the poorly exposed
regions. On the other hand, (a) can recover better details in both under- and
over-exposed regions, and alleviate color distortions and visible artifacts.
An explanation for the removal of color distortions in (a) is that DoLP is a
mask shared among the colored channels, and thus can enforce consistency
between images which helps to remove the colorization artifacts for a better
HDR reconstruction. The quantitative results are shown in Table III, and is
aligned with the observation that (a) achieves a better performance. The key
difference between these two variants is the effect of DoLP. In the (a)
variant, DoLP is used as a strong accurate prior to recover HDR, where for
regions with high DoLP the traditional model based method recovers HDR details
well compared to the deep-learning-based method. Therefore, we achieve HDR
formulation using the (a) variant.

In addition, the results from both (a) and (b) variants can outperform a
recent deep-learning-based HDR reconstruction approach developed using the
polarization camera [13]. Unlike [13] that uses only the polarization images
to achieve HDR reconstruction, and thus neglects the polarimetric cue
available from the polarization camera, the (b) variant integrates the
polarimetric cue as a prior in the form of soft mask for our feature masking
mechanism, to reconstruct better HDR images than [13] as shown in Table V
(fifth row). Then the (a) variant further enhances the reconstruction results
of [13] in Table V by integrating a prior that the traditional model based
method $H_{t}$ is able to recover well for regions with high DoLP to generate
the final HDR image.

TABLE III: Quantitative results on variants of DPHR (the higher the better, except for MSE) | PU-PSNR | PU-SSIM | HDR-VDP2 | MSE
---|---|---|---|---
DPHR w/o $H_{t}$ | 20.34$\pm$1.83 | 0.83$\pm$0.044 | 51.13$\pm$2.94 | 0.0357$\pm$0.0029
DPHR | 30.59$\pm$3.19 | 0.94$\pm$0.068 | 56.15$\pm$5.28 | 0.0081$\pm$0.0006

(a) $\mathcal{L}_{r}$

(b) $\mathcal{L}_{p}$

(c) $\mathcal{L}_{r}+\mathcal{L}_{p}$

(d) ground truth

Figure 7: Qualitative results on different loss functions, where our choice of $\mathcal{L}_{r}+\mathcal{L}_{p}$ loss presents color consistency and preserves details that are better matched with the ground truth.The Photomatix Enhanced tone-mapping operator is used. TABLE IV: Quantitative results on variants of loss function (the higher the better, except for MSE) | PU-PSNR | PU-SSIM | HDR-VDP2 | MSE
---|---|---|---|---
$\mathcal{L}_{r}$ | 29.89$\pm$2.89 | 0.90$\pm$0.034 | 54.15$\pm$3.96 | 0.0080$\pm$0.0081
$\mathcal{L}_{p}$ | 29.94$\pm$2.87 | 0.93$\pm$0.023 | 55.35$\pm$4.80 | 0.0088$\pm$0.0061
$\mathcal{L}_{r}$ \+ $\mathcal{L}_{p}$ | 30.59$\pm$3.19 | 0.94$\pm$0.068 | 56.15$\pm$5.28 | 0.0081$\pm$0.0006

(a) HDR-GAN

(b) DeepHDR

(c) ENet

(d) DrTMO

(e) DRCP

(f) PHDR

(g) DPSHDR

(h) HDRCNN

(i) HDRCNN-Mask

(j) DPHR

(k) ground truth

Figure 8: Qualitative comparative results with state-of-the-art methods. The
proposed DPHR method presents color consistency, free of visible artifacts,
and overall able to recover richer textures in all poorly exposed regions.
This is because DPHR is designed to handle polarization images, as it
integrated the polarimetric cues into the framework. The Photomatix Enhanced
TMO is used.

(a) HDR-GAN

(b) DeepHDR

(c) ENet

(d) DrTMO

(e) DRCP

(f) PHDR

(g) DPSHDR

(h) HDRCNN

(i) HDRCNN-Mask

(j) DPHR

(k) Ground truth

Figure 9: Additional qualitative comparative results with state-of-the-art methods. The proposed DPHR method presents color consistency, free of visible artifacts, and overall able to recover richer textures in all poorly exposed regions. This is because DPHR is designed to handle polarization images, as it integrated the polarimetric cues into the framework. The Photomatix Enhanced TMO is used. TABLE V: Quantitative comparative results with state-of-the-art methods (the higher the better except for MSE) Input sources | Methods | PU-PSNR | PU-SSIM | HDR-VDP2 | MSE
---|---|---|---|---|---
Multiple-shot | HDR-GAN [24] | 15.62 $\pm$ 1.92 | 0.48 $\pm$ 0.018 | 50.54 $\pm$ 4.34 | 0.0599 $\pm$ 0.0037
DeepHDR [20] | 18.69 $\pm$ 2.04 | 0.60 $\pm$ 0.037 | 49.55 $\pm$ 5.08 | 0.0293 $\pm$ 0.0043
Single-shot | ENet [6] | 16.86 $\pm$ 1.45 | 0.88 $\pm$ 0.049 | 49.45 $\pm$ 3.61 | 0.0429 $\pm$ 0.0029
DrTMO [9] | 21.51 $\pm$ 1.74 | 0.84 $\pm$ 0.043 | 53.28 $\pm$ 3.28 | 0.0591 $\pm$ 0.0027
DRCP [11] | 16.48 $\pm$ 1.41 | 0.88 $\pm$ 0.063 | 49.25 $\pm$ 4.64 | 0.0784 $\pm$ 0.0062
HDRCNN [7] | 21.82 $\pm$ 1.90 | 0.90 $\pm$ 0.066 | 53.66 $\pm$ 4.77 | 0.0391 $\pm$ 0.0037
HDRCNN-Mask [8] | 17.87 $\pm$ 1.39 | 0.83 $\pm$ 0.059 | 47.72 $\pm$ 3.22 | 0.0470 $\pm$ 0.0025
PHDR [12] | 16.43 $\pm$ 1.91 | 0.73 $\pm$ 0.067 | 45.92 $\pm$ 4.45 | 0.0639 $\pm$ 0.0048
DPSHDR [13] | 19.22 $\pm$ 2.13 | 0.83 $\pm$ 0.041 | 50.22 $\pm$ 4.36 | 0.0787 $\pm$ 0.0054
DPHR | 30.59 $\pm$ 3.19 | 0.94 $\pm$ 0.068 | 56.15 $\pm$ 5.28 | 0.0081 $\pm$ 0.0006

###  IV-C Evaluation of the Training Loss Function

In this experiment, we analyze the effect of each component of our training
loss function. The results in Table IV and Fig. 7 shows that
$\mathcal{L}_{r}+\mathcal{L}_{p}$ loss is more powerful for preserving details
as discussed in [8]. We thus train our model using
$\mathcal{L}_{r}+\mathcal{L}_{p}$ loss.

###  IV-D Comparison with State-of-the-art Methods

We use the test dataset to evaluate the proposed DPHR method and compare it
with state-of-the-art methods, including seven deep-learning-based methods
developed for the conventional cameras [24, 20, 6, 9, 11, 7, 8], one
traditional model based method developed for the polarization camera [12], and
one deep-learning-based method developed also for the polarization camera
[13]. For all methods, we used the codes and pretrain weights provided by the
authors. Fig. 8, 9 illustrates the qualitative results of the comparative
study. We can observe that images reconstructed with the DPHR method can
recover more accurate details in all properly exposed and poorly exposed
regions, and alleviate color distortions and visible artifacts. The HDR-GAN
[24] and DeepHDR [20] results are relatively bright, and has difficulty
restoring details in the over-exposed areas. The ENet [6] result tends to be
overly bright and smooth, as it over-enhances the extracted illumination
features. Additionally, it’s unable to improve HDR estimation for properly
exposed regions. The DrTMO [9] method suffers from blocking artifacts, and has
difficulty recovering details in all properly exposed and poorly exposed
regions. The DRCP [11] method can restore details in the poorly exposed
regions, but exhibits color distortions. Furthermore, the result is generally
darker, thus contents lost in the under-exposed regions can not be restored.
The PHDR [12] result is overall bright where pixels remain lost in the poorly
exposed regions. The DPSHDR [13] result reveals more details, but remains
unable to reconstruct sufficient details in both under- and over-exposed
regions.

The HDRCNN [7] method can recover some contents in the over-exposed sky
region, but the result is overall dim and presents visual artifacts. The
HDRCNN-Mask [8] method follows the approach by HDRCNN to estimate details in
over-exposed regions, and then reconstruct the final HDR image by combining
with the input. However, unlike HDRCNN, HDRCNN-Mask proposes a feature masking
mechanism to propagate valid features for properly exposed pixels. The output
of HDRCNN-Mask is overly smooth with color artifacts, and reconstructs limited
details of the cloud in the over-exposed sky area. Also, the content remains
lost in the under-exposed areas. On the other hand, the proposed DPHR method
presents color consistency, free of visible artifacts, and overall able to
recover richer textures in both under- and over-exposed regions. This is
because the proposed DPHR method is designed to handle polarization images, as
we have integrated the polarization information (i.e., DoLP) into the network
during training and inference to formulate the final HDR image. In addition to
visual evaluation, the quantitative results are summarized in Table V. Since
the proposed DPHR method uses DoLP as a strong accurate prior to help HDR
recovery for polarization images, it performs favorably compared to state-of-
the-art methods under various evaluation metrics, including both HDR (HDR-
VDP2, MSE) and HDR tone-mapped LDR evaluation metrics (PSNR, SSIM, FSIM).

(a) Low-light

(b) High dynamic-range

Figure 10: Limitation analysis. In the top row, the proposed DPHR method fails
to reconstruct the indoor image captured from an poorly-lit setting. In the
bottom row, the proposed DPHR method also fails to reconstruct the high
dynamic-range image captured from an outdoor scene, where the details of the
clouds in the sky are not recovered. The Photomatix Enhanced TMO is used.

##  V Limitations and Future Work

Despite that our method can reconstruct better details compared to existing
works, its expressive power is still limited. Fig. 10 shows two examples of
difficult scenes where our approach fails to reconstruct. In Fig. 10(a), our
approach is unable to reconstruct the details and incorrectly reconstructs the
color in the underexposed image captured from an indoor poorly-lit setting.
This is mainly because our method is unable to handle noisy pixels. In
particular, shot noise is prevalent when insufficient light reaches the sensor
due to an exposure that is too brief for the lighting condition, and it
results in random pixel variations which we observe as noisy pixels. This
issue can be alleviated by exploring techniques to reduce the pixel noise in
the pre-processing step. In Fig. 10(b), our approach fails to recover the
details of the clouds in the overexposed sky regions from an outdoor high
dynamic-range scene. This is mainly due to images with broad regions of
overexposed sunshine are scarce in our training dataset. This issue can be
mitigated by augmenting our training dataset to incorporate such extreme cases
to improve performance.

To follow up on this study, we will enrich our training dataset to improve the
performance, we can collect a greater variety of scenes to include other
outdoor scenes, such as buildings and people. Additionally, synthesizing HDR
images using 3D computer graphics represents a promising avenue we would like
to explore. Another interesting avenue of future work is to evaluate the
benefits of the polarization camera with other image capturing devices [35,
36, 37, 38, 39, 40, 41] in reconstructing HDR images. We also plan to expand
our method to other camera response functions, investigate demosaicing
artifacts, and explore the use of the AoLP image, as an additional
polarimetric cue, to further the image dynamic range and HDR detail recovery.

##  VI Conclusion

In this paper, we have proposed a novel learning based HDR reconstruction
method using a polarization camera. The proposed DPHR method explores
polarimetric information obtained by a polarization camera to enhance the
reconstruction in all properly exposed and poorly exposed regions, which is a
long-standing challenge in computer vision and robotics applications. In the
proposed DPHR method, we first compute the input and mask image from the
polarization images captured by the polarization camera, and use those cues in
the feature masking mechanism to better propagate the valid features through
the CNN. Then we combined the deep-learning predicted HDR image with a
traditional model based HDR image to formulate the final HDR result. Our
experimental results on the polarization image dataset show that the DPHR
method demonstrated better qualitative and quantitative performances over
state-of-the-art methods.

## References

  * [1] M. Shakeri and H. Zhang, “Illumination invariant representation of natural images for visual place recognition,” pp. 466–472, 2016.
  * [2] ——, “Moving object detection in time-lapse or motion trigger image sequences using low-rank and invariant sparse decomposition,” Oct 2017.
  * [3] ——, “Moving object detection under discontinuous change in illumination using tensor low-rank and invariant sparse decomposition,” June 2019.
  * [4] Q. Yan, D. Gong, Q. Shi, A. van den Hengel, C. Shen, I. Reid, and Y. Zhang, “Attention-guided network for ghost-free high dynamic range imaging,” in _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1751–1760.
  * [5] S. Lee, H. Chung, and N. I. Cho, “Exposure-structure blending network for high dynamic range imaging of dynamic scenes,” _IEEE Access_ , vol. 8, pp. 117 428–117 438, 2020.
  * [6] D. Marnerides, T. Bashford-Rogers, J. Hatchett, and K. Debattista, “Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content,” _Computer Graphics Forum_ , vol. 37, pp. 37–49, 2018\.
  * [7] G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger, “Hdr image reconstruction from a single exposure using deep cnns,” _ACM Transactions on Graphics_ , vol. 36, no. 6, p. 1–15, Nov 2017. [Online]. Available: http://dx.doi.org/10.1145/3130800.3130816
  * [8] M. S. Santos, T. I. Ren, and N. K. Kalantari, “Single image hdr reconstruction using a cnn with masked features and perceptual loss,” _ACM Transactions on Graphics_ , vol. 39, no. 4, Jul. 2020. [Online]. Available: https://doi.org/10.1145/3386569.3392403
  * [9] E. Yuki, K. Yoshihiro, and M. Jun, “Deep reverse tone mapping,” _ACM Transactions on Graphics_ , vol. 36, no. 6, Nov. 2017.
  * [10] S. Lee, G. H. An, and S. Kang, “Deep chain hdri: Reconstructing a high dynamic range image from a single low dynamic range image,” _IEEE Access_ , vol. 6, pp. 49 913–49 924, 2018.
  * [11] Y. L. Liu, W. S. Lai, Y. S. Chen, Y. L. Kao, M. H. Yang, Y. Y. Chuang, and J. B. Huang, “Single-image hdr reconstruction by learning to reverse the camera pipeline,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 1648–1657.
  * [12] X. Wu, H. Zhang, X. Hu, M. Shakeri, C. Fan, and J. Ting, “Hdr reconstruction based on the polarization camera,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5113–5119, 2020.
  * [13] J. Ting, X. Wu, K. Hu, and H. Zhang, “Deep snapshot hdr reconstruction based on the polarization camera,” 2021.
  * [14] P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Goldman, and E. Shechtman, “Robust Patch-Based HDR Reconstruction of Dynamic Scenes,” _ACM Transactions on Graphics_ , vol. 31, no. 6, pp. 203:1–203:11, 2012.
  * [15] J. Hu, O. Gallo, K. Pulli, and X. Sun, “Hdr deghosting: How to deal with saturation?” in _2013 IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 1163–1170.
  * [16] T. Oh, J. Lee, Y. Tai, and I. S. Kweon, “Robust high dynamic range imaging by rank minimization,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 6, pp. 1219–1232, 2015.
  * [17] C. Lee, Y. Li, and V. Monga, “Ghost-free high dynamic range imaging via rank minimization,” _IEEE Signal Processing Letters_ , vol. 21, no. 9, pp. 1045–1049, 2014.
  * [18] Z. Li, J. Zheng, Z. Zhu, and S. Wu, “Selectively detail-enhanced fusion of differently exposed images with moving objects,” _IEEE Transactions on Image Processing_ , vol. 23, no. 10, pp. 4372–4382, 2014.
  * [19] N. K. Kalantari and R. Ramamoorthi, “Deep high dynamic range imaging of dynamic scenes,” _ACM Transactions on Graphics_ , vol. 36, no. 4, 2017.
  * [20] S. Wu, J. Xu, Y.-W. Tai, and C.-K. Tang, “Deep high dynamic range imaging with large foreground motions,” 2018.
  * [21] Z. Liu, W. Lin, X. Li, Q. Rao, T. Jiang, M. Han, H. Fan, J. Sun, and S. Liu, “Adnet: Attention-guided deformable convolutional network for high dynamic range imaging,” in _CVPRW_ , 2021, pp. 463–470.
  * [22] Z. Pu, P. Guo, M. S. Asif, and Z. Ma, “Robust high dynamic range (hdr) imaging with complex motion and parallax,” in _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , November 2020.
  * [23] K. Ma, Z. Duanmu, H. Zhu, Y. Fang, and Z. Wang, “Deep guided learning for fast multi-exposure image fusion,” vol. 29, 2020, pp. 2808–2819.
  * [24] Y. Niu, J. Wu, W. Liu, W. Guo, and R. W. H. Lau, “Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions,” 2020.
  * [25] N. Khademi Kalantari and R. Ramamoorthi, “Deep hdr video from sequences with alternating exposures,” _Computer Graphics Forum_ , vol. 38, pp. 193–205, 05 2019.
  * [26] A. O. Akyüz, R. Fleming, B. Riecke, E. Reinhard, and H. Bülthoff, “Do hdr displays support ldr content? a psychophysical evaluation,” _ACM Transactions on Graphics_ , vol. 26, p. 38, 2007.
  * [27] A. Rempel, M. Trentacoste, H. Seetzen, H. Young, W. Heidrich, L. Whitehead, and G. Ward, “Ldr2hdr: On-the-fly reverse tone mapping of legacy video and photographs,” _ACM Transactions on Graphics_ , vol. 26, p. 39, 08 2007.
  * [28] B. Masia and D. Gutiérrez, “Dynamic range expansion based on image statistics,” _Multimedia Tools and Applications_ , vol. 76, 01 2017.
  * [29] R. P. Kovaleski and M. M. Oliveira, “High-quality reverse tone mapping for a wide range of exposures,” in _2014 27th SIBGRAPI Conference on Graphics, Patterns and Images_ , 2014, pp. 49–56.
  * [30] B. Masia and D. Gutiérrez, “Content-aware reverse tone mapping,” in _Proceedings of the 2016 International Conference on Artificial Intelligence: Technologies and Applications_ , 2016, pp. 235–238. [Online]. Available: https://doi.org/10.2991/icaita-16.2016.58
  * [31] F. Banterle, P. Ledda, K. Debattista, and A. Chalmers, “Inverse tone mapping,” 01 2006, pp. 349–356.
  * [32] H. Kasliwal and S. Modi, “A novel method for tone mapping of hdr images,” 2015\.
  * [33] X. Chen, Y. Liu, Z. Zhang, Y. Qiao, and C. Dong, “Hdrunet: Single image hdr reconstruction with denoising and dequantization,” 2021.
  * [34] Y. Zhang and T. O. Aydin, “Deep hdr estimation with generative detail reconstruction,” _Computer Graphics Forum_ , vol. 40, 2021.
  * [35] J. Han, C. Zhou, P. Duan, Y. Tang, C. Xu, C. Xu, T. Huang, and B. Shi, “Neuromorphic camera guided high dynamic range imaging,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 1727–1736.
  * [36] J. A. Leñero Bardallo, R. Carmona-Galán, and A. Rodríguez-Vázquez, “Hdr image sensor with linear response and asynchronous detection of saturation: Demo,” in _Proceedings of the 10th International Conference on Distributed Smart Camera_ , ser. ICDSC ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 204–205. [Online]. Available: https://doi.org/10.1145/2967413.2974030
  * [37] H. Zhao, B. Shi, C. Fernandez-Cull, S.-K. Yeung, and R. Raskar, “Unbounded high dynamic range photography using a modulo camera,” in _2015 IEEE International Conference on Computational Photography (ICCP)_ , 2015, pp. 1–10.
  * [38] C. Fernandez-Cull, H. Zhao, B. Shi, B. Tyrrell, J. Lin, and R. Raskar, “Snapshot on-chip hdr roic architectures,” 01 2015, p. CM3E.3.
  * [39] S. K. Nayar and T. Mitsunaga, “High dynamic range imaging: spatially varying pixel exposures,” in _Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. 1, 2000, pp. 472–479 vol.1.
  * [40] S. Takeru, M. T., M. Yusuke, and O. Masatoshi, “Deep snapshot hdr imaging using multi-exposure color filter array,” in _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , 2020.
  * [41] M. Rouf, R. Mantiuk, W. Heidrich, M. Trentacoste, and C. Lau, “Glare encoding of high dynamic range images,” in _CVPR 2011_ , 2011, pp. 289–296.
  * [42] C. A. Metzler, H. Ikoma, Y. Peng, and G. Wetzstein, “Deep optics for single-shot high-dynamic-range imaging,” 2019.
  * [43] Q. Sun, E. Tseng, Q. Fu, W. Heidrich, and F. Heide, “Learning rank-1 diffractive optics for single-shot high dynamic range imaging,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020\.
  * [44] J. N. P. Martel, L. K. Müller, S. J. Carey, P. Dudek, and G. Wetzstein, “Neural sensors: Learning pixel exposures for hdr imaging and video compressive sensing with programmable sensors,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 7, pp. 1642–1653, 2020\.
  * [45] M. Alghamdi, Q. Fu, A. Thabet, and W. Heidrich, “Transfer deep learning for reconfigurable snapshot hdr imaging using coded masks,” _Computer Graphics Forum_ , vol. 40, 03 2021.
  * [46] ——, “Reconfigurable Snapshot HDR Imaging Using Coded Masks and Inception Network,” in _Vision, Modeling and Visualization_ , H.-J. Schulz, M. Teschner, and M. Wimmer, Eds. The Eurographics Association, 2019.
  * [47] K. Fotiadou, G. Tsagkatakis, and P. Tsakalides, “Snapshot high dynamic range imaging via sparse representations and feature learning,” _IEEE Transactions on Multimedia_ , vol. 22, no. 3, pp. 688–703, 2020.
  * [48] A. Serrano, F. Heide, D. Gutierrez, G. Wetzstein, and B. Masia, “Convolutional sparse coding for high dynamic range imaging,” _Computer Graphics Forum_ , vol. 35, no. 2, p. 153–163, May 2016. [Online]. Available: http://dx.doi.org/10.1111/cgf.12819
  * [49] D. H. Goldstein, “Polarized light,” _CRC press_ , 2017.
  * [50] C. P. Huynh, A. Robles-Kelly, and E. Hancock, “Shape and refractive index recovery from single-view polarisation images,” in _in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2010, pp. 1229–1236.
  * [51] P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in _Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques_ , 1997, p. 369–378.
  * [52] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [53] “Imx250 cmos sensor,” https://www.sony-semicon.co.jp/e/products/IS/industry/technology/polarization.html, 2021\.
  * [54] T. K. Moon, “The expectation-maximization algorithm,” _IEEE Signal Processing Magazine_ , vol. 13, no. 6, pp. 47–60, 1996.
  * [55] R. Olaf, F. Philipp, and B. Thomas, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , ser. LNCS, vol. 9351. Springer, 2015, pp. 234–241.
  * [56] P. K. Diederik and B. Jimmy, “Adam: A method for stochastic optimization,” _International Conference on Learning Representations (ICLR)_ , 2015.
  * [57] R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich, “Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions,” _ACM Transactions on Graphics_ , vol. 30, no. 4, Jul. 2011.
  * [58] R. Mantiuk and M. Azimi, “Pu21: A novel perceptually uniform encoding for adapting existing quality metrics for hdr,” 06 2021, pp. 1–5.


# Destructive breakdown studies of irradiated LGADs at beam tests for the
ATLAS HGTD

L. A. Beresford  D. E. Boumediene,111Corresponding author. L. Castillo García,
L. D. Corpe  M. J. Da Cunha Sargedas de Sousa  H. El Jarrari  A.
Eshkevarvakili  C. Grieco222now at UCSB S. Grinstein  S. Guindon  A. Howard,
G. Kramberger  O. Kurdysh  R. Mazini  M. Missio  M. Morenas  O. Perrin  V.
Raskina  G. Saito  S. Trincaz-Duvoid

###### Abstract

In the past years, it has been observed at several beam test campaigns that
irradiated LGAD sensors break with a typical star shaped burn mark when
operated at voltages much lower than those at which they were safely operated
during laboratory tests. The study presented in this paper was designed to
determine the safe operating voltage that these sensors can withstand. Many
irradiated sensors from various producers were tested in two test beam
facilities, DESY (Hamburg) and CERN-SPS (Geneva), as part of ATLAS High
Granularity Timing Detector (HGTD) beam tests. The samples were placed in the
beam and kept under bias over a long period of time in order to reach a high
number of particles crossing each sensor. Both beam tests lead to a similar
conclusion, that these destructive events begin to occur when the average
electric field in the sensor becomes larger than
$12\text{\,}\mathrm{V}\text{/}\mathrm{\SIUnitSymbolMicro m}$.

###  1 Introduction

During the high-luminosity phase of the LHC (HL-LHC), the primary interactions
created in proton-proton collisions are accompanied by a large number of zero
and minimum bias interactions (pileup). In order to mitigate the adverse
effects of pileup, the ATLAS experiment will install the High Granularity
Timing Detector (HGTD) [1]. In particular, the high-precision track timing
information from HGTD will allow for the proton-proton collisions to be
distinguished in time. HGTD will use Low Gain Avalanche Detectors (LGADs) [2,
3]. LGADs provide low gain, typically to the order of several tens, by using a
n++-p+-p-p++ structure to create electric fields high enough for impact
ionization to occur at the n++-p+ junction. The doping concentration and
profile shape of the gain layer impacts the gain factor of the detector.

The harsh radiation environment at the HL-LHC, particularly in the forward
region (corresponding to a pseudorapidity333 The ATLAS experiment uses a
right-handed coordinate system with its origin at the nominal interaction
point (IP) in the center of the detector, and the $z$-axis along the beam
line. The $x$-axis points from the IP to the center of the LHC ring, and the
$y$-axis points upwards. Cylindrical coordinates $(r,\phi)$ are used in the
transverse plane, $\phi$ being the azimuthal angle around the $z$-axis.
Observables labelled “transverse” are projected onto the $x$ \- $y$ plane. The
pseudorapidity is defined in terms of the polar angle $\theta$ as
$\eta=-\ln\tan\theta/2$., $\eta$, range of 2.8<$\eta$<4.0) where HGTD will
operate, will lead to the deterioration of LGAD performances. The main concern
is the so-called initial acceptor removal [4], which reduces the gain and
requires an increase of bias voltage to compensate for the loss of acceptors
in the gain layer. By the end of their lifetime at HGTD the most exposed
sensors will have received a $1\text{\,}\mathrm{MeV}\text{/}$ neutron
equivalent fluence of around $\Phi_{eq}=2.5\times 10^{15}$ neq/cm2. This would
require operation of sensors at voltages exceeding
$700\text{\,}\mathrm{V}\text{/}$ at
$-30\text{\,}\mathrm{\SIUnitSymbolCelsius}\text{/}$ for a sensor active area
thickness of $50\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$. It has been
shown in the laboratory using strontium-90 electrons that in such conditions
enough charge is collected and the targeted time resolution of 35 (70) ps at
the start (end) of their lifetime. The minimal bias voltage under which each
sensor reaches the 4 fC charge threshold was first measured in the laboratory
using a strontium-90 source.

The HGTD beam test campaigns at CERN-SPS [5]
($120\text{\,}\mathrm{GeV}\text{/}$ pions) and DESY [6]
($3\text{\,}\mathrm{GeV}\text{/}$ electrons) aimed to reproduce the
strontium-90 measurements. Many of the sensors underwent destructive breakdown
at voltages that were $\sim$100\text{\,}\mathrm{V}\text{/}$$ lower than those
at which the sensors were successfully operated in laboratory tests. A typical
star shape burn mark, see figure 1, appeared in the location of the particle
hitting the sensors.

|
---|---
(a) | (b)
Figure 1: (a) Typical star shaped burn mark observed in beam test sensor
mortalities due to SEB. Sensor pictured here is from run HPK-P2 wafer 31
irradiated to 2.5$\times 10^{15}$ neq/cm2, whose mortality occurred at the
DESY beam test. (b) Microscopic photograph of a typical burn mark, observed in
ATLAS proton beam tests at Fermilab in 2018 in a CNM LDA35 sensor (courtesy of
CNM) [7].

An intensive investigation done within the ATLAS, CMS and RD50 Collaborations
[8, 9] led to the conclusion that a single beam particle hitting the detector
is responsible for the sensor destruction. Both ATLAS and CMS recorded the
signal from such events and associated the location of the crater with the hit
position reconstructed by the beam telescope. An example is shown in figure 2.
Studies with high-power $50\text{\,}\mathrm{fs}\text{/}$ laser pulses were
performed [10] and also confirmed the destructive events at high bias
voltages.

Figure 2: The right plot shows a typical single event burnout mark from a 2019
DESY beam test with $5\text{\,}\mathrm{GeV}\text{/}$ electrons. The
reconstructed track in the destructive event pointed to the location of the
burn mark (middle and right plot). The distribution of all the reconstructed
tracks across the detector before destruction is shown in the left plot.

The mechanism of the destruction is called single event burnout (SEB) [11] and
will be schematically described in the next section. However, it was not clear
what the parameters determining its occurrence were. In order to answer that
question a large set of sensors from various producers with different
thicknesses of the active material, D, irradiation levels, annealing stages
and gain layer designs, were tested in two different beam test campaigns at
DESY and SPS, in 2021. This paper describes the result of these studies.

###  2 Single event burnout

The most probable cause of the sensor mortalities observed is the mechanism
known as SEB. Laboratory tests use a beta beam from a strontium-90 source
which have significantly less energy than the test beams. The maximum energy
of a strontium-90 electron is $2.3\text{\,}\mathrm{MeV}\text{/}$ which sets
the maximum deposited charge in the LGAD sensor. On the other hand, the
deposited charge from a high-momentum particle beam in the active zone of the
LGAD detector can be much larger. According to GEANT4 [12] simulations up to
$100\text{\,}\mathrm{MeV}\text{/}$ can be deposited by a single charged hadron
[9, 13]. This leads to the generation of a large density of carriers. The
screening effect created by such a large carrier density then prevents
carriers from being swept away, leading to a change in the local resistivity
such that the sensor becomes conductive [11]. The field collapses in the
region of high free carrier density, leading to an increased voltage drop in
the region where density is lower. The increase of the field there leads to
avalanche breakdown (field exceeds the critical field). The charge stored on
the sensor electrodes as well as on the high voltage (HV) filtering capacitor
(typically $10\text{\,}\mathrm{nF}\text{/}$) is discharged through the sensor.
The energy available in such a discharge is enough to melt the silicon,
creating a crater and damaging the sensor. The breakdown is eventually
quenched, but the sensor is permanently damaged. This mechanism is illustrated
in figure 3.

Figure 3: Scheme of the SEB mechanism in an LGAD [11]. (1) A large amount of
energy is deposited in the sensor. (2) Large carrier density leads to collapse
of the field. (3) The HV is brought closer to the pad leading to very high
field strength. (4) Avalanche breakdown leads to destruction of the sensor.

###  3 Test beam set-up

The SEB events were initially observed in sensors being operated in test beam
lines where occasionally the signals from the fatal events were recorded. In
order to maximise the number of sensors that can be tested at a given beam
test campaign, a printed circuit board (PCB) was designed that could host two
sensors of different sizes. Figure 4 shows the circuit diagram of the boards,
and the five connections to the boards. The current was measured as a voltage
drop on the bias resistor. The resistor is either
$30\text{\,}\mathrm{k\SIUnitSymbolOhm}\text{/}$ or
$100\text{\,}\mathrm{k\SIUnitSymbolOhm}\text{/}$ based on the size of the
sensors; smaller resistance for larger-sized sensors. A
$2\text{\,}\mathrm{nF}\text{/}$ capacitor was used to simulate the full sensor
capacitance, while the circuitry in figure 4a prevented the voltage drop from
exceeding the maximum tolerable at the analog-to-digital converter (ADC)
input. Up to eight PCBs, i.e. 16 sensors, were aligned to one another with the
use of mechanical rails, as shown in figure 5.

|
---|---
(a) | (b)
Figure 4: (a) Circuit diagram of a single channel of the PCB. (b) Photograph
of the PCB, showing the five connections for: bias applied to the left sensor
(HV1), the voltage output of the left sensor (V1), ground (GND), the voltage
output of the right sensor (V2), and the bias applied to the right sensor
(HV2). Figure 5: The eight PCBs are placed one in front of the other on a
frame, and placed directly in the beam. A scintillator and PMT placed in front
of the train of boards record the beam rate. Another scintillator and PMT are
placed behind the train. The coincidence of both was used to align the sensors
in the beam.

A CAEN N472 HV power supply, which has four HV channels, was used to apply the
bias to the sensors. To allow for more than one sensor to be biased by one HV
channel the HV was first taken to an external box where it is split to allow
up to seven connections per HV channel. Flat cables carried the HV to each of
the PCBs. A major benefit of this design is that when a sensor broke it simply
needed to be disconnected from this box and all the other sensors connected to
the same HV channel could continue to be operated.

A NI-USB 6001 DAQ system was used to measure the voltage drop on the sensors,
as well as the HV channels, temperature and humidity. This setup was used at
both beam tests. At DESY dry ice was used to cool down the sensors. Since the
temperature increases as the dry ice evaporates, detailed temperature
measurement was kept throughout testing, which required a dedicated HV
channel. 3 HV channels were used for sensor testing. The HV was applied only
while the temperature was between
$-40\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{C}$ to
$-25\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{C}$. At the SPS the cooling
box used a chiller to keep the temperature stable at
$-30\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{C}$, which is the planned
operating temperature of the HGTD in the ATLAS detector, and so the choice was
made to measure 4 HV channels and just monitor the temperature by using an
external sensor placed inside the cooling box.

In both beam tests, a 1 cm2 scintillator read out by a photo multiplier tube
(PMT) placed in front of the train of sensors recorded the beam rate, which
allowed for the number of particles crossing through each sensor to be
calculated. This scintillator was used together with a second scintillator
placed behind the train of sensors to align the sensors in the beam by moving
the platform to observe where the coincidence rate was maximum.

###  4 Sensors

####  4.1 LGAD samples

The ATLAS sensors will consist of 15$\times$15 LGAD arrays of 1.3$\times$1.3
mm2 pads. For the tests performed here other pad structures are used such as
single-pads and 2$\times$2 and 5$\times$5 arrays. A complete list of the
sensors sent to the beam tests and their properties is given in appendix A. A
set of sensors from different runs and wafers, produced by HPK444Hamamatsu
Photonics, Japan., IHEP-IME555Institute of High Energy Physics and Institute
of Microelectronics, Chinese Academy of Sciences, China., CNM666Centro
Nacional de Microelectrónica, Spain., FBK777Fondazione Bruno Kessler, Italy.,
NDL888Novel Device Laboratory, China. and USTC-IME999University of Science and
Technology of China and Institute of Microelectronics, China., were chosen to
test a wide variety of LGAD properties. The sensors also vary in size, active
material thickness and in the radiation level they were subjected to, all
given in the tables 1 and 2. Irradiation fluence is given as the
$1\text{\,}\mathrm{MeV}\text{/}$ neutron equivalent [14]. All of the sensors
were irradiated with reactor neutrons at the Jozef Stefan Institute’s TRIGA II
research reactor [15]. Most of the sensors were then annealed for
$80\text{\,}\mathrm{min}\text{/}$ at
$60\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{C}$, unless otherwise stated.
Of the sensors tested, USTC-IME-1.1 wafer 11 sensors, all FBK sensors and all
IHEP-IMEv2 sensors are carbon-enriched [16] while the rest are not.

A few sensors were studied for more specific reasons. The HPK-P1 sensors of
type-3.1 and 3.2 were annealed for an extremely long time, and for this reason
were chosen in order to test if annealing time would have any impact on the
occurrence of mortality. The performance of these type-3.1 and 3.2 sensors
were tested at HGTD beam tests in 2018 and 2019 [17]. HPK-P1 sensors of
type-1.1 and 1.2 were chosen to give a clearer picture if mortality was
impacted by the sensor active material thickness, as they are thinner than the
more recently produced LGADs at $35\text{\,}\mathrm{\SIUnitSymbolMicro
m}\text{/}$ versus the more typical
$\sim$$50\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$. Lastly, irradiated
Positive Intrinsic Negative (PIN) diodes were chosen in order to determine if
gain had an impact on the mortality.

####  4.2 Testing process

In total, 32 samples were sent to DESY and were measured in two batches of 16
sensors across two weeks. Those that survived the DESY test beam were sent to
the SPS test beam. In total, 42 sensors were sent to SPS and measured in four
batches altogether, again across two weeks. In each case, the sensors were
grouped up in such a way that sensors requiring similar biases to achieve the
$4\text{\,}\mathrm{fC}\text{/}$ requirement for the HGTD [1] were connected on
the same HV channel. The bias of each HV channel was then increased until it
was high enough to reach $4\text{\,}\mathrm{fC}\text{/}$ for the best
performing sensor. The sensors were left for a significant amount of time in
the beam, so as to acquire a high number of particles crossing the sensor. At
DESY, where the particle rate was approximately
$2\text{\,}\mathrm{kHz}\text{/}{\mathrm{cm}}^{2}$ through the entire beam test
period, the sensors were left in the beam for 6 to 8 hours so as to reach a
total of about a million particles crossing each sensor pad in this time. At
the SPS, the rate was less stable but overall significantly higher, reaching
up to more than $6\text{\,}\mathrm{kHz}\text{/}{\mathrm{cm}}^{2}$ at times, so
most sensors received many millions of particles per pad. Once at least
$10^{6}$ events had been reached, the bias was increased by about
$20\text{\,}\mathrm{V}\text{/}$. This was repeated until either the sensor
broke or a bias well above the one required to achieve the
$4\text{\,}\mathrm{fC}\text{/}$ charge requirement was reached. Sensor
breakdown was identified through a current divergence.

###  5 Results and discussion

An example of the plots produced from the recorded data is shown in figure 6.
Figure 6a shows the current of a HPK-P2 W25 2$\times$2 sensor as a function of
time measured during the DESY beam test. It can be clearly seen that the
current gradually rises over time and this is due to the temperature increase
as the dry ice evaporates, as previously described. There is also a large
spike in the current at 90 hours which indicates that the sensor broke. Figure
6b shows the current of a IHEP-IMEv2 W4-II 1$\times$3 sensor as a function of
time measured during the SPS beam test. The current stays steady over time in
comparison to the sensor measured at DESY due to the stable temperature. In
both plots, each pause in the current is due to the HV being switched off for
reasons such as replacing the dry ice (at DESY only) or removing a broken
sensor from the same HV channel. Each large jump in current is caused by the
bias being stepped up as part of the testing plan.

|
---|---
(a) | (b)
Figure 6: Example of the plots produced from the data obtained at the beam
tests. (a) The current of a HPK-P2 W25 SE3-IP4 2$\times$2 sensor irradiated to
$1.5\times 10^{15}$ neq/cm2 measured at DESY. It broke at 90 hours. (b) The
current of a IHEP-IMEv2 W4-II 1$\times$3 sensor irradiated to $1.5\times
10^{15}$ neq/cm2 measured at SPS.

The results of both beam tests are represented in figure 7. The graph shows
the bias voltage required for the minimum required charge of
$4\text{\,}\mathrm{fC}\text{/}$ (when known from strontium-90 measurements in
the laboratories), the lowest bias tested, and the highest bias tested. For
the sensors that broke it also marks the bias at which it failed. Based on
these results, the sensors that are able to reach the bias required for
$4\text{\,}\mathrm{fC}\text{/}$ are HPK-P2 W25 and USTC-IME irradiated to
$1.5\times 10^{15}$ neq/cm2, and IHEP-IMEv2 and FBK-UFSD3.2 irradiated to both
$1.5\times 10^{15}$ neq/cm2 and $2.5\times 10^{15}$ neq/cm2. The USTC-IME-1.1
wafer 11, IHEP-IMEv2 and FBK sensors are carbon-enriched. None of the NDL nor
CNM sensors reached the required bias.

Figure 7: Results of both the DESY and SPS beam tests. Green points mark the
bias required to reach $4\text{\,}\mathrm{fC}\text{/}$ and includes the error.
The diamond marks are the lowest bias tested while the line marks are the
highest bias tested. Yellow marks the measurement range performed at SPS,
while blue marks the measurement range performed at DESY. Red crosses mark the
bias at which a sensor broke in the beam. Sensors are grouped up by producer.

In order to determine the conditions under which a sensor breaks in the beam,
various active material thicknesses are compared against the last bias and the
electric field, both shown in figure 8.

As can be seen, sensors with a larger active material thickness were able to
withstand a higher bias, and when an average field in the sensors is
calculated, <E>=Vbias/D, they start to break once they reach
$12\text{\,}\mathrm{V}\text{/}\mathrm{\SIUnitSymbolMicro m}$ regardless of the
LGAD design. This explains why the sensor from FBK-UFSD3.2 wafer 7 (active
thickness $55\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$) irradiated to
$2.5\times 10^{15}$ neq/cm2 survived at biases well above
$600\text{\,}\mathrm{V}\text{/}$ while sensors from other FBK wafers (active
thickness $45\text{\,}\mathrm{\SIUnitSymbolMicro m}\text{/}$) irradiated to
$2.5\times 10^{15}$ neq/cm2 did not. No fatality was observed at
E<$12\text{\,}\mathrm{V}\text{/}\mathrm{\SIUnitSymbolMicro m}$, although this
observation is with a limited number of particles crossing the detector.
However, to account for uncertainty and the fact that breakdown depends
exponentially on the field, a safe zone of operation of
E<$11\text{\,}\mathrm{V}\text{/}\mathrm{\SIUnitSymbolMicro m}$ is proposed.

|
---|---
(a) | (b)
Figure 8: Comparison of (a) the thickness with the last tested bias for all
sensors, and (b) the thickness with the electric field in the sensor. In both
plots the red circles mark sensors that broke and the red dashed line
indicates $12\text{\,}\mathrm{V}\text{/}\mathrm{\SIUnitSymbolMicro m}$.

For the sensors that did not survive, the probability of mortality per
particle crossing the sensor can be calculated. The number of particles
crossing the $1\text{\,}{\mathrm{cm}}^{2}\text{/}$ scintillator was recorded,
and this value was scaled down to the number of particles that crossed a
single-pad of each sensor. The probability was then calculated as the inverse
of the number of particles that passed through the sensor pad before it broke.
The precision with which the rate, and therefore the number of particles, is
known is better than 50% due to the control of the beam profile and the
efficiency of the particle scintillator counter. The probabilities are
presented in figure 9 and are generally in the range of $10^{-5}-10^{-6}$. As
would be expected, the sensors that broke at a bias of
$680\text{\,}\mathrm{V}\text{/}$ have the highest probability of mortality.
However, these sensors were not tested at biases below
$680\text{\,}\mathrm{V}\text{/}$, so it is possible that they would have
broken at a lower bias with a lower probability.

Figure 9: Probability of mortality for each particle crossing a single-pad of
the sensor, calculated for the sensors that broke at DESY and SPS beams. The
bias that the sensor was operated at when mortality occurred is denoted by a
red cross.

###  6 Conclusion

In this study, many sensors from various producers and runs were sent to two
test beam facilities to have their survivability tested in the particle beam.
Some of the sensors broke at biases lower than those tested in the lab,
demonstrating that the particle beam itself causes their mortality. It was
determined that it was not the design of the LGAD that changed the chances of
mortality, but the strength of the electric field. It is concluded that the
safe zone of operation is at electric fields below
$11\text{\,}\mathrm{V}\text{/}\mathrm{\SIUnitSymbolMicro m}$.

### Acknowledgements

The authors gratefully acknowledge CERN and the SPS staff for successfully
operating the North Experimental Area and for continuous supports to the
users. The measurements leading to these results have been performed at the
Test Beam Facility at DESY Hamburg (Germany), a member of the Helmholtz
Association (HGF). This work was partially funded by MINECO, Spanish
Government, under grant RTI2018-094906-B-C21. This project has received
funding from the European Union’s Horizon 2020 research and innovation
programme under the Marie Sklodowska-Curie grant agreement No. 754510. This
work was partially funded by: the Spanish Government, under grant
FPA2015-69260-C3-2-R and SEV-2012-0234 (Severo Ochoa excellence programme);
and by the H2020 project AIDA-2020, GA no. 654168. The authors acknowledge the
financial support from the Slovenian Research Agency (ARRS J1-1699, ARRS
P1-0135). The USP group acknowledges support from FAPESP grant 2020/04867-2
and CAPES.

### References

  * [1] ATLAS Collaboration.  A High-Granularity Timing Detector for the ATLAS Phase-II Upgrade: Technical Design Report, 2020.
  * [2] G. Pellegrini et al.  Technology developments and first measurements of Low Gain Avalanche Detectors (LGAD) for high energy physics applications.  Nucl. Instrum. Meth. A, 765:12–16, 2014.
  * [3] M. Carulla et al.  First 50 $\mu$m thick LGAD fabrication at CNM.  28th RD50 Workshop, Torino, Italy, 2016.
  * [4] G. Kramberger et al.  Radiation effects in Low Gain Avalanche Detectors after hadron irradiations.  JINST, 10(07):P07006, 2015.
  * [5] CERN SPS north area, http://sba.web.cern.ch/sba/BeamsAndAreas/H6/H6_presentation.html.
  * [6] R. Diener et al.  The DESY II Test Beam Facility.  Nucl. Instrum. Meth. A, 922:265–286, 2019.
  * [7] S. Hidalgo.  LGAD 4 ATLAS, IMB—CNM activities, work presented during a meeting within the ATLAS collaboration, attended by IFAE, SCIPP-UCSC, IJS and IMB-CNM, May 2019.
  * [8] G. Kramberger.  LGADs for timing detectors at HL-LHC.  Presented at CERN detector seminars, CERN, 2021.
  * [9] R. Heller.  Studies of LGAD mortality using the fermilab test beam.  38th RD50 Workshop, 2021.
  * [10] Gordana Laštovička-Medin, Mateusz Rebarz, Gregor Kramberger, Jiří Kroll, Kamil Kropielnicki, Tomáš Laštovička, Martin Precek, and Jakob Andreasson.  Femtosecond laser studies of the Single Event Effects in Low Gain Avalanche Detectors and PINs at ELI Beamlines.  Nucl. Instrum. Meth. A, 1041:167321, 2022.
  * [11] G. Laštovička-Medin, G. Kramberger, M. Rebarz, J. Andreasson, K. Kropielnicki, T. Laštovička, and J. Kroll.  A brief overview of the studies on the irreversible breakdown of LGAD testing samples irradiated at the critical LHC-HL fluences.  JINST, 17(07):C07020, 2022.
  * [12] S. Agostinelli et al.  GEANT4–a simulation toolkit.  Nucl. Instrum. Meth. A, 506:250–303, 2003.
  * [13] Mika Huhtinen.  Highly ionising events in silicon detectors.  Technical report, CERN, Geneva, 2002.
  * [14] G. Lindström et al.  Radiation hard silicon detectors–developments by the rd48 (rose) collaboration.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 466(2):308–326, 2001.  4th Int. Symp. on Development and Application of Semiconductor Tracking Detectors.
  * [15] Computational analysis of irradiation facilities at the jsi triga reactor.  Applied Radiation and Isotopes, 70(3):483–488, 2012.
  * [16] M. Ferrero et al.  Radiation resistant LGAD design.  Nucl. Instrum. Meth. A, 919:16–26, 2019.
  * [17] C. Agapopoulou et al.  Performance in beam tests of irradiated Low Gain Avalanche Detectors for the ATLAS High Granularity Timing Detector.  JINST, 17(09):P09026, 2022.

## Appendices

###  Appendix A List of tested sensors

The sensors that were tested in both test beam facilities are listed in this
table. The sensor name is made up of the producer, run, wafer, and in certain
cases other information such as the type or inter-pad (IP) distance. Where the
sensor is a PiN and not an LGAD, this is mentioned in parenthesis next to the
sensor name. The size of the sensor refers to the number of pads. In the case
of LGAD-PiN this means that there were two pads, with one being an LGAD and
the other a PiN. The annealing for the majority of the sensors was done at
$60\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{C}$, unless stated otherwise
in parenthesis next to the annealing time.

Table 1: List of tested sensors from CNM, USTC-IME, NDL and HPK vendors. Sensor name |  | Pad
---
structure
| Active thickness
---
[$\mu$m]
$V_{gl}$ [V] |  | Fluence
---
[neq/cm2]
| Annealing
---
time [min]
(@$60\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{C}$)
Tested at
CNM 12916 W2 |  2$\times$2 | 50 | 13.2 | $1.5\times 10^{15}$ | 80 | DESY
CNM 12916 W2 |  2$\times$2 | 50 | 6.6 | $2.5\times 10^{15}$ | 80 | DESY
CNM 13002 | single large | 55 | unknown | $1.5\times 10^{15}$ | 80 | DESY
USTC-IME-1.1 W8 | single | 50 | 15.6 | $1.5\times 10^{15}$ | 80 | SPS
USTC-IME-1.1 W10 | single | 50 | 19.1 | $1.5\times 10^{15}$ | 80 | SPS
USTC-IME-1.1 W11 | single | 50 | 29.2 | $1.5\times 10^{15}$ | 80 | DESY/ SPS
USTC-IME-1.1 W11 | single | 50 | 23.3 | $2.5\times 10^{15}$ | 80 | DESY
NDLv3 B14-D3 |  2$\times$2 | 50 | 10.8 | $1.5\times 10^{15}$ | 80 | DESY
NDLv3 B14-D5 |  2$\times$2 | 50 | 10.8 | $1.5\times 10^{15}$ | 80 | SPS
NDLv3 B14-B5 |  2$\times$2 | 50 | unmeasurable | $2.5\times 10^{15}$ | 80 | SPS
HPK-P1 Type1.1 SE2-IP9 |  2$\times$2 | 35 | unknown | $1.5\times 10^{15}$ | 80 | DESY
HPK-P1 Type1.2 SE3-IP5 |  2$\times$2 | 35 | unknown | $1.5\times 10^{15}$ | 80 | DESY
HPK-P1 Type3.1 SE5 | single | 50 | unknown | $3\times 10^{15}$ | 20480 | DESY
HPK-P1 Type3.2 SE2 | single | 50 | unknown | $3\times 10^{15}$ | 20480 | DESY
HPK-P2 W25 SE3-IP4 |  2$\times$2 | 50 | 29.5 | $1.5\times 10^{15}$ | 80 | DESY
HPK-P2 W25 SE3-IP7 |  2$\times$2 | 50 | 29.5 | $1.5\times 10^{15}$ | 80 | DESY
HPK-P2 W25 SE3-IP5 |  2$\times$2 | 50 | 29.5 | $1.5\times 10^{15}$ | 80 | DESY
HPK-P2 W25 SE3-IP3 |  2$\times$2 | 50 | 29.5 | $1.5\times 10^{15}$ | 80 | DESY
HPK-P2 W25 SE3-IP5 |  2$\times$2 | 50 | 29.5 | $1.5\times 10^{15}$ | 80 | SPS
HPK-P2 W25 SE3-IP5 |  2$\times$2 | 50 | 29.5 | $1.5\times 10^{15}$ | 80 | SPS
HPK-P2 W25 SE3-IP5 |  5$\times$5 | 50 | 29.5 | $1.5\times 10^{15}$ | 80 | SPS
HPK-P2 W25 SE3-IP7 |  5$\times$5 | 50 | 29.5 | $1.5\times 10^{15}$ | 80 | DESY/ SPS
HPK-P2 W31 SE3-IP5 |  2$\times$2 | 50 | 27.4 | $1.5\times 10^{15}$ | 80 | SPS
HPK-P2 W31 SE3-IP5 |  5$\times$5 | 50 | 27.4 | $1.5\times 10^{15}$ | 80 | SPS
HPK-P2 W31 SE3-IP5 |  5$\times$5 | 50 | 27.4 | $1.5\times 10^{15}$ | 80 | DESY/ SPS
HPK-P2 W36 SE3-IP4 |  2$\times$2 | 50 | 24.8 | $1.5\times 10^{15}$ | 80 | SPS
HPK-P2 W25 (PIN) | single | 50 | N/A | $2.5\times 10^{15}$ | 80 | DESY
HPK-P2 W25 SE3-IP3 |  5$\times$5 | 50 | 18.9 | $2.5\times 10^{15}$ | 80 | DESY
HPK-P2 W31 SE5-IP7 |  5$\times$5 | 50 | 18.9 | $2.5\times 10^{15}$ | 80 | DESY
HPK-P2 W31 SE3-IP5 |  5$\times$5 | 50 | 18.9 | $2.5\times 10^{15}$ | 80 | SPS
HPK-P2 W36 SE3-IP5 |  5$\times$5 | 50 | 15.2 | $2.5\times 10^{15}$ | 80 | SPS
Table 2: List of tested sensors from FBK and IHEP-IME vendors. Sensor name |  | Pad
---
structure
| Thickness
---
[$\mu$m]
$V_{gl}$ [V] |  | Fluence
---
[neq/cm2]
| Annealing
---
time [min]
(@$60\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{C}$)
Tested at
FBK-UFSD3.2 W7A-Type 10 |  2$\times$2 | 55 | 18.0 | $1.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W7-Type 4 |  2$\times$2 | 55 | 19.6 | $1.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W13 | LGAD-PiN | 45 | 34.2 | $1.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W13-Type 10 |  2$\times$2 | 45 | 34.2 | $1.5\times 10^{15}$ | 80 | DESY/ SPS
FBK-UFSD3.2 W13-Type 4 |  2$\times$2 | 45 | 34.2 | $1.5\times 10^{15}$ | 130 | SPS
FBK-UFSD3.2 W18 | LGAD-PiN | 45 | 33.1 | $1.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W18-Type 10 |  2$\times$2 | 45 | 33.1 | $1.5\times 10^{15}$ | 80 | DESY/ SPS
FBK-UFSD3.2 W19-Type 10 |  2$\times$2 | 45 | 37.3 | $1.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W19-Type 4 |  2$\times$2 | 45 | 37.3 | $1.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W19-Type 9 |  5$\times$5 | 45 | 37.3 | $1.5\times 10^{15}$ | 80 | DESY/ SPS
FBK-UFSD3.2 W7 | LGAD-PiN | 55 | 17.6 | $2.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W13 | LGAD-PiN | 45 | 29.1 | $2.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W13-Type 10 |  2$\times$2 | 45 | 29.1 | $2.5\times 10^{15}$ | 80 | DESY/ SPS
FBK-UFSD3.2 W13-Type 4 |  2$\times$2 | 45 | 29.1 | $2.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W18 | LGAD-PiN | 45 | 28.5 | $2.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W18-Type 4 |  2$\times$2 | 45 | 28.5 | $2.5\times 10^{15}$ | 80 | SPS
FBK-UFSD3.2 W19 | LGAD-PiN | 45 | 32.4 | $2.5\times 10^{15}$ | 80 | DESY
FBK-UFSD3.2 W19-Type 10 |  2$\times$2 | 45 | 32.4 | $2.5\times 10^{15}$ | 80 | DESY/ SPS
FBK-UFSD3.2 W3-Type 9 |  5$\times$5 | 45 | unknown | $2.5\times 10^{15}$ | 80 | DESY/ SPS
IHEP-IMEv1 W7 |  2$\times$2 | 50 | 14.3 | $1.5\times 10^{15}$ | 80 | DESY
IHEP-IMEv1 W7 |  2$\times$2 | 50 | 14.3 | $1.5\times 10^{15}$ | 80 | DESY/ SPS
IHEP-IMEv1 W1 | single | 50 | 10.8 | $2.5\times 10^{15}$ | 80 | DESY
IHEP-IMEv1 W1 (PIN) | single | 50 | N/A | $2.5\times 10^{15}$ | 80 | DESY
IHEP-IMEv1 W7 |  2$\times$2 | 50 | 11.1 | $2.5\times 10^{15}$ | 80 | DESY
IHEP-IMEv1 W7 (PIN) | single | 50 | N/A | $2.5\times 10^{15}$ | 80 | DESY
IHEP-IMEv2 W4-II |  1$\times$3 | 50 | 18.3 | $1.5\times 10^{15}$ | 80 | SPS
IHEP-IMEv2 W4-IV |  1$\times$3 | 50 | 19.7 | $1.5\times 10^{15}$ | 80 | SPS
IHEP-IMEv2 W7-I |  1$\times$3 | 50 | 18.7 | $1.5\times 10^{15}$ | 80 | SPS
IHEP-IMEv2 W7-III |  1$\times$3 | 50 | 20.3 | $1.5\times 10^{15}$ | 80 | SPS
IHEP-IMEv2 W4-II |  1$\times$3 | 50 | 16.7 | $2.5\times 10^{15}$ |  80 (@90∘C) | SPS
IHEP-IMEv2 W4-IV |  1$\times$3 | 50 | 17.2 | $2.5\times 10^{15}$ |  80 (@90∘C) | SPS
IHEP-IMEv2 W7-I |  1$\times$3 | 50 | 16.3 | $2.5\times 10^{15}$ |  80 (@90∘C) | SPS
IHEP-IMEv2 W7-III |  1$\times$3 | 50 | 18.7 | $2.5\times 10^{15}$ |  80 (@90∘C) | SPS