Further author information: (Send correspondence to Hector Santos-Villalobos)
Hector Santos-Villalobos: E-mail<EMAIL_ADDRESS>Telephone: 1 865 574 0215
David S. Bolme: E-mail<EMAIL_ADDRESS>Telephone: 1 865 576 0300

Notice: This manuscript has been authored by UT-Battelle, LLC, under contract
DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government
retains and the publisher, by accepting the article for publication,
acknowledges that the US government retains a nonexclusive, paid-up,
irrevocable, worldwide license to publish or reproduce the published form of
this manuscript, or allow others to do so, for US government purposes. DOE
will provide public access to these results of federally sponsored research in
accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-
public-access-plan).

# The Mertens Unrolled Network (MU-Net): A High Dynamic Range Fusion Neural
Network for Through the Windshield Driver Recognition

Max Ruby  Purdue University, 610 Purdue Mall, West Lafayette, USA  David S.
Bolme  Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, USA  Joel
Brogan  Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, USA
David Cornett III  Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak
Ridge, USA  Baldemar Delgado  Texas A&M University-Kingsville, 700 University
Blvd, Kingsville, USA  Gavin Jager  University of Michigan, 500 S. State St,
Ann Arbor, USA  Christi Johnson  Oak Ridge National Laboratory, 1 Bethel
Valley Rd, Oak Ridge, USA  Jose Martinez-Mendoza  Texas A&M University-
Kingsville, 700 University Blvd, Kingsville, USA  Hector Santos-Villalobos
Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, USA  Nisha
Srinivas  Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, USA

###### Abstract

Face recognition of vehicle occupants through windshields in unconstrained
environments poses a number of unique challenges ranging from glare, poor
illumination, driver pose and motion blur. In this paper, we further develop
the hardware and software components of a custom vehicle imaging system to
better overcome these challenges. After the build out of a physical prototype
system that performs High Dynamic Range (HDR) imaging, we collect a small
dataset of through-windshield image captures of known drivers. We then re-
formulate the classical Mertens-Kautz-Van Reeth HDR fusion algorithm as a pre-
initialized neural network, which we name the Mertens Unrolled Network (MU-
Net), for the purpose of fine-tuning the HDR output of through-windshield
images. Reconstructed faces from this novel HDR method are then evaluated and
compared against other traditional and experimental HDR methods in a pre-
trained state-of-the-art (SOTA) facial recognition pipeline, verifying the
efficacy of our approach.

###### keywords:

MU-Net, HDR, Facial Recognition, Through Windshield

##  1 INTRODUCTION

Face recognition of vehicle occupants in unconstrained environments,
particularly through the windshield, poses a number of challenges. Previous
research has shown that artifacts introduced by a windshield can greatly
impact a camera’s ability to image within the vehicle interior [1].

Additionally, images captured in this scenario are typically from long
distances and moderate speeds, which reduces the amount of light available to
the sensor. Low-intensity ultraviolet or near-infrared (NIR) illumination has
become increasingly ineffective in alleviating these issues as tinted
windshields that block these wavelengths gain popularity. Likewise, increasing
the exposure time cannot fix the problems associated with that since the
vehicle is moving, and motion blur artifacts caused by increasing exposure
time significantly degrades face recognition quality.

Furthermore, windshields are reflective, which produces several unique
challenges. First, it further reduces the light available to the sensors.
Second, the windshield will reflect light from other sources into the sensor.
If that light propagates directly from a light source, this will often cause
obstructive glare (Figure 1). Even if that light is reflected off of an object
onto the windshield, the object will appear as an unwanted artifact overlaid
with the desired image of the driver.

Current solutions for this problem include flashing visible lights at the
driver, such as the system devised by Gatekeeper Security [2]. This is highly
undesirable, as it both distracts the driver from safe vehicle operation and
has the potential to cause obstructive glare due to the windshield.

To provide the best possible input to a deep learning algorithm, a custom
multi-camera imaging system was developed to specifically mitigate these
hurdles while remaining non-intrusive [3]. The system is modular in design,
where each unit is composed of both an imaging system and an associated
computational system, seen in Figures 2, 3, and 4. The raw images captured by
the system can subsequently be processed by any HDR method to ultimately
provide the processed input to facial recognition software.

Figure 1: A typical example of an HDR burst captured from the Through the
Windshield System, including heavy glaring artifacts.

This paper extends the previous work in two main ways: First, we collect and
annotate a new dataset for the purpose of training HDR reconstruction models
fine-tuned for our specific imaging hardware. Second, we propose an novel
network architecture for HDR reconstruction by using an unrolling technique to
model the Mertens-Kautz-Van Reeth HDR algorithm [4] as a neural network - we
call this network the Mertens-Unrolled Network, or MU-Net. Finally, to analyze
the end-to-end system, SOTA facial detection and recognition algorithms are
used to evaluate the system’s image capture and HDR reconstruction quality in
its overall goal to detect and identify faces through the windshield of moving
vehicles. We also compare our efficacy of MU-Net against other classical and
deelply-learned HDR methods from previous works in the same recognition
pipeline.

##  2 Motivation

The applications of this system are numerous. For example, the ability to
recognize drivers through the windshield has the potential to speed up
security checkpoints in restricted areas. These can also be permanently
deployed along roads to locate criminals and terrorists. Moreover, a more
agile version of this system can be used to greatly accelerate the flow of
traffic through temporary checkpoints which have been set up to catch
identified criminals who have recently committed a high-profile crime or
escaped from prison. Alternatively, it can be used to help locate a child who
is the focus of an amber alert, which can possibly return them to safety
faster.

However, contrast within the vehicle is a function of the ambient light, the
shadows from the vehicle, and the tint of the windshield. We should expect
that these conditions may change quickly; by the addition or removal of cloud
cover, or by a passing driver with their lights on. If we have a single camera
with fixed optics, then we cannot expect the image to have good contrast most
of the time. Unfortunately, automatic camera gain is difficult given these
situations. This is why HDR fusion techniques for our camera system are so
valuable - we expect that if we take pictures from multiple cameras with
varying optics, we should be able to get a high-quality image by combining
them.

##  3 Related Work

There is a notable body of work in the field of HDR fusion, a review of which
can be found in [5]. Note the algorithms given by Mertens et. al. [4] and
Debevec et. al. [6] usually perform adequate HDR fusion under normal
circumstances. However, when dealing with through-windshield environments,
these methods are not designed to account for the unfortunately common cases
of obstructive glare, which obscures the face within one or more images in the
HDR burst.

Most external driver recognition research has been performed in the commercial
sector [7] and public sector [8]. Beyond an initial case study at Oak Ridge
National Laboratory [3], very little open research has been published on the
topic. While little work has been done in the field of driver recognition
through the windshield , there has been significant related work two related
fields: Vehicle occupancy detection, and glass artifact suppression. In
vehicle occupancy detection [9, 10, 11, 12], algorithms aim to detect and
accurately count the number of occupants in an automobile, usually for the
purpose of automatically enforcing HOV and carpool lane rules. In glass
artifact suppression, algorithms look to effectively remove artifacts such as
reflection, refraction, and glare, that are introduced by glass into real-
world images [13, 14, 15, 16]. While these works are useful within their own
context, they are solutions designed around separate problems and therefore
sub-optimal to solve the through-windshield recognition task, due to less-
stringent time or accuracy constraints on their output

Convolutional Neural Networks (CNNs) are the the current SOTA approaches for a
large number of image processing tasks. Some recent successes of the neural
networks can be attributed to their ability learn and optimize the behavior
previously established deterministic algorithms, as is pointed out in [17].
Moreover, the technique of taking an algorithm and turning it into a neural
network - for example, through “deconstructing” or “Unrolling” - has been
shown to effectively incorporate domain-specific priors into a given model[18,
19]. This technique can provide a straightforward improvement to a problem’s
solution, as long as the solution in question is already acceptably effective
and easily posed as a network architecture. If we have designed and initialize
a network with parameters to accommodate a target algorithm, we can ensure
that the original algorithm can not only be mimicked by the neural network,
but further improve upon it through fine-tuning [20]. We should expect an
algorithm re-formulated as a neural network to perform no worse than the
original.

There is a growing body of work focused around performing HDR reconstruction
using a CNN. Many of them attempt to do this using a single shot [21, 22] or
by simply having a extreme number of weights [21, 23, 24, 22]. For real-time,
robust driver recognition, both of these are unacceptable. In the former case,
we expect that a single shot will have a high probability of either misfiring
or inaccurately capturing the face, due to the potential for obstructive
glare. In the latter case, an extremely large model cannot be run locally,
within an embedded system on-device, and expected to perform with adequate
throughput for the frequency and volume of highway traffic. Previous work
attempts to overcome these issues using a GAN approach [3], which this work
improves upon. While our proposed Deep HDR Pipeline, the MU-Net, is designed
and trained to work specifically with the our custom face imaging system, it’s
architectural design mimicking the general Merten’s HDR algorithm inherently
improves the method’s generalizability across all domains, not just through-
windshield faces.

##  4 Computational Imaging System Overview

Our custom imaging system was developed to withstand harsh environments as
well as produce quick results for biometric assessment in field deployment.
The overall system design is modular, where each computational unit added can
perform stand-alone, or input its data into a central analysis system. A
single computational imaging unit is comprised of two primary components:
weatherproof camera enclosure and a ruggedized computer. The camera enclosure
is shown in Figure 2 and the associated computing system can be seen in Figure
3.

The camera enclosure houses an array of three Basler GigE cameras ($2048\times
1536$ pixel matrix, 3.45 $\mu$m pixel size), mounted on a 3D-printed bracket
and arranged horizontally. The cameras are linked to the associated computer
unit via Power over Ethernet (PoE) connections. They are aligned to have a
common focal point at 10 meters. Gain and exposure time is programmatically
adjusted at the time of data acquisition. The cameras are equipped with
filters as follows: First, all cameras are equipped with a 50 mm focal length
and 34 mm aperture lens. Next, all cameras are equipped with a Midwest Optical
Systems UVIR filter to sample only visible light. Third, Cameras #1 and #2 are
equipped with a Midwest Optical Systems 0.9 and 0.3 ND filter, respectively.
Last, cameras #1, #2, and #3 are equipped with Meadowlark Optics linear
polarizers (extinction ratio greater than 10,000,000:1) placed at angles of
$80^{\circ}$, $90^{\circ}$, and $100^{\circ}$, respectively.

The ruggedized computer is also housed in a 3D-printed, weatherize enclosure.
The computer has 32GB of memory, Intel Core i7 CPU with 8 processing cores and
a NVIDIA GTX 1050 Ti GPU. The computer was chosen for its PoE ports used to
interface with the camera unit as well as its GPU so that it can handle the
computational burden of the implemented deep learning algorithms.

To trigger the imaging system an Optex through beam photoelectric sensor was
used. To ensure frame synchronization across multiple systems the same trigger
signal was distributed to all systems. Once triggered, an imaging unit
captures 20 frame-sets (20 x 3 cameras = 60 images per frame-set).

The system software is also modular by design, and leverages the Google Remote
Procedure Call (gRPC) library. The gRPC library allows for the use of
independent servers so that each task of the image acquistion and processing
pipeline can be managed independently.

For the all experiments mentioned in this paper, two modular units were used -
one on either side of the driving lane that test subjects would drive through.
This configuration was used to account for driver head pose, ensuring that a
sufficiently frontal face shot could be captured for detection and
identification tasks. The layout of the units can be observed in Figure 4.

Figure 2: Computational imaging unit - weatherproof camera array. Figure 3:
Computational imaging unit - weatherproof ruggedized computer. Figure 4: Two-
unit camera setup for dataset collection at security portal.

##  5 Overview of Image Pipeline

Before images are acquired, the cameras are calibrated to give an initial
estimate for coarse image registration; After this, we use FaRO’s [25]
implementation of RetinaFace face detection on the three images. Using these
detection locations, a secondary registration is performed (in the case that a
face was not detected in a given image, this step is skipped). We crop the
images to 600 $\times$ 600 pixels and perform fine registration using a
modified form FFT-based image registration [26]; We modify the algorithm to
multiply the images using a Kaiser window with $\beta=4$ as to “prefer”
smaller shifts (we expect the fine registration to be, in our case, usually
between $30$ and $150$ pixels in any given direction, as we have already been
coarsely registered). We utilize this method of registration to achieve
accurate correspondence and alignment without the need to detect facial
landmark points, which are often noisy or incorrect within our dataset. It
should be noted that this method deviates from the previous work, [3], which
uses Hamming windows to perform alignment instead. The images are then cropped
to $256\times 256$ pixels and then fed into the HDR fusion neural network
outlined in Figure 5. This produces a final face tile which can then be fed to
a biometric template extractor.

##  6 MU-Net: Mertens Unrolled Network

Motivated by the simplicity of the Mertens HDR algorithm, we first wanted to
show that the algorithm itself could be reformulated within a CNN
architecture. Therefore, MU-Net was conceived by applying an “Unrolling”,
first used in [27], to the Mertens algorithm.

For completeness, a rough sketch of the Mertens HDR algorithm follows. First,
for each pixel, the contrast and well-exposedness of each pixel is computed,
by the formulas

$\begin{split}C_{i,j,k}=|R_{i-1,j,k}+R_{i+1,j,k}+R_{i,j-1,k}\\\
+R_{i,j+1,k}-4R_{i,j,k}|\end{split}$ (1)

and

$X_{i,j,k}=\exp\left(-\frac{1}{2}\left(\frac{R_{i,j,k}-0.5}{\sigma}\right)^{2}\right)$
(2)

where $R_{i,j,k}\in[0,1]$ is the $(i,j)$th pixel of the $k$th image, and
$\sigma$ is some standard deviation (in the standard implementation
$\sigma=0.2$.) Concretely, $X$ is a measure of image exposure, while $C$ is a
measure of contrast. We note that for a color image, there is also a measure
of saturation; this network can be extended to include this saturation
measure, but in our greyscale implementation it is not necessary.

After this, the $C$ and $X$ measurements are multiplied to give a measure of
“quality” of each pixel of each image. This quality metric is then normalized,
and then the images are mixed using a Laplacian Blending scheme, as in [28].

We determine that the way to proceed is to develop a network composed of three
blocks, following the scheme above. First, a Contrast Block, or “C-Block”,
which is initialized to compute the contrast as above. Second, a Well-Exposure
Block, or “X-Block,”, which is initialized to compute the Well-Exposure as
above. Third, a Pyramid Blending Block, or “P-Block,”, which is initialized to
perform Laplacian Pyramid Blending. The output of the C-Block and X-Block is
multiplied together, a convolutional layer is applied, and the resulting
“Image Quality” is fed into the P-Block along with the original images. A
sketch of the neural network, as well as the blocks, are in Figure 5. Note
that only 3 “steps” of the P-Block are shown in Figure 5, for brevity. In
reality, our network contains 8 (that is, the network should be naturally
extended so that the output of the upper-right and lower-right 2D
convolutional layers are $1\times 1$ pixel each, as each convolutional layer
has a stride of 2)

Figure 5: An overview of the MU-Net architecture. From left to right: i) The
Contrast Block (C-Block), ii) The Exposure Block (X-Block), iii) the
Concatenation Block (P-Block). Note that P-Block shows only 3 unrolled loops
for brevity. Each convolutional and upsampling layer has a stride of 2 with 3
output channels, unless denoted by a superscript. Each conv layer utilizes
tanh activation unless otherwise denoted with a ReLU block. * - contains an
ELU, Negative, Exp, and Tanh activation.

Unless denoted with a RelU block, each convolutional layer in Figure 5 is
followed by a tanh activation. The C-Block’s first convolutional layer has 6
output channels, while its second has 3 output channels, using $7\times 7$
kernels. Each layer in the X-Block has 3 channels with $3\times 3$ kernels.
The ReLU layer that has been starred within the X-Block has an ELU, Negative,
Exp, and Tanh activation. The output of the last Conv2D layer represents an
exponent, whose value we want to remain negative so that our network’s output
never “explodes”. The ELU and Negative force that condition on that parameter.
The Conv2D layer between the P-Block has $3$ output channels and a $3\times 3$
kernel. Every Conv2D layer in the P-block has a $7\times 7$ kernel and a tanh
activation function. All of them have 3 output channels, with the exception of
the ones which are superscripted.

As we developed this generator by unrolling the Laplacian Blending scheme in
the Mertens algorithm, we call it the Mertens Unrolled Network, or “MU-Net.”
Note that the architecture is rather light-weight: our current architecture
has under $45,000$ parameters.

A significant advantage to this approach is that we avoid treating our neural
network like a “black box.” Specifically, we expect that our C-Block measures
something similar to the contrast, our X-Block measures something similar to
the Well-Exposedness, and that our P-Block performs something similar to
Laplacian Pyramid Blending. This means that it should be possible to debug the
MU-Net, and we can identify when and where our training has gone badly.

###  6.1 MU-Net Discriminator: The NU-Net

We use a Feature Pyramid Network (FPN) architecture to design our
discriminator [29]. The FPN provides a multi-scale framework for learning a
feature to detect real and fake images at different frequency scales. If we
take the maximum element in each channel (that is, if we maxpool down to
$1\times 1$ pixel), this should serve as an indicator of how much each feature
exists within the input image. This pooled result is then feed that a
classifier layer with a single dimension output. We decided to downsample more
aggressively than in the MU-Net to reduce the number of parameters required;
it was found that it was easy for the architecture to overtrain if it was
given more parameters, so we opt to use a smaller network. Moreover, we feed
the contrast well-exposure maps from the C-Block and X-Block into the
discriminator, as these are pertinent features to generate correctly when
producing a fused HDR image.

Figure 6: NU-Net (Discriminator) Architecture.

The architecture for NU-Net is shown in Figure 6. The upper half of the
diagram is the same as in the MU-Net, utilizing the C- and X-Block. The
leftmost Conv2D layers all have a $7\times 7$ kernel, except for the last,
which has a $4\times 4$ kernel, due to reduced resolution. They have 4,8,16,
and 32 output channels, from top to bottom, respectively. The upsampling
Conv2D layers have 16 output channels and a $7\times 7$ kernel, except the
first, which has a $3\times 3$ kernel due to reduced resolution. The rightmost
Conv2D layers all have $8$ output channels, with $5\times 5$ kernels.

As we developed this discriminator by doing something other than unrolling, we
call it the Not Unrolled Network, or “NU-Net.” Both the MU-Net and NU-Net were
built in Keras, using the Tensorflow backend.

###  6.2 Training

We were able to take a number of photos at the security gate of a restricted
access vehicle portal. From this set of photos, we were able to capture $1957$
triples of images which were of faces, as well as $2371$ images of faces which
were of sufficiently high quality for training the discriminator. Of this, we
used $1440$ of the triples and face tiles for training, $160$ for validation,
and the rest as a testing set.

We determined that the use of Batch Normalization layers to overcome the
exploding gradients problem was not a good solution; naive approaches to Batch
Normalization eliminate the advantages of our initialization, due to our
initialization passing values of nonzero mean and nonunit variance between
layers. We opted to perform $L1$ normalization on the layers, with
$\lambda=0.001$, along with gradient clipping, with our clipnorm value being
$0.1$. Moreover, we determined that pretraining the adversary for $10$ epochs
would improve the initial training, as our generator had a “head-start” due to
its good initialization. We used ADAM to optimize, with an initial learning
rate of $0.005$, a learning rate decay of $0.000001$, and a value of epsilon
equal to $0.001$; we set all other parameters to the defaults.

During training, we noticed that the output of our GAN began to degenerate
once the adversary began to learn. It seems that this is due to the generator
learning a “cheap trick” that fools the adversary, but does not produce
realistic looking images. This was solved by taking the training result before
degeneration; we save output images after each epoch, and take the result that
looks qualitatively best. Although we trained for $100$ epochs, we determined
that the result of training after the $71$st epoch provided the best aesthetic
quality.

##  7 Experimental Evaluation and Results

This section details our effort to evaluate and analyze the effectiveness of
the MU-Net HDR algorithm for the purpose of through-windshield facial
recognition.

Figure 7: ROC performance of MU-Net compared to other HDR Methods.

###  7.1 The Driver Dataset

To evaluate MU-Net on the Image pipeline from Section 5, we capture a new
dataset of known drivers, disjoint from the training data captured in Section
6.2. The dataset was captured at a facility security portal using live driving
conditions over the course of two days. For this experiment there were 22
participants enrolled as gallery subjects for the recognition set. These
participants entered the facility through the security portal under normal
driving conditions. The images captured during this experiment represented the
scene illumination changes that occur through sunrise to sundown. There were
471 vehicle triggered events that produced 56,520 raw images. For each of
these events that involved an enrolled participant, each of the three raw
image frame-sets were fused via each of the evaluated HDR methods to be used
as probes by the face identification systems.

###  7.2 Baseline Methods

Classical HDR fusion can be performed in a number of ways, usually combining
images taken at different radiance ranges and fuses them into a single
radiance image. This method allows the fused image to display a range of
radiance levels that could not be captured by a single image. For our
application, HDR imaging provides a means to overcome the artifacts introduced
into face images from windshield glare and reflection.

A brief summary of all methods we will compare to our novel approach is laid
out below:

  * •

Matlab HDR[30] is the standard Matlab implementation for HDR fusion. It is
used as a standard comparison against the other methods.

  * •

Mertens-Kautz-Van HDR (Mertens)[4] is based on the Mertens algorithm utilizes
contrast and exposure estimates to fuse the three raw frames for detection and
recognition.

  * •

Debevec HDR[6] is based on Debevec radiance mapping with Durand tone mapping.
This method is also a physics-based implementation of HDR.

  * •

PeceptionGAN[3] is a GAN network trained specifically for visual perception
and was the initial design for the system used in [3].

Figure 8: Sampled images from cameras (top), input to HDR (middle), Mertens
HDR reconstruction (bottom left), and MU-Net HDR reconstruction (bottom
right).

To analyze the efficacy of MU-Net, we performed a verification study on the
dataset introduced in Section 7.1, comparing performance against the 5
baselines described in Section 7.2. We also include the baseline Single
Camera, which uses a randomly-selected, non-processed image from each HDR
burst. For each method, we detect faces using RetinaFace [31] and subsequently
extract feature templates using Additive Angular Margin Loss for Deep Face
Recognition (ArcFace)[32]. Because our driver dataset is relatively small and
was not used for network training, we report results over the entire set of
subjects. Figure 7 Shows the final results of all 6 methods. The MU-Net
provides the best performance out of all those evaluated, improving upon the
PerceptionGAN by almost $7\%$. As can be seen by the AUC’s of most other
methods, this problem is inherently difficult to solve, and classical HDR
methods do not provide high enough quality reconstructions for accurate face
recognition. Figure 8 shows that classical methods like Debevec and Merten’s
are not robust towards artifacts introduced by windshield glare and
reflection, which significantly impairs verification performance.

Quantitative and Qualitative analysis of the MU-Net pipeline revealed two key
features of the system: First, MU-Net is better able to discriminate areas of
poor quality, including glare and reflection artifacts, than the original
Mertens or Debevec HDR methods, as seen in Figure 8 columns 3,4, and 5.
Second, MU-Net provides output reconstructions that are more stable than that
of PerceptionGan. Because the underlying architecture of MU-Net was designed
and initialized from a state that already performed classical HDR, the network
overfits our data less, resulting in fewer GAN-like artifacts (see Figure 8
columns 6 and 7). We see in general that MU-Net consistently outputs a more
stable image free from windshield artifacts than its competing baselines.

While MU-Net improves upon previous methods for through-windshield
recognition, there is still significant work to be done to achieve a highly
reliable system. For instance, MU-Net can suffer when running in low-light
conditions. This highlights the need for more uniform data collection, ideally
over the course of an entire day. Moreover, the MU-Net was trained on faces -
it does not appear to work properly on images that are not faces.

##  8 Conclusion and Discussion

The MU-Net architecture described in this paper represents a significant step
forward in the use of neural network architectures for HDR fusion for face
recognition through the windshield. Using this method we are able to obtain
higher quality images than classical fusion methods while maintaining a
relatively low computational overhead. While other works have performed HDR
using deep learning, our method is unique in that it is designed on top of the
original Merten’s HDR algorithm, and has a relatively low number of parameters
in contrast to other comparative algorithms. The MU-Net shows that we are able
to successfully incorporate exposure and contrast quality priors into our deep
learning architecture, allowing for faster training, more stable fusion
results that introduce significantly fewer GAN-related generative artifacts.

While the MU-Net is a major improvement over previous methods, significant
work must be done to further enhance facial recognition in the adverse
conditions of through-windshield imaging. We believe it would be beneficial to
train the network stacked end-to-end with a face recognition algorithm,
eliminating the need for training with an ad-hoc discriminator, and optimizing
the HDR fusion expressly for the task of facial recognition.

Efforts are currently being made to expand the custom dataset of through the
windshield images. This expansion would include more enrolled subjects, a more
expansive training set, and a wider array of weather and illumination
conditions. Additionally, the custom imaging system is being redesigned with
better optics, utilizing polarized filter arrays to mitigate glare for
different angles of incidence.

###### Acknowledgements.

This research was supported in part by an appointment to the Oak Ridge
National Laboratory Post-Bachelor’s Research Associate Program, sponsored by
the U.S. Department of Energy and administered by the Oak Ridge Institute for
Science and Education, the U.S. Department of Energy, Office of Science,
Office of Workforce Development for Teachers and Scientists (WDTS) under the
Science Undergraduate Laboratory Internships Program (SULI), and the Texas A&M
University-Kingsville, Office of University Programs, Science and Technology
Directorate, Department of Homeland Security Grant Award # 2012-ST
-062-000054.

## References

  * [1] Bhise, V. and Sethumadhavan, S., “Effect of windshield veiling glare on driver visibility,” Transportation Research Record 2056(1), 1–8 (2008).
  * [2] Harmon, D., “Facial recognition and registration plate reading,” (2017).
  * [3] Cornett, D., Yen, A., Nayola, G., Montez, D., Johnson, C. R., Baird, S. T., Santos-Villalobos, H., and Bolme, D. S., “Through the windshield driver recognition,” Electronic Imaging XVII, 140–1–140–9 (2019).
  * [4] Mertens, T., Kautz, J., and Van Reeth, F., “Exposure fusion,” Pacific Graphics , 382 – 390 (10 2007).
  * [5] Nayana, A. and Johnson, A. K., “High dynamic range imaging- a review,” (2015).
  * [6] Debevec, P. E. and Malik, J., “Recovering high dynamic range radiance maps from photographs,” in [Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques ], SIGGRAPH ’97, 369–378, ACM Press/Addison-Wesley Publishing Co., New York, NY, USA (1997).
  * [7] “Komoto unveils through windscreen face recognition,” (Mar 2018).
  * [8] Brandom, R., “New homeland security system will bring facial recognition to land borders this summer,” (Jun 2018).
  * [9] Hao, X., Chen, H., Yang, Y., Yao, C., Yang, H., and Yang, N., “Occupant detection through near-infrared imaging,” Tamkang Journal of Science and Engineering 14(3), 275–283 (2011).
  * [10] Morris, T., Morellas, V., Canelon-Suarez, D., and Papanikolopoulos, N., “Sensing for hov/hot lanes enforcement,” (2017).
  * [11] Fan, Z., Islam, A. S., Paul, P., Xu, B., and Mestha, L. K., “Front seat vehicle occupancy detection via seat pattern recognition,” (Dec. 17 2013).  US Patent 8,611,608.
  * [12] Artan, Y. and Paul, P., “Occupancy detection in vehicles using fisher vector image representation,” arXiv preprint arXiv:1312.6024 (2013).
  * [13] Kong, N., Tai, Y.-W., and Shin, J. S., “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE transactions on pattern analysis and machine intelligence 36(2), 209–221 (2013).
  * [14] Wieschollek, P., Gallo, O., Gu, J., and Kautz, J., “Separating reflection and transmission images in the wild,” in [European Conference on Computer Vision (ECCV) ], (September 2018).
  * [15] Fan, Q., Yang, J., Hua, G., Chen, B., and Wipf, D., “A generic deep architecture for single image reflection removal and image smoothing,” in [Proceedings of the IEEE International Conference on Computer Vision ], 3238–3247 (2017).
  * [16] Arvanitopoulos, N., Achanta, R., and Susstrunk, S., “Single image reflection suppression,” in [Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ], 4498–4506 (2017).
  * [17] McCann, M. T., Jin, K. H., and Unser, M. A., “Convolutional neural networks for inverse problems in imaging: A review,” IEEE Signal Processing Magazine 34, 85–95 (2017).
  * [18] Adler, J. and Öktem, O., “Learned primal-dual reconstruction,” (2017).
  * [19] Putzky, P. and Welling, M., “Recurrent inference machines for solving inverse problems,” (2017).
  * [20] Rao, Y. and Ni, J., “A deep learning approach to detection of splicing and copy-move forgeries in images,” in [2016 IEEE International Workshop on Information Forensics and Security (WIFS) ], 1–6, IEEE (2016).
  * [21] Huo, Y. and Zhu, X., “High dynamic range image forensics using cnn,” CoRR abs/1902.10938 (2019).
  * [22] Marnerides, D., Bashford-Rogers, T., Hatchett, J., and Debattista, K., “Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content,” CoRR abs/1803.02266 (2018).
  * [23] WANG, J., Wang, W., XU, G., and Liu, H., “End-to-end exposure fusion using convolutional neural network,” IEICE Transactions on Information and Systems E101.D, 560–563 (02 2018).
  * [24] Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R., and Unger, J., “Hdr image reconstruction from a single exposure using deep cnns,” ACM Transactions on Graphics (TOG) 36(6) (2017).
  * [25] Bolme, D. S., III, D. C. C., and Srinivas, N., “FaRO: FAce Recognition from Oak ridge.” https://github.com/ORNL/faro (2019).
  * [26] Reddy, B. S. and Chatterji, B. N., “An fft-based technique for translation, rotation, and scale-invariant image registration,” IEEE Transactions on Image Processing 5, 1266–1271 (Aug 1996).
  * [27] Gregor, K. and LeCun, Y., “Learning fast approximations of sparse coding,” in [Proceedings of the 27th International Conference on International Conference on Machine Learning ], ICML’10, 399–406, Omnipress, USA (2010).
  * [28] ADELSON, H., “Laplacian pyramid as a compact image code peter j. burt,” (07 2019).
  * [29] Lin, T., Dollár, P., Girshick, R. B., He, K., Hariharan, B., and Belongie, S. J., “Feature pyramid networks for object detection,” CoRR abs/1612.03144 (2016).
  * [30] Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., and Myszkowski, K., [High dynamic range imaging: acquisition, display, and image-based lighting ], Morgan Kaufmann (2010).
  * [31] Deng, J., Guo, J., Yuxiang, Z., Yu, J., Kotsia, I., and Zafeiriou, S., “Retinaface: Single-stage dense face localisation in the wild,” in [arxiv ], (2019).
  * [32] Deng, J., Guo, J., Niannan, X., and Zafeiriou, S., “Arcface: Additive angular margin loss for deep face recognition,” in [CVPR ], (2019).