11institutetext: Department of Electronics and Communication Engineering,
ITER, Siksha ‘O’ Anusandhan (Deemed to be University), Bhubaneswar-751030,
Odisha, India
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>https://sites.google.com/site/erkundanec/home 22institutetext: Department of
Electrical Engineering, Sardar Vallabhbhai National Institute of Technology,
Surat, Gujarat-395007
22email<EMAIL_ADDRESS>
# Automated retinal vessel segmentation based on morphological preprocessing
and 2D-Gabor wavelets

Kundan Kumar corresponding author11 Debashisa Samal 11 Suraj 22

###### Abstract

Automated segmentation of vascular map in retinal images endeavors a potential
benefit in diagnostic procedure of different ocular diseases. In this paper,
we suggest a new unsupervised retinal blood vessel segmentation approach using
top-hat transformation, contrast-limited adaptive histogram equalization
(CLAHE), and 2-D Gabor wavelet filters. Initially, retinal image is
preprocessed using top-hat morphological transformation followed by CLAHE to
enhance only the blood vessel pixels in the presence of exudates, optic disc,
and fovea. Then, multiscale 2-D Gabor wavelet filters are applied on
preprocessed image for better representation of thick and thin blood vessels
located at different orientations. The efficacy of the presented algorithm is
assessed on publicly available DRIVE database with manually labeled images. On
DRIVE database, we achieve an average accuracy of 94.32% with a small standard
deviation of 0.004. In comparison with major algorithms, our algorithm
produces better performance concerning the accuracy, sensitivity, and kappa
agreement.

###### Keywords:

Retinopathy, Blood vasculature, Retinal vessel segmentation, 2D-Gabor wavelet,
Top-hat transform.

##  1 Introduction

Change in anatomical structure of retinal blood vessels (vasculature) in
retina is a good indication of the presence of ophthalmic diseases, e.g.,
hypertension, cardiovascular diseases, diabetic retinopathy, glaucoma, etc.
[1, 2]. Vascular map segmentation of fundus images has played a decisive role
in assessing the change in the vasculature for severity of ocular diseases.
However, periodic screening of retinal images for early recognition of the
change in vascular structure can prevent major vision loss [2]. Therefore, an
automatic and accurate retinal vessel segmentation is the prerequisite for the
initial diagnosis of retinal diseases. Extraction of the vascular map from an
uneven illuminated and pigmented fundus image is a challenging problem.
Besides retinal blood vessels, the presence of other structures (e.g.,
exudates, optic disc, fovea, red lesions) under uneven illuminated and
pigmented background makes the vessel detection even more difficult. Also, the
retinal vessel thickness varies in the wide range whereas thin vessels have
low contrast which makes thin vessel detection more challenging [2].

Several, automatic vascular map segmentation of retinal fundus images have
been proposed in literature [3, 4, 5, 6, 7]. Fraz et al. have done a
comprehensive literature survey on retinal blood vessel segmentation in [8].
The retinal blood vessel segmentation techniques are popularly classified as:
(i) supervised and (ii) unsupervised techniques. In supervised category,
$k$-NN-classifier [9], Artificial Neural Network (ANN) [10], trainable COSFIER
filters [11], Support Vector Machine (SVM) [12], Extreme Learning Machine
(ELM) [13], Deep Neural Network (DNN) [6, 14], etc. have been explored for
blood vessel segmentation as identification problem. However, supervised
algorithms rely on the robust feature extraction followed by classification.
In many approaches line detector [12], Gabor filters [1, 10], and gray level
co-occurrence matrix (GLCM) [15] based methods have been explored for feature
extraction purpose. A feature vector is computed for each pixel using these
approaches and classifier classify the pixels as the vessel and non-vessel
pixels. These approaches are time-consuming process due to training. On the
contrary, in the unsupervised category, filtering methods [3, 5, 16],
vasculature tracing methods [17], curvelet based [18], morphological operators
[19], have been used. In these approaches, classification of the vessel or
non-vessel pixels is performed without training process, i.e., training data
do not contribute to finding the model parameter.

In earlier reported works under the unsupervised category, match filter has
received the enormous response of the scholars due to its straightforwardness
in the implementation of the technique. Match filter based retinal vessel
segmentation relies on a 2D kernel with Gaussian profile initially proposed by
Chaudhuri et al. [3]. The kernel is rotated at $15^{o}$ increment, and the
best output of the filter for each pixel is carefully chosen to map all blood
vessels orientated at different angles. After that, thresholding is applied to
get binary vessel map image. Further, pruning is applied as post-processing to
improve the final identification of blood vessels. Hoover et al. [4] have used
local and region-based properties for vessel segmentation where threshold
probing technique is used on match filter response. However, match filter
gives a strong response in terms of vessels and non-vessels edges. Zhang et
al. [5] have exploited the first-order derivative of Gaussian to improve the
performance of matched filter by eliminating non-vessel edges from retinal
images. In literature [17][16], multiscale match filter and its variation are
suggested to identify blood vessels of different thickness. In [2], Zhao et
al. have enhanced the retinal vessels in retinal images by utilizing 2D-Gabor
wavelet filters and a contrast-limited adaptive histogram equalization
(CLAHE). After that, the processing results of region growing method and level
set approach are combined to get final segmentation as a binary image.
However, this approach is unable to remove non-vessel structures also takes
long processing time. Roychowdhury et al. [20] have performed an unsupervised
iterative process to obtain the vessels using top-hat reconstruction followed
by iterative region growing method. Most of the approaches like [5, 2] fail to
remove optic disc and exudates from pathological images. Also, many approaches
remove these structures in postprocessing with the extra burden of
computational time. Thus, an automated unsupervised blood vessel segmentation
approach is needed to identify the blood vessel pixels correctly with a small
false positive rate. Simultaneously, need to remove the anatomical structure
other than blood vessels with high accuracy and less complexity.

We propose an entirely unsupervised approach for automatic segmentation of
blood vessels to obtain the retinal vascular map. The significant contribution
of this paper is to use top-hat transform followed by CLAHE for retinal image
enhancement in preprocessing step. Use of top-hat transform facilitates to
enhance only the blood vessels, simultaneously remove the local intensity
change due to exudates, optic disc, and fovea from the background. For further
image enhancement, CLAHE is applied on top-hat transformed retinal image. The
preprocessed retinal image is passed through the Gabor filters bank, and the
maximum outcome of the filters are chosen for each pixel. Otsu thresholding as
global thresholding is applied to get a binary blood vessel structure. The
proposed technique is proficient in identifying the blood vessels under uneven
pigmentation and illuminance condition in the presence of exudates, optic
disc, and fovea. Our proposed approach suppresses the non-vessel pixels in the
preprocessing step, however multiscale Gabor wavelet filters efficiently
represent the thick and thin vessels in the retinal image. The efficacy of the
presented technique is validated on the publicly available Digital Retinal
Image for Vessels Extraction (DRIVE) database [9].

Rest of the paper is organized as follows. The DRIVE database and the proposed
algorithm are discussed in sections 2 and 3 respectively. The section 4
discusses the experimental results and finally, the paper is concluded in
section 5.

##  2 Materials

We validate our proposed algorithm on the DRIVE database [9] for performance
evaluation. The DRIVE database is publicly available to execute a comparative
study and experimental evaluation of vascular segmentation algorithms. The
gold standard segmented images are provided with the database as manually
labeled images. The DRIVE database contains 40 color retinal images which
include 20 images in training set and 20 images in the test set. All images
were captured by Canon CR5 nonmydriatic 3 charge-coupled-device (CCD) cameras
at $45^{o}$ field of view (FOV). Each color retinal image is having a
resolution of $565\times 584$ pixels with three R, G, and B channels, and each
channel is an 8-bit grayscale image. In this paper, we examined our presented
algorithm on images from the test set. For test set, two subsets of manually
segmented images, i.e., set A and set B, are provided. Set A as the first
observer’s manual segmented images are considered as gold standard which are
utilized as ground truth for performance assessment. The primary objective of
choosing DRIVE database is to perform a comparative analysis of the presented
work with the state-of-art techniques which have been evaluated on the same
database.

##  3 Proposed method

###  3.1 overview

In the presented work, an unsupervised blood vessel segmentation approach is
proposed. Fig. 1 presents the flow diagram of the proposed algorithm.

Figure 1: Flow diagram of the proposed algorithm

The principal idea of the proposed method is that in retinal image, vessels
can be distinguished from other structures like exudates, optic disc, fovea,
etc. during preprocessing. In many approaches, CLAHE is employed in
preprocessing to boost the dynamic range of the retinal images besides
preventing the over-amplification of noise [2]. However, CLAHE improves the
contrast of images by operating on local regions rather than globally due to
which vessel pixels enhance with the other structures too. Therefore, we first
applied the top-hat transformation on the green channel of color retinal
fundus image that intensifies only the blood vessel pixels and simultaneously
suppress the other structure pixels. After that, CLAHE is applied to get the
full advantage of its characteristics. In the second stage, 2D-Gabor wavelet
filter is applied to the preprocessed image. Furthermore, a global Otsu
thresholding method is used to get a binary segmented image.

###  3.2 Preprocessing

Initially, an original RGB color retinal image is split into three channels as
shown in Fig. 2.

Figure 2: Retinal image through FOV: (a) original RGB color image, (b) Red
channel, (c) green channel, and (d) blue channel.

Among these three channels, the green channel image appears having higher
contrast compared to other two channel images. In green channel image, the
blood vessel pixels are visible and easily distinguishable from the background
pixels. Because in our eyes, lens pigments absorb light colors differently
[21, 2]. Therefore, red vessels in color retinal image are more visible in the
green channel image as presented in Fig. 2, however, red channel image is the
brightest image and blue channel image suffers from poor dynamic range as
shown in Fig. 2.

Figure 3: Processing results at each intermediate steps of the proposed
algorithm. (a) Original test image, (b) Inverted green channel image, (c)
white top-hat transformed image of inverted green channel image, (d) CLAHE
processed image, (e) Gabor wavelet response, (f) Binary image after global
thresholding.

In the green channel image ($I_{G}$), the blood vessel pixels due to its
intensities being close to 0 seems to be dark. Image $I_{G}$ is inverted
followed by superposition of fundus mask ($M$) to make the blood vessel pixels
brighter and keep the focus on the region of interest (FOV). The inverted
green channel image through fundus mask is shown in Fig. 3. To enhance only
the vessels, white top-hat transformation is applied on inverted green channel
image ($I^{\prime}_{G}$). Usually, blood vessels have small thickness compared
to the other structures in retinal images. Therefore, a circular structuring
element of diameter at least equal to the diameter of the thickest blood
vessel is preferred for top-hat transformation. The white top-hat transform
can be defined as

$T_{w}(I^{\prime}_{G})=I^{\prime}_{G}-I^{\prime}_{G}\circ b.$ (1)

Where $T_{w}$ is the transformed image of $I^{\prime}_{G}$ using structuring
element $b$, and $\circ$ denote the morphological opening operator. In image
processing, the top-hat transformation is a morphological operation which
highlights the object smaller than the structuring element [22]. The diameter
of the structuring element is chosen 11 pixels wide as the maximum width of
the blood vessel is less than 11 pixels. For the DRIVE database, the width of
the blood vessel varies in the range of 1-10 pixels [23]. The diameter of the
structuring element may vary for the different database having different image
resolution.

Usually, the width of widest blood vessels is less than the width of other
structures, like exudates, fovea, and optic disc, which do not appear in the
top-hat transformed image ($T_{w}$) as shown in Fig. 3. Also, a homogeneous
background is obtained in the transformed image. After applying the top-hat
transformation, the blood vessel pixels do not achieve a good contrast
compared to the background. Therefore, (CLAHE) technique is adopted for
further enhancement of the processed retinal image. Fig 3 shows the CLAHE
response ($I_{c}$) of the top-hat transformed image (Fig. 3). However, CLAHE
also enhances the background noise. For informative representation of the
blood vessels having different thickness and orientation, retinal image is
processed through the multiscale Gabor wavelet filters at different
frequencies and orientations. Gabor wavelet also smoothes the background
noise.

###  3.3 2D-Gabor wavelet filter bank

The 2D-Gabor wavelet transformation is a tool for a complete representation of
an image in terms of radial frequency and orientation [24]. To highlight the
blood vessels of different width placed at different orientations in this
work, we used a bank of 2D-Gabor wavelet filters to the preprocessed retinal
image. Daugman et al. [25] have proposed that ensemble of a simple cell of
visual cortex can be represented as a family of 2D-Gabor wavelets. The
decomposition of an image, $f=I_{c}$, as wavelet transform is defined as

$({T^{wav}}f)(a,\theta,{x_{0}},{y_{0}})={\left\|a\right\|^{-1}}\iint{dxdyf(x,y){\psi_{\theta}}\left({\frac{{x-{x_{0}}}}{a},\frac{{y-{y_{0}}}}{a}}\right)},$
(2)

where $\theta$ is the orientation parameter of the wavelet, and $a$ is the
parameter that defines the standard deviation in $x$ and $y$ directions. In
equation (2),

${\psi_{\theta}}(a,x,y,{x_{0}},{y_{0}})={\left\|a\right\|^{-1}}{\psi_{\theta}}\left({\frac{{x-{x_{0}}}}{a},\frac{{y-{y_{0}}}}{a}}\right)$
(3)

represents the elementary function of the 2D wavelet rotated by an angle
$\theta$. Using the Gabor elementary function, the entire family of Gabor
wavelets can be generated. Lee et al. [24] have derived a specific class of
2D-Gabor wavelets which is used in this paper to obtain a set of Gabor
wavelets to process the retinal images. The Gabor wavelet which satisfies the
neurophysiological restraint of simple cells is defined as

$\displaystyle\psi(x,y,{\omega_{0}},\theta,K)=$
$\displaystyle\frac{{{\omega_{0}}}}{{\sqrt{2\pi}K}}\exp\left({-\frac{{\omega_{0}^{2}}}{{8{K^{2}}}}\left({4{{(xcos\theta+y\sin\theta)}^{2}}+{{(-xsin\theta+ycos\theta)}^{2}}}\right)}\right)$
(4)
$\displaystyle\cdot\left[{\exp\left\\{{i{\omega_{0}}\left({xcos\theta+y\sin\theta}\right)}\right\\}-\exp\left({-\tfrac{{{K^{2}}}}{2}}\right)}\right]$

where, $\theta$ denotes the wavelet orientation in radians and $\omega_{0}$ is
the radial frequency in radian per unit length. The constant $K$ tells about
the frequency bandwidth of octave where $K=\pi$ is for a frequency bandwidth
of one octave and $K\approx 2.5$ for frequency bandwidth of 1.5 octaves. Each
Gabor wavelet filter at radial frequency $\omega_{0}$ and orientation $\theta$
is centered at $(x=0,y=0)$ and normalized by $L^{2}$ norm.

For each pixel in the retinal image, maximum Gabor wavelet outcome over all
possible filters is stored for filtered image. If we consider
$T_{\psi}(\omega_{0},\theta,K)$ as the transformed retinal image at angular
frequency $\omega_{0}$ and orientation $\theta$. Then Gabor wavelet
transformation result is obtained as

$P(K)=\mathop{max}\limits_{{\omega_{0}},\theta}\left|{{T_{\psi}}({\omega_{0}},\theta,K)}\right|$
(5)

where $\theta$ is varied in the range of $[0,180)$ at an equal interval of
$20^{\circ}$. The radial frequency is varied between $[0.7,1.5)$ at an
interval of $0.2$. Lee at el.[24] have suggest to choose $K$ in the range of
[2,2.5]. In this paper, $K=2.2$ is selected to accomplish better
distinguishability between background and vessels. All these parameters are
selected by performing few experiments on retinal images and Gabor wavelet
filters. Fig. 4 shows few Gabor wavelet filters from the bank of filters for
different radial frequency at orientation of $50^{o}$. The Gabor wavelet
filter at $\omega_{0}=0.7$ is efficient to detect thick blood vessel. However,
$\omega_{0}=1.3$ is suitable to detect thin blood vessels. The Gabor wavelet
response of the preprocessed image is shown in Fig. 3.

(a) $\omega_{0}=0.7$

(b) $\omega_{0}=0.9$

(c) $\omega_{0}=1.1$

(d) $\omega_{0}=1.3$

Figure 4: Gabor wavelet filters at $50^{o}$ orientation for four different
radial frequencies

The hard segmented binary image as illustrated in Fig. 3 is obtained by
applying Otsu thresholding on the filtered image $P$.

##  4 Results and Discussions

The presented algorithm is evaluated and compared with competitive algorithms
using five different metrics: accuracy ($Acc$), sensitivity ($Se$),
specificity ($Sp$), kappa agreement ($\kappa$), and area under the curve
($A_{z}$). All the metrics are computed using only pixels inside the FOV.
Accuracy, sensitivity, and specificity are calculated using false positive
($X$), false negative ($Y$), true positive ($Z$), and true negative ($W$)
values. $Z$ denotes the number of pixels correctly identified as vessel
pixels, and $X$ denotes the number of pixels belongs to the background but
wrongly identified as vessel pixel. $W$ represents the number of pixels
correctly identified as background pixels, and $Y$ represents the number of
pixels belongs to the vessel but incorrectly assigned to background pixels.
The evaluation metrics are measured using the following mathematical
expressions as

$Acc=\frac{Z+W}{Z+X+Y+W},~{}~{}~{}~{}~{}~{}Se=\frac{Z}{Z+Y},$ (6)
$Sp=\frac{W}{X+W},~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\kappa=\frac{p_{o}-p_{e}}{1-p_{e}}.$
(7)

Where $p_{e}$ is the hypothetical probability of chance agreement and $p_{o}$
is the relative perceived agreement. These agreement values can be calculated
using the perceived data to estimate the probabilities of each observer
randomly seeing all the classes. $\kappa$ value varies in the range of
$[0,1]$, where $\kappa=0$ relates to no agreement between two rates, whereas
$\kappa=1$ relates to complete agreement between the rates. To compute
$A_{z}$, receiver operating characteristic (ROC) curve is attained by changing
the global threshold between 1 and 0 in steps of 0.01. The global threshold
divides the image into a binary image with labels 0 and 1. For each threshold,
two performance measure false positive rate ($XR=1-Sp$) and true positive rate
($ZR=Sp$) are obtained by comparing the segmented binary image with the
corresponding ground truth. Before applying the global threshold, the Gabor
wavelet response values are normalized to 0-1 range.

The blood vessel segmentation outcome performances are shown in Fig. 3. All
intermediate results and segmented output of the presented algorithm for a
retinal image from DRIVE database are illustrated in Fig. 3. The segmentation
performance of the presented algorithm on the DRIVE test data set is listed in
Table 1.

Table 1: Segmented outcome of the presented work on the DRIVE test database Image |  Se |  Sp |  Acc |  $A_{z}$ |  $\kappa$
---|---|---|---|---|---
1 |  0.8337 |  0.9592 |  0.9427 |  0.9687 |  0.7593
2 |  0.7781 |  0.9767 |  0.9469 |  0.9602 |  0.7837
3 |  0.7583 |  0.9639 |  0.9339 |  0.9432 |  0.7312
4 |  0.6619 |  0.9874 |  0.9440 |  0.9443 |  0.7282
5 |  0.7010 |  0.9837 |  0.9453 |  0.9468 |  0.7460
6 |  0.6975 |  0.9784 |  0.9388 |  0.9371 |  0.7279
7 |  0.7068 |  0.9737 |  0.9383 |  0.9431 |  0.7171
8 |  0.7111 |  0.9721 |  0.9393 |  0.9395 |  0.7120
9 |  0.7289 |  0.9771 |  0.9480 |  0.9525 |  0.7378
10 |  0.7021 |  0.9816 |  0.9483 |  0.9507 |  0.7354
11 |  0.7378 |  0.9660 |  0.9365 |  0.9412 |  0.7142
12 |  0.7936 |  0.9618 |  0.9408 |  0.9517 |  0.7364
13 |  0.6907 |  0.9798 |  0.9388 |  0.9481 |  0.7273
14 |  0.8094 |  0.9612 |  0.9433 |  0.9581 |  0.7386
15 |  0.7800 |  0.9672 |  0.9477 |  0.9566 |  0.7268
16 |  0.7225 |  0.9787 |  0.9451 |  0.9632 |  0.7440
17 |  0.7440 |  0.9684 |  0.9407 |  0.9451 |  0.7221
18 |  0.7920 |  0.9629 |  0.9432 |  0.9613 |  0.7300
19 |  0.8476 |  0.9713 |  0.9564 |  0.9729 |  0.7991
20 |  0.8080 |  0.9623 |  0.9459 |  0.9638 |  0.7306
Average |  0.7503 |  0.9717 |  0.9432 |  0.9524 |  0.7374
Std. deviation |  0.0499 |  0.0081 |  0.0049 |  0.0098 |  0.0206
Se - sensitivity, Sp - specificity, Acc - accuracy, $A_{z}$ \- Area under the
ROC curve, $\kappa$ \- Kappa agreement

The performance metrics illustrate that average segmentation accuracy,
sensitivity, and specificity of the presented technique are 94.32%, 75.03%,
and 97.12% respectively. It can be noticed that a small deviation in accuracy
($\sigma=0.0049$) is achieved. In DRIVE test data set, image 8 is a
pathological image having exudates on which we have achieved 93.93 % accuracy
with 0.7111 $ZR$. However, Zhao et al. [2] have reported 93.59 % of accuracy
with 0.6524 $ZR$ on the same image. The segmented blood vessels of image 8 is
presented in Fig. 5. It confirms that our proposed algorithm is efficient in
segmenting the blood vessels in pathological images also. Average area under
the curve, $A_{z}$, of 0.9524 is obtained with maximum and minimum $A_{z}$ as
0.9729 and 0.9371 respectively. On DRIVE test data set, we achieved average
kappa agreement 0.7374 with 0.0206 standard deviations.

A comparative analysis of the presented algorithm with competing methods is
presented in Table 2 in terms of average accuracy, sensitivity, specificity,
$A_{z}$, and $\kappa$ agreement. The evaluation metrics values of different
methods provided in the table are taken from the respective papers. Numerical
results show that our proposed algorithm outperforms many unsupervised methods
to DRIVE dataset concerning accuracy and sensitivity. Furthermore, the
proposed algorithm is better than few supervised techniques on DRIVE test
dataset concerning sensitivity and specificity.

Table 2: Comparative analysis of the presented work with existing approach on the DRIVE test data set with respect to golden standard ground truth image Methods |  Se |  Sp |  Acc ($\sigma$) |  $A_{z}$ |  $\kappa$
---|---|---|---|---|---
Supervised Methods
Soares et al. [1] |  0.7230 |  0.9762 |  0.9466 (-) |  0.9614 |  -
Staal et al. [9] |  0.7194 |  0.9773 |  0.9441 (-) |  0.9520 |  -
Ricci et al. [12] |  - |  - |  0.9595 (-) |  0.9558 |  -
Lahiri et al. [26] |  - |  - |  0.9530 (0.0030) |  - |  0.7090
Unsupervised Methods
2nd observer |  0.7760 |  0.9725 |  0.9473 (-) |  - |  0.6970
Chaudhuri et al.[3]∗ |  - |  - |  0.8773 (0.0232) |  0.7878 |  0.3357
Zana et al. [27]∗ |  - |  - |  0.9377 (0.0077) |  0.8984 |  0.6971
Martinez-Parez et al. [28] |  0.7246 |  0.9655 |  0.9344 (-) |  - |  -
Zhang et al. [5] |  0.7120 |  0.9724 |  0.9382 (-) |  - |  -
Miri et.al. [18] |  0.7352 |  0.9795 |  0.9458 (-) |  - |  -
Zhao et al. [2] |  0.7354 |  0.9789 |  0.9477 (-) |  - |  -
Gou et al. [7] |  0.7526 |  0.9669 |  0.9393 (-) |  - |  -
Presented method |  0.7503 |  0.9717 |  0.9432 (0.0049) |  0.9524 |  0.7374
∗results are taken from [29]

Fig. 5 shows the segmentation result of four retinal vessel images selected
from the DRIVE test data sets. Among these four retinal images, three retinal
images (Fig. 5(a)-(c)) are healthy retinal images with optic disc and fovea.
Whereas, Fig. 5 is the pathological image having exudates, optic disc, and
fovea. It can be observed that our proposed algorithm is competent in
identifying thin as well as thick vessels in the presence of exudates, fovea,
and optic disc. Besides, the proposed algorithm also preserves the
connectivity of the blood vessels. The segmentation performance can be further
improved by using some local adaptive thresholding technique instead of global
thresholding to detect 1-2 pixel thick vessels.

Figure 5: Segmentation result of retinal images from the DRIVE database. First
row: original image, Second row: golden standard ground truth, Third row:
segmented vessel using the proposed algorithm. (a-c) Healthy retinal image
with different pigmentation and illuminance, (d) Pathological retinal image.

##  5 Conclusions

Blood vessel segmentation in fundus images plays a vital role to detect
different ocular diseases. To distinguish blood vessel from other structure
like exudates, optic disc, fovea, etc., under uneven illuminance condition is
difficult. In this paper, we have been proposed an entirely unsupervised
approach for blood vessel segmentation using the retinal image and validated
on DRIVE database. In preprocessing, retinal image is enhanced in the two-
steps. In the first step, white top-hat transform is applied to enhance only
the blood vessels and concurrently suppress all other structures which are
larger than structuring element. In next step, the processed image is further
enhanced using CLAHE algorithm. After preprocessing, multiscale Gabor wavelet
filters are applied to emphasize the thick and thin vessels in the retinal
image. A global threshold is obtained from the filtered image by using Otsu’s
thresholding technique. Experimental results clearly indicate that the
proposed technique is efficient to recognize the blood vessels in the presence
of exudates, fovea, and optic disc with the average accuracy of 94.32% with a
small standard deviation of 0.0049. Moreover, the suggested algorithm in this
paper is simple and easy to implement.

The outcome result of the proposed technique depends on the diameter selected
as structuring element for the top-hat transform. If some other structures
(like small microaneurysms) which look like vessels and have a thickness less
than the width of the structuring element then our proposed method may fail to
remove those structures in the segmented image. In future, the shape feature
can be incorporated with Gabor wavelet response to discarding such kind of the
small structures. Furthermore, testing is to be performed on different retinal
databases like STARE and CHASE_DB1 to test the robustness of the proposed
method.

## References

  * [1] Soares, J.V., Leandro, J.J., Cesar, R.M., Jelinek, H.F., Cree, M.J.: Retinal vessel segmentation using the 2-D Gabor wavelet and supervised classification. IEEE Transactions on medical Imaging 25(9), 1214–1222 (2006)
  * [2] Zhao, Y.Q., Wang, X.H., Wang, X.F., Shih, F.Y.: Retinal vessels segmentation based on level set and region growing. Pattern Recognition 47(7), 2437–2446 (2014)
  * [3] Chaudhuri, S., Chatterjee, S., Katz, N., Nelson, M., Goldbaum, M.: Detection of blood vessels in retinal images using two-dimensional matched filters. IEEE Transactions on medical imaging 8(3), 263–269 (1989)
  * [4] Hoover, A., Kouznetsova, V., Goldbaum, M.: Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Transactions on Medical imaging 19(3), 203–210 (2000)
  * [5] Zhang, B., Zhang, L., Zhang, L., Karray, F.: Retinal vessel extraction by matched filter with first-order derivative of Gaussian. Computers in biology and medicine 40(4), 438–445 (2010)
  * [6] Liskowski, P., Krawiec, K.: Segmenting retinal blood vessels with deep neural networks. IEEE transactions on medical imaging 35(11), 2369–2380 (2016)
  * [7] Gou, D., Wei, Y., Fu, H., Yan, N.: Retinal vessel extraction using dynamic multi-scale matched filtering and dynamic threshold processing based on histogram fitting. Machine Vision and Applications 29(4), 655–666 (2018)
  * [8] Fraz, M.M., Remagnino, P., Hoppe, A., Uyyanonvara, B., Rudnicka, A.R., Owen, C.G., Barman, S.A.: Blood vessel segmentation methodologies in retinal images–a survey. Computer methods and programs in biomedicine 108(1), 407–433 (2012)
  * [9] Staal, J., Abràmoff, M.D., Niemeijer, M., Viergever, M.A., Van Ginneken, B.: Ridge-based vessel segmentation in color images of the retina. IEEE transactions on medical imaging 23(4), 501–509 (2004)
  * [10] Franklin, S.W., Rajan, S.E.: Retinal vessel segmentation employing ANN technique by Gabor and moment invariants-based features. Applied Soft Computing 22, 94–100 (2014)
  * [11] Azzopardi, G., Strisciuglio, N., Vento, M., Petkov, N.: Trainable COSFIRE filters for vessel delineation with application to retinal images. Medical image analysis 19(1), 46–57 (2015)
  * [12] Ricci, E., Perfetti, R.: Retinal blood vessel segmentation using line operators and support vector classification. IEEE transactions on medical imaging 26(10), 1357–1365 (2007)
  * [13] Zhu, C., Zou, B., Zhao, R., Cui, J., Duan, X., Chen, Z., Liang, Y.: Retinal vessel segmentation in colour fundus images using extreme learning machine. Computerized Medical Imaging and Graphics 55, 68–77 (2017)
  * [14] Sadek, I., Elawady, M., Shabayek, A.E.R.: Automatic classification of bright retinal lesions via deep network features. arXiv preprint arXiv:1707.02022 (2017)
  * [15] Rahebi, J., Hardalaç, F.: Retinal blood vessel segmentation with neural network by using gray-level co-occurrence matrix-based features. Journal of medical systems 38(8), 85 (2014)
  * [16] Li, Q., You, J., Zhang, D.: Vessel segmentation and width estimation in retinal images using multiscale production of matched filter responses. Expert Systems with Applications 39(9), 7600–7610 (2012)
  * [17] Sofka, M., Stewart, C.V.: Retinal vessel centerline extraction using multiscale matched filters, confidence and edge measures. IEEE transactions on medical imaging 25(12), 1531–1546 (2006)
  * [18] Miri, M.S., Mahloojifar, A.: Retinal image analysis using curvelet transform and multistructure elements morphology by reconstruction. IEEE Transactions on Biomedical Engineering 58(5), 1183–1192 (2011)
  * [19] Hassan, G., El-Bendary, N., Hassanien, A.E., Fahmy, A., Snasel, V., et al.: Retinal blood vessel segmentation approach based on mathematical morphology. Procedia Computer Science 65, 612–622 (2015)
  * [20] Roychowdhury, S., Koozekanani, D.D., Parhi, K.K.: Iterative vessel segmentation of fundus images. IEEE Transactions on Biomedical Engineering 62(7), 1738–1749 (2015)
  * [21] Walter, T., Massin, P., Erginay, A., Ordonez, R., Jeulin, C., Klein, J.C.: Automatic detection of microaneurysms in color fundus images. Medical image analysis 11(6), 555–566 (2007)
  * [22] Dougherty, E.R., Lotufo, R.A.: Hands-on morphological image processing, vol. 59. SPIE press (2003)
  * [23] Fathi, A., Naghsh-Nilchi, A.R.: Automatic wavelet-based retinal blood vessels segmentation and vessel diameter estimation. Biomedical Signal Processing and Control 8(1), 71–80 (2013)
  * [24] Lee, T.S.: Image representation using 2D Gabor wavelets. IEEE Transactions on pattern analysis and machine intelligence 18(10), 959–971 (1996)
  * [25] Daugman, J.G.: Complete discrete 2D Gabor transforms by neural networks for image analysis and compression. IEEE Transactions on acoustics, speech, and signal processing 36(7), 1169–1179 (1988)
  * [26] Lahiri, A., Roy, A.G., Sheet, D., Biswas, P.K.: Deep neural ensemble for retinal vessel segmentation in fundus images towards achieving label-free angiography. In: Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th Annual International Conference of the. pp. 1340–1343. IEEE (2016)
  * [27] Zana, F., Klein, J.C.: Segmentation of vessel-like patterns using mathematical morphology and curvature evaluation. IEEE transactions on image processing 10(7), 1010–1019 (2001)
  * [28] Martinez-Perez, M.E., Hughes, A.D., Thom, S.A., Bharath, A.A., Parker, K.H.: Segmentation of blood vessels from red-free and fluorescein retinal images. Medical image analysis 11(1), 47–61 (2007)
  * [29] Niemeijer, M., Staal, J., van Ginneken, B., Loog, M., Abramoff, M.D.: Comparative study of retinal vessel segmentation methods on a new publicly available database. In: Medical Imaging 2004: Image Processing. vol. 5370, pp. 648–657. International Society for Optics and Photonics (2004)


11institutetext: ImViA EA7535, University Burgundy Franche-Comté, 21078 Dijon,
France 11email<EMAIL_ADDRESS>
# CEN-HDR: Computationally Efficient neural Network for real-time High Dynamic
Range imaging

Steven Tel  0000-0002-1487-9381 Barthélémy Heyrman  0000-0003-1642-8311
Dominique Ginhac  0000-0002-5911-2010

###### Abstract

High dynamic range (HDR) imaging is still a challenging task in modern digital
photography. Recent research proposes solutions that provide high-quality
acquisition but at the cost of a very large number of operations and a slow
inference time that prevent the implementation of these solutions on
lightweight real-time systems. In this paper, we propose CEN-HDR, a new
computationally efficient neural network by providing a novel architecture
based on a light attention mechanism and sub-pixel convolution operations for
real-time HDR imaging. We also provide an efficient training scheme by
applying network compression using knowledge distillation. We performed
extensive qualitative and quantitative comparisons to show that our approach
produces competitive results in image quality while being faster than state-
of-the-art solutions, allowing it to be practically deployed under real-time
constraints. Experimental results show our method obtains a score of 43.04
$\mu$-PSNR on the Kalantari2017 dataset with a framerate of 33 FPS using a
Macbook M1 NPU. The proposed network will be available at
https://github.com/steven-tel/CEN-HDR

###### Keywords:

High Dynamic Range Imaging, Efficient computational photography

##  1 Introduction

In the last decades, applications based on computer vision have become
increasingly important in everyday life. Currently, many research works are
conducted to propose more reliable algorithms in areas such as object
detection, action recognition, or scene understanding. However, the accuracy
of these algorithms depends largely on the quality of the acquired images.
Most standard cameras are unable to faithfully reproduce the illuminations
range of a natural scene, as the limitations of their sensors generate a loss
of structural or textural information in under-exposed and over-exposed
regions of the acquired scene. To tackle this challenge, sensors with a higher
dynamic range (HDR) have been proposed [17, 26] to capture more intensity
levels of the scene illumination, but these solutions are expensive,
preventing high dynamic range acquisition from being readily available.

Towards making HDR imaging practical and accessible, software solutions were
proposed, based on the emergence of deep learning in computer vision
applications. They acquire one Low Dynamic Range (LDR) image and try to
expends its dynamic range thanks to a generative adversarial network [12, 37,
8]. Although these methods produce images with a higher illumination range,
they have limitations in extending the dynamic of the input image to the
dynamic of the acquired scene. A more effective approach is to acquire
multiple LDR images with different exposure times and merge them into one
final HDR image. Traditional computer vision algorithms [3] allow the
acquisition of good-quality static scenes when there is no camera or object
motion between images with different exposure times. However, in a lot of use
cases, images are captured in a rapid sequence from a hand-held device
resulting in inevitable misalignments between low dynamic range shots.
Therefore scenes with motions introduce new challenges as ghost-like artifacts
for large motion regions or loss of details in occluded regions. Following
recent advances in the field of deep learning, several methods based on
Convolutional Neural Network (CNN) were proposed to spatially align input
frames to a reference one before merging them into a final HDR image.

State-of-the-art deep learning solutions for multi-frame merging HDR [34, 13]
tend to be based on a previously proposed method[30] and add additional
processing to increase the accuracy of the HDR merging system. As a
consequence, the computational cost and execution time are significantly
increased, preventing these solutions to be used in lightweight systems and/or
in real-time applications. Then, the primary goal of HDR imaging software
solutions, which was to make HDR imaging more widely available compared to
hardware solutions, is therefore not being achieved at all. Therefore, in this
paper, we propose a Computationally Efficient neural Network for High Dynamic
Range imaging (CEN-HDR). CEN-HDR is based on an encoder-decoder neural network
architecture for generating ghost-free HDR images from scenes with large
foreground and camera movements. Unlike previously published solutions, we
decided to develop a new approach keeping in mind the constraint of inference
time and computational cost.

  1. 1.

We propose CEN-HDR a novel efficient convolutional neural network based on a
new attention mechanism and sub-pixel convolution that overcomes ghost-like
artifacts and occluded regions while keeping a low computational cost,
allowing our solution to be implemented in real-time on a lightweight system.

  2. 2.

We demonstrate the efficiency of network compression for the realization of
CEN-HDR by applying a knowledge distillation scheme.

  3. 3.

We perform extensive experiments to determine the best trade-off between
accuracy and inference cost with the main objective to demonstrate the
relevance of CEN-HDR.

##  2 Related Works

We briefly summarize existing HDR merging approaches into two categories: deep
learning-based architectures and efficient learning-based architectures. In
the first category, the proposed methods aim to achieve better quality in HDR
imaging without taking inference cost into account. Approaches belonging to
the second category seek to optimize the compromise between the quality of the
generated images and the computation cost. This leads to the use of new
operators in the proposed deep learning architectures.

Figure 1: Architecture of the proposed CEN-HDR solution. The spatial size of
input features is divided by 2 at the encoding step. The attention module
allows registering non-reference features to the reference ones. The full
spatial size is recovered thanks to the pixel shuffle operation.

###  2.1 Deep learning based HDR merging

Using multiple input images for HDR generation leads to the need to align the
features of the LDR images to the reference image. The first common method for
feature registration is by computing the motion between inputs features using
an optical flow algorithm. Multiple studies [31, 7, 22] used Liu[11] optical
flow algorithm. In Kalantari et al.[7], the input images are aligned by
selecting the image with better pixels as a reference and computing the
optical flow between this reference and other input LDR images. Then, the
warped images are fed to a supervised convolutional neural network (CNN) to
merge them into an HDR image. However, since the optical flow algorithm
initially assumes that the input images have the same exposure time, trying to
warp the different exposures with occluded regions can result in artifacts in
the final HDR image. To address this issue, DeepHDR[28] proposes an image
translation network able to hallucinate information in the occluded regions
without the need for optical flow. Moreover, many solutions have been
developed to correct the ghost effect introduced by the misalignment of the
input images. In AHDRNet[30] an attention module is proposed to emphasize the
alignment of the input images to the reference image. Input images are then
merged using several dilated residual dense blocks. The high performance of
the AHDRNet network led it to be used as a base network for other methods. For
example, ADNet[13] follows the same main architecture that AHDRNet but adds a
Pyramidal alignment module based on deformable convolution allowing a better
representation of the extracted features but in the counterpart of a larger
number of operations.

###  2.2 Efficient learning-based HDR merging architectures

To our best knowledge, the first architecture which aims to be efficient was
proposed by Prabhakar et al.[21] by processing low-resolution images and
upscaling the result to the original full resolution thanks to a bilateral
guided upsampling module. Recently, the HDR community tends to focus more on
efficient HDR image generation[20] and no longer only aims at improving the
image quality but also at significantly limiting the number of processing
operations. This results in efficient solutions such as GSANet[9] that propose
efficient ways to process gamma projections of input images with spatial and
channel attention blocks to increase image quality while limiting the number
of parameters. Yu et al.[36] introduce a multi-frequency lightweight encoding
module to extract features and a progressive dilated u-shape block for
features merging. Moreover, the different standard convolution operations are
replaced by depth-wise separable convolution operations firstly proposed in
[2], they are composed of a depth-wise convolution followed by a pointwise
convolution which allows more efficient use of model parameters. Another
efficient method, proposed by Yan et al.[33] is a lightweight network based on
an u-net[23] like encoder-decoder architecture, allowing for spatially reduced
processed features. While these solutions focus on the number of performed
operations, their inference time still remains too long to be considered as
real-time solutions.

##  3 Proposed Method

We consider three LDR images $I_{i}\in\mathbb{R}^{3\times H\times W}$ with
their respective exposure times $t_{i}$ as inputs. The generated HDR image is
spatially aligned with the central LDR frame $I_{2}$ selected as the reference
image. To make our solution more robust to exposure difference between inputs,
the respective projection of each LDR input frame into the HDR domain is
calculated using the gamma encoding function described in Eq. 1, following
previous works [13, 18, 30]:

.

$H_{i}=\frac{I_{i}^{\gamma}}{t_{i}},\quad\gamma=2,2$ (1)

Where $H_{i}\in\mathbb{R}^{3\times H\times W}$ is the gamma-projected input.
Then, each input is concatenated with their respective gamma-projection to
obtain $L_{i}\in\mathbb{R}^{6\times H\times W}$ :

$L_{i}=I_{i}\oplus H_{i}$ (2)

where $\oplus$ represents the concatenation operation. $L_{i}$ will then be
fed to our proposed merging network whose architecture is detailed in Fig. 1.

###  3.1 Feature encoding

Using high-resolution images as inputs presents an additional challenge in the
design of a real-time HDR merging network. To solve such a problem, previous
works [29, 33] propose to use a U-net[23] like architecture to reduce the
spatial size of the features processed by the merging network. However, a too
large reduction of the spatial dimensions causes the extraction of coarse
features that degrade the final result. So, we decide to limit the spatial
reduction to 2 by using an encoder block composed of 2 sequential convolutions
as described in Eq. 3.

$F_{i}=conv_{E_{1}}(conv_{E_{2}}(L_{i}))$ (3)

where $F_{i}\in\mathbb{R}^{32\times\frac{H}{2}\times\frac{W}{2}}$ is the
features map extracted from the encoder for each LDR input. $conv_{E_{1}}$ and
$conv_{E_{2}}$ are 3x3-convolution layers extracting respectively 16 and 32
features map. The spatial size is divided by 2 setting a stride of 2 for
$conv_{2}$.

Figure 2: Illustration of the attention module composed of 2 branches,
respectively responsible for spatial attention and channel attention. A
sigmoid activation function is used to keep the value between 0 and 1.

###  3.2 Attention module

The final generated HDR image must be aligned with the reference image. To
address this requirement [30] demonstrates the effectiveness of using a
spatial attention module after the encoding step. Since then, many spatial and
channel attention modules have been proposed in the literature which can be
integrated into networks to improve their performance. According to the
inference cost study done in Table 5, we propose the Spatial-Channel Reference
Attention Module (SCRAM) a slightly modified version of the Bottleneck
Attention Module (BAM) proposed in [19]. Indeed while BAM aims to generate a
mask of its input feature maps, in our case we want to generate attention maps
from the concatenation of reference and non-reference features, these
attention maps are then applied to non-reference features only resulting in a
reduction of the number of feature maps in the proposed attention module.
Moreover, batch normalization is not applied in SCRAM. The detailed structure
of SCRAM is illustrated in Fig. 2.

The non-reference features $F_{i\neq 2}$ are concatenated with the features of
the reference image:

$X_{i}=F_{i}\oplus F_{2=ref},\quad i\neq 2$ (4)

where $X_{i}\in\mathbb{R}^{64\times\frac{H}{2}\times\frac{W}{2}}$ is the input
of SCRAM and $\oplus$ is the concatenation operator.

Following [19], SCRAM is composed of two branches respectively responsible for
the spatial and channel features alignment of the non-reference images to the
reference ones:

$A_{i}=\sigma(s(X_{i})+c(X_{i})),\quad i\neq 2$ (5)

where $s$ is the spatial attention branch and $c$ is the channel attention
branch. The sum of produced features by each branch passes through $\sigma$, a
sigmoid activation function to keep the output values between 0 and 1.

Spatial attention: The objective of the spatial branch is to produce an
attention map allowing to keep the most relevant information for the spatial
alignment of the non-reference images to the reference one. To limit the
computation, we first reduce the number of features map by 3 using a pointwise
convolution. With the objective to extract more global features while keeping
the same computation, we then make the receptive field larger by employing 3
dilated convolutions [35] layers with a factor of dilatation set to 2. The
final attention map of size $(1,H,W)$ is produced by using a pointwise
convolution and then expanded across the channel dimension to obtain
$A_{S}\in\mathbb{R}^{32\times H\times W}$.

Channel attention: This branch aims to perform a channel-wise feature
recalibration. We first squeeze the spatial dimension by applying a global
average pooling which sums out the spatial information to obtain the features
vector of size $(64,1,1)$. A multilayer perceptron with three hidden layers is
then used in purpose to estimate cross-channel attention. The last activation
size is set to 32 to fit the number of channels of the non-reference features
$F_{i}$. Finally, the resulting vector map is spatially expanded to obtain the
final feature map $A_{C}\in\mathbb{R}^{32\times H\times W}$.

The Attention features $A_{i}$ are then used to weight the non-reference
features $F_{i}$:

$F^{\prime}_{i}=F_{i}\otimes A_{i},\quad i\neq 2$ (6)

where $\otimes$ is the element-wise product and $F^{\prime}_{i}$ is the
aligned non-reference features. For the reference features we set
$F^{\prime}_{2}=F_{2}$.

###  3.3 Features merging

While most of the computation is usually done in the merging block [13, 18,
30], we propose a novel efficient feature merging block that first focuses on
merging the non-references features. Each feature map $F^{\prime}_{i}$
produced by the encoder goes through a convolution layer:

$M_{i}=conv_{M_{1}}(F^{\prime}_{i})$ (7)

Where $conv_{M_{1}}$ is a 3x3-convolution producing
$M_{i}\in\mathbb{R}^{64\times\frac{H}{2}\times\frac{W}{2}}$. Then we focus on
merging the non-reference features maps by concatenating them and feeding the
result features in a convolution layer:

$M_{\text{non-ref}}=conv_{M_{2}}(M_{1}\oplus M_{3})$ (8)

where $conv_{M_{2}}$ is a 3x3-convolution and $M_{\text{non-
ref}}\in\mathbb{R}^{64\times\frac{H}{2}\times\frac{W}{2}}$ is the non-
reference merged features.

As we emphasize the reference features throughout all our network, here we
merge our reference features with the non-reference $M_{non-ref}$ only by
adding them together to limit the number of features map processed later:

$M=conv_{M_{4}}(conv_{M_{3}}(M_{2}+M_{non-ref}))$ (9)

where $conv_{M_{3}}$ and $conv_{M_{4}}$ are 3x3-convolutions producing each 64
features map and $M\in\mathbb{R}^{64\times\frac{H}{2}\times\frac{W}{2}}$
contains features from all LDR input images.

Figure 3: Illustration of the pixel rearrangement by the pixel shuffle layer
for an upscale factor set to $r=2$ and an input shape of $(4,4,4)$. In the
proposed solution, the input shape is $(64,\frac{H}{2},\frac{W}{2})$, the
produced output size is $(16,H,W)$.

###  3.4 Features decoding

The role of the decoder is to produce the final HDR image from the features
produced by the merger block. At the encoding stage, we divided the spatial
dimensions by 2. While the original spatial size is usually recovered using
bilinear upsampling or transposed convolution[14] operation, we propose to use
the pixel shuffle operation first proposed in [25], it is presented as an
efficient sub-pixel convolution with a stride of $1/r$ where $r$ is the
upscale factor. In our case, we set $r=2$. As illustrated in Fig. 3, the pixel
shuffle layer rearranges elements in a tensor of shape $(C\times r^{2},H,W)$
to a tensor of shape $(C,H\times r,W\times r)$:

$D=PixelShuffle(M)$ (10)

where $D\in\mathbb{R}^{16\times H\times W}$ is the resulting upscaled
features.

The final HDR image is obtained by following the Eq 11.

$HDR=\sigma(conv_{D}(D+S_{2}))$ (11)

where $S_{2}$ is the reference features extracted by the first convolution
layer of our network $conv_{E_{1}}$ to stabilize the training of our network.
Finally, we generate the final HDR image using a 3x3-convolution layer,
followed by a Sigmoid activation function.

##  4 Experimental Settings

###  4.1 Datasets

The CEN-HDR network has been trained using the dataset provided by [7]
composed of 74 training samples and 15 test samples. Each sample represents
the acquisition of a dynamic scene caused by large foreground or camera
motions and is composed of three input LDR images (with EV of -2.00, 0.00,
+2.00 or -3.00, 0.00, +3.00) and a reference HDR image aligned with the medium
exposure image. The network has also been separately trained and tested using
the dataset from the NTIRE[20] dataset, where the 3 LDR images are
synthetically generated from the HDR images provided by [4]. The dataset is
composed of 1500 training samples, 60 validation samples, and 201 testing
samples. The ground-truth images for the testing sample are not provided.

Figure 4: Qualitative comparison of the proposed CEN-HDR solution with other
HDR merging methods. The cropped patch demonstrates that the proposed
efficient network has equivalent capabilities to the state-of-the-art methods
to correct the ghost effect due to the large movement in the scene.

###  4.2 Loss function

Following previous works [7, 30], the images have been mapped from the linear
HDR domain to the LDR domain before evaluating the loss function. In order to
train the network, the tone-mapping function has to be differentiable around
zero, so, the $\mu$-law function is defined as follows:

$T(H)=\frac{log(1+\mu H)}{log(1+\mu)},\quad\mu=5000$ (12)

where H is the linear HDR image and $\mu$ the amount of compression.

To make an efficient network, a network compression method defined as
knowledge distillation proposed in [5] has been used. By using knowledge
distillation, we assume that the capacity of a large network is not fully
exploited, so the objective is to transfer the knowledge of this large teacher
network to our lighter network as described in the Eq. 13.

$\mathcal{L}=\alpha\times\mathcal{L}(T,T_{GT})+(1-\alpha)\times\mathcal{L}(T,T_{Teacher})$
(13)

where $\mathcal{L}$ is the $L_{1}$ loss function. $T$ is the tone mapped
prediction of our network, $T_{GT}$ the tone mapped ground truth provided in
the dataset and $T_{Teacher}$ the tone mapped prediction of the large teacher
network. we use the HDR-GAN [18] model as the teacher. $\alpha$ is a trade-off
parameter set to 0.2. Moreover, the method proposed by [7] to produce the
training dataset focuses mainly on foreground motion. It does not allow the
generation of reliable ground-truth images for chaotic motions in the
background, such as the movement of tree leaves due to wind, which results in
a ground truth image with blurred features that do not reflect reality. This
has the effect of producing a greater error when the predicted image contains
sharper features than in the ground truth image. Using also a predicted image
from a teacher model allows for dealing with this data misalignment. The
comparison of the performance obtained between training done with knowledge
distillation and without is made in Table. 1.

###  4.3 Implementation details

The CEN-HDR network has been trained using cropped patches of size $256\times
256$ pixels with a stride of 128 and evaluated on the full-resolution test
images. During training, random augmentations are applied on the cropped patch
such as horizontal symmetry and rotation of 90, 180, or 270 degrees. Training
has been done using the Adam optimizer with a batch size of 8. The learning
rate is initially set to $10^{-4}$, keep fixed for 80 epochs, and decreased by
0.8 every 20 epochs after. The training lasts for 500 epochs.

Figure 5: Comparison of the proposed CEN-HDR solution with other HDR merging
methods. The X-axis represents the mean runtime using the M1 NPU with an input
size of 1280x720 pixels. The Y-axis is the fidelity score on the test images
from [7] dataset. The best solutions tend to be in the upper left corner. The
radius of circles represents the number of operations, the smaller the better.
Table 1: Comparison of the proposed CEN-HDR architecture performances with and
without knowledge distillation. In the first case, the network is trained with
the HDR ground-truth proposed in [7]. In the second case, we also use the
prediction of HDR-GAN [18] as label (Eq. 13). In both cases, the network is
trained for 500 epochs using the $l_{1}$ criterion.

Training method |  $\mu$-PSNR | PSNR |  $\mu$-SSIM | SSIM | HDR-VDP2
---|---|---|---|---|---
w/o knowledge distillation | 40.8983 | 40.0298 | 0.9772 | 0.9926 | 62.17
with knowledge distillation | 43.0470 | 40.5335 | 0.9908 | 0.9956 | 64.34

##  5 Experimental Results

Table 2: Quantitative comparison with lightweight state-of-the-art methods on
the Kalantari2017[7] test samples. PSNR and SSIM are calculated in the linear
domain while $\mu$-PSNR and $\mu$-SSIM are calculated after $\mu$-law tone
mapping (Eq. 12). For compared methods, the results are from [18]. PU-PSNR and
PU-SSIM are calculated applying the encoding function proposed in [16].

Method |  $\mu$-PSNR | PU-PSNR | PSNR |  $\mu$-SSIM | PU-SSIM | SSIM | HDR-VDP2
---|---|---|---|---|---|---|---
Sen et al.[24] | 40.80 | 32.47 | 38.11 | 0.9808 | 0.9775 | 0.9721 | 59.38
Hu et al.[6] | 35.79 | $-$ | 30.76 | 0.9717 | $-$ | 0.9503 | 57.05
Kalantari et al.[7] | 42.67 | 33.82 | 41.23 | 0.9888 | 0.9832 | 0.9846 | 65.05
DeepHDR[28] | 41.65 | 31.36 | 40.88 | 0.9860 | 0.9815 | 0.9858 | 64.90
NHDRRNet[32] | 42.41 | $-$ | $-$ | 0.9887 | $-$ | $-$ | 61.21
AHDRNet[30] | 43.61 | 33.94 | 41.03 | 0.9900 | 0.9855 | 0.9702 | 64.61
HDRGAN[18] | 43.92 | 34.04 | 41.57 | 0.9905 | 0.9851 | 0.9865 | 65.45
CEN-HDR(our) | 43.05 | 33.23 | 40.53 | 0.9908 | 0.9821 | 0.9856 | 64.34

###  5.1 Fidelity performance

In Table 2 and Fig. 4, the proposed CEN-HDR solution is compared against seven
lightweight state-of-the-art methods: [24] and [6] are based on input patch
registration methods. [7] is based on a sequential CNN, the inputs need first
to be aligned thanks to an optical flow algorithm. For [28], the background of
each LDR input is aligned by homography before being fed to an encoder-
decoder-based CNN. [32] proposes an encoder-decoder architecture with a non-
local attention module. [30] is a CNN based on an attention block for features
registration and on multiple dilated residual dense blocks for merging. [18]
is the first GAN-based approach for HDR merging with a deep supervised HDR
method. Quantitative evaluation is done using objective metrics. The standard
peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are
computed both directly in the linear domain and after tone mapping by applying
the $\mu$-law function (Eq. 12). The HDR-VDP2[15] metric predicts the quality
degradation with the respect to the reference image. We set the diagonal
display size to 24 inches and the viewing distance to 0.5 meter. In addition,
PU-PSNR and PU-SSIM are calculated by applying the encoding function proposed
in [16] with a peak luminance set to 4000.

In Fig. 6, we present the results for 3 test scenes of the NTIRE[20] dataset.
While the network can produce images with a high dynamic range, we notice that
the high sensor noise present in the input images is not fully corrected in
the dark areas of the output HDR images. Moreover, the motion blur introduced
in the dataset produces less sharp characteristics in the output image.

In Table 6 we study the effect of the attention module on the performance of
proposed HDR deghosting architecture. SCRAM-C and SCRAM-S respectively
correspond to the SCRAM module composed only of the channel attention branch
and the spatial attention branch. The proposed SCRAM allows to achieve a
similar quality as the spatial attention module proposed in AHDRNet[28], while
Table 5 shows that the inference cost of SCRAM is lower.

###  5.2 Efficiency comparison

As we want to propose an efficient HDR generation method, in Table 3 we
compare the computation cost and the inference time of our network with state-
of-the-art HDR networks that achieve similar performance on quality metrics.

The number of operations and parameters are measured using the script provided
by the NTIRE[20] challenge. To evaluate runtimes, all the compared networks
are executed on the Neural Processing Unit (NPU) of a MacBook Pro (2021)
powered with an M1 chip. The time shown is the average for 500 inference runs
after a warm-up of 50 runs. The input size is set to $1280\times 720$ pixels.
The gamma-projection of LDR inputs (Eq. 1) and the tone mapping of the HDR
output are included in the inference time measurement. Note that the
background alignment of inputs frame using homography for [28] is not
included.

Table 3: Inference cost comparison of the proposed CEN-HDR solution against state-of-the-art lightweight deep learning-based methods. The number of operations and parameters are measured using the script provided by the NTIRE[20] challenge. To measure the inference time, all the compared networks are executed on an M1 NPU. The input size is set to $1280\times 720$ pixels. Method | Num. of params. | Num. of op. (GMAccs) | Runtime(s) | FPS
---|---|---|---|---
DeepHDR[28] | 14618755 | 843.16 | 0.1075 | 9.30
AHDRNet[30] | 1441283 | 1334.95 | 0.4571 | 2.18
NHDRRNet[32] | 7672649 | 166.11 | 0.0431 | 23.20
HDR-GAN[18] | 2631011 | 479.78 | 0.2414 | 4.14
CEN-HDR(our) | 282883 | 78.36 | 0.0277 | 36.38

Table 4 compares the number of parameters and operations of the proposed CEN-
HDR solution with recent efficient methods [9, 33, 36]. The input size is set
to $1900\times 1060$ pixels corresponding to the size of the inputs from the
dataset proposed by [20]. For compared methods [9, 33, 36] the measurements
are provided by the NTIRE[20] challenge. We could not compare the inference
time of the CEN-HDR solution with these three architectures as they were
recently proposed and their implementation is not yet available.

Fig. 5 compares the trade-off between fidelity to the ground truth label and
runtime of the proposed CEN-HDR solution with other HDR merging methods. The
X-axis represents the mean runtime using an M1 NPU with an input size of
1280x720 pixels. The Y-axis is the fidelity score on the test images from [7]
dataset. The best solutions tend to be in the upper left corner. The radius of
circles represents the number of operations. Our solution is shown as the best
solution for real-time HDR merging with a high-fidelity score.

Table 4: Inference cost comparison of the proposed CEN-HDR solution versus recent efficient merging networks. The number of operations and parameters for [9, 33, 36] and our solution are computed following the method described in [20].The input size is set to $1900\times 1060$ pixels. Method | Num. of params. | Num. of op. (GMAccs)
---|---|---
GSANet[9] | 80650 | 199.39
Yan et al.[33] | 18899000 | 156.12
Yu et al.[36] | 1013250 | 199.88
CEN-HDR(our) | 282883 | 128.78
Table 5: Inference cost comparison of attention modules. Spatial and Channel
attention modules are studied by feeding a tensor of size
$(1,\frac{H}{4},\frac{W}{4})$ corresponding to the concatenation of the
reference and non-reference tensors after the encoding step.

Method | Attention type | params. | GMAccs | Runtime(s)
---|---|---|---|---
| Spatial | Channel |  |  |
AHDRNet attention [30] | ✓ |  | 55392 | 20.772 | 0.0085
EPSANet[38] | ✓ |  | 42560 | 15.768 | 0.0111
SK attention [10] |  | ✓ | 125984 | 43.104 | 0.0155
Double attention[1] | ✓ | ✓ | 33216 | 12.456 | 0.0101
CBAM[27] | ✓ | ✓ | 22689 | 7.525 | 0.0734
BAM[19] | ✓ | ✓ | 17348 | 5.008 | 0.0060

Table 6: Effect of the attention module on the performance of proposed HDR
deghosting network. SCRAM-C and SCRAM-S respectively correspond to the SCRAM
module composed only of the channel attention branch and the spatial attention
branch.

Method |  $\mu$-PSNR | PSNR |  $\mu$-SSIM | SSIM
---|---|---|---|---
Without attention module | 42.12 | 39.95 | 0.9850 | 0.9823
AHDRNet[30] attention module | 42.94 | 40.49 | 0.9903 | 0.9852
SCRAM-C | 42.32 | 40.14 | 0.9854 | 0.9829
SCRAM-S | 42.89 | 40.41 | 0.9884 | 0.9835
SCRAM | 43.05 | 40.53 | 0.9908 | 0.9856

Figure 6: Qualitative results of the proposed CEN-HDR solution on samples from
the NTIRE[20] challenge dataset. The ground truth images are not provided. We
notice that the high sensor noise present in the input images is not fully
corrected in the dark areas of the output HDR images. Moreover, the motion
blur introduced in the dataset produces less sharp characteristics in the
output image.

##  6 Conclusions

In this paper, we propose CEN-HDR, a novel computationally efficient HDR
merging network able to correct the ghost effect caused by large object
motions in the scene and camera motion. The proposed lightweight network
architecture effectively succeeds in generating real-time HDR images with a
dynamic range close to that of the original scene. By integrating the
knowledge distillation methods in our training scheme, we demonstrate that the
majority of the representation capabilities of a large HDR merging network can
be transferred into a lighter network, opening the door to real-time HDR
embedded systems.

## References

  * [1] Chen, Y., Kalantidis, Y., Li, J., Yan, S., Feng, J.: $a^{2}$-nets: Double attention networks (2018). https://doi.org/10.48550/ARXIV.1810.11579, https://arxiv.org/abs/1810.11579
  * [2] Chollet, F.: Xception: Deep learning with depthwise separable convolutions (2016). https://doi.org/10.48550/ARXIV.1610.02357, https://arxiv.org/abs/1610.02357
  * [3] Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. p. 369–378. SIGGRAPH ’97, ACM Press/Addison-Wesley Publishing Co., USA (1997). https://doi.org/10.1145/258734.258884, https://doi.org/10.1145/258734.258884
  * [4] Froehlich, J., Grandinetti, S., Eberhardt, B., Walter, S., Schilling, A., Brendel, H.: Creating cinematic wide gamut hdr-video for the evaluation of tone mapping operators and hdr-displays. Proc. SPIE 9023 (02 2014). https://doi.org/10.1117/12.2040003
  * [5] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015). https://doi.org/10.48550/ARXIV.1503.02531, https://arxiv.org/abs/1503.02531
  * [6] Hu, J., Gallo, O., Pulli, K., Sun, X.: Hdr deghosting: How to deal with saturation? In: 2013 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1163–1170 (2013). https://doi.org/10.1109/CVPR.2013.154
  * [7] Kalantari, N.K., Ramamoorthi, R.: Deep high dynamic range imaging of dynamic scenes. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017) 36(4) (2017)
  * [8] Khan, Z., Khanna, M., Raman, S.: Fhdr: Hdr image reconstruction from a single ldr image using feedback network (2019)
  * [9] Li, F., Gang, R., Li, C., Li, J., Ma, S., Liu, C., Cao, Y.: Gamma-enhanced spatial attention network for efficient high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 1032–1040 (June 2022)
  * [10] Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks (2019). https://doi.org/10.48550/ARXIV.1903.06586, https://arxiv.org/abs/1903.06586
  * [11] Liu, C.: Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Ph.D. thesis, MIT, USA (2009), aAI0822221
  * [12] Liu, Y.L., Lai, W.S., Chen, Y.S., Kao, Y.L., Yang, M.H., Chuang, Y.Y., Huang, J.B.: Single-image hdr reconstruction by learning to reverse the camera pipeline (2020)
  * [13] Liu, Z., Lin, W., Li, X., Rao, Q., Jiang, T., Han, M., Fan, H., Sun, J., Liu, S.: Adnet: Attention-guided deformable convolutional network for high dynamic range imaging (2021)
  * [14] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2015)
  * [15] Mantiuk, R., Kim, K.J., Rempel, A.G., Heidrich, W.: Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. In: ACM SIGGRAPH 2011 Papers. SIGGRAPH ’11, Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/1964921.1964935, https://doi.org/10.1145/1964921.1964935
  * [16] Mantiuk, R.K., Azimi, M.: Pu21: A novel perceptually uniform encoding for adapting existing quality metrics for hdr. In: 2021 Picture Coding Symposium (PCS). pp. 1–5 (2021). https://doi.org/10.1109/PCS50896.2021.9477471
  * [17] Nayar, S., Mitsunaga, T.: High dynamic range imaging: spatially varying pixel exposures. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662). vol. 1, pp. 472–479 vol.1 (2000). https://doi.org/10.1109/CVPR.2000.855857
  * [18] Niu, Y., Wu, J., Liu, W., Guo, W., Lau, R.W.H.: Hdr-gan: Hdr image reconstruction from multi-exposed ldr images with large motions. IEEE Transactions on Image Processing 30, 3885–3896 (2021). https://doi.org/10.1109/TIP.2021.3064433
  * [19] Park, J., Woo, S., Lee, J.Y., Kweon, I.S.: Bam: Bottleneck attention module (2018). https://doi.org/10.48550/ARXIV.1807.06514, https://arxiv.org/abs/1807.06514
  * [20] Pérez-Pellitero, E., Catley-Chandar, S., Shaw, R., Leonardis, A., Timofte, R., Zhang, Z., Liu, C., Peng, Y., Lin, Y., Yu, G., Zhang, J., Ma, Z., Wang, H., Chen, X., Wang, X., Wu, H., Liu, L., Dong, C., Zhou, J., Yan, Q., Zhang, S., Chen, W., Liu, Y., Zhang, Z., Zhang, Y., Shi, J.Q., Gong, D., Zhu, D., Sun, M., Chen, G., Hu, Y., Li, H., Zou, B., Liu, Z., Lin, W., Jiang, T., Jiang, C., Li, X., Han, M., Fan, H., Sun, J., Liu, S., Marín-Vega, J., Sloth, M., Schneider-Kamp, P., Röttger, R., Li, C., Bao, L., He, G., Xu, Z., Xu, L., Zhan, G., Sun, M., Wen, X., Li, J., Li, J., Li, J., Li, C., Li, C., Gang, R., Gang, R., Li, F., Li, F., Liu, C., Liu, C., Feng, S., Lei, F., Liu, R., Ruan, J., Dai, T., Li, W., Lu, Z., Liu, H., Huang, P., Ren, G., Luo, Y., Liu, C., Tu, Q., Ma, S., Cao, Y., Tel, S., Heyrman, B., Ginhac, D., Lee, C., Kim, G., Park, S., Vien, A.G., Mai, T.T.N., Yoon, H., Vo, T., Holston, A., Zaheer, S., Park, C.Y.: Ntire 2022 challenge on high dynamic range imaging: Methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 1009–1023 (June 2022)
  * [21] Prabhakar, K., Agrawal, S., Singh, D.K., Ashwath, B., Babu, R.V.: Towards practical and efficient high-resolution hdr deghosting with cnn. In: ECCV (2020)
  * [22] Prabhakar, K.R., Agrawal, S., Babu, R.V.: Self-gated memory recurrent network for efficient scalable HDR deghosting. CoRR abs/2112.13050 (2021), https://arxiv.org/abs/2112.13050
  * [23] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. CoRR abs/1505.04597 (2015), http://arxiv.org/abs/1505.04597
  * [24] Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shechtman, E.: Robust Patch-Based HDR Reconstruction of Dynamic Scenes. ACM Transactions on Graphics (TOG) (Proceedings of SIGGRAPH Asia 2012) 31(6), 203:1–203:11 (2012)
  * [25] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network (2016)
  * [26] Tumblin, J., Agrawal, A., Raskar, R.: Why i want a gradient camera. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). vol. 1, pp. 103–110 vol. 1 (2005). https://doi.org/10.1109/CVPR.2005.374
  * [27] Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module (2018). https://doi.org/10.48550/ARXIV.1807.06521, https://arxiv.org/abs/1807.06521
  * [28] Wu, S., Xu, J., Tai, Y., Tang, C.: End-to-end deep HDR imaging with large foreground motions. CoRR abs/1711.08937 (2017), http://arxiv.org/abs/1711.08937
  * [29] Wu, S., Xu, J., Tai, Y.W., Tang, C.K.: Deep high dynamic range imaging with large foreground motions. In: The European Conference on Computer Vision (ECCV) (September 2018)
  * [30] Yan, Q., Gong, D., Shi, Q., Hengel, A.v.d., Shen, C., Reid, I., Zhang, Y.: Attention-guided network for ghost-free high dynamic range imaging. IEEE Conference on Compute rVision and Pattern Recognition (CVPR) pp. 1751–1760 (2019)
  * [31] Yan, Q., Gong, D., Zhang, P., Shi, Q., Sun, J., Reid, I., Zhang, Y.: Multi-scale dense networks for deep high dynamic range imaging. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 41–50 (2019). https://doi.org/10.1109/WACV.2019.00012
  * [32] Yan, Q., Zhang, L., Liu, Y., Zhu, Y., Sun, J., Shi, Q., Zhang, Y.: Deep hdr imaging via a non-local network. IEEE Transactions on Image Processing 29, 4308–4322 (2020). https://doi.org/10.1109/TIP.2020.2971346
  * [33] Yan, Q., Zhang, S., Chen, W., Liu, Y., Zhang, Z., Zhang, Y., Shi, J.Q., Gong, D.: A lightweight network for high dynamic range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 824–832 (June 2022)
  * [34] Ye, Q., Xiao, J., Lam, K., Okatani, T.: Progressive and selective fusion network for high dynamic range imaging. CoRR abs/2108.08585 (2021), https://arxiv.org/abs/2108.08585
  * [35] Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions (2015). https://doi.org/10.48550/ARXIV.1511.07122, https://arxiv.org/abs/1511.07122
  * [36] Yu, G., Zhang, J., Ma, Z., Wang, H.: Efficient progressive high dynamic range image restoration via attention and alignment network (2022). https://doi.org/10.48550/ARXIV.2204.09213, https://arxiv.org/abs/2204.09213
  * [37] Zeng, H., Cai, J., Li, L., Cao, Z., Zhang, L.: Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence pp. 1–1 (2020). https://doi.org/10.1109/TPAMI.2020.3026740
  * [38] Zhang, H., Zu, K., Lu, J., Zou, Y., Meng, D.: Epsanet: An efficient pyramid squeeze attention block on convolutional neural network (2021). https://doi.org/10.48550/ARXIV.2105.14447, https://arxiv.org/abs/2105.14447