# Visibility Enhancement for Low-light Hazy Scenarios

Chaoqun Zhuang
Beihang University
Beijing
&Yunfei Liu
Beihang University
Beijing
&Sijia Wen
Beihang University
Beijing
&Feng Lu*
Beihang University
Beijing

###### Abstract

Low-light hazy scenes commonly appear at dusk and early morning. The visual
enhancement for low-light hazy images is an ill-posed problem. Even though
numerous methods have been proposed for image dehazing and low-light
enhancement respectively, simply integrating them cannot deliver pleasing
results for this particular task. In this paper, we present a novel method to
enhance visibility for low-light hazy scenarios. To handle this challenging
task, we propose two key techniques, namely cross-consistency dehazing-
enhancement framework and physically based simulation for low-light hazy
dataset. Specifically, the framework is designed for enhancing visibility of
the input image via fully utilizing the clues from different sub-tasks. The
simulation is designed for generating the dataset with ground-truths by the
proposed low-light hazy imaging model. The extensive experimental results show
that the proposed method outperforms the SOTA solutions on different metrics
including SSIM$\uparrow$ (9.19%) and PSNR$\uparrow$ (5.03%). In addition, we
conduct a user study on real images to demonstrate the effectiveness and
necessity of the proposed method by human visual perception.

_Keywords_ low-light hazy enhancement, deep learning

##  1 Introduction

Haze is a common atmosphere phenomenon, which can be an obstacle for many
computer vision tasks and multi-media applications. Though research has been
proposed to improve the image quality of such phenomenon, haze removal for
low-light scenarios is still a challenging problem. The hazy images captured
under low-light illumination especially suffer from poor visibility, reduced
contrasts, fainted surfaces, color shift, and tremendous noise. In addition,
images with low-light conditions are often unavoidable for multiple computer
vision tasks, such as video surveillance, autonomous driving, urban mapping,
etc. In this regard, visibility enhancement for low-light hazy scenes is
indispensable for different visual-based applications.

Conventional dehazing methods usually rely on the atmospheric scattering model
[1], which is defined as follows:

$I(x)=J(x)t(x)+A(1-t(x)),$ (1)

where $J$ denotes a haze-free scene radiance, $A$ describes the global
atmospheric light indicating the intensity of ambient light, $t$ is the
transmission map, and $x$ represents the pixel position. It notes that the
decomposition of this formulation depends on the $A$ and $t$, which leads to
coarse and inaccurate results under low-light illumination. As a result, the
general dehazing methods cannot perform well for image dehazing under low-
light illumination. Similarly, the former image enhancement methods are often
based on the classic Retinex theory models [2]. The physical imaging model
under low-light illumination can be expressed as:

$J(x)=R(x)L(x)+\mathcal{N},$ (2)

where $R$ is the reflectance, $L$ is illumination and $\mathcal{N}$ is the
illumination-related noise caused by camera intrinsic sensors. In this regard,
the visibility enhancement for low-light hazy scenes can be considered as an
entangled problem, which is hard to decouple.

In this paper, we propose a cross-consistency dehazing-enhancement framework
to solve the aforementioned problems. The proposed method focuses on
integrating dehazing inference and enhancement inference for low-light hazy
scenes simultaneously. Concretely, we apply the multi-level attention-guided
mappings to obtain an implicit dehazing / enhancement inference. To jointly
utilize these learned mappings, we design a cross-consistency framework to
integrate these inferences without affecting each other. As illustrated in
Fig. 1, the haze is removed properly and the result images are enhanced with
high visibility.

Since the paired training data for low-light hazy scenes is impractical to
collect, we further propose a synthetic visibility enhancement dataset. In
order to simulate the visually realistic and physically reliable dataset, we
make fine-tuning on the integration of the hazy imaging model and the low-
light imaging model. Moreover, we consider the noise introduced by the sensor
under low-light illumination.

The contribution of this work can be summarized as the following: 1) We first
explore the low-light hazy scenes and propose a visibility enhancement method
for this particularly difficult task. 2) We develop a cross-consistency
dehazing-enhancement framework to improve the visibility of low-light hazy
scenarios. 3) We propose a physical-based simulation strategy to construct the
low-light hazy dataset. The simulated visibility enhancement dataset consists
of 8970 images. We also capture 200 real low-light hazy scenes for
verification purpose. 4) We evaluate the proposed method on the simulated
dataset and the real scenes. Both experimental results prove that our solution
outperforms the state-of-the-art methods.

Figure 1: The proposed method can deliver pleasing visibility enhancement
result for low-light hazy scene.

##  2 Related Work

Focusing on visibility enhancement for low-light hazy scenes, the related
methods can be generally grouped into three categories: image dehazing, low-
light image enhancement, and image dehazing in the dark.

Image dehazing. Due to the importance of image dehazing, a large number of
related researches have been provided. In the beginning, some methods [3, 4]
treat image dehazing as an image quality enhancement task and improve the
contrast to restore a clear image by removing the noise. Based on the
atmospheric scattering model [1], some methods [5, 6, 7, 8, 9] perform image
dehazing by complying with the physical law. The most representative work is
that He et al. [5] proposed an inventive dark channel prior to estimating the
transmission map for image dehazing.

Since deep learning has made great progress in recent years, the training-
based network is naturally utilized in the area of image dehazing. Cai et al.
[6] propose an end-to-end training model to directly estimate the transmission
map, which can improve the quality of dehazing results. To improve the
effectiveness of the network, Qin et al. [10] propose a feature fusion
attention network, which combines channel attention and pixel attention
mechanism. Dong [11] newly propose a network with dense feature fusion based
on the U-Net architecture. Shao et al. [12], Liu et al. [13] and Chen et al.
[14] focus on real hazy data, and try to improve the performance of dehazing
on real data. Zheng et al. [15] proposed a multi-guide bilateral learning
framework for 4K resolution image dehazing. Sun et al. [16] explore the effect
of the semi-supervised method in the field of image dehazing. There are also
some researches [9, 17] that try to explore real-time dehazing models.

Although these methods can remove haze from general hazy images, due to the
physical complexity of low-light hazy scenes, the image dehazing cannot be
performed correctly in low-light environments. In addition, since the haze is
highly related to the depth information of the scene, these methods cannot
deliver pleasing dehazing results.

Low-light image enhancement. The existent methods [3, 18, 2, 19] try to
address this issue through statistical assumptions and computational
optimization. In recent years, Shen et al. [20] firstly develop an end-to-end
multi-scale Retinex convolutional network to achieve low-light image
enhancement. Lv et al. [21] further propose a multi-branch low-light image
enhancement network to remove the artifacts and noise in dark areas. Zhang et
al. [22] decompose the image into brightness and reflection parts. The former
part is used to adjust the light flexibly, and the latter part is used to deal
with the noise and color distortion. Guo et al. [23] convert this task into a
specific curve estimation problem. Mohit et al. [24] try to deal with this
problem in an extremely dark scenario and achieve it at a real-time level. Yet
these methods will be seriously disturbed by the haze in the dark environment,
which amplifies the noise during the image enhancement.

Image dehazing in the dark. By considering the artificial light sources at
night as the ambient light, some methods develop the night dehazing model for
image dehazing in the dark. These methods normally solved this problem by
setting some priors [25, 26, 27] or simulating the night hazy images [28].
However, the expected results of these nighttime dehazing methods cannot
deliver the fine details in terms of visual perception.

In summary, none of the above-mentioned methods can effectively improve the
image quality of low-light hazy scenes. In this paper, we focus on not only
the dehazing results but also the visibility enhancement for low-light hazy
images.

##  3 Proposed Method

###  3.1 Problem Analysis

Since the low-light hazy images comply with a complex physical imaging model,
the decomposition of such a model is the key to tackling this ill-posed
problem. To begin with, we first formulate the task with precise denotations.
Specifically, we denote $I$ as the well-exposed hazy-free image. $I_{L}$ is
the low-light image without haze. $I^{H}$ is the hazy image under normal
exposure. $I^{H}_{L}$ is the hazy low-light image. Our goal is to learn a
mapping $\mathcal{S}(\cdot,\theta)$ to recover the well-exposed haze-free
image $I$ from the under-exposed hazy image $I^{H}_{L}$, where $\theta$
denotes the learnable parameters for implicit inference.

We firstly propose a low-light enhancement mapping
$\mathcal{E}(\cdot,\theta)$, which converts the images captured under low-
light illumination to well-exposed images with rich details. In addition, we
denote $\mathcal{D}(\cdot,\theta)$ as a dehazing map for haze removal.
Apparently, there are two paths to obtain the well-exposed haze-free image $I$
from $I^{H}_{L}$.

Figure 2: Overview of our cross-consistency dehazing-enhancement framework.

  1. 1.

$LLE\rightarrow Dehaze$. Applying the low-light enhancement mapping
$\mathcal{E}$ on the low-light hazy image $I^{H}_{L}$ can deliver an
intermediate result $I^{H}$, which should be a well-exposed image with haze
residual. Then we adopt the dehazing mapping $\mathcal{D}$ for $I^{H}$ to
obtain the final results. Formally, the forward process is
$I=\mathcal{D}(I^{H},\theta)$, where $I^{H}=\mathcal{E}(I^{H}_{L},\theta)$.

  2. 2.

$Dehaze\rightarrow LLE$. Similarly, we operate the dehazing first, then
perform low-light enhancement. In this regard, the forward process is
$I=\mathcal{E}(I_{L},\theta)$, where $I_{L}=\mathcal{D}(I^{H}_{L},\theta)$,
where the intermediate $I_{L}$ is the under-exposed image without haze.

To better address this problem, we introduce an order-invariant assumption.
From a physical perspective, the sequence of the two natural phenomena of low
light and haze will not impact the imaging. From a mathematical perspective,
the order of dehazing and enhancement theoretically does not affect the
results. Therefore, we assume that the computational process of low-light
enhancement and dehazing is order-invariant, which is proved by our
experimental results in Table 4. Based on this assumption, we propose the
mapping $\mathcal{S}$ to jointly utilize $\mathcal{E}$ and $\mathcal{D}$ with
customized constraints. The computational process can be formulated as:

$\begin{matrix}\mbox{Input image\;}I^{H}_{L}\\\
{\color[rgb]{0.75390625,0,0}\mbox{\, Dim \&
Hazy}}\end{matrix}\begin{array}[]{l}{\nearrow}\\\ \\\
{\searrow}\end{array}\begin{array}[]{l}{\stackrel{{\scriptstyle\mathcal{E}(\cdot,\theta)}}{{\longrightarrow}}I^{H}\stackrel{{\scriptstyle\mathcal{D}(\cdot,\theta)}}{{\longrightarrow}}I_{e-d}}\\\
{\color[rgb]{0.75390625,0,0}Path\,I:LLE\rightarrow Dehaze}\\\ \\\
{\stackrel{{\scriptstyle\mathcal{D}(\cdot,\theta)}}{{\longrightarrow}}I_{L}\stackrel{{\scriptstyle\mathcal{E}(\cdot,\theta)}}{{\longrightarrow}}I_{d-e}}\\\
{\color[rgb]{0.75390625,0,0}Path\,II:Dehaze\rightarrow
LLE}\end{array}\begin{array}[]{l}{\searrow}\\\ \\\
{\nearrow}\end{array}\begin{matrix}\mathcal{F}(\cdot,\theta)\\\
\longrightarrow\\\
{\color[rgb]{0.75390625,0,0}\mbox{Fusion}}\end{matrix}\begin{matrix}\mbox{Output
image\;}I_{final}\,,\\\ {\color[rgb]{0.75390625,0,0}\mbox{\, High-
quality}}\end{matrix}$ (3)

where $I_{e-d}$ is the output of first low-light enhancement then dehaze path,
and $I_{d-e}$ is the output of first dehaze then low-light enhancement path.
We then fuse the two results using a mapping $\mathcal{F}(\cdot,\theta)$ to
get the final result.

###  3.2 Framework

As mentioned above, we propose a cross-consistency dehazing-enhancement
framework, which consists of pre-tuned dehazing and enhancement mappings.
Essentially, these two mappings need to be capable of inferring clean/well-
exposed images from hazy/low-light images. For this purpose, each mapping is
constructed in two parts: 1) a rich-encoder for multi-level feature extraction
and 2) an attention-guided decoder.

Rich-encoder. Based on Resnet-18 [29], we extract multi-level features by
gathering intermediate output from each stage of the backbone to preserve more
detailed information. Then the encoder produces mixed features from different
levels of the encoding process. The mixed features contain rich information,
which can help for a more effective final inference.

Attention-guided decoder. The decoder’s purpose is to map the learned features
from high-dimensional space to RGB space. Specifically, given the input
features, the decoder first aggregates them by bilinear-resizing each of the
features to the same size as the input image, and then channel-wisely
concatenates them together. Inspired by the attention mechanism [30, 31, 32],
we apply the channel-spatial attention [31] for the context between channels
and multi-scale context within channels.

Figure 3: Architecture of the CABF (channel-attention-based fusion block).
This block fuses two different path’s results to get a high-quality final
result.

Cross-consistency dehazing-enhancement architecture. After obtaining these two
mappings, we apply a unified architecture to jointly use these two mappings
with customized constraints. An overview of the proposed method is illustrated
in Fig. 2. According to the order-invariant assumption, we want to jointly use
these two mappings for performance improvement without affecting each other.
To perfectly utilize the consistency of these two paths, we share the
parameters of these mappings, which have the same purpose in different paths.
We use $\mathcal{S-D}(\cdot,\theta)$ and $\mathcal{S-E}(\cdot,\theta)$ to
represent that the two mappings share parameters in Fig. 2. Then, we design
the CABF block to fuse the results of these two paths.

Channel-Attention-Based Fusion block. To achieve better image quality, we
carefully design a channel-attention-based fusion block(CABF) to combine the
results from two paths. The underlying design principle is to introduce
channel-wise context for the final output. As shown in Fig. 3, the fusion
block takes two images as input, i.e., $I_{e-d}$ and $I_{d-e}$. Through
utilizing the imperceptible clues from two inputs, it fuses them in latent
space and generates the refined image $I_{final}$. More specifically, the CABF
block first concatenates two images in the channel dimension. Then it enhances
the local receptive field via a stack of CNNs. Next, we adopt SE-layer to
adaptively predict the relative importance of each channel. The overall
structure is within a residual block, without increasing the parameter count.
Finally, the output layer is a convolutional layer with a ReLU activation for
reconstructing the refined image.

Based on the proposed architecture, the dehazing mapping can infer clean
images from the hazy ones under both low-light and normal-exposed conditions.
Similarly, the enhancement mapping is capable of bridging the gap between low-
light images and normal ones with dense haze. In other words, such shared
parameters can make the mapping focus on their own task and discard the
interference of other irrelevant factors. Our framework implements logical
two-path processing using two basic mappings and a fusion block.

Loss functions. We apply different loss functions for these two types of
mappings. For Dehazing mapping, the loss function is

$\mathcal{L}_{\mathcal{D}}(\mathbf{x},\hat{\mathbf{x}})=|\mathbf{x}-\hat{\mathbf{x}}|_{1}+|\mathbf{x}-\hat{\mathbf{x}}|_{2}+|\nabla\mathbf{x}-\nabla\hat{\mathbf{x}}|_{1},$
(4)

where $\hat{\mathbf{x}}$ and $\mathbf{x}$ are prediction and ground-truth,
correspondingly. $\nabla$ denotes the gradient operation, which is used as a
constraint for the edge of the output.

For Enhancement mapping, the loss function is

$\mathcal{L}_{\mathcal{E}}(\mathbf{x},\hat{\mathbf{x}})=|\mathbf{x}-\hat{\mathbf{x}}|_{1}+|\mathbf{x}-\hat{\mathbf{x}}|_{2}+\mathcal{L}_{exp}(\hat{\mathbf{x}}),$
(5)

where $L_{exp}(\cdot)$ is a self-supervised loss for controlling the exposure
level [33]. This loss leverages the distance between the average intensity
value of a local region to a well-exposed level $\delta$. We use the same
settings as the existing work [23] in our experiment. The $L_{exp}$ is
expressed as:

$\mathcal{L}_{exp}=\frac{1}{M}\sum^{M}_{k=1}|y_{k}-\delta|,$ (6)

where $M$ denotes the number of non-overlapping patches of 16$\times$16,
$y_{k}$ is the average value of k-th patch in the predicted image
$\hat{\mathbf{x}}$.

Moreover, we add a path-invariant loss $L_{pi}$ to keep the outputs from two
paths consistent, i.e.,

$\mathcal{L}_{pi}=|\mathcal{E}(\mathcal{D}(I_{L}^{H}))-\mathcal{D}(\mathcal{E}(I_{L}^{H}))|.$
(7)

Finally, the total loss of our proposed method is

$\mathcal{L}_{total}=\lambda_{11}\mathcal{L}_{\mathcal{E}11}+\lambda_{12}\mathcal{L}_{\mathcal{D}12}+\lambda_{21}\mathcal{L}_{\mathcal{D}21}+\lambda_{22}\mathcal{L}_{\mathcal{E}22}+\lambda_{3}\mathcal{L}_{pi},$
(8)

where $\lambda_{11}$, $\lambda_{12}$, $\lambda_{21}$, $\lambda_{22}$ and
$\lambda_{3}$ are tunable parameters that make the total loss flexible.

##  4 Simulation and Dataset

###  4.1 Low-light Haze Simulation

Figure 4: Pipeline of our data simulation. (a) Clear image. (b) Simulated low-
light image. (c) Simulated low-light hazy image. (d) Simulated haze-only
image. (e) Simulated low-light-only image with noise. (f) Our final low-light
hazy simulation.

It should be noted that suitable datasets for our motivation do not exist.
Moreover, there is no long-range sensor to directly collect the visibility
enhancement training pair. Therefore, we propose a physical-based simulation
strategy to collect the first visibility enhancement dataset for low-light
hazy scenes. As illustrated in Eq. 1, the global atmospheric light $A$ relies
on the intensity of the illumination. Therefore, to simulate the haze under
low-light illumination, we should render the low-light environment first, then
simulate the haze. The pipeline of the simulation can be generally summarized
into three phases: 1) low-light rendering; 2) haze generation; 3) noise
simulation, as shown in Fig. 4.

Low-light rendering. To render the low-light haze in terms of visual
perception and physical constraints, we utilize the Retinex theory [34] for
the low-light imaging model:

$I(x)=R(x)L(x),\quad I_{low}(x)=R(x)L_{low}(x)+\mathcal{N},$ (9)

where $R$ stands for reflection, $L$ stands for illumination. $\mathcal{N}$ is
the unbalanced illumination distribution and the noise caused by camera
intrinsic sensors. Afterwards, we can get the relationship between normal
image $I$ and the corresponding low-light image $I_{low}$ as following:

$I_{low}(x)=I(x)\frac{L_{low}(x)}{L(x)}+\mathcal{N}=I(x)\Delta
L(x)+\mathcal{N},$ (10)

where $\Delta L(x)$ represents the various lightness on objects caused by the
low-light illumination. In conclusion, $\Delta L(x)$ is the key parameter to
render the low-light imagery.

Since just setting a prior value for $\Delta L(x)$ cannot simulate the complex
scenarios with low-light illumination, we introduce auxiliary variables to
reformulate this delicate parameter. Based on the analysis of the imaging
model with different exposure levels and other low-light image synthetic
methods [35, 36], we use a combination of linear adjustment and gamma
adjustment to fit the function of $\Delta L(x)$. In this case, Eq. 10 can be
reformulated as:

$I_{low}(x)=\beta\ (\alpha\ I(x))^{\gamma}+\mathcal{N},$ (11)

where $\alpha$, $\beta$ and $\gamma$ are parameters for adjusting exposure
levels to simulate the under-exposed image. After numerous analytical tests on
real scenes with different levels of exposure, we randomly set these three
parameters from the certain value ranges: $\alpha\in[0.9,1]$,
$\beta\in[0.5,0.7]$ and $\gamma\in[1.5,2.5]$.

Haze generation. After obtaining the low-light data, inspired by RESIDE [37],
we can generate the haze as following:

$I^{hazy}_{low}(x)=I_{low}(x)t(x)+A(1-t(x)),$ (12)

where $I^{hazy}_{low}(x)$ denotes a low-light hazy image. By plugging Eq. 10
into Eq. 12, the low-light hazy image can be generated through:

$I^{hazy}_{low}(x)=\beta\ (\alpha\ I(x))^{\gamma}t(x)+A(1-t(x))+\mathcal{N}.$
(13)

We select the brightest pixels in $I_{low}(x)$ to estimate $A$. Based on the
depth of RESIDE, the scattering coefficient of $t$ is randomly set within
[0.1, 0.2].

Noise simulation. As for noise, we follow a realistic noise model [38], which
considers the influence of signal-dependent noise and the ISP process.

Mathematically, we can obtain the low-light hazy image with an arbitrary
mutually combination of Eq. 10 and Eq. 12. As we explained before, the global
atmospheric light $A$ and the noise $\mathcal{N}$ are highly related to the
intensity of illumination in real scenes. Therefore, it is reasonable to
render the low-light environment before generating the haze.

###  4.2 Datasets

Based on the RESIDE dataset, we simulate 8970 paired low-light hazy scenes by
the above-mentioned simulation strategy. We split the simulated dataset into
8073 training images and 897 testing images. To constrain the dehazing /
enhancement block during the training phase, we also generate the haze-only
dataset and low-light-only dataset. In conclusion, the proposed simulated
dataset includes four groups, which is formulated as
$X=\\{I,I_{L},I^{H},I^{H}_{L}\\}$.

Since there is no real data for this newly raised task, we also collect a real
low-light hazy dataset, which contains 200 images. These images were taken
with a Sonny $\alpha$7RII camera on a tripod, which are as following
considerations: scenes (i.e.houses, campuses, streets), camera viewing angles
(i.e.front view and oblique view), camera apertures (i.e.f/2.0 - f/16). The
dataset and corresponding project will be released soon.

##  5 Experiments

Since our task is for low-light hazy images, which involves two tasks:
dehazing and low-light enhancement. We choose the SOTA dehazing and low-light
enhancement methods for comparison experiments: FFA [10], MSBDN [11], DA [12],
Grid [32], KinD [22], and ZeroDCE [23]. For fair comparisons, these methods
are evaluated on the proposed simulated dataset with retrained models. We
evaluate the quantitative comparison by the commonly used metrics: PSNR and
SSIM [39]. It should be noted that the newly raised task is about visibility
enhancement for low-light hazy scenes, and there is no specific method to
directly solve this problem. In this regard, to demonstrate that the simple
sequential combination of dehazing and enhancement methods cannot solve this
particular task, we utilize two strategies for comparison: $LLE\rightarrow
Dehaze$ (First low-light enhancement and then dehaze strategy) and
$Dehaze\rightarrow LLE$ (First dehaze and then low-light enhancement strategy)
methods. In addition, we directly retrain the above-mentioned methods to prove
the effectiveness of our network.

In this section, firstly, we introduce our experimental implementation
details. Next, we conduct a series of comparisons among our methods and other
SOTA approaches on our synthetic visibility enhancement dataset, including two
different sequential combination ways and the end-to-end way. Then we show the
performance of our method in real low-light hazy scenarios. Furthermore, we
conduct ablation experiments to demonstrate the effectiveness of our
framework. Finally, we do some applied research to discuss the value of our
method.

Figure 5: Qualitative comparisons among the state-of-the-art  $LLE\rightarrow Dehaze$ methods trained and tested on visibility enhancement dataset. Our method recovers a high-quality image, while others have varying degrees of haze residue. Table 1: Quantitative comparison results of $LLE\rightarrow Dehaze$ methods. First(Retrained)- | KinD | ZeroDCE | Ours
---|---|---|---
Second(Original)- | FFA | MSBDN | GridDehaze | FFA | MSBDN | GridDehaze
SSIM | 0.752 | 0.751 | 0.766 | 0.523 | 0.463 | 0.211 | 0.914
PSNR | 18.21 | 17.85 | 18.38 | 11.14 | 11.03 | 7.13 | 26.92
First(Retrained)- | KinD | ZeroDCE | Ours
Second(Finetune)- | FFA${\dagger}$ | MSBDN${\dagger}$ | GridDehaze${\dagger}$ | FFA${\dagger}$ | MSBDN${\dagger}$ | GridDehaze${\dagger}$
SSIM | 0.781 | 0.835 | 0.833 | 0.572 | 0.554 | 0.518 | 0.914
PSNR | 21.14 | 24.72 | 23.49 | 14.16 | 11.43 | 10.18 | 26.92
Figure 6: Qualitative comparisons among the state-of-the-art  $Dehaze\rightarrow LLE$ methods trained and tested on visibility enhancement dataset. Boxes indicate obvious differences. Nearly all haze is removed by our method without generating other noise while other methods are commonly not. Figure 7: Qualitative comparisons among the state-of-the-art end-to-end retrained methods trained and tested on visibility enhancement dataset. Boxes indicate obvious differences. Our method can restore color details well and can remove the noise. Table 2: Quantitative comparison results of $Dehaze\rightarrow LLE$ methods. First(Retrained)- | FFA | MSBDN | GridDehaze | Ours
---|---|---|---|---
Second(Original)- | KinD | ZeroDCE | KinD | ZeroDCE | KinD | ZeroDCE
SSIM | 0.776 | 0.650 | 0.744 | 0.614 | 0.751 | 0.612 | 0.914
PSNR | 18.44 | 15.76 | 19.13 | 15.68 | 18.38 | 15.34 | 26.92
First(Retrained)- | FFA | MSBDN | GridDehaze | Ours
Second(Finetuned)- | KinD${\dagger}$ | ZeroDCE${\dagger}$ | KinD${\dagger}$ | ZeroDCE${\dagger}$ | KinD${\dagger}$ | ZeroDCE${\dagger}$
SSIM | 0.629 | 0.678 | 0.633 | 0.654 | 0.672 | 0.641 | 0.914
PSNR | 14.58 | 14.62 | 16.34 | 15.16 | 15.84 | 14.56 | 26.92

###  5.1 Implementation Details

The training examples are augmented by random rotating -10${}^{\circ}\sim$10∘,
and horizontal flipping. We implement the model using PyTorch and train it on
an NVIDIA Tesla V100. We split the visibility enhancement dataset into two
disjoint parts, 8073 images as the training set, and 897 images as the
validation set. All the training images are resized to the size of 256
$\times$ 256\. We train the framework for 250 epochs and optimize the
parameters by Adam[40] optimizer, where $\beta_{1}$ and $\beta_{2}$ take the
values of 0.5 and 0.999.

Figure 8: Examples of visibility enhancement results on our real dataset,
compared with KinD [22], FFA [10], MSBND [11]. Close-up views are displayed at
the top of each image.

Algorithm 1 Pipeline to train our framework

0: Datasets($I^{H}_{L}$, $I^{H}$, $I_{L}$, $I$), untrained block
$\mathcal{D}(\cdot,\theta)$, untrained block $\mathcal{E}(\cdot,\theta)$,
untrained CABF block

0: Optimal $\mathcal{D}(\cdot,\theta)$, optimal $\mathcal{E}(\cdot,\theta)$
and optimal fushion block Stage I train for $\mathcal{D}(\cdot,\theta)$ and
$\mathcal{E}(\cdot,\theta)$:

1: train $\mathcal{D}(\cdot,\theta)$ with ($I^{H}_{L}$, $I_{L}$) and ($I^{H}$,
$I$) using the dehazing loss

2: train $\mathcal{E}(\cdot,\theta)$ with ($I^{H}_{L}$, $I^{H}$) and ($I_{L}$,
$I$) using the enhancement loss Stage II jointly train for the cross-
consistency framework:

1: load pretrained blocks $\mathcal{D}(\cdot,\theta)$ and
$\mathcal{E}(\cdot,\theta)$

2: train the entire framework with ($I^{H}_{L}$, $I^{H}$, $I_{L}$, $I$) using
the total loss

We first train the dehazing and enhancement blocks using corresponding
data(hazy/clear images under different exposed conditions for dehazing block;
lowlight/normal images under different concentrations of haze for enhancement
block), and then we comprehensively train the entire framework, as shown in
Algorithm 1. In the comprehensive training, we consider the noise problem of
the second stage of each path, and strengthen the weight of the loss of the
second part and the fusion part. We set $\lambda_{11}=\lambda_{21}=0.2$,
$\lambda_{12}=\lambda_{22}=2$ and $\lambda_{3}=5$. The pre-trained blocks’
learning rate is set as $10^{-3}$. In the comprehensive training, the two
blocks’ learning rate is set as $10^{-5}$ since them have been pre-trained
while the fusion part’s is set as $10^{-3}$.

###  5.2 Different Sequential Combination Manner

$LLE\rightarrow Dehaze$.  Recall that the visibility enhancement in this paper
can be decomposed into two relative tasks, i.e., low-light enhancement and
haze removal. To evaluate the different combinations of the above tasks, we
conduct the following experiments. Firstly, we use $LLE\rightarrow Dehaze$
manner to improve the low-visibility images. In detail, the first stage (low-
light enhancement) enhances the low-light hazy image($I^{H}_{L}$) to normal
hazy image($I^{H}$). The current low-light enhancement methods are trained on
the dataset without haze. To make such methods generalize well in our
situation, we re-train the first stage methods on our simulated visibility
enhancement dataset to make them learn the mapping $I^{H}_{L}\mapsto I^{H}$.
We choose two recently proposed methods, KinD [22] and ZeroDCE [23] as the
first stage, respectively. After the first stage, the second stage becomes a
traditional haze removal task. Here we choose the SOTA dehazing methods
(FFA/MSBDN/GridDehaze) as the second stage. We directly adopt their pre-
trained model, which can be downloaded from their project page. Furthermore,
we finetune these methods on our dataset for a better mapping of $I^{H}\mapsto
I$. The finetuned version of these methods is marked with ${\dagger}$.

We conduct the quantitative comparison on the test dataset of visibility
enhancement dataset. Table 1 shows that our method obtains the highest values
of metrics on both SSIM and PSNR. As mentioned above, the top part of the
table is the result of directly using the pre-trained dehazing model, and the
bottom part is using the fine-tuned models. Overall, the experimental results
after fine-tuning are better than those directly using the pre-trained
dehazing model. The visual comparisons are shown in Fig. 5. As expected, the
performance of these methods often suffers from color distortion and haze
residual. The low-light enhancement will not only amplify the haze but corrupt
the physical rule. Therefore, the dehazing methods cannot handle the further
dehazing issue after the low-light enhancement.

$Dehaze\rightarrow LLE$. We alter the order of low-light enhancement and
dehazing models to prove that performing image dehazing before low-light
enhancement also suffers from unexpected results. In the combined
$Dehaze\rightarrow LLE$ methods, the one-stage dehazing method performs on the
low-light hazy image($I^{H}_{L}$) and expects to obtain a low-light image
without haze($I_{L}$). Since the models of traditional dehazing methods are
trained under normal light conditions, which are not very suitable for our
condition, we first retrain the dehazing methods (FFA-Net/MSBDN/GridDehaze) on
our visibility enhancement dataset to make them learn the mapping
$I^{H}_{L}\mapsto I_{L}$. The second stage is the traditional low-light
enhancement task. So we first directly use the SOTA low-light enhancement
methods’ best open source pretrained models(KinD/Zero-DCE). Furthermore, we
finetune these methods on our dataset for better mapping of $I_{L}\mapsto I$.
The fine-tuned version of these methods is marked with ${\dagger}$.

We conduct the quantitative comparison on the test dataset of the visibility
enhancement dataset. Table 2 shows that our method receives the best numerical
scores in both PSBR and SSIM. As mentioned above, the top part of the table is
the result of directly using the pre-trained low-light enhancement model, and
the bottom part is using the fine-tuned models. Since the former dehazing
networks are not designed for the low-light hazy images, directly training
them by visibility enhancement dataset cannot deliver pleasing results. Even
though the enhancement models seem to perform well, the final dehazing results
for low-light scenes still tend to leave haze and darken some regions. From
the experimental results in the table and the visual results in Fig. 6, it can
be seen that the noise and artifacts generated by the residual haze in the
first stage will be more severely amplified in the second stage. When the
results obtained in the first stage are used to fine-tune the low light
enhancement method, it will degrade the performance of the result.

###  5.3 End-to-end Manner

Logically, visibility enhancement for low-light hazy scenes could be included
in both of dehazing task and low-light enhancement task. Therefore, with the
proper dataset, both of these models may solve this task. To verify the
effectiveness of proposed method, we directly retrain the SOTA methods of
dehazing and low-light enhancement on our visibility enhancement dataset for
comparison.

Table 3: Quantitative comparisons among end-to-end retrained methods and ours. Method | KinD | ZeroDCE | FFA | MSBDN | GridDehaze | Ours
---|---|---|---|---|---|---
SSIM | 0.751 | 0.521 | 0.820 | 0.837 | 0.817 | 0.914
PSNR | 18.38 | 10.93 | 23.45 | 25.63 | 21.88 | 26.92

As shown in Fig. 7, directly training these former methods can deliver better
results than the sequential combination way, which verify that applying the
image dehazing and low-light enhancement in a cascade way may amplify the
error. We find that the haze cannot be removed properly through low-light
enhancement methods like KinD and ZeroDCE. Since the target of these methods
aims to make the outputs brighter without focusing on haze removal, there are
still haze residuals contained in the outputs of state-of-the-art dehazing
methods. Based on the idea of cross-consistency, we jointly optimize dehazing
and enhancement in a path-invariant manner. As a result, the proposed method
can deliver clean and bright results. Table 3 shows that our method is more
effective and accurate for joint dehazing and enhancement in low-light hazy
scenes.

Figure 9: Rating distribution of the user study. Results from 1 (best) to 4
(worst). Most participants rated our results as the highest(67.88%) or second-
highest(22.39%) quality. Our method has significant advantages over other
methods, suggesting that our results align with the high-quality standards of
human perception.

###  5.4 Performance on Real Scenarios

To evaluate the proposed method can generalize to real scenes, we also conduct
comparative experiments in real scenes with four methods, which perform well
on the simulated dataset. As shown in Fig. 8, our method can obviously deliver
better results in terms of visual perception for this particular task. The
performance on real scenarios also demonstrates the effectiveness of the
proposed dataset simulation strategy. For real low-light hazy images, the
conventional low-light enhancement methods cannot perform well. Though the
dehazing methods can achieve good visual perception, the results still contain
some artifacts and dark regions.

User study.  To further prove the effectiveness of our method, We invite 36
participants to attend a user study based on randomly selected 16 natural low-
light hazy images. We apply three most competitive methods(KinD/FFA/MSBDN) and
the proposed method on these cases for comparisons. For each case, the input
data and the four results will be shown to the participants at the same time.
The participants need to rank the quality of these four results from 1 (best)
to 4 (worst) on the aspects of brightness, haze removal, artifacts, and noise.
As shown in Fig. 9, the participants consider that the proposed method can
deliver the best results in most of the cases, which shows that our results
are more preferred by human subjects.

Figure 10: (a) Statistical results of ablation study. The legend shows the meaning of each line. The solid line is the original value. The dashed line is the trend fitted by Microsoft Excel. (b) Visual results of ablation study. Our final framework achieves favorable qualitative and quantitative results. Table 4: Ablation study. Method | OnePath-d-e | OnePath-e-d | TwoPath-without-fusion | Total model
---|---|---|---|---
SSIM | 0.9096 | 0.9085 | 0.9116 | 0.9141
PSNR | 26.40 | 26.47 | 26.46 | 26.92

###  5.5 Ablation Analysis

In this section, we analyze the effectiveness of our cross-consistency
dehazing-enhancement framework. All pipelines are trained using the same
setting. As mentioned in the proposed method, there are two paths to obtain
the well-exposed clean image $I$ from the under-exposed hazy image
$I^{H}_{L}$, as indicated in Eq. 3. The experimental results proved that the
order of dehazing and enhancement has a limited impact on the performance, as
shown in Table 4. To perfectly utilize the results of two paths, we design the
CABF block to obtain the final pleasing results. The results in the fourth row
of Table 4 demonstrate the effectiveness of the CABF block. In order to
further analyze the existing numerical results, we count the results of the
ablation experiments under different illumination difficulties. The results of
our three methods are shown in the Fig. 10, and we used the Microsoft Excel to
fit the performance of the three methods trend. It can be seen that as the
illumination decreases, the task difficulty increases, and the performance of
the three methods is in a downward trend as a whole. Still, the results of our
final model always maintain the best performance results. At the same time, we
selected a typical example for visual analysis. We can see that both the one
path-d-e method and the one path-e-d method have different degrees of haze
residue and other artifacts, and our final model can have a better visual
effect. Our total model achieves the best results without increasing the
number of parameters. Overall, the results of the ablation study verify the
validity of the proposed cross-consistency dehazing-enhancement framework.

Figure 11: Results of three applications. (a) Visual comparisons on panoptic
segmentation task with different enhancement and dehazing methods. (b) Google
Vision API for Object Recognition before/after joint enhancement and dehaze.
(c) Image caption before/after joint enhancement and dehaze. All three of
these applications prove that our research is valuable.

###  5.6 Applications

Since the hazy images with low-light illumination are inevitable in real
scenarios, these images can become extremely hard cases for other computer
vision tasks. To demonstrate the motivation and effectiveness of the proposed
method, we conduct three computer vision applications: panoptic segmentation,
object detection and image captioning. To verify the benefit of the our cross-
consistency dehazing-enhancement framework, we directly apply our trained
model as an image pre-processing tool.

For the panoptic segmentation task, we aim to pixel-wisely classify the
category of instances in the image, including things in the foreground and
background. We show the visual comparisons among different methods in Fig. 11.
Due to the pleasing results of joint dehazing and enhancement for low-light
scenes, our method can achieve desirable segmentation results for the input
image. Image captioning task means automatically generating a description of
the image. In this case, we apply the self-critical [41] to generate image
captions. Fig. 11 illustrates that the image captioning method generates more
accurate descriptions when taking the image processed by our method as input.
We also employ Google Vision API to evaluate our results. As can be seen in
Fig. 11, the Google API can recognize most objects in the results image rather
than the original rainy image. Especially, the score of the building is
improved by 5% after visibility enhancement by our method. At the same time,
the input image before processing will have a more serious atmospheric
phenomenon.

##  6 Conclusions

In this paper, we propose a cross-consistency dehazing-enhancement framework,
focusing on integrating dehazing and low-light enhancement inference for low-
light hazy scenes. With the parameter shared dehazing block and enhancement
block, the proposed framework is capable of integrating the dehazing and
enhancement without affecting each other. To tackle the problem of the lack of
paired hazy low-light data, we propose an image simulation strategy to
construct the visibility enhancement dataset. Leveraging on the dataset, we
conduct several experimental comparisons to prove the necessity and
effectiveness of our model. The experimental results demonstrate that the
proposed solution can outperform the state-of-the-art methods for this
particular task. In addition, the analysis of real scenes, user study, and
applications show that the proposed method is practical and effective.

## References

  * [1] Srinivasa G Narasimhan and Shree K Nayar.  Vision and the atmosphere.  International journal of computer vision, 48(3):233–254, 2002.
  * [2] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu.  Deep retinex decomposition for low-light enhancement.  arXiv preprint arXiv:1808.04560, 2018.
  * [3] Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld.  Adaptive histogram equalization and its variations.  Computer vision, graphics, and image processing, 39(3):355–368, 1987.
  * [4] Ali M Reza.  Realization of the contrast limited adaptive histogram equalization (clahe) for real-time image enhancement.  Journal of VLSI signal processing systems for signal, image and video technology, 38(1):35–44, 2004.
  * [5] K. He, J. Sun, and X. Tang.  Single image haze removal using dark channel prior.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12):2341–2353, 2011.
  * [6] Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, and Dacheng Tao.  Dehazenet: An end-to-end system for single image haze removal.  IEEE Transactions on Image Processing, 25(11):5187–5198, 2016.
  * [7] Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng.  An all-in-one network for dehazing and beyond.  arXiv preprint arXiv:1707.06543, 2017.
  * [8] He Zhang and Vishal M Patel.  Densely connected pyramid dehazing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3194–3203, 2018.
  * [9] Wei Liang, Jing Long, Kuan-Ching Li, Jianbo Xu, Nanjun Ma, and Xia Lei.  A fast defogging image recognition algorithm based on bilateral hybrid filtering.  ACM transactions on multimedia computing, communications, and applications (TOMM), 17(2):1–16, 2021.
  * [10] Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia.  Ffa-net: Feature fusion attention network for single image dehazing.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11908–11915. AAAI Press, 2020.
  * [11] Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang.  Multi-scale boosted dehazing network with dense feature fusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2157–2167, 2020.
  * [12] Yuanjie Shao, Lerenhan Li, Wenqi Ren, Changxin Gao, and Nong Sang.  Domain adaptation for image dehazing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [13] Ye Liu, Lei Zhu, Shunda Pei, Huazhu Fu, Jing Qin, Qing Zhang, Liang Wan, and Wei Feng.  From synthetic to real: Image dehazing collaborating with unlabeled real data.  arXiv preprint arXiv:2108.02934, 2021.
  * [14] Zeyuan Chen, Yangchao Wang, Yang Yang, and Dong Liu.  Psd: Principled synthetic-to-real dehazing guided by physical priors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7180–7189, 2021.
  * [15] Zhuoran Zheng, Wenqi Ren, Xiaochun Cao, Xiaobin Hu, Tao Wang, Fenglong Song, and Xiuyi Jia.  Ultra-high-definition image dehazing via multi-guided bilateral learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16185–16194, 2021.
  * [16] Ziyi Sun, Yunfeng Zhang, Fangxun Bao, Ping Wang, Xunxiang Yao, and Caiming Zhang.  Sadnet: semi-supervised single image dehazing method based on an attention mechanism.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(2):1–23, 2022.
  * [17] Gaoming Du, Jiting Wu, Hongfang Cao, Kun Xing, Zhenmin Li, Duoli Zhang, and Xiaolei Wang.  A real-time effective fusion-based image defogging architecture on fpga.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17(3):1–21, 2021.
  * [18] Shih-Chia Huang, Fan-Chieh Cheng, and Yi-Sheng Chiu.  Efficient contrast enhancement using adaptive gamma correction with weighting distribution.  IEEE transactions on image processing, 22(3):1032–1041, 2012.
  * [19] Shijie Hao, Xu Han, Yanrong Guo, and Meng Wang.  Decoupled low-light image enhancement.  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(4):1–19, 2022.
  * [20] Liang Shen, Zihan Yue, Fan Feng, Quan Chen, Shihao Liu, and Jie Ma.  Msr-net: Low-light image enhancement using deep convolutional network.  arXiv preprint arXiv:1711.02488, 2017.
  * [21] Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim.  Mbllen: Low-light image/video enhancement using cnns.  In BMVC, page 220, 2018.
  * [22] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo.  Kindling the darkness: A practical low-light image enhancer.  In Proceedings of the 27th ACM International Conference on Multimedia, MM ’19, pages 1632–1640, New York, NY, USA, 2019. ACM.
  * [23] Chunle Guo Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong.  Zero-reference deep curve estimation for low-light image enhancement.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 1780–1789, June 2020.
  * [24] Mohit Lamba and Kaushik Mitra.  Restoring extremely dark images in real time.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3487–3497, 2021.
  * [25] Jing Zhang, Yang Cao, and Zengfu Wang.  Nighttime haze removal based on a new imaging model.  In 2014 IEEE International Conference on Image Processing (ICIP), pages 4557–4561. IEEE, 2014.
  * [26] Yu Li, Robby T Tan, and Michael S Brown.  Nighttime haze removal with glow and multiple light colors.  In Proceedings of the IEEE international conference on computer vision, pages 226–234, 2015.
  * [27] Jing Zhang, Yang Cao, Shuai Fang, Yu Kang, and Chang Wen Chen.  Fast haze removal for nighttime image using maximum reflectance prior.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7418–7426, 2017.
  * [28] Jing Zhang, Yang Cao, Zheng-Jun Zha, and Dacheng Tao.  Nighttime dehazing with a synthetic benchmark.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 2355–2363, 2020.
  * [29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [30] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In Proceedings of the European conference on computer vision (ECCV), pages 286–301, 2018.
  * [31] Kaixuan Wei, Jiaolong Yang, Ying Fu, Wipf David, and Hua Huang.  Single image reflection removal exploiting misaligned training data and network enhancements.  In CVPR, 2019.
  * [32] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen.  Griddehazenet: Attention-based multi-scale network for image dehazing.  In ICCV, 2019.
  * [33] Tom Mertens, Jan Kautz, and Frank Van Reeth.  Exposure fusion.  In 15th Pacific Conference on Computer Graphics and Applications (PG’07), pages 382–390. IEEE, 2007.
  * [34] Edwin H Land.  The retinex theory of color vision.  Scientific american, 237(6):108–129, 1977.
  * [35] Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar.  Llnet: A deep autoencoder approach to natural low-light image enhancement.  Pattern Recognition, 61:650–662, 2017.
  * [36] Feifan Lv, Bo Liu, and Feng Lu.  Fast enhancement for non-uniform illumination images using light-weight cnns, 2020.
  * [37] Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng, Wenjun Zeng, and Zhangyang Wang.  Reside: A benchmark for single image dehazing.  arXiv preprint arXiv:1712.04143, 1, 2017.
  * [38] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Toward convolutional blind denoising of real photographs.  2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [39] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [40] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [41] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In CVPR, 2017.