11institutetext: National Tsing Hua University
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
22institutetext: The University of Tokyo, Japan
22email<EMAIL_ADDRESS>
33institutetext: RIKEN, AIP
33email<EMAIL_ADDRESS>
# PANet: A Physics-guided Parametric Augmentation Net for Image Dehazing by
Hazing

Chih-Ling Chang∗ 11 Fu-Jen Tsai∗ 11 Zi-Ling Huang 22 Lin Gu 2233
Chia-Wen Lin 11

###### Abstract

Image dehazing faces challenges when dealing with hazy images in real-world
scenarios. A huge domain gap between synthetic and real-world haze images
degrades dehazing performance in practical settings. However, collecting real-
world image datasets for training dehazing models is challenging since both
hazy and clean pairs must be captured under the same conditions. In this
paper, we propose a Physics-guided Parametric Augmentation Network (PANet)
that generates photo-realistic hazy and clean training pairs to effectively
enhance real-world dehazing performance. PANet comprises a Haze-to-Parameter
Mapper (HPM) to project hazy images into a parameter space and a Parameter-to-
Haze Mapper (PHM) to map the resampled haze parameters back to hazy images. In
the parameter space, we can pixel-wisely resample individual haze parameter
maps to generate diverse hazy images with physically-explainable haze
conditions unseen in the training set. Our experimental results demonstrate
that PANet can augment diverse realistic hazy images to enrich existing hazy
image benchmarks so as to effectively boost the performances of state-of-the-
art image dehazing models.

###### Keywords:

Image Dehazing Haze Augmentation

∗ Equal Contribution.

##  1 Introduction

Images taken in hazy environments often suffer from severe degradation,
leading to undesirable contrast and appearance distortions. These hazy
artifacts are often dense and non-homogeneous in real-world scenarios,
significantly impacting the visual quality and visibility of scenes and
downstream computer vision tasks, _e.g_. object detection, tracking, and scene
understanding. Image dehazing aims to restore high-quality and clean images
from single hazy images, which is a highly ill-posed inverse problem due to
information loss caused by the degradation.

To address this problem, image dehazing techniques have found significant
advancements through the success of deep learning. Numerous studies [21, 10,
9, 15, 35, 40, 20, 32, 37] have been dedicated to improving dehazing
performance through network architecture designs. These studies primarily
utilized CNN-based modules to learn haze-related features, such as channel-
wise attention [21], haze-aware feature distillation [10], and dual-domain
selection module [9]. In addition, inspired by the success of Transformers
[36] in vision applications [11, 7, 33], several studies have utilized
Transformer-based architectures for image dehazing, such as transmission-aware
Transformer [15] and window-based Transformer [35]. These methods primarily
rely on synthetic image dehaze datasets [19] that adopt the physical
scattering models [26, 29, 28] to generate homogeneous synthetic hazy images
as

$\begin{gathered}I(z)=J(z)t(z)+A(1-t(z))\mbox{,}\\\ t(z)=e^{-\beta
d(z)}\mbox{,}\end{gathered}$ (1)

where $I(z)$ and $J(z)$ respectively denote the input hazy image and its
corresponding clean version, $A$ and $t(z)$ denote the atmospheric light and
transmission map, $\beta$ and $d(x)$ are the haze density and depth map, and
$z$ is the pixel index.

Figure 1: Left: Examples of the augmented images by PANet (yellow rectangles)
and other augmentation methods (blue rectangles). Existing non-homogeneous
hazy datasets [3, 6] contain a very limited number of training pairs, as shown
in green circles. Previous augmentation methods [38, 39] cannot effectively
generate non-homogeneous hazy images. In contrast, PANet can generate
realistic non-homogeneous and homogeneous hazy images, as shown in yellow
rectangles. Right: Comparison of the dehazing results with and without using
data augmented by PANet.

Although using the physical scattering models can generate abundant hazy and
clean counterparts, a large domain gap between synthetic and real-world hazy
image distributions usually significantly limits image dehazing performances
in practical settings [13, 41, 8], where real-world hazy images often exhibit
dense and non-homogeneous haze [2, 3, 6], as shown in Fig. 1. To address this
problem, existing works [3, 6] mainly resort to collecting real-world non-
homogeneous hazy and clean training image pairs. However, collecting such
datasets is challenging and costly since both the hazy and clean counterparts
must be captured under the same conditions, including identical moving objects
and background light. Therefore, these datasets often contain a very limited
number of training pairs, significantly limiting the performances of deep
dehazing models in real-world scenarios. Some methods have attempted to
enhance the diversity of hazy images via brightness adjustments [38] or global
haze density adjustments [39], as illustrated in Figs. 1(B) and 1(C).
Nevertheless, they ignore the above important fact that real-world haze
distributions are often dense and non-homogeneous, making the domain gap still
large. Therefore, it is crucial to propose a new approach to learn to generate
additional realistic non-homogeneous hazy images with various haze conditions
from existing hazy and clean image pairs without heavily relying on a high-
cost data collection process.

Figure 2: Overview of the proposed PANet. PANet comprises a Haze-to-Parameter
Mapper (HPM) to project real haze images into a parameter space and a
Parameter-to-Haze Mapper (PHM) to revert them back to the real space. In
addition to generating original hazy images (green rectangle box), PANet can
generate real hazy images not provided in the training set (yellow rectangle
box).

In this paper, we propose a novel Physics-guided Parametric Haze Augmentation
Net (PANet) to effectively augment realistic non-homogeneous hazy and clean
training pairs for enhancing dehazing performance in the real world. As
overviewed in Fig. 2, PANet utilizes a Haze-to-Parameter Mapper (HPM) to
project the hazy images of training clean and hazy image pairs into a
parameter space spanned by haze parameters, followed by a Parameter-to-Haze
Mapper (PHM) to resample and then convert the haze parameters in the parameter
space back to haze images. Specifically, inspired by the widely-adopted
Physical Scattering Model (PSM) in Eq. (1), HPM parameterizes the haze
conditions of hazy images into two pixel-wise haze parameters: haze density
and atmospheric light. Next, these estimated parameters are resampled and then
used to translate the clean version into its hazy versions with various haze
conditions in a two-step manner. In the initial step, we utilize the physical
scattering model in Eq. (1) to generate reasonable initial hazy images. In the
refinement step, we further use a Data-driven Haze Refiner (DHR)
$N_{\mathrm{DHR}}(\cdot)$ to refine the initial hazy image to make it
realistic. Note, although with the physics guides, since a scene in a dense
and non-homogeneous haze is usually substantially occluded or distorted, it is
still a highly ill-posed problem to retrieve accurate pixel-wise haze
parameter maps from a clean and hazy image pair. Such ill-posedness makes the
estimated haze parameters inaccurate, thereby leading to unsatisfactory visual
qualities of hazy images generated solely by the PSM [13, 41, 8]. To address
this problem, we propose using the DHR to refine the the input hazy image and
then use the reconstruction error as supervision to form a cyclic haze-
parameter-haze learning path involving both the HPM and DHR as elaborated in
Sec. 3.

Our proposed physics-guided haze augmentation scheme offers several
advantages. First, with the scattering model-based physics guides, the
estimated haze parameters have their physical meanings, making the parameter
resampling and the resulting hazy image generation explainable. Second, our
haze-parameter-haze mapping framework constitutes a cyclic learning path for
haze generation involving haze parameter estimation, model-based
initialization, and data-driven refinement. This leads to more accurate haze
parameter estimation and realistic hazy image generation to complement the
inaccuracy of model-based generation as well as simply simplify the pure data-
driven deep model design thanks to reasonable initialization based on the
scattering models. Third, by resampling the explainable haze parameters, we
can easily augment hazy images with various haze conditions to significantly
boost the performances of existing deep dehazing models while reducing the
cost of training data collection. The contributions of this work can be
summarized as follows:

  * •

We propose a novel physics-guided Parametric Augmentation Net (PANet) that
generates realistic, non-homogeneous hazy images to enhance existing deep
dehazing models.

  * •

The proposed PANet is a controllable network that can retrieve from input hazy
images core haze parameters, including pixel-wise haze density and atmospheric
light, and then resample the haze parameters to generate additional photo-
realistic hazy images with various explainable haze conditions unseen in the
original training set.

  * •

Extensive experimental results demonstrate the high efficacy of PANet in
boosting state-of-the-art dehazing models on four real-world image dehazing
datasets. Cross-dataset evaluations also validate the generability of PANet.

##  2 Related Work

###  2.1 Image Dehazing

Image dehazing techniques have achieved remarkable progress with the fast
growth of CNNs. Specifically, to effectively extract haze-related features,
several studies resorted to attention-based methods using CNNs. For example,
Liu  _et al_. [21] proposed a multi-scale attention-based network that
utilizes channel-wise attention for feature fusion. Xu  _et al_. [31] proposed
a feature fusion attention network with cascaded channel-attention and pixel-
attention modules. Deng  _et al_. [10] proposed a haze-aware representation
distillation module to distill haze-related features through instance
normalization. Fu  _et al_. [12] utilized discrete wavelet transform with a
generative adversarial network (GAN) to preserve high-frequency knowledge in
the feature space. Cui  _et al_. [9] proposed an efficient image restoration
network that contains a dual-domain selection mechanism to emphasize important
regions for restoration.

Recently, motivated by Transformers’ powerful ability to model long-range
dependencies among features, several studies have devised transformer-based
models for image dehazing. Song  _et al_. [35] utilized window-based attention
[22] to design a vision transformer for image dehazing. Guo  _et al_. [15]
proposed a hybrid architecture that integrates CNN and Transformer with a
transmission-aware 3D position embedding to improve image dehazing
performance. Although these methods successfully improve image dehazing
performance through elaborate model designs, they primarily rely on synthetic
hazy datasets, which may lead to a performance decrease when handling real-
world hazy images. Instead of concentrating solely on architectural designs to
improve dehazing performance, our goal is to design a haze augmentation method
applicable across various dehazing models and improve dehazing performance in
real-world scenarios.

###  2.2 Hazy Image Augmentation

In addition to improving dehazing performance through architectural designs,
some studies focused on hazy image augmentation strategies to enhance dehazing
performance. Based on a physical scattering model, the method proposed in [38]
incorporates brightness adjustment, color bias, and Gaussian noise within the
physical scattering model to simulate adverse light conditions in real-world
scenarios. However, this approach modifies additional factors rather than
leveraging the inductive bias of real-world hazy images, which are usually
non-homogeneous with high opacity. To generate diverse hazy images with real-
world haze characteristics, Yang  _et al_. [39] proposed a rehazing model that
incorporates depth and haze density with CycleGAN [42]. By globally sampling
haze density, they can generate additional hazy images as a data augmentation
operation. However, GAN-based architectures often encounter challenges such as
unstable training process [14, 25], model collapse [1, 24], and uncontrollable
outputs [17, 34], which restricts the diversity and usability of the generated
images. Furthermore, their method only allows global haze density adjustment,
making it unsuitable for real-world hazy images that typically exhibit non-
uniformity with high opacity.

In contrast, we propose PANet, a stable and controllable framework to generate
real-world hazy images for data augmentation, leveraging the inductive bias
inherent in real-world hazy images. In addition, our approach allows for
pixel-wisely adjusting haze conditions to generate non-homogeneous haze with
different densities and positions. This enhances the diversity of hazy images
and leads to significant improvements on existing image dehazing models on
several real-world hazy image datasets.

##  3 Proposed Method

###  3.1 Overview

In the real world, haze is often non-homogeneous and exhibits high opacity. To
generate hazy images with real-world characteristics, we propose a PANet that
augments photo-realistic hazy images with diverse haze conditions for
individual hazy and clean training pairs to effectively boost the performances
of real-world image dehazing. Fig. 3 illustrates the block diagram of PANet, a
cyclic Haze-Parameter-Haze mapping framework consisting of a Haze-to-Parameter
Mapper (HPM) followed by a Parameter-to-Haze Mapper (PHM). Given a pair of
hazy and clean images, we first use HPM to project the hazy image into a
learned parameter space that characterizes real-world haze conditions with two
physically-explainable haze parameters: the haze density and atmospheric
light. Next, we use the estimated haze density and atmospheric light maps to
translate the input clean image to an initial hazy image based on the physical
scattering model in Eq. (1). We then refine the initial hazy image to a more
photo-realistic one by using a Data-driven Haze Refiner (DHR)
$N_{\mathrm{DHR}}(\cdot)$ to complement the inaccuracy of the scattering model
due to the ill-posedness of haze parameter estimation.

In this way, besides the hazy images in the original training set, we can
easily augment additional hazy iamges with various explainable haze conditions
unseen in the training set by resampling the pixel-wise parametric haze
conditions (_i.e_., the haze density and atmosphere light). This allows us to
significantly enrich the training data, thereby enhancing the performances of
existing dehazing models in real-world scenarios. Next, we will introduce the
core components of PANet, including HPM, PHM, and the haze augmentation
process.

Figure 3: Block diagram of PANet. PANet utilizes a cyclic haze-parameter-haze
mapping framework consisting of a Haze-to-Parameter Mapper (HPM) followed by a
Parameter-to-Haze Mapper (PHM). Besides the hazy images in the original
training set (green boxes), PANet can augment additional hazy images with
various haze conditions unseen in the training set (yellow boxes).

###  3.2 Haze-to-Parameter Mapper (HPM)

HPM aims to project hazy images into a learned parameter space that
parameterizes real-world haze conditions with two physically-explainable haze
parameters: haze density and atmospheric light. HPM comprises an encoder and
two decoders, as shown in Fig. 4. Given a hazy image
$I_{H}(z)\in\mathbb{R}^{H\times W\times 3}$, where $z$ is the pixel index, the
encoder $E_{\mathrm{haze}}(\cdot)$ extracts the haze-relevant features of
$I_{H}(z)$. Next, the Density Decoder $D_{\mathrm{dense}}(\cdot)$ and the
Atmospheric Light Decoder $D_{\mathrm{AL}}(\cdot)$ are then used to estimate
the pixel-wise haze density map $\beta_{\mathrm{est}}(z)\in\mathbb{R}^{H\times
W\times 3}$ and atmospheric light map
$A_{\mathrm{est}}(z)\in\mathbb{R}^{H\times W\times 1}$, respectively, as

$\begin{gathered}\beta_{\mathrm{est}}(z)=D_{\mathrm{dense}}(E_{\mathrm{haze}}(I_{H}(z))),\end{gathered}$
(2)

and

$\begin{gathered}A_{\mathrm{est}}(z)=D_{\mathrm{AL}}(E_{\mathrm{haze}}(I_{H}(z))).\end{gathered}$
(3)

To derive $d(z)$ in Eq. (1), we choose RA-Depth [27] as the pre-trained depth
estimator $\Psi(\cdot)$ same as RIDCP [38]. Besides, to bridge the domain gap
with the pre-trained depth estimator, we further use a Depth Refinement Module
(DRM) $d_{\mathrm{ref}}$ to refine the depth map as

$d(z)=d_{\mathrm{ref}}(\Psi(I_{C}(z))),$ (4)

where the architecture of $d_{\mathrm{ref}}(\cdot)$ is similar to HPM but with
only one decoder.

Figure 4: Architecture of Haze-to-Parameter Mapper (HPM). HPM consists of a
shared encoder followed by two parallel parameter decoders to estimate the
haze density map $\beta_{\mathrm{est}}(z)$ and atmospheric light map
$A_{\mathrm{est}}(z)$, respectively.

To train the modules in HPM to learn to accurately estimate
$\beta_{\mathrm{est}}(z)$, $A_{\mathrm{est}}(z)$ and $d(z)$, we calculate the
fidelity between the input hazy image and its reconstructed version generated
from its clean version by using the PSM in Eq. (1) and the PHM in Sec. 3.3
based on the predicted $\beta_{\mathrm{est}}(z)$, $A_{\mathrm{est}}(z)$, and
$d(z)$. This reconstruction fidelity reflects the accuracy of the estimated
parameters, thereby allowing us to use it as supervision to guide the learning
of HPM and DRM.

###  3.3 Parameter-to-Haze Mapper (PHM)

After retrieving the haze parameters and scene depth, we further utilize PHM
to map the haze parameters in the parameter space back to real hazy images.
Specifically, based on the estimated $\beta_{\mathrm{est}}(z)$ and
$A_{\mathrm{est}}(z)$, we subsequently translate the input clean image
$I_{C}(z)$ to the initial hazy image
$O_{\mathrm{ini}}(z)\in\mathbb{R}^{H\times W\times 3}$ using the following
Physical Scattering Model:

$\begin{gathered}O_{\mathrm{ini}}(z)=I_{C}(z)t(z)+A_{\mathrm{est}}(z)(1-t(z)),\end{gathered}$
(5)

where

$\begin{gathered}t(z)=e^{-\beta_{\mathrm{est}}(z)d(z)},\end{gathered}$ (6)

where $d(z)\in\mathbb{R}^{H\times W\times 1}$ denotes the depth map estimated
from the clean image $I_{C}(z)$.

By using the Physical Scattering Model, we can generate initial hazy images
$O_{\mathrm{ini}}(z)$ from the clean image $I_{C}(z)$. This model provides
physical meanings for the haze parameter estimated by HPM. However, since a
scene in a dense, non-homogeneous haze is usually substantially occluded or
distorted, retrieving pixel-wise haze parameters from a hazy image is highly
ill-posed, making the model-generated initial hazy images inaccurate and
distorted (_e.g_., incorrect color tone and unrealistic transparency) [13, 41,
8]. Therefore, we propose a Data-driven Haze Refiner (DHR)
$N_{\mathrm{DHR}}(\cdot)$ to further refine the initial hazy images to
mitigate the inaccuracy. To this end, we concatenate $O_{\mathrm{ini}}(z)$
with its corresponding clean image $I_{C}(z)$ and feed them to
$N_{\mathrm{DHR}}(\cdot)$ to get the real hazy image $O_{\mathrm{final}}(z)$
as

$O_{\mathrm{final}}(z)=N_{\mathrm{DHR}}(\mathbf{Concate}(O_{\mathrm{ini}}(z),I_{C}(z))),$
(7)

where $N_{\mathrm{DHR}}(\cdot)$ has a similar architecture to HPM but with
only one decoder.

With the cyclic mapping framework involving HPM and PHM, PANet can
successfully project hazy images into a parameter space and then generate
additional hazy images by pixel-wisely resampling the haze density
$\beta_{\mathrm{est}}(z)$ and atmospheric light $A_{\mathrm{est}}(z)$ with
physically-explainable haze conditions unseen in the training set, as
elaborated in the haze augmentation process.

###  3.4 Loss Function

We choose Charbonnier loss [18] $\mathcal{L}_{\mathrm{char}}$ and perceptual
loss [16] $\mathcal{L}_{\mathrm{perc}}$ for optimizing PANet as follows:

$\begin{gathered}\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{char}}(O_{\mathrm{ini}}(z),I_{H}(z))+\mathcal{L}_{\mathrm{char}}(O_{\mathrm{final}}(z),I_{H}(z))\\\
+\lambda\mathcal{L}_{\mathrm{perc}}(O_{\mathrm{ini}}(z),I_{H}(z))+\lambda\mathcal{L}_{\mathrm{perc}}(O_{\mathrm{final}}(z),I_{H}(z))\mbox{,}\end{gathered}$
(8)

where $O_{\mathrm{ini}}(z)$ denotes the model-generated initial hazy image,
$O_{\mathrm{final}}(z)$ denotes the refined hazy images, $I_{H}(z)$ denotes
the ground-truth hazy image, and $\lambda$ is a weight empirically set to
$\lambda=10^{-6}$.

Figure 5: Visuals of hazy images generated by PANet. Given a hazy image, we
can decrease or amplify its haze density by 2, as shown in (A) and (B). In
addition, we can reverse its haze location or generate a complex hazy image,
as shown in (C) and (D)

###  3.5 Haze Augmentation Process

To generate new hazy images unseen in the training set, given a pair of hazy
$I_{H}(z)$ and clean $I_{C}(z)$ images, we pixel-wisely resample its estimated
haze density map $\beta_{\mathrm{est}}(z)\in\mathbb{R}^{H\times W\times 3}$
and atmospheric light map $A_{\mathrm{est}}(z)\in\mathbb{R}^{H\times W\times
1}$ to obtain their new versions: $\beta^{\prime}(z)$ and $A^{\prime}(z)$. The
two new maps are then used to generate a new hazy image by using PHM.
Specifically, we can alter haze density $\beta_{\mathrm{est}}(z)$ by a scaling
factor $\alpha$ to generate $\beta^{\prime}(z)$ as

$\beta^{\prime}(z)=\alpha\cdot\beta_{\mathrm{est}}(z),$ (9)

For example, two new hazy images with $\alpha=0.5$ and $\alpha=2$ are
illustrated in Figs. 5(A) and 5(B), respectively. As illustrated in Fig. 5(C),
we can also reverse the location of haze patterns by altering the atmospheric
light map $A_{\mathrm{est}}(z)$ as

$A^{\prime}(z)=1-A_{\mathrm{est}}(z),$ (10)

where $A_{\mathrm{est}}(z)$ ranges in $[0,1]$. In this case, for those reverse
regions that do not contain haze in the original hazy image, we sample
$\beta^{\prime}(z)$ to be in $[0.6,1.25]$, the range of $\beta_{\mathrm{est}}$
in the whole training set. Moreover, we can linearly interpolate
$A_{\mathrm{est}}(z)$ and $1-A_{\mathrm{est}}(z)$ to generate $A^{\prime}(z)$
as

$A^{\prime}(z)=\min(\gamma
A_{\mathrm{est}}(z)+\eta(1-A_{\mathrm{est}}(z)),1),$ (11)

where $\gamma$ and $\eta$ denote the weights for $A_{\mathrm{est}}(z)$ and
$1-A_{\mathrm{est}}(z)$, respectively. This allows us to generate complex hazy
images, as shown in Fig. 5(D). In our experiments, we generate additional hazy
images for each hazy and clean training pair by adjusting their haze density
and location based on the above haze augmentation process.

##  4 Experiments

###  4.1 Implementation Details

PANet. For optimizing PANet, we utilize NH-Haze20 dataset [3] that contains
non-homogeneous hazy and clean pairs with real-world outdoor scenes. We follow
the settings in [12, 9] that utilize $50$ training pairs and $5$ testing pairs
in our experiments. During the training process, we employ the Adam optimizer
with the initial learning rate of $5\times 10^{-5}$ that is decayed to
$10^{-7}$ by using the cosine annealing strategy to train PANet for $270$
epochs with a batch size of 2. We also randomly crop $256\times 256$ patches
and utilize random rotation and flipping for augmentation.

Dehazing Models. We adopt three state-of-the-art dehazing models, including
DW-GAN [12], DeHamer [15], and FocalNet [9], to demonstrate the effectiveness
of PANet. To make a fair comparison, we utilize the $50$ training pairs of NH-
Haze20 to train the above-mentioned dehazing models as their baseline
following the default training setting in their methods. To demonstrate the
effectiveness of PANet, we utilize PANet to generate additional $400$ training
pairs, which is $8$ times larger than the original training set. We combine
the augmented $400$ training pairs and the original $50$ training pairs to
train the three dehazing models as the enhanced version of their baseline. To
evaluate the performances of the dehazing models, we choose four real-world
hazy image datasets, including NH-Haze20 [3] test set, NH-Haze21 [6] dataset,
O-Haze [5] test set, and I-Haze [4] test set. Specifically, NH-Haze20 test set
contains $5$ testing pairs with non-homogeneous haze. Since NH-Haze21 does not
provide a test set, we use its training set that contains $25$ pairs collected
in non-homogeneous hazy environments for evaluation. In contrast, O-Haze and
I-Haze test sets contain $5$ outdoor and $5$ indoor testing pairs with
homogeneous haze, respectively.

Table 1: Quantitative performances of different dehazing methods on NH-Haze20
test set, NH-Haze21 dataset, O-Haze test set, and I-Haze test set. "Baseline"
and "+PANet" represent the dehazing performance without and with PANet,
respectively.

|  | NH-Haze20 [3] | NH-Haze21 [6] | O-Haze [5] | I-Haze [4]
---|---|---|---|---|---
Model | PSNR (dB) | SSIM | PSNR (dB) | SSIM | PSNR (dB) | SSIM | PSNR (dB) | SSIM
DW-GAN [12] | Baseline | 21.50 | 0.697 | 18.10 | 0.726 | 18.44 | 0.574 | 14.88 | 0.403
| +PANet | 21.84 (+0.34) | 0.704 | 18.42 (+0.32) | 0.708 | 20.15 (+1.71) | 0.634 | 15.47 (+0.59) | 0.508
DeHamer [15] | Baseline | 20.01 | 0.649 | 16.49 | 0.612 | 20.01 | 0.600 | 15.49 | 0.463
| +PANet | 20.73 (+0.72) | 0.650 | 17.05 (+0.56) | 0.627 | 20.64 (+0.63) | 0.650 | 16.22 (+0.73) | 0.563
FocalNet [9] | Baseline | 20.31 | 0.646 | 16.51 | 0.632 | 18.28 | 0.622 | 15.29 | 0.417
| +PANet | 20.76 (+0.45) | 0.682 | 17.87 (+1.36) | 0.700 | 20.64 (+2.36) | 0.639 | 15.41 (+0.12) | 0.374
Average Gain | +0.50 | +0.015 | +0.75 | +0.022 | +1.57 | +0.042 | +0.48 | +0.054

###  4.2 Performance Evaluations

Quantitative Performance Comparison. In Table 1, we compare the dehazing
performances of three baselines and their PANet-enhanced versions, where
“Baseline” and “+PANet” denote the dehazing performance without and with
PANet, respectively. As shown in Table 1, The PANet-augmented data
significantly improve the average PSNR performances of the three dehazing
models, including DW-GAN [12], DeHamer [15], and FocalNet [9] by $0.50$ dB,
$0.75$ dB, $1.57$ dB, and $0.48$ dB on NH-Haze20, NH-Haze21, O-Haze and,
I-Haze, respectively. These evaluation results demonstrate that PANet can
effectively help boost the performances of deep dehazing models under various
haze conditions.

Figure 6: Qualitative performance comparison on NH-Haze20 [3] testing set.
Figure 7: Qualitative performance comparison on NH-Haze21 [6] dataset.

Qualitative Performance Comparison. We demonstrate the dehazed results of
dehazing models with or without using PANet in Figs. 6, 7, 8, and 9. In Fig. 6
and Fig. 7, we visualize some dehazed results on non-homogeneous haze
datasets: NH-Haze20 and NH-Haze21. Compared to their baselines, PANet-enhanced
models achieve significant visual quality improvements by removing unwanted
hazy artifacts or correcting color distortions. Fig. 8 and Fig. 9 visualize
some dehazing results on homogeneous haze datasets: O-Haze and I-Haze. Again,
the PANet-enhanced dataset can also significantly boost the performances of
state-of-the-art models under homogeneous haze conditions in both outdoor and
indoor scenarios. These visuals show that PANet is effective in augmenting
both homogeneous and non-homogeneous hazy images in real-world scenarios.

###  4.3 Ablation studies

In the ablation studies, we analyze the impact of PANet on the dehazing
performance of FocalNet on NH-Haze20 test set, where“Baseline” denotes the
PSNR performance of FocalNet trained on NH-Haze20 training set without using
the additional training pairs augmented by PANet.

Figure 8: Qualitative Comparison on O-Haze [5] testing set Figure 9: Qualitative Comparison on I-Haze [4] testing set Table 2: Effect of the Data-driven Haze Refiner (DHR) $N_{\mathrm{ref}}(\cdot)$ in PANet. FocalNet [9] | Baseline | w/o DHR | with DHR
---|---|---|---
NH-Haze20 (PSNR) | 20.31 dB | 20.22 dB | 20.76 dB

Effect of the Data-driven Haze Refiner (DHR) $N_{\mathrm{DHR}}(\cdot)$. To
verify the effectiveness of DHR, we compare the dehazing performance enhanced
by PANet with or without using DHR. As shown in Table 2, “w/o DHR” and “w/
DHR” denote the dehazing performances of PANet without and with DHR,
respectively. The results show that PANet without DHR, which generates hazy
images by solely using the physical scattering model, cannot improve the
Baseline, due to the inaccuracy of the estimated haze parameters. Besides, we
compare the qualitative results of the generated hazy images for“w/o DHR” and
“w/ DHR” in Fig. 10. Without using DHR, the generated hazy images contain
unrealistic color tones and transparency compared to the ones using DHR, which
demonstrates the importance of DHR in PANet.

Effect of the Depth Refinement Module (DRM). Since the pre-trained depth
estimator suffers from a domain gap when addressing unseen clean images, this
may degrade the accuracy of the physical scattering model [30, 23]. Therefore,
we use DRM to refine the initial depth map. We compare the performance with or
without using DRM in Table 3, where “w/o DRM” and “w/ DRM” denote PANet
without and with DRM, respectively. The results show that PANet without DRM
cannot improve the Baseline, which demonstrates the importance of DRM in
PANet.

Effect of the Number of Augmented Images. To analyze the impact of the number
of training pairs generated by PANet on the dehazing performance, we use
different numbers of training pairs augmented by PANet to improve the Baseline
trained on $50$ training pairs. As shown in Table 4, we generate additional
$200$ ($400\%$ of the original data size), $400$ ($800\%$ of the original
size), and $600$ training pairs ($1,200\%$ of the original size). The results
show that the dehazing performance improves with the number of additional
training pairs but tend to get saturated when the number of additional pairs
reaches $600$. Therefore, we choose to augment $400$ additional training pairs
considering the tradeoff between the training time and the performance gain.

Figure 10: Effect of the Data-driven Haze Refiner (DHR) in PANet. First row: Hazy images generated by PANet with DHR. Second row: Hazy images generated by PANet without DHR. Table 3: Effectiveness of the Depth Refinement Module (DRM) in PANet. FocalNet [9] | Baseline | w/o DRM | w/ DRM
---|---|---|---
NH-Haze20 (PSNR) | 20.31 | 20.31 | 20.76
Table 4: Effect of the dehazing performance versus the number of augmented pairs by PANet, where the original training set contains $50$ training pairs. FocalNet [9] | Baseline (0%) | 200 (400%) | 400 (800%) | 600 (1200%)
---|---|---|---|---
NH-Haze20 (PSNR) | 20.31 | 20.51 | 20.76 | 20.83

Comparison with different augmentation methods. Finally, we compare PANet with
different haze augmentation methods, including uniform haze method: Physical
Scattering Model (PSM), GAN-based method: $D^{4}$ [39], and RIDCP [38], where
we use these methods to augment the NH-Haze20 training set to improve FocalNet
and evaluate on the NH-Haze20 testing set. As shown in Table 5, our PANet
achieves the best performance compared to other augmentation methods. In
addition, we compare the hazy images generated by these methods in Fig. 11.
Among these methods, PSM and RIDCP [38] solely rely on the physical scattering
models to generate hazy images. Although RIDCP can alter the brightness and
color bias, they both cannot generate non-homogeneous hazy images. $D^{4}$
[39] applies a cycle-GAN-based architecture to generate hazy images. However,
the lack of robustness of GAN increases the difficulty of generating realistic
hazy images. In addition, GAN-based methods cannot pixel-wisely control haze
conditions to generate diverse hazy images. In contrast, our PANet is a robust
network through the physics-guided learning strategy and can pixel-wisely
alter hazy conditions to generate diverse non-homogeneous hazy images, which
also achieve the best dehazing performances as shown in Fig. 12.

Figure 11: Hazy images generated by different augmentation methods, including
Physical Scattering Model (PSM), $D^{4}$ [39], RIDCP [38], and PANet. Figure
12: Dehazing results with different augmentation methods, including Physical
Scattering Model (PSM), $D^{4}$ [39], RIDCP [38], and PANet. Table 5:
Comparison regarding dehazing performance with different augmentation methods,
including Physical Scattering Model (PSM), $D^{4}$ [39], RIDCP [38], and
PANet.

FocalNet [9] | Baseline | w/ PSM | w/ D4 [39] | w/ RIDCP [38] | w/ PANet
---|---|---|---|---|---
NH-Haze20 (PSNR) | 20.31 | 20.41 | 20.26 | 20.57 | 20.76

##  5 Conclusion

In this paper, we proposed a Parametric Augmentation Net (PANet) to generate
diverse non-homogeneous hazy images for boosting the performances of dehazing
models in real-world scenarios. PANet comprises a Haze-to-Parameter Mapper
(HPM) that projects real hazy images into a parameter space and a Parameter-
to-Haze Mapper (PHM) that maps them back to hazy images. With PANet, we can
pixel-wisely resample the estimated haze parameter maps to generate hazy
images with various physically-explainable haze conditions unseen in the
training set. This allows us to generate diverse training pairs to improve
dehazing performance. Extensive experimental results demonstrate the high
efficacy of PANet in enhancing three state-of-the-art dehazing models on four
real-world hazy image datasets.

## References

  * [1] Akash, S., Lazar, V., Chris, R., U., G.M., Charles, S.: Veegan: Reducing mode collapse in gans using implicit variational learning. In: NeurIPS (2017)
  * [2] Ancuti, C.O., Ancuti, C., Sbert, M., Timofte, R.: Dense haze: A benchmark for image dehazing with dense-haze and haze-free images. In: ICIP (2019)
  * [3] Ancuti, C.O., Ancuti, C., Timofte, R.: NH-HAZE: an image dehazing benchmark with non-homogeneous hazy and haze-free images. In: CVPRW (2020)
  * [4] Ancuti, C.O., Ancuti, C., Timofte, R., Vleeschouwer, C.D.: I-haze: a dehazing benchmark with real hazy and haze-free indoor images. In: ACIVS (2018)
  * [5] Ancuti, C.O., Ancuti, C., Timofte, R., Vleeschouwer, C.D.: O-haze: a dehazing benchmark with real hazy and haze-free outdoor images. In: CVPRW (2018)
  * [6] Ancuti, C.O., Ancuti, C., Vasluianu, F.A., Timofte, R.: Ntire 2021 nonhomogeneous dehazing challenge report. In: CVPRW (2021)
  * [7] Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., Gao, W.: Pre-trained image processing transformer. In: CVPR (2021)
  * [8] Chen, Z., Wang, Y., Yang, Y., Liu, D.: Psd: Principled synthetic-to-real dehazing guided by physical priors. In: CVPR (2021)
  * [9] Cui, Y., Ren, W., Cao, X., Knoll, A.: Focal network for image restoration. In: ICCV (2023)
  * [10] Deng, Q., Huang, Z., Tsai, C.C., Lin, C.W.: Hardgan: A haze-aware representation distillation gan for single image dehazing. In: ECCV (2020)
  * [11] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [12] Fu, M., Liu, H., Yu, Y., Chen, J., Wang, K.: Dw-gan: A discrete wavelet transform gan for nonhomogeneous dehazing. In: CVPRW (2021)
  * [13] Gui, J., Cong, X., Cao, Y., Ren, W., Zhang, J., Zhang, J., Cao, J., Tao, D.: A comprehensive survey and taxonomy on single image dehazing based on deep learning. ACM Computing Surveys (2023)
  * [14] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) NeurIPS (2017)
  * [15] Guo, C.L., Yan, Q., Anwar, S., Cong, R., Ren, W., Li, C.: Image dehazing transformer with transmission-aware 3d position embedding. In: CVPR (2022)
  * [16] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV (2016)
  * [17] Kowalski, M., Garbin, S.J., Estellers, V., Baltrušaitis, T., Johnson, M., Shotton, J.: Config: Controllable neural face image generation. In: ECCV (2020)
  * [18] Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: CVPR (2017)
  * [19] Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., Wang, Z.: Benchmarking single-image dehazing and beyond. IEEE TIP (2019)
  * [20] Li, C., Guo, C., Guo, J., Han, P., Fu, H., Cong, R.: Pdr-net: Perception-inspired single image dehazing network with refinement. IEEE TMM (2019)
  * [21] Liu, X., Ma, Y., Shi, Z., Chen, J.: Griddehazenet: Attention-based multi-scale network for image dehazing. In: ICCV (2019)
  * [22] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV (2021)
  * [23] Lou, Z., Xu, H., Mu, F., Liu, Y., Zhang, X., Shang, L., Li, J., Guan, B., Li, Y., Hu, Y.H.: Simhaze: game engine simulated data for real-world dehazing. arXiv preprint arXiv:2305.16481 (2023)
  * [24] Mao, Q., Lee, H.Y., Tseng, H.Y., Ma, S., Yang, M.H.: Mode seeking generative adversarial networks for diverse image synthesis. In: CVPR (2019)
  * [25] Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. In: ICCV (2017)
  * [26] McCartney, E.J.: Optics of the atmosphere: scattering by molecules and particles. New York (1976)
  * [27] Mu, H., Le, H., Yikai, B., Jian, R., Jin, X., Jian, Y.: Ra-depth: Resolution adaptive self-supervised monocular depth estimation. In: ECCV (2022)
  * [28] Narasimhan, S.G., Nayar, S.K.: Contrast restoration of weather degraded images. IEEE TPAMI (2003)
  * [29] Nayar, S.K., Narasimhan, S.G.: Vision in bad weather. In: ICCV (1999)
  * [30] Othman, M.K., Abdulla, A.A.: Enhanced single image dehazing technique based on hsv color space. UHDJST (2022)
  * [31] Qin, X., Wang, Z., Bai, Y., Xie, X., Jia, H.: Ffa-net: Feature fusion attention network for single image dehazing. In: AAAI (2020)
  * [32] Qu, Y., Chen, Y., Huang, J., Xie, Y.: Enhanced pix2pix dehazing network. In: CVPR (2019)
  * [33] Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: ICCV (2021)
  * [34] Shoshan, A., Bhonker, N., Kviatkovsky, I., Medioni, G.: Gan-control: Explicitly controllable gans. In: ICCV (2021)
  * [35] Song, Y., He, Z., Qian, H., Du, X.: Vision transformers for single image dehazing. IEEE TIP (2023)
  * [36] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017)
  * [37] Wu, H., Qu, Y., Lin, S., Zhou, J., Qiao, R., Zhang, Z., Xie, Y., Ma, L.: Contrastive learning for compact single image dehazing. In: CVPR (2021)
  * [38] Wu, R., Duan, Z., Guo, C., Chai, Z., Li, C.: Ridcp: Revitalizing real image dehazing via high-quality codebook priors. In: CVPR (2023)
  * [39] Yang, Y., Wang, C., Liu, R., Zhang, L., Guo, X., Tao, D.: Self-augmented unpaired image dehazing via density and depth decomposition. In: CVPR (2022)
  * [40] Yu, H., Zheng, N., Zhou, M., Huang, J., Xiao, Z., Zhao, F.: Frequency and spatial dual guidance for image dehazing. In: ECCV (2022)
  * [41] Zhang, X., Dong, H., Pan, J., Zhu, C., Tai, Y., Wang, C., Li, J., Huang, F., Wang, F.: Learning to restore hazy video: A new real-world dataset and a new method. In: CVPR (2021)
  * [42] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV (2017)