# Restoring Images Captured in Arbitrary Hybrid Adverse Weather Conditions in
One Go

Ye-Cong Wan, Ming-Wen Shao Yuan-Shuo Cheng, Yue-Xian Liu, Zhi-Yuan Bao, and
De-Yu Meng Y.-C. Wan, M.-W. Shao, Y.-S. Cheng, Y.-X. Liu, and Z.-Y. Bao are
with the College of Computer Science and Technology, China University of
Petroleum, China. D.-Y. Meng is with the Institute for Information and System
Sciences, Xi’an Jiaotong University, China.

###### Abstract

Adverse conditions typically suffer from stochastic hybrid weather
degradations (e.g., rainy and hazy night), while existing image restoration
algorithms envisage that weather degradations occur independently, thus may
fail to handle real-world complicated scenarios. Besides, supervised training
is not feasible due to the lack of comprehensive paired dataset to
characterize hybrid conditions. To this end, we have advanced the
forementioned limitations with two tactics: framework and data. On the one
hand, we present a novel unified framework, dubbed RAHC, to Restore Arbitrary
Hybrid adverse weather Conditions in one go, which can comfortably cope with
hybrid scenarios with insufficient remaining background constituents and
restore arbitrary hybrid conditions with a single trained model flexibly. On
the other hand, we establish a new dataset, termed HAC, for learning and
benchmarking arbitrary Hybrid Adverse Conditions restoration. HAC contains 31
scenarios composed of an arbitrary combination of five common weather, with a
total of $\sim\\!316K$ adverse-weather/clean pairs. As for fabrication, the
training set is automatically generated by a dedicated AdverseGAN with no-
frills labor, while the test set is manually modulated by experts for
authoritative evaluation. Extensive experiments yield superior results and in
particular establish new state-of-the-art results on both HAC and conventional
datasets.

###### Index Terms:

Hybrid weather conditions, Unified framework, Data generation.

##  1 Introduction

In real-world adverse scenarios, different weather degradations often occur
simultaneously in an uncertain way, e.g., snowy night, rainy night. Images
captured under these conditions inevitably incur abysmal visibility and
corrupted feature remnants, and the stochastic hybrid disruption may
dramatically hamper outdoor high-level vision systems such as autonomous
driving systems [1, 2, 3] and surveillance systems [4]. Unfortunately,
prominent algorithms [5, 6, 7, 8] deal with each weather degradation
individually ignoring the hybrid degradation characterization of combined
action. Specifically, early researchers focused on developing task-specific
restoration algorithms for given weather conditions [9, 10, 11, 12, 13, 14]
(Fig. 1(a)). Afterwards various generic networks were proposed to handle
different degradations with identical architecture [15, 16, 17, 18, 19] (Fig.
1(b)), but with different weights for different tasks. To address this issue,
recent researches aim at restoring degradation under multiple adverse
conditions with a single unified model [20, 21, 22, 23, 24] (Fig. 1(c)). While
combined rain and haze scenes were evaluated [21, 23, 24], more hybrid
conditions and more complex hybrid combinations remain untapped by these
methods especially when there are more than two weather degradations mixed
together. Despite few task-specific works [25, 26, 27] have partly explored
restoring two-weather superimposed scenarios, they were designed for
specialized combinations (e.g., rain streak and raindrop) and unable to be
extended to other real-world diverse hybrid scenarios.

To more flexibly and practically deal with real-world scenarios, a turnkey
solution that can restore arbitrary hybrid adverse conditions in one go is
imminent. Compared to this goall, the existing works have the following
limitations. In terms of framework, (i) existing restoration networks are
limited in characterizing hybrid multiple weather degradations simultaneously
due to the lack of a multi-subspace feature extraction mechanism. (ii) Models
utilized in single degradation removal are restricted in restoring hybrid
adverse weather conditions with insufficient remaining background
constituents. (iii) Previous unified learning strategies are designed for non-
overlapping degradations and are constrained to popularize to diverse hybrid
scenarios. In terms of data, (i) existing datasets only cover separate single
or special double weather degradations without various hybrid conditions. (ii)
Collecting real-world paired data would be virtually infeasible, due to
uncontrollable factors, e.g., wind, lighting, and camera shifts. (iii) Manual
synthesis of massive paired data with hybrid adverse weather conditions tends
to be time-consuming and labor-consuming.

With the above observations, there are two posers need to answer: (1) How to
construct a unified framework that can flexibly tackle real-world diverse
hybrid scenarios with a single trained model? (2) How to effectively establish
a comprehensive paired dataset that includes an exhaustive list of all
possible real-world adverse weather conditions?

Figure 1:  Overview of adverse conditions restoration frameworks. (a) Separate
networks designed for specific tasks; (b) generic networks with task-specific
weights; (c) unified all-in-one networks with single trained weights; (d) our
proposed RAHC framework. In contrast to existing approaches that aim to tackle
conditions with a single weather type, RAHC can handle arbitrary hybrid
adverse weather conditions in one go, thus enjoying better flexibility and
practicality in realistic applications.

To tackle the aforementioned problems, in this paper, we propose a novel
unified framework RAHC and a new hybrid dataset HAC to restore arbitrary
hybrid adverse weather conditions in one go (Fig. 1(d)). For the framework
RAHC, there are three innovations for defeating the above limitations. (1)
Multi-Head Blend Block (MHBB) for multi-weather degradation representation:
the multi-head mechanism overriding the blend operator of convolution and
attention can provide multiple “representation subspaces” [28] as well as
complementary feature for multi-weather learning. (2) Reconstruction vectors
aided restoration for hybrid conditions with limited image constituents
retention: we propose using discrete representations from the Codebook [29]
pre-trained on large-scale natural images to. These discrete representations,
which we refer to as reconstruction vectors, can provide additional visual
content cues to auxiliary the reconstruction of realistic and clean output.
(3) Output space discrimination for efficient arbitrary hybrid conditions
restoration: we design a simple multilabel-classification discriminator from
the output space to encourage the restoration network to learn degradation-
agnostic repair capabilities, which enable RAHC to flexibly handle diverse
hybrid scenarios without requiring complex strategies or modules. Notably,
this protocol can be seamlessly integrated into existing universal image
restoration algorithms in a non-destructive manner, enhancing their
performance in the all-in-one multi-weather removal setting.

For the construction of hybrid adverse weather conditions dataset HAC, there
are $\sim\\!316K$ pairs of $2^{5}\\!-\\!1\\!=\\!31$ adverse conditions with
five common weather types (namely haze, rain streak, snow, night, and
raindrop) arranged in combination except for the clean one. To synthesize
sufficient and diverse pairwise data efficiently and at low consumption, we
develop a powerful generator AdverseGAN to learn from adverse conditions so as
to approximate the degradation implicitly rather than expensive manual
labeling. Thus, the training set can be automatically generated by AdverseGAN
with minimal labor cost. On the contrary, the test set is meticulously
handcrafted by our recruited experts for authoritative evaluation. The domain
gap between the training and test sets allows for better evaluation of the
generalization ability which is critical for real-world applications,
especially when the real data is infeasible. Comprehensive experimental
results substantiate the superiority of the RAHC beyond state-of-the-art
restoration methods on both HAC and conventional datasets.

In conclusion, the main contributions are summarized as follows:

  * •

We propose a novel framework, dubbed RAHC, to restore diverse hybrid adverse
weather conditions while enjoying the properties of being concise, flexible,
and powerful.

  * •

We present a multi-head blend block to provide multiple representation
subspaces for multi-degeneration learning. Meanwhile, a reconstruction vectors
aided restoration scheme is devised for severely deteriorated hybrid adverse
conditions, as well as an output space discrimination regime for unified
learning of diverse hybrid degradations.

  * •

A new synthetic dataset HAC for arbitrary hybrid adverse weather conditions
restoration is constructed. To the best of our knowledge, HAC is the first to
encompass such a wide range of scenarios and provides the most comprehensive
benchmark for this task.

  * •

Solid and promising experimental results on HAC and conventional datasets
demonstrate the effectiveness, superiority, and robustness of our proposed
RAHC.

##  2 Related Work

###  2.1 Adverse Weather Conditions Restoration

Numerous algorithms have been proposed to recover images captured in adverse
weather conditions, e.g., rain [12, 30], haze [31, 32, 33], snow [34, 9], etc.
While these methods perform well on the specific weather type, significant
performance degradation was observed when migrating to other tasks. Aiming at
this limitation, a broad spectrum of research [35, 15, 16, 36, 17] has
explored generic frameworks to accommodate different degradation types with an
identical network. Even so, they still require separate training of
independent weights for each task, hindering its generalizability. To repair
multiple degradations in an all-in-one fashion, several unified models are
proposed [20, 21, 22, 23, 24]. However, these approaches ignore that real-
world adverse conditions often suffer from superimposed multiple degradations
[25], e.g., rain streaks and raindrops [26], rain and nighttime [27], etc.
Although several methods [21, 23, 24] include the superposition of two weather
(rain and fog) at test time, they suffer from relatively limited image
constituents retention as well as inadequate multi-degradation
characterization mechanism when there are more than two degradations in the
hybrid condition. Meanwhile, the unified learning strategies of these methods
are constrained in modeling hybrid multiple degradations simultaneously. On
the contrary, in this paper, adverse scenarios with an arbitrary hybrid of
five weather types are considered for a total of 31 conditions, and RAHC can
cope with all conditions in one go driving arbitrary hybrid adverse conditions
restoration in a broad sense.

###  2.2 Pseudo Data Generation

Data-driven vision tasks rely heavily on high-quality datasets. Unfortunately,
the labeling and acquiring of data tends to be expensive, challenging and
time-consuming. To surmount this bottleneck, several recent efforts [37, 38,
39, 40] delve into pseudo-data generation by leveraging Generative Adversarial
Networks (GAN). For instance, Zhang _et al._ [37] proposed a DatasetGAN to
synthesize highly realistic images with pixel-wise annotations, the model
trained on the synthetic dataset can even surpass the ones trained on the real
dataset. Yang _et al._ [38] released SurfelGAN, which facilitates the training
of autonomous driving models by simulating realistic road scenarios. In
addition to high-level tasks, this research boom has also been conveyed to
low-level tasks. Wang _et al._ [39] exceeded SOTA with only $0.17\%$ original
data and synthesized pseudo-data. Yue _et al._ [41] introduced a dual
adversarial network to simultaneously tackle noise removal and noise
generation. Inspired by CycleGAN [42], Wei _et al._ [43] generated a pseudo-
paired dataset Rain200A by cycle translation, and the models trained on it
exhibit better robustness and generalization. Inspired by these pioneer works,
we pursue the efficient and inexpensive synthesis of paired data for training
by utilizing a deliberate GAN.

###  2.3 Perceptual Image Compression

Tremendous success in two-stage image generation [29, 44, 45, 46, 47] based on
perceptual image compression has been witnessed. These works compress images
into discrete latent vectors in the first stage and synthesize high-quality
images leveraging encoded vectors in the second stage. In this paper, we focus
on perceptual image compression in the first stage. VQVAE [46] first presents
an auto-encoder model to implement multi-scale quantization of images. Based
on VQVAE, VQGAN [29] introduces adversarial and perceptual objectives to
obtain higher compression rates while preserving satisfactory perceptual
quality. Furthermore, LDM [47] explores different compression rates and
different kinds of regularizations, deriving a compression model that is more
adept at preserving details. Intrigued by the extensive image priors contained
in the discrete latent vectors of image compression, we advocate utilizing
these auxiliary priors to guide the restoration, since the discrete vectors
comprise context-rich visual parts.

###  2.4 Unsupervised Domain Adaptation (UDA)

UDA algorithms [48, 49, 50, 51, 52, 53, 54] for semantic segmentation aim to
constrain the model trained on the source domain to learn domain-invariant
features, thereby generalizing to the target domain. One category of these
studies [51, 52, 53, 54] was devoted to training a discriminator to
distinguish whether the output results come from the source or target domain,
while the segmentation network learns domain-invariant knowledge to confuse
the discriminator. Motivated by these methods, RAHC interestingly treats the
unified restoration of different degradations as a domain adaptive problem,
i.e., no matter which degradation the image suffers, the restored result
should be a degeneration-agnostic high-quality clean image. As will be
demonstrated, fooling a degradation-type classifier is more straightforward
and concise than sophisticated units and training strategies in degradation-
agnostic feature learning.

##  3 Methodology

We first present an overview of our proposed tactics for arbitrary hybrid
adverse weather conditions restoration in Sec 3.1. Then we introduce the
restoration network as well as the core elements in Sec 3.2, and describe the
output space discriminative learning scheme in Sec 3.3. Finally, we discuss
and detail our adverse conditions generator AdverseGAN in Sec 3.4.1 and the
established HAC dataset in Sec. 3.4.2.

###  3.1 Overview of the Proposed Method

Figure 2: Illustration of the proposed RAHC architecture. The restoration
network consists of an encoder, a decoder, and a feature mapping network. The
encoder and decoder are both cascaded from multiple MHBBs. The mapping network
first maps the encoded feature to the latent clean space and then locates the
reconstruction vectors in the pre-established Codebook by nearest neighbor
matching to provide privileged visual cues for the decoder. To enable
degradation-agnostic learning, we utilize a discriminator to distinguish the
type of the weather degradation from the restored image while the restoration
network struggles to fool the discriminator. Schematic diagrams for Multi-Head
Blend Block (MHBB), Convolution-Attention Module (CAM), and Dual-Path Feed-
Forward Network (DP-FFN) are illustrated in colored dashed boxes.

The training procedure for RAHC is illustrated in Fig. 2. RAHC aims to tackle
arbitrary hybrid adverse conditions restoration via a unified framework, and
the central notion is to generate adequate paired data using AdverseGAN with
minimal labor costs, and training a degradation-agnostic restoration network
through a discriminative learning scheme. Simultaneously, the framework also
incorporates visual constituents embedded in the Codebook to provide auxiliary
cues for restoring highly challenging hybrid conditions. Formally, given an
adverse condition $D\in\mathbb{R}^{H\times W\times 3}$ generated by AdverseGAN
and corresponding clean counterpart $C\in\mathbb{R}^{H\times W\times 3}$, the
degraded image $D$ is first fed into the restoration network to produce the
restored clean image $R\in\mathbb{R}^{H\times W\times 3}$. Meanwhile, the
feature mapping network learns the projection from the encoded feature to
corresponding clean embedding to locate the reconstruction vectors in the
Codebook, supplying additional auxiliary visual atoms for restoration.
Finally, the restored result is input into the discriminator to distinguish
the type of degradation, while the restoration network tries to restore
degradation-agnostic image to confuse the discriminator. The whole procedure
is a snap, and there is no extra inference cost or modification during testing
besides the restoration network.

###  3.2 Network Architecture

The restoration network adheres to a U-shaped structure, which hierarchically
cascades multiple tailored multi-head blend blocks, and the knowledge domain
is broadened at the bottleneck supported by reconstruction vectors, thereby
leading to more optimal repair results. Mathematically, given a degraded image
$D\in\mathbb{R}^{H\times W\times 3}$, a $3\times 3$ convolution is first
adopted to produce shallow feature embeddings $F_{in}\in\mathbb{R}^{H\times
W\times C}$. Then, $F_{in}$ is propagated through four encoder layers built
upon MHBB to obtain the deep feature
$F_{mi}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 8C}$. Thereafter,
$F_{mi}$ is transmitted by the feature mapping network to locate the
reconstruction vectors
$F_{rv}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times N_{z}}$ that most
likely can reconstruct the hidden clean image from the Codebook, where $N_{z}$
represents the dimension of reconstruction vector. Eventually, $F_{mi}$ and
$F_{rv}$ are concatenated together and fed into the symmetric decoder to
recover the final result $R\in\mathbb{R}^{H\times W\times 3}$ via a $3\times
3$ convolution. Next, we will describe the core components of the proposed
restoration network.

_Multi-Head Blend Block (MHBB)._ Existing feature extraction modules lack a
multi-degradation representation mechanism to capture the characteristics of
hybrid multiple weather, leading to inadequate feature modeling. Besides, the
complementary properties of convolution, which has strong local computational
capabilities, and Transformer, which is excellent at capturing long-range
dependencies, make the hybrid structure a better alternative for feature
extraction [55, 56]. To this end, we propose a Multi-Head Blend Block (MHBB)
to provide multiple “representation subspaces” as well as complementary
features for multi-weather learning. Fig. 2 illustrates the two core units
(CAM & DP-FFN) of MHBB. Rather than simply combining Transformer and
Convolution in parallel or series, we treat convolution and self-attention as
equivalent micro-level operators and deploy them in parallel. By applying a
multi-head mechanism that overrides the blend parallel operator, the proposed
block is able to provide multi-degradation representation subspaces for
unified learning. More concretely, an input feature $X$, is first divided into
“heads” which bears a resemblance to the vanilla Transformer [28]. The multi-
head design allows separate branches to learn different representations and
thus adaptively extract different degradation cues to guarantee the ability of
diversity restoration. Then, each head is transformed by a CAM. CAM contains
two branches: the attention path and the convolution path, which are split and
merged for parallel processing with the addition of $1\times 1$ convolution.
To reduce the high computational effort of vanilla self-attention while
preserving the global computational properties, we adopt the pixel-shuffle
[57] operator in the attention path to diminish the number of tokens while
avoiding information loss. The convolution path, on the other hand, consists
of two convolutional layers and a GELU activation function. We formulate such
a process as:

$\displaystyle X_{k}^{a},X_{k}^{b}=Split(Norm(Conv_{1\times 1}(X_{k}))),$ (1)
$\displaystyle\hat{X_{k}^{a}}=PS(Self\mbox{-}Attention(PU(X_{k}^{a}))),$
$\displaystyle\hat{X_{k}^{b}}=Conv_{3\times 3}(GELU(Conv_{3\times
3}(X_{k}^{b}))),$ $\displaystyle\hat{X_{k}}=Conv_{1\times
1}(Cat(\hat{X_{k}^{a}},\hat{X_{k}^{b}}))+X_{k},$

where $k$ represents the $k_{th}$ head, $PU$ and $PS$ denote pixel-unshuffle
and pixel-shuffle operations. Subsequently, the results of different heads are
integrated together with a stacked $1\times 1$ convolutional.

Locally relevant information is crucial for image restoration, while the
original Feed-Forward Network (FFN) is insensitive and inept at this demand
since it processes each token independently without considering the
relationships between them. Considering that the convolution operation can
capture local contextual information by sharing weights over neighboring
pixels. To compensate for the above limitations, we present a Dual-Path FFN
(DP-FFN) to extract local contextual information by introducing convolutional
branches parallel with a fully connected layer. The process of DP-FFN is
formulated as:

$\displaystyle\hat{Y_{1}},\hat{Y_{2}}=Split(Norm(\hat{Y})),$ (2)
$\displaystyle Y_{1}=GELU(Linear(\hat{Y_{1}})),$ $\displaystyle
Y_{2}=GELU(DConv_{3\times 3}(Conv_{1\times 1}(\hat{Y_{2}}))),$ $\displaystyle
Y=Conv_{1\times 1}(Cat(Y_{1},Y_{2}))+\hat{Y},$

where DConv${}_{3}\times 3$ represents $3\times 3$ depthwise convolution with
group number equal to channel dimension.

Overall, the MHBB process can be expressed as:

$\displaystyle\hat{Y}=$ $\displaystyle
Cat(CAM_{1}(X_{1}),CAM_{2}(X_{2}),...CAM_{k}(X_{k})),$ (3) $\displaystyle Y=$
$\displaystyle DP\mbox{-}FFN(Conv_{1\times 1}(\hat{Y})).$

_Reconstruction Vectors Aided Restoration._ Existing image restoration
algorithms intend to recover the degraded image from the remaining ambiguous
background content. In this case, the available features are limited,
especially hybrid conditions, and it is extremely challenging to recover high-
quality images with insufficient information. Inspired by two-stage image
generation models, which build a Codebook with rich context in the first stage
and then generate images with encoded discrete vectors in the second stage. We
utilize the context-rich vectors embedded in the Codebook to assist the
network in repairing hybrid degraded images, which allows the knowledge domain
of restoration to be extended from a single image to the entire vectors
repository. Benefiting from the information-rich image components contained in
the reconstruction vectors, the restoration network can better reconstruct
high-quality images.

Figure 3: Brief illustration of VQGAN model [29]. Figure 4: Detailed
illustration of reconstruction vectors aided scheme.

We first briefly describe the VQGAN model, which is proposed by Esser _et al._
[29] for two-stage image generation. VQGAN consists of an encoder E, a
codebook Q with discrete codes, and a decoder G. Given an input image
$x\in\mathbb{R}^{H\times W\times 3}$, the encoder E first embeds $x$ to latent
representation $Z\in\mathbb{R}^{\frac{H}{f}\times\frac{W}{f}\times N_{z}}$,
where $f$ and $N_{z}$ are downsampling factor and the dimension of latent
vectors, respectively. Then, a quantization $q(\cdot)$ is performed on each
representation vector $z\in\mathbb{R}^{N_{z}}$ to locate the most similar
discrete vector $q_{i}$ in the Codebook.

$\displaystyle Z_{q}=q(Z):=(argmin_{q_{i}\in Q}\parallel
z-q_{i}\parallel)\in\mathbb{R}^{\frac{H}{f}\times\frac{W}{f}\times N_{z}}.$
(4)

Finally, the decoder G reconstructs the output image $\hat{x}$ from the
quantized representation $Z_{q}$. The whole process of encoding, quantizing,
and reconstructing can be expressed as follows:

$\displaystyle\hat{x}=G(Z_{q})=G(q(E(x))).$ (5)

With the knowledge about VQGAN, the innovation of this paper can be better
understood. To be specific, we utilize the VQGAN [29] pre-trained on
OpenImages [58] with 8192 quantization encodings as a library of
reconstruction vectors. The schematic illustration of this scheme is depicted
in Fig. 4. The VQGAN encoder first produces the quantized encoding
$F_{rv}^{c}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times N_{z}}$ of the
clean image, and for the degraded image feature $F_{mi}$ encoded by the
restoration network encoder, the mapping network learns to predict the
possible embedding $F_{rv}^{d}$ that consistent with $F_{rv}^{c}$. Attention
and convolution layers are cascaded to construct the mapping network which can
be optimized by the following cosine similarity loss.

$\displaystyle\mathcal{L}_{map}=\sum_{i=0}^{\frac{W}{8}}\sum_{j=0}^{\frac{H}{8}}[\frac{1-cos(F_{rv}^{d}(i,j),F_{rv}^{c}(i,j))}{\frac{W}{8}\times\frac{H}{8}}].$
(6)

We can then obtain $F_{rv}$ by leveraging a subsequent nearest neighbor
matching $N\\!N\\!M(\cdot)$ of each spatial encoding
$F_{rv}^{d}(i,j)\in\mathbb{R}^{N_{z}}$ from
$F_{rv}^{d}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times N_{z}}$ onto its
closest reconstruction vector $rv_{k}$ in the Codebook.

RAHC approximates quantized reconstruction vectors by mapping networks rather
than predicting logits directly. The advantages are mainly twofold: (1) the
transform dimension of the feature mapping is equal to the length of the
quantized vector 256, while the dimension of the direct prediction is the size
of the whole Codebook 8192, thus our mapping network is more lightweight. (2)
Compared with direct classification to predict the reconstruction vectors,
feature mapping followed by quantization would be less challenging and yield
more accurate results (see Sec. 4.5.3).

###  3.3 Output Space Discrimination for Efficient Arbitrary Hybrid Conditions
Restoration

Figure 5: Detailed illustration of output space discriminative learning
scheme.

Existing all-in-one restoration approaches rely on distillation [20],
degradation guidance [22], and querying [23] to gain knowledge of different
degradations. Although these methods achieve excellent performance for non-
overlapping degradation, they are limited in modeling the joint
characterization of hybrid multiple degradations. Our intuition is that
regardless of the degradation type image suffered, the restored result should
be a degeneration-agnostic high-quality clean image. Thus, we innovatively
treat the unified learning of multiple degradations as a domain adaptive
problem and cultivate the degradation-agnostic adaptive restoration network
via discriminative adversarial learning scheme. In contrast to Li _et al._
[21] that strives to reserve degradation cues to train multiple feature
extractors, we dedicate to back-constraining the restoration network to yield
consistent degradation-agnostic ideal images.

Based on the concept of adversarial gaming, the discriminator aims to identify
from the restored image which degradations it suffered. On the contrary, the
restoration network endeavors to yield consistent untracked images to confuse
the discriminator. The detailed engraving is given in Fig. 5.

_Discriminator Training._ Given the restoration output $R=Network(D)$, we
forward $R$ to discriminator $Dis$ to distinguish the type of weather
degradation from the restored image. The training of the discriminator can be
regarded as a multilabel classification task, and the cross-entropy loss
objective can be defined as:

$\displaystyle\mathcal{L}_{d}=\sum_{i=0}^{n-1}[-t_{i}\log
Dis(R)_{i}-(1-t_{i})\log(1-Dis(R)_{i})],$ (7)

where $n$ denotes the number of degenerate types, $t_{i}=1$ if $I$ suffer from
degradation $i$ else $t_{i}=0$.

_Restoration Network Training._ First, pixel-level L1 loss is equipped to
allow the restoration results $R$ to approximate the ground truth clean image
$C$:

$\displaystyle\mathcal{L}_{net}^{l1}=\parallel R-C\parallel_{1}.$ (8)

Second, to allow restoring degeneration-agnostic results, i.e., results that
can not be recognized by the discriminator what degeneration types suffered, a
discriminative loss is employed:

$\displaystyle\mathcal{L}_{net}^{dis}=\sum_{i=0}^{n}-\log(1-Dis(R)_{i}).$ (9)

The ultimate goal is to minimize the L1 loss while allowing the results to
approximate consistent degradation-agnostic clean distribution. Meanwhile,
perceptual loss $\mathcal{L}_{net}^{per}$ [59] is also employed to weaken the
interference of noise from pseudo-data on the training. Thus, the final
training objective can be expressed as:

$\displaystyle\mathcal{L}_{net}=\mathcal{L}_{net}^{l1}+\lambda_{dis}\mathcal{L}_{net}^{dis}+\mathcal{L}_{net}^{per},$
(10)

where $\lambda_{dis}$ is set to 0.1 to balance relative weight of
$\mathcal{L}_{net}^{dis}$.

The proposed paradigm can cope with 31 hybrid weather conditions by relying
only on a five-class classification discriminator, whereas existing approaches
have to treat 31 scenarios separately and are limited in capturing cross-
weather interrelationships.

###  3.4 Hybrid Adverse Conditions Dataset (HAC)

####  3.4.1 AdverseGAN for Training Set Generation.

Figure 6: Pipeline of our proposed AdverseGAN. The generator is an encoder-
decoder structure, consisting of two latent space injection branches: content
branch (top) and style branch (bottom). And there are two discriminators:
Realism-Discriminator (R-Discriminator) and Pairing-Discriminator
(P-Discriminator). R-Discriminator is designed to constrain the realism of the
weather degradation, while P-Discriminator guarantees the matching with the
input, i.e., the background is consistent, and only the weather is added. For
brevity, we take the generation of rain streaks as an example, the other four
weather types are similar. It is worth noting that AdverseGAN can generate
different adverse conditions by controlling type vector and without have to
train five separate models.

Pseudo-data generation [60, 37, 38, 39, 40, 41, 43] has been shown to be
feasible and has achieved notable performance. Inspired by the above research,
to save time and effort in building dataset for end-to-end training of
arbitrary hybrid adverse conditions restoration, we propose to generate
pseudo-adverse conditions with GAN rather than artificial processing. The key
insight is to firstly train an elaborated AdverseGAN that can generate five
non-hybrid weather conditions and then hybrid conditions can be generated by
recursive calls. Therefore, ultimately we can automatically generate 31
adverse conditions automaticall without manual operation. Schematic
illustration of the AdverseGAN is provided in Fig. 6.

_Generator._ Following recent works [61, 62, 63, 64], our generator is
constructed based on a dual space structure, content space $\mathcal{C}$ and
style space $\mathcal{S}$. The generator first encodes the input clean image
into latent space and then generates the degraded image with the injection of
style vector $z_{s}$ and content vector $z_{c}$ randomly drawn from a normal
distribution. Type vector $z_{t}$ mapping from type label $t$ is integrated
together to control condition type like cGAN [65]. The injection of style
vector and content vector follow StyleGAN2 [66] and SNI [63] respectively.
Conversely, the functions of our content latent space are different from SNI
which can be summarized in the following three aspects: (1) content code and
image code are concatenated together as input feature map of the decoder
achieving natural decoupling of clean background and degradation; (2) a
separate content space separate from the style space can generate richer and
more diverse conditions; (3) dual space allows for better disentanglement of
content and style to generate more realistic and reliable adverse conditions.

After introducing dual space, the generation procedure from clean image $C$ to
corresponding degraded image $D^{t}$ can be depicted by the conditional
distribution $p(D^{t}|C,t,z_{c},z_{s})$. The generator $G$ expresses an
implicit distribution $p_{G}(D|C,t,z_{c},z_{s})$ to approximate the true
distribution $p(D^{t}|C,t,z_{c},z_{s})$. The pseudo degraded image $D$ can be
easily obtained:

$\displaystyle z_{c}\sim p(z_{c}),z_{s}\sim p(z_{s}),D=G(C,t,z_{c},z_{s}),$
(11)

where $p(i)$ denotes the distribution of the latent variable $i$.

_Discriminators._ Although paired datasets of individual degradation types are
readily available, most of them are synthetic data by artificial modulation
and are unable to simulate real degradation scenarios well. In addition,
generative adversarial networks cannot guarantee that the background of the
generated images remains constant except for degradation. In order to remedy
the above limitations, dual discriminators were introduced to generate more
realistic and content-preserved adverse conditions. Realism-discriminator
(_RD_) tries to distinguish between real-world adverse weather conditions and
generated fake weather conditions. And pairing-discriminator (_PD_) tries to
distinguish between real pairing data and pseudo pairing data, i.e.,
constraining the background consistency of the generated images. To allow
multiple adverse conditions generation, our discriminator produces probability
distributions over both sources and condition types [67],
$D:\mu\to{Dsrc(\mu),Dcls(\mu)}$.

For _RD_ , the adversarial loss is first deployed to constrain the generated
image $D$ to be indistinguishable from the real-world image $E$. The loss can
be written as:

$\displaystyle\mathcal{L}_{adv}^{RD}=$ $\displaystyle\mathbb{E}_{r}[\log
RDsrc(E)]+$ (12)
$\displaystyle\mathbb{E}_{C,t,z_{c},z_{s}}[\log(1-RDsrc(G(C,t,z_{c},z_{s})))].$

Apart from adversarial loss, type classification loss is also imposed to
guarantee that the generated weather type meets expectations. We decompose the
loss into two terms: a type classification loss of real-world images used to
optimize _RD_ , and a type classification loss of generated images used to
optimize $G$. In detail, the former is defined as:

$\displaystyle\mathcal{L}_{cls(r)}^{RD}=$
$\displaystyle\mathbb{E}_{E,\hat{t}}[-\log RDcls(\hat{t}|E)].$ (13)

where $\hat{t}$ represents the predicted weather type. On the other hand, the
latter can be defined as:

$\displaystyle\mathcal{L}_{cls(f)}^{RD}=$
$\displaystyle\mathbb{E}_{C,t,z_{c},z_{s}}[-\log
RDcls(t|G(C,t,z_{c},z_{s}))].$ (14)

Likewise, _PD_ also contains two loss objectives, the adversarial loss for
constraining the pairing of generated images and clean counterparts and the
type classification loss for ensuring the pairing type matched expectation.
The adversarial loss for optimizing _PD_ can be expressed as:

$\displaystyle\mathcal{L}_{adv}^{PD}=$ $\displaystyle\mathbb{E}_{C,H}[\log
PDsrc(C,H)]+$ (15)
$\displaystyle\mathbb{E}_{C,t,z_{c},z_{s}}[\log(1-PDsrc(C,G(C,t,z_{c},z_{s})))].$

Correspondingly, the type classification loss utilized to optimize _PD_ and
_G_ , respectively, are represented as:

$\displaystyle\mathcal{L}_{cls(r)}^{PD}=\mathbb{E}_{C,H,\hat{t}}[-\log
PDcls(\hat{t}|C,H)],$ (16)
$\displaystyle\mathcal{L}_{cls(f)}^{PD}=\mathbb{E}_{C,t,z_{c},z_{s}}[-\log
PDcls(t|C,G(C,t,z_{c},z_{s}))].$ (17)

_Full Objective._ Finally, the objective functions to optimize _G_ , _RD_ ,
and _PD_ are written, respectively, as

$\displaystyle\mathcal{L}_{G}=$
$\displaystyle\alpha(-\mathcal{L}_{adv}^{RD}+\lambda_{cls}\mathcal{L}_{cls(f)}^{RD})+$
(18)
$\displaystyle\beta(-\mathcal{L}_{adv}^{PD}+\lambda_{cls}\mathcal{L}_{cls(f)}^{PD}),$
$\displaystyle\mathcal{L}_{RD}=\alpha(\mathcal{L}_{adv}^{RD}+\lambda_{cls}\mathcal{L}_{cls(r)}^{RD}),$
(19)
$\displaystyle\mathcal{L}_{PD}=\beta(\mathcal{L}_{adv}^{PD}+\lambda_{cls}\mathcal{L}_{cls(r)}^{PD}),$
(20)

where $\alpha$, $\beta$, and $\lambda_{cls}$ are hyper-parameters that balance
the weights of each term and are set to 1, 2 and 3, respectively. To unify the
learning rate and avoid mode collapse, both $\mathcal{L}_{RD}$ and
$\mathcal{L}_{PD}$ are modulated by $\alpha$ and $\beta$. Thanks to the
decoupled design and pairing-discriminator which has served as a content
preserver, cycle consistency loss is not introduced since it is time-consuming
and resource-intensive.

In this paper, we have considered $2^{5}-1=31$ combinations for arbitrary
hybrid adverse conditions restoration (five weather types arranged in
combination except for the clean one). To cost-effectively synthesize
sufficient paired data for end-to-end training, the enormous amount of
training data is produced by AdverseGAN almost without any labor cost. We
first train the AdverseGAN which can generate five basic adverse weather
conditions by leveraging existing data, and then superimposed hybrid scenarios
can be generated by recursive calls. Ultimately, the production of the
training set can be done automatically by AdverseGAN. The datasets for
training AdverseGAN are summarized in Tab. I.

TABLE I: Summary of the dataset for AdverseGAN training. It is worth noting that all data are from the training set and 10% of real-world images were split out for the visual testing purpose. Adverse Condition | Paired Data | Real-world Data
---|---|---
Haze | OTS [68] | Li _et al._ [68]
Rain Streak | Rain12000 [7] | Wei _et al._ [69]
Snow | Snow100K [34] | Liu _et al._ [34]
Night | SICE [70] | Loh _et al._ [71]
Raindrop | Raindrop [10] | Qian _et al._ [10]
Algorithm 1 Arbitrary Adverse Condition Generation With AdverseGAN

Input:  $C$: input clean image; $t$: type label initialized to $"00000"$;
$Codes$: five binary codes from $"00001"$ to $"11111"$. Each digit represents
a weather type, and from left to right, it represents haze, rain streak, snow,
night, and raindrop in turn. ’1’ indicates that the condition contains that
type of weather, while ’0’ is the opposite.
Output: $D$: degraded image

1:for $i=0$ to $4$ do

2: if $Codes[i]="1"$ then

3: $t[i]="1"$

4: Sample $z_{c}$ from $\mathcal{N}(0,1)$

5: Sample $z_{s}$ from $\mathcal{N}(0,1)$

6: $C=AdverseGAN(C,t,z_{c},z_{s})$

7: $t="00000"$

8:$D=C$

9:return $D$

After training, we manually curate 5000 well-illuminated outdoor clean images
from three datasets [10, 34, 68] as the groundtruth. Then each image is
randomly cropped to a fixed size of $256\times 256$. Thereafter, for each
cropped image, we generate 31 adverse scenes, each scene was randomly sampled
twice from the style space and content space, resulting in $310K$ paired data.
We also provide the algorithm procedures in Algo. 1 to better illustrate the
training data generation pipeline. Moreover, we demonstrate the amazing
generative power of AdverseGAN in Fig. 7.

Inevitably, the generated fake images would introduce undesired artifacts,
which seems to compromise the training of the model, but we found the hazard
of this to be minimal. As will be demonstrated in Sec. 4.3, RAHC trained only
with data generated by AdverseGAN can achieve competitive results to the one
trained on the standard training set and outperform most state-of-the-art
methods.

Figure 7: Superimposed adverse conditions (top two rows) and interpolation
results (bottom five rows) generated by AdverseGAN. AdverseGAN can generate
infinite, controllable, realistic, and diverse adverse scenarios.

####  3.4.2 Handcrafted Test Set

Since the training set can be generated by AdverseGAN, to guarantee the
authority and accuracy of evaluation, the test set is carefully fabricated
artificially. We first capture 200 high-quality well-curated images
($720\times 480$) as ground truth with Canon EOS 60D, and then synthesize
corresponding 31 adverse conditions for each image, resulting in 6200 image
pairs. The synthesis of haze, rain streak, snow, night, and raindrop follows
previous academic literature [68, 72, 34, 73, 26]. Photoshop-related works
were performed by three photography experts we recruited. Lighting conditions,
scenes, subjects, and styles, e.g., are all taken into account to ensure
realism and diversity.

To the best of our knowledge, HAC covers the wealthiest of adverse conditions
and owns the largest amount of data. In some way, AdverseGAN can generate
infinite adverse conditions. In Tab. II, we compare HAC to other adverse
conditions restoration datasets. These datasets only focus on single or double
degradations, while our HAC dataset covers 31 scenarios combined by five types
of weather.

TABLE II: Comparison of HAC against conventional adverse conditions restoration datasets. Dataset | Haze | Rain | Snow | Night | Raindrop | Total
---|---|---|---|---|---|---
RESIDE [68] | $\checkmark$ |  |  |  |  | $\sim\\!87K$
Rain1200 [7] |  | $\checkmark$ |  |  |  | $13.2K$
Snow100K [34] |  |  | $\checkmark$ |  |  | $100K$
SICE [70] |  |  |  | $\checkmark$ |  | $\sim\\!0.6K$
Raindrop [10] |  |  |  |  | $\checkmark$ | $\sim\\!1.1K$
Outdoor-Rain [25] | $\checkmark$ | $\checkmark$ |  |  |  | $10.5K$
DarkRain [27] |  | $\checkmark$ |  | $\checkmark$ |  | $\sim\\!5K$
RainDS [26] |  | $\checkmark$ |  |  | $\checkmark$ | $5.8K$
HAC | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\sim\\!316K$

##  4 Experiments and Analysis

In this section, we conduct extensive experiments to demonstrate the
effectiveness of the proposed method. We first present a detailed experimental
setup in Section 4.1 and qualitative and quantitative comparisons with state-
of-the-art methods are performed in Sections 4.2 and 4.3. The performance in
real-world hybrid scenarios is verified in Section 4.4. We then performed
ablation experiments on the restoration framework RAHC and the adverse weather
generator AdverseGAN in Section 4.5 and Section 4.6, respectively. The
complexity of the model is subsequently analyzed in Section 4.7. Finally, we
further explored the application to high-level vision tasks in Section 4.8.

###  4.1 Implementation Details

All experiments are implemented with the Pytorch framework on two NVIDIA RTX
3090ti GPUs. For AdverseGAN, we use Adam as the optimizer with a batch size of
16, the learning rate is initialized to 0.00001 and then decreased to 0.000005
after 200 epochs. Once the network structure is determined, the size of the
generated image is fixed, so the training data are randomly cropped to
$256\times 256$. We trained from scratch for 400 epochs on the above-mentioned
datasets. Considering the differences in the number of different datasets, the
resampling strategy is applied throughout the training process. For the
restoration network, the number of MHBBs in different layers $n_{1}\sim n_{8}$
is set to 2, 4, 6, 4, 4, 2, 2, and 2, respectively. The number of heads is
symmetrical, set to 1, 2, 4, and 8 in order from shallow to deep layers. We
train our RAHC for $6\times 10^{5}$ iterations using Adam as the optimizer
with a batch size of 4, except for the first 5K iterations where the mapping
network is solely trained to avoid cold-start. The initial learning rate is
$2\times 10^{-4}$, which is steadily decreased to $1\times 10^{-6}$ using the
cosine annealing strategy [74]. Notably, all methods in the comparison are
trained on the same dataset until convergence with default configurations for
a fair comparison. PSNR [75] and SSIM [76] are utilized to evaluate the
restoration performance. These metrics are calculated in the RGB color space
except for deraining on Rain1200 [7] where PSNR and SSIM are calculated on the
Y channel in the YCbCr color space as in other works [16, 15]. The
implementation code and dataset are available at
https://github.com/Jeasco/RAHC.

###  4.2 Comparison with the State-of-the-Arts on HAC Dataset

TABLE III: Quantitative comparison with the SOTA methods on our proposed benchmark HAC dataset. Top super row: single model instance for single condition (condition-specific). Bottom super row: one model instance for all conditions (all-in-one). Multiplicative numeral indicates the number of weather types contained in an image (e.g., Triple represents the average scores of $C_{5}^{3}=10$ conditions containing three weather types). Method | Single | Double | Triple | Quadruple | Pentuple | Average
---|---|---|---|---|---|---
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
MIRNet [35] | 28.44 | 0.9285 | 22.10 | 0.8749 | 19.11 | 0.7940 | 15.74 | 0.6483 | 14.30 | 0.6257 | 19.94 | 0.7743
HINet [36] | 28.99 | 0.9393 | 23.33 | 0.9101 | 20.48 | 0.8422 | 16.14 | 0.6982 | 14.69 | 0.6689 | 20.73 | 0.8117
MPRNet [15] | 28.84 | 0.9322 | 22.79 | 0.9147 | 20.19 | 0.8375 | 15.99 | 0.6858 | 14.65 | 0.6653 | 20.49 | 0.8071
SwinIR [18] | 29.08 | 0.9411 | 24.21 | 0.9199 | 20.85 | 0.8517 | 16.78 | 0.7599 | 14.87 | 0.6672 | 21.16 | 0.8280
Uformer [19] | 29.17 | 0.9425 | 24.48 | 0.9228 | 21.05 | 0.8533 | 16.89 | 0.7683 | 14.95 | 0.6728 | 21.31 | 0.8319
Restormer [16] | 29.30 | 0.9478 | 24.60 | 0.9239 | 21.08 | 0.8531 | 16.92 | 0.7701 | 15.04 | 0.6783 | 21.39 | 0.8350
NAFNet [17] | 29.35 | 0.9513 | 24.71 | 0.9276 | 21.12 | 0.8583 | 17.01 | 0.7864 | 15.23 | 0.6844 | 21.48 | 0.8416
RAHC | 29.45 | 0.9517 | 24.98 | 0.9298 | 22.03 | 0.8817 | 19.53 | 0.8395 | 18.09 | 0.7502 | 22.82 | 0.8706
NAFNet [17] | 29.10 | 0.9407 | 24.22 | 0.9143 | 20.74 | 0.8465 | 15.83 | 0.6623 | 14.33 | 0.6287 | 20.84 | 0.7985
TransWeather [23] | 29.16 | 0.9433 | 24.25 | 0.9155 | 20.83 | 0.8495 | 16.72 | 0.7555 | 14.72 | 0.6652 | 21.14 | 0.8258
WeatherDiff [24] | 29.19 | 0.9431 | 24.36 | 0.9215 | 20.99 | 0.8530 | 16.90 | 0.7685 | 14.93 | 0.6711 | 21.27 | 0.8314
AirNet [22] | 29.19 | 0.9436 | 24.32 | 0.9207 | 20.97 | 0.8522 | 16.87 | 0.7655 | 15.01 | 0.6741 | 21.27 | 0.8312
TKL [20] | 29.23 | 0.9441 | 24.36 | 0.9229 | 21.05 | 0.8531 | 16.93 | 0.7725 | 14.92 | 0.6734 | 21.30 | 0.8332
RAHC | 29.40 | 0.9514 | 24.91 | 0.9283 | 22.17 | 0.8946 | 19.88 | 0.8425 | 18.24 | 0.7612 | 22.92 | 0.8756

The quantitative results on HAC dataset are reported in Tab. III. As can be
found, our RAHC delivers unparalleled performance gains and outperforms all
competitive models both in the condition-specific setting and in the all-in-
one setting, especially for extremely adverse scenarios such as “Quadruple”
and “Pentuple”. Notably, RAHC exceeds the top-performing unified approach TKL
[20] by 3.32dB on PSNR when there are pentuple degradation types. Furthermore,
RAHC trained in the all-in-one setting even surpasses the results separately
trained in each single condition. This phenomenon can be ascribed to the fact
that the proposed discriminative learning scheme allows the network to learn
more generalized and degradation-agnostic repair capabilities, and that the
reconstruction vectors aided scheme can benefit from more sufficient data. We
also demonstrate visual comparisons in Fig. 8 and Fig. 9. As suggested, RAHC
recovers clean and crisp results while achieving a harmonious global tone
without introducing visible artifacts or color shifts suffered by other
methods, especially for complicated hybrid scenarios.

Figure 8: Visual comparisons with SOTA adverse conditions restoration methods
on outdoor natural scenery. Our RAHC restores more visually pleasing results
with finer details and consistent illumination.  Figure 9: Visual comparisons
with SOTA adverse conditions restoration methods on outdoor building
structures. Our RAHC restores more photorealistic results with coherent fine
structural and textural details.

###  4.3 Comparison with the State-of-the-Arts on Conventional Datasets

TABLE IV: Quantitative comparisons with the SOTA methods on the Snow100K [34] dataset. Top super row: training with task-specific data. Bottom super row: training with mixed data. $\dagger$ indicates training with pure data generated by AdverseGAN. The performance gain from applying AdverseGAN is reported in parenthesis. Method | Snow100K-S | Snow100K-M | Snow100K-L
---|---|---|---
|  PSNR | SSIM |  PSNR | SSIM |  PSNR | SSIM
DesnowNet [34] |  32.33 | 0.9500 |  30.87 | 0.9409 |  27.17 | 0.8983
DDMSNet [77] |  34.34 | 0.9445 |  32.89 | 0.9330 |  28.85 | 0.8772
MPRNet [15] |  35.25 | 0.9537 |  33.02 | 0.9422 |  30.30 | 0.8999
HDCWNet [9] |  35.28 (0.07) | 0.9571 |  33.14 (0.09) | 0.9436 |  30.49 (0.14) | 0.9017
NAFNet [17] |  35.68 (0.06) | 0.9609 |  33.80 (0.11) | 0.9479 |  30.99 (0.17) | 0.9084
RAHC |  35.81 (0.09) | 0.9647 |  34.03 (0.13) | 0.9525 |  31.45 (0.16) | 0.9189
RAHC† |  35.32 | 0.9584 |  33.25 | 0.9421 |  30.56 | 0.9009
All-in-One [21] | – | - | – | - |  28.33 | 0.8820
TransWeather [23] |  35.21 (0.09) | 0.9532 |  33.02 (0.16) | 0.9408 |  30.28 (0.18) | 0.8987
AirNet [22] |  35.37 (0.09) | 0.9584 |  33.24 (0.15) | 0.9419 |  30.41 (0.19) | 0.9087
TKL [20] |  35.44 (0.08) | 0.9597 |  33.38 (0.14) | 0.9428 |  30.52 (0.17) | 0.9098
RAHC |  35.53 (0.11) | 0.9625 |  33.66 (0.17) | 0.9457 |  31.08 (0.23) | 0.9115

TABLE V: Quantitative comparison with the SOTA methods on the Raindrop removal test dataset [10]. Please reffer to Tab. V for table notes. Method | Raindrop
---|---
|  PSNR | SSIM
pix2pix [78] |  28.02 | 0.8547
Quan [8] |  31.44 | 0.9263
DuRN [79] |  31.24 (0.18) | 0.9259
Restormer [16] |  32.69 (0.14) | 0.9407
NAFNet [17] |  32.51 (0.16) | 0.9322
RAHC |  33.37 (0.15) | 0.9445
RAHC† |  32.25 | 0.9326
All-in-One [21] |  31.12 | 0.9268
TransWeather [23] | 31.74 (0.22) | 0.9279
AirNet [22] | 32.27 (0.18) | 0.9315
TKL [20] |  32.35 (0.19) | 0.9341
RAHC |  32.82 (0.24) | 0.9401

TABLE VI: Quantitative comparison with the SOTA methods on the SOTS-outdoor [68] dataset. Please reffer to Tab. V for table notes. Method | SOTS-outdoor
---|---
|  PSNR | SSIM
GridDehazeNet [6] |  31.47 | 0.9779
MPRNet [15] |  32.17 | 0.9799
FFA-Net [31] |  32.89 | 0.9802
AECR-Net [32] |  33.07 (0.14) | 0.9804
DehazeFormer [33] |  33.22 (0.10) | 0.9814
RAHC |  33.89 (0.16) | 0.9897
RAHC† |  33.24 | 0.9809
TransWeather [23] | 33.01 (0.20) | 0.9768
AirNet [22] | 33.09 (0.19) | 0.9808
TKL [20] |  33.13 (0.17) | 0.9792
RAHC |  33.54 (0.22) | 0.9865

TABLE VII: Quantitative comparison with the SOTA methods on the Rain1200 [7] dataset. Please reffer to Tab. V for table notes. Method | Rain1200
---|---
|  PSNR | SSIM
DDN [30] |  30.97 | 0.9116
PReNet [80] |  33.17 | 0.9481
RCDNet [12] |  34.08 | 0.9532
Restormer [72] | 34.67 (0.12) | 0.9623
NAFNet [17] | 34.69 (0.08) | 0.9617
RAHC | 34.93 (0.11) | 0.9652
RAHC† |  34.25 | 0.9572
TransWeather [23] |  34.05 (0.16) | 0.9535
AirNet [22] |  34.19 (0.13) | 0.9562
TKL [20] | 34.25 (0.15) | 0.9549
RAHC | 34.71 (0.14) | 0.9641

TABLE VIII: Quantitative comparison with the SOTA methods on the SICE [70] test dataset. Please reffer to Tab. V for table notes. Method | SICE
---|---
|  PSNR | SSIM
RUAS [13] |  15.26 | 0.5481
EnlightenGAN [81] |  17.55 | 0.5915
SCI [82] |  19.26 | 0.6324
KinD [5] | 20.13 (0.09) | 0.6502
MIRNetv2 [83] | 21.37 (0.15) | 0.7203
RAHC | 22.73 (0.14) | 0.7425
RAHC† |  21.09 | 0.6705
TransWeather [23] | 20.42 (0.14) | 0.6973
AirNet [22] | 20.65 (0.18) | 0.6993
TKL [20] | 21.26 (0.16) | 0.7131
RAHC | 21.56 (0.18) | 0.7269

In addition to HAC, we also evaluate the performance of RAHC on five
conventional adverse weather removal datasets: Snow100K [34], Raindrop [10],
SOTS-outdoor [68], Rain1200 [7], and SICE [70]. And only the training sets
from these datasets are utilized to train AdverseGAN, so there is no
information leakage. In addition, real-world images used for training
AdverseGAN are replaced by handcrafted images in the corresponding datasets to
better verify the data distribution simulation performance of AdverseGAN. The
quantitative experimental results are presented in Tab. V, Tab. V, Tab. VIII,
Tab. VIII, and Tab. VIII respectively.

In the tables, we provide the results of four types of experiments: (1) task-
specific setting that is training with task-specific data only; (2) all-in-one
setting that is training with mixed data from all datasets; (3) training with
pure data generated by AdverseGAN; (4) the performance gain from applying
AdverseGAN as a data augmentor.

As can be seen, our RAHC achieves the best scores on all five tasks both in
the task-specific setting and in the all-in-one setting. Take Tab. VIII for
example, our method outperforms previous SOTA TKL [20] by a PSNR margin of up
to 0.3dB. Moreover, with the all-in-one setting, our RAHC is only slightly
inferior to the task-specific results, demonstrating consistent restoration
performance.

To verify the reliability and distribution simulation capability of
AdverseGAN, we trained RAHC only with data generated by AdverseGAN without
original degraded images. As we have seen, RAHC trained from the generated
data achieves competitive results, although tolerably worse than the standard
results, this is negligible compared to the tedious and labor-intensive data
synthesis process. And after all, both qualitative and quantitative
experimental results on the HAC dataset demonstrate the plausibility of
AdverseGAN. The results also indicate that our proposed AdverseGAN is
trustworthy and excellent in simulating the source distribution, thus can
perform as an augmentor to boost performance gains to existing methods.

Apart from the above experiments, we also provide the performance gain from
applying AdverseGAN as an augmentor in parenthesis. Obviously, AdverseGAN
delivers consistent performance gains for both task-specific and all-in-one
approaches. Compared to task-specific approaches, the unified frameworks can
obtain higher performance gains from augmentation, manifesting the importance
of adequate training data for multi-weather learning.

###  4.4 Comparison with the State-of-the-Arts on Real-World Conditions

TABLE IX: Quantitative comparison results of average NIQE/SSEQ on real-world adverse weather conditions. The smaller scores indicate better perceptual quality. Method | TransWeather [23] | AirNet [22] | TKL [20] | RAHC
---|---|---|---|---
NIQE | 6.918 | 7.266 | 5.625 | 4.514
SSEQ | 50.17 | 45.66 | 40.11 | 30.62
Figure 10: Visual comparisons with SOTA adverse conditions restoration methods
on real-world examples.

We conduct additional comparisons on real-world adverse weather conditions to
further verify the realistic reliability of HAC and the robustness of RAHC. We
use NIQE [84] and SSEQ [85] to quantitatively evaluate the reference-free
restoration performance. Quantitative results are shown in Tab. IX. The
smaller scores of SSEQ and NIQE indicate better perceptual quality and clearer
con- tents. It is observed that our RAHC delievers the best average scores on
real-world samples, outperforming the state-of-the-art reatoration methods
[20, 22, 23] by a large margin. Moreover, Fig. 10 exhibits the visual
comparisons on real-world conditions. It can be seen that RAHC and TKL trained
on HAC dataset can comfortably handle real-world hybrid adverse weather
conditions, adequately indicating the realism and effectiveness of our
proposed dataset. Meanwhile, our RAHC produces cleaner and more pleasing
results compared to TKL and TransWeather, which strongly proves the
superiority of our proposed RAHC. Additionally, this experiment also reveals
the truth that real-world scenarios often suffer from multiple superimposed
degradations rather than simple one corruption, and our method can flexibly
restore arbitrary hybrid conditions in one go.

###  4.5 Analyzing Framework RAHC

####  4.5.1 Effect of Basic Schemes

TABLE X: Average PSNR of RAHC on HAC dataset with four variants. RVA and OSD represent reconstruction vectors aided scheme and output space discriminative learning scheme respectively. Variant | PSNR
---|---
RAHC w/o RVA | 21.96
RAHC w/o OSD | 22.41
RAHC w/o RVA&OSD | 21.47
Ours | 22.92

We first investigate the validity of the reconstruction vectors aided scheme
and the output space discriminative learning scheme proposed in this paper.
Tab. X reports the ablations of the two basic schemes, and as we can see, the
PSNR of RAHC w/o OSD decreases by 0.51dB, and a sharp reduction of 0.96dB was
observed when RVA was discarded. The output space discriminative learning
scheme ensures that the network learns degradation-agnostic generic repair
capabilities while the reconstruction vectors aided scheme provides solid
support for the network to cope with hybrid complicated adverse scenarios. The
experimental results strongly demonstrate the indispensability of these two
schemes, which have been instrumental in achieving arbitrary hybrid adverse
weather conditions restoration.

####  4.5.2 Effect of MHBB

TABLE XI: Average PSNR of RAHC on HAC dataset with five variants. Variant | PSNR
---|---
MHBB w/o multi-head | 22.59
CAM w/o convolution path | 22.66
CAM w/o attention path | 22.61
DP-FFN w/o convolution path | 22.71
Ours | 22.92

In order to verify the effectiveness of each pivotal design element in MHBB,
we then conduct experiments over the following variants: (1) MHBB w/o multi-
head; (2) CAM w/o convolution path; (3) CAM w/o attention path; (4) DP-FFN w/o
convolution path; (5) Ours. It is necessary to note that for each variant we
adjust the width of the network to maintain the total number of parameters. As
can be found in Tab. XI, each of the components plays a pivotal role, and
removing any of them causes significant performance degradation. In
particular, the PSNR decreases by 0.33dB when the multi-head mechanism is
eliminated, indicating the profound effect of multiple representation subspace
for degradation-agnostic learning.

Besides, we also provide a feature map comparison of DP-FFN with or without
convolution path in Fig. 11. It is observed that the original FFN is
insensitive to the local detail context and the extracted features ignore the
building texture in the image, instead, our method extracts the rich building
texture structure by parallelizing a convolutional branch afterward. The
results clearly demonstrate that our proposed DP-FFN can better extract the
detailed contexts of interest in image restoration by parallelizing a
convolution path.

Figure 11: Visual effect of DP-FFN with or without convolution path. Top row:
feature maps extracted by the original FFN without convolutional path. Bottom
row: feature maps extracted by the proposed DP-FFN with convolutional path.

####  4.5.3 Effect of Reconstruction Vectors

Figure 12: Visual effect of reconstruction vectors. Visualizations of
reconstruction vectors were obtained by the VQGAN decoder. RV represents
reconstruction vectors.

We also explored the effect of reconstruction vectors, and the visual results
are provided in Fig. 12. As suggested, RAHC without RV tends to produce
ambiguous contexts while our RAHC can restore sharper structural and textural
details. Naturally, recovering clean images from the reconstruction vectors
directly using the VQGAN decoder may be another option, but in this paper, we
only leverage the reconstruction vectors as auxiliary features and let the
network learn how to utilize them on its own. Such strategy can also be
demonstrated by Fig. 12. We can see that the directly restored images
(Visualization of RV) complement the possible textures of the degraded region
but with low fidelity, while our implicit modeling allows the restoration
model to use the visual atoms embedded in the reconstruction vector according
to its own “experience”, hence restoring more realistic and reliable images
with rich details.

Figure 13: Curve comparison on reconstruction vectors location accuracy.

Additionally, we also proffer the accuracy comparison of the proposed mapping
network and direct classification in reconstruction vectors localization. As
shown in Fig. 13, our method exhibits consistently higher accuracy than
classification, not sensitive to high-precision predictions, the close-fitting
reconstruction vectors can provide rich visual cues for the restoration
process, as best demonstrated in Fig. 12.

####  4.5.4 Objective Evaluation of Output Space Discriminative Learning
Scheme

TABLE XII: Average probability predicted by the weather type classifier. A lower confidence level indicates that the recovery results are perceptually closer to expected clean ones. Method | Avg. Probability $(\downarrow)$
---|---
TransWeather [23] | 0.553
AirNet [22] | 0.528
TKL [20] | 0.496
RAHC w/o OSD | 0.452
RAHC (Ours) | 0.231

To verify whether the output space discriminative learning scheme yields
degradation-agnostic and trace-free clean results, we objectively evaluated
the perceptual satisfaction of the restoration results. We first train a
ResNet-101 [86] classifier with half of the data from the HAC test set to
recognize which weather degradation is contained in the image. Then we input
the rest data repaired by the restoration network to the trained classifier to
get the predicted probability. Tab. XII presents the average probability
predicted by the weather type classifier. As we can see, our RAHC restores
more indiscernible images, suggesting a more consistent and trace-free clean
result. When this scheme is eliminated, the probability becomes significantly
higher indicating the indelible role of the proposed learning scheme in the
degradation-agnostic restoration. Despite OSD being removed, the probability
is still lower than other approaches, which can be attributed to the proposed
restoration network that allows the restored image to preserve more structural
and textural details, thus closer to the desired clean result.

####  4.5.5 Feature-Level Discrimination vs. Output Space Discrimination

TABLE XIII: Comparison of feature-level discrimination and output space discrimination. Variant | PSNR | SSIM
---|---|---
None | 22.41 | 0.8652
Feature-Level | 22.71 | 0.8735
Output Space (Ours) | 22.92 | 0.8756

Analogous to the output space discrimination, we further investigate feature-
level discrimination that is input the encoder-extracted features into the
discriminator to distinguish the type of degradation while the restoration
network tries to confuse the discriminator to extract degradation-agnostic
features. Quantitative experimental results are shown in Tab. XIII, as can be
found, both feature-level and output space contribute to learning degradation-
agnostic restoration capacity while our output space is more favored compared
to the counterpart. We conjecture that intermediate features containing high-
level semantics are more prone to perplex the discriminator, which leads to
weaker binding of the discriminator.

####  4.5.6 Universal Output Space Discrimination

TABLE XIV:  Results of applying TKL strategy proposed by Chen _et al._ [20] (top rows) and output space discriminative learning scheme proposed in this study (middle rows) into SOTA universal image restoration methods. Performance gains compared to pure training process are provided in parenthesis.  Method | PSNR | SSIM
---|---|---
MPRNet [15] | 20.26 $\uparrow$ 0.58) | 0.7812 ($\uparrow$ 0.0383)
SwinIR [18] | 21.01 ($\uparrow$ 0.36) | 0.8163 ($\uparrow$ 0.0256)
Uformer [19] | 21.03 ($\uparrow$ 0.28) | 0.8197 ($\uparrow$ 0.0264)
Restormer [16] | 21.14 ($\uparrow$ 0.30) | 0.8156 ($\uparrow$ 0.0172)
NAFNet [17] | 21.22 ($\uparrow$ 0.38) | 0.8215 ($\uparrow$ 0.0230)
MPRNet [15] | 20.33 ($\uparrow$ 0.65) | 0.7866 ($\uparrow$ 0.0437)
SwinIR [18] | 21.09 ($\uparrow$ 0.44) | 0.8213 ($\uparrow$ 0.0306)
Uformer [19] | 21.16 ($\uparrow$ 0.41) | 0.8217 ($\uparrow$ 0.0284)
Restormer [16] | 21.31 ($\uparrow$ 0.47) | 0.8301 ($\uparrow$ 0.0317)
NAFNet [17] | 21.37 ($\uparrow$ 0.53) | 0.8339 ($\uparrow$ 0.0354)
TransWeather [23] | 21.14 | 0.8258
AirNet [22] | 21.27 | 0.8312
TKL [20] | 21.30 | 0.8332
RAHC | 22.92 | 0.8756

As mentioned above, our proposed output space discriminative learning scheme
can be integrated into existing universal image restoration architectures
boosting their performance under all-in-one setting. Similarly, TKL proposed
by Chen _et al._ can also be applied to existing algorithms, and we have
compared the superiority and inferiority of the two approaches under a fair
backbone. As presented in Tab. XIV, all algorithms show a significant
performance improvement under the all-in-one setting when the learning scheme
is equipped which is more obvious and noticeable than the TKL strategy. In
particular, Restormer [16] and NAFNet [17] even surpass the extant state-of-
the-art unified framework TKL [20] with the original backbone in their paper
and our mechanism is more flexible and straightforward compared to the complex
training process of TKL’s two-stage strategy. The experiments provide strong
evidence of the generality, universality, and effectiveness of our proposed
output space discriminative learning scheme.

####  4.5.7 Robust Hybrid Adverse Weather Conditions Restoration

TABLE XV: Comparison of similarity between reconstruction outputs of different degraded versions. S, D, T, Q, and P represent Single, Double, Triple, Quadruple, and Pentuple respectively (e.g., D$\longleftrightarrow$S denotes similarity calculation between the restoration results of the double-degraded versions and the single-degraded versions that correspond to the same clean image).  Participating | TransWeather | TKL | RAHC (ours)
---|---|---|---
Data | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
D$\longleftrightarrow$S | 22.06 | 0.869 | 22.26 | 0.851 | 25.44 | 0.926
T$\longleftrightarrow$S | 18.53 | 0.799 | 19.02 | 0.811 | 22.36 | 0.901
Q$\longleftrightarrow$S | 15.16 | 0.675 | 15.36 | 0.698 | 19.91 | 0.852
P$\longleftrightarrow$S | 12.11 | 0.612 | 13.78 | 0.621 | 18.22 | 0.745

Since our proposed HAC dataset contains different degraded versions of the
same clean image we can further verify the robustness of different restoration
algorithms by comparing the similarity between the restoration results of
different degraded images. Considering that existing methods are designed
primarily for single degradation scenarios, we calculate the similarity
between the restoration results of the hybrid weather degradation versions and
the corresponding single degradation versions. Higher similarity indicates
more robust hybrid adverse weather conditions restoration capability. It can
be observed from Tab. XV that our RAHC-restored hybrid degraded images are
much more similar to the repaired single degraded images, delivering 3.34dB
higher PSNR than TKL [20] on T$\longleftrightarrow$S. Especially when there
are more hybrid species (e.g., P$\longleftrightarrow$S) for which our
superiority is more obvious. This result strongly demonstrates the superiority
of our method in restoring hybrid adverse weather conditions, in particular,
benefiting from the reconstruction vectors aided scheme, which allows RAHC to
extract the image content features from the auxiliary visual cues to produce
more promising results in hybrid scenarios.

####  4.5.8 Effect and Sensitivity Analysis of $\lambda_{dis}$

TABLE XVI: Average PSNR of RAHC on HAC dataset with different $\lambda_{dis}$. $\lambda_{dis}$ | 0.01 | 0.05 | 0.10 | 0.15 | 0.20
---|---|---|---|---|---
PSNR | 22.69 | 22.86 | 22.92 | 22.83 | 22.81

We also conducted a pilot study to substantiate the effect of $\lambda_{dis}$
on the proposed output space discriminative learning scheme. Tab. XVI shows
the final performances of different $\lambda_{dis}$ values. Obviously, the
highest PSNR is yielded when $\lambda_{dis}$ is set to 0.10, and both
decreasing and increasing result in minor performance degradation.
Furthermore, despite there are minor fluctuations, the final performances
remain stable without collapse, implying the stability and facility of the
proposed discriminative learning scheme.

###  4.6 Analyzing Data Generator AdverseGAN

####  4.6.1 Effect of AdverseGAN

TABLE XVII: Average PSNR of RAHC with different adverse weather conditions generators. DD and DS represent dual discriminitor scheme and dual space injection scheme respectively. Generator | PSNR
---|---
pix2pix [78] | 18.33
pix2pixHD [87] | 18.56
CycleGAN [42] | 19.69
StarGAN [88] | 20.75
StarGAN v2 [89] | 20.82
Kim _et al._ [64] | 20.91
AdverseGAN w/o DD&DS | 20.86
AdverseGAN w/o DD | 21.78
AdverseGAN w/o DS | 21.89
AdverseGAN (Ours) | 22.92
Figure 14: Visual comparison of snow scenarios. Our AdverseGAN generates snow
scenes that more closely resemble real-world condition and with consistent
background.

To substantiate the effectiveness of AdverseGAN and the tailored schemes, we
first conduct experiments with SOTA image translation algorithms as well as
variants of AdverseGAN on adverse conditions generation. As can be found in
Tab. XVII, the model trained with data generated by AdverseGAN achieves the
highest PSNR of 22.92 dB, 2.17 dB higher than StarGAN [88]. In addition, the
dual discriminator scheme and dual space injection scheme are indelible
credits to our design which are specifically adapted to the adverse conditions
generation task presented in this paper. These two tailored schemes bring
consistent performance improvements providing solid data support for the
training of restoration networks.

Furthermore, we also provide a visual comparison in Fig. 14, from which it can
be seen that removing the P-Discriminator leads to inconsistent background
perturbations while removing the R-Discriminator results in unrealistic snow.
On the contrary, can generate snow sceneries containing multiple scales of
snow effects with different density distributions at different scales, which
are visually similar to real snow scenes and significantly more realistic than
hand-made conditions.

####  4.6.2 Disentanglement of Style and Content

Figure 15: Style interpolation and content interpolation. $s$ code and $c$
code are sampled from style space and content space respectively.

Interpolation of style and content are shown in Fig. 15. As we can see, the
$s$ code tend to control the form and direction of the weather degradations
while the $c$ code are mainly responsible for the position and density of the
weather degradations. The disentanglement of style and content can better
control the generation of adverse conditions and therefore produce richer and
more diverse data.

####  4.6.3 The Amount of Training Data

TABLE XVIII: Average PSNR of RAHC on HAC dataset with different amount of training data. Amount | $31K$ | $62K$ | $124K$ | $186K$ | $248K$ | $310K$
---|---|---|---|---|---|---
PSNR | 13.54 | 16.75 | 19.65 | 21.06 | 22.43 | 22.92

We also substantiate the effect of different amounts of training data on the
final performance of the model, and the experimental results are reported in
Tab. XVIII. One can see that more training data can always lead to higher PSNR
values, except that the frequency of growth continues to slow as the volume
increases. In this paper, there are $310K$ training data in HAC, but as we can
see from the trend, continuing to increase the training data will bring
further performance improvements, only more computational resources are
required.

###  4.7 Performance vs. Complexity

Figure 16:  Avergae PSNR vs. computational cost and parameter quantity on our
proposed HAC dataset. The proposed RAHC obtains state-of-the-art performance
with competitive computational cost and parameters.

As it can be seen from Fig. 16, our RAHC obtains state-of-the-art performance
at a moderate complexity and outperforms existing methods by a large margin.
As a comparison, MIRNet [35], Restormer [16], and NAFNet [17] can only deal
with single degradation scenarios, while AirNet [22] does require complicated
additional regulation branch and TKL [20] suffers from two-stage training
bringing unwanted resource consumption.

###  4.8 Results on High-level Application

TABLE XIX: Quantitative results in mean intersection over union (mIoU) on ACDC [90] dataset with five adverse weather conditions (Both rain streaks and raindrops are included in Rain). Method | Fog | Night | Rain | Snow
---|---|---|---|---
Original | 43 | 11 | 37 | 29
Restormer [16] | 46 | 27 | 39 | 31
TransWeather [23] | 47 | 29 | 40 | 33
TKL [20] | 49 | 27 | 41 | 34
RAHC (Ours) | 54 | 36 | 45 | 36

To demonstrate the performance of the proposed restoration algorithm in real-
world vision systems, we evaluate the effect of the restoration process on the
popular adverse conditions segmentation dataset ACDC [90] with a pre-trained
DeepLabv3+ [91]. As can be seen from Tab. XIX, the unified framework TKL [20],
TransWeather [23] and our RAHC achieve better results than single weather
removal algorithm Restormer [16]. This phenomenon once again reveals the truth
that real-world adverse weather conditions are sophisticated, unpredictable,
and often the results of a mixture of weather factors. Single weather removal
algorithm Restormer [16] can only remove specific weather at once, while
unified frameworks can restore the complex scenario adaptively generating more
appealing results. In addition, thanks to the application of reconstruction
vectors and the output space constraint, our proposed RAHC attains
significantly higher mIoU compared to TKL [20] and TransWeather [23]. This
result also indicates the necessity and effectiveness of our proposed
arbitrary hybrid adverse weather conditions restoration issue from the side.
We also provide visual results of segmentation in Fig. 17. As we can see, our
approach can better understand images in adverse conditions and more
accurately identify the semantics of different regions.

Figure 17: Visual comparison of segmentation for driving scene understanding
on ACDC [90] dataset between TKL [20] and our RAHC.

##  5 Discussion

Figure 18: Real-world hybrid adverse weather condition restoration results.
Top row: an example with rain streaks, haze, and night. Bottom row: an example
with rain streaks, snow, and night.

Fig. 18 illustrates the restoration results of two real-world hybrid weather
conditions. It can be easily noticed that processing each weather individually
is unable to fully restore the degraded images, while our method can remove
all degradations in one go to restore more pleasing high quality images.
Besides, our RAHC can handle not only randomly occurring hybrid adverse
weather conditions but also single weather scenarios robustly, thus our
approach can be better deployed to tackle complex and diverse real-world
conditions (without losing the ability to restore single weather degradations
while can handle hybrid weather scenarios that are neglected by existing
approaches). We believe the exploration in this study should be meaningful and
valuable to real-world weather conditions restoration.

##  6 Limitations and Future Work

A common limitation shared with other restoration algorithms is the strong
reliance on large-scale paired data. As stated in the main paper, there are
$310K$ pairs of data for model training. Fortunately, thanks to the proposed
AdverseGAN, the labor cost, time cost, and capital cost is tremendously
compressed. On the other hand, RAHC can handle arbitrary hybrid conditions
combined by five common weather, while other rare adverse weather like frost
on glass, sand and dust has remained to be addressed. In the future, we will
focus on training models that can handle complicated hybrid conditions with
single degraded data, which is extremely challenging but meaningful,
especially when more weather types are taken into account, whose number of
hybrid conditions grows exponentially. In addition, we will also explore more
real-world nighttime adverse weather conditions simulation strategy in future
research, which hopefully training more robust and practical image restoration
models.

##  7 Concluding Remarks

In this paper, we proposed a novel unified framework, namely RAHC to restore
arbitrary hybrid adverse weather conditions in one go. In contrast to existing
frameworks, RAHC can handle severely deteriorated scenarios suffered from
hybrid weather degradations and restore arbitrary hybrid conditions with a
single trained model by a concise and flexible scheme. Meanwhile, MHBB
provides comprehensive degradation characterization and representation support
for multiple simultaneous weather. In addition, we propose a hybrid adverse
conditions generation pipeline, based on which sufficient training data can be
generated cost-effectively. And then finally established HAC dataset contains
$\sim\\!316K$ image pairs of 31 weather conditions, which richness, diversity,
and adequacy render it a competent evaluator. Extensive experiments on our HAC
and conventional datasets manifest the effectiveness, superiority, and
robustness of RACH. We expect this work to provide insights into arbitrary
hybrid adverse conditions restoration and steer future research on this
Gordian knot.

## References

  * [1] M. Liang, B. Yang, S. Wang, and R. Urtasun, “Deep continuous fusion for multi-sensor 3d object detection,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 641–656.
  * [2] A. Prakash, K. Chitta, and A. Geiger, “Multi-modal fusion transformer for end-to-end autonomous driving,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 7077–7087.
  * [3] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum pointnets for 3d object detection from rgb-d data,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 918–927.
  * [4] A. G. Perera, Y. Wei Law, and J. Chahl, “Uav-gesture: A dataset for uav control and gesture recognition,” in _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , 2018, pp. 0–0.
  * [5] Y. Zhang, J. Zhang, and X. Guo, “Kindling the darkness: A practical low-light image enhancer,” in _Proceedings of the 27th ACM international conference on multimedia_ , 2019, pp. 1632–1640.
  * [6] X. Liu, Y. Ma, Z. Shi, and J. Chen, “Griddehazenet: Attention-based multi-scale network for image dehazing,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 7314–7323.
  * [7] H. Zhang and V. M. Patel, “Density-aware single image de-raining using a multi-stream dense network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 695–704.
  * [8] Y. Quan, S. Deng, Y. Chen, and H. Ji, “Deep learning for seeing through window with raindrops,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2463–2471.
  * [9] W.-T. Chen, H.-Y. Fang, C.-L. Hsieh, C.-C. Tsai, I. Chen, J.-J. Ding, S.-Y. Kuo _et al._ , “All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 4196–4205.
  * [10] R. Qian, R. T. Tan, W. Yang, J. Su, and J. Liu, “Attentive generative adversarial network for raindrop removal from a single image,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2482–2491.
  * [11] H. Dong, J. Pan, L. Xiang, Z. Hu, X. Zhang, F. Wang, and M.-H. Yang, “Multi-scale boosted dehazing network with dense feature fusion,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2157–2167.
  * [12] H. Wang, Q. Xie, Q. Zhao, and D. Meng, “A model-driven deep neural network for single image rain removal,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3103–3112.
  * [13] R. Liu, L. Ma, J. Zhang, X. Fan, and Z. Luo, “Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 561–10 570.
  * [14] M.-W. Shao, L. Li, D.-Y. Meng, and W.-M. Zuo, “Uncertainty guided multi-scale attention network for raindrop removal from a single image,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 4828–4839, 2021.
  * [15] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 14 821–14 831.
  * [16] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5728–5739.
  * [17] L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” _arXiv preprint arXiv:2204.04676_ , 2022.
  * [18] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1833–1844.
  * [19] Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17 683–17 693.
  * [20] W.-T. Chen, Z.-K. Huang, C.-C. Tsai, H.-H. Yang, J.-J. Ding, and S.-Y. Kuo, “Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward a unified model,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17 653–17 662.
  * [21] R. Li, R. T. Tan, and L.-F. Cheong, “All in one bad weather removal using architectural search,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 3175–3185.
  * [22] B. Li, X. Liu, P. Hu, Z. Wu, J. Lv, and X. Peng, “All-In-One Image Restoration for Unknown Corruption,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17 452–17 462.
  * [23] J. M. J. Valanarasu, R. Yasarla, and V. M. Patel, “Transweather: Transformer-based restoration of images degraded by adverse weather conditions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2353–2363.
  * [24] O. Özdenizci and R. Legenstein, “Restoring vision in adverse weather conditions with patch-based denoising diffusion models,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [25] R. Li, L.-F. Cheong, and R. T. Tan, “Heavy rain image restoration: Integrating physics model and conditional adversarial learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1633–1642.
  * [26] R. Quan, X. Yu, Y. Liang, and Y. Yang, “Removing raindrops and rain streaks in one go,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9147–9156.
  * [27] Y. Wan, Y. Cheng, M. Shao, and J. Gonzàlez, “Image rain removal and illumination enhancement done in one go,” _Knowledge-Based Systems_ , p. 109244, 2022.
  * [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [29] P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 12 873–12 883.
  * [30] X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removing rain from single images via a deep detail network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3855–3863.
  * [31] X. Qin, Z. Wang, Y. Bai, X. Xie, and H. Jia, “Ffa-net: Feature fusion attention network for single image dehazing,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 11 908–11 915.
  * [32] H. Wu, Y. Qu, S. Lin, J. Zhou, R. Qiao, Z. Zhang, Y. Xie, and L. Ma, “Contrastive learning for compact single image dehazing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 551–10 560.
  * [33] Y. Song, Z. He, H. Qian, and X. Du, “Vision transformers for single image dehazing,” _arXiv preprint arXiv:2204.03883_ , 2022.
  * [34] Y.-F. Liu, D.-W. Jaw, S.-C. Huang, and J.-N. Hwang, “Desnownet: Context-aware deep network for snow removal,” _IEEE Transactions on Image Processing_ , vol. 27, no. 6, pp. 3064–3073, 2018.
  * [35] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Learning enriched features for real image restoration and enhancement,” in _European Conference on Computer Vision_. Springer, 2020, pp. 492–511.
  * [36] L. Chen, X. Lu, J. Zhang, X. Chu, and C. Chen, “Hinet: Half instance normalization network for image restoration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 182–192.
  * [37] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler, “Datasetgan: Efficient labeled data factory with minimal human effort,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 145–10 155.
  * [38] Z. Yang, Y. Chai, D. Anguelov, Y. Zhou, P. Sun, D. Erhan, S. Rafferty, and H. Kretzschmar, “Surfelgan: Synthesizing realistic sensor data for autonomous driving,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 118–11 127.
  * [39] H. Wang, Z. Yue, Q. Xie, Q. Zhao, Y. Zheng, and D. Meng, “From rain generation to rain removal,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 791–14 801.
  * [40] Z. Liu, H. Yin, X. Wu, Z. Wu, Y. Mi, and S. Wang, “From shadow generation to shadow removal,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4927–4936.
  * [41] Z. Yue, Q. Zhao, L. Zhang, and D. Meng, “Dual adversarial network: Toward real-world noise removal and noise generation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 41–58.
  * [42] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [43] Y. Wei, Z. Zhang, Y. Wang, M. Xu, Y. Yang, S. Yan, and M. Wang, “Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 4788–4801, 2021.
  * [44] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas, “Videogpt: Video generation using vq-vae and transformers,” _arXiv preprint arXiv:2104.10157_ , 2021\.
  * [45] J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu, “Vector-quantized image modeling with improved vqgan,” _arXiv preprint arXiv:2110.04627_ , 2021.
  * [46] A. Razavi, A. Van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with vq-vae-2,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [47] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 684–10 695.
  * [48] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” in _International conference on machine learning_. Pmlr, 2018, pp. 1989–1998.
  * [49] G. Kang, Y. Wei, Y. Yang, Y. Zhuang, and A. Hauptmann, “Pixel-level cycle association: A new perspective for domain adaptive semantic segmentation,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 3569–3580, 2020.
  * [50] Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim, “Image to image translation for domain adaptation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4500–4509.
  * [51] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker, “Learning to adapt structured output space for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7472–7481.
  * [52] Y. Luo, P. Liu, L. Zheng, T. Guan, J. Yu, and Y. Yang, “Category-level adversarial adaptation for semantic segmentation using purified features,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [53] Y. Luo, L. Zheng, T. Guan, J. Yu, and Y. Yang, “Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2507–2516.
  * [54] Y. Zou, Z. Yu, B. Kumar, and J. Wang, “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 289–305.
  * [55] K. Zhang, Y. Li, J. Liang, J. Cao, Y. Zhang, H. Tang, R. Timofte, and L. Van Gool, “Practical blind denoising via swin-conv-unet and data synthesis,” _arXiv preprint arXiv:2203.13278_ , 2022.
  * [56] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” _arXiv preprint arXiv:2005.08100_ , 2020.
  * [57] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1874–1883.
  * [58] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov _et al._ , “The open images dataset v4,” _International Journal of Computer Vision_ , vol. 128, no. 7, pp. 1956–1981, 2020.
  * [59] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European conference on computer vision_. Springer, 2016, pp. 694–711.
  * [60] X. Pan, A. Tewari, L. Liu, and C. Theobalt, “Gan2x: Non-lambertian inverse rendering of image gans,” _arXiv preprint arXiv:2206.09244_ , 2022.
  * [61] Y. Xu, Y. Yin, L. Jiang, Q. Wu, C. Zheng, C. C. Loy, B. Dai, and W. Wu, “Transeditor: Transformer-based dual-space gan for highly controllable facial editing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7683–7692.
  * [62] G. Kwon and J. C. Ye, “Diagonal attention and style-based gan for content-style disentanglement in image generation and translation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 13 980–13 989.
  * [63] Y. Alharbi and P. Wonka, “Disentangled image generation through structured noise injection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5134–5142.
  * [64] K. Kim, S. Park, E. Jeon, T. Kim, and D. Kim, “A style-aware discriminator for controllable image translation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 239–18 248.
  * [65] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves _et al._ , “Conditional image generation with pixelcnn decoders,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [66] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8110–8119.
  * [67] A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis with auxiliary classifier gans,” in _International conference on machine learning_. PMLR, 2017, pp. 2642–2651.
  * [68] B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, “Benchmarking single-image dehazing and beyond,” _IEEE Transactions on Image Processing_ , vol. 28, no. 1, pp. 492–505, 2018.
  * [69] W. Wei, D. Meng, Q. Zhao, Z. Xu, and Y. Wu, “Semi-supervised transfer learning for image rain removal,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3877–3886.
  * [70] J. Cai, S. Gu, and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” _IEEE Transactions on Image Processing_ , vol. 27, no. 4, pp. 2049–2062, 2018.
  * [71] Y. P. Loh and C. S. Chan, “Getting to know low-light images with the exclusively dark dataset,” _Computer Vision and Image Understanding_ , vol. 178, pp. 30–42, 2019.
  * [72] X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley, “Clearing the skies: A deep network architecture for single-image rain removal,” _IEEE Transactions on Image Processing_ , vol. 26, no. 6, pp. 2944–2956, 2017.
  * [73] C. Wei, W. Wang, W. Yang, and J. Liu, “Deep retinex decomposition for low-light enhancement,” _arXiv preprint arXiv:1808.04560_ , 2018.
  * [74] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv preprint arXiv:1608.03983_ , 2016.
  * [75] Q. Huynh-Thu and M. Ghanbari, “Scope of validity of psnr in image/video quality assessment,” _Electronics letters_ , vol. 44, no. 13, pp. 800–801, 2008.
  * [76] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [77] K. Zhang, R. Li, Y. Yu, W. Luo, and C. Li, “Deep dense multi-scale network for snow removal using semantic and depth priors,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 7419–7431, 2021.
  * [78] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [79] X. Liu, M. Suganuma, Z. Sun, and T. Okatani, “Dual residual networks leveraging the potential of paired operations for image restoration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7007–7016.
  * [80] D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng, “Progressive image deraining networks: A better and simpler baseline,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3937–3946.
  * [81] Y. Jiang, X. Gong, D. Liu, Y. Cheng, C. Fang, X. Shen, J. Yang, P. Zhou, and Z. Wang, “Enlightengan: Deep light enhancement without paired supervision,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 2340–2349, 2021.
  * [82] L. Ma, T. Ma, R. Liu, X. Fan, and Z. Luo, “Toward fast, flexible, and robust low-light image enhancement,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5637–5646.
  * [83] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Learning enriched features for fast image restoration and enhancement,” _arXiv preprint arXiv:2205.01649_ , 2022.
  * [84] A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” _IEEE Signal processing letters_ , vol. 20, no. 3, pp. 209–212, 2012.
  * [85] L. Liu, B. Liu, H. Huang, and A. C. Bovik, “No-reference image quality assessment based on spatial and spectral entropies,” _Signal processing: Image communication_ , vol. 29, no. 8, pp. 856–863, 2014.
  * [86] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [87] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8798–8807.
  * [88] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8789–8797.
  * [89] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, “Stargan v2: Diverse image synthesis for multiple domains,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8188–8197.
  * [90] C. Sakaridis, D. Dai, and L. Van Gool, “Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 765–10 775.
  * [91] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.

|  Yecong Wan is a B.Eng. at College of Computer Science and Technology, China
University of Petroleum (East China), under the supervision of Prof. Shao. His
current research interests include image restoration and computer vision.
---|---
|  Mingwen Shao received his M.S. degree in mathematics from the Guangxi
University, Guangxi, China, in 2002, and the Ph.D. degree in applied
mathematics from Xi’an Jiaotong University, Xi’an, China, in 2005. He received
the postdoctoral degree in control science and engineering from Tsinghua
University in February 2008. Now he is a professor and doctoral supervisor at
China University of Petroleum (East China). His research interests include
machine learning, computer vision, and data mining.
---|---
|  Yuanshuo Cheng is a B.Eng. at College of Computer Science and Technology,
China University of Petroleum (East China), under the supervision of Prof.
Shao. His current research interests include image restoration, computer
vision, and deep learning.
---|---
|  Yuexian Liu is a B.Eng. at College of Computer Science and Technology,
China University of Petroleum (East China), under the supervision of Prof.
Shao. His current research interests include image restoration and computer
vision.
---|---
|  Zhiyuan Bao is a master student at College of Computer Science and
Technology, China University of Petroleum (East China), under the supervision
of Prof. Shao. He received the bachelor of software engineering from the China
University of Petroleum (East China). His research interests include image
restoration and computer vision.
---|---
|  Deyu Meng received the B.Sc., M.Sc., and Ph.D. degrees from Xi’an Jiaotong
University, Xi’an, China, in 2001, 2004, and 2008, respectively. He was a
Visiting Scholar with Carnegie Mellon University, Pittsburgh, PA, USA, from
2012 to 2014. He is currently a Professor with the Institute for Information
and System Sciences, Xi’an Jiaotong University. His current research interests
include self-paced learning, noise modeling, and tensor sparsity.
---|---