# Masked Unsupervised Self-training for Label-free Image Classification

Junnan Li, Silvio Savarese, Steven Hoi
Salesforce Research
<EMAIL_ADDRESS>
###### Abstract

State-of-the-art computer vision models are mostly trained with supervised
learning using human-labeled images, which limits their scalability due to the
expensive annotation cost. While self-supervised representation learning has
achieved impressive progress, it still requires a second stage of finetuning
on labeled data. On the other hand, models pre-trained with large-scale text-
image supervision (e.g., CLIP) have enabled zero-shot transfer to downstream
image classification tasks. However, the zero-shot performance of CLIP-like
models are often insufficient for real-world adoption. In this paper, we aim
to leverage the abundant unlabeled data from a target domain to improve the
performance of a pre-trained zero-shot classifier, by unsupervised finetuning
of the pre-trained model. We propose Masked Unsupervised Self-Training (MUST),
a new unsupervised adaptation method which leverages two different and
complementary sources of training signals: pseudo-labels and raw images. MUST
jointly optimizes three objectives to learn both class-level global feature
and pixel-level local feature and enforces a regularization between the two.
We demonstrate the efficacy of MUST on a variety of downstream tasks, where it
improves upon CLIP by a large margin. MUST also outperforms supervised few-
shot adaptation methods. It achieves a top-1 accuracy of 77.7% on ImageNet
using ViT-B, +9.4% higher than CLIP, and +6.2% higher than 16-shot CLIP
adaptation. Our code is available at https://github.com/salesforce/MUST.

##  1 Introduction

Zero-shot image classification is a challenging goal that marks the capability
of a vision model to solve tasks without human supervision. Recently, vision-
language pre-training (e.g. CLIP (Radford et al., 2021)) has shown promising
performance on open-vocabulary zero-shot classification, where it leverages
web-scale image-text pairs to train image and text encoders that can be
transferred to downstream tasks through natural language prompting. However,
the zero-shot performance of CLIP is often inadequate for real-world
adoptions, especially when compared to models that are trained with supervised
learning. On the other hand, there are abundant unlabeled data available for
many tasks. In this paper, we aim to improve the performance of an open-
vocabulary zero-shot classifier by finetuning it on unlabeled images from a
downstream task.

Given unlabeled data, the key question is: what is the source of supervision?
Numerous papers have attempted to answer this question. Among them, self-
training and self-supervised learning are two of the most dominant approaches.
In self-training (Zoph et al., 2020; Xie et al., 2020; Lee et al., 2013),
pseudo-labels are generated by a teacher model and then used to supervise
task-specific training of a student model, where the student usually has an
equal model size as the teacher. On the other hand, self-supervised learning
methods are generally task-agnostic. Masked image modeling (Bao et al., 2022;
He et al., 2021; Xie et al., 2022), which trains the model to predict missing
information from masked image patches, has recently emerged as the superior
self-supervised learning method for vision transformers (ViT) (Dosovitskiy et
al., 2021). However, both self-training and self-supervised learning have
their limitations. Self-training overly relies on the pseudo-labels as the
only source of supervision, thus is prone to overfitting to the noise in
pseudo-labels. Self-supervised learning requires an additional stage of task-
specific finetuning on labeled data, thus is not a one-stop solution.

In this paper, we propose Masked Unsupervised Self-Training (MUST), a simple
and effective method for label-free image classification. MUST performs
unsupervised learning using both pseudo-labels and raw images as two different
and complementary training signals. Specifically, MUST jointly optimizes three
objectives to finetune a pre-trained classification model (e.g. CLIP) on
unlabeled images: (1) Self-training objective to learn global task-specific
class prediction; (2) Masked image modeling objective to learn local pixel-
level information; (3) Global-local feature alignment objective to bridge the
knowledge learned from the two sources of supervision.

We validate the efficacy of MUST on 8 image classification tasks across a
variety of domains, showing significant improvement over CLIP (Radford et al.,
2021). MUST also outperforms supervised few-shot adaptation methods (Zhou et
al., 2021; 2022). For instance, MUST achieves 77.7% top-1 accuracy on
ImageNet, +9.4% higher than CLIP, and +6.2% higher than 16-shot CLIP
adaptation. On certain domains, MUST can achieve comparable performance with a
fully-supervised method. We further perform extensive quantitative and
qualitative analysis to examine the effect of each proposed component. MUST is
a low-cost solution for image classification that unlocks the potential of
CLIP-like models for practical scenarios where images are abundant but labels
are scarce.

##  2 Related Work

Zero-shot learning traditionally aims to recognize unseen classes by training
the model on base classes (Xian et al., 2017; Wang et al., 2019), where the
most common approach is to utilize auxiliary information such as attributes
(Huynh & Elhamifar, 2020) or knowledge graphs (Wang et al., 2018). CLIP
(Radford et al., 2021) popularizes a new approach for open-vocabulary zero-
shot image classification by leveraging natural language supervision from web-
scale datasets. Despite its impressive performance, the zero-shot accuracy of
CLIP stills falls far below the supervised method in many domains. Some recent
work tries to adapt CLIP to downstream tasks using labeled data (Zhou et al.,
2021; Gao et al., 2021), which is less scalable compared to our unsupervised
adaptation method. Our method is also orthogonal to CLIP-like research in
model pre-training and can be applied to other image classification models
(Jia et al., 2021; Li et al., 2022; Yao et al., 2022; Li et al., 2021), such
as LiT (Zhai et al., 2022) which performs two-stage pre-training.

Self-training has shown promising progress in many domains including vision
(Zoph et al., 2020; Xie et al., 2020; Sahito et al., 2022), NLP (He et al.,
2020a), and speech (Kahn et al., 2020). Our method is more closely related to
the self-training approaches proposed for semi-supervised learning (Tarvainen
& Valpola, 2017; Sohn et al., 2020; Berthelot et al., 2020), where pseudo-
labels on unlabeled data are used as training targets. We construct our self-
training objective by following three principles: (1) consistency
regularization (Sohn et al., 2020; Laine & Aila, 2017) which enforces the
model to output the same prediction when the input is perturbed; (2) entropy
minimization (Grandvalet & Bengio, 2004) which encourages the model to give
“sharp” predictions with low entropy; (3) prediction fairness (Berthelot et
al., 2020) which alleviates the model’s bias towards certain classes. Our
method leverages a separate supervision signal from raw image pixels to reduce
the confirmation bias in self-training, which is orthogonal to methods for
pseudo-label debiasing (Wei et al., 2021; Wang et al., 2022).

Masked image modeling, fueled by the success of vision transformers
(Dosovitskiy et al., 2021), recently emerged as a more appealing self-
supervised representation learning method over contrastive learning (He et
al., 2020b; Chen et al., 2020; Hjelm et al., 2019). While some methods train
the model to predict discrete tokens (Bao et al., 2022) or contextualized
representations (Baevski et al., 2022) for masked image patches, MAE (He et
al., 2021) and SimMIM (Xie et al., 2022) achieve competitive performance by
simply predicting the pixel values. The MIM objective has also been used for
test-time training (Gandelsman et al., 2022). Different from existing self-
supervised learning methods which require an additional stage of supervised
finetuning on labeled data, we synergically incorporate masked image modeling
into self-training as a one-stage solution for zero-shot image classification.

##  3 Method

Figure 1:  On each downstream task, we convert CLIP’s text embeddings into a
linear classifier, which is fine-tuned together with the image encoder for
unsupervised adaptation.

MUST is a simple unsupervised learning approach that adapts a pre-trained
open-vocabulary classifier to a downstream task using unlabeled images. In
this paper, we consider vision transformers pre-trained by CLIP (Radford et
al., 2021) as the models to be adapted due to their distinctive zero-shot
performance. CLIP pre-trains an image encoder and a text encoder with a
contrastive loss such that paired images and texts have higher similarities in
a shared embedding space. In order to perform zero-shot classification, CLIP
converts a set of class names into text embeddings using ensemble of natural
language prompts (e.g., a photo of a {object}). During inference, it takes the
dot-product between an image embedding and all text embeddings to produce the
prediction logits for that image. As shown in Figure 1, we convert CLIP’s non-
parametric text embeddings into weights of a linear classifier, and directly
finetune the linear classifier together with the image encoder for
unsupervised adaptation.

Figure 2 shows the overall framework of MUST. Given an image, we follow ViT
(Dosovitskiy et al., 2021) to divide it into regular non-overlapping patches.
A [CLS] token is appended to extract global information, which is used by the
classifier for prediction. We then randomly mask image patches by replacing a
patch’s embedding with a learnable [MSK] token. The output embeddings of [CLS]
and [MSK] are used to jointly optimize three objectives: (1) global self-
training, (2) local masked image modeling, and (3) global-local feature
alignment. Next we delineate the details of each objective.

###  3.1 Self-training with an EMA Teacher

The self-training objective is applied to the classifier’s output. Given a
batch of $B$ unlabeled images, we compute a pseudo-label for each image by
passing a weakly-augmented version of the image to a teacher model. The
teacher is parameterized by an exponentially moving average (EMA) (Tarvainen &
Valpola, 2017; Grill et al., 2020) of the model parameters $\theta$.
Specifically, the parameters of the EMA teacher $\Delta$ are intialized as
$\theta$. During each update of $\theta$, $\Delta$ are updated with

$\Delta=\mu\Delta+(1-\mu)\theta$ (1)

Let $q_{b}$ denote the EMA teacher’s softmax prediction for the weakly-
augmented image, we enforce a cross-entropy loss against the model’s
prediction $p_{b}$ for a strongly-augmented version of the same image:

$\mathcal{L}_{\mathrm{cls}}=\frac{1}{B}\sum_{b=1}^{B}\mathbbm{1}(\max
q_{b}\geq\tau)\mathrm{H}(\hat{q}_{b},p_{b})$ (2)

Following FixMatch (Sohn et al., 2020), we only use pseudo-labels with maximum
scores above a threshold $\tau$, and convert the soft labels $q_{b}$ into
“one-hot” hard labels by $\hat{q}_{b}=\operatorname*{arg\,max}(q_{b})$.

Since the pseudo-labels generated by a CLIP model are often biased towards
certain classes (Wang et al., 2022), minimizing $\mathcal{L}_{\mathrm{cls}}$
alone would magnify the bias. To mitigate confirmation bias, we introduce a
“fairness” regularization which encourages that on average, across a batch of
samples, the model’s prediction probability is close to a uniform
distribution:

$\mathcal{L}_{\mathrm{reg}}=-\frac{1}{K}\sum_{k=1}^{K}\log(\bar{p}_{k}),$ (3)

where $K$ is the total number of classes, and $\bar{p}$ is the model’s average
prediction across the batch. In cases where $K>B$, we compute $\bar{p}$ using
moving average instead of batch average. We find this regularization to be
beneficial even for datasets with long-tailed class distribution (see Table
7).

Figure 2:  Unsupervised learning framework of MUST. We randomly replace image
patches with [MSK] tokens, and jointly optimize three objectives on unlabeled
images. (1) Global self-training: the classifier uses the output of [CLS] to
predict the pseudo-label produced by an EMA model. (2) Masked image modeling:
the linear decoder uses the outputs of [MSK] to predict the pixel values of
the masked patches. (3) Global-local feature alignment: minimize the distance
between [CLS] and [MSK] in a normalized embedding space.

###  3.2 Masked Image Modeling

To alleviate the over-reliance on the noisy pseudo-labels in self-training, we
introduce another source of supervision obtained from the raw images. The
masked image modeling (MIM) objective aims to learn local image representation
at masked positions by predicting the missing information using contextual
patches. We follow SimMIM (Xie et al., 2022) and simply predict the RGB pixel
values for masked patches. Specifically, given the output embedding
$z_{b}^{m}$ of the $m$-th [MSK] token, we first pass it through a linear
decoder head to obtain the predicted RGB values $y_{b}^{m}\in\mathbb{R}^{N}$
for that patch, where $N$ denotes the number of RGB pixels per patch. Then we
compute the MIM loss as an $\ell_{1}$-loss between $y_{b}^{m}$ and the ground-
truth RGB values $x_{b}^{m}$:

$\mathcal{L}_{\mathrm{mim}}=\frac{1}{BMN}\sum_{b=1}^{B}\sum_{m=1}^{M}\lVert
y_{b}^{m}-x_{b}^{m}\rVert_{1},$ (4)

where $M$ denotes the number of masked patches per image. We employ the patch-
aligned random masking strategy (Xie et al., 2022) where multiple $s\times s$
patches are randomly masked with a fix masking ratio for each dataset. In
general, we find that a low masking ratio (e.g., 10%) works well for most
datasets, which is consistent to the observations in Xie et al. (2022).

###  3.3 Global-local Feature Alignment

We aim to bridge the two sources of supervision (i.e., pseudo-labels and image
pixels) such that the local [MSK] features learned from MIM can improve the
global [CLS] feature for better classification performance. Let $z_{b}^{c}$
denote the output embedding of the [CLS] token, and $v_{b}^{c}=h(z_{b}^{c})$
denote its normalized embedding after the projection network $h$. Following
CLIP, $h$ is a linear layer followed by $\ell_{2}$ normalization, and
$v_{b}^{c}$ is subsequently used by the classifier to produce the prediction
$p_{b}$ for self-training (see Section 3.1). We also project the embeddings of
the [MSK] tokens to the same space by passing them through $h$:
$v_{b}^{m}=h(z_{b}^{m})$. The global-local feature alignment loss is defined
as the average squared distance between the normalize embeddings of the [CLS]
token and all [MSK] tokens for each image:

$\mathcal{L}_{\mathrm{align}}=\frac{1}{BM}\sum_{b=1}^{B}\sum_{m=1}^{M}\lVert
v_{b}^{c}-v_{b}^{m}\rVert_{2}^{2}$ (5)

During training, MUST jointly optimizes the above three objectives. The
overall loss is

$\mathcal{L}=\mathcal{L}_{\mathrm{cls}}+\lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}+\mathcal{L}_{\mathrm{mim}}+\lambda_{\mathrm{align}}\mathcal{L}_{\mathrm{align}}$
(6)

###  3.4 Implementation Details

We experiment with two ViT models pre-trained by Radford et al. (2021):
ViT-B/16 and ViT-L/14, containing respectively 12 and 24 Transformer blocks
with 768 and 1024 hidden dimensions. The [MSK] token and linear decoder head
are randomly initialized and finetuned together with the entire model. During
finetuning, we use AdamW (Loshchilov & Hutter, 2017) optimizer with a weight
decay of 0.05. We employ a cosine learning rate schedule without any warmup.
Following Bao et al. (2022); He et al. (2021), we use a layer-wise learning
rate decay (Clark et al., 2020) of 0.65 for both ViT models. The batch size is
1024 for ViT-B/16 and 512 for ViT-L/14, and the learning rate is scaled
linearly with the batch size
($lr=\text{base\\_lr}\times\text{batchsize}/256$). We use 16 A100 GPUs, and
the training process of MUST adds only a small amount of computation overhead
after CLIP pre-training. We observe negligible variance on the results between
runs with different random seeds.

The model receives input images of size $224\times 224$. During training, we
use RandomResizedCrop+Flip+RandAug (Cubuk et al., 2020) to augment input
images, while using Resize+RandomCrop as the weak augmentation to generate
pseudo-labels. During test, we simply take a center crop after resizing the
shorter edge of the image to 224. For the EMA teacher, we follow Baevski et
al. (2022) and linearly ramp-up the parameter decay rate $\mu$ from $\mu_{0}$
to 0.9998 in $\mu_{n}$ iterations. Table 10 provides more details of the
hyperparameters used for each downstream task, including the pseudo-label
threshold, mask patch size, mask ratio, etc.

##  4 Experiments

###  4.1 Transfer Learning Datasets

Dataset | Train size | Test size | Metric | Description
---|---|---|---|---
ImageNet (Deng et al., 2009) | 1,281,167 | 50,000 | acc | 1000 categories of objects from WordNet
SUN397 (Xiao et al., 2010) | 76,129 | 21,758 | acc | 397 categories of indoor and outdoor scenes
Food101 (Bossard et al., 2014) | 75,750 | 25,250 | acc | 101 categories of food dishes
GTSRB (Stallkamp et al., 2011) | 26,640 | 12,630 | acc | 43 categories of German traffic signs
DTD (Cimpoi et al., 2014) | 3,760 | 1,880 | acc | 47 categories of describable textures
UCF101 (Soomro et al., 2012) | 9,537 | 3,783 | acc | 101 categories of human action video frames
Oxford Pets (Parkhi et al., 2012) | 3,680 | 3,669 | mean per class | 37 categories of cats and dogs
Caltech101 (Binh, 2011) | 3,060 | 6,085 | mean per class | 101 categories of objects and 1 background

Table 1:  Information on the image classification datasets used for transfer
learning.

We perform experiments on 8 image classification datasets which span many
different domains including common objects (Deng et al., 2009; Binh, 2011),
fine-grained animals (Parkhi et al., 2012), indoor and outdoor scenes (Xiao et
al., 2010), foods (Bossard et al., 2014), traffic signs (Stallkamp et al.,
2011), natural textures (Cimpoi et al., 2014), and human actions (Soomro et
al., 2012). Table 1 shows the detailed information of each dataset.

###  4.2 MUST Results

Table 2 shows the unsupervised classification results on 6 tasks. MUST
substantially improves upon CLIP on all tasks with an average accuracy
improvement of +11.0% / +8.5% for ViT-B / ViT-L. We also experiment with a
widely-adopted supervised learning approach: supervised pre-training on
ImageNet (1k or 21k) followed by supervised finetuning on the downstream
tasks. We finetune the ImageNet model with the same number of epochs and the
same data augmentation (Cubuk et al., 2020) as MUST. Compared to the label-
intensive supervised learning, unsupervised MUST achieves comparable
performance on some datasets (e.g., Food101 and UCF101). MUST still lags
behind supervised learning on some datasets (e.g., GTSRB and DTD). We
hypothesis that CLIP’s low accuracy is the major cause. Since CLIP has not
seen enough data from those niche domains during pre-training, its pseudo-
labels would contain a limited amount of useful information. Despite this,
MUST can narrow the performance gap between unsupervised and supervised
learning by around 50%. Therefore, MUST is a low-cost solution for image
classification that opens up vast opportunities for practical scenarios where
images are abundant but labels are scarce.

|  | ImageNet | SUN397 | Food101 | GTSRB | DTD | UCF101 | Average
---|---|---|---|---|---|---|---|---
ViT-B/16 | CLIP | 68.3 | 64.4 | 88.7 | 43.4 | 44.7 | 68.8 | 63.1
MUST | 77.7 (+9.4) | 73.1 (+8.7) | 92.7 (+4.0) | 65.5 (+22.1) | 54.1 (+9.4) | 81.1 (+12.3) | 74.1 (+11.0)
Supervised | 81.8 | 77.2 | 90.3 | 97.1 | 78.6 | 82.3 | 84.5
ViT-L/14 | CLIP | 75.5 | 67.4 | 92.9 | 50.6 | 55.4 | 77.0 | 69.8
MUST | 82.1 (+6.6) | 75.6 (+8.0) | 95.3 (+2.4) | 68.7 (+18.1) | 62.6 (+7.2) | 85.7 (+8.7) | 78.3 (+8.5)
ViT-L/16 | Supervised | 83.4 | 81.1 | 93.1 | 97.3 | 79.7 | 86.4 | 86.8

Table 2:  Image classification results on a variety of downstream tasks. MUST significantly improves upon CLIP for unsupervised classification on all datasets. Results for CLIP are obtained using the publicly released models. Supervised denotes the label-intensive common practice of supervised pre-training followed by supervised finetuning on downstream tasks. We use the ImageNet-1k model from Touvron et al. (2021) for supervised ViT-B and the ImageNet-21k model from Dosovitskiy et al. (2021) for supervised ViT-L.  |  CLIP |  CoCoOp | CoOp | MUST
---|---|---|---|---
Finetuning data | none | 16 labeled images per class | unlabeled images
ImageNet Acc. | 68.3 |  71.0 | 71.5 | 77.7
Table 3:  Comparison with few-shot CLIP adaptation methods on ImageNet. All
methods use the same CLIP-pretrained ViT-B/16 model as image encoder.

###  4.3 Comparison with Few-shot Adaptation

In Table 3, we compare MUST with two state-of-the-art few-shot adaptation
methods (Zhou et al., 2021; 2022) that adapt a pre-trained CLIP model on
labeled ImageNet images. MUST outperforms 16-shot adaptation by a large margin
(+6.2%), demonstrating its advantage as a low-cost method to improve
classification performance using unlabeled data.

###  4.4 Unsupervised Transfer across Domains

| Pets | Caltech101
---|---|---
CLIP | 88.9 | 89.8
MUST w/o ImageNet warmup | 93.1 (+4.2) | 93.1 (+3.3)
MUST w/ ImageNet warmup | 94.3 (+5.4) | 93.7 (+3.9)

Table 4:  Warmup on ImageNet improves the performance of MUST on other
domains.

Since the [MSK] token and linear decoder are trained from scratch, the
potential of MUST may not be fully exploited for downstream tasks with limited
number of unlabeled images. To address this, we propose to warm-up the model
by training it on a similar domain with more unlabeled images. Specifically,
we first employ MUST to finetune a CLIP ViT-B model on ImageNet for a single
epoch, and then continue finetuning the model on two different datasets with
limited number of samples (Pets and Caltech101). During warmup, we freeze the
linear classifier (i.e., CLIP’s text embeddings) to anchor the normalized
image embeddings in their original space for easier transfer. As shown in
Table 4, warmup on ImageNet further improves the performance of MUST on both
datasets.

###  4.5 Ablation Study

Effect of the proposed objectives. We study the effect of the three
objectives: self-training, mask image modeling, and global-local feature
alignment. The results are shown in Table 5. Removing the alignment loss
reduces the average accuracy by 1.5%, and removing MIM further decreases
accuracy by 0.7%. The results demonstrate the importance to align the features
of [CLS] and [MSK] tokens, which regularizes the model to learn better global
features for classification.

Number of [MSK] tokens to align with.

MUST objectives | ImageNet | SUN397 | Food101 | GTSRB | DTD | UCF101 | Average
---|---|---|---|---|---|---|---
ST+MIM+Align | 77.7 | 73.1 | 92.7 | 65.5 | 54.1 | 81.1 | 74.1
ST+MIM | 77.1 | 72.0 | 92.4 | 62.5 | 52.6 | 79.2 | 72.6 (–1.5)
ST | 76.5 | 71.4 | 92.4 | 59.8 | 52.4 | 79.0 | 71.9 (–2.2)
Table 5:  Effect of the proposed training objectives on unsupervised image classification.  Num. [MSK] for $\mathcal{L}_{\mathrm{align}}$ | ImageNet | SUN397 | Food101 | GTSRB | DTD | UCF101 | Average
---|---|---|---|---|---|---|---
All | 77.7 | 73.1 | 92.7 | 65.5 | 54.1 | 81.1 | 74.1
Nearest-10 | 77.6 | 73.2 | 92.7 | 65.7 | 54.1 | 80.8 | 74.1
Table 6:  Number of [MSK] tokens used to align with the [CLS] token.

Our global-local feature alignment loss aims to align the [CLS] token to all
of the [MSK] tokens for an image (see equation 5). We relax the alignment
strength by only using the 10 [MSK] tokens that are nearest to the [CLS] token
in the embedding space. As shown in Table 6, using the nearest-10 [MSK] tokens
leads to similar performance compared to using all [MSK] tokens, which
suggests that MUST is robust to the strength of alignment.

Effect of fairness regularization.

| ImageNet | SUN397 | Food101 | GTSRB | DTD | UCF101 | Average
---|---|---|---|---|---|---|---
ST w/ $\mathcal{L}_{\mathrm{reg}}$ | 76.5 | 71.4 | 92.4 | 59.8 | 52.4 | 79.0 | 71.9
ST w/o $\mathcal{L}_{\mathrm{reg}}$ | 74.0 | 71.0 | 91.8 | 50.0 | 46.8 | 73.0 | 67.8 (–4.1)
Table 7:  Effect of the fairness regularization loss on self-training
performance.

We propose the fairness regularization loss $\mathcal{L}_{\mathrm{reg}}$ to
counteract the bias in pseudo-labels. In Table 7, we examine its effect on the
self-training objective. The regularization improves the performance of self-
training on all datasets. It is the least helpful on SUN397, which has a long-
tailed class distribution in its training set (Xiao et al., 2010).

###  4.6 Qualitative Analysis

Figure 3:  GradCAM visualization on validation images from multiple datasets.
We visualize the self-attention from [CLS] to all patches in the last ViT
layer. For each triplet, we compare three different models: CLIP (left), ST
(middle), MUST (right). We make two qualitative observations: (1) MUST
generally relies on more patches to make predictions; (2) MUST can better
leverage the true correlation between patches and class labels.

MUST pays attention to more informative regions. In Figure 3, we visualize the
GradCAM (Selvaraju et al., 2017) heatmap for the self-attention from [CLS] to
all patches in the last ViT layer. We compare three models - CLIP, ST, MUST-
and make the following qualitative observations: (1) MUST generally relies on
more patches to make predictions. E.g., the fully body of the dog instead of
only the head in row 1 column 1; all four corners of the traffic sign in row 3
column 1. (2) MUST can better leverage the true correlation between image
patches and class labels, instead of the spurious correlations used by CLIP
and ST. E.g., the guitar is attended for the action “playing guitar” in row 4
column 1, instead of the person or the background.

A helpful MIM is not necessarily good at image recovery. In Figure 4, we show
examples of recovered images from the MIM output on the validation set of
different datasets. On some datasets with abundant samples (e.g., ImageNet),
the model can learn to recover masked patches with high quality. On other
datasets with fewer number of samples (e.g., GTSRB), the recovered images have
lower quality with many artifacts. Despite the low recovery quality, the MIM
objective still provides useful supervision that improves the image
classification performance.

Figure 4:  Examples of mask image modeling on validation images from multiple
datasets. For each triplet, we show the original image (left), the masked
image (middle), and the recovered image (right).

###  4.7 Robustness to Image Corruptions

ImageNet-C Hendrycks & Dietterich (2019) | CLIP | ST | MUST | MAE+Supervised He et al. (2021)
---|---|---|---|---
mCE $\downarrow$ | 70.7 | 55.2 | 53.4 | 51.7
Table 8:  Robustness evaluation on the ImageNet-Corruption dataset.

In Table 8, we study the robustness of MUST to image corruptions. We evaluate
the performance of ImageNet-finetuned models on the ImageNet-C(Hendrycks &
Dietterich, 2019) benchmark, which contains 15 corruption types with various
severity. MUST is more robust than CLIP and ST, as suggested by a lower mean
corruption error (mCE). The mCE of MUST is only slightly higher than a model
that is first trained with self-supervised MAE (He et al., 2021) followed by
supervised finetuning on ImageNet.

###  4.8 Transductive Adaptation to Distribution Shift

| ImageNetV2 | ImageNet-R | ImageNet-A
---|---|---|---
CLIP | 61.9 | 77.7 | 49.9
Inductive MUST on ImageNet | 68.9 (+7.0) | 68.7 (–9.0) | 41.6 (–8.3)
Transductive MUST w/o ImageNet warmup | 64.7 (+2.8) | 87.3 (+9.6) | 56.9 (+7.0)
Transductive MUST w/ ImageNet warmup | 66.9 (+5.0) | 87.6 (+9.9) | 58.2 (+8.3)
Table 9:  Evaluation of MUST under natural distribution shift from ImageNet.
Figure 5:  Example images of pomegranate from ImageNet and other datasets of
natural distribution shifts.

We further investigate the robustness of MUST under natural distribution
shifts. We experiment with three out-of-distribution variants of ImageNet:
ImageNetV2 (Recht et al., 2019) with the same 1000 ImageNet classes, ImageNet-
Rendition (Hendrycks et al., 2021a) and ImageNet-Adversarial (Hendrycks et
al., 2021b) where each contains a subset of 200 classes (see Figure 5 for
example images). First, we directly evaluate a ViT-B model finetuned on
ImageNet using MUST, which has 77.7% accuracy on ImageNet validation set. As
shown in Table 9, MUST on ImageNet improves the classification accuracy on
ImageNetV2, but decreases the performance on ImageNet-R and ImageNet-A. A
similar observation is reported in CLIP (Radford et al., 2021), where
supervised adaptation to ImageNet reduces the model’s average robustness to
out-of-distribution datasets.

To address this, we perform transductive transfer learning (Xian et al., 2017;
Rohrbach et al., 2013) with MUST. Transductive learning assume access to
unlabeled test images from the new distribution, which MUST uses to jointly
update model parameters and infer image labels. The results in Table 9 show
that transductive MUST substantially improves the model’s accuracy to
distribution shift. Furthermore, the warmup on ImageNet (Section 4.4) before
transductive learning leads to more improvement.

##  5 Limitations

In this paper, we show MUST as a promising domain-specific label-free
classification algorithm where a different model is trained for each task.
Although the adaptation process is efficient, the storage of additional model
parameters could be a practical limitation if the number of tasks is large.
There exists a simple way to address this concern: gather unlabeled image from
all the domains of interest, and perform MUST to learn a single model that can
generalize to multiple domains. However, this may sacrifice the model’s
performance on each individual domain.

Due to the use of CLIP pre-trained models, MUST inherits some of the social
implications of CLIP that arose from using minimally-filtered data crawled on
the open internet. Additional analysis on the model is necessary before
deploying it for real-world applications.

##  6 Conclusion

| ImageNet | SUN397 | Food101 | GTSRB | DTD | UCF101 | Pets | Caltech101
---|---|---|---|---|---|---|---|---
| B | L | B | L | B | L | B | L | B | L | B | L | B | B
base lr | 2e-5 | 4e-5 | 1e-4 | 2e-4 | 2e-5 | 4e-5 | 2e-5 | 4e-5 | 2e-5 | 4e-5 | 1e-5 | 2e-5 | 1e-5 | 1e-5
epoch | 30 | 20 | 20 | 20 | 40 | 40 | 20 | 20 | 60 | 60 | 60 | 60 | 60 | 60
threshold $\tau$ | 0.7 | 0.5 | 0.4 | 0.3 | 0.7 | 0.7 | 0.3 | 0.2 | 0.4 | 0.5 | 0.5 | 0.6 | 0.7 | 0.7
mask ratio | 0.3 | 0.3 | 0.1 | 0.1 | 0.1 | 0.1 | 0.5 | 0.3 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
mask patch size | 32 | 56 | 32 | 28 | 32 | 56 | 32 | 56 | 32 | 56 | 16 | 14 | 32 | 32
$\lambda_{\mathrm{align}}$ | 0.2 | 0.5 | 1 | 0.5 | 0.1 | 0.5 | 0.5 | 0.5 | 0.2 | 0.5 | 0.5 | 0.5 | 0.1 | 0.1
$\lambda_{\mathrm{reg}}$ | 1 | 0.2 | 1 | 1 | 1 | 1 | 1 | 1
EMA init. $\mu_{0}$ | 0.999 | 0.99 | 0.999 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99
EMA iters. $\mu_{n}$ | 2000 | 500 | 2000 | 500 | 100 | 500 | 100 | 100

Table 10:  Hyperparameters for MUST on the downstream datasets. B: ViT-B/16;
L: ViT-L/14.

The field of deep learning has been largely driven by simple and effective
methods that can scale well. MUST is a simple method that can effectively
adapt a pre-trained zero-shot classifier to downstream tasks in an
unsupervised manner. The core of MUST is to simultaneously leverage two
different sources of supervision signals obtained from unlabeled data: task-
specific supervision from pseudo-labels and task-agnostic self supervision
from raw images. We propose a simple alignment objective that bridges the two
sources of supervision and enables the model to learn better features.

As a future work, MUST can be naturally extended to other domains such as NLP,
where mask language modeling could act as the task-agnostic objective to
improve self-training. We hope that this perspective could inspire future
methods in unsupervised learning and zero-shot classification.

## References

  * Baevski et al. (2022) Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  _arXiv preprint arXiv:2202.03555_ , 2022.
  * Bao et al. (2022) Hangbo Bao, Li Dong, and Furu Wei.  Beit: BERT pre-training of image transformers.  In _ICLR_ , 2022.
  * Berthelot et al. (2020) David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  In _ICLR_ , 2020.
  * Binh (2011) Nguyen Dang Binh.  Online multiple tasks one-shot learning of object categories and vision.  In David Taniar, Eric Pardede, Hong-Quang Nguyen, J. Wenny Rahayu, and Ismail Khalil (eds.), _MoMM_ , 2011.
  * Bossard et al. (2014) Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101 - mining discriminative components with random forests.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), _ECCV_ , 2014.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _ICML_ , 2020.
  * Cimpoi et al. (2014) Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In _CVPR_ , 2014.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _ICLR_ , 2020.
  * Cubuk et al. (2020) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _CVPR Workshops_ , 2020.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , pp. 248–255, 2009.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Gandelsman et al. (2022) Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros.  Test-time training with masked autoencoders.  In _NeurIPS_ , 2022.
  * Gao et al. (2021) Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _arXiv preprint arXiv:2110.04544_ , 2021.
  * Grandvalet & Bengio (2004) Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  In _NIPS_ , 2004.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  Bootstrap your own latent - A new approach to self-supervised learning.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), _NeurIPS_ , 2020.
  * He et al. (2020a) Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato.  Revisiting self-training for neural sequence generation.  In _ICLR_ , 2020a.
  * He et al. (2020b) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , 2020b.
  * He et al. (2021) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  _arXiv preprint arXiv:2111.06377_ , 2021.
  * Hendrycks & Dietterich (2019) Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _ICLR_ , 2019.
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  _ICCV_ , 2021a.
  * Hendrycks et al. (2021b) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  _CVPR_ , 2021b.
  * Hjelm et al. (2019) R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  In _ICLR_ , 2019.
  * Huynh & Elhamifar (2020) Dat Huynh and Ehsan Elhamifar.  Fine-grained generalized zero-shot learning via dense attribute-based attention.  In _CVPR_ , 2020.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  _arXiv preprint arXiv:2102.05918_ , 2021.
  * Kahn et al. (2020) Jacob Kahn, Ann Lee, and Awni Y. Hannun.  Self-training for end-to-end speech recognition.  In _ICASSP_ , pp. 7084–7088, 2020.
  * Laine & Aila (2017) Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  In _ICLR_ , 2017.
  * Lee et al. (2013) Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _ICML Workshops_ , 2013.
  * Li et al. (2021) Junnan Li, Caiming Xiong, and Steven C.H. Hoi.  Mopro: Webly supervised learning with momentum prototypes.  In _ICLR_ , 2021.
  * Li et al. (2022) Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  In _ICLR_ , 2022.
  * Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Parkhi et al. (2012) Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In _CVPR_ , 2012.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang (eds.), _ICML_ , 2021.
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _ICML_ , 2019\.
  * Rohrbach et al. (2013) Marcus Rohrbach, Sandra Ebert, and Bernt Schiele.  Transfer learning in a transductive setting.  In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), _NeurIPS_ , 2013.
  * Sahito et al. (2022) Attaullah Sahito, Eibe Frank, and Bernhard Pfahringer.  Better self-training for image classification through self-supervision.  In Guodong Long, Xinghuo Yu, and Sen Wang (eds.), _AI_ , 2022.
  * Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _ICCV_ , pp. 618–626, 2017.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In _NeurIPS_ , 2020.
  * Soomro et al. (2012) Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ , 2012.
  * Stallkamp et al. (2011) Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.  The german traffic sign recognition benchmark: A multi-class classification competition.  In _IJCNN_ , 2011.
  * Tarvainen & Valpola (2017) Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In _NIPS_ , pp. 1195–1204, 2017.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _ICML_ , 2021.
  * Wang et al. (2019) Wei Wang, Vincent W. Zheng, Han Yu, and Chunyan Miao.  A survey of zero-shot learning: Settings, methods, and applications.  _TIST_ , 2019.
  * Wang et al. (2018) Xiaolong Wang, Yufei Ye, and Abhinav Gupta.  Zero-shot recognition via semantic embeddings and knowledge graphs.  In _CVPR_ , 2018.
  * Wang et al. (2022) Xudong Wang, Zhirong Wu, Long Lian, and Stella X. Yu.  Debiased learning from naturally imbalanced pseudo-labels for zero-shot and semi-supervised learning.  In _CVPR_ , 2022.
  * Wei et al. (2021) Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan L. Yuille, and Fan Yang.  Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning.  In _CVPR_ , 2021.
  * Xian et al. (2017) Yongqin Xian, Bernt Schiele, and Zeynep Akata.  Zero-shot learning - the good, the bad and the ugly.  In _CVPR_ , 2017.
  * Xiao et al. (2010) Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba.  SUN database: Large-scale scene recognition from abbey to zoo.  In _CVPR_ , 2010.
  * Xie et al. (2020) Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le.  Self-training with noisy student improves imagenet classification.  In _CVPR_ , 2020.
  * Xie et al. (2022) Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  SimMIM: A simple framework for masked image modeling.  In _CVPR_ , 2022.
  * Yao et al. (2022) Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  FILIP: fine-grained interactive language-image pre-training.  In _ICLR_ , 2022.
  * Zhai et al. (2022) Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In _CVPR_ , 2022.
  * Zhou et al. (2021) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _arXiv preprint arXiv:2109.01134_ , 2021.
  * Zhou et al. (2022) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _CVPR_ , 2022.
  * Zoph et al. (2020) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le.  Rethinking pre-training and self-training.  In _NeurIPS_ , 2020.


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: National University of Singapore 22institutetext: Huawei
International Pte Ltd
22email<EMAIL_ADDRESS>
# NightHaze: Nighttime Image Dehazing via Self-Prior Learning

Beibei Lin 1 *1 * Yeying Jin Both authors contributed equally to this
research.11 Wending Yan 22 Wei Ye 22 Yuan Yuan 22
Robby T. Tan 11

###### Abstract

Masked autoencoder (MAE) [13] shows that severe augmentation during training
produces robust representations for high-level tasks. This paper brings the
MAE-like framework to nighttime image enhancement, demonstrating that severe
augmentation during training produces strong network priors that are resilient
to real-world night haze degradations. We propose a novel nighttime image
dehazing method with self-prior learning. Our main novelty lies in the design
of severe augmentation, which allows our model to learn robust priors. Unlike
MAE that uses masking, we leverage two key challenging factors of nighttime
images as augmentation: light effects and noise. During training, we
intentionally degrade clear images by blending them with light effects as well
as by adding noise, and subsequently restore the clear images. This enables
our model to learn clear background priors. By increasing the noise values to
approach as high as the pixel intensity values of the glow and light effect
blended images, our augmentation becomes severe, resulting in stronger priors.
While our self-prior learning is considerably effective in suppressing glow
and revealing details of background scenes, in some cases, there are still
some undesired artifacts that remain, particularly in the forms of over-
suppression. To address these artifacts, we propose a self-refinement module
based on the semi-supervised teacher-student framework. Our NightHaze,
especially our MAE-like self-prior learning, shows that models trained with
severe augmentation effectively improve the visibility of input haze images,
approaching the clarity of clear nighttime images. Extensive experiments
demonstrate that our NightHaze achieves state-of-the-art performance,
outperforming existing nighttime image dehazing methods by a substantial
margin of 15.5% for MUSIQ and 23.5% for ClipIQA.

###### Keywords:

Nighttime Image Dehazing MAE-like Representations

Figure 1: Qualitative results from NightEnhance’23 [17], NightDeFog’20 [53],
DiT’23 [36] and our method, on the real-world dataset. Ours not only suppress
glow but also reveal the detailed textures of the night scenes, including
those under low light and strong glow.

##  1 Introduction

Dehazing night images deals not only with haze but also with nighttime
degradations such as low light, noise, glow, uneven light distribution, etc.
The combination of these degradations and the absence of paired real data make
the task significantly challenging.

Existing supervised and semi-supervised nighttime image dehazing methods [53,
17, 32] use either synthetic data or both synthetic and real data in training.
Unfortunately, as shown in Figure 1, their performance is unsatisfactory when
evaluated on real-world nighttime haze images. The main reason is that there
is a large domain gap between synthetic haze and real-world haze, and unpaired
real data cannot provide pixel-level guidance. As a result, models trained on
these data lead to inferior performance when applied to real haze images.
Table 1 shows five non-reference image quality assessment (NF-IQA) scores of
these methods on real night haze data. Noticeably, there are significant gaps
between the NF-IQA scores of clear night images and those of restored results
obtained from existing methods.

Masked autoencoders (MAE) [13] are known to yield strong representations for
high-level tasks. The core design of MAE is to use severe augmentation for
training. MAE masks 75% of the input image before reconstructing it. This
paper aims to bring MAE-like strong representations to nighttime image
dehazing. However, MAE’s masking strategies are designed to capture structural
spatial priors by filling in the masked regions. In contrast, nighttime image
dehazing aims to restore a clear background behind haze and nighttime
degradations.

To obtain strong priors that are resilient to real-world night haze
degradations, we design the severe augmentation for our learning process. Our
main novelty lies in defining “severe” based on proper augmentation.
Specifically, we first find two key challenging factors of nighttime images
for augmentation: light effects and noise. During training, we intentionally
degrade clear nighttime images by blending them with various light effects,
including glow, and adding noise. As demonstrated in MAE [13], the severity of
augmentation is key to yield strong feature representations or priors.
Accordingly, we define “severe augmentation” as noise values approaching as
high as the pixel intensity values blended with strong glow and other night
light effects.

Based on our severe augmentation, we propose NightHaze, a simple yet effective
nighttime image dehazing method with self-prior learning. Our main novelty,
the design of severe augmentation, allows our network to learn the strong
priors that are robust to real-world haze effects. Once the strong priors are
obtained, our network can effectively enhance the visibility of nighttime haze
images. Like MAE, we apply severe augmentation and then train our network to
restore the clear background. Table 1 shows that the NF-IQA scores of our
self-prior learning are closer to the NF-IQA scores of clear night images.

While our self-prior learning effectively can enhance the visibility of night
haze images, in some cases, artifacts can persist, typically in the form of
over-suppression. To refine the output of our self-prior learning, we
introduce a self-refinement module based on the teacher-student framework
[47]. Unlike standard teacher-student frameworks that directly pass knowledge
from the student to the teacher, we utilize NF-IQA scores to monitor the
quality of knowledge from the student. This allows us to pass only high-
quality information to the teacher, resulting in superior performance. Figure
1 shows our results in comparison to those of the state-of-the-art methods.
The main contributions are as follows:

  * •

We propose self-prior learning to bring MAE-like strong representations to
nighttime image dehazing. Our main novelty lies in the design of severe
augmentation, which enables our model to learn clear background priors.

  * •

We propose self-refinement to refine our model by utilizing real-world
nighttime haze images. Unlike the standard teacher-student framework, we
introduce NF-IQA scores to assess the knowledge quality from our student
model’s learning. This approach avoids inaccurate knowledge transfer from our
student to our teacher.

  * •

Extensive experiments demonstrate that our Nighthaze achieves a significant
performance improvement on real-world nighttime haze images. Our NightHaze
attains outstanding scores of 52.87 in MUSIQ and 76.88 in ClipIQA,
significantly exceeding existing nighttime image dehazing methods with a
substantial margin of 15.5% for MUSIQ and 23.5% for ClipIQA.

##  2 Related Work

#### Nighttime Image Dehazing

Previous image dehazing methods [45, 8, 14, 4, 34] remove haze effects based
on the haze model. Specifically, they estimate transmission maps and
atmosphere lights to recovery the clear background. However, nighttime haze
images always suffer from noise. Thus, these methods are hard to estimate
accurate transmission maps and atmosphere lights, resulting in inferior
performance. Recent deep learning-based daytime image dehazing methods [5, 40,
59, 22, 41, 27, 39, 56, 57, 31, 7, 38, 66, 51, 11, 43, 26, 42, 6, 28, 24, 23,
15, 10, 30, 65, 55] propose different networks to restore haze images. These
methods can address nighttime haze images by training on synthetic nighttime
dehazing datasets. However, these methods struggle to handle nighttime haze
images as they often neglect complex nighttime degradation factors, such as
noise and multi-colored artificial light effects.

Existing optimization-based nighttime dehazing methods address nighttime haze
effects using the atmospheric scattering model (_e.g_., [37, 2, 3]), new
imaging model (_e.g_., [61, 60, 46, 32]), among others. Wang  _et al_. [49]
propose a variational model with the gray haze-line prior to enhance the
visibility of nighttime haze images. However, these methods struggle to handle
challenging hazy regions affected by significant noise and complex lighting
conditions. The main challenge is the difficulty in estimating atmospheric
lights in these regions.

Existing learning-based nighttime image dehazing methods [62, 53, 21, 17]
propose convolutional neural networks for nighttime image dehazing. Zhang  _et
al_. [62] first synthesize nighttime dehazing datasets, then train their
network on these datasets. Yan  _et al_. [53] and Jin  _et al_. [17] use
labeled synthetic dehazing datasets and unlabeled real-world datasets to train
their frameworks. However, these labeled synthetic datasets are built based on
less challenging augmentation, such as regions affected by light noise. Models
trained using these datasets struggle to restore clear backgrounds in
challenging real-world haze images, including those affected by significant
noise and complex lighting conditions. Unsupervised strategies from [53, 17]
aim to reduce the domain gap between synthetic and real-world haze effects.
However, these strategies, based on unpaired real-world data, cannot provide
accurate guidance for restoring challenging real hazy regions.

#### Image Quality Assessment

To qualitatively evaluate the performance of real-world haze images without
the corresponding ground truth, we introduce several image quality assessment
methods, including MUSIQ [19], ManIQA [54], ClipIQA [48], HyperIQA [44], and
TRES [9]. MUSIQ [19] is builted based on a transofmer network and evaluate the
image quality at multiple image scales. ManIQA [54] integrates the global and
local information of images to assess the image quality. HyperIQA [44]
assesses in-the-wild image quality by using a self-adaptive hyper network
architecture. TRES [9] leverages transformers, relative ranking, and self-
consistency losses to effectively evaluate the image quality. ClipIQA [48]
introduces Contrastive Language-Image Pre-training (CLIP) models to evaluate
the quality perception and abstract perception of images.

Figure 2: Overview of our self-prior learning. This approach aims to produce
strong priors that are resilient to real-world night haze degradations. Given
a clear nighttime image, we blend clear images with various maps collected
from real-world haze images and add noise to these blended images. The
augmented input is then fed into an Encoder-Decoder framework to recover the
clear background. The loss is a simple L1 loss between the reconstructed image
and the clear input. The key to self-prior learning is the severity of the
augmentation. By increasing the noise values to approach as high as the pixel
intensity values of the glow and light effect blended images, our augmentation
becomes severe, resulting in stronger priors.

##  3 Proposed Method: NightHaze

The overview of our NightHaze is shown in Figure 2. It includes two
components: self-prior learning and self-refinement. Our self-prior learning
aims to bring the MAE-like framework to nighttime image enhancement, resulting
in strong network priors that are resilient to real-world night haze effects.
Our main novelty is the design of severe augmentation. By employing severe
augmentation, our prior learning can more effectively learn the strong priors.
Once the strong priors are obtained, our network can effectively enhance the
visibility of nighttime haze images. Our self-refinement utilizes unlabeled
nighttime haze images to refine the dehazing ability of our network in the
real scenes in a semi-supervised manner.

###  3.1 Self-Prior Learning

Our self-prior learning aims to learn strong network priors of clear
backgrounds using severe augmentation. As shown in Figure 2, our training
process is similar to the approach in MAE, where we severely augment clear
images and force the network to recover their clear backgrounds. MAE employs
masking strategies to augment the input images. This masking forces the
network to learn the structural spatial priors of the world by predicting the
contents in the masked regions. However, nighttime image dehazing is about
restoring a clear background behind haze and nighttime degradations. Hence,
masking is not a suitable augmentation for dehazing.

Nighttime haze degradations are mainly caused by glow light effects and noise.
Due to this, we degrade clear images by blending them with light effects
including glow and adding noise and then we restore the clear images, as
demonstrated in Figure 3. The augmentation can be formulated as:

$I=W_{b}*J+(1-W_{b})*L+\epsilon,$ (1)

where I is the augmented image, J is the clear image, $W_{b}$ is the blend
weight map, $L$ is the light map, $\epsilon$ is the noise. We explain the
details of each term in Eq. (1) as follows.

Figure 3: Example results on validation sets. For each triplet, we show the
augmented image (left), our restoration (middle), and the ground-truth
(right).

#### Light Map $L$

We manually capture atmosphere light maps from real haze images. When regions
in a night image suffer from dense haze effects (where the signal is less than
the noise), they can be regarded as atmospheric lights. Hence, we collect
these regions from real haze images as light maps. Given an input image
$J_{i}\in\mathbb{R}^{C_{i}\times H_{i}\times W_{i}}$, where $C_{i}$ is the
number of channels and ($H_{i},W_{i}$) is image size, we first sample a light
map and resize it to $\mathbb{R}^{C_{i}\times H_{i}\times W_{i}}$. Moreover,
to simulate glow effects, we use Gaussian kernels to enhance the brightness of
several random regions in the light maps. Figure 4 shows our augmented light
maps.

#### Blending Weight Map $W_{b}$

As the goal of our blending weight $W_{b}$ is to mix a clear image and light
maps, the range of the blending weight $W_{b}$ is (0, 1). To obtain the
blending weight $W_{b}$, we first sample a value $t\in\mathbb{R}^{1}$ from a
uniform distribution in [$T_{\rm Low}$, $T_{\rm High}$], where both $T_{\rm
Low}$ and $T_{\rm High}$ are lower than 1. We then expand $t\in\mathbb{R}^{1}$
to a map $W_{b}\in\mathbb{R}^{C_{i}\times H_{i}\times W_{i}}$. Moreover, we
randomly adjust some regions’ blending weights. Figure 5 visualizes several
blending weight maps. Our blending weight map is inspired by transmission maps
of the haze formulation. However, transmission maps rely on the depth of
images for their generation. Obtaining accurate depth maps in nighttime images
is a challenging problem. A straightforward approach is to use monocular depth
estimates, but this may introduce errors. Unlike transmission maps, we
randomly generate uneven blending weights for an image. This approach enables
us to simply and effectively augment clear images.

#### Noise $\epsilon$

We take Gaussian noise as the noise $\epsilon$, which can be formulated as
$W_{\epsilon}*\mathcal{N}(0,1)$, where $W_{\epsilon}$ is the weight of the
Gaussian noise. Theoretically, $\mathcal{N}(0,1)$ ranges from negative
infinity to positive infinity, with about approximately 99.7% of the values
fall within the range of -3 to 3. We remove the rest 0.3% values and the
negative values of $\mathcal{N}(0,1)$. Thus, the range of our noise term
$\epsilon$ is (0, 3*$W_{\epsilon}$).

Figure 4: Visualization of different light maps that contain glow effects. We
use Gaussian kernels to simulate glow effects. By adjusting the parameters of
kernels, we can control the number, size, and brightness of glow regions.

#### Severe Augmentation

MAE demonstrates that severe augmentation is key to obtaining strong
representations. Motivated by this, we define “severe augmentation” as
$\epsilon$ approaching $W_{b}*J$ according to Eq. (1). The main reason is that
the level of the noise $\epsilon$ affects the difficulty of nighttime image
restoration. To be more specific, we use min-max normalization to normalize
the clear image and thus the range of $J$ is (0, 1). After adding light
effects, the range of the term $W_{b}*J$ becomes to (0, $T_{\rm High}$). Since
its range depend on the value of $T_{\rm High}$, we further define
$W_{\epsilon}$ as $W_{n}*T_{\rm High}$ and thus the range of $\epsilon$
becomes to (0, 3*$W_{n}$*$T_{\rm High}$). We empirically find that when
$W_{n}=0.1$ and $\epsilon\in[0,0.3*T_{\rm High}]$, our augmentation is
difficult enough.

In light maps, we use Gaussian kernels to increase the brightness of several
random regions. These regions can overlap, resulting in extremely uneven light
distribution in our augmented light maps. Moreover, due to the overlapping
brightness enhancement, the glow effects can become over-saturated. Thus, when
we blend clear images with these over-saturated light effects, their signals
become more dominant than those of the clear images, making restoration
challenging.

#### Implementation

Given a clear nighttime image, we first sample a light map $L$ from our
dataset and randomly enhance the brightness of several regions. Second, we
sample a value $t$ from a uniform distribution to generate a uniform map. We
then randomly adjust the values of several regions in this uniform map to
create the blending weight map $W_{b}$. Third, we generate a significant
Gaussian noise map $\epsilon$. Finally, we augment the input clear image using
Eq. (1), as illustrated in Figure 3 which shows several augmented inputs.
During the training stage, we calculate the mean squared error (MSE) between
the reconstructed and original clear images in the pixel space.

(a)

(b)

(c)

(d)

Figure 5: Visualization of blending weight maps, where the black and white
regions represent low and high blending values. By adjusting the values of
blending weights and the noise term, we can control the severity of our
augmentation.

###  3.2 Self-Refinement

Although our self-prior learning effectively handle haze effects, it
occasionally results in artifacts particularly in the forms of over-
suppression. To address this issue, we propose a teacher-student framework to
further refine our network using real haze images. The network parameter from
our self-prior learning is used to initialize a teacher model $w_{T}$ and a
student model $w_{S}$. The purpose of the teacher model is to generate the
pseudo ground truth based on confidence score, while the student model is
trained on high-confidence predictions and their corresponding inputs. By
gradually learning from high-confidence predictions, our network is allowed to
address more challenging regions.

Given a real-world hazy dataset $\mathbf{D}_{\rm uh}=\\{\mathbf{x}^{\rm
uh}_{i}\\}^{N_{\rm uh}}_{i=1}$, where $\mathbf{x}^{\rm uh}_{i}$ is the $i$-th
haze image and $N_{\rm uh}$ is the number of haze images, we utilize the
teacher model $w_{\rm T}$ to generate predictions $\mathbf{y}^{\rm uh}_{i}$
and confidence maps $\mathbf{u}^{\rm uh}_{i}$. To be more specific, we first
sample overlapping regions from $\mathbf{x}^{\rm uh}_{i}$ and input the set of
overlapping regions into our teacher model $w_{T}$ to generate predictions.
Once the predictions of the overlapping regions are obtained, we calculate the
mean values and variance of overlapping regions as predictions and confidence
maps. Subsequently, we use a threshold $\mathbf{v}^{1}_{thr}$ to convert the
confidence map $\mathbf{u}^{\rm uh}_{i}$ to a binary mask $\mathbf{m}^{\rm
uh}_{i}$.

During the training stage, we use the noise and haze augmentation to degrade
the haze images $\mathbf{x}^{\rm uh}_{i}$. The degraded haze images
$\mathbf{\hat{x}}^{\rm uh}_{i}$ are then fed into our student model for
training. The loss function can be formulated as:

${\rm Loss}=\frac{1}{N}\sum_{i=1}^{N}|\mathbf{\hat{y}}^{\rm
uh}_{i}-\mathbf{y}^{\rm uh}_{i}|*\mathbf{m}^{\rm uh}_{i},$ (2)

where $N$ is the number of samples in the training batch and
$\mathbf{\hat{y}}^{\rm uh}_{i}$ is the output of the degraded haze images
$\mathbf{\hat{x}}^{\rm uh}_{i}$. Once the losses are obtained, traditional
teacher-student frameworks use back-propagation to update the student model
$w_{S}$. Then they use the Exponential Moving Average (EMA) to update the
teacher model. However, the student model is trained using augmented real-
world haze inputs. Since the augmentation is random, the updates of the
student model in some training step maybe wrong and thus affect the
performance of the teacher model by EMA. To avoid this, we introduce a non-
reference metric to monitor each updates of the student model, which can be
formulated as:

${\rm Score}=F_{\rm IQA}(F(w^{t+1}_{s},\mathbf{x}^{\rm uh}_{i}))+F_{\rm
IQA}(F(w^{t}_{s},\mathbf{x}^{\rm uh}_{i})),$ (3)

where $F_{\rm IQA}(\cdot)$ represents non-reference image assessment metrics,
$F(w^{t+1}_{s},\mathbf{x}^{\rm uh}_{i})$ and $F(w^{t}_{s},\mathbf{x}^{\rm
uh}_{i})$ are $\mathbf{x}^{\rm uh}_{i}$’s outputs generated using the student
weights $w^{t+1}_{s}$ and $w^{t}_{s}$. $w^{t}_{s}$ is the current weight of
the student model, while $w^{t+1}_{s}$ is the updated weight. If Score
$>\mathbf{v}^{2}_{thr}$, the student model will accept this update and pass
the knowledge to the teacher model. Otherwise, the student model will not
accept this update.

##  4 Experiments

Due to the lack of ground truth, previous methods typically use synthetic
datasets for quantitative evaluation and real-world datasets for qualitative
evaluation. However, the quantitative evaluation on synthetic datasets cannot
effectively represent the real-world dehazing capabilities. Motivated by this,
we introduce a new benchmark to quantitatively evaluate the dehazing
performance on real-world nighttime haze datasets. Specifically, we use five
non-reference image quality assessments, including MUSIQ [19], ManIQA [54],
ClipIQA [48], HyperIQA [44], and TRES [9], along with image contrast, to
evaluate the performance of existing methods.

Our real-world nighttime haze datasets, RealNightHaze, is collected from
internet and existing nighttime dehazing methods [62, 17], and consists of 440
real-world nighttime haze images. We will publicly release our collected
dataset.

Table 1: Quantitative comparison on the RealNightHaze dataset. Five non-reference metrics, including MUSIQ [19], ManIQA [54], ClipIQA [48], HyperIQA [44], and TRES [9], and the image contrast are used for evaluation. For five non-reference metrics, higher is better. In terms of image contrast, the better it is when the contrast closely matches the statistical value of a clear image. NightDeFog [53], NightVDM [32] and NightEnhance [17] are nighttime image dehazing methods, while the rest methods are image restoration backbones. “syn-training” means training our backbone on synthetic datasets. Non-reference metrics | MUSIQ | TRES | Hyperiqa | CLIPIQA | MANIQA | Contrast
---|---|---|---|---|---|---
Statistics from clear images | 53.91 | 78.69 | 0.6819 | 0.6834 | 0.4319 | 50.96
Statistics from haze images | 42.12 | 57.94 | 0.5186 | 0.5267 | 0.2556 | 22.21
Backbone with syn-training | 44.25 | 60.84 | 0.5903 | 0.5449 | 0.2665 | 24.08
NightDeFog [53] | 45.76 | 65.83 | 0.6133 | 0.5146 | 0.3318 | 57.60
NightVDM [32] | 40.90 | 53.87 | 0.5241 | 0.4656 | 0.2332 | 27.47
Restormer [58] | 39.13 | 52.97 | 0.4956 | 0.4763 | 0.2428 | 24.58
Uformer [50] | 39.14 | 52.60 | 0.4902 | 0.4427 | 0.2420 | 25.03
WeatherDiffusion [35] | 38.98 | 52.33 | 0.4868 | 0.5044 | 0.2338 | 24.04
NightEnhance [17] | 44.33 | 60.07 | 0.5379 | 0.4736 | 0.2354 | 24.57
DiT [36] | 41.02 | 55.54 | 0.5259 | 0.5059 | 0.2513 | 23.94
Ours | 52.87 | 76.88 | 0.6747 | 0.6360 | 0.4009 | 44.35

###  4.1 Implementation Details

Our NightHaze includes two parts: self-prior learning and self-refinement. We
adopt the MAE network [13] as our backbone. In our self-prior learning, the
size of input images is 224 $\times$ 224\. During the training stage, we use
Adam as our optimizer and set the initial learning rate to 1.5e-4. The total
training steps and the training batch size are set to 20,000 and 128,
respectively. Our self-prior learning mainly consists of three parts: the
light map, the blending weight map, and noise.

#### Light Map

At each training step, we randomly select an atmospheric light map for every
clear image. Subsequently, we select various regions and amplify their
brightness using Gaussian kernels. The number of selected regions is between 2
and 10, and the size of Gaussian kernels ranges from 15 to 80.

#### Blending Weight Map

For a clear image, we initially select a blending weight $t$ from a uniform
distribution ($T_{\rm low}$, $T_{\rm high}$) and expand it into a map. Then,
we randomly adjust the blending weights of several regions. $T_{\rm low}$ and
$T_{\rm high}$ are set to 0.001 and 0.1, respectively. We set the number of
the selected regions to 8. The size of each region is 64 $\times$ 64 and the
range of the changed value is a uniform distribution (0, 0.04).

#### Noise

Section 3.1 introduces our noise term. During the training stage, we use
Gaussian noise to augment our input images. We set $W_{n}$ to 0.1. Thus, the
value range of our $\epsilon$ becomes to (0, 0.03).

#### Self-Refinement

For a haze image of size 256 $\times$ 256, we first sample overlapping regions
within the input haze image. The image size of the overlapping regions is 224
$\times$ 224 and the stride of the overlap sampling is 4. Once the pseudo
ground truths are obtained, we randomly crop 224 $\times$ 224 regions from
both the haze images and the pseudo ground truths for training our student
model. We use Adam as our optimizer, with the initial learning rate set to
2e-5. We set the training step and the training batch size to 10,000 and 16,
respectively. The Exponential Moving Average (EMA) weight is set to 0.9999.
The thresholds $\mathbf{v}^{1}_{thr}$ and $\mathbf{v}^{2}_{thr}$ are set to
0.005 and 0.

(a)

(b)

(c)

(d)

(e)

Figure 6: Qualitative results from NightEnhance’23 [17], NightDeFog’20 [53],
DiT’23 [36], and our method, on the real-world dataset. Our method not only
handles glow effects but also reveals detailed textures in low-light regions,
such as walls and streets. Overall, results from our method look more
realistic. Zoom in for better visualisation.

###  4.2 Quantitative Evaluation on RealNightHaze

We compare our NightHaze method on RealNightHaze with state-of-the-art (SOTA)
methods, including NightEnhance [17], NightDeFog [53], NightVDM [32], Uformer
[50], Restormer [58], WeatherDiffusion [35] and DiT [36]. NightEnhance [17],
NightDeFog [53] and NightVDM [32] are nighttime image dehazing methods, while
the other methods are image restoration backbones. WeatherDiffusion [35] and
DiT [36] are diffusion-based methods. Table 1 shows that our method achieves a
significant performance improvement in all non-reference metrics. For example,
the MUSIQ and TRES of NightDeFog are 45.12 and 57.64, respectively. Our method
achieves 52.87 of MUSIQ and 76.88 of TRES, which outperforms NightDeFog by
7.11 and 19.24, respectively. Furthermore, it can be observed that our image
quality scores are closer to those of clean night images. This is because we
propose self-prior learning, which brings MAE-like strong representations to
nighttime image enhancement. By utilizing severe augmentation, we can generate
strong network priors that demonstrate robustness against real-world haze
effects, thereby leading to superior performance.

Additionally, we compare our method with other powerful image restoration
backbones, including Uformer [50], Restormer [58], WeatherDiffusion [35] and
DiT [36]. The experimental results are shown in Table 1. It can be observed
that existing image restoration methods trained using synthetic datasets
struggle to restore clear images. The main reason is that their learning
strategies, based on synthetic datasets, struggle to develop strong priors.
This results in inferior performance.

Table 2: Ablation studies on the RealNightHaze dataset. Five non-reference metrics, including MUSIQ [19], ManIQA [54], ClipIQA [48], HyperIQA [44], and TRES [9], and the image contrast are used for evaluation. For five non-reference metrics, higher is better. In terms of image contrast, the better it is when the contrast closely matches the statistical value of a clear image. “Ours(xx%)” means xx% of the training data was severely augmented, while the remaining data was non-severely augmented. “syn-training” means training our backbone on synthetic datasets.  Non-reference metrics | MUSIQ | TRES | Hyperiqa | CLIPIQA | MANIQA | Contrast
---|---|---|---|---|---|---
Statistics from clear images | 53.91 | 78.69 | 0.6819 | 0.6834 | 0.4319 | 50.96
Statistics from haze images | 42.12 | 57.94 | 0.5186 | 0.5267 | 0.2556 | 22.21
Backbone with syn-training | 44.25 | 60.84 | 0.5903 | 0.5449 | 0.2665 | 24.08
Backbone with 0% severe | 44.67 | 62.18 | 0.5773 | 0.5364 | 0.2930 | 23.40
Backbone with 25% severe | 48.64 | 67.96 | 0.6253 | 0.5715 | 0.3361 | 28.73
Backbone with 50% severe | 49.12 | 68.74 | 0.6286 | 0.5750 | 0.3417 | 29.80
Backbone with 75% severe | 49.31 | 68.92 | 0.6289 | 0.5891 | 0.3420 | 31.15
Backbone with 100% severe | 52.87 | 76.88 | 0.6747 | 0.6360 | 0.4009 | 44.35

(a)

(b)

(c)

Figure 7: Ablation studies on real-world haze images. ‘PL’ and ‘SR’ stand for
self-prior learning and self-refinement, respectively. (a) Input (b) Results
obtained using our self-prior learning. (c) Results obtained using our self-
prior learning and self-refinement. Zoom in for better visualisation.

###  4.3 Qualitative Evaluation on RealNightHaze

Figure 9 presents a qualitative comparison on the RealNightHaze dataset. It is
evident that existing nighttime image dehazing methods cannot effectively
recover a clean background in challenging areas, such as those with low-light
and glow effects. The main reason is that there is a large domain gap between
their synthetic datasets and nighttime haze degradation. As a result, these
methods struggle to reveal a clear background in such challenging areas.

In contrast, our NightHaze method not only removes nighttime haze effects but
also enhances the visibility of images. This is because our self-prior
learning effectively forces our network to learn strong priors. Similar to
MAE, our self-prior learning is also built on augmentation. In our case, we
utilize two key challenging aspects of nighttime images as augmentation
factors: light effects and noise. Following the same principle as MAE, we
define and employ severe augmentation to obtain a strong prior, which
significantly enhances performance. Furthermore, we employ a self-refinement
module to further refine our model through learning from real haze images. To
be more specific, our teacher model generates high-confidence predictions from
real haze images to guide the student model’s learning. This method enables
our model to gradually adapt to real-world high-confidence predictions,
thereby addressing more challenging scenarios.

###  4.4 Ablation Studies

To verify the effectiveness of our self-prior learning and self-refinement, we
conduct ablation studies on the RealNightHaze dataset. The experimental
results are shown in Figure 7 and Table 2.

#### Analysis of Self-Prior Learning

To verify the effectiveness of our self-prior learning, we use different
ratios to mix severe and non-severe augmentation. The experimental results are
shown in Table 2. It can be observed that our backbone with non-severe
augmentation cannot sufficiently enhance the visibility of haze images. The
MUSIQ and TRES scores are 44.67 and 62.18, outperforming the original haze
images’ scores by 2.55 and 4.24. In contrast, our backbone with severe
augmentation achieves significant performance improvement. The main reason for
this is that severe augmentation yields strong priors that are robust to real-
world haze.

#### Analysis of Self-Refinement

We conduct ablation studies to verify the effectiveness of our self-
refinement. The experimental results are shown in Figure 7. Although our self-
prior learning can effectively enhance the visibility of night haze images, it
still leaves some artifacts caused by over-suppression. For example, the light
sources in Figure 7(b) are over-suppressed. Figure 7(c) demonstrates that our
self-refinement further refines our outputs.

(a)

(b)

Figure 8: (a) Quantitative results from NightDeFog’20 [53], NightEnhance’23
[17], NightVDM’22 [32], Restormer’22 [58], DiT’23 [36] and our method, on the
real-world dataset, are presented. In this evaluation, we use only the results
from our self-prior learning. We utilize five non-reference image quality
assessment metrics: MUSIQ [19], ManIQA [54], ClipIQA [48], HyperIQA [44], and
TRES [9]. ‘Clean’ and ‘Haze’ denote statistics derived from clear night images
and haze images, respectively. For all metrics, a larger value indicates
better performance. (b) Quantitative results from models trained on synthetic
data (blue lines) and our method with different ratios of severely augmented
samples in training are presented. “Ours(xx%)” means xx% of the training data
was severely augmented, while the remaining data was non-severely augmented.
“syn-training” means training models on synthetic datasets.

##  5 Discussion

In this section, we further discuss the differences between our self-prior
learning and supervised learning based on synthetic data

#### Discussion on Training Data

Synthesizing haze requires accurate depth estimation, which is still an open
challenge in real-world nighttime images. Although game engines can provide
accurate depth for scenes, there remains a significant gap between these game
scenes and real-world scenes. As shown in Figure 8, NF-IQA scores of results
obtained from models trained on synthetic datasets have significant gaps
compared to NF-IQA scores of clear night images.

In contrast, we incorporate the MAE-like framework into nighttime image
enhancement and propose self-prior learning to obtain strong priors that are
robust to real-world nighttime haze effects. Our main novelty is the design of
severe augmentation, allowing our prior learning to become more effective.
Once these robust priors are acquired, high-quality dehazing can be achieved.
In our self-prior learning, we utilize two key challenging factors of
nighttime images as augmentation: light effects and noise. As introduced in
Section 3.1, these augmentations do not rely on any external estimation that
might be inaccurate. As demonstrated in MAE [13], as long as the augmentation
is severe, the priors become stronger. As shown in Figure 8, our self-prior
learning significantly outperforms models trained with synthetic data.

#### Discussion on Degradations

Unlike synthetic data, which randomly adds degradations such as noise, our
self-prior learning emphasizes the importance of augmentation severity during
training. By increasing noise values to approximate the pixel intensity values
of glow and light effect blended images, our learning process yields strong
priors. As demonstrated in Table 2, using severe augmentation achieves
significant performance improvement. Our NightHaze achieves an outstanding
MUSIQ score of 52.87, significantly surpassing methods trained on synthetic
datasets by a substantial margin of 19.5% in MUSIQ.

##  6 Conclusion

This paper brings the MAE-like framework to nighttime image restoration and
shows that severe augmentation during training yields strong priors that are
robust to real-world night haze effects. We present NightHaze, a novel
nighttime dehaizng method with self-prior learning. Our main novelty is the
design of severe augmentation, which enables our prior learning to become more
effective. Our self-prior learning is simple: we intentionally augment clear
nighttime images by blending them with various light effects including glow
and adding noise. Specifically, we blend clear images with various light maps
containing artificial glow and add noise to these blended images. Inspired by
MAE, which indicates that the severity of augmentation is key to obtaining
strong priors, we define severe augmentation using our night-specific
augmentations. We show that severe augmentation in training yields strong
priors that are robust to real-world haze. Although our self-prior learning
can effectively handle haze effects, it still leaves some artifacts caused by
over-suppression. Our self-refinement is proposed to address this issue. It is
built on the semi-supervised teacher-student framework. Unlike traditional
teacher-student frameworks, we incorporate non-reference metrics to monitor
each update of our student model. This approach allows us to prevent
inaccurate knowledge transfer from the student to the teacher. Extensive
experimental results demonstrate that our NightHaze achieves a significant
performance improvement on real-world nighttime hazy datasets.

## Supplementary Material

(a) Input (b) Restormer [58]

(c) DiT [36] (d) NightDeFog [53]

(e) NightEnhance [17] (f) Ours

Figure 9: Qualitative results from NightEnhance’23 [17], NightDeFog’20 [53],
Restormer’22 [58], DiT’23 [36], and our method, on the real-world dataset. Our
method not only handles glow effects but also reveals detailed textures in
low-light regions. Overall, results from our method look more realistic. Zoom
in for better visualisation.

## More Results on Real-world Datasets

In this section, we compare our NightHaze method on the RealNightHaze dataset
with state-of-the-art (SOTA) methods, including NightEnhance [17], NightDeFog
[53], Restormer [58], and DiT [36]. NightEnhance [17] and NightDeFog [53] are
nighttime image dehazing methods, while the other methods are image
restoration backbones. The experimental results are shown in Figs. 9, 11 and
10. It can be observed that existing nighttime image dehazing methods cannot
effectively recover a clean background in challenging areas, such as those
with low-light and glow effects. In contrast, our NightHaze method not only
removes nighttime haze effects but also enhances the visibility of images.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 10: Qualitative results from NightEnhance’23 [17], NightDeFog’20 [53],
Restormer’22 [58], DiT’23 [36], and our method, on the real-world dataset. Our
method not only handles glow effects but also reveals detailed textures in
low-light regions. Overall, results from our method look more realistic. Zoom
in for better visualisation.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 11: Qualitative results from NightEnhance’23 [17], NightDeFog’20 [53],
Restormer’22 [58], DiT’23 [36], and our method, on the real-world dataset. Our
method not only handles glow effects but also reveals detailed textures in
low-light regions. Overall, results from our method look more realistic. Zoom
in for better visualisation.

(a) Input

(b) UNIE [18]

(c) Ours

Figure 12: Qualitative results from UNIE [18] and our method, on the real-
world dataset. Overall, results from our method look more realistic. Zoom in
for better visualisation.

## Extension to Other Tasks

To verify the applicability of our method to other tasks, we also conduct
experiments on backlit image enhancement [29] and night image enhancement
[18].

### Night Image Enhancement

The goal of night image enhancement is to suppress light effects in bright
regions while, simultaneously, boosting the intensity in dark regions. We use
the LightEffects [18] dataset to evaluate the performance. In this experiment,
we use only the results from our self-prior learning. The experimental results
are presented in Figs. 12, 13, 14 and 15. We can observe that our method can
effectively enhance the visibility of night images.

(a) Input

(b) UNIE [18]

(c) Ours

Figure 13: Qualitative results from UNIE [18] and our method, on the real-
world dataset. Overall, results from our method look more realistic. Zoom in
for better visualisation.

(a) Input

(b) UNIE [18]

(c) Ours

Figure 14: Qualitative results from UNIE [18] and our method, on the real-
world dataset. Overall, results from our method look more realistic. Zoom in
for better visualisation.

(a) Input

(b) UNIE [18]

(c) Ours

Figure 15: Qualitative results from UNIE [18] and our method, on the real-
world dataset. Overall, results from our method look more realistic. Zoom in
for better visualisation.

(a) Input

(b) CLIP-LiT [29]

(c) Ours

Figure 16: Qualitative results from CLIP-LiT [29] and our method, on the real-
world dataset. Overall, results from our method look more realistic. Zoom in
for better visualisation.

(a) Input

(b) CLIP-LiT [29]

(c) Ours

Figure 17: Qualitative results from CLIP-LiT [29] and our method, on the real-
world dataset. Overall, results from our method look more realistic. Zoom in
for better visualisation.

### Backlit Image Enhancement

For backlit image enhancement, We evaluate our method using the Backlit300
dataset [29], which consists of 305 backlit images. Following the evaluation
protocol from [29], we use a non-reference metric, MUSIQ [20], to
quantitatively compare our NightHaze method with state-of-the-art (SOTA)
methods. In this evaluation, we use only the results from our self-prior
learning. The experimental results are presented in Table 3, while Figs. 16
and 17 illustrate the qualitative results. It can be observed that our method
can effectively enhance the visibility of backlit images.

Table 3: Quantitative comparison on the Backlit300 test dataset [29]. Higher
is better.

Method | Input | Afifi et al. [1] | MIT5K [64] | URetinex-Net [52] | Zero-DCE [12] | Zero-DCE++ [25]
---|---|---|---|---|---|---
MUSIQ | 51.900 | 51.930 | 50.354 | 51.551 | 51.250 | 48.435
SCI-easy [33] | SCI-medium [33] | SCI-diffucult [33] | EnlightenGAN [16] | ExCNet [63] | CLIP-LIT [29] | Ours
50.642 | 43.493 | 49.428 | 48.308 | 50.278 | 52.921 | 54.470

## References

  * [1] Afifi, M., Derpanis, K.G., Ommer, B., Brown, M.S.: Learning multi-scale photo exposure correction. In: CVPR (2021)
  * [2] Ancuti, C., Ancuti, C.O., De Vleeschouwer, C., Bovik, A.C.: Night-time dehazing by fusion. In: 2016 IEEE International Conference on Image Processing (ICIP). pp. 2256–2260. IEEE (2016)
  * [3] Ancuti, C., Ancuti, C.O., De Vleeschouwer, C., Bovik, A.C.: Day and night-time dehazing by local airlight estimation. IEEE Transactions on Image Processing 29, 6264–6275 (2020)
  * [4] Berman, D., Treibitz, T., Avidan, S.: Single image dehazing using haze-lines. IEEE transactions on pattern analysis and machine intelligence 42(3), 720–734 (2018)
  * [5] Cai, B., Xu, X., Jia, K., Qing, C., Tao, D.: Dehazenet: An end-to-end system for single image haze removal. IEEE Transactions on Image Processing 25(11), 5187–5198 (2016)
  * [6] Chen, Z., Wang, Y., Yang, Y., Liu, D.: Psd: Principled synthetic-to-real dehazing guided by physical priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7180–7189 (2021)
  * [7] Dong, H., Pan, J., Xiang, L., Hu, Z., Zhang, X., Wang, F., Yang, M.H.: Multi-scale boosted dehazing network with dense feature fusion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2157–2167 (2020)
  * [8] Fattal, R.: Single image dehazing. ACM transactions on graphics (TOG) 27(3), 1–9 (2008)
  * [9] Golestaneh, S.A., Dadsetan, S., Kitani, K.M.: No-reference image quality assessment via transformers, relative ranking, and self-consistency. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1220–1230 (2022)
  * [10] Golts, A., Freedman, D., Elad, M.: Unsupervised single image dehazing using dark channel prior loss. IEEE Transactions on Image Processing 29, 2692–2701 (2019)
  * [11] Guo, C.L., Yan, Q., Anwar, S., Cong, R., Ren, W., Li, C.: Image dehazing transformer with transmission-aware 3d position embedding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5812–5820 (2022)
  * [12] Guo, C., Li, C., Guo, J., Loy, C.C., Hou, J., Kwong, S., Cong, R.: Zero-reference deep curve estimation for low-light image enhancement. In: CVPR (2020)
  * [13] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000–16009 (2022)
  * [14] He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence 33(12), 2341–2353 (2011)
  * [15] Huang, L.Y., Yin, J.L., Chen, B.H., Ye, S.Z.: Towards unsupervised single image dehazing with deep learning. In: 2019 IEEE International Conference on Image Processing (ICIP). pp. 2741–2745. IEEE (2019)
  * [16] Jiang, Y., Gong, X., Liu, D., Cheng, Y., Fang, C., Shen, X., Yang, J., Zhou, P., Wang, Z.: EnlightenGAN: Deep light enhancement without paired supervision. IEEE Transactions on Image Processing 30, 2340–2349 (2021)
  * [17] Jin, Y., Lin, B., Yan, W., Yuan, Y., Ye, W., Tan, R.T.: Enhancing visibility in nighttime haze images using guided apsf and gradient adaptive convolution. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 2446–2457 (2023)
  * [18] Jin, Y., Yang, W., Tan, R.T.: Unsupervised night image enhancement: When layer decomposition meets light-effects suppression. arXiv preprint arXiv:2207.10564 (2022)
  * [19] Ke, J., Wang, Q., Wang, Y., Milanfar, P., Yang, F.: Musiq: Multi-scale image quality transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5148–5157 (2021)
  * [20] Ke, J., Wang, Q., Wang, Y., Milanfar, P., Yang, F.: MUSIQ: Multi-scale image quality transformer. In: ICCV (2021)
  * [21] Kuanar, S., Mahapatra, D., Bilas, M., Rao, K.: Multi-path dilated convolution network for haze and glow removal in nighttime images. The Visual Computer 38(3), 1121–1134 (2022)
  * [22] Li, B., Peng, X., Wang, Z., Xu, J., Feng, D.: Aod-net: All-in-one dehazing network. In: Proceedings of the IEEE international conference on computer vision. pp. 4770–4778 (2017)
  * [23] Li, B., Gou, Y., Gu, S., Liu, J.Z., Zhou, J.T., Peng, X.: You only look yourself: Unsupervised and untrained single image dehazing neural network. International Journal of Computer Vision 129(5), 1754–1767 (2021)
  * [24] Li, B., Gou, Y., Liu, J.Z., Zhu, H., Zhou, J.T., Peng, X.: Zero-shot image dehazing. IEEE Transactions on Image Processing 29, 8457–8466 (2020)
  * [25] Li, C., Guo, C., Loy, C.C.: Learning to enhance low-light image via zero-reference deep curve estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
  * [26] Li, L., Dong, Y., Ren, W., Pan, J., Gao, C., Sang, N., Yang, M.H.: Semi-supervised image dehazing. IEEE Transactions on Image Processing 29, 2766–2779 (2019)
  * [27] Li, R., Pan, J., Li, Z., Tang, J.: Single image dehazing via conditional generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8202–8211 (2018)
  * [28] Li, Y., Chang, Y., Gao, Y., Yu, C., Yan, L.: Physically disentangled intra-and inter-domain adaptation for varicolored haze removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5841–5850 (2022)
  * [29] Liang, Z., Li, C., Zhou, S., Feng, R., Loy, C.C.: Iterative prompt learning for unsupervised backlit image enhancement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8094–8103 (2023)
  * [30] Liu, W., Hou, X., Duan, J., Qiu, G.: End-to-end single image fog removal using enhanced cycle consistent adversarial networks. IEEE Transactions on Image Processing 29, 7819–7833 (2020)
  * [31] Liu, X., Ma, Y., Shi, Z., Chen, J.: Griddehazenet: Attention-based multi-scale network for image dehazing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7314–7323 (2019)
  * [32] Liu, Y., Yan, Z., Wu, A., Ye, T., Li, Y.: Nighttime image dehazing based on variational decomposition model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 640–649 (2022)
  * [33] Ma, L., Ma, T., Liu, R., Fan, X., Luo, Z.: Toward fast, flexible, and robust low-light image enhancement. In: CVPR (2022)
  * [34] Meng, G., Wang, Y., Duan, J., Xiang, S., Pan, C.: Efficient image dehazing with boundary constraint and contextual regularization. In: Proceedings of the IEEE international conference on computer vision. pp. 617–624 (2013)
  * [35] Özdenizci, O., Legenstein, R.: Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
  * [36] Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4195–4205 (October 2023)
  * [37] Pei, S.C., Lee, T.Y.: Nighttime haze removal using color transfer pre-processing and dark channel prior. In: 2012 19th IEEE International conference on image processing. pp. 957–960. IEEE (2012)
  * [38] Qin, X., Wang, Z., Bai, Y., Xie, X., Jia, H.: Ffa-net: Feature fusion attention network for single image dehazing. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 11908–11915 (2020)
  * [39] Qu, Y., Chen, Y., Huang, J., Xie, Y.: Enhanced pix2pix dehazing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8160–8168 (2019)
  * [40] Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M.H.: Single image dehazing via multi-scale convolutional neural networks. In: European conference on computer vision. pp. 154–169. Springer (2016)
  * [41] Ren, W., Ma, L., Zhang, J., Pan, J., Cao, X., Liu, W., Yang, M.H.: Gated fusion network for single image dehazing. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3253–3261 (2018)
  * [42] Shao, Y., Li, L., Ren, W., Gao, C., Sang, N.: Domain adaptation for image dehazing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2808–2817 (2020)
  * [43] Song, Y., He, Z., Qian, H., Du, X.: Vision transformers for single image dehazing. IEEE Transactions on Image Processing 32, 1927–1941 (2023)
  * [44] Su, S., Yan, Q., Zhu, Y., Zhang, C., Ge, X., Sun, J., Zhang, Y.: Blindly assess image quality in the wild guided by a self-adaptive hyper network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3667–3676 (2020)
  * [45] Tan, R.T.: Visibility in bad weather from a single image. In: 2008 IEEE conference on computer vision and pattern recognition. pp. 1–8. IEEE (2008)
  * [46] Tang, Q., Yang, J., He, X., Jia, W., Zhang, Q., Liu, H.: Nighttime image dehazing based on retinex and dark channel prior using taylor series expansion. Computer Vision and Image Understanding 202, 103086 (2021)
  * [47] Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems 30 (2017)
  * [48] Wang, J., Chan, K.C., Loy, C.C.: Exploring clip for assessing the look and feel of images. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 2555–2563 (2023)
  * [49] Wang, W., Wang, A., Liu, C.: Variational single nighttime image haze removal with a gray haze-line prior. IEEE Transactions on Image Processing 31, 1349–1363 (2022)
  * [50] Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: A general u-shaped transformer for image restoration. In: CVPR (2022)
  * [51] Wu, H., Qu, Y., Lin, S., Zhou, J., Qiao, R., Zhang, Z., Xie, Y., Ma, L.: Contrastive learning for compact single image dehazing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10551–10560 (2021)
  * [52] Wu, W., Weng, J., Zhang, P., Wang, X., Yang, W., jiang, J.: Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In: CVPR (2022)
  * [53] Yan, W., Tan, R.T., Dai, D.: Nighttime defogging using high-low frequency decomposition and grayscale-color networks. In: European Conference on Computer Vision. pp. 473–488. Springer (2020)
  * [54] Yang, S., Wu, T., Shi, S., Lao, S., Gong, Y., Cao, M., Wang, J., Yang, Y.: Maniqa: Multi-dimension attention network for no-reference image quality assessment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1191–1200 (2022)
  * [55] Yang, Y., Wang, C., Liu, R., Zhang, L., Guo, X., Tao, D.: Self-augmented unpaired image dehazing via density and depth decomposition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2037–2046 (2022)
  * [56] Ye, T., Zhang, Y., Jiang, M., Chen, L., Liu, Y., Chen, S., Chen, E.: Perceiving and modeling density for image dehazing. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIX. pp. 130–145. Springer (2022)
  * [57] Yu, H., Zheng, N., Zhou, M., Huang, J., Xiao, Z., Zhao, F.: Frequency and spatial dual guidance for image dehazing. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIX. pp. 181–198. Springer (2022)
  * [58] Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5728–5739 (2022)
  * [59] Zhang, H., Patel, V.M.: Densely connected pyramid dehazing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3194–3203 (2018)
  * [60] Zhang, J., Cao, Y., Fang, S., Kang, Y., Wen Chen, C.: Fast haze removal for nighttime image using maximum reflectance prior. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7418–7426 (2017)
  * [61] Zhang, J., Cao, Y., Wang, Z.: Nighttime haze removal based on a new imaging model. pp. 4557–4561. IEEE (2014)
  * [62] Zhang, J., Cao, Y., Zha, Z.J., Tao, D.: Nighttime dehazing with a synthetic benchmark. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 2355–2363 (2020)
  * [63] Zhang, L., Zhang, L., Liu, X., Shen, Y., Zhang, S., Zhao, S.: Zero-shot restoration of back-lit images using deep internal learning. In: Proceedings of the 27th ACM International Conference on Multimedia. pp. 1623–1631 (2019)
  * [64] Zhao, L., Lu, S., Chen, T., Yang, Z., Shamir, A.: Deep symmetric network for underexposed image enhancement with recurrent attentional learning. In: ICCV (2021)
  * [65] Zhao, S., Zhang, L., Shen, Y., Zhou, Y.: Refinednet: A weakly supervised refinement framework for single image dehazing. IEEE Transactions on Image Processing 30, 3391–3404 (2021)
  * [66] Zheng, Z., Ren, W., Cao, X., Hu, X., Wang, T., Song, F., Jia, X.: Ultra-high-definition image dehazing via multi-guided bilateral learning. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16180–16189. IEEE (2021)