Some Simulation and Empirical Results for

Semi-Supervised Learning of the Bayes’ Rule of Allocation

Ziyang Lyu

UNSW Data Science Hub, School of Mathematics and Statistics,

University of New South Wales

Daniel Ahfock and Geoffrey J. McLachlan

School of Mathematics and Physics, University of Queensland

> Abstract:
> There has been increasing attention to semi-supervised learning (SSL)
> approaches in machine learning to forming a classifier in situations where
> the training data consists of some feature vectors that have their class
> labels missing. In this study, we consider the generative model approach
> proposed by Ahfock and McLachlan (2020) who introduced a framework with a
> missingness mechanism for the missing labels of the unclassified features.
> In the case of two multivariate normal classes with a common covariance
> matrix, they showed that the error rate of the estimated Bayes’ rule formed
> by this SSL approach can actually have lower error rate than the one that
> could be formed from a completely classified sample. In this study we
> consider this rather surprising result in cases where there may be more than
> two normal classes with not necessarily common covariance matrices.
>
> Key words and phrases: semi-supervised learning , normal mixture model,
> missing at random, entropy, error rate.

##  1 Introduction

In practice, the training data for the construction of a classifier may not be
completely classified with respect to the predefined distinct classes from
which the feature data have been drawn. That is, there are some feature
vectors in the training data with their class labels missing. One reason for
this is that the procurement of classified data can be quite costly due to
high acquisition costs and subsequent financial, time, and ethical issues that
can arise in attempts to provide the true class labels for the feature vectors
in the data that have been acquired. As noted in the brief review of Ahfock
and McLachlan (2022) there is a now an extensive literature on semi-supervised
learning (SSL). Some initial theoretical results on the SSL approach were
given by McLachlan (1975a, 1977) and O’Neill (1978).

Various SSL methods have been proposed for constructing a classifier from
partially classified training data. We consider here the generative model
approach where a model is adopted for the joint distribution of the feature
vector $\mathbf{Y}$ and its label specified by the $g$-dimensional
$\mathbf{Z}$, which has its $i$th element equal to 0 or 1, according as
$\mathbf{Y}$ does not belong, or belong, to the $i$th class
$C_{i}\,(i=1,\,\ldots,\,g)$. In the context of the generative model, Ahfock
and McLachlan (2020) proposed to introduce the concept of missingness for the
unobserved (missing) labels. They showed there is information to be exploited
in the missing labels where the missingness-mechanism of the class labels is
non-ignorable in the pioneering framework of Rubin (1976) for missingness in
incomplete-data analysis. In the case of $g=2$ normal classes with a common
covariance matrix, they showed asymptotically that the error rate of the
estimated Bayes’ rule can have lower error rate than that if it were formed
from a completely classified sample. They also performed some simulations in
support of their result for the asymptotic relative efficiency (ARE) of the
rule so formed by their SSL approach compared to the rule formed from a
completely classified sample. The adoption of the missing-data mechanism to
gain more information about the parameters of the optimal classifier is most
relevant in commonly occurring situations in practice, where the unclassified
data occur primarily in regions of relatively high entropy in the feature
space thereby making it difficult for their class labels to be easily
obtained.

Ahfock and McLachlan (2020) noted from their examination of several real
partially classified datasets that the unclassified feature vectors tended to
fall in regions of the feature space where there is class overlap. As such
points have relatively high entropy, they proposed that the probability of a
feature vector $\mathbf{y}_{j}$ having a missing label $\mathbf{z}_{j}$ to
depend on the entropy of the feature vector $\mathbf{y}_{j}$.

In this paper, we shall examine further the size of the error rate of the
Bayes’ rule estimated for two and three multivariate normal classes that do
not have a common covariance matrix. Due to the analytical difficulties in
pursuing a theoretical study of the error rates of classiifers, we shall used
simulations and the analyses of two real datasets. It is of interest to see if
the SSL rule adopting the missing-data mechanism for the missing labels as
proposed by Ahfock and McLachlan (2020) can perform close to if not better
than the rule using a completely classified sample. It is also of interest to
see how much better this SSL rule performs relative to the rule that is formed
ignoring the missing-data mechanism.

##  2 Notation

The random variable $\mathbf{Y}$ corresponding to the realisation $\mathbf{y}$
is assumed to have density $f_{i}(\mathbf{y};\boldsymbol{\omega}_{i})$ known
up to a vector $\boldsymbol{\omega}_{i}$ of unknown parameters in Class
$C_{i}\,(i=1,\,\ldots,\,g)$. The optimal (Bayes’) rule of allocation
$R(\mathbf{y};\boldsymbol{\theta})$ assigns an entity with feature vector
$\mathbf{y}$ to Class $C_{k}$ (that is, $R(\mathbf{y};\boldsymbol{\theta})=k)$
if

$k=\arg\max_{i}\;\tau_{i}(\mathbf{y};\boldsymbol{\theta}),$ (2.1)

where

$\tau_{i}(\mathbf{y};\boldsymbol{\theta})={\pi_{i}f_{i}(\mathbf{y};\boldsymbol{\omega}_{i})}/{\sum_{h=1}^{g}\pi_{h}f_{h}(\mathbf{y};\boldsymbol{\omega}_{h})}$
(2.2)

is the posterior probability that the entity belongs to Class $C_{i}$ given
$\mathbf{Y}=\mathbf{y}$ and $\pi_{i}$ is the prior probability that the entity
belongs to $C_{i}\,(i=1,\,\dots,\,g)$;
$\boldsymbol{\theta}=(\pi_{1},\,\ldots,\,\pi_{g-1},\boldsymbol{\theta}_{1}^{T},\,\ldots,\,\boldsymbol{\theta}_{g}^{T})^{T}$
is the vector of unknown parameters. The superscript $T$ denotes vector
transpose.

In order to estimate $\boldsymbol{\theta}$, it is customary in practice to
have available a training sample of size $n$. We let $\mathbf{x}_{\rm
CC}=(\mathbf{x}_{1}^{T},\,\ldots,\,\mathbf{x}_{n}^{T})^{T}$ contain $n$
independent realisations of
$\mathbf{X}=(\mathbf{Y}^{T},\mathbf{Z})^{T},\mathbf{x}_{j}=(\mathbf{y}_{j}^{T},\mathbf{z}_{j})^{T}\,(j=1,\,\ldots,\,n)$,
as the completely classified training data. For a partially classified
training sample $\mathbf{x}_{\rm PC}$ in SSL, we introduce the missing-label
indicator $m_{j}$ which equals 1 if $\mathbf{z}_{j}$ is missing and 0 if it is
available $(j=1,\,\ldots,\,n)$. Thus $\mathbf{x}_{\rm PC}$ consists of those
observations $\mathbf{x}_{j}$ in $\mathbf{x}_{\rm CC}$ with $m_{j}=0$, but
only the feature vector $\mathbf{y}_{j}$ in $\mathbf{x}_{\rm CC}$ if $m_{j}=1$
(that is, the label $\mathbf{z}_{j}$ is missing).

The (Shannon) entropy of an entity with feature vector $\mathbf{y}_{j}$ is
defined by

$e(\mathbf{y},\boldsymbol{\theta})=-\sum\tau_{i}(\mathbf{y};\boldsymbol{\theta})\log\tau_{i}(\mathbf{y};\boldsymbol{\theta}).$
(2.3)

We let $M_{j}$ be the random variable corresponding to the realized value
$m_{j}$ for the missing-label indicator for the feature vector
$\mathbf{y}_{j}$. The missing-data mechanism of Rubin (1976) is specified in
the present context by the conditional probabilities

$\displaystyle{\rm pr}\\{M_{j}=m_{j}\mid\mathbf{y}_{j},\,\mathbf{z}_{j}\\}$
$\displaystyle=$ $\displaystyle{\rm pr}\\{M_{j}=m_{j}\mid\mathbf{y}_{j}\\}$
(2.4) $\displaystyle=$ $\displaystyle
q(\mathbf{y}_{j};\boldsymbol{\Psi})\qquad\qquad\qquad(j=1,\,\ldots,\,n),$
(2.5)

where $\boldsymbol{\Psi}=(\boldsymbol{\theta}^{T},\boldsymbol{\xi}^{T})^{T}$
and where the parameter $\boldsymbol{\xi}=(\xi_{0},\xi_{1})^{T}$ is distinct
from $\boldsymbol{\theta}$. In the case where the probability (2.5) does not
depend on $\boldsymbol{\theta}$ but only $\mathbf{y}_{j}$ and
$\boldsymbol{\xi}$, the missingness is ignorable. But now with this
probability depending also on $\boldsymbol{\theta}$, the missingness is non-
ignorable.

Ahfock and McLachlan (2020) proposed to use the logistic function to model the
probability $q(\mathbf{y}_{j};\boldsymbol{\Psi})$ as

$\displaystyle q(\mathbf{y}_{j};\boldsymbol{\Psi})$ $\displaystyle=$
$\displaystyle\frac{\exp\\{\xi_{0}+\xi_{1}\log
e_{j}(\boldsymbol{\theta})\\}}{1+\exp\\{\xi_{0}+\xi_{1}\log
e_{j}(\boldsymbol{\theta})\\}},$ (2.6)

where $\boldsymbol{\xi}=(\xi_{0},\xi_{1})^{T}$.

The expected proportion $\gamma(\boldsymbol{\Psi})$ of unclassified features
in a partially classified sample $\mathbf{x}_{\rm PC}$ is given by

$\displaystyle\gamma(\boldsymbol{\Psi})$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{n}E(M_{j})/n$ (2.7) $\displaystyle=$ $\displaystyle
E[{\rm pr}\\{M_{j}=1\mid\mathbf{Y}_{j}\\}]$ $\displaystyle=$ $\displaystyle
E\\{q(\mathbf{Y};\boldsymbol{\Psi})\\}.$

Considerable simplification is possible under the two-class normal model,

$\mathbf{Y}\mid(\mathbf{z})_{i}=1\;\sim\,N(\boldsymbol{\mu}_{i},\boldsymbol{\Sigma})\quad{\rm
in}\;\;C_{i}\;\;\;{\rm with\;prob.}\;\;\pi_{i}\quad(i=1,2).$ (2.8)

Under (2.8), the Bayes’ rule reduces to depending on just the
$(p+1)$-dimensional vector of discriminant function coefficients
$\boldsymbol{\beta}=(\beta_{0},\boldsymbol{\beta}_{1}^{T})^{T}$, since

${\rm
logit}\\{\tau_{i}(\mathbf{y};\boldsymbol{\beta})\\}=\beta_{0}+\boldsymbol{\beta}_{1}^{T}\mathbf{y},$
(2.9)

where

$\displaystyle\beta_{0}$ $\displaystyle=$
$\displaystyle\log(\pi_{1}/\pi_{2})-{\textstyle\frac{1}{2}}(\boldsymbol{\mu}_{1}+\boldsymbol{\mu}_{2})^{T}\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2}),$
(2.10) $\displaystyle\boldsymbol{\beta}_{1}$ $\displaystyle=$
$\displaystyle\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2}).$
(2.11)

The two-class homoscedastic normal model (2.8) can be reparameterized by
taking
$\boldsymbol{\theta}=(\boldsymbol{\theta}_{1}^{T},\boldsymbol{\beta}^{T})^{T},$
where $\boldsymbol{\theta}_{1}$ contains the elements of
$\boldsymbol{\mu}=\pi_{1}\boldsymbol{\mu}_{1}+\pi_{2}\boldsymbol{\mu}_{2}$ and
the distinct elements of

$\boldsymbol{\Lambda}=\boldsymbol{\Sigma}+\pi_{1}\pi_{2}(\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2})(\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2})^{T}.$

It has the convenient canonical form,

$\boldsymbol{\Sigma}=\mathbf{I}_{p},\;\boldsymbol{\mu}_{1}=(\Delta,0,\,\ldots,\,0)^{T},\;\boldsymbol{\mu}_{2}=(0,\,\ldots,\,0)^{T},$
(2.12)

where $\mathbf{I}_{p}$ is the $p\times p$ identity matrix and

$\Delta^{2}=(\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2})^{T}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2})$

is the squared Mahalanobis distance between the two classes.

##  3 Estimation of the Bayes’ Rule

We now outline the estimation of the Bayes’ rule of allocation under the
missing-label mechanism as specified by (2.6), which implies that the
missingness is non-ignorable since the probability
$q(\mathbf{y}_{j};\boldsymbol{\Psi})$ depends on $\boldsymbol{\theta}$, and
hence, the parameters of the Bayes’ rule.

We let

$\displaystyle\log L_{\rm PC}^{(\rm ig)}(\boldsymbol{\theta})$
$\displaystyle=$ $\displaystyle\log L_{\rm C}(\boldsymbol{\theta})+\log L_{\rm
UC}(\boldsymbol{\theta})$ (3.1)

denote the log partial likelihood function for $\boldsymbol{\theta}$ formed by
ignoring the missing-data mechanism for the missing-label indicators $m_{j}$,
where

$\displaystyle\log L_{\rm C}(\boldsymbol{\theta})$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{n}(1-m_{j})\,\sum_{i=1}^{g}z_{ij}\log\\{\pi_{i}f_{i}(\mathbf{y}_{j};\boldsymbol{\omega}_{i})\\}$
(3.2)

and

$\displaystyle\log L_{\rm UC}(\boldsymbol{\theta})$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{n}m_{j}\log\sum_{i=1}^{g}\pi_{i}f_{i}(\mathbf{y}_{j};\boldsymbol{\omega}_{i}),$
(3.3)

denote the log function for $\boldsymbol{\theta}$ formed on the basis of the
classified data and unclassified data, respectively. The log of the likelihood
$L_{\rm CC}(\boldsymbol{\theta})$ for the completely classified sample
$\mathbf{x}_{\rm CC}$ is given by (3.2) with all $m_{j}=0$,

$\displaystyle\log L_{\rm CC}(\boldsymbol{\theta})$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{n}\,\sum_{i=1}^{g}z_{ij}\log\\{\pi_{i}f_{i}(\mathbf{y}_{j};\boldsymbol{\omega}_{i})\\}.$
(3.4)

The full likelihood function for $\boldsymbol{\Psi}$ is given by

$\log L_{\rm PC}^{(\rm full)}(\boldsymbol{\Psi})=\log L_{\rm PC}^{(\rm
ig)}(\boldsymbol{\theta})+\log L_{\rm PC}^{(\rm miss)}(\boldsymbol{\Psi}),$
(3.5)

where

$\displaystyle\log L_{\rm PC}^{(\rm
miss)}(\boldsymbol{\Psi})=\sum_{j=1}^{n}[(1-m_{j})\log\\{1-q(\mathbf{y}_{j};\boldsymbol{\Psi})\\}+m_{j}\log
q(\mathbf{y}_{j};\boldsymbol{\Psi})]$ (3.6)

is the log likelihood function for $\boldsymbol{\Psi}$ formed on the basis of
the missing-label indicators $m_{j}$.

We let $\hat{\mbox{\boldmath$\boldsymbol{\Psi}$}}_{\rm PC}^{(\rm full)}$ be
the estimate of $\boldsymbol{\Psi}$ formed by consideration of the full
likelihood $L_{\rm PC}^{(\rm full)}(\boldsymbol{\Psi})$ and
$R(\mathbf{y};\hat{\boldsymbol{\theta}}_{\rm PC}^{(\rm full)})$ be the
estimated Bayes’ rule obtained by plugging in $\hat{\boldsymbol{\theta}}_{\rm
PC}^{(\rm full)}$ for $\hat{\boldsymbol{\theta}}$ in the Bayes’ rule
$R(\mathbf{y};\hat{\boldsymbol{\theta}})$. Also, we let
$R(\mathbf{y};\hat{\boldsymbol{\theta}}_{\rm CC})$ and
$R(\mathbf{y};\hat{\boldsymbol{\theta}}_{\rm PC}^{(\rm ig)})$ denote the
estimated Bayes’ rule obtained by plugging in $\hat{\boldsymbol{\theta}}_{\rm
CC}$ and $\hat{\boldsymbol{\theta}}_{\rm PC}^{(\rm ig)}$, respectively, for
$\boldsymbol{\theta}$ in $R(\mathbf{y};\boldsymbol{\theta})$.

The overall conditional error rate of the rule
$R(\mathbf{y};\hat{\boldsymbol{\theta}}_{\rm CC})$ is defined by

${\rm err}(\hat{\boldsymbol{\theta}}_{\rm
CC};\boldsymbol{\theta})=1-\sum_{i=1}^{g}\pi_{i}\,{\rm
pr}\\{R(\mathbf{Y};\hat{\boldsymbol{\theta}}_{CC})=i\mid\hat{\boldsymbol{\theta}}_{CC},(\mathbf{Z})_{i}=1\\}.$
(3.7)

The corresponding conditional error rate ${\rm
err}(\hat{\boldsymbol{\theta}}_{PC}^{(\rm ig)};\boldsymbol{\theta})$ of the
rule $R(\mathbf{y};\hat{\boldsymbol{\theta}}_{\rm PC}^{(\rm ig)})$ is defined
by replacing $\hat{\boldsymbol{\theta}}_{CC}$ with
$\hat{\boldsymbol{\theta}}_{PC}^{(\rm ig)}$ in (3.7). The optimal error rate
${\rm err}(\boldsymbol{\theta})$ is defined by replacing
$\hat{\boldsymbol{\theta}}_{\rm CC}$ by $\boldsymbol{\theta}$ in (3.7).

##  4 Simulation study

We now report the results of some simulation experiments to compare the rules
$R_{\text{PC}}^{(\text{full})}$ and $R_{\text{PC}}^{(\text{ig})}$ on a
partially classified sample and $R_{\text{CC}}$ on a completely classified
sample.

###  4.1 Two-class normal model with unequal covariance matrices

We study a two-class normal model with unequal covariance matrices. We
consider proportional covariance matrices; i.e., we assume
$\boldsymbol{\Sigma}_{1}=\boldsymbol{\Sigma}$ and
$\boldsymbol{\Sigma}_{2}=\lambda\boldsymbol{\Sigma}$. This model has been
studied by Bartlett and Please (1963), Gilbert (1969), Han (1969, 1974) ,Marks
and Dunn (1974) and McLachlan (1975b). The error rate of the Bayes’ rule is
evaluated for various combinations of the parameters in canonical form, where,
$\boldsymbol{\mu}_{1}=(\Delta,0,\ldots,0)^{T}$,
$\boldsymbol{\mu}_{1}=\boldsymbol{0}$, and $\boldsymbol{\Sigma}=\mathbf{I}$.
Marks and Dunn (1974) and McLachlan (1975b) introduced a parameter
$\Delta^{*}=\Delta/(1+\sqrt{\lambda})$ as a measure of the degree of
separation of two populations.

We generated $B=1000$ samples with sample size $n=200$ and $\pi_{1}=\pi_{2}$.
For each combination of $p\in\\{2,3,4\\}$, $\lambda\in\\{2,3,4\\}$,
$\Delta^{*}$ (from 0.2 to 4) and
$\boldsymbol{\xi}\in\\{(3,1)^{T},(3,4)^{T}\\}$, the observations and the
missing label indicators in each sample were generated via normal mixture
model and (2.3), respectively.

To assess the influence on the relative efficiency, we followed the above
simulation setup but replaced $\Delta^{*}$ (degree of separation of the two
populations) with $\Delta\in\\{1,2,3\\}$. In each replication, the estimates
$\hat{\boldsymbol{\theta}}_{\text{CC}}$,
$\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{ig})}$, and
$\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})}$ were computed using
the log likelihood function (3.4), (3.1), and (3.5), respectively. As we are
unable to calculate the conditional error rate (3.7) analytically, we used a
holdout sample of size $0.2\times n$ on each simulation trial to assess the
conditional error rate.

The relative efficiency (RE) of
$R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{full})})$ compared to
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ was estimated as

$\overline{\text{RE}}\\{{R}(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{full})})\\}=\frac{B^{-1}\sum_{b=1}^{B}\\{\text{err}(\hat{\boldsymbol{\theta}}_{\text{CC}}^{(b)})-\text{err}(\boldsymbol{\theta})\\}}{B^{-1}\sum_{b=1}^{B}\\{\text{err}(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full},b)})-\text{err}(\boldsymbol{\theta})\\}},$
(4.1)

and the RE of $R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$
compared to $R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{full})})$ was
estimated as

$\overline{\text{RE}}\\{R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{ig})})\\}=\frac{B^{-1}\sum_{b=1}^{B}\\{\text{err}(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full},b)})-\text{err}(\boldsymbol{\theta})\\}}{B^{-1}\sum_{b=1}^{B}\\{\text{err}(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig},b)})-\text{err}(\boldsymbol{\theta})\\}}.$
(4.2)

The nonparametric bootstrap with 1000 re-samples was used to assess the
variability of the estimates. All the bootstrap standard errors were small
relative to their estimand.

Table 1:  Simulated relative efficiency
$\overline{\text{RE}}\\{{R}(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{full})})\\}$
for $g=2$, $\pi_{1}=\pi_{2}$, $\boldsymbol{\mu}_{1}=(\Delta,0,\ldots,0)^{T}$,
$\boldsymbol{\mu}_{2}=\boldsymbol{0}$, $\boldsymbol{\Sigma}_{1}=\mathbf{I}$
and $\boldsymbol{\Sigma}_{2}=\lambda\mathbf{I}$ for $n=200$ ( proportion of
missing labels in parentheses)

$\lambda$ | $\Delta$ | $\boldsymbol{\xi}=(3,1)^{T}$ | $\boldsymbol{\xi}=(3,4)^{T}$
---|---|---|---
$\it p$=2 |  $\it p$=3 |  $\it p$=4 |  $\it p$=2 |  $\it p$=3 |  $\it p$=4
2 | 1 | 0.21 (0.03) | 0.27 (0.13) | 0.28 (0.11) | 1.2 (0.06) | 1.71 (0.12) | 1.71 (0.18)
2 | 0.26 (0.03) | 0.7 (0.08) | 0.5 (0.07) | 1.93 (0.13) | 1.95 (0.12) | 4.00 (0.32)
3 | 1.07 (0.02) | 1.09 (0.03) | 1.09 (0.16) | 1.71 (0.12) | 1.58 (0.18) | 1.58 (0.18)
3 | 1 | 0.28 (0.04) | 0.53 (0.12) | 0.57 (0.12) | 1.52 (0.16) | 1.54 (0.14) | 1.54 (0.26)
2 | 0.34 (0.19) | 0.73 (0.14) | 0.87 (0.04) | 2.93 (0.14) | 1.71 (0.32) | 1.76 (0.26)
3 | 0.35 (0.07) | 1.34 (0.12) | 1.34 (0.16) | 3.1 (0.24) | 1.39 (0.39) | 1.39 (0.21)
4 | 1 | 0.37 (0.16) | 0.61 (0.23) | 0.75 (0.17) | 1.25 (0.15) | 1.82 (0.25) | 1.79 (0.36)
2 | 0.5 (0.16) | 0.94 (0.09) | 1.08 (0.19) | 1.5 (0.12) | 2.46 (0.27) | 1.59 (0.26)
3 | 0.46 (0.07) | 1.09 (0.16) | 1.09 (0.06) | 3.04 (0.82) | 1.58 (0.13) | 2.03 (0.11)

Table 2:  Simulated relative efficiency $\overline{\text{RE}}\\{{R}(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{ig})})\\}$ for $g=2$, $\pi_{1}=\pi_{2}$, $\boldsymbol{\mu}_{1}=(\Delta,0,\ldots,0)^{T}$, $\boldsymbol{\mu}_{2}=\boldsymbol{0}$, $\boldsymbol{\Sigma}_{1}=\mathbf{I}$ and $\boldsymbol{\Sigma}_{2}=\lambda\mathbf{I}$ for $n=200$ $\lambda$ | $\Delta$ | $\boldsymbol{\xi}=(3,1)^{T}$ | $\boldsymbol{\xi}=(3,4)^{T}$
---|---|---|---
$\it p$=2 |  $\it p$=3 |  $\it p$=4 |  $\it p$=2 |  $\it p$=3 |  $\it p$=4
2 | 1 | 0.78 | 0.87 | 0.78 | 0.28 | 0.24 | 0.24
2 | 0.51 | 0.45 | 0.44 | 0.24 | 0.32 | 0.15
3 | 0.33 | 0.34 | 0.34 | 0.52 | 0.41 | 0.41
3 | 1 | 0.57 | 0.57 | 0.55 | 0.29 | 0.36 | 0.36
2 | 0.67 | 0.45 | 0.44 | 0.21 | 0.34 | 0.39
3 | 0.51 | 0.34 | 0.34 | 0.27 | 0.53 | 0.53
4 | 1 | 0.58 | 0.58 | 0.48 | 0.34 | 0.39 | 0.33
2 | 0.50 | 0.41 | 0.38 | 0.36 | 0.26 | 0.4
3 | 0.34 | 0.40 | 0.36 | 0.83 | 0.51 | 0.55

It cam be seen in Table 4 that when $\xi_{1}=4$, the simulated relative
efficiency of $R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ is
greater than 1, implying its error rate is lower than the rule
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ based on a completely classified
sample. However, the simulated relative efficiency of
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ is generally less
than one for the combinations of the parameters with $\xi_{1}=1.$ This can be
explained by the fact that the parameter $\xi_{1}$ is the coefficient of the
log entropy of a feature vector in the postulated logistic function for the
probability of a missing label. Hence as it increases from 1 to 4 for a given
level of the other parameter $\xi_{0}$ in the logistic function,it implies
that the feature vectors with missing labels in the generated sample for
$\xi=4$ have higher entropy than for $\xi=1$.

Table 2 reports the results for the simulated relative efficiency of
$R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{ig})})$ compared to
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$. All these
simulated values of the relative efficiency are smaller than 1, confirming
that the classifier $R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$
performs better than
$R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{ig})})$.

###  4.2 Three-class normal model with unequal covariance matrices

In this simulation, we consider a three-class normal model with unequal
covariance matrices.We specified $\boldsymbol{\mu}_{1}=(-1,0,\ldots,0)^{T}$,
$\boldsymbol{\mu}_{2}=\boldsymbol{\mu}_{3}=(1,0,\ldots,0)^{T}$, and
$\boldsymbol{\Sigma}_{i}=\lambda_{i}\mathbf{I}$ $(i=1,2,3)$. For each
combination of $\boldsymbol{\pi}$ and $\lambda_{i}$
($\boldsymbol{\pi}=(0.5,0.35,0.15)^{T},(1/3,1/3,1/3)^{T}$;
$\lambda_{1}=0.5,1.5,2.5$; $\lambda_{2}=1,2,3$; $\lambda_{3}=2,4,6$), we
generated $B=1000$ samples with sample size $n=200$ and $p=2$. Note that if
$\lambda_{2}=\lambda_{3}=2$, observations would be actually generated from a
two-component normal model. Therefore, we do not consider this simulation
setting.

For the partially classified sample $\mathbf{x}_{{\text{PC}}}$, the missing
label indicators were generated using the missing label probability in (2.3),
where we set $\boldsymbol{\xi}=(3,7)^{T}$. In each replication, the estimates
$\hat{\boldsymbol{\theta}}_{\text{CC}}$,
$\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{ig})}$, and
$\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})}$ were computed using
the log likelihood function, (3.4), (3.1), and (3.5), respectively, and we
generated a hold-out sample of size $0.2\times n$, to calculate the error rate
via (3.7).

Table 3:  Simulated relative efficiency
$\overline{\text{RE}}\\{R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{full})})\\}$
with $g=3$, $\boldsymbol{\mu}_{1}=(-1,0,\ldots,0)^{T}$,
$\boldsymbol{\mu}_{2}=\boldsymbol{\mu}_{3}=(1,0,\ldots,0)^{T}$, and
$\boldsymbol{\Sigma}_{i}=\lambda_{i}\mathbf{I}$ $(i=1,2,3)$ for $n=200$, and
$p=2$ with $\boldsymbol{\xi}=(3,7)^{T}$ ( Bootstrap standard errors are in
parentheses)

$\lambda_{2}$ | $\lambda_{3}$ | $\boldsymbol{\pi}=(0.5,0.35,0.15)^{T}$ | $\pi_{1}=\pi_{2}=\pi_{3}$
---|---|---|---
$\lambda_{1}=0.5$ | $\lambda_{1}=1.5$ | $\lambda_{1}=2.5$ | $\lambda_{1}=0.5$ | $\lambda_{1}=1.5$ | $\lambda_{1}=2.5$
1 | 2 | 1.7 (0.18) | 0.49 (0.13) | 0.34 (0.11) | 1.05 (0.28) | 0.51 (0.15) | 0.32 (0.09)
| 4 | 2.4 (0.13) | 1.83 (0.24) | 0.9 (0.21) | 2.54 (0.37) | 0.75 (0.19) | 1.07 (0.14)
| 6 | 2.93 (0.45) | 2.27 (0.48) | 1.41 (0.26) | 2.3 (0.39) | 1.81 (0.16) | 1.13 (0.27)
2 | 4 | 1.81 (0.38) | 0.91 (0.05) | 0.31 (0.12) | 1.05 (0.26) | 0.44 (0.18) | 0.31 (0.18)
| 6 | 1.76 (0.12) | 1.3 (0.16) | 0.7 (0.21) | 2.15 (0.13) | 0.61 (0.17) | 0.48 (0.12)
3 | 2 | 0.67 (0.12) | 0.34 (0.12) | 0.22 (0.07) | 0.82 (0.13) | 0.53 (0.18) | 0.37 (0.18)
| 4 | 1.05 (0.12) | 0.55 (0.12) | 0.28 (0.03) | 0.81 (0.13) | 0.5 (0.16) | 0.28 (0.09)
| 6 | 1.22 (0.26) | 1.07 (0.35) | 0.51 (0.16) | 1.12 (0.28) | 0.55 (0.11) | 0.31 (0.07)

Table 4:  Simulated relative efficiency $\overline{\text{RE}}\\{R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{ig})})\\}$ with $g=3$, $\boldsymbol{\mu}_{1}=(-1,0,\ldots,0)^{T}$, $\boldsymbol{\mu}_{2}=\boldsymbol{\mu}_{3}=(1,0,\ldots,0)^{T}$, and $\boldsymbol{\Sigma}_{i}=\lambda_{i}\mathbf{I}$ $(i=1,2,3)$. for $n=200$, and $p=2$ with $\boldsymbol{\xi}=(3,7)^{T}$ $\lambda_{2}$ | $\lambda_{3}$ | $\pi=(0.5,0.35,0.15)^{T}$ | $\pi_{1}=\pi_{2}=\pi_{3}$
---|---|---|---
$\lambda_{1}=0.5$ | $\lambda_{1}=1.5$ | $\lambda_{1}=2.5$ | $\lambda_{1}=0.5$ | $\lambda_{1}=1.5$ | $\lambda_{1}=2.5$
1 | 2 | 0.27 | 0.38 | 0.47 | 0.42 | 0.52 | 0.74
| 4 | 0.23 | 0.17 | 0.33 | 0.15 | 0.33 | 0.25
| 6 | 0.24 | 0.18 | 0.25 | 0.23 | 0.20 | 0.26
2 | 4 | 0.28 | 0.31 | 0.55 | 0.41 | 0.51 | 0.63
| 6 | 0.35 | 0.24 | 0.40 | 0.21 | 0.36 | 0.46
3 | 2 | 0.68 | 0.53 | 0.78 | 0.72 | 0.67 | 0.81
| 4 | 0.38 | 0.44 | 0.62 | 0.84 | 0.58 | .80
| 6 | 0.43 | 0.30 | 0.42 | 0.47 | 0.42 | 0.67

The results for the relative efficiency of
$R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{full})})$ compared to
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ and of
$R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{ig})})$ compared to
$R(\hat{\boldsymbol{\theta}}_{{\text{PC}}}^{(\text{full})})$ are summarized in
Table 3 and 4, respectively.

From the results for the simulated relative efficiency of
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ compared to
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ it would appear that it is higher
for combinations of the parameters in which the classes are occurring in equal
proportions. However, further simulations are needed. Also, it is of interest
to explore the effect of the level of the parameter $\xi_{1}$ on the relative
efficiency. As to be expected the simulated relative efficiency of the rule
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$ that is formed using
the partial likelihood that ignores the missing-data mechanism for the labels
of the unclassified feature vectors in the training sample is always less than
1.

##  5 Real Datasets

In most practical applications, class labels are assigned based on human
expertise. Such manual annotation of a dataset can induce a systematic
missingness mechanism. We present the following results for two real datasets
to demonstrate that a classifier based on this missingness mechanism can
perform better than one that ignores the missingness mechanism, and can even
perform better than the classifier based on the completely classified sample.

###  5.1 Interneuron dataset

There is currently no unique catalogue of cortical GABAergic interneuron
types. Mihaljević et al. (2019) asked 48 leading neuroscientists to classify
320 interneurons by inspecting images of their morphology. Each neuroscientist
used a web application to classify the interneurons, and each image was
resized to $32\times 32$ pixels. A total of 48 neuroscientists participated in
the study. Here, we consider only the 42 neuroscientists who fully classified
all 320 neurons. We consider 2 classes in this dataset: transcolumnar and
intracolumnar. We discard those observations for which more than or equal to
half of the 42 neuroscientists viewed an entry was not applicable to a given
interneuron . Thus, we consider a total of $n=304$ observations. The ground
truth for each image was generated by agreement among at least half of the 42
neuroscientists.

Figure 1:  Analysis of the interneuron dataset with regard to the relationship
between the entropy and the labelled and unlabelled observations.

We select five neuroscientists who reviewed the images and 3D visualizations
to determine whether an interneuron was transcolumnar or intracolumnar. We
form a consensus labeling using the individual neuroscientists’ labels.
Observations for which all five neuroscientists agreed are treated as labelled
data, corresponding to $n_{l}=173$. Observations for which fewer than five
neuroscientists agreed are treated as unlabelled data, corresponding to
$n_{u}=131$. The dimensionality of the dataset is reduced to 3 features via
principal component analysis (PCA) prior to classification.

We fit a skew $\it t$-mixture model (Lee and McLachlan, 2014) to estimate the
entropy of each observation. Figure 1 (a) compares the empirical cumulative
distribution functions of the estimated entropy distributions in the labelled
and unlabelled groups. Figure 1 (b) presents a Nadaraya–Watson kernel estimate
of the probability of missing class labels. From Figure 1 (a), we find that
the unlabelled observations typically have higher entropy than the labelled
observations. Figure 1 (b) shows that the estimated probability of missing
class labels decreases as the negative log entropy increases. This is in
accordance with the relation of equation (2.6). The higher the entropy is, the
higher the probability of the missing class labels. Therefore, we fit a three-
class normal model with unequal covariance matrices to compare the three
classifiers $R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ ,
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$, and
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$.

We apply five-fold cross-validation (CV) to access the error rates of the
three classifiers.The error rate summary is given in Table 5, and we see that
the five-fold CV error rate from the partially classified sample based on the
missingness mechanism is smaller than that obtained when ignoring the
missingness mechanism and close to the fivefold CV error rate from the
completely classified sample.

Table 5: Summary statistics for the interneuron dataset. There are n = 304 feature vector observations with dimensions of $\it p$ = 3. There are g = 2 classes (transcolumnar and intracolumnar). Classifier |  $n_{l}$(classified) |  $n_{u}$(unclassified) | 5-fold CV error rate ($\%$)
---|---|---|---
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ | 304 | 0 | 24.65
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$ | 173 | 131 | 34.06
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ | 173 | 131 | 26.29

###  5.2 Skin lesion dataset

Dermatoscopic images can be accessed and downloaded from Harvard Dataverse
provided by Tschandl (2018). We select 3 classes (Figure 2), i.g., benign
keratosis (BKL), dermatofibroma (DF), and vascular skin lesions (VASC). The
dataset includes a set of ground–truth labels of two types. Some labels have
been confirmed by histopathology (performed by specialized
dermatopathologists), while the rest have been confirmed either by follow-up,
expert consensus, or in-vivo confocal microscopy. We treat the observations
diagnosed by dermatopathologists, as labelled data and the rest of the cases
as data with missing labels. Thus, the subset we consider contains $n=1356$
observations, with $n_{l}=888$ labelled observations and $n_{u}=468$
unlabelled observations. To reduce the dimensionality of the dataset, we use
sparse linear discriminant analysis (Clemmensen et al., 2011), to select a
subset of four features for model based clustering.

Figure 2: (1): ”Benign keratosis” is a generic class that includes seborrheic
keratoses (” senile wart”), solar lentigo. (2): Dermatofibroma is a benign
skin lesion regarded as either a benign proliferation or an inflammatory
reaction to minimal trauma. (3): The vascular skin lesions in the dataset
range from cherry angiomas to angiokeratomas and pyogenic granulomas.

Similarly, we fit a skew $\it t$-mixture model (Lee and McLachlan, 2014) to
estimate the classification entropy of each observation. Figure 3 (a) compares
box-plots of the estimated entropies in the labelled and unlabelled groups.
Figure 3 (b) presents a Nadaraya–Watson kernel estimate of the probability of
missing class labels. From Figure 3 (a), we see that the unlabelled
observations typically have higher entropy than the labelled observations.
Figure 3 (b) shows the estimated probability of missing class labels increases
as the entropy increases. This is in accordance with the relation of equation
(2.3). Therefore, we fit a three-class normal model with unequal covariance
matrices to compare the three classifiers
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ ,
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$, and
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$.

Figure 3:  Analysis of the skin lesion dataset with regard to the relationship between the entropy and the labelled and unlabelled observations. Table 6: Summary statistics for the skin lesion dataset.There are $n=1356$ feature vector observations with $p=4$ dimensions. There are $g=3$ classes (BKL, DF, and VASC). Classifier |  $n_{l}$(classified) |  $n_{u}$(unclassified) | 5-fold CV error rate ($\%$)
---|---|---|---
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ | 1356 | 0 | 19.98
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$ | 888 | 468 | 20.06
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ | 888 | 468 | 19.69

We apply five-fold CV to access the error rates of three classifiers,
splitting the skin lesion dataset into a test set and a training set. The
error rate summary is given in Table 6. We see that the error rate from the
partially classified sample based on the missingness mechanism is smaller than
that obtained when the missingness mechanism is ignored and even better than
that from the completely classified sample. But the difference between the
error rates are very small, suggesting that the unlabelled observations must
be easy to classify in the iterative fitting of the EM algorithm and entropies
in Figure 3(a) are relative low which can explain that
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ does not do much
better than $R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$ and is
close to $R(\hat{\boldsymbol{\theta}}_{\text{CC}})$.

Note that in using cross-validation to estimate the error rate of the various
rules, the rule $R(\hat{\boldsymbol{\theta}}_{\rm CC})$ has an advantage over
$R(\hat{\boldsymbol{\theta}}_{\rm PC})$ in their application to the
unclassified feature vectors since $R(\hat{\boldsymbol{\theta}}_{\rm CC})$ is
formed using the true values of the labels for the unclassified features.
Bearing this in mind, the full rule based on the partially classified sample
does well to be fairly close to that of the rule based on the completely
classified sample in the former Interneuron dataset and even better in this
Skin Lesion dataset.

##  6 Discussion

In the case of two multivariate normal classes with equal covariances, Ahfock
and McLachlan (2020) showed that the SSL rule
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ that can be formed
from a partially classified training sample that can have smaller error rate
than that of the rule $R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ based on the
sample completely classified with respect to the two predefined classes. In
this study we considered the performance of the SSL rule
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ in the case of two
multivariate normal classes with proportional covariance matrices and for
three multivariate normal classes. It confirmed that the SSL rule
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ formed by adopting a missing-data
mechanism for the missing labels can be more efficient than the rule
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ based on the sample completely
classified. The relative efficiency these two rules depends on the entropy of
the feature vectors of the entities with missing labels in the partially
classified sample..

In a subsequent paper, Ahfock and McLachlan (2022) derived the asymptotic
relative efficiency of the rule
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$ formed using a
partial likelihood that ignores the missing-data mechanism. It was compared to
their aforementioned full rule
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ . In our
simulations, we also simulated the relative efficiency of
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$ compared to
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$, and hence, to
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$. As to be expected,
this latter relative efficiency was always less than one and quantified the
loss of information in ignoring the missing-data mechanism for the missing
labels of the feature vectors in the partially classified training sample.

We also reported the analyses of two real datasets in which the two rules
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{full})})$ and
$R(\hat{\boldsymbol{\theta}}_{\text{PC}}^{(\text{ig})})$ were applied and
compared to the results using the rule
$R(\hat{\boldsymbol{\theta}}_{\text{CC}})$ based on using the labels of all
the feature vectors in the dataset.

## References

  * Ahfock and McLachlan (2020) Ahfock, D. and G. J. McLachlan (2020).  An apparent paradox: a classifier based on a partially classified sample may have smaller expected error rate than that if the sample were completely classified.  Statistics and Computing, 1–12.
  * Ahfock and McLachlan (2022) Ahfock, D. and G. J. McLachlan (2022).  Semi-supervised learning of classifiers from a statistical perspective: A brief review.  Econometrics and Statistics.
  * Bartlett and Please (1963) Bartlett, M. and N. Please (1963).  Discrimination in the case of zero mean differences.  Biometrika 50(1/2), 17–21.
  * Clemmensen et al. (2011) Clemmensen, L., T. Hastie, D. Witten, and B. Ersbøll (2011).  Sparse discriminant analysis.  Technometrics 53(4), 406–413.
  * Gilbert (1969) Gilbert, E. S. (1969).  The effect of unequal variance-covariance matrices on fisher’s linear discriminant function.  Biometrics, 505–515.
  * Han (1969) Han, C.-P. (1969).  Distribution of discriminant function when covariance matrices are proportional.  The Annals of Mathematical Statistics 40(3), 979–985.
  * Han (1974) Han, C.-P. (1974).  Asymptotic distribution of discriminant function when covariance matrices are proportional and unknown.  Annals of the Institute of Statistical Mathematics 26(1), 127–133.
  * Lee and McLachlan (2014) Lee, S. and G. J. McLachlan (2014).  Finite mixtures of multivariate skew t-distributions: some recent and new results.  Statistics and Computing 24(2), 181–202.
  * Marks and Dunn (1974) Marks, S. and O. J. Dunn (1974).  Discriminant functions when covariance matrices are unequal.  Journal of the American Statistical Association 69(346), 555–559.
  * McLachlan (1975a) McLachlan, G. J. (1975a).  Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis.  Journal of the American Statistical Association 70(350), 365–369.
  * McLachlan (1975b) McLachlan, G. J. (1975b).  Some expected values for the error rates of the sample quadratic discriminant function1.  Australian Journal of Statistics 17(3), 161–165.
  * McLachlan (1977) McLachlan, G. J. (1977).  Estimating the linear discriminant function from initial samples containing a small number of unclassified observations.  Journal of the American statistical association 72(358), 403–406.
  * Mihaljević et al. (2019) Mihaljević, B., R. Benavides-Piccione, C. Bielza, P. Larrañaga, and J. DeFelipe (2019).  Classification of gabaergic interneurons by leading neuroscientists.  Scientific data 6(1), 1–6.
  * O’Neill (1978) O’Neill, T. J. (1978).  Normal discrimination with unclassified observations.  Journal of the American Statistical Association 73(364), 821–826.
  * Rubin (1976) Rubin, D. B. (1976).  Inference and missing data.  Biometrika 63(3), 581–592.
  * Tschandl (2018) Tschandl, P. (2018).  The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.


iint iint TXFiint TXF iint

# LVRNet: Lightweight Image Restoration for
Aerial Images under Low Visibility

Esha Pahwa
BITS Pilani
<EMAIL_ADDRESS>equal contribution Achleshwar
Luthra11footnotemark: 1
Carnegie Mellon University
<EMAIL_ADDRESS>Pratik Narang
BITS Pilani
<EMAIL_ADDRESS>
###### Abstract

Learning to recover clear images from images having a combination of degrading
factors is a challenging task. That being said, autonomous surveillance in low
visibility conditions caused by high pollution/smoke, poor air quality index,
low light, atmospheric scattering, and haze during a blizzard becomes even
more important to prevent accidents. It is thus crucial to form a solution
that can result in a high-quality image and is efficient enough to be deployed
for everyday use. However, the lack of proper datasets available to tackle
this task limits the performance of the previous methods proposed. To this
end, we generate the LowVis-AFO dataset, containing 3647 paired dark-hazy and
clear images. We also introduce a lightweight deep learning model called Low-
Visibility Restoration Network (LVRNet). It outperforms previous image
restoration methods with low latency, achieving a PSNR value of 25.744 and an
SSIM of 0.905, making our approach scalable and ready for practical use. The
code and data can be found here.

Figure 1: Visual results on the proposed LowVis-AFO dataset. The method used
to obtain each result has been mentioned under the image.

##  1 Introduction

Image enhancement and restoration have been a critical area of research using
both traditional digital image processing techniques[12] [2], and the recent
deep learning frameworks[32][33][44]. The goal of image restoration is to
recover a clear image, whereas image enhancement is to improve the quality of
the degraded image. In this study, we perform recovery of the clear image from
the hazy version while performing low-light image enhancement using a single
convolutional network, which could further be applied to tasks such as search
and rescue operations using object detection.

Using deep learning algorithms for image recovery has many benefits, the most
important being that it can generalize to different variations in the images
captured. Hence, we observe that deep learning-based methods on most benchmark
datasets often outperform traditional methods significantly. However, there
are still challenges that the researchers have to tackle for image
restoration. Publicly available datasets containing a variety of degrading
factors that model real-world scenarios are few. Hence, most previous works
have focused on removing one type of degradation with a specific intensity
level. From the perspective of computational complexity, recent deep learning
methods are computationally expensive, and thus they can’t be deployed on edge
devices. Moreover, image restoration has been a long-standing ill-posed
research problem, as there are infinite mappings between the degraded and the
clear image. Thus, the existing methods still have room for improvement in
finding the correct mapping.

In this work, we focus on developing an end-to-end lightweight deep-learning
solution for the image restoration task. Our major contributions are listed
below:

  * •

Taking inspiration from Non-linear Activation Free Network (NAFNet) [5] and
Level Attention Module [45], we propose a novel algorithm - Low-Visibility
Restoration Network (LVRNet), that can effectively recover high-quality images
from degraded images taken in poor visual conditions (Figure 1).

  * •

Due to the lack of available datasets that exhibit a combination of adverse
effects, we generate a new dataset, namely LowVis-AFO (abbreviation for Low-
Visibility Aerial Floating Objects dataset). We use AFO [15] as our ground
truth dataset and synthesize dark hazy images. The data generation process has
been elaborated in Section 4.1.

  * •

Benchmarking experiments have been provided on the LowVis-AFO dataset to help
future researchers for quantitative comparison. Along with that, LVRNet
surpasses the results obtained using previous image restoration techniques by
a significant margin.

  * •

We perform extensive ablation studies to analyze the importance of various
loss functions existing in current image restoration research. These
experiments are discussed in detail in Section 5.

Figure 2: Model architecture of the proposed LVRNet. Starting from the top-
left: The input image is passed to the pre-processing convolution layers where
feature maps are learned and passed to NAF Groups (here we have used 3
groups). The features extracted from each group are concatenated (or stacked)
along the channel dimension and sent as input to the Level Attention Module
(LAM). Finally, we pass LAM’s output to CNN layers for post-processing, adding
the original image through residual connection and extracting the restored
image at the bottom-left.

##  2 Related Works

This section highlights the previous work done in the fields of image dehazing
and low-light image enhancement and their limitations.

###  2.1 Image Dehazing

Hazy weather is often seen due to floating particles in the environment which
degrade the quality of the image captured. Therefore, many previous works have
tried to recover a clear image from the hazy one. These works can be divided
into two methods, ones that rely on prior assumptions [17] and the atmospheric
scattering model (ASM) [31] and the others which use deep learning to solve
the problem, either by combination with ASM [3][34][36] or independently
[25][26][33][46][50]. Conventional approaches are physically inspired and
apply various types of sharp image priors to regularize the solution space.
However, they exhibit shortcomings when implemented with real-world images and
videos. For example, the dark channel prior method (DCP) [31] does not perform
well in regions containing the sky. These methods [1][11][24] are known to be
computationally expensive and require heuristic parameter-tuning. Supervised
dehazing methods can be divided into two subparts, one is ASM based, and the
other is non-ASM based.

ASM-based Learning: MSCNN[34] solves the task of image dehazing by dividing
the problem into three steps: using CNN to estimate the transmission map
$t(x)$, using statistical methods to find atmospheric light $A$ and then
recover the clear image $J(x)$ using $t(x)$ and $A$ jointly. Methods like LAP-
Net [23] adopt the relation of depth with the amount of haze in the image. The
farther the scene from the camera, the denser the haze would be. Hence it
considers the difference in the haze density in the input image using a stage-
wise loss, where each stage predicts the transmission map from mild to severe
haze scenes. DehazeNet [3] consists of four sequential operations: feature
extraction, multi-scale mapping, calculating local extremum, and non-linear
regression. MSRL-DehazeNet [43] decomposes the problem into recovering high-
frequency and basic components. GCANet [4] employs residual learning between
haze-free and hazy images as an optimization objective.

End-to-end Learning: This subpart of previous work corresponds to non-ASM-
based deep learning methods for recovering the clear image. Back-Projected
Pyramid Network (BPPNet) [39] is a generative adversarial network that
includes iterative blocks of UNets [37] to learn haze features and pyramid
convolution to preserve spatial features of different scales. The reason
behind using iterative blocks of UNets[37] is to avoid increasing the number
of encoder layers in a single UNet[37] as it leads to a decrease in height and
width of latent feature representation hence resulting in loss of spatial
information. Moreover, different blocks of UNet learn different complexities
of haze features, and the final concatenation step ensures that all of them
are taken into account during image reconstruction. The final reconstruction
is done using the pyramid convolution block. The output feature is post-
processed to get a haze-free image. Feature-Fusion Attention Network (FFANet)
[33] adopts the idea of an attention mechanism and skip connections to restore
haze-free images. A combination of channel attention and pixel attention is
introduced, which helps the network, deal with the uneven spatial distribution
of haze and different weighted information across channels. Autoencoders [6],
hierarchical networks [9], and dense block networks [14] has also been
proposed for the task of image dehazing. However, our main comparison lies
with FFANet [33], wherein we show a huge improvement compared to the former
method with a model containing a lesser number of parameters and which can
generalize to different levels of haze.

Figure 3: Architecture of NAF Block and NAF Group. NAF Blocks are the building
blocks of NAF Groups. A detailed description has been provided in Section 3.1
and Section 3.1.1

###  2.2 Low-light Enhancement

Traditional methods for low-light image enhancement (LLIE) include Histogram
Equalization-based methods and Retinex model-based methods. Recent research
has been focused on developing deep learning-based methods following the
success of the first seminal work. Deep learning-based solutions are more
accurate, robust, and have a shorter inference time thus attracting more
researchers. Learning strategies used in these methods are mainly supervised
learning [27, 29, 30, 35, 51, 28, 41], unsupervised learning [20], and zero-
shot learning [49, 13].
Supervised Learning: The first deep learning-based LLIE method LLNet [27] is
an end-to-end network that employs a variant of stacked-sparse denoising
autoencoder to brighten and denoise low-light images simultaneously. LLNet
inspired many other works [29, 30, 35, 51], but they do not consider the
observation that noise exhibits different levels of contrast in different
frequency layers. Later, Xu et al. [41] proposed a network that suppresses
noise in the low-frequency layers and recovers the image contents by inferring
the details in high-frequency layers. There is another division of methods
that is based on the Retinex theory. Deep Retinex-based models [40, 42]
decomposes the image into two separate components - light-independent
reflectance and structure-aware smooth illumination. The final estimated
reflection component is treated as the enhanced result.
Unsupervised Learning: Although the above-mentioned methods perform well on
synthetic data, they show limited generalization capability on real-world low-
light images. This might be the result of overfitting. EnlightenGAN [20]
proposed to solve this issue by adopting an unsupervised learning technique,
i.e., avoiding the use of paired synthetic data. This work uses attention-
guided UNet as a generator and global-local discriminators to achieve the
objective of LLIE.
Zero-short Learning: These methods, in low-level vision tasks, do not require
any paired or unpaired training data. Zero-reference Deep Curve Estimation
[13] formulates image enhancement as a task of image-specific deep curve
estimation, taking into account pixel value range, monotonicity, and
differentiability. It is a lightweight DCE-Net that doesn’t require paired or
unpaired ground truth images during training and relies on non-reference loss
functions that measure the enhancement quality hence driving the learning of
the network. Another such method, Semantic-guided Zero-shot low-light
enhancement Network [49] is a lightweight model for low-light enhancement
factor extraction which is inspired by the architecture of U-Net [37]. The
output of this network is fed to a recurrent image enhancement network, along
with the degraded input image. Each stage in this network considers the
enhancement factor and the output from the previous scale as its input. This
is followed by a feature-pyramid network that aims to preserve the semantic
information in the image.

More recently, researchers have experimented with transformers for Zero-shot
Learning LLIE. Structure-Aware lightweight Transformer (STAR) [47] focuses on
real-time image enhancement without using deep-stacked CNNs or large
transformer models. STAR is formulated to capture long-range dependencies
between separate image patches, facilitating the model to learn structural
relationships between different regions of the images. In STAR, patches of the
image are tokenized into token embeddings. The tokens generated as an
intermediate stage are passed to a long-short-range transformer that outputs
two long and short-range structural maps. These structural maps can further
predict curve estimation or transformation for image enhancement tasks.
Although these methods show impressive results for the study of low-light
image enhancement for which it originally developed, they cannot deal with
foggy low– light images.

###  2.3 Limitations

Previous works have relied on ASM-based methods in the case of dehazing and
Retinex model-based methods for low-light image enhancement. However, these
methods fail to generalize to real-world images. Recent deep learning-based
methods using large networks solve the task of image dehazing and low-light
enhancement separately. To our knowledge, no work is introduced that solves
the two problems in a collaborative network. Deep learning methods also fail
to generalize to different haze levels and darkness.

##  3 Proposed Methodology

In this section, we provide a detailed description of the overall architecture
proposed and the individual components included in the network.

###  3.1 Architecture

Like the group structure in [33], each group in our network consists of a $K$
NAF Block [5] with a skip connection at the end as shown in Figure 3. The
output of each group is concatenated, passed to the level attention module to
find the weighted importance of the feature maps obtained, and post-processed
using two convolutional layers. A long skip connection for global residual
learning accompanies this.

####  3.1.1 NAF-Block

To keep this work self-contained, we explain the NAF Block [5] in this
subsection. NAF Block is the building block of Nonlinear Activation Free
Network. Namely NAFNet [5]. To avoid over-complexity in the architecture, this
block avoids using any activation functions like ReLU, GELU, Softmax, etc.
hence keeping a check on the intra-block complexity of the network.

The input first passes through Layer Normalization as it can help stabilize
the training process. This is followed by convolution operations and a Simple
Gate (SG). SG is a variant of Gated Linear Units (GLU) [10] as evident from
the following equations 1 and 2

$GLU(X,f,g,\sigma)=f(X)\odot\sigma(g(X))$ (1)

$SimpleGate(X,Y)=X\odot Y$ (2)

and a replacement for GELU[18] activation function because of the similarity
between GLU and GELU (Equation 3).

$GELU(x)=x\phi(x)$ (3)

In Simple Gate, the feature maps are divided into two parts along the channel
dimension and then multiplied as shown in Figure 4. Another novelty introduced
in this block is Simplified Channel Attention (SCA). Channel Attention (CA)
can be expressed as:

$CA(X)=X\otimes\sigma(W_{2}max(0,W_{1}pool(X)))$ (4)

where $X$ represents the feature map, $pool$ indicates the global average
pooling operation,$\sigma$ is Sigmoid, $W1,W2$ are fully-connected layers and
$\otimes$ is a channel-wise product operation. This can be taken as a special
case of GLU from which we can derivate the equation for Simplified Channel
Attention:

$SCA(X)=X\otimes Wpool(X)$ (5)

####  3.1.2 Level Attention Module

Once we have extracted features from all the NAF Groups, we concatenate them
and pass them through the Level Attention Module (LAM) [45]. This module
learns attention weights for features obtained at different levels.

In LAM, each feature map is first reshaped to a 2D matrix of the size $K\times
HWC$, where $K,H,W,$ and $C$ are the no. of NAF Groups, height, width, and no.
of channels of the feature maps respectively. We find a correlation matrix of
this 2D matrix by multiplying it with its transpose matrix. Finally, we
multiply the 2D matrix with this correlation matrix and reshape it to $K\times
H\times W\times C$ tensor. Inspired by residual learning, this tensor is
substituted for residual and is added to the original concatenated feature
maps. The resultant features are then reshaped to $H\times W\times KC$,
passing through $1\times 1$ convolution operation to get the $H\times W\times
C$ feature map. This is passed through some post-processing convolutions to
get the final enhanced output. We include its architecture diagram in the
supplementary material for a better understanding.

Figure 4: Simple Gate as represented by Equation 2 $\otimes$ denotes channel-
wise multiplicaWere

###  3.2 Loss Functions

Four loss functions, namely, reconstruction loss, perceptual loss, edge loss
[19], and FFT loss[7], have been used to supervise the task of image
restoration.

The total loss L is defined in Equation 6, where $\lambda_{1}=0.04$,
$\lambda_{2}=1$ and $\lambda_{3}=0.01$.

$L=L_{s}+\lambda_{1}L_{p}+\lambda_{2}L_{e}+\lambda_{3}L_{f}$ (6)

####  3.2.1 Reconstruction Loss:

The restored clear output image is compared with its ground truth value in the
spatial domain using a standard $l_{1}$ loss as demonstrated in Equation 7. We
use $l_{1}$ loss instead of $l_{2}$ loss as it does not over-penalize the
errors and leads to better image restoration performance [48].

$L_{s}=\frac{1}{N}\sum\limits_{i=1}^{n}\parallel
x_{i}^{gt}-NAFNet(x_{i}^{dark,hazy})\parallel_{1}$ (7)

In the above equation, $x_{i}^{gt}$ refers to the ground truth clear image,
and $NAFNet(x_{i}^{dark,hazy})$ denotes the output of our proposed NAFNet when
a dark and hazy image is fed to the network.

####  3.2.2 Perceptual Loss:

To reduce the perceptual loss and improve the image’s visual quality, we
utilize the features of the pre-trained VGG-19 network [38] obtained from the
output of one of the ReLU activation layers. It is defined in Equation 8,
where $w_{ij}$, $h_{ij}$, and $c_{ij}$ refer to the dimensions of the
respective feature maps inside the VGG-19 architecture. $\phi_{ij}$ denotes
the feature maps outputted from the jth convolutional layer inside the i-th
block in the VGG network.

$L_{p}=\frac{1}{w_{ij}h_{ij}c_{ij}}\sum\limits_{x=1}^{w_{ij}}\sum\limits_{y=1}^{h_{ij}}\sum\limits_{z=1}^{c_{ij}}\parallel\phi_{ij}(I_{gt})_{xyz}-\phi_{ij}(I_{out})_{xyz}\parallel$
(8)

####  3.2.3 Edge Loss:

To recover the high-frequency details lost because of the inherent noise in
dark and hazy images, we have an additional edge loss to constrain the high-
frequency components between the ground truth and the recovered image.

$L_{e}=\sqrt{(\nabla^{2}(I_{gt})-\nabla^{2}(I_{out}))^{2}+\epsilon^{2}}$ (9)

In Equation 9, $\nabla^{2}$ refers to the Laplacian operation [22], which is
then applied to the ground truth and the predicted clean image to get the edge
loss.

####  3.2.4 FFT Loss:

To supervise the haze-free results in the frequency domain, we add another
loss called Fast Fourier transform (FFT) loss (denoted by $L_{f}$ in Equation
12. It calculates the loss of both amplitude and phase using the $l_{1}$ loss
function without additional inference cost.

$A_{x_{i}^{gt}},P_{x_{i}^{gt}}=FFT(x_{i}^{gt}),$ (10)

$A_{x_{i}^{out}},P_{x_{i}^{out}}=FFT(x_{i}^{out}),$ (11)

$L_{f}=\frac{1}{N}\sum\limits_{i=1}^{n}(\parallel
A_{x_{i}^{gt}}-A_{x_{i}^{out}}\parallel_{1}+\parallel
P_{x_{i}^{gt}}-P_{x_{i}^{out}}\parallel_{1})$ (12)

Figure 5: Visual illustration of a few sample images from our dataset. Columns
1 and 3 show original images taken from AFO Dataset [15], whereas Columns 2
and 4 show their corresponding images generated as explained in Section 4.1
simulating low-visibility conditions.

##  4 Experimental Results

To demonstrate the outcomes of our model’s approach towards image enhancement
under low-visibility conditions, this section contains a detailed description
of the dataset generated and used in Section 4.1, the experimental settings in
Section 4.2, the metrics used for evaluation in Section 4.3 and a discussion
on the results obtained in Section 5.1 and 5.2.

###  4.1 Dataset Details

Due to the lack of available datasets that meet our requirements, we generate
a new one using the AFO dataset [15]. The dataset generation process has been
elaborated below, and the final images have been shown in Figure 5.

  * •

Haze effect \- To add fog, imgaug [21], a well-known python library was used.
A random integral value between 3, 4, 5 was selected, representing the fog’s
severity. For each image, this random number was chosen and pre-defined
functions within the package were utilized to add a layer of fog to the image.

  * •

Low-light Effect \- Given a normal image, our goal is to output a low-lit
image while preserving the underlying information. We follow the pipeline
introduced [8], which parametrically models the low light-degrading
transformation by observing the image signal processing (ISP) pipeline between
the sensor measurement system and the final image. The low-illumination-
degrading pipeline is a three-stage process:

    * –

Unprocessing procedure - This part aims to synthesize RAW format images from
input sRGB images by invert tone mapping, invert gamma correction, and the
transformation of the image from sRGB space to cRGB space, and invert white
balancing.

    * –

Low Light Corruption - This aims at adding shot and read noises to the output
of the unprocessing procedure, as these are common in-camera imaging systems.
Shot noise is a type of noise generated by the random arrival of photons in a
camera, which is a fundamental limitation. Read noise occurs during the charge
conversion of electrons into voltage in the output amplifier, which can be
approximated using a Gaussian random variable with zero mean and fixed
variance.

    * –

ISP Pipeline - RAW image processing is done after the lowlight corruption
process in the following order: add quantization noise, white balancing from
cRGB to sRGB, and gamma correction, which finally outputs a degraded low-light
image.

  * •

Combination of Haze and Low-light Effect \- Results of implementing the low-
light generation algorithm described above on foggy images generated using
img-aug are shown here. It can be seen that combining the two (fog and low
light) has introduced adversity in finding the location of the objects in the
water bodies. Moreover, finding a unique solution for such a combination has
not been explored to date

###  4.2 Experimental Settings

The images were resized to get the resultant dimensions as $256\times 456$.
Adam optimizer with an initial learning rate of
$1e\scalebox{0.75}[1.0]{$-$}4$, $\beta_{1}$, and $\beta_{2}$ with a value of
0.9 and 0.999 were chosen. The batch size was fixed as 2. We have used 3
groups in all our experiments, each with 16 blocks. Pytorch backend was used
to compile the model and train it.

###  4.3 Evaluation Metrics

We reported the results we obtained using two standard image restoration
metrics (i.e., PSNR and SSIM). These metrics will help us quantitatively
evaluate the performance of our model in terms of feature colors and structure
similarity. High PSNR and SSIM values if indicative of good results.

##  5 Experimental Results

The architecture used is given in Figure 2. This section gives a detailed
analysis of the results obtained by the proposed method.

Method | Year | PSNR | SSIM
---|---|---|---
Zero-DCE[13] | $2020$ | $12.323$ | $0.529$
SGZNet[49] | $2022$ | $12.578$ | $0.519$
BPPNet[39] | $2022$ | $15.507$ | $0.755$
DehazeNet[3] | $2016$ | $15.710$ | $0.391$
Star-DCE[47] | $2021$ | $16.651$ | $0.539$
FFANet[7] | $2020$ | $15.050$ | $0.582$
MSBDN-DFF[16] | $2020$ | $16.686$ | $0.689$
LVRNet (Ours) | $2022$ | $25.744$ | $0.905$
Table 1: Quantitative comparison of our proposed network with previous work. The best results and the second-best results have been highlighted with red color and blue colors, respectively. S.no. | Reconstruction Loss | Perceptual Loss | Edge Loss | FFT Loss | PSNR | SSIM
---|---|---|---|---|---|---
1. | ✓ | ✓ | ✗ | ✗ | 24.070 | 0.870
2. | ✓ | ✓ | ✗ | ✓ | 25.455 | 0.903
3. | ✓ | ✓ | ✓ | ✗ | 25.624 | 0.897
4. | ✓ | ✗ | ✓ | ✓ | 25.719 | 0.900
5. | ✓ | ✓ | ✓ | ✓ | 25.744 | 0.905
Table 2: Ablation experiments: We train our model using different combinations
of loss functions to understand the importance of individual losses for image
restoration. The best results are obtained when the model is trained using all
the loss functions mentioned in this work.

###  5.1 Discussion and Comparison

In this subsection, we discuss the evaluation results obtained by the proposed
pipeline. Previous methods were trained on the newly generated dataset and
tested to compare their metrics with our model’s performance. These methods
were built to enhance the low-light image or obtain a clear image from a hazy
one. The results are mentioned in Table 1.

We observe a huge increase in the PSNR value as compared to Zero-DCE[13],
which enhances the low-light image as a curve estimation problem. However, it
introduces an even amplified noise leading to color degradation as seen in
Figure 1. Notwithstanding its fast processing speed, Zero-DCE has limited
noise suppression and haze removal capacity. Star-DCE[47], which uses a
transformer backbone instead of a CNN one in the Zero-DCE network, shows a
35.12% increase in PSNR value. Owing to the added LAM structure, using which
our model can focus on more important feature maps, we can achieve a 54%
higher PSNR value.

SGZNet[49] uses pretrained networks for enhancement factor estimation, thus
their result is dependent on those pretrained weights, leading to a lower PSNR
value of 12.578 on LowVis-AFO. From Figure 1, we observe that the result
obtained from SGZNet is still degraded by excessive noise and lacks
saturation. DehazeNet[3] is limited by the network’s depth and cannot
generalize to real-world scenarios. Hence, it results in a low PSNR of 15.710.
Methods like BPPNet[39] and FFANet[33] are end-to-end deep learning methods
for image dehazing. BPPNet[39] distorts the color distribution in the
recovered image as it cannot remove the dark regions, whereas FFA-Net[33]
produces image with a lower perceptual quality.

We propose an end-to-end deep learning pipeline (0.43M parameters) that can
perform image dehazing and low-light image enhancement with a significant
decrease in the number of parameters as compared to MSBDN-DFF [16] (31M
parameters) and FFA-Net[33] (4.45M parameters).

The supplementary material has provided a discussion on the number of
parameters of other models. We also trained the model for 10 epochs with fewer
NAF blocks to prove that we achieved better results than the lighter results,
not due to an increase in parameters but because of the self-sufficiency of
the added LAM module, non-linear activation networks, and residual
connections. The results of these experiments are reported in the
supplementary material.

###  5.2 Ablation Studies

To prove the importance of the perceptual loss, edge loss, and fft-loss, added
to supervise the training procedure, we conducted experiments excluding each
of them and reported the values of PSNR and SSIM in Table 2. We keep the
$l_{1}$ loss function constant in all experiments as it is critical in image
restoration tasks. We observe an increase in metric values in the lower rows
compared to row 1. As a result of more supervision in the unchanged
architecture, there is an increase in the quality of clear images obtained,
which are demonstrated in the supplementary material. There is also an
increase in PSNR value (which depends on per-pixel distance) in row 3, once we
train the model without perceptual loss. This is seen as perceptual loss
doesn’t compare individual pixel values but the high-level features obtained
from a pretrained network. In row 4, we get a lower PSNR value on excluding
edge loss compared to row 5, as we get lesser edge supervision. Overall, we
get the best performance when we include all the loss functions, as seen in
row 5.

##  6 Conclusion

In this work, we have presented Low-Visibility Restoration Network (LVRNet), a
new lightweight deep learning architecture for image restoration. We also
introduce a new dataset, LowVis-AFO, that includes a diverse combination of
synthetic darkness and haze. We also performed benchmarking experiments on our
generated dataset and surpassed the results obtained using the previous image
restoration network by a significant margin. Qualitative and quantitative
comparison with previous work has demonstrated the effectiveness of LVRNet. We
believe our work will motivate more research, focused on dealing with a
combination of adverse effects such as haze, rain, snowfall, etc. rather than
considering a single factor. In our future work, we plan to extend LVRNet for
image restoration tasks where more factors, that negatively impact the image
quality, are taken into account.

Supplementary Material

To make our submission self-contained and given the page limitation, this
supplementary material provides additional details. Section 1 gives an
overview of the number of parameters and PSNR obtained by different methods.
Section 2 contains visual results that highlight the significance of the loss
functions. Section 3 contains the ablation experiment with lesser blocks, and
Section 4 demonstrates the architecture diagram of the level attention module.

##  1 PSNR vs Parameters

Figure 6 presents the PSNR vs. Parameters plot that the previous methods and
our method achieved on the testing set of LowVis-AFO. Our model outperforms
the state-of-the-art image dehazing and low-light image enhancement methods by
a good margin while having a lesser number of parameters.

Figure 6: The PSNR vs Number of Parameters of recent image restoration methods on the newly proposed LowVis-AFO dataset. S.no. | #Blocks | PSNR | SSIM | #params | Runtime(s)
---|---|---|---|---|---
1. | 14 | 21.3432 | 0.8626 | 0.38M | 0.035
2. | 12 | 20.4302 | 0.8488 | 0.33M | 0.029
3. | 10 | 20.2965 | 0.8494 | 0.28M | 0.024
Table 3: Results of the experiments conducted on a lesser number of NAF
blocks. The training was done for 10 epochs and the metrics were obtained on
the test set thereafter. Figure 7: Visual illustration of operations performed
by Level Attention Module.

##  2 Ablation Experiment on Different Loss Functions

Figure 8 demonstrates the visual results obtained when we conducted
experiments excluding some loss functions. The motivation behind the
experiment is to highlight the importance of the extra loss functions
(perceptual loss, edge loss, fft-loss) added to supervise our pipeline. The
quantitative results are given in Table 2 in the main manuscript.

Figure 8: Qualitative results obtained from experiments conducted on different
loss functions. In the figure, L = L1 loss, P = Perceptual Loss, E = Edge loss
and F = FFT loss.

##  3 Ablation Experiment with Lesser Number of Blocks

To prove the self-sufficiency of the individual components included in our
architecture such as LAM, we conduct experiments with a lesser number of NAF
blocks [5] and reported the PSNR and SSIM obtained in Table 1. Seeing the
results, we can conclude that our model achieves better results, not because
of an increase in the number of parameters as compared to the lighter model,
but because of the entire pipeline adopted.

##  4 Level Attention Module

As mentioned in the main text, the diagram for LAM[45] has been provided here
in the supplementary material. (refer Figure 7)

## References

  * [1] Codruta O Ancuti, Cosmin Ancuti, Chris Hermans, and Philippe Bekaert.  A fast semi-inverse approach to detect and remove the haze from a single image.  In Asian Conference on Computer Vision, pages 501–514. Springer, 2010.
  * [2] Julian Besag, Jeremy York, and Annie Mollié.  Bayesian image restoration, with two applications in spatial statistics.  Annals of the institute of statistical mathematics, 43(1):1–20, 1991.
  * [3] Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, and Dacheng Tao.  Dehazenet: An end-to-end system for single image haze removal.  IEEE Transactions on Image Processing, 25(11):5187–5198, 2016.
  * [4] Dongdong Chen, Mingming He, Qingnan Fan, Jing Liao, Liheng Zhang, Dongdong Hou, Lu Yuan, and Gang Hua.  Gated context aggregation network for image dehazing and deraining.  In 2019 IEEE winter conference on applications of computer vision (WACV), pages 1375–1383. IEEE, 2019.
  * [5] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun.  Simple baselines for image restoration.  arXiv preprint arXiv:2204.04676, 2022.
  * [6] Rongsen Chen and Edmund M-K Lai.  Convolutional autoencoder for single image dehazing.  In ICIP, pages 4464–4468, 2019.
  * [7] Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko.  Rethinking coarse-to-fine approach in single image deblurring.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 4641–4650, 2021.
  * [8] Ziteng Cui, Guo-Jun Qi, Lin Gu, Shaodi You, Zenghui Zhang, and Tatsuya Harada.  Multitask aet with orthogonal tangent regularity for dark object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2553–2562, 2021.
  * [9] Sourya Dipta Das and Saikat Dutta.  Fast deep multi-patch hierarchical network for nonhomogeneous image dehazing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 482–483, 2020.
  * [10] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier.  Language modeling with gated convolutional networks.  In International conference on machine learning, pages 933–941. PMLR, 2017.
  * [11] Raanan Fattal.  Dehazing using color-lines.  ACM transactions on graphics (TOG), 34(1):1–14, 2014.
  * [12] Stuart Geman and Donald Geman.  Stochastic relaxation, gibbs distributions, and the bayesian restoration of images.  IEEE Transactions on pattern analysis and machine intelligence, (6):721–741, 1984.
  * [13] Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong.  Zero-reference deep curve estimation for low-light image enhancement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1780–1789, 2020.
  * [14] Tiantong Guo, Venkateswararao Cherukuri, and Vishal Monga.  Dense123’color enhancement dehazing network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [15] Jan Gąsienica-Józkowy, Mateusz Knapik, and Boguslaw Cyganek.  An ensemble deep learning method with optimized weights for drone-based water rescue and surveillance.  Integrated Computer-Aided Engineering, pages 1–15, 01 2021.
  * [16] Dong Hang, Pan Jinshan, Hu Zhe, Lei Xiang, Zhang Xinyi, Wang Fei, and Yang Ming-Hsuan.  Multi-scale boosted dehazing network with dense feature fusion.  In CVPR, 2020.
  * [17] Kaiming He, Jian Sun, and Xiaoou Tang.  Single image haze removal using dark channel prior.  IEEE transactions on pattern analysis and machine intelligence, 33(12):2341–2353, 2010.
  * [18] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [19] Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang.  Multi-scale progressive fusion network for single image deraining.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8346–8355, 2020.
  * [20] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang.  Enlightengan: Deep light enhancement without paired supervision.  IEEE Transactions on Image Processing, 30:2340–2349, 2021.
  * [21] Alexander B. Jung, Kentaro Wada, Jon Crall, Satoshi Tanaka, Jake Graving, Christoph Reinders, Sarthak Yadav, Joy Banerjee, Gábor Vecsei, Adam Kraft, Zheng Rui, Jirka Borovec, Christian Vallentin, Semen Zhydenko, Kilian Pfeiffer, Ben Cook, Ismael Fernández, François-Michel De Rainville, Chi-Hung Weng, Abner Ayala-Acevedo, Raphael Meudec, Matias Laporte, et al.  imgaug.  https://github.com/aleju/imgaug, 2020.  Online; accessed 01-Feb-2020.
  * [22] Behzad Kamgar-Parsi and Azriel Rosenfeld.  Optimally isotropic laplacian operator.  IEEE Transactions on Image Processing, 8(10):1467–1472, 1999.
  * [23] Yunan Li, Qiguang Miao, Wanli Ouyang, Zhenxin Ma, Huijuan Fang, Chao Dong, and Yining Quan.  Lap-net: Level-aware progressive network for image dehazing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3276–3285, 2019.
  * [24] Zhuwen Li, Ping Tan, Robby T Tan, Danping Zou, Steven Zhiying Zhou, and Loong-Fah Cheong.  Simultaneous video defogging and stereo reconstruction.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4988–4997, 2015.
  * [25] Xiao Liang, Runde Li, and Jinhui Tang.  Selective attention network for image dehazing and deraining.  In Proceedings of the ACM Multimedia Asia, pages 1–6. 2019.
  * [26] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen.  Griddehazenet: Attention-based multi-scale network for image dehazing.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 7314–7323, 2019.
  * [27] Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar.  Llnet: A deep autoencoder approach to natural low-light image enhancement.  Pattern Recognition, 61:650–662, 2017.
  * [28] Kun Lu and Lihong Zhang.  Tbefn: A two-branch exposure-fusion network for low-light image enhancement.  IEEE Transactions on Multimedia, 23:4093–4105, 2020.
  * [29] Feifan Lv, Bo Liu, and Feng Lu.  Fast enhancement for non-uniform illumination images using light-weight cnns.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 1450–1458, 2020.
  * [30] Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim.  Mbllen: Low-light image/video enhancement using cnns.  In BMVC, volume 220, page 4, 2018.
  * [31] Earl J McCartney.  Optics of the atmosphere: scattering by molecules and particles.  New York, 1976.
  * [32] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883–3891, 2017.
  * [33] Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia.  Ffa-net: Feature fusion attention network for single image dehazing.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11908–11915, 2020.
  * [34] Wenqi Ren, Si Liu, Hua Zhang, Jinshan Pan, Xiaochun Cao, and Ming-Hsuan Yang.  Single image dehazing via multi-scale convolutional neural networks.  In European conference on computer vision, pages 154–169. Springer, 2016.
  * [35] Wenqi Ren, Sifei Liu, Lin Ma, Qianqian Xu, Xiangyu Xu, Xiaochun Cao, Junping Du, and Ming-Hsuan Yang.  Low-light image enhancement via a deep hybrid network.  IEEE Transactions on Image Processing, 28(9):4364–4375, 2019.
  * [36] Wenqi Ren, Jinshan Pan, Hua Zhang, Xiaochun Cao, and Ming-Hsuan Yang.  Single image dehazing via multi-scale convolutional neural networks with holistic edges.  International Journal of Computer Vision, 128(1):240–259, 2020\.
  * [37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015\.
  * [38] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2014\.
  * [39] Ayush Singh, Ajay Bhave, and Dilip K. Prasad.  Single image dehazing for a variety of haze scenarios using back projected pyramid network, 2020.
  * [40] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu.  Deep retinex decomposition for low-light enhancement.  arXiv preprint arXiv:1808.04560, 2018.
  * [41] Ke Xu, Xin Yang, Baocai Yin, and Rynson WH Lau.  Learning to restore low-light images via decomposition-and-enhancement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2281–2290, 2020.
  * [42] Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu.  Sparse gradient regularized deep retinex network for robust low-light image enhancement.  IEEE Transactions on Image Processing, 30:2072–2086, 2021.
  * [43] Chia-Hung Yeh, Chih-Hsiang Huang, and Li-Wei Kang.  Multi-scale deep residual learning-based single image haze removal via image decomposition.  IEEE Transactions on Image Processing, 29:3153–3167, 2019.
  * [44] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE transactions on image processing, 26(7):3142–3155, 2017.
  * [45] Kaihao Zhang, Wenhan Luo, Boheng Chen, Wenqi Ren, Bjorn Stenger, Wei Liu, Hongdong Li, and Ming-Hsuan Yang.  Benchmarking deep deblurring algorithms: A large-scale multi-cause dataset and a new baseline model.  arXiv preprint arXiv:2112.00234, 2021.
  * [46] Xiaoqin Zhang, Jinxin Wang, Tao Wang, and Runhua Jiang.  Hierarchical feature fusion with mixed convolution attention for single image dehazing.  IEEE Transactions on Circuits and Systems for Video Technology, 32(2):510–522, 2021.
  * [47] Zhaoyang Zhang, Yitong Jiang, Jun Jiang, Xiaogang Wang, Ping Luo, and Jinwei Gu.  Star: A structure-aware lightweight transformer for real-time image enhancement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4106–4115, 2021.
  * [48] Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz.  Loss functions for image restoration with neural networks.  IEEE Transactions on computational imaging, 3(1):47–57, 2016.
  * [49] Shen Zheng and Gaurav Gupta.  Semantic-guided zero-shot learning for low-light image/video enhancement.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 581–590, 2022.
  * [50] Zhuoran Zheng, Wenqi Ren, Xiaochun Cao, Xiaobin Hu, Tao Wang, Fenglong Song, and Xiuyi Jia.  Ultra-high-definition image dehazing via multi-guided bilateral learning.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16180–16189. IEEE, 2021.
  * [51] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Eemefn: Low-light image enhancement via edge-enhanced multi-exposure fusion network.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13106–13113, 2020.