TransposedTabularDB Ttabular[1]

# Towards Improving Robustness Against Common Corruptions using Mixture of
Class Specific Experts

Shashank Kotyan Danilo Vasconcellos Vargas Laboratory of Intelligent Systems
Kyushu Univeristy, Fukuoka, Japan

###### Abstract

Neural networks have demonstrated significant accuracy across various domains,
yet their vulnerability to subtle input alterations remains a persistent
challenge. Conventional methods like data augmentation, while effective to
some extent, fall short in addressing unforeseen corruptions, limiting the
adaptability of neural networks in real-world scenarios. In response, this
paper introduces a novel paradigm known as the Mixture of Class-Specific
Expert Architecture. The approach involves disentangling feature learning for
individual classes, offering a nuanced enhancement in scalability and overall
performance. By training dedicated network segments for each class and
subsequently aggregating their outputs, the proposed architecture aims to
mitigate vulnerabilities associated with common neural network structures. The
study underscores the importance of comprehensive evaluation methodologies,
advocating for the incorporation of benchmarks like the common corruptions
benchmark. This inclusion provides nuanced insights into the vulnerabilities
of neural networks, especially concerning their generalization capabilities
and robustness to unforeseen distortions. The research aligns with the broader
objective of advancing the development of highly robust learning systems
capable of nuanced reasoning across diverse and challenging real-world
scenarios. Through this contribution, the paper aims to foster a deeper
understanding of neural network limitations and proposes a practical approach
to enhance their resilience in the face of evolving and unpredictable
conditions.

##  1 Introduction

We’ve leveraged the capabilities of neural networks to achieve high precision
across a variety of tasks applied in diverse fields such as natural language
processing and computer vision. Many of these tasks would be unfeasible
without the assistance of neural networks. Nevertheless, the existing neural
networks (Krizhevsky et al., 2012) lack the resilience observed in the human
visual system (Recht et al., 2018, 2019; Azulay & Weiss, 2019). Neural
networks tend to make mistakes when confronted with minor modifications in
input samples (Szegedy et al., 2014; Carlini & Wagner, 2016, 2017a), in
contrast to the robustness of the human vision system against slight
alterations in images (Dodge & Karam, 2017b). Humans effortlessly navigate
through various corruptions and distortions in images, and even abstract
changes in structure and style do not disrupt their visual perception.

Earlier research suggests enhancing training data by introducing various
distortions to improve overall resilience. However, obtaining training data
for rare conditions is challenging, and it’s impossible to foresee all
potential conditions that might occur in real-world situations. Recent
findings indicate that neural networks struggle to generalize to newly
introduced distortion types, even when trained on a diverse set of other
distortions, as shown by Geirhos et al. (2018b). In a broader context, neural
networks often face difficulty in extending their performance beyond the
domain or distribution of the training data. As a result, the generation of
adversarial examples, as demonstrated by Szegedy et al. (2014), becomes
possible.

Adversarial examples represent the most severe attempts to exploit the
vulnerabilities of neural networks, characterized by a minimal domain shift
that is imperceptible to humans yet sufficient to deceive the neural network.
Therefore, analyzing with adversarial examples involves considering the worst-
case scenario for a neural network rather than the more common average-case
scenarios encountered in real-world situations. To address the prevalent but
less extreme issue of noticeable image distortions, such as blurriness, noise,
or natural distortions like snow, one can utilize the analysis proposed by
Hendrycks & Dietterich (2019) through their suggested common corruptions
benchmarks. This allows for an understanding of how performance degrades in
situations closely resembling real-world conditions.

Research by Kotyan et al. (2022) suggests that current deep neural networks
generalise poorly over the unknown classes which is also related to their
robustness. This suggests that the network overfit on the trained classes such
that they generalise poorly on the unknown classes. Another research by
Mathieu et al. (2019) suggests that disentangling features in a representation
is crucial for understanding and interpreting the underlying factors of
variation in data. Thus, building on the past researches by Kotyan et al.
(2022); Mathieu et al. (2019) we propose to learn the disentagled features by
making a part of the network focus on a particular feature in order to better
understand the underlying variations and learn more robustly.

Contributions.

Mixture of Class-Specific Expert Architecture:

We suggest employing a combination of class-specific expert architectures to
enhance overall robustness by separating the feature learning process for
distinct classes within a dataset. Our approach involves training a segment of
the network exclusively focused on recognizing features specific to a single
class. Subsequently, we aggregate the outcomes from all individual class-
specific experts to determine the image classification. This disentanglement
of feature learning not only enhances performance but also improves
robustness, making it applicable to datasets with a large number of images.

##  2 Related Works

Challenges in the field of Adversarial Machine Learning. Adversarial images
undergo carefully crafted distortions aiming to confuse classifiers,
occasionally deceiving black-box classifiers Kurakin et al. (2017). Algorithms
have been developed to identify minimal additive distortions in RGB space that
can confuse classifiers (Carlini et al., 2018). Neural networks exhibit
peculiar behavior for nearly identical images, showing high confidence when
faced with textures and random noise, revealing vulnerabilities exploited by
adversarial attacks (Szegedy et al., 2014; Nguyen et al., 2015). The
feasibility of universal adversarial perturbations, capable of deceiving a
neural network across most samples, has been demonstrated (Moosavi-Dezfooli et
al., 2017). The introduction of patches into an image can lead to
misclassification by neural networks (Brown et al., 2018), and extreme attacks
have proven effective, causing misclassification with a single-pixel change
(Su et al., 2019; Kotyan & Vargas, 2022). To counter adversarial attacks,
Goodfellow et al. (2015) proposed the Fast Gradient Sign Method (FGSM),
perturbing a target sample to its gradient direction to increase its loss and
using the generated samples to train the model for improved robustness.
Subsequent works introduced iterative variants of the gradient attack with
enhanced adversarial learning frameworks (Madry et al., 2018; Moosavi-Dezfooli
et al., 2016; Kurakin et al., 2017; Carlini & Wagner, 2017b).

Challenges in evaluating natural corruptions, perturbations and distortions.
Various studies highlight the susceptibility of neural networks to common
corruptions, with Hosseini et al. (2017) disrupting Google’s Cloud Vision API
using impulse noise, and Dodge & Karam (2016) demonstrating the sensitivity of
Convolutional Neural Networks to blur and Gaussian noise in image recognition.
Another study by Dodge & Karam (2017b) employs Gaussian noise and blur to
underscore the superior robustness of human vision compared to neural
networks, even after specific fine-tuning for these distortions. Geirhos et
al. (2018b) find that neural network performance declines more rapidly than
human performance in recognizing corrupted images, and fine-tuning on specific
corruptions does not lead to generalization. Hendrycks & Dietterich (2019)
introduce a benchmark of $19$ common corruptions to assess model robustness,
and various approaches, such as preprocessing to eliminate corruptions or data
augmentation by including corrupted data in training, are explored to address
performance degradation. Techniques like matching softmax distributions or
using a mixture of corruption-specific experts, as suggested by Zheng et al.
(2016) and Dodge & Karam (2017a), aim to mitigate issues such as under-fitting
when fine-tuning on noisy images. Geirhos et al. (2018a) train recognition
models on a stylized version of the ImageNet dataset, reporting increased
robustness against different corruptions due to a stronger bias towards object
shape over textures. Additionally, Hendrycks & Dietterich (2019) present
methods like Histogram Equalization (de la Torre et al., 2005; Harvilla &
Stern, 2012), Multi-scale Networks (Ke et al., 2017; Huang et al., 2018),
Adversarial Logit Pairing (Kannan et al., 2018), Feature Aggregating (Xie et
al., 2017), and Larger Networks to enhance performance on their corruption
benchmark. These efforts collectively seek to improve neural network
resilience to a diverse set of common corruptions.

Challenges in disentanglement. Disentanglement in machine learning, refers to
achieving independence among features in learned representations. This is
acheived as features are disentangled each dimension of the representation
captures a specific and independent aspect of the data, promoting transparency
and interpretability (Bengio et al., 2013; Eastwood & Williams, 2018; Higgins
et al., 2018). This concept has roots in traditional methods like Independent
Component Analysis and extends to modern deep learning approaches enableing
the model to learn meaningful and separable representations of complex data
(Schmidhuber, 1992; Yang & Amari, 1997; Hyvärinen & Oja, 2000; Reed et al.,
2014; Cheung et al., 2015; Chen et al., 2016; Makhzani et al., 2016; Mathieu
et al., 2016; Achille & Soatto, 2018; Hjelm et al., 2019; Mathieu et al.,
2019). The primary goal is to create interpretable representations by
capturing meaningful and independent dimensions in the data. Disentanglement
is crucial for identifying true generative factors within the data, enhancing
transparency, and facilitating tasks like classification and generation
(Higgins et al., 2016; N et al., 2017; Alemi et al., 2018; Chen et al., 2018;
Kim & Mnih, 2018; Esmaeili et al., 2019). Challenges arise in capturing the
true generative factors for complex datasets, necessitating richly structured
dependencies between latent dimensions (Eastwood & Williams, 2018). Despite
challenges, the development of decomposition frameworks aims to overcome
limitations in existing disentanglement approaches, highlighting the need for
structured dependencies in more complex datasets (Johnson et al., 2016; N et
al., 2017; Bouchacourt et al., 2018; Esmaeili et al., 2019).

##  3 Formal Mathematical Formulations

Let us suppose that for the image classification problem, we consider a
classifier $f_{\theta}:x\rightarrow y$ trained on samples from distribution
$D$ such that $x\in\mathbb{R}^{m\times n\times c}$ be the image which is to be
classified. Here $m,n$ is the width and the height of the image, and $c$ is
the number of colour channels.

A neural network is composed of several neural layers linked together. Each
neural layer is composed of a set of perceptrons (artificial neurons). Each
perceptron maps a set of inputs to output values with an activation function.
Thus, function of the neural network (formed by a chain) can be defined as:

$\displaystyle f_{\Theta}(x)=g_{\theta_{k}}^{(k)}(\ldots
g_{\theta_{2}}^{(2)}(g_{\theta_{1}}^{(1)}(x)))$

where $g_{\theta_{i}}^{(i)}$ is the function of the $i^{\text{th}}$ layer of
the network, and $i=1,2,3\ldots k$ such that $k$ is the last layer of the
neural network. $\theta_{i}$ is the parameter of the $i^{\text{th}}$ layer
which is optimised and consequently
$\Theta=\\{\theta_{1},\theta_{2},\theta_{3}\ldots\theta_{k}\\}$ is set of all
parameters of the neural network $f$ which is optimised. In the image
classification problem, $f(x)\in\mathbb{R}^{N}$ is the probabilities
(confidence) for all the available $N$ classes. Most classifiers are judged by
their performance (often measured in accuracy) on test queries drawn from
$\mathcal{D}$, i.e.,

$\displaystyle\mathbb{P}_{(x,y)\sim\mathcal{D}}~{}~{}\underset{y}{\operatorname*{arg\,max}}$
$\displaystyle f(x)=y~{}~{}.$

###  3.1 Adversarial Perturbations

Let us define adversarial samples $\hat{x}$ as:

$\displaystyle\hat{x}=x+\epsilon_{x}$
$\displaystyle\\{\hat{x}\in\mathbb{R}^{m\times n\times
3}\mid\underset{y}{\operatorname*{arg\,max}}~{}~{}f(x)\neq\underset{\hat{y}}{\operatorname*{arg\,max}}~{}~{}f(\hat{x})\\}$

in which $\epsilon_{x}\in\mathbb{R}^{m\times n\times c}$ is the perturbation
added to the input. Here, $y$ and $\hat{y}$ are the respective classfications
for $x$ and $\hat{x}$.

Making use of the definition of adversarial samples, adversarial robustness
can be formally defined as the following optimization problem for untargeted
black-box attacks:

$\displaystyle\underset{\epsilon_{x}}{\text{minimize}}$ $\displaystyle
f(\hat{x})_{y}=f(x+\epsilon_{x})_{y}$ subject to
$\displaystyle\|\epsilon_{x}\|_{p}\leq\delta$

Similarly optimization problem for the targeted black-box attacks can be
defined as:

$\displaystyle\underset{\epsilon_{x}}{\text{maximize}}$ $\displaystyle
f(\hat{x})_{\tilde{y}}=f(x+\epsilon_{x})_{\tilde{y}}$ subject to
$\displaystyle\|\epsilon_{x}\|_{p}\leq\delta$

where $f()_{c}$ is the soft-label for the class $c$, and $y$ is the true class
of sample $x$, whereas $\tilde{y}$ is target class for the the sample $x$. $p$
is the constraint on $\epsilon_{x}$ and $\delta$ is the threshold value for
the constraint. Thus, adversarial robustness can be formulated as,

$\displaystyle\underset{\|\epsilon_{x}\|_{p}~{}\leq~{}\delta}{\text{minimize}}$
$\displaystyle{\mathbb{P}}_{(x,y)\sim\mathcal{D}}~{}~{}\underset{\hat{y}}{\operatorname*{arg\,max}}$
$\displaystyle f(\hat{x})=y$

Given such a dataset $\mathcal{D}$ and a model $f$, adversarial attacks aim
towards finding the worst-case examples nearby by searching for the
perturbation $\epsilon$, which maximizes the loss. We can define one such
adversarial attack which tries to find perturbation within a certain radius
from the sample (e.g., norm balls) as follows:

$\displaystyle\hat{x}^{i+1}=\Pi_{\|\epsilon_{x}\|_{p}~{}\leq~{}\delta}(\hat{x}^{i}+\alpha\cdot\text{sign}(\nabla_{\hat{x}^{i}}\mathcal{L}_{\text{CE},\theta}(f(\hat{x}^{i}),y)))$

where $\|\epsilon_{x}\|_{p}~{}\leq~{}th$ is the norm-ball around $x$ with
radius $th$, and $\Pi$ is the projection function for norm-ball. The $\alpha$
is the step-size of the attacks whereas sign(·) returns the sign of the vector
and $\nabla_{p}(q)$ is the gradient of $p$ w.r.t $q$. Further,
$\mathcal{L}_{\text{CE}}$ is the cross-entropy loss for supervised training,
and $i$ is the number of attack iterations. This formulation generalizes
across different types of gradient attacks. For example, Projected Gradient
Descent (PGD) (Madry et al., 2018) starts from a random point within the $x\pm
th$ and perform $i$ gradient steps, to obtain the final adversarial sample
$\hat{x}$.

###  3.2 Common Corruptions

We now consider a set of corruption functions $C$ such that
$\mathbb{P}_{C}(c)$ approximate the real-world frequency of these corruptions.
We can now define, a classifier’s robustness against corruptions as,

$\displaystyle\mathbb{E}_{c\sim
C}[{\mathbb{P}}_{(x,y)\sim\mathcal{D}}~{}~{}\underset{y}{\operatorname*{arg\,max}}$
$\displaystyle f(c(x))=y~{}~{}].$

Thus, corruption robustness measures the classifier’s average-case performance
on classifier-agnostic corruptions $C$, while adversarial robustness measures
the worst-case performance on small, additive, classifier-tailored
perturbations.

###  3.3 Adversarial Training

The simplest and most straightforward way to defend against such adversarial
attacks is to minimize the loss of adversarial examples, which is often called
adversarial learning. The adversarial learning framework proposed by Madry et
al. (2018) does solve the following non-convex outer minimization problem and
non-convex inner maximization problem, as follows:

$\displaystyle\underset{\theta}{\operatorname*{arg\,min}}$
$\displaystyle\mathbb{E}_{(x,y)\sim
D}[~{}~{}\underset{\|\epsilon_{x}\|_{p}~{}\leq~{}th}{\text{maximize}}$
$\displaystyle\mathcal{L}_{\text{CE},\theta}(f(\hat{x}^{i}),y)~{}~{}].$

There are various adversarial learning frameworks, including PGD (Madry et
al., 2018), and TRADES Zhang et al. (2019) depending on how adversarial sample
is optimized and how the classifier is optimized.

###  3.4 Mixture of Expert Network

A Mixture of Experts network consists of a set of $M$ ‘expert networks’,
$f_{1},f_{2},\ldots,f_{M}$ and a gating network which is generally set to be
linear (Shazeer et al., 2017; Fedus et al., 2022). The output of $m$-th expert
network with input $x$ and parameter $W$ can be written as $f_{m}(x;W)$.
Similarly, the output of the gating network parameterised by
$\Theta=[\theta_{1},\ldots,\theta_{m}]\in\mathbb{R}^{d\times M}$ can be
written as $h(x;\Theta)=\sum\Theta^{T}x$. Thus, the output $F$ of the Mixture
of Experts network can be written as follows,

$\displaystyle F(x;\Theta,W)=\sum_{m\in T_{x}}\pi_{m}(x;\Theta)f_{m}(x;W)$

where $T_{x}\subseteq[M]$ is a set of selected indices of experts and
$\pi_{m}(x;\Theta)$ is route gate value given by

$\displaystyle\pi_{m}(x;\Theta)=\frac{\exp(h_{m}(x;\Theta))}{\sum_{m^{\prime}=1}^{M}\exp(h_{m^{\prime}}(x;\Theta))},\quad\forall~{}~{}m\in[M]$

###  3.5 Mixture of Class-Specific Expert Network

We define, a Mixture of Class-Specific Experts network consisting of a set of
$N$ ‘expert networks’, $f_{1},f_{2},\ldots,f_{N}$ and an aggregation network
$F$. The output of $n$-th expert network with input $x$ and parameter
$\theta_{n}$ can be written as $f_{\theta_{n}}(x)$. In our image
classification problem, $f_{\theta_{n}}(x)\in\mathbb{R}^{2}$ is the
probabilities (confidence) for either where the given $x$ belongs to $n$-th
class or not. Similarly, the output of aggregation network $F$ can be written
as,

$\displaystyle F(x)$
$\displaystyle=\\{f_{\theta_{n}}(x)_{\text{True}}:\forall~{}~{}n\in[N]\\}$
$\displaystyle=\begin{pmatrix}f_{\theta_{1}}(x)_{\text{True}}\\\
f_{\theta_{2}}(x)_{\text{True}}\\\ \vdots\\\
f_{\theta_{N}}(x)_{\text{True}}\end{pmatrix}$

where, $f_{\theta_{n}}(x)_{\text{True}}$ is the true probability (confidence)
of $n$-th expert that the sample $x$ belongs to $n$-th class.

The set of all parameters needed to be optimised can be written as
$\Theta={\theta_{1},\ldots,\theta_{n}}$ Thus, the performance of Mixture of
Class-Specific Expert Network on test queries drawn from data distribution
$\mathcal{D}$ is,

$\displaystyle\mathbb{P}_{(x,y)\sim\mathcal{D}}~{}~{}\underset{y}{\operatorname*{arg\,max}}$
$\displaystyle F(x)=y~{}~{}.$

##  4 Methodology

###  4.1 Common Corruptions

The Common Corruptions benchmark (Hendrycks & Dietterich, 2019) introduced
corrupted versions of commonly used classification datasets (ImageNet-C,
CIFAR10-C) as standardized benchmarks which consists of $19$ diverse
corruption types categoried into noise, blur, weather, and digital corruptions
and each corruption type has five levels of severity. This is because in real-
world these corruptions can manifest themselves at varying intensities. Since,
the real-world corruptions also have variation even at a fixed intensity, to
simulate these, the benchmark also introduces variation for each corruption
when possible. We briefly describe each of the $19$ corruption in the
benchmark below,

Gaussian Noise,

this corruption can appear in low-lighting conditions.

Shot Noise,

also called Poisson noise, is electronic noise caused by the discrete nature
of light itself.

Impulse Noise,

is a color analogue of salt-and-pepper noise and can be caused by bit errors.

Speckle Noise,

an additive noise where the noise added to a pixel tends to be larger if the
original pixel intensity is larger.

Defocus Blur,

occurs when an image is out of focus.

Frosted Glass Blur,

appears with “frosted glass” windows or panels.

Motion Blur,

appears when a camera is moving quickly.

Zoom Blur,

occurs when a camera moves toward an object rapidly.

Gaussian Blur,

is a low-pass filter where a blurred pixel is a result of a weighted average
of its neighbors, and farther pixels have decreasing weight in this average.

Snow,

is a visually obstructive form of precipitation.

Frost,

forms when lenses or windows are coated with ice crystals.

Fog,

shrouds objects and is rendered with the diamond-square algorithm.

Brightness,

varies with daylight intensity.

Spatter,

can occlude a lens in the form of rain or mud.

Contrast,

can be high or low depending on lighting conditions and the photographed
object’s color.

Elastic Transformations,

stretch or contract small image regions.

Pixelation,

occurs when upsampling a lowresolution image.

JPEG Compression,

is a lossy image compression format which introduces compression artifacts.

Saturate,

is common in edited images where images are made more or less colorful.

###  4.2 Evaluation Metric for Common Corruptions

Common corruptions can be benign or destructive depending on their severity.
In order to comprehensively evaluate a classifier’s robustness to a given type
of corruption, we score the classifier’s performance across five corruption
severity levels and aggregate these scores. The first evaluation step is to
take a trained classifier $f$, and compute the performance on clean dataset
($P^{f}_{\text{clean}}$). The second step is to test the classifier on each
corruption type $c$ at each level of severity $s$ ($P^{f}_{c,s}$).

We then aggregate the classifier’s performance for each of the $19$ corruption
type $c$ as,

$\displaystyle\overline{P}^{f}_{c}=1/5\times\sum_{s=1}^{5}E_{fs,c}$

Finally, we aggregate the classifier’s average performance on the common
corruptions as,

$\displaystyle\overline{P}^{f}=1/19\times\sum_{c}E_{fs,c}$

###  4.3 Mixture of Class-Specific Experts

Figure 1:  Illustration of the proposed Mixture of Class-Specific Expert
Architecture.

Figure 1 provide an illustration for the proposed Mixture of Class Specific
Expert Architecture. We first create $N$ Class-Specific Expert Networks whose
task will be to identify only if the given input sample is within their class
and respond in binary logit representing whether the input sample belongs to
the class or not. We then create a simple non-parameterised aggregator network
which takes the true soft-label from the Class-Specific Expert Networks and
forms the final logit space of the entire model. We then computed the cross-
entropy loss $\mathcal{L}_{\text{CE}}$ using the aggregated logits, optimizing
the parameters for all the class specific networks. We also compute the binary
cross-entropy loss $\mathcal{L}_{\text{BCE}}$ of individual Class-Specific
Expert Network to optimise their own parameter.

##  5 Results

Table 1: Performance of Pre-activated ResNet-18 with different training
strategies and dataset for image classification task. Performance is measured
in Accuracy.

Dataset | CIFAR-10 | CIFAR-10 Augmented
---|---|---
Training | Standard | Adversarial | Standard
Architecture | Standard | MoCSE | Standard | MoCSE | Standard | MoCSE
Natural Samples | 95.18% | 94.08% | 83.35% | 85.09% | 96.13% | 96.54%
Noise | Gaussian Noise | 17.63% | 26.37% | 52.96% | 59.83% | 16.69% | 25.04%
Shot Noise | 17.66% | 25.66% | 50.51% | 56.81% | 18.03% | 24.67%
Impulse Noise | 25.53% | 33.17% | 52.06% | 57.79% | 28.97% | 37.48%
Speckle Noise | 25.43% | 35.77% | 60.12% | 64.73% | 27.98% | 34.75%
Blur | Defocus Blur | 16.68% | 17.87% | 30.33% | 29.85% | 14.20% | 15.05%
Glass Blur | 20.18% | 23.71% | 39.64% | 38.56% | 17.57% | 22.81%
Motion Blur | 30.69% | 31.87% | 38.40% | 38.29% | 30.73% | 32.94%
Zoom Blur | 71.48% | 77.18% | 75.75% | 76.61% | 76.13% | 80.55%
Gaussian Blur | 19.74% | 24.12% | 36.33% | 35.99% | 17.74% | 21.14%
Weather | Snow | 70.35% | 67.22% | 51.37% | 53.07% | 73.43% | 75.03%
Frost | 74.40% | 67.60% | 40.73% | 42.21% | 77.22% | 79.15%
Fog | 43.51% | 43.10% | 17.41% | 17.67% | 40.32% | 44.43%
Brightness | 88.94% | 85.54% | 64.11% | 66.13% | 91.36% | 92.03%
Spatter | 82.67% | 82.90% | 72.28% | 75.55% | 85.29% | 87.45%
Digital | Contrast | 52.08% | 41.88% | 22.14% | 21.90% | 52.43% | 49.78%
Elastic Transform | 19.59% | 19.70% | 40.98% | 40.95% | 21.80% | 24.03%
Pixelate | 28.10% | 36.73% | 66.55% | 67.33% | 30.96% | 30.63%
JPEG Compression | 52.00% | 55.87% | 76.08% | 77.70% | 46.77% | 51.02%
Saturate | 86.08% | 83.00% | 74.29% | 75.78% | 89.22% | 90.52%
Average of 19 Corruptions | 44.35% | 46.28% | 50.63% | 52.46% | 45.10% | 48.34%

Datasets. CIFAR-10 and CIFAR-10-Augmented

Models. Standard Preact-ResNet-18 and Mixture of Class-Specific Experts
Architecture

Training Strategies. Standard Training and Adversarial Training

Results. Table 1 shows the result of image classifier, Preact-ResNet-18 in
native form and in Mixture of Class-Specific Experts (MoCSE) form trained
using standard procedure and using adversarial training recipe. From the
results, we can see that CIFAR-10-Augmented Dataset offers a boost in
performance of ResNet by around $1\%$ for non-corrupted images and improve
robustness of ViT by around $1\%$ for common corruptions. We can see also see
that Adversarial Training on the standard CIFAR-10 Dataset degrades the
performance of the ResNet on non-corrupted images but improves robustness of
ResNet in general by $8\%$ for common corruptions. This results shows that
Adversarial Training benefits in improving the robustness of the model, when
the model capacity of the model is lower and a small dataset is used. Further,
augmented dataset help pushing the performance of the model even when the
model is trained to the capacity suggesting that existing neural networks
based on Convolution Layers can still benefit from Augmented Dataset.

When our Mixture of Class-Specific Experts Architecture is applied on ResNet,
we notice a general improvement in performance and robustness of ResNet,
across all trainings such as Standard Training with CIFAR-10 Dataset, Standard
Training with CIFAR-10 Augmented Dataset, and Adversarial Training with
CIFAR-10 Dataset. We improve the robustness of the ResNet trained with
Standard Recipe using CIFAR-10 Dataset from $44.35\%$ to $46.28\%$. We also
improve the robustness of ResNet trained with Standard Recipe using CIFAR-10
Augmented Dataset from $45.10\%$ to $48.34\%$. Further, we notice an
improvement of best robustness of the ResNet trained with Adversarial Training
Recipe using CIFAR-10 Dataset from $50.63\%$ to $52.46\%$. This results show
that Mixture of Class-Specific Experts Architecture enables better robustness
of the model by disentangling the features across classes to learn such
features which can better identify a sample belonging to a particular class.

##  6 Conclusion

In conclusion, the study acknowledges the inherent challenges in achieving
neural network robustness to diverse image corruptions. The proposed Mixture
of Class-Specific Expert Architecture introduces a novel paradigm for
enhancing robustness by disentangling feature learning, leading to improved
performance across varied datasets. As the pursuit of resilient models
continues, the need for comprehensive evaluation methodologies, such as the
common corruptions benchmark, becomes evident. By addressing the limitations
of current approaches and advancing our understanding of neural network
vulnerabilities, this research aims to contribute to the development of more
robust learning systems capable of adept reasoning in the face of real-world
distortions.

## References

  * Achille & Soatto (2018) Alessandro Achille and Stefano Soatto.  Emergence of Invariance and Disentanglement in Deep Representations.  _Journal of Machine Learning Research_ , 19(50):1–34, 2018.  ISSN 1533-7928.
  * Alemi et al. (2018) Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy.  Fixing a Broken ELBO.  In _Proceedings of the 35th International Conference on Machine Learning_ , pp. 159–168. PMLR, July 2018.
  * Azulay & Weiss (2019) Aharon Azulay and Yair Weiss.  Why do deep convolutional networks generalize so poorly to small image transformations?  _Journal of Machine Learning Research_ , 20(184):1–25, 2019.  ISSN 1533-7928.
  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation Learning: A Review and New Perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35(8):1798–1828, August 2013.  ISSN 1939-3539.  doi: 10.1109/TPAMI.2013.50.
  * Bouchacourt et al. (2018) Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin.  Multi-Level Variational Autoencoder: Learning Disentangled Representations From Grouped Observations.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), April 2018.  ISSN 2374-3468.  doi: 10.1609/aaai.v32i1.11867.
  * Brown et al. (2018) Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer.  Adversarial Patch, May 2018.
  * Carlini & Wagner (2016) Nicholas Carlini and David Wagner.  Defensive Distillation is Not Robust to Adversarial Examples, July 2016.
  * Carlini & Wagner (2017a) Nicholas Carlini and David Wagner.  Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , AISec ’17, pp. 3–14, New York, NY, USA, November 2017a. Association for Computing Machinery.  ISBN 978-1-4503-5202-4.  doi: 10.1145/3128572.3140444.
  * Carlini & Wagner (2017b) Nicholas Carlini and David Wagner.  Towards Evaluating the Robustness of Neural Networks.  In _2017 IEEE Symposium on Security and Privacy (SP)_ , pp. 39–57, May 2017b.  doi: 10.1109/SP.2017.49.
  * Carlini et al. (2018) Nicholas Carlini, Guy Katz, Clark Barrett, and David L. Dill.  Ground-Truth Adversarial Examples.  February 2018.
  * Chen et al. (2018) Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud.  Isolating Sources of Disentanglement in Variational Autoencoders.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Chen et al. (2016) Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel.  InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets.  In _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc., 2016.
  * Cheung et al. (2015) Brian Cheung, Jesse A. Livezey, Arjun K. Bansal, and Bruno A. Olshausen.  Discovering Hidden Factors of Variation in Deep Networks, June 2015.
  * de la Torre et al. (2005) A. de la Torre, A.M. Peinado, J.C. Segura, J.L. Perez-Cordoba, M.C. Benitez, and A.J. Rubio.  Histogram equalization of speech representation for robust speech recognition.  _IEEE Transactions on Speech and Audio Processing_ , 13(3):355–366, May 2005.  ISSN 1558-2353.  doi: 10.1109/TSA.2005.845805.
  * Dodge & Karam (2016) Samuel Dodge and Lina Karam.  Understanding how image quality affects deep neural networks.  In _2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX)_ , pp. 1–6, June 2016.  doi: 10.1109/QoMEX.2016.7498955.
  * Dodge & Karam (2017a) Samuel Dodge and Lina Karam.  Quality Resilient Deep Neural Networks, March 2017a.
  * Dodge & Karam (2017b) Samuel Dodge and Lina Karam.  A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions.  In _2017 26th International Conference on Computer Communication and Networks (ICCCN)_ , pp. 1–7, July 2017b.  doi: 10.1109/ICCCN.2017.8038465.
  * Eastwood & Williams (2018) Cian Eastwood and Christopher K. I. Williams.  A Framework for the Quantitative Evaluation of Disentangled Representations.  In _International Conference on Learning Representations_ , February 2018.
  * Esmaeili et al. (2019) Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N. Siddharth, Brooks Paige, Dana H. Brooks, Jennifer Dy, and Jan-Willem Meent.  Structured Disentangled Representations.  In _Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics_ , pp. 2525–2534. PMLR, April 2019.
  * Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _The Journal of Machine Learning Research_ , 23(1):120:5232–120:5270, January 2022.  ISSN 1532-4435.
  * Geirhos et al. (2018a) Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  In _International Conference on Learning Representations_ , September 2018a.
  * Geirhos et al. (2018b) Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann.  Generalisation in humans and deep neural networks.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018b.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and Harnessing Adversarial Examples, March 2015.
  * Harvilla & Stern (2012) Mark J. Harvilla and Richard M. Stern.  Histogram-based subband powerwarping and spectral averaging for robust speech recognition under matched and multistyle training.  In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4697–4700, March 2012\.  doi: 10.1109/ICASSP.2012.6288967.
  * Hendrycks & Dietterich (2019) Dan Hendrycks and Thomas Dietterich.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations, March 2019.
  * Higgins et al. (2016) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  Beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.  In _International Conference on Learning Representations_ , November 2016.
  * Higgins et al. (2018) Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner.  Towards a Definition of Disentangled Representations, December 2018.
  * Hjelm et al. (2019) R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization, February 2019.
  * Hosseini et al. (2017) Hossein Hosseini, Baicen Xiao, and Radha Poovendran.  Google’s Cloud Vision API Is Not Robust To Noise, July 2017.
  * Huang et al. (2018) Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger.  Multi-Scale Dense Networks for Resource Efficient Image Classification.  In _International Conference on Learning Representations_ , February 2018.
  * Hyvärinen & Oja (2000) A. Hyvärinen and E. Oja.  Independent component analysis: Algorithms and applications.  _Neural Networks_ , 13(4):411–430, June 2000\.  ISSN 0893-6080.  doi: 10.1016/S0893-6080(00)00026-5.
  * Johnson et al. (2016) Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta.  Composing graphical models with neural networks for structured representations and fast inference.  In _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc., 2016.
  * Kannan et al. (2018) Harini Kannan, Alexey Kurakin, and Ian Goodfellow.  Adversarial Logit Pairing, March 2018.
  * Ke et al. (2017) Tsung-Wei Ke, Michael Maire, and Stella X. Yu.  Multigrid Neural Architectures.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 6665–6673, 2017.
  * Kim & Mnih (2018) Hyunjik Kim and Andriy Mnih.  Disentangling by Factorising.  In _Proceedings of the 35th International Conference on Machine Learning_ , pp. 2649–2658. PMLR, July 2018.
  * Kotyan & Vargas (2022) Shashank Kotyan and Danilo Vasconcellos Vargas.  Adversarial robustness assessment: Why in evaluation both L0 and L$\infty$ attacks are necessary.  _PLOS ONE_ , 17(4):e0265723, April 2022.  ISSN 1932-6203.  doi: 10.1371/journal.pone.0265723.
  * Kotyan et al. (2022) Shashank Kotyan, Moe Matsuki, and Danilo Vasconcellos Vargas.  Transferability of features for neural networks links to adversarial attacks and defences.  _PLOS ONE_ , 17(4):e0266060, April 2022.  ISSN 1932-6203.  doi: 10.1371/journal.pone.0266060.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  In _Advances in Neural Information Processing Systems_ , volume 25. Curran Associates, Inc., 2012.
  * Kurakin et al. (2017) Alexey Kurakin, Ian Goodfellow, and Samy Bengio.  Adversarial examples in the physical world, February 2017.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards Deep Learning Models Resistant to Adversarial Attacks.  In _International Conference on Learning Representations_ , February 2018.
  * Makhzani et al. (2016) Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey.  Adversarial Autoencoders, May 2016.
  * Mathieu et al. (2019) Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh.  Disentangling Disentanglement in Variational Autoencoders.  In _Proceedings of the 36th International Conference on Machine Learning_ , pp. 4402–4412. PMLR, May 2019.
  * Mathieu et al. (2016) Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun.  Disentangling factors of variation in deep representation using adversarial training.  In _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc., 2016.
  * Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2574–2582, June 2016.  doi: 10.1109/CVPR.2016.282.
  * N et al. (2017) Siddharth N, Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr.  Learning Disentangled Representations with Semi-Supervised Deep Generative Models.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Nguyen et al. (2015) Anh Nguyen, Jason Yosinski, and Jeff Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 427–436, June 2015.  doi: 10.1109/CVPR.2015.7298640.
  * Recht et al. (2018) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do CIFAR-10 Classifiers Generalize to CIFAR-10?, June 2018.
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do ImageNet Classifiers Generalize to ImageNet?  In _Proceedings of the 36th International Conference on Machine Learning_ , pp. 5389–5400. PMLR, May 2019.
  * Reed et al. (2014) Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee.  Learning to Disentangle Factors of Variation with Manifold Interaction.  In _Proceedings of the 31st International Conference on Machine Learning_ , pp. 1431–1439. PMLR, June 2014.
  * Schmidhuber (1992) Jürgen Schmidhuber.  Learning Factorial Codes by Predictability Minimization.  _Neural Computation_ , 4(6):863–879, November 1992.  ISSN 0899-7667.  doi: 10.1162/neco.1992.4.6.863.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, January 2017.
  * Su et al. (2019) Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai.  One Pixel Attack for Fooling Deep Neural Networks.  _IEEE Transactions on Evolutionary Computation_ , 23(5):828–841, October 2019.  ISSN 1941-0026.  doi: 10.1109/TEVC.2019.2890858.
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks, February 2014.
  * Xie et al. (2017) Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He.  Aggregated Residual Transformations for Deep Neural Networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1492–1500, 2017.
  * Yang & Amari (1997) Howard Hua Yang and Shun-ichi Amari.  Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information.  _Neural Computation_ , 9(7):1457–1482, July 1997\.  ISSN 0899-7667.  doi: 10.1162/neco.1997.9.7.1457.
  * Zhang et al. (2019) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.  Theoretically Principled Trade-off between Robustness and Accuracy.  In _Proceedings of the 36th International Conference on Machine Learning_ , pp. 7472–7482. PMLR, May 2019.
  * Zheng et al. (2016) Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow.  Improving the Robustness of Deep Neural Networks via Stability Training.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4480–4488, 2016.


# Traffic Surveillance using Vehicle License Plate Detection and Recognition
in Bangladesh

. Md. Saif Hassan Onim1, Muhaiminul Islam Akash2, Mahmudul Haque3,
Raiyan Ibne Hafiz4
Military Institute of Science & Technology (MIST), Dhaka - 1216, Bangladesh
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>.

###### Abstract

Computer vision coupled with Deep Learning (DL) techniques bring out a
substantial prospect in the field of traffic control, monitoring and law
enforcing activities. This paper presents a YOLOv4 object detection model in
which the Convolutional Neural Network (CNN) is trained and tuned for
detecting the license plate of the vehicles of Bangladesh and recognizing
characters using tesseract from the detected license plates. Here we also
present a Graphical User Interface (GUI) based on Tkinter, a python package.
The license plate detection model is trained with mean average precision (mAP)
of 90.50% and performed in a single TESLA T4 GPU with an average of 14 frames
per second (fps) on real time video footage.

###### Index Terms:

License plate detection, OCR, ALPR, YOLOv4, CNN, tesseract, GUI

##  I Introduction

Vehicle surveillance from real time video footage has become an important
topic of research in the field of Artificial Intelligence (AI). It is
difficult to detect whether all the vehicles are obeying traffic laws, giving
tolls by traditional human traffic monitoring and police patrolling. Vehicle
License Plate (VLP) detection, recognition and tracking aid forensic experts
and practitioners in understanding their interested events. Due to preceding
practical applications, Vehicle detection along with Automatic License Plate
Recognition (ALPR) and Optical character recognition (OCR) are now a great
interest for researchers [1], [2]. ALPR typically consists of three
consecutive steps: detection of VLP, character segmentation and character
recognition. A large number of research work in the computer vision field has
carried out significant increase in performance specially for availability of
Graphical Processing Unit (GPU) and annotated dataset which are available in
ImageNet [3], Open Images dataset V6+ and other sources. Main deterrents of
VLP detection is the diversity of types and templates of VLP in different
countries.

A great advancement in object detection is carried out by YOLO stimulated
models [4], [5]. New features: Weighted Residual Connections, Cross State
Partial connections, Cross Mini-Batch Normalization, Self-adversarial
training, Mish Activation are used in YOLOv4 [6].Tesseract 4, a latest
released stable open-source OCR engine has Unicode (UTF-8) support and it has
the ability to recognize more than 100 languages. Tesseract 4 has a new,
neural network-based engine specifically Long short-term memory (LSTM) based
neural network.

In this paper our intentions were to detect and recognize the VLP of
Bangladesh which are of different types and Bangla characters. In order to
reach out our intention, we trained a vehicle classifier and a YOLOv4 VLP
detection model. Tesseract version 4 is used as our OCR engine to recognise
characters from the detected VLP contour. Before passing the detected VLP
contour to the OCR engine, several preprocessing techniques are used to
increase the performance of OCR. Another extension of our work is making the
whole system more user friendly and easier by designing a GUI.

##  II Literature Review

An end to end framework is proposed that can super resolve a sequence of low-
quality real-world traffic video frames to detect and make the unreadable VLP,
readable [7]. In many cases, vehicle detection is performed first then VLP
detection process starts in the contour of the detected vehicle to reduce
false positive detection.

Rayson et al. [8] proposed a new robust real-time YOLOv2 based CNN model.
Their intention was to detect only one class in case of both vehicle and VLP
detection. They first detected the vehicle then its VLP. This technique
results in reduced number of filters and higher speed. Both Fast YOLO and
YOLOv2 were used in their research work that outperformed two commercial ALPR
systems in the dataset of SSIG and their proposed UFPR-ALPR. They
consecutively achieved ALPR with all correct and redundancy accuracy of 64.89%
and 78.33% with 35 fps and time of 28.3022 milliseconds.

In [9] retrieval of text from images of different size, style, orientation,
complex background was presented. Bulan et al. [10] used artificial license
plate image generation and unsupervised domain adaptation. Viterbi algorithm
was used to identify code sequence from localized VLP region.

##  III Methodologies

Our ALPR process was comprised of 3 steps in which firstly, we classified
video frames with four classes: bus, car, motorbike and truck. If the vehicle
classifier detects the vehicle in the video frames, those frames will be fed
to the VLP detection model with one class: VLP. This process makes the system
efficient. At last OCR system gets the detected VLP contour for character
recognition. The complete ALPR system development and deployment is
represented in Fig. 1.

###  III-A Vehicle Classification

We decided to embed Transfer learning to our CNN architecture as suggested in
[11]. The ResNet based model requires heavy hardware configuration which is
not suitable for large scale implementation. Hence, as suggested in [12] we
chose MobileNetV2 as the backbone of our CNN followed by two custom hidden
layers to obtain our desired outcome as it requires relatively lighter
hardware configuration with negligible loss in accuracy. For overall
implementation of our CNN model we chose TensorFlow (TF) v2.3.0 framework.

#### Dataset Collection

We collected images from a video shot in a traffic intersection of Dhaka City,
the capital of Bangladesh. The images were collected by cropping the vehicles
from video frames.

#### Pre-Processing

We used geometrical data augmentation by using keras to create diversity in
the dataset. The collected vehicle images were classified into ’Bus’, ’Car’,
’Motorbike’ and ’Truck’. The collected images had variable sizes. So, they
were resized to (96, 96, 3) by up-sampling or down-sampling according to their
respective former sizes. All images were chosen after cross-validation and
split into training and validation dataset where 6,074 training images being
80% and 1,519 validation images being 20% of the total 7,593 images.

#### Transfer Learning (TL)

The amount of collected images is not sufficient to get the best outcome from
a DNN algorithm. So, to overcome this, along with pre-trained weights from
‘imagenet’ we got additional 2,263,108 parameters. MobileNetV2 was used as the
backbone of the DNN architecture which took an input of shape (96, 96, 3) and
gave an output of shape (3, 3, 1280).

Figure 1: Overall block diagram of the system

#### Additional Layers

On top of the MobileNetV2 layers, an average pooling layer was implemented.
Finally, we got our desired output using a dense layer of DNN which gives the
output i.e. the vehicle classification. The use of these additional layers
contributed to 5,124 trainable parameters resulting in a total of 2,257,984
parameters.

#### Hyper-parameters And Tuning

The training hyper-parameters are given in TABLE. I. Those parameters are
chosen after several trails. RMSprop [13] algorithm was used for optimization
and classification in the final stage. Besides, ‘sparse categorical cross
entropy’ was used. The last stage categorizes input images into classes they
fall under.

###  III-B YOLOv4 VLP Detection

For the purpose of robust object detection, the neural network’s fast
operating speed was the primary aim of YOLOv4. Alexey et al. [6] in YOLOv4
used CSPDarknet53 [14] as backbone. It consists of 29 convolutional layers, a
$725\times 725$ receptive field with parameters of 27.6 million. YOlOv3 [15]
is used as head and SPP [16], PAN [17] played the role of neck of the object
detector. Mosaic and self-adversarial training were used in YOLOv4 as new data
augmentation techniques. We trained our custom YOLOv4 [6] model with an
annotated dataset of 1500 training and 300 validation from Open Images Dataset
V6+. We obtained substantial results of correctly detecting VLP from real
world video footage from the roads in Dhaka, Bangladesh with significant
traffic density. If the number of classes = CL and with a view to predicting
bounding boxes, anchor boxes is A in YOLO each with four coordinates,
confidence and class probabilities [18]. The number of filters (used in three
convolutional layers before YOLO layers) can be represented by equation (1).
Here A = 5 is used. All modified parameters are given in TABLE. I.

$filters=(A+CL)\times 3$ (1)

TABLE I: Modified Parameters for Vehicle Classification & YOLOv4 VLP Detection Modified Parameters | Vehicle Classification | VLP Detection
---|---|---
classes | 4 | 1
batch | 32 | 64
subdivision | - | 16
learning_rate | 0.0001 | 0.001
max_batches (refers |  |
to epochs) | 100 | 3500
steps | 190, 90 | 4800, 5400
width, height | 96, 96 | 416, 416
filters | 32 |  $(5+1)\times 3$ = 18

Training was started with an initially set iteration number of 6000. After
2000 iterations, the decrease of average loss was not significant, moreover
the accuracy line started showing ups and downs after 3000 iterations.The
training was capable of taking backup of weights after each 1000 iterations.
The performance of every backup weights are given in TABLE. II. We took the
weight after 3000 iteration as our best weights for testing. The model was
trained in Google Colab using a single TESLA T4 GPU with 12 GB RAM.

###  III-C VLP Processing And Character Recognition

OCR is used to recognize characters from raw images in a machine-readable
format. There are mainly two steps for text recognition by means of OCR: image
pre-processing and OCR itself. To increase the accuracy and efficiency of OCR,
the identified license plates that we got from the VLP detection algorithm
were binarized using Otsu’s Binarization Method [19] in order to get desired
results despite factors like lighting, background and environment. The
processed image is then fed to Tesseract 4. The binary image is initially
segmented character-wise by tesseract, and individual characters are
recognized. The recognition is mainly focused on Bangla character recognition.
Tesseract has provision of using custom trained data for OCR but since the
license plates have printed texts in them, we have used the default Bangla
trained data which can be obtained through custom installation of Tesseract.

###  III-D GUI

Our GUI was built using 5 types of widgets. A user interacting with our system
using the GUI can perform various functions like monitoring vehicles,
recognizing VLP and query details about a vehicle from its license plate
number. These functions have been implemented by using various widgets and
geometry managers namely label, button, text, pack, frame and grid. In GUI, we
used a total of 5 frames. A main frame which encloses 4 other frames. In Fig.
2, there are a total of 5 buttons on the GUI for performing different
functions on command.

Figure 2: Interface of The GUI

  * •

Start Button: starts classifying the video footage.

  * •

Stop Button: stops classifying the video footage.

  * •

Print Button: displays the detected number plate and other information.

  * •

Warning Button: triggers an alarm to specific responders.

  * •

Record Button: creates a recording of the processed video footage and saves it
on local drive.

##  IV Experimental Result

###  IV-A Vehicle Classification

Training accuracy and loss of the classification model are 96.43% and 0.1065
respectively along with validation accuracy of 96.91% and loss of 0.1023.

TABLE II: Performance Metric of YOLOv4 VLP Detection Epochs | mAP | Precision | recall | F1-score
---|---|---|---|---
1000 | 78.20 | 0.70 | 0.81 | 0.75
2000 | 89.57 | 0.91 | 0.88 | 0.89
3000 | 90.50 | 0.93 | 0.86 | 0.89
4000 | 88.99 | 0.92 | 0.84 | 0.88
5000 | 89.02 | 0.93 | 0.87 | 0.90
6000 | 89.25 | 0.93 | 0.86 | 0.89

Figure 3: Detected VLP samples from real world captured video footage, Dhaka,
Bangladesh.

###  IV-B VLP Detection

We tested our model on recorded video footage captured by a Samsung Galaxy A70
smartphone with video resolution of $1920\times 1080$ with 30 fps. The visual
result of VLP detection with drawing bounding boxes are given in Fig. 3. The
attached images represent that our model was capable of detecting multiple VLP
in the video frames during daylight. In Google Colab TESLA T4 GPU, our model
runs on average 14 fps. The model gained the best mAP of 90.50% and recall of
0.86 during training and other performance parameters are shown in TABLE. II.

###  IV-C Character Recognition

Tesseract’s performance was tested with the test dataset available in
tesseract forum and the performance we got from the test data set is shown in
TABLE. III. The histogram of the VLP contour and output of tesseract are
illustrated in Fig. 4a and 4b. Our ALPR system was tested on a recorded real-
world traffic video footage from Dhaka, Bangladesh. The VLP detection model
detects the VLP contours in the consecutive video frames from the video. Then
those detected contours are extracted from the video and passed to the OCR
system. Detected VLP contours are of different geometric orientation and
brightness. For these reasons, to increase character recognition efficiency,
the images are processed before feeding to tesseract.

TABLE III: Performance Analysis of OCR Image | No of characters | Accuracy of OCR | Time taken for OCR
---|---|---|---
No | extracted | (in %) | (in Seconds)
1 | 5 | 56 | 0.402
2 | 4 | 40 | 0.548
3 | 7 | 70 | 0.402
4 | 4 | 40 | 0.701
5 | 4 | 44 | 0.705
6 | 2 | 22 | 0.7
7 | 8 | 73 | 1.717
8 | 6 | 67 | 0.806
9 | 6 | 55 | 0.596

(a)

(b)

Figure 4: Pre-processing of detected VLP contour and recognized OCR output

##  V Error Analysis

The probability of false positive detection of VLP is not negligible as
different objects existing on real world roads might be similar to VLP, this
result is false positive detection. If any false positive VLP contour is
passed to the OCR system to extract the characters of VLP, the processing will
be redundant. Another noticeable failure to detect VLP is when it is under
shadow or in direct sunlight. Detecting Vehicle contour first and VLP contour
second in the detected vehicle, false positive detection rate can be reduced.
Initial representation in Fig. 4b resembles that Bangla characters output
sequence got disrupted. Which was overcome by writing the result of OCR in a
text file. The angular orientation of VLP contour is one of the causes of
erroneous tesseract output. We noticed that when we manually fixed the
orientation, the result got improved. But the main problem of OCR is handling
blurry images. Motion blur, distance and angle of camera position with respect
to VLP significantly affects the performance of OCR. Performance might be
improved if several consecutive VLP frames are processed and filtered to come
to a single image with better quality.

##  VI Conclusion

This paper emphasized on license plate detection and its character recognition
of the vehicles of Bangladesh. A design of GUI is also presented so that it
can easily be operable by the users. Our initial works started consecutively
from image classification to detect the existence of vehicles in video frames,
VLP detection using YOLOv4 and tesseract as OCR engine. Character recognition
from VLP was one of the most competitive tasks in this ALPR system due to
various orientation, motion blur, lighting condition etc. of detected VLP. As
future work, our intentions are to reduce the effects of blurry VLP and to
overcome the deterrents of OCR by deploying preprocessing and compare our
prospective model with several existing algorithms from published literature,
specially for VLP detection and recognition of vehicles in Bangladesh.

## References

  * [1] S. Du, M. Ibrahim, M. Shehata, and W. Badawy, “Automatic license plate recognition (alpr): A state-of-the-art review,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 23, no. 2, pp. 311–325, 2013\.
  * [2] Ming-Chao Chiang and T. E. Boult, “Efficient image warping and super-resolution,” in _Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV’96_ , 1996, pp. 56–61.
  * [3] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [4] G. Ning, Z. Zhang, C. Huang, X. Ren, H. Wang, C. Cai, and Z. He, “Spatially supervised recurrent convolutional neural networks for visual object tracking,” in _2017 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 2017, pp. 1–4.
  * [5] B. Wu, F. Iandola, P. H. Jin, and K. Keutzer, “Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , July 2017.
  * [6] A. Bochkovskiy, C.-Y. Wang, and H.-y. Liao, “Yolov4: Optimal speed and accuracy of object detection,” 04 2020.
  * [7] H. Seibel, S. Goldenstein, and A. Rocha, “Eyes on the target: Super-resolution and license-plate recognition in low-quality surveillance videos,” _IEEE Access_ , vol. 5, pp. 20 020–20 035, 2017.
  * [8] R. Laroca, E. Severo, L. A. Zanlorensi, L. S. Oliveira, G. R. Gonçalves, W. R. Schwartz, and D. Menotti, “A robust real-time automatic license plate recognition based on the yolo detector,” in _2018 International Joint Conference on Neural Networks (IJCNN)_ , 2018, pp. 1–10.
  * [9] C. Patel, A. Patel, and D. Patel, “Optical character recognition by open source ocr tool tesseract: A case study,” _International Journal of Computer Applications_ , vol. 55, pp. 50–56, 10 2012.
  * [10] O. Bulan, V. Kozitsky, P. Ramesh, and M. Shreve, “Segmentation- and annotation-free license plate recognition with deep localization and failure identification,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 18, no. 9, pp. 2351–2363, 2017.
  * [11] Z. Huang, Z. Pan, and B. Lei, “Transfer learning with deep convolutional neural network for sar target classification with limited labeled data,” _Remote Sensing_ , vol. 9, no. 9, p. 907, Aug 2017. [Online]. Available: http://dx.doi.org/10.3390/rs9090907
  * [12] L. Bai, Y. Zhao, and X. Huang, “A cnn accelerator on fpga using depthwise separable convolution,” _IEEE Transactions on Circuits and Systems II: Express Briefs_ , vol. 65, no. 10, pp. 1415–1419, 2018.
  * [13] Y. Dauphin, H. Vries, J. Chung, and Y. Bengio, “Rmsprop and equilibrated adaptive learning rates for non-convex optimization,” _arXiv_ , vol. 35, 02 2015.
  * [14] C. Wang, H. Mark Liao, Y. Wu, P. Chen, J. Hsieh, and I. Yeh, “Cspnet: A new backbone that can enhance learning capability of cnn,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020, pp. 1571–1580.
  * [15] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” 2018, cite arxiv:1804.02767Comment: Tech Report. [Online]. Available: http://arxiv.org/abs/1804.02767
  * [16] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 9, pp. 1904–1916, 2015\.
  * [17] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8759–8768.
  * [18] J. Redmon and A. Farhadi, “Yolo9000: Better, faster, stronger,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 6517–6525.
  * [19] P. Puneet and N. Garg, “Binarization techniques used for grey scale images,” _International Journal of Computer Applications_ , vol. 71, pp. 8–11, 2013\.