# BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object
Detectors

Ali Harakeh1, Michael Smart2, and Steven L. Waslander1 1Ali Harakeh and Steven
L. Waslander are with The Institute For Aerospace Studies (UTIAS), University
of Toronto, Toronto, Canada<EMAIL_ADDRESS><EMAIL_ADDRESS>Michael Smart is with the Department of Mechanical
and Mechatronics Engineering, University of Waterloo, Waterloo, Canada,
<EMAIL_ADDRESS>
###### Abstract

When incorporating deep neural networks into robotic systems, a major
challenge is the lack of uncertainty measures associated with their output
predictions. Methods for uncertainty estimation in the output of deep object
detectors (DNNs) have been proposed in recent works, but have had limited
success due to 1) information loss at the detectors non-maximum suppression
(NMS) stage, and 2) failure to take into account the multitask, many-to-one
nature of anchor-based object detection. To that end, we introduce BayesOD, an
uncertainty estimation approach that reformulates the standard object detector
inference and Non-Maximum suppression components from a Bayesian perspective.
Experiments performed on four common object detection datasets show that
BayesOD provides uncertainty estimates that are better correlated with the
accuracy of detections, manifesting as a significant reduction of 9.77%-13.13%
on the minimum Gaussian uncertainty error metric and a reduction of
1.63%-5.23% on the minimum Categorical uncertainty error metric. Code will be
released at https://github.com/asharakeh/bayes-od-rc.

##  I Introduction

Due to their high level of performance, deep object detectors have become
standard components of perception stacks for safety critical tasks such as
autonomous driving [1, 2, 3] and automated surveillance [4]. Therefore, the
quantification of how trustworthy these detectors are for subsequent modules,
especially in safety critical systems, is of utmost importance. To encode the
level of confidence in an estimate, a meaningful and consistent measure of
uncertainty should be provided for every detection instance (see Fig. 1).

Two important goals must be met to create a meaningful uncertainty measure.
First, the robotic system should be capable of using the uncertainty measure
to fuse an object detector’s output with prior information from different
sources [5] to connect sequences of detections over time and increase
detection and tracking performance as a result. Second and most importantly,
the robotic system should be able to use its own estimates of detection
uncertainty to reliably identify incorrect detections, including those
resulting from out of distribution instances, where object categories,
scenarios, textures, or environmental conditions have not been seen during the
training phase [5].

Two sources of uncertainty can be identified in any machine learning model.
Epistemic or model uncertainty is the uncertainty in the model’s parameters,
usually as a result of the confusion about which model generated the training
data, and can be explained away given enough representative training data
points [6]. Aleatoric or observation uncertainty results from the stochastic
nature of the observed input, and persists in network output despite expanded
training on additional data [7].

Figure 1: The output from BayesOD, demonstrated on a test image frame from the
KITTI Dataset [8]. Three levels of trust (teal: highly reliable, orange:
slightly reliable and red: unreliable) are determined based on thresholds of
the Gaussian entropy provided by BayesOD. All bounding boxes are shown with
the 95% confidence ellipse of their top-left and bottom-right corners.

Methods to estimate both uncertainty types in DNNs have been recently proposed
by Kendal et al. [7], with applications to pixel-wise perception tasks. Recent
methods [9, 10, 11, 12, 13, 14, 15] extended Kendal’s work [7] to object
detection, but fail to consider the multi-task, many-to-one nature of the
object detection task. To that end, we introduce BayesOD, a framework designed
to estimate the uncertainty in both bounding box and category of detected
object instances. This paper offers the following contributions:

  * •

We provide a Bayesian treatment for every step of the neural network inference
procedure, allowing the incorporation of anchor-level and object-level priors
in closed form.

  * •

We replace standard non-maximum suppression (NMS) with Bayesian inference,
allowing the detector to retain all predicted information for both the
bounding box and the category of a detected object instance.

  * •

We perform comprehensive experiments to quantify the quality of the estimated
uncertainty on four commonly used 2D object detection datasets, COCO, Pascal
VOC, Berkeley Deep Drive and Kitti. We show that BayesOD provides a
significant reduction of $9.77\%-13.13\%$ on the minimum Gaussian uncertainty
error metric, a reduction of $1.63\%-5.23\%$ on the minimum Categorical
uncertainty error metric, and an increase of $0.07\%-3.00\%$ on the
probabilistic detection quality over the next best method from current state
of the art.

Figure 2: The different stages of estimation employed in BayesOD, demonstrated
on a test image frame from the BDD Dataset [16]. The additions by BayesOD to a
standard object detector (grey) are shown in purple. Prior information is
shown in red. Left: Prior bounding boxes. Middle: Object detector results
after processing the prior boxes and incorporating anchor-level non-
informative priors. Right: Final detection results after clustering and
Bayesian Inference. Box corner covariance is visualized as in Fig. 1

##  II Related Work

###  II-A Deep Neural Networks For Object Detection

The object detection problem requires the estimation of both the category to
which an object belongs, and its spatial location and extent, often expressed
as the tightest fitting bounding box. The majority of state of the art object
detectors in 2D [17] or in 3D [1, 2, 3] follow a standard algorithm, which
maps a scene representation to object instances. Since the number of object
instances in the scene is usually unknown a priori, the procedure begins with
a densely sampled grid of prior object bounding boxes, referred to as anchors
[18, 19], where the object detector provides a category and a bounding box
estimate for each anchor element. Since multiple anchors can be mapped to a
single bounding box in space, redundant outputs are eliminated through Non-
Maximum Suppression. BayesOD builds on the RetinaNet 2D object detector [19].

###  II-B Uncertainty Estimation In Deep Object Detectors

To account for epistemic uncertainty, Bayesian Neural Networks [20] usually
apply a prior distribution over their parameters $\bm{\theta}$ to compute a
posterior distribution $p(\bm{\theta}|\mathcal{D})$ over the set of all
possible parameters given the training dataset $\mathcal{D}$. A marginal
distribution can then be computed for any prediction as:

$p(\bm{\hat{y}}_{i}|\textbf{x}_{i},\mathcal{D})=\int\limits_{\theta}p(\bm{\hat{y}}_{i}|\textbf{x}_{i},\mathcal{D},\bm{\theta})p(\bm{\theta}|\mathcal{D})d\bm{\theta},$
(1)

where $\textbf{x}_{i}$ is the input, and $\bm{\hat{y}}_{i}$ is the output of
the neural network. Unfortunately, the calculation of the integral in Eq. (1)
is usually intractable due to the non-linear activation function between
consecutive layers [21]. Tractable approximations can be derived through
Monte-Carlo integration by using ensemble methods [22] or Monte Carlo (MC)
Dropout [6].

To estimate the epistemic uncertainty in the output of deep object detectors,
Miller et al. [9] directly applies MC Dropout, treating the deep object
detector as a black box. Uncertainty is then estimated as sample statistics
from spatially correlated detector outputs. Subsequent work [10] studied the
effect of various correlation and merging algorithms on the quality of the
estimated uncertainty measures from the black box method in [9]. The black box
method is shown to provide weakly correlated estimates for bounding box
uncertainty, mainly because it observes the output bounding box after NMS,
where most of the information from redundant predictions has already been
removed.

Kendall et al. [7] provides one of the first works to address the estimation
of aleatoric uncertainty for computer vision tasks. For regression tasks, a
log likelihood loss is used to estimate heteroscedastic aleatoric uncertainty,
written for every regression target as:

$L_{reg}(\textbf{x},\bm{\theta})=\frac{1}{2\sigma(\textbf{x},\bm{\theta})^{2}}||\textbf{y}-f(\textbf{x},\bm{\theta})||^{2}_{2}+\frac{1}{2}\log\sigma(\textbf{x},\bm{\theta})^{2},$
(2)

where x is the input to, and $f(\textbf{x},\bm{\theta})$ is the output from
the neural network. Furthermore y is the ground truth regression target,
$||.||_{2}$ is the $L_{2}$ norm, $\bm{\theta}$ are the neural network
parameters, and $\sigma(\textbf{x},\bm{\theta})$ is the estimated output
variance.

Le et al. [15] directly apply the formulation in Eq. (2) to estimate the
diagonal elements of the covariance matrix of the bounding box output from
object detectors. Such methods are referred to as sampling free and require
only a single run of the deep object detector to estimate uncertainty. The
estimated variance in Eq. (2) has also been used in [11, 12, 14] to increase
average precision, by incorporating it in the non-maximum suppression stage,
while disregarding the quality of the output uncertainty. The proposed
sampling free methods assume a diagonal covariance matrix and still use NMS to
eliminate low scoring predictions, reducing the quality of their estimated
uncertainty for both objects’ bounding box and category.

Le et al. [15] estimate aleatoric uncertainty in deep object detectors by
exploiting anchor redundancy, where multiple per-anchors predictions map to
the same object. These predictions are clustered using spatial affinity before
NMS, and uncertainty measures are estimated using the cluster associated with
every output prediction. Finally, a straightforward extension of [7] is
typically used to perform joint estimation of epistemic and aleatoric
uncertainty in deep object detectors [13, 23], while still employing NMS to
eliminate rather than fuse information from redundant anchors.

Unlike each of the existing methods, BayesOD replaces NMS with Bayesian
inference significantly improving the quality of its uncertainty estimates. In
addition, BayesOD is the first method to tackle fusion of the category from
redundant output anchors, as well as to provide a multivariate extension of
Eq. (2) to estimate the aleatoric uncertainty of objects’ bounding boxes.

##  III A Bayesian Formulation For Object Detection:

Throughout this section, the bounding box of an object, represented by its top
left and bottom right corners, is denoted as $\mathcal{B}$, whereas its
category, represented by a one-hot vector, is denoted as $\mathcal{S}$. The
index $i$ is used to signify a variable related to the $i$th anchor in the
anchor grid. Variables not indexed with $i$ represent inference output
clustered over several anchors. Finally, predictions provided by the neural
network are denoted with a $\hat{.}$ operator.

###  III-A Computing The Per-Anchor Gaussian Posterior:

Computing the uncertainty in the estimated per-anchor bounding box: Following
[7] and using MC-Dropout as a tractable approximation of the integral in Eq.
(1), the sufficient statistics of the Gaussian marginal probability
distribution describing the estimated per-anchor bounding box
$\hat{\mathcal{B}}_{i}\sim\mathcal{N}(\bm{\mu}(\textbf{x}_{i}),\Sigma(\textbf{x}_{i}))$
can be derived as:

$\displaystyle\bm{\mu}(\textbf{x}_{i})=\frac{1}{T}\sum\limits_{t=1}^{T}f(\textbf{x}_{i},\bm{\theta}_{t})$
(3)
$\displaystyle\Sigma_{e}(\textbf{x}_{i})=\frac{1}{T}\left(\sum\limits_{t=1}^{T}f(\textbf{x}_{i},\bm{\theta}_{t})f(\textbf{x}_{i},\bm{\theta}_{t})^{\intercal}\right)-\bm{\mu}(\textbf{x}_{i})\bm{\mu}(\textbf{x}_{i})^{\intercal},$
(4)

where $T$ is the number of times MC-Dropout sampling is performed, and
$f(\textbf{x}_{i},\bm{\theta}_{t})$ is the bounding box regression output of
the neural network for the $t$th MC-Dropout run. The covariance matrix,
$\Sigma_{e}$, captures the epistemic uncertainty in the estimated bounding box
$\hat{\mathcal{B}}_{i}$.

Eq. (3) is sufficient to compute the output mean of the per-anchor bounding
box $\hat{\mathcal{B}}_{i}$. However, Eq. (4) still needs to account for the
aleatoric component of uncertainty, where the final per-anchor output
covariance $\Sigma(\textbf{x}_{i})$ can be approximated as:

$\Sigma(\textbf{x}_{i})=\Sigma_{e}(\textbf{x}_{i})+\frac{1}{T}\sum\limits_{t=1}^{T}\Sigma_{a}(\textbf{x}_{i},\bm{\theta}_{t}).$
(5)

To estimate the full covariance matrix $\Sigma_{a}(\textbf{x}_{i})$, a novel
multivariate log likelihood regression loss is derived as:

$L_{mv}(\textbf{x}_{i},\bm{\theta})=\\\
\frac{1}{2}(f(\textbf{x}_{i},\bm{\theta})-\textbf{y}_{i})^{\intercal}\Sigma_{a}(\textbf{x}_{i},\bm{\theta})^{-1}(f(\textbf{x}_{i},\bm{\theta})-\textbf{y}_{i})\\\
+\frac{1}{2}\log\det\Sigma_{a}(\textbf{x}_{i},\bm{\theta}),$ (6)

where $\Sigma_{a}(\textbf{x}_{i},\bm{\theta})$ is the predicted per-anchor
aleatoric covaraince matrix, $f(\textbf{x}_{i},\bm{\theta})$ is the predicted
per-anchor bounding box, and $\textbf{y}_{i}$ is the associated regression
target. However, the loss in Eq. (6) is found to be numerically unstable.
Furthermore, there are no guarantees on the positive definiteness of the
predicted covariance matrix $\Sigma_{a}(\textbf{x}_{i},\bm{\theta})$. Using
the $LDL$ decomposition of
$\Sigma_{a}(\textbf{x}_{i},\bm{\theta})=L(\textbf{x}_{i},\theta)D(\textbf{x}_{i},\theta)L(\textbf{x}_{i},\theta)^{\intercal}$,
in conjunction with the Cauchy-Schwarz inequality, a numerically stable
surrogate loss function is derived as:

$L_{mv}(\textbf{x}_{i},\bm{\theta})=\\\
\frac{1}{2}||L(\textbf{x}_{i},\bm{\theta})^{-1}||_{F}^{2}||D(\textbf{x}_{i},\bm{\theta})^{-\frac{1}{2}}(f(\textbf{x}_{i},\bm{\theta})-\textbf{y}_{i})||_{2}^{2}\\\
+\frac{1}{2}tr(\log D(\textbf{x}_{i},\bm{\theta})),$ (7)

where $L(\textbf{x}_{i},\bm{\theta})$ is a lower triangular matrix with ones
for its diagonal entries, and $D(\textbf{x}_{i},\bm{\theta})$ is a diagonal
matrix. The loss function in Eq. (7) is a numerically stable upper bound of
the one in Eq. (6) and can guarantee the positive definiteness of
$\Sigma_{a}(\textbf{x}_{i},\bm{\theta})$ by predicting positive values for the
diagonal elements of $D(\textbf{x}_{i},\bm{\theta})$ through standard
activation functions. The final output distributions after incorporating both
epistemic and aleatoric covariance estimates are plotted as bounding boxes in
the middle image of Fig. 2.

Incorporating per-anchor bounding box priors: The per-anchor bounding box
prior is usually defined based on the training dataset $\mathcal{D}$ as
$p(\mathcal{B}|\textbf{x}_{i})\sim\mathcal{N}(\bm{\mu}_{0},\Sigma_{0})$. The
per-anchor posterior distribution describing the bounding box $\mathcal{B}$
can then be written as:

$p(\mathcal{B}|\textbf{x}_{i},\mathcal{D},\hat{\mathcal{B}}_{i})\propto
p(\hat{\mathcal{B}}_{i}|\textbf{x}_{i},\mathcal{D},\mathcal{B})p(\mathcal{B}|\textbf{x}_{i},\mathcal{D}).$
(8)

$p(\hat{\mathcal{B}}_{i}|\textbf{x}_{i},\mathcal{D},\mathcal{B})$ is a
Gaussian likelihood function described by the sufficient statistics
$[\bm{\mu}(\textbf{x}_{i}),\Sigma(\textbf{x}_{i})]$ in equations Eq. (3) and
Eq. (5). The sufficient statistics can be computed through the multivariate
Gaussian conjugate update, as:

$\displaystyle\Sigma^{\prime}(\textbf{x}_{i})$
$\displaystyle=(\Sigma_{0}^{-1}+\Sigma(\textbf{x}_{i})^{-1})^{-1}$ (9)
$\displaystyle\bm{\mu}^{\prime}(\textbf{x}_{i})$
$\displaystyle=\Sigma^{\prime}(\textbf{x}_{i})(\Sigma_{0}^{-1}\bm{\mu}_{0}+\Sigma(\textbf{x}_{i})\bm{\mu}(\textbf{x}_{i})).$
(10)

The choice of anchor priors depends on the application, and whether object
information is actually available a priori. Since no useful bounding box
information is available from our 2D training datasets, a non-informative
prior, visually shown in the left image of Fig. 2, is chosen for $\mathcal{B}$
following [24].

###  III-B Computing The Per-Anchor Categorical Posterior:

Computing the uncertainty in the estimated per-anchor category: Since the
neural network outputs the parameters of a Categorical distribution rather
than one-hot categorical samples, the parameters for the Categorical marginal
conditional probability distribution $\hat{\mathcal{S}}_{i}\sim
Cat([\hat{p}_{1}\dots\hat{p}_{K}])$ can be computed as:

$\displaystyle\hat{p}_{k}=\frac{1}{T}\sum\limits_{t=1}^{T}\mbox{SoftMax}(g(\textbf{x}_{i},\bm{\theta}_{t}))_{k},$
(11)

where $\mbox{SoftMax}(.)$ is the soft max function, and
$g(\textbf{x}_{i},\bm{\theta}_{t})_{k}$ is the output logit of the $k$th
category, estimated at the $t$th MC-Dropout run of the neural network. No
explicit treatment of the aleatoric classification uncertainty is performed,
since it is already contained within the estimated parameters
$[\hat{p}_{1}\dots\hat{p}_{K}]$ [14].

Incorporating per-anchor category priors: For the object category, a Dirichlet
distribution is set as a prior over the parameters $\mathcal{P}$ of the
categorical distribution $Cat(\mathcal{P})$ generating $\mathcal{S}$, instead
of incorporating a prior distribution directly over the category
$\mathcal{S}$. The posterior distribution of the categorical parameters can be
written as:

$p(\mathcal{P}|\textbf{x}_{i},\mathcal{D},\hat{\bm{Z}}_{i})\propto
p(\hat{\bm{Z}}_{i}|\textbf{x}_{i},\mathcal{D},\mathcal{P})p(\mathcal{P}|\textbf{x}_{i},\mathcal{D}),$
(12)

where $\mathcal{P}$ is the set of updated parameters
$[p^{\prime}_{1},\dots,p^{\prime}_{K}]$, and
$\hat{\bm{Z}_{i}}=[\hat{\bm{z_{1}}},\dots,\hat{\bm{z}}_{H}]$ are $H$ i.i.d.
samples from $Cat([\hat{p}_{1},\dots,\hat{p}_{K}])$. Since the likelihood
function $p(\hat{\bm{Z}_{i}}|\textbf{x}_{i},\mathcal{D},\mathcal{P})$ is a
categorical distribution, the prior distribution
$p(\mathcal{P}|\textbf{x}_{i},\mathcal{D})$ is chosen to be a Dirichlet
distribution allowing a Dirichlet posterior to be computed in closed form as:

$\displaystyle p(\mathcal{P}|\textbf{x}_{i},\mathcal{D},\hat{\bm{Z}}_{i})$
$\displaystyle\propto\prod\limits_{k=1}^{K}p_{k}^{\alpha_{k}-1}\prod\limits_{h=1}^{H}\prod\limits_{k=1}^{K}p_{k}^{\hat{z}_{hk}=1}$
$\displaystyle=Dir(\alpha^{\prime}_{1},\ldots,\alpha^{\prime}_{K}),$ (13)

where $\hat{z}_{hk}$ is the element in instance $\hat{\bm{z}}_{h}$
corresponding to category $k$, and
$[\alpha_{k}^{\prime}=\alpha_{k}+\sum\limits_{h=1}^{H}\hat{z}_{hk}\ \forall\
k=1,\dots,K]$ are the inferred parameters of the Dirichlet posterior
distribution. The per-anchor categorical posterior distribution can be written
as:

$p(\mathcal{S}|\textbf{x}_{i},\mathcal{D},\bm{Z}_{i})=Cat([p^{\prime}_{1},\ldots,p^{\prime}_{K}]),$
(14)

where $p^{\prime}_{k}$ is the mean of the Dirichlet posterior distribution
[24] in Eq. (III-B) written as:

$\displaystyle
p^{\prime}_{k}=\frac{\alpha^{\prime}_{k}}{\sum\limits_{j=1}^{K}\alpha^{\prime}_{j}}.$

Similar to the prior used for the per-anchor bounding box, we choose a non-
informative Dirichlet prior for the per-anchor category following [24].
Although non-informative, the prior still serves an essential purpose by
allowing the derivation of a Dirichlet posterior in Eq. (III-B), which will
allow the fusion of information from multiple clustered categorical variables
in the next section.

###  III-C Bayesian Inference as a Replacement to NMS:

Similar to NMS, BayesOD clusters per-anchor outputs from the neural network
using spatial affinity. However, all elements in the cluster are then combined
regardless of their classification score during inference. Greedy clustering
is chosen as it provides adequate performance when compared to standard NMS,
while maintaining computational efficiency. For better performing but slower
clustering algorithms, see [10].

For the remainder of this section, we will continue the derivation for a
single anchor cluster containing $M$ anchors. The anchor with the highest
categorical score is considered the cluster’s center, is indexed by $1$, and
is described with the posterior distributions in Eq. (8) and Eq. (12). The
rest of the cluster members are assumed to be measurement outputs from the
neural network described by the states $\hat{\mathcal{S}}_{i}$ and
$\hat{\mathcal{B}}_{i}$, and are used to update the bounding box and category
of the cluster center. Specifically, the final posterior distribution
describing an object’s bounding box is:

$\displaystyle
p(\mathcal{B}|\mathcal{X},\mathcal{D},[\hat{\mathcal{B}}_{1},\dots,\hat{\mathcal{B}}_{M}])$
$\displaystyle\propto
p(\mathcal{B}|\textbf{x}_{1},\mathcal{D},\hat{\mathcal{B}}_{1})\prod\limits_{i=2}^{M}p(\hat{\mathcal{B}}_{i}|\textbf{x}_{i},\mathcal{D},\mathcal{B})$
$\displaystyle=\mathcal{N}(\bm{\mu}^{\prime\prime}(\mathcal{X}),\Sigma^{\prime\prime}(\mathcal{X})),$
(15)

where $\mathcal{X}$ is the set of inputs $[\textbf{x}_{i}\ |\ i=1\dots M]$
corresponding to the $M$ cluster members,
$p(\mathcal{B}|\textbf{x}_{1},\mathcal{D},\hat{\mathcal{B}}_{1})$ is the per-
anchor posterior distribution of the cluster center, and
$p(\mathcal{B}|\textbf{x}_{1},\mathcal{D},\hat{\mathcal{B}}_{1})\prod\limits_{i=2}^{M}p(\hat{\mathcal{B}}_{i}|\textbf{x}_{i},\mathcal{D},\mathcal{B})$
is the likelihood derived through a conditional independence assumption of the
$\hat{\mathcal{B}}_{i}$ of the cluster members given $\mathcal{B}$. The
sufficient statistics of Eq. (III-C) can be estimated in closed form as:

$\displaystyle\Sigma^{\prime\prime}(\mathcal{X})$
$\displaystyle=\left(\sum\limits_{i=1}^{M}\Sigma^{\prime}(\textbf{x}_{i})^{-1}\right)^{-1}$
(16) $\displaystyle\bm{\mu}^{\prime\prime}(\mathcal{X})$
$\displaystyle=\Sigma^{\prime\prime}(\mathcal{X})\left(\sum\limits_{i=1}^{M}\Sigma^{\prime}(\textbf{x}_{i})^{-1}\bm{\mu}^{\prime}(\textbf{x}_{i})\right),$
(17)

where $\bm{\mu}^{\prime}(\textbf{x}_{i}),\Sigma^{\prime}(\textbf{x}_{i})$ are
the sufficient statistics of the per anchor posterior distribution derived in
Eq. (8).

To arrive at the final posterior distribution describing the category
$\mathcal{S}$, a similar analysis can be performed to update the sufficient
statistics $\mathcal{P}$ of the cluster center with categorical measurements
$[\hat{\bm{Z}}_{2},\dots,\hat{\bm{Z}}_{m}]$ of the rest of the cluster
members. Specifically, the posterior probability of $\mathcal{P}$ can be
derived as:

$\displaystyle
p(\mathcal{P}|\textbf{x}_{i},\mathcal{D},[\hat{\bm{Z}}_{1},\dots,\hat{\bm{Z}}_{M}])$
$\displaystyle\propto
p(\mathcal{P}|\textbf{x}_{1},\mathcal{D},\hat{\bm{Z}}_{1})\prod\limits_{i=2}^{M}p(\hat{\bm{Z}}_{i}|\textbf{x}_{i},\mathcal{D},\mathcal{P})$
$\displaystyle=Dir(\alpha^{\prime\prime}_{1},\dots,\alpha^{\prime\prime}_{K})$
(18)

where
$\alpha^{\prime\prime}_{k}=\alpha^{\prime}_{k}+\sum\limits_{i=2}^{M}\sum\limits_{h=1}^{H}\hat{z}_{ihk}\
\forall\ k=1\dots K$, and the categorical measurements
$[\hat{\bm{Z}}_{2},\dots,\hat{\bm{Z}}_{m}]$ are assumed to be i.i.d. In
summary, $\alpha^{\prime\prime}_{k}$ is derived by updating the per-anchor
Dirichlet posterior distribution in (12) of the cluster center with index
$i=1$ with categorical measurements $\bm{Z}_{2},\dots,\bm{Z}_{M}$ from all
cluster members. The final categorical distribution describing the state
$\mathcal{S}$ is then:

$p(\mathcal{S}|\mathcal{X},\mathcal{D},[\hat{\bm{Z}}_{1},\dots,\hat{\bm{Z}}_{M}])=Cat(p^{\prime\prime}_{1},\dots,p^{\prime\prime}_{K}),$
(19)

where $[p^{\prime\prime}_{1},\dots,p^{\prime\prime}_{K}]$ is computed as the
mean of the posterior distribution in Eq. (III-C):

$p^{\prime\prime}_{k}=\frac{\alpha^{\prime\prime}_{k}}{\sum\limits_{j=1}^{K}\alpha^{\prime\prime}_{j}},$
(20)

Note that every member of the cluster contributes to the estimation of the
final bounding box and category states of the object. Furthermore, the output
distributions for both the category and bounding box can be updated with
object-level priors using the same equations presented in sections III-A and
III-B. The final output from BayesOD is shown as the rightmost image in Fig.
2.

Training Dataset | Testing Dataset | Method | mAP(%) $\uparrow$ | PDQ Score(%) $\uparrow$ | mGMUE(%) $\downarrow$ | mCMUE(%) $\downarrow$
---|---|---|---|---|---|---
BDD | BDD | Sampling Free | 36.59 | 33.97 | 44.19 | 28.46
Black Box | 36.43 | 32.46 | 47.63 | 30.45
Anchor Redundancy | 32.92 | 29.57 | 48.56 | 35.58
Joint Aleatoric-Epistemic | 36.84 | 29.57 | 46.35 | 28.28
BayesOD | 38.14 | 36.79 | 34.42 | 24.85
BDD | Kitti | Sampling Free | 64.78 | 29.24 | 46.70 | 20.67
Black Box | 62.96 | 32.26 | 49.23 | 22.27
Anchor Redundancy | 64.83 | 29.57 | 48.56 | 35.58
Joint Aleatoric-Epistemic | 62.96 | 29.57 | 46.35 | 28.28
BayesOD | 63.34 | 35.26 | 30.06 | 15.58
COCO | COCO | Sampling Free | 31.89 | 22.43 | 40.39 | 25.76
Black Box | 33.71 | 21.87 | 45.26 | 28.68
Anchor Redundancy | 29.94 | 17.63 | 43.74 | 31.13
Joint Aleatoric-Epistemic | 32.68 | 23.08 | 42.90 | 26.51
BayesOD | 35.41 | 23.15 | 30.23 | 24.13
COCO | Pascal VOC | Sampling Free | 54.94 | 14.18 | 49.49 | 29.63
Black Box | 54.67 | 12.77 | 48.90 | 29.42
Anchor Redundancy | 51.56 | 13.06 | 48.67 | 39.64
Joint Aleatoric-Epistemic | 55.43 | 11.62 | 49.99 | 30.14
BayesOD | 56.00 | 13.23 | 36.36 | 24.19

TABLE I: The results of the evaluation of Sampling Free [14, 15], Black Box
[9, 10], Anchor Redundancy [15], and Joint Aleatoric-Epistemic [13, 23] state
of the art methods compared to BayesOD.

##  IV Experiments and Results

To show the effectiveness of BayesOD in comparison to the state of the art, it
is applied to the problem of 2D object detection in image space. The
evaluation is based on four commonly used datasets:

  * •

Berkley Deep Drive 100K Dataset (BDD) [16] road scene dataset, with $80K$
frames used according to the official $70K/10K$ training/validation split.
Models trained on BDD are also tested on $7,481$ frames of KITTI [8]. Both
datasets contain $7$ common road scene object categories.

  * •

MS COCO [25] dataset, with $223K$ frames that contain instances from $81$
different object categories, and an official $118K/5K$ training/testing split.
Models trained on COCO are also tested on $5,823$ frames from Pascal VOC [26],
which shares $20$ object categories with the COCO dataset.

Models used for testing are not allowed to observe instances from the KITTI or
Pascal VOC datasets.

All baseline uncertainty estimation methods used in comparison are integrated
into the inference process of RetinaNet [19], trained using the regression
loss function in Eq. (2) to estimate a diagonal bounding box covariance
matrix. Full aleatoric covariance matrix results are provided through a second
RetinaNet model, trained using the proposed regression loss in Eq. (7). For
additional information on RetinaNet’s training procedure and hyperparamters,
see [19].

###  IV-A Evaluation Metrics

Three evaluation metrics are used to quantify the performance of uncertainty
estimation methods in comparison to BayesOD. For performance on the detection
task, we use the Mean Average Precision (mAP) [25, 26, 16, 8] at $0.5$ IOU.
The maximum mean average precision achievable by a detector is $100\%$.

The Minimum Uncertainty Error (MUE) [10] at $0.5$ IOU is used to determine the
ability of the detector’s estimated uncertainty to discriminate true positives
from false positives. The lowest MUE achievable by a detector is $0\%$. We
define the Gaussian MUE (GMUE) when the Gaussian entropy is used, Categorical
MUE (CMUE) when the Categorical entropy is used. Finally, we average the GMUE
and CMUE over all categories in a testing dataset to arrive to a single value,
the Mean (Gaussian or Categorical) MUE (mGMUE or mCMUE).

Finally, we use the newly proposed Probability Based Detection Quality (PDQ)
[27] to jointly quantify the bounding box and category probability assigned to
true positives by the detector. The highest PDQ achievable by a detector is
$100\%$, where the PDQ increases as the distributions assigned to a detection
better match those of the ground truth instance. For detailed information on
the three evaluation metrics, we refer the reader to the [26, 10, 27].

###  IV-B Comparison With State of The Art Methods:

BayesOD is compared against four approaches representing the state of the art
methods for uncertainty estimation methods used for object detection. The four
approaches are referred to as: Black Box [9, 10], Sampling Free [15, 14],
Anchor Redundancy [15], and Joint Aleatoric Epistemic [13]. BayesOD, Black
Box, and Joint Aleatoric Epistemic use $10$ stochastic runs of MC-Dropout,
while Sampling Free and Anchor Redundancy use only one non-stochastic run. As
such, BayesOD, Black Box, and Joint Aleatoric Epistemic run at a similar frame
rate, approximately $4\times$ slower than Sampling Free and Anchor Redundancy.
The affinity threshold used for clustering in all methods was set to the $0.5$
IOU, similar to that used for NMS in RetinaNet. The number of categorical
samples $H$ in Eq. (12) is empirically set to $30$.

Table I shows the results of evaluating the four methods in comparison to
BayesOD, on the four testing datasets. BayesOD is seen to outperform all four
methods on mAP when tested on the BDD, COCO and PASCAL VOC datasets by a
margin of $0.57\%-1.7\%$ over the second best method, but is outperformed on
the KITTI dataset by $\sim 1.5\%$ when using the Sampling Free and Anchor
Redundancy methods. Such reduction in performance on KITTI is noted with all
methods using MC-Dropout, implying that MC-Dropout might hurt mAP performance
in cases where the testing dataset is semantically different than the training
dataset.

Similarly, BayesOD also outperforms all four methods on PDQ when tested on the
BDD, KITTI and COCO datasets by a margin of $0.07\%-3.00\%$ over the second
best method. BayesOD is outperformed on the PASCAL VOC dataset by $0.95\%$
when using the sampling free method. Considering the performance only on PDQ,
it cannot be determined if a method is assigning lower probability values to
false positives.

On the other hand, the mGMUE/mCMUE are capable of providing a quantitative
measure of how well the estimated uncertainty can be used to separate correct
and incorrect detections [10]. BayesOD provides a significant reduction of
$9.77\%-13.13\%$ in mGMUE over the next best method on all four testing
datasets. Combined with BayesOD’s performance on the PDQ metric, it can be
inferred that BayesOD not only assigns adequate probability to true positives,
but also assigns a lower probability to false positives when compared to true
positives. Finally, when comparing mCMUE, BayesOD provides a reduction between
$1.63\%-5.23\%$ over the next best method on all four datasets.

$\\#$ | Experiment | mAP(%) $\uparrow$ | PDQ Score(%) $\uparrow$ | mGMUE(%) $\downarrow$ | mCMUE(%) $\downarrow$
---|---|---|---|---|---
1 | Full System | 35.41 | 23.15 | 30.23 | 24.13
2 | Diagonal Covariance | 34.77 | 22.64 | 30.69 | 25.25
3 | Epistemic Only | 34.15 | 22.62 | 35.88 | 26.47
4 | Aleatoric Only | 34.12 | 22.67 | 28.95 | 25.60
5 | Standard NMS | 34.70 | 22.65 | 43.19 | 25.10

TABLE II: The results of ablation studies performed on BayesOD using the COCO
dataset for training and testing.

###  IV-C Ablation Studies:

Table II shows the results of the mAP, PDQ, mGMUE, and mCMUE for the ablation
studies performed on the COCO dataset. The results of the full BayesOD
framework can be seen in experiment $\\#1$. By analyzing the results of the
ablation studies, the following claims are put forth:

Learning the off-diagonal elements of the covariance matrix provides slightly
better uncertainty estimates for the objects’ bounding box. To support this
claim, RetinaNet is trained using the original log likelihood loss in Eq. (2)
instead of the proposed multivariate loss in Eq. (7). The results of BayesOD
using this original loss formulation are shown in experiment $\\#2$. When
compared to the full system, an increase of $0.48\%$ is observed in mGMUE.
Although the improvement is not substantial, the new proposed loss avoids an
explicit independence assumption and allows the neural network to learn to
drive the off-diagonal elements of the covariance matrix towards $0$ if
needed.

Aleatoric uncertainty provides a more discriminative uncertainty estimate for
the objects’ bounding box over epistemic uncertainty estimated from MC-
Dropout. To support this claim BayesOD is implemented without the update step
in Eq. (5), to use only the per-anchor sample variance computed from multiple
stochastic runs of MC-Dropout. The results, presented in experiment $\\#3$,
show an increase of $5.65\%$ and $2.34\%$ is observed in the mGMUE and mCMUE
respectively. Note however that this conclusion is specific to MC-Dropout, and
might not be valid for alternative epistemic uncertainty estimation
mechanisms.

To provide better insight on the effect of epistemic uncertainty from MC-
Dropout on the full system, experiment $\\#4$ is performed by using BayesOD
with a single inference run, and without any epistemic uncertainty estimation
mechanism. The results show a decrease in mGMUE of $6.93\%$ over experiment
$\\#3$, and $1.28\%$ over the full system, further cementing the conclusion
that MC-Dropout might not be a good method to estimate epistemic uncertainty
in deep object detectors.

Greedy Non-Maximum Suppression is detrimental to the discriminative power of
the uncertainty in the objects’ bounding box. To support this claim, the
elimination scheme of NMS is selected to retain only cluster centers, while
discarding the remaining cluster members. The results presented in experiment
$\\#5$ show a large increase of $12.96\%$ mGMUE when compared to the full
system. We conclude that merging information from all cluster members into the
final object estimate is essential for proper quantification of bounding box
uncertainty by a neural network.

##  V Conclusion

This paper presents BayesOD, a Bayesian approach for estimating the
uncertainty in the output of deep object detector. Experiments using BayesOD
show that replacing NMS with Bayesian inference and explicitly incorporating
full aleatoric covariance matrix estimation allows for a much more meaningful
estimated category and bounding box uncertainty in deep object detectors. This
work aims to pave the path for future research directions that would use
BayesOD for active learning, exploration, as well as object tracking. Future
work will study the effect of informative priors originating from multiple
detectors, temporal information, and different sensors on the perception
capabilities of a robotic system.

## References

  * [1] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander.  Joint 3d proposal generation and object detection from view aggregation.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
  * [2] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [3] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia.  Multi-view 3d object detection network for autonomous driving.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [4] Trupti M Pandit, PM Jadhav, and AC Phadke.  Suspicious object detection in surveillance videos for security applications.  In Inventive Computation Technologies (ICICT), International Conference on, 2016.
  * [5] Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, et al.  The limits and potentials of deep learning for robotics.  The International Journal of Robotics Research, 37(4-5):405–420, 2018.
  * [6] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In International Conference on Machine Learning (ICML), 2016.
  * [7] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, 2017.
  * [8] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [9] Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhauf.  Dropout sampling for robust object detection in open-set conditions.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1–7. IEEE, 2018.
  * [10] Dimity Miller, Feras Dayoub, Michael Milford, and Niko Sünderhauf.  Evaluating merging strategies for sampling-based uncertainty techniques in object detection.  arXiv preprint arXiv:1809.06006, 2018.
  * [11] Gregory P Meyer, Ankit Laddha, Eric Kee, Carlos Vallespi-Gonzalez, and Carl K Wellington.  Lasernet: An efficient probabilistic 3d object detector for autonomous driving.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12677–12686, 2019.
  * [12] Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, and Xiangyu Zhang.  Bounding box regression with uncertainty for accurate object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2888–2897, 2019.
  * [13] Di Feng, Lars Rosenbaum, and Klaus Dietmayer.  Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection.  In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018.
  * [14] Di Feng, Lars Rosenbaum, and Klaus Dietmayer.  Leveraging heteroscedastic aleatoric uncertainties for robust real-time lidar 3d object detection.  In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018.
  * [15] Michael Truong Le, Frederik Diehl, Thomas Brunner, and Alois Knol.  Uncertainty estimation for deep neural object detectors in safety-critical applications.  In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2018.
  * [16] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving video database with scalable annotation tooling.  arXiv preprint arXiv:1805.04687, 2018.
  * [17] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy.  Speed/accuracy trade-offs for modern convolutional object detectors.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [18] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in Neural Information Processing Systems 28, 2015.
  * [19] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar.  Focal loss for dense object detection.  In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [20] David JC MacKay.  Probable networks and plausible predictions—a review of practical bayesian methods for supervised neural networks.  Network: computation in neural systems, 6(3):469–505, 1995.
  * [21] Murat Sensoy, Lance Kaplan, and Melih Kandemir.  Evidential deep learning to quantify classification uncertainty.  In Advances in Neural Information Processing Systems, pages 3179–3189, 2018.
  * [22] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6402–6413. Curran Associates, Inc., 2017.
  * [23] Florian Kraus and Klaus Dietmayer.  Uncertainty estimation in one-stage object detection.  arXiv preprint arXiv:1905.10296, 2019.
  * [24] Andrew Gelman, Hal S Stern, John B Carlin, David B Dunson, Aki Vehtari, and Donald B Rubin.  Bayesian data analysis.  Chapman and Hall/CRC, 2013.
  * [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [26] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [27] David Hall, Feras Dayoub, John Skinner, Peter Corke, Gustavo Carneiro, and Niko Sünderhauf.  Probability-based detection quality (PDQ): A probabilistic approach to detection evaluation.  CoRR, abs/1811.10800, 2018.


banner above paper title short description of paper

Mohammad Daniyal Shaiq FAST NUCES<EMAIL_ADDRESS>Musa Dildar Ahmed Cheema
FAST NUCES<EMAIL_ADDRESS>Ali Kamal FAST NUCES<EMAIL_ADDRESS>
# Transformer based Urdu Handwritten Text Optical Character Reader

###### Abstract

Extracting Handwritten text is one of the most important components of
digitizing information and making it available for large scale setting.
Handwriting Optical Character Reader (OCR) is a research problem in computer
vision and natural language processing computing, and a lot of work has been
done for English, but unfortunately, very little work has been done for low
resourced languages such as Urdu. Urdu language script is very difficult
because of its cursive nature and change of shape of characters based on it’s
relative position, therefore, a need arises to propose a model which can
understand complex features and generalize it for every kind of handwriting
style. In this work, we propose a transformer based Urdu Handwritten text
extraction model. As transformers have been very successful in Natural
Language Understanding task, we explore them further to understand complex
Urdu Handwriting.

###### keywords:

Transformers, Urdu Handwriting, OCR

##  1 Problem statement

In these current times, there has been an exponential growth in the adoption
of technology because of the promises it makes - easy and fast access to
information, accuracy, connectivity and security. Hence, there exists a need
to digitize information in order to preserve, and make better use of it. One
of the potential raw information which needs to be digitized, is handwritten
text (HWT), especially low resource text written in low resource languages
such as Urdu. Optical Character Reader (OCR) is one way to digitize printed
text, but recognizing handwritten text is a challenge because of various
different writing styles and other irregularities. There has been a
significant amount of work done for English Handwritten Text extraction,
however, the same cannot be said for low resource languages such as Urdu,
which in itself has a user base of 200 million people. Additionally, Urdu has
its largest user base in Pakistan, where countless businesses and other
processes still involve handwritten Urdu text. Therefore, there’s a need to
build such programs that can digitize Urdu Handwritten text, which can impact
the lives of about 200 million people. Urdu, unlike Latin languages, is
written in Arabic like script, hence it is more challenging because of
ligatures, joining of letters leading to change of shape, and a vast variety
of writing styles.

Figure 1: Challenges in extracting Urdu Script, identified by Anjum and Khan
[2020] Figure 2: Different characteristics of Urdu Script

##  2 Introduction

Urdu language has its second largest user base in the Indian Subcontinent,
amounting to about 200 million people. The Urdu language is written in Arabic
like script, Nastaleeq script. Urdu language has about 45 different
characters, and these characters are joined while writing which changes their
shape. This concept is known as ligatures. Urdu language has, in total, about
26,000 ligatures Ul-Hasan et al. [2013]. Due to this phenomenon, it makes it
exponentially difficult for OCR systems to extract characters from images.
Moreover, there are other complexities in Urdu script, such as lack of word
spacing, diagonal writing style, change of shape with context, and an
assortment of different styles. Considering these problems in Urdu script,
extracting printed Urdu text with a standardised style is a challenge for OCR
systems, let alone extracting Urdu Handwritten text. Handwritten Urdu text
adds more complexity to the text by adding noise, non standardised format,
more overlap, unequal spacing, inconsistent shapes etc. Arabic language, which
has a similar script to Urdu, has gained a lot of interest from the research
community, and in fact there are many commercial products for Arabic text, but
for Urdu there is only one commercially available OCR solution Ul-Hasan et al.
[2013], which works only for printed Urdu text, and so far there has been no
commercial product for Urdu Handwriting. Therefore, there exists a dire need
to propose a solution for extracting Urdu Handwriting, which can serve as the
pillar for commercial handwriting extraction tools. However, the problem for
Urdu Handwriting extraction is till in it’s early days, as there are only two
publicly available data sets - NUST-UHWD ul Sehr Zia et al. [2021] and UHWD.
Written Urdu language is widely used in the Indian subcontinent, as there’s
still a huge gap in digital penetration in the region. Many forms, cheques,
FIR documents, and many historical documents are present in written form,
alongside various business processes. Due to all this, overall process from
government processes to private processes are very slow, as there’s a need of
manual verification. Digitizing this problem will have a huge impact in the
lives of many and can help organizations in providing efficient and quick
service deliveries.

There are two forms of writing scripts available for languages, cursive and
non-cursive. Extraction of non-cursive handwritten text is easier as compared
to cursive as letters are isolated, i-e. they are not joined together, figure
1 illustrates it. Urdu language is written in cursive style, which makes it a
difficult problem. There has been work done for extracting cursive style text
such as Arabic, and there are many publicly available data sets for Arabic
language, but they cannot be used for Urdu, because despite Arabic and Urdu
being sister languages, which share an almost similar script, there exists a
difference in writing style in both the languages. Urdu is written in
Nastaleeq writing style and Arabic is written in Naskh writing style Malik
[2005]. Nastaleeq writing style is more complex than Naskh because it has
diagonality, filled loops, no fixed baseline, large variation of words/sub
words and false loop. Moreover, the number of ligatures and characters in Urdu
language are more than Arabic language. Therefore, the data set and OCR models
for Arabic language cannot certainly satisfy Urdu language. As of now,
collecting a large diverse Urdu Handwritten data set can be a huge
contribution In the handwritten text extraction problem, there are further two
divisions, Online Handwriting recognition and Offline Handwriting recognition.
Online Handwriting is the form of writing text on a digital pad using a
stylus, while on the contrary, Offline Handwriting is written on any piece of
paper. Recognizing Online Handwriting is easier, as pixel level information of
the handwriting is stored in the computer, which can help in identification of
text, while also removing the noise. On the other hand, Offline Handwriting is
written on paper, which has no other information apart from the image itself,
making it much more difficult for the computer to comprehend. Also, images
contain background noise, which adds another layer of difficulty to the
problem. We, in this research, worked on offline handwritten Urdu text, as
very little work is done in it. Our ultimate goal is digitization of Urdu
documents, forms, cheques etc. which come under the umbrella of Offline Urdu
Handwriting. Lastly, the most challenging aspect of Urdu Handwriting is change
of shape, with change of position or context. This is illustrated in the
figure 2.

##  3 Related work

In Ahmed et al. [2019], the authors study the problem of Urdu Handwriting
detection, and identify the issue faced in extraction of Urdu handwriting as
compared with other languages. Moreover, they address the problem of data
scarcity in offline Urdu Handwriting extraction, in the form of a data set
they collected, which handles many different handwriting styles, ligatures and
writing conditions. For the data set, they use red color for handwriting as it
prevents data loss during noise removal. The methodology used is that Urdu
text is passed as a scanned image, afterwards noise is removed and the image
is binarized using image processing techniques. Lastly, text lines are
segmented, and a Deep learning model extracts the information from each text
line segment. The problem of extraction is dealt as a Sequence to Sequence
problem, where a sequence of letters are generated from a an input image,
which is passed as a sequence. In the paper, a one directional BiLSTM is used
as it retains memory for a longer sequence as compared to RNNs, and
connectionist temporal classification loss (CTC) function is used as the loss
function. The model predicts character by character.

In Anjum and Khan [2020], the authors propose an attention based approach for
Urdu Handwriting extraction. In the paper, they argue that attention based
mechanisms have been successful in tasks such as machine translation, speech
recognition, and image captioning, and that attention mechanism helps extract
most relevant features for images. Therefore, in the Handwriting text
extraction problem, it can help extracting the relevant features and context
for predicting the right character. In Urdu, the shape of characters change
with respect to their position and shape, hence the attention based mechanism
is more relevant in that aspect. As far as methodology is concerned, the
authors deal with this as a sequence to sequence to problem, and the problem
is dealt with an encoder decoder architecture. Firstly, a convolutional neural
network is used to extract features from the Urdu Handwritten text images to
get a list of vector representation of the image. Afterwards, the vectors are
based into a Gated Recurrent Unit to predict the sequence of characters in the
image. In the Gated Recurrent Unit, the attention mechanism is used, where a
context vector is used to focus on the relevant part of the image. In this
mechanism, the character is predicted based upon the previous character, the
hidden state and the context vector. The context vector is also convolved with
a learnable filter to smooth out the histories in the context vector. This
entire part is the decoder part, and the other part is the encoder part, where
an input image is encoded to vectors that can be fed into a GRU. To do this,
the authors used DenseNet Convolutional Neural Network. In this specific
model, the convolution is applied to all previous layers, which enables
diverse learning in the model. As DenseNet model is compute intensive,
bottleneck layers (which are 1 X 1 convolutions) are used inside the DenseNet
model. Lastly, CTC loss function is used in the model, alongside word beam
search, which is used to choose the best word given an input (i-e. looks for
word with maximum probability). The authors collected their own data set with
the help of 100 writers to evaluate their model. They compared their model
with non attention based encoder decoder model, and their model ended up
outperforming all the non attention based models.

In Mushtaq et al. [2021], the authors firstly collect an Urdu Handwritten text
data set. In the paper, the authors explain the 3 entire data preprocessing
phase for Urdu Handwriting extraction. The first phase is segmenting text by
drawing bounding boxes on the handwritten text in the images. Afterwards, the
segmented text is normalized, binarized, and denoized for text extraction.
Moreover, the problem of text extraction is dealt as a classification problem,
where the input is passed through a convolutional neural network, and in the
last layer, there is a Fully connected layer which predicts characters in the
image.

In ul Sehr Zia et al. [2021], the authors propose an unconstrained dataset for
Urdu handwriting that covers more than 73Urdu ligatures, NUST-UHW. Moreover,
the authors also propose an encoder decoder architecture, which deals with
this problem as sequence to sequence problem. In the paper, the authors argue
that Urdu Handwritten text contains very complex features that are crucial for
predicting the characters in the image, therefore low resolution images cannot
perform really well. To solve this issue, the author uses high resolution
images for Urdu Handwritten text. The images are first encoded into vectors by
passing it into a Convolutional Neural Network. In the CNN, the high
resolution images are needed to be brought down to low resolution. Max pool
layers are used to achieve this. The issue with Max pool layers is that it can
lead to loss of data from the image. To address this issue, the authors
concatenate the output in the last layers, instead of Max pooling. This saves
information in the last layers as the last layers in CNN hold deeper
information about the image. Afterwards, the output from CNN is passed into
multiple BiLSTM layers, where again vectors are concatenated. Moreover, an
n-gram model is used to correct the language of the output from the BiLSTM
layer. For the loss, CTC loss function is used with beam search for predicting
the correct word. Also, to diversify the data during training time, a random
noise layer is added before the CNN, to add random noise to the image so that
at every training step a different image is sent to the model.

##  4 Methodology

###  4.1 Dataset

In this research work, we use the Dataset collected by Punjab University
College of Information technology titled as PUCIT - Offline Urdu Handwritten
Urdu Lines DatasetAnjum and Khan [2020]. The dataset is a multiple line
written dataset written by 100 writers. In the dataset, text lines from the
long written paragraphs. The dataset considers the slant nature of writing
style in Urdu language. The dataset contains 7309 Urdu text lines and 78,870
Urdu words. Lastly, the dataset is available for only non-commercial academic
purpose

###  4.2 Preprocessing

The images in the dataset were not normalized, there were images with
different sizes and channels. We standardised it by applying image processing
techniques. Moreover, we also applied some image binarization techniques for
better results.

Figure 3: Resnet-Transformer Model Architecture for Urdu Handwritting
Extraction

###  4.3 Procedure

We addressed this problem as a sequence to sequence problem, just like in the
previous studies, but instead of using LSTMS or RNN, we used Transformer
architecture Vaswani et al. [2017] for predicting the text in the image.
Transformer architecture has been popular in the NLP domain as it was able to
provide state of the art results in NLP tasks, such as Machine Translation,
Question Answering, Named Entity Recognition, Language modeling etc.
Transformers are quite good in understanding inter language dependencies, and
it also learns a language model for output text. In the transformer
architecture, there are two blocks, encoder and decoder. In the encoder part,
there’s a self attention mechanism, which focuses on different features of the
input for every segment of the input image, whereas in the decoder part there
is causal attention mechanism, that only looks into the previous tokens. There
also exists attention mechanism between the encoder and decoder. These
attention mechanisms can help handwriting problem by focusing on important
part of the image for predicting the characters, as Urdu characters are
context dependent. Moreover, the language modeling capability of Transformers
can help in correct language prediction in the same pass as text extraction.
There’s no need of separate language model. Moreover, RNNS are not
parallelizable, but transformers are parallelizable, which can make training
large models scalable. Before passing an input image to a transformer, it
needs to be converted into a sequence of vectors as it is required by the
transformer architecture, so we used ResNetHe et al. [2015] 18 architecture
for extracting high dimensional features from the input image. As Urdu is a
complex language, there is a need for extracting complex features, for which
ResNet does a good job, as in ResNet there are residual blocks, which retain
the information of previous layers as we go deep into the network, so
information is not lost. The problem is dealt as a classification problem,
where a sequence of characters will be predicted from the Transformer layer.

In this approach, we first take an image, and pass it through a Resnet18 model
to extract important features from the image. Afterwards, we pass the output
of the Resnet18 model into Linear layers (we tried different combinations) to
get a vectorr (embeddings) representation of the images. We then passed these
representations into the transformer model. The transformer model predicts
character by character. We trained the model with masking the target. Resnet18
model was pretrained on imagenet, which was finetuned during the training
process, while on the other hand transformer was built from scratch without
any pretrained weights.

Figure 4: Sample Input and ground truth for training of dataset by Anjum and
Khan [2020]

##  5 Evaluation and Experiments

For the evaluation, we used character error rate (CER) between input and
output text. It is illustrated below.

$CER=\frac{ins+sub+del}{n}\times 100$ (1)

It is the number of insertions, substitutions, and deletions required to
transform output text into the target label. While n is the number of
characters in the target label. This equation can also be used to calculate
word error rate. The model will be considered good if it achieves low CER or
WER.

The model was ran on a machine with the following specifications:

Processor: AMD Ryzen 5 3600 @4.2 GHz 6 Cores/12 Threads
RAM: 32GB DDR4 3200 MHz
GPU: Nvidia GeForce RTX 3070 Ti 8GB

Despite transformers showing great results in Natural Language Understanding,
there exists one limitation, that it needs huge amounts of data to train.
Unfortunately, that is the limitation of our research, as there is a
substantial lack of availability of large Urdu handwritten text datasets. We
trained our model for around 12 hours, however, it didn’t converge. Due to
very limited data, we achieved more than 85% error rate. In our research, we
were able to explore transformer architecture for handwritten text extraction,
but due to a lack of large datasets, we were not able to achieve good CER.
Moreover, we trained the transformer on character level, due to which output
sequence had a huge length, which may be the reason for not fitting well on
the data. In the future, we can test on pretrained models, where we can use a
pretrained Urdu Language decoder, and train our image feature extractor for
better results. Lastly, our research has opened pathways for researchers to
work on transformer based OCR for low resource languages, such as Urdu.

Firstly, we would like to thank Almighty Allah (S.W.T) for granting us the
strength to work on this project. Moreover, we would like to thank the
contributors to the Urdu datasets that we employed the use of. Lastly, we
would like thank our instructor, Dr. Mirza Omer Beg for giving us the
opportunity to work on this problem, while making us stay motivated throughout
the entire semester.

## References

  * Ahmed et al. [2019] S. Ahmed, S. Naz, S. Swati, and M. Razzak.  Handwritten urdu character recognition using 1-dimensional blstm classifier.  _Neural Computing and Applications_ , 31, 04 2019.  10.1007/s00521-017-3146-x.
  * Ali et al. [2022] R. Ali, U. Farooq, U. Arshad, W. Shahzad, and M. O. Beg.  Hate speech detection on twitter using transfer learning.  _Computer Speech & Language_, 74:101365, 2022.
  * Alvi et al. [2017] H. M. Alvi, H. Sahar, A. A. Bangash, and M. O. Beg.  Ensights: A tool for energy aware software development.  In _2017 13th International Conference on Emerging Technologies (ICET)_ , pages 1–6. IEEE, 2017.
  * Alvi et al. [2021] H. M. Alvi, H. Majeed, H. Mujtaba, and M. O. Beg.  Mlee: Method level energy estimation—a machine learning approach.  _Sustainable Computing: Informatics and Systems_ , 32:100594, 2021.
  * Anjum and Khan [2020] T. Anjum and N. Khan.  An attention based method for offline handwritten Urdu text recognition.  In _International Conference on Frontiers in Handwriting Recognition (ICFHR)_ , September 2020.
  * Anwar and Baig [2020] T. Anwar and O. Baig.  Tac at semeval-2020 task 12: Ensembling approach for multilingual offensive language identification in social media.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 2177–2182, 2020.
  * Arshad et al. [2019] M. U. Arshad, M. F. Bashir, A. Majeed, W. Shahzad, and M. O. Beg.  Corpus for emotion detection on roman urdu.  In _2019 22nd International Multitopic Conference (INMIC)_ , pages 1–6. IEEE, 2019.
  * Asad et al. [2020] M. Asad, M. Asim, T. Javed, M. O. Beg, H. Mujtaba, and S. Abbas.  Deepdetect: detection of distributed denial of service attacks using deep learning.  _The Computer Journal_ , 63(7):983–994, 2020\.
  * Awan and Beg [2021] M. N. Awan and M. O. Beg.  Top-rank: a topicalpostionrank for extraction and classification of keyphrases in text.  _Computer Speech & Language_, 65:101–116, 2021.
  * Bangash et al. [2017] A. A. Bangash, H. Sahar, and M. O. Beg.  A methodology for relating software structure with energy consumption.  In _2017 IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , pages 111–120. IEEE, 2017.
  * Bashir et al. [2022] M. F. Bashir, A. R. Javed, M. U. Arshad, T. R. Gadekallu, W. Shahzad, and M. O. Beg.  Context aware emotion detection from low resource urdu language using deep neural network.  _Transactions on Asian and Low-Resource Language Information Processing_ , 2022.
  * Beg [2008] M. Beg.  Critical path heuristic for automatic parallelization.  _University of Waterloo, David R. Cheriton School of Computer Science, Technical Report CS-2008-16_ , 2008.
  * Beg and Beek [2013] M. Beg and P. v. Beek.  A constraint programming approach for integrated spatial and temporal scheduling for clustered architectures.  _ACM Transactions on Embedded Computing Systems (TECS)_ , 13(1):1–23, 2013.
  * Beg and Dahlin [2001] M. Beg and M. Dahlin.  A memory accounting interface for the java programming language.  _Technical Report CS-TR-01–40, University of Texas at Austin_ , 2001\.
  * Beg and Van Beek [2010] M. Beg and P. Van Beek.  A graph theoretic approach to cache-conscious placement of data for direct mapped caches.  In _Proceedings of the 2010 international symposium on Memory management_ , pages 113–120, 2010.
  * Beg and Van Beek [2011] M. Beg and P. Van Beek.  A constraint programming approach for instruction assignment.  In _2011 15th Workshop on Interaction between Compilers and Computer Architectures_ , pages 25–34. IEEE, 2011.
  * Beg [2013] M. O. Beg.  Combinatorial problems in compiler optimization.  2013\.
  * Beg et al. [2019] M. O. Beg, M. N. Awan, and S. S. Ali.  Algorithmic machine learning for prediction of stock prices.  In _FinTech as a Disruptive Technology for Financial Institutions_ , pages 142–169. IGI Global, 2019.
  * Dilawar et al. [2018] N. Dilawar, H. Majeed, M. O. Beg, N. Ejaz, K. Muhammad, I. Mehmood, and Y. Nam.  Understanding citizen issues through reviews: A step towards data informed planning in smart cities.  _Applied Sciences_ , 8(9):1589, 2018.
  * Farooq et al. [2019a] M. U. Farooq, M. O. Beg, et al.  Bigdata analysis of stack overflow for energy consumption of android framework.  In _2019 International Conference on Innovative Computing (ICIC)_ , pages 1–9. IEEE, 2019a.
  * Farooq et al. [2019b] M. U. Farooq, S. U. R. Khan, and M. O. Beg.  Melta: A method level energy estimation technique for android development.  In _2019 International Conference on Innovative Computing (ICIC)_ , pages 1–10. IEEE, 2019b.
  * He et al. [2015] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  _CoRR_ , abs/1512.03385, 2015.  URL http://arxiv.org/abs/1512.03385.
  * Ismail et al. [2021] S. Ismail, H. Mujtaba, and M. O. Beg.  Spems: A sustainable parasitic energy management system for smart homes.  _Energy and Buildings_ , 252:111429, 2021.
  * Javed et al. [2020a] A. R. Javed, M. O. Beg, M. Asim, T. Baker, and A. H. Al-Bayatti.  Alphalogger: Detecting motion-based side-channel attack using smartphone keystrokes.  _Journal of Ambient Intelligence and Humanized Computing_ , pages 1–14, 2020a.
  * Javed et al. [2020b] A. R. Javed, M. U. Sarwar, M. O. Beg, M. Asim, T. Baker, and H. Tawfik.  A collaborative healthcare framework for shared healthcare plan with ambient intelligence.  _Human-centric Computing and Information Sciences_ , 10(1):1–21, 2020b.
  * Javed et al. [2019] H. T. Javed, M. O. Beg, H. Mujtaba, H. Majeed, and M. Asim.  Fairness in real-time energy pricing for smart grid using unsupervised learning.  _The Computer Journal_ , 62(3):414–429, 2019\.
  * Javed et al. [2021] M. S. Javed, H. Majeed, H. Mujtaba, and M. O. Beg.  Fake reviews classification using deep learning ensemble of shallow convolutions.  _Journal of Computational Social Science_ , 4(2):883–902, 2021.
  * Karsten et al. [2007] M. Karsten, S. Keshav, S. Prasad, and M. Beg.  An axiomatic basis for communication.  _ACM SIGCOMM Computer Communication Review_ , 37(4):217–228, 2007.
  * Khawaja et al. [2018] H. S. Khawaja, M. O. Beg, and S. Qamar.  Domain specific emotion lexicon expansion.  In _2018 14th International Conference on Emerging Technologies (ICET)_ , pages 1–5. IEEE, 2018.
  * Majeed et al. [2020] A. Majeed, H. Mujtaba, and M. O. Beg.  Emotion detection in roman urdu text using machine learning.  In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering Workshops_ , pages 125–130, 2020.
  * Majeed et al. [2021] H. Majeed, A. Wali, and M. Beg.  Optimizing genetic programming by exploiting semantic impact of sub trees.  _Swarm and Evolutionary Computation_ , 65:100923, 2021.
  * Malik [2005] S. K. Malik.  Sa,“urdu online handwriting recognition”, emerging technologies, 2005\.  In _Proceedings of the IEEE Symposium on Volume, Issue_ , pages 17–18, 2005.
  * Mushtaq et al. [2021] F. Mushtaq, M. M. Misgar, M. Kumar, and S. S. Khurana.  Urdudeepnet: Offline handwritten urdu character recognition using deep neural network.  _Neural Comput. Appl._ , 33(22):15229–15252, nov 2021.  ISSN 0941-0643.  10.1007/s00521-021-06144-x.  URL https://doi.org/10.1007/s00521-021-06144-x.
  * Naeem et al. [2020a] B. Naeem, A. Khan, M. O. Beg, and H. Mujtaba.  A deep learning framework for clickbait detection on social area network using natural language cues.  _Journal of Computational Social Science_ , pages 1–13, 2020a.
  * Naeem et al. [2020b] S. Naeem, M. Iqbal, M. Saqib, M. Saad, M. S. Raza, Z. Ali, N. Akhtar, M. O. Beg, W. Shahzad, and M. U. Arshad.  Subspace gaussian mixture model for continuous urdu speech recognition using kaldi.  In _2020 14th International Conference on Open Source Systems and Technologies (ICOSST)_ , pages 1–7. IEEE, 2020b.
  * Qamar et al. [2021] S. Qamar, H. Mujtaba, H. Majeed, and M. O. Beg.  Relationship identification between conversational agents using emotion analysis.  _Cognitive Computation_ , pages 1–15, 2021.
  * Sahar et al. [2019] H. Sahar, A. A. Bangash, and M. O. Beg.  Towards energy aware object-oriented development of android applications.  _Sustainable Computing: Informatics and Systems_ , 21:28–46, 2019.
  * Tariq et al. [2019] M. Tariq, H. Majeed, M. O. Beg, F. A. Khan, and A. Derhab.  Accurate detection of sitting posture activities in a secure iot based assisted living environment.  _Future Generation Computer Systems_ , 92:745–757, 2019\.
  * Ul-Hasan et al. [2013] A. Ul-Hasan, S. B. Ahmed, F. Rashid, F. Shafait, and T. M. Breuel.  Offline printed urdu nastaleeq script recognition with bidirectional lstm networks.  In _Proceedings of the 2013 12th International Conference on Document Analysis and Recognition_ , ICDAR ’13, page 1061–1065, USA, 2013. IEEE Computer Society.  ISBN 9780769549996.  10.1109/ICDAR.2013.212.  URL https://doi.org/10.1109/ICDAR.2013.212.
  * ul Sehr Zia et al. [2021] N. ul Sehr Zia, M. F. Naeem, S. M. K. Raza, M. M. Khan, A. Ul-Hasan, and F. Shafait.  A convolutional recursive deep architecture for unconstrained urdu handwriting recognition.  _Neural Computing and Applications_ , 34(2):1635–1648, sep 2021.  10.1007/s00521-021-06498-2.  URL https://doi.org/10.1007.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  _CoRR_ , abs/1706.03762, 2017.  URL http://arxiv.org/abs/1706.03762.
  * Zafar et al. [2018] A. Zafar, H. Mujtaba, M. O. Beg, and S. Ali.  Deceptive level generator.  In _AIIDE Workshops_ , 2018.
  * Zafar et al. [2019a] A. Zafar, H. Mujtaba, S. Ashiq, and M. O. Beg.  A constructive approach for general video game level generation.  In _2019 11th Computer Science and Electronic Engineering (CEEC)_ , pages 102–107. IEEE, 2019a.
  * Zafar et al. [2019b] A. Zafar, H. Mujtaba, M. T. Baig, and M. O. Beg.  Using patterns as objectives for general video game level generation.  _ICGA Journal_ , 41(2):66–77, 2019b.
  * Zafar et al. [2020] A. Zafar, H. Mujtaba, and M. O. Beg.  Search-based procedural content generation for gvg-lg.  _Applied Soft Computing_ , 86:105909, 2020.