# Offline handwritten mathematical symbol recognition utilising deep learning

Azadeh Nazemi1, Niloofar Tavakolian2, Donal Fitzpatrick1, Chandrika Fernando3,
Ching Y. Suen4
1School of Computing, Dublin City University, 2Department of Computer Science
and Engineering and Information Technology, Shiraz University 3 School of
Electrical Computer and Mathematics, Curtin University 4Centre for Pattern
Recognition and Machine Intelligence, Concordia University

###### Abstract

This paper describes an approach for offline recognition of handwritten
mathematical symbols. The process of symbol recognition in this paper includes
symbol segmentation and accurate classification for over 300 classes. Many
multidimensional mathematical symbols need both horizontal and vertical
projection to be segmented. However, some symbols do not permit to be
projected and stop segmentation, such as the root symbol. Besides, many
mathematical symbols are structurally similar, specifically in handwritten
such as 0 and null. There are more than 300 Mathematical symbols. Therefore,
designing an accurate classifier for more than 300 classes is required. This
paper initially addresses the issue regarding segmentation using “Simple
Linear Iterative Clustering”(SLIC). Experimental results indicate that the
accuracy of the designed kNN classifier is 84% for salient, 57% Histogram of
Oriented Gradient (HOG), 53% for Linear Binary Pattern (LBP) and finally 43%
for pixel intensity of raw image for 66 classes. 87 classes using modified
LeNet represents 90% accuracy. Finally, for 101 classes, SqueezeNet achieved
90% accuracy using pre-trained one.

###### Index Terms:

Super pixel, Deep learning , Hyper parameters ,SqueezeNet , Handwritten
symbols

##  I Introduction

Manyresearchers have worked on Handwritten Mathematical Expression (HME)
recognition for many years such as Anderson’s effort in the
’60s[1][2]Generally, handwritten recognition based on the applied method has
two categories: Offline and Online. Many devices, such as Personal Digital
Assistants(PDAs), tablets, Personal Computers( PCs), and electronic
whiteboards, require online handwriting recognition. However, offline
handwritten recognition is used for the scanned documents; it is less
appealing than online. Therefore, at present little research is published on
offline handwriting. In contrast, there are many comprehensive approaches have
been applied for online HME. Competition on Recognition of Online Handwritten
Mathematical Expressions (CROHME) 2016 concluded that the recognition of HMEs
was still a challenge after six years of competition[3] The online method
works well for connected strokes or cursive strokes[4], while the offline way
can overcome the problem of out-of-order strokes or duplicated strokes using
contextual information[5][6]. Online HME recognition can improve by tracing
some points of symbols from starting to ending as ink Markup Language (inkML)
format indicates. Nevertheless, offline HME recognition involves two-
dimensional layouts, subscript positioning, variable-sized characters, and
unusual maths operators depending on the area of mathematics. In addition to
the variation in handwriting styles, the media such as pen and paper also may
lead to more challenges[7]. The issue regarding mathematical symbol
recognition is not limited to symbol classification, and It requires
interpreting the mathematical expression. Interpretation of mathematical
expression is performed based on considering its structures and mathematical
area even in printed mathematical expression recognition In the recent decade,
deep learning is the state-of-the-art for many machine learning applications
such as classification, detection, identification, and many more. Among
several recognition methods, deep learning with multiple layers of nonlinear
information processing is an inevitable trend. These approaches automatically
learn and solve problems without using any prior knowledge[8]. The results of
recent researchers studies prove that Deep neural networks improve online
mathematical recognition symbols comparing previous methods like Modified
Quadratic Discriminant Functions (MQDFs) [9] with Bidirectional Long Short-
term Memory (BLSTM) and Markov Random Fields (MRFs)[10][11]. However, there is
still a lot of room for researchers to explore. To design and train a deep
network, large scale datasets are required. This issue in many classification
cases can be solved utilising augmentation. In this study, augmentation is not
applicable due to the extreme similarity of some symbols. Besides HME
recognition has common problems with the retrieval and recognition of complex
structures [12][13]. Based on the research for offline handwritten recognition
using Neural Network(NN)[7] identification and verification of off-line
handwritten signatures from images is a difficult problem due to dependency of
signature as bio-metrics factor on psychological factors of the
individual[14]. HME also face the same issue for recognition. This research
performs symbol segmentation with Simple Linear Iterative Clustering
(SLIC)[15]. Recognition uses a modified LeNet network and of pre-trained
SqueezeNet. For the purpose of comparison and evaluation of the proposed
method, some former methods such as Histogram of Oriented Gradient (HOG),
Linear Binary Pattern (LBP) are also implemented. Deep learning starts using a
simple network by only tree fully connected layers and handwritten digits of
the MNIST dataset[16]. Then the MNIST dataset is upgraded to alphanumeric
dataset (66 classes) and network is changed to LeNet. At this point, some
network hyperparameters are modified to increase accuracy and control loss. In
the next state carefully the number of classes is increased up to 87 by adding
21 mathematical symbols and performing supervised and limited augmentation. In
addition, the network is modified to have three convolution layers, three
Maxpooling as well as three fully connected layers. The last stage of using
deep learning is transferring network to SqueeezeNet, training 101 available
mathematical symbol classes applying transfer learning to prevent overfitting
in the lack of sufficient data[17]. The rest of this paper has been organised
as follows. Section II includes related work which is the basis of this
research. Section III contains the methodology followed by the experimental
arech reviewed. For classifications purposes feature based methods such as
Linear Binary Pattern(LBP)[18] and Histogram of Oriented Gradient(HOG) are
studied. Besides two other approaches based on intensity and salience are
considered for comparison.

###  I-A Limitations of symbol recognition

Symbol recognition process, regardless of being online or offline, has some
limitations that do not allow achieving high accuracy. These limitations are
of two types as follows

  * •

Type I: Symbols with different meanings and similar shapes such as: 9q gq bh
GC KR DO UV QO B8 Z2 S5 I1 Uv.

  * •

Type II: Upper and lower case symbols with same shapes such as: nN fF wW zZ xX
yY cC uU sS kK mM pP vV

However, in some cases online HMEs can be properly recognised due to tracking
of coordinate values from start to stop. Mis-recognition occurring in Group I
have stronger negative impacts on the process of Mathematical Expression
Retrieval (MER). The only avenue to accurately recognise these symbols is
concept analysis performed after symbol recognition considering pre and post-
located symbols.

###  I-B Segmentation

In order to obtain accurate results from mathematical symbol classification, a
multi-step segmentation module is required. These steps include block
segmentation, line segmentation, line classification as thress classes of
text-only, embedded maths and maths-only classes thr word segmentation, and
symbol segmentation. Line classification is to classify mathematical multi-
dimensional lines, linear text lines,and mixed lines. In 2016, Nazemi’s thesis
proposed Support Vector Machine (SVM) for line classification[19]. Symbol
segmentation contains two sub-modules, namely, Vertical Symbol
Segmentation(VSS) and Horizontal Symbol Segmentation(HSS). In complex
mathematical expressions, it requires that HSS and VSS recursively be
performed to extract symbols as the primary component of a mathematics. HSS
and VSS are failed for extraction some mathematical symbols such as root sign.

Figure 1. shows steps of recursive HSS VSS for sample expression

Figure 1: recursive HSS VSS for sample expression

###  I-C Symbol categorisation

Symbol categorisation has been performed based on symbol’s aspect ratio. A
symbol aspect ratio is the ratio of height to width and leads to the following
Symbol categorisation

  * •

if Aspect-Ratio is greater than 1.3 then symbol is Tall such as $\prod$
,$\sum$,$\int$

  * •

if Aspect-Ratio is between 0.58 and 1.7 then symbol is Square such as ’exist’
sign ,’for all’ sign and equivalent sign

  * •

if Aspect-Ratio is less than 0.76 then symbol is Short such as ¡¿- $\infty$

The categorisation results for 258 different symbols considering their aspect
ratio were 43 “Tall”, 206 “Square” and 18 “Short”. Then each symbol was re-
scaled to a predefined size as follows:
the “Tall” symbol is resized to 16x28,
the “Square” symbol is converted to 28x28, and
the “Short” symbol is converted to 28x16 when actual size is defined in format
W x H.
categorisation performs to implement the more accurate classification system
and reduce computational cost

###  I-D Classifiers based on HOG, LBP, Salient, and Intensity

For pixel intensity if symbol size is 28x28 then the feature vector is 784. In
the Salient approach, if symbol is 28x28 the saliency map is a binary image
with size of 128x28[20][21]

For LBP, if the symbol size is 28x28 and pixels per cell= 3x3, then the number
of cells will be 9x9. Therefore, the size of feature vector is 81. The
following snippet shows LBP calculation:

$p_{11}\quad p_{12}\quad p_{13}$ $p_{21}\quad p_{22}\quad p_{23}$ $p_{31}\quad
p_{32}\quad\ p_{33}$

For LBP convert symbol image (28x28) to (3x3)
pattern=”
for i=1 ; i¡=3; i++
for j=1 ;j¡=3;j++
if $p_{22}>p_{ij}$
pattern=0pattern
else
pattern=1pattern

For HOG if the symbol size is 28x28 and pixels per cell= 9x9, then the number
of cells will be 3x3.Then value of orientation and cell per blocks determine
the size of feature vector. Supposing the orientation=9 and cell per block=4,
then the size of feature vector is 9x4=36. In the MNISTdataset of zero to nine
handwritten digits, the plainest method for designing a classifier is
converting each symbol image to the vector of intensity and using k-nearest
neighbour. It has an accuracy of 96.88% . Although this method has a
reasonable accuracy for a small number of classes, it canbe expected to keep
the accuracy high when the number of classes is more than 300 due to the
limitations of symbol recognition. To obtain most discriminators features, HOG
or LBP can be used. During implementation of feature extraction in this
classifier, a restriction was observed due to disability to extract such
features from symbols. To address this issue, a morphology technique was
performed to change the boldness of symbols[22] Figure 2 illustrates some
symbols before and after morphology dilation.

Figure 2: Applying morphology technique to change the boldness of symbols

In order to implement classifiers, this research assembled a dataset of alpha
numerical handwritten symbols (66 symbols) together with 21 mathematical
symbols belonging to three different categories, namely,‘short’, ‘tall’ and
‘square’.The results pixel intensity,salient, features extracted using HOG and
features extracted using LBP are presented for comparison.The results indicate
that the accuracy using kNN classifier is 84% for salient, 57% for HOG, 53%
for LBP and finally 43% for pixel intensity for the 87 classes of mathematical
handwritten symbols. The following figures (Figures 3,4) illustrate the
results of t-distributed Stochastic Neighbours Embedding $t_{SNE}$ scatter to
visualise high-level representations obtained with intensity (raw image), and
salience $t_{SNE}$illustrations confirmed the obtained accuracy for two
methods.

Figure 3: Salient visualisation for 10 random classes Figure 4: Pixel
intensity visualisation for 10 random classes

Fig 3 illustrates the inner class similarity of menders is more than their
similarities to other classes (for saliency map). Fig 4 illustrates How the
similarity of members of two different classes are more than the members inner
class similarity(for intensity vectors)

###  I-E Used dataset

The dataset used in this study was made of MNIST[10], 6000 images from inkML
of the CROHME dataset converted to PNG by the developed converter in this
research, 2000 images from the set of Handwritten Digit Images published by
Computer Vision Group of the University of São Paulo and 1000 handwritten
character symbols from the Chars74K dataset

##  II Proposed method

This section initially presents the proposed method in this study for symbol
segmentation followed by a description of all steps of utilising and modifying
designed convolutional network for classification[23][24] over 100 classes in
detail.

###  II-A Segmentation with SLIC

Simple Linear Iterative Clustering(SLIC) Superpixel is a group of connected
pixels with similar colours or gray levels. Superpixel segmentation divides an
image into hundreds of non-overlapping superpixels. For the first time, in
2003, and Malik used superpixels to do image segmentation. Superpixels provide
access to meaningful regions for accurate feature extraction and reduce the
input entities for further processing. Superpixel segmentation is utilised for
semantic segmentation, visual tracking, image classification, and many
computer vision applications. SLIC is an approach that utilises clustering for
superpixel segmentation by combining five-dimensional colours and image plane
space. It is a method to replace multi-step pre-processing for segmentation.
SLIC can solely perform the role of all of the steps stated above to obtain
segmented symbols. The SLIC method can address the issues regarding
segmentation of some symbols such as binding root as Figure 5 illustrates.

Figure 5: SLIC segmentation

###  II-B Classification

In this subsection, two different architectures are described as the
approaches for mathematical handwritten symbol recognition. Data augmentation
means increasing the size of training dataset by changing color channels,
adding noise, applying random regional sharpening filters, and adding mean-
images based on clustering techniques efficiently. Most recent augmentations
are centered on using the Generative Adversarial Network (GAN) model,
sometimes swapping the generator network with a genetic algorithm. However, to
perform augmentation on a dataset of handwritten mathematical symbols, it must
be considered that augmentation does not change the symbol meaning, for
example,$\cup$ when is horizontally flipped, it is converted to $\cap$.
Therefore, augmentation techniques cannot be run on all symbols. In the
result, this research faces lack of sufficient data for deep learning. On the
other hand, due to regularising effect of augmentation, by increasing the
usage of augmented samples, the possibility of network under fitting
increases. Thus, data augmentation is not always efficient and is not an ideal
approach to address the lack of sufficient training sample for deep
convolutional network. Figure 6 illustrates applying shift augmentation
techniques which is almost safe for handwritten mathematical symbol
recognition.

Figure 6: Applying shift augmentation techniques.

Considering the above mentioned obstacles in the design of a deep network to
classify handwritten mathematical symbols, this research started with
networking from a very simple network with only 10 classes using MNIST large
scale dataset for digits only and having sufficient samples. Since training
dataset was enough and only 10 classes need to recognise in this case, even
the network with only three fully connected layers (2 RelU and 1 Softmax) and
without any convolutional layer, provided at least 89% accuracy. The empirical
results of the research demonstrated that the transmission of the network from
a very simple network to LeNet with seven layers including convolutional
layers, even with changing MNIST with 10 classes to alphanumeric dataset still
can maintain high accuracy. Alphanumeric dataset contains the numbers 0 - 9,
the letters A - Z (both uppercase and lowercase), and some common symbols such
as sharp sign,ampere sign, @, and *. Totally, there are 66 alphanumeric case
sensitive symbols. After segmentation, the next major objective for this study
is symbol classification. As long as the number of classes is less than 40,
symbol image saliency map followed by kNN classifier, have reasonable
classification results. Moreover, modification of some conevolutional networks
with a few layers such as LeNet, makes the system efficient for less than 40
classes. The most prominent challenge for classification of the handwritten
mathematical symbols is the real world nature of the training samples. In some
cases, the extracted descriptor from the last layer of the network proves
inner class discrimination more than the outer class. Another issue to design
a neural network to meet the requirement for specific task classification is
the lack of sufficient training samples and imbalance training data sets with
different number of samples in each class. To address the first issue, there
are two avenues, one is augmentation and the other is Transfer Learning
(TL)[25] which means the weights of the pretained networks with the large
scale training data. Making the decision to use one of these two approaches
fairly depends upon the nature of training sample. To find out most
appropriate network architecture to classify handwritten mathematical symbols,
this research initially visualise the output of the last convolution layer for
some networks such as VGG16, LeNet, SqueezeNet applied on some printed
symbols. The results indicated that the VGG networks were not suitable for
this purpose. However, LeNet and SqueezNet had better functionality.
Experimental result of this comparison is presented in Figure 7 .

Figure 7: VGG,LeNet, SqueezeNet comparison.

Gradient-weighted Class Activation Mapping (Grad-CAM) technique used for this
visualisation and includes following steps.

  * •

Picking final convolutional feature map

  * •

Considering a sample image

  * •

Predicting its class by the network

  * •

Calculating the gradient of the class noting width height and colour channels

  * •

Updating the weights of each channel in the sample feature map using class
gradient

[26]

###  II-C LeNet

The proposed network architecture, LeNet, for deep learning for classification
of alphanumeric symbols originally used tangent hyperbolic (tanH) as the
activation function. Considering the number of classes, this research used
rectified linear unit (RelU) to improve classification accuracy. In the
following equation: O=Output, W=Input, F=Filter Size, P=Padding, and S=Stride

$O=\frac{(W-F+2P)}{S}+1$ (1)

The starting image size was 32x32 with filter size 5x5, Stride 2 and padding
was equal to zero. Equation 1 was used to obtain output size by initial size,
filter, stride and padding. Based on Equation 1, the size of the next layer
was calculated as 28x28. In the next setup, for the convolutional layer, the
filter size is 2x2, the padding is zero and stride was two which lead to a
size of 14x14. Generally, the LeNet architecture contains two sets of
convolutional layers, activation, and pooling, followed by two fully connected
activation and end up to classifier function such as Softmax. These sequential
network included input, conv1, RelU, pool1, conv2, RelU, pool2, FC, RelU, FC,
Softmax, and Output respectively. This statement is further illustrated in
Figure 8 and Table I in detail.

Figure 8: The original LeNet used for classifying 66 classes. TABLE I: Seven layers of LeNet Layers | Description
---|---
Input | Size=32x32
Convolutional 1 | I=32x32 P=0 F=5 O=28x28 Feature=6
Activation | RelU / tanH
Pooling 1 | I=28x28S=2 P=0 F=2 O=14x14
Convolutional 2 | I=14x14 S=1 P=0 F=5 O=10x10 Feature=6
Activation | RelU / tanH
Pooling 2 | I=12x12 S=2 P=0 F=2 O=5x5
Fully connected | Input=400
Fully connected | Input=200
Loss function | Sigmod/Softmax
Output | Size= 66 class

Figure 9 and Table II both describe three layers insertion to LeNet(modified
LeNet) for increasing accuracy.

Figure 9: The modified LeNet for classifying 87 classes

Table III compares different performance based on different hyper parameters
in terms of Accuracy.

TABLE II: Modified ten layers of LeNet Layers | Description
---|---
Input | Size= 32x32
Convolution 1 | I=32x32 S=1 P=2 F=9 O=28x28 Feature=6
Activation | RelU / tanH
Pooling 1 | I=28x28 S=2 P=2 F=2 O=16x16
Convolution 2 | I=16x16 S=1 P=2 F=9 O=12x12 Feature=16
Activation | RelU / tanH
Pooling 2 | I=12x12 S =2 P=2 F=2 O=8x8
Convolution 3 | I=8x8 S=1 P=2 F=5 O=8x8 Feature=16
Activation | RelU / tanH
Pooling 3 | I=8x8 S=2 P=2 F=2 O=6x6
Fully connected | Input=576
Fully connected | Input=288
Fully connected | Input=144
Loss function | Softmax
Output | size=87 Classes
TABLE III: LeNet/Modified and Hyper parameters F(Activation) | F(Loss) | F(Optimisation) | LR | Accuracy | Classes
---|---|---|---|---|---
tanH | Sigmoid | SGD | 0.01 | 88% | 66
RelU | Softmax | SGD | 0.01 | 80% | 66
RelU | Softmax | ADAM | 0.00001 | 80% | 66
tanH | Sigmoidx | ADAM | 0.00001 | 90% | 66
RelU | Sigmoid | ADAM | 0.00001 | 90% | 66
RelU | Sofmax | ADAM | 0.00001 | 90% | 87

Changing the parameters such as learning rate(0.01-0.00001), optimization
function (ADAM and SGD), activation functions (RelU, tanH), loss functions
(softmax and sigmoid) resulted in different recognition rates upto 90%.Then,
gradually the number of classes was increased from 66 to 87 with the inclusion
of alpha numeric symbols and 7 symbols of each of the three categories,
namely, ‘tall’, ‘square’ and ‘short’ .The same set of changes resulted in
lower recognition rates.In conclusion, changing parameters for 66 classes
achieved 90% accuracy, but further increase of the number of classes caused a
drop in accuracy. Therefore, in the next stage, LeNet structure was modified
by adding one a fully connected,a convolutional layer and a pooling as
illustrated in Table II. Parameters such as the size of convolution were
revised and maxpooling from two series was changed to three series.
Additionally, extra fully connected layer with loss function RelU was added
before the last layer of softmax. All padding from zero size was changed to
two. All these modifications lead to a change of filter size as indicated in
Table II. Figures 8 and 9 illustrate the layer architectures of original LeNet
for 66 classes and modified LeNet for 87 classes produced in this study. Table
III indicates all changes in Hyper parameters and related results in LeNet and
Modified LeNet with 10 layers.

###  II-D SqueezeNet

This subsection describes SqueezeNet, an alternative architecture for the
network used in order to classify more than 100 classes up to 300. SqueezeNet
is a small CNN architecture that achieves AlexNet-level accuracy on 1000
classes of ImageNet with 50x fewer parameters. SqueezeNet replaces 3x3 filters
in AlexNet with 1x1 filters. The convolutional layer with 1x1 filters is
called squeeze layer. These squeeze layers feed expand layers which contain
convolutional layers with a mix of 1x1 and 3x3 filters. Fire module, the basic
building block of SqueezeNet, is made of squeeze layer and expand layer. The
fire module has two functions to perform:
1\. Reducing the total number of parameters and making a smaller model
2\. Making extra learning step to be embedded within the transformations
between several of “fire modules” in the lack of fully-connected layers.
The fully connected layers learn the relationships between the earlier higher
level features of a CNN and the classes the network is trying to identify.
That is, the fully connected layers are the ones that learn the details of
symbols in most networks such as VGG. SqueezeNet reduces the number of
parameters by decreasing the S with later convolutional layers. It creates a
larger activation/feature map later in the network and increases
classification accuracy. SqueezeNet is in contrast to VGG that activation maps
get smaller as get closer to the end of a network. He and Sun (2014) applied a
delayed down sampling to gain higher classification accuracy[27] SqueezeNet is
a CNN with 18 layers.It is trained on the ImageNet database to classify 1000
classes.The pre-trained network is a version of SqueezNet which has been
trained over more than one million images on ImageNet dataset for 1000 object
classes. Since a pre-trained network has appropriate weights, then the
technique is implemented by freezing some layers. In the next step, a network
using a pre-trained model has been fine-tuned to be converged optimally based
on 101 classes of mathematical symbols.

##  III  Experimental results

To select most appropriate parameters for model in network, objective function
must be monitored to observe how in will be affected by changing
parameters.This observation support system to find proper direction for
iteration to reduce error. This section initially presents different plots to
illustrate how various parameters affect. Sigmoid value is between (0 to 1).
Thus it suitable to predict the probability as an output.On the other hand
Sigmoid as loss function and tangent hyperbolic(tanH) as activation function
in networks with many layers based on vanishing gradient issues are not
efficient.

Figure 10 shows the LeNet performance with the activation function Relu and
the accuracy 80% with loss function softmax, and Figure 11 shows the LeNet
performance with the activation function tanH the accuracy 88% with loss
function sigmoid.

Figure 10:  LeNet - 66 classes,LR=0.01,SGD,RelU,Softmax Figure 11:  LeNet-66
classes,LR=0.01,SGD,tanH,Sigmoid

Since ADAM learns faster than SGD, the starting learning rate for ADAM must be
much less than that of SGD[28]. The learning rate is a critical concept and
must be selected carefully. If it is a short learning process, then it does
not improve, and if it is a long one, then the learning process cannot
converge.[29] learning rate of 0.00001 and using ADAM compares ADAM and SGD as
optimization functions for learning rate 10-5 and 10-2 respectively on LeNet.
Since ADAM learns faster than SGD, starting learning rate for ADAM must be
much less than that of SGD. Learning rate is a critical concept and must be
selected carefully. If it is a short learning process then it does not improve
and if it is a long one, then the learning process cannot converge.

Figure 12 shows the LeNet performance with the activation function Relu and
the accuracy 80% with loss function softmax, and Figure 13 shows the LeNet
performance with the activation function tanH the accuracy 90% with loss
function sigmoid.

Figure 12:  LeNet - 66 classes,LR=0.00001,ADAM,RelU,Softmax Figure 13:
LeNet-66 classes,LR=0.00001,ADAM,tanH,Sigmoid

Figure 14 shows modified LeNet run over 78 classes. Due to increase the number
of layers, Sigmoid and tanH are not the best options so Softmax and RelU are
replaced and ADAM with learning rate =0.00001 has 85% accuracy.

Figure 14:  Modified LeNet-76 classes classes,LR=0.00001

Figure 15 shows the performance of SqueezeNet using pre-trained one over 101
classes. The pre-trained network is a version of SqueezNet which has been
trained over more than one million images on ImageNet dataset for 1000 object
classes[30].

Figure 15:  SqueezeNet-101 classes classes

##  IV Conclusion

The dataset used in this study was made of MNIST 6000 images from inkML of the
CROHME dataset converted to PNG by the developed application in this research,
2000 images from the set of Handwritten Digit Images published by Computer
Vision Group of the University of São Paulo In this study, two different types
of classification methods, were used for offline handwritten mathematical
symbol recognition. Intensity of raw image as well as extracted features using
HOG, LBP, and salient maps were used to see the performance with regard to
recognition. The Neural Network methods used for this research were LeNet and
SqueezeNet. The number of classes was increased gradually 66 to 101. The final
experimental result is 90% using pre-trained SqueezNet by transfer learning.
The future work of the research is aimed at increasing the number of classes
to 300. .

## References

  * [1] R. H. Anderson, “Syntax-directed Recognition of Hand-printed Two-dimensional Mathematics,” in Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium, 1967, pp. 436–459.

  * [2] D. BLOSTEIN and A. N. N. GRBAVEC, “RECOGNITION OF MATHEMATICAL NOTATION,” in Handbook of Character Recognition and Document Image Analysis, pp. 557–582.

  * [3] Mouchère, H., Viard-Gaudin, C., Zanibbi, R. andGarain, U. (2016). CROHME:competition on recognition of online handwritten mathematical expressions, 15th International Conference on Frontiers in Handwriting Recognition, Shenzhen, China, 607-612.
  * [4] L. Hu and R. Zanibbi, “Line-of-Sight Stroke Graphs and Parzen Shape Context Features for Handwritten Math Formula Representation and Symbol Segmentation,” 2016 15th Int. Conf. Front. Handwrit. Recognit., pp. 180–186, 2016
  * [5] Wenhao He et al., “Context-aware mathematical expression recognition: An end-to-end framework and a benchmark,” in 2016 23rd International Conference on Pattern Recognition (ICPR), 2016, pp. 3246–3251
  * [6] M. Watt, “Mathematical document classification via symbol frequency analysis,” Towar. Digit. Math. Libr. Birmingham, United Kingdom, July 27th, 2008, pp. 29–40, 2008.
  * [7] L. G. Hafemann, R. Sabourin, and L. S. Oliveira, “Learning features for offline handwritten signature verification using deep convolutional neural networks,” Pattern Recognit., vol. 70, pp. 163–176, 2017.
  * [8] Nguyen, H. D., Le, A. D., and Nakagawa, M. (2016). Recognition of online handwritten math symbols using deep neural networks, IEICE Transactions on Information and Systems, E99.D, 12, 3110-3118.
  * [9] Phan, K. M., Le, A. D., Indurkhya, B. and Nakagawa, M . (2018). Augmented incremental recognition of online handwritten mathematical expressions, International Journal on Document Analysis and Recognition, 21, 4, 253-268.
  * [10] Julca-Aguilar F. D., and Hirata, N. S. T. (2018). Symbol detection in online handwritten graphics using Faster R-CNN, 13th International Workshop on Document Analysis Systems, Vienna, Austria, 151-156.
  * [11] F. Álvaro, J.-A. Sánchez, and J.-M. Bened, “Recognition of On-line Handwrittenb Mathematical Expressions Using 2D Stochastic Context-free Grammars and Hidden Markov Models,” Pattern Recogn. Lett., vol. 35, pp. 58–67, Jan. 2014.

  * [12] Zhang et al., “Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition,” Pattern Recognit., vol. 71, pp. 196–206, 2017.
  * [13] A. D. Le and M. Nakagawa, “Training an End-to-End System for Handwritten Mathematical Expression Recognition by Generated Patterns,” 2017 14th IAPR Int. Conf. Doc. Anal. Recognit., vol. 01, pp. 1056–1061, 2017
  * [14] Y. Shkarupa, R. Mencis, and M. Sabatelli, “Offline handwriting recognition using LSTM recurrent neural networks,” 28th BNAIC, pp. 88–95, 2016.
  * [15] X.Ren, and J. Malik. ”Learning a classification model for segmentation.” null. IEEE, 2003.
  * [16] L. Deng, “The MNIST database of handwritten digit immages for machine learning research [best of the web],” IEEE Signal Process. Mag., vol. 29, no. 6, pp. 141–142, 2012.

  * [17] J. A. Aghamaleki and S. M. Baharlou, “Transfer learning approach for classification and noise reduction on noisy web data,” Expert Syst. Appl., vol. 105, pp. 221–232, 2018.
  * [18] N. A. Hamid and N. N. A. Sjarif, “Handwritten recognition using SVM, KNN and neural network,” arXiv Prepr. arXiv1702.00723, 2017.
  * [19] A. Nazemi, “Non-Visual Representation of Complex Documents for Use in Digital Talking Books,” Curtin University, 2015.
  * [20] X. Hou and L. Zhang, “Saliency Detection: A Spectral Residual Approach,” 2007 IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1–8, 2007.
  * [21] S. Montabone and A. Soto, “Human Detection Using a Mobile Platform and Novel Features Derived from a Visua
  * [22] D. Mellouli, T. M. Hamdani, M. Ben Ayed, and A. M. Alimi, “Morph-CNN: A Morphological Convolutional Neural Network for Image Classification,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017, vol. 10635 LNCS, pp. 110–117.
  * [23] K. He and J. Sun, “Convolutional neural networks at constrained time cost,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 5353–5360.
  * [24] M. D. Ferreira, D. C. Corrêa, L. G. Nonato, and R. F. de Mello, “Designing architectures of convolutional neural networks to solve practical problems,” Expert Syst. Appl., vol. 94, pp. 205–217, 2018.
  * [25] D. Han, Q. Liu, and W. Fan, “A new image classification method using CNN transfer learning and web data augmentation,” Expert Syst. Appl., vol. 95, pp. 43–56, 2018.
  * [26] R. R. Selvaraju, A. Das, R. Vedantam, M. Cogswell, D. Parikh, and D. Batra, “Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization,” CoRR, vol. abs/1610.02391, 2016.
  * [27] F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer, “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and ¡1MB model size,” CoRR, vol. abs/1602.07360, 2016.
  * [28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv Prepr. arXiv1412.6980, 2014.
  * [29] L. N. Smith, “Cyclical learning rates for training neural networks,” in 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), 2017, pp. 464–472.
  * [30] N. Tavakolian, A. Nazemi, Z. Azimifar, and I. Murray, “Face recognition under occlusion for user authentication and invigilation in remotely distributed online assessments,” Int. J. Intell. Def. Support Syst., vol. 5, no. 4, pp. 277–297, 2018.