Marcel Granero-Moya, Penny Karanasou, Sri Karlapati, Bastian Schnell,
Nicole Peinelt, Alexis Moinet, Thomas Drugman

# A Comparative Analysis of Pretrained Language Models for Text-to-Speech

###### Abstract

State-of-the-art text-to-speech (TTS) systems have utilized pretrained
language models (PLMs) to enhance prosody and create more natural-sounding
speech. However, while PLMs have been extensively researched for natural
language understanding (NLU), their impact on TTS has been overlooked. In this
study, we aim to address this gap by conducting a comparative analysis of
different PLMs for two TTS tasks: prosody prediction and pause prediction.
Firstly, we trained a prosody prediction model using 15 different PLMs. Our
findings revealed a logarithmic relationship between model size and quality,
as well as significant performance differences between neutral and expressive
prosody. Secondly, we employed PLMs for pause prediction and found that the
task was less sensitive to small models. We also identified a strong
correlation between our empirical results and the GLUE scores obtained for
these language models. To the best of our knowledge, this is the first study
of its kind to investigate the impact of different PLMs on TTS.

Index Terms: pretrained language models, text-to-speech

##  1 Introduction

The advent of Transformer-based pretrained language models (PLMs) has
revolutionized natural language processing (NLP). Initially, Transformers [1]
were proposed as an attention-based encoder-decoder architecture for machine
translation. But the Transformer decoder was rapidly adopted for text
generation with GPT-2 [2] and its successors, and the transformer encoder for
natural language understanding (NLU) with BERT [3]. Regarding transformer
encoders for NLU, since the emergence of BERT, many BERT-like models have been
released with more pretraining data [4, 5], novel pretraining methods [6],
more efficient architectures [7], or smaller architectures with faster
inference [8, 9, 10, 11]. Generally, these models are trained on a two-step
setup. First, they are pretrained from scratch on task-agnostic objectives
such as masked language modelling, next sentence prediction, or replaced token
detection. Self-supervised pretraining enables the model to learn from large
unlabeled corpora like Wikipedia or the Book corpus [12]. Second, they are
fine-tuned on a task-specific objective and corpus.

The success of pretrained language models on NLU has inspired research in
other fields. In text-to-speech (TTS), several studies have incorporated
pretrained language models to Tacotron2 [13], an encoder-decoder TTS system
relying on attention. Fang et al. [14] enhance Tacotron2 by adding BERT as a
secondary encoder. Thus, the decoder attends to both the subword-level
representations generated by BERT and the output of the original Tacotron2
encoder. In a similar fashion, Hayashi et al. [15] compare subword-level and
phrase-level representations from BERT-large as extra inputs to Tacotron2
decoder. They conclude that text context helps in generating more natural
speech specially at the granularity of subwords. Xiao et al. [16] include
Chinese BERT to improve prosody in Tacotron2. They also leverage the
pretrained language model to predict pauses between Chinese characters.
Alternatively, Xu et al. [17] propose using BERT embeddings to generate
context vectors from neighboring sentences to improve prosody modelling. Away
from Tacotron 2, Kenter et al. [18] include small versions of BERT in an RNN-
based TTS system and highlight the importance of fine-tuning the pretrained
model. They present an ablation study on the size of the language model for
$F_{0}$ prediction. More recently, Makarov et al. [19] demonstrate that BERT
enhances prosody prediction specially when it is fed with multiple sentences.
Karlapati et al. [20] present CopyCat2, a TTS system that learns a prosodic
space and subsequently predicts prosody representations based on
contextualized subword embeddings from BERT. Similarly, eCat [21] is
introduced as a system that predicts prosody with RoBERTa and blocks of
normalizing flows.

PLMs have been widely surveyed [22, 23, 24, 25] and compared empirically [26,
27, 28] for various NLP tasks. They are often compared on GLUE [29], an NLU
benchmark combining 9 tasks such as natural language inference, sentiment
analysis, and sentence similarity. Notwithstanding the numerous works studying
the impact of PLMs in NLP and the positive results of PLMs in TTS systems,
they have not been surveyed nor compared for TTS tasks, leaving many questions
unanswered. First, although several models have outperformed BERT for NLP
tasks recently, there is almost no empirical data on how PLMs other than BERT
perform in TTS systems. Second, there is little information about the relation
between model properties (e.g., size, architecture, and pretraining
methodology) and their performance in TTS tasks. Third, there is no comparison
of PLMs’ performance on NLU and TTS tasks like prosody prediction. This
relation would be beneficial to link results of PLMs in NLU to potential
findings in TTS.

Language model | HuggingFace's model name | Num. parameters (M)
---|---|---
$BERT_{TINY}$ [11] | prajjwal1/bert-tiny | 4.39
$BERT_{MINI}$ [11] | prajjwal1/bert-mini | 11.17
$ELECTRA_{SMALL}$ [6] | google/electra-small-discriminator | 13.48
$MobileBERT$ [10] | google/mobilebert-uncased | 24.58
$BERT_{SMALL}$ [11] | prajjwal1/bert-small | 28.76
$SqueezeBERT$ [9] | squeezebert/squeezebert-uncased | 51.09
$DistilBERT$ [8] | distilbert-base-cased | 65.19
$BERT_{BASE}$ [3] | bert-base-cased | 108.31
$ELECTRA_{BASE}$ [6] | google/electra-base-discriminator | 108.89
$RoBERTA_{BASE}$ [4] | roberta-base | 124.65
$FunnelTransformer_{SMALL}$ [7] | funnel-transformer/small | 130.97
$FunnelTransformer_{INTERMEDIATE}$ [7] | funnel-transformer/intermediate | 177.06
$XLM-RoBERTa_{BASE}$ [5] | xlm-roberta-base | 278.04
$BERT_{LARGE}$ [3] | bert-large-cased | 333.58
$RoBERTA_{LARGE}$ [4] | roberta-large | 355.36

Table 1: Selection of pretrained language models downloaded from HuggingFace
[30] and their number of parameters.

This work presents a comparative analysis of 15 PLMs for two TTS tasks:
prosody prediction and pause prediction. To the best of our knowledge, this is
the first study of its kind in the field of TTS. Our contributions are
summarized as follows. Firstly, we conduct an experimental analysis of the 15
PLMs for both prosody and pause prediction. Secondly, the results for prosody
prediction demonstrate a logarithmic relation between model size and quality,
and reveal differences in performance for neutral and expressive prosody.
Thirdly, our results on pause prediction indicate that the task is less
sensitive to smaller language models. Lastly, we compare our findings with the
GLUE scores, highlighting similarities between the performance of PLMs on TTS
and NLU.

##  2 Methods

This study investigates how different PLMs impact two distinct TTS tasks:
prosody prediction and pause prediction. To predict prosody, we use the
prosody predictor from eCat [21], an end-to-end TTS model. We delve into the
creation and prediction of the latent prosodic space in subsection 2.1. In
addition, we analyze pause prediction as a separate task in subsection 2.2.
Finally, we present the selected PLMs to be tested on the aforementioned tasks
and explain their differences in subsection 2.3.

###  2.1 Prosody prediction

We investigate the impact of PLMs on prosody prediction using eCat [21]
because it is a state-of-the-art TTS system that operates in a prosodic latent
space. eCat latent space is designed in a way to factor out phoneme and
speaker information and capture prosody. It works in a two-step approach where
it first learns word-level acoustic and duration-based prosodic
representations through an auto-encoder step. In the first step, reference
encoders generate the prosodic space based on the mel-spectrogram, and
decoders synthesize the speech waveform based on the prosodic representations.
The acoustic representations are expected to capture the main prosodic
characteristics such as intonation and tone, whereas duration representations
are expected to focus on rhythm, stress, and duration. In the second step, a
prosody prediction model uses a PLM, followed by Bi-LSTM layers and
normalizing flow blocks, to predict these learned prosodic representations
from text. The language model produces contextualized word embeddings that
capture syntactic and semantic information from text. These embeddings are
then fed through Bi-LSTM layers and combined with speaker embeddings to be
used as conditions for flow layers. The outputs of the prosody prediction
model are used in inference replacing the reference encoder outputs since mel-
spectrograms are not available at test time. In this study, we aim to replace
the default PLM used in the prosody predictor, i.e. RoBERTa-base, with
alternative PLMs and evaluate their effectiveness.

###  2.2 Pause prediction

In order to achieve a diverse range of tasks, we examine various PLMs for
pause prediction. The objective of pause prediction is to predict the
probability of a pause after each word, thereby ensuring that the pauses are
accurately placed. Our pausing model consists of a language model
(specifically, RoBERTa-base), a Bi-LSTM layer, and a dense projection. The
model is trained using binary cross-entropy loss between the predicted values
and the target binary pausing sequence. Our target binary pausing labels are
derived from ground-truth durations, where any silence lasting more than 100ms
is considered a pause. This threshold was chosen empirically. Our aim was to
annotate anything that is reasonably a silence in terms of audio signal,
rather than focusing on pauses from a human perception perspective. During
inference, we binarize the predicted probabilities using a threshold
previously computed on the development set. This model can be used in eCat to
override the duration prosodic representations of pauses. Similar to prosody
prediction, we want to compare the pause prediction model with alternative
PLMs of different sizes and pretrained with different tasks and data.

Figure 1: Euclidean distances between predicted and ground-truth latents for
acoustic (left) and duration (right) representations. Language models are
shown in ascending order of size.

###  2.3 Selection of pretrained language models

We select 15 pretrained language models, including the default PLM (RoBERTa-
base), for both the prosody predictor and the pause prediction model. All
language models are BERT-like models, i.e., transformer encoders trained with
large corpora of data in a self-supervised fashion. Our selection aims to
cover a broad spectrum of model sizes with diverse pretraining or distillation
techniques. In Table 1, we provide a list of the pretrained language models,
along with their corresponding HuggingFace name and model size. Note that the
size refers to the embedding matrix and model architecture weights combined.

We first select medium- and large-size models that target NLU performance.
BERT [3] (BERT-base, BERT-large) was the first pretrained transformer encoder
achieving state-of-the-art performance in multiple NLP tasks. It was
pretrained on 2 tasks: masked language modelling and next sentence prediction.
RoBERTa [4] (RoBERTa-base, RoBERTa-large) leverages BERT's architecture but is
trained longer over more data with longer input sequences and bigger batches.
Authors removed the next sentence prediction task, and used masked language
modelling dynamically. Overall, RoBERTa outperforms BERT in several NLU
benchmarks. XLM-RoBERTa [5] is a multilingual version of RoBERTa. It has the
largest embedding matrix out of the selection, consisting of 192 million
parameters due to its coverage of 100 languages. Despite this, it is
comparable in architecture size to both BERT-base and RoBERTa-base. ELECTRA
[6] (ELECTRA-base, ELECTRA-small) is trained with replaced token detection
instead of masked language modelling. A small generator network corrupts the
text by replacing tokens and a discriminator is trained to identify the
replaced tokens. Authors claim a more efficient training while obtaining
better results than previous models. Funnel-Transformer [7] (Funnel-
intermediate, Funnel-small) makes an architectural change by gradually
compressing the sequence of hidden states. It is trained with a similar
objective as ELECTRA.

Regarding small models, transfer learning from large-scale language models and
more efficient architectures have reduced the size and increased speed of
language models while retaining performance. DistilBERT [8] is a distilled
version of BERT that reduces BERT size by 40%, while preserving 97% of its NLU
performance. SqueezeBERT [9] uses grouped convolutions instead of fully-
connected layers as they account for a great percentage of FLOPs and latency.
MobileBERT [10] is a deep and thin language model that reduces BERT size by 4.
It is trained through progressive knowledge transfer from a bigger network.
Turc et al. [11] introduce the concept of pretrained distillation to improve
the efficiency of knowledge distillation and release several distilled
versions of BERT (BERT-small, BERT-mini, BERT-tiny).

##  3 Experimental setup

###  3.1 Data

We conducted experiments on an internal dataset containing recordings of
speakers reading excerpts from Wikipedia articles, news articles,
conversations, etc. The dataset comprises over 20 hours of speech per speaker,
recorded from 4 female speakers of US English. All recordings were sampled at
24kHz and documents are not shared across data splits (train/dev/test). All
data splits contain utterances with diverse levels of expressivity. We divided
them into utterances with more neutral speech such as discourse and utterances
with highly expressive speech such as quotes. Differences in expressivity were
taken into account during evaluations (see Section 4.1).

###  3.2 System configuration

In this work, we did not tune hyperparameters for every model that was
trained. We used the same learning rate and training steps across all language
models to maintain simplicity and consistency in our experiments. However, we
fine-tuned language models of diverse sizes and pretraining techniques. This
may have resulted in an advantage for the RoBERTa-base model compared to the
other models, as the hyperparameters were tuned specifically for this pre-
trained language model. For instance, large language models may have been
under-trained.

For the prosody predictor, we used the training hyperparameters from the
original eCat [21], which were tuned for RoBERTa-base. The Bi-LSTM layer had a
hidden dimension of 512. To train the pause predictor, we used a batch size of
32, Adam optimizer with a learning rate of $1e\mbox{-}5$ in the language model
and $1e\mbox{-}4$ in the rest of the model, with betas of 0.9 and 0.98. The
model was trained for 50k steps.

We investigated the effect of different language models on both TTS tasks.
First, we trained the prosody predictor with the 15 PLMs listed in Table 1.
Second, we trained and compared the pause prediction model with a subset of 12
PLMs. BERT-small, Funnel-small, and Funnel-intermediate were excluded due to
implementation issues. In both setups, the prosody predictor and pause
prediction model were trained end-to-end, meaning that the losses were back-
propagated into the language models to fine-tune textual representations.

Prosody prediction and pause prediction tasks required the use of word-level
tokens. However, most language model tokenizers operate at the word-piece
level, breaking words into smaller subword units. To bridge this gap, we
downsampled the contextualized word-piece embeddings produced by the language
model. Specifically, we averaged the word-piece embeddings that aligned with
the same word-level tokens, thus obtaining word-level embeddings.

##  4 Results

###  4.1 Language models for prosody prediction

The prosody predictor was trained with every language model from Table 1. To
measure the accuracy of our prosody prediction model objectively, we
calculated the Euclidean distance between the predicted latent samples and the
ground-truth ones. We generated the ground-truth prosody representations by
passing the ground-truth mel-spectrogram through a reference encoder.
Generally, the closer the predicted latent sample is to the ground-truth
latent, the more coherent and appropriate it is. It is worth noting that
during training, we did not use Euclidean distance, as we employed negative
log-likelihood on normalizing flows instead. Figure 1 shows Euclidean
distance, for acoustic (left) and duration (right) latents. On the y axis,
language models are sorted by descending size. Results manifest a negative
correlation between size and distance; prosody predictors with bigger language
models reduce the distance to ground-truth representations for both acoustic
and duration latents. In other words, bigger language models seem to better
predict prosody latents.

Alternative LM | eCat with alt. LM | eCat with RoBERTa-base | Recordings | Gap reduction (%) | p-value | Significance
---|---|---|---|---|---|---
$BERT_{TINY}$ | $73.86\pm 16.37$ | $74.99\pm 15.37$ | $78.32\pm 14.58$ | -33.73 | 0 | TRUE
$BERT_{MINI}$ | $72.77\pm 16.35$ | $73.64\pm 15.79$ | $77.31\pm 15.31$ | -23.78 | 0.0026 | TRUE
$MobileBERT$ | $73.25\pm 18.32$ | $74.15\pm 18.14$ | $78.01\pm 17.40$ | -23.19 | 0.0019 | TRUE
$DistilBERT$ | $69.04\pm 19.75$ | $69.83\pm 19.36$ | $75.96\pm 17.87$ | -12.86 | 0.0017 | TRUE
$RoBERTa_{LARGE}$ | $72.23\pm 16.36$ | $72.67\pm 15.76$ | $75.98\pm 15.30$ | -13.14 | 0.0914 | FALSE
Table 2: MUSHRA mean scores. Every row is a MUSHRA evaluation with 3 systems:
eCat with the alternative LM from the first column, eCat with RoBERTa-base,
and recordings. Gap reduction is the difference between RoBERTa-base and the
alternative language model relative to the difference between recordings and
RoBERTa-base.

In addition, we conducted subjective evaluations on synthetic speech. We
launched 5 MUSHRA evaluations comparing recordings, speech from vanilla eCat
(i.e., with RoBERTa-base in the prosody predictor), and speech generated by
eCat with an alternative language model in the prosody predictor. The
alternative language models were selected keeping a broad range of sizes,
namely, RoBERTa-large, DistilBERT, MobileBERT, BERT-mini, and BERT-tiny. All
evaluations were conducted with 24 testers and the same 80 utterances,
consisting of 20 utterances per speaker. Approximately half of the samples per
speaker comprised neutral speech while the other half consisted of highly
expressive speech. The MUSHRA question was about the naturalness of the voice.
We expect any changes to be due to prosody since we are only modifying the
prosody predictor, i.e., only changing the predicted latents. Table 2
summarises results for each MUSHRA evaluation. It displays mean MUSHRA scores
for 3 systems: recordings, vanilla eCat (with RoBERTa-base), and eCat with an
alternative language model in the prosody predictor. We show the reduction of
gap between vanilla eCat and recordings; negative values indicate a gap
increase. Moreover, a two-sided t-test was computed to compare vanilla eCat
against eCat with an alternative language model. The last two columns show the
$p\mbox{-}value$ and whether it was statistically significant given a
significance level of $\alpha=0.05$.

As a result, all alternative models perform statistically significantly worse
than RoBERTa-base except for RoBERTa-large. Similar to objective metrics,
there is a relation between size and gap reduction. Smaller models have worse
degradation than larger models. Notably, the fourth MUSHRA evaluation yielded
the lowest scores for eCat with DistilBERT and eCat with RoBERTa-base. We
attribute this outcome to the fact that this specific evaluation took place
several weeks after the rest of the evaluations and involved a different set
of listeners.

Furthermore, we examined MUSHRA results for neutral and expressive speech
separately. Our systems were rated worse on expressive speech and received
scores close to recordings for neutral speech. There were no statistically
significant differences between RoBERTa-base and alternative models on neutral
speech, except for BERT-tiny, the smallest one. On the other hand, RoBERTa-
base was statistical significantly better than all alternative systems on
expressive speech. We hypothesize that prosody prediction on expressive speech
is more challenging than in neutral speech because of its variance in pitch,
tone, and speed. Prosody predictors with small language models were able to
capture the expressivity involved for neutral speech but failed to reproduce
expressive prosody. Therefore, we find more suitable to use large language
models for expressive speech, whereas for neutral speech medium-size models
would suffice.

Given the objective metrics and subjective evaluations presented, we identify
a relation between size and prosodic quality. This relation is manifested on
objective metrics as shown in Figures 2(a) and 2(b). We consider Funnel-
intermediate an outlier as it has architectural changes and might have been
under-trained. Subjective metrics demonstrate a similar trend in Figure 2(c).
Overall, the findings suggest that the quality of prosody improves in a
logarithmic fashion as the size of language models increases. While our
results for prosody prediction rely on eCat, we believe that the conclusions
presented in this section will be applicable to other architectures utilizing
prosodic latent spaces.

(a) GLUE - Distance of acoustic prosodic latents.
Correlation score of -0.84.

(b) GLUE - Distance of duration prosodic latents.
Correlation score of -0.86.

(c) GLUE - MUSHRA gap reduction for prosody prediction.
Correlation score of 0.87.

(d) GLUE - F-0.5 score for pause prediction.
Correlation score of 0.83.

Figure 2: Comparison of GLUE scores with distance of (a) acoustic and (b)
duration prosodic latents, (c) MUSHRA gap reduction, and (d) F-0.5 score for
pause prediction. Logarithmic regressions are included for every metric.

###  4.2 Language models for pause prediction

The pause prediction model was trained with 12 language models including
RoBERTa-base. We trained the language models in the pause prediction model to
predict pause probabilities after each word. Once trained, a threshold was
obtained on the dev set. We measured precision, recall, and F-scores for the
binarized pause predictions on the test set. Table 3 presents results for
every pause prediction model trained. It shows precision, recall and
$F\mbox{-}0.5$ and $F\mbox{-}1$ scores. $F\mbox{-}0.5$ is more aligned with
our objectives as we believe that failing to predict a pause is not as bad as
predicting a wrong pause. Results indicate a relation between size and
performance as larger models like RoBERTa-large perform better than smaller
ones. Actually, the smallest models (i.e., BERT-tiny and BERT-mini) were
substantially behind the rest, but from ELECTRA-small until RoBERTa-large
there was a slighter performance increment. Thus, we hypothesize that there is
a minimum size required, e.g., ELECTRA-small's size, to perform on par with
medium-size models like RoBERTa-base. We also attribute the outstanding
results of both ELECTRA versions to their discriminative pretraining task,
replaced token detection, which makes the model learn more efficiently and
robustly from all input tokens at the same time.

In conclusion, we believe that predicting pauses in English only requires a
high-level understanding of language, as the task ambiguity is limited, and
small models like ELECTRA-small are already able to capture it. Compared to
prosody prediction, we surmise that there is an order of complexity between
the tasks. Predicting prosody for expressive speech would be the most
challenging task, followed by predicting prosody for neutral speech. Pause
prediction would be less challenging than predicting prosody. Therefore, small
language models could replace larger ones as they perform similarly well. It's
important to note that our findings are specific to the English language, and
further research is required to determine whether the conclusions hold true
for other languages.

Language Model | Precision | Recall | F-0.5 | F-1
---|---|---|---|---
$BERT_{TINY}$ | 0.895 | 0.828 | 0.881 | 0.860
$BERT_{MINI}$ | 0.920 | 0.825 | 0.899 | 0.870
$ELECTRA_{SMALL}$ | 0.934 | 0.828 | 0.910 | 0.877
$MobileBERT$ | 0.931 | 0.832 | 0.910 | 0.879
$SqueezeBERT$ | 0.938 | 0.802 | 0.907 | 0.865
$DistilBERT$ | 0.929 | 0.831 | 0.907 | 0.877
$BERT_{BASE}$ | 0.916 | 0.852 | 0.902 | 0.883
$ELECTRA_{BASE}$ | 0.941 | 0.811 | 0.912 | 0.871
$RoBERTa_{BASE}$ | 0.933 | 0.831 | 0.911 | 0.879
$XLM\mbox{-}RoBERTa$ | 0.925 | 0.827 | 0.904 | 0.873
$BERT_{LARGE}$ | 0.933 | 0.837 | 0.912 | 0.882
$RoBERTa_{LARGE}$ | 0.924 | 0.858 | 0.910 | 0.889
Table 3: Results for the pause prediction model with different language
models. Models are shown in ascending order of size.

##  5 Further analysis

###  5.1 Relation with GLUE scores

We investigated whether there are comparable trends in language models
performance for TTS and NLU tasks. We compared experimental results for TTS
obtained in this study with scores on GLUE test set. GLUE [29] is a benchmark
composed of 9 diverse language understanding tasks. Note that we collected
GLUE scores from original publications of language models, while the language
models used in this work were downloaded from HuggingFace. We compared the
GLUE scores with the metrics reported for prosody prediction and pause
prediction, and computed correlation scores. Figure 2 shows the comparison of
GLUE scores with (a) distance of acoustic prosody latents to ground-truth, (b)
distance of duration prosody latents to ground-truth, (c) MUSHRA gap
reduction, and (d) F-0.5 score for pause prediction. We computed the
correlation coefficient between each pair of score sequences, and plotted a
logarithmic regression. The comparison reveals a robust correlation between
GLUE scores and the reported metrics, with all absolute correlation scores
exceeding 0.8. These results suggest that language models enhance prosody and
pause prediction similarly to natural language understanding tasks, as
demonstrated by the comparison with GLUE scores. Additionally, the metrics
exhibit a logarithmic relationship with size, meaning that an increase in size
generally corresponds to an increase in quality. Based on the quality-size
relationship and the correlation with GLUE scores described earlier, it seems
reasonable to estimate the performance of BERT-like language models for
prosody and pause prediction by considering architectural size and their GLUE
score.

###  5.2 Model throughput analysis

Batch inference time was measured for all prosody predictors trained with
different language models. We used an NVIDIA Tesla V100 SXM2 32GB GPU with
CUDA 11.6 and Pytorch 1.11.0+cu113. 8 iterations were run in a test set of
2500 samples with a batch size of 32 samples. For a better estimate, the first
batch was discarded to avoid GPU initialization overhead, and the last one was
also discarded as its batch size was smaller.

Figure 3 displays batch inference time for each language model. All values are
relative to the mean batch inference time of the prosody predictor with
RoBERTa-base. The figure on the right depicts mean batch inference time
compared to the prosody predictor size. The results demonstrate a linear
relation between batch inference time and models size besides a few outliers.
For instance, MobileBERT was slower than models with similar size. In fact, it
was slower than what was reported on the original paper. We attribute this gap
to the fact that the model was not the original, but instead was uploaded to
HuggingFace by an external user. On the other hand, even though XLM-RoBERTa
doubled in total size, its inference time remained similar to BERT-base. This
is because their architecture sizes are identical, and the increase in size is
attributed to the embedding matrix as XLM-RoBERTa comprises a wide
multilingual vocabulary.

Figure 3: Batch inference time for the prosody predictor, which varies
depending on the language model used. All the times are presented relative to
the mean inference time for the prosody predictor when using RoBERTa-base.

##  6 Conclusion

This work presents a comparative analysis between 15 PLMs for two TTS tasks in
US-English: prosody prediction and pause prediction. We evaluated language
models performance on prosody prediction with objective and subjective
metrics. Results indicate a logarithmic relation between size and quality. We
expanded to pause prediction, where small language models performed on par
with larger ones. Moreover, we compared this study's results with GLUE scores
and identified similar trends. Thus, improvements in language models for NLU
can inform future developments and improvements in TTS. To the best of our
knowledge, this is the first study of this kind for TTS.

## References

  * [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' _Advances in neural information processing systems_ , vol. 30, 2017.
  * [2] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , ``Language models are unsupervised multitask learners,'' _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``BERT: Pre-training of deep bidirectional transformers for language understanding,'' in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
  * [4] L. Zhuang, L. Wayne, S. Ya, and Z. Jun, ``A Robustly Optimized BERT Pre-training Approach with Post-training,'' in _Proceedings of the 20th Chinese National Conference on Computational Linguistics_. Huhhot, China: Chinese Information Processing Society of China, Aug. 2021, pp. 1218–1227. [Online]. Available: https://aclanthology.org/2021.ccl-1.108
  * [5] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, ``Unsupervised Cross-lingual Representation Learning at Scale,'' in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, 2020, pp. 8440–8451. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.747
  * [6] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, ``ELECTRA: Pre-training text encoders as discriminators rather than generators,'' in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=r1xMH1BtvB
  * [7] Z. Dai, G. Lai, Y. Yang, and Q. V. Le, ``Funnel-transformer: filtering out sequential redundancy for efficient language processing,'' in _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , 2020, pp. 4271–4282.
  * [8] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, ``DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,'' _arXiv preprint arXiv:1910.01108_ , 2019.
  * [9] F. Iandola, A. Shaw, R. Krishna, and K. Keutzer, ``SqueezeBERT: What can computer vision teach NLP about efficient neural networks?'' in _Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing_. Online: Association for Computational Linguistics, 2020, pp. 124–135. [Online]. Available: https://www.aclweb.org/anthology/2020.sustainlp-1.17
  * [10] Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou, ``MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices,'' in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, 2020, pp. 2158–2170. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.195
  * [11] I. Turc, M.-W. Chang, K. Lee, and K. Toutanova, ``Well-read students learn better: On the importance of pre-training compact models,'' _arXiv preprint arXiv:1908.08962_ , 2019.
  * [12] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, ``Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books,'' in _2015 IEEE International Conference on Computer Vision (ICCV)_. Santiago, Chile: IEEE, Dec. 2015, pp. 19–27. [Online]. Available: http://ieeexplore.ieee.org/document/7410368/
  * [13] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu, ``Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions,'' in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. Calgary, AB: IEEE, Apr. 2018, pp. 4779–4783. [Online]. Available: https://ieeexplore.ieee.org/document/8461368/
  * [14] W. Fang, Y.-A. Chung, and J. Glass, ``Towards transfer learning for end-to-end speech synthesis from deep pre-trained language models,'' _arXiv preprint arXiv:1906.07307_ , 2019.
  * [15] T. Hayashi, S. Watanabe, T. Toda, K. Takeda, S. Toshniwal, and K. Livescu, ``Pre-Trained Text Embeddings for Enhanced Text-to-Speech Synthesis,'' in _INTERSPEECH 2019_. ISCA, Sep. 2019, pp. 4430–4434. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2019/hayashi19_interspeech.html
  * [16] Y. Xiao, L. He, H. Ming, and F. K. Soong, ``Improving Prosody with Linguistic and BERT Derived Features in Multi-Speaker Based Mandarin Chinese Neural TTS,'' in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. Barcelona, Spain: IEEE, May 2020, pp. 6704–6708. [Online]. Available: https://ieeexplore.ieee.org/document/9054337/
  * [17] G. Xu, W. Song, Z. Zhang, C. Zhang, X. He, and B. Zhou, ``Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-End Speech Synthesis,'' in _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. Toronto, ON, Canada: IEEE, Jun. 2021, pp. 6079–6083. [Online]. Available: https://ieeexplore.ieee.org/document/9414102/
  * [18] T. Kenter, M. Sharma, and R. Clark, ``Improving the Prosody of RNN-Based English Text-To-Speech Synthesis by Incorporating a BERT Model,'' in _INTERSPEECH 2020_. ISCA, Oct. 2020, pp. 4412–4416. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2020/kenter20_interspeech.html
  * [19] P. Makarov, S. Ammar Abbas, M. Łajszczak, A. Joly, S. Karlapati, A. Moinet, T. Drugman, and P. Karanasou, ``Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosody,'' in _INTERSPEECH 2022_. ISCA, Sep. 2022, pp. 3368–3372. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2022/makarov22_interspeech.html
  * [20] S. Karlapati, P. Karanasou, M. Łajszczak, S. Ammar Abbas, A. Moinet, P. Makarov, R. Li, A. Van Korlaar, S. Slangen, and T. Drugman, ``CopyCat2: A Single Model for Multi-Speaker TTS and Many-to-Many Fine-Grained Prosody Transfer,'' in _INTERSPEECH 2022_. ISCA, Sep. 2022, pp. 3363–3367. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2022/karlapati22_interspeech.html
  * [21] A. Abbas, S. Karlapati, B. Schnell, P. Karanasou, M. Granero Moya, A. Nagaraj, A. Boustati, N. Peinelt, A. Moinet, and T. Drugman, ``eCat: An End-to-End Model for Multi-Speaker TTS & Many-to-Many Fine-Grained Prosody Transfer,'' in _INTERSPEECH 2023_. ISCA, 2023.
  * [22] Q. Liu, M. J. Kusner, and P. Blunsom, ``A survey on contextual embeddings,'' _arXiv preprint arXiv:2003.07278_ , 2020.
  * [23] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, ``Pre-trained models for natural language processing: A survey,'' _Science China Technological Sciences_ , vol. 63, no. 10, pp. 1872–1897, Oct. 2020. [Online]. Available: https://link.springer.com/10.1007/s11431-020-1647-3
  * [24] X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, Y. Yao, A. Zhang, L. Zhang, W. Han, M. Huang, Q. Jin, Y. Lan, Y. Liu, Z. Liu, Z. Lu, X. Qiu, R. Song, J. Tang, J.-R. Wen, J. Yuan, W. X. Zhao, and J. Zhu, ``Pre-trained models: Past, present and future,'' _AI Open_ , vol. 2, pp. 225–250, 2021. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S2666651021000231
  * [25] A. Rogers, O. Kovaleva, and A. Rumshisky, ``A Primer in BERTology: What We Know About How BERT Works,'' _Transactions of the Association for Computational Linguistics_ , vol. 8, pp. 842–866, Dec. 2020. [Online]. Available: https://direct.mit.edu/tacl/article/96482
  * [26] S. Casola, I. Lauriola, and A. Lavelli, ``Pre-trained transformers: an empirical comparison,'' _Machine Learning with Applications_ , vol. 9, p. 100334, Sep. 2022. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S2666827022000445
  * [27] J. Li, T. Tang, Z. Gong, L. Yang, Z. Yu, Z. Chen, J. Wang, X. Zhao, and J.-R. Wen, ``ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models,'' in _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Seattle, United States: Association for Computational Linguistics, 2022, pp. 3519–3539. [Online]. Available: https://aclanthology.org/2022.naacl-main.258
  * [28] V. Lialin, K. Zhao, N. Shivagunde, and A. Rumshisky, ``Life after BERT: What do Other Muppets Understand about Language?'' in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 3180–3193. [Online]. Available: https://aclanthology.org/2022.acl-long.227
  * [29] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, ``GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding,'' in _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. Brussels, Belgium: Association for Computational Linguistics, 2018, pp. 353–355. [Online]. Available: http://aclweb.org/anthology/W18-5446
  * [30] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. Von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, ``Transformers: State-of-the-Art Natural Language Processing,'' in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Online: Association for Computational Linguistics, 2020, pp. 38–45. [Online]. Available: https://www.aclweb.org/anthology/2020.emnlp-demos.6


978-1-nnnn-nnnn-n/yy/mm nnnnnnn.nnnnnnn

banner above paper title short description of paper

Muhammad Kashif National University of Computer and Emerging Sciences E-mail:
<EMAIL_ADDRESS>
# Urdu Handwritten Text Recognition Using ResNet18

(20yy)

###### Abstract

Handwritten text recognition is an active research area in the field of deep
learning and artificial intelligence to convert handwritten text into machine-
understandable. A lot of work has been done for other languages, especially
for English, but work for the Urdu language is very minimal due to the cursive
nature of Urdu characters. The need for Urdu HCR systems is increasing because
of the advancement of technology. In this paper, we propose a ResNet18 model
for handwritten text recognition using Urdu Nastaliq Handwritten Dataset
(UNHD) which contains 3,12000 words written by 500 candidates.

###### keywords:

Handwritten, Urdu Text, Recognition, ResNet

††conference: CONF ’yy Month d–d, 20yy, City, ST, Country

##  1 Problem statement

Urdu is the national language of Pakistan and one of the largest languages in
the world. Enough amount of research work has been done for other languages
but when it comes to Urdu a very limited work has been done especially for
handwritten text recognition. The handwritten text has enormous applications
for instance converting handwritten files into digital, reading house numbers,
postal addresses automatically. National Information and Technology Board
(NITB) has introduced E-office in government offices so a system is required
to convert all handwritten text into digital format.

Figure 1: Urdu text converted into machine formate  Figure 2: Challenges with
urdu text

##  2 Introduction

In the field of deep learning and artificial intelligence, handwritten text
recognition is one of the challenging tasks because of its cursive nature, and
the shape of each character and different writing style is also one of the key
issues.

Figure 3: One stroke characters  Figure 4: Two and three stroke characters

Research work has been done for printed text with higher accuracy rates but
there is limited work for handwritten text especially for Urdu handwritten
text. Urdu is the official language of Pakistan and in most of the government
offices, Urdu is used for writing official orders and directives, documents in
government and private organizations in Pakistan. There is a need for
converting handwritten text into a machine-understandable because the digital
text is safe and we can easily edit it or search it. For example, in
handwritten documents/books and digital books, searching a specific topic in
digital documents is easy but the same task is not easy in handwritten text.
Another example, here in Pakistan we have documents, records of some very
sensitive and important data in handwritten files in government offices. One
major issue is to store all the files, secondly, if you need some old files,
it is very difficult to search for them, and also its time consuming and needs
a human resource. But if this data were in digital form it would have been
very easy to find old documents and to store them.

Figure 5:  Urdu Characters and Numerals

Motivation: The main reason for this work is that Urdu is an official language
of Pakistan and one of the largest languages of the world. secondly, very
limited research has been carried out for handwritten text in Urdu. Thirdly,
converting handwritten text into digital has enormous applications such as
reading postal addresses, reading house numbers, and reading signboards for
driverless cars, also a huge amount of data in government offices are in a
handwritten format, saving or keeping handwritten text is difficult as it
needs a separate place and someone to maintain all these documents and there
is a high chance of losing this data in disasters but keeping these documents
in digital format, one can save a lot of time and resource. The hand-written
text is not like a typical text in one single font, but recognition of
handwritten text is a challenging task due to its writing styles vary from
person to person.
Background: Optical Character Recognition (OCR) is a technology that picks out
handwritten or printed text from digital images of handwritten text or printed
text for example scanned documents. Optical Character Recognition examines the
digital images and generates a code for the text that can be used to retrieve
the text from the image into digital format. Optical Character Recognition
also referred to as text recognition that is used to convert physical
documents into machine-readable text. OCR has typically four different types
i.e. Input acquisition mode (online or offline text), written mode
(handwritten or printed), character connectivity (isolated or joined), and
font constraints (single or Omni font). These types of Optical Character
Recognition are shown in figure 6 khan2018urdu .

Figure 6: OCR System.

  1. 1.

Input acquisition Mode Input acquisition mode is the mode in which an input is
given to the optical character recognition. It is of two types i.e. Online
recognition and Offline recognition.

  2. 2.

Writing Mode The writing mode resources available for OCR can be in any format
that is typically written text which contains tables, headers, footers,
borders, etc. and another format can be a handwritten format which has
different styles of writing for different people.

  3. 3.

Font Constraints Font constraints are the type that is developed for one font
style. It may fail or partially succeed for text written in another style.
Optical Character Recognition that is only for a single font style is known as
font constraints OCR.

  4. 4.

character connectivity Another type of Optical Character Recognition is
character connectivity which is isolated or joined characters. isolated
characters are those that are not joined with each other, while joined
characters are joined and it changes the shape and position of a character.

A handwritten text system is a system that recognizes a handwritten text
pattern and converts the user’s handwritten characters or words into a format
that the computer understands. It has many applications in to-days era i.e.
reading home addresses and reading numbers automatically or reading traffic
signs for driverless cars. Without this technology, we have dependent on
handwriting texts that result in errors. Handwritten text or data needs to
update manually and it is difficult to save, maintain or search for a specific
file. It also needs someone to update it manually. Storing handwritten data is
difficult, people have lost important data because of disasters, and they no
backup of this handwritten data.
Urdu is the national language of Pakistan and one of the largest languages in
the world. it is widely used in government offices, institutions, banks, and
very important documents are in handwritten form and it has many applications
in this technology era such as reading home numbers, reading billboards, etc.
Limited research has been done for Urdu text recognition mostly for printed
text. For handwritten text, a research is being done but there is no system
available to provide a reliable result.
UNHD Database, finding a data set for Urdu Nastaliq handwriting is a separate
challenge. There are very few free available datasets that are large enough to
train a model. In this paper, we use Urdu Nastaliq Handwritten Dataset (UNHD),
which has Urdu numbers, digits, etc, but in Urdu Nasta’liq font. It covers
commonly used ligatures that are written by 500 writers with their handwriting
on A4 size paper. The dataset has 3,12000 words written by 500 people with a
total of 10,000 lines ahmed2019handwritten .

Figure 7: UNHD dataset details

##  3 Related work

Related work for Urdu handwritten text recognition is limited as compared to
other languages, but people are working for the Urdu language, and it’s a good
research area. Urdu text style is usually connected and has many variations
therefore it is a challenging task. The literature shows that for character-
level recognition, the artificial neural network (ANN) is widely used. AN ANN
is based on the concept of working of a human brain, it is a collection of
nodes called artificial neurons van2017artificial . These nodes can send input
from one layer to another. Designing an OCR that can recognize multiple
languages is challenging as every language exhibits different characteristics
and features. Generalizing this kind of issue is nearly impossible. To address
this problem a new approach was proposed by Ali et al. ali2004language  shows
a new technique for the character of a language and presented all the
characters of a language as a geometrical stroke. Their technique is font
independent but it can also apply to standard fonts. The specialty of their
technique is that their model needs to be trained once. Figure 8 shows the
geometrical strokes of Urdu characters. The accuracy rate is 70% to 80% with
25 samples of handwritten text. This technique has some limitations that it
will not work on words having dots and signs.

Figure 8: Basic geometrical strokes of Urdu script

The Urdu language has a large character set, it has the issue of similarity of
strokes. To address this issue, Haider et al. khan2010online  proposed a
methodology for dividing a set of Urdu characters into groups as shown in
figure 9.

Figure 9: Urdu characters (a) is matching order (b) is set of groups of urdu
characters.

For single-character recognition, Shahzad et al. shahzad2009urdu  presented a
technique that set up features when using isolated characters by examining the
keystrokes and secondary strokes. A linear classifier was used for a total of
190 characters which was written by different writers also contains 38 Urdu
characters. 6% error was detected by the classifier due to the similarity of
characters.

Figure 10: Similar characters ”De” (left) and ”Daal” (right)

Similar work has been proposed by Khan et al. khan2014online  considering
characters that change their shape and position. A multidimensional neural
network is widely used in a multidimensional environment for sequence learning
and decomposition husnain2019recognition . The Urdu Printed Text Image (UPTI)
has been used for assessment purposes, which has 10,000 images of handwritten
and printed text sabbour2013segmentation .
In razzak2009multi  razzak2009numeral , the authors presented a similarity
between Urdu and Arabic language handwritten data. They used the logic of HMM
for Urdu and Arabic to recognize numerals. 97%, 96%, and 97.8% accuracies are
obtained by using the fuzzy logic rule.
Recent work: Ahmad et al.ahmad2017offline  has developed a system for Offline
Urdu Nastaleeq Based on Stacked Denoising Autoencoder. Similarly, Rizvi et al.
rizvi2019optical  has proposed a technique for Nastalique like Urdu text by
using supervised learning. Their system has obtained the highest rate ever
achieved by Optical character recognition that is 98.4% training and 97.3%
test results by many experimental settings. A comprehensive Urdu dataset named
Urdu Nastaliq Handwritten Dataset (UNHD) is presented by Ahmed et
al.ahmed2019handwritten . The dataset contains commonly used ligatures and is
written by 500 people on A4 size paper. This is one of the largest datasets of
Urdu which are available free and can be accessed by emailing them. Chhajro et
al.chhajro2020handwritten  uses different techniques of deep learning to
recognize Urdu handwritten characters through pictures. The different
algorithm has been used for evaluation via pictures such as Multilayer
perceptron (MLP), K-Nearest neighbor, Recurrent neural network, etc. Ali et
al.ali2020pioneer  has introduced a dataset for Urdu handwritten characters
which is written by 900 people and they have used a deep autoencoder for
recognition of these characters. Husnain et al.husnain2020urdu  provides a
survey of different techniques for Urdu text recognition, which shows
interesting facts. Figure 11 shows details of Urdu datasets.

Figure 11: Details of Urdu datasets Figure 12: rates reported on common
datasets Figure 13: comparison results on UPTI dataset

Figure 14 shows year-wise published articles distribution.

Figure 14: Year wise distribution of related papers

In husnain2019recognition , proposed a technique for recognition of multi-font
Urdu handwritten characters written by different people from different age
groups and institutions. figure 15 shows the different isolated Urdu
characters.

Figure 15: Urdu characters grouped by their shape

In zahid2020roman , they have proposed a dataset named RU dataset from social
media data. This data is important for companies and institutions for future
prediction. The dataset is manually interpreted where each sentence contains
sentiment labels. majeed2020emotion  has worked for emotion detection in roman
Urdu as these days people express their emotions on social media by posting
and sharing pictures and statuses. Some work has been done for emotion
detection but in roman Urdu, limited work has been done. They have applied
different algorithms such as SVM, KNN, random forest. naeem2020deep  has
presented a technique for detecting fake news, yellow journalism, and social
media as these days it is very easy to post or share fake news and there is a
huge amount of this kind of contents so to manage it intelligently there is a
need for a model that detects these kinds of news and sensationalism. They
have obtained an accuracy of 97% by using Long Short-Term Memory (LSTM).
arshad2019corpus  has presented an evaluation technique for detecting emotion
in a short text. Emotion detection has many applications i.e. intelligent
agents, smart clinical diagnoses, etc. They have collected 10,000 sentences in
roman Urdu to experiment with roman Urdu. qamarrelationship  has presented a
methodology for relationship identification using emotion on social media in
the form of text messages or posts. The use of social media has become a part
of our day-to-day life, people express their emotions in the form of posts,
statuses, or text messages. Using these data they have proposed a method of
Relationship Identification using Emotion Analysis (RIEA). They have obtained
85% accuracy using RIEA. javed2020collaborative  has proposed a framework for
health and fitness evaluation of people by using machine learning techniques.
The framework is called Collaborative Shared Healthcare Plan (CSHCP). The
results show a satisfactory outcome. beg2019algorithmic  has applied machine
learning techniques to predict trading decisions using data of stock markets
which generate a huge amount of data every day. In farooq2019melta  they have
presented a method level energy estimation MELTA for android applications to
compute the energy consumed by android applications. sahar2019towards  has
proposed a methodology for computation of energy consumption by structural
information related to energy consumption. As Energy is one of the main
concerns of the developers these days. They have proposed three object-
oriented suits. farooq2019bigdata  has used a machine learning technique for
energy-related posts on stack overflow. They have analyzed 4 million posts
from stack overflow using the machine learning technique NLTK. In
zafar2019constructive , they have presented a restriction-based generator for
the generation of that work for general games. Their results are quite good
and can be compared with other level generators. In dilawar2018understanding ,
the authors presented a machine learning technique to obtain a suitable
representation of the social media data to help governments with people
opinion and their problems related to local issues, political, religion. This
will help the government to know about people’s opinions and their problems.
They have obtained the highest F1 score of 76.40% in SemEval 2016 task 5 and
for SemEval 2015 task 12 they have achieved 94.99%. In awan2021top , They have
proposed a top rank approach to extract key phrases from documents. The
approach extracts the position of key phrases from the document and enlarges
it with topical key phrases. The top rank attains an F1 score of 0.73 for key
phrases which is ahead of other state-of-the-art approaches. In
javed2020alphalogger , They have developed an approach Alpha logger that
deduces the alphabet keys typed on a keypad. Typing on a keypad of smartphones
generates vibrations that can be used to recognize the keys. In
alvi2017ensights , They have presented a tool named Ensight that helps
software developers by giving them energy consumption details. The tool has
been tested on three open-source android applications and gives the F1 score
of 86%.

##  4 Methodology

  * •

Our initial approach was to apply 1-Dimensional Bidirectional LSTM on the UNHD
dataset, we got the dataset but couldn’t get the desired results. We then
applied a ResNet18 Model on Urdu Nastaliq Handwritten Dataset (UNHD). the
Resnet model is a residual network used for computer vision tasks. ResNet is
based on skip connection. The figure below shows how Resnet uses skip
connection.

Figure 16: Skip Connection  Figure 17: Block diagram  Figure 18: Block diagram
of proposed ResNet

    * –

Pre Processing: Pre-processing is the stage when images go through pre-
processing for further processing. The image was first processed to remove
noise.

    * –

Feature Extraction: In this stage, the Urdu handwritten characters are
processed to extract features such as the shape of the character, the height
of the character, and horizontal and vertical lines of the image.

    * –

ResNet18 Framework: Resnet18 applies 18 convolution layers on dataset images
to learn and predict the output.

Our contribution to this work is that we applied the ResNet18 framework which
has 18 convolution layers on Urdu Nastaliq Handwritten Dataset (UNHD), to the
best of our knowledge we applied ResNet18 on UNHD for the first time. So a new
framework applied to the Urdu dataset, its a good learning work for new
students in this field and us as well.

  * •

The idea was to recognize Urdu handwritten text with better accuracy than
previous work. First to work on any of the previously proposed techniques and
comes up with a better result if it does not work, then apply a new technique
on a different dataset.

  * •

The proposed model is significant because It has 18 convolution layers so it
recognizes Urdu text more in-depth because of its shape and cursive style.

##  5 Evaluation and Experiments

  * •

The experiment is conducted on a Lenovo laptop with an Intel® Core™ i7-10750H
CPU @ 2.60GHz, 16 Gigabyte of ram, with a graphic card NVIDIA GEFORCE GTX 1660
Ti, Window 10 operating system. Colab was used for coding with Python as a
programming language. The UNHD dataset was split as a training set and test
set.

  * •

We consider a proposed work already done on the UNHD dataset with the highest
accuracy rate as our benchmark because the goal was to come up with more
accurate results than previous works.

  * •

This benchmark is appropriate for our results because the dataset matters a
lot in recognition of handwritten text, so we consider a work that is done on
the same dataset.

I would like to thank Almighty Allah who gave me the strength and courage to
accomplish this project in this trying time.
I am very thankful to my advisor Dr. Mirza Omer Beg for his supervision, help,
and his amazing lectures throughout the course.
I am also thankful to all my friends for help.

## References

  * [1] Ibrar Ahmad, Xiaojie Wang, Ruifan Li, and Shahid Rasheed.  Offline urdu nastaleeq optical character recognition based on stacked denoising autoencoder.  China Communications, 14(1):146–157, 2017.
  * [2] Saad Bin Ahmed, Saeeda Naz, Salahuddin Swati, and Muhammad Imran Razzak.  Handwritten urdu character recognition using one-dimensional blstm classifier.  Neural Computing and Applications, 31(4):1143–1151, 2019.
  * [3] Anjum Ali, Mohmood Ahmad, Nasir Rafiq, Javed Akber, Usman Ahmad, and Shahwar Akmal.  Language independent optical character recognition for hand written text.  In 8th International Multitopic Conference, 2004. Proceedings of INMIC 2004., pages 79–84. IEEE, 2004.
  * [4] Hazrat Ali, Ahsan Ullah, Talha Iqbal, and Shahid Khattak.  Pioneer dataset and automatic recognition of urdu handwritten characters using a deep autoencoder and convolutional neural network.  SN Applied Sciences, 2(2):1–12, 2020.
  * [5] Hamza M Alvi, Hareem Sahar, Abdul A Bangash, and Mirza O Beg.  Ensights: A tool for energy aware software development.  In 2017 13th International Conference on Emerging Technologies (ICET), pages 1–6. IEEE, 2017.
  * [6] Muhammad Umair Arshad, Muhammad Farrukh Bashir, Adil Majeed, Waseem Shahzad, and Mirza Omer Beg.  Corpus for emotion detection on roman urdu.  In 2019 22nd International Multitopic Conference (INMIC), pages 1–6. IEEE, 2019.
  * [7] Mubashar Nazar Awan and Mirza Omer Beg.  Top-rank: a topicalpostionrank for extraction and classification of keyphrases in text.  Computer Speech & Language, 65:101116, 2021.
  * [8] Mirza O Beg, Mubashar Nazar Awan, and Syed Shahzaib Ali.  Algorithmic machine learning for prediction of stock prices.  In FinTech as a Disruptive Technology for Financial Institutions, pages 142–169. IGI Global, 2019.
  * [9] MA Chhajro, H Khan, F Khan, K Kumar, AA Wagan, and S Solangi.  Handwritten urdu character recognition via images using different machine learning and deep learning techniques.  Indian Journal of Science and Technology, 13(17):1746–1754, 2020\.
  * [10] Noman Dilawar, Hammad Majeed, Mirza Omer Beg, Naveed Ejaz, Khan Muhammad, Irfan Mehmood, and Yunyoung Nam.  Understanding citizen issues through reviews: A step towards data informed planning in smart cities.  Applied Sciences, 8(9):1589, 2018.
  * [11] Muhammad Umer Farooq, Mirza Omer Beg, et al.  Bigdata analysis of stack overflow for energy consumption of android framework.  In 2019 International Conference on Innovative Computing (ICIC), pages 1–9. IEEE, 2019.
  * [12] Muhammad Umer Farooq, Saif Ur Rehman Khan, and Mirza Omer Beg.  Melta: A method level energy estimation technique for android development.  In 2019 International Conference on Innovative Computing (ICIC), pages 1–10. IEEE, 2019.
  * [13] Mujtaba Husnain, Malik Muhammad Saad Missen, Shahzad Mumtaz, Mickaël Coustaty, Muzzamil Luqman, and Jean-Marc Ogier.  Urdu handwritten text recognition: a survey.  IET Image Processing, 14(11):2291–2300, 2020.
  * [14] Mujtaba Husnain, Malik Muhammad Saad Missen, Shahzad Mumtaz, Muhammad Zeeshan Jhanidr, Mickaël Coustaty, Muhammad Muzzamil Luqman, Jean-Marc Ogier, and Gyu Sang Choi.  Recognition of urdu handwritten characters using convolutional neural network.  Applied Sciences, 9(13):2758, 2019.
  * [15] Abdul Rehman Javed, Mirza Omer Beg, Muhammad Asim, Thar Baker, and Ali Hilal Al-Bayatti.  Alphalogger: Detecting motion-based side-channel attack using smartphone keystrokes.  Journal of Ambient Intelligence and Humanized Computing, pages 1–14, 2020.
  * [16] Abdul Rehman Javed, Muhammad Usman Sarwar, Mirza Omer Beg, Muhammad Asim, Thar Baker, and Hissam Tawfik.  A collaborative healthcare framework for shared healthcare plan with ambient intelligence.  Human-centric Computing and Information Sciences, 10(1):1–21, 2020\.
  * [17] Kamran Ullah Khan et al.  Online urdu handwritten character recognition: Initial half form single stroke characters.  In 2014 12th International Conference on Frontiers of Information Technology, pages 292–297. IEEE, 2014.
  * [18] Kamran Ullah Khan and Ihtesham Haider.  Online recognition of multi-stroke handwritten urdu characters.  In 2010 International Conference on Image Analysis and Signal Processing, pages 284–290. IEEE, 2010.
  * [19] Naila Habib Khan and Awais Adnan.  Urdu optical character recognition systems: Present contributions and future directions.  IEEE Access, 6:46019–46046, 2018.
  * [20] Adil Majeed, Hasan Mujtaba, and Mirza Omer Beg.  Emotion detection in roman urdu text using machine learning.  In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering Workshops, pages 125–130, 2020.
  * [21] Bilal Naeem, Aymen Khan, Mirza Omer Beg, and Hasan Mujtaba.  A deep learning framework for clickbait detection on social area network using natural language cues.  Journal of Computational Social Science, pages 1–13, 2020.
  * [22] Saira Qamar, Hasan Mujtaba, Hammad Majeed, and Mirza Omer Beg.  Relationship identification between conversational agents using emotion analysis.  Cognitive Computation, pages 1–15.
  * [23] Muhammad Imran Razzak, SA Hussain, Abdel Belaïd, and Muhammad Sher.  Multi-font numerals recognition for urdu script based languages.  International Journal of Recent Trends in Engineering (IJRTE), 2009\.
  * [24] Muhammad Imran Razzak, SA Hussain, and Muhammad Sher.  Numeral recognition for urdu script in unconstrained environment.  In 2009 International Conference on Emerging Technologies, pages 44–47. IEEE, 2009.
  * [25] SSR Rizvi, A Sagheer, K Adnan, and A Muhammad.  Optical character recognition system for nastalique urdu-like script languages using supervised learning.  International Journal of Pattern Recognition and Artificial Intelligence, 33(10):1953004, 2019.
  * [26] Nazly Sabbour and Faisal Shafait.  A segmentation-free approach to arabic and urdu ocr.  In Document recognition and retrieval XX, volume 8658, page 86580N. International Society for Optics and Photonics, 2013.
  * [27] Hareem Sahar, Abdul A Bangash, and Mirza O Beg.  Towards energy aware object-oriented development of android applications.  Sustainable Computing: Informatics and Systems, 21:28–46, 2019\.
  * [28] Nabeel Shahzad, Brandon Paulson, and Tracy Hammond.  Urdu qaeda: recognition system for isolated urdu characters.  In Proceedings of the IUI Workshop on Sketch Recognition, Sanibel Island, Florida. Citeseer, 2009.
  * [29] Marcel Van Gerven and Sander Bohte.  Artificial neural networks as models of neural information processing.  Frontiers in Computational Neuroscience, 11:114, 2017.
  * [30] Adeel Zafar, Hasan Mujtaba, Sohrab Ashiq, and Mirza Omer Beg.  A constructive approach for general video game level generation.  In 2019 11th Computer Science and Electronic Engineering (CEEC), pages 102–107. IEEE, 2019.
  * [31] Rabail Zahid, Muhammad Owais Idrees, Hasan Mujtaba, and Mirza Omer Beg.  Roman urdu reviews dataset for aspect based opinion mining.  In 2020 35th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW), pages 138–143. IEEE, 2020.