# Authorship Attribution in Bangla Literature (AABL) via Transfer Learning
using ULMFiT

Aisha Khatun<EMAIL_ADDRESS>0000−0001−6559−4493 ,  Anisur Rahman
<EMAIL_ADDRESS>0000−0002−4616−4559 ,  Md Saiful Islam  saiful-
<EMAIL_ADDRESS>0000−0001−9236−380X ,  Hemayet Ahmed Chowdhury
<EMAIL_ADDRESS>0000−0002−5582−669X and  Ayesha Tasnim  tasnim-
<EMAIL_ADDRESS>0000−0002−7143−3255 Department of Computer Science and
Engineering, Shahjalal University of Science and
TechnologyKumargaonSylhetBangladesh3114

(2022)

###### Abstract.

Authorship Attribution is the task of creating an appropriate characterization
of text that captures the authors’ writing style to identify the original
author of a given piece of text. With increased anonymity on the internet,
this task has become increasingly crucial in various security and plagiarism
detection fields. Despite significant advancements in other languages such as
English, Spanish, and Chinese, Bangla lacks comprehensive research in this
field due to its complex linguistic feature and sentence structure. Moreover,
existing systems are not scalable when the number of author increases, and the
performance drops for small number of samples per author. In this paper, we
propose the use of Average-Stochastic Gradient Descent Weight-Dropped Long
Short-Term Memory (AWD-LSTM) architecture and an effective transfer learning
approach that addresses the problem of complex linguistic features extraction
and scalability for authorship attribution in Bangla Literature (AABL). We
analyze the effect of different tokenization, such as word, sub-word, and
character level tokenization, and demonstrate the effectiveness of these
tokenizations in the proposed model. Moreover, we introduce the publicly
available Bangla Authorship Attribution Dataset of 16 authors (BAAD16)
containing 17,966 sample texts and 13.4+ million words to solve the standard
dataset scarcity problem and release six variations of pre-trained language
models for use in any Bangla NLP downstream task. For evaluation, we used our
developed BAAD16 dataset as well as other publicly available datasets.
Empirically, our proposed model outperformed state-of-the-art models and
achieved 99.8% accuracy in the BAAD16 dataset. Furthermore, we showed that the
proposed system scales much better even with an increasing number of authors,
and performance remains steady despite few training samples.

Authorship attribution, Transfer Learning, Language model, AWD-LSTM, Bangla

††copyright: acmcopyright††journalyear: 2022††doi:
10.1145/1122445.1122456††journal: JACM††journalvolume: 37††journalnumber:
4††article: 111††publicationmonth: 8††ccs: Computing methodologies Information
extraction††ccs: Computing methodologies Supervised learning by
classification††ccs: Computing methodologies Neural networks

##  1\. Introduction

Authorship attribution (AA) is a distinct type of classification task that
deals with identifying the author of an anonymous piece of writing within a
set of probable authors. Identifying the original author depends on capturing
the elusive characteristics of an author’s writing style from their digitized
texts by analyzing stylometric and linguistic features. Despite extensive use
of stylometry to identify the literary styles of authors, AA is challenging
because of (i) the complex structure of language (ii) similarity of the topic
of discussion, (iii) implicit writing styles of authors, (iv) known forms of
writing, such as novels or stories, often have similar structures. Extracting
useful information about an author’s style is difficult, especially to build
an end-to-end system to detect the author of a text.

Anonymity is widespread in recent times, primarily due to the widespread use
of the internet; the use and misuse of anonymity have become an essential
factor to consider. The plethora of unattributed digital footprints makes
authorship attribution indispensable in various fields, and its applications
are constantly growing. Application of authorship attribution covers sectors
such as forensic investigation, plagiarism detection, computer security,
criminal law, cyber-crime, literature, trading, etc. Intelligence agencies can
use it to link intercepted messages to known enemies; original authors of
harassing messages can be identified. In education, student submissions can be
verified, and actual authors can be identified using authorship attribution.

Early approaches of AA followed statistical methods that suffered from text
length limitation problems and failed to capture the text’s semantic meaning.
Most stylometric studies use some language items, such as lexical and
syntactic components. Among them, character n-gram has been used widely and
shown to provide insights into the writing style of the authors (Stamatatos,
2009). In recent studies, machine learning approaches (Zhang et al., 2014) are
used to extract complex and essential features of the text to identify the
original author’s writing style. Some research extracted semantic information
using pre-trained word, and character embeddings (Khatun et al., 2019). With
increasing computing power, various deep learning models have also been
successfully applied to the task, achieving impressive results (Jafariakinabad
et al., 2019; Ruder et al., 2016).

AA is not limited to any specific language and pertains to any written text.
Bangla is an Indic language, and the 7th most spoken language globally, shared
among the people of Bangladesh, the Indian states of West Bengal, Tripura,
Assam, and the global Bangla diaspora communities (Wikipedia, 2021). Despite
success in English and other languages (Jafariakinabad et al., 2019; Ruder et
al., 2016; Kreutz and Daelemans, 2018; Sundararajan and Woodard, 2018), Bangla
lacks significant work in this area due to its high inflection ”with more than
160 different inflected forms for verbs, 36 different forms for nouns, and 24
different forms for pronouns” (Bhattacharya et al., 2005). Moreover, Bangla
exhibits diglossia, Shadhu and Cholito forms: these are two different writing
styles with slightly different vocabularies and syntax. Although the former is
uncommon in modern writings, it constitutes a vast majority of classical
literature in Bangla, and both forms share a large number of common roots. All
these contribute to the complex written form of Bangla and its extended
vocabulary.

Only a few research work was done to conduct Authorship Attribution in Bangla
Literature (AABL), but most of them lack in some important aspect. Most works
apply traditional approaches to extract features like word length, sentence
length, word use frequency, etc., along with n-grams (Das et al., 2015;
Hossain et al., 2017; Islam et al., 2018; Ahmad et al., 2020). Besides,
embedding-based models have also been used to encode words (Chowdhury et al.,
2018b) and characters (Khatun et al., 2019) to extract authorial information
without any manual feature selection. Although a few promising studies have
been done using deep learning, existing systems suffer due to the substantial
dependence on the dataset. The limitations of the present systems are
described below:

  1. (1)

Most AABL works use manual feature engineering, which is corpus dependent and
labour-intensive. High-performing end-to-end systems are not widely available
for AABL.

  2. (2)

Lack of resources and scarcity of open benchmark datasets limit existing
systems’ rapid development and expansion. Most previous works use available
small datasets of a maximum of 10 authors for this purpose (Das et al., 2015;
Chowdhury et al., 2018b; Khatun et al., 2019).

  3. (3)

Current systems are not scalable and perform poorly for increasing number of
authors, especially where number of sample texts per author is limited, which
is the case generally for real-life scenarios.

  4. (4)

Despite the deep neural network-based model’s good performance, they suffer
due to the use of small datasets and short length of texts. They typically
require many samples for each author, which is costly, time-consuming, and not
always available.

A linguistically driven transfer learning method was proposed in (Howard and
Ruder, 2018), which uses pre-trained language models trained on unsupervised
large text corpora to improve performance in various downstream tasks through
representation learning. With the help of transfer learning, it is possible to
the solve existing problems and build a highly scalable, robust system for
authorship attribution. Hence, an elaborate study on the application of
transfer learning for author detection in Bangla using language modeling is
presented in this paper. The language model acts as a re-usable language
learner trained on a large corpus in an unsupervised manner. Then the semantic
and syntactic information learned by the language model is leveraged to
perform authorship attribution in the target datasets.

Figure 1. Schematic diagram of the proposed system. The first two steps are
language modelling tasks, and the last step is authorship attribution. The
embedding layer and the AWD-LSTM base remain the same in all steps, only the
final dense layer changes. The trained weights of the fixed parts are passed
on from pre-training to fine-tuning to the classification step, updated in
each step.

In this paper, we present an effective transfer learning approach that uses
task-specific optimization with language models pre-trained on two different
datasets during the training phase and fine-tuned on target datasets. Figure 1
shows a brief overview of our methodology where we use AWD-LSTM (Average
Stochastic Gradient Descent Weight-Dropped LSTM) (Merity et al., 2018)
architecture as the language model. The language model is first trained with a
general Bangla corpus and then fine-tuned with authorship attribution text in
an unsupervised manner. Later, a classifier is added to the pre-trained
language models, and supervised authorship attribution is carried out.
Additionally, the effects of various tokenization on proposed models are
analyzed in terms of performance; the robustness is tested against the
existing models by increasing the number of authors and decreasing samples per
author. Extensive experiments were done with all the variations, and the best
model for AABL was identified. The results demonstrate a clear superiority of
the transfer-learning-based approach against the other traditional models.

Our contributions are three-fold:

  1. (1)

In this article, we introduce the largest and most varied dataset for AABL
with long text samples of 16 authors in an imbalanced manner, imitating real-
world scenarios more closely.

  2. (2)

We present an intuitively simple but computationally effective transfer
learning approach in which the model is pre-trained on a large corpus, fine-
tuned with the target dataset, and later the classifier is trained with
labelled data for AABL. To the best of our knowledge, no work has been done to
leverage the power of transfer learning for AABL, reducing manual labour and
enhancing model re-usability manifold. Experimental results show that the
proposed model considerably outperforms the existing models and achieves
state-of-the-art performance, efficiently solving the limitations of the
previous works.

  3. (3)

The various language models trained in this work can be used as pre-trained
models for many downstream tasks in Bangla language. All of these pre-trained
models, along with the code and dataset of this paper have been released for
public use111https://github.com/tanny411/Authorship-Attribution-using-
Transfer-Learning.

The paper is organized as follows. Section 2 gives an overview of the previous
related work. Section 3 describes all the datasets used in this work with
brief statistical reports. Section 4 contains the methodology of our work. In
Section 5 we describe the proposed model along with some traditional models
used for comparison. Section 6 shows the results and their analysis. Section 7
contains information about the released pre-trained models. Finally,
conclusions and directions for further research are summarized in Section 8.

##  2\. Related Work

###  2.1. On Authorship Attribution

Authorship attribution task is an essential research topic and has been
prevalent for quite a long time. Research in authorship attribution relies on
detecting authors’ style through stylometric analysis, assuming that the
authors subconsciously use homological idiolect in their writings. Different
kinds of feature extraction based methods are implemented to detect these
patterns. The features are classified as lexical, character, syntactic, and
semantic. Function words, punctuation, ratio of certain words, average word
length, average sentence length, vocabulary richness, bag of n-grams, etc.,
are extracted to determine the authorial styles (Sari et al., 2018; Das et
al., 2015; Hossain et al., 2017; El Bakly et al., 2020). Function words are
parts of speech that do not have lexical meaning but express grammatical
relationships among other words and can be used to identify the authors from
their sub-conscious pattern of the use of these words and other writing
characteristics. Naive Bayes and Support Vector Machines were used with
features such as ratio of punctuation, apostrophes per word, uppercase ratio,
etc. (Barry and Luna, 2012). Word frequency, part-of-speech bigram model, and
preterminal tag bigram model were also used (Goldman and Allison, [n.d.]).
Topic modeling approaches were employed to detect authors of unknown texts
(Seroussi et al., 2014; Phani et al., 2017)

Most of the earlier researchers dealt with small collections where each author
may have been heavily inclined towards particular topics which made the
authorship attribution task border on topic classification (Sari et al.,
2018). Character n-grams can pick up author nuances combining lexical and
syntactic information, and so are an essential set of features (Sari et al.,
2018; Stamatatos, 2009). A combination of lexical and syntactic features
provides valuable information and has shown enhancement in performance (Kreutz
and Daelemans, 2018). Such combinations can be crucial to improve performance
in cross-topic authorship attribution and single-domain attribution
(Sundararajan and Woodard, 2018).

Proceeding with the advancement of deep learning, a large body of work is
available on authorship attribution and stylometry using various deep learning
models. For example, multi-headed recurrent neural network character language
model was used that outperformed the other methods in PAN 2015 (Bagnall,
2016). Some works used syntactic recurrent neural networks, which learn
document representations from parts-of-speech tags and then use attention
mechanisms to detect the authorial writing style (Jafariakinabad et al.,
2019). Convolutional neural networks have also been employed for this task.
Impressive performance was achieved by using character-level and multi-channel
CNN for large-scale authorship attribution (Ruder et al., 2016). Character
n-grams help identify the authors of tweets, and CNN architectures capture the
character-level interactions, representing patterns in higher levels, thus
detecting distinct styles of authors (Shrestha et al., 2017). A combination of
pre-trained word vectors with one hot encoded POS tag was also used (Hitschler
et al., 2017). Others investigate syntactic information in authorship task by
building separate language models for each author using part-of-speech tags
besides word and character-level information (Fourkioti et al., 2019).

###  2.2. On Bangla

Despite the significant progress in English and other western languages for
authorship attribution, not much has been done in the Bangla language,
especially with transfer learning methods. A notable body of literature exists
on the use of hand-drawn features to extract authorial styles from texts.
Features such as POS tag count, word frequency, word and sentence length,
type-token ratio, unique words percentage, common words, spelling variation,
etc., are often used. Some of the such extracted features were used with a
dataset of 4 (Das et al., 2015) and 3 (Chakraborty, 2012) current Bangladeshi
authors. The use of N-gram features is the most common along with other
extracted features and has been used with probabilistic classification methods
(Das and Mitra, 2011), voting classifier with cosine similarity measurements
for 6 authors (Hossain et al., 2017), Naive Bayes for 3 authors (Anisuzzaman
and Salam, 2018; Phani et al., 2017), Random forests for 10 authors (Islam et
al., 2017), and Feed-forward neural networks for 3, 5 and 23 authors (Phani et
al., 2016; Islam et al., 2018; Ahmad et al., 2020). SVM was employed on a
dataset of 6 authors (Pal et al., 2017). Islam et al. used unigram, bigram,
trigram, part-of-speech like pronoun and conjunction as features, removed stop
words, and used Information Gain (IG) to select the most important features.
The classification was compared between Naive Bayes, Decision tree, and Random
forest classifiers (Islam et al., 2017). On the other hand, Phani et al. used
term frequency (tf), term frequency-inverse document frequency (tf-idf), and
binary presence methods on stop words, word and character unigrams, bigrams,
and trigrams to perform author attribution with Naive Bayes, SVM and Decision
Trees (Phani et al., 2017).

Traditional methods of using n-grams counts consider each feature independent
of meaning and context. Instead, all words can be combined under a continuous
vector space by representing each word or token with a high dimensional
vector. These vectors are called word embedding and can better represent
semantic and syntactic similarity among words such that similar words are
closer in the vector space than words that are semantically different. The
effects of various kinds of word embeddings on authorship attribution were
analyzed on a 6 author dataset with multiple neural network architectures
showing that using fastText skip-gram word embeddings with a Convolutional
Neural Network (CNN) performs best to extract authorial information from texts
(Chowdhury et al., 2018b; Chowdhury et al., 2019a, b). Besides, a character-
level CNN model was used for 6 to 14 author detection where character
embeddings were pre-trained on a larger corpus (Khatun et al., 2019).

###  2.3. On Transfer Learning

Transfer learning is the process of reusing a model trained on an initial task
as a starting point for a different task. Deep learning approaches have been
using this method to boost model performance and save an enormous amount of
time on various computer vision tasks and natural language processing (NLP).
Transfer learning has been prevalent in computer vision for a long time, and
pre-trained models on massive datasets are readily available. Natural language
processing started approaching transfer learning by using pre-trained word
embeddings which aim at only the first layer of the model. Some approaches
combine multiple derived embeddings at different layers (Peters et al., 2018).
The idea of using transfer learning from language models has been approached
(Dai and Le, 2015) but not widely adopted due to its need for large-scale
datasets. To address these issues, a method called ULMFiT was proposed that
successfully enables robust transfer learning in any NLP task (Howard and
Ruder, 2018). Later transformer-based transfer learning approaches were able
to reduce training time significantly (Devlin et al., 2018; Liu et al., 2019;
Radford et al., 2018). These architectures use attention mechanism, which
works on the entire sequence simultaneously instead of working token by token
like in LSTM. Many variations of these models have been used in numerous NLP
tasks. Pre-trained models using multiple languages are also made available for
various transformer-based architectures (Devlin et al., 2018; Lample and
Conneau, 2019; Yang et al., 2019; Conneau et al., 2019).

Transfer learning has been adopted in Bangla language tasks in both computer
vision and NLP. Various computer vision tasks exist such as handwritten Bangla
word recognition (Pramanik and Bag, 2020), Bangla sign language recognition
(Nishat and Shopon, 2019), Bengali Ethnicity and Gender Classification (Jewel
et al., 2019) etc. In NLP, transfer learning started with word, and character
embeddings (Chowdhury et al., 2018b; Khatun et al., 2019). BengFastText was
created as the largest Bangla word embedding model using 250 million articles
(Karim et al., 2020). BERT (Devlin et al., 2018) was used for sentiment
analysis in code-mixed English-Bangla social media data (Jamatia et al.,
2020). The contextualized embeddings from BERT were used for Bangla named
entity recognition task (Ashrafi et al., 2020). Transfer learning from pre-
trained models was also used for detecting fake news in the Bangla (Hossain et
al., 2020). No such work on authorship attribution in the Bangla language has
been done to date.

##  3\. Corpora

The scarcity of standard benchmark datasets in the Bangla language makes the
task of authorship attribution quite tricky. Most researchers in Bangla have
used datasets containing 3-6 authors on average. For this purpose, we create a
dataset of 16 authors and name it BAAD16. It is the largest dataset for
authorship attribution in Bangla, containing texts from 16 authors and 750
words per sample with 17966 samples in total (More description in Section
3.1), unlike most datasets which have up to 10 authors at most (Phani et al.,
2016; Das et al., 2015; Islam et al., 2018, 2017). One exception is a dataset
with 23 authors (Ahmad et al., 2020), it is not publicly available for use,
and the word count for each document in this dataset is only 50 words on
average, making it a short-text detection dataset. We also use a dataset of 6
authors called BAAD6, which has 1000 words per document. Both of these
datasets are long-text-containing datasets, which becomes more challenging for
traditional machine learning techniques to handle. A comparison of existing
datasets, along with their respective performance, will be shown in Section 6
Table 9. Besides, a larger corpus of Bangla text is required to perform pre-
training in an unsupervised manner. For this step, we use a News corpus and a
Wikipedia corpus. In total, four datasets were used in different steps of the
experiments. Descriptions of all the datasets used in this work are given
below.

###  3.1. BAAD16: Bangla Authorship Attribution Dataset of 16 authors

This dataset has been created by scraping text from an online Bangla e-library
using a custom web crawler and contains literary works of different Bangla
famous writers. It contains novels, stories, series, and other works of 16
authors. Each sample document is created with 750 words. The aim is to create
a long-text dataset, but the length is often capped by the memory usage of
long input strings and the length of content usable by models to less than
1000 words. Therefore we chose a length of 750 to help provide as much context
as possible for any given piece of text. In Table 1 details are shown about
the dataset, and Figure 2(a) shows the distribution of text samples for each
of the 16 authors. This dataset is imbalanced, as apparent from Figure 2 and
resembles real-world scenarios more closely, where not all the authors will
have a large number of sample texts. This is the largest dataset in Bangla
literature for authorship attribution in terms of the number of words
available per author and the total number of words (13.4+ million). Figure
2(b) shows the distribution of the total number of words and the unique word
count for each author. We further create multiple balanced subsets of this
dataset for various numbers of authors in order to measure model stability
with increasing author numbers. In this case, the datasets are truncated to
the minimum number of samples available per author.

(a) Sample distribution

(b) Words distribution

Figure 2. BAAD16 dataset: Distribution per author. Authors are indicated by their index number. See indices in Table 1 Table 1. BAAD16 dataset statistics Author Id | Author Name | Samples | Word Count | Unique Word
---|---|---|---|---
0 | zahir_rayhan | 185 | 138k | 20k
1 | nazrul | 223 | 167k | 33k
2 | manik_bandhopaddhay | 469 | 351k | 44k
3 | nihar_ronjon_gupta | 476 | 357k | 43k
4 | bongkim | 562 | 421k | 62k
5 | tarashonkor | 775 | 581k | 84k
6 | shottojit_roy | 849 | 636k | 67k
7 | shordindu | 888 | 666k | 84k
8 | toslima_nasrin | 931 | 698k | 76k
9 | shirshendu | 1048 | 786k | 69k
10 | zafar_iqbal | 1100 | 825k | 53k
11 | robindronath | 1259 | 944k | 89k
12 | shorotchandra | 1312 | 984k | 78k
13 | shomresh | 1408 | 1056k | 69k
14 | shunil_gongopaddhay | 1963 | 1472k | 109k
15 | humayun_ahmed | 4518 | 3388k | 161k
Total | 17966 | 13474500 | 590660
Average | 1122.875 | 842156.25 | 71822.25

###  3.2. BAAD6: Bangla Authorship Attribution Dataset of 6 authors

BAAD6 is a 6 Author dataset collected and analyzed by Hemayet et al.
(Chowdhury et al., 2018b). This dataset’s total number of words and unique
words are 2,304,338 and 230,075 respectively. The data was obtained from
different online posts and blogs. This dataset is balanced among the 6 Authors
with 350 sample texts per author. The total word count and number of unique
words per author are shown in Table 2. This is a relatively small dataset but
is noisy given the sources it was collected from and its cleaning procedure.
Nonetheless, it may help evaluate the proposed system as it resembles texts
often available on the Internet.

Table 2. BAAD6 dataset statistics Author | Samples | Word count | Unique word
---|---|---|---
fe | 350 | 357k | 53k
ij | 350 | 391k | 72k
mk | 350 | 377k | 47k
rn | 350 | 231k | 50k
hm | 350 | 555k | 72k
rg | 350 | 391k | 58k
Total | 2100 | 2304338 | 230075
Average | 350 | 384056.33 | 59006.67

###  3.3. News Corpus

This corpus comprises of various Bangla newspaper articles in 12 different
categories. It is significantly large compared to the datasets used in
previous works in Bangla. There are 28.5+ million word tokens in this corpus,
and the number of unique words is 836,509 forming around 3% of the total
vocabulary. Table 3 shows the details of the dataset, along with some
necessary statistical measures.

Table 3. News dataset statistics Category | Samples | Word count | Unique word
---|---|---|---
opinion | 8098 | 4185k | 243k
international | 5155 | 1089k | 86k
economics | 3449 | 909k | 58k
art | 2665 | 1312k | 154k
science | 2906 | 697k | 76k
politics | 20050 | 6167k | 196k
crime | 8655 | 2016k | 128k
education | 12212 | 3963k | 225k
sports | 11903 | 3087k | 174k
accident | 6328 | 1086k | 77k
environment | 4313 | 1347k | 103k
entertainment | 10121 | 2669k | 204k
Average | 7988 | 2377803 | 144342
Total | 95855 | 28533646 | 836509

###  3.4. Wikipedia Corpus

Besides the News corpus, we also test our method by pre-training on a
Wikipedia corpus. A subset of the Bangla Wikipedia text was collected from the
Bangla wiki-dump222collected on 10th June 2019. The files are then merged, and
each article is selected as a sample text. All HTML tags are removed, and the
page’s title is stripped from the beginning of the text. This dataset contains
70,377 samples with the total number of words approximating at 18+ million.
The entire dataset has more than 1.2 million unique words, which is 7% of the
total word count. Compared to the news dataset, this makes the Wikipedia
dataset more varied regarding the types of words used.

##  4\. Methodology

###  4.1. Overview

Transfer learning generally implies three basic steps: (i) Pre-training, (ii)
Fine-tuning, and (iii) Downstream tasks. In this case, the downstream task is
authorship attribution. We pre-train the models from scratch due to the
unavailability of pre-trained Bangla language models. First, we tokenize our
texts in one of three ways - word, subword, or character. Then we train the
language model on a large Bangla text corpus. To perform the downstream task,
we fine-tune the language model using text from the tokenized authorship
attribution dataset in an unsupervised manner and then finally perform
classification by adding a classifier to the trained language model. The
entire procedure, starting from tokenization to author identification, was
previously shown schematically in Figure 1.

The idea of using such a setup is much like the traditional approach of using
a single word-embedding layer, only here, the language model is a multi-
layered deep neural network on its own, which is capable of holding much more
information in its weights than a single layer of embedding matrix could
(Howard and Ruder, 2018). The traditional approaches of authorship attribution
in Bangla Language, even in deep learning methods, implement a classification
model with word representations or other extracted features, taking minimal
regard for the language on its own. However, the general idea of this research
is to capture the essence of the Bangla language first, then proceed to the
more granular task of classification with the help of language models. After
the training phase, we test with some examples and find that the models
created in our experiments can complete entire Bangla paragraphs somewhat
meaningfully with minimal grammatical or semantic flaws. So, it is safe to
assume that the weights of the language models hold enough relevant
information to understand the patterns at which Bangla words appear one after
another instead of a single word-embedding layer, which only represents the
relationship among words.

###  4.2. Tokenization

Traditionally tokenization in Bangla has been carried out at the word level,
separating words by spaces, including punctuation or special characters as
separate tokens. Nevertheless, Bangla language has some distinct
characteristics which make a mere separation by words less meaningful.
Lemmatization and stemming provide a way for removing inflection, but this
process remains arduous, and only basic rule-based systems are available to
perform this task in Bangla. Besides, the removed parts of the words are not
always wholly meaningless and may provide meaning in terms of gender, person,
case, number, or animacy. Information provided by declension is also a
necessary part of the language to consider. Therefore, besides word-level
basic tokenization, we also perform sub-word and character level tokenization
and compare the various methods’ effects in the final result. All these
methods are described elaborately in the following subsections, and the
tokenization of a sample text is shown in Figure 3.

####  4.2.1. Word level

Word level tokenization is performed mainly by considering words separated by
spaces as separate tokens. Besides, words that occur less than three times are
discarded, considering them as rare words, names, or misspellings. We select
the most frequent 60,000 words as the vocabulary to proceed to the network.
Unknown words are replaced with the $<$unk$>$ token. Some specialized tokens
are also added, such as the beginning of a string $<$bos$>$, end of a string
$<$end$>$, padding $<$pad$>$, character repetition, and word repetition
representing tokens, etc.

Figure 3. Example of three kinds of tokenization of a sample text. Punctuation
added for demonstration purposes.

####  4.2.2. Sub-word level

Sub-word tokenization means dividing a word into parts and using each part as
a token. Despite having considerable inflection, there is no specialized sub-
word tokenizer for Bangla language. To tackle this problem, sub-word
tokenization was performed using SentencePiece tokenizer (Kudo and Richardson,
2018). SentencePiece is a language independent tokenizer that tokenizes raw
sentences in an unsupervised manner. Using SentencePiece, the unigram
segmentation algorithm (Kudo, 2018) was employed to create the sub-word
vocabulary. For training, we chose 30,000 most frequent tokens (Czapla et al.,
2019). Tokens appearing less than three times were discarded and replaced with
$<$unk$>$ token. Other tokens include $<$s$>$ as the start and $<$/s$>$ as the
end of a sentence.

####  4.2.3. Character level

In character level tokenization, a sentence is split into characters, and each
character is considered a token. In this case, the total number of tokens is
much smaller, 188 in our case, by combining Bangla, English alpha-numerals,
and special characters. The language model generates one character at a time,
concatenated to form words, sentences, and even paragraphs. Besides the
vocabulary, we also use special tokens, just as was used in word-level
tokenization. The use of character-level tokenization not only reduces the
vocabulary size drastically but also removes the bottleneck for out-of-
vocabulary words, misspellings, etc. Characters can be used to build
correlation among groups of characters despite the inflection and declension
that occur and thus add additional meaning to the word. Related words can be
kept as they are, without discarding any part and losing any information, and
still be recognized as being related.

(a) Language Model Architecture

(b) Classification Model Architecture

Figure 4. Simplified diagram of the architectures. The base (embedding and
LSTM layers) of the model remain the same to transfer learned weights, but the
classifier parts are changed when the tasks are changed.

###  4.3. Proposed Architecture

####  4.3.1. Language Model

The patterns in the writings of authors recur, and this nature of the task
suggests the use of a recurrent neural network (RNN). A simplified diagram of
the architecture is shown in Figure 4(a). The architecture consists of an
encoder and a decoder part. The encoder network starts with an Embedding layer
with an embedding size of 400, followed by 3 regular LSTM layers, each with
1150 hidden nodes. It has a few short-cut connections and numerous drop-out
hyper-parameters but does not use any attention mechanism. The decoder is
formed by a dense and a softmax linear layer that provides the probabilistic
estimations for the next word over the vocabulary. Input passes through the
encoder, and the outputs from the LSTMs are passed on to the decoder, where
the prediction is generated. We used Adam optimizer (Kingma and Ba, 2014) and
flattened cross-entropy loss function to train our models.

In this case, we employed a particular variant of the LSTM called the AWD-LSTM
(Merity et al., 2017). It stands for ASGD(Average Stochastic Gradient Descent)
Weight-Dropped LSTM. This model provides special regularization techniques
such as drop-connect (Wan et al., 2013) and other optimizations that make it a
suitable choice for generalizing context and language modelling. In
traditional models, overfitting is a significant issue. Drop-connect handles
this issue by randomly selecting the activation subset on the hidden-to-hidden
weight matrices. This preserves the LSTMs ability to remember long-term
dependency yet not overfit. Figure 5 provides a graphical example of the drop-
connect network.

Figure 5. DropConnect Network (dro, 2022)

Among some other techniques is the generation of a dropout mask on the first
call, namely the variational dropout (Gal and Ghahramani, 2016). Embedding
Dropout (Gal and Ghahramani, 2016) and reduction of embedding size helped the
architecture in achieving high performance in language modelling (Merity et
al., 2018). This model is therefore chosen for the successful application of
transfer learning (Howard and Ruder, 2018).

####  4.3.2. Classifier

The classifier is built on top of the language model by only changing the
decoder part of the model. Two linear layers are added with batch
normalization, and dropout (Howard and Ruder, 2018). Activation function ReLU
is used for intermediate layers and softmax for the output layer, which
outputs the probability distribution among the given authors. The author with
the highest probability value is then selected as the predicted author. A
diagram of the classifier architecture is shown in Figure 4(b).

###  4.4. Training techniques

In recent research, training neural networks efficiently has become a
cumulative task of proper architecture selection, suitable techniques
application, and fine-tuning. This section briefs the techniques used in this
paper.

####  4.4.1. 1cycle policy

The learning rate is one of the most crucial hyper-parameter that drastically
affects experiments and has been manually tuned for quite some time. A unique
approach was published in this regard (Smith, 2015). In this method, a single
trial is run over the dataset, starting with a low learning rate and is
increased exponentially batch-wise. The loss observed for each value of the
learning rate is recorded. The point of decreasing loss and a high learning
rate is selected as the optimal value of the learning rate. Examples of
learning rate finder diagrams obtained in our different experimental steps are
shown in Figure 6.

(a) Cyclical learning rate plot achieved
during the pre-training phase.

(b) Cyclical learning rate plot achieved
during the fine-tuning phase.

(c) Cyclical learning rate plot achieved
during the classification phase.

Figure 6. The cyclical learning rate for various stages of training. The
entire dataset is swept once, and losses for various learning rates(in log
scale) are plotted as the blue line. The red points show the suggested
learning rate, where the loss has a negative slope and continues as such for a
while.

####  4.4.2. Discriminative learning rate

The chosen learning rate is often fixed throughout the training process and
among all the layers. The various layers in a deep learning model contain
different levels of information, and sometimes it may be necessary to allow
for some information to be preserved or changed slowly or fast compared to the
others. A technique known as discriminative learning rate (Howard and Ruder,
2018) solves the problem. This method ensures that the later layers of a
neural network train faster than the base layers by applying a range of
learning rates across different layers. Building deep learning models on top
of the embedding layers has shown promising results in the recent past, which
implies that the later layers of the model need to be modified. This can be
accomplished using a higher learning rate for the last layers, whereas for the
base layers, we use a low learning rate to slowly change the pre-trained
weights and retain the most learned information. Figure 7 illustrates a
generic deep learning architecture to show discriminative learning rates.

Figure 7. Application of Discriminative Learning Rates (Lynn-Evans, 2019). The
initial layers (red) have a lower learning rate to retain the pre-trained
weights, while later layers (green) have a higher learning rate to tune faster
to the task at hand.

####  4.4.3. Gradual unfreezing

When a task-specific model is trained on top of a pre-trained model in
transfer learning, all the layers’ weights are altered together. This method
carries noise back to the base layers from the newly attached (randomly
initialized) layers. To prevent such abrupt alteration and, therefore,
catastrophic forgetting, we employ gradual unfreezing (Howard and Ruder,
2018). This method first freezes the initial layers and trains only the last
layers; then, the pre-trained layers are unfrozen one by one and trained
further to tune to the problem-specific domain. Thus, the learned information
of the pre-trained model is used efficiently.

####  4.4.4. Slanted triangular learning rates

The original concept of stochastic gradient descent (Loshchilov and Hutter,
2016) tries to employ the idea that the neural networks should be getting
closer to the loss’s global minimum value. As the global minimum for loss gets
closer to the minimum, the learning rate should decrease quite obviously. If
it doesn’t, the system may fall in an infinite loop as it jumps from one side
of the global minima to another because it keeps subtracting the multiple of
the large learning rate selected. Slanted triangular learning rate (STLR)
linearly increases the learning rate to converge quickly and then linearly
decays it according to an updated schedule to tune the parameters (Howard and
Ruder, 2018).

####  4.4.5. Stochastic Gradient Descent with Restarts

The neural network training phase will likely encounter local minima in the
loss curve rather than the global minimum. The model may get stuck at the
local minima instead of reaching for the global minima in such cases. A
technique that approaches this problem involves a sudden increase in learning
rates hoping that it will ’jump’ over the local minima if there are any. The
process is called stochastic gradient descent with restarts (SGDR) and was
introduced in (Loshchilov and Hutter, 2016). SGDR suddenly increases the
learning rate and decreases it again by cosine annealing. Figure 8 shows how
learning rates are restarted after every epoch to avoid the problem.

Figure 8. Resetting Learning Rate after each Epoch and reducing gradually by
cosine annealing (Lynn-Evans, 2019)

###  4.5. Training

The primary approach to performing transfer learning with the help of language
models closely follows ULMFiT (Howard and Ruder, 2018). The entire procedure
is divided into three broad steps, as shown previously in Figure 1. Each step
has been explained in the following subsections.

Figure 9. Text generation from the different trained language models.

####  4.5.1. Language model Pre-training

Two reasonably large Bangla datasets have been used to perform this training
step. The aim is for the model to _learn_ the Bangla language through language
modeling. For this, we used the Wikipedia dataset and News dataset separately;
these contain a variety of Bangla texts that can generalize to most text that
may appear in downstream tasks. For all the models, batch size of 32, back-
propagation-through-time (bptt) of 70, and weight decay of 0.1 are used on all
layers. We used a drop-out multiplier of 0.5 to the ratios selected in (Howard
and Ruder, 2018) for all the layers to avoid over-fitting. An appropriate
learning rate for each model is selected through a cyclic learning rate finder
(Smith, 2015). This is depicted in Figure 6. Sometimes learning rates needed
to be altered as training progressed, as determined by the loss of the
validation set. For example, when loss starts to increase, the model is
trained from a previous checkpoint with a lower learning rate. This is shown
as comma-separated learning rates in Table 4. Each epoch employed stochastic
gradient descent with restarts (SGDR) mentioned in Section 2. All models are
trained until they start to show signs of over-fitting, which is 30 epochs for
the word and sub-word levels and 15 epochs for the character-level models.
Important hyper-parameters are mentioned in Table 4 for all the models.

Table 4. Pre-training Language model hyper-parameters Dataset | Tokenization | Epoch | Batch size | Learning rate
---|---|---|---|---
News | Word | 30 | 32 | 1e-3
Sub-word | 30 | 32 | 1e-2,1e-3
Character | 15 | 32 | 1e-2,1e-6,1e-2
Wiki | Word | 30 | 32 | 1e-2,1e-3
Sub-word | 30 | 32 | 1e-3
Character | 15 | 32 | 1e-2

After training, the models begin to learn Bangla. When provided with one or
more initial words, the models can complete entire paragraphs. Illustrations
of models’ prediction are shown in Figure 9 from the Wikipedia dataset trained
model for different levels of tokenization. All three models were given the
starting words ’Three months’ in Bangla, and the rest were generated. Note
that the outputs were tokens that were bound together, removing extra spaces
to form words where required (except in word-level tokenization, where output
is already in words). English translations for the generated text are provided
under each paragraph. We reproduced all grammatical and semantic errors
produced by the model in English as well. Although the text, on the whole,
does not make much sense, the model has learned to form reasonable phrases and
words. This learning is leveraged in the fine-tuning and classification
stages.

####  4.5.2. Language model Fine-tuning

After training the language model on general text, it is fine-tuned to the
task-specific type of text. Each model from the previous section (pre-trained
on news and Wikipedia dataset) is then fine-tuned on two authorship
attribution datasets as mentioned in Section 3. This step allows the model to
get accustomed to the authorial writing style. The target datasets are split
into training and testing sets in an 80%-20% ratio, and only the training set
is used in this phase. The original language model is first loaded and frozen
to tune the model except the last layers that hold the most task-specific
information. Using cyclic learning rate finder (Smith, 2015), the learning
rate for each model is determined. The models were trained for two epochs and
then trained for two more epochs by unfreezing another layer group, repeating
it one more time before unfreezing the entire model. The models are trained as
long as the loss keeps decreasing, and training is halted on signs of over-
fitting. Learning rates are altered as required by observing the loss of the
validation data, depicted by comma-separated values in Table 5. The rest of
the model settings are kept the same as before. Batch size, along with other
hyper-parameters is summarized in Table 5.

Table 5. Fine-Tuning Language model hyper-parameters | Pre-training
---
Dataset
Tokenization |  | Fine-tuning
---
Dataset
Epoch | Batch size | Learning rate
News | Word | BAAD16 | 19 | 32 | 1e-3
BAAD6 | 10 | 32 | 1e-2
Sub-word | BAAD16 | 22 | 32 | 1e-3,1e-4
BAAD6 | 10 | 32 | 2e-2
Character | BAAD16 | 14 | 32 | 1e-2
BAAD6 | 10 | 128 | 1e-3
Wikipedia | Word | BAAD16 | 26 | 32 | 1e-2,1e-3,1e-4
BAAD6 | 10 | 32 | 1e-2
Sub-word | BAAD16 | 24 | 32 | 1e-2,1e-3
BAAD6 | 10 | 32 | 2e-2
Character | BAAD16 | 20 | 32 | 1e-2
BAAD6 | 10 | 128 | 1e-1,1e-2,1e-3

####  4.5.3. Classification

The final downstream task is classification. The model is modified, as
mentioned in Section 4.3.2. The encoder weights from the fine-tuned language
model are loaded into the classifier. The decoder weights are randomly
initialized. The dropout multiplier is set to 0.5, and the chosen learning
rate is 1e-2 using a cyclical learning rate finder. Momentum values of 0.8 and
0.7 are used for optimization (Howard and Ruder, 2018). The entire model is
frozen except the decoder, which is available for training. After training
with the training set of the authorship attribution dataset for two epochs, a
layer group is unfrozen and trained using sliced learning rates for two
epochs. Slicing mainly distributes learning rates among the layers so that the
initial layers are updated slowly to maintain the pre-trained weights. In
contrast, the later layers, which are the most task-specific layers, are
updated swiftly to learn the task at hand. A general rule of slicing learning
rate has been followed where the slice consists of initial learning rate(lr)
and $\frac{lr}{2.6^{4}}$ (Howard and Ruder, 2018). The unfreezing step is
carried out one more time and trained for two epochs before unfreezing the
entire model and training for six epochs. The training was stopped when it
started to overfit. We used a batch size of 32 for all the models at this
stage.

##  5\. Experiments

Previous state-of-the-art models were re-created to compare the proposed
models against baselines, and results were drawn in terms of accuracy and F1
score. Neither of the previous models uses transfer learning from entire
models. They are based on using pre-trained character and word embeddings to
boost performance. Based on experiments from (Khatun et al., 2019), and
(Chowdhury et al., 2018b), we select the best performing models for word
embeddings (using fastText skip-gram) and character embedding and evaluate
them on BAAD16 and BAAD6 datasets. Besides, we also experiment with a
transformer-based multilingual model called mBERT and a similar model trained
only on Bangla text called Bangla-BERT. All the models used for comparison are
discussed in brief below. Table 6 shows the various hyper-parameters used for
the models described below.

###  5.1. CNN-LSTM Word level classifier

The CNN-LSTM model (Khatun et al., 2019) comprises of a mix of convolutional
and LSTM layers suitable for training the corpus on word-level tokenization.
The architecture consists of an embedding layer that is initialized with pre-
trained word-embeddings using skip-gram of fastText. Vocabulary size of 60,000
and embedding vector size of 300 are used. Two convolutional and max pool
layer pairs follow the embedding layer. The outputs from the convolutional
layers are fed into an LSTM layer of 100 neurons, whose outputs are in turn
fed into a fully connected layer of 512 neurons. Dropout is used in the fully
connected layer to prevent the model from overfitting. A final softmax layer
outputs probabilities for the classification. All parameters for the layers
are kept the same as the originally proposed model. The model is trained for
15 epochs with a learning rate of 0.001, decay of 1e-4, and batch size of 128.
We used Adam optimizer and categorical cross-entropy as the loss function.

###  5.2. CNN Character level classifier

This model aims to use character signals as embedding and passes them on to
several convolutional layers to extract information and later perform
classification. The original CNN model (Khatun et al., 2019) has been
recreated and trained for the entire dataset. The first layer consists of the
embedding layer whose weights are set as the pre-trained weights (Khatun et
al., 2019) obtained from training the news dataset mentioned in Section 3. The
vocabulary and embedding size is 181. The vocabulary is the same as the one
used with the proposed character-level AWD-LSTM model except for the special
tokens. The embedding layer is connected to 4 sequentially connected
convolutional layers. Each convolutional layer is followed by a max pool layer
of kernel size 3. A fully connected layer is stacked on top with the number of
nodes set as 512. The number of filters and kernel sizes for the convolutional
layers is kept the same as in the original model. The last layer is the output
layer with softmax activation. With a batch size of 128 and a maximum text
sample length of 3000, the model has been trained on the dataset for 15
epochs. A dropout rate of 0.5 is used in the fully connected layer as
regularization. The rest of the hyper-parameters are kept consistent with the
CNN-LSTM model.

###  5.3. Multilingual BERT

In recent times transfer learning with transformer-based models has shown
promise in many NLP tasks. These models are large and are pre-trained on huge
amounts of text in an unsupervised manner. One of the most widely used models
is BERT (Bidirectional Encoder Representations from Transformers) (Devlin et
al., 2018). Multilingual BERT (mBERT) has Bangla as part of its training
corpus, among 100+ other languages. BERT’s architecture is a multi-layer
bidirectional Transformer encoder based on the original Transformer (Vaswani
et al., 2017). BERT’s uniqueness is that it uses bi-directional context
instead of only left-right context as in autoregressive language modeling
tasks. BERT uses WordPiece embeddings (Wu et al., 2016) where a word maybe be
broken into multiple pieces, and each piece is prefixed with the symbols ## to
indicate their origins. The first token of all sequences is a special token
[CLS]. The final hidden state of this token is used as an aggregate sequence
representation. BERT has three kinds of embeddings: word embedding, positional
embedding, and sentence embedding. And the final embedding of the model is the
sum of these three embeddings. We used the small version of BERT which has L =
12 attention blocks, A = 12 self-attention heads and the hidden size is H =
768 (Devlin et al., 2018).

For classification, we only concern ourselves with the output of the [CLS]
token present at the beginning of every sentence. We add a single-layer
classifier on top of the BERT encoder. Specifically, the final hidden vector
$C\in\rm I\\!R^{H}$ corresponding to the input token [CLS] is input to the
classifier layer, and the number of outputs is the number of class labels; in
our case, the number of authors in the dataset. The entire model is then fine-
tuned end-to-end along with the new layer. For our task of authorship
attribution in Bangla, we use the trained, multilingual BERT model called
_bert-base-multilingual-cased_. We use batch size of 6 and total sequence
length of 512 as per the original setting. Unlike the original BERT, we use
Slanted Triangular learning rates, Discriminative Learning rates, and gradual
unfreezing, similar to our proposed approach. We select the learning rates at
various stages using a learning rate finder. Thus we choose an initial
learning rate of 0.0004 for the BAAD16 dataset and train the classifier for 40
epochs. For the BAAD6 dataset learning rate of 0.0008 was selected and was
trained for 35 epochs. The number of epochs was selected based on the
validation set errors. When the validation set errors start to increase or
oscillate, the training is stopped.

###  5.4. Bangla-BERT

BERT based models have been used extensively in various tasks recently. To
this end, a BERT variation was trained purely in Bangla by Sarkar (Sarker,
2020). In contrast to mBERT, this model incorporates more Bangla training data
than mBERT. It was trained on the Bangla common crawl corpus from
OSCAR333https://oscar-corpus.com/ as well as the Bangla Wikipedia dump
dataset. This uses the same _bert-base_ architecture as in mBERT and consists
of 110M parameters. We use the _sagorsarker/bangla-bert-base_ checkpoint from
the Hugging Face Transformers library for this purpose. For this model also,
we use a batch size of 6 and a maximum sequence length of 512 as per the
original model’s settings. To train for the BAAD16 dataset, we choose an
initial learning rate of $1.32E-6$ using the learning rate finder. The model
was fine-tuned for a total of 40 epochs using gradual unfreezing, and the
learning rate was adjusted using the learning rate finder at each unfreezing
step. The training was stopped when the validation set performance started
degrading, and the loss started increasing. For BAAD6, the initial learning
rate was $1.74E-3$, and it was trained for a total of 30 epochs following
similar unfreezing and learning rate adjustment methods.

Table 6. Hyper-parameters of other models used for comparison. | Pre-training
---
dataset
Model | Tokenization |  | Fine-tuning
---
dataset
Epoch |  | Batch
---
size
| Learning
---
rate
News (fastText) | CNN-LSTM | Word | BAAD6 | 15 | 128 | 1e-3
BAAD16 | 15 | 128 | 1e-3
News (Embedding layer) | CNN | Character | BAAD6 | 15 | 128 | 1e-3
BAAD16 | 15 | 128 | 1e-3
Wiki dump | mBERT |  WordPiece (Sub-word) | BAAD6 | 35 | 6 | 8e-4
BAAD16 | 40 | 6 | 4e-4
Wiki dump, Commoncrawl | Bangla-BERT |  WordPiece (Sub-word) | BAAD6 | 30 | 6 | 1.74e-3
BAAD16 | 40 | 6 | 1.32e-6

##  6\. Results and Discussions

###  6.1. Train and Test splits

After training, the models were tested with the held-out test set. BAAD6 has a
separate test set with 12.5% data (Chowdhury et al., 2018b). The same test set
was used for the purposes of consistency. BAAD16, on the other hand, holds out
20% of the dataset from each author as a test set. This dataset was structured
in line with the one used in (Khatun et al., 2019) to be able to compare and
draw from its results directly. Since the number of samples is not in the
millions, test sizes smaller than 20% are not representative, and choosing
test sizes greater than 20% reduces training data significantly since the
training set is used for both tuning the language model as well as to learn
classification.

###  6.2. Metrics

All the models were measured in terms of accuracy and F1 score. Although BAAD6
is balanced, BAAD16 is not (refer to Section 3), for which a look at the
F1-score is also necessary. Accuracy measures the percentage of samples
correctly identified, whereas the F1 score is the harmonic sum of the
precision and recall. F1-score thus gives us the ability to look into pieces
correctly identified from each class in a comparable form. Table 7 shows the
summary of the results obtained from various experimented models against all
the variations of the proposed model.

Table 7. Results of Classification Dataset | Model | Tokenization |  | Pre-traning
---
Dataset
| FiT
---
Perplexity
Accuracy % | F1 Score
BAAD16 | ULMFiT | Word | News | 74.67 | 99.58 | 0.9855
Wiki | 60.91 | 99.67 | 0.9967
Sub-word | News | 62.45 | 99.80 | 0.9980
Wiki | 57.85 | 99.72 | 0.9972
Character | News | 3.42 | 98.55 | 0.9855
Wiki | 3.36 | 98.58 | 0.9858
Char-CNN | Character | News | - | 86.28 | 0.7981
CNN-LSTM | Word | News | - | 93.82 | 0.8934
mBERT | Sub-word | Wiki | - | 94.79 | 0.9283
Bangla-BERT | Sub-word |  | Wiki,
---
Common-crawl
- | 96.80 | 0.9533
BAAD6 | ULMFiT | Word | News | 203.11 | 94.67 | 0.9469
Wiki | 149.58 | 94.33 | 0.9437
Sub-word | News | 233.04 | 95.33 | 0.9536
Wiki | 260.90 | 94.67 | 0.9476
Character | News | 4.47 | 83.67 | 0.8360
Wiki | 3.71 | 90.00 | 0.9007
Char-CNN | Character | News | - | 73.33 | 0.7147
CNN-LSTM | Word | News | - | 66.33 | 0.6320
mBERT | Sub-word | Wiki | - | 93.33 | 0.9301
Bangla-BERT | Sub-word |  | Wiki,
---
Commoncrawl
- | 90.33 | 0.9012

###  6.3. Results

From Table 7, we see that the proposed methods of transfer learning using
language modeling with AWD-LSTM architecture outperform the previous high-
performing models in Bangla literature in the target datasets. Specifically,
the sub-word tokenized models have consistently performed well. The mBERT
model performs well considering that it is pre-trained on over 100 languages,
of which only a tiny portion consists of Bangla texts. But counter-
intuitively, a pure Bangla-BERT model does not seem to outperform the mBERT
model. One hypothesis could be the size of the dataset we use to train the
model. The model may be over-parameterized for the current Authorship
Attribution datasets, and training more and more epochs leads to a memorizing
issue, ultimately causing the validation and test sets to perform poorly.
Nevertheless, we only train up to the point where the validation loss stops
decreasing, thus getting the best possible model checkpoint from the available
datasets.

###  6.4. K-fold validation

To test the robustness of our proposed systems, we perform K-fold cross-
validation on BAAD6 and BAAD16 datasets with the best performing model, i.e,
the News dataset pre-trained sub-word level ULMFiT model. We perform a 5-fold
cross-validation and average the accuracies and F1 scores across the folds.
Table 8 shows the results along with the margin of error at a 95% confidence
interval. The authorship-attribution fine-tuned model for each dataset was
used, i.e, we start from the _classification_ level of training. They were
trained with an early stopping callback of 2 epochs. For both datasets and all
folds, the newly initialized layer was trained for 2 epochs, followed by 2
epochs with a layer group unfrozen, followed by another 2 epochs with another
layer group unfrozen. Then the completely unfrozen model was trained until the
loss stopped decreasing. For BAAD6, it was a total of 10 epochs; for BAAD16,
the model trained for about 10-13 epochs in various folds. Besides K-fold
validation, more minor variations of BAAD16 were also tested separately. The
results of this analysis can be found in Section 6.9.

Table 8. Results of 5-fold cross-validation for the subword ULMFiT models on BAAD6 and BAAD16 datasets. The margin of error at 95% confidence interval was reported alongside the mean accuracy and F1-scores | Pre-training
---
Datset
Tokenization | Model | Dataset | Accuracy % | F1 Score
News | Sub-word | ULMFiT | BAAD6 |  | 95.0952
---
±0.339 (±0.36%)
| 0.9511
---
±0.00347 (±0.37%)
BAAD16 |  | 99.2597
---
±0.436 (±0.44%)
| 0.9924
---
±0.00466 (±0.47%)

###  6.5. Comparative analysis

In addition, we compare our proposed model to most existing research in Bangla
on authorship attribution in Table 9, demonstrating that our technique has
outstanding performance when considering the size of the corpus, length of
samples utilized, and the number of authors categorized. Most works use self-
created datasets with very few authors, possibly bordering on topic
classification. Some authors tend to write on specific topics, so
classification becomes very easy and unlike real-life scenarios, therefore
achieving high accuracy. The publicly available dataset BAAD6 is a slightly
noisy dataset and was used to evaluate the proposed system and some of the
previous approaches. Our approach shows improved accuracy in this dataset at
95.33% from the previous 92.9% using fastText word embedding (Chowdhury et
al., 2018b).

Table 9. Comparison of the proposed system with existing works in Bangla authorship attribution (ascending order of the number of authors in the dataset). The * mark indicates that the work was replicated in this paper and applied to the respective dataset. Paper/Dataset |  | No. of
---
author
| Average
---
word /
author
| Average
---
samples /
author
Method |  | Accuracy
---
(%)
Das et al. (Das and Mitra, 2011) | 3 | 325k | 12 |  | Probabilistic classification
---
unigram + vocabulary richness
91.67
| Probabilistic classification
---
bigram + vocabulary richness
100
| Chakraborty
---
et al. (Chakraborty, 2012)
3 | 4921k | 150 | Extracted features with SVM | 83.3
| Anisuzzaman
---
et al. (Anisuzzaman and Salam, 2018)
3 | 35k | - | N-grams with Naive Bayes | 95
Phani et al. (Phani et al., 2017) | 3 | 104k | 1000 |  | Tfidf of N-grams of both word &
---
character, stop words with SVM
98.93
| Topic modeling as features
---
with Naive Bayes
100
Islam et al. (Islam et al., 2018) | 5 | - | 394 |  | N-grams and extracted features
---
with MLP
89
Hossain et al. (Hossain et al., 2017) | 6 | 185k | 100+ |  | N-grams and extracted features
---
with voting classifier
90.67
Chowdhury et al. (Chowdhury et al., 2018b) (BAAD6) | 6 | 384k | 350 | Word embedding with MLP | 85.46
Word embedding with LSTM | 89.6
Word embedding with CNN | 92.9
Chowdhury et al. (Chowdhury et al., 2018a) (BAAD6) |  | Word embedding with
---
Naive Bayes
61
Word embedding with SVM | 84.4
| Word embedding with
---
heirarchical classifier
80.8
| Khatun
---
et al. (Khatun et al., 2019) (BAAD6)*
Character Embedding with CNN | 73.3
| Khatun
---
et al. (Khatun et al., 2019) (BAAD6)*
| Word Embedding with
---
CNN-LSTM
66.3
| Proposed Model
---
(BAAD6)
| Transfer learning using
---
Language Model
95.33
Islam et al. (Islam et al., 2017) | 10 | - | 312 |  | N-grams and POS tags
---
with Random forests
96
| Khatun
---
et al. (Khatun et al., 2019) (BAAD16)*
16 | 842k | 1122 | Character Embedding with CNN | 86.28
| Khatun
---
et al. (Khatun et al., 2019) (BAAD16)*
| Word Embedding with
---
CNN-LSTM
93.82
| Proposed Model
---
(BAAD16)
| Transfer learning using
---
Language Model
99.8

Furthermore, our curated dataset, BAAD16, was also used to compare the
performance of some of the previously published models by replicating those
architectures (details in Section 5). Performance improves from appx. 94% (of
mBERT, CNN-LSTM, etc) to 99.8%, which is a significant improvement.

###  6.6. Error analysis

A small number of false classifications were briefly analyzed to reveal that
the model confuses similar writings of different authors. It is not uncommon
for writers to follow another famous writer or any of their peers. Besides,
some writers of the same period also tend to follow similar writing patterns.
Despite these, most cases were in-fact random errors whose correlations were
not found in this brief analysis, at least not manually. An in-depth analysis
of the errors can help reveal more about what the models learn in terms of
understanding the structure and semantics of a language which remains a scope
for future study.

The effects of various other factors on the obtained results, such as
architecture, tokenization, number of authors, sample distribution, and
dataset used for pre-training, are analysed to test the robustness and
scalability of the proposed system. These are briefly discussed below.

###  6.7. Model Effect

To assess the effects of ULMFiT model for the task of authorship attribution,
we train some previous state-of-the-art models, namely Char-CNN and word-level
CNN-LSTM. From Table 7, it is evident that the ULMFiT model outperforms the
other models by a significant amount on both BAAD6 and BAAD16 datasets. This
clearly shows that the transfer learning approach is effectively applicable to
the task of authorship attribution. The model learns Bangla from language
model training and tunes to authorial writing styles on fine-tuning. Thus,
with the additional steps of teaching the model a language, it can better
detect the text’s original author. Besides this, the AWD-LSTM architecture,
along with the training techniques employed, offers a strong base for the
language model as well as the classifier making the transfer learning thereof
highly effective (Howard and Ruder, 2018).

###  6.8. Tokenization Effect

Tokenization is an integral part of any NLP task. In Bangla, the most common
way so far has been to tokenize by words. In this paper, three types of
tokenizations are attempted, and their effects are analyzed on the task of
author classification. As described in Section 4.2, we have performed word,
sub-word, and character level tokenization. In general, the character level
models perform worse than both word and sub-word models. This reflects that
the character-level model faces difficulty choosing the correct author because
of the long stream of tokens it has to go through before reaching any
conclusion. LSTM layers pass one character at a time, making any sentence a
long set of tokens. Therefore, the LSTM layers may have trouble gathering
enough data about each author’s difference in text or style.

The word and sub-word level models perform nearly equally well, but the sub-
word models begin to outrun the word-level models on further epochs. This
information can be drawn from Figure 11, where we see the progress for each
tokenization type’s model on both BAAD16 and BAAD6 in a zoomed-in scale for
easy understanding. A more zoomed-in graph in Figure 11 shows the rising
accuracy of the sub-word level model on each epoch, indicating a specific
comparison of word and sub-word level models. The sub-word tokenization breaks
down the text into multiple parts but not wholly. As a result, linguistic
information about the words is kept intact, and it also provides information
about the relationship between structural components of words. For example, it
separates inflected prefixes or suffixes from words, so the model retains root
word information as well as gets additional information from the separated
parts. Whereas when tokenized by words, the relationship between related words
with slight structure variations is not easily recognized.

(a) On BAAD16

(b) On BAAD6

Figure 10. The accuracy versus epoch plot shows the models’ performance
comparison with different tokenizations. Here the character tokenized model
cannot reach the level of subword and word tokenized models in both the tested
datasets (BAAD6 and BAAD16). Notice the vertical axis does not start from zero
for clarity.

(c) On BAAD16

(d) On BAAD6

Figure 11. The accuracy versus epoch plot shows the models’ performance
comparison with word and subword tokenizations. It is an extended version of
the previous figure, showing the two models advancing exceptionally closely
together. For clarity, the vertical axis does not begin at zero.

###  6.9. Effect of number of Authors

In this section, the proposed models’ results are compared with varying
numbers of authors to determine the models’ effectiveness against increasing
classes trained with fewer samples. BAAD16 contains 16 authors in an
imbalanced manner. It is chunked into five parts containing 6, 8, 10, 12, and
14 authors, respectively, randomly taking subsets from the original dataset.
The samples per class in the derived datasets are truncated to the minimum
number of samples among the classes. We have only compared the accuracy
because all the models are now being trained and tested on balanced datasets.
Accuracy is measured on 20% held out dataset for each case, after training on
80% of the data. Each of the six pre-trained models of 3 types (word, sub-
word, and character level) is fine-tuned on these five sub-datasets. As we
increase the number of authors, the sample per author also decreases, making
it difficult for any deep learning model to learn each class of text. The
summary is presented in Table 10. A graph depicting the accuracy trend with
increasing number of authors for all the models is shown in Figure 12.

Table 10. Accuracy of authorship attribution for increasing number of Authors. All subsets of the dataset are balanced, and the samples per author are displayed underneath the author number in each column. |  | # of authors on the subset
---
# of samples per author
Tokenization |  Pre-training Dataset | 6 authors | 8 authors | 10 authors | 12 authors | 14 authors
1100 | 931 | 849 | 562 | 469
Word | News | 99.69 | 99.53 | 99.58 | 99.41 | 98.63
Wiki | 99.62 | 99.56 | 99.70 | 99.48 | 99.47
Sub-word | News | 99.47 | 99.40 | 98.71 | 99.56 | 99.39
Wiki | 99.62 | 99.67 | 99.41 | 99.63 | 99.54
Character | News | 98.79 | 98.60 | 98.00 | 98.38 | 97.26
Wiki | 98.79 | 98.66 | 98.18 | 97.78 | 97.11
Figure 12. Performance of different models with increasing number of authors.
The models vary by tokenization and the dataset it was pre-trained on,
depicted in the legend in the form ’tokenization_pretraining-dataset’. The
vertical axis shows the classification accuracy, and the horizontal axis
indicates the number of authors taken from the BAAD16 dataset.

Out of the five subsets created, the sub-word model pre-trained on the
Wikipedia corpus performs best in the majority of cases (3 in this occasion).
More importantly, as the sample number decreases, all other models more or
less start performing worse than the sub-word tokenized model trained on the
Wikipedia corpus. Figure 12 shows the decline in the character model trained
with lesser samples. Moreover, the word and sub-word level wiki models
consistently perform better, and the sub-word model tends to give higher
accuracy. From these results, we can conclude that the Wikipedia pre-trained
sub-word model shows more stability than others. The reason behind this could
be two-fold. Firstly the Wikipedia dataset has more varied text than the news
dataset. This helps the model generalize better. Secondly, the sub-word level
model provides enough information about word and sentence structure yet does
not break it down too much (into characters) to lose sight of the overall
picture of the sampled text.

###  6.10. Effect of pre-training datasets

Although trained models with the Wikipedia dataset show better downstream
generalization, the scenario changes in real-world applications. Data is often
imbalanced, with some authors having very few training samples compared to
others. The classifier’s loss for each epoch is plotted to compare the effects
of news and Wikipedia pre-trained subword models for each target dataset in
Figure 13.

We see that the trained model with the news dataset works significantly better
in terms of classifier’s loss for both BAAD6 and BAAD16 datasets. The loss
tends to decrease consistently with each epoch in both situations, although
the news model’s loss is substantially smaller than the wiki model’s. It shows
that the news dataset’s categories and size enabled the classifier to
distinguish better among the authors.

(a) On BAAD16

(b) On BAAD6

Figure 13. Loss versus epoch plot showing the effect of the dataset used in
pre-training on classification. The news pre-trained model achieves lower loss
for the task of authorship attribution in our case. Both the models are sub-
word tokenized since sub-word tokenization achieves better performance than
others.

##  7\. Released Pre-trained Language Models

A set of language models have been pre-trained on two different datasets,
namely news and Wikipedia, as part of our workflow. Language models’
performance is measured in terms of perplexity. This measure captures the
degree of uncertainty in predicting the next word of the sentence. It is
calculated as the exponentiation of the obtained loss. Low perplexity is a
sign of a well-trained model. Table 11 lists the perplexities of the pre-
trained models. We see that the character-level models perform significantly
better for the task of text generation. An illustration of the produced text
can be seen in Figure 9. The next best performance is obtained by the sub-word
level model being slightly lower than the word-level models. The reason could
be that the tokens’ representation in smaller forms (e.g. characters) may
provide additional information about the words and relationships among the
words than the larger chunks can provide. Nevertheless, such information is
not reliable for downstream tasks, as evident from our experiments where the
character-level models underperformed compared to their counterparts.

All the pre-trained models have been made available for use and can be applied
to any other Bangla NLP task of choice. The heavy task of pre-training has
already been done. The weights have to be loaded, and the final dense layer
has to be changed to perform the downstream task to use these models. It is to
be noted that the downstream task has to be done in steps and preferably
following the discriminative learning rate procedure so that the pre-trained
weights can be efficiently utilized.

Table 11. Perplexity of the trained Language Models. Dataset | Tokenization | Perplexity
---|---|---
News | Word | 47.52
Sub-word | 43.61
Character | 3.20
Wiki | Word | 83.62
Sub-word | 74.20
Character | 3.08

##  8\. Conclusion and future work

Transfer learning has been proven beneficial for domains with inadequate data
labelling as it is first pre-trained on a resource-rich source domain before
being fine-tuned on a downstream task. Therefore, in this work, we analyze
transfer learning applications on authorship attribution using language
modeling in three phases: Pre-training, Fine-tuning, and Author prediction. In
this article, we present an AWD-LSTM and transfer learning-based scalable
framework that addresses the problem of scarce language resources,
scalability, and manual feature engineering for AABL. Towards that goal,
experiments were conducted using the word, sub-word, and character level
tokenization, demonstrating the efficacy of our approach with different
tokenization. Furthermore, comparative analysis shows that our approach
outperforms all existing traditional methods. To test our model with a larger
dataset and longer text, we build a dataset containing the literary works of
16 authors with more than 13.4 million words in total. Our contribution to
this work includes the proposal of a robust and scalable system for AABL using
transfer learning, a comprehensive analysis of the effectiveness of different
tokenization, a standard dataset, and the release of the pre-trained models
for further use. The proposed method provides stability with fewer data and a
large pool of authors. We conclude that the sub-word tokenized models
consistently perform better regardless of the number of samples from our
experiments. We want to investigate the application of transfer learning on
deep CNN-based architectures and compare them with LSTM architecture with an
attention model (Al Azhar et al., 2021) for further work due to its pattern
detection ability and faster training time. We would also like to make use of
various transformer-based models for this task. Additionally, cross-lingual
authorship attributions and improvement of existing models on such tasks also
interest us.

## References

  * (1)
  * dro (2022) 2022\.  _Regularization of Neural Networks Using DropConnect_.  https://cds.nyu.edu/projects/regularization-neural-networks-using-dropconnect/
  * Ahmad et al. (2020) Sumnoon Ibn Ahmad, Lamia Alam, and Mohammed Moshiul Hoque. 2020\.  An Empirical Framework to Identify Authorship from Bengali Literary Works. In _International Conference on Cyber Security and Computer Science_. Springer, 465–476.
  * Al Azhar et al. (2021) Ibrahim Al Azhar, Sohel Ahmed, Md Saiful Islam, and Aisha Khatun. 2021.  Identifying Author in Bengali Literature by Bi-LSTM with Attention Mechanism. In _2021 24th International Conference on Computer and Information Technology (ICCIT)_. IEEE, 1–6.
  * Anisuzzaman and Salam (2018) DM Anisuzzaman and Abdus Salam. 2018.  Authorship attribution for Bengali language using the fusion of N-gram and Naïve bayes algorithms.  _International Journal of Information Technology and Computer Science (IJITCS)_ 10, 10 (2018), 11–21.
  * Ashrafi et al. (2020) Imranul Ashrafi, Muntasir Mohammad, Arani Shawkat Mauree, Galib Md Azraf Nijhum, Redwanul Karim, Nabeel Mohammed, and Sifat Momen. 2020\.  Banner: A Cost-Sensitive Contextualized Model for Bangla Named Entity Recognition.  _IEEE Access_ 8 (2020), 58206–58226.
  * Bagnall (2016) D Bagnall. 2016\.  Authorship clustering using multi-headed recurrent neural networks—notebook for PAN at CLEF 2016. In _CLEF 2016 Evaluation Labs and Workshop–Working Notes Papers_. 5–8.
  * Barry and Luna (2012) Kurt Barry and Katherine Luna. 2012.  Stylometry for online forums.
  * Bhattacharya et al. (2005) Samit Bhattacharya, Monojit Choudhury, Sudeshna Sarkar, and Anupam Basu. 2005.  Inflectional morphology synthesis for bengali noun, pronoun and verb systems. In _Proc. of the National Conference on Computer Processing of Bangla (NCCPB 05)_. Citeseer, 34–43.
  * Chakraborty (2012) Tanmoy Chakraborty. 2012\.  Authorship identification in bengali literature: a comparative analysis.  _arXiv preprint arXiv:1208.6268_ (2012).
  * Chowdhury et al. (2019a) Hemayet Ahmed Chowdhury, Md Azizul Haque Imon, Syed Md Hasnayeen, and Md Saiful Islam. 2019a.  Authorship Attribution in Bengali Literature using Convolutional Neural Networks with fastText’s word embedding model. In _2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT)_. IEEE, 1–5.
  * Chowdhury et al. (2018a) Hemayet Ahmed Chowdhury, Md Azizul Haque Imon, and Md Saiful Islam. 2018a.  Authorship Attribution in Bengali Literature Using fastText’s Hierarchical Classifier. In _2018 4th International Conference on Electrical Engineering and Information & Communication Technology (iCEEiCT)_. IEEE, 102–106.
  * Chowdhury et al. (2018b) Hemayet Ahmed Chowdhury, Md Azizul Haque Imon, and Md Saiful Islam. 2018b.  A Comparative Analysis of Word Embedding Representations in Authorship Attribution of Bengali Literature.  (2018).
  * Chowdhury et al. (2019b) Hemayet Ahmed Chowdhury, Md Azizul Haque Imon, Anisur Rahman, Aisha Khatun, and Md Saiful Islam. 2019b.  A continuous space neural language model for bengali language. In _2019 22nd International Conference on Computer and Information Technology (ICCIT)_. IEEE, 1–6.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_ (2019).
  * Czapla et al. (2019) Piotr Czapla, Sylvain Gugger, Jeremy Howard, and Marcin Kardas. 2019.  Universal Language Model Fine-Tuning for Polish Hate Speech Detection.  _Proceedings ofthePolEval2019Workshop_ (2019), 149.
  * Dai and Le (2015) Andrew M Dai and Quoc V Le. 2015.  Semi-supervised sequence learning. In _Advances in neural information processing systems_. 3079–3087.
  * Das et al. (2015) Prapti Das, Rishmita Tasmim, and Sabir Ismail. 2015\.  An experimental study of stylometry in bangla literature. In _2015 2nd International Conference on Electrical Information and Communication Technologies (EICT)_. IEEE, 575–580.
  * Das and Mitra (2011) Suprabhat Das and Pabitra Mitra. 2011.  Author identification in bengali literary works. In _International Conference on Pattern Recognition and Machine Intelligence_. Springer, 220–226.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * El Bakly et al. (2020) Abeer H El Bakly, Nagy Ramadan Darwish, and Hesham A Hefny. 2020.  Using Ontology for Revealing Authorship Attribution of Arabic Text.  _International Journal of Engineering and Advanced Technology (IJEAT)_ 9, 4 (2020), 143–151.
  * Fourkioti et al. (2019) Olga Fourkioti, Symeon Symeonidis, and Avi Arampatzis. 2019\.  Language models and fusion for authorship attribution.  _Information Processing & Management_ 56, 6 (2019), 102061\.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  A theoretically grounded application of dropout in recurrent neural networks.  _Advances in neural information processing systems_ (2016).
  * Goldman and Allison ([n.d.]) Erik Goldman and Abel Allison. [n.d.].  Using Grammatical Markov Models for Stylometric Analysis.  ([n. d.]).
  * Hitschler et al. (2017) Julian Hitschler, Esther van den Berg, and Ines Rehbein. 2017\.  Authorship attribution with convolutional neural networks and POS-Eliding. In _Proceedings of the Workshop on Stylistic Variation_. 53–58.
  * Hossain et al. (2017) M Tahmid Hossain, Md Moshiur Rahman, Sabir Ismail, and Md Saiful Islam. 2017.  A stylometric analysis on Bengali literature for authorship attribution.  (2017).
  * Hossain et al. (2020) Md Zobaer Hossain, Md Ashraful Rahman, Md Saiful Islam, and Sudipta Kar. 2020.  Banfakenews: A dataset for detecting fake news in bangla.  _arXiv preprint arXiv:2004.08789_ (2020).
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal Language Model Fine-tuning for Text Classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 328–339.
  * Islam et al. (2018) Md Ashikul Islam, Md Minhazul Kabir, Md Saiful Islam, and Ayesha Tasnim. 2018.  Authorship Attribution on Bengali Literature using Stylometric Features and Neural Network. In _2018 4th International Conference on Electrical Engineering and Information & Communication Technology (iCEEiCT)_. IEEE, 360–363.
  * Islam et al. (2017) Nazmul Islam, Mohammed Moshiul Hoque, and Mohammad Rajib Hossain. 2017.  Automatic authorship detection from Bengali text using stylometric approach. In _2017 20th International Conference of Computer and Information Technology (ICCIT)_. IEEE, 1–6.
  * Jafariakinabad et al. (2019) Fereshteh Jafariakinabad, Sansiri Tarnpradab, and Kien A Hua. 2019\.  Syntactic Recurrent Neural Network for Authorship Attribution.  _arXiv preprint arXiv:1902.09723_ (2019).
  * Jamatia et al. (2020) Anupam Jamatia, Steve Durairaj Swamy, Björn Gambäck, Amitava Das, and Swapan Debbarma. 2020.  Deep Learning Based Sentiment Analysis in a Code-Mixed English-Hindi and English-Bengali Social Media Corpus.  _International Journal on Artificial Intelligence Tools_ (2020).
  * Jewel et al. (2019) Md Jewel, Md Ismail Hossain, and Tamanna Haider Tonni. 2019\.  Bengali Ethnicity Recognition and Gender Classification using CNN & Transfer Learning. In _2019 8th International Conference System Modeling and Advancement in Research Trends (SMART)_. IEEE, 390–396.
  * Karim et al. (2020) Md Karim, Bharathi Raja Chakravarthi, Mihael Arcan, John P McCrae, Michael Cochez, et al. 2020\.  Classification Benchmarks for Under-resourced Bengali Language based on Multichannel Convolutional-LSTM Network.  _arXiv preprint arXiv:2004.07807_ (2020).
  * Khatun et al. (2019) Aisha Khatun, Anisur Rahman, Md Saiful Islam, et al. 2019\.  Authorship Attribution in Bangla literature using Character-level CNN. In _2019 22nd International Conference on Computer and Information Technology (ICCIT)_. IEEE, 1–5.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kreutz and Daelemans (2018) Tim Kreutz and Walter Daelemans. 2018.  Exploring classifier combinations for language variety identification. In _Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects, Santa Fe, New Mexico, USA, August 20, 2018_. 191–198.
  * Kudo (2018) Taku Kudo. 2018\.  Subword regularization: Improving neural network translation models with multiple subword candidates.  _arXiv preprint arXiv:1804.10959_ (2018).
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _arXiv preprint arXiv:1808.06226_ (2018).
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  _arXiv preprint arXiv:1901.07291_ (2019).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Loshchilov and Hutter (2016) Ilya Loshchilov and Frank Hutter. 2016.  SGDR: Stochastic Gradient Descent with Restarts.  _CoRR_ (2016).
  * Lynn-Evans (2019) S. Lynn-Evans. 2019\.  _Ten Techniques Learned From fast.ai_.  https://blog.floydhub.com/ten-techniques-from-fast-ai/
  * Merity et al. (2017) Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017.  Regularizing and Optimizing LSTM Language Models.  _CoRR_ (2017).
  * Merity et al. (2018) Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018.  Regularizing and Optimizing LSTM Language Models.  _CoRR_ (2018).
  * Nishat and Shopon (2019) Zinnia Khan Nishat and Md Shopon. 2019.  Unsupervised Pretraining and Transfer Learning-Based Bangla Sign Language Recognition. In _International Joint Conference on Computational Intelligence_. Springer, 529–540.
  * Pal et al. (2017) Urmee Pal, Ayesha Siddika Nipu, and Sabir Ismail. 2017\.  A machine learning approach for stylometric analysis of Bangla literature.  (2017).
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. 2227–2237.
  * Phani et al. (2016) Shanta Phani, Shibamouli Lahiri, and Arindam Biswas. 2016\.  A machine learning approach for authorship attribution for Bengali blogs.  (2016).
  * Phani et al. (2017) Shanta Phani, Shibamouli Lahiri, and Arindam Biswas. 2017\.  A supervised learning approach for authorship attribution of Bengali literary texts.  _ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP)_ 16, 4 (2017), 1–15.
  * Pramanik and Bag (2020) Rahul Pramanik and Soumen Bag. 2020.  Segmentation-based recognition system for handwritten Bangla and Devanagari words using conventional classification and transfer learning.  _IET Image Processing_ 14, 5 (2020), 959–972.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Ruder et al. (2016) Sebastian Ruder, Parsa Ghaffari, and John G Breslin. 2016\.  Character-level and multi-channel convolutional neural networks for large-scale authorship attribution.  _arXiv preprint arXiv:1609.06686_ (2016).
  * Sari et al. (2018) Yunita Sari, Mark Stevenson, and Andreas Vlachos. 2018\.  Topic or style? exploring the most useful features for authorship attribution. In _Proceedings of the 27th International Conference on Computational Linguistics_. 343–353.
  * Sarker (2020) Sagor Sarker. 2020\.  BanglaBERT: Bengali Mask Language Model for Bengali Language Understading.  https://github.com/sagorbrur/bangla-bert
  * Seroussi et al. (2014) Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2014\.  Authorship attribution with topic models.  _Computational Linguistics_ 40, 2 (2014), 269–310.
  * Shrestha et al. (2017) Prasha Shrestha, Sebastian Sierra, Fabio Gonzalez, Manuel Montes, Paolo Rosso, and Thamar Solorio. 2017\.  Convolutional neural networks for authorship attribution of short texts. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_. 669–674.
  * Smith (2015) Leslie N. Smith. 2015\.  No More Pesky Learning Rate Guessing Games.  _CoRR_ (2015).
  * Stamatatos (2009) Efstathios Stamatatos. 2009\.  A survey of modern authorship attribution methods.  _Journal of the American Society for information Science and Technology_ 60, 3 (2009), 538–556.
  * Sundararajan and Woodard (2018) Kalaivani Sundararajan and Damon Woodard. 2018.  What represents “style” in authorship attribution?. In _Proceedings of the 27th International Conference on Computational Linguistics_. 2814–2822.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Wan et al. (2013) Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013.  Regularization of neural networks using dropconnect. In _International conference on machine learning_. 1058–1066.
  * Wikipedia (2021) Wikipedia. 2021\.  _Bengali Language_.  https://en.wikipedia.org/wiki/Bengali_language
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_ (2016).
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019\.  Xlnet: Generalized autoregressive pretraining for language understanding. In _Advances in neural information processing systems_. 5753–5763.
  * Zhang et al. (2014) Chunxia Zhang, Xindong Wu, Zhendong Niu, and Wei Ding. 2014\.  Authorship identification from unstructured texts.  _Knowledge-Based Systems_ 66 (2014), 99–111.