#  ParsiNLU: A Suite of High-level NLP Tasks for Persian

Daniel Khashabi1 Arman Cohan1 Siamak Shakeri2 Pedram Hosseini3 Pouya
Pezeshkpour4 Malihe Alikhani5 Moin Aminnaseri6 Marzieh Bitaab7 Faeze Brahman8
Sarik Ghazarian9 Mozhdeh Gheini9 Arman Kabiri10 Rabeeh Karimi Mahabadi11 Omid
Memarrast12 Ahmadreza Mosallanezhad7 Erfan Noury13 Shahab Raji14 Mohammad
Sadegh Rasooli15 Sepideh Sadeghi2 Erfan Sadeqi Azer2 Niloofar Safi
Samghabadi16 Mahsa Shafaei17 Saber Sheybani18 Ali Tazarv4 Yadollah
Yaghoobzadeh19
1Allen Institute for AI, 2Google, 3George Washington U., 4UC Irvine, 5U. of
Pittsburgh, 6TaskRabbit, 7Arizona State U., 8UC Santa Cruz 9U. of Southern
California, 10IMRSV Data Labs, 11EPFL, 12U. of Illinois - Chicago, 13U. of
Maryland Baltimore County 14Rutgers U., 15U. of Pennsylvania, 16Expedia Inc.,
17U. of Houston, 18Indiana U. - Bloomington, 19Microsoft

#  ParsiNLU: A Suite of High-level NLP Challenges for Persian

Daniel Khashabi1 Arman Cohan1 Siamak Shakeri2 Pedram Hosseini3 Pouya
Pezeshkpour4 Malihe Alikhani5 Moin Aminnaseri6 Marzieh Bitaab7 Faeze Brahman8
Sarik Ghazarian9 Mozhdeh Gheini9 Arman Kabiri10 Rabeeh Karimi Mahabadi11 Omid
Memarrast12 Ahmadreza Mosallanezhad7 Erfan Noury13 Shahab Raji14 Mohammad
Sadegh Rasooli15 Sepideh Sadeghi2 Erfan Sadeqi Azer2 Niloofar Safi
Samghabadi16 Mahsa Shafaei17 Saber Sheybani18 Ali Tazarv4 Yadollah
Yaghoobzadeh19
1Allen Institute for AI, 2Google, 3George Washington U., 4UC Irvine, 5U. of
Pittsburgh, 6TaskRabbit, 7Arizona State U., 8UC Santa Cruz 9U. of Southern
California, 10IMRSV Data Labs, 11EPFL, 12U. of Illinois - Chicago, 13U. of
Maryland Baltimore County 14Rutgers U., 15U. of Pennsylvania, 16Expedia Inc.,
17U. of Houston, 18Indiana U. - Bloomington, 19Microsoft

#  TACL’21
ParsiNLU:
A Suite of Language Understanding Challenges for Persian

Daniel Khashabi1 Arman Cohan1 Siamak Shakeri2 Pedram Hosseini3 Pouya
Pezeshkpour4 Malihe Alikhani5 Moin Aminnaseri6 Marzieh Bitaab7 Faeze Brahman8
Sarik Ghazarian9 Mozhdeh Gheini9 Arman Kabiri10 Rabeeh Karimi Mahabadi11 Omid
Memarrast12 Ahmadreza Mosallanezhad7 Erfan Noury13 Shahab Raji14 Mohammad
Sadegh Rasooli15 Sepideh Sadeghi2 Erfan Sadeqi Azer2 Niloofar Safi
Samghabadi16 Mahsa Shafaei17 Saber Sheybani18 Ali Tazarv4 Yadollah
Yaghoobzadeh19
1Allen Institute for AI, 2Google, 3George Washington U., 4UC Irvine, 5U. of
Pittsburgh, 6TaskRabbit, 7Arizona State U., 8UC Santa Cruz 9U. of Southern
California, 10IMRSV Data Labs, 11EPFL, 12U. of Illinois - Chicago, 13U. of
Maryland Baltimore County 14Rutgers U., 15U. of Pennsylvania, 16Expedia Inc.,
17U. of Houston, 18Indiana U. - Bloomington, 19Microsoft

###### Abstract

††$\star$ The point of view of the authors are their own and not attributable
to the company they work for.

Despite the progress made in recent years in addressing natural language
understanding (NLU) challenges, the majority of this progress remains to be
concentrated on resource-rich languages like English. This work focuses on
Persian language, one of the widely spoken languages in the world, and yet
there are few NLU datasets available for this language. The availability of
high-quality evaluation datasets is a necessity for reliable assessment of the
progress on different NLU tasks and domains. We introduce ParsiNLU, the first
benchmark in Persian language that includes a range of language understanding
tasks — _Reading Comprehension_ , _Textual Entailment_ , etc. These datasets
are collected in a multitude of ways, often involving manual annotations by
native speakers. This results in over 14.5$k$ new instances across 6 distinct
NLU tasks. Besides, we present the first results on state-of-the-art
monolingual and multi-lingual pre-trained language models on this benchmark
and compare them with human performance, which provides valuable insights into
our ability to tackle natural language understanding challenges in Persian. We
hope ParsiNLU fosters further research and advances in Persian language
understanding.111 https://git.io/JIuRO

##  1 Introduction

In recent years, considerable progress has been made in building stronger NLU
models, particularly supported by high-quality benchmarks Bowman et al.
(2015); Rajpurkar et al. (2016); Wang et al. (2019) for resourceful languages
like English. However, in many other languages, such benchmarks remain scarce,
unfortunately, stagnating the progress towards language understanding in these
languages.

In this work, we focus on developing NLU benchmarks for Persian (also known as
“Farsi”). This language has many attributes that make it distinct from other
well-studied languages. In terms of script, Persian is similar to Semitic
languages (e.g., Arabic). Linguistically, however, Persian is an Indo-European
language Masica (1993) and thus distantly related to most of the languages of
Europe as well as the northern part of the Indian subcontinent. Such
attributes make Persian a unique case to study in terms of language
technologies. Although Persian is a widely spoken language Simons and Fennig
(2017), our ability to evaluate performance and measure the progress of NLU
models on this language remains limited. This is mainly due to the lack of
major language understanding benchmarks that can evaluate progress on a
diverse range of tasks.

In this work, we present ParsiNLU, a collection of NLU challenges for
Persian.222 We focus on the standard Iranian Persian, spoken by over 80
million people. There are other dialects of Persian spoken in other countries,
e.g., Afghanistan and Tajikistan.  ParsiNLU contains challenges for _reading
comprehension_ , _multiple-choice question-answering_ , _textual entailment_ ,
_sentiment analysis_ , _question paraphrasing_ , and _machine translation_
(examples in Fig. 1). ParsiNLU offers data for tasks that have never been
explored before in the context of the Persian language. We are not aware of
any publicly available dataset for Persian _question answering_ (§3.2.2),
_reading comprehension_ (§3.2.1), and _paraphrasing_ (§3.2.5). For the rest of
the tasks, we improve at least one aspect of the existing datasets (e.g.,
better data construction, more comprehensive evaluation, and evaluation of
less investigated genres or domains). To ensure the quality of the presented
challenge tasks, we rely on the annotations from native Persian speakers or
novel data collection techniques, such as, search engine auto-complete
(§3.2.1), and past collegiate exams (§3.2.2). To the best of our knowledge,
this is the first comprehensive collection of its own, composed of a variety
of Persian NLU tasks.

We conduct a collection of empirical work (§4) to establish the difficulty of
ParsiNLU. We benchmark each ParsiNLU task via collecting state-of-the-art
multi-lingual and mono-lingual LMs, as well as estimating the human upper
bound scores. The gap between human and machine baselines indicate the need
for further research and stronger models for Persian. We hope that the release
of ParsiNLU will encourage more research on Persian NLP.

Figure 1: Examples of the ParsiNLU tasks. For each task (other than Machine
Translation which already contains English phrases) we show the English
translations for ease of communication to non-Persian readers. The purple tags
indicate the example category, according to their construction (explained in
the main text under Section 3.2).

##  2 Related Work

##### Cross-lingual benchmarks.

There are several recent cross-lingual benchmarks; however, almost none
includes Persian: XNLI Conneau et al. (2018) for entailment, PWNS-X Yang et
al. (2019) for paraphrasing, XCOPA Ponti et al. (2020) for choice of plausible
alternatives, XQuAD, MLQA, TyDI and MKQA Artetxe et al. (2020b); Lewis et al.
(2020); Clark et al. (2020a); Longpre et al. (2020) for reading comprehension.
These datasets have also been integrated as part of multi-task multilingual
evaluation suites such as the XTREME Hu et al. (2020) and XGLUE Liang et al.
(2020). Unfortunately, the Persian portion of the former benchmark covers only
two tagging tasks (POS and NER) and the latter does not cover Persian.

##### NLU benchmarks for other languages.

Benchmarks like GLUE Wang et al. (2019) encourage development of better and
stronger models on a diverse set of challenges. There have been several
efforts to create GLUE-like benchmarks for other languages; for example, CLUE
for Chinese Xu et al. (2020), GLUECoS for Hindi Khanuja et al. (2020), and
RussianSuperGLUE Shavrina et al. (2020). We view ParsiNLU in the same family
of benchmarks, dedicated to the Persian language.

##### NLU Datasets for Persian.

Prior work on creating evaluation resources for the Persian language has
focused on low-level tasks in narrow domains (e.g., datasets for POS Bijankhan
(2004), NER Shahshahani et al. (2019), Parsing Seraji et al. (2013)).
Complementary to these efforts, we aim at providing an NLU evaluation
benchmark for Persian, consisting of a wide variety of tasks. Below we mention
several related works and how we build upon them.

FarsTail Amirkhani et al. (2020) is a concurrent work on the _entailment_
task, where the dataset is constructed semi-automatically based on existing
multiple-choice exams. Different from this work, our entailment datasets are
built with the annotations of native speakers of Persian and some use of
machine translation (§3.2.4). Therefore, we hypothesize our construction
represents a slightly different distribution than that of FarsTail.

There is a rich set of works on Persian _sentiment analysis_. We build upon
these works and differ from them in the following manners: (a) The existing
work mainly focuses on _document-level_ sentiment identification which does
not capture the nuanced judgements with respect to aspects and entities of the
context (HosseinzadehBendarkheili et al., 2019; Sharami et al., 2020, inter
alia). In addition to such _document-level_ annotations, we provide _aspect-
level_ sentiment annotations (§3.2.3). (b) The majority of existing resources,
such as MirasOpinion Ashrafi Asli et al. (2020) focus on binary or ternary
sentiment classes. However, our annotations contain a more granular sentiment
intensity with five labels (§3.2.3). (c) Compared to the aspect-level datasets
Hosseini et al. (2018); Ataei et al. (2019), we cover two relatively less
investigated domains: food & beverages and movies, each posing new challenges
for Persian sentiment analysis.

_Machine translation_ of Persian $\rightleftarrows$ English is one of the few
tasks that has enjoyed decent attention Tiedemann and Nygaard (2004);
Mohaghegh et al. (2010); Pilevar et al. (2011); Mohaghegh et al. (2011);
Rasooli et al. (2013); Karimi et al. (2018); Kashefi (2018); Khojasteh et al.
(2020). Unfortunately, most published work for this task focus on niche
domains and datasets. Our contribution to this task is compiling a set of
high-quality evaluation sets from a broad range of domains, based on the
existing datasets as well as datasets introduced in this work. The hope is
that this will help future work on Persian MT to evaluate their systems on a
variety of domains to get a more realistic measure of machine translation.

To the best of our knowledge, this is the first work that publishes an
evaluation benchmark for Persian language, promoting future studies on several
NLU tasks such as _question answering_ (§3.2.2), _reading comprehension_
(§3.2.1), and _paraphrasing_ (§3.2.5), among others.

##  3 ParsiNLU

###  3.1 Design Considerations

We now discuss possible design choices for constructing the dataset and the
underlying reasons.

##### Naturally-occurring instances.

A common way of collecting data for low-resource languages has been using
automated translation of the benchmark datasets of high-resource languages
Artetxe et al. (2020b); Ponti et al. (2020). This can be a poor practice as
recent investigations have shown translation artifacts in data gathered via
translation of existing tasks Artetxe et al. (2020a). It is important for any
NLP dataset to reflect the _natural_ distribution of the target language
tokens and their associated cultural contexts. Therefore, one should _avoid_
over-reliance on automatic conversion of resources from high-resource
languages to minimize any unnatural instances or artifacts Khvalchik and
Malkin (2020).

##### Experts, over crowdworkers.

While crowdsourcing has been the common approach for building datasets, we
choose to work with few native Persian speakers to construct the dataset.
Crowdworkers are difficult to train and often generate more noisy annotations.
However, expert annotators that are closely familiar with the task at hand
often generate better quality annotations. Using crowdworkers is further
complicated by the fact that crowdsourcing platforms do not have an active
community of Persian-speaking workers due to limited international financial
transactions and crowdsourcing platforms. A study done by Pavlick et al.
(2014, Table 6) shows that there are almost no crowd-workers for Persian on
the Amazon Mechanical Turk platform.

###  3.2 Constructing ParsiNLU tasks

Examples are shown in Fig. 1. We now explain the data construction of each
task.

####  3.2.1 Reading Comprehension

We use the commonly used definition of reading-comprehension task: extracting
a substring from a given context _paragraph_ that answers a given _question_.

SQuAD Rajpurkar et al. (2016) is one of the most popular reading comprehension
datasets in English. Similar datasets to SQuAD are developed in other
languages using varying degrees of human or semi-automatic translation
techniques: KorQuAD for Korean Lim et al. (2019), MMQA for Hindi Gupta et al.
(2018), etc. For constructing our reading comprehension tasks, we avoid using
SQuAD as a source and employ a process resembling that of Kwiatkowski et al.
(2019) that would lead to more natural questions.

##### Collecting questions.

Our efforts to translate questions from the English dataset indicated that
such questions are often about topics that are not of much importance in
Persian. For instance, there are many questions in SQuAD Rajpurkar et al.
(2016) about major US sports events (e.g., Superbowl, NFL) or western
civilization history that might not be common among Persian speakers. Instead,
we follow a pipeline that is more similar to the one introduced by Kwiatkowski
et al. (2019), setting our goal to annotate answers for an existing
naturalistic set of questions in Persian, as opposed to writing questions for
existing paragraphs.

Unlike Kwiatkowski et al. (2019), we do not have direct access to query logs.
Thus we follow the approach of Berant et al. (2013); Khashabi et al. (2021)
which relies on a query auto-completion API for collecting questions.
Similarly, we use Google’s auto-
completion333http://google.com/complete/search?client=chrome&q=… which enables
us to mine a rich, yet a natural set of questions in Persian as it is
reflective of popular questions posed by users of Google.

We start with a seed set of question terms (e.g., “^che kesI>” [che kasI]
meaning “who”, and “kojA>” [kojA] meaning “where”) We bootstrap based on this
set, by repeatedly querying parts of previously-extracted questions, in order
to discover a longer and richer set of questions. We hypothesize that such
questions extracted from the auto-complete algorithm, are highly reflective of
popular questions posed by Persian-speaking users of Google. We filter out any
results shorter than 5 tokens as they are often incomplete questions. This
process yields over 50$k$ questions.

Subsequently, we automatically filter out open-ended questions with no
concrete answers (e.g., “natIjehye bAzI bA ^zApon?>” [nætIdZe ye bAzI bA
ZApon?] meaning “What is the results of the game with Japan?”). Our filtering
was guided by the observation that typically more complete questions lead to
Google results that include well-established sources (such as Wikipedia).
Hence, we perform this filtering by retrieving the Google search
results444https://github.com/MarioVilas/googlesearch for each question and
checking if any of the top 10 search results overlap with a pre-defined list
of credible websites.555fa.wikipedia.org, bbcpersian.com, etc. We keep only
the questions that match this criterion.

##### Annotating paragraphs and answers.

In this step, native speakers of Persian select a paragraph and an answer span
within the paragraph that answers each of the questions. At the first step,
the annotators read the question and correct any grammatical errors and typos
(e.g., “otsAn>” [otsAn] is corrected to “ostAn>” [ostAn] “state”). Next, they
annotate all _the minimal and coherent spans_ that contains the answer to the
question, from a paragraph obtained from a relevant web page (from the Google
search results retrieved from an earlier step). Whenever possible, we annotate
all valid spans as the answer (for example, “hamedAn>” [hæmedAn] and “ostAn
hamedAn>” [ostAn e hæmedAn], as shown in Fig. 1). The paragraph that contains
this answer is also annotated as the context of the question.

Overall, 6 native-speaker annotators annotated a collection of 1.3$k$
question-answer-paragraph triplets (Table 2).

##### Annotation quality.

To ensure the quality of the annotations, the answers to each question were
labeled by two independent annotators. Any misalignment of the answer spans or
missing any valid spans were indicated as disagreements. Such disagreements
were resolved in further adjudication.

####  3.2.2 Multiple-Choice QA

Multiple-choice questions are one of the common formats for evaluation of
fact-retrieval and reasoning Richardson et al. (2013); Clark et al. (2020b).
Following prior works, we define the task as: given a natural language
question, pick the correct answer among a list of multiple candidates. A key
difference from reading comprehension (§3.2.1) is that the instances are open-
domain (i.e., no context paragraph is provided). Hence, a model would either
need to retrieve external supporting documents or have stored the necessary
knowledge internally to be able to answer the question.

##### Sources of questions.

We use existing sources of multiple-choice questions, rather than annotating
new ones. We collect the questions from a variety of sources: (i) The
literature questions of the annual college entrance exams in Iran, for the
past 15 years. These questions often involve the understanding of poetry and
their implied meaning, knowledge of Persian grammar, and the history of
literature. (ii) Employment exams that are expected to assess an individual’s
depth in various topics (accounting, teaching, mathematics, logic, etc). (iii)
Common knowledge questions, which involve questions about topics such as basic
science, history, or geography.

Most of the above sources are scanned copies of the original exams in image
format. We use an existing Persian OCR tool to convert the image data to a
textual format.666 https://www.sobhe.ir/alefba/  Then 4 annotators fix any
mistakes made by the OCR system and convert the result into a structured
format. Overall, this yields 2460 questions with an average of 4.0 candidate
answers (Table 2). Additionally, the task comes with a label indicating the
type of knowledge it requires: ‘literature’ (understanding of literary
expressions), ‘common-knowledge’ (encyclopedic knowledge or everyday
activities), and ‘math & logic’ (logical or mathematical problems). Examples
from each category of questions are included in Fig. 1.

##### Annotation quality.

To further examine the quality of the annotations, we randomly sampled 100
questions from the annotations and cross-checked the OCR output with the
original data. We discovered that 94 of such questions exactly matched the
original data, and the rest required minor modifications. We thus conclude
that the annotated data has a high quality.

####  3.2.3 Aspect-Based Sentiment Analysis

Sentiment Analysis (SA) is the study of opinions (i.e., positive, negative, or
neutral sentiment) expressed in a given text Liu (2012). Aspect-based
Sentiment Analysis (ABSA) is a more fine-grained SA that aims to extract
aspects of entities mentioned in the text and determine sentiment toward these
aspects Pontiki et al. (2014). For instance, _“it tastes good but it’s so
expensive …”_ (Fig. 1) conveys _positive_ and _negative_ sentiments with
respect to _taste_ and _price_ aspects of the mentioned product (entity),
respectively.

##### Annotation scheme.

We follow the existing ABSA scheme Pontiki et al. (2014). For every review, we
do two types of annotations: (1) we assign an overall sentiment to each
review, selecting from one of the following values: _very-negative, negative,
neutral, positive, very positive,_ and _mixed_. The _mixed_ category indicates
reviews where none of the sentiments are dominant (mix of positive and
negative, or borderline cases), hence it is hard to detect the primary
sentiment of a review. We also assign neutral label to reviews that express no
clear sentiment toward an entity or any aspect of it. (2) we annotate pairs of
$(a,s)$ where, $a$ is an aspect that belongs to a predefined set of aspects
for each domain and $s$ expresses the sentiment toward the aspect $a$.

##### Collecting reviews.

At first, we collect reviews from two different domains: (1) _food &
beverages_ and 2) _movies_. We chose these domains since they are relatively
less investigated in the existing literature (see §2 for the past work). For
the _food & beverages_ category, we
extracted777https://github.com/rajabzz/digikala-crawler reviews from the
online grocery section of Digikala,888https://www.digikala.com/ and for the
_movie_ reviews category, we crawled reviews from
Tiwall.999https://www.tiwall.com/ Both of these websites are well-known and
popular websites among Persian speakers.

##### Defining aspects.

Following ABSA scheme, we predefined a set of aspects for each domain. For
food & beverages, we crawled Digikala and retrieved all listed aspects for
product reviews in the food & beverages category. Subsequently, we manually
aggregated the extracted aspects and merged those with significant semantic
overlap. We also added taste/smell as a new aspect category since users
frequently commented on this aspect. For _movie_ reviews, we created an
initial list of aspects based on the movie review aspects defined by Thet et
al. (2010). In consultation with a movie critic, we resolved the potential
overlaps among aspect categories and created a set of aspects that capture
various perspectives of movie reviews. Overall, this process resulted in 6 and
7 aspects for food & beverages and movie review domains, respectively (Table
1).

Table 1: The predefined sentiment aspects (§3.2.3).

After defining the sentiment aspects, we trained four native speaker
annotators for the final round of annotations. This results in 2423 instances
for the sentiment task (Table 2).

##### Annotation quality.

To measure the quality of the annotations, we randomly selected 100 samples
from each domain and calculated the Inter-Annotator Agreement (IAA) using
Cohen’s kappa Cohen (1960) on annotations elicited from two independent
annotators. Based on the computed IAA values, there is a substantial agreement
on sub-task 1 (0.76), and moderate agreement on sub-tasks 2 and 3 (0.49 and
0.47, resp.).

Task | Attribute | Statistic
---|---|---
Reading Comprehension | # of instances | 1300
avg. question length (tokens) | 6.3
avg. paragraph length (tokens) | 94.6
avg. answer length (tokens) | 7.6
Multiple-Choice QA | # of instances | 2460
% of ‘literature’ questions | 834
% of ‘common-knowledge’ questions | 949
% of ‘math & logic’ questions | 677
avg. # of candidates | 4.0
Sentiment Analysis | # of instances | 2423
% of ‘food & beverages’ reviews | 1917
% of ‘movie’ reviews | 506
avg. length of reviews (words) | 22.01
# of annotated pairs of (aspect, sentiment) | 2539
Textual Entailment | # of instances | 2,700
% of ‘natural’ instances | 1,370
% of ‘mnli’ instances | 1,330
avg. length of premises (tokens) | 23.4
avg. length of hypotheses (tokens) | 11.8
Question Paraphrasing | # of instances | 4,644
% of ‘natural’ instances | 2,521
% of ‘qqp’ instances | 2,123
avg. length of Q1 (tokens) | 10.7
avg. length of Q2 (tokens) | 11.0
Machine Translation | # of instances | 47,745
% of ‘QP’ subset | 489
% of ‘Quran’ subset | 6,236
% of ‘Bible’ subset | 31,020
% of ‘Mizan’ subset (eval. only) | 10,000

Table 2: Statistics on various subsets of the dataset.

##### Distribution of the labels.

Here we report the distribution of the labels for this task. Fig. 2 shows the
distribution of the document-level sentiment labels. As expected, most reviews
are associated with extreme sentiments (very positive or very negative) and a
relatively small portion of them are neutral. There is also a non-negligible
portion of the reviews that contains mixed sentiments (partially positive and
partially negative).

Figure 2: The distribution of the overall sentiments labels (document-level).

####  3.2.4 Textual Entailment

Textual Entailment Dagan et al. (2013); Bowman et al. (2015) is typically
defined as a 3-way classification to determine whether a _hypothesis_ sentence
_entails_ , _contradicts_ , or is _neutral_ with respect to a given _premise_
sentence.

We construct two subsets: (i) based on available natural sentences, and (ii)
based on the available English entailment dataset. The former approach yields
high-quality instances, however, it is a relatively slower annotation task.
The latter is slightly easier, but yields less interesting instances.

##### Based on natural sentences.

We start with randomly sampled raw sentences, selected from 3 different
resources: Miras,101010https://github.com/miras-tech/MirasText Persian
Wikipedia and VOA corpus.111111https://jon.dehdari.org/corpora/ In this random
sampling process, we specifically sample sentences that contain conjunctive
adverbs (e.g, “ammA>” [amA] meaning “but”), along with their preceding
sentences. We chose such examples as there is a higher chance that these
sentences naturally contain inference relationships. We ask annotators to
consider both sentences and write a premise and corresponding entailing,
contradicting, and neutral sentences, whichever they deem appropriate. To
minimize annotation artifacts and avoid creating an artificially easy dataset,
we specifically instruct annotators to avoid using simple modifications, such
as simply negating a sentence or changing a word to its synonym. For the rest
of the work, we refer to this set as the ‘natural’ set.

##### Based on existing datasets.

In this approach, we use existing datasets in English. We start with MNLI
dataset Williams et al. (2018) and translate them with the publicly available
Google Translate API.121212https://cloud.google.com/translate Subsequently,
expert annotators carefully review and fix inaccurate translations.
Furthermore, each translated document is reviewed by a native-speaker
annotator to correct the translational mistakes. Our annotations show that
about $66.4\%$ of the translated documents have gone through some form of
correction by our annotators. For the rest of the draft, we refer to this set
as ‘mnli’.

Overall, our two-pronged construction with 6 annotators results in 2.7$k$
entailment instances (Table 2). Examples from each collected subset are
included in Fig. 1.

##### Annotation quality.

To verify the annotation quality, we quantify the agreement of 3 independent
annotators, on 150 random examples. On this subset, we observe a Fleiss Kappa
Fleiss (1971) of 0.77, indicating a _substantial_ inter-annotator agreement
Landis and Koch (1977).

##### Distribution of the labels.

As the label distribution (Fig. 3) shows, the distribution of the labels
across the three categories are not far from uniform distribution.

Figure 3: The distribution of the labels for the entailment task.

####  3.2.5 Question Paraphrasing

This task is defined as determining whether two given questions are
paraphrases or not. This task has been previously used to improve down-stream
applications like document retrieval Zukerman and Raskutti (2002); Callison-
Burch et al. (2006); Duboue and Chu-Carroll (2006).

Similar to the construction of the entailment task (§3.2.4), we take two
different approaches: (i) based on available natural sentences, (ii) based an
existing English question paraphrasing dataset.

##### Based on natural sentences.

We start with questions mined using Google auto-complete (§3.2.1) as well as
an additional set of questions mined from Persian discussion
forums.131313http://javabkoo.com/ We create pairs of questions with high token
overlap. Each pair is annotated as _paraphrase_ or _not-paraphrase_ by native-
speakers. We drop the pair if any of the questions is incomplete. For the rest
of this document, we refer to this subset as ‘natural’.

##### Based on existing datasets.

We start with the QQP dataset,141414 https://www.kaggle.com/c/quora-question-
pairs  which is a dataset of English question-pairs, and translate it with
Google Translate API. Later, expert annotators carefully review the
translations and amend any inaccuracies. We observe that about $65.6\%$ of the
translated documents have gone through some form of correction by our
annotators.

Overall, the annotations involved 4 annotators and resulted in 4682 question
paraphrasing instances (Table 2). Examples from each collected subset are
included in Fig. 1.

##### Annotation quality.

After the annotation of the earlier steps, the example were reviewed by
another annotators familiar with the task. The disagreements were labeled and
adjudicated among the annotators, in order to ensure the quality of the
resulting labels.

##### Distribution of the labels.

As the label distribution shows (Fig. 4), the label distributions of the two
splits (‘qqp’ vs ‘natural’) are not much different.

Figure 4: Label distribution for the query paraphrasing task. Task | Train | Dev | Eval
---|---|---|---
Reading Comprehension | 600 | 125 | 575
Multiple-Choice | 1271 | 139 | 1050
Sentiment Analysis | 1894 | 235 | 294
Textual Entailment | 756 | 271 | 1,751
Question Paraphrasing | 1,830 | 898 | 1,916
Machine Translation | $1.6m$ | $2k$ | $47k$
Table 3: Split sizes for different tasks.

Setup |  Model $\downarrow$ \- Task $\rightarrow$ |  Reading Comprehension |  Multiple-Choice Question Answering |  Textual Entailment |  Question Paraphrasing
---|---|---|---|---|---
Subtask $\rightarrow$ | all | literature | com-know | math & logic | natural | mnli | natural | qqp
trained on Persian | mBERT (base) | 49.0 | 30.1. | 28.7 | 33.8 | 48.7 | 51.6 | 80.4 | 75.3
WikiBERT (base) | 39.2 | 36.9 | 30.2 | 34.1 | 52.8 | 52.6 | 80.0 | 75.5
ParsBERT (base) | 40.7 | 33.4 | 28.6 | 32.5 | 51.8 | 53.9 | 79.4 | 72.0
mT5 (small) | 30.9 | 33.7 | 23.7 | 39.1 | 51.9 | 51.0 | 75.2 | 72.0
mT5 (base) | 42.6 | 34.0 | 24.0 | 36.9 | 57.8 | 59.9 | 79.1 | 75.1
mT5 (large) | 49.2 | 32.6 | 27.1 | 38.9 | 69.1 | 71.6 | 84.6 | 76.6
mT5 (XL) | 70.4 | 33.7 | 27.7 | 38.9 | 77.2 | 74.5 | 88.6 | 80.3
trained on English | mT5 (small) | 33.0 | 20.9 | 25.7 | 28.9 | 45.1 | 55.6 | 73.5 | 75.1
mT5 (base) | 53.4 | 23.4 | 23.4 | 24.3 | 44.4 | 43.3 | 83.2 | 81.8
mT5 (large) | 67.4 | 27.4 | 33.1 | 25.4 | 46.5 | 54.9 | 88.1 | 86.6
mT5 (XL) | 68.2 | 28.3 | 38.6 | 22.0 | 66.2 | 77.8 | 89.2 | 87.0
trained on Per + Eng | mT5 (small) | 45.3 | 30.9 | 24.9 | 36.6 | 53.3 | 56.2 | 77.9 | 71.3
mT5 (base) | 63.9 | 32.3 | 24.0 | 37.7 | 57.8 | 63.9 | 80.2 | 73.4
mT5 (large) | 73.6 | 30.6 | 28.9 | 38.6 | 70.9 | 72.5 | 85.3 | 78.9
mT5 (XL) | 74.7 | 38.0 | 33.7 | 38.0 | 75.5 | 78.7 | 88.2 | 80.3
| Human | 86.2 | 80.0 | 85.0 | 85.0 | 87.1 | 90.2 | 92.3 | 88.4

Setup |  Model $\downarrow$ \- Task $\rightarrow$ |  Sentiment (sentence sent.) |  Sentiment (aspect ext.) |  Sentiment (aspect sent.) |  Machine Translation (Eng $\rightarrow$ Per) |  Machine Translation (Per $\rightarrow$ Eng)
---|---|---|---|---|---|---
Subtask $\rightarrow$ | food | movies | food | movies | food | movies | quran | bible | qqp | mizan | quran | bible | qqp | mizan
trained on our data | mBERT (base) | 55.2 | 48.6 | 87.1 | 73.24 | 53.9 | 34.7 | - | - | - | - | - | - | - | -
WikiBERT (base) | 52.0 | 58.5 | 91.9 | 78.0 | 56.5 | 41.6 | - | - | - | - | - | - | - | -
ParsBERT (base) | 59.1 | 56.8 | 91.1 | 76.8 | 53.9 | 37.6 | - | - | - | - | - | - | - | -
mT5 (small) | 54.6 | 49.4 | 86.4 | 78.6 | 52.4 | 40.6 | 10.2 | 2.1 | 22.2 | 8.4 | 20.6 | 2.5 | 22.9 | 14.6
mT5 (base) | 56.6 | 52.9 | 88.6 | 80.5 | 52.9 | 46.5 | 11.4 | 2.1 | 27.3 | 9.4 | 22.8 | 2.5 | 34.6 | 14.9
mT5 (large) | 62.9 |  72.5 | 92.2 | 85.0 | 58.1 | 53.5 | 11.9 | 2.1 | 24.8 | 10.6 | 24.7 | 2.4 | 35.1 | 16.4
mT5 (XL) | 63.1 | 70.6 | 92.0 |  85.8 | 58.9 |  54.5 | 13.5 | 2.2 | 20.0 | 11.0 | 30.0 | 2.6 | 33.7 | 19.3
trained on English | mT5 (small) | - | - | - | - | - | - | - | - | - | - | 6.6 | 1.9 | 7.7 | 3.7
mT5 (base) | - | - | - | - | - | - | - | - | - | - | 11.5 | 2.1 | 14.0 | 5.7
mT5 (large) | - | - | - | - | - | - | - | - | - | - | 20.2 | 2.3 | 21.0 | 7.4
mT5 (XL) | - | - | - | - | - | - | - | - | - | - | 25.6 | 2.3 | 30.7 | 9.7
trained on Per + Eng | mT5 (small) | - | - | - | - | - | - | - | - | - | - | 19.2 | 2.5 | 25.6 | 12.1
mT5 (base) | - | - | - | - | - | - | - | - | - | - | 24.1 | 2.4 | 36.0 | 14.8
mT5 (large) | - | - | - | - | - | - | - | - | - | - | 29.9 | 2.6 | 36.5 | 18.1
mT5 (XL) | - | - | - | - | - | - | - | - | - | - | 33.4 | 2.6 | 41.0 | 18.2
| Human | 88.4 | 90.3 | 93.1 | 91.6 | 71.0 | 61.6 | - | - | - | - | - | - | - | -

Table 4:  Evaluation of _Persian_ -only models (top), _English_ -only (middle)
and _Persian+English_ (bottom) models on Persian tasks. Best baseline scores
are indicated as bold.

####  3.2.6 Machine Translation

We consider the task of translating a given English sentence into Persian, and
vice versa.

This task is one of the few for which several resources are available in the
literature Kashefi (2018); Prokopidis et al. (2016); Pilevar et al. (2011).
One major limitation is that there is no widely adopted comprehensive
assessment of this task: most of the works are often limited to narrow
domains, and the generalization across different styles of text is rarely
studied. Our contribution is to put together a collection of evaluation sets,
from various domains to encourage a more holistic _evaluation_ set.

Our proposed evaluation sets consist of the followings: (i) _Quran_ : Quran
has been translated into many languages, including English and Persian
Tiedemann and Nygaard (2004). We use several different translations of Quran
to create high-quality evaluation sets (10 gold standard translations for each
direction). Having multiple gold standards is particularly helpful for the
automatic evaluation of machine translation since such metrics work best when
provided with several gold standards Gupta et al. (2019). (ii) _Bible_ :
similarly, we use Persian and English versions of
Bible151515https://github.com/christos-c/bible-corpus as another evaluation
set. (iii) _QQP_ : using the data obtained in the construction of question
paraphrasing task (§3.2.5) to create an evaluation set for translating
language questions. (iv) _Mizan_ : we use the evaluation subset of the Mizan
corpus Kashefi (2018), which is acquired based on a manual alignment of famous
literary works and their published Persian translations. Overall, the
combination of these four high-quality subsets yields an evaluation set that
contains $47k$ sentences, from $4$ different domains (Table 2.)

While our main contribution here is providing a more comprehensive
_evaluation_ of machine translation, we also provide training/dev sets to let
the future work create comparable experiments to that of ours. We compile our
training set at the union of the following datasets: (i) questions obtained
from the question paraphrasing task (§3.2.5, by translating the QQP
instances), (ii) the training set of _Mizan_ dataset Kashefi (2018), (iii) TEP
dataset Pilevar et al. (2011) and Global Voices dataset Prokopidis et al.
(2016). The latter two are not included in our evaluation set due to their
noisy translations to prevent any inaccurate evaluations. Note that the
_Quran_ and _Bible_ documents are intentionally not included in the training
data, in order to measure models’ generalization to unseen documents.

##  4 Experiments

We experiment with several recent LMs, to assess the difficulty of the
ParsiNLU tasks (compared to human expert performance) and also to establish
baseline performance of the state-of-the-art mono- and multi-lingual pre-
trained models.

All the baseline models used in this work are available online.161616Included
in the repository mentioned in footnote 1.

##### Evaluation metrics.

For each task, we pick a common set of existing metrics: For reading-
comprehension, we use _F1_ between gold answer and the response string
Rajpurkar et al. (2016); for question paraphrasing, textual entailment,
multiple-choice question-answering, and sentiment analysis, we use _accuracy_.
For the first two sub-tasks of sentiment analysis (document-level sentiment,
aspect extraction), we use _macro-F1_. For the third sub-task (aspect-specific
sentiment) we use _accuracy_ as our target evaluation metric Angelidis and
Lapata (2018); Sun et al. (2019). For machine translation we use _SacreBLEU_
Post (2018).

##### Task splits.

For each task, we have provided statistics on eval, train, and dev splits in
Table 3. In doing so, we have ensured that enough instances are included in
our evaluation sets.

##### Human performance.

To have an estimate of the performance and the difficulty of the challenges,
we report human performance on a random subset (100-150) of instances from
each task. Similar to Wang et al. (2019), we collect annotations from three
human annotators, adjudicate the inconsistencies and evaluate it against the
gold labels to estimate human performance for each task.

##### Models.

For evaluation of our baselines, we use state-of-the-art LMs. Multilingual
BERT (mBERT) Devlin et al. (2019) is pre-trained on the masked LM task over
104 languages. Additionally, we use two specialized variants of BERT for
Persian: wikiBERT171717https://github.com/TurkuNLP/wikibert (trained on
Persian Wiki) and ParsBERT Farahani et al. (2020).181818
https://github.com/hooshvare/parsbert  We also use mT5 Xue et al. (2021),
which is a multilingual variant of T5 Raffel et al. (2020).

##### Model selection.

We train each model with various hyper-parameters and select the best one
according to their developement set performance. For the BERT-based models, we
fine-tune them according to the cross product of the following hyper-
parameters: (1) Batch sizes: $\\{8,16\\}$ for small/base models and
$\\{1,2\\}$ for large models; (2) Training epochs: $\\{3,7\\}$; (3) Learning-
rates: $\\{3\times 10^{-5},5\times 10^{-5}\\}$. For mT5 models, we fine-tune
them for $20k$ steps, dumping checkpoints every $1k$ step. For the translation
task, we trained the models for $200k$ steps since the task has much larger
training data. We use $10^{-3}$ learning-rate.

##### Input/output encoding.

We formulate question paraphrasing (§3.2.5) and entailment (§3.2.4) tasks as
text classification tasks.191919 https://git.io/JYTNr  For sentiment-analysis
(§3.2.3), we follow formulation of Sun et al. (2019) and encode the instances
as questions per aspect. The expected output is the sentiment polarity of the
input review with respect to the input aspect-specific question. This
formulation has the benefit that it is not restricted to a particular domain
and its associated set of aspects, unlike alternatives such as multi-class
classification.

##### Experimental setups.

First, we fine-tune our models on _Persian_ (our dataset). The results of this
setup are listed in the top segment of Table 4.

Following the recent works on generalization across languages Artetxe et al.
(2020b), we evaluate _English_ models on our Persian benchmark. We use the
commonly used English datasets to supervise mT5 on each task and evaluate the
resulting model on the evaluation section of ParsiNLU. The English datasets
used here are as follows: SQuAD 1.1 Rajpurkar et al. (2016) for reading
comprehension (size: $88k$), the union of ARC Clark et al. (2018), OpenBookQA
Mihaylov et al. (2018) and CommonsenseQA Talmor et al. (2019) for multiple-
choice question-answering (size: $18k$), SNLI Bowman et al. (2015) for textual
entailment (size: $550k$), QQP202020See footnote 14. for question paraphrasing
(size: $350k$), and Arabic-English subset of OPUS-100 Zhang et al. (2020) for
machine translation (size: $1m$). We don’t do such mixing for sentiment
analysis since existing English datasets are not quite compatible with our
sentiment schema. The results are reported in the middle section of Table 4.

Finally, we train models on the union of Persian and English datasets. Since
English datasets tend to be much bigger than Persian ones, we make sure that
the batches of training data, on average, contain the same number of instances
from each language. Similar treatments of task mixing have also been adopted
by Khashabi et al. (2020); Raffel et al. (2020). The results of this setup are
at the bottom segment of Table 4.

###  4.1 Results

Below are key insights from the empirical work:

##### Humans do well on ParsiNLU.

As shown in the last row of Table 4, the human upper-bound scores are
relatively high across the board. This is an indication of a reasonable degree
of consensus between the ground-truth and judgments of native speakers and
hence, the quality of our dataset.

##### Models haven’t solved ParsiNLU yet.

The majority of the models significantly lag behind human performance. This is
especially true for the mid-sized (‘large’ or smaller) models that are
commonly used. It is encouraging that our largest model (mT5-XL) achieves
close to human performance, for certain tasks (e.g., question paraphrasing),
however, this model is prohibitively large and it requires a massive amount of
compute. However, even these large models still struggle for most of the
remaining tasks, particularly multiple-choice QA.

##### English models successfully transfer to Persian.

Consistent with the prior observations Artetxe et al. (2020b), multilingual
models (mT5, in this case) trained with English data show a surprising degree
of generalization to other languages (to Persian, in our case). Training on
English data is particularly helpful for challenges that were originally
translated from English datasets (such as ‘qqp‘ and ‘mnli‘).

##### Joint training on English and Persian helps.

For most of the tasks, combining Persian and English yields better results
than training solely on Persian or English data.

While joint training generally helps, such combinations are not guaranteed to
lead to positive gains all the times. Whether the “Eng + Per” models will beat
either of the Persian-only or English-only models depends on whether their
strengths (large size of “Eng” and distributional alignment of “Per”) align or
go against each other. Because of this issue, the combined models are not
always better than the individual models.

##  5 Discussion

We now discuss several limitations of the current dataset and the experiments.
We then outline several directions for future work.

##### Beyond current models.

As shown in the earlier experiments, for most of the tasks the current mid-
sized models perform significantly worse than humans. This is particularly
pronounced for the multiple-choice QA task where there is over 40% gap between
the model and human performance, and increasing the model size (number of
parameters) shows minimal benefits.

We hypothesize that the difficulty of our multiple-choice questions (and other
tasks, to some extent) for the models are partly due to the reasoning and
abstraction needed to answer them. For example, the ‘literature’ questions
often demand creating connection several pieces of poetry, based on abstract
interpretations of their meanings. Likewise, most of the ‘math & logic’
questions require several ‘hops’ of algebraic operations to get to the final
answer. We hypothesizes these challenges (multi-hop reasoning over high-level
abstractions of language) cannot be solely be addressed with more training
data. and likely require a dramatic rethinking of our architectures design.
For example, the poor performance on ‘math & logic’ questions might be due to
models’ inability to comprehend Persian numbers and do logical reasoning with
them, a topic that is briefly studied in English Geva et al. (2020). There
might also be value in exploring multi-task setups across our various tasks
Zaremoodi et al. (2018), which we delegate to the future work. We hope this
benchmark will encourage more of such studies, especially in the context of
the Persian language.

##### Coverage of dialects.

There are other dialects of Persian, including Dari and Tajiki dialects, that
are not covered by our dataset. We acknowledge this limitation and hope the
future work will create broader and more inclusive collections.

##  6 Conclusion

This work introduced ParsiNLU, a benchmark for high-level language
understanding tasks in Persian. We present a careful set of steps we have
followed to construct each of the tasks with the help of native speakers
(§3.2). We have presented human scores to establish estimated upper-bounds for
each task. This is followed by evaluating state-of-the-art models on each task
and quantifying the human-machine gap (§4).

To the best of our knowledge, this is the first work that publishes a language
understanding benchmark for Persian language. We hope that ParsiNLU inspires
more activity in the Persian NLU tasks, as well as contributing to the latest
efforts in multilingual NLU.

## Acknowledgement

The authors would like to thank Alireza Nourian for providing the OCR system
used in the work and the anonymous reviewers for their constructive feedback.
Thanks to Google’s TensorFlow Research Cloud (TFRC) for making research TPUs
available.

## References

  * Amirkhani et al. (2020) Hossein Amirkhani, Mohammad Azari Jafari, Azadeh Amirak, Zohreh Pourjafari, Soroush Faridan Jahromi, and Zeinab Kouhkan. 2020.  Farstail: A Persian natural language inference dataset.  _arXiv preprint arXiv:2009.08820_.
  * Angelidis and Lapata (2018) Stefanos Angelidis and Mirella Lapata. 2018.  Multiple instance learning networks for fine-grained sentiment analysis.  _Transactions of the Association for Computational Linguistics_ , 6:17–31.
  * Artetxe et al. (2020a) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a.  Translation artifacts in cross-lingual transfer learning.  In _Proceedings of EMNLP_ , pages 7674–7684.
  * Artetxe et al. (2020b) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of ACL_ , pages 4623–4637.
  * Ashrafi Asli et al. (2020) Seyed Arad Ashrafi Asli, Behnam Sabeti, Zahra Majdabadi, Preni Golazizian, Reza Fahmi, and Omid Momenzadeh. 2020.  Optimizing annotation effort using active learning strategies: A sentiment analysis case study in Persian.  In _Proceedings of LREC_ , pages 2855–2861.
  * Ataei et al. (2019) Taha Shangipour Ataei, Kamyar Darvishi, Soroush Javdan, Behrouz Minaei-Bidgoli, and Sauleh Eetemadi. 2019.  Pars-absa: an aspect-based sentiment analysis dataset for Persian.  _arXiv preprint arXiv:1908.01815_.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on freebase from question-answer pairs.  In _Proceedings of EMNLP_ , pages 1533–1544.
  * Bijankhan (2004) Mahmood Bijankhan. 2004.  The role of the corpus in writing a grammar: An introduction to a software.  _Iranian Journal of Linguistics_ , 19(2):48–67.
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of EMNLP_.
  * Callison-Burch et al. (2006) Chris Callison-Burch, Philipp Koehn, and Miles Osborne. 2006.  Improved statistical machine translation using paraphrases.  In _Proceedings of NAACL_ , pages 17–24.
  * Clark et al. (2020a) Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020a.  TyDi QA: a benchmark for information-seeking question answering in ty pologically di verse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Clark et al. (2020b) Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2020b.  From ‘F’ to ‘A’ on the NY Regents Science Exams: An Overview of the Aristo Project.  _AI Magazine_ , 41(4):39–53.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ , 20(1):37–46.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  In _Proceedings of EMNLP_ , pages 2475–2485.
  * Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013.  _Recognizing Textual Entailment: Models and Applications_.  Morgan & Claypool Publishers.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of NAACL_ , pages 4171–4186.
  * Duboue and Chu-Carroll (2006) Pablo Duboue and Jennifer Chu-Carroll. 2006.  Answering the question you wish they had asked: The impact of paraphrasing for question answering.  In _Proceedings of NAACL_ , pages 33–36.
  * Farahani et al. (2020) Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, and Mohammad Manthouri. 2020.  Parsbert: Transformer-based model for Persian language understanding.  _arXiv preprint arXiv:2005.12515_.
  * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378.
  * Geva et al. (2020) Mor Geva, Ankit Gupta, and Jonathan Berant. 2020.  Injecting numerical reasoning skills into language models.  In _Proceedings of ACL_ , pages 946–958.
  * Gupta et al. (2018) Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. 2018.  Mmqa: A multi-domain multi-lingual question-answering framework for english and hindi.  In _Proceedings of LREC_.
  * Gupta et al. (2019) Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019.  Investigating evaluation of open-domain dialogue systems with human generated multiple references.  In _Proceedings of SIGDIAL_ , pages 379–391.
  * Hosseini et al. (2018) Pedram Hosseini, Ali Ahmadian Ramaki, Hassan Maleki, Mansoureh Anvari, and Seyed Abolghasem Mirroshandel. 2018.  Sentipers: A sentiment analysis corpus for Persian.  _arXiv preprint arXiv:1801.07737_.
  * HosseinzadehBendarkheili et al. (2019) Fatemeh HosseinzadehBendarkheili, Rezvan MohammadiBaghmolaei, and Ali Ahmadi. 2019\.  Product quality assessment using opinion mining in persian online shopping.  In _Proceedings of ICEE_ , pages 1917–1921. IEEE.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of ICML_ , pages 4411–4421. PMLR.
  * Karimi et al. (2018) Akbar Karimi, Ebrahim Ansari, and Bahram Sadeghi Bigham. 2018.  Extracting an english-persian parallel corpus from comparable corpora.  In _Proceedings of LREC_.
  * Kashefi (2018) Omid Kashefi. 2018.  Mizan: a large Persian-english parallel corpus.  _arXiv preprint arXiv:1801.02107_.
  * Khanuja et al. (2020) Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020.  GLUECoS: An evaluation benchmark for code-switched NLP.  In _Proceedings of ACL_.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  Unifiedqa: Crossing format boundaries with a single qa system.  In _Proceedings of EMNLP (Findings)_ , pages 1896–1907.
  * Khashabi et al. (2021) Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021.  Gooaq: Open question answering with diverse answer types.  _arXiv preprint arXiv:2104.08727_.
  * Khojasteh et al. (2020) Hadi Abdi Khojasteh, Ebrahim Ansari, and Mahdi Bohlouli. 2020.  Lscp: Enhanced large scale colloquial persian language understanding.  In _Proceedings of LREC_ , pages 6323–6327.
  * Khvalchik and Malkin (2020) Maria Khvalchik and Mikhail Malkin. 2020.  Departamento de nosotros: How machine translated corpora affects language models in mrc tasks.  In _Proceedings of the Workshop on Hybrid Intelligence for Natural Language Processing Tasks (HI4NLP 2020) co-located with 24th European Conference on Artificial Intelligence (ECAI 2020): Santiago de Compostela, Spain, August 29, 2020_ , pages 29–33. CEUR Workshop Proceedings.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977.  The measurement of observer agreement for categorical data.  _biometrics_ , pages 159–174.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of ACL_ , pages 7315–7330.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020.  Xglue: A new benchmark datasetfor cross-lingual pre-training, understanding and generation.  In _Proceedings of EMNLP_ , pages 6008–6018.
  * Lim et al. (2019) Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019.  Korquad1. 0: Korean qa dataset for machine reading comprehension.  _arXiv preprint arXiv:1909.07005_.
  * Liu (2012) Bing Liu. 2012.  Sentiment analysis and opinion mining.  _Synthesis lectures on human language technologies_ , 5(1):1–167.
  * Longpre et al. (2020) Shayne Longpre, Yi Lu, and Joachim Daiber. 2020.  MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering.  _arXiv preprint arXiv:2007.15207_.
  * Masica (1993) Colin P Masica. 1993.  _The indo-aryan languages_.  Cambridge University Press.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of EMNLP_.
  * Mohaghegh et al. (2010) Mahsa Mohaghegh, Abdolhossein Sarrafzadeh, and Tom Moir. 2010.  Improved language modeling for English-Persian statistical machine translation.  In _Proceedings of the Workshop on Syntax and Structure in Statistical Translation_ , pages 75–82.
  * Mohaghegh et al. (2011) Mahsa Mohaghegh, Abdolhossein Sarrafzadeh, and Tom Moir. 2011.  Improving Persian-English statistical machine translation:experiments in domain adaptation.  In _Proceedings of the Workshop on South Southeast Asian Natural Language Processing (WSSANLP)_ , pages 9–15.
  * Pavlick et al. (2014) Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch. 2014\.  The language demographics of amazon mechanical turk.  _Transactions of the Association for Computational Linguistics_ , 2:79–92.
  * Pilevar et al. (2011) Mohammad Taher Pilevar, Heshaam Faili, and Abdol Hamid Pilevar. 2011.  Tep: Tehran english-Persian parallel corpus.  In _International Conference on Intelligent Text Processing and Computational Linguistics_ , pages 68–79. Springer.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  Xcopa: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of EMNLP_ , pages 2362–2376.
  * Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014.  SemEval-2014 task 4: Aspect based sentiment analysis.  In _Proceedings of the International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 27–35.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of WMT_ , pages 186–191.
  * Prokopidis et al. (2016) Prokopis Prokopidis, Vassilis Papavassiliou, and Stelios Piperidis. 2016.  Parallel global voices: a collection of multilingual corpora with citizen media stories.  In _Proceedings of LREC_ , pages 900–905.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of EMNLP_.
  * Rasooli et al. (2013) Mohammad Sadegh Rasooli, Ahmed El Kholy, and Nizar Habash. 2013.  Orthographic and morphological processing for Persian-to-English statistical machine translation.  In _Proceedings of IJCNLP_ , pages 1047–1051.
  * Richardson et al. (2013) Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013.  Mctest: A challenge dataset for the open-domain machine comprehension of text.  In _Proceedings of EMNLP_ , pages 193–203.
  * Seraji et al. (2013) Mojgan Seraji, Carina Jahani, Beáta Megyesi, and Joakim Nivre. 2013.  Uppsala Persian dependency treebank annotation guidelines.  Technical report, Uppsala University.
  * Shahshahani et al. (2019) Mahsa Sadat Shahshahani, Mahdi Mohseni, Azadeh Shakery, and Heshaam Faili. 2019\.  Payma: A tagged corpus of Persian named entities.  _Signal and Data Processing_ , 16(1):91–110.
  * Sharami et al. (2020) Javad PourMostafa Roshan Sharami, Parsa Abbasi Sarabestani, and Seyed Abolghasem Mirroshandel. 2020.  Deepsentipers: Novel deep learning models trained over proposed augmented persian sentiment corpus.  _arXiv preprint arXiv:2004.05328_.
  * Shavrina et al. (2020) Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020.  Russiansuperglue: A russian language understanding evaluation benchmark.  In _Proceedings of EMNLP_ , pages 4717–4726.
  * Simons and Fennig (2017) Gary F Simons and Charles D Fennig. 2017.  _Ethnologue: languages of Asia_.  sil International Dallas.
  * Sun et al. (2019) Chi Sun, Luyao Huang, and Xipeng Qiu. 2019.  Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence.  In _Proceedings of NAACL_ , pages 380–385.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  In _Proceedings of NAACL_ , pages 4149–4158.
  * Thet et al. (2010) Tun Thura Thet, Jin-Cheon Na, and Christopher SG Khoo. 2010.  Aspect-based sentiment analysis of movie reviews on discussion boards.  _Journal of information science_ , 36(6):823–848.
  * Tiedemann and Nygaard (2004) Jörg Tiedemann and Lars Nygaard. 2004.  The OPUS corpus - parallel and free: http://logos.uio.no/opus.  In _Proceedings of LREC_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Proceedings of NourIPS_ , pages 3266–3280.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of NAACL_ , pages 1112–1122.
  * Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020.  Clue: A chinese language understanding evaluation benchmark.  In _Proceedings of COLING_ , pages 4762–4772.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mt5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  Paws-x: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of EMNLP-IJCNLP_ , pages 3678–3683.
  * Zaremoodi et al. (2018) Poorya Zaremoodi, Wray Buntine, and Gholamreza Haffari. 2018.  Adaptive knowledge sharing in multi-task learning: Improving low-resource neural machine translation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 656–661.
  * Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020.  Improving massively multilingual neural machine translation and zero-shot translation.  In _Proceedings of ACL_ , pages 1628–1639.
  * Zukerman and Raskutti (2002) Ingrid Zukerman and Bhavani Raskutti. 2002.  Lexical query paraphrasing for document retrieval.  In _Proceedings of COLING_.


# Grid HTM: Hierarchical Temporal Memory for Anomaly Detection in Videos

Vladimir Monakhov  University of Oslo and SimulaMetNorway ,  Vajira Thambawita
SimulaMetNorway ,  Pål Halvorsen  SimulaMet and OsloMetNorway and  Michael A.
Riegler  SimulaMet and UiTNorway

(2018)

###### Abstract.

The interest for video anomaly detection systems has gained traction for the
past few years. The current approaches use deep learning to perform anomaly
detection in videos, but this approach has multiple problems. For starters,
deep learning in general has issues with noise, concept drift, explainability,
and training data volumes. Additionally, anomaly detection in itself is a
complex task and faces challenges such as unknowness, heterogeneity, and class
imbalance. Anomaly detection using deep learning is therefore mainly
constrained to generative models such as generative adversarial networks and
autoencoders due to their unsupervised nature, but even they suffer from
general deep learning issues and are hard to train properly. In this paper, we
explore the capabilities of the Hierarchical Temporal Memory (HTM) algorithm
to perform anomaly detection in videos, as it has favorable properties such as
noise tolerance and online learning which combats concept drift. We introduce
a novel version of HTM, namely, Grid HTM, which is an HTM-based architecture
specifically for anomaly detection in complex videos such as surveillance
footage.

HTM, deep learning, surveillance, anomaly detection

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Computer
vision

##  1\. Introduction

As the global demand for security and automation increases, many seek to use
video anomaly detection systems. In the US alone, the surveillance market is
expected to reach $\$23.60$ Billion by 2027 (us_video_stats, 1). Leveraging
modern computer vision, modern anomaly detection systems play an important
role in increasing monitoring efficiency and reducing the need for expensive
live monitoring. Their use cases can vary from detecting faulty products on an
assembly line to detecting car accidents on a highway.

The most important component in video anomaly detection systems is the
intelligence behind it. The intelligence ranges from simple on-board
algorithms to advanced deep learning models, where the latter has experienced
increased popularity in the past few years. Yet, despite the major progress
within the field of deep learning, there are still many tasks where humans
outperform models, especially in anomaly detection where the anomalies are
often undefined. Deep learning approaches also perform poorly when dealing
with noise and concept drift.

The cause for the discrepancy lies in the difference between how humans and
machine learning algorithms represent data and learn. Most machine learning
algorithms use a dense representation of the data and apply back-propagation
in order to learn. Human learning happens in the neocortex, where evidence
points to that the neocortex uses a sparse representation and performs
Hebbian-style learning. For the latter, there is a growing field of machine
learning dedicated to replicating the inner mechanics of the neocortex, namely
Hierarchical Temporal Memory (HTM) theory (BAMI, 2). This theory outlines its
advantages over standard machine learning, such as noise-tolerance and the
ability to adapt to changing data.

With the advantages of HTM and the rise of video anomaly detection in mind, a
natural question one could pose is whether it is possible to apply HTM for
anomaly detection in videos. Combined with a lack of related works, it is this
very question that is the motivation behind this paper. In this paper, we
therefore propose and evaluate Grid HTM which is a novel expansion of the base
HTM algorithm that allows for unsupervised anomaly detection in videos.

##  2\. Background

Anomaly detection is often defined as detecting data points that deviate from
the general distribution (anomaly_detection, 3). Unlike most other problems in
deep learning, anomaly detection deals with unpredictable and rare events
which makes it hard to apply traditional deep learning for anomaly detection.
A subset of anomaly detection is smart surveillance (anomalyvideo, 4), which
is the use of video analysis specifically in surveillance.

An issue for deep-learning models in general is that they are susceptible to
noise in the dataset (noise1, 5, 6), which leads to decreased model accuracy
and poor prediction results. Due to the nature of training deep learning
models, they are also in most cases not self-supervised and therefore require
constant tuning in order to stay effective on changing data. In addition, they
require a lot of data before they can be considered effective, and performance
increases logarithmically based on the volume of training data
(deeplearning_dataset, 7). Deep learning models also suffer from issues with
out-of-distribution generalization (deeplearning_ood_generalization_survey,
8), where a model might perform great on the dataset it is tested on, but
performs poorly when deployed in real life. This could be caused by selection
bias in the dataset or when there are differences in the causal structure
between the training domain and the deployment domain (deeplearning_ood, 9).
Another challenge with deep learning models is that they generally suffer from
a lack of explainability (XAI, 10). While it is known _how_ the models make
their decisions, their huge parametric spaces make it unfeasible to know _why_
they make those predictions. Combined with the vast potential that deep
learning offers in critical sectors such as medicine, makes approaches that
offer explainability highly attractive.

The HTM theory (BAMI, 2) introduces a machine learning algorithm which works
on the same principles as the brain and therefore solves some of the issues
that deep learning has. HTM is considered noise resistant and can perform
online learning, meaning that it learns as it observes more data. HTM
replicates the structure of the neocortex which is made up of cortical
regions, which in turn are made up of mini-columns and then neurons.

The data in an HTM model is represented using a Sparse Distributed
Representation (SDR), which is a sparse bit array. An encoder converts real
world values into SDRs, and there are currently encoders for numbers,
geospatial locations, categories, and dates. One of the difficulties with HTM
is making it work on visual data, where creating a good encoder for visual
data is still being researched (CNN_HTM, 11, 12, 13). The learning mechanism
consists of two parts, the Spatial Pooler (SP) and the Temporal Memory (TM).
The SP learns to extract semantically important information into output SDRs.
The TM learns sequences of patterns of SDRs and forms a prediction in the form
of a predictive SDR. A research study (AHMAD2017134, 14) has shown that HTM is
very capable of performing anomaly detection on low-dimensional data and is
able to outperform other anomaly detection methods. However, related works,
such as (MotionAnomalyDetection, 13), show that HTM struggles with higher
dimensional data. Therefore, a natural conclusion is that HTM should be
applied differently, and that a new type of architecture using HTM should be
explored for the purpose of video anomaly detection and surveillance.

##  3\. Grid HTM

This paper proposes and explores a new type of architecture, named Grid HTM,
for anomaly detection in videos using HTM, and proposes to use segmentation
techniques to simplify the data into an SDR-friendly format. These
segmentation techniques could be anything from simple binary thresholding to
deep learning instance segmentation. Even keypoint detectors such as Oriented
FAST and Rotated BRIEF (ORB) (orb_detector, 15) could in theory be applied.
When explaining Grid HTM, the examples will be taken from deep learning
instance segmentation of cars on a video from the VIRAT (VIRAT, 16) dataset.
An example segmentation is shown in Figure 1.

Figure 1. Segmentation result of cars, which is suited to be used as an SDR.
Original frame taken from VIRAT (VIRAT, 16).

The idea is that the SP will learn to find an optimal general representation
of cars. How general this representation is can be configured using the
various SP parameters, but ideally they should be set so that different cars
will be represented similarly while trucks and motorcycles will be represented
differently. An example representation by the SP is shown in Figure 2.

Figure 2. The SDR (left) and its corresponding SP representation (right). Note
that the SP is untrained.

The task of the TM will then be to learn the common patterns that the cars
exhibit, their speed, shape, and positioning will be taken into account.
Finally, the learning will be set so that new patterns are learned quickly,
but forgotten slowly. This will allow the model to quickly learn the norm,
even if there is little activity, while still reacting to anomalies. This
requires that the input is stationary, in our example this means that the
camera is not moving.

It is possible to split different segmentation classes into their respective
SDRs. This will give the SP and the TM the ability to learn different things
for each of the classes. For instance, if there are two classes ”person” and
”car”, then the TM will learn that it is normal for objects belonging to
”person” to be on the sidewalk, while objects belonging to ”car” will be
marked as anomalous when on the sidewalk.

Ideally, the architecture will have a calibration period spanning several days
or weeks, during which the architecture is not performing any anomaly
detection, but is just learning the patterns.

##  4\. Improvements

(MotionAnomalyDetection, 13) tested only the base HTM version and showed that
the algorithm cannot handle subtle anomalies, therefore multiple improvements
needed to be introduced to increase effectiveness.

Invariance. One issue that becomes evident is the lack of invariance, due to
the TM learning the global patterns. Using the example, it learns that it is
normal for cars to drive along the road but only in the context of there being
cars parked in the parking lot. It is instead desired that the TM learns that
it is normal for cars to drive along the road, regardless of whether there are
cars in the parking lot. We proposes a solution based on dividing the encoder
output into a grid and have a separate SP and TM for each cell in the grid.
The anomaly scores of all the cells are then aggregated into a single anomaly
score using an aggregation function.

Aggregation Function.  Selecting the correct aggregation function is important
because it affects the final anomaly output. For instance, it might be
tempting to use the mean of all the anomaly scores as the aggregation
function:

$\displaystyle X$ $\displaystyle:\\{x\in\mathbb{R}:x\geq 0\\}$ $\displaystyle
Anomaly\\_Score$ $\displaystyle=\dfrac{\sum\limits_{x\in X}x}{|X|}$

Where $X$ denotes the set of anomaly scores $x$ from each individual grid.
However, this leads to problems with normalization, meaning that an overall
anomaly score of 1 is hard to achieve due to many cells having a zero anomaly
score. In fact, it becomes unclear what a high anomaly score is anymore. Using
the mean also means that anomalies that take up a lot of space will be
weighted higher than anomalies that take up a little space, which might not be
desirable. To solve the aforementioned problem and if the data has little
noise, a potential aggregation function could be the non-zero mean:

$\displaystyle X$ $\displaystyle:\\{x\in\mathbb{R}:x>0\\}$ $\displaystyle
Anomaly\\_Score$ $\displaystyle=\begin{cases}\dfrac{\sum\limits_{x\in
X}x}{|X|}&\text{if }|X|>0\\\ \hfil 0&\text{otherwise}\end{cases}$

Meaning that only the cells with a strictly positive anomaly score, will be
contributing to the overall anomaly score which helps solve the aforementioned
normalization and weighting problem. On the other hand, the non-zero mean will
perform poorly when the architecture is exposed to noisy data which could lead
to there always being one or more cells with a high anomaly score.

Noisy data

(a) Mean.

(b) Non-zero mean.

Figure 3. Aggregation function performance on noisy data.

Clean data

(a) Mean.

(b) Non-zero mean.

Figure 4. Aggregation functions performance on clean data.

Figure 3 illustrates the effect of an aggregation function for noisy data,
where the non-zero mean is rendered useless due to the noise. On the other
hand, Figure 4 shows how the non-zero mean gives a clearer anomaly score when
the data is clean.

Explainability.  Having the encoder output divided into a grid has the added
benefit of introducing explainability into the model. By using Grid HTM it is
now possible to determine where in the input an anomaly has occurred by simply
observing which cell has a high anomaly score. It is also possible to estimate
the number of predictions for each cell which can be used as a measure of
certainty, where fewer predictions means higher certainty. Making it possible
to measure certainty per cell creates a new source of information which can be
used for explainability or robustness purposes.

Flexibility and Performance.  In addition, it is also possible to configure
the SP and the TM in each cell independently, giving the architecture
increased flexibility and to use a non-uniform grid, meaning that some cells
can have different sizes. Last but not least, dividing the frame into smaller
cells makes enables it to run each cell in parallel for increased performance.

Reviewing Encoder Rules.  A potential challenge with the grid approach is that
the rules for creating a good encoder, may not be respected and therefore
should be reviewed:

  * •

Semantically similar data should result in SDRs with overlapping active bits.
In this example, a car at one position will produce an SDR with a high amount
of overlapping bits as another car at a similar position in the input image.

  * •

The same input should always produce the same SDR. The segmentation model
produces a deterministic output given the same input.

  * •

The output must have the same dimensionality (total number of bits) for all
inputs. The segmentation model output has a fixed dimensionality.

  * •

The output should have similar sparsity (similar number of one-bits) for all
inputs and have enough one-bits to handle noise and subsampling. The
segmentation model does not respect this. An example is that there can be no
cars (zero active bits), one car ($n$ active bits), or two cars ($2n$ active
bits), and that this will fluctuate over time.

The solution for the last rule is two-fold, and consists of imposing a soft
upper bound and a hard lower bound for the number of active pixels within a
cell. The purpose is to lower the variation of number of active pixels, while
also containing enough semantic information for the HTM to work:

  * •

Pick a cell size so that the distribution of number of active pixels is as
tight as possible, while containing enough semantic information and also being
small enough so that the desired invariance is achieved. The cell size acts as
a soft upper bound for the possible number of active pixels.

  * •

Create a pattern representing emptiness, where the number of active bits is
similar to what can be expected on average when there are cars inside a cell.
This acts as a hard lower bound for the number of active pixels.

There could be situations where a few pixels are active within a cell, which
could happen when a car has just entered a cell, but this is acceptable as
long as it does not affect the distribution too much. If it does affect the
distribution, which can be the case with noisy data, then an improvement would
be to add a minimum sparsity requirement before a cell is considered not
empty, e.g. less than 5 active pixels means that the cell is empty. In the
following example, the number of active pixels within a cell centered in the
video was used to build the distributions seen in Figure 5:

(a) Without empty pattern.

(b) With empty pattern and a minimum sparsity requirement of $5$.

Figure 5. Distribution of number of active pixels within a cell of size
$12\times 12$.

With a carefully selected empty pattern sparsity, the standard deviation of
active pixels was lowered from $\mathbf{3.78}$ to $\mathbf{1.41}$. It is
possible to automate this process by developing an algorithm which finds the
optimal cell size and empty pattern sparsity which causes the least variation
of number of active pixels per cell. This algorithm would run as a part of the
calibration process.

The visual output resulting from these changes, which is an equally important
output as the aggregated anomaly score, can be seen in Figure 6 (for each cell
red means higher anomaly score, green lower anomaly score).

Figure 6. Example Grid HTM output and the corresponding input. The color
represents the anomaly score for each of the cells, where red means high
anomaly score and green means zero anomaly score. Two of the cars are marked
as anomalous because they are moving, which is something Grid HTM has not seen
before during its 300 frame (top right) long lifetime.

Since there are now cells that are observing an empty pattern for a lot of the
time in sparse data, boosting is recommended to be turned off, otherwise the
SP output for the empty cells would change back and forth in order to adjust
the active duty cycle.

Stabilizing Anomaly Output.

Figure 7. High anomaly score when an empty cell (represented with an empty
pattern with a sparsity value of 5) changes to being not empty, as something
enters the cell.

Another issue with the grid based approach is when a car first comes into a
cell. The TM in that cell has no way of knowing that a car is about to enter,
since it does not see outside its own cell, and therefore the first frame that
a car enters a cell will cause a high anomaly output. This is illustrated in
Figure 7 where it can be observed that this effect causes the anomaly output
to needlessly fluctuate. The band-aid solution is to ignore the anomaly score
for the frame during which the cell goes from being empty to being not empty,
which is illustrated in Figure 8.

Figure 8. The anomaly score is ignored (set to 0) for the frame in which the
cell changes state from empty to not empty.

A more proper solution could be to allow the TM to grow synapses to the TMs in
the neighboring cells, but this is not documented in any research papers and
might also hinder invariance.

Multistep Temporal Patterns.  Since the TM can only grow segments to cells
that were active in the previous timestep, it will struggle to learn temporal
patterns across multiple timesteps. This is especially evident in high
framerate videos, where an object in motion has a similar representation at
timestep $t$ and $t+1$, as an object standing still.

This could cause situations where an object that is supposed to be moving,
suddenly stands still, yet the TM will not mark it as an anomaly due to it
being stuck in a contextual loop. A contextual loop is when one of the
predictions at $t$ becomes true at $t+1$, and then one of the predictions at
$t+1$ is almost identical to the state at $t$, which becomes true if the
object is not moving, causing the TM to enter the same state that it was in at
$t$. A solution is to concatenate the past $n$ SP outputs as input into the
TM, which is made possible by keeping a buffer of past SP outputs and shifting
its contents out as new SP outputs are inserted. This follows the core idea
behind encoding time in addition to the data, which makes time act as a
contextual anchor. However, in this case there are no timestamps that are
suitable to be used as contextual anchors, so as a replacement, the past
observations are encoded instead.

Concatenating past observations together will force the TM input, for when an
object is in motion and when an object is still, to be unique. High framerate
videos can benefit the most from this, and the effect will be more pronounced
for higher values of $n$.

A potential side effect of introducing temporal patterns, is that because the
TM is now exposed to multiple frames at once, it will be more tolerant to
temporal noise. An example of temporal noise is when an object disappears for
a single frame due to falling below the classification threshold of the deep
learning segmentation model encoder. The reason for the noise tolerance is
that instead of the temporal noise making up the entire input for the TM, it
now only makes up $\frac{1}{n}$ of the TM input.

Use Cases.  The most intuitive use case is to use Grid HTM for semi-active
surveillance, where personnel only have to look at segments containing
anomalies, leading to drastically increased efficiency. One example is making
it possible to have an entire city be monitored by a few people. This is made
possible by making it so that people only have to look at segments that Grid
HTM has found anomalous, which is what drastically lowers the manpower
requirement for active monitoring of the entire city.

##  5\. Experimental details and results

As stated earlier, one of the use cases of Grid HTM is anomaly detection in
surveillance, and we using a video from the VIRAT (VIRAT, 16) video dataset
with long duration and a stationary camera, we demonstrate our system. The
video consists of technical anomalies in the form of several segments with
sudden frame skips in between. There is also a synthetic anomaly introduced in
the form of a frame repeat lasting a couple of seconds, essentially ”freezing”
time, in order to test whether Grid HTM is able to understand how objects
should be moving in time.

In this experiment, a segmentation model which can extract classes into their
respective SDRs is employed. Meaning that there could be an SDR for cars and
an SDR for persons, that are then concatenated before being fed into the
system. The segmentation model used is PointRend (pointrend, 17) with a
ResNet101 (resnet, 18) backbone, pretrained on ImageNet (imagenet, 19), and
implemented using PixelLib (pixellib, 20). For the sake of simplicity, this
experiment will focus only on the segmentation of cars. While on the topic of
segmentation, it is important to mention that the segmentation model is not
perfect and that there are cases where objects are misclassified as well as
cases where cars repeatedly go above and below the confidence threshold.

Figure 9. Anomaly score output from Grid HTM.

We can see in Figure 9 that Grid HTM is detecting when segments begin and end,
however it is not possible to use a threshold value to isolate them, and they
also have vastly different anomaly scores compared to each other. This is due
to the way the aggregation function works, which means that the anomaly output
is dependent on the physical size of the anomaly. It should also be noted that
a moving average ($n=200$) was applied to smooth out the anomaly score output,
otherwise the graph would be too noisy.

With the aggregation functions presented in this paper in mind, it is safe to
conclude that looking at the anomaly score output is meaningless for complex
data such as a surveillance video. This however does not mean that Grid HTM is
completely useless, and this can be observed by looking at the visual output
of Grid HTM. The visual output during which the first segment anomaly occurs
can be seen in Figure 10. Here, it is observed that Grid HTM correctly marks
the sudden change of cars when the current segment ends and a new segment
begins.

Figure 10. The first segment anomaly, which is marked with red text, and the
corresponding changes detected by Grid HTM. The numbers beneath each frame
represent the relative frame number and the current anomaly score
respectively.

In the original video, there is a road on which cars regularly drive. By
observing the visual output, it becomes evident that after some time Grid HTM
has mostly learned that behavior and does not report those moving cars as
anomalies. This is shown in Figure 11.

Figure 11. Visual output when a car is driving along a road.

To prove that Grid HTM has learned that cars on the road should be moving, it
is possible to look at the visual output during the period when the video is
repeating the same frame and observe if the architecture marks the cars
standing still on the road as anomalies.

Figure 12. Anomaly output during the repeating frame, the start of the frame
repeat is marked with red text. The blue circle highlights the object of
interest.

It can be observed in Figure 12 that the cars along the main road are not
marked as anomalies, but this could be attributed to the fact that there is a
crossing there and that cars periodically have to stop at that point to let
pedestrians cross.

On the other hand, when looking at the anomaly marked with a blue circle, the
car on the road in the parking lot is marked as an anomaly that increases in
severity as the time goes on during the frame repeat. The reason why that car
causes an anomaly is because, unlike the cars on the main road, a car is
rarely observed as standing still at that position. To prove that the anomaly
was actually directly caused by the repeating frame, and not just due to
repeating the anomaly in time, it should be compared to the anomaly output if
there was no repeating frame.

Figure 13. Anomaly output when there is no frame repeating, where it should
have repeated is marked in red. The blue circle highlights the object of
interest.

It can be observed in Figure 13 that the anomaly output is minor compared to
when there was a repeating frame, proving that the anomaly was indeed a
product of the repeating frame and that Grid HTM was able to learn how objects
should be moving in time.

Finally, it is interesting to look at how Grid HTM handles the repeating
frames without multistep temporal patterns, which is shown in Figure 14.

Figure 14. Anomaly output during the repeating frame, the start of the frame
repeat is marked with red text. The blue circle highlights the object of
interest. This time without multistep temporal patterns.

Unfortunately, simply disabling multistep temporal patterns without adjusting
the other TM parameters causes the same car to be marked as an anomaly before
and during the frame repeat. In fact, as previously mentioned, disabling
multistep temporal patterns causes Grid HTM to be less noise tolerant which
causes a lot more anomalies to be wrongly detected. This is evident in Figure
14, where a higher number of severe anomalies can be observed compared to
previous examples. This also highlights how sensitive HTM can be regarding
parameters. The working code for Grid HTM and the parameters for the
experiments conducted in for this paper can be found on
GitHub111https://github.com/vladim0105/GridHTM.

##  6\. Conclusion

We presented a novel method to perform anomaly detection in videos.
Experiments showed that the proposed Grid HTM can be used for unsupervised
anomaly detection in complex videos such as surveillance footage. One of the
most important future work would be to create a dataset with videos that are
several days long and contain anomalies such as car accidents, jaywalking, and
other similar anomalous behaviors. For Grid HTM, more time can be spent
exploring other aggregation functions so that the aggregated anomaly score can
be used more efficiently. Additionally, it would be a big benefit to create an
algorithm which can decide the parameters for each cell during the calibration
phase. It is also possible to improve explainability and robustness by
implementing a measure of certainty for each cell.

Finally, experiments should be performed to validate the possibility of having
the TM in each cell grow synapses to neighboring cells in order to solve the
issue with unstable anomaly output.

## References

  * (1) Divyanshi Tewari  “U.S. Video Surveillance Market by Component (Solution, Service, and Connectivity Technology), Application (Commercial, Military & Defense, Infrastructure, Residential, and Others), and Customer Type (B2B and B2C): Opportunity Analysis and Industry Forecast, 2020–2027”, Online, 2019  URL: https://www.alliedmarketresearch.com/us-video-surveillance-market-A06741
  * (2) J. Hawkins, S. Ahmad, S. Purdy and A. Lavin  “Biological and Machine Intelligence (BAMI)” Initial online release 0.4, 2016  URL: https://numenta.com/resources/biological-and-machine-intelligence/
  * (3) Guansong Pang, Chunhua Shen, Longbing Cao and Anton Van Den Hengel  “Deep Learning for Anomaly Detection”  In _ACM Computing Surveys_ 54.2 Association for Computing Machinery (ACM), 2021, pp. 1–38  DOI: 10.1145/3439950
  * (4) Sijie Zhu, Chen Chen and Waqas Sultani  “Video Anomaly Detection for Smart Surveillance”  arXiv, Online, 2020  DOI: 10.48550/ARXIV.2004.00222
  * (5) Shivani Gupta and Atul Gupta  “Dealing with Noise Problem in Machine Learning Data-sets: A Systematic Review” The Fifth Information Systems International Conference, 23-24 July 2019, Surabaya, Indonesia  In _Procedia Computer Science_ 161, 2019, pp. 466–474  DOI: https://doi.org/10.1016/j.procs.2019.11.146
  * (6) Dan Hendrycks and Thomas Dietterich  “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”  arXiv, Online, 2019  DOI: 10.48550/ARXIV.1903.12261
  * (7) Chen Sun, Abhinav Shrivastava, Saurabh Singh and Abhinav Gupta  “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era”  arXiv, Online, 2017  DOI: 10.48550/ARXIV.1707.02968
  * (8) Zheyan Shen et al.  “Towards Out-Of-Distribution Generalization: A Survey”  arXiv, 2021  DOI: 10.48550/ARXIV.2108.13624
  * (9) Alexander D’Amour et al.  “Underspecification Presents Challenges for Credibility in Modern Machine Learning”  arXiv, 2020  DOI: 10.48550/ARXIV.2011.03395
  * (10) Alejandro Barredo Arrieta et al.  “Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI”  In _Information Fusion_ 58, 2020, pp. 82–115  DOI: https://doi.org/10.1016/j.inffus.2019.12.012
  * (11) Y. Zou et al.  “Hierarchical Temporal Memory Enhanced One-Shot Distance Learning for Action Recognition”  In _Proceedings of the 2018 IEEE International Conference on Multimedia and Expo (ICME)_ , 2018, pp. 1–6  DOI: 10.1109/ICME.2018.8486447
  * (12) David McDougall 9000-times), Online, 2019  URL: https://github.com/htm-community/htm.core/issues/259#issuecomment-533333336
  * (13) Ilya Daylidyonok, Anastasiya Frolenkova and Aleksandr I. Panov  “Extended Hierarchical Temporal Memory for Motion Anomaly Detection”  In _Biologically Inspired Cognitive Architectures 2018_ Cham: Springer International Publishing, 2019, pp. 69–81  DOI: 10.1007/978-3-319-99316-4˙10
  * (14) Subutai Ahmad, Alexander Lavin, Scott Purdy and Zuha Agha  “Unsupervised real-time anomaly detection for streaming data” Online Real-Time Learning Strategies for Data Streams  In _Neurocomputing_ 262, 2017, pp. 134–147  DOI: https://doi.org/10.1016/j.neucom.2017.04.070
  * (15) Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary Bradski  “ORB: An efficient alternative to SIFT or SURF”  In _Proceedings of the 2011 International Conference on Computer Vision (ICCV)_ , 2011, pp. 2564–2571  DOI: 10.1109/ICCV.2011.6126544
  * (16) Sangmin Oh et al.  “A large-scale benchmark dataset for event recognition in surveillance video”  In _Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2011, pp. 3153–3160  DOI: 10.1109/CVPR.2011.5995586
  * (17) Alexander Kirillov, Yuxin Wu, Kaiming He and Ross Girshick  “PointRend: Image Segmentation as Rendering”  arXiv, Online, 2019  DOI: 10.48550/ARXIV.1912.08193
  * (18) Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep Residual Learning for Image Recognition”  In _Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778  DOI: 10.1109/CVPR.2016.90
  * (19) Jia Deng et al.  “ImageNet: A large-scale hierarchical image database”  In _Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009, pp. 248–255  DOI: 10.1109/CVPR.2009.5206848
  * (20) Ayoola Olafenwa  “Simplifying Object Segmentation with PixelLib Library”, Online, 2021  URL: https://vixra.org/abs/2101.0122