11institutetext: Department of Digital Humanities,
Faculty of Arts, University of Helsinki, Finland
11email<EMAIL_ADDRESS>
# When Word Embeddings Become Endangered

Khalid Alnajjar  0000-0002-7986-2994

###### Abstract

Big languages such as English and Finnish have many natural language
processing (NLP) resources and models, but this is not the case for low-
resourced and endangered languages as such resources are so scarce despite the
great advantages they would provide for the language communities. The most
common types of resources available for low-resourced and endangered languages
are translation dictionaries and universal dependencies. In this paper, we
present a method for constructing word embeddings for endangered languages
using existing word embeddings of different resource-rich languages and the
translation dictionaries of resource-poor languages. Thereafter, the
embeddings are fine-tuned using the sentences in the universal dependencies
and aligned to match the semantic spaces of the big languages; resulting in
cross-lingual embeddings. The endangered languages we work with here are
Erzya, Moksha, Komi-Zyrian and Skolt Sami. Furthermore, we build a universal
sentiment analysis model for all the languages that are part of this study,
whether endangered or not, by utilizing cross-lingual word embeddings. The
evaluation conducted shows that our word embeddings for endangered languages
are well-aligned with the resource-rich languages, and they are suitable for
training task-specific models as demonstrated by our sentiment analysis model
which achieved a high accuracy. All our cross-lingual word embeddings and the
sentiment analysis model have been released openly via an easy-to-use Python
library.

###### Keywords:

Cross-lingual Word Embeddings Endangered Languages Sentiment Analysis.

##  1 Introduction

The interest in building natural language processing (NLP) solutions for low-
resourced languages is constantly increasing [1], not only because of the
challenges associated with dealing with scarce resources but also because NLP
solutions facilitate documenting and analysing languages. Examples of such
solutions are applying optical character recognition to scan books [45],
normalizing historical variation [7], using speech recognition [19] and more.
However, most of the existing research is conducted in a simulated setting
[15, 24, 22] where a reduced portion of the resource-rich language is used to
represent a low-resourced language. Other approaches consider Wikipedias of
languages having a small number of articles (i.e., $<$ 500,000) such as Latin,
Hindi, Thai and Swahili [9, 48].

In this paper, we are dealing with languages that are classified as endangered
based on UNESCO Atlas111http://www.unesco.org/languages-atlas/index.php. These
languages are Erzya222See [37] for an insightful description of the situation
of the language. (myv), Moksha (mdf), Komi-Zyrian (kpv) and Skolt Sami (sms).
The most common methodology for documenting endangered languages is
constructing translation dictionaries, whether digitizing physical
dictionaries or reaching to native speakers. Universal dependencies (UD)
written by dedicated researchers studying such endangered languages might also
be available, and, in a fortunate scenario, they would include translations to
a language with more speakers. The bigger languages that endangered languages
are translated to are very inconsistent and vary depending on the language
family, geographically close languages and the languages spoken by the
documenter.

English, without a doubt, is currently the most resourced language in the
field of NLP. However, English translations are not frequently found for
endangered and low-resourced languages. To overcome this and make using
existing English resources possible, we leverage the recent advances in the
field of NLP for aligning word embeddings of big languages such as Finnish and
Russian with English word embeddings.

The contributions in this paper are:

  * •

Proposing a method for constructing word embeddings for low-resourced and
endangered languages, which are also aligned with word embeddings of big
languages.

  * •

Building a universal sentiment analysis model that achieves high accuracy in
both endangered and resource-rich languages covered in this work.

  * •

Releasing an open-source and easy-to-use Python library with all the word
embeddings and the sentiment analyzer model to support the community and
researchers333https://github.com/mokha/semantics.

This paper is structured as follows. Section 2 contains a brief description of
the related work on building cross-lingual low-resource word embeddings.
Thereafter, we describe the linguistic resources used in this work, including
the translation dictionaries, universal dependencies and existing word
embeddings of resource-rich. The proposed method for constructing cross-
lingual word embeddings for endangered languages is elaborated then, followed
by the description of the sentiment analysis model. We then present the
results and evaluation for word embeddings and sentiment analysis model.
Lastly, we discuss and highlight our remarks in the conclusions.

##  2 Related work

The largest scale model for capturing the computational semantics of
endangered Uralic languages, Erzya, Moksha, Komi-Zyrian and Skolt Sami, is,
perhaps, SemUr [16]. The database consists of words that are connected to each
other based on their syntactic co-occurrences in a large internet corpus for
Finnish. The extracted relations have been automatically translated by using
Jack Rueter’s XML dictionaries. In human evaluation, the quality was
surprisingly acceptable given that the method was based on word-level
translations. This gives hope in using these high-quality dictionaries in
building computational semantic models.

Apart from SemUr, there has not been any other attempts in automatically
modelling semantics for endangered Uralic languages. Some recent work, however
presents interesting work on higher-resourced languages using word embeddings
[2, 11]. In general, word embeddings based methods such as word2vec [28] and
fastText [6] are optimal for the task of applying high-resource language data
to endangered languages as they work on word-level.

Several recent approaches such as GPT-2 [34], ELMo [33] and BERT [10] aim to
capture richer semantic representations from text. However, they are very data
intensive and their representation is no longer on the level of individual
words. This makes it more difficult to use them for endangered languages.

Recently, neural networks have been used heavily in the field of NLP due to
their great capabilities in learning a generalization, which resulted in high
accuracies. However, neural networks demand a large amount of data, which
usually is not available for low-resource languages. Despite this, researchers
have employed neural networks in a low-resource setting by producing synthetic
data. For instance, Hämäläinen and Rueter have built a neural network to
detect cognates for between two endangered languages [18], Skolt Sami and
North Sami. Their approach reached to a better accuracy when they combined
data, synthetically produced by a statistical model, with real data.

##  3 Linguistic resources

Here, we describe the linguistic resources used throughout the research
presented in this paper. We will focus on resources related to the endangered
languages (i.e., Erzya, Moksha, Komi-Zyrian and Skolt Sami), while still
providing a brief introduction to resource-rich resources. The resources for
endangered languages that we cover here are: 1) translation dictionaries, 2)
universal dependencies and 3) finite-state transducers. This list by no means
is inclusive of all available and useful resources for endangered languages,
as additional resources might exist such as the work of Jack Rueter on Online
dictionaries [41] and making them usable even through click-in-text interfaces
[38]. In terms of resource-languages, we describe their word-embeddings.

###  3.1 Translation dictionaries

Low-resource and endangered languages commonly have translation dictionaries
to a bigger language. For our case, such dictionaries are multilingual and are
provided in an Extensible Markup Language (XML) format. Fortunately, the
target languages of the translations are mostly consistent in all the
dictionaries (which is not the typical case), but each dictionary contains
different portions of translations.

Table 1 shows a statistical summary of the translations existing in the
dictionaries. The source language represents the endangered languages and the
target language indicates the resource-rice language. A meaning group in the
dictionaries may contain multiple translations that can be used
interchangeably as they share the same meaning. The analysis shows that for
Erzya (myv) and Skolt Sami (sms), Finnish (fin) translations are the most
common ones, whereas Russian (rus) and English (eng) translations are the most
frequent ones for Komi-Zyrian (kpv) and Moksha (mdf).

Table 1: An overview of translation in the XML dictionaries of the low-resourced languages. Language codes are given in ISO 639-3. Source | Target | Meaning | Translations | Total
---|---|---|---|---
language | language | groups
myv | fin | 8388 | 14344 (59.89%) | 23950
rus | 5631 | 7608 (31.77%)
eng | 1917 | 1998 (8.34%)
kpv | fin | 1352 | 2046 (14.89%) | 13744
rus | 15492 | 10585 (77.01%)
eng | 1078 | 1113 (8.10%)
sms | fin | 20503 | 27522 (68.02%) | 40461
rus | 4872 | 6026 (14.89%)
eng | 5824 | 6913 (17.09%)
mdf | rus | 37 | 37 (0.27%) | 13626
eng | 6587 | 13589 (99.73%)

Entries in the dictionaries are in the lemma form, and, typically, their part-
of-speech tags are provided. Further metadata information might exist, such as
stems and example usages of the word in the source language. We use the Giella
[29] dictionaries that have been mainly authored by Jack Rueter through
UralicNLP [21]. While Moksha has Finnish translations, the Moksha dictionary
in UralicNLP did not contain any of these translations because the data was
missing from the repository.

###  3.2 Universal dependencies

Universal dependencies (UD) [47] is a standard framework for annotating the
grammar (parts of speech, morphological features, and syntactic dependencies
of sentences. Additionally, UD allows annotators to supply their own comments.
In the UD we are dealing with, translation sentences might appear in the
comments. The UD of the endangered languages can be obtained directly from
Universal Dependencies’ website444https://universaldependencies.org/. At the
time of writing, 1,690, 167, 104 and 435 sentences were in Erzya’s [44],
Moksha’s [42], Skolt Sami’s [30] and Komi-Zyrian’s UDs555There is a UD for
Komi-Permyak [43] which is close to Komi-Zyrian. [32], respectively. These
numbers highlight the insufficient amount of data present for training machine
learning or NLP models for endangered languages. We have used the UralicNLP
[21], a Python library, to read the universal dependencies.

###  3.3 Finite-state transducers

The common automatic tools found for endangered languages are finite-state
transducers (FSTs), as they are rule-based which allows language experts to
define how the finite-state machine should behave depending on the language.
As a result, FSTs make it possible to lemmatize words and produce mini- and
full-paradigms. In this work, we use Jack Rueter’s FSTs for Skolt Sami [39],
Erzya and Moksha [40], and Komi-Zyrian [36]. The FSTs are supplied as part of
the UralicNLP [21] Python library.

###  3.4 Word embeddings of resource-rich languages

Word embeddings are a vector representation of words, which are built based on
the surrounding context of the word. Semantic similarity between words
captured in the word embeddings can be measured using cosine similarity, which
can then be utilized to cluster meanings in text [17]. Common usages for word
embeddings is to acquire semantically similar words to an input word. For
example, the most 5 similar words to “king” are “queen”, “monarch”, “prince”,
“sultan”, and “ruler”. The vector nature of these words makes it possible to
perform vector operations such as addition, multiplication and subtractions.
With such operations, analogies could be predicted such as “king” - “man” +
“woman” = “queen”. Simply, this asks what is the equivalent of a king that is
not a man but rather a woman in the semantic space, the answer is a queen.

When building word embeddings, there are many preprocessing configurations and
hyperparameters that influence the performance of the models, such as
lemmatization, part-of-speech tagging, window size, the dimension size of the
embeddings, minimum and maximum thresholds for word frequencies and so on.
There is no fixed nor optimal configuration that is apt for all applications.

In the translation dictionaries, words and their translations are provided in
their lemma form. Due to this reason, the vocabulary in any word embeddings we
will be using has to be lemmatized. Ideally, all the hyperparameters and
configurations for word embeddings should be the same to capture similar
features and semantics, which would yield better results across models once
they are aligned. For the scope of this research, we use the most similar
models we could get our hands on.

We utilize the Russian and English [12], and Finish [25] word embeddings. The
Russian embeddings are trained on a news corpus, while the English is based on
Wikipedia and Gigaword 5th Edition corpora [31]. The Finnish word embeddings
are trained on Common Crawls. The dimension size of the English and Russian
embeddings is 300 but 200 is the size of the Finnish one. The window size is 5
for all embeddings but Finnish, which is 2. These differences, in addition to
other reasons, end up affecting the quality of the models we will build of
endangered languages. We discuss them more in the Discussion section.

##  4 Cross-lingual word embeddings for endangered languages

Cross-lingual word embeddings are word embeddings where vectors across
multiple languages are aligned. For instance, the vector for “dog” in the
English embeddings points roughly to the same direction for the same word in
other languages (i.e., “koira” and “ собака” for Finnish and Russian,
respectively). Example applications for employing cross-lingual word
embeddings are: headline generation [5], loan word identification [27] and
cognate identification [26].

Before we build and align the word embeddings, we apply a dimensionality
reduction using the method proposed in [35] to the three pre-trained models
(i.e., English, Russian and Finnish). We set the target dimension to 100. This
is to ensure that the vectors in all the embeddings share the same size.
Subsequently, we process the vocabulary of the Finnish by removing all
occurrences of the hashtag symbol “#”, which is there to mark compounds.
Regarding the Russian word embeddings, the vocabulary contained part-of-speech
information and, hence, each lemma might be present multiple times. To address
this, we discard the part-of-speech information and use all vectors matching
the target lemma.

To align the main three word embedding models, we employ the state-of-the-art
supervised multilingual word embeddings alignment technique introduced in MUSE
[8]. Figure 1 illustrates transforming the word embeddings of the source
language $X$ with the target language $Y$ so that words in both languages are
aligned together. In this example the source language is English and the
target language is Italian. What supervised means in this context is that the
alignment process relies on a bilingual dictionary that guides the
transformation process. In our work, we set the target language to English and
align both Russian and Finnish models with it using the bilingual dictionaries
released as part of MUSE. The models are refined over 20 iterations.

Figure 1: A visualization of the transformation process of aligning word
embeddings in $X$ in accordance to the ones in $Y$, taken from [8].

Following the alignment of the resource-rich models, we construct the word
embeddings for the endangered languages: Erzya, Moksha, Komi-Zyrian and Skolt
Sami. In doing so, we iterate over all the lexemes in the dictionary of a
given endangered language. In the case where a lexeme had translations to any
of the three resource-rich languages and the translation existed in the word
embeddings of the corresponding language, a vector for the lexeme is
constructed as the centroid –an average vector– of all translation vectors.

Once the word embeddings for the endangered languages have been constructed,
we fine-tune them using the sentences in their universal dependencies. Lastly,
we realign each word embeddings model with the resource-rich language having
most translations to. In other words, Erzya and Skolt Sami are aligned with
Finnish but Komi-Zyrian and Moksha are aligned with Russian and English,
respectively. The models are aligned over 5 refinement steps.

Table 2: The top 3 semantically similar words to the English vector, in all
languages. The score after the colon is the semantic similarity score (the
higher, the more similar).

eng (input) fin rus myv mdf sms kpv dog koira: 0.7100 поймать: 0.4354
пинелевкс: 0.8172 пине: 0.9977 piânnai: 0.7197 гиджгысь: 0.5691 kissa: 0.6618
убивать: 0.4310 псарня: 0.6691 кутерь: 0.8220 piânnaž: 0.7078 барсук: 0.5271
namipalo: 0.6263 .родственник: 0.4271 киска: 0.6340 ката: 0.7547 kaazzâž:
0.6521 вежель: 0.5238 cat rotta: 0.6631 щенок: 0.5246 обизьган: 0.7474 ката:
0.9990 kaazzâž: 0.6800 черепаха: 0.6121 Fretti: 0.6484 бася: 0.4885 лыркай:
0.7018 зака: 0.9990 žee$\prime$vet: 0.6672 питон: 0.5996 kissa: 0.6461
детеныша: 0.4794 пинелевкс: 0.6993 пине: 0.7478 kue$\prime$ttžeevai: 0.6665
ӧбезьяна: 0.5937 king Ahasveros: 0.5551 наследник: 0.4433 инеазормастор:
0.6751 кароль: 0.9971 daam: 0.5301 королева: 0.4869 kuningas: 0.5522 гордо:
0.4188 озадкс: 0.6643 тюштя: 0.8601 koonõs: 0.5214 принц: 0.4736 kuninkas:
0.5243 исходемик: 0.4122 инеазоронь: 0.6315 оцязор: 0.7768 lââ$\prime$ssnõmm:
0.5035 герцог: 0.4648 queen kuningatar: 0.5902 паркер-боулз: 0.5578
инеазорава: 0.9954 оцязорава: 0.9972 koonõskaav: 0.7180 королева: 0.7227
prinsessa: 0.5867 энистон: 0.5314 инеазоронь: 0.5428 kevärj: 0.6191
prince$\prime$ss: 0.5865 принцесса: 0.6614 kruununprinsessa: 0.5686 чад:
0.5063 венчакай: 0.5360 лемдяз: 0.6191 kå$\prime$ll-lå$\prime$dd: 0.5457
принц: 0.4903 car auto: 0.7728 машина: 0.6621 автомобиль: 0.7716 машина:
0.8568 autt: 0.6826 мотик: 0.6299 kottero: 0.6843 бмв: 0.6234 автомашина:
0.7716 автокрандаз: 0.6957 mõõnnâmneävv: 0.6572 водитель: 0.5915 katumaasturi:
0.6627 bmw: 0.6170 уаз: 0.7438 ардомбяль: 0.6377 luâđastvuejjamautt: 0.6438
автобусса: 0.5914 man spolle: 0.5062 пожизненно: 0.4450 муюкт: 0.4911 аля:
0.7974 so$\prime$rmmjeei: 0.4548 айулов: 0.4970 pedofiiliä: 0.5029
остин::пауэрс: 0.4377 нарт: 0.4911 ава: 0.5362 upsee$\prime$r: 0.4522
катаржик: 0.4538 puukottaja: 0.4986 кривенко: 0.4291 гурямка: 0.4869 сакал:
0.5212 nuõrrooumaž: 0.4468 допроситны: 0.4063 woman romaninainen: 0.5813
юлия::печерская: 0.5349 аваломань: 0.5539 ава: 0.9988 neezzan: 0.6134 айулов:
0.4585 somalinainen: 0.5713 столбова&mdash: 0.5157 авасыме: 0.5428 авакань:
0.6255 ååumai: 0.4610 мам: 0.4035 maahanmuuttajanainen: 0.5436 воспитатель:
0.5079 пекиязь: 0.4938 ни: 0.5704 åålm: 0.4610 колготки: 0.3830 France Ranska:
0.6330 франция: 0.5325 французонь: 0.4922 Кранцмастор: 0.9964 Franskkjânnam:
0.6357 забастовка: 0.4077 Belgia: 0.6097 деша: 0.4916 француз: 0.4922 кранц:
0.7155 Jõnn-Britann: 0.5778 кӧрень: 0.3972 Iso-Britannia: 0.5757 арно: 0.4801
австриец: 0.4586 кранцава: 0.7155 Itaal: 0.5331 японец: 0.3698 Finland Tanska:
0.5735 тудегешев: 0.4599 Финляндия: 0.4457 шведонь: 0.6399 Lä$\prime$dd:
0.6780 ненеч: 0.4165 Norja: 0.5732 инсбрук: 0.4462 Суоми: 0.4457 шведава:
0.6399 Lää$\prime$ddjânnam: 0.6780 вужкыв: 0.3451 Viro: 0.5612 либерец: 0.4398
Россия: 0.4384 швед: 0.6399 Taarr: 0.6424 подувкыв: 0.3451 see Muuttu: 0.4886
видеть: 0.5243 покш: 0.5228 няемс: 0.9982 õinn: 0.5315 дзик: 0.4642 tämä:
0.4860 тренд: 0.5057 те: 0.5200 няема: 0.5860 o$\prime$ddjõõttâd: 0.4936 эсся:
0.4590 ainavain: 0.4824 тенденция: 0.4935 но: 0.4920 ила-крда: 0.5248 tiett-
aa: 0.4881 шензьӧдлыны: 0.4540 want haluta: 0.5709 жить: 0.5960 мирямс: 0.5852
пиштемс: 0.6032 soovšed: 0.5562 гажавны: 0.8506 siksi: 0.5219 хотеть: 0.5225
секс: 0.5654 мезевок: 0.5864 haa$\prime$leed: 0.5319 желайтны: 0.6250 molempi:
0.5146 актриса::юлия::михалков: 0.5222 одямс: 0.5654 мезе-мезе: 0.5861
jee$\prime$res: 0.5257 ньӧтчыдысь: 0.5561 day lomaaamu: 0.5336 утра: 0.4900
поздаямс: 0.4964 цяс: 0.5621 minut: 0.4695 мӧдасув: 0.4533 reissupäivä: 0.5230
7days.ru: 0.4613 покшнэ: 0.4948 ой: 0.5621 jâđđa: 0.4684 вежонпом: 0.4377
lähtöa: 0.5088 выплакать: 0.4506 час: 0.4730 шиньгучка: 0.5351
kõskkpei$\prime$vv: 0.4673 салют: 0.3980

##  5 Sentiment analysis

In this section, we describe an experiment with the newly produced word
embeddings. We apply them in the task of sentiment analysis. We hand pick all
positive and negative sentences from the Erzya treebank [44] based on the
translations provided in the treebank in English and Finnish. This constitutes
our Erzya test corpus that contains 23 negative sentences and 22 positive
sentences, giving us a total of 45 sentences.

We use the Stanford Sentiment Treebank for English [46] to train our sentiment
analyzer model. As the Erzya test data is binary – negative and positive
sentences – we treat the sentiment information in the treebank as binary as
well, ignoring any neutral examples. It is important to note that we do not
use any examples written in Erzya during the training, only sentences in
English.

We train a neural model that takes in a sentence in English as a source and a
sentiment label (positive or negative) as a target. We train the neural model
with the aligned embeddings by substituting the words in the input sentences
with their vectors. As our models are lemmatized, we need to ensure that all
words are lemmatized in the in input as well. We use spaCy [20] for this
lemmatization step. The architecture and training of the neural model is
inspired by the work presented in [23], where bi-grams are added to the input
sentences during the training phase and the neural network is a linear
classifier.

Table 3: Example sentences in Erzya and their translations in English, along
with the predicted sentiment by our method for each sentence.

Erzya | English | Sentiment
---|---|---
Зярошкаль цёрыненть кенярксозо! | You can imagine the boy’s delight! | Positive
Чизэ лембе. | It is a warm day.
Сехте паро шка. | The best time of all.
Цёрынентень аламодо визькс теевсь. | The boy felt a little ashamed. | Negative
Баягинень ёмавтомась — пек берянь тешксэсь. | Losing a bell was a really bad sign.
Весе те — апаро вийтнень тандавтнемс. | This is all meant to scare away the evil spirits.

Table 3 shows some examples of the input in Erzya, its translation in English
and the correctly predicted label. For Erzya, we use the lemmas from the
treebank, and get their closest English vectors through the aligned word
embeddings. This way, the model treats the Erzya sentences as though they were
English and it can predict the sentiment in the language it did not see during
the training. The resulting model was trained for 30 epochs and it reached to
53.3% accuracy for Erzya and 75.5% accuracy for English in the treebank
sentences and an accuracy of 83.5% in English in the Stanford Sentiment
Treebank dataset. We have obtained an accuracy boost for Erzya predictions,
reaching 57.8%, when we also considered vectors of other resource-rich
languages with the aid of the translation dictionary (Finnish in this case, as
Erzya has many translations to Finnish).

The resulting accuracy is respectable given that the test data is
fundamentally different from the training data. First of all, the testing and
training are in different languages. Second of all, they represent very
different genres: the training data is based on movie reviews, whereas the
testing data has sentences from novels.

##  6 Discussion and Conclusions

The work conducted in this paper has been a first step for using machine
learning in modelling the semantics of some of the endangered Uralic
languages. It is evident that these aligning based approaches embraced before
in the literature cannot get us too far in truly representing the semantics do
to socio-cultural mismatches in concepts. For instance, we saw that Finland,
which is a very important concept for a Finnish model was completely
misaligned with geographically close countries such as Denmark, Norway and
Estonia. Alignment can only get us so far and using models trained on larger
languages has its inherent problems when applied to completely new domains in
a completely different language.

Even the starting quality for the pretrained embeddings was low. The Russian
model was unacceptably bad and the Finnish model has too many words that are
not lemmatized at all, or are lemmatized to a wrong lemma. When the quality of
the models available for a high-resourced languages is substandard, one cannot
expect any sophisticated machine learning method to come to the rescue.
Unfortunately in our field, too little attention is paid to the quality of
resources and more attention is paid into single values representing overall
accuracies and overall performance.

As there is no shortcut to happiness, we should look into the data available
in the endangered languages themselves. For instance, FU-Lab has a plethora of
resources for Komi languages [14, 13] that are just waiting for lemmatization.
Once lemmatized, these resources could be used to build word embeddings
directly in that language. Of course, this requires collaboration between many
parties and willingness to make data openly available. While this might not be
an issue with FU-Lab, it might be with some other instances holding onto their
immaterial rights too tight.

At the current, stage our dictionary editing system, Ve$\prime$rdd [4, 3],
contains words for multiple endangered languages and their translations in a
graph structure. This data could be extended by predicting new relations into
the graph with semantic models such as word embeddings. This could help at
least in resolving meaning groups and polysemy of the lexical entries.
However, the word embeddings available for the endangered languages in
question has not yet reached to a stage mature enough for their incorporation
as a part of the lexicon.

##  7 Acknowledgement

I would like to dedicate the acknowledgement section to Jack Rueter, for all
his work on endangered languages and his brilliant ideas on improving the
current technological state of endangered languages. Jack’s enthusiasm and
dedication to endangered languages is clearly shown in all the various
dictionaries and FSTs built and maintained by him. He supervised my work on
building the dictionary editing system, Ve$\prime$rdd [4, 3]. He was always
available for discussing and supporting my work, without him Ve$\prime$rdd
would not be in the great level it is at at the moment. He truly is a pioneer
in the field, and the entire community appreciates all of his work.

## References

  * [1] Proceedings of the 3rd Workshop on the Use of Computational Methods in the Study of Endangered Languages Volume 1 (Papers). Association for Computational Linguistics, Honolulu (Feb 2019), https://www.aclweb.org/anthology/W19-6000
  * [2] Adams, O., Makarucha, A., Neubig, G., Bird, S., Cohn, T.: Cross-lingual word embeddings for low-resource language modeling. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. pp. 937–947. Association for Computational Linguistics, Valencia, Spain (Apr 2017), https://www.aclweb.org/anthology/E17-1088
  * [3] Alnajjar, K., Hämäläinen, M., Rueter, J.: On editing dictionaries for uralic languages in an online environment. In: Proceedings of the Sixth International Workshop on Computational Linguistics of Uralic Languages. p. 26–30. The Association for Computational Linguistics (2020)
  * [4] Alnajjar, K., Hämäläinen, M., Rueter, J., Partanen, N.: Ve’rdd. narrowing the gap between paper dictionaries, low-resource nlp and community involvement. In: Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations. International Committee on Computational Linguistics (2020)
  * [5] Alnajjar, K., Leppänen, L., Toivonen, H.: No time like the present: Methods for generating colourful and factual multilingual news headlines. In: Proceedings of the 10th International Conference on Computational Creativity. pp. 258–265. Association for Computational Creativity (Jun 2019)
  * [6] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, 135–146 (2017)
  * [7] Bollmann, M.: A large-scale comparison of historical text normalization systems. arXiv preprint arXiv:1904.02036 (2019)
  * [8] Conneau, A., Lample, G., Ranzato, M., Denoyer, L., Jégou, H.: Word translation without parallel data. arXiv preprint arXiv:1710.04087 (2017)
  * [9] Das, A., Ganguly, D., Garain, U.: Named entity recognition with word embeddings and wikipedia categories for a low-resource language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 16(3), 1–19 (2017)
  * [10] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423, https://www.aclweb.org/anthology/N19-1423
  * [11] Duong, L., Kanayama, H., Ma, T., Bird, S., Cohn, T.: Learning crosslingual word embeddings without bilingual corpora. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016). Association for Computational Linguistics, Texas, USA (November 2016)
  * [12] Fares, M., Kutuzov, A., Oepen, S., Velldal, E.: Word vectors, reuse, and replicability: Towards a community repository of large-text resources. In: Proceedings of the 21st Nordic Conference on Computational Linguistics, NoDaLiDa, 22-24 May 2017, Gothenburg, Sweden. vol. 0, pp. 271–276. Linköping University Electronic Press, Linköpings universitet (2017)
  * [13] Федина, МС: Создание официально-делового подкорпуса национального корпуса коми языка. In: УПРАВЛЕНИЕ СОЦИАЛЬНО-ЭКОНОМИЧЕСКИМ РАЗВИТИЕМ СУБЪЕКТА РОССИЙСКОЙ ФЕДЕРАЦИИ, pp. 205–216 (2015)
  * [14] Федина, Марина Серафимовна: Корпус Коми Языка Как База Для Научных Исследований. In: II Международная научная конференция «Электронная письменность народов Российской Федерации: опыт, проблемы и перспективы» проводится в рамках реализации Государственной программы «Сохранение и развитие государственных языков Республики Башкортостан и языков народов Республики Башкортостан» на 2019–2024 гг. Ответственный редактор: Ахмадеева АУ. p. 45 (2019)
  * [15] Gu, J., Hassan, H., Devlin, J., Li, V.O.: Universal neural machine translation for extremely low resource languages. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 344–354 (2018)
  * [16] Hämäläinen, M.: Extracting a semantic database with syntactic relations for finnish to boost resources for endangered uralic languages. The Proceedings of Logic and Engineering of Natural Language Semantics 15 (LENLS15) (2018)
  * [17] Hämäläinen, M., Alnajjar, K.: Let’s FACE it: Finnish poetry generation with aesthetics and framing. In: 12th International Conference on Natural Language Generation. pp. 290–300. The Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/w19-8637
  * [18] Hämäläinen, M., Rueter, J.: Finding Sami cognates with a character-based NMT approach. In: Proceedings of the 3rd Workshop on the Use of Computational Methods in the Study of Endangered Languages Volume 1 (Papers). pp. 39–45. Association for Computational Linguistics, Honolulu (Feb 2019), https://www.aclweb.org/anthology/W19-6006
  * [19] Hjortnaes, N., Partanen, N., Rießler, M., Tyers, F.M.: Towards a speech recognizer for komi, an endangered and low-resource uralic language. In: Proceedings of the Sixth International Workshop on Computational Linguistics of Uralic Languages. pp. 31–37 (2020)
  * [20] Honnibal, M., Montani, I., Van Landeghem, S., Boyd, A.: spaCy: Industrial-strength Natural Language Processing in Python (2020). https://doi.org/10.5281/zenodo.1212303
  * [21] Hämäläinen, M.: UralicNLP: An NLP library for Uralic languages. Journal of Open Source Software 4(37), 1345 (2019). https://doi.org/10.21105/joss.01345
  * [22] Hämäläinen, M., Alnajjar, K.: A template based approach for training nmt for low-resource uralic languages - a pilot with finnish. In: ACAI 2019: Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence. pp. 520–525. ACM, United States (Dec 2019). https://doi.org/10.1145/3377713.3377801
  * [23] Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. pp. 427–431. Association for Computational Linguistics, Valencia, Spain (Apr 2017), https://www.aclweb.org/anthology/E17-2068
  * [24] Khayrallah, H., Thompson, B., Post, M., Koehn, P.: Simulated multiple reference training improves low-resource machine translation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 82–89. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.7, https://www.aclweb.org/anthology/2020.emnlp-main.7
  * [25] Laippala, V., Ginter, F.: Syntactic n-gram collection from a large-scale corpus of internet finnish. In: Human Language Technologies-The Baltic Perspective: Proceedings of the Sixth International Conference Baltic HLT. vol. 268, p. 184 (2014)
  * [26] Lefever, E., Labat, S., Singh, P.: Identifying cognates in English-Dutch and French-Dutch by means of orthographic information and cross-lingual word embeddings. In: Proceedings of the 12th Language Resources and Evaluation Conference. pp. 4096–4101. European Language Resources Association, Marseille, France (May 2020), https://www.aclweb.org/anthology/2020.lrec-1.504
  * [27] Mi, C., Yang, Y., Wang, L., Zhou, X., Jiang, T.: Toward better loanword identification in Uyghur using cross-lingual word embeddings. In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 3027–3037. Association for Computational Linguistics, Santa Fe, New Mexico, USA (Aug 2018), https://www.aclweb.org/anthology/C18-1256
  * [28] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  * [29] Moshagen, S., Rueter, J., Pirinen, T., Trosterud, T., Tyers, F.M.: Open-Source Infrastructures for Collaborative Work on Under-Resourced Languages (2014), the LREC 2014 Workshop “CCURL 2014 - Collaboration and Computing for Under-Resourced Languages in the Linked Open Data Era”
  * [30] Nivre, J., Zeman, D., Rueter, J., Juutinen, M., Hämäläinen, M.: Ud_skolt_sami-giellagas 2.7 (Nov 2020)
  * [31] Parker, R., Graff, D., Kong, J., Chen, K., Maeda, K.: English gigaword fifth edition, 2011. Linguistic Data Consortium, Philadelphia, PA, USA (2011)
  * [32] Partanen, N., Blokland, R., Lim, K., Poibeau, T., Rießler, M.: The first Komi-Zyrian Universal Dependencies treebanks. In: Proceedings of the Second Workshop on Universal Dependencies (UDW 2018). pp. 126–132. Association for Computational Linguistics, Brussels, Belgium (Nov 2018). https://doi.org/10.18653/v1/W18-6015, https://www.aclweb.org/anthology/W18-6015
  * [33] Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Proc. of NAACL (2018)
  * [34] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [35] Raunak, V., Gupta, V., Metze, F.: Effective dimensionality reduction for word embeddings. In: Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019). pp. 235–243. Association for Computational Linguistics, Florence, Italy (Aug 2019). https://doi.org/10.18653/v1/W19-4328, https://www.aclweb.org/anthology/W19-4328
  * [36] Rueter, J.: Хельсинкиса университетын кыв туялысь Ижкарын перымса кывъяс симпозиум вылын лыддьӧмтор. Permistika pp. 154–158 (2000)
  * [37] Rueter, J.: The erzya language. where is it spoken? Études finno-ougriennes (45) (2013)
  * [38] Rueter, J.: Giellatekno open-source click-in-text dictionaries for bringing closely related languages into contact. In: Proceedings of the Third Workshop on Computational Linguistics for Uralic Languages. pp. 8–9 (2017)
  * [39] Rueter, J., Hämäläinen, M.: FST morphology for the endangered Skolt Sami language. In: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL). pp. 250–257. European Language Resources association, Marseille, France (May 2020), https://www.aclweb.org/anthology/2020.sltu-1.35
  * [40] Rueter, J., Hämäläinen, M., Partanen, N.: Open-source morphology for endangered mordvinic languages. In: Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS). pp. 94–100. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.nlposs-1.13, https://www.aclweb.org/anthology/2020.nlposs-1.13
  * [41] Rueter, J., Hämäläinen, M.: Synchronized mediawiki based analyzer dictionary development. In: Proceedings of the Third Workshop on Computational Linguistics for Uralic Languages. pp. 1–7 (2017)
  * [42] Rueter, J., Nivre, J., Zeman, D., Kabaeva, N., Levina, M.: Ud_moksha-jr 2.7 (Nov 2020)
  * [43] Rueter, J., Partanen, N., Ponomareva, L.: On the questions in developing computational infrastructure for Komi-permyak. In: Proceedings of the Sixth International Workshop on Computational Linguistics of Uralic Languages. pp. 15–25. Association for Computational Linguistics, Wien, Austria (10–11 Jan 2020), https://www.aclweb.org/anthology/2020.iwclul-1.3
  * [44] Rueter, J., Tyers, F.: Towards an open-source universal-dependency treebank for Erzya. In: Proceedings of the Fourth International Workshop on Computational Linguistics of Uralic Languages. pp. 106–118. Association for Computational Linguistics, Helsinki, Finland (Jan 2018). https://doi.org/10.18653/v1/W18-0210, https://www.aclweb.org/anthology/W18-0210
  * [45] Silfverberg, M., Rueter, J.: Can morphological analyzers improve the quality of optical character recognition? In: Septentrio Conference Series. pp. 45–56. No. 2 (2015)
  * [46] Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pp. 1631–1642. Association for Computational Linguistics, Seattle, Washington, USA (Oct 2013), https://www.aclweb.org/anthology/D13-1170
  * [47] Zeman, D., Nivre, J., Abrams, M., Ackermann, E., Aepli, N., Aghaei, H., Agić, Ž., Ahmadi, A., Ahrenberg, et al.: Universal dependencies 2.7 (2020), http://hdl.handle.net/11234/1-3424, LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University
  * [48] Zhou, S., Rijhwani, S., Wieting, J., Carbonell, J., Neubig, G.: Improving candidate generation for low-resource cross-lingual entity linking. Transactions of the Association for Computational Linguistics 8, 109–124 (2020), https://www.aclweb.org/anthology/2020.tacl-1.8


# Light meson nuclear modification factor in p-Pb collisions over an
unprecedented $\bm{p_{\mathrm{T}}}$ range with ALICE

, on behalf of the ALICE collaboration
Oak Ridge National Laboratory & IKF Frankfurt
E-mail

###### Abstract:

Light neutral meson differential invariant cross section and nuclear
modification factor measurements have been carried out with the ALICE detector
at the CERN LHC in pp collisions at $\sqrt{s}=8$ TeV and p–Pb collisions at
$\sqrt{s_{\mbox{\tiny NN}}}=8.16$ TeV. The analysis combines results from
several partially independent reconstruction techniques where the $\pi^{0}$
and $\eta$ meson decay photons were detected with the electromagnetic
calorimeter, EMCal, the photon spectrometer, PHOS, or via reconstruction of
$e^{+}e^{-}$ pairs from conversions in the ALICE detector material using the
central tracking system. The neutral pion measurement reaching a
$p_{\mbox{\tiny T}}$ of 200 GeV/$c$ poses as the highest measured identified
particle spectrum to date while the $\eta$ meson is measured to an
unprecedented $p_{\mbox{\tiny T}}$ of 50 GeV/$c$. The spectra are found to be
generally overestimated by NLO pQCD calculations. The nuclear modification
factors of both mesons exhibit a suppression for $p_{\mbox{\tiny T}}<10$
GeV/$c$ which is stronger compared to previous measurements at
$\sqrt{s_{\mbox{\tiny NN}}}=5.02$ TeV and consistent with CGC and cold nuclear
matter energy loss calculations. For $p_{\mbox{\tiny T}}>10$ GeV/$c$,
$R_{\mathrm{pPb}}$ is consistent with unity and theory predictions.

##  1 Introduction

Light neutral meson measurements in small systems like proton-proton and
proton-nucleus collisions at LHC energies over a large transverse momentum
range provide important insights into particle production mechanisms and their
modifications at small parton fractional momentum $x$ and high parton
densities. The nuclear modification factor $R_{\mathrm{pPb}}$ presents these
modifications by deviating from unity and shows distinct features depending on
$p_{\mbox{\tiny T}}$ as seen in previous measurements at RHIC and LHC
energies. The low $p_{\mbox{\tiny T}}$ region has been covered by several
measurements and is well understood for various particle species where nuclear
shadowing and saturation effects are dominant and can be described using
nuclear parton distribution functions (nPDFs) [1] and the Color-Glass-
Condensate (CGC) framework [2], respectively. The high transverse momentum
region of identified particle measurements is so far very little explored and
the effects of energy loss in cold nuclear matter or anti-shadowing are not
yet fully constrained. New measurements covering large $p_{\mbox{\tiny T}}$
ranges can therefore provide additional constraints for theory calculations
and allow one to disentangle initial and final state effects in the
modification of particle production.

##  2 Detector description and datasets

The central barrel of ALICE [3] provides multiple detector systems for
particle tracking, identification and reconstruction. The presented neutral
meson measurements require the reconstruction of decay photons for which ALICE
provides two calorimeter-based methods using the Electromagnetic Calorimeter
(EMCal) and the Photon Spectrometer (PHOS) as well as the reconstruction of
photons from the e+e- pairs produced by conversions in the detector material
using the Inner Tracking System (ITS) and the Time Projection Chamber (TPC).
The data used in the presented measurements is from pp ($\sqrt{s}=8$ TeV) and
p–Pb ($\sqrt{s_{\mbox{\tiny NN}}}=8.16$ TeV) collisions recorded in 2012 and
2016, respectively. Both systems have slightly different detector
configurations due to upgrades installed during the LHC long shutdown 1 which
are mentioned in the following. The EMCal is a lead-scintillator sampling
calorimeter that covered $\Delta\varphi=100^{\circ}$ for $|\eta|<0.7$ in 2012
and was extended by the DCal in 2015 adding an additional
$\Delta\varphi=68^{\circ}$ for $0.22<|\eta|<0.7$ opposite in azimuth. Together
they consist of 18240 cells with $\Delta\eta\times\Delta\varphi=0.0143\times
0.0143$ rad2 in size which provide an energy resolution of
$\sigma_{E}/E=4.8\%/E\oplus 11.3\%/\sqrt{E}\oplus 1.7\%$ with $E$ in units of
GeV. The PHOS is a lead tungstate crystal calorimeter that covered in its 2012
configuration $\Delta\varphi=60^{\circ}$ for $|\eta|<0.13$ and is made of
cells with $22\times 22$ mm2 which are close to the Molière radius of 2 cm
that provide an excellent resolution of $\sigma_{E}/E=1.8\%/E\oplus
3.3\%/\sqrt{E}\oplus 1.1\%$. The ITS is a six layered cylindrical silicon
detector system with two layers each of Silicon Pixel (SPD), Silicon Drift
(SDD) and Silicon Strip (SSD) detectors used for the reconstruction of primary
and secondary vertices. The TPC is a large cylindrical drift chamber that
covers $|\eta|<0.9$ and provides together with the ITS particle tracking as
well as particle identification via energy loss signals d$E$/d$x$ down to
$p_{\mbox{\tiny T}}\approx 100$ and 50 MeV/$c$ for primary and secondary
tracks, respectively. The V0 detectors are two scintillator arrays covering
the full azimuth and the forward pseudorapidities $2.8<\eta<5.1$ and
$-3.7<\eta<-1.7$. They are used to define the minimum bias trigger which
requires a coincidence signal in both arrays and for determining the event
multiplicity. Further event triggers are provided by both calorimeters based
on large energy deposits in small arrays of cells.

##  3 Photon and meson reconstruction in ALICE

The decay photons of the $\pi^{0}$ and $\eta$ mesons are reconstructed with
three reconstruction techniques. The photon conversion method (PCM) uses the
ITS and TPC to reconstruct photons from e+e- pairs produced in photon
conversions that occur with an approximate 8.9% probability in the inner
detector material of ALICE. The identification criteria are similar to those
given in Refs. [4, 5] with additional improvements to increase the
reconstruction efficiency at high $p_{\mbox{\tiny T}}$ by loosening the
constraints on the variable $q_{\mathrm{T}}$ which is the projection of the
electron momentum to the photon candidate momentum. The photon reconstruction
with PHOS and EMCal is based on the clusterization of energy deposits in the
calorimeter cells from electromagnetic showers induced by photons or
electrons. The selection criteria for the calorimetric analyses are adapted
from Refs. [4, 5] for the EMCal while the PHOS has only been used in the pp
reference measurement. For EMCal additional improvements were made regarding
the nonlinearity of the calorimeter response at high energies. The
corresponding studies resulted in two nonlinearity corrections which account
for a hardware nonlinearity of up to 14% at cluster energies of 200 GeV and a
residual energy and position correction which is applied in simulations to
match the $\pi^{0}$ invariant mass peak positions in data resulting in a per-
mille level of agreement.

The neutral mesons are reconstructed using an invariant mass technique where
photons reconstructed with the same technique are combined, called PCM, EMC or
PHOS, or following a hybrid approach where a PCM photon is paired with an
EMCal photon called PCM-EMC. The latter profits from the high resolution of
the conversion photon and the high efficiency of the EMC method and is able to
cover a large transverse momentum range while the pure PCM method is strongest
at low $p_{\mbox{\tiny T}}$ and the pure EMC method suffers, despite its high
efficiency, from cluster merging starting from 16 GeV/$c$ for the $\pi^{0}$.
Combinatorial background in this method is removed using an event mixing
technique and raw yields are obtained by integration of wide ranges around the
peak position which is obtained from a combined gaussian and one-sided
exponential fit on the invariant mass distributions.

In addition to the techniques based on the invariant mass, a particle
identification method based on merged photon clusters from $\pi^{0}$ decays is
employed, called mEMC [7]. This method exploits that the opening angle of high
momentum ($p_{\mbox{\tiny T}}>16$ GeV/$c$) $\pi^{0}$ mesons are too small to
induce two separate showers in the EMCal. Instead the showers overlap and thus
produce a single shower of “elliptical” shape which can be distinguished from
single photon clusters via the shower shape parameter
$\sigma_{\mathrm{long}}^{2}$ which can be interpreted as the long axis of the
shower ellipse. Photon clusters are located at small values of
$\sigma_{\mathrm{long}}^{2}$ thus the application of a selection with
$\sigma_{\mathrm{long}}^{2}>0.2$ allows us to reject single photon background
resulting in a purity between 81–87% depending on $p_{\mbox{\tiny T}}$. As
this method relies on a good description of $\sigma_{\mathrm{long}}^{2}$ in
simulation compared to data, an additional detector effect, called cross-talk,
had to be emulated within the same readout card as described in [6].

Figure 1: Top: Differential invariant cross section spectra of $\pi^{0}$ and
$\eta$ mesons in pp collisions at $\sqrt{s}=8$ TeV and p–Pb collisions at
$\sqrt{s_{\mbox{\tiny NN}}}=8.16$ TeV together with two-component model fits
and Pythia 8 calculations with different nPDFs [1] as well as NLO calculations
using CT10, MSTW08 and CTEQ6M5 PDFs with DSS14 and AESSS fragmentation
functions. Lower panels: Ratio of the p–Pb $\sqrt{s_{\mbox{\tiny NN}}}=8.16$
TeV $\pi^{0}$ and $\eta$ meson spectra and their corresponding theory
calculations to their respective two-component model fit. Statistical
uncertainties are given by the vertical lines while systematic uncertainties
are shown as boxes. Bottom right: $\eta/\pi^{0}$ ratio in both collision
systems together with NLO calculations.

##  4 Spectra and nuclear modification measurements

The $\pi^{0}$ and $\eta$ meson differential invariant cross sections have been
measured in pp collisions at $\sqrt{s}=8$ TeV [5] and p–Pb collisions at
$\sqrt{s_{\mbox{\tiny NN}}}=8.16$ TeV with several partially independent
reconstruction techniques. The results of the individual measurements were
combined using the Best Linear Unbiased Estimates (BLUE) method accounting for
correlations of uncertainties. The p–Pb measurement is therefore a combination
of the PCM, EMC, PCM-EMC and mEMC methods while for the pp measurement also
PHOS was used to reconstruct the $\pi^{0}$. The spectra together with two-
component model (TCM) fits and NLO [1] as well as Pythia 8 calculations are
shown in Figure 1. In addition, the ratios of the spectra and theory
predictions to the TCM fits are given which show an overestimation of the
spectra by NLO calculations especially for the $\eta$ meson. The
$\eta/\pi^{0}$ ratio is presented in the same figure (bottom right) and found
to be consistent between both collision systems with a constant value of about
$0.49$ for $p_{\mbox{\tiny T}}>4$ GeV/$c$ in the p–Pb measurement.

The nuclear modification factors for both mesons were calculated using the
measured spectra in pp and p–Pb with
$R_{\mathrm{pPb}}=\sigma_{\mathrm{pPb}}^{\pi^{0},\eta}/(208\cdot\sigma_{\mathrm{pp}}^{\pi^{0},\eta})$
where the reference measurement was scaled to the p–Pb collision energy as
well as corrected for the rapidity shift using Pythia8 Monash2013. The
resulting $R_{\mathrm{pPb}}$ are shown in Fig. 2 (left) for both mesons
together with NLO calculations using different nPDFs [1]. In addition,
calculations from the CGC framework [2] and cold nuclear matter energy loss
(FCEL) [8] are able to describe the $p_{\mbox{\tiny T}}<10$ GeV/$c$ region.
Furthermore, a comparison to a previous measurement in p–Pb
$\sqrt{s_{\mbox{\tiny NN}}}=5.02$ TeV, given in Fig. 2 (right) shows a $7\%$
stronger suppression for $p_{\mbox{\tiny T}}<6$ GeV/$c$ in the new data which
is supported by the CGC calculations hinting at gluon saturation effects. The
neutral meson $R_{\mathrm{pPb}}$ are found to be consistent with unity for
$p_{\mbox{\tiny T}}>10$ GeV/$c$ while the moderate enhancement of the CMS
charged hadron measurement which was interpreted as an anti-shadowing effect
shown in the same figure is not observed in the presented measurement.

Figure 2: Left: Nuclear modification factor of $\pi^{0}$ and $\eta$ mesons in
p–Pb collisions at $\sqrt{s_{\mbox{\tiny NN}}}=8.16$ TeV together with two NLO
calculations as well as calculations from the Color-Glass-Condensate (CGC)
framework [2] and cold nuclear matter energy loss calculations (FCEL) [8].
Right: Nuclear modification factors of neutral pions in p–Pb collisions at
$\sqrt{s_{\mbox{\tiny NN}}}=5.02$ and 8.16 TeV and central Pb–Pb collisions at
$\sqrt{s_{\mbox{\tiny NN}}}=5.02$ TeV together with charged hadron
measurements from ALICE and CMS in p–Pb collisions at $\sqrt{s_{\mbox{\tiny
NN}}}=5.02$ TeV. Statistical uncertainties are given by the vertical lines
while systematic uncertainties are shown as boxes.

##  5 Conclusions

The differential invariant cross sections and nuclear modification factors of
$\pi^{0}$ and $\eta$ mesons have been measured in p–Pb collisions at
$\sqrt{s_{\mbox{\tiny NN}}}=8.16$ TeV over unprecedented transverse momentum
ranges of $0.4<p_{\mbox{\tiny T}}<200$ GeV/$c$ and $1<p_{\mbox{\tiny T}}<50$
GeV/$c$, respectively. The reference $\pi^{0}$ measurement has been improved
and extended up to the same $p_{\mbox{\tiny T}}$ using the merged cluster
analysis in order to calculate $R_{\mathrm{pPb}}$. The nuclear modification
factors were found to be consistent with CGC and cold nuclear matter energy
loss calculations at low $p_{\mbox{\tiny T}}$ with a stronger suppression
compared to previous measurements at a lower center-of-mass energy. For high
transverse momenta of $p_{\mbox{\tiny T}}>10$ GeV/$c$, $R_{\mathrm{pPb}}$ is
found to be consistent with unity and theory predictions. The moderate
enhancement observed of the CMS charged hadron measurement is not observed.

## References

  * [1] Kovarik et al., Phys. Rev. D 93, 085037 (2016)
  * [2] Lappi and Mäntysaari, Phys. Rev. D 88, 114020 (2013)
  * [3] Abelev et al., ALICE Collaboration, Int. J. Mod. Phys. A29 (2014) 1430044
  * [4] Acharya et al., ALICE Collaboration, EPJC 78, 8 (2018) 624
  * [5] Acharya et al., ALICE Collaboration, EPJC 78, 3 (2018) 263
  * [6] Acharya et al., ALICE Collaboration, EPJC 79, 11 (2019) 896
  * [7] Acharya et al., ALICE Collaboration, EPJC 77, 9 (2017) 586
  * [8] Arleo et al., arXiv:2003.06337 [hep-ph]


11institutetext: Alibaba Group, Hangzhou, China
11email: {chenben.cb,cb242829,dehong.gdh,qijin.cqj,
chengfu.hcf,xiaonan.mengxn, afei, yngzhou}
@alibaba-inc.com

# Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News
Detection

Ben Chen  Bin Chen contribute equally with first author. Dehong Gao  Qijin
Chen  Chengfu Huo
Xiaonan Meng  Weijun Ren  Yang Zhou

###### Abstract

With the pandemic of COVID-19, relevant fake news is spreading all over the
sky throughout the social media. Believing in them without discrimination can
cause great trouble to people’s life. However, universal language models may
perform weakly in these fake news detection for lack of large-scale annotated
data and sufficient semantic understanding of domain-specific knowledge. While
the model trained on corresponding corpora is also mediocre for insufficient
learning. In this paper, we propose a novel transformer-based language model
fine-tuning approach for these fake news detection. First, the token
vocabulary of individual model is expanded for the actual semantics of
professional phrases. Second, we adapt the heated-up softmax loss to
distinguish the hard-mining samples, which are common for fake news because of
the disambiguation of short text. Then, we involve adversarial training to
improve the model’s robustness. Last, the predicted features extracted by
universal language model RoBERTa and domain-specific model CT-BERT are fused
by one multiple layer perception to integrate fine-grained and high-level
specific representations. Quantitative experimental results evaluated on
existing COVID-19 fake news dataset show its superior performances compared to
the state-of-the-art methods among various evaluation metrics. Furthermore,
the best weighted average F1 score achieves 99.02$\%$.

###### Keywords:

COVID-19 Fake news Adversarial training Knowledge Fusion.

##  1 Introduction

The development of social media, such as Twitter and MicroBlog, has greatly
facilitated people’s lives. We can get real-time news from almost anywhere in
the world. However, fabrications, satires, and hoaxes mixed in real reports
often mislead people’s judgments, especially during the pandemic. For example,
”CDC Recommends Mothers Stop Breastfeeding To Boost Vaccine
Efficacy”111https://www.snopes.com/fact-check/breast-practices/ and ”Consuming
alcohol beverages or vodka will reduce risk of COVID-19
infection”222https://www.usatoday.com/story/news/factcheck/2020/03/20/ are two
common rumors during the epidemic and caused panic among the masses.
Therefore, fake news detection is necessary and we hope to design an effective
detector which could quickly distinguish whether the news is fake or not
according to its title or summary. It is usually formulated as the sequence
classification problem in general.

Text classification is a fundamental task in natural language processing
(NLP), and transformer-based language models have achieved excellent
performance in general domains thanks to large corresponding corpora and
fined-designed pre-training skills (MLM/NSP/SOP) [13, 15, 14]. However, they
usually perform weak for specific domain. One main reason is the professional
phrases are rare in general corpora and existing tokenizers (e.g. byte-pair-
encoding, wordpiece, and
sentencepiece333https://github.com/huggingface/tokenizers) would split them
into many sub-tokens, and this operation hampers their actual semantics. Even
if data of specific domain is collected and used for down-stream fine-tuning,
the limited token vocabulary also fails to get the full meaning. Recently,
[12, 18, 16] have devoted to collected amounts of COVID-19 data. Especially
for [16], it also further trains one transformer-based model named CT-BERT
with part of these annotated data, making a 10-30$\%$ marginal improvement
compared to its base model on classification, question-answering and chatbots
tasks related to COVID-19. But for the specific fake news detection, its
insufficient learning of limited corpus contents and incomplete hard samples
mining make it hard to achieves one impressive result. Furthermore, excessive
further-training weakens the model’s understanding of common sense, resulting
in some incomprehensible mistakes.

In these paper, we try to optimize the performance of transformed-based
language models for COVID-19 fake news detection. For individual model,
firstly the token vocabulary is expanded with most frequent professional
phrases for getting the actual semantics without no split. Second, we adapt
the heated-up softmax loss to distinguish the hard-mining samples, which are
common for fake news because of the disambiguation of short text. Then,
adversarial training [6] is involved to improve the model’s generalization and
robustness. Last, the predicted features extracted by universal language model
RoBERTa [15] and domain-specific model CT-BERT [16] are fused by one multiple
layer perception to integrate fine-grained universal and high-level specific
representations. Quantitative experimental results evaluated on existing
COVID-19 fake news dataset [12] show these methods superior performances
compared to the state-of-the-art methods among various evaluation metrics.
Furthermore, the best weighted average F1 score achieves 99.02$\%$.

##  2  Related Work

###  2.1 Text classification task with adversarial training methods

Adversarial training is firstly designed to increase the model robustness
through adding small perturbations to training data, but it also increases the
generalization ultimately [6]. In the field of computer vision, mainstream
gradient-based attack [1], optimization-based attack [2] and generation-based
attack [3] have achieved impressive results. In recent years, more and more
adversarial training tips [4, 5, 6] have been proposed for natural language
processing tasks. Different from computer vision, where the image pixel is
continuous in the fixed space, so it is suitable to add noise perturbation
based on gradient method. However, in natural language processing tasks, the
input is usually a discrete text sequence, so it is impossible to realize the
antagonistic training by adding disturbance to the one-hot layer of the input
text. From the Goodfellow’s work [4], they realized the adversarial training
by adding embedding disturbance into the embedding layer. Usually in the CV
task, according to empirical conclusions, the adversarial training tends to
make the performance of the model on the normal samples worse, while in the
NLP task, the work of [4] shows that the generalization ability of the model
is stronger. In this paper, we use the method of adversarial training to
improve the generalization ability of the model in the task of Fake New
detection.

###  2.2 Model fusion approaches for text classification

Traditional machine learning models have proved that ensemble learning play an
important roles in improving the model effect, such as Bagging and Boosting.
The main reason lies in the complementary feature among models helps the model
to make correct judgment on the results. In recent years, a series of model
fusion methods have also appeared in the field of deep learning. The methods
for model fusion mainly include feature-based methods[7] and score-based
methods[8]. Feature level fusion method is suitable for models of different
categories, such as CNN and BiGRU, and can extract word-level features and
syntax-level features simultaneously[7]. While the fusion methods based on the
score level are more applicable to similar structure for models, which obtain
the final result by voting. This paper combines CT-BERT and RoBERTa by using
the fusion method at the score level, while ensuring the universality of the
model and its professionalism simultaneously in the task of Fake News
Detection.

###  2.3 Fake news detection

Fake news is intentionally written to mislead readers to believe false
information, which makes it difficult and nontrivial to detect based on news
content. Most existing methods identify false news by combining richer data,
which provide a data repository that include not only news contents and social
contents, but also dynamic information[18]. However, such methods lack
universality and cannot play a role in new fields. Hence, semi-supervised and
unsupervised methods are proposed[9], which try to make a trade-off between
the amount of training data and the final training accuracy. Based on the
excellent representational ability of deep pre-trained model (E.g, BERT,
ALBERT), our method tries to get a well result by utilizing a small amount of
data from domain special fields.

##  3 Methodology

###  3.1 Problem Definition

As described above, in this paper we convert COVID-19 fake news detection as
one typical single sentence classification, which means the proposed detector
can judge whether one news sentence is true or false according to its semantic
meaning. It can be expressed formally as: giving a news sentence
$x=t_{1},t_{2},t_{3},t_{4},t_{5},....$, the detector should predict one
correct label $y\subseteq{\\{0,1\\}}$. And so the corresponding optimization
goal is to learn the $\theta$ and maximize the loss function $L(y|x,\theta)$.

###  3.2  Our proposed network

As shown in Fig.1 is the structure of our proposed network. It is derived from
the most famous language model $-$ BERT, and we involve some practical methods
to enhance the ultimate performance from various perspectives. Below we will
introduce each module in detail.

####  3.2.1 Training with additional tokens

For specific domain text, there are many professional phrases and existing
tokenizers will split them into many sub-tokens, resulting in a
misunderstanding of their actual meanings. In this paper, we count 6 most
frequent tokens in train and validation data which will be split with original
method and add them in the existing token vocabulary of CT-BERT. There are:

$\bullet$ covid$-$19, covid19, coronavirus, pandemic, indiafightscorona,
lockdown.

Subsequent ablation experiments will prove the effectiveness of this method.

Figure 1: The framework of our proposed method. (a) is the overall framework
of the fused models and (b) is three improved modules added to each model. The
bottom part shows the result of adding new tokens to the existing vocabulary.
The middle red boxes represent the gradient perturbations added to the word
embedding. And the top blue histograms exhibit how the heated-up softmax
affects the distribution.

####  3.2.2 Optimization with heated-up softmax loss function

For binary classification task, we utilize the cross entropy loss as our loss
function. In order to remind model to pay more attention to the hard mining
examples, we introduce the heated-up softmax to replace the origin activation
function[10]. Initially, the heated-up softmax is expressed as follow:

$p(m|x)=\frac{exp(z_{m}/T)}{\sum_{j=1}^{M}exp(z_{j}/T)}=\frac{exp(\alpha
z_{m})}{\sum_{j=1}^{M}exp(\alpha z_{m})}$ (1)

where $\alpha$ denotes the temperature parameter. As can be seen from Eq.1,
the larger the parameter $\alpha$ is, the greater the gradient value is for
the hard mining samples, and the model pays more attention to the hard mining
samples. The smaller the $\alpha$ value, the smaller the attention to hard
mining sample, boundary sample and easy sample. Therefore, at the beginning of
the training, we first set a large $\alpha$ and let the model focus on
learning the difficult sample, and then reduce the $\alpha$. The model began
to pay attention to the boundary sample, and when the hard mining sample and
boundary sample were almost completed, then further reduced $\alpha$ to fine
tuning. As depicted in Fig.1(b), the factor $\alpha$ affect the distribution
of the last output layer, which is efficient for classify the hard example to
adjust the $\alpha$ as training going.

####  3.2.3 Gradient-based adversarial training

Adversarial training [11] is a novel regularization method for classifiers to
improve model robustness for small, approximately worst case perturbations.
The loss function we used with adversarial training unit can be formulated as:

$-log\;p(y|x+r_{adv};\theta)\;\;\;where\;\;r_{adv}=argmin_{r,||r||<=\epsilon}\;log\;p(x|x+r;\hat{\theta})$
(2)

where $r_{adv}$ denotes the perturbations , $\hat{\theta}$ denotes the
parameters of current network, and $\theta$ denotes the parameters after one-
step gradient optimization of the network. We can find from the Eq.2 that our
core objective is to add a small perturbation $r$ which could disable the
current classifier, then optimize the classifier, and optimize the network to
maximize the model’s ability to correctly classify samples. However, we cannot
calculate this value exactly in general, because exact minimization with
respect to $r$ is intractable for many interesting models such as neural
networks. Goodfellow et al. [2] proposed to approximate this value by
linearizing $log\;p(y|x;\hat{\theta})$ around $x$. With a linear approximation
and a L2 norm constraint in Eq.(2), the final adversarial perturbation is

$r_{adv}=-\epsilon\frac{g}{||g||_{2}}\;\;where\;\;g=\nabla_{x}logp(y|x;\hat{\theta})$
(3)

This perturbation can be easily computed using backpropagation in neural
networks.

In Fig.1(b), the red box represents the perturbation calculated by Eq.3.
perturbation is added to the embedding layer, and the robustness of the model
is enhanced by adversarial training.

##  4 Experimental Results

In this section, we will evaluate performance of our method in Fake News
Detection task. We compared our approach to several pre-trained models that
perform well on Glue tasks444https://gluebenchmark.com/, including BERT,
ALBERT, and RoBERTa, and each model included the basic version (X-Base) and
the Large version (X-Large).

In order to reflect the importance of each module in our model, we also
perform the ablation studies on our experiments. Specifically, we divide our
approach into 3 different methods and test the performance of each module:
COVID-Twitter-BERT (benchmark model, refer to as CT-BERT model below) [16],
CT-BERT-FGM(including adversarial training module with fast gradient
method),CT-BERT-HL (including heated-up softmax module),CT-BERT-New-Tokens
(including new tokens training ), Ro-CT-BERT (ours).

###  4.1 Experimental setting

The data we used for training and evaluation is the online-collected COVID-19
fake news dataset [12]. The sources of them are various social-media platforms
such as Twitter, Facebook, Instagram, etc. It contains 6420 / 2140 / 2140 raw
news sentence for training / validation / test. The real news sentence is as
follow:

$Wearing\;mask\;can\;protect\;you\;from\;the\;virus.$

While the fake one is shown as follow:

$If\;you\;take\;Crocin\;thrice\;a\;day\;you\;are\;safe.$

As a side note, for the data preprocessing, we follow the baselines in [12] to
remove all links, non alphanumeric characters (e.g. unicode emotions) and
English stop words, which all would bring great interference to the effective
detection. In order to train the more distinguishable models, After each
evaluation we will reserve the mis-classified samples in training and
validation set and replace some (1$\sim$2) words with their synonyms or remove
these words directly. The extended data will be added to the training set for
the next round of training.

To evaluate the performance of different methods, three popular metrics are
adopted, namely weighted Precision, weighted Recall and weighted F1. The
definitions are as follows:

$Precision_{weighted}=\frac{\sum_{i=1}^{n}Precision_{i}*w_{i}}{n}$ (4)

$Recall_{weighted}=\frac{\sum_{i=1}^{n}Recall_{i}*w_{i}}{n}$ (5)

$F1_{weighted}=\frac{2*Precision_{weighted}*Recall_{weighted}}{Precision_{weighted}+Recall_{weighted}}$
(6)

where $n$ represents the number of classes, $w_{i}$ represents the radio of
true instances for each label.

Our implementation is based on the online available natural language library
Transformers555https://github.com/huggingface/transformers. We adopt the
initial learning rate of 2e-5 with warm-up rate of 0.1. The batch size is
selected as 64 for training and 128 for validation and test. The temperature
parameter $\alpha$ is set as 4 in first 10 epochs, as 1 in middle 10 epochs
and as 0.5 for last 10 epochs. Each sequence length is limited to 128 tokens.
All experiments are performed using PyTorch on a Telsa V100 GPU with the
optimizer selected as Adam.

###  4.2 Performance in Fake News Detection in English

In this paper, we investigate the most cutting-edge models to tackle fake New
Detection tasks, including the highly versatile BERT [13], ALBERT [14], and
RoBERTa [15] and their large versions; In addition, we investigate COVID-
Twitter-BERT (CT-BERT), which is trained on a large corpus of Twitter messages
on the topic of COVID-19. We utilize the pre-trained model files provided on
the official website to initialize corresponding parameters, and then fine
tuning on the COVID-19 dataset [12]. Each model used the best result as the
final experimental result. For the sake of description, we call our model as
Robust-COVID-Twitter-BERT (Ro-CT-BERT). The experimental results are shown in
Table 1.

Table 1: Experimental comparison results on fake news detection task. Method | Accuracy | Precision | Recall | F1
---|---|---|---|---
BERT-base | 0.978505 | 0.978574 | 0.978505 | 0.978497
BERT-large | 0.980374 | 0.980407 | 0.980374 | 0.980369
RoBERTa-base | 0.983645 | 0.983755 | 0.983644 | 0.983638
RoBERTa-large | 0.985981 | 0.986081 | 0.985981 | 0.985976
ALBERT-base | 0.973365 | 0.973419 | 0.973365 | 0.973356
ALBERT-large | 0.973832 | 0.973897 | 0.973832 | 0.973823
ALBERT-xlarge | 0.974299 | 0.974665 | 0.974299 | 0.974276
CT-BERT | 0.984112 | 0.984161 | 0.984112 | 0.984115
Ro-CT-BERT | 0.990187 | 0.990218 | 0.990187 | 0.990185

From the experimental results, We can see that Ro-CT-BERT can get superior
performance than state-of-the-art methods on the metrics weighted average
accuracy, precision, recall and F1 score. Universal language models BERT,
ALBERT get all metric value lower than 0.981, while RoBERTa is much better
than them because of fine-designed key hyperparameters and larger training
data size. Although the CT-BERT model has been trained in a large number of
Messages on the Topic of COVID-19, our model’s F1 score is still 0.006 points
higher than it’s F1 score. This promotion is very difficult because we need to
make the correct classification for hard mining samples. We attribute the
improvement of the model to the hard mining samples learning and effective
fusion of fine-grained and high-level representations. In the next section, we
will demonstrate the effect of each module on the model through ablation
experiments.

###  4.3 Ablation studies for Ro-CT-BERT

In order to verify the effectiveness of the improved modules involved in Ro-
CT-BERT, we also conduct several ablation experiments. We mainly compare the
influence of three modules on the model, which called adversarial training,
heated-up softmax loss function and addition of new Token. For fairly
comparison, we take CT-BERT as the benchmark model and add three modules for
subsequent experiments, respectively. These three models are successively
referred as attack-training-CT-BERT (CT-BERT-FGM), heated-up softmax loss CT-
BERT (CT-BERT-HL) and new-taken CT-BERT (CT-BERT-New-Tokens). The model with
all three modules is referred as three-modules-CT-BERT (CT-BERT-TRM). Results
of ablation experiments are shown in Table 2.

Table 2: Experimental comparison results on Fake News Detection task. Method | Accuracy | Precision | Recall | F1
---|---|---|---|---
CT-BERT | 0.984112 | 0.984161 | 0.984112 | 0.984115
CT-BERT-FGM | 0.986449 | 0.986451 | 0.986449 | 0.986448
CT-BERT-HL | 0.986916 | 0.986971 | 0.986916 | 0.986912
CT-BERT-New-Tokens | 0.984579 | 0.984623 | 0.984579 | 0.984575
CT-BERT-TRM | 0.987851 | 0.987888 | 0.987851 | 0.987848
Ro-CT-BERT | 0.990187 | 0.990218 | 0.990187 | 0.990185

It can be seen that compared with the TD-BERT model without any other tricks,
the classification effect of the other three models is improved to a certain
extent, especially the heated-up softmax loss, which increases the
generalization ability of the model and has a strong classification ability
for the hard mining samples. Furthermore, these three modules combined
produces one better result. Lastly, Ro-CT-BERT fuse the predicted features of
CT-BERT-TRM and RoBERTa-TRM (three-module RoBERTa) and get the highest score,
indicating that the integration of fine-grained and high-level specific
representations helps to understand the text semantics more comprehensively.

##  5 Conclusions

In this work, we propose a transformer-based Language Model Fine-tuning
approach for COVID-19 Fake News Detection[19]. The length of adopted news
sentences for this task is short, and lots of professional phrases are rare in
common corpora. These two distinct features make universal and specific
language models all fail to make a correct distinction whether news is fake or
real. To address these problems, we respectively introduce new tokens for the
specific model vocabulary for better understanding of professional phrases,
model adversarial training to improve the robustness, and heated-up softmax
loss function to distinguish the hard-mining sample. Lastly, we also fuse the
predicted features extracted by universal language model and domain-specific
model to integrate fine-grained and high-level specific representations. These
methods are verified to be useful for improving the transformer-based model’s
ability, and finally, our approach achieves super performance compare with
state-of-the-art methods on the COVID-19 fake news detection.

## References

  * [1] Carlini, Nicholas, and David Wagner. ”Towards evaluating the robustness of neural networks.” 2017 ieee symposium on security and privacy (sp). IEEE, 2017.
  * [2] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. ”Explaining and harnessing adversarial examples.” arXiv preprint arXiv:1412.6572 (2014).
  * [3] Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, Dawn Song: Generating Adversarial Examples with Adversarial Networks. CoRR abs/1801.02610 (2018) a service of Schloss Dagstuhl - Leibniz Center for Informatics homebrowsesearchabout
  * [4] Miyato, Takeru, Andrew M. Dai, and Ian Goodfellow. ”Adversarial training methods for semi-supervised text classification.” arXiv preprint arXiv:1605.07725 (2016).
  * [5] Wang, Wenqi, et al. ”Towards a Robust Deep Neural Network in Texts: A Survey.” arXiv preprint arXiv:1902.07285 (2019).
  * [6] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, Jingjing Liu: FreeLB: Enhanced Adversarial Training for Natural Language Understanding. ICLR 2020
  * [7] Xie, Jinbao, et al. ”Chinese text classification based on attention mechanism and feature-enhanced fusion neural network.” Computing 102.3 (2020): 683-700.
  * [8] Bhushan S N B, Danti A. Classification of text documents based on score level fusion approach[J]. Pattern Recognition Letters, 2017, 94: 118-126.
  * [9] Bhattacharjee, Sreyasee Das, Ashit Talukder, and Bala Venkatram Balantrapu. ”Active learning based news veracity detection with feature weighting and deep-shallow fusion.” 2017 IEEE International Conference on Big Data (Big Data). IEEE, 2017.
  * [10] Xu Zhang, Felix X. Yu, Svebor Karaman, Wei Zhang, Shih-Fu Chang: Heated-Up Softmax Embedding. CoRR abs/1809.04157 (2018)
  * [11] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. ”Explaining and harnessing adversarial examples.” arXiv preprint arXiv:1412.6572 (2014).
  * [12] Patwa, Parth, et al. ”Fighting an Infodemic: COVID-19 Fake News Dataset.” arXiv preprint arXiv:2011.03327 (2020).
  * [13] Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.
  * [14] Lan, Zhenzhong, et al. ”ALBERT: A lite BERT for self-supervised learning of language representations.” arXiv preprint arXiv:1909.11942 (2019).
  * [15] Liu, Yinhan, et al. ”RoBERTa: A robustly optimized BERT pretraining approach.” arXiv preprint arXiv:1907.11692 (2019).
  * [16] Müller, Martin, Marcel Salathé, and Per E. Kummervold. ”COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter.” arXiv preprint arXiv:2005.07503 (2020).
  * [17] Sun, Chi , et al. ”How to Fine-Tune BERT for Text Classification?.” China National Conference on Chinese Computational Linguistics Springer, Cham, 2019.
  * [18] Gautam Kishore Shahi, Durgesh Nandini: FakeCovid-A Multilingual Cross-domain Fact Check News Dataset for COVID-19. CoRR abs/2006.11343 (2020)
  * [19] Parth Patwa and Mohit Bhardwaj, et al. Overview of CONSTRAINT 2021 Shared Tasks: Detecting English COVID-19 Fake News and Hindi Hostile Posts. Proceedings of the First Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situation (CONSTRAINT-2021)