# Two-Stage Classifier for COVID-19 Misinformation Detection Using BERT: a
Study on Indonesian Tweets

Douglas Raevan Faisal  Rahmad Mahendra

###### Abstract

The COVID-19 pandemic has caused globally significant impacts since the
beginning of 2020. This brought a lot of confusion to society, especially due
to the spread of misinformation through social media. Although there were
already several studies related to the detection of misinformation in social
media data, most studies focused on the English dataset. Research on COVID-19
misinformation detection in Indonesia is still scarce. Therefore, through this
research, we collect and annotate datasets for Indonesian and build prediction
models for detecting COVID-19 misinformation by considering the tweet’s
relevance. The dataset construction is carried out by a team of annotators who
labeled the relevance and misinformation of the tweet data. In this study, we
propose the two stage classifier model using IndoBERT pre-trained language
model for the Tweet misinformation detection task. We also experiment with
several other baseline models for text classification. The experimental
results show that the combination of the BERT sequence classifier for
relevance prediction and Bi-LSTM for misinformation detection outperformed
other machine learning models with an accuracy of 87.02%. Overall, the BERT
utilization contributes to the higher performance of most prediction models.
We release a high-quality COVID-19 misinformation Tweet corpus in the
Indonesian language, indicated by the high inter-annotator agreement.

###### keywords:

COVID-19 , Misinformation , Tweet , Indonesian , Text Classification , BERT

††journal: Elsevier Journal

##  1 Introduction

The global effect of the COVID-19 pandemic had been one of the most
significant events starting early 2020s. This also applies even more severely
in Indonesia, with over 4 million confirmed cases nationally at the end of
2021 [1]. The pandemic situation that has been on for almost 2 years resulted
in more discoveries being made as it goes on. Those discoveries are ranging
from medical breakthroughs such as vaccines [2], to the mutations that
introduced new coronavirus variants [3].

While new information and findings get spread widely, other false information
or misinformation is also spread across mediums. In Indonesia, social media
has been one of the technologies that have the fastest adoption rate and also
became the easiest medium for misinformation to propagate quickly [4].
However, on the other hand, it is found that as much as 76% of their survey
respondents believed social media was a trusted information source [5]. The
pandemic also affected their findings in which the health and medical category
became the second most prominent issue after politics in the spread of
misinformation in Indonesia [5]. As the technology became more accessible,
there was a tendency for people to look for information through social media
as opposed to other traditional media [6].

Detecting misinformation in COVID-19 topics using machine learning approaches
has been examined by several research works, specifically in the English
social media text corpus. Using the social media post features without any
prior knowledge of the facts apart from the dataset itself, machine learning
models could achieve above 90% scores in accuracy [7, 8]. The use of pre-
trained language models, i.e., Bidirectional Encoder Representations from
Transformers (BERT) [9] improved the misinformation classification performance
when compared to predecessor methods, such as word embeddings and $n$-grams
models [10].

Following the COVID-19 pandemic, Panacea Lab worked on collecting COVID-19
social media posts on Twitter [11] and updated their dataset periodically
111http://www.panacealab.org/covid19/. As of December 2021, the Indonesian
language was the sixth most written language among 65 distinct languages in
their dataset. Other work inspected that Indonesian was the third most popular
language in their COVID-19 Twitter dataset from January to May 2020 [12].
Their findings indicated that the Indonesian language has a large proportion
of the public discourse in social media.

This research fills the gap in detecting misinformation in Indonesian social
media Twitter posts by using the BERT language model. There are two main
contributions of this research:

  1. 1.

Dataset building. We first collected Indonesian social media text data related
to the COVID-19 pandemic from Twitter. We annotated the sample of tweets for
the misinformation detection task. The data collection spanned more than one
year of Indonesian COVID-19-related tweets, and the annotation process follows
a comprehensive annotation guideline. The Indonesian COVID-19 Twitter
corpus222https://github.com/douglasraevan/covid19-indonesian-tweets and
misinformation detection
dataset333https://github.com/douglasraevan/covid19-indonesian-misinformation-
tweets are publicly available for research purpose.

  2. 2.

Model building. This research revolves around detecting misinformation with
the machine learning approach. The text mining experiments in this research
present insights into how well a machine learning model combined with a pre-
trained language model can learn and detect misinformation in social media.

##  2 Related Work

Misinformation is defined as a false or inaccurate information [13, 14]. In
the science and health context, misinformation is defined as information that
contradicts the consensus agreement of scientific communities on a phenomenon
[15]. Other related terms are hoax and disinformation. Disinformation is
intentionally made to be deceptive, while misinformation may or may not be
deceptive. Karlova and Fisher argued that misinformation and disinformation
should be viewed as a separate set [13], whereas Kumar and Geethakumari [14]
and Swire-Thompson and Lazer [15] considered disinformation as the subset of
misinformation. Due to the difficulty of the current computational model in
determining someone’s deceptive intention in text, most automatic
misinformation detection researches lean towards the latter definition and
ignored the deceptive factors of disinformation.

There are four mediums in which misinformation can propagate, which are direct
to online sources, search engines, user-generated content, and mobile apps
[15]. A study found that misinformation was spreading faster than the true
information by tracking around 126,000 online rumors on Twitter between 2006
and 2017 [16].

As a platform that played a big role in user-generated content, social media
is defined as the use of electronic and Internet tools to share and discuss
information and experiences with other people [17]. Among the variety of data
types shared in social media, the text is one of the most used data types and
therefore can be seen as a fundamental part of most platforms [18]. The large
amount of text data stored in social media became the main motive for the
research in social media analytics through text mining. Following the
traditional framework of text analytics, [18] defined the social media text
analytics framework consisted of three consecutive phases, which are text
preprocessing, text representation, and knowledge discovery.

###  2.1 Data Annotation

Machine learning relies heavily on data samples to be able to create
meaningful inferences. Annotating data for the misinformation classification
task is challenging. Full Fact and academic institution in the UK designed a
guide for crowdsourcing data annotation on verifiable factual claims [19].
Each claim is categorized into seven different classes, which are ”personal
experience”, ”quantity in the past or present”, ”correlation or causation”,
”current laws or rules of operation”, ”prediction”, ”other type of claim”, and
”not a claim”.

QCRI team proposed several improvements to the annotation process of Tweet
misinformation data [20]. They adjusted the guideline and asked seven
questions to annotators when annotating the Tweet data. The main idea is to
guide the annotators to determine whether a tweet contains any verifiable
factual claim, then proceed to verify whether the claim is true or not and
estimate the significance of the impact if it were to be seen by the majority
of social media users. One thing that differentiates the QCRI’s annotation
guideline from Full Fact‘s is the exclusion of ”personal experience” from
being labeled as misinformation. The argument is that personal experiences
mostly consist of claims about facts that could not be verified with any
publicly-available information.

###  2.2 Detecting Misinformation

Detecting misinformation on social media data has been done by several works
on English text corpus. Al-Rakhami and Al-Amri collected around 409,484 tweets
that revolved around the COVID-19 topic and constructed an ensemble-based
model to classify whether they contain any misinformation [7]. They only used
structured data as the features that became the input for the ensemble model.
Another work [21] performed a similar method in classifying fake news based on
the clickbait elements that are contained within the news titles and URLs.
They extracted shape features of the text (e.g. uppercase, lowercase, numeric)
and keywords matching as the features for the machine learning.

Other work explored the use of textual data as the main feature for the
machine learning algorithm. The Indian researchers found that SVM model using
TF-IDF representation yields satisfying accuracy [22] when being tested on the
annotated fake news corpus built from several established fact-checking
websites [23].

More modern approaches explored the use of word embeddings and pre-trained
language models. As current state-of-the-art language models, the Transformer
[24] and BERT [9] have been highly referenced and used by multiple NLP
downstream tasks. Several works have utilized BERT for misinformation
detection [25, 10]. Song constructed classification-aware neural topic model
which hybridizes a deep-learning-based classifier and topic modeling [10]. The
model was then trained to classify 10 categories of misinformation and then
compared to BERT as the existing state-of-the-art model.

COVIDLies was introduced as the dataset for stance detection task in which the
model is challenged to identify whether a tweet support, refute, or has no
stance towards retrieved misconceptions about COVID-19 [25]. The BERT model
was utilized with domain adaptation to improve the accuracy performance for
this domain specific task. Another work collected particular Bulgarian news
articles and then classified them into credible or fake news [26]. They
experimented with three types of features, i.e., linguistic feature (TF-IDF),
credibility features (orthographic), and semantic features (Word2Vec).

Two key points that were overlooked by the most previous works are the
imbalanced ratio nature between true information compared to misinformation
and lack of topic relevance judgement in misinformation dataset. Naturally,
the number of misinformation is less significant than the number of available
true information. It has implication that the misinformation detection should
be considered as an imbalanced classification problem. On the other hand,
misinformation detection should not be viewed only as a binary classification
task, but rather as two binary classification tasks. The first classification
task is to determine whether the data is relevant enough to be verified. If it
fulfills the criteria of being relevant, then the second task is to verify and
classify whether the claim is true to the fact or false.

###  2.3 Bidirectional Encoder Representations from Transformers

Bidirectional Encoder Representations from Transformers (BERT) is a language
model or text representation model that learn the context and semantic meaning
of documents bidirectionally [9]. This model utilizes the self-attention
layers architecture from the Vaswani Transformer model [24]. In addition, BERT
has bidirectional architecture for fine-tuning multiple tasks that require
context from two directions such as question answering. This model also
implements language representation with a pre-training approach for reuse in
several different language processing tasks. Similar to the Word2Vec model,
the output representation of this model is a numeric vector for each token in
the document.

The initial phase involves a pre-training process by performing two
unsupervised tasks, namely the Masked Language Model (MLM) and Next Sentence
Prediction (NSP). The next phase is the fine-tuning process to adjust to the
language processing task.

For Indonesian language, the IndoBERT model [27] is trained with a dataset of
Indonesian language texts, which includes social media texts, blogs, news
articles, and online sites. The IndoBERT model outperforms several baseline
models which includes fastText and mBERT in several NLP tasks in Indonesian
language.

##  3 Methodology

Figure 1: Overall View of The Research Process.

The design of this research process, as shown in Figure 1, followed the
framework for social media text mining by [18]. The process consists of data
collection, data annotation, preprocessing, feature extraction, and machine
learning. The main contribution of this research is twofold, i.e., Indonesian
COVID-19 misinformation dataset and machine learning model to detect
misinformation. In addition, we also release large scale COVID-19 Indonesian
Tweets collection.

We formulate the misinformation detection task as a text classification task.
Each tweet is seen as a document instance containing a sequence of words. For
each tweet, the input is a sequence of $N$ tokens $W=(w_{1},w_{2},...,w_{N})$
and the output is a single label prediction $y$ where $y\in\\{$irrelevant,
true, misinformation$\\}$.

###  3.1 Data Collection

####  3.1.1 Tweet Data Retrieval

As the primary dataset for this research, tweet data were collected directly
through Twitter API similar to previous works [11, 12, 28]. The data
collection was conducted from July 21, 2020 to February 7, 2021. Due to the
limitations of the Twitter API’s free tier, we did not manage to retrieve
tweets before July 21, 2020. To retrieve the Tweets collection, we defined the
keywords that are related to the COVID-19 pandemic. We first started with
several keywords as query seed and then we updated the keywords iteratively by
evaluating the top n-grams periodically during the collection process.

We only considered tweets that were identified as written in the Indonesian
language based on Twitter’s language identification algorithm. However,
Twitter language tags are not always accurate [29], especially for closely
related language pairs, e.g. Indonesian and Malay [30]. We found that several
tweets were actually written in the Malaysian language detected as Indonesian.
To exclude those tweets from our collection, we identified tweets containing
keywords that were specific to the Malaysian context, such as ”malaysia”,
”KKMPutrajaya” (the Twitter handle for the Malaysian Ministry of Health), and
”kes baharu” (meaning: new case(s)).

To complement the missing data before July 2020, we used the Indonesian
portion dataset published by Panacea Lab [11]. It covered COVID-19 tweets in
multiple languages from January 27, 2020. The dataset has metadata of tweet
IDs, timestamps, geolocation, and language, so a hydration process was
required to collect the respective tweets based on the given tweet IDs.

Furthermore, we filtered the Panacea Lab data using the query we used to
collect the data from Twitter API. We merged filtered Panacea Lab data with
crawled data and obtain a total of 13,274,237 tweets.

####  3.1.2 Misconception Retrieval

We followed COVIDLies [25] methodology to harness a set of refuted false
claims or misconceptions of COVID-19 in order to limit only related discussion
topics to be included in the misinformation experiment. We use the
misconceptions topics published by Kominfo, the Indonesian Ministry of
Communication and Information Technology, during the timeframe of our Tweet
collection (from January 27, 2020 to February 7,
2021)444https://www.kominfo.go.id/content/all/laporan_isu_hoaks.

In addition, we used the misconceptions from COVIDLies for the global context.
The data that they retrieved were originally published on a Wikipedia page on
COVID-19 misinformation. We translated the misconceptions into Indonesian
using the Google Translate API and then incorporated them into the Indonesian
local misconception topics. Other COVID-19 misconceptions from the medical
science perspective were also retrieved [31] and [32], e.g., ”5g”,
”hydroxychloroquine”, ”mosquito”, and ”garlic”.

We sampled some of the misconceptions and then manually defined the list of
keywords related to that misconception. The keywords are then used to filtered
the Tweets to be included in our misinformation dataset.

###  3.2 Data Annotation

The data annotation involve two steps, namely relevance annotation and
misinformation annotation. The first step was to annotate each tweet whether
it is relevant to be included in the misinformation classification or not.
Once the tweet had been annotated as relevant, then the tweet would be
included in the second step, which was misinformation annotation. The final
annotated data is also published on
Github555https://github.com/douglasraevan/covid19-indonesian-misinformation-
tweets.

####  3.2.1 Relevance Annotation

For this part, 4,500 tweets were randomly sampled from the tweets that matched
the misconception-based query. Two annotators were recruited to annotate the
relevance aspect of each tweet. Both annotated the same set of tweets. The
annotators were given the tweet hyperlink, the posting time and date, the full
text, and the URL contained in each tweet. In order to reduce any human bias,
the annotators should not consider any belief inferred from the Twitter
profile user when doing annotation.

The relevance annotation was done by asking the annotator three questions
towards the Tweet, i.e., whether it is written in Indonesian, whether it is
topically relevant, and whether it contains verifiable factual claim [20, 19].

##  4 Experimental Results

###  4.1 Annotation Results

The annotation process consisted of two steps; the first step was to annotate
the relevance of the tweet and the second step was to annotate whether the
claim within the tweet was misinformation or not. For the relevance
annotation, two annotators annotated 4,500 tweets and obtained a Kappa
agreement score of 0.8840. This means the annotators had a high level of
agreement in annotating the relevance of the tweet [33]. In the misinformation
annotation, two annotators verified the truth of claims for 2,441 Tweets with
relevant label. The Kappa score of the two annotators was 0.8467, which can be
interpreted as strong inter-annotator agreement.

The tweet count and the label distributions based on the annotation are
presented in Table 1. The results show that there is a notable imbalance
between claims that were true and misinformations in which the ”true” label
appeared in the annotations almost four times more than the ”misinformation”
label. There were also 237 tweets in which the majority of the annotators
within a group voted as ”not sure” and 22 tweets where the annotators thought
that the tweets needed expert annotations. Lastly, there were 3.24% out of all
the annotated tweets where the votes of the annotators could not reach a
consensus.

Figure 2: Annotated Tweets Wordcloud.

We found several keywords or phrases that indicate different topics discussed
for each of the labels as shown in the word clouds in Figure 2. Tweets that
were annotated as misinformations were mostly discussing the vaccinations
(keyword(s): ”vaksin”, ”vaksinasi”), which at the time was still filled with
rumors. Meanwhile, the tweets that were annotated as containing true
information mostly discussed the recovery aspects from the pandemic, such as
hospitals (rumah sakit or rsud), health workers (tenaga kesehatan), and COVID
patients (pasien covid). As for the tweets that were labeled as irrelevant,
they were mostly tweets on health protocols reminders, indicated by the
dominance of keywords such as health protocol (protokol kesehatan), face masks
(masker), and vitamin(s).

Table 1: Annotation Label Distribution Annotation Label | Tweet Count | Percentage
---|---|---
Irrelevant | 2,059 | 45.76%
True Information | 1,632 | 35.60%
Misinformation | 404 | 8.98%
Not Sure | 237 | 5.93%
No Consensus | 146 | 3.24%
Need Expert Judgement | 22 | 0.49%

We only include the Tweets that were labeled as irrelevant, true, and
misinformation in final dataset and used it for the experiments. We did not
perform further adjudication process for the remaining labels because the
label counts for the other labels were not significant when compared to the
main labels. We also further sampled several tweets which were already
filtered out during the misconception filtering and then labeled the new
samples as irrelevant. We aimed for the irrelevant label count to be around
twice as much compared to the true label count. The final dataset contains
3,127 irrelevant Tweets. The dataset was splitted into three sets; train,
validation, and test set with each label count shown in Table 2. We also
splitted them stratifiedly based on the label count, so each split has a near-
identical label distribution.

Table 2: Dataset Split | Irrelevant | True | Misinformation
---|---|---|---
Train (60%) | 1,876 | 979 | 242
Val (20%) | 625 | 327 | 81
Test (20%) | 626 | 326 | 81

###  4.2 Feature Selection Results

Using the constructed dataset, we then experimented with some of the feature
options combined with the classifiers. We tested the performance based on the
F1 scores on classifying the validation set. The models were run using
$k$-fold cross-validation and then prepared a pair of samples based on the
features that we plan to compare. We assumed the null hypothesis ($H_{0}$) to
denote the equal mean between the two sets and rejected the $H_{0}$ if the
$P$-value produced by the statistical significance test is below 0.05.

The results are shown on Table 3. The first comparison was to evaluate the
effect of implementing a balancing technique into the training dataset. Both
SMOTE and random undersampling were used in a chain as the balancing
technique. Based on the $t$-test results, we found that there was a
significant effect by balancing the label distribution. Hence, $H_{0}$ was
rejected. The mean of the balanced sample set was also found to be higher than
the mean of the non-balanced sample set. Therefore, we determined to implement
the balancing technique for the traditional features and classifiers.

The second comparison was between the Bag-of-Words (BoW) and TF-IDF. For the
BoW, we implemented the binary type as opposed to the token occurrence count.
We found that there was no significant difference in performance between BoW
and TF-IDF. However, we investigated the results further and it turns out that
the classifier models that were used in each feature type seemed to affect the
scores. So the tests were then divided based on the classifiers. One group was
specific to the Naive Bayes classifier, and the other group was all the other
traditional classifiers. Both groups’ results indicated that there were
significant differences in different features (BoW and TF-IDF). It was found
that the Naive Bayes classifier results were improved by using BoW, while the
other classifiers results were improved by using TF-IDF.

Table 3: Feature Selection Statistical Significance Test Results Sample Sets | $\bar{x}_{1}$ | $\bar{x}_{2}$ |  $t$-value |  $P$-value
---|---|---|---|---
Non-Balanced | Balanced | $22.29$ | $36.25$ | $8.8448$ | 0.0000*
BoW | TF-IDF |
\- All Classifiers | $35.56$ | $33.94$ | $1.2487$ | $0.2129$
\- Naive Bayes | $44.49$ | $25.70$ | $7.8314$ | 0.0000*
\- SVM, LR, DT, RF, XGB | $31.99$ | $37.23$ | $4.1978$ | 0.0000*
$\bar{x}_{i}$: The mean of F1 scores from sample set $i$
*: $P$-value $<0.05$

###  4.3 Single Classifier Results

We first experimented with only a single classifier as our baseline to perform
a 3-class classification task. The results were also divided into traditional
machine learning classifiers and deep learning classifiers. Based on the
results in Table 4, the BERT pre-trained by [27] achieved the highest accuracy
score, misinformation recall, and misinformation F1 score. The XGBoost
classifier with TF-IDF as its feature had the highest precision but with a
high recall tradeoff. The opposite occurred with the SVM-Word2Vec combination
where it had a high recall (the same as BERT) but with a low precision score.

Most of the deep learning classifiers resulted a higher score when compared to
the traditional classifiers. However, there were still several traditional
classifiers that outperformed the deep learning classifiers. Based on the F1
scores, the best model for the traditional machine learning was the XGBoost
model combined with the Word2Vec feature with an F1 score of 49.41%, which was
still lower than BERT, but higher than the other deep learning models.

Table 4: Single Classifier Results Classifier | Feature | Acc. | Target = Misinformation
---|---|---|---
Prec. | Rec. | F1
Traditional Machine Learning
NB | BoW | 76.38 | 39.77 | 43.21 | 41.42
SVM | TF-IDF | 79.28 | 49.21 | 38.27 | 43.06
Word2Vec | 74.44 | 39.20 | 60.49 | 47.57
IndoBERT | 75.12 | 35.65 | 50.62 | 41.84
LR | TF-IDF | 80.25 | 53.19 | 30.86 | 39.06
Word2Vec | 73.86 | 33.07 | 51.85 | 40.38
IndoBERT | 75.51 | 35.54 | 53.09 | 42.57
DT | TF-IDF | 71.54 | 30.23 | 32.10 | 31.14
Word2Vec | 59.63 | 12.23 | 20.99 | 15.45
IndoBERT | 64.67 | 19.84 | 30.86 | 24.15
RF | TF-IDF | 80.35 | 63.16 | 29.63 | 40.34
Word2Vec | 79.48 | 51.56 | 40.74 | 45.52
IndoBERT | 78.51 | 53.33 | 39.51 | 45.39
XGB | TF-IDF | 77.93 | 64.29 | 22.22 | 33.03
Word2Vec | 79.38 | 47.19 | 51.85 | 49.41
IndoBERT | 78.80 | 44.19 | 46.91 | 45.51
Deep Learning
BERT | IndoBERT | 85.00 | 55.68 | 60.49 | 57.99
Bi-LSTM | IndoBERT | 83.16 | 57.63 | 41.98 | 48.57
DNN | IndoBERT | 81.99 | 56.39 | 38.27 | 45.59
CNN | IndoBERT | 81.61 | 57.14 | 29.63 | 39.02
*Acc.: Accuracy; Prec.: Precision; Rec.: Recall; F1: F1-score

###  4.4 Two-Stage Classifier Results

Next, we experimented with a pipeline model consisting of two sequential
classifiers in which each was trained for different binary classification
tasks. The first task was tweet’s relevance classification and the second task
was the misinformation detection.

####  4.4.1 Relevance Classification Results

The experimental results for the relevance classification task is shown on
Table 5. Based on that result, the IndoBERT model had the highest performance
when compared to the other feature-classifier combinations. All of the
performance metrics for this task were also led by IndoBERT. The difference
between IndoBERT and the other models was relatively high.

Table 5: Relevance Classification Results Classifier | Feature | Acc. | Target = Relevance
---|---|---|---
Prec. | Rec. | F1
Traditional Machine Learning
NB | BoW | 80.83 | 74.13 | 78.87 | 76.43
SVM | TF-IDF | 83.16 | 79.35 | 77.40 | 78.36
Word2Vec | 80.45 | 73.46 | 78.87 | 76.07
IndoBERT | 79.48 | 78.93 | 65.36 | 71.51
LR | TF-IDF | 83.54 | 83.76 | 72.24 | 77.57
Word2Vec | 81.12 | 74.42 | 79.36 | 76.81
IndoBERT | 81.61 | 76.27 | 77.40 | 76.83
DT | TF-IDF | 78.99 | 74.11 | 71.74 | 72.91
Word2Vec | 71.93 | 63.57 | 67.32 | 65.39
IndoBERT | 70.76 | 63.43 | 60.93 | 62.16
RF | TF-IDF | 83.64 | 80.67 | 76.90 | 78.74
Word2Vec | 82.28 | 80.27 | 72.97 | 76.45
IndoBERT | 80.93 | 81.44 | 66.83 | 73.41
XGB | TF-IDF | 79.38 | 83.45 | 59.46 | 69.44
Word2Vec | 83.83 | 78.30 | 81.57 | 79.90
IndoBERT | 81.90 | 77.09 | 76.90 | 77.00
Deep Learning
BERT | IndoBERT | 90.03 | 87.62 | 86.98 | 87.30
Bi-LSTM | IndoBERT | 85.09 | 79.91 | 83.05 | 81.45
DNN | IndoBERT | 84.22 | 78.50 | 82.56 | 80.48
CNN | IndoBERT | 83.45 | 78.23 | 80.34 | 79.27
*Acc.: Accuracy; Prec.: Precision; Rec.: Recall; F1: F1-score

####  4.4.2 Misinformation Classification Results

For this task, only the tweets with true or misinformation labels were
included in the training, validation, and testing process. The results are
shown in Table 6. We found that IndoBERT did not perform as well as on the
previous tasks. Instead, the Bi-LSTM with IndoBERT gained the highest
accuracy, precision, and F1 scores. Other combinations that performed nearly
as well as the best one were XGBoost-IndoBERT, Logistic Regression-IndoBERT,
Random Forest-IndoBERT, and CNN-IndoBERT. The top five combinations were using
IndoBERT embeddings as their feature input.

Table 6: Misinformation Classification Results Classifier | Feature | Acc. | Target = Misinformation
---|---|---|---
Prec. | Rec. | F1
Traditional Machine Learning
NB | BoW | 86.98 | 66.67 | 69.14 | 67.88
SVM | TF-IDF | 88.21 | 71.43 | 67.90 | 69.62
Word2Vec | 84.28 | 58.25 | 74.07 | 65.22
IndoBERT | 84.52 | 59.00 | 72.84 | 65.19
LR | TF-IDF | 88.94 | 80.00 | 59.26 | 68.09
Word2Vec | 86.98 | 64.89 | 75.31 | 69.71
IndoBERT | 89.43 | 71.59 | 77.78 | 74.56
DT | TF-IDF | 80.84 | 51.65 | 58.02 | 54.65
Word2Vec | 79.85 | 49.53 | 65.43 | 56.38
IndoBERT | 77.64 | 46.03 | 71.60 | 56.04
RF | TF-IDF | 85.75 | 68.25 | 53.09 | 59.72
Word2Vec | 88.21 | 70.37 | 70.37 | 70.37
IndoBERT | 89.93 | 72.73 | 79.01 | 75.74
XGB | TF-IDF | 84.28 | 64.91 | 45.68 | 53.62
Word2Vec | 86.98 | 65.22 | 74.07 | 69.36
IndoBERT | 90.42 | 74.42 | 79.01 | 76.65
Deep Learning
BERT | IndoBERT | 88.48 | 74.29 | 64.20 | 68.87
Bi-LSTM | IndoBERT | 91.15 | 80.00 | 74.07 | 76.92
DNN | IndoBERT | 89.93 | 78.57 | 67.90 | 72.85
CNN | IndoBERT | 90.17 | 75.95 | 74.07 | 75.00
*Acc.: Accuracy; Prec.: Precision; Rec.: Recall; F1: F1-score
Table 7: Pipeline Results Classifier | Feature | Acc. | Target = Misinformation
---|---|---|---
Relev. | Misinfo. | Prec. | Rec. | F1
Single Classifier
BERT | IndoBERT | 85.00 | 55.68 | 60.49 | 57.99
XGB | Word2Vec | 79.38 | 47.19 | 51.85 | 49.41
Bi-LSTM | IndoBERT | 83.16 | 57.63 | 41.98 | 48.57
SVM | Word2Vec | 74.44 | 39.20 | 60.49 | 47.57
DNN | IndoBERT | 81.99 | 56.36 | 38.27 | 45.59
Two-Stage Classifier Pipeline
BERT | Bi-LSTM | IndoBERT | 87.02 | 59.75 | 60.49 | 60.12
XGB | IndoBERT | 85.77 | 55.43 | 62.96 | 58.96
LR | IndoBERT | 85.58 | 54.17 | 64.20 | 58.76
RF | IndoBERT | 85.77 | 54.84 | 62.96 | 58.62
CNN | IndoBERT | 86.74 | 54.35 | 61.73 | 57.80
*Relev.: Relevance; Misinfo.: Misinformation
**Acc.: Accuracy; Prec.: Precision; Rec.: Recall; F1: F1-score

####  4.4.3 Pipeline Comparison

We compared our proposed two-stage classifier pipeline with the baseline
single 3-class classifier. For each method, five of the best combinations were
picked to be compared. Based on the results in Table 7, the best outcome was
achieved by combining BERT for relevance classification and Bi-LSTM for
misinformation classification using IndoBERT as its feature with an accuracy
of 87.02% and F1 of 60.12%.

Apart from BERT+CNN, we found that the other top four models on the two-stage
Classifier pipeline achieved higher scores compared to the single classifier
model. Our proposed method by splitting the specific tasks yielded
substantially better results compared to using a multiclass classification.
Other than using the two-stage classifier pipeline, it was also indicated that
using pre-trained embeddings (Word2Vec [34] and IndoBERT [27]) produced better
results when compared to using more traditional features such as BoW and TF-
IDF.

###  4.5 Error Analysis

Figure 3: Ternary Class Confusion Matrix from The Best Model.

Figure 4: Binary Class Confusion Matrix from The Best Model. Annotation | Predicted |  Tweet Text |  Translated
---|---|---|---
Irrelevant | Misinformation |  Anies Bersyukur DKI Pecah Rekor Positif Corona, ini hasil Kita kerja Senyap dan Sunyi,,, kerja Senyap dan Sunyi kok anggaran juga Amblas |  ”Anies is grateful that DKI has a positive record of Corona, this is the result by persevering quietly”… If so, explain why the budgeting failed
Misinformation | Irrelevant |  VAKSIN CORONA !!! SUDAH ADA YANG. HALAL !!! KENAPA MAKSA !!!YANG VAKSIN HARAM !!! JELAS SEKALI KEJAHATANNYA !!! |  CORONA VACCINE !!! THERE IS THE HALAL OPTION !!! WHY ARE WE FORCED WITH THE NON-HALAL !!! OBVIOUSLY A CRIME !!!
Misinformation | True |  RSUD Banten Karantina 7 Mahasiwa Dari Wuhan Soal Virus Corona |  The Public Hospital in Banten Quarantined 7 Students from Wuhan Due to Coronavirus precaution
Table 8: Samples of False Predictions

We took the best model, which was BERT+Bi-LSTM to analyze the output
predictions. The confusion matrix for the three classes and two classes
classification are shown in Figure 4 and Figure 4 respectively.

The imbalanced nature of the dataset had an impact on the classifiers
especially in classifying the misinformation class. Both the true and
irrelevant classes were relatively easy to predict by the model. On the other
hand, the misinformation class was relatively more difficult to predict.

If the performance is to be viewed solely as a misinformation detection
performance, as shown in Figure 4, the model still performed well despite a
highly imbalanced dataset. Based on the numbers of false positives and false
negatives, the model also managed to find a good balance between precision and
recall, which resulted in almost no performance trade-off in either metric
(precision and recall).

Several false positives and false negatives were sampled to provide a case-by-
case error analysis shown in Table 8. In the first case, the tweet author
expressed their disapproval towards The Governor of Jakarta Capital Region,
Anies Baswedan, who had stated that Jakarta was showing improvements in
controlling the spread of the coronavirus thanks to the government’s effort.
During the annotation process, we decided to mark any politically-related
tweet as irrelevant. However, the model predicted the tweet as a
misinformation tweet.

Both the latter cases were false negatives. For the second case, it should be
noted that Indonesia is majorly populated by Muslims. Their belief (Islam)
does not allow the consumption or any means of inserting non-halal or haram
substances such as pork or dog meat into one’s body. During the beginning of
2021 when COVID-19 vaccines were starting to be distributed, there were
uncertainties regarding whether the vaccines were made out of halal materials.
The tweet author claimed that the vaccines distributed by the government at
that time, which was CoronaVac by Sinovac, was not halal, and blamed the
government for forcing the Muslims to get vaccinated. However, in this case,
the model predicted this as irrelevant and failed to flag this as
misinformation.

The third case was from a news outlet that had shared a headline on Twitter.
It turns out that there was a revision to the headline in the actual article
itself. Before the revision, the headline stated that 7 students were
quarantined. It was found later that there were 9 students instead of 7. The
annotators annotated this as misinformation due to the discrepancy between the
tweet and the revised article. However, the model predicted this as true. The
most possible explanation is that the text’s semantics were similar to other
headlines which were labeled as true.

##  5 Conclusion

It is difficult to predict when the spread of the COVID-19 would ease up and
no longer be declared a pandemic. False claims about the disease would
continue to be made and spread through different types of mediums. Moving
forward, the need to detect misinformation spread on social media would remain
not only around COVID-19 but also for different topics.

In this research, we have constructed dataset and conducted the experiments
for detecting COVID-19 misinformation that is spreading through social media.
Previous works were studied and synthesized to design a robust dataset
following a thorough annotation guideline. We formulated the misinformation
detection task as two serialized subtasks, i.e., relevance classification and
misinformation classification. Splitting the different tasks and using a pre-
trained language model improved the overall prediction performance.

For future research, the misinformation detection task may incorporate
language models and information retrieval techniques to automatically verify
the factual claims that are contained in the social media posts. To broaden
the research scope, the inclusion of other social media platforms can also be
considered. Lastly, the lack of annotated data could become an issue,
especially in low-resource languages, and approaches such as semi-supervised
learning can be explored for this task.

##  Appendix A Annotation Guideline

The following guidelines were shared with the annotators involved in this
research. The guidelines were originally written in Indonesian and then
translated into English by the authors.

Figure 5: Annotation Diagram.

###  A.1 Relevance Annotation Guideline

####  A.1.1 Task Definition

Annotations are to be made on Indonesian tweet data with topics revolving
around COVID-19. The expected output is a collection of labeled tweets that
have been annotated based on topic relevance and whether the tweet contains
verifiable factual claim.

The term verifiable factual claim refers to the definition made by [20] in
their annotation guideline. A verifiable factual claim is a statement that
claims something or some events are true and its truth can be verified using
any publicly available factual verified information such as statistics,
specific examples, or personal testimonials. The factual claim itself
includes:

  * 1.

A statement

  * 2.

Any mentions of a quantity value in the present or the past

  * 3.

Any predictions of the future that is verifiable

  * 4.

A reference to a product of law, procedures, or operational rules

  * 5.

A correlation or causal statement

####  A.1.2 Data Columns

  1. 1.

Tweet Columns

     * (a)

tweet_url: Anonymized link to the related tweet

     * (b)

text: The text in the tweet post

     * (c)

urls: Links that are written in the tweet

     * (d)

date: The date when the tweet was posted

  2. 2.

Annotation Columns

     * (a)

filter: Determine the initial relevance of the tweet (relevant / non-
Indonesian / out-of-topic)

     * (b)

personal: Whether the tweet contains something personal [35, 36] (true/false)

     * (c)

humor: Whether the tweet written for humor purposes [37] (true/false)

     * (d)

factual_claim: Whether the tweet fulfills the criteria of a verifiable factual
claim [20] (true / not sure / false)

####  A.1.3 Annotation Steps

  1. 1.

Identify the languages used in the tweet.

    1. (a)

If at least one of the following criteria is fulfilled, proceed to step 2.

       * i.

The entire tweet is written in Indonesian.

       * ii.

The Tweet is code-mixed, e.g., Indonesian-English [38] with the majority text
content is written in Indonesian.

    2. (b)

If the criteria is not fulfilled, annotate the column filter as non-indonesia.

  2. 2.

Identify the topic discussed in the tweet.

    1. (a)

If the tweet discussed any of the following topics, annotate the column filter
as out-of-topic.

       * i.

Political campaign or critic.

       * ii.

Discussed a certain topic but used a highly unrelated yet trending hashtag.

       * iii.

Any forms of advertisements.

    2. (b)

If the main topic discussed in the tweet involved one of the following topics,
annotate the column filter as relevant and proceed to step 3.

       * i.

The COVID-19 pandemic situation in Indonesia or globally.

       * ii.

Scientific discoveries, vaccines, medicines, or medical treatments related to
COVID-19.

       * iii.

Any official policies related to handling the COVID-19 pandemic in Indonesia
or globally.

       * iv.

Other affected sectors by the pandemic (socioeconomic for example)

    3. (c)

If it does not fulfill the previous two criterias, annotate the columns filter
as out-of-topic.

  3. 3.

Identify Verifiable Factual Claim [20] on the three columns as follow:

    1. (a)

personal. If the tweet discussed a personal experience, annotate this column
as true.

    2. (b)

humor. If the tweet has a humorous intent, annotate this column as true.

    3. (c)

factual_claim. If the tweet contains a factual claim that can be verified with
publicly available information, annotate this column as true. If you are
unsure, annotate this column as not sure. Otherwise, annotate false.

###  A.2 Misinformation Annotation Guideline

####  A.2.1 Task Definition

Annotations are to be made on Indonesian tweet data with topics revolving
around COVID-19. The expected output is a collection of tweets that have been
annotated based on the truth or misinformation of the verifiable factual claim
contained within the tweets. The tweets that are given are already filtered
based on their relevance. It should be noted that the annotation should only
address the claim mentioned in the text. The other form of media attached to
the tweet should not affect your judgment during the annotation process.

The term verifiable factual claim refers to the definition made by [20] in
their annotation guideline. A verifiable factual claim is a statement that
claims something or some events are true and its truth can be verified using
any publicly available factual verified information such as statistics,
specific examples, or personal testimonials. The factual claim itself
includes:

  * 1.

A statement

  * 2.

Any mentions of a quantity value in the present or the past

  * 3.

Any predictions on the future that is verifiable

  * 4.

A reference to a product of law, procedures, or operational rules

  * 5.

A correlation or causal statement

In this context, the misinformation in question is a factual claim that is:

  * 1.

Partly or completely false. Example: ”CAREFUL!! MICROCHIPS are planted within
the VACCINES!”.

  * 2.

Contains misleading or clickbait elements. Example: ”MUI (Indonesian Ulema
Council) declared COVID vaccines as haram!?”.

  * 3.

Rumors or conspiracy theories that have not been proven. Example: ”COVID will
end in 2022”.

####  A.2.2 Data Columns

  1. 1.

Tweet Columns

     * (a)

tweet_url: Anonymized link to the related tweet

     * (b)

text: The text in the tweet post

     * (c)

urls: Links that are written in the tweet

     * (d)

date: The date when the tweet was posted

  2. 2.

Annotation Columns

     * (a)

truth: The truth of the tweet (true / misinformation / not sure / need expert)

####  A.2.3 Steps

  1. 1.

Verification Steps

Tweet example: ”It turned out that Chloroquine has been used to treat several
COVID-19 patients in Indonesia” (March 23, 2020).

    1. (a)

Determine the information that needs to be verified. Example based on the
tweet above: ”Indonesia chloroquine COVID treatment”.

    2. (b)

Look for the information using a trusted search engine.

    3. (c)

If the tweet information matches the source of the article that can be
considered reliable (reputable media or medical institutions), then the
information can be considered true. Otherwise, the tweet may contain
misinformation.

    4. (d)

You can open the external hyperlinks (tweet_url or urls) to get more context
of the tweet. However, please note that only the text content is considered
for the annotation.

  2. 2.

Annotation Steps

    1. (a)

If the information is true, annotate as true.

    2. (b)

If the information is false, partly false, or misleading, annotate as
misinformation.

    3. (c)

If you are not sure about the truth of the claim, annotate as not sure.

    4. (d)

If you think an expert’s annotation is needed for this tweet, annotate as need
expert.

## References

  * [1] R. N. Chaterine, Update 17 desember: Ada 4.912 kasus aktif covid-19 di indonesia (Dec 2021).
URL https://nasional.kompas.com/read/2021/12/17/19190051/update-17-desember-
ada-4912-kasus-aktif-covid-19-di-indonesia

  * [2] Covid-19 vaccine tracker and landscape, [Online]. Available: https://www.who.int/publications/m/item/draft-landscape-of-covid-19-candidate-vaccines [Accessed: 18-Dec-2021] (Dec 2021).
URL https://www.who.int/publications/m/item/draft-landscape-of-
covid-19-candidate-vaccines

  * [3] Tracking sars-cov-2 variants (Nov 2021).
URL https://www.who.int/en/activities/tracking-SARS-CoV-2-variants/

  * [4] C. Juditha, Interaksi komunikasi hoax di media sosial serta antisipasinya hoax communication interactivity in social media and anticipation, Jurnal Pekommas 3 (1) (2018) 31–44.
  * [5] K. I. Center, Status Literasi Digital Indonesia 2020, Katadata Insight Center, 2020.
URL https://katadata-s3-public.s3.ap-
southeast-1.amazonaws.com/media/kic/kominfo/StatusLiterasiDigital_Nasional.pdf

  * [6] W. Jacobs, A. O. Amuta, K. C. Jeon, Health information seeking in the digital age: An analysis of health information seeking behavior among us adults, Cogent Social Sciences 3 (1) (2017) 1302785.  doi:10.1080/23311886.2017.1302785.
  * [7] M. S. Al-Rakhami, A. M. Al-Amri, Lies kill, facts save: Detecting covid-19 misinformation in twitter, IEEE Access 8 (2020) 155961–155970.  doi:10.1109/ACCESS.2020.3019600.
  * [8] M. K. Elhadad, K. F. Li, F. Gebali, Detecting misleading information on covid-19, IEEE Access 8 (2020) 165201–165215.  doi:10.1109/ACCESS.2020.3022867.
  * [9] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), 2019, pp. 4171–4186.  doi:10.18653/v1/N19-1423.
  * [10] X. Song, J. Petrak, Y. Jiang, I. Singh, D. Maynard, K. Bontcheva, Classification aware neural topic model for covid-19 disinformation categorisation, PloS One 16 (2) (2021) e0247086.  doi:10.1371/journal.pone.0247086.
  * [11] J. M. Banda, R. Tekumalla, G. Wang, J. Yu, T. Liu, Y. Ding, E. Artemova, E. Tutubalina, G. Chowell, A large-scale covid-19 twitter chatter dataset for open scientific research—an international collaboration, Epidemiologia 2 (3) (2021) 315–324.  doi:10.3390/epidemiologia2030024.
  * [12] E. Chen, K. Lerman, E. Ferrara, Tracking social media discourse about the covid-19 pandemic: Development of a public coronavirus twitter data set, JMIR Public Health and Surveillance 6 (2) (2020) e19273.  doi:10.2196/19273.
  * [13] N. A. Karlova, K. E. Fisher, A social diffusion model of misinformation and disinformation for understanding human information behaviour, Information Research (2013).
URL http://InformationR.net/ir/18-1/paper573.html

  * [14] K. K. Kumar, G. Geethakumari, Detecting misinformation in online social networks using cognitive psychology, Human-centric Computing and Information Sciences 4 (1) (2014) 1–22.  doi:10.1186/s13673-014-0014-x.
  * [15] B. Swire-Thompson, D. Lazer, Public health and online misinformation: challenges and recommendations, Annual Review of Public Health 41 (2020) 433–451.  doi:10.1146/annurev-publhealth-040119-094127.
  * [16] S. Vosoughi, D. Roy, S. Aral, The spread of true and false news online, Science 359 (6380) (2018) 1146–1151.  doi:10.1126/science.aap9559.
  * [17] S. Moturu, Quantifying the trustworthiness of user-generated social media content, Ph.D. thesis, Arizona State University Phoenix, Arizona (2009).
  * [18] X. Hu, H. Liu, Text analytics in social media, in: Mining text data, Springer, 2012, pp. 385–414.  doi:10.1007/978-1-4614-3223-4_12.
  * [19] L. Konstantinovskiy, O. Price, M. Babakar, A. Zubiaga, Toward automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection, Digital Threats 2 (2) (apr 2021).  doi:10.1145/3412869.
URL https://doi.org/10.1145/3412869

  * [20] F. Alam, F. Dalvi, S. Shaar, N. Durrani, H. Mubarak, A. Nikolov, G. D. S. Martino, A. Abdelali, H. Sajjad, K. Darwish, P. Nakov, Fighting the covid-19 infodemic in social media: A holistic perspective and a call to arms, in: Proceedings of the International AAAI Conference on Web and Social Media, ICWSM ’21, 2021, pp. 913–922.
URL https://ojs.aaai.org/index.php/ICWSM/article/view/18114

  * [21] M. Aldwairi, A. Alwahedi, Detecting fake news in social media networks, Procedia Computer Science 141 (2018) 215–222.  doi:10.1016/j.procs.2018.10.171.
  * [22] O. Sharif, E. Hossain, M. M. Hoque, Combating hostility: Covid-19 fake news and hostile post detection in social media, arXiv preprint arXiv:2101.03291 (2021).
  * [23] P. Patwa, S. Sharma, S. Pykl, V. Guptha, G. Kumari, M. S. Akhtar, A. Ekbal, A. Das, T. Chakraborty, Fighting an infodemic: Covid-19 fake news dataset, in: International Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situation, Springer, 2021, pp. 21–29.  doi:10.1007/978-3-030-73696-5_3.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008.
URL
https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-
Paper.pdf

  * [25] T. Hossain, R. L. Logan IV, A. Ugarte, Y. Matsubara, S. Young, S. Singh, Covidlies: Detecting covid-19 misinformation on social media, in: Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, 2020.  doi:10.18653/v1/2020.nlpcovid19-2.11.
  * [26] M. Hardalov, I. Koychev, P. Nakov, In search of credible news, in: International conference on Artificial intelligence: methodology, systems, and applications, Springer, 2016, pp. 172–180.  doi:10.1007/978-3-319-44748-3_17.
  * [27] B. Wilie, K. Vincentio, G. I. Winata, S. Cahyawijaya, X. Li, Z. Y. Lim, S. Soleman, R. Mahendra, P. Fung, S. Bahar, A. Purwarianti, Indonlu: Benchmark and resources for evaluating indonesian natural language understanding, in: Proceedings of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing (AACL-IJCNLP), 2020, pp. 843–857.
URL https://aclanthology.org/2020.aacl-main.85

  * [28] U. Qazi, M. Imran, F. Ofli, Geocov19: a dataset of hundreds of millions of multilingual covid-19 tweets with location information, SIGSPATIAL Special 12 (1) (2020) 6–15.  doi:10.1145/3404820.3404823.
  * [29] A. Zubiaga, I. S. Vicente, P. Gamallo, J. R. P. Campos, I. Alegria, N. Aranberri, A. Ezeiza, V. Fresno-Fernández, Tweetlid: a benchmark for tweet language identification, Language Resources and Evaluation 50 (2016) 729–766.  doi:10.1007/s10579-015-9317-4.
  * [30] B. Ranaivo-Malancon, Automatic identification of close languages - case study: Malay and indonesian, ECTI Transactions on Computer and Information Technology (ECTI-CIT) 2 (2) (2016) 126–134.  doi:10.37936/ecti-cit.200622.53288.
  * [31] E. O. Nsoesie, N. Cesare, M. Müller, A. Ozonoff, Covid-19 misinformation spread in eight countries: Exponential growth modeling study, Journal of Medical Internet Research 22 (12) (2020) e24425.  doi:10.2196/24425.
  * [32] N. M. Nasir, B. Baequni, M. I. Nurmansyah, Misinformation related to covid-19 in indonesia, Jurnal Administrasi Kesehatan Indonesia 8 (2) (2020) 51–59.  doi:10.20473/jaki.v8i0.2020.51-59.
  * [33] M. L. McHugh, Interrater reliability: the kappa statistic, Biochemia Medica 22 (3) (2012) 276–282.
URL https://hrcak.srce.hr/89395

  * [34] D. N. Karyana, Membuat model word2vec bahasa indonesia dari wikipedia menggunakan gensim, [Online]. Available: https://medium.com/@diekanugraha/membuat-model-word2vec-bahasa-indonesia-dari-wikipedia-menggunakan-gensim-e5745b98714d (Jun 2018).
URL https://medium.com/@diekanugraha/membuat-model-word2vec-bahasa-indonesia-
dari-wikipedia-menggunakan-gensim-e5745b98714d

  * [35] S. Dann, et al., Twitter content classification (2010).
  * [36] A.-Z. Yen, H.-H. Huang, H.-H. Chen, Detecting personal life events from twitter by multi-task lstm, in: Companion Proceedings of the The Web Conference 2018, WWW ’18, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 2018, p. 21–22.  doi:10.1145/3184558.3186909.
URL https://doi.org/10.1145/3184558.3186909

  * [37] R. Zhang, N. Liu, Recognizing humor on twitter, in: Proceedings of the 23rd ACM international conference on conference on information and knowledge management, 2014, pp. 889–898.  doi:10.1145/2661829.2661997.
  * [38] A. M. Barik, R. Mahendra, M. Adriani, Normalization of indonesian-english code-mixed twitter data, in: Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), 2019, pp. 417–424.  doi:10.18653/v1/D19-5554.