# Challenges and Considerations with Code-Mixed NLP for Multilingual Societies

Vivek Srivastava
TCS Research
Pune, Maharashtra, India
<EMAIL_ADDRESS>
Mayank Singh
IIT Gandhinagar
Gandhinagar, Gujarat, India
<EMAIL_ADDRESS>

###### Abstract

Multilingualism refers to the high degree of proficiency in two or more
languages in the written and oral communication modes. It often results in
language mixing, a.k.a. code-mixing, when a multilingual speaker switches
between multiple languages in a single utterance of a text or speech. This
paper discusses the current state of the NLP research, limitations, and
foreseeable pitfalls in addressing five real-world applications for social
good—crisis management, healthcare, political campaigning, fake news, and hate
speech—for multilingual societies. We also propose futuristic datasets,
models, and tools that can significantly advance the current research in
multilingual NLP applications for societal good. As a representative example,
we consider English-Hindi code-mixing but draw similar inferences for other
language pairs.

##  1 Introduction

The term “multilingualism” refers both to an individual’s ability to use
several languages and the coexistence of different language communities in one
geographical area. Contrary to the general belief, most of the world’s
population is bilingual or multilingual. Monolingualism is characteristic only
of a minority of the world’s population Valdés (2007). Multilingualism
sometimes uses elements of multiple languages when conversing with each other,
resulting in a code-mixed conversation. Code-mixing (aka code-switching), as
formally defined by Bokamba (1989), is the mixing of words, phrases, and
sentences from two distinct grammatical (sub)systems within the same speech
event. With the wide-reaching popularity of social media platforms, code-
mixing has emerged as one of the significant linguistic phenomena among
multilingual communities Das and Gambäck (2015). Throughout this paper, we
only consider code-mixing as intra-sentential wherein language switching
occurs inside the sentence.

In this paper, we present five real-world applications for social good—crisis
management, healthcare, political campaigning, fake news, and hate speech —and
discuss the current ability of the NLP tools in handling code-mixed data. We
critically analyze the progress in NLP in processing code-mixed data and
present immediate research directions and potential pitfalls. In contrast to
existing survey literature on code-mixing, we primarily focus on the NLP
applications for societal good and evaluate NLP venues’ contribution towards
such applications from a code-mixing perspective.

English: We have a fully autonomous vehicle. Romanized Hindi: hamaare paas poori tarah se svaayatt vaahan hai Hinglish: Hamare paas fully autonomous vaahan hai Figure 1: A representative Hinglish sentence and the corresponding parallel Hindi-English sentences Srivastava and Singh (2020b). Application | Example
---|---
2.1cmCrisis Management |  12.2cmHinglish: Delhi NCR may bhukamp? Assam Bengal ke baad ab Delhi may
12.2cmEnglish: Earthquake in Delhi NCR? After Assam Bengal now in Delhi.
2.1cmHealthcare |  12.2cmHinglish: Nahi bole toh bhi kam se kam 250-300 nuske bata chuke hai yeh log Corona khatam karne ke liye. Aur WHO ainwayi keh raha hai ke Corona ke liye vaccine nahi bani abtak
12.2cmEnglish: These people have suggested atleast 250-300 prescriptions to
cure CORONA. And WHO, without any reason, is saying we have no vaccine for
CORONA.
2.3cmPolitical Campaigning |  11cmHinglish: “Beti Bachao” is a myth. “Beti BJP Se Bachao” is reality!
11cmEnglish: Save daughter is a myth. Save daughter from BJP is a reality.
2.3cmFake News |  12cmHinglish: WHO ne Bharat ko Karuna free declare Kar Diya hai. Humare PM narinder Modi is being awarded best PM award by UNESCO
12cmEnglish: WHO has declared India as Corona free. Our Prime minister
Narindra Modi is being awarded best PM award by UNESCO..
2cmHate Speech |  11cmHinglish: Look ye politicians suvar jaise baithe rahte hain sirf money ke liye kaam karte hain. They don’t care about public.
11cmEnglish: Look these politicians are sitting like pigs and they only work
for money. They don’t care about public.
Table 1: Representative examples for each application as described in Section
2.

As a representative example, in this paper, we primarily focus on Hinglish, a
portmanteau of Hindi and English. Highlish is highly prevalent in the Indian
sub-continent. Baldauf (2004) projected in 2004 that the world’s Hinglish
speakers ($>$350 million) might soon outnumber native English speakers.
Hinglish presents significant linguistic challenges. In contrast to other
code-mixed pairs such as Spanglish (a portmanteau of Spanish and English),
which carries a relative similarity in Spanish and English syntactic
structure, Hinglish is constructed over two syntactically divergent languages
Berk-Seligson (1986). Linguistic theories such as equivalence constraint
theory POPLACK (1980) and size-of-constituent constraint theory Gumperz and
Hernandez (1969) and the associated NLP algorithms and tools that were
proposed for syntactically similar languages poorly perform for a code-mixed
text generated from syntactically divergent languages. Figure 1 shows the
Hinglish text and the corresponding parallel Hindi-English sentences.

We present a general introduction to the five applications for social good in
the next section. Section 3 describes the current state of the research in
these application areas. We present limitations of existing NLP research for
code-mixing in Section 4 and identify futuristic application-specific
datasets, models and tools in Section 5. Towards the end, Section 6 concludes
the discussion.

##  2 Applications for societal good

This paper focuses on five popular applications for social good and discusses
these applications from a multilingual community perspective, wherein the data
generated comprises code-mixed text, largely available over the internet.
Next, we describe each application in detail.

###  2.1 Crisis Management

Crisis encompasses both man-made and natural events. Man-made crisis events
include military war, explosions, acts of terrorism, extreme pollution levels,
hazardous material spills, fires, transportation accidents, structure
failures, and mining accidents. In contrast, natural crisis events include
catastrophic events with atmospheric, geological, and hydrological origins
(e.g., droughts, earthquakes, floods, hurricanes, landslides) Prasad and
Francescutti (2017). Interestingly, the current COVID-19 crisis can be
attributed to both man-made and natural crises.

As expected, these crisis events affect a large population, resulting in a
large volume of multi-modal data such as text messages, videos, and images.
The generated text data includes real-time casualty and loss statistics,
donation announcements, volunteering calls, relief camp locations, along with
negative aspects such as misinformation, disinformation, and rumors during a
health emergency. Thus, the primary role of NLP systems in crisis data
management is not only to enable a large proportion of the affected population
to consume the factually correct data but also to reduce the effect of
incorrect/false information before it reaches the affected population
(hereafter, crisis data misinformation (CDM)). Another important NLP goal is
disaster response, wherein the system identifies information related to
emotional support to the affected population, donations and volunteering and
caution and advice from government bodies and social organizations, government
announcements related to infrastructure (hereafter, crisis data response
(CDR)). The third major goal is to summarize the large temporal data volume in
real-time and present it in a meaningful fashion for easier consumption
(hereafter, crisis data summarization (CDS)).

###  2.2 Healthcare

The Healthcare domain has witnessed tremendous advancements in the last two
decades. Many healthcare systems allow patients to access their electronic
health record (EHR) notes online through patient portals. Applications such as
personalized healthcare chatbots are increasingly becoming popular Futurist
(2020). Online medical portals such as Prato111http://www.practo.com and
Plushcare222https://plushcare.com are helping patients to connect and find
doctors. Online healthcare blogs present well-rounded, perceptive, and
informative news sources and opinions from healthcare experts.

Understanding healthcare information from the above sources requires NLP at
its core. The major goal of NLP is to understand the medical jargon and
construct a mapping between non-standard terminologies used in different
geographies. This understanding helps in constructing association between
medical processes (hereafter, healthcare knowledge graph construction (HKG)).
The second major goal is in developing personalized healthcare chat agents
(hereafter, healthcare chatbots (HCB)). The third major goal is to debunk the
misinformation associated with the health and medical terms (hereafter,
healthcare misinformation (HMI)).

###  2.3 Political campaigning

Social platforms play a critical role in the new political campaigning
paradigm. Politicians exchange views on the latest partisan developments and
invite the public and citizens to comment, share ideas, and adhere to their
political program Phillips and Young (2009). Previous research shows that the
successful social media campaign positively correlates with the election
outcomes Baxter and Marcella (2012); Barclay et al. (2015); Kanungo (2015);
Heredia et al. (2018). Similarly, in the recent U.S. elections, a study
Wharton Business Daily (2020) shows that within the first month of using
Twitter, politicians were able to raise between 1% and 3% of what they would
have raised in a traditional two-year campaign. Some of the positive aspects
of social media-based political campaigns are wider reach, no traditional
media filter, two-way discussions, and negligible funding requirements. Some
of the negative aspects include malicious propaganda against rival parties and
exaggerated achievements.

NLP systems can be deployed to understand the ongoing political campaigns and
to make people aware of the factually correct information and red-flag the
misinformation (hereafter, political data misinformation (PDM)). Another
important goal is to measure the public perception about the ongoing political
campaigns and present it to the political parties and the general mass. These
opinions help a political party to plan its programs and visions and a voter
to see both sides of campaigns (hereafter, political opinion extraction
(POE)).

###  2.4 Fake news

Fake news is a critical yet challenging problem in NLP. The rapid rise of
social networking platforms has yielded a vast increase in information
accessibility and accelerated the spread of fake news Oshikawa et al. (2020).
We witness fake news spreaders on diverse platforms ranging from political
discussions, movie recommendations, e-retail product reviews. In the last
decade, we witness a surge in fake news on sensitive and critical topics like
pandemic Kar et al. (2020), disasters Kwanda and Lin (2020), and war Salem et
al. (2019). In the ongoing pandemic scenario, where major business, education,
and political activities have been confined to online settings, the volume of
fake news has grown substantially Sujeet K Sharma and Chandwani (2020).

As the menace of fake news encompasses multiple domains, the primary goal of
the NLP system is to detect such fake messages, posts, or reviews (hereafter,
fake news detection (FND)). FND is highly challenging in a real-time setting,
where incoming messages are flagged according to the likelihood of being fake.
The second major goal is the course correction, i.e., to summarize and present
facts against the fake news (hereafter, fake news explanation (FNE)).

###  2.5 Hate speech

Hate speech is commonly defined as any communication that disparages a person
or a group based on some characteristic such as race, color, ethnicity,
gender, sexual orientation, nationality, or religion Nockleby (2000). Due to
the massive rise of user-generated web content, the amount of hate speech is
also steadily increasing Schmidt and Wiegand (2017). Government organisations
Singh (2020); Bayer and Bárd (2020) have framed strict laws to deal with
offensive messages sent through social media and online messaging
applications. The major challenge in hate speech detection lies in the
diversity of characteristics. An NLP system, depending on the characteristics
of hate speech, should be able to process and detect such hateful content
effectively (hereafter, hate speech detection (HSD)).

To summarize, in this section, we elaborate on five applications of NLP for
societal good. For each application, we propose multiple sub-tasks. Table 1
presents representative Hinglish examples for each application described in
this section. Since we focus on the applications from a code-mixed data
perspective, the next section describes the current NLP research status
addressing the above issues in code-mixed data.

##  3 The current state of the research

We study two different perspectives to understand the current state of the
computational linguistic research for the societal good applications in
multilingual societies. First, we analyze the five major ACL and non-ACL
conferences to understand the NLP community’s global trend. Next, we present
an analysis of the currently available resources and some top-performing
systems for each of the five societal good applications.

###  3.1 Are we organizing enough workshops for the societal good?

Workshops are major sources to promote the research in a dedicated area of
study. In this paper, we analyze five major NLP conferences (ACL, EMNLP,
NAACL333NAACL happened in the year 2016, 2018, and 2019., COLING, and
LREC444LREC happened in the year 2016, 2018, and 2020.) organized in the past
five years (2016–2020). Over the years, almost all the top preferred
conferences in the NLP community are organizing workshops to promote and
include community members’ diverse research interests (see Table 2).
Specifically, in the healthcare domain, we see a significant number of
workshops in ACL, EMNLP, and LREC. For other societal good applications, the
number of organized workshops is considerably low. Also, except the workshop
on computational approaches to linguistic code-switching (CALCS), none of
these workshops primarily focus on code-mixed text.

|  Crisis Management |  Healthcare |  Political Campaigning |  Fake News |  Hate Speech | Total
---|---|---|---|---|---|---
ACL | 3 | 10 | 1 | 1 | 2 | 82
EMNLP | 1 | 8 | 2 | 3 | 3 | 65
NAACL | 0 | 2 | 1 | 0 | 0 | 51
COLING | 0 | 3 | 1 | 3 | 2 | 40
LREC | 0 | 7 | 3 | 2 | 2 | 96

Table 2: Distribution of the societal good workshops organized during the
five-year interval (2016–2020) in the five major NLP conferences. Total
represent total number of workshops in the five-year interval.

###  3.2 What is accomplished so far? Is it really “state-of-the-art”?

Here, we explore some major works in the five societal good applications for
multilingual societies. We provide an empirical study to highlight various
strengths and limitations of these notable works. Since we consider the
majority of the popular and recent works in this discussion, it will help
capture the community’s diverse research interests. It will also pave the way
for developing effective systems in the future.

####  3.2.1 Crisis management

In Section 2.1, we discuss several NLP tasks in crisis and disaster
management. To the best of our knowledge, we do not find any work primarily
focusing on code-mixed crisis data. However, we find several works that
process noisy social-media text for generating insights and responses. We list
these works below:

  * •

CDM: We witness a large volume of work on addressing the crisis data
misinformation. Particularly, majority of the recent works focus on curating
misinformed social-media posts during COVID-19 pandemic Shahi and Nandini
(2020); Li et al. (2020); Kar et al. (2020); Patwa et al. (2020b); Qazi et al.
(2020). Although all these works report multilingual data curation, we do not
find any proposed misinformation identification model that leverages these
datasets to train misinformation detection models. Additionally, we find
several other monolingual datasets discussing events like Ukrainian conflict
Khaldarova and Pantti (2016), Syrian War Salem et al. (2019) and Palu
earthquake Kwanda and Lin (2020). The majority of the crisis datasets are
monolingual and present limited opportunities to develop tools to identify
multilingual misinformed content. This limitation fosters the code-mixed
misinformation spreader to spread the false information in a time of crisis
easily.

  * •

CDR: We witness very few works addressing crisis data response. For example,
Imran et al. (2013) extracted valuable information nuggets relevant to
disaster response from microblogs. They curated a small dataset of 1,233
English sentences. However, we do not find any work that curates the above
extremely time-sensitive information available in the code-mixed language.

  * •

CDS: We witness few recent works on summarizing crisis-related data available
as microblogs Rudra et al. (2015, 2016, 2018a). All these works address the
problem in two phases. In the first phase, they extract the relevant
situational information from among the large volume of tweets. In the second
phase, they summarize the extracted information. Note that the proposed
methods only work for the English Language. Off-the-shelf usage of these works
for the code-mixed languages might not give the desired results.

####  3.2.2 Healthcare

In Section 2.2, we discussed the importance and the necessities to address the
various healthcare-related challenges. We layout three major sub-fields (HKG,
HCB, and HMI) where computational linguistic could play a major role. We could
not find any dedicated study pertaining to the code-mixed languages to the
best of our knowledge. Here, we present some well-known works in other
languages and draw insights from them that present future research directions
in the code-mixing domain.

  * •

HKG: Healthcare knowledge graphs Lindberg et al. (1993); Oliveira et al.
(2017) are extremely useful in converting enormous healthcare data in the
machine-understandable format. HKG is also useful in informing healthcare
practitioners of the latest research results Sadeghi and Lehmann (2019). It
has proven useful in multiple instances such as diagnosis prediction Choi et
al. (2017); Ma et al. (2018) and analytics Aasman and Mirhaji (2018). Code-
mixed data available on various social media platforms (e.g., Twitter and
Facebook) and healthcare discussion forums (e.g., India medical
hub555http://www.indiamedicalhub.com/ and patient666https://patient.info) can
be used to expand existing monolingual HKGs.

  * •

HCB: Healthcare chatbots are the future for query resolution Amato et al.
(2017); Bates (2019), consultation Kowatsch et al. (2017), medical history
analysis Divya et al. (2018), and treatment Cameron et al. (2018); Bibault et
al. (2019); Chaix et al. (2019). The majority of the HCBs are designed for
formal communication, making it difficult for the multilingual speakers to
have a human-like conversation with these chatbots as they lack the human
emotions Palanica et al. (2019).

  * •

HMI: Misinformation in the healthcare domain can result in a serious negative
impact on the well-being of the people Tasnim et al. (2020). Recently, a wave
of misinformation related to the COVID-19 pandemic has propagated and studied
Brennen et al. (2020); Pennycook et al. (2020); Kouzy et al. (2020). Several
domain-specific misinformation detection systems Cui et al. (2020); Hou et al.
(2019); Bal et al. (2020); Cui and Lee (2020) have been proposed to check the
healthcare-related misinformation. The majority of these works focus on the
monolingual data available on various social media platforms. However, due to
the popularity of code-mixing conversations in multilingual communities, the
risk of misinformation spread increases manifold. Most healthcare-related
information is generated in monolingual language at the source (hospitals,
healthcare centers, and diagnostic labs).

####  3.2.3 Political campaigning

The recent thrust on social media and online news platforms has contributed
significantly to the change of medium for political campaigning. These
platforms greatly facilitate reaching the masses quickly, but it adds several
challenges such as misinformation and hate speech. Here, we explore two sub-
fields (PDM and POE) where computational linguistic could help improve the
situation. Since we cannot find any study for the code-mixed languages, we
will explore monolingual and multilingual works.

  * •

PDM: Misinformation during election campaigning misleads people with
manipulated facts and rumors Badawy et al. (2018); Cantarella et al. (2020).
Multiple works identify the spread of political misinformation on various
platform such as WhatsApp Garimella and Eckles (2020); Machado et al. (2019);
Resende et al. (2019), Facebook Guess et al. (2019); Mena (2020), and Twitter
Shin et al. (2017); Grinberg et al. (2019); Neudert et al. (2017). In order to
address political misinformation, various fact-checking platforms have been
introduced Hassan et al. (2015); Myslinski (2015); Rashkin et al. (2017);
Hassan et al. (2017); Thorne and Vlachos (2018). Almost all of the above works
curate and classify misinformation in English language.

  * •

POE: Extracting opinion from the political campaigning data could be extremely
helpful in building efficient election result forecasting systems and real-
time visualizations Kaschesky et al. (2011). Various works have explored
multiple sources to extract people’s political opinions such as Twitter Asghar
et al. (2014); Joshi et al. (2016); Gull et al. (2016), news sources (such as
NPR, Mother Jones, and Politico) Bamman and Smith (2015) and YouTube Kandias
et al. (2013); Acharoui et al. (2020).

####  3.2.4 Fake news

Fake news is one of the biggest challenges on social media platforms. Over the
years, it has been an active area of research for the computational linguistic
community. Unfortunately, we could not find any work for the fake news
propagation in the code-mixed languages. Here, we explore some prominent works
in other languages for the two major subareas (FND and FNE).

  * •

FND: Detecting fake news and the fake news spreaders is the primary step in
checking the propagation of fake news Shu et al. (2017); Meel and Vishwakarma
(2019). Over the years, various benchmark datasets have been proposed Wang
(2017b); Golbeck et al. (2018); Nakamura et al. (2020) to detect fake news on
various platforms. Recently, FND for low-resource Hossain et al. (2020); Amjad
et al. (2020) and multilingual Schwarz et al. (2020); Shahi and Nandini (2020)
languages have gained interest from the computational linguistic community.

  * •

FNE: Black-box models for the fake news detection adds to the complexity of
identifying and filtering the fake news Reis et al. (2019). The reasoning,
interpretability, and the explanation Shu et al. (2019) of the fake news
detection models are extremely important for websites with extensive outreach
(e.g., social media and news websites). Real-time prediction and the
visualization of the fake news and its interpretation Yang et al. (2019) could
be extremely useful in several cases for multilingual societies.

####  3.2.5 Hate speech

In contrast to the above societal good applications, hate speech detection in
the code-mixed languages has been an active area of exploration Santosh and
Aravind (2019); Chopra et al. (2020); Kamble and Joshi (2018). Here, we
discuss two very popular datasets (for the code-mixed languages) by Bohra et
al. (2018) (hereafter, HD1) and Mandl et al. (2019)) (hereafter, HD2) to
detect hate speech on social media platforms. HD1 presents a dataset of 4,575
Hinglish tweets annotated with either of the binary labels: hate speech or
normal speech. HD2 is a fine-grained hate-speech detection dataset in
Hinglish. Labels in HD2 can be used to identify hate speech, offensive
language, and the profanity in the 5,983 tweets. Even though these works
address the data scarcity issue for the code-mixed languages, the quality and
the quantity of the examples in the dataset remains debatable. The size of HD1
and HD2 datasets is significantly small as compared to HD datasets in other
monolingual languages Davidson et al. (2017); Mathew et al. (2020).

##  4 Limitations

As discussed in Section 3.2, the majority of the research for good societal
applications for multilingual societies has enormous opportunities. This
section discusses some major limitations that text NLP researchers and
practitioners encounter very often in code-mixing. Srivastava and Singh
(2020b) highlights six major limitations with the machine translation systems
on the code-mixed text. The majority of these limitations stand true in
dealing with societal good applications as well. Here, we present a
comprehensive list of challenges and limitations that need to be carefully
addressed to build robust and efficient systems and an encouraging environment
in the multilingual research community.

  * •

Limited text processing tools: The limited availability of the text processing
tools build especially for the code-mixed language makes it extremely
challenging to create large scale robust and efficient systems. Efficient
tools with functionalities like POS tagging, named entity recognition,
language identification, transliteration, key phrase extraction, and topic
modeling are yet to be developed at a large scale. The scarcity of high-
quality datasets for these tasks makes it a vicious cycle to build
multilingual societies’ solutions.

  * •

Identifying and filtering code-mixed data: One of the major challenges with
the code-mixed data is the coexistence with other languages. We rarely find
any platform where people communicate solely in code-mixed languages. The non-
availability of automatic filtering tools makes it extremely challenging to
create a large-scale code-mixed corpus. In most cases, people employ humans in
the loop to identify and filter the code-mixed data Kumar et al. (2018); Vyas
et al. (2014); Chandu et al. (2019), but this approach is highly time and cost
extensive.

  * •

Human bias: Inherent human bias is one of the most significant and dangerous
challenges for most NLP systems. Code-mixed languages also suffer from such
biases very often. In addition to the bias present in the data generated on
the various platforms (Twitter, Facebook, WhatsApp, Reddit, and Quora), we
observe a very high bias in the dataset annotation by the human annotators
Geva et al. (2019); Srivastava and Singh (2020a). On manual inspection, we
observe a significantly high ambiguity in the sentiment classification
datasets for the Hinglish language (see Example I in Figure 2). The primary
reason for this ambiguous behavior can be attributed to the underlying bias of
the human annotators.

  * •

Annotator’s proficiency: The lack of grammatical standardization (see Example
II & IV in Figure 2) for the code-mixed languages presents the challenge of
spelling variation, dialect, and readability. This standardization scarcity
makes it challenging to employ human annotators for an annotation task. Also,
evaluating an annotator’s proficiency in a particular code-mixed language is a
challenge. The natural language generation tasks such as machine translation,
text generation, data to text generation, and summarization suffer heavily due
to this constraint.

  * •

Evaluation metrics: The evaluation metrics developed for the monolingual
languages does not seem to capture the linguistic diversity of the code-mixed
languages. The majority of the widely used NLG related metrics such as
Bilingual Evaluation Understudy (BLEU) Papineni et al. (2002), Word Error Rate
(WER) Levenshtein (1966), and Translation Error Rate (TER) Snover et al.
(2006), gives a false impression of the system performance (see Example III in
Figure 2). We need metrics that can account for the various attributes of the
code-mixed languages, such as spelling variations, informal writing style, and
contextual information.

  * •

Reproducibility: Reproducibility is a major issue for the majority of the NLP
research Wieling et al. (2018). Code-mixing, which is a relatively
understudied area of research, suffers significantly due to the non-
reproducible research. Be it off the self-usage or baseline comparison, the
non-reproducible research is a significant bottleneck for the code-mixing
research. Also, the lack of publicly available datasets limits the scope of
future opportunities.

  * •

Benchmark: Recently, we witness several efforts Aguilar et al. (2020); Khanuja
et al. (2020) to benchmark various datasets and tasks in code-mixed language.
The majority of these benchmarks leverage very small-scale datasets, which
make generalizability and large-scale applicability nearly impossible. Most
critical NLP tasks, such as question-answering, machine translation, and
summarization, are still unaddressed due to either lack of research or the
unavailability of publicly available datasets and systems.

  * •

Official status: Since the code-mixed languages do not possess an official
status, it becomes challenging to enforce standardization. The majority of the
recent efforts in NLP for the multilingual communities Kakwani et al. (2020);
Hu et al. (2020); Xue et al. (2020) do not include the code-mixed languages
even though the number of speakers for such languages is very high.

Example I (Human bias) Hinglish Patwa et al. (2020a): Twitter k baghair apna
roza mumkin nahi hota ? Apna chutiyaap dusron per thopna band karo Bhai !
Annotated sentiment: Positive Possible sentiment(s): Negative, Neutral Example
II (Annotator’s proficiency) Hinglish Patra et al. (2015): uske baap ko mobile
ki factory tha kya. Incorrect annotator’s language label: mobile- Hindi
Correct language label: mobile-English Example III (Evaluation metrics) Source
Hinglish Srivastava and Singh (2020b): kaun hai ye zaleel insaan? Reference
Hinglish: Who is this zaleel person? BLEU: 0.7598 Human score: 1 Example IV
(Lack of standardization) Spelling variation Dhar et al. (2018): Jaldi
$\rightarrow$ [Jldi, Jalde, Jald], Jyada $\rightarrow$ [Jada, Jyda, Jyaada]
Language ambiguity Patra et al. (2015): {to, us, is, the} $\rightarrow$
{English, Hindi} Figure 2: Example of the various limitations in the existing
code-mixed datasets. The cited works are a representative example and not an
exhaustive list.

| Possible code-mixed data source(s) | Already existing datasets (Language)
---|---|---
CDM |  Shahi and Nandini (2020), Li et al. (2020), Li et al. (2020), Kar et al. (2020), Patwa et al. (2020b) |  Montesi (2020) (Es)
CDR | crisisnlp.qcri.org |  Imran et al. (2013) (En)
CDS | crisisnlp.qcri.org |  Rudra et al. (2015) (En), Rudra et al. (2016) (En)
HKG | patient.info, emedinexus, indiamedicalhub, healthvision.in |  Lindberg et al. (1993) (En), Oliveira et al. (2017) (En)
HCB | nhp.gov.in, nhsrcindia.org, nathealthindia.org, ima-india.org |  Ur Rahman Khilji et al. (2020) (En)
HMI | thip.media, poynter.org/coronavirusfactsalliance |  Zhao et al. (2021) (En)
PDM | newschecker.in, politifact.com |  Wang (2017a) (En), Nørregaard et al. (2019) (En)
POE | guides.nyu.edu/polisci/public-opinion-data, ucsd.libguides.com/data-statistics/publicopinion |  Kannangara (2019) (En)
FND | altnews.in, boomlive.in, smhoaxslayer.com, factchecker.in |  Wang (2017a) (En),Golbeck et al. (2018) (En)
FNE | factcrescendo.com, vishvasnews.com, factly.in, maldita.es |  Wang (2017a) (En)
HSD | hatespeechdata.com, NEWS comments Conversation (2016); csmonitor (2016) |  Rudra et al. (2018b) (En)

Table 3: Various possible code-mixed data source(s) for the societal good
applications. Already existing datasets in various languages could be used to
generate the synthetic data for each of these tasks. Here ‘En’ represents
English and ‘Es’ represents Spanish language.

##  5 The Futuristic NLP for Applications of Social Good

This section proposes futuristic datasets, models, and tools that can
significantly advance the current state of the research in multilingual NLP
applications for societal good.

###  5.1 Datasets

As noted in the previous sections, natural code-mixed datasets are rare and
small in size, with a major focus on selected applications. Here, we present
some primary data sources for all the five societal good applications. We
summarize our findings777Due to the ease of accessibility of the code-mixed
text on the social media platforms (WhatsApp, Facebook, and Twitter), majority
of the works considers these platforms. Here, we present the platforms which
could also be a good alternative to curate the code-mixed datasets. in Table
3. There are several good quality datasets already available in various other
languages for these tasks (see Table 3, third column). We could effectively
utilize these monolingual datasets to create the code-mixed datasets using
various code-mixed text generation systems Gupta et al. (2020); Pratapa et al.
(2018). Since most of these systems use parallel sentences to generate code-
mixed text, we can create parallel data from state-of-the-art translation
systems by first translating the original monolingual source data and then use
these systems to generate the code-mixed text. We can also use this technique
to convert the monolingual data from the possible code-mixed data sources
(mentioned in Table 3, second column) to get the code-mixed data. This
approach can significantly address the data scarcity problem in the code-mixed
languages.

###  5.2 Models

The recent NLP advancements contribute to popular contextual representations
constructed using (semi-)unsupervised language models Vaswani et al. (2017).
As these language models are trained on large monolingual datasets, they do
not perform well in code-mixing tasks Khanuja et al. (2020). We, therefore,
propose the construction of pre-trained models from the large-scale code-mixed
text. The proposed datasets (see Section 5.1) can be used for this pretraining
and further task-specific fine-tuning.

###  5.3 Tools

In the future, we envisage several NLP tools deployed in the field that can
efficiently curate and process code-mixed data. For example, significant
information is available to debunk a fake tweet by just looking into the
replies. In most cases, the replies are comparatively more informal and
contain facts in code-mixed language. The facts can be extracted and encoded
into completely data-driven automatic fake news detectors for code-mixed text.
This also helps in reducing political and ethical bias towards the source of
fake news van der Linden et al. (2020). Several media websites that curate and
track hate speech are currently shutdown under political pressure. Thus, an
automatic curation and tracking tool for hate speech in a multilingual
community—where the text is mostly informal and contains large volumes of
code-mixing—can benefit society.

##  6 Conclusion

This paper discusses five applications for social good and presents the
current state of the research in each application area. We critically analyze
the code-mixing research in each application area and show an extensive
opportunity for future research for dataset curation, modeling, and tool
development. We aim to develop and extend our expertise in handling the large
volume of code-mixed data in the future.

## References

  * Aasman and Mirhaji (2018) Jans Aasman and Parsa Mirhaji. 2018.  Knowledge graph solutions in healthcare for improved clinical outcomes.  In _CEUR Workshop Proceedings_ , volume 2180. CEUR-WS.
  * Acharoui et al. (2020) Zakia Acharoui, Altaf Alaoui, Badia Ettaki, Jamal Zerouaoui, and Mohamed Dakkon. 2020.  Identifying political influencers on youtube during the 2016 moroccan general election.  _Procedia Computer Science_ , 170:1102–1109.
  * Aguilar et al. (2020) Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020.  Lince: A centralized benchmark for linguistic code-switching evaluation.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 1803–1813.
  * Amato et al. (2017) Flora Amato, Stefano Marrone, Vincenzo Moscato, Gabriele Piantadosi, Antonio Picariello, and Carlo Sansone. 2017.  Chatbots meet ehealth: Automatizing healthcare.  In _WAIAH@ AI* IA_ , pages 40–49.
  * Amjad et al. (2020) Maaz Amjad, Grigori Sidorov, and Alisa Zhila. 2020.  Data augmentation using machine translation for fake news detection in the urdu language.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 2537–2542.
  * Asghar et al. (2014) Muhammad Zubair Asghar, Ahmad B RahmanUllah, Aurangzeb Khan, Shakeel Ahmad, and Irfan Ullah Nawaz. 2014.  Political miner: opinion extraction from user generated political reviews.  _Sci. Int (Lahore)_ , 26(1):385–389.
  * Badawy et al. (2018) Adam Badawy, Emilio Ferrara, and Kristina Lerman. 2018.  Analyzing the digital traces of political manipulation: The 2016 russian interference twitter campaign.  In _2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_ , pages 258–265. IEEE.
  * Bal et al. (2020) Rakesh Bal, Sayan Sinha, Swastika Dutta, Risabh Joshi, Sayan Ghosh, and Ritam Dutt. 2020.  Analysing the extent of misinformation in cancer related tweets.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 14, pages 924–928.
  * Baldauf (2004) Scott Baldauf. 2004.  A hindi-english jumble, spoken by 350 million.  _The Christian Science Monitor_ , 1123(1):3.
  * Bamman and Smith (2015) David Bamman and Noah A Smith. 2015.  Open extraction of fine-grained political statements.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 76–85.
  * Barclay et al. (2015) Francis P Barclay, C Pichandy, Anusha Venkat, and Sreedevi Sudhakaran. 2015.  India 2014: Facebook ‘like’as a predictor of election outcomes.  _Asian Journal of Political Science_ , 23(2):134–160.
  * Bates (2019) Mary Bates. 2019.  Health care chatbots are here to help.  _IEEE pulse_ , 10(3):12–14.
  * Baxter and Marcella (2012) Graeme Baxter and Rita Marcella. 2012.  Does scotland ‘like’this? social media use by political parties and candidates in scotland during the 2010 uk general election campaign.  _Libri_ , 62(2):109–124.
  * Bayer and Bárd (2020) Judit Bayer and Petra Bárd. 2020.  Hate speech and hate crime in the eu and the evaluation of online content regulation approaches.
  * Berk-Seligson (1986) Susan Berk-Seligson. 1986.  Linguistic constraints on intrasentential code-switching: A study of spanish/hebrew bilingualism.  _Language in society_ , pages 313–348.
  * Bibault et al. (2019) Jean-Emmanuel Bibault, Benjamin Chaix, Pierre Nectoux, Arthur Pienkowski, Arthur Guillemasé, and Benoît Brouard. 2019.  Healthcare ex machina: Are conversational agents ready for prime time in oncology?  _Clinical and translational radiation oncology_ , 16:55–59.
  * Bohra et al. (2018) Aditya Bohra, Deepanshu Vijay, Vinay Singh, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018.  A dataset of hindi-english code-mixed social media text for hate speech detection.  In _Proceedings of the second workshop on computational modeling of people’s opinions, personality, and emotions in social media_ , pages 36–41.
  * Bokamba (1989) Eyamba G Bokamba. 1989.  Are there syntactic constraints on code-mixing?  _World Englishes_ , 8(3):277–292.
  * Brennen et al. (2020) J Scott Brennen, Felix Simon, Philip N Howard, and Rasmus Kleis Nielsen. 2020.  Types, sources, and claims of covid-19 misinformation.  _Reuters Institute_ , 7:3–1.
  * Cameron et al. (2018) Gillian Cameron, David Cameron, Gavin Megaw, Raymond Bond, Maurice Mulvenna, Siobhan O’Neill, Cherie Armour, and Michael McTear. 2018.  Best practices for designing chatbots in mental healthcare–a case study on ihelpr.  In _Proceedings of the 32nd International BCS Human Computer Interaction Conference 32_ , pages 1–5.
  * Cantarella et al. (2020) Michele Cantarella, Nicolò Fraccaroli, and Roberto Volpe. 2020.  Does fake news affect voting behaviour?
  * Chaix et al. (2019) Benjamin Chaix, Jean-Emmanuel Bibault, Arthur Pienkowski, Guillaume Delamon, Arthur Guillemassé, Pierre Nectoux, and Benoît Brouard. 2019.  When chatbots meet patients: one-year prospective study of conversations between patients with breast cancer and a chatbot.  _Jmir Cancer_ , 5(1):e12856.
  * Chandu et al. (2019) Khyathi Chandu, Ekaterina Loginova, Vishal Gupta, Josef van Genabith, Günter Neumann, Manoj Chinnakotla, Eric Nyberg, and Alan W Black. 2019.  Code-mixed question answering challenge: Crowd-sourcing data and techniques.  In _Third Workshop on Computational Approaches to Linguistic Code-Switching_ , pages 29–38. Association for Computational Linguistics (ACL).
  * Choi et al. (2017) Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, and Jimeng Sun. 2017\.  Gram: graph-based attention model for healthcare representation learning.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 787–795.
  * Chopra et al. (2020) Shivang Chopra, Ramit Sawhney, Puneet Mathur, and Rajiv Ratn Shah. 2020.  Hindi-english hate speech detection: Author profiling, debiasing, and practical perspectives.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 386–393.
  * Conversation (2016) The Conversation. 2016.  Mobile messaging app map-february 2018.  [Online; accessed 30-Jan-2021].
  * csmonitor (2016) csmonitor. 2016.  How news sites’ online comments help fuel hatred.  [Online; accessed 30-Jan-2021].
  * Cui and Lee (2020) Limeng Cui and Dongwon Lee. 2020.  Coaid: Covid-19 healthcare misinformation dataset.  _arXiv preprint arXiv:2006.00885_.
  * Cui et al. (2020) Limeng Cui, Haeseung Seo, Maryam Tabar, Fenglong Ma, Suhang Wang, and Dongwon Lee. 2020.  Deterrent: Knowledge guided graph attention network for detecting healthcare misinformation.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 492–502.
  * Das and Gambäck (2015) Amitava Das and Björn Gambäck. 2015.  Code-mixing in social media text: the last language identification frontier?
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 11.
  * Dhar et al. (2018) Mrinal Dhar, Vaibhav Kumar, and Manish Shrivastava. 2018.  Enabling code-mixed translation: Parallel corpus creation and mt augmentation approach.  In _Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing_ , pages 131–140.
  * Divya et al. (2018) S Divya, V Indumathi, S Ishwarya, M Priyasankari, and S Kalpana Devi. 2018.  A self-diagnosis medical chatbot using artificial intelligence.  _Journal of Web Development and Web Designing_ , 3(1):1–7.
  * Futurist (2020) The Medical Futurist. 2020.  The top 12 health chatbots.
  * Garimella and Eckles (2020) Kiran Garimella and Dean Eckles. 2020.  Images and misinformation in political groups: Evidence from whatsapp in india.  _arXiv preprint arXiv:2005.09784_.
  * Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019.  Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1161–1166.
  * Golbeck et al. (2018) Jennifer Golbeck, Matthew Mauriello, Brooke Auxier, Keval H Bhanushali, Christopher Bonk, Mohamed Amine Bouzaghrane, Cody Buntain, Riya Chanduka, Paul Cheakalos, Jennine B Everett, et al. 2018.  Fake news vs satire: A dataset and analysis.  In _Proceedings of the 10th ACM Conference on Web Science_ , pages 17–21.
  * Grinberg et al. (2019) Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019.  Fake news on twitter during the 2016 us presidential election.  _Science_ , 363(6425):374–378.
  * Guess et al. (2019) Andrew Guess, Jonathan Nagler, and Joshua Tucker. 2019.  Less than you think: Prevalence and predictors of fake news dissemination on facebook.  _Science advances_ , 5(1):eaau4586.
  * Gull et al. (2016) Ratab Gull, Umar Shoaib, Saba Rasheed, Washma Abid, and Beenish Zahoor. 2016.  Pre processing of twitter’s data for opinion mining in political context.  _Procedia Computer Science_ , 96:1560–1570.
  * Gumperz and Hernandez (1969) John Joseph Gumperz and Edward Hernandez. 1969.  _Cognitive aspects of bilingual communication_.  28\. Institute of International Studies, University of California.
  * Gupta et al. (2020) Deepak Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2020.  A semi-supervised approach to generate the code-mixed text using pre-trained encoder and transfer learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 2267–2280.
  * Hassan et al. (2015) Naeemul Hassan, Bill Adair, James T Hamilton, Chengkai Li, Mark Tremayne, Jun Yang, and Cong Yu. 2015.  The quest to automate fact-checking.  In _Proceedings of the 2015 Computation+ Journalism Symposium_.
  * Hassan et al. (2017) Naeemul Hassan, Gensheng Zhang, Fatma Arslan, Josue Caraballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan, Minumol Joseph, Aaditya Kulkarni, Anil Kumar Nayak, et al. 2017.  Claimbuster: the first-ever end-to-end fact-checking system.  _Proceedings of the VLDB Endowment_ , 10(12):1945–1948.
  * Heredia et al. (2018) Brian Heredia, Joseph D Prusa, and Taghi M Khoshgoftaar. 2018.  The impact of malicious accounts on political tweet sentiment.  In _2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC)_ , pages 197–202. IEEE.
  * Hossain et al. (2020) Md Zobaer Hossain, Md Ashraful Rahman, Md Saiful Islam, and Sudipta Kar. 2020.  Banfakenews: A dataset for detecting fake news in bangla.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 2862–2871.
  * Hou et al. (2019) Rui Hou, Verónica Pérez-Rosas, Stacy Loeb, and Rada Mihalcea. 2019.  Towards automatic detection of misinformation in online medical videos.  In _2019 International conference on multimodal interaction_ , pages 235–243.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pages 4411–4421. PMLR.
  * Imran et al. (2013) Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz, and Patrick Meier. 2013.  Extracting information nuggets from disaster-related messages in social media.  In _Iscram_.
  * Joshi et al. (2016) Aditya Joshi, Pushpak Bhattacharyya, and Mark Carman. 2016.  Political issue extraction model: A novel hierarchical topic model that uses tweets by political and non-political authors.  In _Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 82–90.
  * Kakwani et al. (2020) Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020.  IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages.  In _Findings of EMNLP_.
  * Kamble and Joshi (2018) Satyajit Kamble and Aditya Joshi. 2018.  Hate speech detection from code-mixed hindi-english tweets using deep learning models.  In _15th International Conference on Natural Language Processing_ , page 155.
  * Kandias et al. (2013) Miltiadis Kandias, Lilian Mitrou, Vasilis Stavrou, and Dimitris Gritzalis. 2013\.  Youtube user and usage profiling: Stories of political horror and security success.  In _International Conference on E-Business and Telecommunications_ , pages 270–289. Springer.
  * Kannangara (2019) Sandeepa Harshanganie Kannangara. 2019.  Political opinion mining and analysis for social media.
  * Kanungo (2015) Neena Talwar Kanungo. 2015.  India’s digital poll battle: Political parties and social media in the 16th lok sabha elections.  _Studies in Indian Politics_ , 3(2):212–228.
  * Kar et al. (2020) Debanjana Kar, Mohit Bhardwaj, Suranjana Samanta, and Amar Prakash Azad. 2020.  No rumours please! a multi-indic-lingual approach for covid fake-tweet detection.  _arXiv preprint arXiv:2010.06906_.
  * Kaschesky et al. (2011) Michael Kaschesky, Pawel Sobkowicz, and Guillaume Bouchard. 2011.  Opinion mining in social media: modeling, simulating, and visualizing political opinion formation in the web.  In _Proceedings of the 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times_ , pages 317–326.
  * Khaldarova and Pantti (2016) Irina Khaldarova and Mervi Pantti. 2016.  Fake news.  _Journalism Practice_ , 10(7):891–901.
  * Khanuja et al. (2020) Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020.  Gluecos: An evaluation benchmark for code-switched nlp.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3575–3585.
  * Kouzy et al. (2020) Ramez Kouzy, Joseph Abi Jaoude, Afif Kraitem, Molly B El Alam, Basil Karam, Elio Adib, Jabra Zarka, Cindy Traboulsi, Elie W Akl, and Khalil Baddour. 2020\.  Coronavirus goes viral: quantifying the covid-19 misinformation epidemic on twitter.  _Cureus_ , 12(3).
  * Kowatsch et al. (2017) Tobias Kowatsch, Marcia Nißen, Chen-Hsuan Iris Shih, Dominik Rüegger, Dirk Volland, Andreas Filler, Florian Künzler, Filipe Barata, Dirk Büchter, Björn Brogle, et al. 2017.  Text-based healthcare chatbots supporting patient and health professional teams: preliminary results of a randomized controlled trial on childhood obesity.
  * Kumar et al. (2018) Ritesh Kumar, Aishwarya N Reganti, Akshit Bhatia, and Tushar Maheshwari. 2018.  Aggression-annotated corpus of hindi-english code-mixed data.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Kwanda and Lin (2020) Febbie Austina Kwanda and Trisha TC Lin. 2020.  Fake news practices in indonesian newsrooms during and after the palu earthquake: a hierarchy-of-influences approach.  _Information, Communication & Society_, pages 1–18.
  * Levenshtein (1966) Vladimir I Levenshtein. 1966.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , volume 10, pages 707–710.
  * Li et al. (2020) Yichuan Li, Bohan Jiang, Kai Shu, and Huan Liu. 2020.  Mm-covid: A multilingual and multidimensional data repository for combating covid-19 fake news.  _arXiv preprint arXiv:2011.04088_.
  * Lindberg et al. (1993) Donald AB Lindberg, Betsy L Humphreys, and Alexa T McCray. 1993.  The unified medical language system.  _Methods of information in medicine_ , 32(4):281.
  * van der Linden et al. (2020) Sander van der Linden, Costas Panagopoulos, and Jon Roozenbeek. 2020.  You are fake news: political bias in perceptions of fake news.  _Media, Culture & Society_, 42(3):460–470.
  * Ma et al. (2018) Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao. 2018\.  Kame: Knowledge-based attention model for diagnosis prediction in healthcare.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , pages 743–752.
  * Machado et al. (2019) Caio Machado, Beatriz Kira, Vidya Narayanan, Bence Kollanyi, and Philip Howard. 2019\.  A study of misinformation in whatsapp groups with a focus on the brazilian presidential elections.  In _Companion proceedings of the 2019 World Wide Web conference_ , pages 1013–1019.
  * Mandl et al. (2019) Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019.  Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages.  In _Proceedings of the 11th Forum for Information Retrieval Evaluation_ , pages 14–17.
  * Mathew et al. (2020) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2020.  Hatexplain: A benchmark dataset for explainable hate speech detection.  _arXiv preprint arXiv:2012.10289_.
  * Meel and Vishwakarma (2019) Priyanka Meel and Dinesh Kumar Vishwakarma. 2019.  Fake news, rumor, information pollution in social media and web: A contemporary survey of state-of-the-arts, challenges and opportunities.  _Expert Systems with Applications_ , page 112986.
  * Mena (2020) Paul Mena. 2020.  Cleaning up social media: The effect of warning labels on likelihood of sharing false news on facebook.  _Policy & internet_, 12(2):165–183.
  * Montesi (2020) Michela Montesi. 2020.  Understanding fake news during the covid-19 health crisis from the perspective of information behaviour: The case of spain.  _Journal of Librarianship and Information Science_ , page 0961000620949653\.
  * Myslinski (2015) Lucas J Myslinski. 2015.  Method of and system for validating a fact checking system.  US Patent 9,087,048.
  * Nakamura et al. (2020) Kai Nakamura, Sharon Levy, and William Yang Wang. 2020.  Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 6149–6157.
  * Neudert et al. (2017) L Neudert, Bence Kollanyi, and Philip N Howard. 2017.  Junk news and bots during the german parliamentary election: What are german voters sharing over twitter?
  * Nockleby (2000) John T Nockleby. 2000.  Hate speech.  _Encyclopedia of the American constitution_ , 3(2):1277–1279.
  * Nørregaard et al. (2019) Jeppe Nørregaard, Benjamin D Horne, and Sibel Adalı. 2019.  Nela-gt-2018: A large multi-labelled news dataset for the study of misinformation in news articles.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 13, pages 630–638.
  * Oliveira et al. (2017) Jonice Oliveira, Carla Delgado, and Ana Carolina Assaife. 2017.  A recommendation approach for consuming linked open data.  _Expert Systems with Applications_ , 72:407–420.
  * Oshikawa et al. (2020) Ray Oshikawa, Jing Qian, and William Yang Wang. 2020.  A survey on natural language processing for fake news detection.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 6086–6093.
  * Palanica et al. (2019) Adam Palanica, Peter Flaschner, Anirudh Thommandram, Michael Li, and Yan Fossat. 2019.  Physicians’ perceptions of chatbots in health care: cross-sectional web-based survey.  _Journal of medical Internet research_ , 21(4):e12887.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting on association for computational linguistics_ , pages 311–318. Association for Computational Linguistics.
  * Patra et al. (2015) Braja Gopal Patra, Dipankar Das, Amitava Das, and Rajendra Prasath. 2015.  Shared task on sentiment analysis in indian languages (sail) tweets-an overview.  In _International Conference on Mining Intelligence and Knowledge Exploration_ , pages 650–655. Springer.
  * Patwa et al. (2020a) Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, PYKL Srinivas, Björn Gambäck, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. 2020a.  Semeval-2020 task 9: Overview of sentiment analysis of code-mixed tweets.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 774–790.
  * Patwa et al. (2020b) Parth Patwa, Shivam Sharma, Srinivas PYKL, Vineeth Guptha, Gitanjali Kumari, Md Shad Akhtar, Asif Ekbal, Amitava Das, and Tanmoy Chakraborty. 2020b.  Fighting an infodemic: Covid-19 fake news dataset.  _arXiv preprint arXiv:2011.03327_.
  * Pennycook et al. (2020) Gordon Pennycook, Jonathon McPhetres, Yunhao Zhang, Jackson G Lu, and David G Rand. 2020.  Fighting covid-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention.  _Psychological science_ , 31(7):770–780.
  * Phillips and Young (2009) David Phillips and Philip Young. 2009.  _Online public relations: A practical guide to developing an online strategy in the world of social media_.  Kogan Page Publishers.
  * POPLACK (1980) S POPLACK. 1980.  Sometimes i’ll start a sentence in spanish y termino en espanol: Toward a typology of code-switching.  _Linguistics. An Interdisciplinary Journal of the Language Sciences La Haye_ , 18(7-8):581–618.
  * Prasad and Francescutti (2017) Abhaya S. Prasad and Louis Hugo Francescutti. 2017.  Natural disasters.  In Stella R. Quah, editor, _International Encyclopedia of Public Health (Second Edition)_ , second edition edition, pages 215 – 222. Academic Press, Oxford.
  * Pratapa et al. (2018) Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018.  Language modeling for code-mixing: The role of linguistic theory based synthetic data.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1543–1553.
  * Qazi et al. (2020) Umair Qazi, Muhammad Imran, and Ferda Ofli. 2020.  Geocov19: a dataset of hundreds of millions of multilingual covid-19 tweets with location information.  _SIGSPATIAL Special_ , 12(1):6–15.
  * Rashkin et al. (2017) Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017\.  Truth of varying shades: Analyzing language in fake news and political fact-checking.  In _Proceedings of the 2017 conference on empirical methods in natural language processing_ , pages 2931–2937.
  * Reis et al. (2019) Julio CS Reis, André Correia, Fabrício Murai, Adriano Veloso, and Fabrício Benevenuto. 2019.  Explainable machine learning for fake news detection.  In _Proceedings of the 10th ACM Conference on Web Science_ , pages 17–26.
  * Resende et al. (2019) Gustavo Resende, Philipe Melo, Julio CS Reis, Marisa Vasconcelos, Jussara M Almeida, and Fabrício Benevenuto. 2019.  Analyzing textual (mis) information shared in whatsapp groups.  In _Proceedings of the 10th ACM Conference on Web Science_ , pages 225–234.
  * Rudra et al. (2016) Koustav Rudra, Siddhartha Banerjee, Niloy Ganguly, Pawan Goyal, Muhammad Imran, and Prasenjit Mitra. 2016.  Summarizing situational tweets in crisis scenario.  In _Proceedings of the 27th ACM Conference on Hypertext and Social Media_ , pages 137–147.
  * Rudra et al. (2015) Koustav Rudra, Subham Ghosh, Niloy Ganguly, Pawan Goyal, and Saptarshi Ghosh. 2015\.  Extracting situational information from microblogs during disaster events: a classification-summarization approach.  In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_ , pages 583–592.
  * Rudra et al. (2018a) Koustav Rudra, Pawan Goyal, Niloy Ganguly, Prasenjit Mitra, and Muhammad Imran. 2018a.  Identifying sub-events and summarizing disaster-related information from microblogs.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 265–274.
  * Rudra et al. (2018b) Koustav Rudra, Ashish Sharma, Niloy Ganguly, and Saptarshi Ghosh. 2018b.  Characterizing and countering communal microblogs during disaster events.  _IEEE Transactions on Computational Social Systems_ , 5(2):403–417.
  * Sadeghi and Lehmann (2019) Afshin Sadeghi and Jens Lehmann. 2019.  Linking physicians to medical research results via knowledge graph embeddings and twitter.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 622–630. Springer.
  * Salem et al. (2019) Fatima K Abu Salem, Roaa Al Feel, Shady Elbassuoni, Mohamad Jaber, and May Farah. 2019.  Fa-kes: A fake news dataset around the syrian war.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 13, pages 573–582.
  * Santosh and Aravind (2019) TYSS Santosh and KVS Aravind. 2019.  Hate speech detection in hindi-english code-mixed social media text.  In _Proceedings of the ACM India Joint International Conference on Data Science and Management of Data_ , pages 310–313.
  * Schmidt and Wiegand (2017) Anna Schmidt and Michael Wiegand. 2017.  A survey on hate speech detection using natural language processing.  In _Proceedings of the Fifth International workshop on natural language processing for social media_ , pages 1–10.
  * Schwarz et al. (2020) Stephane Schwarz, Antônio Theóphilo, and Anderson Rocha. 2020.  Emet: Embeddings from multilingual-encoder transformer for fake news detection.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 2777–2781. IEEE.
  * Shahi and Nandini (2020) Gautam Kishore Shahi and Durgesh Nandini. 2020.  Fakecovid–a multilingual cross-domain fact check news dataset for covid-19.  _arXiv preprint arXiv:2006.11343_.
  * Shin et al. (2017) Jieun Shin, Lian Jian, Kevin Driscoll, and François Bar. 2017.  Political rumoring on twitter during the 2012 us presidential election: Rumor diffusion and correction.  _new media & society_, 19(8):1214–1235.
  * Shu et al. (2019) Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019.  defend: Explainable fake news detection.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 395–405.
  * Shu et al. (2017) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017.  Fake news detection on social media: A data mining perspective.  _ACM SIGKDD explorations newsletter_ , 19(1):22–36.
  * Singh (2020) Vijaita Singh. 2020.  Centre plans law on online hate speech.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A study of translation edit rate with targeted human annotation.  In _Proceedings of association for machine translation in the Americas_ , volume 200. Cambridge, MA.
  * Srivastava and Singh (2020a) Vivek Srivastava and Mayank Singh. 2020a.  Iit gandhinagar at semeval-2020 task 9: Code-mixed sentiment classification using candidate sentence generation and selection.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 1259–1264.
  * Srivastava and Singh (2020b) Vivek Srivastava and Mayank Singh. 2020b.  PHINC: A parallel Hinglish social media code-mixed corpus for machine translation.  In _Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)_ , pages 41–49, Online. Association for Computational Linguistics.
  * Sujeet K Sharma and Chandwani (2020) Jang Bahadur Singh Sujeet K Sharma and Rajesh Chandwani. 2020.  Rumors vs fake news: How to address misinformation in crisis?
  * Tasnim et al. (2020) Samia Tasnim, Md Mahbub Hossain, and Hoimonty Mazumder. 2020.  Impact of rumors and misinformation on covid-19 in social media.  _Journal of preventive medicine and public health_ , 53(3):171–174.
  * Thorne and Vlachos (2018) James Thorne and Andreas Vlachos. 2018.  Automated fact checking: Task formulations, methods and future directions.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3346–3359.
  * Ur Rahman Khilji et al. (2020) A. F. Ur Rahman Khilji, S. R. Laskar, P. Pakray, R. A. Kadir, M. S. Lydia, and S. Bandyopadhyay. 2020.  Healfavor: Dataset and a prototype system for healthcare chatbot.  In _2020 International Conference on Data Science, Artificial Intelligence, and Business Analytics (DATABIA)_ , pages 1–4.
  * Valdés (2007) G Valdés. 2007.  Multilingualism. linguistic society of america.  _On WWW at http://www. lsadc. org/info/ling-fields-multi. cfm. Accessed_ , 9.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pages 6000–6010.
  * Vyas et al. (2014) Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014.  Pos tagging of english-hindi code-mixed social media content.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 974–979.
  * Wang (2017a) William Yang Wang. 2017a.  “liar, liar pants on fire”: A new benchmark dataset for fake news detection.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 422–426, Vancouver, Canada. Association for Computational Linguistics.
  * Wang (2017b) William Yang Wang. 2017b.  “liar, liar pants on fire”: A new benchmark dataset for fake news detection.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 422–426.
  * Wharton Business Daily (2020) Podcasts Wharton Business Daily. 2020.  How social media is shaping political campaigns.
  * Wieling et al. (2018) Martijn Wieling, Josine Rawee, and Gertjan van Noord. 2018.  Reproducibility in computational linguistics: Are we willing to share?  _Computational Linguistics_ , 44(4):641–649.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020.  mt5: A massively multilingual pre-trained text-to-text transformer.  _arXiv preprint arXiv:2010.11934_.
  * Yang et al. (2019) Fan Yang, Shiva K Pentyala, Sina Mohseni, Mengnan Du, Hao Yuan, Rhema Linder, Eric D Ragan, Shuiwang Ji, and Xia Hu. 2019.  Xfake: explainable fake news detector with visualizations.  In _The World Wide Web Conference_ , pages 3600–3604.
  * Zhao et al. (2021) Yuehua Zhao, Jingwei Da, and Jiaqi Yan. 2021.  Detecting health misinformation in online health communities: Incorporating behavioral features into machine learning based approaches.  _Information Processing & Management_, 58(1):102390.