# Uni-Parser: Unified Semantic Parser for Question Answering on Knowledge Base
and Database

Ye Liu1, Semih Yavuz1, Rui Meng1, Dragomir Radev2, Caiming Xiong1, Yingbo
Zhou1
1 Salesforce Research, 2 Yale University
{yeliu, syavuz, ruimeng, yingbo.zhou<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Parsing natural language questions into executable logical forms is a useful
and interpretable way to perform question answering on structured data such as
knowledge bases (KB) or databases (DB). However, existing approaches on
semantic parsing cannot adapt to both modalities, as they suffer from the
exponential growth of the logical form candidates and can hardly generalize to
unseen data. In this work, we propose Uni-Parser, a unified semantic parser
for question answering (QA) on both KB and DB. We introduce the primitive
(relation and entity in KB, and table name, column name and cell value in DB)
as an essential element in our framework. The number of primitives grows
linearly with the number of retrieved relations in KB and DB, preventing us
from dealing with exponential logic form candidates. We leverage the generator
to predict final logical forms by altering and composing top-ranked primitives
with different operations (e.g. select, where, count). With sufficiently
pruned search space by a contrastive primitive ranker, the generator is
empowered to capture the composition of primitives enhancing its
generalization ability. We achieve competitive results on multiple KB and DB
QA benchmarks more efficiently, especially in the compositional and zero-shot
settings.

##  1 Introduction

With the recent advances in deep neural networks, question answering (QA)
systems enable users to interact with massive data using queries in natural
language. However, it remains challenging to assess structured data, such as
knowledge bases and databases. Semantic parsing is a core step of question
answering for structured data. The goal is to convert a natural language
question to an executable logical form Berant et al. (2013); Yih et al.
(2015), e.g., SQL for databases and S-expression for knowledge bases.

To improve the accuracy and faithfulness in execution of semantic parsing,
recent KBQA studies propose to generate logical form candidates by enumerating
and selecting the best logical form by ranking Berant and Liang (2014); Yih et
al. (2015); Sun et al. (2020); Ye et al. (2021). However, the number of
logical form candidates may grow exponentially with the increase of reasoning
depth for complex questions. Thus this approach can suffer from poor runtime
performance due to the time-consuming logical form enumeration Gu et al.
(2021) and inefficient candidate ranking Ye et al. (2021). For example, given
an entity in a KB, we collect its logical form candidates by enumerating paths
up to two hops. If the first hop with respect to the given entity contains N
relations and the second hop contains M relations, the enumeration would
result in $N\times M$ logical forms. As KBs typically contain massive
structured knowledge, an entity can have hundreds of linked relations
Bollacker et al. (2008). Moreover, for complex questions, which require
combining logic or aggregation operations, such as COUNT, ARGMIN, or ARGMAX,
into the logical form, the situation could be even worse. Therefore,
traditional enumeration methods may fail in cases requiring complicated
reasoning (e.g., involving large amounts of entities and long reasoning
chains). The situation becomes more severe and prohibitive if one wants to
apply the enumeration method to other structured data with dense connections
between entities such as in databases. Designing a unified semantic parsing
method for various modalities of structured data have significant theoretical
and practical value, yet it is still an understudied topic.

To avoid the problem of exponential growth in the logical form enumeration, we
consider logical forms composed of two types of elements – primitives and
operations. Primitives are defined by the schema of the structured data source
and operations are a set of grammars associated with primitives. For example,
in the context of knowledge bases, primitives are defined as relations and
entities in the knowledge graph. Whereas in databases, primitives are
presented as tables, columns, and cells. Through this formulation, the number
of candidates can be greatly reduced, going down from $N\times M$ to $N+M$.

In this study, we present Uni-Parser, a unified semantic parser for question
answering on both knowledge bases (KBs) and databases (DBs). Our model follows
the framework of Enumeration-Ranker-Generator proposed in RnG-KBQA Ye et al.
(2021). We first enumerate possible question-relevant primitives of a given KB
or DB. Then a cross-encoder ranker is utilized to select the best candidates
with contrastive learning Chang et al. (2020), and it is further enhanced by a
special hard negative sampling strategy. After getting the top-k ranked
primitives for each hop, we filter out the high-order primitives that cannot
be reached from the KB or do not exist in the DB through selected low-order
primitives. Next, we introduce a generator that consumes both the question and
filtered top-k primitives with predicted operations to compose the final
logical form. Starting from primitives rather than logical forms, our
generator needs to understand the semantic meaning of each primitive to
compose them into the logical form.

Our contributions can be summarized as follows:
$\bullet$ We propose a unified semantic parser working for both KB and DB
question answering.
$\bullet$ We enumerate primitives rather than logical forms, which greatly
reduces the search space and makes candidate generation and ranking more
efficient and scalable.
$\bullet$ The composition of logical forms from primitives and operations is
postponed to the generation phase. Thus, the generator is required to learn
the compositional relations among primitives. This leads to a more generalized
model that can work on complex logical forms and generalize to questions
involving unseen schema.
$\bullet$ Extensive empirical results on four KB and DB QA datasets
demonstrate the effectiveness, flexibility and scalability of our unified
framework.

##  2 Problem Formulation

Given a structured data source $D$ and a question $X$ in natural language, a
semantic parser model is tasked to generate the corresponding logical form
$Y$. Specifically, we illustrate the details depending on the type of $D$ as
follows: (1) Knowledge Base: Data is stored in the form of subject-relation-
object $(s,r,o)$, where $s$ is an entity, $r$ is a relation and $o$ is an
entity or a literal (e.g., integer values, data, etc.). We use S-expressions
Gu et al. (2021) to represent logical forms for KB. S-expression is used to
query a KB with the entity type, and the operations on the KB are treated as
operations on a set of entities. This formulation greatly reduces the number
of operations in traditional lambda DCS Liang (2013). (2) Database: A DB QA
dataset typically consists of multiple tables $T=\\{t_{1},\cdots,t_{N}\\}$.
Each table $T_{i}$ contains $M$ columns: $C=\\{c_{1},\cdots,c_{M}\\}$. Each
column includes multiple cell values $V=\\{v_{1},\cdots,v_{L}\\}$, where $L$
is the number of rows. For tabular data, SQL is used to represent logical
forms Yu et al. (2018).

Logical forms can be decomposed into the primitives and operations, both of
which are defined by the schema of the structured data. We define primitives
as the atomic elements that are entities themselves or can be used to navigate
to entities. Operations are a set of grammars to associate primitives. Thus, a
logical form can be decomposed to primitives and operations. We list specific
primitives and operations considered in this study in Table 1.

KB | DB
---|---
Primitive on S-expression | Primitive on SQL
Relation, Entity | Table Name, Column Name, Cell Value
Operation on S-expression | Operation on SQL
Logical Operation
AND, JOIN, R | SELECT, WHERE, ORDER/GROUP BY
Aggregation Operation
COUNT, ARGMAX, ARGMIN | AVG, COUNT, MAX, MIN, SUM
Conditional Operation
<, <=, >, >= | between, =, >, <, >=, <=,
| !=, in, like, is, exists, not in,
| not like, not between, is not
Conjunction Operation
| and, or, except, intersect, union

Table 1: Primitives and Operations in KB and DB logical form.

##  3 Methodology

Figure 1: Given the question and its knowledge source, the primitive
enumeration process produces different categories of primitives for KB and DB.

Our model first obtains the relevant primitives from the given structured data
and question, and sets them into different categories (Sec 3.1). Then a ranker
filters out the irrelevant primitives (Sec 3.2) and provides the top-ranked
primitives to the generator to produce the final logical form (Sec 3.3). We’ll
first explain how to extract the primitive candidates from KB or DB based on
the question.

###  3.1 Primitive Enumeration

Rather than enumerating all possible logical forms as in RnG-KBQA Ye et al.
(2021), here we only enumerate primitives that are relevant to the question.

In cases of knowledge bases, we start by detecting the entity mentioned in the
question with the help of an out-of-the-box NER system and then run fuzzy
matching Lin et al. (2020) with the entity names in the knowledge base to
identify relevant entities. This technique is also used in Gu et al. (2021);
Chen et al. (2021). To alleviate the issue of entity disambiguation, we follow
Ye et al. (2021) to use a ranker model to select entity candidates based on
the similarity between the question and the one-hop in/out relations of the
entity. Nevertheless, since most questions contain two-hop reasoning in KBs,
we also extract two-hop paths associated with question entities as a related
sub-graph. We define <|firsthop|> category primitives as the entities with
first-hop relation and likewise for <|secondhop|> category primitives
(examples are shown at the top of Figure 1).

As for databases, we consider two formats of primitives. The first category is
<|tb_cl|>, denoting the format table$\\_$name.column$\\_$name. The second one
is <|tb_cl_vl|>, representing the format table$\\_$name.column$\\_$name <op>
cell$\\_$value. The <op> represents a conditional operation as shown in Table
1. To enumerate the first category of primitives, we can simply use all table
names together with their column names. However, for the second category,
including cell values in enumeration will lead to a vast amount of candidates.
For instance, on the Spider dataset Yu et al. (2018), if we treat every cell
value as a candidate, we can get up to 263K candidates for one question. To
address this issue, following Lin et al. (2020), we perform a fuzzy string
match between question $X$ and the cell value $V$ under each column name $C$,
and pair the matched value with its corresponding column name. One shortcoming
of using string match is, that we can only obtain coverage of 15$\%$ for the
Spider dataset. This is because many cell values are of numeric type, where
string match fails to detect. For example, for the question "How many heads of
departments are older than 56?", there is no cell value that matches 56.
Therefore, if a question contains numbers, we pair them with all column names
in the table.

Figure 2: An illustration of Uni-Parser ranker.

###  3.2 Primitive Ranker

Our ranker model learns to filter out irrelevant primitives by measuring the
similarity between questions and primitive candidates. We utilize the cross-
encoder architecture Chang et al. (2020) for ranker, which has shown to be
more performant than bi-encoder architectures Thakur et al. (2020); Lei et al.
(2022). As shown in Figure 2, we use a special category token as a prompt in
the input to differentiate primitives in the input. Note that all primitives
from different categories share the same ranker. Specifically, given a
question $X$ and a primitive $p$ with a category token $p_{c}$, we use a BERT-
based encoder that takes as input the concatenation of their vector
representations, and outputs a logit representing the similarity between the
primitive and the question:

$\displaystyle s(X,p,p_{c})=\operatorname{FFN}(\psi_{\theta}(p_{c}\oplus
X\oplus p))$ (1)

where $\oplus$ denotes a concatenation operation. $\psi_{\theta}$ denotes the
[CLS] representation of the concatenated input after BERT embedding;
$\operatorname{FNN}$ is a projection layer reducing the representation to a
scalar similarity score. $p_{c}$ is the special token to distinguish the
category of the primitive (in KB, $p_{c}\in\\{$<|firsthop|>,
<|secondhop|>$\\}$ and in DB, $p_{c}\in\\{$<|tb$\\_$cl|>,
<|tb$\\_$cl$\\_$vl|>$\\}$).

The ranker is optimized to minimize the contrastive loss:

$\displaystyle\mathcal{L}_{\theta}\left(X,\leavevmode\nobreak\
p^{+},\mathcal{P}^{-},p_{c}\right)=-\frac{e^{s\left(X,\leavevmode\nobreak\
p^{+},\leavevmode\nobreak\
p_{c}\right)}}{\sum_{p\in\left\\{p^{+}\right\\}\cup\mathcal{P}^{-}}e^{s\left(X,\leavevmode\nobreak\
p,\leavevmode\nobreak\ p_{c}\right)}}$ (2)

where $p^{+}$ is the positive primitive extracted from the gold logical form
and $\mathcal{P}^{-}$ is the set of negative primitives from the same category
$p_{c}$.

####  3.2.1 Negative Sampling

Since a large number of negative primitive candidates can be paired with a
positive example, it is necessary to apply negative sampling. A
straightforward way for this is random sampling, however it may suffer from
the domination of uninformative negatives Xiong et al. (2020). To this end, we
design a strategy to sample hard negative candidates for training the ranker
Liu et al. (2021b). In cases of KBs, the number of second hop relations can
grow exponentially compared to the first hop. Thus the hard negative
candidates of the second hop can only be sampled from the primitives connected
to the ground truth first hop. In cases of DBs, for <|tb$\\_$cl|> category
primitives, we treat those having the same table name with ground truth but
different column names as the hard negatives. And for the <|tb$\\_$cl$\\_$vl|>
category, we treat candidates with the same table and column name with ground
truth, but having a different cell value as the hard negatives. Moreover, the
bootstrap negative sampling strategy is leveraged; that is, the model is
trained recursively using the false positive candidates generated from the
last training epoch.

####  3.2.2 Primitive Candidates Filtering

In KBs, the top ranked first hop primitives and second hop primitives can be
formed into two-hop paths by combining one first hop primitive with each of
the second hop. However, the resulting paths may not exist in the KB. To
provide valid primitive candidates to the generator, we filter out the second
hop primitives that cannot be reached from any of the first hop primitives.

Figure 3: An illustration of Uni-Parser generator.

###  3.3 Logical Form Generation by Composing Primitives

In the final stage, we need to employ a generator model to predict the target
logical form by composing the top primitive candidates provided by the ranker.
We use a T5 model Raffel et al. (2020) as the basis of our logical form
generator, as it demonstrates strong performance on various text generation
tasks. We construct the inputs by concatenating the question and the top-k
primitive candidates. As shown in Figure 3, the input for KBs is formatted as:
[X; <|second_hop|> primitives ; <|first_hop|> primitives]. As for DBs, its
input is formatted as: [X; |table_name1| column_name1, column_name2 <op> value
$\cdots$ |table_name|2 $\cdots$. We train the model by teacher forcing – the
target logical form is generated token by token and the model is optimized
with the loss of cross-entropy. At inference time, we use beam-search to
decode top-k target logical forms in an autoregressive manner.

During training, it is often the case that the ranker performs well and
predicts the gold primitive at the top place, but this positional information
can be misused by the generator. If the generator is biased by the ranking of
primitives, it may generalize poorly to unseen data. In order to encourage the
generator to focus on the semantic meaning of each primitive rather than their
positions, we shuffle the order of primitives in the input during training.
According to our experiments, we find that the generator can benefit from this
shuffle augmentation and perform robustly against the noise in the ranked
primitives.

Our generator learns to generate logical forms by understanding the meaning of
its elements – primitives and operations – and composing them. Compared to
RnG-KBQA Ye et al. (2021), whose generator predicts logical forms based on a
list of logical form candidates, the compositionality on the basis of
primitives and operations can make our model likely to generalize better on
unseen structured data.

##  4 Experiments

###  4.1 Dataset and Evaluation

KBQA (1) GRAILQA Gu et al. (2021) contains 64,331 questions and carefully
splits the data to evaluate three levels of generalization in the task of
KBQA, including i.i.d. setting, compositional generalization to unseen
composition of KB schema, and zero-shot generalization to unseen KB schema.
The fraction of each setting in the test set is 25$\%$, 25$\%$, and 50$\%$,
respectively. (2) WebQSP Yih et al. (2016) is a dataset that evaluates KBQA
approaches in i.i.d. setting. It contains 4,937 questions and requires
reasoning chains with up to 2 hops. Similar to Ye et al. (2021), we randomly
sample 200 examples from the training set for validation.
DBQA (1) Spider Yu et al. (2018) is a multi-table text-to-SQL dataset, which
contains 10,181 questions and 5,693 complex SQL queries on 200 databases.
There is no overlap between train/dev/test databases. (2) WikiSQL Zhong et al.
(2017) is a single-table text-to-SQL dataset, which contains 80,654 questions
and SQL queries distributed across 24,241 tables from Wikipedia. 49.6$\%$ of
its dev tables and 45.1$\%$ of its test tables are not in the training set.
Therefore, both datasets require models to generalize to the unseen schema
composition (compositional generalization) and unseen schema (zero-shot
generalization).

Evaluation Metrics. We use their official evaluation script for each dataset
with two metrics to measure logical form of program exact match accuracy (EM)
and answer accuracy (F1).

###  4.2 Results on KBQA

We first test our approach on KBQA with the GrailQA and WebQSP datasets.

| Overall | I.I.D | Compositional | Zero-Shot
---|---|---|---|---
| EM | F1 | EM | F1 | EM | F1 | EM | F1
Bert Ranking(Test) Gu et al. (2021) | 50.6 | 58.0 | 59.9 | 67.0 | 45.5 | 53.9 | 48.6 | 55.7
ReTrack (Test) Chen et al. (2021) | 58.1 | 65.3 | 84.4 | 87.5 | 61.5 | 70.9 | 44.6 | 52.5
UnifiedSKG (Test) Xie et al. (2022) | 62.4 | - | - | - | - | - | - | -
RNG-KBQA (Test) Ye et al. (2021) | 68.8 | 74.4 | 86.2 | 89.0 | 63.8 | 71.2 | 63.0 | 69.2
UnifiedSKG (Dev) Xie et al. (2022) | 60.0 | - | - | - | - | - | - | -
RNG-KBQA (Dev) | 70.7 | 75.7 | 86.4 | 88.6 | 61.7 | 68.3 | 67.4 | 73.1
Uni-Parser (Dev) | 70.8 | 76.5 | 85.7 | 88.3 | 62.8 | 71.4 | 67.7 | 73.4
Uni-Parser (Test) | 69.5 | 74.6 | 85.5 | 88.5 | 65.1 | 71.1 | 64.0 | 69.8

Table 2: Exact match (EM) and F1 scores on test/dev split of the GRAILQA. The
numbers of the baselines are taken from leaderboard and their research works.
The reported models are based on BERT-base model for ranker and T5-base for
generator. Best results among dev are bolded and the results of test better
than dev are underlined.

| EM | F1
---|---|---
Topic Units Lan et al. (2019) | - | 67.9
STAGG Yih et al. (2015) | 63.9 | 71.7
QGG Lan and Jiang (2020) | - | 74.0
CBR Das et al. (2021) | 70.0 | 72.8
ReTrack Chen et al. (2021) | - | 71.0
RNG-KBQA Ye et al. (2021) | 71.1 | 75.6
ArcaneQA Gu and Su (2022) | - | 75.3
Uni-Parser | 71.4 | 75.8

Table 3: Exact match (EM) and F1 scores on the test split of WebQSP. The
reported models are based on BERT-base model for ranker and T5-base for
generator.

####  4.2.1 Implementation Details

For GrailQA and WebQSP, we use the entity linking results provided by Ye et
al. (2021). After identifying a set of entities, we extract the primitives
within 2 hops from the question entities. We initiate the primitive ranker
using BERT-base-uncased. For each primitive category, 96 negative candidates
are sampled. We trained the ranker for 3 epochs using a learning rate of 1e-5
and a batch size of 8. Bootstrap sampling is applied after every epoch. It is
also noteworthy that we perform teacher-forcing when training the ranker,
i.e., we use ground truth entity linking for enumerating training candidates.
We base our generation model on T5-base Raffel et al. (2020). We use top-10
primitives from each category returned by the ranker and finetune the T5
generation model for 10 epochs using a learning rate of 3e-5 and a batch size
of 8. A vanilla T5 generation model is used without syntactic constraints,
which does not guarantee the syntactic correctness nor executability of the
produced logical forms. Therefore, we use an execution-augmented inference
procedure, which is commonly used in previous semantic parsing related work
Devlin et al. (2017); Ye et al. (2020). We first decode top-k logical forms
using beam search and then execute each logical form until finding one that
yields a valid (non-empty) answer. In case none of the top-k logical forms is
valid, the top-ranked primitives obtained using the ranker is returned. The
rule-based method is used to formulate the final logical form, which is
guaranteed to be executable. This inference schema can ensure finding one
valid logical form for each problem.

####  4.2.2 Overall Evaluation

Table 2 and 3 summarize the results on GrailQA and WebQSP, respectively. Our
approach achieves the highest overall performance among all approaches.
Compared with the methods enumerating logical forms like Bert Ranking and RnG-
KBQA, our approach achieves better performance in compositional and zero-shot
settings. Especially, we get 3.1$\%$ improvement over baselines on F1 on dev
set and 1.3$\%$ improvement over baselines on EM on test set in the
compositional setting. This matches our expectation that our generator learns
the composition of the primitives. RnG-KBQA enumerates logical forms rather
than primitives. Therefore its generation module acts more like an auxiliary
input to complement the enumerated logical forms. However, the rationale
behind our generation module is to compose logical forms with basic semantic
units(primitives). Thus, with the sense of primitive composition, our model is
more capable of dealing with unseen composition than RnG-KBQA. For i.i.d.
setting, our approach underperforms RnG-KBQA by 0.7$\%$ on EM and 0.5$\%$ on
F1. We speculate that our model needs to understand whether a question implies
a one or two hops reasoning on the KB, which is a difficult task. But the
generator of RnG-KBQA already sees logical form candidates in the input, it
does not need to deal with this problem.

####  4.2.3 Efficiency Analysis

We compare the running time of the Uni-Parser and ranking-based model. To make
a fair comparison, we time the process from the enumeration step to the
logical form generation, using 1,000 randomly sampled questions on GrailQA
datasets. We also report the average running time per question on an A100 GPU.
Our model uses 19.4s, which is considerably faster than BERT+Ranking (76.3s)
and RnG-KBQA (53.5s). Unlike logical form-based models, our model doesn’t need
to enumerate a large amount of logical forms. Instead, only a small number of
relevant primitives are considered, which leads to faster tokenization and
efficient ranking.

| EM | F1
---|---|---
Global-GNN Bogin et al. (2019) | 52.7 | -
EditSQL Zhang et al. (2019b) | 57.6 | -
T5-Base Scholak et al. (2021) | 57.2 | 57.9
UnifiedSKG(T5-Base) Xie et al. (2022) | 58.1 | -
Uni-Parser(T5-Base) | 61.2 | 62.8
RAT-SQL Wang et al. (2019) | 69.7 | -
BRIDGE(Large) Lin et al. (2020) | 70.0 | 68.0
T5-3B* Scholak et al. (2021) | 71.5 | 74.4
UnifiedSKG(T5-3B) Xie et al. (2022) | 71.7 | -
Uni-Parser(T5-3B) | 71.2 | 71.8

Table 4: Exact match (EM) and F1 scores on the dev split of Spider. The upper
block shows the comparison of the small pre-trained models, and the lower
block shows the comparison of the large pre-trained models. Our model trains
T5-3B model 100 epochs while T5-3B * trains 3072 epochs
111https://github.com/ServiceNow/picard/blob/main/configs/train.json.

| EM | F1
---|---|---
SQLova Hwang et al. (2019) | 80.7 | 86.2
X-SQL He et al. (2019) | 83.3 | 88.7
IE-SQL Ma et al. (2020) | 84.6 | 88.8
NL2SQL Guo and Gao (2019) | 83.7 | 89.2
HydraNet Lyu et al. (2020) | 83.8 | 89.2
BRIDGE(Large) Lin et al. (2020) | 85.7 | 91.1
TAPEX Liu et al. (2021a) | - | 89.5
Uni-Parser(T5-Base) | 85.8 | 91.3
Uni-Parser(T5-Large) | 86.9 | 92.1

Table 5: Exact match (EM) and F1 scores on the test split of WikiSQL.

###  4.3 Results on DBQA

We also evaluate our approach to the DBQA task with Spider and WikiSQL
datasets.

####  4.3.1 Implementation Details

To construct the <|tb$\\_$cl$\\_$vl|> category primitives mentioned in Section
3.1, we find the relevant cell values related to the question. Given a
question and DB, we compute the string matching between the arbitrary length
of phrase in question and the list of cell values under each column of all
tables. We followed Lin et al. (2020) to use a fuzzy matching algorithm to
match a question to a possible cell value mentioned in the DB. We also detect
the number value in the question and form all column names with the value as
the primitives. We find that the column name in the WikiSQL dataset is vague,
like “No.”, “Pick #”, and “Rank”, so we use the cell value to supplement the
meaning of the column name. We use the matching cell value to locate the row
and match the column name with the cell value in the same row.

We initiate the primitive ranker using BERT-base-uncased. We sample 48
negative candidates for each primitive category. We trained the ranker for 10
epochs using a learning rate of 1e-5 and a batch size of 8. Bootstrap hard
negative sampling is conducted after every two epochs. We also use ground
truth entity linking for enumerating training candidates. For the generator,
we trained it using T5-base and 3B on Spider datasets. We use top-15
<|tb$\\_$cl|> category primitives and top-5 <|tb$\\_$cl$\\_$vl|> category
primitives returned by the ranker and finetune the T5-base model for 200
epochs using a learning rate of 5e-5 and a batch size of 64. For the T5-3B
model, we run it on 16 A100 GPUs with 100 epochs using a batch size of 1024.
And on the WikiSQL dataset, we use T5-base and T5-large, and use top-5
<|tb$\\_$cl|> category primitives and top-3 <|tb$\\_$cl$\\_$vl|> category
primitives as the input of the generator. We finetune the T5-base/large model
for 20 epochs using a learning rate of 3e-5 and a batch size of 16.

####  4.3.2 Overall Evaluation

Table 1 and 5 summarize the results on Spider and WikiSQL respectively. On the
challenging Spider dataset, our model achieves competitive performance among
all baseline models. Compared with generation models that use whole DB table
schema as input like BRIDGE and UnifiedSKG on T5-base models, our model
achieves 3$\%$ improvement, suggesting the advantage of our method. During
primitive enumeration and ranking, we filter out irrelevant candidates based
on the DB table schema. Compared with the other T5-3B models, our model
achieves comparable performance with fewer training epochs where T5-3B*
trained 3K epochs, while we train only 100 epochs. For WikiSQL, we compare the
Text2SQL methods and answer generation method (TAPEX) in Table 5, and Uni-
Parser outperforms all the baselines.

| w/o CG | w/o HN | w/ HN
---|---|---|---
KBQA: WebQSP |  |  |
Top-1 | 54.5 | 56.2 | 56.2
Top-10 | 67.8 | 68.8 | 69.0
Top-20 | 70.4 | 71.4 | 71.8
TableQA: WikiSQL |  |  |
Top-1 | 80.5 | 81.9 | 84.0
Top-3 | 86.5 | 87.9 | 91.6
Top-5 | 88.4 | 90.0 | 94.0

Table 6: The Recall of the top-K ranked primitive on KB and Table. CG means
category, HN means Hard Negative. Figure 4: Ranker output (shown in dotted
boxes) and Generator output (shown in Final output) comparison between our
primitive-based method Uni-Parser and logical form-based method RnG-KBQA on
GrailQA dev set. Our model generates the correct output while the logical
form-based model produces a wrong output that is the same as the top-1 ranked
output.

###  4.4 Analysis

Ablation Study We perform an ablation study on the effects of hard negative
strategies in ranking on WebQSP and WikiSQL, and the result is shown in Table
6. No CG means that we do not differentiate primitives by their categories.
The intention of this design is to help the ranker to distinguish whether a
question has two or one hop reasoning. No CG shows lower performance than the
setting using categories in the input. By comparing the settings of with and
without hard negative (rightmost two columns), we can see that the proposed
hard negative sampling can help the ranker to better determine the positive
primitive from the negative ones. Moreover, the accuracy of entity linking is
WebQSP is 72.5, which means the upper bound of the ranking stage. Therefore,
our model achieves 69.0 among Top-10 primitive is close to the oracle
performance.

Dataset | LF | Primitive
---|---|---
KB |  |  | <|firsthop|> | <|secondhop|>
GrailQA | 1194 | 308 | 133
WebQSP | 4309 | 103 | 1546
DB |  |  | <|tb_cl|> | <|tb_cl_vl|>
Spider | 2726 | 38 | 22
WikiSQL | 1516 | 18 | 21

Table 7: The average numbers of candidate logical form and two types of
primitives in each dataset. LF represents logical form

Candidate Size To better understand the benefit of enumerating primitives in
reducing the size of candidates, we compare the numbers of enumerated
primitives and logical forms on both KB and DB. In Table 7, we show that in
KBQA datasets, the number of our primitives is three times less than the
number of logical forms used in previous SOTAs Ye et al. (2021); Gu et al.
(2021). In DB where the number of operations is usually larger, this
efficiency advantage is more obvious. While a logical form in KB is usually
composed of no more than two entities and two hops, a logical form in DB is
generally more complex with some uncertainty and the number of primitives and
operations in a logical form would be larger. Specifically, many frequently
used operations can appear in one logical form in SQL, like SELECT, WHERE,
ORDER BY, GROUP BY, JOIN ON, etc and each of them has a unique functionality.
In our model, we only need to enumerate two types of primitives (tb_cl,
tb_cl_vl) , which are sufficient to generate the complex logical form. As a
result, the number of primitives in DB is 30 to 40 times less than that of
logical forms. This shows the benefit of enumerating primitives in Uni-Parser
is universal across both KB and DB despite their significant differences in
structure and complexity.

Case study For a more intuitive understanding of our model, we show a concrete
example to illustrate the results of our model and the logical form
enumeration based model RnG-KBQA Ye et al. (2021) in Figure 4. The top-5
ranked logical forms in RnG-KBQA contain much redundant information, and none
of them equals the gold S-expression. In contrast, our output from the ranker
is simple and includes the ingredient of the gold s-expression. The output of
the ranker is used as the input to the generator. The generator output of the
RnG-KBQA is as same as the top-1 logical form from the ranker. This indicates
that their generator is more like a correctness rewriter that performs minor
edits on the input candidates. In comparison, our model can find the correct
first and second hop primitives and generate the correct logical form. It’s
worth noticing that even though proper primitive is not ranked as the top-1,
our generator has the capability to find the correct.

##  5 Related Work

We focus on semantic parsing rather than directly getting the answer, as
semantic parsing is more explainable Zhang and Balog (2020); Lin et al.
(2020). Many papers have applied seq2seq models to solve semantic parsing in
either KB Gu and Su (2022); Ye et al. (2021) or Table scenarios Dong and
Lapata (2016); Lin et al. (2018), treating it as a translation problem that
takes a natural question as input and outputs a logical form.

Semantic Parsing on KBQA Past works have attempted to generate logical forms
using a grammar-based bottom-up parser Berant et al. (2013); Pasupat and Liang
(2015) or a seq2seq network Hao et al. (2017); Zhang et al. (2019a). An
alternative approach is to produce a list of logical form candidates and then
use a ranker to find the ones that best match the intent of the question Lan
and Jiang (2020); Sun et al. (2020); Luo et al. (2018). Ye et al. (2021)
further employs a generation stage beyond the rank to remedy or supplement
existing logical form candidates. Rather than enumerating the complete logical
form, our Uni-Parser only gets the relevant primitives, which greatly improves
the parser’s efficiency and compositional generalization ability. Recently,
ArcaneQA proposed a generation-based model with dynamic programming induction
in the KB search space to improve the faithfulness of the generated programs
Gu and Su (2022) but it is still not as accurate as the rank-based model.
DECAF Yu et al. (2022) jointly generates both logical forms and direct
answers, which help them leverage both KB and text to get better final
answers.

Semantic Parsing on DBQA (Text2SQL) Text2SQL models take both the natural
language question and the database table schema as input Dou et al. (2022);
Zhang et al. (2020b). To get the sequential version of the table schema, prior
work commonly linearizes the input as a table name followed by all the column
names. Lin et al. (2020); Zhang et al. (2020a) further show that using table
content as supplemental information in Seq2Seq model can provide a better
understanding of the table schema. Moreover, it supports the prediction of the
conditional part in the logical form as mentioned in Yavuz et al. (2018). Shaw
et al. (2020) first shows that the pre-trained Seq2Seq model Raffel et al.
(2020) with 3 Billion parameters achieves competitive performance on the
Spider dataset. Scholak et al. (2021) proposes a constrained decoding method
that can be compatible with various large pre-trained language models and
achieves promising performance on Spider.

Unified Question Answering Many unified QA models convert the structured (KB)
or semi-structured data (DB) to unstructured texts, which provides additional
information for missing knowledge in the open-domain textual QA by directly
linearizing the structured schema into text Oguz et al. (2020); Xie et al.
(2022); Tay et al. (2022). Ma et al. (2021) further uses the data-to-text
generator to revise the linearized schema into natural language. Li et al.
(2021) proposes a hybrid QA model that either answers questions using text or
generates the SQL queries from table schema on the textual and Tabular QA
datasets. Our Uni-Parser works in a different direction that efficiently
parses the questions into the executable logical forms on both KB and DB in a
unified framework.

##  6 Conclusion

For unified semantic parsing on both KB and DB structured data, we propose
Uni-Parser, which has three modules: primitive enumeration, ranker, and
compositional generator. Our enumeration at the primitive level rather than
the logical-form level produces a smaller number of potential candidates,
leading to high efficiency in the enumeration and ranker steps. Moreover,
training a generator to produce the logical form from the primitives leads to
a more generalized and robust compositional generator. Experimental results on
both KB and DB QA demonstrate the advantages of Uni-Parser, especially in the
compositional and zero-shot settings.

##  7 Acknowledge

The authors would like to thank the members of Salesforce AI Research team for
fruitful discussions, as well as the anonymous reviewers for their helpful
feedback.

## Limitations

The current Uni-Parser model needs to independently train on each of the
datasets. In this work, we test it on four datasets. But if having more
datasets, this process will be very time costing. A more unified way is having
one model trained on all the datasets, either from KB or DB, once and
producing a good performance on each dataset.

The other limitation is that the current model needs to indicate whether the
question is from KB or DB. This makes the model hard to be applied to reality
where which source can answer the question is unknown. Those limitations are
challenging and we leave them for further explorations.

## References

  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on freebase from question-answer pairs.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1533–1544.
  * Berant and Liang (2014) Jonathan Berant and Percy Liang. 2014.  Semantic parsing via paraphrasing.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1415–1425.
  * Bogin et al. (2019) Ben Bogin, Matt Gardner, and Jonathan Berant. 2019.  Global reasoning over database structures for text-to-sql parsing.  _arXiv preprint arXiv:1908.11214_.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008\.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , pages 1247–1250.
  * Chang et al. (2020) Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020\.  Pre-training tasks for embedding-based large-scale retrieval.  _arXiv preprint arXiv:2002.03932_.
  * Chen et al. (2021) Shuang Chen, Qian Liu, Zhiwei Yu, Chin-Yew Lin, Jian-Guang Lou, and Feng Jiang. 2021\.  Retrack: a flexible and efficient framework for knowledge base question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 325–336.
  * Das et al. (2021) Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay-Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021.  Case-based reasoning for natural language queries over knowledge bases.  _arXiv preprint arXiv:2104.08762_.
  * Devlin et al. (2017) Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017.  Robustfill: Neural program learning under noisy i/o.  In _International conference on machine learning_ , pages 990–998. PMLR.
  * Dong and Lapata (2016) Li Dong and Mirella Lapata. 2016.  Language to logical form with neural attention.  _arXiv preprint arXiv:1601.01280_.
  * Dou et al. (2022) Longxu Dou, Yan Gao, Mingyang Pan, Dingzirui Wang, Jian-Guang Lou, Wanxiang Che, and Dechen Zhan. 2022.  Unisar: A unified structure-aware autoregressive language model for text-to-sql.  _arXiv preprint arXiv:2203.07781_.
  * Gu et al. (2021) Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021.  Beyond iid: three levels of generalization for question answering on knowledge bases.  In _Proceedings of the Web Conference 2021_ , pages 3477–3488.
  * Gu and Su (2022) Yu Gu and Yu Su. 2022.  Arcaneqa: Dynamic program induction and contextualized encoding for knowledge base question answering.  _COLING_.
  * Guo and Gao (2019) Tong Guo and Huilin Gao. 2019.  Content enhanced bert-based text-to-sql generation.  _arXiv preprint arXiv:1910.07179_.
  * Hao et al. (2017) Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. 2017.  An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 221–231.
  * He et al. (2019) Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019.  X-sql: reinforce schema representation with context.  _arXiv preprint arXiv:1908.08113_.
  * Hwang et al. (2019) Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Seo. 2019.  A comprehensive exploration on wikisql with table-aware word contextualization.  _arXiv preprint arXiv:1902.01069_.
  * Lan and Jiang (2020) Yunshi Lan and Jing Jiang. 2020.  Query graph generation for answering multi-hop complex questions from knowledge bases.  Association for Computational Linguistics.
  * Lan et al. (2019) Yunshi Lan, Shuohang Wang, and Jing Jiang. 2019.  Knowledge base question answering with topic units.
  * Lei et al. (2022) Jie Lei, Xinlei Chen, Ning Zhang, Mengjiao Wang, Mohit Bansal, Tamara L Berg, and Licheng Yu. 2022.  Loopitr: Combining dual and cross encoder architectures for image-text retrieval.  _arXiv preprint arXiv:2203.05465_.
  * Li et al. (2021) Alexander Hanbo Li, Patrick Ng, Peng Xu, Henghui Zhu, Zhiguo Wang, and Bing Xiang. 2021.  Dual reader-parser on hybrid textual and tabular evidence for open domain question answering.  _arXiv preprint arXiv:2108.02866_.
  * Liang (2013) Percy Liang. 2013.  Lambda dependency-based compositional semantics.  _arXiv preprint arXiv:1309.4408_.
  * Lin et al. (2020) Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020.  Bridging textual and tabular data for cross-domain text-to-sql semantic parsing.  _arXiv preprint arXiv:2012.12627_.
  * Lin et al. (2018) Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. 2018.  Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system.  _arXiv preprint arXiv:1802.08979_.
  * Liu et al. (2021a) Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian-guang Lou. 2021a.  Tapex: Table pre-training via learning a neural sql executor.  _arXiv preprint arXiv:2107.07653_.
  * Liu et al. (2021b) Ye Liu, Kazuma Hashimoto, Yingbo Zhou, Semih Yavuz, Caiming Xiong, and Philip S Yu. 2021b.  Dense hierarchical retrieval for open-domain question answering.  _EMNLP_.
  * Luo et al. (2018) Kangqi Luo, Fengli Lin, Xusheng Luo, and Kenny Zhu. 2018.  Knowledge base question answering via encoding of complex query graphs.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2185–2194.
  * Lyu et al. (2020) Qin Lyu, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. 2020.  Hybrid ranking network for text-to-sql.  _arXiv preprint arXiv:2008.04759_.
  * Ma et al. (2020) Jianqiang Ma, Zeyu Yan, Shuai Pang, Yang Zhang, and Jianping Shen. 2020.  Mention extraction and linking for sql query generation.  _arXiv preprint arXiv:2012.10074_.
  * Ma et al. (2021) Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2021.  Open domain question answering over virtual documents: A unified approach for data and text.  _arXiv preprint arXiv:2110.08417_.
  * Oguz et al. (2020) Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2020.  Unik-qa: Unified representations of structured and unstructured knowledge for open-domain question answering.  _arXiv preprint arXiv:2012.14610_.
  * Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015.  Compositional semantic parsing on semi-structured tables.  _arXiv preprint arXiv:1508.00305_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Scholak et al. (2021) Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021.  Picard: Parsing incrementally for constrained auto-regressive decoding from language models.  _arXiv preprint arXiv:2109.05093_.
  * Shaw et al. (2020) Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2020.  Compositional generalization and natural language variation: Can a semantic parsing approach handle both?  _arXiv preprint arXiv:2010.12725_.
  * Sun et al. (2020) Yawei Sun, Lingling Zhang, Gong Cheng, and Yuzhong Qu. 2020.  Sparqa: skeleton-based semantic parsing for complex questions over knowledge bases.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8952–8959.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022.  Unifying language learning paradigms.  _arXiv preprint arXiv:2205.05131_.
  * Thakur et al. (2020) Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2020.  Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks.  _arXiv preprint arXiv:2010.08240_.
  * Wang et al. (2019) Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019.  Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers.  _arXiv preprint arXiv:1911.04942_.
  * Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. 2022\.  Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  _arXiv preprint arXiv:2201.05966_.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  _arXiv preprint arXiv:2007.00808_.
  * Yavuz et al. (2018) Semih Yavuz, Izzeddin Gür, Yu Su, and Xifeng Yan. 2018.  What it takes to achieve 100% condition accuracy on wikisql.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1702–1711.
  * Ye et al. (2020) Xi Ye, Qiaochu Chen, Xinyu Wang, Isil Dillig, and Greg Durrett. 2020.  Sketch-driven regular expression generation from natural language and examples.  _Transactions of the Association for Computational Linguistics_ , 8:679–694.
  * Ye et al. (2021) Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2021.  Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering.  _arXiv preprint arXiv:2109.08678_.
  * Yih et al. (2015) Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015.  Semantic parsing via staged query graph generation: Question answering with knowledge base.  In _Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP_.
  * Yih et al. (2016) Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016.  The value of semantic parse labeling for knowledge base question answering.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 201–206.
  * Yu et al. (2022) Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Wang, Zhiguo Wang, and Bing Xiang. 2022.  Decaf: Joint decoding of answers and logical forms for question answering over knowledge bases.  _arXiv preprint arXiv:2210.00063_.
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.  _arXiv preprint arXiv:1809.08887_.
  * Zhang et al. (2019a) Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019a.  Complex question decomposition for semantic parsing.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4477–4486.
  * Zhang et al. (2019b) Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b.  Editing-based sql query generation for cross-domain context-dependent questions.  _arXiv preprint arXiv:1909.00786_.
  * Zhang and Balog (2020) Shuo Zhang and Krisztian Balog. 2020.  Web table extraction, retrieval, and augmentation: A survey.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 11(2):1–35.
  * Zhang et al. (2020a) Xiaoyu Zhang, Fengjing Yin, Guojie Ma, Bin Ge, and Weidong Xiao. 2020a.  F-sql: fuse table schema and table content for single-table text2sql generation.  _IEEE Access_ , 8:136409–136420.
  * Zhang et al. (2020b) Xiaoyu Zhang, Fengjing Yin, Guojie Ma, Bin Ge, and Weidong Xiao. 2020b.  M-sql: Multi-task representation learning for single-table text2sql generation.  _IEEE Access_ , 8:43156–43167.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017.  Seq2sql: Generating structured queries from natural language using reinforcement learning.  _arXiv preprint arXiv:1709.00103_.


# Sociolinguistically Informed Interpretability:
A Case Study on Hinglish Emotion Classification

Kushal Tatariya1 Heather Lent2 Johannes Bjerva2 Miryam de Lhoneux1
1 Department of Computer Science, KU Leuven, Belgium
2 Department of Computer Science, Aalborg University, Denmark
{kushaljayesh.tatariya<EMAIL_ADDRESS>
{hcle<EMAIL_ADDRESS>

###### Abstract

Emotion classification is a challenging task in NLP due to the inherent
idiosyncratic and subjective nature of linguistic expression, especially with
code-mixed data. Pre-trained language models (PLMs) have achieved high
performance for many tasks and languages, but it remains to be seen whether
these models learn and are robust to the differences in emotional expression
across languages. Sociolinguistic studies have shown that Hinglish speakers
switch to Hindi when expressing negative emotions and to English when
expressing positive emotions. To understand if language models can learn these
associations, we study the effect of language on emotion prediction across 3
PLMs on a Hinglish emotion classification dataset. Using LIME Ribeiro et al.
(2016) and token level language ID, we find that models do learn these
associations between language choice and emotional expression. Moreover,
having code-mixed data present in the pre-training can augment that learning
when task-specific data is scarce. We also conclude from the
misclassifications that the models may overgeneralise this heuristic to other
infrequent examples where this sociolinguistic phenomenon does not apply.

Disclaimer: This paper contains some examples of language use that readers may
find offensive.

Figure 1: Our workflow. We train 3 emotion classification models, then obtain
LIME scores for each token (positive scores in red, negative scores in blue,
and zero scores in grey). These same samples are then tagged with token-level
language ID, which enables us to examine LIME distributional differences by
language.

##  1 Introduction

An open-ended goal of the NLP community is to develop language technologies
robust to the vast and various idiosyncrasies of authentic human
communication. Understanding emotion requires knowledge of the subtleties of
linguistic expression and inherent human subjectivity, making emotion
classification a challenging task. It is further complicated when working with
code-mixed utterances. Every language participating in code-mixed
communication comes with its own cultural and linguistic baggage that oversees
the verbalization of emotion Kachru (1978); Hershcovich et al. (2022). The
adoption of pre-trained language models (PLMs) has improved performance across
the board for this task, but the PLMs still remain black boxes. While research
in interpretability aims to address this shortcoming, most analyses remain
centered around English Ruder et al. (2022). In this work, we aim to make
explicit what associations are learned when PLMs are trained on code-mixed
data, and whether established differences in linguistic expression across
languages indeed influence model prediction.

We approach this interpretability problem through the lens of
sociolinguistics. In particular, we focus on Hindi-English (Hinglish) code-
mixing, prevalent in India and in the Indian diaspora Orsini (2015). In a
study on Hindi-English bilinguals on Twitter, Rudra et al. (2016) observed
that English was the language of choice for expression of a positive emotion
and Hindi was more used for negative emotion. Moreover, Hindi was the
preferred language for swearing online, a finding also echoed by Agarwal et
al. (2017). Rudra et al. (2016) explain the reason behind this to be the fact
that bilinguals prefer to express strong emotions Dewaele (2010) and swear
Dewaele (2007) in L1, which happens to be Hindi for most Hinglish speakers.
Conversely, Rudra et al. (2016) speculate that since English is the language
of aspiration in India, it becomes the preferred language for positive
emotion.

In this context, we formulate our main questions as: (RQ1) Are PLMs likely to
associate different emotions with different languages? (RQ2) Are English
tokens more likely to influence a model to predict a positive emotion? (RQ3)
Are Hindi tokens more likely to influence a model to predict a negative
emotion, and if so, what is the role of Hindi swear words? To this end, we
fine-tune 3 different PLMs on a Hinglish emotion classification dataset and
leverage LIME and token-level language identification for an interpretability
analysis.

##  2 Related Work

##### Code-Mixing

Previous works in emotion classification and sentiment analysis have
demonstrated that processing code-mixed text is more difficult than
monolingual text Sitaram et al. (2019); Zaharia et al. (2020); Yulianti et al.
(2021). This is in part due to the complexities of processing emotion from two
different languages with varying socio-cultural and grammatical structures at
play Younas et al. (2020); Sasidhar et al. (2020); Ilyas et al. (2023). In
this context, Doğruöz et al. (2021) published a survey on the linguistic and
social perspectives on code-mixing for language technologies. They emphasized
the importance of incorporating the social context of a code-mixed language
pair into systems processing code-mixed text.

##### Interpretability with LIME

LIME (Local Interpretable Model-Agnostic Explanations) Ribeiro et al. (2016)
is a popular tool for interpretability that is model agnostic and employable
for classification tasks. It learns a linear classifier locally around a
model’s prediction, leveraging token weights (also learned by the linear
classifier) to assign a "LIME score" between 1 and -1 to each token. A
positive score indicates that the token influenced the model towards the
predicted label, and a negative score indicates that the token influenced the
model to not predict that label. We leverage LIME due to its availability and
easy-to-use implementations, for instance in the Language Interpretability
Tool (LIT; Tenney et al., 2020), which we used for this work. Previous work
has also indicated the accuracy of its approximation of the models and its
ability to provide human-friendly explanations Madsen et al. (2022); Hajiyan
et al. (2023).

##  3 Methodology

##### Dataset

We utilize a dataset for sentiment analysis of code-mixed tweets by Patwa et
al. (2020), later annotated with emotion labels by Ghosh et al. (2023). Each
example is annotated with the six basic Ekman emotions Ekman et al. (1999) \-
joy, sadness, fear, surprise, disgust and anger. When an example does not fit
any of these emotions, or expresses no emotion, it is labelled as others. This
dataset contains 14,000 examples in the train set, 3,000 in the validation,
and 3,000 in the test set. For this work, we randomly sample 1,000 examples
from the validation set to enable manual verification of the automatic token
tagging described below, maintaining the default distribution across labels
(see Appendix B).

##### Models

We fine-tune 3 different PLMs for the task of emotion classification with the
Hinglish training data:

  * •

XLMR Conneau et al. (2020), pre-trained on Common Crawl, spanning 100
languages, including English and Hindi, both in the Devanagari script and
additional romanized Hindi.

  * •

IndicBERT v2 Doddapaneni et al. (2022), pre-trained on data from 24 Indic
languages, including Hindi and English that is local to the Indian
subcontinent. For Hindi, this model has only seen the Devanagari script, and
no romanized Hindi.

  * •

HingRoBERTa Nayak and Joshi (2022), an XLM-R model that has been further pre-
trained on romanized, code-mixed Hindi-English. Thus, in addition to having
seen romanized Hindi, this model is specifically intended for code-mixed text.

The full details on model training and performance are given in Appendix A.

##### Token Tagging

For each of the 1,000 samples (20,835 tokens in total) drawn from the
validation set, we first obtain LIME scores for each token using LIT Tenney et
al. (2020). We then run the samples through CodeSwitch Sarkar (2020), a
Hinglish language identification tool which tags each token as Hindi, English,
or Other, to get the language ID tags.111Besides the Hindi and English labels,
CodeSwitch also tags tokens as ”Named-Entity”, ”Foreign words”, and ”Other”
for punctuation, emojis, and other non-textual tokens. For this work, we
combine these 3 additional tags into one category.

Figure 2: Frequencies of Hindi (green), English (purple) and Other (orange)
tokens to be assigned a positive (solid) or a negative (striped) LIME score
for examples predicted as joy and anger, for all models.

##  4 Results and Analysis

First, to answer whether the models learn to meaningfully distinguish between
languages for emotion prediction (RQ1), we examine the distributions of LIME
scores across each language ID tag (English, Hindi, Other). Concretely, we
inspect the frequency with which tokens received a positive or a negative LIME
score in our sample, for each language. We then conduct a $\chi^{2}$ test of
independence to determine whether these two variables have some dependency.
Table 1 shows the $p$-values for the entire sample. For all models, we find
this dependence to be statistically significant ($p<0.05$), indicating that
there is some influence of language over the LIME scores. We also confirm this
with a 1-Way ANOVA test, which can be found in Appendix C, along with our
entire statistical analysis.

Next, we examine this dependency on a more granular level to determine whether
the presence of English tokens influence the Hinglish emotion classification
models to predict more positive emotions (RQ2), and whether Hindi tokens
influence them to predict negative emotions (RQ3). We observe the distribution
of language ID across LIME scores for examples that the models predicted as
joy, anger, and sadness. These labels were selected in particular as they have
the most examples in the dataset (after others), and provide the positive
(joy) and negative (anger and sadness) polarity discussed in the
sociolinguistics literature.

|  $p$-values
---|---
Model | Entire Sample | Joy | Anger | Sadness
XLM-R | 7.06e-12 | 1.44e-15 | 6.18e-7 | 1.78e-3
IndicBERT | 1.22e-22 | 3.28e-4 | 1.69e-5 | 3.30e-1
HingRoBERTa | 3.30e-7 | 4.00e-18 | 1.71e-8 | 2.18e-5
Table 1: We test the null hypothesis that language ID tags and LIME scores are
independent of each other using $\chi^{2}$. This table contains the $p$-values
for tests done on the entire sample, and also on examples predicted as joy,
anger, and sadness.

##### (RQ2) Do English tokens influence models to predict positive emotions?

Figure 2 shows which languages tend to have more positive and more negative
LIME scores. As observed for joy, English tokens have the highest frequency
with positive LIME scores. Table 1 shows that there is a significant
dependency between language ID and LIME score for all models. Thus, English
tokens influence the model significantly more than Hindi and Others when
predicting joy.

##### (RQ3) Do Hindi tokens influence models to predict negative emotions?

When predicting anger, Table 1 again shows that there is dependency between
language ID and LIME score for all models. From Figure 2, we can see that
Hindi tokens influence the model significantly towards predicting anger. When
predicting sadness, however, we only observe significance with the XLM-R and
HingRoBERTa models, but not with IndicBERT. Moreover, for XLM-R, the $p$-value
is not much lower than the threshold. Thus, we cannot make strong conclusions
for this label.

Token | Lang_ID | Swear Word?222As decided by a native speaker, and also compared with the lexicon lists of Hindi and English swear words used by Agarwal et al. (2017).
---|---|---
Fuck | eng | Yes
Chutiye | hin | Yes
Fakeionist | eng | No
Bsdk | hin | Yes
Sadly | eng | No
Bakwas | hin | No
Kutta | hin | Yes
Gaddar | hin | No
Shame | eng | No
Sala | hin | Yes
Table 2: Top 10 tokens with the highest LIME scores when predicting negative
emotions, (anger, sadness, disgust andfear) for all models. They have been
mapped to a canonical form and are in descending order of LIME score.

##### Swear Words

Previous works demonstrate that Hinglish speakers prefer to swear in Hindi
over English, in a code-mixed setting Rudra et al. (2016); Agarwal et al.
(2017). To check whether this finding is similarly echoed by our fine-tuned
models, we examine the top 10 tokens with the highest LIME scores when
predicting a negative emotion (anger, sadness, disgust, fear), across all
models (see Table 2). While the first among these is an English swear word
(owing to it being the most used swear word by Hinglish speakers online
Agarwal et al. (2017)) there are 4 Hindi swear words in this list of tokens.
As such, we can see that the models not only learn the negative connotation of
the Hindi swear words, but also that these Hindi swear words are the most
negative of all other tokens, regardless of language, thus confirming
observations from the sociolinguistics literature.

##  5 Discussion

From the section above, it can be concluded that the models are able to
distinguish patterns of speaker preference detailed by Rudra et al. (2016)
when predicting emotion for code-mixed data. English tokens influence the
models more towards predicting a positive emotion, and Hindi tokens influence
the models more towards predicting a negative emotion. An example of this is
provided in Figure 3, where all the models exhibit a strong degree of
influence from the English tokens in their prediction of the joy label. At the
same time, most of the tokens assigned a negative LIME score come from Hindi.

Figure 3: An example from the dataset labelled as joy, with the translation
and language ID tags. The 3 tokens with the highest LIME scores are marked in
blue, and the 3 tokens with the lowest scores are marked in red.

For sadness, we surmise that a shortage of training data is responsible for
models’ failure to learn meaningful differences across the languages. About
10% of the entire dataset consists of examples labelled sadness. In contrast,
joy is 30% and anger is about 20% (see Appendix B). Even with less data,
however, we still observe a dependency between language and LIME score with
HingRoBERTa. It is the only model we examine with code-mixed data present in
the pre-training. Thus, when there is less data for a model to learn these
associations, it can help to have code-mixed data in the pre-training.

###  5.1 Do PLMs overgeneralize these learnt associations?

McCoy et al. (2019) found that language models can adapt to heuristics that
are valid for frequent cases and fail on the less frequent ones. In a similar
vein, we investigate whether these sociolinguistic associations learnt by the
models overgeneralise to the less frequent examples where this phenomenon is
not seen. We examine instances where the models have misclassified examples
labelled as joy and anger, highlighted in Figure 4.

For both joy and anger, the models generally either predict another label of
the same emotional polarity (for example, disgust instead of anger), or they
predict them as others. The dataset is highly imbalanced, and thus we can say
that although the models can discern the polarity difference between positive
and negative emotion labels (as seen in Figure 4 where the values in the lower
left and upper right quadrants are low), they struggle with granular
distinctions between them.

Figure 4: Confusion matrix containing the percentage of correctly and
incorrectly classified examples for each label combination. The blue cells
represent correct classifications, and the pink cells represent incorrect
classifications.

We also manually look into the few instances where joy examples were assigned
a negative emotion label, and anger examples were assigned a positive emotion
label. Out of the total instances, 15 involve scenarios where either Hindi
words with a negative connotation led the model to attribute a negative label
to joy, or English words with a positive connotation influenced the model to
assign a positive emotion label to anger. This suggests that examples
featuring English words indicating positive emotions on their own can mislead
the model into predicting a positive emotion label despite an overall negative
tone in the expression (and vice versa for Hindi words), as illustrated in
Figure 5.

Figure 5: An example labelled anger that was misclassified as joy owing to the
English phrase (English \- purple; Hindi \- green; Other \- orange) in the
sentence having a positive connotation, even though the sentence itself
conveys anger.

On a broader scale, we examine the distribution of English, Hindi and Other
tokens in the misclassified joy and anger examples. As seen in Table 3, the
normalised frequency of Hindi tokens is higher in the misclassified joy
examples than the overall distribution. Consequently, more Hindi tokens have a
positive LIME score. Thus, McCoy et al. (2019)’s conclusions stated earlier
are echoed here as well. While the extreme cases where the models
overgeneralise to predict an emotion label of the opposite polarity are few,
there is a bias learnt in the models against predicting joy for Hindi tokens.
For examples labelled anger, although there is less difference seen in the
frequency of English tokens in the misclassified examples, more English tokens
have a positive LIME score. Thus, a similar bias against predicting anger for
English could be inferred.

Overall, the fact that these associations are learnt by the models, to the
extent that they can overgeneralise them, could also be seen as substantiating
the sociolinguistic phenomena. If speakers tend to switch to Hindi to express
negative emotions, the ability of language models to detect this reinforces
the existence of such a tendency. This also encourages deeper engagement
between sociolinguistics and interpretability, with both fields offering
valuable insights to each other.

Joy
---
Distribution of tokens in all examples
| All examples | Correct | Misclassified
English | 0.40 | 0.44 | 0.32
Hindi | 0.34 | 0.29 | 0.44
Other | 0.26 | 0.27 | 0.24
Distribution of tokens assigned a positive LIME score
| All examples | Correct | Misclassified
English | 0.43 | 0.48 | 0.32
Hindi | 0.32 | 0.28 | 0.42
Other | 0.25 | 0.24 | 0.32
Anger
Distribution of tokens in all examples
| All examples | Correct | Misclassified
English | 0.15 | 0.14 | 0.17
Hindi | 0.63 | 0.65 | 0.61
Other | 0.22 | 0.21 | 0.22
Distribution of tokens assigned a positive LIME score
| All examples | Correct | Misclassified
English | 0.15 | 0.13 | 0.18
Hindi | 0.64 | 0.68 | 0.60
Other | 0.21 | 0.19 | 0.23
Table 3: Normalized frequencies of English, Hindi, and Other tokens for
instances labeled joy and anger for correct and incorrect classification.
Additionally, the count of tokens in each language category assigned a
positive LIME score for all models.

##  6 Conclusion

In this work, we use sociolinguistics theories to understand what PLMs learn
when training emotion classifiers for code-mixed data. We found that the
models indeed learn the differences in language use and emotional expression
detailed in the sociolinguistics literature. Concretely, these are the
associations of English tokens with positive emotions, and Hindi tokens with
negative emotions. Adding code-mixed data to the pre-training can help augment
this learning when task-specific data is scarce. However, the models can
overgeneralise this learning to infrequent examples where it does not apply.
In future work, it would be interesting to see if this understanding can be
leveraged to help improve systems designed for code-mixed languages.

##  7 Acknowledgements

The computational resources and services used in this work were provided by
the VSC (Flemish Supercomputer Center), funded by the Research Foundation -
Flanders (FWO) and the Flemish Government - department EWI (for Kushal
Tatariya and Miryam de Lhoneux). Heather Lent and Johannes Bjerva are
supported by the Carlsberg Foundation, under the Semper Ardens: Accelerate
programme (project nr. CF21-0454).

## References

  * Agarwal et al. (2017) Prabhat Agarwal, Ashish Sharma, Jeenu Grover, Mayank Sikka, Koustav Rudra, and Monojit Choudhury. 2017.  I may talk in english but gaali toh hindi mein hi denge : A study of english-hindi code-switching and swearing pattern on social networks.  In _2017 9th International Conference on Communication Systems and Networks (COMSNETS)_ , pages 554–557.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Dewaele (2007) Jean-Marc Dewaele. 2007.  Blistering barnacles! what language do multilinguals swear in?! _Sociolinguistic Studies_ , 5.
  * Dewaele (2010) Jean-Marc Dewaele. 2010.  _Emotions in multiple languages._ Palgrave Macmillan, Basingstoke.
  * Doddapaneni et al. (2022) Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2022.  Indicxtreme: A multi-task benchmark for evaluating indic languages.
  * Doğruöz et al. (2021) A. Seza Doğruöz, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021.  A survey of code-switching: Linguistic and social perspectives for language technologies.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1654–1666, Online. Association for Computational Linguistics.
  * Ekman et al. (1999) Paul Ekman et al. 1999.  Basic emotions.  _Handbook of cognition and emotion_ , 98(45-60):16.
  * Ghosh et al. (2023) Soumitra Ghosh, Amit Priyankar, Asif Ekbal, and Pushpak Bhattacharyya. 2023.  Multitasking of sentiment detection and emotion recognition in code-mixed hinglish data.  _Knowledge-Based Systems_ , 260:110182.
  * Hajiyan et al. (2023) Hooria Hajiyan, Heidar Davoudi, and Mehran Ebrahimi. 2023.  A comparative analysis of local explainability of models for sentiment detection.  In _Proceedings of the Future Technologies Conference (FTC) 2022, Volume 3_ , pages 593–606, Cham. Springer International Publishing.
  * Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022.  Challenges and strategies in cross-cultural NLP.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics.
  * Ilyas et al. (2023) Abdullah Ilyas, Khurram Shahzad, and Muhammad Kamran Malik. 2023.  Emotion detection in code-mixed roman urdu - english text.  _ACM Trans. Asian Low-Resour. Lang. Inf. Process._ , 22(2).
  * Kachru (1978) Braj B. Kachru. 1978.  Toward structuring code-mixing: An indian perspective.  _International Journal of the Sociology of Language_ , 1978(16):27–46.
  * Madsen et al. (2022) Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022.  Post-hoc interpretability for neural nlp: A survey.  _ACM Comput. Surv._ , 55(8).
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Nayak and Joshi (2022) Ravindra Nayak and Raviraj Joshi. 2022.  L3Cube-HingCorpus and HingBERT: A code mixed Hindi-English dataset and BERT language models.  In _Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference_ , pages 7–12, Marseille, France. European Language Resources Association.
  * Orsini (2015) Francesca Orsini. 2015.  Dil maange more: Cultural contexts of hinglish in contemporary india.  _African Studies_ , 74(2):199–220.
  * Patwa et al. (2020) Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas PYKL, Björn Gambäck, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. 2020.  SemEval-2020 task 9: Overview of sentiment analysis of code-mixed tweets.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 774–790, Barcelona (online). International Committee for Computational Linguistics.
  * Ribeiro et al. (2016) Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  "why should I trust you?": Explaining the predictions of any classifier.  _CoRR_ , abs/1602.04938.
  * Ruder et al. (2022) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2022.  Square one bias in NLP: Towards a multi-dimensional exploration of the research manifold.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics.
  * Rudra et al. (2016) Koustav Rudra, Shruti Rijhwani, Rafiya Begum, Kalika Bali, Monojit Choudhury, and Niloy Ganguly. 2016.  Understanding language preference for expression of opinion and sentiment: What do Hindi-English speakers do on Twitter? In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1131–1141, Austin, Texas. Association for Computational Linguistics.
  * Sarkar (2020) Sagor Sarkar. 2020.  Code switch.
  * Sasidhar et al. (2020) T Tulasi Sasidhar, Premjith B, and Soman K P. 2020.  Emotion detection in hinglish(hindi+english) code-mixed social media text.  _Procedia Computer Science_ , 171:1346–1352.  Third International Conference on Computing and Network Communications (CoCoNet’19).
  * Sitaram et al. (2019) Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, and Alan W. Black. 2019.  A survey of code-switched speech and language processing.  _CoRR_ , abs/1904.00784.
  * Tenney et al. (2020) Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020.  The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 107–118, Online. Association for Computational Linguistics.
  * Younas et al. (2020) Aqsa Younas, Raheela Nasim, Saqib Ali, Guojun Wang, and Fang Qi. 2020.  Sentiment analysis of code-mixed roman urdu-english social media text using deep learning approaches.  In _2020 IEEE 23rd International Conference on Computational Science and Engineering (CSE)_ , pages 66–71.
  * Yulianti et al. (2021) Evi Yulianti, Ajmal Kurnia, Mirna Adriani, and Yoppy Setyo Duto. 2021.  Normalisation of indonesian-english code-mixed text and its effect on emotion classification.  _International Journal of Advanced Computer Science and Applications_ , 12(11).
  * Zaharia et al. (2020) George-Eduard Zaharia, George-Alexandru Vlad, Dumitru-Clementin Cercel, Traian Rebedea, and Costin Chiru. 2020.  UPB at SemEval-2020 task 9: Identifying sentiment in code-mixed social media texts using transformers and multi-task learning.  In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , pages 1322–1330, Barcelona (online). International Committee for Computational Linguistics.

##  Appendix A Model Details

We used Huggingface to fine-tune the pre-trained language models described in
Section 3 on the emotion classification dataset. Our hyperparameters are
listed in Table 4 and the performance of our models over the development set
are in Table 5, below.

Hyperparameter | Value
---|---
Dropout | 0.2
Learning Rate | 2e-05
Number of Epochs | 50
Batch Size | 32
Table 4: The hyperparameters used to train all three emotion classification models.  Model | Accuracy
---|---
XLM-R | 0.57
IndicBERT | 0.55
HingRoBERTa | 0.58
Table 5: Accuracy scores on the test sets of each pre-trained language model
fine-tuned on the Hinglish emotion classification dataset.

##  Appendix B Label Distributions

The sample of 1,000 examples used in the analysis was selected by maintaining
the label distribution from the validation set. The distribution is detailed
in Table 6.

Label Distributions
---
| Our Sample | Validation Set
others | 347 | 1048
joy | 325 | 973
anger | 204 | 607
sadness | 102 | 307
disgust | 19 | 55
surprise | 2 | 6
fear | 1 | 4
Total | 1,000 | 3,000
Table 6: Distribution of emotion labels in our random sample versus the
original validation set.

##  Appendix C Statistical Analysis

###  C.1 $\chi^{2}$

We performed a $\chi^{2}$ test of independence on the samples for each model
to understand the relationship between the two variables - language ID and
LIME score. We constructed the contingency tables with the frequencies of how
many times each language ID label - eng, hin and other had a positive or a
negative LIME score. We did this for the entire sample to confirm a dependency
between those variables. We further examined this dependency on a more
granular level by conducting the same $\chi^{2}$ test for examples that were
predicted as joy, anger and sadness by the models. The contingency table for
the entire sample is in Table 7, and per label is in Table 8.

| Contingency Tables - All Samples
---|---
| XLM-R | IndicBERT | HingRoBERTa
| Positive | Negative | Positive | Negative | Positive | Negative
English | 3658 | 2264 | 3840 | 2082 | 3728 | 2194
Hindi | 5759 | 4242 | 5914 | 4087 | 6127 | 3874
Other | 2709 | 2203 | 3281 | 1631 | 2843 | 2069
Table 7: $\chi^{2}$ contingency tables for all samples, across all models | Contingency Tables - Per Label
---|---
| Joy
| XLM-R | IndicBERT | HingRoBERTa
| Positive | Negative | Positive | Negative | Positive | Negative
English | 1730 | 759 | 1669 | 643 | 1643 | 639
Hindi | 1127 | 648 | 974 | 500 | 1061 | 527
Other | 876 | 668 | 1010 | 429 | 849 | 618
| Anger
| XLM-R | IndicBERT | HingRoBERTa
| Positive | Negative | Positive | Negative | Positive | Negative
English | 365 | 173 | 332 | 165 | 482 | 288
Hindi | 1760 | 689 | 1487 | 789 | 1926 | 840
Other | 473 | 293 | 393 | 307 | 551 | 370
| Sadness
| XLM-R | IndicBERT | HingRoBERTa
| Positive | Negative | Positive | Negative | Positive | Negative
English | 248 | 139 | 176 | 80 | 233 | 135
Hindi | 596 | 258 | 389 | 187 | 597 | 272
Other | 187 | 129 | 135 | 49 | 169 | 143
Table 8: $\chi^{2}$ contingency tables for examples predicted as joy, anger
and sadness by each model

###  C.2 ANOVA and Tukey HSD

####  C.2.1 Entire Sample

ANOVA - All Samples
---
Model |  $p$-value
XLM-R | 2.09e-31
IndicBERT | 3.35e-45
HingRoBERTa | 3.97e-20
Table 9: We test the null hypothesis that language ID tags and LIME scores are
independent of each other using 1-Way ANOVA. This table contains the
$p$-values for tests done on the entire sample.

The $p$-values from the ANOVA results are in Table 9. They confirm $\chi^{2}$
results that for the entire sample size, there is dependency between language
and LIME score for all models. The key difference between our ANOVA and the
$\chi^{2}$ tests is that, while the $\chi^{2}$ treats LIME score polarity as a
categorical variable (positive versus negative scores), in our ANOVA we
directly compute over the numerical values, ranging from -1 to 1.

In order to better understand the relationship between languages (i.e., Hindi
versus English; Hindi versus Other; English versus Other), we also performed
an additional post-hoc Tukey HSD Test to test which pairs of language ID have
means that are significantly different from each other. Results for all
samples are in Table 10. For all models, the means for Hindi and English
tokens are meaningfully different from each other, and thus we can say that
all models are able to distinguish between these two languages. For XLM-R, we
cannot reject the null hypothesis that hin and other have independent
distributions, and for IndicBERT, we cannot reject that eng and other have
independent distributions. It is only for HingRoBERTa that we can reject the
null hypothesis for all pairs of language ID. Thus, HingRoBERTa, having seen
code-mixed data in the pre-training, is the only one that can meaningfully
distinguish across eng, hin and other.

Tukey HSD - All Samples
---
XLMR
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.013 | 0 | -0.0157 | -0.0102 | True
en | other | -0.0156 | 0 | -0.0188 | -0.0124 | True
hin | other | -0.0026 | 0.0854 | -0.0055 | 0.0003 | False
$p$-values: [1.218e-11, 1.218e-11, 8.538e-02]
IndicBERT
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.0144 | 0 | -0.0169 | -0.0118 | True
en | other | -0.0027 | 0.095 | -0.0057 | 0.0003 | False
hin | other | 0.0117 | 0 | 0.009 | 0.0144 | True
$p$-values: [1.22E-11, 9.50E-02, 1.22E-11]
HingRoBERTa
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.0069 | 0 | -0.0096 | -0.0042 | True
en | other | -0.0126 | 0 | -0.0158 | -0.0095 | True
hin | other | -0.0057 | 0 | -0.0086 | -0.0029 | True
$p$-values: [4.29E-09, 1.22E-11, 8.12E-06]
Table 10: Results for Tukey HSD for the entire sample size, for all models,
along with the adjusted $p$-values.

####  C.2.2 Per Label

ANOVA - Per Label
---
Model | Joy | Anger | Sadness
XLM-R | 2.09e-31 | 2.86e-10 | 1.25e-2
IndicBERT | 1.53e-19 | 3.20e-7 | 5.57e-1
HingRoBERTa | 3.74e-37 | 1.74e-9 | 2.14e-3
Table 11: $p$-values for 1-Way ANOVA on examples predicted as joy, anger and
sadness by each model.

We also conduct ANOVA tests for one positive label (joy) and two negative
labels anger, sadness, to see whether there is agreement with the $\chi^{2}$
results. Table 11 shows the $p$-values for each model. Both ANOVA and
$\chi^{2}$ find dependency between language and LIME score for the joy and
anger labels. Moreover, for sadness, both ANOVA and $\chi^{2}$ also agree that
there is a significant dependency of language and LIME score with HingRoBERTa,
and for IndicBERT there is no dependency. Where they differ slightly is with
XLM-R, where there is no dependency found with the ANOVA test, but with
$\chi^{2}$, the $p$-value is slightly below the significance threshold.

A further fine-grained analysis of these conclusions is presented with Tukey
HSD in Tables 12, 13 and 14. To summarise the results per label:

  1. 1.

Joy For both XLM-R and IndicBERT, hin and other have no meaningful difference,
but do show significant distinction between hin and eng. HingRoBERTa, on the
other hand, is able to distinguish between all language ID tags.

  2. 2.

Anger We see a significant difference between all language ID pairs and across
all models for anger.

  3. 3.

Sadness No meaningful difference is observed between hin and eng for both
XLM-R and HingRoBERTa, and for IndicBERT, there is no meaningful difference
across any of the language ID pairs.

Tukey HSD - Joy
---
XLMR
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.0222 | 0 | -0.0281 | -0.0164 | True
en | other | -0.0284 | 0 | -0.0345 | -0.0223 | True
hin | other | -0.0062 | 0.0717 | -0.0127 | 0.0004 | False
$p$-values: [0, 0, 0.718]
IndicBERT
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.0218 | 0 | -0.0277 | -0.0158 | True
en | other | -0.0169 | 0 | -0.0229 | -0.011 | True
hin | other | 0.0048 | 0.2004 | -0.0018 | 0.0114 | False
$p$-values: [0, 8.56E-11, 2.00E-01]
HingRoBERTa
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.0198 | 0 | -0.0257 | -0.0139 | True
en | other | -0.0326 | 0 | -0.0387 | -0.0266 | True
hin | other | -0.0129 | 0 | -0.0194 | -0.0063 | True
$p$-values: [8.54E-13, 8.42E-13, 1.15E-05]
Table 12: Results for Tukey HSD for examples predicted as joy by each model,
along with the adjusted $p$-values. Tukey HSD - Anger
---
XLMR
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | 0.0076 | 0.0411 | 0.0002 | 0.015 | True
en | other | -0.0103 | 0.0152 | -0.019 | -0.0016 | True
hin | other | -0.0179 | 0 | -0.0243 | -0.0115 | True
$p$-values: [4.11E-02, 1.52E-02, 1.83E-10]
IndicBERT
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.0129 | 0.0003 | -0.0207 | -0.0051 | True
en | other | -0.0216 | 0 | -0.0308 | -0.0124 | True
hin | other | -0.0087 | 0.0076 | -0.0155 | -0.0019 | True
$p$-values: [3.23E-04, 1.37E-07, 1.37E-07]
HingRoBERTa
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | 0.008 | 0.0184 | 0.0011 | 0.0149 | True
en | other | -0.0092 | 0.0258 | -0.0175 | -0.0009 | True
hin | other | -0.0172 | 0 | -0.0236 | -0.0107 | True
$p$-values: [1.84E-02, 2.58E-02, 1.50E-09]
Table 13: Results for Tukey HSD for examples predicted as anger by each model,
along with the adjusted $p$-values. Tukey HSD - Sadness
---
XLMR
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.0003 | 0.9955 | -0.0092 | 0.0085 | False
en | other | -0.0117 | 0.0323 | -0.0227 | -0.0008 | True
hin | other | -0.0114 | 0.0139 | -0.0209 | -0.0019 | True
$p$-values: [0.996, 0.032, 0.014]
IndicBERT
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.004 | 0.5855 | -0.0135 | 0.0055 | False
en | other | -0.0008 | 0.9868 | -0.0131 | 0.0114 | False
hin | other | 0.0032 | 0.7648 | -0.0075 | 0.0139 | False
$p$-values: [0.585, 0.987, 0.765]
HingRoBERTa
group1 | group2 | meandiff | p-adj | lower | upper | reject
en | hin | -0.0011 | 0.9558 | -0.0104 | 0.0081 | False
en | other | -0.0149 | 0.0067 | -0.0263 | -0.0034 | True
hin | other | -0.0137 | 0.003 | -0.0236 | -0.0039 | True
$p$-values: [0.956, 0.007, 0.003]
Table 14: Results for Tukey HSD for examples predicted as sadness by each
model, along with the adjusted $p$-values.