# Towards ASR Robust Spoken Language Understanding Through In-Context Learning
With Word Confusion Networks

###### Abstract

In the realm of spoken language understanding (SLU), numerous natural language
understanding (NLU) methodologies have been adapted by supplying large
language models (LLMs) with transcribed speech instead of conventional written
text. In real-world scenarios, prior to input into an LLM, an automated speech
recognition (ASR) system generates an output transcript hypothesis, where
inherent errors can degrade subsequent SLU tasks. Here we introduce a method
that utilizes the ASR system’s lattice output instead of relying solely on the
top hypothesis, aiming to encapsulate speech ambiguities and enhance SLU
outcomes. Our in-context learning experiments, covering spoken question
answering and intent classification, underline the LLM’s resilience to noisy
speech transcripts with the help of word confusion networks from lattices,
bridging the SLU performance gap between using the top ASR hypothesis and an
oracle upper bound. Additionally, we delve into the LLM’s robustness to
varying ASR performance conditions and scrutinize the aspects of in-context
learning which prove the most influential.

Index Terms—  spoken language understanding, in-context learning, zero-shot
learning, large language models, ASR confusion networks

##  1 Introduction

In spoken language understanding (SLU) systems, typically a spoken utterance
is first transcribed by automated speech recognition (ASR) to yield a
hypothesis, called the “$1$-best”, before being fed into a more conventional
NLU pipeline. The $1$-best often has errors that negatively impact downstream
performance.

To mitigate this issue, there has been research on using additional
information from ASR. Relevant strategies include feeding n-best alternative
hypotheses to a transformer language model [1], extending RNN language models
to lattices [2], flattening lattices and masking attention between parallel
lattice paths [3], and using word confusion networks (WCNs) to compute
weighted sums of word options or paths [4, 5].

The idea to supply additional information from ASR to decoder-based LLMs [6,
7] for SLU is under-explored, as previous work has focused on encoder models.
One recent work addressed this by providing noisy in-context examples from an
ASR model, with optional chain-of-thought (CoT) error correction for intent
classification [8]. In the CoT setup, authors give ASR transcripts of commands
from the SLURP benchmark [9], with descriptions of ASR errors found in the in-
context examples (e.g. “‘Stop price’ doesn’t make sense, which could be a
mistranscription of ‘Stock price’…”), instructing the model to correct such
errors in test examples before determining the intent. The performance with
and without CoT error correction lagged behind use of the ground-truth text.
While results improved after incorporating CoT error correction into the
prompt, only the $1$-best transcript is provided for test examples, lacking
any specific ambiguities or word options. Furthermore, the range of potential
ASR error types broadly found in test data is unlikely to be adequately
represented through a few in-context examples.

In this paper, we propose a simple representation of WCNs which can be fed to
off-the-shelf LLMs for downstream SLU tasks. Specifically, we study whether
through in-context learning [10] and our representations, LLMs can achieve
improved robustness to ASR errors and ambiguities compared to using the
$1$-best transcript. Our findings indicate that model size is critical, and
that in-context instruction and examples each provide improved performance.

Fig. 1: Proposed in-context learning framework using a WCN question transcript
for Spoken Language Understanding (SLU).

(b) how many towel|two|total|tow|tall yards did denver game|gain?

(c) how many total yards did denver game|gain?

Fig. 2: Constructing LLM input from raw WCN: (a) Raw WCN, (b) LLM input from
WCN, and (c) LLM input from WCN after filtering based on word posteriors
($p\geq 0.3$).

##  2 Methods

As shown in the pipeline depicted in Figure 1, we use an ASR model to extract
a WCN from speech, and insert a representation of this WCN into a prompt
template before passing it to an LLM.

###  2.1 Word confusion networks (WCN) input representation

####  2.1.1 ASR system and WCN construction

We use an off-the-shelf Emformer-RNNT ASR
model,111https://pytorch.org/audio/main/generated/torchaudio.models.emformer
_rnnt_model pretrained on the LibriSpeech dataset [11] to transcribe spoken
data and extract lattices, which can be used to create WCNs using the Kaldi
toolkit [12, 13]. As shown in the example in Fig. 2(a), WCNs are derived from
lattices and contain ambiguous word options aligned along the contextual
content axis. Each word option is accompanied by a posterior, which sum to one
over all alternative hypotheses at a given position in the utterance. We use
WCNs rather than lattices because they contain more paths, while lending
themselves to a simpler, more compact string representation.

####  2.1.2 LLM inputs from WCNs

Starting with a WCN for each example, we create an input that can be
interpreted by an LLM by flattening the ASR confusability axis. We separate
alternative words with a separator symbol, either a slash “$/$” or pipe “$|$”,
chosen due to their function in common written text to represent alternative
options. Optionally, we can also filter options based on a posterior
threshold. This process of constructing an input from a WCN is demonstrated in
Fig. 2(a-c). Our method requires minimal preprocessing and only modifications
to the input itself. We note that a limitation of this format is that it
cannot represent ambiguity between word hypotheses and word deletions (null
words), which can also result from ASR output. For the present study we simply
delete null alternatives.

###  2.2 In-context learning for SLU with WCNs

Since our separators may be represented in different contexts within the
training data, we optionally include a short description of the in-context
significance of the WCN separator symbol (“$/$” or “$|$”) in our prompt (known
as “WCN instruction”). In one-shot experiments, WCN instruction is always
included, and in-context examples use the same transcript source as the test
example, i.e., we use the $1$-best transcript for the in-context example when
testing with a $1$-best transcript, a WCN representation when testing on a WCN
representation, etc. Thus, the in-context example provides not only a task
demonstration, but also an example of ASR output representations.

###  2.3 Language models

For our experiments, we use the BLOOMZ-560M and BLOOMZ-3B models [14], which
were fine-tuned on a multilingual multi-task dataset. We also use a fixed
GPT-3.5-turbo model [10].

BLOOMZ ($560$M & $3$B): BLOOM stands as the inaugural open-source LLM trained
on a French government-sponsored supercomputer [15]. Its successor, BLOOMZ,
undergoes continuous fine-tuning using multi-task incremental learning [16]
and generative-aware prompting paradigms [14]. The foundational training of
the model uses an extensive public repository of HuggingFace datasets [17],
which aggregates up to $1.61$ TB of text across $46$ natural languages and
$13$ programming languages. Notably, our evaluation datasets are exempted from
this collection.

GPT-3.5 (approx. $175$B): Building upon the foundational GPT-3 model [10, 18],
GPT-3.5 employs the principles of reinforcement learning from human feedback
(RLHF) [19] to further its training. This paradigm culminates in
GPT-3.5-turbo, showcasing enhanced zero-shot learning capabilities. It’s
important to clarify that in our investigations, we have opted for a
consistent version of GPT-3.5, rather than directly interfacing with various
versions of ChatGPT.

We opt to use each LLM off-the-shelf, rather than fine-tuning them with ASR
transcripts or our WCN transcript notation. Hence, the efficacy of our methods
hinges on the models’ ability to recall from its training data that our chosen
separators (“$/$” and “$|$”) indicate word choices, as well as the capacity of
the models to learn in-context.

###  2.4 Baselines

To evaluate the efficacy of our WCN representation, we compare results to the
following baseline setups, with varying LLM inputs:

Ground-truth transcripts: the ground-truth transcript provides an upper bound,
as performance is naturally expected to degrade when using any ASR output.

ASR n-best oracle: the hypothesis from the $n$-best list with the lowest edit
distance to the ground-truth transcript (using the ASR score as a tie-
breaker). This provides a more realistic upper bound, since the WCN input is
unlikely to provide more useful information than the most correct $n$-best
hypothesis.

ASR 1-best: the simplest approach when using ASR output, this provides a
baseline for comparison, with the goal to improve over ASR 1-best by enriching
the LLM input with WCNs.

##  3 A Study of Spoken Question Answering

###  3.1 NMSQA dataset and task

The Natural Multi-speaker Spoken Question Answering (NMSQA) dataset [20]
contains spoken versions of examples from the SQuAD dataset [21, 22]. The
train and dev splits were created using a text-to-speech (TTS) system, while
crowdworkers were hired to compile the spoken test set. As with SQuAD, the
examples in the NMSQA dataset consist of a paragraph of context, a question,
and a corresponding answer which can be found directly in the context. Due to
computing limitations and small size of the original test set, we partition
the original NMSQA dev set into a small development set consisting of the
final $1000$ examples, for tuning the posterior threshold, and a test set
consisting of the first $4,800$ examples. Details on the NMSQA dataset (and
our dev/test splits) are shown in Table 1.

Table 1: NMSQA statistics Split | # questions |  $1$-best WER% |  $n$-best oracle WER%
---|---|---|---
train | $87125$ | $21.52\%$ | $14.73\%$
dev | $1000$ | $22.09\%$ | $15.43\%$
test | $4800$ | $19.35\%$ | $12.82\%$

Both SQuAD and NMSQA were created with extractive question answering in mind,
in which a model is tasked with identifying the span in the context that
contains the answer. Under our setup, we prompt the LLM to provide the answer
given a ground-truth text context and transcribed question, and measure
performance using unigram F1 and exact match (EM) metrics.

###  3.2 Prompt design and spoken QA results

The LLM prompt [23, 24, 25, 26, 27] includes a short description of the SQA
task, followed by the context and question transcript. We envision a scenario
in which an agent has access to a context document, and a user asks questions
that can be answered from the context. Accordingly, the ground-truth context
is always used, while the question uses the various transcript sources
previously described (with optional WCN instruction, as indicated in Section
2.2).

Table 2: NMSQA test-set zero-shot results.

Transcript WCN BLOOMZ-560M BLOOMZ-3B GPT-3.5 source instr. F1 EM F1 EM F1 EM
Ground-truth 58.9% 37.0% 67.1% 35.6% 71.7% 42.5% $1$-best 48.3% 29.9% 58.4%
38.2% 64.5% 36.5% “$/$” $44.0\%$ $27.0\%$ $57.2\%$ $37.1\%$ $50.5\%$ $21.0\%$
“$/$” ✓ $44.4\%$ $26.8\%$ $54.3\%$ $30.2\%$ $53.6\%$ $25.5\%$ “$/$” ($p\geq
0.3$) $47.6\%$ $29.3\%$ $58.0\%$ $37.7\%$ $56.1\%$ $27.7\%$ “$/$” ($p\geq
0.3$) ✓ $47.0\%$ $28.6\%$ $56.7\%$ $31.6\%$ $61.9\%$ $33.7\%$ “$|$” $42.1\%$
$26.2\%$ $57.1\%$ $37.1\%$ $53.0\%$ $24.6\%$ “$|$” ✓ $43.5\%$ $26.6\%$
$53.6\%$ $31.5\%$ $57.4\%$ $31.0\%$ “$|$” ($p\geq 0.3$) $47.6\%$ $29.3\%$
$58.0\%$ $37.7\%$ $56.1\%$ $27.3\%$ “$|$” ($p\geq 0.3$) ✓ $46.4\%$ $28.5\%$
$56.4\%$ $31.0\%$ $63.2\%$ $35.3\%$

Based on development set performance, we use a posterior threshold of $0.3$.
Zero-shot results with optional WCN instruction are reported in Table 2. It is
immediately evident that with each model, none of the WCN representations
achieve better performance than the $1$-best. In addition, the BLOOMZ models
show limited ability to learn via WCN instruction, with results generally
degrading when using WCN input with posterior filtering. However, use of the
WCN instruction in the prompt led to significantly improved results for
GPT-3.5, with relative improvements ranging from $5\%$ to $29\%$ over
prompting without WCN instruction.

Table 3: NMSQA test-set one-shot results..

BLOOMZ-3B GPT-3.5 Transcript source F1 EM F1 EM Ground-truth 58.1% 36.7% 76.2%
47.9% $n$-best oracle [HTML]EFEFEF47.7% [HTML]EFEFEF29.7% [HTML]EFEFEF72.4%
[HTML]EFEFEF45.1% $1$-best $\textbf{43.6}\%$ $\textbf{27.1}\%$ $70.2\%$
$43.0\%$ “$/$” $31.6\%$ $18.7\%$ $70.4\%$ $43.8\%$ “$/$” ($p\geq 0.3$)
$38.0\%$ $23.9\%$ $70.3\%$ $44.0\%$ “$|$” $32.1\%$ $19.1\%$ $69.6\%$ $44.1\%$
“$|$” ($p\geq 0.3$) $38.6\%$ $24.3\%$ $\textbf{70.5}\%$ $\textbf{44.7}\%$

One-shot performance with WCN instruction is reported in Table 3. We use one
randomly-chosen (context, question, answer) example from the training split,
with the same transcript source for the in-context example and the test
question. The BLOOMZ-560M model displayed poor ability to learn from in-
context examples, and is thus omitted. While $1$-best transcripts still
produce the best results for BLOOMZ-3B, the WCN inputs produce similar or
better results with GPT-3.5. Specifically, using “$|$” with posterior
filtering improved between $0\%$ and $4\%$ (relative) over the $1$-best
result. For the EM metric, the best WCN method closes $81\%$ of the
performance gap between the $1$-best baseline and the $n$-best oracle upper
bound.

##  4 A Study of Intent Classification

###  4.1 ATIS dataset and task

The Airline Travel Information Systems (ATIS) corpus [28] contains spoken
requests that were designed to be answered via a corresponding database, and
has been widely used in spoken intent classification (IC) and slot filling
(SF) tasks. The label space for the IC task consists of seventeen individual
intents and is highly skewed, with the top seven classes combined covering
$94\%$ of the entire dataset. The test set contains $809$ command/intent
pairs, with a $1$-best WER of $8.40\%$ and $n$-best oracle WER of $5.31\%$.

###  4.2 Prompt design and results

Similar to the NMSQA task, the LLM prompt for ATIS consists of a short
description of the task with a list of the intent classes in alphabetical
order, followed by the user command transcript.

Table 4: ATIS zero-shot intent classification results (accuracy).

Transcript source BLOOMZ-560M BLOOMZ-3B GPT-3.5 Ground-truth 69.5% 73.6% 88.1%
$n$-best oracle [HTML]EFEFEF67.5% [HTML]EFEFEF73.4% [HTML]EFEFEF86.0% $1$-best
66.8% $73.6\%$ $84.6\%$ “$/$” $51.9\%$ $73.6\%$ $84.8\%$ “$/$” ($p\geq 0.3$)
$65.4\%$ 73.7% $84.8\%$ “$|$” $61.3\%$ $73.3\%$ $84.8\%$ “$|$” ($p\geq 0.3$)
$64.3\%$ $73.6\%$ 85.0%

We report the accuracy for each model using each transcript source in Table 4.
For the smallest model, using $1$-best transcripts resulted in the best
performance, with WCN performance lagging by $1.5-15\%$. In addition,
posterior filtering was essential for achieving results that approach the
$1$-best performance. Meanwhile, the GPT-3.5 model achieved the best
performance using the WCN with “$|$” and posterior filtering, with all WCN
setups producing slightly better performance than the $1$-best. Considering
the $n$-best oracle as an upper bound, the best WCN setup is able to close
$33\%$ of the gap between the $1$-best baseline and this upper bound.

##  5 Analysis and Discussion

Based on the findings in Sections 3.2 and 4.2, the following trends emerge
that predict under what conditions WCN representations may be able to achieve
comparable or better performance than $1$-best transcripts:

  * •

Instruction: using WCN instructions improves performance with GPT-3.5.

  * •

In-context examples: adding a single in-context example improves performance
with GPT-3.5.

  * •

Posterior filtering: especially in the noisier NMSQA setting, filtering WCNs
by posteriors achieves the best performance.

  * •

Model size: the large GPT-3.5 model is able to achieve WCN performance
exceeding that from the $1$-best. Given that the key difference between
GPT-3.5 and the BLOOMZ models is size (in addition to RLHF training), we
hypothesize that the ability to disambiguate between confused word options
could be an emergent property of larger models.

###  5.1 Impact of ASR errors and failure analysis

To assess the impact of ASR errors on the performance of our methods, we
divide the NMSQA dev set into two partitions: examples where the $1$-best
$\text{WER\%}=0\%$ or $\text{WER\%}\neq 0\%$. The EM results when using
$1$-best transcripts and WCN inputs are reported in Table 5, with WCN
transcripts showing over $4\%$ relative improvement over using the $1$-best
when ASR errors exist. Meanwhile, improvements are more modest in the no-error
scenario.

Table 5: NMSQA test-set one-shot EM in examples with and without ASR errors
(overall WER$=22\%$).

Subset $\\#$ examples $1$-best “$|$” ($p\geq 0.3$) Rel. Improv w/out ASR
errors $548$ $48.2\%$ $48.6\%$ $0.83\%$ w/ ASR errors $4252$ $42.3\%$ $44.1\%$
$4.26\%$

Fig. 3: NMSQA test-set one-shot EM by question WER bin.

We more closely examine performance across a range of ASR conditions in Figure
3. By sorting NMSQA test examples into bins based on their question WER%, it
is shown that while performance is similar among examples with no word errors
(echoing Table 5), the WCN representation performance exceeds that of $1$-best
transcripts over a wide range of WER% scenarios. However, it is important to
note that under very noisy conditions (over $\sim 70\%$ WER), using the
$1$-best is preferable to WCNs.

To illustrate the trends in Figure 3, selected examples are displayed in Table
6, for (a) zero, (b) medium, and (c) high WER% scenarios. While our WCN
methods are comparable to using the $1$-best (see Table 5) in $0\%$ WER
examples, (a) contains an example in which using the WCN adds unnecessary word
options and even removes relevant words which were present in the $1$-best,
resulting the incorrect answer. Conversely, the WCN question in (b) recovers
relevant words that were deleted in the $1$-best, and the LLM generates the
correct response. Finally, in the high WER% example shown in (c), the WCN
input introduces noise with incorrect options for several words. To this
point, we note that with posterior filtering ($p\geq 0.3$), WCN questions in
the $0\%$ WER bin contain an average of $1.04$ options per word, with this
metric steadily increasing to $1.12$ options/word in the highest WER bin, in
which transcripts can become saturated with noisy (and often incorrect)
options.

Table 6: NMSQA test-set examples with varying $1$-best WER%.

Transcript source Question? Generated answer (correct/incorrect) (a) Zero WER%
Ground-truth what do these teachers not do? teach by rote $1$-best what do
these teachers not do? teach by rote “$|$” ($p\geq 0.3$) what do these
teachers knocked$|$knock to$|$due? knock (b) Medium WER% Ground-truth when was
his article published in century magazine? nineteen hundred $1$-best one was
his article published in century magaz? the problem of increasing human energy
“$|$” ($p\geq 0.3$) one$|$when was his article published in century$|$sanctary
magazine$|$magaz? nineteen hundred (c) High WER% Ground-truth who did denver
beat in the afc championship? the new england patriots $1$-best who did them
for beating the a f sea champions? the new england patriots “$|$” ($p\geq
0.3$) who did them for beating in the a$|$aye f i see$|$sea
champions$|$championship? the arizona cardinals

###  5.2 In-context example transcript format

In one-shot NMSQA experiments, the in-context example that we provide has the
same transcript representation as the test example. Thus, the one-shot setup
potentially helps via two avenues: (a) demonstration of the SQA task and (b)
demonstration of the ASR or WCN representation. To assess the impact of each,
we evaluate the NMSQA test set with WCN inputs (using “$|$” and $p\geq 0.3$),
using an in-context example with ground-truth transcripts rather than the
corresponding WCN transcripts, effectively removing the impact of (b) above.
As shown in Table 7, the performance difference between using ground-truth
transcripts versus WCN transcripts in the in-context examples is minimal,
indicating that most of the improvement when adding an in-context example is
due to demonstration of the task, rather than demonstration of the WCN
representation.

Table 7: NMSQA test-set one-shot results with “$|$” ($p\geq 0.3$) transcripts
and varying in-context transcript type.

In-context transcript type F1 EM N/A (zero-shot) $63.2\%$ $35.3\%$ Ground-
truth $70.2\%$ $\textbf{44.8}\%$ “$|$” ($p\geq 0.3$) $\textbf{70.5}\%$
$44.7\%$

###  5.3 Current limitations and future work

We consider simplicity and lack of a need for model fine-tuning to be the key
advantages of our approach, showing that current off-the-shelf LLMs can
exhibit improved robustness to ASR transcripts by representing ambiguity in
the input. However, this success was limited to GPT-3.5, and was not mirrored
by the smaller models. Since the use of WCN yielded worse results with BLOOMZ
models, the ability of smaller models to ingest and understand WCN
representations remains an open issue. It is likely that fine-tuning the LLMs
directly on ASR transcripts and WCN representations would give better results
in future studies.

##  6 Conclusions

We have proposed a simple method for representing ambiguities from ASR output
for use in SLU tasks using LLMs. Our strategy requires only preprocessing the
input, and yields improved results over 1-best transcripts when using larger
models.

## References

  * [1] Karthik Ganesan, Pakhi Bamdev, Jaivarsan B, Amresh Venugopal, and Abhinav Tushar,  “N-best ASR transformer: Enhancing SLU performance using multiple ASR hypotheses,”  in Proc. of NAACL. Aug. 2021, pp. 93–98, Association for Computational Linguistics.
  * [2] Faisal Ladhak, Ankur Gandhe, Markus Dreyer, Lambert Mathias, Ariya Rastrow, and Björn Hoffmeister,  “Lattice rnn: Recurrent neural networks over lattices,”  2016\.
  * [3] Chao-Wei Huang and Yun-Nung Chen,  “Adapting pretrained transformer to lattices for spoken language understanding,” 2020.
  * [4] Ryo Masumura et al.,  “Neural confnet classification: Fully neural network based spoken utterance classification using word confusion networks,”  in Proc. of ICASSP, 2018, pp. 6039–6043.
  * [5] Chen Liu, Su Zhu, et al.,  “Jointly encoding word confusion network and dialogue context with bert for spoken language understanding,” 2020.
  * [6] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al.,  “Voicebox: Text-guided multilingual universal speech generation at scale,”  arXiv preprint arXiv:2306.15687, 2023.
  * [7] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.,  “Neural codec language models are zero-shot text to speech synthesizers,”  arXiv preprint arXiv:2301.02111, 2023.
  * [8] Mutian He and Philip N. Garner,  “Can chatgpt detect intent? evaluating large language models for spoken language understanding,” 2023.
  * [9] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser,  “Slurp: A spoken language understanding resource package,” 2020.
  * [10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.,  “Language models are few-shot learners,”  Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  * [11] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: an asr corpus based on public domain audio books,”  in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 5206–5210.
  * [12] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, et al.,  “The kaldi speech recognition toolkit,”  in IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. Dec. 2011, IEEE Signal Processing Society,  IEEE Catalog No.: CFP11SRW-USB.
  * [13] Haihua Xu, Daniel Povey, Lidia Mangu, and Jie Zhu,  “Minimum bayes risk decoding and system combination based on a recursion for edit distance,”  Computer Speech Language, vol. 25, no. 4, pp. 802–828, 2011.
  * [14] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, et al.,  “Crosslingual generalization through multitask finetuning,”  arXiv preprint arXiv:2211.01786, 2022.
  * [15] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.,  “Bloom: A 176b-parameter open-access multilingual language model,”  arXiv preprint arXiv:2211.05100, 2022.
  * [16] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, et al.,  “Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks,”  Proc. of EMNLP, 2022.
  * [17] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al.,  “Datasets: A community library for natural language processing,”  in Proc. of EMNLP, 2021, pp. 175–184.
  * [18] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, et al.,  “Training language models to follow instructions with human feedback,”  in Advances in Neural Information Processing Systems, 2022.
  * [19] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei,  “Deep reinforcement learning from human preferences,”  NeurIPS, vol. 30, 2017.
  * [20] Guan-Ting Lin, Yung-Sung Chuang, et al.,  “DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering,”  in Proc. Interspeech 2022, 2022, pp. 5165–5169.
  * [21] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang,  “Squad: 100,000+ questions for machine comprehension of text,” 2016.
  * [22] Chia-Hsuan Lee, Szu-Lin Wu, Chi-Liang Liu, and Hung-yi Lee,  “Spoken squad: A study of mitigating the impact of speech recognition errors on listening comprehension,”  Proc. Interspeech 2018, pp. 3459–3463, 2018.
  * [23] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai,  “Promptchainer: Chaining large language model prompts through visual programming,”  in Proc. of CHI, 2022, pp. 1–10.
  * [24] Kai-Wei Chang, Yu-Kai Wang, Hua Shen, Iu-thing Kang, Wei-Cheng Tseng, Shang-Wen Li, and Hung-yi Lee,  “Speechprompt v2: Prompt tuning for speech classification tasks,”  arXiv preprint arXiv:2303.00733, 2023.
  * [25] Chao-Han Huck Yang, Yun-Yun Tsai, and Pin-Yu Chen,  “Voice2series: Reprogramming acoustic models for time series classification,”  in International Conference on Machine Learning. PMLR, 2021, pp. 11808–11819.
  * [26] Haibin Wu, Kai-Wei Chang, Yuan-Kuei Wu, and Hung-yi Lee,  “Speechgen: Unlocking the generative power of speech language models with prompts,”  arXiv preprint arXiv:2306.02207, 2023.
  * [27] Kai-Wei Chang, Wei-Cheng Tseng, Shang-Wen Li, and Hung-yi Lee,  “Speechprompt: An exploration of prompt tuning on generative spoken language model for speech processing tasks,”  arXiv preprint arXiv:2203.16773, 2022.
  * [28] Charles T. Hemphill, John J. Godfrey, and George R. Doddington,  “The ATIS spoken language systems pilot corpus,”  in Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990, 1990.


# Data-Augmentation-Based Dialectal Adaptation for LLMs

Fahim Faisal, Antonios Anastasopoulos
Department of Computer Science, George Mason University
<EMAIL_ADDRESS>
###### Abstract

This report presents GmuNLP’s participation to the DIALECT-COPA shared task at
VarDial 2024 Chifu et al. (2024), which focuses on evaluating the commonsense
reasoning capabilities of large language models (LLMs) on South Slavic micro-
dialects. The task aims to assess how well LLMs can handle non-standard
dialectal varieties, as their performance on standard languages is already
well-established. We propose an approach that combines the strengths of
different types of language models and leverages data augmentation techniques
to improve task performance on three South Slavic dialects: Chakavian,
Cherkano, and Torlak. We conduct experiments using a language-family-focused
encoder-based model (BERTić) and a domain-agnostic multilingual model
(AYA-101). Our results demonstrate that the proposed data augmentation
techniques lead to substantial performance gains across all three test
datasets in the open-source model category. This work highlights the practical
utility of data augmentation and the potential of LLMs in handling non-
standard dialectal varieties, contributing to the broader goal of advancing
natural language understanding in low-resource and dialectal settings.111Code
and data are publicly available: https://github.com/ffaisal93/dialect_copa

##  1 Introduction

Recent advancements in large language models (LLMs) have led to remarkable
performance on a wide range of natural language understanding tasks,
particularly in standard languages. However, the effectiveness of these models
on non-standard dialectal varieties remains an open question Faisal et al.
(2024). The DIALECT-COPA shared task, introduced by Ljubešić et al. (2024),
aims to bridge this gap by evaluating the commonsense reasoning capabilities
of LLMs on South Slavic dialects.

Commonsense reasoning, as originally proposed by Gordon et al. (2011),
requires models to make plausible inferences based on everyday knowledge and
understanding of the world. Extending this task to dialects poses unique
challenges, as models must capture the nuances and variations specific to
these language varieties. The DIALECT-COPA shared task provides a platform to
explore the adaptability and generalization capabilities of LLMs in this
context.

In this GmuNLP submission, we explore the potential of data augmentation
techniques in enhancing the performance of language models on dialectal
commonsense reasoning tasks. Our approach harnesses the power of state-of-the-
art LLMs to generate synthetic training data, which we combine with the
provided training dataset. By employing a diverse set of language models, we
aim to quantify the performance gains achievable through data augmentation.
Specifically, we utilize three categories of language models to maximize
dialectal task performance: (1) smaller language models that are well-suited
for low-resource settings and can be easily customized, (2) mid-size language
models that strike a balance between task-specific performance and language
understanding capabilities, and (3) closed-source language models that
generate high-quality synthetic task data to further enhance the performance
of the other two categories of language models.

We achieved the highest scores across all three test datasets in the open-
source model category. In addition, our solution performed on par with the
GPT-4 zero-shot iterative prompting approach employed by one of the teams,
demonstrating the competitiveness of the proposed approach against state-of-
the-art closed-source models. Furthermore, we achieved substantial performance
improvements for the small-scale, language-family-focused model BERTić by
combining it with our data augmentation strategy, showcasing the effectiveness
of our approach in boosting the performance of language models tailored for
low-resource settings.

The remainder of this paper is organized as follows: Section 2 provides an
overview of the DIALECT-COPA shared task and dataset, Section 3 describes our
methodology and experimental setup, Section 4 presents our results and
analysis, and Section 5 concludes the paper and discusses future directions.

##  2 The DIALECT-COPA shared task

#### Task Information

In the DIALECT-COPA shared task, a premise sentence is provided along with a
question that can be either a cause or an effect. The objective is to build a
classifier that selects the most plausible response from two candidate answer
choices based on the given premise and question. To illustrate, consider the
following training example in English, where the task is to identify the most
plausible cause:

{"premise": "My body cast a shadow over the grass.", "choice1": "The sun was
rising.", "choice2": "The grass was cut.", "question": "cause", "label": 0,
"idx": 0}

The DIALECT-COPA dataset consists of such cause-effect examples across 8
languages and dialects, challenging models to perform commonsense reasoning in
non-standard language varieties.

code |  language | train | val. | test
---|---|---|---|---
en |  English | 400 | 100 |
sl |  Slovenian | 400 | 100 |
sl-cer |  Cerkno | 400 | 100 | 500
hr |  Croatian | 400 | 100 |
hr-ckm |  Chakavian | - | - | 500
sr |  Serbian | 400 | 100 |
sr-trans |  Serbian (transliterated) | 400 | 100 |
sr-tor |  Torlak | 400 | 100 |
sr-tor-trans |  Torlak (transliterated) | 400 | 100 | 500
mk |  Macedonian | 400 | 100 |
mk-trans |  Macedonian (transliterated) | 400 | 100 |
Table 1: DIALECT-COPA dataset statistics for different languages and their
dialectal varieties.

#### Languages

The DIALECT-COPA dataset encompasses training and validation data in 7
languages, including English, 6 moderately resourced South Slavic languages,
and two related micro-dialects. The test dataset features these two micro-
dialects along with an additional previously unseen dialect. The three
dialects in the test set are as follows:

  1. 1.

The Cerkno dialect of Slovenian, spoken in the Slovenian Littoral region,
specifically in the town of Idrija.

  2. 2.

The Chakavian dialect of Croatian from the northern Adriatic, particularly
from the town of Žminj.

  3. 3.

The Torlak dialect, spoken in southeastern Serbia, northeastern North
Macedonia, and northwestern Bulgaria, with the specific test instances coming
from the town of Lebane.

Cerkno and Torlak dialects are present in all three dataset splits (training,
validation, and test) whereas, the Chakavian dialect is intentionally held out
from the training and validation splits and is exclusively encountered during
the test phase. Each dialect in the test dataset comprises 500 instances.
Table 1 presents the detailed statistics of the DIALECT-COPA dataset,
providing an overview of the distribution of instances across languages and
dialects.

##  3 Experimental Phases

In this section, we report different phases of our experiments. We step by
step perform experiments to choose appropriate base models followed by data
augmentation, combination and task-specific model tuning.

Base model |  Fine-tuning (FT)/Prompting | Epoch |  Acc. (%)
---|---|---|---
|  |  | en | hr
Aya-101 |  4 shot (2 cause, 2 effect) | - | 80 | 75
MaLA-500 |  4 shot (2 cause, 2 effect) | - | 50 | –
Llama2-CHAT (7B) |  4 shot (2 cause, 2 effect) | - | 75 | 50
BERT |  FT (eval. lang) | 3 | 66 | 55
mBERT |  FT (eval. lang) | 3 | 55 | 57
XLM-R |  FT (eval. lang) | 3 | 54 | 54
BERTić |  FT (eval. lang) | 3 | 48 | 64
Table 2: Preliminary evaluation results on the English and Croatian validation set for different base models. Data Identifier |  Description | Covered Language
---|---|---
[lang]-train |  Original DIALECT-COPA training data | en, hr, mk, sl, sl-cer, sr, sr-tor
[lang]-trans |  Transliterated (Cyrillic $\rightarrow$ Latin) training data | mk, sr, sr-tor
[lang]-claude |  Providing grammar rules and few-shot Croatian-Chakavian examples to generate synthetic parallel hr-ckm-train examples given the hr-train examples | hr-ckm
[lang]-gpt4 |  Additional synthetic English training data generated by GPT-4 Whitehouse et al. (2023) | en
[lang]-reverse |  Reverse-augmentation on [lang]-train, [lang]-trans, and [lang]-claude data | en, hr, mk, sl, sl-cer, sr, sr-tor
[lang]-nllb |  Machine translation of en-gpt4 source data to other languages using the NLLB-6B model | hr, mk, sl, sr
Table 3: Training data augmentation approaches

#### Phase 1: Model Selection

In the preliminary phase of our experiments, we conduct a series of trials to
identify base language models that demonstrate strong performance on language
understanding tasks in a multilingual context. To achieve this, we fine-tune
widely-used encoder-based models, such as BERT (Devlin et al., 2019), mBERT,
and XLM-R (Conneau et al., 2020), on the English and Croatian subsets of the
DIALECT-COPA training dataset. Additionally, we explore the potential of more
recently open-sourced large language models (LLMs) of varying sizes, such as
LLaMA-2 (Touvron et al., 2023), Aya-101 Üstün et al. (2024) and MaLA-500 (Lin
et al., 2024), to gauge their effectiveness on the task.

Our key observations from this preliminary phase are as follows:

  1. →

BERT, mBERT, and XLM-R exhibit comparable performance on the Croatian subset,
achieving an accuracy of around 55%(+/-) after 3 epochs of in-language fine-
tuning. However, the monolingual English BERT model surpasses the multilingual
models on the English subset when fine-tuned for the same number of epochs.

  2. →

BERTić (Ljubešić and Lauc, 2021), a transformer-based model pre-trained on
Bosnian, Croatian, Montenegrin, and Serbian languages, aligns well with the
target languages of the DIALECT-COPA test set. Fine-tuning BERTić on the
Croatian subset yields a notable performance improvement of approximately 12
percent (i.e. 7 percentage points) compared to the aforementioned multilingual
models.

  3. →

Employing 4-shot prompting with the LLaMA-2 7B parameter model results in
better performance on the English subset. However, for the Croatian subset,
LLaMA-2 generates random inferences. This finding aligns with expectations, as
LLaMA-2 is primarily an English-centric model and not inherently multilingual.
In an effort to address the multilingual limitations of LLaMA-2, Lin et al.
(2024) proposed MaLA-500, a multilingual adaptation of the model that
underwent fine-tuning using a causal language modeling objective. However,
after this adaptation, MaLA-500 produces random-level inferences on the
English subset.

  4. →

Aya-101, a 13B parameter mt5-xxl-based model Xue et al. (2021) instruction-
tuned in 101 languages. It shows superior performance both in English and
Croatian.

Based on these preliminary findings, we select the two best-performing models,
Aya-101 and BERTić , for further experimentation in the subsequent phases of
our study. We report our preliminary experimental findings in Table 2. The
results of our preliminary experiments are summarized in Table 2.

Setting |  Description |  Data Combination
---|---|---
o |  All original dialect-copa training data mixed together |  [en, hr, mk, sl, sl-cer, sr, sr-tor]-train [sr, sr-tor]-trans
otrsl |  Combining all original, transliterated as well as reverse-augmented and synthetic training data (only latin script ones) |  [en, hr, sl, sl-cer]-train [sr, sr-tor, mk]-trans [en, hr, mk-trans, sl, sl-cer, sr-trans, sr-tor-trans]-reverse en-gpt4, hr-ckm-claude [hr, sl, mk-trans, sr-trans]-nllb
otrslc |  Combining all original, transliterated as well as reverse-augmented and synthetic training data (Both latin and cyrillic script) |  all available training data
$\texttt{otrsl}_{\texttt{mk-hr-ckm}}$ |  Selective otrsl setting with upsampled data count by repetition for mk, hr and hr-ckm |  hr-train, mk-trans, hr-ckm-claude [hr-train, mk-trans, hr-ckm-claude]-reverse [hr, mk-trans]-nllb
$\texttt{otrsl}_{\texttt{hr-ckm}}$ |  Selective otrsl setting with upsampled data count by repetition for hr and hr-ckm |  hr-train, hr-ckm-claude [hr-train, hr-ckm-claude]-reverse hr-nllb
$\texttt{otrsl}_{\texttt{sl-cer}}$ |  Same as previous but for sl and sl-cer |  [sl, sl-cer]-train [sl, sl-cer]-reverse sl-nllb
$\texttt{otrsl}_{\texttt{sr-tor}}$ |  Same as previous but for sr and sr-tor |  [sr, sr-tor]-trans [sr-trans, sr-tor-trans]-reverse sr-nllb-trans
$\texttt{otrslc}_{\texttt{sr-tor}}$ |  Same as previous but we include both transliterated as well as Cyrillic script data |  [sr, sr-tor]-train, [sr, sr-tor]-trans [sr, sr-trans, sr-tor, sr-tor-trans]-reverse sr-nllb, sr-nllb-trans
$\texttt{otrsl}_{\texttt{mix}}$ |  Cross-lingual mix and match using all data from otrsl setting |  [en, hr, sl, sl-cer]-train [sr, sr-tor, mk]-trans [en, hr, mk-trans, sl, sl-cer, sr-trans, sr-tor-trans]-reverse en-gpt4, hr-ckm-claude [hr, sl, mk-trans, sr-trans]-nllb
$\texttt{otrsl}_{\texttt{mix-mk-hr-ckm}}$ |  Cross-lingual mix and match using all data from $\texttt{otrsl}_{\texttt{mk-hr-ckm}}$ setting |  hr-train, mk-trans, hr-ckm-claude [hr-train, mk-trans, hr-ckm-claude]-reverse [hr, mk-trans]-nllb
$\texttt{otrsl}_{\texttt{mix-hr-ckm}}$ |  Cross-lingual mix and match using all data from $\texttt{otrsl}_{\texttt{hr-ckm}}$ setting |  hr-train, hr-ckm-claude [hr-train, hr-ckm-claude]-reverse hr-nllb
$\texttt{otrslc}_{\texttt{mix-testset}}$ |  Cross-lingual mix and match using all data from otrslc setting except English |  all available training data except English
Table 4: After performing data augmentation, we create various data
combinations by merging the augmented data blocks described in Table 3 with
the original training datasets. These carefully designed data settings are
then employed to conduct task-specific fine-tuning or instruction tuning on
the selected base models, enabling us to evaluate the impact of different data
configurations and therefore, select the suitable ones for the test-set
evaluation phase.

#### Phase 2: Data Augmentation

To address the limited size of the DIALECT-COPA training dataset, which
consists of only 400 instances per language, we employ various data
augmentation techniques to expand the available training data. This step is
crucial in mitigating the data scarcity bottleneck and improving the models’
ability to generalize across diverse dialectal variations. By augmenting the
training data, we aim to provide a more representative dataset for task-
specific fine-tuning and instruction tuning of our selected language models.
The data augmentation approaches we explore include:

  1. →

The test dataset primarily contains instances written using the Latin script.
Hence, we transliterate the Macedonian (mk) dataset from Cyrillic to Latin
script to maintain consistency with the already available Serbian, and Torlak
transliterated datasets.

  2. →

For each instance in the training data, we swap the premise and the correct
answer choice, effectively transforming cause examples into effect examples
and vice versa, thereby doubling the number of training instances. For example
consider the following premise and two ‘effect’ choices:

premise: I poured water on my sleeping friend. choice1: My friend awoke. ✓
choice2: My friend snored. $\times$

Now our proposed reverse-augmentation method will transform the above example
in a ‘casue-specific’ question as follows:

premise: My friend awoke. choice1: I poured water on my sleeping friend.✓
choice2: My friend snored. $\times$

  3. →

We utilize a publicly available English COPA-style synthetic dataset generated
by GPT-4 Achiam et al. (2023), as introduced by Whitehouse et al. (2023). To
expand the coverage of this synthetic data to other languages, we translate
the English examples using the NLLB-6B machine translation model Team et al.
(2022) to all the four DIALECT-COPA standard languages: Croatian, Macedonian,
Serbian and Slovenian.

  4. →

The DIALECT-COPA dataset does not provide any training or validation data for
the Chakavian dialect. To overcome this limitation, we compile a set of
Croatian to Chakavian conversion rules and corresponding examples from online
language community forums (uni, ). In addition to these rules, we also gather
a few Croatian to Chakavian lyrics translations lyr . We then prompt the
Claude-3 language model (ant, ) with these rules and examples, instructing it
to translate the Croatian sentences from the DIALECT-COPA training set into
their Chakavian equivalents. Through this process, we create a synthetic
Chakavian training set in the style of DIALECT-COPA, which we refer to as
[lang]-claude. Here is an example with ground truth Croatian to Chakavian
translation (correctly translated words are bolded):

→ Croatian (source): Djevojka je pronašla kukca u žitaricama. Izgubila je
apetit. → Chakavian (gold-translation): Mlada je našla neko blago va
žitaricah. Je zgubila tiek. → Chakavian (claude-translation): Divojka je našla
buba u žitarican. Zgubila je tiek.

We observe that only a small number of words, specifically three in this
instance, are correctly translated from Croatian to Chakavian. Despite the
limited accuracy of the translation, this synthetic translated dataset enables
us to train and evaluate models on the Chakavian dialect, despite the absence
of original training data for this specific dialect. The detailed report on
the dialect conversion rules and the Claude-3 prompt template used for
generating the synthetic Chakavian dataset can be found in Appendix A.

Table 3 provides a comprehensive overview of the data augmentation techniques
employed and the languages covered by each approach.

#### Phase 3: Data Selection

Following the data augmentation process, we create various data combinations
by merging the augmented data with the original training datasets. Table 4
provides a comprehensive overview of the various training data combination
settings we employ, along with their respective descriptions and the specific
data sources included in each combination. These combinations are designed to
investigate the impact of different data characteristics on the performance of
our models. For instance, the otrsl setting combines all original,
transliterated, reverse-augmented, and synthetic data while excluding any data
written in the Cyrillic script. The rationale behind this combination is to
assess whether our Latin-only DIALECT-COPA test set benefits from the absence
of script variations in the training data. Additionally, we introduce a
language-agnostic data combination denoted as $\texttt{otrsl}_{\texttt{mix}}$,
in which we perform cross-lingual modifications by ensuring that the premise,
choice1, and choice2 for each example are presented in different languages.
This combination allows us to evaluate the models’ ability to handle language-
agnostic reasoning.

#### Phase 4: Prompt Design

Encoder-based models can be fine-tuned using any of the data settings created
in the previous steps. However, to perform few-shot prompting or instruction
tuning with generative language models (LLMs), we need to design prompt-based
instructions. During our preliminary experiments, we observed that using
4-shot same-class prompting (i.e., providing 4 cause examples for a cause-
based question) yields slightly better results compared to combining 2 cause
and 2 effect examples in the prompt. Specifically, this approach led to a 4.9%
improvement on the English validation set. So we opted for 4-shot same-class
prompting to perform inference.

The following prompt template is used for inference and instruction tuning of
the Aya-101 model:

Instruction: Given the premise, {premise}, What is the correct {question}
{‘before’/‘after’} this? A: {choice1} B: {choice2} Correct {question}:
{correct_answer}

By designing the prompt in this manner, we provide the model with a clear
instruction, the premise, and the two answer choices. The model is then
expected to select the correct answer based on the given question type (cause
or effect). This template is employed both during inference and instruction
tuning of the Aya-101 model to ensure consistency and optimize performance on
the DIALECT-COPA dataset.

#### Phase 5: Task-Specific Tuning

We employ two distinct approaches for task-specific tuning of our selected
models. The first approach, known as full model fine-tuning, involves updating
all the weights of the model during the training process. We apply this method
to the BERTić model, fine-tuning it for 5-10 epochs on the DIALECT-COPA
dataset. However, for mid-size models like Aya-101, full fine-tuning may be
unnecessarily computationally expensive, especially considering the limited
amount of training data available. To address this concern, we use LoRA (Low-
Rank Adaptation) adapter tuning (Hu et al., 2022) which is a more parameter-
efficient tuning approach. LoRA introduces a small number of trainable
parameters in the form of low-rank matrices, which are inserted between the
layers of the pre-trained model. Note that this draws from a long history of
efficient adaption using dedicated units Houlsby et al. (2019); Pfeiffer et
al. (2020); Faisal and Anastasopoulos (2022). During training, only these
newly introduced parameters are updated, while the original model weights
remain frozen. This approach significantly reduces the number of trainable
parameters, making it more suitable for fine-tuning on smaller datasets. By
employing LoRA adapter tuning, we can effectively adapt the Aya-101 model to
the DIALECT-COPA dataset without the need for full model fine-tuning, thereby
striking a balance between performance and computational efficiency.

##  4 Results and Discussion

In this section, we present and discuss the results of our experiments on the
DIALECT-COPA dataset.

Base Model | Setting | en | hr | mk | sl | sl-cer | sr | sr-tor | Avg. (acc)
---|---|---|---|---|---|---|---|---|---
Takeaway 1: Combining all training data helps
BERTić | Finetune (hr) | - | 64 | - | - | - | - | - | -
BERTić | o | 0.65 | 0.67 | 0.55 | 0.67 | 0.49 | 0.66 | 0.59 | 0.61
Takeaway 2: Data augmentation helps for low-resource languages in most cases
BERTić | otrslc | 0.46 | 0.7 | 0.69 | 0.67 | 0.42 | 0.68 | 0.65 | 0.61
Takeaway 3: Script choice makes a difference (Using only Latin script performs
better on Latin script evaluation)
BERTić | otrsl | 0.51 | 0.77 | 0.64 | 0.64 | 0.59 | 0.72 | 0.64 | 0.65
Takeaway 4: Cross-lingual mix-and-match effect: inconclusive
BERTić | $\texttt{otrsl}_{\texttt{mix}}$ | 0.56 | 0.76 | 0.68 | 0.57 | 0.52 | 0.66 | 0.63 | 0.63
Takeaway 5: Upsampling certain language groups might help targeted evaluation
in some cases
BERTić | $\texttt{otrsl}_{\texttt{sr-tor}}$ | 0.55 | 0.74 | 0.63 | 0.61 | 0.48 | 0.64 | 0.66 | 0.62
BERTić | $\texttt{otrsl}_{\texttt{sl-cer}}$ | 0.54 | 0.68 | 0.65 | 0.57 | 0.58 | 0.66 | 0.64 | 0.62
Takeaway 6: Instruction tuning helps
Aya-101 | 4-shot | 0.83 | 0.77 | 0.75 | 0.76 | 0.62 | 0.81 | 0.73 | 0.75
Takeaway 7: Further task-specific instruction tuning helps even more
Aya-101 | otrsl | 0.86 | 0.79 | 0.81 | 0.91 | 0.7 | 0.82 | 0.77 | 0.81

Takeaway 7: Training for 10 epochs instead of 5: inconclusive |
---|---
setting | epochs | en | hr | mk | sl | sl-cer | sr | sr-tor | mean | max count
otrsl | 10 | 0.52 | 0.74 | 0.65 | 0.56 | 0.61 | 0.67 | 0.59 | 0.62 | 5
| 5 | 0.50 | 0.72 | 0.62 | 0.57 | 0.58 | 0.65 | 0.61 | 0.61 | 2
$\texttt{otrsl}_{\texttt{mk-hr-ckm}}$ | 10 | 0.49 | 0.76 | 0.66 | 0.62 | 0.51 | 0.67 | 0.70 | 0.63 | 1
| 5 | 0.48 | 0.76 | 0.68 | 0.64 | 0.52 | 0.71 | 0.67 | 0.64 | 4
$\texttt{otrsl}_{\texttt{sl-cer}}$ | 10 | 0.56 | 0.64 | 0.65 | 0.62 | 0.58 | 0.65 | 0.60 | 0.62 | 5
| 5 | 0.54 | 0.68 | 0.65 | 0.57 | 0.58 | 0.66 | 0.64 | 0.62 | 5
$\texttt{otrslc}_{\texttt{sr-tor}}$ | 10 | 0.50 | 0.70 | 0.65 | 0.55 | 0.48 | 0.62 | 0.68 | 0.60 | 4
| 5 | 0.49 | 0.69 | 0.63 | 0.60 | 0.51 | 0.62 | 0.70 | 0.61 | 4

Table 5: Takeaways from incremental experiments performed on the DIALECT-COPA validation dataset. The best language-specific scores for each setting are underlined (Takeaway 7). Team | Base Model |  System Description | sl-cer | hr-ckm | sr-tor | Avg. (acc)
---|---|---|---|---|---|---
Closed Source Model Weights
JSI | GPT-4 |  10-shot with first 10 test instances (without answer) | 0.734 | 0.890 | 0.974 | 0.866
UNIRI | GPT-4 |  RAG implementation; Chakavian and Cerkno lexical dictionary; Reasoning instruction and self referral grading task | 0.708 | 0.764 | - | -
UNIRI | GPT-4 |  0-shot iterative prompt | 0.664 | 0.774 | 0.894 | 0.777
Open Source Model Weights
GmuNLP | Aya-101 |  4-shot prompting | 0.694 | 0.756 | 0.840 | 0.763
GmuNLP | Aya-101 |  LORA adapter tuning on $\texttt{otrsl}_{\texttt{hr-ckm}}\rightarrow$ 4-shot prompting | 0.700 | 0.750 | 0.824 | 0.758
GmuNLP | Aya-101 |  LORA adapter tuning on $\texttt{otrsl}\rightarrow$ 4-shot prompting | 0.682 | 0.760 | 0.824 | 0.755
GmuNLP | Aya-101 |  LORA adapter tuning on $\texttt{otrsl}_{\texttt{hr-ckm}}\rightarrow$ 4-shot prompting | 0.660 | 0.742 | 0.848 | 0.750
WueNLP | Mixtral |  LORA adapter tuning on standard variety of target dialect | 0.556 | 0.606 | 0.738 | 0.633
CLaC | XLM-R |  Fine-tuning XLM-RoBERTa base for multiple choice QA task | 0.564 | 0.522 | 0.570 | 0.552
Table 6: Performance comparison of different submissions on DIALECT-COPA test set. base model | setting | epoch | sl-cer | hr-ckm | sr-tor | Avg. (acc)
---|---|---|---|---|---|---
Aya-101 | 4-shot | - | 0.694 | 0.756 | 0.840 | 0.763
Aya-101 | $\texttt{otrslc}_{\texttt{mix-mk-hr-ckm}}$ | 5 | 0.690 | 0.756 | 0.836 | 0.761
Aya-101 | $\texttt{otrsl}_{\texttt{mix-hr-ckm}}$ | 5 | 0.700 | 0.750 | 0.824 | 0.758
Aya-101 | otrsl | 5 | 0.682 | 0.760 | 0.824 | 0.755
Aya-101 | $\texttt{otrsl}_{\texttt{mk-hr-ckm}}$ | 5 | 0.660 | 0.742 | 0.848 | 0.750
Aya-101 | $\texttt{otrsl}_{\texttt{sl-cer}}$ | 5 | 0.686 | 0.718 | 0.836 | 0.747
BERTić | $\texttt{otrsl}_{\texttt{hr-ckm}}$ | 10 | 0.572 | 0.626 | 0.722 | 0.640
BERTić | $\texttt{otrsl}_{\texttt{mk-hr-ckm}}$ | 5 | 0.582 | 0.634 | 0.682 | 0.633
BERTić | $\texttt{otrslc}_{\texttt{mix-testset}}$ | 5 | 0.576 | 0.622 | 0.692 | 0.630
BERTić | otrsl | 10 | 0.540 | 0.622 | 0.700 | 0.621
Table 7: GMUNLP system submissions for test-set evaluation. The best dialect-
specific scores for each base-model type are bolded.

###  4.1 Validation Set Insights

Table 5 summarizes the key takeaways from our incremental experiments
conducted on the validation dataset using BERTić and Aya-101. First, we
observe that combining datasets from multiple languages boosts performance on
the Croatian subset, as opposed to training on a single language. This finding
motivates us to utilize all available training data and prepare different data
combinations, as described in Table 4. Second, we find that increasing the
data quantity through various data augmentation techniques (Table 3) primarily
improves performance for most languages and low-resource dialects.
Furthermore, discarding instances written in the Cyrillic script can boost
performance for certain languages and dialects (e.g., Croatian, Cerkno, and
Serbian), while hurting others.

We also explore cross-lingual mix-and-match strategies, but we do not find any
conclusive patterns indicating that this approach consistently makes the model
more language-agnostic, as it helps in some cases while hindering performance
in others. Additionally, we experiment with discarding English examples and
upsampling specific language groups (e.g., Serbian and Torlak examples for the
Torlak dialect), which leads to slight performance improvements for the Torlak
dialect.

Notably, we observe that full fine-tuning of the comparatively smaller, non-
instruction-tuned, but language-specific BERTić model cannot surpass the
performance of the multilingual, instruction-tuned Aya-101 model. Finally, we
apply the same data combinations to perform instruction tuning on the Aya-101
model and observe an overall performance boost. However, our experiments with
different numbers of training epochs (5 and 10) yield inconclusive findings.

###  4.2 Test Set Insights

#### Team-specific ranking

Table 6 presents a comparison of the best-performing submissions from
different teams on the DIALECT-COPA test set. We categorize the submissions
into two groups: Category 1 includes teams that utilize closed-source model
weights, while Category 2 consists of teams that rely on open-source model
weights. Our submissions belong to the latter category. We observe that the
closed-source GPT-4 model achieves the best overall performance. Team JSI
employs GPT-4 with a 10-shot prompting approach, where they provide the first
10 test instances without revealing the answers. Interestingly, even the
0-shot prompting using GPT-4 (by team UNRI) outperforms all submissions in
Category 2 using open-source models. Among the Category 2 submissions, GMUNLP
(our submission) achieves the highest performance on all varieties. The base
Aya-101 model with 4-shot prompting yields the best average score across all
languages. However, LoRA adapter tuning on different data combinations results
in language-specific best scores.

####  GmuNLP submission

Table 7 presents the results of our selected 10 system submissions. We observe
that the best performance achieved by the BERTić model on the otrsl setting is
62%, which is approximately 17% lower compared to the otrcl-tuned Aya-101
model. When comparing language-specific results, we find that the Torlak (sr-
tor) dialect is the easiest to predict for both the Aya-101 and BERTi’c
models, while the Cerkno dialect (sl-cer) proves to be the most challenging to
learn.

Interestingly, upsampling the Cerkno dialect-related data
($\texttt{otrsl}_{\texttt{sl-cer}}$-tuned) does not yield the best score for
the Cerkno test-set. Instead, upsampling the Chakavian dialect-related data
using the $\texttt{otrsl}_{\texttt{mk-hr-ckm}}$ setting leads to better scores
on the Cerkno test set. This observation holds true for both the Aya-101 and
BERTić base models, indicating that leveraging data from more closely related
languages does not always provide the most significant benefit. We believe
this phenomenon warrants further investigation to gain a deeper understanding
of the complex interplay between language-relatedness and task-specific model
performance.

##  5 Conclusion

In this study, we explored the impact of data augmentation techniques on fine-
tuning multilingual language models for improving common sense reasoning in
dialectal variations. Our experiments encompassed a range of language models,
from smaller to mid-sized architectures, to investigate their adaptability to
dialectal nuances. The observed variations in performance and the upper limits
achieved by different models reflect the diverse ways in which language models
handle and adapt to dialectal variations. The insights gained from this work
may contribute to the development of more robust and adaptable language models
that can handle the challenges posed by dialectal variations. Future work can
explore advanced data augmentation techniques, investigate the impact of
domain-specific knowledge integration, and develop novel architectures
tailored to the unique characteristics of dialects.

## Acknowledgements

This work has been generously supported by the National Science Foundation
under grants IIS-2125466 and IIS-2327143. We are thankful to the anonymous
reviewers and area chairs for their constructive feedback. This project was
supported by resources provided by the Office of Research Computing at George
Mason University (https://orc.gmu.edu) and funded in part by grants from the
National Science Foundation (Awards Number 1625039 and 2018631).

## References

  * (1) Claude — anthropic.com.  https://www.anthropic.com/claude.  [Accessed 25-03-2024].
  * (2) Croatian-chakavian features -UniLang — forum.unilang.org.  https://forum.unilang.org/viewtopic.php?t=14771.  [Accessed 28-03-2024].
  * (3) Elitni Odredi - Ljubavi Moja lyrics + Croatian (Chakavian dialect) translation — lyricstranslate.com.  https://lyricstranslate.com/en/ljubavi-moja-jubavi-moja.html.  [Accessed 28-03-2024].
  * Achiam et al. (2023) OpenAI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim’on Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Adeola Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M’ely, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O’Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer’on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023.  Gpt-4 technical report.
  * Chifu et al. (2024) Adrian Chifu, Goran Glavaš, Radu Ionescu, Nikola Ljubešić, Aleksandra Miletić, Filip Miletić, Yves Scherrer, and Ivan Vulić. 2024.  VarDial evaluation campaign 2024: Commonsense reasoning in dialects and multi-label similar language identification.  In _Eleventh Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2024)_ , Mexico City, Mexico. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Faisal et al. (2024) Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024.  Dialectbench: A nlp benchmark for dialects, varieties, and closely-related languages.
  * Faisal and Anastasopoulos (2022) Fahim Faisal and Antonios Anastasopoulos. 2022.  Phylogeny-inspired adaptation of multilingual models to new languages.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 434–452, Online only. Association for Computational Linguistics.
  * Gordon et al. (2011) Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Lin et al. (2024) Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, and Hinrich Schütze. 2024.  Mala-500: Massive language adaptation of large language models.
  * Ljubešić and Lauc (2021) Nikola Ljubešić and Davor Lauc. 2021.  BERTić - the transformer language model for Bosnian, Croatian, Montenegrin and Serbian.  In _Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing_ , pages 37–42, Kiyv, Ukraine. Association for Computational Linguistics.
  * Ljubešić et al. (2024) Nikola Ljubešić, Nada Galant, Sonja Benčina, Jaka Čibej, Stefan Milosavljević, Peter Rupnik, and Taja Kuzman. 2024.  DIALECT-COPA: Extending the standard translations of the COPA causal commonsense reasoning dataset to south slavic dialects.  In _Eleventh Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2024)_ , Mexico City, Mexico. Association for Computational Linguistics.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020.  Adapterhub: A framework for adapting transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations_ , pages 46–54, Online. Association for Computational Linguistics.
  * Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Whitehouse et al. (2023) Chenxi Whitehouse, Monojit Choudhury, and Alham Aji. 2023.  LLM-powered data augmentation for enhanced cross-lingual performance.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 671–686, Singapore. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Üstün et al. (2024) Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024.  Aya model: An instruction finetuned open-access multilingual language model.

##  Appendix A Croatian to Chakavian conversion rules

Here we report the collected Croatian to Chakavian conversion rules and their
corresponding examples uni , the lyrics translations lyr  and the Claude-3
prompt we used to generate the Cahakavian synthetic translations.

#### Conversion rules

    Rule: m’s at the end of words become n’s
    Examples:
    Ja sam = Ja san
    osam = osan
    s enom = s enon
    vidim = vidin

    Rule:  becomes j
    Examples:
    mlai=>mlaji
    meu=>meju

    Rule: The sounds  and  actually do not
    exist separately from  and d

    Rule: genitive plural in the feminine and
    neuter takes a zero ending:
    Examples: ena = en, sela = sel

    Rule: dative/instrumental plurals are
    also somewhat different.

    Rule: Masculine and neuter nouns can have
    an alternate ending.
    Examples:
    it can be gradovima or gradoviman

    Rule: For feminine nouns, it’s shortened.
    Examples:
    enama=>enan

    Rule:-ao endings are shortened to just -a
    Examples:
    iao je => ia je
    rekao sam => reka san

    Rule:-io endings change to -ija
    Examples:
    govorio je => govorija je
    vidio sam => vidija san

    Rule:a, aside from meaning "what" can
    also be used as a particle meaning "away"
    or "out".
    Examples:
    gremo a (let’s get out of here)

    Rule: some of the third person plural
    forms can often be extended into longer
    -u ending forms.
    Examples:
    govore -> govoru -> govoridu
    rade -> radu -> radidu
    piu -> piedu

    Rule: Infinitive is shortened .
    There is no I at the end.
    In some speeches there is neither T/ at
    the end
    Examples:
    bit =>bi
    pivat =>piva
    znat => zna
    plivat => pliva
    ronit => roni

    Rule: change lj=>j
    Examples:
    zaljubiti se=>zajubit se
    ljubav =>jubav
    ljuska =>juska
    ljudi=>judi

    Rule: Change O => E in some cases
    Example:
    nekoga-nikega
    svakoga-svakega
    tomu-temu
    toga-tega
    bijeloga-bilega
    jednoga-jenega
    jednomu-jenemu

    Rule: In standard croatian third person
    plural has ending in some verbs E.
    In chakavian it is always U
    Examples:
    vide =>vidu
    hoe =>hou
    stoje =>stoju
    stave =>stavu
    motre =>motru
    lee=>leu

    Rule: Change H=>V
    Examples:
    kruh =>kruv
    kuhati =>kuvati
    suh =>suv
    gluh =>gluv[/b]

#### Lyrics translations

We collected Croatian to Chakavian lyrics translations from the
lyricstranslate.com site lyr .

#### Claude-3 prompt

We use the following prompt consisting the the above mentioned conversion
rules, examples and lyrics translations:

    <conversion_file.txt>

    This file contains Croatian to Chakavian
    dialect conversion grammar rules with
    examples.

    Now here are some Croatian sentences and
    it’s parallel Chakavian sentences:

    <Croatian_lyrics.txt>
    <Chakavian_lyrics.txt>.

    Given these resources, I want you to translate
    the following Croatian sentences to Chakavian
    dialect.

    <Croatian_training_sentences.txt>


# CAISA at SemEval-2023 Task 8: Counterfactual Data Augmentation for
Mitigating Class Imbalance in Causal Claim Identification

Akbar Karimi1,2 Lucie Flek1,2
1Conversational AI and Social Analytics (CAISA) Lab, University of Bonn,
Germany
2Conversational AI and Social Analytics (CAISA) Lab, University of Marburg,
Germany
{akkarimi<EMAIL_ADDRESS>
https://caisa-lab.github.io

###### Abstract

The class imbalance problem can cause machine learning models to produce an
undesirable performance on the minority class as well as the whole dataset.
Using data augmentation techniques to increase the number of samples is one
way to tackle this problem. We introduce a novel counterfactual data
augmentation by verb replacement for the identification of medical claims. In
addition, we investigate the impact of this method and compare it with 3 other
data augmentation techniques, showing that the proposed method can result in a
significant (relative) improvement in the minority class.

##  1 Introduction

Automatic identification of medical claims Khetan et al. (2023); Wadhwa et al.
(2023) is a task with various real-life applications in industries such as
healthcare Herland et al. (2017) and insurance Wang and Xu (2018) as well as
content moderation Schlicht et al. (2023). However, it can be a difficult task
due to the lack of data for all or some categories. One solution for such an
issue is increasing the number of data points in each category, especially the
one that has significantly fewer samples. We can do this using data
augmentation techniques Temraz and Keane (2022), which modify certain
characteristics of an input sequence (or its representation in the embedding
space) in order to create different versions of it. One example is entity
replacement Zeng et al. (2020), where entities in one sequence can be swapped
with equivalent ones from another sequence. The advantage of this type of
augmentation is that it provides more real context to the target entities.

Given that the task at hand is claim detection, we hypothesize that the verb
in a sentence can be determinant in its category. Therefore, we address the
problem of class imbalance using a novel data augmentation technique where we
replace a verb in a sentence with other verbs from the training data. Our
experiments show that verb replacement can improve the performance of a model
on the target category. In addition, for more comparison, we experiment with
several other data augmentation techniques, namely noise insertion Karimi et
al. (2021b), entity replacement Zeng et al. (2020), augmentation with
YouChat111https://you.com/chat, and augmentation in the embedding space Karimi
et al. (2021a).

##  2 Background

Class Imbalance Problem. This problem frequently comes up in many domains and
applications. As a result, it has been tackled by a variety of methods such as
oversampling Ling and Li (1998) and undersampling He and Garcia (2009). The
former method randomly selects some of the samples in the minority class and
uses them multiple times for training in addition to the original samples.
Contrarily, the latter randomly ignores some of the training examples from the
majority class. However, the issue with them is that one (oversampling) might
not always add new information to the training data, and the other
(undersampling) might lose valuable information by not using some of the data
points.

Data Augmentation. Another solution to tackling the class imbalance problem is
to create synthetic instances from the existing ones Chawla et al. (2002).
With this method, the resulting samples can be more diverse which can help
avoid overfitting. However, the trade-off is that it can also introduce noise
to the system although introducing noise is not always harmful Karimi et al.
(2021b). In counterfactual data augmentation, words (or phrases) in a sentence
are replaced with opposite (or different) ones from other sentences. This way,
the focus parts of sentences are combined with different contexts, helping
models in a better generalization to unseen sentences and combinations in the
original data. For example, Zeng et al. (2020) replace named entities from one
sentence with another one to create new samples for the task of named entity
recognition. We use the same approach for tackling PIO extraction (Subtask 2).
To do so, we first create a dictionary of all the PIOs. Then, to augment a
sentence in the training set, we replace its PIOs randomly with other ones
from the dictionary.

We compare the performance of the entity replacement with two other
augmentation techniques. One is our counterfactual verb replacement method
that we also used for Subtask 1 and the second one is an augmentation
technique called BAT Karimi et al. (2021a) that takes place in the embedding
space instead of the input space.

###  2.1 Data Exploration

The dataset Wadhwa et al. (2023) for the first task consists of 5710 texts
that we split into two sets of training and development. Tables 1 and 2 show
the number of samples for each set as well as the test set. As we can see from
the tables, some texts can be longer than 1000 tokens. However, due to their
low frequency (Figure 1), we train our baseline (DistilBERT) with 512 tokens.

Data | Texts | Unique words | Max length
---|---|---|---
Train | 5016 | 39685 | 1777
Dev | 694 | 11287 | 1040
Test | 1424 | 16948 | 7876
Table 1: Dataset statistics in Subtask 1 Data | Texts | Unique words | Max length
---|---|---|---
Train | 501 | 10834 | 1802
Dev | 96 | 3570 | 887
Test | 150 | 4418 | 657
Table 2: Dataset statistics in Subtask 2

Figure 1: Sentence length distributions in the training data for Subtasks 1
and 2.

The categories in the dataset in Subtask 1 include Claims (CLA), Claim per
experience (EXP), Per experience (PER), and Questions (QUE). Additionally, the
dataset for Subtask 2 includes the categories of Population (POP),
Intervention (INT), and Outcome (OUT). Tables 3 and 4 show the number of
tokens for each category in Subtasks 1 and 2, respectively. In the former, the
claims (CLA) category has significantly fewer samples than other categories.
In the latter, on the other hand, all the entities are vastly outnumbered by
the outside (O) class.

Data | CLA | EXP | O | PER | QUE
---|---|---|---|---|---
Train | 8183 | 33358 | 316676 | 138359 | 51707
Dev | 1045 | 3995 | 44172 | 18298 | 6803
Table 3: Label distribution of train and dev data in Subtask 1. Data | INT | O | OUT | POP
---|---|---|---|---
Train | 800 | 69594 | 768 | 444
Dev | 185 | 12883 | 143 | 78
Table 4: Label distribution of train and dev data in Subtask 2

###  2.2 Data Augmentation Methods

We experiment with four data augmentation methods for the identification of
causal claims (Subtask 1) and three methods for PIO extraction (Subtask 2).
Table 5 shows an example for each method.

Original sentence |  80% of people diagnosed with IBS have Sibo.
---|---
ER |  100 percent of people diagnosed with IBS have Sibo.
VR (random) |  80 % of people diagnosed with IBS cause Sibo .
VR (antonym) |  80 % of people diagnosed with IBS abstain Sibo .
AEDA |  80% of people diagnosed with IBS ! have Sibo.
YouChat |  Only a small fraction of those diagnosed with IBS actually have Small Intestinal Bacterial Overgrowth (SIBO).
Table 5: An example augmentation from the claims class by the four
augmentation methods, ER (entity replacement), VR (verb replacement), AEDA (an
easier data augmentation).

AEDA Karimi et al. (2021b). This approach is based on inserting punctuation
marks into the input sentences. This can help improve the generalization
capability of the model by changing the position of the words in the input
sentence.

Entity Replacement Zeng et al. (2020). This method replaces the existing named
entities with similar ones in the training dataset. In order to implement it,
we first need to extract the entities from the sequences. We can do this using
an off-the-shelf model for named entity recognition such as the FLAIR model
Akbik et al. (2018). Then, we create a dictionary of the named entities and in
augmentation, we randomly choose one from the dictionary to replace with the
original entity. One problem with this approach is that some sentences might
not contain any entities. This will result in some sentences being repeated or
discarded from the augmented batch.

Verb Replacement. Some verbs might be more indicative of the category to which
they belong. To take advantage of this, we can create more diverse sequences
by just replacing the verbs that are present in them. When replacing the
verbs, we keep their tense intact. In addition, we experiment with two
different ways for verb replacement. In one case, we first create a dictionary
of verbs in the training data and select randomly from them when replacing a
verb in a sentence. In the second case, we replace the verb with an antonym
using WordNet Miller (1995). The reason for not using the training data is
that antonyms are rare and they might not be found in the data.

Augmentation with YouChat. YouChat is a chatbot that can perform various
guided actions such as augmenting sentences by producing contradictory ones.
To do that we come up with a framing for our prompts that encourages diversity
in the output as well as a contradiction. The reason for producing
contradictory sentences is that, for the categories that, both the original
and its contradiction can belong to the same category. For instance, for the
claims category, if one sentence is considered to be a claim, then its
contradiction can also be seen as a different claim.

We use two prompts for pushing YouChat to produce diverse and counterfactual
sentences: 1) Contradict this sentence with colorful words "original
sentence", and 2) Without using despite, while, and although, contradict this
sentence with colorful words "original sentence". We use the first prompt to
augment half of the sentences in the claims category. However, one problem
that we notice with the outcome of this prompt is that after a couple of
outputs, YouChat begins all sentences with expressions such as although,
despite, and while. In order to change this, we augment the second half using
the second prompt. This results in augmentations with different sentence
structures.

Augmentation with Adversarial Examples. The BAT model Karimi et al. (2021a)
trains the pre-trained language model in an adversarial manner where
adversarial examples are created during the training in the embedding space.

Figure 2: Workflow of our system333Figure drawn using https://draw.io. Data | CLA | EXP | O | PER | QUE
---|---|---|---|---|---
Train | 401 | 1917 | 19826 | 7824 | 5064
Dev | 49 | 235 | 2666 | 995 | 633
Table 6: Number of sentences for each category after sentence tokenization in Subtask 1. | Method | CLA | EXP | O | PER | QUE | Precision | Recall | F1
---|---|---|---|---|---|---|---|---|---
Baseline | CRF | 11.1 | 12.7 | 76.5 | 49.5 | 71.0 | 51.1 | 42.1 | 44.1
DistilBERT | 25.8 | 32.0 | 77.6 | 56.8 | 76.4 | 57.9 | 52.0 | 53.7
400 | CRF + ER | 13.7 | 12.1 | 77.4 | 48.2 | 70.3 | 56.1 | 41.7 | 44.3
CRF + VR (random) | 11.3 | 15.1 | 76.4 | 50.2 | 69.5 | 50.5 | 42.4 | 44.5
CRF + VR (antonym) | 11.7 | 8.9 | 75.9 | 49.6 | 69.2 | 47.2 | 41.6 | 43.1
CRF + AEDA | 5.4 | 4.8 | 77.0 | 47.8 | 67.7 | 44.5 | 39.5 | 40.6
CRF + YouChat | 9.2 | 12.0 | 75.3 | 49.1 | 68.7 | 46.1 | 41.4 | 42.9
100 | DB + ER | 27.9 | 29.3 | 77.5 | 56.5 | 76.8 | 56.7 | 52.3 | 53.6
DB + VR (random) | 27.9 | 34.2 | 77.5 | 56.2 | 76.6 | 58.3 | 52.8 | 54.5
DB + VR (antonym) | 27.5 | 29.1 | 77.6 | 56.3 | 77.2 | 57.7 | 52.0 | 53.5
DB + AEDA | 25.5 | 28.5 | 77.8 | 56.7 | 76.9 | 56.3 | 51.9 | 53.1
DB + YouChat | 26.2 | 35.8 | 77.6 | 56.9 | 76.1 | 59.0 | 52.9 | 54.5
400 | DB + ER | 29.7 | 29.8 | 77.1 | 57.1 | 77.2 | 56.7 | 53.1 | 54.2
DB + VR (random) | 22.3 | 29.9 | 77.7 | 56.5 | 77.2 | 56.5 | 51.3 | 52.7
DB + VR (antonym) | 18.6 | 32.1 | 77.4 | 56.6 | 76.3 | 57.0 | 50.7 | 52.2
DB + AEDA | 28.9 | 35.2 | 77.6 | 56.7 | 76.7 | 59.1 | 53.1 | 55.0
DB + YouChat | 21.0 | 30.1 | 77.0 | 57.0 | 76.3 | 54.8 | 51.1 | 52.3
Table 7: Subtask 1. Experiments with 100 and 400 augmented samples for the
claims class with DistilBERT (DB) and CRF models using Verb Replacement (VR),
Entity Replacement (ER), AEDA, and YouChat augmentations. Green shows the best
performer, blue is the second best, and red is the worst.

##  3 System Overview

The annotated data gives us the span of each category. The spans can be
complete sentences or part of a sentence. One approach to address the task is
to frame it as a token classification task, similar to named entity
recognition tasks. However, named entities seem to be easier to spot because
of their locality. On the contrary, given that a longer sequence of words
could belong to a category, we take a broader look to recognize them. As a
result, we formulate the problem as sentence classification. The dataset
statistics indicate that in more than 87 percent of the resulting sentences,
all the words have the same label.

###  3.1 Workflow

As can be seen in Figure 3, we first split the texts into sentences using a
sentence tokenizer toolkit from the NLTK library Loper and Bird (2002). Then,
for training, we assign the label of the majority of the tokens to the
sentence and train the model with the resulting data. For the inference part,
after sentence tokenization, we classify each sentence using the trained model
and assign the sentence label to the individual tokens.

###  3.2 Sentence-Tokenized Dataset Statistics

Separating the input texts into sentences results in just over 35K sentences
which are distributed heavily in favor of the outside (O) class. Table 6 shows
the statistics of the resulting data. As we can see, only a small proportion
of the sentences belong to the claims category. We augmented the samples in
this category to mitigate the class imbalance.

##  4 Baseline Models

We compare the performance of augmentation methods with two baseline models,
namely a conditional random fields Lafferty et al. (2001) model and the
DistilBERT pre-trained language model for Subtask 1 Sanh et al. (2019), and
the BioBERT model Lee et al. (2020) for Subtask 2.

Conditional Random Fields (CRF). This model is particularly suited for
sequence labeling tasks. It considers a set of manually defined feature
functions to predict the label of a token. In our case, we only consider some
simple features such as the word itself, the word endings, whether the word is
uppercase or lowercase, and if it is a number or not. With the same features,
we also consider bigrams.

DistilBERT. This model is a lighter and more robust version of the BERT model
Devlin et al. (2019). We also consider the performance of this model without
any augmentation as one of the baselines.

BioBERT. This is another variant of the BERT model that has been trained on
medical texts in addition to the general text used for training BERT.

##  5 Results and Analysis

We perform augmentation on the claims class for Subtask 1 and the whole
dataset for Subtask 2.

###  5.1 Subtask 1: Causal Claim Identification

Method | CLA | EXP | O | PER | QUE | Precision | Recall | F1
---|---|---|---|---|---|---|---|---
DistilBERT | 25.8 | 32.0 | 77.6 | 56.8 | 76.4 | 57.9 | 52.0 | 53.7
DB + ER | 23.8 | 27.8 | 76.6 | 55.1 | 77.1 | 54.7 | 51.1 | 52.1
DB + VR (random) | 15.4 | 36.5 | 77.7 | 56.3 | 77.4 | 57.7 | 51.2 | 52.7
DB + AEDA | 24.4 | 32.1 | 76.9 | 55.6 | 76.9 | 57.7 | 51.6 | 53.2
Table 8: Subtask 1. Results with 4 augmentations for all 400 samples in claims
class with DistilBERT (DB) using Verb Replacement (VR), Entity Replacement
(ER), and AEDA methods.

For this task, the CRF model provides a relatively well-performing baseline
despite its simplicity. Notably, from Table 7, we can see that it does well on
the QUE class with 71 percent. This can be attributed to the fact that the CLA
class is easier to detect although the number of samples in this category is a
lot lower than the PER class that has a performance of 49.5 percent. Quite
understandably, the lowest performing classes were the CLA and EXP classes,
which can be due to having only a small number of samples in addition to their
difficulty.

DistilBERT, on the other hand, shows an almost 10 percent overall improvement
as well as on individual classes over the CRF model, which is expected given
the large number of parameters it has compared to CRF ($\approx$68M vs.
$\approx$10M).

Impact of augmentation on CRF. The impact of augmentations with the CRF model
is somewhat mixed. While ER helps the claims class improve by two percent,
others show no improvement or negative improvement. It is possible that the
CRF model is more vulnerable to out-of-distribution changes.

Impact of augmentation on DistilBERT. Considering the effect of the
augmentation methods on the overall performance of DistilBERT, we experiment
with two scenarios: first with 100 augmented sentences and then with 400
augmentations. Table 7 shows that, in the first case, VR (random) and YouChat
have helped the model improve by almost one point while the ER method has had
a slightly negative effect. The effect on the minority class, however, was
positive for all the augmentation methods except for AEDA, with ER and VR
(random) showing more than two percent improvement and YouChat 0.4 percent.
With 400 augmentations, we see that only ER and AEDA have improved the class
performance. This can be attributed to the increase in the amount of noise as
we include more augmentations.

Impact of multiple augmentations. In this experiment, we investigate how
multiple augmentations can impact the DistilBERT model on the studied dataset.
Therefore, for each tokenized sentence, we produce four augmentations. We do
this only for ER, VR (random), and AEDA since for YouChat the manual work is
time-consuming and for VR (antonym), there is only one antonym for a verb. As
we can see from Table 8, more augmentations of the claims class have a
negative effect on the class itself while improving the results on the EXP
(claim per experience) class. Given that this class also contains claims, it
seems that more data for the claims class could also help claims per
experience class.

###  5.2 Subtask 2: PIO Frame Extraction

Method | Precision | Recall | F1
---|---|---|---
BioBERT | 47.2 | 11.7 | 18.8
BioBERT + ER | 32.5 | 11.7 | 17.2
BioBERT + BAT | 20.8 | 17.7 | 19.1
BioBERT + VR | 25.7 | 16.4 | 20.1
Table 9: Subtask 2. Results with one augmentation using ER, BAT, and VR with
BioBERT. We augment the entire dataset.

For this experiment, we utilized the BioBERT model Lee et al. (2020) as the
baseline. Table 9 shows the effect of three augmentation methods on this model
with 100 examples augmented from the claims category. As we can see, overall,
verb replacement is more effective than other methods although entity
replacement makes more sense for this task since in ER, we increase the number
of sentences using similar entities. This should provide a more diverse
context for the existing entities in the training data.

##  6 Conclusion

We proposed verb replacement as a novel counterfactual data augmentation
technique to increase the number of samples in the minority class for causal
claim identification. Then, we showed that this method can significantly
improve the performance of the machine learning model in the minority class.
Comparing it with three other augmentation methods, we also found out that the
proposed method can outperform them in some cases.

## References

  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th international conference on computational linguistics_ , pages 1638–1649.
  * Chawla et al. (2002) Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002\.  Smote: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_ , 16:321–357.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * He and Garcia (2009) Haibo He and Edwardo A Garcia. 2009.  Learning from imbalanced data.  _IEEE Transactions on knowledge and data engineering_ , 21(9):1263–1284.
  * Herland et al. (2017) Matthew Herland, Richard A Bauder, and Taghi M Khoshgoftaar. 2017.  Medical provider specialty predictions for the detection of anomalous medicare insurance claims.  In _2017 IEEE international conference on information reuse and integration (IRI)_ , pages 579–588. IEEE.
  * Karimi et al. (2021a) Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021a.  Adversarial training for aspect-based sentiment analysis with bert.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pages 8797–8803. IEEE.
  * Karimi et al. (2021b) Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021b.  AEDA: An easier data augmentation technique for text classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2748–2754, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Khetan et al. (2023) Vivek Khetan, Somin Wadhwa, Byron Wallace, and Silvio Amir. 2023.  Semeval-2023 task 8: Causal medical claim identification and related pio frame extraction from social media posts.  In _Proceedings of the 17th International Workshop on Semantic Evaluation_ , Toronto, Canada. Association for Computational Linguistics.
  * Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Ling and Li (1998) Charles X Ling and Chenghui Li. 1998.  Data mining for direct marketing: Problems and solutions.  In _Kdd_ , volume 98, pages 73–79.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  Nltk: The natural language toolkit.  In _Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics_ , pages 63–70.
  * Miller (1995) George A Miller. 1995.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Schlicht et al. (2023) Ipek Baris Schlicht, Lucie Flek, and Paolo Rosso. 2023.  Multilingual detection of check-worthy claims using world languages and adapter fusion.  _arXiv preprint arXiv:2301.05494_.
  * Temraz and Keane (2022) Mohammed Temraz and Mark T Keane. 2022.  Solving the class imbalance problem using a counterfactual method for data augmentation.  _Machine Learning with Applications_ , 9:100375.
  * Wadhwa et al. (2023) Somin Wadhwa, Vivek Khetan, Silvio Amir, and Byron Wallace. 2023.  Redhot: A corpus of annotated medical questions, experiences, and claims on social media.  In _European Association of Computational Linguistics (EACL)_.
  * Wang and Xu (2018) Yibo Wang and Wei Xu. 2018.  Leveraging deep learning with lda-based text analytics to detect automobile insurance fraud.  _Decision Support Systems_ , 105:87–95.
  * Zeng et al. (2020) Xiangji Zeng, Yunliang Li, Yuchen Zhai, and Yin Zhang. 2020.  Counterfactual generator: A weakly-supervised method for named entity recognition.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7270–7280, Online. Association for Computational Linguistics.