[1]

# Investigations on Speech Recognition Systems for Low-Resource Dialectal
Arabic-English Code-Switching Speech

Injy Hamed<EMAIL_ADDRESS>Pavel Denisov
<EMAIL_ADDRESS>Chia-Yu Li<EMAIL_ADDRESS>Mohamed Elmahdy<EMAIL_ADDRESS>Slim Abdennadher
<EMAIL_ADDRESS>Ngoc Thang Vu<EMAIL_ADDRESS>Institute for Natural Language Processing, University of Stuttgart, Germany
Data Science Department, Raisa Energy LLC, Cairo, Egypt  Computer Science
Department, The German University in Cairo, Egypt

###### Abstract

Code-switching (CS), defined as the mixing of languages in conversations, has
become a worldwide phenomenon. The prevalence of CS has been recently met with
a growing demand and interest to build CS ASR systems. In this paper, we
present our work on code-switched Egyptian Arabic-English automatic speech
recognition (ASR). We first contribute in filling the huge gap in resources by
collecting, analyzing and publishing our spontaneous CS Egyptian Arabic-
English speech corpus. We build our ASR systems using DNN-based hybrid and
Transformer-based end-to-end models. In this paper, we present a thorough
comparison between both approaches under the setting of a low-resource,
orthographically unstandardized, and morphologically rich language pair. We
show that while both systems give comparable overall recognition results, each
system provides complementary sets of strength points. We show that
recognition can be improved by combining the outputs of both systems. We
propose several effective system combination approaches, where hypotheses of
both systems are merged on sentence- and word-levels. Our approaches result in
overall WER relative improvement of 4.7%, over a baseline performance of 32.1%
WER. In the case of intra-sentential CS sentences, we achieve WER relative
improvement of 4.8%. Our best performing system achieves 30.6% WER on ArzEn
test set.

###### keywords:

speech recognition, speech corpus, code-switching, Arabic-English, low-
resource, end-to-end ASR, DNN-based ASR, system combination

utf8

##  1 Introduction

It is common for speakers in multilingual societies to alternate between
languages in conversations, a multilingual phenomenon, referred to as “code-
switching” (CS). In the middle east, colonization and international businesses
and education have played a major role in introducing English and French into
conversational language. CS is prevalent in Arab countries such as Arabic-
French in Morocco [19] and Algeria [20], Arabic-English in Egypt [1], Saudi
Arabia [76], Jordan [74], Kuwait [3], Oman [4] and UAE [56] and a high level
of multilingualism is found in Lebanon [15] and Tunisia [16] with the mixing
of Arabic and both English and French. CS can occur at the levels of
sentences, words, and morphemes in the case of morphologically rich languages,
such as Arabic. In the following examples, we present the different CS types.
As Arabic follows a right-to-left writing system, we use arrows to denote the
starting direction of a sentence.

  * •

Inter-sentential CS: defined as switching languages from one sentence to
another. For example:
“→It was really nice. اتعلمت كتير> .”
(It was really nice. I learnt a lot.)

  * •

Extra-sentential CS: where a loan word is borrowed from the secondary
language. For example:
“project انا خلصت ال > ←”
(I’m done with the project).

  * •

Intra-sentential CS (code-mixing): defined as using multiple languages within
the same sentence. For example:
“→I do not think أني عاوز أبقى> student anymore.”
(I don’t think I want to be a student anymore.)

  * •

Intra-word CS (morphological CS): where switching occurs at the level of
morphemes. For example:
“.[conference ال>] [target هن>] احنا> ←”
(We [will target] [the conference])

The global prevalence of this multilingual phenomenon has placed great demand
over ASR systems to be able to handle such mixed input. Despite the increasing
attention CS ASR has received in the past decade [21, 95, 11, 102, 91, 87,
103, 63, 64, 30], it is still considered to be in a rather fledgling state,
with only few language pairs covered, and a huge space for research
investigations and advancements. One of the main bottlenecks hindering the
development of CS NLP applications is the lack of resources [27]. It is
usually the case that CS data is scarce, with resources only covering few
language pairs, and are usually very limited in size. In the case of Arabic
CS, the diglossic nature of the Arabic language magnifies this problem even
more. Besides the Modern Standard Arabic (MSA), which is the formal Arabic
that is taught in schools and acts as a linguistic unifying force to all
Arabic speakers, each country has its own dialect (and sub-dialects), which is
used in daily communication. Since CS is language-dependent, a CS corpus for
each dialect is needed covering the foreign languages associated with this
dialect. Few corpora are available for Arabic CS, where Egyptian Arabic-
English suffers from lack of speech corpora. In order to build our Egyptian-
Arabic ASR system, our first task was to collect our ArzEn CS speech corpus.
The corpus is described in Section 3.

Most of the previous work on CS ASR has used hybrid ASR systems [63, 64, 103];
following the traditional acoustic modeling, language modeling and
pronunciation dictionary (lexicon) pipeline. Recently, E2E systems are
preferred by researchers for their simplicity and success in monolingual [29]
and multilingual [94] settings. While E2E systems have been previously
investigated for CS ASR [67, 104, 85, 63, 99, 64, 103], their competency
versus DNN-based hybrid systems is still questionable, where literature still
lacks a thorough comparison between both systems for the low-resourced CS ASR
task. In this work, we present our work on building Arabic-English CS ASR
systems using both E2E and DNN-based hybrid models. In Section 5, we present a
thorough comparison between both systems with regards to the extent of
multilingual as well as crosslingual knowledge transfer achieved by both
systems, their tolerance to unstandardized orthography, and their reliance on
CS data for the success of cross-lingual knowledge transfer from monolingual
to CS context and the success of transfer learning in E2E systems.

Building ASR systems for CS Egyptian Arabic-English is a rather challenging
task. From the dialectal Arabic (DA) side, the system is challenged by two
main DA issues: unstandardized orthography and morphological richness. From
the English side, despite the fact that there are huge amounts of English
speech corpora available, our system can only make use of a few hours of such
data. This restriction is placed as a result of our primary language (Egyptian
Arabic) being a low-resource language, a limit is then consequently placed on
the amount of secondary language (English) data that can be used in the
models, after which addition of further data harms recognition. We show that
DNN-based hybrid systems show superiority in English recognition. The problem
of limited English training data is alleviated with the use of lexicons which
act as a valuable knowledge resource, enabling the system to recognize out-of-
vocabulary (OOV) and infrequent words. However, the reliance on lexicons
limits the performance of DNN-based in the case of DA, where they fall short
against the challenges of unstandardized orthography and morphological
richness. E2E systems on the other hand, as their performance is not
restricted with lexicons, show superiority in DA recognition.

Motivated by our findings and the fact that system combination is found to be
most effective in the case of ASR systems that provide similar performance and
are complementary in their errors [51], we apply system combination to utilize
the strengths of both systems. Combining hypotheses of ASR systems has been
previously explored, where researchers investigated several techniques
including Recognizer Output Voting Error Reduction (ROVER) [42, 62], iROVER
[50], N-best ROVER [93], Confusion Network Combination (CNC) [41], iCNC [52],
Bayesian model combination (BAYCOM) [83], lattice rescoring [71], and lattice
combination [100, 65]. An overview on previous work for ASR system combination
is presented in [23]. Up to our knowledge this work represents the first
efforts towards combining outputs of hybrid and E2E systems for improving CS
ASR. In our work, we present three approaches for combining outputs from both
systems on the sentence- and word-levels. We show how such approaches can be
used to leverage the strengths of both systems to reach a better list of
hypotheses and significantly improve CS ASR.

In the scope of this work, we make the following contributions:

  * •

Release our ArzEn corpus to motivate further research in this direction. The
corpus is primarily designed for ASR, however, it also serves as a useful
resource for other NLP tasks as well as linguistic, sociolinguistic and
psycholinguistic CS studies.

  * •

Present the first ASR system for Egyptian Arabic-English speech, while
reflecting on language-specific challenges and CS ASR limitations. Our final
system achieves a WER of 30.6% and CER of 18.7% on test set.

  * •

Present a thorough comparison between E2E and DNN-based hybrid systems. While
the performance of E2E systems is usually tied to the availability of huge
amounts of training data, we show that they provide comparable results to DNN-
based hybrid systems in our low-resource setting. Through our error analysis,
we show that each system brings its own set of strengths. The superiority of
E2E systems in Arabic recognition is attributed to their tolerance to DA
issues, which greatly affect the performance of hybrid systems due to their
reliance on lexicons. In the case of English recognition, systems suffered
from the restricted amount of English training data. While this problem
greatly affected E2E systems, DNN-based hybrid systems performed significantly
better as knowledge was gained from lexicons. The performance of the E2E
system is further improved using transfer learning.

  * •

Propose effective approaches for combining outputs of E2E and hybrid systems.
The best performance is achieved by a hybrid system combination, where
hypotheses are merged on both sentence- and word-levels. Our approach
leverages the flexibility of E2E systems with regards to DA recognition as
well as the strengths provided by the DNN-based systems lexicon supporting
English recognition and infrequent and OOV words. Our results confirm systems’
complementarity, where system combination results in an overall WER relative
improvement of 4.8% in the case of intra-sentential CS.

##  2 Related Work

In this section we give an overview on how CS ASR systems have evolved over
the years, we present the current work done on CS ASR for the Arabic language,
as well as currently available CS speech corpora.

###  2.1 CS ASR

Over the past years, CS ASR systems have remarkably evolved. The first efforts
included a multi-pass approach [68], where a language boundary detection (LBD)
algorithm is used to divide the input utterance into segments that are
language-homogeneous. The language identity of each segment is identified
using a language identity detection (LID) algorithm. The corresponding
language-dependent ASR system is then used. While this approach is suitable
for recognizing intersentential CS, it does not perform well for
intrasentential CS. Researchers then opted for a one-pass approach, where one
multilingual ASR system -consisting of multilingual acoustic model,
pronunciation dictionary and language model- is capable of recognizing such
mixed speech. The majority of the work done was based on the GMM-HMM model
[21, 95]. With the current advances in deep learning, researchers have
directed their work towards DNN-based approaches, which have proven to
outperform the traditional GMM-HMM models [91, 87, 102]. While hybrid systems
have achieved great success, they require language-specific lexicons (which
may not be present for low-resource languages) and linguistic knowledge. E2E
approach, on the other hand, has simplified the training process by removing
the hand-engineered components requiring human expertise. E2E ASR systems have
proved to be as successful as the hybrid ASR systems for resource-rich
languages [53]. However, the limit of acoustic resources hinders the
performance of E2E CS ASR systems more severely than hybrid ASR systems [103].
While a number of researchers have investigated the use of E2E systems for CS
ASR, literature still lacks a thorough comparison between E2E and DNN-based
approaches under the low-resource CS setting. Moreover, only few language
pairs have been investigated so far, including Mandarin-English [63, 67, 104,
85], Chinese-English [64, 99], and Frisian-Dutch [103]. Research lacks such an
investigation for CS Egyptian Arabic-English language pair.

###  2.2 Arabic CS ASR

Despite Arabic being one of the most popular languages (ranking $6^{th}$
according to total number of speakers [37]) and the prevalence of CS across
Arab countries [13, 46, 12], there is a huge research gap in the field of the
Arabic CS ASR. While many researchers have worked on ASR for Modern Standard
Arabic [38, 58, 72, 9, 25, 5] and dialectal Arabic [59, 60, 89, 40, 5, 8, 75,
10, 6, 7], work on CS Arabic ASR is still in its initial stages. In [18],
Bayeh et al. presented an ASR system for Arabic Broadcast News (BN), where
they rely on crosslingual techniques to allow the baseline systems trained on
MSA data to recognize dialectal languages (mainly Levantine/Maghrebian) as
well as French embedded language. Barik and Lestari [17] developed an ASR
system for Indonesian-Arabic, where Arabic is embedded into Indonesian
conversations. The system’s WER was improved by including lexicon entries for
Arabic words, adding CS sentences in the text corpus used for language
modeling (LM) training, and merging phonemes using International Phonetic
Association (IPA). Elfahal [39] worked on developing CS ASR system for read
Sudanese Arabic-English speech. The authors built the ASR system using mixed
LM, mixed phonetic dictionary and GMM-HMM acoustic model. The output of the
ASR system, along with the hypotheses confidence scores were then passed to a
second stage where the language of each word is identified using lexicon look-
up. In [30], Chowdhury et al. present their work on dialectal Arabic-MSA ASR
system. Also related to the scope of our research is the work of Amazouz et
al. [11], where the authors do not work on ASR, but rather on language
identification in speech for the Algerian Arabic-English language pair. Our
system presents the first work for the Egyptian Arabic-English language pair.

###  2.3 CS Speech Corpora

The vast majority of the available CS speech corpora have covered Chinese-
English [28, 86, 66, 70, 2], Hindi-English [34, 82, 87, 90, 78] and Spanish-
English [88, 33, 82] language pairs. Less work has covered Arabic-English [54,
46, 47], Arabic-French [12, 14], Frisian-Dutch [101], Mandarin-Taiwanese [69,
68], Turkish-German [26], English-Malay [2], English-isiZulu [98] and Sepedi-
English [73]. For Arabic CS, speech corpora are very limited in size as well
as covered language-pairs. Ismail [54] collected an 89-minutes speech corpus
by recording and transcribing informal dinner gatherings involving 6
participants. Amazouz et al. [12] gathered a 7.5 hours of read speech from
books and movie transcripts, as well as informal conversations, gathered from
20 speakers. The corpus contains transcriptions, sentence segmentation,
language boundary and phone-level time codes information. A Maghrebian Arabic-
French speech corpus [14] was also collected by Amazouz et al. containing 53
hours of spontaneous speech gathered from TV entertainment and talk shows,
involving speakers from Algeria, Morocco and Tunisia. The corpus contains
sentence segmentation and language annotation. For Egyptian Arabic, several
corpora were collected in the scope of our project. In [46], we present a
6-hours corpus of informal interviews with 12 speakers in technical domain,
where transcriptions were obtained for 4 hours. The corpus is still under
processing to be used in ASR, where the remaining transcriptions are being
obtained and the whole corpus is being segmented. In [47], we present a
12-hours speech corpus that is transcribed, sentence-segmented, and annotated
for speaker information. Information about the corpus is given in Section 3.

##  3 ArzEn Speech Corpus

(a) Using Praat for the annotation process.

(b) Example of the TextGrid output from Praat for the transcription tier. The
arrows beside the sentences show the sentence starting direction, while the
arrows on top of the words show the direction of the language-homogeneous
blocks.

Figure 1: Corpus transcription and annotation process.

In this Section, we will briefly present the ArzEn111Arz and En are the codes
for Egyptian Arabic and English in Ethnologue. speech corpus, discussing the
collection and annotation process as well as corpus code-switching statistics.
We extend our work in [47] with: (1) releasing the corpus
222https://github.com/DigitalPhonetics and (2) improved annotation and further
analysis with regards to intra-word CS in conversational Egyptian Arabic
speech.

ArzEn is a 12-hours Egyptian Arabic-English CS spontaneous speech corpus. The
corpus is collected through informal interviews with 38 Egyptian bilingual
speakers, who are fluent in English. The interviews were held at The German
University in Cairo (GUC), a private university where English is the
instruction language. Participants are in the age range of 18-35, where 61.5%
are males and 38.5% are females. All participants are students (55%) or
employees (45%) of the GUC. The interviewees were asked questions covering
several topics, including education, personality, personal life, career,
hobbies, traveling, work and life experiences, role model and technology. To
ensure good audio quality, all recordings were carried out in a soundproof
room and at a sampling rate of 48kHz. The corpus is mainly designed to be used
in Automatic Speech Recognition (ASR) systems, however, the interview setup is
designed so that the corpus acts as a useful resource for analyzing the CS
phenomenon from linguistic, sociological, and psychological perspectives.
Interviews included one interviewee and two interviewers; a male and a female.
The set of questions as well as the way the questions were asked were also
fixed throughout the interviews. After the interview, the participant was
asked to fill in three forms:

  * •

Questionnaire: gathering information about the participants including their
demographics, education, work, travelling experiences, and their perceptions
and opinions regarding code-switching.

  * •

Big Five Personality Test [44]: assesses five major dimensions of personality:
Openness, Conscientiousness, Agreeableness, Extraversion, and Neuroticism. The
Big Five Personality test was chosen as it is the most widely used and
extensively researched model of personality [45] and because it consists of 50
questions, thus only requires around 10 minutes from the participants.

  * •

Self-Assessment Manikin Test [24]: a picture-oriented questionnaire developed
to measure emotional response.

These forms were used in order to gather more information about the
interviewees, motivating further socio- and psycho-linguistic research to be
conducted for this under-resourced CS language pair.

###  3.1 Corpus Transcription and Annotation

The corpus was transcribed by professional transcribers, and revised by one of
the authors, who is a fluent Arabic-English bilingual. In order to address the
unstandardized issue of Dialectal Arabic orthography, we based our
transcription guidelines on the conventions developed and used by the Egyptian
Arabic Wikipedia community
333https://arz.wikipedia.org/wiki/ويكيبيديا¿:Introduction_in_English
#Rules_of_writing. In the case of ambiguous cases where several orthographic
varieties are allowed, we made decisions to restrict the number of
possibilities to usually one variant only. We use the following tags for non-
speech parts: [HES] for hesitation, [HUM] for humming, [COUGH], [LAUGHTER],
and [NOISE]. Sentence segmentation was performed such that each segment spans
of maximum 25 seconds, and each segment was annotated with the speaker ID.
Praat444http://www.fon.hum.uva.nl/praat/ is used for annotation, where we use
two tiers: transcription and speaker. Figure 1(a) shows an example of the
annotation process. Figure 1(b) shows the TextGrid file produced by Praat.
Since Arabic adopts a right-to-left writing system, reading code-mixed Arabic-
English sentences could be confusing as language-homogeneous segments are read
in different directions. Also, the starting direction of the sentence, depends
on the language of the first word, where sentences beginning with an Arabic
character start from the right side, while sentences beginning with a Latin
character start from the left side. In order to make it easier throughout the
paper, we will use a small arrow to denote the starting direction of
sentences, and we will place arrows on top of words to guide the reader
through the word sequences. For example, in Figure 1, the text in the third
interval starts from the left side and is read in the following order: I -
actually - made - more - friends - من> \- ال> \- course \- بتاع> \- ال> \-
Deutsch \- اللي> \- انا> \- كنت> \- باخده>.

###  3.2 Corpus Statistics

In Table 1(a), we provide an overview on the corpus, presenting the number of
speakers, speech and non-speech duration, average sentence length and
duration, and average speaking rate, as well as the number of sentences,
words, and tokens. The difference between the number of words and tokens is
due to intra-word CS, where a compound word consisting of Arabic affixes and
an English word is treated as separate tokens. In Table 1(b), we report the
level of code-switching available in the ArzEn corpus. In order to provide a
thorough CS analysis on the ArzEn corpus, we report code-switching metrics
reflecting all 4 types of code-switching.

Category |  Value
---|---
# Speakers |  40
# Interviewers |  38
# Interviewees |  2
Average Interview Duration |  0.32 hours
Total Duration |  12 hours
Speech Duration |  11.4 hours
Non-speech Duration |  0.7 hours
# Sentences |  6,290
# Words |  99,340
# Tokens |  102,332
Sentence Duration Range (s) |  0.3-24.9
Av. Sentence Duration (s) |  6.6
Sentence Length Range (words) |  1-95
Av. Sentence Length (words) |  16.3
Average speaking rate (words per minute) |  149.1
(a) Overall statistics

Category |  Value
---|---
Inter-sentential CS
% Monolingual Arabic Sentences |  33.1%
% Monolingual English Sentences |  3.7%
Intra-sentential CS
% CS Sentences |  63.2%
% Arabic Tokens |  84.9%
% English Tokens |  15.1%
Morphological CS
% Sentences with morphological CS |  28.4%
% CS sentences with morphological CS |  45.0%
% Arabic Words |  84.9%
% English Words |  12.3%
% Morphological CS words |  2.7%
# Morphological CS Words |  2,684
(b) Code-switching statistics

Table 1: Corpus statistics.

####  3.2.1 Inter-sentential CS:

Inter-sentential CS is seen in the corpus, however, with very low frequency,
where pure monolingual English sentences only constitute 3.7% of the
sentences. However, it is seen that some sentences are mainly in English but
contain few Arabic conjugations. For example: “I would say probably martial
arts أو> (or) tennis.”

####  3.2.2 Intra-sentential and Extra-sentential CS:

A total of 63.2% of the sentences are code-mixed, where 88.9% are mainly in
Arabic with English embeddings, while 7.4% have more English than Arabic
words. On average, in each CS sentence, there are 1.98 switches from Arabic to
English and 1.91 switches from English to Arabic. Among the CS sentences,
80.5% of the sentences start in Arabic and 19.5% start in English. Throughout
the corpus, there are 15,481 English words, which are 15.1% of the total
words. Among the CS sentences, 18.8% of the words are in English. We also
report the Code-Mixing Index (CMI) [31], which represents the level of mixing
between languages in text. CMI is defined as:

$CMI=\frac{\sum_{i=1}^{N}(w_{i})-max\\{w_{i}\\}}{n-u}$

where $\sum_{i=1}^{N}(w_{i})$ is the total number of words over all languages,
$max(w_{i})$ is the highest number of words across the languages, $n$ is the
total number of words, and $u$ is the total number of language-independent
words. Monolingual sentences would have a CMI of 0 and sentences with equal
word distributions across languages would have a CMI of $n/N$, which is 0.5 in
the case of bilingual utterances. The CMI is 0.12 over the whole corpus and
0.17 over CS sentences.

Set Interviews F M Size(h) wpm CMI %CS Snt. %En Snt. % En words Train 12 4+1
8+1 5.6 156.9 0.14 68.8% 3.9% 16.0% Dev 13 5 8 2.9 143.9 0.14 70.0% 2.6% 14.8%
Test 13 5 8 2.9 148.7 0.14 68.7% 2.1% 17.0%

Table 2: Overview on training, development and test sets. For each set, we
show the number of interviews, gender distribution, duration, word per minute
rate (wpm), CMI as well as the percentages of CS sentences, English sentences,
and English words.

####  3.2.3 Intra-word CS:

We extend our analysis presented in [47] by improving our annotation for
morphological CS. In [47], no annotation was done for intra-word CS, where
English words and Arabic affixes/clitics were separated with spaces, for
example “ال> TASK ات>” (the tasks). In this paper, We improve our annotation
for morphological CS words as follows: Arabic prefixes/proclitics + English
words # Arabic suffixes/encl-itics, for example “ال>+TASK#ات>”. Such an
annotation allows further investigations on intra-word CS in the case of
Egyptian Arabic-English language pair, and allows us to measure the
recognition improvements in our ASR experiments specifically on this type of
CS. We extend our corpus with morphological CS analysis. The corpus contains
2,684 morphological CS words. In Table 12, we present the list of Arabic
clitics and affixes in the context of morphological CS, as well as their
frequencies. Morphological CS occurs most frequently in the form of Arabic
prefixes/proclitics attached to English words, where this form is seen 2,440
times. The attachment of Arabic suffixes/enclitics to English words is less
frequent, occurring only 14 times. The attachment of Arabic circumfixes to
English words is present 10 times. Other combinations of Arabic
prefixes/proclicits and suffix-es/enclitics attached to English words occur 7
times, such as ال>+ROBOT#ات> (the robots), لل>+TASK#ات> (to the tasks),
ا>+ENHANCE#ه> ((I) enhance it), ى>+PROPAGATE#ها> ((he) propagates it),
بى>+EVALUATE#نى> ((he) evaluates me).

###  3.3 Adaptation, Development, and Test Sets

The corpus is divided into train, dev and test sets. We take into
consideration having balanced dev and test sets in terms of gender
distribution, number of interviews, duration, word per minute (wpm) and CS
metrics where we consider the CMI, the percentage of code-mixed sentences, and
the percentage of English words. In Table 2, we present the figures across
training, development, and test sets. In order to avoid including the
interviewers as common speakers across all sets, their utterances are placed
in the train set. Therefore, the train set includes 4 female and 8 male
interviewees, in addition to 1 female and 1 male interviewer.

##  4 Experimental setup

###  4.1 Data

####  4.1.1 Speech Corpora

For CS data, we use ArzEn [47] speech corpus for training, development and
testing. For monolingual Egyptian Arabic data, we use CALLHOME [43]
(19.4h-Telephone conversations) and MGB-3 [8] (16h-Egyptian YouTube vid-eos).
For the MGB-3 corpus, we excluded utterances having overlapping speech, thus
ended up with a subset of 9-hours. For monolingual English data, we use
Librispeech [77] (1,000h-Read audiobooks). For monolingual MSA data, we use
MGB-2 [5] (1,137h-Aljazeera Arabic TV programs (News)). For MGB-2 and
Librispeech, we used subsets of the corpora, as outlined later in Section 5.

####  4.1.2 Text Corpora

For language modeling, we use the transcriptions of ArzEn train set ($3,346$
sentences, $55,432$ tokens), Callhome ($29,829$ sentences, $213,260$ tokens),
and MGB-3 ($3,346$ sentences, $87,093$ tokens), in addition to further CS
transcriptions [46]. In order to obtain more text data for CS and monolingual
Egyptian Arabic, we leverage data collected from social media platforms
($1,574,208$ sentences, $22,025,671$ tokens) [48].

####  4.1.3 Data Processing

Unlike MSA, Dialectal Arabic does not have a standardized orthography, where
words can have multiple correct spellings and some letters can be used
interchangeably. Regarding the former problem, we identified major differences
in the orthography guidelines used in Callhome, ArzEn and MGB-3, and
standardized the text in regards to these issues. Regarding the latter
problem, it is common that variants of Hamzated Alif (أ>, إ>, آ>) are often
written without their Hamza and the Alif-Maqsura (or dotless Ya) (ى>) and
dotted Ya (ي>) are used interchangeably when occurring at the end of a word.
This leads to various forms of a word being interpreted as different words,
thus higher sparsity and out-of-vocabulary (OOV) rates. In hybrid ASR systems,
these inconsistencies affect a word being mapped to its corresponding entry in
the lexicon. Therefore, it is important to handle such discrepancies through
normalization. In this step, we normalize the textual data and pronunciation
dictionary by mapping Hamzated Alifs to bare Alif (ا>) and dotted Ya at word
end to dotless Ya. For the English language, normalization included mapping
all letters to uppercase. Punctuation symbols were also removed.

###  4.2 Hybrid ASR

The hybrid ASR systems were built using Kaldi toolkit [81]. In this Section,
we discuss the multilingual acoustic model, pronunciation dictionary and
language model.

####  4.2.1 Acoustic Modeling

We train hybrid systems using both GMM-HMM and TDNN-HMM [80] architectures.
TDNN-DNN was chosen as it has shown superiority over other DNN-based
architectures [102]. The GMM-HMM model is used as point of reference. For the
GMM-HMM acoustic model, the Egyptian Callhome Kaldi recipe was used, which
follows a Triphone+SGMM+SAT+fMLLR pipeline. The recipe uses 5 triphone
training with 30k Gaussians using 13 Mel-frequency cepstral coefficients
(MFCC). The acoustic features are extracted by applying Hamming windowing with
a frame length of 25 ms and frame shift of 10 ms. For the TDNN-HMM acoustic
model, we use the "s5c" TDNN Switchboard recipe "tdnn_7q", using the same
configurations for training parameters. The alignments produced by the GMM-HMM
are used for TDNN training. The default MFCC features combined with i-vectors
for speaker adaptation [84] are used, in addition to Standard 3-way speed
perturbation [61] with 0.9, 1.0 and 1.1 speed ratios for 3-fold augmentation
of the training data. The TDNN consisted of 15 context splicing layers, each
with 1536 nodes, with the following splicing indices: {-1, 0, 1} for layers
2-4, and {-3, 0, 3} for layers 6-15. The final softmax output layer computes
posteriors for 5,728 triphone states. Motivated by the work of Biswas et al.
[22], we add 6 CNN layers prior to the TDNN layer, providing a CNN-TDNN
architecture.

####  4.2.2 Pronunciation Dictionary

The Egyptian Colloquial Arabic lexicon [57], containing 42 phones and 56,913
words was used for Egyptian Arabic. Librispeech lexicon [77], containing 70
phones and 206,508 words was used for English. It is to be noted that
Librispeech lexicon is not the most appropriate choice as it does not take
into consideration accented Egyptian English, however it was used due to the
absence of an Egyptian English lexicon. Phone merging was performed based on
linguistic knowledge of a bilingual speaker, where 26 similar phones were
merged. This method has shown to outperform multiple data-driven methods [87].
The lexicon was filtered to words that appear in the speech and text corpora.
The OOV rates are 11.0% (dev) and 11.4% (test).

####  4.2.3 Language Modeling

For the GMM-HMM models, we use n-gram language modeling using SRILM [92]
toolkit. The LM was obtained using interpolation of two models: (1) 4-gram
model trained on ArzEn train set transcriptions + extra transcriptions and (2)
3-gram model trained on Callhome and MGB-3 transcriptions, in addition to
social media text. Both models used Kneser-Ney smoothing technique. Several
interpolation weights were tried and a weight of 0.7 for the first model was
found to achieve the best perplexity on the development set. The resulting LM
has a perplexity of 566.0 and an OOV rate of 1.1% on dev set. For CNN-TDNN
models, we use recurrent neural network language model (RNNLM). The RNNLM is
trained on the same data sets used for training the n-gram LM. The RNNLM
consists of 1 LSTM layer with 1000 hidden units. The network is trained for 20
epochs. The perplexity is 208.1 on dev set.

###  4.3 E2E ASR

For E2E systems, there are three popular types: Connectionist Temporal
Classification (CTC), Attention Encoder-Decoder (AED), and RNN Transducer
(RNN-T). We use joint CTC/attention based E2E framework as it has demonstrated
superior performance over CTC and AED separate methods [97]. The E2E system
was trained using ESPnet [96]. The encoder and decoder consist of 12 and 6
Transformer blocks (layers) with 4 heads, feed-forward inner dimension 2048
and attention dimension 256 [55]. The CTC/attention weight $(\lambda_{1})$ is
set to 0.3. We applied SpecAugment [79] for data augmentation. For decoding,
the beam size is 20 and the CTC weight is 0.2. Similar to the CNN-TDNN models,
an RNNLM is trained and used in decoding. The RNNLM consists of 1 LSTM layer
with 1000 hidden units and is trained for 20 epochs. The perplexity is 131.2
on dev set. Despite our efforts in unifying hyperparameters across systems,
there is a difference in perplexities due to implementation differences.

####  4.3.1 Transfer Learning

In our experiments, we first explore multi-condition training, which involves
pooling training samples from different conditions and training them
simultaneously [32]. In multi-condition training, we train one model using
pooled CS and monolingual data. We then explore transfer learning, where we
train the first model using monolingual data, and pass the parameters to a
second model trained on CS data. We also combine both approaches, training the
first model using CS and monolingual data, and transferring the parameters to
the second model which is solely trained on CS data. In Table 5, we report
results for multi-condition training, transfer learning, and their
combination.

##  5 Experimental Results and Analysis

We compare the two systems (hybrid vs E2E) with regards to: (1) the extent of
multilingual and crosslingual knowledge transfer achieved by both systems (2)
their tolerance to orthography un-standardization issue; and (3) their
dependence on in-domain CS data for the success of cross-lingual knowledge
transfer.

###  5.1 Extent of Multilingual and Crosslingual Knowledge Transfer

In this section, we evaluate hybrid and E2E systems with regards to their
ability to transfer knowledge from additional monolingual data, including
primary (dialectal Egyptian Arabic), secondary (English), as well as a
closely-related (MSA) languages. We examine the effect of adding data from
primary, secondary as well as closely-related languages. The corresponding
overall as well as language-specific WER figures are presented in Table 3. We
also present the WER figures as well as relative improvements in Figures 2 and
3.

Table 3: WER(%) on ArzEn dev and test sets.

#Exp Datasets Overall WER Arabic WER English WER GMM CNN-TDNN E2E CNN-TDNN E2E
CNN-TDNN E2E dev test dev test dev test test test test test [5.6h] ArzEn Data
1 CS 53.7 54.3 43.4 44.0 53.5 55.0 44.1 56.0 52.9 80.1 \+ [28.4h] Egyptian
Arabic Data (Callhome + MGB-3) 2 CS+Eg 54.4 55.1 35.1 35.8 33.6 35.3 35.6 32.9
44.6 63.5 \+ [5-10h] English Data (Librispeech) 3 CS+Eg+En(5h) 56.0 55.9 33.6
34.1 32.9 33.8 34.7 32.3 37.5 54.7 4 CS+Eg+En(10h) 57.6 57.4 32.9 33.2 32.1
33.0 33.8 31.6 36.3 51.8 \+ [5-40h] MSA Data (MGB-2) 5 CS+Eg+En(5h)+MSA(5h)
56.5 56.0 32.9 33.2 31.9 32.8 33.7 31.1 36.3 53.5 6 CS+Eg+En(5h)+MSA(10h) 58.0
56.8 32.5 33.2 32.3 33.2 33.5 31.3 37.0 55.5 7 CS+Eg+En(5h)+MSA(20h) 58.0 57.6
32.6 32.9 32.4 33.3 33.4 31.5 36.4 54.9 8 CS+Eg+En(10h)+MSA(5h) 58.0 57.6 32.2
32.8 34.0 35.3 33.3 33.5 35.6 55.9 9 CS+Eg+En(10h)+MSA(10h) 58.6 58.2 32.2
32.7 32.2 33.5 33.1 32.0 36.9 54.6 10 CS+Eg+En(10h)+MSA(20h) 59.5 58.6 31.9
32.1 32.8 34.2 32.6 32.3 34.2 51.6 11 CS+Eg+En(10h)+MSA(40h) 59.5 59.0 31.7
32.6 35.8 37.1 33.0 35.9 36.4 58.3

In Exp2-4, we explore the effect of adding monolingual data for primary and
secondary languages. CNN-TDNN and E2E models benefit from the addition of
monolingual data, while GMM-HMM models show a deterioration in performance.
CNN-TDNN models show superiority over E2E systems with limited (5.6 hours) CS
data (Exp1). However, E2E systems show higher relative improvements with the
addition of monolingual data, and are able to achieve comparable results after
adding only 28.4h of monolingual Egyptian Arabic data. As shown in Figure 3,
CNN-TDNN and E2E models achieve overall relative improvements of 18.6% and
35.8% on test set with the addition of Egyptian Arabic data, and 7.2% and 6.5%
with English data. As shown in Figure 2, while the performance of E2E systems
is usually tied to the availability of vast amount of training data, they were
able to achieve comparable results to CNN-TDNN hybrid systems with only 34h of
training data (5.6h CS + 28.4h Egyptian Arabic).

Figure 2: WER achieved by hybrid and E2E systems. Figure 3: Relative WER
improvements achieved by adding Egyptian Arabic (Eg), English (En) and MSA
monolingual data.

In Exp5-11, we investigate the impact of leveraging MSA speech data. While MSA
and dialectal Arabic are considered in literature to be separate languages,
they share common alphabet and words, presenting a promising source for
further acoustic knowledge. Results show that both CNN-TDNN and E2E systems
are able to learn from closely-related languages, where the addition of MSA
data -up to a certain limit- results in relative improvements of 3.3% and 3.0%
on test sets, respectively. While the improvements in Arabic recognition can
be foreseen (3.6% and 3.7% for CNN-TDNN and E2E systems), English recognition
also, unexpectedly, improved (5.8% and 2.2% for CNN-TDNN and E2E systems). It
is to be noted though that the addition of MSA data deteriorates English
recognition in some utterances (as shown in Table 6-Example 2), however, it
provides a general improvement in English recognition for both systems.

###  5.2 Tolerance to Un-standardized Orthography

Table 4: Tolerance to Un-standardized Orthography: WER(%) on ArzEn dev and
test sets.

#Exp Datasets Overall WER Arabic WER English WER GMM CNN-TDNN E2E CNN-TDNN E2E
CNN-TDNN E2E dev test dev test dev test test test test test 2 CS+Eg 54.4 55.1
35.1 35.8 33.6 35.3 35.6 32.9 44.6 63.5 12 CS+Eg (Unstandardized) 59.6 59.3
47.3 47.8 34.7 36.2 49.8 33.5 45.8 63.5

In order to further compare the performance of both systems, we investigate
their tolerance to un-standardization of dialectal Arabic. In Table 4, we
report the WER achieved by both systems when training models on CS and
Egyptian data with (Exp2) and without (Exp12) the pre-processing step of
normalization and standardization. While E2E systems show high tolerance to
un-standardization where the performance is only slightly affected (0.9%
absolute), hybrid systems suffer from a great (12.0% absolute) performance
deterioration. This can be traced back to their reliance on lexicons, where
the lexicon OOV rates improved by 31% (dev) and 28% (test) after normalization
and standardization. This result reflects: (1) the reliance of hybrid systems
on linguistic knowledge (in addition to linguistic resources) and (2) the
flexibility of E2E systems, where corpora following different orthography
guidelines can be directly used, without the need for standardization.

###  5.3 Reliance on in-domain CS speech data

Table 5: Reliance on in-domain CS speech data: WER(%) on ArzEn dev and test
sets.

#Exp Datasets Overall WER Arabic WER English WER GMMM CNN-TDNN E2E CNN-TDNN
E2E CNN-TDNN E2E dev test dev test dev test test test test test Multi-
condition Training 5 CS+Eg+En(5h)+MSA(5h) 56.5 56.0 32.9 33.2 31.9 32.8 33.7
31.1 36.3 53.5 13 Eg+En(5h)+MSA(5h) 65.3 64.2 54.2 53.0 54.5 54.7 51.1 54.4
101.0 113.8 Transfer Learning 14 [Eg+En(5h)+MSA(5h)]→[CS] 32.8 34.3 32.6 56.1
Multi-condition Training + Transfer Learning 15 [CS+Eg+En(5h)+MSA(5h)]→[CS]
31.6 32.7 31.1 53.7

Figure 4: WER of hybrid and E2E systems trained by adding Egyptian Arabic
(Eg), English (En) and MSA monolingual data.

Given the sparsity of CS data and the lack of CS data for many language pairs,
We investigate the importance of in-domain CS data and whether we can dispense
such time-consuming and expensive data and rely only on monolingual speech
data. Most of the work on CS ASR has involved training the ASR systems on CS
data. Thus, it is still unclear whether CS ASR systems can be built without CS
data, and to what extent is CS data essential for the success of cross-lingual
knowledge transfer from monolingual to CS context. We explore the ability of
the systems to recognize CS speech, relying only on monolingual data. In
Exp13, we train our models using only monolingual data. Although in-domain
data constitutes only 12.7% of the training data, we report that excluding it
leads to great performance reductions; 29.6% (CNN-TDNN) and 32.6% (E2E) on
test set, as shown in Table 5. While both CNN-TDNN and E2E systems were still
able to recognize Arabic speech with 51.1% and 54.4% WER (on test set), they
fail to recognize English words, where English WER drops to 101.0% and 113.8%
(on test set). This demonstrates the necessity of in-domain CS data for the
success of cross-lingual knowledge transfer, enabling systems to predict CS
switch points. This experiment also highlights the importance of reporting
language-specific WER in CS NLP systems.

We further extend our experiments to investigate the importance of in-domain
CS data in the success of transfer learning in E2E systems. We train an
initial model using solely monolingual data, and then fine-tune this
pretrained model on CS data (Exp14). Results show that multi-condition
training outperforms transfer learning in our task. The problem with the
transfer learning approach can be attributed to the low performance of the
pretrained model (Exp13-trained only on monolingual data) and its failure with
regards to English recognition, as previously discussed, thus limiting the
transferred knowledge. Accordingly, we apply transfer learning after
multicondition training, as outlined in Exp15, achieving further minor
improvements. The effect of incorporating the limited amount of CS data in
both approaches on Arabic, English and overall recognition is demonstrated in
Figure 4.

##  6 Qualitative Analysis

We provide further insight into the systems’ performances, identifying the
strengths and weaknesses of both system and highlighting their complementary
performance in the case of dialectal Arabic-English. We also discuss factors
resulting in incorrect/harsh evaluation using WER.

Table 6: Hypotheses (test set) reflecting strengths and weaknesses of hybrid
and E2E systems.

# Systems Hypothesis Ground-truth I’M NOT A TEAMS PERSON فلأ> ← Translation
(so no I’M NOT A TEAMS PERSON) CNN-TDNN[CS] →NOT A TEAM PERSON CNN-TDNN[+Eg]
→I’M NOT A TEAM PERSON CNN-TDNN[+Eg+En] →I’M NOT A TEAM PERSON 1 CNN-
TDNN[+Eg+En+MSA] →I’M NOT A TEAM PERSON E2E[CS] THINK PERSONAL لأ عندنا> ←
E2E[+Eg] I NOT A TEAM PERSON لأ> ← E2E[+Eg+En] I NOT A TEAM PERSON لأ> ←
E2E[+Eg+En+MSA] I’M NOT A TEAM PERSON لأ> ← Ground-truth DEEP REINFORCEMENT
LEARNING اسمه> TOPIC بتاعى عن> BACHELOR PROJECT ال> ← Translation (my BACHELOR
PROJECT is about a TOPIC called DEEP REINFORCEMENT LEARNING) CNN-TDNN[+Eg+En]
LEARNING من ال> DIP DREAM FORCE اسمه> TOPIC بتاعى عن> BACHELOR PROJECT و ال> ←
2 CNN-TDNN[+Eg+En+MSA] DEEP REINFORCEMENT LEARNING اسمه> TOPIC بتاعى عن>
BACHELOR PROJECT ال> ← E2E[+Eg+En] LEARNING ال> DEEP REFORCEMENT اسمه> TOPIC
بتاعى عن> BACHELOR PROJECT ال> ← E2E[+Eg+En+MSA] PERIENFORCEMENT IN LEARNING
دى> اسمه> TOPIC بتاعى عن> BACHELOR PROJECT ال> ← Transfer Learning
PERIENFORCEMENT LEARNING دى> اسمه> TOPIC بتاعى عن> BACHELOR PROJECT ال> ←

###  6.1 Hybrid Systems’ Strength over End-to-end Systems

The strength of TDNN-based systems is seen in their ability to recognize
infrequent and OOV words that exist in the lexicon. This allows them to
outperform E2E systems in extreme low-resource scenarios. This is shown in the
significantly better recognition achieved by TDNN-based systems over E2E with
models only trained on CS data (Table 3-Example 1). This is demonstrated in
Table 6\- Example 1 as well as in Table 6-Example 2, where deep and learning
are infrequent words (in Librispeech and Arzen corpora) and reinforcement is
an OOV word. The superiority of TDNN-based systems in English recognition over
E2E systems can be attributed to the low frequency of English words. As shown
in Figure 5, 49% of English words in the test set occur only 1-50 times in
train set (with 38% occurring only 1-25 times), while 42% of Arabic words
occur 1000+ times. The limited amount of English data is found to negatively
affect the performance of E2E systems’ English recognition. DNN-based hybrid
systems on the other hand are less affected by this problem as they compensate
the lack of speech data by obtaining knowledge from lexicons. Accordingly,
DNN-based systems show superiority in English recognition.

Figure 5: Percentage of words in test set against their word frequencies in
train set

###  6.2 End-to-end Systems’ Strength over Hybrid Systems

Table 7: Hypotheses (test set) reflecting issues with WER as an evaluation
metric for the Egyptian Arabic-English language pair.

# Systems Hypothesis Ground-truth موبايل>ال> اهم م> LAPTOP انا بالنسبالى ال >
← Translation (for me the LAPTOP is more important than the MOBILE) 1 CNN-
TDNN[+Eg+En+MSA] MOBILE اهم من ال > LAPTOP انا بالنسبة لى ال> ←
E2E[+Eg+En+MSA] MOBILE اهم من ال > LAPTOP انا بالنسبالى ال> ← Ground-truth
BACHELOR عشان ال> دلوقت> ENGLISH كتير بال> PAPERS بقرا> كتاب قبل كده و> قاريت>
← Translation (I read a book before and I’m reading a lot of PAPERS now for
the BACHELOR) 2 CNN-TDNN[+Eg+En+MSA] BACHELOR عشان ال> دلوقتى> ENGLISH كتير
بال> PAPERS كتاب قبل كده و> قريت> ← E2E[+Eg+En+MSA] BACHELOR عشان> دلوقت>
ENGLISH كتير بال> PAPER بقرا> كتاب قبل كده و> قريت> ← Ground-truth هناك>
حاسافر> فعلا لما يبقى عندى فرصة تانية انا لازم> ACTUALLY و> ← Translation (and
ACTUALLY truly when I get another opportunity I must go there) 3 CNN-
TDNN[+Eg+En+MSA] اسافر> فعلا لما يبقى عندى فرصة تانية انا لازم> ACTUALLY و> ←
E2E[+Eg+En+MSA] هناك> اسافر> فعلا لما يبقى عندى فرصة تانية انا لازم> ACTUALLY
و> ←

We identify two main scenarios where E2E systems’ superiority is observed.
Firstly, their recognition is not limited to lexicon words. This limitation
disables TDNN-based systems to recognize frequently occurring words in the
speech corpus that are absent in the lexicon. This strength is particularly
advantageous in the case of dialectal Arabic as it allows E2E systems to
better handle two main DA challenges: morphological richness and
unstandardized orthography. Given that Arabic is a morphologically rich
language, the limitation of lexicon dependence is highlighted, as it becomes
more challenging to cover all words with all possible affixes combinations.
This limits the performance of TDNN-based systems where (1) the system fails
to recognize morphological compound words that are not present in the lexicon,
even if they are frequently present in the speech corpus and (2) splitting of
stems and prefixes in Arabic words is more frequently observed (for example
“الكتابة>” meaning “the writing” was recognized as “ال كتابة>”). This problem
occurs when the lexicon includes each token separately, however the whole
compound morphological word is absent. This limitation also affects English
compound words, where they are recognized as stand-alone words (for example
“data base”). Although this issue does not occur frequently, it is to be noted
as a disadvantage of the reliance on lexicons. The lexicon dependence of
hybrid systems also limits their Arabic recognition in the case of
orthographically unstandardized words. Spelling inconsistencies result in
discrepancies between the speech corpora and the pronunciation lexicon,
affecting lexicon OOV rates and recognition. The second strength point of E2E
systems is their ability to recognize some OOV words with incorrect spelling,
but very close pronunciation, such as coordinator-courdinater and
spontaneously-spontinuestly. This is reflected in the lower Character Error
Rate (CER) achieved on test set by the E2E system (18.4%) in comparison to the
TDNN-based system (22.0%).

###  6.3 Incorrect/Harsh Evaluation Using WER

We identify cases where WER acts as an incorrect or harsh evaluation metric
for dialectal Arabic-English ASR recognition. Firstly, the confusion between
loanwords and single-word switches. Loanwords are single words that are
borrowed from one language and become, possibly adapted, and integrated into
another [49], such as doctor-دكتور> (d-u-k-t-o-o-r) and Paris-باريس> (p-a-r-e-
e-s/b-a-r-e-e-s). It is a difficult task for humans to distinguish between
both multilingual phenomena, especially with CS words often pronounced with
foreign accent, resulting in three cases: loan words and accented/non-accented
single-word switches. It is interesting to see how our ASR systems handle the
distinction between loanwords and single-word switches and their ability to
use the correct orthography in each case; Arabic for loanwords, and Latin for
CS. Our models are mostly able to transcribe loanwords and non-accented
single-word switches with the correct orthography. The confusion however
arises with accented single-word switches. We also note that in some cases,
loanwords have the same pronunciation in both languages, such as mobile-
موبايل>. In this case, both transcriptions are acceptable, and the confusion
between both incorrectly affects WER, as shown in Table 7-Example 1. Secondly,
due to the issue of unstandardized orthography, correctly recognized words
having a different orthography than the reference words are incorrectly
penalized. This is seen in Table 7-Example 2, where underlined words are all
acceptable spellings. Thirdly, with Arabic being a morphologically rich
language, words can be combined with several affixes, and consequently, each
stem produces numerous words with minor differences, which poses a challenge
for ASR systems. WER then acts as a harsh evaluation metric in the case of
morphological compound words, where words with a mistake in one affix are
considered to be incorrect. As shown in Table 7-Example 3, the word “حاسافر>”
(will travel) was recognized as “اسافر>” (to travel). In this case, the future
proclitic “ح>” that was dropped in the hypothesis does not affect the meaning
of the sentence as the tense is already clear from the sentence context, and
the use of both words in this example is correct. These factors contribute to
a significantly lower CER, where our best models achieve CER of 22.0%/18.4%
(CNN-TDNN/E2E) on test set, in comparison to WER of 32.1%/32.7% (CNN-
TDNN/E2E).

##  7 Merging Hypotheses

Motivated by the different strengths provided by each system, we propose and
evaluate combination approaches for hypotheses on both levels: sentence and
word. We report the results of the combination approaches in Table 8.

Table 8: WER improvements achieved on dev and test sets.

Model Overall WER Arabic WER English WER dev test test test CNN-TDNN 31.9 32.1
32.6 34.2 E2E 31.6 32.7 31.1 53.7 Sentence-level Combination Oracle WER 26.5
26.9 26.5 35.8 Confidence Scores 31.1 32.0 31.4 43.3 Discriminant Classifier
30.8 31.0 30.9 38.5 Word-level Combination E2E →CNN-TDNN 31.4 31.5 31.9 34.1
CNN-TDNN →E2E 31.0 32.0 31.1 48.8 Hybrid Combination sentence+word level 30.4
30.6 30.6 37.2

###  7.1 Sentence-level Combination

In this approach, we propose merging the results of both systems through
rescoring. In order to validate the possible improvements that can be
achieved, we check the oracle WER. The Oracle WER is obtained by choosing the
hypothesis with lower WER among the 1-best hypotheses from both systems. The
oracle WER presents an indicator for the upper bound improvement that can be
achieved through sentence-level systems combination. As shown in Table 8,
there is 5.2-5.8% absolute WER difference between the TDNN-based and E2E
systems and Oracle WER. This supports our claim that both systems contribute
in providing the best hypothesis.

####  7.1.1 Confidence Scores

In the first combination method, we rely on the confidence scores assigned to
the 1-best hypotheses from both systems, to choose the hypothesis with the
highest score. ESPNET provides one confidence score per hypothesis, while
Kaldi provides separate scores from acoustic and language models, where scores
are unnormalized log probabilities. For each utterance, we obtain comparable
normalized probabilities for 1-best hypotheses by applying Softmax function to
the N-best (N=20) hypotheses. The final confidence score of hypothesis $i$ is
calculated as follows:

$\displaystyle score_{ESPNET}=\max_{1\leq i\leq N}Softmax(\textbf{e}_{i})$ (1)
$\displaystyle score_{KALDI}=\max_{1\leq i\leq N}Softmax(\textbf{k}_{i})$ (2)
$\displaystyle\textbf{k}_{i}=(-1\times\textbf{lm}_{i})+(-1\times\textbf{am}_{i}\times\frac{1}{w})$
(3) $\displaystyle
Softmax(\textbf{z}_{i})=\frac{exp(\textbf{z}_{i})}{\sum_{j=1}^{N}exp(\textbf{z}_{j})}$
(4)

where e and k are ESPNET and Kaldi scores’ vectors of 20-best hypotheses for
the utterance, respectively. For Kaldi, the overall confidence score of
hypothesis $i$ in the N-best hypotheses is calculated from the separate
language model $\textbf{lm}_{i}$ and acoustic model $\textbf{am}_{i}$ scores.
In this calculation, the acoustic score needs to first be scaled down by $w$,
which is the LM weight for the best WER ($w=8$ for our best Kaldi model).
After obtaining sentence confidence scores for the best hypothesis from each
system, we choose the hypothesis with the higher score, giving slight absolute
WER improvements of 0.1(CNN-TDNN)-0.7(E2E)% on test set, as shown in Table 8.

####  7.1.2 Discriminant Classifier

The confidence score-based approach allows the decision to rely on one factor
only; the confidence score. However, other factors can also play an important
role, such as sentence CMI and the percentage of English words in the
sentence. Therefore, we propose treating the problem as a Binary
Classification problem. Given the following information about 1-best
hypotheses; confidence scores, CMI and English percentage, we train a
Discriminant Classifier to learn which system to choose. The model achieves an
accuracy of 62.1% and improves WER by 1.1(CNN-TDNN)-1.7(E2E)% (absolute) and
3.4(CNN-TDNN)-5.2(E2E)% (relative) on test set, as shown in Table 8. We also
experiment with $n>1$, however, we observe a reduction in the accuracy of the
Discriminant Classifier for $n>1$ and an increase in WER. In Table 9, we show
the results of training the Discriminant Classifier for n=1,2,5. We report the
accuracy of the Discriminant Classifier as well as the percentage of
hypotheses chosen from TDNN-based and E2E systems and the corresponding WER on
test set.

Table 9: Results of sentence-level combination using Discriminant Classifier for n=1,2,5. n | ML Accuracy | % CNN-TDNN | % E2E | WER
---|---|---|---|---
n=1 | 62.1 | 41.6 | 58.4 | 31.0
n=2 | 34.0 | 26.5 | 73.5 | 31.8
n=5 | 31.6 | 33.0 | 67.0 | 32.5
Table 10: Effect of merging approaches (test set example).

Model Sentence Score WER reference معين فى دماغى دلوقتى> QUOTE يعنى بس انا
مافيش> STANDARDS هو فى> CNN-TDNN يعنى> دلوقت> معين فى دماغى> QUOTE ما فيش>
يعنى بس انا > STANDARDS فى> هو> 0.20 0.33 E2E معين فى دماغى دلوقتى> QUOTE يعنى
بس انا مافيش> STUDENTS هو فى> 0.43 0.08 sentence-level (E2E chosen) معين فى
دماغى دلوقتى> QUOTE يعنى بس انا مافيش> STUDENTS هو فى> 0.43 0.08 +word-level
معين فى دماغى دلوقتى> QUOTE يعنى بس انا مافيش> STANDARDS هو فى> 0.0

Table 11: WER improvements achieved on test set with regards to CS types.

Model All sentences Intra-sentential CS Intra-word CS overall Eg En Overall Eg
En Overall Eg affixes En words CNN-TDNN 32.1 32.6 34.2 31.5 32.2 32.8 32.6
37.2 29.8 E2E 32.7 31.1 53.7 32.9 31.0 52.7 39.2 28.8 48.8 Hybrid Combination
30.6 30.6 37.2 30.0 30.0 35.8 35.2 38.1 34.7

###  7.2 Word-level Combination

In this approach, we take advantage of the strengths of both systems to merge
top hypotheses on the word level. We aim at utilizing the English recognition
provided by TDNN-based systems, and overcome the limitation of TDNN-based
systems’ of failing at recognizing words that are not present in the lexicon.
We investigate the effect of (1) borrowing English words from CNN-TDNN
hypotheses into ESPNET hypotheses as well as infrequent and OOV Arabic words
present in lexicon and (2) borrowing Arabic words that are not present in the
lexicon from E2E hypotheses into CNN-TDNN hypotheses. We first obtain
alignments of 1-best hypotheses using fast-align [36], and perform the
replacements accordingly. While replacing words having 1-to-many alignments
deteriorated the WER, replacing 1-to-1 alignments gave WER improvements, as
shown in Table 8. By looking into the replacements, we find that this approach
improves English E2E recognition by inserting OOV words in lexicon recognized
by TDNN-based models (please →peers and interpanered →entrepreneur), as well
as correcting mis-spelt words (alcommist →alchemist and chaveling →traveling).
On the other hand, TDNN-based hypotheses were improved by inserting more
accurate words provided by the E2E model where affixes were corrected (هواية>
(hobby) →هوايات> (hobbies) and صاحبتى> (my friend) →صاحبتها> (her friend),
دخلت> (entered) →دخلته> (entered it)). This further validates the strength of
E2E systems over hybrid systems in the case of morphologically rich languages.
The word-level combination resulted in a 1.9% and 2.1% relative improvements
for TDNN-based and E2E systems.

###  7.3 Hybrid Combination

In the hybrid combination, the merged 1-best hypotheses list obtained from
sentence-level combination is used as input to word-level combination. In
Table 10, we show an example for the effect of sentence-level and word-level
merge approaches, where words colored in red are incorrect and words colored
in blue are correct however differ from the reference due to unstandardized
orthography. As shown in Table 8, the final model achieves absolute WER
improvements of 2.0% on Arabic over TDNN-based model and 16.5% on English over
E2E models. Relative overall improvements of 4.7% and 6.4% are achieved over
TDNN-based and E2E systems. We have also performed hybrid combination for
$n>1$, however, it does not outperform results for $n=1$, where WER on test
set drops to 31.3% (n=2) and 32.1% (n=5).

We further evaluate the effect of our hybrid merging approach on the ASR
system performance with regards to code-switching. We report the WER for
intra-sentential and intra-word CS. We report the WER figures in Table 11. In
the case of intra-sentential CS, we report WER for code-mixed sentences only.
We show that the hybrid combination achieves a relative improvement of 4.8%
and 8.8% over TDNN-based hybrid and E2E systems for code-mixed sentences. In
order to calculate WER for morphological CS words, we use fast-align [35] to
obtain word level alignments between the tokens making up the compound
morphological CS word in the reference transcription and their corresponding
tokens in the merged hypothesis. The tokens were then manually revised to fix
alignment mistakes. We report that in the case of intra-word CS, E2E system is
superior in recognizing Arabic prefixes, while TDNN-based system is superior
in recognizing English embedded words. For overall recognition of
morphological code-switched words, TDNN-based system shows superiority over
E2E and combined systems. Through our error analysis, we observe that similar
to words, TDNN-based systems were unable to recognize the following Arabic
affixes { ي>, هت>, كال>, بى>, م>, فب>, هن>, ات>, وا>, ى>, ها>, نى>} as most of
these affixes are not present in the lexicon, while E2E systems were not able
to recognize the following infrequent affixes: { هت>, كال>, م>, فب>, هن>, ات>,
وا>, ها>, نى> }.

.

##  8 Conclusions

In this paper, we develop the first Egyptian Arabic-English CS ASR system.
This is a challenging task due to the scarcity of CS corpora, the fact that
the Egyptian Arabic primary language is under-resourced as well as Dialectal
Arabic issues. Our work aims at filling the gap in resources and NLP research
for Dialectal Arabic-English code-switching. We present the ArzEn speech
corpus, provide CS analyses, and release the corpus to motivate further NLP
and linguistic research for this language pair. We extend CS ASR research by
presenting a thorough comparison between TDNN-based hybrid and Transformer-
based end-to-end systems under the setting of this low-resourced,
morphologically rich, orthographically unstandardized, code-switched language
pair. We compare both systems with regards to their extent of multilingual and
crosslingual knowledge transfer, their reliance on in-domain CS data, and
their tolerance to unstandardized orthography. In general, E2E systems
demonstrate great potential, providing comparable results to TDNN-based
systems. We also show that end-to-end and hybrid systems both offer
complementary sets of strength points. On the one hand, E2E systems are robust
against Dialectal Arabic issues, outperforming TDNN-based systems in Arabic
recognition. On the other hand, TDNN-based systems show significant
superiority in English recognition. We therefore propose system combination as
an effective method for leveraging the strengths of both systems through word-
and sentence-level hypotheses combination. Our results prove the
complementarity of both systems, where system combination achieves WER
relative improvements of 4.7% in overall recognition and 4.8% in the case of
code-mixed sentences. By looking into the Oracle WER for sentence-level
combination, we show that further gains can be achieved, and thus we motivate
further research in this direction. Our final ASR system achieves 30.6% WER
and 18.7% CER on ArzEn test set.

## Acknowledgements

This work was supported by the DAAD (German Academic Exchange Service). We
would like to thank Nizar Habash, Özlem Çetinoğlu, and Mohamed Balabel for
their valuable suggestions and linguistic advice. We would also like to thank
the reviewers for their insightful comments and constructive feedback.

##  Appendix A Arzen Intra-word CS

In Table 12, we provide a list of the affixes and clitics occurring in
morphological CS words in ArzEn corpus, along with their frequencies.

Affix Meaning Frequency Examples Translation Proclitics ال> Definite article
1,995 WEEKENDS+ال> the weekends ف> Conjunction ‘so/then’ 83 THEREFORE+ف> so
therefore ك> Preposition ‘as’ 60 STUDENT+ك> as (a) student ل> Preposition
‘to/for’ 45 LEVEL+ل> to (a) level ب> Preposition ‘with’ 36 ATTITUDE+ب> with
(an) attitude ه> or ح> Imperfect future verb 7 SUBMIT+ه> will submit Enclitics
ه> 3rd person possessive pronoun (masc.) 4 CAREER#ه> his career ها> 3rd person
direct object pronoun (fem.) 1 ها>#PROPAGATE+ي> (he) propagates it ى> 1st
person possessive pronoun 1 MOBILE#ى> my mobile Prefixes ا> 1st person
imperfect verb (sg.) 71 IMPLEMENT+ا> to implement ي> 3rd person imperfect verb
(masc.) 49 BREAK+ي> to break ب> Present tense marker 46 USE+ب> to use ت> 2nd
person imperfect verb (masc.) 33 PARTICIPATE+ت> to participate 3rd person
imperfect verb (fem.) SUPPORT+ت> to participate ن> 1st person imperfect verb
(pl.) 14 DETECT+ن> to detect م> Present participle 1 EXPECT+م> EXPECTING
Suffixes ات> Feminine plural 7 EVENT#ات> events نى> 1st person direct object
pronoun 1 نى>#EVALUATE+بي> he evaluates me Circumfixes وا> \+ ي> 3rd person
imperfect verb (pl.) 4 وا>#ADAPT+ي> (they) adapt ى> \+ ت> 2rd person imperfect
verb (fem.) 2 ى>#CODE+ت> to code وا> \+ ت> 2rd person imperfect verb (pl.) 1
وا>#PRESENT+ت> to present Combinations بال> Combination of the morphemes: ب>
(preposition) and ال> 85 Mobile+بال> with the mobile لل> Combination of the
morphemes: ل> and ال> 36 LIMIT+لل> to the limit بي> Combination of the
morphemes: ب> (present tense) and ي> 27 APPRECIATE+بي> (he/it) appreciates بت>
Combination of the morphemes: ب> (present tense) 19 OVERCOME+بت> (you)
overcome and ت> (2nd person imperfect verb (masc.)) Combination of the
morphemes: ب> (present tense) REQUIRE+بت> (she/it) requires and ت> (3rd person
imperfect verb (fem.)) بن> Combination of the morphemes: ب> (present tense)
and ن> 10 TARGET+بن> (we) target فال> Combination of the morphemes: ف> and ال>
6 TEAM+فال> so the team هي> Combination of the morphemes: ه> and ي> 6
REACT+هي> (he/it) will react هت> Combination of the morphemes: ه> and ت> 6
AFFECT+هت> (she/it) will affect كال> Combination of the morphemes: ك> and ال>
2 WORKING LIFE+كال> as (the) working life فب> Combination of the morphemes: ف>
and ب> (present tense) 2 GROW+فب> so (I) grow هن> Combination of the
morphemes: ه> and ن> 2 GUARANTEE+هن> (we) will guarantee وا> \+ بي>
Combination of the morphemes: ب> (present tense) and وا> \+ ي> circumfix 2
وا>#target+بي> (they) target ى> \+ بت> Combination of the morphemes: ب>
(present tense) and ى> \+ ت> circumfix 1 ى>#HANDLE+بت> to handle

Table 12: Affixes and clitics occurring in ArzEn Intra-word CS.

## References

  * Abu-Melhim [1991] Abu-Melhim, A.R., 1991.  Code-switching and linguistic accommodation in arabic, in: Perspectives on Arabic Linguistics III: Papers from the Third Annual Symposium on Arabic Linguistics, John Benjamins Publishing. pp. 231–250.
  * Ahmed and Tan [2012] Ahmed, B.H., Tan, T.P., 2012\.  Automatic speech recognition of code switching speech using 1-best rescoring, in: International Conference on Asian Language Processing (IALP), IEEE. pp. 137–140.
  * Akbar [2007] Akbar, R., 2007.  Students’ and teachers’ attitudes towards Kuwaiti English code-switching.  Cardiff University.
  * Al-Qaysi [2016] Al-Qaysi, N.J.M., 2016.  Examining Students’ and Educators’ Attitudes Towards the use of Code-Switching within Higher Educational Environments in Oman.  Ph.D. thesis. The British University in Dubai (BUiD).
  * Ali et al. [2016] Ali, A., Bell, P., Glass, J., Messaoui, Y., Mubarak, H., Renals, S., Zhang, Y., 2016\.  The mgb-2 challenge: Arabic multi-dialect broadcast media recognition, in: SLT, IEEE. pp. 279–284.
  * Ali and Renals [2018] Ali, A., Renals, S., 2018.  Word error rate estimation for speech recognition: e-wer, in: the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 20–24.
  * Ali et al. [2019] Ali, A., Shon, S., Samih, Y., Mubarak, H., Abdelali, A., Glass, J., Renals, S., Choukri, K., 2019.  The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech, in: ASRU, IEEE. pp. 1026–1033.
  * Ali et al. [2017] Ali, A., Vogel, S., Renals, S., 2017.  Speech recognition challenge in the wild: Arabic mgb-3, in: ASRU, IEEE. pp. 316–322.
  * Ali et al. [2014] Ali, A., Zhang, Y., Cardinal, P., Dahak, N., Vogel, S., Glass, J., 2014\.  A complete kaldi recipe for building arabic speech recognition systems, in: SLT, IEEE. pp. 525–529.
  * Ali [2018] Ali, A.M.A.M., 2018.  Multi-dialect arabic broadcast speech recognition .
  * Amazouz et al. [2017] Amazouz, D., Adda-Decker, M., Lamel, L., 2017.  Addressing code-switching in french/algerian arabic speech, in: Interspeech, pp. 62–66.
  * Amazouz et al. [2018] Amazouz, D., Adda-Decker, M., Lamel, L., 2018.  The french-algerian code-switching triggered audio corpus (facst), in: LREC, pp. 1468–1473.
  * Amazouz et al. [2016a] Amazouz, D., Martine, A.D., Lamel, L., 2016a.  Arabic-french code-switching across maghreb arabic dialects: a quantitative analysis, in: Corpus-driven Studies of Heterogeneous and Multilingual Corpora Workshop.
  * Amazouz et al. [2016b] Amazouz, D., Martine, A.D., Lamel, L., 2016b.  Arabic-french code-switching across maghreb arabic dialects: a quantitative analysis .
  * Bacha and Bahous [2011] Bacha, N.N., Bahous, R., 2011\.  Foreign language education in lebanon: A context of cultural and curricular complexities.  Journal of Language Teaching and Research 2, 1320–1328.
  * Baoueb [2009] Baoueb, L.B., 2009.  Social factors for code-switching in tunisian business companies: A case study.
  * Barik and Lestari [2019] Barik, R.E., Lestari, D.P., 2019\.  Text corpus and acoustic model addition for indonesian-arabic code-switching in automatic speech recognition system, in: International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), IEEE. pp. 1–6.
  * Bayeh et al. [2008] Bayeh, R., Mokbel, C., Chollet, G., 2008.  Broadcast news transcription baseline system using the nemlar database, in: Sixth International Conference on Language Resources and Evaluation (LREC): the workshop in HLT & NLP within the Arabic world, Marrakech, Morocco.
  * Bentahila [1983] Bentahila, A., 1983.  Motivations for code-switching among arabic-french bilinguals in morocco.  Language & communication 3, 233–243.
  * Bentahila and Davies [1983] Bentahila, A., Davies, E.E., 1983\.  The syntax of arabic-french code-switching.  Lingua 59, 301–330.
  * Bhuvanagirir and Kopparapu [2012] Bhuvanagirir, K., Kopparapu, S.K., 2012\.  Mixed language speech recognition without explicit identification of language.  American Journal of Signal Processing 2, 92–97.
  * Biswas et al. [2019] Biswas, A., Yılmaz, E., de Wet, F., van der Westhuizen, E., Niesler, T., 2019.  Semi-supervised acoustic model training for five-lingual code-switched asr, in: Interspeech, pp. 3745–3749.
  * Breslin [2008] Breslin, C., 2008.  Generation and combination of complementary systems for automatic speech recognition.  Ph.D. thesis. University of Cambridge.
  * Bynion and Feldner [2017] Bynion, T.M., Feldner, M.T., 2017\.  Self-assessment manikin.  Encyclopedia of personality and individual differences , 1–3.
  * Cardinal et al. [2014] Cardinal, P., Ali, A., Dehak, N., Zhang, Y., Hanai, T.A., Zhang, Y., Glass, J.R., Vogel, S., 2014\.  Recent advances in asr applied to an arabic transcription system for al-jazeera, in: Fifteenth annual conference of the international speech communication association, pp. 2088–2092.
  * Çetinoğlu [2017] Çetinoğlu, Ö., 2017.  A code-switching corpus of turkish-german conversations, in: Eleventh Linguistic Annotation Workshop, pp. 34–40.
  * Çetinoğlu et al. [2016] Çetinoğlu, Ö., Schulz, S., Vu, N.T., 2016.  Challenges of computational processing of code-switching, in: Second Workshop on Computational Approaches to Code Switching, pp. 1–11.
  * Chan et al. [2009] Chan, J.Y., Cao, H., Ching, P., Lee, T., 2009.  Automatic recognition of cantonese-english code-mixing speech.  International Journal of Computational Linguistics & Chinese Language Processing 14, 281–304.
  * Chiu et al. [2018] Chiu, C.C., Sainath, T.N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R.J., Rao, K., Gonina, E., et al., 2018\.  State-of-the-art speech recognition with sequence-to-sequence models, in: ICASSP, IEEE. pp. 4774–4778.
  * Chowdhury et al. [2020] Chowdhury, S.A., Samih, Y., Eldesouki, M., Ali, A., 2020\.  Effects of dialectal code-switching on speech modules: A study using egyptian arabic broadcast speech, in: Interspeech, pp. 2382–2386.
  * Das and Gambäck [2014] Das, A., Gambäck, B., 2014\.  Identifying languages at the word level in code-mixed indian social media text, in: Eleventh International Conference on Natural Language Processing, pp. 378–387.
  * Denisov and Vu [2019] Denisov, P., Vu, N.T., 2019\.  End-to-end multi-speaker speech recognition using speaker embeddings and transfer learning, in: Interspeech, pp. 4425–4429.
  * Deuchar et al. [2014] Deuchar, M., Davies, P., Herring, J., Couto, M.C.P., Carter, D., 2014.  Building bilingual corpora.  Advances in the Study of Bilingualism , 93–111.
  * Dey and Fung [2014] Dey, A., Fung, P., 2014.  A hindi-english code-switching corpus., in: LREC, pp. 2410–2413.
  * Dyer et al. [2013] Dyer, C., Chahuneau, V., Smith, N.A., 2013.  A simple, fast, and effective reparameterization of ibm model 2, in: NAACL-HLT, pp. 644–648.
  * Dyer et al. [2010] Dyer, C., Weese, J., Setiawan, H., Lopez, A., Ture, F., Eidelman, V., Ganitkevitch, J., Blunsom, P., Resnik, P., 2010.  cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models, in: ACL, pp. 7–12.
  * Eberhard et al. [2019] Eberhard, D., Simons, G.F., Fennig, C.D., 2019.  Ethnologue: Languages of the world (22nd edn.) dallas: Sil international .
  * El Choubassi et al. [2003] El Choubassi, M., El Khoury, H., Alagha, C.J., Skaf, J., Al-Alaoui, M., 2003.  Arabic speech recognition using recurrent neural networks, in: Third IEEE International Symposium on Signal Processing and Information Technology, IEEE. pp. 543–547.
  * Elfahal [2019] Elfahal, M.O.E., 2019.  Automatic Recognition and Identification for Mixed Sudanese Arabic–English Languages Speech.  Ph.D. thesis. Sudan University of Science & Technology.
  * Elmahdy et al. [2012] Elmahdy, M., Gruhn, R., Minker, W., 2012.  Novel techniques for dialectal arabic speech recognition.  Springer Science & Business Media.
  * Evermann and Woodland [2000] Evermann, G., Woodland, P., 2000\.  Posterior probability decoding, confidence estimation and system combination, in: Speech Transcription Workshop, Citeseer. pp. 78–81.
  * Fiscus [1997] Fiscus, J.G., 1997.  A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover), in: ASRU, IEEE. pp. 347–354.
  * Gadalla et al. [1997] Gadalla, H., Kilany, H., Arram, H., Yacoub, A., El-Habashi, A., Shalaby, A., Karins, K., Rowson, E., MacIntyre, R., Kingsbury, P., Graff, D., McLemore, C., 1997\.  Callhome egyptian arabic transcripts.  Linguistic Data Consortium, Philadelphia .
  * Goldberg [1992] Goldberg, L.R., 1992.  The development of markers for the big-five factor structure.  Psychological assessment 4, 26–42.
  * Gosling et al. [2003] Gosling, S.D., Rentfrow, P.J., Swann Jr, W.B., 2003.  A very brief measure of the big-five personality domains.  Journal of Research in personality 37, 504–528.
  * Hamed et al. [2018] Hamed, I., Elmahdy, M., Abdennadher, S., 2018.  Collection and analysis of code-switch egyptian arabic-english speech corpus, in: LREC, pp. 208–216.
  * Hamed et al. [2020] Hamed, I., Vu, N.T., Abdennadher, S., 2020.  Arzen: A speech corpus for code-switched egyptian arabic-english, in: LREC.
  * Hamed et al. [2019] Hamed, I., Zhu, M., Elmahdy, M., Abdennadher, S., Vu, N.T., 2019.  Code-switching language modeling with bilingual word embeddings: A case study for egyptian arabic-english, in: International Conference on Speech and Computer, pp. 160–170.
  * Haspelmath [2009] Haspelmath, M., 2009.  Lexical borrowing: Concepts and issues.  Loanwords in the world’s languages: A comparative handbook , 35–54.
  * Hillard et al. [2007] Hillard, D., Hoffmeister, B., Ostendorf, M., Schlüter, R., Ney, H., 2007.  irover: Improving system combination with classification, in: NAACL-HLT, pp. 65–68.
  * Hoffmeister et al. [2006] Hoffmeister, B., Klein, T., Schlüter, R., Ney, H., 2006\.  Frame based system combination and a comparison with weighted rover and cnc, in: Ninth International Conference on Spoken Language Processing, pp. 537–540.
  * Hoffmeister et al. [2008] Hoffmeister, B., Schlüter, R., Ney, H., 2008.  icnc and irover: The limits of improving system combination with classification?, in: Interspeech, pp. 232–235.
  * Hori et al. [2017] Hori, T., Watanabe, S., Zhang, Y., Chan, W., 2017\.  Advances in joint ctc-attention based end-to-end speech recognition with a deep cnn encoder and rnn-lm, in: Interspeech, pp. 949–953.
  * Ismail [2015] Ismail, M.A., 2015.  The sociolinguistic dimensions of code-switching between arabic and english by saudis.  International Journal of English Linguistics 5, 99–109.
  * Karita et al. [2019] Karita, S., Soplin, N.E.Y., Watanabe, S., Delcroix, M., Ogawa, A., Nakatani, T., 2019\.  Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration, in: Interspeech, pp. 1408–1412.
  * Khuwaileh [2003] Khuwaileh, A.A., 2003.  Code switching and multilingualism in a small multi-ethnic group society (uae).  Journal of Language for International Business 14, 32–49.
  * Kilany et al. [1997] Kilany, H., Gadalla, H., Arram, H., Yacoub, A., , El-Habashi, A., McLemore, C., 1997\.  Egyptian colloquial arabic lexicon.  Linguistic Data Consortium, Philadelphia .
  * Kingsbury et al. [2011] Kingsbury, B., Soltau, H., Saon, G., Chu, S., Kuo, H.K., Mangu, L., Ravuri, S., Morgan, N., Janin, A., 2011\.  The ibm 2009 gale arabic speech transcription system, in: ICASSP, IEEE. pp. 4672–4675.
  * Kirchhoff et al. [2003] Kirchhoff, K., Bilmes, J., Das, S., Duta, N., Egan, M., Ji, G., He, F., Henderson, J., Liu, D., Noamany, M., et al., 2003.  Novel approaches to arabic speech recognition: report from the 2002 johns-hopkins summer workshop, in: ICASSP, IEEE. pp. 344–347.
  * Kirchhoff et al. [2006] Kirchhoff, K., Vergyri, D., Bilmes, J., Duh, K., Stolcke, A., 2006.  Morphology-based language modeling for conversational arabic speech recognition.  Computer Speech & Language 20, 589–608.
  * Ko et al. [2015] Ko, T., Peddinti, V., Povey, D., Khudanpur, S., 2015\.  Audio augmentation for speech recognition, in: Sixteenth Annual Conference of the International Speech Communication Association, pp. 3586–3589.
  * Launay et al. [2002] Launay, B., Siohan, O., Surendran, A., Lee, C.H., 2002\.  Towards knowledge-based features for hmm based large vocabulary automatic speech recognition, in: ICASSP, IEEE. pp. 817–820.
  * Li and Vu [2019] Li, C.Y., Vu, N.T., 2019.  Integrating knowledge in end-to-end automatic speech recognition for mandarin-english code-switching, in: International Conference on Asian Language Processing (IALP), pp. 160–165.
  * Li et al. [2019] Li, K., Li, J., Ye, G., Zhao, R., Gong, Y., 2019.  Towards code-switching asr for end-to-end ctc models, in: ICASSP, IEEE. pp. 6076–6080.
  * Li et al. [2002] Li, X., Singh, R., Stern, R.M., 2002.  Lattice combination for improved speech recognition, in: ICSLP, pp. 405–408.
  * Li et al. [2012] Li, Y., Yu, Y., Fung, P., 2012\.  A mandarin-english code-switching corpus., in: LREC, pp. 2515–2519.
  * Luo et al. [2018] Luo, N., Jiang, D., Zhao, S., Gong, C., Zou, W., Li, X., 2018.  Towards end-to-end code-switching speech recognition.  arXiv preprint arXiv:1810.13091 .
  * Lyu and Lyu [2008] Lyu, D.C., Lyu, R.Y., 2008.  Language identification on code-switching utterances using multiple cues, in: Interspeech, pp. 711–714.
  * Lyu et al. [2006] Lyu, D.C., Lyu, R.Y., Chiang, Y.c., Hsu, C.N., 2006\.  Speech recognition on code-switching among the chinese dialects, in: ICASSP, IEEE. pp. 1105–1108.
  * Lyu et al. [2015] Lyu, D.C., Tan, T.P., Chng, E.S., Li, H., 2015.  Mandarin–english code-switching speech corpus in south-east asia: Seame.  Language Resources and Evaluation 49, 581–600.
  * Ma et al. [2010] Ma, C., Kuo, H.K.J., Soltau, H., Cui, X., Chaudhari, U., Mangu, L., Lee, C.H., 2010.  A comparative study on system combination schemes for lvcsr, in: ICASSP, IEEE. pp. 4394–4397.
  * Mangu et al. [2011] Mangu, L., Kuo, H.K., Chu, S., Kingsbury, B., Saon, G., Soltau, H., Biadsy, F., 2011\.  The ibm 2011 gale arabic speech transcription system, in: ASRU, IEEE. pp. 272–277.
  * Modipa et al. [2015] Modipa, T.I., Davel, M.H., De Wet, F., 2015.  Implications of sepedi/english code switching for asr systems, in: Pattern Recognition Association of South Africa, pp. 112–117.
  * Mustafa and AL-KHATIB [1994] Mustafa, Z., AL-KHATIB, M., 1994\.  Code-mixing of arabic and english in teaching science.  World Englishes 13, 215–224.
  * Najafian et al. [2017] Najafian, M., Hsu, W.N., Ali, A., Glass, J., 2017.  Automatic speech recognition of arabic multi-genre broadcast media, in: ASRU, IEEE. pp. 353–359.
  * Omar and Ilyas [2018] Omar, A., Ilyas, M., 2018.  The sociolinguistic significance of the attitudes towards code-switching in saudi arabia academia.  International Journal of English Linguistics 8, 79–91.
  * Panayotov et al. [2015] Panayotov, V., Chen, G., Povey, D., Khudanpur, S., 2015\.  Librispeech: an asr corpus based on public domain audio books, in: ICASSP, IEEE. pp. 5206–5210.
  * Pandey et al. [2017] Pandey, A., Srivastava, B.M.L., Gangashetty, S.V., 2017.  Adapting monolingual resources for code-mixed hindi-english speech recognition, in: International Conference on Asian Language Processing (IALP), IEEE. pp. 218–221.
  * Park et al. [2019] Park, D.S., Chan, W., Zhang, Y., Chiu, C.C., Zoph, B., Cubuk, E.D., Le, Q.V., 2019.  Specaugment: A simple data augmentation method for automatic speech recognition, in: Interspeech, pp. 2613–2617.
  * Peddinti et al. [2015] Peddinti, V., Povey, D., Khudanpur, S., 2015.  A time delay neural network architecture for efficient modeling of long temporal contexts, in: Sixteenth Annual Conference of the International Speech Communication Association, pp. 3214–3218.
  * Povey et al. [2011] Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., Hannemann, M., Motlicek, P., Qian, Y., Schwarz, P., et al., 2011\.  The kaldi speech recognition toolkit, in: ASRU, IEEE Signal Processing Society. pp. 1–4.
  * Ramanarayanan and Suendermann-Oeft [2017] Ramanarayanan, V., Suendermann-Oeft, D., 2017\.  Jee haan, i’d like both, por favor: Elicitation of a code-switched corpus of hindi-english and spanish-english human-machine dialog., in: Interspeech, pp. 47–51.
  * Sankar [2005] Sankar, A., 2005.  Bayesian model combination (baycom) for improved recognition, in: ICASSP, IEEE. pp. 845–848.
  * Saon et al. [2013] Saon, G., Soltau, H., Nahamoo, D., Picheny, M., 2013\.  Speaker adaptation of neural network acoustic models using i-vectors, in: ASRU, IEEE. pp. 55–59.
  * Shan et al. [2019] Shan, C., Weng, C., Wang, G., Su, D., Luo, M., Yu, D., Xie, L., 2019.  Investigating end-to-end speech recognition for mandarin-english code-switching, in: ICASSP, IEEE. pp. 6056–6060.
  * Shen et al. [2011] Shen, H.P., Wu, C.H., Yang, Y.T., Hsu, C.S., 2011\.  Cecos: A chinese-english code-switching speech database, in: International Conference on Speech Database and Assessments (Oriental COCOSDA), IEEE. pp. 120–123.
  * Sivasankaran et al. [2018] Sivasankaran, S., Srivastava, B.M.L., Sitaram, S., Bali, K., Choudhury, M., 2018.  Phone merging for code-switched speech recognition, in: Third Workshop on Computational Approaches to Linguistic Code-Switching, pp. 11–19.
  * Solorio and Liu [2008] Solorio, T., Liu, Y., 2008.  Learning to predict code-switching points, in: EMNLP, pp. 973–981.
  * Soltau et al. [2011] Soltau, H., Mangu, L., Biadsy, F., 2011.  From modern standard arabic to levantine asr: Leveraging gale for dialects, in: ASRU, IEEE. pp. 266–271.
  * Sreeram et al. [2018] Sreeram, G., Dhawan, K., Sinha, R., 2018.  Hindi-english code-switching speech corpus.  arXiv preprint arXiv:1810.00662 .
  * Srivastava and Sitaram [2018] Srivastava, B.M.L., Sitaram, S., 2018\.  Homophone identification and merging for code-switched speech recognition., in: Interspeech, pp. 1943–1947.
  * Stolcke [2002] Stolcke, A., 2002.  Srilm-an extensible language modeling toolkit, in: Seventh International Conference on Spoken Language Processing, pp. 901–904.
  * Stolcke et al. [2000] Stolcke, A., Bratt, H., Butzberger, J., Franco, H., Gadde, V.R., Plauché, M., Richey, C., Shriberg, E., Sönmez, K., Weng, F., et al., 2000\.  The sri march 2000 hub-5 conversational speech transcription system, in: NIST Speech Transcription Workshop, Citeseer.
  * Toshniwal et al. [2018] Toshniwal, S., Sainath, T.N., Weiss, R.J., Li, B., Moreno, P., Weinstein, E., Rao, K., 2018.  Multilingual speech recognition with a single end-to-end model, in: ICASSP, IEEE. pp. 4904–4908.
  * Vu et al. [2012] Vu, N.T., Lyu, D.C., Weiner, J., Telaar, D., Schlippe, T., Blaicher, F., Chng, E.S., Schultz, T., Li, H., 2012.  A first speech recognition system for mandarin-english code-switch conversational speech, in: ICASSP, IEEE. pp. 4889–4892.
  * Watanabe et al. [2018] Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y., Soplin, N.E.Y., Heymann, J., Wiesner, M., Chen, N., et al., 2018\.  Espnet: End-to-end speech processing toolkit, in: Interspeech, pp. 2207–2207.
  * Watanabe et al. [2017] Watanabe, S., Hori, T., Kim, S., Hershey, J.R., Hayashi, T., 2017.  Hybrid ctc/attention architecture for end-to-end speech recognition.  IEEE Journal of Selected Topics in Signal Processing 11, 1240–1253.
  * van der Westhuizen and Niesler [2016] van der Westhuizen, E., Niesler, T., 2016\.  Automatic speech recognition of english-isizulu code-switched speech from south african soap operas.  Procedia Computer Science 81, 121–127.
  * Winata et al. [2018] Winata, G.I., Madotto, A., Wu, C.S., Fung, P., 2018.  Towards end-to-end automatic code-switching speech recognition.  arXiv preprint arXiv:1810.12620 .
  * Xu et al. [2010] Xu, H., Povey, D., Mangu, L., Zhu, J., 2010.  An improved consensus-like method for minimum bayes risk decoding and lattice combination, in: ICASSP, IEEE. pp. 4938–4941.
  * Yilmaz et al. [2016] Yilmaz, E., Andringa, M., Kingma, S., Dijkstra, J., Kuip, F., Velde, H., Kampstra, F., Algra, J., Heuvel, H., van Leeuwen, D.A., 2016\.  A longitudinal bilingual frisian-dutch radio broadcast database designed for code-switching research, in: LREC, pp. 4666–4669.
  * Yılmaz et al. [2018] Yılmaz, E., Biswas, A., van der Westhuizen, E., de Wet, F., Niesler, T., 2018.  Building a unified code-switching asr system for south african languages, in: Interspeech, pp. 1923–1927.
  * Yue et al. [2019] Yue, X., Lee, G., Yılmaz, E., Deng, F., Li, H., 2019.  End-to-end code-switching asr for low-resourced language pairs, in: ASRU, pp. 972–979.
  * Zeng et al. [2018] Zeng, Z., Khassanov, Y., Pham, V.T., Xu, H., Chng, E.S., Li, H., 2018.  On the end-to-end solution to mandarin-english code-switching speech recognition, in: Interspeech, pp. 2165–2169.