# ArzEn-ST: A Three-way Speech Translation Corpus for
Code-Switched Egyptian Arabic - English

Injy Hamed,1,2 Nizar Habash,1 Slim Abdennadher,3 Ngoc Thang Vu2
1Computational Approaches to Modeling Language Lab, New York University Abu
Dhabi
2Institute for Natural Language Processing, University of Stuttgart
3Computer Science Department, The German University in Cairo
<EMAIL_ADDRESS>
###### Abstract

utf8

##  1 Introduction

Code-switching (CSW), defined as the alternation of language in text or
speech, is a common linguistic phenomenon in multilingual societies. CSW can
occur on the boundaries of sentences, words (within the same sentence), or
morphemes (within the same word). While the worldwide prevalence of CSW has
been met with increasing efforts in NLP systems trying to handle such mixed
input, data sparsity remains one of the main bottlenecks hindering the
development of such systems Çetinoğlu et al. (2016). In this paper, we present
ArzEn-ST,111Arz is the ISO 639-3 code for Egyptian Arabic. a speech
translation (ST) corpus for code-switched Egyptian Arabic (Egy) - English. We
extend the ArzEn Egyptian Arabic-English CSW conversational speech corpus
Hamed et al. (2020) with translations going to both directions; the primary
(Egyptian Arabic) as well as secondary (English) languages. See Figure 1. This
corpus is a valuable resource filling an important gap, given the naturalness
and high frequency of CSW in it. It can be used for the purpose of linguistic
investigations as well as for building and evaluating NLP systems. We provide
benchmark baseline results for the tasks of automatic speech recognition
(ASR), machine translation (MT), and ST. We make the translation guidelines
and full corpus available, as well as the experiments’ scripts and data
splits.222http://arzen.camel-lab.com/ The paper is organized as follows. In
Section 2, we provide an overview of previous work done for code-switched ASR,
MT, and ST tasks as well as corpora collection. In Section 3, we provide an
overview of the ArzEn speech corpus. In Section 4, we elaborate on the
translation guidelines used to create the three-way parallel ST corpus.
Finally, in Section 5, we report the performance of the ASR, MT, and ST
baseline systems.

Figure 1: An example from the corpus, showing the four representations for
each utterance: audio, transcription, Egyptian Arabic translation, and English
translation.

##  2 Related Work

Language | Citations
---|---
Translating CSW $\rightarrow$ monolingual
Hindi-English $\rightarrow$ English |  Dhar et al. (2018); Srivastava and Singh (2020); Tarunesh et al. (2021); Chen et al. (2022)
Sinhala-English $\rightarrow$ Sinhala |  Kugathasan and Sumathipala (2021)
English-Spanish $\rightarrow$ English |  Chen et al. (2022)
MSA-Egyptian Arabic $\rightarrow$ English |  Chen et al. (2022)
English-Bengali $\rightarrow$ both |  Mahata et al. (2019)
Egyptian Arabic -English $\rightarrow$ both |  ArzEn-ST (the corpus presented in this paper)
Translating monolingual $\rightarrow$ CSW
Hindi $\rightarrow$ Hindi-English |  Tarunesh et al. (2021); Banerjee et al. (2018)
English $\rightarrow$ Bengali-English |  Banerjee et al. (2018)
English $\rightarrow$ Gujarati-English |  Banerjee et al. (2018)
English $\rightarrow$ Tamil-English |  Banerjee et al. (2018)
English, Hindi $\rightarrow$ Hindi-English |  Srivastava and Singh (2021)
Table 1: Overview on available human-annotated CSW-focused parallel corpora.

###  2.1 CSW Automatic Speech Recognition

CSW ASR has gained a considerable amount of research Vu et al. (2012); Li and
Vu (2019); Ali et al. (2021); Hamed et al. (2022a); Hussein et al. (2022),
where several CSW speech corpora have been collected, covering multiple
language pairs, including Chinese-English Lyu et al. (2015), Hindi-English
Ramanarayanan and Suendermann-Oeft (2017), Spanish-English Solorio and Liu
(2008), Arabic-English Ismail (2015); Hamed et al. (2018, 2020); Chowdhury et
al. (2021), Arabic-French Djegdjiga et al. (2018), Frisian-Dutch Yilmaz et al.
(2016), Mandarin-Taiwanese Lyu et al. (2006), Turkish-German Çetinoğlu (2017),
English-Malay Ahmed and Tan (2012), English-isiZulu van der Westhuizen and
Niesler (2016) and Sepedi-English Modipa et al. (2013). In this work, we build
on our ArzEn speech corpus Hamed et al. (2020), and enrich it with multiple
translations.

###  2.2 CSW Machine Translation

While research in CSW MT has been gaining attention over the past years Sinha
and Thakur (2005); Dhar et al. (2018); Mahata et al. (2019); Menacer et al.
(2019); Song et al. (2019); Tarunesh et al. (2021); Xu and Yvon (2021); Chen
et al. (2022); Hamed et al. (2022b); Gaser et al. (2022), the collected CSW
parallel corpora are limited. By looking into the reported corpora, we
identify a number of dimensions in which they vary. First is synthetic or
human-annotated data. Second, for human-annotated data, it can be either
collected, or especially commissioned for MT/NLP. Third, for collected data,
it can be obtained from textual or speech sources. And finally, the data set
may include translations to one or more languages. To circumvent the data
scarcity issue, researchers investigated the use of synthetically generated
CSW parallel data for training and testing Gupta et al. (2020); Yang et al.
(2020); Xu and Yvon (2021). While this is acceptable for training purposes,
synthetic data should not be used for testing, as it does not reflect real-
world CSW distributions. For collecting human-annotated parallel corpora,
researchers have tried either asking bilingual speakers to translate naturally
occurring CSW sentences into monolingual sentences, or as another solution to
data scarcity, have commissioned annotators to translate monolingual sentences
into CSW sentences. In Table 1, we present a summary of available human-
annotated parallel corpora that are focused on CSW. For the latter approach,
we note that generating CSW data in a human-commissioned fashion could differ
from naturally-occurring CSW sentences. Such data could be biased to the
grammatical structure of the monolingual sentences, and could be dominated by
single noun switches, being the easiest CSW type to generate. The former
approach of translating naturally-occurring CSW sentences into monolingual
sentences is the most optimal way to collect a CSW parallel corpus; however,
most of the collected corpora rely on CSW sentences obtained from textual
sources (mostly from social media platforms). The main concern here is that
CSW phenomena occurring in text are more restricted than those occurring in
natural speech. In text, people are usually dissuaded from changing scripts,
and therefore either avoid switching languages, or switch languages without
switching scripts. The latter issue was tackled by Shazal et al. (2020), where
the authors used a sequence-to-sequence deep learning model to transliterate
SMS/chat text collected by Chen et al. (2017) from Arabizi (where Arabic words
are written in Roman script) to Arabic orthography. While this corpus is not
focused on CSW, it contains CSW sentences. Finally, we categorize the
collected corpora in terms of the translation direction. As shown in Table 1,
most of the corpora include translations for CSW sentences to the secondary
language, which is most commonly English. A smaller number of researchers
investigated translating CSW sentences into the primary language. And even
fewer researchers included translations to both directions.The work of Menacer
et al. (2019) is also relevant to our work. The authors extracted Modern
Standard Arabic (MSA)-English CSW sentences from the UN documents, to which
English translations are available Eisele et al. (2010). The Arabic
translations were generated by translating the English segments using the
Google Translate API. While this can be used for training purposes, these
translations should not be used as gold reference. Moreover, given the nature
of the corpus, it contained limited types of CSW, as opposed to the types that
occur in conversational speech.

###  2.3 CSW Speech Translation

Work on CSW ST is still in its early stages, with little prior work Nakayama
et al. (2019); Weller et al. (2022); Huber et al. (2022). For CSW ST corpora,
two corpora are available for Spanglish: Bangor Miami Cieri et al. (2004) and
Fisher Deuchar et al. (2014). While the Fisher dataset is not a CSW-focused
corpus, it contains a considerable amount of CSW Weller et al. (2022).
Similarly, for CSW Egyptian Arabic-English, the Callhome dataset also contains
some amount of CSW Gadalla et al. (1997); Kumar et al. (2014). A Japanese-
English ST corpus Nakayama et al. (2019) was also collected, however it
includes read-speech and not spontaneous speech. Huber et al. (2022) collected
a one-hour German-English code-switching speech translation corpus containing
read-speech. Our new corpus, ArzEn-ST, fills an important resource gap,
providing an ST corpus for code-switched Egyptian Arabic-English. The corpus
is human-annotated where the source sentences are collected through interviews
with bilingual speakers, and contain naturally-generated CSW sentences; they
are then translated in both directions: monolingual Egyptian Arabic and
monolingual English.

CSW Type | Example
---|---
Inter-sentential CSW | It’s very difficult making friends at work. .عملت صحاب بس مش فى الشغل>
|  I made friends, but not at work. It’s very difficult making friends at
work.
Extra-sentential CSW |  أنا مولود فى مصر فى القاهرة > Okay
| Okay I was born in Egypt, in Cairo.
Intra-sentential CSW |  related to research جزء> actually كان فى>
|  There was actually a part related to research
Intra-word CSW |  ؟> project+لل> بتاعك> expectation+ال> ايه>
|  What is your expectation for the project?
Explicatory CSW |  أو مقولة قرتيها فى كتاب وعجبتك؟> quote طيب فى مثلا > Okay
| Okay okay is there a quote or a quote that you read in a book and liked?
Elaboratory CSW |  عربي> author او هي شرقية فهقراها اه من .. من> oriental دي يعني هى> novel لو ال>
|  If this novel is like oriental or it is [originally written] in Arabic then
I will
|  read it ah from .. from an Arab author [in its Arabic version].
Table 2: Examples of different CSW types in ArzEn followed by their English
translation. The originally Arabic phrases are italicized in the English
translation. For Explicatory and Elaboratory CSW, the underlining marks the
repeated phrases.

##  3 Overview of the ArzEn Corpus

ArzEn is a conversational speech corpus that is collected through informal
interviews. The interviews were held at the German University in Cairo, which
is a private university where English is the instruction language. The topics
discussed were general topics such as education, work and life experiences,
career, technology, personal life, hobbies, and travelling experiences. No
instructions were given to participants regarding code-switching; they were
not asked to produce nor avoid code-switching. Interviews were held with 38
Egyptian Arabic-English bilingual speakers (61.5% males, 38.5% females), in
the age range of 18-35, who are students (55%) and employees (45%) at the
university. The speech corpus comprises of 12 hours of speech, containing
6,216 sentences.

###  3.1 Code-switching Types in ArzEn

The four main CSW types mentioned in the literature are present in ArzEn
Poplack (1980); Stefanich et al. (2019). We present a corpus example for each
of the types in Table 2.

#### Inter-sentential CSW

This type of CSW is defined as switching languages from one sentence to
another.

#### Extra-sentential CSW

This type of CSW, also called tag-switching, is where tag elements from one
language are inserted into a monolingual sentence in another language, without
the need for grammatical considerations. It mostly involves the use of
fillers, interjections, tags, and idiomatic expressions. This type of CSW
requires only minimal knowledge of the grammar of the secondary language.

#### Intra-sentential CSW

This type, also referred to as code-mixing, is defined as using multiple
languages within the same sentence, where the CSW segments must conform to the
underlying syntactic rules of both languages. This type of CSW requires a
better understanding of the grammar of both languages, compared to extra-
sentential CSW.

#### Intra-word CSW

This type, also called morphological CSW, is where switching occurs at the
level of morphemes. Given that Egyptian Arabic is a morphologically rich
language Habash et al. (2012b), morphological code-switching occurs where
Egyptians attach Arabic clitics and affixes to English words. In addition to
the above, and motivated by our interest in translation from CSW texts, we
identify two types of repetitive CSW phenomena in terms of their communicative
purposes.

#### Explicatory CSW

This type of CSW is where the speaker simply repeats the same word in another
language.

#### Elaboratory CSW

This type of CSW is where the speaker code-switches to further elaborate on
the meaning. Both types are challenging in terms of handling the CSW
repetitions when translating into a single language. We address these issues
in Section 4.

###  3.2 Code-switching Statistics in ArzEn

ArzEn contains a considerable amount of CSW. On the sentence level, 33.2% of
the sentences are monolingual Arabic, 3.1% are monolingual English, and 63.7%
code-mixed. Among the code-mixed sentences, 46.0% have morphological CSW. On
the word level, in the code-mixed sentences, 81.3% of the words are Arabic,
15.2% are English, and 3.4% are morphologically CSW words. Morphological CSW
in ArzEn involves the use of both Arabic clitics and affixes. A list of the
clitics and affixes occurring in morphological code-switched words present in
the ArzEn corpus and their frequencies are provided in Hamed et al. (2022a).

###  3.3 Input Transcription

The ArzEn collected interviews were manually transcribed by Egyptian Arabic-
English bilingual speakers. The transcribers were requested to use Arabic
script for Arabic words and Roman script for English words. For morphological
CSW words, Arabic clitics and affixes are written in Arabic script and English
words are written in Roman script, as follows: Arabic prefixes/proclitics +
English words # Arabic suffixes/enclitics, for example ات>#TASK+ال>
Al+TASK#At333Transliteration in the HSB scheme Habash et al. (2007).
‘the+task#s’. While the transcribers generally followed the rules in a strict
manner, we observe script confusion in the case of borrowed words that have
become strongly embedded in Egyptian Arabic. In such cases, transcriptions can
contain occurrences of the same words in both scripts, such as mobile and
موبايل> mwbAyl, film and فيلم> fylm, and camera and كاميرا> kAmyrA. Given the
spontaneous nature of the corpus, disfluencies were found due to repetitions,
corrections, and changing course/structure mid-sentence. Such disfluencies
were marked with ‘..’, which occurs in more than 26% of the corpus sentences.
The following tags were also used for non-speech parts: [HES] for hesitation,
[HUM] for humming, [COUGH], [LAUGHTER], and [NOISE].

##  4 ArzEnST Translation Guidelines

The transcriptions are translated to monolingual English and monolingual
Egyptian Arabic sentences by human translators.444English translations are
performed by one translator, and the dev and test sets are revised by one of
the authors. Egyptian Arabic translations are performed by one translator and
revised by another. In this section, we discuss the translation guidelines. In
general, our decisions are mainly guided by giving a higher priority to
fluency over accuracy. We opt for producing as natural as possible outputs
that reflect the style of the original sentence. Even though we acknowledge
that some of our decisions can make the translation task harder for MT
systems, our goal is to produce natural translations. The guidelines cover
three categories, general translation rules (denoted by GR), conversational
speech translation rules (denoted by SR), and code-switching translation rules
(denoted by CSWR). In Table 3, we present translation examples covering some
of the rules.

###  4.1 General Translation Rules (GR)

#### [GRintended]

Translators are requested to provide natural translations with the intended
meaning rather than literal translations. This also covers the case of
idiomatic expressions. See Table 3 (a).

#### [GRdifficult]

Similar to the LDC Arabic-to-English Translation Guidelines LDC (2013),
segments that are difficult to translate should be indicated using ((text)).
Such cases usually contain highly dialectal Arabic words or Arabic idioms. See
Table 3 (b).

#### [GRabbrev]

For all abbreviations, we made the decision to provide transliteration as
pronounced instead of translation,555We plan to annotate these cases with full
translations in the future, to assist in tasks interested in removing
English/Arabic text. for example NLP is transliterated as ان ال بي> An Al by,
and AIESEC is transliterated as آيزيك> Āyzyk.

#### [GRpropn]

Non-abbreviated proper nouns should be transliterated, unless they have
meaning. In that case, they should be translated as long as the meaning of the
sentence remains coherent, otherwise, should be transliterated.666The
translators were advised to refer to Wikipedia Arabic for the translations of
titles of books and films. See Table 3 (c).

###  4.2 Conversational Speech Translation Rules (SR)

#### [SRstyle]

Translations should capture the same fluency and style of the original text.
This means that disfluencies such as repetitions should also be included in
translations. See Table 3 (d).

#### [SRpunc+]

Punctuation, non-speech tags, and disfluency marks ‘..’ present in the source
text should be kept the same and in the same relative position in the sentence
in the target translation.

#### [SRpartial]

Due to disfluencies, it is common to have partial Arabic words. We
transliterate such partial words, and similar to LDC (2013), we mark them with
a preceding ‘%’ sign. See Table 3 (e).

###  4.3 Code-switching Translation Rules (CSWR)

Table 3: Translation examples following different guideline rules.

#### [CSWRborrowed]

For English words that are commonly used in Arabic, an attempt should first be
made to identify a commonly used reasonable translation, otherwise,
translators are allowed to transliterate. Examples of the latter case,
included loanwords such as mobile and laptop that have become strongly
integrated in Arabic, as opposed to online and presentation which can be
translated to عبر الأنترنت> and عرض تقديمي>, respectively.

#### [CSWRrewrites]

We allow modifications to CSW segments when translating into English for the
purpose of achieving better fluency. Similarly, when translating into Arabic,
we also allow slight modifications to the original Arabic words. We elaborate
on such cases for both directions below.

####  CSW$\rightarrow$En:

We allow modification to the original English words. This is mainly needed to
handle difference in grammatical structures across languages as well as
morphological CSW. For example, ASK+بي> by+ASK is translated as ‘he asks’. See
Table 3 (f-g).

####  CSW$\rightarrow$Ar:

It is allowed to slightly modify the original Arabic words for better fluency.
The following are common cases where this is needed.

  * •

Since Arabic makes heavy use of the definite article +ال> Al+ to mark
different constructions such as adjectival modification and idafa (possessive
construct), translators are given permission to drop/reassign the placement of
definite articles for the purpose of maximizing fluency. For example, the
adjectival construction working life +ال> gets translated to الحياة المهنية>
(with two definite article instances).

  * •

Given that Arabic is a gender-marking language, gender reinflection is
sometimes needed to guarantee fluent translation into Arabic. See Table 3 (h).

  * •

Modifying Arabic prepositions following English words. See Table 3 (i).

#### [CSWRreorder]

We allow changing the order between the original Arabic and English words to
handle syntactic divergences between the two languages and achieve better
fluency. See Table 3 (j).

#### [CSWRstyle]

The SRstyle rule gets further compounded in the case of CSW when repetition
occurs at a location where there is syntactic divergence between both
languages, such as adjectival phrases in our language pair. In this case,
since the order of words changes during translation, the word to be repeated
at that position could also change. Following our fluency preference, we
prefer the translation that gives higher fluency over providing an accurate
literal translation. See Table 3 (k).

#### [CSWRdisfluency]

Another interesting translation challenge arises in the context of CSW when
speakers repeat words using different languages. In the case of explicatory
CSW, where the English and Arabic words have the exact same meaning, we allow
translating the English word into the same present Arabic word, treating it as
a case of repetition due to disfluency. In the case of elaboratory CSW, where
CSW is used to further elaborate on meaning, we ask the translator to find
another translation of the word that would better capture the subtle
difference between both words. If that is not possible, we allow the
repetition of the word.

| ASR | MT | ST
---|---|---|---
|  | CSW$\rightarrow$En | CSW$\rightarrow$Egy | CSW$\rightarrow$En | CSW$\rightarrow$Egy
Training Set | WER | CER | BLEU | BLEU | BLEU | BLEU
ArzEn-ST | 57.9 | 36.2 | 8.6 | 48.0 | 4.5 | 13.0
ArzEn-ST + Extra | 34.7 | 20.0 | 34.3 | 79.8 | 16.5 | 31.1
Table 4: Summary of results for baseline systems evaluated on ArzEn-ST test
set. We present baseline systems for both settings: (1) training using ArzEn-
ST data only and (2) training using ArzEn-ST data with Extra monolingual
speech corpora for the ASR system and Extra monolingual Egyptian Arabic-
English parallel sentences for the MT systems. We report Word Error Rate (WER)
and Character Error Rate (CER) for ASR systems, and BLEU score Papineni et al.
(2002) using SacrebleuBLEU Post (2018) for MT and ST systems.

##  5 Benchmarking Baseline Systems

In this section, we discuss the ASR, MT, and ST baseline systems. We describe
the experimental setup for each and present the results in Table 4.

###  5.1 Experimental Setup

We follow the same train, dev, and test splits defined in Hamed et al. (2020).
For all the experiments, we use ArzEn-ST dev set (1,402 sentences) for tuning
and ArzEn-ST test set (1,470 sentences) for testing. For training, we use
ArzEn-ST train set (3,344 sentences), in addition to other monolingual data
which we mention below.

#### Automatic Speech Recognition

We train a joint CTC/attention based E2E ASR system using ESPnet Watanabe et
al. (2018). The encoder and decoder consist of 12 and 6 Transformer blocks
with 4 heads, feed-forward inner dimension 2048 and attention dimension 256.
The CTC/attention weight $(\lambda_{1})$ is set to 0.3. SpecAugment Park et
al. (2019) is applied for data augmentation. For the Language Model (LM), the
RNNLM consists of 1 LSTM layer with 1000 hidden units and is trained for 20
epochs. For decoding, the beam size is 20 and the CTC weight is 0.2. In
addition to using ArzEn-ST for training, we also train the ASR system and LM
using Callhome Gadalla et al. (1997), MGB-3 Ali et al. (2017), a 5-hours
subset from Librispeech Panayotov et al. (2015), and a 5-hours subset from
MGB-2 Ali et al. (2016).777We followed the setup used in Hamed et al. (2022a).
We perform Alif/Ya normalization (Arabic), remove punctuation and corpus-
specific annotations, and lower-case English words.888For the Callhome corpus,
we removed partial words.

#### Machine Translation

We train Transformer models using Fairseq Ott et al. (2019) on a single
GeForce RTX 3090 GPU. We use the hyperparameters from the FLORES benchmark for
low-resource machine translation Guzmán et al. (2019). The hyperparameters are
given in Appendix A. For each MT model, we use a BPE model trained jointly on
source and target sides. The BPE model is trained using Fairseq with
character_coverage set to $1.0$. We tune the vocabulary size for each
experiment for the values of $1k,3k,5k,8k,$ and $16k$. In addition to ArzEn-
ST, we also train the MT system using $324k$ extra Egyptian Arabic-English
parallel sentences obtained from the following parallel corpora: Callhome
Egyptian Arabic-English Speech Translation Corpus Kumar et al. (2014),
LDC2012T09 Zbib et al. (2012), LDC2017T07 Chen et al. (2017), LDC2019T01 Chen
et al. (2019), LDC2020T05 Li et al. (2020), and MADAR Bouamor et al.
(2018).999For corpora with no defined data splits, we follow the guidelines
provided in Diab et al. (2013). These extra corpora include $15k$ sentences
with CSW instances. When translating into En, we use all these extra corpora
as Arabic-English training. However, when translating into Egy, we use these
extra corpora as English-Arabic training, but we exclude the $15k$ sentences
producing CSW Arabic as our reference does not have CSW sentences. Data
preprocessing involved removing all corpus-specific annotations, URLs and
emoticons, lowercasing, running Moses’ Koehn et al. (2007) tokenizer, MADAMIRA
Pasha et al. (2014) simple tokenization (D0) and Alif/Ya normalization
(Arabic).

#### Speech Translation

We build a cascaded speech translation system, where we train an ASR system
and use an MT system to translate the ASR system’s outputs. We opt for a
cascaded system over an end-to-end system due to the limitation of available
resources to build an end-to-end system, in addition to the fact that cascaded
systems have been shown to outperform end-to-end systems in low-resource
settings Denisov et al. (2021).

###  5.2 Results

Table 4 presents the results for the MT and ST baseline systems. We also
report results for the ASR system used to build the cascaded ST
system.101010ASR results are different than those reported in Hamed et al.
(2022a) as we limit the data to publicly-available corpora, use different
preprocessing, and different data splits. We report results for both settings:
(1) when training only using ArzEn-ST corpus and (2) when training using
ArzEn-ST corpus in addition to the extra monolingual data specified for each
task (Extra). As expected, adding extra monolingual data greatly improves
results. We observe that translating into Arabic achieves higher BLEU scores
than translating into English. This is expected, as in the case of translating
from CSW text, Arabic words (around 85% of words) remain mostly the same with
possible slight modifications required. We also observe that for the ST
models, the performance is nearly reduced by half compared to the MT results.
This highlights the difficulty of the task. Given that CSW ST has only been
slightly tackled by other researchers, we hope that this corpus will motivate
further research on this task.

##  6 Conclusion and Future Work

Code-switching has become a worldwide prevalent phenomenon. This created a
need for NLP systems to be able to handle such mixed input. Code-switched data
is typically scarce, which is evident in the limited number of available
corpora for machine translation and speech translation tasks. In this paper,
we extend the previously collected ArzEn speech corpus with translations to
both its primary and secondary languages, providing a three-way code-switched
Egyptian Arabic-English speech translation corpus. We have discussed the
translation guidelines, particularly with regards to issues arising due to the
spontaneous nature of the corpus as well as code switching. We reported
benchmark results for baseline ASR, MT, and ST systems. We make this corpus
available to motivate and facilitate further research in this area. For future
work, we plan on improving the corpus and using it for code-switching
linguistic investigations as well as NLP tasks. With regards to corpus
improvements, we plan on adding additional translation references and
CODAfying Habash et al. (2012a); Eskander et al. (2013) the corpus. From a
linguistic perspective, having signals from the monolingual Arabic and English
translations, we plan to further understand why code-switching occurs at the
given points. Finally, we plan to use this corpus for NLP tasks, working on
data augmentation for the purpose of improving machine translation and speech
translation.

## Acknowledgements

This project has benefited from financial support by DAAD (German Academic
Exchange Service). We also thank the reviewers for their insightful comments
and constructive feedback.

## References

  * Ahmed and Tan (2012) Basem HA Ahmed and Tien-Ping Tan. 2012.  Automatic speech recognition of code switching speech using 1-best rescoring.  In _Proceedings of the International Conference on Asian Language Processing_ , pages 137–140.
  * Ali et al. (2016) Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2016.  The mgb-2 challenge: Arabic multi-dialect broadcast media recognition.  In _Proceedings of the Spoken Language Technology Workshop_ , pages 279–284.
  * Ali et al. (2021) Ahmed Ali, Shammur Chowdhury, Amir Hussein, and Yasser Hifny. 2021.  Arabic code-switching speech recognition using monolingual data.  In _Proceedings of Interspeech_.
  * Ali et al. (2017) Ahmed Ali, Stephan Vogel, and Steve Renals. 2017.  Speech recognition challenge in the wild: Arabic mgb-3.  In _Proceedings of the Automatic Speech Recognition and Understanding Workshop_ , pages 316–322.
  * Banerjee et al. (2018) Suman Banerjee, Nikita Moghe, Siddhartha Arora, and Mitesh M Khapra. 2018.  A dataset for building code-mixed goal oriented conversation systems.  In _Proceedings of the International Conference on Computational Linguistics_.
  * Bouamor et al. (2018) Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018.  The MADAR Arabic dialect corpus and lexicon.  In _Proceedings of the International Conference on Language Resources and Evaluation_.
  * Çetinoğlu (2017) Özlem Çetinoğlu. 2017.  A code-switching corpus of turkish-german conversations.  In _Proceedings of the Linguistic Annotation Workshop_ , pages 34–40.
  * Çetinoğlu et al. (2016) Özlem Çetinoğlu, Sarah Schulz, and Ngoc Thang Vu. 2016.  Challenges of computational processing of code-switching.  In _Proceedings of the Workshop on Computational Approaches to Linguistic Code Switching_.
  * Chen et al. (2022) Shuguang Chen, Gustavo Aguilar, Anirudh Srinivasan, Mona Diab, and Thamar Solorio. 2022.  Calcs 2021 shared task: Machine translation for code-switched data.  _arXiv preprint arXiv:2202.09625_.
  * Chen et al. (2017) Song Chen, Dana Fore, Stephanie Strassel, Haejoong Lee, and Jonathan Wright. 2017\.  BOLT Egyptian Arabic sms/chat and transliteration LDC2017T07.  Philadelphia: Linguistic Data Consortium.
  * Chen et al. (2019) Song Chen, Jennifer Tracey, Christopher Walker, and Stephanie Strassel. 2019.  BOLT Egyptian Arabic parallel discussion forums data.  Linguistic Data Consortium (LDC) catalog number LDC2019T01, ISBN 1-58563-871-4.
  * Chowdhury et al. (2021) Shammur Absar Chowdhury, Amir Hussein, Ahmed Abdelali, and Ahmed Ali. 2021.  Towards one model to rule all: Multilingual strategy for dialectal code-switching Arabic asr.  In _Proceedings of Interspeech_.
  * Cieri et al. (2004) Christopher Cieri, David Miller, and Kevin Walker. 2004.  The fisher corpus: A resource for the next generations of speech-to-text.  In _Proceedings of the International Conference on Language Resources and Evaluation_ , pages 69–71.
  * Denisov et al. (2021) Pavel Denisov, Manuel Mager, and Ngoc Thang Vu. 2021.  IMS’systems for the IWSLT 2021 low-resource speech translation task.  _Proceedings of the International Conference on Spoken Language Translation_.
  * Deuchar et al. (2014) Margaret Deuchar, Peredur Davies, Jon Herring, M Carmen Parafita Couto, and Diana Carter. 2014.  Building bilingual corpora.  _Advances in the Study of Bilingualism_ , pages 93–111.
  * Dhar et al. (2018) Mrinal Dhar, Vaibhav Kumar, and Manish Shrivastava. 2018.  Enabling code-mixed translation: Parallel corpus creation and MT augmentation approach.  In _Proceedings of the Workshop on Linguistic Resources for Natural Language Processing_ , pages 131–140.
  * Diab et al. (2013) Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013.  LDC Arabic treebanks and associated corpora: Data divisions manual.  _arXiv preprint arXiv:1309.5652_.
  * Djegdjiga et al. (2018) Amazouz Djegdjiga, Martine Adda-Decker, and Lori Lamel. 2018.  The French-Algerian code-switching triggered audio corpus (FACST).  In _Proceedings of the International Conference on Language Resources and Evaluation_.
  * Eisele et al. (2010) Andreas Eisele, Yu Chen, and UN Multi. 2010.  a multilingual corpus from united nation documents.  In _Proceedings of the International Conference on Language Resources and Evaluation_ , pages 924–929.
  * Eskander et al. (2013) Ramy Eskander, Nizar Habash, Owen Rambow, and Nadi Tomeh. 2013.  Processing spontaneous orthography.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , pages 585–595, Atlanta, Georgia.
  * Gadalla et al. (1997) Hassan Gadalla, Hanaa Kilany, Howaida Arram, Ashraf Yacoub, Alaa El-Habashi, Amr Shalaby, Krisjanis Karins, Everett Rowson, Robert MacIntyre, Paul Kingsbury, David Graff, and Cynthia McLemore. 1997.  Callhome Egyptian Arabic transcripts.  _Linguistic Data Consortium, Philadelphia_.
  * Gaser et al. (2022) Marwa Gaser, Manuel Mager, Injy Hamed, Nizar Habash, Slim Abdennadher, and Ngoc Thang Vu. 2022.  Exploring segmentation approaches for neural machine translation of code-switched Egyptian Arabic-English text.  _arXiv preprint arXiv:2210.06990_.
  * Gupta et al. (2020) Deepak Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2020.  A semi-supervised approach to generate the code-mixed text using pre-trained encoder and transfer learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2267–2280.
  * Guzmán et al. (2019) Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019.  The flores evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
  * Habash et al. (2012a) Nizar Habash, Mona Diab, and Owen Rambow. 2012a.  Conventional orthography for dialectal Arabic.  In _Proceedings of the International Conference on Language Resources and Evaluation_ , pages 711–718.
  * Habash et al. (2012b) Nizar Habash, Ramy Eskander, and Abdelati Hawwari. 2012b.  A morphological analyzer for Egyptian Arabic.  In _Proceedings of the meeting of the special interest group on computational morphology and phonology_ , pages 1–9.
  * Habash et al. (2007) Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007.  On Arabic Transliteration.  In A. van den Bosch and A. Soudi, editors, _Arabic Computational Morphology: Knowledge-based and Empirical Methods_ , pages 15–22. Springer, Netherlands.
  * Hamed et al. (2022a) Injy Hamed, Pavel Denisov, Chia-Yu Li, Mohamed Elmahdy, Slim Abdennadher, and Ngoc Thang Vu. 2022a.  Investigations on speech recognition systems for low-resource dialectal Arabic–English code-switching speech.  _Computer Speech & Language_, 72:101278.
  * Hamed et al. (2018) Injy Hamed, Mohamed Elmahdy, and Slim Abdennadher. 2018.  Collection and analysis of code-switch Egyptian Arabic-English speech corpus.  In _Proceedings of the International Conference on Language Resources and Evaluation_.
  * Hamed et al. (2022b) Injy Hamed, Nizar Habash, Slim Abdennadher, and Ngoc Thang Vu. 2022b.  Investigating lexical replacements for Arabic-English code-switched data augmentation.  _arXiv preprint arXiv:2205.12649_.
  * Hamed et al. (2020) Injy Hamed, Ngoc Thang Vu, and Slim Abdennadher. 2020.  Arzen: A speech corpus for code-switched Egyptian Arabic-English.  In _Proceedings of the International Conference on Language Resources and Evaluation_ , pages 4237–4246.
  * Huber et al. (2022) Christian Huber, Enes Yavuz Ugan, and Alexander Waibel. 2022.  Code-switching without switching: Language agnostic end-to-end speech translation.  _arXiv preprint arXiv:2210.01512_.
  * Hussein et al. (2022) Amir Hussein, Shammur Absar Chowdhury, Ahmed Abdelali, Najim Dehak, and Ahmed Ali. 2022.  Code-switching text augmentation for multilingual speech processing.  _arXiv preprint arXiv:2201.02550_.
  * Ismail (2015) Manal A Ismail. 2015.  The sociolinguistic dimensions of code-switching between Arabic and English by saudis.  _International Journal of English Linguistics_ , 5(5):99.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics Demo and Poster Sessions_ , pages 177–180.
  * Kugathasan and Sumathipala (2021) Archchana Kugathasan and Sagara Sumathipala. 2021.  Neural machine translation for Sinhala-English code-mixed text.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing_ , pages 718–726.
  * Kumar et al. (2014) Gaurav Kumar, Yuan Cao, Ryan Cotterell, Chris Callison-Burch, Daniel Povey, and Sanjeev Khudanpur. 2014.  Translations of the callhome Egyptian Arabic corpus for conversational speech translation.  In _Proceedings of International Workshop on Spoken Language Translation_.
  * LDC (2013) Linguistic Data Consortium LDC. 2013.  BOLT program: Arabic to English translation guidelines.
  * Li and Vu (2019) Chia-Yu Li and Ngoc Thang Vu. 2019.  Integrating knowledge in end-to-end automatic speech recognition for Mandarin-English code-switching.  In _International Conference on Asian Language Processing_ , pages 160–165.
  * Li et al. (2020) Xuansong Li, Stephen Grimes, and Stephanie Strassel. 2020.  BOLT Egyptian Arabic-English word alignment – conversational telephone speech training.  Linguistic Data Consortium (LDC) catalog number LDC2020T05, ISBN 1-58563-920-6.
  * Lyu et al. (2006) Dau-Cheng Lyu, Ren-Yuan Lyu, Yuang-chin Chiang, and Chun-Nan Hsu. 2006.  Speech recognition on code-switching among the Chinese dialects.  In _Proceedings of the International Conference on Acoustics Speech and Signal Processing Proceedings_ , volume 1, pages I–I.
  * Lyu et al. (2015) Dau-Cheng Lyu, Tien-Ping Tan, Eng-Siong Chng, and Haizhou Li. 2015.  Mandarin–English code-switching speech corpus in south-east asia: Seame.  _Language Resources and Evaluation_ , 49(3):581–600.
  * Mahata et al. (2019) Sainik Kumar Mahata, Soumil Mandal, Dipankar Das, and Sivaji Bandyopadhyay. 2019\.  Code-mixed to monolingual translation framework.  In _Proceedings of the Forum for Information Retrieval Evaluation_ , pages 30–35.
  * Menacer et al. (2019) Mohamed Amine Menacer, David Langlois, Denis Jouvet, Dominique Fohr, Odile Mella, and Kamel Smaïli. 2019.  Machine translation on a parallel code-switched corpus.  In _Proceedings of the Canadian Conference on Artificial Intelligence_ , pages 426–432. Springer.
  * Modipa et al. (2013) Thipe I Modipa, Marelie H Davel, and Febe De Wet. 2013.  Implications of Sepedi/English code switching for ASR systems.  In _Proceedings of Pattern Recognition Association of South Africa_.
  * Nakayama et al. (2019) Sahoko Nakayama, Takatomo Kano, Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2019.  Recognition and translation of code-switching speech utterances.  In _Proceedings of the Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques_ , pages 1–6.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  Fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: an ASR corpus based on public domain audio books.  In _Proceedings of the International Conference on Acoustics, Speech and Signal Processing_ , pages 5206–5210.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Park et al. (2019) Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019.  Specaugment: A simple data augmentation method for automatic speech recognition.  In _Proceeding of Interspeech_ , pages 2613–2617.
  * Pasha et al. (2014) Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014.  Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic.  In _Proceedings of the International Conference on Language Resources and Evaluation_ , pages 1094–1101.
  * Poplack (1980) Shana Poplack. 1980.  Sometimes i’ll start a sentence in Spanish y termino en espanol: toward a typology of code-switching.  _Linguistics_ , 18:581–618.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Conference on Machine Translation: Research Papers_ , pages 186–191.
  * Ramanarayanan and Suendermann-Oeft (2017) Vikram Ramanarayanan and David Suendermann-Oeft. 2017.  Jee haan, i’d like both, por favor: Elicitation of a code-switched corpus of Hindi-English and Spanish-English human-machine dialog.  In _Proceedings of Interspeech_ , pages 47–51.
  * Shazal et al. (2020) Ali Shazal, Aiza Usman, and Nizar Habash. 2020.  A unified model for Arabizi detection and transliteration using sequence-to-sequence models.  In _Proceedings of the Arabic Natural Language Processing Workshop_ , pages 167–177.
  * Sinha and Thakur (2005) R Mahesh K Sinha and Anil Thakur. 2005.  Machine translation of bi-lingual Hindi-English (Hinglish) text.  In _Proceedings of the Machine Translation summit (MT Summit X)_ , pages 149–156.
  * Solorio and Liu (2008) Thamar Solorio and Yang Liu. 2008.  Learning to predict code-switching points.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 973–981.
  * Song et al. (2019) Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. 2019.  Code-switching for enhancing NMT with pre-specified translation.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 449–459.
  * Srivastava and Singh (2020) Vivek Srivastava and Mayank Singh. 2020.  Phinc: A parallel Hinglish social media code-mixed corpus for machine translation.  In _Proceedings of the Workshop on Noisy User-generated Text_.
  * Srivastava and Singh (2021) Vivek Srivastava and Mayank Singh. 2021.  Hinge: A dataset for generation and evaluation of code-mixed Hinglish text.  In _Proceedings of the Workshop on Evaluation and Comparison of NLP Systems_.
  * Stefanich et al. (2019) Sara Stefanich, Jennifer Cabrelli, Dustin Hilderman, and John Archibald. 2019.  The morphophonology of intraword codeswitching: Representation and processing.  _Frontiers in Communication_ , page 54.
  * Tarunesh et al. (2021) Ishan Tarunesh, Syamantak Kumar, and Preethi Jyothi. 2021.  From machine translation to code-switching: Generating high-quality code-switched text.  In _Proceedings of The Joint Conference of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing_.
  * van der Westhuizen and Niesler (2016) Ewald van der Westhuizen and Thomas Niesler. 2016.  Automatic speech recognition of English-isiZulu code-switched speech from South African soap operas.  _Procedia Computer Science_ , 81:121–127.
  * Vu et al. (2012) Ngoc Thang Vu, Dau-Cheng Lyu, Jochen Weiner, Dominic Telaar, Tim Schlippe, Fabian Blaicher, Eng-Siong Chng, Tanja Schultz, and Haizhou Li. 2012.  A first speech recognition system for Mandarin-English code-switch conversational speech.  In _Proceedings of the International Conference on Acoustics, Speech and Signal Processing_ , pages 4889–4892. IEEE.
  * Watanabe et al. (2018) Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. 2018.  Espnet: End-to-end speech processing toolkit.  In _Proceeding of Interspeech_ , pages 2207–2207.
  * Weller et al. (2022) Orion Weller, Matthias Sperber, Telmo Pires, Hendra Setiawan, Christian Gollan, Dominic Telaar, and Matthias Paulik. 2022.  End-to-end speech translation for code switched speech.  _Findings of the Association for Computational Linguistics_.
  * Xu and Yvon (2021) Jitao Xu and François Yvon. 2021.  Can you traducir this? machine translation for code-switched input.  In _Proceedings of the Workshop on Computational Approaches to Linguistic Code-Switching_ , pages 84–94.
  * Yang et al. (2020) Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. 2020.  Csp: Code-switching pre-training for neural machine translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , pages 2624–2636.
  * Yilmaz et al. (2016) Emre Yilmaz, Maaike Andringa, Sigrid Kingma, Jelske Dijkstra, F Kuip, H Velde, Frederik Kampstra, Jouke Algra, H Heuvel, and David A van Leeuwen. 2016.  A longitudinal bilingual Frisian-Dutch radio broadcast database designed for code-switching research.  In _Proceedings of the International Conference on Language Resources and Evaluation_.
  * Zbib et al. (2012) Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris CallisonBurch. 2012\.  Machine translation of Arabic dialects.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 49–59.

##  Appendix A MT Hyperparameters

The following is the train command:
python3 fairseq_cli/train.py $DATA_DIR –source-lang src –target-lang tgt –arch
transformer –share-all-embeddings –encoder-layers 5 –decoder-layers 5
–encoder-embed-dim 512 –decoder-embed-dim 512 –encoder-ffn-embed-dim 2048
–decoder-ffn-embed-dim 2048 –encoder-attention-heads 2 –decoder-attention-
heads 2 –encoder-normalize-before –decoder-normalize-before –dropout 0.4
–attention-dropout 0.2 –relu-dropout 0.2 –weight-decay 0.0001 –label-smoothing
0.2 –criterion label_smoothed_cross_entropy –optimizer adam –adam-betas ’(0.9,
0.98)’ –clip-norm 0 –lr-scheduler inverse_sqrt –warmup-updates 4000 –warmup-
init-lr 1e-7 –lr 1e-3 –stop-min-lr 1e-9 –max-tokens 4000 –update-freq 4 –max-
epoch 100 –save-interval 10 –ddp-backend=no_c10d


Fig. 13 shows the difference between the true and predicted state trajectories
(median and 95th percentiles) for each state in
$[x,\cos(\vartheta),\sin(\vartheta),\dot{x},\dot{\vartheta}]$. Since we start
each branched rollout from a state on the real environment, the initial error
is always zero. Across all states, the errors are drastically reduced when
using opc. Fig. 14 shows the predicted trajectories for the cosine of the
pole’s angle, $\cos(\vartheta)$. For mbpo($\star$), we observe that the
trajectories often diverge and attain values that are clearly out of
distribution, i.e., $\cos(\vartheta)>1$.

Figure 13: Difference in state trajectories
$\mathbf{s}_{t}^{\mathrm{true}}-\mathbf{s}_{t}^{\mathrm{pred}}$ from branched
rollouts using a fixed policy of length $H=20$ on the RoboSchool environment
(cf. Fig. 4 (left)) with
$\mathbf{s}=[x,\cos(\vartheta),\sin(\vartheta),\dot{x},\dot{\vartheta}]$. The
solid lines show the median across 100 rollouts and the shaded areas represent
the 95th percentiles. With opc ( ), the simulated rollouts follows the true
state trajectories much closer, whereas with mbpo($\star$) ( ) the prediction
errors quickly accumulate over time. Figure 14:  Trajectories of the second
state ($\cos(\vartheta)$) from 100 branched rollouts using a fixed policy of
length $H=20$ on the RoboSchool environment (cf. Fig. 4 (left)). Both plots
present the same data but the differ in terms of scaling of the ordinate. With
opc ( ), the respective trajectories remain around values close to one, which
corresponds to the upright position of the pendulum. When using the standard
predictive model from mbpo($\star$) ( ), the state trajectories often diverge
and the rollouts are terminated prematurely.

####  D.4 Improvement Bound: Empirical Investigation

In this section, we empirically investigate to what extent opc is able to
tighten the policy improvement bound Eq. 4 compared to pure model-based
approaches. As mentioned in the main paper, the motivation behind opc is to
reduce the on-policy error and we assume that the off-policy error is not
affected too badly by the corrected transitions. Generally speaking, it is
difficult to quantify a-priori how opc compares to a purely model-based
approach as this depends on the generalization capabilities of the learned
dynamics model, the environment itself and the reward signal.

Here, we analyze the CartPole environment (PyBullet implementation) and
estimate the respective error terms that appear in the policy improvement
bound Eq. 4. To this end, we roll out a sequence of policies $\pi_{n}$ on the
true environment (to estimate $\eta_{n}$ and $\eta_{n+1}$) and on the learned
model with and without opc (to estimate $\tilde{\eta}_{n}$ and
$\tilde{\eta}_{n+1}$). The sequence of policies was obtained during a full
policy optimization run (the policy was logged after each update) and we roll
out each policy 100 times on the respective environment/models. The
corresponding learned transition models were similarly logged during a full
policy optimization. For opc, we estimate the off-policy return
$\tilde{\eta}_{n+1}$ using the learned model from iteration $n$ and reference
trajectories from the true environment that were collected under $\pi_{n}$,
but then roll out the model with $\pi_{n+1}$. The results are shown in Fig.
15.

Figure 15:  Empirical evaluation of the error terms in the policy improvement
bound in Eq. 4 for the CartPole environment (PyBullet). We evaluate the
respective terms for a sequence of policies that were obtained during
different iterations $n$ from a full policy optimization run. The respective
returns $\tilde{\eta}_{n+1},\eta_{n},\tilde{\eta}_{n+1},\eta_{n+1}$ are
approximated as the mean from 100 rollouts on the true environment and the
respective models, $\tilde{p}^{\mathrm{opc}}_{n}$ ( ) and
$\tilde{p}^{\mathrm{model}}_{n}$ ( ). For the return on the true environment
$\eta_{n}$ (top), we additionally show the sample distribution of the
rollouts’ returns. This nicely demonstrates how the policy smoothly
transitions from failing consistently ($n\leq 8$) to successfully stabilizing
the pole ($n\geq 12$). Additionally, note that the on-policy model error is
almost always smaller for opc compared to mbpo($\star$), which supports the
theoretical motivation that our method is build upon.

####  D.5 Off-Policy Analysis

In this section, we investigate the robustness of opc towards ‘off-policy-
ness’ of the reference trajectories that are simulated under the data-
generating policy $\pi_{n}$. To this end, we manually increase the
stochasticity of the behavior policy $\pi_{\mathrm{rollout}}$ by a factor
$\beta$ such that

$\displaystyle\mathbb{V}[\pi_{\mathrm{rollout}}(\cdot\mid\mathbf{s})]=\beta^{2}\mathbb{V}[\pi_{n}(\cdot\mid\mathbf{s})],$
(92)

and we keep the mean the same for both policies. Fig. 16 shows the
distributions of the returns from 100 rollouts with varying degree of ‘off-
policy-ness’ on the CartPole environment (PyBullet). Note that the data-
generating policy consistently leads to the maximum return of 1000. For
$\beta$ close to one, both opc and mbpo($\star$) lead to almost ideal behavior
and correctly predict the return in more than 95% of the cases. As we increase
the policy’s stochasticity (from left to right), the respective performance of
the policy decreases until all rollouts terminate prematurely ($\beta\geq
2.3$). Notably, the extent of this effect is almost identical between opc and
mbpo($\star$). We conclude that opc is, at least empirically, robust towards
‘off-policy-ness’ of the reference trajectories. Otherwise, we should observe
a more pronounced degradation of the policy’s performance with increased
stochasticity of the behavior policy, since this leads to observing more off-
policy state/actions.

Figure 16:  Sample distributions of the return on the CartPole environment
(PyBullet) with increasing stochasticity of the behaviour policy
$\pi_{\mathrm{rollout}}$ when rolling out with
$\tilde{p}^{\mathrm{opc}}_{n}$/opc (top) and
$\tilde{p}^{\mathrm{model}}_{n}$/mbpo($\star$) (bottom). The multiplier
$\beta$ quantifies the stochasticity of $\pi_{\mathrm{rollout}}$ relative to
the reference policy $\pi_{n}$ (Eq. 92) such that higher values lead to more
‘off-policy-ness’.

####  D.6 Implementation Changes to Original mbpo

Here, we provide details to the changes we made to the original implementation
of mbpo. We denote our variant as mbpo($\star$).

###### Episodic Setting

mbpo updates the policy during rollouts on the real environment. Algorithms 3
and 4 show the original and our version of mbpo, respectively. While the
original version might be more sample efficient because it allows for more
policy gradient updates based on more recent data, it is not realistic to
update the policy during a rollout on the real environment.

Algorithm 3 Original mbpo algorithm

1:Initialize policy $\pi_{\phi}$, predictive model $p_{\theta}$, environment
dataset $\mathcal{D}_{\mathrm{env}}$, model dataset
$\mathcal{D}_{\mathrm{model}}$

2:for $N$ epochs do

3: Train model $p_{\theta}$ on $\mathcal{D}_{\mathrm{env}}$ via maximum
likelihood

4: for $E$ steps do

5: Take action in environment according to $\pi_{\phi}$; add to
$\mathcal{D}_{\mathrm{env}}$

6: for $M$ model rollouts do

7: Sample $s_{t}$ uniformly from $\mathcal{D}_{\mathrm{env}}$

8: Perform $k$-step model rollout starting from $s_{t}$ using policy
$\pi_{\phi}$; add to $\mathcal{D}_{\mathrm{model}}$

9: for $G$ gradient updates do

10: Update policy parameters on model data:
$\phi\leftarrow\phi-\lambda_{\pi}\hat{\nabla}_{\phi}J_{\pi}(\phi,\mathcal{D}_{\mathrm{model}})$

Algorithm 4 Our version of mbpo algorithm, denoted as mbpo($\star$)

1:Initialize policy $\pi_{\phi}$, predictive model $p_{\theta}$, environment
dataset $\mathcal{D}_{\mathrm{env}}$, model dataset
$\mathcal{D}_{\mathrm{model}}$

2:for $N$ epochs do

3: Train model $p_{\theta}$ on $\mathcal{D}_{\mathrm{env}}$ via maximum
likelihood

4: for $E$ steps do

5: Take action in environment according to $\pi_{\phi}$; add to
$\mathcal{D}_{\mathrm{env}}$

6: for $M$ model rollouts do

7: Sample $s_{t}$ uniformly from $\mathcal{D}_{\mathrm{env}}$

8: Perform $k$-step model rollout starting from $s_{t}$ using policy
$\pi_{\phi}$; add to $\mathcal{D}_{\mathrm{model}}$

9: for $G$ gradient updates do

10: Update policy parameters on model data:
$\phi\leftarrow\phi-\lambda_{\pi}\hat{\nabla}_{\phi}J_{\pi}(\phi,\mathcal{D}_{\mathrm{model}})$

###### Mix-in of Real Data

As mentioned in the main paper, one of the key problems with MBRL is that the
generated data might exhibit so-called model-bias. Biased data can be
problematic for policy optimization as the transition tuples possibly do not
come from the true state distribution of the real environment, thus misleading
the RL algorithm. In the original implementation of mbpo, the authors use a
mix of simulated data from the model as well as observed data from the true
environment for policy optimization
(https://github.com/jannerm/mbpo/blob/22cab517c1be7412ec33fbe5c510e018d5813ebf/mbpo/algorithms/mbpo.py#L430)
to alleviate the issue of model bias. While this design choice might help in
practice, it adds another hyperparameter and the exact influence of this
parameter is difficult to interpret. We therefore refrain from mixing in data
from the true environment, but instead only use simulated transition tuples –
staying true to the spirit of MBRL.

###### Fix Replay Buffer

The replay buffer that stores the simulated data
$\mathcal{D}_{\mathrm{model}}$ for policy optimization is allocated to a fixed
size of $\texttt{rollout length}\times\texttt{rollout batch
size}\times\texttt{retain epochs}$, meaning that per epoch $\texttt{rollout
length}\times\texttt{rollout batch size}$ transition tuples are simulated and
only the data from the last retain epochs are kept in the buffer
(https://github.com/jannerm/mbpo/blob/22cab517c1be7412ec33fbe5c510e018d5813ebf/mbpo/algorithms/mbpo.py#L351).
As the buffer is implemented as a FIFO queue and rollouts might terminate
early such that the actual number of simulated transitions is less than
rollout length, data from older episodes as specified by retain epochs are
retained in the buffer. We assume that this is not the intended behavior and
correct for that in our implementation.

####  D.7 Results for Original mbpo Implementation

The following results were obtained with the original mbpo implementation
without the changes described in Section D.6. Consequently, the results for
opc in this section also do not include these changes.

###### Comparative Evaluation

opc (ours)mbpo $\beta=5\%$mbpo $\beta=0\%$sac

Figure 17: Results based on original implementation Section D.7: Comparison of
opc to the model-based mbpo and model-free sac algorithms. The two mbpo
variants differ in terms of the mix-in ratio $\beta$ of real off-policy
transitions in the training data – a critical hyperparameter. All model-based
approaches outperform sac in terms of convergence for the high-dimensional
tasks. Moreover, on the highly stochastic Hopper environment, opc outperforms
both mbpo variants and does not require additional real off-policy data.

We evaluate the original implementation on three continuous control benchmark
tasks from the MuJoCo control suite (Todorov et al., 2012). The results for
opc, two variants of mbpo and sac are presented in Fig. 3. Both opc and mbpo
use a rollout horizon of $H=10$ to generate the training data. The difference
between the two mbpo variants lies in the mix-in ratio $\beta$ of real off-
policy transitions to the simulated training data. Especially for highly
stochastic environments such as the Hopper, this mix-in ratio is a critical
hyperparameter that requires careful tuning (see also Fig. 18). Fig. 3
indicates that opc is on par with mbpo on both the InvertedPendulum and
HalfCheetah environments that exhibit little stochasticity. On the Hopper
environment, opc outperforms both mbpo variants. Note that the mix-in ratio
for mbpo is critical for successful learning (the original implementation uses
$\beta=5\%$). opc on the other hand, does not require any mixed-in real data.
sac learns slower on the more complex Hopper and HalfCheetah environments, re-
iterating that model-based approaches are significantly more data-efficient
than model-free methods.

###### Large Ablation Study – Hopper

The full study investigates the influence of the following hyperparameters and
design choices:

  * •

Rollout length $H$ and total number of simulated transitions $N$.

  * •

Mix-in ratio of real transitions into the training data $\beta$.

  * •

Deterministic or stochastic rollouts (for mbpo): Current state-of-the-art
methods in MBRL rely on probabilistic dynamic models to capture both aleatoric
and epistemic uncertainty. Accordingly, when rolling out the model, these two
sources of uncertainty are accounted for. However, we show that in terms of
evaluation return, the stochastic rollouts do not always lead to the best
outcome.

  * •

Re-setting the buffer of simulated data after a policy optimization step: We
found that re-setting the replay buffer for simulated data after each
iteration of Algorithm 3 can have a large influence. In particular, the replay
buffer is implemented as a FIFO queue with a fixed size. Hence, if the buffer
is not emptied after each iteration, it still contains (simulated) off-policy
transitions.

Fig. 18 presents the results of the ablation study. We want to highlight a few
core insights:

  1. 1.

When choosing the best setting for each method, opc improves mbpo by a large
margin (bottom row, right). Generally, for long rollouts $H=20$ (bottom row),
opc improves mbpo.

  2. 2.

Across all settings, opc performs well and is more robust with respect to the
choices of hyperparameters (e.g., bottom row left and center, third row left).
Only for few exceptions, re-setting the buffer can have detrimental effects
(e.g., top right, third row right).

  3. 3.

Mixing in real transition data can be highly beneficial for mbpo (second row
left and center) but it can also have the opposite effect (bottom row).

  4. 4.

Using deterministic rollouts can be beneficial (third row right and left),
detrimental (third row center) or have no influence (bottom row left) for
mbpo.

  5. 5.

It is not clear, if re-setting the buffer after each iteration should overall
be recommended or not. This remains an open question left for future research.

opc \+ reset bufferopc \+ not reset buffermbpo \+ reset buffer +
deterministicmbpo \+ not reset buffer + deterministicmbpo \+ reset buffer +
stochasticmbpo \+ not reset buffer + stochastic

Figure 18:  Results based on original implementation Section D.7: Large
ablation study on the Hopper environment, investigating the influence various
hyperparameters and design choices. Mix-in ratio of real data $\beta$
(columns): ${0\%,5\%,10\%}$ from left to right. Rollout length $H$ (rows):
${1,5,10,20}$ from top to bottom.

###  Appendix E Connection between MBRL and ILC

In this section we compare optimization-based or so-called _norm-optimal_ ILC
(NO-ILC) with MBRL. In particular, we show that under certain assumptions we
can reduce the MBRL setting to NO-ILC. This comparison is structured as
follows: First, we review the basic assumptions and notations for NO-ILC.
While there are many variations on NO-ILC, we will only consider the very
basic setting, i.e., linear dynamics, fully observed state and deterministic
state evolution. Then, based on the lifted state representation of the
problem, we derive the solution to the optimization problem that leads to the
input sequence of the next iteration / rollout. Next, we will state the
general MBRL problem and pose the simplifications that we need to make in
order to be equivalent to the NO-ILC problem. Last, we show that the solution
to the reduced MBRL problem is equivalent to the one of NO-ILC.

####  E.1 Norm-optimal ILC

The goal of NO-ILC is to find a sequence of inputs
$\mathbf{a}=[\mathbf{a}_{0}^{\top},\dots,\mathbf{a}_{T-1}^{\top}]^{\top}$ with
length $T+1$ such that the outputs
$\mathbf{y}=[\mathbf{y}_{0}^{\top},\dots,\mathbf{y}_{T}^{\top}]^{\top}$ follow
a desired output trajectory
$\hat{\mathbf{y}}=[\hat{\mathbf{y}}_{0}^{\top},\dots,\hat{\mathbf{y}}_{T}^{\top}]^{\top}$.
In the simplest setting we assume that the system evolves according to the
following linear dynamics

$\displaystyle\mathbf{s}_{t+1}$
$\displaystyle=\mathbf{A}\mathbf{s}_{t}+\mathbf{B}\mathbf{a}_{t}+\mathbf{d}_{t}$
(93) $\displaystyle\mathbf{y}_{t}$ $\displaystyle=\mathbf{C}\mathbf{s}_{t},$
(94)

with state $\mathbf{s}_{t}\in\mathbb{R}^{d_{\mathcal{S}}}$, action
$\mathbf{a}_{t}\in\mathbb{R}^{d_{\mathcal{A}}}$, output
$\mathbf{y}_{t}\in\mathbb{R}^{p}$ and disturbance
$\mathbf{d}_{t}\in\mathbb{R}^{d_{\mathcal{S}}}$. One of the major assumption
in ILC is that the disturbances $\mathbf{d}_{t}$ are _repetitive_ , meaning
that the sequence
$\mathbf{d}=[\mathbf{d}_{0}^{\top},\dots,\mathbf{d}_{T}^{\top}]^{\top}$ does
not change (or only varies slightly) across multiple rollouts. While these
disturbances can be considered to come from some exogenous error source, one
can also interpret these as unmodeled effects of the dynamics, e.g.,
nonlinearities stemming from friction, aerodynamic effects, etc.

For the derivation, let’s assume $\mathbf{C}=\mathbf{I}$ so that we’re
operating on an MDP. Consequently, the goal of ILC is to track a sequence of
desired states $\hat{\mathbf{s}}$ instead of outputs $\hat{\mathbf{y}}$. In
order to find an optimal input sequence, we minimize the squared 2-norm of a
state’s deviation from the reference for each time-step $t$ of the sequence,
i.e., $\mathbf{e}_{t}=\hat{\mathbf{s}}_{t}-\mathbf{s}_{t}$. As is common in
the ILC literature, we make use of the so-called _lifted system_ formulation
such that we can conveniently write the state evolution for one rollout using
a single matrix/vector multiplication. Assuming zero initial state (which can
always be done in the linear setting by just shifting the state by a constant
offset), we obtain the following formulation

$\displaystyle\mathbf{s}=\mathbf{F}\mathbf{a}+\mathbf{d},\quad\text{with}\quad\mathbf{F}=\begin{bmatrix}0&\cdots&&\\\
\mathbf{B}&0&&\\\ \mathbf{A}\mathbf{B}&\mathbf{B}&0&\\\
\mathbf{A}^{2}\mathbf{B}&\mathbf{A}\mathbf{B}&\mathbf{B}&\ddots\\\
\vdots&&\ddots&\ddots\end{bmatrix}\in\mathbb{R}^{{d_{\mathcal{S}}}(T+1)\times{d_{\mathcal{A}}}T}.$
(95)

Thus, we can write the state-error at the $j$-th iteration of ILC in the
lifted representation as (recall that the disturbance $\mathbf{d}$ does not
change across iterations)

$\displaystyle\mathbf{e}^{(i)}$
$\displaystyle=\hat{\mathbf{s}}-\mathbf{s}^{(i)}=\hat{\mathbf{s}}-\mathbf{F}\mathbf{a}^{(i)}-\mathbf{d},$
(96) $\displaystyle\mathbf{e}^{(i+1)}$
$\displaystyle=\mathbf{e}^{(i)}-\mathbf{F}(\mathbf{a}^{(i+1)}-\mathbf{a}^{(i)}).$
(97)

Now, the resulting optimization problem then becomes

$\displaystyle\mathbf{a}_{*}^{(i+1)}=\operatorname*{arg\,min}_{\mathbf{a}^{(i+1)}}J\left(\mathbf{a}^{(i+1)}\right)\text{
with
}J\left(\mathbf{a}^{(i+1)}\right)=\frac{1}{2}\lVert\mathbf{e}^{(i+1)}\rVert^{2}.$

In order to be less sensitive to noise and the inherent stochasticity of real-
world problems, one typically also adds a regularizing term that penalizes the
changes in the input sequence. While this additional penalization term can
slow down the learning process it makes it more robust by avoiding
overcompensation to the disturbances. The full objective then becomes

$\displaystyle
J\left(\mathbf{a}^{(i+1)}\right)=\frac{1}{2}\lVert\mathbf{e}^{(i+1)}\rVert_{\mathbf{M}}^{2}+\frac{1}{2}\lVert\mathbf{a}^{(i+1)}-\mathbf{a}^{(i)}\rVert_{\mathcal{W}}^{2},$
(98)

where we additionally added positive semi-definite cost matrices
$\mathbf{M},\mathcal{W}$ for the respective norms in order to facilitate
tuning of the corresponding terms in the objective. Given the regularized cost
function we obtain the optimal sequence of inputs at the next iteration as

$\displaystyle\mathbf{a}_{*}^{(i+1)}=\mathbf{a}_{*}^{(i)}+\left(\mathbf{F}^{\top}\mathbf{M}\mathbf{F}+\mathcal{W}\right)^{-1}\mathbf{F}^{\top}\mathbf{M}\mathbf{e}^{(i)}.$
(99)

####  E.2 Model-based RL to Norm-optimal ILC

###### Assumptions

In order to show the equivalence of MBRL and NO-ILC we need to make some
assumptions to the above stated optimization problem.

  * •

No aleatoric uncertainty in the model, i.e., no transition noise
$\omega_{t}=0\quad\forall t$.

  * •

Typically in RL we assume the policy to be stationary, however, in this
setting we will allow for non-stationary policies that are indexed by time $t$
(the result will not necessarily depend on the state such that we essentially
just obtain a feedforward control sequence. To include feedback one can always
just combine the feedforward signal with a local controller that tracks the
desired state/action trajectory).

  * •

The reward is given as the negative quadratic error of the state w.r.t. a
desired state trajectory,
$r({\mathbf{s}}_{t},{\mathbf{a}}_{t})=-\frac{1}{2}\lVert\hat{\mathbf{s}}_{t}-{\mathbf{s}}_{t}\rVert^{2}$

  * •

Typically in RL we have constraints on the policy such that it does not change
too much after every iteration, see e.g. TRPO, REPS, etc. While in the
mentioned approaches, the policy are often constrained in terms of their
parameterization vectors, we constrain it as
$\lVert\pi_{t}^{(i+1)}-\pi_{t}^{(i)}\rVert^{2}\leq\epsilon_{\pi}\quad\forall
t$.

  * •

Assume that the model is given by
$\tilde{f}_{t}({\mathbf{s}},{\mathbf{a}})=\mathbf{A}{\mathbf{s}}+\mathbf{B}{\mathbf{a}}+\mathbf{d}_{t}$,
where $\mathbf{A}$ and $\mathbf{B}$ are fixed system matrices and
$\mathbf{d}_{t}$ is a time-dependent offset that we learn.

###### The learned dynamics model

Given state/input pairs of the $i$-th trajectory
$\tau^{(i)}=\\{(\mathbf{s}_{t}^{(i)},\mathbf{a}_{t}^{(i)})\\}_{t=0}^{T}$ from
the true system, we can now improve our dynamics model. In particular, if we
minimize the prediction error over $\tau^{(i)}$, we obtain

$\mathbf{d}_{t}^{(i)}=\mathbf{s}_{t+1}^{(i)}-\left(\mathbf{A}\mathbf{s}_{t}^{(i)}+\mathbf{B}\mathbf{a}_{t}^{(i)}\right)$
(100)

The resulting dynamics for the optimal control problem in Eq. 1 become

$\displaystyle\tilde{f}_{t}^{(i)}({\mathbf{s}},{\mathbf{a}})=\mathbf{s}_{t+1}^{(i)}+\mathbf{A}({\mathbf{s}}-\mathbf{s}_{t}^{(i)})+\mathbf{B}({\mathbf{a}}-\mathbf{a}_{t}^{(i)})$
(101)

which are the error dynamics around the trajectory. Now in the noisy case,
taking the last trajectory is not necessarily the best thing one can do. E.g.,
(Schöllig & D’Andrea, 2009) instead integrate the information of all past
trajectories via Kalman-filtering. In the fully observed case, one way to
think of this is as low-pass filtering $\mathbf{d}_{t}$ in order to account
for the transition noise $\omega$.

###### The resulting MBRL problem

Now, let’s plug in all assumptions into the MBRL problem and have a look at
how to solve it.

$\displaystyle\begin{split}\pi_{*}^{(i+1)}=\operatorname*{arg\,min}_{\pi=\\{\pi_{0},\dots,\pi_{T}\\}}\,&\sum_{t=0}^{T}\frac{1}{2}\lVert\hat{\mathbf{s}}_{t}-{\mathbf{s}}_{t}\rVert^{2}\\\
{\mathbf{s}}_{t+1}&=\mathbf{s}_{t+1}^{(i)}+\mathbf{A}({\mathbf{s}}-\mathbf{s}_{t}^{(i)})+\mathbf{B}({\mathbf{a}}-\mathbf{a}_{t}^{(i)})\\\
{\mathbf{a}}_{t}&=\pi_{t}\\\
\lVert\pi_{t}^{(i)}-\pi_{t}\rVert^{2}&\leq\epsilon_{\pi}\quad\forall
t\end{split}$ (102)

where we have flipped the reward’s sign to transform it into a minimization
problem. In the presented form, this optimization problem has a well-defined
unique solution, however, it is a-priori not clear if the trust-region
constraint is active for some time-steps. To facilitate an analytical solution
to equation 102, we incorporate the constrain on the policy’s stepsize by a
soft-constraint such that

$\displaystyle\begin{split}\pi_{*}^{(i+1)}=\operatorname*{arg\,min}_{\pi=\\{\pi_{0},\dots,\pi_{T}\\}}\,&\sum_{t=0}^{T}\frac{1}{2}\lVert\hat{\mathbf{s}}_{t}-{\mathbf{s}}_{t}\rVert^{2}+\frac{C^{\pi}_{t}}{2}\lVert\pi_{t}^{(i)}-\pi_{t}\rVert^{2}\\\
{\mathbf{s}}_{t+1}&=\mathbf{s}_{t+1}^{(i)}+\mathbf{A}({\mathbf{s}}-\mathbf{s}_{t}^{(i)})+\mathbf{B}({\mathbf{a}}-\mathbf{a}_{t}^{(i)})\\\
{\mathbf{a}}_{t}&=\pi_{t},\end{split}$ (103)

with $C^{\pi}_{t}$ being constants that weight the respective trust-region
terms.

Generally, the MBRL problem cannot be solved analytically due to the (possibly
highly non-linear, non-differentiable, etc.) reward signal and the need for
propagating the (uncertain) dynamics model forwards in time. However, using
the simplifying assumptions above, the reduced MBRL problem equation 103 can
in fact be solved analytically. We can circumvent the equality constraints by
predicting the state trajectory using the error corrected dynamics such that
we obtain a lifted dynamics formulation similar to the analysis for ILC,

$\displaystyle{\mathbf{s}}^{(i)}=\mathbf{F}{\mathbf{a}}^{(i)}+\mathbf{d}^{(i)},\quad\text{with}\quad\mathbf{d}^{(i)}=\mathbf{s}^{(i)}-\mathbf{F}\mathbf{a}^{(i)},$
(104)

with $\mathbf{s},\mathbf{a}$ denoting the stacked states and actions of the
recorded trajectory. Using this notation, the optimization problem reduces to

$\displaystyle\pi_{*}^{(i+1)}=\operatorname*{arg\,min}_{\pi}\,$
$\displaystyle\frac{1}{2}\lVert\hat{\mathbf{s}}-{\mathbf{s}}^{(i)}\rVert^{2}+\frac{1}{2}\lVert\pi^{(i)}-\pi\rVert_{\mathbf{C}^{\pi}}^{2},$
(105)

with $\mathbf{C}^{\pi}=\operatorname{diag}[C^{\pi}_{0},\dots,C^{\pi}_{H}]$. By
inserting equation 104 into equation 105 we obtain the closed-form solution as

$\displaystyle\pi_{*}^{(i+1)}=\left(\mathbf{F}^{\top}\mathbf{F}+\mathbf{C}^{\pi}\right)^{-1}\mathbf{F}^{\top}\left(\hat{\mathbf{s}}-\mathbf{d}^{(i)}\right)=\pi_{*}^{(i)}+\left(\mathbf{F}^{\top}\mathbf{F}+\mathbf{C}^{\pi}\right)^{-1}\mathbf{F}^{\top}\left(\hat{\mathbf{s}}-{\mathbf{s}}^{(i)}\right),$
(106)

which, clearly, is equivalent to equation 99 for $\mathbf{M}=\mathbf{I}$ and
$\mathcal{W}=\mathbf{C}^{\pi}$.

####  E.3 Extensions

In the previous section, we analyzed the most basic setting for NO-ILC, i.e.,
linear dynamics, no transition noise in the dynamics and no state/input
constraints. Some of these assumptions can easily be liftd to genearlized the
presented framework.

###### Nonlinear Dynamics

Instead of the system being defined by fixed matrices $\mathbf{A},\mathbf{B}$,
we can just linearize a non-linear dynamics model $f(\mathbf{s},\mathbf{a})$
around the last state/input trajectory such that

$\displaystyle\mathbf{A}^{(i)}_{t}=\left.\frac{\partial
f}{\partial\mathbf{s}}\right|_{\mathbf{s}_{t}=\mathbf{s}_{t}^{(i)},\mathbf{a}_{t}=\mathbf{a}_{t}^{(i)}},\quad\mathbf{B}^{(i)}_{t}=\left.\frac{\partial
f}{\partial\mathbf{a}}\right|_{\mathbf{s}_{t}=\mathbf{s}_{t}^{(i)},\mathbf{a}_{t}=\mathbf{a}_{t}^{(i)}}$
(107)

and the corresponding dynamics matrix for the lifted representation becomes
(dropping the superscript notation indicating the iteration for clarity)

$\displaystyle\mathbf{F}=\begin{bmatrix}0&\cdots&&\\\ \mathbf{B}_{0}&0&&\\\
\mathbf{A}_{0}\mathbf{B}_{0}&\mathbf{B}_{1}&0&\\\
\mathbf{A}_{0}\mathbf{A}_{1}\mathbf{B}_{0}&\mathbf{A}_{0}\mathbf{B}_{1}&\mathbf{B}_{2}&\ddots\\\
\vdots&&\ddots&\ddots\end{bmatrix}\in\mathbb{R}^{{d_{\mathcal{S}}}(T+1)\times{d_{\mathcal{A}}}T}.$
(108)

###### State/Input Constraints

Recall that we could solve the quadratic problems equation 98 and equation 103
analytically because we assumed no explicit inequality constraints on the
state and inputs. In practice, however, both states and actions are limited by
physical or safety constraints typically given by polytopes. While a closed-
form solution is not readily available for this case, one can easily employ
any numerical solver to deal with such constraints. See, e.g.,
https://en.wikipedia.org/w/index.php?title=Quadratic_programming for a
comprehensive list of available solvers.

###### Stochastic Dynamics

Schöllig & D’Andrea (2009) generalize the ILC setting by considering two
separate sources of noise: 1) transition noise in the dynamics, i.e.,
$\mathbf{s}_{t+1}=f(\mathbf{s}_{t},\mathbf{a}_{t})+\omega^{f}_{t}$ as well as
2) varying disturbances, i.e.,
$\mathbf{d}^{(i+1)}=\mathbf{d}^{(i)}+\omega^{d}$. Based on this model, a
Kalman-Filter in the _iteration-domain_ is developed that estimates the
respective random variables and the ILC scheme is adapted to account for the
estimated quantities.