# Enhancing Translation for Indigenous Languages: Experiments with
Multilingual Models

Atnafu Lambebo Tonja1, Hellina Hailu Nigatu2, Olga Kolesnikova1,
Grigori Sidorov1, Alexander Gelbukh1, Jugal Kalita3
1Instituto Politécnico Nacional (IPN), Mexico,
2University of California, Berkeley, USA,
3University of Colorado, Colorado Springs, USA

###### Abstract

This paper describes CIC NLP’s submission to the AmericasNLP 2023 Shared Task
on machine translation systems for indigenous languages of the Americas. We
present the system descriptions for three methods. We used two multilingual
models, namely M2M-100 and mBART50, and one bilingual (one-to-one) — Helsinki
NLP Spanish-English translation model, and experimented with different
transfer learning setups. We experimented with 11 languages from America and
report the setups we used as well as the results we achieved. Overall, the
mBART setup was able to improve upon the baseline for three out of the eleven
languages.

##  1 Introduction

While machine translation systems have shown commendable performance in recent
years, the performance is lagging for low-resource languages Hadgu et al.
(2022); Tonja et al. (2023). Since low-resource languages suffer from a lack
of sufficient data Siddhant et al. (2022); Haddow et al. (2022), most models
and methods that are developed for high-resource languages do not work well in
low-resource settings. Additionally, low-resource languages are linguistically
diverse and have divergent properties from the mainstream languages in NLP
studies Zheng et al. (2021).

Though low-resource languages lack sufficient data to train large models, some
such languages still have a large number of native speakers Zheng et al.
(2021). While the availability of language technologies such as machine
translation systems can be helpful for such linguistic communities, they could
also bring harm and exposure to exploitation Hovy and Spruit (2016). Borrowing
from human-computer interaction (HCI) studies Schneider et al. (2018), we want
to acknowledge our belief that low-resource language speakers should be
empowered to create technologies that benefit their communities. Many
indigenous communities have community-rooted efforts for preserving their
languages and building language technologies for their communities
111https://papareo.nz/ and we hope that methods from Shared Tasks like this
will contribute to their efforts.

Improving machine translation systems for low-resource languages is an active
research area and different approaches Zoph et al. (2016); Karakanta et al.
(2018); Ortega et al. (2020a); Goyal et al. (2020); Tonja et al. (2022);
Imankulova et al. (2017) have been to improve the performance of systems
geared forward low-resource languages. We participated in the AmericasNLP 2023
Shared Task in hopes of contributing new approaches for low-resource machine
translation that are likely to be helpful for community members interested in
developing and adapting these technologies for their languages.

In recent years, large pre-trained models have been used for downstream NLP
tasks, including machine translation Brants et al. (2007) because of the
higher performance in downstream tasks compared to traditional approaches Han
et al. (2021). One trend is to use these pre-trained models and fine-tune them
on smaller data sets for specific tasks Sun et al. (2019). This method has
shown promising results in downstream NLP tasks for languages with low or
limited resources Tars et al. (2022); Zhao and Zhang (2022). In our
experiments, we used multilingual and bilingual models and employed different
fine-tuning strategies for the eleven languages in the 2023 Shared Task
Ebrahimi et al. (2023).

In this paper, we describe the system setups we used and the results we
obtained from our experiments. One of our systems improves upon the baseline
for three languages. We also reflect on the setups we experimented with but
ended up not submitting in hopes that future work could improve upon them.

##  2 Languages and Datasets

In this section, we present the languages and datasets used in our shared task
submission. Table 1 provides an overview of the languages, their linguistic
families, and the numbers of parallel sentences.

Language | ISO | Family | Train | Dev | Test
---|---|---|---|---|---
Aymara | aym | Aymaran | 6,531 | 996 | 1,003
Bribri | bzd | Chibchan | 7,508 | 996 | 1,003
Asháninka | cni | Arawak | 3,883 | 883 | 1002
Chatino | czn | Zapotecan | 357 | 499 | 1,000
Guarani | gn | Tupi-Guarani | 26,032 | 995 | 1,003
Wixarika | hch | Uto-Aztecan | 8,966 | 994 | 1003
Nahuatl | nah | Uto-Aztecan | 410,000 | 672 | 996
Hñähñu | oto | Oto-Manguean | 4,889 | 599 | 1,001
Quechua | quy | Quechuan | 125,008 | 996 | 1,003
Shipibo-Konibo | shp | Panoan | 14,592 | 996 | 1,002
Rarámuri | tar | Uto-Aztecan | 14721 | 995 | 1002
Table 1: This table provides information about the languages with which we
experimented including ISO language code and language family as well as the
number of sentences in training, development, and test sets for each language.

#### Aymara

is an Aymaran language spoken by the Aymara people of the Bolivian Andes. It
is one of only a handful of Native American languages with over one million
speakers Homola (2012). Aymara, along with Spanish and Quechua, is an official
language in Bolivia and Peru. The data for the Aymara-Spanish come from the
Global Voices Tiedemann (2012).

#### Bribri

The Bribri language is spoken in Southern Costa Rica. Bribri has two major
orthographies: Jara222https://www.lenguabribri.com/se-tt%C3%B6-bribri-ie-
hablemos-en-bribri and Constenla333https://editorial.ucr.ac.cr/index.php and
the writing is not standardized which results in spelling variations across
documents. In this case, the sentences use an intermediate representation to
unify existing orthographies. The Bribri-Spanish data Feldman and Coto-Solano
(2020) came from six different sources.

#### Asháninka

Asháninka is an Arawakan language spoken by the Asháninka people of Peru and
Acre, Brazil444https://www.everyculture.com/wc/Norway-to-Russia/Ash-
ninka.html. It is primarily spoken in the Satipo Province located in the
Amazon forest. The parallel data for Asháninka-Spanish come mainly from three
sources Cushimariano Romano and Sebastián Q. (2008); Ortega et al. (2020b);
Mihas (2011) and translations by Richard Castro.

#### Chatino

Chatino is a group of indigenous Mesoamerican languages. These languages are a
branch of the Zapotecan family within the Oto-Manguean language family. They
are natively spoken by 45,000 Chatino people Cruz and Woodbury (2006) whose
communities are located in the southern portion of the Mexican state of
Oaxaca. The parallel data for Chatino-Spanish can be accessed
here555https://scholarworks.iu.edu/dspace/handle/2022/21028.

#### Guarani

Guarani is a South American language that belongs to the Tupi-Guarani family
Britton (2005) of the Tupian languages. It is one of the official languages of
Paraguay (along with Spanish), where it is spoken by the majority of the
population, and where half of the rural population are monolingual speakers of
the language Mortimer (2006).

#### Wixarika

Wixarika is an indigenous language of Mexico that belongs to the Uto-Aztecan
language family de la Federación (2003). It is spoken by the ethnic group
widely known as the Huichol (self-designation Wixaritari), whose mountainous
territory extends over portions of the Mexican states of Jalisco, San Luis
Potosí, Nayarit, Zacatecas, and Durango, but mostly in Jalisco. United States:
La Habra, California; Houston, Texas.

#### Nahuatl

Nahuatl is a Uto-Aztecan language and was spoken by the Aztec and Toltec
civilizations of Mexico666www.elalliance.org/languages/nahuatl. The Nahuatl
language has no standard orthography and has wide dialectical variations Zheng
et al. (2021).

#### Hñähñu

Hñähñu, also known as Otomí, belongs to the Oto-Pamean family and lived in
central Mexico for many centuries Lastra (2001). Otomí is a tonal language
with a Subject-Verb-Object (SVO) word order Ebrahimi et al. (2022). It is
spoken in several states across Mexico.

#### Quechua

The Quechua-Spanish data Agić and Vulić (2019); Tiedemann (2012) has three
different sources: the Jehova’s Witnesses texts, the Peru Minister of
Education, and dictionary entries and samples collected by Diego Huarcaya. The
Quechua language, also known as Runasimi is spoken in Peru and is the most
widely spoken pre-Columbian language family of the Americas Ebrahimi et al.
(2022).

(a) Spanish-English (Helsinki) model

(b) mBART50 model

(c) m2m100-48inter model

Figure 1: Experiments on (a) fine-tuning bilingual model, (b) and (c) fine-
tuning multilingual models. For (a) we fine-tuned the bilingual Spanish-
English model on Spanish-Indigenous pairs, for (b) we fine-tuned the
multilingual mBART50 model on Spanish-Indigenous pairs, and for (c) we fine-
tuned the multilingual m2m100-48 model first on Spanish-All to produce
m2m100-48inter model and then fine-tuned the m2m100-48inter model on Spanish-
Indigenous pairs.

#### Shipibo-Konibo

Shipibo-Konibo - Spanish data Montoya et al. (2019); Galarreta et al. (2017)
come from three different sources: samples from flashcards translated to
Shipibo-Konibo, sentences translated from books for bilingual education, and
dictionary entries.

#### Rarámuri

Rarámuri, also known as Tarahumara is a Uto-Azetcan language spoken in
Northern Mexico Caballero (2017). Rarámuri is a polysynthetic and
agglutinative language spoken mainly in the Sierra Madre Occidental region of
Mexico Ebrahimi et al. (2022).

Data | Model | aym | bzd | cni | czn | gn | hch | nah | oto | quy | shp | tar | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Baseline | 28.3 | 16.5 | 25.8 | - | 33.6 | 30.4 | 26.6 | 14.7 | 34.3 | 32.9 | 18.4 | -
| M1 | 12.25 | 20.3 | 26.65 | - | 23.83 | 11.09 | 29.55 | 6.57 | 35.04 | 20.99 | 14.12 | 20.03
| M2 | 20.65 | 18.59 | 20.63 | - | 20.40 | 12.7 | 18.66 | 10.17 | 33.53 | 21.03 | 13.54 | 18.99
| M3 | 14.70 | 19.9 | 25.62 | - | 23.62 | 11.82 | 29.94 | 7.94 | 35.3 | 21.32 | 14.19 | 20.43
Dev | M4 | 20.89 | 12.17 | 23.59 | - | 20.84 | 13.51 | 22.63 | 7.16 | 30.86 | 18.02 | 12.60 | 18.22
| M2 | 19.05 | 19.90 | 23.50 | 14.41 | 19.35 | 12.05 | 21.88 | 9.22 | 34.15 | 20.43 | 13.86 | 18.89
| M3 | 18.52 | 21.17 | 25.85 | 15.61 | 21.75 | 13.88 | 26.57 | 7.40 | 35.62 | 21.26 | 14.87 | 20.22
Test | M4 | 18.59 | 13.24 | 23.79 | 13.64 | 20.94 | 14.67 | 22.60 | 7.28 | 32.75 | 18.13 | 12.07 | 17.97
Table 2: chrF2 scores for the three submissions, computed on the development
and test sets. M1, M2, M3, and M4 represent M2M100-48, M2M100-48inter, mBART50
and Helsinki-NLP models respectively. The development set evaluations are used
to select the best-performing model before working on submission data. The
development set was not trained when evaluating the dev set, but we included
the dev set during training for the final submission. The bold results show
the models that out-preforms the baseline Vázquez et al. (2021) results. The
bold results show out-preforming models from our three model setups(excluding
the baseline) for each individual language.

##  3 Models

We experimented with two multilingual and one bilingual translation model with
different transfer learning setups. We used M2M-100 and mBART50 for the
multilingual experiment and the Helsinki-NLP Spanish-English model for the
bilingual experiment. Figure 1 shows the models used in this experiment.

###  3.1 Bilingual models

For the bilingual model, as shown in Figure 1(a), we use a publicly available
Spanish - English777https://huggingface.co/Helsinki-NLP/opus-mt-es-en pre-
trained model from Huggingface888https://huggingface.co/ trained by Helsinki-
NLP. The pre-trained MT models released by Helsinki-NLP are trained on OPUS,
an open-source parallel corpus for covering 500 languages Tiedemann and
Thottingal (2020); Tiedemann (2020). This model is trained using the framework
of Marian NMT Junczys-Dowmunt et al. (2018). Each model has six self-attention
layers in the encoder and decoder parts, and each layer has eight attention
heads.

We used this model with the intention that the model trained with high-
resource languages will improve the translation performance of low-resource
indigenous languages when using a model trained with high-resource languages.
We fine-tuned the Spanish-English model for each of the Spanish-to-Indigenous
language pairs.

###  3.2 Multilingual models

For multilingual models, we used the Many-to-Many multilingual translation
model that can translate directly between any pair of 100 languages (M2M100)
Fan et al. (2021) with 48M parameters and a sequence-to-sequence denoising
auto-encoder pre-trained on large-scale monolingual corpora in 50 languages
(mBART50) Tang et al. (2020). We fine-tuned multilingual models in two ways:

  1. 1.

We fine-tuned two multilingual models on each Spanish-Indigenous language pair
for 5 epochs and evaluated their performance using the development data before
training the final submission system. As shown in Figure 1(b), for the final
system, we only fine-tuned mBART50 on Spanish-indigenous data based on the
development set evaluation performance.

  2. 2.

Fine-tuning multilingual models first on the Spanish - All (mixture of all
indigenous language data) dataset to produce an intermediate model and then
fine-tuning the intermediate model for each of the Spanish-Indigenous language
pairs as shown in Figure 1(c). For this experiment, we combined all language
pairs’ training data to form a Spanish - all parallel corpus, and then we
first fine-tuned m2m100-48 using a combined dataset for five epochs and saved
the model, here referred to as m2m100-48inter model. We fine-tuned the
m2m100-48inter model again on each Spanish-Indigenous language pair for
another 5 epochs and evaluated the performance on the development set before
training the final submission system.

Evaluation We used chrF2 Popović (2017) evaluation metric to evaluate our MT
systems.

##  4 Results

We submitted three (two multilingual and one bilingual) systems, as shown in
Table 2, namely m2m100-48inter, mBART50, and Helsinki-NLP. We included the dev
set performance for all the models we trained before the final model to
compare the results with the final model evaluated by using test set data.
From the dev set result, it can be seen that fine-tuning the multilingual
model on the Spanish-Indigenous language pair outperforms the fine-tuned
result of the bilingual and m2m100-48inter models. From all the models
evaluated using the dev set, mBART50 outperformed the others on average.

Our test results show comparable results when compared to the strongest
baseline shared by the AmericasNLP 2023, and our model outperformed the
baseline for Spanish-Bribri (es-bzd), Spanish-Asháninka (es-cni), and Spanish-
Quechua (es-quy) pairs. Similarly, mBART50 outperformed the other models on
average on the test set.

##  5 Conclusion

In this work, we present the system descriptions and results for our
submission to the 2023 AmericasNLP Shared Task on Machine Translation into
Indigenous Languages. We used pre-trained models and tested different fine-
tuning strategies for the eleven languages provided for the shared task. We
used one bilingual (Helsinki NLP English-Spanish model) and two multilingual
(M2M-100 and mBART50) models for our experiments. In addition to fine-tuning
the individual languages’ data, we concatenated the data from all eleven
languages to create a Spanish-All dataset and fine-tuned the M2M-100 model
before fine-tuning for the individual languages. Our mBAERT50 model beat the
strong baseline in three languages.

## Acknowledgements

## References

  * Agić and Vulić (2019) Željko Agić and Ivan Vulić. 2019.  JW300: A wide-coverage parallel corpus for low-resource languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3204–3210, Florence, Italy. Association for Computational Linguistics.
  * Brants et al. (2007) Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007.  Large language models in machine translation.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 858–867, Prague, Czech Republic. Association for Computational Linguistics.
  * Britton (2005) A Scott Britton. 2005.  Guaraní-english/english-guaraní: Concise dictionary.  _New York: Hippocrene_.
  * Caballero (2017) Gabriela Caballero. 2017.  Choguita rarámuri (tarahumara) language description and documentation: a guide to the deposited collection and associated materials. language documentation and conservation.
  * Cruz and Woodbury (2006) Emiliana Cruz and Anthony C Woodbury. 2006.  The sandhi of tones in the chatino de quiahije.  In _Memoirs of the Congress of Indigenous Languages of Latin Americaírica-II_.
  * Cushimariano Romano and Sebastián Q. (2008) Rubén Cushimariano Romano and Richer C. Sebastián Q. 2008.  Ñaantsipeta asháninkaki birakochaki. diccionario asháninka-castellano. versión preliminar.  http://www.lengamer.org/publicaciones/diccionarios/.
  * de la Federación (2003) Diario Oficial de la Federación. 2003.  Ley general de derechos lingüísticos de los pueblos indígenas.  _Dirección General de Bibliotecas_.
  * Ebrahimi et al. (2022) Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Meza-Ruiz, Gustavo A. Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Ngoc Thang Vu, and Katharina Kann. 2022.  Americasnli: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages.
  * Ebrahimi et al. (2023) Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Enora Rice, Cynthia Montaño, John Ortega, Shruti Rijhwani, Alexis Palmer, Rolando Coto-Solano, Hilaria Cruz, and Katharina Kann. 2023.  Findings of the AmericasNLP 2023 shared task on machine translation into indigenous languages.  In _Proceedings of the Third Workshop on Natural Language Processing for Indigenous Languages of the Americas_. Association for Computational Linguistics.
  * Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021.  Beyond english-centric multilingual machine translation.  _The Journal of Machine Learning Research_ , 22(1):4839–4886.
  * Feldman and Coto-Solano (2020) Isaac Feldman and Rolando Coto-Solano. 2020.  Neural machine translation models with back-translation for the extremely low-resource indigenous language bribri.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3965–3976.
  * Galarreta et al. (2017) Ana-Paula Galarreta, Andrés Melgar, and Arturo Oncevay. 2017.  Corpus creation and initial SMT experiments between Spanish and Shipibo-konibo.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 238–244, Varna, Bulgaria. INCOMA Ltd.
  * Goyal et al. (2020) Vikrant Goyal, Sourav Kumar, and Dipti Misra Sharma. 2020.  Efficient neural machine translation for low-resource languages via exploiting related languages.  In _Proceedings of the 58th annual meeting of the association for computational linguistics: student research workshop_ , pages 162–168.
  * Haddow et al. (2022) Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindřich Helcl, and Alexandra Birch. 2022.  Survey of Low-Resource Machine Translation.  _Computational Linguistics_ , 48(3):673–732.
  * Hadgu et al. (2022) Asmelash Teka Hadgu, Abel Aregawi, and Adam Beaudoin. 2022.  Lesan – machine translation for low resource languages.  In _Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track_ , volume 176 of _Proceedings of Machine Learning Research_ , pages 297–301. PMLR.
  * Han et al. (2021) Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. 2021.  Pre-trained models: Past, present and future.  _AI Open_ , 2:225–250.
  * Homola (2012) Petr Homola. 2012.  Building a formal grammar for a polysynthetic language.  In _Formal Grammar: 15th and 16th International Conferences, FG 2010, Copenhagen, Denmark, August 2010, FG 2011, Ljubljana, Slovenia, August 2011, Revised Selected Papers_ , pages 228–242. Springer.
  * Hovy and Spruit (2016) Dirk Hovy and Shannon L. Spruit. 2016.  The social impact of natural language processing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 591–598, Berlin, Germany. Association for Computational Linguistics.
  * Imankulova et al. (2017) Aizhan Imankulova, Takayuki Sato, and Mamoru Komachi. 2017.  Improving low-resource neural machine translation with filtered pseudo-parallel corpus.  In _Proceedings of the 4th Workshop on Asian Translation (WAT2017)_ , pages 70–78.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Karakanta et al. (2018) Alina Karakanta, Jon Dehdari, and Josef van Genabith. 2018.  Neural machine translation for low-resource languages without parallel corpora.  _Machine Translation_ , 32:167–189.
  * Lastra (2001) Y. Lastra. 2001.  _Chapter 6\. Otomí Language Shift and Some Recent Efforts to Reverse It_, pages 142–165. Multilingual Matters, Bristol, Blue Ridge Summit.
  * Mihas (2011) Elena Mihas. 2011.  _Añaani katonkosatzi parenini, El idioma del alto Perené_.  Milwaukee, WI: Clarks Graphics.
  * Montoya et al. (2019) Héctor Erasmo Gómez Montoya, Kervy Dante Rivas Rojas, and Arturo Oncevay. 2019.  A continuous improvement framework of machine translation for Shipibo-konibo.  In _Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages_ , pages 17–23, Dublin, Ireland. European Association for Machine Translation.
  * Mortimer (2006) Katherine Mortimer. 2006.  Guaraní académico or jopará? educator perspectives and ideological debate in paraguayan bilingual education.  _Working Papers in Educational Linguistics (WPEL)_ , 21(2):3.
  * Ortega et al. (2020a) John E Ortega, Richard Castro Mamani, and Kyunghyun Cho. 2020a.  Neural machine translation with a polysynthetic low resource language.  _Machine Translation_ , 34(4):325–346.
  * Ortega et al. (2020b) John E Ortega, Richard Alexander Castro-Mamani, and Jaime Rafael Montoya Samame. 2020b.  Overcoming resistance: The normalization of an Amazonian tribal language.  In _Proceedings of the 3rd Workshop on Technologies for MT of Low Resource Languages_ , pages 1–13, Suzhou, China. Association for Computational Linguistics.
  * Popović (2017) Maja Popović. 2017.  chrf++: words helping character n-grams.  In _Proceedings of the second conference on machine translation_ , pages 612–618.
  * Schneider et al. (2018) Hanna Schneider, Malin Eiband, Daniel Ullrich, and Andreas Butz. 2018.  Empowerment in hci - a survey and framework.  CHI ’18, page 1–14, New York, NY, USA. Association for Computing Machinery.
  * Siddhant et al. (2022) Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Xu Chen, Isaac Caswell, and Xavier Garcia. 2022.  Towards the next 1000 languages in multilingual machine translation: Exploring the synergy between supervised and self-supervised learning.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019.  How to fine-tune bert for text classification?  In _Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18_ , pages 194–206. Springer.
  * Tang et al. (2020) Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020.  Multilingual translation with extensible multilingual pretraining and finetuning.  _arXiv preprint arXiv:2008.00401_.
  * Tars et al. (2022) Maali Tars, Taido Purason, and Andre Tättar. 2022.  Teaching unseen low-resource languages to large translation models.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 375–380.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Tiedemann (2020) Jörg Tiedemann. 2020.  The tatoeba translation challenge – realistic data sets for low resource and multilingual MT.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 1174–1182, Online. Association for Computational Linguistics.
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  OPUS-MT – building open translation services for the world.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 479–480, Lisboa, Portugal. European Association for Machine Translation.
  * Tonja et al. (2022) Atnafu Lambebo Tonja, Olga Kolesnikova, Muhammad Arif, Alexander Gelbukh, and Grigori Sidorov. 2022.  Improving neural machine translation for low resource languages using mixed training: The case of ethiopian languages.  In _Advances in Computational Intelligence: 21st Mexican International Conference on Artificial Intelligence, MICAI 2022, Monterrey, Mexico, October 24–29, 2022, Proceedings, Part II_ , pages 30–40. Springer.
  * Tonja et al. (2023) Atnafu Lambebo Tonja, Olga Kolesnikova, Alexander Gelbukh, and Grigori Sidorov. 2023\.  Low-resource neural machine translation improvement using source-side monolingual data.  _Applied Sciences_ , 13(2).
  * Vázquez et al. (2021) Raúl Vázquez, Yves Scherrer, Sami Virpioja, and Jörg Tiedemann. 2021\.  The helsinki submission to the americasnlp shared task.  In _Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas_. The Association for Computational Linguistics.
  * Zhao and Zhang (2022) Jing Zhao and Wei-Qiang Zhang. 2022.  Improving automatic speech recognition performance for low-resource languages with self-supervised models.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1227–1241.
  * Zheng et al. (2021) Francis Zheng, Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. 2021.  Low-resource machine translation using cross-lingual language model pretraining.  In _Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas_ , pages 234–240.
  * Zoph et al. (2016) Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016.  Transfer learning for low-resource neural machine translation.  _arXiv preprint arXiv:1604.02201_.


# Mitigating Negative Side Effects in Multi-Agent Systems Using Blame
Assignment

Pulkit Rustagi &Sandhya Saisubramanian
Oregon State University
{rustagip<EMAIL_ADDRESS>
###### Abstract

When agents that are independently trained (or designed) to complete their
individual tasks are deployed in a shared environment, their joint actions may
produce negative side effects (NSEs). As their training does not account for
the behavior of other agents or their joint action effects on the environment,
the agents have no prior knowledge of the NSEs of their actions. We model the
problem of mitigating NSEs in a cooperative multi-agent system as a
Lexicographic Decentralized Markov Decision Process with two objectives. The
agents must optimize the completion of their assigned tasks while mitigating
NSEs. We assume independence of transitions and rewards with respect to the
agents’ tasks but the joint NSE penalty creates a form of dependence in this
setting. To improve scalability, the joint NSE penalty is decomposed into
individual penalties for each agent using credit assignment, which facilitates
decentralized policy computation. Our results in simulation on three domains
demonstrate the effectiveness and scalability of our approach in mitigating
NSEs by updating the policies of a subset of agents in the system.

##  1 Introduction

Many real-world settings involve the operation of multiple autonomous agents
that do not require coordination for the completion of their individual
assigned tasks in an environment, such as robots with distinct areas of
operation in factory warehouses [?]. Such agents are often trained in
isolation and their training does not account for potential negative
interactions with other agents or their cumulative effects on the environment.
Consequently, the agents may act in ways that are collectively undesirable
when deployed, such as producing _negative side effects_ (NSEs) [?; ?]. This
paper focuses on mitigating NSEs in cooperative multi-agent settings where the
agents produce no (or negligible) NSEs when executing their policy in
isolation, but their joint policy results in NSEs.

Consider warehouse robots, operating alongside humans, that optimize moving
shelves between two locations. Each robot’s model provides the necessary
information, including reward and transition dynamics, to complete its task
optimally. The models may lack information about the effects of agents’ joint
actions in the environment, such as a narrow corridor being blocked for human
access when multiple robots simultaneously move large shelves through it.
Thus, even when the agents are adept at completing their tasks and produce no
NSEs when acting in isolation, their joint actions may have undesirable
effects.

Figure 1: Overview of our solution approach. The agents independently compute
optimal policies to complete their assigned tasks described by $R_{1}^{i}$
(Naive policy). The NSE Monitor computes the NSE penalty associated with the
joint policy $\vec{\pi}$. The Blame Resolver then performs blame assignment to
decompose the penalty in each joint state-action pair into penalties
associated with each agent’s local state-action pair, which yields
$R^{i}_{N}$. The agents finally recompute their policies by solving the bi-
objective problem using _lexicographic value iteration_ (LVI) with
$R_{1}^{i}\succ R_{N}^{i}$.

Mitigating NSEs in multi-agent settings is challenging because: (1) NSEs are
often discovered after deployment, since they depend on the environment, agent
interactions and their assigned tasks; (2) NSEs and corresponding penalties
are defined over joint actions, introducing a dependency between agents; and
(3) the computational complexity of mitigating NSEs, without significantly
affecting task completion, increases with the number of agents in the system.
Prior research on NSEs primarily target single-agent settings [?; ?; ?; ?]. It
is not straightforward to apply them to multi-agent settings as they do not
account for the agent interactions that produce NSEs. Recently, [?] proposed
an approach to avoid NSEs by optimizing the future returns by other agents in
the environment. However, it cannot avoid NSEs that require policy update for
multiple agents and do not fully scale to large settings.

We formulate the problem of mitigating NSEs in a multi-agent system as a
decentralized Markov decision process (DEC-MDP) with _two objectives_ and a
_lexicographic ordering_ over them. The primary objective for each agent is to
optimize its assigned task and the secondary objective is to minimize the
NSEs. We assume independence of transition and rewards with respect to the
agents’ assigned tasks [?]. Each agent’s reward for task completion is
determined by its local state and actions and the overall reward for the
system is the sum of individual agent’s rewards, and agents’ state transitions
are affected only by their own actions. We use a lexicographic formulation
since it is more intuitive and many real-world problems have an inherent
ordering over objectives. It has been also been shown that lexicographic MDPs
can be converted to a constrained MDP [?].

The agents do not have prior knowledge about NSEs but incur a _joint penalty_
for them when executing certain actions together. Thus, while agents can
independently compute policies to complete their tasks, mitigating NSEs
requires addressing a form of dependence created by the joint penalty.

We present a _metareasoning_ [?; ?] approach to detect and mitigate NSEs. The
metareasoner is a centralized entity that monitors the agents’ behaviors and
has two components: (1) NSE Monitor that estimates the NSEs associated with
agents’ joint policy, and outputs the corresponding penalty, and (2) a Blame
Resolver that decomposes the joint penalty into individual agent penalties,
using our algorithm Reward Estimation using Counterfactual Neighbors (RECON).
It is assumed that the NSE monitor has access to a model of NSEs and the
associated penalty, either provided as part of its design or acquired through
human feedback. The blame resolver, using RECON, performs blame (credit)
assignment to determine each agent’s relative contribution towards the NSE,
based on which the joint NSE penalty is decomposed into local penalties for
each agent [?; ?; ?]. This decomposition facilitates updating the model of
each agent with NSE information via a penalty function, thereby enabling
decentralized policy computation to mitigate NSEs.

Our solution framework uses a four-step approach to mitigate NSEs (Figure 1):
(1) the agents first calculate optimal policies to complete their assigned
tasks (referred to as naive policies); (2) the NSE monitor estimates the NSE
penalty associated with the joint policy of the agents; (3) the blame resolver
then decomposes the joint penalty into individual penalties for each agent,
using blame (credit) assignment; and (4) the agents recompute their policies
by solving a decentralized, bi-objective problem, with the prescribed reward
for their task and the estimated local penalty for NSE, using lexicographic
value iteration [?].

Our primary contributions are: (1) formalizing the problem of mitigating NSE
in cooperative multi-agent settings as a decentralized, bi-objective problem;
(2) presenting a metareasoning approach to update agent models with a penalty
function corresponding to NSEs; (3) introducing an algorithm to decompose
joint NSE penalty into individual penalties, using counterfactual-based credit
assignment; and (4) empirical evaluation on three domains in simulation.

Credit Assignment Technique |  Scalable | Supports decentralized | Compatible with | Can generate NSE-
---|---|---|---|---
planning | heterogeneous agents | specific counterfactuals
Shapley Value |  ✗ |  ✓ |  ✓ |  ✗
Difference Reward |  ✓ |  ✓ |  ✓ | $-$
Action Not Taken |  ✓ |  ✓ |  ✓ |  ✗
D++ |  ✗ |  ✓ |  ✓ |  ✗
Wonderful Life Utility |  ✗ |  ✗ |  ✗ |  ✗
Aristrocratic Utility |  ✗ |  ✗ |  ✗ |  ✗
Value Decomposition |  ✗ |  ✗ |  ✓ |  ✗
Our Approach |  ✓ |  ✓ |  ✓ |  ✓
Table 1: Overview of different credit assignment techniques for mitigating
side effects. A technique is scalable if it is computationally inexpensive to
generate counterfactuals for problems with a few hundreds of agents. “$-$”
indicates that the approach cannot be applied directly but can be modified to
meet the requirements of our setting.

##  2 Related Works

### Negative Side Effects

Prior works have considered different definitions of NSEs such as hindering
the completion of future tasks or the operation of other agents [?; ?; ?; ?],
and model incompleteness in the form of missing information or incomplete
reward function [?; ?; ?; ?]. Our setting is similar to the latter works in
that the agents have incomplete information about the cumulative effects of
their joint actions. A common approach in such settings is to modify the
existing reward function or to consider a secondary reward function associated
with NSEs [?; ?; ?; ?] and solve a multi-objective problem. The information
about NSEs may be gathered using human feedback [?; ?] . However, these
approaches have been presented for a single agent setting or consider other
agents as part of the environment, and it is not straightforward to extend
them to multi-agent settings where multiple agent policies must be updated.
Recently, a distributed coordination approach has been proposed for mitigating
NSEs in multi-agent settings [?]. They target NSEs arising due to local agent
interactions and formulate it as a distributed constraint optimization problem
and the agents learn a policy to avoid NSEs using coordinated Q-learning.
Their approach is not scalable as distributed constraint optimization can
become intractable for large problems [?]. We present a metareasoning approach
that scales to large multi-agent systems.

### Credit Assignment.

Credit assignment is a popular approach to measure the contribution of an
agent to team performance, which is used to convert a joint reward into
individual agent rewards [?]. Difference Reward [?] and its variants such as
D++ [?], Wonderful Life Utility and Aristrocratic Utility [?], perform credit
assignment by comparing the joint rewards before and after the agent is
removed from the system. Another type of credit assignment uses Shapley value
and involves computing a value function for each agent by considering all
combinations of possible agent interactions [?; ?; ?; ?]. Table 1 summarizes
the characteristics of different credit assignment techniques. Blame
assignment for NSEs must be based on counterfactuals generated by considering
NSE-relevant features only. The existing methods calculate counterfactuals by
considering all state features which leads to incorrect blame assignment for
NSEs.

##  3 Preliminaries

### Decentralized Markov Decision Process (Dec-MDP).

A Dec-MDP is widely used to model decentralized multi-agent decision-making
problems [?]. A Dec-MDP is defined by the tuple
$\langle\mathcal{A},S,A,T,R\rangle$ with $\mathcal{A}$ denoting the finite set
of $k$ agents in the system;
$S\\!=\\!\hat{S}_{1}\\!\times...\times\\!\hat{S}_{k}$ denoting the joint state
space, where $\hat{S}_{i}$ denotes the state space of agent $i$;
$A=\hat{A}_{1}\\!\times\ldots\\!\times\\!\hat{A}_{k}$ denoting the joint
action space, where $\hat{A}_{i}$ denotes agent $i$’s actions; $T:S\times
A\times S\rightarrow[0,1]$ denoting the transition function; and $R$ denoting
the reward function. A joint policy $\vec{\pi}\\!=\\!(\pi_{1},...,\pi_{k})$ is
a set of policies, one for each agent in the system.

Transition and reward independence.  A Dec-MDP with _transition independence
and reward independence_ [?] is a class of problems in which agents operate
independently but are tied together through a reward structure that depends on
all of their execution histories. A transition and reward-independent MDP
satisfies the following:

$T(\hat{s}_{i}^{\prime}|{s},{a},{s}^{\prime}_{-i})=T_{i}(\hat{s}_{i}^{\prime}|\hat{s}_{i},\hat{a}_{i}),\forall
i\in\mathcal{A}\vspace{-3pt}$
$R({s},{a},{s}^{\prime})=\sum_{i\in\mathcal{A}}R_{i}(\hat{s}_{i},\hat{a}_{i},\hat{s}_{i}^{\prime}).$

A transition and reward-independent Dec-MDP can be solved as $k$ single agent
MDPs [?].

### Lexicographic MDP (LMDP).

LMDPs are particularly convenient to model problems with potentially competing
objectives with an inherent lexicographic ordering over them, such as ours
where task completion is prioritized over NSE mitigation [?; ?]. An LMDP is
denoted by the tuple $M=\langle S,A,T,\boldsymbol{\mathrm{R}},o\rangle$ with
finite set of states $S$, finite set of actions $A$, transition function
denoted by $T\\!:\\!S\\!\times\\!A\times\\!S\\!\rightarrow\\![0,1]$ and a
vector of reward functions
$\boldsymbol{\mathrm{R}}\\!=\\![R_{1},...,R_{k}]^{T}$ with
$R_{i}\\!:\\!S\\!\times\\!A\\!\rightarrow\\!\mathbb{R}$, and $o$ denotes the
strict preference ordering over the $k$ objectives. The set of value functions
is denoted by $\boldsymbol{V}\\!=\\![V_{1},...,V_{k}]^{T}$, with $V_{i}$
corresponding to $o_{i}$,

$\boldsymbol{V}^{\pi}(s)\\!=\\!\boldsymbol{\mathrm{R}}(s,\pi(s))+\gamma\sum_{s^{\prime}\in
S}T(s,\pi(s),s^{\prime})\boldsymbol{V}^{\pi}(s^{\prime}),\forall s\in S.$

A slack $\boldsymbol{\Delta}\\!=\\!\langle\delta_{1},...,\delta_{k}\rangle$
with $\delta_{i}\\!\geq\\!0$, denotes the acceptable deviation from the
optimal expected reward for objective $o_{i}$ so as to improve the lower
priority objectives. Objectives are processed in the lexicographic order. The
set of restricted actions for $o_{i+1}$ is
$A_{i+1}(s)\\!=\\!\\{a\\!\in\\!A|\max\limits_{a^{\prime}\in
A_{i}}Q_{i}(s,a^{\prime})-Q_{i}(s,a)\leq\eta_{i}$ where
$\eta_{i}\\!=\\!(1-\gamma)\delta_{i}$,$\gamma\in[0,1)$. Refer [?] for a
detailed background on LMDP.

##  4 Problem Formulation

Problem Setting.  Consider a setting where $m$ agents are independently
performing their assigned tasks which is their primary objective
$o_{1}\\!=\\!\\{o_{1}^{1},...,o_{1}^{m}\\}$. The agents operate based on a
transition and reward independent decentralized Markov decision process (Dec-
MDP), denoted by $M$ [?]. A meta-level process, _metareasoner_ , monitors and
controls the agents performance (object-level process), by intervening if
agent actions have undesirable consequences [?; ?].

While the model $M$ contains all the information necessary to optimize
$o_{1}$, it may not describe the joint effects of agents’ actions, since it is
irrelevant to task completion in this setting. Due to the limited fidelity of
$M$, _negative side effects_ (NSEs) occur when agents execute their policies
jointly. The agents incur a penalty for NSEs, determined by a function $R_{N}$
which is known to the metareasoner (either by design or learned using human
feedback) but unknown to the agents.

We make the following assumptions about the NSEs: (1) there are no (or
negligible) NSEs when agents execute their policy in isolation but their joint
policy produces NSEs that must be mitigated to the extent possible; (2) agents
have no prior knowledge about NSEs of the joint actions, besides receiving a
penalty assigned by the metareasoner; and (3) NSEs do not interfere with the
agents’ task completion—they are undesirable but are not catastrophic.We
target settings where the completion of the agents’ assigned tasks ($o_{1}$)
are _prioritized_ over minimizing NSEs ($o_{2}$), $o_{1}\\!\succ\\!o_{2}$.

MASE-MDP.  The problem of mitigating NSEs in a cooperative multi-agent system
is formulated as a Dec-MDP with two objectives and a lexicographic ordering
over them.

######  Definition 1.

A multi-agent side effects MDP (MASE-MDP) is a bi-objective Dec-MDP with
lexicographic ordering over the objectives, denoted by
$M=\langle\mathcal{A},S,A,T,\bm{R},o\rangle$, where:

  * •

$\mathcal{A}=\\{1,\ldots,m\\}$ is a finite set of agents in the system;

  * •

$S=\hat{S}_{1}\times\ldots\times\hat{S}_{m}$ is the joint state space;

  * •

$A=\hat{A}_{1}\times\ldots\times\hat{A}_{m}$ is the joint action space;

  * •

$T:S\times A\times S\rightarrow[0,1]$ is the transition function;

  * •

$\bm{R}\\!=\\![R_{1},R_{N}]$ is the reward function with
$R_{1}\\!:\\!S\\!\times\\!A\\!\rightarrow\\!\mathbb{R}$ denoting the reward
for task performance and
$R_{N}\\!:\\!S\\!\times\\!A\\!\rightarrow\\!\mathbb{R}$ denoting the penalty
function for NSEs; and

  * •

$o=[o_{1},o_{2}]$ denotes the objectives, where $o_{1}$ is the primary
objective denoting agents’ assigned tasks and $o_{2}$ is minimizing NSEs, with
$o_{1}\succ o_{2}$.

We consider independence of transition function and task completion reward
$R_{1}$ in the MASE-MDP, meaning each agent’s transitions and task reward
depend only on its local state and action. Hence, each agent can independently
compute a policy for its assigned task by solving an MDPs [?; ?]. However, the
agents incur a _joint penalty_ for the NSEs, which introduces a form of
dependence and prevents decentralized computation of individual agent
policies. In Section 5, we present an approach to decompose joint penalties
into individual penalties, thereby facilitating decentralized planning.

Local and Global State Features.  We consider a factored state representation.
Let $F$ denote the set of features in the environment, which are categorized
into _local features_ $F_{l}$ and _global features_ $F_{g}$,
$F\\!=\\!F_{l}\cup\\!F_{g}$. The local features of an agent $i$ are denoted by
$F^{i}_{l}$. Local features are agent-specific features that are controlled by
the agent’s actions and affect its performance (e.g. an agent’s $x,y$
location). Global features are shared among agents and denote the overall
state of the system. They are further divided into static and dynamic global
features, denoted by $F_{gs}$ and $F_{gd}$ respectively, based on whether they
are exogenous or can be modified by agent actions. An agent’s state $\hat{s}$
is described by $\vec{f}\\!=\vec{f}_{l}^{i}\cup\vec{f}_{gd}\cup\vec{f}_{gs}$.

######  Definition 2.

Static global features $F_{gs}$ are exogenous factors that affect all agents
and are observable to all agents but not changed by the agents’ actions (e.g.
ocean currents).

######  Definition 3.

Dynamic global features $F_{gd}$ describe properties of the environment that
affect agent operation directly or indirectly, and can be modified by agent
actions (e.g. locations and sizes of the shelves moved by the agents).

In this work, the penalty for NSE associated with a joint action is modeled as
a _function of the dynamic global features_ of the joint state,
$R_{N}({s})=\Omega(\vec{f}_{gd})$. However, the proposed framework and
approach can be extended to handle penalties defined over joint states and
actions.

##  5 Penalty Decomposition via Blame Assignment

Our algorithm for NSE penalty decomposition, Reward Estimation using
Counterfactual Neighbors (RECON), is outlined in Algorithm 1. RECON first
initializes each agent’s penalty function $R_{N}^{i}$ to zero (Line 1). After
the agents have calculated their naive policies independently, the NSE Monitor
component in the metareasoner calculates the joint penalty $\vec{r}_{N}$ for
NSEs incurred from joint policy $\vec{\pi}$ (Line 2). If the penalty exceeds a
pre-defined NSE tolerance threshold $\eta$, then the Blame Resolver decomposes
the joint penalty into local penalties for each agent, based on their relative
contribution to NSE, calculated using blame (credit) assignment (Lines 3-5).
The penalty decomposition resolves the dependency induced by the joint penalty
and enables decentralized policy computation to optimize task completion while
minimizing NSEs.

Blame Estimation. Blame (credit) assignment estimates an agent’s contribution
to a global penalty (or reward) using counterfactual states. The blame $B_{i}$
(Eqn. 1) is calculated for each agent $i$ in the joint state $s$, based on the
difference between the current NSE penalty and the minimum NSE penalty that
could have been achieved by the agent. The existing credit assignment
techniques calculate counterfactual states by considering all state features.
In our setting, NSE occurrence is determined only by the dynamic global
features $(\vec{f}_{gd})$. Therefore, the counterfactuals must be calculated
only over $\vec{f}_{gd}$. Generating counterfactual states for NSEs by
considering all the state features could result in incorrect attribution.
Consider multiple warehouse robots, with some carrying large shelves,
navigating in a narrow corridor. The NSE of blocked paths for human access is
determined by number of agents carrying large shelves. Generating
counterfactual states traditionally will involve changing agent location but
that does not provide information on NSE associated with multiple agents
carrying large shelves. Hence, counterfactuals must be generated over NSE-
relevant state features. We present a modified formulation of credit
assignment that uses _counterfactual neighbors_ instead of counterfactual
states.

######  Definition 4.

The counterfactual neighbors of a joint state $s$, denoted by $s_{c}$, is the
set of all states that vary only in the values of dynamic global features
$(\vec{f}_{gd})$ while all other feature values are same as in $s$.

Counterfactual neighbors are therefore a subset of counterfactual states.
Since the NSE penalty depends only on the state in our formulation, we
calculate counterfactual neighbors over the state variables. The set of
counterfactual neighbors that are reachable from the start state in the
environment are referred to as _valid counterfactual neighbors_ and are
denoted by ${s}_{c}^{v}$.

######  Definition 5.

Agent-specific counterfactual neighbors, denoted by $s^{i}_{c}\\!\subset
s_{c}$, are states that differ in those dynamic global features that can be
controlled by agent $i$, while other feature values are same as in the current
joint state $s$.

Agent-specific counterfactuals allow us to estimate an individual agent’s
contribution to NSE. Note that this is a typical form of counterfactual state
generation for credit assignment, in that the agent contribution is estimated
by either removing it or replacing its actions, while the other agents’
behaviors are fixed [?; ?].

Algorithm 1 RECON

Input: NSE Tolerance $\eta$, $\vec{\pi}$ for primary assigned task

1: Initialize $R^{i}_{N}(\hat{s})=0,\forall\hat{s}\in\hat{S}_{i},\forall
i\in\mathcal{A}$

2: Calculate joint penalty $r_{N}^{\vec{\pi}}$ for $\vec{\pi}$

3: if $r_{N}^{\vec{\pi}}$ $>$ $\eta$ then

4: Compute blame $B_{i}(s)$ using Eqn. 1, $\forall i\in\mathcal{A},\forall
s\in S$

5: $R^{i}_{N}(\hat{s}_{i})\leftarrow B_{i}(s)$, $\forall
i\in\mathcal{A},\forall\hat{s}_{i}\in s,\forall s\in S$

6: end if

7: return $[R^{1}_{N},...,R^{m}_{N}]$

### Counterfactual neighbors vs state counterfactuals.

Let ${s}^{i,v}_{c}\subset s^{v}_{c}$ be the set of valid (reachable)
counterfactual neighbors for agent $i$. The blame $B_{i}$ for agent $i$ in the
joint state $s$ is calculated as,

$B_{i}({s})=\frac{b_{i}({s})}{\sum_{i\in\mathcal{A}}b_{i}(s)}\cdot{R_{N}}(s),\text{
with}$ (1)
$b_{i}(s)=\frac{1}{2}\left({R_{N}^{*}}+\epsilon+\left(R_{N}(s)-\min_{s^{\prime}\in
s^{i,v}_{c}}R_{N}(s^{\prime})\right)\right),$ (2)

where $R_{N}$ is the joint NSE penalty function, $R_{N}^{*}=\max_{s\in
S}R_{N}(s)$ denotes the maximum joint NSE penalty possible which is used to
rescale blame in the range $[0,R_{N}^{*}]$ to avoid impractical values that
might be negative or exceed the joint NSE penalty itself. $\epsilon$ is a
small fixed value to avoid singularities during normalization and rescaling,
and $b_{i}(s)$ is an intermediate value used to calculate $B_{i}(s)$. The
above equation assigns blame proportional to the agent’s ability to mitigate
NSEs. This ensures that agents already making their best efforts are penalized
less compared to other agents.

The metareasoner’s Blame Resolver compiles a local penalty function
$R^{i}_{N}$ for each agent $i$ as follows:

$R^{i}_{N}(\hat{s}_{i})=B_{i}(s),\forall\hat{s}_{i}\in s,\forall s\in
S,i\in\mathcal{A}.$ (3)

The agents then solve the MASE-MDP, with the prescribed $R_{1}^{i}$
corresponding to their assigned task and $R^{i}_{N}$ provided by the
metareasoner, using lexicographic value iteration (LVI) [?].

### Generalizing $R^{i}_{N}$

The penalty function $R^{i}_{N}$ obtained from RECON in Algorithm 1, is
calculated based on the blame values corresponding to $\vec{\pi}$ and does not
provide any information about potential NSEs that may occur if the agents
followed a _different joint policy_. As a result, NSEs may persist when the
agents update their policy by solving the MASE-MDP with $R^{i}_{N}$. To
overcome this limitation, we consider a supervised learning approach to
generalize the NSE penalty to unseen situations by using the $R^{i}_{N}$,
based on the initial $\vec{\pi}$, as training data. The prediction accuracy
can further be improved by including the counterfactuals as part of the
training [?]. In our experiments, learning to generalize using counterfactuals
improves the performance in some settings, compared to generalizing without
using counterfactual information. Generalization helps scale the algorithm to
systems with large number of agents and eliminates the requirement of having
an iterative approach where agents could discover new NSE in unseen states. An
iterative RECON will work but it incurs a high computational overhead and does
not guarantee a monotonic improvement in policy with each iteration.

(a) Salp agents , inspired from [?; ?], are tasked with collecting physical
samples from sites ,, and depositing them at lab facility  surrounded by
corals  prone to damage.

(b) Overcooked environment, inspired from [?], shows 4 agents tasked with soup
orders. The kitchen shown above has tomatoes , onions , cooking pots , clean
dishes , and a serving counter , dirty dishes , and garbage bins  for waste.

(c) Warehouse environment, inspired from [?], shows agents  tasked with
processing shelves , at counter . The narrow corridors  with human workers
that are inconvenienced around multiple robots carrying shelves.

Figure 2: Example problem instances from domains used in our experiments.

##  6 Experimental Setup

We evaluate our approach in simulation using three proof-of-concept grid-world
domains (salp, overcooked, and warehouse). Figure 2 shows sample instances
from each domain. All algorithms are implemented in Python and the simulations
were run on an Ubuntu machine with $32$ GB RAM.The MASE-MDP problem is solved
using lexicographic value iteration [?] and results are averaged over five
instances in each domain. We test with zero NSE tolerance $\eta\\!=\\!0$ to
evaluate the extent of NSE mitigation using RECON, and using
$\epsilon=10^{-4}$ for rescaling the blame values.

### Baselines

We compare the performance of RECON with four baselines. First is a _Naive
policy_ that is optimal for agent task completion and does not optimize NSE
mitigation, providing an upper bound on NSE penalty. Second is performing
credit assignment using the _Difference Reward_ technique [?] that calculates
blame by removing the agent from the system,
$B_{i}(s)=R_{N}(s)-R_{N}(s_{-i})$. A direct comparison with this technique is
not feasible since it calculates counterfactuals based on assumptions that do
not hold in our setting. Therefore, we modify the approach to calculate
counterfactuals over dynamic global features of the agents, and calculate
blame as $B_{i}(s)=R_{N}(s)-\max_{s^{\prime}\in
s^{i,v}_{c}}R_{N}(s^{\prime})$. We use max over all valid counterfactual
neighbors to get an aggressive blame assignment, producing conservative
(safer) behavior. Third, we compare with a variant of _considerate reward_
inspired from [?], where they augment agents’ reward functions with value
functions of subsequently acting agent to avoid their definition of NSE. Since
in our setting, agents operate simultaneously and NSE is defined for multiple
agents, we augment agents’ prescribed rewards with blame contributions from
the rest of the agents as shown in Equation 4.

$R^{i}_{r}(\hat{s}_{i})=\alpha_{1}\frac{R_{1}^{i}(\hat{s}_{i})}{R_{1}^{*}}+\alpha_{2}\frac{R_{N}(s)-B_{i}(s)}{R_{N}^{*}}$
(4)

where $R_{1}^{*}=\max_{\hat{s}\in\hat{S}_{i}}R_{1}(\hat{s})$ is the maximum
possible reward for assigned task for agent $i$, $\alpha_{1}$ and $\alpha_{2}$
are the selfish and care coefficients respectively. $R_{1}^{*}$ and
$R_{N}^{*}$ are used to normalization so that their relative scales are not an
inherent factor, but a controlled one (using $\alpha_{1}$, $\alpha_{2}$).
Fourth, we compare with _generalized RECON w/o counterfactual data_ , which
uses only the $R^{i}_{N}$ values from RECON to generalize the penalty
situations using supervised learning on global features of the local states
and attributed blame values. Finally, we compare with _generalized RECON w/
counterfactual (cf) data_ , which uses both counterfactuals and $R^{i}_{N}$
values to generalize $R^{i}_{N}$ to unseen situations using supervised
learning similar to the previous baseline. The techniques are evaluated on the
following domains.

### Sample Collection using Salps

Salps are marine organisms that can form chains and coordinate movements [?].
We consider a setting where salp-inspired underwater robots must collect
chemical samples from different locations in the seabed. Each agent is tasked
with collecting a specific type of sample and depositing it to a designated
location for analysis. An agent’s state is denoted by $\langle
x,y,sample,coral,status\rangle$, where $x,y$ denote agent’s location, $sample$
indicates the sample type, $coral$ indicates presence of coral at $x,y$, and
$status$ is a binary variable indicating if the sample has been deposited at
the destination. In our experiments, samples can be of type $A,B$ or $X$
indicating no sample. Agents can move in all four directions, and $pick$ and
$drop$ samples, each with a reward of $-1$. The reward for task completion is
$+100$. The move actions succeed with a probability of $0.8$ or fail and slide
to an adjacent location with a probability of $0.2$. Features used for
training when generalizing $R^{i}_{N}$ are $\langle sample,coral\rangle$. We
test with five instances of $20\\!\times\\!20$ grids that vary in coral
locations. This allows us to evaluate the different approaches to mitigate
NSEs, while keeping other conditions fixed.

NSE: The salp-like robots may have chemical residues floating around them when
transporting samples. A joint NSE occurs when multiple robots carrying
chemical samples are in immediate vicinity of corals, potentially damaging it.

### Overcooked

In this domain, agents are tasked with preparing and serving food [?]. The
agents work together to deliver a fixed number of tomato and onion soup
orders, while keeping the kitchen clean. Each agent is assigned a cleaning or
a cooking task, and it receives a reward of $+100$ upon task completion.
Actions in non-goal state have a reward of $-1$. In each problem instance,
$20\%$ agents are assigned to cleaning and the rest are assigned the cooking
task. An agent’s state is represented as $\langle
x,y,dir,object,bin,status\rangle$, where $x,y$ denote its location, $dir$
denotes its orientation, $bin$ indicates the presence of garbage bins at
$x,y$, and $status$ indicates the completion status of its assigned task.
Agents can move forward in all directions, and $interact$ with objects in the
kitchen. Interactions vary from picking and putting down ingredients and
dishes, to using the cooking pots and dumping garbage in bins. Agent’s _move_
actions succeed with a probability of $0.8$ or slide to an adjacent location
with $0.2$ probability. The _interact_ actions succeed with probability of
$0.8$ or fail and remain in the same state with probability $0.2$. Features
used for generalizing $R^{i}_{N}$ are $\langle object,bin\rangle$. We test
with five instances of $15\\!\times 15$ grids that vary in the locations of
the garbage bins. NSE: The garbage bins emit bad odors and attract flies. Any
object involved in soup preparation must therefore be kept away from the
garbage bins or the waste must be disposed at a farther bin, otherwise the
food quality is affected.

(a) Salp domain

(b) Overcooked domain

(c) Warehouse domain

Figure 3: Average NSE penalty and standard deviation for varying % of agents
undergoing policy update in each domain with $25$ agents.

### Warehouse Inventory Management

In the multi-robot warehouse environment [?], agents are tasked with getting
the requested shelves of different sizes (_big_ or _small_) that will be
processed at the counter and must be returned to their location. We consider
the setting where these robot agents are assigned specific shelves that they
need to locate, pick up, transport, process, and bring back, in order to
complete their task. Each agent receives a reward of $+100$ for returning its
processed shelf, and $-1$ for each step before completion. An agent’s state is
represented as $\langle x,y,\emph{shelf size},\emph{shelf
status},\emph{corridor},done\rangle$, where $x,y$ denote its location, _shelf
size_ is the size of the shelf transported by the agent and can be one of
_big_ , _small_ , or _X_ for no shelf. _shelf status_ denotes the current
processing stage of the assigned shelf which could be one of _{ picked up,
processed, delivered}_, $corridor$ denotes presence of a narrow corridor at a
given location $x,y$, and $done$ indicates the status of task completion. The
corridors in the warehouse are structural features unrelated to shelves and do
not impact the robots’ delivery tasks. Robots can fulfill their tasks with or
without going through a corridor; shelves are organized separately in another
part of the warehouse. The robots can move in all directions and _toggle load_
that allows them to pick up and drop shelves. A shelf is labeled as processed
if it is dropped at the counter. Agent’s _move_ actions succeed with a
probability of $0.8$ and slide to an adjacent location with $0.2$ probability.
The _toggle load_ action succeeds with a probability of $0.8$ or fails with a
probability of $0.2$, in which case the agent remains in the same state.
Features used to generalize $R^{i}_{N}$ are $\langle\emph{shelf
size},\emph{shelf status},corridor\rangle$. We test with five instances of
$15\times 15$ grids maintaining the same location for shelves and counters,
but changing the location of corridors across the warehouse.
NSE: When multiple robots carrying shelves are simultaneously in the same
narrow corridor, it inconveniences human workers to navigate and access the
area. The NSE penalty depends on shelf size and the number of agents carrying
shelves in the same corridor.

(a) Salp domain

(b) Overcooked domain

(c) Warehouse domain

Figure 4: Average NSE penalties and standard deviations, averaged over five
problem instances and varying number of agents.

(a) Salp domain

(b) Overcooked domain

(c) Warehouse domain

Figure 5: Average runtime of all techniques, with increasing number of agents
across five problem instances, with standard deviation.

### NSE Penalty

We employ a logarithmic NSE penalty function (Equation 5) for the domains
described earlier. The NSE penalty calculated as,

$R_{N}(s)=\sum_{d\in F_{gd}}\sum_{k\in
d}\beta_{k}\cdot\log\left(\alpha_{d}N_{k}+1\right),$ (5)

where $F_{gd}$ is the set of dynamic global features in the environment, $d$
is a feature in $F_{gd}$, and $k$ is the value of feature $d$. $N_{k}$ is the
number of agents with dynamic global feature value $d\\!=\\!k$ in the joint
state $s$. $\beta_{k}$ is a scaling factor that is directly related to the NSE
penalty for a specific feature value $d=k$. Essentially, higher NSE penalties
correspond to higher values of $\beta_{k}$, reflecting that mitigating more
critical features yields a more substantial reduction in NSE penalty, aligning
with the nuanced dynamics of real-world environments. $\alpha_{d}\,(>0)$ is a
sensitivity parameter for NSE associated with feature $d$. A larger
$\alpha_{d}$ denotes that the NSE penalty is more sensitive to an increasing
number of agents with a specific feature value.

We use a logarithmic function to effectively model scenarios where the NSE
impact plateaus with a certain number of agents performing undesirable actions
[?; ?]. For example, there may be no significant difference in the NSE impact
caused by 10 agents blocking a corridor versus 11 agents blocking a corridor.
A log penalty will lead to prioritizing the mitigation of severe NSEs, by
updating the policy of a subset of agents. However, note that RECON can work
with any penalty function in practice.

##  7 Results and Discussion

We compare the effectiveness of different techniques in mitigating NSEs, along
multiple dimensions such as the number of agents undergoing policy update to
mitigate the impact produced, NSE avoidance by generalizing $R^{i}_{N}$ to
unseen situations, and run time of the techniques.

Effect of number of agents undergoing policy update  First, we evaluate the
effectiveness of our approach in mitigating NSEs, by varying the number of
agents whose policy must be updated after $R^{i}_{N}$ is calculated. For each
domain, we consider $25$ agents in the environment and vary the percentage of
agents undergoing policy update to minimize NSEs, from $10\%$ to $100\%$. We
select agents to update policies by ranking them in the decreasing order of
their blame values. The results, in Figure 3, show naive policy as a straight
line in all the domains, since it is indifferent to NSEs. For the salp sample
collection domain, Figure 3(a) shows that the NSE penalty plateaus at the
$50\%$ mark, where the difference reward baseline shows its best performance
against the naive policy. For the warehouse domain, in Figure 3(c), we see a
similar trend of NSE penalty plateauing at the $50\%$ mark however, difference
reward fails to mitigate NSE and performs same as the naive policy. In case of
the overcooked domain, Figure 3(b) shows a steady decline of NSE penalty using
RECON and its generalized versions but similar to the warehouse domain
results, difference reward fails to mitigate NSE and performs same as the
naive policy.

Note that in some cases, NSE may not be avoided even when we update the
policies of $100\%$ of the agents in the system. This is because we prioritize
completing the task optimally over minimizing NSEs. Hence in some cases, it
may be impossible for the agents to avoid NSEs, while optimally completing
their tasks. This is a problem characteristic and not a limitation of RECON.

Overall, the results show that RECON is able to mitigate NSEs, without
updating the policies of a large number of agents. Based on these results, we
update the policies of $50\%$ of agents in all the following experiments,
across all domains, since we consistently see a plateau and drop in the NSE
penalty at $50\%$. For some settings, updating all the agents’ policies may be
effective but it is not practically feasible.

Effect of generalizing $R^{i}_{N}$ Since the penalty function $R^{i}_{N}$ is
calculated based on the blame values corresponding to $\vec{\pi}$, policy
updates may produce novel NSEs that were not encountered earlier. We evaluate
the effectiveness of generalizing $R^{i}_{N}$ to unseen situations by
comparing the performances of RECON, with and without generalization using
counterfactual data, against the baselines. Based on results from Figure 3, we
conduct experiments with $50\%$ of agents undergoing policy update for each
technique. Figure 4 shows that both the generalized versions of RECON
outperform other methods, consistently reducing NSE penalty across domains. In
few instances, generalized RECON with counterfactual data outperforms the
generalization without counterfactual data. These results show that
generalization is useful and can mitigate NSEs considerably even when
counterfactual data is not used for training. We also tested the techniques
when $100\%$ of agents undergo policy update. We observe that the NSE
penalties further reduce by over $60\%$. While this may be practically
infeasible for large systems, it shows the effectiveness of our approach in
mitigating NSEs.

Scalability  We evaluate the scalability of our approach in terms of run-time
to solve the problems as we increase the number of agents in the system from
$10$ to $100$. Figure 5 plots the runtime of different techniques, in the
three domains, with varying number of agents. The run times increase in an
approximately linear manner as the number of agents increases. The considerate
reward baseline takes the least time possible because it involves solving only
for one agent as other agents are treated as part of the environment.
Difference reward baseline takes longer since policy computation is done in
two stages but it does not evaluate all counterfactual neighbors and defaults
to the counterfactual producing maximum NSE penalty. RECON takes longer than
difference reward since it involves computing and comparing counterfactuals.
As expected, RECON with $R_{N}$ generalizations take longer than the other
approaches, since they generalize NSE penalties over the entire state space.
However, the time increase is small, compared to the reduction in NSE
penalties they offer, even in settings with $100$ agents. Since RECON is an
offline approach, the slightly higher runtimes for additional processing to
generalize $R_{N}$ may be tolerable in many settings. Since RECON is an
offline approach, the slightly higher runtimes for additional processing to
generalize $R^{i}_{N}$ may be tolerable in many settings.

##  8 Summary and Future Work

This paper formalizes the problem of mitigating NSEs in cooperative multi-
agent settings as a decentralized, bi-objective problem. The agents’ assigned
tasks follow transition and reward independence. The agents produce no NSE
when operating in isolation but their joint actions produce NSE and incur a
joint penalty. The agents have no prior knowledge about the side effects of
their joint actions. We present a metareasoning approach to detect NSEs and
update agent policies in a decentralized manner, by decomposing the joint NSE
penalty into individual penalties. Our algorithm, RECON, uses a
counterfactual-based blame attribution to estimate each agent’s contribution
towards the joint penalty. Our experiments demonstrate the effectiveness of
our approach in mitigating NSEs, by updating the policies of a subset of
agents.

Our framework currently supports Dec-MDPs with transition and reward
independence. In the future, we aim to relax this assumption and extend our
approach to settings with tightly coupled task assignment. Another interesting
future direction is to exploit the agent dependencies and interactions for
task completion to mitigate NSEs, by leveraging their complementary skills.

## Acknowledgments

This work was supported in part by ONR grant number N00014-23-1-2171.

## References

  * [Alamdari et al., 2022] Parand A. Alamdari, Toryn Q. Klassen, Rodrigo T. Icarte, and Sheila A. McIlraith.  Be considerate: Avoiding negative side effects in reinforcement learning.  In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 18–26, 2022.
  * [Becker et al., 2004] Raphen Becker, Shlomo Zilberstein, Victor Lesser, and Claudia V. Goldman.  Solving transition independent decentralized markov decision processes.  Journal of Artificial Intelligence Research (JAIR), 22:423–455, 2004.
  * [Bilbao and Edelman, 2000] Jesus M. Bilbao and Paul H. Edelman.  The shapley value on convex geometries.  Discrete Applied Mathematics, 103(1-3):33–40, 2000.
  * [Carlin and Zilberstein, 2012] Alan Carlin and Shlomo Zilberstein.  Bounded rationality in multiagent systems using decentralized metareasoning.  pages 1–28, 2012.
  * [Carroll et al., 2019] Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan.  On the utility of learning about humans for human-ai coordination.  Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
  * [Choudhury et al., 2024] Moumita Choudhury, Sandhya Saisubramanian, Hao Zhang, and Shlomo Zilberstein.  Minimizing negative side effects in cooperative multi-agent systems using distributed coordination.  In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 2213–2215, 2024.
  * [D’Andrea, 2012] Raffaello D’Andrea.  A revolution in the warehouse: A retrospective on kiva systems and the grand challenges ahead.  IEEE Transactions on Automation Science and Engineering, 9(4):638–639, 2012.
  * [Fioretto et al., 2018] Ferdinando Fioretto, Enrico Pontelli, and William Yeoh.  Distributed constraint optimization problems and applications: A survey.  Journal of Artificial Intelligence Research (JAIR), 61:623–698, 2018.
  * [Gemp et al., 2022] Ian Gemp, Thomas Anthony, Janos Kramar, Tom Eccles, Andrea Tacchetti, and Yoram Bachrach.  Designing all-pay auctions using deep learning and multi-agent simulation.  Scientific Reports, 12(1):16937, 2022.
  * [Goldman and Zilberstein, 2004] Claudia V. Goldman and Shlomo Zilberstein.  Decentralized control of cooperative systems: Categorization and complexity analysis.  Journal of Artificial Intelligence Research (JAIR), 22:143–174, 2004.
  * [Hadfield-Menell et al., 2017] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca Dragan.  Inverse reward design.  Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
  * [Klassen et al., 2022] Toryn Q. Klassen, Sheila A. McIlraith, Christian Muise, and Jarvis Xu.  Planning to avoid side effects.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 36, pages 9830–9839, 2022.
  * [Krakovna et al., 2018] Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg.  Penalizing side effects using stepwise relative reachability.  2018\.
  * [Krakovna et al., 2020] Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg.  Avoiding side effects by considering future tasks.  Advances in Neural Information Processing Systems (NeurIPS), 33:19064–19074, 2020.
  * [Meng, 2012] Fan-Yong Meng.  The core and shapley function for games on augmenting systems with a coalition structure.  International Journal of Mathematical and Computational Sciences, 6(8):813–818, 2012.
  * [Nguyen et al., 2018] Duc T. Nguyen, Akshat Kumar, and Hoong C. Lau.  Credit assignment for collective multiagent RL with global rewards.  Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
  * [Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht.  Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks.  In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021.
  * [Pineda et al., 2015] Luis Enrique Pineda, Kyle Hollins Wray, and Shlomo Zilberstein.  Revisiting multi-objective mdps with relaxed lexicographic preferences.  In 2015 AAAI Fall Symposium Series, 2015.
  * [Proper and Tumer, 2012] Scott Proper and Kagan Tumer.  Modeling difference rewards for multiagent learning.  In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1397–1398, 2012.
  * [Rahmattalabi et al., 2016] Aida Rahmattalabi, Jen Jen Chung, Mitchell Colby, and Kagan Tumer.  D++: Structural credit assignment in tightly coupled multiagent domains.  In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4424–4429. IEEE, 2016.
  * [Saisubramanian et al., 2020] Sandhya Saisubramanian, Ece Kamar, and Shlomo Zilberstein.  A multi-objective approach to mitigate negative side effects.  In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence (JAIR), 2020.
  * [Saisubramanian et al., 2022] Sandhya Saisubramanian, Ece Kamar, and Shlomo Zilberstein.  Avoiding negative side effects of autonomous systems in the open world.  Journal of Artificial Intelligence Research (JAIR), 74:143–177, 2022.
  * [Shapley, 1953] Lloyd S. Shapley.  A value for n-person games.  1953\.
  * [Sundararajan and Najmi, 2020] Mukund Sundararajan and Amir Najmi.  The many shapley values for model explanation.  In International Conference on Machine Learning (ICML), pages 9269–9278. PMLR, 2020.
  * [Sutherland and Madin, 2010] Kelly R. Sutherland and Laurence P. Madin.  Comparative jet wake structure and swimming performance of salps.  Journal of Experimental Biology, 213(17):2967–2975, 2010.
  * [Sutherland and Weihs, 2017] Kelly R. Sutherland and Daniel Weihs.  Hydrodynamic advantages of swimming by salp chains.  Journal of The Royal Society Interface, 14(133):20170298, 2017.
  * [Svegliato et al., 2022] Justin Svegliato, Connor Basich, Sandhya Saisubramanian, and Shlomo Zilberstein.  Metareasoning for safe decision making in autonomous systems.  In International Conference on Robotics and Automation (ICRA), pages 11073–11079. IEEE, 2022.
  * [Turner et al., 2020] Alexander M. Turner, Dylan Hadfield-Menell, and Prasad Tadepalli.  Conservative agency via attainable utility preservation.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 385–391, 2020.
  * [Wray et al., 2015] Kyle Wray, Shlomo Zilberstein, and Abdel-Illah Mouaddib.  Multi-objective MDPs with conditional lexicographic reward preferences.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 29, 2015.
  * [Zerbel and Tumer, 2023] Nicholas Zerbel and Kagan Tumer.  Counterfactual focused learning.  2023\.
  * [Zhang et al., 2018] Shun Zhang, Edmund H. Durfee, and Satinder Singh.  Minimax-regret querying on side effects for safe optimality in factored markov decision processes.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), 2018.
  * [Zhang et al., 2020] Shun Zhang, Edmund Durfee, and Satinder Singh.  Querying to find a safe policy under uncertain safety constraints in markov decision processes.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 34, pages 2552–2559, 2020.
  * [Zhu et al., 2018] Ming Zhu, Xiao-Yang Liu, and Xiaodong Wang.  Joint transportation and charging scheduling in public vehicle systems—a game theoretic approach.  IEEE Transactions on Intelligent Transportation Systems, 19(8):2407–2419, 2018.
  * [Zilberstein, 2011] Shlomo Zilberstein.  Metareasoning and bounded rationality.  In Metareasoning: Thinking about Thinking, pages 27–40. MIT Press, Cambridge, MA, USA, 2011.