* Deng et al. [2023] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing.  Multilingual jailbreak challenges in large language models.  _ArXiv_ , abs/2310.06474, 2023.  URL https://api.semanticscholar.org/CorpusID:263831094.
  * desik98 [2023] desik98.  Telugu Riddles.  https://huggingface.co/datasets/desik98/TeluguRiddles, 2023.  Accessed: 2023-11-30.
  * Dhole et al. [2021] Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, et al.  Nl-augmenter: A framework for task-sensitive natural language augmentation.  _arXiv preprint arXiv:2112.02721_ , 2021.
  * Doddapaneni et al. [2023] Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar.  Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 12402–12426, Toronto, Canada, July 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.acl-long.693.
  * Dodge et al. [2021] Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner.  Documenting the english colossal clean crawled corpus.  _CoRR_ , abs/2104.08758, 2021.  URL https://arxiv.org/abs/2104.08758.
  * Doğruöz et al. [2023] A Seza Doğruöz, Sunayana Sitaram, and Zheng-Xin Yong.  Representativeness as a forgotten lesson for multilingual and code-switched data collection and preparation.  _arXiv preprint arXiv:2310.20470_ , 2023.
  * Dolan & Brockett [2005] Bill Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_. Asia Federation of Natural Language Processing, January 2005.  URL https://www.microsoft.com/en-us/research/publication/automatically-constructing-a-corpus-of-sentential-paraphrases/.
  * Durmus et al. [2023] Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al.  Towards measuring the representation of subjective global opinions in language models.  _arXiv preprint arXiv:2306.16388_ , 2023.
  * el2e10 [2023a] el2e10.  Aya Indic Sentiment.  https://huggingface.co/datasets/el2e10/aya-indicsentiment, 2023a.  Accessed: 2023-11-28.
  * el2e10 [2023b] el2e10.  Aya Paraphrase.  https://huggingface.co/datasets/el2e10/aya-paraphrase, 2023b.  Accessed: 2023-11-28.
  * Fabbri et al. [2019] Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R. Radev.  Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model, 2019.
  * Falck et al. [2012] Oliver Falck, Stephan Heblich, Alfred Lameli, and Jens Südekum.  Dialects, cultural identity, and economic exchange.  _Journal of urban economics_ , 72(2-3):225–239, 2012.
  * Fan et al. [2021] Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al.  Beyond english-centric multilingual machine translation.  _Journal of Machine Learning Research_ , 22(107):1–48, 2021.
  * Farahani et al. [2021] Mehrdad Farahani, Mohammad Gharachorloo, and Mohammad Manthouri.  Leveraging parsbert and pretrained mt5 for persian abstractive text summarization.  In _2021 26th International Computer Conference, Computer Society of Iran (CSICC)_ , pp. 1–6. IEEE, 2021.
  * Ferrara [2023] Emilio Ferrara.  Should chatgpt be biased? challenges and risks of bias in large language models.  _arXiv preprint arXiv:2304.03738_ , 2023.
  * $\forall$ et al. [2020] $\forall$, Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir.  Participatory research for low-resourced machine translation: A case study in African languages.  In Trevor Cohn, Yulan He, and Yang Liu (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2020_ , Online, November 2020. Association for Computational Linguistics.  10.18653/v1/2020.findings-emnlp.195.  URL https://aclanthology.org/2020.findings-emnlp.195.
  * Franzoni & Sauermann [2014] Chiara Franzoni and Henry Sauermann.  Crowd science: The organization of scientific research in open collaborative projects.  _Research Policy_ , 43(1):1–20, 2014.  ISSN 0048-7333.  https://doi.org/10.1016/j.respol.2013.07.005.  URL https://www.sciencedirect.com/science/article/pii/S0048733313001212.
  * ganeshjcs [2023a] ganeshjcs.  Hindi Article Summarization.  https://huggingface.co/datasets/ganeshjcs/hindi-article-summarization, 2023a.  Accessed: 2023-11-28.
  * ganeshjcs [2023b] ganeshjcs.  Hindi Headline Article Generation.  https://huggingface.co/datasets/ganeshjcs/hindi-headline-article-generation, 2023b.  Accessed: 2023-11-28.
  * Gao & Liu [2023] Ya Gao and WenQi Liu.  Measures to sustain endangered languages: A bilingual competition model with sliding mode control.  _PLOS ONE_ , 18(6):1–16, 06 2023.  10.1371/journal.pone.0287850.  URL https://doi.org/10.1371/journal.pone.0287850.
  * Garrette et al. [2013] Dan Garrette, Jason Mielens, and Jason Baldridge.  Real-world semi-supervised learning of pos-taggers for low-resource languages.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 583–592, 2013.
  * Gerosa et al. [2021] Marco Gerosa, Igor Wiese, Bianca Trinkenreich, Georg Link, Gregorio Robles, Christoph Treude, Igor Steinmacher, and Anita Sarma.  The shifting sands of motivation: Revisiting what drives contributors in open source.  In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_ , pp. 1046–1058. IEEE, 2021.
  * Ghosh & Caliskan [2023] Sourojit Ghosh and Aylin Caliskan.  Chatgpt perpetuates gender bias in machine translation and ignores non-gendered pronouns: Findings across bengali and five other low-resource languages, 2023.
  * Gliwa et al. [2019] Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer.  SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pp. 70–79, Hong Kong, China, November 2019. Association for Computational Linguistics.  10.18653/v1/D19-5409.  URL https://aclanthology.org/D19-5409.
  * Goodwin [2017] Charles Goodwin.  _Co-Operative Action_.  Learning in Doing: Social, Cognitive and Computational Perspectives. Cambridge University Press, 2017.
  * Goyal et al. [2021a] Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau.  Larger-Scale Transformers for Multilingual Masked Language Modeling.  In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_. Association for Computational Linguistics, 2021a.
  * Goyal et al. [2021b] Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan.  The flores-101 evaluation benchmark for low-resource and multilingual machine translation, 2021b.
  * Graff et al. [2003] David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda.  English gigaword.  _Linguistic Data Consortium, Philadelphia_ , 4(1):34, 2003.
  * Grano et al. [2017] Giovanni Grano, Andrea Di Sorbo, Francesco Mercaldo, Corrado A Visaggio, Gerardo Canfora, and Sebastiano Panichella.  Android apps and user feedback: a dataset for software evolution and quality improvement.  In _Proceedings of the 2nd ACM SIGSOFT international workshop on app market analytics_ , pp. 8–11, 2017.
  * Groeneveld et al. [2024] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi.  OLMo: Accelerating the Science of Language Models.  _arXiv preprint_ , 2024.
  * Gu et al. [2022] Yuling Gu, Bhavana Dalvi, and Peter Clark.  DREAM: Improving situational QA by first elaborating the situation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 1115–1127, Seattle, United States, July 2022. Association for Computational Linguistics.  10.18653/v1/2022.naacl-main.82.  URL https://aclanthology.org/2022.naacl-main.82.
  * Guevara-Rukoz et al. [2020] Adriana Guevara-Rukoz, Isin Demirsahin, Fei He, Shan-Hui Cathy Chu, Supheakmungkol Sarin, Knot Pipatsrisawat, Alexander Gutkin, Alena Butryna, and Oddur Kjartansson.  Crowdsourcing Latin American Spanish for low-resource text-to-speech.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pp. 6504–6513, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://aclanthology.org/2020.lrec-1.801.
  * Gulli [2005] Antonio Gulli.  AG’s Corpus of News Articles.  _Dipartimento di Informatica, University of Pisa, Nov_ , 2005.  URL http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html.
  * Gunasekar et al. [2023] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li.  Textbooks are all you need, 2023.
  * Gupta et al. [2023] Himanshu Gupta, Kevin Scaria, Ujjwala Anantheswaran, Shreyas Verma, Mihir Parmar, Saurabh Arjun Sawant, Chitta Baral, and Swaroop Mishra.  Targen: Targeted data generation with large language models, 2023.
  * Hanley et al. [2020] Margot Hanley, Apoorv Khandelwal, Hadar Averbuch-Elor, Noah Snavely, and Helen Nissenbaum.  An ethical highlighter for people-centric dataset creation.  _arXiv preprint arXiv:2011.13583_ , 2020.
  * Hardmeier & Guillou [2018] Christian Hardmeier and Liane Guillou.  Pronoun translation in english-french machine translation: An analysis of error types, 2018.
  * Hartung et al. [2023] Kai Hartung, Aaricia Herygers, Shubham Kurlekar, Khabbab Zakaria, Taylan Volkan, Sören Gröttrup, and Munir Georges.  Measuring sentiment bias in machine translation, 2023.
  * Hasan et al. [2021] Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pp. 4693–4703, Online, August 2021. Association for Computational Linguistics.  10.18653/v1/2021.findings-acl.413.  URL https://aclanthology.org/2021.findings-acl.413.
  * Haugen [1959] Einar Haugen.  Planning for a standard language in modern norway.  _Anthropological linguistics_ , pp. 8–21, 1959.
  * Held et al. [2023] William Held, Camille Harris, Michael Best, and Diyi Yang.  A material lens on coloniality in nlp, 2023.
  * Hermann et al. [2015] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  Teaching machines to read and comprehend.  In _Advances in neural information processing systems_ , pp. 1693–1701, 2015.
  * hghader1 [2023] hghader1.  FarsTail-Instruct-LLM.  https://huggingface.co/datasets/hghader1/FarsTail-Instruct-LLM, 2023.  Accessed: 2023-11-28.
  * Holmström & Doostmohammadi [2023] Oskar Holmström and Ehsan Doostmohammadi.  Making instruction finetuning accessible to non-English languages: A case study on Swedish models.  In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pp. 634–642, Tórshavn, Faroe Islands, May 2023. University of Tartu Library.  URL https://aclanthology.org/2023.nodalida-1.62.
  * Honovich et al. [2023] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick.  Unnatural instructions: Tuning language models with (almost) no human labor.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 14409–14428, Toronto, Canada, July 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.acl-long.806.
  * Hovy & Prabhumoye [2021] Dirk Hovy and Shrimai Prabhumoye.  Five sources of bias in natural language processing.  _Language and Linguistics Compass_ , 15(8):e12432, 2021.
  * Hovy et al. [2001] Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran.  Toward semantics-based answer pinpointing.  In _Proceedings of the First International Conference on Human Language Technology Research_ , 2001.  URL https://aclanthology.org/H01-1069.
  * Hsueh et al. [2009] Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani.  Data quality from crowdsourcing: a study of annotation selection criteria.  In _Proceedings of the NAACL HLT 2009 workshop on active learning for natural language processing_ , pp. 27–35, 2009.
  * Hu et al. [2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models, 2021.
  * Huang et al. [2023] Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al.  Acegpt, localizing large language models in arabic.  _arXiv preprint arXiv:2309.12053_ , 2023.
  * Huang et al. [2022] Kuan-Hao Huang, I-Hung Hsu, Premkumar Natarajan, Kai-Wei Chang, and Nanyun Peng.  Multilingual generative language models for zero-shot cross-lingual event argument extraction, 2022.
  * Huang et al. [2019] Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Cosmos QA: Machine reading comprehension with contextual commonsense reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2391–2401, Hong Kong, China, November 2019. Association for Computational Linguistics.  10.18653/v1/D19-1243.  URL https://aclanthology.org/D19-1243.
  * Hussain et al. [2021] Khalid Hussain, Nimra Mughal, Irfan Ali, Saif Hassan, and Sher Muhammad Daudpota.  Urdu News Dataset 1M.  Technical report, Mendeley Data, V3, 2021.
  * Hämäläinen [2021] Mika Hämäläinen.  _Endangered Languages are not Low-Resourced!_ , pp. 1–11.  University of Helsinki, March 2021.  10.31885/9789515150257.1.  URL http://dx.doi.org/10.31885/9789515150257.1.
  * Iftitahu [2023a] Iftitahu.  Indonesian Instruct Stories.  https://huggingface.co/datasets/Iftitahu/indonesian_instruct_stories, 2023a.  Accessed: 2023-11-28.
  * Iftitahu [2023b] Iftitahu.  Javanese Instruct Stories.  https://huggingface.co/datasets/Iftitahu/javanese_instruct_stories, 2023b.  Accessed: 2023-11-28.
  * Iftitahu [2023c] Iftitahu.  Sudanese Instruct Stories.  https://huggingface.co/datasets/Iftitahu/sundanese_instruct_stories, 2023c.  Accessed: 2023-11-28.
  * Ilhomovna & Yuldasheva [2023] Amirova Gulruh Ilhomovna and S Yuldasheva.  You have got to know your language to understand your culture.  In _Interdiscipline Innovation and Scientific Research Conference_ , volume 1, pp. 103–106, 2023.
  * Iyer et al. [2012] Shankar Iyer, Nikhil Dandekar, and Kornäl Csernai.  Quora question pairs, 2012.
  * Iyer et al. [2022] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al.  Opt-iml: Scaling language model instruction meta learning through the lens of generalization.  _arXiv preprint arXiv:2212.12017_ , 2022.
  * Ji et al. [2023a] Meng Ji, Meng Ji, Pierrette Bouillon, and Mark Seligman.  _Cultural and Linguistic Bias of Neural Machine Translation Technology_ , pp. 100–128.  Studies in Natural Language Processing. Cambridge University Press, 2023a.  10.1017/9781108938976.005.
  * Ji et al. [2023b] Yunjie Ji, Yan Gong, Yong Deng, Yiping Peng, Qiang Niu, Baochang Ma, and Xiangang Li.  Towards better instruction following language models for chinese: Investigating the impact of training data and evaluation, 2023b.
  * jjzha [2023] jjzha.  IMDB Dutch Instruct.  https://huggingface.co/datasets/jjzha/imdb-dutch-instruct, 2023.  Accessed: 2023-11-28.
  * Jørgensen et al. [2015] Anna Jørgensen, Dirk Hovy, and Anders Søgaard.  Challenges of studying and processing dialects in social media.  In _Proceedings of the Workshop on Noisy User-generated Text_ , pp. 9–18, Beijing, China, July 2015. Association for Computational Linguistics.  10.18653/v1/W15-4302.  URL https://aclanthology.org/W15-4302.
  * Joseph Cheung [2023] Joseph Cheung.  GuanacoDataset (Revision 8cf0d29) , 2023.  URL https://huggingface.co/datasets/JosephusCheung/GuanacoDataset.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.  triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.  _arXiv e-prints_ , art. arXiv:1705.03551, 2017.
  * Joshi et al. [2019] Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali.  Unsung challenges of building and deploying language technologies for low resource language communities.  In _Proceedings of the 16th International Conference on Natural Language Processing_ , pp. 211–219, International Institute of Information Technology, Hyderabad, India, December 2019. NLP Association of India.  URL https://aclanthology.org/2019.icon-1.25.
  * Joshi et al. [2020] Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 6282–6293, Online, July 2020. Association for Computational Linguistics.  10.18653/v1/2020.acl-main.560.  URL https://aclanthology.org/2020.acl-main.560.
  * jxmorris12 et al. [2023] jxmorris12, thomwolf, lhoestq, and lewtun.  ag_news.  https://huggingface.co/datasets/ag_news, 2023.  Accessed: 2023-11-28.
  * Kanerva et al. [2021] Jenna Kanerva, Filip Ginter, Li-Hsin Chang, Iiro Rastas, Valtteri Skantsi, Jemina Kilpeläinen, Hanna-Mari Kupari, Jenna Saarni, Maija Sevón, and Otto Tarkka.  Finnish paraphrase corpus.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pp. 288–298, Reykjavik, Iceland (Online), May 31–2 June 2021. Linköping University Electronic Press, Sweden.  URL https://aclanthology.org/2021.nodalida-main.29.
  * Khandelwal et al. [2023] Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale.  Casteist but not racist? quantifying disparities in large language model bias between india and the west.  _ArXiv_ , abs/2309.08573, 2023.  URL https://api.semanticscholar.org/CorpusID:262013517.
  * Khashabi et al. [2018] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth.  Looking beyond the surface:a challenge set for reading comprehension over multiple sentences.  In _Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2018.
  * Khashabi et al. [2020] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.  Unifiedqa: Crossing format boundaries with a single qa system.  _arXiv preprint arXiv:2005.00700_ , 2020.
  * Khondaker et al. [2023] Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed.  Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp, 2023.
  * Khot et al. [2020] Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal.  Qasc: A dataset for question answering via sentence composition.  _arXiv:1910.11473v2_ , 2020.
  * Kim et al. [2022] Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi.  Soda: Million-scale dialogue distillation with social commonsense contextualization.  _ArXiv_ , abs/2212.10465, 2022.
  * Kim et al. [2021] Joongwon Kim, Mounica Maddela, Reno Kriz, Wei Xu, and Chris Callison-Burch.  BiSECT: Learning to split and rephrase sentences with bitexts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 6193–6209, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  10.18653/v1/2021.emnlp-main.500.  URL https://aclanthology.org/2021.emnlp-main.500.
  * Kim et al. [2023] Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo.  The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning.  _arXiv preprint arXiv:2305.14045_ , 2023.
  * Köpf et al. [2023] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al.  Openassistant conversations–democratizing large language model alignment.  _arXiv preprint arXiv:2304.07327_ , 2023.
  * Kotek et al. [2023] Hadas Kotek, Rikker Dockum, and David Q. Sun.  Gender bias and stereotypes in large language models.  _Proceedings of The ACM Collective Intelligence Conference_ , 2023.  URL https://api.semanticscholar.org/CorpusID:261276445.
  * Kreutzer et al. [2022] Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72, 2022.  10.1162/tacl_a_00447.  URL https://aclanthology.org/2022.tacl-1.4.
  * Krishna [2020] Venni V. Krishna.  Open science and its enemies: Challenges for a sustainable science–society social contract.  _Journal of Open Innovation: Technology, Market, and Complexity_ , 6(3):61, 2020.  ISSN 2199-8531.  https://doi.org/10.3390/joitmc6030061.
  * Kudugunta et al. [2023] Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, et al.  Madlad-400: A multilingual and document-level large audited dataset.  _arXiv preprint arXiv:2309.04662_ , 2023.
  * Kunchukuttan et al. [2021] Anoop Kunchukuttan, Siddharth Jain, and Rahul Kejriwal.  A large-scale evaluation of neural machine transliteration for Indic languages.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pp. 3469–3475, Online, April 2021. Association for Computational Linguistics.  10.18653/v1/2021.eacl-main.303.  URL https://aclanthology.org/2021.eacl-main.303.
  * Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466, 2019.  10.1162/tacl_a_00276.  URL https://aclanthology.org/Q19-1026.
  * Köpf et al. [2023] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick.  Openassistant conversations – democratizing large language model alignment, 2023.
  * Ladhak et al. [2020] Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 4034–4048, Online, November 2020. Association for Computational Linguistics.  10.18653/v1/2020.findings-emnlp.360.  URL https://aclanthology.org/2020.findings-emnlp.360.
  * Lahoti et al. [2023] Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, and Jilin Chen.  Improving diversity of demographic representation in large language models via collective-critiques and self-voting, 2023.
  * Lai et al. [2017] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy.  RACE: Large-scale ReAding comprehension dataset from examinations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 785–794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  10.18653/v1/D17-1082.  URL https://aclanthology.org/D17-1082.
  * Lai et al. [2023] Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen.  Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.  _arXiv preprint arXiv:2307.16039_ , 2023.
  * Lample & Conneau [2019] Guillaume Lample and Alexis Conneau.  Cross-lingual language model pretraining.  _arXiv preprint arXiv:1901.07291_ , 2019.
  * Laurençon et al. [2022] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al.  The bigscience roots corpus: A 1.6 tb composite multilingual dataset.  _Advances in Neural Information Processing Systems_ , 35:31809–31826, 2022.
  * Lebret et al. [2016] Rémi Lebret, David Grangier, and Michael Auli.  Generating text from structured data with application to the biography domain.  _CoRR_ , abs/1603.07771, 2016.  URL http://arxiv.org/abs/1603.07771.
  * Lee et al. [2023] Nayeon Lee, Chani Jung, and Alice Oh.  Hate speech classifiers are culturally insensitive.  In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_ , pp. 35–46, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.c3nlp-1.5.
  * Lehmann et al. [2014] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, and Christian Bizer.  Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia.  _Semantic Web Journal_ , 6, 01 2014.  10.3233/SW-140134.
  * Lenart-Gansiniec et al. [2023] Regina Lenart-Gansiniec, Wojciech Czakon, Łukasz Sułkowski, and Jasna Pocek.  Understanding crowdsourcing in science.  _Review of Managerial Science_ , 17(8):2797–2830, Nov 2023.  ISSN 1863-6691.  10.1007/s11846-022-00602-z.  URL https://doi.org/10.1007/s11846-022-00602-z.
  * Lent et al. [2022] Heather Lent, Kelechi Ogueji, Miryam de Lhoneux, Orevaoghene Ahia, and Anders Søgaard.  What a creole wants, what a creole needs.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 6439–6449, Marseille, France, June 2022. European Language Resources Association.  URL https://aclanthology.org/2022.lrec-1.691.
  * Levenshtein et al. [1966] Vladimir I Levenshtein et al.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , volume 10, pp. 707–710. Soviet Union, 1966.
  * Lewis et al. [2020] Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk.  Mlqa: Evaluating cross-lingual extractive question answering, 2020.
  * Li et al. [2023a] Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin.  Bactrian-x: A multilingual replicable instruction-following model with low-rank adaptation.  _arXiv preprint arXiv:2305.15011_ , 2023a.
  * Li et al. [2023b] Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, and Yangqiu Song.  Privacy in large language models: Attacks, defenses and future directions.  _ArXiv_ , abs/2310.10383, 2023b.  URL https://api.semanticscholar.org/CorpusID:264145758.
  * Li et al. [2023c] Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu.  M3it: A large-scale dataset towards multi-modal multilingual instruction tuning, 2023c.
  * Li et al. [2023d] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.  Starcoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_ , 2023d.
  * Li & Roth [2002] Xin Li and Dan Roth.  Learning question classifiers.  In _COLING 2002: The 19th International Conference on Computational Linguistics_ , 2002.  URL https://aclanthology.org/C02-1150.
  * Li et al. [2023e] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen.  Making large language models better reasoners with step-aware verifier, 2023e.
  * Li et al. [2022] Yudong Li, Yuqing Zhang, Zhe Zhao, Linlin Shen, Weijie Liu, Weiquan Mao, and Hui Zhang.  Csl: A large-scale chinese scientific literature dataset, 2022.
  * Lignos et al. [2022] Constantine Lignos, Nolan Holley, Chester Palen-Michel, and Jonne Sälevä.  Toward more meaningful resources for lower-resourced languages.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 523–532, Dublin, Ireland, May 2022. Association for Computational Linguistics.  10.18653/v1/2022.findings-acl.44.  URL https://aclanthology.org/2022.findings-acl.44.
  * Lin et al. [2020] Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren.  CommonGen: A constrained text generation challenge for generative commonsense reasoning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 1823–1840, Online, November 2020. Association for Computational Linguistics.  10.18653/v1/2020.findings-emnlp.165.  URL https://aclanthology.org/2020.findings-emnlp.165.
  * Lin et al. [2021] Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, and Xiang Ren.  Common sense beyond English: Evaluating and improving multilingual language models for commonsense reasoning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1274–1287, Online, August 2021. Association for Computational Linguistics.  10.18653/v1/2021.acl-long.102.  URL https://aclanthology.org/2021.acl-long.102.
  * Lin et al. [2019] Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner.  Reasoning over paragraph effects in situations.  In _EMNLP 2019 MRQA Workshop_ , pp. 58, 2019.
  * Lin et al. [2022] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 9019–9052, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.616.
  * Liu et al. [2021] Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott.  Visually grounded reasoning across languages and cultures.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 10467–10485, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  10.18653/v1/2021.emnlp-main.818.  URL https://aclanthology.org/2021.emnlp-main.818.
  * Longpre et al. [2023a] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts.  The flan collection: Designing data and methods for effective instruction tuning.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org, 2023a.
  * Longpre et al. [2023b] Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, and Sara Hooker.  The data provenance initiative: A large scale audit of dataset licensing & attribution in ai, 2023b.
  * Longpre et al. [2023c] Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al.  A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity.  _arXiv preprint arXiv:2305.13169_ , 2023c.
  * Loukissas [2019] Yanni Alexander Loukissas.  _All Data Are Local: Thinking Critically in a Data-Driven Society_.  MIT Press, Cambridge, Massachusetts, 2019.
  * Lowphansirikul et al. [2022] Lalita Lowphansirikul, Charin Polpanumas, Attapol T Rutherford, and Sarana Nutanong.  A large English–Thai parallel corpus from the web and machine-generated text.  _Language Resources and Evaluation_ , 56(2):477–499, 2022.
  * Luccioni & Viviano [2021] Alexandra Luccioni and Joseph Viviano.  What’s in the box? an analysis of undesirable content in the Common Crawl corpus.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pp. 182–189, Online, August 2021. Association for Computational Linguistics.  10.18653/v1/2021.acl-short.24.  URL https://aclanthology.org/2021.acl-short.24.
  * Lucia Specia et al. [2010] Lucia Specia, Nicola Cancedda, and Marc Dymetman.  A Dataset for Assessing Machine Translation Evaluation Metrics.  _International Conference on Language Resources and Evaluation_ , 2010.
  * Lucy et al. [2024] Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, and Jesse Dodge.  Aboutme: Using self-descriptions in webpages to document the effects of english pretraining data filters, 2024.
  * Lukas et al. [2023] Nils Lukas, A. Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-B’eguelin.  Analyzing leakage of personally identifiable information in language models.  _2023 IEEE Symposium on Security and Privacy (SP)_ , pp. 346–363, 2023.  URL https://api.semanticscholar.org/CorpusID:256459554.
  * Luo et al. [2023] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct, 2023.
  * Luukkonen et al. [2023] Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al.  Fingpt: Large generative models for a small language.  _arXiv preprint arXiv:2311.05640_ , 2023.
  * Maas et al. [2011] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pp. 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.  URL http://www.aclweb.org/anthology/P11-1015.
  * Maillard et al. [2023] Jean Maillard, Cynthia Gao, Elahe Kalbassi, Kaushik Ram Sadagopan, Vedanuj Goswami, Philipp Koehn, Angela Fan, and Francisco Guzmán.  Small data, big impact: Leveraging minimal data for effective machine translation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2740–2756, Toronto, Canada, 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.acl-long.154.
  * Marion et al. [2023] Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker.  When less is more: Investigating data pruning for pretraining llms at scale, 2023.
  * Marivate et al. [2020] Vukosi Marivate, Tshephisho Sefara, Vongani Chabalala, Keamogetswe Makhaya, Tumisho Mokgonyane, Rethabile Mokoena, and Abiodun Modupe.  Investigating an approach for low resource language dataset creation, curation and classification: Setswana and sepedi.  In _Proceedings of the first workshop on Resources for African Indigenous Languages_ , pp. 15–20, Marseille, France, May 2020. European Language Resources Association (ELRA).  ISBN 979-10-95546-60-3.  URL https://aclanthology.org/2020.rail-1.3.
  * maxbartolo [2023a] maxbartolo.  AdversarialQA D(BERT).  https://huggingface.co/datasets/adversarial_qa/viewer/dbert, 2023a.  Accessed: 2023-11-28.
  * maxbartolo [2023b] maxbartolo.  AdversarialQA D(BiDAF).  https://huggingface.co/datasets/adversarial_qa/viewer/dbidaf, 2023b.  Accessed: 2023-11-28.
  * maxbartolo [2023c] maxbartolo.  AdversarialQA D(RoBERTa).  https://huggingface.co/datasets/adversarial_qa/viewer/droberta, 2023c.  Accessed: 2023-11-28.
  * Maxwell & Hughes [2006] Mike Maxwell and Baden Hughes.  Frontiers in linguistic annotation for lower-density languages.  In _Proceedings of the workshop on frontiers in linguistically annotated corpora 2006_ , pp. 29–37, 2006.
  * Mayhew et al. [2023] Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Šuppa, Hila Gonen, Joseph Marvin Imperial, Börje F. Karlsson, Peiqin Lin, Nikola Ljubešić, LJ Miranda, Barbara Plank, Arij Riabi, and Yuval Pinter.  Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark.  _arXiv preprint arXiv:2311.09122_ , 2023.
  * McCann et al. [2018] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher.  The natural language decathlon: Multitask learning as question answering.  _arXiv preprint arXiv:1806.08730_ , 2018.
  * Mehrabi et al. [2021] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM computing surveys (CSUR)_ , 54(6):1–35, 2021.
  * Mihaylov et al. [2018] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _EMNLP_ , 2018.
  * Min et al. [2021] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Metaicl: Learning to learn in context.  _arXiv preprint arXiv:2110.15943_ , 2021.
  * Mishra et al. [2022] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.  Cross-task generalization via natural language crowdsourcing instructions.  In _ACL_ , 2022.
  * Moran & Chiarcos [2020] Steven Moran and Christian Chiarcos.  4 linguistic linked open data and under-resourced languages: From collection to application.  _Development of Linguistic Linked Open Data Resources for Collaborative Data-Intensive Research in the Language Sciences_ , pp. 39, 2020.
  * Morschheuser et al. [2017] Benedikt Morschheuser, Juho Hamari, Jonna Koivisto, and Alexander Maedche.  Gamified crowdsourcing: Conceptualization, literature review, and future agenda.  _International Journal of Human-Computer Studies_ , 106:26–43, 2017.  ISSN 1071-5819.  https://doi.org/10.1016/j.ijhcs.2017.04.005.  URL https://www.sciencedirect.com/science/article/pii/S1071581917300642.
  * Muennighoff et al. [2023a] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre.  Octopack: Instruction tuning code large language models.  _arXiv preprint arXiv:2308.07124_ , 2023a.
  * Muennighoff et al. [2023b] Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel.  Scaling data-constrained language models.  _arXiv preprint arXiv:2305.16264_ , 2023b.
  * Muennighoff et al. [2023c] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 15991–16111, Toronto, Canada, July 2023c. Association for Computational Linguistics.  URL https://aclanthology.org/2023.acl-long.891.
  * Muhammad et al. [2023] Shamsuddeen Muhammad, Idris Abdulmumin, Abinew Ayele, Nedjma Ousidhoum, David Adelani, Seid Yimam, Ibrahim Ahmad, Meriem Beloucif, Saif Mohammad, Sebastian Ruder, Oumaima Hourrane, Alipio Jorge, Pavel Brazdil, Felermino Ali, Davis David, Salomey Osei, Bello Shehu-Bello, Falalu Lawan, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Belay, Wendimu Messelle, Hailu Balcha, Sisay Chala, Hagos Gebremichael, Bernard Opoku, and Stephen Arthur.  AfriSenti: A Twitter sentiment analysis benchmark for African languages.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 13968–13981, Singapore, December 2023. Association for Computational Linguistics.  10.18653/v1/2023.emnlp-main.862.  URL https://aclanthology.org/2023.emnlp-main.862.
  * Myers-Scotton [2017] Carol Myers-Scotton.  Code-switching.  _The handbook of sociolinguistics_ , pp. 217–237, 2017.
  * Naik et al. [2023] Ranjita Naik, Varun Chandrasekaran, Mert Yuksekgonul, Hamid Palangi, and Besmira Nushi.  Diversity of thought improves reasoning abilities of large language models, 2023.
  * Nakamura et al. [2023] Gabriel Nakamura, Bruno Soares, Valério Pillar, José Diniz-Filho, and Leandro Duarte.  Three pathways to better recognize the expertise of global south researchers.  _npj Biodiversity_ , 08 2023.  10.1038/s44185-023-00021-7.
  * Nallapati et al. [2016] Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang.  Abstractive text summarization using sequence-to-sequence rnns and beyond, 2016.
  * Naous et al. [2023] Tarek Naous, Michael Joseph Ryan, and Wei Xu.  Having beer after prayer? measuring cultural bias in large language models.  _ArXiv_ , abs/2305.14456, 2023.  URL https://api.semanticscholar.org/CorpusID:258865272.
  * Narayan et al. [2018] Shashi Narayan, Shay B. Cohen, and Mirella Lapata.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  _ArXiv_ , abs/1808.08745, 2018.
  * Nasr et al. [2023] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee.  Scalable extraction of training data from (production) language models, 2023.
  * Nguyen et al. [2023] Huu Nguyen, Sameer Suri, Ken Tsui, and Christoph Schuhmann.  The open instruction generalist (oig) dataset.  https://laion.ai/blog/oig-dataset/, 2023.
  * Ni et al. [2023] Jinjie Ni, Fuzhao Xue, Kabir Jain, Mahir Hitesh Shah, Zangwei Zheng, and Yang You.  Instruction in the wild: A user-based instruction dataset.  https://github.com/XueFuzhao/InstructionWild, 2023.
  * Nicholas & Bhatia [2023] Gabriel Nicholas and Aliya Bhatia.  Lost in translation: Large language models in non-english content analysis, 2023.
  * Niklaus et al. [2023] Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E Ho.  Multilegalpile: A 689gb multilingual legal corpus.  _arXiv preprint arXiv:2306.02069_ , 2023.
  * NLLB-Team et al. [2022] NLLB-Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang.  No language left behind: Scaling human-centered machine translation, 2022.
  * Nogara et al. [2023] Gianluca Nogara, Francesco Pierri, Stefano Cresci, Luca Luceri, Petter Törnberg, and Silvia Giordano.  Toxic bias: Perspective api misreads german as more toxic, 2023.
  * Ogundepo et al. [2023] Odunayo Ogundepo, Tajuddeen Gwadabe, Clara Rivera, Jonathan Clark, Sebastian Ruder, David Adelani, Bonaventure Dossou, Abdou Diop, Claytone Sikasote, Gilles Hacheme, Happy Buzaaba, Ignatius Ezeani, Rooweither Mabuya, Salomey Osei, Chris Emezue, Albert Kahira, Shamsuddeen Muhammad, Akintunde Oladipo, Abraham Owodunni, Atnafu Tonja, Iyanuoluwa Shode, Akari Asai, Anuoluwapo Aremu, Ayodele Awokoya, Bernard Opoku, Chiamaka Chukwuneke, Christine Mwase, Clemencia Siro, Stephen Arthur, Tunde Ajayi, Verrah Otiende, Andre Rubungo, Boyd Sinkala, Daniel Ajisafe, Emeka Onwuegbuzia, Falalu Lawan, Ibrahim Ahmad, Jesujoba Alabi, Chinedu Mbonu, Mofetoluwa Adeyemi, Mofya Phiri, Orevaoghene Ahia, Ruqayya Iro, and Sonia Adhiambo.  Afriqa: Cross-lingual open-retrieval question answering for african languages.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 14957–14972, Singapore, December 2023. Association for Computational Linguistics.  10.18653/v1/2023.findings-emnlp.997.  URL https://aclanthology.org/2023.findings-emnlp.997.
  * Ojo et al. [2023] Jessica Ojo, Kelechi Ogueji, Pontus Stenetorp, and David I. Adelani.  How good are large language models on african languages?, 2023.
  * Ortiz Su’arez et al. [2019] Pedro Javier Ortiz Su’arez, Benoit Sagot, and Laurent Romary.  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.  In Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz, Harald L"ungen, and Caroline Iliadi (eds.), _7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7)_ , Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pp. 9 – 16, Mannheim, 2019. Leibniz-Institut f"ur Deutsche Sprache.  10.14618/ids-pub-9021.  URL http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215.
  * osyvokon [2023] osyvokon.  UA-GEC instruction tuning .  https://huggingface.co/datasets/osyvokon/ua_gec_instruction_tuning, 2023.  Accessed: 2023-11-28.
  * Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022.
  * Pang & Lee [2005] Bo Pang and Lillian Lee.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the ACL_ , 2005.
  * Paperno et al. [2016] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández.  The lambada dataset: Word prediction requiring a broad discourse context, 2016.
  * Park et al. [2023] Michael Park, Erin Leahey, and Russell J. Funk.  Papers and patents are becoming less disruptive over time.  _Nature_ , 613:138–144, 2023.  URL https://api.semanticscholar.org/CorpusID:255466666.
  * Peng et al. [2021] Kenny Peng, Arunesh Mathur, and Arvind Narayanan.  Mitigating dataset harms requires stewardship: Lessons from 1000 papers.  _arXiv preprint arXiv:2108.02922_ , 2021.
  * Perez-Beltrachini & Lapata [2021] Laura Perez-Beltrachini and Mirella Lapata.  Models and datasets for cross-lingual summarisation.  In _Proceedings of The 2021 Conference on Empirical Methods in Natural Language Processing_ , Punta Cana, Dominican Republic, 2021.
  * Petroni et al. [2021] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick S. H. Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel.  KILT: a benchmark for knowledge intensive language tasks.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pp. 2523–2544. Association for Computational Linguistics, 2021.  URL https://www.aclweb.org/anthology/2021.naacl-main.200/.
  * Petty [2023] Martin Petty.  Explainer: Why is Myanmar’s military holding an election?, 2023.  URL https://www.reuters.com/world/asia-pacific/why-is-myanmars-military-holding-an-election-2023-03-29/.  Accessed on Jan. 17, 2024.
  * Pilehvar & Camacho-Collados [2019] Mohammad Taher Pilehvar and Jose Camacho-Collados.  Wic: the word-in-context dataset for evaluating context-sensitive meaning representations, 2019.
  * Pinel C [2020] McKevitt C Pinel C, Prainsack B.  Caring for data: Value creation in a data-intensive research laboratory.  _Social Studies of Science_ , 50(2):175–197, April 2020.
  * Ponti et al. [2020] Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen.  Xcopa: A multilingual dataset for causal commonsense reasoning.  _arXiv preprint arXiv:2005.00333_ , 2020.
  * Popović [2015] Maja Popović.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pp. 392–395, Lisbon, Portugal, September 2015. Association for Computational Linguistics.  10.18653/v1/W15-3049.  URL https://aclanthology.org/W15-3049.
  * Post [2018] Matt Post.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pp. 186–191, Brussels, Belgium, October 2018. Association for Computational Linguistics.  10.18653/v1/W18-6319.  URL https://aclanthology.org/W18-6319.
  * Pratapa et al. [2022] Adithya Pratapa, Rishubh Gupta, and Teruko Mitamura.  Multilingual event linking to Wikidata.  In _Proceedings of the Workshop on Multilingual Information Access (MIA)_ , pp. 37–58, Seattle, USA, July 2022. Association for Computational Linguistics.  10.18653/v1/2022.mia-1.5.  URL https://aclanthology.org/2022.mia-1.5.
  * Pudjiati et al. [2022] Danti Pudjiati, Ninuk Lustyantie, Ifan Iskandar, and Tira Nur Fitria.  Post-editing of machine translation: Creating a better translation of cultural specific terms.  _Language Circle: Journal of Language and Literature_ , 17(1):61–73, 2022.
  * Puschmann & Burgess [2014] Cornelius Puschmann and Jean Burgess.  Big data, big questions| metaphors of big data.  _International Journal of Communication_ , 8(0), 2014.
  * Pushkarna et al. [2022] Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson.  Data cards: Purposeful and transparent dataset documentation for responsible ai.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, pp. 1776–1826, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450393522.  10.1145/3531146.3533231.  URL https://doi.org/10.1145/3531146.3533231.
  * PyThaiNLP [2023a] PyThaiNLP.  scb_mt_2020_en2th_prompt.  https://huggingface.co/datasets/pythainlp/scb_mt_2020_en2th_prompt, 2023a.  Accessed: 2023-11-29.
  * PyThaiNLP [2023b] PyThaiNLP.  scb_mt_2020_th2en_prompt.  https://huggingface.co/datasets/pythainlp/scb_mt_2020_th2en_prompt, 2023b.  Accessed: 2023-11-29.
  * PyThaiNLP [2023c] PyThaiNLP.  Thai-Pos-prompt.  https://huggingface.co/datasets/pythainlp/Thai-Pos-prompt, 2023c.  Accessed: 2023-11-29.
  * PyThaiNLP [2023d] PyThaiNLP.  thai_usembassy_en2th_prompt.  https://huggingface.co/datasets/pythainlp/thai_usembassy_en2th_prompt, 2023d.  Accessed: 2023-11-29.
  * PyThaiNLP [2023e] PyThaiNLP.  thai_usembassy_th2en_prompt.  https://huggingface.co/datasets/pythainlp/thai_usembassy_th2en_prompt, 2023e.  Accessed: 2023-11-29.
  * PyThaiNLP [2023f] PyThaiNLP.  thai-wiktionary-prompt.  https://huggingface.co/datasets/pythainlp/thai-wiktionary-prompt, 2023f.  Accessed: 2023-11-29.
  * Raganato et al. [2020] Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar.  XL-WiC: A multilingual benchmark for evaluating semantic contextualization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 7193–7206, Online, November 2020. Association for Computational Linguistics.  10.18653/v1/2020.emnlp-main.584.  URL https://aclanthology.org/2020.emnlp-main.584.
  * Rajani et al. [2019] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher.  Explain yourself! leveraging language models for commonsense reasoning.  In _Proceedings of the 2019 Conference of the Association for Computational Linguistics (ACL2019)_ , 2019.  URL https://arxiv.org/abs/1906.02361.
  * Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  _arXiv e-prints_ , art. arXiv:1606.05250, 2016.
  * Reddy et al. [2019] Siva Reddy, Danqi Chen, and Christopher D. Manning.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266, 2019.  10.1162/tacl_a_00266.  URL https://aclanthology.org/Q19-1016.
  * Reimers & Gurevych [2019] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ , 2019.
  * Reuters [2023] Reuters.  Explainer: What is happening between Armenia and Azerbaijan over Nagorno-Karabakh?, 2023.  URL https://www.reuters.com/world/what-is-happening-between-armenia-azerbaijan-over-nagorno-karabakh-2023-09-19/.  Accessed on Jan. 17, 2024.
  * Rickford et al. [2012] John R Rickford, Julie Sweetland, Angela E Rickford, and Thomas Grano.  _African American, Creole, and other vernacular Englishes in education: A bibliographic resource_.  Routledge, 2012.
  * Robinson et al. [2023] Nathaniel R. Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig.  Chatgpt mt: Competitive for high- (but not low-) resource languages.  _ArXiv_ , abs/2309.07423, 2023.  URL https://api.semanticscholar.org/CorpusID:261824661.
  * Rogers et al. [2020] Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky.  Getting closer to AI complete question answering: A set of prerequisite real tasks.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pp. 8722–8731. AAAI Press, 2020.  URL https://aaai.org/ojs/index.php/AAAI/article/view/6398.
  * Rooy [2021] Raf Van Rooy.  _Language or Dialect? The History of a Conceptual Pair_.  Oxford University Press, 2021.
  * Rush et al. [2015] Alexander M. Rush, Sumit Chopra, and Jason Weston.  A neural attention model for abstractive sentence summarization.  _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2015.  10.18653/v1/d15-1044.  URL http://dx.doi.org/10.18653/v1/D15-1044.
  * Saad-Falcon et al. [2023] Jon Saad-Falcon, Joe Barrow, Alexa Siu, Ani Nenkova, Ryan A Rossi, and Franck Dernoncourt.  Pdftriage: Question answering over long, structured documents.  _arXiv preprint arXiv:2309.08872_ , 2023.
  * Sabou et al. [2012] Marta Sabou, Kalina Bontcheva, and Arno Scharl.  Crowdsourcing research opportunities: Lessons from natural language processing.  In _Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies_ , i-KNOW ’12, New York, NY, USA, 2012. Association for Computing Machinery.  ISBN 9781450312424.  10.1145/2362456.2362479.  URL https://doi.org/10.1145/2362456.2362479.
  * Saha et al. [2018] Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan.  DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension.  In _Meeting of the Association for Computational Linguistics (ACL)_ , 2018.
  * Sambasivan et al. [2021] Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo.  “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , CHI ’21, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450380966.  10.1145/3411764.3445518.  URL https://doi.org/10.1145/3411764.3445518.
  * Sanh et al. [2022] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=9Vrb9D0WI4.
  * Sap et al. [2019] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.  Socialiqa: Commonsense reasoning about social interactions, 2019.
  * Savoldi et al. [2021] Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi.  Gender bias in machine translation, 2021.
  * Scao et al. [2022a] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_ , 2022a.
  * Scao et al. [2022b] Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al.  What language model to train if you have one million gpu hours?  _arXiv preprint arXiv:2210.15424_ , 2022b.
  * Schwartz et al. [2022] Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, Patrick Hall, et al.  Towards a standard for identifying and managing bias in artificial intelligence.  _NIST special publication_ , 1270(10.6028), 2022.
  * Seaver [2021] Nick Seaver.  Care and scale: Decorrelative ethics in algorithmic recommendation.  _Cultural Anthropology_ , 36(3):509–537, 2021.
  * Secretariat [2022] UN Secretariat.  International decade of indigenous languages, 2022–2032: Global action plan: note/by the secretariat, 2022.
  * See et al. [2017] Abigail See, Peter J. Liu, and Christopher D. Manning.  Get to the point: Summarization with pointer-generator networks.  _CoRR_ , abs/1704.04368, 2017.  URL http://arxiv.org/abs/1704.04368.
  * Sen et al. [2022] Priyanka Sen, Alham Fikri Aji, and Amir Saffari.  Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pp. 1604–1619, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.  URL https://aclanthology.org/2022.coling-1.138.
  * Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.  10.18653/v1/P16-1162.  URL https://aclanthology.org/P16-1162.
  * Shafagh [2023a] Shafagh.  Aya Persian Instruction pn Summary.  https://huggingface.co/datasets/Shafagh/aya_persian_instruction_pn-summary, 2023a.  Accessed: 2023-11-28.
  * Shafagh [2023b] Shafagh.  Aya Persian Instruction pn Summary Title.  https://huggingface.co/datasets/Shafagh/aya_persian_instruction_pn-summary-title, 2023b.  Accessed: 2023-11-28.
  * Shao et al. [2019] Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai.  Drcd: a chinese machine reading comprehension dataset, 2019.
  * Sidnell & Enfield [2012] Jack Sidnell and N. J. Enfield.  Language diversity and social action: A third locus of linguistic relativity.  _Current Anthropology_ , 53(3):302–333, 2012.
  * Siminyu et al. [2021] Kathleen Siminyu, Godson Kalipe, Davor Orlic, Jade Z. Abbott, Vukosi Marivate, Sackey Freshia, Prateek Sibal, Bhanu Neupane, David Ifeoluwa Adelani, Amelia V. Taylor, Jamiil Toure Ali, Kevin Degila, Momboladji Balogoun, Thierno Ibrahima Diop, Davis David, Chayma Fourati, Hatem Haddad, and Malek Naski.  AI4D - african language program.  In Kathleen Siminyu, Julia Kreutzer, Hady Elsahar, Vukosi Marivate, Nishant Subramani, Jade Z. Abbott, and Bernardt Duvenhage (eds.), _2nd AfricaNLP Workshop Proceedings, AfricaNLP@EACL 2021, Virtual Event, April 19, 2021_ , 2021.  URL https://arxiv.org/abs/2104.02516.
  * Simons [2019] Gary F. Simons.  Two centuries of spreading language loss.  _Proceedings of the Linguistic Society of America_ , 4(1):27:1–12, Mar. 2019.  10.3765/plsa.v4i1.4532.  URL https://journals.linguisticsociety.org/proceedings/index.php/PLSA/article/view/4532.
  * Singh et al. [2022] Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman.  SciRepEval: A Multi-Format Benchmark for Scientific Document Representations.  _ArXiv_ , abs/2211.13308, 2022.
  * Soldaini et al. [2024] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Pete Walsh, Hannaneh Hajishirzi, Noah A. Smith, Luke Zettlemoyer, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo.  Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research.  _arXiv preprint_ , 2024.
  * Specia & Farzindar [2010] Lucia Specia and Atefeh Farzindar.  Estimating machine translation post-editing effort with HTER.  In _Proceedings of the Second Joint EM+/CNGL Workshop: Bringing MT to the User: Research on Integrating MT in the Translation Industry_ , pp. 33–43, Denver, Colorado, USA, November 4 2010. Association for Machine Translation in the Americas.  URL https://aclanthology.org/2010.jec-1.5.
  * Spolsky [2018] Bernard Spolsky.  Language policy in french colonies and after independence.  _Current Issues in Language Planning_ , 19(3):231–315, 2018.  10.1080/14664208.2018.1444948.
  * Srivastava & Singh [2021] Vivek Srivastava and Mayank Singh.  Challenges and considerations with code-mixed nlp for multilingual societies, 2021.
  * Stark & Hoffmann [2019] Luke Stark and Anna Lauren Hoffmann.  Data is the new what? popular metaphors & professional ethics in emerging data culture.  _Journal of Cultural Analytics_ , 4(1), 2019.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Strassel & Tracey [2016] Stephanie Strassel and Jennifer Tracey.  LORELEI language packs: Data, tools, and resources for technology development in low resource languages.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pp. 3273–3280, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA).  URL https://aclanthology.org/L16-1521.
  * Sun et al. [2023] Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, and Sebastian Gehrmann.  Dialect-robust evaluation of generated text.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 6010–6028, Toronto, Canada, July 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.acl-long.331.
  * SuryaKrishna02 [2023a] SuryaKrishna02.  Aya Telugu Food Recipes.  https://huggingface.co/datasets/SuryaKrishna02/aya-telugu-food-recipes, 2023a.  Accessed: 2023-11-28.
  * SuryaKrishna02 [2023b] SuryaKrishna02.  Aya Telugu Jokes.  https://huggingface.co/datasets/SuryaKrishna02/aya-telugu-jokes, 2023b.  Accessed: 2023-11-28.
  * SuryaKrishna02 [2023c] SuryaKrishna02.  Aya Telugu News Articles.  https://huggingface.co/datasets/SuryaKrishna02/aya-telugu-news-articles, 2023c.  Accessed: 2023-11-28.
  * SuryaKrishna02 [2023d] SuryaKrishna02.  Aya Telugu Paraphrase.  https://huggingface.co/datasets/SuryaKrishna02/aya-telugu-paraphrase, 2023d.  Accessed: 2023-11-28.
  * SuryaKrishna02 [2023e] SuryaKrishna02.  Aya Telugu Poems.  https://huggingface.co/datasets/SuryaKrishna02/aya-telugu-poems, 2023e.  Accessed: 2023-11-28.
  * Suzuki et al. [2023] Masahiro Suzuki, Masanori Hirano, and Hiroki Sakaji.  From base to conversational: Japanese instruction dataset and tuning large language models.  _arXiv preprint arXiv:2309.03412_ , 2023.
  * syntaxshill [2023] syntaxshill.  Arpa aya.  https://huggingface.co/datasets/syntaxshill/arpa-aya, 2023.  Accessed: 2023-11-28.
  * Syvokon et al. [2023] Oleksiy Syvokon, Olena Nahorna, Pavlo Kuchmiichuk, and Nastasiia Osidach.  UA-GEC: Grammatical error correction and fluency corpus for the Ukrainian language.  In _Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP)_ , pp. 96–102, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.unlp-1.12.
  * Tafjord et al. [2019a] Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal.  Quarel: A dataset and models for answering questions about qualitative relationships.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 7063–7071, 2019a.
  * Tafjord et al. [2019b] Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark.  Quartz: An open-domain dataset of qualitative relationship questions.  _arXiv preprint arXiv:1909.03553_ , 2019b.
  * TahmidH [2023] TahmidH.  Annotated News Summary.  https://huggingface.co/datasets/TahmidH/annotated_news_summary, 2023.  Accessed: 2023-11-28.
  * Tandon et al. [2019] Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, and Peter Clark.  Wiqa: A dataset for "what if…" reasoning over procedural text.  _arXiv:1909.04739v1_ , 2019.
  * Tang et al. [2021] Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan.  Multilingual translation from denoising pre-training.  _Findings of the Association for Computational Linguistics: ACL-IJCNLP_ , 2021.  10.18653/v1/2021.findings-acl.304.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Tellarin.ai [2023a] Tellarin.ai.  LLM Japanese Dataset Vanilla Aya Format.  https://huggingface.co/datasets/tellarin-ai/llm-japanese-dataset-vanilla-aya-format, 2023a.  Accessed: 2023-11-28.
  * Tellarin.ai [2023b] Tellarin.ai.  NTX LLM Instructions.  https://huggingface.co/datasets/tellarin-ai/ntx_llm_instructions, 2023b.  Accessed: 2023-11-28.
  * theblackcat102 [2023] theblackcat102.  Joke explaination.  https://huggingface.co/datasets/theblackcat102/joke_explaination, 2023.  Accessed: 2023-11-29.
  * Tiedemann [2012] Jörg Tiedemann.  Parallel data, tools and interfaces in opus.  In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis (eds.), _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey, may 2012. European Language Resources Association (ELRA).  ISBN 978-2-9517408-7-7.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Tu et al. [2019] Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, and Bowen Zhou.  Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs, 2019.
  * TurkuNLP [2023] TurkuNLP.  Turku Paraphrase Corpus.  https://huggingface.co/datasets/TurkuNLP/turku_paraphrase_corpus, 2023.  Accessed: 2023-11-28.
  * Universal NER [2023] Universal NER.  UNER LLM Instructions.  https://huggingface.co/datasets/universalner/uner_llm_instructions, 2023.  Accessed: 2023-11-28.
  * Urbizu et al. [2023] Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, and Ander Corral.  Not enough data to pre-train your language model? MT to the rescue!  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 3826–3836, Toronto, Canada, July 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.findings-acl.235.
  * Vanmassenhove et al. [2021] Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam.  Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pp. 2203–2213, Online, April 2021. Association for Computational Linguistics.  10.18653/v1/2021.eacl-main.188.  URL https://aclanthology.org/2021.eacl-main.188.
  * Vashishtha et al. [2023] Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram.  On evaluating and mitigating gender biases in multilingual settings, 2023.
  * Vigouroux [2013] Cécile B. Vigouroux.  Francophonie.  _Annual Review of Anthropology_ , 42(1):379–397, 2013.  10.1146/annurev-anthro-092611-145804.  URL https://doi.org/10.1146/annurev-anthro-092611-145804.
  * Wan et al. [2023] Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein.  Poisoning language models during instruction tuning, 2023.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. [2019] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_ , 2019.
  * Wang et al. [2023] Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, and Xiaohu Li.  Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation, 2023.
  * Wang et al. [2022a] Jun Wang, Benjamin Rubinstein, and Trevor Cohn.  Measuring and mitigating name biases in neural machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2576–2590, Dublin, Ireland, May 2022a. Association for Computational Linguistics.  10.18653/v1/2022.acl-long.184.  URL https://aclanthology.org/2022.acl-long.184.
  * Wang et al. [2022b] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language model with self generated instructions.  _ArXiv preprint_ , abs/2212.10560, 2022b.  URL https://arxiv.org/abs/2212.10560.
  * Wang et al. [2022c] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi.  Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, 2022c.
  * Wang et al. [2022d] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al.  Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.  _arXiv preprint arXiv:2204.07705_ , 2022d.
  * Wang et al. [2022e] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 5085–5109, Abu Dhabi, United Arab Emirates, December 2022e. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.340.
  * Warstadt et al. [2018] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.  Neural network acceptability judgments.  _arXiv preprint arXiv:1805.12471_ , 2018.
  * Wei et al. [2022a] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2022a.  URL https://openreview.net/forum?id=gEZrGCozdqR.
  * Wei et al. [2022b] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022b.
  * Wei et al. [2023] Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al.  Polylm: An open source polyglot large language model.  _arXiv preprint arXiv:2307.06018_ , 2023.
  * Welbl et al. [2017] Johannes Welbl, Nelson F Liu, and Matt Gardner.  Crowdsourcing multiple choice science questions.  _arXiv preprint arXiv:1707.06209_ , 2017.
  * West et al. [2020] Richard E West, Timothy Newby, Zui Cheng, Alyssa Erickson, and Kyle Clements.  Acknowledging all learning: Alternative, micro, and open credentials.  _Handbook of Research in Educational Communications and Technology: Learning Design_ , pp. 593–613, 2020.
  * Whitehouse et al. [2023] Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji.  Llm-powered data augmentation for enhanced crosslingual performance.  _arXiv preprint arXiv:2305.14288_ , 2023.
  * Willis [1998] W Bruce Willis.  _The Adinkra dictionary: A visual primer on the language of Adinkra_.  Pyramid Complex, 1998.
  * Wilson [2023] Joseph Wilson.  Voicing an algorithm: trials of strength in artificial intelligence research.  _Anthropology News_ , 2023.  URL https://www.anthropology-news.org/articles/combat-by-algorithm-trials-of-strength-in-artificial-intelligence-research/.
  * Winata et al. [2023a] Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio.  The decades progress on code-switching research in NLP: A systematic survey on trends and challenges.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 2936–2978, Toronto, Canada, July 2023a. Association for Computational Linguistics.  URL https://aclanthology.org/2023.findings-acl.185.
  * Winata et al. [2023b] Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder.  NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pp. 815–834, Dubrovnik, Croatia, May 2023b. Association for Computational Linguistics.  URL https://aclanthology.org/2023.eacl-main.57.
  * Wittenburg [2021] Peter Wittenburg.  Open Science and Data Science.  _Data Intelligence_ , 3(1):95–105, 02 2021.
  * Witteveen & Andrews [2019] Sam Witteveen and Martin Andrews.  Paraphrasing with large language models.  _arXiv preprint arXiv:1911.09661_ , 2019.
  * Wolfram [1997] Walt Wolfram.  Issues in dialect obsolescence: An introduction.  _American speech_ , 72(1):3–11, 1997.
  * Wu et al. [2007] Chorng-Guang Wu, James H Gerlach, and Clifford E Young.  An empirical analysis of open source software developers’ motivations and continuance intentions.  _Information & Management_, 44(3):253–262, 2007.
  * Wu et al. [2021] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano.  Recursively summarizing books with human feedback.  _arXiv preprint arXiv:2109.10862_ , 2021.
  * Xie et al. [2022] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al.  Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  _arXiv preprint arXiv:2201.05966_ , 2022.
  * Xu et al. [2023a] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions, 2023a.
  * Xu et al. [2023b] Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen.  Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models, 2023b.
  * Xu et al. [2020] Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan.  CLUE: A Chinese language understanding evaluation benchmark.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pp. 4762–4772, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  10.18653/v1/2020.coling-main.419.  URL https://aclanthology.org/2020.coling-main.419.
  * Xue et al. [2021] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 483–498, Online, June 2021. Association for Computational Linguistics.  10.18653/v1/2021.naacl-main.41.  URL https://aclanthology.org/2021.naacl-main.41.
  * Yang et al. [2023] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li.  Dataset pruning: Reducing training data by examining generalization influence, 2023.
  * Yang et al. [2015] Yi Yang, Wen-tau Yih, and Christopher Meek.  WikiQA: A challenge dataset for open-domain question answering.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pp. 2013–2018, Lisbon, Portugal, September 2015. Association for Computational Linguistics.  10.18653/v1/D15-1237.  URL https://aclanthology.org/D15-1237.
  * Yang et al. [2019] Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge.  PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification.  In _Proc. of EMNLP_ , 2019.
  * Yang et al. [2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2018.
  * Ye et al. [2021] Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren.  CrossFit: A few-shot learning challenge for cross-task generalization in NLP.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 7163–7189, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.  10.18653/v1/2021.emnlp-main.572.  URL https://aclanthology.org/2021.emnlp-main.572.
  * Yong et al. [2023a] Zheng Xin Yong, Cristina Menghini, and Stephen Bach.  Low-resource languages jailbreak GPT-4.  In _Socially Responsible Language Modelling Research_ , 2023a.  URL https://openreview.net/forum?id=pn83r8V2sv.
  * Yong et al. [2023b] Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina.  BLOOM+1: Adding language support to BLOOM for zero-shot prompting.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 11682–11703, Toronto, Canada, July 2023b. Association for Computational Linguistics.  URL https://aclanthology.org/2023.acl-long.653.
  * Yong et al. [2023c] Zheng Xin Yong, Ruochen Zhang, Jessica Zosa Forde, Skyler Wang, Arjun Subramonian, Holy Lovenia, Samuel Cahyawijaya, Genta Indra Winata, Lintang Sutawika, Jan Christian Blaise Cruz, et al.  Prompting multilingual large language models to generate code-mixed texts: The case of south east asian languages.  In _Sixth Workshop on Computational Approaches to Linguistic Code-Switching_ , 2023c.
  * Yu et al. [2023] Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang.  Large language model as attributed training data generator: A tale of diversity and bias, 2023.
  * Yuan et al. [2021] Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann.  Synthbio: A case study in human-ai collaborative curation of text datasets.  _CoRR_ , abs/2111.06467, 2021.  URL https://arxiv.org/abs/2111.06467.
  * Yusuf [2022] Tajudeen Yusuf.  Politeness in arabic and yoruba: Personal pronouns as a case study.  _Asian Journal of Language, Literature and Culture Studies_ , 5(2):82–88, 2022.
  * Zampieri et al. [2017] Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli.  Findings of the VarDial evaluation campaign 2017.  In _Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)_ , pp. 1–15, Valencia, Spain, April 2017. Association for Computational Linguistics.  10.18653/v1/W17-1201.  URL https://aclanthology.org/W17-1201.
  * Zampieri et al. [2020] Marcos Zampieri, Preslav Nakov, and Yves Scherrer.  Natural language processing for similar languages, varieties, and dialects: A survey.  _Natural Language Engineering_ , 26(6):595–612, 2020.
  * Zhang et al. [2022] Daniel Zhang, Nestor Maslej, Erik Brynjolfsson, John Etchemendy, Terah Lyons, James Manyika, Helen Ngo, Juan Carlos Niebles, Michael Sellitto, Ellie Sakhaee, Yoav Shoham, Jack Clark, and Raymond Perrault.  The ai index 2022 annual report, 2022.
  * Zhang et al. [2023a] Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, Wenhao Huang, and Jie Fu.  Chinese open instruction generalist: A preliminary release, 2023a.
  * Zhang et al. [2023b] Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng.  Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models, 2023b.
  * Zhang et al. [2018a] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.  Record: Bridging the gap between human and machine commonsense reading comprehension, 2018a.
  * Zhang et al. [2023c] Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing.  M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models, 2023c.
  * Zhang et al. [2015] Xiang Zhang, Junbo Zhao, and Yann LeCun.  Character-level convolutional networks for text classification.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc., 2015.  URL https://proceedings.neurips.cc/paper_files/paper/2015/file/250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf.
  * Zhang et al. [2019] Yuan Zhang, Jason Baldridge, and Luheng He.  Paws: Paraphrase adversaries from word scrambling, 2019.
  * Zhang et al. [2018b] Zhirui Zhang, Shujie Liu, Mu Li, Ming Zhou, and Enhong Chen.  Joint training for neural machine translation models with monolingual data.  _CoRR_ , abs/1803.00353, 2018b.  URL http://arxiv.org/abs/1803.00353.
  * Zhou et al. [2023] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_ , 2023.
  * Zhuo et al. [2024] Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff.  Astraios: Parameter-efficient instruction tuning code large language models.  _arXiv preprint arXiv:2401.00788_ , 2024.
  * Ziegler et al. [2020] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences, 2020.
  * Zoph et al. [2016] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight.  Transfer learning for low-resource neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pp. 1568–1575, Austin, Texas, November 2016. Association for Computational Linguistics.  10.18653/v1/D16-1163.  URL https://aclanthology.org/D16-1163.

##  Appendix A Annotation Platform

In this section, we discuss the detailed design and development of the
Annotation Platform and the gamification strategy employed. Together, these
attempts aimed to ensure high-quality curation of the Dataset.

Figure 17: The onboarding page for a new user. We collect some demographic
information and ask them to specify the languages they are comfortable
annotating data in. These are the language options they are presented with
later on in the UI.

###  A.1 Engagement Strategies

We decided to employ gamification methods to enhance annotator engagement [de
Franga et al., 2015; Bastanfard et al., 2023; Morschheuser et al., 2017]. Our
strategy involved using a points-based rewards system, motivating contributors
through social media recognition, and fostering friendly competition with
leaderboards. Regular mini-challenges and sprints helped to create collective
achievement goals and fostered a sense of community [Bastanfard et al., 2023].
Real-time feedback reinforced positive behavior and customization options,
such as avatars, provided a personalized experience. Overall, these
gamification strategies aimed to boost engagement, improve data quality, and
created a more enjoyable experience for the crowd-sourcing participants.
[Morschheuser et al., 2017].

To recognize and incentivize the efforts of our contributors, we established a
tiered reward system based on contribution milestones: 500, 1,000, and 5,000
contributions. Contributors who achieved these goals on the project
leaderboard were rewarded with certificates and specially designed project
apparel. The attire varied according to the contribution level, with different
packages for each milestone. Additionally, the most active contributors were
prominently acknowledged in the project’s dataset documentation, highlighting
their significant role as key contributors to the project. This system not
only motivated contributors but also served as a token of appreciation for
their dedication and hard work.

In addition to the leaderboard, the Discord Bot was developed to recognize
contributors with a high number of points. This bot recognized the daily top
10 contributors by tagging them in a message that was posted on the Discord
server. It also aggregated daily total contributions per region and specified
how many days were left until the data collection phase ended. As shown in
Figure 18, these messages provided a regular snapshot of progression that
allowed annotators to see the dataset grow across all languages.

Figure 18: Aya Discord Bot message aggregating daily statistics and top 10
annotators

Figure 19: Leaderboard. Daily and weekly leaderboards are shown on the top
left and right respectively. At the bottom are language-specific and overall
leader boards, where annotators are ranked based on their score.

###  A.2 Accessibility of Registration Tools

The accessibility and popularity of registration tools differed from country
to country and this had an impact on where the UI users joined from. Figure 20
compares the percentage of registered users using Discord and Gmail to sign up
in the top 10 countries. After introducing Google SSO, we observed a
significant jump in the number of registered users from several new countries
(Figure 21).

Figure 20: Percentage of UI users that registered via Left: Discord and Right:
Gmail. Figure 21: Percentage of the UI users that registered after the
introduction of Google SSO in the platform.

##  Appendix B UI Tasks

###  B.1 Reviewing Annotators

In the UI, we display the original and re-annotated versions of both prompt
and completion along with the name of the annotator who did the re-annotation.
Reviewers are tasked with rating the re-annotated prompt and completion on a
scale from 1 to 5. The ratings are defined as follows:

  1. (1)

Much worse than the original annotation;

  2. (2)

Worse than the original annotation;

  3. (3)

No noticeable difference compared to the original annotation;

  4. (4)

Better than the original annotation;

  5. (5)

Much better than the original annotation.

If any of the prompt/completion pairs receive a rating below 5 (i.e., anything
lower than “Much better than the original annotation”), the reviewer is
provided with an option to modify the provided prompt and completion pair to
improve its quality. An “Additional Feedback” text box is also available for
reviewers to explain the reasoning behind their chosen rating.

###  B.2 Aya Score

To encourage annotators to make high-quality submissions, we designed a
ranking score favoring contributions that received high peer ratings. The
Score for an individual annotator is:

$\text{Score}_{\aya}=w_{1}\times(\mathcal{E})+w_{2}\times\mathcal{C},$ (1)

with weights $w_{1}$, $w_{2}$ defined as:

$w_{1}=\max\left(0,(\hat{Q}-3)\right),w_{2}=\left(\frac{\mathcal{T_{+}}}{\mathcal{T}}\right)$
(2)

where:

  * •

$\mathcal{E}$ represents the total number of re-annotations in which the
annotator edited prompts and completions

  * •

$\mathcal{C}$ represents the total number of original annotations submitted by
an annotator

  * •

$\hat{Q}$ represents the average quality rating derived for an annotator
through peer review via Annotation Feedback task

  * •

$\mathcal{T_{+}}$ represents the total number of original annotations made by
an annotator that received positive feedback in Re-annotations task

  * •

$\mathcal{T}$ represents the total number original annotations made by an
annotator that feedback received in Re-annotations task.

The rationale behind introducing the $\text{Score}_{\aya}$ was to boost a
competitive environment among annotators and encourage them to focus on
submitting high-quality data, consequently improving the overall quality of
the Dataset.

###  B.3 Annotation Guidelines

The annotators were provided with the following evaluation criteria for what a
good prompt and completion pair must look like. Re-Annotations were then
performed if they determined that the prompts or completions needed editing.

  1. 1.

No grammatical or spelling mistakes in both the prompt and completion text.

  2. 2.

The prompt provides clear instruction on what the task is.

  3. 3.

The completion answers the prompt correctly. Both the prompts and completions
should be in full sentences and coherent, with reasonable length.

  4. 4.

For original annotations, the prompts and completions should not be generated
from other language models.

Re-Annotations  Before editing, annotators rated the quality of existing
prompt and completion pairs by choosing either the thumbs-up or thumbs-down
option. If the provided prompt and completion pair were already of good
quality according to the criteria above, then annotators rated them with
thumbs up and moved ahead without editing. Overall, annotators were encouraged
to re-annotate the completions, in particular by adding more details and
context to them since many of them were often short one-word answers.

##  Appendix C Language Representation via Community

###  C.1 Division by Regions

We chose to divide languages into four primary regions: Africa, Asia, Europe,
and Latin America. These four regions were established in order to facilitate
the administration of user contributions and were not intended to prescribe
boundaries within which certain languages are exclusively spoken.

The language statistics by region are as follows: Africa (14 languages), Asia
(41 languages), Europe (42 languages), and Latin America (4 languages). Almost
all the languages were assigned to a region but there are some exceptions,
Maori and Samoan were unassigned to any specific region as they didn’t align
with the predefined regions. English was left unassigned, serving as a common
language across all regions. Additionally, contributions in Spanish and
Portuguese were distributed between Europe and Latin America based on
contributors’ countries. Similarly, Arabic contributions were shared between
Africa and Asia depending on the contributors’ country of origin. Additional
dialects of Arabic were included in regions separate from that of their parent
language because we had a significant number of speakers from these regions
eager to contribute to their respective dialects. Each region had at least one
“Regional Lead” responsible for coordinating “Language Ambassadors,” and for
recruiting fluent speakers for the languages within their area.

###  C.2 Language Ambassadors

The Language Ambassador’s role was pivotal in bridging the gap between the
data collected in a language and its speakers. An essential criterion for
selection was native fluency in the specific language. The Language
Ambassador’s expertise in specific languages and familiarity with the cultures
of the language speakers was invaluable. They assisted not only in spreading
awareness among participants but also in identifying and addressing potential
data issues specific to each language, such as languages incorrectly assigned
to their region. Their cultural and linguistic insights enabled them to make
informed decisions, like choosing suitable data sources for collection in
their respective languages. Not every language had a designated Language
Ambassador, and some had more than one. In total, we had 84 Language
Ambassadors over the course of the initiative. Their combined efforts played a
vital role in broadening the contributor base for each language. Support for
the Language Ambassadors’ progress and trouble-shooting challenges they faced
was coordinated asynchronously and through weekly online meetups, discussed in
Sec. D.1 and Sec. D.2.

###  C.3 Regional Leads

There were a total of six Regional Leads: two for Latin America, one for
Africa, one for Asia, and two for Europe. The selection for Regional Lead
roles was on a voluntary basis, with the only requirement being that they must
originate from the regions they intended to lead. The invitation for this role
was specifically extended to individuals who were already actively
participating in Cohere For AI projects or engaged in other open science
projects. Regional Leads had several key roles throughout the project, such as
selecting Language Ambassadors and aiding their efforts in attracting more
annotators and maintaining their engagement.

##  Appendix D Communication

###  D.1 Platforms

We established a Discord server for coordination between Regional Leads,
Language Ambassadors and annotators. The server provided basic channels for
internal communications: introductions, inquiries, and announcements, as well
as specific channels for Language Ambassadors, for each region, and for each
language, along with any other channels that proved useful for the particular
region. For external communications, we used social media platforms (e.g., X,
LinkedIn, WhatsApp, Facebook), recognizing that the choice of communication
platform varied based on cultural and regional preferences. Using multiple
platforms not only facilitated internal organization but also broadened our
project’s outreach by providing flexible and inclusive means of outreach to
diverse communities and audiences.

###  D.2 Meetings

In addition to asynchronous communication through Discord, we conducted
meetings to maintain team collaboration and cohesion:

  1. 1.

Regional Leads and Language Ambassadors Meeting: A weekly meeting in which
Regional Leads and Language Ambassadors shared project updates, exchanged
ideas, and addressed questions from Language Ambassadors. It served as an
excellent platform for gathering ideas from Language Ambassadors and
brainstorming new strategies to engage annotators effectively.

  2. 2.

New Contributor Introduction Meeting: Held weekly, this meeting aimed to
introduce new contributors to the project’s specifics. It included
explanations about the motivations behind the project, a walk-through of the
UI, and a sharing of regional statistics. Additionally, this meeting provided
examples of both good and bad annotations and edits to guide new annotators in
their work. It concluded with a synchronous challenge for the annotators to
submit a few initial annotations in real time to familiarize them with the
process and allow them to ask questions if they got stuck.

  3. 3.

Regional Leads Meeting: Held bi-weekly, this meeting brought together Regional
Leads to assess progress, discuss upcoming steps, and provide advice on how to
engage and sustain contributions for their respective regions. Furthermore,
this meeting facilitated collaborative troubleshooting efforts and helped make
important decisions for the following week.

  4. 4.

Technical Update: This meeting was dedicated to sharing technical updates,
with a focus on recent UI progress, data, and benchmarking. The purpose of
this monthly update was to ensure all team members and annotators were well-
informed about the project’s current status and upcoming priorities. It was a
place for open discussion to hear feedback from everyone interested in the
project.

  5. 5.

Language Specific Meeting: Held weekly or biweekly, these meetings were co-
working sessions or datathons led by the language ambassadors with their
respective annotators to submit annotations synchronously. It also acted as an
onboarding session to welcome new contributors from regions that could not
join the New Contributor Introduction Meeting due to conflicting time zones.
Demonstrations on using the UI, as well as brainstorming sessions, were
conducted to improve the representation of specific languages in the project.

##  Appendix E Language Groupings

In this work we will refer to groups of languages to be “lower-”, “mid-” or
“higher”-resourced according to their recorded, written, and catalogued NLP
resources [Joshi et al., 2020]. Joshi et al. [2020] group languages into 5
distinct clusters based on the amount of data from a combined range of source
(LDC catalog161616https://catalog.ldc.upenn.edu/, ELRA
Map171717https://catalog.elra.info/en-us/, Wikipedia
181818https://wikipedia.org/), which we interpret as a proxy for data
availability for pretraining and IFT training of LLMs. We group these 5
distinct clusters into a rough taxonomy of lower-resourced (LR), mid-resourced
(MR) and higher-resourced (HR) (See Table 6). See Table
LABEL:tab:language_codes for full mapping of languages to categories. We note
that this grouping is inevitably imperfect; languages and their varieties
cannot absolutely nor universally be classified based on this single dimension
[Hämäläinen, 2021; Lignos et al., 2022; Bird, 2022]. The categorization in our
case serves the purpose of aggregation in our analysis of the data
distribution.

Group | Category | Languages | Examples
---|---|---|---
Higher-Resourced | 5 | 7 | Arabic, Chinese, English, French, Spanish
4 | 18 | Hindi, Italian, Portuguese, Russian, Turkish
Mid-Resourced | 3 | 25 | Afrikaans, Indonesian, Kazakh, Malay, Latvian
Lower-Resourced | 2 | 13 | Hausa, Icelandic, Irish, Lao, Maltese
1 | 39 | Albanian, Gujarati, Igbo, Luxembourgish
0 |  12∗ | Kurdish, Kyrgyz, Sinhala, Yiddish
Table 6: Language grouping for the Collection. We assign categories to
languages based on [Joshi et al., 2020]. (*) We assign label 0 to two
languages not found in Joshi et al. [2020]’s taxonomy (manipuri and ngaju).

##  Appendix F Post-Editing the dolly-machine-translated Test Set

###  F.1 Annotators

Annotator Selection The primary demographic make-up of the participants in the
evaluations was recruited based on their proficiency in the language groups.
The proficiency was self-reported, and our requirements were natively
proficient or professionally proficient in the specific languages needed for
the project. Outside of this, the participants come from diverse social
backgrounds comprised of students and individuals with full-time or part-time
jobs that do annotation as a “side gig”.

Socio-Demographics The annotator pool is comprised of people from diverse
backgrounds, and this spans across socioeconomic backgrounds, careers, levels
of education, and self-reported gender and sexual identities. We do not ask
any annotators to share or report any of these statistical pieces of
information in a formal way; any insights into this are gathered organically
and through self-reporting by the annotators.

Quality Considerations We do not believe that any socio-demographic
characteristics have led to any impact on the data that has been annotated.
Through every part of the project we have reiterated the importance of this
work and the fact that this is helping to support a global-scale research
project. We are confident in the trust we have built with the annotators in
this project, and they care greatly about the overall outcome and therefore
have been diligent in completing the task with a high degree of accuracy.
Where possible, we have done our best to have annotators work on this project
and be representatives of the communities that the project aims to support.

Compensation The annotators were paid 30 CAD per hour. No special
consideration was made to the hourly rate as that is the standard rate offered
to Cohere’s annotators who work on highly complex tasks.

###  F.2 Annotation Process

Communication Annotators were briefed by one of the authors in a virtual
introduction session, and were able to ask questions and raise issues
throughout the annotation task in a Slack channel. They were also encouraged
to share frequent error patterns, artifacts, or hard decisions that they
encountered throughout the task with the authors and other annotators.

Schedule There was no fixed time schedule for the annotations and annotators
contributed a varying amount of hours and ratings, depending on their
availabilities and speed. Each translation was post-edited by one annotator,
and there were 3–4 annotators involved in each task. After post-edits were
completed, a second annotator (not the original post-editor) assessed the
post-edit for quality and proposed new final edits if necessary.

Interface Post-edits were collected on Google Sheets with an interface built
in Google Apps Script.

###  F.3 Instructions

The instructions given to professional annotators for the dolly-machine-
translated test set post-edits were the following: “As an annotator, you have
the task to improve the quality of the prompts for our multilingual model! The
prompts are originally machine-translated from English, and sometimes the
translation introduces errors in the prompts that make them hard to follow.

We need your help to identify these cases, and to edit these translations so
that they…

  1. 1.

Convey the same instruction/task/request as the English original — not more
and not less.

  2. 2.

Are grammatically correct.

  3. 3.

Are free from phrases too literally translated from English (we call this
“Translationese”).

This is how:
For each pair of English prompt and translated prompt shown, decide whether
the prompt is okay as it is (according to the above criteria), or needs an
edit.

  * •

If it needs an edit, edit the prompt until the quality is satisfactory (in the
field “Edited Prompt”). Try to keep your edits minimal. Then confirm that the
edited prompt fulfills the above three criteria.

  * •

If it’s okay as is, just proceed (without editing the “Edited Prompt” field)
to confirm that it fulfills the above three criteria.”

Annotations were done through an interface built on top of Google Sheets. One
annotator edited each prompt, and another verified the edit, if necessary had
a discussion and edited the original edit. Three to four editors collaborated
on each language.

###  F.4 Post-Editing Effort

For each prompt, we measure the post-editing effort with Human-targeted
Translation Error Rate (HTER) [Specia & Farzindar, 2010], an edit-distance
metric that compares the original machine translation with the post-edited
version in terms of edit operations on units of words. This also gives us an
estimate of how severe the errors in the original translations were, and how
critically the post-editors assessed the original translations. Analogously,
we estimate with a Human-targeted Character F-Score (HChrF) score how much the
original translation overlaps with the final post-edited translation. This
metric is based on the ChrF score [Popović, 2015] and operates on character-
level matches. Computations of HTER and HChrF are based on the sacrebleu
implementation [Post, 2018].

Table 7 reports these statistics for the six languages of the dolly-machine-
translated test set. We find that editors edited at least 41% of prompts in
all languages, a surprisingly high number. This indicates that translation
errors in the dolly-machine-translated test set are quite common. For Russian,
the post-editing effort was overall largest, with an average of 37.43 HTER,
which means that 37.43% of words in the final post-edit had to be edited from
the original. This stands in contrast with the post-edits for French, where a
similar ratio of original prompts was edited (84.5% compared to 86.5% for
Russian), but to a much lesser extent (5.56 HTER).

Language | % of Prompts Edited | HTER | HChrF
---|---|---|---
Arabic | 41.0% | 10.78 | 92.74
French | 84.5% | 5.56 | 96.81
Hindi | 60.0% | 6.16 | 95.00
Russian | 86.5% | 37.43 | 75.92
Serbian | 72.5% | 9.06 | 92.79
Spanish | 75.5% | 9.13 | 93.25
Table 7: Post-editing effort measured by the overall percentage of edited
dolly test prompts, HTER (Human-targeted Translation Error Rate: the higher,
the more effort), and HChrF (Human-Targeted Character F-Score: the lower, the
more effort).

##  Appendix G Translation using NLLB

Additional Generation details One caveat with using NLLB is that since the
model was trained on single sentence pairs, the translations tend to cut off
abruptly when full paragraphs are translated. To get around this, we sentence-
tokenize the paragraphs using the sentence-splitter Python package (similar to
[NLLB-Team et al., 2022]) and concatenate them post-translation. To avoid
known translation introduced artefacts, We also filter any samples which have
<unk> tokens introduced by the NLLB tokenizer or model.

###  G.1 Translation Quality of NLLB

Figure 22 illustrates NLLB translation quality as measured by ChrF++ on the
FLORES benchmark for the languages of interest for , grouped by their
resourcefulness according to [Joshi et al., 2020]. The scores were extracted
from https://github.com/facebookresearch/fairseq/blob/nllb/README.md for the
dense 3.3B model.

(a) Low-resource Languages: [Joshi et al., 2020] classes 0, 1, 2

(b) Mid-resource Languages: [Joshi et al., 2020] class 3

(c) High-resource Languages: [Joshi et al., 2020] classes 4, 5

Figure 22: NLLB Translation Quality: ChrF++ scores on FLORES for translations
from English into the target languages that are covered by NLLB, grouped by
their resourcefulness according to [Joshi et al., 2020].

##  Appendix H Additional Figures

Figure 23: Average Approval Ratio per dataset, constrained to datasets
receiving at least 20 votes.

(a) Average prompt length

(b) Average completion length

Figure 24: Average prompt and completion length of instances in the Dataset
before and after re-annotation across different language categories.

(a) Average completion length for every language in the Dataset.

(b) Average prompt length for every language in the Dataset

Figure 25: Average prompt and completion length for every language in the
Dataset Figure 26: Average prompt and completion length across different
languages in Dataset.

##  Appendix I Additional Tables

Table 8: List of datasets in Collection (templated datasets). Dataset |  #Langs |  Template lang |  Dataset lang |  $\bar{L}_{prompt}$ |  $\bar{L}_{compl.}$ |  License |  Task
---|---|---|---|---|---|---|---
AfriQA-inst [Ogundepo et al., 2023] |  12 |  |  bem, fon, hau, ibo, kin, swh, twi, wol, yor, zul, eng, fra |  46 |  15 |  CC BY 4.0 |  Question Answering
AfriSenti-inst [Muhammad et al., 2023] |  9 |  |  amh, arq, hau, ibo, kin, ary, por, swh, twi |  168 |  44 |  CC BY 4.0 |  Sentiment Analysis
Amharic QA [Abedissa et al., 2023] |  1 |  amh |  amh |  1114 |  33 |  MIT license |  Question Answering
News-summary-instruct [TahmidH, 2023] |  1 |  ben |  ben |  174 |  67 |  CC0 1.0 |  Summarization
Arpa-instruct [syntaxshill, 2023] |  1 |  hye |  hye |  165 |  118 |  Artistic-2.0 |  Paraphrasing
Telugu-food-recipes [SuryaKrishna02, 2023a] |  1 |  tel |  tel |  70 |  870 |  Apache 2.0 |  Generation
Telugu-jokes [SuryaKrishna02, 2023b] |  1 |  tel |  tel |  80 |  276 |  Apache 2.0 |  Generation
Telugu-news-articles [SuryaKrishna02, 2023c] |  1 |  tel |  tel |  448 |  426 |  Apache 2.0 |  Generation
Telugu-poems [SuryaKrishna02, 2023e] |  1 |  tel |  tel |  357 |  198 |  Apache 2.0 |  Generation
FarsTail-Instruct [Amirkhani et al., 2023; hghader1, 2023] |  1 |  pes |  pes |  224 |  112 |  Apache 2.0 |  Natural Language Inference
Hindi-article-summarization [ganeshjcs, 2023a] |  1 |  hin |  hin |  3813 |  175 |  CC BY-SA 4.0 |  Summarization
Hindi-article-generation [ganeshjcs, 2023b] |  1 |  hin |  hin |  102 |  3683 |  CC BY-SA 4.0 |  Generation
IMDB-Dutch-instruct [Maas et al., 2011; jjzha, 2023] |  1 |  nld |  nld |  1470 |  31 |  Apache 2.0 |  Sentiment Analysis
IndicSentiment-inst [Doddapaneni et al., 2023; el2e10, 2023a] |  11 |  eng |  ben, guj, hni, kan, mal, mar, pan, tam, tel, urd, eng |  174 |  141 |  MIT |  Translation
IndicXParaphrase-inst [Doddapaneni et al., 2023; el2e10, 2023b; SuryaKrishna02, 2023d] |  7 |  ben, guj, hin, mar, pan, mal, tel |  ben, guj, hin, mar, pan, mal, tel |  132 |  93 |  MIT |  Paraphrase Identification
Indo-stories-instruct [Iftitahu, 2023a, b, c] |  3 |  ind, sun, jav |  ind, sun, jav |  345 |  322 |  CC BY 4.0 |  Translation
Joke-explaination-inst [theblackcat102, 2023] |  1 |  |  eng |  118 |  548 |  MIT |  Generation
Lijnews-instruct [ConseggioLigure, 2023a, b] |  2 |  ita, lij |  it, lij |  893 |  898 |  CC BY 4.0 |  Translation
LLM-Japanese-Vanilla-inst [Suzuki et al., 2023; Tellarin.ai, 2023a] |  1 |  jpn |  jpn |  60 |  97 |  CC BY-SA 4.0 |  Question Answering
MasakhaNEWS-inst [Adelani et al., 2023] |  16 |  |  amh, eng, fra, hau, ibo, lin, cgg, orm, pcm, run, sna, som, swh, tir, xho, yor |  1483 |  1459 |  AFL-3.0 |  Text Classification
Mintaka-inst [Sen et al., 2022] |  9 |  eng |  arb, deu, spa, fra, jpn, por, hin, ita, eng |  102 |  49 |  CC BY 4.0 |  Question Answering
NTX-LLM-inst [Chen et al., 2023c; Tellarin.ai, 2023b], |  13 |  arb, zho, nld, eng, fra, deu, hin, ita, jpn, kor, por, spa, tur |  arb, zho, nld, eng, fra, deu, hin, ita, jpn, kor, por, spa, tur |  917 |  493 |  CC BY-SA 4.0 |  Information Extraction
NusaX-senti-inst [Winata et al., 2023b] |  12 |  |  ace, ban, bjn, bug, eng, ind, jav, mad, min, nij, sun, bbc |  219 |  22 |  Apache 2.0 |  Sentiment Analysis
Persian-instruct-pn [Farahani et al., 2021; Shafagh, 2023a, b] |  1 |  pes |  pes |  1713 |  128 |  MIT |  Summarization
SCB-MT-2020-prompt [Lowphansirikul et al., 2022; PyThaiNLP, 2023a, b] |  2 |  tha, eng |  tha, eng |  181 |  127 |  CC BY-SA 4.0 |  Translation
Scirepeval-biomimicry-inst [Singh et al., 2022] |  1 |  |  eng |  996 |  523 |  ODC-BY |  Scientific Document Representation
Seed-instruct-lij [Maillard et al., 2023; ConseggioLigure, 2023c, d] |  2 |  lij, eng |  lij, eng |  184 |  186 |  CC BY-SA 4.0 |  Translation
SODA-inst [Kim et al., 2022] |  1 |  |  eng |  412 |  328 |  CC BY 4.0 |  Dialogue
TamilStories [AI Tamil Nadu, 2023a] |  1 |  tam |  tam |  2266 |  2172 |  Apache 2.0 |  Generation
TeluguRiddles [desik98, 2023] |  1 |  tel |  tel |  74 |  44 |  Apache 2.0 |  Question Answering
Thai-USEmbassy-prompt [PyThaiNLP, 2023d, e] |  2 |  tha, eng |  tha, eng |  2131 |  2077 |  CC0 1.0 |  Translation
Thai-POS-inst [PyThaiNLP, 2023c] |  1 |  tha |  tha |  72 |  36 |  CC BY-SA 3.0 |  Generation
Thai-Wiktionary-inst [PyThaiNLP, 2023f] |  1 |  tha |  tha |  35 |  147 |  CC BY-SA 3.0 |  Generation
Thirukkural-instruct [AI Tamil Nadu, 2023b] |  1 |  tam |  tam |  133 |  542 |  Apache 2.0 |  Generation
Turku-paraphrase-inst [Kanerva et al., 2021; TurkuNLP, 2023] |  1 |  fin |  fin |  108 |  59 |  CC BY-SA 4.0 |  Paraphrase Identification
UA-Gec-inst [Syvokon et al., 2023; osyvokon, 2023] |  1 |  ukr |  ukr |  192 |  148 |  CC BY 4.0 |  Generation
UNER-LLM-inst [Mayhew et al., 2023; Universal NER, 2023] |  11 |  zho, hrv, dan, eng, deu, por, rus, srp, slk, swe, tgl |  zho, hrv, dan, eng, deu, por, rus, srp, slk, swe, tgl |  768 |  109 |  CC BY-SA 4.0 |  Named Entity Recognition
Urdu-News-Gen-Article [Hussain et al., 2021; AhmadMustafa, 2023a] |  1 |  urd |  urd |  109 |  1313 |  CC BY 4.0 |  Generation
Urdu-News-Category-Class [Hussain et al., 2021; AhmadMustafa, 2023b] |  1 |  urd |  urd |  1407 |  43 |  CC BY 4.0 |  Text Classification
Urdu-News-Gen-Headline [Hussain et al., 2021; AhmadMustafa, 2023c] |  1 |  urd |  urd |  1314 |  94 |  CC BY 4.0 |  Generation
Wiki-split-inst [Botha et al., 2018] |  1 |  |  eng |  200 |  166 |  CC BY 4.0 |  Text Simplification
X-CSQA-inst [Lin et al., 2021] |  16 |  |  eng, zho, deu, spa, fra, ita, jpn, nld, pol, por, rus, arb, vie, hin, swa, urd |  197 |  21 |  MIT |  Question Answering
Xlel_wd-inst [Pratapa et al., 2022] |  44 |  |  |  379 |  190 |  CC BY 4.0 |  Event Linking
XWikis-inst [Perez-Beltrachini & Lapata, 2021] |  4 |  |  ces, fra, eng, deu |  5662 |  346 |  MIT |  Summarization
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Table 9: List of datasets in Aya Collection (translated datasets). Dataset |  #Langs |  $\bar{L}_{prompt}$ |  $\bar{L}_{compl.}$ |  License |  Task
---|---|---|---|---|---
Adversarial QA (T) [Bartolo et al., 2020] |  101 |  159 |  721 |  CC BY-SA 3.0 |  Question Answering
CNN-Daily-Mail (T) [See et al., 2017] [Hermann et al., 2015] |  101 |  1980 |  305 |  Apache 2.0 |  Summarization
Flan-Coqa (T) [Wei et al., 2022a; Reddy et al., 2019] |  101 |  2143 |  364 |  Multiple* |  Question Answering
Flan-CoT-submix (T) [Wei et al., 2022a] |  101 |  239 |  160 |  Unknown |  Generation
Flan-GEM-wiki-lingua (T) [Wei et al., 2022a; Ladhak et al., 2020] |  101 |  1732 |  572 |  CC BY-NC-SA 3.0 |  Summarization
Flan-lambada (T) [Wei et al., 2022a; Paperno et al., 2016] |  101 |  232 |  7 |  CC BY 4.0 |  Generation
Flan-unified-QA (T) [Wei et al., 2022a; Khashabi et al., 2020] |  101 |  281 |  13 |  Apache 2.0 |  Question Answering
HotpotQA (T) [Yang et al., 2018] |  101 |  129 |  15 |  CC BY-SA 4.0 |  Question Answering
Joke-explaination-inst (T) [theblackcat102, 2023] |  101 |  111 |  545 |  MIT |  Generation
Mintaka-inst (T) [Sen et al., 2022] |  101 |  54 |  67 |  CC BY 4.0 |  Question Answering
MLQA-en (T) [Lewis et al., 2020] |  101 |  819 |  20 |  CC BY-SA 3.0 |  Question Answering
NQ-Open (T) [Kwiatkowski et al., 2019] |  101 |  68 |  14 |  CC BY-SA 3.0 |  Question Answering
PAWS-Wiki (T) [Zhang et al., 2019] |  101 |  308 |  6 |  Custom license, attribution |  Paraphrase Identification
PIQA (T) [Bisk et al., 2020] |  101 |  304 |  100 |  Unknown |  Question Answering
SODA-inst (T) [Kim et al., 2022] |  101 |  86 |  208 |  CC BY 4.0 |  Dialogue
WIKI QA (T) [Yang et al., 2015] |  101 |  205 |  36 |  MSR DLA* |  Question Answering
Wiki-split-inst (T) [Botha et al., 2018] |  101 |  126 |  220 |  CC BY-SA 4.0 |  Text Simplification
Xlel_wd-inst (T) [Pratapa et al., 2022] |  101 |  300 |  274 |  CC BY 4.0 |  Event Linking
Dolly-v2 (T) [Conover et al., 2023] |  101 |  427 |  357 |  CC BY-SA 3.0 |  Generation
|  |  |  |  |
Table 10: Task Taxonomy of Templated Datasets (Collection). We classify the templated datasets with a standard task taxonomy of three main tasks: Question Answering, Natural Language Generation, and Text Classification (Table 4). We then have a fine-grained task taxonomy within each task, such as Summarization, Translation, Paraphrasing, Sentiment Analysis, Information Extraction, and Named Entity Recognition. If there is not a recognized fine-grained task taxonomy for a specific dataset, we put it in the main task type category. Main Task Type | Fine-grained Task Type | Dataset
---|---|---
Question Answering |  |  AfriQA-inst [Ogundepo et al., 2023]
|  |  Amharic QA [Abedissa et al., 2023]
|  |  LLM-Japanese-Vanilla-inst [Tellarin.ai, 2023a]
|  |  Mintaka-inst [Sen et al., 2022]
|  |  X-CSQA-inst [Lin et al., 2021]
|  |  TeluguRiddles [desik98, 2023]
Natural Language | Summarization |  News-summary-instruct [TahmidH, 2023]
Generation |  |  Persian-instruct-pn [Shafagh, 2023a]
|  |  Hindi-article-summarization [ganeshjcs, 2023a]
|  |  XWikis-inst [Perez-Beltrachini & Lapata, 2021]
| Translation |  IndicSentiment-inst [el2e10, 2023a]
|  |  Indo-stories-instruct [Iftitahu, 2023a, b, c]
|  |  Lijnews-instruct [ConseggioLigure, 2023a, b]
|  |  SCB-MT-2020-prompt [PyThaiNLP, 2023a, b]
|  |  Thai-USEmbassy-prompt [PyThaiNLP, 2023d, e]
|  |  SEED-instruct-lij [ConseggioLigure, 2023c, d]
| Paraphrasing |  Arpa-instruct [syntaxshill, 2023]
|  |  IndicXParaphrase-inst [el2e10, 2023b; SuryaKrishna02, 2023d]
|  |  Turku-paraphrase-inst [TurkuNLP, 2023]
| Text Simplification |  Wiki-split-inst [Botha et al., 2018]
| Dialogue |  SODA-inst [Kim et al., 2022]
| NL Generation |  Telugu-food-recipes [SuryaKrishna02, 2023a]
|  |  Telugu-jokes [SuryaKrishna02, 2023b]
|  |  Telugu-news-articles [SuryaKrishna02, 2023c]
|  |  Telugu-poems [SuryaKrishna02, 2023e]
|  |  TamilStories [AI Tamil Nadu, 2023a]
|  |  Joke-explaination-inst [theblackcat102, 2023]
|  |  Thirukkural-instruct [AI Tamil Nadu, 2023b]
|  |  Hindi-article-generation [ganeshjcs, 2023b]
|  |  Thai-Wiktionary-inst [PyThaiNLP, 2023f]
|  |  UA-Gec-inst [osyvokon, 2023]
|  |  Urdu-News-Gen-Article [AhmadMustafa, 2023a]
|  |  Urdu-News-Gen-Headline [AhmadMustafa, 2023c]
|  |  Thai-POS-inst [PyThaiNLP, 2023c]
Text Classification | Sentiment Analysis |  AfriSenti-inst [Muhammad et al., 2023]
|  |  IMDB-Dutch-instruct [jjzha, 2023]
|  |  NusaX-senti-inst [Winata et al., 2023b]
| Information Extraction |  NTX-LLM-inst [Tellarin.ai, 2023b]
| Named Entity Recognition |  UNER-LLM-inst [Universal NER, 2023]
| Natural Language Inference |  FarsTail-Instruct [hghader1, 2023]
| Event Linking |  Xlel_wd-inst [Pratapa et al., 2022]
| Sci. Doc. Representation |  Scirepeval-biomimicry-inst [Singh et al., 2022]
| Text Classification |  Urdu-News-Category-Class [AhmadMustafa, 2023b]
|  |  MasakhaNEWS-inst [Adelani et al., 2023]
Table 11: Task Taxonomy of Translated Datasets (Aya Collection). We classify the translated datasets similar to templated datasets (Table LABEL:tab:templated_taxonomy). If there is not a recognized fine-grained task taxonomy for a specific dataset, we put it in the main task type category. Main Task Type |  Fine-grained Task Type |  Dataset
---|---|---
Question Answering |  |  Adversarial QA (T) [Bartolo et al., 2020]
|  |  Flan-Coqa (T) [Wei et al., 2022a; Reddy et al., 2019]
|  |  Flan-unified-QA (T) [Wei et al., 2022a; Khashabi et al., 2020]
|  |  HotpotQA (T) [Yang et al., 2018]
|  |  Mintaka-inst (T) [Sen et al., 2022]
|  |  MLQA-en (T) [Lewis et al., 2020]
|  |  NQ-Open (T) [Kwiatkowski et al., 2019]
|  |  PIQA (T) [Bisk et al., 2020]
|  |  WIKI QA (T) [Yang et al., 2015]
Natural Language |  Summarization |  CNN-Daily-Mail (T) [See et al., 2017] [Hermann et al., 2015]
Generation |  |  Flan-GEM-wiki-lingua (T)[Wei et al., 2022a; Ladhak et al., 2020]
|  Text Simplification |  Wiki-split-inst (T) [Botha et al., 2018]
|  Dialogue |  SODA-inst (T) [Kim et al., 2022]
|  NL Generation |  Joke-explaination-inst (T) [theblackcat102, 2023]
|  |  Flan-CoT-submix (T)[Wei et al., 2022a]
|  |  Flan-lambada (T) [Wei et al., 2022a; Paperno et al., 2016]
|  |  Dolly-v2 (T) [Conover et al., 2023]
Text Classification |  Event Linking |  Xlel_wd-inst (T) [Pratapa et al., 2022]
|  Paraphrase Identification |  PAWS-Wiki (T) [Zhang et al., 2019]
|  |