* Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018.  Colorless green recurrent networks dream hierarchically.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703, Online. Association for Computational Linguistics.
  * He et al. (2018) Yu He, Jianxin Li, Yangqiu Song, Mutian He, and Hao Peng. 2018.  Time-evolving text classification with deep neural networks.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 2241–2247. International Joint Conferences on Artificial Intelligence Organization.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Hu et al. (2019) Junjie Hu, Mengzhou Xia, Graham Neubig, and Jaime Carbonell. 2019.  Domain adaptation of neural machine translation by lexicon induction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2989–3001, Florence, Italy. Association for Computational Linguistics.
  * Huang and Paul (2018) Xiaolei Huang and Michael J. Paul. 2018.  Examining temporality in document classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 694–699, Melbourne, Australia. Association for Computational Linguistics.
  * Huang and Paul (2019) Xiaolei Huang and Michael J. Paul. 2019.  Neural temporality adaptation for document classification: Diachronic word embeddings and domain adaptation models.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4113–4123, Florence, Italy. Association for Computational Linguistics.
  * Hupkes et al. (2018) Dieuwke Hupkes, Sara , and Willem Zuidema. 2018.  Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure.  _Journal of Artificial Intelligence Research_ , 61:907–926.
  * Hupkes (2020) Dieuwke Hupkes. 2020.  _Hierarchy and interpretability in neural models of language processing_.  Ph.D. thesis, University of Amsterdam.
  * Hupkes et al. (2020) Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020.  Compositionality decomposed: How do neural networks generalise? _Journal of Artificial Intellgence Research_ , 67:757–795.
  * Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Social biases in NLP models as barriers for persons with disabilities.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5491–5501, Online. Association for Computational Linguistics.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
  * Joty et al. (2015) Shafiq Joty, Hassan Sajjad, Nadir Durrani, Kamla Al-Mannai, Ahmed Abdelali, and Stephan Vogel. 2015.  How to avoid unwanted pregnancies: Domain adaptation using neural network models.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1259–1270, Lisbon, Portugal. Association for Computational Linguistics.
  * Jumelet et al. (2021) Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert-Threlkeld. 2021.  Language models use monotonicity to assess NPI licensing.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4958–4969, Online. Association for Computational Linguistics.
  * Kaushik et al. (2019) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019.  Learning the difference that makes a difference with counterfactually-augmented data.  In _International Conference on Learning Representations_.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C. Lipton. 2018.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5010–5015, Brussels, Belgium. Association for Computational Linguistics.
  * Kavumba et al. (2022) Pride Kavumba, Ryo Takahashi, and Yusuke Oda. 2022.  Are prompt-based models clueless? In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2333–2352, Dublin, Ireland. Association for Computational Linguistics.
  * Keysers et al. (2019) Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. 2019.  Measuring compositional generalization: A comprehensive method on realistic data.  In _International Conference on Learning Representations_.
  * Khishigsuren et al. (2022) Temuulen Khishigsuren, Gábor Bella, Khuyagbaatar Batsuren, Abed Alhakim Freihat, Nandu Chandran Nair, Amarsanaa Ganbold, Hadi Khalilia, Yamini Chandrashekar, and Fausto Giunchiglia. 2022.  Using linguistic typology to enrich multilingual lexicons: the case of lexical gaps in kinship.  _CoRR_ , abs/2204.05049.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021.  Dynabench: Rethinking benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4110–4124, Online. Association for Computational Linguistics.
  * Kim and Linzen (2020) Najoung Kim and Tal Linzen. 2020.  COGS: A compositional generalization challenge based on semantic interpretation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9087–9105, Online. Association for Computational Linguistics.
  * Kirk et al. (2021) Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. 2021.  A survey of generalisation in deep reinforcement learning.  _CoRR_ , abs/2111.09794.
  * Kirov and Cotterell (2018) Christo Kirov and Ryan Cotterell. 2018.  Recurrent neural networks in linguistic theory: Revisiting pinker and prince (1988) and the past tense debate.  _Transactions of the Association for Computational Linguistics_ , 6:651–665.
  * Kneser and Ney (1995) Reinhard Kneser and Hermann Ney. 1995.  Improved backing-off for m-gram language modeling.  In _1995 international conference on acoustics, speech, and signal processing_ , volume 1, pages 181–184. IEEE.
  * Kodner et al. (2022) Jordan Kodner, Salam Khalifa, Khuyagbaatar Batsuren, Hossep Dolatian, Ryan Cotterell, Faruk Akkus, Antonios Anastasopoulos, Taras Andrushko, Aryaman Arora, Nona Atanalov, Gábor Bella, Elena Budianskaya, Yustinus Ghanggo Ate, Omer Goldman, David Guriel, Simon Guriel, Silvia Guriel-Agiashvili, Witold Kieraś, Andrew Krizhanovsky, Natalia Krizhanovsky, Igor Marchenko, Magdalena Markowska, Polina Mashkovtseva, Maria Nepomniashchaya, Daria Rodionova, Karina Scheifer, Alexandra Sorova, Anastasia Yemelina, Jeremiah Young, and Ekaterina Vylomova. 2022.  SIGMORPHON–UniMorph 2022 shared task 0: Generalization and typologically diverse morphological inflection.  In _Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 176–203, Seattle, Washington. Association for Computational Linguistics.
  * Koehn and Schroeder (2007) Philipp Koehn and Josh Schroeder. 2007.  Experiments in domain adaptation for statistical machine translation.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 224–227, Prague, Czech Republic. Association for Computational Linguistics.
  * Koh et al. (2021) Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. 2021.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pages 5637–5664. PMLR.
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4034–4048, Online. Association for Computational Linguistics.
  * Lake and Baroni (2018) Brenden Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _proceedings of the 35th International Conference on Machine Learning (ICML)_ , pages 4487–4499.
  * Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017\.  Building machines that learn and think like people.  _Behavioral and brain sciences_ , 40.
  * Lakretz et al. (2021a) Yair Lakretz, Theo Desbordes, Dieuwke Hupkes, and Stanislas Dehaene. 2021a.  Causal transformers perform below chance on recursive nested constructions, unlike humans.  _CoRR_ , abs/2110.07240.
  * Lakretz et al. (2021b) Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. 2021b.  Mechanisms for handling nested dependencies in neural-network language models and humans.  _Cognition_ , 213:104699.  Special Issue in Honour of Jacques Mehler, Cognition’s founding editor.
  * Lakretz et al. (2019) Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019.  The emergence of number and syntax units in LSTM language models.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 11–20, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  _Advances in Neural Information Processing Systems_ , 34:29348–29363.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Lewis et al. (2021) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021.  Question and answer test-train overlap in open-domain question answering datasets.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1000–1008, Online. Association for Computational Linguistics.
  * Li et al. (2022) Belinda Li, Jane Yu, Madian Khabsa, Luke Zettlemoyer, Alon Halevy, and Jacob Andreas. 2022.  Quantifying adaptability in pre-trained language models with 500 tasks.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4696–4715, Seattle, United States. Association for Computational Linguistics.
  * Li et al. (2021a) Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021a.  MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2950–2962, Online. Association for Computational Linguistics.
  * Li and Wilson (2021) Jane S.Y. Li and Colin Wilson. 2021.  Leveraging paradigmatic information in inflection acceptability prediction: The JHU-SFU submission to SIGMORPHON shared task 0.2.  In _Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 289–294, Online. Association for Computational Linguistics.
  * Li et al. (2021b) Yafu Li, Yongjing Yin, Yulong Chen, and Yue Zhang. 2021b.  On compositional generalization of neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4767–4780, Online. Association for Computational Linguistics.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020.  XGLUE: A new benchmark datasetfor cross-lingual pre-training, understanding and generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6008–6018, Online. Association for Computational Linguistics.
  * Libovický et al. (2022) Jindřich Libovický, Helmut Schmid, and Alexander Fraser. 2022.  Why don’t people use character-level machine translation? In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2470–2485, Dublin, Ireland. Association for Computational Linguistics.
  * Lin et al. (2021) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2021.  Few-shot learning with multilingual language models.
  * Linzen (2020) Tal Linzen. 2020.  How can we accelerate progress towards human-like linguistic generalization? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5210–5217, Online. Association for Computational Linguistics.
  * Liu et al. (2019a) Jiahua Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2019a.  XQA: A cross-lingual open-domain question answering dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2358–2368, Florence, Italy. Association for Computational Linguistics.
  * Liu and Hulden (2022) Ling Liu and Mans Hulden. 2022.  Can a transformer pass the wug test? tuning copying bias in neural morphological inflection models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 739–749, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2021) Linqing Liu, Patrick S. H. Lewis, Sebastian Riedel, and Pontus Stenetorp. 2021.  Challenges in generalization in open domain question answering.  _CoRR_ , abs/2109.01156.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  RoBERTa: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Longpre et al. (2021) Shayne Longpre, Yi Lu, and Joachim Daiber. 2021.  MKQA: A linguistically diverse benchmark for multilingual open domain question answering.  _Transactions of the Association for Computational Linguistics_ , 9:1389–1406.
  * Ludwig et al. (2022) Florian Ludwig, Klara Dolos, Torsten Zesch, and Eleanor Hobley. 2022.  Improving generalization of hate speech detection systems to novel target groups via domain adaptation.  In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_ , pages 29–39, Seattle, Washington (Hybrid). Association for Computational Linguistics.
  * Lukes and Søgaard (2018) Jan Lukes and Anders Søgaard. 2018.  Sentiment analysis under temporal shift.  In _Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 65–71, Brussels, Belgium. Association for Computational Linguistics.
  * Luong and Manning (2015) Minh-Thang Luong and Christopher Manning. 2015.  Stanford neural machine translation systems for spoken language domains.  In _Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign_ , pages 76–79, Da Nang, Vietnam.
  * Magerman (1995) David M. Magerman. 1995.  Statistical decision-tree models for parsing.  In _33rd Annual Meeting of the Association for Computational Linguistics_ , pages 276–283, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
  * Malinin et al. (2021) Andrey Malinin, Neil Band, Yarin Gal, Mark J. F. Gales, Alexander Ganshin, German Chesnokov, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Denis Roginskiy, Mariya Shmatova, Panagiotis Tigas, and Boris Yangel. 2021.  Shifts: A dataset of real distributional shift across multiple large-scale tasks.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Malouf (2017) Robert Malouf. 2017.  Abstractive morphological learning with a recurrent neural network.  _Morphology_ , 27(4):431–458.
  * Marcus (2018) Gary Marcus. 2018.  Deep learning: A critical appraisal.  _CoRR_ , abs/1801.00631.
  * Marcus (1998) Gary F. Marcus. 1998.  Rethinking eliminative connectionism.  _Cognitive Psychology_ , 37(3):243–282.
  * Marcus (1999) Gary F Marcus. 1999.  Connectionism: with or without rules?: Response to jl mcclelland and dc plaut (1999).  _Trends in Cognitive Sciences_ , 3(5):168–170.
  * Marcus (2003) Gary F Marcus. 2003.  _The algebraic mind: Integrating connectionism and cognitive science_.  MIT press.
  * Marcus et al. (1995) Gary F Marcus, Ursula Brinkmann, Harald Clahsen, Richard Wiese, and Steven Pinker. 1995.  German inflection: The exception that proves the rule.  _Cognitive psychology_ , 29(3):189–256.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018.  The natural language decathlon: Multitask learning as question answering.  _arXiv preprint arXiv:1806.08730_.
  * McCarthy et al. (2019) Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019.  The SIGMORPHON 2019 shared task: Morphological analysis in context and cross-lingual transfer for inflection.  In _Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 229–244, Florence, Italy. Association for Computational Linguistics.
  * McClelland and Plaut (1999) James L McClelland and David C Plaut. 1999.  Does generalization in infant learning implicate abstract algebra-like rules?  _Trends in Cognitive Sciences_ , 3(5):166–168.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * McCurdy et al. (2020) Kate McCurdy, Sharon Goldwater, and Adam Lopez. 2020.  Inflecting when there’s no majority: Limitations of encoder-decoder neural networks as cognitive models for German plurals.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1745–1756, Online. Association for Computational Linguistics.
  * Michel and Neubig (2018) Paul Michel and Graham Neubig. 2018.  MTNT: A testbed for machine translation of noisy text.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 543–553, Brussels, Belgium. Association for Computational Linguistics.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting.  In _Proceedings of the conference on fairness, accountability, and transparency_ , pages 220–229.
  * Montague (1970) Richard Montague. 1970.  Universal grammar.  _Theoria_ , 36(3):373–398.
  * Moreno-Torres et al. (2012) Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. 2012.  A unifying view on dataset shift in classification.  _Pattern recognition_ , 45(1):521–530.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 839–849, San Diego, California. Association for Computational Linguistics.
  * Mrini et al. (2020) Khalil Mrini, Franck Dernoncourt, Quan Hung Tran, Trung Bui, Walter Chang, and Ndapa Nakashole. 2020.  Rethinking self-attention: Towards interpretability in neural parsing.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 731–742, Online. Association for Computational Linguistics.
  * Mul and Zuidema (2019) Mathijs Mul and Willem Zuidema. 2019.  Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization.  In _CoRR, abs/1906.00180_.
  * Muller et al. (2021) Benjamin Muller, Luca Soldaini, Rik Koncel-Kedziorski, Eric Lind, and Alessandro Moschitti. 2021.  Cross-lingual genqa: Open-domain question answering with answer sentence generation.
  * Nguyen and Daumé III (2019) Khanh Nguyen and Hal Daumé III. 2019.  Global Voices: Crossing borders in automatic news summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 90–97, Hong Kong, China. Association for Computational Linguistics.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4034–4043, Marseille, France. European Language Resources Association.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  * Papadimitriou and Jurafsky (2020) Isabel Papadimitriou and Dan Jurafsky. 2020.  Learning Music Helps You Read: Using transfer to study linguistic structure in language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6829–6839, Online. Association for Computational Linguistics.
  * Park et al. (2018) Ji Ho Park, Jamin Shin, and Pascale Fung. 2018.  Reducing gender bias in abusive language detection.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2799–2804, Brussels, Belgium. Association for Computational Linguistics.
  * Parrish et al. (2022) Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022.  BBQ: A hand-built bias benchmark for question answering.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2086–2105, Dublin, Ireland. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Petrov and Klein (2007) Slav Petrov and Dan Klein. 2007.  Improved inference for unlexicalized parsing.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_ , pages 404–411, Rochester, New York. Association for Computational Linguistics.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Pimentel et al. (2021a) Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, Eleanor Chodroff, Brian Leonard, Garrett Nicolai, Yustinus Ghanggo Ate, Salam Khalifa, Nizar Habash, Charbel El-Khaissi, Omer Goldman, Michael Gasser, William Lane, Matt Coler, Arturo Oncevay, Jaime Rafael Montoya Samame, Gema Celeste Silva Villegas, Adam Ek, Jean-Philippe Bernardy, Andrey Shcherbakov, Aziyana Bayyr-ool, Karina Sheifer, Sofya Ganieva, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Andrew Krizhanovsky, Natalia Krizhanovsky, Clara Vania, Sardana Ivanova, Aelita Salchak, Christopher Straughn, Zoey Liu, Jonathan North Washington, Duygu Ataman, Witold Kieraś, Marcin Woliński, Totok Suhardijanto, Niklas Stoehr, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Richard J. Hatcher, Emily Prud’hommeaux, Ritesh Kumar, Mans Hulden, Botond Barta, Dorina Lakatos, Gábor Szolnok, Judit Ács, Mohit Raj, David Yarowsky, Ryan Cotterell, Ben Ambridge, and Ekaterina Vylomova. 2021a.  Sigmorphon 2021 shared task on morphological reinflection: Generalization across languages.  In _Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 229–259, Online. Association for Computational Linguistics.
  * Pimentel et al. (2021b) Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, Eleanor Chodroff, Brian Leonard, Garrett Nicolai, Yustinus Ghanggo Ate, Salam Khalifa, Nizar Habash, Charbel El-Khaissi, Omer Goldman, Michael Gasser, William Lane, Matt Coler, Arturo Oncevay, Jaime Rafael Montoya Samame, Gema Celeste Silva Villegas, Adam Ek, Jean-Philippe Bernardy, Andrey Shcherbakov, Aziyana Bayyr-ool, Karina Sheifer, Sofya Ganieva, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Andrew Krizhanovsky, Natalia Krizhanovsky, Clara Vania, Sardana Ivanova, Aelita Salchak, Christopher Straughn, Zoey Liu, Jonathan North Washington, Duygu Ataman, Witold Kieraś, Marcin Woliński, Totok Suhardijanto, Niklas Stoehr, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Richard J. Hatcher, Emily Prud’hommeaux, Ritesh Kumar, Mans Hulden, Botond Barta, Dorina Lakatos, Gábor Szolnok, Judit Ács, Mohit Raj, David Yarowsky, Ryan Cotterell, Ben Ambridge, and Ekaterina Vylomova. 2021b.  Sigmorphon 2021 shared task on morphological reinflection part 2: Are we there yet? A shared task on cognitively plausible morphological inflection.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Plank (2016) Barbara Plank. 2016.  What to do about non-standard (or non-canonical) language in NLP.  _arXiv preprint arXiv:1608.07836_.
  * Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.  Hypothesis only baselines in natural language inference.  In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ , pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  XCOPA: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rae et al. (2021) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021.  Scaling language models: Methods, analysis & insights from training gopher.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Raganato et al. (2020) Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2020.  XL-WiC: A multilingual benchmark for evaluating semantic contextualization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7193–7206, Online. Association for Computational Linguistics.
  * Rahimi et al. (2019) Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019.  Massively multilingual transfer for NER.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 151–164, Florence, Italy. Association for Computational Linguistics.
  * Raunak et al. (2020) Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. 2020.  On long-tailed phenomena in neural machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3088–3095, Online. Association for Computational Linguistics.
  * Raunak et al. (2019) Vikas Raunak, Vaibhav Kumar, Florian Metze, and Jaimie Callan. 2019.  On compositionality in neural machine translation.  In _NeurIPS 2019 Context and Compositionality in Biological and Artificial Neural Systems Workshop_.
  * Razeghi et al. (2022) Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022.  Impact of pretraining term frequencies on few-shot reasoning.  _CoRR_ , abs/2202.07206.
  * Rijhwani and Preotiuc-Pietro (2020) Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020.  Temporally-informed analysis of named entity recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7605–7617, Online. Association for Computational Linguistics.
  * ROSENFELD (1996) R ROSENFELD. 1996.  A maximum entropy approach to adaptive statistical language modelling.  _Computer speech & language (Print)_, 10(3):187–228.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021\.  XTREME-R: Towards more challenging and nuanced multilingual evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ryu et al. (2018) Seonghan Ryu, Sangjun Koo, Hwanjo Yu, and Gary Geunbae Lee. 2018.  Out-of-domain detection based on generative adversarial network.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 714–718, Brussels, Belgium. Association for Computational Linguistics.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Winogrande: An adversarial Winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106.
  * Sangati and Zuidema (2011) Federico Sangati and Willem Zuidema. 2011.  Accurate parsing with compact tree-substitution grammars: Double-DOP.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 84–95, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Saxton et al. (2019) David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019.  Analysing mathematical reasoning abilities of neural models.  In _Proceedings of the 7th International Conference on Learning Representations (ICLR)_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Schmidhuber (1990) Jürgen Schmidhuber. 1990.  Towards compositional learning in dynamic networks.  Technical report, Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA).
  * Scialom et al. (2020) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020.  MLSUM: The multilingual summarization corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8051–8067, Online. Association for Computational Linguistics.
  * Shaw et al. (2021) Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021.  Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 922–938, Online. Association for Computational Linguistics.
  * Shen et al. (2021) Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 2021.  Towards out-of-distribution generalization: A survey.  _CoRR_ , abs/2108.13624.
  * Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019.  The woman worked as a babysitter: On biases in language generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3407–3412, Hong Kong, China. Association for Computational Linguistics.
  * Sinha et al. (2021) Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021.  Masked language modeling and the distributional hypothesis: Order word matters pre-training for little.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2888–2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Søgaard et al. (2021) Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2021\.  We need to talk about random splits.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1823–1832, Online. Association for Computational Linguistics.
  * Specia et al. (2020) Lucia Specia, Zhenhao Li, Juan Pino, Vishrav Chaudhary, Francisco Guzmán, Graham Neubig, Nadir Durrani, Yonatan Belinkov, Philipp Koehn, Hassan Sajjad, Paul Michel, and Xian Li. 2020.  Findings of the WMT 2020 shared task on machine translation robustness.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 76–91, Online. Association for Computational Linguistics.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _CoRR_ , abs/2206.04615.
  * Storkey (2009) Amos Storkey. 2009.  When training and test sets are different: characterizing learning transfer.  _Dataset shift in machine learning_ , 30:3–28.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Sun et al. (2020) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020.  LAMOL: LAnguage MOdeling for Lifelong Language Learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Szolnok et al. (2021) Gábor Szolnok, Botond Barta, Dorina Lakatos, and Judit Ács. 2021.  Bme submission for sigmorphon 2021 shared task 0. a three step training approach with data augmentation for morphological inflection.  In _Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 268–273, Online. Association for Computational Linguistics.
  * Talat et al. (2018) Zeerak Talat, James Thorne, and Joachim Bingel. 2018.  Bridging the gaps: Multi task learning for domain transfer of hate speech detection.  In _Online harassment_ , pages 29–55. Springer.
  * Talman and Chatzikyriakidis (2019) Aarne Talman and Stergios Chatzikyriakidis. 2019.  Testing the generalization power of neural network models across NLI benchmarks.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 85–94, Florence, Italy. Association for Computational Linguistics.
  * Talmor et al. (2020) Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020.  oLMpics-on what language model pre-training captures.  _Transactions of the Association for Computational Linguistics_ , 8:743–758.
  * Tan et al. (2019) Ming Tan, Yang Yu, Haoyu Wang, Dakuo Wang, Saloni Potdar, Shiyu Chang, and Mo Yu. 2019.  Out-of-domain detection for low-resource text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3566–3572, Hong Kong, China. Association for Computational Linguistics.
  * Tian et al. (2021) Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021\.  Diagnosing the first-order logical reasoning ability through LogicNLI.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3738–3747, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tu et al. (2020) Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020.  An empirical study on robustness to spurious correlations using pre-trained language models.  _Transactions of the Association for Computational Linguistics_ , 8:621–633.
  * Varab and Schluter (2021) Daniel Varab and Natalie Schluter. 2021.  MassiveSumm: a very large-scale, very multilingual, news summarisation dataset.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10150–10161, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Veldhoen et al. (2016) Sara Veldhoen, Dieuwke Hupkes, and Willem Zuidema. 2016.  Diagnostic classifiers: Revealing how neural networks process hierarchical structure.  In _Proceedings of the NIPS2016 Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches_.
  * Vylomova et al. (2020) Ekaterina Vylomova, Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky, Tiago Pimentel, Lucas Torroba Hennigen, Christo Kirov, Garrett Nicolai, Adina Williams, Antonios Anastasopoulos, Hilaria Cruz, Eleanor Chodroff, Ryan Cotterell, Miikka Silfverberg, and Mans Hulden. 2020.  SIGMORPHON 2020 shared task 0: Typologically diverse morphological inflection.  In _Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 1–39, Online. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2021) Bailin Wang, Mirella Lapata, and Ivan Titov. 2021.  Meta-learning for domain generalization in semantic parsing.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 366–379, Online. Association for Computational Linguistics.
  * Wang et al. (2017a) Rui Wang, Andrew Finch, Masao Utiyama, and Eiichiro Sumita. 2017a.  Sentence embedding for neural machine translation domain adaptation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 560–566, Vancouver, Canada. Association for Computational Linguistics.
  * Wang et al. (2017b) Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017b.  Instance weighting for neural machine translation domain adaptation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1482–1488, Copenhagen, Denmark. Association for Computational Linguistics.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. 2022.  Benchmarking generalization via in-context instructions on 1,600+ language tasks.
  * Weber et al. (2021) Lucas Weber, Jaap Jumelet, Elia Bruni, and Dieuwke Hupkes. 2021.  Language modelling as a multi-task problem.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2049–2060, Online. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Wei et al. (2021) Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021.  Frequency effects on syntactic rule learning in transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 932–948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wilcox et al. (2021) Ethan Wilcox, Pranali Vani, and Roger Levy. 2021.  A targeted assessment of incremental processing in neural language models and humans.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 939–952, Online. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wilson and Li (2021) Colin Wilson and Jane S.Y. Li. 2021.  Were we there already? applying minimal generalization to the sigmorphon-unimorph shared task on cognitively plausible morphological inflection.  In _Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 283–291, Online. Association for Computational Linguistics.
  * Wong and Wang (2007) Francis CK Wong and William SY Wang. 2007.  Generalisation towards combinatorial productivity in language acquisition by simple recurrent networks.  In _2007 International Conference on Integration of Knowledge Intensive Multi-Agent Systems_ , pages 139–144. IEEE.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 833–844, Hong Kong, China. Association for Computational Linguistics.
  * Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022.  Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  _CoRR_ , abs/2201.05966.
  * Xu et al. (2020) Weijia Xu, Batool Haider, and Saab Mansour. 2020.  End-to-end slot alignment and recognition for cross-lingual NLU.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5052–5063, Online. Association for Computational Linguistics.
  * Yanaka et al. (2021) Hitomi Yanaka, Koji Mineshima, and Kentaro Inui. 2021.  Exploring transitivity in neural NLI models through veridicality.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 920–934, Online. Association for Computational Linguistics.
  * Yang and Deng (2020) Kaiyu Yang and Jia Deng. 2020.  Strongly incremental constituency parsing with graph neural networks.  _Advances in Neural Information Processing Systems_ , 33:21687–21698.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
  * Yogatama et al. (2019) Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019.  Learning and evaluating general linguistic intelligence.  _arXiv preprint arXiv:1901.11373_.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  SWAG: A large-scale adversarial dataset for grounded commonsense inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 93–104, Brussels, Belgium. Association for Computational Linguistics.
  * Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020.  Improving massively multilingual neural machine translation and zero-shot translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1628–1639, Online. Association for Computational Linguistics.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  PAWS: Paraphrase adversaries from word scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zhong et al. (2021) Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021.  Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2856–2878, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhou et al. (2021) Xiang Zhou, Heba Elfardy, Christos Christodoulopoulos, Thomas Butler, and Mohit Bansal. 2021.  Hidden biases in unreliable news detection datasets.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2482–2492, Online. Association for Computational Linguistics.
  * Zhou et al. (2018) Zhong Zhou, Matthias Sperber, and Alexander Waibel. 2018.  Massively parallel cross-lingual learning in low-resource target language translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 232–243, Brussels, Belgium. Association for Computational Linguistics.

##  Appendix A Annotation setup

In this section, we describe the procedures we used for the selection of the
papers in our review and their annotation.

###  A.1 Paper selection

An initial selection of manuscripts was made through a substantive preliminary
literature review by the main authors of this paper. We then carried out a
search through the ACL anthology. We started by retrieving all papers that
have the (sub)words _generalisation_ , _generalization_ , _generalise_ or
_generalize_ in their title or abstract. In Figure 7, we see that the number
of papers with those keywords grew substantially over time, both in absolute
and relative terms. We manually checked the abstracts and titles of the
resulting papers to remove those that were not, in fact, addressing a
generalisation question (for instance, because they proposed a generalisation
of a _method_ , or because they used random train–test splits). Furthermore,
we restricted ourselves to papers with one modality. We then annotated the
resulting papers using the taxonomy presented in the previous sections. During
the annotation process, we sometimes removed entries that upon further reading
did not contain generalisation experiments, and we duplicated entries that
contained multiple experiments with different values on one of our axes. The
findings presented in this section encompass in total 619 generalisation
experiments, presented in 449 papers. The full list of papers can be found in
the second bibliography at the end of this paper, as well as on our
website181818https://genbench.github.io/references. While the conclusions in
this – static – paper pertain only to this specific selection of papers, we
intend to keep expanding the number of entries on our website with existing
papers we missed or as new generalisation papers are published.

###  A.2 Annotation

The annotation of all selected papers was done collectively by the authors of
this article. Each paper was given five labels by a first annotator, one for
every axis of our taxonomy, and these labels were then checked by a second
annotator. Disagreements were discussed among the two annotators, and for
unresolved cases, a third annotator was used. As a guide, we used the diagram
presented in Figure 11. An FAQ with common questions that occurred while using
this diagram, which intends to capture our taxonomy but is naturally a
simplified version of it, can be found on our website. In addition to the
taxonomy axes values, we also annotated which task(s) the studies considered.
If a paper performed the same experiment with multiple different tasks, we
label it _multiple tasks_ , use the overarching category (e.g.  _NLU_) when
possible, or mark it as _multitask_ if the purpose is to show that a paper can
do those all at the same time. If a paper contained multiple studies with
different values on the same axis – e.g. a paper considers both cross-domain
and compositional generalisation or uses both natural shifts and synthetic
data – we record those experiments separately.

Figure 11: A graphical representation of our annotation process and an
indication of where in a paper you might find the information required to
complete the annotation. One paper can potentially contain multiple
generalisation questions – e.g. both cross-domain and cross-task
generalisation, or both generated shifts and splits using natural data. In
that case, the diagram has to be walked through twice. Of course, the diagram
is an aid that helps characterise papers but also simplifies the full
taxonomy. On our website, we keep track of common questions that arise when
using the diagram to characterise papers in an FAQ.

##  Appendix B Evaluation cards

In the main text of this paper, we have argued several times that we believe
standardisation is an important requirement for state-of-the-art evaluation in
NLP. To further push in this direction, we propose, on top of our taxonomy, a
standardised way to indicate what kind of generalisation experiments a paper
reports: _evaluation cards_. Evaluation cards (for an example, see Figure 2)
allow visualising all generalisation experiments conducted in a study in a
comprehensive way and thus to easily shows how extensively a model was
evaluated. In contrast to our review, in which multiple loci and shifts are
grouped under one category for visualisation and analysis purposes, the
evaluation cards do allow recording which shifts and loci are investigated in
the same experiments. In the example card in Figure 2, for instance, the
experiment indicated with the triangle $\bigtriangleup$ considers a covariate
shift in the finetune stage (from one language to another), but through
investigating multiple pretrained models it also investigates a label shift
from pretraining to training. On our website, we provide a tool to generate
(one or two-column) evaluation cards, that authors can include in the appendix
of their papers. In Figure 12, we show how this tool is rendered in our web
interface.

Figure 12: Our online interface generates LaTeX code for one- or two-column
evaluation cards.

##  Appendix C Author contributions

To recognise individual author contributions, we detail those contributions
below, following the Contributor Roles Taxonomy
(CRediT)191919https://www.elsevier.com/authors/policies-and-guidelines/credit-
author-statement introduced by Elsevier. Authors are listed in the order they
appear on the author list.
2121footnotetext: Work done outside of Amazon

Dieuwke Hupkes |  | Conceptualisation, Methodology, Software, Validation,
---
Formal Analysis, Investigation, Data Curation,
Writing – Original Draft, Writing – Review & Editing,
Visualisation, Supervision, Project Administration
Mario Giulianelli |  | Conceptualisation, Methodology, Validation, Formal Analysis,
---
Investigation, Data Curation, Writing – Original Draft,
Writing – Review & Editing, Project Administration
Verna Dankers |  | Conceptualisation, Software, Formal Analysis, Data Curation,
---
Writing – Review & Editing, Visualisation
Mikel Artetxe |  | Conceptualisation, Data Curation, Validation,
---
Writing – Original Draft, Writing – Review & Editing
Yanai Elazar |  | Conceptualisation, Data Curation, Writing – Original Draft
---
Writing – Review & Editing
Tiago Pimentel |  | Conceptualisation, Data Curation,
---
Writing – Original Draft, Writing – Review & Editing
Christos Christodoulopoulos2020footnotemark: 20 |  | Conceptualisation, Data Curation, Writing – Original Draft
---
Karim Lasri |  | Conceptualisation, Data Curation, Writing – Original Draft,
---
Visualisation
Naomi Saphra |  | Conceptualisation, Writing – Original Draft
---
Arabella Sinclair |  | Conceptualisation, Data Curation
---
Dennis Ulmer |  | Data Curation, Writing – Review & Editing, Visualisation
---
Florian Schottmann |  | Data Curation
---
Khuyagbaatar Batsuren |  | Data Curation
---
Kaiser Sun |  | Data Curation
---
Koustuv Sinha |  | Data Curation
---
Leila Khalatbari |  | Data Curation
---
Maria Ryskina |  | Software, Data Curation
---
Rita Frieske |  | Data Curation
---
Ryan Cotterell |  | Data Curation, Writing – Review & Editing
---
Zhijing Jin |  | Data Curation
---
Table 1: Individual author contributions using CRediT.

##  Appendix D Multi-lingual benchmarks

While the field of multilingual modelling is vast and associated with many
interesting generalisation questions, papers in this area do not often focus
explicitly on generalisation. In this section, we provide a list of the most
important available multilingual benchmarks which can be used to evaluate
cross-lingual generalisation. Multilingual benchmarks or datasets are created
in a variety of ways. Several benchmarks are created by translating
monolingual benchmarks into different languages, usually through a
professional translation service (Ponti et al., 2020; Lin et al., 2021;
Mostafazadeh et al., 2016; Yang et al., 2019; Zhang et al., 2019; Conneau et
al., 2018; Williams et al., 2018; Ebrahimi et al., 2022; Artetxe et al., 2020;
Lewis et al., 2020; Longpre et al., 2021; Xu et al., 2020; Li et al., 2021a;
FitzGerald et al., 2022). Other multilingual benchmarks, instead, have been
built by separately annotating each language via its native speakers (e.g.
Adelani et al., 2021; Clark et al., 2020; Asai et al., 2021; Muller et al.,
2021). Yet another way to construct multilingual benchmarks is to leverage
existing resources that cover multiple languages. For instance, Wikipedia has
been used as a resource to derive multilingual benchmarks (Pan et al., 2017;
Rahimi et al., 2019; Botha et al., 2020; Liu et al., 2019a), and several
multilingual summarisation datasets have been created by extracting article-
summary pairs from online newspapers or how-to guides (e.g. Nguyen and Daumé
III, 2019; Scialom et al., 2020; Ladhak et al., 2020; Hasan et al., 2021;
Varab and Schluter, 2021). Various linguistic resources have also been
exploited: for instance, the Universal Dependencies treebank (Nivre et al.,
2020) has been used to evaluate cross-lingual part-of-speech tagging, and
multilingual WordNet and Wiktionary have been used to build XL-WiC (Raganato
et al., 2020), an extension of WiC (Pilehvar and Camacho-Collados, 2019) that
reformulates word sense disambiguation in 12 languages as a binary
classification task. Finally, in the same spirit of GLUE and SuperGLUE for
English, several aggregated benchmarks include selected sets of benchmarks
previously proposed by others (e.g. Hu et al., 2020; Ruder et al., 2021; Liang
et al., 2020; Wang et al., 2022), which allow for evaluating cross-task and
cross-language generalisation simultaneously.

Figure 13: An overview figure of our literature review, including
interactions. An interactive version of this plot can be found on our website
https://genbench.github.io/visualisations. For more detailed explanations and
analyses, we refer to § 3.

##  Appendix E A concise summary of our taxonomy

For the convenience of the reader, in this section of the supplementary
materials we provide a concise summary of our generalisation taxonomy. The
taxonomy we propose is based on a detailed analysis of a large number of
existing studies on generalisation in NLP, and it includes the main five axes
along which those studies differ. The five axes capture different aspects of
generalisation studies, that together form a comprehensive picture of the
motivation and goal of the study and provide information on important choices
in the experimental setup.

The first axis of our generalisation taxonomy (§ 2.1) is the high-level
motivation for the study. The motivation of a study impacts or even determines
what type of generalisation is desirable, as well as what kind of conclusions
can be drawn from a model’s display or lack of generalisation. Furthermore,
the motivation of a study shapes its experimental design. It is therefore
important for researchers to be explicitly aware of it, to ensure that the
experimental setup aligns with the questions they seek to answer. We consider
four different types of motivations: the _practical_ motivation, the
_cognitive_ motivation, the _intrinsic_ motivation, and the _fairness and
inclusivity_ motivation.

The second axis in our taxonomy (§ 2.2) indicates the type of generalisation
the test is addressing. This axis describes on a high level what exactly it is
that a generalisation test is intended to capture, rather than considering why
or how, making it one of the most important axes of our taxonomy. In the
literature, we have found six main types of generalisation: _compositional_
generalisation, _structural_ generalisation, _cross-task_ generalisation,
_cross-lingual_ generalisation, _cross-domain_ generalisation, and
_robustness_ generalisation.

The third axis in our taxonomy (§ 2.3) describes what kind of data shift is
considered in the generalisation test. This axis adds a statistical
interpretation to our taxonomy and derives its importance from the fact that
data shift plays an essential formal role in defining and understanding
generalisation from a statistical perspective, as well as from the fact that
different types of shifts are best addressed with different kinds of
experimental setups. On the data shift axis, we consider three shifts which
are well-attested in the literature: _covariate shift_ , _label shift_ and
_full shift_. We further include two additional types of shift – assumed shift
and multiple shifts – to account for studies that cannot be labelled with any
of the three main shift types.

In the fourth axis of our taxonomy (§ 2.4), we consider what is the source of
the data shift used in the experiment. The source of the data shift determines
how much control the experimenter has over the training and testing data and,
consequently, what kind of conclusions can be drawn from an experiment. We
distinguish four different sources of shifts: _naturally occurring shifts_ ,
_artificially partitioned natural corpora_ , _generated shifts_ and _fully
generated datasets_.

In the last axis of our taxonomy (§ 2.5), we consider what is the locus of the
data shift, or, in other words, for what part of the modelling pipeline
generalisation is investigated. The locus of the shift, together with the
shift type, forms the last piece of the puzzle, as it determines what part of
the modelling pipeline is investigated and thus the kind of generalisation
question that can be asked. On this axis, we consider shifts between all
stages in the contemporary modelling pipeline – pretraining, training and
testing – as well as studies that consider shifts between multiple stages
simultaneously.

See pages 2- of lit_review_refs.pdf