* Dhingra et al. (2019) Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019.  Handling divergent reference texts when evaluating table-to-text generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4884–4895, Florence, Italy. Association for Computational Linguistics.
  * Dhole et al. (2021) Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Wu, Jascha Sohl-Dickstein, Jinho D. Choi, Eduard Hovy, Ondrej Dusek, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Banjade, Lisa Barthe, Hanna Behnke, Ian Berlot-Attwell, Connor Boyle, Caroline Brun, Marco Antonio Sobrevilla Cabezudo, Samuel Cahyawijaya, Emile Chapuis, Wanxiang Che, Mukund Choudhary, Christian Clauss, Pierre Colombo, Filip Cornell, Gautier Dagan, Mayukh Das, Tanay Dixit, Thomas Dopierre, Paul-Alexis Dray, Suchitra Dubey, Tatiana Ekeinhor, Marco Di Giovanni, Rishabh Gupta, Rishabh Gupta, Louanes Hamla, Sang Han, Fabrice Harel-Canada, Antoine Honore, Ishan Jindal, Przemyslaw K. Joniak, Denis Kleyko, Venelin Kovatchev, Kalpesh Krishna, Ashutosh Kumar, Stefan Langer, Seungjae Ryan Lee, Corey James Levinson, Hualou Liang, Kaizhao Liang, Zhexiong Liu, Andrey Lukyanenko, Vukosi Marivate, Gerard de Melo, Simon Meoni, Maxime Meyer, Afnan Mir, Nafise Sadat Moosavi, Niklas Muennighoff, Timothy Sum Hon Mun, Kenton Murray, Marcin Namysl, Maria Obedkova, Priti Oli, Nivranshu Pasricha, Jan Pfister, Richard Plant, Vinay Prabhu, Vasile Pais, Libo Qin, Shahab Raji, Pawan Kumar Rajpoot, Vikas Raunak, Roy Rinberg, Nicolas Roberts, Juan Diego Rodriguez, Claude Roux, Vasconcellos P. H. S., Ananya B. Sai, Robin M. Schmidt, Thomas Scialom, Tshephisho Sefara, Saqib N. Shamsi, Xudong Shen, Haoyue Shi, Yiwen Shi, Anna Shvets, Nick Siegel, Damien Sileo, Jamie Simon, Chandan Singh, Roman Sitelew, Priyank Soni, Taylor Sorensen, William Soto, Aman Srivastava, KV Aditya Srivatsa, Tony Sun, Mukund Varma T, A Tabassum, Fiona Anting Tan, Ryan Teehan, Mo Tiwari, Marie Tolkiehn, Athena Wang, Zijian Wang, Gloria Wang, Zijie J. Wang, Fuxuan Wei, Bryan Wilie, Genta Indra Winata, Xinyi Wu, Witold Wydmański, Tianbao Xie, Usama Yaseen, M. Yee, Jing Zhang, and Yue Zhang. 2021\.  NL-Augmenter: A framework for task-sensitive natural language augmentation.
  * Dinan et al. (2021) Emily Dinan, Gavin Abercrombie, A. Stevie Bergman, Shannon L. Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2021.  Anticipating safety issues in E2E conversational AI: framework and tooling.  _CoRR_ , abs/2107.03451.
  * Djolonga et al. (2020) Josip Djolonga, Mario Lucic, Marco Cuturi, Olivier Bachem, Olivier Bousquet, and Sylvain Gelly. 2020.  Precision-recall curves using information divergence frontiers.  In _The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]_ , volume 108 of _Proceedings of Machine Learning Research_ , pages 2550–2559. PMLR.
  * Dodge et al. (2019) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019\.  Show your work: Improved reporting of experimental results.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2185–2194, Hong Kong, China. Association for Computational Linguistics.
  * Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1286–1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dorr et al. (2005) Bonnie Dorr, Christof Monz, Stacy President, Richard Schwartz, and David Zajic. 2005\.  A methodology for extrinsic evaluation of text summarization: Does ROUGE correlate? In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 1–8, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_.
  * Dras (2015) Mark Dras. 2015.  Squibs: Evaluating human pairwise preference judgments.  _Computational Linguistics_ , 41(2):309–317.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Dušek et al. (2019) Ondřej Dušek, David M. Howcroft, and Verena Rieser. 2019.  Semantic noise matters for neural natural language generation.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 421–426, Tokyo, Japan. Association for Computational Linguistics.
  * Dušek et al. (2018) Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018.  Findings of the E2E NLG challenge.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 322–328, Tilburg University, The Netherlands. Association for Computational Linguistics.
  * Dusek et al. (2020) Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2020.  Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG challenge.  _Comput. Speech Lang._ , 59:123–156.
  * Ethayarajh and Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky. 2020.  Utility is in the eye of the user: A critique of NLP leaderboards.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4846–4853, Online. Association for Computational Linguistics.
  * Eyal et al. (2019) Matan Eyal, Tal Baumel, and Michael Elhadad. 2019.  Question answering as an automatic evaluation metric for news article summarization.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3938–3948, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021.  SummEval: Re-evaluating Summarization Evaluation.  _Transactions of the Association for Computational Linguistics_ , 9:391–409.
  * Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2214–2220, Florence, Italy. Association for Computational Linguistics.
  * Fort et al. (2011) Karën Fort, Gilles Adda, and K. Bretonnel Cohen. 2011.  Amazon Mechanical Turk: Gold Mine or Coal Mine? _Computational Linguistics_ , 37(2):413–420.
  * Freitag et al. (2021a) Markus Freitag, George F. Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _CoRR_ , abs/2104.14478.
  * Freitag et al. (2020) Markus Freitag, David Grangier, and Isaac Caswell. 2020.  BLEU might be guilty but references are not innocent.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 61–71, Online. Association for Computational Linguistics.
  * Freitag et al. (2021b) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , Online. Association for Computational Linguistics.
  * Gabriel et al. (2021) Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021\.  GO FIGURE: A meta evaluation of factuality in summarization.  In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 478–487. Association for Computational Linguistics.
  * Gatt et al. (2008) Albert Gatt, Anja Belz, and Eric Kow. 2008.  The TUNA challenge 2008: Overview and evaluation results.  In _Proceedings of the Fifth International Natural Language Generation Conference_ , pages 198–206, Salt Fork, Ohio, USA. Association for Computational Linguistics.
  * Gatt and Krahmer (2018) Albert Gatt and Emiel Krahmer. 2018.  Survey of the state of the art in natural language generation: Core tasks, applications and evaluation.  _J. Artif. Intell. Res._ , 61:65–170.
  * Gebru et al. (2021) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021.  Datasheets for datasets.  _Communications of the ACM_ , 64(12):86–92.
  * Gehrmann (2020) Sebastian Gehrmann. 2020.  _Human-AI Collaboration for Natural Language Generation with Interpretable Neural Networks_.  Ph.D. thesis, Harvard University.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021.  The GEM benchmark: Natural language generation, its evaluation and metrics.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 96–120, Online. Association for Computational Linguistics.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.  Bottom-up abstractive summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4098–4109, Brussels, Belgium. Association for Computational Linguistics.
  * Gehrmann et al. (2019a) Sebastian Gehrmann, Steven Layne, and Franck Dernoncourt. 2019a.  Improving human text comprehension through semi-Markov CRF-based neural section title generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1677–1688, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gehrmann et al. (2019b) Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019b.  GLTR: Statistical detection and visualization of generated text.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 111–116, Florence, Italy. Association for Computational Linguistics.
  * Giannakopoulos et al. (2015) George Giannakopoulos, Jeff Kubina, John Conroy, Josef Steinberger, Benoit Favre, Mijail Kabadjov, Udo Kruschwitz, and Massimo Poesio. 2015.  MultiLing 2015: Multilingual summarization of single and multi-documents, on-line fora, and call-center conversations.  In _Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 270–274, Prague, Czech Republic. Association for Computational Linguistics.
  * Gillick and Liu (2010) Dan Gillick and Yang Liu. 2010.  Non-expert evaluation of summarization systems is risky.  In _Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk_ , pages 148–151, Los Angeles. Association for Computational Linguistics.
  * Gkatzia and Mahamood (2015) Dimitra Gkatzia and Saad Mahamood. 2015.  A snapshot of NLG evaluation practices 2005 - 2014.  In _Proceedings of the 15th European Workshop on Natural Language Generation (ENLG)_ , pages 57–60, Brighton, UK. Association for Computational Linguistics.
  * Goel et al. (2021) Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christopher Ré. 2021.  Robustness gym: Unifying the NLP evaluation landscape.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations_ , pages 42–55, Online. Association for Computational Linguistics.
  * Gorman and Bedrick (2019) Kyle Gorman and Steven Bedrick. 2019.  We need to talk about standard splits.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2786–2791, Florence, Italy. Association for Computational Linguistics.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1449–1462, Online. Association for Computational Linguistics.
  * Graham (2015) Yvette Graham. 2015.  Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 128–137, Lisbon, Portugal. Association for Computational Linguistics.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 708–719, New Orleans, Louisiana. Association for Computational Linguistics.
  * Harman and Over (2004) Donna Harman and Paul Over. 2004.  The effects of human variation in DUC summarization evaluation.  In _Text Summarization Branches Out_ , pages 10–17, Barcelona, Spain. Association for Computational Linguistics.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021.  XL-sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4693–4703, Online. Association for Computational Linguistics.
  * Hashimoto et al. (2019) Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. 2019.  Unifying human and statistical evaluation for natural language generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1689–1701, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Higashinaka et al. (2015) Ryuichiro Higashinaka, Masahiro Mizukami, Kotaro Funakoshi, Masahiro Araki, Hiroshi Tsukahara, and Yuka Kobayashi. 2015.  Fatal or not? finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2243–2248, Lisbon, Portugal. Association for Computational Linguistics.
  * Hirschman (1998) Lynette Hirschman. 1998.  The evolution of evaluation: Lessons from the message understanding conferences.  _Comput. Speech Lang._ , 12(4):281–305.
  * Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $q^2$: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 7856–7870. Association for Computational Linguistics.
  * Hovy et al. (2005) Eduard Hovy, Chin-Yew Lin, and Liang Zhou. 2005.  Evaluating duc 2005 using basic elements.  In _Proceedings of DUC_ , volume 2005. Citeseer.
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Howcroft and Rieser (2021) David M. Howcroft and Verena Rieser. 2021.  What happens if you treat ordinal ratings as interval data? human evaluations in NLP are even more under-powered than you think.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8932–8939, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hutchinson et al. (2021) Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021.  Towards accountability for machine learning datasets: Practices from software engineering and infrastructure.  In _FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021_ , pages 560–575. ACM.
  * Huynh et al. (2021) Jessica Huynh, Jeffrey Bigham, and Maxine Eskénazi. 2021.  A survey of nlp-related crowdsourcing hits: what works and what does not.  _CoRR_ , abs/2111.05241.
  * Ippolito et al. (2020) Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020.  Automatic detection of generated text is easiest when humans are fooled.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1808–1822, Online. Association for Computational Linguistics.
  * Iskender et al. (2020) Neslihan Iskender, Tim Polzehl, and Sebastian Möller. 2020.  Best practices for crowd-based evaluation of German summarization: Comparing crowd, expert and automatic evaluation.  In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_ , pages 164–175, Online. Association for Computational Linguistics.
  * Jones and Galliers (1995) Karen Sparck Jones and Julia R Galliers. 1995.  _Evaluating natural language processing systems: An analysis and review_ , volume 1083.  Springer Science & Business Media.
  * Jørgensen et al. (2015) Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015.  Challenges of studying and processing dialects in social media.  In _Proceedings of the Workshop on Noisy User-generated Text_ , pages 9–18, Beijing, China. Association for Computational Linguistics.
  * Jørgensen et al. (2016) Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016.  Learning a POS tagger for AAVE-like language.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1115–1120, San Diego, California. Association for Computational Linguistics.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kane et al. (2020) Hassan Kane, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. 2020.  NUBIA: NeUral based interchangeability assessor for text generation.  In _Proceedings of the 1st Workshop on Evaluating NLG Evaluation_ , pages 28–37, Online (Dublin, Ireland). Association for Computational Linguistics.
  * Karpinska et al. (2021) Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021.  The perils of using Mechanical Turk to evaluate open-ended text generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kasai et al. (2021) Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander R. Fabbri, Yejin Choi, and Noah A. Smith. 2021.  Bidimensional leaderboards: Generate and evaluate language hand in hand.  _CoRR_ , abs/2112.04139.
  * Kaster et al. (2021) Marvin Kaster, Wei Zhao, and Steffen Eger. 2021.  Global explainability of bert-based evaluation metrics by disentangling along linguistic factors.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 8912–8925. Association for Computational Linguistics.
  * Kedzie et al. (2018) Chris Kedzie, Kathleen McKeown, and Hal Daumé III. 2018.  Content selection in deep learning models of summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1818–1828, Brussels, Belgium. Association for Computational Linguistics.
  * Khashabi et al. (2021) Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. 2021.  GENIE: A leaderboard for human-in-the-loop evaluation of text generation.  _CoRR_ , abs/2101.06561.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021.  Dynabench: Rethinking benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4110–4124, Online. Association for Computational Linguistics.
  * Kim et al. (2021) Jihyuk Kim, Myeongho Jeong, Seungtaek Choi, and Seung-won Hwang. 2021.  Structure-augmented keyphrase generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2657–2667, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975\.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.  Technical report, Naval Technical Training Command Millington TN Research Branch.
  * Kiritchenko and Mohammad (2017) Svetlana Kiritchenko and Saif Mohammad. 2017.  Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 465–470, Vancouver, Canada. Association for Computational Linguistics.
  * Koch et al. (2021) Bernard Koch, Emily Denton, Alex Hanna, and Jacob G. Foster. 2021.  Reduced, reused and recycled: The life of a dataset in machine learning research.  _CoRR_ , abs/2112.01716.
  * Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021.  To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 478–494, Online. Association for Computational Linguistics.
  * Koehn et al. (2019) Philipp Koehn, Francisco Guzmán, Vishrav Chaudhary, and Juan Pino. 2019.  Findings of the WMT 2019 shared task on parallel corpus filtering for low-resource conditions.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2)_ , pages 54–72, Florence, Italy. Association for Computational Linguistics.
  * Koehn and Monz (2006) Philipp Koehn and Christof Monz. 2006.  Manual and automatic evaluation of machine translation between European languages.  In _Proceedings on the Workshop on Statistical Machine Translation_ , pages 102–121, New York City. Association for Computational Linguistics.
  * Krishna et al. (2021) Kundan Krishna, Sopan Khosla, Jeffrey Bigham, and Zachary C. Lipton. 2021.  Generating SOAP notes from doctor-patient conversations using modular summarization techniques.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4958–4972, Online. Association for Computational Linguistics.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Kusner et al. (2015) Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015.  From word embeddings to document distances.  In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pages 957–966. JMLR.org.
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4034–4048, Online. Association for Computational Linguistics.
  * Langkilde and Knight (1998) Irene Langkilde and Kevin Knight. 1998.  Generation that exploits corpus-based statistical knowledge.  In _36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1_ , pages 704–710, Montreal, Quebec, Canada. Association for Computational Linguistics.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213, Austin, Texas. Association for Computational Linguistics.
  * Lee et al. (2021) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021.  Deduplicating training data makes language models better.  _CoRR_ , abs/2107.06499.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021.  Datasets: A community library for natural language processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liao et al. (2021) Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. 2021.  Are we learning yet? a meta review of evaluation failures across machine learning.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin and Och (2004) Chin-Yew Lin and Franz Josef Och. 2004.  ORANGE: a method for evaluating automatic evaluation metrics for machine translation.  In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_ , pages 501–507, Geneva, Switzerland. COLING.
  * Liu et al. (2021) Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. 2021.  ExplainaBoard: An explainable leaderboard for NLP.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 280–289, Online. Association for Computational Linguistics.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical transformers for multi-document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5070–5081, Florence, Italy. Association for Computational Linguistics.
  * Lommel et al. (2014) Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014.  Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics.  _Tradumàtica_ , 1(12):0455–463.
  * Luccioni and Viviano (2021) Alexandra Luccioni and Joseph Viviano. 2021.  What’s in the box? an analysis of undesirable content in the Common Crawl corpus.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 182–189, Online. Association for Computational Linguistics.
  * Lyu et al. (2021) Xinglin Lyu, Junhui Li, Zhengxian Gong, and Min Zhang. 2021.  Encouraging lexical translation consistency for document-level neural machine translation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3265–3277, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ma et al. (2018) Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018.  Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 671–688, Belgium, Brussels. Association for Computational Linguistics.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.  Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 62–90, Florence, Italy. Association for Computational Linguistics.
  * Macháček and Bojar (2014) Matouš Macháček and Ondřej Bojar. 2014.  Results of the WMT14 metrics shared task.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 293–301, Baltimore, Maryland, USA. Association for Computational Linguistics.
  * Mani et al. (1999) Inderjeet Mani, David House, Gary Klein, Lynette Hirschman, Therese Firmin, and Beth Sundheim. 1999.  The TIPSTER SUMMAC text summarization evaluation.  In _Ninth Conference of the European Chapter of the Association for Computational Linguistics_ , pages 77–85, Bergen, Norway. Association for Computational Linguistics.
  * Mathur et al. (2020) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020\.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * McMillan-Major et al. (2021) Angelina McMillan-Major, Salomey Osei, Juan Diego Rodriguez, Pawan Sasanka Ammanamanchi, Sebastian Gehrmann, and Yacine Jernite. 2021.  Reusable templates and guides for documenting datasets and models for natural language processing and generation: A case study of the HuggingFace and GEM data and model cards.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 121–135, Online. Association for Computational Linguistics.
  * Mille et al. (2021) Simon Mille, Kaustubh Dhole, Saad Mahamood, Laura Perez-Beltrachini, Varun Gangal, Mihir Kale, Emiel van Miltenburg, and Sebastian Gehrmann. 2021.  Automatic construction of evaluation suites for natural language generation datasets.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, GA, USA, January 29-31, 2019_ , pages 220–229. ACM.
  * Moradi and Samwald (2021) Milad Moradi and Matthias Samwald. 2021.  Evaluating the robustness of neural language models to input perturbations.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 1558–1570. Association for Computational Linguistics.
  * Nakayama and Halualani (2011) Thomas K Nakayama and Rona Tamiko Halualani. 2011.  _The handbook of critical intercultural communication_.  John Wiley & Sons.
  * Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_ , pages 3075–3081. AAAI Press.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu̇lçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Napoles et al. (2012) Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012.  Annotated Gigaword.  In _Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX)_ , pages 95–100, Montréal, Canada. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Narayan et al. (2021) Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simoes, Vitaly Nikolaev, and Ryan McDonald. 2021.  Planning with learned entity prompts for abstractive summarization.  _arXiv preprint arXiv:2104.07606_.
  * Nenkova (2005) Ani Nenkova. 2005.  Automatic text summarization of newswire: Lessons learned from the document understanding conference.  In _Proceedings, The Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Applications of Artificial Intelligence Conference, July 9-13, 2005, Pittsburgh, Pennsylvania, USA_ , pages 1436–1441. AAAI Press / The MIT Press.
  * Nenkova and Passonneau (2004) Ani Nenkova and Rebecca Passonneau. 2004.  Evaluating content selection in summarization: The pyramid method.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 145–152, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Novikova et al. (2017a) Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017a.  Why we need new evaluation metrics for NLG.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics.
  * Novikova et al. (2017b) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017b.  The E2E dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206, Saarbrücken, Germany. Association for Computational Linguistics.
  * Novikova et al. (2018) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2018.  RankME: Reliable human ratings for natural language generation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 72–78, New Orleans, Louisiana. Association for Computational Linguistics.
  * Over and Yen (2004) Paul Over and James Yen. 2004.  An introduction to duc-2004.  In _Proceedings of the Document Understanding Conference_.
  * Owczarzak et al. (2012) Karolina Owczarzak, Peter A. Rankel, Hoa Trang Dang, and John M. Conroy. 2012.  Assessing the effect of inconsistent assessors on summarization evaluation.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 359–362, Jeju Island, Korea. Association for Computational Linguistics.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4812–4829, Online. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020.  ToTTo: A controlled table-to-text generation dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1173–1186, Online. Association for Computational Linguistics.
  * Pasunuru and Bansal (2018) Ramakanth Pasunuru and Mohit Bansal. 2018.  Multi-reward reinforced summarization with saliency and entailment.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 646–653, New Orleans, Louisiana. Association for Computational Linguistics.
  * Paullada et al. (2020) Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2020.  Data and its (dis)contents: A survey of dataset development and use in machine learning research.  _CoRR_ , abs/2012.05345.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _International Conference on Learning Representations_.
  * Perez-Beltrachini and Lapata (2021) Laura Perez-Beltrachini and Mirella Lapata. 2021.  Models and datasets for cross-lingual summarisation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9408–9423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Peyrard (2019) Maxime Peyrard. 2019.  Studying summarization evaluation metrics in the appropriate scoring range.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5093–5100, Florence, Italy. Association for Computational Linguistics.
  * Phang et al. (2021) Jason Phang, Angelica Chen, William Huang, and Samuel R. Bowman. 2021.  Adversarially constructed evaluation sets are more challenging, but may not be fair.  _CoRR_ , abs/2111.08181.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaïd Harchaoui. 2021.  MAUVE: measuring the gap between neural text and human text using divergence frontiers.  _CoRR_ , abs/2102.01454.
  * Pitler et al. (2010) Emily Pitler, Annie Louis, and Ani Nenkova. 2010.  Automatic evaluation of linguistic quality in multi-document summarization.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 544–554, Uppsala, Sweden. Association for Computational Linguistics.
  * Popovic and Ney (2011) Maja Popovic and Hermann Ney. 2011.  Towards automatic error analysis of machine translation output.  _Comput. Linguistics_ , 37(4):657–688.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Prabhakaran et al. (2021) Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021.  On releasing annotator-level labels and information in datasets.  In _Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop_ , pages 133–138, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Pratapa et al. (2021) Adithya Pratapa, Antonios Anastasopoulos, Shruti Rijhwani, Aditi Chaudhary, David R. Mortensen, Graham Neubig, and Yulia Tsvetkov. 2021.  Evaluating the morphosyntactic well-formedness of generated texts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 7131–7150. Association for Computational Linguistics.
  * Pu et al. (2021) Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021\.  Learning compact metrics for MT.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 751–762, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Puduppully et al. (2019) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.  Data-to-text generation with content selection and planning.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 6908–6915. AAAI Press.
  * Quiñonero-Candela et al. (2009) Joaquin Quiñonero-Candela, Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer. 2009.  _Dataset shift in machine learning_.  Mit Press.
  * Rankel et al. (2013) Peter A. Rankel, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2013.  A decade of automatic content evaluation of news summaries: Reassessing the state of the art.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 131–136, Sofia, Bulgaria. Association for Computational Linguistics.
  * Rankel et al. (2012) Peter A. Rankel, John M. Conroy, and Judith D. Schlesinger. 2012.  Better metrics to automatically predict the quality of a text summary.  _Algorithms_ , 5(4):398–420.
  * Rashkin et al. (2021) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021.  Measuring attribution in natural language generation models.  _CoRR_ , abs/2112.12870.
  * Rebuffel et al. (2021) Clément Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari. 2021.  Data-questeval: A referenceless metric for data-to-text semantic evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 8029–8036. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Reiter (2018) Ehud Reiter. 2018.  A structured review of the validity of BLEU.  _Computational Linguistics_ , 44(3):393–401.
  * Reiter and Belz (2009a) Ehud Reiter and Anja Belz. 2009a.  An investigation into the validity of some metrics for automatically evaluating natural language generation systems.  _Comput. Linguistics_ , 35(4):529–558.
  * Reiter and Belz (2009b) Ehud Reiter and Anja Belz. 2009b.  An investigation into the validity of some metrics for automatically evaluating natural language generation systems.  _Computational Linguistics_ , 35(4):529–558.
  * Reiter and Dale (1997) Ehud Reiter and Robert Dale. 1997.  Building applied natural language generation systems.  _Natural Language Engineering_ , 3(1):57–87.
  * Reiter and Sripada (2002) Ehud Reiter and Somayajulu Sripada. 2002.  Should corpora texts be gold standards for NLG? In _Proceedings of the International Natural Language Generation Conference_ , pages 97–104, Harriman, New York, USA. Association for Computational Linguistics.
  * Reiter and Thomson (2020) Ehud Reiter and Craig Thomson. 2020.  Shared task on evaluating accuracy.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 227–231, Dublin, Ireland. Association for Computational Linguistics.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Rogers (2021) Anna Rogers. 2021.  Changing the world by changing the data.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2182–2194, Online. Association for Computational Linguistics.
  * Rogers and Augenstein (2020) Anna Rogers and Isabelle Augenstein. 2020.  What can we do to improve peer review in NLP? In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1256–1262, Online. Association for Computational Linguistics.
  * Rogers et al. (2021) Anna Rogers, Timothy Baldwin, and Kobi Leins. 2021.  ‘just what do you think you’re doing, dave?’ a checklist for responsible data use in NLP.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4821–4833, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Rosa and Flores (2017) Jonathan Rosa and Nelson Flores. 2017.  Unsettling race and language: Toward a raciolinguistic perspective.  _Language in Society_ , 46:621 – 647.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
  * Saggion et al. (2010) Horacio Saggion, Juan-Manuel Torres-Moreno, Iria da Cunha, Eric SanJuan, and Patricia Velázquez-Morales. 2010.  Multilingual summarization evaluation without human models.  In _Coling 2010: Posters_ , pages 1059–1067, Beijing, China. Coling 2010 Organizing Committee.
  * Sai et al. (2021) Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021.  Perturbation checklists for evaluating NLG evaluation metrics.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 7219–7234. Association for Computational Linguistics.
  * Sambasivan et al. (2021) Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen K. Paritosh, and Lora Aroyo. 2021.  ”everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI.  In _CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021_ , pages 39:1–39:15. ACM.
  * Scheuerman et al. (2021) Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton. 2021.  Do datasets have politics? disciplinary values in computer vision dataset development.  _Proceedings of the Conference on ACM Human Computer Interaction_ , 5(CSCW2):1–37.
  * Schmidt (2013) Florian Alexander Schmidt. 2013.  The good, the bad and the ugly: Why crowdsourcing needs ethics.  In _2013 International Conference on Cloud and Green Computing, Karlsruhe, Germany, September 30 - October 2, 2013_ , pages 531–535. IEEE Computer Society.
  * Schoch et al. (2020) Stephanie Schoch, Diyi Yang, and Yangfeng Ji. 2020.  “this is a problem, don’t you agree?” framing and bias in human evaluation for natural language generation.  In _Proceedings of the 1st Workshop on Evaluating NLG Evaluation_ , pages 10–16, Online (Dublin, Ireland). Association for Computational Linguistics.
  * Scialom et al. (2020) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020.  MLSUM: The multilingual summarization corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8051–8067, Online. Association for Computational Linguistics.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021.  Questeval: Summarization asks for fact-based evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 6594–6604. Association for Computational Linguistics.
  * Scialom et al. (2019) Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019\.  Answers unite! unsupervised metrics for reinforced summarization models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3246–3256, Hong Kong, China. Association for Computational Linguistics.
  * Scott and Moore (2007) Donia Scott and Johanna Moore. 2007.  An nlg evaluation competition? eight reasons to be cautious.  In _Proceedings of the Workshop on Shared Tasks and Comparative Evaluation in Natural Language Generation_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shapira et al. (2019) Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019.  Crowdsourcing lightweight pyramids for manual summary evaluation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 682–687, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Sheng et al. (2008) Victor S. Sheng, Foster J. Provost, and Panagiotis G. Ipeirotis. 2008.  Get another label? improving data quality and data mining using multiple, noisy labelers.  In _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008_ , pages 614–622. ACM.
  * Shimorina and Belz (2021) Anastasia Shimorina and Anya Belz. 2021.  The human evaluation datasheet 1.0: A template for recording details of human evaluation experiments in NLP.  _CoRR_ , abs/2103.09710.
  * Shmueli et al. (2021) Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021.  Beyond fair pay: Ethical implications of NLP crowdsourcing.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3758–3769, Online. Association for Computational Linguistics.
  * Shu et al. (2021) Raphael Shu, Kang Min Yoo, and Jung-Woo Ha. 2021.  Reward optimization for neural machine translation with learned metrics.  _CoRR_ , abs/2104.07541.
  * Smith et al. (2022) Eric Michael Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022.  Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents.  _CoRR_ , abs/2201.04723.
  * Snow et al. (2008) Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008.  Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 254–263, Honolulu, Hawaii. Association for Computational Linguistics.
  * Søgaard et al. (2021) Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2021\.  We need to talk about random splits.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1823–1832, Online. Association for Computational Linguistics.
  * Stanojević et al. (2015) Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015\.  Results of the WMT15 metrics shared task.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 256–273, Lisbon, Portugal. Association for Computational Linguistics.
  * Stent et al. (2005) Amanda Stent, Matthew Marge, and Mohit Singhai. 2005.  Evaluating evaluation methods for generation in the presence of variation.  In _Computational Linguistics and Intelligent Text Processing, 6th International Conference, CICLing 2005, Mexico City, Mexico, February 13-19, 2005, Proceedings_ , volume 3406 of _Lecture Notes in Computer Science_ , pages 341–351. Springer.
  * Stevens-Guille et al. (2020) Symon Stevens-Guille, Aleksandre Maskharashvili, Amy Isard, Xintong Li, and Michael White. 2020.  Neural NLG for methodius: From RST meaning representations to texts.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 306–315, Dublin, Ireland. Association for Computational Linguistics.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020.  Learning to summarize from human feedback.  _CoRR_ , abs/2009.01325.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Sun et al. (2019) Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019.  How to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature.  In _Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation_ , pages 21–29, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Svensson (2000) Elisabeth Svensson. 2000.  Comparison of the quality of assessments using continuous and discrete ordinal rating scales.  _Biometrical Journal: Journal of Mathematical Methods in Biosciences_ , 42(4):417–434.
  * Syed et al. (2019) Shahbaz Syed, Michael Völske, Nedim Lipka, Benno Stein, Hinrich Schütze, and Martin Potthast. 2019.  Towards summarization for social media - results of the TL;DR challenge.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 523–528, Tokyo, Japan. Association for Computational Linguistics.
  * Tauchmann and Mieskes (2020) Christopher Tauchmann and Margot Mieskes. 2020.  Language agnostic automatic summarization evaluation.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 6656–6662, Marseille, France. European Language Resources Association.
  * Tenney et al. (2020) Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020.  The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 107–118, Online. Association for Computational Linguistics.
  * Thompson and Post (2020) Brian Thompson and Matt Post. 2020.  Automatic machine translation evaluation in many languages via zero-shot paraphrasing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 90–121, Online. Association for Computational Linguistics.
  * Thomson and Reiter (2020) Craig Thomson and Ehud Reiter. 2020.  A gold standard methodology for evaluating accuracy in data-to-text systems.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 158–168, Dublin, Ireland. Association for Computational Linguistics.
  * Thomson and Reiter (2021) Craig Thomson and Ehud Reiter. 2021.  Generation challenges: Results of the accuracy evaluation shared task.  In _Proceedings of the 14th International Conference on Natural Language Generation, INLG 2021, Aberdeen, Scotland, UK, 20-24 September, 2021_ , pages 240–248. Association for Computational Linguistics.
  * van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021.  Human evaluation of automatically generated text: Current trends and best practice guidelines.  _Computer Speech & Language_, 67:101151.
  * van der Lee et al. (2019) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019.  Best practices for the human evaluation of automatically generated text.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 355–368, Tokyo, Japan. Association for Computational Linguistics.
  * van Miltenburg et al. (2021) Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, and Luou Wen. 2021.  Underreporting of errors in NLG output, and what to do about it.  In _Proceedings of the 14th International Conference on Natural Language Generation_ , pages 140–153, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Watson (2019) David Watson. 2019.  The rhetoric and reality of anthropomorphism in artificial intelligence.  _Minds Mach._ , 29(3):417–440.
  * Wegmann and Nguyen (2021) Anna Wegmann and Dong Nguyen. 2021.  Does it capture STEL? a modular, similarity-based linguistic style evaluation framework.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7109–7130, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wei and Jia (2021) Johnny Wei and Robin Jia. 2021.  The statistical advantage of automatic NLG metrics at the system level.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6840–6854, Online. Association for Computational Linguistics.
  * Wieting et al. (2019) John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019.  Beyond BLEU:training neural machine translation with semantic similarity.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4344–4355, Florence, Italy. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017.  Challenges in data-to-document generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics.
  * Xie et al. (2021) Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021.  Factual consistency evaluation for text summarization via counterfactual estimation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 100–110, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Yuan et al. (2021) Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. 2021.  Synthbio: A case study in faster curation of text datasets.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019.  Defending against neural fake news.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 9051–9062.
  * Zeng et al. (2021) Zhiyuan Zeng, Jiaze Chen, Weiran Xu, and Lei Li. 2021.  Gradient-based adversarial factual consistency evaluation for abstractive summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 4102–4108. Association for Computational Linguistics.
  * Zhang and Bansal (2021) Shiyue Zhang and Mohit Bansal. 2021.  Finding a balanced degree of automation for summary evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 6617–6632. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Surveying recent ACL, INLG, and EMNLP papers

Here we describe the annotation instructions for our analysis of 66 ACL,
EMNLP, and INLG papers from 2021. The instructions were defined such that
results are an upper bound to the criteria. We avoid judging quality of a
particular evaluation aspect and instead only annotate its presence.

###  A.1 Paper selection

The analyzed papers were selected from the proceedings of the three
conferences. A paper was selected if it included in its title any reference to
working on a generation problem. To avoid an over-emphasis on machine
translation in the analysis, we did not specifically select translation papers
unless their title was related to generation as a whole. As a result, about
10–15 translation papers were part of the analysis which we considered an
appropriate amount. The selected papers were subsequently filtered if they did
not provide modeling results, e.g., because it was an analysis-focused paper
or if the modeling task was not a generation task covered by our definition in
section 2.

###  A.2 Instructions

Make informed evaluation choices and document them

  * •

Evaluate on multiple datasets: Select yes if the paper reports results on more
than one dataset. Select N/A if the paper explicitly states that there is only
one dataset available for the addressed task.

  * •

Motivate dataset choice(s): Select yes if the paper states why each particular
dataset was chosen. If the only reasoning is that previous work uses it,
select no. If the paper introduces a dataset, select N/A.

  * •

Motivate metric choice(s): Select yes if the paper states why each particular
metric was chosen. If the only reasoning is that previous work uses it, select
no.

  * •

Evaluate on non-English language: If at least one of the evaluated datasets
includes non-English language, select yes.

Measure specific generation effects

  * •

Use a combination of metrics from at least two different categories: Select
yes, if the automatic evaluation results include at least two metrics from
different families (e.g., one QA-based one and one lexical one). Reporting
ROUGE and BLEU would not count while ROUGE and BLEURT would.

  * •

Avoid claims about overall “quality”: Select no if the abstract of the paper
reports improvements generally and not in terms of specific generation aspects
(e.g., “we outperform baselines”)

  * •

Discuss limitations of using the proposed method: Select yes, if there is at
least one paragraph dedicated to the limitations of the proposed method in the
results or discussion section or as its own section.

Analyze and address issues in the used dataset(s)

  * •

Discuss or identify issues with the data: Select yes, if there is at least a
mention of problematic artefacts with the data or what or who it represents.

  * •

Contribute to the data documentation or create it if it does not yet exist:
Select yes, if the paper is accompanied by a data card or if there is a
mention that original documentation was updated.

  * •

Address these issues and release an updated version: Select yes, if the paper
is accompanied by a release of updated data or points to a loader that
retrieves the updated dataset. If the paper introduces a dataset, select N/A.

  * •

Create targeted evaluation suite(s): Select yes, if the paper describes the
creation of a fine-grained breakdown of subpopulations or multiple training or
test splits.

  * •

Release evaluation suite or analysis script: Select yes, if the resources in
the previous points were released in the form of data or code.

Evaluate in a comparable setting

  * •

Re-train or -implement most appropriate baselines: Select yes, if the paper
explicitly mentions that it trains or implements baselines from prior papers.

  * •

Re-compute evaluation metrics in a consistent framework: Select yes, if all
the reported scores were computed by the authors or by another centralized
framework (e.g., through upload to a leaderboard). If only a subset was
recomputed, select no.

Select N/A for both questions above if a new dataset was introduced and the
only one evaluated in the paper.

Run a well-documented human evaluation

  * •

Run a human evaluation to measure important quality aspects: Select yes, if a
human evaluation of any kind was conducted.

  * •

Document the study setup (questions, measurement instruments, etc.): Select
yes, if, at the minimum, the specific questions and the way that participants
answer them are reported.

  * •

Document who is participating in the study: Select yes, if, at the minimum,
the annotation platform used and the number of participants are stated.

Produce robust human evaluation results

  * •

Estimate the effect size and conduct a power analysis: Select yes, if any
effect size estimate or power analysis is mentioned (we assume that not
mentioning it implies it absence).

  * •

Run significance test(s) on the results: Select yes, if the human annotation
results are accompanied by a statistical significance test.

  * •

Conduct an analysis of result validity (agreement, comparison to gold
ratings): Select yes, if there is any kind of analysis of the quality of the
human annotations themselves.

  * •

Discuss the required rater qualification and background: Select yes, if the
required knowledge of raters is discussed and compared to the qualifications
selected for in the study.

Document results in model cards

  * •

Report disaggregated results for subpopulations: Select yes, if the paper
reports fine-grained results on subsets of the test set(s) (note that the
paper does not need to introduce these breakdowns as in the point above).

  * •

Evaluate on non-i.i.d. test set(s): Select yes, if there is an evaluation on a
non-i.i.d. test set. If the paper does not specifically mention this fact,
select no (i.e., if the used dataset has such a test set but this is not
mentioned).

  * •

Analyze the causal effect of modeling choices on outputs with specific
properties: Select yes, if the results include a breakdown that allow for
insights of the form “if input has feature X, model output has Y”. An ablation
study counts as a yes, if the ablation focuses on feature representations
(i.e. what data a model sees), but not if the ablation is on model
architecture choices.

  * •

Conduct an error analysis and/or demonstrate failures of a model: Select yes,
if there is any kind of error analysis or qualitative samples of where the
model fails.

Release model outputs and annotations

In this section, select yes, if the paper is accompanied by data releases that
include the following.

  * •

Release outputs on the validation set

  * •

Release outputs on the test set

  * •

Release outputs for non-English dataset(s): Select N/A if the paper does not
include evaluation on any non-English data.

  * •

Release human evaluation annotations

###  A.3 Limitations

There are a few limitation of this setup. (1) Due to the phrasing as recall-
oriented prompts, nuanced errors pointed out in earlier sections are
implicitly ignored. For example, “Document the study setup” is marked as
positive even if the exact definition of each measurement category is not
provided. The lack of providing a definition was identified as a source of
confusion by Howcroft et al. (2020). In other cases, our prompts may not be
covering all possibilities. For example, a study that releases not an improved
version of a corpus, but instead a tailored pretraining set would not count as
“Address dataset issues and release an updated version”. (2) Each paper is
only annotated by one co-author of this survey. This means that there could be
misunderstandings of the different dimensions. We tried to address this
problem by refining definitions when unclear points arose and by discussing
the definitions before starting the annotation which led to the instructions
above. Nevertheless, the exact percentage results may differ from the ground-
truth by a few points and we thus consider only the overall trends.