* Ebrahimi et al. (2018) Ebrahimi, Javid, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018.  HotFlip: White-box adversarial examples for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 31–36, Association for Computational Linguistics, Melbourne, Australia.
  * Eger and Benz (2020) Eger, Steffen and Yannik Benz. 2020.  From hero to zéroe: A benchmark of low-level adversarial attacks.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 786–803, Association for Computational Linguistics, Suzhou, China.
  * Eger et al. (2019) Eger, Steffen, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019.  Text processing like humans do: Visually attacking and shielding NLP systems.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1634–1647, Association for Computational Linguistics, Minneapolis, Minnesota.
  * Fan et al. (2020) Fan, Angela, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020.  Beyond english-centric multilingual machine translation.
  * Faruqui et al. (2015) Faruqui, Manaal, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015.  Retrofitting word vectors to semantic lexicons.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1606–1615, Association for Computational Linguistics, Denver, Colorado.
  * Fico (2018) Fico. 2018.  FICO Announces xML Challenge.  https://www.prnewswire.com/news-releases/fico-announces-xml-challenge-300626571.html.  [Online; accessed 01.06.2021].
  * Fomicheva et al. (2021) Fomicheva, Marina, Piyawat Lertvittayakumjorn, Wei Zhao, Steffen Eger, and Yang Gao. 2021.  The eval4nlp shared task on explainable quality estimation: Overview and results.  In _Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems_.
  * Fomicheva, Specia, and Aletras (2021) Fomicheva, Marina, Lucia Specia, and Nikolaos Aletras. 2021.  Translation error detection as rationale extraction.
  * Fomicheva et al. (2020a) Fomicheva, Marina, Shuo Sun, Erick Fonseca, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2020a.  MLQE-PE: A multilingual quality estimation and post-editing dataset.  _arXiv preprint arXiv:2010.04480_.
  * Fomicheva et al. (2020b) Fomicheva, Marina, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020b.  Unsupervised quality estimation for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:539–555.
  * Freiesleben (2021) Freiesleben, Timo. 2021.  The intriguing relation between counterfactual explanations and adversarial examples.  _Minds and Machines_.
  * Freitag et al. (2021) Freitag, Markus, George F. Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _CoRR_ , abs/2104.14478.
  * Freitag, Grangier, and Caswell (2020) Freitag, Markus, David Grangier, and Isaac Caswell. 2020.  BLEU might be guilty but references are not innocent.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 61–71, Association for Computational Linguistics, Online.
  * Freitas (2014) Freitas, Alex A. 2014.  Comprehensible classification models: a position paper.  _ACM SIGKDD explorations newsletter_ , 15(1):1–10.
  * Gao et al. (2021) Gao, Yang, Steffen Eger, Wei Zhao, Piyawat Lertvittayakumjorn, and Marina Fomicheva, editors. 2021.  _Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems_. Association for Computational Linguistics, Punta Cana, Dominican Republic.
  * Gehrmann, Clark, and Sellam (2022) Gehrmann, Sebastian, Elizabeth Clark, and Thibault Sellam. 2022.  Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text.
  * Goodfellow, Shlens, and Szegedy (2015) Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and harnessing adversarial examples.
  * Goodman and Flaxman (2017) Goodman, Bryce and Seth Flaxman. 2017.  European union regulations on algorithmic decision-making and a “right to explanation”.  _AI Magazine_ , 38(3):50–57.
  * Graham et al. (2016) Graham, Yvette, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016.  Can machine translation systems be evaluated by the crowd alone.  _Natural Language Engineering_ , FirstView:1–28.
  * Gunning (2016) Gunning, David. 2016.  Explainable artificial intelligence (xai).
  * Han, Wallace, and Tsvetkov (2020) Han, Xiaochuang, Byron C. Wallace, and Yulia Tsvetkov. 2020.  Explaining black box predictions and unveiling data artifacts through influence functions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5553–5563, Association for Computational Linguistics, Online.
  * Hewitt and Manning (2019) Hewitt, John and Christopher D. Manning. 2019.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138, Association for Computational Linguistics, Minneapolis, Minnesota.
  * Hoffman et al. (2018) Hoffman, Robert R, Shane T Mueller, Gary Klein, and Jordan Litman. 2018.  Metrics for explainable ai: Challenges and prospects.  _arXiv preprint arXiv:1812.04608_.
  * Jacovi and Goldberg (2020) Jacovi, Alon and Yoav Goldberg. 2020.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205, Association for Computational Linguistics, Online.
  * Jacovi and Goldberg (2021) Jacovi, Alon and Yoav Goldberg. 2021.  Aligning faithful interpretations with their social attribution.  _Transactions of the Association for Computational Linguistics_ , 9:294–310.
  * Jacovi et al. (2021) Jacovi, Alon, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021.  Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai.  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 624–635.
  * Jain et al. (2020) Jain, Sarthak, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020.  Learning to faithfully rationalize by construction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4459–4473, Association for Computational Linguistics, Online.
  * Jiang et al. (2021) Jiang, Yuchen, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, and Ming Zhou. 2021.  Blond: An automatic evaluation metric for document-level machinetranslation.  _arXiv preprint arXiv:2103.11878_.
  * Jin et al. (2020) Jin, Di, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):8018–8025.
  * Johnson, Carenini, and Murray (2020) Johnson, David, Giuseppe Carenini, and Gabriel Murray. 2020.  Njm-vis: Interpreting neural joint models in nlp.  In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ , IUI ’20, page 286–296, Association for Computing Machinery, New York, NY, USA.
  * Joty et al. (2017) Joty, Shafiq, Francisco Guzmán, Lluís Màrquez, and Preslav Nakov. 2017\.  Discourse structure in machine translation evaluation.  _Computational Linguistics_ , 43(4):683–722.
  * Kaster, Zhao, and Eger (2021) Kaster, Marvin, Wei Zhao, and Steffen Eger. 2021.  Global explainability of bert-based evaluation metrics by disentangling along linguistic factors.  In _Emnlp 2021_ , Association for Computational Linguistics, Online.
  * Keller, Mackensen, and Eger (2021) Keller, Yannik, Jan Mackensen, and Steffen Eger. 2021.  BERT-defense: A probabilistic model based on BERT to combat cognitively inspired orthographic adversarial attacks.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1616–1629, Association for Computational Linguistics, Online.
  * Kim et al. (2018) Kim, Been, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , pages 2668–2677, PMLR.
  * Kim et al. (2020) Kim, Siwon, Jihun Yi, Eunji Kim, and Sungroh Yoon. 2020.  Interpretation of NLP models through input marginalization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3154–3167, Association for Computational Linguistics, Online.
  * Kiritchenko and Mohammad (2017) Kiritchenko, Svetlana and Saif Mohammad. 2017.  Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 465–470, Association for Computational Linguistics, Vancouver, Canada.
  * Kobayashi et al. (2020) Kobayashi, Goro, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is not only a weight: Analyzing transformers with vector norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7057–7075, Association for Computational Linguistics, Online.
  * Koh and Liang (2017) Koh, Pang Wei and Percy Liang. 2017.  Understanding black-box predictions via influence functions.  In _International Conference on Machine Learning_ , pages 1885–1894, PMLR.
  * Krause, Perer, and Ng (2016) Krause, Josua, Adam Perer, and Kenney Ng. 2016.  Interacting with predictions: Visual inspection of black-box machine learning models.  In _Proceedings of the 2016 CHI conference on human factors in computing systems_ , pages 5686–5697.
  * Kulesza et al. (2015) Kulesza, Todd, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015.  Principles of explanatory debugging to personalize interactive machine learning.  In _Proceedings of the 20th international conference on intelligent user interfaces_ , pages 126–137.
  * Kumar and Talukdar (2020) Kumar, Sawan and Partha Talukdar. 2020.  NILE : Natural language inference with faithful natural language explanations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8730–8742, Association for Computational Linguistics, Online.
  * Lai, Liu, and Tan (2020) Lai, Vivian, Han Liu, and Chenhao Tan. 2020.  “why is’ chicago’deceptive?” towards building model-driven tutorials for humans.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , pages 1–13.
  * Lakkaraju et al. (2019) Lakkaraju, Himabindu, Ece Kamar, Rich Caruana, and Jure Leskovec. 2019.  Faithful and customizable explanations of black box models.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 131–138.
  * Lan, Jiang, and Xu (2021) Lan, Wuwei, Chao Jiang, and Wei Xu. 2021.  Neural semi-Markov CRF for monolingual word alignment.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6815–6828, Association for Computational Linguistics, Online.
  * Lavie, Sagae, and Jayaraman (2004) Lavie, Alon, Kenji Sagae, and Shyamsundar Jayaraman. 2004.  The significance of recall in automatic metrics for mt evaluation.  In _Machine Translation: From Real Users to Research_ , pages 134–143, Springer Berlin Heidelberg, Berlin, Heidelberg.
  * Lei, Barzilay, and Jaakkola (2016) Lei, Tao, Regina Barzilay, and Tommi Jaakkola. 2016.  Rationalizing neural predictions.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 107–117, Association for Computational Linguistics, Austin, Texas.
  * Lertvittayakumjorn (2021) Lertvittayakumjorn, Piyawat. 2021.  Explainable nlp for human-ai collaboration.
  * Lertvittayakumjorn et al. (2021) Lertvittayakumjorn, Piyawat, Ivan Petej, Yang Gao, Yamuna Krishnamurthy, Anna Van Der Gaag, Robert Jago, and Kostas Stathis. 2021.  Supporting complaints investigation for nursing and midwifery regulatory agencies.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 81–91, Association for Computational Linguistics, Online.
  * Lertvittayakumjorn and Toni (2019) Lertvittayakumjorn, Piyawat and Francesca Toni. 2019.  Human-grounded evaluations of explanation methods for text classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5195–5205, Association for Computational Linguistics, Hong Kong, China.
  * Lertvittayakumjorn and Toni (2021) Lertvittayakumjorn, Piyawat and Francesca Toni. 2021.  Explanation-Based Human Debugging of NLP Models: A Survey.  _Transactions of the Association for Computational Linguistics_ , 9:1508–1528.
  * Lewis et al. (2020) Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Association for Computational Linguistics, Online.
  * Li et al. (2016) Li, Jiwei, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016.  Visualizing and understanding neural models in NLP.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 681–691, Association for Computational Linguistics, San Diego, California.
  * Li et al. (2020) Li, Linyang, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020.  BERT-ATTACK: Adversarial attack against BERT using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6193–6202, Association for Computational Linguistics, Online.
  * Lin (2004) Lin, Chin-Yew. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Association for Computational Linguistics, Barcelona, Spain.
  * Linardatos, Papastefanopoulos, and Kotsiantis (2021) Linardatos, Pantelis, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. 2021.  Explainable ai: A review of machine learning interpretability methods.  _Entropy_ , 23(1).
  * Lipton (2016) Lipton, Zachary. 2016.  The mythos of model interpretability.  _Communications of the ACM_ , 61.
  * Liu, Yin, and Wang (2019) Liu, Hui, Qingyu Yin, and William Yang Wang. 2019.  Towards explainable NLP: A generative explanation framework for text classification.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5570–5581, Association for Computational Linguistics, Florence, Italy.
  * Liu et al. (2021) Liu, Pengfei, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. 2021.  ExplainaBoard: An explainable leaderboard for NLP.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 280–289, Association for Computational Linguistics, Online.
  * Lo and Larkin (2020) Lo, Chi-kiu and Samuel Larkin. 2020.  Machine translation reference-less evaluation using YiSi-2 with bilingual mappings of massive multilingual language model.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 903–910, Association for Computational Linguistics, Online.
  * Lommel, Burchardt, and Uszkoreit (2014) Lommel, Arle, Aljoscha Burchardt, and Hans Uszkoreit. 2014.  Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics.  _Tradumàtica: tecnologies de la traducció_ , 0:455–463.
  * Lommel, Burchardt, and Uszkoreit (2015) Lommel, Arle, Aljoscha Burchardt, and Hans Uszkoreit. 2015.  Multidimensional quality metrics (mqm) definition.
  * Lundberg et al. (2020) Lundberg, Scott M, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020\.  From local explanations to global understanding with explainable ai for trees.  _Nature machine intelligence_ , 2(1):56–67.
  * Lundberg, Erion, and Lee (2018) Lundberg, Scott M, Gabriel G Erion, and Su-In Lee. 2018.  Consistent individualized feature attribution for tree ensembles.  _arXiv preprint arXiv:1802.03888_.
  * Lundberg and Lee (2017) Lundberg, Scott M and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_. Curran Associates, Inc., pages 4765–4774.
  * Mac Aodha et al. (2018) Mac Aodha, Oisin, Shihan Su, Yuxin Chen, Pietro Perona, and Yisong Yue. 2018.  Teaching categories to human learners with visual explanations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3820–3828.
  * Magnolini, Feltracco, and Magnini (2016) Magnolini, Simone, Anna Feltracco, and Bernardo Magnini. 2016.  FBK-HLT-NLP at SemEval-2016 task 2: A multitask, deep learning approach for interpretable semantic textual similarity.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 783–789, Association for Computational Linguistics, San Diego, California.
  * Mahendran and Vedaldi (2014) Mahendran, Aravindh and Andrea Vedaldi. 2014.  Understanding deep image representations by inverting them.
  * Marie, Fujita, and Rubino (2021) Marie, Benjamin, Atsushi Fujita, and Raphael Rubino. 2021.  Scientific credibility of machine translation research: A meta-evaluation of 769 papers.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7297–7306, Association for Computational Linguistics, Online.
  * Mathur et al. (2020) Mathur, Nitika, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Association for Computational Linguistics, Online.
  * McCoy, Pavlick, and Linzen (2019) McCoy, Tom, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Association for Computational Linguistics, Florence, Italy.
  * Mehrabi et al. (2021) Mehrabi, Ninareh, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021.  A survey on bias and fairness in machine learning.  _ACM Comput. Surv._ , 54(6).
  * Mikolov et al. (2013) Mikolov, Tomás, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
  * Mohseni, Block, and Ragan (2018) Mohseni, Sina, Jeremy E Block, and Eric D Ragan. 2018.  A human-grounded evaluation benchmark for local explanations of machine learning.  _arXiv preprint arXiv:1801.05075_.
  * Moosavi et al. (2021) Moosavi, Nafise Sadat, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021\.  Scigen: a dataset for reasoning-aware text generation from scientific tables.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Morris et al. (2020a) Morris, John, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020a.  Reevaluating adversarial examples in natural language.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3829–3839, Association for Computational Linguistics, Online.
  * Morris et al. (2020b) Morris, John, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020b.  TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 119–126, Association for Computational Linguistics, Online.
  * Mu and Andreas (2020) Mu, Jesse and Jacob Andreas. 2020.  Compositional explanations of neurons.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 17153–17163, Curran Associates, Inc.
  * Naber (2003) Naber, Daniel. 2003.  _A Rule-Based Style and Grammar Checker_.  Ph.D. thesis, Universität Dortmund.
  * Narang et al. (2020) Narang, Sharan, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020.  Wt5?! training text-to-text models to explain their predictions.  _arXiv preprint arXiv:2004.14546_.
  * Ng and Abrecht (2015) Ng, Jun-Ping and Viktoria Abrecht. 2015.  Better summarization evaluation with word embeddings for ROUGE.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1925–1930, Association for Computational Linguistics, Lisbon, Portugal.
  * Nguyen (2018) Nguyen, Dong. 2018.  Comparing automatic and human evaluation of local explanations for text classification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1069–1078, Association for Computational Linguistics, New Orleans, Louisiana.
  * Papineni et al. (2002) Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA.
  * Pennington, Socher, and Manning (2014) Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Peyrard (2019) Peyrard, Maxime. 2019.  Studying summarization evaluation metrics in the appropriate scoring range.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5093–5100, Association for Computational Linguistics, Florence, Italy.
  * Peyrard et al. (2021) Peyrard, Maxime, Wei Zhao, Steffen Eger, and Robert West. 2021.  Better than average: Paired evaluation of NLP systems.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2301–2315, Association for Computational Linguistics, Online.
  * Poerner, Roth, and Schütze (2018) Poerner, Nina, Benjamin Roth, and Hinrich Schütze. 2018.  Interpretable textual neuron representations for NLP.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 325–327, Association for Computational Linguistics, Brussels, Belgium.
  * Poerner, Schütze, and Roth (2018) Poerner, Nina, Hinrich Schütze, and Benjamin Roth. 2018.  Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 340–350, Association for Computational Linguistics, Melbourne, Australia.
  * Polák, Singh, and Bojar (2021) Polák, Peter, Muskaan Singh, and Ondřej Bojar. 2021.  Explainable quality estimation: Cuni eval4nlp submission.  In _In Proceedings of the 2nd Workshop on Evaluation and Comparison for NLP systems._
  * Poliak et al. (2018) Poliak, Adam, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018.  Collecting diverse natural language inference problems for sentence representation evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 67–81, Association for Computational Linguistics, Brussels, Belgium.
  * Pruthi, Dhingra, and Lipton (2019) Pruthi, Danish, Bhuwan Dhingra, and Zachary C. Lipton. 2019.  Combating adversarial misspellings with robust word recognition.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5582–5591, Association for Computational Linguistics, Florence, Italy.
  * Pu and Chen (2006) Pu, Pearl and Li Chen. 2006.  Trust building with explanation interfaces.  In _Proceedings of the 11th international conference on Intelligent user interfaces_ , pages 93–100.
  * Ranasinghe, Orasan, and Mitkov (2020) Ranasinghe, Tharindu, Constantin Orasan, and Ruslan Mitkov. 2020.  TransQuest: Translation quality estimation with cross-lingual transformers.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5070–5081, International Committee on Computational Linguistics, Barcelona, Spain (Online).
  * Ranasinghe, Orasan, and Mitkov (2021) Ranasinghe, Tharindu, Constantin Orasan, and Ruslan Mitkov. 2021.  An exploratory analysis of multilingual word-level quality estimation with cross-lingual transformers.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 434–440, Association for Computational Linguistics, Online.
  * Rei et al. (2020) Rei, Ricardo, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Association for Computational Linguistics, Online.
  * Reimers and Gurevych (2019) Reimers, Nils and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Association for Computational Linguistics, Hong Kong, China.
  * Reimers and Gurevych (2020) Reimers, Nils and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , Association for Computational Linguistics.
  * Reiter (2018) Reiter, Ehud. 2018.  A structured review of the validity of BLEU.  _Computational Linguistics_ , 44(3):393–401.
  * Ribeiro, Singh, and Guestrin (2016) Ribeiro, Marco, Sameer Singh, and Carlos Guestrin. 2016.  “why should I trust you?”: Explaining the predictions of any classifier.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_ , pages 97–101, Association for Computational Linguistics, San Diego, California.
  * Ribeiro, Singh, and Guestrin (2018a) Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. 2018a.  Anchors: High-precision model-agnostic explanations.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Ribeiro, Singh, and Guestrin (2018b) Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. 2018b.  Semantically equivalent adversarial rules for debugging NLP models.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 856–865, Association for Computational Linguistics, Melbourne, Australia.
  * Ribeiro et al. (2020a) Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020a.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Association for Computational Linguistics, Online.
  * Ribeiro et al. (2020b) Ribeiro, Marco Tulio, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020b.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Association for Computational Linguistics, Online.
  * Rubino, Fujita, and Marie (2021) Rubino, Raphael, Atsushi Fujita, and Benjamin Marie. 2021.  Error identification for machine translation with metric embedding and attention.  In _In Proceedings of the 2nd Workshop on Evaluation and Comparison for NLP systems._
  * Rudin (2019) Rudin, Cynthia. 2019.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1:206–216.
  * Sachan et al. (2020) Sachan, Swati, Jian-Bo Yang, Dong-Ling Xu, David Eraso Benavides, and Yang Li. 2020\.  An explainable ai decision-support-system to automate loan underwriting.  _Expert Systems with Applications_ , 144:113100.
  * Sai et al. (2021) Sai, Ananya B., Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021.  Perturbation checklists for evaluating nlg evaluation metrics.
  * Sai, Mohankumar, and Khapra (2020) Sai, Ananya B., Akash Kumar Mohankumar, and Mitesh M. Khapra. 2020.  A survey of evaluation metrics used for nlg systems.
  * Samek et al. (2017) Samek, Wojciech, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2017.  Evaluating the visualization of what a deep neural network has learned.  _IEEE Transactions on Neural Networks and Learning Systems_ , 28(11):2660–2673.
  * Samek et al. (2019) Samek, Wojciech, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller. 2019.  _Explainable AI: interpreting, explaining and visualizing deep learning_ , volume 11700.  Springer Nature.
  * Samek, Wiegand, and Müller (2018) Samek, Wojciech, Thomas Wiegand, and Klaus-Robert Müller. 2018.  Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models.  _ITU Journal: ICT Discoveries - Special Issue 1 - The Impact of Artificial Intelligence (AI) on Communication Networks and Services_ , 1(1):39–48.
  * Schulz et al. (2020) Schulz, Karl, Leon Sixt, Federico Tombari, and Tim Landgraf. 2020.  Restricting the flow: Information bottlenecks for attribution.  In _International Conference on Learning Representations_.
  * Sellam, Das, and Parikh (2020) Sellam, Thibault, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Association for Computational Linguistics, Online.
  * Serrano and Smith (2019) Serrano, Sofia and Noah A. Smith. 2019.  Is attention interpretable?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2931–2951, Association for Computational Linguistics, Florence, Italy.
  * Setzu et al. (2021) Setzu, Mattia, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti. 2021.  Glocalx-from local to global explanations of black box ai models.  _Artificial Intelligence_ , 294:103457.
  * Shenoy et al. (2021) Shenoy, Raksha, Nico Herbig, Antonio Krüger, and Josef van Genabith. 2021.  Investigating the helpfulness of word-level quality estimation for post-editing machine translation output.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10173–10185.
  * Snover et al. (2006) Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A study of translation edit rate with targeted human annotation.  In _Proceedings of association for machine translation in the Americas_ , volume 200, Citeseer.
  * Song, Zhao, and Specia (2021) Song, Yurun, Junchen Zhao, and Lucia Specia. 2021.  SentSim: Crosslingual semantic evaluation of machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3143–3156, Association for Computational Linguistics, Online.
  * Stanojević et al. (2015) Stanojević, Miloš, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015.  Results of the WMT15 metrics shared task.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 256–273, Association for Computational Linguistics, Lisbon, Portugal.
  * Sundararajan, Taly, and Yan (2017) Sundararajan, Mukund, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , Icml’17, pages 3319–3328, JMLR.org.
  * Sushil, Šuster, and Daelemans (2018) Sushil, Madhumita, Simon Šuster, and Walter Daelemans. 2018.  Rule induction for global explanation of trained models.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 82–97, Association for Computational Linguistics, Brussels, Belgium.
  * Szegedy et al. (2014) Szegedy, Christian, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014.  Intriguing properties of neural networks.
  * Tan et al. (2018) Tan, Sarah, Rich Caruana, Giles Hooker, and Yin Lou. 2018.  Distill-and-compare: Auditing black-box models using transparent model distillation.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 303–310.
  * Tang and Surdeanu (2021) Tang, Zheng and Mihai Surdeanu. 2021.  Interpretability rules: Jointly bootstrapping a neural relation extractorwith an explanation decoder.  In _Proceedings of the First Workshop on Trustworthy Natural Language Processing_ , pages 1–7, Association for Computational Linguistics, Online.
  * Tättar and Fishel (2017) Tättar, Andre and Mark Fishel. 2017.  bleu2vec: the painfully familiar metric on continuous vector space steroids.  In _Proceedings of the Second Conference on Machine Translation_ , pages 619–622, Association for Computational Linguistics, Copenhagen, Denmark.
  * Thompson and Post (2020) Thompson, Brian and Matt Post. 2020.  Automatic machine translation evaluation in many languages via zero-shot paraphrasing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , Association for Computational Linguistics, Online.
  * Toreini et al. (2020) Toreini, Ehsan, Mhairi Aitken, Kovila Coopamootoo, Karen Elliott, Carlos Gonzalez Zelaya, and Aad van Moorsel. 2020.  The relationship between trust in ai and trustworthy machine learning technologies.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 272–283.
  * Treviso et al. (2021) Treviso, Marcos V., Nuno M. Guerreiro, Ricardo Rei, and André F.T Martins. 2021\.  Ist-unbabel 2021 submission for the explainable quality estimation sharedtask.  In _In Proceedings of the 2nd Workshop on Evaluation and Comparison for NLP systems._
  * Tsai, Yang, and Chen (2019) Tsai, Yi-Ting, Min-Chu Yang, and Han-Yu Chen. 2019.  Adversarial attack on sentiment classification.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 233–240, Association for Computational Linguistics, Florence, Italy.
  * Turchi et al. (2014) Turchi, Marco, Antonios Anastasopoulos, José G. C. de Souza, and Matteo Negri. 2014.  Adaptive quality estimation for machine translation.  In _ACL_.
  * Vaughan and Wallach (2020) Vaughan, Jennifer Wortman and Hanna Wallach. 2020.  A human-centered agenda for intelligible machine learning.  _Machines We Trust: Getting Along with Artificial Intelligence_.
  * Voita, Sennrich, and Titov (2019) Voita, Elena, Rico Sennrich, and Ivan Titov. 2019.  When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1198–1212, Association for Computational Linguistics, Florence, Italy.
  * Voita and Titov (2020) Voita, Elena and Ivan Titov. 2020.  Information-theoretic probing with minimum description length.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 183–196, Association for Computational Linguistics, Online.
  * Wachter, Mittelstadt, and Russell (2018) Wachter, Sandra, Brent Mittelstadt, and Chris Russell. 2018.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.
  * Wallace, Feng, and Boyd-Graber (2018) Wallace, Eric, Shi Feng, and Jordan Boyd-Graber. 2018.  Interpreting neural networks with nearest neighbors.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 136–144, Association for Computational Linguistics, Brussels, Belgium.
  * Wallace et al. (2019) Wallace, Eric, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Association for Computational Linguistics, Hong Kong, China.
  * Wang et al. (2019) Wang, Alex, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Superglue: a stickier benchmark for general-purpose language understanding systems.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pages 3266–3280.
  * Wang et al. (2020) Wang, Yuxuan, Yutai Hou, Wangxiang Che, and Ting Liu. 2020.  From static to dynamic word representations: a survey.  _Int. J. Mach. Learn. & Cyber._, 11:1611–1630.
  * Wiegreffe and Marasovic (2021) Wiegreffe, Sarah and Ana Marasovic. 2021.  Teach me to explain: A review of datasets for explainable natural language processing.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Wiegreffe and Pinter (2019) Wiegreffe, Sarah and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Association for Computational Linguistics, Hong Kong, China.
  * Yuan, Neubig, and Liu (2021) Yuan, Weizhe, Graham Neubig, and Pengfei Liu. 2021.  BARTScore: Evaluating generated text as text generation.  In _Thirty-Fifth Conference on Neural Information Processing Systems_.
  * Zeng et al. (2021) Zeng, Guoyang, Fanchao Qi, Qianrui Zhou, Tingji Zhang, Bairu Hou, Yuan Zang, Zhiyuan Liu, and Maosong Sun. 2021.  Openattack: An open-source textual adversarial attack toolkit.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 363–371.
  * Zhang et al. (2020) Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang, Liao, and Bellamy (2020) Zhang, Yunfeng, Q Vera Liao, and Rachel KE Bellamy. 2020.  Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 295–305.
  * Zhao et al. (2020) Zhao, Wei, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020.  On the limitations of cross-lingual encoders as exposed by reference-free machine translation evaluation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1656–1671, Association for Computational Linguistics, Online.
  * Zhao et al. (2019) Zhao, Wei, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Association for Computational Linguistics, Hong Kong, China.
  * Zhao, Strube, and Eger (2022) Zhao, Wei, Michael Strube, and Steffen Eger. 2022.  Discoscore: Evaluating text generation with bert and discourse coherence.
  * Zhu et al. (2019) Zhu, Junnan, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019.  NCLS: Neural cross-lingual summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3054–3064, Association for Computational Linguistics, Hong Kong, China.

##  Appendix A Inverse Metric Techniques

The approach we use for inverse metrics applies simple, greedy perturbations
on a word-level. To do so, it randomly searches through mask replacements by a
language model. Hence, it is similar to language-model-based adversarial
attacks (e.g. Li et al. 2020). Given a metric MTE, a hypothesis $h$ with
tokens $h=(h_{1},...,h_{n})$, a perturbed hypothesis $h^{\ast}$, a target
score $t$, and a source (and/or reference) $s$, the method aims to minimize
the following equation: argmin_h^∗ —t - MTE(s, h^∗)— Further, following the
perturbation setup in LIME (Ribeiro, Singh, and Guestrin 2016), let
$m=(m_{1},...,m_{n})$ be a mask for $h$ where each element indicates whether
the respective token in $h$ is perturbed. Instead of $m$ being a binary
vector, we choose each $m_{i}$ in $m$ to represent the $k$-th likeliest mask
replacement based on a language model. E.g., a mask $m=(0,2,1)$ indicates that
the first token is not perturbed, the second token is perturbed with the 2nd
most likely word and the third token is perturbed with the most likely word.
We apply the perturbations one by one in a random order. The algorithm
searches for the mask that produces the best hypothesis solving the
minimization problem above. We search masks for $x$ iterations. Further, we
keep a list of $b$ masks per iteration, which is initialized with masks
without perturbation (all zero). In each iteration, for every mask in $b$, we
randomly increase each mask element by $1$ with a perturbation probability of
$p$. Setting $p$ small, and $b$ high should lead to a better consideration of
perturbations that are close to the original hypothesis. Instead, setting a
high perturbation rate and a small number of samples per iteration will lead
to far examples being explored earlier. While being easy to implement,
shortcomings are that the number of tokens (words or sub-words) is already
set, making some options unreachable. Additionally, the algorithm does not
directly consider the source sentence.

##  Appendix B Explanation with respect REF and/or HYP

In this section of the Appendix we provide an example that demonstrates the
differences in explaining different parts of the input with additive feature
attribution methods as described in section 4.2. To do so, we extract feature
importance scores from BLEURT (Sellam, Das, and Parikh 2020) using exact SHAP
(Lundberg and Lee 2017) 141414SHAP: https://github.com/slundberg/shap. BLEURT:
https://github.com/google-research/bleurt. We mask removed words with an empty
string. Further, we explain the score (0.113) that is assigned for the
hypothesis “I have a cat .” and the reference “I have a dog .”. In figure 12
we fix the reference sentence (i.e. SHAP only perturbs the hypothesis and
assigns importance scores to the words in the hypothesis). At the bottom left,
we see the Baseline value of -1.859. I.e., BLEURT assigns this score, when the
reference sentence is compared to an empty hypothesis. The red arrows indicate
how important each of the words is in order to achieve the given score.
Compared to its removal, the word cat gets assigned the highest importance,
even though it is the only word that was incorrectly translated. This is an
interesting inside on the BLEURT model. Further, in 13, we explain the
reference sentence with regards to a fixed hypothesis. Here we similarly see
that the word dog was assigned with the highest importance. Finally, in 14, we
explain BLEURT with respect to a free reference and hypothesis. I.e. the
importance of all words for achieving a high score in general is evaluated.
Here, the baseline is at 1.5 and most words have a negative importance. The
reason for this is the similarity of empty sequences, such that the baseline
gets highs scores. In general the result is more difficult to interpret.

Figure 12: Explanation with respect to a fixed reference Figure 13:
Explanation with respect to a fixed hypothesis Figure 14: Explanation with
free reference and hypothesis