* Hastie and Tibshirani (2014) Trevor Hastie and R. Tibshirani. 2014.  Generalized Additive Models.  In _Wiley StatsRef: Statistics Reference Online_. John Wiley & Sons, Ltd.  https://doi.org/10.1002/9781118445112.stat03141
  * Health (1246) Center for Devices and Radiological Health. Fri, 10/20/2023 - 12:46.  Artificial Intelligence and Machine Learning in Software as a Medical Device.  _FDA_ (Fri, 10/20/2023 - 12:46).
  * Ho et al. (2007) Cristy Ho, Nick Reed, and Charles Spence. 2007.  Multisensory In-Car Warning Signals for Collision Avoidance.  _Human Factors_ 49, 6 (2007), 1107–1114.  https://doi.org/10.1518/001872007X249965
  * Hoffman et al. (2018) Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018.  Metrics for Explainable AI: Challenges and Prospects.  _arXiv preprint arXiv:1812.04608_ (2018).  arXiv:1812.04608
  * Hoffman et al. (2023) Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. 2023.  Measures for Explainable AI: Explanation Goodness, User Satisfaction, Mental Models, Curiosity, Trust, and Human-AI Performance.  _Frontiers in Computer Science_ 5 (2023), 1096257.  https://doi.org/10.3389/fcomp.2023.1096257
  * Holzinger et al. (2021) Andreas Holzinger, Bernd Malle, Anna Saranti, and Bastian Pfeifer. 2021.  Towards Multi-Modal Causability with Graph Neural Networks Enabling Information Fusion for Explainable AI.  _Information Fusion_ 71 (2021), 28–37.  https://doi.org/10.1016/j.inffus.2021.01.008
  * Holzinger and Müller (2021) Andreas Holzinger and Heimo Müller. 2021.  Toward Human–AI Interfaces to Support Explainability and Causability in Medical AI.  _Computer_ 54, 10 (2021), 78–86.  https://doi.org/10.1109/MC.2021.3092610
  * Hong et al. (2023) Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2023.  MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework.  arXiv:2308.00352 [cs.AI]
  * Hong et al. (2020) Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. 2020.  Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs.  https://doi.org/10.1145/3392878 arXiv:2004.11440 [cs]
  * Hu et al. (2022a) Zhongxu Hu, Shanhe Lou, Yang Xing, Xiao Wang, Dongpu Cao, and Chen Lv. 2022a.  Review and Perspectives on Driver Digital Twin and Its Enabling Technologies for Intelligent Vehicles.  _IEEE Transactions on Intelligent Vehicles_ 7, 3 (2022), 417–440.  https://doi.org/10.1109/TIV.2022.3195635
  * Hu et al. (2022b) Zhongxu Hu, Chen Lv, Peng Hang, Chao Huang, and Yang Xing. 2022b.  Data-Driven Estimation of Driver Attention Using Calibration-Free Eye Gaze and Scene Features.  _IEEE Transactions on Industrial Electronics_ 69, 2 (2022), 1800–1808.  https://doi.org/10.1109/TIE.2021.3057033
  * Huang et al. (2023) Wei Huang, Xingyu Zhao, Gaojie Jin, and Xiaowei Huang. 2023.  SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretability.  https://doi.org/10.48550/arXiv.2208.09418 arXiv:2208.09418 [cs]
  * Huber et al. (2023) Tobias Huber, Maximilian Demmler, Silvan Mertes, Matthew L. Olson, and Elisabeth André. 2023.  GANterfactual-RL: Understanding Reinforcement Learning Agents’ Strategies through Visual Counterfactual Explanations.  https://doi.org/10.48550/arXiv.2302.12689 arXiv:2302.12689 [cs]
  * Ibrahim and Shafiq (2023) Rami Ibrahim and M. Omair Shafiq. 2023.  Explainable Convolutional Neural Networks: A Taxonomy, Review, and Future Directions.  _Comput. Surveys_ 55, 10 (2023), 1–37.  https://doi.org/10.1145/3563691
  * Idahl et al. (2021) Maximilian Idahl, Lijun Lyu, Ujwal Gadiraju, and Avishek Anand. 2021.  Towards Benchmarking the Utility of Explanations for Model Debugging.  arXiv:2105.04505 [cs]
  * Inglis et al. (2021) Alan Inglis, Andrew Parnell, and Catherine Hurley. 2021.  Visualizing Variable Importance and Variable Interaction Effects in Machine Learning Models.  arXiv:2108.04310 [stat]
  * Ismail et al. (2021) Aya Abdelsalam Ismail, Héctor Corrada Bravo, and Soheil Feizi. 2021.  Improving Deep Learning Interpretability by Saliency Guided Training.  https://doi.org/10.48550/arXiv.2111.14338 arXiv:2111.14338 [cs]
  * Itoh and Inagaki (2014) Makoto Itoh and Toshiyuki Inagaki. 2014.  Design and Evaluation of Steering Protection for Avoiding Collisions during a Lane Change.  _Ergonomics_ 57, 3 (2014), 361–373.  https://doi.org/10.1080/00140139.2013.848474
  * Javed et al. (2023) Abdul Rehman Javed, Waqas Ahmed, Sharnil Pandya, Praveen Kumar Reddy Maddikunta, Mamoun Alazab, and Thippa Reddy Gadekallu. 2023.  A Survey of Explainable Artificial Intelligence for Smart Cities.  _Electronics_ 12, 4 (2023), 1020.  https://doi.org/10.3390/electronics12041020
  * Jeon (2023) Myounghoon Jeon. 2023.  The Effects of Emotions on Trust in Human-Computer Interaction: A Survey and Prospect.  _International Journal of Human–Computer Interaction_ 0, 0 (2023), 1–19.  https://doi.org/10.1080/10447318.2023.2261727
  * Jha and Busso (2023) Sumit Jha and Carlos Busso. 2023.  Estimation of Driver’s Gaze Region from Head Position and Orientation Using Probabilistic Confidence Regions.  _IEEE Transactions on Intelligent Vehicles_ 8, 1 (2023), 59–72.  https://doi.org/10.1109/TIV.2022.3141071
  * Jin et al. (2023b) Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, Shirui Pan, Vincent S. Tseng, Yu Zheng, Lei Chen, and Hui Xiong. 2023b.  Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook.  https://doi.org/10.48550/arXiv.2310.10196 arXiv:2310.10196 [cs]
  * Jin et al. (2023a) Weina Jin, Jianyu Fan, Diane Gromala, Philippe Pasquier, and Ghassan Hamarneh. 2023a.  Invisible Users: Uncovering End-Users’ Requirements for Explainable AI via Explanation Forms and Goals.  https://doi.org/10.48550/arXiv.2302.06609 arXiv:2302.06609 [cs]
  * Jing et al. (2022) Chunhui Jing, Chenguang Shang, Dongyu Yu, Yaodong Chen, and Jinyi Zhi. 2022.  The Impact of Different AR-HUD Virtual Warning Interfaces on the Takeover Performance and Visual Characteristics of Autonomous Vehicles.  _Traffic Injury Prevention_ 23, 5 (2022), 277–282.  https://doi.org/10.1080/15389588.2022.2055752
  * Karimi et al. (2022) Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. 2022.  A Survey of Algorithmic Recourse: Contrastive Explanations and Consequential Recommendations.  _Comput. Surveys_ 55, 5 (2022), 95:1–95:29.  https://doi.org/10.1145/3527848
  * Kashevnik et al. (2021) Alexey Kashevnik, Roman Shchedrin, Christian Kaiser, and Alexander Stocker. 2021.  Driver Distraction Detection Methods: A Literature Review and Framework.  _IEEE access : practical innovations, open solutions_ 9 (2021), 60063–60076.  https://doi.org/10.1109/ACCESS.2021.3073599
  * Kaur et al. (2022) Davinder Kaur, Suleyman Uslu, Kaley J Rittichier, and Arjan Durresi. 2022.  Trustworthy Artificial Intelligence: A Review.  _ACM Computing Surveys (CSUR)_ 55, 2 (2022), 1–38.
  * Keane et al. (2021) Mark T. Keane, Eoin M. Kenny, Eoin Delaney, and Barry Smyth. 2021.  If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniques.  https://doi.org/10.48550/arXiv.2103.01035 arXiv:2103.01035 [cs]
  * Kenny et al. (2021) Eoin M. Kenny, Eoin D. Delaney, Derek Greene, and Mark T. Keane. 2021.  Post-Hoc Explanation Options for XAI in Deep Learning: The Insight Centre for Data Analytics Perspective. In _Pattern Recognition. ICPR International Workshops and Challenges_ , Alberto Del Bimbo, Rita Cucchiara, Stan Sclaroff, Giovanni Maria Farinella, Tao Mei, Marco Bertini, Hugo Jair Escalante, and Roberto Vezzani (Eds.). Springer International Publishing, Cham, 20–34.  https://doi.org/10.1007/978-3-030-68796-0_2
  * Kenny et al. (2022) Eoin M Kenny, Mycal Tucker, and Julie Shah. 2022.  Towards interpretable deep reinforcement learning with human-friendly prototypes. In _The Eleventh International Conference on Learning Representations_.
  * Kim et al. (2016) Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016.  Examples Are Not Enough, Learn to Criticize! Criticism for Interpretability. In _Advances in Neural Information Processing Systems_ , Vol. 29. Curran Associates, Inc.
  * Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  https://doi.org/10.48550/arXiv.1711.11279 arXiv:1711.11279 [stat]
  * Kim et al. (2023b) Gwangbin Kim, Dohyeon Yeo, Taewoo Jo, Daniela Rus, and SeungJun Kim. 2023b.  What and When to Explain? On-road Evaluation of Explanations in Highly Automated Vehicles.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 7, 3 (2023), 104:1–104:26.  https://doi.org/10.1145/3610886
  * Kim et al. (2024) Minjung Kim, Saebyeol Kim, Jinwoo Kim, Tae-Jin Song, and Yuyoung Kim. 2024.  Do Stakeholder Needs Differ? - Designing Stakeholder-Tailored Explainable Artificial Intelligence (XAI) Interfaces.  _International Journal of Human-Computer Studies_ 181 (2024), 103160.  https://doi.org/10.1016/j.ijhcs.2023.103160
  * Kim et al. (2023a) Sunnie S. Y. Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023a.  ”Help Me Help the AI”: Understanding How Explainability Can Support Human-AI Interaction.  https://doi.org/10.1145/3544548.3581001 arXiv:2210.03735 [cs]
  * Klien et al. (2004) G. Klien, D.D. Woods, J.M. Bradshaw, R.R. Hoffman, and P.J. Feltovich. 2004.  Ten Challenges for Making Automation a ”Team Player” in Joint Human-Agent Activity.  _IEEE Intelligent Systems_ 19, 6 (2004), 91–95.  https://doi.org/10.1109/MIS.2004.74
  * Kobayashi et al. (2023) Kazuma Kobayashi, Bader Almutairi, Md Nazmus Sakib, Souvik Chakraborty, and Syed B. Alam. 2023.  Explainable, Interpretable & Trustworthy AI for Intelligent Digital Twin: Case Study on Remaining Useful Life.  https://doi.org/10.48550/arXiv.2301.06676 arXiv:2301.06676 [cs, stat]
  * Kommiya Mothilal et al. (2021) Ramaravind Kommiya Mothilal, Divyat Mahajan, Chenhao Tan, and Amit Sharma. 2021.  Towards Unifying Feature Attribution and Counterfactual Explanations: Different Means to the Same End. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. ACM, Virtual Event USA, 652–663.  https://doi.org/10.1145/3461702.3462597
  * Kravaris et al. (2023) Theocharis Kravaris, Konstantinos Lentzos, Georgios Santipantakis, George A Vouros, Gennady Andrienko, Natalia Andrienko, Ian Crook, Jose Manuel Cordero Garcia, and Enrique Iglesias Martinez. 2023.  Explaining deep reinforcement learning decisions in complex multiagent settings: towards enabling automation in air traffic flow management.  _Applied Intelligence_ 53, 4 (2023), 4063–4098.
  * Krishna et al. (2022) Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. 2022.  The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective.  https://doi.org/10.48550/arXiv.2202.01602 arXiv:2202.01602 [cs]
  * Kroeger et al. (2023) Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. 2023.  Are Large Language Models Post Hoc Explainers?  arXiv:2310.05797 [cs]
  * Kyriakidis et al. (2019) M. Kyriakidis, J. C. F. de Winter, N. Stanton, T. Bellet, B. van Arem, K. Brookhuis, M. H. Martens, K. Bengler, J. Andersson, N. Merat, N. Reed, M. Flament, M. Hagenzieker, and R. Happee. 2019.  A Human Factors Perspective on Automated Driving.  _Theoretical Issues in Ergonomics Science_ 20, 3 (2019), 223–249.  https://doi.org/10.1080/1463922X.2017.1293187
  * Lai et al. (2019) Vivian Lai, Zheng Cai, and Chenhao Tan. 2019.  Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 486–495.  https://doi.org/10.18653/v1/D19-1046
  * Langer et al. (2021) Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum. 2021.  What Do We Want from Explainable Artificial Intelligence (XAI)? – A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research.  _Artificial Intelligence_ 296 (2021), 103473.  https://doi.org/10.1016/j.artint.2021.103473
  * Lash (2022) Michael T. Lash. 2022.  HEX: Human-in-the-loop Explainability via Deep Reinforcement Learning.  arXiv:2206.01343 [cs, stat]
  * Lee and See (2004) John D. Lee and Katrina A. See. 2004.  Trust in Automation: Designing for Appropriate Reliance.  _Human Factors_ 46, 1 (2004), 50–80.  https://doi.org/10.1518/hfes.46.1.50_30392
  * Lee et al. (2023) Sangwon Lee, Jeonguk Hong, Gyewon Jeon, Jeongmin Jo, Sanghyeok Boo, Hwiseong Kim, Seoyoon Jung, Jieun Park, Inheon Choi, and Sangyeon Kim. 2023.  Investigating Effects of Multimodal Explanations Using Multiple In-vehicle Displays for Takeover Request in Conditionally Automated Driving.  _Transportation Research Part F: Traffic Psychology and Behaviour_ 96 (2023), 1–22.  https://doi.org/10.1016/j.trf.2023.05.014
  * Leichtenstern et al. (2010) Karin Leichtenstern, Elisabeth André, and Ekatarina Kurdyukova. 2010.  Managing User Trust for Self-Adaptive Ubiquitous Computing Systems. In _Proceedings of the 8th International Conference on Advances in Mobile Computing and Multimedia_ _(MoMM ’10)_. Association for Computing Machinery, New York, NY, USA, 409–414.  https://doi.org/10.1145/1971519.1971589
  * Lerner et al. (2015) Jennifer S. Lerner, Ye Li, Piercarlo Valdesolo, and Karim S. Kassam. 2015.  Emotion and Decision Making.  _Annual Review of Psychology_ 66, 1 (2015), 799–823.  https://doi.org/10.1146/annurev-psych-010213-115043
  * Li et al. (2023) Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. 2023.  Trustworthy AI: From Principles to Practices.  _Comput. Surveys_ 55, 9 (2023), 177:1–177:46.  https://doi.org/10.1145/3555803
  * Li et al. (2019) Fan Li, Chun-Hsien Chen, Gangyan Xu, Li Pheng Khoo, and Yisi Liu. 2019.  Proactive Mental Fatigue Detection of Traffic Control Operators Using Bagged Trees and Gaze-Bin Analysis.  _Advanced Engineering Informatics_ 42 (2019), 100987.  https://doi.org/10.1016/j.aei.2019.100987
  * Li et al. (2022) Mengyao Li, Isabel M Erickson, Ernest V Cross, and John D Lee. 2022.  Estimating Trust in Conversational Agent with Lexical and Acoustic Features. In _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ , Vol. 66. SAGE Publications Sage CA: Los Angeles, CA, 544–548.
  * Li and Kesavadas (2018) Xiao Li and Thenkurussi Kesavadas. 2018.  Surgical Robot with Environment Reconstruction and Force Feedback. In _2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. 1861–1866.  https://doi.org/10.1109/EMBC.2018.8512695
  * Liao et al. (2020) Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020.  Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. ACM, Honolulu HI USA, 1–15.  https://doi.org/10.1145/3313831.3376590
  * Lieberman et al. (2020) Isador H. Lieberman, Stanley Kisinde, and Shea Hesselbacher. 2020.  Robotic-Assisted Pedicle Screw Placement During Spine Surgery.  _JBJS Essential Surgical Techniques_ 10, 2 (2020), e0020.  https://doi.org/10.2106/JBJS.ST.19.00020
  * Loh (2011) Wei-Yin Loh. 2011.  Classification and Regression Trees.  _Wiley interdisciplinary reviews: data mining and knowledge discovery_ 1, 1 (2011), 14–23.
  * Lundberg et al. (2020) Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020.  From Local Explanations to Global Understanding with Explainable AI for Trees.  _Nature Machine Intelligence_ 2, 1 (2020), 56–67.  https://doi.org/10.1038/s42256-019-0138-9
  * Luo et al. (2020) Yunan Luo, Jian Peng, and Jianzhu Ma. 2020.  When Causal Inference Meets Deep Learning.  _Nature Machine Intelligence_ 2, 8 (2020), 426–427.  https://doi.org/10.1038/s42256-020-0218-x
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective Approaches to Attention-based Neural Machine Translation.  arXiv:1508.04025 [cs]
  * Lyons et al. (2018) Joseph B Lyons, Nhut T Ho, Anna Lee Van Abel, Lauren C Hoffmann, W Eric Fergueson, Garrett G Sadler, Michelle A Grigsby, and Amy C Burns. 2018.  Exploring trust barriers to future autonomy: a qualitative look. In _Advances in Human Factors in Simulation and Modeling: Proceedings of the AHFE 2017 International Conference on Human Factors in Simulation and Modeling, July 17–21, 2017, The Westin Bonaventure Hotel, Los Angeles, California, USA 8_. Springer, 3–11.
  * Lyons et al. (2021) Joseph B. Lyons, Katia Sycara, Michael Lewis, and August Capiola. 2021.  Human–Autonomy Teaming: Definitions, Debates, and Directions.  _Frontiers in Psychology_ 12 (2021).
  * Ma et al. (2021) Hongbin Ma, Shuyuan Yang, Dongzhu Feng, Licheng Jiao, and Luping Zhang. 2021.  Progressive Mimic Learning: A New Perspective to Train Lightweight CNN Models.  _Neurocomputing_ 456 (2021), 220–231.  https://doi.org/10.1016/j.neucom.2021.04.086
  * Madhav and Tyagi (2023) A. V. Shreyas Madhav and Amit Kumar Tyagi. 2023.  Explainable Artificial Intelligence (XAI): Connecting Artificial Decision-Making and Human Trust in Autonomous Vehicles. In _Proceedings of Third International Conference on Computing, Communications, and Cyber-Security_ _(Lecture Notes in Networks and Systems)_ , Pradeep Kumar Singh, Sławomir T. Wierzchoń, Sudeep Tanwar, Joel J. P. C. Rodrigues, and Maria Ganzha (Eds.). Springer Nature, Singapore, 123–136.  https://doi.org/10.1007/978-981-19-1142-2_10
  * Manawadu et al. (2016) Udara E. Manawadu, Mitsuhiro Kamezaki, Masaaki Ishikawa, Takahiro Kawano, and Shigeki Sugano. 2016.  A Haptic Feedback Driver-Vehicle Interface for Controlling Lateral and Longitudinal Motions of Autonomous Vehicles. In _2016 IEEE International Conference on Advanced Intelligent Mechatronics (AIM)_. 119–124.  https://doi.org/10.1109/AIM.2016.7576753
  * Mao et al. (2019) Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. 2019.  The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision.  arXiv:1904.12584 [cs]
  * Markus et al. (2021) Aniek F. Markus, Jan A. Kors, and Peter R. Rijnbeek. 2021.  The Role of Explainability in Creating Trustworthy Artificial Intelligence for Health Care: A Comprehensive Survey of the Terminology, Design Choices, and Evaluation Strategies.  _Journal of Biomedical Informatics_ 113 (2021), 103655.  https://doi.org/10.1016/j.jbi.2020.103655
  * Melis and Jaakkola (2018) David Alvarez Melis and Tommi Jaakkola. 2018.  Towards Robust Interpretability with Self-Explaining Neural Networks. In _Advances in Neural Information Processing Systems_ , Vol. 31. Curran Associates, Inc.
  * Meng and Spence (2015) Fanxing Meng and Charles Spence. 2015.  Tactile Warning Signals for In-Vehicle Systems.  _Accident; Analysis and Prevention_ 75 (2015), 333–346.  https://doi.org/10.1016/j.aap.2014.12.013
  * Meske and Bunde (2022) Christian Meske and Enrico Bunde. 2022.  Design Principles for User Interfaces in AI-Based Decision Support Systems: The Case of Explainable Hate Speech Detection.  _Information Systems Frontiers_ (2022).  https://doi.org/10.1007/s10796-021-10234-5
  * Messina et al. (2022) Pablo Messina, Pablo Pino, Denis Parra, Alvaro Soto, Cecilia Besa, Sergio Uribe, Marcelo Andía, Cristian Tejos, Claudia Prieto, and Daniel Capurro. 2022.  A survey on deep learning and explainability for automatic report generation from medical images.  _ACM Computing Surveys (CSUR)_ 54, 10s (2022), 1–40.
  * Methnani et al. (2024) Leila Methnani, Manolis Chiou, Virginia Dignum, and Andreas Theodorou. 2024.  Who’s in Charge Here? A Survey on Trustworthy AI in Variable Autonomy Robotic Systems.  _Comput. Surveys_ (2024), 32.  https://doi.org/10.1145/3645090
  * Michon (1985) John A. Michon. 1985.  A Critical View of Driver Behavior Models: What Do We Know, What Should We Do?  In _Human Behavior and Traffic Safety_ , Leonard Evans and Richard C. Schwing (Eds.). Springer US, Boston, MA, 485–524.  https://doi.org/10.1007/978-1-4613-2173-6_19
  * Milani et al. (2023) Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. 2023.  Explainable Reinforcement Learning: A Survey and Comparative Review.  _Comput. Surveys_ (2023), 1–35.  https://doi.org/10.1145/3616864
  * Minh et al. (2022) Dang Minh, H. Xiang Wang, Y. Fen Li, and Tan N. Nguyen. 2022.  Explainable Artificial Intelligence: A Comprehensive Review.  _Artificial Intelligence Review_ 55, 5 (2022), 3503–3568.  https://doi.org/10.1007/s10462-021-10088-y
  * Mishra et al. (2018) Abhijit Mishra, Srikanth Tamilselvam, Riddhiman Dasgupta, Seema Nagar, and Kuntal Dey. 2018.  Cognition-Cognizant Sentiment Analysis With Multitask Subjectivity Summarization Based on Annotators’ Gaze Behavior.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 32, 1 (2018).  https://doi.org/10.1609/aaai.v32i1.12068
  * Mohseni et al. (2021) Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. 2021.  A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems.  _ACM Transactions on Interactive Intelligent Systems_ 11, 3–4 (2021), 1–45.  https://doi.org/10.1145/3387166
  * Monaco and Brennan (2020) Chris D. Monaco and Sean N. Brennan. 2020.  RADARODO: Ego-Motion Estimation From Doppler and Spatial Data in RADAR Images.  _IEEE Transactions on Intelligent Vehicles_ 5, 3 (2020), 475–484.  https://doi.org/10.1109/TIV.2020.2973536
  * Money and Agius (2009) Arthur G. Money and Harry Agius. 2009.  Analysing User Physiological Responses for Affective Video Summarisation.  _Displays_ 30, 2 (2009), 59–70.  https://doi.org/10.1016/j.displa.2008.12.003
  * Mossing et al. (2024) Dan Mossing, Steven Bills, Henk Tillman, Tom Dupré la Tour, Nick Cammarata, Leo Gao, Joshua Achiam, Catherine Yeh, Jan Leike, Jeff Wu, and William Saunders. 2024.  Transformer Debugger.
  * Mulgan (2000) Richard Mulgan. 2000.  ‘Accountability’: An Ever-Expanding Concept?  _Public Administration_ 78, 3 (2000), 555–573.  https://doi.org/10.1111/1467-9299.00218
  * Mutzenich et al. (2021) Clare Mutzenich, Szonya Durant, Shaun Helman, and Polly Dalton. 2021.  Updating Our Understanding of Situation Awareness in Relation to Remote Operators of Autonomous Vehicles.  _Cognitive Research: Principles and Implications_ 6, 1 (2021), 9.  https://doi.org/10.1186/s41235-021-00271-8
  * Naujoks et al. (2017) Frederik Naujoks, Yannick Forster, Katharina Wiedemann, and Alexandra Neukum. 2017.  Improving Usefulness of Automated Driving by Lowering Primary Task Interference through HMI Design.  _Journal of Advanced Transportation_ 2017 (2017), e6105087.  https://doi.org/10.1155/2017/6105087
  * Nauta et al. (2023) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. 2023.  From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI.  _Comput. Surveys_ 55, 13s (2023), 1–42.  https://doi.org/10.1145/3583558
  * Neerincx et al. (2018) Mark A. Neerincx, Jasper van der Waa, Frank Kaptein, and Jurriaan van Diggelen. 2018.  Using Perceptual and Cognitive Explanations for Enhanced Human-Agent Team Performance. In _Engineering Psychology and Cognitive Ergonomics_ _(Lecture Notes in Computer Science)_ , Don Harris (Ed.). Springer International Publishing, Cham, 204–214.  https://doi.org/10.1007/978-3-319-91122-9_18
  * Neubauer (2023) Catherine Neubauer. 2023.  HAT3: The Human Autonomy Team Trust Toolkit. In _Companion Publication of the 25th International Conference on Multimodal Interaction_ (¡conf-loc¿, ¡city¿Paris¡/city¿, ¡country¿France¡/country¿, ¡/conf-loc¿) _(ICMI ’23 Companion)_. Association for Computing Machinery, New York, NY, USA, 115–118.  https://doi.org/10.1145/3610661.3620660
  * Neupane et al. (2023) Subash Neupane, Ivan A. Fernandez, Wilson Patterson, Sudip Mittal, Milan Parmar, and Shahram Rahimi. 2023.  TwinExplainer: Explaining Predictions of an Automotive Digital Twin.  arXiv:2302.00152 [cs]
  * Okamura and Yamada (2020) Kazuo Okamura and Seiji Yamada. 2020.  Adaptive Trust Calibration for Human-AI Collaboration.  _Plos one_ 15, 2 (2020), e0229132.
  * Oliveira et al. (2018) Luis Oliveira, Jacob Luton, Sumeet Iyer, Chris Burns, Alexandros Mouzakitis, Paul Jennings, and Stewart Birrell. 2018.  Evaluating How Interfaces Influence the User Interaction with Fully Autonomous Vehicles. In _Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_. ACM Association for Computing Machinery.  https://doi.org/10.1145/3239060.3239065
  * Omeiza et al. (2022) Daniel Omeiza, Helena Webb, Marina Jirotka, and Lars Kunze. 2022.  Explanations in Autonomous Driving: A Survey.  _IEEE Transactions on Intelligent Transportation Systems_ 23, 8 (2022), 10142–10162.  https://doi.org/10.1109/tits.2021.3122865
  * O’Neill et al. (2022) Thomas O’Neill, Nathan McNeese, Amy Barron, and Beau Schelble. 2022.  Human–Autonomy Teaming: A Review and Analysis of the Empirical Literature.  _Human Factors_ 64, 5 (2022), 904–938.  https://doi.org/10.1177/0018720820960865
  * O’Sullivan et al. (2019) Shane O’Sullivan, Nathalie Nevejans, Colin Allen, Andrew Blyth, Simon Leonard, Ugo Pagallo, Katharina Holzinger, Andreas Holzinger, Mohammed Imran Sajid, and Hutan Ashrafian. 2019.  Legal, Regulatory, and Ethical Frameworks for Development of Standards in Artificial Intelligence (AI) and Autonomous Robotic Surgery.  _The international journal of medical robotics + computer assisted surgery: MRCAS_ 15, 1 (2019), e1968.  https://doi.org/10.1002/rcs.1968
  * Oviatt and Cohen (2015) Sharon Oviatt and Philip Cohen. 2015.  _The Paradigm Shift to Multimodality in Contemporary Computer Interfaces_.  Morgan & Claypool Publishers.  https://doi.org/10.2200/S00636ED1V01Y201503HCI030
  * Panganiban et al. (2020) April Rose Panganiban, Gerald Matthews, and Michael D. Long. 2020.  Transparency in Autonomous Teammates: Intention to Support as Teaming Information.  _Journal of Cognitive Engineering and Decision Making_ 14, 2 (2020), 174–190.  https://doi.org/10.1177/1555343419881563
  * Panigutti et al. (2023) Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, Alan Perotti, and Salvatore Rinzivillo. 2023.  Co-Design of Human-Centered, Explainable AI for Clinical Decision Support.  _ACM Transactions on Interactive Intelligent Systems_ (2023), 3587271.  https://doi.org/10.1145/3587271
  * Panigutti et al. (2022) Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi. 2022.  Understanding the Impact of Explanations on Advice-Taking: A User Study for AI-based Clinical Decision Support Systems. In _CHI Conference on Human Factors in Computing Systems_. ACM, New Orleans LA USA, 1–9.  https://doi.org/10.1145/3491102.3502104
  * Park et al. (2018) Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018.  Multimodal Explanations: Justifying Decisions and Pointing to the Evidence. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE, Salt Lake City, UT, 8779–8788.  https://doi.org/10.1109/CVPR.2018.00915
  * Park et al. (2019) Heejong Park, Arvind Easwaran, and Sidharta Andalam. 2019.  Challenges in Digital Twin Development for Cyber-Physical Production Systems.  , 28–48 pages.  https://doi.org/10.1007/978-3-030-23703-5_2 arXiv:2102.03341 [cs]
  * Park et al. (2022) Seungeun Park, Betty Bekemeier, Abraham Flaxman, and Melinda Schultz. 2022.  Impact of Data Visualization on Decision-Making and Its Implications for Public Health Practice: A Systematic Literature Review.  _Informatics for Health & Social Care_ 47, 2 (2022), 175–193.  https://doi.org/10.1080/17538157.2021.1982949
  * Parvatharaju et al. (2021) Prathyush S. Parvatharaju, Ramesh Doddaiah, Thomas Hartvigsen, and Elke A. Rundensteiner. 2021.  Learning Saliency Maps to Explain Deep Time Series Classifiers. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_ _(CIKM ’21)_. Association for Computing Machinery, New York, NY, USA, 1406–1415.  https://doi.org/10.1145/3459637.3482446
  * Patnaik et al. (2019) Biswaksen Patnaik, Andrea Batch, and Niklas Elmqvist. 2019.  Information Olfactation: Harnessing Scent to Convey Data.  _IEEE Transactions on Visualization and Computer Graphics_ 25, 1 (2019), 726–736.  https://doi.org/10.1109/TVCG.2018.2865237
  * Patrício et al. (2023) Cristiano Patrício, João C. Neves, and Luís F. Teixeira. 2023.  Explainable Deep Learning Methods in Medical Image Classification: A Survey.  _Comput. Surveys_ 56, 4 (2023), 1–41.  https://doi.org/10.1145/3625287
  * Pearl (2009) Judea Pearl. 2009.  Causal Inference in Statistics: An Overview.  _Statistics Surveys_ 3, none (2009).  https://doi.org/10.1214/09-SS057
  * Pearl and Mackenzie (2018) Judea Pearl and Dana Mackenzie. 2018.  _The Book of Why: The New Science of Cause and Effect_.  Basic books.
  * Pesapane et al. (2018) Filippo Pesapane, Caterina Volonté, Marina Codari, and Francesco Sardanelli. 2018.  Artificial Intelligence as a Medical Device in Radiology: Ethical and Regulatory Issues in Europe and the United States.  _Insights into Imaging_ 9, 5 (2018), 745–753.  https://doi.org/10.1007/s13244-018-0645-y
  * Petsiuk et al. (2018) Vitali Petsiuk, Abir Das, and Kate Saenko. 2018.  RISE: Randomized Input Sampling for Explanation of Black-box Models.  arXiv:1806.07421 [cs]
  * Pierce et al. (2022) Robin L. Pierce, Wim Van Biesen, Daan Van Cauwenberge, Johan Decruyenaere, and Sigrid Sterckx. 2022.  Explainability in Medicine in an Era of AI-based Clinical Decision Support Systems.  _Frontiers in Genetics_ 13 (2022), 903600.  https://doi.org/10.3389/fgene.2022.903600
  * Plutchik (2003) Robert Plutchik. 2003.  _Emotions and Life: Perspectives from Psychology, Biology, and Evolution_.  American Psychological Association, Washington, DC, US. xix, 381 pages.
  * Prado-Romero et al. (2023) Mario Alfonso Prado-Romero, Bardh Prenkaj, Giovanni Stilo, and Fosca Giannotti. 2023.  A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges.  _Comput. Surveys_ (2023), 1–35.  https://doi.org/10.1145/3618105
  * Prosperi et al. (2020) Mattia Prosperi, Yi Guo, Matt Sperrin, James S. Koopman, Jae S. Min, Xing He, Shannan Rich, Mo Wang, Iain E. Buchan, and Jiang Bian. 2020.  Causal Inference and Counterfactual Prediction in Machine Learning for Actionable Healthcare.  _Nature Machine Intelligence_ 2, 7 (2020), 369–375.  https://doi.org/10.1038/s42256-020-0197-y
  * Purificato et al. (2022) Erasmo Purificato, Cataldo Musto, Pasquale Lops, and Ernesto William De Luca. 2022.  First Workshop on Adaptive and Personalized Explainable User Interfaces (Apex-Ui 2022). In _27th International Conference on Intelligent User Interfaces_. 1–3.
  * Rago et al. (2023) Antonio Rago, Hengzhi Li, and Francesca Toni. 2023.  Interactive Explanations by Conflict Resolution via Argumentative Exchanges.  https://doi.org/10.48550/arXiv.2303.15022 arXiv:2303.15022 [cs]
  * Rajavenkatanarayanan et al. (2019) Akilesh Rajavenkatanarayanan, Varun Kanal, Konstantinos Tsiakas, James Brady, Diane Calderon, Glenn Wylie, and Fillia Makedon. 2019.  Towards a Robot-Based Multimodal Framework to Assess the Impact of Fatigue on User Behavior and Performance: A Pilot Study. In _Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments_ _(PETRA ’19)_. Association for Computing Machinery, New York, NY, USA, 493–498.  https://doi.org/10.1145/3316782.3322776
  * Ramgopal et al. (2023) Sriram Ramgopal, L. Nelson Sanchez-Pinto, Christopher M. Horvat, Michael S. Carroll, Yuan Luo, and Todd A. Florin. 2023.  Artificial Intelligence-Based Clinical Decision Support in Pediatrics.  _Pediatric Research_ 93, 2 (2023), 334–341.  https://doi.org/10.1038/s41390-022-02226-1
  * Raykar et al. (2023) Vikas C. Raykar, Arindam Jati, Sumanta Mukherjee, Nupur Aggarwal, Kanthi Sarpatwar, Giridhar Ganapavarapu, and Roman Vaculin. 2023.  TsSHAP: Robust Model Agnostic Feature-Based Explainability for Time Series Forecasting.  https://doi.org/10.48550/ARXIV.2303.12316
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier.  https://doi.org/10.48550/arXiv.1602.04938 arXiv:1602.04938 [cs, stat]
  * Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018.  Anchors: High-Precision Model-Agnostic Explanations.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 32, 1 (2018).  https://doi.org/10.1609/aaai.v32i1.11491
  * Ribeiro and Costa (2019) Rafael F. Ribeiro and Paula D. P. Costa. 2019.  Driver Gaze Zone Dataset With Depth Data. In _2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)_. 1–5.  https://doi.org/10.1109/FG.2019.8756592
  * Robb et al. (2018) David A. Robb, Francisco J. Chiyah Garcia, Atanas Laskov, Xingkun Liu, Pedro Patron, and Helen Hastie. 2018.  Keep Me in the Loop: Increasing Operator Situation Awareness through a Conversational Multimodal Interface. In _Proceedings of the 20th ACM International Conference on Multimodal Interaction_. ACM, Boulder CO USA, 384–392.  https://doi.org/10.1145/3242969.3242974
  * Rosenfeld (2021) Avi Rosenfeld. 2021.  Better Metrics for Evaluating Explainable Artificial Intelligence. In _Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems_. 45–50.
  * Rosenfeld and Richardson (2019) Avi Rosenfeld and Ariella Richardson. 2019.  Explainability in Human–Agent Systems.  _Autonomous Agents and Multi-Agent Systems_ 33, 6 (2019), 673–705.  https://doi.org/10.1007/s10458-019-09408-y
  * Rožanec et al. (2022) Jože M. Rožanec, Blaž Fortuna, and Dunja Mladenić. 2022.  Knowledge Graph-Based Rich and Confidentiality Preserving Explainable Artificial Intelligence (XAI).  _Information Fusion_ 81 (2022), 91–102.  https://doi.org/10.1016/j.inffus.2021.11.015
  * Rudin (2019) Cynthia Rudin. 2019.  Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead.  _Nature machine intelligence_ 1, 5 (2019), 206–215.  https://doi.org/10.1038/s42256-019-0048-x
  * Sado et al. (2023) Fatai Sado, Chu Kiong Loo, Wei Shiung Liew, Matthias Kerzel, and Stefan Wermter. 2023.  Explainable Goal-driven Agents and Robots - A Comprehensive Review.  _Comput. Surveys_ 55, 10 (2023), 211:1–211:41.  https://doi.org/10.1145/3564240
  * Salmon et al. (2006) Paul Salmon, Neville Stanton, Guy Walker, and Damian Green. 2006.  Situation Awareness Measurement: A Review of Applicability for C4i Environments.  _Applied Ergonomics_ 37, 2 (2006), 225–238.  https://doi.org/10.1016/j.apergo.2005.02.001
  * Salubre and Nathan-Roberts (2021) Kevin Joel Salubre and Dan Nathan-Roberts. 2021.  Takeover Request Design in Automated Driving: A Systematic Review.  _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ 65, 1 (2021), 868–872.  https://doi.org/10.1177/1071181321651296
  * Sanneman (2023) Lindsay Sanneman. 2023.  _Transparent Value Alignment: Foundations for Human-Centered Explainable AI in Alignment_.  Thesis. Massachusetts Institute of Technology.
  * Sanneman and Shah (2022) Lindsay Sanneman and Julie A. Shah. 2022.  The Situation Awareness Framework for Explainable AI (SAFE-AI) and Human Factors Considerations for XAI Systems.  _International Journal of Human–Computer Interaction_ 38, 18-20 (2022), 1772–1788.  https://doi.org/10.1080/10447318.2022.2081282
  * Schelble et al. (2020) Beau G. Schelble, Christopher Flathmann, and Nathan McNeese. 2020.  Towards Meaningfully Integrating Human-Autonomy Teaming in Applied Settings. In _Proceedings of the 8th International Conference on Human-Agent Interaction_. ACM, Virtual Event USA, 149–156.  https://doi.org/10.1145/3406499.3415077
  * Schneider et al. (2021) Tobias Schneider, Sabiha Ghellal, Steve Love, and Ansgar R.S. Gerlicher. 2021.  Increasing the User Experience in Autonomous Driving through Different Feedback Modalities. In _26th International Conference on Intelligent User Interfaces_. ACM, College Station TX USA, 7–10.  https://doi.org/10.1145/3397481.3450687
  * Schölkopf et al. (2021) Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021.  Towards Causal Representation Learning.  https://arxiv.org/abs/2102.11107v1.
  * Schraagen et al. (2020) Jan Maarten Schraagen, Pia Elsasser, Hanna Fricke, Marleen Hof, and Fabyen Ragalmuto. 2020.  Trusting the X in XAI: Effects of Different Types of Explanations by a Self-Driving Car on Trust, Explanation Satisfaction and Mental Models.  _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ 64, 1 (2020), 339–343.  https://doi.org/10.1177/1071181320641077
  * Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 618–626.  https://doi.org/10.1109/ICCV.2017.74
  * Seyyedi et al. (2023) Azra Seyyedi, Mahdi Bohlouli, and Seyedehsan Nedaaee Oskoee. 2023.  Machine Learning and Physics: A Survey of Integrated Models.  _Comput. Surveys_ 56, 5 (2023), 115:1–115:33.  https://doi.org/10.1145/3611383
  * Sheth et al. (2021) Amit Sheth, Manas Gaur, Kaushik Roy, and Keyur Faldu. 2021.  Knowledge-Intensive Language Understanding for Explainable AI.  https://doi.org/10.48550/arXiv.2108.01174 arXiv:2108.01174 [cs]
  * Shrikumar et al. (2019) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2019.  Learning Important Features Through Propagating Activation Differences.  arXiv:1704.02685 [cs]
  * Shukla et al. (2020) Bibhudhendu Shukla, Ip-Shing Fan, and Ian Jennions. 2020.  Opportunities for explainable artificial intelligence in aerospace predictive maintenance. In _PHM Society European Conference_ , Vol. 5. 11–11.
  * Siau and Wang (2018) Keng Siau and Weiyu Wang. 2018.  Building Trust in Artificial Intelligence, Machine Learning, and Robotics.  _Cutter Business Technology Journal_ 31 (2018), 47–53.
  * Simorov et al. (2012) Anton Simorov, R. Stephen Otte, Courtni M. Kopietz, and Dmitry Oleynikov. 2012.  Review of Surgical Robotics User Interface: What Is the Best Way to Control Robotic Surgery?  _Surgical Endoscopy_ 26, 8 (2012), 2117–2125.  https://doi.org/10.1007/s00464-012-2182-y
  * Slack et al. (2020) Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020.  Fooling LIME and SHAP: Adversarial Attacks on Post Hoc Explanation Methods. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’20)_. Association for Computing Machinery, New York, NY, USA, 180–186.  https://doi.org/10.1145/3375627.3375830
  * Souza et al. (2022) Victor Feitosa Souza, Ferdinando Cicalese, Eduardo Sany Laber, and Marco Molinaro. 2022.  Decision Trees with Short Explainable Rules. In _Advances in Neural Information Processing Systems_.
  * Speith (2022) Timo Speith. 2022.  A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods. In _2022 ACM Conference on Fairness, Accountability, and Transparency_. ACM, Seoul Republic of Korea, 2239–2250.  https://doi.org/10.1145/3531146.3534639
  * Stowers et al. (2021) Kimberly Stowers, Lisa L. Brady, Christopher MacLellan, Ryan Wohleber, and Eduardo Salas. 2021.  Improving Teamwork Competencies in Human-Machine Teams: Perspectives From Team Science.  _Frontiers in Psychology_ 12 (2021), 590290.  https://doi.org/10.3389/fpsyg.2021.590290
  * Subramonyam et al. (2021) Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021.  ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In _26th International Conference on Intelligent User Interfaces_ _(IUI ’21)_. Association for Computing Machinery, New York, NY, USA, 48–58.  https://doi.org/10.1145/3397481.3450640
  * Sudjianto and Zhang (2021) Agus Sudjianto and Aijun Zhang. 2021.  Designing Inherently Interpretable Machine Learning Models.  https://doi.org/10.48550/arXiv.2111.01743 arXiv:2111.01743 [cs, stat]
  * Sun et al. (2023) Jimin Sun, Wenqi Shi, Felipe O. Giuste, Yog S. Vaghani, Lingzi Tang, and May D. Wang. 2023.  Improving Explainable AI with Patch Perturbation-Based Evaluation Pipeline: A COVID-19 X-ray Image Analysis Case Study.  _Scientific Reports_ 13, 1 (2023), 19488.  https://doi.org/10.1038/s41598-023-46493-2
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic Attribution for Deep Networks.  arXiv:1703.01365 [cs]
  * Szymanski et al. (2021) Maxwell Szymanski, Martijn Millecamp, and Katrien Verbert. 2021.  Visual, Textual or Hybrid: The Effect of User Expertise on Different Explanations. In _26th International Conference on Intelligent User Interfaces_ _(IUI ’21)_. Association for Computing Machinery, New York, NY, USA, 109–119.  https://doi.org/10.1145/3397481.3450662
  * Szymanski et al. (2022) Maxwell Szymanski, Vero Vanden Abeele, and Katrien Verbert. 2022.  Explaining Health Recommendations to Lay Users: The Dos and Dont’s. In _Joint Proceedings of the IUI 2022 Workshops: APEx-UI, HAI-GEN, HEALTHI, HUMANIZE, TExSS, SOCIALIZE Co-Located with the ACM International Conference on Intelligent User Interfaces (IUI 2022)_. CEUR Workshop Proceedings, 1–10.
  * Tahboub (2006) Karim A. Tahboub. 2006.  Intelligent Human-Machine Interaction Based on Dynamic Bayesian Networks Probabilistic Intention Recognition.  _Journal of Intelligent and Robotic Systems_ 45, 1 (2006), 31–52.  https://doi.org/10.1007/s10846-005-9018-0
  * Tang and Xu (2021) Yifan Tang and Yan Xu. 2021.  Multi-Agent Deep Reinforcement Learning for Solving Large-scale Air Traffic Flow Management Problem: A Time-Step Sequential Decision Approach. In _2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC)_. IEEE, San Antonio, TX, USA, 1–10.  https://doi.org/10.1109/DASC52595.2021.9594329
  * Tao et al. (2018) Fei Tao, Jiangfeng Cheng, Qinglin Qi, Meng Zhang, He Zhang, and Fangyuan Sui. 2018.  Digital Twin-Driven Product Design, Manufacturing and Service with Big Data.  _The International Journal of Advanced Manufacturing Technology_ 94, 9 (2018), 3563–3576.  https://doi.org/10.1007/s00170-017-0233-1
  * TAPAS (2022) TAPAS. 2022.  https://tapas-atm.eu/.  Accessed on 2024-03-31.
  * Tekkesinoglu (2024) Sule Tekkesinoglu. 2024.  Exploring Evaluation Methodologies for Explainable AI: Guidelines for Objective and Subjective Assessment.  _SSRN Electronic Journal_ (2024).  https://doi.org/10.2139/ssrn.4667052
  * Teso et al. (2023) Stefano Teso, Öznur Alkan, Wolfgang Stammer, and Elizabeth Daly. 2023.  Leveraging Explanations in Interactive Machine Learning: An Overview.  _Frontiers in Artificial Intelligence_ 6 (2023).  https://doi.org/10.3389/frai.2023.1066049
  * Torens et al. (2022) Christoph Torens, Umut Durak, and Johann C. Dauer. 2022.  Guidelines and Regulatory Framework for Machine Learning in Aviation. In _AIAA SCITECH 2022 Forum_. American Institute of Aeronautics and Astronautics.  https://doi.org/10.2514/6.2022-1132
  * Torrey et al. (2006) Cristen Torrey, Aaron Powers, Matthew Marge, Susan R Fussell, and Sara Kiesler. 2006.  Effects of Adaptive Robot Dialogue on Information Exchange and Social Relations. In _Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-robot Interaction_. 126–133.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _arXiv preprint arXiv:2307.09288_ (2023).  arXiv:2307.09288
  * Turbé et al. (2023) Hugues Turbé, Mina Bjelogrlic, Christian Lovis, and Gianmarco Mengaldo. 2023.  Evaluation of Post-Hoc Interpretability Methods in Time-Series Classification.  _Nature Machine Intelligence_ 5, 3 (2023), 250–260.  https://doi.org/10.1038/s42256-023-00620-w
  * Umerenkov et al. (2023) D. Umerenkov, G. Zubkova, and A. Nesterov. 2023.  Deciphering Diagnoses: How Large Language Models Explanations Influence Clinical Decision Making.  arXiv:2310.01708 [cs]
  * van Lent et al. (2004) Michael van Lent, William Fisher, and Michael Mancuso. 2004.  An Explainable Artificial Intelligence System for Small-unit Tactical Behavior.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need. In _Advances in Neural Information Processing Systems_ , Vol. 30. Curran Associates, Inc.
  * Verhagen et al. (2024) Ruben S Verhagen, Mark A Neerincx, and Myrthe L Tielman. 2024.  Meaningful human control and variable autonomy in human-robot teams for firefighting.  _Frontiers in Robotics and AI_ 11 (2024), 1323980.
  * Vilone and Longo (2021a) Giulia Vilone and Luca Longo. 2021a.  Classification of Explainable Artificial Intelligence Methods through Their Output Formats.  _Machine Learning and Knowledge Extraction_ 3, 3 (2021), 615–661.  https://doi.org/10.3390/make3030032
  * Vilone and Longo (2021b) Giulia Vilone and Luca Longo. 2021b.  Notions of Explainability and Evaluation Approaches for Explainable Artificial Intelligence.  _Information Fusion_ 76 (2021), 89–106.  https://doi.org/10.1016/j.inffus.2021.05.009
  * Voigt and Von dem Bussche (2017) Paul Voigt and Axel Von dem Bussche. 2017.  The eu general data protection regulation (gdpr).  _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ 10, 3152676 (2017), 10–5555.
  * Vøllestad (1997) N. K. Vøllestad. 1997.  Measurement of Human Muscle Fatigue.  _Journal of Neuroscience Methods_ 74, 2 (1997), 219–227.  https://doi.org/10.1016/s0165-0270(97)02251-6
  * Vouros (2022) George A. Vouros. 2022.  Explainable Deep Reinforcement Learning: State of the Art and Challenges.  _Comput. Surveys_ 55, 5 (2022), 92:1–92:39.  https://doi.org/10.1145/3527448
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017.  Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.  _SSRN Electronic Journal_ (2017).  https://doi.org/10.2139/ssrn.3063289
  * Waller et al. (2020) Kassie Waller, Nellie Hill, Courtney Meyers, Amber McCord, and Courtney D. Gibson. 2020.  The Effect of Infographics on Recall of Information about Genetically Modified Foods.  _Journal of Agricultural Education_ 61, 3 (2020), 22–37.  https://doi.org/10.5032/jae.2020.03022
  * Walliser et al. (2019) James C. Walliser, Ewart J. de Visser, Eva Wiese, and Tyler H. Shaw. 2019.  Team Structure and Team Building Improve Human–Machine Teaming With Autonomous Agents.  _Journal of Cognitive Engineering and Decision Making_ 13, 4 (2019), 258–278.  https://doi.org/10.1177/1555343419867563
  * Wang et al. (2019) Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019.  Designing Theory-Driven User-Centric Explainable AI. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3290605.3300831
  * Wang et al. (2023) Liuping Wang, Zhan Zhang, Dakuo Wang, Weidan Cao, Xiaomu Zhou, Ping Zhang, Jianxing Liu, Xiangmin Fan, and Feng Tian. 2023.  Human-Centered Design and Evaluation of AI-empowered Clinical Decision Support Systems: A Systematic Review.  _Frontiers in Computer Science_ 5 (2023).
  * Wang et al. (2022a) Qianwen Wang, Zhutian Chen, Yong Wang, and Huamin Qu. 2022a.  A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization.  _IEEE Transactions on Visualization and Computer Graphics_ 28, 12 (2022), 5134–5153.  https://doi.org/10.1109/TVCG.2021.3106142 arXiv:2012.00467 [cs]
  * Wang et al. (2022b) Yan Wang, Wei Song, Wei Tao, Antonio Liotta, Dawei Yang, Xinlei Li, Shuyong Gao, Yixuan Sun, Weifeng Ge, Wei Zhang, and Wenqiang Zhang. 2022b.  A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances.  arXiv:2203.06935 [cs]
  * Warren and Hillas (2020) Aiden Warren and Alek Hillas. 2020.  Friend or Frenemy? The Role of Trust in Human-Machine Teaming and Lethal Autonomous Weapons Systems.  _Small Wars & Insurgencies_ 31, 4 (2020), 822–850.  https://doi.org/10.1080/09592318.2020.1743485
  * Weber et al. (2023) Leander Weber, Sebastian Lapuschkin, Alexander Binder, and Wojciech Samek. 2023.  Beyond Explaining: Opportunities and Challenges of XAI-based Model Improvement.  _Information Fusion_ 92 (2023), 154–176.  https://doi.org/10.1016/j.inffus.2022.11.013
  * Wehner et al. (2022) Christoph Wehner, Francis Powlesland, Bashar Altakrouri, and Ute Schmid. 2022.  Explainable Online Lane Change Predictions on a Digital Twin with a Layer Normalized LSTM and Layer-wise Relevance Propagation.  https://doi.org/10.48550/arXiv.2204.01292 arXiv:2204.01292 [cs]
  * Weidemann and Rußwinkel (2021) Alexandra Weidemann and Nele Rußwinkel. 2021.  The Role of Frustration in Human–Robot Interaction – What Is Needed for a Successful Collaboration?  _Frontiers in Psychology_ 12 (2021), 640186.  https://doi.org/10.3389/fpsyg.2021.640186
  * Weitz et al. (2021) Katharina Weitz, Dominik Schiller, Ruben Schlagowski, Tobias Huber, and Elisabeth André. 2021.  “Let Me Explain!”: Exploring the Potential of Virtual Agents in Explainable AI Interaction Design.  _Journal on Multimodal User Interfaces_ 15, 2 (2021), 87–98.  https://doi.org/10.1007/s12193-020-00332-0
  * Wells and Bednarz (2021) Lindsay Wells and Tomasz Bednarz. 2021.  Explainable AI and Reinforcement Learning—A Systematic Review of Current Approaches and Trends.  _Frontiers in Artificial Intelligence_ 4 (2021).  https://doi.org/10.3389/frai.2021.550030
  * Wintersberger et al. (2020) Philipp Wintersberger, Hannah Nicklas, Thomas Martlbauer, Stephan Hammer, and Andreas Riener. 2020.  Explainable Automation: Personalized and Adaptive UIs to Foster Trust and Understanding of Driving Automation Systems. In _12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications_. ACM, Virtual Event DC USA, 252–261.  https://doi.org/10.1145/3409120.3410659
  * Wright et al. (2010) Adam Wright, Shobha Phansalkar, Meryl Bloomrosen, Robert A. Jenders, Anne M. Bobb, John D. Halamka, Gilad Kuperman, Thomas H. Payne, S. Teasdale, A. J. Vaida, and D. W. Bates. 2010.  Best Practices in Clinical Decision Support.  _Applied Clinical Informatics_ 1, 3 (2010), 331–345.  https://doi.org/10.4338/ACI-2010-05-RA-0031
  * Wright et al. (2020) Julia L. Wright, Jessie Y. C. Chen, and Shan G. Lakhmani. 2020.  Agent Transparency and Reliability in Human–Robot Interaction: The Influence on User Confidence and Perceived Reliability.  _IEEE Transactions on Human-Machine Systems_ 50, 3 (2020), 254–263.  https://doi.org/10.1109/THMS.2019.2925717
  * Wu et al. (2023a) Chenwang Wu, Xiting Wang, Defu Lian, Xing Xie, and Enhong Chen. 2023a.  A Causality Inspired Framework for Model Interpretation. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ _(KDD ’23)_. Association for Computing Machinery, New York, NY, USA, 2731–2741.  https://doi.org/10.1145/3580305.3599240
  * Wu et al. (2024) Xuansheng Wu, Haiyan Zhao, Yaochen Zhu, Yucheng Shi, Fan Yang, Tianming Liu, Xiaoming Zhai, Wenlin Yao, Jundong Li, Mengnan Du, and Ninghao Liu. 2024.  Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era.  arXiv:2403.08946 [cs]
  * Wu et al. (2023b) Zijun Wu, Zi Xuan Zhang, Atharva Naik, Zhijian Mei, Mauajama Firdaus, and Lili Mou. 2023b.  Weakly Supervised Explainable Phrasal Reasoning with Neural Fuzzy Logic.  arXiv:2109.08927 [cs]
  * Wysocki et al. (2023) Oskar Wysocki, Jessica Katharine Davies, Markel Vigo, Anne Caroline Armstrong, Dónal Landers, Rebecca Lee, and André Freitas. 2023.  Assessing the Communication Gap between AI Models and Healthcare Professionals: Explainability, Utility and Trust in AI-driven Clinical Decision-Making.  _Artificial Intelligence_ 316 (2023), 103839.  https://doi.org/10.1016/j.artint.2022.103839
  * Xiang et al. (2021) Ziyu Xiang, Mingzhou Fan, Guillermo Vázquez Tovar, William Trehern, Byung-Jun Yoon, Xiaofeng Qian, Raymundo Arroyave, and Xiaoning Qian. 2021.  Physics-Constrained Automatic Feature Engineering for Predictive Modeling in Materials Science.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 12 (2021), 10414–10421.  https://doi.org/10.1609/aaai.v35i12.17247
  * Xie et al. (2019) Yao Xie, Ge Gao, and Xiang’Anthony’ Chen. 2019.  Outlining the Design Space of Explainable Intelligent Systems for Medical Diagnosis.  _arXiv preprint arXiv:1902.06019_ (2019).  arXiv:1902.06019
  * Xing et al. (2023) Yang Xing, Zhongxu Hu, Peng Hang, and Chen Lv. 2023.  Learning from the Dark Side: A Parallel Time Series Modelling Framework for Forecasting and Fault Detection on Intelligent Vehicles.  _IEEE Transactions on Intelligent Vehicles_ (2023), 1–15.  https://doi.org/10.1109/TIV.2023.3342648
  * Xing et al. (2024) Yang Xing, Zhongxu Hu, Xiaoyu Mo, Peng Hang, Shujing Li, Yahui Liu, Yifan Zhao, and Chen Lv. 2024.  Driver Steering Behaviour Modelling Based on Neuromuscular Dynamics and Multi-Task Time-Series Transformer.  _Automotive Innovation_ 7, 1 (2024), 45–58.  https://doi.org/10.1007/s42154-023-00272-x
  * Xing et al. (2020) Yang Xing, Chen Lv, and Dongpu Cao. 2020.  _Advanced Driver Intention Inference: Theory and Design_.  Elsevier.
  * Xing et al. (2021) Yang Xing, Chen Lv, Dongpu Cao, and Peng Hang. 2021.  Toward Human-Vehicle Collaboration: Review and Perspectives on Human-Centered Collaborative Automated Driving.  _Transportation Research Part C: Emerging Technologies_ 128 (2021), 103199.  https://doi.org/10.1016/j.trc.2021.103199
  * Xiong et al. (2024) Yu Xiong, Zhipeng Hu, Ye Huang, Runze Wu, Kai Guan, Xingchen Fang, Ji Jiang, Tianze Zhou, Yujing Hu, Haoyu Liu, Tangjie Lyu, and Changjie Fan. 2024.  XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniques.  arXiv:2402.12685 [cs]
  * Xu et al. (2016) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2016.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.  arXiv:1502.03044 [cs]
  * Xu et al. (2022) Shiyun Xu, Zhiqi Bu, Pratik Chaudhari, and Ian J. Barnett. 2022.  Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsity.  https://doi.org/10.48550/arXiv.2202.12482 arXiv:2202.12482 [cs, math, stat]
  * Xu and Dainoff (2023) Wei Xu and Marvin Dainoff. 2023.  Enabling Human-Centered AI: A New Junction and Shared Journey between AI and HCI Communities.  https://doi.org/10.48550/arXiv.2111.08460 arXiv:2111.08460 [cs]
  * Xu et al. (2023) Xuhai Xu, Anna Yu, Tanya R. Jonker, Kashyap Todi, Feiyu Lu, Xun Qian, João Marcelo Evangelista Belo, Tianyi Wang, Michelle Li, Aran Mun, Te-Yen Wu, Junxiao Shen, Ting Zhang, Narine Kokhlikyan, Fulton Wang, Paul Sorenson, Sophie Kim, and Hrvoje Benko. 2023.  XAIR: A Framework of Explainable AI in Augmented Reality. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, 1–30.  https://doi.org/10.1145/3544548.3581500
  * Yang et al. (2017) Guang-Zhong Yang, James Cambias, Kevin Cleary, Eric Daimler, James Drake, Pierre E. Dupont, Nobuhiko Hata, Peter Kazanzides, Sylvain Martel, Rajni V. Patel, Veronica J. Santos, and Russell H. Taylor. 2017.  Medical Robotics—Regulatory, Ethical, and Legal Considerations for Increasing Levels of Autonomy.  _Science Robotics_ 2, 4 (2017), eaam8638.  https://doi.org/10.1126/scirobotics.aam8638
  * Yang et al. (2023) Qian Yang, Yuexing Hao, Kexin Quan, Stephen Yang, Yiran Zhao, Volodymyr Kuleshov, and Fei Wang. 2023.  Harnessing Biomedical Literature to Calibrate Clinicians’ Trust in AI Decision Support Systems. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3544548.3581393
  * Yau et al. (2020) Herman Yau, Chris Russell, and Simon Hadfield. 2020.  What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes.  https://doi.org/10.48550/arXiv.2011.05064 arXiv:2011.05064 [cs, stat]
  * Yin et al. (2019) Changchang Yin, Rongjian Zhao, Buyue Qian, Xin Lv, and Ping Zhang. 2019.  Domain Knowledge Guided Deep Learning with Electronic Health Records. In _2019 IEEE International Conference on Data Mining (ICDM)_. IEEE, Beijing, China, 738–747.  https://doi.org/10.1109/ICDM.2019.00084
  * Yu et al. (2023) Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin D. Haeffele, and Yi Ma. 2023.  White-Box Transformers via Sparse Rate Reduction.  https://doi.org/10.48550/arXiv.2306.01129 arXiv:2306.01129 [cs]
  * Yuan et al. (2022) Luyao Yuan, Xiaofeng Gao, Zilong Zheng, Mark Edmonds, Ying Nian Wu, Federico Rossano, Hongjing Lu, Yixin Zhu, and Song-Chun Zhu. 2022.  In Situ Bidirectional Human-Robot Value Alignment.  _Science Robotics_ 7, 68 (2022), eabm4183.  https://doi.org/10.1126/scirobotics.abm4183 arXiv:https://www.science.org/doi/pdf/10.1126/scirobotics.abm4183
  * Yun and Yang (2020) Hanna Yun and Ji Hyun Yang. 2020.  Multimodal Warning Design for Take-over Request in Conditionally Automated Driving.  _European Transport Research Review_ 12, 1 (2020), 34.  https://doi.org/10.1186/s12544-020-00427-5
  * Zaheer et al. (2021) Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2021.  Big Bird: Transformers for Longer Sequences.  arXiv:2007.14062 [cs, stat]
  * Zhang et al. (2021b) Dandan Zhang, Ruoxi Wang, and Benny Lo. 2021b.  Surgical Gesture Recognition Based on Bidirectional Multi-Layer Independently RNN with Explainable Spatial Feature Extraction.  https://doi.org/10.48550/arXiv.2105.00460 arXiv:2105.00460 [cs]
  * Zhang et al. (2021a) Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin I. P. Rubinstein. 2021a.  Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors.  https://doi.org/10.48550/arXiv.2006.15417 arXiv:2006.15417 [cs]
  * Zhang et al. (2023) Xijia Zhang, Yue Guo, Simon Stepputtis, Katia Sycara, and Joseph Campbell. 2023.  Explaining Agent Behavior with Large Language Models.  arXiv:2309.10346 [cs]
  * Zheng et al. (2020) Min Zheng, Jessecae K. Marsh, Jeffrey V. Nickerson, and Samantha Kleinberg. 2020.  How Causal Information Affects Decisions.  _Cognitive Research: Principles and Implications_ 5, 1 (2020), 6.  https://doi.org/10.1186/s41235-020-0206-z
  * Zhong et al. (2023) Zhiqiang Zhong, Yangqianzi Jiang, and Davide Mottin. 2023.  On the Robustness of Post-hoc GNN Explainers to Label Noise.  https://doi.org/10.48550/arXiv.2309.01706 arXiv:2309.01706 [cs]
  * Zhu et al. (2018) Jichen Zhu, Antonios Liapis, Sebastian Risi, Rafael Bidarra, and G Michael Youngblood. 2018.  Explainable AI for designers: A human-centered perspective on mixed-initiative co-creation. In _2018 IEEE conference on computational intelligence and games (CIG)_. IEEE, 1–8.
  * Zini and Awad (2022) Julia El Zini and Mariette Awad. 2022.  On the Explainability of Natural Language Processing Deep Models.  _Comput. Surveys_ 55, 5 (2022), 1–31.  https://doi.org/10.1145/3529755 arXiv:2210.06929 [cs]

Online Appendix to: Explainable Interface for Human-Autonomy Teaming: A Survey

##  Appendix A EXPLAINABLE INTERFACE APPLICATION SCENARIOS IN HAT: CASE
STUDIES

Thougn EIs play a pivotal role in enhancing the efficiency of HAT in high-risk
sectors like healthcare, transportation, and manufacturing, there is a lack of
empirical studies examining end-users’ explainability needs and behaviors
around XAI explanations in real-world contexts (Kim et al., 2023a). Research
(Beer et al., 2014) outlines a framework to evaluate robot autonomy levels in
human-robot interaction (HRI). It redefines autonomy as a robot’s ability to
perceive, plan, and act in its environment for specific tasks. A 10-point
taxonomy categorizes levels of robot autonomy (LORA), assessing how these
levels influence key HRI factors like acceptance and reliability. The study
also underscores autonomy’s interplay with constructs like safety and trust in
HRI, essential for comprehending autonomy’s role and impacts in these
interactions. We summarize related researches as Table 2 which illustrates
how, across varying levels of autonomy, the roles of humans and autonomous
systems differ, thereby influencing the functionality of EIs in these
contexts. Notably, the concept of variable autonomy in HAT for firefighting,
introduced in (Verhagen et al., 2024), allows for the dynamic adjustment of
robot autonomy based on situational demands. This approach leads to efficient
task sharing and role adaptation between humans and robots, implying a need
for similarly dynamic EIs in these contexts.

Table 2. Design of Explainable Interface for Different Autonomy Levels Level |  Human Role |  Autonomy Role |  Functions of EIs
---|---|---|---
Low Autonomy |  Direct Controller: Executes immediate control, frequent commands, continuous monitoring |  Active Executor: Acts upon human input nearly real-time, no independent decision-making. |  Clarify operational mechanisms and respond to direct human commands. (Beer et al., 2014; Berlin et al., 2006)
Medium Autonomy |  Supervisor/Guide: Involves setting parameters, providing intermittent input, and intervening in complex decisions beyond the agent’s programming. |  Assistant: Offers insights into decision-making under specific conditions |  Provide insights into decision-making, especially under specific conditions or parameters. (Beer et al., 2014; Lieberman et al., 2020)
High Autonomy |  Overseer/Monitor: Engages in high-level monitoring and intervenes in unexpected situations. |  Self-regulated Actor/Adaptive Manager: Performs tasks independently, seeks input when uncertain |  Offer detailed explanations of complex decisions, highlighting the influencing algorithms, behaviors, or data. Justify actions retrospectively (Lyons et al., 2018).
Full Autonomy |  Initializer/Auditor: Establishes initial parameters and high-level goal setting. Beyond the setup, humans are mostly hands-off, only intervening in exceptional cases or for maintenance. |  Sovereign Operator: Manages all tasks autonomously, constantly improving or adapting its strategy based on goals and feedback from its environment. Human input is rare. |  Explain overall strategies and objectives, clarify adjustments and rationale in autonomous strategies, and ensure consistency between human intentions and autonomous actions. (Lyons et al., 2018)

###  A.1. Healthcare

####  A.1.1. Clinical Decision Support Systems

EIs are vital in Clinical Decision Support Systems (CDSS) for clinician
acceptance and transparency of AI’decision-making is more effective than
merely presenting AI confidence levels (Panigutti et al., 2022). Factors
influencing the effectiveness of explainability in CDSS include technical
integration, algorithm validation, deployment context, decision-making roles,
and user demographics (Amann et al., 2022). Computer-based CDSS differ in
operation, knowledge sources, and workflow impact. A substantial portion
focuses on patient-centered decision support, while a smaller percentage
provides online support for clinicians. Most require additional personnel for
data management (Berlin et al., 2006). Implementing CDSS involves considering
delivery methods, governance, incentives, workflow integration, content
updates, and data quality (Wright et al., 2010). AI-based CDSS are evolving
with complex algorithms, analyzing large datasets for precise, personalized
support, improving healthcare quality and efficiency. However, issues such as
data privacy, transparency of algorithms, and integration into existing
workflows present ongoing challenges (Ramgopal et al., 2023). Key elements in
the design and implementation of EIs for CDSS include:

  * •

Real-time visualization: EIs should display system’s decision-making processes
and actions in real-time, allowing users to observe and understand how the
system arrives at its conclusions. This feature enables clinicians, operators,
or users to monitor and intervene when necessary (Amann et al., 2022).

  * •

Alerts and notifications: EIs generate alerts and notifications to inform
users about potential complications, anomalies, or critical events. This
feature enables timely intervention by support staff, who can take appropriate
actions based on the provided information.

  * •

Context-aware assistance: EIs should deliver situation-specific information,
like patient condition, current procedures, and relevant literature, aiding
support staff, nurses, and anesthesiologists in recognizing potential issues
and making informed decisions (Yang et al., 2023). External Explanations also
matters, such as biomedical literature or prediction process allow clinicians
to validate system reasoning against their own knowledge and determine if
recommendations are clinically appropriate (Pierce et al., 2022; Yang et al.,
2023).

  * •

Natural language explanation: Providing natural language explanations and
visualizations can help clinicians understand the robot’s control logic and
reasoning, improving trust and confidence in the technology (Markus et al.,
2021). Study (Umerenkov et al., 2023) shows that LLMs in CDSS can clarify
medical diagnoses but must be used cautiously.

  * •

User-Centered Design: Emphasizing the importance of aligning with clinical
workflows and user needs, human-centered design is crucial in CDSS development
(Wang et al., 2023). This approach involves iterative processes, including
prototyping, testing, and redesigning (Panigutti et al., 2023).

####  A.1.2. Surgical Robotics

The development of autonomy in surgical robots can be categorized based on a
combination of human involvement and the robot’s level of proficiency. At
Level 0, the robot is completely under the control of the surgeon, serving
merely as a tool. In contrast, Level 5 envisions a robot performing an entire
surgery independently. Intermediate levels involve the robot transitioning
from providing mechanical assistance to autonomously executing specified
tasks, suggesting surgical methods awaiting human confirmation, and even
making medical judgments under the supervision of a doctor (Yang et al.,
2017). This range from Level 0 to Level 5 encapsulates a progression from
total human oversight to complete self-sufficiency of the robot. Currently,
most surgical robots operate at the level 1 and level 2, with the surgeon
playing a significant role in the procedure. However, advancements in AI and
sensors are improving the precision, grip, feedback, and autonomy of robotic
surgery systems (Fiorini et al., 2022).

Figure 8 illustrates Level 1, ’Robot Assistant’, where a continuous
interaction loop exists between the developer and the doctor. In this level,
EI provides tool status updates and enhanced visualizations, like 3D
renderings of the surgical area, for clearer insights (Dappa et al., 2016).
Level 2, ’Task Autonomy’, positions the doctor in a supervisory role. Here,
control over the robot is discrete rather than continuous. An application of
this is in spinal surgery, where robotic guidance is integrated with
preoperative planning software (Lieberman et al., 2020). EIs for doctor
include feedback on decisions, reasons, confidence levels, alternative
suggestions, and real-time updates. In the future, it is possible that
surgical robots will achieve higher levels of autonomy, allowing them to
perform more complex tasks independently while still being supervised by human
surgeons. However, there also are researches arguing for a shift in focus from
complete autonomy to enhancement of human capabilities in the context of
autonomous surgery (Battaglia et al., 2021). As AI becomes more integrated
into healthcare, challenges arise, especially concerning explainability,
utility, trust, legal, regulatory, and ethical (Yang et al., 2017; O’Sullivan
et al., 2019; Wysocki et al., 2023).

Figure 8. Explainable Interface in level 1 and level 2 surgical robotics

Figure 1 illustrates Level 1, ’Robot Assistant’, where a continuous
interaction loop exists between the developer and the doctor. In this level,
EI provides tool status updates and enhanced visualiza- tions, like 3D
renderings of the surgical area, for clearer insights. Level 2, ’Task
Autonomy’, positions the doctor in a supervisory role. Here, control over the
robot is discrete rather than continuous.

The surgical robotics field is characterized by varied interaction methods to
enhance surgical precision or improve experience. Developments include robots
capable of environment reconstruction and force feedback (Li and Kesavadas,
2018). Minimally invasive surgery primarily consists of a series of sub-tasks,
which can be broken down into basic gestures or contexts. Surgical gesture
recognition can aid in motion planning and decision-making, as well as
establish context-aware knowledge to improve the control quality of surgical
robots (Zhang et al., 2021b). Furthermore, as robots become more autonomous,
the importance of strategic task planning becomes increasingly clear (Ginesi
et al., 2020). There is a trend towards optimizing shared autonomy, advocating
for a more collaborative approach between humans and robots. This human-in-
the-loop collaborative method enhances human capabilities alongside robotic
autonomy (Gopinath et al., 2017). A study reviews current surgical robotic
interfaces, highlighting advancements and limitations in feedback and
visualization, and proposes a novel design featuring bedside access, sterile
procedure, 3D head-mounted visualization, and intuitive control for enhanced
surgical outcomes (Simorov et al., 2012).

####  A.1.3. Challenges fo Explainable Interface in Healthcare

Based Advancements in healthcare suggest that their future depends not only on
technological progression but also on creating interfaces that integrate
seamlessly with AI, prioritizing clarity and comprehensibility. Design and
implementing EIs in healthcare faces unique challenges:

  * •

Complexity of Medical Data: The inherently intricate and varied nature of
medical data, including electronic health records, medical imaging, genomics,
and sensor data, presents a significant challenge. XAI systems must be adept
at interpreting and explaining decisions based on this diverse, high-
dimensional data (Arora et al., 2023). Furthermore, the lack of standardized
datasets exacerbates the difficulty of model training.

  * •

Real-time Clinical Decision Making: Given the high stakes involved in
healthcare decisions, the critical nature of surgical procedures, the
necessity for real-time decision-making is paramount.(Amann et al., 2022).

  * •

Varied User Expertise: The range of technical and medical knowledge among XAI
system users in healthcare, such as doctors, nurses, and patients, varies
widely (Kim et al., 2024). Crafting explanations that are understandable to
this diverse user base is a considerable challenge, necessitating
customization to suit different comprehension levels.

  * •

Regulatory Compliance and Ethical Considerations: Healthcare’s strict
regulatory environment demands compliance, especially as the FDA starts to
view certain AI models as medical devices, necessitating stringent regulatory
frameworks (Health, 1246).

  * •

Integration with Clinical Workflows: The effectiveness of EIs in healthcare
hinges on their seamless integration with existing clinical workflows. This
necessitates interdisciplinary collaboration to ensure that the implementation
of these interfaces does not introduce additional complexity or time-consuming
burdens for healthcare professionals (Panigutti et al., 2023).

  * •

Evaluating Explanation Effectiveness: The difficulty in evaluating healthcare
explanations lies in their subjective and objective combined nature, as
effectiveness may vary based on their role in clinical decision-making,
patient understanding, and improving outcomes.

  * •

Data Privacy and Security: Ensuring the privacy and security of sensitive
health data used in XAI systems is paramount. This includes addressing
concerns related to data sharing, consent, and anonymization (Albahri et al.,
2023).

###  A.2. Transportation

####  A.2.1. Autonomous Vehicles

HAT in transportation and autonomous vehicles (AVs) can be categorized into
six levels of driving automation, as defined by the Society of Automotive
Engineers (SAE) (Committee, 2021). As the level of automation increases, the
role of the human driver shifts from active control to supervision. This
transition can lead to challenges in maintaining driver engagement, situation
awareness, and trust in the automation system (Beggiato et al., 2015). Level 3
and higher AVs are equipped to handle full control, allowing drivers to focus
on non-driving activities. However, human intervention is still required in
certain situations. Providing explanations within AVs enhances the driver’s
sense of control and understanding of the vehicle’s status and capabilities,
reducing potential anxiety. These intervention needs differ depending on non-
driving tasks, driving conditions, and takeover methods. Personalized AVs,
tailored to individual driver behaviors, enhance reliability and foster
greater trust in the system (Salubre and Nathan-Roberts, 2021; Kyriakidis et
al., 2019). Shahin et al.’s survey about XAI in autonomous driving, proposes
an framework for a safer, transparent, publicly approved, and environmentally
friendly intelligent vehicles (Atakishiyev et al., 2023). As self-driving cars
become more prevalent, regulatory bodies might see the value in having these
types of interfaces (Xu and Dainoff, 2023). Implementing such interfaces not
only be beneficial for users but can also help manufacturers meet demands and
gain approvals from regulatory agencies (Biondi et al., 2019). To present EIs
inside and outside vehicles, visual, auditory, and haptic/tactile aspects
should be considered:

  * •

Visual interface: Visual interfaces in AVs have evolved from traditional
physical controls to advanced touchscreens and augmented reality head-up
displays (AR-HUD). They enable passengers to set destinations, monitor
progress, environment, vehicle status, and manage settings like climate and
entertainment, thereby improving efficiency in control takeover, enhancing
user satisfaction, and trust (Oliveira et al., 2018; Jing et al., 2022).

  * •

Auditory interface: Speech output is highly relevant to improve driver
attitudes that affect acceptance of automated systems is brought forth
(Forster et al., 2017). In AVs, auditory user interfaces are designed to
ensure safe control transitions, especially during the conditional self-
driving phase. These interfaces use a combination of voice and sound feedback
to communicate with the user. Compared to generic auditory output,
communicating upcoming automated manoeuvres additionally by speech led to a
decrease in self-reported visual workload and decreased monitoring of the
visual HMI (Naujoks et al., 2017).

  * •

Haptic/tactile interface: AVs can enhance user experience and control through
the integration of haptic feedback interfaces. A study compared two types of
driver-vehicle interfaces for controlling vehicles: one using a haptic
interface with both kinesthetic and tactile feedback, and the other using a
hand-gesture interface with augmented reality feedback. Compared to manual
driving, semi-autonomous control significantly reduced the perceived workload.
The haptic interface, in particular, proved to be especially reliable and
effective in comparison to the gesture interface, as it provides direct and
immediate feedback to the driver (Manawadu et al., 2016).

Multimodal explanations utilizing two types of displays are more effective
than unimodal or three-display multimodal explanations, with speech-based
displays producing the best takeover performance. The combination of visual,
auditory, and tactile cues, especially visual-auditory warnings, improves
response efficiency in scenarios like vehicle malfunctions and highway exits
(Lee et al., 2023; Yun and Yang, 2020). The MIRIAM study demonstrates a
multimodal interface combining visual indicators with a conversational agent,
maintaining situation awareness across various cognitive styles and enhancing
user engagement through natural language interaction (Robb et al., 2018). The
timing of information delivery is as crucial as its content, with traffic risk
probability guiding when explanations are provided, particularly to avoid
overwhelming passengers with excess information (Kim et al., 2023b). Figure 9
delineates the requirements for EIs in AVs across three operational
conditions: ’good condition’ for standard autonomous operation, ’specific
condition’ for complex scenarios like adverse weather requiring increased
notifications, and ’takeover condition’ triggered by system errors or
unpredictable circumstances necessitating control transfer. EIs should avoid
overloading users with complex technical details, instead clearly presenting
current and anticipated environmental conditions, vehicle behaviors, and
control information. The interface should also allow users to retrospectively
track significant environmental changes, behaviors, and control shifts during
the journey. Incorporating predictive maintenance data, such as remaining
useful life and time-to-failure, is important (Shukla et al., 2020). As for
remote control, more comprehensive dynamic team situation awareness is needed
(Mutzenich et al., 2021).

Figure 9. Explainable Interface Requirements for Human Vehicle Interaction in
Conditional Driving Automation

Figure 2 delineates the requirements for EIs in AVs across three operational
conditions: ’good condition’ for standard autonomous operation, ’specific
condition’ for complex scenarios like adverse weather requiring increased
notifications, and ’takeover condition’ triggered by system errors or
unpredictable circumstances necessitating control transfer. EIs should avoid
overloading users with complex technical details, instead clearly presenting
current and anticipated environmental conditions, vehicle behaviors, and
control information. The interface should also allow users to retrospectively
track significant environmental changes, behaviors, and control shifts during
the journey.

In evaluating autonomous systems, it is imperative to assess understanding,
trust, and satisfaction in real-time interactions during the operation of AVs.
This approach ensures more accurate and useful results compared to traditional
questionnaires, which may not be effective in this context. Measurements
should integrate the use of existing vehicle devices, such as cameras, and
wearable devices capable of monitoring human psychological activities for
dynamic modelling of human behavior in HAT.

####  A.2.2. Air Transportation Systems

The integration of EIs is pivotal in enhancing the functionality and safety of
transportation systems. The European Union Aviation Safety Agency (EASA) is
actively working towards developing guidelines for AI applications intended
for safety-related applications in aviation. The stakeholders, including air
traffic controllers, airport operators, and Uncrewed Traffic Management (UTM)
service providers, seek assurance that the AI solutions are trustworthy and
safe (Torens et al., 2022). Developing robust solutions to manage, measure,
and maintain trust between humans and autonomous systems is important,
particularly in maintaining existing safety standards in aviation.

In Air Traffic Management (ATM), XAI is instrumental in managing air traffic
flow and capacity, ensuring efficient resource utilization, and minimizing
delays. leading to the funding of various projects by SESAR (Single European
Sky ATM Research). Among these, the TAPAS specializes in air traffic flow,
capacity management, and air traffic control services within ATM. The
project’s scope includes establishing transparency requirements, developing a
prototype, implementing an explainable layer, and formulating principles for
transparency in ATM automation processes, as referenced in (TAPAS, 2022).
ARTIMATION improves runway delay predictions and optimizes departures and
inspections by enhancing explainability with heatmaps and decision-based
storytelling. Additionally, the Mindtooth tool evaluates air traffic
controllers’ reactions, including Approach-Withdrawal, stress, and workload
using neurophysiological metrics. This is particularly relevant considering
that time pressure is a critical factor in high-stakes environments like air
traffic control, where rapid and accurate decision-making is imperative
(ARTIMATION, 2022).

Concerning Demand-Capacity Balancing (DCB), challenges such as congestion or
hotspots occur when airspace demand exceeds capacity. Research in this area
includes a multi-agent RL approach that uses a rule-based environment and a
modified proximal policy optimization (MAPPO) framework. This approach
integrates a supervisor for enhanced learning, effectively balancing demand,
and capacity in air traffic flow management (Tang and Xu, 2021). In (Kravaris
et al., 2023), EI enhances the explainability of deep multi-agent
reinforcement learning solutions for demand-capacity balance (DCB) problems,
employing visual analytics for transparent decision-making. Additionally, XAI
can drive secure data chains in conjunction with blockchain networks for
autonomous ATM, ensuring data integrity in ATM operations (Axon et al., 2023).

####  A.2.3. Challenges for Explainable Interface in Transportation

Summarizing key findings, integrating XAI into transportation systems is
crucial for high safety and operational standards. Major challenges include:

  * •

Integration with Existing Infrastructure: Merging XAI with existing
transportation infrastructure, such as traffic signals and GPS systems.

  * •

Adapting to Dynamic Environments such as unpredictable traffic, weather, and
pedestrian behavior.

  * •

Real-Time Data Processing and Interaction: In high-stake real-time HAT,
swiftly processing dynamic data, provide understandable explanations in proper
time and adjust EIs according to feedback.

  * •

Real-time Evaluation: Collecting evaluation data during HAT process, without
distracting the user.

  * •

Challenges in Advanced Urban Transportation: The emergence of intricate urban
air traffic, featuring drones and eVTOLs (electric vertical take-off and
landing vehicles), presents unique challenges and prospects for incorporating
EIs in AVs and traffic control systems. Addressing these issues calls for
creative solutions to preserve both efficiency and safety within complex urban
transportation networks.

###  A.3. Explainable Interface and Digital Twin

####  A.3.1. Relationship of Physical System, Digital Twin, and XAI

The relationship between a physical system and its digital twin (DT) is
dynamic and symbiotic. The DT mirrors the life of its corresponding physical
entity, adapting its real-time behavior in response to inputs from the
physical system. In the context of Industry 4.0, Cyber-Physical Systems (CPS)
provide real-time data and control capabilities (Groshev et al., 2021).
Utilizing DT for decision-making, they enable the simulation and modelling of
various aspects of a physical system, offering valuable insights and
predictions. These simulations can serve multiple purposes, such as optimizing
performance, predicting maintenance needs, and simulating potential scenarios
in smart cities, manufacturing, healthcare (Fuller et al., 2020). EIs play a
crucial role in this relationship by ensuring transparency and
understandability in the decision-making processes within the DT.

####  A.3.2. XAI Methods for Digital Twin

XAI methods play a pivotal role in DT. For example, LSTM neural networks,
employed for lane change prediction in autonomous driving DT, use techniques
such as layer-wise relevance propagation for better clarity in their
predictions (Wehner et al., 2022). Furthermore, architectures like
TwinExplainer integrate XAI into deep learning-powered DT systems,
prioritizing the interpretability of their predictive outcomes (Neupane et
al., 2023). The study (Kobayashi et al., 2023) investigated the remaining
useful life prediction in DT systems through interpretable machine learning,
employing the PiML library for transparent and reliable AI decision-making.

####  A.3.3. Digital Twin as an Explainable Interface

DTs surpass traditional explainable interfaces (EIs) by offering interactive
features and counterfactual analysis. This improves understanding of AI-driven
decisions across various industries, like healthcare and manufacturing. For
instance, In drug discovery, DTs explain AI’s reasoning behind diagnoses and
treatments, boosting trust and confident decision-making by healthcare
professionals. (Askr et al., 2023). In the manufacturing industry, DTs can
provide real-time explanations for quality control decisions, helping
operators identify the root causes of defects and optimize production
processes (Park et al., 2019).

###  A.4. Challenges for XAI and Digital Twin Integration

The integration of XAI methodologies into DT systems presents interface design
and optimization challenges (Javed et al., 2023). First, incorporating XAI not
only as a post-hoc approach to explain the outputs, but also integrating
internal interpretable AI from the initial stages of design. Moreover, the
progression of DT technology is hindered by a general lack of understanding of
its advantages and a deficiency in technical expertise. Contributing
technologies such as 3D simulations, IoT, AI, big data, and cloud computing
are also in developmental stages, which adds to the complexity (Tao et al.,
2018). Furthermore, advancements in high-performance computing and real-time
interaction technologies are crucial for the enhancement of XAI-enabled DTs.
This necessitates a high level of technical understanding from users. To
address these issues, there is a need for the development of new methods and
frameworks. These should enable the seamless integration of XAI into DT
systems, thus lowering the learning curve for users and making it easier for
them to understand and trust these systems.