# Improving User Mental Models of XAI Systems with Inclusive Design Approaches

Md Montaser Hamid<EMAIL_ADDRESS>Oregon State UniversityUSA
0000-0002-5701-621X ,  Fatima Moussaoui<EMAIL_ADDRESS>Oregon State
UniversityUSA ,  Jimena Noa Guevara<EMAIL_ADDRESS>Oregon State
UniversityUSA ,  Andrew Anderson<EMAIL_ADDRESS>0000-0003-4964-6059
Oregon State UniversityUSA and  Margaret Burnett<EMAIL_ADDRESS>Oregon State UniversityUSA

(9999)

###### Abstract.

Explainable Artificial Intelligence (XAI) systems aim to improve users’
understanding of AI but rarely consider the inclusivity aspects of XAI.
Without inclusive approaches, improving explanations might not work well for
everyone. This study investigates leveraging users’ diverse problem-solving
styles as an inclusive strategy to fix an XAI prototype, with the ultimate
goal of improving users’ mental models of AI. We ran a between-subject study
with 69 participants. Our results show that the inclusivity fixes increased
participants’ engagement with explanations and produced significantly improved
mental models. Analyzing differences in mental model scores further
highlighted specific inclusivity fixes that contributed to the significant
improvement in the mental model.

Diversity, Inclusive AI, Explainable AI, Mental Model

††copyright: acmcopyright††journalyear: 9999††doi:
99.9999/9999999.9999999††journal: TIIS††publicationmonth: 99††ccs: Human-
centered computing Empirical studies in HCI

##  1\. Introduction

> “The curb-cut effect underscores the foundational belief that we are one
> nation, that we rise or fall together. Without equity, there can be neither
> progress nor prosperity.” (Blackwell, 2016) —Angela Glover Blackwell, author
> of the book “Curb-Cut Effect”.

Angela Glover Blackwell’s thoughts on equity resonate with our focus on
inclusive XAI. While Blackwell talks about equity, we’re focusing on
inclusivity in this paper. As researchers are working to make XAI better,
inclusivity asks the question “better for whom?”

By “better”, we mean promoting better mental models of how a user’s AI system
is reasoning. In XAI, mental model is defined as the user’s understanding of
how an AI system works (Anderson et al., 2020). More formally, mental models
are “internal representations that people build based on their experiences in
the real world” (Norman and Gentner, 1983). Helping users form suitable mental
models is the aim of XAI systems. Although such XAI systems are gaining
attention as AI becomes more widespread, the challenges of understanding these
systems are also on the rise (Weld and Bansal, 2019).

Improving explanations for one specific user group may not always benefit
others if we don’t adopt inclusive approaches in existing AI development
practices. One particular inclusive approach involves recognizing users’
diverse problem-solving styles. While demographic diversity is important,
considering problem-solving diversity generates actionable design directions
(Hamid et al., 2024). Several studies showed how addressing users’ diverse
problem-solving approaches reduced inclusivity biases from non-AI applications
and systems (Vorvoreanu et al., 2019; Guizani et al., 2022; Burnett et al.,
2016). This paper investigates whether focusing on users’ diverse problem-
solving styles can reduce inclusivity issues across an information-rich XAI
system, ultimately aiming to improve users’ mental models.

To do so, we evaluated how well a group of end users fared in two versions of
an XAI prototype. These versions were the work of a team of AI practitioners
to address inclusivity issues within their XAI prototype. Their prototype
showed a modified version of Tic-Tac-Toe games played sequentially by two AI
agents. The AI practitioners had leveraged the GenderMag inclusivity method
(Hamid et al., 2024; Burnett et al., 2016; Guizani et al., 2022; Vorvoreanu et
al., 2019) to find and fix the inclusivity issues. We ran a between-subject
study where a group of 34 participants used the “Original” version and 35
participants used the “Post-GenderMag” version. We investigated participants’
engagement with the explanations of the prototype, their ability to detect and
specify flaws, their perception of the AI agents’ behavior, and lastly their
mental models. Our investigation centered around these three research
questions:

  * •

RQ1: Did explanation usage have any impact on participants’ mental models?

  * •

RQ2: Which group of participants had a higher engagement with the
explanations?

  * •

RQ3: Which group of participants had a better mental model?

##  2\. Background and Related Works

In recent years, Human-AI Interaction (HAI) research has focused on user
performance, system reliability, trustworthiness, and safety (Shneiderman,
2020), while also exploring the dynamics of human and AI control levels
(Shneiderman, 2022). The growing body of HAI works aims to increase
transparency (Rader et al., 2018), explain AI behaviors (Herlocker et al.,
2000; Kulesza et al., 2015), provide guidelines for AI practitioners (Amershi
et al., 2019), and address ethical principles in governance (Shneiderman,
2021). Additionally, HAI research advocates for human-centric concerns such as
well-being, privacy, and cognitive capacities in the adoption of AI
technologies (Ozmen Garibay et al., 2023; Shneiderman, 2022). In our study, we
are specifically interested in bringing diverse users’ problem-solving styles
together with HAI to improve XAI’s inclusivity.

###  2.1. Problem-Solving Style and HAI/XAI

There’s been limited investigation into how users’ problem-solving styles
influence HAI/XAI. Although AI researchers have examined users’ explanation
preferences, they’ve largely overlooked individual differences within these
preferences (Dodge et al., 2019). Some research used users’ cognitive
diversity to understand why users behave or judge differently when interacting
with AI systems. For example, when looking at how fair end-users’ perception
of AI was, Dodge et al. (Dodge et al., 2019) assessed the impacts of two
factors: i) four types of explanations and ii) data processing. Their findings
revealed that participants’ need for cognition significantly impacted their
perception of the AI’s fairness. Similarly, Millecamp et al.’s work (Millecamp
et al., 2020) found that diverse end-users’ cognitive styles showed
differences in their trends to interact with the AI’s explanations. This
investigation discussed that the variation in the definitions of cognitive
styles did not have much effect and any of the definition variations could
help identify valuable insights.

Goldberg et al. (Goldberg, 1990) presented an alternative approach to
capturing individual differences through five personality traits— Openness,
Conscientiousness, Extraversion, Agreeableness, and Neuroticism (OCEAN)
(Goldberg, 1990). They found that one of the personality traits, “Openness”
influenced users’ interaction with the interface. Jiang et al. (Jiang et al.,
2000) also used a personality trait, self-confidence to evaluate participants’
acceptance of the expert system’s output. Other researchers also used some
form of problem-solving styles, cognitive styles, and personality traits to
explain user behavior (Giboney et al., 2012; Kulesza et al., 2012; Felmingham
et al., 2021; Djulbegovic et al., 2014). However, none of these works
considered a wider range of problem-solving styles as we did in our study.

In this study, we used problem-solving styles defined by users’— attitude
towards risk, computer self-efficacy, motivations, information processing
style, and learning style.

###  2.2. GenderMag

We used the GenderMag method to define the end users’ five problem-solving
style ranges used in this study. GenderMag, also known as the Gender-
Inclusiveness Magnifier, is a usability inspection method to find and fix
gender-inclusivity bugs in problem-solving software (Hamid et al., 2024;
Burnett et al., 2016; Guizani et al., 2022; Vorvoreanu et al., 2019). This
method focuses on five facets of gender differences (statistical), brings them
to life through three faceted personas, and encapsulates the use of these
facets into a systematic process through a gender specialization of the
Cognitive Walkthrough (CW) (Wharton et al., 1994). Software practitioners have
used this method to evaluate the software they design and develop from a
gender-inclusiveness perspective (Burnett et al., 2016; Guizani et al., 2022;
Vorvoreanu et al., 2019).

The GenderMag method’s five facets capture five factors in an individual’s
problem-solving style: their motivations for using technology, information
processing styles, computer self-efficacy, attitude toward risk in using
unfamiliar technology features, and learning style (by process vs. by
tinkering) (Burnett et al., 2016; Hamid et al., 2024). The “inclusivity bugs”
uncovered by GenderMag are instances where a technology product fails to fully
accommodate the complete range of values associated with the five facets,
leading to a disproportionate impact on individuals whose problem-solving
styles are unsupported. These bugs are also considered gender-inclusivity
issues because the facets reflect statistical differences among gender
preferences in problem-solving styles (Burnett et al., 2016; Guizani et al.,
2022; Vorvoreanu et al., 2019; Hamid et al., 2024). GenderMag reveals
inclusivity issues that are statistically associated with gender preferences.
GenderMag does not use peoples’ gender identities to find inclusivity
concerns, it uses only their problem-solving styles.

The GenderMag method has three personas: Abi (Abigail/Abishek), Pat
(Patricia/Patrick), and Tim (Timara/Timothy) (see Table 1). Statistically, the
problem-solving styles of Abi disproportionately skew toward women, for Tim
they disproportionately skew toward men, and Pat provides a third set of
values within Abi and Tim (Burnett et al., 2016; Hamid et al., 2024).

Table 1. Three GenderMag personas and their problem-solving styles. |  |  |
---|---|---|---
|  Abigail/Abishek (”Abi”) |  Patricia/Patrick (”Pat”) |  Timara/Timothy (”Tim”)
Attitude Toward Risk |  Risk-Averse |  Risk-Averse |  Risk-Tolerant
Self-Efficacy |  Lower |  Medium |  Higher
Motivations |  Task-Oriented |  Task-Oriented |  Tech-Oriented
Information Processing Style |  Comprehensive |  Comprehensive |  Selective
Learning: by Process vs. by Tinkering |  Process-Oriented |  Learns by tinkering: tinkers reflectively |  Learns by tinkering (sometimes to excess)
Abi’s and Tim’s problem-solving styles are at the two ends and Pat is in the
middle with a mix of possible values.

The GenderMag faceted personas are integrated into a systematic specialized
Cognitive Walkthrough process. A group of evaluators adopt a specific
GenderMag persona, say Abi, customize the persona’s background (optional), and
systematically “walkthrough” each step of a use case. At each step, they
answer three questions: i) will ¡Abi¿ have formed this subgoal as a step to
their overall goal? why?, ii) will ¡Abi¿ take the action the developers
expected at this step? Why?, and iii) if ¡Abi¿ took the action, will they know
that they did the right thing and are making progress toward their goal? Why?
If the evaluators find any answer is negative because of Abi’s particular
problem-solving styles, they report an inclusivity bug, provide their reasons,
and note down the problem-solving styles that support their reasonings. Later,
when fixing an inclusivity bug, these notes ensure that the fixes are designed
around Abi’s problem-solving styles that highlighted the bug.

The evaluators relevant to our study were a group of AI practitioners whom we
called “Team J” in this paper. Team J evaluated their own XAI prototype from
the perspective of the Abi persona. They chose the Abi persona as Abi
represents the group of users who are most often left out of the development
process which results in more barriers for these users. Team J’s before
GenderMag and after GenderMag version of the prototype were then used in our
study.

###  2.3. XAI in Sequential Domains

The AI practitioners of our study designed their XAI prototype specifically
for a sequential domain. A sequential domain refers to a specific situation or
area where actions or events unfold in a particular order, following a series
of steps or stages in a predetermined sequence. For instance, consider a board
game like Chess, where players take turns making moves and the progression of
the game follows an ordered sequence of moves. In recent years, there has been
a growing interest in developing AI systems that can play sequential games
like Chess, Shogi, and Go. This has led to the development of AI players that
can compete with and even outperform the world champions of these games
(Silver et al., 2016, 2018). Real-Time Strategy (RTS) games have also seen
similar achievements (Vinyals et al., 2019b, a).

Despite such progress, explaining AI players’ actions in sequential domains
remains a major challenge. In an attempt to understand this challenge, Metoyer
et al. (Metoyer et al., 2010) studied how experienced users explain strategies
and gameplay in RTS games. In another study, Penney et al. (Penney et al.,
2018) found that even experienced users encounter challenges when seeking
information about an intelligent agent in an RTS setting. One result was that
participants showed a preference for “What” information over “Why”. In a
sequential game domain, understanding the causal relationships among relevant
variables within the game can also help users (Madumal et al., 2020). As
Madumal et al. (Madumal et al., 2020) showed offering causal explanations of
AI agents can enhance participants’ ability to predict tasks, as well as
increase their satisfaction and trust. In another study, Dodge et al. (Dodge
et al., 2022) observed that when ranking AI agents playing a sequential game
like Tic-Tac-Toe, participants highlighted the importance of detailed
explanations to differentiate between closely matched agents. That study also
noted that participants showed a preference for receiving multiple
explanations over a single one when making decisions.

###  2.4. Mental Models and Explainable AI (XAI)

The opaque nature of AI systems poses a challenge for end-users with low or no
expertise in AI (Wortham et al., 2017). This especially hinders users’ ability
to seamlessly interpret and trust the system’s output (Kulesza et al., 2015).
Lim et. al (Lim et al., 2009) regarded such lack of clarity as an
intelligibility concern for AI systems.

To improve the clarity of an AI system, researchers and developers often
provide explanations of the system to its end-users. According to Hoffman et
al., explanations that are effective and satisfactory for users can facilitate
the development of a good mental model (Hoffman et al., 2019). Kulesza et al.
demonstrated that providing explanations about the debugging process of a
machine-learning system improved participants’ understanding of the system
(Kulesza et al., 2015, 2010). Also, the explanations improved their
participants’ ability to rectify errors. Other studies also found additional
benefits of adding explanations such as improvements in users’ self-reported
satisfaction (Kapoor et al., 2010; Amershi et al., 2014) and feelings of trust
towards the AI systems (Lim et al., 2009).

Hoffman et al. (Hoffman et al., 2019) proposed measures that evaluate the
quality, user satisfaction, and user understanding of the explanations.
According to Hoffman et al., mental models can be deduced from empirical
evidence, even though individuals might find it challenging to fully
articulate their understanding (Hoffman et al., 2019). However, the authors
suggested that through guided task reflection, people can effectively
communicate their understanding of an event or system, including its
fundamental concepts and principles. The authors provided a list of 11 methods
that researchers have used to derive mental models across various domains,
including cognitive psychology, expert system knowledge acquisition,
instructional design, and educational computing. From these 11 methods,
Anderson et al. used the “Prediction Task” and “Task Reflection” to understand
users’ mental models of a Reinforcement Learning agent (Anderson et al.,
2019). In our paper, we have utilized these two methods and also integrated
the “Glitch Detector Task”, and “Self-Explanation Task” from Hoffman et al.’s
11 methods to know if users were able to find any flaws in the given
explanations and their own understanding of the AI agent.

Thus, although HAI and XAI research have made significant progress in
exploring users’ mental models, they often overlooked the potential impact of
users’ diverse problem-solving styles on their overall understanding of AI
systems. Recognizing this gap, we wanted to check the effectiveness of
GenderMag-driven cognitive walkthroughs that put users’ diverse problem-
solving styles at the center stage of finding and fixing inclusivity issues.
Historically, HCI practitioners leveraged GenderMag for a wide variety of user
experience improvements such as inclusivity bug removal (Guizani et al.,
2022), task performance improvement (Vorvoreanu et al., 2019), customer rating
improvement (Burnett et al., 2017), etc. However, no prior GenderMag research
has evaluated GenderMag’s applicability and effectiveness within AI-powered
systems. Addressing this gap in GenderMag studies, our goal is to investigate
how successful GenderMag fixes can be in improving users’ engagement and their
mental models of AI.

##  3\. Methodology

###  3.1. Experimental Design

We set up a between-subject study in a lab to evaluate the two versions of
Team J’s XAI prototype. We discuss the details of our study here.

####  3.1.1. Participants

For recruiting participants, we first sent out flyers through the internal
mailing lists of our university. We also posted flyers in different locations
around campus. The recruitment emails and flyers had the invitation link to a
Qualtrics survey where we collected demographic information. We also asked 2
eligibility questions to make sure our participants were at least 18 years of
age and had not taken any AI courses as we wanted to recruit people with no AI
background. Based on these criteria, we dropped 41 responses out of 151 and
sent out emails to the remaining 110 respondents individually. We offered them
the flexibility to choose from multiple days and times that suited their
schedule. 77 respondents accepted our invitation and provided their
availability. In the end, 69 of them participated in our study.

Our study participants came from diverse backgrounds, spanning different age
groups, academic disciplines, and genders. To illustrate, we had participants
from 39 unique college majors. Table 2 provides a summary of their demographic
information.

Table 2. Summary of participants’ demographic information.

Gender | Participants#
---|---
Woman | 36
Man | 28
Non-Binary | 2
Prefer not to state | 2
Woman, Transgender | 1

Age | Participants#
---|---
18-19 | 9
20-29 | 38
30-39 | 17
40-49 | 2
50-59 | 3

College Standing | Participants#
---|---
Undergraduate student | 31
Graduate student | 22
Already graduated | 9
Other | 8

####  3.1.2. Procedures

As we described in Section 3.2, Team J designed 2 versions of their game
prototype: the “Original” and the “Post-GenderMag”. We assigned participants
to sessions based on their availability. We then randomly assigned either the
Original or the Post-GenderMag version of the prototype to each session. The
Original sessions included a total of 34 participants, while the Post-
GenderMag sessions had 35 participants. Each session lasted 2 hours and had 2
to 6 participants. We conducted a total of 17 sessions from August 18 to
August 28, 2023.

Upon arriving at the lab, participants filled out the GenderMag survey
(provided in the supplemental document of (Vorvoreanu et al., 2019)) in which
participants answered Likert-scale items about their problem-solving styles.
Then we began the session with a 25-minute hands-on tutorial covering the
game, agents, explanations, and participants’ tasks. Afterward, participants
went through the first three moves of a practice game to get familiar with the
prototype and the study questions.

In the main part of the study session, participants observed 3 games of Tic-
Tac-Toe played by 2 AI agents: Agent Blue (X) and Agent Red (O). Before Agent
Blue made a move, participants predicted the move, provided reasoning, and
their confidence in it. After the move, they identified flaws and suggested
changes. After each game, they answered two After-Action Review for AI
(AAR/AI) questions. Once all three games concluded, they completed post-task
questions covering agent descriptions, explanation feedback, and subjective
workload assessment using NASA TLX (NASA, [n. d.]). Table 3 lists all the
questions.

Table 3. The questions we asked both groups of participants. Before Agent Blue
made a move
---
\- Which square do you think the blue agent will place its next move?
\- Why do you think that the agent will place its move there? (A few
sentences, as detailed as possible)
\- How confident are you that the agent will select the square that you
predicted? On a scale of 1 (not confident at all) to 10 (extremely confident).
After Agent Blue made a move
\- From looking at the explanations (right), do you think there is a flaw in
the agent’s reasoning? If so, what is that flaw? (A few sentences, as detailed
as possible)
\- What CHANGES would you make in the decisions taken by the agent? (A few
sentences, as detailed as possible.)
After each game: AAR/AI
\- What happened in this game (write any good, bad, or interesting things
you’ve observed in these past moves and/or games)? (A few sentences, as
detailed as possible.)
\- Is there anything in the explanation that helps you understand why the AI
you’re assessing did the things it did? Please specify which explanation you
are referring to. ($\sim$2 sentences)
After finishing all 3 games
\- Please describe the steps in the agent’s approach or method. (A few
sentences, as detailed as possible.)
\- Explanation feedback 111Participants provided feedback for each of the
three explanations, BTW, STT, and OTB. They responded on a 7-point Likert-
scale from Very Strongly Disagree to Strongly Agree.: i) the explanation
helped in understanding Agent Blue, and ii) the explanation was sufficiently
detailed.
\- NASA TLX.

From these questions, we have not used the responses from “Please describe the
steps in the agent’s approach or method…”. When the user study was over, we
found that Agent Blue’s noise levels were different between each game. This
means participants would be describing multiple agents rather than one.
Despite Team J’s lead (who created the agent) suggesting that the noise
differences didn’t affect the agent’s behavior much, we excluded these
responses from our analysis to ensure clarity in our evaluation of users’
understanding of AI.

####  3.1.3. Qualitative Analysis Methods

After all the study sessions were over, we used affinity diagramming to
understand and organize the participant responses. Five of the researchers
independently marked responses that they thought described participants’
mental model. We then discussed the results as a group and developed the
initial codeset. We also took feedback from Team J’s lead (who created the AI
agents) to further modify the initial codeset. Finally, the codeset had 35
codes divided into four code concepts: “Prediction Why”, “Strategy Should Be”,
“Flaws”, and “Agent Sense, Game Sense” (Table LABEL:tab:codes_definitions).
Four of the researchers (R1, R2, R3, and R4) in a combination of two then
independently coded a small portion of the participant responses for each of
the four code concepts. Table 4 lists the inter-rater reliability rates (IRR)
(Stemler, 2004) of the agreement tests. Given these high IRRs, one researcher
from each coders group finished the rest of the coding for their assigned code
concept.

Table 4. Agreement test results. Code Concept | IRR% | Test Data% | Coders
---|---|---|---
Prediction Why (7) | 82% | 20% | R1, R2
Flaws (7) | 80% | 26% | R1, R3
Strategy Should be (4) | 81% | 25% | R1, R2
Agent Sense (14), Game Sense (3) | 81% | 20% | R1, R4
The number inside the parentheses is the number of codes inside that code concept. |

###  3.2. Where the Two Versions of the Prototype Came From

Before our study, Team J used GenderMag to find and fix inclusivity bugs in
their game prototype. During the “find” sessions, Team J identified
inclusivity bugs within the Original prototype (Figure 1). In the “fix”
sessions, they proposed design solutions to potentially address these issues.
These sessions took place over a period of five weeks via Zoom and are
described in detail in Andrew Anderson’s forthcoming dissertation.

Figure 1. The Original prototype during a game. Columns are labeled A-I and
rows are labeled 1-4. Squares can be identified using their column name
followed by the row number (e.g., E1). The game progression is controlled by
clicking the prior move and next move buttons below the gameboard.

Team J started the find sessions by customizing the Abi persona. They then
used Abi persona during the GenderMag cognitive walkthrough to identify
inclusivity issues within their prototype. They identified issues that spanned
all five problem-solving styles outlined in Table 1. Team J engaged in find
sessions totaling about 5 hours.

In the fix sessions, Team J systematically reviewed all identified inclusivity
issues from the find sessions spending about 8 hours in total. Team J devised
15 proposed fixes during these sessions. Table 5 presents 3 of these fixes
which we later utilized to illustrate our study findings. All the 15 fixes are
listed in Table LABEL:tab:fixes_all in the appendix.

Table 5. Three fixes, why Team J fixed them (problem-solving styles impacted), and the changes made for the fixes. Fix-ID & Fix Name |  Fixed-Why + Problem Solving Styles(PSS) |  Changes Made for the Fixes
---|---|---
Fix-2: a) Exact values, b) Exact actions |  “She is not clear on what she should do…..will not take this step.” PSS: Risk, Learn, Info. |  – a) Showed exact win%, loss%, and draw% for every rectangle in the BTW, STT, and OTB explanations using tooltips. – b) Added tooltips that informed about the exact actions offered by the next and the prior move buttons.
Fix-7: Top Moves Comparison |  “…but they may not be sure how to reason with this information.” PSS: Motiv, Info. |  – Added “Top 5” moves with their score, win, loss, and draw percentages.
Fix-10: BTW Legend |  “The relationship between…board and the BTW…isn’t clear.” PSS: SE, Risk. |  – Added legend to the top right for btw. – Changed the coloring of the move lines. Darker color for more recent moves. – Made the legend interactive.
Rest of the Fixes are in Table LABEL:tab:fixes_all.

We then implemented 13 of Team J’s 15 fixes. We could not implement Fix-14 and
Fix-15 due to their technical difficulties and our time constraints. Figure 2
shows the Post-GenderMag version of Team J’s prototype.

Figure 2. The Post-GenderMag prototype during a game. Some fixes resulted in
new additions such as the Game Log, Game History, and Top 5 Moves.

We call the original version “Original” and the redesigned version “Post-
GenderMag” as its changes came from the GenderMag-driven cognitive
walkthroughs. With these two versions, we aimed to address our research
questions, which revolved around whether Team J’s fixes improved user
engagement and mental models.

###  3.3. Game Overview and Research Prototype

Team J was working in the sequential decision domain of MNK games, based on
Dodge et al.’s original prototype (Dodge et al., 2022). MNK games are an
abstraction of Tic-Tac-Toe where M and N represent the dimensions of the
gameboard ($M\times N$) and the goal is to get a sequence of length K. To
simplify the prototype presented in Dodge et al. (Dodge et al., 2022) and
enable Team J to make changes, we re-created the original prototype in a web-
based format using Svelte (JavaScript framework) seen in Figure 1. In this MNK
game, the gameboard—on the left half of the prototype—has 9 columns and 4 rows
(36 squares) where the winning condition is getting 4 in-a-row (9-4-4). The
game is played by two AI agents, Agent Blue (X) and Agent Red (O), that use a
convolutional neural network to predict outcome tuples
$O=(Win\%,Loss\%,Draw\%)$ for each square on the gameboard before making a
move. Each square is then assigned a score using the function $Win\%-Loss\%$.
The agents make moves based on the scores. To the right of the gameboard, the
prototype provides three explanations for Agent Blue’s move but not for Agent
Red. This prototype had the same explanations presented in Dodge et al. (Dodge
et al., 2022), namely Scores Best-To-Worst (BTW), Scores Through Time (STT),
and Scores On-the-Board (OTB).

####  3.3.1. Explanation Type: Scores Best-to-Worst (BTW)

The Scores Best-to-Worst (BTW) explanation presents all of Agent Blue’s
predicted scores for the 36 gameboard squares ranked from the best move to the
worst move on a graph (Figure 3). The X-axis is the order of scores, from the
best move on the left to the worst moves on the right and the Y-axis
represents the score from -100% to 100%. Agent Blue assigns very low scores
for the illegal moves (moves that are already taken). After each move by Agent
Blue, a new series of rectangles will appear in the explanation and the prior
set of scores will get a lighter color. The most recent data series will have
the darkest color and can be interacted with by hovering. Hovering over a
score in the most recent data series will highlight its corresponding square
on the gameboard and its past scores in all the explanations. Hovering over a
square on the gameboard will highlight its corresponding scores in BTW (and
also in other 2 explanations) over the past moves.

Figure 3. BTW in the Original prototype. Three data series of scores by Agent
Blue after the agent made its third move. The blue data series is the most
recent scores the agent calculated.

####  3.3.2. Explanation Type: Scores Through Time (STT)

The Scores Through Time (STT) explanation shows how the distribution of scores
changes throughout the game whenever Agent Blue makes a move (Figure 4). The
X-axis is time (decision points) and the Y-axis is the score. When Agent Blue
makes a move a new column of scores appears. The score with the maximum score
in the column is highlighted in a darker color for the current move. Hovering
over a score in the column highlights its corresponding square in the
gameboard and the score of that square in prior decisions. Like the prior
explanation BTW, only the most recent series of scores can be hovered over.
Hovering over a gameboard square will highlight its score for every decision
through time.

Figure 4. STT in the Original prototype. Agent Blue has played 3 moves so
there are 3 columns and the number under each column represents the move
number.

####  3.3.3. Explanation Type: Scores On-the-Board (OTB)

The last explanation, Scores On-the-Board (OTB), shows the scores through time
at the level of each square where each square has its own version of STT
(Figure 5). Each little dot shows Agent Blue’s evaluation of that square over
different moves or decision points. Squares that have been used for moves have
a highlighted border in the same color (blue or red) as the agent that picked
the move. Hovering over a square in the gameboard highlights the corresponding
chart in the explanation and vice versa. A new data point is added to each
square’s chart every time Agent Blue makes a move.

Figure 5. OTB in the Original prototype. The square highlighted in yellow is
the most recent move by Agent Blue which has a high score.

###  3.4. Measuring Mental Models for this Game

####  3.4.1. Mental Model Rubric

We needed a mental model rubric to calculate the mental model score of each
participant. From the 4 code concepts from our initial codeset, Team J’s lead,
who created the AI agents, selected 2 code concepts, “Flaws” and “Agent Sense”
for the mental model rubric (Table 6).

“Flaws”: The agent creator picked the following 3 codes under “Flaws” for the
rubric:
– Square X is miscalculated.
– Agent miss selection.
– Agent chose the wrong action.
The agent creator pointed out that to identify and specify flaws, participants
needed to understand the scoring system, the agent’s decision-making process,
and the gameboard. They also mentioned both local and global understanding of
the Agents’ moves were necessary for detecting flaws. For these reasons, the
agent creator picked the concept “Flaws” and the above 3 codes. They assigned
positive weights to these codes in the mental model rubric.

“Agent Sense”: The agent creator picked 7 codes from the “Agent Sense” concept
for the mental model rubric and divided them into two groups. “Agent”
mentioned in the following codes is Agent Blue and “Opponent” is Agent Red:
i) Agent Sense codes which are Correct:
– Agent picks high score.
– Agent doesn’t pick high score.
– Agent doesn’t consider opponent’s moves.
ii) Agent Sense codes which are INcorrect:
– Opponent is random/illogical.
– Agent is random/illogical.
– Agent prolongs game.
– Agent is/can learn

The agent creator considered that these 7 codes offer insights into
participants’ perceptions of the AI agents’ gameplay, behavior, priorities,
strengths, and weaknesses. According to the agent creator, these codes
contributed to participants’ comprehensive understanding of the AI agents,
justifying their integration into the mental model rubric. The agent creator
also pointed out that mentioning the correct codes from group-i indicated
participants’ better understanding of the AI agents, while INcorrect codes
from group-ii indicated a lack of understanding. That’s why they assigned
positive weights for group-i and negative weights for group-ii in the mental
model rubric.

Table 6. Codes used in the mental model rubric. Code |  Definition
---|---
Square “X” is miscalculated |  Participant says the score is wrong/agent miscalculated the score/there are prediction flaws and they specified which particular square they were referring to.
Agent miss-selection |  Participant says the agent selected the wrong square because it is not the high/highest score (not following its reasoning).
Agent chose the wrong action |  Participant says there is a lost win opportunity, missed a critical move, or missed the opportunity to block.
Agent picks high score |  Participant says the agent is picking the square with the highest score or within the top 5.
Agent doesn’t pick high score |  Participant says the agent is not picking the square with the highest score or within the top 5.
Agent doesn’t consider opponent’s moves |  Participant says the agent does not consider the opponent or consider their effect.
Opponent is random/illogical |  Participant says red is dumb, not smart, making weird moves, illogical, random.
Agent is random/illogical |  Participant says generally the agent is random, behaving illogically, making weird moves, or doesn’t know what it is doing.
Agent prolongs game |  Participant says the agent prolongs/stalls/lengthens the game (e.g., not doing finishing or winning move).
Agent is/can learn |  Participant says the agent maybe/is learning.

####  3.4.2. Assignment of Weights: Finding the Weight Ranges

Our team collaborated with the agent creator (Team J’s lead) to assign weights
for the mental model rubric. Team J’s lead finalized specific weights for
certain codes and a range of weights for others (Table 7). A code with a range
of weights indicates that any values within the range would be appropriate for
the mental model rubric, as defined by Team J’s lead. Our team also provided
feedback on possible weights when Team J’s lead made the decisions.

Table 7. Mental model rubric: proposed weights and ranges by Team J’s lead. Code | Weight Range
---|---
Square X is miscalculated | [6, 10]
Agent miss selection* | 7
Agent chose the wrong action | [11, 15]
Agent picks high score* | 2
Agent doesn’t pick high score | [6, 10]
Agent doesn’t consider opponent’s moves | [11, 15]
Opponent is random/illogical | [-2, -1]
Agent is random/illogical | [-5, -3]
Agent prolongs game | [-5, -3]
Agent is/can learn* | -15
For 3 codes with *, Team J’s lead assigned exact weights as given in the
table.

From the weight ranges assigned by Team J’s lead, we got four categories of
weight ranges (Table 9). We labeled these categories based on how Team J’s
lead intended the weights to map the importance of the code concepts. From
these, we derived Table 9’s final weights as follows.

Table 8. Weight categories. Category |  Weight Range
---|---
Low Importance |  [1, 2], [-2,-1]
Low-Medium Importance |  [3, 5], [-5, -3]
Medium Importance |  [6, 10]
High Importance |  [11, 15]

Table 9. Criteria for weight selection. Criteria |  Flaws |  Agent Sense (Correct) |  Agent Sense (INcorrect)
---|---|---|---
Criteria 1: Max within constraints |  Middle |  Max |  Max
Criteria 2: Min within constraints |  Min |  Max |  Min
Criteria 3: INcorrect-check |  Max |  Min |  Middle
Criteria 4: Flaw-check |  Min |  Max |  Max

####  3.4.3. Assignment of Weights: Finalizing the weights

First, we came up with 4 weighting strategies, described via Table 9’s 4
criteria. These 4 criteria analyzed the possible effects of the three code
concepts within the mental model rubric: i) Flaws, ii) Agent Sense (Correct),
and iii) Agent Sense (INcorrect). To check the effects of these codes, we have
considered different combinations of weights (min, middle, max). For Criteria
1 and 2, we have taken the maximum and minimum weights respectively within the
ranges. Criteria 3 varies only in INcorrect weights from Criteria 2 to check
if “INcorrect” weight makes much difference. Criteria 4: Flaw-check: varies
only Flaws’ weight from Criteria 1 to check if “Flaws” weight makes much
difference. To give an example of Min, Middle, and Max, let’s consider the
code “Square X is miscalculated” that has a weight range of [6, 10]. Within
this range, we considered its min weight as 6/7, middle weight as 8, and max
weight as 9/10.

The rationale behind the weights in the columns of Table 9 were:

For the Flaws codes, the agent creator believed it is important to detect the
flaws, and detecting the flaws highlights a participant’s understanding of the
agent’s gameplay. However, the agent creator mentioned that detecting the
flaws was not a very difficult task. That’s why we mostly scored them with low
values from their respective weight ranges.

Agent sense(correct) provides the most important aspects of the participant’s
overall understanding of the agent. That’s why the agent creator decided that
these codes should have high weights and based on the importance imposed by
the agents’ creators, we gave them the maximum possible values within their
respective weight range for all 4 criteria.

The agent creator wanted the agent sense (INcorrect) codes to account for
participants’ incorrect understanding of the AI agents. Thus, we mostly rated
them as Max within their ranges to maximize the penalty for having an
incorrect understanding of the AI agents.

Table 10 shows the final set of weights using Table 9’s weighting strategies.

Table 10. Exact code weights assigned in the 4 criteria. Code | Criteria 1 | Criteria 2 | Criteria 3 | Criteria 4
---|---|---|---|---
Square X is miscalculated | 8 | 7 | 7 | 7
Agent miss selection* | 7 | 7 | 7 | 7
Agent chose the wrong action | 13 | 11 | 11 | 11
Agent picks high score* | 2 | 2 | 2 | 2
Agent doesn’t pick high score | 10 | 10 | 10 | 10
Agent doesn’t consider opponent’s moves | 15 | 15 | 15 | 15
Opponent is random/illogical | -2 | -1 | -1 | -2
Agent is random/illogical | -5 | -3 | -4 | -5
Agent prolongs game | -5 | -3 | -4 | -5
Agent is/can learn* | -15 | -15 | -15 | -15
For 3 codes with *, Team J’s lead assigned exact weights.

Out of the four criteria, we have picked Criteria 4: Flaw-Check for our final
set of weights for the mental model rubric. Before making the final selection,
we compared the mental model scores of the participants generated using the
weights of all 4 criteria (Figure 6).

Figure 6. Mental model score comparison for the four weight selection
criteria.

In this graph (Figure 6), each participant has a position along the x-axis
(“index”). For ease of visual comparison, we ordered the participants in
descending order of mental model score as per Criteria-1. Since Participant-4
(PID-4) was the highest scorer by Criteria-1, at index[1] is PID 4’s score
(y-axis) for each of the four criteria. The second-highest scorer by
Criteria-1 was PID-57, so, the index[2] shows PID 57’s scores for each of the
four criteria. And so on.

As Figure 6 shows, all criteria generated similar participant rankings. For
example, 9/10 of the top ten participants by Criteria-1 are also in the top
ten participants of the other criteria. Likewise, 10/10 of the bottom ten are
always in the bottom ten. The lines for Criteria 2,3,4 are almost the same in
terms of their crests and troughs.

When we made the final criteria selection, we rejected Criteria-1 as its line
graph (Figure 6) was different from the other 3 criteria. Among, Criteria 2,3,
and 4, the trends of the mental model linegraphs were almost the same. We
picked Criteria 4: Flaw-Check for two reasons: i) It did not overly emphasize
a “Flaw” instance, because these were only intermittent points of constantly
evolving understanding of the AI agents after every move of the game. ii) It
had Max weights for “Agent Sense”. Agent sense responses mostly came from the
AAR/AI questions at the end of the game which is the end point of their mental
model until that point. Therefore, Agent Sense(Correct and INcorrect) codes
should have high priority and we should assign Max values within their ranges.
From here on, “mental model scores” refer to the scores calculated using
Criteria 4.

##  4\. Results

###  4.1. Explanations Mattered

We initiated our analysis by examining if the explanations had any impacts on
our participants. Team J considered the explanations as the most important
element of the prototype, as they help users understand the decision-making
process of the AI agents. So we investigated if using explanations more
frequently had any positive impacts on users’ understanding of the AI agents.
In this study, we assessed users’ understanding through their mental model
scores. So we checked if using explanations mattered to have a better mental
model score.

For every participant, we calculated their explanation usage. As outlined in
Table LABEL:tab:codes_definitions, we define the code “Participant used prior
explanation” as “Participants drew on something they’d seen in the previous
explanation(s) including the explanations from a previous game.” We calculated
the frequency of each participant’s responses falling under this code to
determine their overall explanation usage. A participant’s explanation usage
count reflects how frequently they mentioned using explanations.

We used linear regression to investigate the impacts of explanation usage on
participants’ mental model scores. We started with the following linear
regression model where we wanted to explain the mental model score of the
participants based on their explanation usage:

(1) $\displaystyle Model1:\mu($ $\displaystyle\text{Mental Model
Score}|\text{Explanation Usage})=\beta_{0}+\beta_{1}\times\text{Explanation
Usage}$

From this linear regression model, we got a multiple R-squared value of 0.202
with 67 degrees of freedom. This suggests that explanation usage could explain
about 20.2% variability in the mental model score. To improve the R-squared
value, we constructed a more complicated linear regression model by adding two
new predictor variables. One is the categorical variable “Group” (Original or
Post-GenderMag) and another is the interaction term between the group and the
explanation usage. This gives us the following linear regression model:

$\displaystyle Model2:\mu($ $\displaystyle\text{Mental Model
Score}|\text{Explanation Usage},\text{GroupOriginal})$
$\displaystyle=\beta_{0}+\beta_{1}\times\text{Explanation Usage}$
$\displaystyle\quad+\beta_{2}\times\text{GroupOriginal}$ (2)
$\displaystyle\quad+\beta_{3}\times(\text{Explanation
Usage}\times\text{GroupOriginal})$

However, compared to model 1, the explanatory power of model 2 did not improve
much. This model has a multiple R-squared value of 0.212 which means we only
had (21.21-20.2)=1.01% increase in the explanatory power from model 1.
Moreover, Model 1 had 67 degrees of freedom and for Model 2 it dropped to 65
as this model has two additional terms.

Before making our final model selection, we performed analysis of variance
(ANOVA) to compare the two models (Table 11). Specifically, we wanted to check
if adding the group and the interaction term significantly improves the
model’s explanatory power.

Table 11. Comparison between Model 1 and Model 2 using ANOVA. Model | Res.Df | RSS | Df | Sum of Sq | F | Pr(¿F)
---|---|---|---|---|---|---
1 | 67 | 135554 |  |  |  |
2 | 65 | 133838 | 2 | 1716.5 | 0.4168 | 0.6609

The ANOVA results (Table 11) show no significant differences between the two
models. The residual sum of squares (RSS) for model 1 is 135554 and for model
2 it was slightly lower with a value of 133838. The F statistic of 0.4168 gave
a p-value of 0.6609 which is much higher than the significance level of 0.05.
This lack of significance suggests that the simpler Model 1 performs just as
well as the more complex Model 2. Considering all these, we decided to go with
model 1 for further analysis of our study.

With the finalized linear regression model, we investigated if the explanation
usage had any impacts on participants’ mental model scores and found
significant results (Table 12).

Table 12. Model 1 results summary. Variable | Estimate | Std. Error | t value | Pr(¿—t—)
---|---|---|---|---
(Intercept) | 69.781 | 13.174 | 5.297 | 1.41e-06
Explanation Usage | 2.899 | 0.704 | 4.118 | 0.000107
Residual standard error: 44.98 on 67 degrees of freedom(DF)
Multiple R-squared: 0.202, Adjusted R-squared: 0.19
F-statistic: 16.96 on 1 and 67 DF, and p-value: 0.0001073

The results show that the regression model was statistically significant,
overall ($R^{2}=0.202,F(1,67)=16.96,p=0.0001073$). Also, the explanation usage
significantly predicted mental model scores ($coefficient=2.899,p=0.000107$).

In essence, using explanations really mattered in shaping participants’ mental
models.

###  4.2. Who Used Explanations More? Original or Post-GenderMag Participants.

After observing the significant impacts of explanation usage on participants’
mental models, we were curious to know if there was any significant difference
between the Original and the Post-GenderMag participants’ explanation usage,
as measured by the number of times each participant mentioned explanations. To
find out, we ran a two-sided Student’s t-test with the following hypotheses:

  * •

Null Hypothesis (H0): The mean explanation usage is equal for the Original and
the Post-GenderMag participants.

  * •

Alternative Hypothesis (H1): The mean explanation usage is not equal for the
Original and the Post-GenderMag participants.

The results revealed marginal significance between the Post-GenderMag
($mean=18.8$) and the Original ($mean=15.26$) participants’ explanation usage,
$t(67)=1.9327,p=.0575$. Figure 7 shows the boxplots of the explanation usage.

Figure 7. Box Plots of the explanation usage counts. Post-GenderMag
participants were more engaged with explanations than the Original
participants.

The difference between the mean values indicates that the Post-GenderMag group
was on average $\sim 23\%$ more engaged with the explanations than the
Original group. If so, the increased engagement may be tied to Team J’s fixes,
where 9 out of 15 fixes targeted the explanations.

One explanation fix, Fix-10: “BTW Legend” is in Figure 8. In Fix-10 from Table
5, Team J focused on resolving the difficulty in distinguishing between
multiple data series222Every time Agent Blue makes a move, it is considered a
decision point. For every new decision point, a data series containing the
scores of all 36 squares on the game board during that decision point is
generated in BTW.in the BTW explanation. To fix this, Team J decided to add a
legend alongside the BTW explanation. While making this specific adjustment,
they took into account two problem-solving styles of Abi: Computer Self-
Efficacy and Attitude Towards Risk. During the find session, Team J identified
a user with Abi’s low computer self-efficacy and risk-averse attitude might
struggle to differentiate which data series came from which decision point.
Team J expected that an interactive legend would give users an option to
hover, highlighting the corresponding data series in BTW to identify which
data series came from which decision point.

Figure 8. (Left) Original version: Team J found it hard for Abi to
differentiate between the data series. (Right) Post-GenderMag version: Team J
added Fix-10, a legend with BTW. The call-out shows hovering over move number
7 on the legend highlighted the data series generated in the 7th move in
yellow.

Team J’s expectation was later verified in our study. P66-OR333P66-OR: 66th
participant of our study who was assigned to the Original group., a
participant from the Original group, explicitly stated the need to be able to
better follow the moves in BTW:

> P66-OR: ¡When giving feedback on the BTW explanation¿ Sometimes the lines
> overlapped and I couldn’t follow the move that I was on when it was
> highlighted.

Another Original participant P59-OR, mentioned how they got confused when
there were multiple data series in BTW:

> P59-OR: Past the first/second move the chart is very difficult to interpret.

Section 4.1, revealed that explanation mattered. Now, the current section’s
results suggest that the Post-GenderMag participants engaged more with the
explanations.

###  4.3. Who has a Better Mental Model Score? Original or Post-GenderMag
participants

Next, we wanted to know, what were the impacts of more explanation usage on
the Post-GenderMag participants. To find out, we compared the mental model
scores of the two groups using t-test.

####  4.3.1. Post-GenederMag participants’ mental model score is significantly
greater than the other

We had to choose between a one-tailed vs a two-tailed t-test to compare the
mental model scores. Two-tailed tests are more common in statistical analyses
as they can detect both positive and negative effects whereas one-tailed
t-tests are more suitable when the effects can occur only in one direction
(Frost, 2021). Considering these attributes of the two tests, we turned to the
literature related to our study to investigate whether a one-tailed or two-
tailed approach was more appropriate. The literature unanimously reports
GenderMag improving the user experience of people with diverse problem-solving
styles. For instance, Guizani et al.’s research demonstrated a 90% reduction
in software-related inclusivity issues following GenderMag fixes (Guizani et
al., 2022). In another study, Vorvoreanu et al. observed substantial
improvement in users’ task performance resulting from GenderMag-guided design
changes (Vorvoreanu et al., 2019). Furthermore, Burnett et al.’s field study
found a 40% increase in customer ratings when a team of developers used
GenderMag to address gender biases (Burnett et al., 2017). Lastly, Hilderbrand
et al. reported an improvement (decline) in help desk questions after a team
of web developers integrated GenderMag into their existing practices
(Hilderbrand et al., 2020). Other members of the cognitive walkthrough family
also show cognitive walkthroughs improving user experience. For example,
Blackmon et al. used Cognitive Walkthrough for the Web (CWW) to repair
usability issues of three web pages and found that all three repaired pages
were significantly better than the unrepaired pages ($p<0.0001$) (Blackmon et
al., 2003). In another study, Green et al. used cognitive walkthroughs to
identify and fix issues within an empirical study, resulting in a more
effective study that produced much cleaner results (Green et al., 2000). All
of these studies report positive improvements from using GenderMag-driven and
other cognitive walkthroughs. We could not find no studies that reported
negative impacts for these methods. Thus, we chose one-tailed greater t-test
with the following hypotheses:

  * •

Null Hypothesis (H0): The mean mental model score of the Post-GenderMag
participants is less than or equal to the mean score of the Original
participants.

  * •

Alternative Hypothesis (H1): The mean mental model score of the Post-GenderMag
participants is greater than the mean score of the Original participants.

To investigate the hypotheses, we used Student’s one-tailed t-test on
participants’ mental model scores described in Section 3.4.3. The test results
showed that Post-GenderMag participants ($mean=129.06$) compared to the
Original participants ($mean=109.12$) demonstrated significantly better mental
model scores, $t(67)=1.6788,p=0.04892$. Figure 9 shows the boxplots of the
mental model scores.

Figure 9. Box Plots of the mental model score. Post-GenderMag participants had
significantly better mental model scores than the Original participants.

####  4.3.2. Why did the Post-GenderMag group have a better mental model than
the Original group?

The post-GenderMag group had a significantly better mental model than the
Original group, but why? What factors caused the two groups to have different
understandings of the AI agents?

Findings from Sections 4.1 and 4.2 point to one factor, “Explanation Usage”.
Section 4.1 revealed that participants with higher explanation usage had
significantly higher mental model scores. Further, section 4.2, reported the
Post-GenderMag group had marginally higher explanation usage than the Original
group. These findings suggest that the Post-GenderMag participants had better
mental models because they were more engaged with the explanations than the
Original participants.

But what did the explanations reveal that could explain why Post-GenderMag
participants had a better mental model? To find out, we considered group
differences for each code from the mental model codeset (Table 6).

The results (Figure 10) reveal three codes in which the Post-GenderMag and the
Original groups differ by nearly double or more participants: “Agent picks
high score”, “Agent doesn’t consider opponent’s moves,” and “Agent is
random/illogical”.

Figure 10. Percentage of Original(n=34) and Post-GenderMag(n=35) participants
for each code from the mental model codeset. Percentage indicates the
percentage of participants whose responses had at least one instance of a
statement falling under that code. Positive codes increase the mental model
score, while negative codes decrease it. Differences in % were nearly double
or more (*).

Since the only difference between the Original and the Post-GenderMag versions
of the prototype lay in the fixes, we next investigate how the fixes related
to these 3 codes.

####  4.3.3. GenderMag fixes are tied to the differences within mental model
codes.

Two of Team J’s fixes— Fix-2a: “Exact Values” and Fix-7: “Top Moves
Comparison” seem directly implicated.

In Fix-2a (Figure 5), Team J addressed the issue of incomplete score
information in the explanations. In the Original version (Figure 5: Top),
participants could only estimate the score% (win%-loss%) of the rectangles
directly from the graph. To fix this, Team J added the option to see the exact
values with each rectangle of the explanations (Figure 5: Bottom). Upon
hovering over a rectangle, the participants could see what are the exact win%,
loss%, and draw% if Agent Blue picked the square represented by this
rectangle. The fact that the Post-GenderMag participants were twice as likely
to notice “Agent picks high score” suggests that showing them the exact values
was indeed helpful.

Figure 11. Top: (a) Original version: Hovering over a rectangle from the
latest data series highlights the rectangles representing the same gameboard
square in every data series555Every time agent blue makes a move, the BTW
explanation generates a data series containing 36 small rectangles, 1
rectangle for each gameboard square. Where the rectangle lies vertically tells
how good the agent thinks moving that square would be. Multiple data series
show this info for different decision points.from the previous decision
points. However, it does not show the exact scores and which rectangles came
from which data series. Bottom: (b) Post-GenderMag version: Hovering over a
rectangle from the latest data series now explicitly shows the win%, loss%,
and draw% of the gameboard square represented by the rectangle.

Showing the exact values might have also influenced why more than twice Post-
GenderMag participants compared to the Original participants mentioned “Agent
doesn’t consider opponent’s moves”, which is a true observation of Agent Blue.
Seeing the exact probabilities of winning, losing, or drawing a game for each
potential move could have helped Post-GenderMag participants understand that
the agent emphasizes its own calculations rather than considering what the
opponent agent is doing.

Fix-2a’s emphasis on the agent’s own calculations also may have helped most
Post-GenderMag participants avoid an error that Original participants made:
thinking that the “Agent is random/illogical”, which is not true. 38% of the
Original participants considered Agent Blue to be random/illogical, whereas
only 14% of the Post-GenderMag participants held that assumption.

Turning to Fix-7 (Figure 12), Team J addressed the absence of a feature that
could compare the top-ranked squares and explain their ratings. To fix this,
Team J added the score%, win%, loss%, and draw% of the “Top 5” ranked squares
from Agent Blue’s most recent move. With this feature, the Post-GenderMag
group had the option to specifically examine why Agent Blue considered some
moves to be better than others. They could see the exact score% which is one
of the major factors behind Agent Blue’s decision-making process. As Agent
Blue truly followed its scores and ranking for most of the moves, seeing the
exact scores and comparing the top-ranked moves could be the reasons why more
Post-GenderMag participants said that “Agent picks high score” and “Agent
doesn’t consider opponent’s moves”. The same reasons could explain why knowing
more information prevented more Post-GenderMag participants from thinking that
the “Agent is random/illogical”.

Figure 12. Post-GenderMag version showed the top 5 moves with their score%,
win%, loss%, and draw%.

Team J recognized the need to add these 2 fixes because of Abi’s two problem-
solving styles: 1) Motivations and 2) Information Processing Style. In the
find session, Team J identified that Abi’s low motivation to use unfamiliar
technology and comprehensive information processing style would create the
following scenario when Abi would see the explanations for the first time:

> Abi is not sure what this new information in the explanation means. Certain
> squares are higher rated and some are low, Abi may not be sure why there is
> this difference. Abi may not be sure how to reason with this info.:- Team
> J’s critiques on Abi’s first encounter with the explanations.

Team J believed using tooltips for showing the exact values would add
familiarity as Abi prefers to use familiar features. Team J also believed
providing additional information about the moves would benefit Abi, as she
comprehensively gathers information before attempting to solve a problem.

Some of our Original participants later verified Team J’s concerns. Two
examples:

> P61-OR: I wish BTW included a feature where I could see what the win
> percentage was and what the lose percentage was. So I can understand how
> they got the values.

> P31-OR: Need to see the actual scores!

Post-GenderMag participants’ responses reflected how they benefitted from
Fix-2a: “Exact Values” and Fix-7: “Top Moves Comparison”. The following
responses show how the Post-GenderMag participants used the additional
information provided by the 2 GenderMag fixes to understand that Agent Blue
picks high-scoring squares:

> P68-PostGM: No Flaw. Chose A2 because of the win% and loss% ¡Percentage
> values are only available in Post-GenderMag version¿. The AI is following
> the best move for the entire game.

> P7-PostGM: The “scores best-to-worst” and the “top 5 moves” ¡Only Post-
> GenderMag version explicitly showed the top 5 moves¿ help explain it. It
> shows that it’s picking squares based on their scores.

> P39-PostGM: Yes - it is clear that X was trying to place moves based on top
> score (Top5 Moves) ¡Only Post-GenderMag version explicitly showed the top 5
> moves¿. It worked in the end.

The above evidence suggests that GenderMag fixes brought positive changes to
Team J’s XAI prototype. This is evident from Post-GenderMag participants’
significantly better mental models (Section 4.3.1). Further, participant
percentages within the mental model codes showed that Original and Post-
GenderMag groups had nearly double or more differences for multiple codes
(Section 4.3.2). Lastly, the tie between the GenderMag fixes and the mental
model codes showed why Team J’s decision to consider Abi’s problem-solving
styles led to their fixes (Section 4.3.3).

##  5\. Future Work

In this ongoing research, our next step involves analyzing the GenderMag
survey responses that we gathered from both groups of participants. Findings
from the previous section suggested the positive impact of GenderMag fixes on
user experiences that led to better mental models for the Post-GenderMag
participants. Now, we plan to closely examine participants’ problem-solving
styles. In the GenderMag survey, participants responded what are their
problem-solving styles in different scenarios. We plan to analyze their
reported problem-solving styles and the usability issues they mentioned during
the user study.

During the GenderMag-driven cognitive walkthroughs, Team J adopted the Abi
persona, recognizing the support needed for Abi-like users’ problem-solving
styles. Our aim is to investigate whether the Original and the Post-GenderMag
versions posed challenges for Abi-like users, what specific issues they
encountered, and the differences between the two versions in terms of the
usability issues. Subsequently, we want to check if Team J’s GenderMag fixes
worked for our Abi-like Post-GenderMag participants. Finally, we aim to know
that team J’s GenderMag fixes did not inadvertently cause any usability issues
for users with Tim’s problem-solving styles.

##  6\. Threats to validity

Every study encounters potential challenges to its validity (Wohlin et al.,
2000), and ours is no exception. We acknowledge several factors that could
limit the generalizability and reliability of our findings.

The first issue is the demographic homogeneity. Although we had 69
participants, most of them were students and came from the same university.
This may raise concerns about the broader application of our results. Our
participants may not properly reflect the diversity present among potential
users of MNK or XAI prototypes.

Moreover, the MNK domain itself presents a potential limitation as this domain
is not common in XAI and HAI research. The MNK games Team J used might not
sufficiently replicate real-world scenarios. This could potentially compromise
participants’ ability to accurately reason or understand the agents’ behavior.
We decided to work with an MNK prototype as people are more familiar with Tic-
Tac-Toe which allowed for straightforward judgment on agents’ decisions. Also,
MNK gave us the option to ask a wide range of questions related to
predictions, flaws, changes, and after-action reviews.

Another concern is our participants’ passive involvement. Our study
participants only observed AI agents in gameplay without any control over the
agents’ decision-making. While this reduced cognitive load and allowed more
focus on explanations, it could have limited us from creating immersive, high-
stakes scenarios. Additionally, this observational role might have limited
participants’ interactions with the prototype.

We did not gather click log data for explanation interactions. Instead, we
relied on participants’ mentions of using the explanations as an engagement
indicator. Maintaining a click log could have offered more dependable insights
on explanation usage. While we have screen recordings from each session,
determining the time spent specifically on explanations would be difficult and
time-consuming.

Addressing such threats would require additional studies using diverse
empirical methods. Those methods have to make sure the findings apply widely
to various XAI systems and participant demographics.

##  7\. Conclusion

This paper investigated if inclusivity fixes can improve users’ mental models
of the AI system. Our setting was an XAI prototype where participants observed
two AI agents sequentially play a modified version of Tic-Tac-Toe. We report
the results from our between-subject study where the “Original” group used the
original version of the prototype and the “Post-GenderMag” group used the
version with the inclusivity fixes:

  * •

RQ1: Participants’ engagement with the explanations mattered in shaping their
mental models for both groups (Section 4.1).

  * •

RQ2: Participants who used the Post-GenderMag version of the prototype were on
average $\sim 23\%$ more engaged with the explanations (Section 4.2).

  * •

RQ3: Participants who used the Post-GenderMag version of the prototype
demonstrated significantly better mental models than the Original version
participants (Section 4.3).

These positive outcomes indicate the impacts of inclusivity fixes in enhancing
user engagement and mental models within AI systems. Our study highlights the
potential of diverse users’ problem-solving style-centric inclusive design
approaches to create more accessible and user-centric XAI prototypes,
benefiting diverse users’ understanding of AI.

> P37-PostGM: The BTW ¡Best to Worst explanation¿ and top 5 moves score
> ¡available in the Post-GenderMag version only¿ helped me gain a better
> understanding of why the AI did the things it did.

## References

  * (1)
  * Amershi et al. (2014) Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014.  Power to the People: The Role of Humans in Interactive Machine Learning.  _AI Magazine_ 35, 4 (Dec. 2014), 105–120.  https://doi.org/10.1609/aimag.v35i4.2513
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Anderson et al. (2019) Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fern, and Margaret Burnett. 2019.  Explaining Reinforcement Learning to Mere Mortals: An Empirical Study. In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ (Macao, China) _(IJCAI’19)_. AAAI Press, 1328–1334.
  * Anderson et al. (2020) Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Matthew Olson, Alan Fern, and Margaret Burnett. 2020.  Mental models of mere mortals with explanations of reinforcement learning.  _ACM Transactions on Interactive Intelligent Systems (TiiS)_ 10, 2 (2020), 1–37.
  * Blackmon et al. (2003) Marilyn Hughes Blackmon, Muneo Kitajima, and Peter G. Polson. 2003.  Repairing Usability Problems Identified by the Cognitive Walkthrough for the Web. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Ft. Lauderdale, Florida, USA) _(CHI ’03)_. Association for Computing Machinery, New York, NY, USA, 497–504.  https://doi.org/10.1145/642611.642698
  * Blackwell (2016) A. G. Blackwell. 2016.  The Curb-Cut Effect.  _Stanford Social Innovation Review_ 15, 1 (2016), 28–33.  https://doi.org/10.48558/YVMS-CC96
  * Burnett et al. (2017) Margaret Burnett, Robin Counts, Ronette Lawrence, and Hannah Hanson. 2017.  Gender HCl and microsoft: Highlights from a longitudinal study. In _2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. 139–143.  https://doi.org/10.1109/VLHCC.2017.8103461
  * Burnett et al. (2016) Margaret Burnett, Simone Stumpf, Jamie Macbeth, Stephann Makri, Laura Beckwith, Irwin Kwan, Anicia Peters, and William Jernigan. 2016.  GenderMag: A Method for Evaluating Software’s Gender Inclusiveness.  _Interacting with Computers_ forthcoming (01 2016).  https://doi.org/10.1093/iwc/iwv046
  * Djulbegovic et al. (2014) Benjamin Djulbegovic, Jason W. Beckstead, Shira Elqayam, Tea Reljic, Iztok Hozo, Ambuj Kumar, Janis Cannon-Bowers, Stephanie Taylor, Athanasios Tsalatsanis, Brandon Turner, et al. 2014\.  Evaluation of Physicians’ Cognitive Styles.  _Medical Decision Making_ 34, 5 (2014), 627–637.
  * Dodge et al. (2022) Jonathan Dodge, Andrew A. Anderson, Matthew Olson, Rupika Dikkala, and Margaret Burnett. 2022.  How Do People Rank Multiple Mutant Agents?. In _27th International Conference on Intelligent User Interfaces_ (Helsinki, Finland) _(IUI ’22)_. Association for Computing Machinery, New York, NY, USA, 191–211.  https://doi.org/10.1145/3490099.3511115
  * Dodge et al. (2019) Jonathan Dodge, Q. Vera Liao, Yunfeng Zhang, Rachel K. E. Bellamy, and Casey Dugan. 2019.  Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_ (Marina del Ray, California) _(IUI ’19)_. Association for Computing Machinery, New York, NY, USA, 275–285.  https://doi.org/10.1145/3301275.3302310
  * Felmingham et al. (2021) Christopher M. Felmingham, Nikki R. Adler, Zhidong Ge, Rachael L. Morton, Monika Janda, and Victoria J. Mar. 2021.  The Importance of Incorporating Human Factors in the Design and Implementation of Artificial Intelligence for Skin Cancer Diagnosis in the Real World.  _American Journal of Clinical Dermatology_ 22, 2 (2021), 233–242.  https://doi.org/10.1007/s40257-020-00574-4
  * Frost (2021) Jim Frost. 2021.  _When Can I Use One-Tailed Hypothesis Tests?_ https://statisticsbyjim.com/hypothesis-testing/use-one-tailed-tests/ Accessed on November 21, 2023.
  * Giboney et al. (2012) Justin Giboney, Susan Brown, and Jay Jr. 2012.  User Acceptance of Knowledge-Based System Recommendations: Explanations, Arguments, and Fit.  _Decision Support Systems_ 72 (01 2012), 3719–3727.  https://doi.org/10.1016/j.dss.2015.02.005
  * Goldberg (1990) Lewis R. Goldberg. 1990.  An alternative ”description of personality”: the big-five factor structure.  _Journal of personality and social psychology_ 59 6 (1990), 1216–29.  https://api.semanticscholar.org/CorpusID:9034636
  * Green et al. (2000) T.R.G. Green, M.M. Burnett, A.J. Ko, K.J. Rothermel, C.R. Cook, and J. Schonfeld. 2000.  Using the cognitive walkthrough to improve the design of a visual programming experiment. In _Proceeding 2000 IEEE International Symposium on Visual Languages_. 172–179.  https://doi.org/10.1109/VL.2000.874381
  * Guizani et al. (2022) Mariam Guizani, Igor Steinmacher, Jillian Emard, Abrar Fallatah, Margaret Burnett, and Anita Sarma. 2022.  How to Debug Inclusivity Bugs? A Debugging Process with Information Architecture. In _2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS)_. 90–101.  https://doi.org/10.1145/3510458.3513009
  * Hamid et al. (2024) Md Montaser Hamid, Amreeta Chatterjee, Mariam Guizani, Andrew Anderson, Fatima Moussaoui, Sarah Yang, Isaac Escobar, Anita Sarma, and Margaret Burnett. 2024.  _Equity, Diversity, and Inclusion in Software Engineering: Best Practices and Insights_.  Apress, Chapter : How to Measure Diversity Actionably in Technology.
  * Herlocker et al. (2000) Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. 2000.  Explaining Collaborative Filtering Recommendations. In _Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work_ (Philadelphia, Pennsylvania, USA) _(CSCW ’00)_. Association for Computing Machinery, New York, NY, USA, 241–250.  https://doi.org/10.1145/358916.358995
  * Hilderbrand et al. (2020) Claudia Hilderbrand, Christopher Perdriau, Lara Letaw, Jillian Emard, Zoe Steine-Hanson, Margaret Burnett, and Anita Sarma. 2020.  Engineering Gender-Inclusivity into Software: Ten Teams’ Tales from the Trenches. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. Association for Computing Machinery, New York, NY, USA, 433–444.  https://doi.org/10.1145/3377811.3380371
  * Hoffman et al. (2019) Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. 2019.  Metrics for Explainable AI: Challenges and Prospects.  arXiv:1812.04608
  * Jiang et al. (2000) J.J Jiang, Gary Klein, and Richard Vedder. 2000.  Persuasive expert systems: The influence of confidence and discrepancy.  _Computers in Human Behavior_ 16 (03 2000), 99–109.  https://doi.org/10.1016/S0747-5632(00)00014-5
  * Kapoor et al. (2010) Ashish Kapoor, Bongshin Lee, Desney Tan, and Eric Horvitz. 2010.  Interactive Optimization for Steering Machine Classification. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA, 1343–1352.  https://doi.org/10.1145/1753326.1753529
  * Kulesza et al. (2015) Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015.  Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In _Proceedings of the 20th International Conference on Intelligent User Interfaces_. Association for Computing Machinery, New York, NY, USA, 126–137.  https://doi.org/10.1145/2678025.2701399
  * Kulesza et al. (2012) Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012.  Tell Me More? The Effects of Mental Model Soundness on Personalizing an Intelligent Agent. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Austin, Texas, USA) _(CHI ’12)_. Association for Computing Machinery, New York, NY, USA, 1–10.  https://doi.org/10.1145/2207676.2207678
  * Kulesza et al. (2010) Todd Kulesza, Simone Stumpf, Margaret Burnett, Weng-Keen Wong, Yann Riche, Travis Moore, Ian Oberst, Amber Shinsel, and Kevin McIntosh. 2010.  Explanatory Debugging: Supporting End-User Debugging of Machine-Learned Programs. In _2010 IEEE Symposium on Visual Languages and Human-Centric Computing_. 41–48.  https://doi.org/10.1109/VLHCC.2010.15
  * Lim et al. (2009) Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009.  Why and Why Not Explanations Improve the Intelligibility of Context-Aware Intelligent Systems. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Boston, MA, USA) _(CHI ’09)_. Association for Computing Machinery, New York, NY, USA, 2119–2128.  https://doi.org/10.1145/1518701.1519023
  * Madumal et al. (2020) Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. 2020.  Explainable Reinforcement Learning through a Causal Lens.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34, 03 (Apr. 2020), 2493–2500.  https://doi.org/10.1609/aaai.v34i03.5631
  * Metoyer et al. (2010) Ronald Metoyer, Simone Stumpf, Christoph Neumann, Jonathan Dodge, Jill Cao, and Aaron Schnabel. 2010.  Explaining How to Play Real-Time Strategy Games.  _Know.-Based Syst._ 23, 4 (may 2010), 295–301.  https://doi.org/10.1016/j.knosys.2009.11.006
  * Millecamp et al. (2020) Martijn Millecamp, Robin Haveneers, and Katrien Verbert. 2020.  Cogito Ergo Quid? The Effect of Cognitive Style in a Transparent Mobile Music Recommender System. In _Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization_ (Genoa, Italy) _(UMAP ’20)_. Association for Computing Machinery, New York, NY, USA, 323–327.  https://doi.org/10.1145/3340631.3394871
  * NASA ([n. d.]) NASA. [n. d.].  _NASA TLX: Task Load Index_.  Retrieved December 4, 2023 from https://humansystems.arc.nasa.gov/groups/TLX/
  * Norman and Gentner (1983) Donald Norman and Dedra Gentner. 1983.  _Mental Models_.  Lawrence Erlbaum Associates, Hillsdale, NJ.
  * Ozmen Garibay et al. (2023) Ozlem Ozmen Garibay, Brent Winslow, Salvatore Andolina, Margherita Antona, Anja Bodenschatz, Constantinos Coursaris, Gregory Falco, Stephen M. Fiore, Ivan Garibay, Keri Grieman, John C. Havens, Marina Jirotka, Hernisa Kacorri, Waldemar Karwowski, Joe Kider, Joseph Konstan, Sean Koon, Monica Lopez-Gonzalez, Iliana Maifeld-Carucci, Sean McGregor, Gavriel Salvendy, Ben Shneiderman, Constantine Stephanidis, Christina Strobel, Carolyn Ten Holter, and Wei Xu. 2023.  Six Human-Centered Artificial Intelligence Grand Challenges.  _International Journal of Human–Computer Interaction_ 39, 3 (2023), 391–437.  https://doi.org/10.1080/10447318.2022.2153320
  * Penney et al. (2018) Sean Penney, Jonathan Dodge, Claudia Hilderbrand, Andrew Anderson, Logan Simpson, and Margaret Burnett. 2018.  Toward Foraging for Understanding of StarCraft Agents: An Empirical Study. In _23rd International Conference on Intelligent User Interfaces_ (Tokyo, Japan) _(IUI ’18)_. Association for Computing Machinery, New York, NY, USA, 225–237.  https://doi.org/10.1145/3172944.3172946
  * Rader et al. (2018) Emilee Rader, Kelley Cotter, and Janghee Cho. 2018.  Explanations as Mechanisms for Supporting Algorithmic Transparency. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3173574.3173677
  * Shneiderman (2020) Ben Shneiderman. 2020.  Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy.  _International Journal of Human–Computer Interaction_ 36, 6 (2020), 495–504.  https://doi.org/10.1080/10447318.2020.1741118
  * Shneiderman (2021) Ben Shneiderman. 2021.  Responsible AI: Bridging from Ethics to Practice.  _Commun. ACM_ 64, 8 (jul 2021), 32–35.  https://doi.org/10.1145/3445973
  * Shneiderman (2022) B. Shneiderman. 2022.  _Human-Centered AI_.  OUP Oxford.  https://books.google.com/books?id=mSRXEAAAQBAJ
  * Silver et al. (2016) David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ 529 (01 2016), 484–489.  https://doi.org/10.1038/nature16961
  * Silver et al. (2018) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2018.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.  _Science_ 362 (12 2018), 1140–1144.  https://doi.org/10.1126/science.aar6404
  * Stemler (2004) Steven Stemler. 2004.  A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability.  _Practical Assessment, Research, and Evaluation_ 9 (01 2004), 1–19.
  * Vinyals et al. (2019a) Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojtek Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. 2019a.  AlphaStar: Mastering the Real-Time Strategy Game StarCraft II.  https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/.
  * Vinyals et al. (2019b) Oriol Vinyals, Igor Babuschkin, Wojciech Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, and David Silver. 2019b.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ 575 (11 2019).  https://doi.org/10.1038/s41586-019-1724-z
  * Vorvoreanu et al. (2019) Mihaela Vorvoreanu, Lingyi Zhang, Yun-Han Huang, Claudia Hilderbrand, Zoe Steine-Hanson, and Margaret Burnett. 2019.  From Gender Biases to Gender-Inclusive Design: An Empirical Investigation. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3290605.3300283
  * Weld and Bansal (2019) Daniel S. Weld and Gagan Bansal. 2019.  The Challenge of Crafting Intelligible Intelligence.  _Commun. ACM_ 62, 6 (may 2019), 70–79.  https://doi.org/10.1145/3282486
  * Wharton et al. (1994) Cathleen Wharton, John Rieman, Clayton H. Lewis, and Peter G. Polson. 1994.  The cognitive walkthrough method: a practitioner’s guide.  https://api.semanticscholar.org/CorpusID:53925427
  * Wohlin et al. (2000) Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Bjöorn Regnell, and Anders Wesslén. 2000.  _Experimentation in Software Engineering: An Introduction_.  Kluwer Academic Publishers, USA.
  * Wortham et al. (2017) Robert H Wortham, Andreas Theodorou, and Joanna J Bryson. 2017.  Improving robot transparency: real-time visualisation of robot AI substantially improves understanding in naive observers. In _2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)_ _(IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN))_. IEEE, USA United States.  https://doi.org/10.1109/ROMAN.2017.8172491

## Appendix

Table 13. Codes and definitions. Code |  Definition
---|---
Prediction Why |
Participant used current game state |  Participant drew on aspects of the current game state, such as gameboard, move placement, sequence orientation, win/loss position/state (condition of gameboard).
Participant used prior actions |  Participants drew on something they’d seen the agent do (games/moves).
Participant used prior explanation |  Participants drew on something they’d seen in the previous explanation(s) including the explanations from a previous game.
Participant predicted next explanation |  Participants hypothesized what the explanation would look like after the agent took the action they predicted.
Participant said agent was random or baseless |  Participant stated something about the agent that seemed random/wasn’t made for a “good” reason.
Participant stated what they’d do |  Participants placed themselves as a pilot, directing what they want to attack.
Participant is randomly guessing |  Participant gives a random guess, particularly for an empty gameboard or early game.
Flaws |
Yes failure |  Participant detected an issue, manifested an incorrect output i.e., wrong move, value too high/low, missing opportunity, missing an opportunity to prevent a loss.
Yes fault |  Participant identified why the agent performed poorly or something wrong about how the agent processed its input, or agent is focusing too much on its own moves.
General miscalculation |  Participant says or perceives the score is wrong/agent miscalculated the score/there are prediction flaws but they did not specify which particular square they were referring to.
Square “X” is miscalculated |  Participant says the score is wrong/agent miscalculated the score/there are prediction flaws and they specified which particular square they were referring to.
Agent miss-selection |  Participant says the agent selected the wrong square because it is not the high/highest score (not following its reasoning).
Agent chose the wrong action |  Participant says there is a lost win opportunity, missed a critical move, or missed the opportunity to block.
General flaw |  Participant only mentioned that there is a flaw but did not specify why it’s a flaw.
Strategy Should be |  Participant says some general idea about how the agent SHOULD play the game or suggestions that the agent should follow.
Agent should pick high score |  Participant says the agent should use the highest / high score or top 5.
Agent should do location-based Strategy |  Participant says the agent should pick based on the gameboard, past moves, orientation, or should continue the sequence to 4 in a row, prioritize established sequence unless under threat, take winning move, block opponent’s move, follow offensive or defensive strategy.
Agent should create multiple options |  Participant says the agent should create multiple options, e.g., 2 2 X in a row.
Agent should follow other strategy |  Participant says the agent should follow another strategy such as changing how it calculates, reduce the number of moves, and/or consider more options/data (e.g., consider opponent’s moves).
Agent Sense |  Participant’s perception of the agent’s behavior, such as how the agent plays the game, how it behaves during the game, what its priority, strengths and weaknesses are.
Agent picks high score |  Participant says the agent is picking the square with the highest score or within the top 5.
Agent doesn’t pick high score |  Participant says the agent is not picking the square with the highest score or within the top 5.
Agent does location-based Strategy |  Participant says the agent picks based on the gameboard, past moves, orientation, or will continue the sequence to 4 in a row, prioritize established sequence unless under threat, take winning move, block opponent’s move, follow offensive or defensive strategy.
Agent does not do location-based Strategy |  Participant says the agent does not pick based on the gameboard, past moves, orientation, continue the sequence to 4 in a row, prioritize established sequence unless under threat, take winning move, block opponent’s move, follow offensive or defensive strategy.
Agent doesn’t consider opponent’s moves |  Participant says the agent does not consider the opponent or consider their effect.
Agent does make multiple options |  Participant says the agent makes multiple options, e.g., 2 2 X in a row.
Agent thinks ahead |  Participant says explicitly that the agent can think ahead.
Agent doesn’t think ahead |  Participant says that the agent does not think ahead.
Opponent is random/illogical |  Participant says red is dumb, not smart, making weird moves, illogical, random.
Agent is random/illogical |  Participant says generally the agent is random, behaving illogically, making weird moves, or doesn’t know what it is doing.
Agent prolongs game |  Participant says the agent prolongs/stalls/lengthens the game (e.g., not doing finishing or winning move).
Agent is/can learn |  Participant says the agent maybe/is learning.
Participant has uncertainty/confusion |  Participant said they are unsure/confused about something with respect to how the agent works (possibly speculatively).
Participant has a question |  Participant explicitly asks a question about the agent, game, or explanations and may want more information.
Game Sense |  Participants’ perception of the game itself (not about the agent).
Starting first has advantages |  Participant says that going first has advantages or going second has fewer advantages towards winning the game.
Start doesn’t matter |  Participant says that starting moves do not matter.
Score should be symmetrical |  Participant says scores should be symmetrical with respect to the gameboard.
Table 14. All of Team J’s Fixes, why they fixed them (problem-solving styles impacted), and the changes made for the fixes. Fix-ID & Fix Name |  Fixed-Why + Problem Solving Styles(PSS) |  Changes Made for the Fixes
---|---|---
Fix-1: Blurb of Instructions |  ”Attitude toward technology is by process,…look for documentation..” PSS: Info, Risk. |  – Added instructions on getting started. – Added summary of each explanation. – Added minimize button for instructions.
Fix-2: a) Exact values, b) Exact actions |  ”She is not clear on what she should do…..will not take this step.” PSS: Risk, Learn, Info. |  – a) Showed exact win%, loss%, and draw% for every rectangle in the BTW, STT, and OTB explanations using tooltips. – b) Added tooltips that informed about the exact actions offered by the next and the prior move buttons.
Fix-3: Buttons Design and Colors |  ”The game info…gives the hint….needs further indications.” PSS: Motiv, Info. |  – Made buttons look more like buttons. – Increased color contrast between the button and the instructions background. – Used undo icon with the prior move button.
Fix-4: Info Button |  “Not sure if Abi knows….what type of info it would provide.” PSS: SE, Learn. |  – Added circular info button with each explanation that gives a short summary of the explanation.
Fix-5: Colors |  “Hovering does….This falls into her learning style..” PSS: SE, Learn. |  – Changed the color of the interactive elements. – Increased the contrast between the interactive element and the interface background.
Fix-6: Labels and Titles |  “She sees new information…needs to understand what….trying to provide.” PSS: Motiv, Info. |  – Changed labels for STT x-axis: Move Number instead of Time. – Changed the BTW and STT y-axis: 100% win and 100% loss instead of just win and loss. – Added a title bar “Spot the Broken AI”.
Fix-7: Top Moves Comparison |  “…but they may not be sure how to reason with this information.” PSS: Motiv, Info. |  – Added “Top 5” moves with their score, win, loss, and draw percentages.
Fix-8: Tying Arrows |  “..no update for the other player…she may consider….something wrong.” PSS: Info, Risk, SE. |  – Added arrows on BTW, STT, OTB that point toward the rectangles representing the scores of the most recent move. – These arrows appeared for a few seconds when the user clicked the next/prior move button.
Fix-9: Game Log |  “…nothing has changed….so Abi might be confused.” PSS: Risk. |  – Added a game log beside the move rank list. – Added info on all the moves in the log. – Added a scroll bar and a separator between the game log and the top 5 moves.
Fix-10: BTW Legend |  “The relationship between…board and the BTW…isn’t clear.” PSS: SE, Risk. |  – Added legend to the top right for btw. – Changed the coloring of the move lines. Darker color for more recent moves. – Made the legend interactive.
Fix-11: Text and Prompts |  “Nothing is clearly stating that the game is over…” PSS: SE. |  – Added a text message at the beginning shows the game is starting. – Added a text message at the end that shows the game has ended. – Alongside the last move’s number added who won. – Removed who moves next for the last move. – Mentioned the game number with the move, e.g., Game 1 Move 1.
Fix-12: Chart Highlight changes |  “When highlighting…, there are multiple….This causes confusion for Abi..likes to have a complete understanding…” PSS: Info, SE, Learn. |  – Added the same highlighting on OTB for the winning sequence. – Added a thick vertical bar on STT indicates that this is the last move. – Changed BTW interaction to just highlight the rectangle from the current data series.
Fix-13: Game History |  “Because of IP/Risk, …rethink what they’re doing before moving on.” PSS: Info, SE, Risk. |  – Provided a drop-down game history at the top. – Provided a summary of what happened in the earlier games inside the game history drop-down.
Fix-14: Move History |  “Abi will want to spend more time, because with the prior move…what it’s telling her.” PSS: Info, SE, Risk. |  – Team J wanted to have the feature to go back to any move from an older game. We have not implemented it.
Fix-15: Zoom in/out |  “Not sure that Abi would get the information…given her information processing style. PSS: Info, SE. |  – Team J wanted to have a zoom-in/out feature for the explanation charts. We have not implemented it.
9 of the 15 fixes: Fix-1, 2, 4, 5, 6, 7, 8, 10, and 15 were targeted to
improve the explanations.