# PADTHAI-MM: A Principled Approach for Designing Trustable, Human-centered AI
systems using the MAST Methodology

###### Abstract

Designing for AI trustworthiness is challenging, with a lack of practical
guidance despite extensive literature on trust. The Multisource AI Scorecard
Table (MAST), a checklist rating system, addresses this gap in designing and
evaluating AI-enabled decision support systems. We propose the Principled
Approach for Designing Trustable Human-centered AI systems using MAST
Methodology (PADTHAI-MM), a nine-step framework what we demonstrate through
the iterative design of a text analysis platform called the REporting
Assistant for Defense and Intelligence Tasks (READIT). We designed two
versions of READIT, high-MAST including AI context and explanations, and low-
MAST resembling a “black box” type system. Participant feedback and state-of-
the-art AI knowledge was integrated in the design process, leading to a
redesigned prototype tested by participants in an intelligence reporting task.
Results show that MAST-guided design can improve trust perceptions, and that
MAST criteria can be linked to performance, process, and purpose information,
providing a practical and theory-informed basis for AI system design.

###### keywords:

decision support system, artificial intelligence, trust, standards,
intelligence community, use-inspired

[1, 1, 1, 1, 1, 1, 3, 2, 1, 1]Nayoung Kim, Myke C. Cohen, Yang Ba, Anna Pan,
Shawaiz Bhatti, Pouria Salehi, James Sung, Erik Blasch, Michelle V. Mancenido,
Erin K. Chiou Arizona State University, Air Force Office of Scientific
Research, U. S. Department of Homeland Security***Email addresses:
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Weiss

##  1 Introduction

Recent strides in artificial intelligence (AI) have raised concerns of AI
trust and transparency given the potential impact that AI use may have in
various decision settings. For example, ChatGPT is a generative AI interface
capable of natural language interactions through text chat with its users.
Although its users can engage ChatGPT across a broad range of topics, many do
not fully appreciate its merits without understanding how it reaches its
conclusions (Zhai,, 2023; Jousselme et al.,, 2023). In contrast, white-box
models—algorithms whose outputs are intrinsically interpretable from their
decision-making processes—have been in demand for high-risk areas like
healthcare and finance where trust and transparency are crucial (Loyola-
González,, 2019). However, much of the recent growth in AI applications is
attributed to black-box deep learning techniques, which can outperform white-
box models on many complex tasks such as face recognition or machine
translation. Although deep learning models have been widely adopted in recent
years, the debate persists over the potential risks posed by their largely
inexplicable nature (Rudin,, 2019; Durán & Jongsma,, 2021; London,, 2019).

Various technical approaches exist to improve the trustworthiness of black-box
models while preserving their performance over white-box alternatives.
Explainable Artificial Intelligence (XAI) (Gunning & Aha,, 2019) methods aim
to increase the transparency of black-box models by ”patching” them with post-
hoc explanations to make their decisions more understandable to people.
Examples of post-hoc explanations include representative predictions (Ribeiro
et al.,, 2016), IF-THEN rules (Ribeiro et al.,, 2018), or feature
contributions (Lundberg & Lee,, 2017). However, these explanations may not
always increase peoples’ trust because they are typically designed from an
engineer’s perspective, not the end users. Recent research supports this view:
Alufaisan et al., (2021) found that while providing AI predictions can
generally improve peoples’ decision-making, offering additional explanations
on why a prediction was made does not always impact decision-making
performance.

Several studies (Schmidt et al.,, 2020; Zerilli et al.,, 2022; von
Eschenbach,, 2021) have shown that an AI system’s level of transparency is
closely related to the trust that operators have in its decisions. A lack of
trust may result in people interacting with an AI system more naïvely or
avoidantly as they consider the potential utility and costs that its failures
could incur. Trust has thus become a focal point of discussion alongside
transparency and the groundbreaking progress of black-box models. Given the
known potential trade-off between AI transparency and performance (Lai & Tan,,
2019), trust is becoming an evaluative factor of interest for determining
which algorithms to used within an AI system.

Theoretical frameworks based on empirical studies generally suggest that well-
calibrated trust in AI involves a complex interplay of multiple factors beyond
simply providing explanations (Chiou & Lee,, 2023; de Visser et al.,, 2020).
Specifically, trust in automation depends, in part, on a person’s
understanding of its processes (i.e., how it performs tasks), purpose (i.e.,
why the system was developed), and performance (i.e., how effectively it
carries out those tasks) (Lee & See,, 2004). Recent work has shown these three
categories of information can be used by designers and evaluators to gain
insights into how operators trust a system and make informed decisions about
its usage (Chancey et al.,, 2017). Therefore, in addition to making AI
processes more explainable as XAI approaches suggest, theoretical trust
frameworks have long suggested that effectively designing for calibrated trust
also requires operators’ understanding of a system’s intended function (i.e.,
purpose) and reliability (i.e., performance) (Shneiderman,, 2020).

Theoretical frameworks may proffer general design guidelines to address human
information needs associated with trusting AI systems (e.g., Wheelock et al.,,
2020; Göttgens & Oertelt-Prigione,, 2021; Lee & See,, 2004). However, these
guidelines are subject to broad and varied interpretation or require
experienced Human Factors professionals for effective synthesis and execution.
Conversely, many actionable guidelines for designing trustworthy AI seem based
on industry or tradecraft standards rather than empirical and theoretical
works. Among these is the Multisource AI Scorecard Table (MAST), which was
developed by a group from the 2019 Public-Private Analytic Exchange program
sponsored by the U.S. Department of Homeland Security’s Office of Intelligence
and Analysis, on behalf of the Office of the Director of National Intelligence
(ODNI) (Blasch et al.,, 2021). MAST encompasses nine criteria derived from
analytic tradecraft standards, ICD 203 (ODNI,, 2015) and was intended to
inform the design and evaluation of AI system trustworthiness. Using a four-
point Likert scale (0-3, with 0 representing “poor” and 3 “excellent”), MAST
is one of few tradecraft-derived scales for AI system trustworthiness that
have been validated against existing theoretically-derived and empirically-
tested trust questionnaires (Chiou et al.,, 2022).

In this paper, we propose a design framework called PADTHAI-MM (Principled
Approach for Designing Trustable, Human-centered AI systems using the MAST
Methodology) that integrates the MAST criteria with an iterative, user-
centered design process for AI-enabled decision support systems (AI-DSS). We
demonstrate the PADTHAI-MM framework through our own design and development of
a prototype text summarization system called READIT (REporting Assistant for
Defense and Intelligence Tasks). To evaluate the effectiveness of the MAST
criteria as a set of design principles, we created two versions of READIT: a
“High-MAST” version designed to be rated highly on as many MAST criteria as
possible, and a “Low-MAST” version that delivers only basic system
functionalities to allow operators to complete a decision-making task. Our
process involved showing participants video mockups of the initial versions of
READIT (“READIT 1.0”). Our multi-disciplinary team then incorporated READIT
1.0 participants’ MAST evaluations and trust questionnaire responses with our
AI and data knowledge to build a functional version (“READIT 2.0”). Finally,
through Principal Component Analysis (PCA) and feature categorization, we
demonstrate not only the practical usefulness of our framework, but also the
relationship between MAST criteria and process, purpose, and performance
information.

##  2 PADTHAI-MM: A MAST-inspired Design Framework

The PADTHAI-MM design framework (Figure 1) integrates AI and data knowledge
with current trust scholarship to develop AI-DSSs that are trustworthy
according to the involved stakeholder needs. PADTHAI-MM comprises nine major
steps and is rooted in iterative design frameworks for product development
(Ulrich et al.,, 2008). We posit that each step contributes to the qualities
of reliability, responsibility, and system alignment to organizational values.
Design ideation and execution steps of PADTHAI-MM cover how to connect these
qualities to corresponding AI features.

Figure 1: PADTHAI-MM Design Framework

A foundational step of PADTHAI-MM is to focus on the _use case_ or _context_ ,
i.e., identifying the needs and expectations of operators and other peripheral
stakeholders where the AI-DSS will be deployed. This step addresses Customer
Relevance, one of the MAST criteria. After this ”opportunity recognition”
step, the design process moves to ”goal setting” for system functionality and
acceptable MAST ratings. This step is followed by ”iterative evaluation” which
involves design, test, and evaluation sub-processes until the previously set
goals are satisfied.

Throughout the iterative evaluation step, design concepts or features attuned
to the MAST criteria are generated to address stakeholder needs and
expectations. These concepts or features must also be informed by the design
team’s knowledge of AI and data system concepts; including the principles,
tools, and processes linked to the datasets that will be used by the AI-DSS.
This requires the design team to identify how to fulfill system functionality
and MAST rating goals without compromising system reliability

In this study, we demonstrate the PADTHAI-MM framework in our design of
READIT, a prototype text analysis system that uses modern AI technology and
data visualization techniques to assist human decision-making. For example, to
satisfy the MAST Sourcing criterion, we chose high-performing and
interpretable AI algorithms and verified datasets to ensure the quality and
credibility of underlying sources and methods. To assess the theoretical
validity of MAST as the foundational basis for PADTHAI-MM, we also investigate
three types of information known to affect trust in supervised automated
systems (i.e., purpose, process, performance; Lee & See,, 2004), and their
associations with our designed READIT features and MAST rating goals. We first
detail our design process and READIT case study below, before reporting the
findings of our investigation.

###  2.1 Step 0: Opportunity Recognition

In Step 0, valid opportunities for implementing an AI-DSS must first be
identified. In this step, key stakeholders should be identified and
systematically interviewed to gather information about the relevant working
processes and pain points in the existing work system.

####  2.1.1 Case study: READIT, a text summarization AI-DSS

To demonstrate PADTHAI-MM, we chose to develop a platform in the natural
language processing (NLP) domain within which Blasch et al., (2021) first
demonstrated the utility of MAST as an evaluation checklist. This required
that we also develop a plausible use case that would be of interest to our
project sponsors. Our initial use case envisioned that analysts could use
READIT as a text summarization platform to aid in real-time intelligence
reporting. One such task might be to quickly glean security information from
social media platforms, e.g., assess public sentiment towards a new U.S.
consulate building project in London.

###  2.2 Step 1: Determine system functionality goals

Step 1 involves determining work system goals and generating a clear
understanding of which work processes can be addressed by an AI-DSS solution.
This includes identifying which work processes should be performed by the AI-
DSS technology and which human work processes are introduced, expanded,
improved, or eliminated by incorporating the AI-DSS; and vice versa. Other
considerations include identifying what are the acceptable outputs for each
work process, and what are the system reliability expectations (e.g., error
rates over expected periods of operation). Potential consequences of
unacceptable outputs or failures, and other ethical, regulatory, and legal
considerations should inform stated limits of use.

####  2.2.1 Case study: Determining system functionality goals for READIT 1.0

Our initial READIT use case was designed for intelligence analysts to search
and summarize large corpora of text-based social media content to accurately
gauge public sentiment regarding a new London consulate building. We
determined that the key outputs of the system are: (1) a list of the most
influential content in terms of reactions and re-posts; and (2) an AI-
generated extractive summary that describes a corpus of posts. The primary
function allows users to search social media posts by keywords that link to
related content.

###  2.3 Step 2: Determine MAST criteria goals

Step 2 involves determining the importance of each MAST criterion and setting
rating goals. First, assess how the MAST criteria align with the system goals
identified in Step 1, followed by comparing the relative significance of each
criterion. Establishing the relative importance of each criterion is important
for resolving potential design trade-offs during subsequent feature generation
and configuration steps.

Step 2 also requires specifying the desired ratings for each MAST criterion
and an overall MAST score. Target MAST ratings can be compared against
Appendix A as a guide for generating relevant design concepts in subsequent
steps. In cases where an existing AI-DSS system is being redesigned, this step
also includes rating the current system using MAST to identify gaps that may
be independent of stakeholder-requested changes.

####  2.3.1 Case study: Determining MAST criteria goals for READIT 1.0

Our objective for the READIT architecture and interface was to achieve a
convenient and functional AI-DSS that addresses the MAST criteria. To assess
our design framework, we purposefully designed two versions of READIT, a High-
MAST version and a Low-MAST version. The High-MAST version was intended to
include the necessary features and level of information that would result in a
maximal MAST rating (i.e., scoring 3 or “excellent”) on as many criteria as
possible. The Low-MAST version was intended to include only the necessary
features and level of information to fulfill the identified system
functionality goals, with a minimum MAST target rating of 1 (“fair”) for each
criterion.

###  2.4 Step 3. Conceptualize MAST-informed features

After determining the system functionality and MAST rating goals, Step 3
involves conceptualizing AI system features or identifying existing solutions
to generate feature concepts. As features are conceptualized to address each
identified candidate process in Step 1, their likely MAST ratings may also be
estimated to identify their potential contributions to the MAST goals
identified in Step 2. Features may address multiple candidate processes or
multiple MAST criteria; thus, design teams should map how feature concepts
address each of its target candidate processes and assign MAST rating
estimates to the relevant MAST criteria. Note that feature concepts generated
at this step, including their interoperability, will be improved upon in
further iterations before their inclusion in system-level configurations.

####  2.4.1 Case study: Conceptualizing MAST-informed features for READIT 1.0

In our use case, we focused on generating features that allow experienced and
naïve operators to quickly evaluate the underlying data context, algorithmic
performance, and algorithmic processes used by the AI-DSS to derive its
results. An individual contributor (co-author A.P.) wrote the initial feature
descriptions and proposed a MAST rating for each. Then, over a series of
meetings, our project team of computer scientists, industrial engineers, and
human systems engineers discussed and eventually reached a consensus on each
of the feature descriptions and their respective MAST ratings. Project teams
working on critical, industrial-scale systems would benefit from documenting
these initial ratings to estimate the range of variation for each MAST
criterion; this constitutes another iterative step in the design process that
is not explicitly depicted in Figure 1. For future research, we hypothesize
that capturing these context-specific variations may indicate a need for
further scrutiny (i.e., defining and agreeing on feature concepts and how to
operationalize them) when aligning the AI-DSS features with MAST criteria and
rating goals.

Below we describe the system feature concepts for High-MAST READIT 1.0, which
are arranged according to each of the nine MAST criteria and are illustrated
in Figure 3. Feature concepts for Low-MAST READIT 1.0 (Figure 4) are also
provided for a visual comparison of the features. Appendix A provides a more
detailed definition of each MAST item and corresponding feature in READIT
High-MAST and Low-MAST versions.

  1. 1.

Sourcing focuses on the quality and credibility of the system’s underlying
sources, data, and methodologies. This can be addressed by a Raw content
feature that displays the raw datasets along with the resulting summaries
(2(a)). The Diagnostics, Data Sheet, and Date Range features (3(a)) also
display basic information about the training data, its cleaning and extraction
process, and metadata about the source data, respectively. The Diagnostics
feature displays the original data source locations, the number of original
poster accounts, and the number of pieces of content or posts with uncertainty
factors. Comparison Summary feature (2(b)) provides the function to
distinguish between content from larger organizations (e.g., news media firms)
from individual accounts.

  2. 2.

Uncertainty addresses uncertainty factors and their potential impact on major
analytic judgments. The Diagnostics feature provides diagnostic information
about uncertainties in the underlying content (e.g., misspelled words, foreign
characters or emojis, excessive punctuation, and machine- or human-generated
fake news) and highlights the factors in the raw content. The Alternative
Summary needed feature also pops up an alert in the left window, as shown in
3(b), if data integrity issues likely affected the final summary.

  3. 3.

Distinguishing gauges how the system differentiates underlying intelligence
information from its assumptions and judgments. For example, stating clearly
which information is generated from the AI-DSS, and which information might be
from corrupted sources, can help to distinguish between system inputs and
system outputs to assist with decision-making. Based on off-the-shelf
algorithms, the Diagnostics feature provides inferences on the likelihood of
whether data is human- or machine-generated fake news, which can help draw
analysts’ attention to the quality or intent behind the information posted,
rather than taking the results at face-value.

  4. 4.

Analysis of Alternatives refers to how the system assesses various decision
options and provides alternative outcomes if the data exceeds a certain
uncertainty threshold. The Alternative Summary needed feature alerts users if
the uncertainty factors warrant an alternative summary. For instance, if the
content contains numerous misspellings or primarily comprises fake news, this
feature will prompt the user to generate another summary using an alternative
set of keywords. When this alert feature is triggered, uncertainty factors are
also highlighted in the Raw content feature (3(b)).

  5. 5.

Customer Relevance addresses the implications of system outputs and the
system’s ability to support users’ needs and expectations. READIT 1.0 is
presumably catered toward a college-educated U.S. population, particularly
working intelligence analysts of varying levels of experience. Therefore, in
conjunction with the data and AI models we knew would be available, we
designed informative, short text summaries for the Most Influential Content
and Extractive Summary features (2(a)). The Content Shown feature (3(a))
displays the number of social media posts shown in the summaries, and the
Diagnostics and Word Cloud features respectively display information about the
content-creators (e.g., how long it has been active, how often content is
created by the account, etc.) and the content frequency.

  6. 6.

Logical Argumentation denotes how the system uses clear and logical reasoning,
elucidating how the AI generated information. Given the potential effects of
social media influence on the intelligence task outcome, search results in the
Summary and Raw Content features are ordered based on the number of followers.
Extractive Summaries are ranked based on their significance scores (i.e., the
frequency of the search terms appearing in each piece of content) that are
meant to align with the user’s manual inspection of raw content.

  7. 7.

Consistency focuses on how systems explain relevance or changes in analytic
judgments over time. One way to address this criterion is to include
comparisons between AI recommendations (or operator decisions) within or
across working sessions. The Comparison Summary feature provides statistics
for two selected summaries to compare and contrast results. Through this
feature, participants can also choose to create a comparison summary with
customized keywords to see how a new summary is similar or different from
previous results (top of 2(b)).

  8. 8.

Accuracy pertains to the quality of a system’s judgments and assessments, and
how clearly these judgments are presented by reducing ambiguity from the
likelihood, timing, and nature of decision parameters. The Date Range and
Diagnostics features show the date, time, and location of each piece of
content, as well as a comprehensive metadata report. These offer users the
ability to assess content relevance and quality when they perform the analysis
task.

  9. 9.

Visualization describes how a system incorporates effective visual information
displays where appropriate, e.g., as graphical representations of the raw data
or resulting statistical analyses. The Visualizations feature (bottom of 2(b))
includes various graphical representations of the data, including word
frequency bar charts, histograms of content by date, word clouds, webs of
original poster networks, etc. Visualizations are labeled and designed to be
easy to understand without needing to reference additional information.

###  2.5 Step 4. Generate MAST-informed system mockups

Step 4 focuses on generating system-level mockups, by first assessing the
compatibility of each feature concept from Step 3. Consider whether: (1)
features can be displayed simultaneously within interface space constraints
and human perceptual limitations (Fitts,, 1954); (2) feature workflows and
interaction modalities impact performance through process or information
redundancies, and; (3) feature concepts represent possible sources of
confusion for a user when gauging system trustworthiness, usability, or
algorithmic performance.

After assessing feature compatibility, system mockups are generated with a
features configuration that covers the candidate processes identified in Step
1. The goal is to generate several audiovisual or storyboard-like mockups
rather than working prototypes. These mockups should communicate user
interactions with the system in terms of necessary inputs and expected
outputs. The mockups ensure that alternative configurations can be assessed
quickly by stakeholders, in terms of how each configuration meets system
functionality and other goals defined in Steps 1 and 2.

####  2.5.1 Case study: Generating MAST-informed system mockups for READIT

From our final set of MAST-informed features, we generated system mockups of
each High-MAST and Low-MAST version, as shown in Figure 3 and Figure 4,
respectively.

The High-MAST mockup comprises three windows: (1) the Main Page (2(a))
consists of a keyword search bar and the resulting summaries and corresponding
source texts that are returned; (2) the Tools window (2(b)) can be accessed by
clicking a button on the right side of the main page and offers detailed
search options and content visualizations; and (3) the Options (3(a)) window
provides data information, such as the date range, the number of the dataset,
and content containing false information, special characters, or misspellings.
The High-MAST version also includes a feature that alerts users when the
search result exceeds a certain threshold, which prompts users to check
alternative summaries.

In contrast, the Low-MAST mockup includes only one window comprising only the
minimum features needed to complete the task: (1) a keyword search bar; and
(2) a search results display that consists of the most influential content and
their extractive summaries.

(a) Main page of READIT 1.0

(b) Tools window

(a) Options window

(b) Alert in READIT 1.0

Figure 3: (a) Main page of READIT 1.0 High-MAST, (b) two options of Tools
window, (c) four Options window, and (d) alert. Blue boxes highlight the
features fulfilling the corresponding MAST criteria. Figure 4: Low-MAST READIT
1.0.

###  2.6 Step 5. Determine MAST ratings for AI system mockups

After generating a number of system mockups as design alternatives (i.e.,
until the range of implementable solutions reaches saturation), Step 5
involves collecting stakeholder perceptions, including MAST criteria ratings,
perceived usability, and functionality. The process may involve creating
additional informational materials (e.g., audio-visual aids) to help
communicate how the processes identified in Step 1 are carried out in each
mockup, and to establish a baseline understanding of the system’s
functionality based on the features included. Potential stakeholders (e.g.,
operators and subject matter experts) should be recruited as participants in
an evaluative study, randomly assigned to each system alternative, and shown
the relevant informational materials. Participants should then complete their
MAST ratings of the system mockup, and complete questionnaires about perceived
usability and other assessment tools depending on the goals and processes
identified in Step 1. Interviews and focus group techniques can also be
employed to generate rich feedback that may identify unanticipated tradeoffs
that can inform decisions to continue iterating the design or to finalize the
design.

####  2.6.1 Case study: Determining MAST ratings for READIT system mockups

To determine MAST ratings for READIT 1.0 system mockups, we conducted an
experimental study as shown in Figure 5.

  1. 1.

Procedure. Participants were recruited through Prolific (www.prolific.com). To
access the study, participants were directed to a Qualtrics survey link
(Qualtrics,, 2020). The first page asked participants to provide their
Prolific IDs and an IRB-approved informed consent was administered.
Participants were then directed to a task description page that asked them to
imagine a fictional scenario in which they were U.S. intelligence analysts,
and their main task was to monitor the news for any ongoing threats to public
safety. They were told that to aid them in this task, they would be using an
AI-enabled decision support system (i.e., READIT), a technological aid meant
to supplement their own skills. Participants were informed that they would not
be able to interact with READIT, but that they would be presented with an
audio-visual walk-through and description of the tool’s interface and
features. To check for engagement and comprehension, participants were asked
to complete a short pass/fail quiz relating to the content of the task and
READIT descriptions. Participants who did not pass were removed from the
remainder of the study. Lastly, participants were asked to complete an
evaluation of the system using MAST and to respond to questionnaires with
items measuring trust, task engagement, and usability.

Figure 5: Evaluation Study Procedure for READIT 1.0.

  2. 2.

Independent Variable: System design. Designed mockups of the High-MAST and
Low-MAST versions of READIT 1.0 were administered as a randomly assigned
between-subjects condition. Detailed descriptions of the features and
descriptions for each version can be found in the preceding sections, and are
illustrated in Figure 3 and Figure 4.

  3. 3.

Dependent variables: MAST criteria and trust perceptions. The nine MAST
criteria were administered using an adaptation of the Blasch et al., (2021)
four-point MAST scale, with “1” representing “poor” and “4” representing
“excellent”. After initial pilot testing, our team decided to rephrase each
MAST criterion as a question accompanied by its relevant feature description,
as shown in Appendix B. The MAST-total score was calculated by adding these 9
criteria with a min - max range of 9 - 36. We also administered two well-known
questionnaires used to assess people’s trust in automation, with 12 items
(Jian et al.,, 2000) assessing general trust and 15 items (Chancey et al.,,
2017) assessing three types of information known to affect trust in
automation: performance, process, and purpose. Each item was rated on a seven-
point Likert scale, with “7” indicating “extremely agree” and “1” indicating
“extremely disagree”. We calculated the average rating, including the average
from each subcategory in (Chancey et al.,, 2017), to form the trust perception
scores.

  4. 4.

Control Variables: Engagement, and usability. To avoid common confounding
factors that could affect the MAST and trust ratings for READIT, we also
measured perceptions of READIT usability and task engagement. Usability had a
10-item questionnaire (Brooke,, 2020) and engagement had a 17-item
questionnaire (Schaufeli et al.,, 2002).

  5. 5.

Results. Study results showed that the High-MAST group compared to the Low-
MAST group was significantly different (descriptively higher) on the MAST
ratings (on 8/9 criteria, except Customer Relevance) and also significantly
different (descriptively higher) on the trust responses. In addition,
usability and engagement were not significantly different between High-MAST
and Low-MAST groups.

###  2.7 Step 6. Iterating feature and mockup designs

Step 6 focuses on refining the initial mockups into working prototypes after
analyzing stakeholder responses (Step 5) of the initial mockups and
alternatives generated in Step 4. The results of the analysis should be
compared to the system and MAST criteria goals determined in Steps 1 and 2,
respectively. These should also be interpreted in light of the system features
and MAST criteria mapping in Step 3. In some cases, as in our READIT case
study, Step 6 may result in the AI-DSS use case being reconceptualized to
better suit stakeholder needs identified in Step 0 of the PADTHAI-MM
framework.

Depending on the analysis results, the design team (ideally including AI
engineers, human systems engineers, and task domain experts) should recommend
which system alternatives are ready for prototyping, need further design
iterations, or should be discarded. Feature concepts and system mockups in
need of further design iterations should roughly follow the procedure outlined
in Steps 3 and 4. Step 6 ensures that expectations about the system’s
qualities remain within reasonable boundaries that are acceptable to the team.

####  2.7.1 Case study: Iterating feature and mockup designs for READIT

After analyzing our study data, our team revisited the READIT 1.0 system
mockups and discussed its use case and the feasibility of implementation in
light of our available resources (e.g., datasets, servers, project personnel,
timeline, and state-of-the-art models with similar functions). We assessed
READIT 1.0 from three perspectives: feasibility, task performance ability, and
usability of the Low-MAST and High-MAST versions. We concluded that the
initial mockup presented to participants was not implementable as a functional
or credible working prototype for actual intelligence analyst use cases.
Consequently, based on numerous project team discussions and feedback from
pilot participants and project stakeholders, we redesigned the use case and
system as READIT 2.0.

##### READIT 2.0 Use Case.

One of the main constraints was that the original use case (i.e., gauging
public sentiment from social media) could not be tested with an appropriate
dataset to support the analysis and data visualization features we envisioned,
and generating our own datasets would have gone beyond the scope of our
project. Therefore, we adopted a new but similar use case for READIT 2.0,
based on a corresponding available dataset from the VAST Challenge 2011 MC3
(IEEE SEMVAST Project,, 2011): Investigation into Terrorist Activity. In this
task, participants play the role of intelligence analysts searching for
information related to potential terrorist activity among thousands of given
news reports. Consultation with project stakeholders confirmed that this
revised task sufficiently aligned with real-world situations encountered by
analysts.

##### READIT 2.0 Features and Mockups.

We again developed Low-MAST and High-MAST versions for READIT 2.0 to further
demonstrate the utility of PADTHAI-MM, illustrated in Figure 6. Instead of the
extractive summaries that select and rearrange existing content from original
text sources, we designed READIT 2.0 to present abstractive summaries that
generate new content to describe news data. For the High-MAST version (6(a)),
we designed three data visualization features: topic clusters, a topic
similarity matrix, and a topic timeline. A feature was also designed to
provide users with raw content when operators click on the summary box, as
well as a list of all topics along with checkboxes to narrow down the range of
summary results. The Low-MAST version (6(b)) does not provide topic clusters
nor additional information buttons for each section but maintains basic
functionality for operators to complete the given task within a reasonable
time frame.

After designing the READIT 2.0 mockups, our team assessed the two designs to
ensure that the MAST rating goals would be met (“excellent” for High-MAST,
“fair” for Low-MAST). Table 1 presents the results of this process, in which
each rating represents the outcome of a group discussion and is presented as a
binary rating (yes/no) to support efficient decision-making and minimize
ambiguity. To fulfill MAST criteria goals, additional features were added to
the High-MAST version, including information about the platform and
technologies used, as well as visualization of the topics from the raw
documents. To further streamline the Low-MAST version, certain components were
simplified, resulting in some functions being omitted. For example, the Low-
MAST Topic Similarity matrix lacked a selection feature, the Topic Timeline
lacked a timeline selection option, and the Documents & Summaries did not
visualize the documents (presenting only a summary). Appendix B presents the
detailed features in READIT 2.0 matching each MAST criterion.

(a) High-MAST READIT 2.0

(b) Low-MAST READIT 2.0

Figure 6: (a) Main page of High-MAST and (b) Low-MAST of READIT 2.0. Compared with the Low-MAST, High-MAST has more interactive features (topic clusters, information buttons, dropdown option in topic similarity, etc.) and information (Documents and About tab) to demonstrate MAST criterion. Interface Items |  Data Preprocess |  Documents Tab |  About Tab |  Documents & Summaries |  Topic Clusters |  Topic Similarity Matrix |  Topic Filtering |  Topic Timeline
---|---|---|---|---|---|---|---|---
Sourcing |  |  |  |  |  |  |  |
Uncertainty |  |  |  |  |  |  |  |
Distinguishing |  |  |  |  |  |  |  |
Analysis of Alternatives |  |  |  |  |  |  |  |
Customer Relevance |  |  |  |  |  |  |  |
Logic |  |  |  |  |  |  |  |
Change |  |  |  |  |  |  |  |
Accuracy |  |  |  |  |  |  |  |
Visualization |  |  |  |  |  |  |  |
Table 1: MAST criteria in READIT 2.0. Red checkmarks and blue checkmarks
denote High-MAST and Low-MAST versions respectively. Each row represents a
specific MAST criterion and the columns show details for the front-end
interface, which include various functions presented by the AI system, such as
descriptive and calculated details, alerts, pop-up boxes, and buttons.

###  2.8 Step 7. Generating MAST-informed working prototypes.

Step 7 develops the system design identified at the end of Step 6 into a
working prototype. At this step, minor modifications approved by the design
team should still be expected as designs are translated into working programs.

####  2.8.1 Case study: Generating MAST-informed working prototypes for READIT

READIT 2.0 was prototyped as a JavaScript-based client to facilitate
interactivity between participants and the AI-DSS. The server was developed
using Python 3 and Flask library and hosted on the Google Cloud Platform. To
enhance operators’ ability to explore the dataset, READIT 2.0 included various
data visualizations. These visualizations were implemented using React, a
well-known JavaScript library used for creating personalized and interactive
data visualizations.

We designed the abstractive summary generation feature of READIT 2.0 using
PEGASUS (Zhang et al.,, 2020), a state-of-the-art deep learning-based text
summarization algorithm. The model uses the transformer encoder-decoder
structure. Input text is encoded as a numerical context vector, which is then
decoded to generate the abstractive summaries. The model is pre-trained to
choose and mask important sentences and then recover them, a process called
Gap Sentence Generation. We used the cnn_dailymail dataset for pre-training
PEGASUS. Both High-MAST and Low-MAST working prototypes of READIT 2.0 included
a functional front-end interface that allowed access to various natural
language processing and visualization features executed in the back-end. The
code is available here: https://github.com/nayoungkim94/PADTHAI-MM.

###  2.9 Step 8. Determining MAST ratings for working prototypes

Similar to Step 5, potential system users and stakeholders should be recruited
to use and evaluate the working system prototypes developed in Step 7. This
should involve designing a task scenario that closely resembles the
operational environment in which the system will be deployed. Along with
asking participants to evaluate the system through the questionnaires and
interview techniques specified in Step 5, people’s performance in the
decision-making task should be evaluated according to the system functionality
goals and metrics in Step 1.

####  2.9.1 Case study: Determining MAST ratings for READIT working prototypes

We recruited 23 intelligence analysts (IAs) from the U. S. Department of
Homeland Security (DHS) to operate and test READIT 2.0. Each analyst was
randomly assigned to a High-MAST or Low-MAST version and was asked to interact
with READIT, in a fictional scenario, to identify possible causes for a
terrorist attack. Afterward, study participants were asked to evaluate the
system by assigning MAST criteria ratings and filling out the Jian et al.,
(2000) and Chancey et al., (2017) trust questionnaires which were used to
evaluate READIT 1.0.

Our results show that the High-MAST group compared to the Low-MAST group were
significantly different (descriptively higher) on the MAST criteria ratings
(on 8/9 criteria except Customer Relevance) and significantly different
(descriptively higher) trust scores. Regarding usability and engagement, there
was no significant difference between the Low-MAST and High-MAST groups. The
High-MAST group required, on average, an additional 10 minutes to complete the
task compared to the Low-MAST group. Further details can be found in the
experimental results in Salehi et al., (2023).

###  2.10 Step 9. Determining if further design iterations are needed

For each of the working prototypes evaluated in Step 8, statistical analyses
of the questionnaires and performance metrics should be compared to the system
functionality and MAST rating goals set in Steps 1 and 2, respectively. If the
intended performance and MAST ratings are met by one or more of the working
prototypes, the design team and the stakeholders should determine which of
these to further develop towards an implementation plan. If none of the
working prototypes meet either performance or MAST criteria goals, another
iteration of the design process should be undertaken starting from Step 7. If
the design team collectively determines that the performance and MAST criteria
goals cannot be met realistically, design improvements are iterated starting
from Step 1 or 2.

####  2.10.1 Case study: Determining further iteration design needs for READIT
2.0

After analyzing the participant responses with READIT 2.0, our design team
determined that our system functionality and MAST criteria goals were met.
Because the primary purpose of our work was to conduct research, no further
design iterations were completed.

##  3 Theoretical Evaluation of MAST-Inspired Design Framework

To validate the theoretical basis of our proposed design framework, we
investigated the relationship between the MAST-inspired design features of
READIT 2.0 and participant responses to a trust questionnaire developed based
on trust in automation theory (Chancey et al.,, 2017).

###  3.1 Results

We used the collected responses to the MAST items (described in Step 9) for
further analysis. Our team created a mapping matrix using Table 1 to draw out
relationships between the MAST criteria and the specific features we designed
in READIT 2.0. To convert the ratings of MAST to the ratings of features, we
applied a linear transformation using the inner product of the two matrices,
the matrix of MAST evaluations and the matrix of MAST-READIT feature mapping.

###  3.2 Method: Principal Component Regression

Members of our team (EC, MM, NK, YB, SB, PS) reviewed and categorized each of
the MAST criteria sub-items, into the three information types known to affect
trust: process, purpose, and performance (PPPs). We completed the
categorization by voting, discussion, and consensus. These three categories
derive from scholarship on interpersonal trust in organizations (i.e.,
ability, integrity, benevolence respectively) in Mayer et al., (1995) and have
been connected to empirical studies in a literature review on trust in
automation Lee & See, (2004).

Because the MAST criteria informed several of our design features in READIT
2.0 (see Table 1), to identify connections of the predominant features and the
types of information (PPPs) captured by Chancey et al., (2017), we used a
standard dimensionality reduction method to obtain the most representative
components of our data, principal component analysis (PCA). PCA can transform
a high-dimensional dataset into a lower-dimensional space by identifying
relationships among variables while capturing the maximum data variance. As a
criterion to select the most important components, we set 0.3 as the threshold
for the absolute values of factor loadings and to ensure at least 80%
cumulative variance of the data would be explained by the first $k$ principal
components. Thus, the features with values over the 0.3 threshold would be
considered the most important contributors to the PPPs.

We then employed a linear regression using the PPPs as the target variable and
principal components (PC) as the predictor variables. Modeling was conducted
in R using dplyr (Wickham et al.,, 2019) and psych (Revelle,, 2018). The
approach of our validation analysis includes five steps:

  1. 1.

Transform ratings of MAST to ratings of features.

  2. 2.

Apply PCA to generate principal components from ratings of features.

  3. 3.

Select the first $k$ principal components that can explain at least 80% of the
variance, where $k=2$ in our case.

  4. 4.

Fit a linear regression model on the selected principal components as
predictor variables.

  5. 5.

Choose the significant principal components ($\textrm{p-value}\leq 0.05$) and
calculate the nested coefficients with a criterion of the absolute value of
PCA factor loadings over 0.3.

Here, the principal component regression is formulated as:

$PPPs=\beta_{1}\cdot PC_{1}+\beta_{2}\cdot PC_{2}$ (1)

where $PC_{i}=loadings_{i}\cdot features$ ($i={1,2}$) and $\beta_{i}$ is the
coefficients of the regression model. The nested coefficients $\beta_{i}\cdot
loadings_{i}$ denote correlations between features of READIT 2.0 and the PPPs.

###  3.3 PCA Results

Table 2 represents the relationship between features in READIT 2.0 and the
PPPs, which can be interpreted as the average change in ratings of PPPs for a
one-standard-deviation increase of MAST-informed feature ratings by
participants of READIT 2.0 while holding other predictors constant. For
example, a one-standard-deviation increase rating of the About Tab is
associated with an increase rating of purpose by 0.68 units on average.

Interface Items |  Process |  Purpose |  Performance
---|---|---|---
Data Preprocessing |  |  |
Documents Tab |  |  |
About Tab |  0.477 |  0.68 |
Topic Clusters |  0.287 |  0.328 |  0.479
Topic Similarity Matrix |  0.314 |  0.464 |  0.51
Topic Filtering |  |  0.412 |
Topic Timeline |  |  0.464 |  0.388
Documents and Summaries |  0.246 |  0.328 |
Table 2: Correlation between READIT 2.0 features and trust theory (Chancey et
al.,, 2017). The values in the table represent the nested coefficients
obtained from PCA loadings multiplied by regression coefficients $\beta$.
(Detailed information about loadings and coefficients are found here: Appendix
7 and 8.) Blank spaces indicate that certain features do not meet the 0.3
threshold of PCA loadings, suggesting they are not as significant as other
features for PPPs.

PCA results show that dominant features contributing to process, purpose, and
performance vary and several of our READIT 2.0 features contributed to all
three PPPs. Specifically, the About Tab, including descriptions about
technologies used in READIT 2.0, along with three other features: Topic
Clusters, Topic Similarity Matrix, and Documents and Summaries, supported by
underlying algorithms, is associated with process. Moreover, purpose connects
to all features except for Data Preprocessing and Documents Tab, and About Tab
is the more important feature. Intuitively, operators who check the detailed
descriptions on the About Tab to understand how the features in READIT 2.0
work, align with what purpose information provides. Finally, Topic Clusters,
Topic Similarity Matrix, and Topic Timeline all connect to performance and
assist operators in better understanding the system output through data
visualization.

In general, each MAST-informed feature in READIT 2.0 exhibited a moderate
positive correlation with performance, process, or purpose, indicating a
discernible impact. However, these features interplay to reach high trust
ratings. In READIT 2.0 when participants select a specific topic, three
visualization features (Topic Clusters, Topic Similarity Matrix and Topic
Timeline) display corresponding information at the same time, which then
interact to accomplish an enhanced outcome in the form of performance
information. Thus, we infer from our analysis that system designers who employ
MAST criterion as guidance for developing and selecting AI-DSS features can
also effectively address the PPPs of information known to affect trust
calibration.

##  4 Discussion

To bridge the gap between tradecraft-derived guidelines for trustworthy AI
systems and a human-centered approach to system design, we adopted MAST as a
design tool integrated with user-centered design principles and called this
process PADTHAI-MM. We used PADTHAI-MM to create a text summarization system,
READIT, for research purposes and as a way to evaluate the utility of MAST. We
designed two distinct levels of READIT, a High-MAST version with a rich set of
features to help operators perform a task, and a Low-MAST version with minimum
features necessary to complete the task. We also used MAST as a rating tool
within the goal setting, concept generation, and iterative evaluation steps of
our overall design framework. Although scholarship-derived frameworks for
trustworthy AI systems provide valuable insights into what factors contribute
to appropriately calibrated trust, they often lack detailed, actionable steps
that would be more accessible to a broader practitioner population. At the
same time, tradecraft and other field-based best practices may be more
narrowly suited for a particular area, and therefore can suffer from a lack of
transparent rigor and generalizability that more scholarly-based approaches
are attuned to. The integration of MAST as part of a design framework, and our
subsequent assessment of our designed features with respect to what is known
from the scholarly literature to affect trust in technology, is a step toward
bridging the gap between tradecraft and trust scholarship.

The results of our stakeholder evaluation study show that the designed levels
of High-MAST and Low-MAST were significantly different in both the MAST
criteria ratings and trust ratings. Moreover, our exploration of the
relationships between MAST and process, purpose, and performance information
of the designed system provides a more comprehensive understanding of how the
MAST criteria address trustworthiness. By employing techniques such as PCA, we
categorized the MAST-informed features of our READIT 2.0 design. This analysis
allowed us to assess the alignment of these features with the types of
information that are known to affect trust in technology. As a result, we can
establish a connection between the practical application of MAST as part of a
design process, and current trust scholarship.

###  4.1 Limitations

Although our resulting design served our main objective to assess the utility
of MAST and its relationship to trust in AI systems, we acknowledge several
limitations for future work. First, although we demonstrate a clear
distinction between the High-MAST and Low-MAST versions in terms of operator
perceptions, it is important to note that higher ratings in MAST criteria and
trust scores do not guarantee improved performance outcomes. Second, applying
and interpreting MAST items for black-box AI models can be challenging.
Certain models may employ highly intricate algorithms or techniques making it
difficult to assess the accuracy, reliability, and relevance of their results.
Additionally, evaluating the quality and credibility of underlying sources,
data, and methods can pose difficulties, especially for AI systems that use
extensive data from multiple sources. Third, the rapid pace of innovation and
development in the AI field necessitates continuous updates and refinements to
the MAST criteria to ensure ongoing effectiveness and relevance. Fourth, some
MAST items may exhibit high correlations in certain task contexts, suggesting
the potential for reducing the number of criteria in future work. Lastly,
striking a balance between potentially competing MAST items can present
challenges. For instance, some AI models may prioritize accuracy at the
expense of interpretability, hindering the understanding of their decision-
making process and potentially introducing biases or errors. Conversely,
models designed for interpretability may sacrifice accuracy, speed, or
relevance. Therefore, to advance the evaluation and development of trustworthy
AI systems we encourage future researchers to consider the complex nature of
AI models, the task environments in which they are intended to be deployed,
refine the MAST criteria, and address the trade-offs between competing
factors. Future projects, especially those developing systems intended for
fielding rather than for research purposes, should also consider the design
framework proposed here against other approaches like ecological interface
design or work-centered approaches Roth et al., (2021).

##  5 Conclusion

Our paper highlights the practical and theoretical merits of a tradecraft-
derived tool, the Multisource AI Scorecard Table (MAST), as the basis for
designing more trustable AI-enabled decision support systems. We propose a
design framework, PADTHAI-MM, that integrates the original MAST tool with
user-centered design principles, and demonstrate how to operationalize that
framework in our design and development of an AI-enabled decision support
system, READIT. Our previous work addresses in more detail the use of MAST as
a tool to assess operator trust in various types of AI-enabled decision
support systems Salehi et al., (2023). In this work, our proposed design
framework and investigation of the theoretical basis of MAST-informed features
further support MAST as an informative tool for researchers and practitioners
across human-AI system domains. In conclusion, our study demonstrates the
practical applicability and theoretical grounding of the PADTHAI-MM framework
to support the development of more trustable AI.

## Acknowledgments

This material is based on work supported by the U. S. Department of Homeland
Security under Grant Award Number 17STQAC00001-05-00. The views and
conclusions contained in this document are those of the authors and should not
be interpreted as representing the official policies, either expressed or
implied, of the Department of Homeland Security, Department of Defense, or the
university.

##  6 Disclosure of Potential Conflicts of Interest

The authors have no conflicts of interest to declare.

## References

  * Alufaisan et al., (2021) Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2021).  Does explainable artificial intelligence improve human decision-making?  Proceedings of the AAAI Conference on Artificial Intelligence, 35(8), 6618–6626.
  * Blasch et al., (2021) Blasch, E., Sung, J., & Nguyen, T. (2021).  Multisource AI scorecard table for system evaluation.  eprint arXiv:2102.03985.  Presented at AAAI FSS-20: Artificial Intelligence in Government and Public Sector, Washington, DC, USA.
  * Brooke, (2020) Brooke, J. (2020).  SUS: A ”Quick and Dirty” Usability Scale.  In Usability Evaluation In Industry (pp. 207–212). London, UK: CRC Press.
  * Chancey et al., (2017) Chancey, E. T., Bliss, J. P., Yamani, Y., & Handley, H. A. (2017).  Trust and the compliance–reliance paradigm: The effects of risk, error bias, and reliability on trust and dependence.  Human Factors, 59(3), 333–345.
  * Chiou & Lee, (2023) Chiou, E. K. & Lee, J. D. (2023).  Trusting automation: Designing for responsivity and resilience.  Human Factors, 65(1), 137–165.
  * Chiou et al., (2022) Chiou, E. K., Salehi, P., Blasch, E., Sung, J., Cohen, M. C., Pan, A., Mancenido, M., Mosallanezhad, A., Ba, Y., & Bhatti, S. (2022).  Trust in AI-enabled decision support systems: Preliminary validation of MAST criteria.  2022 IEEE 3rd International Conference on Human-Machine Systems (ICHMS), (pp. 1–1).
  * de Visser et al., (2020) de Visser, E. J., Peeters, M. M., Jung, M. F., Kohn, S., Shaw, T. H., Pak, R., & Neerincx, M. A. (2020).  Towards a theory of longitudinal trust calibration in human–robot teams.  International Journal of Social Robotics, 12(2), 459–478.
  * Durán & Jongsma, (2021) Durán, J. M. & Jongsma, K. R. (2021).  Who is afraid of black box algorithms? on the epistemological and ethical basis of trust in medical ai.  Journal of Medical Ethics, 47(5), 329–335.
  * Fitts, (1954) Fitts, P. M. (1954).  The information capacity of the human motor system in controlling the amplitude of movement.  Journal of Experimental Psychology, 47(6), 381–391.
  * Göttgens & Oertelt-Prigione, (2021) Göttgens, I. & Oertelt-Prigione, S. (2021).  The application of human-centered design approaches in health research and innovation: A narrative review of current practices.  JMIR mHealth and uHealth, 9(12), e28102.
  * Gunning & Aha, (2019) Gunning, D. & Aha, D. (2019).  Darpa’s explainable artificial intelligence (xai) program.  AI Magazine, 40(2), 44–58.
  * IEEE SEMVAST Project, (2011) IEEE SEMVAST Project (2011).  IEEE VAST Challenge 2011, Mini Challenge 3 (MC3).  Retrieved May 15, 2023, from https://www.vgtc.org/activities/vastcontest2011/.
  * Jian et al., (2000) Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000).  Foundations for an empirically determined scale of trust in automated systems.  International Journal of Cognitive Ergonomics, 4(1), 53–71.
  * Jousselme et al., (2023) Jousselme, A.-L., de Villiers, J., de Freitas, A., Blasch, E., Dragos, V., Pavlin, G., Costa, P. C., Laskey, K. B., & Laudy, C. (2023).  Uncertain about chatgpt: Enabling the uncertainty evaluation of large language models.  In 2023 26th International Conference on Information Fusion (FUSION) (pp. 1–8).
  * Lai & Tan, (2019) Lai, V. & Tan, C. (2019).  On human predictions with explanations and predictions of machine learning models: A case study on deception detection.  In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19 (pp. 29–38). New York, NY, USA: Association for Computing Machinery.
  * Lee & See, (2004) Lee, J. D. & See, K. A. (2004).  Trust in automation: Designing for appropriate reliance.  Human Factors, 46(1), 50–80.
  * London, (2019) London, A. J. (2019).  Artificial intelligence and black-box medical decisions: Accuracy versus explainability.  Hastings Center Report, 49(1), 15–21.
  * Loyola-González, (2019) Loyola-González, O. (2019).  Black-box vs. white-box: Understanding their advantages and weaknesses from a practical point of view.  IEEE Access, 7, 154096–154113.
  * Lundberg & Lee, (2017) Lundberg, S. M. & Lee, S.-I. (2017).  A unified approach to interpreting model predictions.  Advances in Neural Information Processing Systems, 30, 4668–4777.
  * Mayer et al., (1995) Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995).  An integrative model of organizational trust.  Academy of management review, 20(3), 709–734.
  * ODNI, (2015) ODNI (2015).  Intelligence Community Directive 203.  Retrieved from https://www.dni.gov/files/documents/ICD/ICD-203_TA_Analytic_Standards_21_Dec_2022.pdf.
  * Qualtrics, (2020) Qualtrics (2020).  Qualtrics.  Provo, UT.  https://www.qualtrics.com.
  * Revelle, (2018) Revelle, W. R. (2018).  psych: Procedures for Personality and Psychological Research.  R package Version 1.18. 10.
  * Ribeiro et al., (2016) Ribeiro, M. T., Singh, S., & Guestrin, C. (2016).  ”Why should i trust you?” Explaining the predictions of any classifier.  Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, (pp. 1135–1144).
  * Ribeiro et al., (2018) Ribeiro, M. T., Singh, S., & Guestrin, C. (2018).  Anchors: High-precision model-agnostic explanations.  Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
  * Roth et al., (2021) Roth, E. M., Bisantz, A. M., Wang, X., Kim, T., & Hettinger, A. Z. (2021).  A work-centered approach to system user-evaluation.  Journal of Cognitive Engineering and Decision Making.
  * Rudin, (2019) Rudin, C. (2019).  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1(5), 206–215.
  * Salehi et al., (2023) Salehi, P., Ba, Y., Kim, N., Mosallanezhad, A., Pan, A., Cohen, M. C., Wang, Y., Zhao, J., Bhatti, S., Sung, J., Blasch, E., Mancenido, M. V., & Chiou, E. K. (2023).  Evaluating trustworthiness of ai-enabled decision support systems: Validation of the multisource ai scorecard table (mast).
  * Schaufeli et al., (2002) Schaufeli, W. B., Salanova, M., González-Romá, V., & Bakker, A. B. (2002).  The measurement of engagement and burnout: A two sample confirmatory factor analytic approach.  Journal of Happiness Studies, 3, 71–92.
  * Schmidt et al., (2020) Schmidt, P., Biessmann, F., & Teubner, T. (2020).  Transparency and trust in artificial intelligence systems.  Journal of Decision Systems, 29(4), 260–278.
  * Shneiderman, (2020) Shneiderman, B. (2020).  Human-centered artificial intelligence: Reliable, safe & trustworthy.  International Journal of Human–Computer Interaction, 36(6), 495–504.
  * Ulrich et al., (2008) Ulrich, K. T., Eppinger, S. D., & Yang, M. C. (2008).  Product design and development, volume 4.  McGraw-Hill higher education Boston.
  * von Eschenbach, (2021) von Eschenbach, W. J. (2021).  Transparency and the black box problem: Why we do not trust AI.  Philosophy and Technology, 34(4), 1607–1622.
  * Wheelock et al., (2020) Wheelock, A., Bechtel, C., & Leff, B. (2020).  Human-centered design and trust in medicine.  JAMA, 324(23), 2369–2370.
  * Wickham et al., (2019) Wickham, H., François, R., Henry, L., & Müller, K. (2019).  dplyr: A Grammar of Data Manipulation.  R package version 0.8. 1.
  * Zerilli et al., (2022) Zerilli, J., Bhatt, U., & Weller, A. (2022).  How transparency modulates trust in artificial intelligence.  Patterns, 3(4), 100455.
  * Zhai, (2023) Zhai, X. (2023).  Chatgpt for next generation science learning.  XRDS: Crossroads, The ACM Magazine for Students, 29(3), 42–46.
  * Zhang et al., (2020) Zhang, J., Zhao, Y., Saleh, M., & Liu, P. (2020).  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In International Conference on Machine Learning (pp. 11328–11339).: PMLR.

##  Appendix A MAST Goal for READIT 1.0

Table 3: The goals of High-MAST and Low-MAST READIT 1.0.  MAST item |  High-MAST |  Low-MAST
---|---|---
Sourcing |  \- The system provides a description of the data that was used to derive the result including agencies, serial numbers, number of reports from an agency, intelligence type, source quality. \- System allows you to click on text in summary and it will bring up underlying report and relevant passages in it. \- Datasheet describes the quality of data used, which data repositories were searched, and data used to train and validate the system/methodology used by algorithm and describes potential biases if applied to data not trained for. \- User/data owner filters reporting so system does not consider duplicates, derivatives, or summaries. \- Systems has detailed knowledge of agencies, organizations, sources, collection methods, source evaluations and can provide summary of factors affecting quality and credibility of sources used to obtain result. |  \- The system will not provide a description of the type of content or number of posts used in the summary or any descriptive information regarding the content’s sourcing. \- The system does not have the ability to create reports about the data.
Uncertainty |  \- The system provides a detailed description of the accuracy of its summary with an accuracy score and explanation on how the score was derived. \- The system provides detailed diagnostic information about uncertainties in derived results and why they came about, including problems with structure of raw reporting or inconsistent/contrary information in the reporting. \- System can tell what type of data would be needed to increase confidence in results, i.e. better quality reporting in a certain area would increase accuracy of results. |  \- The system will not provide any information about uncertainties in the underlying content, nor will it provide any information about how those uncertainties may have affected the accuracy of the resulting summary.
Distinguishing |  \- The NLP system identifies structural or semantic indicators in the reporting that, if detected, could validate or refute assumptions used to develop the model. \- The developer includes a datasheet or factsheet that details aspects of the training data that were critical to the development of the model, how the system could react to use in unfamiliar environments, and suggestions for how to reduce risks of bias and error. \- The developer of the system includes detailed information that explains how press and government reporting are different, how the model might react, and remedies to be taken to account for the differences in the two types of reporting. \- The datasheet has information on assumptions that framed how data was chosen, curation rationale, annotations, and collection process. \- There is a model card showing type of model to understand the type of assumptions, domain-relevant expert input/judgment, and current model version. \- There is a factsheet on explainability, such as statement of purpose, if there is direct/indirect use of data and algorithm, and recommended use, users, and restrictions. \- System can explicitly state assumptions it used to fill in gaps when deriving summaries from limited or noisy reporting. |  \- The system does not provide any indicators to help distinguish between derived results (summary) and underlying data (individual pieces of content or posts). For example, there is no way to determine which parts of the summary are from different sources. The whole summary could be from a single or source, or many sources. This is not distinguishable. \- There is no additional information describing how the system was trained to analyze results. Also, there is no information on the specific sources or type of content that was used in the system’s training algorithm.
Analysis of Alternatives |  \- The system presents alternative possibilities for results when uncertainties, lack of data, etc. warrant alternatives, indicates which alternative is best, and likelihood that alternatives are correct. \- The system explains evidence and reasoning that underpin the alternative by providing a list of source documents and highlighting most relevant text in alternative summary. \- System indicates which data or keywords which, if detected, affect the likelihood of any identified alternatives. |  \- The system does not suggest any alternative summaries, even if uncertainties in the content warrant an alternative summary.
Customer Relevance |  \- Identifies trends or prospects for follow-up work by analyst/user. \- System provides useful information and results that are tailored to needs of analyst/user, can provide supplementary information (sourcing, visuals, etc.) if needed. \- System adds value by synthesizing large volumes of information across disciplines to draw conclusions about trends/prospects for additional analysis, connections between data, classification of data. \- System can also show trends in reporting (use timelines or visuals), extract entities or show connections between them, generate written reports based on analysis of data. \- System analyzes how current topics relate to other topics searched by analyst/user in past (information of interest to analyst/user). |  \- The system provides an extractive summary as well as most influential content in the output window. But the system will not provide any further information on the content used in the summary, such as details about the user accounts, location of content creator, or timestamp of content. \- The system cannot identify trends for follow up work.
Logic |  \- System can interpret data correctly and produce conclusions that are backed up by data. \- System can demonstrate how it derived its results, summary is clearly based on cited evidence. \- System provides information on logic used to derive the results and shows the source documents used to derive summary (covered in Sourcing criteria). \- Logical approach, consistent and meaningful. \- System uses appropriate language for intended analysts/users, displays results in a logical progression and results are contextualized in an understandable way. \- System displays topics derived from analysis and allows user to drill down to see underlying titles/summaries, reporting used to derive titles/summaries and other supplementary information (covered in Sourcing criteria). \- System provides detailed description of contrary or outlying data and why it chose the given result over other possibilities (covered in Uncertainty and Analysis of Alternatives criteria). \- System applies notable skill or sophistication in deriving results by combing different data, contexts, assumptions. \- System describes methodological framework from which it derives judgments and the evidence and assumptions used are relevant to the results. |  \- The system does not display results in a logical progression and results are not contextualized in a way the user can understand. All content is mixed up in the summary window and it is unclear, for example, if a post title is actually a tweet.
Change |  \- System notes changes or consistency with previous results. \- System provides a ‘similarity score’ between current and previous summaries on similar searches and notes any semantic shifts. \- System explains in simple terms how the new data caused system to maintain or change past derived results. \- System compares current results with results from outside systems or agencies on similar topics and describes similarities and differences to user/analyst. |  \- The system does not provide a way to create additional summaries and compare them with each other. The system also does not provide a way to note any changes with previous analyses or similar topics.
Accuracy |  \- The system’s derived results are clear and understandable. \- The system’s derived result factors in timing, likelihood and other conditional factors supporting an actionable conclusion. \- There are no ill-defined conditions and analyst/user is comfortable that if there were conditions, they would be expressed by the system. \- The system presents a measure of accuracy (percent confidence that a judgment is correct) which is sufficiently high. \- Analyst/user can independently determine that the systems results are accurate, complete, and consistent. \- Analyst/user can select portions of summary and test for accuracy. |  \- The system does not display results in a logical progression and rather presents them in a way that is confusing to the user. Different types of content are mixed together in a disorganized fashion. The summary displays mostly content related to the keywords used, but some content is remotely related. This makes the resultant summary non-actionable for a user, since it is not possible to simply remove irrelevant content because it is not possible to determine why that content was included (i.e. the criteria for being included in the summary).
Visualization |  \- Includes understandable and directable visual information where appropriate. \- Visual is pertinent to information conveyed in summary and helps show key concepts (like graphs, scatterplots, or visual that links items grouped in a topic). \- Visual has proper labels and data points are explained. \- Analyst/user can learn something new from visual produced. \- Visualization highlights data to clarify, complement, or augment understanding. \- The system chooses the most effective type of visual to display because it knows the analysts’ goals and data available. \- The system can display micro and macro trends in visuals in a way that is easily digestible and with a visual representation of why trends are significant. \- The visual displayed does not require analyst/user to look through other sources, etc. to understand and it does not overly reference other addendums, etc. |  \- The system cannot provide any visualizations.

##  Appendix B The Nine MAST Criteria for READIT 2.0

Table 4: MAST and READIT 2.0 feature descriptions for High-MAST and Low-MAST MAST item |  Questions and Feature Descriptions
---|---
Sourcing |  How well can the system identify underlying sources and methodologies upon which results are based?
High-MAST: In the Documents page, you can see descriptive information about
the documents (data) used to gather the clusters including basic information
and detailed descriptions of the sources. The About page includes information
on the clustering model, models for summarization, training data, possible
biases, pre-processing of data, and quality of the data used in training to
derive results. In the main dashboard view, you can view the data used to
derive the cluster either by hovering or clicking on it including the cluster
title, number of documents, top terms, and representative documents. The
representative documents can be viewed as a summary (derived result) or raw
version.
Low-MAST: For any given cluster in the main dashboard view, you can view more
details about it by clicking on it. The title of the cluster, number of
documents, and summaries of the documents will be displayed in the documents
and summaries pane. Only the derived results are shown, not the underlying
sources and data used to derive the clusters or summaries.
Uncertainty |  How well can the system indicate and explain the basis for the uncertainties associated with derived results?
High-MAST: READIT indicates levels of uncertainty with derived results in two
ways, as described in the About page. First, READIT includes keywords per
cluster to show how documents in clusters are related to each other. Keywords
are displayed with a term frequency–inverse document frequency (tf-idf) score
which measures the certainty the word fits with the cluster. Second, READIT
includes similarity scores to assess the similarity between clusters. This
score is calculated using cosine similarity to show the certainty that
clusters are related to each other.
Low-MAST: In the topic similarity visualization, the relationship between two
topics is colored from white to dark blue with dark blue indicating a higher
certainty the two topics are related. These relationships are not labeled with
numbers, neither is it explained how this similarity is calculated.
Distinguishing |  How well can the system clearly distinguish derived results and underlying data?
High-MAST: For any given cluster you can view more details about the data used
to derive the cluster either by hovering or clicking on it. The About page for
READIT includes information on the clustering model, models for summarization,
training data, underlying assumptions for choice of training data, quality of
the data used in training to derive results, possible biases, pre-processing
of data, recommended uses and users, and restrictions on use. The About page
was created with domain expert input.
Low-MAST: When opening or clicking on clusters, you can view more details
about that cluster. The title and summary of representative documents will
appear. The raw data used to derive the title and summaries is not displayed.
There is no About page with information on how these titles or summaries are
calculated.
Analysis of Alternatives |  How well can the system identify and assess plausible alternative results?
High-MAST: In the topic similarity visualization, users initially view the
visualization where the topics are ordered alphabetically. By factoring in the
similarity score and uncertainties, READIT can reorder the view in this
visualization such that highly related topics will appear together to present
an alternative view.
Low-MAST: READIT is not able to show alternative results when uncertainties in
the data warrant them. There is no way to reorder visualizations based on any
criteria.
Customer Relevance |  How well can the system provide information and insight to users?
High-MAST: READIT synthesizes large corpora of documents and produces clusters
of similar documents. The topic similarity visualization shows which clusters
are most highly related to each other. Users can examine the clusters and
their relationships in the topic similarity view for trends for follow-up
work. READIT is also able to suggest locations to filter by if the documents
contain multiple locations. Users can also filter all visualizations by topic.
There is a topic filtering pane where users can check all, or some topics and
the corresponding selected topics will be highlighted in the visualizations.
Low-MAST: READIT synthesizes large corpora of documents and produces clusters
of similar documents. The topic similarity visualization shows which clusters
are most highly related to each other. Users can examine the clusters and
their relationships in the topic similarity view for trends for follow-up
work.
Logic |  How well can the system help the user understand how it derived its results?
High-MAST: For any given cluster you can view more details about the data used
to derive the cluster either by hovering or clicking on it. The About page
includes information on pre-processing of data. READIT includes an option to
filter results by location if location information is detected in the
document. To give location options, READIT must consider the location
information in the context of the document, and other assumptions about the
embedding of the location in the document.
Low-MAST: When clicking on clusters in the main view, you can view the title
and representative documents in summary form. The titles and summaries are
understandable to users. Information on how clusters, titles, and summaries
are formed is not included. There is also no information on the pre-processing
of data.
Change |  How well can the system help the user understand how derived results on a topic are consistent with or represent a change from previous analysis of the same or similar topic?
High-MAST: In the Documents page, READIT includes information on similar
searches from other agencies. Similar searches may be based on the average
length of the document, number of documents, or number of clusters generated.
Low-MAST: READIT does not have a way to note changes from previous analyses or
similar analyses. READIT also cannot compare current results with those of
other agencies which had similar results.
Accuracy |  How well can the system make the most accurate judgments and assessments possible, based on the information available and known information gaps?
High-MAST: The READIT About page includes information on system verification
and validation methodology, and results from the training data where the
system achieved sufficiently high accuracy. To assess the accuracy of READIT,
users can view the full documents used in each cluster and compare them
against the top terms to independently determine whether the documents match
the top terms. Likewise, users can view a summary of the document and compare
it against the full version of the document in the documents and summaries
view to see if the summary is accurate.
Low-MAST: READIT does not include information on system verification,
validation methodology, or information on the training of the system where it
achieved sufficient accuracy. Since underlying sourcing information and raw
data are not included in the system, it is difficult to assess whether the
topics and summaries are accurate.
Visualization |  How well can the system incorporate visual information if it will clarify an analytic message and complement or enhance the presentation of data and analysis? Is visual information clear and pertinent to the product’s subject?
High-MAST: READIT uses three main visualizations to enhance users’
understanding of the clusters. First, in the topic overview visualization,
clusters are displayed as bubbles where the size of the bubbles can indicate
anomalies. Next, READIT also creates and displays a topic similarity
visualization to help understand the connections between clusters. Lastly,
there is a timeline view in READIT to display clusters on a timeline (if
documents contain date information). All visualizations are simple and labeled
properly. Users can view more details about the visualizations by clicking on
them or hovering over them or filtering all visualizations by cluster using
the filtering option.
Low-MAST: READIT uses two visualizations. The similarity matrix shows the
similarity scores between topics. Darker colors indicate more similarity but
score values are not shown. The timeline shows the clusters on the timeline.
Visualizations contain no interactivity and users are not able to click or
hover on items to view more details about the visualizations.

##  Appendix C Study Questionnaires

Table 5: Dependent and Control Variables Variables |  Example item(s) |  Number of items / Reverse items |  Scale
---|---|---|---
MAST-total (Blasch et al.,, 2021) |  Sourcing, uncertainty, distinguishing, analysis of alternatives, customer relevance, logical argumentation, consistency, accuracy, and visualization |  9/0 |  9 - 36
Trust (Jian) (Jian et al.,, 2000) |  “I can trust the system.”; “The system looks deceptive.” |  12/5 |  1 - 7
Trust (Chancey) (Chancey et al.,, 2017) |  “I understand how the system will help me perform well. “; “The information the system provides reliably helps me perform well. |  15/0 |  1 - 7
Engagement (Schaufeli et al.,, 2002) |  “I was immersed in this research task.”; “To me, this research task was challenging.” |  17/0 |  1 - 7
Usability (SUS) (Brooke,, 2020) |  “I felt very confident using the system.”; “I thought the system was easy to use.” |  10/5 |  1 - 5

##  Appendix D PCA Evaluation Results

(a) Performance

(b) Process

(c) Purpose

Figure 7: PCA loadings for three dimensions in trust theory

(a) Performance

(b) Process

(c) Purpose

Figure 8: Principal Component Regression results