# xPath: Human-AI Diagnosis in Pathology with Multi-Criteria Analyses and
Explanation by Hierarchically Traceable Evidence

Hongyan Gu<EMAIL_ADDRESS>0000-0001-8962-9152 ,  Yuan Liang
<EMAIL_ADDRESS>,  Yifan Xu<EMAIL_ADDRESS>580 Portola Plaza, Room
1538University of California, Los AngelesLos AngelesCalifornia90095USA ,
Christopher Kazu Williams<EMAIL_ADDRESS>,  Shino Magaki
<EMAIL_ADDRESS>,  Negar Khanlou<EMAIL_ADDRESS>,  Harry
Vinters<EMAIL_ADDRESS>,  Zesheng Chen
<EMAIL_ADDRESS>UCLA David Geffen School of Medicine757
Westwood Plaza, Pathology and Laboratory MedicineLos AngelesCaliforniaUSA90095
,  Shuo Ni<EMAIL_ADDRESS>580 Portola Plaza, Room 1538University of
California, Los Angeles and University of Southern CaliforniaLos
AngelesCalifornia90095USA ,  Chunxu Yang<EMAIL_ADDRESS>580 Portola
Plaza, Room 1538University of California, Los Angeles and Peking
UniversityCalifornia90095USA ,  Xinhai Robert Zhang<EMAIL_ADDRESS>1653 West Congress Parkway, Department of PathologyRush University Medical
CenterChicagoIllinois60612USA ,  Yang Li<EMAIL_ADDRESS>Google
ResearchMountain ViewCalifornia94043USA ,  Mohammad Haeri<EMAIL_ADDRESS>3901 Rainbow BoulevardUniversity of Kansas Medical CenterKansas
CityKansas66103USA and  Xiang ‘Anthony’ Chen<EMAIL_ADDRESS>University of
California, Los Angeles580 Portola Plaza, Room 6730ALos
AngelesCalifornia90095USA

(2021)

###### Abstract.

Data-driven AI promises support for pathologists to discover sparse tumor
patterns in high-resolution histological images. However, from a pathologist’s
point of view, existing AI suffers from three limitations: (i) a lack of
comprehensiveness where most AI algorithms only rely on a single criterion;
(ii) a lack of explainability where AI models tend to work as ‘black boxes’
with little transparency; and (iii) a lack of integrability where it is
unclear how AI can become part of pathologists’ existing workflow. Based on a
formative study with pathologists, we propose two designs for a human-AI
collaborative tool: (i) presenting joint analyses of multiple criteria at the
top level while (ii) revealing hierarchically traceable evidence on-demand to
explain each criterion. We instantiate such designs in xPath — a brain tumor
grading tool where a pathologist can follow a top-down workflow to oversee
AI’s findings. We conducted a technical evaluation and work sessions with
twelve medical professionals in pathology across three medical centers. We
report quantitative and qualitative feedback, discuss recurring themes on how
our participants interacted with xPath, and provide initial insights for
future physician-AI collaborative tools.

Human-AI collaboration; digital pathology; medical AI; meningioma

††copyright: acmcopyright††journalyear: 2021††ccs: Human-centered computing
Human computer interaction (HCI)††ccs: Applied computing Life and medical
sciences††ccs: Computing methodologies Machine learning

##  1\. Introduction

One critical step for cancer diagnosis and treatment is pathologists’ analysis
of histological glass slides obtained from a patient’s tissue sections to
identify evidence of tumor patterns and determine the diagnosis (e.g., benign
vs. malignant) and grade based on medical guidelines.

Such an analysis is often challenging for pathologists, due to the sheer
amount of effort it requires to identify sparse, small-scaled, non-homogeneous
histological patterns of tumors in each patient’s case. Further, the process
suffers from subjectivity due to the intra- and inter-observer variations,
e.g., different interpretations of the grading guidelines and different ways
of sampling and examining the slides to ‘implement’ a given guideline (Cai et
al., 2019b).

To overcome these challenges, digital pathology, enabled by advanced whole
slide imaging techniques, promises to transform traditional manual optical
microscopic examinations to be automated by data-driven artificial
intelligence (AI) (Holzinger et al., 2017). Specifically, there is a recent
development of deep learning techniques to detect or grade carcinoma with
digitized Whole Slide Images (WSI) (Pantanowitz et al., 2011), such as breast
cancer (Wang et al., 2016; Huang and Chung, 2018; Rakhlin et al., 2018),
prostate cancer (Ström et al., 2020), lung cancer (Wang et al., 2019), uterine
cervical cancer (De et al., 2013), and glioblastoma (Barker et al., 2016).

Although multiple AI-aided diagnostic systems have been designed preliminarily
for pathologists (Schneider et al., 2012; Bankhead et al., 2017; Carpenter et
al., 2006; Saltz et al., 2017; Martel et al., 2017), there remain barriers
that prevent their adoption into a clinical setting. Studies indicate that
various factors are in play, including a lack of concerns for clinicians’
needs that undermine their motivation to accept AI for clinical usage, and a
lack of workflow integration where the system is disruptive and time-consuming
to use in practice (Yang et al., 2016). Other research pointed out that
medical users desire more transparency to overcome subjectivity and more
interpretability to correct the model when the model makes mistakes (Cai et
al., 2019b; Xie et al., 2020). Building upon this prior work, we conducted a
formative study with four board-certified pathologists that further summarizes
three gaps of employing AI-enabled analysis of histological data:

  1. (1)

The gap of comprehensiveness. Most AI models tend to focus on one specific
criterion inferred from one specific type of histological data, while
pathologists in practice use multiple criteria and data types, e.g.,
Hematoxylin and Eosin (H&E) staining111A primary type of staining used for
histopathology examinations. for initial examination, and for instance, Ki-67
immunohistochemistry (IHC) staining222A more precise type of staining that
utilizes the antibodies to highlight specific histological features, in this
case, the histological pattern of mitosis. (Abry et al., 2010) to collect
different information to come up with a differential diagnosis (Gurda et al.,
2014; Shih and Kurman, 1998);

  2. (2)

The gap of explainability. Most AI models function as ‘black boxes’ and lack
transparency in how they arrive at certain findings, including, locally, how
an individual criterion is computed and, globally, how different criteria are
combined to arrive at a diagnosis;

  3. (3)

The gap of integrability: most AI models abstract histological analysis as a
computational problem, yet it remains unclear how to integrate such models
into pathologists’ existing workflow and practices.

Figure 1. A pathologist uses xPath in a top-down workflow to oversee how
multiple AI models have contributed to the grading diagnosis of meningioma — a
type of brain tumor: (a) at the top-level, an AI-suggested diagnosis (i.e.,
WHO Grade III) according to the World Health Organization guidelines, which
consist of (b) a list of meningioma grading criteria examined by multiple AI
models; (c) an arrow highlights the main contributing criterion to the current
grading result; (d) as the user selects the histological pattern of ‘mitosis’
criterion, they see a list of sampled evidence based on which AI grades this
criterion; (e) each piece of evidence consists of AI’s output probability,
confidence level, and a saliency map that highlights the spatial support for
the mitosis class; (f) clicking a pair of thumbnail images registers the
evidence to a whole slide image viewer with continuous magnification; the
yellow outer box corresponds to the area of one high power field in the
optical microscope, the blue box in the middle corresponds to a patch that
includes the positive detection (a mitotic cell from xPath’s AI), while the
inner red box points out a more precise location of such a detection,
corresponding to the evidence in the list; (g) a user can verify each piece of
sampled evidence by clicking on ‘approve’, ‘decline’ or ‘declare-uncertain’
button; (h) a heatmap can be enlarged to see a global distribution of each
criterion; (i) the user can also override each histological feature manually.
Correspondingly, the findings demonstrated in (a,b) would be updated as the
user overrides or modifies the AI’s results.

To fill in these gaps, we propose a human-AI collaborative workflow for
pathologists with two key design ingredients.

  * •

Joint-analyses of multiple criteria: at the top level, we present AI’s
findings based on multiple juxtaposed criteria across multiple data types,
which are combined to produce a single diagnosis based on rules derived from
existing medical guidelines. Such a design addresses comprehensiveness by
following how pathologists often examine more than one criterion; the
presentation of rules addresses global explainability, i.e., how different AI-
computed criteria contribute to a diagnosis.

  * •

Explanation by hierarchically traceable evidence: for each criterion, we
present a trace of evidence hierarchically across three levels: clicking a
top-level finding based on a specific criterion (Figure 1b) brings a user to
the mid-level list of samples (Figure 1d); The mid-level sample for the
mitosis criterion consists of AI probability, confidence level and a saliency
map (Chattopadhay et al., 2018) that highlights the spatial support for the
mitosis class (Figure 1e); clicking one sample further brings a user to the
original WSI for examining low-level details (Figure 1f). Such a design
addresses local explainability by making the provenance of a criterion
traceable and transparent; further, the top-down workflow is similar to (thus
integrable with) pathologists’ existing practices of delegating work to
trainees and overseeing their findings.

###  1.1. System Overview

We instantiate these two design ingredients in the implementation of xPath — a
human-AI collaborative brain tumor grading tool333Currently, we focus on the
grading of meningioma—the most common primary type of brain tumor—as a point
of departure for exploring the design of xPath. The goal is to aid
pathologists to grade meningioma with the aid of AI, which assumes that tumor
areas have already been identified on a slide (e.g., from an early radiology
examination). for pathologists to perform top-down, multi-criterion
histological analysis. In this work, the medical task of meningioma grading is
selected since it represents one of the hardest practices of the pathologists
— grading meningiomas is particularly challenging because it covers three
aspects of difficulties for pathologists: (i) pathologists are required to
locate and examine multiple histological features from cross-data-typed H&E
and IHC slides, (ii) slides’ high resolutions ($\sim$$(10^{5})^{2}$) vs.
small-sized histological patterns (e.g., mitosis, ($\sim$$90^{2}$)), and,
(iii) the sparse, non-homogeneous distribution of the histological patterns.
As such, the practice of grading meningiomas is a favorable arena of studying
how AI should be applied to assist pathologists in carrying out the task.

Figure 1 shows an overview of xPath ’s interface. A typical workflow starts
with a pathologist first seeing the top-level suggested tumor grade (Figure
1a), where an arrow (Figure 1c) highlights the main contributing criterion
that leads to the suggested grading. The analysis of different criteria
(Figure 1b) is produced by multiple AI models that examine histological data —
H&E and Ki-67 — based on the World Health Organization (WHO) meningioma
grading guideline (Louis et al., 2007). The pathologist can further select and
drill down to a specific criterion, which retrieves a set of examples as
evidence (Figure 1d) to explain AI’s findings. For example, for the criterion
of mitotic count (the count number of a type of cell within a fixed size of
area), the pathologist can see pieces of evidence (Figure 1d) detected by
xPath’s mitosis classification model. Each piece of evidence (Figure 1e)
demonstrates multiple explainable components, including probability,
confidence level, and a saliency map. Moreover, the pathologist can open a
heatmap (Figure 1h) overlaying the whole slide image to overview the density
distribution of positive mitotic cells recognized by the AI. Selecting a piece
of evidence directs the pathologists’ attention to a high power field444One
high power field corresponds to a field of view under x400 magnification from
the optical microscope. (HPF, yellow box) of the mitosis on the original whole
slide image, where they can further examine the low-level histological
features and approve/decline/declare-uncertain AI’s analysis with one click
(Figure 1g,i), which in turn will update AI’s findings on individual criterion
and, if necessary, the overall suggested grading as well based on WHO
guidelines.

We conducted two evaluations to validate xPath:

(i) A technical evaluation shows AI models identifying multiple meningioma
grading criteria, achieving F1 scores of 0.755, 0.904, 0.763, 0.946 in
classifying four histological patterns of mitosis, necrosis, prominent
nucleolus, and sheeting tissues, respectively. Moreover, models used in xPath
achieve averaged error rates of 12.08% and 29.36% in counting nuclei (for the
histological pattern of hypercellularity) and calculating Ki-67 proliferation
index;

(ii) Work sessions with twelve555We would like to point out that our
participants are highly-specialized medical experts that come from a much
smaller population than general users and are very difficult to recruit due to
their busyness. medical professionals666, which includes two attendings, two
fellows, seven senior residents, and one junior resident. across three medical
centers, comparing the performance of xPath with an off-the-shelf whole slide
image viewer as the baseline. The result shows that, with less than an hour of
learning, participants were able to use xPath to make more accurate grading
decisions. Specifically, participants gave correct gradings for 7/12 cases
with the baseline interface. In comparison, participants gave 17/20 cases
correct gradings using xPath. In the meantime, a post-study questionnaire
shows that participants found xPath more comprehensive ($p$=0.001), more
integrable with their existing workflow ($p$=0.006), requiring less effort
($p$=0.002), and more effective on reducing the workload ($p$=0.002) in
grading meningiomas. Moreover, they gave xPath high ratings on explainability
($\mu$=5.58/7) and trust ($\mu$=5.83 and 6.00/7). Last but not least,
pathologists were more likely to use xPath in the future ($p$=0.002), and gave
more overall preference on xPath (9/12 “totally prefer”, 3/12 “much more
prefer”).

###  1.2. Contributions

In terms of system input, xPath goes beyond previous work that merely relies
on a single data type (e.g., H&E (Rakhlin et al., 2018; Liu et al., 2017; Wang
et al., 2016; Gu et al., 2020), or immunohistochemistry examinations (Saha et
al., 2017; Xing et al., 2019; Anari et al., 2010)); instead, xPath’s pipeline
utilizes multiple H&E-based criteria (Figure 1b) and a complementary
immunohistochemistry data type — Ki-67 slides as input. In terms of system
output, xPath goes beyond prior work on automating diagnosis by ‘black box’ AI
(Irshad, 2013; Lu and Mandal, 2013; Mishra et al., 2018; Yap et al., 2015);
instead, xPath examines whole slide images with joint analyses across multiple
criteria where explanatory evidence can be traced hierarchically from top-
level grading to mid-level samples and to low-level details on a whole slide
image. Moreover, xPath differs from previous medical human-AI collaboration
systems (Cai et al., 2019a; Corvo et al., 2017; Krueger et al., 2019) (i) by
focusing on a different clinical task of diagnosing with AI, which requires an
aggregation of detections and a fusion of decisions from multiple criteria and
(ii) by contributing a top-down diagnosis workflow that assists pathologists
in overseeing and verifying AI as a task integrable to their day-to-day work.

Our main contribution is a generalizable tool design for human-AI
collaborative diagnosis that employs a workflow with joint-analyses of
multiple criteria and explanation by hierarchically traceable evidence,
addressing three existing gaps of comprehensiveness, explainability and
integrability. Our study provides initial insights into the application of the
proposed AI-assistive diagnosis, which share the common requirements of
processing multiple types of medical information and overseeing AI’s
performance explainably from the global findings to traces of local evidence.

##  2\. Background of Meningioma

In this work, we target the challenging task of meningioma grading to probe
the design of human-AI collaborative tools for pathology diagnosis.

Meningioma is the most common primary brain tumor in adults and, according to
the World Health Organization (WHO) guidelines (2007), can be graded as Grade
I, Grade II, or Grade III (Louis et al., 2007). The future grading of
meningioma in the new WHO guideline (2021) still recommends the same criteria
for grading, although the nomenclature is slightly different.

The accurate grading of meningioma is vital for the treatment planning: the
Grade I tumors can be treated with either surgery or external beam radiation,
while Grade II/III ones often need both treatments (Walcott et al., 2013);
meanwhile, research shows that patients with Grade III meningiomas suffer a
higher recurrence rate as well as lower survival rate in comparison to Grade
II patients (Palma et al., 1997).

Pathologists need to search and locate features across various magnifications
in order to grade meningiomas. Specifically, they first localize the regions
of interest (ROIs) in low magnification (x40), then switch to the patch level
with a higher magnification (x100), and sometimes zoom further into the
highest magnification (x400) into the cell level. These steps are usually
repeated multiple times until pathologists have collected sufficient findings
to conclude a grading and sign-off the case. Figure 2 briefly visualizes such
a workflow with typical pathology slides for demonstration. Pathologists’
grading workflow starts with a resected session from a patient that cuts the
tumor into slices and then stains it with Hematoxylin and Eosin (H&E) solution
on a glass slide (Figure 2a). Apart from the H&E, Ki-67 immunohistochemistry
(IHC) (Abry et al., 2010) is an additional staining method that is often used
(Figure 2b) to provide an estimated proliferation index (Figure 2d,k), which
is highly correlated to meningioma grading. Next, we describe the WHO
guidelines for the meningioma grading (Louis et al., 2007):

  * •

Grade I (benign) meningiomas include “histological variant other than clear
cell, chordoid, papillary, and rhabdoid ”(Brat et al., 2008) and a lack of
criteria from grade II and III meningiomas.

  * •

Grade II (atypical) meningiomas are recognized by meeting at least one of the
four following criteria:

    1. (1)

the appearance of 4 to 19 mitoses (Figure 2c,l) in 10 consecutive high-power
fields (HPFs). Moreover, since mitoses are challenging to recognize in H&E,
the Ki-67-positive nucleus (Figure 2k) in the corresponding areas of Ki-67
(Figure 2d) are often referred to for disambiguation;

    2. (2)

at least three out of five histological features are observed:
hypercellularity — an abnormal excess of cells in an HPF (Figure 2f),
prominent nucleoli — enlarged nucleoli in a cell (Figure 2g,m), sheeting —
loss of ‘whirling’ architecture (Figure 2h), necrosis — irreversible injury to
cells (Figure 2i), and small cell — cell aggregation with high
nuclear/cytoplasmic ratio (Figure 2j);

    3. (3)

brain invasion — invasive tumor cells in brain tissue is observed (Figure 2e);

    4. (4)

the appearance of clear cell or chordoid histological subtype.

  * •

Grade III meningiomas are justified if at least one of the following criteria
met (Backer-Grøndahl et al., 2012):

    1. (1)

20 or more mitoses per 10 consecutive HPFs;

    2. (2)

the appearance of frank anaplasia, papillary or rhabdoid histological subtype.

Figure 2. An example workflow pf pathologists’ grading with whole slide images
(WSIs) from our formative study. (a) The resected tissues are first stained
with H&E solution and scanned into WSIs. (b) An additional Ki-67 IHC staining
is usually used to locate mitoses. Pathologists then zoom into the patch level
and seek certain histological patterns listed in the WHO grading guidelines.
Specifically, pathologists look for (c) mitotic cells (marked in the red box)
in high power fields with the help of (d) Ki-67 stains; (e) brain invasion
(invasive tumor cells in brain tissue); five histological patterns, including
(f) hypercellularity (an abnormal excess of cells), (g) prominent nucleoli
(enlarged nucleoli pointed by the arrow), (h) sheeting (loss of ‘whirling’
architecture), (i) necrosis (irreversible injury to cells marked in the red
box), (j) small cells (tumor cell aggregation with high nuclear/cytoplasmic
ratio marked in the red box). For some criteria, e.g., mitosis (k,l) and
prominent nucleoli (m)), pathologists are required to zoom further into cell
level for examination.

As shown above, meningioma grading is not only challenging but high-stake — an
overestimated study would incur unnecessary treatment on patients, and an
overlooked one would cause a delay of necessary treatment.

##  3\. Related Work

In this section, we review the related work of xPath from three areas: (i)
data-driven AI algorithms for digital pathology, (ii) tools for digital
pathology, and (iii) human-AI collaborative tools.

###  3.1. Data-Driven AI Algorithms for Digital Pathology

According to Komura et al. (Komura and Ishikawa, 2018), there are three
primary categories of applications of data-driven AI algorithms in digital
pathology: (i) Computer-Aided Diagnosis (CAD), (ii) Content-Based Image
Retrieval (CBIR), and (iii) feature-triggered biomarker discovery. We will
review the representative works on CAD, since xPath falls into this category.

Current AI algorithms are primarily based on H&E slides, the most commonly
used stained slides for providing a detailed view of the tissue. In
particular, several existing approaches utilize AI algorithms to detect a
single criterion in different diseases. While not targeted at meningiomas
specifically, they seek the same criterion as in WHO grading guidelines (Louis
et al., 2007) (e.g., mitosis). For example, Irshad et al. include selected
color spaces and morphological features into the mitotic cell detection
pipeline to support breast cancer grading (Irshad, 2013); Lu et al. use
Bayesian modeling and local-region threshold method to detect mitotic cells
(Lu and Mandal, 2013); Mishra et al. propose a CNN network to identify
necrosis tissues in osteosarcoma tumor (Mishra et al., 2018); Zhou et al.
enhance the traditional U-Net models by applying nested, dense skip pathways
for nuclei segmentation (Zhou et al., 2018); Yap et al. use RankBoost-
permutations to integrate multiple base classifiers to detect prominent
nucleoli patterns from multiple tumor tissues (Yap et al., 2015).

Besides H&E slides, AI algorithms have also been devised on other stainings
that can be applied in clinical settings to assist decision-making, e.g.,
Ki-67 IHC tests. For example, Saha et al. use CNN as a feature extractor with
Gamma Mixture Model to detect immuno-positive and negative cells in breast
cancer (Saha et al., 2017). Xing et al. train a fully connected convolutional
network that can perform nucleus detection and classification from Ki-67
slides in a single stage (Xing et al., 2019). Anari et al. utilize fuzzy
c-means clustering to extract positive and negative cells for meningioma
tissues (Anari et al., 2010).

Different from prior work that simply applies an AI model for predicting and
red-flagging the target of a clinical task, the design of xPath has two unique
features. First, previous work has been mainly based on a single staining
examination as input, e.g., H&E images, which differs from clinical settings
where pathologists usually refer to multiple IHC examinations to acquire
comprehensive information (Gurda et al., 2014; Shih and Kurman, 1998).
Following such practices, xPath’s AI models accept multiple examinations and
direct users across different input sources during diagnosis. Second, previous
studies have focused on the direct prediction of the diagnostic target (e.g.,
classifying whether the tissue is tumor). Although some AI has reported
performance on par with pathologists, the non-transparent, non-explainable
characteristics of data-driven AI algorithms can still lead to distrust in
high-stake medical decision processes (Bera et al., 2019). In contrast, xPath
decomposes the diagnosis as rule-based joint analyses based on multiple
juxtaposed WHO criteria. By demonstrating the example-based explanations
according to the criteria, xPath supports pathologists to collaborate with AI
by seeing, modifying, and verifying hierarchical evidence that leads to AI’s
findings.

###  3.2. Tools for Digital Pathology

In the domain of digital pathology, multiple tools have been designed to
assist pathologists for the purpose of clinical diagnosis. Rather than taking
a pixel-based data-driven approach, most of these tools rely on morphological
features (e.g., nuclear shape and texture). Since such features are typically
associated with attributes of the disease, they tend to have greater
explainability and a stronger morphological underpinning than the ones based
on data-driven models. Meanwhile, because such features are shallow and from a
low level, these tools might suffer from inferior performance in diagnostic
tasks and are mostly only applied for general image analysis (Bera et al.,
2019). For example, ImageJ (Schneider et al., 2012), one of the most commonly
used scientific image analysis tools with extensions for computational
pathology, provides functions of nuclei segmentation and nuclei
characteristics analysis (e.g., intensity distribution and texture);
Cellprofiller (Carpenter et al., 2006) automates morphological analysis and
classifies cell phenotypes; QuPath (Bankhead et al., 2017) and CaMicroscope
(Saltz et al., 2017) provide extensive annotation and automation for nuclei
segmentation and positive cell counting; Pathology Image Informatics Platform
(PIIP) (Martel et al., 2017) extends Sedeen
viewer777https://pathcore.com/sedeen/ by adding plugins on out-of-focus
detection, region of interest transformation, and IHC slide analysis.

Despite the recent development of AI algorithms, only a few have been adapted
into tools. For example, Steiner et al.’s tool can red-flag skeptical regions
on slides for grading prostate biopsies and evaluate the impact of the
presence of AI on inter-observer consistency and time cost (Steiner et al.,
2020). Other models still lack integrability into physicians’ existing
workflow, which has been recognized as a long-standing issue. Teach et al.
have studied physicians’ attitudes on clinical consultation systems and offer
suggestions on computer-based decision support systems, e.g., “minimizing
changes to current clinical practices” and “enhancing the interactive
capabilities” (Teach and Shortliffe, 1981). Middleton et al. have reviewed
research on clinical decision support since 1990 and point out that the poor
integration in clinicians’ workflow is becoming a key barrier preventing the
application of such tools (Middleton et al., 2016). Specifically related to
histological diagnosis, there is limited research on tool design for
pathologist-AI collaboration on a clinical task (Cai et al., 2019a). Different
from prior work, xPath addresses the integrability of AI using a specific task
of meningioma grading as a case study. By working closely with pathologists,
xPath proposes a top-down workflow design for the tool inspired by how
pathologists oversee trainees’ work — presenting pathologists with a trace of
evidence hierarchically from each computed criterion to contextual information
and allowing pathologists to correct the AI on the fly.

###  3.3. Human-AI Collaborative Tools

Recent HCI research has demonstrated numerous examples of human-AI
collaboration, where AI takes human input and conducts automation to ease
humans’ burden on performing repeated routines (Chen et al., 2018; Willett et
al., 2018; Lee et al., 2019). In the digital pathology domain, multiple works
have shown that the human + AI team has the potential to increase the quality
of diagnoses. For example, Wang et al. report that combining AI and human
diagnoses improves pathologists’ performance on breast cancer metastasis
classification with an $\sim$85% reduction of human error rate (Wang et al.,
2016). More recent work by Bulten et al. points out that the introduction of
AI assistance increases pathologists’ agreement with the expert reference
standard in the prostate cancer grading (Bulten et al., 2021). Meanwhile,
Buccinca et al. show that users might over-rely on AI by failing to recognize
or correct AI when its predictions are wrong (Buçinca et al., 2021). Such a
contradiction in the performance opens up a question: how should human-AI
collaboration be employed to harness the AI without incurring bias? To answer
this question, Horvitz first sheds light on the design of human-AI
collaboration by proposing a series of principles of mixed-initiative
interactions (Horvitz, 1999). Amershi et al. further enhance Horvitz’s work
with 18 design guidelines on how humans could better interact with AI, such as
“support efficient correction” and “make clear why the system did what it did”
(Amershi et al., 2019).

However, Yang et al. point out that implementing these guidelines is a non-
trivial task (Yang et al., 2020) — specifically, the uncertainty and
complexity in AI’s inference make it hard for users to control the data
reasoning process. Going beyond collaborative research in the general domain,
Cai et al. highlight medical experts’ need for information from AI, which
includes the AI’s capabilities measured in well-defined metrics and
transparency to overcome subjectivity (Cai et al., 2019b). Meanwhile, merely
automating a part of work would not be sufficient to motivate the medical
users — a medical human-AI collaborative tool should set the explicit goal of
helping medical users increase the overall quality of work (Yang et al.,
2016).

A number of existing human-AI collaboration projects on pathology have been
focused on Content-Based Image Retrieval (CBIR). With a given slide (or patch)
from pathologists, such tools retrieve previous examples of a similar pattern
to help the decision-making. For instance, Hegde et al. propose a reversed
image searching tool to help pathologists find histological image patches of
similar histological features or disease states (Hegde et al., 2019); Cai et
al. enable pathologists to specify custom concepts that guide the retrieval of
similar annotated patches of histological patterns (Cai et al., 2019a).
However, CBIR focuses on image searching, which is a low-level task: what
images to search, how to use the searching results, and what to conclude
according to searching results. On the contrary, diagnosing/grading carcinoma
in digital pathology is a high-level task, automating an aggregation of
detections and a fusion of decisions from multiple criteria. This is also
confirmed by Tschandl et al.’s work, which discovers that the CBIR tools
require significantly more time to interact with than those giving diagnostic
predictions (Tschandl et al., 2020). In contrast to above-mentioned CBIR tools
(Hegde et al., 2019; Cai et al., 2019a), xPath is considered a tool of
Computer-Aided Diagnosis (CAD) or Clinical Decision Support System (CDSS).

Going beyond CBIR, existing CAD/CDSS tools enhance the detection in digital
pathology with visualization. For example, Corvo et al. develop PathoVA that
provides AI support for breast cancer grading by visualizing three types of
pixel-level clues (Corvo et al., 2017). The system can also track
pathologists’ interactions and help them generate reports by providing
snapshots for confirmed areas. Krueger et al. enhance users’ exploration of
multi-channel fluorescence images to support cell phenotype analysis (Krueger
et al., 2019). Specifically, the tool maintains hierarchical statistics about
the number of cell-level findings to help a user keep track of analysis and
interactively update the statistics with machine learning algorithms on the
fly. These tools provide a bottom-up approach to assist pathologists in making
a diagnosis: pathologists are only prompted with low-level AI-generated clues
(e.g., highlighting tumor cells with a segmentation map); then, the diagnosis
is drawn by pathologists from fusing observations with these clues. In
contrast, xPath implements a top-down workflow, where pathologists first see
an overall grading based on joint analyses of multiple criteria, then drill
down to localized areas with traceable evidence and further to low-level
patterns for verification and correction. Such a design provides “actionable
information” (Gu et al., 2020) and reduces the total areas of study for
pathologists. Moreover, to support such a workflow, xPath makes comprehensive
AI detections based on medical guidelines, whereas the above-mentioned prior
studies (Krueger et al., 2019; Corvo et al., 2017) only provide partial clues.

##  4\. Formative Study

We conducted a formative study with four board-certified neuropathologists
(average experience $\mu=21.25$ years) from a local medical center888Please
see the supplementary material for the demographic information for the
participants. to reveal the system requirements for human-AI collaborative
histological diagnosis. We started by describing the project’s motivation and
then asked pathologists to describe their typical process of examining a
patient’s case. Next, we asked them to describe the challenges in their
practice and their expectations on an AI-enabled system to assist such a
process.

###  4.1. Existing Challenges for Pathologists

We found two major challenges in the current pathology practice of meningioma
grading, which validate our motivation for introducing AI into the diagnosis
process.

Time consumption. The small-scaled characteristics in the patterns of interest
and the very high resolution of slides make the meningioma grading highly
time-consuming for pathologists. A resected section from a patient’s brain
tissue would generate eight to twelve H&E slides, and pathologists need to
look through all those slides and integrate the information found on each
slide. Except for the few experienced pathologists, meningioma grading can
take up to several hours to go through a single patient’s case that often
consists of 10+ slides. Automating portions of the slide examination process
by AI can potentially reduce such time consumption, alleviate pathologists’
workload, and increase the overall throughput.

Subjectivity. There are high intra- and inter-observer variations during the
grading of tumors. Pathologists summarize three factors contributing to such
subjectivity: (i) a lack of precise definitions — the WHO guidelines do not
always provide a quantified description for the five histological features for
high-grade meningioma. For example, for the ‘prominent nucleoli’ criterion,
the WHO guideline does not specify how large the nucleolus should be
considered as ‘prominent’; (ii) implementation of the examination process —
for example, the mitotic count for grade II meningioma is defined as 4 to 19
mitotic cells in 10 consecutive HPFs. However, the guideline does not specify
the sampling rules of these 10 HPFs. As a result, different pathologists are
likely to sample different areas on the slide; (iii) natural variability in
people, such as the level of experience, time constraint, and fatigue
(Croskerry et al., 2017). For AI, the definition and implementation of
guidelines can be codified into the model and visualized in the system that
performs consistently to overcome people’s variability.

###  4.2. System Requirements for xPath

In addition to the two innate requirements of reducing time and overcoming
subjectivity, we further identified the following requirements related to the
human-AI collaborative aspects.

Comprehensiveness. The grading of tumors involves multiple sources of
information (from different staining, e.g., H&E, Ki-67) and criteria (e.g.,
mitosis, nuclei density, necrosis, small cell, sheeting, prominent nucleoli,
brain invasion). To incorporate xPath into the current practice, the system
should comprehensively support all these sources and criteria.

Explainability. In lieu of a single grading result from a black-box AI model,
the system should provide visual evidence to justify the AI’s findings
according to the medical definition of the criterion. This is because each
criterion (based on histological features of an HPF) requires examining lower-
level details in order to interpret an AI’s finding and further needs to be
traceable to the original location in the whole slide image for a review with
more contextualized information. Overall, there should be explainability both
globally (how results from multiple criteria are combined to yield a grading)
and locally (, which includes (i) what evidence leads to the computed result
of each criterion, e.g., where mitoses are detected that lead to the number of
mitosis counts, and (ii) why a specific piece of evidence is captured by AI,
e.g., which part of the evidence convinces the AI that it contains mitoses).

Integrability Similar to how attending pathologists oversee trainees’ work,
the system should allow pathologists to oversee AI’s findings by retrieving
detailed contextualized evidence on-demand. When showing the evidence of
grading, the system should not overwhelm pathologists with all evidence from a
whole slide; rather, it should direct pathologists to the representative
regions of interest. Given that errors are inevitable for most existing AI
models, the system should enable pathologists to cross-check each criterion
and override the results manually when they detect an error.

##  5\. The Design of xPath

Guided by the aforementioned system requirements, we developed xPath with two
key designs: (i) joint-analyses of multiple criteria and (ii) explanation by
hierarchically traceable evidence. We first detail the two designs and then
describe how a pathologist uses xPath to perform a meningioma grading task.

Figure 3. Joint-analyses of multiple criteria in xPath’s design: (a) the
overall suggested grading; (b) a structured overview of each WHO criterion
with (c) an arrow highlighting the main contributing criterion to the
suggested grading; (d) users can override criteria by right-clicking on each
item and change the result to ‘found’, ‘not found’ or ‘uncertain’; xPath
provides color bars to indicate the status of each criterion: (e) red
indicates a confirmed abnormal criterion (or presence), (f) green indicates a
confirmed normal criterion (or absence), (g) orange indicates the criterion is
unconfirmed/confirmed uncertain, and (h) gray indicates the criterion is not
applicable in this case.

###  5.1. Joint Analyses of Multiple Criteria

Based on the formative study, we found that pathologists rely on the WHO
guideline to grade meningiomas, a process that involves reasoning jointly from
multiple criteria. Thus xPath’s design follows the WHO meningioma guideline
and employs AI to compute eight criteria for meningioma grading, i.e., mitotic
count, Ki-67 proliferation index, hypercellularity, necrosis, small cell,
prominent nucleoli, sheeting, and brain invasion999This work does not consider
using AI to identify the subtypes (e.g., clear cell, frank anaplasia) since
they are easy to be discovered and justified by pathologists.. Details on the
AI implementation are described in Section 6. These criteria can be split into
two categories: quantitative and qualitative. For the quantitative criteria of
the mitotic count and Ki-67 proliferation index, we show their predicted
quantitative values directly. For the rest of the criteria concerned with the
presence or absence of a specific histological pattern, xPath provides
recommendations of regions of interest (ROI) hotspots according to the largest
aggregations of AIs’ probabilities.

Figure 3 demonstrates the interface of multiple criteria. Besides showing the
current suggested grading result (i.e., the suggested ‘WHO grade II’, Figure
3a) and a structured overview of each WHO criterion (e.g., mitosis, Ki-67,
Figure 3b), xPath displays an arrow to indicate the main contributing
criterion that leads to the grading based on the underpinning WHO guidelines,
e.g., the mitotic count (Figure 3c). All the criteria are linked with ROIs
related to the findings101010Sampling rules of ROIs vary from different
criteria. Please refer to Section 5.2 and the supplementary material for
details.. Moreover, AI’s recommendation on all the criteria can be confirmed
and modified by the pathologist. xPath uses color bars (Figure 3e,f,g,h) to
indicate the status: as shown in Figure 3, red indicates a confirmed abnormal
criterion (or presence), green indicates a confirmed normal criterion (or
absence), orange indicates that the criterion is unconfirmed/uncertain
(neither presence nor absence), and gray indicates the criterion is not
applicable in this case. Once a pathologist overrides the result of any
criterion (Figure 3d), the color bar and the final grading is updated
correspondingly.

In summary, the joint analysis of multiple criteria addresses
comprehensiveness by following how pathologists often examine more than one
criterion; the presentation of rules addresses global explainability, i.e.,
how different AI-computed criteria are combined to arrive at a diagnosis.

###  5.2. Explanation by Hierarchically Traceable Evidence for Each Criterion

Another finding from the formative study is that, besides a global explanation
of the overall grading, pathologists also would like to see evidence that
justifies AI’s grading, e.g., how AI processes the image of a local patch (for
local explainability). Hence, we designed xPath to provide such explanations
by top-down, hierarchically traceable evidence. Further, such a workflow
mimics a scenario that we found in the formative study: pathologists
reviewing/overseeing trainees’ work. By replacing trainees with AI’s findings,
we emulated the relationship between the pathologists and trainees to
pathologist-AI collaboration, thus making AI more integral to pathologists’
current practices.

Figure 4. We designed a top-down human-AI collaboration workflow for
pathologists to interact with xPath (left) and pathologists’ corresponding
footprints on the xPath’s frontend user interface (right). A pathologist user
starts from (a) the automatically-generated suggested grading result and then
examines (b) the main contributing criterion. They can further examine (c) the
evidence list, and register back into the original whole slide image in higher
magnifications (d,e). Furthermore, users can (f) approve/decline/declare-
uncertain on the evidence, or (g) override AI results directly by right-
clicking on each criterion. For the rest of the criteria, the user could
repeat (c-g) until they have collected sufficient confidence for a grading
diagnosis.

As shown in Figure 4, xPath presents a hierarchical trace of evidence for each
criterion with positive findings, which allows a pathologist to see a list of
mid-level samples (Figure 4c) that lead to a computed criterion (Figure 4b).
For the most important criterion — mitosis, xPath demonstrates a series of
explanations in each mid-level sample, including AI’s output probability
(Figure 5a), AI’s confidence level (Figure 5b), and a saliency map (Figure 5c)
that highlights the spatial support for the mitosis class in the reference
image111111Please refer to the supplementary material for the implementation
of calculating the confidence level and the saliency map., allowing
pathologists to check AI’s validity on each sample quickly. Further, at the
low-level, xPath supports registering each sample of the mid-level evidence
into the whole slide image (WSI) to enable pathologists to examine in even
higher magnification and search nearby for more contextual information (Figure
4d,e). With the provided mid- and low-level information, a pathologist can
approve/decline/declare-uncertain a sample for a criterion with one click
(Figure 4f), or return to the top-level and directly override AI’s results on
each criterion (Figure 4g). Correspondingly, the overall suggested grading
(Figure 4a) is updated dynamically upon the user’s input.

Figure 5. For the most important criterion, mitosis, xPath demonstrates a
series of explanations in each mid-level sample, including the (a) AI’s
probability, (b) AI’s confidence level, which is justified by the probability
thresholds, and (c) a saliency map (calculated by the Grad-CAM++ algorithm
(Chattopadhay et al., 2018)) that highlights the spatial support for the
mitosis class in the reference image on the left. Figure 6. Selected pieces of
sampled evidence from in-the-wild detection where we applied trained models
directly on multiple H&E and Ki-67 IHC slides: (a) a highest focal region
sampling result of mitotic count on H&E slide (red box, 1HPF), the small blue
frames are the mid-level samples (that are shown on the evidence list), the
smaller red boxes in the blue frames mark the positions of mitoses found by
xPath’s AI; (b) a highest focal region sampling result on the Ki-67 IHC slide
(red box, 1HPF); (c) a highest region sampling result of mitotic count on H&E
slide (red box, 10HPFs), the small blue frames are the mid-level samples; (d)
highest region sampling result on the Ki-67 IHC slide (red box, 10HPFs); (e) a
hypercellularity ROI sample (blue box); (f) a necrosis ROI sample (blue box);
(g) a small cell ROI sample (the inner blue box, the outer yellow box marks
the dimension of 1HPF); (h) a prominent nucleoli ROI sample (blue box).

Figure 6 demonstrates typical examples of evidence provided by xPath during
the “in-the-wild” test, where we applied trained models directly on multiple
H&E and Ki-67 IHC slides. Particularly, for the mitosis-related criteria
(i.e., mitotic count and Ki-67 proliferation index) — the most reliable and
commonly used criteria in the grading — we introduce the two ‘shortcuts’ for
pathologists to look into AI’s results:

  * •

Highest Region Sampling. One WHO criterion is “mitotic count in 10 consecutive
HPFs”. Our formative study found that the inter-observer consistency of “10
consecutive HPFs” is low. To address this problem, xPath provides the highest
region sampling tool. The highest region is defined as a $2\times 5$ HPF area
with the highest number of mitotic counts (Figure 6c) or the highest Ki-67
proliferation index (Figure 6d). This tool speeds up a pathologist’s work by
helping them locate 10 consecutive HPFs from the WHO guidelines.

  * •

Highest Focal Region Sampling. From our formative study, pathologists
mentioned that the high-grade meningiomas share a common feature of increased
mitotic activities in a localized area. Hence, xPath provides the highest
focal sampling tool to help pathologists better localize highly concentrated
mitosis/Ki-67 proliferation index areas. In xPath, the highest focal region is
calculated as the one HPF with the highest number of mitotic counts (Figure
6a) or the highest Ki-67 proliferation index (Figure 6b). Using this tool,
pathologists can locate foci of highly-mitotic areas that the highest region
sampling might miss.

Pathologists can go beyond the sampled areas and navigate the high-heat areas
using heatmaps generated for the whole slide (please see the supplementary
material for details). For example, the mitosis heatmap registers all AI-
detected positive mitotic cells as a mitotic density atlas, where high-heat
areas indicate a high density of mitotic cells. As such, the heatmap would
serve as a ‘screening tool’ to help pathologists filter out unrelated areas
and rapidly narrow down to the ROIs that are scattered in an entire WSI. xPath
provides such ‘screening tools’ for all criteria.

In summary, in contrast to prior work that enables pathologists to define
their own criteria for finding similar examples (Cai et al., 2019a), xPath
aims at making examination based on an existing criterion traceable and
transparent with evidence, allowing pathologists to view comprehensive AI
findings from multiple criteria; further, the top-down workflow is integrable
with pathologists’ existing practices of delegating work to trainees, which
enables pathologists to oversee AI’s performance.

##  6\. Implementation of xPath’s AI Backend

xPath implements an AI-aided pathology image processing backend to compute the
eight WHO criteria of the mitotic count, Ki-67 proliferation index,
hypercellularity, necrosis, small cell, prominent nucleoli, sheeting, and
brain invasion. In this section, we describe the tasks, problem formulations,
datasets, and training details for the AI models. Finally, we report the
performance for each of the AI models from a technical evaluation.

###  6.1. Tasks & Problem Formulations

In contrast to performing an image search by user-defined concepts (Cai et
al., 2019a), xPath aims to screen the entire WSI to select regions of interest
(ROIs) and then determine grades based on such ROIs. To achieve this, xPath
includes six AI models and two detection rules, one for each criterion, to
distill quantized information on histological features. Below we describe how
xPath uses these techniques to process a whole slide image (WSI).

Figure 7. Data processing pipeline of xPath: (i) xPath takes H&E and Ki-67
whole slide images (WSIs) as input. (ii) For each WSI, xPath uses a non-
overlapping sliding window method to acquire (a) H&E and (b) Ki-67 patches
(size=$512\times 512\times 3$); Furthermore, each $512\times 512\times 3$ H&E
patch is processed with (c) resizing, (d) sliding window ($240\times 240\times
3$), and (c) another sliding window ($96\times 96\times 3$) to fit the inputs
of the down-stream models. (iii) xPath’s AI backend takes over the pre-
processed tiles and employs multiple AI models to detect WHO meningioma
grading criteria from each patch. Given a $512\times 512\times 3$ H&E patch,
xPath uses (f) a nuclei segmentation model to count the number of nuclei (for
hypercellularity judgment), (g) a necrosis classification model to calculate
necrosis probability, and (h) a sheeting classification model to calculate
sheeting probability. xPath further utilizes the nuclei counting results for
(k) small cell recommendation, and (l) brain invasion recommendation. For a
$240\times 240\times 3$ tile, xPath uses (i) a mitosis classification model to
obtain mitosis probability. For a $96\times 96\times 3$ tile, xPath uses (j) a
prominent nucleoli classification model to predict prominent nuclei
probability. For each $512\times 512\times 3$ Ki-67 patch, xPath detects
positive and negative nucleus and calculates the Ki-67 proliferation index;
(iv) All AI-computed results (marked in the green boxes) are shown in xPath’s
frontend for pathologist users to oversee.

####  6.1.1. Pre-process whole slide images

Figure 7 shows a pipeline of how xPath pre-processes a whole slide image. At
first, xPath cuts a high-resolution ($\sim$$(10^{5})^{2}$) whole slide image
into $512\times 512\times 3$-pixel121212For the criteria of sheeting and Ki-67
proliferation index, the dimension of each pixel is 0.5$\mu$m. For other
criteria, the dimension of each pixel is 0.25$\mu$m. patches with non-
overlapping sliding windows. A patch with an average pixel value greater than
240 is considered as background and is discarded. Otherwise, xPath further
processes each patch separately to fit the tasks of detecting the eight
criteria:

  * •

For the criteria of Ki-67 proliferation index, hypercellularity, necrosis,
small cell, brain invasion, xPath directly applies each AI model or detection
rule to the patch;

  * •

For detecting sheeting patterns, xPath resizes each patch to $224\times
224\times 3$ to reduce the computation burden;

  * •

For counting the mitotic figures, xPath further processes each patch using a
$240\times 240$ sliding window with a step size of 120 to fit the input of the
mitosis classification model;

  * •

For counting prominent nucleolus, xPath cuts each patch with a $96\times 96$
non-overlapping sliding window to fit the prominent nucleoli classification
model.

####  6.1.2. Detecting each criterion with machine learning techniques

After xPath pre-processes the whole slide image, it runs eight techniques to
detect eight criteria, which covers three types of tasks: classification (to
justify whether a given image is positive or not), semantic segmentation (to
recognize and segment nucleus from the tissue background), and rule-based
image recommendation131313xPath uses such an unsupervised approach to detect
the small cell pattern because of a shortage of IRB-approved annotated data.
(to recommend candidates based on fixed rules). Below we describe the target
of each task and its formulation. Specific thresholds are justified by the
maximum F1 scores achieved by each model in the validation set.

  * •

Mitotic Count (Classification). xPath uses an EfficientNet-b7 model (Tan and
Le, 2019) to identify mitosis (Figure 7i). A $240\times 240$ tile with a
prediction probability ¿ 0.78 is counted as positive. xPath further applies a
non-maximum suppression technique to post-process the overlapping positive
tiles. The mitotic distribution of the slide is calculated by merging the
results from each $512\times 512\times 3$ H&E patch.

  * •

Ki-67 Proliferation Index (Semantic Segmentation). xPath uses a pre-trained
Cycle-GAN model (Ghahremani et al., 2021) to detect both Ki-67 positive and
negative nucleus (Figure 7m). Given a $512\times 512\times 3$ Ki-67 patch as
the observation region, the Ki-67 proliferation index is calculated as
$\frac{\text{positive-count}}{\text{positive-count}+\text{negative-
count}}\times 100\%$.

  * •

Hypercelluarity (Semantic Segmentation). xPath uses a pre-trained deep neural
network (Graham et al., 2019) to segment and count the number of nuclei in a
$512\times 512\times 3$ H&E patch (Figure 7f).

  * •

Necrosis (Classification). xPath uses an EfficientNet-b5 model to justify
whether a $512\times 512\times 3$ H&E patch contains the necrosis pattern
(Figure 7g). A patch is considered necrosis-positive if the prediction
probability ¿ 0.74.

  * •

Small Cell (Rule-Based Recommendation). xPath applies the rules for
recognizing small cell patterns: selecting and recommending the top-10
$512\times 512\times 3$ H&E patches with the highest nuclei count within each
slide (Figure 7k). For each recommended patch, if it has ¿125 nucleus/patch,
then xPath includes it in the recommendation.

  * •

Prominent Nucleoli (Classification). Similar to mitosis classification, xPath
uses an EfficientNet-b0 model to classify prominent nucleoli (Figure 7j). To
avoid false-positive cases influencing the result, only tiles that have ¿0.9
prediction probabilities are counted as positive141414We choose precision
rather than recall in the prominent nucleoli classification because (i) the
performance of the prominent nucleoli classification model is not satisfactory
(see Section 6.3); (ii) unlike mitosis, this criterion is justified by the
presence of cell clusters that have prominent nucleolus, and missing one or a
few detections would not significantly influence the overall result.. xPath
counts positive tiles in each $512\times 512\times 3$ patch for calculating
the distribution of prominent nucleoli.

  * •

Sheeting (Classification). xPath uses an EfficientNet-b1 model (Tan and Le,
2019) to classify whether the patch includes a sheeting pattern (Figure 7h). A
sheeting patch is justified as positive if its prediction probability is
¿0.52.

  * •

Brain Invasion (Classification). xPath outlines the brain invasion pattern by
classifying whether a given $512\times 512\times 3$ H&E patch is tumor, brain,
or background (Figure 7l). If the tumor cells are invading the normal brain
tissues, it can be seen clearly with a heatmap visualization of tumors vs.
brain areas. Because meningioma is a high-cellular tumor, xPath classifies
tumor patches with the following rule: (i) patches that have ¿55 nucleus in
each H&E patch are counted as the tumor; (ii) patches that have [10,55]
nucleus are counted as the brain; (iii) otherwise, count as the background.

###  6.2. Dataset and Model Training

We built an in-house dataset consisting of 30 WSIs from a local medical center
for the model training and evaluation. The WSIs were cropped into patches of
the expected dimensions for each model. For the supervised tasks, we created
the ground-truth labels for all the patches by working with a board-certified
neuropathologist. For the necrosis and sheeting patterns, we used a random-
crop technique when generating patches from an annotated region. Note that
patches in different sets were generated from a different group of regions. To
train the models, we further randomly selected a subset of the training set to
be the validation set, and utilized it to determine the optimal
hyperparameters. Please find the supplementary material for more specific
training details.

Dataset |  | Dimension
---
(in pixels)
| # of Samples
---
(Training)
| # of Samples
---
(Testing)
Mitosis Nuclei | $240\times 240\times 3$ | 33,562 (1,925 positive, 31,637 negative) | 8,223 (336 positive, 7,887 negative)
Necrosis | $512\times 512\times 3$ |  | 4,383 (from 190 regions)
---
(651 positive, 3,732 negative)
| 3,587 (from 162 regions)
---
(770 positive, 2,817 negative)
Prominent Nucleoli | $96\times 96\times 3$ | 15,042 (2,447 positive, 12,595 negative) | 3,753 (609 positive, 3,144 negative)
Sheeting | $224\times 224\times 3$ |  | 3,660 (from 55 regions)
---
(1605 positive, 2055 negative)
| 2,340 (from 45 regions)
---
(1,185 positive, 1,155 negative)
Table 1. Dataset descriptions for each task. The input patch dimension in
pixels, the size of training/testing sets, and the distribution of
positive/negative patches are provided.  Figure 8. Classification performance
for (a) mitosis, (b) necrosis, (c) prominent nucleoli, (d) sheeting. The solid
lines in each sub-figure illustrate the Precision-Recall curves of each model.
The blue crosses indicate the performance achieved by the models using the
thresholds that maximized the F1 scores on the validation sets.

###  6.3. Technical Evaluation

We report the performance of AI models on testing sets. Specifically, we test
the supervised models for recognizing mitosis, necrosis, prominent nucleoli,
and sheeting criteria, and report the Precision-Recall curve, as shown in
Figure 8. In summary, xPath achieved F1 scores of 0.755, 0.904, 0.763, 0.946
in identifying the histological patterns of mitosis, necrosis, prominent
nucleolus, and sheeting. The scores indicate the effectiveness of our models
to implement xPath. Moreover, for the tasks of cell-counting in
hypercellularity and Ki-67 proliferation index criteria, we tested their
performance with 150 randomly-selected $512\times 512\times 3$ patches each
and report the average error rate. The results show that the average error
rate of nuclei counting (hypercellularity) and Ki-67 proliferation index is
12.08% and 29.36%, respectively.

Due to a lack of data at present, for brain invasion and small cell patterns,
rather than drawing a definitive conclusion, xPath uses a rule-based,
unsupervised approach to recommend areas for pathologists to examine. We
planned to validate the performance on these two criteria later in the work
session with pathologists; however, it was hard for the participants to
justify the small cell formation vs. inflammation areas without proper IHC
tests. As such, xPath’s AI performance on detecting small cell patterns was
not validated. For the brain invasion, most pathologists felt it was faster to
examine it manually and did not rely on AI’s recommendations.

##  7\. Work Sessions with Pathologists

The technical evaluation reported in the previous session validated the
effectiveness of xPath’s AI backend in the in-house dataset. However, it
remains unanswered whether xPath is beneficial to pathologist users in real
clinical settings. Notably, many previous cases showed how easily AI models
could break although they showed high accuracy in training/test data
(Strickland, 2019; Kandula and Shaman, 2019). To address these concerns, we
conducted work sessions with 12 medical professionals in pathology across
three medical centers and studied their behavior of grading meningiomas using
a traditional interface — an open-source whole slide image viewer called
ASAP151515https://computationalpathologygroup.github.io/ASAP/. This tool was
selected because it is open-source and has gained popularity in the digital
pathology research domain (Litjens et al., 2018). and xPath. In this study, we
referred to the traditional interface as system 1 and xPath as system 2 to
avoid biasing of participants. The main research questions are:

RQ1: Can xPath enable pathologists to achieve accurate diagnoses?

One reason of utilizing AI in xPath is because it can highlight ROIs of
multiple histological patterns, freeing pathologists from examining the entire
slide. However, it is still yet unclear whether introducing AI will have a
positive or a negative effect on pathologists’ diagnoses: On one hand,
multiple previous work show that the introduction of human-AI collaboration
improves pathologists’ performance (Wang et al., 2016; Bulten et al., 2021);
On the other hand, due to the existing limitations in AI models’ accuracy,
users face the risk to generating wrong diagnoses if they over-rely on the
non-perfect AI (Bansal et al., 2019; Buçinca et al., 2021). As such, we
hypothesize that —

  * •

[H1] Pathologists’ grading decisions with xPath will be as accurate as those
with manual examinations.

RQ2: Do pathologists work more efficiently with xPath?

Another reason for using AI in xPath is that it can improve the pathologists’
throughput by alleviating their workload. However, it remains unanswered how
AI will assist pathologists in xPath, given that previous work shows less-
carefully-designed AI might incur extra burdens (Gu et al., 2020). As such, it
is also necessary to find out whether pathologists can work efficiently with
xPath’s AI. We hypothesize that —

  * •

[H2a] Pathologists will spend less time examining meningioma cases using
xPath.

  * •

[H2b] Pathologists will perceive less effort using xPath.

RQ3: Overall, does xPath add value to pathologists’ existing workflow?

Going beyond the influence brought by AI, we introduce two design ingredients
— joint-analyses of multiple criteria and explanation by hierarchically
traceable evidence — to fulfill the three system requirements (i.e.,
comprehensiveness, explainability, integrability). In this study, we
investigate whether such designs will add value to pathologists’ existing
workflow. Specifically, we hypothesize that:

  * •

[H3a] xPath will improve comprehensiveness with the joint-analyses of multiple
criteria.

  * •

[H3b] xPath will improve explainability with explanation by hierarchically
traceable evidence.

  * •

[H3c] xPath will improve integrability with the top-down human-AI
collaboration workflow.

###  7.1. Participants

We recruited 12 medical professionals in pathology across three medical
centers. Our participants’ experience ranged from two to 10 years ($\mu$=4.38,
$\sigma$=2.16), including two attendings (A), two fellows (F), seven senior
residents (SR, $\geq$ PGY-3), and one junior resident (JR, $\leq$
PGY-2)161616Please see the supplementary material for the demographic
information for the participants.. All participants had received training for
examining meningiomas prior to the work session. Regarding familiarity with
digital pathology tools, pathologists had more experience in glass slides than
interfaces in digital pathology: six of them used
ImageScope171717https://www.leicabiosystems.com/digital-
pathology/manage/aperio-imagescope/ occasionally for training or reviewing
remote cases.

Figure 9. We used the ‘virtual cookie cut’ technique to generate the tests
cases. Specifically, we first collected (a) pairs of H&E (in x400) and Ki-67
(in x200) WSIs from a local medical center. Then, we generated ‘virtual cuts’
by (b) selecting 30,000$\times$30,000-pixel regions in H&E WSIs, and (c)
15,000$\times$15,000-pixel regions from the same position as their H&E
counterparts. (d) Each virtual case consists of one mandatory H&E slide with
two nodes and one optional Ki-67 slide with two corresponding ones.

###  7.2. Test Data

We collected 18 IRB-approved meningioma WSIs181818…which include eleven H&E
WSIs (scanned in x400), and seven Ki-67 WSIs (scanned in x200). from the same
medical center to generate the test cases. In normal conditions, each
patient’s case consisted of more than 10 WSIs, and an averaged-experienced
resident pathologist typically needs to spend about one hour to finish
examining an averaged-difficult case (i.e., criteria found in the case do not
lie on the grading borderlines). As such, we generated nine ‘virtual cases’
with the ‘virtual cookie cut’ technique (see Figure 9) to fit the task of
grading meningiomas191919We would like to point out that all cases used in
this study are meningioma-positive. Pathologists’ task was to grade
meningiomas instead of justifying each case as positive vs. negative. into the
hour-long work sessions. Each virtual case consists of a mandatory H&E slide
(in x400), and an optional Ki-67 slide (in x200). Each H&E slide consists of
two nodes (each has a size of 30,000$\times$30,000, which is relatively
smaller than the real cases), while each Ki-67 slide has two corresponding
Ki-67 nodes (each has a size of 15,000$\times$15,000) that were cut from the
same position as their H&E counterparts, if available. In total, nine virtual
cases have nine H&E slides and six Ki-67 slides, i.e., three virtual cases do
not come with a corresponding Ki-67 slide. As for the ground-truth grading
diagnosis, 2/9 is grade I, 5/9 is grade II, and 2/9 is grade III. We selected
three from the grade II cases for the tutorial purpose for the work session,
leaving the test set with two cases for each grade.

###  7.3. Task & Procedure

All sessions were conducted online because of the COVID-19 pandemic. We first
introduced the mission of the project and provided a detailed walkthrough of
the traditional interface and xPath with three pairs of H&E and Ki-67 slides
as an example. Participants used Microsoft Remote Desktop to interact with
both systems that ran on a remote server. After that, we ran a testing session
for the participants to grade one virtual case with the traditional interface,
and one-four other virtual cases with xPath with the time cost
logged202020Variation in the number of cases in xPath was caused by the
difference in the participants’ ability. Details of the arrangement are
reported in the supplementary material.. The order was counterbalanced across
participants. For each case, the time was counted from when participants first
clicked the WSI case until they reached the grading diagnosis. After
participants finished each case, we asked them to report their grading
diagnosis as well as their findings through a questionnaire adapted from the
College of American Pathologists (CAP) cancer protocol
template212121https://documents.cap.org/protocols/cp-cns-18protocol-4000.pdf.
In this session, we did not compare xPath with traditional optical microscopes
because of the difficulty of instrumentation and observation given the remote
situation. After participants had examined all the cases, we conducted a semi-
structured interview to elicit their responses to xPath’s perceived effort and
added value. The average duration of each work session was $\sim$70 minutes.
Each participant was paid with a $100 gift card as compensation for their
time.

###  7.4. Measurements

In this study, we collected participants’ grading decisions from the CAP
questionnaire and analyzed the time log. We also asked them to fill in a post-
study questionnaire (see Table 2) with seven-point Likert questions following
(Cai et al., 2019a; Jordan et al., 1996; Hart and Staveland, 1988). We tested
our hypotheses via the following measurements:

For H1, we compared the gradings reported by participants and the ‘ground-
truth’ gradings provided by a board-certified neuropathologist. We measured
the accuracy of both systems by calculating the error rates of gradings.

For H2a, we calculated the average time participants spent on each case using
xPath and the traditional interface. For H2b, we asked them to give both
systems ratings to the effort needed for grading (Table 2, W1), and the effect
of the system to reduce the workload (Table 2, W2) in the post-study
questionnaire.

H3a-c was evaluated by the post-study questionnaire. For H3a, we asked
participants to rate the comprehensiveness of xPath and the traditional
interface (Table 2, C1). For H3b, we asked them to rate the explainability of
xPath only since the traditional interface did not provide AI detections
(Table 2, E1). For H3c, we asked participants to rate the integrability of
both systems (Table 2, I1).

Apart from the hypotheses, we further investigated whether the participants
trusted xPath by asking them the following two questions: (i) How capable is
the system at helping grade meningiomas? (Table 2, T1), (ii) How confident do
you feel about the accuracy of your diagnoses using the system? (Table 2, T2).
We also studied whether the participants would like to use both systems in the
future (Table 2, F1), and also let the participants to rate the overall
preference of system 1 vs. system 2 (Table 2, F2).

##  8\. Results & Findings

In this section, we first discuss our initial research questions and
hypotheses. Then, we summarize the recurring themes that we have found in the
working sessions.

###  8.1. RQ1: Can xPath enable pathologists to achieve accurate diagnoses?

We summarize the CAP questionnaire responses from our participants and collect
12 grading decisions from the traditional interface, and 20 from xPath. We
then follow previous works on digital histology (Tschandl et al., 2020;
Steiner et al., 2020) and compare the difference between pathologists’
responses and the ground truth diagnoses justified by a board-certified
neuropathologist. In summary, with the traditional interface, participants
gave correct gradings for 7/12 cases, lower-than-ground-truth gradings for
4/12 cases, and higher-than-ground-truth grading for 1/12 case. In comparison,
using xPath, pathologists gave 17/20 cases correct gradings and lower-than-
ground-truth gradings in 3/20 diagnoses. Upon further analysis, we found that
all three errors that participants made with xPath were caused by their over-
reliance on the AI. In these cases specifically, participants spent their
majority of effort examining the evidence reported by xPath and missed the
false-negative features that xPath failed to detect —

> It’s just that I got caught up in looking at the boxes and I would forget
> that I should look at the entire case myself. (P4)

In sum, based on the data collected by the study, we report that pathologists
could make more accurate grading decisions with xPath in comparison to the
traditional interface (H1).

###  8.2. RQ2: Do pathologists work more efficiently with xPath?

Contrary to our hypothesis (H2a), participants spent an average of 7min13s
examining each case using xPath, which is 1min17s higher than the traditional
interface (ASAP). Our study suggests that pathologists tended to ($p$=0.050)
invest more time in xPath than the traditional interface. We believe this is
partly because xPath brings pathologists an extra workload to comprehend and
oversee the AI findings. In the traditional interface, our participants share
a similar workflow of examining the WSI — they first scanned the entire WSI in
low magnification, then prioritized studying one criterion (such as the brain
invasion or the mitotic count) to ascertain a probable diagnosis as quickly as
possible. They also checked Ki-67 slides to assist the diagnosis. In this
process, they collected evidence that accounts for a higher grade and
memorized them in their minds. Once they acquired enough evidence, they would
stop and make a grading decision. When using xPath, participants did not
abandon their standard workflow as in the traditional interface. Rather, on
top of their standard workflow, participants would perform the differential
diagnosis based on AI’s findings — they clicked through each piece of evidence
in xPath, justified it by registering into the WSI, and at times overrode AI
by clicking the approve/decline/declare-uncertain buttons. These extra steps
of interactions prolong participants’ workflow —

> System 2 (xPath) actually makes it longer because some of the images have
> sort of competing opinions — whether this is mitosis or not …So I’d better
> take a closer look at what the machine suggests. (P3)

Regarding the perceived effort (H2b), participants reported significantly less
effort (as shown in Table 2, W1, xPath: $\mu$=0.91, ASAP: $\mu$=3.67,
$p$=0.002) and a stronger effect on reducing the workload (Table 2, W2, xPath:
$\mu$=5.83, ASAP: $\mu$=2.17, $p$=0.002) while using xPath. Pathologists
mentioned that automating the process of finding small-scaled histological
features, especially mitosis, would save their time and effort —

> I spend a lot more time crawling around the slide in the high-power looking
> for mitosis (for system 1), which you don’t have to do as much in system 2
> (xPath). (P8)

Question | ASAP | xPath
---|---|---
C1: Rate the comprehensiveness of the system. | 2.83(1.27) | 5.75(0.75)
E1: Rate the explainability of the system. | N/A | 5.58(0.90)
I1: Rate the integrability of the system. | 4.17(1.70) | 5.91(1.08)
W1: Rate the effort needed to grade meningiomas when using the system. | 3.67(1.37) | 0.91(0.90)
W2: Rate the effect of the system on your workload to reach a diagnosis. | 2.17(1.40) | 5.83(1.03)
T1: How capable is the system at helping grade meningiomas? | N/A | 5.83(0.94)
T2: How confident do you feel about the accuracy of your diagnoses using the system? | N/A | 6.00 (0.95)
F1: If approved by the FDA, I would like to use this system in the future. | 3.75(1.76) | 6.42(0.79)
F2: Overall preference | 6.75(0.45)

Table 2. Participant response on the quantitative measurements of a
traditional interface (ASAP) and xPath with seven-point Likert scores. For the
rating questions (C1, E1, I1, W1, W2), 1=lowest and 7=highest. For question
T1, T2, F1, 1=very strongly disagree, 2=strongly disagree, 3=slightly
disagree, 4=neutral, …, and 7=very strongly agree. For question F2, score
1=totally prefer system 1 over system 2, 2=much more prefer system 1 over
system 2, 3=slightly prefer system 1 over system 2, 4=neutral, …, and 7=very
strongly prefer system 2 over system 1. Note that for question W1, a higher
rating indicates users perceive more effort while using the system. Question
E1, T1, T2 are not applicable to ASAP, since it does not have AI detections
for users.

###  8.3. RQ3: Overall, does xPath add value to pathologists’ existing
workflow?

For the comprehensiveness dimension (H3a), xPath received a significantly
higher rating than the traditional interface (Table 2, C1, xPath: $\mu$=5.75,
ASAP: $\mu$=2.83, $p$=0.001). Specifically, pathologists responded positively
that xPath provides sufficient information (i.e., criteria and evidence) to
assist the diagnosis —

> …it (xPath) kind of gives you a step-wise checklist to make sure that it’s
> the correct diagnosis, and also provides you what is most likely a
> diagnosis. (P11)

For the explainability dimension (H3b), xPath obtained an average rating of
5.58/7 (Table 2, E1). In general, pathologists could understand the logical
relationship between the evidence and the suggested grading (global
explainability). However, some (P1, P5) found it hard to interpret the
saliency map, especially for the cases where cues of attentions scattered
across the entire evidence (see Figure 11a) —

> For the heatmap …it is also a little bit confusing …it takes some time
> getting used to it and there are some false positives. (P1)

For the integrability dimension (H3c), pathologists gave overall higher scores
for xPath (Table 2, I1, xPath: $\mu$=5.91, ASAP: $\mu$=4.17, $p$=0.006).
Specifically, pathologists were able to perform diagnoses based on the xPath’s
AI findings similar to their workflow of collaborating with human trainees —

> It’s kind of like a first-year resident marking everything. (P1)

> I’m a cytology fellow, and cases are pre-screened for us. And essentially
> this is doing similarly. (P4)

For the trust dimension, participants responded positively to xPath’s
capability of helping to grade meningiomas (T1: $\mu$=5.83) and the accuracy
of the diagnoses while using the system (T2: $\mu$=6.00). However, some (P3,
P4, P5) pointed out that they might spend more time examining the WSI entirely
if more time had been granted —

> I just went to the areas that the system suggested. If I had more time, I
> would like to just go to all the areas, just to feel more comfortable that
> I’m not missing anything. (P5)

Last but not least, participants were more likely to use xPath than the
traditional interface (Table 2, F1, xPath: $\mu$=6.41, ASAP: $\mu$=3.75,
$p$=0.002). Overall, 9/12 of the participants “totally” preferred xPath over
the traditional interface, while 3/12 “much more” preferred xPath (Table 2,
F2).

However, it is noteworthy that this study is based on pathologists’
examination of WSIs, while pathologists use the optical microscope in their
daily practice. During the study, 7/12 of our participants expressed that they
preferred using an optical microscope with the glass slide vs. a digital
interface with the WSI — “…it’s much faster (in the microscope) than moving on
the computer …we would prefer to look at a real slide instead of using a scan
picture.” (P2). As such, further comparison between xPath and the optical
microscope is considered as future work.

###  8.4. Recurring Themes

Based on our observations of pathologists’ using xPath and the interview with
them, we discuss the following recurring themes that characterize how
pathologists interacted with xPath.

####  8.4.1. How pathologists use xPath’s multiple criteria: prioritizing one,
referring to others on-demand

We noted that pathologists tended to focus on a specific criterion. If that
criterion alone did not meet the bar of a diagnosis for a higher grade,
pathologists would use xPath to browse other criteria, looking for evidence of
a differential diagnosis, until they identify sufficient evidence to support
their hypothesis.

> I’m done. Because with the mitosis that high, you’re done, you don’t have to
> go through that stuff (other criteria). (P12)

However, some pathologists would also like to see other criteria and examine
the slide comprehensively —

> With the mitosis rate that high, you don’t actually need it (Ki-67) for the
> diagnosis. But I will have a look at it. (P1)

> I will just look at (other criteria) because I don’t want to grade by one
> single criterion (mitosis). (P3)

Such a relationship between criteria is analogous to ‘focus + context’ (Card,
1999) in information visualization — different pathologists might focus on a
few different criteria. Still, the other criteria are also important to serve
as context at their disposal to support an existing diagnosis or find an
alternative.

####  8.4.2. xPath’s top-down workflow with hierarchical explainable evidence
enables pathologists to navigate between high-level AI results & low-level WSI
details

One of the main reasons limiting the throughput of histological diagnosis is
that criteria like mitotic count have very small-scaled histological features.
As a result, pathologists have to switch to high power magnification to
examine such small features in detail. Given the high resolution of the WSI,
it is possible to ‘get lost’ in the narrow scope of HPF, resulting in a time-
consuming process to go through the entire WSI. With xPath, pathologists found
its hierarchical design and the provision of mid-level evidence (e.g., AI’s
ROI samples) the most helpful for diagnosis as it connects high-level findings
and low-level details —

> It (xPath) finds the best area to look at. …You can jump there, and if it is
> a grade III, then it is a grade III. You don’t have to look at other areas.
> (P6)

Furthermore, pathologists appreciated that xPath provided heatmap
visualizations to assist them in navigating the WSI out of the ROI samples —

> The heatmap is very useful to assist pathologists to go through the entire
> slide …which saves time and makes sure not missing anything. (P12)

####  8.4.3. xPath’s explainable design helps pathologists see what AI is
doing

We found xPath’s evidence-based justification of AI findings assisted
pathologists to relate AI-computed results with evidence, which added
explainability —

> System 2 (xPath) does find some evidence and assigns it to a particular
> observation that is related to the grading, so that it helps with
> explainability. (P3)

In xPath, the AI might make two kinds of mistakes that may incur potential
bias: (i) false positive, where AI mistakenly identifies negative areas as
positive for a given criterion; (ii) false negative, where the AI misses
positive areas corresponding to a criterion. We observed a number of false-
positive detections that confused some participants. We also found out that
the participants would rather deal with more false positives than false
negatives so that signs of more severe grade would not be missed —

> It’s better that it picks them up and gives me the opportunity to decline
> it. (P10)

Furthermore, although some participants found the saliency map hard to
interpret in some cases, the others used it to locate the cells that led to
AI’s grading —

> There were a couple of instances where it was a bit more difficult to figure
> out what it (the saliency map) was trying to point out to me. But for the
> majority of the time, I could tell which area they (the saliency maps) were
> trying to show me. (P9)

Further, with the aid of the saliency map, participants could understand AI’s
limitations and what might have misled the AI —

> You can see what this system counted as mitosis …the heatmap (the saliency
> map) helps to understand why AI chose this or that area, for example, I
> think AI chose neutrophils as mitotic figures in some areas. (P6)

####  8.4.4. How pathologists oversee xPath: incrementing human findings onto
justified AI results

Given the explainable evidence provided by xPath, it was straightforward for
pathologists to recognize and modify AI results when there was a disagreement.
Specifically, pathologists could oversee AI by clicking on the
approve/decline/declare-uncertain buttons or modifying AI results directly on
the criteria panel. If the overseen AI result were sufficient to conclude a
grading decision (e.g., seven mitoses in 10 HPFs, enough to make the case as
grade II (¿4), but still far from grade III (¡¡20)), they would stop examining
and report the grading. However, if the overseen AI result appeared to be
marginal (e.g., 19 mitoses in 10 HPFs, which is only one mitosis away from
upgrading the case to a grade III), pathologists would continue to search
based on the AI findings and add their new insights to grade —

> I count a total of number of five …adding previous 19 makes it 24 …this is
> grade III. (P2)

What’s more, for the cases where xPath did not actively report positive
detections, pathologists would examine the WSI manually as in a traditional
interface — that is, pathologists would use their experience to evaluate the
case further and make a grading decision.

##  9\. Limitations, Design Recommendations & Future Work

###  9.1. Limitations & Future Works

In this section, we discuss limitations of this work and outline the possible
directions of future works.

####  9.1.1. Increasing the scope of study

One limitation of this work is the materials used. Specifically, we used the
data from one institute to train and test the AI of xPath. This leaves the
performance of xPath’s AI questionable while applying the AI to the WSIs from
other institutes. This is because other institutes might use a different
staining process or a different type of scanner, causing a difference in the
domain/distribution (see Figure 10).

Figure 10. Mitoses from meningiomas (in x400), scanned by (a) the medical
center in this study and (b) a different medical center. The difference in
appearance is caused by the difference in processing procedures and scanners
used.

Another limitation is from the whole slide images. During the work sessions,
more than half of the participants expressed that they preferred using an
optical microscope with the glass slide vs. a digital interface with the WSI.
Remarkably, participants found it challenging to navigate a digital WSI, which
has also been described and discussed by Ruddle & Thomas et al. (Ruddle et
al., 2016). We believe such a difficulty in navigation is partly related to
the pathologists’ unfamiliarity with the traditional (digital) interface. As
such, we suggest future work also compare with the optical microscope aside
from the traditional digital interface. Therefore, with more data points
collected, we can validate xPath’s performance and the generalizability more
comprehensively.

####  9.1.2. Enabling adjusting the thresholds in the frontend

Currently, xPath does not support directly adjusting the threshold on the
frontend. In our user study, one participant mentioned that different
pathologists might have different thresholds to justify whether a piece of
evidence is positive —

> “I only call the characteristic mitoses …other pathologists might have
> different thresholds. (P7)

Further, dealing with false positives and false negatives is another issue
with the fixed-threshold scheme. From our study, we found out that
pathologists would prefer high-sensitivity results that include some false
positives rather than high-specificity results that have false negatives —

> I could have more faith if it could find all the candidates. And I could
> pretty easily click through and accept/reject, and know that it wasn’t
> missing anything. (P8)

Therefore, the system by default should be designed to err on the side of
caution, e.g., showing a wide range of ROIs despite that some are inevitably
false positives. Pathologists are fast in examining ROIs (and ruling out false
positives), whereas missing important features would come with a much higher
cost (e.g., delayed or missed treatment).

####  9.1.3. Improving the quality and granularity of explanations

In the study, we found a number of cases where the saliency maps failed to
explain the detection results and caused confusion to the users. As shown in
Figure 11, the failed saliency maps showed either scattered attention across
the evidence (Figure 11a), or concentrated attention at a wrong place (Figure
11b). Such errors can be explained as the attentions are reasoned from patch-
wise annotations rather than localized ones because the localized annotations
of positive findings are extremely labor-costly to obtain. The quality of the
saliency maps can be potentially improved with the increment of training data
for higher model generalization and the advent of the methodologies of
unsupervised attention reasoning (Arrieta et al., 2020).

Besides, knowing the location of a potential positive finding can be
insufficient for pathologists. Since the pathological imaging of tissues is
merely an approximation of the real condition, there can often exist
uncertainty in diagnosis even for well-trained pathologists. As such,
explaining why an area contains positive findings, e.g., a highlighted cell is
detected to stage as mitosis since its boundary is jagged, can be critical for
systems in the future. Such causality enables a system to imitate how
pathologists discuss with their peers, which can improve the collaboration
between a system and its users. Moreover, future work should also employ more
formal measurements (e.g., System Causability Scale (Holzinger et al., 2020))
to evaluate the quality of explanations.

Figure 11. Examples of failure explanation cases, where the saliency shows (a)
scattered attentions across the image or (b) misleading hot spots. The green
arrows point to the location of a mitosis figure marked by a human
pathologist.

###  9.2. Design Recommendations for Physician-AI Collaborative Systems

####  9.2.1. Showing the logical relationship amongst multiple types of
evidence at the top level

Carcinoma grading usually involves examining multiple criteria from multiple
data sources (e.g., H&E slides, IHC slides, FISH (fluorescence in situ
hybridization) test, patient’s health record). As such, one-size-fits-all AI
models are not sufficient. In practice, multiple AI models are employed to
locate different types of disease markers. To organize these AI-computed
results, medical AI systems (such as xPath) should seek to present the logical
relationship that connects these multiple criteria/features/sources of
information and update final results dynamically given any pathologists’ input
(e.g., acceptance or rejection of how AI computes each criterion). Such a
design is more likely to match the clinical practice of pathologists and cost
minimal extra learning when users onboard a system. It is noteworthy that the
‘multiple criteria’ design is not limited to this research but can also
generalize to other tasks in digital pathology, such as breast cancer grading
(Rakha et al., 2008).

####  9.2.2. Making AI’s finding traceable with hierarchically organized
evidence

There is a pressing need to deal with the transparency of a black-box model
and the traceability of the explanation evidence in the high-stake tasks
(e.g., medical diagnosis). As such, AI systems should provide local
explainability where each piece of low-level evidence is traceable. In xPath,
we employ the design of hierarchically traceable evidence for each criterion.
Such an organization forms an ‘evidence chain’ where each direct evidence is
accountable for the high-level system output. Similar intuitions can also be
applied to medical applications in a more general context, such as cancer
staging (Lydiatt et al., 2017) and cancer scoring (Humphrey, 2004), where the
evidence is accumulated to arrive at a diagnosis.

####  9.2.3. Employing a focus+context design towards presenting and/or
interacting with multiple criteria

Medical diagnosis involves accumulating evidence from multiple criteria — our
study observed that pathologists started with focusing on one criterion while
continuing to examine the others for a differential diagnosis. Thus, medical
AI systems should make multiple criteria available, and support the navigation
of such criteria following a ‘focus+context’ design (Card, 1999), which is
commonly used in information visualization. The major design goal is to strike
the dichotomy between juxtaposing the focused criterion with sufficient
contextual criteria and overwhelming the pathologists with too much
information. It is also possible for a system to, based on a patient’s prior
history and the pre-processing of their data, recommend a pathologist to start
focusing on specific criteria followed by examining some others as context.

##  10\. Conclusion

In this work, we identify three gaps of comprehensiveness, explainability, and
integrability that prevent AI from being adopted in a clinical setting for
pathologists. To close these gaps, we implement xPath with two key design
ingredients: (i) joint-analyses of multiple criteria and (ii) explanation by
hierarchically traceable evidence. To validate xPath, we conducted work
sessions with twelve medical professionals in pathology across three medical
centers. Our findings suggest that xPath can leverage AI to reduce
pathologists’ cognitive workload for meningioma grading. Meanwhile,
pathologists learned the tool and benefited from the design (e.g., working
with multiple criteria in parallel while drilling down to more evidence for
individual criteria), and made fewer mistakes. By observing pathologists’ use
of xPath with quantitative and qualitative feedback, we shed light on how
pathologists collaborate with AI and summarize design recommendations. We
believe this work can help future research on tool design for physician-AI
collaborative diagnosis.

## References

  * (1)
  * Abry et al. (2010) Ellen Abry, Ingrid Ø Thomassen, Øyvind O Salvesen, and Sverre H Torp. 2010.  The significance of Ki-67/MIB-1 labeling index in human meningiomas: a literature study.  _Pathology-Research and Practice_ 206, 12 (2010), 810–815.
  * Amershi et al. (2019) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019\.  Guidelines for human-AI interaction. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. ACM, 3.
  * Anari et al. (2010) Vahid Anari, Parvin Mahzouni, and Rasoul Amirfattahi. 2010\.  Computer-aided detection of proliferative cells and mitosis index in immunohistichemically images of meningioma. In _2010 6th Iranian Conference on Machine Vision and Image Processing_. IEEE, 1–5.
  * Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020\.  Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  _Information Fusion_ 58 (2020), 82–115.
  * Backer-Grøndahl et al. (2012) Thomas Backer-Grøndahl, Bjørnar H Moen, and Sverre H Torp. 2012\.  The histopathological spectrum of human meningiomas.  _International journal of clinical and experimental pathology_ 5, 3 (2012), 231.
  * Bankhead et al. (2017) Peter Bankhead, Maurice B Loughrey, José A Fernández, Yvonne Dombrowski, Darragh G McArt, Philip D Dunne, Stephen McQuaid, Ronan T Gray, Liam J Murray, Helen G Coleman, et al. 2017\.  QuPath: Open source software for digital pathology image analysis.  _Scientific reports_ 7, 1 (2017), 16878.
  * Bansal et al. (2019) Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019\.  Beyond accuracy: The role of mental models in human-AI team performance. In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , Vol. 7. 2–11.
  * Barker et al. (2016) Jocelyn Barker, Assaf Hoogi, Adrien Depeursinge, and Daniel L Rubin. 2016.  Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles.  _Medical image analysis_ 30 (2016), 60–71.
  * Bera et al. (2019) Kaustav Bera, Kurt A Schalper, David L Rimm, Vamsidhar Velcheti, and Anant Madabhushi. 2019\.  Artificial intelligence in digital pathology—new tools for diagnosis and precision oncology.  _Nature reviews Clinical oncology_ 16, 11 (2019), 703–715.
  * Brat et al. (2008) Daniel J Brat, Joseph E Parisi, Bette K Kleinschmidt-DeMasters, Anthony T Yachnis, Thomas J Montine, Philip J Boyer, Suzanne Z Powell, Richard A Prayson, and Roger E McLendon. 2008.  Surgical neuropathology update: a review of changes introduced by the WHO classification of tumours of the central nervous system.  _Archives of pathology & laboratory medicine_ 132, 6 (2008), 993–1007.
  * Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021.  To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (2021), 1–21.
  * Bulten et al. (2021) Wouter Bulten, Maschenka Balkenhol, Jean-Joël Awoumou Belinga, Américo Brilhante, Aslı Çakır, Lars Egevad, Martin Eklund, Xavier Farré, Katerina Geronatsiou, Vincent Molinié, et al. 2021\.  Artificial intelligence assistance significantly improves Gleason grading of prostate biopsies by pathologists.  _Modern Pathology_ 34, 3 (2021), 660–671.
  * Cai et al. (2019a) Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S Corrado, Martin C Stumpe, et al. 2019a.  Human-centered tools for coping with imperfect algorithms during medical decision-making. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Cai et al. (2019b) Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019b.  ” Hello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making.  _Proceedings of the ACM on Human-computer Interaction_ 3, CSCW (2019), 1–24.
  * Card (1999) Mackinlay Card. 1999\.  _Readings in information visualization: using vision to think_.  Morgan Kaufmann.
  * Carpenter et al. (2006) Anne E Carpenter, Thouis R Jones, Michael R Lamprecht, Colin Clarke, In Han Kang, Ola Friman, David A Guertin, Joo Han Chang, Robert A Lindquist, Jason Moffat, et al. 2006\.  CellProfiler: image analysis software for identifying and quantifying cell phenotypes.  _Genome biology_ 7, 10 (2006), R100.
  * Chattopadhay et al. (2018) Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. 2018\.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In _2018 IEEE winter conference on applications of computer vision (WACV)_. IEEE, 839–847.
  * Chen et al. (2018) Xiang’Anthony’ Chen, Ye Tao, Guanyun Wang, Runchang Kang, Tovi Grossman, Stelian Coros, and Scott E Hudson. 2018.  Forte: User-driven generative design. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Corvo et al. (2017) Alberto Corvo, Marc A van Driel, and Michel A Westenberg. 2017\.  PathoVA: A visual analytics tool for pathology diagnosis and reporting. In _2017 IEEE Workshop on Visual Analytics in Healthcare (VAHC)_. IEEE, 77–83.
  * Croskerry et al. (2017) Pat Croskerry, Karen Cosby, Mark L Graber, and Hardeep Singh. 2017.  _Diagnosis: Interpreting the shadows_.  CRC Press.
  * De et al. (2013) Soumya De, R Joe Stanley, Cheng Lu, Rodney Long, Sameer Antani, George Thoma, and Rosemary Zuna. 2013.  A fusion-based approach for uterine cervical cancer histology image classification.  _Computerized medical imaging and graphics_ 37, 7-8 (2013), 475–487.
  * Ghahremani et al. (2021) Parmida Ghahremani, Yanyun Li, Arie Kaufman, Rami Vanguri, Noah Greenwald, Michael Angelo, Travis J Hollmann, and Saad Nadeem. 2021\.  DeepLIIF: Deep Learning-Inferred Multiplex ImmunoFluorescence for IHC Image Quantification.  _bioRxiv_ (2021).
  * Graham et al. (2019) Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. 2019.  Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images.  _Medical Image Analysis_ (2019), 101563.
  * Gu et al. (2020) Hongyan Gu, Jingbin Huang, Lauren Hung, and Xiang’Anthony’ Chen. 2020.  Lessons learned from designing an AI-enabled diagnosis tool for pathologists.  _arXiv preprint arXiv:2006.12695_ (2020).
  * Gurda et al. (2014) Grzegorz T Gurda, Alexander S Baras, and Robert J Kurman. 2014.  Ki-67 index as an ancillary tool in the differential diagnosis of proliferative endometrial lesions with secretory change.  _International Journal of Gynecological Pathology_ 33, 2 (2014), 114–119.
  * Hart and Staveland (1988) Sandra G Hart and Lowell E Staveland. 1988.  Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research.  In _Advances in psychology_. Vol. 52. Elsevier, 139–183.
  * Hegde et al. (2019) Narayan Hegde, Jason D Hipp, Yun Liu, Michael Emmert-Buck, Emily Reif, Daniel Smilkov, Michael Terry, Carrie J Cai, Mahul B Amin, Craig H Mermel, et al. 2019\.  Similar image search for histopathology: SMILY.  _NPJ digital medicine_ 2, 1 (2019), 1–9.
  * Holzinger et al. (2020) Andreas Holzinger, André Carrington, and Heimo Müller. 2020.  Measuring the quality of explanations: the system causability scale (SCS).  _KI-Künstliche Intelligenz_ (2020), 1–6.
  * Holzinger et al. (2017) Andreas Holzinger, Bernd Malle, Peter Kieseberg, Peter M Roth, Heimo Müller, Robert Reihs, and Kurt Zatloukal. 2017.  Towards the augmented pathologist: Challenges of explainable-ai in digital pathology.  _arXiv preprint arXiv:1712.06657_ (2017).
  * Horvitz (1999) Eric Horvitz. 1999\.  Principles of mixed-initiative user interfaces. In _Proceedings of the SIGCHI conference on Human Factors in Computing Systems_. 159–166.
  * Huang and Chung (2018) Yongxiang Huang and Albert Chi-shing Chung. 2018.  Improving high resolution histology image classification with deep spatial fusion network.  In _Computational Pathology and Ophthalmic Medical Image Analysis_. Springer, 19–26.
  * Humphrey (2004) Peter A Humphrey. 2004\.  Gleason grading and prognostic factors in carcinoma of the prostate.  _Modern pathology_ 17, 3 (2004), 292–306.
  * Irshad (2013) Humayun Irshad. 2013\.  Automated mitosis detection in histopathology using morphological and multi-channel statistics features.  _Journal of pathology informatics_ 4 (2013).
  * Jordan et al. (1996) Patrick W Jordan, Bruce Thomas, Ian Lyall McClelland, and Bernard Weerdmeester. 1996.  _Usability evaluation in industry_.  CRC Press.
  * Kandula and Shaman (2019) Sasikiran Kandula and Jeffrey Shaman. 2019.  Reappraising the utility of Google flu trends.  _PLoS computational biology_ 15, 8 (2019), e1007258.
  * Komura and Ishikawa (2018) Daisuke Komura and Shumpei Ishikawa. 2018.  Machine learning methods for histopathological image analysis.  _Computational and structural biotechnology journal_ 16 (2018), 34–42.
  * Krueger et al. (2019) Robert Krueger, Johanna Beyer, Won-Dong Jang, Nam Wook Kim, Artem Sokolov, Peter K Sorger, and Hanspeter Pfister. 2019.  Facetto: Combining unsupervised and supervised learning for hierarchical phenotype analysis in multi-channel image data.  _IEEE transactions on visualization and computer graphics_ 26, 1 (2019), 227–237.
  * Lee et al. (2019) Bokyung Lee, Taeil Jin, Sung-Hee Lee, and Daniel Saakes. 2019\.  SmartManikin: Virtual Humans with Agency for Design Tools. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Litjens et al. (2018) Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob van de Loo, Rob Vogels, et al. 2018\.  1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset.  _GigaScience_ 7, 6 (2018), giy065.
  * Liu et al. (2017) Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Aleksei Timofeev, Philip Q Nelson, Greg S Corrado, et al. 2017\.  Detecting cancer metastases on gigapixel pathology images.  _arXiv preprint arXiv:1703.02442_ (2017).
  * Louis et al. (2007) David N Louis, Hiroko Ohgaki, Otmar D Wiestler, Webster K Cavenee, Peter C Burger, Anne Jouvet, Bernd W Scheithauer, and Paul Kleihues. 2007.  The 2007 WHO classification of tumours of the central nervous system.  _Acta neuropathologica_ 114, 2 (2007), 97–109.
  * Lu and Mandal (2013) Cheng Lu and Mrinal Mandal. 2013.  Toward automatic mitotic cell detection and segmentation in multispectral histopathological images.  _IEEE Journal of Biomedical and Health Informatics_ 18, 2 (2013), 594–605.
  * Lydiatt et al. (2017) William M Lydiatt, Snehal G Patel, Brian O’Sullivan, Margaret S Brandwein, John A Ridge, Jocelyn C Migliacci, Ashley M Loomis, and Jatin P Shah. 2017.  Head and neck cancers—major changes in the American Joint Committee on cancer eighth edition cancer staging manual.  _CA: a cancer journal for clinicians_ 67, 2 (2017), 122–137.
  * Martel et al. (2017) Anne L Martel, Dan Hosseinzadeh, Caglar Senaras, Yu Zhou, Azadeh Yazdanpanah, Rushin Shojaii, Emily S Patterson, Anant Madabhushi, and Metin N Gurcan. 2017.  An image analysis resource for cancer research: PIIP—pathology image informatics platform for visualization, analysis, and management.  _Cancer research_ 77, 21 (2017), e83–e86.
  * Middleton et al. (2016) B Middleton, DF Sittig, and A Wright. 2016.  Clinical decision support: a 25 year retrospective and a 25 year vision.  _Yearbook of medical informatics_ Suppl 1 (2016), S103.
  * Mishra et al. (2018) Rashika Mishra, Ovidiu Daescu, Patrick Leavey, Dinesh Rakheja, and Anita Sengupta. 2018.  Convolutional neural network for histopathological analysis of osteosarcoma.  _Journal of Computational Biology_ 25, 3 (2018), 313–325.
  * Palma et al. (1997) Lucio Palma, Paolo Celli, Carmine Franco, Luigi Cervoni, and Giampaolo Cantore. 1997.  Long-term prognosis for atypical and malignant meningiomas: a study of 71 surgical cases.  _Journal of neurosurgery_ 86, 5 (1997), 793–800.
  * Pantanowitz et al. (2011) Liron Pantanowitz, Paul N Valenstein, Andrew J Evans, Keith J Kaplan, John D Pfeifer, David C Wilbur, Laura C Collins, and Terence J Colgan. 2011\.  Review of the current state of whole slide imaging in pathology.  _Journal of pathology informatics_ 2 (2011).
  * Rakha et al. (2008) Emad A Rakha, Maysa E El-Sayed, Andrew HS Lee, Christopher W Elston, Matthew J Grainge, Zsolt Hodi, Roger W Blamey, and Ian O Ellis. 2008\.  Prognostic significance of Nottingham histologic grade in invasive breast carcinoma.  _Journal of clinical oncology_ 26, 19 (2008), 3153–3158.
  * Rakhlin et al. (2018) Alexander Rakhlin, Alexey Shvets, Vladimir Iglovikov, and Alexandr A Kalinin. 2018.  Deep convolutional neural networks for breast cancer histology image analysis. In _International Conference Image Analysis and Recognition_. Springer, 737–744.
  * Ruddle et al. (2016) Roy A Ruddle, Rhys G Thomas, Rebecca Randell, Philip Quirke, and Darren Treanor. 2016.  The design and evaluation of interfaces for navigating gigapixel images in digital pathology.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 23, 1 (2016), 1–29.
  * Saha et al. (2017) Monjoy Saha, Chandan Chakraborty, Indu Arun, Rosina Ahmed, and Sanjoy Chatterjee. 2017.  An advanced deep learning approach for Ki-67 stained hotspot detection and proliferation rate scoring for prognostic evaluation of breast cancer.  _Scientific reports_ 7, 1 (2017), 1–14.
  * Saltz et al. (2017) Joel Saltz, Ashish Sharma, Ganesh Iyer, Erich Bremer, Feiqiao Wang, Alina Jasniewski, Tammy DiPrima, Jonas S Almeida, Yi Gao, Tianhao Zhao, et al. 2017\.  A containerized software system for generation, management, and exploration of features from whole slide tissue images.  _Cancer research_ 77, 21 (2017), e79–e82.
  * Schneider et al. (2012) Caroline A Schneider, Wayne S Rasband, and Kevin W Eliceiri. 2012\.  NIH Image to ImageJ: 25 years of image analysis.  _Nature methods_ 9, 7 (2012), 671.
  * Shih and Kurman (1998) Ie-Ming Shih and Robert J Kurman. 1998.  Ki-67 labeling index in the differential diagnosis of exaggerated placental site, placental site trophoblastic tumor, and choriocarcinoma: a double immunohistochemical staining technique using Ki-67 and Mel-CAM antibodies.  _Human pathology_ 29, 1 (1998), 27–33.
  * Steiner et al. (2020) David F Steiner, Kunal Nagpal, Rory Sayres, Davis J Foote, Benjamin D Wedin, Adam Pearce, Carrie J Cai, Samantha R Winter, Matthew Symonds, Liron Yatziv, et al. 2020\.  Evaluation of the Use of Combined Artificial Intelligence and Pathologist Assessment to Review and Grade Prostate Biopsies.  _JAMA Network Open_ 3, 11 (2020), e2023267–e2023267.
  * Strickland (2019) Eliza Strickland. 2019\.  IBM Watson, heal thyself: How IBM overpromised and underdelivered on AI health care.  _IEEE Spectrum_ 56, 4 (2019), 24–31.
  * Ström et al. (2020) Peter Ström, Kimmo Kartasalo, Henrik Olsson, Leslie Solorzano, Brett Delahunt, Daniel M Berney, David G Bostwick, Andrew J Evans, David J Grignon, Peter A Humphrey, et al. 2020\.  Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study.  _The Lancet Oncology_ 21, 2 (2020), 222–232.
  * Tan and Le (2019) Mingxing Tan and Quoc Le. 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks. In _International Conference on Machine Learning_. PMLR, 6105–6114.
  * Teach and Shortliffe (1981) Randy L Teach and Edward H Shortliffe. 1981.  An analysis of physician attitudes regarding computer-based clinical consultation systems.  _Computers and Biomedical Research_ 14, 6 (1981), 542–558.
  * Tschandl et al. (2020) Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, et al. 2020\.  Human–computer collaboration for skin cancer recognition.  _Nature Medicine_ 26, 8 (2020), 1229–1234.
  * Walcott et al. (2013) Brian Patrick Walcott, Brian V Nahed, Priscilla K Brastianos, and Jay S Loeffler. 2013.  Radiation treatment for WHO grade II and III meningiomas.  _Frontiers in oncology_ 3 (2013), 227.
  * Wang et al. (2016) Dayong Wang, Aditya Khosla, Rishab Gargeya, Humayun Irshad, and Andrew H Beck. 2016.  Deep learning for identifying metastatic breast cancer.  _arXiv preprint arXiv:1606.05718_ (2016).
  * Wang et al. (2019) Shidan Wang, Donghan M Yang, Ruichen Rong, Xiaowei Zhan, Junya Fujimoto, Hongyu Liu, John Minna, Ignacio Ivan Wistuba, Yang Xie, and Guanghua Xiao. 2019\.  Artificial intelligence in lung cancer pathology image analysis.  _Cancers_ 11, 11 (2019), 1673.
  * Willett et al. (2018) Nora S Willett, Rubaiat Habib Kazi, Michael Chen, George Fitzmaurice, Adam Finkelstein, and Tovi Grossman. 2018.  A mixed-initiative interface for animating static pictures. In _Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology_. 649–661.
  * Xie et al. (2020) Yao Xie, Melody Chen, David Kao, Ge Gao, and Xiang’Anthony’ Chen. 2020.  CheXplain: Enabling Physicians to Explore and Understand Data-Driven, AI-Enabled Medical Imaging Analysis. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Xing et al. (2019) Fuyong Xing, Toby C Cornish, Tell Bennett, Debashis Ghosh, and Lin Yang. 2019.  Pixel-to-pixel learning with weak supervision for single-stage nucleus recognition in Ki67 images.  _IEEE Transactions on Biomedical Engineering_ 66, 11 (2019), 3088–3097.
  * Yang et al. (2020) Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020.  Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In _Proceedings of the 2020 chi conference on human factors in computing systems_. 1–13.
  * Yang et al. (2016) Qian Yang, John Zimmerman, Aaron Steinfeld, Lisa Carey, and James F Antaki. 2016.  Investigating the heart pump implant decision process: opportunities for decision support tools to help. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. 4477–4488.
  * Yap et al. (2015) Choon K Yap, Emarene M Kalaw, Malay Singh, Kian T Chong, Danilo M Giron, Chao-Hui Huang, Li Cheng, Yan N Law, and Hwee Kuan Lee. 2015.  Automated image based prominent nucleoli detection.  _Journal of pathology informatics_ 6 (2015).
  * Zhou et al. (2018) Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. 2018\.  Unet++: A nested u-net architecture for medical image segmentation.  In _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_. Springer, 3–11.