Amaya<EMAIL_ADDRESS>2 Luca<EMAIL_ADDRESS>Felice<EMAIL_ADDRESS>Costantino
<EMAIL_ADDRESS>Myles<EMAIL_ADDRESS>Michael
Barnesm.r.barnes@qmul.ac.uk1,2 Gregory<EMAIL_ADDRESS>Digital
Environment Research Institute
Queen Mary University of London
London, UK William Harvey Research Institute
Queen Mary University of London
London, UK Department of Electronic and Information Engineering
The Hong Kong Polytechnic University
Hong Kong MUSTANG

# Multi-Stain Self-Attention Graph Multiple Instance Learning Pipeline for
Histopathology Whole Slide Images

###### Abstract

Whole Slide Images (WSIs) present a challenging computer vision task due to
their gigapixel size and presence of numerous artefacts. Yet they are a
valuable resource for patient diagnosis and stratification, often representing
the gold standard for diagnostic tasks. Real-world clinical datasets tend to
come as sets of heterogeneous WSIs with labels present at the patient-level,
with poor to no annotations. Weakly supervised attention-based multiple
instance learning approaches have been developed in recent years to address
these challenges, but can fail to resolve both long and short-range
dependencies. Here we propose an end-to-end multi-stain self-attention graph
(MUSTANG) multiple instance learning pipeline, which is designed to solve a
weakly-supervised gigapixel multi-image classification task, where the label
is assigned at the patient-level, but no slide-level labels or region
annotations are available. The pipeline uses a self-attention based approach
by restricting the operations to a highly sparse $k$-Nearest Neighbour Graph
of embedded WSI patches based on the Euclidean distance. We show this approach
achieves a state-of-the-art F1-score/AUC of $0.89$/$0.92$, outperforming the
widely used CLAM model [Lu et al.(2021)Lu, Williamson, Chen, et al.]. Our
approach is highly modular and can easily be modified to suit different
clinical datasets, as it only requires a patient-level label without
annotations and accepts WSI sets of different sizes, as the graphs can be of
varying sizes and structures. The source code can be found at
https://github.com/AmayaGS/MUSTANG.

##  1 Introduction

In recent years, deep learning techniques have become the preferred
methodology for analysing medical images, particularly in histology image
classification [Ahmedt-Aristizabal et al.(2022)Ahmedt-Aristizabal, Armin,
Denman, Fookes, and Petersson]. Histopathology slide analysis is a time-
consuming task that requires a qualified histopathologist. However, the
digitisation of histopathology slides into Whole Slide Images (WSIs) has made
it possible to automate analysis using deep learning techniques for routine
workflows like subtyping, grading, and localising Regions of Interest (ROI).
This automation accelerates analysis, enhances inter-observer concordance by
offering a consistent benchmark, and can help overcome limitations due to
restricted access to qualified professionals [Lucchesi et al.(2020)Lucchesi,
Pontarini, Donati, et al.].

WSIs are large gigapixel multi-resolution image files, which present several
challenges such as their large size, high memory requirements, and
heterogeneity of artefacts (variations in staining intensity, scanner used,
pen marks, etc). Since WSIs can be as large as 100k $\times$ 100k pixels,
relevant information may be localised in small regions of the image (such as
micro-tumours or Ectopic Lymphoid Structures (ELS)), but can also depend on
interactions between far away parts (macro-tissue architecture) [Chen et
al.(2021)Chen, Lu, Weng, Chen, Williamson, Manz, Shady, and Mahmood]. Labels
are often only present at the slide or patient-level (represented by a set of
WSIs), which can make it very challenging to develop classification models
which accurately capture both micro and macro information.

##  2 Related Work

#### Multiple Instance Learning.

Given the gigapixel size and heterogeneity of WSIs, they present a challenging
computer vision task, with many successful deep learning methods rendered
computationally intractable. Most approaches use variations on the weakly
supervised Multiple Instance Learning (MIL) algorithm, where the gigapixel
image is divided into a set of smaller patches (e.g. $224\times 224$ pixels),
inheriting noisy slide/patient labels. Patches are then embedded into a
feature vector and classified at the slide/patient level using some form of
non-trainable global (e.g. max or mean) pooling on the set of instances [Chen
et al.(2022)Chen, Chen, Li, Chen, Trister, Krishnan, and Mahmood]. This weakly
supervised learning approach has yielded clinical grade performance, despite
low instance level accuracy [Ilse et al.(2018)Ilse, Tomczak, and Welling,
Coudray et al.(2018)Coudray, Ocampo, Sakellaropoulos, Narula, Snuderl, Fenyö,
Moreira, Razavian, and Tsirigos, Ghaffari Laleh et al.(2022)Ghaffari Laleh,
Muti, Loeffler, Echle, Saldanha, Mahmood, Lu, Trautwein, Langer, Dislich,
Buelow, Grabsch, Brenner, Chang-Claude, Alwers, Brinker, Khader, Truhn, Gaisa,
Boor, Hoffmeister, Schulz, and Kather]. Furthermore, these approaches fail at
capturing long-range dependencies and in learning which regions are most
relevant to the final classification.

#### Attention.

Recent methods have introduced linear attention pooling layers [Ilse et
al.(2018)Ilse, Tomczak, and Welling] and clustering-constrained attention
pooling [Lu et al.(2021)Lu, Williamson, Chen, et al.], which replaces non-
trainable global pooling by a trainable weighted average aggregation layer
where weights are given by a two-layered neural network [Ilse et
al.(2018)Ilse, Tomczak, and Welling]. This approach has demonstrated high
performance on publicly available datasets and crucially provides information
on key instances, permitting heatmap visualisation of the attention weights
associated with each image patch [Lu et al.(2021)Lu, Williamson, Chen, et
al.].

#### Self-attention.

In [Ilse et al.(2018)Ilse, Tomczak, and Welling] and [Lu et al.(2021)Lu,
Williamson, Chen, et al.] the attention pooling layer measures the importance
of a patch embedding given the whole sequence of patch embeddings. In
contrast, self-attention measures the importance of a given instance compared
to all other pairwise instances, which can provide a better understanding of
long-range dependencies [Vaswani et al.(2017)Vaswani, Shazeer, Parmar,
Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]. However, applying self-
attention to the full sequence of embedded patches is computationally
infeasible as the complexity grows quadratically to sequence length
$O(N^{2})$, rendering both runtime and memory usage problematic [Wang et
al.(2020)Wang, Li, Khabsa, Fang, and Ma]. Several approaches have been
developed to improve the computational complexity of self-attention, such as
relying on restricting the number of self-attention operations by inducing
sparsity or by reformulating the problem via matrix factorisation for example
[Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez,
Kaiser, and Polosukhin, Child et al.(2019)Child, Gray, Radford, and Sutskever,
Wang et al.(2020)Wang, Li, Khabsa, Fang, and Ma, Kitaev et al.(2020)Kitaev,
Kaiser, and Levskaya]. Sparse attention methods reduce the complexity by only
considering a subset of the computations in the $N\times N$ self-attention
matrix $P$. In matrix factorisation approaches, $P$ is assumed to be of low
rank, meaning not all elements of the matrix are linearly independent of each
other [Wang et al.(2020)Wang, Li, Khabsa, Fang, and Ma]. Intuitively, the idea
that WSIs patches are not independent of each other is reasonable, suggesting
that restricting self-attention operations via induced sparsity could achieve
good results. The question then becomes how to restrict operations to subsets
of relevant patches, while incurring minimal information loss. For example in
[Kitaev et al.(2020)Kitaev, Kaiser, and Levskaya] finding the nearest
neighbours of tokens in high-dimensional space is achieved via locality-
sensitive hashing.

#### Graph Neural Networks.

Graph Neural Networks (GNNs) are capable of learning hierarchical
representations of graphs by propagating node features through a series of
message-passing and aggregation operations. Given a graph over a set of nodes
$V$, during the $k$-th message-passing iteration, the embedding
$\mathbf{h}_{u}^{(k)}$ corresponding to each node $u\in V$ is updated
according to information aggregated from the neighbours of $u$, i.e.

$\displaystyle\mathbf{h}_{u}^{(k+1)}$
$\displaystyle=\operatorname{UPDATE}^{(k)}\left(\mathbf{h}_{u}^{(k)},\operatorname{AGGREGATE}^{(k)}\left(\left\\{\mathbf{h}_{v}^{(k)},\forall
v\in\mathcal{N}(u)\right\\}\right)\right)$ (1)
$\displaystyle=\operatorname{UPDATE}^{(k)}\left(\mathbf{h}_{u}^{(k)},\mathbf{m}_{\mathcal{N}(u)}^{(k)}\right)\,,$

where the neighbourhood $\mathcal{N}(u)$ is defined as the set of nodes that
share an edge with $u$, UPDATE and AGGREGATE are arbitrary differentiable
functions, and $\mathbf{m}_{\mathcal{N}(u)}$ is the “message” that is
aggregated from $\mathcal{N}(u)$. At each iteration, the AGGREGATE function
takes as input the set of embeddings of the nodes in $\mathcal{N}(u)$
[Hamilton()].

#### Graphs in Histopathology.

Although Convolutional Neural Networks (CNNs) have shown impressive
performance in histopathology analysis, they are less able to capture complex
neighbourhood information as they analyse local areas determined by the size
of the convolutional kernel [Ahmedt-Aristizabal et al.(2022)Ahmedt-
Aristizabal, Armin, Denman, Fookes, and Petersson]. GNNs can better exploit
these irregular relationships by preserving neighbouring information and are
thus ideally suited to representing relational information despite a graph
structure not being explicitly present in the data [Ahmedt-Aristizabal et
al.(2022)Ahmedt-Aristizabal, Armin, Denman, Fookes, and Petersson, Li et
al.(2022)Li, Huang, and Zitnik, Han et al.(2022)Han, Wang, Guo, Tang, and Wu,
Senior et al.(2023)Senior, Slabaugh, Yuan, and Rossi]. As presented in
[Ahmedt-Aristizabal et al.(2022)Ahmedt-Aristizabal, Armin, Denman, Fookes, and
Petersson], applications in histopathology can be divided into cell, patch or
tissue-level graphs, with both node and graph classification approaches being
employed. In particular, the MIL problem can be reformulated as a patch-graph
with graph-level classification, where GNN layers are combined with pooling
and readout layers to produce an end-to-end framework. Patch-graphs can be
constructed using features extracted from a WSI or a set of WSIs, and then
connected via edges. This has been done by connecting selected or spatially
adjacent patches, as well as “super-pixels” [Achanta et al.(2012)Achanta,
Shaji, Smith, Lucchi, Fua, and Süsstrunk, Adnan et al.(2020)Adnan, Kalra, and
Tizhoosh, Zheng et al.(2022a)Zheng, Gindra, Green, Burks, Betke, Beane, and
Kolachalama, Zheng et al.(2022b)Zheng, Jiang, Shi, Xie, Zhang, Luo, Hu, Sun,
Jiang, and Xue]. However, in real-world settings, information across multiple
WSIs is integrated by the pathologist for the purpose of disease diagnosis and
subtyping [Dwivedi et al.(2022)Dwivedi, Nofallah, Pouryahya, Iyer, Leidal,
Chung, Watkins, Billin, Myers, Abel, and Behrooz]. Thus, combining the
relevant information from multiple images more accurately reproduces the
pathology pipeline. This only complexifies the MIL problem, as the labels are
at the patient and not the slide level. Furthermore, large bodies of WSIs
coming from clinical trials have few to no annotations, yet still present a
valuable source of information. Histopathology deep learning frameworks
therefore need to extend to real-world datasets with little curation and very
noisy labels.

###  2.1 Contributions

  * •

Here we propose a novel end-to-end MUlti-STain self-AtteNtion Graph (MUSTANG)
multiple instance learning pipeline. MUSTANG solves a weakly-supervised
gigapixel multi-image classification task, where the label is assigned at the
patient-level (across the multiple images), but no slide-level labels or
region annotations are available.

  * •

The pipeline introduces a self-attention based approach on multiple full
gigapixel WSIs by restricting the attention operations to a highly sparse
$k$-Nearest Neighbour Graph ($k$-NNG) of embedded WSI patches based on
Euclidean distance.

  * •

Importantly, our approach does not require registration of WSIs, preprocessing
or labelling of ROIs, nor any feature engineering for the embedded feature
vectors, making it straightforward and flexible to apply to real-world
clinical datasets.

##  3 Methods

###  3.1 Dataset

#### Rheumatoid Arthritis.

MUSTANG is designed for the Rheumatoid Arthritis R4RA clinical trial dataset
[Rivellese et al.(2022)Rivellese, Surace, Goldmann, Sciacca, Çubuk, Giorli,
John, Nerviani, Fossati-Jimack, Thorborn, Ahmed, Prediletto, Church, Hudson,
Warren, McKeigue, Humby, Bombardieri, Barnes, Lewis, and Pitzalis, Humby et
al.(2021)Humby, Durez, Buch, Lewis, Rizvi, Rivellese, Nerviani, Giorli, Mahto,
Montecucco, Lauwerys, Ng, Ho, Bombardieri, Romão, Verschueren, Kelly,
Sainaghi, Gendi, Dasgupta, Cauli, Reynolds, Cañete, Moots, Taylor, Edwards,
Isaacs, Sasieni, Choy, Pitzalis, Thompson, Bugatti, Bellan, Congia, Holroyd,
Pratt, Fonseca, White, Warren, Peel, Hands, Fossati-Jimack, Hadfield,
Thorborn, Ramirez, and Celis]. 20 European centres recruited a total of 164
patients who underwent ultrasound-guided synovial biopsy of a clinically
active joint. The synovial tissue samples were then stained with Hematoxylin &
Eosin (H&E) and Immunochemistry (IHC) stains. H&E provides information on
overall tissue architecture and composition, while IHC identifies specific
immune cells, such as CD20+ B cells, CD68+ macrophages and CD138+ plasma cells
[Humby et al.(2021)Humby, Durez, Buch, Lewis, Rizvi, Rivellese, Nerviani,
Giorli, Mahto, Montecucco, Lauwerys, Ng, Ho, Bombardieri, Romão, Verschueren,
Kelly, Sainaghi, Gendi, Dasgupta, Cauli, Reynolds, Cañete, Moots, Taylor,
Edwards, Isaacs, Sasieni, Choy, Pitzalis, Thompson, Bugatti, Bellan, Congia,
Holroyd, Pratt, Fonseca, White, Warren, Peel, Hands, Fossati-Jimack, Hadfield,
Thorborn, Ramirez, and Celis]. Each dye contains complimentary information
about the underlying disease process. Pathologists then semi-quantitatively
assigned patients to one of three groups: Fibroid, Myeloid, and Lymphoid, each
corresponding to a disease subtype linked to drug response and patient
trajectory [Dennis et al.(2014)Dennis, Holweg, Kummerfeld, Choy, Setiadi,
Hackney, Haverty, Gilbert, Lin, Diehl, Fischer, Song, Musselman, Klearman,
Gabay, Kavanaugh, Endres, Fox, Martin, and Townsend, Lewis et al.(2019)Lewis,
Barnes, Blighe, Goldmann, Rana, Hackney, Ramamoorthi, John, Watson,
Kummerfeld, Hands, Riahi, Rocher-Ros, Rivellese, Humby, Kelly, Bombardieri,
Ng, DiCicco, van der Heijde, Landewé, van der Helm-van Mil, Cauli, McInnes,
Buckley, Choy, Taylor, Townsend, and Pitzalis]. The different stain types and
disease pathotypes can be seen in Figure 1. For the purpose of this study, the
pathotypes Fibroid and Myeloid are aggregated and compared to the Lymphoid
pathotype, as the latter more substantially differs in disease presentation,
trajectory, and drug response. Samples were scanned into WSIs with .ndpi
format with Hamamatsu digital scanners under 20x objectives. The dataset has a
total of 651 WSIs, with a variable number of WSIs per patient.

Figure 1: Rheumatoid Arthritis inflammatory pathotypes Fibroid, Myeloid &
Lymphoid based on semi-quantitative analysis of synovial tissue biopsies
stained with H&E, CD20+ B cells, CD68+ macrophages and IHC+ CD138 plasma cells
[Humby et al.(2021)Humby, Durez, Buch, Lewis, Rizvi, Rivellese, Nerviani,
Giorli, Mahto, Montecucco, Lauwerys, Ng, Ho, Bombardieri, Romão, Verschueren,
Kelly, Sainaghi, Gendi, Dasgupta, Cauli, Reynolds, Cañete, Moots, Taylor,
Edwards, Isaacs, Sasieni, Choy, Pitzalis, Thompson, Bugatti, Bellan, Congia,
Holroyd, Pratt, Fonseca, White, Warren, Peel, Hands, Fossati-Jimack, Hadfield,
Thorborn, Ramirez, and Celis].

###  3.2 MUSTANG

The MUSTANG pipeline, which is graphically represented in Figure 2 is composed
of:

  * •

A - Segmentation: A automated segmentation step, where UNet is used to segment
tissue areas on the WSIs. The user can use the trained weights provided on our
GitHub repository or use their own.

  * •

B - Patching: After segmentation, the tissue area is divided into patches at a
size chosen by the user, which can be overlapping or non-overlapping.

  * •

C - Feature extraction: Each image patch is passed through a VGG16 CNN feature
extractor and embedded into a [$1\times 1024$] feature vector. All feature
vectors from a given patient are aggregated into a matrix. The number of rows
in the matrix will vary as each patient has a variable set of WSIs, each with
their own dimensions.

  * •

D - $k$-Nearest-Neighbour Graph: The matrix of feature vectors of each patient
is used to create a sparse directed $k$-NNG using the Euclidean distance
metric, with a default of $k=5$. The attribute of each node corresponds to a
[$1\times 1024$] feature vector. This graph is used as input to the GNN.

  * •

E - Graph classification: The $k$-NNG is successively passed through four
Graph Attention Network layers (GAT) [Veličković et al.(2018)Veličković,
Cucurull, Casanova, Romero, Liò, and Bengio] and SAGPooling layers [Lee et
al.(2019)Lee, Lee, and Kang]. The SAGPooling readouts from each layer are
concatenated and passed through three MLP layers and finally classified.

  * •

F - Prediction: A pathotype or diagnosis prediction is obtained at the
patient-level.

Figure 2: MUSTANG pipeline. Schemas in E1 and E2 are reproduced from [Lee et
al.(2019)Lee, Lee, and Kang, Veličković et al.(2018)Veličković, Cucurull,
Casanova, Romero, Liò, and Bengio]

#### Segmentation and patch extraction.

From the 651 WSIs, a total 309,248 non-overlapping $224\times 224$ pixel
patches were extracted at 10x magnification from the tissue area segmented by
UNet [Gallagher-Syed et al.(2023)Gallagher-Syed, Khan, Rivellese, Pitzalis,
Lewis, Slabaugh, and Barnes]. The 10x magnification was chosen based on our
domain knowledge of RA as a compromise to show both the macro/micro-
architecture of the tissue and to reduce the number of patches for storage and
computation [Gallagher-Syed et al.(2023)Gallagher-Syed, Khan, Rivellese,
Pitzalis, Lewis, Slabaugh, and Barnes].

####  $k$-Nearest Neighbour Graph.

The $k$-NNG is a directed graph (digraph) in which node $p$ is connected by a
directed edge to node $q$, if $q$ is among the $k$ closest nodes to $p$. Here
we measure the distance between two nodes in terms of the Euclidean distance
between the corresponding embedded feature vectors. Notice that not all
connections are reciprocal because if node $p$ is $q$’s closest neighbour, it
does not follow that $q$ is also $p$’s closest neighbour. A digraph is weakly
connected if there exists a path between every pair of nodes in the underlying
undirected graph.

#### Graph Attention Network.

Graph Attention Networks (GATs), based on the self-attention mechanism
[Bahdanau et al.(2016)Bahdanau, Cho, and Bengio, Vaswani et al.(2017)Vaswani,
Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin], incorporate
masked self-attention layers into graph convolutions and use attention weights
to define a weighted sum of the neighbours:

$\mathbf{m}_{\mathcal{N}(u)}=\sum_{v\in\mathcal{N}(u)}\alpha_{u,v}\mathbf{h}_{v}\,,$
(2)

where $\alpha_{u,v}$ denotes the attention on neighbour $v\in\mathcal{N}(u)$
when aggregating information at node $u$. In the original GAT paper, the
attention weights are defined as:

$\alpha_{u,v}=\frac{\exp\left(\mathsf{LeakyReLU}\left(\mathbf{a}^{\top}\left[\mathbf{Wh}_{u}\oplus\mathbf{W}\mathbf{h}_{v}\right]\right)\right)}{\sum_{v^{\prime}\in\mathcal{N}(u)}\exp\left(\mathsf{LeakyReLU}\left(\mathbf{a}^{\top}\left[\mathbf{W}\mathbf{h}_{u}\oplus\mathbf{W}\mathbf{h}_{v^{\prime}}\right]\right)\right)},$
(3)

where $\mathbf{a}$ is a trainable attention vector, $\mathbf{W}$ is a
trainable matrix and $\oplus$ denotes the concatenation operation. To
stabilise the learning process multi-head attention can be used, where $m$
different attention heads are applied to compute mutually independent features
in parallel, and their features are then averaged.

#### Graph Attention Pooling.

Graph pooling is used to downsample a graph, reducing its size while seeking
to preserve topological information. Global pooling methods use max or mean
aggregation layers to pool all the representations of nodes in each layer,
which enables graphs of different structures to be processed but tends to lose
topological information. Similarly, hierarchical methods such as DiffPool
[Ying et al.(2019)Ying, You, Morris, Ren, Hamilton, and Leskovec] or gPool
[Gao et al.(2019)Gao, Chen, and Ji], which use a learnable vector to calculate
projection scores and select the top-ranked nodes, do not fully take into
account graph topology [Gao and Ji(2019), Lee et al.(2019)Lee, Lee, and Kang,
Cangea et al.(2018)Cangea, Veličković, Jovanović, Kipf, and Liò]. SAGPool [Lee
et al.(2019)Lee, Lee, and Kang] uses the GCN defined in [Kipf and
Welling(2017)] to calculate the self-attention scores $Z\in\mathbb{R}^{N\times
1}$ as follows:

$Z=\sigma\left(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}X\Theta_{att}\right)\,,$
(4)

where $\sigma$ is the activation function, $\tilde{A}\in\mathbb{R}^{N\times
N}$ is the adjacency matrix with self-connections,
$\tilde{D}\in\mathbb{R}^{N\times N}$ is the degree matrix of $\tilde{A}$,
$X\in\mathbb{R}^{N\times F}$ is the matrix of input features of the graph with
$N$ nodes and $F$-dimensional features, and
$\Theta_{att}\in\mathbb{R}^{F\times 1}$ is the only parameter of the SAGPool
layer [Lee et al.(2019)Lee, Lee, and Kang]. By utilizing graph convolution to
obtain self-attention scores, the result of the pooling is based on both graph
and topological features. The node selection method follows [Gao and Ji(2019),
Cangea et al.(2018)Cangea, Veličković, Jovanović, Kipf, and Liò, Knyazev et
al.(2019)Knyazev, Taylor, and Amer] by retaining a portion of nodes of the
input graph even when graphs of varying sizes and structures are input. The
pooling ratio $k\in(0,1]$ is a hyperparameter that determines the number of
nodes to keep. The top $\lceil kN\rceil$ nodes are selected based on the value
of $Z$, i.e.

$\mathrm{idx}=\operatorname{top}-\operatorname{rank}(Z,\lceil kN\rceil),\quad
Z_{\text{mask }}=Z_{\mathrm{idx}}\,,$ (5)

where $\operatorname{top}-\operatorname{rank}$ is the function that returns
the indices of the top $\lceil kN\rceil$ values, $idx$ is an indexing
operation, and $Z_{mask}$ is the feature attention mask. Finally, the readout
layer, adopted from [Cangea et al.(2018)Cangea, Veličković, Jovanović, Kipf,
and Liò], aggregates node features to make a fixed size representation. The
summarized output feature of the readout layer is

$s=\frac{1}{N}\sum_{i=1}^{N}x_{i}\oplus\max_{i=1}^{N}x_{i}\,,$ (6)

where $N$ is the number of nodes, $x_{i}$ id the feature vector of the $i$-th
node, and $\oplus$ denotes concatenation [Cangea et al.(2018)Cangea,
Veličković, Jovanović, Kipf, and Liò].

#### Benchmarking and ablation studies.

We benchmark our method against CLAM [Lu et al.(2021)Lu, Williamson, Chen, et
al.], which is a clustering-constrained gated attention based MIL method,
widely used within the histopathology community. We modify the framework
slightly to accommodate the patient WSI set’s DataLoader and use VGG16 instead
of ResNet50 for feature extraction. We also try several different GNN
architectures using combinations of GAT or GCN, with SAGPooling or TopKPooling
(GCN_SAG, CGN_TopK, GAT_SAG), to assess the importance of each component.
Finally, we look at F1-scores for different values of $k$ in the $k$-NNG.

#### Training schedule.

This is a binary classification task with weakly-supervised labels. To obtain
our results we use a 70/30 train/test split and train with the Adam optimizer
$\beta_{1}=0.9$, $\beta_{2}=0.98$ and $\epsilon=10^{-9}$. We use the default
parameters recommended for CLAM [Lu et al.(2021)Lu, Williamson, Chen, et al.]:
$lr_{CLAM}=0.0001$, with no dropout. We train MUSTANG with
$lr_{MUSTANG}=0.0001$, pooling ratio=$0.8$, attention heads=$2$. We train each
method for 50 epochs and keep the best F1-score, whilst checking the loss is
stable (i.e. that we are not underfitting). The training was prototyped
locally on a commercial workstation with NVidia GPU RTX3080 and trained with
QMUL’s Apocrita HPC facility on an NVidia A100 GPU supported by QMUL Research-
IT [King et al.(2017)King, Butcher, and Zalewski].

##  4 Experimental Results

#### Multi-stain WSIs.

In Table 1 we present the results obtained by both MUSTANG and CLAM on the
multi-stain R4RA Rheumatoid Arthritis test set. MUSTANG outperforms the
benchmark by $5$ percentage points for both F1-score and AUC and runtime on
inference is substantially similar to that of CLAM, despite having a larger
number of total parameters and relying on self-attention operations. In Figure
3, we show MUSTANG performs well at identifying both correct and true
positives (Sensitivity=$0.93$), an important consideration for healthcare.

| F1-score | AUC | Sens | Spec | Params [M] | Test runtime [min]
---|---|---|---|---|---|---
CLAM | 0.84 | 0.88 | 0.86 | 0.82 | 0.47 | 10
MUSTANG (ours) | 0.89 | 0.92 | 0.93 | 0.82 | 3.29 | 11
Table 1: Multi-stain F1-score, AUC, Sensitivity (Sens) and Specificity (Spec)
results, total parameter number, and test runtime results.

Figure 3: Area Under the ROC and Precision-Recall curves for both MUSTANG (full line) and CLAM (dotted line). | Single Stain
---|---
| CD138 | CD68 | CD20 | HE
CLAM | 0.85 | 0.87 | 0.88 | 0.76
MUSTANG (ours) | 0.89 | 0.89 | 0.87 | 0.78
Table 2: Single-stain F1-score results

#### Single-stain WSIs.

To understand if our model is able to outperform single-stain F1-score, we
propagate the patient WSIs set label to each slide and rerun the method on the
single-stain. The results are presented in Table 2. MUSTANG outperforms CLAM,
except for CD20+ staining, and obtains similar accuracy results to the multi-
stain problem with 89% F1-score for CD138 and CD68. H&E is the worst
performing stain in both cases, suggesting any IHC staining performed in the
clinic would be beneficial for patient pathotype assignment. CLAM performs
better on the single-stain problem than on the multi-stain one, confirming it
does not fully capture long-range dependencies as the patient matrix size
increases. MUSTANG does not show increased performance compared to the single-
stain problem, but successfully integrates information across stains,
identifying complex relations and spatial arrangements pertaining to disease
subtyping. This is valuable because we do not know a priori which stains
contain the most information about the disease pathotype, as most clinical
datasets have not received curation.

#### GNN model ablation.

In Table 3 we check which component parts of the GNN model provide the most
accuracy gain. From the high accuracy of the GAT/GCN + SAGPool models, it is
clear that the self-attention pooling topology preserving method is crucial in
aggregating graph topological information which preserves the relations
between nodes.

GNN Model | GAT + SAGPool | GAT + TopK | GCN + SAGPool | GCN + TopK
---|---|---|---|---
Multi Stain | 0.89 | 0.67 | 0.82 | 0.76
Table 3: GNN model ablation F1-score results

####  $k$-NNG ablation on $k$.

In Figure 4 we show the $k$-NNG layout for different values of $k$. Up to
$k=4$, the graph is not a weakly connected digraph, meaning that it has
isolated nodes and subgraphs. For message-passing purposes, we reason that the
initial graph should be weakly-connected in order to obtain better results. We
check this assumption running MUSTANG for several values of $k$ and present
the results in Table 4. The $k$-NNG with $k=5$ obtains the highest score, with
$k=1$ the lowest. $k=2$ already shows good accuracy, evidencing even highly
sparse poorly connected graphs suffice to extract relevant signals. Higher
values of $k$ also obtain good results, but a denser graphs comes at the cost
of higher memory requirements and the potential to over-smooth the signal,
suggesting that using a minimally weakly connected graph is a good strategy.
In Figure 5 we show how the $k$-NNG graph connectivity structure changes after
each GAT + SAGPool layer in the GNN model: after each layer the graph loses
structure, restricting message passing to increasingly many small subgraphs.
We posit the early layers aggregate macro-tissue topological information while
the later layers concentrate on micro-tissue information, suggesting that
graph disaggregation is not a phenomenon to avoid.

Figure 4: $k$-NNG layout for different values of $k$. From left to right, $k=1$, $k=2$, $k=3$, $k=4$ and $k=5$. Figure 5: $k$-NNG graph connectivity layout after each GAT + SAGPool layer, with pooling ratio of 0.5 $k$ | 1 | 2 | 3 | 4 | 5 | 10 | 20 | 50 | 100
---|---|---|---|---|---|---|---|---|---
Accuracy | 0.71 | 0.85 | 0.87 | 0.87 | 0.89 | 0.83 | 0.87 | 0.85 | 0.87
Table 4: $k$-NNG model $k$ ablation F1-score results

#### Graph connectivity.

We empirically check graph connectivity to see what each node is connecting to
in Figure 6. On the left, we show a set of WSIs for a single patient and on
the right the corresponding $k$-NNG ($k=5$) graph structure plotted using
Networkx spring-layout [Hagberg et al.(2008)Hagberg, Schult, and Swart], with
nodes coloured in function of their provenance. The force-directed spring
layout uses the Fruchterman-Reingold algorithm, where edges act as “springs”
and nodes repel each other, hence resulting in closely connected nodes
clustering together [Fruchterman and Reingold(1991)]. As expected nodes tend
to connect to other nodes in the same WSI, but there is a good degree of
mixing between WSIs, indicating information can flow between them.

Figure 6: On left a set of WSI corresponding to a patient and on right its
corresponding $k$-NNG ($k=5$) plotted using Networkx’s spring-layout.

#### Limitations.

Ideally we would apply MUSTANG to publicly available datasets for comparison
purposes. However, we are not aware of a public dataset with multi-stain WSI
sets labelled at the patient-level, despite this being a common histopathology
pipeline. We address this limitation by comparing our method against CLAM,
which has strong performance on a variety of public datasets. Another
limitation is that we have not yet developed a visualisation of heatmap
overlays indicating the regions of interest that the model attends to.
Visualisation is a crucial step for the translation of these tools into
clinical practice and which remains as future work.

#### Supplementary material.

Our supplementary material has additional results and analysis.

##  5 Conclusion

We introduced a novel end-to-end MUlti-STain self-AtteNtion Graph Multiple
Instance Learning pipeline, which we call MUSTANG. The pipeline employs a
self-attention based approach on multiple full gigapixel Whole Slide Images by
restricting the attention operations to a highly sparse $k$-NNG of embedded
WSI patches based on Euclidean distance. We show this approach achieves state-
of-the-art accuracy, outperforming the widely used CLAM model. This shows that
when data is highly correlated, applying self-attention operations on a very
sparse matrix is sufficient to integrate both long-range dependencies and
local behaviour. This approach is highly modular and can easily be modified to
suit different clinical datasets, as it only requires a patient-level label
without any WSI annotations and accepts WSI sets of different sizes, as the
graphs can be of varying sizes and structures.

## Acknowledgements

This work was supported with funding from the Wellcome Trust (grant no.
218584/Z/19/Z). The R4RA trial was funded by the Efficacy and Mechanism
Evaluation (EME) Programme, a partnership between the Medical Research Council
(MRC) and the National Institute for Health and Care Research (NIHR) (grant
no. 11/100/76). This work acknowledges the support of the National Institute
for Health Research Barts Biomedical Research Centre (NIHR203330).

## References

  * [Achanta et al.(2012)Achanta, Shaji, Smith, Lucchi, Fua, and Süsstrunk] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 34(11):2274–2282, November 2012.  ISSN 1939-3539.  10.1109/TPAMI.2012.120.
  * [Adnan et al.(2020)Adnan, Kalra, and Tizhoosh] Mohammed Adnan, Shivam Kalra, and Hamid R. Tizhoosh.  Representation Learning of Histopathology Images using Graph Neural Networks.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 4254–4261, June 2020.  10.1109/CVPRW50498.2020.00502.  ISSN: 2160-7516.
  * [Ahmedt-Aristizabal et al.(2022)Ahmedt-Aristizabal, Armin, Denman, Fookes, and Petersson] David Ahmedt-Aristizabal, Mohammad Ali Armin, Simon Denman, Clinton Fookes, and Lars Petersson.  A survey on graph-based deep learning for computational histopathology.  _Computerized Medical Imaging and Graphics_ , 95:102027, January 2022.  ISSN 0895-6111.  10.1016/j.compmedimag.2021.102027.  URL https://www.sciencedirect.com/science/article/pii/S0895611121001762.
  * [Bahdanau et al.(2016)Bahdanau, Cho, and Bengio] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural Machine Translation by Jointly Learning to Align and Translate.  Technical report, May 2016.  URL http://arxiv.org/abs/1409.0473.  arXiv:1409.0473 [cs, stat] type: article.
  * [Cangea et al.(2018)Cangea, Veličković, Jovanović, Kipf, and Liò] Cătălina Cangea, Petar Veličković, Nikola Jovanović, Thomas Kipf, and Pietro Liò.  Towards Sparse Hierarchical Graph Classifiers.  Technical report, November 2018.  URL http://arxiv.org/abs/1811.01287.  arXiv:1811.01287 [cs, stat] type: article.
  * [Chen et al.(2021)Chen, Lu, Weng, Chen, Williamson, Manz, Shady, and Mahmood] Richard J. Chen, Ming Y. Lu, Wei-Hung Weng, Tiffany Y. Chen, Drew F.K. Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood.  Multimodal co-attention transformer for survival prediction in gigapixel whole slide images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 4015–4025, October 2021.
  * [Chen et al.(2022)Chen, Chen, Li, Chen, Trister, Krishnan, and Mahmood] Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, and Faisal Mahmood.  Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16123–16134, June 2022.  10.1109/CVPR52688.2022.01567.  ISSN: 2575-7075.
  * [Child et al.(2019)Child, Gray, Radford, and Sutskever] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.  Generating Long Sequences with Sparse Transformers.  Technical report, April 2019.  URL http://arxiv.org/abs/1904.10509.  arXiv:1904.10509 [cs, stat] type: article.
  * [Coudray et al.(2018)Coudray, Ocampo, Sakellaropoulos, Narula, Snuderl, Fenyö, Moreira, Razavian, and Tsirigos] Nicolas Coudray, Paolo Santiago Ocampo, Theodore Sakellaropoulos, Navneet Narula, Matija Snuderl, David Fenyö, Andre L. Moreira, Narges Razavian, and Aristotelis Tsirigos.  Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning.  _Nature Medicine_ , 24(10):1559–1567, October 2018.  ISSN 1546-170X.  10.1038/s41591-018-0177-5.  URL https://www.nature.com/articles/s41591-018-0177-5.
  * [Dennis et al.(2014)Dennis, Holweg, Kummerfeld, Choy, Setiadi, Hackney, Haverty, Gilbert, Lin, Diehl, Fischer, Song, Musselman, Klearman, Gabay, Kavanaugh, Endres, Fox, Martin, and Townsend] Glynn Dennis, Cécile TJ Holweg, Sarah K Kummerfeld, David F Choy, A Francesca Setiadi, Jason A Hackney, Peter M Haverty, Houston Gilbert, Wei Yu Lin, Lauri Diehl, S Fischer, An Song, David Musselman, Micki Klearman, Cem Gabay, Arthur Kavanaugh, Judith Endres, David A Fox, Flavius Martin, and Michael J Townsend.  Synovial phenotypes in rheumatoid arthritis correlate with response to biologic therapeutics.  _Arthritis Research & Therapy_, 16(2):R90, 2014\.  ISSN 1478-6354.  10.1186/ar4555.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4060385/.
  * [Dwivedi et al.(2022)Dwivedi, Nofallah, Pouryahya, Iyer, Leidal, Chung, Watkins, Billin, Myers, Abel, and Behrooz] Chaitanya Dwivedi, Shima Nofallah, Maryam Pouryahya, Janani Iyer, Kenneth Leidal, Chuhan Chung, Timothy Watkins, Andrew Billin, Robert Myers, John Abel, and Ali Behrooz.  Multi Stain Graph Fusion for Multimodal Integration in Pathology.  pages 1835–1845, 2022.  URL https://openaccess.thecvf.com/content/CVPR2022W/CVMI/html/Dwivedi_Multi_Stain_Graph_Fusion_for_Multimodal_Integration_in_Pathology_CVPRW_2022_paper.html.
  * [Fruchterman and Reingold(1991)] Thomas M. J. Fruchterman and Edward M. Reingold.  Graph drawing by force-directed placement.  _Software: Practice and Experience_ , 21(11):1129–1164, 1991.  ISSN 1097-024X.  10.1002/spe.4380211102.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.4380211102.
  * [Gallagher-Syed et al.(2023)Gallagher-Syed, Khan, Rivellese, Pitzalis, Lewis, Slabaugh, and Barnes] Amaya Gallagher-Syed, Abbas Khan, Felice Rivellese, Costantino Pitzalis, Myles J. Lewis, Gregory Slabaugh, and Michael R. Barnes.  Automated segmentation of rheumatoid arthritis immunohistochemistry stained synovial tissue.  Technical report, September 2023.  URL http://arxiv.org/abs/2309.07255.  arXiv:2309.07255 [cs, eess, q-bio] type: article.
  * [Gao and Ji(2019)] Hongyang Gao and Shuiwang Ji.  Graph U-Nets.  pages 2083–2092. PMLR, May 2019.  URL https://proceedings.mlr.press/v97/gao19a.html.
  * [Gao et al.(2019)Gao, Chen, and Ji] Hongyang Gao, Yongjun Chen, and Shuiwang Ji.  Learning Graph Pooling and Hybrid Convolutional Operations for Text Representations.  In _The World Wide Web Conference_ , WWW ’19, pages 2743–2749, New York, NY, USA, May 2019. Association for Computing Machinery.  ISBN 9781450366748.  10.1145/3308558.3313395.  URL https://doi.org/10.1145/3308558.3313395.
  * [Ghaffari Laleh et al.(2022)Ghaffari Laleh, Muti, Loeffler, Echle, Saldanha, Mahmood, Lu, Trautwein, Langer, Dislich, Buelow, Grabsch, Brenner, Chang-Claude, Alwers, Brinker, Khader, Truhn, Gaisa, Boor, Hoffmeister, Schulz, and Kather] Narmin Ghaffari Laleh, Hannah Sophie Muti, Chiara Maria Lavinia Loeffler, Amelie Echle, Oliver Lester Saldanha, Faisal Mahmood, Ming Y. Lu, Christian Trautwein, Rupert Langer, Bastian Dislich, Roman D. Buelow, Heike Irmgard Grabsch, Hermann Brenner, Jenny Chang-Claude, Elizabeth Alwers, Titus J. Brinker, Firas Khader, Daniel Truhn, Nadine T. Gaisa, Peter Boor, Michael Hoffmeister, Volkmar Schulz, and Jakob Nikolas Kather.  Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology.  _Medical Image Analysis_ , 79:102474, July 2022.  ISSN 1361-8415.  10.1016/j.media.2022.102474.  URL https://www.sciencedirect.com/science/article/pii/S1361841522001219.
  * [Hagberg et al.(2008)Hagberg, Schult, and Swart] Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart.  Exploring network structure, dynamics, and function using networkx.  In Gaël Varoquaux, Travis Vaught, and Jarrod Millman, editors, _Proceedings of the 7th Python in Science Conference_ , pages 11 – 15, Pasadena, CA USA, 2008.
  * [Hamilton()] William L. Hamilton.  Graph representation learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , 14(3):1–159.
  * [Han et al.(2022)Han, Wang, Guo, Tang, and Wu] Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu.  Vision GNN: An image is worth graph of nodes.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=htM1WJZVB2I.
  * [Humby et al.(2021)Humby, Durez, Buch, Lewis, Rizvi, Rivellese, Nerviani, Giorli, Mahto, Montecucco, Lauwerys, Ng, Ho, Bombardieri, Romão, Verschueren, Kelly, Sainaghi, Gendi, Dasgupta, Cauli, Reynolds, Cañete, Moots, Taylor, Edwards, Isaacs, Sasieni, Choy, Pitzalis, Thompson, Bugatti, Bellan, Congia, Holroyd, Pratt, Fonseca, White, Warren, Peel, Hands, Fossati-Jimack, Hadfield, Thorborn, Ramirez, and Celis] Frances Humby, Patrick Durez, Maya H. Buch, Myles J. Lewis, Hasan Rizvi, Felice Rivellese, Alessandra Nerviani, Giovanni Giorli, Arti Mahto, Carlomaurizio Montecucco, Bernard Lauwerys, Nora Ng, Pauline Ho, Michele Bombardieri, Vasco C. Romão, Patrick Verschueren, Stephen Kelly, Pier Paolo Sainaghi, Nagui Gendi, Bhaskar Dasgupta, Alberto Cauli, Piero Reynolds, Juan D. Cañete, Robert Moots, Peter C. Taylor, Christopher J. Edwards, John Isaacs, Peter Sasieni, Ernest Choy, Costantino Pitzalis, Charlotte Thompson, Serena Bugatti, Mattia Bellan, Mattia Congia, Christopher Holroyd, Arthur Pratt, João Eurico Cabral da Fonseca, Laura White, Louise Warren, Joanna Peel, Rebecca Hands, Liliane Fossati-Jimack, Gaye Hadfield, Georgina Thorborn, Julio Ramirez, and Raquel Celis.  Rituximab versus tocilizumab in anti-TNF inadequate responder patients with rheumatoid arthritis (R4RA): 16-week outcomes of a stratified, biopsy-driven, multicentre, open-label, phase 4 randomised controlled trial.  _The Lancet_ , 397(10271):305–317, January 2021\.  ISSN 0140-6736, 1474-547X.  10.1016/S0140-6736(20)32341-2.  URL https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)32341-2/fulltext.
  * [Ilse et al.(2018)Ilse, Tomczak, and Welling] M. Ilse, J. Tomczak, and M. Welling.  Attention-based deep multiple instance learning.  In _Proc. 35th ICML_ , volume 80, pages 2127–2136, 2018.  URL https://proceedings.mlr.press/v80/ilse18a.html.
  * [King et al.(2017)King, Butcher, and Zalewski] Thomas King, Simon Butcher, and Lukasz Zalewski.  Apocrita - High Performance Computing Cluster for Queen Mary University of London.  March 2017.  URL https://zenodo.org/record/438045.
  * [Kipf and Welling(2017)] Thomas N. Kipf and Max Welling.  Semi-Supervised Classification with Graph Convolutional Networks.  Technical report, February 2017.  URL http://arxiv.org/abs/1609.02907.  arXiv:1609.02907 [cs, stat] type: article.
  * [Kitaev et al.(2020)Kitaev, Kaiser, and Levskaya] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The Efficient Transformer.  Technical report, February 2020.  URL http://arxiv.org/abs/2001.04451.  arXiv:2001.04451 [cs, stat] type: article.
  * [Knyazev et al.(2019)Knyazev, Taylor, and Amer] Boris Knyazev, Graham W Taylor, and Mohamed Amer.  Understanding Attention and Generalization in Graph Neural Networks.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html.
  * [Lee et al.(2019)Lee, Lee, and Kang] Junhyun Lee, Inyeop Lee, and Jaewoo Kang.  Self-attention graph pooling.  In _Proceedings of the 36th International Conference on Machine Learning_ , 09–15 Jun 2019.
  * [Lewis et al.(2019)Lewis, Barnes, Blighe, Goldmann, Rana, Hackney, Ramamoorthi, John, Watson, Kummerfeld, Hands, Riahi, Rocher-Ros, Rivellese, Humby, Kelly, Bombardieri, Ng, DiCicco, van der Heijde, Landewé, van der Helm-van Mil, Cauli, McInnes, Buckley, Choy, Taylor, Townsend, and Pitzalis] Myles J. Lewis, Michael R. Barnes, Kevin Blighe, Katriona Goldmann, Sharmila Rana, Jason A. Hackney, Nandhini Ramamoorthi, Christopher R. John, David S. Watson, Sarah K. Kummerfeld, Rebecca Hands, Sudeh Riahi, Vidalba Rocher-Ros, Felice Rivellese, Frances Humby, Stephen Kelly, Michele Bombardieri, Nora Ng, Maria DiCicco, Désirée van der Heijde, Robert Landewé, Annette van der Helm-van Mil, Alberto Cauli, Iain B. McInnes, Christopher D. Buckley, Ernest Choy, Peter C. Taylor, Michael J. Townsend, and Costantino Pitzalis.  Molecular Portraits of Early Rheumatoid Arthritis Identify Clinical and Treatment Response Phenotypes.  _Cell Reports_ , 28(9):2455–2470.e5, August 2019\.  ISSN 2211-1247.  10.1016/j.celrep.2019.07.091.  URL https://www.sciencedirect.com/science/article/pii/S2211124719310071.
  * [Li et al.(2022)Li, Huang, and Zitnik] Michelle M. Li, Kexin Huang, and Marinka Zitnik.  Graph Representation Learning in Biomedicine.  Technical report, June 2022.  URL http://arxiv.org/abs/2104.04883.  arXiv:2104.04883 [cs, q-bio] type: article.
  * [Lu et al.(2021)Lu, Williamson, Chen, et al.] M. Y. Lu, D. F. K. Williamson, T. Y. Chen, et al.  Data-efficient and weakly supervised computational pathology on whole-slide images.  _Nat. Biomed. Eng_ , 5(6):555–570, June 2021\.  ISSN 2157-846X.  10.1038/s41551-020-00682-w.  URL https://www.nature.com/articles/s41551-020-00682-w.
  * [Lucchesi et al.(2020)Lucchesi, Pontarini, Donati, et al.] D. Lucchesi, E. Pontarini, V. Donati, et al.  The use of digital image analysis in the histological assessment of Sjögren’s syndrome salivary glands improves inter-rater agreement and facilitates multicentre data harmonisation.  _Clin. and Exp. Rheumatology_ , 38 Suppl 126(4):180–188, 2020.  ISSN 0392-856X.
  * [Rivellese et al.(2022)Rivellese, Surace, Goldmann, Sciacca, Çubuk, Giorli, John, Nerviani, Fossati-Jimack, Thorborn, Ahmed, Prediletto, Church, Hudson, Warren, McKeigue, Humby, Bombardieri, Barnes, Lewis, and Pitzalis] Felice Rivellese, Anna E. A. Surace, Katriona Goldmann, Elisabetta Sciacca, Cankut Çubuk, Giovanni Giorli, Christopher R. John, Alessandra Nerviani, Liliane Fossati-Jimack, Georgina Thorborn, Manzoor Ahmed, Edoardo Prediletto, Sarah E. Church, Briana M. Hudson, Sarah E. Warren, Paul M. McKeigue, Frances Humby, Michele Bombardieri, Michael R. Barnes, Myles J. Lewis, and Costantino Pitzalis.  Rituximab versus tocilizumab in rheumatoid arthritis: synovial biopsy-based biomarker analysis of the phase 4 R4RA randomized trial.  _Nature Medicine_ , 28(6):1256–1268, June 2022\.  ISSN 1546-170X.  10.1038/s41591-022-01789-0.  URL https://www.nature.com/articles/s41591-022-01789-0.
  * [Senior et al.(2023)Senior, Slabaugh, Yuan, and Rossi] Henry Senior, Gregory Slabaugh, Shanxin Yuan, and Luca Rossi.  Graph neural networks in vision-language image understanding: A survey.  _arXiv preprint arXiv:2303.03761_ , 2023.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * [Veličković et al.(2018)Veličković, Cucurull, Casanova, Romero, Liò, and Bengio] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph Attention Networks.  Technical report, February 2018.  URL http://arxiv.org/abs/1710.10903.  arXiv:1710.10903 [cs, stat] type: article.
  * [Wang et al.(2020)Wang, Li, Khabsa, Fang, and Ma] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-Attention with Linear Complexity.  Technical report, June 2020.  URL http://arxiv.org/abs/2006.04768.  arXiv:2006.04768 [cs, stat] version: 1 type: article.
  * [Ying et al.(2019)Ying, You, Morris, Ren, Hamilton, and Leskovec] Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec.  Hierarchical Graph Representation Learning with Differentiable Pooling.  Technical report, February 2019.  URL http://arxiv.org/abs/1806.08804.  arXiv:1806.08804 [cs, stat] type: article.
  * [Zheng et al.(2022a)Zheng, Gindra, Green, Burks, Betke, Beane, and Kolachalama] Yi Zheng, Rushin H. Gindra, Emily J. Green, Eric J. Burks, Margrit Betke, Jennifer E. Beane, and Vijaya B. Kolachalama.  A Graph-Transformer for Whole Slide Image Classification.  _IEEE Transactions on Medical Imaging_ , 41(11):3003–3015, November 2022a.  ISSN 1558-254X.  10.1109/TMI.2022.3176598.
  * [Zheng et al.(2022b)Zheng, Jiang, Shi, Xie, Zhang, Luo, Hu, Sun, Jiang, and Xue] Yushan Zheng, Zhiguo Jiang, Jun Shi, Fengying Xie, Haopeng Zhang, Wei Luo, Dingyi Hu, Shujiao Sun, Zhongmin Jiang, and Chenghai Xue.  Encoding histopathology whole slide images with location-aware graphs for diagnostically relevant regions retrieval.  _Medical Image Analysis_ , 76:102308, February 2022b.  ISSN 1361-8415.  10.1016/j.media.2021.102308.  URL https://www.sciencedirect.com/science/article/pii/S1361841521003534.


# Why Data Science Projects Fail
Understanding of Data Science Project failure and reasoning

Balaram Panda  Software Engineering
University of Auckland
Auckland, New Zealand
<EMAIL_ADDRESS>
###### Abstract

Data Science is a modern Data Intelligence practice, which is the core of many
businesses and helps businesses build smart strategies around to deal with
businesses challenges more efficiently. Data Science practice also helps in
automating business processes using the algorithm, and it has several other
benefits, which also deliver in a non-profitable framework. In regards to data
science, three key components primarily influence the effective outcome of a
data science project. Those are

  1. 1.

Availability of Data

  2. 2.

Algorithm

  3. 3.

Processing power / infrastructure

In today’s technology world, there is no limitation on data as well as
processing power, and we have a much more efficient algorithm to produce the
desired output. In spite of the success of Data Science projects, many Data
Science projects still fail and are unable to produce the desired outcome. In
this paper, we have explored the bottleneck of Data Science projects and
provided some recommendations to make data science projects more successful.
Standard Data Science project development life-cycle CRISP-DM [1] is old in
this agile development[31] world, and but most Data Science practices still
follow CRISP-DM. In general, Data Scientist analyses scenarios where a
predictive model or machine learning model might fail. But this study is to
analyze when and why the Data Science project fails despite an excellent
model. Data Science is a diverse field. It needs technical as well as business
knowledge to deliver a project. Hence, to understand why the Data Science
project fails, we need to understand challenges from the business side and
technical side.

  1. 1.

Technical perspective

  2. 2.

Business leader perspective or stakeholder perspective

Also, it has been observed that the success of the Data science project
depends on business domain. Example market propensity model, a strategic
market campaign method, is more successful in any retail use case than a fraud
analytic model in the banking fraud domain. So domain agnostic framework was
implemented in this research to make this research independent of the business
domain.

###### Index Terms:

Software Engineering, Data Science, Machine Learning, CRISP-DM

##  I Introduction

Data Science is comparable to applied research. Often in research, there are
lots of repetitive processes to be followed until it generates the desired
outcome. Similarly, data science processes are often repetitive until they
meet the business objectives. Those business objectives are set during the
repeated Business Understanding and Data Understanding phase of CRISP-DM.
CRISP-DM developed during 90’s, and since then, components of Data Science
have evolved a lot in regards to the way data is collected, volume, velocity,
and variety of the information being analyzed(big data [24]), processing
methodology, algorithms, and outcome delivery. Hence it is crucial to
understand in detail the current day’s Data Science challenges, deep dive into
the reasoning of failures, and find out the best practice. Data Science is an
evolving field, and lots have been changed since last decade in regards to
various keys areas, such as data collection, big data processing on cloud
and/or parallel or distributed processing framework, evolution of new
algorithm, data visualization, deployment and productionizing Data Science
solutions as a digital product etc. There are several challenges in those key
areas; as an example, because of big data, the Data Science project collects
lots of irrelevant and uncleaned data. Sometimes the intermediate data source
doesn’t have the correct business logic, which is mapped to the business
objective of the data science project. “big data brings lots of “big errors”
in data quality and data usage, which cannot be used as a substitute for sound
research design and solid theories.” [4] Data quality is one of the common
challenges in the Data Science Project. Another major challenge is domain
understanding, and many times domain understanding is a bottleneck for Data
Science project success. Also, in many cases, data science issues are seen
differently and vary from one domain to another, stakeholder view and data
scientist view. As mentioned by Dr.Om Deshmukh [9] ” Data Science project
success also depends on what stakeholder knows about the expected outcome of
the data science project and how effectively stakeholder expectations being
managed though effectivthroughunication.” Data Science projects struggles with
these types of challenges and more other challenges in industry settings, but
not enough research has been done in this space to highlight those problems
and their possible solutions. For better quality research in this domain,
years of experience in the Data Science field and role/designation level
diversity in data science projects such as key stakeholders and data
scientists have been included.

We prepared a list of questions and interviewed experienced Data Scientists,
Senior Leader who actively manage data science and data analytic team, and
stakeholders part of data science projects, in a motivation to understand the
perception on data science project failure. Then did the thematic analysis to
analyze and draw conclusions from data. Then came up with the most popular
theme by thoroughly analyzing the discussion.

In this research, we have discovered and highlighted the key challenging areas
where Data Science practice and Data scientists should focus. Also, some
suggestions on tackling those challenges. Also, we came up with a state of art
framework to tackle those challenges in efficient ways.

##  II Research Methodology

###  II-A Research methodology

In this research Interview based semi-structured Qualitative research
methodology [8] have been used.

###  II-B Interview Design

In this research the objective is also to capture the diversity in regards to
the people involved in a data science project such as Data Scientist and
business owner and/or data science capability leader and/or stakeholder. Hence
for better outcome, two types of interview questions have prepared

  1.     1.       1. Group A:

Data Scientist with some years of experience working in Data Science project

      2. Group B:

Business owner / leader / stakeholder

The interviews had done via zoom software. During the interview, answers and
key points were captured. Also, interview voice recording has been stored in
google drive for reference.

###  II-C Candidate selection for interview and Sample Diversity

Interview candidates contacted via reference and Linkedin(professional
networking website). Manually reviewed their background and put them into the
Group A or Group B Category based on their experience. Some candidates were
also selected through reference. While selecting a reference candidate,
Snowball technique [5] had been used to select the reference candidate.
Snowball helped to capture unbiased interviews candidate samples. During the
selection process, candidates domain experience deiversity such as Banking and
finance, Retail, Software Product, IT Services, Telecommunication etc. had
been considered. In this research the objective also is to add domain
diversity so that Data Science problems can be viewed from a normalized and
generic approach. Also to capture more diversity candidate selected from
various geography such as India, USA, Canada in addition to New Zeland.

TABLE I: List of Interview Candidates ID |  | Year Of
---
Experience
| Group A or
---
Group B
| Business
---
Domain
| Working
---
Geography
C1 | 5 + | Group A |  | IT and
---
Consulting
USA
C2 | 10 + | Group B |  | Banking and
---
Finance
NZ
C3 | 10 + | Group A |  | Software
---
Product
India
C4 | 1 + | Group A | Research | NZ
C5 | 5 + | Group B |  | Software
---
Product
Canada
C6 | 10 + | Group B |  | IT Service and
---
Consulting
India
C7 | 10 + | Group B |  | Banking and
---
Finance
NZ
C8 | 5 + | Group A |  | Software
---
Product
USA
C9 | 10 + | Group B | Consulting | NZ

###  II-D Preparation of research Questions

In first iteration a list of questions has been prepared from prior knowledge
and after going through literature review papers [10] [11] [12] Then questions
are discussed with the research supervisor. Based on the discussion few
questions are added, and a few are modified. A final list of questions has
been prepared and shared with the supervisor in the second iteration. Then
after 2nd interview, 2 more questions were added to make the interview more
target-oriented.

  1.     1.       1. Group A:

Data Scientist with some years of experience working in Data Science project

      2. Group B:

Business owner / leader / stakeholder

###  II-E Research Questions

  1. 1.

What are the key reasons for a Data Science project success as per your
experience

  2. 2.

What are key reasons for Data Science project failure as per your experience

  3. 3.

What steps Data Science should follow to match user expectations? Can you
refer to any specific project and explain it in detail?

  4. 4.

How stakeholder can contribute towards Data Science project success, and how
important is it?

  5. 5.

How value measured in the Data Science project and who are involved and what
are the challenges? Any example based on a specific project?

  6. 6.

Do you think infrastructure is a challenge for any data science project?

  7. 7.

How important Data Quality is for Data Science projects?

  8. 8.

What are the hindrances to maintaining data quality?

  9. 9.

What are the solutions to overcome those hindrances?

  10. 10.

What testing approaches do you take to test the model based on business user
expectations?

  11. 11.

Some detailed discussion on failure mentioned in open ended. Like how to solve
model explainability? Is there any approach you followed, or can you suggest
from your experience how to maintain model explainability and without
compromising the model accuracy?

  12. 12.

Does the level of competencies affect project outcome and how to solve that?

  13. 13.

How often do Data Science projects productionize and the pitfalls you
experienced during productionization ?

  14. 14.

Many Data Science projects help automate the existing process; what are the
challenges you face during knowledge elicitation from business users in those
cases?

  15. 15.

How challenging is change management in Data Science?

  16. 16.

There are limitations of Data usability due to governance restriction? How
often do you see this as a limitation, and how do you solve these?

In Group A, questions from 1 to 6 are open ended questions and 6 to 16 are
focused questions.

  1. 1.

What are the essential things Data Scientists must do to make a Data Science
project successful?

  2. 2.

What is your view on Stakeholder’s contribution to a Data Science project?

  3. 3.

What are the critical gaps between a successful and unsuccessful Data Science
Project from stakeholder/business owner’s perspective?

  4. 4.

What improvement do you want to make towards the traditional Data Science
approach to make it more valuable for business?

  5. 5.

How important change management is for a Data Science Project? Any suggestions
to make change management successful?

  6. 6.

Data Science projects have lots of intangible benefits; how do you create
value metrics for Data Science projects? How often are you able to achieve
that?

  7. 7.

Do you follow any ROI metrics for Data Science projects? (question 6
extension)

  8. 8.

How do you define a Data Science project different from a Business
Intelligence project? (Optional question)

###  II-F Interview Process and Data Collection

Interviewer contacted via Linked-in and via emails. Once they confirmed their
participation, the consent document was shared, and formal consent approval
was taken before the interview. Interviews are done via Zoom and in-person
meetings. During the interview, answers and critical points has been captured.
Also, interview voice recording has been stored in google drive for reference.
Only the question-answer part of the interview has been recorded. In an
average interview, only the question-answer part lasted for 30 minutes. For
the introduction, including research motivation, we spent around 8 to 10
minutes in each interview.

###  II-G Data Understanding and Analysis

Thematic analysis[6] has been used to analysis the data. Thematic analytic is
a popular method for qualitative research. As suggested by Braun and Clark
[30] there are 6 steps of thematic analyses as follows

  1.     1.       1. Step 1:

Become familiar with the data

      2. Step 2:

Generate initial codes

      3. Step 3:

Search for themes

      4. Step 4:

Review themes

      5. Step 5:

Define themes

      6. Step 6:

Write-up

As the questions were designed before the interviews and interview, we
familiarised ourselves with the data during the interview. Also, note has been
taken during the interview. The code generation to theme generation is
described in the following Theme Generation section and the write-up step
mentioned in the result section.

###  II-H Theme Generation

There are various orientations in thematic analytic mentioned in [26, 27]. In
this research, we have used a combination of the Inductive and Deductive
approaches. We have prior knowledge from experience and literature review;
based on that, questions have been prepared on the most common issues that
have been known so far. After the interview, get to know more new issues that
the Data Science project is experiencing and solutions around how to tackle
that. After the interview, we had iterate over the recordings, list down all
the key issues that experts mentioned in the interview, and mapped that to the
questions. Then cluster those issues together to identify the common themes in
those issues. In addition to the theme, we also got solutions and workaround
to tackle those issues.

##  III Results

TABLE II: Resulted Themes Themes |  | Questions
---
where themes
were found
| Key concerns
---
for
Data Sciecne
Project Success
| Effective Stakeholder
---
Managment
AQ4,AQ10,BQ1,BQ2 | Yes
| Clarity in Business
---
problem understanding
AQ1,BQ6,AQ2 | Yes
Data quality Issues | AQ7, AQ2, AQ9 | Yes
| Model deployment
---
and productionzaiton
AQ3,AQ6,AQ13,BQ3 | Yes
| Department level
---
intigration while
productionizing the model
BQ3,BQ5 | Yes
Change Managment | AQ15, BQ5 | No
Data governance road blocker | AQ17 | No

###  III-A Effective Stakeholder Management:

Stakeholders are the people representing a department or owning a business
product or a group of people who benefit from a project’s success. In Data
Science, Stakeholder can be business process owner or product owner or end-
user who is going to use the Data science outcome or product. Many
participants mentioned how crucial it is to communicate and manage Stakeholder
effectively. C1 also mentioned that data scientists need to communicate well
across different project stakeholders like project managers, business
stakeholders, and data engineers. ”Stakeholders should share equal
responsibility as data scientist to make the project successful” - C1. But
often, stakeholders may not be available all the time as they have other
priorities. Also its often found the there is a gap what Stakeholder thinks
about Data Science can solve vs what data science can actually solve for
business.

C1 mentioned that from her experience, she found that data science projects
are more likely to successful where data scientist and stake holder work
together towards a common business goal. C2 mentioned that Stakeholder should
take equal responsibilities as data scientist to make the Data Science project
successful.

[13] Data science teams often struggle to use their model into business
processes. Also stakeholders often can’t articulate the problems they are
aiming to solve. [13] there is a significant gap between stakeholder and data
science teams that needs to be recognized and addressed. [13] suggest data
science need to bridge the gap between an organizational structure and
leadership commitment to develop better communication, processes, and gain
trust among all stakeholders.

###  III-B Business Problem Understanding:

All the participants mentioned that Business problem understanding is the most
critical step in a data science project. Business problem understanding with
effective stakeholder communication improves the success criteria of a data
science project. During the business problem understanding process, the data
science team gets an overview of the business process and tries to articulate
the problem that the business is trying to solve. It helps the data science
team get more clarity around the business objective, business process, and
some understanding of data. Then Data Science team prepares the plan to check
for data availability. This also helps define the target variable or dependent
variable in the case of a supervised modeling approach. In this process Data
Science team understands the business problem well; more accurately and define
the business objective and target/dependent/outcome variable for the
predictive model or another type of modeling task. More accurately, the
outcome variable is defined, more chances of success of the data science
project.

###  III-C Data Quality issues:

Data quality issue another common issue which came out as a theme. Most
participants from group a mentioned about Data Quality issue. Also they
mentioned most teams don’t want to spend much time on fixing data quality
issues and focusing more on deliverable, which can be measured. Because the
time spend is not mapped to any tangible benefit at this stage. So the idea is
to create something which can be used for multiple projects. Example the
DW(Data Warehousing) and Cube concept got more attention when BI (Business
Intelligence) generate value for organisation and BI/ DW team started creating
something which can be reused for many different projects.

###  III-D Model Deployment and Production:

Several participants from group A and group B, described that many Data
Science project is not deployable or failed due to complexity in
productionzing the model or the outcome. Example an antivirus software company
built a predictive model build with 99.1% accuracy on test data. No doubt that
is a that is one of the best model, but if the expectation is model should be
integrated into a real-time environment, that might become tricky. That
requite a software engineering skill, also the key thing is to how a model
file can be integrated in the existing programming environment. These things
need to be plan ahead before the project starts during the business
understanding and planning phase. Based on the complexity of implementation
project kick-off. If its is feasible to implement, then the production
implementation plan need to be discussed and agreed with the stakeholder at
the beginning of the project.

###  III-E Department level Collaboration:

Several group B candidates highlighted that cross department collaboration is
one of the major challenges while delivering the Data Science outcome. Many
times model outperform and implementation done but due to lack in department
level integration project may not be useful for the organization. Example a
market propensity model can produce most accurate target customer, but that
need to integrated in the marketing channel. Even though its integrated, the
forntline channel might de-prioritize the request due to some other priority
they are having. These situation are out of control of a Data Science and
Stakeholder.

###  III-F Change Management:

Most of the data science project are either used for strategy or time saving
or cost saving using automation to reduce manual effort. Sometimes this
creates resistance from the people affected. Several candidates from group B
mentioned that Change Management is one of the challenge in any data science
implementation. But this is not the key challenge as there are strategy to
tackle it. Also few participants mentioned that the solution for the change
management sometimes out of control of Data Science team. As suggested by C2
”Change management specialized area and need specialized people to handle such
things.”

”Change management is not a key challenge” -C3. May be because he is working
in google where automation using algorithm is always promoted.

To handle change management, ”DS team need to engage with the team going to
affect very early and try to gain confidence from the team affected by
illustrating an explaining how this is going to improve their work” - C1.

###  III-G Data Governance:

Data governance processes, roles, policies and standard are getting more
stronger after GDPR(General Data Protection Regulation)[21] law. ”Around 30%
of data science projects unable to go to production due to Data Governance
rules” C3. Same issues due to data governance also mentioned by various other
participants from group A and Group B. Data governance is a must have standard
for any organization using data as a asset. But to avoid data science project
late failure, data governance team engagement need to be discussed at the
early stage of the project. So that Data governance team can advise on
alternate solution to get the data, or they can advise not to opt for this
project due to the data used in Data Science project are not meeting the
governance standard. In addition to that organisation need to adopt PII
(personal identifiable information) masking approaches to mask the data.

##  IV Discussion and Recommendations

There are the top 7 themes generated [Table II] in this research. More
emphasis was given to Effective Stakeholder management, Most of the interview
participants mentioned the challenges during stakeholder management, and few
mentioned that stakeholder management is not that challenging. Also, business
leaders who sometimes act as stakeholders (in our interview panel) also agreed
that there is a gap in understanding the data science process from a
stakeholder lens.

Usually, Data science projects have more than one stakeholder. In this multi-
stakeholder environment, it’s always good to have a strategy around
stakeholder management. The stakeholder can have different types of posture
[15] based on the RDAP(reactive, defensive, accommodative, or proactive)
strategy (table Fig 2). The same can be applied to Data Science projects. As
C2 mentioned, stakeholders should share equal responsibility. Hence for a
successful data science project, the strategy should be Accommodative or
Proactive where stakeholder postures are Accept responsibility and Anticipate
responsibility, respectively, as described in fig1. But to do that as
suggested by few candidated, every Data Science project should have top
management buying to make the DS project successful.

Figure 1: Stakeholder Management strategy [15]

Sometimes these postures of stakeholders due to the nature of the organization
of the nature of data science application. Sometimes the posture varies at the
department level, even in the same organization. Example in a Banking and
finance business posture of the same stakeholder may vary from a Credit Risk
project to a pure customer marketing project. Hence stakeholder communication
and management should be strategic, and one strategy can’t be generalized for
all data science projects.

Another key challenge most data scientists mention is that most projects
struggle to articulate the business problem well enough. Business problem
understanding is not easy; it usually happens with stakeholders, business
subject matter experts, and the data science team. This isn’t easy because it
requires a substantial amount of prior business knowledge to understand the
business objective clearly. Another challenge sometimes it is hard to define a
uniform business process or rule even in the same department. For example, in
a credit risk modeling scenario, the objective is to identify the risky
customer based on historical default customer data. But defining a default
customer sometimes varies from one business SME to another business SME. Some
SMEs can define a customer as a default customer if someone has not made the
payment 60 days after the due date. In some cases, SME may consider 90 days or
sometimes 1st-time defaulter is not true defaulter as 1st time can be due to
negligence or due to some mistake. So it’s critical to understand the business
problem definition clearly and once you get some level of understanding,
validate that understanding in a meeting with business SME. Sometimes
stakeholder comes from a mindset predictive model or machine learning a
solution to any type of business problem. In some cases, it is true, but the
definition of the business rule should be clear enough to make any Data
Science project successful. [13] stakeholders are sometimes not sure what they
want. This usually happens in business strategy projects. In this business
understanding process, Data scientists should also take the opportunity to
help refine the stakeholder objective in several meetings. But often, business
understanding is not given priority because of the following reason.

  1. 1.

Stakeholder and Data scientist are both very excited to start working on the
project

  2. 2.

Data scientists cannot ask the right questions initially due to a lack of
domain knowledge. So to solve this issue, business SME and Data Scientists
should spend a good amount of time articulating the business problem clearly.

Another challenge in Data Science we found from this research is that the data
quality. Data quality has always been an issue. Spending a good amount of time
on fixing Data Quality is always difficult because effort/time to value
mapping in data quality framework is always missing. Hence fixing data quality
is never get prioritized. To understand data quality, we need to understand
what data quality means. Data quality has three dimensions as mentioned in
[20].

  1. 1.

Accuracy : Are the data free of error ?

  2. 2.

Consistency in Timeliness : Are the data up-to-date ?

  3. 3.

Completeness : Are necessary data present ?

Reasons: Data Quality issues arise due to three main reasons.

  1. 1.

Standardized Data collection process across organization margin someone’s in
saying about group Simpson someone’s in a number of energy

  2. 2.

Error due to human or software

  3. 3.

Error in business logic to process the data

Solution we proposed to fix data quality :

  1. 1.

Better data collection standard from application level data generation

  2. 2.

Centralized data store : An organization needs to spend time building a
feature store with the help of a robust data pipeline. The objective of the
feature store is to create a dataset which can be of high quality and useful
for various DS projects. Ideally the team should list down all the possible DS
projects and start mapping the possible features/ data which need to be
created.

Another key issue identified from this research is the lack of department-
level collaboration, leading to low-value outcomes. Because the department
which should use the product doesn’t use the model as they might have other
priorities to attend. The solution is to list down all the departments going
to be affected by this outcome and department who are going to use the outcome
and which department is going to be involved in hosting this solution. This
need to be discussed keeping stakeholder in loop. After this department level
engagement Data Science team need to share the deployment plan mention how the
outcome can be deployed or reusable. Other issues such as change management
and data governance came out of the theme are not the key bottleneck for the
data science project success. However one participant mentioned that ”around
30% of the time they loose the project due to data governance and privacy
limitation” - C3. The earlier Data Science assess the data privacy and
governance issues, that is better for the project. So that Data Science can
have the opportunity to negotiate or can find other source which can be
approved by governance.

We discussed the problems and the solution in the discussion section. In this
section, we have summarized the solutions based on interviews, my knowledge,
and my experience in Data Science. Based on this research work, we would
recommend there are three key areas where improvement needed

  1. 1.

Stakeholder Management

  2. 2.

Data Quality

  3. 3.

Durable and deployable outcome

A potential data science project might fail due to the above reasons what we
discussed so far. The Data Science team needs to have good stakeholder
management skills to set the expectation of the stakeholder at an early stage
based on the business problem understanding. Also, a data scientist should
have the ability to highlight the potential risk ahead of time that might
occur due to data privacy and/or data security and/or data governance
restrictions. Also, Data scientists should have visibility of the risk of
failure due to factors such as data quality issues and/or deployment
limitations due to the complexity of the software platform and/or IT
infrastructure. To create a fail-proof data quality management for the Data
Science project, the Data Science team needs to spend dedicated time checking
the data quality and plan to engage the data engineer to improve the data
quality standard. Also, to validate the requirement with stakeholder Data
Scientist need to build a deployment plan ahead and share it with the
stakeholder in the early phase of the project. By combining all, we are
proposing a new Data Science methodology called HYBRID CRISP DS(Hybrid Cross
Industry Standard Process for Data Science), which will take care of all the
above hindrances and make Data Science project less prone to failure.

Following figure-II represents HYBRID-CRISP-DS.

Figure 2: HYBRID-CRISP-DS

In this HYBRID-CRISP-DS methodology stakeholder and business SME always be in
the loop during the various phases of the Data Science project. As mentioned
in the figure-II, emphasis has been given on variable selection after
consulting a business SME(subject matter expert), which is an iterative
process. Soon after, that needs to be validated through DGP (Data Governance
and Privacy) in collaboration with DG(Data Governance) team. Based on the
outcome from DGP, proceed to the next step, DDP(deployment and delivery plan),
or negotiate on variable selection for those variables rejected by DG. Based
on the complexity of requirements in DDP SE(software engineer) or Data
engineer, dependencies will be created. Once the deployment plan is ready,
that needs to be validated by stakeholders. Then, the project flows through
DCQ(Data collection and quality) to find the right data source and fix the
quality issues where needed. After DCQ, next is to prepare the data, will
follow through with model building and validation. Then the emphasis is given
to outcome validation by the business user. The accuracy of a predictive model
or any unsupervised model may not make sense to the business. Businesses
should always test the model with live data or any validation data on which
businesses can rely upon. Once the business is satisfied with the outcome,
then the project moves to deployment. Once the project is deployed, a proper
drift monitoring technique is implemented to monitor the drift in an automated
fashion.

##  V Future work

Although several industry experts and researchers suggested alternate
approaches to CRISP-DM such as TDSP(team data science process) by Microsoft
[25] SEMMA(Sampling, Exploring, Modifying, Modeling, and Assessing) by SAS
[28], each comes with the limitation. HYBRID-CRISP-DS is a new approach that
came out of this research. In the future, this approach needs to be applied
and can be observed where issues might occur while applying HYBRID-CRISP-DS in
real-world data science projects. In this process, the Data Science project
outcome never looked from the Data Science project as a software product lens.
So the testing ends in step 8 of this proposed methodology after the business
validates the modeling outcome. So the HYBRID-CRISP-DS approach can be
researched and analyzed further to the Data Science as a product scenario,
where the methodology can be slightly different with respect to business
requirement, deployment, and software product testing. In addition to that,
the proposed HYBRID-CRISP-DS is a series of processes. Hence this research can
be further extended to understand how HYBRID-CRISP-DS process can be
paralleled to make the Data Science process more efficient.

Data Science experts participated in this interview process, not emphasizing
model explainability. However, model explainability can be a big roadblock in
other domains such as healthcare, critical safety environment. Even though
this study has been conducted by interviewing experts from various domains and
geography, this study can be further extended by including more professionals
from a data science background.

## Acknowledgment

Special thanks to my supervisor Dr Kelly Blincoe (Senior Lecturer in Software
Engineering, University of Auckland) for her expert advise, guidance and
excellent support during this research. We acknowledged all the candidates
((Pragyansmita Nayak (C1), Confidential (C2), Mani Kanteswara Rao Garlapati
(C3), Peter Devine (C4), Tural Gulmammadov (C5), Suresh Bommu (C6),
Confidential (C7), Siddhant Gawsane (C8), Confidential (C9)) for their
participation in this research interview.

## References

  * [1] R Wirth, J Hipp, “CRISP-DM: Towards a standard process model for data mining” Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining, 2000.
  * [2] https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/
  * [3] https://blogs.gartner.com/andrew_white/2019/01/03/our-top-data-and-analytics-predicts-for-2019/
  * [4] Jianzheng Liu, Jie Li, Weifeng Li, Jiansheng Wu, Rethinking big data: A review on the data quality and usage issues, ISPRS Journal of Photogrammetry and Remote Sensing, Volume 115, 2016, Pages 134-142, ISSN 0924-2716, https://doi.org/10.1016/j.isprsjprs.2015.11.006. (https://www.sciencedirect.com/science/article/pii/S0924271615002567)
  * [5] @article10.1214/aoms/1177705148, Leo A. Goodman, Snowball Sampling, volume = 32, journal = The Annals of Mathematical Statistics, number = 1, publisher = Institute of Mathematical Statistics, pages = 148 – 170, year = 1961, 10.1214/aoms/1177705148 https://doi.org/10.1214/aoms/1177705148
  * [6] @articlebraun2012thematic, title=Thematic analysis., author=Braun, Virginia and Clarke, Victoria, year=2012, publisher=American Psychological Association
  * [7] M. Young, The Technical Writer’s Handbook. Mill Valley, CA: University Science, 1989.
  * [8] @articledicicco2006qualitative, title=The qualitative research interview, author=DiCicco-Bloom, Barbara and Crabtree, Benjamin F, journal=Medical education, volume=40, number=4, pages=314–321, year=2006, publisher=Wiley Online Library
  * [9] https://www.analyticsvidhya.com/blog/2019/08/data-science-leader-guide-managing-stakeholders/
  * [10] Longbing Cao. 2017. Data Science: A Comprehensive Overview. ACM Comput. Surv. 50, 3, Article 43 (October 2017), 42 pages. DOI:https://doi-org.ezproxy.auckland.ac.nz/10.1145/3076253
  * [11] Jordan, M. I., and T. M. Mitchell. “Machine Learning: Trends, Perspectives, and Prospects.” Science, vol. 349, no. 6245, American Association for the Advancement of Science, 2015, pp. 255–60, http://www.jstor.org/stable/24748571
  * [12] @inproceedingsvogelsang2019requirements, title=Requirements engineering for machine learning: Perspectives from data scientists, author=Vogelsang, Andreas and Borg, Markus, booktitle=2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), pages=245–251, year=2019, organization=IEEE
  * [13] https://sloanreview.mit.edu/article/to-succeed-with-data-science-first-build-the-bridge/
  * [14] https://sloanreview.mit.edu/article/why-so-many-data-science-projects-fail-to-deliver/
  * [15] Geunchan Lim, Hyunchul Ahn, Heeseok Lee, Formulating strategies for stakeholder management: a case-based reasoning approach, Expert Systems with Applications, Volume 28, Issue 4, 2005, Pages 831-840, utanISSN 0957-4174, https://doi.org/10.1016/j.eswa.2004.12.038.
  * [16] Malone, K. (2020). When Translation Problems Arise Between Data Scientists and Business Stakeholders, Revisit Your Metrics. Harvard Data Science Review, 2(1). https://doi.org/10.1162/99608f92.c2fc310d
  * [17] C. Cichy and S. Rass, ”An Overview of Data Quality Frameworks,” in IEEE Access, vol. 7, pp. 24634-24648, 2019, doi: 10.1109/ACCESS.2019.2899751.
  * [18] https://hbr.org/2019/07/building-the-ai-powered-organization
  * [19] https://journals.sagepub.com/doi/full/10.1177/1096348017753521
  * [20] Benjamin T. Hazen, Christopher A. Boone, Jeremy D. Ezell, L. Allison Jones-Farmer, Data quality for data science, predictive analytics, and big data in supply chain management: An introduction to the problem and suggestions for research and applications, International Journal of Production Economics,Volume 154, 2014, Pages 72-80, ISSN 0925-5273, https://doi.org/10.1016/j.ijpe.2014.04.018. (https://www.sciencedirect.com/science/article/pii/S0925527314001339)
  * [21] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
  * [22] https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining
  * [23] ftp://public.dhe.ibm.com/software/analytics/spss/documentation/modeler/18.0/en/ModelerCRISPDM.pdf
  * [24] S. Sagiroglu and D. Sinanc, ”Big data: A review,” 2013 International Conference on Collaboration Technologies and Systems (CTS), 2013, pp. 42-47, doi: 10.1109/CTS.2013.6567202.
  * [25] https://docs.microsoft.com/en-us/azure/architecture/data-science-process/overview
  * [26] https://www.psych.auckland.ac.nz/en/about/thematic-analysis.html
  * [27] Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
  * [28] http://documentation.sas.com/doc/en/emref/14.3/n061bzurmej4j3n1jnj8bbjjm1a2.htm
  * [29] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.58.9085&rep=rep1&type=pdf
  * [30] @articlebraun2006using, title=Using thematic analysis in psychology, author=Braun, Virginia and Clarke, Victoria, journal=Qualitative research in psychology, volume=3, number=2, pages=77–101, year=2006, publisher=Taylor & Francis
  * [31] https://agilemanifesto.org/