# Hierarchical Cell-to-Tissue Graph Representations for Breast Cancer
Subtyping in Digital Pathology

Pushpak Pati*1,2 Guillaume Jaume*1,3 Antonio Foncubierta-Rodríguez1 Florinda
Feroce4 Anna Maria Anniciello4 Giosue Scognamiglio4 Nadia Brancati5 Maryse
Fiche6 Estelle Dubruc7 Daniel Riccio5 Maurizio Di Bonito4 Giuseppe De Pietro5
Gerardo Botti4 Jean-Philippe Thiran3 Maria Frucci5 Orcun Goksel2 Maria
Gabrani1 1IBM Research Zurich,
Switzerland  2ETH Zurich,
Switzerland  3EPFL,
Switzerland  4IRCCS-Fondazione Pascale,
Italy  5ICAR-CNR,
Italy  6Aurigen,
Switzerland  7CHUV,
Switzerland

###### Abstract

Cancer diagnosis and prognosis for a tissue specimen are heavily influenced by
the phenotype and topological distribution of the constituting histological
entities. Thus, adequate tissue representation by encoding the histological
entities, and quantifying the relationship between the tissue representation
and tissue functionality is imperative for computer aided cancer patient care.
To this end, several approaches have leveraged cell-graphs, that encode cell
morphology and cell organization, to denote the tissue information, and
utilize graph theory and machine learning to map the representation to tissue
functionality. Though cellular information is crucial, it is incomplete to
comprehensively characterize the tissue. Therefore, we consider a tissue as a
hierarchical composition of multiple types of histological entities from fine
to coarse level, that depicts multivariate tissue information at multiple
levels. We propose a novel multi-level hierarchical entity-graph
representation to depict a tissue specimen, which encodes multiple
pathologically relevant entity types, intra- and inter-level entity-to-entity
interactions. Subsequently, a hierarchical graph neural network is proposed to
operate on the hierarchical entity-graph representation to map the tissue
structure to tissue functionality. Specifically, we utilize cells and tissue
regions in a tissue to build a HierArchical Cell-to-Tissue (HACT) graph
representation, and HACT-Net, an instance of message passing graph neural
network, to classify histology images. As part of this work, we propose the
BReAst Carcinoma Subtyping (BRACS) dataset, a large cohort of Haematoxylin &
Eosin stained breast tumor regions-of-interest, to evaluate and benchmark our
proposed methodology against pathologists and state-of-the-art computer-aided
diagnostic approaches. Thorough comparative assessment and ablation studies
demonstrated the superior classification efficacy of the proposed methodology.

###### Index Terms:

Digital pathology, Breast cancer classification, Hierarchical tissue
representation, Hierarchical graph neural network, Breast cancer dataset

##  I Introduction

Breast cancer is the second most commonly diagnosed cancer and registers the
second-highest number of deaths for women with cancer [1]. A study by [2]
exhibits that intensive early diagnostic activities have improved 5-year
survival to 85% during 2005–09 for breast cancer patients. Early diagnosis of
cancer, primarily through manual inspection of pathology slides by
pathologists, enables the acute assessment of risk and facilitates optimal
treatment plan. Though the diagnostic criteria for breast cancer are
established, the continuum of histologic features phenotyped across the
diagnostic spectrum prevents the distinct demarcation. Thus, manual inspection
is tedious and time-consuming with significant intra- and inter-observer
variability [3, 4]. Increasing incidence rate of breast cancer cases per year
[5] and the complications in manual diagnosis demand for automated computed
aided diagnostics.

Whole-slide scanning systems empowered rapid digitization of pathology slides
into high-resolution whole-slide images (WSIs), thereby profoundly
transforming pathologists’ daily practice [6]. Further, they enabled computed
aided diagnostics to leverage artificial intelligence [7, 8], especially deep
learning, to address various digital pathology tasks in breast diagnosis [9],
such as nuclei segmentation [10, 11], nuclei classification [12, 13], gland
segmentation [14, 15], tumor detection [16, 17, 18], tumor staging [16, 19]
etc. Deep learning techniques primarily employ Convolutional Neural Networks
($\mathrm{CNN}$) [20, 21] to process pathology images in a patch-wise manner.
$\mathrm{CNN}$s extract representative patterns from patches and aggregate the
patch representations to perform image-level tasks. However, patch-wise
processing suffers from the trade-off between the resolution of operation and
the acquisition of adequate context [22, 23]. Operating at a higher resolution
captures local cellular information but limits the field-of-view due to
computational burden, thereby limits the accessibility of global information
capturing the tissue microenvironment. In contrast, operating at a lower
resolution hinders access to the cellular microenvironment. [22, 23, 24] have
proposed $\mathrm{CNN}$-based methods to address the issue with visual
context, but the $\mathrm{CNN}$s, operating on fix-sized input patches, are
confined to a fix-sized field-of-view. The fix-sized inputs also restrict the
diagnosis to adapt to regions-of-interest with varying dimensions. Further,
the pixel-based processing in $\mathrm{CNN}$s disregards the notion of
pathologically comprehensible entities [25], such as cells, tissue types, or
glands. The absence of the notion of relevant entities limits the
interpretability of the $\mathrm{CNN}$’s input space and the utilization of
well-established entity-level prior pathological knowledge in $\mathrm{CNN}$s’
operation. Additionally, $\mathrm{CNN}$s disregard the structural composition
of tissue, where fine entities hierarchically constitute to form coarse
entities, such as, epithelial cells organize to form epithelium, which further
constitutes to form glands.

In this paper, we address the aforementioned limitations by shifting the
analytical paradigm from pixel to entity-based processing. In an entity
paradigm, a histology image is described as an entity-graph, where nodes and
edges of the graph denote biological entities and inter-entity interactions.
An entity-graph can be configured in terms of the type of entity set, entity
attributes, graph topology, etc. by leveraging task-specific prior
pathological knowledge. Thus, the graph representation enables pathology-
specific interpretability and human-machine co-learning. Additionally, the
graph representation is memory efficient compared to pixelated pathology
images and can seamlessly describe a large tissue region. [26] first
introduced cell-graphs using cells as the entity type. Though a cell-graph
efficiently encodes the cell microenvironment, it cannot extensively capture
the tissue microenvironment, i.e., the distribution of tissue regions such as
necrosis, stroma, epithelium, etc. Similarly, a tissue-graph comprising of the
set of tissue regions cannot depict the cell microenvironment. Therefore, an
entity-graph representation using a single type of entity set is insufficient
to comprehensively describe the tissue structure in a histology image. To
address the limitation, we propose a multi-level entity-graph representation,
i.e., HierArchical Cell-to-Tissue ($\mathrm{HACT}$), consisting of multiple
types of entity sets, i.e., cells and tissue regions, to encode both cell and
tissue microenvironment. The multiset of entities is inherently coupled
depicting tissue composition and tissue information at multiple scales. The
$\mathrm{HACT}$ graph encodes the individual entity attributes and the intra-
and inter-entity relationships among the multiset of entities to
hierarchically describe a pathology image from fine cells to coarse tissue
regions. Upon the graph construction, a graph neural network ($\mathrm{GNN}$),
a deep learning technique operating on graph-structured data, processes the
entity-graph to perform image analysis. Specifically, we introduce a
hierarchical $\mathrm{GNN}$, HierArchical Cell-to-Tissue Network
($\mathrm{HACT}$-Net), to sequentially operate on $\mathrm{HACT}$ graph, from
fine-level to coarse-level, to result in a fixed dimensional embedding for the
image. The embedding encodes the morphological and topological distribution of
the multiset of entities in the tissue. Interestingly, the proposed
methodology resembles the tissue diagnostic procedure in clinical practice,
where a pathologist hierarchically analyzes a tissue.

We propose a methodology that consists of the $\mathrm{HACT}$ graph
construction and $\mathrm{HACT}$-Net based histology image analysis. We
characterize breast tumor regions-of-interest ($\mathrm{TRoI}$s) to evaluate
our methodology. The preliminary version of this work is presented by [27].
The work has been substantially extended by, 1) improving the $\mathrm{HACT}$
representation and $\mathrm{HACT}$-Net architecture, 2) dataset extension
(2$\times$), 3) detailed ablation studies and evaluation on public data, and
4) benchmarking against independent pathologists. More specifically, the major
contributions of this paper are,

  * •

A novel hierarchical entity-graph representation ($\mathrm{HACT}$) and
hierarchical learning ($\mathrm{HACT}$-Net) methodology for analyzing
histology images.

  * •

BReAst Carcinoma Subtyping (BRACS) dataset 111The BRACS dataset for breast
cancer subtyping will be shortly released to the community, a large cohort of
breast $\mathrm{TRoI}$s annotated with seven breast cancer subtypes. BRACS
includes challenging atypical cases and a wide variety of $\mathrm{TRoI}$s
representing a more realistic breast cancer analysis.

  * •

The proposed methodology is benchmarked on the BRACS dataset by comparing with
three independent pathologists for breast cancer subtyping. Upon extensive
assessment, we demonstrate comparable classification performance as the
pathologists on per-class and aggregated classification. Also, our methodology
outperforms several recent $\mathrm{CNN}$ and $\mathrm{GNN}$ approaches for
cancer subtyping.

##  II Related work

Tumor subtyping in digital pathology: Several deep learning algorithms have
been proposed to categorize pathology images or whole-slide images into cancer
subtypes [28, 29, 8, 30, 31, 16]. Most of the algorithms employ
$\mathrm{CNN}$s to address the task in a patch-wise manner. [31, 32, 33, 19]
utilize $\mathrm{CNN}$ to categorize breast cancer histology images into
various subtypes. These methodologies employ single stream patch-wise
approaches to capture local patch-level context, unify patch-level information
via several aggregation strategies, and classify the image using aggregated
information. However, single-stream approaches do not capture adequate context
from neighboring tissue microenvironment to aptly classify a patch. In [23]
this issue is addressed by including multi-scale information from concentric
patches across different magnifications. [24] propose a neural image
compression methodology, where WSIs are compressed using a neural network
trained in an unsupervised fashion, followed by a $\mathrm{CNN}$ training on
the compressed image representations to predict image labels. [34] include an
attention module and auxiliary task to improve neural image compression for
classifying histology images. [35] propose a hybrid convolutional and
recurrent deep neural network to utilize the spatial correlations among
patches for pathological image classification. [22] propose a stacked
$\mathrm{CNN}$ architecture to capture large context and perform end-to-end
processing of large-sized histology images. [36] propose a streaming
$\mathrm{CNN}$ to accommodate multi-megapixel images. [37] utilize multiple
instance learning approach to process whole-slide images in an end-to-end
manner. [38] extend the multiple instance learning to automatically identify
sub-regions of high diagnostic value, through an attention mechanism, and
improve the classification. Though the aforementioned methodologies employ
different strategies to capture useful context information, they still operate
on a square and fix-sized input image. However, in reality, $\mathrm{TRoI}$s
can be of highly varying dimensions and shapes depending on the cancer subtype
and the site of tissue extraction. Our proposed entity-graph methodology can
acquire both local and global context from arbitrary-sized $\mathrm{TRoI}$s to
address the aforementioned limitations.

Graphs in digital pathology: Entity-graph-based tissue representation can
effectively describe the phenotypical and structural properties of tissue by
incorporating morphology, topology, and interactions among biologically
defined tissue entities. The graph topology is heuristically defined using
k-Nearest Neighbors, probabilistic modeling, Waxman model etc. [39].
Subsequently, a $\mathrm{CG}$ is processed by classical machine learning
techniques in [39] or $\mathrm{GNN}$s in [40, 27, 41, 42] to map to tissue
function. Recently, improved graph representations using patches [43] and
tissue regions [27] are proposed to enhance the tissue structure-function
mapping. Other graph-based applications in computational pathology include
cellular community detection [44], WSI classification [45, 46] etc. Notably,
entity-graph representation consists of biological entities, which the
pathologists can readily relate to. Thus, graph-based analysis allows to
incorporate pathologically defined task-specific entity-level prior knowledge
to construct meaningful tissue representation. Further interesting research is
conducted to enable the interpretability and explainability of the graph-based
networks to pathologists. To this end, [40] analyzes the cluster assignment of
the nodes in $\mathrm{CG}$ to group cells according to their appearance and
tissue types. [47] introduce a post-hoc graph-pruning explainer to identify
decisive cells and interactions. [48] employ robust spatial filtering that
utilizes an attention-based $\mathrm{GNN}$ and node occlusion to highlight
cell contributions. [49] propose a set of quantitative metrics using
pathologically measurable cellular properties to characterize several graph
explainability algorithms in $\mathrm{CG}$ analysis.

##  III Preliminaries

###  III-A Notation

We define an attributed undirected entity graph $G:=(V,E,H)$ as a set of nodes
$V$, edges $E$, and node features $H$. Each node $v\in V$ is represented by a
feature vector $h(v)\in\mathbb{R}^{d}$, thus,
$H\in\mathbb{R}^{|V|\,\times\,d}$. $d$ denotes the number of features per
node, and $|\,.\,|$ denotes set cardinality. An edge between two nodes $u,v\in
V$ is denoted as $e_{uv}$. The graph topology is described by a symmetric
adjacency matrix $A\in\mathbb{R}^{|V|\,\times\,|V|}$, where $A_{u,v}=1$ if
$e_{uv}\in E$. The neighborhood of a node $v\in V$ is denoted as
$\mathcal{N}(v):=\\{u\in V\;|\;v\in V,\;e_{uv}\in E\;\\}$.

###  III-B Graph Neural Networks

$\mathrm{GNN}$ [50, 51, 52, 53, 54] defines a class of neural networks that
extend the concept of convolution to operate on graph-structured data.
Specifically, we employ message passing based $\mathrm{GNN}$ in this work. In
message passing $\mathrm{GNN}$ [55], the node features $h(v),\,\forall v\in V$
are iteratively updated in a two-step procedure, i) AGGREGATE, and ii) UPDATE.
In the AGGREGATE step for node $v$, the features of neighboring nodes
$\mathcal{N}(v)$ are aggregated into a single feature representation. In the
UPDATE step, the features of node $v$ is updated by using the current node
features and the aggregated representation from the AGGREGATE step. A series
of $T$ such iterations, i.e., the number of $\mathrm{GNN}$ layers, are
employed to obtain updated node features $\forall\,v\in V$ up to $T$-hops.
Finally, the node features $h^{T}(v)$ are pooled in the READOUT step to build
a fix-sized graph-level embedding $h_{G}$. The AGGREGATE, UPDATE and READOUT
function must be differentiable to allow back-propagation. Additionally, the
AGGREGATE and READOUT function must be permutation-invariant such that the
aggregated representation is invariant to the node ordering. Formally, the
three steps are presented as,

$\begin{split}&a^{\,t\,+\,1}(v)=\mathit{AGGREGATE}\,(\,\\{\,h^{\,t}(u):u\in\mathcal{N}(v)\,\\}\,)\\\
&h^{\,t\,+\,1}(v)=\mathit{UPDATE}\,(\,h^{\,t}(v),\,a^{\,t\,+\,1}(v)\,)\\\
&h_{\,G}=\mathit{READOUT}\,(\,\\{\,h^{\,T}(v):v\in V\,\\}\,)\end{split}$ (1)

An important aspect of designing a $\mathrm{GNN}$ is the characterization of
its expressive power. A $\mathrm{GNN}$ has a strong expressive power if it can
map two non-isomorphic graphs to two unique graph embeddings, thus imparting
an injective mapping between the graph space and the graph-embedding space. A
line-of-research exploring the expressive power of $\mathrm{GNN}$s [52, 56,
57] highlight the connection between iterative message passing procedure of
$\mathrm{GNN}$ and the popular Weisfeiler-Lehman (WL) [58] test for graph
isomorphism. It is established that architectures such as the Graph
Isomorphism Network [52] ($\mathrm{GIN}$) can perform as well as the
1-dimensional WL test for _countable_ node feature spaces, i.e., when the node
features are discrete. An example of graph with discrete node features can be
the study of molecule design, where the nodes represent atoms that are
discrete in nature. Recent studies show that in case of _continuous_ node
features, e.g., $\mathrm{CNN}$-based node embeddings, the use of multiple
permutation invariant aggregators, such as sum, max, mean etc., can help to
build expressive $\mathrm{GNN}$s [59, 60]. To this end, [60] proposed the
Principal Neighbourhood Aggregation ($\mathrm{PNA}$) network by using a
combination of aggregators followed by degree-scalers. The series of
aggregators replace the sum operation in $\mathrm{GIN}$ and the degree-scalers
amplify or dampen neighboring aggregated message according to the node’s
degree. Overviews of the $\mathrm{GIN}$ and $\mathrm{PNA}$ architectures are
presented in Figure 1.

(a) $\mathrm{GIN}$

(b) $\mathrm{PNA}$

Figure 1: Overview of $\mathrm{GIN}$ and $\mathrm{PNA}$ layers. $h_{v}^{t}$
and $\\{h_{u}^{t}\\}$ denote the representation of node $v$ and its neighbors
at layer $t$. $\mathrm{GIN}$ uses sum as the AGGREGATE function, followed by a
sum and multi-layer perceptron ($\mathrm{MLP}$) for the UPDATE function.
$\mathrm{PNA}$ uses a set of aggregators (element-wise mean, standard
deviation, maximum and minimum) followed by degree-scalers (identity,
amplifier and dampener) as the AGGREGATE function. The UPDATE function
consists of a concatenation followed by a $\mathrm{MLP}$.

Figure 2: Overview of the proposed hierarchical entity-graph based tissue
analysis methodology. Post pre-processing measures, the methodology constructs
a hierarchical entity-graph representation of a tissue, and processes via a
hierarchical graph neural network to map the tissue composition to respective
tissue category.

##  IV Methodology

In this section, we specify the details of the proposed methodology, as shown
in Figure 2, for hierarchical tissue analysis. For an input Hematoxylin and
Eosin (H&E) stained pathology $\mathrm{TRoI}$, first, we apply pre-processing
measures to standardize the input. Subsequently, we identify pathologically
relevant entities and construct a $\mathrm{HACT}$ graph representation for the
$\mathrm{TRoI}$ by encoding the morphological and topological information of
the identified entities. Finally, $\mathrm{HACT}$-Net, a hierarchical
$\mathrm{GNN}$, is employed to map the $\mathrm{HACT}$ graph representation to
the corresponding cancer subtype.

###  IV-A Pre-processing

H&E stained tissue specimens exhibit appearance variability due to various
reasons, such as specimen preparation technique, staining protocols like
temperature of the adopted solutions, fixation characteristics, imaging device
characteristics, etc. Such variability in appearance adversely impacts the
model performance of downstream diagnosis as demonstrated by [61, 62]. To
alleviate the appearance variability, we employ the unsupervised reference-
free stain normalization algorithm proposed by [63]. The algorithm is based on
the principle that the RGB color of each pixel is a linear combination of two
unknown stain vectors, Hematoxylin and Eosin, which need to be estimated.
First, the algorithm estimates the stain vectors of the $\mathrm{TRoI}$ by
using a Singular Value Decomposition of the non-background pixels. Second, the
algorithm applies a correction to account for the intensity variations due to
noise. The algorithm does not involve any intermediate step that requires
training of model parameters, thus it is computationally inexpensive.
Specifically, we employ a scalable and fast pipeline proposed by [64] to
conduct the stain normalization.

###  IV-B Graph representation

Stain normalized $\mathrm{TRoI}$s are processed to identify relevant entities
and construct the hierarchical entity-graph representations. In this work, we
consider _nuclei_ and _tissue_ components as the entities to build
$\mathrm{HACT}$ graph representations. Specifically, a $\mathrm{HACT}$ graph
consists of three components, 1) a low-level cell-graph, capturing cell
morphology and interactions, 2) a high-level tissue-graph, capturing
morphology and spatial distribution of tissue regions, and 3) cells-to-tissue
hierarchies, encoding the relative spatial distribution of the cells with
respect to the tissue distribution. The details of the components are
presented in the following subsections.

####  IV-B1 Cell-graph representation

A cell-graph ($\mathrm{CG}$) characterizes low-level cell information, where
the nodes represent cells encoding cell morphology and edges encode cellular
interactions depicting cell topology. A $\mathrm{CG}$ is constructed in three
steps, i) nuclei detection, ii) nuclei feature extraction, and iii)
configuration of $\mathrm{CG}$ topology. The steps are demonstrated in Figure
3.

Precise nuclei detection leads to reliable $\mathrm{CG}$ representation. To
this end, we use HoVer-Net, a nuclei segmentation network proposed by [11],
pre-trained on MoNuSeg dataset by [10]. HoVer-Net leverages the instance-rich
information encoded within the vertical and horizontal distances of nuclear
pixels to their centers of mass. These distances are used to separate
clustered nuclei, resulting in an accurate segmentation, particularly in areas
with overlapping instances. Notably, HoVer-Net identifies several nuclei
subtypes despite large inter- and intra-subtype variability. The centroids of
the segmented instances are used as the spatial coordinates of the nodes in
$\mathrm{CG}$.

Following nuclei detection, morphological features are extracted by processing
patches of size $h\times w$ centered around nuclei centroids via ResNet [65]
architecture pre-trained on ImageNet dataset [66]. Spatial features of the
nuclei are extracted as the spatial coordinates of the nuclei normalized by
the $\mathrm{TRoI}$ dimensions. The morphological and spatial features
constitute the nuclei features grouped in a node feature matrix as,
$H_{\mathrm{CG}}\in\mathbb{R}^{\;|V_{\mathrm{CG}}|\,\times\,d_{\mathrm{CG}}}$.

To generate $\mathrm{CG}$ topology $E_{\mathrm{CG}}$, we build upon the fact
that spatially close cells have stronger interactions [67] and should
therefore be connected, and distant cells have weak cellular interactions and
should remain disconnected. To this end, we use the k-Nearest Neighbors (kNN)
algorithm to build an initial topology, that we subsequently prune by removing
edges longer than a threshold distance $d_{\min}$. We use $L2$-norm between
nuclei centroids in the image space to quantify the cellular distance.
Formally, for each node $v$, an edge $e_{vu}$ is built if,

$\begin{split}u\in\\{w\;|\;\mathrm{dist}(v,w)\leq
d_{k}\wedge\mathrm{dist}(v,w)<d_{\min},\;\forall w,v\in V_{\mathrm{CG}},\;\\\
,d_{k}=k\text{-th smallest distance in }\mathrm{dist}(v,w)\\}\end{split}$ (2)

The $\mathrm{CG}$ topology is represented by a binary adjacency matrix,
$E_{\mathrm{CG}}\in\mathbb{R}^{\;|V_{\mathrm{CG}}|\;\times\;|V_{\mathrm{CG}}|}$.
Figure 3 presents the $\mathrm{CG}$ representation for a sample
$\mathrm{TRoI}$. Formally, a $\mathrm{CG}$ representation is formulated as,
$G_{\mathrm{CG}}:=$ { $V_{\mathrm{CG}},E_{\mathrm{CG}},H_{\mathrm{CG}}$ }.

Figure 3: Overview of hierarchical cell-to-tissue ($\mathrm{HACT}$) graph
construction for a $\mathrm{TRoI}$. The $\mathrm{HACT}$ graph representation
consists of a cell-graph, a tissue-graph and cell-to-tissue hierarchies, and
encodes the phenotypical and topological distribution of tissue entities to
describe the cell and tissue microenvironment.

####  IV-B2 Tissue-graph representation

A tissue graph ($\mathrm{TG}$) depicts a high-level tissue microenvironment,
where nodes and edges of $\mathrm{TG}$ denote tissue regions and their
interactions respectively. Similar to a $\mathrm{CG}$, a $\mathrm{TG}$ is
constructed by first identifying tissue regions, i.e., epithelium, stroma,
lumen, necrosis, etc., followed by feature representation of the tissue
regions, and $\mathrm{TG}$ topology construction. The steps are demonstrated
in Figure 3.

Tissue regions are identified in a two-step process. First, we oversegment the
tissue at a low magnification to detect non-overlapping homogeneous
superpixels. Operating at low magnification avoids noisy pixels and renders
computational efficiency. To this end, we employ the Simple Linear Iterative
Clustering (SLIC) superpixel algorithm [68]. Formally, SLIC follows an
unsupervised approach to associate each pixel with a feature vector and merges
them using a spatially localized version of the standard k-means clustering.
Second, we iteratively merge neighboring superpixels that have similar color
attributes, i.e., channel-wise mean and standard deviation, to create
superpixels capturing meaningful tissue information. A sample of tissue region
instance map is presented in Figure 3.

We follow a two-step procedure to extract feature representations of tissue
regions. First, we extract $\mathrm{CNN}$-based features for the oversegmented
superpixels. Patches of size $h\times w$ centered around the oversegmented
superpixel centroids are processed by a ResNet. Second, morphological features
of a tissue region are obtained by averaging the deep features of its
constituting superpixels. Similar to $\mathrm{CG}$, we include spatial
features as the normalized centroids of the tissue region. For a
$\mathrm{TRoI}$ with a set of $V_{\mathrm{TG}}$ tissue regions, we denote the
node feature matrix as,
$H_{\mathrm{TG}}\in\mathbb{R}^{\;|V_{\mathrm{TG}}|\,\times\,d_{\mathrm{TG}}}$.

To generate $\mathrm{TG}$ topology, we assume that adjacent tissue regions
biologically interact and should be connected. To this end, we construct a
Region Adjacency Graph [69] where an edge is built between each adjacent
region. The $\mathrm{TG}$ topology is denoted by a binary adjacency matrix,
$E_{\mathrm{TG}}\in\mathbb{R}^{\;|V_{\mathrm{TG}}|\,\times\,|V_{\mathrm{TG}}|}$.
Formally, a $\mathrm{TG}$ representation is formulated as, $G_{\mathrm{TG}}:=$
{$V_{\mathrm{TG}},E_{\mathrm{TG}},H_{\mathrm{TG}}$}.

####  IV-B3 Hierarchical Cell-to-Tissue graph representation

The tissue in histopathology can be considered as a hierarchical organization
of biological entities ranging from fine-level, i.e., cells, to coarse-level,
i.e., tissue regions. There exist intra- and inter-level coupling depicting
topological distribution and interactions among the entities. Following the
motivation, we propose $\mathrm{HACT}$, a HierArchical Cell-to-Tissue
($\mathrm{HACT}$) graph representation to jointly represent low-level
$\mathrm{CG}$ and high-level $\mathrm{TG}$. Intra-level topology is captured
by standalone $\mathrm{CG}$ and $\mathrm{TG}$. Inter-level topology is
presented by a binary assignment matrix, i.e., cell-to-tissue hierarchies
matrix,
$A_{\mathrm{CG}\rightarrow\mathrm{TG}}\in\mathbb{R}^{\;|V_{\mathrm{CG}}|\;\times\;|V_{\mathrm{TG}}|}$
that utilizes the relative spatial distribution of the nuclei with respect to
the tissue regions. For the $i^{th}$ nucleus and $j^{th}$ tissue region, the
corresponding assignment is given as,

$\begin{split}&A_{\mathrm{CG}\rightarrow\mathrm{TG}}[\,i,j\,]=1,\;\text{if}\,i^{th}\,\text{nucleus
centroid}\in j^{th}\,\text{tissue region}\\\
&A_{\mathrm{CG}\rightarrow\mathrm{TG}}[\,i,j\,]=0,\;\text{otherwise}\end{split}$
(3)

Cell-to-tissue hierarchies for a tissue region are presented in Figure 3. Note
that, each nucleus is assigned to only one tissue region. For a segmented
nucleus overlapping with multiple tissue regions, the nucleus is assigned to
the tissue region that maximally overlaps with the nucleus. Formally for a
given $\mathrm{TRoI}$, a $\mathrm{HACT}$ representation is formulated as,
$G_{\mathrm{HACT}}:=$
{$G_{\mathrm{CG}},G_{\mathrm{TG}},A_{\mathrm{CG}\rightarrow\mathrm{TG}}$}.

Figure 4: Overview of HACT-Net.

###  IV-C Graph learning

The $\mathrm{HACT}$ graph representation for a $\mathrm{TRoI}$ is processed by
a hierarchical $\mathrm{GNN}$ to map the $\mathrm{TRoI}$ composition to
respective $\mathrm{TRoI}$ subtype. To this end, we propose a HierArchical
Cell-to-Tissue Network ($\mathrm{HACT}$-Net), a hierarchical $\mathrm{GNN}$
architecture as presented in Figure 4.

####  IV-C1 HACT-Net architecture & learning

$\mathrm{HACT}$-Net intakes a $\mathrm{HACT}$ representation
$G_{\mathrm{HACT}}$ and outputs a graph-level representation
$h_{\mathrm{HACT}}\in\mathbb{R}^{d_{\mathrm{HACT}}}$. Subsequently, a multi-
layer perceptron ($\mathrm{MLP}$) classifies $h_{\mathrm{HACT}}$ into
respective subtype. Formally, $\mathrm{HACT}$-Net consists of two
$\mathrm{GNN}$s, namely Cell-$\mathrm{GNN}$ ($\mathrm{CG}$-$\mathrm{GNN}$) and
Tissue-$\mathrm{GNN}$ ($\mathrm{TG}$-$\mathrm{GNN}$), to hierarchically
process the $\mathrm{HACT}$ graph from fine to coarse level.
$\mathrm{HACT}$-Net can operate with any $\mathrm{GNN}$ architecture
pertaining to $\mathrm{CG}$-$\mathrm{GNN}$ and $\mathrm{TG}$-$\mathrm{GNN}$.
In this work, we leverage the recent advances in $\mathrm{GNN}$s and model
$\mathrm{HACT}$-Net using the Principal Neighbourhood Aggregation
($\mathrm{PNA}$) layer [60].

First, $\mathrm{CG}$-$\mathrm{GNN}$ intakes $G_{\mathrm{CG}}:=$
$\\{V_{\mathrm{CG}},E_{\mathrm{CG}},H_{\mathrm{CG}}\\}$, and applies
$T_{\mathrm{CG}}$ $\mathrm{PNA}$ layers to build contextualized cell-node
embeddings. Inline with Equation (1), we iteratively update each node
embedding $h^{(t)}(v)$, $\forall v\in V_{\mathrm{CG}}$ as,

$\begin{split}&h_{\,\mathrm{CG}}^{\,(0)}(v)=H_{\,\mathrm{CG}}(v)\\\
&a_{\,\mathrm{CG}}^{\,(t\,+\,1)}(v)={{{\oplus}}}_{\,u\,\in\,\mathcal{N}_{\,\mathrm{CG}}(v)}\,M_{\,\mathrm{CG}}^{\,(t)}\,\Big{(}\,h_{\,\mathrm{CG}}^{\,(t)}(v),\,h_{\,\mathrm{CG}}^{\,(t)}(u)\,\Big{)}\\\
&h_{\,\mathrm{CG}}^{\,(t\,+\,1)}(v)=U_{\,\mathrm{CG}}^{\,(t)}\,\Bigg{(}\,h_{\,\mathrm{CG}}^{\,(t)}(v),\,a_{\,\mathrm{CG}}^{\,(t\,+\,1)}(v)\,\Bigg{)}\end{split}$
(4)

where, $t=0,\dots,T_{\mathrm{CG}}$. $\mathcal{N}_{\,\mathrm{CG}}(v)$ is the
set of neighborhood cell-nodes of $v$. The functions $U_{\,\mathrm{CG}}^{\,t}$
and $M_{\,\mathrm{CG}}^{\,t}$ are $\mathrm{MLP}$s. ${{{\oplus}}}$ denotes the
combination of multiple degree-scalers and aggregators, i.e.,

$\begin{split}&{{{\oplus}}}=\Big{[}I,\mathcal{S}(D,\alpha=1),\mathcal{S}(D,\alpha=-1)\Big{]}\,{{{\otimes}}}\,\Big{[}\mu,\sigma,\mathrm{max},\mathrm{min}\Big{]}\,\\\
&\mathcal{S}(D,\,\alpha)=\frac{\log\;(D+1)^{\,\alpha}}{\delta}\,\\\
&\delta=\frac{1}{|\text{train}|}\sum_{i\,\in\,\text{train}}\log\;(d_{i}+1)\end{split}$
(5)

where, $I$ is the identity matrix, $\mathcal{S}$ is the degree-scaler matrix,
$D$ is the degree matrix of cell-nodes,
$[\,\mu,\sigma,\mathrm{max},\mathrm{min}\,]$ is the list of aggregators, and
${{{\otimes}}}$ is the tensor product. $\delta$ is the normalization factor
computed as the average log-scale cell-node degree from the training dataset.
$\alpha$ is a variable parameter that is negative for attenuation, positive
for amplification, or zero for no scaling. The aggregators compute the
statistics on the neighboring nodes. The schematic diagram of a $\mathrm{PNA}$
layer is presented in Figure 1. Following $T_{\mathrm{CG}}$ $\mathrm{PNA}$
layers, an $\mathrm{LSTM}$-based jumping knowledge technique [70] is employed
to adapt to different sub-graph structures in the $\mathrm{CG}$, given as,

$\begin{split}&h^{\,(T_{\mathrm{CG}}+1)}_{\mathrm{CG}}(v)=\mathrm{LSTM}\,\Big{(}\,\Big{\\{}\,h_{\,\mathrm{CG}}^{\,(t)}(v)\;\Big{|}\;t=1,\dots,T_{\mathrm{CG}}\,\Big{\\}}\,\Big{)}\end{split}$
(6)

The cell node embeddings $h^{\,T_{\mathrm{CG}}+1}_{\mathrm{CG}}(v)\;|\;v\in
V_{\mathrm{CG}}$ and the assignment matrix
$A_{\mathrm{CG}\rightarrow\mathrm{TG}}$ are used as additional hierarchical
information to initialize the tissue-node features in the $\mathrm{TG}$, i.e.,

$h_{\,\mathrm{TG}}^{\,(0)}(w)=\mathrm{CONCAT}\,\Big{(}\,H_{\,\mathrm{TG}}(w),\sum_{v\,\in\,\mathcal{M}\,(w)}h_{\mathrm{CG}}^{\,(T_{\mathrm{CG}}+1)}(v)\,\Big{)}\\\
$ (7)

where, $\mathrm{CONCAT}$ denotes concatenation operation.
${\mathcal{M}}\,(w):=$ $\\{\,v\in
V_{\mathrm{CG}}\;|\;A_{\mathrm{CG}\rightarrow\mathrm{TG}}\,(v,w)=1\,\\}$ is
the set of nodes in $G_{\mathrm{CG}}$ mapping to a node $w\in
V_{\mathrm{TG}}$. Analogous to Equation (4), we process $G_{\mathrm{TG}}$
using the $\mathrm{TG}$-$\mathrm{GNN}$, based on $\mathrm{PNA}$ layers, to
compute the tissue region node embeddings $h^{(t)}_{\mathrm{TG}}(w),\;\forall
w\in V_{\mathrm{TG}}$. At $t=T_{\mathrm{TG}}$, the embedding of each tissue
component node $w$ encodes the cell and tissue information up to
$T_{\mathrm{TG}}$-hops from $w$.

The tissue-node embeddings are further processed with an LSTM-based jumping
knowledge technique to aggregate the intermediate tissue-node representations.
Finally, the graph-level embedding $h_{\mathrm{HACT}}$ is extracted by summing
all the tissue-node representations. An $\mathrm{MLP}$ layer followed by
softmax operation maps $h_{\mathrm{HACT}}$ to the respective $\mathrm{TRoI}$
label. The model is trained end-to-end to minimize the cross-entropy loss
between the ground truth $\mathrm{TRoI}$ label and softmax output.

Following [71], we include a graph normalization (GraphNorm) layer followed by
a batch normalization (BatchNorm) layer after each $\mathrm{PNA}$ layer. Graph
normalization scales the node feature representation by the number of nodes in
the graph. Intuitively, it prevents the node representations from graphs of
different sizes to be at different scales. This normalization helps the
network learning discriminative topological patterns when the number of nodes
varies significantly within a class.

Figure 5: Overview of class-wise samples in BRACS.

Figure 6: Overview of the variability in DCIS category in BRACS. TABLE I: Key statistics of the BRACS dataset. | Metric | Normal | Benign | UDH | ADH | FEA | DCIS | Invasive | Total
---|---|---|---|---|---|---|---|---|---
Image | Number of images | 512 | 758 | 471 | 568 | 783 | 749 | 550 | 4391
Number of pixels (in million) |  2.8$\pm$2.7 |  5.7$\pm$4.5 |  2.4$\pm$2.9 |  2.2$\pm$2.0 |  1.2$\pm$1.1 |  5.0$\pm$5.0 |  8.2$\pm$5.4 |  3.9$\pm$4.3
| Max/Min pixel ratio | 75.3 | 97.9 | 180.1 | 75.3 | 58.3 | 128.6 | 62.4 | 235.6
$\mathrm{CG}$ | Number of nodes |  994$\pm$732 |  1826$\pm$1547 |  903$\pm$910 |  863$\pm$730 |  470$\pm$352 |  1723$\pm$1598 |  3609$\pm$2393 |  1468$\pm$1642
Number of edges |  3759$\pm$2643 |  6103$\pm$5420 |  3371$\pm$3675 |  3098$\pm$2781 |  1738$\pm$1395 |  5728$\pm$5811 |  12490$\pm$10011 |  5102$\pm$6089
Max/Min node ratio | 71.9 | 126.6 | 133.3 | 104.2 | 45.2 | 161.3 | 113.6 | 256.4
$\mathrm{TG}$ | Number of nodes |  107$\pm$106 |  217$\pm$233 |  88$\pm$93 |  100$\pm$91 |  45$\pm$32 |  225$\pm$217 |  423$\pm$317 |  172$\pm$217
Number of edges |  509$\pm$545 |  1012$\pm$1236 |  393$\pm$450 |  480$\pm$474 |  194$\pm$159 |  1111$\pm$1123 |  2025$\pm$1741 |  815$\pm$1125
Max/Min node ratio | 169.5 | 312.5 | 125.0 | 178.6 | 416.7 | 312.5 | 101.0 | 434.8
Image split | Train | 342 | 586 | 303 | 405 | 599 | 562 | 366 | 3163
Validation | 86 | 87 | 88 | 77 | 85 | 97 | 82 | 602
Test | 84 | 85 | 80 | 86 | 99 | 90 | 102 | 626
WSI split | Train | 67 | 86 | 59 | 38 | 37 | 33 | 41 | 198
Validation | 28 | 24 | 24 | 28 | 17 | 21 | 19 | 68
Test | 15 | 16 | 20 | 17 | 12 | 16 | 16 | 59

##  V Datasets

BRACS dataset: As part of this work, we introduce a new dataset and term as
the BReAst Cancer Subtyping (BRACS) dataset. BRACS contains 4391
$\mathrm{TRoI}$s acquired from 325 H&E stained breast carcinoma WSIs. The WSIs
are selected from the archives of the Department of Pathology at National
Cancer Institute- IRCCS-Fondazione Pascale, Naples, Italy, and are scanned
with an Aperio AT2 scanner at 0.25 $\mu$m/pixel resolution. The
$\mathrm{TRoI}$s are selected and annotated as, Normal, Benign, Usual ductal
hyperplasia (UDH), Atypical ductal hyperplasia (ADH), Flat epithelial atypia
(FEA), Ductal carcinoma in situ (DCIS) and Invasive using QuPath software
[72]. The $\mathrm{TRoI}$s are annotated independently by three pathologists.
$\mathrm{TRoI}$s with conflict in annotations are further discussed and
annotated by the consensus of the three pathologists. Note that the
pathologists utilize the entire WSI-level context during the $\mathrm{TRoI}$
annotation. Figure 5 presents sample $\mathrm{TRoI}$s from all the cancer
subtypes in BRACS. Figure 6 demonstrates a few DCIS samples in BRACS to depict
the appearance variability per category. Figure 6(a,b,c) display the
variability in the $\mathrm{TRoI}$ sizes. Figure 6(d,e) display the
variability in staining appearance. Figure 6(f,g,h,i) display different
patterns of low-grade, moderate-grade and high-grade DCIS in the dataset,
i.e., Papillary DCIS, Cribriform DCIS, Solid DCIS and Comedo DCIS
respectively. Figure 6(j,k) present DCIS $\mathrm{TRoI}$s with single and
multiple glandular regions. Figure 6(l,m,n) presents some notable artifacts in
tissue and slide preparation, such as tissue-fold, tears, ink stains and blur.
Similar variability in $\mathrm{TRoI}$s is maintained for other cancer
subtypes in the dataset to represent wholesome realistic scenarios for each
category.

Table I presents category-wise statistics of $\mathrm{TRoI}$s in BRACS. The
image statistics demonstrate a high variation in the $\mathrm{TRoI}$
dimensions. Additionally, we present statistics for the $\mathrm{CG}$ and
$\mathrm{TG}$ representations that are constructed as part of the proposed
framework. These statistics indicate a high variation in the size of
constructed graph representations. To evaluate our framework on BRACS, we
partition the $\mathrm{TRoI}$s into train, validation, and test sets at WSI-
level, such that two $\mathrm{TRoI}$s from the same WSI do not belong to
different sets. The WSI-level splitting is performed randomly to maintain a
comparable number of $\mathrm{TRoI}$s per cancer subtype. The partitioning
ensures a fair development and evaluation of computer-aided diagnostic methods
on the dataset.

BACH dataset: We also evaluate the proposed methodology on the publicly
available microscopy image dataset from the Grand Challenge on BreAst Cancer
Histology images BACH [16]. The dataset consists of 400 training and 100 test
images from four breast cancer subtypes, i.e., Normal, Benign, DCIS, and
Invasive. All images are acquired using a Leica DM 2000 LED microscope and a
Leica ICC50 HD camera. The provided images are on RGB .tiff format and have a
size of 2048 $\times$ 1536 pixels and a pixel scale of 0.42 $\mu$m $\times$
0.42 $\mu$m. Notably, the introduced BRACS dataset has three major advantages
over the BACH dataset.

  * •

The train and test sets of BRACS are nearly 10 times and 6 times the size of
the train and test sets of BACH respectively. The large test set ensures a
reliable evaluation of computer-aided diagnostic methods.

  * •

BRACS includes diagnostically complex pre-cancerous atypical (ADH, FEA)
category that possesses a great diagnostic relevance due to its high risk of
becoming cancerous. The seven cancer subtypes in BRACS represent a broad
spectrum of breast pathology.

  * •

The aforementioned high variability in the BRACS dataset in terms of
$\mathrm{TRoI}$ appearances and dimensions presents a realistic scenario for
breast cancer subtyping.

##  VI Results

In this section, we comprehensively evaluate the proposed methodology for
breast cancer subtyping. First, we introduce several state-of-the-art
$\mathrm{CNN}$ and $\mathrm{GNN}$ baselines and their implementation
strategies. Second, we perform ablation studies on BRACS dataset to examine
the impact of various components in the proposed methodology. Third, we
evaluate the classification performance of our methodology and compare with
the stated baselines on BRACS and BACH datasets for different classification
settings. Finally, we propose a human comparison by analyzing the performance
of $\mathrm{HACT}$-Net against three independent expert-pathologists.

###  VI-A Baseline methodologies

The evaluated $\mathrm{CNN}$ and $\mathrm{GNN}$ baselines are:

$\bullet$ Single-scale CNN: It processes $\mathrm{TRoI}$s at a single
magnification. A $\mathrm{CNN}$ is trained to predict patch-wise cancer
subtypes and aggregate the patch-wise predictions to produce a
$\mathrm{TRoI}$-level prediction. We experiment for three scales, i.e.,
10$\times$, 20$\times$ and 40$\times$, and denoted as,
$\mathrm{CNN}$(10$\times$), $\mathrm{CNN}$(20$\times$) and
$\mathrm{CNN}$(40$\times$). Same network architecture and training strategy is
employed for all scales. For each scale, we extract patches of size
128$\times$128 pixels with stride 64 pixels. The $\mathrm{CNN}$ follows the
single-scale training procedure from [23] and patch-wise predictions are
aggregated using the Agg-Penultimate strategy proposed by [19]. We use
transfer learning with a ResNet-50 architecture, pre-trained on ImageNet
dataset, as our $\mathrm{CNN}$ backbone. Following the feature extraction from
ResNet-50, we employ a two-layer $\mathrm{MLP}$ of 128 channels to classify
the patches. The ResNet-50 parameters are fine-tuned to improve
classification. Adam optimizer [73] with $10^{-3}$ learning rate and batch
size 16 is used to optimize the categorical cross-entropy objective.

$\bullet$ Multi-scale CNN: This baseline processes the $\mathrm{TRoI}$s at
multiple scales. We extract concentric patches of size 128$\times$128 pixels
from multiple magnifications and follow the “Late fusion with single-stream +
$\mathrm{LSTM}$” training procedure from [23]. The multi-scale approach uses
concentric patches to acquire context information from multiple magnifications
for improving the patch classification. We operate at two settings, i.e.,
(10$\times$ \+ 20$\times$) setting and (10$\times$ \+ 20$\times$ \+
40$\times$) setting, and denote them as Multi-scale $\mathrm{CNN}$(10$\times$
\+ 20$\times$) and Multi-scale $\mathrm{CNN}$(10$\times$ \+ 20$\times$ \+
40$\times$). The patch-wise predictions are aggregated using the Agg-
Penultimate strategy [19]. Following the concatenated feature representation
from the $\mathrm{LSTM}$, we employ a two-layer $\mathrm{MLP}$ of 128 channels
to classify the patches. The training strategy and hyperparameters are the
same as Single-scale CNN.

$\bullet$ CGC-Net: We implement the Cell Graph Convolutional Network (CGC-Net)
proposed by [40], the state-of-the-art method for classifying $\mathrm{CG}$
representations for $\mathrm{TRoI}$s. We construct the $\mathrm{CG}$ topology
for a $\mathrm{TRoI}$ using thresholded kNN strategy presented in Section
IV-B1. We initialize the $\mathrm{CG}$ nodes with hand-crafted features,
employ the Adaptive GraphSage-based CGC-Net architecture, and follow the
training strategy proposed by [40].

$\bullet$ Patch-GNN: This baseline implements the methodology proposed by
[43], the state-of-the-art $\mathrm{GNN}$ method for classifying patch-graph
representations of $\mathrm{TRoI}$s. The methodology incorporates local inter-
patch context through a $\mathrm{GNN}$ to construct a graph-level feature
representation which is further processed by a $\mathrm{MLP}$ to classify the
$\mathrm{TRoI}$. We experiment with Patch-$\mathrm{GNN}$ at three scales,
i.e., 10$\times$, 20$\times$ and 40$\times$, and denote the baselines as,
Patch$-\mathrm{GNN}$(10$\times$), Patch$-\mathrm{GNN}$(20$\times$) and
Patch$-\mathrm{GNN}$(40$\times$). At each magnification, we extract patches of
size 128$\times$128 to construct a $\mathrm{TRoI}$ specific patch-graph. We
follow the same network architecture and training strategy as proposed by
[43].

$\bullet$ CG-GNN: This baseline is implemented to compare the proposed
hierarchical learning with standalone cell-graph based learning.
$\mathrm{CG}$-$\mathrm{GNN}$ architecture utilizes $\mathrm{PNA}$ layers, an
$\mathrm{LSTM}$-based jumping knowledge, sum readout, and a two-layer
$\mathrm{MLP}$ classifier. We follow the same $\mathrm{CG}$ representation
strategy as proposed in Section IV-B1.

$\bullet$ TG-GNN: We implement this baseline to compare the proposed
hierarchical learning with standalone tissue graph-based learning.
$\mathrm{TG}$-$\mathrm{GNN}$ employs the same architecture as the
$\mathrm{CG}$-$\mathrm{GNN}$. The $\mathrm{TG}$ node features are directly
initialized by $H_{\,\mathrm{TG}}$ instead of Equation (7).

$\bullet$ CONCAT-GNN: This baseline is implemented to evaluate the impact of
hierarchical graph representation and learning.
$\mathrm{CONCAT}$-$\mathrm{GNN}$ utilizes standalone $\mathrm{CG}$ and
$\mathrm{TG}$ representations as input to standalone
$\mathrm{CG}$-$\mathrm{GNN}$ and $\mathrm{TG}$-$\mathrm{GNN}$ respectively to
produce $h_{\mathrm{CG}}$ and $h_{\mathrm{TG}}$ graph-level embeddings. The
$\mathrm{TRoI}$ level embedding is constructed by concatenating the graph-
level embeddings, i.e., $h_{\,\mathrm{CONCAT}}$ =
$\mathrm{CONCAT}(\,h_{\mathrm{CG}},\,h_{\,\mathrm{TG}})$. Finally, a two-layer
$\mathrm{MLP}$ classifies $h_{\,\mathrm{CONCAT}}$ into cancer subtype.

###  VI-B Implementation

Resources: All the experiments are conducted using PyTorch [74] and Deep Graph
Library (DGL) [75]. We use NVIDIA Tesla P100 GPUs with POWER8 processors to
run the experiments.

Graph representation hyperparameters: $\mathrm{CG}$ representations (Section
IV-B1) use, i) patches of size 72$\times$72, and ii) $\mathrm{CNN}$:
{ResNet-34, ResNet-50} to initialize node features. $\mathrm{TG}$
representations (Section IV-B2) use, i) patches of size 144$\times$144, and
ii) $\mathrm{CNN}$: {ResNet-34, ResNet-50} to initialize node features.

Graph architecture and learning hyperparameters: $\mathrm{CG}$-$\mathrm{GNN}$,
$\mathrm{TG}$-$\mathrm{GNN}$, $\mathrm{CONCAT}$-$\mathrm{GNN}$ and
$\mathrm{HACT}$-Net share the same hyperparameters and respective options,
i.e., ,

  * •

# $\mathrm{PNA}$ layers in $\mathrm{GNN}$: [3, 4, 5]

  * •

# $\mathrm{MLP}$ layers in a $\mathrm{PNA}$ layer: 2

  * •

# channels in a $\mathrm{PNA}$ layer $\mathrm{MLP}$: 64

  * •

Graph-level embedding dimension: 128

  * •

# $\mathrm{MLP}$ layers in output classifier: 2

  * •

# channels in output $\mathrm{MLP}$ classifier: 128

  * •

Training parameters: Adam optimizer [73] with $10^{-3}$ learning rate, 16
batch size and a categorical cross-entropy objective.

Evaluation metrics: Considering the imbalanced number of $\mathrm{TRoI}$s per
class in train, validation and test set (see Table I), we use weighted
F1-score to evaluate the classification performance. The class-wise weights
are computed using the class-wise number of $\mathrm{TRoI}$s in each set.
During training, networks with the best validation weighted F1-scores are
selected as the final trained models. To ensure the statistical significance
of our results, all the models are trained three times using random weight
initializations. The reported scores are calculated as the mean and standard
deviation over the three runs. Further, we present precision, recall and
confusion matrix to indicate the distribution of the predictions across
different classes.

###  VI-C Ablation studies

We conduct thorough ablation studies to evaluate the impact of three
components in the proposed methodology on $\mathrm{TRoI}$ classification
performance. The components are, i) node feature initialization, ii)
$\mathrm{GNN}$ layer type, and iii) jumping knowledge technique. Each
component is analyzed individually, while fixing the other components. The
ablation studies are performed on the BRACS dataset for classifying the
$\mathrm{TRoI}$s into seven classes.

####  VI-C1 Impact of node feature initialization

The performance of $\mathrm{GNN}$s eminently rely on the initial node features
[51]. In context of entity-based tissue analysis, we focus on the impact of
initial morphological features embedded in the nodes. To this end, we
experiment with three morphological feature initialization settings, given as,

$\bullet$ No morphological features: In this setting, the nodes of an entity-
graph representation are initialized with only spatial features. Experiments
with this setting demonstrate the impact of standalone graph topology on the
$\mathrm{GNN}$ performance.

TABLE II: Ablation: Impact of node features. Mean and standard deviation of 7-class weighted F1-scores. Results expressed in $\%$. | Weighed F1
---|---
CG-$\mathrm{GNN}$: No morphological features |  45.24$\pm$1.51
CG-$\mathrm{GNN}$: Hand-crafted morphological features |  48.34$\pm$5.22
CG-$\mathrm{GNN}$: $\mathrm{CNN}$ morphological features | 55.94$\pm$1.01
TG-$\mathrm{GNN}$: No morphological features |  36.81$\pm$0.71
TG-$\mathrm{GNN}$: Hand-crafted morphological features |  51.62$\pm$2.11
TG-$\mathrm{GNN}$: $\mathrm{CNN}$ morphological features | 56.62$\pm$1.35
CONCAT-$\mathrm{GNN}$: No morphological features |  47.62$\pm$1.56
CONCAT-$\mathrm{GNN}$: Hand-crafted morphological features |  51.55$\pm$1.32
CONCAT-$\mathrm{GNN}$: $\mathrm{CNN}$ morphological features | 57.01$\pm$2.27
$\mathrm{HACT}$-Net: No morphological features |  48.70$\pm$0.16
$\mathrm{HACT}$-Net: Hand-crafted morphological features |  52.46$\pm$0.19
$\mathrm{HACT}$-Net: $\mathrm{CNN}$ morphological features | 61.53$\pm$0.87

$\bullet$ Hand-crafted morphological features: The entity-graph nodes are
initialized with hand-crafted morphological features, i.e., i) texture
features: average foreground and background difference; standard deviation,
skewness and mean entropy of intensity; dissimilarity, homogeneity, energy and
angular second moment from Gray-Level Co-occurrence Matrix, and ii) shape
features: eccentricity, area, maximum and minimum length of axis, perimeter,
solidity and orientation. Note that, for $\mathrm{CG}$ and $\mathrm{TG}$, the
hand-crafted features are computed from the segmented instances of the nuclei
and tissue regions respectively.

$\bullet$ CNN morphological features: The morphological features of the graph
nodes are initialized with $\mathrm{CNN}$ features (ResNet-34 pre-trained on
ImageNet) extracted from patches around the centroids of the nodes.

Results of the experiments in Table II indicate that the $\mathrm{CG}$
topology alone is more discriminative for cancer subtyping than $\mathrm{TG}$
topology. The combination of $\mathrm{CG}$ and $\mathrm{TG}$ topologies
further improves the discriminability. The best performance with
$\mathrm{HACT}$ topology confirms the positive impact of hierarchical
representation. Further, including morphological features significantly
improves the discriminability of the graph representations. The superiority of
graphs with $\mathrm{CNN}$-based morphological features indicates the richness
of morphological information acquired from $\mathrm{CNN}$ compared to hand-
crafted measures.

TABLE III: Ablation: Impact of $\mathrm{GNN}$ layer. Mean and standard deviation of 7-class weighted F1-scores. Results expressed in $\%$. | Weighed F1
---|---
CG-$\mathrm{GNN}$: GIN |  55.70$\pm$0.51
CG-$\mathrm{GNN}$: PNA | 55.94$\pm$1.01
TG-$\mathrm{GNN}$: GIN |  55.33$\pm$1.36
TG-$\mathrm{GNN}$: PNA | 56.62$\pm$1.35
CONCAT-$\mathrm{GNN}$: GIN |  56.20$\pm$2.12
CONCAT-$\mathrm{GNN}$: PNA | 57.01$\pm$2.27
$\mathrm{HACT}$-Net: GIN |  59.73$\pm$1.20
$\mathrm{HACT}$-Net: PNA | 61.53$\pm$0.87
TABLE IV: Ablation: Impact of $\mathrm{GNN}$ jumping knowledge strategy. Mean and standard deviation of 7-class weighted F1-scores. Results expressed in $\%$. | Weighed F1
---|---
CG-$\mathrm{GNN}$: No aggregator |  55.53$\pm$0.75
CG-$\mathrm{GNN}$: Concatenation |  55.82$\pm$0.97
CG-$\mathrm{GNN}$: LSTM | 55.94$\pm$1.01
TG-$\mathrm{GNN}$: No aggregator |  55.30$\pm$0.81
TG-$\mathrm{GNN}$: Concatenation |  56.07$\pm$0.80
TG-$\mathrm{GNN}$: LSTM | 56.62$\pm$1.35
CONCAT-$\mathrm{GNN}$: No aggregator |  57.67$\pm$4.66
CONCAT-$\mathrm{GNN}$: Concatenation |  56.28$\pm$2.75
CONCAT-$\mathrm{GNN}$: LSTM | 57.01$\pm$2.27
$\mathrm{HACT}$-Net: No aggregator |  49.16$\pm$1.15
$\mathrm{HACT}$-Net: Concatenation |  59.78$\pm$1.59
$\mathrm{HACT}$-Net: LSTM | 61.53$\pm$0.87

####  VI-C2 Impact of $\mathrm{GNN}$ layer type

We investigate the impact of two state-of-the-art $\mathrm{GNN}$ layers, i.e.,
$\mathrm{GIN}$ and $\mathrm{PNA}$ (Figure 1), on the classification
performance. The experiments use $\mathrm{CNN}$-based node feature
initialization and $\mathrm{LSTM}$-based jumping knowledge to compute the
graph-level embedding. The ablation results in Table III demonstrate that
$\mathrm{GNN}$ with $\mathrm{PNA}$ layer outperform $\mathrm{GNN}$ with
$\mathrm{GIN}$ layer for all four $\mathrm{GNN}$s in this work. It can be
accounted to the higher expressive power of $\mathrm{PNA}$ layer that operates
on graphs with continuous node features. The series of aggregators with
degree-scalers in a $\mathrm{PNA}$ layer includes additional network
parameters compared to single aggregator-based $\mathrm{GIN}$ layer, which
allows more flexibility and learning ability in the network.

TABLE V: Mean and standard deviation of class-wise F1-scores, and 7-class weighted F1-scores. Results expressed in $\%$. The best result is highlighted in bold and the second best is underlined. | Method | Normal | Benign | UDH | ADH | FEA | DCIS | Invasive | Weighted F1
---|---|---|---|---|---|---|---|---|---
$\mathrm{CNN}$ | $\mathrm{CNN}(10\times)$ |  48.67$\pm$1.71 |  44.33$\pm$1.89 | 45.00$\pm$4.97 |  24.00$\pm$2.83 |  47.00$\pm$4.32 |  53.33$\pm$2.62 | 86.67$\pm$2.64 |  50.85$\pm$2.64
$\mathrm{CNN}(20\times)$ |  42.00$\pm$2.16 |  42.33$\pm$3.09 |  39.33$\pm$2.05 |  22.67$\pm$2.49 |  47.67$\pm$1.25 |  50.33$\pm$3.09 |  77.00$\pm$1.41 |  46.85$\pm$2.19
$\mathrm{CNN}(40\times)$ |  32.33$\pm$4.64 |  39.00$\pm$0.82 |  23.67$\pm$1.70 |  18.00$\pm$0.82 |  37.67$\pm$2.87 |  47.33$\pm$2.05 |  70.67$\pm$0.47 |  39.41$\pm$1.89
Multi-scale $\mathrm{CNN}(10\times+20\times)$ |  48.33$\pm$2.05 |  45.67$\pm$0.47 |  41.67$\pm$4.99 |  32.33$\pm$0.94 |  46.33$\pm$1.41 |  59.33$\pm$2.05 |  85.67$\pm$1.89 |  52.27$\pm$1.93
Multi-scale $\mathrm{CNN}(10\times+20\times+40\times)$ |  50.33$\pm$0.94 |  44.33$\pm$1.25 |  41.33$\pm$2.49 |  31.67$\pm$3.30 |  51.67$\pm$3.09 |  57.33$\pm$0.94 |  86.00$\pm$1.41 |  52.83$\pm$1.92
$\mathrm{GNN}$ | CGG-Net |  30.83$\pm$5.33 |  31.63$\pm$4.66 |  17.33$\pm$3.38 |  24.5$\pm$5.24 |  58.97$\pm$3.56 |  49.36$\pm$3.41 |  75.30$\pm$3.20 |  43.63$\pm$0.51
Patch-$\mathrm{GNN}(10\times)$ |  52.53$\pm$3.27 |  47.57$\pm$2.25 |  23.67$\pm$4.65 |  30.66$\pm$1.79 |  60.73$\pm$5.35 |  58.76$\pm$1.15 |  81.63$\pm$2.17 |  52.10$\pm$0.61
Patch-$\mathrm{GNN}(20\times)$ |  43.86$\pm$4.23 |  43.37$\pm$3.21 |  19.47$\pm$2.31 |  25.73$\pm$2.87 |  55.57$\pm$2.08 |  52.86$\pm$1.85 |  79.2$\pm$1.04 |  47.10$\pm$0.70
Patch-$\mathrm{GNN}(40\times)$ |  41.7$\pm$3.06 |  32.93$\pm$1.04 |  25.07$\pm$3.74 |  25.63$\pm$2.01 |  49.47$\pm$3.46 |  48.6$\pm$4.23 |  71.57$\pm$5.15 |  43.23$\pm$0.57
Ours |  CG-$\mathrm{GNN}$ |  58.77$\pm$6.82 |  40.87$\pm$3.05 | 46.82$\pm$1.95 | 39.99$\pm$3.56 |  63.75$\pm$10.48 |  53.81$\pm$3.89 |  81.06$\pm$3.33 |  55.94$\pm$1.01
TG-$\mathrm{GNN}$ | 63.59$\pm$4.88 | 47.73$\pm$2.87 |  39.41$\pm$4.70 |  28.51$\pm$4.29 | 72.15$\pm$1.35 |  54.57$\pm$2.23 |  82.21$\pm$3.99 |  56.62$\pm$1.35
CONCAT-$\mathrm{GNN}$ |  60.97$\pm$4.54 |  43.06$\pm$2.26 |  41.96$\pm$4.67 |  26.10$\pm$3.73 |  71.29$\pm$2.09 | 60.83$\pm$3.71 |  85.42$\pm$2.70 | 57.01$\pm$2.27
$\mathrm{HACT}$-Net (Proposed) | 61.56$\pm$2.15 | 47.49$\pm$2.94 |  43.60$\pm$1.86 | 40.42$\pm$2.55 | 74.22$\pm$1.41 | 66.44$\pm$2.57 | 88.40$\pm$0.19 | 61.53$\pm$0.87

(a)

(b)

Figure 7: 7-class. Precision and Recall (mean, standard deviation) for CG-GNN,
TG-GNN, CONCAT-GNN and HACT-Net.

####  VI-C3 Impact of jumping knowledge technique

To investigate the impact of the jumping knowledge technique, we experiment
with three settings, i.e., no jumping knowledge, $\mathrm{CONCAT}$-based, and
$\mathrm{LSTM}$-based. $\mathrm{LSTM}$-based technique follows Equation (6).
The $\mathrm{CONCAT}$-based technique replaces $\mathrm{LSTM}$ with
concatenation operation in Equation (6). The experiments use
$\mathrm{CNN}$-based node feature initialization and $\mathrm{PNA}$ layer in
the $\mathrm{GNN}$s. The ablation results in Table IV indicate the positive
impact of the jumping knowledge technique. Compared to
$\mathrm{CONCAT}$-based, the $\mathrm{LSTM}$-based technique learns better
dependencies between $\mathrm{GNN}$ layers, thus generates more discriminable
graph-level feature representations.

####  VI-C4 Ablation summary

The ablation experiments conclude the following choice of components for
designing our methodology, i) $\mathrm{CNN}$-based initialization of node-
level morphological features, ii) $\mathrm{PNA}$ layers, and iii) an
$\mathrm{LSTM}$-based jumping knowledge technique.

###  VI-D Classification results on BRACS dataset

In this section, we evaluate the proposed methodology on BRACS dataset for
$\mathrm{TRoI}$ classification task and compare the classification performance
with $\mathrm{CNN}$ and $\mathrm{GNN}$ baselines. We analyze the performance
for three classification settings, i.e.,

$\bullet$ Setting 1: 7-class classification: This setting classifies
$\mathrm{TRoI}$s into 7-classes, i.e., Normal, Benign, UDH, ADH, FEA, DCIS and
Invasive, to differentiate the spectrum of breast cancer subtypes. Table V
highlights the classification performance of $\mathrm{HACT}$-Net,
$\mathrm{CNN}$ and $\mathrm{GNN}$ baselines. $\mathrm{CNN}$(10$\times$) is the
best performing single-scale $\mathrm{CNN}$ indicating the importance of
global context information for $\mathrm{TRoI}$ classification. Multi-scale
$\mathrm{CNN}$s using both global and local context information outperforms
single scale $\mathrm{CNN}$s. The gain is significant for ADH, FEA and DCIS
categories that require both local and global context information for the
diagnosis. Multi-scale $\mathrm{CNN}$s also outperforms CGC-Net and
Patch-$\mathrm{GNN}$ baselines. Interestingly for each magnification,
Patch-$\mathrm{GNN}$ outperforms single-scale $\mathrm{CNN}$. It conveys the
importance of relational and topological distribution information of patches
for $\mathrm{TRoI}$ classification.

TABLE VI: Mean and standard deviation of class-wise F1-scores, and 4-class weighted F1-scores. Results expressed in $\%$. The best result is highlighted in bold and the second best is underlined. | Method | Normal | Non-cancerous | Precancerous | Cancerous | Weighted F1
---|---|---|---|---|---|---
$\mathrm{CNN}$ | $\mathrm{CNN}(10\times)$ |  54.33$\pm$3.68 |  56.00$\pm$0.82 |  56.33$\pm$1.25 |  83.67$\pm$0.94 |  64.36$\pm$1.37
$\mathrm{CNN}(20\times)$ |  45.33$\pm$4.64 |  55.33$\pm$0.47 |  52.33$\pm$1.89 |  81.67$\pm$2.05 |  61.18$\pm$1.93
$\mathrm{CNN}(40\times)$ |  42.00$\pm$4.89 |  51.00$\pm$0.82 |  47.67$\pm$4.11 |  77.67$\pm$2.05 |  56.99$\pm$2.72
Multi-scale $\mathrm{CNN}(10\times+20\times)$ |  51.67$\pm$5.79 |  55.33$\pm$1.25 |  52.67$\pm$2.87 |  80.67$\pm$1.89 |  61.82$\pm$2.53
Multi-scale $\mathrm{CNN}(10\times+20\times+40\times)$ |  51.33$\pm$3.27 |  56.33$\pm$2.05 |  57.00$\pm$1.64 |  81.33$\pm$3.68 |  63.52$\pm$2.59
$\mathrm{GNN}$ | CGG-Net |  34.53$\pm$2.93 |  47.23$\pm$3.72 |  62.9$\pm$2.81 |  82.2$\pm$1.04 |  59.87$\pm$2.30
Patch-$\mathrm{GNN}(10\times)$ |  53.13$\pm$4.40 |  46.23$\pm$2.45 |  63.96$\pm$3.82 |  77.43$\pm$3.22 |  61.93$\pm$2.51
Patch-$\mathrm{GNN}(20\times)$ |  53.46$\pm$1.81 |  47.16$\pm$2.81 |  63.20$\pm$3.78 |  74.90$\pm$3.36 |  61.26$\pm$2.90
Patch-$\mathrm{GNN}(40\times)$ |  40.90$\pm$2.75 |  38.67$\pm$2.76 |  56.77$\pm$3.91 |  72.20$\pm$2.61 |  54.60$\pm$1.9
Ours |  CG-$\mathrm{GNN}$ |  52.95$\pm$12.11 | 56.55$\pm$3.70 |  61.53$\pm$3.03 | 84.47$\pm$0.87 |  66.10$\pm$2.58
TG-$\mathrm{GNN}$ |  52.96$\pm$6.81 |  56.52$\pm$2.85 | 64.36$\pm$1.05 |  82.21$\pm$0.78 | 66.24$\pm$1.11
CONCAT-$\mathrm{GNN}$ | 54.54$\pm$1.64 | 56.63$\pm$1.68 |  62.58$\pm$1.45 |  81.80$\pm$0.77 |  65.83$\pm$0.04
$\mathrm{HACT}$-Net (Proposed) | 66.08$\pm$3.69 |  55.28$\pm$1.74 | 66.21$\pm$0.87 | 84.91$\pm$0.79 | 69.04$\pm$0.46

Analyzing the proposed $\mathrm{GNN}$s, we observe that
$\mathrm{CG}$-$\mathrm{GNN}$ significantly beats CGC-Net concluding the
superiority of $\mathrm{CNN}$-based node feature initialization over
handcrafted features, and the significance of $\mathrm{GNN}$ with expressive
$\mathrm{PNA}$ layers over Adaptive GraphSage in CGC-Net. We notice that
$\mathrm{CG}$-$\mathrm{GNN}$ and $\mathrm{TG}$-$\mathrm{GNN}$ provide overall
comparable performance. However, $\mathrm{TG}$-$\mathrm{GNN}$ performs better
for Normal, Benign, and FEA subtypes indicating the utility of tissue
microenvironment information for these categories. In contrast,
$\mathrm{CG}$-$\mathrm{GNN}$ surpasses $\mathrm{TG}$-$\mathrm{GNN}$ for UDH
and ADH that rely on local nuclei-level information. For DCIS and Invasive,
both $\mathrm{CG}$-$\mathrm{GNN}$ and $\mathrm{TG}$-$\mathrm{GNN}$ perform
similar showing that both low-level and high-level information is useful for
these categories. Further, both $\mathrm{HACT}$-Net and
$\mathrm{CONCAT}$-$\mathrm{GNN}$ provide overall superior performance compared
to all $\mathrm{CNN}$ and $\mathrm{GNN}$ baselines. $\mathrm{HACT}$-Net
significantly outperforms $\mathrm{CONCAT}$-$\mathrm{GNN}$ indicating the
significance of the proposed hierarchical modeling and learning.
$\mathrm{CONCAT}$-$\mathrm{GNN}$ produces overall comparable performance to
$\mathrm{CG}$-$\mathrm{GNN}$ and $\mathrm{TG}$-$\mathrm{GNN}$, however, in
terms of class-wise performance, $\mathrm{CONCAT}$-$\mathrm{GNN}$ seems to
utilize complementary information from $\mathrm{CG}$ and $\mathrm{TG}$
representation. The complementary information is effectively utilized by
$\mathrm{HACT}$-Net to improve the per-class and the overall classification
performance. In general, all the proposed $\mathrm{GNN}$s comprehensively
outperform the $\mathrm{CNN}$ baselines establishing the potential of entity-
based analysis in digital pathology.

Figure 7 presents the per-class precision and recall for
$\mathrm{CG}$-$\mathrm{GNN}$, $\mathrm{TG}$-$\mathrm{GNN}$,
$\mathrm{CONCAT}$-$\mathrm{GNN}$ and $\mathrm{HACT}$-Net. $\mathrm{HACT}$-Net
is observed to produce the best precision values for most of the classes. The
ranking of per-class recall values for $\mathrm{CG}$-$\mathrm{GNN}$ and
$\mathrm{TG}$-$\mathrm{GNN}$ are inconsistent, whereas, $\mathrm{HACT}$-Net
consistently results in better recall values. Further, the standard deviation
of class-wise precision and recall values are persistently the lowest for
$\mathrm{HACT}$-Net for most of the classes. Figure 8 demonstrates the 7-class
confusion matrix of precision and recall values for $\mathrm{HACT}$-Net. The
network presents high ambiguity in precision between i) Normal and Benign, ii)
UDH and ADH, and iii) ADH and DCIS. Additionally, intermediate ambiguity in
recall appears for Benign, UDH, and ADH with FEA. Notably, these pair-wise
classes bear high pathological ambiguity and are diagnostically very
challenging.

Figure 8: 7-class. Precision and Recall (mean $\pm$ std) confusion matrix for HACT-Net. TABLE VII: Mean and standard deviation of Binary-class weighted F1-scores. Results expressed in $\%$. The best result is highlighted in bold and the second best is underlined. | Method | I vs | N+B+U vs | N vs | B vs | A+F vs | A vs
---|---|---|---|---|---|---|---
|  | N+B+A+U+F+D | A+F+D | B+U | U | D | F
$\mathrm{CNN}$ | $\mathrm{CNN}(10\times)$ |  95.66$\pm$0.48 |  81.24$\pm$0.42 |  69.83$\pm$0.38 |  76.12$\pm$1.13 |  73.44$\pm$2.56 |  77.59$\pm$1.73
$\mathrm{CNN}(20\times)$ |  92.39$\pm$0.37 |  80.84$\pm$0.36 |  66.52$\pm$2.14 |  74.75$\pm$1.51 |  67.87$\pm$1.82 |  71.78$\pm$2.53
$\mathrm{CNN}(40\times)$ |  90.74$\pm$0.59 |  79.92$\pm$1.66 |  62.36$\pm$2.14 |  68.13$\pm$4.30 |  64.86$\pm$2.98 |  66.91$\pm$1.68
Multi-scale $\mathrm{CNN}(10\times+20\times)$ |  94.31$\pm$1.26 |  80.89$\pm$1.31 |  67.99$\pm$1.86 |  75.58$\pm$2.06 |  72.07$\pm$1.85 |  76.91$\pm$2.22
Multi-scale $\mathrm{CNN}(10\times+20\times+40\times)$ |  95.12$\pm$1.15 |  82.21$\pm$0.34 |  70.87$\pm$2.07 |  72.89$\pm$2.26 |  72.08$\pm$3.17 |  75.47$\pm$3.69
$\mathrm{GNN}$ | CGG-Net |  91.60$\pm$2.09 |  79.73$\pm$1.53 |  63.67$\pm$3.12 |  62.37$\pm$3.00 |  81.56$\pm$1.56 |  73.80$\pm$5.41
Patch-$\mathrm{GNN}(10\times)$ |  95.8$\pm$0.43 |  76.53$\pm$0.32 |  72.57$\pm$1.10 |  72.87$\pm$3.07 |  77.17$\pm$0.85 |  78.26$\pm$2.60
Patch-$\mathrm{GNN}(20\times)$ |  93.7$\pm$0.36 |  76.63$\pm$1.40 |  70.1$\pm$1.90 |  69.77$\pm$3.13 |  74.10$\pm$0.10 |  81.03$\pm$1.85
Patch-$\mathrm{GNN}(40\times)$ |  92.4$\pm$0.95 |  74.43$\pm$0.64 |  71.1$\pm$1.74 |  67.4$\pm$2.46 |  72.97$\pm$0.66 |  76.4$\pm$1.95
Ours |  CG-$\mathrm{GNN}$ (Ours) |  94.52$\pm$0.43 | 83.79$\pm$0.31 | 75.71$\pm$1.68 |  73.15$\pm$3.32 |  77.48$\pm$1.68 |  84.33$\pm$0.54
TG-$\mathrm{GNN}$ | 96.00$\pm$0.56 |  80.38$\pm$0.80 |  69.51$\pm$3.12 | 76.12$\pm$0.99 | 80.67$\pm$0.22 |  84.18$\pm$3.56
CONCAT-$\mathrm{GNN}$ |  95.91$\pm$0.56 |  83.21$\pm$0.68 |  71.84$\pm$1.46 |  75.67$\pm$1.81 |  80.14$\pm$2.60 | 88.88$\pm$3.86
$\mathrm{HACT}$-Net (Proposed) | 96.32$\pm$0.64 | 83.63$\pm$0.73 | 76.84$\pm$0.68 | 77.66$\pm$0.37 | 81.11$\pm$0.72 | 89.35$\pm$0.26

$\bullet$ Setting 2: 4-class classification: This setting categorizes
$\mathrm{TRoI}$s into 4-classes based on the risk of attaining cancerous
state. We group the $\mathrm{TRoI}$s in BRACS into four pathologically
relevant categories, i.e., Normal, Non-cancerous (Benign + UDH), Precancerous
(ADH + FEA), and Cancerous (DCIS + Invasive). Classification performance of
$\mathrm{HACT}$-Net, $\mathrm{CNN}$ and $\mathrm{GNN}$ baselines are presented
in Table VI. Single scale $\mathrm{CNN}$s exhibit the same behavior as in the
7-class setting. However, combining multiple magnifications in multi-scale
$\mathrm{CNN}$s does not improve the classification performance over the
single scale $\mathrm{CNN}$s. CGC-Net performs inferior to the $\mathrm{CNN}$
baselines and the best Patch-$\mathrm{GNN}$ results in comparable performance
to the $\mathrm{CNN}$s. Similar to the 7-class classification setting,
$\mathrm{CG}$-$\mathrm{GNN}$, $\mathrm{TG}$-$\mathrm{GNN}$ and
$\mathrm{CONCAT}$-$\mathrm{GNN}$ provide similar classification performance
and outperform the $\mathrm{CNN}$ and $\mathrm{GNN}$ baselines.
$\mathrm{HACT}$-Net significantly outperforms the $\mathrm{CNN}$ and
$\mathrm{GNN}$ baselines. Analysis of the per-class performance indicates that
$\mathrm{HACT}$-Net provides the best classification performance for Normal,
Precancerous, and Cancerous categories. To highlight, $\mathrm{HACT}$-Net
achieves $66\%$ F1-score for the diagnostically challenging Precancerous
category.

$\bullet$ Setting 3: Binary classifications: In this setting, we follow a
diagnostic decision tree, presented in Figure 9, to make one diagnosis at a
time. It is inspired by the classification scheme presented by [76] which
follows the pathologist’s approach for breast cancer subtyping. The individual
binary tasks are less constrained compared to multi-class classification
tasks, thereby presenting better discrimination between a selected pair of
classes. The binary classifiers can assist pathologists in categorizing
ambiguous cases while traversing the decision tree. Table VII highlights the
classification performance of $\mathrm{HACT}$-Net, $\mathrm{CNN}$ and
$\mathrm{GNN}$ baselines for six binary classification tasks in the decision
tree. Performance of the networks for individual binary tasks is consistent
with the 7-class and 4-class classification settings. $\mathrm{HACT}$-Net
consistently outperforms the $\mathrm{CNN}$ and $\mathrm{GNN}$ baselines.
These analyses establish the superiority of the proposed $\mathrm{HACT}$
representation and $\mathrm{HACT}$-Net workflow for subtyping breast cancer
$\mathrm{TRoI}$s.

Figure 9: Decision tree used by pathologists to make their diagnosis. The 7-class classification is simplified to a series of binary decision tasks. While going through the tree, the diagnosis become more and more specific until the leaves, i.e., the 7-classes, are reached. TABLE VIII: Comparison with pathologist. Weighted F1 score on 7-class breast cancer subtyping. Results expressed in $\%$. | Normal | Benign | UDH | ADH | FEA | DCIS | Invasive | Weighted F1 | Accuracy
---|---|---|---|---|---|---|---|---|---
Pathologist 1 | 67.53 | 53.92 | 41.90 | 36.00 | 19.13 | 71.59 | 94.00 | 55.30 | 56.71
Pathologist 2 | 47.83 | 52.94 | 25.00 | 35.37 | 65.22 | 68.00 | 94.00 | 57.07 | 57.99
Pathologist 3 | 39.66 | 49.59 | 49.43 | 42.29 | 54.12 | 65.19 | 89.47 | 56.71 | 56.55
Pathologist statistics |  51.57$\pm$11.70 | 52.15$\pm$1.85 |  38.78$\pm$10.22 |  37.89$\pm$3.12 |  46.16$\pm$19.64 | 68.26$\pm$2.62 | 92.49$\pm$2.14 |  56.36$\pm$0.76 |  57.08$\pm$0.64
$\mathrm{HACT}$-Net statistics | 61.56$\pm$2.15 |  47.49$\pm$2.94 | 43.60$\pm$1.86 | 40.42$\pm$2.55 | 74.22$\pm$1.41 |  66.44$\pm$2.57 |  88.40$\pm$0.19 | 61.53$\pm$0.87 | 63.21$\pm$0.27

####  VI-D1 Domain expert comparison on BRACS dataset

To assess the quality of the BRACS dataset and benchmark the proposed
$\mathrm{HACT}$-Net, we acquire independent pathologist’ annotations on the
BRACS test set. We include the participation of three board-certified
pathologists (excluding the pathologists providing the initial annotations,
namely ground truth labels) in the domain expert comparison experiment by
following the evaluation protocols by [4]. The participating pathologists are
from three medical centers, 1) National Cancer Institute- IRCCS-Fondazione
Pascale, Naples, Italy, 2) Lausanne University Hospital, CHUV, Lausanne,
Switzerland, and 3) Aurigen- Centre de Pathologie, Lausanne, Switzerland. They
are specialized in breast pathology and have been in practice for over twenty
years. The pathologists independently and remotely annotated the
$\mathrm{TRoI}$s in the BRACS test set without having access to the respective
WSIs. This protocol ensures equal field-of-view for all the pathologists and
our proposed methodology.

The pathologists’ annotations and $\mathrm{HACT}$-Net predictions are compared
to the ground truth labels of the BRACS test set to evaluate their
classification performances, presented in Table VIII. We include per-class
F1-scores, overall weighted F1-score, and accuracy for the individual
pathologist. Further, we include the aggregated statistics of the
pathologists’ performances for benchmarking $\mathrm{HACT}$-Net with respect
to the domain expert annotators. Table VIII indicates that $\mathrm{HACT}$-Net
provides per-class comparable classification performance to the pathologists.
The $\mathrm{HACT}$-Net actually outperforms the domain experts in the cases
that are typically more difficult to distinguish, namely atypia and
hyperplasias, which was also one of the main aims of the developed
technologies. Also, the per-class standard deviation conveys that
$\mathrm{HACT}$-Net reduces uncertainty and provides more reproducible and
objective results compared to the independent domain expert annotators. The
independent pathologists’ have 57% concordance with the ground truth diagnoses
for the 7-class classification task and $\mathrm{HACT}$-Net results in a
better concordance rate.

TABLE IX: Concordance among independent pathologists’ annotations. Results expressed in $\%$. | Pathologist 1 | Pathologist 2 | Pathologist 3 | Ground truth
---|---|---|---|---
Pathologist 1 | - | 47.60 | 50.96 | 56.71
Pathologist 2 | - | - | 64.38 | 57.99
Pathologist 3 | - | - | - | 56.55

To benchmark the BRACS dataset with respect to the dataset by [4], we compare
the aggregated pathologists’ statistics on both datasets for the same set of
classes, i.e., Benign without atypia (Normal + Benign + UDH), Atypia (ADH +
FEA), DCIS and Invasive. Note that, the [4]’s dataset consists of 240 breast
biopsy slides, and the BRACS dataset consists of 626 $\mathrm{TRoI}$ images.
For the dataset by [4], the pathologists’ class-wise concordance rates are,
87%, 48%, 84% and 96% for the aforementioned classes. For the BRACS dataset,
the pathologists’ class-wise concordance rates are respectively, 87%, 50%,
72%, and 90% for the aforementioned classes. The class-wise concordance rates
exhibit a similar trend for both datasets. The difference in the pathologists’
performance in both datasets can be accounted for by the difference in the
field-of-view, i.e., $\mathrm{TRoI}$ and WSI, accessible to the pathologist
during the diagnoses.

Table IX presents the inter-observer concordance rates for the BRACS test set.
We notice significant differences in concordance rates between pathologist (1,
2) and pathologist (2, 3), and comparable concordance between pathologist (1,
2) and pathologist (1, 3). The observation can be reasoned to the diagnostic
differences and practices across different regions. Pathologist 1 belongs to
Naples, Italy, and 2 and 3 belong to Lausanne, Switzerland.

####  VI-D2 Classification results on BACH dataset

We evaluate the proposed methodology on the public BACH dataset. Considering
the small training set, i.e., 400 images, we employ different image
augmentation techniques for training $\mathrm{HACT}$-Net. To this end, we
employ rotation, mirroring, and color augmentations on the training images
before extracting the $\mathrm{HACT}$ graph representations. We do not use
other graph augmentation techniques, such as random node dropping, random edge
dropping, etc., as these augmentations can hamper the meaningful topological
distribution of the biological entities. The implementation strategies and
hyperparameters from Section VI-B are employed for training
$\mathrm{HACT}$-Net. Classification performance of $\mathrm{HACT}$-Neat and
the state-of-the-art results on the BACH dataset are summarized in Table X.
Our model predictions are independently evaluated by the organizers of the
BACH challenge. This ensured fair and independent comparison with state-of-
the-art methods. The proposed methodology outperforms the state-of-the-art
performance on the BACH dataset. Notably, our methodology employs a single
network compared to the ensemble strategy of employing multiple networks
during the inference.

TABLE X: BACH dataset results expressed in $\%$. | Methods | Accuracy
---|---|---
Ensemble |  Chennamsetty et al. [77] | 87.00
Kwok [78] | 87.00
Brancati et al. [79] | 86.00
Single |  $\mathrm{HACT}$-Net | 92.00

##  VII Conclusion

Pixel-based processing of pathology images suffers from the context-resolution
trade-off, and misses the notion of biological entity and tissue composition.
In this work, we proposed an entity-based tissue representation and learning
to address the issues with pixel-based processing. To achieve this, we
proposed two novel contributions: (i) a hierarchical entity-graph
representation of a tissue image by incorporating multisets of pathologically
intuitive biological entities, and (ii) a hierarchical graph neural network
for sequentially processing the entity-graph representation and mapping the
tissue composition to respective tissue functionality. Further, we introduce
BReAst Cancer Subtyping (BRACS), a large cohort of breast tumor regions-of-
interest annotated with breast cancer subtypes. BRACS encompasses seven breast
cancer subtypes to represent a realistic breast cancer diagnosis scenario.
Using BRACS and BACH, a public breast cancer subtyping dataset, we demonstrate
the performance of the proposed methodology for classifying breast tumor
regions-of-interest into cancer subtypes. Upon assessment through various
classification settings, our methodology outperformed several state-of-the-art
pixel-based and entity-graph based classification methodologies in terms of
classification performance. Further, we benchmarked the performance of our
methodology on the BRACS dataset by comparing it to three independent
pathologists. Notably, the proposed methodology achieved better per-cancer
subtype and overall aggregated classification performance. Although we have
evaluated our method for breast cancer classification, the technology is
easily extendable to other tissue types and diseases. Notably, the proposed
hierarchical graph methodology can be also adapted to other imaging types,
such as natural images, multiplexed images, hyperspectral images, satellite
images, and other medical imaging domains, by incorporating adequate domain
and task-specific entities.

## References

  * [1] F. Bray _et al._ , “Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” in _Cancer J. Clinicians_ , vol. 68, 2018, pp. 394–424.
  * [2] C. Allemani _et al._ , “Global surveillance of cancer survival 1995-2009: Analysis of individual data for 25 676 887 patients from 279 population-based registries in 67 countries (concord-2),” in _The Lancet_ , vol. 385, no. 9972, 2015, pp. 977–1010.
  * [3] D. Gomes _et al._ , “Inter-observer variability between general pathologists and a specialist in breast pathology in the diagnosis of lobular neoplasia, columnar cell lesions, atypical ductal hyperplasia and ductal carcinoma in situ of the breast,” in _Diagnostic Pathology_ , vol. 9, no. 121, 2014.
  * [4] J. Elmore _et al._ , “Diagnostic concordance among pathologists interpreting breast biopsy specimens,” in _JAMA_ , vol. 313, no. 11, 2015, pp. 1122–1132.
  * [5] R. Siegel, K. Miller, and A. Jemal, “Cancer statistics, 2020,” in _CA: A Cancer Journal for Clinicians_ , vol. 70, 2020, pp. 7–30.
  * [6] S. Mukhopadhyay _et al._ , “Whole slide imaging versus microscopy for primary diagnosis in surgical pathology: a multicenter blinded randomized noninferiority study of 1992 cases (pivotal study),” in _Am J Surg Pathol_ , vol. 42, no. 1, 2017, pp. 39–52.
  * [7] G. Litjens _et al._ , “A survey on deep learning in medical image analysis,” in _Medical Image Analysis_ , vol. 42, 2017, pp. 60–88.
  * [8] S. Deng _et al._ , “Deep learning in digital pathology image analysis: A survey,” in _Frontiers of Medicine_ , 2020.
  * [9] A. Ibrahim _et al._ , “Artificial intelligence in digital breast pathology: Techniques and applications,” in _The Breast_ , vol. 49, 2020, pp. 267–273.
  * [10] N. Kumar _et al._ , “A dataset and a technique for generalized nuclear segmentation for computational pathology,” in _IEEE Transactions on Medical Imaging_ , vol. 36, no. 7, 2017, pp. 1550–1560.
  * [11] S. Graham _et al._ , “Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images,” in _Medical Image Analysis_ , vol. 58, 2019.
  * [12] P. Pati _et al._ , “Reducing annotation effort in digital pathology: A co-representation learning framework for classification tasks,” in _Medical Image Analysis_ , vol. 67, 2021.
  * [13] R. Verma _et al._ , “Multi-organ nuclei segmentation and classification challenge,” 2020.
  * [14] S. Graham _et al._ , “Mild-net: Minimal information loss dilated network for gland instance segmentation in colon histology images,” in _Medical Image Analysis_ , vol. 52, 2019, pp. 199–211.
  * [15] T. Binder _et al._ , “Multi-organ gland segmentation using deep learning,” in _Frontiers in Medicine_ , 2019.
  * [16] G. Aresta _et al._ , “Bach: Grand challenge on breast cancer histology images,” in _Medical Image Analysis_ , vol. 56, 2019, pp. 122–139.
  * [17] B. Bejnordi _et al._ , “Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer,” in _JAMA_ , vol. 318, no. 22, 2019, pp. 2199–2210.
  * [18] P. Pati _et al._ , “Deep positive-unlabeled learning for region of interest localization in breast tissue images,” in _Proc. SPIE 10581, Medical Imaging 2018: Digital Pathology_ , vol. 1058107, 2018.
  * [19] C. Mercan _et al._ , “From patch-level to roi-level deep feature representations for breast histopathology classification,” in _Proc. SPIE 10956, Medical Imaging 2019: Digital Pathology_ , vol. 109560H, 2019.
  * [20] A. Madabhushi and G. Lee, “Image analysis and machine learning in digital pathology: Challenges and opportunities,” in _Medical Image Analysis_ , vol. 33, 2016, pp. 170–175.
  * [21] A. Parwani, “Next generation diagnostic pathology: use of digital pathology and artificial intelligence tools to augment a pathological diagnosis,” in _Diagnostic Pathology_ , vol. 14, no. 138, 2019.
  * [22] B. Bejnordi _et al._ , “Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images,” in _Journal of Medical Imaging_ , vol. 4, no. 4, 2017\.
  * [23] K. Sirinukunwattana _et al._ , “Improving whole slide segmentation through visual context - a systematic study,” in _Medical Image Computing and Computer Assisted Intervention_ , vol. 11071, 2018.
  * [24] D. Tellez _et al._ , “Neural image compression for gigapixel histopathology image analysis,” in _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 58, 2019.
  * [25] M. Hagele _et al._ , “Resolving challenges in deep learning-based analyses of histopathological images using explanation methods,” in _Scientific Reports 10_ , vol. 6423, 2020.
  * [26] C. Demir _et al._ , “The cell graphs of cancer,” in _Bioinformatics_ , vol. 20, 2004, pp. 145–151.
  * [27] P. Pati _et al._ , “Hact-net: A hierarchical cell-to-tissue graph neural network for histopathological image classification,” in _MICCAI Graphs in Biomedical Image Analysis Workshop_ , 2020.
  * [28] D. Komura and S. Ishikawa, “Machine learning methods for histopathological image analysis,” in _Computational and Structural Biotechnology Journal_ , 2018, pp. 34–42.
  * [29] C. Srinidhi _et al._ , “Deep neural network models for computational histopathology: A survey,” in _arXiv:1912.12378_ , 2019.
  * [30] F. Spanhol _et al._ , “A dataset for breast cancer histopathological image classification,” in _IEEE transactions on biomedical engineering_ , vol. 63, no. 7, 2016, pp. 1455–1462.
  * [31] T. Araujo _et al._ , “Classification of breast cancer histology images using convolutional neural networks,” in _PloS one_ , vol. 12, no. 6, 2005\.
  * [32] D. Bardou _et al._ , “Classification of breast cancer based on histology images using convolutional neural networks,” in _IEEE Access_ , vol. 6, 2018, pp. 24 680–24 693.
  * [33] K. Roy _et al._ , “Patch-based system for classification of breast histology images using deep learning,” in _Computerized Medical Imaging and Graphics_ , vol. 71, 2019, pp. 90–103.
  * [34] M. Shaban _et al._ , “Context-aware convolutional neural network for grading of colorectal cancer histology images,” in _IEEE Transactions on Medical Imaging_ , vol. 39, no. 7, 2020, pp. 2395 – 2405.
  * [35] R. Yan _et al._ , “Breast cancer histopathological image classification using a hybrid deep neural network,” in _Methods_ , vol. 173, 2020, pp. 52–60.
  * [36] H. Pinckaers, B. van Ginneken, and G. Litjens, “Streaming convolutional neural networks for end-to-end learning with multi-megapixel images,” in _IEEE Transactions on Medical Imaging_ , vol. 39, no. 5, 2020, pp. 1306–1315.
  * [37] G. Campanella _et al._ , “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images,” in _Nature Medicine_ , vol. 25, no. 8, 2019, pp. 1301–1309.
  * [38] M. Lu _et al._ , “Data efficient and weakly supervised computational pathology on whole slide images,” in _arXiv:2004.09666v2_ , 2019.
  * [39] H. Sharma _et al._ , “A review of graph-based methods for image analysis in digital histopathology,” in _Diagnostic Pathology_ , 2015.
  * [40] Y. Zhou _et al._ , “CGC-net: Cell graph convolutional network for grading of colorectal cancer histology images,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2019.
  * [41] R. Chen _et al._ , “Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,” in _IEEE Transactions on Medical Imaging_ , 2020.
  * [42] D. Anand _et al._ , “Histographs: graphs in histopathology,” in _Proc. SPIE 11320, Medical Imaging 2020: Digital Pathology_ , vol. 113200O, 2020.
  * [43] B. Aygunes _et al._ , “Graph convolutional networks for region of interest classification in breast histopathology,” in _Proc. SPIE 11320, Medical Imaging 2020: Digital Pathology_ , vol. 113200K, 2020.
  * [44] S. Javed _et al._ , “Cellular community detection for tissue phenotyping in colorectal cancer histology images,” in _Medical Image Analysis_ , vol. 63, 2020.
  * [45] Y. Zhao _et al._ , “Predicting lymph node metastasis using histopathological images based on multiple instance learning with deep graph convolution,” in _IEEE CVPR_ , 2020, pp. 4837–4846.
  * [46] M. Adnan _et al._ , “Representation learning of histopathology images using graph neural networks,” in _IEEE CVPR Workshops_ , 2020, pp. 4254–4261.
  * [47] G. Jaume _et al._ , “Towards explainable graph representations in digital pathology,” in _ICML Workshop on Computational Biology_ , 2020.
  * [48] M. Sureka _et al._ , “Visualization for histopathology images using graph convolutional neural networks,” in _arXiv:2006.09464_ , 2020.
  * [49] G. Jaume _et al._ , “Quantifying explainers of graph neural networks in computational pathology,” in _arXiv:2011.12646_ , 2020.
  * [50] M. Defferrard _et al._ , “Convolutional neural networks on graphs with fast localized spectral filtering,” in _NeurIPS_ , 2016, pp. 3844–3852.
  * [51] T. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in _ICLR_ , 2017.
  * [52] K. Xu _et al._ , “How powerful are graph neural networks?” in _ICLR_ , 2019.
  * [53] W. Hamilton _et al._ , “Inductive representation learning on large graphs,” in _NeurIPS_ , 2017, pp. 1024–1034.
  * [54] P. Velickovic _et al._ , “Graph attention networks,” in _International inproceedings on Learning Representations, ICLR_ , 2018.
  * [55] J. Gilmer _et al._ , “Neural message passing for quantum chemistry,” in _ICML_ , vol. 70, 2017, pp. 1263–1272.
  * [56] C. Morris _et al._ , “Weisfeiler and leman go neural: Higher-order graph neural networks,” in _AAAI_ , 2018.
  * [57] G. Jaume _et al._ , “edgnn: a simple and powerful gnn for directed labeled graphs,” in _ICLR Workshop on Representation Learning on Graphs and Manifolds_ , 2019.
  * [58] B. Weisfeiler and A. A. Lehman, “A reduction of a graph to a canonical form and an algebra arising during this reduction,” in _Nauchno-Technicheskaya Informatsia_ , vol. 2, no. 9, 1968, pp. 12–16.
  * [59] N. Dehmamy _et al._ , “Understanding the representation power of graph neural networks in learning graph topology,” in _NeurIPS_ , 2019, pp. 15 413–15 423.
  * [60] G. Corso _et al._ , “Principal neighbourhood aggregation for graph nets,” in _NeurIPS_ , 2020.
  * [61] M. Veta _et al._ , “Breast cancer histopathology image analysis: A review,” in _IEEE Transactions on Biomedical Engineering_ , no. 5, 2014, pp. 1400–1411.
  * [62] D. Tellez _et al._ , “Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology,” in _Medical Image Analysis_ , vol. 58, 2019.
  * [63] M. Macenko _et al._ , “A method for normalizing histology slides for quantitative analysis,” in _IEEE International Symposium on Biomedical Imaging: From Nano to Macro_ , 2009, pp. 1107–1110.
  * [64] M. Stanisavljevic _et al._ , “A fast and scalable pipeline for stain normalization of whole-slide images in histopathology,” in _ECCV Workshop_ , 2018.
  * [65] K. He _et al._ , “Deep residual learning for image recognition,” in _IEEE CVPR_ , 2016.
  * [66] J. Deng _et al._ , “Imagenet: A large-scale hierarchical image database,” in _IEEE CVPR_ , 2009.
  * [67] K. Francis and B. Palsson, “Effective intercellular communication distances are determined by the relative time constants for cyto/chemokine secretion and diffusion,” in _Proceedings of the National Academy of Sciences_ , vol. 94, no. 23, 1997, pp. 12 258–12 262.
  * [68] R. Achanta _et al._ , “Slic superpixels compared to state-of-the-art superpixel methods,” in _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 34, no. 11, 2012, pp. 2274–2282.
  * [69] F. Potjer, “Region adjacency graphs and connected morphological operators,” in _Mathematical Morphology and its Applications to Image and Signal Processing. Computational Imaging and Vision_ , vol. 5, 1996, p. 111–118.
  * [70] K. Xu _et al._ , “Representation learning on graphs with jumping knowledge networks,” in _ICML_ , 2018.
  * [71] V. P. Dwivedi _et al._ , “Benchmarking graph neural networks,” in _CoRR,abs/2003.00982_ , 2020.
  * [72] P. Bankhead _et al._ , “Qupath: Open source software for digital pathology image analysis,” in _Scientific reports_ , vol. 7, no. 1, 2017, pp. 1–7.
  * [73] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015.
  * [74] A. Paszke _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” in _NeurIPS_ , 2019, pp. 8024–8035.
  * [75] M. Wang _et al._ , “Deep graph library: Towards efficient and scalable deep learning on graphs,” in _CoRR_ , vol. abs/1909.01315, 2019.
  * [76] E. Mercan _et al._ , “Automated diagnosis of breast cancer and pre-invasive lesions on digital whole slide images,” in _ICPRAM_ , 2018.
  * [77] S. Chennamsetty, M. Safwan, and V. Alex, “Classification of breast cancer histology image using ensemble of pre-trained neural networks,” in _International Conference Image Analysis and Recognition_ , 2018, pp. 804–811.
  * [78] S. Kwok, “Multiclass classification of breast cancer in whole-slide images,” 2018\.
  * [79] N. Brancati, M. Frucci, and D. Riccio, “Multi-classification of breast cancer histology images by using a fine-tuning strategy,” in _International Conference Image Analysis and Recognition_ , 2018, pp. 771–778.