# Unleashing the Infinity Power of Geometry: A Novel Geometry-Aware
Transformer (GOAT) for Whole Slide Histopathology Image Analysis

###### Abstract

The histopathology analysis is of great significance for the diagnosis and
prognosis of cancers, however, it has great challenges due to the enormous
heterogeneity of gigapixel whole slide images (WSIs) and the intricate
representation of pathological features. However, recent methods have not
adequately exploited geometrical representation in WSIs which is significant
in disease diagnosis. Therefore, we proposed a novel weakly-supervised
framework, Geometry-Aware Transformer (GOAT), in which we urge the model to
pay attention to the geometric characteristics within the tumor
microenvironment which often serve as potent indicators. In addition, a
context-aware attention mechanism is designed to extract and enhance the
morphological features within WSIs. Extensive experimental results
demonstrated that the proposed method is capable of consistently reaching
superior classification outcomes for gigapixel whole slide images.

Index Terms—  Computational Pathology, Graph Convolution Network, Weakly-
Supervised Learning.

##  1 Introduction

Pathological slide examination is widely considered the "gold standard" for
cancer diagnosis and prognosis, making histopathology whole slide image
analysis a crucial deep learning application in clinical practice [1]. In
recent years, the rapid advancement of computational pathology presents an
excellent prospect for researchers to design and develop deep learning
algorithms for diverse spheres of application, encompassing but not limited to
the Gleason grading, survival prediction, and cancer subtyping [2, 3, 4, 5, 6,
7, 8, 9, 10, 11].

The advent of deep learning techniques in recent years has facilitated AI-
assisted diagnosis in computational pathology, demonstrating comparable
performance to experienced pathologists. Consequently, numerous whole slide
image classification techniques based on deep learning have emerged [4, 5, 6,
7, 8, 9, 10]. For example, Ilse et al. [12] proposed an attention based
aggregation operator, giving each instance additional contribution information
through trainable attention weights. Li et al. [6] designed a novel framework
that jointly trained a patch and an image classifier, where the patches are
selected softly with instance-level attention. In addition, Shao et al. [10]
presented a transformer-enabled multi-instance learning (MIL) framework to
explore both morphological and spatial information among instances. However,
these approaches are mostly based on traditional MIL framework and utilize
convolutional neural networks (CNNs) as the feature extractor to solely
provide limited histology features, which leads to such techniques are not
sufficiently context-aware and unable to capture the key morphological feature
interactions between cell and tissue that are predictive of cancer diagnosis
[13].

In this paper, we propose a novel geometry-aware, weakly-supervised,
transformer-enabled framework, termed Geometry-Aware Transformer (GOAT) as
shown in Fig. 1, to learn contextual histopathological features and complex
spatial relationships in the tumor microenvironment for improving the
performance of gigapixel WSI classification. The three main contributions of
this paper are as follows: (1) We formulated the gigapixel WSI as a fully
connected dense graph structure where each node represents a non-overlapping
patch in the image and each edge represents the spatial relationship between
two patches. (2) We proposed a context-aware attention module (MHGA) to
leverage the graph structure to learn the spatial relationships between cell
identities and identify the different tissue types. (3) We performed extensive
experiments on two benchmarks from TCGA to demonstrate the advantages of GOAT
over cutting-edge approaches.

##  2 Methodology

Fig. 1: Overview of the proposed Geometry-Aware Transformer framework.

###  2.1 Graph Construction of Whole Slide Image

To construct the graph $\bm{\mathcal{G}}$ for the input WSI X, we first
utilize Otsu’s Binarization on the saturation channel to separate H&E-stained
tissue from the background and employ the CLAM [4] for automated tissue
segmentation. Following segmentation, we extract 256$\times$256 image patches
$\left\\{x_{k}\right\\}_{k=1}^{N}$ without spatial overlapping at the
20$\times$ equivalent pyramid level from all tissue regions identified. To
create feature embeddings for the extracted patches, we utilize an ImageNet-
pretrained ResNet-50 [14] as a CNN encoder to convert each patch into a
1024-dimensional feature embedding $\mathbf{h}_{k}\in\mathbb{R}^{1024}$,
truncated after the third residual block and adaptive average pooling layer
and is then packed into a node feature matrix $\bm{M}\in\mathbb{R}^{d\times
N}$ for the patches in X.

We construct a $k$-NN graph $\bm{\mathcal{G}}=(\bm{M},\bm{A})$ based on the
Euclidean distance between the node features, where $\bm{A}$ denotes the
adjacency matrix for each X (defined by the coordinates from the tissue
segmentation). Finally, we obtain a geometric representation according to the
morphological feature of input WSI, which often serves as a powerful cue for
cancer diagnosis in clinical pathology practice.

###  2.2 Geometry-Aware Transformer

Graph Embedding We follow [15] to utilize the graph embedding layer to
transform the graph structure as the input of GOAT. As far as we know, besides
the vertices, edges also have rich structural information in many types of
graphs. Therefore, for exploiting the relationship among nodes and edges in
graph data, we transform the graph into edge feature embedding $\bm{E}$ and
node feature embedding $\bm{V}$ respectively.

Multi-Head Geometry Attention In recent years, a range of works show global
self-attention could serve as a flexible alternative to graph convolution and
improve graph representation learning [16, 17]. While most of them only
consider part of the graph data and ignore the edge information which is also
crucial for geometric structure learning. To this end, we introduce Multi-Head
Geometry Attention (MHGA) (shown in Fig. 2) to fully model the relationship
among vertices and edges and represent the graph structure. We first use
linear layer to project the node embedding $\bm{V}$ to query $\mathrm{Q}$, key
$\mathrm{K}$, and value $\mathrm{V}$, then we follow [15] to predict layer-
specific attention biases $\bm{E}_{\theta}$ from the edge embedding $\bm{E}$.
After that, we add it to the result of the query-key dot product to generate
the attention map $\mathbb{A}$ and update the output node embedding. This
process can be written as follows:

$\mathbb{A}=\frac{\mathrm{Q}\cdot\mathrm{K}^{\top}}{\sqrt{d_{head}}}+\bm{E}_{\theta}=\frac{\mathrm{Q}\cdot\mathrm{K}^{\top}}{\sqrt{d_{head}}}+\theta\left(\bm{E}\right)\textbf{W}_{\theta}$
(1)

where $d_{head}$ refers to the dimension of each head. Thus, the proposed MHGA
can be calculated as:

$\displaystyle\mathbf{MHGA}\left(\bm{E},\bm{V}\right)=\bm{E}^{\prime}\otimes\bm{V}^{\prime}$
(2)
$\displaystyle=\Big{(}\mathbb{A}+\Phi\left(\mathbb{A}\right)\Big{)}\otimes\Big{(}\Phi\left(\mathbb{A}\right)\odot\mathrm{V}\otimes\Phi\left(\bm{E}_{\theta}\right)\Big{)}\longrightarrow\textbf{h}$

where $\otimes$ and $\odot$ refers kronecker product and element-wise
multiplication respectively, $\Phi$ denotes the softmax function, h denotes
the result of MHGA.

Graph Convolution Network We further design our graph learning module based on
Topology Adaptive Graph Convolutional Network (TAGCN) [18] to alleviate the
computational complexity without information loss. The TAGCN layer in MHGA can
be formulated as follows:

$\mathbf{TAGCN}\left(\textbf{h}\right)=\sum_{f=1}^{3}\Big{(}\sum_{n=1}^{N}\textbf{GCN}_{n}^{f}\left(\textbf{h}\right)+\textbf{h}_{f-1}\Big{)}\longrightarrow\textbf{h}_{\mathrm{GCN}}$
(3)

where $\textbf{GCN}\left(\cdot\right)$ is the matrix-vector product,
$n=1,2,\cdots,N$ and $N$ denotes the number of vertices, $f$ refers the number
of GCN layers ($f=3$ here), $\textbf{h}_{f-1}$ denotes the result of previous
TAGCN layer. In order to alleviate the problems of over-smoothing and
vanishing gradient as the network goes deeper, we add the residual mapping to
improve the training.

Global Attention Pooling We leverage a global attention-based pooling layer
$\mathcal{F}_{\mathrm{GAP}}\left(\cdot\right)$ [12] to adaptively computes a
weighted summation of all node features in the graph and pool the node feature
embedding $\textbf{h}_{\mathrm{GCN}}$ to a WSI-level embedding:

$\displaystyle\textbf{h}_{\mathrm{GAP}}=\mathcal{F}_{\mathrm{GAP}}\left(\sum\limits_{i=1}^{N}\bm{\alpha}_{i}\right)\cdot\textbf{h}_{\mathrm{GCN}}\quad
where$ (4)
$\displaystyle\bm{\alpha}_{i}=\frac{\mathrm{exp}\left\\{\textbf{W}\Big{(}\mathrm{tanh}\left(\textbf{V}\cdot\textbf{h}_{i}^{\top}\right)\odot\mathrm{sigm}\left(\textbf{U}\cdot\textbf{h}_{i}^{\top}\right)\Big{)}\right\\}}{\sum\limits_{j=1}^{N}\mathrm{exp}\left\\{\textbf{W}\Big{(}\mathrm{tanh}\left(\textbf{V}\cdot\textbf{h}_{j}^{\top}\right)\odot\mathrm{sigm}\left(\textbf{U}\cdot\textbf{h}_{j}^{\top}\right)\Big{)}\right\\}}$

where W, V, and U are trainable weight matrices, $\bm{\alpha}_{i}$ is the
learnable scalar weight for global attention pooling layer.

Finally, we utilize a two-layer multi-layer perceptron as the feed-forward
network (FFN) in transformer with a fully connected linear layer to downsample
and aggregate the final cancer subtyping result, which can be calculated as
follows:

$\textbf{h}_{\mathrm{Final}}=\mathrm{Linear}\Big{(}\mathrm{FFN}\left(\textbf{h}_{\mathrm{GAP}}\right)\Big{)}$
(5)

where the
$\textbf{h}_{\mathrm{Final}}\in\mathbb{R}^{1024\times\\#\mathrm{CLS}}$ refers
the final embedding, $\\#\mathrm{CLS}$ denotes the number of classes in
current dataset.

##  3 Experimental Setup

###  3.1 Datasets and Evaluation Metrics

TCGA-NSCLC The TCGA-NSCLC (Non-Small Cell Lung Cancer) dataset includes two
sub-types of lung cancer, Lung Adenocarcinoma (LUAD) and Lung Squamous Cell
Carcinoma (LUSC). The dataset contains a total of 993 diagnostic WSIs,
including 507 LUAD slides and 486 LUSC slides.

TCGA-RCC The TCGA-RCC (Renal Cell Carcinoma) dataset includes three sub-types
of kidney cancer, Kidney Chromophobe Renal Cell Carcinoma (KICH), Kidney Renal
Clear Cell Carcinoma (KIRC), and Kidney Renal Papillary Cell Carcinoma (KIRP).
The dataset contains a total of 884 diagnostic WSIs, including 111 CRCC
slides, 489 CCRCC slides, and 284 PRCC slides.

Evaluation Metrics To evaluate GOAT, we conducted experiments using 10-fold
Monte Carlo cross-validation which each dataset was divided into 60/15/25
partitions for training, validation, and testing. The overall accuracy and the
area under the curve (AUC) across the testing splits were employed to measure
the diagnosis ability of the models.

Fig. 2: The proposed Multi-Head Geometry Attention.

###  3.2 Implementation Details

GOAT is implemented using PyTorch 1.13.1 and PyTorch-Geometric (PyG) 2.0.3,
the model is trained on a workstation equipped with an NVIDIA Quadro GV100
GPU. We utilize the cross-entropy loss function, and the model parameters are
optimized via stochastic gradient descent using the Adam optimizer with a
learning rate of 2e-4 and weight decay of 1e-5. Our related code will be
publicly made available at https://github.com/lmxmercy/GOAT.

##  4 Results and Analysis

###  4.1 Comparison with State-of-the-Art Methods

The two used datasets, TCGA-NSCLC and TCGA-RCC, share a common characteristic
that the large areas of tumor region in the positive slide (average area >80%
per slide), making most of the methods have similar performance, and even the
most naive baselines max-pooling and mean-pooling also achieved a relatively
high performance. However, GOAT outperformed all comparative baselines in two
metrics on these two benchmarks by a significant margin. In comparison with
the second best-performing baseline, TransMIL [10], GOAT achieved a
performance increase of 4.32% and 2.19% on accuracy and AUC respectively.
Furthermore, the highest performance of GOAT on the TCGA-RCC dataset
demonstrated that our model also excels in the multi-class classification
problem. The superior improvements of the proposed method showcases its
ability to discriminate the cancer regions in the tumor microenvironment,
which is often crucial for clinical cancer diagnosis.

Table 1: Quantitative evaluation on TCGA-NSCLC and TCGA-RCC two datasets. The
highest results are in bold.

Methods | TCGA-NSCLC | TCGA-RCC
---|---|---
Accuracy $\uparrow$ | AUC $\uparrow$ | Accuracy $\uparrow$ | AUC $\uparrow$
Max-pooling | 0.7282 | 0.8401 | 0.9054 | 0.9786
Mean-pooling | 0.8593 | 0.9463 | 0.9378 | 0.9879
ABMIL [12] | 0.7719 | 0.8656 | 0.8934 | 0.9702
PT-MTA [5] | 0.7379 | 0.8299 | 0.9059 | 0.9700
StreamingCNN [9] | 0.8692 | 0.9260 | 0.8817 | 0.9660
DSMIL [6] | 0.8058 | 0.8925 | 0.9294 | 0.9841
CLAM-SB [4] | 0.8180 | 0.8818 | 0.8816 | 0.9723
CLAM-MB [4] | 0.8422 | 0.9377 | 0.8966 | 0.9799
TransMIL [10] | 0.8835 | 0.9603 | 0.9466 | 0.9882
Zhang et al. [8] | 0.8785 | 0.9377 | 0.9140 | 0.9740
GOAT (Ours) | 0.9217 | 0.9813 | 0.9705 | 0.9956

###  4.2 Ablation Study

To validate the effectiveness of our designed modules like MHGA, TAGCN,
residual mapping, and GAP, we add these components one by one to the baseline.
Experimental results are shown in Table 2. As can be seen, the proposed MHGA
has significant improvement (14.29% on accuracy on TCGA-NSCLC) for cancer
subtyping which demonstrated the necessity of the introduced graph structure.
In addition, the introduction of TAGCN is also beneficial for WSI
classification, performance dramatically degrades for two evaluation metrics
when it was removed. Then we add the residual mappings to Model B, the
experiment result shows that it plays an essential role in GOAT. We further
add a global attention pooling layer, and 2.56% and 2.07% improvement is
observed for cancer subtyping on accuracy and AUC, respectively.

Table 2: Ablation study on TCGA-NSCLC. MHGA: Multi-Head Geometry Attention.
TAGCN: topology adaptive graph convolution network. Residual: residual
connections in GCN layers. GAP: global attention pooling layer.

Model | Designs in GOAT | TCGA-NSCLC
---|---|---
MHGA | TAGCN | Residual | GAP | Accuracy $\uparrow$ | AUC $\uparrow$
A |  |  |  |  | 0.7532 | 0.8703
B | ✓ |  |  |  | 0.8616 | 0.9317
C | ✓ | ✓ |  |  | 0.8801 | 0.9564
D | ✓ | ✓ | ✓ |  | 0.8987 | 0.9614
E | ✓ | ✓ | ✓ | ✓ | 0.9217 | 0.9813

###  4.3 Interpretability and Attention Visualization

We demonstrate the interpretability of GOAT by visualizing the attention
scores as heatmaps, which indicate the locations of the discriminative
regions. We generate an attention heatmap by converting the attention scores
for the predicted class of the model into percentiles and mapping the
normalized scores to their corresponding spatial location in the original
slide. Additionally, we visualize the top-$k$ patches of the most highly
attended regions in the input slide which can generally exhibit the
corresponding tumor morphology pattern. As shown in Fig. 3, the proposed GOAT
can accurately predict the subtyping and localize the related regions in
different cancer types which demonstrates that GOAT has the potential to be
used for meaningful WSI interpretability and visualization in cancer subtyping
problems for clinical practice.

Fig. 3: Interpretability and visualization for NSCLC and RCC subtyping. From
left to right, the first, second and third column denotes original slide, the
attention heatmap, and the top-$k$ highly attended patches with attention
scores respectively.

##  5 Conclusion

In this paper, we present an innovative geometry-aware, weakly-supervised,
transformer-enabled framework called Geometry-Aware Transformer (GOAT) for
cancer diagnosis in computational pathology. Our GOAT leverages multi-head
geometry attention (MHGA) to exploit the geometric representation from
pathological morphological features to improve cancer subtyping performance
significantly. Extensive experiments on TCGA-NSCLC and TCGA-RCC two public
datasets demonstrate the advantages of the proposed GOAT over cutting-edge
methods in improving WSI classification performance. Further ablation study
validates the effectiveness of the proposed modules.

## References

  * [1] Tahsin Kurc, Spyridon Bakas, Xuhua Ren, Aditya Bagari, Alexandre Momeni, Yue Huang, Lichi Zhang, Ashish Kumar, Marc Thibault, Qi Qi, et al.,  “Segmentation and classification in digital pathology for glioma research: challenges and deep learning approaches,”  Frontiers in neuroscience, vol. 14, pp. 27, 2020.
  * [2] Jingwen Wang, Richard J Chen, Ming Y Lu, Alexander Baras, and Faisal Mahmood,  “Weakly supervised prostate tma classification via graph convolutional networks,”  in 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020, pp. 239–243.
  * [3] Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al.,  “Pan-cancer integrative histology-genomic analysis via multimodal deep learning,”  Cancer Cell, vol. 40, no. 8, pp. 865–878, 2022.
  * [4] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood,  “Data-efficient and weakly supervised computational pathology on whole-slide images,”  Nature biomedical engineering, vol. 5, no. 6, pp. 555–570, 2021.
  * [5] Weijian Li, Viet-Duy Nguyen, Haofu Liao, Matt Wilder, Ke Cheng, and Jiebo Luo,  “Patch transformer for multi-tagging whole slide histopathology images,”  in Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part I 22. Springer, 2019, pp. 532–540.
  * [6] Bin Li, Yin Li, and Kevin W Eliceiri,  “Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14318–14328.
  * [7] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas J Fuchs,  “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images,”  Nature medicine, vol. 25, no. 8, pp. 1301–1309, 2019.
  * [8] Jingwei Zhang, Xin Zhang, Ke Ma, Rajarsi Gupta, Joel Saltz, Maria Vakalopoulou, and Dimitris Samaras,  “Gigapixel whole-slide images classification using locally supervised learning,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, pp. 192–201.
  * [9] Hans Pinckaers, Bram Van Ginneken, and Geert Litjens,  “Streaming convolutional neural networks for end-to-end learning with multi-megapixel images,”  IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 3, pp. 1581–1590, 2020.
  * [10] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al.,  “Transmil: Transformer based correlated multiple instance learning for whole slide image classification,”  Advances in neural information processing systems, vol. 34, pp. 2136–2147, 2021.
  * [11] Mingxin Liu, Yunzan Liu, Hui Cui, Chunquan Li, and Jiquan Ma,  “Mgct: Mutual-guided cross-modality transformer for survival outcome prediction using integrative histopathology-genomic features,”  arXiv preprint arXiv:2311.11659, 2023.
  * [12] Maximilian Ilse, Jakub Tomczak, and Max Welling,  “Attention-based deep multiple instance learning,”  in International conference on machine learning. PMLR, 2018, pp. 2127–2136.
  * [13] A Tie and O PE,  “Contextual learning is nearly all you need,” 2022.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [15] Zhe Chen, Hao Tan, Tao Wang, Tianrun Shen, Tong Lu, Qiuying Peng, Cheng Cheng, and Yue Qi,  “Graph propagation transformer for graph representation learning,”  arXiv preprint arXiv:2305.11424, 2023.
  * [16] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu,  “Do transformers really perform badly for graph representation?,”  Advances in Neural Information Processing Systems, vol. 34, pp. 28877–28888, 2021.
  * [17] Vijay Prakash Dwivedi and Xavier Bresson,  “A generalization of transformer networks to graphs,”  arXiv preprint arXiv:2012.09699, 2020.
  * [18] Jian Du, Shanghang Zhang, Guanhang Wu, José MF Moura, and Soummya Kar,  “Topology adaptive graph convolutional networks,”  arXiv preprint arXiv:1710.10370, 2017.