# Morphological Prototyping for Unsupervised Slide Representation Learning
in Computational Pathology

Andrew H. Song1,2, , Richard J. Chen1,2,∗, Tong Ding1,2, Drew F.K.
Williamson1,2, ,
Guillaume Jaume1,2, Faisal Mahmood1,2
1Mass General Brigham and 2Harvard University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contributionPresently at Emory University
School of Medicine

###### Abstract

Representation learning of pathology whole-slide images (WSIs) has been has
primarily relied on weak supervision with Multiple Instance Learning (MIL).
However, the slide representations resulting from this approach are highly
tailored to specific clinical tasks, which limits their expressivity and
generalization, particularly in scenarios with limited data. Instead, we
hypothesize that morphological redundancy in tissue can be leveraged to build
a task-agnostic slide representation in an unsupervised fashion. To this end,
we introduce Panther, a prototype-based approach rooted in the Gaussian
mixture model that summarizes the set of WSI patches into a much smaller set
of morphological prototypes. Specifically, each patch is assumed to have been
generated from a mixture distribution, where each mixture component represents
a morphological exemplar. Utilizing the estimated mixture parameters, we then
construct a compact slide representation that can be readily used for a wide
range of downstream tasks. By performing an extensive evaluation of Panther on
subtyping and survival tasks using 13 datasets, we show that 1) Panther
outperforms or is on par with supervised MIL baselines and 2) the analysis of
morphological prototypes brings new qualitative and quantitative insights into
model interpretability. The code is available at
https://github.com/mahmoodlab/Panther.

##  1 Introduction

Representation learning of whole-slide images (WSIs) is a fundamental task in
Computational Pathology (CPath) [77]. Given a WSI, the goal is to learn a
slide-level representation that can be used for various downstream tasks, such
as diagnosis, prognostication, and therapeutic response prediction. The
standard approach is weakly supervised learning based on Multiple Instance
Learning (MIL) [29, 40, 13], in which the gigapixel WSI is tokenized into a
large set of patch embeddings ($N>10,000$) with a pretrained vision encoder,
followed by aggregation of the embeddings [40]. Current advances in CPath have
examined: (1) learning better patch embeddings with domain-specific vision
encoders based on self-supervised learning (SSL) [21, 83, 31, 1, 46, 44, 39,
18, 58] and (2) developing new aggregation strategies for pooling patch
embeddings into a slide representation [55, 60, 73, 56]. As many histology
datasets have limited samples for supervised MIL, an emerging goal in CPath is
to (3) shift slide representation learning from _weakly-supervised_ to
_unsupervised_ [17, 52, 81, 69, 41], which may help mitigating data and label
scarcity and improving generalization.

Figure 1: Slide decomposition into morphological prototypes Due to
morphological redundancy across and within the tissue, a slide can be
decomposed into prototypes. We introduce Panther, a method that can identify
and extract morphological prototypes to form a compact and unsupervised slide
representation.

We postulate that such models are particularly suited for fine-grained
classification tasks, such as survival outcome prediction that require
holistic modeling of morphologies found in the tissue microenvironment [88,
96]. In contrast to “needle-in-a-haystack” tasks (_e.g.,_ micro-metastasis
detection) that require localizing tumor patches, “panoramic” tasks require
integrating spatial heterogeneity (diversity of distinct tumor populations)
[35, 61], interactions and context (immune infiltrate near invasive tumor
margin) [3, 71], and size (number and size of masses) [6]. Attention-based
architectures [40, 60, 43] demonstrate clinical-grade performance in the
former task (selectively focusing on diagnostic patches of a single visual
concept) [4]; however, they have limited expressivity in the panoramic tasks
that benefit from understanding proportions and mixtures of visual concepts
[20, 7, 15].

Based on these insights, we propose an unsupervised slide representation
framework that can accurately capture the proportions and mixtures of
morphological visual concepts. Specifically, we build on the observation that
WSI patches show morphological redundancy and thus a handful of morphological
patterns are repeated (Fig. 1). Formally, we hypothesize that a concise set of
key descriptors (prototypes), coupled with distribution characterizing the
extent and variation of each descriptor, would comprehensively summarize the
WSI. As this summary only relies on the statistical characteristics of each
WSI, this yields a generic and unbiased slide embedding applicable across
multiple downstream tasks. To faithfully summarize the WSI, the goal becomes
to construct (1) a mapping between each patch and the prototypes and (2) a
slide embedding with the learned mapping that includes representation of each
prototype and its extent, _i.e.,_ its cardinality.

To this end, we introduce Panther, a Prototype AggregatioN-based framework for
compacT HEterogenous slide set Representation (Panther). Inspired by previous
work in prototype-based set representation learning [64, 26, 49, 89], Panther
builds an unsupervised slide embedding by assuming that each patch embedding
is generated from the Gaussian mixture model (GMM), with each morphological
prototype representing a mixture component. By employing GMM, the two
aforementioned goals are seamlessly satisfied through the parameter estimation
procedure with Expectation-Maximization (EM) [27, 49]: (1) the estimated
posterior probability of mixture assignment for each patch defines the mapping
between a patch embedding and a prototype, and (2) mixture probability
represents cardinality, and mixture mean & covariance represent the
representation of corresponding morphological pattern. The slide
representation is formed as a concatenation of the estimated GMM parameters
across all prototypes. Since the representation can be decomposed along each
prototype, this allows for per-prototype nonlinear modeling and interpretation
of the slide centered around each histology visual concept.

To summarize, our contributions are (1) the first prototypical framework for
learning compact and unsupervised slide representations in WSI based on GMM;
(2) a comprehensive evaluation of four diagnostic and nine prognostic tasks,
demonstrating the outperformance against nearly all unsupervised and
supervised baselines; (3) post-hoc interpretability with the quantification
and visualization of morphological prototype spread within the tissue.

##  2 Related work

###  2.1 Multiple instance learning in CPath

While initial histology-based outcome prediction was centered on pathologist-
annotated region-of-interests [65, 11, 47], later works have utilized WSIs for
clinical prediction tasks with MIL [40, 75, 13, 22, 17]. There is a sustained
effort for new MIL schemes, with a focus on new patch aggregation strategies
to learn more representative and task-specific slide embedding, towards better
predictive accuracy [73, 55, 84, 78, 94, 59] or interpretability [43, 79]. MIL
methods based on multiscale representation slides have recently shown promise
for “panoramic” tasks [33, 37, 5]. Panther is similar to MIL in that the
slide-level embedding is constructed from patches for outcome prediction.
However, Panther constructs an unsupervised set embedding and is agnostic to
downstream tasks, in contrast to supervised MIL frameworks.

###  2.2 Quantification of distance between sets

Measuring the distance between two distinct sets (Wasserstein distance),
_e.g.,_ between supports of empirical probability distributions, has received
increasing attention from a diverse range of disciplines such as signal
processing [50], vision-language tasks [68], and computational biology [2, 72,
10]. Given a similarity (or cost) metric between set elements such as
$\mathcal{L}_{2}$ distance, the Wasserstein distance is defined as
transporting (or matching) elements of one set to the other, incurring a
minimum cost. Computing such distance is commonly called the optimal transport
(OT) [25].

It is natural to extend this idea to CPath, where the biological entities of
interest (e.g., WSI and genomic data) are typically modeled as a set of
biological concepts. Different from MIL setting where a set of WSI patches is
matched to a patient-level clinical label, this concerns matching between two
sets: 1) sets of WSI patches for slide retrieval [24] or domain adaptation
[30], 2) different cancer datasets to quantify morphological distance between
different cancer types [90], and 3) a set of WSI patches and a set of genomic
tokens to learn optimal fusion for improved prognosis [85]. Similarly, Panther
can be seen as the matching problem between a set of WSI patches and
prototypes.

###  2.3 Prototype-based set representation

Prototypes are representative examples of data points that share the same
class, usually formulated as centroids from clustering that describe unique
human-interpretable concepts and other semantic information [19, 76, 16].
Recently, it has been applied to compactly represent large set data in
bioinformatics and NLP [54, 64, 49, 26]. The desiderata for prototype-based
set representation is to model: (1) cardinality, i.e., how many elements in
the set are associated with a prototype, and (2) description, i.e., prototype
identity.

Posed also in many related forms such as signatures [53, 95] and bag of visual
words (BoVW) [74, 23, 12], learning prototypical representations is a natural
problem in CPath as repeating histology patterns often reflect the same
morphology [82, 86, 36, 87, 67, 91, 45]. Recent prototypical MIL approaches
for pathology include AttnMISL [88], which aggregates patch embeddings within
the same cluster, followed by aggregating the pooled cluster embeddings.
Following recent advances in visual self-supervised learning, prototypes have
been used for constructing unsupervised slide features via pooling similar
patch embeddings into a concatenated representation (H2T [81]) or measuring
the proportion of prototypes assignments in WSIs (HPL [69]). We note that H2T
and HPL have limitations in not encoding cardinality or not including deep
visual features, which are relevant for interpretability and solving panoramic
tasks. Moreover, as many pretrained vision encoders in CPath are pretrained on
TCGA, prototypical patterns lack extensive evaluation of out-of-domain
performance.

Figure 2: Overview of Panther workflow. Whole-slide image (WSI) is segmented
and patched into a set of WSI patches. A compressed feature for each patch is
encoded through a feature extractor pretrained on a large histopathology
dataset. Panther uses the Gaussian mixture model for patch embedding
distribution, with each mixture corresponding to a morphologically distinct
prototype. The estimated model parameters are concatenated to form the slide
representation, which can be used as input to a predictor module for clinical
downstream tasks and visualized as a prototypical assignment map.

##  3 Methods

We present Panther, an unsupervised slide representation learning framework
based on a compact set of prototypes with GMM (Fig. 2). We first explain the
GMM setup and its connection to a slide embedding (Section 3.1). We then
present how it is used for downstream tasks (Section 3.2) and prototype-based
interpretation (Section 3.3).

###  3.1 Prototype-based aggregation

Given a WSI for subject $j$, we tessellate it into small non-overlapping
patches
$\mathbf{X}^{j}=\\{\mathbf{x}_{1}^{j},\cdots,\mathbf{x}^{j}_{N_{j}}\\}$ with
$\mathbf{x}_{n}^{j}\in\mathbb{R}^{W\times H\times 3}$. We then employ a
feature extractor $f_{\text{enc}}(\cdot)$ pretrained on large archives of
histopathology images [18], to extract a representative and compressed
embedding from each patch. The set of extracted embeddings
$\mathbf{Z}^{j}=\\{\mathbf{z}_{1}^{j},\cdots,\mathbf{z}_{N_{j}}^{j}\\}$ with
$\mathbf{z}_{n}^{j}=f_{\text{enc}}(\mathbf{x}_{n}^{j})\in\mathbb{R}^{d}$ is
then aggregated to construct a slide embedding $\mathbf{z}_{\text{WSI}}^{j}$.

We aim to represent the set $\mathbf{Z}^{j}$ with a small set of prototypes
$\mathbf{H}=\\{\mathbf{h}_{1},\cdots,\mathbf{h}_{C}\\}$ with
$\mathbf{h}_{c}\in\mathbb{R}^{d}$, $C\ll N_{j}$, without compromising
essential morphological information. Using the prototypes as references, each
patch is aggregated (or mapped) to the references and form
$\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{C\cdot M}$ [49, 64]

$\mathbf{z}_{\text{WSI}}^{j}=[\sum_{n=1}^{N_{j}}\phi^{j}(\mathbf{z}_{n}^{j},\mathbf{h}_{1}),\cdots,\sum_{n=1}^{N_{j}}\phi^{j}(\mathbf{z}_{n}^{j},\mathbf{h}_{C})],$
(1)

where
$\phi^{j}(\cdot,\cdot):\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{M}$
is a function that maps a pair of patch embedding and reference, based on the
similarity between the two, to a post-aggregation prototype embedding. In our
work, the typical dimensions are $C=8\sim 32$ and $N_{j}=10,000\sim 20,000$.
The $\mathbf{z}_{\text{WSI}}^{j}$ dimension is fixed such that a variable-
length set of $N_{j}$ features can always be represented in fixed-length.

To define and estimate the mapping function $\phi^{j}$, we introduce a
probabilistic framework for patch embedding distribution and assume each
$\mathbf{z}_{n}^{j}$ is generated from a GMM,

$\begin{split}p(\mathbf{z}_{n}^{j};\theta^{j})&=\sum_{c=1}^{C}p(c_{n}^{j}=c;\theta^{j})\cdot
p(\mathbf{z}_{n}^{j}|c_{n}^{j}=c;\theta^{j})\\\
&=\sum_{c=1}^{C}\pi_{c}^{j}\cdot\mathcal{N}(\mathbf{z}_{n}^{j};\boldsymbol{\mu}_{c}^{j},\Sigma_{c}^{j}),\,\,s.t.\sum_{c=1}^{C}\pi_{c}^{j}=1,\\\
\end{split}$ (2)

where $\pi_{c}^{j}$ refers to the mixture probability of component $c$ in WSI
set $j$ and $\theta^{j}$ refers to the set of parameters to be estimated,
i.e.,
$\theta^{j}=\\{\pi_{c}^{j},\boldsymbol{\mu}_{c}^{j},\Sigma_{c}^{j}\\}_{c=1}^{C}$.
For ease of computation, we use the diagonal covariance $\Sigma_{c}^{j}$.
Intuitively, Eq. 2 states that a morphological prototype and its variations
correspond to a mixture component, with $\pi_{c}^{j}$ indicating the extent to
which the pattern manifests in the $j^{\text{th}}$ WSI.

We formulate the slide embedding construction as that of estimating
$\widehat{\theta}^{j}$ from $\mathbf{Z}^{j}$. To this end, we maximize the
log-likelihood (or log-posterior if prior is introduced for $\theta^{j}$)

$\max_{\theta^{j}}\log
p(\mathbf{Z}^{j};\theta^{j})=\max_{\theta^{j}}\sum_{n=1}^{N_{j}}\log
p(\mathbf{z}^{j}_{n};\theta^{j}).$ (3)

We use the expectation-maximization (EM) algorithm to obtain the maximum
likelihood estimate $\theta^{j}$ [27, 49], with derivation provided in
Supplementary Information. The algorithm produces a posterior distribution
$q(c_{n}^{j}=c|\mathbf{z}_{n}^{j})$, which represents the probability that
$\mathbf{z}_{n}^{j}$ is associated with prototype $c$. At EM iteration $t+1$,
it is given as

$q^{(t+1)}(c_{n}^{j}=c|\mathbf{z}_{n}^{j})=\dfrac{\pi_{c}^{j,(t)}\mathcal{N}(\mathbf{z}_{n}^{j};\boldsymbol{\mu}_{c}^{j,(t)},\Sigma_{c}^{j,(t)})}{\sum_{c=1}^{C}\pi_{c}^{j,(t)}\mathcal{N}(\mathbf{z}_{n}^{j};\boldsymbol{\mu}_{c}^{j,(t)},\Sigma_{c}^{j,(t)})}.$
(4)

Eq. (4) indicates that a patch is assigned to a certain prototype $c$ that is
most similar, with the similarity measured in terms of weighted
$\mathcal{L}_{2}$ distance. Moreover, the soft prototype assignment, i.e.,
$q^{(t+1)}(c_{n}^{j}=c|\mathbf{z}_{n}^{j})>0,\,\forall c$, allows each patch
to contribute towards all prototypes, in contrast to hard prototype assignment
approaches [81, 91, 69, 88]. Using
$q^{j,(t+1)}_{n,c}=q^{(t+1)}(c_{n}^{j}=c|\mathbf{z}_{n}^{j})$ for notational
simplicity, we can estimate $\theta^{j,(t+1)}$ as

$\begin{split}\pi_{c}^{j,(t+1)}&=\frac{\sum_{n=1}^{N_{j}}q^{j,(t+1)}_{n,c}}{N_{j}},\,\,\,\boldsymbol{\mu}_{c}^{j,(t+1)}=\frac{\sum_{n=1}^{N_{j}}q^{j,(t+1)}_{n,c}\cdot\mathbf{z}_{n}^{j}}{\sum_{n=1}^{N_{j}}q^{j,(t+1)}_{n,c}}\\\
{\Sigma}_{c}^{j,(t+1)}&=\frac{\sum_{n=1}^{N_{j}}q^{j,(t+1)}_{n,c}\cdot(\mathbf{z}_{n}^{j}-\boldsymbol{\mu}_{c}^{j,(t+1)})^{2}}{\sum_{n=1}^{N_{j}}q^{j,(t+1)}_{n,c}}\\\
\end{split}$ (5)

We set $\pi_{c}^{j,(0)}=1/C$, $\boldsymbol{\mu}_{c}^{j,(0)}=\mathbf{h}_{c}$,
and $\Sigma_{c}^{j,(0)}=\mathbf{I}$, which serves as a morphology-aware
initialization. Due to its iterative nature, EM can be placed as a neural
network module. The initialization for $\\{\mathbf{h}_{c}\\}_{c=1}^{C}$ is
performed with K-means clustering on the entire patch training set,
constructed by aggregating patch embeddings from all training slides in a
cohort.

Based on the final estimate $\widehat{\theta}^{j}$ after the EM convergence,
the slide embedding $\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{C\cdot M}$ with
$M=1+2d$ can be represented as a concatenation of the elements in
$\widehat{\theta}^{j}$, following set representation learning literature [64,
49],

$\begin{split}\mathbf{z}_{\text{WSI}}^{j}&=[\mathbf{z}_{\text{WSI},1}^{j},\cdots,\mathbf{z}_{\text{WSI},C}^{j}]\\\
&=[\underbrace{\widehat{\pi}_{1}^{j},\widehat{\boldsymbol{\mu}}_{1}^{j},\widehat{\Sigma}_{1}^{j}}_{\sum_{n=1}^{N_{j}}\phi^{j}(\mathbf{z}_{n}^{j},\mathbf{h}_{1})},\cdots,\underbrace{\widehat{\pi}_{C}^{j},\widehat{\boldsymbol{\mu}}_{C}^{j},\widehat{\Sigma}_{C}^{j}}_{\sum_{n=1}^{N_{j}}\phi^{j}(\mathbf{z}_{n}^{j},\mathbf{h}_{C})}].\end{split}$
(6)

We emphasize that while the prototypes $\\{\mathbf{h}_{c}\\}_{c=1}^{C}$ are
shared across different WSIs, the parameter estimation and the slide embedding
construction are performed per WSI. Overall, the embedding
$\mathbf{z}_{\text{WSI}}^{j}$ satisfies two essential principles for a
faithful WSI representation and good downstream performance. First, it
accounts for the cardinality of each prototype explicitly through
$\widehat{\pi}_{c}^{j}$ and implicitly through
$\widehat{\boldsymbol{\mu}}_{c}^{j}$ and $\widehat{\Sigma}_{c}^{j}$. In
addition, by concatenating the features (rather than averaging), feature
vector for each morphological prototype is directly accessible for downstream
tasks.

####  3.1.1 Connection to optimal transport

The aggregation in Panther can be seen as matching the empirical distribution
of patch embeddings and the prototypes, defined as $\hat{p}_{j}$ and
$\hat{q}_{j}$ respectively. Specifically, we have
$\hat{p}_{j}=\sum_{n=1}^{N_{j}}a_{n}^{j}\cdot\delta_{\mathbf{z}_{n}^{j}}\quad\text{and}\quad\hat{q}_{j}=\sum_{c=1}^{C}\pi_{c}^{j}\cdot\delta_{\mathbf{h}_{c}},$
where $\sum_{n=1}^{N_{j}}a_{n}^{j}=\sum_{c=1}^{C}\pi_{c}^{j}=1$. OT aggregates
$\\{\mathbf{z}_{n}^{j}\\}_{n=1}^{N_{j}}$ to $\\{\mathbf{h}_{c}\\}_{c=1}^{C}$
by minimizing the Wasserstein distance between $\hat{p}_{j}$ and
$\hat{q}_{j}$, typically assuming uniform distribution for
$\\{a_{n}^{j}\\}_{n=1}^{N_{j}}$ and $\\{\pi_{c}^{j}\\}_{c=1}^{C}$, i.e.,
$a_{n}^{j}=1/N_{j}$ and $\pi_{c}^{j}=1/C$ [25, 64]. While Panther also assumes
$a_{n}^{j}=1/N_{j}$, the mixture probability $\pi_{c}^{j}$ is estimated (Eq.
5). Furthermore, the OT solution can be seen as a special case of GMM solution
with uniform prototype distribution [49].

###  3.2 Downstream evaluation

The embedding $\mathbf{z}_{\text{WSI}}^{j}$ can be used as the input to a
predictor module $g(\cdot)$ for various downstream tasks. The predictor
$g(\cdot)$ can be a linear layer (linear probing), or implemented as a
multilayer perceptron (MLP). Instead of using the entire
$\mathbf{z}_{\text{WSI}}^{j}$ as an input to a MLP, we propose a structured
MLP

$\begin{split}\mathbf{z}_{\text{WSI}}^{\prime}&=[g^{\text{indiv.}}_{1}(\mathbf{z}_{\text{WSI},1}),\ldots,g^{\text{indiv.}}_{C}(\mathbf{z}_{\text{WSI},C})]\\\
g(\mathbf{z}_{\text{WSI}})&=g^{\text{pred.}}(\mathbf{z}_{\text{WSI}}^{\prime}),\\\
\end{split}$ (7)

where $j$ is dropped for notational simplicity, and $g^{\text{pred.}}$ and
$\\{g^{\text{indiv.}}_{c}\\}_{c=1}^{C}$ assume one of
$\\{\text{Identity},\text{Linear},\text{MLP}\\}$. Eq. 7 leverages that
$\mathbf{z}_{\text{WSI}}^{j}$, which is a concatenation of mixture estimates,
can be decomposed along each prototype and learns per-prototype nonlinear
mapping $g^{\text{indiv.}}_{c}$. This is not possible with a typical MIL:
First, the large and variable size of $N$ prohibits the learning of
$\\{g_{n}^{\text{indiv.}}\\}_{n=1}^{N}$. Moreover, the permutation invariance
makes finding an “appropriate” function $g_{n}^{\text{indiv.}}$ for a given
patch embedding $\mathbf{z}_{n}$ non-trivial.

###  3.3 Interpretability

Based on the estimated prototype assignment probability $q$, we propose two
approaches for interpretability. First, for each patch embedding
$\mathbf{z}_{n}^{j}$, we assign the prototype with the highest posterior
probability,

$c_{n}^{j}=\operatorname{argmax}_{c}q(c_{n}^{j}=c\mid\mathbf{z}_{n}^{j}),$ (8)

and overlay the prototype assignments onto WSI, visualizing how pathology
visual concepts are distributed within each WSI (prototypical assignment map,
Fig. 3), with $\widehat{\pi}_{c}^{j}$ quantifying the extent of the
distribution. For a specific prototype $c^{\prime}$, we can also visualize how
morphologically similar each patch embedding is to the prototype using
$q(c_{n}^{j}=c^{\prime}\mid\mathbf{z}_{n}^{j})$ (Fig. 1).

##  4 Experiments

###  4.1 Datasets

Subtyping We evaluate Panther on four different subtyping tasks: EBRAINS fine
subtyping (30 classes) and coarse subtyping (12 classes) for rare brain cancer
types [32, 70], Non-Small Cell Lung Carcinoma (NSCLC) subtyping on TCGA and
CPTAC (2 classes), and ISUP grading based on Prostate cancer grade assessment
(PANDA) challenge (6 classes) [9, 8]. We use balanced accuracy and weighted F1
metrics for evaluation on EBRAINS and NSCLC subtyping task and Cohen’s
$\kappa$ for the ISUP grading task.

Survival We evaluate Panther on TCGA across several cancer types: Breast
Invasive Carcinoma (BRCA), Colon and Rectum Adenocarcinoma (CRC), Bladder
Urothelial Carcinoma (BLCA), Uterine corpus endometrial carcinoma (UCEC),
Kidney renal clear cell carcinoma (KIRC), and Lung adenocarcinoma (LUAD). For
TCGA, we use the 5-fold site-stratified cross-validation. For cancer types
with external validation datasets (KIRC: CPTAC, LUAD: CPTAC, NLST), we use the
models trained on TCGA and evaluate on the external dataset. We use the
concordance index (c-index) for evaluation. To address the shortcomings of
overall survival accounting for non-cancerous deaths [57, 14, 42], we use
disease-specific survival (DSS). Additional details can be found in
Supplementary Information.

Table 1: Subtyping prediction Results of Panther and baselines for four different subtyping tasks. All methods use UNI features [18]. Best performance in bold, second best underlined. AttnMISL, ProtoCounts, H2T, OT, and Panther use $C=16$ prototypes. | Train on | EBRAINS | EBRAINS | TCGA-NSCLC | PANDA
---|---|---|---|---|---
|  | (fine, 30 classes) | (coarse, 12 classes) | (2 classes) | (6 classes)
| Test on | EBRAINS | EBRAINS | TCGA | CPTAC | Karolinska | Radboud
|  | (Bal. acc.) | (F1) | (Bal. acc.) | (F1) | (Bal. acc.) | (Bal. acc.) |  (Cohen’s $\kappa$) |  (Cohen’s $\kappa$)
Supervised. |  ABMIL [40] | 0.674 | 0.744 | 0.834 | 0.906 | 0.949 | 0.904 | 0.935 | 0.918
TransMIL [73] | 0.701 | 0.758 | 0.848 | 0.921 | 0.959 | 0.867 | 0.942 | 0.922
DSMIL [55] | 0.648 | 0.698 | 0.824 | 0.882 | 0.980 | 0.791 | 0.909 | 0.911
AttnMISL [88] | 0.534 | 0.636 | 0.647 | 0.823 | 0.888 | 0.823 | 0.882 | 0.894
ILRA [84] | 0.618 | 0.695 | 0.820 | 0.896 | 0.939 | 0.887 | 0.931 | 0.925
Unsup. |  DeepSets [93] | 0.033 | 0.073 | 0.082 | 0.2 | 0.571 | 0.707 |  $<$ 0 |  $<$ 0
ProtoCounts [91, 69] | 0.038 | 0.018 | 0.097 | 0.079 | 0.429 | 0.569 |  $<$ 0 | 0.13
H2T [81] | 0.117 | 0.223 | 0.181 | 0.421 | 0.929 | 0.821 | 0.457 | 0.755
OT [64] | 0.700 | 0.756 | 0.837 | 0.915 | 0.950 | 0.867 | 0.817 | 0.883
Ours |  $\textsc{Panther}_{\text{Top}}$ \+ lin. | 0.471 | 0.571 | 0.554 | 0.758 | 0.857 | 0.833 | 0.631 | 0.689
$\textsc{Panther}_{\text{Bot.}}$ \+ lin. | 0.038 | 0.080 | 0.138 | 0.329 | 0.602 | 0.705 | 0.071 | 0.000
$\textsc{Panther}_{\text{WA}}$ \+ lin. | 0.497 | 0.598 | 0.569 | 0.784 | 0.888 | 0.860 | 0.663 | 0.787
$\textsc{Panther}_{\text{All}}$ \+ lin. | 0.691 | 0.756 | 0.829 | 0.904 | 0.939 | 0.882 | 0.866 | 0.909
$\textsc{Panther}_{\text{All}}$ \+ MLP | 0.693 | 0.760 | 0.854 | 0.908 | 0.980 | 0.906 | 0.923 | 0.931

###  4.2 Baselines

We employ 1) unsupervised baselines, which use unsupervised slide
representation followed by the task-specific linear network, and 2) supervised
baselines, which construct supervised slide representation for each task. For
the unsupervised baselines, we use the following: 1) DeepSets [93] The slide
embedding $\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{d}$ is formed by
averaging all the features in the set. 2) ProtoCounts [69, 91] K-means
clustering is performed on the cohort-aggregated set of features. The slide
embedding $\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{C}$ is a count vector of
the number of patches assigned to each cluster. 3) H2T [81] The patch
embeddings are clustered and averaged within each cluster. The averaged
cluster centroids are concatenated, with
$\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{C\cdot d}$. 4) Optimal Transport
(OT) [64] The patch features of a WSI is aggregated to a set of prototypes
with OT [25], with $\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{C\cdot d}$. OT
assumes uniform mixture probability, i.e., $\pi_{c}^{j}=1/C,\forall c$.

We also implement the following supervised baselines: Attention-based MIL
(ABMIL) [40], Transformer-based MIL (TransMIL) [73], Prototype-clustering
based MIL (AttnMISL) [88], and low-rank MIL (ILRA) [84].

For Panther, we experiment with variations of $\mathbf{z}_{\text{WSI}}^{j}$ to
better understand our model: 1) All (original): All mixture parameters are
concatenated, $\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{C(1+2d)}$. 2)
Weighted avg. (WA): $\boldsymbol{\mu}_{c}$ and $\Sigma_{c}$ weighted-averaged
by $\pi_{c}$ and concatenated,
$\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{2d}$. 3) Top (Bottom): Parameters
for mixture component with the largest (smallest) $\widehat{\pi}^{j}_{c}$ is
selected, $\mathbf{z}_{\text{WSI}}^{j}\in\mathbb{R}^{(1+2d)}$. We use either
linear (+lin.) or nonlinear head (+MLP) on top of
$\mathbf{z}_{\text{WSI}}^{j}$.

For the feature extractor $f_{\text{enc}}(\cdot)$, which is the same for all
baselines used in this work, we used UNI [18], a ViT-L/16 DINOv2 [28, 66] that
was pre-trained on a large internal histology dataset of $1\times 10^{8}$
patches from $1\times 10^{6}$ WSIs***Accessible at:
https://github.com/mahmoodlab/UNI. We also experiment with other feature
encoders, CTransPath [83] and ResNet50 [34], the results of which can be found
in Supplementary Information.

###  4.3 Implementation

WSIs at $20\times$ magnification ($0.5\,\mu m$/pixel) are patched with non-
overlapping patches of $256\times 256$ pixels. For each WSI, we use all
patches without sampling. We found that a single EM step is sufficient for
convergence. The prototypes $\mathbf{H}$ are constructed from K-means
clustering on the set of patches aggregated from the training cohort (all
slides) for each task. The same $\mathbf{H}$ is used for AttnMISL,
ProtoCounts, H2T, OT, and Panther. Additional details on training and loss
functions can be found in Supplementary Information.

##  5 Results

###  5.1 Subtyping and survival prediction

Subtyping and survival prediction results are shown in Table 1 and Table 2.
Overall, Panther consistently outperforms or is on par with all supervised and
unsupervised baselines. We highlight key insights and provide hypotheses for
the high performance of Panther.

Table 2: Survival prediction Results of Panther and baselines for measuring patient disease-specific survival based on c-index. All methods use UNI features [18]. Best performance in bold, second best underlined. All models and prototypes are trained on TCGA. AttnMISL, ProtoCounts, H2T, OT, and Panther use $C=16$ prototypes. Standard deviation (in parentheses) are reported over five runs. | Dataset | BRCA | CRC | BLCA | UCEC | KIRC | LUAD
---|---|---|---|---|---|---|---
| Test on | TCGA | TCGA | TCGA | TCGA | TCGA | CPTAC | TCGA | CPTAC | NLST
Supervised | ABMIL [40] | 0.644 | 0.608 | 0.550 | 0.669 | 0.684 | 0.613 | 0.654 | 0.572 | 0.519
($\pm$ 0.05) |  ($\pm$ 0.09) |  ($\pm$ 0.06) |  ($\pm$ 0.07) |  ($\pm$ 0.06) |  ($\pm$ 0.06) |  ($\pm$ 0.06) |  ($\pm$ 0.03) |  ($\pm$ 0.04)
TransMIL [73] | 0.612 | 0.684 | 0.595 | 0.695 | 0.671 | 0.639 | 0.665 | 0.555 | 0.484
($\pm$ 0.07) |  ($\pm$ 0.06) |  ($\pm$ 0.06) |  ($\pm$ 0.08) |  ($\pm$ 0.10) |  ($\pm$ 0.04) |  ($\pm$ 0.10) |  ($\pm$ 0.03) |  ($\pm$ 0.05)
DSMIL [55] | 0.496 | 0.5 | 0.501 | 0.497 | 0.5 | 0.5 | 0.501 | 0.502 | 0.5
($\pm$ 0.00) |  ($\pm$ 0.00) |  ($\pm$ 0.00) |  ($\pm$ 0.00) |  ($\pm$ 0.00) |  ($\pm$ 0.00) |  ($\pm$ 0.00) |  ($\pm$ 0.00) |  ($\pm$ 0.00)
AttnMISL [88] | 0.627 | 0.639 | 0.485 | 0.581 | 0.649 | 0.608 | 0.673 | 0.632 | 0.577
($\pm$ 0.08) |  ($\pm$ 0.10) |  ($\pm$ 0.06) |  ($\pm$ 0.12) |  ($\pm$ 0.09) |  ($\pm$ 0.06) |  ($\pm$ 0.10) |  ($\pm$ 0.03) |  ($\pm$ 0.04)
ILRA [84] | 0.649 | 0.555 | 0.550 | 0.632 | 0.637 | 0.611 | 0.586 | 0.651 | 0.482
($\pm$ 0.10) |  ($\pm$ 0.10) |  ($\pm$ 0.04) |  ($\pm$ 0.02) |  ($\pm$ 0.14) |  ($\pm$ 0.03) |  ($\pm$ 0.06) |  ($\pm$ 0.05) |  ($\pm$ 0.01)
Unsupervised | DeepSets [93] | 0.673 | 0.563 | 0.581 | 0.730 | 0.715 | 0.634 | 0.652 | 0.550 | 0.509
($\pm$ 0.11) |  ($\pm$ 0.10) |  ($\pm$ 0.05) |  ($\pm$ 0.05) |  ($\pm$ 0.08) |  ($\pm$ 0.01) |  ($\pm$ 0.05) |  ($\pm$ 0.01) |  ($\pm$ 0.04)
ProtoCounts [91, 69] | 0.490 | 0.552 | 0.533 | 0.441 | 0.461 | 0.503 | 0.460 | 0.577 | 0.500
($\pm$ 0.11) |  ($\pm$ 0.06) |  ($\pm$ 0.09) |  ($\pm$ 0.06) |  ($\pm$ 0.06) |  ($\pm$ 0.09) |  ($\pm$ 0.11) |  ($\pm$ 0.11) |  ($\pm$ 0.01)
H2T [81] | 0.672 | 0.639 | 0.566 | 0.715 | 0.703 | 0.631 | 0.662 | 0.583 | 0.603
($\pm$ 0.07) |  ($\pm$ 0.11) |  ($\pm$ 0.05) |  ($\pm$ 0.09) |  ($\pm$ 0.11) |  ($\pm$ 0.04) |  ($\pm$ 0.09) |  ($\pm$ 0.03) |  ($\pm$ 0.04)
OT [64] | 0.755 | 0.622 | 0.603 | 0.747 | 0.695 | 0.650 | 0.687 | 0.641 | 0.495
($\pm$ 0.06) |  ($\pm$ 0.09) |  ($\pm$ 0.04) |  ($\pm$ 0.08) |  ($\pm$ 0.09) |  ($\pm$ 0.02) |  ($\pm$ 0.08) |  ($\pm$ 0.02) |  ($\pm$ 0.04)
Ours | $\textsc{Panther}_{\text{Top}}$ \+ lin. | 0.718 | 0.534 | 0.543 | 0.707 | 0.741 | 0.608 | 0.575 | 0.607 | 0.417
($\pm$ 0.11) |  ($\pm$ 0.08) |  ($\pm$ 0.05) |  ($\pm 0.08$) |  ($\pm$ 0.11) |  ($\pm$ 0.01) |  ($\pm$ 0.05) |  ($\pm$ 0.02) |  ($\pm$ 0.06)
$\textsc{Panther}_{\text{Bot.}}$ \+ lin. | 0.578 | 0.452 | 0.494 | 0.570 | 0.524 | 0.532 | 0.592 | 0.539 | 0.519
($\pm$ 0.13) |  ($\pm$ 0.08) |  ($\pm$ 0.08) |  ($\pm$ 0.11) |  ($\pm$ 0.14) |  ($\pm$ 0.08) |  ($\pm$ 0.10) |  ($\pm$ 0.12) |  ($\pm$ 0.02)
$\textsc{Panther}_{\text{WA}}$ \+ lin. | 0.670 | 0.647 | 0.586 | 0.753 | 0.730 | 0.623 | 0.654 | 0.461 | 0.482
($\pm$ 0.09) |  ($\pm$ 0.12) |  ($\pm$ 0.04) |  ($\pm$ 0.09) |  ($\pm$ 0.07) |  ($\pm$ 0.01) |  ($\pm$ 0.07) |  ($\pm$ 0.01) |  ($\pm$ 0.06)
$\textsc{Panther}_{\text{All}}$ \+ lin. | 0.722 | 0.645 | 0.602 | 0.751 | 0.703 | 0.649 | 0.672 | 0.568 | 0.623
($\pm$ 0.07) |  ($\pm$ 0.07) |  ($\pm$ 0.05) |  ($\pm$ 0.11) |  ($\pm$ 0.13) |  ($\pm$ 0.04) |  ($\pm$ 0.06) |  ($\pm$ 0.05) |  ($\pm$ 0.07)
$\textsc{Panther}_{\text{All}}$ \+ MLP | 0.758 | 0.665 | 0.612 | 0.757 | 0.716 | 0.691 | 0.685 | 0.653 | 0.634
($\pm$ 0.06) |  ($\pm$ 0.10) |  ($\pm$ 0.07) |  ($\pm$ 0.10) |  ($\pm$ 0.10) |  ($\pm$ 0.03) |  ($\pm$ 0.06) |  ($\pm$ 0.04) |  ($\pm$ 0.04)

Panther _vs._ supervised MIL $\textsc{Panther}_{\text{All}}$+MLP outperforms
or is on par with the best-performing supervised baseline on subtyping
(TransMIL) and survival prediction tasks (mix). With linear probing,
$\textsc{Panther}_{\text{All}}$+lin. remains competitive against MIL on
subtyping and performs better on most cancer types in survival prediction,
demonstrating the strong representation quality of
$\mathbf{z}_{\text{WSI}}^{j}$. This is encouraging as Panther builds a slide
representation in an unsupervised fashion, unlike MIL which learns a patch
aggregation end-to-end with the downstream tasks. Interestingly, despite
relying on a similar prototype construction as Panther, AttnMISL [88] performs
consistently lower than Panther. We attribute this difference to AttnMISL
_averaging_ the prototypes with attention weights, whereas Panther builds a
slide embedding by _concatenating_ them. Our baseline
$\textsc{Panther}_{\text{WA}}$+lin. further confirms this by showing that
averaging leads to a consistently lower performance.

Panther vs. unsupervised baselines We observe that
$\textsc{Panther}_{\text{All}}$+MLP outperforms most unsupervised baselines on
subtyping and survival prediction tasks. We attribute this gain to two design
principles behind Panther: (1) prototypes are represented as low-dimensional
feature vectors, and (2) the resulting slide embedding encodes the cardinality
of each prototype, _i.e.,_ their extent in the WSI. In comparison, ProtoCounts
only encodes the count information, leading to poor performance.
Interestingly, DeepSets, which builds slide embeddings as the sum of all patch
embeddings, and similarly our baseline $\textsc{Panther}_{\text{WA}}$+lin.,
which takes weighted averaging of the prototype features, lead to poor
subtyping performance despite implicitly encoding both deep patch
representations and cardinality. We hypothesize that subtyping requires a
mechanism to “isolate” discriminative information, which can be implemented
using attention (as in ABMIL and TransMIL), or using prototype concatenation
as in Panther.

Unsurprisingly, H2T and OT come closest to Panther on subtyping tasks as they
also aggregate the patches to the prototypes (albeit with different mechanisms
from Panther) and use concatenation. However, on ISUP grading in prostate
cancer which is clinically assessed using the primary and secondary Gleason
patterns, Panther sees significant performance increases. Lastly, H2T and OT
lack explicit mechanisms to incorporate cardinality into slide representation,
an important feature of Panther for interpretability (Section 5.2).

In this context, Panther appears as a comprehensive unsupervised slide
representation method that concatenates deep prototype representations along
with their cardinality.

Panther ablations We further ablate Panther by retaining only a single
component with the highest (lowest) $\widehat{\pi}_{c}$, _i.e.,_
$\textsc{Panther}_{\text{Top}}$ ($\textsc{Panther}_{\text{Bot.}}$). We observe
that both $\textsc{Panther}_{\text{Top}}$ and $\textsc{Panther}_{\text{Bot.}}$
performs consistently poorly. This reaffirms that capturing morphological
heterogeneity is crucial for accurate prediction of a patient’s clinical
outcome [63, 80]. That $\textsc{Panther}_{\text{Bot.}}$ is the lowest-
performing agrees with our intuition, as subtypes and grades are most often
determined by pathologists based on visual cues that must be integrated across
the entirety of the tumor, rather than utilizing only a particular region or
morphology within the whole. Interestingly, $\textsc{Panther}_{\text{Top}}$
performs relatively well for the NSCLC subtyping task, which we attribute to
it being a relatively simple binary classification task and the most populous
component (highest $\widehat{\pi}_{c}$) for the NSCLC WSIs on average being
tumor.

Panther with linear vs. non-linear head In all tasks,
$\textsc{Panther}_{\text{All}}$+MLP consistently boosts the performance over
$\textsc{Panther}_{\text{All}}$+lin., demonstrating additional predictive
capability enabled by per-prototype non-linearity modeling.

###  5.2 Interpretability

To understand which prototypes are used in learning slide representations, we
visualize (1) the patch-level prototype assignments per WSI via probability
$q(c_{n}^{j}=c\mid\mathbf{z}_{n}^{j})$, and (2) the distribution of mixture
components $\widehat{\pi}^{j}_{c}$ within and across all WSIs in the cohort.

Overall, we find that GMMs are a simple yet powerful framework for mapping the
spatial organization of histologic visual concepts in the tissue
microenvironment. Visual assessment by a board-certified pathologist
(D.F.K.W.) revealed that prototypical patterns reflect distinct morphological
phenotypes of tumor-, tumor-associated stromal, and immune-cell populations as
well as normal tissue components (Fig. 3C). In NSCLC, we discover prototypes
that correspond to adenocarcinoma (turquoise, C2 and C15) and squamous cell
carcinoma (orange, C12) patterns (Fig. 3A,B). Analysis of $\widehat{\pi}_{c}$
shows that these patterns almost exclusively appeared in LUAD and LUSC slides
(Fig. 3C). In CRC, the distribution of prototypical patterns had strong
concordance with existing tissue annotations for CRC tissue types (CRC-100K)
[47], with further visualizations presented in the Supplementary Information.

Figure 3: Prototype-oriented heatmap interpretation. (A) Examples of WSIs and
prototypical assignment maps from LUAD and LUSC, with estimated prototype
distribution $\hat{\pi}_{c}$ for each WSI. (B) Prototype distribution and
morphological annotations by a board-certified pathologist in the NSCLC
cohort. The adenocarcinoma prototypes (C2, C15) and squamous cell carcinoma
(C12) appear exclusively in LUAD and LUSC respectively, showing that Panther
can correctly capture essential morphological cues in the tissue.

###  5.3 Further ablations

We run further ablation studies and present additional insights in
Supplementary Information. Overall, Panther is robust across different choices
of the number of prototypes $C$, survival loss functions, and feature
encoders.

##  6 Conclusion and limitations

We present Panther, a prototype-based aggregation framework for learning
unsupervised slide representations with Gaussian mixtures as the patch
distribution. We believe this is an important addition to the emerging group
of slide representation studies, with the unsupervised nature of Panther
making it readily applicable to diverse tasks. Limitations include using
$C=16$ for all tasks, which may lead to over- or under-clustering for certain
cancers. Future work includes introducing more expressive mixture models for
patch distributions, determining the number of prototypes in a data-driven
manner, and evaluating on rare cancer cohorts with small sample sizes.

## References

  * [1] Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Nenad Tomasev, Jovana Mitrović, Patricia Strachan, et al.  Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging.  Nature Biomedical Engineering, pages 1–24, 2023.
  * [2] Saurav Basu, Soheil Kolouri, and Gustavo K Rohde.  Detecting and visualizing cell phenotype differences from microscopy images using transport-based morphometry.  Proceedings of the National Academy of Sciences, 111(9):3448–3453, 2014.
  * [3] Andrew H Beck, Ankur R Sangoi, Samuel Leung, Robert J Marinelli, Torsten O Nielsen, Marc J Van De Vijver, Robert B West, Matt Van De Rijn, and Daphne Koller.  Systematic analysis of breast cancer morphology uncovers stromal features associated with survival.  Science Translational Medicine, 3(108):108ra113–108ra113, 2011.
  * [4] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al.  Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.  JAMA, 318(22):2199–2210, 2017.
  * [5] Gianpaolo Bontempo, Angelo Porrello, Federico Bolelli, Simone Calderara, and Elisa Ficarra.  Das-mil: Distilling across scales for mil classification of histological wsis.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 248–258. Springer, 2023.
  * [6] James D Brierley, Mary K Gospodarowicz, and Christian Wittekind.  TNM classification of malignant tumours.  John Wiley & Sons, 2017.
  * [7] Christian Bueno and Alan Hylton.  On the representation power of set pooling networks.  Advances in Neural Information Processing Systems, 34:17170–17182, 2021.
  * [8] Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F Steiner, Hester van Boven, Robert Vink, et al.  Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge.  Nature Medicine, 28(1):154–163, 2022.
  * [9] Wouter Bulten, Hans Pinckaers, Hester van Boven, Robert Vink, Thomas de Bel, Bram van Ginneken, Jeroen van der Laak, Christina Hulsbergen-van de Kaa, and Geert Litjens.  Automated deep-learning system for gleason grading of prostate cancer using biopsies: a diagnostic study.  The Lancet Oncology, 21(2):233–241, 2020.
  * [10] Charlotte Bunne, Stefan G Stark, Gabriele Gut, Jacobo Sarabia Del Castillo, Mitch Levesque, Kjong-Van Lehmann, Lucas Pelkmans, Andreas Krause, and Gunnar Rätsch.  Learning single-cell perturbation responses using neural optimal transport.  Nature methods, pages 1–10, 2023.
  * [11] Dmitrii Bychkov, Nina Linder, Riku Turkki, Stig Nordling, Panu E Kovanen, Clare Verrill, Margarita Walliander, Mikael Lundin, Caj Haglund, and Johan Lundin.  Deep learning based tissue analysis predicts outcome in colorectal cancer.  Scientific Reports, 8(1), 2018.
  * [12] Juan C Caicedo, Angel Cruz, and Fabio A Gonzalez.  Histopathology image classification using bag of features and kernel functions.  In Artificial Intelligence in Medicine: 12th Conference on Artificial Intelligence in Medicine, AIME 2009, Verona, Italy, July 18-22, 2009. Proceedings 12, pages 126–135. Springer, 2009.
  * [13] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas J Fuchs.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images.  Nature medicine, 25(8):1301–1309, 2019.
  * [14] Iain Carmichael, Andrew H Song, Richard J Chen, Drew FK Williamson, Tiffany Y Chen, and Faisal Mahmood.  Incorporating intratumoral heterogeneity into weakly-supervised deep learning models via variance pooling.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 387–397. Springer, 2022.
  * [15] Lyndon Chan, Mahdi S Hosseini, Corwyn Rowsell, Konstantinos N Plataniotis, and Savvas Damaskinos.  Histosegnet: Semantic segmentation of histological tissue type in whole slide images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10662–10671, 2019.
  * [16] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su.  This looks like that: deep learning for interpretable image recognition.  Advances in neural information processing systems, 32, 2019.
  * [17] Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood.  Scaling vision transformers to gigapixel images via hierarchical self-supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16144–16155, 2022.
  * [18] Richard J. Chen, Tong Ding, Ming Y. Lu, Drew F. K. Williamson, Guillaume Jaume, Andrew H. Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, Mane Williams, Lukas Oldenburg, Luca L. Weishaupt, Judy J. Wang, Anurag Vaidya, Long Phi Le, Georg Gerber, Sharifa Sahai, Walt Williams, and Faisal Mahmood.  Towards a general-purpose foundation model for computational pathology.  Nature Medicine, 2024.
  * [19] Yixin Chen and James Z Wang.  Image categorization by learning and reasoning with regions.  The Journal of Machine Learning Research, 5:913–939, 2004.
  * [20] Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li.  Towards understanding the mixture-of-experts layer in deep learning.  Advances in neural information processing systems, 35:23049–23062, 2022.
  * [21] Ozan Ciga, Tony Xu, and Anne Louise Martel.  Self supervised contrastive learning for digital histopathology.  Machine Learning with Applications, 7:100198, 2022.
  * [22] Nicolas Coudray, Paolo Santiago Ocampo, Theodore Sakellaropoulos, Navneet Narula, Matija Snuderl, David Fenyö, Andre L Moreira, Narges Razavian, and Aristotelis Tsirigos.  Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning.  Nature medicine, 24(10):1559–1567, 2018.
  * [23] Angel Cruz-Roa, Juan C Caicedo, and Fabio A González.  Visual pattern mining in histology image collections using bag of features.  Artificial intelligence in medicine, 52(2):91–106, 2011.
  * [24] Yufei Cui, Ziquan Liu, Yixin Chen, Yuchen Lu, Xinyue Yu, Xue Liu, Tei-Wei Kuo, Miguel R. D. Rodrigues, Chun Jason Xue, and Antoni B. Chan.  Retrieval-augmented multiple instance learning.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [25] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  Advances in neural information processing systems, 26, 2013.
  * [26] Dan dan Guo, Long Tian, Minghe Zhang, Mingyuan Zhou, and Hongyuan Zha.  Learning prototype-oriented set representations for meta-learning.  In International Conference on Learning Representations, 2022.
  * [27] Arthur P Dempster, Nan M Laird, and Donald B Rubin.  Maximum likelihood from incomplete data via the em algorithm.  Journal of the royal statistical society: series B (methodological), 39(1):1–22, 1977.
  * [28] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [29] M Murat Dundar, Sunil Badve, Vikas C Raykar, Rohit K Jain, Olcay Sertel, and Metin N Gurcan.  A multiple instance learning approach toward optimal classification of pathology slides.  In 2010 20th International Conference on Pattern Recognition, pages 2732–2735. IEEE, 2010.
  * [30] Kianoush Falahkheirkhah, Alex Xijie Lu, David Alvarez-Melis, and Grace Huynh.  Domain adaptation using optimal transport for invariant learning using histopathology datasets.  In Medical Imaging with Deep Learning, 2023.
  * [31] Alexandre Filiot, Ridouane Ghermi, Antoine Olivier, Paul Jacob, Lucas Fidon, Alice Mac Kain, Charlie Saillard, and Jean-Baptiste Schiratti.  Scaling self-supervised learning for histopathology with masked image modeling.  medRxiv, pages 2023–07, 2023.
  * [32] Gemma Gatta, Riccardo Capocaccia, Laura Botta, Sandra Mallone, Roberta De Angelis, Eva Ardanaz, Harry Comber, Nadya Dimitrova, Maarit K Leinonen, Sabine Siesling, et al.  Burden and centralised treatment in europe of rare tumours: results of rarecarenet—a population-based study.  The Lancet Oncology, 18(8):1022–1039, 2017.
  * [33] Noriaki Hashimoto, Daisuke Fukushima, Ryoichi Koga, Yusuke Takagi, Kaho Ko, Kei Kohno, Masato Nakaguro, Shigeo Nakamura, Hidekata Hontani, and Ichiro Takeuchi.  Multi-scale domain-adversarial multiple-instance cnn for cancer subtype classification with unannotated histopathological images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3852–3861, 2020.
  * [34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [35] Andreas Heindl, Sidra Nawaz, and Yinyin Yuan.  Mapping spatial heterogeneity in the tumor microenvironment: a new era for digital pathology.  Laboratory investigation, 95(4):377–384, 2015.
  * [36] Le Hou, Dimitris Samaras, Tahsin M Kurc, Yi Gao, James E Davis, and Joel H Saltz.  Patch-based convolutional neural network for whole slide tissue image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2424–2433, 2016.
  * [37] Wentai Hou, Lequan Yu, Chengxuan Lin, Helong Huang, Rongshan Yu, Jing Qin, and Liansheng Wang.  H^ 2-mil: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis.  In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 933–941, 2022.
  * [38] Frederick M Howard, James Dolezal, Sara Kochanny, Jefree Schulte, Heather Chen, Lara Heij, Dezheng Huo, Rita Nanda, Olufunmilayo I Olopade, Jakob N Kather, et al.  The impact of site-specific digital histology signatures on deep learning model accuracy and bias.  Nature communications, 12(1):4423, 2021.
  * [39] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou.  A visual–language foundation model for pathology image analysis using medical twitter.  Nature medicine, 29(9):2307–2316, 2023.
  * [40] Maximilian Ilse, Jakub Tomczak, and Max Welling.  Attention-based deep multiple instance learning.  In International conference on machine learning, pages 2127–2136. PMLR, 2018.
  * [41] Guillaume Jaume, Lukas Oldenburg, Anurag J Vaidya, Richard J Chen, Drew FK Williamson, Thomas Peeters, Andrew H Song, and Faisal Mahmood.  Transcriptomics-guided slide representation learning in computational pathology.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
  * [42] Guillaume Jaume, Anurag Vaidya, Richard Chen, Drew Williamson, Paul Liang, and Faisal Mahmood.  Modeling dense multimodal interactions between biological pathways and histology for survival prediction.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  * [43] Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Limin Yu, and Aaditya Prakash.  Additive mil: intrinsically interpretable multiple instance learning for pathology.  Advances in Neural Information Processing Systems, 35:20689–20702, 2022.
  * [44] Cheng Jiang, Xinhai Hou, Akhil Kondepudi, Asadur Chowdury, Christian W Freudiger, Daniel A Orringer, Honglak Lee, and Todd C Hollon.  Hierarchical discriminative learning improves visual representations of biomedical microscopy.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19798–19808, 2023.
  * [45] Shivam Kalra, Hamid R Tizhoosh, Charles Choi, Sultaan Shah, Phedias Diamandis, Clinton JV Campbell, and Liron Pantanowitz.  Yottixel–an image search engine for large archives of histopathology whole slide images.  Medical Image Analysis, 65:101757, 2020.
  * [46] Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo, and Sérgio Pereira.  Benchmarking self-supervised learning on diverse pathology datasets.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3344–3354, 2023.
  * [47] Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, et al.  Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study.  PLoS Medicine, 16(1), 2019.
  * [48] Jared L Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger.  Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network.  BMC medical research methodology, 18(1):1–12, 2018.
  * [49] Minyoung Kim.  Differentiable expectation-maximization for set representation learning.  In International Conference on Learning Representations, 2022.
  * [50] Soheil Kolouri, Se Rim Park, Matthew Thorpe, Dejan Slepcev, and Gustavo K. Rohde.  Optimal mass transport: Signal processing and machine-learning applications.  IEEE Signal Processing Magazine, 34(4):43–59, 2017.
  * [51] Daisuke Komura, Akihiro Kawabe, Keisuke Fukuta, Kyohei Sano, Toshikazu Umezaki, Hirotomo Koda, Ryohei Suzuki, Ken Tominaga, Mieko Ochi, Hiroki Konishi, et al.  Universal encoding of pan-cancer histology by deep texture representations.  Cell Reports, 38(9), 2022.
  * [52] Tristan Lazard, Marvin Lerousseau, Etienne Decencière, and Thomas Walter.  Giga-ssl: Self-supervised learning for gigapixel images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4304–4313, 2023.
  * [53] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce.  A sparse texture representation using local affine regions.  IEEE transactions on pattern analysis and machine intelligence, 27(8):1265–1278, 2005.
  * [54] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In International conference on machine learning, pages 3744–3753. PMLR, 2019.
  * [55] Bin Li, Yin Li, and Kevin W Eliceiri.  Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14318–14328, 2021.
  * [56] Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang-Wen Chen.  Interventional bag multi-instance learning on whole-slide pathological images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19830–19839, 2023.
  * [57] Jianfang Liu, Tara Lichtenberg, Katherine A Hoadley, Laila M Poisson, Alexander J Lazar, Andrew D Cherniack, Albert J Kovatich, Christopher C Benz, Douglas A Levine, Adrian V Lee, et al.  An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcome analytics.  Cell, 173(2):400–416, 2018.
  * [58] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al.  A visual-language foundation model for computational pathology.  Nature Medicine, pages 1–12, 2024.
  * [59] Ming Y Lu, Bowen Chen, Andrew Zhang, Drew FK Williamson, Richard J Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood.  Visual language pretrained multiple instance zero-shot transfer for histopathology images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19764–19775, 2023.
  * [60] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood.  Data-efficient and weakly supervised computational pathology on whole-slide images.  Nature biomedical engineering, 5(6):555–570, 2021.
  * [61] Wenqi Lu, Simon Graham, Mohsin Bilal, Nasir Rajpoot, and Fayyaz Minhas.  Capturing cellular topology in multi-gigapixel pathology images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 260–261, 2020.
  * [62] Margaux Luck, Tristan Sylvain, Joseph Paul Cohen, Heloise Cardinal, Andrea Lodi, and Yoshua Bengio.  Learning to rank for censored survival data.  arXiv preprint arXiv:1806.01984, 2018.
  * [63] Andriy Marusyk, Michalina Janiszewska, and Kornelia Polyak.  Intratumor heterogeneity: the rosetta stone of therapy resistance.  Cancer cell, 37(4):471–484, 2020.
  * [64] Grégoire Mialon, Dexiong Chen, Alexandre d’Aspremont, and Julien Mairal.  A trainable optimal transport embedding for feature aggregation and its relationship to attention.  In International Conference on Learning Representations, 2021.
  * [65] Pooya Mobadersany, Safoora Yousefi, Mohamed Amgad, David A Gutman, Jill S Barnholtz-Sloan, José E Velázquez Vega, Daniel J Brat, and Lee AD Cooper.  Predicting cancer outcomes from histology and genomics using convolutional networks.  Proceedings of the National Academy of Sciences, 115(13):E2970–E2979, 2018.
  * [66] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  DINOv2: Learning robust visual features without supervision.  Transactions on Machine Learning Research, 2024.
  * [67] Wentao Pan, Jiangpeng Yan, Hanbo Chen, Jiawei Yang, Zhe Xu, Xiu Li, and Jianhua Yao.  Human-machine interactive tissue prototype learning for label-efficient histopathology image segmentation.  In International Conference on Information Processing in Medical Imaging, pages 679–691. Springer, 2023.
  * [68] Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik J Shah, Yann LeCun, and Rama Chellappa.  VoLTA: Vision-language transformer with weakly-supervised local-feature alignment.  Transactions on Machine Learning Research, 2023.
  * [69] Adalberto Claudio Quiros, Nicolas Coudray, Anna Yeaton, Xinyu Yang, Bojing Liu, Hortense Le, Luis Chiriboga, Afreen Karimkhan, Navneet Narula, David A. Moore, Christopher Y. Park, Harvey Pass, Andre L. Moreira, John Le Quesne, Aristotelis Tsirigos, and Ke Yuan.  Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unlabeled, unannotated pathology slides, 2023.
  * [70] Thomas Roetzer-Pejrimovsky, Anna-Christina Moser, Baran Atli, Clemens Christian Vogel, Petra A Mercea, Romana Prihoda, Ellen Gelpi, Christine Haberler, Romana Höftberger, Johannes A Hainfellner, et al.  The digital brain tumour atlas, an open histopathology resource.  Scientific Data, 9(1):55, 2022.
  * [71] Joel Saltz, Rajarsi Gupta, Le Hou, Tahsin Kurc, Pankaj Singh, Vu Nguyen, Dimitris Samaras, Kenneth R Shroyer, Tianhao Zhao, Rebecca Batiste, et al.  Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images.  Cell reports, 23(1):181–193, 2018.
  * [72] Geoffrey Schiebinger, Jian Shu, Marcin Tabaka, Brian Cleary, Vidya Subramanian, Aryeh Solomon, Joshua Gould, Siyan Liu, Stacie Lin, Peter Berube, et al.  Optimal-transport analysis of single-cell gene expression identifies developmental trajectories in reprogramming.  Cell, 176(4):928–943, 2019.
  * [73] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al.  Transmil: Transformer based correlated multiple instance learning for whole slide image classification.  Advances in Neural Information Processing Systems, 34:2136–2147, 2021.
  * [74] Sivic and Zisserman.  Video google: A text retrieval approach to object matching in videos.  In Proceedings ninth IEEE international conference on computer vision, pages 1470–1477. IEEE, 2003.
  * [75] Ole-Johan Skrede, Sepp De Raedt, Andreas Kleppe, Tarjei S Hveem, Knut Liestøl, John Maddison, Hanne A Askautrud, Manohar Pradhan, John Arne Nesheim, Fritz Albregtsen, et al.  Deep learning for prediction of colorectal cancer outcome: a discovery and validation study.  The Lancet, 395(10221):350–360, 2020.
  * [76] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in neural information processing systems, 30, 2017.
  * [77] Andrew H Song, Guillaume Jaume, Drew FK Williamson, Ming Y Lu, Anurag Vaidya, Tiffany R Miller, and Faisal Mahmood.  Artificial intelligence for digital and computational pathology.  Nature Reviews Bioengineering, 1(12):930–949, 2023.
  * [78] Wenhao Tang, Sheng Huang, Xiaoxian Zhang, Fengtao Zhou, Yi Zhang, and Bo Liu.  Multiple instance learning framework with masked hard instance mining for whole slide image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4078–4087, 2023.
  * [79] Kevin Thandiackal, Boqi Chen, Pushpak Pati, Guillaume Jaume, Drew FK Williamson, Maria Gabrani, and Orcun Goksel.  Differentiable zooming for multiple instance learning on whole-slide images.  In European Conference on Computer Vision, pages 699–715. Springer, 2022.
  * [80] Ilio Vitale, Efrat Shema, Sherene Loi, and Lorenzo Galluzzi.  Intratumoral heterogeneity in cancer progression and response to immunotherapy.  Nature Medicine, 27(2):212–224, 2021.
  * [81] Quoc Dang Vu, Kashif Rajpoot, Shan E. Ahmed Raza, and Nasir Rajpoot.  Handcrafted histological transformer (h2t): Unsupervised representation of whole slide images.  Medical Image Analysis, 85:102743, 2023.
  * [82] Tiep Huu Vu, Hojjat Seyed Mousavi, Vishal Monga, Ganesh Rao, and UK Arvind Rao.  Histopathological image classification using discriminative feature-oriented dictionary learning.  IEEE transactions on medical imaging, 35(3):738–751, 2015.
  * [83] Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han.  Transformer-based unsupervised contrastive learning for histopathological image classification.  Medical image analysis, 81:102559, 2022.
  * [84] Jinxi Xiang and Jun Zhang.  Exploring low-rank property in multiple instance learning for whole slide image classification.  In The Eleventh International Conference on Learning Representations, 2023.
  * [85] Yingxue Xu and Hao Chen.  Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21241–21251, 2023.
  * [86] Yan Xu, Jun-Yan Zhu, Eric Chang, and Zhuowen Tu.  Multiple clustered instance learning for histopathology cancer image classification, segmentation and clustering.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 964–971. IEEE, 2012.
  * [87] Litao Yang, Deval Mehta, Sidong Liu, Dwarikanath Mahapatra, Antonio Di Ieva, and Zongyuan Ge.  TPMIL: Trainable prototype enhanced multiple instance learning for whole slide image classification.  In Medical Imaging with Deep Learning, 2023.
  * [88] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang.  Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks.  Medical Image Analysis, 65:101789, 2020.
  * [89] Hangting Ye, Wei Fan, Xiaozhuang Song, Shun Zheng, He Zhao, Dan dan Guo, and Yi Chang.  PTaRL: Prototype-based tabular representation learning via space calibration.  In The Twelfth International Conference on Learning Representations, 2024.
  * [90] Anna Yeaton, Rahul G Krishnan, Rebecca Mieloszyk, David Alvarez-Melis, and Grace Huynh.  Hierarchical optimal transport for comparing histopathology datasets.  In Medical Imaging with Deep Learning, 2022.
  * [91] Jin-Gang Yu, Zihao Wu, Yu Ming, Shule Deng, Yuanqing Li, Caifeng Ou, Chunjiang He, Baiye Wang, Pusheng Zhang, and Yu Wang.  Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images.  Medical Image Analysis, 85:102748, 2023.
  * [92] Shekoufeh Gorgi Zadeh and Matthias Schmid.  Bias in cross-entropy-based training of deep survival networks.  IEEE transactions on pattern analysis and machine intelligence, 43(9):3126–3137, 2020.
  * [93] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola.  Deep sets.  Advances in neural information processing systems, 30, 2017.
  * [94] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng.  Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18802–18812, 2022.
  * [95] Jianguo Zhang, Marcin Marszałek, Svetlana Lazebnik, and Cordelia Schmid.  Local features and kernels for classification of texture and object categories: A comprehensive study.  International journal of computer vision, 73:213–238, 2007.
  * [96] Xinliang Zhu, Jiawen Yao, Feiyun Zhu, and Junzhou Huang.  Wsisa: Making survival prediction from whole slide histopathological images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7234–7242, 2017.

Supplementary Material

##  1 Mathematical derivations of EM algorithm

Given the Gaussian mixture model (GMM) for a generative model for each
individual patch embedding, we provide a detailed derivation for estimation of
1) the posterior probability for the prototype assignment
$q(c|\mathbf{z}_{n}^{j};\theta_{j})$ and 2) the GMM parameters
$\theta^{j}=\\{\pi_{c}^{j},\boldsymbol{\mu}_{c}^{j},\Sigma_{c}^{j}\\}$. Given
the GMM specification,

$\begin{split}p(\mathbf{z}_{n}^{j};\theta^{j})&=\sum_{c=1}^{C}p(c_{n}^{j}=c;\theta^{j})\cdot
p(\mathbf{z}_{n}^{j}|c_{n}^{j}=c;\theta^{j})\\\
&=\sum_{c=1}^{C}\pi_{c}^{j}\cdot\mathcal{N}(\mathbf{z}_{n}^{j};\boldsymbol{\mu}_{c}^{j},\Sigma_{c}^{j}),\,\,s.t.\sum_{c=1}^{C}\pi_{c}^{j}=1,\\\
\end{split}$ (1)

the goal is to estimate $\theta^{j}$ that maximizes the log-likelihood

$\max_{\theta^{j}}\log
p(\mathbf{Z}^{j};\theta^{j})=\max_{\theta^{j}}\sum_{n=1}^{N_{j}}\log
p(\mathbf{z}^{j}_{n};\theta^{j}).$ (2)

Using Jensen’s inequality, we obtain the following lower bound for the log-
likelihood,

$\begin{split}&\sum_{n=1}^{N_{j}}\log p(\mathbf{z}^{j}_{n};\theta^{j})\\\
&=\sum_{n=1}^{N_{j}}\log\sum_{c=1}^{C}p(\mathbf{z}_{n}^{j},c_{n}^{j}=c;\theta^{j})\\\
&=\sum_{n=1}^{N_{j}}\log\sum_{c=1}^{C}q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})\cdot\frac{p(\mathbf{z}_{n}^{j},c_{n}^{j}=c;\theta^{j})}{q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})}\\\
&\geq\sum_{n=1}^{N_{j}}\sum_{c=1}^{C}q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})\log\frac{p(\mathbf{z}_{n}^{j},c_{n}^{j}=c;\theta^{j})}{q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})}\\\
&=\sum_{n=1}^{N_{j}}\underbrace{E_{q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})}\left[\log
p(\mathbf{z}_{n}^{j},c_{n}^{j}=c;\theta^{j})\right]}_{Q(\theta^{j};\theta^{j}_{\text{old}})}\\\
&\,\,\quad-\sum_{n=1}^{N_{j}}\underbrace{E_{q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})}\left[q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})\right]}_{-H(C;\theta^{j}_{\text{old}})}.\\\
\end{split}$ (3)

This allows us to substitute the problem of maximizing the log-likelihood with
that of maximizing a surrogate function, which in our case is the lower bound
given by Jensen’s inequality. It can be shown that increasing the lower bound
with respect to $\theta^{j}$ leads to monotonically increasing log-likelihood
[27, 49]. The optimization of the surrogate function towards maximizing the
log-likelihood is often referred to as the Expectation-Maximization (EM)
algorithm, which iteratively alternates between the E-step and the M-step.

The surrogate function is comprised of the two terms,
$Q(\theta^{j};\theta^{j}_{\text{old}})$ and $H(C;\theta^{j}_{\text{old}})$,
both of which are expectations with respect to the posterior probability of
prototype assignment, _i.e.,_
$q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})$. In the E-step, we
use Bayes’ rule to compute the posterior probability and consequently the
expectations,

$\begin{split}&q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})\\\
&=\frac{q(\mathbf{z}_{n}^{j}|c_{n}^{j}=c;\theta^{j}_{\text{old}})\cdot
q(c_{n}^{j}=c;\theta^{j}_{\text{old}})}{q(\mathbf{z}_{n}^{j};\theta^{j}_{\text{old}})}\\\
&=\frac{q(\mathbf{z}_{n}^{j}|c_{n}^{j}=c;\theta^{j}_{\text{old}})\cdot
q(c_{n}^{j}=c;\theta^{j}_{\text{old}})}{\sum_{c=1}^{C}q(\mathbf{z}_{n}^{j}|c_{n}^{j}=c;\theta^{j}_{\text{old}})\cdot
q(c_{n}^{j}=c;\theta^{j}_{\text{old}})}\\\
&=\frac{\pi_{c}^{j}\cdot\mathcal{N}(\mathbf{z}_{n}^{j};\boldsymbol{\mu}_{c}^{j},\Sigma_{c}^{j})}{\sum_{c=1}^{C}\pi_{c}^{j}\cdot\mathcal{N}(\mathbf{z}_{n}^{j};\boldsymbol{\mu}_{c}^{j},\Sigma_{c}^{j})}.\\\
\end{split}$ (4)

In the M-step, we find $\theta^{j}_{\text{new}}$ that maximizes the surrogate
function. Since the entropy term $H(C;\theta^{j}_{\text{old}})$ is not a
function of $\theta_{j}$ (it is a function of $\theta^{j}_{\text{old}}$), we
only need to optimize $Q(\theta^{j};\theta^{j}_{\text{old}})$ by taking the
derivative with respect to $\theta_{j}$,

$\begin{split}&\sum_{n=1}^{N_{j}}\frac{\partial
Q(\theta^{j};\theta^{j}_{\text{old}})}{\partial\pi_{c}^{j}}=0\\\
&\Rightarrow\pi_{c}^{j,\text{new}}=\frac{\sum_{n=1}^{N_{j}}q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta_{\text{old}}^{j})}{N_{j}}\\\
&\sum_{n=1}^{N_{j}}\frac{\partial
Q(\theta^{j};\theta^{j}_{\text{old}})}{\partial\boldsymbol{\mu}_{c}^{j}}=0\\\
&\Rightarrow\boldsymbol{\mu}_{c}^{j,\text{new}}=\frac{\sum_{n=1}^{N_{j}}q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta_{\text{old}}^{j})\cdot\mathbf{z}_{n}^{j}}{\sum_{n=1}^{N_{j}}q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta_{\text{old}}^{j})}\\\
&\sum_{n=1}^{N_{j}}\frac{\partial
Q(\theta^{j};\theta^{j}_{\text{old}})}{\partial\Sigma_{c}^{j}}=0\\\
&\Rightarrow\Sigma_{c}^{j,\text{new}}=\frac{\sum_{n=1}^{N_{j}}q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta_{\text{old}}^{j})\cdot(\mathbf{z}_{n}^{j}-\boldsymbol{\mu}_{c}^{j,\text{new}})^{2}}{\sum_{n=1}^{N_{j}}q(c_{n}^{j}=c|\mathbf{z}_{n}^{j};\theta_{\text{old}}^{j})}.\\\
\end{split}$ (5)

##  2 Training details

For training, we use weight decay of $1\times 10^{-5}$ and AdamW optimizer
with a learning rate of $1\times 10^{-4}$ with the cosine decay scheduler. For
slide classification experiments, we use cross-entropy loss and a maximum of
20 epochs with early stopping if the validation loss does not decrease for 10
epochs. For the supervised baselines, due to the variable-length WSI set, we
use a batch size of 1 and a gradient accumulation of 32 steps. For
unsupervised baselines (including Panther), we use a batch size of 32. For
survival prediction experiments, we use negative log-likelihood loss (NLL)
[92] with a batch size of one patient over 20 epochs for supervised baselines.
For unsupervised baselines (including Panther), we use Cox proportional
hazards loss [48] with a batch size of 64 patients over 50 epochs.

##  3 Computational considerations

Two NVIDIA 3090 GPUs were used for training Panther. Panther pre-extracts
32,784-dim slide features (16 prototypes $\times$ 2,049-dim for concatenated
$\pi_{c}$, $\boldsymbol{\mu}_{c},\Sigma_{c}$) for linear or MLP probing,
468$\times$ smaller than $[15\text{K}\times 1024]$-dim patch patch embeddings
used for MIL training. We pre-extract Panther features with batch size of 1
(10 WSIs/sec), and can compress 11K TCGA slides (4 TB) with $\sim$ 1.4 GB.
While more prototypes imply more features to concatenate, training a linear
classifier with Panther features still has less number of parameters (32,784)
than ABMIL ($\sim$500K).

##  4 Datasets

We provide brief explanations for the datasets that were used for the
evaluation of Panther.

###  4.1 Slide classification

EBRAINS [70]: For fine-grained (30 classes) and coarse-grained (12 classes)
brain tumor subtyping tasks, we used Hematoxylin and Eosin (H&E) Formalin-
fixed and paraffin-embedded (FFPE) WSIs ($n=2,319$) collected from the
University of Vienna. We label-stratify the dataset into train/val/test fold
of 50:25:25 and use the same fold for both the fine-grained and coarse-grained
subtyping tasks. Performance was evaluated using balanced accuracy and
weighted F1.

NSCLC: For the non-small cell lung carcinoma (NSCLC) subtyping task, we use
H&E WSIs from TCGA and CPTAC for classifying lung adenocarcinoma (LUAD) and
lung squamous cell carcinoma (LUSC) cases. The TCGA cohort contains a total of
1,041 slides (LUAD: 529, LUSC: 512) and the CPTAC cohort consists a total of
1,091 slides (LUAD: 578, LUSC: 513). We label-stratify the TCGA cohort into
train/val/test fold of 80:10:10, with CPTAC used for external validation.
Performance was evaluated using balanced accuracy and weighted F1.

PANDA [8, 9]: For the ISUP grading task, we used prostate cancer core needle
biopsies (n=10,616) from the Prostate Cancer Grade Assessment (PANDA)
challenge. Each biopsy is given an ISUP grade, making this a 6-class
classification task. These biopsies are collected from Karolinska Institute
(KRLS) and Radboud University Medical Center (RUMC). We label-stratify the
PANDA dataset into train/val/test fold of 80:10:10, with the evaluation
performed on KRLS and RUMC cohorts separately. Performance was evaluated using
Cohen’s quadratic weighted Kappa $\kappa^{2}$ metric.

###  4.2 Survival prediction

TCGA: We perform site-stratified 5-fold CV [38] evaluation on the following
cancer types from TCGA: Breast Invasive Carcinoma (BRCA,
$n=1,041,\text{WSI}=1,111$), Colon and Rectum Adenocarcinoma (CRC,
$n=566,\text{WSI}=575$), Bladder Urothelial Carcinoma (BLCA,
$n=373,\text{WSI}=437$), Uterine corpus endometrial carcinoma (UCEC,
$n=504,\text{WSI}=565$), Kidney renal clear cell carcinoma (KIRC,
$n=511,\text{WSI}=517$), and Lung adenocarcinoma (LUAD,
$n=456,\text{WSI}=1,024$). The train/val split is performed on the patient
level.
External dataset (CPTAC, NLST): Using the models trained on TCGA cohort, we
perform external validation on KIRC (CPTAC: $n=180,\text{WSI}=341$) and LUAD
(CPTAC: $n=185,\text{WSI}=486$, NLST: $n=244,\text{WSI}=686$). We note that
evaluation on CPTAC and NLST is much more difficult due to dataset shifts in
image acquisition (differences in H&E stain variability), geographic location
and demographics (social determinants of health affecting access to
healthcare), and other potential biases (differences in follow-up procedures
between TCGA and CPTAC/NLST).

##  5 Additional experiments

Ablation over different feature encoders: We evaluate Panther, which relied on
features extracted with ViT-L/16 DINOv2 pre-trained on a large internal
histology dataset (UNI) [18], with other baselines using features extracted
from 1) CTransPath encoder [83], which is a Swin Transformer pretrained on
29,753 WSIs from TCGA and 2,457 WSIs from the Pathology AI Platform (PAIP),
and 2) ResNet50 encoder pretrained on natural images (ImageNet) [60]. The
results can be found in Table S1.

Ablation over a different number of clusters $C$: We evaluate how Panther and
other baselines (AttnMISL, ProtoCounts, H2T, and OT) that depend on the number
of prototypes $C$ perform across different choices. We report both the
classification and survival prediction results for $C=\\{8,16,32\\}$ in Table
S2.

Ablation over different survival loss functions: For survival prediction
tasks, we also train our unsupervised baselines with 1) the negative log-
likelihood (NLL) loss [92] and 2) the ranking loss [62]. These survival loss
functions have been frequently used as alternatives to the Cox loss in
survival analysis problems, especially in medical imaging literature. To
maintain consistency with the Cox loss experimental setting, we use a batch
size of 64 for training with the NLL and ranking loss. The results can be
found in Table S3, where we include the supervised baseline results with NLL
loss for completeness.

Evaluation was performed on several representative classification and survival
tasks: EBRAINS (challenging difficulty), PANDA (depends on understanding
mixture proportions of tissue patterns), CRC survival prediction (tissue can
be annotated using CRC-100K [47]), and LUAD survival prediction (assessing
out-of-domain generalization).

##  6 Results, interpretation, and insights

Stronger feature encoders improve supervised MIL baselines: We observe
consistent trends that stronger feature encoders improve slide-level tasks,
with models trained using UNI reaching the best performance (Table S1). Across
all MIL architectures and in all classification tasks (except for DSMIL on
RUMC evaluation in PANDA), UNI consistently outperforms ResNet-50 and
CTransPath in head-to-head comparisons. On survival tasks, we note that
CTransPath was additionally pretrained on TCGA, which may produce optimistic
bias in evaluation. However, we find that UNI still outperforms CTransPath on
TCGA-CRC and TCGA-LUAD survival prediction across many architectures.
Interestingly, AttnMISL, which generally underperformed against ABMIL and
TransMIL using ResNet-50 features, becomes one of the top-ranked MIL models in
survival tasks when using UNI features (second-highest c-index in CRC survival
prediction, and the highest c-index in LUAD survival prediction on the TCGA
and NLST cohorts). This can be attributed to the prototypical formulation of
AttnMISL, which depends on the representation quality of the data centroids
for prototypical pooling, which would improve with stronger feature encoders.

Stronger feature encoders enable unsupervised baselines to compete with MIL:
Similar to MIL methods, Panther and other unsupervised slide representation
methods have consistent improvement using UNI over ResNet-50 and CTransPath
(aside from evaluation on PANDA and evaluation using DeepSets/ProtoCounts,
Table S1). Interestingly, we note that OT and Panther using CTransPath
features significantly underperforms against many weakly-supervised baselines
(0.377 / 0.369 / 0.518 balanced accuracies comparing OT, Panther, ABMIL
respectively on EBRAINS; 0.677 / 0.782 / 0.901 $\kappa^{2}$ comparing OT,
Panther, ABMIL respectively on PANDA-KRLS). When using a stronger feature
encoder such as UNI, unsupervised slide representation methods have
significant gains and consequently outperform ABMIL and other MIL baselines.
As in the case of AttnMISL, this can be attributed to the need for strong
pretrained encoders that are able to retrieve similar patch embeddings for
prototypical pooling.

Panther trains stable survival models: Across all evaluation settings
(different number of prototypes $C$ and survival loss functions), we find that
Panther is able to develop high-performing survival models with out-of-domain
generalization (Table S2 and S3). In CRC survival prediction,
$\textsc{Panther}_{\text{All}}$+MLP consistently outperforms all cluster-based
methods within each setting. In comparison to MIL, though
$\textsc{Panther}_{\text{All}}$+MLP with $C$=16 has lower c-index than
TransMIL (0.684), we note that $\textsc{Panther}_{\text{All}}$+MLP with $C$=8
reaches higher performance (0.691). In LUAD survival prediction,
$\textsc{Panther}_{\text{All}}$+MLP is consistently the second best-performing
model on TCGA evaluation, behind OT (best c-index of 0.715 of with $C$=8).
Though many baselines such as OT, H2T, and even DeepSets can reach strong
performance on TCGA, we note that almost all of these methods have unstable
performance on external cohorts, with c-index falling under 0.5 on CPTAC,
NLST, or both.

Panther prototypes capture distinct tumor morphologies: In Fig. S1, S2, and
S3, we visualize prototypical assignment maps and heatmap visualizations of
various cancer types. Consistent with our findings in Fig. 3, Panther is able
to map the spatial organization of histologic visual concepts. In particular,
Panther finds several unique tumor populations, delineating: tumor-invading
muscle and tumor with immune infiltration in BLCA, nested tumor and tumor-
associated connective tissue in BRCA, and clear cell RCC with and without
presentation of poorly-differentiated glands in KIRC. Furthermore, we also
show concordance of our visualizations using a supervised classifier
(developed using patch-level tumor annotations in the TCGA Uniform Tumor
dataset [51]) for tumor tissue classification. Visualizing the posterior
probability heatmap of the tumor prototype with the highest mixture
probability $\widehat{\pi}_{c}$ (greatest presence), we find that our tumor
heatmap visualizations have strong concordance with those generated based on
the results from supervised classifiers.

Panther prototypes capture distribution of tissue classes in CRC-100K: In Fig.
S4, we visualize prototypical assignment maps and their correspondence to
diverse tissue annotations in COADREAD tissue. Using the CRC-100K dataset
(containing 9 tissue classes) [47], we developed a supervised patch-level
classifier to predict tissue assignments for all patches in TCGA-COADREAD
slides. To match the prototypical assignment maps from Panther with the label
distribution in CRC-100K, we applied the previous classifier to the learned
prototypes of Panther, to predict CRC-100K tissue labels. Overall, we find
that the learned prototypes of Panther have strong concordance with
morphologically-relevant and diverse histopathology tissue patterns annotated
by supervised classifiers.

Table S1: Varying pretrained feature extractors. We compare the performance of supervised (top) and unsupervised (bottom) methods with different pretrained encoders, ResNet50 with ImageNet transfer (RN50), CTransPath (CTP), and UNI, on classification tasks (EBRAINS and PANDA) and survival tasks (CRC and LUAD). | Train on | EBRAINS | PANDA | CRC | LUAD
---|---|---|---|---|---
|  | (fine, 30 classes) | (grading, 6 classes) | (survival) | (survival)
| Test on | EBRAINS | KRLS | RUMC | TCGA | TCGA | CPTAC | NLST
|  | (Bal. acc.) | (F1) |  ($\kappa^{2}$) |  ($\kappa^{2}$) | (C-Index) | (C-Index) | (C-Index) | (C-Index)
Sup. (RN50) |  ABMIL [40] | 0.197 | 0.267 | 0.792 | 0.789 |  0.540 $\pm$ 0.08 |  0.537 $\pm$ 0.18 |  0.684 $\pm$ 0.01 |  0.482 $\pm$ 0.03
TransMIL [73] | 0.516 | 0.614 | 0.841 | 0.854 |  0.501 $\pm$ 0.05 |  0.628 $\pm$ 0.11 |  0.555 $\pm$ 0.06 |  0.462 $\pm$ 0.01
DSMIL [55] | 0.401 | 0.510 | 0.681 | 0.728 |  0.494 $\pm$ 0.01 |  0.487 $\pm$ 0.03 |  0.524 $\pm$ 0.02 |  0.519 $\pm$ 0.02
ILRA [84] | 0.509 | 0.599 | 0.860 | 0.880 |  0.588 $\pm$ 0.09 |  0.522 $\pm$ 0.17 |  0.559 $\pm$ 0.07 |  0.443 $\pm$ 0.06
AttnMISL [88] | 0.033 | 0.073 | 0.128 | 0.005 |  0.493 $\pm$ 0.04 |  0.519 $\pm$ 0.16 |  0.666 $\pm$ 0.01 |  0.535 $\pm$ 0.01
Sup. (CTP) |  ABMIL [40] | 0.518 | 0.594 | 0.901 | 0.908 |  0.642 $\pm$ 0.10 |  0.607 $\pm$ 0.03 |  0.545 $\pm$ 0.05 |  0.560 $\pm$ 0.02
TransMIL [73] | 0.642 | 0.71 | 0.911 | 0.918 |  0.611 $\pm$ 0.12 |  0.614 $\pm$ 0.06 |  0.479 $\pm$ 0.05 |  0.508 $\pm$ 0.04
DSMIL [55] | 0.515 | 0.584 | 0.890 | 0.916 |  0.499 $\pm$ 0.03 |  0.552 $\pm$ 0.05 |  0.466 $\pm$ 0.04 |  0.438 $\pm$ 0.01
ILRA [84] | 0.580 | 0.655 | 0.917 | 0.920 |  0.590 $\pm$ 0.09 |  0.602 $\pm$ 0.05 |  0.427 $\pm$ 0.05 |  0.456 $\pm$ 0.03
AttnMISL [88] | 0.033 | 0.073 | 0.402 | 0.837 |  0.627 $\pm$ 0.12 |  0.602 $\pm$ 0.05 |  0.427 $\pm$ 0.05 |  0.456 $\pm$ 0.03
Sup. (UNI) |  ABMIL [40] | 0.674 | 0.744 | 0.935 | 0.918 |  0.608 $\pm$ 0.09 |  0.654 $\pm$ 0.06 |  0.572 $\pm$ 0.03 |  0.519 $\pm$ 0.04
TransMIL [73] | 0.701 | 0.758 | 0.942 | 0.922 |  0.684 $\pm$ 0.06 |  0.665 $\pm$ 0.10 |  0.555 $\pm$ 0.03 |  0.484 $\pm$ 0.05
DSMIL [55] | 0.648 | 0.698 | 0.909 | 0.911 |  0.500 $\pm$ 0.00 |  0.501 $\pm$ 0.00 |  0.502 $\pm$ 0.00 |  0.500 $\pm$ 0.00
ILRA [84] | 0.618 | 0.695 | 0.931 | 0.925 |  0.555 $\pm$ 0.10 |  0.586 $\pm$ 0.06 |  0.651 $\pm$ 0.05 |  0.482 $\pm$ 0.01
AttnMISL [88] | 0.534 | 0.636 | 0.882 | 0.894 |  0.639 $\pm$ 0.10 |  0.673 $\pm$ 0.10 |  0.632 $\pm$ 0.03 |  0.577 $\pm$ 0.04
Unsup. (RN50) |  DeepSets [93] | 0.033 | 0.073 | $<0$ | $<0$ |  0.574 $\pm$ 0.08 |  0.565 $\pm$ 0.12 |  0.715 $\pm$ 0.02 |  0.591 $\pm$ 0.01
ProtoCounts [91, 69] | 0.045 | 0.077 | 0.016 | 0.183 |  0.516 $\pm$ 0.05 |  0.546 $\pm$ 0.04 |  0.499 $\pm$ 0.08 |  0.502 $\pm$ 0.07
H2T [81] | 0.047 | 0.087 | 0.262 | 0.329 |  0.501 $\pm$ 0.10 |  0.585 $\pm$ 0.14 |  0.512 $\pm$ 0.07 |  0.545 $\pm$ 0.04
OT [64] | 0.063 | 0.088 | 0.211 | 0.540 |  0.578 $\pm$ 0.08 |  0.575 $\pm$ 0.14 |  0.581 $\pm$ 0.05 |  0.544 $\pm$ 0.02
$\textsc{Panther}_{\text{WA}}$ \+ lin. | 0.033 | 0.073 | 0.150 | 0.057 |  0.525 $\pm$ 0.09 |  0.586 $\pm$ 0.11 |  0.552 $\pm$ 0.04 |  0.491 $\pm$ 0.01
$\textsc{Panther}_{\text{All}}$ \+ lin. | 0.063 | 0.112 | 0.207 | 0.535 |  0.554 $\pm$ 0.07 |  0.586 $\pm$ 0.12 |  0.548 $\pm$ 0.04 |  0.505 $\pm$ 0.02
$\textsc{Panther}_{\text{All}}$ \+ MLP | 0.142 | 0.216 | 0.550 | 0.665 |  0.585 $\pm$ 0.07 |  0.601 $\pm$ 0.07 |  0.396 $\pm$ 0.04 |  0.465 $\pm$ 0.03
Unsup. (CTP) |  DeepSets [93] | 0.033 | 0.073 | $<0$ | $<0$ |  0.522 $\pm$ 0.10 |  0.629 $\pm$ 0.06 |  0.463 $\pm$ 0.03 |  0.554 $\pm$ 0.05
ProtoCounts [91, 69] | 0.057 | 0.037 | 0.059 | 0.539 |  0.521 $\pm$ 0.05 |  0.479 $\pm$ 0.11 |  0.537 $\pm$ 0.10 |  0.522 $\pm$ 0.11
H2T [81] | 0.053 | 0.124 | 0.333 | 0.704 |  0.545 $\pm$ 0.11 |  0.586 $\pm$ 0.05 |  0.552 $\pm$ 0.07 |  0.562 $\pm$ 0.02
OT [64] | 0.377 | 0.482 | 0.677 | 0.738 |  0.607 $\pm$ 0.09 |  0.690 $\pm$ 0.06 |  0.466 $\pm$ 0.02 |  0.547 $\pm$ 0.06
$\textsc{Panther}_{\text{WA}}$ \+ lin. | 0.033 | 0.073 | 0.203 | 0.586 |  0.533 $\pm$ 0.11 |  0.673 $\pm$ 0.05 |  0.474 $\pm$ 0.03 |  0.515 $\pm$ 0.05
$\textsc{Panther}_{\text{All}}$ \+ lin. | 0.398 | 0.493 | 0.661 | 0.757 |  0.614 $\pm$ 0.09 |  0.672 $\pm$ 0.05 |  0.491 $\pm$ 0.04 |  0.540 $\pm$ 0.06
$\textsc{Panther}_{\text{All}}$ \+ MLP | 0.369 | 0.483 | 0.782 | 0.869 |  0.661 $\pm$ 0.11 |  0.655 $\pm$ 0.08 |  0.584 $\pm$ 0.04 |  0.511 $\pm$ 0.03
Unsup. (UNI) |  DeepSets [93] | 0.033 | 0.073 |  $<$ 0 |  $<$ 0 |  0.563 $\pm$ 0.10 |  0.652 $\pm$ 0.05 |  0.550 $\pm$ 0.01 |  0.509 $\pm$ 0.04
ProtoCounts [91, 69] | 0.038 | 0.018 | $<0$ | 0.13 |  0.552 $\pm$ 0.06 |  0.460 $\pm$ 0.11 |  0.577 $\pm$ 0.11 |  0.500 $\pm$ 0.01
H2T [81] | 0.117 | 0.223 | 0.457 | 0.755 |  0.639 $\pm$ 0.11 |  0.662 $\pm$ 0.09 |  0.583 $\pm$ 0.03 |  0.603 $\pm$ 0.04
OT [64] | 0.700 | 0.756 | 0.817 | 0.883 |  0.622 $\pm$ 0.09 |  0.687 $\pm$ 0.08 |  0.641 $\pm$ 0.02 |  0.495 $\pm$ 0.04
$\textsc{Panther}_{\text{WA}}$ \+ lin. | 0.497 | 0.598 | 0.663 | 0.787 |  0.647 $\pm$ 0.12 |  0.654 $\pm$ 0.07 |  0.461 $\pm$ 0.01 |  0.482 $\pm$ 0.06
$\textsc{Panther}_{\text{All}}$ \+ lin. | 0.691 | 0.756 | 0.866 | 0.909 |  0.645 $\pm$ 0.07 |  0.672 $\pm$ 0.06 |  0.568 $\pm$ 0.05 |  0.623 $\pm$ 0.07
$\textsc{Panther}_{\text{All}}$ \+ MLP | 0.693 | 0.760 | 0.923 | 0.931 |  0.665 $\pm$ 0.10 |  0.685 $\pm$ 0.06 |  0.653 $\pm$ 0.04 |  0.634 $\pm$ 0.04

Table S2: Varying $C$ in cluster-based methods. We compare the performance of cluster-based methods with $C=\\{8,16,32\\}$ on classification tasks (EBRAINS and PANDA) and survival tasks (CRC and LUAD). Top. MIL baselines with no clustering (NC). Bottom. Cluster-based methods with $C=\\{8,16,32\\}$, which include weakly-supervised MIL (AttnMISL) and unsupervised slide representation learning approaches (ProtoCounts, H2T, OT, PANTHER). | Train on | EBRAINS | PANDA | CRC | LUAD
---|---|---|---|---|---
|  | (fine, 30 classes) | (grading, 6 classes) | (survival) | (survival)
| Test on | EBRAINS | KRLS | RUMC | TCGA | TCGA | CPTAC | NLST
|  | (Bal. acc.) | (F1) |  ($\kappa^{2}$) |  ($\kappa^{2}$) | (C-Index) | (C-Index) | (C-Index) | (C-Index)
NC |  ABMIL [40] | 0.674 | 0.744 | 0.935 | 0.918 |  0.608 $\pm$ 0.09 |  0.654 $\pm$ 0.06 |  0.572 $\pm$ 0.03 |  0.519 $\pm$ 0.04
TransMIL [73] | 0.701 | 0.758 | 0.942 | 0.922 |  0.684 $\pm$ 0.06 |  0.665 $\pm$ 0.10 |  0.555 $\pm$ 0.03 |  0.484 $\pm$ 0.05
DSMIL [55] | 0.648 | 0.698 | 0.909 | 0.911 |  0.500 $\pm$ 0.00 |  0.501 $\pm$ 0.00 |  0.502 $\pm$ 0.00 |  0.50 $\pm$ 0.00
ILRA [84] | 0.618 | 0.695 | 0.931 | 0.925 |  0.555 $\pm$ 0.10 |  0.586 $\pm$ 0.06 |  0.651 $\pm$ 0.05 |  0.482 $\pm$ 0.01
DeepSets [93] | 0.033 | 0.073 |  $<$ 0 |  $<$ 0 |  0.563 $\pm$ 0.10 |  0.652 $\pm$ 0.05 |  0.550 $\pm$ 0.01 |  0.509 $\pm$ 0.04
C=8 |  AttnMISL [88] | 0.560 | 0.655 | 0.857 | 0.874 |  0.595 $\pm$ 0.08 |  0.643 $\pm$ 0.07 |  0.644 $\pm$ 0.04 |  0.545 $\pm$ 0.01
ProtoCounts [91, 69] | 0.022 | 0.03 | 0.0 | 0.284 |  0.479 $\pm$ 0.11 |  0.561 $\pm$ 0.06 |  0.625 $\pm$ 0.13 |  0.562 $\pm$ 0.15
H2T [81] | 0.045 | 0.105 | 0.286 | 0.767 |  0.622 $\pm$ 0.09 |  0.638 $\pm$ 0.08 |  0.525 $\pm$ 0.04 |  0.563 $\pm$ 0.03
OT [64] | 0.689 | 0.756 | 0.803 | 0.869 |  0.626 $\pm$ 0.09 |  0.715 $\pm$ 0.10 |  0.609 $\pm$ 0.03 |  0.523 $\pm$ 0.05
$\textsc{Panther}_{\text{WA}}$ \+ lin. | 0.490 | 0.593 | 0.671 | 0.787 |  0.600 $\pm$ 0.10 |  0.670 $\pm$ 0.05 |  0.526 $\pm$ 0.01 |  0.502 $\pm$ 0.07
$\textsc{Panther}_{\text{All}}$ \+ lin. | 0.668 | 0.742 | 0.843 | 0.9 |  0.678 $\pm$ 0.12 |  0.643 $\pm$ 0.06 |  0.660 $\pm$ 0.02 |  0.615 $\pm$ 0.03
$\textsc{Panther}_{\text{All}}$ \+ MLP | 0.674 | 0.753 | 0.918 | 0.936 |  0.691 $\pm$ 0.11 |  0.648 $\pm$ 0.06 |  0.669 $\pm$ 0.03 |  0.603 $\pm$ 0.03
C=16 |  AttnMISL [88] | 0.534 | 0.636 | 0.882 | 0.894 |  0.639 $\pm$ 0.10 |  0.673 $\pm$ 0.10 |  0.632 $\pm$ 0.03 |  0.577 $\pm$ 0.04
ProtoCounts [91, 69] | 0.038 | 0.018 | $<0$ | 0.13 |  0.552 $\pm$ 0.06 |  0.460 $\pm$ 0.11 |  0.577 $\pm$ 0.11 |  0.500 $\pm$ 0.01
H2T [81] | 0.117 | 0.223 | 0.457 | 0.755 |  0.639 $\pm$ 0.11 |  0.662 $\pm$ 0.09 |  0.583 $\pm$ 0.03 |  0.603 $\pm$ 0.04
OT [64] | 0.700 | 0.756 | 0.817 | 0.883 |  0.622 $\pm$ 0.09 |  0.687 $\pm$ 0.08 |  0.641 $\pm$ 0.02 |  0.495 $\pm$ 0.04
$\textsc{Panther}_{\text{WA}}$ \+ lin. | 0.497 | 0.598 | 0.663 | 0.787 |  0.647 $\pm$ 0.12 |  0.654 $\pm$ 0.07 |  0.461 $\pm$ 0.01 |  0.482 $\pm$ 0.06
$\textsc{Panther}_{\text{All}}$ \+ lin. | 0.691 | 0.756 | 0.866 | 0.909 |  0.645 $\pm$ 0.07 |  0.672 $\pm$ 0.06 |  0.568 $\pm$ 0.05 |  0.623 $\pm$ 0.07
$\textsc{Panther}_{\text{All}}$ \+ MLP | 0.693 | 0.760 | 0.923 | 0.931 |  0.665 $\pm$ 0.10 |  0.685 $\pm$ 0.06 |  0.653 $\pm$ 0.04 |  0.634 $\pm$ 0.04
C=32 |  AttnMISL [88] | 0.492 | 0.598 | 0.901 | 0.889 |  0.572 $\pm$ 0.10 |  0.666 $\pm$ 0.06 |  0.591 $\pm$ 0.03 |  0.587 $\pm$ 0.02
ProtoCounts [91, 69] | 0.073 | 0.105 | 0.301 | 0.54 |  0.578 $\pm$ 0.09 |  0.498 $\pm$ 0.14 |  0.566 $\pm$ 0.12 |  0.529 $\pm$ 0.09
H2T [81] | 0.244 | 0.363 | 0.626 | 0.779 |  0.621 $\pm$ 0.12 |  0.665 $\pm$ 0.05 |  0.599 $\pm$ 0.04 |  0.650 $\pm$ 0.03
OT [64] | 0.687 | 0.746 | 0.841 | 0.898 |  0.605 $\pm$ 0.00 |  0.689 $\pm$ 0.08 |  0.664 $\pm$ 0.02 |  0.518 $\pm$ 0.04
$\textsc{Panther}_{\text{WA}}$ \+ lin. | 0.489 | 0.593 | 0.670 | 0.782 |  0.606 $\pm$ 0.11 |  0.677 $\pm$ 0.06 |  0.522 $\pm$ 0.01 |  0.469 $\pm$ 0.06
$\textsc{Panther}_{\text{All}}$ \+ lin. | 0.676 | 0.751 | 0.883 | 0.896 |  0.649 $\pm$ 0.07 |  0.677 $\pm$ 0.06 |  0.583 $\pm$ 0.06 |  0.594 $\pm$ 0.05
$\textsc{Panther}_{\text{All}}$ \+ MLP | 0.674 | 0.741 | 0.935 | 0.931 |  0.656 $\pm$ 0.13 |  0.676 $\pm$ 0.04 |  0.665 $\pm$ 0.06 |  0.614 $\pm$ 0.05

Table S3: Varying loss function in survival tasks. We compare the performance of all methods with different loss functions, NLL (top), ranking (middle), and Cox loss (bottom), on survival outcome prediction in CRC and LUAD. | Train on | CRC | LUAD
---|---|---|---
| Test on | TCGA | TCGA | CPTAC | NLST
Sup. (NLL) |  ABMIL [40] |  0.608 $\pm$ 0.09 |  0.654 $\pm$ 0.06 |  0.572 $\pm$ 0.03 |  0.519 $\pm$ 0.04
TransMIL [73] |  0.684 $\pm$ 0.06 |  0.665 $\pm$ 0.10 |  0.555 $\pm$ 0.03 |  0.484 $\pm$ 0.05
DSMIL [55] |  0.500 $\pm$ 0.00 |  0.501 $\pm$ 0.00 |  0.502 $\pm$ 0.00 |  0.500 $\pm$ 0.00
ILRA [84] |  0.555 $\pm$ 0.10 |  0.586 $\pm$ 0.06 |  0.651 $\pm$ 0.05 |  0.482 $\pm$ 0.01
AttnMISL [88] |  0.639 $\pm$ 0.10 |  0.673 $\pm$ 0.10 |  0.632 $\pm$ 0.03 |  0.577 $\pm$ 0.04
Unsup. (NLL) |  DeepSets [93] |  0.559 $\pm$ 0.11 |  0.560 $\pm$ 0.19 |  0.659 $\pm$ 0.02 |  0.587 $\pm$ 0.02
ProtoCounts [91, 69] |  0.517 $\pm$ 0.03 |  0.493 $\pm$ 0.04 |  0.496 $\pm$ 0.15 |  0.591 $\pm$ 0.05
H2T [81] |  0.563 $\pm$ 0.08 |  0.498 $\pm$ 0.17 |  0.547 $\pm$ 0.01 |  0.520 $\pm$ 0.02
OT [64] |  0.626 $\pm$ 0.12 |  0.681 $\pm$ 0.09 |  0.615 $\pm$ 0.03 |  0.462 $\pm$ 0.02
$\textsc{Panther}_{\text{WA}}$ \+ lin. |  0.508 $\pm$ 0.10 |  0.647 $\pm$ 0.09 |  0.643 $\pm$ 0.02 |  0.433 $\pm$ 0.04
$\textsc{Panther}_{\text{All}}$ \+ lin. |  0.647 $\pm$ 0.11 |  0.670 $\pm$ 0.08 |  0.651 $\pm$ 0.04 |  0.614 $\pm$ 0.07
$\textsc{Panther}_{\text{All}}$ \+ MLP |  0.649 $\pm$ 0.11 |  0.668 $\pm$ 0.08 |  0.638 $\pm$ 0.08 |  0.607 $\pm$ 0.05
Unsup. (Rank) |  DeepSets [93] |  0.608 $\pm$ 0.11 |  0.614 $\pm$ 0.05 |  0.556 $\pm$ 0.04 |  0.538 $\pm$ 0.04
ProtoCounts [91, 69] |  0.476 $\pm$ 0.05 |  0.503 $\pm$ 0.05 |  0.482 $\pm$ 0.14 |  0.507 $\pm$ 0.08
H2T [81] |  0.598 $\pm$ 0.11 |  0.661 $\pm$ 0.11 |  0.558 $\pm$ 0.02 |  0.620 $\pm$ 0.04
OT [64] |  0.670 $\pm$ 0.11 |  0.643 $\pm$ 0.03 |  0.595 $\pm$ 0.03 |  0.488 $\pm$ 0.05
$\textsc{Panther}_{\text{WA}}$ \+ lin. |  0.626 $\pm$ 0.13 |  0.637 $\pm$ 0.06 |  0.445 $\pm$ 0.02 |  0.518 $\pm$ 0.07
$\textsc{Panther}_{\text{All}}$ \+ lin. |  0.661 $\pm$ 0.07 |  0.677 $\pm$ 0.06 |  0.575 $\pm$ 0.05 |  0.625 $\pm$ 0.06
$\textsc{Panther}_{\text{All}}$ \+ MLP |  0.671 $\pm$ 0.09 |  0.684 $\pm$ 0.06 |  0.651 $\pm$ 0.03 |  0.628 $\pm$ 0.05
Unsup. (Cox) |  DeepSets [93] |  0.563 $\pm$ 0.10 |  0.652 $\pm$ 0.05 |  0.550 $\pm$ 0.01 |  0.509 $\pm$ 0.04
ProtoCounts [91, 69] |  0.552 $\pm$ 0.06 |  0.460 $\pm$ 0.11 |  0.577 $\pm$ 0.11 |  0.500 $\pm$ 0.01
H2T [81] |  0.639 $\pm$ 0.11 |  0.662 $\pm$ 0.09 |  0.583 $\pm$ 0.03 |  0.603 $\pm$ 0.04
OT [64] |  0.622 $\pm$ 0.09 |  0.687 $\pm$ 0.08 |  0.641 $\pm$ 0.02 |  0.495 $\pm$ 0.04
$\textsc{Panther}_{\text{WA}}$ \+ lin. |  0.647 $\pm$ 0.12 |  0.654 $\pm$ 0.07 |  0.461 $\pm$ 0.01 |  0.482 $\pm$ 0.06
$\textsc{Panther}_{\text{All}}$ \+ lin. |  0.645 $\pm$ 0.07 |  0.672 $\pm$ 0.06 |  0.568 $\pm$ 0.05 |  0.623 $\pm$ 0.07
$\textsc{Panther}_{\text{All}}$ \+ MLP |  0.665 $\pm$ 0.10 |  0.685 $\pm$ 0.06 |  0.653 $\pm$ 0.04 |  0.634 $\pm$ 0.04

Figure S1: Prototype-oriented heatmap interpretation of BLCA. (A)
Visualization of prototypical assignment map in an exemplar BLCA H&E WSI, with
zoomed-in histopathology ROI of tumor-invading muscle (C2, C8, C11, C16). We
show the posterior probability heatmap for the tumor-containing C2 prototype,
which has strong concordance with a tumor probability heatmap obtained by a
supervised patch-level classifier for BLCA tumor prediction. (B) Prototype
distribution $\hat{\pi}_{c}$ of the exemplar slide. (C) Morphological
annotations of all prototypes by a board-certified pathologist in the BLCA
cohort.  Figure S2: Prototype-oriented heatmap interpretation of BRCA. (A)
Visualization of prototypical assignment map in an exemplar BRCA H&E WSI, with
zoomed-in histopathology ROI of dense tumor nests (C16) with surrounding
connective tissue (C10), adipose tissue (C9) with tumor presence (C3). We show
the posterior probability heatmap for the tumor-containing C16 prototype,
which has strong concordance with a tumor probability heatmap obtained by a
supervised patch-level classifier for BRCA tumor prediction. (B) Prototype
distribution $\hat{\pi}_{c}$ of the exemplar slide. (C) Morphological
annotations of all prototypes by a board-certified pathologist in the BRCA
cohort.  Figure S3: Prototype-oriented heatmap interpretation of KIRC. (A)
Visualization of prototypical assignment map in an exemplar KIRC/CCRCC H&E
WSI, with zoomed-in histopathology ROI of CCRCC in small-to-medium glands (C7,
C10, C13). We show the posterior probability heatmap for the tumor-containing
C7 prototype, which has strong concordance with a tumor probability heatmap
obtained by a supervised patch-level classifier for CCRCC tumor prediction.
(B) Prototype distribution $\hat{\pi}_{c}$ of the exemplar slide. (C)
Morphological annotations of all prototypes by a board-certified pathologist
in the KIRC cohort.  Figure S4: Prototype-oriented heatmap interpretation of
COADREAD and correspondence with CRC-100K. For exemplar COADREAD slides, we
visualize their prototypical assignment maps and their correspondence with
tissue classes in CRC-100K. Using a supervised patch-level classifier for
predicting the 9 tissue classes in CRC-100K, we predicted tissue classes in
TCGA-COADREAD slides, shown in the far-right column. To match the prototypical
assignment maps from Panther with the label distribution in CRC-100K, we
applied the same classifier to predict CRC-100K tissue labels for the learned
prototypes in Panther (middle-right column). Across all 9 classes, we find
that Panther’s prototypes correspond to morphologically-relevant and semantic
histopathology tissue patterns annotated by supervised classifiers.