#  What makes for good morphology representations for spatial omics?

Eduard Chelebian, Christophe Avenel, Carolina Wählby
Department of Information Technology and SciLifeLab
Uppsala University
Uppsala, Sweden
{eduard.chelebian, christophe.avenel<EMAIL_ADDRESS>

###### Abstract

Spatial omics has transformed our understanding of tissue architecture by
preserving spatial context of gene expression patterns. Simultaneously,
advances in imaging AI have enabled extraction of morphological features
describing the tissue. The intersection of spatial omics and imaging AI
presents opportunities for a more holistic understanding. In this review we
introduce a framework for categorizing spatial omics-morphology combination
methods, focusing on how morphological features can be translated or
integrated into spatial omics analyses. By translation we mean finding
morphological features that spatially correlate with gene expression patterns
with the purpose of predicting gene expression. Such features can be used to
generate super-resolution gene expression maps or infer genetic information
from clinical H&E-stained samples. By integration we mean finding
morphological features that spatially complement gene expression patterns with
the purpose of enriching information. Such features can be used to define
spatial domains, especially where gene expression has preceded morphological
changes and where morphology remains after gene expression. We discuss
learning strategies and directions for further development of the field.

_K_ eywords spatial omics $\cdot$ morphology $\cdot$ representation learning
$\cdot$ integration $\cdot$ translation

##  1 Introduction

Before the advent of spatial omics technologies, single-cell omics relied on
tissue dissociation which resulted in the loss of spatial information for
understanding cellular environments and interactions [40, 54, 7, 49].
Preserving the spatial context of cells provides a more comprehensive view of
cellular heterogeneity and tissue architecture, leading to deeper insights
into the molecular landscape of disease [56, 82, 83]. The early analysis
methods in spatial omics naturally evolved from established analysis
techniques by the single-cell community [69]. However, this reliance often led
to an oversight of the spatial and morphological relationships present within
tissues.

Parallelly, the artificial intelligence (AI) imaging community have
significantly enhanced our ability to analyze and interpret image data. Deep
learning algorithms, convolutional neural networks (CNN) and image
segmentation techniques have enabled for more precise analysis of biological
tissues [19, 60]. Spatial omics, which captures spatially-resolved data and
usually includes imaging data, naturally intersects with such technologies. By
combining the gene expression information from spatial omics and morphological
features from imaging, we can achieve a more holistic understanding of tissue
architecture [41]. Nonetheless, there are some challenges due in part to the
disconnect between the expertise of both communities. It is inherently hard to
simultaneously utilize two different data sources of such high dimensionality
as spatial omics and imaging data. While obtaining features from spatial omics
is relatively established and interpretable, the patterns often directly
representing local gene expression, extracting relevant morphological features
from images requires more understanding and careful validation.

Combining advancements in imaging AI into spatial omics would benefit from a
structured approach to leverage these combined data sources effectively.
Previous reviews and benchmarks have explored separate aspects of the
combination [8, 78] or more generally the applications of AI for spatial omics
[39, 79]. This Review introduces a comprehensive framework to understand such
combination methods by analyzing the distinct ways morphological features can
be utilized. Image morphology can either be translated into features that
correlate with spatial omics, taking advantage of the ease and cost-
effectiveness of acquiring it; or integrated with spatial omics to provide a
richer description of the sample. These approaches are inherently conflicting:
translation focuses on gene-correlated features, while integration benefits
from complementary information for a fuller understanding. This Review aims to
clarify these distinctions to optimize the use of morphological and molecular
data together.

(a)

(b)

Figure 1: (a) Feature extraction from spatial omics and morphology on 10X
Visium spatial transcriptomics and H&E [23]. We use the $n$th spot in Visium
$x_{G}^{n}$ as the center to extract the $n$th image patch $x_{M}^{n}$. The
modality specific encoders $e_{\theta_{G}}$ and $e_{\theta_{M}}$ will output
the molecular feature vector $h_{G}^{n}$ and the morphological feature vector
$h_{M}^{n}$ for position $(i^{n},j^{n})$. (b) Intuition of the framework. By
assuming the amount of relevant spatial omics information is fixed (solid blue
area), we get four different scenarios depending on the morphological
features. We express these four scenarios by the quadrants formed when
presenting the relevance of the morphological features as y-axis and their
shared information with the spatial omics features as x-axis.

##  2 Translation-integration framework for joint analysis of morphology and
spatial omics

Spatial omics, whether imaging-based or sequencing-based, in practice yields a
grid of spatial positions together with gene expression information [54]. The
process for obtaining the gene expression is quite standardized for the
different modalities of spatial omics. What users receive from sequecing-based
solutions such as 10X Visium, slide-seq [55] or stereo-seq [12] is a matrix of
spots or cells representing gene expression together with their spatial
coordinates, ready for downstream analysis. Similarly, imaging-based
solutions, regardless if they are in situ hybridization-based [14, 22], or in
situ sequencing-based [36, 37], yield a stack of images with signals that need
to be detected and decoded to retrieve the location of specific genes. Even if
the specific methods to do so can vary, the general workflow is usually the
same. The individual detections are then aggregated per cell to, again,
construct a matrix of gene expression together with their spatial coordinates.
The gene expression from spatial omics approaches on their own have shown
their biological relevance on countless works [56, 82, 83]. But this
information is fixed and, once the appropriate analyses are in place, it is
not possible to tweak the gene expression to obtain more relevant information.

Many spatial omics approaches are paired with some kind of morphological
imaging, usually stained with hematoxilyn-eosin (H&E) or DAPI. In order to
combine the gene expression with the images, it is common to pair them by
extracting patches using the gene expression coordinates as patch centers, as
shown in Figure 1a. The process of obtaining meaningful characteristics from
images is known as feature extraction. Traditionally, features were extracted
using hand-crafted algorithms with known specific outcomes in mind. However,
learning-based algorithms have proven to capture more powerful in a variety of
medical tasks [19, 60]. Contrary to gene expression from spatial omics, the
features from images can be learned to be task-specific. For instance, the
morphological features extracted for performing cancer grading can be very
different from the morphological features for detecting mitosis for the same
piece of tissue [33]. Thus, the framework that we propose focuses on the
design choices regarding the learned morphological features, their relevance
and their interaction with gene expression.

The features we extract from morphological images should clearly be as
relevant as possible, which is in itself a noteworthy task. Further, relevance
can usually only be measured in these cases by assessing the performance of
the features in a downstream task. But relevance in itself is not sufficient
for a joint analysis of morphology and spatial omics. The morphological
features need to have specific synergies with gene expression patterns in
order to consider them appropriate for different joint analyses. If we
construct a plot with these two dimensions, relevant information in the
morphology features on the y-axis and shared information between morphology
and gene expression on the x-axis, we obtain the four quadrants depicted in
Figure 1b. The two first quadrants (top: translation and integration)
represent scenarios in which features can be used for joint analysis with
spatial omics, while the two last quadrants (bottom: noise and overestimation)
represent scenarios in which features should not be used for joints analyses
with spatial omics:

  * I

Translation. In this scenario, the morphological features contain a high
amount of relevant information and share a high amount of information with
gene expression. These features include the same information as the task-
relevant part of the gene expression. This scenario is ideal for gene
expression prediction to, for instance, generate super-resolution maps or
infer genetic information from clinical H&E-stained samples without incurring
in the costs. These features are typically obtained by specifically training
deep learning models for that purpose.

  * II

Integration. In this scenario, the morphological features contain a high
amount of relevant information but do not share information with gene
expression. These features contain relevant information that has not been
captured by the gene expression. This scenario is ideal for spatial domain
identification due to the time uncoupling between changes in gene expression
and its effect on morphology. These features need to be more general as they
should complement the information in the gene expression and not be redundant.

  * III

Noise. In this scenario, the morphological features do not contain relevant
information nor they share information with gene expression. These features
capture non-relevant variations such as sample-to-sample or staining
variations which cannot be leveraged in a joint analysis with spatial omics as
they will confuse the analysis.

  * IV

Overestimation. In this scenario, the morphological features do not contain
relevant information but they do share information with gene expression. These
features cover the gene expression that is not task-relevant and thus can lead
to an overestimation of the predictive power in, for instance, translation
tasks. These can be housekeeping or constitutive genes that may not be
relevant for the specific downstream task.

This framework already enables rethinking the way we design the morphological
features (e.g. by training a neural network) for the different tasks and
whether the joint analysis is even valuable or not. For instance, if we have
morphological features that are highly correlated with gene expression,
integration is a meaningless, if not harmful, as we would be including
redundant information in our model. Equally, doing translation with
morphological features that do not share information with gene expression
would not be effective.

The formal definition of this framework is presented in Box 1 and we present a
practical example in Box 2.

### Box 1 – Formal framework definition

Let
$\mathcal{D}=\\{(x_{G}^{1},x_{M}^{1},y^{1}),...,(x_{G}^{N},x_{M}^{N},y^{N})\\}$
be the paired multi-modal dataset of two modalities (gene expression $G$ and
morphology $M$) and $N$ number of data points with coordinates
$\\{(i^{1},j^{1}),...,(i^{N},j^{N})\\}$. $y^{n}$ represents the potential
task-relevant information that is contained in each area. The spatial omics
workflow usually consists on processing the data until we obtain, per each
spatial position, a feature corresponding to the point- or aggregate-gene
expression in the area. For the sake of notation, we represent these features
as the result of encoding the spatial omics data with a gene expression-
specific encoder $\boldsymbol{h_{G}}=e_{\theta_{G}}(\boldsymbol{x_{G}})$.
Parallelly, we can represent the morphological features obtained from the
paired images by an image-specific encoder
$\boldsymbol{h_{M}}=e_{\theta_{M}}(\boldsymbol{x_{M}})$. Finally, based on the
ideas of minimal sufficient statistics [59] and information bottleneck theory
[1, 66] proposed by Tian et al. [65], we define
$I(\boldsymbol{a};\boldsymbol{b})$ as the shared information between
$\boldsymbol{a}$ and $\boldsymbol{b}$. As the spatial omics workflow is quite
established by the bioinformatics community, we can assume that the amount of
task-relevant information contained in the gene expression is fixed and
unknown, but not negligible: $I(\boldsymbol{h_{G}};\boldsymbol{y})>0$ (1) The
only thing we can control then is how we train our feature extractor for the
morphology data. Thus, we assume that we can only control $e_{\theta_{M}}$,
the morphology-specific encoder, and the only relationship we can capture is
$I(\boldsymbol{h_{G}};\boldsymbol{h_{M}})$. Undoubtedly, we always want the
morphological features to be as informative as possible:
$\max_{\theta_{M}}I(\boldsymbol{h_{M}};\boldsymbol{y})$ (2) But depending on
what we are using these features for, they will interact differently with gene
expression. This creates four different scenarios depending on the interaction
between the morphology-spatial omics and morphology-relevant spaces. We
interpret these four scenarios by the four quadrants, as depicted in Figure
1b. Translation. We want to ensure that the morphology contains the maximum
task-relevant information. For this we can use the information included in
gene expression as a proxy for relevance. This would require to maximize the
shared information between both modalities while maintaining the relevance of
morphology: $\max_{\theta_{M}}I(\boldsymbol{h_{G}};\boldsymbol{h_{M}})$ (3) so
that the information shared between modalities is relevant, achieving the
sweet point [65]:
$I(\boldsymbol{h_{G}};\boldsymbol{h_{M}})=I(\boldsymbol{h_{G}};\boldsymbol{y})=I(\boldsymbol{h_{M}};\boldsymbol{y})$
(4) Integration. We want to ensure that the fusion, by a fusion module
$f_{\psi}$, between the modalities carries more information than the
individual modalities, minimizing redundancies between modalities, without
adding nuisance information.
$\min_{\theta_{M}}I(\boldsymbol{h_{G}};\boldsymbol{h_{M}})$ (5) so that the
addition of morphology to gene expression carries more task relevant
information than gene expression alone:
$I(f_{\psi}(\boldsymbol{h_{G}},\boldsymbol{h_{M}});\boldsymbol{y})>I(\boldsymbol{h_{G}};\boldsymbol{y})$
(6)

### Box 2 – Practical example for the framework

We want to visualize these four quadrants in practice. For this, we need a
proxy for measuring the relevance of the morphological features and another
for measuring the shared information with gene expression. For relevance, a
simple approach would be to use pathologist annotations on an H&E slide as the
ground truth and with a mutual information-based feature selection approach,
see the information the morphological features share with the annotations.
Thus, the higher in the y-axis the more descriptive the feature is towards
pathologist annotations. For shared information with gene expression, we could
use the absolute Pearson correlation of the feature with its most correlated
gene. This would show if there is a morphological feature that shows the same
spatial patterns expression of a specific gene. We can compare this plot for
two common feature extraction scenarios, using a model pretrained on ImageNet
[20] and using the same model pretrained in a self-supervised way on millions
of H&E slides [17]. What we can immediately see from the plot below is a shift
of the centroid of the distribution further from the third quadrant (noise)
and towards the first quadrant (translation). So, from this simple analysis,
we can conclude for instance that self-supervised pretraining on histology
images would be more appropriate for a translation task than pretraining on
natural images.

##  3 Morphology translation for gene expression prediction

Morphology translation involves identifying morphological features that
spatially correlate with gene expression patterns. The primary application in
this scenario is gene expression prediction. The goal is to learn
morphological features that are highly correlated with gene expression so that
these features can be used independently of gene expression data, either on
new samples or in regions of a sample where spatial gene expression data is
unavailable or sparse (see Figure 2).

Figure 2: The most common task involving morphology translation is the
prediction of gene expression. This can be done either by inferring the gene
expression in a new sample or by imputing the gene expression in between areas
containing gene expression.

Note that, within our framework, we specifically refer to the relationship of
morphological images with spatial omics. Previous work has also tried to
predict spatial gene expression using non-spatial sources [44, 18, 72, 73].

Using spatial gene data, the most common application has been to try and
predict 10X Visium and other sequencing-based spatial trascriptomics
expression from H&E images and we will start this section with an historical
perspective of such methods.

To our knowledge, this field began to take shape with the development of ST-
Net [26], paradoxically entitled "Integrating spatial gene expression and
breast tumor morphology via deep learning" which, per our definition, was a
translation application rather than integration. The authors identified known
breast cancer biomarkers and, even though the correlation values were
moderate, they showed that the translation was relevant.

The still unpublished HisToGene [47] is one of the first works available
introducing the concept of super-resolution for spatial transcriptomics. The
authors translate the gene expression to a dense map of the whole image,
theoretically being able to achieve pixel-level expression. Monjo et al.
trained DeepSpaCE [45] to achieve super-resolution gene expression of areas
between the spots. Additionally, they performed translation on consecutive
sections in a semi-supervised way.

As the field matured, researchers began to incorporate more sophisticated
comparisons. The authors of BrST-Net [52] extended ST-Net by analyzing various
feature extracting techniques, while SEPAL [43] evaluated different approaches
for predicting gene expression: global for the whole-image, patch-based local
and spatially-informed, introducing the need of context in the prediction.
These advances led to the developement of other graph-based methods that
considered the local context such as THItoGene [35], EGN [77] or ErwaNet [13].

Advances in the AI imaging community enabled leveraging the multi-resolution
nature of H&E images. M2ORT [71] and TRIPLEX [16] proposed multi-scale feautre
extractors, achieveing notable correlation improvements accross various
datasets. And, recently, the authors of BLEEP [75] explicitly trained a bi-
modal embedding-based framework for spatial transcriptomics prediction from
H&E.

###  3.1 Learning features correlated with gene expression

The general task for morphology translation methods is to input an image patch
and output the gene expression patterns in that area. This task requires a
series of design choices to effectively capture and predict the complex
relationships between morphological features and gene expression.

####  3.1.1 Selection of training genes

Selecting which genes the model should learn is crucial because sequencing-
based spatial transcriptomics methods typically cover the entire
transcriptome, encompassing nearly 20,000 genes. However, many of these genes
do not exhibit spatial patterns or have low expression levels. To ensure the
learned features are meaningful and fall within relevant quadrants of our
framework, careful gene selection is necessary.

Three main approaches, set by the first works, have been established for gene
selection, which have the same goal:

Highest mean expression. This method involves selecting genes that have the
highest average expression levels across the dataset. The rationale is that
genes with higher mean expression are likely to be more robust and exhibit
clearer spatial patterns. ST-Net and BrST-Net utilized this approach to focus
on the most consistently expressed genes.

Highly variable genes. This strategy targets genes with high variability in
their expression levels across different spatial locations. Genes with
significant expression variability are more likely to capture meaningful
spatial differences. Methods like HisToGene, THItoGene, M2ORT, TRIPLEX, EGC or
ErwaNet have adopted this approach, aiming to enhance the model’s ability to
learn diverse and informative patterns.

Manual selection. In this approach, researchers manually select genes based on
prior biological knowledge or specific research goals. This method allows for
the inclusion of genes known to be relevant to particular conditions or
tissues, ensuring that the selected genes are biologically significant.
DeepSpaCE or BLEEP used manual selection to tailor their model to specific
biomarkers of interest.

These approaches highlight that current methods are not yet capable of
predicting the entire transcriptome due to the excessive noise present in such
highly dimensional data.

####  3.1.2 Training regimes

Once we have selected our training genes, the next crucial step is to choose
how to learn morphological features that correlate with these genes. The
general approach involves a straightforward deep learning regression pipeline:
input – model – output. Here, the input consists of image patches, and the
output is the gene expression patterns for those patches. The main variable in
this setup is the type of model used.

Early methods in this field leaned heavily on convolutional neural networks
(CNNs) due to their proven effectiveness in image analysis. These models excel
in capturing spatial hierarchies through their convolutional layers. For
instance, ST-Net employed DenseNet-121 [30]. DeepSpaCE, another pioneering
method, utilized the simpler VGG16 [58] which offered a straightforward yet
powerful tool for feature extraction. BrST-Net expanded on this approach by
comparing various CNNs architectures and ultimately identified EfficientNet
[63] as the top performer.

One significant shift was towards Transformer-based models, which are adept at
capturing long-range dependencies in data. HisToGene was an early adopter of
the Vision Transformer (ViT) [21], which applies the transformer architecture
directly to image patches, treating them as sequences of tokens. This ability
to capture global context more effectively than traditional CNNs was
beneficial for understanding complex spatial patterns in gene expression data.

To further enhance the understanding of spatial organization, researchers
introduced Graph Neural Networks (GNNs) [74]. SEPAL combined the strengths of
ViTs for obtaining image embeddings with GNNs to learn spatial patterns. This
approach leveraged the global feature capturing of transformers and the
spatial relationship modeling of GNNs, providing a comprehensive framework for
morphology translation. Similarly, THItoGene used Efficient-Capsule Networks
[42], designed to better capture spatial hierarchies, to generate embeddings
and combined ViTs and GNNs for reconstructing gene expression.

The pyramidal nature of histopathological images, with features at multiple
scales, led to the development of multi-level models. M2ORT, for example, used
a hierarchical ViT [15] to accommodate the multi-scale nature of these images.
By processing images at different scales, the hierarchical ViT could capture
fine-grained details as well as broader contextual information, making it
particularly effective for gene expression prediction. TRIPLEX adopted a
similar approach with a multi-resolution architecture based on ResNet-18 [27].

Recent advancements introduced bi-modal embedding-based frameworks, which aim
to create joint representations of images and gene expression data. BLEEP was
the first to explicitly train such a framework, similar to CLIP [51].

####  3.1.3 Training splits

The choice of training splits is critical for the validation of the
generalization abilities of deep learning models, especially given the
typically small size of these datasets.

A common approach is leave-one-out validation. It involves using each sample
as the test set, one at a time, while the remainder are used as a training
set. While it maximizes the use of the dataset, it provides limited view on
the generalization of the model. ST-Net, HisToGene, and THitoGene have
employed leave-one-out validation to assess their models.

Another approach for small datasets is k-fold cross validation, where the
dataset is divided in $k$ subsets and, one at a time, are used as test and the
remaining for training. This ensures that the performance is at least averaged
accross different partitions and it is the approach of BrST-Net, TRIPLEX, EGN
and ErwaNet.

Finally, methods such as SEPAL, BLEEP and M2ORT use traditional
train–validation–test partitions which can suffer from bad splitting choices
but show generalization ability on independent samples. BrST-Net and TRIPLEX
validated their method using leave-one-out and k-fold cross validation but
additionally tested on an independent sample.

###  3.2 Other tasks involving morphology translation

Although super-resolution can be obtained by dense prediction of gene
expression, some authors have proposed other methods to achieve this. The
earliest work published actually achieving pixel-level super-resolution
spatial transcriptomics from H&E is, to our knowledge, XFuse [5]. At the time,
super-resolution lacked a ground-truth benchmark for comparison, prompting the
authors to demonstrate the robustness and potential of their method by
comparing their results with in situ hybridization. iStar by Zhang et al. [81]
is another super-resolution method that aims at predicting gene expression at
super-resolution level. The newer Xenium technology [32] with higher gene
expression resolution, enabled a quantitative comparison with XFuse, showing
that their method enabled predictions that were closer to the ground truth.

One of the few instances where translation by deep learning was made from DAPI
images instead of H&E was demonstrated by us [2]. We utilized imaging-based
spatial omics to annotate marker genes, training CNNs to classify tissue
morphology. This adaptation indicates the versatility of deep learning methods
in handling different imaging types for morphology translation. DAPI imaging
data is often neglected but holds a lot of potential as shown in [9].

We also explored the correlation of latent features in different networks with
gene expression [11]. We discovered that networks trained for cancer
classification tasks [61] already contained features correlating with genes
associated with prostate cancer [23]. Building on this, we introduced MHAST, a
framework that employs self-supervised features to guide the re-assignment of
deconvolved spatial transcriptomics [10]. Our validation on Tangram [6]
demonstrated that MHAST could retrieve more accurate cell instances compared
to original random allocations, showcasing its potential for achieving cell-
level resolution.

More recently, Gao et al. presented IGI-DL [24], a method that uses predicted
gene expression from H&E to predict patient prognosis. This is done by
generating a graph, with predicted gene expression as nodes, to develop a
survival model that outperforms others in breast and colorectal cancer
cohorts.

##  4 Morphology integration for spatial domain identification

Morphology integration involves identifying morphological features that
spatially complement gene expression patterns. The primary application in this
sceneario is spatial domain identification. The goal is to learn morphological
features that are not correlated with gene expression but still add additional
information to define meaningful regions (see Figure 3).

Figure 3: The most common task involving morphology integration is the
identification of spatial domains which can be then used in downstream tasks.

To the best of our knowledge, SpaCell [64] is the first method that combines
morphology with spatial gene expression. They propose a pretrained CNN to
extract morphology that was then combined with gene expression to obtain a
joint latent space. This integrated space was then used for clustering and
domain identification, setting a foundational precedent for future studies.

Hu et al. proposed SpaGCN [29], a graph-based method to further integrate
spatial location with gene expression and histology. In this case, they use
RGB intensity values as morphology descriptors and, through iterative
clustering of the resulting graph they obtain spatial domains.

The still unpublished conST [85] proposes a contrastive approach for
integrating gene expression, spatial information and morphology. The authors
use a pretrained autoencoder to obtain histology representations which they
then combine with gene expression in a graph neural network to obtain common
features. The authors also implemented an interpretability module to explore
the correlation between spots.

In a similar fashion as SpaCell, stLearn [50] also uses a pretrained CNN as a
feature extractor. Together with the spot location, these features are used
for normalization to adjust the gene expression values. They use this adjusted
gene expression for tasks such as trajectory inference of cell-cell
interaction analysis.

More recently, ConGI [80] proposed contrastive learning for obtaining a joint
representation of histology and gene expression. The authors use a pretrained
network as image encoder and three different contrastive losses to model the
relationships between the two modalities.

###  4.1 Extracting features that complement gene expression

A good morphological feature extractor for integration ensures that the
obtained features have complementary information to gene expression and do not
add noise, so as to obtain more meaningful domains compared to using gene
expression alone.

Using CNN pre-trained on large datasets like ImageNet [20] is by far the most
common approach to obtain morphological descriptors for integration and other
tasks. Authors typically use the penultimate layer before prediction. It is
the case for SpaCell and stLearn, which use a pretrained ResNet50 [27] and
ConGI, which uses DenseNet-121 [30]. conST employed an ImageNet pre-trained
masked autoencoder (MAE) [28] to obtain morphological features. In contrast,
SpaGCN does not use any deep learning approach and instead extracts RGB values
from the patches.

As these features are not explicitly trained to capture gene expression
patterns, one could argue that they probably fall in the left half (quadrants
I and III) of Figure 1(b). Authors probably expect that these features are
general enough to add morphological information that is not already contained
in the gene expression. But it is easy to see that networks trained on natural
images or RGB values might not provide features which are relevant enough for
the integration task [9].

###  4.2 Fusion modules for morphological and gene expression features

Apart from the choice of image encoder, it is important to decide how to
integrate the modalities: the fusion module. It is worth noting that all
integration methods use modality specific encoders, due to the fixed workflow
established for spatial omics, but a transformer-based architecture to handle
input tokens for both modalities and use the same encoder for both is an
interesting avenue [84].

No explicit fusion. SpaGCN does not have an explicit fusion module, instead,
it constructs a graph with molecular features as the nodes and a combination
of the spatial location and RGB image intensity features as the distance.
stLearn also does not use a module for fusing the data and instead it uses the
50 principal components (PCs) from the 2048 features pretrained ResNet50
together with the spatial locations to normalize the gene expression.

Representation stitching. SpaCell concatenates the latent representations of
two auto-encoders that input the 2048 last layer features from the pretrained
ResNet50 on the imaging side and the 2048 top highly variable genes from
spatial transcriptomics.

Representation fusion. ConST generates a graph with KNN of spatial coordinates
as edges and a concatenation of the 768 features extracted from the pretrained
MAE and the 300 PCs from the genes as node features. It then generates the
lower dimensional graph with them by self-supervised learning. ConGI also
concatenates the representations from a gene expression MLP encoder and a
DenseNet121 CNN for the images and feeds it into a small neural network to
reduce the dimensionality in a self-supervised way as well.

##  5 Evaluation metrics, datasets and benchmarks

###  5.1 Metrics for assessing the learned representations

Selecting the best possible representation of tissue morphology, both for
translation and integration tasks, requires metrics to evaluate how well the
learned features may solve the task at hand. The metrics introduced by the
pioneering publications in the field are still widely used, despite their
limitations [8, 78]. Figure 4 shows an overview of these commonly used
metrics.

Figure 4: Commonly used metrics for morphology translation and integration.
This synthetic example presents tissue regions as concentric circles. (a)
Translation metrics usually measure the agreement of the true gene expression
with the gene expression predicted from morphology. Pearson’s correlation
coefficient (PCC) and regression metrics such as the root mean squared error
(RMSE) are usually employed. (b) Integration metrics measure the agreement of
expert annotations with the domains defined jointly morphology and spatial
transcriptomics via the adjusted Rand index (ARI). It is common to also define
spatially variable genes that define the identified domains and measure their
degree of spatial auto-correlation with Moran’s I or Geary’s C.

####  5.1.1 Metrics for assessing morphological translation

In our framework, the ideal metric would measure how well the morphological
features describe the gene patterns but, ideally, also include some
information about their biological relevance. Authors have typically
circumvented this by either selecting meaningful genes for prediction or
analyzing the predictions ad hoc. However, there is currently no way of
assessing this in a quantitative manner, so many studies achieving high
performance metrics might be in the overestimation quadrant of the framework,
having good results on non-informative genes.

Pearson’s correlation coefficient

The Pearson’s correlation coefficient (PCC), also represented by $\rho$, is a
measure of the linear relationships between the observed and the predicted
gene expression. Mathematically, it is computed as the covariance (cov) of the
two variables divided by the product of their standard deviations ($\sigma$).
The covariance is a measure of the tendency between the observed and predicted
expression i.e. if higher true expression levels match higher predicted
levels, the covariance is higher. The division by their $\sigma$ is a way of
normalizing so that the metric is bound between $+1$ perfect positive and $-1$
perfect negative correlation.

Let $y$ be the observed true gene expression for a gene, $\hat{y}$ be the
predicted gene expression for that gene, and $\sigma_{y}$ and
$\sigma_{\hat{y}}$ their standard deviations, then the PCC can be expressed
by:

$\text{PCC}=\frac{\text{cov}(y,\hat{y})}{\sigma_{y}\sigma_{\hat{y}}}$ (7)

Regression metrics

The problem of gene expression prediction is typically framed as a regression
problem. More recent methods also report the performance of different
regression metrics, namely mean absolute error (MAE), mean square error (MSE)
and its root MSE (RMSE). Let $y_{i}$ be the observed true gene expression for
a gene at position $i$ and $\hat{y}_{i}$ be the predicted gene expression for
that gene at the same position. In other words, a small error means that we
have learned features that predict gene expression well.

The MAE measures the average magnitude of the errors in a set of predictions,
without considering their direction. It is the average over the test sample of
the absolute differences between prediction and actual observation where all
individual differences have equal weight.

$\text{MAE}=\frac{1}{n}\sum_{i=1}^{n}|y_{i}-\hat{y}_{i}|$ (8)

The MSE measures the average of the squares of the errors—that is, the average
squared difference between the estimated values and the actual value. MSE is a
risk function corresponding to the expected value of the squared error loss.

$\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}$ (9)

The RMSE is the square root of the average of squared differences between
prediction and actual observation. It provides a measure of how spread out
these residuals are. In other words, it tells you how concentrated the data is
around the line of best fit.

$\text{RMSE}=\sqrt{MSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}$
(10)

Each of these metrics offers a different perspective on the accuracy of the
model predictions. MAE gives an idea of the magnitude of errors, MSE
emphasizes larger errors more than smaller ones due to squaring the
differences, and RMSE provides an overall measure of fit in the same units as
the response variable. However, the values are not bounded, which can be
harder to interpret and compare.

####  5.1.2 Metrics for assessing morphological integration

In our framework, the ideal metric for identifying strong features for
morphological integration would measure how the morphological features,
despite not being correlated with gene expression patterns, are still relevant
and thus add additional information to the spatial domain identification. On
the one side, authors typically measure how well their clustered domains agree
with expert annotations. On the other, the defined domains can be linked to a
specific spatially variable gene (SVG). This gene can be validated by its
amount of auto-correlation.

Clustering evaluation metrics

Since spatial domain identification is not a classification problem in which
there is a one-to-one connection between regions, we need metrics that take
this into account. The Rand index (RI) and its adjusted version (ARI) [53, 31,
70] measure the similarity of two clusters, accommodating for scenarios with
unlabeled and different amounts of categories. To evaluate the similarity
between two clustering results, we use metrics that account for such
complexities.

The ARI is a measure of the similarity between two data clusterings,
correcting for the chance grouping of elements. The ARI ranges from -1 to 1,
where 1 indicates perfect agreement between the two clusterings, 0 indicates
random clustering, and negative values indicate less agreement than expected
by chance.

$\text{ARI}=\frac{\text{RI}-\text{Expected RI}}{\max(\text{RI})-\text{Expected
RI}}$ (11)

where RI is the measure of the similarity between two data clusterings, and
Expected RI is the expected value of the RI for random clustering.

Auto-correlation metrics

Gene expressions at different locations may exhibit spatial auto-correlation,
where nearby locations have similar expression levels. To assess this, authors
use Moran’s I [46, 38] and Geary’s C [25, 3] statistics.

Moran’s I quantifies overall spatial auto-correlation of gene expression. It
ranges from -1 to 1, where 1 indicates a clear spatial pattern, 0 indicates
random spatial expression, and -1 indicates a chessboard-like pattern.

$\text{Moran's
I}=\frac{N}{W}\frac{\sum_{i}\sum_{j}w_{ij}(x_{i}-\bar{x})(x_{j}-\bar{x})}{\sum_{i}(x_{i}-\bar{x})^{2}}$
(12)

where $x_{i}$ and $x_{j}$ are the gene expressions at spots $i$ and $j$,
$\bar{x}$ is the mean expression, $N$ is the total number of spots, $w_{ij}$
is the spatial weight between spots $i$ and $j$, and $W$ is the sum of
$w_{ij}$. We set $w_{ij}=1$ for the 4 nearest neighbors of spot $i$ and
$w_{ij}=0$ otherwise.

Geary’s C also measures spatial autocorrelation, but focuses on local
differences. Its value ranges from 0 to 2. To align it with Moran’s I, we
scale it to [-1, 1]:

$\text{Geary's
C}=\frac{(N-1)}{2W}\frac{\sum_{i}\sum_{j}w_{ij}(x_{i}-x_{j})^{2}}{\sum_{i}(x_{i}-\bar{x})^{2}}$
(13)

The scaled version is:

$\text{Scaled Geary's C}=1-\text{Geary's C}$ (14)

Here, 1 indicates perfect positive autocorrelation, 0 indicates no
autocorrelation, and -1 indicates perfect negative autocorrelation. Both
metrics provide insight into the spatial patterns of gene expression.

###  5.2 Public datasets and benchmarks

As with the metrics, the datasets that the community has used for developing
new methods draw inspiration from the first works. The scarce availability of
big spatial transcriptomics datasets further enforced this.

Many translation methods methods use breast cancer datasets, either the two
ductal carcinoma samples from 10X Visium and the 23 breast cancer patients
used by ST-Net [26], or the 32 HER2-positive breast cancer samples presented
by Andersson et al. [2]. Another commonly used dataset has been the human
squamous cell carcinoma dataset [34]. This is also the case for the benchmark
presently available for translation studies [8]. Even though the authors also
include methods that do not train on spatial data, they analyze the
performance of ST-Net, DeepSpaCE and HisToGene on the HER2-positive and
squamous cell carcinoma datasets.

Super-resolution tasks are typically harder to evaluate quantitatively, as
there is no ground-truth for per-pixel gene expression. New technologies such
as 10X Xenium were utilized as a surrogate for this by iStar and MHAST. This
enabled the authors to have a denser gene expression map that could be then
compared to their methods, even though 10X Xenium is a imaging-based method
and thus requires predefining a gene panel of 100s of genes, far from the
whole transcriptome.

The two main datasets used for integration methods are 10X Visium mouse brain
samples and the 10X Visium human dorsolateral prefrontal cortex available at
the spatialLIBD library [48]. The latter was used in the benchmark by Yuan et
al. [78]. As the integration tasks is commonly spatial domain identification,
it requires some type of annotations for comparison, limiting the amount of
available datasets.

##  6 Technical considerations

Working with histological images comes with its own set of challenges that are
far from solved [67, 4, 68]. One of the primary issues is the inherent
variability in histological image data, which arises from differences in
staining protocols, sample preparation techniques, and imaging conditions
across different laboratories and studies. This variability complicates the
generalization of models, as algorithms trained on one dataset may not perform
well on another due to these inconsistencies. Pretraining models on a wide
variety of data from different sources can help improve their robustness and
generalization capabilities [76].

To this we add the challenges from working with such high dimensional data as
spatial transcriptomics. Each modality individually produces complex, high-
dimensional data, and integrating these datasets compounds the complexity.
Dimensionality reduction techniques are usually employed to reduce the
complexity of the data while trying to retain the essential information [57,
62].

These challenges make it hard to benchmark and compare methods. Many authors
compare against previous work consistently outperforming them, but this is
done in new datasets and with different validations schemes. The low
availability of big datasets limits the generalization of current benchmarks,
at times yielding contradictory results with the original authors [8, 78].
Bigger annotated datasets are necessary for method development and comparison.
Transparent reporting of training and validation procedures is also crucial to
ensure reproducibility and allow for meaningful comparisons across studies.

An important consideration for our study is the reassessment of the metrics
currently utilized, with the possibility of incorporating an additional
dimension. Traditional translation tasks often employ PCC or regression
metrics, which primarily evaluate the shared information between learned
morphological features and gene expression patterns, analogous to the x-axis
in our proposed framework. This approach can lead to an overestimation of
predictive performance for genes that are not task-relevant, such as
housekeeping or constitutive genes, neglecting genes that may hold significant
clinical value. Similarly, integration methods typically assess the shared
information between the fusion of morphological features and gene expression
with expert annotations, comparable to the y-axis in our framework. This
methodology can entangle the individual contributions of each modality and
potentially diminish performance by incorporating noisy or correlated
morphological features.

Finally, an often overlooked aspect is the extensive training and
computational resources required in these applications. In predicting gene
expression, especially with internal validation as opposed to an independent
test set, additional training can substantially enhance results. This concern
also extends to integration tasks, where it becomes difficult to determine
whether improved performance is attributable to the actual contribution of
morphological data or merely the result of overfitting to the specific
problem.

##  7 Outlook and perspectives

The combination of morphology and spatial transcriptomics is at its dawn. In
this work we presented a mental framework to conceptualize and guide future
developments in this field, as many challenges remain unresolved.

Correlation values for gene expression prediction are rather moderate. Even
with modern implementations, performances are far from clinically
transferable. If we want to be able to apply such methods to predict the gene
expression in large H&E-stained cohorts, we have to make sure we are falling
in the first quadrant of Figure 1(b). Achieving this will require not just an
improvement in average model performance, but a focus on ensuring that methods
are truly effective for clinically important genes. This might involve
developing specialized models tailored to specific clinical problems, ensuring
their applicability in real-world scenarios.

The task of spatial domain identification has not yet shown significant
benefits from morphological integration. This is likely due to the fact that
the morphological features currently used are either not relevant enough or
provide redundant information already captured by gene expression data. Future
method development should prioritize the creation of morphological descriptors
that are both highly relevant and complementary to gene expression, ensuring
that they add value rather than noise to the analysis. Until such improvements
are made, the integration of morphology should be approached with caution,
avoiding the inclusion of irrelevant or redundant features.

The framework proposed in this review is not limited to the integration of
morphology and spatial transcriptomics; it can be extended to other multi-
omics or multi-modal data integration tasks. In cases of translation, the goal
is to maximize the shared information between modalities while focusing on the
specific downstream task. For integration, the aim is to minimize redundant
information while ensuring that both modalities contribute meaningfully to the
final analysis. This flexible framework can guide future research in multi-
modal data integration, ensuring a more systematic and effective approach.

Ultimately, we hope this framework will serve as a compass for researchers,
facilitating the joint use of morphology and spatial transcriptomics as
imaging AI and bioinformatics continue to advance. By providing a structured
approach to integrating these complex data types, this framework aims to guide
efforts to gain deeper insights into biological processes and disease
mechanisms.

## Acknowledgments

We would like to thank current and former collaborators and co-workers in the
field of spatial omics and imaging AI for sharing their insights leading up to
this review.

## References

  * [1] Alexander A Alemi, Ian Fischer, Joshua V Dillon and Kevin Murphy  “Deep variational information bottleneck”  In _arXiv preprint arXiv:1612.00410_ , 2016
  * [2] Axel Andersson, Gabriele Partel, Leslie Solorzano and Carolina Wählby  “Transcriptome-supervised classification of tissue morphology using deep learning”  In _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_ , 2020, pp. 1630–1633  IEEE
  * [3] Luc Anselin  “A local indicator of multivariate spatial association: extending Geary’s C”  In _Geographical Analysis_ 51.2 Wiley Online Library, 2019, pp. 133–150
  * [4] Kaustav Bera et al.  “Artificial intelligence in digital pathology—new tools for diagnosis and precision oncology”  In _Nature reviews Clinical oncology_ 16.11 Nature Publishing Group UK London, 2019, pp. 703–715
  * [5] Ludvig Bergenstråhle et al.  “Super-resolved spatial transcriptomics by deep data fusion”  In _Nature biotechnology_ 40.4 Nature Publishing Group US New York, 2022, pp. 476–479
  * [6] Tommaso Biancalani et al.  “Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram”  In _Nature methods_ 18.11 Nature Publishing Group US New York, 2021, pp. 1352–1362
  * [7] Dario Bressan, Giorgia Battistoni and Gregory J Hannon  “The dawn of spatial omics”  In _Science_ 381.6657 American Association for the Advancement of Science, 2023, pp. eabq4964
  * [8] Adam S Chan et al.  “Benchmarking the translational potential of spatial gene expression prediction from histology”  In _bioRxiv_ Cold Spring Harbor Laboratory, 2023, pp. 2023–12
  * [9] Eduard Chelebian, Christophe Avenel and Carolina Wahlby  “Self-supervised learning for genetically relevant domain identificaiton in morphological images”  In _2024 IEEE 21st International Symposium on Biomedical Imaging (ISBI)_ , 2024  IEEE
  * [10] Eduard Chelebian et al.  “Learned morphological features guide cell type assignment of deconvolved spatial transcriptomics”  In _Medical Imaging with Deep Learning_ , 2024
  * [11] Eduard Chelebian et al.  “Morphological features extracted by AI associated with spatial transcriptomics in prostate cancer”  In _Cancers_ 13.19 MDPI, 2021, pp. 4837
  * [12] Ao Chen et al.  “Spatiotemporal transcriptomic atlas of mouse organogenesis using DNA nanoball-patterned arrays”  In _Cell_ 185.10 Elsevier, 2022, pp. 1777–1792
  * [13] Cui Chen et al.  “Edge-relational window-attentional graph neural network for gene expression prediction in spatial transcriptomics analysis”  In _Computers in Biology and Medicine_ Elsevier, 2024, pp. 108449
  * [14] Kok Hao Chen et al.  “Spatially resolved, highly multiplexed RNA profiling in single cells”  In _Science_ 348.6233 American Association for the Advancement of Science, 2015, pp. aaa6090
  * [15] Richard J Chen et al.  “Scaling vision transformers to gigapixel images via hierarchical self-supervised learning”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16144–16155
  * [16] Youngmin Chung, Ji Hun Ha, Kyeong Chan Im and Joo Sang Lee  “Accurate Spatial Gene Expression Prediction by integrating Multi-resolution features”  In _arXiv preprint arXiv:2403.07592_ , 2024
  * [17] Ozan Ciga, Tony Xu and Anne Louise Martel  “Self supervised contrastive learning for digital histopathology”  In _Machine Learning with Applications_ 7 Elsevier, 2022, pp. 100198
  * [18] Charles Comiter et al.  “Inference of single cell profiles from histology stains with the Single-Cell omics from Histology Analysis Framework (SCHAF)”  In _BioRxiv_ Cold Spring Harbor Laboratory, 2023, pp. 2023–03
  * [19] Miao Cui and David Y Zhang  “Artificial intelligence and computational pathology”  In _Laboratory Investigation_ 101.4 Elsevier, 2021, pp. 412–422
  * [20] Jia Deng et al.  “Imagenet: A large-scale hierarchical image database”  In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255  Ieee
  * [21] Alexey Dosovitskiy et al.  “An image is worth 16x16 words: Transformers for image recognition at scale”  In _arXiv preprint arXiv:2010.11929_ , 2020
  * [22] Chee-Huat Linus Eng et al.  “Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH+”  In _Nature_ 568.7751 Nature Publishing Group UK London, 2019, pp. 235–239
  * [23] Andrew Erickson et al.  “Spatially resolved clonal copy number alterations in benign and malignant tissue”  In _Nature_ 608.7922 Nature Publishing Group UK London, 2022, pp. 360–367
  * [24] Ruitian Gao et al.  “Harnessing TME depicted by histological images to improve cancer prognosis through a deep learning system”  In _Cell Reports Medicine_ Elsevier, 2024
  * [25] Robert C Geary  “The contiguity ratio and statistical mapping”  In _The incorporated statistician_ 5.3 JSTOR, 1954, pp. 115–146
  * [26] Bryan He et al.  “Integrating spatial gene expression and breast tumour morphology via deep learning”  In _Nature biomedical engineering_ 4.8 Nature Publishing Group UK London, 2020, pp. 827–834
  * [27] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [28] Kaiming He et al.  “Masked autoencoders are scalable vision learners”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 16000–16009
  * [29] Jian Hu et al.  “SpaGCN: Integrating gene expression, spatial location and histology to identify spatial domains and spatially variable genes by graph convolutional network”  In _Nature methods_ 18.11 Nature Publishing Group US New York, 2021, pp. 1342–1351
  * [30] Gao Huang, Zhuang Liu, Laurens Van Der Maaten and Kilian Q Weinberger  “Densely connected convolutional networks”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708
  * [31] Lawrence Hubert and Phipps Arabie  “Comparing partitions”  In _Journal of classification_ 2 Springer, 1985, pp. 193–218
  * [32] Amanda Janesick et al.  “High resolution mapping of the tumor microenvironment using integrated single-cell, spatial and in situ analysis”  In _Nature Communications_ 14.1 Nature Publishing Group UK London, 2023, pp. 8353
  * [33] Andrew Janowczyk and Anant Madabhushi  “Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases”  In _Journal of pathology informatics_ 7.1 Elsevier, 2016, pp. 29
  * [34] Andrew L Ji et al.  “Multimodal analysis of composition and spatial architecture in human squamous cell carcinoma”  In _Cell_ 182.2 Elsevier, 2020, pp. 497–514
  * [35] Yuran Jia et al.  “THItoGene: a deep learning method for predicting spatial transcriptomics from histological images”  In _Briefings in Bioinformatics_ 25.1 Oxford University Press, 2024, pp. bbad464
  * [36] Rongqin Ke et al.  “In situ sequencing for RNA analysis in preserved tissue and cells”  In _Nature methods_ 10.9 Nature Publishing Group US New York, 2013, pp. 857–860
  * [37] Chatarina Larsson, Ida Grundberg, Ola Söderberg and Mats Nilsson  “In situ detection and genotyping of individual mRNA molecules”  In _Nature methods_ 7.5 Nature Publishing Group US New York, 2010, pp. 395–397
  * [38] Hongfei Li, Catherine A Calder and Noel Cressie  “Beyond Moran’s I: testing for spatial dependence based on the spatial autoregressive model”  In _Geographical analysis_ 39.4 Wiley Online Library, 2007, pp. 357–375
  * [39] Yijun Li, Stefan Stanojevic and Lana X Garmire  “Emerging artificial intelligence applications in Spatial Transcriptomics analysis”  In _Computational and Structural Biotechnology Journal_ 20 Elsevier, 2022, pp. 2895–2908
  * [40] Sophia K Longo, Margaret G Guo, Andrew L Ji and Paul A Khavari  “Integrating single-cell and spatial transcriptomics to elucidate intercellular tissue dynamics”  In _Nature Reviews Genetics_ 22.10 Nature Publishing Group UK London, 2021, pp. 627–644
  * [41] Shaina Lu, Daniel Fürth and Jesse Gillis  “Integrative analysis methods for spatial transcriptomics”  In _Nature Methods_ 18.11 Nature Publishing Group US New York, 2021, pp. 1282–1283
  * [42] Vittorio Mazzia, Francesco Salvetti and Marcello Chiaberge  “Efficient-capsnet: Capsule network with self-attention routing”  In _Scientific reports_ 11.1 Nature Publishing Group UK London, 2021, pp. 14634
  * [43] Gabriel Mejia et al.  “SEPAL: Spatial Gene Expression Prediction from Local Graphs”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 2294–2303
  * [44] Raktim Kumar Mondol et al.  “hist2rna: an efficient deep learning architecture to predict gene expression from breast cancer histopathology images”  In _Cancers_ 15.9 MDPI, 2023, pp. 2569
  * [45] Taku Monjo et al.  “Efficient prediction of a spatial transcriptomics profile better characterizes breast cancer tissue sections without costly experimentation”  In _Scientific reports_ 12.1 Nature Publishing Group UK London, 2022, pp. 4133
  * [46] Patrick AP Moran  “Notes on continuous stochastic phenomena”  In _Biometrika_ 37.1/2 JSTOR, 1950, pp. 17–23
  * [47] Minxing Pang, Kenong Su and Mingyao Li  “Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors”  In _BioRxiv_ Cold Spring Harbor Laboratory, 2021, pp. 2021–11
  * [48] Brenda Pardo et al.  “spatialLIBD: an R/Bioconductor package to visualize spatially-resolved transcriptomics data”  In _BMC genomics_ 23.1 Springer, 2022, pp. 434
  * [49] Jiwoon Park et al.  “Spatial omics technologies at multimodal and single cell/subcellular level”  In _Genome Biology_ 23.1 Springer, 2022, pp. 256
  * [50] Duy Pham et al.  “Robust mapping of spatiotemporal trajectories and cell–cell interactions in healthy and diseased tissues”  In _Nature communications_ 14.1 Nature Publishing Group UK London, 2023, pp. 7739
  * [51] Alec Radford et al.  “Learning transferable visual models from natural language supervision”  In _International conference on machine learning_ , 2021, pp. 8748–8763  PMLR
  * [52] Md Mamunur Rahaman, Ewan KA Millar and Erik Meijering  “Breast cancer histopathology image-based gene expression prediction using spatial transcriptomics data and deep learning”  In _Scientific Reports_ 13.1 Nature Publishing Group UK London, 2023, pp. 13604
  * [53] William M Rand  “Objective criteria for the evaluation of clustering methods”  In _Journal of the American Statistical association_ 66.336 Taylor & Francis, 1971, pp. 846–850
  * [54] Anjali Rao, Dalia Barkley, Gustavo S França and Itai Yanai  “Exploring tissue architecture using spatial transcriptomics”  In _Nature_ 596.7871 Nature Publishing Group UK London, 2021, pp. 211–220
  * [55] Samuel G Rodriques et al.  “Slide-seq: A scalable technology for measuring genome-wide expression at high spatial resolution”  In _Science_ 363.6434 American Association for the Advancement of Science, 2019, pp. 1463–1467
  * [56] Zaira Seferbekova, Artem Lomakin, Lucy R Yates and Moritz Gerstung  “Spatial biology of cancer evolution”  In _Nature Reviews Genetics_ 24.5 Nature Publishing Group UK London, 2023, pp. 295–313
  * [57] Lulu Shang and Xiang Zhou  “Spatially aware dimension reduction for spatial transcriptomics”  In _Nature communications_ 13.1 Nature Publishing Group UK London, 2022, pp. 7203
  * [58] Karen Simonyan and Andrew Zisserman  “Very deep convolutional networks for large-scale image recognition”  In _arXiv preprint arXiv:1409.1556_ , 2014
  * [59] Stefano Soatto and Alessandro Chiuso  “Visual representations: Defining properties and deep approximations”  In _arXiv preprint arXiv:1411.7676_ , 2014
  * [60] Andrew H Song et al.  “Artificial intelligence for digital and computational pathology”  In _Nature Reviews Bioengineering_ 1.12 Nature Publishing Group UK London, 2023, pp. 930–949
  * [61] Peter Ström et al.  “Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study”  In _The Lancet Oncology_ 21.2 Elsevier, 2020, pp. 222–232
  * [62] Yidi Sun et al.  “A comprehensive survey of dimensionality reduction and clustering methods for single-cell and spatial transcriptomics data”  In _Briefings in Functional Genomics_ Oxford University Press, 2024, pp. elae023
  * [63] Mingxing Tan and Quoc Le  “Efficientnet: Rethinking model scaling for convolutional neural networks”  In _International conference on machine learning_ , 2019, pp. 6105–6114  PMLR
  * [64] Xiao Tan, Andrew Su, Minh Tran and Quan Nguyen  “SpaCell: integrating tissue morphology and spatial gene expression to predict disease cells”  In _Bioinformatics_ 36.7 Oxford University Press, 2020, pp. 2293–2294
  * [65] Yonglong Tian et al.  “What makes for good views for contrastive learning?”  In _Advances in neural information processing systems_ 33, 2020, pp. 6827–6839
  * [66] Naftali Tishby, Fernando C Pereira and William Bialek  “The information bottleneck method”  In _arXiv preprint physics/0004057_ , 2000
  * [67] Hamid Reza Tizhoosh and Liron Pantanowitz  “Artificial intelligence and digital pathology: challenges and opportunities”  In _Journal of pathology informatics_ 9.1 Elsevier, 2018, pp. 38
  * [68] Jeroen Van der Laak, Geert Litjens and Francesco Ciompi  “Deep learning in histopathology: the path to the clinic”  In _Nature medicine_ 27.5 Nature Publishing Group US New York, 2021, pp. 775–784
  * [69] Katy Vandereyken, Alejandro Sifrim, Bernard Thienpont and Thierry Voet  “Methods and applications for single-cell and spatial multi-omics”  In _Nature Reviews Genetics_ 24.8 Nature Publishing Group UK London, 2023, pp. 494–515
  * [70] Nguyen Xuan Vinh, Julien Epps and James Bailey  “Information theoretic measures for clusterings comparison: is a correction for chance necessary?”  In _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 1073–1080
  * [71] Hongyi Wang et al.  “M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images”  In _arXiv preprint arXiv:2401.10608_ , 2024
  * [72] Yinxi Wang et al.  “Predicting molecular phenotypes from histopathology images: A transcriptome-wide expression–morphology analysis in breast cancer”  In _Cancer research_ 81.19 AACR, 2021, pp. 5115–5126
  * [73] Philippe Weitz et al.  “Transcriptome-wide prediction of prostate cancer gene expression from histopathology images using co-expression-based convolutional neural networks”  In _Bioinformatics_ 38.13 Oxford University Press, 2022, pp. 3462–3469
  * [74] Zonghan Wu et al.  “A comprehensive survey on graph neural networks”  In _IEEE transactions on neural networks and learning systems_ 32.1 IEEE, 2020, pp. 4–24
  * [75] Ronald Xie et al.  “Spatially Resolved Gene Expression Prediction from Histology Images via Bi-modal Contrastive Learning”  In _Advances in Neural Information Processing Systems_ 36, 2024
  * [76] Hanwen Xu et al.  “A whole-slide foundation model for digital pathology from real-world data”  In _Nature_ Nature Publishing Group UK London, 2024, pp. 1–8
  * [77] Yan Yang, Md Zakir Hossain, Eric Stone and Shafin Rahman  “Spatial transcriptomics analysis of gene expression prediction using exemplar guided graph neural network”  In _Pattern Recognition_ 145 Elsevier, 2024, pp. 109966
  * [78] Zhiyuan Yuan et al.  “Benchmarking spatial clustering methods with spatially resolved transcriptomics data”  In _Nature Methods_ Nature Publishing Group US New York, 2024, pp. 1–11
  * [79] Roxana Zahedi et al.  “Deep learning in spatially resolved transcriptfomics: a comprehensive technical view”  In _Briefings in Bioinformatics_ 25.2 Oxford University Press, 2024, pp. bbae082
  * [80] Yuansong Zeng et al.  “Identifying spatial domain by adapting transcriptomics with histology through contrastive learning”  In _Briefings in Bioinformatics_ 24.2 Oxford University Press, 2023, pp. bbad048
  * [81] Daiwei Zhang et al.  “Inferring super-resolution tissue architecture by integrating spatial transcriptomics with histology”  In _Nature Biotechnology_ Nature Publishing Group US New York, 2024, pp. 1–6
  * [82] Tongtong Zhao et al.  “Spatial genomics enables multi-modal study of clonal heterogeneity in tissues”  In _Nature_ 601.7891 Nature Publishing Group UK London, 2022, pp. 85–91
  * [83] Ran Zhou, Gaoxia Yang, Yan Zhang and Yuan Wang  “Spatial transcriptomics in development and disease”  In _Molecular Biomedicine_ 4.1 Springer, 2023, pp. 32
  * [84] Yongshuo Zong, Oisin Mac Aodha and Timothy Hospedales  “Self-supervised multimodal learning: A survey”  In _arXiv preprint arXiv:2304.01008_ , 2023
  * [85] Yongshuo Zong et al.  “conST: an interpretable multi-modal contrastive learning framework for spatial transcriptomics”  In _BioRxiv_ Cold Spring Harbor Laboratory, 2022, pp. 2022–01