11institutetext: Department of Clinical Neurosciences, University of
Cambridge, UK 22institutetext: Department of Applied Mathematics and
Theoretical Physics, University of Cambridge, UK 33institutetext: School of
Science and Engineering, University of Dundee, UK 44institutetext: School of
Medicine, University of Dundee, UK 55institutetext: Zhejiang Lab, China

# Cross-modal Diffusion Modelling for Super-resolved Spatial Transcriptomics

Xiaofei Wang 11 Xingxu Huang 55 Stephen Price 11 Chao Li 11223344

###### Abstract

The recent advancement of spatial transcriptomics (ST) allows to characterize
spatial gene expression within tissue for discovery research. However, current
ST platforms suffer from low resolution, hindering in-depth understanding of
spatial gene expression. Super-resolution approaches promise to enhance ST
maps by integrating histology images with gene expressions of profiled tissue
spots. However, current super-resolution methods are limited by restoration
uncertainty and mode collapse. Although diffusion models have shown promise in
capturing complex interactions between multi-modal conditions, it remains a
challenge to integrate histology images and gene expression for super-resolved
ST maps. This paper proposes a cross-modal conditional diffusion model for
super-resolving ST maps with the guidance of histology images. Specifically,
we design a multi-modal disentangling network with cross-modal adaptive
modulation to utilize complementary information from histology images and
spatial gene expression. Moreover, we propose a dynamic cross-attention
modelling strategy to extract hierarchical cell-to-tissue information from
histology images. Lastly, we propose a co-expression-based gene-correlation
graph network to model the co-expression relationship of multiple genes.
Experiments show that our method outperforms other state-of-the-art methods in
ST super-resolution on three public datasets.

###### Keywords:

Spatial Transcriptomics Digital Pathology Diffusion Models Cross-Modal Super-
Resolution.

##  1 Introduction

Spatial transcriptomics (ST), the spatial distribution of gene expressions,
enables genomic profiling while preserving structural information within
original tissue, promising to understand complex conditions with spatial
heterogeneity. However, popular ST platforms, e.g., Visium [20] and Stereo-seq
[4], only measure gene expression in tissue spots, and the very low spatial
resolution limits their ability to study in-depth gene expression. Novel
biotechnology is developed for targeted in-situ sequencing, e.g., IISS [21],
to generate higher resolution ST maps. However, these methods are expensive
and time-consuming. Also, they are limited by the technical bottleneck of low
capture rates and throughput.

Computational approaches promise to enhance the spatial resolution of ST maps
and accelerate scientific discovery [25, 26, 23]. For instance, Zhao et al.
[25] proposed a Bayesian statistical method using spot neighborhood
information for enhancing the resolution of ST maps. Despite encouraging
results, this method is purely based on genomics without leveraging the
histology information at the tissue and cellular levels available from ST
maps. Therefore, the fine-grained ST is challenged by the biological
confidence of imputed resolution.

Histology features observed from tissue sections are enriched with phenotypic
structure and morphology information. Previous studies show that image-level
histology features are associated with tissue gene expression[19, 1, 2, 22].
This intrinsic link establishes the feasibility of super-resolving ST maps
using histology features extracted from tissue images as additional guidance
combined with profiled tissue spots. A few studies [17, 3, 10] have attempted
this direction, e.g., Pang et al. [25] designed a two-stage transformer-based
framework where low-resolution (LR) ST is firstly predicted from histology
images and then used to infer high-resolution (HR) ST. However, due to the
two-stage design, this model fails to establish the link between the multi-
modalities of histology images and gene expression maps. Further,
Bergenstraahle et al. [3] proposed a multi-scale latent generative model to
enhance ST resolution by integrating histology images with gene expression.
Nevertheless, the unstable optimization process of the proposed generative
model may present significant challenges for enhancing ST maps.

Conditional diffusion models are a class of deep generative models that have
achieved state-of-the-art performance in natural and medical images [15, 11,
6]. The model incorporates a Markov chain-based diffusion process along with
conditional variables, i.e., LR images, to restore HR images. The conditioning
mechanisms in the diffusion model enable the possibility of incorporating
multi-modal conditional data, promising better super-resolved outputs.
Additionally, the objective function of diffusion models is a variant of the
variational lower bound that yields stable optimization processes. Given these
advantages, conditional diffusion models promise to effectively enhance the
resolution of ST maps by integrating genomics and histology images.

However, several challenges remain in developing diffusion models to integrate
multi-modal information in super-resolving ST maps: 1) Traditional methods
regard multi-modal data as multiple diffusion conditions, which are integrated
via simple concatenation or disentanglement [13]. These methods treat each
modality equally and ignore the modulation across modalities, which may not
effectively leverage complementary information in histology images and gene
expression; 2) Diffusion models are primarily intended for the generation of a
single image, i.e., treating the ST map of each gene separately. Intrinsic
expression associations across multiple genes can result in inconsistent
diffusion processes, posing challenges to the efficient learning of relevant
features; 3) The complex structure and morphology information at cellular and
tissue levels pose challenges to integrating multi-modal data in the
conditioning process of the diffusion model.

To address the challenges, we propose a novel cross-modal conditional
diffusion model (Diff-ST) for ST super-resolution (SR). To the best of our
knowledge, this is the first diffusion-based multi-modal SR method for ST
maps. The main contribution of our work is threefold:

  * •

We propose a new backbone, i.e., a multi-modal disentangling network with
cross-modal adaptive modulation, for the conditional diffusion model. The
novel disentangled representation learning enables the modulation of
complementary information from tissue images and ST maps.

  * •

We propose a co-expression intensity-based gene-correlation graph (CIGC-Graph)
network to model the co-expression relationship of multiple genes, enabling
jointly reconstructing SR images of multiple genes.

  * •

We propose a cross-attention modelling strategy based on curriculum learning
mechanisms, which enables extracting hierarchical cell-to-tissue level
information from tissue images.

Our extensive experiments on the three public datasets demonstrate that our
method outperforms other state-of-the-art (SOTA) methods.

##  2 Methodology

Figure 1: Conceptual workflow of Diff-ST. The forward diffusion process $q$
perturbs HR ST $\mathbf{x}$ by gradually adding Gaussian noise. The backward
diffusion process $p$ denoises the perturbed ST, conditioning on its paired LR
version $\mathbf{y}$ and histology image $\mathbf{h}$.

###  2.1 Framework

The proposed Diff-ST achieves multi-modal ST SR through forward and backward
diffusion processes, illustrated in Fig. 1, inspired by [6]. Given the multi-
channel111Channel number denotes the number of genes that are predicted
together. HR ST images $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$, the forward
process gradually adds Gaussian noise to $\mathbf{x}_{0}$ over $T$ diffusion
steps according to a noise variance schedule $\beta_{1},\ldots,\beta_{T}$.
Specifically, each step of the forward diffusion process produces noisier
images $\mathbf{x}_{t}$ with distribution
$q(\mathbf{x}_{t}\mid\mathbf{x}_{t-1})$, formulated as:

$\displaystyle
q\left(\mathbf{x}_{1:T}\mid\mathbf{x}_{0}\right)=\prod_{t=1}^{T}q\left(\bm{x}_{t}\mid\bm{x}_{t-1}\right),q\left(\mathbf{x}_{t}\mid\mathbf{x}_{t-1}\right)=\mathcal{N}\left(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}\right)$
(1)

For sufficiently large $T$, the perturbed HR $\mathbf{x}_{T}$ can be
considered as a close approximation of isotropic Gaussian distribution. On the
other hand, the reverse diffusion process $p$ aims to generate new HR ST maps
from $\mathbf{x}_{T}$. This is achieved by constructing the reverse
distribution
$p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{y},\mathbf{h}\right)$,
conditioned on its paired LR ST maps $\mathbf{y}$ and histology image
$\mathbf{h}$ as follows:

$\displaystyle
p_{\theta}\left(\mathbf{x}_{0:T}\right)=p_{\theta}\left(\mathbf{x}_{T}\right)\prod_{t=1}^{T}p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}\right)$
$\displaystyle
p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{y},\mathbf{h}\right)=\mathcal{N}\left(\mathbf{x}_{t-1};\bm{\mu}_{\theta}\left(\mathbf{x}_{t},\mathbf{y},\mathbf{h},t\right),\sigma^{2}_{t}\mathbf{I}\right)$
(2)

where $p_{\theta}$ denotes a parameterized model, $\theta$ is its trainable
parameters, while $\bm{\mu}_{\theta}$ and $\sigma^{2}_{t}$ can be learned.
Specifically, the detailed conditioning mechanism for reverse diffusion is
introduced as follows.

###  2.2 Conditioning mechanisms for reverse diffusion

1) Overall process: Taking the multi-modal histology image and LR ST maps as
input conditions, our proposed Diff-ST learns to reconstruct the HR ST maps in
the reverse diffusion process. Fig. 2(a) shows the detailed conditioning
process, in which the reverse distribution is estimated by learning multi-
modal representations. Specifically, we first extract hierarchical cell-to-
tissue level features from histology images via cross-attention modelling.
Then, the extracted features are fused with LR ST features via the cross-modal
adaptive modulation and multi-modal disentangling strategy. Finally, the fused
multi-modal features are fed into the reverse diffusion process as conditions
for reconstructing HR ST.

2) Hierarchical cell-to-tissue feature extraction: To obtain the cell-level
information, we first crop the input histology image $\mathbf{h}$ into
$M$222Here, $M$ is set to 256, with each patch of 320 pixels, reflecting
160$\mu$m regions in real tissue, consistent with the average cellular
organization scale in histology [5]. patches
$\\{\mathbf{h}_{\mathrm{cell}}\\}_{1}^{M}$ at the cellular scale. Due to the
varied cellular complexity across patches, we propose a dynamic patch
selection strategy based on curriculum learning. Specifically, given encoded
features $\\{\mathbf{F}_{\mathrm{cell}}\\}_{1}^{M}$ of patches, we select and
retain the features as
$\mathbf{F}_{\mathrm{cell}}^{\mathrm{s}}=\\{\mathbf{F}_{\mathrm{cell}}^{m}\big{|}\mathcal{E}(\mathbf{h}_{\mathrm{cell}}^{m})>\gamma\\}_{m=1}^{M}$,
where $\mathcal{E}(\cdot)$ is the entropy-based image complexity estimation
function. As training progresses, $\gamma$ gradually increases, indicating
increased complexity of sampled patches.

To generate the final histology features, the screened cell-level features
$\mathbf{F}_{\mathrm{cell}}^{\mathrm{s}}$ are further integrated with the
tissue-level features $\mathbf{F}_{\mathrm{tissue}}$ via the cross-attention
modelling
$\text{Atten}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\cdot
V$, with

$Q=f(\mathbf{F}_{\mathrm{tissue}})\cdot
W_{Q},\;K=f(\mathbf{F}_{\mathrm{cell}}^{\mathrm{s}})\cdot
W_{K},\;V=f(\mathbf{F}_{\mathrm{cell}}^{\mathrm{s}})\cdot W_{V}.$

Here, $f(\cdot)$ denotes the pixel flattening function, and
$W_{V}\in\mathbb{R}^{d_{c}\times d_{t}}$, $W_{Q}\in\mathbb{R}^{d_{t}\times d}$
& $W_{K}\in\mathbb{R}^{d_{c}\times d}$ are learnable projection matrices,
where $d_{t}$ and $d_{c}$ denote the features channels of
$\mathbf{F}_{\mathrm{tissue}}$ and $\mathbf{F}_{\mathrm{cell}}^{\mathrm{s}}$,
respectively.

3) Cross-modal adaptive modulation & multi-modal disentangling: Given the
extracted multi-modal features of histology image
$\mathbf{F}_{\mathbf{h}}^{\mathrm{in}}$ and LR ST maps
$\mathbf{F}_{\mathbf{y}}^{\mathrm{in}}$, we propose a cross-modal adaptive
modulation strategy to share the modal-specific information with each other,
i.e., cellular and tissue information from histology to ST, and gene
expression patterns from ST to histology. Specifically, in the histology-to-ST
modulation (Fig 2(b)), to each pixel in
$\mathbf{F}_{\mathbf{y}}^{\mathrm{in}}$ (denoted as $\textbf{y}(w,h)$), we
learn a modulation filter $f^{h2y}_{(w,h)}(\cdot)$ constricted to its adjacent
pixels in an $a\times a$ region (denoted as $\textbf{A}_{\textbf{y}(w,h)}$)
and target its counterpart region in $\mathbf{F}_{\mathbf{h}}^{\mathrm{in}}$
(denoted as $\textbf{A}_{\textbf{h}(w,h)}$). As such, the filter can fully
exploit the local dependency between histology images and ST maps. The
filtering operation is expressed as

$\textbf{A}^{\prime}_{\textbf{y}(w,h)}=\textbf{A}_{\textbf{y}_{(w,h)}}\textbf{w}^{h2y}_{(w,h)},\text{where}\;\textbf{w}^{h2y}_{(w,h)}=FC((\textbf{A}_{\textbf{y}_{(w,h)}})^{T}\textbf{A}_{\textbf{h}(w,h)})$
(3)

where $FC$ is fully connected layer and the resulting
$\textbf{A}^{\prime}_{\textbf{y}(w,h)}$ is the update of
$\textbf{A}_{\textbf{y}_{(w,h)}}$ and is induced to spatially refer to
$\textbf{A}_{\textbf{h}(w,h)}$, which is in HR domain.

To further capture the unique and shared representation of multi-modal data, a
disentangling network is designed after the cross-modal adaptive modulation.
With the modal-unique ($U_{\mathbf{h}}$ for histology and $U_{\mathbf{y}}$ for
ST) and modal-sharing features ($S_{\mathbf{h}}$ for histology and
$S_{\mathbf{y}}$ for ST), a cross-modal disentangling loss
$\mathcal{L}_{\text{cm-
dis}}=\|S_{\mathbf{y}}-S_{\mathbf{h}}\|_{2}/\|U_{\mathbf{y}}-U_{\mathbf{h}}\|_{2}$
is designed for minimizing the disparity among shared representations while
maximizing that among unique representations.

Figure 2: Left: Illustration of the multi-modal conditioned reverse diffusion
process of Diff-ST. Right: Pipeline of cross-modal (histology-to-ST) adaptive
modulation strategy. CL is curriculum learning, while FC denotes fully
connected.

###  2.3 Co-expression intensity-based gene-correlation graph

In predicting expressions of multiple genes, existing diffusion models that
separately infer multi-gene ST maps may ignore the intrinsic co-expression
correlation among genes. We propose a co-expression intensity-based gene-
correlation graph (CIGC-Graph) network for effective modelling of co-
expression among genes.

CIGC-Graph (Supplementary Fig. 1) is defined as
$\mathcal{G}=(\mathbf{V},\mathbf{E})$, where $\mathbf{V}$ indicates the nodes,
while $\mathbf{E}$ represents the edges. Given the encoded features of the
noised HR ST maps of $N$ genes
$\mathbf{F}_{\mathbf{x}}=[\mathbf{F}_{\mathbf{x}}^{i}]_{i=1}^{N}\in\mathbb{R}^{N\times
C}$ as input nodes, we construct a co-expression intensity based correlation
matrix $\mathbf{I}\in\mathbb{R}^{N\times N}$ to reflect the relationships
among each node feature, with a weight matrix
$\mathbf{W}_{g}\in\mathbb{R}^{C\times C}$ to update the value of
$\mathbf{F}_{\mathbf{x}}$. Formally, the output nodes
$\mathbf{F}_{\mathbf{x}}^{\mathrm{out}}\in\mathbb{R}^{N\times C}$ are
formulated by a single graph convolutional network layer as

$\mathbf{F}_{\mathbf{x}}^{\mathrm{mid}}=\mathcal{\delta}(\mathbf{I}\mathbf{F}_{\mathbf{x}}\mathbf{W}_{g}),\text{where}\;\mathbf{I}=[I_{i}^{j}]_{i,j=1}^{N},A_{i}^{j}=\frac{1}{2}\big{(}p(\mathbf{F}_{\mathbf{x}}^{i}|\mathbf{F}_{\mathbf{x}}^{j})+p(\mathbf{F}_{\mathbf{x}}^{j}|\mathbf{F}_{\mathbf{x}}^{i})\big{)}.$
(4)

In (4), $\mathcal{\delta}(\cdot)$ is an activation function and
$p(\mathbf{F}_{\mathbf{x}}^{i}|\mathbf{F}_{\mathbf{x}}^{j})$ denotes the
relative expression intensity of $i$-th gene in terms of $j$-th gene. Besides,
residual structure is utilized to generate the final output
$\mathbf{F}_{\mathbf{x}}^{\mathrm{out}}$ of CIGC-Graph network, defined as
$\mathbf{F}_{\mathbf{x}}^{\mathrm{out}}=\alpha\mathbf{F}_{\mathbf{x}}^{\mathrm{mid}}+(1-\alpha)\mathbf{F}_{\mathbf{x}},$
where $\alpha$ is a graph balancing hyper-parameter.

##  3 Experiments & Results

###  3.1 Datasets & Implementation Details

Datasets: We evaluated our model on three public datasets, i.e., Xenium [9],
SGE [8] and Breast-ST [7]. In all datasets, we use LR ST maps and paired HR
histology images to restore $5\times$ and $10\times$ HR ST maps, aligning with
settings in [10, 24]. Totally, we include 502 histology images and 12,550 ST
maps of 25 genes.

In the Xenium dataset, We randomly split the 232 histology images (with 5,800
ST maps) into 99 (with 2,475 ST maps) for training, 49 (with 1,225 ST maps)
for validation, and 84 (with 2,100 ST maps) for testing. For both SR times,
the histology images are of 5,120 $\times$ 5,120 pixels at 0.5 $\mu$m
$\textrm{px}^{-1}$, while the LR ST maps are of 26 $\times$ 26 pixels at 100
$\mu$m $\textrm{px}^{-1}$. Besides, the HR ST maps are of 256 $\times$ 256
pixels at 10 $\mu$m $\textrm{px}^{-1}$ and of 128 $\times$ 128 pixels at 20
$\mu$m $\textrm{px}^{-1}$ for $10\times$ and $5\times$ SR, respectively.
Moreover, the SGE (47 histology and 1,175 ST) and Breast-ST (223 histology and
5,575 ST) are with the same resolution setting as Xenium and are both set as
external validation datasets.

Implementation details: We trained our model for 200 epochs on two NVIDIA RTX
A5000 24 GB GPUs, with batch size 4 and learning rate 0.0001 with AdamW
optimizer [12] together with the weight decay. Following the sampling strategy
in [6], Diff-ST uses the sample steps of 1,000 in both forward and reverse
diffusion processes. Key hyper-parameters are in Table I of supplementary
material. All hyper-parameters are tuned to achieve the best performance over
the validation set. Our method is implemented on PyTorch with the Python
environment.

Table 1: Performance comparisons on three datasets with $5\times$ and
$10\times$ enlargement scales. Bold numbers indicate the best results.

Dataset Xenium SGE Breast-ST Scale $5\times$ $10\times$ $5\times$ $10\times$
$5\times$ $10\times$ Metrics RMSE PCC RMSE PCC RMSE PCC RMSE PCC RMSE PCC RMSE
PCC U-Net 0.376 0.198 0.392 0.208 0.384 0.463 0.414 0.510 0.356 0.540 0.401
0.514 U-Net++ 0.288 0.256 0.319 0.296 0.354 0.509 0.406 0.538 0.303 0.586
0.358 0.492 AttenU-Net 0.396 0.153 0.459 0.133 0.407 0.413 0.486 0.407 0.364
0.510 0.412 0.477 Guided-DM 0.307 0.263 0.324 0.311 0.365 0.492 0.429 0.483
0.276 0.576 0.279 0.548 HistoGene 0.268 0.284 0.310 0.321 0.346 0.517 0.412
0.512 0.245 0.610 0.237 0.608 TESLA 0.223 0.328 0.266 0.384 0.293 0.542 0.335
0.550 0.197 0.672 0.208 0.684 Ours 0.120 0.403 0.173 0.471 0.171 0.613 0.195
0.626 0.132 0.743 0.148 0.758

Figure 3: Visual comparisons at $5\times$ and $10\times$ scales on the Xenium
dataset. The ST maps are overlayed on the paired histology image for better
visualisation. Note that ANKRD30A, TPD52, GATA3 and SERPINA3 denote different
genes.

###  3.2 Performance evaluation

Quantitative comparison on Xenium dataset: We compare our model with six other
SOTA methods, i.e., TESLA [10], HistoGene [17], Guided Diffusion [13], U-Net
[18], U-Net++ [27] and AttenU-Net [16], at both 5$\times$ and 10$\times$
enlargement scales. Note TESLA [10] and HistoGene [17] are specifically
designed for ST SR tasks, while other common image SR methods are baselines.
Besides, to ensure the fairness, all the comparison methods use both the HR
histology image and LR ST maps for enhancing ST resolution. Experimental
results are shown in Table 1, in terms of Root MSE (RMSE) and Pearson
correlation coefficient (PCC), consistent with [10]. As shown, at 5$\times$
scale, Diff-ST performs the best, achieving improvement of at least 0.103 in
RMSE and 0.075 in PCC over others, indicating that Diff-ST could effectively
integrate histological features and gene expressions for ST SR. Similar
results can also be found in 10$\times$ scale. Notably, due to the extremely
high heterogeneity in spatial gene expression [14], the expression patterns
vary greatly across different tissues and genes, making the ST SR task
challenging due to complex data distribution and severe class imbalance.

Network generalizability analysis: We compare our model with SOTA methods on 2
external validation datasets, i.e., SGE and Breast-ST, without fine-tuning.
Results are shown in Table1, where both 5$\times$ and 10$\times$ SR scale
settings are tested. We observe that at 10$\times$ scale, our method achieves
an RMSE increment of 0.14 and 0.06 and a PCC increment of 0.076 and 0.074
compared to the best SOTA method, respectively, suggesting our superior
robustness and generalizability.

Visual comparison: Fig. 3 shows the restoration results of different methods
at both $5\times$ and $10\times$ scales on Xenium dataset. Diff-ST outperforms
all other methods, producing HR ST images with sharper edges and finer
details. More visual comparisons are presented in supplementary Fig. 2.

###  3.3 Ablation experiments

Table 2: Ablation Study on the Xenium dataset with $5\times$ and $10\times$
enlargement scale. CAM denotes cross-modal adaptive modulation.

Scale |  5$\times$ |  10$\times$
---|---|---
Metrics |  RMSE |  PCC |  RMSE |  PCC
$w/o$ CAM |  0.186 |  0.388 |  0.203 |  0.434
$w/o$ CIGC-Graph |  0.172 |  0.384 |  0.195 |  0.428
$w/o$ hierarchical modelling |  0.164 |  0.377 |  0.213 |  0.456
Diff-ST (Ours) |  0.120 |  0.403 |  0.173 |  0.471

We assess the contribution of three key components in Diff-ST: 1) $w/o$ cross-
modal adaptive modulation - integrate histology and ST features via feature
concatenation; 2) $w/o$ CIGC-Graph - treat each gene of the ST map separately
with simple gene channel concatenation; 3) $w/o$ hierarchical modelling -
extract only tissue-level features without cell- level image patching. The
5$\times$ and 10$\times$ scale results on the Xenium dataset are in Table 2.
All three models perform worse than Diff-ST, suggesting that these components
can enhance the overall model performance. The effectiveness of hierarchical
modelling demonstrates that hierarchical histology feature extraction can
successfully leverage tissue and cellular information for ST restoration.
Moreover, $w/o$ cross-modal adaptive modulation performs the worst, consistent
with our hypothesis that traditional conditional diffusion models may not
effectively leverage complementary information in multi-modal data of
histology and gene expression.

##  4 Summary

ST is an edge-cutting biotechnology but is limited by low spatial resolution
for in-depth research. This paper presents Diff-ST, a novel multi-modal
conditional diffusion model for ST super-resolution. We propose a cross-modal
adaptive modulation strategy to model modality interaction for effective
integration, which enables the modulation of complementary information from
multi-modalities for effective conditioned diffusion modeling. To effectively
extract intra-modal features from histology and gene expressions, we propose a
CIGC-graph network to model gene-to-gene relationships alongside a
hierarchical histological feature extraction to capture cell-to-tissue
relationships. Our experiments demonstrate that our model achieves superior
and robust performance over other state-of-the-art methods, serving as a
potential tool for in silico enhancing ST maps, facilitating downstream
discovery research and clinical translation.

## References

  * [1] Ash, J.T., Darnell, G., Munro, D., Engelhardt, B.E.: Joint analysis of expression levels and histological images identifies genes associated with tissue morphology. Nature communications 12(1), 1609 (2021)
  * [2] Badea, L., Stănescu, E.: Identifying transcriptomic correlates of histology using deep learning. PloS one 15(11), e0242858 (2020)
  * [3] Bergenstråhle, L., He, B., Bergenstråhle, J., Abalo, X., Mirzazadeh, R., Thrane, K., Ji, A.L., Andersson, A., Larsson, L., Stakenborg, N., et al.: Super-resolved spatial transcriptomics by deep data fusion. Nature biotechnology 40(4), 476–479 (2022)
  * [4] Chen, A., Liao, S., Cheng, M., Ma, K., Wu, L., Lai, Y., Qiu, X., Yang, J., Xu, J., Hao, S., et al.: Spatiotemporal transcriptomic atlas of mouse organogenesis using dna nanoball-patterned arrays. Cell 185(10), 1777–1792 (2022)
  * [5] Chen, R.J., Chen, C., Li, Y., Chen, T.Y., Trister, A.D., Krishnan, R.G., Mahmood, F.: Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16144–16155 (2022)
  * [6] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, 8780–8794 (2021)
  * [7] https://data.mendeley.com/datasets/29ntw7sh4r/5/:
  * [8] https://www.10xgenomics.com/datasets/fresh-frozen-visium-on-cytassist-human-breast-cancer-probe-based-whole-transcriptome-profiling-2-standard/:
  * [9] https://www.10xgenomics.com/products/xenium-in-situ/preview-dataset-human-breast/:
  * [10] Hu, J., Coleman, K., Zhang, D., Lee, E.B., Kadara, H., Wang, L., Li, M.: Deciphering tumor ecosystems at super resolution from spatial transcriptomics with tesla. Cell systems 14(5), 404–417 (2023)
  * [11] Li, H., Yang, Y., Chang, M., Chen, S., Feng, H., Xu, Z., Li, Q., Chen, Y.: Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing 479, 47–59 (2022)
  * [12] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [13] Mao, Y., Jiang, L., Chen, X., Li, C.: Disc-diff: Disentangled conditional diffusion model for multi-contrast mri super-resolution. arXiv preprint arXiv:2303.13933 (2023)
  * [14] Miller, B.F., Bambah-Mukku, D., Dulac, C., Zhuang, X., Fan, J.: Characterizing spatial gene expression heterogeneity in spatially resolved single-cell transcriptomic data with nonuniform cellular denasities. Genome research 31(10), 1843–1855 (2021)
  * [15] Moser, B.B., Shanbhag, A.S., Raue, F., Frolov, S., Palacio, S., Dengel, A.: Diffusion models, image super-resolution and everything: A survey. arXiv preprint arXiv:2401.00736 (2024)
  * [16] Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., Kainz, B., et al.: Attention u-net: Learning where to look for the pancreas. arxiv 2018. arXiv preprint arXiv:1804.03999 (1804)
  * [17] Pang, M., Su, K., Li, M.: Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors. bioRxiv pp. 2021–11 (2021)
  * [18] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234–241. Springer (2015)
  * [19] Schmauch, B., Romagnoni, A., Pronier, E., Saillard, C., Maillé, P., Calderaro, J., Kamoun, A., Sefta, M., Toldo, S., Zaslavskiy, M., et al.: A deep learning model to predict rna-seq expression of tumours from whole slide images. Nature communications 11(1), 3877 (2020)
  * [20] Ståhl, P.L., Salmén, F., Vickovic, S., Lundmark, A., Navarro, J.F., Magnusson, J., Giacomello, S., Asp, M., Westholm, J.O., Huss, M., et al.: Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353(6294), 78–82 (2016)
  * [21] Tang, X., Chen, J., Zhang, X., Liu, X., Xie, Z., Wei, K., Qiu, J., Ma, W., Lin, C., Ke, R.: Improved in situ sequencing for high-resolution targeted spatial transcriptomic analysis in tissue sections. Journal of Genetics and Genomics (2023)
  * [22] Wang, X., Price, S., Li, C.: Multi-task learning of histology and molecular markers for classifying diffuse glioma. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 551–561. Springer (2023)
  * [23] Wei, Y., Chen, X., Zhu, L., Zhang, L., Schönlieb, C.B., Price, S., Li, C.: Multi-modal learning for predicting the genotype of glioma. IEEE Transactions on Medical Imaging (2023)
  * [24] Zhang, D., Schroeder, A., Yan, H., Yang, H., Hu, J., Lee, M.Y., Cho, K.S., Susztak, K., Xu, G.X., Feldman, M.D., et al.: Inferring super-resolution tissue architecture by integrating spatial transcriptomics with histology. Nature Biotechnology pp. 1–6 (2024)
  * [25] Zhao, E., Stone, M.R., Ren, X., Guenthoer, J., Smythe, K.S., Pulliam, T., Williams, S.R., Uytingco, C.R., Taylor, S.E., Nghiem, P., et al.: Spatial transcriptomics at subspot resolution with bayesspace. Nature biotechnology 39(11), 1375–1384 (2021)
  * [26] Zhou, Z., Zhong, Y., Zhang, Z., Ren, X.: Spatial transcriptomics deconvolution at single-cell resolution using redeconve. Nature Communications 14(1), 7930 (2023)
  * [27] Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J.: Unet++: A nested u-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4. pp. 3–11. Springer (2018)

Figure 4:  Pipelines of CIGC-Graph network.  Table 3: Implementation details of our proposed method. Number of genes $N$ for ST maps | $25$
---|---
Number of cell-level patches $M$ per histology image | $256$
Features channels $d_{t}$ of $\mathbf{F}_{\mathrm{tissue}}$ | $128$
Features channels $d_{c}$ of $\mathbf{F}_{\mathrm{cell}}^{\mathrm{s}}$ | $128$
Intermediate channels $d$ in cross-attention modelling | $128$
Region size $a$ in cross-modal adaptive modulation | $32$
Number of features $C$ for the input nodes of CPLC-Graph | $676$
Graph balancing weight $\alpha$ | $0.2$
Exponential decay rate $\beta_{1}$ and $\beta_{2}$ for AdamW optimization | 0.9 and 0.999
Epsilon $\epsilon$ for AdamW optimization | $1\times 10^{-8}$
Weight decay for AdamW optimization | $1\times 10^{-5}$
Figure 5:  Additional visual comparisons at $5\times$ and $10\times$ scales on
the Xenium dataset. The ST maps are overlayed on the paired histology image
for better visualisation. Note that KRT7, ERBB2, POSTN, LUM, TACSTD2 and CLIC6
denote different genes.