# Embedding-based Multimodal Learning on Pan-Squamous Cell Carcinomas for
Improved Survival Outcomes

Asim Waqas  Department of Machine Learning, Moffitt Cancer Center & Research
Institute.  Department of Electrical Engineering, University of South Florida.
Aakash Tripathi  Department of Machine Learning, Moffitt Cancer Center &
Research Institute.  Department of Electrical Engineering, University of South
Florida.  Paul Stewart  Department of Biostatistics and Bioinformatics,
Moffitt Cancer Center & Research Institute.  Mia Naeini  Department of
Electrical Engineering, University of South Florida.  Ghulam Rasool
Department of Machine Learning, Moffitt Cancer Center & Research Institute.
Department of Electrical Engineering, University of South Florida.  Department
of Neuro-Oncology, Moffitt Cancer Center & Research Institute.  Department of
Oncologic Sciences, University of South Florida.

###### Abstract

Cancer clinics capture disease information at varying spatial (genetic to
molecular to tissue to organ and beyond) and temporal scales (disease
progression). The existing bioinformatic mechanisms do not encapsulate the
complete spectrum of the heterogeneous nature of the data, especially under
the challenge of missing data modalities. We propose an embeddings-based
relational learning through Graph Neural Network (GNN) that can efficiently
learn from multimodal, heterogeneous datasets for improved prediction of
clinical outcomes. Our framework, PARADIGM, (1) generates embeddings from
multi-resolution datasets using modality-specific foundation models, (2)
aggregates the sample embeddings into patient-level representations in latent
space, (3) fuses the aggregated embeddings into a unified graph
representation, and (4) provides improved performance metrics for downstream
tasks such as survival analysis. Our solution fuses unobserved but
interrelated cancer variables in non-Euclidean space through a scalable
framework. We train GNNs on survival prediction task using pan-Squamous Cell
Carcinomas (SCC) in the head and neck (HNSC), lung (LUSC), bladder (BLCA),
cervical (CESC), and esophageal (ESCA) cancers having 527; 496; 408; 294; and
184 patients, respectively. We also validate our approach for lung SCC data
collected at Moffitt Cancer Center, comprising 103 patients. The data
modalities for training and evaluation include EHR data (age at diagnosis,
gender, ethnicity, race, smoking status, etc.), whole slide images, pathology
reports, and molecular data (gene expression, miRNA expression, DNA
methylation, DNA mutation, and protein expression). The model evaluation used
the concordance index (C-index) of patient survival prediction and 7-fold
cross-validation. We compared the multimodal GNN results with other unimodal
and multimodal machine learning models, such as multilayer perceptron,
Transformers, and XGBoost, and got improved predictions compared to other
methods. We observe that the convergence of individual data modalities and
integration across varying scales creates a unified view of the disease that
is more insightful than the individual view or modality. Our solution aims to
converge the entire spectrum of the disease and understand the patient’s
genetic, physiological, and psychosocial circumstances in a unified framework.
The proposed method can help the community by providing helpful insights on
heterogeneous data integration and showing that the convergence of maximum
data views across varying occurrences can accrue remarkable discoveries about
the disease.

††footnotetext: *Equal contribution.††footnotetext: $\dagger$Correspondence:
<EMAIL_ADDRESS>
_Keywords_ Deep Learning $\cdot$ Multimodal learning $\cdot$ Oncology $\cdot$
Graph Neural Networks $\cdot$ Squamous Cell Carcinoma $\cdot$ Embeddings

##  1 Introduction

Clinical information about cancerous tumors is routinely recorded at different
scales and resolutions throughout the progression of the disease, treatment,
and survivorship Chen et al. (2021); Vanguri et al. (2022); Boehm et al.
(2021). The resulting data may include multiple diverse modalities Waqas et
al. (2024a), including (1) molecular and -omics information recorded from
genome, proteome, transcriptome, epigenome, and microbiome, (2) diagnostic
radiological imaging, e.g., ultrasound, computed tomography (CT), magnetic
resonance imaging (MRI), or positron emission tomography (PET) Fass (2008),
(3) histopathology, immunohistochemistry (IHC), and immunofluorescence (IF)
images and data, e.g., whole slide images (WSI) recorded from stained tumor
tissue samples Gutman et al. (2017); Magi-Galluzzi (2018), and (4) Clinical
data including Electronic Health Records (EHR) that consist of structured and
unstructured data about the patient, their disease, clinical notes from
routine visits, labs tests and vitals, and clinical reports from radiology,
pathology or biopsy Aschebrook-Kilfoy et al. (2022); Xie et al. (2022a).
Jointly learning from such multimodal, multiscale, heterogeneous information
with the possibility of out-of-distribution (OOD) inputs (e.g., unknown
disease or cancer type), incomplete, noisy, and missing data is challenging
but crucial for tackling complex diseases such as cancer. The state-of-the-art
multimodal artificial intelligence/machine learning (AI/ML) techniques use
data fusion methods and various flavors of deep neural networks, including
Transformers, convolutions neural networks (CNNs), multilayer perceptions
(MLPs), etc. Vanguri et al. (2022); Chen et al. (2020); Ahmed et al. (2023);
Boehm et al. (2022); Luo et al. (2022); Chen et al. (2022); Lipkova et al.
(2022); Lu et al. (2022a); Xu et al. (2023); Gao et al. (2020). None of these
models are intrinsically designed to handle heterogeneous multimodal datasets
with noisy, incomplete, and missing data during training and after deployment.
Given the enormous growth in the volume and variety of medical data and the
inherent noise in the data recording procedures/instruments, there is a need
to develop robust AI/ML models that can learn simultaneously from multimodal
heterogeneous datasets to answer clinical questions.

###  1.1 Multiscale, Heterogeneous Data in Cancer

The challenge of understanding cancer exists at different scales, including
(1) genetic and molecular aspects of cancer and its micro-environment, (2)
pathological information about tissue, (3) radiological information about the
organ, (4) physiology and health of the patient, and (5) their lifestyle. We
must also understand the dynamic changes happening over time during the
development and progression of the disease Vanguri et al. (2022); Acosta et
al. (2022). The advanced MedTech hardware/software allows us to take snapshots
of cancer development from a normal cell to a pre-malignant lesion to a
malignant tumor in many different ways and at different scales and
resolutions. The challenge is to coherently ingest, process, denoise, and
learn from these datasets El Naqa and Rollison (2021); Steyaert et al. (2023).
Integrating data from heterogeneous modalities is vital for creating a unified
view of cancer, which can be more insightful and predictive than a view
created by a single data modality.

Cancer patients undergo various diagnostic imaging scans, lab tests, medical
procedures, biopsies, and treatment regimens, including surgical resection,
chemotherapy, radiation therapy, immunotherapy, or targeted therapies. During
these activities, healthcare facilities collect and store various forms or
modalities of data, which are later analyzed by medical professionals for
treatment planning, disease monitoring, surveillance, and post-treatment
survivorship. In this work, we focus on three types of data modalities: (1)
digitized histopathology slides saved in the WSI file format, (3) -omics data
that includes genomics, proteomics, and transcriptomics, (4) clinical data,
which consists of patients’ demographic information, clinical notes (including
pathology reports), and lab results/vitals. Evidently, we are confronted with
diverse data types that capture different yet complementary views of the
underlying disease at different scales. That is, from molecular scale
(captured using -omics data) to tissue (quantified with histopathology data)
to demographic data (from EHR) and finally, a mixture of everything in semi-
structured or unstructured text format (in the form of clinical notes and
medical reports captured in the EHR).

###  1.2 Squamous Cell Carcinoma

Squamous cell carcinoma (SCC) is a type of cancer that can arise in various
organs and tissues beyond the skin, including the lungs, bladder, cervix,
esophagus, and head and neck region. It arises from squamous cells, which are
flat cells that line many surfaces in the body. Lung SCC accounts for
approximately 15-20% of all lung cancers, and head and neck SCC is the seventh
most common cancer worldwide Sabbula et al. (2023); Barsouk et al. (2023).
These cancers can be aggressive and have significant mortality rates,
highlighting the need for better understanding and treatment strategies. SCCs,
particularly in the head and neck, can severely impact the quality of life due
to their effects on essential functions like breathing, eating, and speaking.
The survival rate for SCC is very high when detected early, with a 5-year
survival rate of 99% Center (2024). The management of advanced SCCs often
involves a multidisciplinary approach combining surgery, radiation therapy,
chemotherapy, and targeted or immunotherapies. Ongoing research aims to
optimize these multimodal treatment strategies and identify the most effective
combinations for different SCC types and stages. Researchers are actively
exploring potential biomarkers for early detection, prognosis, and treatment
response prediction in various SCCs Campbell et al. (2018); Song et al.
(2022). In this work, we studied pan-squamous cell carcinoma (PanSCC)
comprising lung, bladder, cervix, esophagus, and head and neck subtypes.

Figure 1: The schematic layout of the multi-modal data integration framework
is presented. The framework integrates multi-omics data, pathology reports,
clinical/EHR data, and histopathology imaging data to generate sample
embeddings using modality-specific foundation models. These embeddings are
pooled and aggregated in patient embeddings which are then used to construct
patient graphs. These patient graphs are analyzed using Graph Neural Network
(GNN) to predict overall survival outcome.

###  1.3 Multimodal Learning (MML)

MML techniques combine information from various modalities to improve the
accuracy and reliability of a given ML task Acosta et al. (2022); Kline et al.
(2022); Lipkova et al. (2022); Tripathi et al. (2022); Soenksen et al. (2022);
Steyaert et al. (2023); Pei et al. (2023). We can define five stages of
multimodal learning, including preprocessing, feature extraction, data fusion,
primary learning, and final classification Sleeman IV et al. (2022). The “data
fusion” combines raw/extracted features or class prediction vectors from
multiple modalities to create a single data representation. Data fusion can be
performed in different ways: (1) _early fusion_ involves merging the modality-
specific features (or embeddings) into a single feature vector before training
AI/ML model, (2) _intermediate fusion_ involves training separate models for
each data modality and combining model outputs for prediction, and (3) in
_late fusion_ , the output of each model is used to produce a separate
decision, which are then combined to make a final decision. The choice of
fusion technique depends on the characteristics of the data and the specific
problem being addressed Acosta et al. (2022); Jiang et al. (2022); Kline et
al. (2022).

###  1.4 State-of-the-Art and Challenges in Oncology MML

Modern and classical AI/ML models have been used to fuse various modalities of
oncology data Liu et al. (2022); Hasanzadeh et al. (2022); Vanguri et al.
(2022); Boehm et al. (2022); Luo et al. (2022); Chen et al. (2022); Lipkova et
al. (2022); Lu et al. (2022a); Xu et al. (2023); Waqas et al. (2024a, b).
However, these techniques are generally ad-hoc and limit their analysis to a
selected set of modalities instead of using all available data.

The state-of-the-art MML models use Transformer-based architectures to fuse
and jointly learn from image and text data. Using these models out-of-the-box
or their variants for fusing medical data that includes various types of
radiology images, gigapixel histopathology/IHC/IF images, a variety of -omics
data, and semi-structured EHR data is neither straightforward nor optimal.

Most MML models targeted for oncology applications are ad-hoc by design, use
various types of AI/ML models and fusion methods subjectively, and involve a
significant amount of manual feature/model engineering focused on a specific
cancer type (or sub-type) and an organ Liu et al. (2022); Hasanzadeh et al.
(2022); Vanguri et al. (2022); Boehm et al. (2022).

Oncology MML models generally target a limited number of data modalities. For
example, radiology and genomics are fused for radio-genomics analysis,
pathology and -omics are mixed and referred to as pathomics, or any one
modality (e.g., radiology or pathology imaging) is fused with demographic data
(e.g., age or smoking status) from the EHR Vanguri et al. (2022); Li and Zhou
(2022); Brancato et al. (2022); Zhang et al. (2022).

The current state-of-the-art models use varying methods to address the
challenges of missing and noisy data, which is pervasive in medical datasets.
These models are not designed and trained to be robust to routine changes in
the data in medical settings, e.g., changes in the data recording protocols,
machine hardware/software/firmware updates, changes in patients’ demographics
owing to changes in the algorithm deployment site or a new mutation or disease
variant.

Answering cancer-related clinical questions using multimodal oncology datasets
has its unique challenges, nuances, and subtleties and therefore warrants its
own unified data integration framework that can (1) handle data heterogeneity,
missingness, and noise, (2) learn relations between different modalities of a
patient and between patients, and (3) make accurate predictions.

###  1.5 Contributions

Our proposed framework has six distinct components. First, we introduce an
embedding-based flexible and robust approach to tackle multi-modality cancer
data. Second, we introduce graphs on the generated embeddings and perform
graph structure learning to identify intra-/inter-cancer
relationships/patterns using GNNs. Third, graph-based and learning-based
methods with supervised techniques are introduced to fuse embeddings and
handle missing/incomplete data. Fourth, the proposed graph structure enables
multiscale learning across modalities. Fifth, we introduce a fusion mechanism
that keeps the maximum amount of information intact while capturing relational
patterns about the disease across different views of data. Sixth, we use the
self-normalizing weights initialization on graph convolutional layers as well
as use exponential linear unit (ELU) activation that ensures the self-
normalizing property of GNNs. The evaluation metric for our framework is
predicting accuracy for the given tasks of predicting OS quantified using the
concordance index (C-index) Uno et al. (2011). Our work will potentially play
a transformative role in the area of learning from multimodal, heterogeneous
data in general and oncology data. The overview of our multimodal embeddings-
based learning framework is illustrated in Figure 1.

##  2 Methods

We propose PARADIGM (Pan-Squamous Cell Carcinoma Representation using Advanced
Multimodal learning with Graph-based Modeling). Our proposed framework
benefits from the state-of-the-art AI/ML models for learning modality-specific
embeddings and combine these embeddings hierarchically using GNNs for learning
inter- and intra-cancer patterns.

###  2.1 PARADIGM Architecture

We use standard mathematical notations, i.e., lowercase letters ($e$) for
scalar values, lowercase bold (e) for column vectors, and uppercase letters
($E$) for matrices. The graphs are depicted by $G_{\text{sub}}$=$(V,C)$ having
node-set $V$=$\\{v_{1},v_{2},...,v_{n}\\}$, where node $v$ has feature vector
xv, edge set $C$=$\\{(v_{i},v_{j})\mid v_{i},v_{j}\in V\\}$, and subscript
${}_{\text{sub}}$ represents the purpose or task associated with the graph
$G$. Fig. 1 presents the main components of PARADIGM, (a) modality-specific
sample embeddings, (b) aggregation and fusion into patient embeddings, (c)
patient-specific graphs, and (d) joint graph learning. We describe these
components in detail in the following:

####  2.1.1 Modality-Specific Embeddings

The first component of the PARADIGM architecture consists of a set of pre-
trained, locally fine-tuned modality-specific AI/ML models, e.g., various
types of Transformers, CNNs, SNNs, ViTs, etc. Dosovitskiy et al. (2021);
Vaswani et al. (2017); Yang et al. (2022); Waqas et al. (2024b). Our
motivation is to leverage the state-of-the-art AI/ML pre-trained modality-
specific models to find the most representative embeddings for all four data
modalities. We prefer the models that have been pre-trained using related
medical datasets Yang et al. (2022); Azizi et al. (2022); Chen et al. (2024);
Waqas et al. (2023); Wasserthal et al. (2023); Qiu et al. (2023); Wornow et
al. (2023). After the first data processing step of modality-specific
operations, we get:
$E_{\text{Path}}=[\textbf{e}_{\text{p}}^{(1)},\textbf{e}_{\text{p}}^{(2)},\ldots,\textbf{e}_{\text{p}}^{(T_{p})}]$,
$E_{\text{Omics}}=[\textbf{e}_{\text{o}}^{(1)},\textbf{e}_{\text{o}}^{(2)},\ldots,\textbf{e}_{\text{o}}^{(T_{o})}]$,
$E_{\text{EHR}}=[\textbf{e}_{\text{e}}^{(1)},\textbf{e}_{\text{e}}^{(2)},\ldots,\textbf{e}_{\text{e}}^{(T_{e})}]$,
and
$E_{\text{PRep}}=[\textbf{e}_{\text{pr}}^{(1)},\textbf{e}_{\text{pr}}^{(2)},\ldots,\textbf{e}_{\text{pr}}^{(T_{pr})}]$
for WSI patches, -omics, EHR, and pathology report datasets, respectively. We
have $E_{\text{Path}}\in\mathbb{R}^{D_{p}\times T_{p}}$,
$E_{\text{Omics}}\in\mathbb{R}^{D_{o}\times T_{o}}$,
$E_{\text{EHR}}\in\mathbb{R}^{D_{e}\times T_{e}}$, and
$E_{\text{PRep}}\in\mathbb{R}^{D_{pr}\times T_{pr}}$, which gives
$\textbf{e}_{\text{p}}\in\mathbb{R}^{D_{p}}$,
$\textbf{e}_{\text{o}}\in\mathbb{R}^{D_{o}}$,
$\textbf{e}_{\text{e}}\in\mathbb{R}^{D_{e}}$, and
$\textbf{e}_{\text{pr}}\in\mathbb{R}^{D_{pr}}$. Here,
$T_{p},T_{o},T_{e},\text{and}T_{pr}$ represent the total number of patients,
and $D_{p},D_{o},D_{e},$ and $D_{pr}$ represent the size of embedding vector
for the WSI patches, -omics, EHR, and pathology reports datasets,
respectively. $D_{p},D_{o},D_{e},$ and $D_{pr}$ can have different values
based on the complexity of the data modality. The EHR data is considered the
baseline and is always available for all patients. Other data modalities may
be missing, i.e., $T_{p}$, $T_{o}$, $T_{pr}$ $\leq T_{e}$.

####  2.1.2 Concatenation and Aggregation

We used a two-step process (1) aggregate the sample embeddings into patient-
level embeddings and (2) concatenate resulting embeddings across all
modalities. In the first step, we aggregate the different samples’ embedding
vectors for the same patient into a single patient embedding for each
modality. We use a learning-based aggregation function to combine all
available WSI sample embeddings for each patient to produce
$E_{\text{M}}=AGG_{\text{avail}}(E_{\text{M}}^{1},E_{\text{M}}^{2},E_{\text{M}}^{3},...,E_{\text{M}}^{u})$,
where $AGG_{\text{avail}}$ represents an aggregator function implemented with
a multiple-instance learning network for modality $M$ having total samples $u$
for each patient. $AGG_{\text{avail}}$ is trained using the ground-truth data
of ’age at index’ and MSE loss function. For the -omics and pathology reports
data modalities, we simply aggregate the sample embeddings by mean pooling to
generate patient embeddings for the respective modalities. As the result of
this aggregation stage, we get aggregated modality embeddings
$\textbf{e}_{\text{o}}\in\mathbb{R}^{48}$,
$\textbf{e}_{\text{p}}\in\mathbb{R}^{1024}$,
$\textbf{e}_{\text{e}}\in\mathbb{R}^{1024}$, and
$\textbf{e}_{\text{pr}}\in\mathbb{R}^{1024}$ for -omics, WSIs, EHR, and
pathology reports, respectively. The second step involves concatenating the
embedding vectors from different modalities into a single latent
representation for each patient. The aggregated and concatenated patient
embedding is represented by
$E_{\text{pat}}=cat(E_{\text{Omics}}^{u},E_{\text{PRep}}^{u},E_{\text{EHR}}^{u},E_{\text{Path}}^{u})$,
where $cat$ represents the concatenation operation such that
$E_{\text{pat}}\in\mathbb{R}^{3120\times T_{e}}$. Thus, the new embedding
vector for each patient now has information from the initial modality
embeddings, $E_{\text{M}}$.

####  2.1.3 Patient-Specific Graphs

We construct patient graph $G_{F}$ where each node of $G_{F}$ represents a
patient and the node embeddings are given by
$E_{f}=[\textbf{x}_{f}^{(1)},\textbf{x}_{f}^{(2)},\ldots,\textbf{x}_{f}^{(T_{e})}]$.
The (weighted) adjacency matrix for the patient graph is calculated using the
Euclidean distance between embedding vectors Stamile et al. (2021). For the
patient graph $G_{F}$, the adjacency matrix can be found using:
$A_{\text{pat}}=Euc_{ij}(E_{M})=\sqrt{\sum_{k=1}^{D_{e}}{\left(E_{M_{ki}}-E_{M_{kj}}\right)^{2}}}$.
The patient graph has the number of nodes equal to the number of patients in
the clinical (EHR) data $\lvert V_{\text{pat}}\rvert=T_{e}$. We also generated
the adjacency matrix using cosine distance between patient embeddings for
comparing the effect of selecting different distance metrics on the downstream
analysis.

####  2.1.4 Joint Graph Learning

Finally, the joint patient graph is used to train GNN using supervised
learning and the MSE loss function. As the result of the graph learning, the
node feature vector for each patient is updated from its neighbors in the
message passing mechanism.

Figure 2: A schematic layout of the multimodal oncology database system
(MINDS) Tripathi et al. (2024a) to collate, retrieve, harmonize, and serve
multimodal data to PARADIGM models for training. We integrate data from
National Cancer Institute (NCI)’s publicly available resources including,
Cancer Research Data Commons (CRDC), Genomic Data Commons (GDC), Imaging Data
Commons (IDC), and Proteomic Data Commons (PDC) Hinkson et al. (2017); Ellis
et al. (2013); Tonsing-Carter et al. (2023); Tomczak et al. (2015).

###  2.2 Learning Strategy

Having presented the architecture of the PARADIGM framework, we now present
the learning strategy of our framework as follows.

####  2.2.1 Modality-Specific learning

The modality-specific learning involves three tasks, (1) dataset development,
(2) evaluating and selecting the modality-specific publicly available
(foundation) models, and (3) transfer learning and fine-tuning of modality-
specific models.

  1. 1.

Datasets Development: Training multimodal models requires multimodal cancer
datasets with EHR data as the base modality and one or more of the following:
radiology, pathology, or -omics. We used public databases developed and shared
by the National Institute of Health (NIH) and National Cancer Institute (NCI)
and Moffitt’s local datasets to train and validate our framework. Generally,
the publicly available datasets may have only one or two data modalities. In
case multiple modalities are present, they are scattered across different
databases, and it is difficult to identify and link patients across databases.
We use the already developed multimodal database, MINDS, to curate and build
patient cohorts across five squamous cell carcinoma datasets (TCGA-LUSC, TCGA-
HNSC, TCGA-CESC, TCGA-ESCA, and TCGA-BLCA) Tripathi et al. (2024a). We
acquired four different types of data modalities. The multi-omics data
includes protein expression, DNA mutation, miRNA expression, gene expression,
and DNA methylation, along with clinical data We used the preprocessing steps
for multiomics data adopted by SeNMo Waqas et al. (2024b). Pathology reports
are processed through text extraction and text splitting techniques to obtain
meaningful segments of text data. Clinical/EHR, which includes structured
clinical records, and histopathology data comprising WSIs have been curated
from the MINDS system Tripathi et al. (2024a). We also evaluated the framework
by training on the data collected at Moffitt ($T=103$) and consisted of EHR
data (including age at diagnosis, gender, ethnicity, race, smoking status,
year of diagnosis, vital status (alive/dead), and tumor cellularity),
pathology images, and -omics data (included RNA-Seq expression and protein
expression) Paul and Komarova (2021).

  2. 2.

Modality-Specific Models: We leverage pre-trained models for initial modality-
specific data processing. Based on our analysis, we have selected pre-trained
models for obtaining initial embeddings after supervised and self-supervised
fine-tuning of these models individually and jointly Yang et al. (2022); Waqas
et al. (2024b). For the WSIs, we evaluated various models and selected UNI for
generating embeddings Chen et al. (2024). For the clinical data such as EHR
and pathology reports, we evaluated GatorTron and ClinicalT5 models Yang et
al. (2022); Lu et al. (2022b). Based on our experiments, we selected GatorTron
as our embedding model because of its superior performance on medical language
tasks and text interpretation from pathology reports Yang et al. (2022). For
the multi-omics model, we selected the only model that has been trained on the
five molecular data types across 33 cancer sites and is publicly available,
SeNMo Waqas et al. (2024b). All of these embeddings are publicly available
through HoneyBee framework and hosted on Hugging Face Tripathi et al. (2024b).

  3. 3.

Fine-tuning Modality Models and Embeddings Generation: The preprocessed data
was then used to fine-tune the modality-specific models on different tasks
such as OS, or self-supervised contrastive loss. After fine-tuning, we
generated the embeddings for each modality model for each data sample. It is
pertinent to mention here that these are sample-level vectors in the case of
molecular and pathology report data and patch-level embeddings in the case of
WSIs.

####  2.2.2 Concatenation and Aggregation of Embeddings

As a result of the processing presented above, we get the initial embeddings
for each data sample in each category of modality. Next, we combine these for
each modality to build a single feature vector for each patient, referred to
as patient-level embedding
$E_{M}=AGG_{\text{avail}}(E_{\text{M}}^{1},E_{\text{M}}^{2},E_{\text{M}}^{3},...,E_{\text{M}}^{u})$.
In the case of patch-level embeddings for WSIs generated from the GatorTron
model, we employed multiple-instance learning (MIL) where a neural network is
trained in a weakly-supervised fashion to generate the slide-level embeddings
Gadermayr and Tschuchnig (2024). We experimented with attention-based MIL
(ABMIL), mean-pooling, and max-pooling learning strategies and selected ABMIL
for its superior performance in predicting the patient’s age at diagnosis Ilse
et al. (2018). The molecular and pathology reports embeddings had only a few
patients with more than one sample, so we employed a simple pooling using the
mean of the sample vectors. As the result of aggregation, we get the uniform-
dimensional embedding vectors for each patient in each modality type. After
aggregation, we concatenated the patient’s modality embeddings into a single
embedding vector per patient,
$E_{\text{pat}}=cat(E_{\text{Omics}}^{u},E_{\text{PRep}}^{u},E_{\text{EHR}}^{u},E_{\text{Path}}^{u})$.
It is important to note here that the concatenation operation caters for the
missing modalities by padding zeros to the missing feature vectors. This way,
the framework takes the union of patients across different modalities to
ensure that patients with missing data modalities, as in the real-world
scenario, are not excluded from the analysis. The concatenation approach
allows the retention of all the available information for the given patient.
The embedding aggregation and concatenation operations are needed for three
reasons, (1) to reduce the size of modality-specific embeddings to represent
patient-level embeddings, (2) to combine embeddings from different data
modalities in a joint space for pathology, -omics, and EHR data, and (3) to
produce a joint embedding of the same size despite the possibility of missing
some data modalities for some patients.

####  2.2.3 Patient Graphs and Joint Graph learning

The final set of embeddings are used as nodal features in the joint graph,
which are trained to predict the selected endpoints of OS. It is important to
highlight that the choice of the endpoint (i.e., OS) does not limit the
PARADIGM framework. Based on the datasets, cancer type, and organ/site, our
framework can support various other endpoints: tumor response rate, tumor
shrinkage, disease-free survival, time to treatment discontinuation, toxicity,
and time to next treatment. We use joint embedding for each patient to build a
graph that represents one disease, i.e., one graph for each of the five cancer
types. We then combine these subgraphs into a joint PanSCC graph data and use
it to train the GNN. We used various measures to quantify the change in the
embeddings, e.g., Euclidean or cosine distance. For the prediction of OS, we
used convolution-based GNNs as they aggregate neighboring nodes’ embeddings
through a stack of multiple layers Xie et al. (2022b). We tested commonly used
spatial and spectral convolutional GNNs such as GraphSAGE Hamilton et al.
(2017), Graph Attention Network (GAT) Veličković et al. (2017), Graph
Convolutional Network (GCN) Kipf and Welling (2016), and Graph Isomorphism
Network (GIN) Xu et al. (2019). Based on our preliminary experiments, we
selected the GCN model as our downstream model Kipf and Welling (2016).

###  2.3 Model Evaluation

We have five trained models based on our selection of five cancers (cervical,
bladder, head and neck, lung, and esophageal) and one model for PanSCC. We
also compare the performance of PARADIGM models with the single-modality, two-
modality, and three-modality models to highlight the benefits of using the
multimodal approach. We also compare the performance of our models with the
state-of-the-art multimodal models. We evaluate the models’ performance using
the C-Index Lifelines (2023), which compares the predicted outcome
probabilities from the model with the ground-truth data. C-Index ranges from 0
to 1, where a value of 1 indicates the model can perfectly distinguish between
individuals who experience an event earlier and those who experience it later.
A value of 0.5 suggests that the model performs no better than random chance,
while a value below 0.5 indicates that the model’s predictions are inversely
related to the observed outcomes. We use logrank statistical test to quantify
the performance of the models to establish statistical significance.

Figure 3: C-Index on training different models on the SCC data. The multimodal
datasets consist of clinical, pathology reports, whole slide images (WSIs),
and molecular data.

##  3 Results

###  3.1 Multimodal Integration of Oncology Database System (MINDS)

Accessing, storing, pre-processing, harmonizing, clearing, and feeding
multimodal data, consisting of various images, text, real numbers, and
categories, to our models for training is a significant challenge. Given that
no such system existed in oncology, we developed an in-house database system
called MINDS and presented it in Fig. 2 Tripathi et al. (2024a). MINDS can
handle all types of data, including -omics (e.g., genomics, transcriptomics,
and proteomics), diagnostic radiological imaging, histopathology/IHC/IF
imaging, and EHR. MINDS employs data harmonization techniques to integrate all
data types seamlessly and enables training PARADIGM models with up to
petabytes of data. MINDS allows building cohorts for various cancer sites and
types/sub-types. We followed the following scheme to build MINDS:

  1. 1.

Data Ingestion: We collected (semi-)structured data from Genetic Data Commons
(GDC), Imaging Data Commons (IDC), and Proteomic Data Commons (PDC) Fedorov et
al. (2021); Grossman et al. (2016); Thangudu et al. (2020). AWS Glue was used
to pre-process and aggregate data Amazon Web Services (a, b).

  2. 2.

Data Storage: The pre-processed semi-structured data were uploaded to an
Amazon S3 ingest buckets Amazon Web Services (c). The structured data was
stored in an AWS RDS instance Amazon Web Services (d).

  3. 3.

Data Lake Formation: We used AWS Data Lake Formation tool Amazon Web Services
(e), which automated transforming the semi-structured data stored in the S3
bucket into a query-able data lake using AWS Glue crawlers and AWS Athena
Amazon Web Services (f).

  4. 4.

Data Warehouse Integration: The structured data stored in AWS RDS was
integrated into a data warehouse using Amazon Redshift Amazon Web Services
(g).

  5. 5.

Cohort Selection and Data Export: We used AWS step functions to coordinate and
manage the data export process for the selected cohort. We also provide
manifest files to interact with GDC, IDC, and PDC and download all the
unstructured data using their APIs.

###  3.2 Modality-Specific Models, Fine-tuning, and Transfer Learning

We investigated various modality-specific models for our data modalities,
pathology, -omics, pathology reports, and EHR. Our criterion for evaluating
the representativeness or optimality of these modality-specific models
includes their predictive performance against the ground-truth data of OS. As
a result of preliminary analysis, we used: (1) UNI for histopathology images
Chen et al. (2024). UNI is a vision transformer model pre-trained on whole
slide images using the DINOv2 self-supervised framework Chen et al. (2024). We
selected UNI for its ability to capture intricate details and patterns in
histopathology images through self-supervised pre-training Chen et al. (2024).
(2) GatorTron for the EHR data (including clinical notes and pathology
reports) Yang et al. (2022). GatorTron is a large language model specifically
designed for clinical NLP. It was trained on a massive corpus of 277 billion
words, including 82 billion words from de-identified clinical notes and 195
billion words from various English texts. (3) Self-Normalizing Networks
(SeNMo) for -omics data Waqas et al. (2024b), which is a mini-foundation model
that has been trained on five molecular data types for more than 13,000
patients across 33 cancer types. Our experiments showed that UNI and GatorTron
worked well with fine-tuning using small datasets Yang et al. (2022); Azizi et
al. (2022); Paul and Komarova (2021); however, SeNMo always required transfer
learning with larger datasets, potentially linked to the complexity and
variability of the -omics datasets Waqas et al. (2024b, a).

Figure 4: C-Index for OS predictions from different models on the SCC data
comprising clinical, pathology reports, and WSIs data. Figure 5: C-Index for
OS predictions from different models on the SCC data comprising clinical,
molecular, and WSIs. Figure 6: C-Index for OS predictions from different
models on the SCC data comprising clinical, pathology reports, and molecular
data.

###  3.3 Prediction of OS in PanSCC Cohort

We predicted OS for pan-squamous cell carcinoma (PanSCC) patients using our
framework, and a host of multimodal and uni-modal MLPs, Transformers, and
classical ML algorithms such as eXtreme Gradient Boosting (XGBoost), and
Support Vector Regression (SVR).

####  3.3.1 Four-modality Analysis

Figure 3 shows the performance of different ML and DL models trained on
multimodal datasets for SCC comprising four modalities; clinical, pathology
reports, whole slide images (WSIs), and molecular data. The models include
Multilayer Perceptrons (MLPs), Transformers, XGBoost, Support Vector Machines
(SVMs), and GNNs. Each model was tested across various types of squamous cell
carcinoma, lung squamous cell carcinoma (LUSC), head and neck squamous cell
carcinoma (HNSC), esophageal carcinoma (ESCA), bladder carcinoma (BLCA), and
pan-squamous cell carcinoma (PanSCC) comprising all five cancer types. The
performance metric used is the C-Index for OS predictions, shown on the
vertical axis. The MLP models, represented in blue, exhibit moderate
performance with C-Index values around 0.5 to 0.6, with the best performance
on the panSCC cohort. Transformer models, shown in beige, demonstrate similar
performance to MLPs. The XGBoost models, depicted in red, generally outperform
the MLP and Transformer models, achieving higher C-Index values, indicating
better prediction accuracy. The SVM models, represented in gray, show
consistent but slightly lower performance compared to XGBoost models. The GNN
models, shown in green, achieve the highest C-Index values across all cancer
types, suggesting that GNNs are particularly effective in leveraging
multimodal data for accurate survival predictions. The error bars indicate the
variability in the performance across different folds, providing insight into
the reliability of the predictions.

####  3.3.2 Three-modality Analysis

Figure 4 illustrates the performance of different ML and DL models on
multimodal datasets for SCC, encompassing clinical data, pathology reports,
and WSIs. The MLP models, show moderate performance with C-Index values around
0.5, indicating the random predictions. Transformer models, depicted in beige,
demonstrate similar performance to MLPs, with C-Index values clustering around
the same range. The XGBoost models outperform both MLP and Transformer models,
achieving higher C-Index values, which suggests better survival prediction
accuracy. The SVM models, represented in gray, exhibit comparable performance
to MLPs and Transformers. The GNN models achieve the highest C-Index values
across all cancer types, significantly outperforming the other models.

Figure 5 illustrates the performance of MLPs, Transformers, XGBoost, SVMs, and
GNNs SCC data incorporating clinical data, molecular data, and WSIs. The MLP
models show random performance for LUSC, HNSC, and ESCA with C-Index values
$<$0.5, while a gradual improvement as we move from CESC, BLCA to panSCC data
cohorts with C-Index $<$0.6. Transformer models do not perform well with all
cohorts having C-Index $<$0.5. The XGBoost models consistently outperform the
MLPs, Transformers, and SVM models. GNNs achieved the highest C-Index values
across all cancer types, significantly outperforming the other models.

Figure 6 shows the MLPs, Transformers, XGBoost, SVMs, and GNNs trained on SCC
data incorporating clinical data, pathology reports, and molecular data. The
MLP models performed better than all previous three-modality cohorts having
C-Index $>$0.5 in five out of six datasets. Transformer models have similar
ranges in C-Indices as before. The XGBoost models generally outperform the MLP
and Transformer models, achieving higher C-Index values. The SVM models have
shown better performance in this cohort with comparable performance to the
XGBoost models. The GNN models achieved the highest C-Indices across all
cancer types, significantly outperforming the other models.

####  3.3.3 Two-modality Analysis

Figure 7 shows the performance of MLPs, Transformers, XGBoost, SVMs, and GNNs
applied to multimodal datasets for SCC, integrating clinical data and WSIs.
The MLPs, Transformers, and SVM models exhibit random performance with C-Index
$<$0.5 for most of the cohorts. The XGBoost models outperform the MLPs,
Transformers, and SVMs, achieving C-Index values from 0.5 to 0.6. GNNs achieve
the highest C-Index values across all cancer types, ranging from 0.65 to 0.72.

Figure 8 illustrates that MLPs achieved the predictive performance for C-Index
between 0.5 and 0.59, while XGBoost achieved values between 0.49 for ESCA to
0.63 for LUSC. Transformers fail to perform well, while SVMs have moderate
performance range of 0.49-0.56. GNN models, illustrated in green, achieve the
highest C-Index values across all cancer types, with all values around 0.7.

Figure 9 presents the performance of various models applied to SCC data with
clinical and molecular data types. As with the previous results, the order of
performance is Transformers, SVMs, MLPs, XGBoost, and GNNs, in increasing
C-Index on predicting OS.

####  3.3.4 Internal Data Analysis

Figure 10 shows the results from analyzing Lung SCC data collected at Moffitt
Cancer Center Stewart et al. (2019). The data consists of four modalities,
including pathology images, clinical data, and two subtypes of molecular data,
RNA-Seq expression, and protein expressions Stewart et al. (2019). We
generated pathology embeddings using REMEDIS Azizi et al. (2022) and EHR
embeddings using GatorTron Yang et al. (2022), without fine-tuning the models.
We trained SeNMo for -omics data to generate RNA-Seq expression and protein
expression embeddings. Model evaluation was done using C-index and 10-fold
cross-validation, as shown in Figure 10. The models selected for evaluation
are MLP, Transformer, self-normalizing network (SNN), and GNN. For two-
modality combinations, MLP models show varying performance, with C-Index
values ranging from approximately 0.5 to 0.7. Transformer models with two
modalities exhibit poor performance compared to MLPs. For the three-modality
combinations, the models generally show improved performance, especially the
Transformers models exhibit better performance compared to two modalities. The
four-modality combinations exhibit the highest performance. The MLP model
trained on EHR, RNA, Protein, and WSIs achieves a C-Index of around 0.65,
indicating strong predictive accuracy. The Transformer model with the same
four modalities achieves a high C-Index, 0.91. The GNN model further improves
the C-Index to 0.93, demonstrating the benefit of integrating all four data
modalities.

Figure 7: Performance Comparison of OS prediction by different models train on
SCC datasets comprising clinical and WSIs data, measured by C-Index.

##  4 Discussion

Disease-related information for oncology resides at varying scales and
resolutions of data. Clinicians routinely fuse such diverse information
mentally for decision-making, but there are limits to human processing
capabilities. On the other hand, AI/ML models struggle when ingesting such
heterogeneous, multiscale information for critical decision-making. Graphs and
GNNs have been shown to perform well in contextual learning as well as
represent the robust architectures of deep neural networks Waqas et al. (2022,
2024a). Jointly learning from such multimodal, multiscale, heterogeneous
information with the possibility of incomplete and missing data modalities is
challenging but crucial for understanding and tackling complex diseases like
cancer. Our proposed framework for multimodal learning from multiscale,
heterogeneous oncology datasets, has been trained and validated for accurately
and robustly predicting overall survival (OS) for different tumor types (e.g.,
head and neck, lung, esophageal, cervical, and bladder cancers). We have also
evaluated the framework on the Moffitt Cancer Center & Research Institute’s
internal lung squamous cell carcinoma cohort comprising 103 patients. Our
framework is built on pre-trained AI/ML models, Graph Neural Networks (GNNs),
and self-surprised and supervised techniques to learn from multimodal,
heterogeneous datasets. Our multimodal framework’s potential applications go
beyond cancer and healthcare settings, such as the deployment of AI/ML in many
mission-critical application areas, including aviation safety, autonomous
control of vehicles, automated decision-making in human-machine integrative
environments, data fusion applications, and financial systems. Overall, the
results indicate that models trained with more modalities tend to achieve
higher C-Index values, reflecting better predictive accuracy for OS panSCC
data. The highest performance is observed with the four-modality combinations,
particularly for the GNNs-based PARADIGM model. All models and processed data
have been made available via Hugging Face and GitHub.

Figure 8: Performance Comparison of OS prediction by different models train on
SCC datasets comprising clinical and pathology report data, measured by
C-Index. Figure 9: Performance Comparison of OS prediction by different models
train on SCC datasets comprising clinical and molecular data, measured by
C-Index. Figure 10: C-indices from different models trained combinations of
modalities. The horizontal axis legend is a tuple: number-(model, modalities):
1-(MLP, EHR + RNA), 2-(MLP, EHR + Protein), 3-(MLP, EHR + Path), 4-(MLP, RNA +
Protein), 5-(MLP, RNA + Path), 6-(MLP, Protein + Path), 7-(Transformer, EHR +
RNA), 8-(Transformer, EHR + Protein), 9-(Transformer, EHR + Path),
10-(Transformer, RNA + Protein), 11-(Transformer, RNA + Path),
12-(Transformer, Protein + Path), 13-(MLP, EHR + RNA + Protein), 14-(MLP, EHR
+ RNA + Path), 15-(MLP, EHR + Protein + Path), 16-(MLP, RNA + Protein + Path),
17-(Transformer, EHR + RNA + Protein), 18-(Transformer, EHR + RNA + Path),
19-(Transformer, EHR + Protein + Path), 20-(Transformer, RNA + Protein +
Path), 21-(MLP, EHR + RNA + Protein + Path), 22-(Transformer, EHR + RNA +
Protein + Path), 23-(GNN, EHR + RNA + Protein + Path).

###  4.1 Embedding-based Approach

The proposed framework introduced an embedding-based flexible and robust
approach for multimodal learning. It benefits from the open-source models,
pre-trained with histopathology, -omics, and EHR data Azizi et al. (2022);
Yang et al. (2022); Waqas et al. (2024b); Chen et al. (2024), later fine-
tuning these models to use them as the feature (or embeddings) extractors.
This helped with rich feature extraction from all modalities at a
significantly reduced computational cost.

###  4.2 Graph Structure and Contextual Learning

Our framework introduced graph structure and GNNs on these embeddings to
perform intra-/inter-cancer learning from heterogeneous datasets. The GNN-
based approach enabled the representation of patient embeddings in the form of
graph-structured data, where nodes represented patients and weighted edges
between nodes represented inter-patient similarities. The process of updating
the features of the current node based on its neighbors and the strength of
the edges between these nodes helped in intra-/inter-cancer learning even
though neighboring nodes may represent different cancers.

Figure 11: Impact of input graph sparsity on the predictive performance of
GNNs. The C-Index is displayed for dense and sparse graph configurations using
various combinations of clinical, pathology reports, molecular, and WSIs data.

###  4.3 Effect of Graph Sparsity

In analyzing the graph structures and their effects on the model’s
performance, we varied the sparsity of the input graphs. Sparsity is
controlled by thresholding the weights on the edges quantified by the distance
metrics, such as Euclidean or cosine distance. Figure 11 illustrates the
effect of changing the graph sparsity on the prediction of OS by the same GNN
model. Across all combinations of modalities and cancer data types, the dense
graph configuration generally achieves significant C-Index values. When we
increase the graph sparsity, there is a notable increase in the model’s
performance, with C-Index values increasing to $>$0.8 in almost all the cases.
This indicates that sparse graphs carry high signal-to-noise ratio and better
predictive power for the GNN models, compared to dense graphs. Moreover, the
inclusion of more data modalities generally enhances predictive accuracy, but
the benefit is maximized when the graph is sparsely connected.

###  4.4 Handling Missing Modalities

The challenge of missing modalities is handled in our framework. Some patients
may not have all the modalities; however, our framework allows learning with
the available data and updating nodal features.

###  4.5 Hierarchical Learning

The framework’s hierarchical nature enables it to handle sub-modalities
seamlessly. We can add another data modality to fuse with the existing data
types. For example, radiological scans such as MRI, CT, PET scans can be
processed through REMEDIS foundation model to generate sample embeddings, or
different stains of histopathology images such as IHC or IF slides can be
added to the existing pipeline seamlessly. This innovative methodology of
hierarchically fusing heterogeneous information based on the data modalities
and sub-modalities is vital for learning from multiscale datasets that capture
various aspects of underlying phenomena in space and time.

##  5 Conclusion

Our proposed framework, PARADIGM, demonstrates a significant advancement in
the field of multimodal, heterogeneous data integration for cancer outcome
prediction. By leveraging graph neural networks and modality-specific
foundation models, we successfully generated comprehensive patient-level
representations that capture the intricate relationships across various data
modalities, including EHR, whole slide images, pathology reports, and
molecular data. Our approach not only improved the predictive accuracy for
survival analysis in pan-Squamous Cell Carcinomas (SCC) across multiple cancer
types but also showcased the robustness and scalability of GNNs in handling
complex, multimodal datasets. The superior performance of PARADIGM compared to
traditional machine learning models underscores the importance of integrating
diverse data sources to achieve a holistic understanding of cancer
progression. Our findings suggest that the convergence of genetic,
physiological, and psychosocial data into a unified framework can provide
deeper insights and more accurate predictions, ultimately contributing to
personalized and precision oncology. This research paves the way for future
studies to explore the full potential of GNNs and multimodal data integration
in various clinical applications, offering a promising direction for improving
patient outcomes in cancer and beyond.

## References

  * Chen et al. [2021] Richard J Chen, Ming Y Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood.  Multimodal co-attention transformer for survival prediction in gigapixel whole slide images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4015–4025, 2021.
  * Vanguri et al. [2022] Rami S Vanguri, Jia Luo, Andrew T Aukerman, Jacklynn V Egger, Christopher J Fong, Natally Horvat, Andrew Pagano, Jose de Arimateia Batista Araujo-Filho, Luke Geneslaw, Hira Rizvi, et al.  Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L) 1 blockade in patients with non-small cell lung cancer.  _Nature Cancer_ , 3(10):1151–1164, 2022.
  * Boehm et al. [2021] Kevin M Boehm, Pegah Khosravi, Rami Vanguri, Jianjiong Gao, and Sohrab P Shah.  Harnessing multimodal data integration to advance precision oncology.  _Nature Reviews Cancer_ , pages 1–13, 2021.
  * Waqas et al. [2024a] Asim Waqas, Aakash Tripathi, Ravi P Ramachandran, Paul Stewart, and Ghulam Rasool.  Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review.  _arXiv preprint arXiv:2303.06471_ , 2024a.  https://arxiv.org/abs/2303.06471.
  * Fass [2008] Leonard Fass.  Imaging and cancer: a review.  _Molecular oncology_ , 2(2):115–152, 2008.
  * Gutman et al. [2017] David A Gutman, Mohammed Khalilia, Sanghoon Lee, Michael Nalisnik, Zach Mullen, Jonathan Beezley, Deepak R Chittajallu, David Manthey, and Lee AD Cooper.  The digital slide archive: A software platform for management, integration, and analysis of histology for cancer research.  _Cancer research_ , 77(21):e75–e78, 2017.
  * Magi-Galluzzi [2018] Cristina Magi-Galluzzi.  Prostate cancer: diagnostic criteria and role of immunohistochemistry.  _Modern Pathology_ , 31:12–21, 2018.
  * Aschebrook-Kilfoy et al. [2022] Briseis Aschebrook-Kilfoy, Paul Zakin, Andrew Craver, Sameep Shah, Muhammad G Kibriya, Elizabeth Stepniak, Andrea Ramirez, Cheryl Clark, Elizabeth Cohn, Lucila Ohno-Machado, et al.  An overview of cancer in the first 315,000 all of us participants.  _PloS one_ , 17(9):e0272522, 2022.
  * Xie et al. [2022a] Feng Xie, Han Yuan, Yilin Ning, Marcus Eng Hock Ong, Mengling Feng, Wynne Hsu, Bibhas Chakraborty, and Nan Liu.  Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies.  _Journal of Biomedical Informatics_ , 126:103980, 2022a.  ISSN 1532-0464.  doi:https://doi.org/10.1016/j.jbi.2021.103980.  URL https://www.sciencedirect.com/science/article/pii/S1532046421003099.
  * Chen et al. [2020] Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Lindeman, and Faisal Mahmood.  Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis.  _IEEE Transactions on Medical Imaging_ , 2020.
  * Ahmed et al. [2023] Sabeen Ahmed, Ian E Nielsen, Aakash Tripathi, Shamoon Siddiqui, Ravi P Ramachandran, and Ghulam Rasool.  Transformers in time-series analysis: A tutorial.  _Circuits, Systems, and Signal Processing_ , 42(12):7433–7466, 2023.
  * Boehm et al. [2022] Kevin M Boehm, Emily A Aherne, Lora Ellenson, Ines Nikolovski, Mohammed Alghamdi, Ignacio Vázquez-García, Dmitriy Zamarin, Kara Long Roche, Ying Liu, Druv Patel, et al.  Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer.  _Nature cancer_ , 3(6):723–733, 2022.
  * Luo et al. [2022] Jia Luo, Rami S Vanguri, Andrew T Aukerman, Jacklynn V Egger, Christopher J Fong, Natally Horvat, Andrew Pagano, Jose Araujo-Filho, Luke Geneslaw, Hira Rizvi, et al.  Multimodal integration of radiology, pathology, and genomics for prediction of response to pd-1 blockade in patients with non–small cell lung cancer., 2022.
  * Chen et al. [2022] Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al.  Pan-cancer integrative histology-genomic analysis via multimodal deep learning.  _Cancer Cell_ , 40(8):865–878, 2022.
  * Lipkova et al. [2022] Jana Lipkova, Richard J Chen, Bowen Chen, Ming Y Lu, Matteo Barbieri, Daniel Shao, Anurag J Vaidya, Chengkuan Chen, Luoting Zhuang, Drew FK Williamson, et al.  Artificial intelligence for multimodal data integration in oncology.  _Cancer Cell_ , 40(10):1095–1110, 2022.
  * Lu et al. [2022a] Haoyu Lu, Qiongyi Zhou, Nanyi Fei, Zhiwu Lu, Mingyu Ding, Jingyuan Wen, Changde Du, Xin Zhao, Hao Sun, Huiguang He, et al.  Multimodal foundation models are better simulators of the human brain.  _arXiv preprint arXiv:2208.08263_ , 2022a.  https://arxiv.org/abs/2208.08263.
  * Xu et al. [2023] Peng Xu, Xiatian Zhu, and David A. Clifton.  Multimodal learning with transformers: A survey, 2023.
  * Gao et al. [2020] Jianliang Gao, Tengfei Lyu, Fan Xiong, Jianxin Wang, Weimao Ke, and Zhao Li.  MGNN: A multimodal graph neural network for predicting the survival of cancer patients.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1697–1700, 2020.
  * Acosta et al. [2022] Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol.  Multimodal biomedical ai.  _Nature Medicine_ , 28(9):1773–1784, 2022.
  * El Naqa and Rollison [2021] Issam El Naqa and Dana Rollison.  Moffitt Cancer Center: Why we are building the first machine learning department in oncology.  _The Cancer Letter_ , 47(7):5–10, 2021.
  * Steyaert et al. [2023] Sandra Steyaert, Marija Pizurica, Divya Nagaraj, Priya Khandelwal, Tina Hernandez-Boussard, Andrew J Gentles, and Olivier Gevaert.  Multimodal data fusion for cancer biomarker discovery with deep learning.  _Nature machine intelligence_ , 5(4):351–362, 2023.
  * Sabbula et al. [2023] Bhanusivakumar R Sabbula, David P Gasalberti, and Fatima Anjum.  Squamous cell lung cancer.  In _StatPearls [Internet]_. StatPearls Publishing, 2023.
  * Barsouk et al. [2023] Adam Barsouk, John Sukumar Aluru, Prashanth Rawla, Kalyan Saginala, and Alexander Barsouk.  Epidemiology, risk factors, and prevention of head and neck squamous cell carcinoma.  _Medical Sciences_ , 11(2):42, 2023.
  * Center [2024] Moffitt Cancer Center.  Squamous cell carcinoma survival rate, June 2024.  Available at: https://www.moffitt.org/cancers/squamous-cell-carcinoma/survival-rate/. Last accessed on Jun 2, 2024.
  * Campbell et al. [2018] Joshua D Campbell, Christina Yau, Reanne Bowlby, Yuexin Liu, Kevin Brennan, Huihui Fan, Alison M Taylor, Chen Wang, Vonn Walter, Rehan Akbani, et al.  Genomic, pathway network, and immunologic features distinguishing squamous carcinomas.  _Cell reports_ , 23(1):194–212, 2018.
  * Song et al. [2022] Qi Song, Ye Yang, Dongxian Jiang, Zhaoyu Qin, Chen Xu, Haixing Wang, Jie Huang, Lingli Chen, Rongkui Luo, Xiaolei Zhang, et al.  Proteomic analysis reveals key differences between squamous cell carcinomas and adenocarcinomas across multiple tissues.  _Nature Communications_ , 13(1):4167, 2022.
  * Kline et al. [2022] Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo.  Multimodal machine learning in precision health: A scoping review.  _npj Digital Medicine_ , 5(1):171, 2022.
  * Tripathi et al. [2022] Satvik Tripathi, Ethan Jacob Moyer, Alisha Isabelle Augustin, Alex Zavalny, Suhani Dheer, Rithvik Sukumaran, Daniel Schwartz, Brandon Gorski, Farouk Dako, and Edward Kim.  RadGenNets: Deep learning-based radiogenomics model for gene mutation prediction in lung cancer.  _Informatics in Medicine Unlocked_ , 33:101062, 2022.  ISSN 2352-9148.  doi:https://doi.org/10.1016/j.imu.2022.101062.
  * Soenksen et al. [2022] Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos Carballo, Liangyuan Na, Holly M Wiberg, Michael L Li, Ignacio Fuentes, and Dimitris Bertsimas.  Integrated multimodal artificial intelligence framework for healthcare applications.  _NPJ digital medicine_ , 5(1):149, 2022.
  * Pei et al. [2023] Xiangdong Pei, Ke Zuo, Yuan Li, and Zhengbin Pang.  A review of the application of multi-modal deep learning in medicine: Bibliometrics and future directions.  _International Journal of Computational Intelligence Systems_ , 16(1):44, 2023.
  * Sleeman IV et al. [2022] William C Sleeman IV, Rishabh Kapoor, and Preetam Ghosh.  Multimodal classification: Current landscape, taxonomy and future directions.  _ACM Computing Surveys_ , 55(7):1–31, 2022.
  * Jiang et al. [2022] Peng Jiang, Sanju Sinha, Kenneth Aldape, Sridhar Hannenhalli, Cenk Sahinalp, and Eytan Ruppin.  Big Data in basic and translational cancer research.  _Nature Reviews Cancer_ , 22(11):625–639, 2022.  doi:10.1038/s41568-022-00502-0.
  * Liu et al. [2022] T Liu, J Huang, T Liao, R Pu, S Liu, and Y Peng.  A hybrid deep learning model for predicting molecular subtypes of human breast cancer using multimodal data.  _Irbm_ , 43(1):62–74, 2022.
  * Hasanzadeh et al. [2022] Arman Hasanzadeh, Ehsan Hajiramezanali, Nick Duffield, and Xiaoning Qian.  MoReL: Multi-omics Relational Learning.  _arXiv preprint arXiv:2203.08149_ , 2022.
  * Waqas et al. [2024b] Asim Waqas, Aakash Tripathi, Sabeen Ahmed, Ashwin Mukund, Hamza Farooq, Matthew B Schabath, Paul Stewart, Mia Naeini, and Ghulam Rasool.  SeNMo: A Self-Normalizing Deep Learning Model for Enhanced Multi-Omics Data Analysis in Oncology.  _arXiv e-prints_ , pages arXiv–2405, 2024b.  URL https://doi.org/10.48550/arXiv.2405.08226.
  * Li and Zhou [2022] Simin Li and Baosen Zhou.  A review of radiomics and genomics applications in cancers: the way towards precision medicine.  _Radiation Oncology_ , 17(1):1–10, 2022.
  * Brancato et al. [2022] Valentina Brancato, Carlo Cavaliere, Nunzia Garbino, Francesco Isgrò, Marco Salvatore, and Marco Aiello.  The relationship between radiomics and pathomics in glioblastoma patients: Preliminary results from a cross-scale association study.  _Frontiers in Oncology_ , 12:1005805, 2022.
  * Zhang et al. [2022] Zhenhua Zhang, Enqi Chen, Xueyuan Zhang, Jing Yang, Xianqi Wang, PinZhen Chen, Meng Zeng, Mingshan Du, Senlin Xu, Zhiqing Yang, Fei Ren, and Wei Chen.  Multi-modal fusion of radiomics and pathomics to predict the survival of pancreatic cancer patients based on asymmetric twinning information interaction network.  _SSRN Electronic Journal_ , 2022.  http://dx.doi.org/10.2139/ssrn.4260135.
  * Uno et al. [2011] Hajime Uno, Tianxi Cai, Michael J Pencina, Ralph B D’Agostino, and Lee-Jen Wei.  On the c-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data.  _Statistics in medicine_ , 30(10):1105–1117, 2011.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  _arXiv preprint arXiv:2010.11929_ , 2021.  URL https://openreview.net/forum?id=YicbFdNTTy.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Yang et al. [2022] Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al.  A large language model for electronic health records.  _npj Digital Medicine_ , 5(1):194, 2022.
  * Azizi et al. [2022] Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Patricia MacWilliams, S Sara Mahdavi, Ellery Wulczyn, et al.  Robust and efficient medical imaging with self-supervision.  _arXiv preprint arXiv:2205.09723_ , 2022.  https://arxiv.org/abs/2205.09723.
  * Chen et al. [2024] Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al.  Towards a general-purpose foundation model for computational pathology.  _Nature Medicine_ , 30(3):850–862, 2024.
  * Waqas et al. [2023] Asim Waqas, Marilyn M Bui, Eric F Glassy, Issam El Naqa, Piotr Borkowski, Andrew A Borkowski, and Ghulam Rasool.  Revolutionizing digital pathology with the power of generative artificial intelligence and foundation models.  _Laboratory Investigation_ , page 100255, 2023.
  * Wasserthal et al. [2023] Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth.  TotalSegmentator: robust segmentation of 104 anatomical structures in CT images, 2023.  https://arxiv.org/abs/2303.12961.
  * Qiu et al. [2023] Jianing Qiu, Lin Li, Jiankai Sun, Jiachuan Peng, Peilun Shi, Ruiyang Zhang, Yinzhao Dong, Kyle Lam, Frank P. W. Lo, Bo Xiao, Wu Yuan, Dong Xu, and Benny Lo.  Large AI Models in Health Informatics: Applications, Challenges, and the Future, 2023.  https://arxiv.org/abs/2303.11568 and https://github.com/Jianing-Qiu/Awesome-Healthcare-Foundation-Models.
  * Wornow et al. [2023] Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A. Pfeffer, Jason Fries, and Nigam H. Shah.  The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs, 2023.  https://arxiv.org/abs/2303.12961.
  * Stamile et al. [2021] Claudio Stamile, Aldo Marzullo, and Enrico Deusebio.  _Graph Machine Learning: Take graph data to the next level by applying machine learning techniques and algorithms_.  Packt Publishing Ltd, 2021.
  * Tripathi et al. [2024a] Aakash Tripathi, Asim Waqas, Kavya Venkatesan, Yasin Yilmaz, and Ghulam Rasool.  Building flexible, scalable, and machine learning-ready multimodal oncology datasets.  _Sensors_ , 24(5):1634, 2024a.
  * Hinkson et al. [2017] Izumi V. Hinkson, Tanja M. Davidsen, Juli D. Klemm, Ishwar Chandramouliswaran, Anthony R. Kerlavage, and Warren A. Kibbe.  A comprehensive infrastructure for big data in cancer research: Accelerating cancer research and precision medicine.  _Frontiers in Cell and Developmental Biology_ , 5, 2017.  ISSN 2296-634X.  doi:10.3389/fcell.2017.00083.  URL https://datacommons.cancer.gov.
  * Ellis et al. [2013] Matthew J. Ellis, Michael Gillette, Steven A. Carr, Amanda G. Paulovich, Richard D. Smith, Karin K. Rodland, R. Reid Townsend, Christopher Kinsinger, Mehdi Mesri, Henry Rodriguez, and Daniel C. Liebler.  Connecting Genomic Alterations to Cancer Biology with Proteomics: The NCI Clinical Proteomic Tumor Analysis Consortium.  _Cancer Discovery_ , 3(10):1108–1112, 10 2013.  ISSN 2159-8274.
  * Tonsing-Carter et al. [2023] Eva Tonsing-Carter, Rachana Agarwal, Cindy W. Kyi, Julyann Perez-Mayoral, Conrado T. Soria, and Jean Claude Zenklusen.  Abstract 4681: Human Cancer Models Initiative (HCMI): A community resource of next-generation cancer models and associated data.  _Cancer Research_ , 83(7_Supplement):4681–4681, 04 2023.  ISSN 0008-5472.  doi:10.1158/1538-7445.AM2023-4681.  URL https://doi.org/10.1158/1538-7445.AM2023-4681.
  * Tomczak et al. [2015] Katarzyna Tomczak, Patrycja Czerwińska, and Maciej Wiznerowicz.  Review The Cancer Genome Atlas (TCGA): An immeasurable source of knowledge.  _Contemporary Oncology_ , 2015(1):68–77, 2015.
  * Paul and Komarova [2021] Doru Paul and Natalia L Komarova.  Multi-scale network targeting: A holistic systems-biology approach to cancer treatment.  _Progress in biophysics and molecular biology_ , 165:72–79, 2021.
  * Lu et al. [2022b] Qiuhao Lu, Dejing Dou, and Thien Nguyen.  Clinicalt5: A generative language model for clinical text.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 5436–5443, 2022b.
  * Tripathi et al. [2024b] Aakash Tripathi, Asim Waqas, Yasin Yilmaz, and Ghulam Rasool.  Honeybee: A scalable modular framework for creating multimodal oncology datasets with foundational embedding models.  _arXiv preprint arXiv:2405.07460_ , 2024b.
  * Gadermayr and Tschuchnig [2024] Michael Gadermayr and Maximilian Tschuchnig.  Multiple instance learning for digital pathology: A review of the state-of-the-art, limitations & future potential.  _Computerized Medical Imaging and Graphics_ , page 102337, 2024.
  * Ilse et al. [2018] Maximilian Ilse, Jakub Tomczak, and Max Welling.  Attention-based deep multiple instance learning.  In _International conference on machine learning_ , pages 2127–2136. PMLR, 2018.
  * Xie et al. [2022b] Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji.  Self-supervised learning of graph neural networks: A unified review.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(2):2412–2429, 2022b.
  * Hamilton et al. [2017] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30, 2017.
  * Veličković et al. [2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ , 2017.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ , 2016.
  * Xu et al. [2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=ryGs6iA5Km.
  * Lifelines [2023] Lifelines.  Concordance Index for Survival Analysis, 2023.  URL https://lifelines.readthedocs.io/en/latest/lifelines.utils.html#lifelines.utils.concordance_index.
  * Fedorov et al. [2021] A. Fedorov, W. Longabaugh, D. Pot, D. Clunie, S. Pieper, R. Lewis, H. Aerts, A. Homeyer, M. Herrmann, U. Wagner, T. Pihl, K. Farahani, and R. Kikinis.  Nci imaging data commons.  _International Journal of Radiation Oncology*Biology*Physics_ , 111(3, Supplement):e101, 2021.  ISSN 0360-3016.  doi:10.1016/j.ijrobp.2021.07.495.  2021 Proceedings of the ASTRO 63rd Annual Meeting.
  * Grossman et al. [2016] Robert L. Grossman, Allison P. Heath, Vincent Ferretti, Harold E. Varmus, Douglas R. Lowy, Warren A. Kibbe, and Louis M. Staudt.  Toward a shared vision for cancer genomic data.  _New England Journal of Medicine_ , 375(12):1109–1112, 2016.  doi:10.1056/nejmp1607591.
  * Thangudu et al. [2020] Ratna Rajesh Thangudu, Paul A. Rudnick, Michael Holck, Deepak Singhal, Michael J. MacCoss, Nathan J. Edwards, Karen A. Ketchum, Christopher R. Kinsinger, Erika Kim, and Anand Basu.  Abstract LB-242: Proteomic Data Commons: A resource for proteogenomic analysis.  _Cancer Research_ , 80(16_Supplement):LB–242, 08 2020.  ISSN 0008-5472.  doi:10.1158/1538-7445.AM2020-LB-242.  URL https://pdc.cancer.gov.
  * Amazon Web Services [a] Amazon Web Services.  AWS Glue, a.  Available online: https://aws.amazon.com/glue/. (accessed on 1 March 2023).
  * Amazon Web Services [b] Amazon Web Services.  Data Catalog and crawlers in AWS Glue, b.  Available online: https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html. (accessed on 1 March 2023).
  * Amazon Web Services [c] Amazon Web Services.  Amazon S3, c.  Available online: https://aws.amazon.com/s3/. (accessed on 1 March 2023).
  * Amazon Web Services [d] Amazon Web Services.  Amazon RDS, d.  Available online: https://aws.amazon.com/rds/. (accessed on 1 March 2023).
  * Amazon Web Services [e] Amazon Web Services.  AWS Lake Formation.  Online, e.  Available online: https://aws.amazon.com/lake-formation/. (accessed on 1 March 2023).
  * Amazon Web Services [f] Amazon Web Services.  Amazon Athena, f.  Available online: https://aws.amazon.com/athena/. (accessed on 1 March 2023).
  * Amazon Web Services [g] Amazon Web Services.  Amazon Redshift, g.  Available online: https://aws.amazon.com/redshift/. (accessed on 1 March 2023).
  * Stewart et al. [2019] Paul A Stewart, Eric A Welsh, Robbert JC Slebos, Bin Fang, Victoria Izumi, Matthew Chambers, Guolin Zhang, Ling Cen, Fredrik Pettersson, Yonghong Zhang, et al.  Proteogenomic landscape of squamous cell lung cancer.  _Nature communications_ , 10(1):1–17, 2019.
  * Waqas et al. [2022] Asim Waqas, Hamza Farooq, Nidhal C Bouaynaya, and Ghulam Rasool.  Exploring Robust Architectures for Deep Artificial Neural Networks.  _Communications Engineering_ , 1(1):46, 2022.