[1] [1]Maryamalsadat Mahootiha

1]organization=The Intervention Centre, addressline=Oslo University Hospital,
city=Oslo, postcode=0372, country=Norway 2]organization=Faculty of Medicine,
addressline=University of Oslo, city=Oslo, postcode=0372, country=Norway
3]organization=Department of Electronic Systems, addressline=Norwegian
University of Science and Technology, city=Trondheim, country=Norway

# Multimodal Deep Learning for Personalized Renal Cell Carcinoma Prognosis:
Integrating CT Imaging and Clinical Data

Maryamalsadat Mahootiha<EMAIL_ADDRESS>Hemin Ali Qadir  Jacob Bergsland
Ilangko Balasingham  [  [  [

###### Abstract

Background and Objective: Renal cell carcinoma represents a significant global
health challenge characterized by a low survival rate. The aim of this
research was to devise a comprehensive deep-learning model capable of
predicting survival probabilities in patients with renal cell carcinoma by
integrating CT imaging and clinical data and addressing the limitations
observed in prior studies. The aim is to facilitate the identification of
patients requiring urgent treatment.
Methods: The proposed framework comprises three modules: a 3D image feature
extractor, clinical variable selection, and survival prediction. The feature
extractor module, based on the 3D CNN architecture, predicts the ISUP grade of
renal cell carcinoma tumors linked to mortality rates from CT images. A
selection of clinical variables is systematically chosen using the Spearman
score and random forest importance score as criteria. A deep learning based
network, trained with discrete LogisticHazard-based loss, performs the
survival prediction. Nine distinct experiments are performed, with varying
numbers of clinical variables determined by different thresholds of the
Spearman and importance scores.
Results: Our findings demonstrate that the proposed strategy surpasses the
current literature on renal cancer prognosis based on CT scans and clinical
factors. The best-performing experiment yielded a concordance index of 0.84
and an area under the curve value of 0.8 on the test cohort, which suggests
strong predictive power.
Conclusions: The multimodal deep-learning approach developed in this study
shows promising results in estimating survival probabilities for renal cell
carcinoma patients using CT imaging and clinical data. This may have potential
implications in identifying patients who require urgent treatment, potentially
improving patient outcomes. The code created for this project is available for
the public on: GitHub

###### keywords:

RadiomicsRenal Cell CarcinomaDeep Learning Survival Analysis Cancer Prognosis
ISUP Grading

##  1 Introduction

###  1.1 Overview

Renal cell carcinoma (RCC) is a prevalent malignancy in adults and constitutes
around 90% of all kidney tumors (Saad et al., 2019). RCC develops in the
tubules that filter blood and produce urine in the kidney (Saad et al., 2019).
If not detected and treated early, RCC can metastasize to other organs, such
as lungs and bones, and become life-threatening (Sung et al., 2021). The
global incidence of RCC has been rising, which may be attributable to the easy
availability of more improved diagnostic modalities, greater use of medical
imaging, and changes in lifestyle factors (Siegel et al., 2020; Znaor et al.,
2015). Treating RCC early is crucial for improving patient outcomes and
enhancing both survival rates and quality of life (Znaor et al., 2015).

Survival analysis is a statistical technique used to investigate the time
duration until a critical event occurs, such as death or disease recurrence,
and is widely used in oncology. The analysis involves examining time-to-event
data to estimate the probability of an event occurring over a specified period
while accounting for censoring. This statistical technique allows for the
inclusion of individuals who did not experience the event of interest by the
end of the study period (Lee and Wang, 2003).

Survival analysis is vital for RCC patients as it informs treatment decisions
and enables clinicians to determine the optimal course of action, including
therapy type, the intensity of treatment, and the need for palliative care or
supportive measures (Hui et al., 2019). Radiological data is essential for
cancer survival analysis and prognosis, revealing tumor features,
heterogeneity, therapy planning, and response evaluation. Clinicians can use
this data to improve patient outcomes and survival prospects (Lambin et al.,
2012). Clinical experts may make erroneous predictions or misinterpret medical
images, which can result in incorrect prognosis and treatment decisions. In
fact, approximately 20 million radiology reports are estimated to contain
clinically significant errors annually (Brady, 2017). Furthermore, there may
be a shortage of expert radiologists in certain regions or healthcare
settings. Therefore, the implementation of artificial intelligence (AI)
technologies can potentially aid in addressing these issues (Liu et al.,
2019).

AI has the potential to improve the accuracy and efficiency of medical image
analysis, particularly through the utilization of convolutional neural
networks (CNN), which can capture patterns and features that may not be easily
detectable by human observers (Coppola et al., 2021). These algorithms can
analyze large amounts of data quickly and accurately, reducing the potential
for human error and improving diagnostic accuracy (n Montero et al., 2021).
The use of AI in survival analysis has also shown promise since it has the
potential to enhance the precision of prognostic models and facilitate
personalized treatment (Wang et al., 2019a).

This study seeks to devise a multimodal AI-driven algorithm capable of
predicting personalized survival probabilities utilizing CT images and
clinical data, addressing challenges such as potential inaccuracies by
clinicians and the scarcity of experts in radiological image interpretation.
Our objective is to utilize a multimodal survival analysis strategy to achieve
enhanced precision in forecasting survival probabilities. To investigate this,
we classify RCC tumors in CT images according to the International Society of
Urological Pathology (ISUP) grading (Srigley et al., 2013) system. This system
serves as a means to evaluate cancer severity by examining the morphological
characteristics of tumor cells under microscopic observation, and it is
closely associated with mortality rates (Samaratunga et al., 2014). After the
classification process, radiomic features are extracted and subsequently
incorporated as input factors within our proposed survival model. Additional
inputs encompass pertinent clinical variables pertaining to individual
patients. By integrating radiomic features and clinical variables, we endeavor
to estimate survival probabilities employing a methodology that is non-linear
and non-proportional, offering a more robust, realistic, and accurate survival
estimation.

###  1.2 Related Work

In statistics, the Cox proportional hazards (CPH) model (Cox, 1972) is the
gold standard for modeling survival analysis using censored observations. CPH
is limited by its linear nature, which fails to capture non-linear
relationships between input data and the risk of an event occurring, e.g.,
death. However, the advent of AI and deep learning (DL) has opened new avenues
for modeling survival analysis, allowing for the exploration of complex, non-
linear relationships. DL-based models, such as Cox-nnet (Ching et al., 2018)
and DeepSurv (Katzman et al., 2018), have been developed to address the
limitations of the CPH model and enable the identification of novel prognostic
factors. But they still face a fundamental constraint imposed by the
proportional hazards assumption of CPH. CPH assumes that the effect of a
patient’s covariates on the risk of death remains constant over time,
resulting in proportional predictions for all patients. However, this
assumption may not be reflective of the true clinical situation, leading to
survival curves that do not intersect.

Recent developments in statistical modeling have led to innovative solutions
to address the limitations of the CPH model in survival analysis. Two
important methods that have been proposed to address the linearity and
proportionality constraints of CPH are multivariate time-to-event logistic
regression (MTLR) (Fotso, 2018), and Nnet-survival (Gensheimer and Narasimhan,
2019). MTLR is a method that extends logistic regression to time-to-event data
by modeling the joint probability of multiple events. This approach allows for
the incorporation of time-dependent covariates and can handle non-proportional
hazards, making it a valuable tool for survival analysis. Nnet-survival, on
the other hand, involves calculating the discrete conditional hazard rate at
each time period. This concept has been established for several decades (Brown
et al., 1997) and was recently applied to contemporary DL approaches, leading
to the development of Nnet-survival. This approach makes it possible to have
non-proportional hazard probability curves for different patients.

Multimodal DL (Ngiam et al., 2011), a framework that leverages DL techniques
to learn from multiple data modalities, including tabular, images, and audio,
can be particularly useful in medical applications. With the availability of
diverse data types such as clinical information, radiological images, and
medication records, the application of multimodal DL can help capture complex
relationships between the model inputs and outputs.

Previous studies have employed various approaches to conduct survival
analysis, focusing on using radiological images or integrating radiological
images with clinical variables to enhance survival estimation. Mukherjee et
al. (2020) developed a shallow CNN in conjunction with Cox loss to predict the
prognosis of lung cancer patients using computed tomography (CT) image data
alone. Wang et al. (2019b) presented a CNN autoencoder-based survival model
incorporating Cox loss for predicting recurrence in patients with high-grade
serous ovarian cancer, relying solely on CT scans. Wu et al. (2021) developed
a regression-based survival model for non-small cell lung cancer patients,
effectively integrating imaging and clinical data to enhance the accuracy of
survival predictions by employing the mean squared error (MSE) loss function.
Zhang et al. (2020) introduced a risk prediction model for assessing overall
survival in gastric cancer patients, incorporating both CT images and clinical
variables as inputs and utilizing a specialized loss function. Zhong et al.
(2020) presented a CNN-based model using Cox survival loss to predict survival
outcomes in patients diagnosed with stage T3N1M0 nasopharyngeal carcinoma
using magnetic resonance (MR) imaging and clinical variables. Lastly, Chaddad
et al. (2017) explored the potential of radiomic features and clinical
variables in predicting the survival group of lung cancer patients. The
authors employed image analysis techniques, rather than DL methods, to extract
radiomic features, and utilized a random forest classifier.

###  1.3 Our Contributions

This study differs from the previous study by presenting a novel multimodal
approach to predicting nonlinear and non-proportional survival curves for
patients afflicted with RCC by employing both CT images and clinical data.
Moreover, our study is distinguished as the first to systematically explore
the impact of varying combinations of clinical variables and CT images on
survival prediction performance, thereby shedding light on the importance of
selecting appropriate data sources for accurate survival estimations.

Our proposed survival model offers several notable advantages over previous
studies, which can be delineated in the following manner: 1) By incorporating
3D inputs and 3D convolutional layers, our model retains comprehensive
information from the data, mitigating any potential loss of critical details
pertaining to the interface between tumor and healthy tissue. 2) Our
methodology enables the forecasting of non-proportional survival analyses,
producing outcomes that are more relevant to clinical situations. 3) In
comparison to previously reported literature, our survival model demonstrates
superior performance indices, highlighting its efficacy. 4) A key feature of
the proposed model is its ability to generate individualized survival curves
for each patient, allowing for a more personalized assessment. 5) To elucidate
the nuances of survival model performance, we conduct an analysis of varying
combinations of clinical variables and CT images, providing valuable insights
into the optimization of survival estimation. 6) In addition to conventional
metrics for evaluating survival models, we also employ the violin diagram to
visualize the distribution of survival probabilities in our survival model’s
outputs.

##  2 Methods

Fig. 1 illustrates our entire approach for modeling survival analysis. It
takes as inputs two data modalities: 1) CT volumes and 2) clinical variables.
Motivated by the success of CNNs in image analysis and cancer prognosis, we
present a CNN-based architecture for CT image feature extraction relevant to
prognosis in our methodology. We utilize 3D CNNs to extract features from the
three dimensions within the tumor volume motivated by Zhu et al. (2018).

Figure 1: The comprehensive framework presented herein is composed of three
primary modules. Module 1 encompasses feature extraction from CT volumes,
wherein features are derived through the classification of CT images based on
ISUP grades. Subsequently, a fully connected layer consisting of 1000 neurons
is employed to reduce the radiomic feature size from 2560. Module 2 focuses on
the judicious selection of clinical variables, which are merged with CT image
features utilizing the Spearman correlation score and random forest importance
score. Module 3 pertains to the survival network, which accepts input from
three sources: CT image features, clinical variables, and a combination of
both. The survival network’s output consists of survival probabilities for 15
discrete time intervals, which are subsequently converted into 1500 time
points through interpolation. This process facilitates the visualization of
continuous survival curves for individual patients.

Subsequently, we integrate clinical information with the CT image features for
survival analysis. Our method comprises three modules: (1) CT image feature
extraction, (2) clinical variables selection, and (3) survival prediction.
Within the scope of our scholarly investigation, the feature extractor network
and the survival network are subjected to independent training processes as
opposed to being trained concurrently.

###  2.1 Radiomic Feature Extraction from CT Volumes

We suggest classifying RCC tumors in CT images into ISUP grades (1, 2, 3, and
4) to obtain radiomic features relevant to prognosis. The CT volumes go
through a 3D CNN feature extractor network to pull out these features. After
that, we can integrate the clinical variables with the extracted radiomic
features. We choose ISUP grade for classification as it has been shown to have
a strong correlation to tumor recurrence, metastasis, and mortality (Warren
and Harrison, 2018). Higher ISUP grades are indicative of a worse prognosis
and higher mortality rate, whereas lower grades are associated with a better
prognosis, and lower mortality rate (Costantini et al., 2021).

For the feature extractor network, the classifier, in our study, we select
EfficientNet (Tan and Le, 2019), which is a state-of-the-art CNN architecture
developed by Google researchers for image classification. This architecture
employs the compound coefficient method to scale up models efficiently. The
largest model, EfficientNet B7, achieved the best performance compared to
other variants. The EfficientNet layers utilize MBConv (Sandler et al., 2018),
a type of convolutional block that can capture complex features in images
while using fewer parameters and less computation compared to traditional
convolutional blocks.

To accommodate three-dimensional (3D) image data such as CT volumes, we adapt
the exact architecture of EfficientNet B7 and transform it into a 3D CNN
model. By doing so, features are extracted in all three-dimensional directions
within the tumor volume, taking the third-dimensional spatial information into
account. Hence, the employed feature extraction network operates in a three-
dimensional (3D) domain, wherein the input comprises image volumes that have
undergone preprocessing and concatenated with the annotations of tumor
segmentations. This network classifies the RCC tumors into four ISUP grades.
Our group has undertaken a separate, comprehensive study focused on the
classification of RCC according to ISUP grading systems (Mahootiha et al.,
2022). The architecture encompasses a combination of convolutional layers,
MBConv layers, an Adaptive Average Pooling layer, and a series of fully
connected (FC) layers, respectively.

The Adaptive Average Pooling layer, which acts as a bridge between CNN and FC
layers, can be used for feature extraction. This layer reduces the number of
parameters and computational complexity required for classification while
preserving crucial information about image features (Russakovsky et al.,
2015). We extract the outputs from the Adaptive Average Pooling layer to
create feature vectors for every patient. Subsequently, the output of the
Adaptive Average Pooling layer is flattened, and the resulting image features
are converted to feature vectors. The initial feature vector dimension is
2560, and our objective is to reduce it to 1000 to streamline integration with
clinical variables. We attempt to achieve this reduction by employing an FC
layer with 2560 input features and 1000 output features. These vectors are
then saved as a CSV file for feeding to the survival network. Following the
feature extraction and storage in a CSV file, normalization is performed to
standardize the data based on the mean and standard deviation.

###  2.2 Clinical Variables Selection

Our objective is not to incorporate all clinical variables with CT image
features for the purpose of survival prediction. Rather, we intend to explore
the feasibility of using a smaller subset of variables (those that are more
relevant to prognosis) in conjunction with CT image features to achieve
improved results in survival prediction. To this end, we aim to evaluate
various combinations of clinical variables. In order to identify the most
relevant clinical variables for predicting survival times, we employ two well-
established methods: (1) the Spearman correlation score (Pirie, 2006) and (2)
the importance score of a random forest regressor (Wehenkel et al., 2018).
These approaches help us to identify the most informative clinical variables
to include in our survival model and achieve more accurate predictions of
survival outcomes for each patient.

Spearman’s rank correlation coefficient is a non-parametric statistical method
that quantifies the strength and direction of the relationship between two
variables. It is primarily used to assess the existence of a monotonic
association between two variables and is less sensitive to non-linear
relationships and non-normal distributions compared to the parametric Pearson
correlation coefficient. We calculate the Spearman correlation coefficients
between clinical variables and survival times, forming a correlation matrix.
This matrix represents the pairwise correlation coefficients between each
clinical variable and survival times. The correlation coefficient values vary
from -1 to 1, with -1 representing a strong negative connection, 1
representing a strong positive connection, and 0 representing no correlation.

On the other hand, random forest regression can generate an importance score
for each predictor variable. To acquire importance scores, we develop a random
forest model consisting of 100 decision trees to estimate survival times based
on clinical variables. The importance scores originate from the average
decrease in the model’s prediction error due to each feature, considering all
the random forest’s decision trees. Subsequently, the clinical variables are
ranked according to their importance scores to discern the most influential
variables for survival time prediction. Higher importance scores signify a
more substantial impact of a variable on the model’s predictive performance.

###  2.3 Modeling Survival Estimation

In this subsection, we focus on the critical aspects of modeling survival
estimation, an essential component of our proposed method. We have organized
this subsection into three parts: 1) survival network, where we describe the
architecture and design choices for the survival network, which is responsible
for estimating survival probabilities; 2) input to the survival network, which
details the features and data used as input to the network, such as clinical
variables and radiomic features extracted from the 3D CNN feature extractor;
3) loss function for modeling survival estimation, in which we discuss the
choice of loss function employed to optimize the survival network.

####  2.3.1 Survival Network

The survival network, shown in Fig. 1, consists of three FC layers, comprising
500, 100, and 15 neurons, respectively. As our model is a discrete-time
survival model, the final layer contains 15 neurons representing survival
probabilities for 15 distinct time intervals. The network utilizes a rectified
linear unit (ReLU) activation function in the intermediate layers and a
sigmoid activation function in the last layer. In an effort to enhance the
generalization capabilities of the model, a dropout rate of 0.3 is
incorporated, accompanied by the implementation of batch normalization
subsequent to the initial two FC layers. Subsequently, linear interpolation
with 100 points is employed to transform the outputs into a set of 1500
values, enabling the generation of continuous survival curves for patients. We
achieve the optimal architecture through a grid search of hyperparameters to
find the best evaluation metrics for survival analysis.

####  2.3.2 Input to the Survival Network

The inputs to the survival network are derived from one of three sources: CT
image features, clinical variables, or a combination of CT image features and
clinical variables. In this study, we do a series of nine experiments, each
using one of these three sources for survival prediction. In Section 3.4, a
full explanation of these experiments will be given.

####  2.3.3 Loss Function for Modeling Survival Estimation

We adapt our survival model loss function based on discrete logistic hazards
similar to the loss used in Nnet survival (Gensheimer and Narasimhan, 2019) to
predict survival probabilities over M days (weeks, months, or years) which M
is the maximum follow-up period. In order to employ the discretized hazard
function, it is essential to convert continuous survival times into discrete
intervals. To achieve this, a judicious selection of appropriate time
intervals is undertaken to discretize the continuous survival times, with the
preferred choice being equidistant intervals. Subsequently, each observed
survival time is allocated to its respective time interval, effectively
transforming the continuous data into a discrete format. We developed a loss
function that used a vectorized form of likelihoods for censored and
uncensored patients. The loss function is given by:

$\operatorname{L}=\mathrm{-}\sum_{x=1}^{p}\sum_{i=1}^{n}\left(\begin{array}[]{l}\ln\left(1+\operatorname{surv}_{s}(x)(i)\cdot\left(\operatorname{surv}_{\text{pred
}}(x)(i)-1\right)\right)\\\
+\ln\left(1-\operatorname{surv}_{f}(x)(i)\cdot\operatorname{surv}_{\text{pred
}}(x)(i)\right)\end{array}\right),$

where $p$ denotes the number of patients in a batch, and $n$ represents the
number of discrete time intervals (15). $\operatorname{surv_{pred}}(x)(i)$
signifies the predicted outcome of the survival model for patient $x$ at time
interval $i$, which can be either 0 for a patient who died during interval $i$
or 1 for a patient who remained alive in interval $i$. Each patient’s death or
censoring time, $t$, is determined based on the ground truth survival time
given in a dataset. The ground truth vectors $\operatorname{surv_{s}}$ and
$\operatorname{surv_{f}}$ for the survival model are of length $n$ for every
patient. Vector $\operatorname{surv_{s}}$ corresponds to the time intervals
when the patient survived, while vector $\operatorname{surv_{f}}$ denotes the
specific time interval when the death occurred. For uncensored patients in the
time interval $i$:

$\displaystyle\operatorname{surv}_{s}(x)(i)=\begin{cases}1,&\text{ if
}t_{x}\geq t_{i}\\\ 0,&\text{ otherwise }\end{cases}$
$\displaystyle\operatorname{surv}_{f}(x)(i)=\begin{cases}1,&\text{ if
}t_{i-1}\leq t_{x}<t_{i}\\\ 0,&\text{ otherwise }\end{cases}$

for censored patients in the time interval $i$:

$\operatorname{surv}_{s}(x)(i)=\begin{cases}1,&\text{ if
}t_{x}\geq\frac{1}{2}\left(t_{i-1}+t_{i}\right)\\\ 0,&\text{ otherwise
}\end{cases}$

and

$\operatorname{surv}_{f}(x)(i)=0.$

The dot product within the loss function assesses the similarities between the
predicted vector and the ground truth vector. We trained the survival networks
with the help of pycox v0.2.0.3 library 111https://github.com/havakv/pycox.

##  3 Experimental Setup

In this section, we describe the experimental setup employed in our study,
which is divided into four main parts: experimental dataset, training the 3D
CNN feature extractor network, training the survival network, and the
experiments conducted. First, we present the datasets used in our study and
discuss their characteristics, source, and any preprocessing steps undertaken.
Next, we outline the process of training the 3D CNN feature extractor network,
followed by the training of the survival network. Finally, we describe the
experiments conducted. A comprehensive experimental setup ensures the
reproducibility of our results and allows for a fair comparison with other
studies in the field.

###  3.1 Experimental Dataset

The selection of appropriate datasets and their preparation plays a crucial
role in the evaluation of our proposed method. In this subsection, we provide
an overview of the dataset used in our experiments and the steps taken to
prepare the data for our study. We have divided this subsection into three
parts: the KiTS21 dataset, dataset splitting, and clinical data preparation.
First, we discuss the KiTS21 dataset, its characteristics, and its source.
Next, we describe the dataset-splitting process, explaining the rationale
behind the chosen method and the proportions used for training, validation,
and testing. Finally, we detail the clinical data preparation, including any
necessary preprocessing and data normalization procedures.

####  3.1.1 KiTS21 Dataset

We used the KiTS21 (Heller et al., 2021) dataset to train and test our
proposed framework. The dataset comprises 300 patients who underwent either
partial or complete nephrectomy for suspected kidney cancer between 2010 and
2020 at the M Health Fairview or Cleveland Clinic medical facility and
includes both clinical data and CT scans with manually annotated kidneys and
tumors (ground-truth labels). The primary objective of collecting this dataset
was to apply segmentation algorithms.

We selected this dataset for its comprehensive clinical information, precise
annotations, and ample subject numbers. The dataset contains three files,
including CT scan volumes (NIFTI format), annotation volumes (NIFTI format),
and clinical data (JSON format). The annotation volumes consist of manual
segmentations of the kidneys, tumor(s), and cyst(s). In this study, we used 41
clinical variables from this JSON file. All critical clinical information,
such as pathology results, is included in this file (Heller et al., 2019).
Notably, this data was originally obtained from the Cancer Imaging Archive in
DICOM format, while the clinical data was provided in a single CSV file.

####  3.1.2 Dataset Splitting

To train the classify network that can be used as the radiomic feature
extraction for survival prediction, we excluded 56 patients with empty ISUP
grade values from the original dataset. The remaining dataset contained 244
patients, of which 32 had dead events and 212 had censored time. The maximum
observation time was 3000 days (which refers to the M variable in Section
2.3.3), and the median observation time was 644 days. We performed three-fold
cross-validation for the ISUP grading classification to create three different
subsets for training, validation, and testing. The division of the dataset
into three folds was based on the number of deceased and censored patients to
ensure that each subset contained the same proportion of deceased individuals.
Each fold included 57% of the total dataset for training, 10% for validation,
and 33% for testing. The training subset had 10% of patients who died, the
validation subset had 33%, and the test subset had 13%. After dividing the
dataset into three folds, we increased the number of samples in each train and
validation subset by doing multiple augmentations (discussed in 3.2.1).

The optimal fold for the classification model was determined based on the
F1-score, as delineated in 3.2.3. This selected fold was subsequently employed
for training, validation, and testing within the survival network, excluding
the utilization of augmented samples. Two distinct networks were employed for
ISUP grade classification and survival analysis; however, they were trained
using identical subjects within the training, validation, and test datasets.
This approach was adapted to preclude the introduction of the classification
network’s training data as the validation or test dataset for the survival
analysis network, thereby avoiding the overestimation of the survival analysis
network’s performance due to heightened accuracy in detecting ISUP grades
within the training dataset.

####  3.1.3 Clinical Data Preparation

The clinical data used in training the survival network consisted of 38
variables classified into two categories: continuous numerical and
categorical. In order to facilitate their usage in the survival model, the
categorical variables were transformed into discrete numerical values, such as
gender. In contrast, the continuous numerical variables, such as pathologic
size, were normalized based on the mean and standard deviation to facilitate
effective interpretation by the survival model.

###  3.2 Training the 3D CNN Feature Extractor

In this subsection, we elaborate on the process of training the 3D CNN feature
extractor, a critical component in our proposed method. This subsection is
divided into three parts: 1) preprocessing of CT image volumes, which is a
necessary step before training the 3D CNN feature extractor to guarantee
consistent input data and enhance the network’s performance; 2) training
details of the classifier, encompassing aspects such as the chosen loss
function, number of epochs, optimizer, and learning rate; 3) best fold
selection for radiomic feature extraction, a crucial step following the
training of the 3D CNN feature extractor, which involves selecting the optimal
fold to ensure the highest quality features for the subsequent survival
network.

####  3.2.1 Preprocessing of CT Image Volumes

Before commencing the preprocessing phase for CT volumes, image augmentations
were implemented as a strategy to address the inherent imbalance in the
dataset, as well as the paucity of training samples. A combination of
positional augmentations, such as flipping, rotation, and affine
transformations, along with noise augmentations, including Gaussian noise,
Gibbs noise, and space spike noise, were employed to enhance the diversity and
generalizability of the dataset. Before the ISUP grade classification, image
preprocessing is applied to improve the quality of the input images and their
radiomic features for better interpretation of the input (Pérez-García et al.,
2021; Akar et al., 2017). As recommended in the MIT
challenge222http://6.869.csail.mit.edu/fa17/miniplaces.html, all volumes were
resized to $128\times 128\times 128$. We also resampled the volumes based on
one millimeter isotropic voxel size, which has been recommended as a standard
voxel size by previous studies in medical imaging (Alom et al., 2019;
Vankdothu and Hameed, 2022). Additionally, all volumes were reoriented to the
RAS (Right, Anterior, and Superior) orientation, which is the most commonly
used orientation in medical images (Alom et al., 2019; Vankdothu and Hameed,
2022; Litjens et al., 2017). We utilized intensity normalization based on the
Z-score in medical imaging (Pérez-García et al., 2021; Tustison et al., 2010).
For kidney image and tumor segmentation, identical image preprocessing steps
were employed, with the exception that intensity normalization was not applied
for tumor segmentation.

As part of our image preprocessing pipeline, we employed a concatenation step
to enhance the performance of our 3D EfficientNet-B7 model in identifying
kidney tumors. Specifically, we combined the extracted kidney images with
their corresponding manual tumor segmentations to enable the model to focus on
the surface patterns of the tumors (Akar et al., 2017). This image
concatenation approach serves to enrich the input volume with additional
information pertaining to the location and size of the tumors. If the model
were to be trained solely on the kidney images without the inclusion of tumor
location data, it could potentially pick up on irrelevant features and perform
poorly on previously unseen data. Thus, the concatenation step helps to
improve the model’s generalizability and overall accuracy.

####  3.2.2 Training Details

To validate the robustness of the radiomic feature extractor network, we
conducted three-fold cross-validation with three distinct train, validation,
and test subsets, while maintaining the same hyperparameters for each training
iteration. For training the 3D CNN feature extractor, we used the ADAM
optimizer (Kingma and Ba, 2014) with a fixed learning rate of $1\times
10^{\mathrm{-4}}$, and 50 epochs were run to optimize the network parameters.
In addition, we employed the Cross-Entropy loss given by:

$L=\mathrm{-}\sum_{i=1}^{n}t_{i}\times log(p_{i}),$ (1)

where $t_{i}$ is the true ISUP class and $p_{i}$ is the softmax probability
for the $ith$ class, and $n$ is the number of ISUP classes (4 in this study).
The 3D feature extractor was trained using PyTorch v1.11.0 on a workstation
equipped with an Nvidia GeForce RTX 3090 GPU, an AMD Ryzen 7 5800X 8-Core
Processor, and 32 GB of RAM.

####  3.2.3 Best Fold Selection for Radiomic Feature Extraction

We used precision, recall, and F-score in the evaluation of our feature
extractor network, as these fundamental metrics are indispensable for
assessing classification model performance.

Precision, also known as the positive predictive value, quantifies the
fraction of true positives out of the total instances predicted as positive by
the model. Mathematically, precision can be defined as:

$\text{Precision}=\frac{\text{TP}}{(\text{TP}+\text{FP})},$ (2)

where TP denotes true positives and FP denotes false positives.

Recall, alternatively referred to as sensitivity or true positive rate,
measures the fraction of true positive instances among the total number of
actual positive instances within the dataset. Recall can be mathematically
represented as:

$\text{Recall}=\frac{\text{TP}}{(\text{TP}+\text{FN})},$ (3)

where FN denotes false negatives.

The F-score, specifically the F1-score, constitutes the harmonic mean of
precision and recall, delivering a single metric that balances both measures.
The F1-score is particularly advantageous in situations with uneven class
distributions, as it accounts for the trade-off between precision and recall.
The F1-score can be calculated using the following equation:

$\text{F1-score}=2*\frac{(\text{Precision}*\text{Recall})}{(\text{Precision}+\text{Recall})}.$
(4)

We calculated the average of four Precision, Recall, and F-scores that we
gained for each ISUP class. We repeated this process three times for each of
our three folds, giving us three average Precision, Recall, and F-scores. The
second fold, with an average F-score of 0.84, was the best and selected as our
final radiomic feature extractor that can be used the the input for the
survival network.

###  3.3 Training the Survival Network

In the present study, we used a total of 500 epochs for training the survival
network. To prevent overfitting and enhance generalization, early stopping was
implemented with a patience level of 10. The model was optimized utilizing the
Adam optimizer, accompanied by a learning rate of 0.01. The optimal learning
rate selection was determined by applying the method put forth by Smith
(Smith, 2017).

###  3.4 Experiments

In our study to demonstrate the performance improvement of our proposed
survival analysis framework, we conduct nine distinct experiments with
different combinations of inputs. The first experiment involves solely CT
image features, the second only involves clinical variables, and the third
combines CT image features and clinical variables. The remaining six
experiments are created by applying three distinct thresholds for each the
Spearman correlation and the random forest regression importance score. The
clinical variables are selected based on the thresholds in the last six
experiments and then fed to the survival network. These experiments are then
compared to each other to evaluate their effectiveness in predicting survival
outcomes. Further details on the results of these experiments will be
presented in Section 4.2.

##  4 Results

In this section, we present the evaluation of our survival model’s
performance, the experimental results, and a comparison with related previous
studies. We have organized this section into four parts: 1) metrics for
survival model performance evaluation, where we describe the evaluation
metrics used to assess the performance of our proposed survival model; 2)
experimental results from nine different experiments, in which we report and
analyze the results obtained from a series of nine distinct experiments
conducted to evaluate our method; 3) plotting violin diagram for survival
distribution, which involves the visualization of survival distribution data
using violin diagrams to provide a comprehensive understanding of the results;
4) discussion, where we compare our findings with those from related previous
studies, highlighting the improvements and contributions made by our proposed
method.

###  4.1 Metrics for Performance Evaluation

To assess the performance of our survival model, we used two key metrics: the
time-dependent concordance index ($C^{td}$) and the cumulative dynamic area
under the curve (AUC). $C^{td}$ extends Harrell’s concordance index (Harrell
et al., 1982), a widely utilized measure for evaluating the discriminative
power of survival models. The time-dependent C-index is specifically designed
to address situations in which a model’s predictive accuracy may vary over
time. It gauges the model’s capacity to accurately rank the predicted survival
probabilities of subject pairs at a specific time point, taking censoring into
account. The computation of $C^{td}(t)$ involves dividing the count of
accurately ordered pairs by the total count of comparable pairs. The $C^{td}$
range between 0 and 1, where values approaching 1 signify superior predictive
accuracy, while those nearing 0.5 indicate the model possesses no greater
discriminative power than random chance. It has been established that the
concordance index is excessively optimistic, particularly with an increasing
number of censored patients in the dataset (Uno et al., ).

The cumulative dynamic AUC (Lambert and Chevret, 2016) extends the
conventional AUC metric, a prominent measure for assessing binary
classification models. This extension is tailored to specifically address
censored data and time-varying predictions in the realm of survival analysis.
Within this context, the cumulative dynamic AUC is computed for a designated
time point t, quantifying the model’s discriminatory capacity to distinguish
subjects experiencing the event of interest by time t from those who do not.
The cumulative dynamic AUC represents the area under the time-dependent
Receiver Operating Characteristic (ROC) curve, which delineates the
sensitivity (true positive rate) against 1-specificity (false positive rate)
for different time points. Ranging from 0 to 1, the cumulative dynamic AUC
reveals greater predictive accuracy as it approaches 1, while values nearing
0.5 indicate that the model’s discriminatory power is no better than random
chance.

In addition to standard metrics, we use violin plots, a novel approach, to
observe survival model output distributions. This is the first study proposing
the application of violin plots for the evaluation of survival models. High
evaluation metrics may be misleading, as predicted survival probabilities may
not match ground truth times of death. Violin plots serve as a valuable tool
in visualizing model performance by exhibiting the distribution of predicted
probabilities at the time of mortality for deceased individuals, as well as
the distribution of predicted probabilities at the ultimate time point for
censored subjects. For example, a distribution approximating zero for deceased
patients signifies satisfactory model training, which consequently yields
probability predictions in close proximity to zero.

###  4.2 Experimental Results

One of our study aims to investigate the impact of various combinations of
clinical variables on the prediction of survival outcomes in patients with
RCC. Specifically, we seek to identify the clinical features that contribute
most significantly to the accurate prediction of patients’ survival times.
Initially, we conducted two independent analyses to evaluate the effectiveness
of CT image features and clinical data individually with respect to their
impact on the performance of our survival model. Subsequently, we explore the
impact of merging CT image features with various combinations of selected
clinical variables on the performance of the survival model.

To this end, we developed nine distinct experiments (Exp). Table 1 shows the
difference between these nine experiments in terms of their inputs and
thresholds used for choosing the combination of clinical variables. Table 1
also reports the C-index and AUC obtained on the test subset from each
experiment. We used the same survival network architecture in the nine
experiments for a fair comparison. From experiment 4 to experiment 9, we
applied different thresholds for the Spearman correlation score (S_score) and
random forest regression importance score (I_score).

Table 1: Differences of Experiments used for RCC survival analysis. Exp |  Inputs | Theresholds |  C-index |  AUC
---|---|---|---|---
Exp1 |  CT images Features |  |  0.72 |  0.73
Exp2 |  38 clinical variables |  |  0.72 |  0.74
Exp3 |  CT images Features |  |  0.82 |  0.74
38 clinical variables |
Exp4 |  CT images Features |  |  0.79 |  0.76
4 clinical variables | $|\text{S\\_score}|\geqslant 0.1$
Exp5 |  CT images Features |  |  0.83 |  0.75
13 clinical variables | $|\text{S\\_score}|\geqslant 0.05$
Exp6 |  CT images Features |  |  0.81 |  0.77
30 clinical variables | $|\text{S\\_score}|\geqslant 0.01$
Exp7 |  CT images Features |  |  0.77 |  0.74
4 clinical variables | $\text{I\\_score}\geqslant 0.1$
Exp8 |  CT images Features |  |  0.84 |  0.8
17 clinical variables | $\text{{{\color[rgb]{1,0.04,0.61}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.04,0.61}\pgfsys@color@cmyk@stroke{0}{0.96}{0.39}{0}\pgfsys@color@cmyk@fill{0}{0.96}{0.39}{0}I\\_score}}}{\color[rgb]{1,0.04,0.61}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.04,0.61}\pgfsys@color@cmyk@stroke{0}{0.96}{0.39}{0}\pgfsys@color@cmyk@fill{0}{0.96}{0.39}{0}\geqslant}\textbf{{\color[rgb]{1,0.04,0.61}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.04,0.61}\pgfsys@color@cmyk@stroke{0}{0.96}{0.39}{0}\pgfsys@color<EMAIL_ADDRESS>
Exp9 |  CT images Features |  |  0.84 |  0.76
29 clinical variables | $\text{I\\_score}\geqslant 0.001$

We adjusted three different thresholds for Spearman’s correlation coefficient.
As the threshold values decreased, we incorporated more clinical variables
with weaker correlations to the patient survival time into the survival model.
In contrast, we utilized three different thresholds for the importance score
of the decision tree regressor. By lowering these threshold values, we
gradually incorporated less important clinical variables in predicting
survival times into the survival model.

According to Table 1, the best evaluation metrics were obtained in experiment
8, in which the C-index and AUC are 0.84 and 0.8, respectively. The inputs to
experiment 8 are the followings: CT images features, Localized Solid Tumor,
Age at Nephrectomy, Congestive Heart Failure, Body Mass Index, Uncomplicated
Diabetes Mellitus, Pathologic Size, Myocardial Infarction, Radiographic Size,
Metastatic Solid Tumor, Hospitalization, Mild Liver Disease, Smoking History,
Surgery Type, Gender, Tumor Histologic Subtype, Pathology T Stage, and
Surgical Approach.

In order to evaluate the effectiveness of the survival model, ten unique
individuals from the test cohort were selected, of which five had deceased
from RCC, and five had censoring time to event. Subsequently, the survival
curves for these patients were plotted, utilizing the survival probabilities
derived from experiment 8. Fig. 2 illustrates five distinct survival curves
generated by our survival model, corresponding to five different patients from
the test cohort with events equal to one (deceased). Based on the ground truth
survival time, patient 1 died after 645 days, patient two after 688 days,
patient three after 102 days, patient four after 2,000 days, and patient five
after 39 days.

At the time of their respective deaths, the model predicted survival
probabilities of 0.42, 0.15, 0.3, 0.05, and 0.5 for patients 1 through 5.
These values indicate varying degrees of accuracy in predicting the survival
probabilities at the actual time of death, with patient 4 exhibiting the
lowest probability and patient 5 the highest. At 500 days, the model’s
survival probability predictions for patients 1 to 5 were 0.57, 0.2, 0.06,
0.3, and 0.05, respectively. At 1000 days, these probabilities decreased to
0.1, 0.07, 0, 0.18, and 0 for the same patients. At 1500 days, all survival
probability predictions reached 0, except for patient 4, whose probability
reached 0 at 2000 days. The above findings suggest that the model demonstrates
varying performance in predicting survival probabilities for the five patients
at different time points. Some predictions align closely with the ground truth
survival times, while others exhibit a bit of discrepancy.

$0$$500$$1000$$1500$$2000$$2500$$3000$$0$$0.2$$0.4$$0.6$$0.8$$1$Time
(Days)Survival ProbabilitiesPatient 1Patient 2Patient 3Patient 4Patient 5
Figure 2: Survival Probabilities for five patients in the test cohort who
died.

Fig. 3 illustrates five distinct survival curves generated by our survival
model for five different patients from the test cohort with events equal to
zero (censored) and censoring time greater than 2000 days. Based on the ground
truth survival time, their censoring times are 2473 days for patient 6, 2045
days for patient 7, 2900 days for patient 8, 2600 days for patient 9, and 2298
days for patient 10.

For patient 6, the model indicates a high probability of survival (0.95) at
the censoring time of 2473, while patient 7 has a slightly lower survival
probability of 0.9 at the censoring time of 2045. Patients 8, 9, and 10
exhibit survival probabilities of 0.75, 0.68, and 0.87 at their censoring
times of 2900, 2600, and 2298, respectively. These predictions suggest that
patient 6 has the highest likelihood of survival at their censoring time,
followed by patients 7 and 10. Conversely, patients 8 and 9 possess relatively
lower survival probabilities, with patient 9 exhibiting the lowest probability
of survival among the five patients at their respective censoring times.

$0$$500$$1000$$1500$$2000$$2500$$3000$$0$$0.2$$0.4$$0.6$$0.8$$1$Time
(Days)Survival ProbabilitiesPatient 6Patient 7Patient 8Patient 9Patient 10
Figure 3: Survival Probabilities for five patients in the test cohort who had
censored events.

###  4.3 Violin Diagram for Survival Distribution

Fig. 4 presents the violin plot for censored and uncensored subjects in the
testing subset, showcasing the survival probability on the vertical axis for
Exp8, which emerged as the optimal experimental outcome. As we mentioned in
Section 4.1, with violin plots, we can comprehend the distribution of survival
probabilities predicted by our survival model.

Figure 4: Violin plots for censored & deceased events in train & test sets.

Censored_Test relates to the patients who did not experience the event in the
test subset. Regarding the Censored_Test, we are uncertain about the outcomes
at the final time point (whether death occurred or not). Based on the median,
it can be inferred that for half of the subjects, a survival probability lower
than 0.45 would be predicted, with a higher concentration around 0.1.
Conversely, for the remaining half, a survival probability greater than 0.45
would be anticipated, with a greater distribution around 0.8. Given the
symmetrical distribution around the median for Censored_Test, the model
predicts that half of the censored patients would exhibit high survival
probability at the last observation time. In contrast, the other half would
demonstrate low survival probability. Dead_Test refers to patients who died
within the test subset. This group’s ideal output survival probabilities
distribution is at zero. The median survival probability predicted by our
survival model is around 0.3. Our survival model accurately predicted near-
zero survival probabilities for half of the patients whose predicted
probabilities were below the median. The other half of the patients with
predicted probabilities higher than the median had distributions mostly near
the median. Those nearer to the median had accurate survival predictions but
with a small time shift. Those close to 1 are those patients whose survival
probabilities were not accurately calculated. Upon analyzing the violin plots
of the test subset for both censored and deceased patients, it can be
concluded that our proposed multimodal survival model yields satisfactory
outcomes that mostly align closely with the actual follow-up times of
patients.

##  5 Discussion

The hypotheses underlying our study were twofold. Firstly, we aimed to
investigate whether the selective provision of the most relevant clinical
variables to the model would enhance the performance evaluation of survival
analysis, as opposed to indiscriminately supplying all clinical variables. As
evidenced by Table 1 in Section 4.2, our findings revealed that the most
favorable results were obtained in Exp 8, wherein clinical variables were
judiciously chosen. In contrast, Exp 3, which involved the inclusion of all
clinical variables, yielded a lower C-index (by 0.02) and a reduced AUC (by
0.06). Our second hypothesis posited that multimodal survival analysis would
yield superior results when compared to single-modality approaches. In support
of this hypothesis, Table 1 in Section 4.2 demonstrates that using single-
modality data, such as solely clinical data or CT image features, led to lower
performance metrics. In contrast, Exp 3 through 9, which incorporated a
combination of clinical data and CT image features, resulted in significantly
improved performance outcomes.

To demonstrate that the integration of clinical data and CT image features
results in superior performance compared to using CT image features or
clinical data alone; we selected a single patient from the test cohort whose
survival curve was incorrectly plotted in Exp 1 and Exp 2, in which both used
a single data modality. This patient had an ISUP grade of 4 and a survival
duration of 2,000 days. Subsequently, we generated survival curves for this
patient from our nine defined experiments as illustrated in Fig. 5. The
estimated survival probabilities for the selected patient at the time of death
(2,000 days) were approximately 0.77 and 0.82 for Exp 1 and Exp 2,
respectively. In contrast, the survival probabilities at the time of death for
Exp 3 through 9 were as follows 0.18 for Exp 3, 0.6 for Exp 4, 0.61 for Exp 5,
0.19 for Exp 6, 0.55 for Exp 7, 0.05 for Exp 8, and 0 for Exp 9. This result
demonstrates that multimodal data can yield superior results compared to
single-modality experiments.

$0$$500$$1000$$1500$$2000$$2500$$3000$$0$$0.2$$0.4$$0.6$$0.8$$1$Time
(Days)Survival ProbabilitiesExperiment 1Experiment 2Experiment 3Experiment
4Experiment 5Experiment 6Experiment 7Experiment 8Experiment 9 Figure 5:
Survival Probabilities from 9 different experiments for one patient.

In the context of our study, we sought to draw comparisons with other studies
that employed radiological images and clinical variables as inputs for their
survival models. A summary of these methodologies can be found in Section 1.2.
Table 2 presents a comparison between our approach and previous studies,
focusing on the C-index and AUC metrics. Our method outperforms the others in
terms of both C-index and AUC, as demonstrated in Table 2. Our methodology,
utilizing 17 clinical variables, yielded the highest C-index and AUC values,
demonstrating its superior performance. As indicated in the second row of the
table 2 for Exp4, our approach’s effectiveness remains evident even when only
four clinical variables are employed. The C-index and AUC values in Exp4
scenario continue to surpass those of alternative methods, despite the
constrained number of clinical variables utilized. Additionally, it is worth
noting that none of the aforementioned studies provided a methodology capable
of generating non-proportional individualized survival curves for distinct
patients. Furthermore, these studies relied on traditional methodologies that
were susceptible to proportionality issues. In contrast, our approach not only
yielded superior performance in terms of C-index and AUC but also addressed
the limitations inherent in previous studies.

Table 2: Comparison of this study results with previous related studies. Studies |  Number of Clinical Variables |  C-index |  AUC
---|---|---|---
Our Method |  17 (Exp8) |  0.84 |  0.8
Our Method |  4 (Exp4) |  0.79 |  0.76
Chaddad et al. (2017) |  2 (Age, TNM Stage) |  - |  0.76
Wu et al. (2021) |  5 (Age, Histology, TNM Stage, Overall Stage, Gender) |  0.65 |  -
Zhang et al. (2020) |  3 (Tumor Size,
Tumor Localization, TNM Stage) |  0.78 |  -
Zhong et al. (2020) |  3 (Age, LDH,
Pre-EBV DNA) |  0.78 |  -

In addition to the benefits of our method, our study has a number of
limitations. Firstly, for Experiment 8, which achieved the highest C-index and
AUC, 17 clinical variables were employed during the training process. In order
to generate survival predictions for a new patient, it is essential to obtain
all 17 clinical variables to ensure the accuracy of the survival estimation.
Secondly, precise feature extraction necessitates not only whole abdomen
images but also segmentation annotations of the target organ and associated
tumors. Thirdly, to generalize this study’s findings to other types of cancer,
it is essential to pinpoint a clinical variable comparable to the ISUP grade,
enabling tumor classification in relation to survival estimation.

In future research, we aim to explore the feasibility of integrating RCC ISUP
grade classification and survival prediction within a unified training
framework, eliminating the need for separate tumor grading. Furthermore, we
intend to investigate innovative approaches for feature extraction that
circumvent the necessity for organ and tumor annotations, thereby enhancing
the applicability and efficiency of the proposed methodology.

##  6 Conclusion

This study presents a novel multimodal AI-based framework for predicting
individualized survival probabilities of patients with renal cell carcinoma.
The proposed framework utilizes CT imaging and clinical data as inputs. We
demonstrated that relevant features for survival estimation could be extracted
from CT scans and combined with clinical data to improve performance. Our
proposed framework can generate personalized, non-linear, and non-proportional
survival probability curves for different patients, achieving higher accuracy
and outperforming previously published methods. We showed that using a
multimodal strategy for survival analysis leads to higher accuracy than a
single-modality approach. Moreover, we presented that carefully selecting
significant clinical factors as inputs to the survival model can further
enhance the performance of survival prediction. This study lays the path for
enhanced clinical decision-making for renal cell carcinoma patients, allowing
for more precise and individualized therapy options based on the combination
of radiological imaging and clinical data. Future research in this field may
build upon these findings, resulting in even more complex and reliable
survival prediction models.

##  7 Acknowledgement

The authors acknowledge the CIRCLE grant no. 287112 and the Health South-East
Trust grant no. 2023069 for funding this study. We thank Hvard Kvamme, a
previous Ph.D. student at the University of Oslo, for his invaluable guidance
in effectively utilizing the pycox library he created.

## References

  * Akar et al. (2017) Akar, E., Kara, S., Akdemir, H., Kırış, A., 2017\.  Fractal analysis of mr images in patients with chiari malformation: The importance of pre-processing.  Biomedical Signal Processing and Control 31, 63–70.  doi:https://doi.org/10.1016/j.bspc.2016.07.005.
  * Alom et al. (2019) Alom, M.Z., Yakopcic, C., Hasan, M., Taha, T.M., Asari, V.K., 2019.  Recurrent residual U-Net for medical image segmentation.  J. Med. Imaging (Bellingham) 6, 014006\.  doi:https://doi.org/10.1117/1.JMI.6.1.014006.
  * Brady (2017) Brady, A.P., 2017.  Error and discrepancy in radiology: inevitable or avoidable?  Insights Imaging 8, 171–182.  doi:https://doi.org/10.1007/s13244-016-0534-1.
  * Brown et al. (1997) Brown, S., Branford, A., Moran, W., 1997.  On the use of artificial neural networks for the analysis of survival data.  IEEE Transactions on Neural Networks 8, 1071–1077.  doi:https://doi.org/10.1109/72.623209.
  * Chaddad et al. (2017) Chaddad, A., Desrosiers, C., Toews, M., Abdulkarim, B., 2017\.  Predicting survival time of lung cancer patients using radiomic analysis.  Oncotarget 8, 104393–104407.  doi:https://doi.org/10.18632%2Foncotarget.22251.
  * Ching et al. (2018) Ching, T., Zhu, X., Garmire, L.X., 2018.  Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data.  PLoS Comput Biol 14, e1006076.  doi:https://doi.org/10.1371/journal.pcbi.1006076.
  * Coppola et al. (2021) Coppola, F., Faggioni, L., Gabelloni, M., De Vietro, F., Mendola, V., Cattabriga, A., Cocozza, M.A., Vara, G., Piccinino, A., Lo Monaco, S., Pastore, L.V., Mottola, M., Malavasi, S., Bevilacqua, A., Neri, E., Golfieri, R., 2021\.  Human, All Too Human? An All-Around Appraisal of the Artificial Intelligence Revolution in Medical Imaging.  Front Psychol 12, 710982\.  doi:https://doi.org/10.3389/fpsyg.2021.710982.
  * Costantini et al. (2021) Costantini, M., Poeta, M.L., Pfeiffer, R.M., Hashim, D., Callahan, C.L., Sentinelli, S., Mendoza, L., Vicari, M., Pompeo, V., Pesatori, A.C., DellaValle, C.T., Simone, G., Fazio, V.M., Gallucci, M., Landi, M.T., 2021.  Impact of histology and tumor grade on clinical outcomes beyond 5 years of follow-up in a large cohort of renal cell carcinomas.  Clinical Genitourinary Cancer 19, e280–e285.  doi:https://doi.org/10.1016/j.clgc.2021.07.003.
  * Cox (1972) Cox, D.R., 1972.  Regression models and life-tables.  Journal of the Royal Statistical Society: Series B (Methodological) 34, 187–202.  doi:https://doi.org/10.1111/j.2517-6161.1972.tb00899.x.
  * Fotso (2018) Fotso, S., 2018.  Deep neural networks for survival analysis based on a multi-task framework doi:https://doi.org/10.48550/arXiv.1801.05512.
  * Gensheimer and Narasimhan (2019) Gensheimer, M.F., Narasimhan, B., 2019\.  A scalable discrete-time survival model for neural networks.  PeerJ 7, e6257.  doi:https://doi.org/10.7717/peerj.6257.
  * Harrell et al. (1982) Harrell, F.E., Califf, R.M., Pryor, D.B., Lee, K.L., Rosati, R.A., 1982.  Evaluating the yield of medical tests.  JAMA 247, 2543–2546.  doi:https://doi.org/10.1001/jama.1982.03320430047030.
  * Heller et al. (2021) Heller, N., Isensee, F., Maier-Hein, K.H., Hou, X., Xie, C., Li, F., Nan, Y., Mu, G., Lin, Z., Han, M., Yao, G., Gao, Y., Zhang, Y., Wang, Y., Hou, F., Yang, J., Xiong, G., Tian, J., Zhong, C., Ma, J., Rickman, J., Dean, J., Stai, B., Tejpaul, R., Oestreich, M., Blake, P., Kaluzniak, H., Raza, S., Rosenberg, J., Moore, K., Walczak, E., Rengel, Z., Edgerton, Z., Vasdev, R., Peterson, M., McSweeney, S., Peterson, S., Kalapara, A., Sathianathen, N., Papanikolopoulos, N., Weight, C., 2021.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 challenge.  Med. Image Anal. 67, 101821\.  doi:https://doi.org/10.1016/j.media.2020.101821.
  * Heller et al. (2019) Heller, N., Sathianathen, N., Kalapara, A., Walczak, E., Moore, K., Kaluzniak, H., Rosenberg, J., Blake, P., Rengel, Z., Oestreich, M., Dean, J., Tradewell, M., Shah, A., Tejpaul, R., Edgerton, Z., Peterson, M., Raza, S., Regmi, S., Papanikolopoulos, N., Weight, C., 2019\.  The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes doi:https://doi.org/10.48550/arXiv.1904.00445.
  * Hui et al. (2019) Hui, D., Paiva, C.E., Del Fabbro, E.G., Steer, C., Naberhuis, J., van de Wetering, M., ndez Ortega, P., Morita, T., Suh, S.Y., Bruera, E., Mori, M., 2019.  Prognostication in advanced cancer: update and directions for future research.  Support Care Cancer 27, 1973–1984.  doi:https://doi.org/10.1007/s00520-019-04727-y.
  * Katzman et al. (2018) Katzman, J.L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., Kluger, Y., 2018\.  Deepsurv: personalized treatment recommender system using a cox proportional hazards deep neural network.  BMC Medical Research Methodology 18\.  doi:https://doi.org/10.1186/s12874-018-0482-1.
  * Kingma and Ba (2014) Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  doi:https://doi.org/10.48550/arXiv.1412.6980.
  * Lambert and Chevret (2016) Lambert, J., Chevret, S., 2016\.  Summary measure of discrimination in survival models based on cumulative/dynamic time-dependent roc curves.  Statistical Methods in Medical Research 25, 2088–2102.  doi:https://doi.org/10.1177/0962280213515571.
  * Lambin et al. (2012) Lambin, P., Rios-Velazquez, E., Leijenaar, R., Carvalho, S., van Stiphout, R.G., Granton, P., Zegers, C.M., Gillies, R., Boellard, R., Dekker, A., Aerts, H.J., 2012.  Radiomics: extracting more information from medical images using advanced feature analysis.  Eur J Cancer 48, 441–446.  doi:https://doi.org/10.1016/j.ejca.2011.11.036.
  * Lee and Wang (2003) Lee, E.T., Wang, J.W., 2003\.  Statistical Methods for Survival Data Analysis.  John Wiley & Sons, Inc.  doi:https://doi.org/10.1002/0471458546.
  * Litjens et al. (2017) Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A.W.M., van Ginneken, B., Sánchez, C.I., 2017.  A survey on deep learning in medical image analysis.  Med. Image Anal. 42, 60–88.  doi:https://doi.org/10.1016/j.media.2017.07.005.
  * Liu et al. (2019) Liu, X., Faes, L., Kale, A.U., Wagner, S.K., Fu, D.J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., Ledsam, J.R., Schmid, M.K., Balaskas, K., Topol, E.J., Bachmann, L.M., Keane, P.A., Denniston, A.K., 2019\.  A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.  The Lancet Digital Health 1, e271–e297.  doi:https://doi.org/10.1016/S2589-7500(19)30123-2.
  * Mahootiha et al. (2022) Mahootiha, M., Qadir, H., Bergsland, J., Balasingham, I., 2022\.  Classification of kidney tumor grading on preoperative computed tomography scans.  16th EAI International Conference on Pervasive Computing Technologies for Healthcare , 1–15.
  * n Montero et al. (2021) n Montero, A., Javaid, U., s, G., Nguyen, D., Desbordes, P., Macq, B., Willems, S., Vandewinckele, L., m, M., fman, F., Michiels, S., Souris, K., Sterpin, E., Lee, J.A., 2021\.  Artificial intelligence and machine learning for medical imaging: A technology review.  Phys Med 83, 242–256.  doi:https://doi.org/10.1016/j.ejmp.2021.04.016.
  * Mukherjee et al. (2020) Mukherjee, P., Zhou, M., Lee, E., Schicht, A., Balagurunathan, Y., Napel, S., Gillies, R., Wong, S., Thieme, A., Leung, A., Gevaert, O., 2020.  A Shallow Convolutional Neural Network Predicts Prognosis of Lung Cancer Patients in Multi-Institutional CT-Image Data.  Nat Mach Intell 2, 274–282.  doi:https://doi.org/10.1038/s42256-020-0173-6.
  * Ngiam et al. (2011) Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y., 2011.  Multimodal deep learning, in: Proceedings of the 28th International Conference on International Conference on Machine Learning, Omnipress. p. 689–696.  doi:https://dl.acm.org/doi/10.5555/3104482.3104569.
  * Pirie (2006) Pirie, W., 2006.  Spearman Rank Correlation Coefficient. volume 8.  doi:https://doi.org/10.1002/0470011815.b2a15150.
  * Pérez-García et al. (2021) Pérez-García, F., Sparks, R., Ourselin, S., 2021.  Torchio: A python library for efficient loading, pre-processing, augmentation and patch-based sampling of medical images in deep learning.  Computer Methods and Programs in Biomedicine 208, 106236.  doi:https://doi.org/10.1016/j.cmpb.2021.106236.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L., 2015\.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision 115, 211–252.  doi:https://doi.org/10.1007/s11263-015-0816-y.
  * Saad et al. (2019) Saad, A.M., Gad, M.M., Al-Husseini, M.J., Ruhban, I.A., Sonbol, M.B., Ho, T.H., 2019\.  Trends in renal-cell carcinoma incidence and mortality in the united states in the last 2 decades: A seer-based study.  Clinical Genitourinary Cancer 17, 46–57.e5.  doi:https://doi.org/10.1016/j.clgc.2018.10.002.
  * Samaratunga et al. (2014) Samaratunga, H., Gianduzzo, T., Delahunt, B., 2014.  The isup system of staging, grading and classification of renal cell neoplasia.  Journal of kidney cancer and VHL 1, 26.  doi:https://doi.org/10.15586%2Fjkcvhl.2014.11.
  * Sandler et al. (2018) Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C., 2018.  Mobilenetv2: Inverted residuals and linear bottlenecks, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510–4520.  doi:https://doi.org/10.48550/arXiv.1801.04381.
  * Siegel et al. (2020) Siegel, R.L., Miller, K.D., Jemal, A., 2020.  Colorectal Cancer statistics, 2020.  CA Cancer J Clin 70, 7–30.  doi:https://doi.org/10.3322/caac.21601.
  * Smith (2017) Smith, L., 2017.  Cyclical learning rates for training neural networks, pp. 464–472.  doi:https://doi.org/10.1109/WACV.2017.58.
  * Srigley et al. (2013) Srigley, J.R., Delahunt, B., Eble, J.N., Egevad, L., Epstein, J.I., Grignon, D., Hes, O., Moch, H., Montironi, R., Tickoo, S.K., Zhou, M., Argani, P., ISUP Renal Tumor Panel, 2013.  The international society of urological pathology (ISUP) vancouver classification of renal neoplasia.  Am. J. Surg. Pathol. 37, 1469–1489.  doi:https://doi.org/10.1097/PAS.0b013e318299f2d1.
  * Sung et al. (2021) Sung, H., Ferlay, J., Siegel, R.L., Laversanne, M., Soerjomataram, I., Jemal, A., Bray, F., 2021.  Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries.  CA: a cancer journal for clinicians 71, 209–249.  doi:https://doi.org/10.3322/caac.21660.
  * Tan and Le (2019) Tan, M., Le, Q.V., 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks doi:https://doi.org/10.48550/arXiv.1905.11946.
  * Tustison et al. (2010) Tustison, N.J., Avants, B.B., Cook, P.A., Zheng, Y., Egan, A., Yushkevich, P.A., Gee, J.C., 2010.  N4ITK: improved N3 bias correction.  IEEE Trans. Med. Imaging 29, 1310–1320.  doi:https://doi.org/10.1109/TMI.2010.2046908.
  * (39) Uno, H., Cai, T., Pencina, M.J., D’Agostino, R.B., Wei, L.J., .  On the c-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data.  Statistics in Medicine 30, 1105–1117.  doi:https://doi.org/10.1002/sim.4154.
  * Vankdothu and Hameed (2022) Vankdothu, R., Hameed, M.A., 2022\.  Brain tumor mri images identification and classification based on the recurrent convolutional neural network.  Measurement: Sensors , 100412doi:https://doi.org/10.1016/j.measen.2022.100412.
  * Wang et al. (2019a) Wang, P., Li, Y., Reddy, C.K., 2019a.  Machine learning for survival analysis: A survey.  ACM Comput. Surv. 51.  doi:https://doi.org/10.1145/3214306.
  * Wang et al. (2019b) Wang, S., Liu, Z., Rong, Y., Zhou, B., Bai, Y., Wei, W., Wei, W., Wang, M., Guo, Y., Tian, J., 2019b.  Deep learning provides a new computed tomography-based prognostic biomarker for recurrence prediction in high-grade serous ovarian cancer.  Radiother Oncol 132, 171–177.  doi:https://doi.org/10.1016/j.radonc.2018.10.019.
  * Warren and Harrison (2018) Warren, A.Y., Harrison, D., 2018\.  WHO/ISUP classification, grading and pathological staging of renal cell carcinoma: standards and controversies.  World J. Urol. 36, 1913–1926.  doi:https://doi.org/10.1007/s00345-018-2447-8.
  * Wehenkel et al. (2018) Wehenkel, M., Sutera, A., Bastin, C., Geurts, P., Phillips, C., 2018.  Random forests based group importance scores and their statistical interpretation: Application for alzheimer's disease.  Frontiers in Neuroscience 12.  doi:https://doi.org/10.3389/fnins.2018.00411.
  * Wu et al. (2021) Wu, Y., Ma, J., Huang, X., Ling, S.H., Weidong Su, S., 2021\.  Deepmmsa: A novel multimodal deep learning method for non-small cell lung cancer survival analysis, in: 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1468–1472.  doi:https://doi.org/10.1109/SMC52423.2021.9658891.
  * Zhang et al. (2020) Zhang, L., Dong, D., Zhang, W., Hao, X., Fang, M., Wang, S., Li, W., Liu, Z., Wang, R., Zhou, J., Tian, J., 2020.  A deep learning risk prediction model for overall survival in patients with gastric cancer: A multicenter study.  Radiother Oncol 150, 73–80.  doi:https://doi.org/10.1016/j.radonc.2020.06.010.
  * Zhong et al. (2020) Zhong, L.Z., Fang, X.L., Dong, D., Peng, H., Fang, M.J., Huang, C.L., He, B.X., Lin, L., Ma, J., Tang, L.L., Tian, J., 2020\.  A deep learning mr-based radiomic nomogram may predict survival for nasopharyngeal carcinoma patients with stage t3n1m0.  Radiotherapy and Oncology 151, 1–9.  doi:https://doi.org/10.1016/j.radonc.2020.06.050.
  * Zhu et al. (2018) Zhu, W., Liu, C., Fan, W., Xie, X., 2018.  Deeplung: Deep 3d dual path nets for automated pulmonary nodule detection and classification , 673–681doi:https://doi.org/10.1109/WACV.2018.00079.
  * Znaor et al. (2015) Znaor, A., Lortet-Tieulent, J., Laversanne, M., Jemal, A., Bray, F., 2015.  International variations and trends in renal cell carcinoma incidence and mortality.  European Urology 67, 519–530.  doi:https://doi.org/10.1016/j.eururo.2014.10.002.