# Segmentation-Free Outcome Prediction in Head and Neck Cancer: Deep Learning-
based Feature Extraction from Multi-Angle Maximum Intensity Projections (MA-
MIPs) of PET Images

Amirhosein Toosi
Department of Radiology
University of British Columbia
Department of Integrative Oncology
BC Cancer Research Institute
Vancouver, BC
<EMAIL_ADDRESS>
&Isaac Shiri
Division of Nuclear Medicine and Molecular Imaging
Geneva University Hospital
Department of Cardiology
University Hospital Bern
Switzerland
<EMAIL_ADDRESS>
&Habib Zaidi
Division of Nuclear Medicine and Molecular Imaging
Geneva University Hospital
Geneva, Switzerland
<EMAIL_ADDRESS>
&Arman Rahmim
Departments of Radiology and Physics
University of British Columbia
Department of Integrative Oncology
BC Cancer Research Institute
Vancouver, BC
<EMAIL_ADDRESS>

###### Abstract

We introduce an innovative, simple, effective segmentation-free approach for
outcome prediction in head & neck cancer (HNC) patients. By harnessing deep
learning-based feature extraction techniques and multi-angle maximum intensity
projections (MA-MIPs) applied to Fluorodeoxyglucose Positron Emission
Tomography (FDG-PET) volumes, our proposed method eliminates the need for
manual segmentations of regions-of-interest (ROIs) such as primary tumors and
involved lymph nodes. Instead, a state-of-the-art object detection model is
trained to perform automatic cropping of the head and neck region on the PET
volumes. A pre-trained deep convolutional neural network backbone is then
utilized to extract deep features from MA-MIPs obtained from 72 multi-angel
axial rotations of the cropped PET volumes. These deep features extracted from
multiple projection views of the PET volumes are then aggregated and fused,
and employed to perform recurrence-free survival analysis on a cohort of 489
HNC patients. The proposed approach outperforms the best performing method on
the target dataset for the task of recurrence-free survival analysis. By
circumventing the manual delineation of the malignancies on the FDG PET-CT
images, our approach eliminates the dependency on subjective interpretations
and highly enhances the reproducibility of the proposed survival analysis
method.

_K_ eywords Artificial intelligence $\cdot$ machine learning $\cdot$ deep
learning $\cdot$ PET imaging $\cdot$ head and neck cancer

##  1 Introduction

Head and neck cancer (HNC) is the seventh most common cancer worldwide. It
encompasses malignancies involving the anatomic sites that compose the upper
aerodigestive tract, including the oral cavity, pharynx, larynx, nasal cavity,
paranasal sinuses, and salivary glands [1]. In 2016, over 1.1 million new
cases and 4.1 million prevalent cases were reported, resulting in
approximately 500,000 deaths worldwide [2]. Although the median diagnosis age
for HNC is around 60 years, in recent years the incidence of these cancers has
increased in adults younger than 45 years, mainly due to higher numbers of
oropharyngeal cancers associated with oncogenic human papillomavirus (HPV) [3]
.

Accurate prognosis and staging is crucial for improving patient care,
treatment planning, and disease management, ultimately leading to improved
survival rates [4]. Recurrence-free survival (RFS) analysis is a common
outcome prediction method used to assess the effectiveness of treatments for
HNC. RFS measures the length of time after treatment during which the cancer
does not recur. RFS plays a vital role in the management of HNC patients by
providing information on the durability of treatment effects and helps to
determine the optimal treatment strategy for individual patients [5]. RFS
analysis is typically performed utilizing clinical and pathological features
like tumor stage, grade, histological sub-type, and nodal involvement.
However, relying solely on these factors may not fully capture the complex
tumor heterogeneity and dynamic changes in the tumor micro-environment, which
can limit the accuracy of prognostication [6].

Biomedical imaging methods such as computed tomography (CT), magnetic
resonance imaging (MRI), and positron emission tomography (PET) can provide
valuable information in various steps of HNC management [7]. Specifically, PET
imaging plays an active role in staging, treatment planning, monitoring
response, recurrence detection and surveillance for HNC patients [8, 9].
Several quantitative parameters could be extracted from fluorodeoxyglucose
(FDG) positron emission tomography (PET) and CT images, providing information
about the metabolic activity and aggressiveness of the tumor. This information
can be helpful in predicting treatment response, disease progression, and
patient survival. Some commonly used imaging biomarkers are Standardized
Uptake Value (SUV) thresholds, such as SUVmean, SUVmax, SUVpeak, volumetric
measures, such as Metabolic Tumor Volume (MTV) and Total Lesion Glycolysis
(TLG) [10]. Additionally, the high-throughput extraction of radiomics features
from PET-CT image modalities provides massive amount of hidden information
that could help personalized management of the HNC [11, 12].

Imaging biomarkers and radiomics features are tumor specific; i.e. they need
to be extracted from the tumor region of intertest (ROI). Therefore, accurate
manual segmentation of ROIs within medical images is a crucial step for
extracting these features. Manual segmentation has been the fundamental step
in various medical imaging analysis tasks [13, 14]. However, it presents
several challenges that can impact the accuracy, reproducibility, and
efficiency of the analysis. Some of the key challenges of manual segmentation
can be categorized as follows:

Subjectivity and Inter-observer Variability: Manual segmentation is a
subjective process that relies on the expertise and judgment of the human
annotators. Different individuals may interpret the image data differently,
leading to inconsistencies in ROI delineation. Inter-observer variability,
where different annotators produce different segmentations for the same image,
can significantly affect the reliability and generalizability of the analysis
[15].

Time and Labor Intensive: Manual segmentation is a time-consuming task that
requires significant human effort and expertise. The process of precisely
outlining complex anatomical structures and tumor regions can be tedious and
prone to errors [16]. Moreover, the availability of skilled personnel for
manual segmentation may be limited, especially in high-volume clinical
settings, potentially causing delays in the analysis pipeline [17].

Limited Scalability and Generalizability: Manual segmentation is not easily
scalable to large datasets or multiple centers due to its labor-intensive
nature. Generating manual segmentations for a large cohort of patients can be
impractical and time-prohibitive [18]. Additionally, the segmentation outcomes
may be influenced by the experience and training of the annotators, limiting
the generalizability of the analysis across different sites and populations
[19, 20].

Variability in Tumor Characteristics: Tumors in HNC can exhibit diverse
characteristics, including irregular shapes, heterogeneous intensity patterns,
and variable contrast uptake [21]. Accurate delineation of these tumors solely
based on visual cues can be challenging, leading to uncertainties and errors
in manual segmentation [22]. This variability in tumor characteristics further
complicates the segmentation process and introduces additional sources of
variability and bias.

Limited Representation of Tumor Heterogeneity: Manual segmentation often
focuses on delineating the tumor or the region of interest as a whole,
neglecting potential subregions within the tumor that exhibit distinct
characteristics [23, 24]. This limited representation of tumor heterogeneity
may overlook critical information relevant to recurrence-free survival
prediction, as various subregions might contribute differently to the
prognosis.

To address these challenges, we propose a segmentation-free approach for
recurrence-free survival analysis in HNC. By circumventing the manual
segmentation step, our approach eliminates the dependency on subjective
interpretations and enhances the reproducibility of the analysis. Instead, our
method utilizes a deep learning-based object detection model to automatically
crop the head & neck region in the FDG PET volumes, enabling a more efficient,
objective, and reproducible analysis.

In the subsequent sections, we describe the methodology employed to
automatically crop the PET volumes (sections 2.2 to 2.4), generate multi-angel
maximum intensity projections of the cropped PET volumes (section 2.5),
extract deep features using a pre-trained deep convolutional neural network
(sections 2.6 to 2.8), and perform recurrence-free survival analysis (section
2.9). We present the results of our study in section 3, discuss their
implications, and highlight the advantages of the segmentation-less approach
in overcoming the challenges associated with manual segmentation in medical
imaging analysis in section 4.

##  2 Materials and Methods

###  2.1 Dataset

For this work we used the training dataset of the third MICCAI Head and Neck
Tumor segmentation and outcome prediction (HECKTOR2022) challenge [22, 25,
26]. The dataset comprises FDG PET/CT images, along with the clinical and
survival data for a cohort of 489 patients with histologically proven H&N
cancer located in the oropharynx region. Scans were gathered across 7
different medical centers, with variations in the scanner manufacturers and
acquisition protocols [27]. All 489 patients underwent radiotherapy treatment
planning and had complete responses to the treatment (i.e. disappearance of
all signs of local, regional and distant lesions). CT images had an original
median voxel size of ${0.98\times 0.98\times 2.80}$ ${mm}^{3}$, and the PET
images had a median voxel size of ${4.00\times 4.00\times 3.27}$ ${mm}^{3}$.
To ensure consistency, for all PET/CT images, both CT and PET volumes were
resampled to a voxel size of ${1.0\times 1.0\times 1.0}$ ${mm}^{3}$ using a
third-order spline method.

###  2.2 Method Overview

Here we describe our proposed segmentation-free outcome prediction method in
detail (see figure 1). First, we train the object detection model [28] on
manually drawn bounding boxes containing the anatomical region of patient’s
head and neck, on both coronal and sagittal maximum intensity projections of
CT images, as described in section 2.4. The trained network is then used for
cropping the head and neck region of the corresponding registered FDG PET
image volume of the patient, as described in section 2.3 (See figure 1-A).
Then, we employ a pre-trained CNN network with frozen weights to extract so-
called “deep features” from 72 maximum intensity projections computed from
5-degree steps of axial rotations of the cropped head and neck PET volumes,
resulting in 72 feature vectors per each patient, as elaborated in detail in
sections 2.6 and 2.7. Next, we aggregate the extracted feature vectors of all
views (MIPs) and process them through an ML pipeline for outcome prediction
(figure 1-B). All the steps are described in detail in the following sections.

Figure 1: Overview of the proposed method for fully automated segmentation-
free outcome prediction for head and neck cancer patients. A) Overall
representation of the automated head and neck region detection and cropping.
B) Extracting deep features from MA-MIP projections followed by feature fusion
step.

###  2.3 Automatic Cropping of the Head and Neck Region

HNC tumors mainly affect the upper aerodigestive tract. in the early-stage
disease the risk of distant metastasis is very low, where only approximately
10% of patients have distant metastasis at the time of diagnosis [29]. Given
that, in order to focus on the main site of the disease affection, we employed
a deep learning-based object detection model to automatically locate the
anatomical upper aerodigestive region. To this end, we first generate a
coronal and a sagittal maximum intensity projection of the CT scan of each
patient for the whole dataset. Then, we manually annotate the anatomical head
and neck region bounding boxes on both coronal and sagittal CT MIPs. On the
coronal MIP, we placed a bounding box vertically between the sternoclavicular
(SC) joint (the link between the clavicle or collarbone, and the sternum, or
breastbone) and the top of the head, or the top of the image, if the full
skull is not covered in the CT scan image. And horizontally between the left
and right acromioclavicular (AC) joint (the joint where the scapula or
shoulder blade, articulates with the clavicle or collarbone). (See figure 2).
We target the clavicle bone to extend the oropharyngeal region down, mainly to
contain all the primary tumors and the involved local lymph nodes in the
anatomical region of interest. Since we defined the anatomical region of
interest based on the regional bones, for generating coronal and sagittal CT
MIPs, we clipped all CT voxel values in the corresponding range of hard tissue
(bone) hounsfield units (700 - 2000).

Figure 2: Visual summary of the proposed bounding box definition and cropping
for locating the anatomical head and neck region on the PET volumes of the
patients based on the coronal and Sagittal MIP CT projections. Locating the
bounding boxes on coronal and sagittal CT MIPs based on the AC and SC clavicle
joints (left), and the 3D bounding box cropping on the PET volume based on the
two 2D bounding boxes on the CT mips (right).

###  2.4 Training the Object Detection Model on CT Projections

For the object detection architecture we used TridentNet [28]. TridentNet is a
multi-stage deep learning-based 2D object detection model initially proposed
for handling the highly variating object scales. Utilizing deformable
convolution in its "Trident" blocks, TridentNet extracts multi-scale features
with different receptive fields for different object scales from features of
the same semantic layers. Based on our previous experience with this
architecture for detecting very small and highly ambiguous objects in PET
images [30, 31], here we picked the same architecture for localizing the
anatomical head and neck region in CT coronal and sagittal MIPs. To this end,
first the training set was divided into 5 folds, 4 for training a network and
one for testing. Both coronal and sagittal MIPs were fed to the networks for
training, along with the manually made bounding boxes as ground truth. The
predicted bounding boxes per each patient in 5 testing folds were then used
for cropping the head and neck region on the corresponding PET volumes.

###  2.5 PET Multi-Angle Maximum Intensity Projection (MA-MIP)

After cropping the head and neck region on the PET volume, the cropped regions
underwent the feature extraction step. The common practice for extracting
features from biomedical volumetric images are either extracting features
directly from the the 3D volume, or the pixels of the transaxial layers. But
in our proposed method we instead extract image features from multiple maximum
intensity projections (MA-MIPs) of the axial rotations of the cropped PET
volume. To this end, we step by step rotate the cropped PET volume axially for
5 degrees, and per each rotation a MIP is captured from the rotated volume.
This step is repeated 72 times in every 5 degrees to cover one complete axial
rotation (see figure 1, B). This enables us to leverage strong 2D
convolutional neural networks backbones pretrained on large-scale public
datasets like ImageNet, for extracting "deep" features.

###  2.6 Deep Feature Extraction from MA-MIP Projections

The vast size and diversity of ImageNet, with millions of natural images
belonging to thousands of categories, have allowed training of models with
robust image recognition capabilities that can effectively generalize across
different visual domains, capturing complex patterns, textures, and structural
relationships. By training a deep neural network on this dataset, the model
learns to recognize features that are not specific to any particular class or
modality, but rather generalize well across different visual domains.

Through transfer learning, the knowledge gained from one task can be
transferred to another related task. By doing so, the model can leverage the
knowledge it has acquired and adapt it to the new task, leading to improved
performance compared to training a model from scratch. When using a pre-
trained CNN backbone for biomedical image analysis, the early layers can
capture low-level features like texture, color, and shape, while higher layers
can capture more abstract features like organs, tissues, or lesions. This
hierarchical representation enables the model to capture both local and global
information in medical images, leading to better performance in various tasks
[32].

In spite of the significant variability of medical images, mainly due to
factors like patient demographics, acquisition protocols, and pathological
conditions, the robustness of pre-trained CNN backbones on ImageNet can
translate to medical images, helping the model to generalize across different
populations, modalities, and diseases [33].

####  2.6.1 EfficientNet as the feature extraction CNN backbone

Looking at the trends of the proposed convolutional neural network
architectures on the ImageNet [34], from the emerge of AlexNet [35] in 2012
till present, the proposed CNN architectures are becoming larger in order to
gain higher accuracy. However, the more the architectures get larger the more
the number of parameters they reach, so much as it seems they have hit the
hardware limits [36].

The most common approach for scaling the convolutional neural networks is
increasing the depth, in order to help the network to learn more complex
features throughout the depth and to generalize well on new tasks. This comes
with the cost of difficulties in terms of training such deep models due to the
well-known problems such as gradient vanishing [37]. Despite the proposed
techniques to overcome this issue such as batch normalization and skip
connections, the accuracy of deeper networks seems to saturate at certain
point [38].

Among the state-of-the-art architectures designed for ImageNet, EfficientNet
[39] is an attempt to achieve higher accuracy, by uniformly scaling the CNN
architecture’s depth, width, and resolution, forming a compound scaling
framework. In order to scaling up the CNN network while keeping the number of
parameters of the network low, as the main building block, EfficientNet use
Inverted Bottleneck Block previously proposed in MobileNetV2 architecture
[40], along with squeeze and excitation optimization [41]. The essence of
Inverted Bottleneck Block is using the Depthwise separable convolution layer
instead of regular convolution block, which splits the convolution operation
into two separate operation, a depth wise convolution and a pointwise
convolution. This splitting lets the model to perform same convolution
operation in each block with much less computation. First introduced in
MobileNet architecture [42], the depthwise separable convolution enables the
EfficientNet architecture to scale up and gain higher accuracy with much lower
number of parameters.

Building on top of a baseline model, called EfficientNetB0, seven scales of
EfficientNet is provided with different uniform scales (EfficientNet B1 - B7)
which could achieve higher accuracies compared to the state-of-the-art CNN
architectures with same depth on the ImageNet dataset, while having lower
parameters and being faster. Besides the superior performance on the ImageNet,
EfficientNets also achieved state of-the-art accuracy on many other widely
used datasets by transfer learning, while having up to 21x less parameters
compared to other CNNs [39].

The remarkable performance of the EfficientNet architecture in terms of
transfer learning, along with the uniform scale-up of the baseline
architecture, motivated us to apply this model for feature extraction in our
work. To this end, we selected 3 different scales of the EfficientNet pre-
trained on ImageNet, namely B1, B4, and B7 (small, medium, and large) for deep
features extraction from the MA-MIPs of the cropped PET volumes, to compare
the effect of model scale on the effectiveness of the extracted features for
the outcome prediction task.

###  2.7 Deep Features Preparation

The output of the pre-trained EfficientNet used for feature extraction is the
last convolutional block before the classifier head of the network which is in
shape of a 4D tensor (a batch of 3D tensors). In order to make a feature
vector from the 3D feature tensor for each image in the batch, different
methods are commonly used. The most common approach is to apply a global
pooling layer on the output of the convolution block, either an average
pooling or a max pooling, to summarize the feature tensor to a one dimensional
feature vector. The global max pooling operation essentially computes the
maximum value over the entire height and width of a 2D convolutional block’s
channel in the 3D tensor output’s depth. The global max pooling operation can
be formulated as follows:

$\text{GlobalMaxPool}(\mathbf{T})_{c}=\max_{h,w}(\mathbf{T}_{c,h,w})$

Here, $\mathbf{T}\in\mathbb{R}^{C\times H\times W}$ represents the 3D tensor
output of the 2D convolutional block, and the index $c$ corresponds to the
channels dimension. The indices $h$ and $w$ iterate over the spatial
dimensions of the tensor, and the $max$ operation finds the maximum value
among all the spatial locations in each channel for every instance. In global
average pooling, similar to global max pooling operation, the $mean$ among all
the spatial locations in each channel is computed as following.

$\text{GlobalAveragePool}(\mathbf{T})_{c}=\frac{1}{h\times
w}\sum_{i=1}^{h}\sum_{j=1}^{w}\mathbf{T}_{c,i,j}$

In this work we also extended the global pooling layer to median and also
standard deviation pooling in order examine different possible global pooling
operators.

###  2.8 MA-MIPs Extracted Deep Features Fusion

Features extracted from MA-MIPs taken from different rotations of the cropped
PET volume, can be seen as the information extracted from different views of
the PET volume. Therefore, combining the information extracted from different
views could give a more informative set of information about the region.

In order to aggregate the feature vectors extracted from 72 MA-MIPs from
different views of the cropped PET volume, feature fusion methods are applied.
Multiple feature fusion methods is used in this work in order to aggregate
feature vectores from multiple views into a single feature vector. The
utilized methods can be categorized into three main groups; Statistical
summarizing methods, linear transformation methods, and non-linear
transformation methods.

Statistical Summarizing methods leverage simple statistical measures to
aggregate information from the 72 feature vectors into a single feature
vector. To this end the index-wise (channel-wise) maximum, mean, median, and
standard deviation of the feature vectors is computed after stacking all the
72 feature vectors of the different views. The maximum value represents the
most prominent feature across all views, while the mean, median, and standard
deviation capture different aspects of the overall statistical distribution of
features from different views. These methods simplify the representation of
the data by distilling it down to key statistical characteristics.

Linear Transformation methods on the other hand, encompass dimensionality
reduction techniques applied index-wise (channel-wise) to the stacked feature
vectors. Initially, the 72 feature vectors are stacked vertically to form a
matrix, similar to the statistical summarizing methods, where each row
represents a feature from one view, and there are 72 columns, each
corresponding to a specific feature index (channel). Independent Component
Analysis (ICA) is then individually applied to each column of the stacked
feature vectors. This process summarizes the 72 cells within each feature
index into a single value. By applying these linear transformations to each
index separately, a reduced-dimensional representations for each feature index
is obtained, in order to capture essential patterns and reduce dimensionality,
while preserving unique characteristics specific to each view. This approach
analyses how each feature contributes to the fused representation.

As for non-linear transformation methods we used an auto-encoder. This
learning-based method is applied on the concatenated 72 feature vectors to
learn a compact, lower-dimensional representation in an unsupervised end-to-
end manner. The auto-encoder comprises an encoder network that maps the input
data into a reduced-dimensional latent space and a decoder network that
reconstructs the input from this latent representation. By training the auto-
encoder, it learns to capture the most salient features in the data while
discarding noise and redundancies, resulting in an optimized fused feature
vector that retains essential information. These three categories of fusion
methods represent a well-rounded approach to fusing feature vectors from
multiple views of the HNC PET images. Each category offers unique advantages
and insights into the data, allowing for a comprehensive analysis that may
enhance the predictive power of the proposed model for outcome prediction.

###  2.9 ML Prediction Pipeline for Recurrence-Free Survival Analysis

Fused multi-view features are then fed to an outcome prediction pipeline in
order to perform recurrence-free survival prediction. Feature vectors of all
489 patients fist undergone a standard scaling by removing the mean and
scaling to unit variance. A nested 5-fold cross validation with 20 times
repetition is applied to evaluate the outcome prediction pipeline. Per each
inner fold, highly correlated features were eliminated, then an independent
component analysis (ICA) were applied on the remaining features to further
reduce the dimensionality of the features also to avoid over-fitting. Cox
proportional hazard survival method were employed on the reduced dimension
features to predict the survival risk of each patient. Grid search was used
for parameter tuning. The trained models were tested on each of the outer
unseen testing folds and the performance of the trained outcome prediction
models are reported in terms of the mean and the standard deviation of the
concordance index (c-index) of the five unseen test folds. In the next section
we report the results of our proposed pipeline for recurrence-free survival
analysis, and evaluate the effect of using different methods for each step
throughout the proposed segmentation-free outcome prediction pipeline.

##  3 Results

In this section we report the result of our proposed segmentation-free outcome
prediction model for recurrence-free survival prediction of HNC patients using
the multi-angle maximum intensity projections of their FDG PET images. Table 1
summarizes the result of all possible combinations of the feature extraction
backbones, pooling methods, and deep feature fusion methods, over all 5 test
folds, in terms of mean and standard deviation c-index. Rows of the Table 1
are divided into four sections each corresponding to one of four global
pooling methods. On average over all three different sizes of the feature
extraction method (EfficientNet B1, B4, and B7), and using all six different
feature fusion methods, global average pooling seems to be slightly more
effective compared to other pooling methods, with the mean c-index of $0.659$
compared to $0.653$, $0.645$, and $0.630$ for standard deviation, median, and
max pooling, respectively. However, the difference is minimal.

Comparing feature fusion methods, independent component analysis (ICA) method
seems to work poorly with respect to other fusion methods ($0.519$ on average
over four pooling methods and three backbone sizes). Among two other fusion
methods, on average, statistical summarization methods seems to outperform
non-linear transformation method (auto-encoder) with mean value over all five
channel-wise methods equal to $0.674$ compared to $0.668$ for the auto-
encoder. Between the four statistical summarization fusion methods, channel-
wise maximum fusion with the mean c-index of 0.686 outperforms other methods
with mean c-index of $0.673$, $0.672$, and $0.668$ for channel-wise standard
deviation, mean, and median, respectively. In order to compare the effect of
using the EfficientNet as feature extraction backbone with different sizes,
looking at the average of the c-index values over all the proposed pooling and
fusion methods, EfficientNet Large (B7) and EfficientNet Small (B1) with mean
c-index of $0.658$ and $0.654$ respectively, slightly outperforming the
EfficientNet Medium (B4) with the mean c-index of $0.630$, with the large size
performing minimally better than EfficienNet Small.

Feature Extractor | Mean | Median | Max | STD-Dev | ICA | AutoEncoder
---|---|---|---|---|---|---
Global Average Pooling |  |  |  |  |  |
EfficientNet Small (B1) |  $0.706\pm$0.05 |  $0.707\pm$0.05 |  $0.707\pm$0.05 |  $0.678\pm$0.04 |  $0.593\pm$0.06 |  $0.689\pm$0.08
EfficientNet Medium (B4) |  $0.615\pm$0.03 |  $0.638\pm$0.03 |  $0.626\pm$0.05 |  $0.640\pm$0.04 |  $0.590\pm$0.08 |  $0.644\pm$0.05
EfficientNet Large (B7) |  $0.718\pm$0.03 |  $0.710\pm$0.03 |  $0.719\pm$0.03 |  $0.705\pm$0.04 |  $0.455\pm$0.10 |  $0.721\pm$0.03
Global Max Pooling |  |  |  |  |  |
EfficientNet Small (B1) |  $0.613\pm$0.08 |  $0.613\pm$0.07 |  $0.703\pm$0.06 |  $0.680\pm$0.03 |  $0.541\pm$0.03 |  $0.639\pm$0.06
EfficientNet Medium (B4) |  $0.650\pm$0.07 |  $0.633\pm$0.06 |  $0.667\pm$0.04 |  $0.646\pm$0.04 |  $0.576\pm$0.07 |  $0.673\pm$0.05
EfficientNet Large (B7) |  $0.625\pm$0.03 |  $0.634\pm$0.03 |  $0.697\pm$0.05 |  $0.665\pm$0.04 |  $0.459\pm$0.03 |  $0.651\pm$0.03
Global Median Pooling |  |  |  |  |  |
EfficientNet Small (B1) |  $0.684\pm$0.06 |  $0.666\pm$0.07 |  $0.679\pm$0.04 |  $0.666\pm$0.04 |  $0.501\pm$0.08 |  $0.671\pm$0.05
EfficientNet Medium (B4) |  $0.652\pm$0.02 |  $0.663\pm$0.03 |  $0.651\pm$0.04 |  $0.641\pm$0.05 |  $0.491\pm$0.03 |  $0.630\pm$0.06
EfficientNet Large (B7) |  $0.688\pm$0.06 |  $0.678\pm$0.05 |  $0.701\pm$0.04 |  $0.697\pm$0.04 |  $0.556\pm$0.07 |  $0.698\pm$0.06
Global STD-Dev Pooling |  |  |  |  |  |
EfficientNet Small (B1) |  $0.702\pm$0.05 |  $0.701\pm$0.06 |  $0.701\pm$0.06 |  $0.709\pm$0.02 |  $0.456\pm$0.05 |  $0.685\pm$0.08
EfficientNet Medium (B4) |  $0.674\pm$0.04 |  $0.655\pm$0.03 |  $0.669\pm$0.04 |  $0.655\pm$0.05 |  $0.503\pm$0.03 |  $0.649\pm$0.03
EfficientNet Large (B7) |  $0.708\pm$0.04 |  $0.706\pm$0.04 |  $0.713\pm$0.04 |  $0.689\pm$0.06 |  $0.506\pm$0.04 |  $0.665\pm$0.04
Table 1:  This table summarizes the result of all combinations of feature
extraction CNN backbones, pooling methods, and deep feature fusion methods,
over all 5 test folds, in terms of mean and standard deviation c-index. Rows
of the table are divided into four sections, each corresponding to one of four
global pooling methods. Columns are also divided into three separate sections
each corresponding to one fusion method, from left to right channel-wise
statistical summarization methods, linear transformation method, and non-
linear transformation method.

Picking the EfficienNet Large as the better performing feature extraction
method, in order to compare the effectiveness of the proposed pooling methods,
figure 3 shows box plots of the c-index values of five test folds. Each plot
shows the results using one of the four proposed pooling methods, for
EfficientNet large, and all six different fusion techniques. Looking at the
median and the variability of the box plots, channel-wise maximum fusion
method seems performing slightly better than other statistical summarization
methods and the auto-encoder method. However, for the global average pooling
method, as the best pooling method among all proposed poolings, auto-encoder-
based fusion seems to perform on par with the channel-wise maximum and mean
fusion techniques. This is not true in other three diagrams showing results of
methods using other pooling techniques. On the other hand, looking at the four
statistical summarization methods and the non-linear transformation method
(auto-encoder) in all four diagrams of the figure 3, it seems that boxes are
more compact with relatively higher median c-index values and lower
variability compared to the boxes in other three diagrams corresponding to
other pooling methods.

Figure 3: Box plots of the c-index values over all five test folds. Each plot
shows the results using one of the four proposed pooling methods, for
EfficientNet large as the feature extraction method, and all six different
fusion techniques. Global average pooling, top left. Global max pooling top
right. Global median pooling, buttom left, and global standard deviation
pooling on the buttom right.

Likewise, figure 4 shows in box plots the performance of channel-wise maximum
fusion technique and the auto-encoder as the two best fusion methods over all
four different global pooling methods, over all five testing folds. Here as
can be seen, global average pooling has higher median c-index and lower
variability compared to other three pooling methods on both channel-wise
maximum fusion and the auto-encoder.

Figure 4: Box plots summarizing the performance of channel-wise maximum fusion technique(left) and the auto-encoder(right) as the two best fusion methods over all four different global pooling methods, and all five testing folds. Outcome Prediction Method | Mean c-index | Max | Min
---|---|---|---
L. Rebaud et al. [43] | $0.688$ | $0.732$ | $0.642$
EfficientNet Large + GAvgPool + CW-Mean (Ours) | $0.718$ | $0.749$ | $0.662$
EfficientNet Large + GAvgPool + CW-Median (Ours) | $0.710$ | $0.736$ | $0.655$
EfficientNet Large + GAvgPool + CW-Max (Ours) | $0.719$ | $0.751$ | $0.682$
EfficientNet Large + GAvgPool + CW-STD-Dev (Ours) | $0.705$ | $0.748$ | $0.657$
EfficientNet Large + GAvgPool + AutoEncoder (Ours) | $0.721$ | $0.742$ | $0.668$
Table 2: Comparison of the results of our proposed method, using global
average pooling and EfficientNet large as feature extractor, to the baseline
method [43] winner of the HECKTOR 2022 challenge for recurrence-free survival
analysis task, on the training part of the dataset, in terms of mean, maximum,
and minimum c-index over the test folds.

Table 2 shows the results of the method proposed in [43] who won the best
place in the HECKTOR 2022 challenge leaderboard for recurrence-free survival
analysis task, on the training part of the dataset, in terms of mean, maximum,
and minimum c-index over the test folds. Using global average pooling over the
deep features extracted using EfficientNet with large size, and all
statistical summarization and non-linear transformation fusion methods
outperforms the result reported by [43] on the dataset.

Feature Extractor | c-index
---|---
InceptionResNetV2 + GAvgPool + CW-Max |  $0.641\pm$0.06
InceptionV3 + GAvgPool + CW-Max |  $0.649\pm$0.05
Resnet-152 + GAvgPool + CW-Max |  $0.662\pm$0.06
VGG16 + GAvgPool + CW-Max |  $0.640\pm$0.07
VGG19 + GAvgPool + CW-Max |  $0.635\pm$0.03
Xception + GAvgPool + CW-Max |  $0.615\pm$0.09
DenseNet-201 + GAvgPool + CW-Max |  $0.663\pm$0.06
NASNet-Large + GAvgPool + CW-Max |  $0.644\pm$0.04
ConvNeXt-Base + GAvgPool + CW-Max |  $0.660\pm$0.06
EfficientNet Small (B1) + GAvgPool + CW-Max |  $0.707\pm$0.05
EfficientNet Medium (B4) + GAvgPool + CW-Max |  $0.626\pm$0.05
EfficientNet Large (B7) + GAvgPool + CW-Max |  $0.719\pm$0.03
Table 3: Result of using other architectures as feature extractors compared to
EfficientNet in terms of mean c-index over the five test folds.

We have also tried a few other well-known pre-trained CNN backbones for deep
feature extraction in order to compare to the EfficientNet used in our
proposed method. Table 3 summarizes the results of using 9 other pre-trained
CNN backbones in terms of mean c-index over the five test folds. For all these
feature extractors we used global average pooling and then channel-wise
maximum fusion method for the PFS prediction pipeline. While EfficientNet with
small and large sizes seem to outperform all the other feature extractors,
other backbones are performing better or on par with EfficientNet medium. We
report the result of all used CNN backbones, using all the proposed pooling
methods and feature fusion methods in Table 4.

##  4 Discussion

The present study proposed a segmentation-free outcome prediction method for
HNC patients. Instead of relying on manual segmentations, this work utilizes
an object detection method that is trained to find the anatomical head and
neck region of the patient on both coronal and sagittal MIPs of the patient’s
CT scan. The benefit of using such method for automatically cropping the
anatomical head and neck region is that, compared to the time and resource
demanding task of manual delineation of all the tumors and involved lymph
nodes, detecting the anatomical site of the primary disease is a less time and
resource demanding task. Defining the manual bounding boxes used in training
the object detection method does not demand an expert physician. Constraining
the entire space of the patient’s body to only the primary site of the disease
has also the advantage of helping the feature extraction methods to focus more
on the relevant information, in such a way that no delineation of the disease
related uptakes on the FDG PET will be needed.

This point also can be seen in previous works on the second round of MICCAI’s
HECKTOR challenge, where all the DICOM volumes were accompanied with 3D
bounding boxes with size of ${144\times 144\times 144}$ voxels containing the
extended oropharyngeal region of the patients. The provided bounding boxes not
only facilitated the use of 3D methods for automatic tumor segmentation due to
the high computation and memory demands of such methods, but also made room
for proposing outcome prediction methods extracting information directly from
the given bounding box, instead of relying on the segmentation masks of the
primary tumors, which resulted in the first and second best performing methods
for the outcome prediction task [44, 45] outperforming methods relying on
predicting the segmentation masks of the primary tumors. It is worth noting
that thanks to the recent success of methods such as TotalSegmentator [46],
training an object detection network to crop the region is not necessarily
needed. Instead, targeting the bones used in this work for defining the
anatomical head and neck site (section 2.3) one can use the segmentation masks
of TotalSegmentator and use it to crop the ROI.

Beside using a bounding box of the entire disease site for extracting
information (rather than manual delineations of the tumors/lymph nodes), in
the present study, instead of extracting image features from the voxels of the
3D volumes of interest, or pixels of trans-axial slices of the 3D volume,
feature extraction was performed on multiple maximum intensity projections of
the VOI from different trans-axial angels. MIP projections from multiple axial
angles can potentially enhance the representation of the malignancies (tumor
and involved lymph nodes) and capture a broader view of its metabolic
activity, while these information could be lost in common radiomics workflows.
We have experienced the effect of enhanced representation of the malignancies
in previous works for detection and segmentation of small metastatic lesions
of biochemically recurrent prostate cancer [31, 30]. The present work follows
the idea of using multi-angle PET MIPs in cancer diagnosis tasks, but here
specifically for the task of outcome prediction.

MIPs have been used in the literature for the task of outcome prediction. FDG
PET coronal and sagittal MIPs were used in [47] for calculating two surrogate
features to be replaced by their 3D equivalents, TMTV and Dmax, which were
shown to be highly predictive in lymphoma cancer management. In [48], a CNN
network was trained on MIP projections of FDG PET scans to predict the
probability of 2-year time-to-progression (TTP). However both mentioned works
used only the coronal and sagittal MIPs compared to 72 multi-angle MIPs in our
work. In [47], MIPs were used indirectly as a mean for estimating volumetric-
based biomarkers through defining surrogate biomarkers, while our proposed
method is using the information extracted from the MA-MIPs directly for the
task of outcome prediction. Moreover, efforts in both [48] and [47] were
either explicitly or implicitly aware of tumor segmentation masks, while our
proposed method is completely segmentation-free.

Beyond the aim of our work in proposing a segmentation-free approach for HNC
outcome prediction, there is a body of research on segmentation-free methods
but for other aims. For instance, [49] proposed a segmentation-free method for
directly estimating tumor volume and total amount of metabolic activity from
patient PET images, which are two important imaging biomarkers in disease
management and survival analysis of the patients. However, our present work
bypasses the step of estimating the aforementioned biomarkers and directly
predicts the outcome of the patients.

One of the main benefits of the proposed method in this work is the use of
pre-trained CNN backbones with frozen weights for feature extraction. This
property has multiple benefits. Firstly, using the pre-trained CNN backbones
with frozen weights and without fine-tuning on the target dataset, makes the
outcome prediction pipeline highly reproducible. This property is perfectly
aligned with efforts such as the Image Biomarker Standardization Initiative
(IBSI) for standardising the extraction of image biomarkers for high-
throughput quantitative image analysis (radiomics) [50]. Moreover, it can also
be beneficial in terms of computational resources.

Using the pre-trained CNN backbones as a pure transfer learning method,
without the need for training on the target domain dataset, will make the
outcome prediction pipeline highly portable, without need of high-end
computational resources like GPUs. This helps the proposed methods be more
generalizeable from limited training data and thus applicable in the clinic,
using limited computational resources or embedded/edge devices. In any case,
we plan to also study fine-tuning of the CNN backbones before feature
extraction to evaluate the effect of domain adaptation on the target dataset
on the task of outcome prediction.

| CW-Mean Fusion | CW-Median Fusion | CW-MAX Fusion | CW-STD-Dev Fusion | AutoEncoder Fusion
---|---|---|---|---|---
Global Max Pooling |  |  |  |  |
InceptionResNetV2 |  $0.604\pm$0.071 |  $0.613\pm$0.047 |  $0.549\pm$0.061 |  $0.559\pm$0.070 |  $0.630\pm$0.071
InceptionV3 |  $0.640\pm$0.056 |  $0.627\pm$0.041 |  $0.643\pm$0.049 |  $0.638\pm$0.037 |  $0.637\pm$0.050
Resnet |  $0.672\pm$0.064 |  $0.673\pm$0.058 |  $0.654\pm$0.058 |  $0.666\pm$0.056 |  $0.675\pm$0.062
VGG16 |  $0.644\pm$0.051 |  $0.644\pm$0.044 |  $0.686\pm$0.032 |  $0.631\pm$0.066 |  $0.599\pm$0.026
VGG19 |  $0.637\pm$0.027 |  $0.624\pm$0.042 |  $0.625\pm$0.023 |  $0.625\pm$0.054 |  $0.638\pm$0.045
Xception |  $0.613\pm$0.064 |  $0.607\pm$0.068 |  $0.608\pm$0.051 |  $0.615\pm$0.054 |  $0.618\pm$0.071
DenseNet201 |  $0.685\pm$0.060 |  $0.679\pm$0.053 |  $0.689\pm$0.054 |  $0.612\pm$0.037 |  $0.683\pm$0.026
EfficientNetB1 |  $0.613\pm$0.083 |  $0.613\pm$0.068 |  $0.703\pm$0.056 |  $0.680\pm$0.033 |  $0.639\pm$0.056
EfficientNetB4 |  $0.650\pm$0.068 |  $0.633\pm$0.055 |  $0.667\pm$0.036 |  $0.646\pm$0.044 |  $0.673\pm$0.053
EfficientNetB7 |  $0.625\pm$0.031 |  $0.634\pm$0.032 |  $0.697\pm$0.052 |  $0.665\pm$0.041 |  $0.651\pm$0.033
Global Average Pooling |  |  |  |  |
InceptionResNetV2 |  $0.605\pm$0.066 |  $0.612\pm$0.065 |  $0.641\pm$0.059 |  $0.627\pm$0.070 |  $0.612\pm$0.078
InceptionV3 |  $0.635\pm$0.052 |  $0.630\pm$0.049 |  $0.649\pm$0.048 |  $0.654\pm$0.043 |  $0.651\pm$0.050
Resnet |  $0.676\pm$0.076 |  $0.675\pm$0.076 |  $0.662\pm$0.055 |  $0.658\pm$0.053 |  $0.680\pm$0.077
VGG16 |  $0.641\pm$0.052 |  $0.624\pm$0.056 |  $0.640\pm$0.072 |  $0.614\pm$0.092 |  $0.655\pm$0.061
VGG19 |  $0.641\pm$0.053 |  $0.646\pm$0.080 |  $0.635\pm$0.034 |  $0.652\pm$0.046 |  $0.636\pm$0.047
Xception |  $0.614\pm$0.080 |  $0.593\pm$0.081 |  $0.615\pm$0.088 |  $0.614\pm$0.080 |  $0.640\pm$0.077
DenseNet201 |  $0.668\pm$0.061 |  $0.665\pm$0.066 |  $0.663\pm$0.064 |  $0.653\pm$0.055 |  $0.683\pm$0.074
EfficientNetB1 |  $0.706\pm$0.048 |  $0.707\pm$0.051 |  $0.707\pm$0.051 |  $0.678\pm$0.035 |  $0.689\pm$0.076
EfficientNetB4 |  $0.615\pm$0.028 |  $0.638\pm$0.029 |  $0.626\pm$0.051 |  $0.640\pm$0.041 |  $0.644\pm$0.048
EfficientNetB7 |  $0.718\pm$0.033 |  $0.710\pm$0.033 |  $0.719\pm$0.032 |  $0.705\pm$0.036 |  $0.721\pm$0.034
Global Median Pooling |  |  |  |  |
InceptionResNetV2 |  $0.628\pm$0.062 |  $0.636\pm$0.063 |  $0.635\pm$0.033 |  $0.641\pm$0.048 |  $0.630\pm$0.045
InceptionV3 |  $0.632\pm$0.038 |  $0.655\pm$0.044 |  $0.660\pm$0.036 |  $0.645\pm$0.065 |  $0.628\pm$0.057
Resnet |  $0.613\pm$0.070 |  $0.625\pm$0.100 |  $0.645\pm$0.093 |  $0.633\pm$0.090 |  $0.596\pm$0.078
VGG16 |  $0.571\pm$0.038 |  $0.574\pm$0.025 |  $0.557\pm$0.041 |  $0.556\pm$0.049 |  $0.582\pm$0.055
VGG19 |  $0.524\pm$0.091 |  $0.531\pm$0.056 |  $0.544\pm$0.069 |  $0.553\pm$0.101 |  $0.561\pm$0.061
Xception |  $0.579\pm$0.073 |  $0.595\pm$0.080 |  $0.612\pm$0.095 |  $0.606\pm$0.092 |  $0.613\pm$0.058
DenseNet201 |  $0.644\pm$0.066 |  $0.653\pm$0.068 |  $0.634\pm$0.088 |  $0.618\pm$0.065 |  $0.665\pm$0.061
EfficientNetB1 |  $0.684\pm$0.061 |  $0.666\pm$0.056 |  $0.679\pm$0.038 |  $0.666\pm$0.041 |  $0.671\pm$0.051
EfficientNetB4 |  $0.652\pm$0.017 |  $0.663\pm$0.027 |  $0.651\pm$0.040 |  $0.641\pm$0.045 |  $0.630\pm$0.055
EfficientNetB7 |  $0.688\pm$0.057 |  $0.678\pm$0.048 |  $0.701\pm$0.042 |  $0.697\pm$0.041 |  $0.698\pm$0.057
Global STD-Dev Pooling |  |  |  |  |
InceptionResNetV2 |  $0.620\pm$0.070 |  $0.624\pm$0.046 |  $0.572\pm$0.083 |  $0.546\pm$0.069 |  $0.638\pm$0.062
InceptionV3 |  $0.634\pm$0.634 |  $0.635\pm$0.058 |  $0.643\pm$0.040 |  $0.638\pm$0.040 |  $0.639\pm$0.066
Resnet |  $0.677\pm$0.071 |  $0.679\pm$0.072 |  $0.643\pm$0.061 |  $0.663\pm$0.058 |  $0.641\pm$0.053
VGG16 |  $0.654\pm$0.046 |  $0.648\pm$0.035 |  $0.661\pm$0.064 |  $0.630\pm$0.051 |  $0.628\pm$0.042
VGG19 |  $0.644\pm$0.055 |  $0.635\pm$0.073 |  $0.627\pm$0.058 |  $0.629\pm$0.056 |  $0.638\pm$0.024
Xception |  $0.607\pm$0.066 |  $0.618\pm$0.082 |  $0.622\pm$0.077 |  $0.606\pm$0.067 |  $0.617\pm$0.082
DenseNet201 |  $0.670\pm$0.049 |  $0.664\pm$0.032 |  $0.657\pm$0.045 |  $0.655\pm$0.031 |  $0.682\pm$0.045
EfficientNetB1 |  $0.702\pm$0.051 |  $0.701\pm$0.058 |  $0.701\pm$0.058 |  $0.709\pm$0.015 |  $0.685\pm$0.075
EfficientNetB4 |  $0.674\pm$0.041 |  $0.655\pm$0.032 |  $0.669\pm$0.044 |  $0.655\pm$0.047 |  $0.649\pm$0.026
EfficientNetB7 |  $0.708\pm$0.041 |  $0.706\pm$0.038 |  $0.713\pm$0.036 |  $0.689\pm$0.062 |  $0.665\pm$0.040
Table 4: Complete result of all used pre-trained CNN backbones, using all the
four proposed pooling methods and five feature fusion methods from two groups
of channel-wise statistical summarization fusions and non-linear
transformation fusion method in terms of mean and std-dev of c-index over all
five test folds.

##  5 Conclusion

We successfully developed an easy-to-perform segmentation-free outcome
prediction method, specifically for predicting progression-free survival of
head and neck cancer patients from their FDG PET-CT image volumes. We found
that by using multiple rotation maximum intensity projections of the PET
images, and focusing the anatomical head and neck region guided by CT scan
images of the patients, outcome prediction is not only feasible, but is also
able to outperform conventional methods, without the need of manual
segmentation of the primary tumors or the involved/metastatic lymph nodes by
the expert nuclear medicine physicians.

## Acknowledgment

This work was supported by the Canadian Institutes of Health Research (CIHR)
Project Grant PJT-162216, Natural Sciences and Engineering Research Council of
Canada (NSERC) Discovery Grant RGPIN-2019–06467, and through computational
resources and services provided by Microsoft for Health.

## References

  * [1] Mark Gormley, Grant Creaney, Andrew Schache, Kate Ingarfield, and David I Conway.  Reviewing the epidemiology of head and neck cancer: definitions, trends and risk factors.  Br. Dent. J., 233(9):780–786, November 2022.
  * [2] Mayur D Mody, James W Rocco, Sue S Yom, Robert I Haddad, and Nabil F Saba.  Head and neck cancer.  The Lancet, 398(10318):2289–2299, 2021.
  * [3] Shanthi Marur, Gypsyamber D’Souza, William H Westra, and Arlene A Forastiere.  Hpv-associated head and neck cancer: a virus-related cancer epidemic.  The lancet oncology, 11(8):781–789, 2010.
  * [4] Paolo Bossi, Salvatore Alfieri, Primoz Strojan, Robert P Takes, Fernando López, Antti Mäkitie, Nabil F Saba, Juan Pablo Rodrigo, Carol Bradford, Carlos Suarez, Mark Zafereo, Arlene A Forastiere, Jan B Vermorken, Miquel Quer, Alvaro Sanabria, Ricard Simo, Remco de Bree, Alessandra Rinaldo, and Alfio Ferlito.  Prognostic and predictive factors in recurrent and/or metastatic head and neck squamous cell carcinoma: A review of the literature.  Crit. Rev. Oncol. Hematol., 137:84–91, May 2019.
  * [5] Atif A Hashmi, Syeda N Iftikhar, Rimsha Haider, Nabeel N Baig, Muhammad Ghani Asif, and Muhammad Irfan.  Recurrence and Disease-Free survival in head and neck squamous cell carcinoma after Margin-Free resection on frozen section: An institutional perspective.  Cureus, 12(11):e11385, November 2020.
  * [6] Julia A Woolgar.  Histopathological prognosticators in oral and oropharyngeal squamous cell carcinoma.  Oral Oncol., 42(3):229–239, March 2006.
  * [7] Panagiota Economopoulou, Remco de Bree, Ioannis Kotsantis, and Amanda Psyrri.  Diagnostic tumor markers in head and neck squamous cell carcinoma (HNSCC) in the clinical setting.  Front. Oncol., 9:827, August 2019.
  * [8] Akram Al-Ibraheem, Andreas Buck, Bernd Joachim Krause, Klemens Scheidhauer, Markus Schwaiger, et al.  Clinical applications of fdg pet and pet/ct in head and neck cancer.  Journal of oncology, 2009, 2009.
  * [9] Paola Castaldi, Lucia Leccisotti, Francesco Bussu, Francesco Micciche, and Vittoria Rufini.  Role of 18f-fdg pet-ct in head and neck squamous cell carcinoma.  Acta Otorhinolaryngologica Italica, 33(1):1, 2013.
  * [10] Sarah Differding, François-Xavier Hanin, and Vincent Grégoire.  Pet imaging biomarkers in head and neck cancer.  European journal of nuclear medicine and molecular imaging, 42:613–622, 2015.
  * [11] Martin Vallières, Emily Kay-Rivest, Léo Jean Perrin, Xavier Liem, Christophe Furstoss, Hugo JWL Aerts, Nader Khaouam, Phuc Felix Nguyen-Tan, Chang-Shu Wang, Khalil Sultanem, et al.  Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer.  Scientific reports, 7(1):10117, 2017.
  * [12] Andrew J Wong, Aasheesh Kanwar, Abdallah S Mohamed, and Clifton D Fuller.  Radiomics in head and neck cancer: from exploration to application.  Translational cancer research, 5(4):371, 2016.
  * [13] S Kevin Zhou, Daniel Rueckert, and Gabor Fichtinger.  Handbook of Medical Image Computing and Computer Assisted Intervention.  Academic Press, October 2019.
  * [14] E A Zanaty and Said Ghoniemy.  Medical image segmentation techniques: an overview.  International Journal of informatics and medical data processing, 1(1):16–37, 2016.
  * [15] Giorgio Davico, Francesca Bottin, Alberto Di Martino, Vanita Castafaro, Fabio Baruffaldi, Cesare Faldini, and Marco Viceconti.  Intra-operator repeatability of manual segmentations of the hip muscles on clinical magnetic resonance images.  J. Digit. Imaging, 36(1):143–152, February 2023.
  * [16] Azka Rehman, Muhammad Usman, Abdullah Shahid, Siddique Latif, and Junaid Qadir.  Selective deeply supervised Multi-Scale attention network for brain tumor segmentation.  Sensors, 23(4), February 2023.
  * [17] Saeed Iqbal, Adnan N Qureshi, Jianqiang Li, and Tariq Mahmood.  On the analyses of medical images using traditional machine learning techniques and convolutional neural networks.  Arch. Comput. Methods Eng., 30(5):3173–3233, April 2023.
  * [18] Xènia Albà, Karim Lekadir, Marco Pereañez, Pau Medrano-Gracia, Alistair A Young, and Alejandro F Frangi.  Automatic initialization and quality control of large-scale cardiac MRI segmentations.  Med. Image Anal., 43:129–141, January 2018.
  * [19] Jillian Cardinell.  Investigating the Effect of Annotation Styles on the Generalizability of Medical Deep Learning Algorithms.  McGill University (Canada), 2022.
  * [20] Prabhakar Rajiah and Michael A Bolen.  Cardiovascular MR imaging at 3 t: opportunities, challenges, and solutions.  Radiographics, 34(6):1612–1635, October 2014.
  * [21] Daniel E Johnson, Barbara Burtness, C René Leemans, Vivian Wai Yan Lui, Julie E Bauman, and Jennifer R Grandis.  Head and neck squamous cell carcinoma.  Nat Rev Dis Primers, 6(1):92, November 2020.
  * [22] Valentin Oreiller, Vincent Andrearczyk, Mario Jreige, Sarah Boughdad, Hesham Elhalawani, Joel Castelli, Martin Vallières, Simeng Zhu, Juanying Xie, Ying Peng, Andrei Iantsen, Mathieu Hatt, Yading Yuan, Jun Ma, Xiaoping Yang, Chinmay Rao, Suraj Pai, Kanchan Ghimire, Xue Feng, Mohamed A Naser, Clifton D Fuller, Fereshteh Yousefirizi, Arman Rahmim, Huai Chen, Lisheng Wang, John O Prior, and Adrien Depeursinge.  Head and neck tumor segmentation in PET/CT: The HECKTOR challenge.  Med. Image Anal., 77:102336, April 2022.
  * [23] Flóra John, Edit Bosnyák, Natasha L Robinette, Alit J Amit-Yousif, Geoffrey R Barger, Keval D Shah, Sharon K Michelhaugh, Neil V Klinger, Sandeep Mittal, and Csaba Juhász.  Multimodal imaging-defined subregions in newly diagnosed glioblastoma: impact on overall survival.  Neuro. Oncol., 21(2):264–273, February 2019.
  * [24] Saima Rathore, Hamed Akbari, Martin Rozycki, Kalil G Abdullah, Maclean P Nasrallah, Zev A Binder, Ramana V Davuluri, Robert A Lustig, Nadia Dahmane, Michel Bilello, Donald M O’Rourke, and Christos Davatzikos.  Radiomic MRI signature reveals three distinct subtypes of glioblastoma with different clinical and molecular characteristics, offering prognostic value beyond IDH1.  Sci. Rep., 8(1):5087, March 2018.
  * [25] Valentin Oreiller, Vincent Andrearczyk, Mario Jreige, Sarah Boughdad, Hesham Elhalawani, Joel Castelli, Martin Vallières, Simeng Zhu, Juanying Xie, Ying Peng, et al.  Head and neck tumor segmentation in pet/ct: the hecktor challenge.  Medical image analysis, 77:102336, 2022.
  * [26] Vincent Andrearczyk, Valentin Oreiller, Mathieu Hatt, and Adrien Depeursinge.  Head and neck tumor segmentation and outcome prediction.  UK: Springer, 2022.
  * [27] Vincent Andrearczyk, Valentin Oreiller, Moamen Abobakr, Azadeh Akhavanallaf, Panagiotis Balermpas, Sarah Boughdad, Leo Capriotti, Joel Castelli, Catherine Cheze Le Rest, Pierre Decazes, Ricardo Correia, Dina El-Habashy, Hesham Elhalawani, Clifton D Fuller, Mario Jreige, Yornna Khamis, Agustina La Greca, Abdallah Mohamed, Mohamed Naser, John O Prior, Su Ruan, Stephanie Tanadini-Lang, Olena Tankyevych, Yazdan Salimi, Martin Vallières, Pierre Vera, Dimitris Visvikis, Kareem Wahid, Habib Zaidi, Mathieu Hatt, and Adrien Depeursinge.  Overview of the HECKTOR challenge at MICCAI 2022: Automatic head and neck tumor segmentation and outcome prediction in PET/CT.  Head Neck Tumor Chall (2022), 13626:1–30, March 2023.
  * [28] Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhao-Xiang Zhang.  Scale-aware trident networks for object detection.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6054–6063. IEEE, October 2019.
  * [29] Thomas Melchardt, Teresa Magnes, Clemens Hufnagl, Aaron R Thorner, Matthew Ducar, Daniel Neureiter, Wolfgang Tränkenschuh, Eckhard Klieser, Alexander Gaggl, Sebastian Rösch, et al.  Clonal evolution and heterogeneity in metastatic head and neck cancer—an analysis of the austrian study group of medical tumour therapy study group.  European Journal of Cancer, 93:69–78, 2018.
  * [30] Amirhosein Toosi, Sara Harsini, Francois Benard, Carlos Uribe, and Arman Rahmim.  Advanced deep learning-based lesion detection on rotational 2d maximum intensity projection (mip) images coupled with reverse mapping to the 3d pet domain, 2023.
  * [31] Amirhosein Toosi, Sara Harsini, Shadab Ahamed, Fereshteh Yousefirizi, François Bénard, Carlos Uribe, and Arman Rahmim.  State-of-the-art object detection algorithms for small lesion detection in psma pet: use of rotational maximum intensity projection (mip) images.  In Medical Imaging 2023: Image Processing, volume 12464, pages 771–778. SPIE, 2023.
  * [32] Padmavathi Kora, Chui Ping Ooi, Oliver Faust, U Raghavendra, Anjan Gudigar, Wai Yee Chan, K Meenakshi, K Swaraja, Pawel Plawiak, and U Rajendra Acharya.  Transfer learning techniques for medical image analysis: A review.  Biocybernetics and Biomedical Engineering, 42(1):79–107, 2022.
  * [33] Laith Alzubaidi, Muthana Al-Amidie, Ahmed Al-Asadi, Amjad J Humaidi, Omran Al-Shamma, Mohammed A Fadhel, Jinglan Zhang, Jesus Santamaría, and Ye Duan.  Novel transfer learning approach for medical imaging with limited labeled data.  Cancers, 13(7):1590, 2021.
  * [34] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [35] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * [36] Laith Alzubaidi, Jinglan Zhang, Amjad J Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma, José Santamaría, Mohammed A Fadhel, Muthana Al-Amidie, and Laith Farhan.  Review of deep learning: concepts, cnn architectures, challenges, applications, future directions.  Journal of big Data, 8:1–74, 2021.
  * [37] Hong Hui Tan and King Hann Lim.  Vanishing gradient mitigation with deep learning neural network optimization.  In 2019 7th international conference on smart computing & communications (ICSCC), pages 1–4. IEEE, 2019.
  * [38] George Philipp, Dawn Song, and Jaime G Carbonell.  The exploding gradient problem demystified-definition, prevalence, impact, origin, tradeoffs, and solutions.  arXiv preprint arXiv:1712.05577, 2017.
  * [39] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [40] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [41] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [42] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [43] Louis Rebaud, Thibault Escobar, Fahad Khalid, Kibrom Girum, and Irène Buvat.  Simplicity is all you need: out-of-the-box nnunet followed by binary-weighted radiomic model for segmentation and outcome prediction in head and neck pet/ct.  In 3D Head and Neck Tumor Segmentation in PET/CT Challenge, pages 121–134. Springer, 2022.
  * [44] Numan Saeed, Roba Al Majzoub, Ikboljon Sobirov, and Mohammad Yaqub.  An ensemble approach for patient prognosis of head and neck tumor using multimodal data.  In Head and Neck Tumor Segmentation and Outcome Prediction, pages 278–286. Springer International Publishing, 2022.
  * [45] Mohamed A Naser, Kareem A Wahid, Abdallah S R Mohamed, Moamen Abobakr Abdelaal, Renjie He, Cem Dede, Lisanne V van Dijk, and Clifton D Fuller.  Progression free survival prediction for head and neck cancer using deep learning based on clinical and PET/CT imaging data.  Head Neck Tumor Segm Chall (2021), 13209:287–299, March 2022.
  * [46] Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al.  Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images.  Radiology: Artificial Intelligence, 5(5), 2023.
  * [47] Kibrom B Girum, Louis Rebaud, Anne-Ségolène Cottereau, Michel Meignan, Jérôme Clerc, Laetitia Vercellino, Olivier Casasnovas, Franck Morschhauser, Catherine Thieblemont, and Irène Buvat.  18f-fdg pet maximum-intensity projections and artificial intelligence: a win-win combination to easily measure prognostic biomarkers in dlbcl patients.  Journal of Nuclear Medicine, 63(12):1925–1932, 2022.
  * [48] Maria C Ferrández, Sandeep SV Golla, Jakoba J Eertink, Bart M de Vries, Pieternella J Lugtenburg, Sanne E Wiegers, Gerben JC Zwezerijnen, Simone Pieplenbosch, Lars Kurch, Andreas Hüttmann, et al.  An artificial intelligence method using fdg pet to predict treatment outcome in diffuse large b cell lymphoma patients.  Scientific Reports, 13(1):13111, 2023.
  * [49] Saeid Asgari Taghanaki, Noirin Duggan, Hillgan Ma, Xinchi Hou, Anna Celler, Francois Benard, and Ghassan Hamarneh.  Segmentation-free direct tumor volume and metabolic activity estimation from pet scans.  Computerized Medical Imaging and Graphics, 63:52–66, 2018.
  * [50] Alex Zwanenburg, Martin Vallières, Mahmoud A Abdalah, Hugo JWL Aerts, Vincent Andrearczyk, Aditya Apte, Saeed Ashrafinia, Spyridon Bakas, Roelof J Beukinga, Ronald Boellaard, et al.  The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping.  Radiology, 295(2):328–338, 2020.