# An end-to-end deep learning pipeline to derive blood input with partial
volume corrections for automated parametric brain PET mapping

Rugved Chavan*
Department of Computer Science and Engineering
Department of Radiology and Medical Imaging
University of Virginia
<EMAIL_ADDRESS>
&Gabriel Hyman*
Department of Biomedical Engineering
Department of Computer Science
Department of Radiology and Medical Imaging
University of Virginia
&Zoraiz Qureshi*
Department of Computer Science and Engineering
Department of Radiology and Medical Imaging
University of Virginia
&Nivetha Jayakumar*
Department of Computer Science and Engineering
Department of Radiology and Medical Imaging
University of Virginia
&William Terrell
Department of Computer Science and Engineering
Department of Radiology and Medical Imaging
University of Virginia
&Stuart Berr
Department of Radiology and Medical Imaging
University of Virginia
&David Schiff
Department of Neurology
University of Virginia
&Megan Wardius
Brain Institute
University of Virginia
&Nathan Fountain
Department of Neurology
University of Virginia
&Thomas Muttikkal
Department of Radiology and Medical Imaging
University of Virginia
&Mark Quigg
Department of Neurology
University of Virginia
&Miaomiao Zhang
Department of Computer Science and Engineering
University of Virginia
&Bijoy Kundu
Department of Radiology and Medical Imaging
Department of Biomedical Engineering
University of Virginia
<EMAIL_ADDRESS>
###### Abstract

Dynamic 2-[18F] fluoro-2-deoxy-D-glucose positron emission tomography (dFDG-
PET) for human brain imaging has considerable clinical potential, yet its
utilization remains limited. A key challenge in the quantitative analysis of
dFDG-PET is characterizing a patient-specific blood input function,
traditionally reliant on invasive arterial blood sampling. This research
introduces a novel approach employing non-invasive deep learning model-based
computations from the internal carotid arteries (ICA) with partial volume (PV)
corrections, thereby eliminating the need for invasive arterial sampling. We
present an end-to-end pipeline incorporating a 3D U-Net based ICA-net for ICA
segmentation, alongside a Recurrent Neural Network (RNN) based MCIF-net for
the derivation of a model-corrected blood input function (MCIF) with PV
corrections. The developed 3D U-Net and RNN was trained and validated using a
5-fold cross-validation approach on 50 human brain FDG PET datasets. The ICA-
net achieved an average Dice score of 82.18% and an Intersection over Union of
68.54% across all tested scans. Furthermore, the MCIF-net exhibited a minimal
root mean squared error of 0.0052. The application of this pipeline to ground
truth data for dFDG-PET brain scans resulted in the precise localization of
seizure onset regions, which contributed to a successful clinical outcome,
with the patient achieving a seizure-free state after treatment. These results
underscore the efficacy of the ICA-net and MCIF-net deep learning pipeline in
learning the ICA structure’s distribution and automating MCIF computation with
PV corrections. This advancement marks a significant leap in non-invasive
neuroimaging.

* Indicate authors contributed equally to this work.

_K_ eywords Dynamic FDG-PET $\cdot$ Non-invasive Brain Imaging $\cdot$ Deep
Learning models $\cdot$ 3D U-Net $\cdot$ LSTM $\cdot$ PET Seizure Localization

##  1 Introduction

Positron emission tomography (PET) is a medical imaging modality that has been
used over three decades [1]. Unlike other medical imaging modalities,
including magnetic resonance imaging (MRI) and computed tomography (CT)—which
provide high-resolution information about morphological and anatomical
structures—PET reveals in vivo physiological and biochemical function of
target organs [2]. The fundamental element that facilitates this scintigraphic
scan is the radiotracer, a molecule known to target the biological pathway of
interest linked to a radioisotope. Upon intravenous injection, the radiotracer
is transported and uptaken by tissue of interest. The most commonly used
radiotracer is 2-[18F] fluoro-2-deoxy-D-glucose (FDG), which is an analog of
glucose and, through its use in PET, allows for in vivo visualization of
glucose metabolism. With FDG-PET, clinicians can identify pathophysiological
regions characterized by abnormal glucose regulation, ranging from states of
infection [3] to diseases like rheumatoid arthritis [4] and cancer [2], [5].
In particular, FDG-PET of the human brain has become considerably popular in
the recent years and can improve neuropathological evaluation of a variety of
brain disease states, including glioblastoma [6] [7][8], dementias [9], [10],
medically intractable epilepsy [11], [12] [13], traumatic brain injury [14],
and most recently, COVID-19 [15].

sFDG-PET vs. dFDG-PET : Static FDG-PET (sFDG-PET) is the most conventionally
used form of PET for the human brain [12], [16]. The patient is injected with
the radiotracer and, typically after a 45-minute window of waiting, is imaged
in the scanner for 15 minutes [12]. The result of this process is a single
snapshot (averaged over the 15-minute scan) of neuronal glucose metabolism,
which is generally only qualitatively interpretable [2]. Dynamic FDG-PET
(dFDG-PET) of the human brain is a newer and emerging form of PET that allows
for both the visualization and quantification of the same biological pathway
information, but instead as a function of time, rather than a single snapshot.
In dFDG-PET the patient is first put on the scanner and a 60-90 minute scan
initiated followed by a slow injection of the radiotracer over a period of
5-10 seconds, thereby capturing data from the point of injection. Glucose
metabolic activity of the brain is captured at discrete time points throughout
the scan, which results in a 4D-object consisting of multiple 3D-volumes of
pathway activity [12]. Recent studies have highlighted the additional clinical
utility of dFDG-PET compared to sFDG-PET across a variety of disease states
[5], [11], [12], [16]– [17]. By capturing both the temporal and spatial
pattern of radiotracer uptake, dFDG-PET provides clinicians with substantially
more information about in vivo pathway activity and even reduces the negative
impact of undesired artifacts present in sFDG-PET [2].

IDIF: Quantitative analysis of these dynamic scans requires an understanding
of patient-specific radiotracer biochemistry, which is given by the blood
input function. The blood input function is a time-activity curve (TAC) that
describes the amount of radiotracer in the blood available to tissue as a
function of time throughout the course of the scan. The gold standard for
measuring this function is through time-distributed arterial blood sampling
during the scan, which is costly [18] and risks patient infection or arterial
occlusion [19]. Researchers, however, have recently discovered a way by which
this kinetic radiotracer information can be successfully extracted directly
from the image data [17]. This non-invasive measurement is known as an image-
derived blood input function (IDIF) and monitors how the concentration of
radiotracer (which, in dFDG-PET, is highly correlated to voxel intensity)
varies across a region-of-interest (ROI) during the scan. The carotid
arteries, which supply the brain with oxygenated blood and, thus, radiotracer,
are the gold standard ROI to segment out of the scan and from which to derive
the IDIF [12], [20]–[21]. Despite the benefit of using an IDIF in the place of
arterial blood sampling to quantify radiotracer kinetics, the associated
annotation process serves as a major roadblock to the wider adoption of dFDG-
PET. Manually segmenting out the carotid arteries is a highly imprecise and
time-consuming process and requires a trained technician to perform [21].
Recognizing these limitations, one of the two objectives of this work is to
implement supervised machine learning (ML) techniques to develop a pipeline
capable of automatically segmenting carotid ROIs and performing subsequent
IDIF calculations, ideally facilitating broader adoption of dFDG-PET for human
brain imaging.

MCIF: However, PET scanners typically have inherently low resolution [33],
which makes raw IDIFs vulnerable to spillover (SP) from adjacent tissues and
partial volume (PV) effects. Spillover (SP) effects occur when the signal from
a high-activity region ’spills over’ into adjacent lower-activity regions,
leading to overestimation of tracer concentration in these adjacent areas.
This phenomenon is particularly evident in small structures adjacent to high
uptake tissues. On the other hand, partial volume (PV) effects arise due to
the finite spatial resolution of PET scanners, where the measured signal is a
blend of activities from different tissues within the same voxel. This
blending leads to an underestimation of peak tracer concentration in small
structures, as the signal from these structures is diluted by surrounding
tissue with lower tracer uptake. So these issues arise when analyzing image
regions with radii smaller than the spatial resolution of the PET scanner,
such as the carotid arteries. To more accurately estimate the blood input
function, we correct the IDIF with a 3-compartment model to obtain the Model
Corrected Blood Input Function (MCIF). In our previous research, we determined
the MCIF by regressing the 3-compartment model to the IDIF and the TAC of the
tissue surrounding the ICA, utilizing a custom loss function. Due to the large
number of parameters of the 3-compartment model (15) and the non-convexity of
this optimization task, prior bounds for the parameters had to be manually
determined. The second objective of this work is to advance this approach by
implementing a deep learning-based calculation of this model. We developed a
recurrent neural network (RNN) based architecture, MCIF-net, designed
specifically for predicting MCIF from IDIF.

##  2 Materials and Methods

In this section, we will discuss the dataset’s collection and preparation, the
comprehensive end-to-end pipeline, the architecture of the model, and the
evaluation metrics employed for assessing performance.

###  2.1 Dataset

The dFDG-PET imaging was conducted on a cohort comprising of 50 participants.
This procedure was performed using the Siemens Biograph time-of-flight mCT
scanner. The scans featured a resolution of 400 pixels x 400 pixels x 110
slices x 38 timeframes, and were conducted with time-dependent attenuation
correction. The dynamic acquisition process involved the initiation of a
60-minute scan followed immediately with an intravenous administration of
approximately 10 mCi FDG, injected over a duration of 10 seconds. All data
were reconstructed in a list-mode format. Prior to the dFDG-PET procedure, all
50 patients underwent T1-weighted MPRAGE MPI (256 pixels x 256 pixels x 192
slices) using a Siemens 3T scanner for co-registration and anatomic mapping.

Figure 1: Abstract overview of the complete pipeline with two disjoint
supervised models for segmentation and blood input correction

###  2.2 End to End Pipeline

The pipeline comprises four distinct phases. The first phase encompasses data
preprocessing, which includes motion correction and co-registration of the
dynamic FDG-PET (dFDG-PET) data. The second phase utilizes the 4D motion-
corrected and co-registered data for internal carotid arteries (ICA)
segmentation through ICA-net. The third phase involves the computation of the
Image-Derived Blood Input Function (IDIF), monitoring changes in intensity
within the ICA region. This phase also includes the application of MCIF-net
for correcting PV and SP errors, resulting in the final MCIF. The fourth phase
employs the MCIF to calculate the Ki Map using the Patlak model. Subsequently,
for patients with epilepsy and known surgical ground truth, we compute the
Z-score, normalizing it against the mean and standard deviation (SD) for the
whole brain, encompassing 18 super regions per side. Figure 1 delineates the
workflow of this pipeline.

In this work, we automated an integrated, end-to-end pipeline by developing
ICA-net for ICA segmentation and MCIF-net for partial volume corrections. This
system is specifically tailored for the automatic identification of seizure
foci in human dynamic FDG brain PET imaging.

####  2.2.1 Data Pre-processing

The initial step in pre-processing started with motion correction for the
60-minute acquisition to align and lock the anatomy in the same 3 dimensional
space throughout the entire time period. PET data (400 pixels x 400 pixels x
111 slices x 38-time frames) was averaged across the first 14 time frames to
create a reference for the proper alignment in the image space. This reference
was used to perform a rigid body transform across the 38 frames for motion
correction. Next, a new average frame of all the motion corrected PET frames
was resliced and co-registered into MRI space using the T1 weighted MRI using
non-rigid transforms to generate a transformation matrix. This was used, in
turn, to generate a co-registered dynamic PET volume. All the above
registration processes were designed using software from the FMRIB’s Software
Library (FSL) tool kit [22] [23] [24] [25] [26] [12] (Figure 1).

####  2.2.2 Segmentation of Internal Carotid Arteries

Frame-selection:

This first step of this algorithm was to perform a center crop of the first
frame based on the central position of the carotids with respect to the neck
and brain. Next, the algorithm calculated the sum of intensity across all
voxels in the cropped region. These two steps were then performed iteratively
across the first 10 frames, generating a plot of total intensity as a function
of scan timeframe. From this discrete function, the difference between each
frame’s summed intensity and the previous frame’s summed intensity was
calculated, generating a plot of summed intensity difference with respect to
the previous frame as a function of scan timeframe (with the first frame
having an intensity difference of its original summed intensity value). With
these differences, the algorithm then selected the frame that occurred one
frame before the first local maximum. This local maximum indicated a large
radiotracer presence distributed throughout the brain, meaning that the
majority of the radiotracer had already been transported through the carotids.
The previous frame, therefore, represented the frame in which a considerable
amount of radiotracer was still being transported through the carotids and the
arteries were, thus, most clear for segmentation. All incorrectly selected
frames were no more than 1 frame off.

Semi-automated segmentation:

To optimize the efficiency of this research, addressing the time-consuming
nature of manual segmentation of carotids, which typically required 35-45
minutes per scan, was essential. Hence, a semi-automated method was devised
for generating ground truth annotations across all 50 datasets. This approach
utilized the imaging visualization and analysis capabilities of 3D Slicer,
particularly leveraging various functions within its Segment Editor module.

Given that the carotid arteries are responsible for supplying blood and,
consequently, radiotracer to the brain, they exhibit high-intensity signals in
their lumen. Utilizing 3D Slicer’s Threshold function, an intensity threshold
was empirically determined—sometimes through trial-and-error—to segment
regions of the scan with voxel intensities equal to or greater than this
threshold. The ‘Show 3D’ option was then activated to facilitate the
visualization of the 3D-rendered segmentation.

Recognizing that the carotids are relatively large and continuous structures
within the scanned area, another empirically determined threshold was set
using the Islands function. This function, particularly the ’Split islands to
segment’ option, was employed to isolate structures with a continuous length
meeting or exceeding this threshold. Following the execution of these
functions, typically only two structures remained. These were amalgamated
using the Logical Operator function, applying the ‘Operation: Add to each.’
Subsequently, one of the objects under ‘Name’ was deleted. Manual removal of
extraneous regions and noise was performed using the Scissors function.

To ensure accurate annotations, each slice was meticulously reviewed and
verified, resulting in carotid segmentations that generally encompassed 40-60
slices. Notably, this methodology consistently produced results in less than 5
minutes. To advance our research, we resolved to utilize the segmented
internal carotid arteries (ICA) as the ground truth and the 3D dynamic FDG-PET
(dFDG-PET) data, selected by the frame selector algorithm, as the label for
our proposed ICA-net model. This model is specifically designed to perform the
segmentation task in a matter of seconds, thereby significantly enhancing the
efficiency and accuracy of the segmentation process in comparison to Semi-
automated methods.

####  2.2.3 ICA-net:

The ICA-net (Figure 2) is a customized model designed for the segmentation of
ICA in 3D dFDG-PET scans. The ICA-net model leverages a modified 3D U-Net
architecture [27], specifically adapted for handling the unique challenges
presented by dFDG-PET data. The preprocessing of ICA annotations, derived from
a semi-automated segmentation method, includes binarization of the volume
data. This is achieved through a custom ’binarizeVolume’ function, which
applies a threshold to convert the scan data into a binary format, thereby
simplifying the learning process for the ICA-net segmentation model.

ICA-net’s architecture is built on the backbone of VGG16, a well-established
model in the field of deep learning. The input to the network is a 3D volume
of shape 128x128x128, with a single channel representing grayscale images. The
network is trained using a combined loss function consisting of Dice loss and
binary cross-entropy (BCE) loss.

The combined loss function $L(y_{\text{true}},y_{\text{pred}})$ is given by:

$L(y_{\text{true}},y_{\text{pred}})=\text{DiceLoss}(y_{\text{true}},y_{\text{pred}})+\text{BCE}(y_{\text{true}},y_{\text{pred}})$
(1)

where Dice loss is defined as:

$\text{DiceLoss}(y_{\text{true}},y_{\text{pred}})=1-\text{DiceCoefficient}(y_{\text{true}},y_{\text{pred}})$
(2)

and the Dice coefficient is calculated as:

$\text{DiceCoefficient}(y_{\text{true}},y_{\text{pred}})=\frac{2\cdot\sum{(y_{\text{true}}\cdot
y_{\text{pred}})}+\text{smooth}}{\sum{y_{\text{true}}}+\sum{y_{\text{pred}}}+\text{smooth}}$
(3)

with smooth being a small constant added to prevent division by zero.

The Combined Dice and BCE Loss function integrates the advantages of both Dice
and BCE loss functions to address ICA segmentation challenges:

  1. 1.

Class Imbalance Compensation: Dice Loss is sensitive to the class imbalance
due to its formulation:

$\mathcal{L}_{\text{Dice}}=1-\frac{2\sum_{i}^{N}p_{i}g_{i}}{\sum_{i}^{N}p_{i}+\sum_{i}^{N}g_{i}}$

However, BCE compensates by providing a strong gradient for each pixel:

$\mathcal{L}_{\text{BCE}}=-\frac{1}{N}\sum_{i}^{N}\left[g_{i}\log(p_{i})+(1-g_{i})\log(1-p_{i})\right]$

  2. 2.

Gradient Behavior: BCE provides a consistent gradient flow for
backpropagation, which can help in scenarios where Dice Loss gradients may
become sparse, particularly when the overlap between $p_{i}$ and $g_{i}$ is
minimal.

  3. 3.

Comprehensive Error Signal: Combining Dice and BCE Loss functions offers a
more holistic error signal for optimization:

$\mathcal{L}_{\text{combined}}=\mathcal{L}_{\text{Dice}}+\mathcal{L}_{\text{BCE}}$

  4. 4.

Optimization Landscape: The combined loss function creates an optimization
landscape that leverages the benefits of both loss functions, potentially
avoiding the local minima that could be encountered when using Dice Loss
alone.

  5. 5.

Robustness: The addition of BCE Loss ensures that the model is penalizing
individual pixel misclassifications, which can make the segmentation process
more robust to noise and outliers.

This combined loss function can therefore facilitate improved model
convergence and generalization. Furthermore, the ICA-net employs several
advanced techniques to bolster its performance and robustness. Data
augmentation plays a pivotal role, wherein each 3D MRI brain image is
transformed through randomized rotations (up to ±4 degrees), shifts (up to 5
voxels), and zooms (±0.05 scale factor), subsequently adjusted to a uniform
size of 128x128x128 voxels through zero-intensity padding and cropping. This
process effectively expands the training dataset, enhancing the model’s
exposure to diverse clinical scenarios. Additionally, the Adam optimizer is
utilized for efficient network training, with its adaptive learning rate
capabilities. The model also implements early stopping and model checkpointing
strategies to prevent overfitting and to retain the best model weights during
training. Finally, the model’s parameters, totaling over 71 million,
underscore its complexity and capacity to capture detailed features essential
for accurate ICA segmentation.

Figure 2: Deep Learning Models

####  2.2.4 MCIF-net:

To generate the parametric brain PET maps, MCIF is computed by optimizing the
IDIF derived from the ICA as described [28], to account for partial volume
recovery of the blood input.

The MCIF-net model (Figure 2) is developed using a hybrid recurrent neural
network architecture, specifically designed to handle the time-series data
inherent in dynamic PET imaging. This architecture integrates both LSTM [29]
and Bi-directional GRU layers [30], leveraging the strengths of each to
enhance the processing of PET scan data. The LSTM layers, with increasing
complexity from 50 to 200 neurons, are adept at capturing long-term
dependencies, while the inclusion of Bi-directional GRU layers provides a
richer context by processing data in both forward and backward directions.
Dropout layers are interspersed throughout to prevent overfitting.
TimeDistributed Dense layers are incorporated to maintain the temporal
structure of the output, aligning with the sequential nature of the input.

Data preparation for this combined architecture involved reshaping it into a
format suitable for time-series analysis, specifically [samples, time steps,
features]. The data was split according 5 fold cross validation.

Compiled with the Adam optimizer and trained on a mean squared error loss
function, MCIF-net underwent fine-tuning over 1000 epochs with a batch size of
32. With a total of 5,127,801 trainable parameters, the model’s sophisticated
architecture is strategically designed to maximize accuracy in predicting
MCIF, a critical factor in PET scan analysis. This implementation of a
combined LSTM and Bi-directional GRU approach in MCIF-net exemplifies the
integration of advanced machine learning techniques in medical imaging,
highlighting its capability to provide nuanced and accurate analysis of
complex time series data.

###  2.3 Evaluation Metrics

ICA-net: The evaluation of the model’s performance is rigorous. We utilize the
5-Fold cross-validation technique, ensuring a thorough and unbiased assessment
of the model. Metrics such as the Jaccard score and Dice coefficient are
calculated for each fold, providing a comprehensive view of the model’s
segmentation capabilities. The overall metrics, including the mean and
standard deviation across all folds, provide a holistic view of the model’s
performance, demonstrating its efficacy in segmenting the carotid arteries in
dFDG-PET scans.

MCIF-net: The evaluation of the MCIF-net, an RNN-based regression model, is
centered around two key metrics: Mean Squared Error (MSE) and Mean Absolute
Error (MAE). These metrics are particularly suited for regression models as
they provide a clear indication of the model’s accuracy in predicting
continuous values.

##  3 Results

We trained and evaluated the ICA-net and MCIF-net models separately, employing
a standard 5-fold cross-validation approach. Furthermore, we validated our
complete pipeline against ground truth epilepsy data by calculating z-scores
and identifying the seizure onset zone.

Table 1: ICA-net Fold-wise Metrics with Mean and Standard Deviation Metric | Train Loss | Val Loss | Val Dice | Val IoU
---|---|---|---|---
Fold 1 | 0.0473 | 0.221 | 0.7865 | 0.6529
Fold 2 | 0.0531 | 0.1005 | 0.8602 | 0.7367
Fold 3 | 0.052 | 0.1186 | 0.8575 | 0.7232
Fold 4 | 0.0431 | 0.154 | 0.8466 | 0.7008
Fold 5 | 0.044 | 0.2401 | 0.7586 | 0.6136
Mean | 0.047901 | 0.16685 | 0.8218 | 0.685423
Std | 0.00404 | 0.05513 | 0.04147 | 0.04582

###  3.1 ICA-net

The ICA-net model, developed on the foundation of the UNet architecture,
underwent training across 120 epochs, leveraging a 5-fold cross-validation
with a leave-one-out approach. This rigorous training regimen yielded a mean
Dice coefficient of 82.18% and a Jaccard score of 68.5423 % as shown in Table
1. Figure 3A illustrates the segmented results, where it is distinctly
observable how the predicted ICA closely aligns with the original ICA,
demonstrating the model’s precision in segmentation. Notably, the analysis
reveals that the predicted IDIF points sometimes adds more clusters, slightly
distinct from the primary two ICA clusters as shown in Fig 3 B (arrow). Upon
exclusion of these outlier clusters, we anticipate an increase in accuracy
ranging from approximately 1.5% to 3.5%, further enhancing the model’s
performance.

Figure 3: Prediction results of ICA-net and MCIF-net on two test subjects Table 2: Performance of MCIF-net with Different Architectures Architecture | MAE | MSE
---|---|---
LSTM | 0.1277 | 0.0991
GRU | 0.1253 | 0.0478
Bi-directional GRU | 0.1183 | 0.0351
Comb Bi-GRU + LSTM | 0.0526 | 0.0052

###  3.2 MCIF-net

The MCIF-net, initially built on a customized LSTM architecture, was trained
for 1000 epochs again using 5 fold cross validation. This initial approach
resulted in a Mean Absolute Error (MAE) of 0.1277073 and a Mean Squared Error
(MSE) of 0.09911411. We then modified the MCIF-net architecture to incorporate
GRU, which achieved an MAE of 0.1253767 and an MSE of 0.04786217. Further
adaptations using a Bi-directional GRU resulted in an MAE of 0.1183242 and an
MSE of 0.03519516. Additionally, a combination of Bi-directional GRU and LSTM
yielded the best results with an MAE of 0.0526 and an MSE of 0.0052, as
detailed in Table 2 . Example predicted MCIF compared to model or original
MCIF are shown for 2 example data sets Figure 3 C-D.

The performance of both the networks (ICA-net and MCIF-net) in this work are
an improvement over our previous work due to improved networks and an increase
in data sets for model training [31].

Table 3: MCIF-net Model Performance Metrics Across Folds Metric | Train Loss | Val Loss | Val MSE | Val MAE | Val RMSE
---|---|---|---|---|---
Fold 1 | 0.0094 | 0.0063 | 0.0063 | 0.0640 | 0.0791
Fold 2 | 0.0042 | 0.0035 | 0.0035 | 0.0302 | 0.0594
Fold 3 | 0.0078 | 0.0039 | 0.0039 | 0.0448 | 0.0621
Fold 4 | 0.0129 | 0.0072 | 0.0072 | 0.0707 | 0.0846
Fold 5 | 0.0090 | 0.0050 | 0.0050 | 0.0532 | 0.0710
Average | 0.0087 | 0.0052 | 0.0052 | 0.0526 | 0.0712
Figure 4: Comprehensive End-to-End Predictive Analysis of Ground Truth Data
’E30’

###  3.3 Z-score results

In this study, we focused on a single epilepsy patient, referred to as ’e30’,
who had confirmed surgical ground truth. We employed our end-to-end pipeline
models, ICA-net and MCIF-net, to analyze this case. The predictive models were
utilized to calculate Z-scores for the Ki values, which were normalized
against the mean and standard deviation (SD) of the whole brain, encompassing
18 super regions on each side [12]. A Z-score cutoff of less than -2 SD was
established to identify hypometabolic regions. This approach successfully
highlighted significant hypometabolism in the patient’s left hippocampus, with
a noteworthy Z-score of -2.027041.

The accuracy of our models was further validated by comparing the Z-score
calculations against the ground truth derived from a 15 parameter compartment
model. The resultant average root mean square error (RMSE) was 0.068,
indicating a high degree of precision in the model’s predictions. This precise
localization as shown in Figure4 was critical, as traditional diagnostic
methods, including standardized uptake value (SUV) and static PET (sPET), had
previously failed to identify the affected area. Our findings were further
corroborated by intracranial EEG results, which localized the seizure onset
zone (SOZ) to the left hippocampus.

The clinical relevance of our study was underscored by the patient’s treatment
outcome. Patient ’e30’ underwent laser interstitial thermal therapy (LITT)
targeting the identified hypometabolic region based on invasive intracranial
monitoring in the right hippocampus, which agreed with our model predictions
(Figure 4). This intervention resulted in a significant clinical improvement,
with the patient achieving a seizure-free status for a period of 3 months by
the follow-up on September 9. This outcome not only validates the
effectiveness of our model but also highlights its potential in enhancing
clinical decision-making for epilepsy treatment.

##  4 Discussion

These results demonstrate the ability of a 3D-U-Net-based segmentation
approach to effectively identify and segment out the carotid arteries in dFDG-
PET of the human brain. Additionally, using network-predicted IDIFs to
calculate important downstream dFDG-PET metrics, like the Ki value [11], [15],
would serve as more clinically relevant measurements on which to base the
evaluation of this model with reference to these same downstream metrics
resulting from respective ground truth annotations.

Continuing, testing more datasets would also allow for better evaluation of
this model’s generalizability. A dataset of 50 scans is extremely limited and,
having already demonstrated successful model learning on so few data,
performance would likely improve substantially with the acquisition of a more
extensive and variable dataset. Additionally, it would be of interest to
assess model performance on scans that made use of different radiotracers. As
discussed earlier, radiotracer selection is specific to the biological pathway
relevant to the disease state being assessed and there are many important
diseases that are not typically studied using dFDG-PET. By both training and
testing on datasets that use radiotracers other than FDG, the utility of
automatic carotid segmentation and IDIF computation could be extended to the
study of a larger scope of neurological disorders, making dPET yet more
clinically robust and attractive.

Another way in which this segmentation model could be further optimized is
through additional exploration of other training parameters, network
architectures and loss functions. Due to the limitations of computational
power available for this project, it was not possible to more exhaustively
investigate the effect of modifying how the model was trained, like factor of
data augmentation (like using a generative adversarial network (GAN) [32]
instead of simply applying affine transformations) and batch size, which could
potentially lead to the discovery of a more ideal training protocol and
effectuate more robust segmentation. These computational limitations also
prevented the use of the full reference frame in training, which needed to be
cropped, likely leaving out important features from which the model could
learn the structure of carotids. Similarly, running a grid-search, for
example, on different combinations of architectural hyper-parameters, like
depth, base filter and dropout layer rate, could facilitate the discovery of a
more effective model.

In the realm of network architectures, while 3D U-Net enjoys popularity, the
integration of transformer-based architectures [33] (UNetR [34] ) could bring
substantial benefits to medical imaging segmentation, particularly in dPET.
Transformers, known for their proficiency in capturing long-range dependencies
and context, could significantly improve the delineation of complex structures
in medical images. Their ability to focus on relevant parts of an image makes
them particularly suitable for handling the intricacies and variability in
dPET images.

Similarly, the potential of sequence-to-sequence architectures [35] for our
MCIF-net should be considered. Given their success in processing and
predicting time-series data, these architectures could accurately model the
dynamic tracer distribution in PET scans, which is vital for precise MCIF
estimation. This approach could lead to more accurate corrections of SP and PV
effects, which are crucial in small vessel structures like carotids.

Additionally, a systematic exploration of architectural hyperparameters
through techniques like grid search could lead to the discovery of a more
effective model. Adjusting parameters such as network depth, base filter
count, and dropout rates could unveil optimal configurations for the unique
challenges of dPET image segmentation. Furthermore, considering alternative
architectures like feature-pyramid-based models [36], and statistical-
analysis-based techniques like local means analysis (LMA) and soft-decision
similar component analysis (SCA) [37], may provide novel avenues for carotid
artery segmentation. Finally, the exploration of different loss functions,
especially hybrid loss functions that have demonstrated success in relevant
segmentation tasks [38], [39], could reveal which metrics most closely
correlate with low IDIF error resulting from network predictions and guide the
selection of the most optimal loss function for this problem.

To successfully allow for fully automatic IDIF computation, the developed
frame-selection, as is explained under Frame-selection, must be automated.
Frame-selection is crucial as it provides the model with the most ideal frame
for selection, making accurate automatic segmentation more likely if
effective. Automated algorithms based on intensity analysis, which include
intensity and continuous-structure-length thresholding, as well as the use of
a trained 3D CNN model, could be explored to enhance the alignment and
accuracy of automatic frame-selection. These approaches aim to more closely
match the results of an automated algorithm with those of manually determined
ideal reference frames. Furthermore, the MCIF-net’s evolution through various
LSTM, GRU and Fusion architectures culminates in its superior performance in
error metrics, signifying its effectiveness in dPET imaging analysis.

##  5 Conclusion

This study’s effective training and evaluation of the ICA-net and MCIF-net
models represent a significant stride in medical imaging and epilepsy
treatment. Both models, validated through a 5-fold cross-validation approach,
have shown high accuracy. The ICA-net excelled in segmentation accuracy, and
the MCIF-net, through various LSTM and GRU architectures, demonstrated
improved performance in error metrics.

Crucially, in a case study of an epilepsy patient, the models accurately
identified hypometabolic regions, with the calculated Z-scores aligning with
the seizure onset zone. This precision, especially in comparison to
traditional diagnostic methods, highlights the models’ clinical relevance.

This success underscores the potential of our approach in enhancing clinical
decision-making and indicates a promising direction for future research in
medical diagnostics and patient-specific treatment strategies.

## Acknowledgments

This work was supported in part by grants from the University of Virginia
Brain Institute (to DS, TM, MQ and BK), Ivy innovation funds (MQ and BK),
Commonwealth commercialization funds (CCF) (to BK) from Virginia Innovation
Partnership Corporation and start-up funds from the Department of Radiology
and Medical Imaging at the University of Virginia (to BK).

## References

  * [1] M. E. Phelps, E. J. Hoffman, N. A. Mullani, and M. M. Ter-Pogossian.  Application of annihilation coincidence detection to transaxial reconstruction tomography.  J. Nucl. Med. Off. Publ. Soc. Nucl. Med., 16(3):210–224, Mar 1975.
  * [2] M. Muzi et al.  Quantitative assessment of dynamic pet imaging data in cancer imaging.  Magn. Reson. Imaging, 30(9):1203–1215, Nov 2012.
  * [3] C. Love, M. B. Tomas, G. G. Tronco, and C. J. Palestro.  Fdg pet of infection and inflammation.  RadioGraphics, 25(5):1357–1368, Sep 2005.
  * [4] X. Zhou, Y. Li, and Q. Wang.  Fdg pet/ct used in identifying adult-onset still’s disease in connective tissue diseases.  Clin. Rheumatol., 39(9):2735–2742, Sep 2020.
  * [5] M. Grkovski et al.  Monitoring early response to chemoradiotherapy with 18f-fmiso dynamic pet in head and neck cancer.  Eur. J. Nucl. Med. Mol. Imaging, 44(10):1682–1691, Sep 2017.
  * [6] A. Binneboese et al.  Correlation between fdg-pet uptake and survival in patients with primary brain tumors.  Am. J. Nucl. Med. Mol. Imaging, 11(3):196–206, 2021.
  * [7] T Hossain, Z Qureshi, N Jayakumar, TE Muttikkal, S Patel, D Schiff, M Quigg, M Zhang, and BK Kundu.  Multimodal deep learning to differentiate tumor progression from treatment effect in human glioblastoma.  In International Symposium on Biomedical Imaging (ISBI), April 2023.
  * [8] R. S. Schetlick et al.  Parametric fdg pet quantification, segmentation and classification of primary brain tumors in human gbm.  In 2021 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC), pages 1–5, Piscataway, NJ, USA, 2021. IEEE.
  * [9] R. K. J. Brown, N. I. Bohnen, K. K. Wong, S. Minoshima, and K. A. Frey.  Brain pet in suspected dementia: Patterns of altered fdg metabolism.  RadioGraphics, 34(3):684–701, May 2014.
  * [10] SM Landau, D Harvey, CM Madison, RA Koeppe, EM Reiman, NL Foster, MW Weiner, and WJ Jagust.  Associations between cognitive, functional, and fdg-pet measures of decline in ad and mci.  Neurobiology of Aging, 32(7):1207–1218, 2011.
  * [11] H. Tang et al.  Cpne1 is a target of mir-335-5p and plays an important role in the pathogenesis of non-small cell lung cancer.  J. Exp. Clin. Cancer Res., 37(1), Jul 2018.
  * [12] Vikram Seshadri, Katherine A. Zarroli, Robert S. Schetlick, James C. Massey, Jose M. Reyes, Thomas J. Eluvathingal Muttikal, James T. Patrie, Stuart S. Berr, Nathan B. Fountain, Bijoy K. Kundu, and Mark Quigg.  Dynamic fdg-pet in localization of focal epilepsy: A pilot study.  Epilepsy and Behavior, 122:108204, 2021.
  * [13] M Quigg and B Kundu.  Dynamic fdg-pet demonstration of functional brain abnormalities.  Annals of Clinical and Translational Neurology, 9(9):1487–1497, 2022.
  * [14] KR Byrnes, CM Wilson, F Brabazon, R von Leden, JS Jurgens, TR Oakes, and RG Selwyn.  Fdg-pet imaging in mild traumatic brain injury: a critical review.  Frontiers in Neuroenergetics, 5:13, 2014.
  * [15] E. Guedj et al.  18f-fdg brain pet hypometabolism in patients with long covid.  Eur. J. Nucl. Med. Mol. Imaging, 48(9):2823–2833, Aug 2021.
  * [16] A. Dimitrakopoulou-Strauss, L. Pan, and L. G. Strauss.  Quantitative approaches of dynamic fdg-pet and pet/ct studies (dpet/ct) for the evaluation of oncological patients.  Cancer Imaging, 12(1):283–289, 2012.
  * [17] K. A. Wangerin et al.  A virtual clinical trial comparing static versus dynamic pet imaging in measuring response to breast cancer therapy.  Phys. Med. Biol., 62(9):3639–3655, May 2017.
  * [18] J. J. Vaquero and P. Kinahan.  Positron emission tomography: Current challenges and opportunities for technological advances in clinical and preclinical imaging systems.  Annu. Rev. Biomed. Eng., 17(1):385–414, Dec 2015.
  * [19] B. G. Oertel et al.  Necessity and risks of arterial blood sampling in healthy volunteer studies.  Clin. Pharmacokinet., 51(10):629–638, Oct 2012.
  * [20] E. Croteau et al.  Image-derived input function in dynamic human pet/ct: methodology and validation with 11c-acetate and 18f-fluorothioheptadecanoic acid in muscle and 18f-fluorodeoxyglucose in brain.  Eur. J. Nucl. Med. Mol. Imaging, 37(8):1539–1550, Aug 2010.
  * [21] C. H. Lyoo et al.  Image-derived input function derived from a supervised clustering algorithm: Methodology and validation in a clinical protocol using [11c](r)-rolipram.  PLoS ONE, 9(2):e89101, Feb 2014.
  * [22] S.M. Smith, M. Jenkinson, M.W. Woolrich, et al.  Advances in functional and structural mr image analysis and implementation as fsl.  Neuroimage, 23:S208–S219, 2004.
  * [23] M.W. Woolrich, S. Jbabdi, B. Patenaude, et al.  Bayesian analysis of neuroimaging data in fsl.  Neuroimage, 45:S173–S186, 2009.
  * [24] M. Jenkinson, P. Bannister, M. Brady, and S. Smith.  Improved optimization for the robust and accurate linear registration and motion correction of brain images.  Neuroimage, 17:825–841, 2002.
  * [25] M. Jenkinson, C.F. Beckmann, T.E. Behrens, et al.  Fsl.  Neuroimage, 62:782–790, 2012.
  * [26] M. Jenkinson and S. Smith.  A global optimisation method for robust affine registration of brain images.  Medical Image Analysis, 5:143–156, 2001.
  * [27] Özgün Çiçek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: Learning dense volumetric segmentation from sparse annotation.  In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, pages 424–432. Springer, 2016.
  * [28] J. C. Massey, V. Seshadri, S. Paul, K. Mińczuk, C. Molinos, J. Li, and B. K. Kundu.  Model corrected blood input function to compute cerebral fdg uptake rates from dynamic total-body pet images of rats in vivo.  Front. Med. (Lausanne), 8:618645, Apr 2021.
  * [29] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [30] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  On the properties of neural machine translation: Encoder-decoder approaches.  In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, 2014.
  * [31] N Jayakumar, Z Qureshi, D Schiff, M Quigg, M Zhang, and BK Kundu.  A deep learning pipeline for parametric fdg brain pet mapping in localization of focal epilepsy.  Journal of Nuclear Medicine, 64(supplement 1):P808, June 2023.
  * [32] V. Sandfort, K. Yan, P. J. Pickhardt, and R. M. Summers.  Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in ct segmentation tasks.  Sci. Rep., 9(1):16884, Dec 2019.
  * [33] J. M. J. Valanarasu, P. Oza, I. Hacihaliloglu, and V. M. Patel.  Medical transformer: Gated axial-attention for medical image segmentation, 2021.
  * [34] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu.  Unetr: Transformers for 3d medical image segmentation.  arXiv preprint arXiv:2103.10504, Oct 2021.
  * [35] Syed Ahmed, Ida E Nielsen, Anurag Tripathi, Hifza Siddiqui, Ghulam Rasool, and Ravi Prakash Ramachandran.  Transformers in time-series analysis: A tutorial.  arXiv preprint arXiv:2205.01138, Apr 2022.
  * [36] H. Li et al.  3d ifpn: Improved feature pyramid network for automatic segmentation of gastric tumor.  Front. Oncol., 11:618496, May 2021.
  * [37] P. Zanotti-Fregonara et al.  Comparison of 3 methods of automated internal carotid segmentation in human brain pet studies: Application to the estimation of arterial input function.  J. Nucl. Med., 50(3):461–467, Mar 2009.
  * [38] D. Huang et al.  Learning rich features with hybrid loss for brain tumor segmentation.  BMC Med. Inform. Decis. Mak., 21(S2):63, Jul 2021.
  * [39] Y. Yang et al.  Mh-net: Model-data-driven hybrid-fusion network for medical image segmentation.  Knowl.-Based Syst., page 108795, Apr 2022.