# VINNA for Neonates - Orientation Independence through Latent Augmentations

Leonie Henschel  David Kügler  Lilla Zöllei  Martin Reuter  German Center for
Neurodegenerative Diseases (DZNE), Bonn, Germany  A.A. Martinos Center for
Biomedical Imaging, Massachusetts General Hospital, Boston MA, USA  Department
of Radiology, Harvard Medical School, Boston MA,USA

###### Abstract

A robust, fast, and accurate segmentation of neonatal brain images is highly
desired to better understand and detect changes during development and
disease, specifically considering the rise in imaging studies for this cohort.
Yet, the limited availability of ground truth datasets, lack of standardized
acquisition protocols, and wide variations of head positioning in the scanner
pose challenges for method development. A few automated image analysis
pipelines exist for newborn brain MRI segmentation, but they often rely on
time-consuming non-linear spatial registration procedures and require
resampling to a common resolution, subject to loss of information due to
interpolation and down-sampling. Without registration and image resampling,
variations with respect to head positions and voxel resolutions have to be
addressed differently. In deep-learning, external augmentations such as
rotation, translation, and scaling are traditionally used to artificially
expand the representation of spatial variability, which subsequently increases
both the training dataset size and robustness. However, these transformations
in the image space still require resampling, reducing accuracy specifically in
the context of label interpolation. We recently introduced the concept of
resolution-independence with the Voxel-size Independent Neural Network
framework, VINN. Here, we extend this concept by additionally shifting all
rigid-transforms into the network architecture with a four degree of freedom
(4-DOF) transform module, enabling resolution-aware internal augmentations
(VINNA) for deep learning. In this work we show that VINNA (i) significantly
outperforms state-of-the-art external augmentation approaches, (ii)
effectively addresses the head variations present specifically in newborn
datasets, and (iii) retains high segmentation accuracy across a range of
resolutions (0.5-$1.0\text{\,}\mathrm{mm}$). Furthermore, the 4-DOF transform
module together with internal augmentations is a powerful, general approach to
implement spatial augmentation without requiring image or label interpolation.
The specific network application to newborns will be made publicly available
as VINNA4neonates.

###### keywords:

Computational Neuroimaging , Deep Learning , Structural MRI , Artificial
Intelligence , High-Resolution , Newborn Brain

††journal: Imaging Neuroscience

CSF short = CSF, long = cerebrospinal fluid dHCP short = dHCP, long =
Developing Human Connectome Project MRI short = MRI, long = Magnetic Resonance
Image MR short = MR, long = Magnetic Resonance CNN short = CNN, long =
Convolutional Neural Network VINN short = VINN, long = Voxel-size Independent
Neural Network VINNA short = VINNA, long = VINN with ”internal augmentations”
infantFS short = infantFS, long = infantFreeSurfer iBEAT short = iBEAT, long =
Infant Brain Extraction and Analysis Toolbox FastInfantSurfer short = FIS,
long = FastInfantSurfer FastInfantSurfer WM short = WM, long = white matter GM
short = GM, long = gray matter PVE short = PVE, long = partial volume effect
abide-ii short = ABIDE-II, long = Autism Brain Imaging Data Exchange II NN
short = NN, long = nearest-neighbour ASD short = ASD, long = Average Surface
Distance DSC short = DSC, long = Dice Similarity Coefficient TODO short =
TODO, long = compile fix still todo HiRes short = HiRes, long = high-
resolution LowRes short = LowRes, long = low-resolution MSDA short = MSDA,
long = multi-source domain adaptation

##  1 Introduction

Collections of neonatal brain MRI are indispensable to understand brain
development and to detect early signs of potential developmental disorders.
One of the key tasks in MRI analysis is automated segmentation, the labeling
of anatomical regions of interest (ROIs) that can be used for quantitative
modeling of healthy development, for analyses in population studies, for
understanding disease effects as well as a starting point for further
neuroimaging tasks. The segmentation of infant MRI is a challenging and non-
trivial undertaking due to the rapid non-linear changes during the postnatal
brain growth period, elevated levels of head motion, limited availability of
congruent datasets, varying intensity profiles across scanners, protocols and
modalities, as well as the inversion of gray-white contrast around the age of
5-9 month [1, 2, 3, 4]. In this paper, we focus on a sub-group of the infant
population – newborns – and present a four-degree of freedom (4-DOF) transform
module to address two core challenges within this cohort: non-uniform image
resolutions (scaling) and increased variability of head positions (rigid
transformations = rotation and translation) during image acquisition. The
4-DOF transform module is directly integrated into the network-architecture
and addresses the variability of head positions internally. As such, it
expands and generalizes the distinguishing feature, resolution independence,
of the recently published VINN [5] by rotation and translation
transformations. We refer to our new framework as VINNA.

Figure 1: Spatial augmentations in deep learning networks: A. One single
resolution-ignorant CNN or resolution-independent voxel size independent
network (VINN) can learn to segment multiple resolutions and head positions by
training on a diverse dataset. External scale, rotation, and translation
augmentation (+ external augmentation, A. bottom) diversifies the existing
training samples by resampling the images and the reference segmentation maps.
Here, however, lossy interpolation and resulting artefacts, especially from
nearest-neighbour interpolation of discrete label maps, may result in a loss
of structural details and sub-optimal performance. B. Our 4-DOF transform
module (VINNA) completely avoids interpolation of the images and discrete
labels by integrating the interpolation step into the network architecture
itself (B. bottom). Rotations, translations, and scalings applied in the first
interpolation block are later reversed, assuring segmentations to be in the
original image orientation. Furthermore, the explicit transition from the
native resolution to a normalized internal resolution facilitates an
understanding of the difference between image features (MultiRes blocks with
distances measured in voxels) and anatomical features (FixedRes inner blocks
with normalized distances).

In contrast to adults, newborn head positions in the scanner are far more
diverse due to the scanning conditions (asleep) and overall smaller anatomy.
While padding is often used to stabilize the child’s head and to occupy the
space between head coil and participant (e.g. foam cushions, pads, mats,
pillows, or visco-elastic matters) [6], its standardization is difficult. This
results in diverse head orientations within the scanner and potentially high
variations among imaging studies.

In addition, there is no de-facto standard resolution for newborn imaging. In
the case of research protocols, when more time is available, MRI are often
acquired at higher-resolutions to address the small size of brain structures
and stronger partial volume effects [7, 8, 9]. However, the range of recorded
resolutions across research and clinical studies is relatively large and
heterogeneous, ranging from $0.5\text{\,}\mathrm{mm}$ to
$3.0\text{\,}\mathrm{mm}$ in past and recent research studies (e.g. NIH-PD
[10], BCP [11], dHCP [12, 13], HBCD [14]). Similarly, resolutions are not
standardized across atlases (UNC 0-1-2 Infant atlases [15], Imperial Brain
Development atlases [16, 17, 18, 19]), that are often used to guide the
automated labeling algorithms.

Traditional tools for newborn or infant segmentation predominantly interpolate
images to a single chosen resolution and harmonize the spatial head position
via atlas registrations [20, 21, 22, 23, 12, 24]. Resampling of images can,
however, result in loss of information, especially in the context of high-
resolution label maps. Furthermore, atlas-guided approaches are usually highly
dependent on registration accuracy. For newborns, registration is particularly
challenging due to lower tissue contrast, specifically in the T1w scans.
Errors in the process are hence common and improvement of the registration,
e.g. with spatio-temporal information, anatomical constraints, and surface
models, is an active field of research [25, 24, 16, 8, 26, 27, 28, 29].

The explicit definition of spatial and intensity features can be avoided by
using CNN. In fact, fast deep-learning methods for semantic segmentation, are
becoming increasingly popular for infant segmentation [30, 31, 32, 33, 34, 35,
36, 37, 38, 39, 40]. Applicability of deep learning approaches, however, is
generally restricted to domains where sufficiently large training datasets
exist. While there have been several initiatives to collect larger
neuroimaging cohorts of newborns and infants in recent years [12, 11, 14, 13],
their size is still relatively small compared to equivalent cohorts in the
adult population. Additionally, accompanying manual labels are sparse, due to
high annotation costs (time and money) and non-uniform labeling protocols,
limiting the pool for supervised training options further. Considering the
newborn cohort, the variability in resolution and head positioning is likely
underrepresented in the publicly available datasets, questioning whether a
network trained on the available pairs of scans and labels can be robust
enough without additional augmentation.

The most widely used solution to artificially increase the training set size,
robustness, and generalizability of deep-learning methods has been traditional
data augmentation, such as rotation, scaling, or translation (Figure 1A). In
this case, both images and their labelmaps are interpolated to a new random
position during training. Interpolation, however, in the native image space
requires resampling of the discrete ground truth segmentations, resulting in
information loss (e.g. from lossy NN interpolation) and reduction in accuracy
[5].

With the VINN architecture [5], we recently established the first network for
resolution-independent deep learning, which effectively circumvents scaling
augmentation and subsequent external resampling, while leveraging information
across datasets of varying resolutions. In VINN, the classic fixed-factor
integer down- and up-scale transitions, often implemented via pooling
operations in UNet-like architectures [41], are replaced with a flexible re-
scaling for the first and last scale transitions. This network-integrated
resolution-normalization allows for segmentation in the native space during
both, training and inference. In adults, this approach has been shown to
outperform fixed-resolution CNN as well as resolution-ignorant CNN trained
with external scaling augmentation, and to improve performance both for sub-
millimeter and one-millimeter scans.

Since newborn datasets are often acquired at various native resolutions and
are particularly subject to partial volume effects, the resolution-
normalization feature offers a basis to improve segmentation performance here
as well. As is, VINN only addresses scaling and would still require external
augmentations, and hence label interpolation, to address the increased
variability of head positions and limited availability of training data for
newborns. With our VINNA and its 4-DOF transform module, we now close this gap
and propose a paradigm shift away from classical data augmentation towards a
detail-preserving internal augmentation scheme (Figure 1B). While avoiding any
type of label interpolation, we extend VINN´s network-integrated resolution-
normalization with spatial augmentations (i.e., rotation and translation). At
the first layer scale transition, the feature maps are hence not only flexibly
rescaled, but also randomly transformed to imitate the position variations
commonly found in newborns, subsequently increasing the training distribution.

In conclusion, VINNA and its 4-DOF transform module effectively address the
challenges associated with newborn segmentation, namely variation in head
positions and resolutions in the context of limited data availability. The
four key contributions of VINNA presented in this work are as follows:

  1. (i)

We provide the first publicly available open-source deep-learning pipeline for
a combined subcortical segmentation as well as cortical, and white matter
parcellation for newborn T1w or T2w MRI.

  2. (ii)

We introduce a novel augmentation paradigm, which for the first time moves
spatial augmentations into the network (instead of being performed outside).
Our experimental results compare various spatial augmentation approaches side-
by-side to isolate their effects.

  3. (iii)

We ensure fair comparisons throughout, for example, by fixed dataset splits,
retraining methods under equal data and parameter settings, comparing
architectures and setups with minimal differences, and quantifying real-world
performance

  4. (iv)

We, further, provide extensive comparison with state-of-the-art deep-learning
methods (2D and 3D nnUNet) adapted for newborn segmentation (retrained on the
same data) and present an extensive comparison to the publicly available
newborn segmentation pipelines iBEAT and infantFS.

The specific application of VINNA to newborns will be made available as
VINNA4neonates within our open source repository111github.com/Deep-
MI/NeonateVINNA upon publication including Docker containers offering easy
accessibility for the community.

###  1.1 Related work

While various reliable and sensitive traditional [42, 43, 44, 45] and fast
deep-learning solutions exist [46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57,
5] for adult whole brain segmentation, application of these methods to younger
ages is hampered by the significant differences in size, MRI contrast profiles
and rapidly changing postnatal anatomy that is challenging to model with
static templates.

###  1.2 Traditional tools for infant segmentation

Infant-specific traditional atlas-guided tools [24, 21, 22, 23, 12, 58] are
predominantly optimized for a specific age range, resolution, and modality.
Further, they differ significantly in the number of segmented classes and
structure definitions.

The more recent iBEAT V2.0 [30] is a combination of age-specific CNN for
tissue segmentation, traditional surface generation and parcellation, based on
atlas registration, into 34 regions following the Desikan-Killiany protocol
[59]. It supports a large age range (0-6 years), and allows segmentation of
both, T1w and T2w MRI. While multiple input resolutions are supported, iBEAT
internally reorients and resamples each image to a standard format (RAS
orientation and $0.8\text{\,}\mathrm{mm}$ isotropic resolution). Hence, it
does not support native resolution segmentation and image interpolation is
required to map segmentations back to the original input space. The resampling
step is automatically included in the pipeline such that in- and output
resolutions are flexible. Furthermore, in its publicly available docker
pipeline222https://github.com/iBEAT-V2/iBEAT-V2.0-Docker, segmentation is
limited to WM, GM and CSF.

infantFS [23], on the other hand, mimics the FreeSurfer [60] processing
pipeline for adults and processes images from the first two years postnatally.
It supports anatomical segmentation into 32 classes based on multi-atlas label
fusion strategy including registration to the infantFreeSurfer training data
set [61]. The entire pipeline is publicly
available333https://surfer.nmr.mgh.harvard.edu/fswiki/infantFS and allows
processing of T1w images at a resolution of $1.0\text{\,}\mathrm{mm}$, where
the atlas training data is defined. For newborns, T1w images often suffer from
poor tissue contrast due to the underlying myelination process, aggravating
accurate registration from the atlases onto individual brains. This age group
can therefore be a challenge for infantFS´s mono-modality approach.

The dHCP minimal-processing-pipeline [12] is an optimized framework for
cortical and sub-cortical volume segmentation, cortical surface extraction,
and cortical surface inflation, which has been specifically designed for high-
resolution T2w MRI of newborns [62]. Here, an Expectation-Maximization
algorithm, including an atlas-based spatial prior term, labels 87 classes
based on a modified version of the ALBERTs atlas [18, 25]. The segmentations
include subcortical structures, cortical and WM parcellations. Due to the
cubic increase in voxel-size for high-resolution images, processing times are
in the order of hours to days for a single subject. This is a common
limitation among traditional methods.

###  1.3 Deep-Learning for infant segmentation

####  1.3.1 Newborns

Overall, networks for cortical parcellations and subcortical structure
segmentations in newborns are limited. The few existing CNN support a single
modality (T2w), fixed resolution, and segment a single [63] or eight [39]
tissue classes. One recent exception is the deep-learning based neuroimaging
pipeline by Shen et al. [64], which is trained with the dHCP data. Here, the
authors propose a 3D multi-task deep learning model with a U-Net like
architecture to segment structural T1w and T2w images on both thin and thick
sliced images. Unfortunately, the network follows a fixed-resolution scheme
($0.8\text{\,}\mathrm{mm}$), it does not support native segmentation across
resolutions commonly encountered in newborn cohorts, and it is not readily
available online.

####  1.3.2 Isointense phase

The vast majority of deep-learning tools focus on processing of images at the
isointense phase around 6 months after birth [31, 32, 33, 34, 35, 36, 37, 65,
66]. Via the iSeg-challenge [67, 68], data for training and validation is
conveniently available partly explaining this predominance. While many
interesting architectural solutions have arisen, the main focus of the works
is the effective combination of information from both T1w and T2w images to
generate a broad segmentation into CSF, GM and WM. This modality combination
is specifically important in the isointense phase, which is characterized by
changes in the myelination strongly effecting the appearance of the recorded
MRI [69, 70, 71]. The inversion of the WM-GM signal results in extremely low
tissue contrast. The newborn cohort, on the other hand, demonstrates good
contrast between GM and WM, specifically on the T2w images. While the age
difference is small, the two cohorts as well as the associated challenges are
distinct and networks trained on the one can not easily be applied to the
other. Subsequently, neither resolution-independence nor the stronger
variation of head positions are specifically accounted for in network
solutions for the isointense phase.

####  1.3.3 Cross-age generalizability

To address generalizability across different age groups, recent research has
suggested optimized training strategies for neonates, such as multi-task
learning of tissue segmentation and geodesic distances [38] or the use of
inductive biases in the form of pre-trained weights (i.e. fine-tuning to the
target domain) [40]. Both approaches improve segmentation accuracy, but they
are still limited in their generalizability. They require retraining and hence
a sufficient amount of labeled data; additionally they rely on private
datasets, limiting their reproducibility. Recently, a contrast agnostic
segmentation via synthetic images, originally proposed for adult brain
segmentation [55, 56], has also been adopted for infant segmentation [72].
Unfortunately, the output resolution is fixed for the network, and native
resolution segmentations are not supported. Furthermore, while the model was
able to generalize across a broader age range, the synthetic images still
differ considerably from real data and the network therefore underperformed
compared to age-specific models trained on existing MRI.

###  1.4 Resolution-independence and position transforms in deep-learning

A general resolution-ignorant framework addressing position transforms via
external augmentations is nnUNet [73]. This network has successfully been
applied for a variety of segmentation tasks due to its inherent ability to
construct optimal parameter settings based on the input data itself. It
provides different network set-ups (2D, 3D and a cascaded 3D approach for
large images) as well as a number of external image augmentations including
random rotation, scaling, mirroring, and gamma transformation. Interestingly,
while the trained network also follows a fixed-resolution scheme, pre- and
post-processing automatically resamples between original image and network
resolution. While native resolution segmentation is not supported, in- and
output resolutions are not fixed and the method is therefore a valid
alternative to our VINNA. Both, the 2D and 3D nnUNet + exA therefore serve as
a state-of-the-art baseline for the newborn segmentation task.

A siamese network for semi-supervised training of a network to become
equivariant to elastic transformation has been proposed in a single-resolution
setting [74]. A dedicated loss function assures that segmentations are
consistent under a given class of transformation applied first to the image,
and second to the output. The approach therefore relies on external
augmentation and applies the transformation in the image space (before and
after a UNet). The proposed VINNA, on the other hand, is fundamentally
different. It shifts this step into the network itself, hence creating an
internal augmentation. Overall, the approach by Bortsova et al [74] does
therefore assure consistency across transformations in the labeling space,
while VINNA targets spatial consistency of the feature maps.

In spatial transformers [75], transformations attempt to harmonize or re-
orient the image into a better position. To this end, an affine transformation
is implicitly learned via a dedicated localisation network. Subsequent
application of the calculated coordinate grid resamples the source feature
maps via bi-linear interpolation to the new position. While our approach
shares grid calculation and interpolation within the network with spatial
transformers, our internal augmentation approach is inherently different.
First, spatial transformers do not diversify or augment feature maps, but
rather try to reach a harmonized position with respect to the data seen during
training. External augmentations are still necessary to expose the network to
a wide data variety and approximate equivariance. Second, instead of a
localisation network, we directly determine the sampling-grid based on a
transformation matrix, which allows for an explicit integration of knowledge
about the image, such as the resolution. As a result, computational complexity
is reduced while achieving the desired position diversification and
resolution-independence.

##  2 Material and methods

Figure 2: Comparison of variation in head position in newborns (orange) and
adults (blue). Newborns show greater variation with respect to rotation
angles. Rotation transformation is based on alignment of each individual
subject to their midspace. N=90 for both cohorts Figure 3: Harmonization of
inconsistent label protocols between iBEAT, infantFS, and dHCP. Reduction of
the original dHCP segmentation from 88 labels (left) by first merging all
cortical parcels to cortex, WM parcels to WM and removal of subcortical
structures, is followed by addition of CSF and ventricles (top) or amygdala
and hippocampus (bottom) for comparison to iBEAT (3 labels: CSF (blue), WM
(white) and GM (red), top second from left top) or infantFS (4 labels: WM
(white), cortex (red), hippocampus (yellow) and amygdala (blue), bottom second
from left), respectively. For iBEAT (right top) all GM and WM are modified
using the dHCP segmentation for the subcortical structures resulting in three
labels for the final mapped version (3 labels, top second from right). For
infantFS (right bottom), all structures except WM, cortex, hippocampus, and
amygdala are removed (4 labels, bottom second from right).

###  2.1 Datasets

As the largest publicly available newborn dataset with intensity images,
accompanying subcortical segmentations as well as cortical and WM
parcellations at the time of our experiments, we randomly assign participants
from the dHCP cohort with corresponding T1w and T2w MRI to the training,
testing, and validation sets, while ensuring equal distribution of age and
gender. Additionally, the Melbourne Children’s Regional Infant Brain (M-CRIB)
atlas cohort serves as an independent testing set for external validation of
the final method. Written informed consent to participate in this study was
provided by the participants’ legal guardian or next of kin in accordance with
the Institutional Review Board. Complete ethic statements are available at the
respective study webpages.

dHCP: The developing Human Connectome Project [12] includes T1w and T2w MRI of
newborns imaged without sedation on a 3 T Philips Achieva scanner. It provides
$0.5\text{\,}\mathrm{mm}$ isotropic de-faced scans of individuals imaged
postnatally between 24 to 45 weeks post-conceptional age. Imaging data for 578
participants with matching T2w and T1w were selected. The original images were
acquired in sagittal and axial slice stacks with in-plane resolution of
$0.8\text{\,}\mathrm{mm}$ $\times$ $0.8\text{\,}\mathrm{mm}$ and
$1.6\text{\,}\mathrm{mm}$ slices overlapped by $0.8\text{\,}\mathrm{mm}$.
Motion correction and super-resolution reconstruction techniques [76, 77]
created isotropic volumes of resolution $0.5\text{\,}\mathrm{mm}$. All T1w
scans follow the same inversion recovery multi-slice fast spin-echo protocol
with TR $4.795\text{\,}\mathrm{s}$, TE $8.7\text{\,}\mathrm{ms}$, TI
$1.740\text{\,}\mathrm{s}$, and SENSE factor 2.27 (axial) / 2.56 (sagittal).
The parameters for the T2w scans are TR $12\text{\,}\mathrm{s}$, TE
$156\text{\,}\mathrm{ms}$ and SENSE factor 2.11 (axial) / 2.66 (sagittal). The
full dataset is available online444https://data.developingconnectome.org/. In
the present study, 318 quality checked images are used for network training
and 90 for validation. A total of 170 images are used in the final test set.

Even though the dHCP follows a well-defined protocol, standardization of
positioning during scanning is still a challenge. As shown in Figure 2, inter-
subject head position diversity in the newborn cohort is larger than an
equally standardized adult cohort (HCP).

M-CRIB: The Melbourne Children’s Regional Infant Brain (M-CRIB) atlas [78] is
constructed from 10 T2w MRI and corresponding manual segmentations of healthy
term-born neonates (four females, six males) with gestational age-at-scan
between 40.29–43.00 weeks. The atlas comprises 94 neonatal brain regions
compatible with the widely-used Desikan-Killiany-Tourville adult cortical
atlas [79]. The T2w MRI scanning protocols include the usage of a transverse
T2 restore turbo spin echo sequence with $1.0\text{\,}\mathrm{mm}$ axial
slices, a TR of $8.910\text{\,}\mathrm{s}$, TE of $152\text{\,}\mathrm{ms}$,
flip angle of 120 degrees, Field Of View of $192\text{\,}\mathrm{mm}$ ×
$192\text{\,}\mathrm{mm}$, and in-plane resolution of $1\text{\,}\mathrm{mm}$
(zero-filled interpolated to $0.5\text{\,}\mathrm{mm}$ ×
$0.5\text{\,}\mathrm{mm}$ × $1\text{\,}\mathrm{mm}$). The T2w images are bias-
corrected, skull-stripped and resampled to $0.63\text{\,}\mathrm{mm}$ ×
$0.63\text{\,}\mathrm{mm}$ × $0.63\text{\,}\mathrm{mm}$ isotropic voxels. All
10 participants are used as an independent testing set for our external
validation experiments.

###  2.2 Generation of reference segmentation with the dhcp-minimal-
processing-pipeline

To imitate various resolutions and create the desired reference segmentations
for training, we processed all raw dHCP MRI with the dhcp-minimal-processing-
pipeline [12] at $1.0\text{\,}\mathrm{mm}$, $0.8\text{\,}\mathrm{mm}$ and
$0.5\text{\,}\mathrm{mm}$. The structure definitions follow the ALBERTs atlas
[18] with the subdivision of the WM and cortex proposed by Makropolus et al.
[25], resulting in a total of 87 structures (3 background labels, 20
subcortical regions, 32 cortical parcels, 32 WM parcels). We further
lateralized the intracranial background based on the average Euclidean
distance to neighbouring labels resulting in a final count of 88 labels. We
provide a list of all segmentation labels used for training in the Appendix
(see Section A.3). As the dHCP cohort includes both, T2w and a co-registered
T1w MRI, we trained dedicated networks for each modality. Note, that the dhcp-
minimal-processing-pipeline relies on the original T2w images to create its
segmentations, which are generally of higher quality in this collection.

###  2.3 Traditional infant segmentation tools

To evaluate VINNA against state-of-the-art traditional segmentation methods,
we further process the testing set with the docker version of the iBEAT V2.0
pipeline [30] and infantFS [23].

####  2.3.1 iBEAT

The iBEAT V2.0 pipeline [30] combines both traditional and deep-learning
models to create tissue segmentations into three classes (GM, WM, and CSF),
surface models, and cortical parcellations of the pial surface into 34 regions
based on the Desikan-Killiany protocol [59]. For tissue segmentation, iBEAT
uses seven age-specific CNN trained on data for the representative age group
($\leq 1$ month, 3 months, 6 months, 9 months, 12 months, 18 months, and 24+
months of age). Neither the source code nor the training data and labels are
publicly available. Hence, retraining of the models is not possible and
comparisons are limited to the iBEAT pipeline output as is. For processing
with iBEAT, submissions via a webserver555https://ibeat.wildapricot.org/ or
processing with a docker image666https://github.com/iBEAT-V2/iBEAT-V2.0-Docker
are possible. The docker version does not currently support the cortical
parcellations of the surface models. Due to the large number of participants,
privacy concerns, and longer processing times when submitting via the
webserver, we decided to use the docker version to process the T2w images of
the testing set at the original $0.5\text{\,}\mathrm{mm}$ resolution. The
resulting 3-label tissue segmentations form the basis for comparison to the
other tools in this paper.

####  2.3.2 infantFS

To allow comparison of segmentation performance to VINNA, all available T1w
images from the dHCP testing set are processed with infantFS with default
settings. The neuroimaging pipeline infantFS creates surface models,
anatomical segmentations and cortical parcellations based on the Desikan-
Killiany-Tourville atlas [79] for 0-2 year old infants akin to the version for
adults (FreeSurfer [60]). The tool runs on T1w MRI at a resolution of
$1.0\text{\,}\mathrm{mm}$ (non-conforming images are resampled). infantFS
relies on a registration-based multi-atlas label fusion strategy and returns
an anatomical segmentation into 32 classes, including two labels for GM and
WM.

####  2.3.3 Label harmonization

As the dHCP-ALBERTs atlas differs from the resulting segmentations of both
iBEAT and infantFS, we merge, remove, and mask classes to reach an approximate
consensus across predictions. Figure 3 shows the merging protocol on a
representative participant with the original and mapped ground truth dHCP
labels (left side) together with iBEAT (top right side) and infantFS (bottom
right side). First, the 32 cortical and WM parcels from the dHCP ground truth
segmentation are reduced to two labels (cortex and WM) (top left in Figure 3).
For iBEAT, the WM additionally includes the corpus callosum while GM also
encompasses the hippocampus and amygdala. The CSF label corresponds to the
union of lateral-ventricles and CSF in the dHCP-ALBERTs atlas. In the dHCP
ground truth, these labels are consequently merged to create the final three
classes (GM, WM, and CSF; top second to left image). All other subcortical
structures without a single possible assignment to GM, WM, or CSF are masked
in the iBEAT prediction using the dHCP ground truth (Figure 3, top right two
images). For infantFS, the hippocampus and amygdala label remain, while
individual cortex and WM parcels of the dHCP ground truth are merged (Figure
3, bottom left images). Hence, in the infantFS predictions the following four
labels remain: cortex, WM, hippocampus and amygdala (Figure 3, bottom two
images to the right).

###  2.4 Network architectures

Figure 4: Network-integrated position variation and scaling normalization in
VINNA. Flexible transitions between resolutions and head positions become
possible by replacing (un)pooling with our network-integrated 4-DOF transform
module (purple) after the first input dense block in the encoder (pre-IDB) and
before the last competitive dense block in the decoder (post-CDB). A
transformation matrix composed of rotation angle $\theta$, translation
parameter T and the scaling factor SF defines the feature alterations. Scale
transitions between the other competitive dense blocks (CDB) remain standard
MaxPool and UnPool operations. Each CDB is composed of four sequences of
parametric rectified linear unit (PReLU), convolution (Conv), and batch
normalization (BN). In the first two encoder blocks ((pre)-IDB), the PReLU is
replaced with a BN to normalize the inputs.

####  2.4.1 Macro architecture

Figure 4 shows the macro architecture for VINNA. While the proposed 4-DOF
transform module (purple) can, in theory, be included in any UNet-like
architecture, we use the same basic setup for all trained models to assure
maximum comparability (i.e. same number of parameters, same kernel sizes,
etc.). CNN*, VINN, and VINNA, all contain an encoder and decoder consisting of
five competitive dense blocks, respectively, which are separated by a
bottleneck layer. In the encoder, max pooling operations rescale the feature
maps at each level by one half between the blocks. In contrast, index-
unpooling doubles the feature map size in the decoder. Skip connections
between the blocks at each level allow the gradient to flow efficiently. In
CNN* [5], pooling and unpooling operations transition between all levels (i.e.
the purple block in Figure 4 is substituted with the gray maxpooling/unpooling
operation). In VINN [5], the first layer pooling and unpooling operation is
replaced with a resolution-normalization module. This network-integrated
flexible interpolation step allows transitions between resolutions without
restrictions to pre-defined fixed voxel sizes, both during training and
inference. Hence, images can be processed at their native resolution without
prior resampling. Similar to spatial transformers [75], the interpolation-
based transition is divided into two parts: (i) calculation of the sampling
coordinates (grid generator) and (ii) interpolation operation (sampler) to
retrieve the spatially transferred output feature maps. Here, the sampling
calculation relies only on the scale factor $SF$: the quotient of the
resolution information of the inner normalized scale
$\text{Res}_{\text{inner}}$, a tune-able hyperparameter set to
$0.8\text{\,}\mathrm{mm}$ throughout our experiments, and the input image
$\text{Res}_{\text{native}}$. The addition of parameter $\alpha$ sampled from
a Gaussian distribution with parameters sigma=0.1 and mean=0 slightly augments
the scale factor SF
($SF=\text{Res}_{\text{inner}}/\text{Res}_{\text{native}}+\alpha$), introduces
small resolution variations to the sampling, and increases the robustness of
the latent space interpolation. Specifically, the presence of alpha allows for
augmentations at the actual anatomical rather than the voxel size as the
normalization of the resolution inside VINN disentangles perceived voxel
versus actual structure size differences.

####  2.4.2 Network-integrated 4-DOF transform module

In VINNA, the transition is implemented via the new network-integrated 4-DOF
transform module shown in purple in Figure 4. Here, the sampling coordinate
calculation is based on a transformation matrix $M\in\mathbf{R}^{3\times 3}$
with four degrees of freedom, encoding not only scaling, but also in-plane
rotation, and translation. Parameters for the rotation angle
$\theta\in\mathbb{R}$ and translation $T\in\mathbb{R}^{2}$ are randomly
sampled on the fly during training. The scale factor $SF$ is calculated as in
VINNs resolution-normalization module, i.e. by dividing the inner normalized
scale by the image resolution with augmentation by the parameter $\alpha$. In
the first transition step (Figure 4, pre-IDB to IDB), the affine per-channel
mapping ${M\colon{\mathcal{U}}\to{\mathcal{V}}}$ samples the input feature
maps $U\in\mathbb{R}^{H_{\text{native}}\times W_{\text{native}}}$ to the
output feature maps $V\in\mathbb{R}^{H_{\text{inner}}\times
W_{\text{inner}}}$. In the final transition step (Figure 4, competitive dense
block (CDB) to post-CDB), this spatial transformation is reversed by using the
inverse transformation matrix $M^{-1}\colon{\mathcal{V}\to{\mathcal{U}}}$. The
interpolation itself is performed by applying a sampling kernel to the input
map $U$ to retrieve the value at a particular pixel in the output map $V$. The
sampling is identical for each channel, hence, conserving spatial consistency.

####  2.4.3 Network blocks

Competitive Dense Block (CDB) design In VINNA, a CDB is formed by repetitions
of the basic composite function consisting of a probabilistic rectified linear
unit (pReLU) activation function, a 3$\times$3 convolution, and a batch-
normalization (BN). Feature competition within the block is achieved by using
maxout [80] instead of concatenations [81] in the local skip connections. The
maxout operation requires normalized inputs and is therefore always performed
after the BN (see position of maxout in CDB design in Figure 4).

Input Competitive Dense Block (IDB) design In contrast to the described CDB,
the first two network blocks follow a different order of operation. Here, the
raw inputs are normalized by first passing them through a BN-Conv-BN
combination before adhering to the original composite function scheme (Conv-
BN-pReLU) (see Figure 4, IDB).

Pre-IDB The first encoder block in VINNA called pre-IDB (see Figure 4)
transfers image intensity information from the native image to the latent
space and encodes voxel size and subject-space-dependent information before
the internal interpolation step. The composite function scheme is identical to
the IDB and the added prefix simply allows differentiation of the block
placements.

Post-CDB Akin to the pre-IDB, an additional CDB block in the decoder merges
the non-interpolated feature information returned from the pre-IDB skip
connection and the upsampled feature maps from the network-integrated 4-DOF
transform modules. A concatenation operation combines both feature maps,
before passing them to a standard CDB block (see Figure 4, (post-)CDB). After
the final 1$\times$1 convolution a softmax operation returns the desired class
probabilities.

###  2.5 Loss function

The network is trained with a weighted composite loss function of logistic
loss and Dice loss [82] combined with the high-resolution specific weighting
from VINN [5]. In short, erosion and dilation of the cortex labels creates a
binary mask of the outer cortex, small WM strands and deep sulci. Wrong
predictions in these areas result in a higher loss, hence guiding the network
to focus on areas particularly affected by PVE. The original publications’
ablation experiments evaluated the impact of the different function elements:
the logistic loss and Dice loss combination improves overall segmentation
performance [82], while the high-resolution weighting leads to higher
segmentation accuracy on the cortical parcels [5].

If we consider $p_{l,i}(x)$ as the estimated probability of pixel $i$ that
belongs to class $l$, $y$ as the corresponding ground truth probability, and
$\omega_{i}$ as the associated weight given to the pixel $i$ based the loss
function can be formulated as

$\mathcal{L}={\underbrace{-\sum_{l,i}\omega_{i}y_{l,i}\log
p_{l,i}(x)}_{\text{Logistic
loss}}-\underbrace{\sum_{l\vphantom{,}}\frac{2\sum_{i}p_{l,i}(x)y_{l,i}}{\sum_{i}p_{l,i}(x)+\sum_{i}y_{l,i}}}_{\text{Soft
Dice loss}}}$ (1)

with $\omega_{i}=\omega_{\text{median
freq.}}+\omega_{\text{gradient}}+\omega_{\text{GM}}+\omega_{\text{WM/Sulci}}$.

Here, $\omega_{\text{median freq.}}$ represents median frequency balancing
addressing the class imbalance and $\omega_{\text{gradient}}$ boundary
refinement through a 2D gradient vector [82], while $\omega_{\text{GM}}$ and
$\omega_{\text{WM/Sulci}}$ assign higher weights to PVE affected areas [5].

###  2.6 View aggregation

In order to account for the inherent 3D geometry of the brain, we adopt the
2.5D view aggregation scheme from [57, 5] for CNN*, VINN, and VINNA. In short,
we train one network instance per anatomical plane and calculate a weighted
average of the resulting softmax probability maps. The weight of the sagittal
predictions is reduced by one half compared to the other two views to account
for the missing lateralization in the sagittal view. In this plane, the
network predicts 23 instead of 88 classes.

###  2.7 Augmentations

External augmentation (exA) The current state-of-the-art approach to introduce
robustness to position changes into neural networks is extensive external
augmentation (see Figure 1B). Therefore, we contrast our proposed network-
integrated 4-DOF transform module against this approach. We use random
transforms with rotation parameters sampled from a uniform distribution of the
predefined range -180∘ to 180∘ and translation by 0 to 15px to augment images
during the training phase and interpolate linearly. For CNN* and nnUNet
augmentation also includes sampling of scaling parameters from a uniform
distribution of the predefined range 0.8 to 1.15. VINN´s resolution-
normalization module makes this step obsolete. Every minibatch hence consist
of a potentially transformed MRI (using bi-linear interpolation) and a
corresponding label map (using NN sampling). By exposing a network to a large
variety of possible image positions during training, orientation-robustness
can be achieved. All external augmentation routines are implemented using
torchIO [83].

Image intensity augmentation To allow generalization outside of the dHCP
cohort, we apply a number of intensity or texture augmentations on the fly to
the training batch, namely bias field changes, random gamma alterations,
ghosting, spiking, blurring, and Gaussian noise. Each batch sampled from the
original data is transformed by any of the operations above with a probability
of 0.4. As before, all augmentations are implemented using torchIO.

###  2.8 Evaluation metrics

We use the DSC [84, 85] and ASD to compare different network architectures and
modifications against each other, and to estimate similarity of the
predictions with a number of previously unseen scans. Both are standard
metrics to evaluate segmentation performance. We establish improvements by
statistical testing (Wilcoxon signed-rank test [86] after Benjamini-Hochberg
correction [87] for multiple testing) referred to as ”corrected p” throughout
the paper.

###  2.9 Training setup

Training dataset:  For training, we select 318 representative participants
from the dHCP cohort. Resolutions are equally represented with 106 MRI at
$1.0\text{\,}\mathrm{mm}$, $0.8\text{\,}\mathrm{mm}$, and
$0.5\text{\,}\mathrm{mm}$, respectively. Empty slices are filtered from the
volumes, leaving on average 137 single view planes per subject and a total
training size of at least 20k images per network. We train all directly
compared networks (CNN*, VINN, VINNA, 2D nnUNet and 3D nnUNet) under the same
conditions.

Training parameters:  We implement and train independent models to convergence
for the coronal, axial, and sagittal planes with PyTorch [88], using one
NVIDIA V100 GPU with 32 GB RAM. During training, the modified Adam optimizer
[89] is used with a learning rate of 0.001. Using a cosine annealing schedule
[90] with warm restarts, the learning rate is adapted after initially 10
epochs. The epoch offset is subsequently increased by a factor of two. The
momentum parameter is fixed at 0.95 to compensate for the relatively small
mini batch size of 16 images. To ensure a fair comparison, all networks (CNN*,
VINN, VINNA, 2D nnUNet and 3D nnUNet) have been trained under equal hardware
and hyper-parameter settings.

##  3 Results

We group the presentation of results into three blocks: 1. ablative
architecture improvements to determine the best performing module for
orientation and position transformation (Section 3.1), 2. performance analysis
to comprehensively characterize the advantages of VINNA with respect to state-
of-the-art traditional atlas- and deep-learning-based methods (Section 3.2)
and 3. external validation on M-CRIB [78] to asses generalizability and
performance with respect to manual labels (Section 3.3). Following best
practice in data-science, we utilize completely separate datasets during the
evaluations: the validation set for Section 3.1 (Table A: Validation), and
various test sets for Sections 3.2 and 3.3 (Table A: Testing). This avoids
data leakage and ensures that training, method design decisions, and final
testing do not influence each other, which could otherwise lead to overly
optimistic results (overfitting).

###  3.1 External augmentation vs network-integrated 4-DOF transform module in
VINNA

As high variances with respect to head orientations and spatial resolutions
are common in newborns and are likely to be underrepresented in the limited
available data cohorts, we first compare multiple approaches for extension of
the training distribution for accurate (sub)millimeter newborn whole brain
segmentation. Traditionally, external data augmentation (exA), such as
scaling, rotation, or translation, addresses this problem by interpolating
both, the image and label maps, to a new, random position during training. Due
to the discrete nature of the label maps, lossy NN interpolation cannot be
avoided. In contrast, our new 4-DOF transform module in VINNA internally
emulates possible head transformations and acts directly on the encoded
feature maps. To evaluate effectiveness of the exA versus VINNA, we compare
VINNA with parameter-identical CNN*, VINN, and VINNA equipped with exA. Each
subsequent improvement in segmentation performance is confirmed by statistical
testing (corrected $p<0.05$).

Figure 5: Comparison of approaches for external and internal spatial
augmentation on T2w images: Our VINNA method – with the 4-DOF transform module
– (third box in group) outperforms both state-of-the-art approaches with
external Augmentation (CNN* + exA and VINN + exA, first and second box) in
Dice Similarity Coefficient (DSC, upper row) and average surface distance
(ASD, lower row). This performance advantage is significant (corrected
$p<10^{-9}$, indicated with *) and consistent across resolutions and
structures. Combining VINNA with external augmentation (VINNA + exA, last box)
reduces performance. exA: external Augmentation.

In Figure 5, we compare the model performance of four approaches: CNN* and
VINN with exA (Section 2.4, CNN* + exA, left box; VINN + exA, second box from
the left), VINNA with the new 4-DOF transform module (VINNA, second box from
the right) and finally VINNA with exA (VINNA + exA, right box). The analysis
of the DSC (top) and ASD (bottom) is grouped for three groups of structures
(cortex averages 32 labels, WM averages 32 labels, and subcortical structures
average 20 labels) and three resolutions (from left to right
$0.5\text{\,}\mathrm{mm}$, $0.8\text{\,}\mathrm{mm}$ and
$1.0\text{\,}\mathrm{mm}$). We present performance for T2w MRI, but we found
the same ranking for T1w MRI777To reduce redundancy, we relegate that analysis
to the Appendix (Section A.1)..

Looking at the T2w segmentation and focusing on the different resolutions, the
differences between the approaches are largest on the subcortical structures.
The CNN* with exA reaches an average DSC of 91.91, 89.95 and 91.14 and an ASD
of $0.238\text{\,}\mathrm{mm}$, $0.192\text{\,}\mathrm{mm}$ and
$0.193\text{\,}\mathrm{mm}$ for input data of $0.5\text{\,}\mathrm{mm}$,
$0.8\text{\,}\mathrm{mm}$ and $1.0\text{\,}\mathrm{mm}$, respectively. The
slight reduction in performance for $0.8\text{\,}\mathrm{mm}$ resolution
consistently occurs for all evaluated models and is probably caused by the
necessary image resampling from the original resolution of
$0.5\text{\,}\mathrm{mm}$ and subsequent reprocessing with the dhcp-pipeline
(Section 2.2). Interpolation from $0.5\text{\,}\mathrm{mm}$ to
$1.0\text{\,}\mathrm{mm}$ results in a well aligned grid due to the even
division by factor 2 (8 voxels get averaged into a single larger voxel).
Resampling to $0.8\text{\,}\mathrm{mm}$ on the other hand, requires an uneven
interpolation grid with weighted averages and original voxels that contribute
to multiple larger voxels. This more challenging setting could result in the
slightly reduced segmentation performance. Optimization of the architecture
design towards multi-resolution (VINN, Section 2.4) leads to significant
improvement in the DSC and ASD across the cortical, WM, and subcortical
structures (Figure 5, VINN + exA). Particularly, the subcortical segmentations
are improved by around 0.5%. Importantly, the internal 4-DOF transform module
(VINNA), which avoids label interpolation all together, further reduces the
error by one half and increases segmentation performance significantly
compared to both CNN* and VINN with exA. This effect is consistent across all
structures and resolutions. Specifically, the ASD at the high-resolution
benefits from the new module. Here, performance can be improved by 4.47% on
the cortex, 5.19% on the WM and 5.89% on the subcortical structures. For the
lower resolution, the improvement on the cortex and WM is slightly lower
(around 2%) while the subcortical structures benefit from the 4-DOF transform
module similarly to the $0.5\text{\,}\mathrm{mm}$ resolution experiments.
Overall, VINNA reaches the highest DSC and lowest ASD for the cortical
structures (96.24, $0.079\text{\,}\mathrm{mm}$), WM (98.18,
$0.063\text{\,}\mathrm{mm}$) and subcortical structures (91.87,
$0.190\text{\,}\mathrm{mm}$) across all resolutions. The addition of exA to
the framework (VINNA + exA, right box in each plot) again reduces performance.
The DSC drops by 0.4 %, 0.28 % and 1.15 % on the cortex, WM and subcortical
structures on average, while the ASD worsens by 4.75 %, 3.92 % and 4.84 %.
Overall, results with VINNA are significantly better compared to all ablations
on the validation set (corrected $p<10^{-9}$).

###  3.2 Comparison to state-of-the-art neonate segmentation tools

To evaluate how VINNA compares to state-of-the-art neonate MRI segmentation
tools, namely nnUNet (3D and 2D), iBEAT, and infantFS, we take a closer look
at the DSC and ASD on the testing sets.

####  3.2.1 Comparison of deep-learning networks

Figure 6 shows a detailed comparison of three different deep-learning based
methods for neonate segmentation across modalities (T2w top, T1w bottom) and
resolutions. All models are trained under equal parameter and dataset
settings. Comparing performance between the two modalities shows, that all
models perform better on the T2w (top) than the T1w MRI (bottom) across all
structures. With VINNA (right box in each plot), the reduction is similar
across all resolutions, with an average difference in DSC of 5.85, 3.47, and
5.16 on the cortex, WM, and subcortical structures. The ASD is, on average,
improved by $0.09\text{\,}\mathrm{mm}$ when predicting on the T2w instead of
T1w inputs. The nnUNet framework in 2D (left box) and 3D (second from left)
has less improvement on the T2w images with an average difference between T1w
and T2w of 3.75, 2.44 and 3.70 in DSC and $0.04\text{\,}\mathrm{mm}$ ASD on
the aforementioned structures.

Figure 6: SOTA Segmentation performance: Our VINNA with the 4-DOF transform
module (last box in group) outperforms the three state-of-the-art deep-
learning approaches, 2D nnUNet + exA and 3D (first and second box) in Dice
Similarity Coefficient (DSC, upper row) and average surface distance (ASD,
lower row). This performance advantage is significant (corrected $p<10^{-10}$,
indicated with *) and consistent across three resolutions
($0.5\text{\,}\mathrm{mm}$, $0.8\text{\,}\mathrm{mm}$, and
$1.0\text{\,}\mathrm{mm}$), two modalities (T2w, top and T1w, bottom), and
three structure groups (cortex, WM and subcortical structures).

When comparing the four models, the 2D nnUNet + exA (left box) version
performs worse than the 3D (second from left box), and 2.5D VINNA (right box)
across all resolutions, structures, and modalities. Particularly notable are
the large variations of 2D nnUNet + exA in prediction performance (large
standard deviation) and large ASD (see Figure 6), especially at the highest
resolution ($0.42\text{\,}\mathrm{mm}$, for the cortex,
$0.46\text{\,}\mathrm{mm}$ for WM and $0.41\text{\,}\mathrm{mm}$ for
subcortical structures). This difference is less prominent in the DSC scores,
but 2D nnUNet + exA also performs worst across all resolutions with respect to
this metric (i.e. 92.39, 95.70, and 87.50 for a resolution of
$0.5\text{\,}\mathrm{mm}$). The 3D nnUNet + exA (second from left) improves
accuracy by 24.5 %, 12.3 %, and 22.9 % for ASD and 1.75 %, 1.06 %, and 1.64 %
for DSC across the three different resolutions. VINNA with its 4-DOF transform
module (VINNA, right box) is again the best performing model, significantly
outperforming all other networks. Compared to the 3D nnUNet + exA, ASD and DSC
scores are significantly improved with the highest gain on the cortical
structures (56 %, 35 %, and 45 % ASD and 3.8 %, 2.7 %, and 3.5 % DSC for
$0.5\text{\,}\mathrm{mm}$, $0.8\text{\,}\mathrm{mm}$ and
$1.0\text{\,}\mathrm{mm}$, respectively).

Figure 7: SOTA Segmentation performance across age groups: VINNA equipped with
the 4-DOF transform module (right box) consistently outperform state-of-the-
art deep-learning approaches 3D nnUNet with external augmentation (+exA, left
box) across four age groups. Improvement in dice similarity coefficient (DSC,
top) and average surface distance (ASD, bottom) on T2w at
$0.5\text{\,}\mathrm{mm}$ is significant (corrected $p<10^{-4}$, indicated
with *) for $<36$, $<40$ and $<46$ week old newborns.

To evaluate if this trend is consistent across age groups, the
$0.5\text{\,}\mathrm{mm}$ MRI images are split into three approximately equal
sized groups based on the participants’ age-at-scan information (32-36, 36-40
and 40-46 weeks). Figure 6 shows DSC (top) and ASD (bottom) calculated for T2w
(Figure 7) for the 3D nnUNet + exA (left box) and VINNA with its 4-DOF
transform module (right box) in each of the categories.

Consistent with the previous section, 3D nnUNet + exA reaches the weakest ASD
and DSC across all age groups. On average, the VINNA with its 4-DOF transform
module (right box) improves performance compared to nnUNet by 2.8, 2.9 and 2.8
% DSC and 41.5, 43.6 and 43.1 % ASD from the youngest (32-36) to the oldest
($<46$) age group and reaches a DSC of 96.86, 96.913, 97.17 for the cortical
structures, 98.96, 98.652, 98.362 for the WM structures and 92.18, 92.69,
92.80 for the subcortical structures across all age groups (youngest to
oldest) on the T2w MRI (Figure 7, top). Here, VINNA also reaches the lowest
ASD ($0.080\text{\,}\mathrm{mm}$, $0.081\text{\,}\mathrm{mm}$,
$0.081\text{\,}\mathrm{mm}$ for the cortical structures,
$0.067\text{\,}\mathrm{mm}$, $0.068\text{\,}\mathrm{mm}$,
$0.066\text{\,}\mathrm{mm}$, for the WM structures and
$0.231\text{\,}\mathrm{mm}$, $0.212\text{\,}\mathrm{mm}$,
$0.212\text{\,}\mathrm{mm}$, for the subcortical structures). The results are
significantly better compared to 3D nnUNet + exA (corrected $p<10^{-9}$). As
seen by the increasing DSC and decreasing ASD, the younger age groups
($<32-36$) have proved to be more challenging to segment. For VINNA, the
performance decreases most significantly for the subcortical structures (DSC
by 1.82 % and ASD by 20.09 %) and least on the cortex (0.35 % DSC and 4.65 %
ASD). This trend is consistent for the other two models. Assessment of
qualitative differences between the 3D nnUNet + exA and VINNA on a
representative participant at 40 weeks of age (Figure 10) shows slight over-
segmentation of the cortex and loss of small WM strands with the 3D nnUNet +
exA (third row, second column, arrows). Overall, the segmentation with VINNA
(fourth row) appears less smoothed and closer to the ground truth (second
row).

####  3.2.2 Comparison to iBEAT

In Figure 8, the deep-learning models are compared to the docker version of
iBEAT on the T2w images at $0.5\text{\,}\mathrm{mm}$. iBEAT is officially
designed for ages 0-6 years and the docker version we used for processing
returns three labels (WM, GM, CSF). The definition of these labels is
different from that of the dHCP-atlas (see Section 2.3.3), so we map the
ground truth as well as the predictions from the deep-learning networks
(nnUNet3D and VINNA) to be able to compare segmentation similarity. As
described in Section 2.3, retraining the CNN part of iBEAT under same data and
label definitions is not possible, as neither the source code nor the original
training data is available online. Note, even though we do not need to
interpolate, cross-protocol comparisons include atlas differences and may
introduce additional errors due to the mapping. While results should be
interpreted with the caveat that iBEAT uses a different atlas and training
dataset than 3D nnUNet + exA and VINNA, the label harmonization allows an as-
fair-as-possible comparison with this state-of-the-art method.

Figure 8: Deep-Learning networks versus iBEAT: Similarity to the dHCP
reference is higher with VINNA (third box) and 3D nnUNet + exA (second box)
than iBEAT (first box) with respect to dice similarity coefficient (DSC, top)
and average surface distance (ASD, bottom) on T2w MRI at 0.5 mm across three
age groups. Segmentation results with VINNA are significantly closer to dHCP
(corrected $p<10^{-6}$, indicated with *) for CSF, GM, and WM. Note, that
iBEAT´s structure definition is not identical to the dHCP-ALBERTs atlas and
analysis is based on harmonized, merged labels.

With respect to the mapped dHCP-reference segmentation, DSC (top) and ASD
(bottom) are lower for iBEAT (left box in each plot) compared to the the deep-
learning methods (3D nnUNet + exA, middle box, and VINNA, right box in each
plot). Performance of iBEAT on the GM, WM, and CSF improves with age. GM and
WM are closest to the reference at 36-40 weeks (DSC 80.29/87.94 and ASD
$0.684\text{\,}\mathrm{/}$$0.813\text{\,}\mathrm{mm}$) while CSF peaks at
32-26 weeks (73.49 and $0.896\text{\,}\mathrm{mm}$). The 3D nnUNet + exA and
VINNA show a similar trend, but performance is more consistent. As mentioned
before, the differences are not necessarily due to wrong predictions made by
iBEAT. Looking at the qualitative comparison in Figure 10, differences appear
small, with iBEAT (third row) missing a few WM strands (arrow) and slightly
over-segmenting the cortex compared to the mapped ground truth (second row).

####  3.2.3 Comparison to infantFS

Figure 9 shows performance comparison between infantFS (left box) and the
deep-learning methods, 3D nnUNet + exA (middle box) and VINNA (left box), but
in contrast to previous evaluations on T1w images at 1.0 mm, the operating
resolution and modality of infantFS. Note, that the infantFS labels are also
different from those of dHCP and the ground truth labels must be mapped (see
Section 2.3.3 for details). As for iBEAT, the cross-protocol comparison can
put the traditional method at an unfair disadvantage and results should be
considered with this caveat.

Figure 9: Deep-learning networks versus infantFS: VINNA with the 4-DOF
transform module (third box) and 3D nnUNet + exA (second box) are closer to
the dHCP reference than infantFS (first box) on T1w at
$1.0\text{\,}\mathrm{mm}$, the supported modality and resolution for infantFS.
Dice Similarity Coefficient (DSC, top) and average surface distance (ASD,
bottom) significantly improve with VINNA (corrected $p<10^{-6}$, indicated
with *) on the cortex, WM, hippocampus and amygdala across all age groups.
Note, that definition of subcortical structures differs in infantFS and
predictions are harmonized to allow comparison to the deep-learning models.

Overall, the infantFS predictions differ strongly from the mapped dHCP ground
truth, specifically for the younger age ranges. The highest similarity is
reached for the subcortical structures (amygdala and hippocampus) at 40-46
weeks (DSC of 70.72/70.56 and ASD of
$0.736\text{\,}\mathrm{/}$$1.23\text{\,}\mathrm{mm}$, respectively). The
cortex and WM reach a maximum DSC of 60.60/41.62 and ASD of
$1.0\text{\,}\mathrm{/}$$13.20\text{\,}\mathrm{mm}$. Qualitative comparison
(see Figure 10, third row, left panel) shows difficulties with the correct
location of the GM and WM border on the dHCP T1w MRI. Larger portions of the
cortex are under-segmented and strands of WM are lost compared to the ground
truth. The deep-learning methods reach a DSC above 80 and an ASD below
$0.5\text{\,}\mathrm{mm}$ for all structures and age groups. The method
closest to the dHCP reference is again VINNA with the 4-DOF transform module
(left box), followed by 3D nnUNet + exA (middle box). For a detailed
comparison between nnUNet and VINNA on T1w, see Figure 7.

Figure 10: Qualitative T1w and T2w MRI segmentations on a representative scan
at 41 weeks. VINNA with the 4-DOF transform module (last row) captures
structural details lost in other methods. Comparison of the mapped ground
truth (top) and segmentations from iBEAT (third row), 3D nnUNet + exA (fourth
row) on a representative participants T2w MRI (left). The right side shows the
T1w-scan from the same participant at 1.0 mm with ground truth (top), infantFS
(third row) and the deep-learning methods.

###  3.3 External validation on manual labels (M-CRIB)

To assess generalizability to a different dataset in our target age range
(20-40 weeks) and to provide results with respect to a manual reference, we
compare the segmentations produced by VINNA, 3D nnUNet + exA, and the dhcp-
minimal-processing-pipeline to the $0.62\text{\,}\mathrm{mm}$ high-resolution
T2w scans forming the M-CRIB atlas [78]. This dataset contains T2w MRI from 10
participants and accompanying label maps based on the Desikian-Killiany-
Tourville (DKT) atlas [79]. Note, that the labels are not identical to the
dHCP-ALBERTs atlas. Hence, we combine the cortical parcels to one label
(cortex) for the segmentation comparison and mask all but three structures
(WM, hippocampus, and lateral ventricles). As iBEAT does not differentiate
between subcortical structures and GM or WM, nor CSF and ventricles, mapping
of both, the ground truth and prediction, would be different compared to
nnUNet, VINNA, and the dhcp-minimal-processing-pipeline. A fair comparison is
therefore only possible between the latter methods, and iBEAT is thus not
included in the following section.

Figure 11: External validation of segmentation performance on the M-CRIB
dataset. VINNA (second box) outperforms nnUNet3D (first box) on all four
structures and the dhcp-strucutral-pipeline (third box) on the hippocampus.
Dice Similarity Coefficient (DSC) and average surface distance (ASD) for
cortex, WM, hippocampus and lateral-ventricles are calculated with respect to
harmonized manual labels on ten subjects from M-CRIB. Note, that atlas
definition differs between ground truth and predictions.

In Figure 11, the DSC (top) and ASD (bottom) are compared over four structures
(from left to right: cortex, WM, hippocampus and lateral-ventricles) for the
deep-learning methods (3D nnUNet + exA and VINNA) as well as the dhcp-minimal-
processing-pipeline. Predictions for these methods are all based on the same
label definition (dHCP-ALBERTs atlas [18, 25]). As for the dHCP test set,
VINNA outperforms the 3D versions of nnUNet across all four structures with a
DSC of 83.83 and an ASD of $0.387\text{\,}\mathrm{mm}$ on the cortex, 78.41
and $4.268\text{\,}\mathrm{mm}$ on the WM, 61.73 and
$3.682\text{\,}\mathrm{mm}$ on the hippocampus, and 84.62 and
$0.585\text{\,}\mathrm{mm}$ on the lateral ventricles. Compared to nnUNet, the
performance improves on average by 3.36 % for the DSC and 19.40 % for the ASD.
Furthermore, 3D nnUNet + exA incorrectly flips left-right labels for five
participants (P02-P04, P08-P09). We restore the lateralization in the
presented results as DSC and ASD would have otherwise been close to zero for
half of the participants. On the hippocampus, ventricles, and WM performance
of VINNA is similar to the dhcp-minimal-processing-pipeline, which is the best
performing method. Specifically, predictions of the cortex are closer to the
M-CRIB manual labels with the dhcp-pipeline (DSC of 89.60 and ASD of
$0.257\text{\,}\mathrm{mm}$), which is likely due to the reliance on surface
models, while the trained deep-learning models seem to over-segment the cortex
(indicated by the similarity of the ASD, but larger differences in the DSC).
Note that VINNA performs slightly better on the hippocampus with an increase
in DSC by 1.25 % and ASD by 0.58 %. Due to the low number of participants,
significance tests are not applicable for these experiments.

##  4 Discussion

In this paper, we present VINNA – a resolution-independent network for native-
resolution neonatal brain MRI segmentation. With VINNA and our novel network-
integrated 4-DOF transform module, we address two main difficulties associated
with neonate segmentation: resolution non-uniformity across data cohorts and
the extended range of head positions in infants.

In contrast to adults, newborn head positioning in the scanner varies
significantly due to imaging during sleep, smaller head sizes, and relevant
necessary modifications to scanner equipment, such as the padding of head
coils. Additionally, while scans are commonly recorded at high resolutions, no
uniform standard exists across cohorts. The availability of newborn datasets
is also scarcer than that of adult subjects and the existing collections to
date are unlikely to represent that wide diversity in resolutions and head
positions.

The current state-of-the-art to address spatial variability such as head
positions is data augmentation, which applies randomly sampled scale,
rotation, and translation transformations in the native imaging space (i.e.,
externally to both intensity and label map). In VINNA, we introduce the 4-DOF
transform module that can apply such transformations internally as part of the
network. As the parameters to the transformation are inputs to the network,
they can be randomized during training, similarly to external data
augmentation methods. Moving the augmentation operation into the network, so
it acts upon feature maps instead of inputs, marks a paradigm shift for data
augmentation strategies. While we only implemented an augmentation of 4 DOFs
here, the concept may be generalized to 9 DOFs for 3D or even to warp fields
as well as to other tasks such as classification, regression, etc.

We demonstrate that the new network-integrated 4-DOF transform with internal
augmentation outperforms state-of-the-art external augmentation approaches in
CNN*, VINN, and nnUNet [73] on the dHCP cohort. Across three different
resolutions and two modalities, our VINNA achieves the highest DSC (95.33 on
average), as well as the lowest ASD ($0.102\text{\,}\mathrm{mm}$ on average).
Metric evaluation combined with qualitative inspection indicate that the
internal 4-DOF transform module in VINNA better retains high-level details
across all resolutions and age groups.

To better explain the factors and mechanisms driving the performance
improvements of VINNA, we review the observation from FastSurferVINN [5] that
motivated the extension presented here: in one-to-one comparisons, external
augmentation reduces the segmentation performance on sub-millimeter MRI. While
– at first sight – the addition of data augmentation reducing performance
seems contradictory, the one-to-one comparison of VINNA and VINNA + exA (see
Figure 5, the only difference is added external augmentation) robustly
confirms the observation and extends it from just scaling to rigid transforms.

The positive effect of data augmentation is usually associated with an
expansion of the input dataset through equivariant operations. Implementing
operations for image and label pairs that are truly equivariant can be
difficult. We believe that the loss of information due to image interpolation
(lossy interpolation of the label map and image) is larger than previously
believed. Internal augmentation, for the first time, offers an alternative
approach with interpolation of continuous values in multiple feature maps,
reducing the information loss.

Furthermore, the 4-DOF transform module together with the internal
augmentation regularizes the latent space of the network, because it imposes
an additional constraint: spatial consistency of the feature maps. Compared to
equivalent CNN architectures, VINNA (and VINN) also benefit from a reduced
capacity requirement to capture a large range of resolutions.

Our comparison to nnUNet highlights, that 2D approaches lack contextual
information, and fail to provide reliable predictions for whole brain
segmentation. The compromise between mid-range and long-range context in the
2.5D VINNA recovers structural information better and achieves higher
segmentation performance across all age groups and structures - even compared
to 3D methods. As full-view 3D networks are currently not applicable for high-
resolution MRI segmentation due to memory requirements, nnUNet and other 3D
networks rely on patch-based processing. In this case, the increased 3D
context comes at the cost of limited long-range information and features a
smaller field of view, potentially explaining the observed reduction in
accuracy compared to 2.5D networks. This finding is in line with previous
investigations which found limited performance differences between 2.5D and 3D
approaches, even after extensive optimization of the 3D network architectures
[91].

On the dHCP cohort, VINNA and its 4-DOF transform module also emulates the
segmented structures better than traditional state-of-the-art infant
pipelines, namely infantFS[23] and iBEAT[30]. Notably, infantFS relies on
traditional atlas-based segmentation while iBEAT uses a combination of deep-
learning and traditional tools with a number of CNN trained on defined target
age-ranges. While the re-trained networks (nnUNet + exA and VINNA) reach
better results with respect to DSC and ASD, it should be noted that both,
infantFS and iBEAT, differ significantly with respect to the returned number
and definition of segmented regions. The necessary mapping between the
segmentations is bound to introduce a bias, which can not be easily assessed.
Additionally, both pipelines cater to a slightly different, larger age range
(0-2 years for infantFS and 0-6 years for iBEAT). Consequently, predictions
from both methods of the cortex and WM improve for participants closer to the
officially supported age-range ($>$ 40 weeks). Qualitative assessment also
shows good performance for the older newborns in iBEAT. infantFS unfortunately
fails to correctly capture the cortex and WM on the majority of participants.
The original iBEAT v2.0 paper[30] also evaluates performance on the dHCP data
and reports a DSC of 0.9 for the WM and 0.85 for the GM. Our results are in
concordance with this assessment for the $>$ 40 weeks old participants (DSC of
0.83 on the GM and 0.88 on the WM). The authors do not provide information on
their label harmonization, therefore we can not infer their reference
standard. In contrast to the docker v2.0 version, the cloud version of iBEAT
(not available for local data processing) does provide cortical parcellations.
Extracting the cortex from the GM label (a combination of both cortical and
subcortical GM in the docker version) allows direct comparison to the dHCP
solution after merging its cortical structures, possibly explaining
performance differences.. In summary, iBEAT seems to work well on the
supported age range while infantFS is less precise on the dHCP population.
Other features included in the pipelines, such as surface generations are an
advantage compared to the proposed VINNA and can help to refine the
segmentation of convoluted structures such as the cortex [60]. For our target
domain in this paper, however, the VINNA architecture appears to emulate the
investigated tissue classes more precisely.

Due to the limited extrapolation capabilities of neural networks,
generalizability beyond the training set distribution is, however, uncertain.
While the 4-DOF transform module in VINNA serves as a diversification of the
training distribution with respect to spatial orientations and image
resolution, the base cohort is still only a representation of the dHCP
population, i.e. all scans encountered during training represent newborns
between 20-40 weeks post-gestational age from a control cohort acquired on the
same 3 T Phillips scanner. Therefore, dedicated experimental validation is
required to confirm the models’ effectiveness under differing conditions. As
for all automated methods, manual quality checks of the predictions are
recommended. While VINNA does perform well on M-CRIB, which covers the same
age range as the dHCP, generalization to other cohorts is not necessarily
guaranteed. Specifically, the T1w image intensities in dHCP appear
significantly different from other cohorts which might also explain why
infantFS performs poorly on the testing set. The T2w MRI in dHCP are, on
average, of better quality and the dhcp-minimal-processing-pipeline builds the
ground truth segmentations based on it [12]. Additionally, in the early weeks
of life, tissue contrast is higher in T2w recordings as the brain is not fully
matured and myelination is still ongoing [8, 92]. Structural details and
specifically tissue boundaries might be missing, blurred, or ambiguous in T1w
MRI. Hence, the imaging data may lack sufficient information to allow correct
delineation of the (sub-)cortical structures. This may also explain why the
deep-learning networks (i.e. nnUNet, CNN*, VINN), and VINNA are not able to
emulate the ground truth on the T1w MRI as closely as on the T2w images.

Better accessibility of newborn datasets would allow diversification of the
training sets and subsequently a better representation of the newborn MRI
distribution with respect to both, T1w and T2w modalities. It has been shown
that an increase in the training corpus alone is extremely effective to boost
performance [93, 5]. Age specific models, as our VINNA or iBEAT´s CNN, are
another way to reduce variations and therefore, segregate the problem (i.e.
less variations within one age group). However, the limited data availability
and non-uniform segmentation labels still pose significant barriers. Models
for more specific segmentations than just CSF, WM and GM are currently not
trainable in a supervised fashion due to missing ground truth. In addition,
definition of these three structures alone already varies across different
atlases and tools, which makes fair method comparisons challenging. Neither
manual labels nor automated segmentation tools exist for a unified
segmentation definition across different resolutions, modalities and age-
ranges. The M-CRIB atlas [78], an infant-specific version of the Desikan-
Killiany-Tourville-Atlas [79] that is commonly used in adults, provides a
first step towards this goal. A consistent structure definition across
different stages of life is especially important in the context of
longitudinal studies, as segmentation with age-dependent models can induce
biases and reduce anatomical consistency [94]. How to solve this conundrum is
an open question for the future. Several NIH- and internationally-funded
initiatives have recently been dedicated to acquire data from newborns [12],
infants and pediatric age ranges [11, 14, 95] as well as adolescence [96]. Due
to the easy integration of varying data resolutions and accommodation for head
position variations between infants, toddlers, and adults, our VINNA
architecture might prove to be useful in this area once the data and label
availability problem is resolved.

Overall, with VINNA, we provide a fast and accurate method for high-resolution
subcortical structure segmentation, cortical and WM parcellation of neonatal
T1w and T2w MRI which generalizes well across the dHCP cohort. The presented
4-DOF transform module is also easy to integrate into other network
architectures and might prove useful in different areas dealing with strong
orientation variations. The application to neonates will be made available
under VINNA4neonates as an open source package888on our github pages.
Adaptation of the infantFS surface pipeline to the VINNA4neonates predictions,
similar to the approach taken in FastSurfer[57] for adults, is an exciting
direction for future work.

##  5 Data and Code Availability

All MRI datasets used within this article are publicly available and the open
source repositories are cited within the article (Section 2.1).

The source code of VINNA4neonates will be made publicly available on Github
(https://github.com/deep-mi/NeonateVINNA) upon acceptance.

##  6 Author Contributions

Leonie Henschel: Conceptualization, Methodology, Software, Validation, Formal
analysis, Investigation, Writing - original draft, Writing - review & editing,
Visualization.

David Kügler: Conceptualization, Methodology, Software, Writing - review &
editing, Supervision.

Martin Reuter: Conceptualization, Methodology, Resources, Writing - review &
editing, Supervision, Project administration, Funding acquisition.

Lilla Zöllei: Conceptualization, Methodology, Resources, Writing - review &
editing, Supervision, Project administration, Funding acquisition.

##  7 Competing Interests

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

##  8 Acknowledgements and Funding

This work was supported by DZNE institutional funds, by the Federal Ministry
of Education and Research of Germany (031L0206, 01GQ1801), by NIH (R01
LM012719, R01 AG064027, R56 MH121426, P41 EB030006, and 5R01 HD109436-01),
NICHD, and a NIH Shared Instrument Grant (S10RR023043). Funding was further
received from the Chan-Zuckerberg Initiatives Essential Open Source Software
for Science RFA (EOSS5 2022-252594).

Data were provided by the Melbourne Children´s Regional Infant Brain Atlas and
the Developing Human Connectome Project, KCL-Imperial-Oxford Consortium funded
by the European Research Council under the European Union Seventh Framework
Programme (FP/2007-2013) / ERC Grant Agreement no. [319456]. We are grateful
to the families who generously supported this trial.

## References

  * [1] M. Ajayi-Obe, N. Saeed, F. Cowan, M. Rutherford, A. Edwards, Reduced development of cerebral cortex in extremely preterm infants, The Lancet 356 (9236) (2000) 1162–1163.  doi:10.1016/S0140-6736(00)02761-6.
URL https://linkinghub.elsevier.com/retrieve/pii/S0140673600027616

  * [2] M. Prastawa, J. H. Gilmore, W. Lin, G. Gerig, Automatic segmentation of MR images of the developing newborn brain, Medical Image Analysis 9 (5) (2005) 457–466.  doi:10.1016/j.media.2005.05.007.
URL https://linkinghub.elsevier.com/retrieve/pii/S1361841505000630

  * [3] J. Dubois, G. Dehaene-Lambertz, S. Kulikova, C. Poupon, P. Hüppi, L. Hertz-Pannier, The early development of brain white matter: A review of imaging studies in fetuses, newborns and infants, Neuroscience 276 (2014) 48–71.  doi:10.1016/j.neuroscience.2013.12.044.
URL https://linkinghub.elsevier.com/retrieve/pii/S0306452213010695

  * [4] J. H. Gilmore, F. Shi, S. L. Woolson, R. C. Knickmeyer, S. J. Short, W. Lin, H. Zhu, R. M. Hamer, M. Styner, D. Shen, Longitudinal Development of Cortical and Subcortical Gray Matter from Birth to 2 Years, Cerebral Cortex 22 (11) (2012) 2478–2485.  doi:10.1093/cercor/bhr327.
URL https://academic.oup.com/cercor/article-lookup/doi/10.1093/cercor/bhr327

  * [5] L. Henschel, D. Kügler, M. Reuter, Fastsurfervinn: Building resolution-independence into deep learning segmentation methods—a solution for highres brain mri, NeuroImage 251 (2022) 118933.  doi:https://doi.org/10.1016/j.neuroimage.2022.118933.
URL https://www.sciencedirect.com/science/article/pii/S1053811922000623

  * [6] A. Copeland, E. Silver, R. Korja, S. J. Lehtola, H. Merisaari, E. Saukko, S. Sinisalo, J. Saunavaara, T. Lähdesmäki, R. Parkkola, S. Nolvi, L. Karlsson, H. Karlsson, J. J. Tuulari, Infant and Child MRI: A Review of Scanning Procedures, Frontiers in Neuroscience 15 (2021) 666020\.  doi:10.3389/fnins.2021.666020.
  * [7] J. Dubois, J. Lefèvre, H. Angleys, F. Leroy, C. Fischer, J. Lebenberg, G. Dehaene-Lambertz, C. Borradori-Tolsa, F. Lazeyras, L. Hertz-Pannier, J.-F. Mangin, P. S. Hüppi, D. Germanaud, The dynamics of cortical folding waves and prematurity-related deviations revealed by spatial and spectral analysis of gyrification, NeuroImage 185 (2019) 934–946.  doi:https://doi.org/10.1016/j.neuroimage.2018.03.005.
URL https://www.sciencedirect.com/science/article/pii/S1053811918301903

  * [8] J. Dubois, M. Alison, S. J. Counsell, L. Hertz-Pannier, P. S. Hüppi, M. J. Benders, Mri of the neonatal brain: A review of methodological challenges and neuroscientific advances, Journal of Magnetic Resonance Imaging 53 (5) (2021) 1318–1343.  doi:https://doi.org/10.1002/jmri.27192.
URL https://onlinelibrary.wiley.com/doi/abs/10.1002/jmri.27192

  * [9] P. Liu, C. Parkinson, D. Jiang, M. Ouyang, J. B. De Vis, F. J. Northington, A. Tekes, H. Huang, T. A. Huisman, W. C. Golden, Characterization of mri techniques to assess neonatal brain oxygenation and blood flow, NMR in Biomedicine 32 (7) (2019) e4103, e4103 NBM-18-0273.R1.  arXiv:https://analyticalsciencejournals.onlinelibrary.wiley.com/doi/pdf/10.1002/nbm.4103, doi:https://doi.org/10.1002/nbm.4103.
URL
https://analyticalsciencejournals.onlinelibrary.wiley.com/doi/abs/10.1002/nbm.4103

  * [10] A. C. Evans, The nih mri study of normal brain development, NeuroImage 30 (1) (2006) 184–202.  doi:https://doi.org/10.1016/j.neuroimage.2005.09.068.
URL https://www.sciencedirect.com/science/article/pii/S105381190500710X

  * [11] B. R. Howell, M. A. Styner, W. Gao, P.-T. Yap, L. Wang, K. Baluyot, E. Yacoub, G. Chen, T. Potts, A. Salzwedel, G. Li, J. H. Gilmore, J. Piven, J. K. Smith, D. Shen, K. Ugurbil, H. Zhu, W. Lin, J. T. Elison, The unc/umn baby connectome project (bcp): An overview of the study design and protocol development, NeuroImage 185 (2019) 891–905.  doi:https://doi.org/10.1016/j.neuroimage.2018.03.049.
URL https://www.sciencedirect.com/science/article/pii/S1053811918302593

  * [12] A. Makropoulos, E. C. Robinson, A. Schuh, R. Wright, S. Fitzgibbon, J. Bozek, S. J. Counsell, J. Steinweg, K. Vecchiato, J. Passerat-Palmbach, G. Lenz, F. Mortari, T. Tenev, E. P. Duff, M. Bastiani, L. Cordero-Grande, E. Hughes, N. Tusor, J.-D. Tournier, J. Hutter, A. N. Price, R. P. A. Teixeira, M. Murgasova, S. Victor, C. Kelly, M. A. Rutherford, S. M. Smith, A. D. Edwards, J. V. Hajnal, M. Jenkinson, D. Rueckert, The developing human connectome project: A minimal processing pipeline for neonatal cortical surface reconstruction, NeuroImage 173 (2018) 88–112.  doi:https://doi.org/10.1016/j.neuroimage.2018.01.054.
URL https://www.sciencedirect.com/science/article/pii/S1053811918300545

  * [13] M. Bastiani, J. L. Andersson, L. Cordero-Grande, M. Murgasova, J. Hutter, A. N. Price, A. Makropoulos, S. P. Fitzgibbon, E. Hughes, D. Rueckert, S. Victor, M. Rutherford, A. D. Edwards, S. M. Smith, J.-D. Tournier, J. V. Hajnal, S. Jbabdi, S. N. Sotiropoulos, Automated processing pipeline for neonatal diffusion mri in the developing human connectome project, NeuroImage 185 (2019) 750–763.  doi:https://doi.org/10.1016/j.neuroimage.2018.05.064.
URL https://www.sciencedirect.com/science/article/pii/S1053811918304889

  * [14] N. D. Volkow, J. A. Gordon, M. P. Freund, The Healthy Brain and Child Development Study—Shedding Light on Opioid Exposure, COVID-19, and Health Disparities, JAMA Psychiatry 78 (5) (2021) 471–472.  arXiv:https://jamanetwork.com/journals/jamapsychiatry/articlepdf/2774000/jamapsychiatry\\_volkow\\_2020\\_vp\\_200048\\_1619663118.80156.pdf, doi:10.1001/jamapsychiatry.2020.3803.
URL https://doi.org/10.1001/jamapsychiatry.2020.3803

  * [15] F. Shi, P.-T. Yap, G. Wu, H. Jia, J. H. Gilmore, W. Lin, D. Shen, Infant brain atlases from neonates to 1- and 2-year-olds, PLOS ONE 6 (4) (2011) 1–11.  doi:10.1371/journal.pone.0018746.
URL https://doi.org/10.1371/journal.pone.0018746

  * [16] M. Kuklisova-Murgasova, P. Aljabar, L. Srinivasan, S. J. Counsell, V. Doria, A. Serag, I. S. Gousias, J. P. Boardman, M. A. Rutherford, A. D. Edwards, J. V. Hajnal, D. Rueckert, A dynamic 4d probabilistic atlas of the developing brain, NeuroImage 54 (4) (2011) 2750–2763.  doi:https://doi.org/10.1016/j.neuroimage.2010.10.019.
  * [17] A. Serag, P. Aljabar, G. Ball, S. J. Counsell, J. P. Boardman, M. A. Rutherford, A. D. Edwards, J. V. Hajnal, D. Rueckert, Construction of a consistent high-definition spatio-temporal atlas of the developing brain using adaptive kernel regression, NeuroImage 59 (3) (2012) 2255–2265.  doi:https://doi.org/10.1016/j.neuroimage.2011.09.062.
  * [18] I. S. Gousias, A. D. Edwards, M. A. Rutherford, S. J. Counsell, J. V. Hajnal, D. Rueckert, A. Hammers, Magnetic resonance imaging of the newborn brain: Manual segmentation of labelled atlases in term-born and preterm infants, NeuroImage 62 (3) (2012) 1499–1509.  doi:https://doi.org/10.1016/j.neuroimage.2012.05.083.
URL https://www.sciencedirect.com/science/article/pii/S1053811912005915

  * [19] A. Makropoulos, P. Aljabar, R. Wright, B. Hüning, N. Merchant, T. Arichi, N. Tusor, J. V. Hajnal, A. D. Edwards, S. J. Counsell, D. Rueckert, Regional growth and atlasing of the developing human brain, NeuroImage 125 (2016) 456–478.  doi:https://doi.org/10.1016/j.neuroimage.2015.10.047.
URL https://www.sciencedirect.com/science/article/pii/S1053811915009581

  * [20] Y. Dai, F. Shi, L. Wang, G. Wu, D. Shen, iBEAT: A Toolbox for Infant Brain Magnetic Resonance Image Processing, Neuroinformatics 11 (2) (2013) 211–225.  doi:10.1007/s12021-012-9164-z.
URL http://link.springer.com/10.1007/s12021-012-9164-z

  * [21] M. Cherel, F. Budin, M. Prastawa, G. Gerig, K. Lee, C. Buss, A. Lyall, K. Zaldarriaga Consing, M. Styner, Automatic tissue segmentation of neonate brain MR Images with subject-specific atlases, Orlando, Florida, United States, 2015, p. 941311.  doi:10.1117/12.2082209.
URL
http://proceedings.spiedigitallibrary.org/proceeding.aspx?doi=10.1117/12.2082209

  * [22] J. C. Prieto, M. A. Styner, K. Pham, X. Yang, M. Niethammer, Multiseg pipeline: automatic tissue segmentation of brain MR images with subject-specific atlases, in: B. Gimi, A. Krol (Eds.), Medical Imaging 2019: Biomedical Applications in Molecular, Structural, and Functional Imaging, SPIE, San Diego, United States, 2019, p. 18.  doi:10.1117/12.2513237.
URL https://www.spiedigitallibrary.org/conference-proceedings-of-
spie/10953/2513237/Multiseg-pipeline--automatic-tissue-segmentation-of-brain-
MR-images/10.1117/12.2513237.full

  * [23] L. Zöllei, J. E. Iglesias, Y. Ou, P. E. Grant, B. Fischl, Infant freesurfer: An automated segmentation and surface extraction pipeline for t1-weighted neuroimaging data of infants 0–2 years, NeuroImage 218 (2020) 116946\.  doi:https://doi.org/10.1016/j.neuroimage.2020.116946.
URL https://www.sciencedirect.com/science/article/pii/S1053811920304328

  * [24] F. Shi, P.-T. Yap, Y. Fan, J. H. Gilmore, W. Lin, D. Shen, Construction of multi-region-multi-reference atlases for neonatal brain MRI segmentation, NeuroImage 51 (2) (2010) 684–693.  doi:10.1016/j.neuroimage.2010.02.025.
URL https://linkinghub.elsevier.com/retrieve/pii/S105381191000193X

  * [25] A. Makropoulos, I. S. Gousias, C. Ledig, P. Aljabar, A. Serag, J. V. Hajnal, A. D. Edwards, S. J. Counsell, D. Rueckert, Automatic whole brain mri segmentation of the developing neonatal brain, IEEE Transactions on Medical Imaging 33 (9) (2014) 1818–1831.  doi:10.1109/TMI.2014.2322280.
  * [26] S. Ahmad, Z. Wu, G. Li, L. Wang, W. Lin, P.-T. Yap, D. Shen, Surface-constrained volumetric registration for the early developing brain, Medical Image Analysis 58 (2019) 101540.  doi:https://doi.org/10.1016/j.media.2019.101540.
URL https://www.sciencedirect.com/science/article/pii/S1361841519300763

  * [27] J. Lebenberg, M. Labit, G. Auzias, H. Mohlberg, C. Fischer, D. Rivière, E. Duchesnay, C. Kabdebon, F. Leroy, N. Labra, F. Poupon, T. Dickscheid, L. Hertz-Pannier, C. Poupon, G. Dehaene-Lambertz, P. Hüppi, K. Amunts, J. Dubois, J.-F. Mangin, A framework based on sulcal constraints to align preterm, infant and adult human brain images acquired in vivo and post mortem, Brain Structure and Function 223 (9) (2018) 4153–4168.  doi:10.1007/s00429-018-1735-9.
URL http://link.springer.com/10.1007/s00429-018-1735-9

  * [28] K. E. Garcia, E. C. Robinson, D. Alexopoulos, D. L. Dierker, M. F. Glasser, T. S. Coalson, C. M. Ortinau, D. Rueckert, L. A. Taber, D. C. V. Essen, C. E. Rogers, C. D. Smyser, P. V. Bayly, Dynamic patterns of cortical expansion during folding of the preterm human brain, Proceedings of the National Academy of Sciences 115 (12) (2018) 3156–3161.  doi:10.1073/pnas.1715451115.
  * [29] J. Bozek, A. Makropoulos, A. Schuh, S. Fitzgibbon, R. Wright, M. F. Glasser, T. S. Coalson, J. O’Muircheartaigh, J. Hutter, A. N. Price, L. Cordero-Grande, R. P. A. Teixeira, E. Hughes, N. Tusor, K. P. Baruteau, M. A. Rutherford, A. D. Edwards, J. V. Hajnal, S. M. Smith, D. Rueckert, M. Jenkinson, E. C. Robinson, Construction of a neonatal cortical surface atlas using multimodal surface matching in the developing human connectome project, NeuroImage 179 (2018) 11–29.  doi:https://doi.org/10.1016/j.neuroimage.2018.06.018.
  * [30] L. Wang, Z. Wu, L. Chen, Y. Sun, W. Lin, G. Li, iBEAT V2.0: a multisite-applicable, deep learning-based pipeline for infant cerebral cortical surface reconstruction, Nature Protocols 18 (5) (2023) 1488–1509.  doi:10.1038/s41596-023-00806-x.
URL https://www.nature.com/articles/s41596-023-00806-x

  * [31] J. Dolz, C. Desrosiers, L. Wang, J. Yuan, D. Shen, I. Ben Ayed, Deep CNN ensembles and suggestive annotations for infant brain MRI segmentation, Computerized Medical Imaging and Graphics 79 (2020) 101660.  doi:10.1016/j.compmedimag.2019.101660.
URL https://linkinghub.elsevier.com/retrieve/pii/S0895611119300771

  * [32] S. Qamar, H. Jin, R. Zheng, P. Ahmad, M. Usama, A variant form of 3D-UNet for infant brain segmentation, Future Generation Computer Systems 108 (2020) 613–623.  doi:10.1016/j.future.2019.11.021.
URL https://linkinghub.elsevier.com/retrieve/pii/S0167739X18332291

  * [33] D. Nie, L. Wang, Y. Gao, D. Shen, Fully convolutional networks for multi-modality isointense infant brain image segmentation, in: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), IEEE, Prague, 2016, pp. 1342–1345.  doi:10.1109/ISBI.2016.7493515.
URL https://ieeexplore.ieee.org/document/7493515/

  * [34] W. Zhang, R. Li, H. Deng, L. Wang, W. Lin, S. Ji, D. Shen, Deep convolutional neural networks for multi-modality isointense infant brain image segmentation, NeuroImage 108 (2015) 214–224.  doi:10.1016/j.neuroimage.2014.12.061.
URL https://linkinghub.elsevier.com/retrieve/pii/S1053811914010660

  * [35] Z. Zeng, T. Zhao, L. Sun, Y. Zhang, M. Xia, X. Liao, J. Zhang, D. Shen, L. Wang, Y. He, 3D‐MASNet: mixed‐scale asymmetric convolutional segmentation network for 6‐month‐old infant brain MR images, Human Brain Mapping 44 (4) (2023) 1779–1792.  doi:10.1002/hbm.26174.
URL https://onlinelibrary.wiley.com/doi/10.1002/hbm.26174

  * [36] G. Zeng, G. Zheng, Multi-stream 3d fcn with multi-scale deep supervision for multi-modality isointense infant brain mr image segmentation, in: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 136–140.  doi:10.1109/ISBI.2018.8363540.
  * [37] S. Kumar, S. Conjeti, A. G. Roy, C. Wachinger, N. Navab, Infinet: Fully convolutional networks for infant brain mri segmentation, in: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 145–148.  doi:10.1109/ISBI.2018.8363542.
  * [38] T. D. Bui, L. Wang, J. Chen, W. Lin, G. Li, D. Shen, Multi-task Learning for Neonatal Brain Segmentation Using 3D Dense-Unet with Dense Attention Guided by Geodesic Distance, in: Q. Wang, F. Milletari, H. V. Nguyen, S. Albarqouni, M. J. Cardoso, N. Rieke, Z. Xu, K. Kamnitsas, V. Patel, B. Roysam, S. Jiang, K. Zhou, K. Luu, N. Le (Eds.), Domain Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data, Vol. 11795, Springer International Publishing, Cham, 2019, pp. 243–251.  doi:10.1007/978-3-030-33391-1_28.
URL http://link.springer.com/10.1007/978-3-030-33391-1_28

  * [39] P. Moeskops, M. J. Benders, S. M. Chiţǎ, K. J. Kersbergen, F. Groenendaal, L. S. De Vries, M. A. Viergever, I. Išgum, Automatic segmentation of MR brain images of preterm infants using supervised classification, NeuroImage 118 (2015) 628–641.  doi:10.1016/j.neuroimage.2015.06.007.
URL https://linkinghub.elsevier.com/retrieve/pii/S1053811915004838

  * [40] Y. Wang, F. S. Haghpanah, X. Zhang, K. Santamaria, G. K. Da Costa Aguiar Alves, E. Bruno, N. Aw, A. Maddocks, C. S. Duarte, C. Monk, A. Laine, J. Posner, program collaborators for Environmental influences on Child Health Outcomes, ID-Seg: an infant deep learning-based segmentation framework to improve limbic structure estimates, Brain Informatics 9 (1) (2022) 12.  doi:10.1186/s40708-022-00161-9.
URL
https://braininformatics.springeropen.com/articles/10.1186/s40708-022-00161-9

  * [41] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, Medical Image Computing and Computer Assisted Intervention - MICCAI 2015 9351 (2015) 234–241.  doi:10.1007/978-3-319-24574-4_28.
URL https://link.springer.com/chapter/10.1007%2F978-3-319-24574-4_28

  * [42] M. Jenkinson, C. F. Beckmann, T. E. Behrens, M. W. Woolrich, S. M. Smith, Fsl, NeuroImage 62 (2) (2012) 782 – 790.
  * [43] Y. Zhang, M. Brady, S. Smith, Segmentation of brain MR images through a hidden markov random field model and the expectation-maximization algorithm 20 (1) (2001) 45–57.  doi:10.1109/42.906424.
  * [44] K. Friston, J. Ashburner, S. Kiebel, T. Nichols, W. Penny, Statistical Parametric Mapping: The Analysis of Functional Brain Images, Academic Press, 2007\.
  * [45] B. Fischl, D. H. Salat, E. Busa, M. Albert, M. Dieterich, C. Haselgrove, A. Van Der Kouwe, R. Killiany, D. Kennedy, S. Klaveness, et al., Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain, Neuron 33 (3) (2002) 341–355.
  * [46] C. Wachinger, M. Reuter, T. Klein, Deepnat: deep convolutional neural network for segmenting neuroanatomy, NeuroImage 170 (2018) 434–445.
  * [47] A. G. Roy, S. Conjeti, N. Navab, C. Wachinger, A. D. N. Initiative, et al., Quicknat: A fully convolutional network for quick and accurate segmentation of neuroanatomy, NeuroImage 186 (2019) 713–727.
  * [48] Y. Huo, Z. Xu, Y. Xiong, K. Aboud, P. Parvathaneni, S. Bao, C. Bermudez, S. M. Resnick, L. E. Cutting, B. A. Landman, 3d whole brain segmentation using spatially localized atlas network tiles, NeuroImage 194 (2019) 105–119.
  * [49] R. Mehta, A. Majumdar, J. Sivaswamy, BrainSegNet: a convolutional neural network architecture for automated segmentation of human brain structures, Journal of Medical Imaging 4 (2) (2017) 1 – 11.
  * [50] H. Chen, Q. Dou, L. Yu, J. Qin, P.-A. Heng, Voxresnet: Deep voxelwise residual networks for brain segmentation from 3d mr images, NeuroImage 170 (2018) 446–455.
  * [51] L. Sun, W. Ma, X. Ding, Y. Huang, D. Liang, J. Paisley, A 3d spatially-weighted network for segmentation of brain tissue from mri, IEEE Transactions on Medical Imaging (2019) 1–1.
  * [52] R. Ito, K. Nakae, J. Hata, H. Okano, S. Ishii, Semi-supervised deep learning of brain tissue segmentation, Neural Networks 116 (2019) 25 – 34.
  * [53] P. McClure, N. Rho, J. A. Lee, J. R. Kaczmarzyk, C. Y. Zheng, S. S. Ghosh, D. M. Nielson, A. G. Thomas, P. Bandettini, F. Pereira, Knowing what you know in brain segmentation using bayesian deep neural networks, Frontiers in Neuroinformatics 13 (2019) 67.  doi:10.3389/fninf.2019.00067.
URL https://www.frontiersin.org/article/10.3389/fninf.2019.00067

  * [54] P. Coupé, B. Mansencal, M. Clément, R. Giraud, B. Denis de Senneville, V.-T. Ta, V. Lepetit, J. V. Manjon, Assemblynet: A large ensemble of cnns for 3d whole brain mri segmentation, NeuroImage 219 (2020) 117026.  doi:https://doi.org/10.1016/j.neuroimage.2020.117026.
URL https://www.sciencedirect.com/science/article/pii/S1053811920305127

  * [55] B. Billot, E. Robinson, A. V. Dalca, J. E. Iglesias, Partial volume segmentation of brain MRI scans of any resolution and contrast, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Springer International Publishing, 2020, pp. 177–187.  doi:10.1007/978-3-030-59728-3_18.
URL https://doi.org/10.1007%2F978-3-030-59728-3_18

  * [56] J. E. Iglesias, B. Billot, Y. Balbastre, A. Tabari, J. Conklin, R. Gilberto González, D. C. Alexander, P. Golland, B. L. Edlow, B. Fischl, Joint super-resolution and synthesis of 1 mm isotropic mp-rage volumes from clinical mri exams with scans of different orientation, resolution and contrast, NeuroImage 237 (2021) 118206.  doi:https://doi.org/10.1016/j.neuroimage.2021.118206.
  * [57] L. Henschel, S. Conjeti, S. Estrada, K. Diers, B. Fischl, M. Reuter, FastSurfer - a fast and accurate deep learning based neuroimaging pipeline, NeuroImage 219 (2020) 117012.  doi:10.1016/j.neuroimage.2020.117012.
  * [58] R. J. Beare, J. Chen, C. E. Kelly, D. Alexopoulos, C. D. Smyser, C. E. Rogers, W. Y. Loh, L. G. Matthews, J. L. Y. Cheong, A. J. Spittle, P. J. Anderson, L. W. Doyle, T. E. Inder, M. L. Seal, D. K. Thompson, Neonatal Brain Tissue Classification with Morphological Adaptation and Unified Segmentation, Frontiers in Neuroinformatics 10 (Mar. 2016).  doi:10.3389/fninf.2016.00012.
URL http://journal.frontiersin.org/Article/10.3389/fninf.2016.00012/abstract

  * [59] R. S. Desikan, F. Ségonne, B. Fischl, B. T. Quinn, B. C. Dickerson, D. Blacker, R. L. Buckner, A. M. Dale, R. P. Maguire, B. T. Hyman, M. S. Albert, R. J. Killiany, An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest, NeuroImage 31 (3) (2006) 968 – 980.
  * [60] B. Fischl, Freesurfer, Neuroimage 62 (2) (2012) 774–781.
  * [61] K. de Macedo Rodrigues, E. Ben-Avi, D. D. Sliva, M.-s. Choe, M. Drottar, R. Wang, B. Fischl, P. E. Grant, L. Zöllei, A freesurfer-compliant consistent manual segmentation of infant brains spanning the 0–2 year age range, Frontiers in Human Neuroscience 9 (2015).  doi:10.3389/fnhum.2015.00021.
URL https://www.frontiersin.org/articles/10.3389/fnhum.2015.00021

  * [62] E. J. Hughes, T. Winchman, F. Padormo, R. Teixeira, J. Wurie, M. Sharma, M. Fox, J. Hutter, L. Cordero-Grande, A. N. Price, J. Allsop, J. Bueno-Conde, N. Tusor, T. Arichi, A. D. Edwards, M. A. Rutherford, S. J. Counsell, J. V. Hajnal, A dedicated neonatal brain imaging system: A Dedicated Neonatal Brain Imaging System, Magnetic Resonance in Medicine 78 (2) (2017) 794–804.  doi:10.1002/mrm.26462.
URL https://onlinelibrary.wiley.com/doi/10.1002/mrm.26462

  * [63] M. Rajchl, M. C. H. Lee, O. Oktay, K. Kamnitsas, J. Passerat-Palmbach, W. Bai, M. Damodaram, M. A. Rutherford, J. V. Hajnal, B. Kainz, D. Rueckert, Deepcut: Object segmentation from bounding box annotations using convolutional neural networks, IEEE Transactions on Medical Imaging 36 (2) (2017) 674–683.  doi:10.1109/TMI.2016.2621185.
  * [64] D. D. Shen, S. L. Bao, Y. Wang, Y. C. Chen, Y. C. Zhang, X. C. Li, Y. C. Ding, Z. Z. Jia, An automatic and accurate deep learning-based neuroimaging pipeline for the neonatal brain, Pediatric Radiology (Mar. 2023).  doi:10.1007/s00247-023-05620-x.
  * [65] W. Ding, M. Abdel-Basset, H. Hawash, W. Pedrycz, Multimodal infant brain segmentation by fuzzy-informed deep learning, IEEE Transactions on Fuzzy Systems 30 (4) (2022) 1088–1101.  doi:10.1109/TFUZZ.2021.3052461.
  * [66] S. Pasban, S. Mohamadzadeh, J. Zeraatkar-Moghaddam, A. K. Shafiei, Infant brain segmentation based on a combination of vgg-16 and u-net deep neural networks, IET Image Processing 14 (17) (2020) 4756–4765.  arXiv:https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/iet-ipr.2020.0469, doi:https://doi.org/10.1049/iet-ipr.2020.0469.
URL https://ietresearch.onlinelibrary.wiley.com/doi/abs/10.1049/iet-
ipr.2020.0469

  * [67] L. Wang, D. Nie, G. Li, E. Puybareau, J. Dolz, Q. Zhang, F. Wang, J. Xia, Z. Wu, J.-W. Chen, K.-H. Thung, T. D. Bui, J. Shin, G. Zeng, G. Zheng, V. S. Fonov, A. Doyle, Y. Xu, P. Moeskops, J. P. W. Pluim, C. Desrosiers, I. B. Ayed, G. Sanroma, O. M. Benkarim, A. Casamitjana, V. Vilaplana, W. Lin, G. Li, D. Shen, Benchmark on Automatic Six-Month-Old Infant Brain Segmentation Algorithms: The iSeg-2017 Challenge, IEEE Transactions on Medical Imaging 38 (9) (2019) 2219–2230.  doi:10.1109/TMI.2019.2901712.
URL https://ieeexplore.ieee.org/document/8654000/

  * [68] Y. Sun, K. Gao, Z. Wu, G. Li, X. Zong, Z. Lei, Y. Wei, J. Ma, X. Yang, X. Feng, L. Zhao, T. Le Phan, J. Shin, T. Zhong, Y. Zhang, L. Yu, C. Li, R. Basnet, M. O. Ahmad, M. N. S. Swamy, W. Ma, Q. Dou, T. D. Bui, C. B. Noguera, B. Landman, I. H. Gotlib, K. L. Humphreys, S. Shultz, L. Li, S. Niu, W. Lin, V. Jewells, D. Shen, G. Li, L. Wang, Multi-Site Infant Brain Segmentation Algorithms: The iSeg-2019 Challenge, IEEE Transactions on Medical Imaging 40 (5) (2021) 1363–1376.  doi:10.1109/TMI.2021.3055428.
URL https://ieeexplore.ieee.org/document/9339962/

  * [69] L. Wang, F. Shi, Y. Gao, G. Li, W. Lin, D. Shen, Isointense infant brain segmentation by stacked kernel canonical correlation analysis, in: G. Wu, P. Coupé, Y. Zhan, B. Munsell, D. Rueckert (Eds.), Patch-Based Techniques in Medical Imaging, Springer International Publishing, Cham, 2015, pp. 28–36.
  * [70] N. I. Weisenfeld, S. K. Warfield, Automatic segmentation of newborn brain mri, Neuroimage 47 (2) (2009) 564–572.
  * [71] L. Gui, R. Lisowski, T. Faundez, P. S. Hüppi, F. Lazeyras, M. Kocher, Morphology-driven automatic segmentation of mr images of the neonatal brain, Medical image analysis 16 (8) (2012) 1565–1579.
  * [72] Z. Shang, M. A. Turja, E. Feczko, A. Houghton, A. Rueter, L. A. Moore, K. Snider, T. Hendrickson, P. Reiners, S. Stoyell, O. Kardan, M. Rosenberg, J. T. Elison, D. A. Fair, M. A. Styner, Learning Strategies for Contrast-agnostic Segmentation via SynthSeg for Infant MRI data, in: Proceedings of Machine Learning Research, Vol. 172 of Proceedings of Machine Learning Research, Zürich, Switzerland, 2022, pp. 1075–1084.
  * [73] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, K. H. Maier-Hein, nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation, Nature Methods 18 (2) (2021) 203–211.  doi:10.1038/s41592-020-01008-z.
URL http://www.nature.com/articles/s41592-020-01008-z

  * [74] G. Bortsova, F. Dubost, L. Hogeweg, I. Katramados, M. de Bruijne, Semi-supervised medical image segmentation via learning consistency under transformations, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019.
URL https://api.semanticscholar.org/CorpusID:204539180

  * [75] M. Jaderberg, K. Simonyan, A. Zisserman, k. kavukcuoglu, Spatial transformer networks, in: C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, R. Garnett (Eds.), Advances in Neural Information Processing Systems, Vol. 28, Curran Associates, Inc., 2015.
URL
https://proceedings.neurips.cc/paper/2015/file/33ceb07bf4eeb3da587e268d663aba1a-Paper.pdf

  * [76] L. Cordero-Grande, E. J. Hughes, J. Hutter, A. N. Price, J. V. Hajnal, Three-dimensional motion corrected sensitivity encoding reconstruction for multi-shot multi-slice MRI: Application to neonatal brain imaging: Aligned Multi-Shot Multi-Slice MRI, Magnetic Resonance in Medicine 79 (3) (2018) 1365–1376.  doi:10.1002/mrm.26796.
URL https://onlinelibrary.wiley.com/doi/10.1002/mrm.26796

  * [77] M. Kuklisova-Murgasova, G. Quaghebeur, M. A. Rutherford, J. V. Hajnal, J. A. Schnabel, Reconstruction of fetal brain mri with intensity matching and complete outlier removal, Medical Image Analysis 16 (8) (2012) 1550–1564.  doi:https://doi.org/10.1016/j.media.2012.07.004.
URL https://www.sciencedirect.com/science/article/pii/S1361841512000965

  * [78] B. Alexander, W. Y. Loh, L. G. Matthews, A. L. Murray, C. Adamson, R. Beare, J. Chen, C. E. Kelly, P. J. Anderson, L. W. Doyle, A. J. Spittle, J. L. Y. Cheong, M. L. Seal, D. K. Thompson, Desikan-killiany-tourville atlas compatible version of m-crib neonatal parcellated whole brain atlas: The m-crib 2.0, Frontiers in Neuroscience 13 (2019).  doi:10.3389/fnins.2019.00034.
URL https://www.frontiersin.org/articles/10.3389/fnins.2019.00034

  * [79] A. Klein, J. Tourville, 101 labeled brain images and a consistent human cortical labeling protocol, Frontiers in Neuroscience 6 (2012) 171.
  * [80] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, Y. Bengio, Maxout networks, in: Proceedings of the 30th International Conference on International Conference on Machine Learning-Volume 28, JMLR. org, 2013, pp. III–1319.
  * [81] S. Jégou, M. Drozdzal, D. Vazquez, A. Romero, Y. Bengio, The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation, in: Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, IEEE, 2017, pp. 1175–1183.
  * [82] A. G. Roy, S. Conjeti, D. Sheet, A. Katouzian, N. Navab, C. Wachinger, Error corrective boosting for learning fully convolutional networks with limited data, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2017, pp. 231–239.
  * [83] F. Pérez-García, R. Sparks, S. Ourselin, TorchIO: a python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning, Computer Methods and Programs in Biomedicine (2021) 106236doi:10.1016/j.cmpb.2021.106236.
  * [84] L. R. Dice, Measures of the Amount of Ecologic Association Between Species, Ecology 26 (3) (1945) 297–302.  doi:10.2307/1932409.
URL http://doi.wiley.com/10.2307/1932409

  * [85] T. J. Sørensen, A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegetation on Danish commons 5 (1948) 1–34.
  * [86] F. Wilcoxon, Individual comparisons by ranking methods, Biometrics Bulletin 1 (6) (1945) 80–83.
URL http://www.jstor.org/stable/3001968

  * [87] Y. Benjamini, Y. Hochberg, Controlling the false discovery rate: A practical and powerful approach to multiple testing, Journal of the Royal Statistical Society: Series B (Methodological) 57 (1) (1995) 289–300.  arXiv:https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1995.tb02031.x, doi:https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
URL
https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.2517-6161.1995.tb02031.x

  * [88] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer, Automatic differentiation in pytorch, in: NIPS Workshop Autodiff, 2017.
  * [89] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: International Conference on Learning Representations, 2019\.
URL https://openreview.net/forum?id=Bkg6RiCqY7

  * [90] I. Loshchilov, F. Hutter, Sgdr: Stochastic gradient descent with warm restarts, in: International Conference on Learning Representations, 2017.
URL https://openreview.net/forum?id=Skq89Scxx

  * [91] S. Roy, D. Kügler, M. Reuter, Are 2.5d approaches superior to 3d deep networks in whole brain segmentation?, in: Medical Imaging with Deep Learning, 2022.
URL https://openreview.net/forum?id=Ob62JPB_CDF

  * [92] D. J. Miller, T. Duka, C. D. Stimpson, S. J. Schapiro, W. B. Baze, M. J. McArthur, A. J. Fobbs, A. M. M. Sousa, N. Šestan, D. E. Wildman, L. Lipovich, C. W. Kuzawa, P. R. Hof, C. C. Sherwood, Prolonged myelination in human neocortical evolution, Proceedings of the National Academy of Sciences 109 (41) (2012) 16480–16485.  arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1117943109, doi:10.1073/pnas.1117943109.
URL https://www.pnas.org/doi/abs/10.1073/pnas.1117943109

  * [93] C. Sun, A. Shrivastava, S. Singh, A. Gupta, Revisiting unreasonable effectiveness of data in deep learning era, in: 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 843–852.  doi:10.1109/ICCV.2017.97.
  * [94] M. Reuter, N. J. Schmansky, H. D. Rosas, B. Fischl, Within-subject template estimation for unbiased longitudinal image analysis, NeuroImage 61 (4) (2012) 1402–1418.  doi:https://doi.org/10.1016/j.neuroimage.2012.02.084.
URL https://www.sciencedirect.com/science/article/pii/S1053811912002765

  * [95] T. L. Jernigan, T. T. Brown, D. J. Hagler, N. Akshoomoff, H. Bartsch, E. Newman, W. K. Thompson, C. S. Bloss, S. S. Murray, N. Schork, D. N. Kennedy, J. M. Kuperman, C. McCabe, Y. Chung, O. Libiger, M. Maddox, B. Casey, L. Chang, T. M. Ernst, J. A. Frazier, J. R. Gruen, E. R. Sowell, T. Kenet, W. E. Kaufmann, S. Mostofsky, D. G. Amaral, A. M. Dale, The pediatric imaging, neurocognition, and genetics (ping) data repository, NeuroImage 124 (2016) 1149–1154.  doi:https://doi.org/10.1016/j.neuroimage.2015.04.057.
  * [96] N. R. Karcher, D. M. Barch, The ABCD study: understanding the development of risk for mental and physical health outcomes, Neuropsychopharmacology 46 (1) (2021) 131–142.  doi:10.1038/s41386-020-0736-6.
URL https://www.nature.com/articles/s41386-020-0736-6

## Appendix

###  A.1 Ablation on T1w

Figure 12: Comparison of approaches for external and internal spatial
augmentation on T1w images: Our VINNA method – with the 4-DOF transform module
– (third box in group) outperforms both state-of-the-art approaches with
external Augmentation (CNN* + exA and VINN + exA, first and second box) in
Dice Similarity Coefficient (DSC, upper row) and average surface distance
(ASD, lower row). This performance advantage is significant (corrected
$p<0.05$) and consistent across resolutions and structures. Combining VINNA
with external augmentation (VINNA + exA, last box) reduces performance. exA:
external Augmentation.

Figure 12 shows DSC (top) and ASD (bottom) for the ablative evaluation
external augmentation (exA) versus the internal 4-DOF transform modul (VINNA)
on the T1w. CNN* + exA (first box) reaches the weakest results, followed by
VINN + exA (second box) and VINNA (third box). The performance improves the
most on the subcortical structures (2.44 % ASD and 0.4 % DSC when switching
from CNN* to VINN, 5.91 % ASD and 0.85 % ASD when switching to VINNA).
Performance improvements with the 4-DOF transform module in VINNA, is
significant (corrected $p<0.05$) and results in an average final DSC of 90.95,
90.03 and 90.36 and ASD of $0.295\text{\,}\mathrm{mm}$,
$0.2\text{\,}\mathrm{mm}$, $0.2\text{\,}\mathrm{mm}$ for the
$0.5\text{\,}\mathrm{mm}$. $0.8\text{\,}\mathrm{mm}$, and
$1.0\text{\,}\mathrm{mm}$ resolution, respectively. Re-addition of exA to
VINNA (fourth box) reduces performance to a similar level as CNN* + exA.

Overall, all models perform better on T2w (Figure 5) rather than T1w MRI
(Figure 12) across all structures. When segmenting a T2w MRI of the same
individual, peformance improves for all networks, on average, by 5.7, 3.63 and
5.09 DSC points for the cortex, WM, and subcortical structures, respectively.
The ASD difference between T1w and T2w applications nearly doubles for the
highest resolution ($0.5\text{\,}\mathrm{mm}$) with an average of
$0.18\text{\,}\mathrm{mm}$ compared to both lower resolutions (ASD of
$0.9\text{\,}\mathrm{mm}$ for $1.0\text{\,}\mathrm{a}$nd
$0.8\text{\,}\mathrm{mm}$) for the three structure averages. The DSC point
improvement for the T2w images is similar across all resolutions. The trend is
consistent for both networks (CNN* and VINN) and augmentation schemes
(external versus internal).

###  A.2 Dataset Summary

Table A: Summary of datasets used for training, validation, and testing. Table lists the dataset, number of participants (#P), range of post-gestational age at scan in weeks and isotropic resolution in millimeter (Res [mm]) used in the paper.  Usage | Dataset | #P |  | Age
---
[weeks]
Res [mm]
Training | dHCP | 320 | 26-44 | 0.5, 0.8, 1.0
Validation |  [HTML]EFEFEFdHCP |  [HTML]EFEFEF90 |  [HTML]EFEFEF27-41 |  [HTML]EFEFEF0.5, 0.8, 1.0
| dHCP | 170 | 30-44 | 0.5, 0.8, 1.0
Testing |  [HTML]EFEFEFM-CRIB |  [HTML]EFEFEF10 |  [HTML]EFEFEF40-43 |  [HTML]EFEFEF0.63

###  A.3 Labels

Table B: Subcortical segmentations of the dHCP-ALBERTs atlas and matching
identifier (ID) equaling the number returned by VINNA for the structure. Left
and Right indicate the respective hemisphere. Four structures are not
lateralized (same ID for Left and Right).

ID |
---|---
Left | Right | Subcortical Structure
[HTML]EFEFEF [HTML]EFEFEF0 | 0 | Background
1 | 2 | Hippocampus
[HTML]EFEFEF [HTML]EFEFEF3 | 4 | Amygdala
17 | 18 | Cerebellum
[HTML]EFEFEF [HTML]EFEFEF19 | 19 | Brainstem
41 | 40 | Caudate_nucleus
[HTML]EFEFEF [HTML]EFEFEF43 | 42 | Thalamus_high_intensity_part_in_T2
45 | 44 | Subthalamic_nucleus
[HTML]EFEFEF [HTML]EFEFEF47 | 46 | Lentiform_Nucleus
48 | 48 | Corpus_Callosum
[HTML]EFEFEF [HTML]EFEFEF49 | 50 | Lateral-Ventricle
83 | 83 | CSF
[HTML]EFEFEF [HTML]EFEFEF84 | 84 | Extra_cranial_background
88 | 85 | IntraCranialBackground
[HTML]EFEFEF [HTML]EFEFEF87 | 86 | Thalamus_low_intensity_part_in_T2

Table C: Segmentation parcels of the dHCP-ALBERTs atlas for gray matter (GM)
and white matter (WM). Left and Right indicate the lateralization. The
identifier (ID) equals the number returned by VINNA for the structure.

GM ID | WM ID |
---|---|---
Left | Right | Left | Right | Structure
[HTML]EFEFEF [HTML]EFEFEF5 | 6 |  [HTML]EFEFEF52 | 51 | Anterior_temporal_lobe_medial_part
7 | 8 | 54 | 53 | Anterior_temporal_lobe_lateral_part
[HTML]EFEFEF [HTML]EFEFEF9 | 10 |  [HTML]EFEFEF56 | 55 | Gyri_parahippocampalis_et_ambiens_anterior_part
11 | 12 | 58 | 57 | Superior_temporal_gyrus_middle_part
[HTML]EFEFEF [HTML]EFEFEF13 | 14 |  [HTML]EFEFEF60 | 59 | Medial_and_inferior_temporal_gyri_anterior_part
15 | 16 | 62 | 61 | Lateral_occipitotemporal_gyrus_gyrus_fusiformis_anterior_part
[HTML]EFEFEF [HTML]EFEFEF21 | 20 |  [HTML]EFEFEF64 | 63 | Insula
23 | 22 | 66 | 65 | Occipital_lobe
[HTML]EFEFEF [HTML]EFEFEF25 | 24 |  [HTML]EFEFEF68 | 67 | Gyri_parahippocampalis_et_ambiens_posterior_part
27 | 26 | 70 | 69 | Lateral_occipitotemporal_gyrus_gyrus_fusiformis_posterior_part
[HTML]EFEFEF [HTML]EFEFEF29 | 28 |  [HTML]EFEFEF72 | 71 | Medial_and_inferior_temporal_gyri_posterior_part
31 | 30 | 74 | 73 | Superior_temporal_gyrus_posterior_part
[HTML]EFEFEF [HTML]EFEFEF33 | 32 |  [HTML]EFEFEF76 | 75 | Cingulate_gyrus_anterior_part
35 | 34 | 78 | 77 | Cingulate_gyrus_posterior_part
[HTML]EFEFEF [HTML]EFEFEF37 | 36 |  [HTML]EFEFEF80 | 79 | Frontal_lobe
39 | 38 | 82 | 81 | Parietal_lobe