# Anatomy-aware and acquisition-agnostic joint registration
with SynthMorph

Malte Hoffmanna–d,*, Andrew Hoopesa–b,e,
Douglas N. Grevea–c, Bruce Fischla–e,1, Adrian V. Dalcaa–c,e,1

( a Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA, USA
b Department of Radiology, Massachusetts General Hospital, Boston, MA, USA
c Department of Radiology, Harvard Medical School, Boston, MA, USA
d Harvard-MIT Health Sciences and Technology, Massachusetts Institute of
Technology, Cambridge, MA, USA
e Computer Science & Artificial Intelligence Laboratory, Massachusetts
Institute of Technology, Cambridge, MA, USA
∗ Corresponding author<EMAIL_ADDRESS>1 These authors contributed
equally.
)

Abstract Affine image registration is a cornerstone of medical-image analysis.
While classical algorithms can achieve excellent accuracy, they solve a time-
consuming optimization for every image pair. Deep-learning (DL) methods learn
a function that maps an image pair to an output transform. Evaluating the
function is fast, but capturing large transforms can be challenging, and
networks tend to struggle if a test-image characteristic shifts from the
training domain, such as resolution. Most affine methods are agnostic to the
anatomy the user wishes to align, meaning the registration will be inaccurate
if algorithms consider all structures in the image. We address these
shortcomings with SynthMorph, a fast, robust, and easy-to-use DL tool for
joint affine-deformable registration of any brain image without preprocessing,
right off the MRI scanner. First, we leverage a strategy to train networks
with wildly varying images synthesized from label maps, yielding robust
performance across acquisition specifics unseen at training. Second, we
optimize the spatial overlap of select anatomical labels. This enables
networks to distinguish anatomy of interest from irrelevant structures,
removing the need for preprocessing that excludes content which would impinge
on anatomy-specific registration. Third, we combine the affine model with a
deformable hypernetwork that lets users choose the optimal deformation-field
regularity for their specific data, at registration time, in a fraction of the
time required by classical methods. This framework is applicable to learning
anatomy-aware, acquisition-agnostic registration of any anatomy with any
architecture, as long as label maps are available for training. We rigorously
analyze how competing architectures learn affine transforms and compare state-
of-the-art registration tools across an extremely diverse set of neuroimaging
data, aiming to truly capture the behavior of methods in the real world.
SynthMorph demonstrates consistent and improved accuracy and is available at
https://w3id.org/synthmorph, as a single complete end-to-end solution for
registration of brain MRI.

Keywords Affine registration, deformable registration, deep learning,
hypernetwork, domain shift, neuroimaging.

##  1 Introduction

Image registration is an essential component of medical image processing and
analysis that estimates a mapping from the space of the anatomy in one image
to the space of another image (Cox, 1996; Fischl et al., 2002, 2004; Jenkinson
et al., 2012; Tustison et al., 2013). Such transforms generally include an
affine component accounting for global orientation such as different head
positions, which are typically not of clinical interest. Transforms often
include a deformable component that may represent anatomically meaningful
differences in geometry (Hajnal and Hill, 2001). Many techniques analyze these
further, for example voxel-based morphometry (Ashburner and Friston, 2000;
Whitwell, 2009).

Iterative registration has been extensively studied, and the available methods
can achieve excellent accuracy both within and across magnetic resonance
imaging (MRI) contrasts (Friston et al., 1995; Jiang et al., 1995; Cox and
Jesmanowicz, 1999; Rueckert et al., 1999; Rohr et al., 2001; Ashburner, 2007;
Lorenzi et al., 2013). Approaches differ in how they measure image similarity
and the strategy chosen to optimize it, but the fundamental algorithm is the
same: fit a set of parameters modeling the spatial transformation between an
image pair by iteratively minimizing a dissimilarity metric. While classical
deformable registration can take tens of minutes to several hours, affine
registration optimizes only a handful of parameters and is generally faster
(Jenkinson and Smith, 2001; Reuter et al., 2010; Modat et al., 2014; Hoffmann
et al., 2015). For these reasons, classical affine methods are still widely
used both within analysis pipelines and also for more specialized applications
such as correcting head motion during image acquisition (Thesen et al., 2000;
Tisdall et al., 2012; Gallichan et al., 2016). However, these approaches solve
an optimization problem for every new image pair, which is inefficient:
depending on the algorithm, affine registration of higher-resolution
structural MRI, for example, can easily take 5-10 minutes (Table 3). Further,
iterative pipelines can be laborious to use. The user typically has to tailor
the optimization strategy and choose a similarity metric appropriate for the
image appearance (Pustina and Cook, 2017). Often, images require
preprocessing, including intensity normalization or removal of structures that
the registration should exclude. These shortcomings have motivated work on
deep-learning (DL) based registration.

Recent advances in DL have enabled registration with unprecedented efficiency
and accuracy (Dalca et al., 2018; Balakrishnan et al., 2019; Li and Fan, 2017;
Rohé et al., 2017; Krebs et al., 2017; Sokooti et al., 2017; Yang et al.,
2016, 2017; Eppenhof and Pluim, 2019). In contrast to classical approaches, DL
models learn a function that maps an input registration pair to an output
transform. Whle evaluating this function on a new pair of images is fast, most
existing DL methods focus on the deformable component. Affine registration of
the input images is often assumed (Balakrishnan et al., 2019; de Vos et al.,
2017) or incorporated ad hoc, and thus given less attention than deformable
registration (Hu et al., 2018; De Vos et al., 2019; Zhao et al., 2019b, d; Mok
and Chung, 2022). Although state-of-the-art deformable algorithms are capable
of compensating for sub-optimal affine alignment to some extent, they cannot
always fully recover the lost accuracy, as the experiment of Section 4.6 will
show. Further, any inaccuracy in the affine transform will make it harder to
interpret the deformable component (Bookstein, 2001; Ou et al., 2014), which
will now include an undesired affine residual.

The learning-based models encompassing both affine and deformable components
usually do not consider network generalization to modality variation (De Vos
et al., 2019; Zhao et al., 2019b, d; Shen et al., 2019; Zhu et al., 2021).
That is, networks trained on one type of data, such as T1-weighted (T1w) MRI,
tend to inaccurately register other types of data, such as T2-weighted (T2w)
scans. Even for similar MRI contrast, the domain shift caused by different
noise or smoothness levels alone has the potential to reduce accuracy at test
time. In contrast, learning frameworks capitalizing on generalization
techniques and domain adaptation often do not incorporate the fundamental
affine transform (Chen et al., 2017; Iglesias et al., 2013; Tanner et al.,
2018; Qin et al., 2019).

A separate challenge for affine registration consists in accurately aligning
specific anatomy of interest in the image while ignoring irrelevant content.
Any undesired structure that moves independently or deforms non-linearly will
reduce the accuracy of the anatomy-specific transform unless an algorithm has
the ability to ignore it. For example, neck and tongue tissue can confuse
rigid brain registration when they deform non-rigidly (Fein et al., 2006;
Fischmeister et al., 2013; Andrade et al., 2018; Hoffmann et al., 2020).
Identifying a suitable architecture for affine registration and formulating
the problem in a differentiable manner will enable embedding and jointly
learning affine registration with other tasks, for example creating
conditional atlases representing a subject population, from non-aligned input
images (Dalca et al., 2019a; Ding and Niethammer, 2022; Sinclair et al.,
2022).

Figure 1:  Representative examples of anatomy-aware SynthMorph affine 3D
registration showing the moving brain transformed onto the fixed brain (red
overlay). Trained with highly variable synthetic data, SynthMorph generalizes
across a diverse array of real-world contrasts, resolutions, and subject
populations without any preprocessing.

###  1.1 Contribution

In this work we present a single, easy-to-use DL tool for end-to-end affine
and deformable brain registration for images right off the MRI scanner,
without preprocessing (Figure 1). The tool performs robustly across MRI
contrasts, intensity scales, and resolutions. We address the domain dependency
and anatomical non-specificity of affine registration: while invariance to
acquisition specifics will enable networks to generalize to new image types
without retraining, our anatomy-specific training strategy alleviates the need
for pre-processing segmentation steps that remove image content which would
distract a majority of registration methods—as Section 4.5 will show for the
example of skull-stripping (Smith, 2002; Iglesias et al., 2011; Eskildsen et
al., 2012; Salehi et al., 2017; Hoopes et al., 2022b).

Our work builds on ideas from DL-based registration, affine registration, and
a recent synthesis-based training strategy that promotes data independence by
exposing networks to arbitrary image contrasts (Billot et al., 2020; Hoffmann
et al., 2021a; Hoopes et al., 2022b).

First, we rigorously analyze three fundamental network architectures in
Section 4.4, to provide insight into how DL models learn and best represent
the affine component. Second, we select an optimized architecture and train
the network with synthetic data only, making it robust across a landscape of
acquired image types without exposing it to any real images during training.
Third, we combine the affine model with a deformable hypernetwork to create an
end-to-end registration tool, enabling users to choose a regularization
strength that is optimal for their own data without retraining and in a
fraction of the time required by classical methods. Fourth, we test the
resulting model across a very diverse set of images, aiming to truly capture
the diversity of real-world data. We compare its performance against popular
affine and deformable toolboxes in Sections 4.5 and 4.6, respectively, to
thoroughly assess the registration accuracy users can achieve with off-the-
shelf implementations on images unseen at training. We freely distribute our
code and stand-alone tool, SynthMorph, at https://w3id.org/synthmorph and
within the upcoming FreeSurfer release (Fischl, 2012).

Figure 2:  Training strategy for affine registration. At each iteration, we
augment a pair of moving and fixed label maps $\\{s_{m},s_{f}\\}$ and
synthesize images $\\{m,f\\}$ from them. The network $h_{\theta}$ predicts an
affine transform $T$ that we compute the moved label map $s_{m}\circ T$ from.
Loss $\mathcal{L}_{o}$ recodes the labels in $\\{s_{m},s_{f}\\}$ to optimize
the overlap of select anatomy of interest only, such as WM, GM, and CSF. We
emphasize the wide applicability of this strategy to learning anatomy-aware
and acquisition-agnostic registration of any anatomy and with any network
architecture, as long as label maps are available for training.

##  2 Related work

There is substantial work on medical image registration. While this section
provides an overview of successful and widely adopted strategies, more in-
depth review articles are available (Oliveira and Tavares, 2014; Wyawahare et
al., 2009; Fu et al., 2020).

###  2.1 Classical registration

Classical registration is driven by an objective function, which measures
similarity in appearance between the moving and the fixed image. A simple and
effective choice for images of the same contrast is the mean squared error
(MSE). Normalized cross-correlation (NCC) is also widely used, because it
provides excellent accuracy independent of the intensity scale (Avants et al.,
2008). Registration of images across contrasts or modalities generally employs
objective functions such as normalized mutual information (NMI) (Wells III et
al., 1996; Maes et al., 1997) or the correlation ratio (Roche et al., 1998),
as these do not assume similar appearance of the input images. Another class
of classical registration methods uses metrics based on patch similarity
(Glocker et al., 2008; Ou et al., 2011; Glocker et al., 2011), which can
outperform simpler metrics across modalities (Hoffmann et al., 2021a).

To improve computational efficiency and avoid local minima, many classical
techniques perform multi-resolution searches (Nestares and Heeger, 2000;
Hellier et al., 2001). First, this strategy coarsely aligns smoothed
downsampled versions of the input images. This initial solution is
subsequently refined at higher resolutions until the original images align
precisely (Avants et al., 2011; Reuter et al., 2010; Modat et al., 2014).
Additionally, an initial grid search over a set of rotation parameters can
help constrain this scale-space approach to a neighborhood around the global
optimum (Jenkinson and Smith, 2001; Jenkinson et al., 2012).

Instead of optimizing image similarity, another registration paradigm detects
landmarks and matches these across the images (Myronenko and Song, 2010).
Early work relied on user assistance to identify fiducials (Besl and McKay,
1992; Meyer et al., 1995). More recent computer-vision approaches
automatically extract features (Toews and Wells III, 2013; Machado et al.,
2018), for example from entropy (Wachinger and Navab, 2010, 2012) or
difference-of-Gaussians images (Lowe, 2004; Rister et al., 2017; Wachinger et
al., 2018). The performance of this strategy depends on the invariance of
landmarks across viewpoints and intensity scales (Matas et al., 2004).

###  2.2 Deep-learning registration

Analogous to classical registration, unsupervised deformable DL methods fit
the parameters of a deep neural network by optimizing a loss function that
measures image similarity—but across many image pairs (Balakrishnan et al.,
2019; Dalca et al., 2019b; Guo, 2019; Krebs et al., 2019; De Vos et al., 2019;
Hoffmann et al., 2021a). In contrast, supervised DL strategies (Eppenhof and
Pluim, 2019; Krebs et al., 2017; Rohé et al., 2017; Sokooti et al., 2017; Yang
et al., 2016, 2017) train a network to reproduce ground-truth transforms, for
example obtained with classical tools, and tend to underperform relative to
their unsupervised counterparts (Hoffmann et al., 2021a; Young et al., 2022),
although warping features at the end of each U-Net (Ronneberger et al., 2015)
level can close the performance gap (Young et al., 2022).

####  2.2.1 Affine deep-learning registration

Similar to the deformable case, affine registration strategies can be
supervised or unsupervised but require a different network architecture. A
straightforward option combines a convolutional encoder with a fully connected
(FC) layer to predict the parameters of an affine transform in one shot (Zhao
et al., 2019b, d; Zhu et al., 2021; Shen et al., 2019). A series of
convolutional blocks successively halve the image dimension, such that the
output of the final convolution has substantially fewer voxels than the input
images. This facilitates the use of the FC layer with the desired number of
output units, preventing the number of network parameters from becoming
intractably large. Networks typically concatenate the input images before
passing them through the encoder. To benefit from weight sharing, twin
networks pass the fixed and moving images separately though the same encoder
and connect their outputs at the end (De Vos et al., 2019; Chen et al., 2021).

As affine transforms have a global effect on the image, some architectures
replace the locally operating convolutional layers with vision transformers
(Dosovitskiy et al., 2020; Mok and Chung, 2022). These models subdivide their
inputs into patch embeddings and pass them through the transformer, before a
multi-layer perceptron (MLP) outputs a transformation matrix. Multiple such
modules in series can successively refine the affine transform if each module
applies its output transform to the moving image before passing it onto the
next stage (Mok and Chung, 2022). Composition of the transforms from each step
produces the final output matrix.

Another affine DL strategy (Moyer et al., 2021; Yu et al., 2022) derives an
affine transform without requiring MLP or FC layers, similar to the classical
feature extraction and matching approach (Section 2.1). This method separately
passes the moving and the fixed image through a convolutional encoder to
detect two corresponding sets of feature maps. Computing the barycenter of
each feature map yields moving and fixed point clouds, and a least-squares
(LS) fit provides a transform aligning them. The approach is robust across
large transforms (Yu et al., 2022), while removing the FC layer alleviates the
dependency of the architecture on a specific image size.

In this work, we will thoroughly test these fundamental DL architectures and
extend them to build an end-to-end solution for joint affine and deformable
registration that is aware of the anatomy of interest.

###  2.3 Robustness and
anatomical specificity

Indiscriminate registration of images as a whole can limit the accurate
alignment of specific substructures, such as the brain in whole-head MRI. One
group of classical methods avoids this problem by down-weighting image regions
that cannot be mapped accurately with the chosen transformation model, for
example using an iteratively re-weighted LS algorithm (Nestares and Heeger,
2000; Gelfand et al., 2005; Reuter et al., 2010; Puglisi and Battiato, 2011;
Modat et al., 2014; Billings et al., 2015). Few approaches focus on specific
anatomical features, for example by restricting the registration to regions of
an atlas with high prior probability for belonging to a particular tissue
class (Fischl et al., 2002). The affine registration tools most commonly used
in neuroimage analysis (Friston et al., 1995; Cox, 1996; Jenkinson and Smith,
2001; Modat et al., 2014) instead expect—and require—that distracting image
content be removed from the input data as a preprocessing step for optimal
performance (Klein et al., 2009; Smith, 2002; Iglesias et al., 2011; Eskildsen
et al., 2012). Similarly, many DL registration algorithms assume intensity-
normalized and skull-stripped input images (Balakrishnan et al., 2019; Zhao et
al., 2019d; Yu et al., 2022), preventing their applicability to diverse
unprocessed data.

###  2.4 Domain generalizability

The adaptability of neural networks to out-of-distribution data generally
presents a challenge to their deployment (Sun et al., 2016; Wang and Deng,
2018). Mitigation strategies include augmenting the variability of the
training distribution, for example by adding random noise or applying
geometric transforms (Perez and Wang, 2017; Shorten and Khoshgoftaar, 2019;
Chaitanya et al., 2019; Zhao et al., 2019a). Transfer learning adapts a
trained network to a new domain by fine-tuning deeper layers on the target
distribution (Kamnitsas et al., 2017; Zhuang et al., 2020). These methods
require training data from the target domain. By contrast, within medical
imaging, a recent strategy synthesizes wildly variable training images to
promote data independence. The resulting networks generalize beyond dataset
specifics and perform with high accuracy on tasks including segmentation
(Billot et al., 2020), deformable registration (Hoffmann et al., 2021a), and
skull-stripping (Hoopes et al., 2022b). We build on this technology to achieve
end-to-end registration incorporating the affine component.

##  3 Method

###  3.1 Background

####  3.1.1 Learning-based registration

Let $m$ be a moving and $f$ a fixed image in $N$-dimensional ($N$D) space. We
train a deep neural network $h_{\theta}$ with learnable parameters $\theta$ to
predict a global transform $T_{\theta}:\Omega\rightarrow\mathbb{R}^{N}$ that
maps the spatial domain $\Omega$ of $f$ onto $m$. The transform
$T_{\theta}=h_{\theta}(m,f)$ is a matrix

$T_{\theta}=\left(\begin{array}[]{ccc|c}&&&\\\ &A_{\theta}&&v_{\theta}\\\
&&&\\\ \hline\cr 0&\cdots&0&1\\\
\end{array}\right)=\left(\begin{array}[]{cccc}&&&\\\ \lx@intercol\hfil
t_{\theta}\hfil\lx@intercol\\\ &&&\\\ \hline\cr 0&\cdots&0&1\\\
\end{array}\right),$ (1)

where matrix $A_{\theta}\in\mathbb{R}^{N\times N}$ represents rotation,
scaling, and shear, and $v_{\theta}\in\mathbb{R}^{N\times 1}$ is a vector of
translational shifts, such that $t_{\theta}\in\mathbb{R}^{N\times(N+1)}$. We
fit the network weights $\theta$ to training set $\mathcal{D}$ subject to

$\hat{\theta}=\operatorname*{arg\,min}_{\theta}\,\operatorname*{\mathbb{E}}_{(m,f)\,\in\,\mathcal{D}^{2}}\Big{[}\mathcal{L}_{s}\big{(}m\circ
h_{\theta}(m,f),\,f\big{)}\Big{]},$ (2)

where the loss $\mathcal{L}_{s}$ measures the similarity in appearance between
two input images, and $m\circ T_{\theta}$ means $m$ transformed by
$T_{\theta}=h_{\theta}(m,f)$.

####  3.1.2 Synthesis-based training

A recent strategy (Billot et al., 2020; Hoffmann et al., 2021b, a; Hoopes et
al., 2022b; Hoffmann et al., 2023) achieves robustness to preprocessing and
acquisition specifics by training networks exclusively with synthetic images
generated from label maps. From a set of label maps $\\{s_{m},s_{f}\\}$, a
generative model synthesizes corresponding wildly variable images $\\{m,f\\}$
as network inputs. Instead of image similarity, the strategy optimizes spatial
label overlap with a (soft) Dice-based loss $\mathcal{L}_{o}$ (Milletari et
al., 2016), completely independent of image appearance:

$\mathcal{L}_{o}(\theta,s_{m},s_{f})=-\frac{2}{|J|}\sum_{\begin{subarray}{c}j\in
J\\\ x\in\Omega\end{subarray}}\frac{(s_{m}\big{|}_{j}\circ
T_{\theta})(x)\times s_{f}\big{|}_{j}(x)}{(s_{m}\big{|}_{j}\circ
T_{\theta})(x)+s_{f}\big{|}_{j}(x)},$ (3)

where $s\big{|}_{j}$ represents the one-hot encoded label $j\in J$ of label
map $s$ defined at the voxel locations $x\in\Omega$ in the discrete spatial
domain $\Omega$ of $f$.

The generative model requires only a few input label maps to produce a stream
of diverse training images, helping the task network accurately generalize to
real medical images of any contrast at test time, which it can then register
without needing label maps.

###  3.2 Anatomy-aware joint registration

As we build on our recent work on deformable registration, SynthMorph
(Hoffmann et al., 2021a), here we only provide a high-level overview and focus
on what is new for affine and joint affine-deformable registration. Figure 2
illustrates our learning setup for affine registration.

####  3.2.1 Label maps

Every training iteration, we draw a pair of moving and fixed brain
segmentations. We apply random spatial transformations to each of them to
augment the range of head orientations and anatomical variability in the
training set. Specifically, we construct an affine matrix from random
translation, rotation, scaling, and shear as detailed in Appendix A. We
compose the affine transform with a randomly sampled and randomly smoothed
deformation field (Hoffmann et al., 2021a) and apply the composite transform
in a single interpolation step. Finally, we simulate acquisitions with a
partial field of view (FOV) by randomly cropping the label map content,
yielding label maps $\\{s_{m},s_{f}\\}$.

####  3.2.2 Anatomical specificity

Let $K$ be the complete set of labels in $\\{s_{m},s_{f}\\}$. To encourage
networks to register specific anatomy while ignoring irrelevant image content,
we propose to recode $\\{s_{m},s_{f}\\}$ such that the label maps include only
a subset of labels $J\subset K$. For brain-specific registration, $J$ consists
of individual brain structures in the deformable case or larger tissue classes
in the affine case. At training, the loss $\mathcal{L}$ optimizes only the
overlap of $J$, whereas we synthesize images from the complete set of labels
$K$, providing rich image content outside the brain as illustrated in Figure
2.

Figure 3:  Synthetic 3D training data with arbitrary contrast, resolution, and
artifact level, generated from brain label maps. The image characteristics
exceed the realistic range to promote network generalization across
acquisition protocols. All examples are based on the same label map. In
practice, we use several different subjects.  Figure 4:  Training strategy for
joint registration. As in Figure 2, network $h_{\theta}$ predicts an affine
transform $T$ between moving and fixed images $\\{m,f\\}$ synthesized from
label maps $\\{s_{m},s_{f}\\}$. Hypernetwork $\Gamma_{\xi}$ takes the
regularization weight $\lambda$ as input and outputs the parameters
$\eta=\Gamma_{\xi}(\lambda)$ of network $g_{\eta}$. The moved image $m\circ T$
and $f$ are inputs to $g_{\eta}$, which predicts a diffeomorphic warp field
$\phi$. We form the joint transform $\psi=T\circ\phi$ by composition and
compute the moved label map $s_{m}\circ\psi$. Loss $\mathcal{L}_{o}$ recodes
the labels of $\\{s_{m},s_{f}\\}$ to optimize the overlap of select anatomy of
interest—in this case brain labels only. We emphasize the wide applicability
of this strategy to learning anatomy-aware and acquisition-agnostic
registration of any anatomy and with any network architecture, as long as
label maps are available for training.

####  3.2.3 Image synthesis

Given label map $s_{m}$, we generate image $m$ with random contrast, noise,
and artifact corruption (and similarly $f$ from $s_{f}$). Following
SynthMorph, we first sample a mean intensity for each label $j\in K$ in
$s_{m}$ and assign this value to all voxels associated with label $j$. Second,
we corrupt $m$ by randomly applying additive Gaussian noise, anisotropic
Gaussian blurring, a multiplicative spatial intensity bias field, intensity
exponentiation with a global parameter, and downsampling along randomized
axes. In aggregate, these steps produce widely varying intensity distributions
within each anatomical label (Figure 3).

####  3.2.4 Generation hyperparameters

We choose the affine augmentation range such that it encompasses real-world
transforms. Figure C.1 (Appendix C) shows the distribution of affine
transformation parameters measured across public datasets. We adapt all other
values from prior work, which thoroughly analyzed their impact on registration
accuracy (Hoffmann et al., 2021a): Table B.1 (Appendix B) lists
hyperparameters for label-map augmentation and image synthesis.

####  3.2.5 Joint registration

For joint registration, we combine the affine model $h_{\theta}$ with the
deformable SynthMorph architecture. Let $g_{\eta}$ be a convolutional neural
network model with parameters $\eta$ that predicts the warp field
$\phi_{\eta}=g_{\eta}(m,f)$. Usually, approaches to learning deformable
registration would directly fit weights $\eta$ by optimizing a loss of the
form

$\mathcal{L}(\eta,s_{m},s_{f})=(1-\lambda)\mathcal{L}_{o}(\phi_{\eta},s_{m},s_{f})+\lambda\mathcal{L}_{r}(\phi_{\eta}),$
(4)

where $\mathcal{L}_{o}$ quantifies label overlap as before, $\mathcal{L}_{r}$
is a regularization term that encourages smooth warps, and the hyperparameter
$\lambda\in[0,1]$ controls the weighting of both terms.

Because directly fitting weights $\eta$ yields an inflexible network
$g_{\eta}$ predicting warps of fixed regularity, we parameterize $\eta$ using
a hypernetwork approach. Let $\Gamma_{\xi}$ be a neural network with trainable
parameters $\xi$. Following our prior work (Hoopes et al., 2022a),
hypernetwork $\Gamma_{\xi}$ takes the regularization weight $\lambda$ as input
and outputs the weights $\eta=\Gamma_{\xi}(\lambda)$ of the deformable task
network $g_{\eta}$. Consequently, $g_{\eta}$ has no learnable parameters in
our setup—its convolutional kernels $\eta$ can flexibly adapt in response to
the value $\lambda$ takes at test time.

As shown in Figure 4, for joint registration we move image $m$ based on the
affine transform $T_{\theta}=h_{\theta}(m,f)$, and $g_{\eta}$ predicts the
warp field $\phi_{\xi}=g_{\eta}(m\circ T_{\theta},f)$ using kernels
$\eta=\Gamma_{\xi}(\lambda)$ specific to input $\lambda$. The total transform
is $\psi_{\theta\xi}=T_{\theta}\circ\phi_{\xi}$, while the loss of Equation
(4) becomes

$\mathcal{L}(\lambda,\theta,\xi,s_{m},s_{f})=(1-\lambda)\mathcal{L}_{o}(\psi_{\theta\xi},s_{m},s_{f})+\lambda\mathcal{L}_{r}(\phi_{\xi}),$
(5)

We choose $\mathcal{L}_{r}(\phi)=\lVert{\nabla}u\rVert_{2}$, where $u$ is the
displacement of the deformation $\phi=id+u$, and $id$ is the identity field.

####  3.2.6 Overlap loss

In this work, we replace the Dice-based overlap loss term of Equation (3) with
a simpler term (Heinrich, 2019; Wang et al., 2021) that measures MSE between
one-hot encoded labels $s\big{|}_{j}$,

$\mathcal{L}_{o}(\theta,s_{m},s_{f})=\frac{1}{|\Omega|}\sum_{\begin{subarray}{c}j\in
J\\\ x\in\Omega\end{subarray}}\Big{[}(s_{m}\big{|}_{j}\circ
T_{\theta})(x)-s_{f}\big{|}_{j}(x)\Big{]}^{2}$ (6)

where we replace weights $\theta$ with $\\{\theta,\xi\\}$ and transform
$T_{\theta}$ with $\psi_{\theta\xi}$ for joint registration. MSE is sensitive
to the proportionate contribution of each label $j\in J$ to overall alignment,
whereas Equation (3) normalizes the contribution of each label by its
respective size. As a result, the MSE loss term discourages the optimization
to disproportionately focus on aligning smaller structures, which we find
favorable for warp regularity at structure boundaries.

Figure 5:  Affine registration architectures. A recurrent Encoder estimates
refinements to the current transform $T_{i}$ from moved image $m_{i}=m\circ
T_{i}$ and fixed image $f$. Decomposer predicts a one-shot displacement field
(no activation) with corresponding voxel weights (ReLU), that we decompose in
a weighted least-squares (WLS) sense to estimate affine transform $T$.
Detector outputs ReLU-activated feature maps for a single image. We compute
their centers of mass (COM) and weights separately for $m$ and $f$, to fit a
transform $T$ that aligns these point sets. Parentheses specify filter
numbers. We LeakyReLU-activate the output of unnamed convolutional blocks
(param. $\alpha=0.2$). Stacked convolutional blocks of decreasing size
indicate subsampling by a factor of 2 via max pooling following each
activation.

###  3.3 Affine architectures

Estimating an affine transform $T$ from a pair of medical images in $N$D
requires reducing a large input space of the order of 100k–10M voxels to only
$N(N+1)$ output parameters. We analyze three competing network architectures
(Figure 5) that represent state-of-the art methods (Shen et al., 2019; Zhu et
al., 2021; Balakrishnan et al., 2019; De Vos et al., 2019; Moyer et al., 2021;
Yu et al., 2022).

####  3.3.1 Parameter encoder

We first build on networks combining a convolutional encoder with an FC layer
(Zhu et al., 2021; Shen et al., 2019) whose $N(N+1)$ output units we interpret
as parameters for translation, rotation, scale, and shear. We refer to a
cascade of $C$ such subnetworks $h_{i}$, with $i\in\\{1,2,...,C\\}$ as
“Encoder”. Each $h_{i}$ outputs a matrix constructed from the affine
parameters as shown in Appendix A, to incrementally update the total
transform. We obtain transform $T_{i}$ by matrix multiplication after invoking
subnetwork $h_{i}$,

$T_{i}=h_{1}(m_{0},f)\;h_{2}(m_{1},f)\;\cdots\;h_{i}(m_{i-1},f)$ (7)

where $m_{i}=m\circ T_{i}$ is the moving image transformed by $T_{i}$, and
$T_{0}=I_{N}$ is the identity matrix. As the subnetworks $h_{i}$ are
architecturally identical, weight sharing is possible, and we evaluate
versions of the model with and without weights shared across cascades.

For balanced gradient steps, we complete each subnetwork with a layer
involving learnable local rescaling weights applied to the affine parameters
before matrix construction.

####  3.3.2 Warp decomposer

We propose a second architecture building on deformable registration models
(Balakrishnan et al., 2019; De Vos et al., 2019). “Decomposer” estimates a
dense deformation field $\phi_{\theta}$ with corresponding non-negative voxel
weights $\kappa_{\theta}$ that we decompose into the affine output transform
$T_{\theta}=h_{\theta}(m,f)$ and a (discarded) residual component
$\delta_{\theta}$, i.e. $\phi_{\theta}=\delta_{\theta}\circ T_{\theta}$. The
voxel weights $\kappa_{\theta}$ enable the network $h_{\theta}$ to focus the
decomposition on the anatomy of interest. Both
$(\phi_{\theta},\kappa_{\theta})$ are the output of a single fully
convolutional network and thus benefit from weight sharing. We decompose
$\phi_{\theta}$ in a weighted least-squares (WLS) sense over the discrete
spatial domain $\Omega$ of $f$, using the definition of $t$ from Equation (1)
as the submatrix of $T$ that excludes the last row:

$\hat{t}_{\theta}=\operatorname*{arg\,min}_{t}\,\sum_{x\,\in\,\Omega}\kappa_{\theta}(x)\,\big{\lVert}\phi_{\theta}(x)^{\top}-(\begin{matrix}x^{\top}&1\end{matrix})\,t^{\top}\big{\rVert}^{2},$
(8)

where $t^{\top}$ is the matrix transpose of $t$. Denoting
$W=\operatorname*{diag}(\kappa_{\theta})$, and by $X$ and $y$ the matrices
whose corresponding rows are $(\begin{matrix}x^{\top}&1\end{matrix})$ and
$\phi_{\theta}(x)^{\top}$ for each $x\in\Omega$, respectively, the closed-form
solution of Equation (8) is,

$\hat{t}_{\theta}^{\top}=(X^{\top}WX)^{-1}X^{\top}Wy.$ (9)

####  3.3.3 Feature detector

Third, we extend a recent architecture (Moyer et al., 2021; Yu et al., 2022)
that takes a single image as input and predicts a set of $k$ non-negative
spatial feature maps $F_{i}$ with $i\in\\{1,2,...,k\\}$, to support full
affine transforms (Yu et al., 2022) and WLS (Moyer et al., 2021).

Following a series of convolutions, we compute the center of mass $a_{i}$ and
channel power $p_{i}\big{|}_{m}$ for each feature map $F_{i}\big{|}_{m}$ of
the moving image,

$a_{i}=p_{i}^{-1}\sum_{x\,\in\,\Omega}xF_{i}\big{|}_{m}(x),\quad
p_{i}\big{|}_{m}=\sum_{x\,\in\,\Omega}F_{i}\big{|}_{m}(x),$ (10)

and separately center of mass $b_{i}$ with channel power $p_{i}\big{|}_{f}$
for each $F_{i}\big{|}_{f}$ of the fixed image. We interpret the sets
$\\{a_{i}\\}$ and $\\{b_{i}\\}$ as corresponding moving and fixed point
clouds. “Detector” refers to a network $h_{\theta}$ that returns the affine
transform $t_{\theta}=h_{\theta}(m,f)$ aligning these point clouds subject to

$\hat{t}_{\theta}=\operatorname*{arg\,min}_{t}\,\sum_{i=1}^{k}\epsilon_{i}\,\big{\lVert}a_{i}^{\top}-(\begin{matrix}b_{i}^{\top}&1\end{matrix})\,t^{\top}\big{\rVert}^{2},$
(11)

where we define the normalized scalar weight $\epsilon_{i}$ as

$\epsilon_{i}=p_{i}\big{|}_{m}\big{(}\sum_{j=1}^{k}p_{j}\big{|}_{m}\big{)}^{-1}\,p_{i}\big{|}_{f}\big{(}\sum_{j=1}^{k}p_{j}\big{|}_{f}\big{)}^{-1}.$
(12)

Let $X$ and $y$ be matrices whose $i$th rows are
$(\begin{matrix}a_{i}^{\top}&1\end{matrix})$ and $b_{i}^{\top}$, respectively.
With $W=\operatorname*{diag}(\\{\epsilon_{i}\\})$, Equation (9) yields the
closed-form solution $\hat{t}_{\theta}$ as above.

####  3.3.4 Implementation

Except for the final convolutions indicated in Figure 5, each affine model
$h_{\theta}$ uses convolutional blocks with $w$ filters; the network width $w$
does not vary within a model. Unless stated otherwise, we activate the output
of each block with LeakyReLU (parameter $\alpha=0.2$) and downsample by a
factor of 2 using max pooling.

As in our prior work, the deformable model $g_{\eta}$ implements a U-Net
(Ronneberger et al., 2015) architecture of width $w=256$ and integrates a
stationary velocity field (SVF) $\nu$ to predict a diffeomorphism (Ashburner,
2007; Dalca et al., 2018). Hypermodel $\Gamma_{\xi}$ is a simple feed-forward
network with 4 ReLU-activated hidden FC layers that have 32 output units each.

All kernels are of size $3^{N}$. For computational efficiency, our 3D models
linearly downsample the network inputs $\\{m,f\\}$ and loss inputs
$\\{s_{m},s_{f}\\}$ by a factor of 2. We min-max normalize input images such
that their intensities fall in the interval $[0,1]$. Affine coordinate
transforms operate in a zero-centered index space, and we have Encoder predict
rotation parameters in degrees. This parameterization ensures varying rotation
angles has an effect of similar magnitude as translations in millimeters, at
the scale of the brain, which we find helps networks converge faster. Appendix
A includes further details.

####  3.3.5 Optimization

We fit model parameters with stochastic gradient descent using Adam (Kingma
and Ba, 2014) over consecutive training strips $S_{i}$ ($i\in\\{1,2,...\\}$)
of $10^{6}$ batches each. At the beginning of each strip or in the event of
divergence, we choose successively smaller learning rates from
$l\in\\{10^{-4},10^{-5},10^{-6}\\}$. All models of the 2D network comparison
train for a single strip with a batch size of 2.

To avoid non-invertible matrices $M=X^{\top}WX$ at the start of training, we
pretrain Decomposer for 500 iterations, temporarily replacing the output
transform with the field $T_{\theta}=\phi_{\theta}\;\odot\;\kappa_{\theta}$,
where $\kappa_{\theta}$ are the voxel weights predicted by the network
(Section 3.3.2), and $\odot$ denotes voxel-wise multiplication. We initialize
the rescaling weights of Encoder to 1 for translations and rotations, and to
0.05 for scaling and shear, which we find favorable to faster convergence.

Because SynthMorph training is generally not prone to overfitting, it uses a
simple stopping criterion measuring progress $P_{i}$ over batches $t\in S_{i}$
in terms of validation Dice overlap $D$ (Section 4.3). The 3D models train
with a batch size of 1 until the mean overlap across $S_{i}$ exceeds
$P_{i}=99.9\%$ of the maximum, that is,

$P_{i}=\big{\\{}|S_{i}|\max\limits_{t\in S_{i}}D(t)\big{\\}}^{-1}\sum_{t\in
S_{i}}D(t).$ (13)

For joint registration, we uniformly sample hyperparameter values
$\lambda\in[0,1]$. We freeze parameters $\theta$ of the trained affine
submodel $h_{\theta}$ to fit only the weights $\xi$ of hypernetwork
$\Gamma_{\xi}$, optimizing the loss of Equation (5). While unfreezing the
affine parameters $\theta$ within the setup of Figure 4 results in unchanged
deformable accuracy, affine accuracy decreases. Although adding an affine loss
term can prevent this decrease, we choose to train both stages separately—the
third loss term increases complexity at no benefit, and simultaneous affine-
deformable training requires more GPU memory.

##  4 Experiments

In a first experiment, we thoroughly analyze the performance of the different
architectures across a broad range of variants and transformations, to
understand how networks learn and best represent the affine component. In a
second experiment, we select and train an architecture with synthetic data
only. This experiment shifts the focus from network architectures to building
a readily usable tool, and we assess the resulting accuracy in various affine
registration tasks. In a third experiment, we complete the affine model with a
deformable hypernetwork to produce a joint registration solution and compare
its performance to readily usable baseline tools.

Table 1: Acquired test data for baseline comparisons spanning a range of MRI contrasts, resolutions, and subject populations. QIN includes contrast-enhanced clinical stacks of thick slices from patients with glioblastoma, whereas the other acquisitions use 3D sequences. While HCP-D and MASi include pediatric data, the remaining datasets consist of adult populations. Dataset | Type |  Voxel size (mm3) | Pairs
---|---|---|---
UKBB | T1w, age 40–75 a | $1.0{\times}1.0{\times}1.0$ | 200
GSP | T1w, age 18–35 a | $1.2{\times}1.2{\times}1.2$ | 100
IXI | T1w | $0.9{\times}0.9{\times}1.2$ | 100
| T2w | $0.9{\times}0.9{\times}1.2$ | 100
| PDw | $0.9{\times}0.9{\times}1.2$ | 100
HCP-D | T1w, age 5–21 a | $0.8{\times}0.8{\times}0.8$ | 80
MASi | T1w, age 5–8 a | $1.0{\times}1.0{\times}1.0$ | 80
QIN | post-contrast T1w | $0.4{\times}0.4{\times}6.0$ | 50

###  4.1 Data

The training-data synthesis and analyses use 3D brain MRI from a very diverse
collection of public data, aiming to truly capture the behavior of the methods
facing the diversity of real-world images. While users of SynthMorph do not
need to preprocess their data, our experiments use images conformed to the
same isotropic $256\times 256\times 256$ 1-mm voxel space using trilinear
interpolation, and by cropping and zero-padding symmetrically. We rearrange
the voxel data to produce gross left-inferior-anterior (LIA) orientation with
respect to the volume axes. Experiments conducted in 2D use corresponding mid-
sagittal slices extracted from 3D images and label maps.

####  4.1.1 Generation label maps

For training-data synthesis, we compose a set of 100 whole-head tissue
segmentations, each derived from T1w acquisitions with isotropic $\sim$1-mm
resolution (although we do not use the T1w images in our experiments). The
training segmentations include 30 locally scanned adult FSM subjects (Greve et
al., 2021), 30 participants of the cross-sectional Open Access Series of
Imaging Studies (OASIS, Marcus et al. 2007), 30 teenage subjects from the
Adolescent Brain Cognitive Development study (ABCD, Casey et al. 2018), and 10
infants scanned at Boston Children’s Hospital at age 0-18 months (de Macedo
Rodrigues et al., 2015; Hoopes et al., 2022b).

We derive brain label maps from the conformed T1w scans using SynthSeg (Billot
et al., 2020). We emphasize that inaccuracies in the segmentations have little
impact on our strategy, as the images synthesized from the segmentations will
be in perfect voxel-wise registration with the labels by construction. To
facilitate the synthesis of spatially complex image signals outside the brain,
we use a simple thresholding procedure to add non-brain labels to each label
map. The procedure sorts non-zero image voxels outside the brain into one of
six intensity bins, equalizing bin sizes on a per-image basis.

####  4.1.2 Analysis and training images

For architecture analysis, we use T1w training images from 5000 adult
participants of the UK Biobank (UKBB) study (Sudlow et al., 2015; Miller et
al., 2016; Alfaro-Almagro et al., 2018). We also randomly pool 1000 distinct
registration pairs from OASIS and another distinct 1000 pairs from ABCD to
analyze real-world transforms.

####  4.1.3 Evaluation images

For baseline comparisons, we pool adult and pediatric T1w images from UKBB,
the Brain Genomics Superstruct Project (GSP, Holmes et al. 2015), the Lifespan
Human Connectome Project Development (HCP-D, Somerville et al. 2018; Harms et
al. 2018), MASiVar (MASi, Cai et al. 2021), and IXI (Imperial College London,
2015). The evaluation set also includes IXI scans with T2w and PDw contrast.
As all these images are near-isotropic $\sim$1-mm acquisitions, we complement
the dataset with contrast-enhanced clinical T1w stacks of axial 6-mm slices
from subjects with newly diagnosed glioblastoma (QIN, Clark et al. 2013; Prah
et al. 2015; Mamonov and Kalpathy-Cramer 2016).

Our experiments use the held-out test images listed in Table 1. For monitoring
and model validation, we use a handful of images pooled from the same
datasets, which do not overlap with the test subjects. We do not consider QIN
at validation and validate performance in pediatric data with held-out ABCD
subjects.

To measure registration accuracy, we compute anatomical brain label maps
individually for each conformed image volume using SynthSeg (Billot et al.,
2020). Although SynthMorph does not require skull-stripping, we skull-strip
all images with SynthStrip (Hoopes et al., 2022b) for a fair comparison across
images that have undergone the preprocessing steps expected by the baseline
methods—unless explicitly noted.

####  4.1.4 Labels

The training segmentations encompass a set $K$ of 38 different labels, 32 of
which are standard (lateralized) FreeSurfer labels (Fischl et al., 2002).
Parenthesizing their average size over FSM subjects relative to the total
brain volume and combining the left and right hemispheres, these structures
are: cerebral cortex (43.4%) and white matter (36.8%), cerebellar cortex
(9.2%) and white matter (2.2%), brainstem (1.8%), lateral ventricle (1.7%),
thalamus (1.2%), putamen (0.8%), ventral DC (0.6%), hippocampus (0.6%),
caudate (0.6%), amygdala (0.3%), pallidum (0.3%), 4th ventricle (0.1%),
accumbens (0.1%), inferior lateral ventricle (0.1%), 3rd ventricle (0.1%), and
background.

The remaining labels map to variable image features outside the brain (Section
4.1.1). These added labels do not necessarily represent distinct or meaningful
anatomical structures but expose the networks to non-brain image content at
training. We use all labels $K$ to synthesize training images but optimize the
overlap of brain-specific labels $J\subset K$ based on Equation (3).

For affine training and evaluation, we merge brain structures such that $J$
consists of larger tissue classes: left and right cerebral cortex, left and
right subcortex, and cerebellum. These classes ensure that small labels like
the caudate do not have a disproportionate influence on global brain
alignment—different groupings may work equally well. In contrast, deformable
registration redefines $J$ to include the 21 largest brain structures up to
and including caudate. We use these labels for deformable training and
evaluation, as prior analyses report that “only overlap scores of localized
anatomical regions reliably distinguish reasonable from inaccurate
registrations” (Rohlfing, 2011).

For a rigorous assessment of deformable registration accuracy, we separately
consider the set of the 10 finest-grained structures above whose overlap we do
not optimize at training, including the labels from amygdala through 3rd
ventricle.

###  4.2 Baselines

We test affine and joint classical registration with ANTs (Avants et al.,
2011) version 2.3.5 using recommended parameters (Pustina and Cook, 2017) for
the NCC metric within and MI across MRI contrasts. We test NiftyReg (Modat et
al., 2014) version 1.5.58 with the NMI metric and enable SVF integration for
joint registration, as in our approach.

We also run the patch-similarity method Deeds (Heinrich et al., 2012),
2022-04-12 version. For a rigorous baseline assessment, we reduce the default
grid spacing from $8\times 7\times 6\times 5\times 4$ to $6\times 5\times
4\times 3\times 2$. This setting effectively trades a shorter runtime for
increased accuracy as recommended by the author, since it optimizes the
parametric B-spline model on a finer control point grid (Heinrich et al.,
2013). The modification results in a 1-2% accuracy improvement for most
datasets as in prior work (Hoffmann et al., 2021a).

We test affine registration with mri_robust_register (“Robust”) from
FreeSurfer 7.3 (Fischl, 2012) using its robust cost functions (Reuter et al.,
2010), as only the robust cost functions can down-weight the contribution of
regions that deform non-linearly. However, we highlight that the robust-
entropy metric for cross-modal registration is experimental. We use Robust
with up to 100 iterations and initialize the affine registration with a rigid
run. Finally, we also test affine and deformable registration with the FSL
(Jenkinson et al., 2012) tools FLIRT (Jenkinson and Smith, 2001) version 6.0
and FNIRT (Andersson et al., 2007) build 507. While the recommended cost
function of FLIRT, correlation ratio, is suitable within and across
modalities, we emphasize that users cannot change FNIRT’s MSE objective, which
specifically targets within-contrast registration.

We compare DL model variants covering popular registration architectures in
Section 4.4. This analysis uses the same capacity and training set for each
model. For our final synthesis-based tool in Sections 4.5 and 4.6, we consider
readily available machine-learning baselines trained by their respective
authors, to assess their generalization capabilities to the diverse data we
have gathered. This strategy evaluates what level of accuracy a user can
expect from off-the-shelf methods without retraining, as retraining is
generally challenging for users (see Section 5.4). We test: KeyMorph (Yu et
al., 2022) and C2FViT (Mok and Chung, 2022) models trained for pair-wise
affine, and the 10-cascade Volume Tweening Network (VTN) (Zhao et al., 2019b,
d) trained for joint affine-deformable registration. Each network receives
inputs with the expected image orientation, resolution, and intensity
normalization.

Table 2: Network capacity for model comparison. Each network uses a different width $w$ held constant across its layers to reach a target capacity of 250k or 500k parameters, up to a small deviation. Architecture | Config. | $w$ | Capacity | Dev. (%)
---|---|---|---|---
Encoder |  $C$ = | $2^{0}$ | 72 | 252k | $+0.8$
Encoder |  $C$ = | $2^{1}$ | 45 | 250k | $0.0$
Encoder |  $C$ = | $2^{2}$ | 27 | 247k | $+1.2$
Encoder |  $C$ = | $2^{3}$ | 16 | 255k | $+2.0$
Encoder |  $C$ = | $2^{4}$ | 9 | 260k | $+4.0$
Decomposer |  $n$ = | 0 | 63 | 253k | $+1.2$
Decomposer |  $n$ = | 1 | 59 | 254k | $+1.6$
Decomposer |  $n$ = | 2 | 55 | 248k | $-0.8$
Decomposer |  $n$ = | 3 | 52 | 246k | $-1.6$
Detector |  $k$ = | $2^{3}$ | 62 | 248k | $-0.8$
Detector |  $k$ = | $2^{4}$ | 62 | 252k | $+0.8$
Detector |  $k$ = | $2^{5}$ | 61 | 253k | $+1.2$
Detector |  $k$ = | $2^{6}$ | 58 | 246k | $-1.6$
Encoder |  $C$ = | $2^{0}$ | 110 | 498k | $-0.4$
Decomposer |  $n$ = | 0 | 89 | 504k | $+0.8$
Detector |  $k$ = | $2^{5}$ | 87 | 503k | $+0.6$

###  4.3 Evaluation metrics

To measure registration accuracy, we propagate the moving label map $s_{m}$
using the predicted transform $T$ to obtain the moved label map $s_{m}\circ T$
and compute its (hard) Dice overlap $D$ (Dice, 1945) with the fixed label map
$s_{f}$. Since all baseline methods tested optimize an image-based objective,
for a balanced comparison we also evaluate NCC of the modality-independent
neighborhood descriptor (MIND, Heinrich et al. 2012) between the moved image
$m\circ T$ and the fixed image $f$. As we seek to measure brain-specific
registration accuracy, we remove any image content external to the brain
labels prior to evaluating the image-based metrics. We use paired two-sided
$t$-tests to determine whether differences in mean scores between methods are
significant.

We analyze the regularity of deformation field $\phi$ in terms of the mean
absolute value of the logarithm of the Jacobian determinant $J_{\phi}$ over
spatial domain $\Omega$. This quantity is sensitive to the deviation of
$J_{\phi}$ from the ideal value 1 and thus provides a measure of the width of
the distribution of log-Jacobian determinants, the “log-Jacobian spread”
$\delta$:

$\delta(\phi)=\frac{1}{|\Omega^{*}|}\sum_{x\,\in\,\Omega^{*}}{\big{|}\ln\big{(}|J_{\phi}(x)|\big{)}\big{|}},$
(14)

where $\Omega^{*}=\\{x\in\Omega\,|\,J_{\phi}(x)\neq 0\\}$. We also determine
the proportion of folding voxels, that is, locations $x\in\Omega$ where
$J_{\phi}(x)<0$. We compare the inverse consistency of registration methods by
means of the average displacement $E$ that voxels undergo upon subsequent
application of transforms $\\{T_{1},T_{2}\\}$,

$E(T_{1},T_{2})=\frac{1}{|\Omega|}\sum_{x\,\in\,\Omega}{\big{\lVert}\big{(}T_{2}\circ
T_{1}\big{)}(x)-x\big{\rVert}}_{2}.$ (15)

Specifically, we compute the inverse consistency of method $h$ with
$T_{1}=h(m,f)$ and $T_{2}=h(f,m)$ for any pair of input images $\\{m,f\\}$.

###  4.4 Experiment 1: network analysis

In the first experiment, we rigorously analyze variants and ablations of the
three competing architectures from Section 3.3. Assuming a network capacity of
$\sim$250k learnable parameters, our goal is to explore the relative strengths
and weaknesses of the affine architectures, before we move on to training a
network using the SynthMorph strategy.

####  4.4.1 Setup

The experiment compares networks in a 2D context, that is, at estimating 2D
transforms with six degrees of freedom for pairs of image slices. A 2D setup
for learning unsupervised registration by minimizing an NCC loss enables us to
reduce the computational burden and thus explore numerous model
configurations. We train networks drawing $\\{m,f\\}$ from a set of 5000 T1w
UKBB images, and test registration on a validation set of 100 cross-subject
pairs that does not overlap with the training data. To keep network capacities
similar, each model has a different width $w$, held constant across its
convolutional layers as shown in Table 2. Training uses only affine
augmentation as indicated in Table B.1.

First, we assess to what extent Encoder benefits from weight sharing. We train
models with different numbers $C\in\\{1,2,4,8,16\\}$ of subnetworks that
either share or use separate weights.

Second, we compare Decomposer variants that fit $T$ in an OLS sense, using
weights $\forall x\in\Omega$, $\kappa_{\theta}(x)=1$, or in a WLS sense. For
both, we asses how increasing the resolution $\varrho$ of the field
$\phi_{\theta}$ relative to $f$ affects performance, by upsampling by a factor
of 2 after each of the first $n\in\\{0,1,2,3\\}$ convolutional blocks
following the encoder, using skip connections where possible, such that
$\varrho(n)=1/2^{4-n}$.

Third, we analyze OLS and WLS variants of Detector predicting
$k\in\\{8,16,32,64\\}$ feature maps to compute the corresponding
$(a_{i},p_{i})$ and $(b_{i},q_{i})$ for $i\in\\{1,2,...,k\\}$.

Finally, we select a suitable configuration per architecture and analyze its
performance across a range of transformation magnitudes. We investigate how
models adapt to larger transforms, by fine-tuning pretrained weights to twice
the affine augmentation amplitudes of Table B.1 until convergence, and we also
repeat the experiment with doubled capacity. We test on copies of the UKBB
validation set, each injected with random affine transforms of maximum
strength $\gamma\in[0,2]$ relative to the augmentation range of Table B.1. For
example, at a given $\gamma$, we uniformly sample a rotation angle
$r\sim\mathcal{U}(-\gamma\alpha,\gamma\alpha)$ for each of the 200 moving and
each fixed images, where $\alpha=45^{\circ}$, and similarly for all other
degrees of freedom (DOF), that we apply by resampling the image (Appendix A).

Figure 6:  Network analysis. We assess Encoder with different numbers of
subnetworks $C$. We also analyze Decomposer and Detector variants using
ordinary (OLS) or weighted least squares (WLS), exploring the effect of
varying the warp resolution $\varrho$ relative to the input image size and
different numbers of output feature maps $k$, respectively. Dice scores are
averages over 100 UKBB cross-subject 2D pairs. Shaded areas indicate the
standard error of the mean.  Figure 7:  Network robustness across affine
transform strengths $\gamma$ relative to the augmentation range of Table B.1.
At a given $\gamma$, we resample each image of the validation set with affine
parameters drawn from a uniform distribution $\mathcal{U}$ modulated by
$\gamma$, for example rotation angle
$r\sim\mathcal{U}(-\gamma\alpha,\gamma\alpha)$, where $\alpha=45^{\circ}$. We
also test networks trained with doubled augmentation (aug) and doubled
capacity (cap). Dice scores are averages over 100 UKBB cross-subject 2D pairs.
Shaded areas indicate the standard error of the mean.  Figure 8:
Representative affine 3D registration examples showing the image moved by each
method overlaid with the fixed brain mask (red). Each row is an example from a
different dataset.

####  4.4.2 Results

Figure 6 compares registration accuracy for the NCC-trained model
configurations in terms of Dice overlap. Encoder achieves the highest
accuracy, surpassing the best Detector configuration by up to 0.4 and the best
Decomposer by up to 1 Dice point. Using more subnetworks improves Encoder
performance, albeit with diminishing returns after $C=4$ and at the cost of
substantially longer training times that roughly scale with $C$.

The local rescaling weights of subnetwork $h_{1}$ converge to values around 1
for translations and rotations and around 0.01 for scaling and shear. The
values tend to decrease for subsequent $h_{i}$ ($i>1)$, most noticeably the
translational weights. For $h_{C}$, the translational weights converge to
roughly 50% of those of $h_{1}$, suggesting that the first subnetworks perform
the bulk of the alignment, whereas the subsequent $h_{i}$ refine it by smaller
amounts.

Although keeping subnetwork weights separate might also enable each $h_{i}$ to
specialize in increasingly fine adjustments to the final transform, in
practice we observe no benefit in distributing capacity over the subnetworks
compared to weight sharing. Decomposer shows a clear trend towards lower
output resolutions $\varrho$ improving accuracy. While decomposing the field
$\phi_{\theta}$ in a WLS sense boosts performance by 0.6–1.3 points over OLS,
the model still lags behind the other architectures while requiring 2–3 times
more training iterations to converge. There is little difference across
numbers $k$ of output feature maps, and choosing WLS over OLS slightly
increases in accuracy.

Figure 7 shows network robustness across a range of maximum transform
strengths $\gamma$, where we compare Encoder with $C=4$ subnetworks sharing
weights to the WLS variants of Decomposer without upsampling and Detector with
$k=32$ output channels, to balance performance and efficiency.

Detector proves most robust to large transforms, remaining unaffected up to
$\gamma\approx 1.2$, i.e. shifts and rotations up to 36 mm and 54∘ for each
axis, respectively, and scale and shear up to 0.12. In contrast, accuracy
declines substantially for Encoder and Decomposer after $\gamma\approx 0.8$,
corresponding to maximum transforms of 24 mm and $36^{\circ}$ (blue). Doubling
the affine augmentation extends Encoder and Decomposer robustness to
$\gamma\approx 1.2$ but comes at the cost of a drop of 1 and 2 Dice points for
all $\gamma<1.2$, respectively (orange). Decomposer performance is capacity-
bound, as doubling the number of parameters restores $\sim$50% of the drop in
accuracy, whereas increasing capacity does not improve Encoder accuracy for
$\gamma<1.2$ (green). Detector optimally benefits from the doubled affine
augmentation, which enables the network to perform robustly across the test
range (orange). Doubling capacity has no effect (green).

In conclusion, the marginal lead of Encoder only manifests for small
transforms and at $C=16$ cascades. The 16 interpolation steps of this variant
render it intractably inefficient for 3D applications. In contrast, Detector
performs with high accuracy across transformation strengths, making it a more
suitable architecture for a general registration tool.

Figure 9:  Affine 3D registration accuracy as mean Dice scores and in terms of
image similarity. Each violin shows the distribution across the skull-stripped
cross-subject pairs from Table 1. For comparison, the asterisk indicates
SynthMorph performance without skull-stripping. Downward arrows indicate
median scores outside the plotted range. Metric values range from 0 to 100.
Higher is better.

###  4.5 Experiment 2:
affine tool performance

Based on the accurate performance of Detector and its robustness across
transform strengths shown in Section 4.4 in 2D and prior work (Yu et al.,
2022) in 3D, we train the WLS-based architecture in 3D using the generative
strategy of Figure 2, leading to “affine SynthMorph”. We focus on the
development of an anatomy-aware affine registration tool that generalizes
across MRI acquisition protocols while enabling brain registration without
requiring the user to preprocess data.

####  4.5.1 Setup

First, to give the reader an idea of the accuracy achievable with off-the-
shelf algorithms for data unseen at training, we compare affine SynthMorph to
classical and DL baselines trained by the respective authors. We test affine
registration of skull-stripped images across MRI contrasts, for a variety of
different imaging resolutions and populations, including adults, children, and
patients with glioblastoma. We also compare the symmetry of each method with
regard to reversing the order of input images. Each test involves held-out
image pairs from separate subjects, summarized in Table 1.

Affine SynthMorph implements Detector (Figure 5) with $w=256$ convolutional
filters, as this network width proved adequate for learning registration from
synthetic data (Hoffmann et al., 2021a). Using 3D convolutions, we choose
$k=64$ output feature maps, shown to yield best performance for large 3D
transforms (Yu et al., 2022). We train SynthMorph solely with synthetic images
generated from label maps using the hyperparameters of Table B.1.

Second, we analyze the effect of thick-slice acquisitions on affine SynthMorph
accuracy compared to classical baselines. This experiment retrospectively
reduces the through-plane resolution of the moving image of each
GSP$\rightarrow$IXIT1 pair to produce stacks of axial slices of thickness
${\Delta}z{}\in\\{1,2,...,10\\}$ mm. At each ${\Delta}z{}$, we simulate
partial voluming (Kneeland et al., 1986; Simmons et al., 1994) by smoothing
all moving images in slice-normal direction with a 1D Gaussian kernel of full-
width at half-maximum (FWHM) ${\Delta}z{}$ and by extracting slices
${\Delta}z{}$ apart using linear interpolation. Finally, we restore the
initial volume size by linearly upsampling the through-plane.

Third, we evaluate the importance of skull-stripping the input images for
accurate registration. With the exception of skull-stripping, we preprocess
full-head GSP$\rightarrow$IXIT1 pairs as expected by each method and assess
brain-specific registration accuracy by evaluating image-based metrics within
the brain only.

(a) Affine

(b) Deformable

Figure 10: Dependency of 3D 10(a) affine and 10(b) deformable registration accuracy on slice thickness. For comparability, we initialize all deformable tools with affine transforms estimated by NiftyReg. Each value indicates the mean over 100 skull-stripped pairs (higher is better). Shaded areas indicate the standard error of the mean. Table 3: Single-threaded runtimes on a 2.2-GHz Intel Xeon Silver 4114 CPU, averaged over $n=10$ runs. Errors indicate standard deviations. On an NVIDIA V100 GPU, all affine and deformable DL runtimes (below midline) are about 1 minute, including setup. Method | Affine (sec) | Deformable (sec)
---|---|---
ANTs |  777.8$\,\pm\,$ | 36.0 |  17189.5$\,\pm\,$ | 52.7
NiftyReg |  293.7$\,\pm\,$ | 0.5 |  7021.0$\,\pm\,$ | 21.3
Deeds |  142.8$\,\pm\,$ | 0.3 |  383.1$\,\pm\,$ | 0.6
Robust |  1598.9$\,\pm\,$ | 0.8 | –
FSL |  151.7$\,\pm\,$ | 0.4 |  8141.5$\,\pm\,$ | 195.7
C2FViT1 |  43.7$\,\pm\,$ | 0.3 | –
KeyMorph |  36.2$\,\pm\,$ | 2.6 | –
VTN2 | – |  63.5$\,\pm\,$ | 0.3
SynthMorph |  72.4$\,\pm\,$ | 0.8 |  887.4$\,\pm\,$ | 2.5
1 Timed on the GPU as the device is hard-coded.
2 Implementation performs joint registration only.

####  4.5.2 Results

Figure 8 shows representative registration examples for the tested dataset
combinations, while Figure 9 quantitatively compares affine registration
accuracy across skull-stripped image pairings. Although affine SynthMorph has
not seen any real MRI data at training, it achieves the highest Dice score for
every dataset tested.

For the GSP$\rightarrow$IXIT1 and MASi$\rightarrow$HCP-D pairs that most
baselines are optimized for, SynthMorph exceeds the best-performing baseline
NiftyReg by ${\Delta}D=0.1$ points ($p<10^{-4}$ and $p<0.03$ for paired two-
sided $t$-tests). Across all other pairings, SynthMorph matches the Dice score
achieved by the most accurate affine baseline, which is NiftyReg in every
case.

Deeds performs least accurately, lagging behind the second last classical
baselines by ${\Delta}D=10.1$ or more. The other classical methods perform
robustly across all testsets, generally within 1–2 Dice points of each other.

On the MASi$\rightarrow$HCP-D testset, FLIRT’s performance exceeds Robust by
${\Delta}D=3$ ($p=10^{-23}$) and matches it across GSP$\rightarrow$IXIT1 pairs
($p=0.8$). Across the remaining testsets, FLIRT ranks fourth among classical
baselines.

In contrast, the DL baselines do not reach the same accuracy. Even for the T1w
pairs they were trained with, SynthMorph leads by ${\Delta}D=3.7$ or more,
likely due to domain shift between the test and baseline training data. As
expected, DL-baseline performance continues to decrease as the test-image
characteristics deviate further from those at training. Interestingly, VTN
consistently ranks among the least accurate baselines, although its
preprocessing effectively initializes the translation and scaling parameters
by separately adjusting the moving and fixed images such that the brain fills
the whole FOV.

Even though affine SynthMorph does not directly optimize image similarity at
training, it surpasses NiftyReg for GSP$\rightarrow$IXIT1 ($p<2\times
10^{-11}$) and MASi$\rightarrow$HCP-D ($p<0.02$) pairs in terms of the image-
based NCC-MIND metric. Generally, NCC-MIND ranks the methods similarly to Dice
overlap, as does NCC (not shown) across the T1w registration pairs.

Figure 11 shows that SynthMorph’s affine symmetry outperforms all baselines
tested across GSP$\rightarrow$IXIT1 pairs. When we reverse the order of the
input images, the inconsistency between forward and backward transforms is
$E=0.003\%$ of the voxel size, tightly followed by NiftyReg. Robust also uses
an inverse-consistent algorithm, leading to $E=0.4\%$. The remaining baselines
are substantially less symmetric, with inconsistencies of $E=100\%$ of the
voxel size for KeyMorph or more.

Figure 10(a) shows how registration accuracy evolves with increasing moving-
image slice thickness ${\Delta}z{}$. SynthMorph and ANTs remain the most
robust for ${\Delta}z{}\leq 6$ mm, reducing to 99% at ${\Delta}z{}=10$ mm. For
${\Delta}z{}\in[2,5]$ mm, ANTs accuracy even improves slightly, likely
benefiting from the smoothing effect on the images. The classical baselines
FLIRT and Robust are only mildly affected by thicker slices. While their Dice
scores decrease more rapidly for ${\Delta}z{}\leq 8$, their accuracy reduces
to 99% and about 98.5% at ${\Delta}z{}=10$ mm. Deeds is noticeably more
susceptible to resolution changes, decreasing to less than 95% at
${\Delta}z{}>6.5$ mm.

Figure 12 compares the drop in median Dice overlap the affine methods undergo
for full-head as opposed to skull-stripped GSP$\rightarrow$IXIT1 images.
Except for Deeds, brain-specific accuracy reduces substantially, by 3% in the
case of NiftyReg and up to 8% for ANTs. Affine SynthMorph remains most robust:
its Dice overlap changes by less than 0.05%. Deeds is the only method that
sees accuracy increase but still yields the lowest score for the testset.

Table 3 lists the registration time required by each affine method on a
2.2-GHz Intel Xeon Silver 4114 CPU using a single computational thread. The
values shown reflect averages over $n=10$ uni-modal runs. Classical runtimes
range between 2 and 27 minutes with Deeds being the fastest and Robust being
the slowest, although we highlight that we substantially increased the number
of Robust iterations. Complete single-threaded DL runtimes are about 1 minute,
including model setup. However, inference only takes a few seconds and reduces
to well under a second on an NVIDIA V100 GPU.

Figure 11: Forward-backward inconsistency between transforms when reversing
the order of input images. We compare the mean displacement of voxels upon
subsequent application of both transforms, relative to the voxel size. Lower
values are better.  Figure 12: Relative reduction in brain-specific accuracy
when registering full-head instead of skull-stripped images. Lower values are
better. Although affine Deeds is the only method whose Dice overlap increases,
it still ranks as the least accurate on the GSP$\rightarrow$IXIT1 testset.
Error bars show the standard error of the mean.

###  4.6 Experiment 3: joint registration

Motivated by the affine performance of SynthMorph, we complete the model with
a hypernetwork-powered deformable module to achieve 3D joint affine-deformable
registration (Figure 4). Our focus is on building a complete and readily
usable tool that generalizes across scan protocols while requiring minimal
preprocessing.

####  4.6.1 Setup

First, we compare deformable registration using the held-out image pairs from
separate subjects for each of the datasets of Table 1. The comparison employs
skull-stripped images initialized with affine transforms estimated by
NiftyReg, the most accurate baseline in Figure 9. We compare deformable
SynthMorph performance to classical baselines and VTN, the only joint DL
baseline trained by the original authors that is available to us—we seek to
gauge the accuracy achievable with off-the-shelf algorithms for data unseen at
training.

Second, we analyze robustness to sub-optimal affine initialization. In order
to cover realistic affine inaccuracies and assess the most likely and intended
use case, we repeat the previous experiment, this time initializing each
method with the affine transform obtained with the same method—that is, we
test end-to-end joint registration with each tool. Similarly, we evaluate the
importance of removing non-brain voxels from the input images. In this
experiment, we initialize each method with affine transforms estimated by
NiftyReg from skull-stripped data, and test deformable registration of full-
head images.

Third, we analyze the effect of reducing through-plane resolution
${\Delta}z{}$ on SynthMorph performance compared to classical baselines,
following the steps outlined in Section 4.5. In this experiment, we initialize
each method with affine transforms estimated by NiftyReg from skull-stripped
images, such that the comparison solely reflects deformable registration
accuracy.

Fourth, we analyze warp-field regularity and registration accuracy over
dataset GSP$\rightarrow$IXIT1 as a function of the regularization weight
$\lambda$. We also compare the symmetry of each method with regard to
reversing the order of the input images.

Figure 13:  Deformable 3D registration examples comparing the moved image
$m\circ\phi$ and the deformation field $\phi$ across methods. Each row is an
example from a different dataset. For comparability, we initialize all methods
with NiftyReg’s affine registration.

Figure 14: Deformable 3D registration accuracy as mean Dice scores over the 21
largest brain regions (large-21), 10 fine-grained structures not optimized at
SynthMorph training (small-10), and image similarity. Each violin shows the
distribution across the skull-stripped cross-subject pairs from Table 1. For
comparability, we initialize all deformable tools with affine transforms
estimated by NiftyReg. The asterisk indicates SynthMorph performance without
skull-stripping. Downward arrows show median scores outside the plotted range.
Higher values are better.

####  4.6.2 Results

Figure 13 shows typical deformable registration examples for each method, and
Figure 14 quantitatively compares registration accuracy across testsets in
terms of mean Dice overlap $D$ over the 21 largest anatomical structures
(large-21), 10 fine-grained structures (small-10) not optimized at training,
and image similarity measured with NCC-MIND. Supplemental Figures S.1–S.5 show
deformable registration accuracy across individual brain structures.

Although SynthMorph trains with synthetic images only, it achieves the highest
large-21 score for every skull-stripped testset. For all cross-contrast
pairings and the pediatric testset, SynthMorph leads by at least 1.7 Dice
points compared to the highest baseline score (MASi$\rightarrow$HCP-D,
$p<10^{-22}$ for paired two-sided $t$-test) and often much more. Across these
testsets, SynthMorph performance remains largely invariant, whereas the other
methods except Deeds struggle. Crucially, the distribution of SynthMorph
scores for isotropic data is substantially narrower than the baseline scores,
indicating the absence of gross inaccuracies such as pairs with $D<65$ that
several baselines yield across all isotropic contrast pairings. On the
clinical testset QIN$\rightarrow$IXIT1, SynthMorph surpasses the baselines by
at least ${\Delta}D=5.1$. For GSP$\rightarrow$IXIT1, it outperforms the best
classical baseline ANTs by 1 Dice point ($p<10^{-18}$).

Across the T1w testsets, FNIRT outperforms NiftyReg by several Dice points and
also ANTs for MASi$\rightarrow$HCP-D pairs. Surprisingly, FNIRT beats
NiftyReg’s NMI implementation for GSP$\rightarrow$IXIT2, even though FNIRT’s
cost function targets within-contrast registration. The most robust baseline
is Deeds, which ranks third at adult T1w registration. Its performance reduces
the least for the cross-contrast and clinical testsets, where it achieves the
highest Dice overlap after SynthMorph.

The only joint DL baseline with trained weights that we had access to, VTN,
yields relatively low accuracy across all testsets. This was expected for the
cross-contrast pairings, since the model was trained with T1w data, confirming
the data dependency introduced with standard training. However, VTN lags
behind the worst-performing classical baseline for GSP$\rightarrow$IXIT1 data,
NiftyReg, too (${\Delta}D=2.1$, $p<3\times 10^{-7}$), likely due to domain
shift as in the affine case.

Considering the fine-grained small-10 brain structures held out at training,
SynthMorph consistently ranks among the two best-performing methods, matching
the performance of Deeds for GSP$\rightarrow$IXIT1 (${\Delta}D=0.1$, $p=0.7$)
and GSP$\rightarrow$IXIPD (${\Delta}D=0.2$, $p=0.3$), and leading by at least
${\Delta}D>5$ ($p<3\times 10^{-10}$) on the clinical testset.

Interestingly, SynthMorph outperforms all baselines across testsets in terms
of NCC-MIND ($p<2\times 10^{-17}$), although it is the only method not
optimizing or trained with an image-based loss.

Figure 15 shows the relative change in large-21 Dice for each tool when run
end-to-end compared to affine initialization with NiftyReg. SynthMorph’s drop
in performance is 0.05% or less across all datasets. For
GSP$\rightarrow$IXIT1, classical-baseline accuracy decreases by no more than
$0.3\%$. Across the other datasets, the classical methods generally cannot
make up for the discrepancy between their own and NiftyReg’s affine transform:
accuracy drops by up to $4.9\%$, whereas SynthMorph remains robust. The
performance of VTN reduces by at least $7.3\%$ across testsets and often much
more, highlighting the detrimental effect an inaccurate affine transform can
have on the subsequent deformable step. Figure 12 shows the importance of
skull-stripping for deformable registration accuracy. Generally, deformable
accuracy suffers less than affine registration when switching to full-head
images, as these algorithms can deform different image regions independently.
SynthMorph remains most robust to the change in preprocessing; its large-21
Dice overlap reduces by less than 0.05%. With a drop of 0.08%, Deeds is
similarly robust. In contrast, FNIRT’s performance is most affected and
reduces by 5%—a decline of the same order as for most affine methods.

Figure 16 analyzes SynthMorph warp smoothness. As expected, image-based NCC-
MIND and large-21 Dice accuracy peak for weak regularization of $\lambda<0.2$.
In contrast, overlap of the small-10 regions not optimized at training
benefits from smoother warps, with an optimum at $\lambda=0.4$. The fields
predicted by SynthMorph achieve the lowest log-Jacobian spread across all
baselines for $\lambda>0.4$ and are thus closest to the ideal value 1.
Similarly, the proportion of folding voxels decreases with higher $\lambda$
and drops to $0\%$ at $\lambda=0.5$ (10 integration steps). Deeds is the
weakest-regularized method and yields $6\times 10^{-5}\%$ folding voxels,
whereas the other baselines achieve $0\%$. For realistic warp fields with
characteristics that match or exceed the tested baselines, we conduct all
comparisons in this study with a default weight $\lambda=0.5$. We highlight
that $\lambda$ is an input to SynthMorph, enabling users to choose the optimal
regularization strength for their specific data without retraining.

Deformable registration with SynthMorph is highly symmetric (Figure 11), with
a forward-backward inconsistency of only $1\%$ of the voxel size, that closely
follows ANTs (0.2%) and NiftyReg (0.5%). In contrast, the other methods have
inconsistencies of the order of the voxel size.

Figure 15: Mean decrease in Dice scores for end-to-end joint registration
relative to affine initialization with NiftyReg. Except for adult T1w
registration pairs, the classical tools in blue generally cannot compensate
for the discrepancy between their own and NiftyReg’s affine transform, showing
the devastating effect inaccurate affine initialization can have on subsequent
deformable registration.

Figure 10(b) assesses the dependency of registration performance on slice
thickness ${\Delta}z{}$. Similar to the affine case, deformable accuracy
decreases for thicker slices, albeit faster. SynthMorph performs most
robustly. Its accuracy remains unchanged up to ${\Delta}z{}\leq 3$ mm and
reduces 95% at ${\Delta}z{}=8.5$ mm. ANTs is the most robust classical method,
but its accuracy drops considerably faster than SynthMorph. FLIRT and NiftyReg
are most affected at reduced resolution, performing at less than 95% accuracy
for ${\Delta}z{}\geq 4$ mm and ${\Delta}z{}\geq 4.5$, respectively.

Deformable registration often requires substantially more time than affine
registration (Table 3). On the GPU, SynthMorph takes less than 8 seconds per
image pair for registration, IO, and resampling. One-time model setup requires
about 1 minute, after which the user could register any number of image pairs
without reinitializing the model. On the CPU, the fastest classical method
Deeds requires only about 6 minutes in single-threaded mode, whereas ANTs
takes almost 5 hours. While VTN’s joint runtime is 1 minute, SynthMorph needs
about 15 minutes for deformable registration on a single thread.

Figure 16: Regularization analysis of SynthMorph registration accuracy, the
proportion of folding voxels with a negative Jacobian determinant, and the
spread of the distribution of absolute log-Jacobian determinants as a function
of the regularization weight $\lambda$. The dots indicate maximum accuracy.
For the other metrics, lower values are better.

##  5 Discussion

We present an easy-to-use DL tool for end-to-end affine and deformable brain
registration. SynthMorph achieves robust performance across imaging contrast,
resolution, and pathology, enabling accurate registration for brain scans
right off the scanner without preprocessing. The SynthMorph strategy
alleviates the dependency on acquired training data by generating widely
variable images from anatomical label maps—and there is no need for label maps
at registration time.

###  5.1 Architectures

We performed a rigorous analysis of popular affine architectures. The
comparison shows that Encoder is an excellent network architecture if the
expected transforms are small, especially at a number of cascades $C\geq 4$.
For medium to large transforms, Encoder accuracy suffers. While our
experiments indicate that the reduction in accuracy can be mitigated by
simultaneously optimizing a separate loss for each cascade, doing so
substantially increases training times compared to the other architectures.
Another drawback of Encoder is the image-size dependence introduced by the FC
layer. We find that Detector is a more flexible and robust alternative for
medium to large transforms. A limitation of the 2D network analysis is that
the results may not generalize fully to 3D registration. The comparison only
serves as an indication of the relative strengths and weaknesses of the
architectures. Yet, prior work confirms the robustness of Detector across
large transforms in 3D (Yu et al., 2022).

Vision transformers (Dosovitskiy et al., 2020) are another popular approach to
overcoming the local receptive field of convolutions with small kernel sizes,
querying information across distributed image patches. However, in practice
the sophisticated architecture is often unnecessary for many computer-vision
tasks (Pinto et al., 2022): while simple small-kernel U-Nets generally perform
well as their multi-resolution convolutions effectively widen the receptive
field (Liu et al., 2022b), increasing the kernel size can boost the
performance of convolutional networks beyond that achieved by vision
transformers across multiple tasks (Liu et al., 2022a; Ding et al., 2022).

###  5.2 Anatomy-specific registration

Accurate registration of the specific anatomy of interest requires down-
weighting the contribution of irrelevant image content to the optimization
metric. SynthMorph learns what anatomy is pertinent to the task, as we
optimize the overlap of select labels of interest only. It is likely that the
model learns an implicit segmentation of the image, in the sense that it
focuses on deforming the anatomy of interest, warping the remainder of the
image only to satisfy regularization constraints. In contrast, most classical
and DL methods cannot distinguish between relevant and irrelevant image
features, and thus have to rely on explicit segmentation to remove distracting
content prior to registration, such as skull-stripping in neuroimage
processing (Smith, 2002; Iglesias et al., 2011; Eskildsen et al., 2012; Salehi
et al., 2017; Hoopes et al., 2022b).

SynthMorph does not enforce or assume a hard segmentation: while our
implementation starts with discrete labels, the input could, in principle, be
a set of probability maps indicating the likelihood that each image region
belongs to a particular tissue class. In fact, the interpolation of the one-
hot encoded label maps during training already introduces soft edges that vary
smoothly from 0 outside to 1 inside each structure.

Pathology missing from the training labels does not necessarily hamper overall
registration accuracy, as the experiments with scans from patients with
glioblastoma show. In fact, SynthMorph outperforms all deformable baselines
tested on these data. However, we do not expect these missing structures to be
mapped with high accuracy, in particular if the structure is absent in one of
the test images—this is no different from the behavior of methods optimizing
image similarity. In this work, we train SynthMorph as a general tool for
cross-subject registration. For specialized applications such as tumor
tracking, performance may be limited relative to a dedicated model. Pathology-
specific models could be trained by extending our strategy, adding synthesized
pathology labels to segmentation maps from healthy subjects.

###  5.3 Baseline performance

Networks trained with the SynthMorph strategy do not have access to the MRI
contrasts of the testsets nor in fact to any MRI data at all. Yet SynthMorph
matches or outperforms classical and DL-baseline performance across the real-
world datasets tested, while being substantially faster than the classical
methods. For deformable registration, the fastest classical method Deeds
requires 6 minutes, while SynthMorph takes about 1 minute for one-time model
setup and just under 8 seconds for each subsequent registration. This speed-up
may be particularly useful for processing large datasets like ABCD, enabling
end-to-end registration of hundreds of image pairs per hour—the time that some
established tools like ANTs require for a single registration.

The DL baselines tested have runtimes comparable to SynthMorph. Combining them
with skull-stripping would generally be a viable option for fast brain-
specific registration: brain extraction with a tool like SynthStrip only takes
about 30 seconds. However, we are not aware of any existing DL tool that would
enable deformable registration of unseen data with adjustable regularization
strength without retraining. While the DL baselines break down for contrasts
unobserved at training, they also cannot match the accuracy of classical tools
for the T1w contrast they were trained on, likely due to domain shift.

In contrast, SynthMorph performance is relatively unaffected by changes in
imaging contrast, resolution, or subject population. These results demonstrate
that the SynthMorph strategy produces powerful networks that can register new
image types unseen at training. We emphasize that our focus is on leveraging
the training strategy to build a robust and accurate registration tool. It is
possible that other architectures, such as the trained DL baselines tested in
this work, perform equally well when trained using our strategy.

Although Robust down-weights the contribution of image regions that cannot be
mapped with the linear transformation model of choice, its accuracy dropped by
several points for data without skull-stripping. The poor performance in
cross-contrast registration may be due to the experimental nature of its
robust-entropy cost function. We initially experimented with the recommended
NMI metric, but registration failed for a number of cases as Robust produced
non-invertible matrix transforms, and we hoped that the robust metrics would
deliver accurate results in the presence of non-brain image content—which the
NMI metric cannot ignore during optimization.

###  5.4 Challenges with retraining baselines

Retraining DL baselines to improve performance for specific user data involves
substantial practical challenges. For example, users have to reimplement the
architecture and training setup from scratch if code is not available. If code
is available, the user may be unfamiliar with the specific programming
language or machine-learning library, and building on the original authors’
implementation typically requires setting up an often complex development
environment with matching package versions. In our experience, not all authors
make this version information readily available, such that users may have to
resort to trial and error. Additionally, the user’s hardware might not be on
par with the authors’. If a network exhausts the memory of the user’s GPU,
avoiding prohibitively long training times on the CPU necessitates reducing
model capacity, which can affect performance. In principle, users could
retrain DL methods despite these challenges. However, the burden is usually
sufficiently large that users will turn to methods that distribute pre-trained
models. For this reason, we specifically compare DL baselines pre-trained by
the respective authors, to gauge the performance attainable without
retraining. We hope the broad applicability of SynthMorph may help alleviate
the historically limited reusability of DL methods.

###  5.5 Joint registration

The joint baseline comparison highlights that deformable algorithms cannot
always fully compensate for real-world inaccuracies in affine initialization.
Generally, the median Dice overlap drops by a few percent when we initialize
each tool with affine transforms estimated by the same package instead of
NiftyReg, the most accurate affine baseline tested. This experiment
demonstrates the importance of accurate affine registration for joint
accuracy—choosing affine and deformable algorithms from the same package is
likely the most common use case.

In Section 4.5, the affine subnetwork of the 10-cascade VTN model consistently
ranks among the least accurate methods even for the T1w image type it trained
with. The authors of VTN do not independently compare the affine component to
baselines and instead focus on joint affine-deformable accuracy (Zhao et al.,
2019b, d). While the VTN publication presents the affine cascade as an Encoder
architecture ($C=1$, Section 3.3) terminating with an FC layer (Zhao et al.,
2019d), the public implementation omits the FC layer (Zhao et al., 2019c).
Some of our early experiments with this architecture indicated that the FC
layer is critical to competitive performance.

###  5.6 An ill-posed problem

Compared to within-subject affine registration, cross-subject affine
registration is an ill-defined optimization problem. The human cortex, for
example, exhibits complex folding patterns that vary considerably between
individuals, and transforms limited to translation, rotation, scaling, and
shear can only establish a gross match between subjects. This implies that a
transform maximizing NCC may not necessarily lead to optimal Dice scores, and
vice versa. While the Dice coefficient is certainly not the ultimate measure
of accuracy, many neuroimage registration studies assess anatomical overlap,
since there is little argument where most volumetric borders are. Label-based
losses may have a difficult optimization landscape when images are far apart,
or when structures have no initial overlap. For example, our models did not
converge when augmenting label maps across the full range of rotations
$r_{i}\in[-180^{\circ},180^{\circ}]$ about any axis $i$.

###  5.7 Rotational range

At training, SynthMorph sees registration pairs rotated apart by angles in
excess of $|r_{i}|=90^{\circ}$ about any axis $i$, due to the combined effect
of spatial augmentation (Table B.1) and the rotational offset already present
between any two input label maps. This limits SynthMorph’s rotational range to
just over half of the possible $|r_{i}|=180^{\circ}$ range, considering voxel
data alone. Although prior work (Yu et al., 2022) and the analysis of Section
4.4 demonstrate that with an NCC loss Detector can effectively capture
rotations up to $|r_{i}|=180^{\circ}$, medical applications will not typically
require the full range. For example, the rotational ranges measured within
OASIS and ABCD do not exceed $|r_{i}|\leq 43.1^{\circ}$ (Figure C.1). Even in
situations where rotations up to $|r_{i}|\leq 180^{\circ}$ can occur, the
registration problem reduces to an effective 45∘ range when transposing the
moving image to match the fixed-image orientation based on the orientation
information typically stored in medical image headers.

###  5.8 Degrees of freedom

While we estimate the full affine transformation matrix with 12 DOF in 3D
space, some applications require fewer DOF. An example is the correction of
head motion during neuroimaging with MRI (White et al., 2010; Tisdall et al.,
2012; Gallichan et al., 2016), where the bulk motion and its possible
mitigation through pulse-sequence adjustments are physically constrained to 6
DOF accounting for translation and rotation. Although our evaluation is
limited to affine registration, the Detector architecture used by SynthMorph
could support lower-DOF registration in several ways. First, the WLS problem
(11) has closed-form solutions for various numbers of DOF, and prior work
evaluated a 6-DOF model (Moyer et al., 2021). Second, we can decompose the
affine matrix $\hat{t}$ of Equation (9) into translation, rotation, scaling,
and shearing parameters for each axis of space (Appendix A). Using a modified
transform $\hat{t}_{vr}$ in the loss (6)—reconstructed from translations
$v_{i}$ and rotation parameters $r_{i}$ only—would encourage the network to
detect features optimal for rigid alignment.

###  5.9 Future work

We plan to expand our work in several ways. First, we will provide a trained
6-DOF model for rigid registration, as many applications require translations
and rotations only, and the most accurate rigid transform does not necessarily
correspond to the translation and rotation encoded in the most accurate affine
transform.

Second, we will employ the proposed strategy and affine architecture to train
specialized models for within-subject registration for navigator-based motion
correction of neuroimaging with MRI (White et al., 2010; Tisdall et al., 2012;
Hoffmann et al., 2016; Gallichan et al., 2016). These models need to be
efficient for real-time use but do not have to be invariant to MRI contrast or
resolution when employed to track head-pose changes between navigators
acquired with a fixed protocol. However, the brain-specific registration made
possible by SynthMorph will improve motion tracking and correction in the
presence of jaw movement (Hoffmann et al., 2020).

Third, another application that can dramatically benefit from anatomy-specific
registration is fetal neuroimaging, where the fetal brain is surrounded by
features such as amniotic fluid and maternal tissue. We plan to tackle
registration of the fetal brain, which is challenging, partly due to its small
size, and which currently relies on brain extraction prior to registration to
remove confounding image content (Gaudfernau et al., 2021).

##  6 Conclusion

We present an easy-to-use DL tool for end-to-end registration of images right
off the scanner, without any preprocessing. Our study demonstrates the
feasibility of training accurate affine and joint registration networks that
generalize to image types unseen at training, outperforming established
baselines across a landscape of image contrasts and resolutions. In a rigorous
analysis approximating the diversity of real-world data, we find that our
networks achieve invariance to protocol-specific image characteristics by
leveraging a strategy that synthesizes wildly variable training images from
label maps.

Optimizing the spatial overlap of select anatomical labels enables anatomy-
specific registration without the need for segmentation that removes
distracting content from the input images. We believe this independence from
complex preprocessing has great promise for time-critical applications, such
as real-time motion correction of MRI. Importantly, SynthMorph is a widely
applicable learning strategy for anatomy-aware and acquisition-agnostic
registration of any anatomy with any network architecture, as long as label
maps are available for training—there is no need for these at registration
time.

## Data and code availability

A stand-alone SynthMorph utility and the source code are available at
https://w3id.org/synthmorph. We also distribute SynthMorph as part of the
open-source FreeSurfer package at https://freesurfer.net. The experiments
presented in this study retrospectively analyze public datasets whose original
sources we indicate in Section 4.1.

## Ethics

We retrospectively re-analyze public datasets only. The Mass General Brigham
Internal Review Board (IRB) approved the FSM study and the infant study. The
Committee on Clinical Investigation at Boston Children’s Hospital approved the
infant study. Retrospective analysis of de-identified open-access data did not
require ethical approval. All subjects gave written informed consent.

## Author contributions

Malte Hoffmann worked on conceptualization, data curation, formal analysis,
funding acquisition, investigation, methodology, project administration,
software, writing (original draft, review, editing). Andrew Hoopes assisted
with data curation, methodology, resources, software, validation, writing
(review, editing). Douglas N. Greve helped with data curation, funding
acquisition, resources, validation, writing (review, editing). Bruce Fischl
worked on conceptualization, funding acquisition, investigation, methodology,
resources, software, supervision, writing (review, editing). Adrian V. Dalca
contributed to conceptualization, methodology, resources, software,
supervision, writing (review, editing).

## Acknowledgments

The authors thank Lilla Zöllei and OASIS Cross-Sectional for sharing data (PIs
D. Marcus, R. Buckner, J. Csernansky, J. Morris, NIH awards P50 AG05681, P01
AG03991, P01 AG026276, R01 AG021910, P20 MH071616, U24 R021382). We are
grateful for funding from the National Institute of Biomedical Imaging and
Bioengineering (P41 EB015896, P41 EB030006, R01 EB019956, R01 EB023281, R01
EB033773, R21 EB018907), the National Institute of Child Health and Human
Development (R00 HD101553), the National Institute on Aging (R01 AG016495, R01
AG070988, R21 AG082082, R56 AG064027), the National Institute of Mental Health
(RF1 MH121885, RF1 MH123195), the National Institute of Neurological Disorders
and Stroke (R01 NS070963, R01 NS083534, R01 NS105820, U01 NS132181, UM1
NS132358), and the National Center for Research Resources (S10 RR019307, S10
RR023043, S10 RR023401). We acknowledge additional support from the BRAIN
Initiative Cell Census Network (U01 MH117023) and the NIH Blueprint for
Neuroscience Research (U01 MH093765), part of the multi-institutional Human
Connectome Project. The study also benefited from computational hardware
provided by the Massachusetts Life Sciences Center.

## Declaration of interests

Bruce Fischl has financial interests in CorticoMetrics, a company whose
medical pursuits focus on brain imaging and measurement technologies. Malte
Hoffmann and Bruce Fischl receive salary support from GE HealthCare.
Massachusetts General Hospital and Mass General Brigham manage this interest
in accordance with their conflict of interest policies. The authors have no
other interests that could have inappropriately influenced the work reported
in this paper.

##  Appendix A Affine parameterization

Let $f$ be a fixed $N$D image of side lengths $d_{i}$, where
$i\in\\{1,...,N\\}$ indexes the right-handed axes of the spatial image domain
$\Omega$. This work uses zero-centered index voxel coordinates $x\in\Omega$.
That is,

$\Omega=\prod_{i=1}^{N}{\Big{\\{}-{\Delta}d_{i},1-{\Delta}d_{i},...,d_{i}-1-{\Delta}d_{i}\Big{\\}}}$
(A.1)

with ${\Delta}d_{i}=(d_{i}-1)/2$, placing the center of rotations at the
center of $f$. Let $T:\Omega\rightarrow\mathbb{R}^{N}$ be the affine
coordinate transform of Equation (1), which maps a moving image $m$ onto the
domain of $f$. We parameterize $T$ as

$T=VRZE,$ (A.2)

where $V$, $R$, $Z$, $E$ are matrices describing translation, rotation,
scaling, and shear, respectively. Denoting $v_{i}$ the translation and $z_{i}$
the scaling parameter along axis $i$, we define

$V=\left(\begin{array}[]{ccc|c}&&&v_{1}\\\ &I_{N}&&\vdots\\\ &&&v_{N}\\\
\hline\cr 0&\cdots&0&1\\\ \end{array}\right),$ (A.3)

and

$Z=\left(\begin{array}[]{cccc}z_{1}&&&\\\ &\ddots&&\\\ &&z_{N}&\\\ &&&1\\\
\end{array}\right).$ (A.4)

For rotations and shear, we distinguish between the 2D and 3D case. Let
$r_{i}$ be the angle of rotation about axis $i$, where the direction of
rotation follows the right-hand rule. We abbreviate $c_{i}=\cos(r_{i})$ and
$s_{i}=\sin(r_{i})$.

###  A.1 Two-dimensional case

In 2D, we apply the rotation angle $r=r_{3}$ and shear $e$ using matrices

$R=\left(\begin{array}[]{ccc}c_{3}&-s_{3}&0\\\ s_{3}&c_{3}&0\\\ 0&0&1\\\
\end{array}\right),\quad E=\left(\begin{array}[]{ccc}1&e&0\\\ 0&1&0\\\
0&0&1\\\ \end{array}\right).$ (A.5)

###  A.2 Three-dimensional case

We consider intrinsic 3D rotations represented as the matrix product
$R=R_{1}\,R_{2}\,R_{3}$, where

$R_{1}=\left(\begin{array}[]{cccc}1&0&0&0\\\ 0&c_{1}&-s_{1}&0\\\
0&s_{1}&c_{1}&0\\\ 0&0&0&1\\\ \end{array}\right),$ (A.6)
$R_{2}=\left(\begin{array}[]{cccc}c_{2}&0&s_{2}&0\\\ 0&1&0&0\\\
-s_{2}&0&c_{2}&0\\\ 0&0&0&1\\\ \end{array}\right),$ (A.7)
$R_{3}=\left(\begin{array}[]{cccc}c_{3}&-s_{3}&0&0\\\ s_{3}&c_{3}&0&0\\\
0&0&1&0\\\ 0&0&0&1\\\ \end{array}\right),$ (A.8)

and we apply the shears $\\{e_{i}\\}$ using the parameterization:

$E=\left(\begin{array}[]{cccc}1&e_{1}&e_{2}&0\\\ 0&1&e_{3}&0\\\ 0&0&1&0\\\
0&0&0&1\\\ \end{array}\right).$ (A.9)

###  A.3 Transforming coordinates

With the notation introduced in Equation (1), we transform the coordinates of
an $N$D point

$x=(\begin{matrix}x_{1}&x_{2}&\dots&x_{N}\end{matrix})^{\top}\in\Omega$ (A.10)

as $x^{\prime}=Ax+v$, or, using a single matrix product,

$\left(\begin{array}[]{c}\\\ x^{\prime}\\\ \\\ \hline\cr 1\\\
\end{array}\right)=T\left(\begin{array}[]{c}\\\ x\\\ \\\ \hline\cr 1\\\
\end{array}\right).$ (A.11)

##  Appendix B Generation hyperparameters

Table B.1 lists the generation hyperparameters ranges that SynthMorph training
uses for label-map augmentation and image synthesis.

Table B.1: Uniform hyperparameter sampling ranges $[a,b]$ for synthesizing training images from source segmentation maps. We abbreviate standard deviation (SD), full width at half maximum (FWHM), and field of view (FOV). Hyperparameter | Unit | $a$ | $b$
---|---|---|---
Translation | mm | $-30$ | 30
Rotation | ∘ | $-45$ | 45
Scaling | % | 90 | 110
Shear | % | 90 | 110
Warp sampling SD | mm | 0 | 4
Warp blurring FWHM | mm | 8 | 32
Label intensity mean | a.u. | 0 | 1
Noise intensity SD | % | 3 | 10
Image blurring FWHM | mm | 0 | 8
Bias field sampling SD | % | 0 | 10
Bias field blurring FWHM | mm | 48 | 64
FOV cropping | % | 0 | 20
Downsampling factor | % | 1 | 8
Gamma exponent | – | $0.5$ | 1.5

##  Appendix C Transform analysis

In this supplementary experiment, we analyze the range of typical transforms a
registration tool may have to cope with. We register 1000 distinct and
randomly pooled subject pairs from OASIS and another 1000 pairs from ABCD. The
estimated transformation matrix $T$ decomposes into the translation, rotation,
scaling, and shearing parameters defined in Appendix A.

Figure C.1 presents the range of absolute transformation parameters measured
within OASIS and ABCD, along any axis $i\in\\{1,2,3\\}$ of the space
introduced in Section 4.1. Within OASIS, the mean translations and rotations
are $\overline{|v_{i}|}=(8.5\pm 9.9)$ mm and $\overline{|r_{i}|}=(5.6\pm
6.0)^{\circ}$, respectively ($\pm$ standard deviation, SD).

The average scaling and shearing parameters are $\overline{|z_{i}-1|}=(5.0\pm
3.8)$% and $\overline{|e_{i}|}=(3.3\pm 3.0)$%, respectively. While most
transforms are small, a subset of the OASIS subjects are far apart, leading to
large total ranges of translation $|v_{i}|\leq 61.1$ mm, rotation $|r_{i}|\leq
43.1^{\circ}$, scaling $|z_{i}-1|\leq 22.6$%, and shear $|e_{i}|\leq 22.1$%.
Transforms within ABCD follow a similar distribution.

Therefore, we choose to augment input label maps at training with affine
transforms drawn from the ranges of Table B.1, ensuring that SynthMorph covers
the transformation parameters measured across public datasets.

Figure C.1: Absolute affine transformation range across $n=1000$ registration
pairs randomly selected from either OASIS or ABCD. Each panel pools parameters
relative to all axes $i\in\\{1,2,3\\}$ of 3D space. Black bars indicate median
values. Circles represent parameters farther than 1.5 inter-quartile ranges
from the median.

## References

  * Alfaro-Almagro et al. (2018) Fidel Alfaro-Almagro, Mark Jenkinson, Neal K Bangerter, Jesper LR Andersson, Ludovica Griffanti, Gwenaëlle Douaud, Stamatios N Sotiropoulos, Saad Jbabdi, Moises Hernandez-Fernandez, Emmanuel Vallee, et al.  Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank.  _NeuroImage_ , 166:400–424, 2018.
  * Andersson et al. (2007) Jesper LR Andersson, Mark Jenkinson, Stephen Smith, et al.  Non-linear registration, aka spatial normalisation fmrib technical report tr07ja2.  _FMRIB Analysis Group of the University of Oxford_ , 2(1):e21, 2007.
  * Andrade et al. (2018) Natan Andrade, Fabio Augusto Faria, and Fábio Augusto Menocci Cappabianco.  A practical review on medical image registration: From rigid to deep learning based approaches.  In _2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_ , pages 463–470. IEEE, 2018.
  * Ashburner (2007) John Ashburner.  A fast diffeomorphic image registration algorithm.  _NeuroImage_ , 38(1):95–113, 2007.
  * Ashburner and Friston (2000) John Ashburner and Karl J Friston.  Voxel-based morphometry—the methods.  _NeuroImage_ , 11(6):805–821, 2000.
  * Avants et al. (2008) Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee.  Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain.  _Medical Image Analysis_ , 12(1):26–41, 2008.
  * Avants et al. (2011) Brian B Avants, Nicholas J Tustison, Gang Song, Philip A Cook, Arno Klein, and James C Gee.  A reproducible evaluation of ANTs similarity metric performance in brain image registration.  _NeuroImage_ , 54(3):2033–2044, 2011.
  * Balakrishnan et al. (2019) Guha Balakrishnan, Amy Zhao, Mert Sabuncu, John Guttag, and Adrian V. Dalca.  VoxelMorph: A Learning Framework for Deformable Medical Image Registration.  _IEEE Transactions on Medical Imaging_ , 38(8):1788–1800, 2019.
  * Besl and McKay (1992) Paul J Besl and Neil D McKay.  Method for registration of 3-D shapes.  In _Sensor fusion IV: control paradigms and data structures_ , volume 1611, pages 586–606. SPIE, 1992.
  * Billings et al. (2015) Seth D Billings, Emad M Boctor, and Russell H Taylor.  Iterative most-likely point registration (IMLP): A robust algorithm for computing optimal shape alignment.  _PLOS One_ , 10(3):e0117688, 2015.
  * Billot et al. (2020) Benjamin Billot, Douglas N Greve, Koen Van Leemput, Bruce Fischl, Juan Eugenio Iglesias, and Adrian Dalca.  A Learning Strategy for Contrast-agnostic MRI Segmentation.  In _Medical Imaging with Deep Learning_ , pages 75–93. PMLR, 2020.
  * Bookstein (2001) Fred L Bookstein.  “Voxel-based morphometry” should not be used with imperfectly registered images.  _NeuroImage_ , 14(6):1454–1462, 2001.
  * Cai et al. (2021) Leon Y Cai, Qi Yang, Praitayini Kanakaraj, Vishwesh Nath, Allen T Newton, Heidi A Edmonson, Jeffrey Luci, Benjamin N Conrad, Gavin R Price, Colin B Hansen, et al.  MASiVar: Multisite, multiscanner, and multisubject acquisitions for studying variability in diffusion weighted MRI.  _Magnetic Resonance in Medicine_ , 86(6):3304–3320, 2021.
  * Casey et al. (2018) BJ Casey, Tariq Cannonier, May I Conley, Alexandra O Cohen, Deanna M Barch, Mary M Heitzeg, Mary E Soules, Theresa Teslovich, Danielle V Dellarco, Hugh Garavan, et al.  The adolescent brain cognitive development (ABCD) study: imaging acquisition across 21 sites.  _Developmental Cognitive Neuroscience_ , 32:43–54, 2018.
  * Chaitanya et al. (2019) Krishna Chaitanya, Neerav Karani, Christian F Baumgartner, Anton Becker, Olivio Donati, and Ender Konukoglu.  Semi-supervised and task-driven data augmentation.  In _International Conference on Information Processing in Medical Imaging_ , pages 29–41. Springer, 2019.
  * Chen et al. (2017) Min Chen, Aaron Carass, Amod Jog, Junghoon Lee, Snehashis Roy, and Jerry L Prince.  Cross contrast multi-channel image registration using image synthesis for MR brain images.  _Medical Image Analysis_ , 36:2–14, 2017.
  * Chen et al. (2021) Xu Chen, Yanda Meng, Yitian Zhao, Rachel Williams, Srinivasa R Vallabhaneni, and Yalin Zheng.  Learning Unsupervised Parameter-Specific Affine Transformation for Medical Images Registration.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 24–34. Springer, 2021.
  * Clark et al. (2013) Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al.  The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository.  _Journal of digital imaging_ , 26(6):1045–1057, 2013.
  * Cox (1996) Robert W Cox.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages.  _Computers and Biomedical Research_ , 29(3):162–173, 1996.
  * Cox and Jesmanowicz (1999) Robert W Cox and Andrzej Jesmanowicz.  Real-time 3D image registration for functional MRI.  _Magnetic Resonance in Medicine_ , 42(6):1014–1018, 1999.
  * Dalca et al. (2019a) Adrian Dalca, Marianne Rakic, John Guttag, and Mert Sabuncu.  Learning conditional deformable templates with convolutional networks.  _Advances in Neural Information Processing Systems_ , 32, 2019a.
  * Dalca et al. (2018) Adrian V Dalca, Guha Balakrishnan, John Guttag, and Mert R Sabuncu.  Unsupervised learning for fast probabilistic diffeomorphic registration.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 729–738. Springer, 2018.
  * Dalca et al. (2019b) Adrian V. Dalca, Guha Balakrishnan, John Guttag, and Mert Sabuncu.  Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfaces.  _Medical Image Analysis_ , 57:226–236, 2019b.
  * de Macedo Rodrigues et al. (2015) Katyucia de Macedo Rodrigues, Emma Ben-Avi, Danielle D Sliva, Myong-sun Choe, Marie Drottar, Ruopeng Wang, Bruce Fischl, Patricia E Grant, and Lilla Zöllei.  A FreeSurfer-compliant consistent manual segmentation of infant brains spanning the 0–2 year age range.  _Frontiers in Human Neuroscience_ , 9:21, 2015.
  * de Vos et al. (2017) Bob D de Vos, Floris F Berendsen, Max A Viergever, Marius Staring, and Ivana Išgum.  End-to-end unsupervised deformable image registration with a convolutional neural network.  In _DLMIA_ , pages 204–212. Springer, 2017.
  * De Vos et al. (2019) Bob D De Vos, Floris F Berendsen, Max A Viergever, Hessam Sokooti, Marius Staring, and Ivana Išgum.  A deep learning framework for unsupervised affine and deformable image registration.  _Medical Image Analysis_ , 52:128–143, 2019.
  * Dice (1945) Lee R Dice.  Measures of the amount of ecologic association between species.  _Ecology_ , 26(3):297–302, 1945.
  * Ding et al. (2022) Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding.  Scaling up your kernels to 31x31: Revisiting large kernel design in cnns.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11963–11975, 2022.
  * Ding and Niethammer (2022) Zhipeng Ding and Marc Niethammer.  Aladdin: Joint Atlas Building and Diffeomorphic Registration Learning with Pairwise Alignment.  _arXiv preprint arXiv:2202.03563_ , 2022.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Eppenhof and Pluim (2019) K. A. J. Eppenhof and J. P. W. Pluim.  Pulmonary CT registration through supervised learning with convolutional neural networks.  _IEEE Transactions on Medical Imaging_ , 38(5):1097–1105, 2019.
  * Eskildsen et al. (2012) Simon F Eskildsen, Pierrick Coupé, Vladimir Fonov, José V Manjón, Kelvin K Leung, Nicolas Guizard, Shafik N Wassef, Lasse Riis Østergaard, D Louis Collins, Alzheimer’s Disease Neuroimaging Initiative, et al.  BEaST: brain extraction based on nonlocal segmentation technique.  _NeuroImage_ , 59(3):2362–2373, 2012.
  * Fein et al. (2006) George Fein, Bennett Landman, Hoang Tran, Jerome Barakos, Kirk Moon, Victoria Di Sclafani, and Robert Shumway.  Statistical parametric mapping of brain morphology: sensitivity is dramatically increased by using brain-extracted images as inputs.  _NeuroImage_ , 30(4):1187–1195, 2006.
  * Fischl (2012) Bruce Fischl.  FreeSurfer.  _NeuroImage_ , 62(2):774–781, 2012.  ISSN 1053-8119.  20 YEARS OF fMRI.
  * Fischl et al. (2002) Bruce Fischl, David H Salat, Evelina Busa, Marilyn Albert, Megan Dieterich, Christian Haselgrove, Andre Van Der Kouwe, Ron Killiany, David Kennedy, Shuna Klaveness, et al.  Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain.  _Neuron_ , 33(3):341–355, 2002.
  * Fischl et al. (2004) Bruce Fischl, David H Salat, André JW Van Der Kouwe, Nikos Makris, Florent Ségonne, Brian T Quinn, and Anders M Dale.  Sequence-independent segmentation of magnetic resonance images.  _NeuroImage_ , 23:S69–S84, 2004.
  * Fischmeister et al. (2013) F Ph S Fischmeister, I Höllinger, N Klinger, A Geissler, MC Wurnig, E Matt, J Rath, SD Robinson, S Trattnig, and R Beisteiner.  The benefits of skull stripping in the normalization of clinical fMRI data.  _NeuroImage: Clinical_ , 3:369–380, 2013.
  * Friston et al. (1995) Karl J Friston, John Ashburner, Christopher D Frith, J-B Poline, John D Heather, and Richard SJ Frackowiak.  Spatial registration and normalization of images.  _Human Brain Mapping_ , 3(3):165–189, 1995.
  * Fu et al. (2020) Yabo Fu, Yang Lei, Tonghe Wang, Walter J Curran, Tian Liu, and Xiaofeng Yang.  Deep learning in medical image registration: a review.  _Physics in Medicine & Biology_, 65(20):20TR01, 2020.
  * Gallichan et al. (2016) Daniel Gallichan, José P Marques, and Rolf Gruetter.  Retrospective correction of involuntary microscopic head movement using highly accelerated fat image navigators (3D FatNavs) at 7T.  _Magnetic Resonance in Medicine_ , 75(3):1030–1039, 2016.
  * Gaudfernau et al. (2021) Fleur Gaudfernau, Eléonore Blondiaux, and Stéphanie Allassonière.  Analysis of the Anatomical Variability of Fetal Brains with Corpus Callosum Agenesis.  In _Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis_ , pages 274–283, Cham, 2021. Springer.
  * Gelfand et al. (2005) Natasha Gelfand, Niloy J Mitra, Leonidas J Guibas, and Helmut Pottmann.  Robust global registration.  In _Symposium on Geometry Processing_ , volume 2, page 5. Eurographics, 2005.
  * Glocker et al. (2008) Ben Glocker, Nikos Komodakis, Georgios Tziritas, Nassir Navab, and Nikos Paragios.  Dense image registration through mrfs and efficient linear programming.  _Medical image analysis_ , 12(6):731–741, 2008.
  * Glocker et al. (2011) Ben Glocker, Aristeidis Sotiras, Nikos Komodakis, and Nikos Paragios.  Deformable medical image registration: setting the state of the art with discrete methods.  _Annual review of biomedical engineering_ , 13:219–244, 2011.
  * Greve et al. (2021) Douglas N Greve, Benjamin Billot, Devani Cordero, Andrew Hoopes, Malte Hoffmann, Adrian V Dalca, Bruce Fischl, Juan Eugenio Iglesias, and Jean C Augustinack.  A deep learning toolbox for automatic segmentation of subcortical limbic structures from MRI images.  _NeuroImage_ , 244:118610, 2021.
  * Guo (2019) Courtney K Guo.  _Multi-modal image registration with unsupervised deep learning_.  PhD thesis, Massachusetts Institute of Technology, 2019.
  * Hajnal and Hill (2001) Joseph V Hajnal and Derek LG Hill.  _Medical image registration_.  CRC Press, 2001.
  * Harms et al. (2018) Michael P. Harms et al.  Extending the Human Connectome Project across ages: Imaging protocols for the Lifespan Development and Aging projects.  _NeuroImage_ , 183:972–984, 2018.  ISSN 1053-8119.
  * Heinrich (2019) Mattias P Heinrich.  Closing the gap between deep and conventional image registration using probabilistic dense displacement networks.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part VI 22_ , pages 50–58. Springer, 2019.
  * Heinrich et al. (2012) Mattias P Heinrich, Mark Jenkinson, Manav Bhushan, Tahreema Matin, Fergus V Gleeson, Michael Brady, and Julia A Schnabel.  MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration.  _Medical Image Analysis_ , 16(7):1423–1435, 2012.
  * Heinrich et al. (2013) Mattias P Heinrich, Mark Jenkinson, Michael Brady, and Julia A Schnabel.  MRF-based deformable registration and ventilation estimation of lung CT.  _IEEE Transactions on Medical Imaging_ , 32(7):1239–1248, 2013.
  * Hellier et al. (2001) Pierre Hellier, Christian Barillot, Etienne Mémin, and Patrick Pérez.  Hierarchical estimation of a dense deformation field for 3-D robust registration.  _IEEE Transactions on Medical Imaging_ , 20(5):388–402, 2001.
  * Hoffmann et al. (2015) Malte Hoffmann, T Adrian Carpenter, Guy B Williams, and Stephen J Sawiak.  A survey of patient motion in disorders of consciousness and optimization of its retrospective correction.  _Magnetic Resonance Imaging_ , 33(3):346–350, 2015.
  * Hoffmann et al. (2016) Malte Hoffmann, Marius Mada, T Adrian Carpenter, Stephen J Sawiak, and Guy B Williams.  Additional sampling directions improve detection range of wireless radiofrequency probes.  _Magnetic Resonance in Medicine_ , 76(3):913–918, 2016.
  * Hoffmann et al. (2020) Malte Hoffmann, Robert Frost, David Salat, M Dylan Tisdall, Jonathan Polimeni, and André van der Kouwe.  Real-time brain masking algorithm improves motion tracking accuracy in scans with volumetric navigators (vNavs).  In _Proceedings of the 28th Annual Meeting of ISMRM_ , page 3367. ISMRM, 2020.
  * Hoffmann et al. (2021a) Malte Hoffmann, Benjamin Billot, Douglas N Greve, Juan Eugenio Iglesias, Bruce Fischl, and Adrian V Dalca.  SynthMorph: learning contrast-invariant registration without acquired images.  _IEEE Transactions on Medical Imaging_ , 2021a.
  * Hoffmann et al. (2021b) Malte Hoffmann, Benjamin Billot, Juan E Iglesias, Bruce Fischl, and Adrian V Dalca.  Learning MRI contrast-agnostic registration.  In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , pages 899–903. IEEE, 2021b.
  * Hoffmann et al. (2023) Malte Hoffmann, Andrew Hoopes, Bruce Fischl, and Adrian V Dalca.  Anatomy-specific acquisition-agnostic affine registration learned from fictitious images.  In _Medical Imaging 2023: Image Processing_ , volume 12464, page 1246402. SPIE, 2023.
  * Holmes et al. (2015) Avram J Holmes, Marisa O Hollinshead, Timothy M O’keefe, Victor I Petrov, Gabriele R Fariello, Lawrence L Wald, Bruce Fischl, Bruce R Rosen, Ross W Mair, Joshua L Roffman, et al.  Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measures.  _Scientific Data_ , 2(1):1–16, 2015.
  * Hoopes et al. (2022a) Andrew Hoopes, Malte Hoffmann, Douglas N Greve, Bruce Fischl, John Guttag, Adrian V Dalca, et al.  Learning the effect of registration hyperparameters with hypermorph.  _Machine Learning for Biomedical Imaging_ , 1(IPMI 2021 special issue):1–30, 2022a.
  * Hoopes et al. (2022b) Andrew Hoopes, Jocelyn S Mora, Adrian V Dalca, Bruce Fischl, and Malte Hoffmann.  SynthStrip: skull-stripping for any brain image.  _NeuroImage_ , 260:119474, 2022b.  ISSN 1053-8119.  doi: https://doi.org/10.1016/j.neuroimage.2022.119474.
  * Hu et al. (2018) Yipeng Hu, Marc Modat, Eli Gibson, Nooshin Ghavami, Ester Bonmati, Caroline M Moore, Mark Emberton, J Alison Noble, Dean C Barratt, and Tom Vercauteren.  Label-driven weakly-supervised learning for multimodal deformable image registration.  In _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_ , pages 1070–1074. IEEE, 2018.
  * Iglesias et al. (2011) Juan Eugenio Iglesias, Cheng-Yi Liu, Paul M Thompson, and Zhuowen Tu.  Robust brain extraction across datasets and comparison with publicly available methods.  _IEEE Transactions on Medical Imaging_ , 30(9):1617–1634, 2011.
  * Iglesias et al. (2013) Juan Eugenio Iglesias, Ender Konukoglu, Darko Zikic, Ben Glocker, Koen Van Leemput, and Bruce Fischl.  Is synthesizing MRI contrast useful for inter-modality analysis?  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 631–638. Springer, 2013.
  * Imperial College London (2015) Imperial College London.  IXI Dataset, 2015.  URL https://brain-development.org/ixi-dataset/.
  * Jenkinson and Smith (2001) Mark Jenkinson and Stephen Smith.  A global optimisation method for robust affine registration of brain images.  _Medical Image Analysis_ , 5(2):143–156, 2001.
  * Jenkinson et al. (2012) Mark Jenkinson, Christian F Beckmann, Timothy EJ Behrens, Mark W Woolrich, and Stephen M Smith.  Fsl.  _NeuroImage_ , 62(2):782–790, 2012.
  * Jiang et al. (1995) Aiping Jiang, David N Kennedy, John R Baker, Robert M Weisskoff, Roger BH Tootell, Roger P Woods, Randall R Benson, Kenneth K Kwong, Thomas J Brady, Bruce R Rosen, et al.  Motion detection and correction in functional MR imaging.  _Human Brain Mapping_ , 3(3):224–235, 1995.
  * Kamnitsas et al. (2017) Konstantinos Kamnitsas, Christian Baumgartner, Christian Ledig, Virginia Newcombe, Joanna Simpson, Andrew Kane, David Menon, Aditya Nori, Antonio Criminisi, Daniel Rueckert, et al.  Unsupervised domain adaptation in brain lesion segmentation with adversarial networks.  In _International Conference on Information Processing in Medical Imaging_ , pages 597–609. Springer, 2017.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Klein et al. (2009) Arno Klein, Jesper Andersson, Babak A Ardekani, John Ashburner, Brian Avants, Ming-Chang Chiang, Gary E Christensen, D Louis Collins, James Gee, Pierre Hellier, et al.  Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration.  _NeuroImage_ , 46(3):786–802, 2009.
  * Kneeland et al. (1986) JB Kneeland, A Shimakawa, and FW Wehrli.  Effect of intersection spacing on MR image contrast and study time.  _Radiology_ , 158(3):819–822, 1986.
  * Krebs et al. (2017) Julian Krebs, Tommaso Mansi, Hervé Delingette, Li Zhang, Florin C Ghesu, Shun Miao, Andreas K Maier, Nicholas Ayache, Rui Liao, and Ali Kamen.  Robust non-rigid registration through agent-based action learning.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 344–352, 2017.
  * Krebs et al. (2019) Julian Krebs, Hervé Delingette, Boris Mailhé, Nicholas Ayache, and Tommaso Mansi.  Learning a probabilistic model for diffeomorphic registration.  _IEEE Transactions on Medical Imaging_ , 38(9):2165–2176, 2019.
  * Li and Fan (2017) Hongming Li and Yong Fan.  Non-rigid image registration using fully convolutional networks with deep self-supervision.  _arXiv preprint arXiv:1709.00799_ , 2017.
  * Liu et al. (2022a) Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang.  More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity.  _arXiv preprint arXiv:2207.03620_ , 2022a.
  * Liu et al. (2022b) Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11976–11986, 2022b.
  * Lorenzi et al. (2013) M. Lorenzi, N. Ayache, G.B. Frisoni, and X. Pennec.  LCC-Demons: A robust and accurate symmetric diffeomorphic registration algorithm.  _NeuroImage_ , 81:470–483, 2013.  ISSN 1053-8119.
  * Lowe (2004) David G Lowe.  Distinctive image features from scale-invariant keypoints.  _International Journal of Computer Vision_ , 60(2):91–110, 2004.
  * Machado et al. (2018) Inês Machado, Matthew Toews, Jie Luo, Prashin Unadkat, Walid Essayed, Elizabeth George, Pedro Teodoro, Herculano Carvalho, Jorge Martins, Polina Golland, et al.  Non-rigid registration of 3D ultrasound for neurosurgery using automatic feature detection and matching.  _International Journal of Computer Assisted Radiology and Surgery_ , 13(10):1525–1538, 2018.
  * Maes et al. (1997) Frederik Maes, Andre Collignon, Dirk Vandermeulen, Guy Marchal, and Paul Suetens.  Multimodality image registration by maximization of mutual information.  _IEEE Transactions on Medical Imaging_ , 16(2):187–198, 1997.
  * Mamonov and Kalpathy-Cramer (2016) AB Mamonov and J Kalpathy-Cramer.  Data From QIN GBM Treatment Response, 2016.
  * Marcus et al. (2007) Daniel S Marcus, Tracy H Wang, Jamie Parker, John G Csernansky, John C Morris, and Randy L Buckner.  Open access series of imaging studies (oasis): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults.  _Journal of Cognitive Neuroscience_ , 19(9):1498–1507, 2007.
  * Matas et al. (2004) Jiri Matas, Ondrej Chum, Martin Urban, and Tomás Pajdla.  Robust wide-baseline stereo from maximally stable extremal regions.  _Image and Vision Computing_ , 22(10):761–767, 2004.
  * Meyer et al. (1995) Charles R Meyer, Gregg S Leichtman, James A Brunberg, Richard L Wahl, and Leslie E Quint.  Simultaneous usage of homologous points, lines, and planes for optimal, 3-D, linear registration of multimodality imaging data.  _IEEE Transactions on Medical Imaging_ , 14(1):1–11, 1995.
  * Miller et al. (2016) Karla L Miller, Fidel Alfaro-Almagro, Neal K Bangerter, David L Thomas, Essa Yacoub, Junqian Xu, Andreas J Bartsch, Saad Jbabdi, Stamatios N Sotiropoulos, Jesper LR Andersson, et al.  Multimodal population brain imaging in the uk biobank prospective epidemiological study.  _Nature neuroscience_ , 19(11):1523–1536, 2016.
  * Milletari et al. (2016) Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In _2016 4th International Conference on 3D Vision (3DV)_ , pages 565–571. IEEE, 2016.
  * Modat et al. (2014) Marc Modat, David M Cash, Pankaj Daga, Gavin P Winston, John S Duncan, and Sébastien Ourselin.  Global image registration using a symmetric block-matching approach.  _Journal of Medical Imaging_ , 1(2):024003, 2014.
  * Mok and Chung (2022) Tony C. W. Mok and Albert C. S. Chung.  Affine Medical Image Registration with Coarse-to-Fine Vision Transformer.  _arXiv preprint arXiv:2203.15216_ , 2022.
  * Moyer et al. (2021) Daniel Moyer, Esra Abaci Turk, P Ellen Grant, William M Wells, and Polina Golland.  Equivariant Filters for Efficient Tracking in 3D Imaging.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 193–202. Springer, 2021.
  * Myronenko and Song (2010) Andriy Myronenko and Xubo Song.  Point set registration: Coherent point drift.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 32(12):2262–2275, 2010.
  * Nestares and Heeger (2000) Oscar Nestares and David J Heeger.  Robust multiresolution alignment of MRI brain volumes.  _Magnetic Resonance in Medicine_ , 43(5):705–715, 2000.
  * Oliveira and Tavares (2014) Francisco PM Oliveira and Joao Manuel RS Tavares.  Medical image registration: a review.  _Computer Methods in Biomechanics and Biomedical Engineering_ , 17(2):73–93, 2014.
  * Ou et al. (2011) Yangming Ou, Aristeidis Sotiras, Nikos Paragios, and Christos Davatzikos.  Dramms: Deformable registration via attribute matching and mutual-saliency weighting.  _Medical image analysis_ , 15(4):622–639, 2011.
  * Ou et al. (2014) Yangming Ou, Hamed Akbari, Michel Bilello, Xiao Da, and Christos Davatzikos.  Comparative evaluation of registration algorithms in different brain databases with varying difficulty: results and insights.  _IEEE Transactions on Medical Imaging_ , 33(10):2039–2065, 2014.
  * Perez and Wang (2017) Luis Perez and Jason Wang.  The effectiveness of data augmentation in image classification using deep learning.  _arXiv preprint arXiv:1712.04621_ , 2017.
  * Pinto et al. (2022) Francesco Pinto, Philip HS Torr, and Puneet K Dokania.  An impartial take to the cnn vs transformer robustness contest.  In _European Conference on Computer Vision_ , pages 466–480. Springer, 2022.