# Surface Analysis with Vision Transformers

Simon Dahan
King’s College London
<EMAIL_ADDRESS>Logan Z. J. Williams
King’s College London
<EMAIL_ADDRESS>Abdulah Fawaz
King’s College London
<EMAIL_ADDRESS>Daniel Rueckert
Imperial College London
<EMAIL_ADDRESS>Emma C. Robinson
King’s College London
<EMAIL_ADDRESS>
###### Abstract

The extension of convolutional neural networks (CNNs) to non-Euclidean
geometries has led to multiple frameworks for studying manifolds. Many of
those methods have shown design limitations resulting in poor modelling of
long-range associations, as the generalisation of convolutions to irregular
surfaces is non-trivial. Recent state-of-the-art performance of Vision
Transformers (ViTs) demonstrates that a general-purpose architecture, which
implements self-attention, could replace the local feature learning operations
of CNNs. Motivated by the success of attention-modelling in computer vision,
we extend ViTs to surfaces by reformulating the task of surface learning as a
sequence-to-sequence problem and propose a patching mechanism for surface
meshes. We validate the performance of the proposed Surface Vision Transformer
(SiT) on two brain age prediction tasks in the developing Human Connectome
Project (dHCP) dataset and investigate the impact of pre-training on model
performance. Experiments show that the SiT outperforms many surface CNNs,
while indicating some evidence of general transformation invariance. Code
available at: https://github.com/metrics-lab/surface-vision-transformers

##  1 Introduction

Studying surfaces and manifolds is critical for many applications in computer
graphics [38, 41, 34], protein interaction [19, 39] and biomedical analysis
including cardiac [37, 55], brain [18, 21, 22, 45, 44, 14] imaging, with
applications in biophysical and shape modelling [8, 20, 54], and mapping of
cortical organisation [22, 14, 53]. While the shapes of meshes may vary
greatly, ultimately, all problems may be reduced to analysis of functions over
tessellated, deformable meshes. Despite this, there is no unified geometric
deep learning method (gDL) for studying all these problems, and many gDL
frameworks would in principle be suitable for estimating surface convolutions
[3, 41, 7, 12]. A recent work [17] benchmarked various surface CNN methods on
cortical phenotype regression and segmentation, and showed that they typically
involve trade-offs between computational complexity, feature expressivity, and
rotational equivariance of the models.

While the locality and weight sharing properties of the convolution operation
have pushed forward the computer vision field, especially in natural imaging,
and created sample-efficient architectures that can generalise to a broad
range of tasks [23, 33, 50], these inductive biases towards locality and
scale-invariance in CNN architectures also induce a limited receptive field
that impairs the modelling of long-range spatial dependencies between distant
parts of an image [58, 47]. This prevents CNNs from efficiently modelling
processes that are diffuse in space and/or time; something that is known to be
true for a wide range of applications, specifically for biomedical
applications where conditions may span over large areas [28, 40, 14].

Recently, Dosovitskiy et al. proposed the Vision Transformer (ViT), which
sought to extend the use of self-attention transformer architectures, used in
Natural Language Processing (NLP), to imaging data, by treating computer
vision tasks as a sequence-to-sequence learning problem. By doing so, they
showed that a general-purpose transformer architecture [52] could be used for
natural image classification; thereby demonstrating the benefits of using
self-attention (SA) on image patches to improve modelling of global-context
without relying on strong spatial priors. Subsequent modifications to this
vanilla Transformer [52] have improved the architecture for vision
applications: by modelling contextual information at different scales [24, 6],
revisiting self-attention with regional or local attention [35, 5], or re-
introducing some inductive biases [11]. Such methods have returned state-of-
the-art performances for many image or video understanding tasks [51, 35, 60,
32, 5], where some of this success may be attributed to the scalability of ViT
to be pre-trained on very large datasets [15], but also to efficient
(pre-)training schemes [51, 27, 4, 1].

Similarly, transformer model [52] and self-attention modules have been adapted
to improve the context-modelling over gDL methods for non-Euclidean manifolds:
in point-clouds [16, 42], shape [56], or graphs networks [57, 29].

In this paper, we propose the Surface Vision Transformer (_SiT_), a
methodology for modelling functions on surfaces by extending the ViT to
surface meshes through proposing a mechanism for surface patching. To do so,
surface data are projected onto a sphere and patched using a regular
icospheric tessellation. This reformulates any surface learning task that can
adapt to genus-zero surfaces as a sequence-to-sequence problem. We validate
this methodology for two cortical phenotype regression tasks, against number
of geometric deep learning (gDL) methods, benchmarked in [17] on cortical
phenotype regression and segmentation. The key contributions of this paper are
as follows:

  * •

We introduce a framework for sequence-to-sequence modelling of surfaces, which
patch surfaces via projection to a regularly tessellated icosphere.

  * •

Surface Vision Transformers (_SiT_) are compared against geometric CNNs,
benchmarked in [17], and demonstrate superior performance for regression of
developmental phenotypes.

  * •

_SiT_ also exhibits some degree of transformation invariance by performing
closely on registered and unregistered scans, without incorporating strong
inductive bias in the architecture.

##  2 Methods

Figure 1: The cortical data is first resampled (a), using barycentric
interpolation, from its template resolution (32492 vertices) to a sixth order
icosphere (mesh of 40962 equally spaced vertices). The regular icosphere is
divided into triangular patches of equal vertex count (b, c) that fully cover
the sphere (not shown), which are flattened into feature vectors (d), and then
fed into the transformer model.

###  2.1 Architecture

The _SiT_ model translates surface understanding to a sequence-to-sequence
learning task by reshaping the high-resolution grid of the input domain $X$,
into a sequence of $N$ flattened patches
$\widetilde{X}=\left[\widetilde{X}^{(0)}_{1},...,\widetilde{X}^{(0)}_{N}\right]\in\mathbb{R}^{N\times(VC)}$
(V vertices, C channels). These are first projected onto a $D-$dimensional
sequence
$X^{(0)}=\left[X^{(0)}_{1},...,X^{(0)}_{N}\right]\in\mathbb{R}^{N\times D}$,
using a trainable linear layer. Then, an extra $D$-dimensional token for
regression is concatenated ($X^{(0)}_{0}$), and a positional embedding
($E_{pos}\in\mathbb{R}^{(N+1)\times D}$) is added, such that the input
sequence of the transformer becomes
$X^{(0)}=\left[X^{(0)}_{0},...,X^{(0)}_{N}\right]+E_{pos}$ (see Fig 1(b-e)).
The sequence of embeddings is then processed by a vanilla Transformer encoder
as in [15] with consecutive transformer encoder blocks of Multi-Head Self-
Attention (MHSA) and Feed Forward Network (FFN) layers, with residual layers
in-between. The architecture of the _SiT_ is illustrated in Figure 1. Here,
the proposed _SiT_ model builds upon two variants of the data efficient image
transformer or DeiT [51]: _DeiT-Tiny_ , _DeiT-Small_ , adapted into smaller
versions from the vanilla Vision Transformer (ViT) [15]. A number of $L=12$
layers or transformer encoder blocks is used for both SiT versions; however,
they differ in their number of heads, hidden size or embedding dimension $D$,
and in the number of neurons (_MLP size_) in the FFN, details in Table 1.

###  2.2 Surface Patching

The _SiT_ can generate patches from any regularly tessellated reference grid
that supports down-sampling. For the cortical surface, this is achieved by
imposing a low-resolution triangulated grid on the input mesh, using a
regularly tessellated icosphere (Fig 1(b)).

Here, cortical surface data were first projected to a regularly-tessellated
sphere (with 32,492 vertices) as part of the dHCP structural pipeline [36].
Spherical data were then resampled onto a regularly-tessellated sixth-order
icosahedron with 40,962 vertices, then split into triangular patches where
each patch corresponds to all data points within one face of a second-order
icosphere (153 vertices per patch). The sequence is thus made of 320 non-
overlapping patches that only share common edges (Fig.1 (a-c)).

###  2.3 Optimisation

To mitigate the lack of inductive biases in the architecture, transformers
typically require large training datasets or efficient (pre-)training
strategies [15, 1, 51, 49]. Therefore, we explore techniques for improving
model generalisation in a context of a neuroimaging dataset of limited size:
specifically pre-training and augmentation.

#### Pre-training

is relevant for biomedical imaging tasks, as datasets are usually smaller than
in natural imaging, and can benefit from pre-training before transferring to
downstream tasks. In this paper we evaluate different training strategies: 1)
training from scratch; 2) initialising from ImageNet weights (to support
training on small datasets through incorporation of some spatial priors) and
3) fine-tuning after BERT-like pretraining, a well-known self-supervised
pretraining strategy. For ImageNet, we used pretrained models from the timm
open-source library111pretrained models on ImageNet available at
http://github.com/rwightman/pytorch-image-models/, where models were
pretrained on ImageNet2012 (1 million images, 1000 classes) on patches of size
$16\times 16\times 3$. Self-supervision is implemented as a _masked patch
prediction_ (MPP) task, following the approach proposed in BERT [13], which
consists of corrupting at random some input patches in the sequence; then
training the network to learn how to reconstruct the full corrupted patches.
In this setting, we corrupt at random 50% of the input patches, either
replacing them with a learnable mask token (80%), another patch embedding from
the sequence at random (10%) or keeping their original embeddings (10%). To
optimise the reconstruction, the mean square error (MSE) loss is computed only
for the patches in the sequence that were masked.

#### Data augmentation

Following previous work [10], we additionally propose to augment the
icosahedral patch selection by implementing
$\pm\left\\{5^{\circ},10^{\circ},15^{\circ},20^{\circ},25^{\circ},30^{\circ}\right\\}$
rotations of the sphere before patching around one of the x,y,z axes. Dropout
was also used before the transformer encoder and inside the FFN networks,
compared to [10].

Models | Layers | Heads |  Hidden size D | MLP size | Params.
---|---|---|---|---|---
SiT-Tiny | 12 | 3 | 192 | 768 | 5.5M
SiT-Small | 12 | 6 | 384 | 1536 | 21.6M
Table 1: Architectures of _SiT-tiny_ and _SiT-small_ , inspired by DeiT models
in [51]

##  3 Experiments & Results

Methods | Pretraining | PMA | GA - deconfounded | Average
---|---|---|---|---
Template | Native | Avg | Template | Native | Avg
S2CNN | ✗ |  0.63 $\pm 0.02$ |  0.73 $\pm 0.25$ | 0.68 |  1.35 $\pm 0.68$ |  1.52 $\pm 0.60$ | 1.44 | 1.06
ChebNet | ✗ |  0.59 $\pm 0.37$ |  0.77 $\pm 0.49$ | 0.68 |  1.57 $\pm 0.15$ |  1.70 $\pm 0.36$ | 1.64 | 1.16
GConvNet | ✗ |  0.75 $\pm 0.13$ |  0.75 $\pm 0.26$ | 0.75 |  1.77 $\pm 0.26$ |  2.30 $\pm 0.74$ | 2.04 | 1.39
Spherical UNet | ✗ |  0.57 $\pm 0.18$ |  0.87 $\pm 0.50$ | 0.75 |  0.85 $\pm 0.17$ |  2.16 $\pm 0.57$ | 1.51 | 1.11
MoNet | ✗ |  0.57 $\pm 0.02$ |  0.61 $\pm 0.05$ | 0.59 |  1.44 $\pm 0.08$ |  1.58 $\pm 0.06$ | 1.51 | 1.05
SiT-tiny | ✗ |  0.63 $\pm 0.01$ |  0.77 $\pm$0.03 | 0.70 |  1.17 $\pm$0.04 |  1.36 $\pm$0.01 | 1.27 | 0.98
SiT-tiny | ImageNet |  0.67 $\pm$0.02 |  0.70 $\pm$0.04 | 0.69 |  1.11 $\pm$0.02 |  1.20 $\pm$0.10 | 1.16 | 0.92
SiT-tiny | MPP |  0.58 $\pm$0.01 |  0.64 $\pm$0.06 | 0.61 |  1.03 $\pm$0.09 |  1.31 $\pm$0.01 | 1.17 | 0.89
SiT-small | ✗ |  0.60 $\pm$0.02 |  0.75 $\pm$0.01 | 0.68 |  1.14 $\pm$0.05 |  1.22 $\pm$0.04 | 1.18 | 0.93
SiT-small | ImageNet |  0.59 $\pm$0.03 |  0.71 $\pm$0.02 | 0.65 |  1.13 $\pm$0.03 |  1.30 $\pm$0.08 | 1.22 | 0.93
SiT-small | MPP |  0.55 $\pm$0.04 |  0.63 $\pm$0.06 | 0.59 |  1.02 $\pm$0.06 |  1.21 $\pm$0.12 | 1.12 | 0.85
Table 2: Results of _SiT-tiny_ and _SiT-small_ models for the task of PMA and
GA on template and native space, for three training configurations. Best MAE
on the test set and standard deviations (over three trainings) are reported.
Two configurations of the data were used: _template_ where data are aligned
and _native_ (unregistered).

We evaluate the performance of _SiT_ on two challenging tasks using neonatal
cortical surface data from the developing Human Connectome Project (dHCP)
[26]; 1) prediction of postmenstrual age at scan (PMA), and 2) gestational age
at birth (GA). Stochastic gradient descent (SGD) with momentum was used for
model optimisation, compared to Adam optimisation for gDL models [17]. _SiT_
models were trained for 2000 iterations from scratch and only 1000 iterations
following pre-training as convergence appeared to be faster. All experiments
were run on a single NVIDIA RTX3090 24GB GPU. A batch size of 256 was used for
_SiT-tiny_ and 128 for _SiT-small_.

#### Data & Training

Data for this experiment corresponds to cortical surface data from the third
release of the developing Human Connectome Project (dHCP) [26]. Surfaces were
extracted from T2- and T1-weighted Magnetic Resonance Imaging (MRI) scans
using the dHCP structural pipeline [36, 31, 46, 26, 9]. Four cortical surface
metrics were used: sulcal depth, curvature, cortical thickness and T1w/T2w
ratio (intracortical myelination). Data were registered using Multimodal
Surface Matching [45, 44] to the left-right symmetric dHCP spatiotemporal
cortical atlas [2, 53].

A total of 588 images were included, acquired from term (born $\geq 37$ weeks
gestational age, GA) and preterm (born $<37$ weeks GA) neonatal subjects,
scanned between 24 and 45 weeks postmenstrual age (PMA). Some of the preterm
neonates were scanned twice: once after birth and again around term-equivalent
age. The proposed framework was benchmarked on two phenotype regression tasks:
prediction of postmenstrual age (PMA) at scan, and gestational age (GA) at
birth, where since the objective was to model PMA and GA as markers of healthy
development, all preterms’ second scans were excluded from the PMA prediction
task, and all first scans were excluded from the GA prediction task. This
resulted in 530 neonatal subjects for the PMA prediction task (419 term/111
preterm), and 514 subjects (419 term/95 preterm) for the GA prediction task.
The dHCP dataset is heavily unbalanced with more term babies than preterm
babies. In extension to previous work [10], this class imbalance was addressed
by adapting sampling during training. Subjects were split into 3 categories,
which reflect the clinical subcategories of preterm birth [48]: over 37 weeks,
between 32 and 37 and below 32 weeks. The original ratio of examples in each
of these three categories was 1/7/11. Experiments were run on both _template_
-aligned data and unregistered (_native_) data, and train/test/validation
splits parallel those used in [17].

Changes to cortical organisation are implicated in numerous neurological and
developmental disorders [43, 25]. Such disorders are diffuse processes,
heterogeneous between individuals and populations and cannot be studied
effectively using traditional approaches (based on spatial normalisation to
global average template) since human brains vary in ways that violate the
assumptions of traditional image registration [22], and thereby limit the
sensitivity of population-based comparisons. This motivates the use of _SiT_
as an attention-modelling tool for cortical analysis on two neonatal imaging
tasks that exhibit high variability in cortical development between subjects.

#### Deconfounding strategy

The task of GA prediction is arguably more complicated than the PMA task, as
it is run on scans acquired around term-equivalent age (37-45 weeks PMA) for
both term and preterm neonates, and therefore is highly correlated to PMA at
scan. Here, a deconfounding strategy was employed, following [10], where the
scan age information was incorporated into the patch sequence by adding an
extra embedding to all patches in the sequence before the transformer encoder.
This was implemented using a fully connected network to project scan age to a
vector embedding of dimension D after batch-normalisation.

#### Results

The proposed _SiT_ models were compared against the best performing surface
CNNs reported in [17]: Spherical U-Net [59], MoNet [38], GConvNet [30],
ChebNet [12] and S2CNN [7] (Table 2). We should stress that these gDL models
were trained with both rotational and non-linear data augmentations [17].

Overall, _SiT-small_ and _SiT-tiny_ configurations consistently outperformed
three of the gDL methods (S2CNN, GConvNet, and ChebNet) for all tasks. On
average, the 6 _SiT_ configurations achieved prediction errors below 0.98 MAE,
compared to 1.05 MAE for the best gDL model on average: MoNet. Best
performance overall (0.85 MAE across tasks) was obtained with _SiT-small_ pre-
trained with MPP, with large improvement for GA prediction: 1.12 MAE (on
average template & native) against 1.44 MAE for S2CNN.

For the task of PMA, _SiT-small_ pretrained obtained performances on template
and native data (0.55/0.63) comparable to the best gDL model MoNet: 0.57/0.61.
The use of dropout and rotation augmentation did not seem to improve the
performances of _SiT_ for the task of PMA, which already achieved good results
without regularisation, whereas augmentation (specifically rotations
$\pm\left\\{5^{\circ},10^{\circ}\right\\}$) and dropout greatly improved
_SiTs’_ performance for GA prediction where _SiT_ models outperformed gDL
methods for all native configuration and template configuration (except for
Spherical UNet-template that under-performs greatly in native space).

Across all tasks, _SiTs_ demonstrate consistent performance across training
runs with smaller variability compared to surface CNNs. The methodology also
demonstrates robustness between template-aligned and native data, dropping
less in performance than some gDL methods, such as Spherical UNet which
obtained 0.85 MAE on GA-template but does not build rotational equivariance
(2.16 MAE on GA-native). All _SiTs_ also outperform MoNet (the best gDL
method) for both GA-template and GA-native, which although rotationally
equivariant and consistent between native and template, learns less expressive
convolutional filters (parameterised as a mixture of Gaussians). Finally, pre-
training generally improves performances of _SiTs_ compared to training from
scratch. This is the case for all _SiTs_ trained following the MPP self-
supervision task, and for 6/8 configurations of _SiTs_ following ImageNet
initialisation, but with slighter improvements in the later case.

##  4 Discussion

In this paper, we demonstrated that surface understanding is possible with
vision transformers. This was obtained by introducing a patching methodology
for surface data that can be projected onto a spherical manifold. The _SiT_
methodology surpasses in performance many geometric deep learning methods in
the context of cortical analysis, showing some degree of transformation
invariance with far less drop in performance on unregistered data than most
performing gDL frameworks, and greatly improved by training with
augmentations, comparatively to [10].

The use of vision transformers constitutes an exciting opportunity for many
surface learning applications, especially in the context of biomedical data to
study diffuse processes in cardiac [37], or neurodevelopmental modelling [14];
and where surface deep learning models are usually limited by the receptive
field of convolution operations. Various improvement of the method could be
explored as the _SiT_ only employs a vanilla Transformer encoder [52]. Latest
developments around multi-scale feature learning in ViT [24, 6, 35, 5] would
further benefit the context-modelling of cortical surface, as new
(pre-)training schemes [51, 49].

## References

  * [1] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  6 2021.
  * [2] Jelena Bozek, Antonios Makropoulos, Andreas Schuh, Sean Fitzgibbon, Robert Wright, Matthew F. Glasser, Timothy S. Coalson, Jonathan O’Muircheartaigh, Jana Hutter, Anthony N. Price, Lucilio Cordero-Grande, Rui Pedro A.G. Teixeira, Emer Hughes, Nora Tusor, Kelly Pegoretti Baruteau, Mary A. Rutherford, A. David Edwards, Joseph V. Hajnal, Stephen M. Smith, Daniel Rueckert, Mark Jenkinson, and Emma C. Robinson.  Construction of a neonatal cortical surface atlas using multimodal surface matching in the developing human connectome project.  NeuroImage, 179:11–29, 10 2018.
  * [3] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic.  Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.  CoRR, abs/2104.13478, 2021.
  * [4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  CoRR, abs/2104.14294, 2021.
  * [5] Chun-Fu Chen, Rameswar Panda, and Quanfu Fan.  Regionvit: Regional-to-local attention for vision transformers.  In International Conference on Learning Representations, 2022.
  * [6] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda.  Crossvit: Cross-attention multi-scale vision transformer for image classification.  pages 347–356, 3 2021.
  * [7] Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling.  Spherical cnns, 2018.
  * [8] Tim F Cootes and Christopher J Taylor.  Statistical models of appearance for medical image analysis and computer vision.  In Medical Imaging 2001: Image Processing, volume 4322, pages 236–248. International Society for Optics and Photonics, 2001.
  * [9] Lucilio Cordero-Grande, Emer J. Hughes, Jana Hutter, Anthony N. Price, and Joseph V. Hajnal.  Three-dimensional motion corrected sensitivity encoding reconstruction for multi-shot multi-slice MRI: Application to neonatal brain imaging.  Magnetic resonance in medicine, 79(3):1365–1376, mar 2018.
  * [10] Simon Dahan, Abdulah Fawaz, Logan Zane John Williams, Chunhui Yang, Timothy S. Coalson, Matthew Glasser, A David Edwards, Daniel Rueckert, and Emma Claire Robinson.  Surface vision transformers: Attention-based modelling applied to cortical analysis.  In Medical Imaging with Deep Learning, 2022.
  * [11] Stéphane d’Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun.  Convit: Improving vision transformers with soft convolutional inductive biases, 2021.
  * [12] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering, 2017.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [14] Ralica Dimitrova, Maximilian Pietsch, Judit Ciarrusta, Sean P. Fitzgibbon, Logan Z.J. Williams, Daan Christiaens, Lucilio Cordero-Grande, Dafnis Batalle, Antonios Makropoulos, Andreas Schuh, Anthony N. Price, Jana Hutter, Rui PAG Teixeira, Emer Hughes, Andrew Chew, Shona Falconer, Olivia Carney, Alexia Egloff, J. Donald Tournier, Grainne McAlonan, Mary A. Rutherford, Serena J. Counsell, Emma C. Robinson, Joseph V. Hajnal, Daniel Rueckert, A. David Edwards, and Jonathan O’Muircheartaigh.  Preterm birth alters the development of cortical microstructure and morphology at term-equivalent age.  NeuroImage, 243:118488, 11 2021.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  CoRR, abs/2010.11929, 2020.
  * [16] Nico Engel, Vasileios Belagiannis, and Klaus Dietmayer.  Point transformer.  IEEE Access, 9:134826–134840, 12 2020.
  * [17] Abdulah Fawaz, Logan Z. J. Williams, Amir Alansary, Cher Bass, Karthik Gopinath, Mariana da Silva, Simon Dahan, Chris Adamson, Bonnie Alexander, Deanne Thompson, Gareth Ball, Christian Desrosiers, Hervé Lombaert, Daniel Rueckert, A. David Edwards, and Emma C. Robinson.  Benchmarking geometric deep learning for cortical segmentation and neurodevelopmental phenotype prediction.  bioRxiv, 2021.
  * [18] Bruce Fischl, Niranjini Rajendran, Evelina Busa, Jean Augustinack, Oliver Hinds, B. T.Thomas Yeo, Hartmut Mohlberg, Katrin Amunts, and Karl Zilles.  Cortical folding patterns and predicting cytoarchitecture.  Cerebral cortex (New York, N.Y. : 1991), 18(8):1973–1980, aug 2008\.
  * [19] P. Gainza, F. Sverrisson, F. Monti, E. Rodolà, D. Boscaini, M. M. Bronstein, and B. E. Correia.  Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning.  Nature Methods 2019 17:2, 17:184–192, 12 2019.
  * [20] Kara E Garcia, Emma C Robinson, Dimitrios Alexopoulos, Donna L Dierker, Matthew F Glasser, Timothy S Coalson, Cynthia M Ortinau, Daniel Rueckert, Larry A Taber, David C Van Essen, et al.  Dynamic patterns of cortical expansion during folding of the preterm human brain.  Proceedings of the National Academy of Sciences, 115(12):3156–3161, 2018.
  * [21] Matthew Glasser, Stamatios Sotiropoulos, J. Wilson, Timothy Coalson, Bruce Fischl, Jesper Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan Polimeni, Van DC, and Mark Jenkinson.  The minimal preprocessing pipelines for the human connectome project.  NeuroImage, 80:105, 10 2013.
  * [22] Matthew F. Glasser, Timothy S. Coalson, Emma C. Robinson, Carl D. Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F. Beckmann, Mark Jenkinson, Stephen M. Smith, and David C. Van Essen.  A multi-modal parcellation of human cerebral cortex.  Nature, 536(7615):171–178, aug 2016.
  * [23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [24] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh.  Rethinking spatial dimensions of vision transformers.  CoRR, abs/2103.16302, 2021.
  * [25] Seok-Jun Hong, Sofie L Valk, Adriana Di Martino, Michael P Milham, and Boris C Bernhardt.  Multidimensional neuroanatomical subtyping of autism spectrum disorder.  Cerebral Cortex, 28(10):3578–3588, 2018.
  * [26] Emer J. Hughes, Tobias Winchman, Francesco Padormo, Rui Teixeira, Julia Wurie, Maryanne Sharma, Matthew Fox, Jana Hutter, Lucilio Cordero-Grande, Anthony N. Price, Joanna Allsop, Jose Bueno-Conde, Nora Tusor, Tomoki Arichi, A. D. Edwards, Mary A. Rutherford, Serena J. Counsell, and Joseph V. Hajnal.  A dedicated neonatal brain imaging system.  Magnetic resonance in medicine, 78(2):794–804, aug 2017.
  * [27] Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng.  Token labeling: Training a 85.4% top-1 accuracy vision transformer with 56m parameters on imagenet.  CoRR, abs/2104.10858, 2021.
  * [28] Byung-Hoon Kim, Jong Chul Ye, and Jae-Jin Kim.  Learning dynamic graph representation of brain connectome with spatio-temporal attention.  CoRR, abs/2105.13495, 2021.
  * [29] Byung-Hoon Kim, Jong Chul Ye, and Jae-Jin Kim.  Learning dynamic graph representation of brain connectome with spatio-temporal attention, 2021.
  * [30] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks, 2017\.
  * [31] Maria Kuklisova-Murgasova, Gerardine Quaghebeur, Mary A. Rutherford, Joseph V. Hajnal, and Julia A. Schnabel.  Reconstruction of fetal brain MRI with intensity matching and complete outlier removal.  Medical image analysis, 16(8):1550–1564, 2012.
  * [32] Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao.  Uniformer: Unified transformer for efficient spatiotemporal representation learning.  CoRR, abs/2201.04676, 2022.
  * [33] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.  Feature pyramid networks for object detection.  CoRR, abs/1612.03144, 2016.
  * [34] Or Litany, Alex Bronstein, Michael Bronstein, and Ameesh Makadia.  Deformable shape completion with graph convolutional autoencoders.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1886–1895, 12 2017.
  * [35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  CoRR, abs/2103.14030, 2021.
  * [36] Antonios Makropoulos, Emma C. Robinson, Andreas Schuh, Robert Wright, Sean Fitzgibbon, Jelena Bozek, Serena J. Counsell, Johannes Steinweg, Katy Vecchiato, Jonathan Passerat-Palmbach, Gregor Lenz, Filippo Mortari, Tencho Tenev, Eugene P. Duff, Matteo Bastiani, Lucilio Cordero-Grande, Emer Hughes, Nora Tusor, Jacques Donald Tournier, Jana Hutter, Anthony N. Price, Rui Pedro A.G. Teixeira, Maria Murgasova, Suresh Victor, Christopher Kelly, Mary A. Rutherford, Stephen M. Smith, A. David Edwards, Joseph V. Hajnal, Mark Jenkinson, and Daniel Rueckert.  The Developing Human Connectome Project: A Minimal Processing Pipeline for Neonatal Cortical Surface Reconstruction, apr 2018.
  * [37] C. Mauger, K. Gilbert, A. Lee, M. Sanghvi, N. Aung, K. Fung, V. Carapella, S. Piechnik, S. Neubauer, S. Petersen, A. Suinesiaputra, and A. Young.  Right ventricular shape and function: Cardiovascular magnetic resonance reference morphology and biventricular risk factor morphometrics in uk biobank.  Journal of Cardiovascular Magnetic Resonance, 21:1–13, 2019.
  * [38] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein.  Geometric deep learning on graphs and manifolds using mixture model cnns, 2016.
  * [39] Alex Morehead, Chen Chen, and Jianlin Cheng.  Geometric transformers for protein interface contact prediction.  10 2021.
  * [40] Dimitrios Perperidis, Raad H. Mohiaddin, and Daniel Rueckert.  Spatio-temporal free-form registration of cardiac mr image sequences.  Medical image analysis, 9:441–456, 2005.
  * [41] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space, 2017.
  * [42] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu.  Geometric transformer for fast and robust point cloud registration.  2 2022.
  * [43] Armin Raznahan, Roberto Toro, Eileen Daly, Dene Robertson, Clodagh Murphy, Quinton Deeley, Patrick F Bolton, Tomáš Paus, and Declan GM Murphy.  Cortical anatomy in autism spectrum disorder: an in vivo mri study on the effect of age.  Cerebral cortex, 20(6):1332–1340, 2010.
  * [44] Emma C. Robinson, Kara Garcia, Matthew F. Glasser, Zhengdao Chen, Timothy S. Coalson, Antonios Makropoulos, Jelena Bozek, Robert Wright, Andreas Schuh, Matthew Webster, Jana Hutter, Anthony Price, Lucilio Cordero Grande, Emer Hughes, Nora Tusor, Philip V. Bayly, David C. Van Essen, Stephen M. Smith, A. David Edwards, Joseph Hajnal, Mark Jenkinson, Ben Glocker, and Daniel Rueckert.  Multimodal surface matching with higher-order smoothness constraints.  NeuroImage, 167:453–465, feb 2018.
  * [45] Emma C. Robinson, Saad Jbabdi, Matthew F. Glasser, Jesper Andersson, Gregory C. Burgess, Michael P. Harms, Stephen M. Smith, David C. Van Essen, and Mark Jenkinson.  MSM: a new flexible framework for Multimodal Surface Matching.  NeuroImage, 100:414–426, oct 2014.
  * [46] Andreas Schuh, Antonios Makropoulos, Robert Wright, Emma C. Robinson, Nora Tusor, Johannes Steinweg, Emer Hughes, Lucilio Cordero Grande, Anthony Price, Jana Hutter, Joseph V. Hajnal, and Daniel Rueckert.  A deformable model for the reconstruction of the neonatal cortex.  Proceedings - International Symposium on Biomedical Imaging, pages 800–803, jun 2017.
  * [47] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [48] Catherine Y Spong.  Defining “term” pregnancy: recommendations from the defining “term” pregnancy workgroup.  Jama, 309(23):2445–2446, 2013.
  * [49] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  CoRR, abs/2106.10270, 2021.
  * [50] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  CoRR, abs/1905.11946, 2019.
  * [51] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  CoRR, abs/2012.12877, 2020.
  * [52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  CoRR, abs/1706.03762, 2017.
  * [53] Logan Z. J. Williams, Sean P. Fitzgibbon, Jelena Bozek, Anderson M. Winkler, Ralica Dimitrova, Tanya Poppe, Andreas Schuh, Antonios Makropoulos, John Cupitt, Jonathan O’Muircheartaigh, Eugene P. Duff, Lucilio Cordero-Grande, Anthony N. Price, Joseph V. Hajnal, Daniel Rueckert, Stephen M. Smith, A. David Edwards, and Emma C. Robinson.  Structural and functional asymmetry of the neonatal cerebral cortex.  bioRxiv, 2021.
  * [54] Tomos G Williams, Andrew P Holmes, John C Waterton, Rose A Maciewicz, Charles E Hutchinson, Robert J Moots, Anthony FP Nash, and Chris J Taylor.  Anatomically corresponded regional analysis of cartilage in asymptomatic and osteoarthritic knees by statistical shape modelling of the bone.  IEEE transactions on medical imaging, 29(8):1541–1559, 2010.
  * [55] Hao Xu, Steven A. Niederer, Steven E. Williams, David E. Newby, Michelle C. Williams, and Alistair A. Young.  Whole heart anatomical refinement from ccta using extrapolation and parcellation.  11 2021.
  * [56] Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.  Shapeformer: Transformer-based shape completion via sparse representation.  1 2022.
  * [57] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim.  Graph transformer networks.  Advances in Neural Information Processing Systems, 32, 11 2019.
  * [58] Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  CoRR, abs/1311.2901, 2013.
  * [59] Fenqiang Zhao, Shunren Xia, Zhengwang Wu, Dingna Duan, Li Wang, Weili Lin, John H Gilmore, Dinggang Shen, and Gang Li.  Spherical u-net on cortical surfaces: Methods and applications, 2019.
  * [60] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: deformable transformers for end-to-end object detection.  CoRR, abs/2010.04159, 2020.