11institutetext: Institute of Biomedical Engineering, Department of
Engineering Science, University of Oxford, Oxford, UK
22institutetext: Wellcome Centre for Integrative Neuroimaging,
FMRIB,University of Oxford, Oxford, UK
33institutetext: Australian Institute for Machine Learning (AIML), Department
of Computer Science, University of Adelaide, Adelaide, Australia
44institutetext: South Australian Health and Medical Research Institute
(SAHMRI), North Terrace, Adelaide, Australia
44email<EMAIL_ADDRESS>

# TEDS-Net: Enforcing Diffeomorphisms in Spatial Transformers to Guarantee
Topology Preservation in Segmentations

Madeleine K. Wyburd 11 Nicola K. Dinsdale 22 Ana I.L. Namburete† 11 Mark
Jenkinson† 223344

###### Abstract

Accurate topology is key when performing meaningful anatomical segmentations,
however, it is often overlooked in traditional deep learning methods. In this
work we propose TEDS-Net: a novel segmentation method that guarantees accurate
topology. Our method is built upon a continuous diffeomorphic framework, which
enforces topology preservation. However, in practice, diffeomorphic fields are
represented using a finite number of parameters and sampled using methods such
as linear interpolation, violating the theoretical guarantees. We therefore
introduce additional modifications to more strictly enforce it. Our network
learns how to warp a binary prior, with the desired topological
characteristics, to complete the segmentation task. We tested our method on
myocardium segmentation from an open-source 2D heart dataset. TEDS-Net
preserved topology in 100$\%$ of the cases, compared to 90$\%$ from the U-Net,
without sacrificing on Hausdorff Distance or Dice performance. Code will be
made available at: www.github.com/mwyburd/TEDS-Net.

###### Keywords:

Diffeomorphic Topology Segmentation Spatial Transformers

##  1 Introduction

Anatomical segmentations with incorrect topology can be clinically problematic
and highly impractical. For example, when segmenting the heart’s myocardium,
topological mistakes may present as isolated segmented regions, small holes
within the wall or a disconnected perimeter. These inaccuracies can all lead
to incorrect measurements of circumference and wall thickness, which are often
required when diagnosing heart conditions such as hypertrophic cardiomyopathy
[11]. Post-processing morphological operations, such as binary closing, can be
used to correct some small gaps and disconnected regions. However, they are
unaware of the structure’s overall topology and need to be customised
accordingly, therefore, an unified method is preferred.

Semantic segmentation of medical images is regularly performed using deep
convolutional neural networks (CNNs), such as the U-Net [15]. One alternative
segmentation method uses a combination of CNNs and spatial transformers to
learn the spatial warping required to transform a set of prior shapes into the
desired class labels [10, 18, 17, 8, 19]. Such methods have outperformed
conventional encoder-decoder and state-of-the-art architectures, however, they
have no topological guarantees.

Moreover, both these methods are commonly trained and evaluated using loss
functions such as Dice or binary cross-entropy, which evaluate each pixel
individually and not the higher level structure, often resulting in large
topological errors. Recent work has recognised the importance of accurate
whole-structure segmentation, with the development of loss functions that
encourage the preservation of topology [5, 14, 9]. Although these techniques
report an improved topology accuracy compared to standalone pixel-wise loss
functions, they only encourage topology preservation as opposed to enforcing
it, with the best performing method achieving 96.7% topology accuracy in
myocardium segmentation [5].

In theory, topology will always be preserved by deforming a prior, with the
correct topological characteristic, with a continuous diffeomorphic field [2].
Diffeomorphic fields are continuous deformation fields that result in a one-
to-one mapping. Their derivatives are invertible, resulting in positive
Jacobian determinants, giving an unambiguous mapping between coordinates and
therefore preserving topology [2].

Resampling one diffeomorphic field by another, also known as a composition,
always results in a new diffeomorphic field [2]. In VoxelMorph’s methods [6,
7], this elegant property has been utilised for brain registration, by
initialising with a small diffeomorphic field and amplifying it through a
series of novel integration layers, based on the squaring and scaling approach
[1, 13], to generate a topology-preserving large-deformation field. However,
when diffeomorphisms are applied in discrete settings, such as images stored
as discrete voxels, their topological guarantees can begin to break down. This
is often indicated by the emergence of non-positive Jacobian determinants in
the deformation fields, which correspond to folding voxels in the warped space
[7]. In brain registration a small fraction of violations may be manageable,
conversely, for segmentation tasks these can lead to topological errors in the
structure of interest, defeating the purpose. Here, we adapt VoxelMorph’s
methods to be suitable for segmentation tasks by introducing a novel topology-
preserving activation function and additional smoothing terms to counteract
the negative effects of discrete sampling. These modifications enforce true
diffeomorphisms, shown to have only positive Jacobian determinants, which
results in 100% topology preservation.

In this work, we present the novel Topology Enforcing Diffeomorphic
Segmentation Network (TEDS-Net), which to the best of our knowledge is the
first deep learning technique to achieve 100% topology accuracy, and to
combine spatial transformer networks (STN) and diffeomorphic displacement
fields to complete a segmentation as the primary task. Our method uses CNNs to
learn a warp field that maps a binary prior on to the input image, with the
warp field being diffeomorphic by construction. We show that in the discrete
setting, continuous properties of diffeomorphisms are no longer guaranteed,
therefore, we introduce model components that enforce topology preservation in
all cases.

##  2 Method

The aim of the proposed TEDS-Net is to predict a segmentation label,
$\hat{\mathbf{Y}}$, using learnt diffeomorphic transformations,
$\mathbf{\Phi}$, applied to a binary prior image, $\mathbf{P}$. The network is
comprised of three main parts, shown in the bottom left corner of Fig 1.
Firstly, an encoder-decoder style network is used to learn the initial
velocity fields. These fields are then enforced to be diffeomorphic and
amplified using the squaring and scaling approach [13, 1]. Finally, the
diffeomorphic fields are used to sample a prior binary shape to complete the
segmentation task.

Figure 1: Schematic of TEDS-Net architecture. A CNN learns two initial fields,
u, at different resolutions, from an input image X. The fields are enforced to
be diffeomorphic using an activation function, $\textbf{v}=f(\textbf{u})$,
amplified through composition layers and “super” upsampled to 2x the
resolution of the input. The bulk displacement,
$\mathbf{\Phi}_{\mathrm{Bulk}}$, samples a binary prior, $\mathbf{P}$,
generating $\mathbf{Y}_{\mathrm{Bulk}}$, which is then sampled by the fine
tuning field, $\mathbf{\Phi}_{\mathrm{FT}}$. The asterisks show the elements
removed during ablation studies.

####  2.0.1 CNN.

An encoder-decoder network was used to extract the relevant features from an
input image, $\mathbf{X}\in\mathbb{R}^{h\times w}$ with dimensions $[h\times
w]$, in order to predict two initial velocity fields: a bulk field and a fine-
tuning field. We used an architecture similar to the U-Net but with two
decoder branches [15, 16]. Each branch consisted of a series of convolutions
followed by an instance normalisation and ReLU activation function and then
repeated, referred to as a convolutional block.

Branching off at the bottleneck were two decoder streams, made up of
convolutional blocks and a final 1x1 convolution, used to generate a two-
dimensional field, u. The first stream, with one upsampling convolutional
block, predicts a low resolution bulk-velocity,
$\textbf{u}_{\mathrm{Bulk}}\in\mathbb{R}^{2\times(h\times w)/8}$, which
focuses on warping the prior shape into the correct region. The second, with
three upsampling blocks, predicts a higher resolution fine-tuning field, used
to fine-tune the warped shape,
$\textbf{u}_{\mathrm{FT}}\in\mathbb{R}^{2\times(h\times w)/2}$.

####  2.0.2 Diffeomorphic Layers.

To enforce that the initial fields, $\mathbf{u}$, were diffeomorphic, and
therefore suitable for the scaling and squaring approach, a customised tanh
activation function, $f(\mathbf{u})$, was applied to the output of each
decoder branch to enforce that the displacements were between $-0.5$ voxel and
$+0.5$ voxel and therefore topology preserving:

$\mathbf{v}(\mathbf{x})=f(\mathbf{u}(\mathbf{x}))=0.5\left(\frac{e^{\mathbf{u}(\mathbf{x})}-e^{\mathbf{-u}(\mathbf{x})}}{e^{\mathbf{u}(\mathbf{x})}+e^{\mathbf{-u}(\mathbf{x})}}\right).$
(1)

A series of composition layers were then used to amplify the initial
diffeomorphic fields, $\mathbf{v}$, adapted from the VoxelMorph’s
implementation [6, 7].

##### Gaussian Smoothing:

As the diffeomorphic fields are represented using a finite number of
parameters and sampled using linear interpolation, the theoretical guarantees
are violated, risking key properties such as topological preservation.
Moreover, these inaccuracies and imperfections can be amplified with the
number of compositions performed [2]. To reduce these violations, we
introduced Gaussian smoothing between each integration layer. For this work we
used a 5 by 5 kernel and $\sigma=2$ and the impact of this will be
investigated in future work.

##### Super Upsample:

To further smooth the fields and minimise any abnormalities, the amplified
diffeomorphic flow fields were both “super upsampled” to double the resolution
of the input image, $\mathbf{X}$, using bilinear interpolation.

####  2.0.3 Spatial Transformers.

The diffeomorphic field, $\mathbf{\Phi}_{\mathrm{Bulk}}\in\mathbb{R}^{2\times
2h\times 2w}$, generated from the first decoder branch sampled the prior
shape, $\mathbf{P}\in\mathbb{R}^{h\times w}$, using bilinear interpolation:
$\mathbf{Y}_{\mathrm{Bulk}}=\mathbf{\Phi}_{\mathrm{Bulk}}(\mathbf{P})$, as
shown in Fig 2. The resulting warped image was then sampled by the fine-tuning
field generated by the second decoder branch,
$\mathbf{\hat{Y}}=\mathbf{\Phi}_{\mathrm{FT}}(\mathbf{Y}_{\mathrm{Bulk}})$,
where $\mathbf{\Phi}_{\mathrm{FT}}\in\mathbb{R}^{2\times 2h\times 2w}$. The
prediction, $\mathbf{\hat{Y}}$, was then downsampled to the same resolution as
the label, $\textbf{Y}\in\mathbb{R}^{h\times w}$, using Max Pooling.

Figure 2: A prior is sampled with a displacement field at double the
resolution,
$\mathbf{Y}_{\mathrm{Bulk}}=\mathbf{\Phi}_{\mathrm{Bulk}}(\mathbf{P})$, before
the second super upsampled displacement is applied to the shifted image
$\hat{\mathbf{Y}}=\mathbf{\Phi}_{\mathrm{FT}}(\mathbf{Y}_{\mathrm{Bulk}})$.
The displacement panels show the shift in both directions.

##### Field Regularisation:

The network outputs the final label prediction, $\hat{\mathbf{Y}}$, and the
two diffeomorphic transformation fields:
$\mathbf{\Phi}_{\mathrm{Bulk}},\mathbf{\Phi}_{\mathrm{FT}}$, which were all
used for the training loss. A combination of Dice Loss
($\mathcal{L}_{\mathrm{Dice}}$) and a field regularisation
($\mathcal{L}_{\mathrm{Grad}}$) function were used for training:

$\mathcal{L}=\mathcal{L}_{\mathrm{Dice}}(\mathbf{Y},\hat{\mathbf{Y}})+\beta\mathcal{L}_{\mathrm{Grad}}(\mathbf{\Phi}_{\mathrm{FT}})+\beta\mathcal{L}_{\mathrm{Grad}}(\mathbf{\Phi}_{\mathrm{Bulk}}),$
(2)

where
$\mathcal{L}_{\mathrm{Grad}}=\sum_{i,j=1}^{2h,2w}\left\|\nabla\mathbf{\Phi}(i,j)\right\|^{2}$,
which encourages smooth flow fields by penalising large spatial gradients
between neighbouring voxels, adapted from [3]. A weighting parameter, $\beta$,
was used to balance the contributions between $\mathcal{L}_{\mathrm{Grad}}$
and $\mathcal{L}_{\mathrm{Dice}}$, in this work we used a constant $\beta$ of
$10,000$, in order to give all the loss function components similar
magnitudes, as measured empirically.

###  2.1 Experimental Setup

TEDS-Net was used to segment the myocardium from an open source dataset of MRI
heart slices in the short axis111The ACDC database: www.creatis.insa-
lyon.fr/Challenge/acdc [4]. This dataset consisted of 2 scans from 100
patients, across five different pathologies. Five myocardium-containing slices
were taken from each scan, cropped to 144 by 208 pixels and augmented using
rotations, shifts, zooming and a combination of the three, resulting in 8,000
images split 75%, 15% and 10% for training, validation and testing,
respectively. It was assured that slices from the same patient were assigned
to the same subset.

The topology of the myocardium is equivalent to a hollow circle, which we used
as the prior shape, $\mathbf{P}$, shown in Fig 2. It should be noted that the
patients’ scans were not aligned and therefore the location of the myocardium
was inconsistent. Additionally, different pathologies have varying myocardium
thickness. Despite this, the same arbitrary prior shape was used throughout,
illustrating the robustness of the method.

Two main experiments were completed and evaluated on 100 unseen heart slices
without augmentations. Firstly, ablation studies were performed to show the
contribution of each additional element used in TEDS-Net to tackle the
limitations of diffeomorphic sampling, shown with the asterisks in Fig 1,
using a prior shape of radius 30 and 8 integration layers, which were
empirically set. As a comparison, we used a U-Net [15] and VoxelMorph
diffeomorphic network [6, 7], using the shape prior P as the second input
channel. Secondly, we investigated the effect of the number of integration
layers and radius of the binary prior.

The networks were all trained end-to-end on a NVIDIA GeForce GTX 1080 GPU and
implemented in Pytorch using Python 3.6. Training was performed over 200
epochs, using the Adam optimiser with a learning rate of 0.0001 and a batch
size of 5. TEDS-Net and the U-Net were both made up of 5 layers, with 12
initial feature maps, and a 20% dropout applied to each map.

##  3 Results and Discussion

To evaluate segmentation performance, Dice, Hausdorff Distance (HD), and a
count of incorrect topologies were used, as seen in Table 1. A predicted
segmentation’s topology was classed as incorrect if their Betti numbers [12]
varied from the known topological properties of the myocardium. To assess the
discrete fields, the Jacobian determinants were also measured, as perfect
diffeomorphic fields always have strictly positive determinants.

Table 1: Comparison of myocardium segmentation performed by the baselines and
TEDS-Net, accompanied by ablation studies (A1-6). The mean and standard
deviation are given for Dice, Hausdorff Distance (HD) and percentage of non-
positive Jacobian determinants from each field. The best result from each
measure is shown in bold.

Network | Dice | HD | $\%\left|J_{\mathbf{\Phi}_{\mathrm{Bulk}}}\right|\leq 0$ | $\%\left|J_{\mathbf{\Phi}_{\mathrm{FT}}}\right|\leq 0$ |  | Incorrect
---
Topology
U-Net (baseline) | $\mathbf{0.87\pm 0.16}$ | $5.25\pm 13.11$ | N/A | N/A | 10/100
VoxelMorph (baseline) | $0.79\pm 0.18$ | $3.0\pm 2.4$ | N/A | $6.7e^{-05}\pm 0.7e^{-03}$ | 6/100
(A1) TEDS-Net: No Gaussian, $\mathcal{L}_{\mathrm{Grad}}$ or Super Upsample | $0.82\pm 0.19$ | $4.26\pm 5.76$ | $56.32\pm 1.97$ | $43.8\pm 3.46$ | 17/100
(A2) TEDS-Net: No Gaussian Smoothing | $0.86\pm 0.13$ | $3.73\pm 8.34$ | $67.10\pm 7.07$ | $35.81\pm 4.41$ | 3/100
(A3) TEDS-Net: No $\mathcal{L}_{\mathrm{Grad}}$ | $0.85\pm 0.14$ | $2.77\pm 2.01$ | $5.94\pm 2.57$ | $22.15\pm 3.67$ | 35/100
(A4) TEDS-Net: No Super Upsample | $0.85\pm 0.11$ | $2.89\pm 1.61$ | $\mathbf{0.0\pm 0.0}$ | $\mathbf{0.0\pm 0.0}$ | 0/100
(A5) TEDS-Net: Only Bulk Branch | $0.77\pm 0.15$ | $3.86\pm 3.50$ | $0.03\pm 0.17$ | N/A | 0/100
(A6) TEDS-Net: Only Fine-Tune Branch | $0.85\pm 0.14$ | $3.01\pm 2.16$ | N/A | $\mathbf{0.0\pm 0.0}$ | 0/100
TEDS-Net (ours) | $0.86\pm 0.12$ | $\mathbf{2.76\pm 2.13}$ | $\mathbf{0.0\pm 0.0}$ | $\mathbf{0.0\pm 0.0}$ | 0/100

Large topological errors were found when segmenting the myocardium with the
U-Net, as seen from Table 1. These errors were mainly expressed as large gaps
separating the myocardium, shown in Fig 3, making clinical measures of
circumference extremely challenging. VoxelMorph was designed to maximise
registration performance using diffeomorphisms, however, when applied to
segmentation a small fraction of topology errors are observed, shown in Table
1: VoxelMorph. Unlike the U-Net predictions, these topology violations are
seen at the boundaries of the myocardium, forming holes or disconnected
regions when defined using 4-pixel connectivity, as done here, but not when
using 8-pixel connectivity. Conversely, our TEDS-Net enforced diffeomorphisms,
preserving topology for all cases, which to the best of our knowledge, is the
first deep learning segmentation technique to achieve 100% topological
accuracy. Therefore, TEDS-Net additional modifications are required to
prioritise topology preservation.

Paired t-tests were computed between TEDS-Net and U-Net and found that TEDS-
Net significantly outperformed the U-Net in Hausdorff Distance, with a p-value
of 0.01. Although the Dice accuracies were competitive, the U-Net performed
significantly better (p=0.04). Figure 3c-d shows particularly challenging
examples, where the U-Net fails to return the correct topology. However,
although TEDS-Net returns labels with accurate topology, the myocardial wall
appears too thick in some parts. This is likely due to the smoothing
modification, whose parameter will be further investigated in future work.

Figure 3: Examples of myocardium manual annotations (green), compared to U-Net
and TEDS-Net segmentations (pink) from the ACDC dataset [4].

Ablation studies were performed to show the effect of the additional smoothing
modifications used to encourage perfect diffeomorphisms, as shown in Table 1
(A1-6). The continuous guarantees of diffeomorphic sampling, such as topology
preservation, break down when applied in the discrete setting, shown by the
emergence of non-positive Jacobian determinants that can then lead to
topological errors. There are two sources of such violations in TEDS-Net: the
composition layers and the image sampling.

To limit the numerical inaccuracies brought about by the discrete composition
of two diffeomorphic fields, we introduced Gaussian smoothing between each
layer. Without this addition, a large fraction of the Jacobian determinants no
longer remained positive, corresponding to a number of topological defects, as
shown in Table 1 (A2).

Sampling a discrete binary image with a finite warp field often results in
disconnects, due to interpolations and the use of thresholds on the resultant
image. To reduce this effect, we regularised the smoothness of the final
deformation fields with $\mathcal{L}_{\mathrm{Grad}}$. Without this term, 35
out of 100 images were found to have topological defects and therefore, it
played a vital role in mimicking perfect diffeomorphisms, as shown in Table 1
(A3).

Removing either the bulk or fine-tuning sampling branch was found to reduce
the performance of both Dice and HD, as shown Table 1 (A5-6). As the images
are unaligned, with varying sizes of myocardium, shown in Fig 3, both branches
are required to first align the prior before fine-tuning the warped shape.
TEDS-Net performs the worst without the fine-tuning branch, which is likely
due to the lack of flexibility in the bulk transformation, as the deformation
fields are generated at a much lower resolution before being upsampled.

Figure 4: The effect that the number of integration layers (a) and the radius
of the binary prior (b) had on segmentation performance and the diffeomorphic
nature of the generated fields. Due to the image dimensions, 50 was the
maximum radius used.

####  3.0.1 Number of Integration Layers:

Including Gaussian smoothing between the integration layers played a key role
in enforcing topology preservation, as shown in Table 1 (A2). Although
theoretically accuracy should increase with the number of composition layers,
this requires performing more compositions that can each bring about small
violations, due to the discrete nature of the fields [2]. However, limiting
the number of composition layers, limits the size of deformations. To
investigate this further, we varied the number of integration layers used in
TEDS-Net and measured the segmentation performance and Jacobian determinants
of the resulting fields, shown by Fig 4a. When using 12 or fewer integration
layers, the segmentation performance is stable whilst the Jacobian
determinants all remain positive and all topology is preserved. However, when
the number of layers is increased beyond 12, non-positive Jacobian
determinants emerge. Therefore, whilst Gaussian smoothing has been shown to be
essential in the integration layers, there is a limit to the number of
compositions that can be used whilst enforcing diffeomorphic sampling.

####  3.0.2 Prior Radius:

In the test set, the myocardium’s radius ranged between approximately 10 to 60
voxels, so we investigated the impact of varying the radius on the final
segmentation performance, as shown in Fig 4b. The segmentation performance
remained consistent as the radius was increased between 20 and 50 voxels.
However, when the radius was set to 10 voxels, a small percentage of the
Jacobian determinants of the fine-tuning fields were non-positive,
corresponding to 2/100 incorrect topologies. This is likely due to the
decreased number of voxels in the prior representing the myocardium and the
area enclosed by it, in combination with the additional smoothing modification
that restrict the flexibility of the diffeomorphic fields.

##  4 Conclusion

We have shown that TEDS-Net is able to achieve highly accurate myocardium
segmentations whilst ensuring topology preservation. We introduced additional
diffeomorphic-encouraging modifications, which were found to play a crucial
role in enforcing an one-to-one mappings in the generated discrete fields. Our
method successfully segmented the myocardium in unaligned MRI heart slices,
with different pathologies that had different thicknesses and circumferences,
using the same general prior shape for all. This flexible, easy to train
method has the potential to have a high impact in future clinical segmentation
work.

## Acknowledgments:

MW and ND is supported by the Engineering and Physical Sciences Research
Council (EPSRC) and Medical Research Council (MRC) [grant number
EP/L016052/1]. MJ is supported by the National Institute for Health Research
(NIHR) Oxford Biomedical Research Centre (BRC), and this research was funded
by the Well- come Trust [215573/Z/19/Z]. The Wellcome Centre for Integrative
Neuroimaging is supported by core funding from the Wellcome Trust
[203139/Z/16/Z]. AN is grateful for support from the UK Royal Academy of
Engineering under the Engineering for Development Research Fellowships scheme.

## References

  * [1] Arsigny, V., Commowick, O., Pennec, X., Ayache, N.: A log-euclidean polyaffine framework for locally rigid or affine registration. In: International Workshop on Biomedical Image Registration. pp. 120–127. Springer (2006)
  * [2] Ashburner, J.: A fast diffeomorphic image registration algorithm. Neuroimage 38(1), 95–113 (2007)
  * [3] Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging 38(8), 1788–1800 (2019)
  * [4] Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.A., Cetin, I., Lekadir, K., Camara, O., Ballester, M.A.G., et al.: Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging 37(11), 2514–2525 (2018)
  * [5] Clough, J., Byrne, N., Oksuz, I., Zimmer, V.A., Schnabel, J.A., King, A.: A topological loss function for deep-learning based image segmentation using persistent homology. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
  * [6] Dalca, A.V., Balakrishnan, G., Guttag, J., Sabuncu, M.R.: Unsupervised learning for fast probabilistic diffeomorphic registration. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 729–738. Springer (2018)
  * [7] Dalca, A.V., Balakrishnan, G., Guttag, J., Sabuncu, M.R.: Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces. Medical image analysis 57, 226–236 (2019)
  * [8] Dinsdale, N.K., Jenkinson, M., Namburete, A.I.: Spatial warping network for 3d segmentation of the hippocampus in mr images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 284–291. Springer (2019)
  * [9] Hu, X., Fuxin, L., Samaras, D., Chen, C.: Topology-preserving deep image segmentation. arXiv preprint arXiv:1906.05404 (2019)
  * [10] Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. Advances in neural information processing systems 28, 2017–2025 (2015)
  * [11] Marian, A.J., Braunwald, E.: Hypertrophic cardiomyopathy: genetics, pathogenesis, clinical manifestations, diagnosis, and therapy. Circulation research 121(7), 749–770 (2017)
  * [12] Millson, J.J.: On the first betti number of a constant negatively curved manifold. Annals of Mathematics pp. 235–247 (1976)
  * [13] Moler, C., Van Loan, C.: Nineteen dubious ways to compute the exponential of a matrix, twenty-five years later. SIAM review 45(1), 3–49 (2003)
  * [14] Mosinska, A., Marquez-Neila, P., Koziński, M., Fua, P.: Beyond the pixel-wise loss for topology-aware delineation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3136–3145 (2018)
  * [15] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [16] Stergios, C., Mihir, S., Maria, V., Guillaume, C., Marie-Pierre, R., Stavroula, M., Nikos, P.: Linear and deformable image registration with 3d convolutional neural networks. In: Image Analysis for Moving Organ, Breast, and Thoracic Images, pp. 13–22. Springer (2018)
  * [17] Vigneault, D.M., Xie, W., Ho, C.Y., Bluemke, D.A., Noble, J.A.: $\omega$-net (omega-net): fully automatic, multi-view cardiac mr detection, orientation, and segmentation with deep neural networks. Medical image analysis 48, 95–106 (2018)
  * [18] Wickramasinghe, U., Knott, G., Fua, P.: Probabilistic atlases to enforce topological constraints. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 218–226. Springer (2019)
  * [19] Zeng, Q., Karimi, D., Pang, E.H., Mohammed, S., Schneider, C., Honarvar, M., Salcudean, S.E.: Liver segmentation in magnetic resonance imaging via mean shape fitting with fully convolutional neural networks. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 246–254. Springer (2019)


# Functor String Diagrams: A Novel Approach to Flexible Diagrams for Applied
Category Theory

Vincent Abbott College of Engineering, Computing and Cybernetics
Australian National University
Canberra, ACT, Australia<EMAIL_ADDRESS>Laboratory for Information
and Decision Systems
Massachusetts Institute of Technology
Cambridge, MA, USA Gioele Zardini  Laboratory for Information and Decision
Systems
Massachusetts Institute of Technology
Cambridge, MA, USA<EMAIL_ADDRESS>
###### Abstract

The study of abstraction and composition—the focus of category
theory—naturally leads to sophisticated diagrams which can encode complex
algebraic semantics. Consequently, these diagrams facilitate a clearer visual
comprehension of diverse theoretical and applied systems. Complex algebraic
structures—otherwise represented by a forest of symbols—can be encoded into
diagrams with intuitive graphical rules. The prevailing paradigm for
diagrammatic category theory are monoidal string diagrams whose specification
reflects the axioms of monoidal categories. However, such diagrams struggle in
accurately portraying crucial categorical constructs such as functors or
natural transformations, obscuring central concepts such as the Yoneda lemma
or the simultaneous consideration of hom-functors and products. In this work,
we introduce _functor string diagrams_ , a systematic approach for the
development of categorical diagrams which allows functors, natural
transformations, and products to be clearly represented. We validate their
practicality in multiple dimensions. We show their usefulness for theoretical
manipulations by proving the Yoneda lemma, show that they encompass monoidal
string diagrams and hence their helpful properties, and end by showing their
exceptional applied utility by leveraging them to underpin neural circuit
diagrams, a method which, at last, allows deep learning architectures to be
comprehensively and rigorously expressed.

##  1 Introduction

Category theory enables a rigorous and abstract understanding of composition,
offering tools for studying a wide range of systems. One of such tools are
diagrams. Cohesive diagrams need to obey strict compositionality rules, making
category theory the natural means by which to approach them. In describing
systems, diagrams offer a rigorous framework to visualize and analyze them.
Furthermore, by diagramming a system, its fundamental abstract structure is
revealed.

Categorical diagrams encode the compositional structure of systems, naturally
reflecting their algebraic semantics and exchanging the management of symbols
for the manipulation of diagrams. The most prominent diagrams in category
theory are monoidal string diagrams [18, 15].

These have been applied, or found to underlie, diagrams in various domains.
This includes Feynman diagrams in physics, Petri nets for engineered systems,
diagrams for quantum computing, and shared theoretical structures in physics,
logic, and topology [4, 6, 13]. Monoidal string diagrams offer a common
unified approach, letting theorems be applied across domains.

Current diagrammatic approaches, however, are specialized to focus on certain
algebraic concepts. They employ topological isotopy rules which ensure that
diagrams correspond to topological manipulations which shift around
components. This limits them. Monoidal string diagrams, for instance, rely on
a privileged product to combine morphisms. This struggles to represent systems
where the actions of both linear and non-linear operations are critical to
represent, for instance. Furthermore, monoidal string diagrams lack a natural
expression for functors and, hence, fail to express many insightful
categorical concepts.

Alternative string diagrams [12, 14] address functors and natural
transformations but lack the means to readily express products. These diagrams
use colors to represent separate categories, and strings are functors showing
boundaries between them. Natural transformations are nodes on these
boundaries, and may introduce additional functors, acting as intersections
between boundaries. These diagrams are focused on advanced categorical
concepts, the behaviour of monads and adjunctions, for instance. A variety of
existing diagrammatic approaches are showcased in the appendix, Section A.

For applied cases, however, streamlined diagrams are preferred. While
morphisms, functors, and natural transformations frequently arise in practice,
color coding categories impose distraction but little benefit. Furthermore,
expressing both functors and products in either scheme is unclear and
obfuscates the underlying expression. Proofs related to areas in which
diagrams are not-specialized are difficult to parse. The topological formality
of current diagrams, then, can be seen as a hindrance to generalization,
cross-domain understanding, and application.

Instead, it will be valuable to distill the construction of diagrams down to
clear principles which ensures we can construct a reliable correspondence
between categorical expressions and diagrams. Rather than specializing
diagrams for a particular case, we will ensure they can robustly represent
categorical expressions and will introduce new features as needed. These new
features will each provide local graphical intuition and will correspond with
algebraic rules, but will not be required to be consistent with an overarching
topological regularity. We will prefer to use colors for annotation rather
than expression, granting additional utility for explaining and applying
diagrams.

This leads to functor string diagrams [3], a principled approach to
constructing diagrams for various systems. We will demand that diagrams can be
readily corresponded to standard object-morphism-object-etc. expressions. New
features will be introduced as needed, with each implementing an algebraic
rule in a graphically intuitive manner that motivates its use. Therefore, they
can reveal the algebraic structure in a range of cases, from formal proofs to
the design of deep learning models, in a clearly understandable manner.

### Our Contributions

In this work, the two principles underlying functor string diagrams, vertical
section decomposition and equivalent expression for graphical intuition, are
introduced. The disciplined diagrams constructed from these principles are
shown to provide both theoretical and applied utility. We show this utility by
presenting a straightforward proof of the Yoneda lemma, showing existing
monoidal string diagrams to be a specific instance of functor string diagrams,
and finally show that they provide mathematical rigor to neural circuit
diagrams, addressing the open-problem of comprehensively communicating the
details of deep learning models.

##  2 The Foundations of Functor String Diagrams

Functor string diagrams fundamentally take the perspective that categorical
diagrams should be flexible yet rigorous, adaptable to various use cases while
maintaining clear correspondence to underlying symbolic expressions and having
features which guide intuition regarding the application of theorems. This is
achieved by establishing two principles for the construction of diagrams.

This section will introduce and highlight these principles. If they are
followed, we will attain flexible, disciplined diagrams that can represent
just about any categorical concept. Compared to monoidal string diagrams which
strictly demand isotopy, we have the flexibility to introduce functors and
natural transformations in a rigorous, intuitive manner. Furthermore, these
diagrams can be adapted to different use cases. As long as we are aware of the
equivalent expressions employed, we always have diagrams corresponding to
unique up to isomorphism categorical expressions.

###  2.1 Vertical Section Decomposition

Functor string diagrams can be decomposed into vertical columns. Vertical
columns either represent the current object, or a morphism acting on that
object. Below in Figure 1, we can see a basic functor string diagram mapping
through $\displaystyle f:x\rightarrow y$ and $\displaystyle g:x\rightarrow z$
to construct $\displaystyle f;g:x\rightarrow z$. (; representing forward
composition.) Note how we have vertical columns for the objects $\displaystyle
x,y,z$, and other vertical columns for the morphisms $\displaystyle f$ and
$\displaystyle g$.

[capbesideposition=left,top]figure[]

Figure 1:  Here, we have a morphism with multiple functor wires present. Using
the principle of equivalent expression, this can be re-expressed into an
easier-to-understand form.

This gives us the first principle for functor string diagrams, vertical
section composition, which states that all vertical sections of a diagram
correspond to either an object or a morphism. Furthermore, as we express
identities by straight lines, the same as objects, the construction of
diagrams via vertical section decomposition corresponds with the composition
and identity axioms of a category. Therefore, there is a one-to-one
correspondence between valid composed functor string diagrams and valid
composed symbolic categorical expressions.

###  2.2 Equivalent Expression

In practice, we will want to introduce additional features to diagrams to
represent functors, natural transformations, and other constructs. To maintain
cohesion, we will demand that our means of representing these additional
features ultimately corresponds to columns of objects and morphisms, abiding
by the first principle. This grants the second principle, that all additional
features are equivalent expressions of existing features to support graphical
intuition.

This guides us towards expressing functors. Functors are morphisms from
$\displaystyle\mathbf{C}$ to $\displaystyle\mathbf{D}$ which map objects and
morphisms so that composition is preserved. Fundamentally, we require that
$\displaystyle f:x\rightarrow y$ maps to $\displaystyle Ff:Fx\rightarrow Fy$
and that $\displaystyle F(f;g)=Ff;Fg$. We express an object $\displaystyle Fx$
by a vertical column with a functor wire $\displaystyle F$ placed above
$\displaystyle x$, and $\displaystyle Ff$ by a vertical column with the
functor wire $\displaystyle F$ placed above $\displaystyle f$, as shown in
Figure 2. When constructing $\displaystyle Ff;Fg$, we see that graphical
intuition supports its equality to $\displaystyle F(f;g)$.

Figure 2: Objects and morphisms in the image of functors can be explicitly
expressed. Functors obey the axioms that $f:x\rightarrow y$ maps to
$Ff:Fx\rightarrow Fy$ and $F(f;g)=Ff;Fg$. Therefore, we develop an equivalent
expression which graphically enforces these axioms, necessarily mapping a
morphism along with its objects and necessarily mapping composed morphisms.

Similarly, we find that natural transformations also have an intuitive
equivalent expression. We can write the defining axiom of natural
transformations at the top, wherein the functors can be equivalently expressed
with functor wires. We notice that the natural transformation can move along
the functor line, hence, we establish an equivalent expression for natural
transformation components by writing them on the functor wire. This is similar
to the methods used by Marsden and Nakahira [12, 14] and ensures that the
axioms of diagrams reflect the axioms of symbols.

[capbesideposition=left,top]figure[]

Figure 3:  We develop equivalent expressions to consider natural
transformations. We see that natural transformations move along the functor
wire and, hence, we show them as morphisms on the functor wire.

##  3 The Yoneda Lemma with Functor String Diagrams

To prove the algebraic utility of functor string diagrams, we will be
expressing the Yoneda lemma in a natural manner. The Yoneda lemma, a
fundamental concept in category theory, has its relevance to various domains
potentially obscured when the role of natural transformations is not evident.
As traditional monoidal string diagrams can not consider functors or natural
transformations, a novel diagrammatic method is needed to understand it. Here,
we will provide a proof of the Yoneda lemma, borrowing from Nakahira, but
which uses the principles laid out to offer a brief, intuitive proof.

The Yoneda lemma relates to hom-functors and the category
$\displaystyle\mathbf{Set}$. In $\displaystyle\mathbf{Set}$, objects are sets
and morphisms are functions between them. In $\displaystyle\mathbf{Set}$, the
elements of sets correspond to morphisms from the set of one element,
$\displaystyle 1$, to the object corresponding to the set. In
$\displaystyle\mathbf{Set}$, morphisms are functions. Functions
$f:x\rightarrow y$ are defined by the output element $\varphi;f:1\rightarrow
y$ produced for each input element $\varphi:1\rightarrow x$, and any function
pairing each input to any output can be constructed.

For any category $\displaystyle\mathbf{C}$ where morphisms between objects
forms a set (locally small), there is a hom-functor for each object
$\displaystyle x$ which maps from $\displaystyle\mathbf{C}$ to
$\displaystyle\mathbf{Set}$, sending objects $\displaystyle y$ to the set of
morphisms $\displaystyle\mathbf{C}(x,y)$ and morphisms $\displaystyle
g:y\rightarrow z$ to a morphism
$\displaystyle\mathbf{C}(x,g):\mathbf{C}(x,y)\rightarrow\mathbf{C}(x,z)$. The
hom-functored morphism corresponds to the following operation in Figure 4.

[capbesideposition=left,top]figure[]

Figure 4:  The hom-functor maps morphisms $g:y\rightarrow z$ to a function
between sets $\mathbf{C}(x,g):\mathbf{C}(x,y)\rightarrow\mathbf{C}(x,z)$. This
function implements composition with $h:x\rightarrow y$ when provided with the
element of the set $\mathbf{C}(x,y)$ corresponding to $h$. We can introduce a
number of equivalent expressions to more clearly visualize these expressions.
This includes not drawing $\displaystyle 1$ object wires.

Regarding hom-functors, the Yoneda lemma makes the following claim.

######  Theorem 1 (Yoneda Lemma)

The natural transformations
$\displaystyle\eta:\mathbf{C}(x,\\_)\rightarrow\Phi$, both being functors
$\displaystyle\mathbf{C}\rightarrow\mathbf{Set}$, are in one-to-one
correspondence with the elements $\displaystyle\varphi\in\Phi x$, the set the
object $\displaystyle x\in\text{Ob}(\mathbf{C})$ is mapped to by
$\displaystyle\Phi$:

$\text{Nat}(\mathbf{C}(x,\\_),\Phi)\cong\Phi x.$

Proof. We start with an element $\displaystyle\varphi\in\Phi x$. We represent
this as a morphism $\displaystyle 1\rightarrow\Phi x$. Diagrammatically, we
see that this provides a hook for a morphism $\displaystyle f:x\rightarrow y$
to attach, giving an element $\displaystyle(\Phi f)(\varphi)\in\Phi y$. We
will define a function mapping from any morphism $\displaystyle f:x\rightarrow
y$ to the outcome of this operation, hence, its a morphism
$\displaystyle\varphi_{[y]}^{*}:\mathbf{C}(x,y)\rightarrow\Phi y$. This
morphism $\displaystyle\mathbf{C}(x,y)\rightarrow\Phi y$ exists by virtue of
the fact that a function mapping from any set of inputs to outputs is allowed
in $\displaystyle\mathbf{Set}$.

We need to confirm whether this is a natural transformation. We do this by
testing the above with a generic function $\displaystyle g:y\rightarrow z$.

These functions are equal for all elements in the domain $\mathbf{C}(x,y)$,
making them equal.

Therefore, we have a procedure “$\displaystyle*$” which maps from elements
$\displaystyle\varphi\in\Phi x$ to natural transformations
$\displaystyle\varphi^{*}:\mathbf{C}(x,\\_)\rightarrow\Phi$. The next phase of
the proof is to show that there are elements corresponding to natural
transformations, and that taken together, these processes form a bijection.

Starting with a natural transformation
$\displaystyle\eta:\mathbf{C}(x,\\_)\rightarrow\Phi$, we use the identity to
obtain an element $\displaystyle 1\rightarrow\Phi x$. This gives a map
“$\displaystyle\dagger$” from natural transforms to elements,
$\displaystyle\dagger:\text{Nat}(\mathbf{C}(x,\\_),\Phi)\rightarrow\Phi x$.

Now, we will find the natural transformation $\displaystyle\eta^{\dagger*}$
corresponding to the element $\displaystyle\eta^{\dagger}$.

This shows that mapping $\displaystyle\eta^{\dagger}$ back to a natural
transformation recovers it. Therefore, the map from elements to natural
transformations is a bijection. This implies there is a one-to-one
correspondence between natural transformations and elements, completing the
proof. $\blacksquare$

The Yoneda lemma, fundamentally, says nothing more or less than that the
following graphical move shown in Figure 5 is allowed. Elements can become
natural transformation components, and vice-versa.

Figure 5: The Yoneda lemma stated with our diagrammatic approach. It shows
that elements $\displaystyle\varphi_{\eta}\in\Phi x$ correspond to natural
transformations
$\displaystyle\eta_{\varphi}:\mathbf{C}(x,\\_)\rightarrow\Phi$. This is
equivalent to the graphically intuitive rule where elements can “jump up” and
become natural transformations, and vice-versa.

##  4 Functor String Diagrams, Products, and Monoidal String Diagrams

In addition to having utility in streamlining algebraic proofs and
understanding, functor string diagrams encompass monoidal string diagrams as a
specific case. Hence, they are shown to capture existing diagrams and provide
a strict improvement over current methods which have found success
representing Feynman diagrams, Petri nets, and other structures mentioned in
the introduction.

Monoidal string diagrams rely on a category $\displaystyle\mathbf{C}$ having a
functor $\displaystyle\otimes$ from
$\displaystyle\mathbf{C}\times\mathbf{C}\rightarrow\mathbf{C}$.
$\displaystyle\mathbf{C}\times\mathbf{D}$ is the product category wherein
objects are tuples and morphisms are tuples from $\displaystyle\mathbf{C}$ and
$\displaystyle\mathbf{D}$. We equivalently express objects and morphisms in a
product category by a double-dashed line separating the constituent objects or
morphisms as in Figure 6.

[capbesideposition=left,top]figure[]

Figure 6:  Product categories such as
$\displaystyle\mathbf{C}\times\mathbf{D}$ are represented by a double-dashed
line. This can represent the combination of expressions from any two
categories.

Therefore, $\displaystyle\mathbf{C}\times\mathbf{C}$ is represented as two
expressions from the same category separated by a double dashed line. A
functor applied onto a product expression needs to be indicated as distinct
from a functor on the upmost expression, and this is achieved with a bold
line. Thereby,
$\displaystyle\otimes:\mathbf{C}\times\mathbf{C}\rightarrow\mathbf{C}$ can be
represented with a bold wire labeled “$\displaystyle\otimes$” (see appendix
Figure 19). Furthermore, monoidal categories typically have a privileged
product. In this case, it is represented with a single dashed line separating
expressions.

Second, we can represent monoidal categories without a dashed line. This is
done by realizing that a monoidal product
$\displaystyle\otimes:\mathbf{C}\times\mathbf{C}\rightarrow\mathbf{C}$ grants
every object in $\displaystyle x\in\text{Ob}(\mathbf{C})$ a functor
$\displaystyle x\otimes\\_:\mathbf{C}\rightarrow\mathbf{C}$ and every morphism
$\displaystyle f:x\rightarrow y$ a natural transformation $\displaystyle
f\otimes\\_:x\otimes\\_\rightarrow y\otimes\\_$. This fact can be observed
from the diagram where $\displaystyle f$ can naturally move over
$\displaystyle h$, with components $\displaystyle f\otimes\text{Id}_{z}^{z}$
and $\displaystyle f\otimes\text{Id}_{w}^{w}$. This is shown in Figure 7.
Monoidal string diagrams, therefore, are functor string diagrams where
$\displaystyle x\otimes\\_$ is equivalently expressed by a wire labeled
$\displaystyle x$, and natural transformations $\displaystyle f\otimes\\_$ are
expressed with boxes labeled $\displaystyle f$ on functor wires $\displaystyle
x\otimes\\_$ to $\displaystyle y\otimes\\_$.

[capbesideposition=left,top]figure[]

Figure 7:  By a certain choice of equivalent expression, wherein we choose to
view monoidal products as functors and natural transformations, one can see
that monoidal string diagrams are a certain specification of functor string
diagrams.

If we look at how monoidal string diagrams are defined, we see that they obey
vertical section decomposition and equivalent expression. A survey by Selinger
[18] represents a standard introduction. It introduces the features of
monoidal string diagrams as re-expressions of symbolic categories, as seen in
appendix Figure 19.

Monoidal string diagrams focus on powerful algebraic correspondence between
diagrammatic and categorical expressions. This coherence property states that
well-formed diagrams exactly correspond to well-formed expressions. This
property is so powerful that it allows all the details of monoidal categories,
isomorphisms, associators, and so forth, to follow from the axioms of the
graphical language and planar isotopy. Monoidal string diagrams, then, can be
used to define what monoidal categories are and hence offer an extremely
powerful perspective from which to understand them.

This level of correspondence is not the focus of functor string diagrams.
Demanding this isotopy raises a barrier to diagramming and understanding
additional structures. To the degree that graphical intuition is useful, it is
far simpler to define properties by their local interactions. For instance,
natural transformations acting on functor wires follows from a simple local
definition. This approach, which does not demand isotopy, allows more features
to be introduced with each having some appropriate form of graphical
intuition. The graphical intuition embedded into equivalent expressions can
coincidentally serve as axioms for the feature, however, this is not the focus
of functor string diagrams.

Furthermore, functor string diagrams are sufficiently coherent. All
categorical expressions are, ultimately, a chain of morphisms acting on
objects, which vertical section decomposition ensures we can perfectly
represent. When certain expressions are isomorphic, it is often
straightforward to encode that isomorphism as an equivalent expression as we
ultimately demand that diagrams correspond to a unique categorical expression
up to isomorphism.

##  5 Neural Circuit Diagrams

Deep learning systems are on the forefront of human technology, with
applications in a variety of fields ranging from the scientific investigation
of protein folding [19, 5] to immense commercial developments. Currently, ad-
hoc methods are used to explain these systems, limiting our understanding and
obfuscating implementation. Existing categorical machine learning, and
theoretical machine learning more generally, often focus on toy models which
do not reflect those used in practice given the limit tools to express the
details of models [17, 8, 22, 20] (see also appendix Figure 18). This stems
from a fundamental limitation of monoidal string diagrams, as they are unable
to express both the details of separate data and the interaction of axes [7].
Functor string diagrams, then, have an exciting application regarding
formalizing neural circuit diagrams [2, 3], a categorical method for
comprehensively communicating deep learning models.

#### The Form of Deep Learning Models.

Deep learning models are functions between data types, meaning they are
expressions in Set. Typically, data types are tensors of an underlying data
type $\mathbb{R}$. Therefore, objects are almost always of hom-functors
applied to $\mathbb{R}$. We develop an equivalent expression, whereby wires
withour arrows represent those objects applied as hom-functors to
$\mathbb{R}$. This allows broadcasted functions between tensors to be
represented as in Figure 8. Furthermore, to represent functions
$\mathbb{R}\rightarrow\mathbb{R}$, we exploit that the $1$-hom-functor is an
isomorphism, letting it be freely introduced and removed (unsqueezing and
squeezing), which we introduce and remove with a wire with an arrowhead.

[capbesideposition=left,top]figure[]

Figure 8:  For deep learning models, objects typically consist of hom-functors
to $\mathbb{R}$. We develop an equivalent expression which lets this be
readily expressed.

#### Understanding Expressions with Indexes.

As we are operating in $\displaystyle\mathbf{Set}$, every object
$\displaystyle a$ has a set of coprojections given by its elements
$\displaystyle\bra{i_{a}}:1\rightarrow a$. Morphisms $\displaystyle
a\rightarrow b$ can be uniquely identified and freely constructed from pre-
compositions with elements. By the Yoneda lemma, natural transformations
$\displaystyle\mathbf{Set}(a,\\_)\rightarrow\mathbf{Set}(1,\\_)$ correspond to
elements of $\displaystyle\mathbf{Set}(1,a)$. Therefore, elements
$\displaystyle\bra{i_{a}}:1\rightarrow a$ become natural transformations
$\displaystyle|i^{\overleftarrow{a}}\rangle^{x}_{x}:\mathbf{Set}(a,x)\rightarrow\mathbf{Set}(1,x)$
which we call indexes. They form a set of projections (see appendix Figure 22
and 23), letting morphisms to $\textbf{Set}(a,x)$ be uniquely identified and
freely constructed from post-composition with indexes.

#### Broadcasting.

Using indexes, we can define and construct various useful forms of functions.
This includes broadcasting, corresponding to hom-functors, and inner
broadcasting, corresponding to broadcasting within tuple segments. These
expressions use continuous wires to indicate the naturality of indexes, as in
Figure 9.

[capbesideposition=left,top]figure[]

Figure 9:  Continuous wires implement naturality, letting natural
transformations, including indexes, be transmitted, defining these
expressions.

#### Natural Transformations.

By the Yoneda lemma, functions on sets $x\rightarrow y$ correspond to
functions manipulating the indexes of tensors
$\mathbb{R}^{y}\rightarrow\mathbb{R}^{x}$. We represent natural
transformations in neural circuit diagrams using hexagons (see Figure 10), and
note that they travel along continuous wires the same way as indexes (see
Figure 11). Continuous wires defined by naturality with indexes, then, can be
seen to grant naturality to natural transformations more generally.

[capbesideposition=left,top]figure[]

Figure 10:  Natural transformations can be shown with hexagons, and they have
naturality regarding hom-functor broadcasting.

[capbesideposition=left,top]figure[]

Figure 11:  By testing this inner broadcasting expression with indexes
$k_{b}$, we see that the naturality of indexes as indicated in Figure 9
implies a naturality of natural transformations along continuous wires.

#### Einops.

Einstein operations refers to a common class of rearranging and contracting
(summation) over axes. This covers a common class of operations, the details
of which are critical yet challenging to express with traditional methods.
Einstein operations can be expressed by associating desired input with output
indexes symbolically, such as with Einstein tensor notation or the Einops
package [16]. With neural circuit diagrams, they correspond to continuous
wiring associating input and output axes. These operations are linear and
often require an outer product to first be taken (element-wise
multiplication), which we diagram by terminating a single dotted line.

Figure 12: Natural operations, including indexes, travel along continuous
wires. This means the above expression, for some choice of indexes
$i_{\bar{x}}$, $j_{\bar{x}}$, $k_{h}$ placed after the Einstein operation, can
be seen to be a contraction over the indexes placed before the operation.

#### Scaled dot-product attention.

Using these tools, we can express a significant yet complicated deep learning
algorithm, multi-head attention, clearly. This algorithm requires interaction
between specific axes and independent data, which is hard to express with
traditional methods (see appendix Figure 24 for a typical presentation). With
neural circuit diagrams, the specific interactions for scaled dot-product
attention (SDPA) can be clearly seen as in Figure 13. Furthermore, multi-head
attention which performs SDPA $h$-times and aggregates the result, can be
shown be shown with neural circuit diagrams in Figure 14.

Figure 13: Scaled dot-product attention can be far more clearly expressed with
neural circuit diagrams. The interaction between axes and the changing shape
of data is naturally shown by diagrams. Figure 14: Conceptually, multi-head
attention involved broadcasting SDPA over $\displaystyle h$ parallel
operations and aggregating the result. The queries, keys, and valued are
produced and recombined by adding an additional axis to the learned linear
layers.

##  6 Conclusion

In this work, we stated the need for a general means of diagrammatically
expressing categorical ideas. Functor string diagrams are a general and
principled approach to developing categorical diagrams that have robust
mathematics for theoretical investigations, encompass existing monoidal string
diagram, and have clear utility for application. Compared to the existing
paradigm of monoidal string diagrams and string diagrams, functor string
diagrams offer a flexible yet principled approach that ensures complex ideas
can be clearly expressed. Their ability to express both functors and products
allows for theoretical ideas to be clearly understood, and allows for applied
systems to be straightforwardly modeled.

## References

  * [1]
  * [2] Vincent Abbott (2023): _Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures_.  Transactions on Machine Learning Research.  Available at https://openreview.net/forum?id=RyZB4qXEgt.
  * [3] Vincent Abbott (2023): _Robust Diagrams for Deep Learning Architectures: Applications and Theory_.  Honours Thesis, The Australian National University, Canberra.
  * [4] Samson Abramsky & Bob Coecke (2008): _Categorical quantum mechanics_ , 10.48550/arXiv.0808.1023.  Available at http://arxiv.org/abs/0808.1023.  ArXiv:0808.1023 [quant-ph].
  * [5] Jumper et al. (2021): _Highly accurate protein structure prediction with AlphaFold_.  Nature 596(7873), pp. 583–589, 10.1038/s41586-021-03819-2.  Available at https://www.nature.com/articles/s41586-021-03819-2.  Number: 7873 Publisher: Nature Publishing Group.
  * [6] John C. Baez & Mike Stay (2010): _Physics, Topology, Logic and Computation: A Rosetta Stone_.  813, pp. 95–172, 10.1007/978-3-642-12821-9-2.  Available at http://arxiv.org/abs/0903.0340.  ArXiv:0903.0340 [quant-ph].
  * [7] David Chiang, Alexander M. Rush & Boaz Barak (2022): _Named Tensor Notation_.  Transactions on Machine Learning Research.  Available at https://openreview.net/forum?id=hVT7SHlilx.
  * [8] Robin Cockett, Geoffrey Cruttwell, Jonathan Gallagher, Jean-Simon Pacaud Lemay, Benjamin MacAdam, Gordon Plotkin & Dorette Pronk (2019): _Reverse derivative categories_ , 10.48550/arXiv.1910.07065.  Available at http://arxiv.org/abs/1910.07065.  ArXiv:1910.07065 [cs, math].
  * [9] G. S. H. Cruttwell, Bruno Gavranović, Neil Ghani, Paul Wilson & Fabio Zanasi (2021): _Categorical Foundations of Gradient-Based Learning_ , 10.48550/arXiv.2103.01931.  Available at http://arxiv.org/abs/2103.01931.  ArXiv:2103.01931 [cs, math].
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren & Jian Sun (2015): _Deep Residual Learning for Image Recognition_.  CoRR abs/1512.03385.  Available at http://arxiv.org/abs/1512.03385.  ArXiv: 1512.03385.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren & Jian Sun (2016): _Identity Mappings in Deep Residual Networks_.  In Bastian Leibe, Jiri Matas, Nicu Sebe & Max Welling, editors: Computer Vision \- ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, Lecture Notes in Computer Science 9908, Springer, pp. 630–645, 10.1007/978-3-319-46493-0_38.  Available at https://doi.org/10.1007/978-3-319-46493-0_38.
  * [12] Dan Marsden (2014): _Category Theory Using String Diagrams_.  CoRR abs/1401.7220.  Available at http://arxiv.org/abs/1401.7220.  ArXiv: 1401.7220.
  * [13] Jose Meseguer & Ugo Montanari (1990): _Petri nets are monoids_.  Information and Computation 88(2), pp. 105–155, 10.1016/0890-5401(90)90013-8.  Available at https://www.sciencedirect.com/science/article/pii/0890540190900138.
  * [14] Kenji Nakahira (2023): _Diagrammatic category theory_ , 10.48550/arXiv.2307.08891.  Available at http://arxiv.org/abs/2307.08891.  ArXiv:2307.08891 [math].
  * [15] Robin Piedeleu & Fabio Zanasi (2023): _An Introduction to String Diagrams for Computer Scientists_ , 10.48550/arXiv.2305.08768.  Available at http://arxiv.org/abs/2305.08768.  ArXiv:2305.08768 [cs].
  * [16] Alex Rogozhnikov (2021): _Einops: Clear and Reliable Tensor Manipulations with Einstein-like Notation_.  Available at https://openreview.net/forum?id=oapKSVM2bcj.
  * [17] Andrew M. Saxe, James L. McClelland & Surya Ganguli (2014): _Exact solutions to the nonlinear dynamics of learning in deep linear neural networks_.  In Yoshua Bengio & Yann LeCun, editors: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.  Available at http://arxiv.org/abs/1312.6120.
  * [18] Peter Selinger (2009): _A survey of graphical languages for monoidal categories_ , 10.1007/978-3-642-12821-9-4.  Available at https://arxiv.org/abs/0908.3347v1.
  * [19] Andrew W. et al. Senior (2020): _Improved protein structure prediction using potentials from deep learning_.  Nature 577(7792), pp. 706–710, 10.1038/s41586-019-1923-7.  Available at https://www.nature.com/articles/s41586-019-1923-7.  Number: 7792 Publisher: Nature Publishing Group.
  * [20] Dan Shiebler, Bruno Gavranovic & Paul W. Wilson (2021): _Category Theory in Machine Learning_.  CoRR abs/2106.07032.  Available at https://arxiv.org/abs/2106.07032.  ArXiv: 2106.07032.
  * [21] Ashish et al. Vaswani (2017): _Attention is all you need_.  In: Advances in neural information processing systems, 30, Curran Associates, Inc.  Available at https://proceedings.neurips.cc/paper-files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * [22] Paul Wilson & Fabio Zanasi (2022): _Categories of Differentiable Polynomial Circuits for Machine Learning_ , 10.48550/arXiv.2203.06430.  Available at http://arxiv.org/abs/2203.06430.  ArXiv:2203.06430 [cs, math].

## Appendix

##  Appendix A Current Categorical Diagrams

Figure 15: A traditional monoidal string diagram from page 18 of [15]. This
expression clearly shows the objects and morphisms at each stage, and has
graphical isotopy whereby topological transforms of diagrams lead to valid,
isomorphic expressions. However, it does not show the details of axes should
the objects be Cartesian tuples. Furthermore, it cannot show functors which
are not monoidal products of objects nor natural transformations which are not
morphisms between the monoidal product of objects. The useful properties of
monoidal string diagrams are encompassed by functor string diagrams, as shown
in Section 4. Figure 16: A string diagram from Marsden [12]. These expressions
use coloured regions to show the current category, and display functors as
left-to-right boundaries between coloured regions. Natural transformations are
dots on functors or their intersection. These can clearly display the action
of functors, natural transformations, and more complicated structures such as
adjunctions, monads, and (co)limits, and display graphical isotopy,
implementing the axioms of these structures graphically. However, they fail to
be clearly decomposable for applied cases, and have a cumbersome
representation of products. The behaviour of functors and natural
transformations using functor string diagrams are shown in Section 2. Figure
17: Nakahira’s diagrams [14] extends Marsden’s diagrams to consider
substiutions and various algebraic proofs, including the Yoneda lemma shown
above. These diagrams have powerful tools for the consideration of products,
bifunctors, and other constructs. However, streamlined simplicity is preferred
for practical application and intuitive understanding, which is shown in the
functor string diagram proof for the Yoneda lemma in Section 3. Figure 18:
Categorical machine learning such as from [9] uses monoidal string diagrams
with Cartesian products, along with the Para construct which allows parameters
to be introduced from above. These do not express the details of axes, and
hence focus on toy models rather than those used in practice. Since 2015 [10,
11], for example, deep learning has employed residual connections which are
significantly different from linear-bias-activation toy models. Functor string
diagrams underly neural circuit diagrams (see Section 5), which can rigorously
and comprehensively express deep learning models.

##  Appendix B Equivalent Expression of Products

With functor string diagrams, we can express bifunctors
$\textbf{C}\times\textbf{C}\rightarrow\textbf{C}$ as bolded functor wires over
an expression in $\textbf{C}\times\textbf{C}$, as in Figure 19. As we can see,
functor string diagrams provide a flexible framework to understand the
connections between various means of viewing expressions.

Figure 19: These equivalent expressions for the diagramming of product
categories and monoidal products offer immense utility. We see how different
expressions are related and can freely adopt whichever approach is the most
useful.

##  Appendix C Typical Introduction to Monoidal String Diagrams

Typical introductions to monoidal string diagrams introduce the fundamental
structures by elaborating how objects and morphisms are represented. Below,
tables 1 and 2 from [18] are shown in Figure 20 and 21. This lets us see that
monoidal string diagrams are, fundamentally, diagrams with vertical sections
corresponding to objects and morphisms equipped with equivalent expressions.

[capbesideposition=left,top]figure[]

Figure 20:  Table 1 from [18], “The graphical language of categories”.

[capbesideposition=left,top]figure[]

Figure 21:  Table 2 from [18], “The graphical language of monoidal
categories”.

##  Appendix D Elements, Indexes, and Projections

[capbesideposition=left,top]figure[]

Figure 22:  Elements of a set $1\rightarrow a$ form a set of coprojections
which can uniquely identify and construct functions $a\rightarrow\\_$.

[capbesideposition=left,top]figure[]

Figure 23:  The associated natural transformations
$\textbf{Set}(a,\\_)\rightarrow\\_$, for a set of projections, letting
functions $\\_\rightarrow\textbf{Set}(a,\\_)$ be uniquely identified.

##  Appendix E Typical Presentation of Attention

A typical presentation of scaled dot-product attention can be drawn from [21]
are shown in Figure 24. This work introduced transformer models. However, its
diagrams are difficult to understand, especially with regards to the
interaction of axes and the shape of data throughout the model. As can be
seen, Figure 13 and 14 are a significant improvement.

Scaled Dot-Product Attention

Multi-Head Attention

Figure 24: The prevailing ad-hoc diagrams used to represent scaled dot-product
attention and multi-head attention obfuscate the interaction of axes and how
the shape of data changes across operations, making understanding and
implementation difficult.