# Homogeneous Vector Bundles and
$G$-equivariant Convolutional Neural Networks

Jimmy Aronsson  Chalmers University of Technology, Department of Mathematical
Sciences
SE-412 96 Gothenburg, Sweden<EMAIL_ADDRESS>
###### Abstract.

$G$-equivariant convolutional neural networks (GCNNs) is a geometric deep
learning model for data defined on a homogeneous $G$-space $\mathcal{M}$.
GCNNs are designed to respect the global symmetry in $\mathcal{M}$, thereby
facilitating learning. In this paper, we analyze GCNNs on homogeneous spaces
$\mathcal{M}=G/K$ in the case of unimodular Lie groups $G$ and compact
subgroups $K\leq G$. We demonstrate that homogeneous vector bundles is the
natural setting for GCNNs. We also use reproducing kernel Hilbert spaces to
obtain a precise criterion for expressing $G$-equivariant layers as
convolutional layers. This criterion is then rephrased as a bandwidth
criterion, leading to even stronger results for some groups.

###### Contents

  1. 1 Introduction
  2. 2 Foundations of equivariant neural networks
    1. 2.1 Convolutional neural networks
    2. 2.2 Gauge theory and the equivariant framework
  3. 3 $G$G-equivariant convolutional neural networks
    1. 3.1 Homogeneous spaces
    2. 3.2 Homogeneous vector bundles
    3. 3.3 Induced representations
    4. 3.4 $G$G-equivariant and convolutional layers
    5. 3.5 RKHS and bandlimited functions
      1. 3.5.1 Abelian groups
      2. 3.5.2 Compact groups
  4. 4 Discussion

##  1\. Introduction

Developments in deep learning have increased dramatically in recent years.
Even though multilayer perceptrons Alpaydin (2014) and other general-
architecture models work well for some tasks, achieving higher levels of
performance often requires models that are more tailored to each application,
and which incorporate some level of understanding of the data. _Geometric deep
learning_ Bronstein et al. (2017, 2021); Cao et al. (2020); Cohen et al.
(2019); Monti et al. (2017) is the approach of using inherent geometric
structure in data, and symmetry derived from geometry, to improve deep
learning models.

_Convolutional neural networks (CNNs)_ are among the simplest and most broadly
applicable general-architecture models. They have been successfully applied to
image classification and segmentation Peng et al. (2017); Xu et al. (2017);
Zhang et al. (2018), text summarization Song et al. (2019), pose estimation
Mehta et al. (2017), sign language recognition Koller et al. (2018), and many
other tasks. One reason why CNNs are so useful is that _convolutional layers_
, the basic building blocks of CNNs, commute with the translation operator in
$\mathbb{Z}^{2}$; convolutional layers are _translation equivariant_. In image
classification tasks, for instance, $\mathbb{Z}^{2}$ represents the underlying
pixel lattice, and translation equivariance helps CNNs identify objects in
images regardless of their exact pixel coordinates. As convolutional layers
respect the global translation symmetry in $\mathbb{Z}^{2}$, CNNs are examples
of geometric deep learning models.

_$G$ -equivariant convolutional neural networks (GCNNs)_ Cohen and Welling
(2016); Cohen et al. (2018b) are generalizations of CNNs to data points
defined on homogeneous $G$-spaces $\mathcal{M}$. Convolutional layers that
commute with the action $G\times\mathcal{M}\to\mathcal{M}$ of the global
symmetry group $G$, remove the need for GCNNs to learn about the global
symmetry. It is already built into the network. This enables GCNNs to focus on
learning other relevant features in data, potentially improving performance.
One example is the detection of tumors in digital pathology. Images of tumors
can have any orientation, and GCNNs with both translation and rotation
equivariant layers have higher accuracy than ordinary CNNs Veeling et al.
(2018). Rotation equivariance is also highly useful in 3D inference problems
Worrall and Brostow (2018), in point cloud recognition Li et al. (2019), and
in other tasks.

_Gauge equivariant_ neural networks Cheng et al. (2019); Cohen et al. (2019);
Favoni et al. (2020); Luo et al. (2021) are instead designed to respect
_local_ symmetries. For example, computations involving vector fields - in
meteorology or other areas - require vectors to be expressed in components.
This requires a frame; a smooth assignment of a basis to each tangent space.
However, the sphere and other non-parallelizable manifolds do not admit a
global frame, so the computations must be performed locally, using different
local frames for different regions on the manifold. It is then important that
any numerical results obtained in one frame are compatible with those obtained
in any other frame on overlapping regions. In other words, the computations
should be equivariant with respect to the choice of local frame, which is a
viewed as a gauge degree of freedom; a local symmetry. Gauge equivariant
neural networks have also been introduced for problems exhibiting other local
symmetries, primarily in lattice gauge theory.

In this paper, we study the mathematical foundations of GCNNs and characterize
convolutional layers in terms of more abstract layers. Our contributions are
threefold:

  * •

We analyze a general framework that include both gauge equivariant neural
networks and GCNNs, that only differ in whether layers respect a local gauge
symmetry or a global translation symmetry. Moreover, we show that GCNNs are
naturally expressed in terms of homogeneous vector bundles.

  * •

In general, not all $G$-equivariant layers can be written as convolutional
layers. We investigate the relation between these types of layers for all
homogeneous spaces $\mathcal{M}=G/K$ when $G$ is a unimodular Lie group and
$K\leq G$ is a compact subgroup. As a result of this investigation, we find a
criterion for expressing $G$-equivariant layers as convolutional layers
(Theorem 14).

  * •

We highlight the close relationship between convolutional layers in GCNNs,
reproducing kernel Hilbert spaces (RKHS), and bandwidth. We reformulate the
criterion in Theorem 14 as a bandwidth criterion and prove that, when $G$ is
discrete abelian or finite,111Discrete Lie groups are countable, in this
paper, as we assume smooth manifolds to be Hausdorff and second-countable. all
$G$-equivariant layers are indeed convolutional layers (Corollaries 19-21).

This work was inspired by a number of papers Cheng et al. (2019); Cohen and
Welling (2016); Cohen et al. (2018a, b, 2019); Weiler et al. (2018). The
theoretical papers Cheng et al. (2019); Cohen et al. (2018b) have been of
particular importance, as our work grew from a desire to understand the
mathematics of equivariant neural networks in even greater detail.

In the case of compact groups $G$, the Peter-Weyl theorem and other powerful
tools have allowed researchers to study GCNNs using harmonic analysis. Among
the most well-known results in this direction is Theorem 1 in Kondor and
Trivedi (2018), which uses Fourier analysis on $G$ to establish that the
layers in a $G$-equivariant feed-forward neural network must be generalized
convolutional layers, when $G$ is compact. This result is similar to our
second contribution above and we discuss the distinction in Section 3.4.
Others have used the well-known representation theory of the compact group
$G=SO(3)$ to study rotation equivariant GCNNs for spherical data Esteves
(2020); Esteves et al. (2017, 2018).

The paper is structured as follows. We summarize the relevant machine learning
background in Section 2.1, and discuss a framework for equivariant neural
networks in Section 2.2. In Section 3, we restrict attention to homogeneous
spaces $G/K$ where $G$ is a unimodular Lie group and $K\leq G$ is a compact
subgroup. Section 3.2 explains the relation between GCNNs, homogeneous vector
bundles, and induced representations. This relation is used to motivate the
definition of $G$-equivariant layers in Section 3.4, where we also discuss
convolutional layers and prove the aforementioned Theorem 14; this result
characterizes when a $G$-equivariant layer is a convolutional layer, in terms
of RKHS. Section 3.5 then relates RKHS to bandwidth, leading to a
reformulation of Theorem 14 (Corollary 19) as well as a few stronger results.
Finally, in Section 4, we summarize our work and end with a discussion.

##  2\. Foundations of equivariant neural networks

In this section, we give an introduction to convolutional neural networks
(CNNs) and discuss a simple framework for equivariant neural networks.

###  2.1. Convolutional neural networks

CNNs were first introduced in 1979 under the name of Neocognitrons, and were
used to study visual pattern recognition Fukushima and Miyake (1982). In the
1990s, CNNs were successfully applied to problems such as automatic
recognition of handwritten digits LeCun et al. (1995) and face recognition
Lawrence et al. (1997). However, it was arguably not until 2012, when the GPU-
based AlexNet CNN outperformed all competition on the ImageNet Large Scale
Visual Recognition Challenge Krizhevsky et al. (2012), that CNNs and other
neural networks truly caught the public eye. Industrial work and academic
research on deep learning has since soared, and current state-of-the-art deep
learning architectures are significantly more powerful and more complex than
AlexNet. Yet, convolutional layers remain important components.

In this introduction, we focus on data that can be represented by finitely
supported functions

$f:\mathbb{Z}^{2}\to\mathbb{R}^{m}.$ (2.1)

Digital images, for example, are of this form since each pixel
$x\in\mathbb{Z}^{2}$ is associated with a color array $f(x)\in\mathbb{R}^{m}$,
and finite support is analogous to finite image resolution. Note that $m=1$
corresponds to grayscale images and $m=3$ to RGB images, but we allow any
number of channels $m$. In general, any data represented by a finite 2D
($m=1$) or 3D ($m>1$) array with real-valued entries is of the form (2.1).

_Convolutional layers_ act on data points (2.1) by222The name _convolutional
layer_ is used even though (2.4) more closely resembles a cross-correlation.
It can be expressed as a convolution if we replace the kernel with its
involution $\kappa^{*}(y)=\kappa(-y)$.

$[\kappa\star f](x)=\sum_{y\in\mathbb{Z}^{2}}\kappa(y-x)f(y),$ (2.2)

given a matrix-valued kernel
$\kappa:\mathbb{Z}^{2}\to\mathrm{Hom}(\mathbb{R}^{m},\mathbb{R}^{n})$ for some
$n\in\mathbb{N}$. The kernel is also finitely supported in practice, so the
maps $\kappa\star f:\mathbb{Z}^{2}\to\mathbb{R}^{n}$ are themselves data
points (2.1) with $n$ channels. Broadly speaking, CNNs consist of
convolutional layers (2.2) combined with other transformations, such as non-
linear activation functions and batch normalization layers. We are mainly
interested in convolutional layers, so we do not go into detail about non-
linear activation functions or other types of layers. For more extensive
descriptions of CNNs, see Albawi et al. (2017); Goodfellow et al. (2016); Wu
(2017).

In image classification tasks, for instance, CNNs categorize digital images
into a predefined number $k$ of distinct classes, based on what the images
depict. The CNN maps each digital image $f:\mathbb{Z}^{2}\to\mathbb{R}^{m}$ to
a probability vector in $\mathbb{R}^{k}$ estimating the probability that $f$
belongs to any given class. During training, this probability vector is
compared to the correct answer (which is known) and the discrepancy is
computed using a _loss_ norm or distance function. A gradient descent-based
algorithm minimizes the loss function, thereby _learning_ the kernel matrix
elements and any other trainable network parameters. The result of this
training procedure is a CNN that accurately classifies images in the training
data set. Finally, the predictive power of the CNN is evaluated by using it to
classify images from a test data set; images that were not used during
training and which the CNN has not encountered before.

CNNs perform very well on image classification and similar machine learning
tasks, and are important parts of many state-of-the-art network architectures
on such tasks Brock et al. (2021); Jia et al. (2021); Tan and Le (2021); Wu et
al. (2021). One reason for their success is _translation equivariance_ :
Convolutional layers (2.2) commute with the translation operator in the image
plane,

$L_{x}:\mathbb{Z}^{2}\to\mathbb{Z}^{2},\qquad L_{x}(y)=y+x,\qquad
x\in\mathbb{Z}^{2}.$ (2.3)

Translation equivariance makes CNNs agnostic to the specific locations of
individual pixels, while still taking into account the relative positions of
different pixels; images are more easily classified based on relevant features
of their subjects, and not based on technical artifacts such as specific pixel
coordinates. This observation motivates the introduction of more general
convolutional layers that act equivariantly on data points $f:\mathcal{M}\to
V$, where the domain $\mathcal{M}$ is homogeneous with respect to a locally
compact group $G$ Cohen and Welling (2016); Cohen et al. (2018b). Given
finite-dimensional vector spaces $V,W$, convolutional layers are defined as
certain vector-valued integrals333For a summary on vector-valued integration
on locally compact groups, see (Folland, 2016, Appendix 4).

$\kappa\star f:\mathcal{M}\to W,\qquad(\kappa\star
f)(g)=\int_{G}\kappa(g^{-1}g^{\prime})f(g^{\prime})\ \differential
g^{\prime},$ (2.4)

with operator-valued kernels $\kappa:G\to\mathrm{Hom}(V,W)$.

######  Remark 1.

In (2.4), we integrate with respect to a Haar measure on the unimodular Lie
group $G$.

Broadly speaking, $G$-equivariant convolutional neural networks (GCNNs)
consist of sequences of convolutional layers (2.4) mixed with non-linear
activation functions, and possibly other layers that are equivariant with
respect to the global symmetry. This characterization is intentionally vague
as we want to avoid making unnecessarily restrictive assumptions on the
layers. For this reason, we will not study GCNNs from a holistic perspective,
as a sequence of multiple layers, but instead focus on individual layers. We
give a formal definition of abstract, $G$-equivariant layers in Definition 9,
before defining the more specific convolutional layers in Definition 10.

###  2.2. Gauge theory and the equivariant framework

Before going into detail about GCNNs in Section 3, let us describe a
mathematical framework for equivariant neural networks. The framework is based
on gauge theoretic concepts but is equally suitable for GCNNs. Both gauge
equivariant neural networks and GCNNs will thus be described by this
framework, their main difference being the specific equivariance properties
imposed on layers.

######  Remark 2.

This framework is already being used in GCNNs and gauge equivariant neural
networks separately Cheng et al. (2019); Cohen et al. (2018b). We are simply
presenting the unified theory that includes both types of equivariance as
separate cases.

Gauge theory originated in physics as a way to model local symmetry. In
quantum electrodynamics (QED), for example, the electron wave function can be
locally phase shifted, $\psi\mapsto e^{i\alpha}\psi$, with no physically
observable consequence, and so QED is said to possess a $U(1)$ gauge symmetry.
Mathematicians have later adopted gauge theory in order to study other types
of local symmetries. The introduction of gauge equivariant deep learning
models has been suggested by deep learning practitioners and physicists alike.
For example, Cheng et al. (2019) investigates the structure of gauge
equivariant layers used for vector fields, tensor fields, and more general
fields. Physicists have introduced gauge equivariant neural networks for
applications in, e.g., lattice gauge theory Boyda et al. (2021); Favoni et al.
(2020); Luo et al. (2021).

We assume some familiarity with fiber bundles,444Introductions to fiber
bundles can be found in Kolár et al. (2013); Lee (2013); Nakahara (2003). but
we still present a few relevant definitions and examples.

######  Definition 1.

Let $K$ be a Lie group. A smooth fiber bundle $\pi:P\to\mathcal{M}$ is called
a _principal $K$-bundle_ with _structure group_ $K$ if there is a free, smooth
right $K$-action

$P\times K\to P,\qquad(p,k)\mapsto p\triangleleft k,$ (2.5)

with the following properties for each $x\in\mathcal{M}$.

  1. (i)

Let $P_{x}=\pi^{-1}(\\{x\\})$ be the fiber at $x$. Then

$p\in P_{x},\ k\in K\quad\Rightarrow\quad p\triangleleft k\in P_{x}.$ (2.6)

That is, the $K$-action preserves fibers.

  2. (ii)

For each $p\in P_{x}$, the mapping $k\mapsto p\triangleleft k$ is a
diffeomorphism $K\to P_{x}$.

Principal bundles are natural tools for understanding local symmetries, i.e.,
gauge degrees of freedom. In theoretical physics, gauge degrees of freedom are
redundancies in the mathematical theory with no physical relevance. This is
both a blessing and a curse: Solving the Yang-Mills equations of motion as an
initial value problem, for example, is an underdetermined problem that cannot
be solved without taking the gauge degrees of freedom into account; without
choosing a _gauge_ Müller (2019). This is similar to our example in the
introduction, that computations involving vector fields may require a choice
of basis in each tangent space, even if this choice is irrelevant for the
underlying application. On the other hand, problems may also become easier to
solve by choosing a gauge with some finesse.555See, for example, the temporal
gauge in lattice gauge theory (Gattringer and Lang, 2009, §3.3.2).

######  Definition 2.

Let $\pi:P\to\mathcal{M}$ be a principal $K$-bundle and assume
$U\subseteq\mathcal{M}$ is open.

  1. (i)

A _gauge_ is a local section $\omega:U\to P$.

  2. (ii)

A _gauge transformation_ is an automorphism $\chi:P\to P$ that is equivariant,

$\chi(p\triangleleft k)=\chi(p)\triangleleft k,\qquad p\in P,k\in K,$ (2.7)

and which preserves fibers: $\pi\circ\chi=\pi$.

We will go into more detail about the vector field example in Example 1.
However, we first need to define associated bundles. To this end, let
$\pi:P\to\mathcal{M}$ be a principal $K$-bundle and let $\rho:K\to
GL(V_{\rho})$ be a finite-dimensional representation. Define an equivalence
relation $\sim$ on $P\times V_{\rho}$ by

$(p,v)\sim(p\triangleleft k,\rho(k)^{-1}v),\qquad p\in P,v\in V_{\rho},k\in
K.$ (2.8)

Let $P\times_{\rho}V_{\rho}=(P\times V_{\rho})/\sim$ denote the quotient
space, whose elements are equivalence classes

$[p,v]=[p\triangleleft k,\rho(k)^{-1}v],\qquad p\in P,v\in V_{\rho},k\in K,$
(2.9)

and consider the projection $\pi_{\rho}:P\times_{\rho}V_{\rho}\to\mathcal{M}$
defined by $\pi_{\rho}([p,v])=\pi(p)$. Observe that each fiber
$\pi_{\rho}^{-1}(\\{x\\})$ has a natural vector space structure such that the
mapping

$V_{\rho}\to\pi_{\rho}^{-1}(\\{x\\}),\qquad v\mapsto[p,v],$ (2.10)

is a linear isomorphism for each fixed $p\in P_{x}$.

######  Lemma 1 ((Kolár et al., 2013, §10.7)).

The _associated bundle_ $\pi_{\rho}:P\times_{\rho}V_{\rho}\to\mathcal{M}$ is a
smooth vector bundle.

######  Example 1.

Let $d=\dim\mathcal{M}$ and consider a coordinate chart
$\left(u^{1},\ldots,u^{d}\right):U\to\mathbb{R}^{d}$, for
$U\subseteq\mathcal{M}$. Recall that for each $x\in U$, the induced coordinate
basis in $T_{x}\mathcal{M}$,

$\omega(x):=\left(\frac{\partial}{\partial
u^{1}}\Big{|}_{x},\ldots,\frac{\partial}{\partial u^{d}}\Big{|}_{x}\right),$
(2.11)

lets us express tangent vectors $X_{x}\in T_{x}\mathcal{M}$ in components
$X_{x}^{1},\ldots,X_{x}^{d}\in\mathbb{R}$. Moreover, (2.11) defines a local
frame $\omega:U\to F\mathcal{M}$ that sends each point $x\in U$ to its
coordinate basis in $T_{x}\mathcal{M}$. Local frames are sections of the frame
bundle $F\mathcal{M}$, which is a principal $GL(d,\mathbb{R})$-bundle, hence
local frames are examples of gauges.

Fix $x\in U$ and expand $X_{x}\in T_{x}\mathcal{M}$ in the coordinate basis:

$X_{x}=\sum_{i=1}^{d}X_{x}^{i}\frac{\partial}{\partial u^{i}}\Big{|}_{x}.$
(2.12)

By decomposing (2.12) into components and basis vectors, we can view the
tangent vector $X_{x}$ as one giant tuple

$X_{x}=\left(\frac{\partial}{\partial
u^{1}}\Big{|}_{x},\ldots,\frac{\partial}{\partial
u^{d}}\Big{|}_{x},X_{x}^{1},\ldots,X_{x}^{d}\right)=(\omega(x),X(x))\in
F\mathcal{M}\times\mathbb{R}^{d}.$ (2.13)

Another choice of coordinate chart produces another local frame
$\omega^{\prime}:U^{\prime}\to F\mathcal{M}$ and another decomposition (2.12),
assuming that $x\in U^{\prime}$. These decompositions are related by a change
of basis

$\left(\omega^{\prime}(x),X^{\prime}(x)\right)=(\omega(x)B(x),B(x)^{-1}X(x)),$
(2.14)

for some $B(x)\in GL(d,\mathbb{R})$. Now observe that (2.14) is of the form

$(p^{\prime},v^{\prime})=(p\triangleleft k,\rho(k)^{-1}v),$ (2.15)

where $p,p^{\prime}\in F\mathcal{M}$, $v,v^{\prime}\in\mathbb{R}^{d}$, $k\in
GL(d,\mathbb{R})$, the right-action $\triangleleft$ is right-multiplication,
and $(\rho,\mathbb{R}^{d})$ is the standard representation $\rho(k)=k$ of
$GL(d,\mathbb{R})$. The basis-dependent description (2.12) of $X_{x}$ thus
resemble the pairs $(p,v)$ in the construction of associated bundles. Passing
to the quotient $(F\mathcal{M}\times_{\rho}\mathbb{R}^{d})/\sim$ instead gives
a basis-_in_ dependent description of $X_{x}$, since it identifies all
possible decompositions (2.12) in all possible bases. That is, the tangent
bundle is isomorphic to $F\mathcal{M}\times_{\rho}\mathbb{R}^{d}$. ∎

Equivariant neural networks use the language of principal and associated
bundles. In the remainder of this subsection, let
$E_{\rho}=P\times_{\rho}V_{\rho}$ and $E_{\sigma}=P\times_{\sigma}V_{\sigma}$
be associated bundles, given a principal bundle $\pi:P\to\mathcal{M}$ over a
smooth manifold $\mathcal{M}$. Further let $\Gamma_{c}(E_{\rho})$ and
$\Gamma_{c}(E_{\sigma})$ be the vector spaces of compactly supported
continuous sections of $E_{\rho}$ and $E_{\sigma}$, respectively.

######  Definition 3.

A _data point_ is a section $s\in\Gamma_{c}(E_{\rho})$.

######  Remark 3.

Even though data is typically real-valued, we primarily consider complex
representations $(\rho,V_{\rho})$ so to simplify the mathematical theory. The
harmonic analysis in Section 3.5 especially benefits from this choice.

Our decision to restrict attention to compactly supported sections was also
made for mathematical reasons: $G$-equivariant layers are defined in Section
3.4 in terms of an induced representation, which lives on the completion of
$\Gamma_{c}(E_{\rho})$ with respect to a certain inner product. This is not a
serious restriction from an application viewpoint.

######  Definition 4.

A _feature map_ is a compactly supported continuous map $f:P\to V_{\rho}$ that
satisfies the transformation property

$f(p\triangleleft k)=\rho(k)^{-1}f(p),$ (2.16)

for all $p\in P$, $k\in K$. The vector space of such feature maps is denoted
$C_{c}(P;\rho)$

Data points and feature maps are, in a sense, dual to each other: Each data
point in $\Gamma_{c}(E_{\rho})$ is of the form

$s_{f}(x)=[p,f(p)],$ (2.17)

for a feature map $f\in C_{c}(P;\rho)$, where $p\in P_{x}$ is any element of
the fiber at $x\in\mathcal{M}$. Note that (2.17) does not depend on the choice
of $p$: Given another element $p^{\prime}\in P_{x}$, there exists a unique
$k\in K$ such that $p^{\prime}=p\triangleleft k$ and

$[p^{\prime},f(p^{\prime})]=[p\triangleleft k,f(p\triangleleft
k)]=[p\triangleleft k,\rho(k^{-1})f(p)]=[p,f(p)].$ (2.18)

That is, the equivalence class $[p,f(p)]$ only depends on the basepoint $x$.

######  Lemma 2 ((Kolár et al., 2013, §10.12)).

The linear map $C_{c}(P;\rho)\to\Gamma_{c}(E_{\rho})$, $f\mapsto s_{f}$ is a
vector space isomorphism.

We are almost ready to define general and gauge equivariant layers. Before
doing so, however, we must say how gauge transformations $\chi:P\to P$ act on
data points. Let $\theta_{\chi}:P\to K$ be the uniquely defined map satisfying
$\chi(p)=p\triangleleft\theta_{\chi}(p)$ for all $p\in P$, and define the
following action on the associated bundle $E_{\rho}$:

$\chi\cdot[p,v]=[\chi(p),v]=[p,\rho(\theta_{\chi}(p))v],\qquad[p,v]\in
E_{\rho}.$ (2.19)

The corresponding action on data points is given by

$(\chi\cdot
s_{f})(p)=[p,\rho(\theta_{\chi}(p))f(p)]=s_{\rho(\theta_{\chi})f}(x),\qquad
s_{f}\in\Gamma_{c}(E_{\rho}).$ (2.20)

We distinguish between general layers and more specific gauge equivariant
layers, as $G$-equivariant layers in GCNNs will only be a special case of the
former.

######  Definition 5.

Let $E_{\rho}=P\times_{\rho}V_{\rho}$ and
$E_{\sigma}=P\times_{\sigma}V_{\sigma}$ be associated bundles.

  1. (i)

A _(linear) layer_ is a linear map
$\Phi:\Gamma_{c}(E_{\rho})\to\Gamma_{c}(E_{\sigma})$.

  2. (ii)

A layer $\Phi$ is _gauge equivariant_ if, for all gauge transformations
$\chi:P\to P$,

$\Phi\circ\chi=\chi\circ\Phi.$ (2.21)

In equivariant neural networks, data points are sent through a sequence of
layers, which are mixed with non-linear activation functions. Again, we focus
on individual layers in this paper, and leave the analysis of equivariant
activation functions and multi-layer networks for an upcoming paper Gerken et
al. (Forthcoming). The fiber bundle-theoretic concepts discussed in this part
describe two kinds of equivariant neural networks:

  1. (i)

Gauge equivariant neural networks, which respect local gauge symmetry and
whose layers are gauge equivariant.

  2. (ii)

GCNNs, which respect global translation symmetry in homogeneous $G$-spaces
$\mathcal{M}$, and whose layers are $G$-equivariant (Definition 9).

######  Remark 4.

Our definition of layers is almost identical to the linear maps in Cheng et
al. (2019). The difference is that we focus on compactly supported sections,
whereas Cheng et al. (2019) use sections that are supported on a single
coordinate chart. Also, Cheng et al. (2019) investigates the structure of
their linear maps under additional assumptions of so-called locality,
covariance, and weight-sharing. Covariance is analogous to gauge equivariance
in our setting.

A consequence of Lemma 2 is that (gauge equivariant) layers
$\Phi:\Gamma_{c}(E_{\rho})\to\Gamma_{c}(E_{\sigma})$ induce a unique linear
map $\phi:C_{c}(P;\rho)\to C_{c}(P;\sigma)$ such that $\Phi s_{f}=s_{\phi f}$.
Writing data points as $s_{f}=[\cdot,f]$ allows us to also express this
relation as $\Phi[\cdot,f]=[\cdot,\phi f]$. We think of $\Phi$ and $\phi$ as
two sides of the same coin, and use the name (gauge equivariant) layer for
both maps.

${\Gamma_{c}(E_{\rho})}$${\Gamma_{c}(E_{\sigma})}$${C_{c}(P;\rho)}$${C_{c}(P;\sigma)}$$\scriptstyle{\Phi}$$\scriptstyle{\phi}$

######  Example 2.

Let $T:V_{\rho}\to V_{\sigma}$ be a linear transformation and consider the
layer

$\phi:C_{c}(P;\rho)\to C_{c}(P;\sigma),\qquad(\phi f)(p)=T\big{(}f(p)\big{)},$
(2.22)

for $p\in P$, $f\in C_{c}(P;\rho)$. Since $f$ and $\phi f$ are feature maps
and thereby satisfy (2.16), the linear transformation $T$ must satisfy

$\displaystyle\sigma(k)T\big{(}f(p)\big{)}=T\big{(}f(p\triangleleft
k^{-1})\big{)}=T\big{(}\rho(k)f(p)\big{)},$ (2.23)

for all $k\in K,p\in P,f\in C_{c}(P;\rho)$. This can be seen to imply that
$\sigma\circ T=T\circ\rho$, so $T$ intertwines the representations $\rho$ and
$\sigma$. Another way to arrive at this conclusion is to analyze when the
corresponding layer

$\Phi:\Gamma_{c}(E_{\rho})\to\Gamma_{c}(E_{\sigma}),\qquad\Phi
s_{f}=[\cdot,\phi f],$ (2.24)

is well-defined.

Now consider a gauge transformation $\chi:P\to P$ and its induced map
$\theta_{\chi}:P\to K$. Because $T$ is an intertwiner,

$(\Phi\circ\chi)s_{f}=s_{\phi\rho(\theta_{\chi})f}=s_{\sigma(\theta_{\chi})\phi
f}=(\chi\circ\Phi)s_{f},$ (2.25)

hence the layer $\Phi$ is automatically gauge equivariant. ∎

As this example illustrates, gauge equivariance is tightly connected to
intertwining properties of $\phi$. Rearranging (2.25) gives the following
result.

######  Lemma 3.

A general layer $\Phi:\Gamma_{c}(E_{\rho})\to\Gamma_{c}(E_{\sigma})$ is gauge
equivariant iff

$\phi\circ\rho(\theta_{\chi})f=\sigma(\theta_{\chi})\circ\phi f,$ (2.26)

for all gauge transformations $\chi:P\to P$ and all feature maps $f\in
C_{c}(P;\rho)$.

This concludes our discussion of gauge theory and of equivariant neural
networks. The framework for the latter is evidently very general, consisting
of layers and non-linear activation functions between data points. There are
advantages of working at this level of generality: Ordinary (non-equivariant)
neural networks have a multitude of different types of layers, many of them
linear. Equivariant analogues of such layers are likely to satisfy either
Definition [def:layer]5(ii) or Definition 9, depending on the relevant type of
equivariance. Any result that can be proven using this general framework, will
thus be true for many different instances of equivariant neural networks. One
example is Theorem 14 below, that characterizes the structure of abstract
$G$-equivariant layers in any GCNN.

##  3\. $G$G-equivariant convolutional neural networks

Recall that GCNNs generalize ordinary CNNs to data points $f:\mathcal{M}\to V$
defined on homogeneous $G$-spaces $\mathcal{M}$. Let us give a brief recap on
homogeneous spaces and global symmetry, before moving on to discuss
homogeneous vector bundles, sections, and induced representations. We will
demonstrate that GCNNs and $G$-equivariant layers (originally defined in Cohen
et al. (2018b)) are most naturally understood from the perspective of
homogeneous vector bundles. We then use reproducing kernel Hilbert spaces and
bandwidth to understand which $G$-equivariant layers are expressible as
convolutional layers.

###  3.1. Homogeneous spaces

######  Definition 6.

Let $G$ be a Lie group. A smooth manifold $\mathcal{M}$ is called a
_homogeneous $G$-space_ if there exists a smooth, transitive left $G$-action

$G\times\mathcal{M}\to\mathcal{M},\qquad(g,x)\mapsto g\cdot x.$ (3.1)

Since the action (3.1) is transitive, we may choose an arbitrary basepoint
$x_{0}\in\mathcal{M}$ and express any other point $x\in\mathcal{M}$ as
$x=g\cdot x_{0}$ for some $g\in G$. This group element is typically not
unique, but observe that

$g\cdot x_{0}=g^{\prime}\cdot x_{0}\quad\iff\quad g^{-1}g^{\prime}\in
H_{x_{0}},$ (3.2)

where $H_{x_{0}}=\left\\{g\in G\ \middle|\ g\cdot x_{0}=x_{0}\right\\}$ is the
isotropy group of $x_{0}$. In other words, there is a one-to-one
correspondence between points $x\in\mathcal{M}$ and left cosets $gH_{x_{0}}\in
G/H_{x_{0}}$.

######  Proposition 4 ((Lee, 2013, Theorem 21.18)).

Let $\mathcal{M}$ be a homogeneous $G$-space and choose a basepoint
$x_{0}\in\mathcal{M}$. The isotropy group $H_{x_{0}}$ is a closed subgroup of
$G$, and the map

$F_{x_{0}}:G/H_{x_{0}}\to\mathcal{M},\qquad gH_{x_{0}}\mapsto g\cdot x_{0},$
(3.3)

is an equivariant diffeomorphism.

Homogeneous spaces are globally symmetric in the sense that any point
$x_{0}\in\mathcal{M}$ may be chosen as basepoint. Given another choice of
basepoint $x_{0}^{\prime}\in\mathcal{M}$, the spaces
$G/H_{x_{0}^{\prime}}\simeq G/H_{x_{0}}$ are diffeomorphically related by a
translation in $G$ \- more precisely, by the composition $F_{x_{0}}^{-1}\circ
F_{x_{0}^{\prime}}$. Euclidean space $\mathcal{M}=\mathbb{R}^{d}$, for
example, possesses a global translation symmetry, allowing any point to be
considered as origin. Similarly, the rotationally symmetric sphere
$\mathcal{M}=S^{2}$ does not have a unique north pole.

We end this part with the following proposition, which is instrumental in
relating homogeneous vector bundles to the equivariance framework in Section
2.2.

######  Proposition 5 ((Steenrod, 1960, §7.5)).

Let $G$ be a Lie group and let $H\leq G$ be a closed subgroup. Then the
quotient map

$q:G\to G/H,\qquad g\mapsto gH,$ (3.4)

defines a smooth principal $H$-bundle over the homogeneous $G$-space
$\mathcal{M}=G/H$.

###  3.2. Homogeneous vector bundles

Vector bundles may inherit global symmetry from a homogeneous base space; the
transitive action $(g,x)\mapsto g\cdot x$ may induce linear maps $E_{x}\mapsto
E_{gx}$ between fibers. Such bundles are naturally called _homogeneous_ and,
because this symmetry is also encoded in its sections (data points), we will
show that homogeneous vector bundles is the natural setting for studying
GCNNs.

From this point on, we restrict attention to homogeneous spaces
$\mathcal{M}=G/K$ where $G$ is a unimodular Lie group and $K\leq G$ is a
compact subgroup. Elements of the homogeneous space is interchangably denoted
as $x\in\mathcal{M}$ or $gK\in G/K$.

######  Remark 5.

Examples of unimodular Lie groups include all finite, discrete, compact, or
abelian Lie groups, the Euclidean groups, and many others. See Folland (2016);
Führ (2005) for details.

######  Definition 7 ((Wallach, 2018, 5.2.1)).

Let $\mathcal{M}$ be a homogeneous $G$-space and let $\pi:E\to\mathcal{M}$ be
a smooth vector bundle with fibers $E_{x}$. We say that $E$ is _homogeneous_
if there is a smooth left $G$-action $G\times E\to E$ satisfying

$g\cdot E_{x}=E_{gx},$ (3.5)

and such that the induced map $L_{g,x}:E_{x}\to E_{gx}$ is linear, for all
$g\in G,x\in\mathcal{M}$.

######  Example 3.

The frame bundle $F\mathcal{M}$ is a homogeneous vector bundle whenever
$\mathcal{M}$ is a homogeneous space, and the same is true of any associated
bundle $F\mathcal{M}\times_{\rho}V_{\rho}$. In particular, the tangent bundle
$T\mathcal{M}$ is a homogeneous vector bundle. ∎

######  Example 4.

If $(\rho,V_{\rho})$ is a finite-dimensional $K$-representation, then the
associated bundle $E_{\rho}=G\times_{\rho}V_{\rho}$ is a homogeneous vector
bundle with respect to the left action

$g\cdot[g^{\prime},v]=[gg^{\prime},v].$ (3.6)

∎

All homogeneous vector bundles $E$ are of the form $G\times_{\rho}V_{\rho}$,
up to isomorphism. To understand why, consider the fiber $E_{K}=E_{eK}$ and
observe that the restriction of (3.5) to $E_{K}$ and elements $k\in K$ yields
invertible linear maps

$L_{k}:E_{K}\to E_{K}.$ (3.7)

The defining properties of group actions ensure that $\rho(k)=L_{k}$ is a
finite-dimensional $K$-representation on $E_{K}$. Moreover, because the linear
maps $L_{g,x}$ are isomorphisms, any element $v^{\prime}$ of any fiber $E_{x}$
can be obtained as the image $v^{\prime}=L_{g,K}(v)=:L_{g}(v)$ for some
choices of $g\in q^{-1}(\\{x\\})$ and $v\in E_{K}$. The mapping

$\xi:G\times E_{K}\to E,\qquad(g,v)\mapsto L_{g}(v),$ (3.8)

is thus surjective. It is not injective, though, since the relation

$L_{g}=L_{g}\circ L_{k}\circ L_{k^{-1}}=L_{gk}\circ\rho(k^{-1}),$ (3.9)

implies that $\xi(g,v)=L_{g}(v)=L_{gk}(\rho(k^{-1})v)=\xi(gk,\rho(k^{-1})v)$
for $k\in K$. However, the same argument shows that $\xi$ is made injective by
passing to the quotient $G\times_{\rho}E_{K}$.

######  Lemma 6 ((Wallach, 2018, 5.2.3)).

The map

$G\times_{\rho}E_{K}\to E,\qquad[g,v]\mapsto L_{g}(v),$ (3.10)

is an isomorphism of homogeneous vector bundles.

We now have two perspectives on bundles $G\times_{\rho}V_{\rho}$: As bundles
associated to the principal bundle $P=G$, and as homogeneous vector bundles
(up to isomorphism). The former perspective offers a connection to the
framework in Section 2.2, whereas the latter motivates the definition of
$G$-equivariant layers in Section 3.4 below.

###  3.3. Induced representations

Let us show the relationship between homogeneous vector bundles and induced
representations, which will be an essential ingredient in the definition of
$G$-equivariant layers. To this end, let $(\rho,V_{\rho})$ be a finite-
dimensional unitary $K$-representation and consider the homogeneous vector
bundle $E_{\rho}=G\times_{\rho}V_{\rho}$.

We will need inner products on $\Gamma_{c}(E_{\rho})$ and $C_{c}(G;\rho)$, the
former of which is defined using the following unitary structure:

######  Lemma 7 ((Wallach, 2018, 5.2.7)).

The unitary structure

$\langle[g,v],[g,w]\rangle_{gK}:=\langle v,w\rangle_{\rho},$ (3.11)

defines a complete inner product on each fiber $E_{gK}$, making $E_{\rho}$
into a Hilbert bundle with $L_{g,x}$ unitary. This unitary structure is unique
in that, if we identify $V_{\rho}$ with $E_{K}$ in the canonical manner, then
the inner product on $V_{\rho}$ so induced agrees with $\langle\ ,\
\rangle_{\rho}$.

We also need the following measure on $G/K$:

######  Theorem 8 (Quotient Integral Formula (Deitmar and Echterhoff, 2014,
§1.5)).

There is a unique $G$-invariant, nonzero Radon measure $\differential x$ on
$G/K$ such that the following _quotient integral formula_ holds for every
$f\in C_{c}(G)$:

$\int_{G}f(g)\ \differential g=\int_{G/K}\int_{K}f(xk)\ \differential k\
\differential x.$ (3.12)

Using these two ingredients, we make $\Gamma_{c}(E_{\rho})$ into a pre-Hilbert
space with respect to the inner product

$\langle s,s^{\prime}\rangle_{L^{2}(E_{\rho})}:=\int_{G/K}\langle
s(x),s^{\prime}(x)\rangle_{x}\ \differential x,\qquad
s,s^{\prime}\in\Gamma_{c}(E_{\rho}),$ (3.13)

and we denote its completion $L^{2}(E_{\rho})$. Similarly, $C_{c}(G;\rho)$ is
a pre-Hilbert space with respect to the inner product

$\langle f,f^{\prime}\rangle_{L^{2}(G;\rho)}=\int_{G}\langle
f(g),f^{\prime}(g)\rangle_{\rho}\ \differential g,\qquad f,f^{\prime}\in
C_{c}(G;\rho),$ (3.14)

the completion of which is denoted $L^{2}(G;\rho)$.

######  Definition 8.

The $G$-representations

$\displaystyle\mathrm{ind}_{K}^{G}\rho(g):L^{2}(E_{\rho})\to
L^{2}(E_{\rho}),\qquad$ $\displaystyle(\mathrm{ind}_{K}^{G}\rho(g)s)(x)$
$\displaystyle=g\cdot s(g^{-1}x),$ (3.15)
$\displaystyle\mathrm{Ind}_{K}^{G}\rho(g):L^{2}(G;\rho)\to
L^{2}(G;\rho),\qquad$
$\displaystyle(\mathrm{Ind}_{K}^{G}\rho(g)f)(g^{\prime})$
$\displaystyle=f(g^{-1}g^{\prime}).$ (3.16)

are called _induced representations_ , or _representations induced by $\rho$_.

Both $\mathrm{ind}_{K}^{G}\rho$ and $\mathrm{Ind}_{K}^{G}\rho$ are unitary
(Wallach, 2018, 5.3.2) and may be identified:

######  Lemma 9.

The induced representations $\mathrm{ind}_{K}^{G}(\rho)$,
$\mathrm{Ind}_{K}^{G}(\rho)$ are unitarily equivalent.

###### Proof.

This is (Wallach, 2018, 5.3.4), but let us write down a proof for clarity.
First observe that the isomorphism $C_{c}(G;\rho)\to\Gamma_{c}(E_{\rho})$,
$f\mapsto s_{f}$ is unitary, which follows by combining the quotient integral
formula (3.12), the unitarity of $\rho$, and the compactness of $K$: For all
$f,f^{\prime}\in C_{c}(G;\rho)$, the map $g\mapsto\langle
f(g),f^{\prime}(g)\rangle_{\rho}$ lies in $C_{c}(G)$ and so

$\displaystyle\langle f,f^{\prime}\rangle_{L^{2}(G;\rho)}$
$\displaystyle=\int_{G/K}\int_{K}\langle f(xk),f^{\prime}(xk)\rangle_{\rho}\
\differential k\ \differential x$ (3.17) $\displaystyle=\int_{G/K}\langle
f(x),f^{\prime}(x)\rangle_{\rho}\ \differential x$
$\displaystyle=\int_{G/K}\langle[x,f(x)],[x,f^{\prime}(x)]\rangle_{x}\
\differential x=\langle s_{f},s_{f^{\prime}}\rangle_{L^{2}(E_{\rho})}.$

The same map $f\mapsto s_{f}$ satisfies

$\left(\mathrm{ind}_{K}^{G}(\rho)s_{f}\right)(x)=g\cdot
s_{f}(g^{-1}x)=[x,f(g^{-1}x)]=s_{\mathrm{Ind}_{K}^{G}(\rho)f}(x),$ (3.18)

so it extends to a unitary isomorphism $L^{2}(G;\rho)\to L^{2}(E_{\rho})$
intertwining the induced representations. ∎

To gain a better understanding of the induced representations, consider the
Bochner space $L^{2}(G,V)$, the space of square-integrable functions $f:G\to
V$ that take values in a finite-dimensional Hilbert space $V$. It is itself a
Hilbert space with inner product

$\langle f,f^{\prime}\rangle_{L^{2}(G,V)}=\int_{G}\langle
f(g),f^{\prime}(g)\rangle_{V}\ \differential g^{\prime}.$ (3.19)

The induced representation $(\mathrm{Ind}_{K}^{G}\rho,L^{2}(G;\rho))$ is
nothing but the restriction of the left regular representation $\Lambda$ on
$L^{2}(G,V_{\rho})$ to a closed, invariant subspace. Furthermore, $\Lambda$ is
intimately related to the left regular representation $\lambda$ on $L^{2}(G)$,
as the following lemma shows. The proof of this lemma is a short calculation.

######  Lemma 10.

Let $V$ be a finite-dimensional Hilbert space and equip $L^{2}(G)\otimes V$
with the tensor product inner product. Then the natural unitary isomorphism

$\displaystyle A:L^{2}(G)\otimes V$ $\displaystyle\to L^{2}(G,V)$ (3.20)
$\displaystyle f\otimes v$ $\displaystyle\mapsto fv$

intertwines $\lambda\otimes\operatorname{Id}_{V}$ with $\Lambda$.

This lemma also shows that, if we choose an orthonormal basis
$e_{1},\ldots,e_{\dim V}\in V$, elements of $L^{2}(G,V)$ are simply linear
combinations $f=\sum_{i}f^{i}e_{i}$ with component functions $f^{i}\in
L^{2}(G)$. We use this fact in some calculations of vector-valued integrals,
and the component functions will also be important in Section 3.5.

###  3.4. $G$G-equivariant and convolutional layers

Given a homogeneous $G$-space $\mathcal{M}$, we observed that vector bundles
$\pi:E\to\mathcal{M}$ may inherit the global symmetry of $\mathcal{M}$. We
took a closer look at such homogeneous vector bundles and found that they are
isomorphic to associated bundles $G\times_{\rho}V_{\rho}$, and therefore fit
within the equivariance framework of Section 2.2 . We also saw how the global
symmetry of $\mathcal{M}$ is encoded in data points and feature maps via
induced representations, and we want $G$-equivariant layers to preserve this
global symmetry.

Consider homogeneous vector bundles $E_{\rho}=G\times_{\rho}V_{\rho}$ and
$E_{\sigma}=G\times_{\sigma}V_{\sigma}$, and recall Definition 5 of layers as
general linear maps $\Phi:\Gamma_{c}(E_{\rho})\to\Gamma_{c}(E_{\sigma})$. We
are mainly interested in _bounded_ layers from an application point of view,
and we can make this restriction now that the domain and codomain are normed
spaces. Furthermore, any bounded layer can be uniquely extended to a bounded
linear map

$\Phi:L^{2}(E_{\rho})\to L^{2}(E_{\sigma}),$ (3.21)

and we assume this extension has already been made.

######  Definition 9.

A bounded linear map $\Phi:L^{2}(E_{\rho})\to L^{2}(E_{\sigma})$ is called a
_$G$ -equivariant layer_ if it intertwines the induced representations:

$\Phi\circ\mathrm{ind}_{K}^{G}\rho=\mathrm{ind}_{K}^{G}\sigma\circ\Phi.$
(3.22)

That is, $G$-equivariant layers are elements
$\Phi\in\mathrm{Hom}_{G}(L^{2}(E_{\rho}),L^{2}(E_{\sigma}))$.

######  Remark 6.

We could also have defined $G$-equivariant layers as bounded linear maps
$\phi:L^{2}(G;\rho)\to L^{2}(G;\sigma)$ that intertwine the induced
representations:

$\phi\circ\mathrm{Ind}_{K}^{G}\rho=\mathrm{Ind}_{K}^{G}\sigma\circ\Phi,$
(3.23)

i.e., elements $\phi\in\mathrm{Hom}_{G}(L^{2}(G;\rho),L^{2}(G,\sigma))$. These
definitions are clearly equivalent.

Apart from minor technical differences, Definition 9 coincides with the
definition of equivariant maps in Cohen et al. (2018b). We have thus obtained
GCNNs almost directly from the definition of homogeneous vector bundles and a
desire for layers to respect the global symmetry. This shows that homogeneous
vector bundles is the natural setting for GCNNs.

Let us now define convolutional layers.

######  Definition 10.

A _convolutional layer_ $L^{2}(G;\rho)\to L^{2}(G;\sigma)$ is a bounded
operator

$[\kappa\star f](g)=\int_{G}\kappa(g^{-1}g^{\prime})f(g^{\prime})\
\differential g^{\prime},\qquad f\in L^{2}(G;\rho),$ (3.24)

with an operator-valued kernel $\kappa:G\to\mathrm{Hom}(V_{\rho},V_{\sigma})$.

Of course, not any function $\kappa:G\to\mathrm{Hom}(V_{\rho},V_{\sigma})$ can
be chosen as the kernel of a convolutional layer. The kernel must ensure both
that (3.24) is bounded and that $\phi f\in L^{2}(G;\sigma)$ for each $f\in
L^{2}(G;\rho)$. We give a sufficient condition for boundedness in Lemma 12 and
the other requirement has been studied in detail in Cohen et al. (2018b); Lang
and Weiler (2020).

The next result is an almost immediate consequence of the Fubini-Tonelli
theorem.

######  Proposition 11.

The adjoint of (3.24) is the integral operator

$[f*\kappa^{*}](g)=\int_{G}\kappa^{*}({g^{\prime}}^{-1}g)f(g^{\prime})\
\differential g^{\prime},\qquad f\in L^{2}(G;\sigma),$ (3.25)

where $\kappa^{*}$ is the pointwise adjoint of $\kappa$. That is,
$(\kappa\star\cdot)^{*}=\cdot*\kappa^{*}$.

One way to ensure that the operators (3.24)-(3.25) are bounded, is to put a
bound on the kernel matrix elements $\kappa_{ij}:G\to\mathbb{C}$ for any given
choice of bases in $V_{\rho}$, $V_{\sigma}$.

######  Lemma 12.

The operators (3.24) and (3.25) are bounded if $\kappa_{ij}\in L^{1}(G)$ for
all $i,j$.

###### Proof.

We need only prove that (3.25) is bounded, its adjoint (3.24) will be bounded
as well. Choose bases $e_{1},\ldots,e_{\dim V_{\rho}}\in V_{\rho}$ and
$\tilde{e}_{1},\ldots,\tilde{e}_{\dim V_{\sigma}}\in V_{\sigma}$ and observe
that, because $L^{2}(G;\sigma)\subset L^{2}(G,V_{\sigma})$, Lemma 10 enables
the decomposition of $f\in L^{2}(G;\sigma)$ into component functions $f^{i}\in
L^{2}(G)$:

$f=\sum_{i=1}^{\dim V_{\sigma}}f^{i}\tilde{e}_{i}.$ (3.26)

To be clear, the kernel $\kappa$ is similarly decomposed into matrix elements
$\kappa_{ij}=\langle\tilde{e}_{j},\kappa e_{i}\rangle_{\sigma}$ and we have
$\kappa_{ji}^{*}=\overline{\kappa_{ij}}$. The integral (3.25) now takes the
form

$[f*\kappa^{*}](g)=\sum_{j=1}^{\dim V_{\rho}}\left(\int_{G}\sum_{i=1}^{\dim
V_{\sigma}}\kappa_{ji}^{*}(g^{\prime-1}g)f^{i}(g)\ \differential
g\right)e_{j},$ (3.27)

so by Young’s convolution inequality,

$\displaystyle\|f*\kappa^{*}\|_{L^{2}(G;\rho)}^{2}$
$\displaystyle\leq\sum_{i,j}\int_{G}\left|\int_{G}\kappa_{ji}^{*}({g^{\prime}}^{-1}g)f^{i}(g^{\prime})\
\differential g^{\prime}\right|^{2}\differential
g=\sum_{i,j}\|f^{i}*\kappa_{ji}^{*}\|_{2}^{2}$ (3.28)
$\displaystyle\leq\sum_{i,j}\|\kappa_{ij}\|_{1}^{2}\|f^{i}\|_{2}^{2}\leq
M\sum_{i}\|f^{j}\|_{2}^{2}=M\|f\|_{L^{2}(G;\sigma)}^{2},$

where $M=\sum_{i,j}\|\kappa_{ij}\|_{1}^{2}<\infty$ if $\kappa_{ij}\in
L^{1}(G)$ for all $i,j$. ∎

We are interested in convolutional layers partly because they are concrete
examples of $G$-equivariant layers, which we show next.

######  Proposition 13.

Convolutional layers are $G$-equivariant layers.

###### Proof.

Convolutional layers $\kappa\star\cdot:L^{2}(G;\rho)\to L^{2}(G;\sigma)$ are
bounded linear operators by definition, so the only thing we need to prove is
that $\kappa\star\cdot$ intertwines the induced representations. This follows
immediately from left-invariance of the Haar measure: For each $f\in
L^{2}(G;\rho)$ and all $g,h\in G$,

$\displaystyle\big{[}\kappa\star\big{(}\mathrm{Ind}_{K}^{G}\rho(g)f\big{)}\big{]}(h)$
$\displaystyle=\int_{G}\kappa(h^{-1}g^{\prime})f(g^{-1}g^{\prime})\
\differential g^{\prime}\qquad(g^{\prime}\mapsto gg^{\prime})$ (3.29)
$\displaystyle=\int_{G}\kappa\big{(}(g^{-1}h)^{-1}g^{\prime}\big{)}f(g^{\prime})\
\differential g^{\prime}=[\kappa\star f](g^{-1}h),$

hence
$\big{[}\kappa\star\mathrm{Ind}_{K}^{G}\rho(g)f\big{]}=\mathrm{Ind}_{K}^{G}\sigma(g)[\kappa\star
f]$. ∎

######  Example 5.

Let us describe where ordinary CNNs fit in the present context. CNNs represent
the case $G=\mathbb{Z}^{2}$ when $K=\\{0\\}$ is the trivial subgroup. The
corresponding homogeneous space is $G/K=\mathbb{Z}^{2}/\\{0\\}=\mathbb{Z}^{2}$
and the quotient map $q:G\to G/K$ is thus the identity map on
$\mathbb{Z}^{2}$. Its inverse, the identity map $\omega:G/K\to G$, is a
globally defined gauge that eliminates the need for gauge equivariance, as we
may choose to work exclusively in this one gauge. This is just a reflection of
the fact that

$G=\mathbb{Z}^{2}=\mathbb{Z}/\\{0\\}\times\\{0\\}=G/K\times K,$ (3.30)

is (obviously) trivial as a principal bundle. Its associated bundles
$E_{\rho}=\mathbb{Z}^{2}\times_{\rho}V_{\rho}$ are also trivial: partly
because the finite-dimensional $K$-representation $\sigma$ must be trivial,
and partly because each equivalence class $[g,v]$ only contains a single
representative. These reasons are, of course, due to the triviality of $K$.

This is not to say that the equivariant framework of Section 2.2 is
uninteresting when dealing with CNNs, or with GCNNs for other homogeneous
spaces $\mathcal{M}=G/K$ with $K$ trivial. We saw in Sections 3.2-3.3 how the
homogeneity give rise to induced representations, which encode the global
symmetry in both data points and feature maps. This is a useful perspective to
have, and $G$-equivariant layers are interesting even when the bundles are
trivial.

Triviality of the associated bundles,
$E_{\rho}\simeq\mathbb{Z}^{2}\times\mathbb{C}^{m}$ where $m=\dim
V_{\sigma}$,666Recall that we focus on complex vector bundles, hence the use
of $\mathbb{C}^{m}$ instead of $\mathbb{R}^{m}$. implies that data points and
feature maps are general square-integrable functions,

$L^{2}(E_{\rho})\simeq L^{2}(\mathbb{Z}^{2};\rho)\simeq
L^{2}(\mathbb{Z}^{2},\mathbb{C}^{m}),$ (3.31)

and are thereby extensions of compactly supported functions
$f:\mathbb{Z}^{2}\to\mathbb{C}^{m}$. This ties well into the discussion in
Section 2.1. Convolutional layers (3.24) reduce to bounded linear operators
$L^{2}(\mathbb{Z}^{2},\mathbb{C}^{m})\to L^{2}(\mathbb{Z}^{2},\mathbb{C}^{n})$
and take the form

$(\kappa\star f)(x)=\sum_{y\in\mathbb{Z}^{2}}\kappa(y-x)f(y),$ (3.32)

as the Haar measure on $\mathbb{Z}^{2}$ is the counting measure. The kernel
$\kappa:\mathbb{Z}^{2}\to\mathrm{Hom}(\mathbb{C}^{m},\mathbb{C}^{n})$ is
finitely supported in practice, so boundeness of (3.32) is ensured by Lemma
12.

Interestingly, all $\mathbb{Z}^{2}$-equivariant layers are convolutional
layers; there are no other types of $\mathbb{Z}^{2}$-equivariant layers than
(3.32). This is a consequence of Theorem 14 and is proven in Corollary 20
below. ∎

For more general groups $G$, it is no longer true that all $G$-equivariant
layers are convolutional layers; we give an example of this fact in Example 6.
Implementations of GCNNs, however, are usually based on convolutional layers,
or on analogous layers in the Fourier domain. What consequences does the
restriction to convolutional layers have for the expressivity of GCNNs? Can we
tell whether a given $G$-equivariant layer is expressible as a convolutional
layer? The answer to this last question, it turns out, requires the following
notion of reproducing kernel Hilbert spaces.

######  Definition 11.

Let $G$ be a group, let $V$ be a finite-dimensional normed vector space, and
let $\mathcal{H}$ be a Hilbert space of functions $G\to V$. Then $\mathcal{H}$
is a _reproducing kernel Hilbert space (RKHS)_ if the evaluation operator

$\mathcal{E}_{g}:\mathcal{H}\to V,\qquad f\mapsto f(g),$ (3.33)

is bounded for all $g\in G$. Moreover, by _left-invariant RKH subspace_
$\mathcal{H}\subseteq L^{2}(G,V)$ we mean a closed subspace that is both a
RKHS and an invariant subspace for the left regular representation $\Lambda$
on $L^{2}(G,V)$.

######  Remark 7.

The term RKHS is typically reserved for the scalar case $V=\mathbb{C}$, when
the evaluation operator is a linear functional. Our version would instead be
dubbed _vector-valued_ RKHS. We see little benefit from distinguishing between
these cases, however, so we use the term RKHS all-encompassingly.

The name RKHS is due to the existence of a kernel-type function that
reproduces all elements of $\mathcal{H}$. To see how, choose an orthonormal
basis $e_{1},\ldots,e_{\dim V}\in V$ and write elements $v\in V$ as linear
combinations $v=\sum_{i}v^{i}e_{i}$. The projection $P_{i}(v)=v^{i}$ onto the
$i$’th component is always continuous, so the composition
$\mathcal{E}_{g,i}:=P_{i}\circ\mathcal{E}_{g}$ is a continuous linear
functional

$\mathcal{E}_{g}^{i}:\mathcal{H}\to\mathbb{C},\qquad f\mapsto f^{i}(g),$
(3.34)

for all $g\in G$ and $i=1,\ldots,\dim V$. By the Riesz representation theorem,
there are elements $\varphi_{g,i}\in\mathcal{H}$ such that
$f^{i}(g)=\mathcal{E}_{g,i}(f)=\langle f,\varphi_{g,i}\rangle$, hence

$f(g)=\sum_{i=1}^{\dim V}\langle f,\varphi_{g,i}\rangle e_{i}.$ (3.35)

Now, if $\mathcal{H}\subseteq L^{2}(G,V)$ is a left-invariant RKH subspace,
expanding the functions $\varphi_{g,i}$ in the orthonormal basis,
$\varphi_{g,i}=\sum_{j}\varphi_{g,i}^{j}e_{j}$, yields the formula

$\displaystyle f(g)$
$\displaystyle=\sum_{i}\left(\int_{G}\sum_{j}\overline{\varphi_{g,i}^{j}(g^{\prime})}f^{j}(g^{\prime})\
\differential
g^{\prime}\right)e_{i}=\int_{G}\varphi_{g}^{*}(g^{\prime})f(g^{\prime})\
\differential g^{\prime},$ (3.36)

where $\varphi_{g}^{*}$ is the conjugate transpose of the matrix
$(\varphi_{g})_{i}^{j}=\varphi_{g,i}^{j}$. By left-invariance,

$\displaystyle f(g)$
$\displaystyle=\big{(}\Lambda(g^{-1})f\big{)}(e)=\int_{G}\varphi_{e}^{*}(g^{-1}g^{\prime})f(g^{\prime})\
\differential g^{\prime},$ (3.37)

hence $f\in\mathcal{H}$ is _reproduced_ by the operator-valued _kernel_
$\varphi_{e}:G\to\mathrm{Hom}(V)$.

######  Remark 8.

The reproducing kernel $\varphi_{e}$ is unique and thus independent of the
choice of basis in $V$. This follows from uniqueness in the Riesz
representation theorem.

It is now clear why left-invariant RKH subspaces of $L^{2}(G,V)$ are relevant
when discussing convolutional layers, as the latter are given by integral
operators similar to (3.37). In order to show that an abstract $G$-equivariant
layer $\phi:L^{2}(G;\rho)\to L^{2}(G;\sigma)$ can be written as a
convolutional layer, it is almost necessary for it to act in a RKHS:

######  Example 6.

The identity operator $\phi:L^{2}(G;\sigma)\to L^{2}(G;\sigma)$ is clearly a
$G$-equivariant layer regardless of $G$, $K$, $\sigma$, but it is only a
convolutional layer if $L^{2}(G;\sigma)$ is a RKHS. This is because when
$\phi$ is the identity, (3.24) becomes the reproducing property

$f(g)=\int_{G}\kappa(g^{-1}g^{\prime})f(g^{\prime})\ \differential
g^{\prime},\qquad f\in L^{2}(G;\sigma).$ (3.38)

It follows that not every $G$-equivariant layer is a convolutional layer,
because $L^{2}(G;\sigma)$ is not always a RKHS. When $\sigma$ is the trivial
representation, for instance, $L^{2}(G;\sigma)$ reduces to $L^{2}(G)$ which is
not a RKHS when $G$ is nondiscrete (Führ, 2005, Theorem 2.42). ∎

At this point, we know that global symmetry manifests itself in feature maps
and data points through the induced representation, and we used this knowledge
to define $G$-equivariant layers. We also defined convolutional layers and
showed that these are special cases of $G$-equivariant layers, but the
converse problem is much more subtle: When can a $G$-equivariant layer be
expressed as a convolutional layer? The answer, as we have just seen, is
directly related to the concept of RKHS and our next result makes this
relation precise. It can be considered our main theorem.

######  Theorem 14.

Let $G$ be a unimodular Lie group, let $K\leq G$ be a compact subgroup, and
consider homogeneous vector bundles $E_{\rho},E_{\sigma}$ over
$\mathcal{M}=G/K$. Suppose that

$\phi:L^{2}(G;\rho)\to L^{2}(G;\sigma),$ (3.39)

is a $G$-equivariant layer. If $\phi$ maps into a left-invariant RKH subspace
$\mathcal{H}\subseteq L^{2}(G;\sigma)$, then $\phi$ is a convolutional layer.

###### Proof.

Fix orthonormal bases in $V_{\rho}$, $V_{\sigma}$. For
$i=1,\ldots,\dim\sigma$, consider the functionals

$\mathcal{E}^{i}:L^{2}(G;\rho)\to\mathbb{C},\qquad\mathcal{E}^{i}(f)=(\phi
f)^{i}(e),$ (3.40)

composing $\phi$ with evaluation at the identity element $e\in G$ and
projection onto the $i$’th component. As $\phi$ maps into a left-invariant RKH
subspace $\mathcal{H}\subseteq L^{2}(G;\sigma)$, (3.40) is a bounded linear
functional: $|\mathcal{E}^{i}(f)|\leq\|(\phi f)(e)\|_{\sigma}\leq\|\phi
f\|_{L^{2}(G;\sigma)}\leq\|\phi\|\|f\|_{L^{2}(G;\rho)}$. By the Riesz
representation theorem, there is a unique $\varphi_{i}\in L^{2}(G;\rho)$ such
that

$\mathcal{E}^{i}(f)=\int_{G}\langle f(g),\varphi_{i}(g)\rangle_{\rho}\
\differential
g=\int_{G}\sum_{j=1}^{\dim\rho}f^{i}(g)\overline{\varphi_{i}^{j}(g)}\
\differential g,$ (3.41)

and proceeding as in (3.36)-(3.37) with $\kappa:=\varphi_{e}^{*}$ yields the
desired relation

$(\phi f)(g)=\int_{G}\kappa(g^{-1}g^{\prime})f(g^{\prime})\ \differential
g^{\prime}.$ (3.42)

∎

######  Remark 9.

Theorem 14 is a generalization of (Cohen et al., 2018b, Theorem 6.1), which
was proven under the assumption that $\phi$ is an integral operator $(\phi
f)(g)=\int_{G}\kappa(g,g^{\prime})f(g^{\prime})\ \differential g^{\prime}$.

######  Remark 10.

While Theorem 14 is similar in spirit to (Kondor and Trivedi, 2018, Theorem
1), there are also some clear differences. For example, we work with
unimodular Lie groups whereas Kondor and Trivedi (2018) use compact groups,
but (Kondor and Trivedi, 2018, Theorem 1) is also stronger in this case as
there is no criterion on the layer. Another difference is that Kondor and
Trivedi (2018) analyzes the whole network structure while we focus on
individual layers. We also assume that the homogeneous space $G/K$ is the same
before and after each layer, in constrast to Kondor and Trivedi (2018).

In the special case of single-layer networks with compact $G$, (Kondor and
Trivedi, 2018, Theorem 1) states that any $G$-equivariant layer is a
convolutional layer. Example 6 seems to contradict this statement when $G$ is
non-discrete compact. This conflict is possibly due to minor technical
differences in the assumptions on layers and data points, but we have not
identified the precise cause.

We end this section with a result that could simplify the numerical
computations of convolutional layers, as integrals over $G/K$ are sometimes
easier to compute than integrals over $G$. For example when $G=SO(3)$,
$K=SO(2)$, and $G/K\simeq S^{2}$. This result is similar to the generalized
convolutions described in (Kondor and Trivedi, 2018, Section 4.1)

######  Corollary 15.

Let $\phi:L^{2}(G;\rho)\to L^{2}(G;\sigma)$ be as in Theorem 14 and let
$\kappa$ be the kernel of the resulting convolutional layer (3.42). Then

$(\phi f)(g)=\int_{G/K}\kappa(g^{-1}x)f(x)\ \differential x.$ (3.43)

###### Proof.

In the proof of Theorem 14, we constructed the kernel $\kappa$ from the
components of $\varphi_{i}\in L^{2}(G;\rho)$, and unitarity of $\rho$ clearly
implies that the expression $\langle f(x),\varphi_{i}(x)\rangle_{\rho}$ is
well-defined. We may therefore use the unitary structure (3.11) to get the
following relation for all component functions $(\phi f)^{i}$ and all $g\in
G$:

$\displaystyle(\phi f)^{i}(g)$ $\displaystyle=\langle
f,\mathrm{Ind}_{K}^{G}\rho(g)\varphi_{i}\rangle_{L^{2}(G;\rho)}=\langle
s_{f},s_{\mathrm{Ind}_{K}^{G}\rho(g)\varphi_{i}}\rangle_{L^{2}(E_{\rho})}$
(3.44) $\displaystyle=\int_{G/K}\langle
s_{f}(x),s_{\mathrm{Ind}_{K}^{G}\rho(g)\varphi_{i}}(x)\rangle_{x}\
\differential x$ (3.45) $\displaystyle=\int_{G/K}\langle
f(x),\mathrm{Ind}_{K}^{G}\rho(g)\varphi_{i}(x)\rangle_{\rho}\ \differential x$
(3.46)
$\displaystyle=\int_{G/K}\sum_{j=1}^{\dim\rho}\overline{\varphi_{i}^{j}(g^{-1}x)}f^{j}(x)\
\differential x.$ (3.47)

We now obtain (3.43) by reconstructing $\kappa$ from its components
$\kappa_{ij}=\overline{\varphi_{i}^{j}}$. ∎

###  3.5. RKHS and bandlimited functions

The strength of Theorem 14 naturally depends on how common left-invariant RKH
subspaces of $L^{2}(G;\sigma)$ are. Our analysis of $G$-equivariant layers
would not be complete without a discussion on this topic.

Let us proceed by investigating when the component functions $f^{i}$ of $f\in
L^{2}(G;\sigma)$ are contained in a left-invariant RKH subspace
$\mathcal{H}\subset L^{2}(G)$; these subspaces have been fully characterized
when the unimodular Lie group $G$ is of _type I_ Carey (1978); Führ (2005).
The unitary equivalence (3.20) then ensures that $A(\mathcal{H}\otimes
V_{\sigma})\subset L^{2}(G,V_{\sigma})$ is a left-invariant RKH subspace, and
so is the closed subspace

$A(\mathcal{H}\otimes V_{\sigma})\cap L^{2}(G;\sigma)\subset L^{2}(G;\sigma).$
(3.48)

######  Remark 11.

Groups of type I are, in a sense, groups with manageable representation
theory. They include the most common groups, such as all finite, discrete,
compact, or abelian groups, the Euclidean groups and many other groups. In
particular, there is a considerable overlap between type I groups and the
unimodular Lie groups that we already consider. See Folland (2016); Führ
(2005) for more details.

######  Remark 12.

While $\rho,\sigma$ still denote finite-dimensional unitary representations of
$K$, we reserve the letter $\gamma$ for elements of the unitary dual
$\widehat{G}$, i.e., the space of equivalence classes of unitary
representations. Specific representatives of $\gamma$ are written as
$(\pi_{\gamma},V_{\gamma})$, and note that $V_{\gamma}$ need not be finite-
dimensional unless $G$ is compact. The unimodular Lie group $G$ is assumed to
be of type I throughout this section.

######  Proposition 16 ((Führ, 2005, Proposition 2.40)).

Let $\mathcal{H}\subseteq L^{2}(G)$ be a left-invariant RKH subspace. The
kernel $\varphi\in\mathcal{H}$ is then a self-adjoint convolution
idempotent,777That is, $\varphi=\varphi*\varphi^{*}=\varphi^{*}$ where
$\varphi^{*}(g):=\overline{\varphi(g^{-1})}$ denotes involution. and

$\mathcal{H}=L^{2}(G)*\varphi=\left\\{f*\varphi\ \middle|\ f\in
L^{2}(G)\right\\}\subset C(G).$ (3.49)

Conversely, if $\varphi\in L^{2}(G)$ is a self-adjoint convolution idempotent,
then $\mathcal{H}=L^{2}(G)*\varphi$ is a left-invariant RKH subspace of
$L^{2}(G)$.

######  Example 7.

Consider the real line $G=\mathbb{R}$ and suppose $\mathcal{H}\subseteq
L^{2}(\mathbb{R})$ is a left-invariant RKH subspace with kernel
$\varphi\in\mathcal{H}$. The calculation in (3.37) with $V=\mathbb{C}$ shows
that, for all $f\in\mathcal{H}$,

$f(x)=\int_{-\infty}^{\infty}\overline{\varphi(y-x)}f(y)\ \differential
y=\langle f,\lambda(x)\varphi\rangle=(f*\varphi^{*})(x).$ (3.50)

Since the regular representation $\lambda$ is continuous, $f$ must be
continuous, so $\mathcal{H}\subset C(\mathbb{R})$. Setting $f=\varphi$ shows
that the kernel is a self-adjoint convolution idempotent:

$\varphi=\varphi*\varphi^{*}=(\varphi*\varphi^{*})^{*}=\varphi^{*}.$ (3.51)

Combining the Plancherel transform on $L^{2}(\mathbb{R})$ (see Theorem 17 and
Section 3.5.1) with the convolution theorem in Fourier analysis, we observe
that, for all $f\in\mathcal{H}$,

$\hat{f}=\widehat{f*\varphi}=\hat{f}\hat{\varphi}.$ (3.52)

In particular, $\hat{\varphi}=\hat{\varphi}^{2}$, so $\hat{\varphi}$ is the
characteristic function $1_{E}$ on a subset
$E\subset\widehat{\mathbb{R}}\simeq\mathbb{R}$. Inserting
$\hat{\varphi}=1_{E}$ in (3.52) immediately tells us that
$\operatorname{supp}(\hat{f})\subset E$, so $\mathcal{H}$ is a space of
bandlimited functions. Moreover, the set $E$ has finite Lebesgue measure
according to the Plancherel theorem:
$\mathrm{vol}(E)=\|1_{E}\|_{2}^{2}=\|\varphi\|_{2}^{2}<\infty$. ∎

This example illustrates that any measurable subset $E\subset\mathbb{R}$ with
finite Lebesgue measure induces a left-invariant closed RKH subspace

$\mathcal{H}_{E}=\left\\{f\in L^{2}(\mathbb{R})\ \middle|\
\operatorname{supp}(\hat{f})\subset E\right\\}=L^{2}(\mathbb{R})*\varphi_{E},$
(3.53)

$\varphi_{E}$ being the inverse Plancherel transform of $1_{E}$ (Führ, 2005,
2.63-2.65). This relation between left-invariant RKH subspaces
$\mathcal{H}\subseteq L^{2}(G)$ and bandlimited functions generalizes to
unimodular Lie groups $G$ of type I, although the necessary harmonic analysis
becomes significantly more advanced. Going into detail on this rather
technical subject would distract from the topic at hand, so we refer curious
readers to the relevant literature instead Führ (2005). Let us take the short
route of stating a theorem on the direct integral decomposition of the left
regular representation $\lambda$ and its commutant

$\lambda(G)^{\prime}=\left\\{T\in\mathcal{B}(L^{2}(G))\ \middle|\
T\circ\lambda(g)=\lambda(g)\circ T\text{ for all }g\in G\right\\},$ (3.54)

and discuss a few consequences of this decomposition, before restricting
attention to two important cases where we can be more explicit: Abelian and
compact groups.

######  Definition 12 ((Führ, 2005, §3.5)).

The _operator-valued Fourier transform_ on $G$ maps each $f\in L^{1}(G)$ to
the family $\mathcal{F}(f)=(\hat{f}(\gamma))_{\gamma\in\widehat{G}}$, where
each $\hat{f}(\gamma)\in\mathcal{B}(V_{\gamma})$ is a bounded operator given
by the Bochner integral

$\hat{f}(\gamma)=\int_{G}f(g)\pi_{\gamma}(g)\ \differential g.$ (3.55)

######  Theorem 17 ((Führ, 2005, Theorem 3.48)).

There is a canonical _Plancherel measure_ $\nu$ for the unitary dual
$\widehat{G}$ with the following properties:

  1. (a)

$\mathcal{F}$ extends to a unitary operator

$\mathcal{P}:L^{2}(G)\to\int_{\widehat{G}}^{\oplus}V_{\gamma}\otimes
V_{\gamma}\ \differential\nu(\gamma),$ (3.56)

called the _Plancherel transform_ of $G$.

  2. (b)

$\mathcal{P}$ implements the following unitary equivalences:

$\displaystyle\lambda\simeq$
$\displaystyle\int_{\widehat{G}}^{\oplus}\pi_{\gamma}\otimes\operatorname{Id}\
\differential\nu(\gamma)$ (3.57) $\displaystyle\lambda(G)^{\prime}\simeq$
$\displaystyle\int_{\widehat{G}}^{\oplus}\operatorname{Id}\otimes\mathcal{B}(V_{\gamma})\
\differential\nu(\gamma)$ (3.58)

Observe that if $\mathcal{H}\subseteq L^{2}(G)$ is a left-invariant closed
subspace, then the projection $P:L^{2}(G)\to\mathcal{H}$ commutes with the
left-regular representation and is thus an element of the commutant
$\lambda(G)^{\prime}$. It therefore has a direct integral decomposition

$P=\int_{\widehat{G}}^{\oplus}\operatorname{Id}\otimes\hat{P}_{\gamma}\
\differential\nu(\gamma),$ (3.59)

where $\hat{P}_{\gamma}\in\mathcal{B}(V_{\gamma})$ for each
$\gamma\in\widehat{G}$.

######  Theorem 18 ((Führ, 2005, Theorem 4.22, Proposition 2.40)).

Suppose that $\mathcal{H}\subseteq L^{2}(G)$ is a left-invariant closed
subspace with equation (3.59) denoting the projection onto $\mathcal{H}$. Then
$\mathcal{H}$ is a RKHS iff

$\int_{\widehat{G}}\rank(\hat{P}_{\gamma})\ \differential\nu(\gamma)<\infty.$
(3.60)

We interpret this theorem as a bandwidth restriction, similar to Example 7.
The integrand in (3.60) is an integer-valued function on $\widehat{G}$, so the
integral is finite only if the projection (3.59) is supported on a set
$E\subseteq\widehat{G}$ of finite Plancherel measure,

$\nu\Big{(}\underbrace{\overline{\left\\{\gamma\in\widehat{G}:\hat{P}_{\gamma}\neq
0\right\\}}}_{E}\Big{)}=\int_{\widehat{G}}1_{E}(\gamma)\
\differential\nu(\gamma)\leq\int_{\widehat{G}}\rank(\hat{P}_{\gamma})\
\differential\nu(\gamma).$ (3.61)

That is, the left-invariant RKH subspaces $\mathcal{H}\subseteq L^{2}(G)$ are
precisely those subspaces whose elements are bandlimited on a set
$E\subseteq\widehat{G}$, in the sense that, for all $f\in\mathcal{H}$ and each
equivalence class $\gamma\not\in E$,

$\hat{f}(\gamma)=\widehat{Pf}(\gamma)=\hat{f}(\gamma)\circ\hat{P}_{\gamma}=0.$
(3.62)

######  Remark 13.

The second equality in (3.62) is (Führ, 2005, Corollary 4.17).

Bandlimited functions are thus central to the theory of RKHS and, by
extension, to the mathematical theory of GCNNs. Indeed, by extending the
concept of bandwidth to feature maps, through (3.48), we obtain the following
rephrasing of Theorem 14.

######  Corollary 19.

Let $G$ be a unimodular Lie group of type I, let $K\leq G$ be a compact
subgroup, and consider homogeneous vector bundles $E_{\rho},E_{\sigma}$ over
$G/K$. Suppose that

$\phi:L^{2}(G;\rho)\to L^{2}(G;\sigma),$ (3.63)

is a $G$-equivariant layer. If $\phi$ maps into a space of bandlimited
functions, then $\phi$ is a convolutional layer.

######  Remark 14.

The relevance of bandwidth for convolutional layers has already been
recognized in the case of azimuthally equivariant linear operators on
$L^{2}(S^{2})$ Toft et al. (2021). In our setting, these operators translate
to certain $G$-equivariant layers

$\Phi:L^{2}(E_{\rho})\to L^{2}(E_{\sigma}),$ (3.64)

when $G=SO(3)$, $K=SO(2)$, and $\rho,\sigma$ are the trivial representation.

######  Remark 15.

Some implementations of GCNNs use Fourier transforms and a variant of the
convolution theorem $\widehat{f_{1}*f_{2}}=\hat{f}_{1}\hat{f}_{2}$ to compute
convolutional layers Esteves et al. (2017); Kondor et al. (2018); Toft et al.
(2021); Weiler et al. (2018). Feature maps $f$ are then represented by their
Fourier transform $\hat{f}$ which, for numerical reasons, is only approximated
up to a finite bandlimit. That is, bandwidth is already being used in
implementations.

####  3.5.1. Abelian groups

The irreducible representations $\gamma\in\widehat{G}$ of any abelian group
$G$ are 1-dimensional, and may thus be identified with their character
$\chi_{\gamma}=\tr\pi_{\gamma}$. There are several useful consequences of this
fact.

First, the unitary dual $\widehat{G}$ is now the set of continuous
homomorphisms $\chi:G\to\mathbb{T}$, where $\mathbb{T}$ is the circle group.
This is a locally compact group with respect to pointwise multiplication and,
as $\gamma\in\widehat{G}$ is unitary, we may write
$\chi_{\gamma}=e^{i\xi_{\gamma}}$ where $\xi_{\gamma}:G\to\mathbb{R}$. The
Fourier transform then takes the more familiar form

$\hat{f}(\gamma)=\int_{G}f(g)e^{-i\xi_{\gamma}(g)}\ \differential g,$ (3.65)

for $f\in L^{1}(G)\cap L^{2}(G)$. Moreover, the Haar measure on $\widehat{G}$
can be made to coincide with the Plancherel measure such that (3.56) becomes a
unitary equivalence

$\mathcal{P}:L^{2}(G)\to L^{2}(\widehat{G}).$ (3.66)

Another consequence of the fact that irreducible representations are
1-dimensional, is that the integrand in (3.60) takes values in $\\{0,1\\}$ and
(3.61) becomes an equality. By the same arguments as in Example 7, we see that
the left-invariant RKH subspaces $\mathcal{H}\subset L^{2}(G)$ are the spaces
of bandlimited functions, $\operatorname{supp}(\hat{f})\subset E$, for subsets
$E\subset\widehat{G}$ of finite Haar/Plancherel measure. Also, the kernel
$\varphi_{E}\in\mathcal{H}$ is the inverse Plancherel transform of the
characteristic function $1_{E}$.

######  Corollary 20.

If $G$ is a discrete and abelian group, then any $G$-equivariant layer is a
convolutional layer.

###### Proof.

Discrete groups are unimodular Lie group of type I, so we may use results from
the current section. We note that the integral (3.60) converges for any left-
invariant, closed subspace $\mathcal{H}\subseteq L^{2}(G)$, as the unitary
dual $\widehat{G}$ is compact when $G$ is discrete and abelian (Deitmar and
Echterhoff, 2014, Proposition 3.1.5), and because the integrand is bounded.
Consequently, $\mathcal{H}=L^{2}(G)$ is itself a RKHS,888In fact, the kernel
is simply the Kronecker delta $\varphi(g)=\delta_{g,e}$. and the same is true
for both $L^{2}(G,V_{\sigma})\simeq L^{2}(G)\otimes V_{\sigma}$ and its
closed, left-invariant subspace $L^{2}(G;\sigma)$, independently of $K\leq G$
and $(\sigma,V_{\sigma})$. The result now follows from Theorem 14. ∎

By setting $G=\mathbb{Z}^{2}$, Corollary 20 establishes that convolutional
layers are the only possible translation equivariant layers in the ordinary
CNN setting.

####  3.5.2. Compact groups

When the group $G$ is compact, all irreducible representations are finite-
dimensional. Furthermore, the unitary dual $\widehat{G}$ is discrete and the
Plancherel measure on $\widehat{G}$ is simply the counting measure. For these
reasons, the integral (3.60) reduces to a discrete sum with finite summands,
and converges iff $\hat{P}_{\gamma}=0$ for all but finitely many
$\gamma\in\widehat{G}$.

######  Corollary 21.

If $G$ is finite, then any $G$-equivariant layer is a convolutional layer.

###### Proof.

When $G$ is a finite group, $\widehat{G}$ is also finite (Folland, 2016,
Proposition 5.27) and the integral (3.60) reduces to a finite sum. That is,
$L^{2}(G)$ is a RKHS and the result now follows in the same way as Corollary
20. ∎

##  4\. Discussion

In this paper, we have investigated the mathematical foundations of
$G$-equivariant convolutional neural networks (GCNNs), which are designed for
deep learning tasks exhibiting global symmetry. We presented a basic framework
for equivariant neural networks that include both gauge equivariant neural
networks and GCNNs as special cases. We also demonstrated how GCNNs can be
obtained from homogeneous vector bundles, when $G$ is a unimodular Lie group
and $K\leq G$ is a compact subgroup.

In Theorem 14, we gave a precise criterion for when a given $G$-equivariant
layer is, in fact, a convolutional layer. This criterion uses reproducing
kernel Hilbert spaces (RKHS) and cannot be circumvented, as shown in Example
6. After discussing the relation between RKHS and bandwidth, we were able to
reformulate Theorem 14 to get an analogous bandlimit-criterion in Corollary
19. In Corollaries 20-21, we showed that the criterion is automatically
satisfied when $G$ is discrete abelian or finite, hence all $G$-equivariant
layers are convolutional layers for these groups.

One limitation of the current paper, compared to Cohen et al. (2018b); Kondor
and Trivedi (2018), is that the homogeneous space $G/K$ does not change
between layers. This restriction was made in order to limit the scope of our
analysis, and the same goes for our restriction to unimodular Lie groups. It
would be interesting to go beyond these restrictions in the future.

I am grateful for the support from my research group: Oscar Carlsson, Jan
Gerken, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and last but
not least, my advisor Daniel Persson. Thanks also to David Müller for the
interesting discussions on gauge equivariance in physics. This work was
supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP)
funded by the Knut and Alice Wallenberg Foundation.

## References

  * Albawi et al. [2017] S. Albawi, T. A. Mohammed, and S. Al-Zawi.  Understanding of a convolutional neural network.  In _International Conference on Engineering and Technology (ICET)_. IEEE, 2017.
  * Alpaydin [2014] E. Alpaydin.  _Introduction to machine learning_.  MIT Press, 2014.
  * Boyda et al. [2021] D. L. Boyda, M. N. Chernodub, N. V. Gerasimeniuk, V. A. Goy, S. D. Liubimov, and A. V. Molochkov.  Finding the deconfinement temperature in lattice yang-mills theories from outside the scaling window with machine learning.  _Physical Review D_ , 2021.
  * Brock et al. [2021] A. Brock, S. De, S. L. Smith, and K. Simonyan.  High-performance large-scale image recognition without normalization.  _arXiv preprint arXiv:2102.06171_ , 2021.
  * Bronstein et al. [2017] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst.  Geometric deep learning: going beyond euclidean data.  _IEEE Signal Processing Magazine_ , 2017.
  * Bronstein et al. [2021] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković.  Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.  _arXiv preprint arXiv:2104.13478_ , 2021.
  * Cao et al. [2020] W. Cao, Z. Yan, Z. He, and Z. He.  A comprehensive survey on geometric deep learning.  _IEEE Access_ , 2020.
  * Carey [1978] A. L. Carey.  Group representations in reproducing kernel hilbert spaces.  _Reports on Mathematical Physics_ , 1978.
  * Cheng et al. [2019] M. Cheng, V. Anagiannis, M. Weiler, P. de Haan, T. Cohen, and M. Welling.  Covariance in physics and convolutional neural networks.  _arXiv preprint arXiv:1906.02481_ , 2019.
  * Cohen and Welling [2016] T. Cohen and M. Welling.  Group equivariant convolutional networks.  In _International conference on machine learning_. PMLR, 2016.
  * Cohen et al. [2018a] T. Cohen, M. Geiger, J. Köhler, and M. Welling.  Spherical cnns.  _arXiv preprint arXiv:1801.10130_ , 2018a.
  * Cohen et al. [2018b] T. Cohen, M. Geiger, and M. Weiler.  A general theory of equivariant cnns on homogeneous spaces.  _arXiv preprint arXiv:1906.02481_ , 2018b.
  * Cohen et al. [2019] T. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling.  Gauge equivariant convolutional networks and the icosahedral cnn.  In _International Conference on Machine Learning_. PMLR, 2019.
  * Deitmar and Echterhoff [2014] A. Deitmar and S. Echterhoff.  _Principles of harmonic analysis_.  Springer International Publishing, 2014.
  * Esteves [2020] C. Esteves.  Theoretical aspects of group equivariant neural networks.  _arXiv preprint arXiv:2004.05154_ , 2020.
  * Esteves et al. [2017] C. Esteves, C.e Allen-Blanchette, A. Makadia, and K. Daniilidis.  3D object classification and retrieval with spherical cnns.  _arXiv preprint arXiv:1711.06721_ , 2017.
  * Esteves et al. [2018] C. Esteves, C. Allen-Blanchette, A. Makadia, and K. Daniilidis.  Learning so(3) equivariant representations with spherical cnns.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Favoni et al. [2020] M. Favoni, A. Ipp, D. Müller, and D. Schuh.  Lattice gauge equivariant convolutional neural networks.  _arXiv preprint arXiv:2012.12901_ , 2020.
  * Folland [2016] G. B. Folland.  _A course in abstract harmonic analysis_.  CRC press, 2016.
  * Führ [2005] H. Führ.  _Abstract harmonic analysis of continuous wavelet transforms_.  Springer Science & Business Media, 2005.
  * Fukushima and Miyake [1982] K. Fukushima and S. Miyake.  Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition.  In _Competition and cooperation in neural nets_. Springer, 1982.
  * Gattringer and Lang [2009] C. Gattringer and C. Lang.  _Quantum chromodynamics on the lattice: an introductory presentation_.  Springer Science & Business Media, 2009.
  * Gerken et al. [Forthcoming] J. E. Gerken, J. Aronsson, O. Carlsson, H. Linander, F. Ohlsson, C. Petersson, and D. Persson.  Geometric deep learning and equivariant neural networks.  Forthcoming.
  * Goodfellow et al. [2016] I. Goodfellow, Y. Bengio, and A. Courville.  _Deep learning_.  MIT Press, 2016.  URL http://www.deeplearningbook.org.
  * Jia et al. [2021] C. Jia, Y. Yang, Y. Xia, Y-T. Chen, Z. Parekh, H. Pham, Q.V. Le, Y. Sung, Z. Li, and T. Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  _arXiv preprint arXiv:2102.05918_ , 2021.
  * Kolár et al. [2013] I. Kolár, P. W. Michor, and J. Slovák.  _Natural operations in differential geometry_.  Springer Science & Business Media, 2013.
  * Koller et al. [2018] O. Koller, S. Zargaran, H. Ney, and R. Bowden.  Deep sign: Enabling robust statistical continuous sign language recognition via hybrid cnn-hmms.  _International Journal of Computer Vision_ , 2018.
  * Kondor and Trivedi [2018] R. Kondor and S. Trivedi.  On the generalization of equivariance and convolution in neural networks to the action of compact groups.  In _International Conference on Machine Learning_. PMLR, 2018.
  * Kondor et al. [2018] R. Kondor, Z. Lin, and S. Trivedi.  Clebsch-gordan nets: a fully fourier space spherical convolutional neural network.  _arXiv preprint arXiv:1806.09231_ , 2018.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 2012.
  * Lang and Weiler [2020] L. Lang and M. Weiler.  A wigner-eckart theorem for group equivariant convolution kernels.  _arXiv preprint arXiv:2010.10952_ , 2020.
  * Lawrence et al. [1997] S. Lawrence, C.L. Giles, A.C. Tsoi, and A.D. Back.  Face recognition: A convolutional neural-network approach.  _IEEE transactions on neural networks_ , 1997.
  * LeCun et al. [1995] Y. LeCun, L.D. Jackel, L. Bottou, C. Cortes, J.S. Denker, H. Drucker, I. Guyon, U.A. Muller, E. Sackinger, P. Simard, et al.  Learning algorithms for classification: A comparison on handwritten digit recognition.  _Neural networks: the statistical mechanics perspective_ , 1995.
  * Lee [2013] J. M. Lee.  _Introduction to smooth manifolds_.  Springer, 2013.
  * Li et al. [2019] J. Li, Y. Bi, and G.H. Lee.  Discrete rotation equivariance for point cloud recognition.  In _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019.
  * Luo et al. [2021] D. Luo, G. Carleo, B. Clark, and J. Stokes.  Gauge equivariant neural networks for quantum lattice gauge theories.  _Bulletin of the American Physical Society_ , 2021.
  * Mehta et al. [2017] D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei, H-P. Seidel, W. Xu, D. Casas, and C. Theobalt.  Vnect: Real-time 3d human pose estimation with a single rgb camera.  _ACM Transactions on Graphics (TOG)_ , 2017.
  * Monti et al. [2017] F. Monti, D. Boscaini, J. Masci, E. Rodola, J. Svoboda, and M. M. Bronstein.  Geometric deep learning on graphs and manifolds using mixture model cnns.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017.
  * Müller [2019] D. Müller.  Yang-mills theory, lattice gauge theory and simulations, 2019.  URL https://www.jku.at/en/institut-of-analysis/research/guest-lecture-series/.
  * Nakahara [2003] M. Nakahara.  _Geometry, topology and physics_.  CRC press, 2003.
  * Peng et al. [2017] C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun.  Large kernel matters - improve semantic segmentation by global convolutional network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017.
  * Song et al. [2019] S. Song, H. Huang, and T. Ruan.  Abstractive text summarization using lstm-cnn based deep learning.  _Multimedia Tools and Applications_ , 2019.
  * Steenrod [1960] N. Steenrod.  _The topology of fibre bundles_.  Princeton University Press, 1960.
  * Tan and Le [2021] M. Tan and Q.V. Le.  Efficientnetv2: Smaller models and faster training.  _arXiv preprint arXiv:2104.00298_ , 2021.
  * Toft et al. [2021] C. Toft, G. Bökman, and F. Kahl.  Azimuthal rotational equivariance in spherical cnns.  2021\.  URL https://openreview.net/forum?id=sp3Z1jiS2vn.
  * Veeling et al. [2018] B.S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling.  Rotation equivariant cnns for digital pathology.  In _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2018.
  * Wallach [2018] N. R. Wallach.  _Harmonic analysis on homogeneous spaces_.  M. Dekker, 2018.
  * Weiler et al. [2018] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. Cohen.  3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data.  _arXiv preprint arXiv:1807.02547_ , 2018.
  * Worrall and Brostow [2018] D. Worrall and G. Brostow.  Cubenet: Equivariance to 3d rotation and translation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Wu et al. [2021] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang.  Cvt: Introducing convolutions to vision transformers.  _arXiv preprint arXiv:2103.15808_ , 2021.
  * Wu [2017] J. Wu.  Introduction to convolutional neural networks.  _National Key Lab for Novel Software Technology. Nanjing University. China_ , 2017.
  * Xu et al. [2017] Y. Xu, Z. Jia, L-B. Wang, Y. Ai, F. Zhang, M. Lai, I Eric, and C. Chang.  Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features.  _BMC bioinformatics_ , 2017.
  * Zhang et al. [2018] Y. Zhang, Z. Qiu, T. Yao, D. Liu, and T. Mei.  Fully convolutional adaptation networks for semantic segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.