# Deformation Robust Roto-Scale-Translation Equivariant CNNs

Liyao Gao  Department of Statistics, University of Washington  Guang Lin
Department of Mathematics and School of Mechanical Engineering, Purdue
University  Wei Zhu111Email<EMAIL_ADDRESS>Department of Mathematics and
Statistics, University of Massachusetts Amherst

###### Abstract

Incorporating group symmetry directly into the learning process has proved to
be an effective guideline for model design. By producing features that are
guaranteed to transform covariantly to the group actions on the inputs, group-
equivariant convolutional neural networks (G-CNNs) achieve significantly
improved generalization performance in learning tasks with intrinsic symmetry.
General theory and practical implementation of G-CNNs have been studied for
planar images under either rotation or scaling transformation, but only
individually. We present, in this paper, a roto-scale-translation equivariant
CNN ($\mathcal{RST}$-CNN), that is guaranteed to achieve equivariance jointly
over these three groups via coupled group convolutions. Moreover, as symmetry
transformations in reality are rarely perfect and typically subject to input
deformation, we provide a stability analysis of the equivariance of
representation to input distortion, which motivates the truncated expansion of
the convolutional filters under (pre-fixed) low-frequency spatial modes. The
resulting model provably achieves deformation-robust $\mathcal{RST}$
equivariance, i.e., the $\mathcal{RST}$ symmetry is still “approximately”
preserved when the transformation is “contaminated” by a nuisance data
deformation, a property that is especially important for out-of-distribution
generalization. Numerical experiments on MNIST, Fashion-MNIST, and STL-10
demonstrate that the proposed model yields remarkable gains over prior arts,
especially in the small data regime where both rotation and scaling variations
are present within the data.

##  1 Introduction

Symmetry is ubiquitous in machine learning. For instance, in image
classification, the class label of an image remains the same when the image is
spatially translated. Convolutional neural networks (CNNs) through spatial
weight sharing achieve built-in translation-equivariance, i.e., a shift of the
input leads to a corresponding shift of the output, a property that improves
the generalization performance and sample complexity of the model for computer
vision tasks with translation symmetry, such as image classification [25],
object detection [36], and segmentation [27, 37].

Inspired by the standard CNNs, researchers in recent years have developed both
theoretical foundations and practical implementations of group equivariant
CNNs (G-CNNs), i.e., generalized CNN models that guarantee a desired
transformation on layer-wise features under a given input group
transformation, for signals defined on Euclidean spaces [11, 12, 46, 43, 42,
45, 39, 9, 51, 41, 19, 50, 44, 20, 48, 32], manifolds [13, 14, 24, 15], point
clouds [40, 8, 49], and graphs [23, 2, 21]. In particular, equivariant CNNs
for either rotation [41, 9, 19, 46, 50, 33, 43] or scaling [20, 32, 48, 45,
39, 51] transforms on 2D inputs have been well studied separately, and their
advantage has been empirically verified in settings where the data have rich
variance in either rotation or scale individually.

However, for many vision tasks, it is beneficial for a model to simultaneously
incorporate translation, rotation, and scaling symmetry directly into its
representation. For example, a self-driving vehicle is required to recognize
and locate pedestrians, objects, and road signs under random translation
(e.g., moving pedestrians), rotation (e.g., tilted road signs), and scaling
(e.g., close and distant objects) [6]. Moreover, in realistic settings,
symmetry transformations are rarely perfect; for instance, a tilted stop sign
located faraway can be modeled in reality as if it were transformed through a
sequence of translation, rotation and scaling following a local data
deformation, which results from (unavoidable) changing view angle and/or
digitization. It is thus crucial to design roto-scale-translation equivariant
CNNs ($\mathcal{RST}$-CNNs) with provably robust equivariant representation
such that the $\mathcal{RST}$ symmetry is still “approximately” preserved when
the transformation is “contaminated” by a nuisance data deformation. Such
deformation robustness is especially important for out-of-distribution
generalization when the equivariant model is trained on, for instance, only
upright images within a particular scale range but tested on randomly rotated
images at a different scale.

The purpose of this paper is to address both the theoretical and practical
aspects of constructing deformation robust $\mathcal{RST}$-CNNs, which, to the
best of our knowledge, have not been studied in the computer vision community.
Specifically, our contribution is three-fold:

  1. 1.

We propose roto-scale-translation equivariant CNNs with joint convolutions
over the space $\mathbb{R}^{2}$, the rotation group $SO(2)$, and the scaling
group $\mathcal{S}$, which is shown to be sufficient and necessary for
equivariance with respect to the regular representation of the group
$\mathcal{RST}$.

  2. 2.

We provide a stability analysis of the proposed model, guaranteeing its
ability to achieve equivariant representations that are robust to nuisance
data deformation.

  3. 3.

Numerical experiments are conducted to demonstrate the superior (both in-
distribution and out-of-distribution) generalization performance of our
proposed model for vision tasks with intrinsic $\mathcal{RST}$ symmetry,
especially in the small data regime.

##  2 Related Works

Group-equivariant CNNs (G-CNNs). Since its introduction by Cohen and Welling
[11], a variety of works on G-CNNs have been conducted that consistently
demonstrate the benefits of bringing equivariance prior into network designs.
Based on the idea proposed in [11] for discrete symmetry groups, G-CNNs with
group convolutions which achieve equivariance under regular representations of
the group have been studied for the 2D (and 3D) roto-translation group $SE(2)$
(and $SE(3)$) [41, 9, 19, 46, 50, 33, 43, 44], scaling-translation group [20,
32, 48, 45, 39, 51], rotation $SO(3)$ on the sphere [13, 24, 15], and
permutation on graphs [23, 2, 21]. B-spline CNNs [3] and LieConv [17]
generalize group convolutions to arbitrary Lie groups on generic spatial data,
albeit achieving slightly inferior performance compared to G-CNNs specialized
for Euclidean inputs [17]. Steerable CNNs further generalize the network
design to realize equivariance under induced representations of the symmetry
group [12, 42, 41, 14], and the general theory has been summarized in [14] for
homogeneous spaces.

Representation robustness to input deformations. Input deformations typically
introduce noticeable yet uninformative variability within the data. Models
that are robust to data deformation are thus favorable for many vision
applications. The scattering transform network [7, 30, 31], a multilayer
feature encoder defined by average pooling of wavelet modulus coefficients,
has been proved to be stable to both input noise and nuisance deformation.
Using group convolutions, scattering transform has also been extended in [34,
38] to produce rotation/translation-invariant features. Despite being a
pioneering mathematical model, the scattering network uses pre-fixed wavelet
transforms in the model, and is thus non-adaptive to the data. Stability and
invariance have also been studied in [4, 5] for convolutional kernel networks
[28, 29]. DCFNet [35] combines the regularity of a pre-fixed filter basis and
the trainability of the expansion coefficients, achieving both representation
stability and data adaptivity. The idea is later adopted in [9] and [51] in
building deformation robust models that are equivariant to either rotation or
scaling individually.

Despite the growing body of literature in G-CNNs, to the best of our
knowledge, no G-CNNs have been specifically designed to simultaneously achieve
roto-scale-translation equivariance. More importantly, no stability analysis
has been conducted to quantify and promote robustness of such equivariant
model to nuisance input deformation.

##  3 Roto-Scale-Translation Equivariant CNNs

We first explain, in Section 3.1 , the definition of groups, group
representations, and group-equivariance, which serves as a background for
constructing $\mathcal{RST}$-CNNs in Section 3.2.

###  3.1 Preliminaries

Group. A group is a set $G$ equipped with a binary operator $\cdot:G\times
G\to G$, satisfying associativity and the existence of an identity $e$ as well
as an inverse element $g^{-1}$ for all $g\in G$. In this paper, we consider in
particular the roto-scale-translation group
$\mathcal{RST}=(SO(2)\times\mathbb{R})\ltimes\mathbb{R}^{2}=\\{(\eta,\beta,v):\eta\in[0,2\pi],\beta\in\mathbb{R},v\in\mathbb{R}^{2}\\}$,
with the group multiplication

$\displaystyle(\eta,\beta,v)\cdot(\theta,\alpha,u)=(\theta+\eta,\alpha+\beta,v+R_{\eta}2^{\beta}u),$
(1)

where $R_{\eta}u$ is a counterclockwise rotation (around the origin) by angle
$\eta$ applied to a point $u\in\mathbb{R}^{2}$.

Group action and representation. Given a group $G$ and a set $\mathcal{X}$,
$D_{g}:\mathcal{X}\to\mathcal{X}$ is called a $G$-action on $\mathcal{X}$ if
$D_{g}$ is invertible for all $g\in G$, and $D_{g_{1}}\circ
D_{g_{2}}=D_{g_{1}\cdot g_{2}},~{}\forall g_{1},g_{2}\in G$, where $\circ$
denotes map composition. A $G$-action $D_{g}$ is called a $G$-representation
if $\mathcal{X}$ is further assumed to be a vector space and $D_{g}$ is linear
for all $g\in G$. In particular, given an input RGB image $x^{(0)}(u,\lambda)$
modeled in the continuous setting (i.e., $x^{(0)}$ is the intensity of the RGB
color channel $\lambda\in\\{1,2,3\\}$ at the pixel location
$u\in\mathbb{R}^{2}$), a roto-scale-translation transformation on the image
$x^{(0)}(u,\lambda)$ can be understood as an $\mathcal{RST}$-action
(representation) $D_{g}^{(0)}=D_{\eta,\beta,v}^{(0)}$ acting on the input
$x^{(0)}$:

$\displaystyle[D_{\eta,\beta,v}^{(0)}x^{(0)}](u,\lambda)=x^{(0)}\left(R_{-\eta}2^{-\beta}(u-v),\lambda\right),$
(2)

i.e., the transformed image $[D_{\eta,\beta,v}x^{(0)}]$ is obtained through an
$\eta$ rotation, $2^{\beta}$ scaling, and $v$ translation.

Group Equivariance. Let $f:\mathcal{X}\to\mathcal{Y}$ be a map between
$\mathcal{X}$ and $\mathcal{Y}$, and $D_{g}^{\mathcal{X}},D_{g}^{\mathcal{Y}}$
be $G$-actions on $\mathcal{X}$ and $\mathcal{Y}$ respectively. The map $f$ is
said to be $G$-equivariant if

$\displaystyle
f(D_{g}^{\mathcal{X}}x)=D_{g}^{\mathcal{Y}}(f(x)),\quad\forall~{}g\in
G,~{}x\in\mathcal{X}.$ (3)

A special case of (3) is $G$-invariance, when $D_{g}^{\mathcal{Y}}$ is set to
$\text{Id}_{\mathcal{Y}}$, the identity map on $\mathcal{Y}$. For vision tasks
where the output $y\in\mathcal{Y}$ is known a priori to transform covariantly
through $D_{g}^{\mathcal{Y}}$ to a $D_{g}^{\mathcal{X}}$ transformed input,
e.g., the class label $y$ remains identical for a rotated/rescaled/shifted
input $x\in\mathcal{X}$, it is beneficial to consider only $G$-equivariant
models $f$ to reduce the statistical error of the learning method for improved
generalization.

###  3.2 Equivariant Architecture

Since the composition of equivariant maps remains equivariant, to construct an
$L$-layer $\mathcal{RST}$-CNN, we only need to specify the
$\mathcal{RST}$-action $D_{\eta,\beta,v}^{(l)}$ on each feature space
$\mathcal{X}^{(l)}$, $0\leq l\leq L$, and require the layer-wise mapping to be
equivariant:

$\displaystyle
x^{(l)}[D^{(l-1)}_{\eta,\beta,v}x^{(l-1)}]=D^{(l)}_{\eta,\beta,v}x^{(l)}[x^{(l-1)}],~{}\forall
l\geq 1,$ (4)

where we slightly abuse the notation $x^{(l)}[x^{(l-1)}]$ to denote the $l$-th
layer output given the $(l-1)$-th layer feature $x^{(l-1)}$. In particular, we
define $D_{\eta,\beta,v}^{(0)}$ on the input as in (2); for the hidden layers
$1\leq l\leq L$, we let $\mathcal{X}^{(l)}$ be the feature space consisting of
features in the form of $x^{(l)}(u,\theta,\alpha,\lambda)$, where
$u\in\mathbb{R}^{2}$ is the spatial position, $\theta\in[0,2\pi]$ is the
rotation index, $\alpha\in\mathbb{R}$ is the scale index,
$\lambda\in[M_{l}]\coloneqq\left\\{1,\ldots,M_{l}\right\\}$ corresponds to the
unstructured channels (similar to the RGB channels of the input), and we
define the action $D_{\eta,\beta,v}^{(l)}$ on $\mathcal{X}^{(l)}$ as

$\displaystyle[D_{\eta,\beta,v}^{(l)}x^{(l)}](u,\theta,\alpha,\lambda)=x^{(l)}\left(R_{-\eta}2^{-\beta}(u-v),\theta-\eta,\alpha-\beta,\lambda\right),~{}\forall
l\geq 1.$ (5)

We note that (5) corresponds to the regular representation of $\mathcal{RST}$
on $\mathcal{X}^{(l)}$ [14], which is adopted in this work as its ability to
encode any function on the group $\mathcal{RST}$ leads typically to better
model expressiveness and stronger generalization performance [41]. The
following proposition outlines the general network architecture to achieve
$\mathcal{RST}$-equivariance under the representations
$D_{\eta,\beta,v}^{(l)}$ (2) (5).

######  Proposition 1.

An $L$-layer feedforward neural network is $\mathcal{RST}$-equivariant under
the representations (2) (5) if and only if the layer-wise operations are
defined as (6) and (7):

$\displaystyle
x^{(1)}[x^{(0)}](u,\theta,\alpha,\lambda)=\sigma\bigg{[}\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}x^{(0)}(u+u^{\prime},\lambda^{\prime})2^{-2\alpha}W_{\lambda^{\prime},\lambda}^{(1)}\left(2^{-\alpha}R_{-\theta}u^{\prime}\right)du^{\prime}+b^{(1)}(\lambda)\bigg{]},$
(6) $\displaystyle
x^{(l)}[x^{(l-1)}](u,\theta,\alpha,\lambda)=\sigma\bigg{[}\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}\int_{S^{1}}\int_{\mathbb{R}}x^{(l-1)}(u+u^{\prime},\theta+\theta^{\prime},\alpha+\alpha^{\prime},\lambda^{\prime})$
$\displaystyle\quad\quad\quad\quad\cdot
2^{-2\alpha}W_{\lambda^{\prime},\lambda}^{(l)}\left(2^{-\alpha}R_{-\theta}u^{\prime},\theta^{\prime},\alpha^{\prime}\right)d\alpha^{\prime}d\theta^{\prime}du^{\prime}+b^{(l)}(\lambda)\bigg{]}$
(7)

where $\sigma:\mathbb{R}\to\mathbb{R}$ is a pointwise nonlinearity,
$W_{\lambda^{\prime},\lambda}^{(1)}(u)$ is the spatial convolutional filter in
the first layer with output channel $\lambda$ and input channel
$\lambda^{\prime}$, $W_{\lambda^{\prime},\lambda}^{(l)}(u,\theta,\alpha)$ is
the $\mathcal{RST}$ joint convolutional filter for layer $l>1$, and
$\int_{S^{1}}f(\alpha)d\alpha$ denotes the normalized $S^{1}$ integral
$\frac{1}{2\pi}\int_{0}^{2\pi}f(\alpha)d\alpha$.

We note that the joint-convolution (7) is the group convolution over
$\mathcal{RST}$ (whose subgroup $SO(2)\times\mathbb{R}$ is a non-compact Lie
group acting on $\mathbb{R}^{2}$), which is known to achieve equivariance
under regular representations (5) [14]. We provide an elementary proof of
Proposition 1 in the appendix for completeness.

##  4 Robust Equivariance to Input Deformation

Proposition 1 details the network architecture to achieve
$\mathcal{RST}$-equivariance for images modeled on the continuous domain
$\mathbb{R}^{2}$ undergoing a “perfect” $\mathcal{RST}$-transformation (2).
However, in practice, symmetry transformations are rarely perfect, as they
typically suffer from numerous source of input deformation coming from, for
instance, unstable camera position, change of weather, as well as practical
issues such as numerical discretization and truncation (see, for example,
Figure 1.) We explain, in this section, how to quantify and improve the
representation stability of the model such that it stays “approximately”
$\mathcal{RST}$-equivariant even if the input transformation is “contaminated”
by minute local distortion (see, for instance, Figure 2.)

Figure 1: $\mathcal{RST}$-transformations in reality, e.g., the blue arrow,
are rarely perfect, and they can be modeled as a composition of input
deformation $D_{\tau}$ [cf. (12)] and an exact $\mathcal{RST}$-transformation
$D_{\eta,\beta,v}^{(0)}$.

###  4.1 Decomposition of Convolutional Filters

In order to quantify the deformation stability of representation equivariance,
motivated by [35, 9, 51], we leverage the geometry of the group
$\mathcal{RST}$ and decompose the convolutional filters
$W_{\lambda,\lambda^{\prime}}^{(l)}(u,\theta,\alpha)$ under the separable
product of three orthogonal function bases, $\\{\psi_{k}(u)\\}_{k}$,
$\\{\varphi_{m}(\theta)\\}_{m}$, and $\\{\xi_{n}(\alpha)\\}_{n}$. In
particular, we choose $\\{\varphi_{m}\\}_{m}$ as the Fourier basis on $S^{1}$,
and set $\\{\psi_{k}\\}_{k}$ and $\\{\xi_{n}\\}_{n}$ to be the eigenfunctions
of the Dirichlet Laplacian over, respectively, the unit disk
$D\subset\mathbb{R}^{2}$ and the interval $I_{\alpha}=[-1,1]$, i.e.,

$\displaystyle\left\\{\begin{aligned}
\Delta\psi_{k}=-\mu_{k}\psi_{k}~{}~{}&\text{in}~{}D,\\\
\psi_{k}=0~{}~{}&\text{on}~{}~{}\partial
D,\end{aligned}\right.~{}~{}\left\\{\begin{aligned}
&\xi_{n}^{\prime\prime}=-\nu_{m}\xi_{n}~{}~{}\text{in}~{}I_{\alpha}\\\
&\xi_{n}(\pm 1)=0,\end{aligned}\right.$ (8)

where $\mu_{k}$ and $\psi_{n}$ are the corresponding eigenvalues.

######  Remark 1.

One has flexibility in choosing the spatial function basis
$\\{\psi_{k}\\}_{k}$. We consider mainly, in this work, the rotation-steerable
Fourier-Bessel (FB) basis [1] defined in (8), as the spatial regularity of its
low-frequency modes leads to robust equivariance to input deformation in an
$\mathcal{RST}$-CNN, which will be shown in Theorem 1. One can also choose
$\\{\psi_{k}\\}_{k}$ to be the eigenfunctions of Dirichlet Laplacian over the
cell $[-1,1]^{2}$ [51], i.e., the separable product of the solutions to the 1D
Dirichlet Sturm–Liouville problem, which leads to a similar stability
analysis. We denote such basis the Sturm-Liouvielle (SL) basis, and its
efficacy will be compared to FB basis in Section 6.

Since spatial pooling can be modeled as rescaling the convolutional filters in
space, we assume the filters $W^{(l)}_{\lambda^{\prime},\lambda}$ are
compactly supported on a rescaled domain as follows

$\displaystyle W_{\lambda^{\prime},\lambda}^{(1)}\in
C_{c}(2^{j_{1}}D),~{}W_{\lambda^{\prime},\lambda}^{(l)}\in
C_{c}(2^{j_{l}}D\times S^{1}\times I_{\alpha}),$ (9)

where $j_{l}<j_{l+1}$ models a sequence of filters with decreasing size. Let
$\psi_{j,k}(u)=2^{-2j}\psi_{k}(2^{-j}u)$ be the rescaled spatial basis
function, and we can decompose $W_{\lambda^{\prime},\lambda}^{(l)}$ under
$\\{\psi_{j_{l},k}\\}_{k}$, $\\{\varphi_{m}\\}_{m}$, $\\{\xi_{n}\\}_{n}$ into

$\displaystyle
W_{\lambda^{\prime},\lambda}^{(1)}(u)=\sum_{k}a_{\lambda^{\prime},\lambda}^{(1)}(k)\psi_{j_{1},k}(u)$
(10) $\displaystyle
W_{\lambda^{\prime},\lambda}^{(l)}(u,\theta,\alpha)=\sum_{k,m,n}a_{\lambda^{\prime},\lambda}^{(l)}(k,m,n)\psi_{j_{1},k}(u)\varphi_{m}(\theta)\xi_{n}(\alpha),$

where $a_{\lambda^{\prime},\lambda}^{(l)}$ are the expansion coefficients of
the filters. We explain, in Section 4.2, the effect of
$a_{\lambda,\lambda^{\prime}}^{(l)}$ on the stability of the
$\mathcal{RST}$-equivariant representation to input deformation.

###  4.2 Stability under Input Deformation

First, in order to gauge the distance between different inputs and features,
we define the layer-wise feature norm as

$\displaystyle\left\|x^{(0)}\right\|^{2}=\frac{1}{M_{0}}\sum_{\lambda=1}^{M_{0}}\int_{\mathbb{R}^{2}}|x^{(0)}(u,\lambda)|^{2}du,$
(11)
$\displaystyle\left\|x^{(l)}\right\|^{2}=\sup_{\alpha}\frac{1}{M_{l}}\sum_{\lambda=1}^{M_{l}}\iint_{S^{1}\times\mathbb{R}^{2}}|x^{(l)}(u,\theta,\alpha,\lambda)|^{2}dud\theta,\quad\forall
l\geq 1,$

i.e., the norm is a combination of an $L^{2}$-norm over the roto-translation
group $SE(2)\cong S^{1}\times\mathbb{R}^{2}$ and an $L^{\infty}$-norm over the
scaling group $\mathcal{S}\cong\mathbb{R}$.

We next define the spatial deformation of an input image. Given a $C^{2}$
vector field $\tau:\mathbb{R}^{2}\to\mathbb{R}^{2}$, the spatial deformation
$D_{\tau}$ on $x^{(0)}$ is defined as

$\displaystyle D_{\tau}x^{(0)}(u,\lambda)=x^{(0)}(\rho(u),\lambda),$ (12)

where $\rho(u)=u-\tau(u)$. Thus $\tau(u)$ is understood as the local image
distortion (at pixel location $u$), and $D_{\tau}$ is the identity map if
$\tau(u)\equiv 0$, i.e., not input distortion.

The deformation stability of the equivariant representation can be quantified
in terms of (11) after we make the following three mild assumptions on the
model and the input distortion $D_{\tau}$:

(A1): The nonlinearity $\sigma:\mathbb{R}\to\mathbb{R}$ is non-expansive,
i.e., $|\sigma(x)-\sigma(y)|\leq|x-y|,~{}\forall x,y\in\mathbb{R}$. For
instance, the rectified linear unit (ReLU) satisfies this assumption.

(A2): The convolutional filters are bounded in the following sense: $A_{l}\leq
1,\forall l\geq 1$, where

$\displaystyle A_{1}$
$\displaystyle\coloneqq\pi\max\Big{\\{}\sup_{\lambda}\sum_{\lambda^{\prime}}\|a_{\lambda^{\prime},\lambda}^{(1)}\|_{\text{FB}},\frac{M_{0}}{M_{1}}\sup_{\lambda^{\prime}}\sum_{\lambda^{\prime}}\|a_{\lambda^{\prime},\lambda}^{(1)}\|_{\text{FB}}\Big{\\}}$
$\displaystyle A_{l}$
$\displaystyle\coloneqq\pi\max\Big{\\{}\sup_{\lambda}\sum_{\lambda^{\prime}}\sum_{n}\|a_{\lambda^{\prime},\lambda}^{(l)}(\cdot,n)\|_{\text{FB}},\frac{2M_{l-1}}{M_{l}}\sum_{n}\sup_{\lambda^{\prime}}\sum_{\lambda}\|a_{\lambda^{\prime},\lambda}^{(l)}(\cdot,n)\|_{\text{FB}}\Big{\\}},~{}\forall
l>1,$

in which the FB-norm $\|\cdot\|_{\text{FB}}$ of a sequence $\\{a(k)\\}_{k\geq
0}$ and double sequence $\\{b(k,m)\\}_{k,m\geq 0}$ is the weighted
$l_{2}$-norm defined as

$\displaystyle\|a\|_{\text{FB}}^{2}=\sum_{k}\mu_{k}a(k)^{2},\|b\|_{\text{FB}}^{2}=\sum_{k}\sum_{m}\mu_{k}b(k,m)^{2},$
(13)

$\mu_{k}$ being the eigenvalues defined in (8).

(A3): The local input distortion is small:

$\displaystyle|\nabla\tau|_{\infty}\coloneqq\sup_{u}\|\nabla\tau(u)\|<1/5,$
(14)

where $\|\cdot\|$ is the operator norm.

Figure 2: Approximate $\mathcal{RST}$-equivariance in the presence of nuisance
input deformation $D_{\tau}$.

Theorem 1 below quantifies the deformation stability of the equivariant
representation in an $\mathcal{RST}$-CNN under the assumptions (A1)-(A3):

######  Theorem 1.

Let $D_{\eta,\beta,v}^{(l)}$, $0\leq l\leq L$, be the $\mathcal{RST}$ group
actions defined in (2)(5), and let $D_{\tau}$ be a small input deformation
define in (12). If an $\mathcal{RST}$-CNN satisfies the assumptions (A1)-(A3),
we have, for any $L$,

$\displaystyle\left\|x^{(L)}[D_{\eta,\beta,v}^{(0)}\circ
D_{\tau}x^{(0)}]-D_{\eta,\beta,v}^{(L)}x^{(L)}[x^{(0)}]\right\|\leq
2^{\beta+1}\left(4L|\nabla\tau|_{\infty}+2^{-j_{L}}|\tau|_{\infty}\right)\|x^{(0)}\|.$
(15)

The proof of Theorem 1 is deferred to the Appendix. An important message from
Theorem 1 is that as long as (A1)-(A3) are satisfied, the model stays
approximately $\mathcal{RST}$-equivariant, i.e.,
$x^{(L)}[D_{\eta,\beta,v}^{(0)}\circ D_{\tau}x^{(0)}]\approx
D_{\eta,\beta,v}^{(L)}x^{(L)}[x^{(0)}]$, even with the presence of non-zero
(yet small) input deformation $D_{\tau}$ (see, e.g., Figure 2.)

######  Remark 2.

The fact that the right hand side of (15) grows exponentially with $\beta$ is
inevitable, as it comes naturally from the definition of the norm (11): if an
image is spatially rescaled by $2^{\beta}$ without correspondingly scaling the
color range (i.e., pixel intensity), its $L^{2}$-norm is enlarged by
$2^{\beta}$.

######  Remark 3.

According to the definition of the FB-norm (13), the main assumption (A2) can
be facilitated by truncating the filter expansion (10) to include only low-
frequency (small $\mu_{k}$) components. The implementation detail of such
truncated filter expansion will be explained in detail in Section 5.

##  5 Implementation Details

We next discuss the implementation details of the $\mathcal{RST}$-CNN outlined
in Proposition 1.

Discretization. To implement $\mathcal{RST}$-CNN in practice, we first need to
discretize the features $x^{(l)}$ modeled originally under the continuous
setting. First, the input signal $x^{(0)}(u,\lambda)$ is discretized on a
uniform grid into a 3D array of shape $[M_{0},H_{0},W_{0}]$, where
$H_{0},W_{0},M_{0}$, respectively, are the height, width, and the number of
the unstructured channels of the input (e.g., $M_{0}=3$ for RGB images.) For
$l\geq 1$, the rotation group $S^{1}$ is uniformly discretized into $N_{r}$
points; the scaling group $\mathcal{S}\cong\mathbb{R}^{1}$, unlike $S^{1}$, is
unbounded, and thus features $x^{(l)}$ are computed and stored only on a
truncated scale interval $I=[-T,T]\subset\mathbb{R}$, which is uniformly
discretized into $N_{s}$ points. The feature $x^{(l)}$ is therefore stored as
a 5D array of shape $[M_{l},N_{r},N_{s},H_{l},W_{l}]$.

Filter expansion. The analysis in Section 4 suggests that robust
$\mathcal{RST}$-equivariance is achieved if the convolutional filters are
expanded with only the first $K$ low-frequency spatial modes
$\\{\psi_{k}\\}_{k=1}^{K}$. More specifically, the first $K$ spatial basis
functions as well as their rotated and rescaled versions
$\\{2^{-2\alpha}\psi_{k}(2^{-\alpha}R_{-\theta}u^{\prime})\\}_{k,\theta,\alpha,u^{\prime}}$
are sampled on a uniform grid of size $L\times L$ and stored as an array of
size $[K,N_{r},N_{s},L,L]$, which is fixed during training. The expansion
coefficients $a_{\lambda^{\prime},\lambda}^{(l)}$, on the other hand, are the
trainable parameters of the model, which are used together with the fixed
basis to linearly expand the filters. The resulting filters
$\\{2^{-2\alpha}W_{\lambda^{\prime},\lambda}^{(1)}(2^{-\alpha}R_{-\theta}u^{\prime})\\}_{\lambda^{\prime},\lambda,\theta,\alpha,u^{\prime}}$
and
$\\{2^{-2\alpha}W_{\lambda^{\prime},\lambda}^{(l)}(2^{-\alpha}R_{-\theta}u^{\prime},\theta^{\prime},\alpha^{\prime})\\}_{\lambda^{\prime},\lambda,\theta,\theta^{\prime},\alpha,\alpha^{\prime},u^{\prime}}$
are stored, respectively, as tensors of size $[M_{0},M_{1},N_{r},N_{s},L,L]$
and $[M_{l-1},M_{l},N_{r},L_{\theta},N_{s},L_{\alpha},L,L]$, where
$L_{\alpha}$ is the number of grid points sampling the interval $I_{\alpha}$
in (8), and $L_{\theta}$ is the number of grid points sampling $S^{1}$ on
which the integral $\int_{S^{1}}d\theta$ in (7) is performed.

######  Remark 4.

The number $L_{\alpha}$ measures the support $I_{\alpha}$ (8) of the
convolutional filters in scale, which corresponds to the amount of “inter-
scale” information transfer when performing the convolution over scale
$\int_{\mathbb{R}}(\cdots)d\alpha^{\prime}$ in (7). It is typically chosen to
be a small number (e.g., 1 or 2) to avoid the “boundary leakage effect” [45,
39, 51], as one needs to pad unknown values beyond the truncated scale channel
$[-T,T]$ during convolution (7) when $L_{\alpha}>1$. The number $L_{\theta}$,
on the other hand, corresponds to the “inter-rotation” information transfer
when performing the convolution over the rotation group
$\int_{S^{1}}(\cdots)d\theta^{\prime}$ in (7); it does not have to be small
since periodic-padding of known values is adopted when conducting integrals on
$S^{1}$ with no “boundary leakage effect”. We only require $L_{\theta}$ to
divide $N_{r}$ such that $\int_{S^{1}}(\cdots)d\theta^{\prime}$ is computed on
a (potentially) coarser grid (of size $L_{\theta}$) compared to the finer grid
(of size $N_{r}$) on which we discretize the rotation channel of the feature
$x^{(l)}$.

Discrete convolution. After generating, in the previous step, the discrete
joint convolutional filters together with their rotated and rescaled versions,
the continuous convolutions in Proposition 1 can be efficiently implemented
using regular 2D discrete convolutions.

More specifically, let $x^{(0)}(u,\lambda)$ be an input image of shape
$[M_{0},H_{0},W_{0}]$. A total of $N_{r}\times N_{s}$ discrete 2D convolutions
with the rotated and rescaled filters
$\\{2^{-2\alpha}\psi_{k}(2^{-\alpha}R_{-\theta}u^{\prime})\\}_{k,\theta,\alpha,u^{\prime}}$,
i.e., replacing the spatial integrals in (7) by summations, are conducted to
obtain the first-layer feature $x^{(1)}(u,\theta,\alpha,\lambda)$ of size
$[M_{1},N_{r},N_{s},H_{1},W_{1}]$. For the subsequent layers, given a feature
$x^{(l-1)}(u,\theta,\alpha,\lambda)$ of shape
$[M_{l-1},N_{r},N_{s},H_{l-1},W_{l-1}]$ and the joint filters
$F^{(l)}=\\{2^{-2\alpha}W_{\lambda^{\prime},\lambda}^{(l)}(2^{-\alpha}R_{-\theta}u^{\prime},\theta^{\prime},\alpha^{\prime})\\}_{\lambda^{\prime},\lambda,\theta,\theta^{\prime},\alpha,\alpha^{\prime},u^{\prime}}$
of size $[M_{l-1},M_{l},N_{r},L_{\theta},N_{s},L_{\alpha},L,L]$, the next-
layer feature $x^{(l)}$ is computed in the following way: for each
$l_{\alpha}\in[0,L_{\alpha}-1]$ and $l_{\theta}\in[0,L_{\theta}-1]$, we shift
the signal $x^{(l-1)}$ in the scale channel by $l_{\alpha}$ and in the
rotation channel by $l_{\theta}N_{r}/L_{\theta}$, which is then convolved with
the filter $F[:,:,:,l_{\theta},:,l_{\alpha},:,:]$ (after proper reshaping and
combining adjacent dimensions) to produce an output array of shape
$[M_{l},N_{r},N_{s},H_{l},W_{l}]$. The $l$-th layer feature map
$x^{(l)}(u,\theta,\alpha,\lambda)$ is then computed as the sum of the
$L_{\theta}\times L_{\alpha}$ tensors obtained by iterating over
$l_{\theta}\in[0,L_{\theta}-1]$ and $l_{\alpha}\in[0,L_{\alpha}-1]$.

Group pooling. For learning tasks where the outputs are supposed to remain
unchanged to $\mathcal{RST}$-transformed inputs, e.g., image classification, a
max-pooling over the entire group $\mathcal{RST}$ is performed on the last-
layer feature $x^{(L)}(u,\theta,\alpha,\lambda)$ of shape
$[M_{L},N_{r},N_{s},H_{L},M_{L}]$ to produce an $\mathcal{RST}$-invariant 1D
output of length $M_{L}$. We only perform the $\mathcal{RST}$ group-pooling in
the last layer without explicit mention.

##  6 Numerical Experiments

We conduct numerical experiments, in this section, to demonstrate:

  * •

The proposed model indeed achieves robust $\mathcal{RST}$-equivariance under
realist settings.

  * •

$\mathcal{RST}$-CNN yields remarkable gains over prior arts in vision tasks
with intrinsic $\mathcal{RST}$-symmetry, especially in the small data regime.

Software implementation of the experiments is included in the supplementary
materials.

###  6.1 Data Sets and Models

We conduct the experiments on the Rotated-and-Scaled MNIST (RS-MNIST),
Rotated-and-Scaled Fashion-MNIST (RS-Fashion), as well as the STL-10 data sets
[10].

RS-MNIST and RS-Fashion are constructed through randomly rotating (by an angle
uniformly distributed on $[0,2\pi]$) as well as rescaling (by a uniformly
random factor from [0.3, 1]) the original MNIST [26] and Fashion-MNIST [47]
images. The transformed images are zero-padded back to a size of $28\times
28$. We upsize the image to $56\times 56$ for better comparison of the models.

The STL-10 data set has 5,000 training and 8,000 testing RGB images of size
$96\times 96$ belonging to 10 different classes such as cat, deer, and dog. We
use this data set to evaluate different models under both in-distribution (ID)
and out-of-distribution (OOD) settings. More specifically, the training set
remains unchanged, while the testing set is either unaltered for ID testing,
or randomly rotated (by an angle uniformly distributed on $[-\pi/2,\pi/2]$)
and rescaled (by a factor uniformly distributed on $[0.8,1]$) for OOD testing.

We evaluate the performance of the proposed $\mathcal{RST}$-CNN against other
models that are equivariant to either roto-translation ($SE(2)$) or scale-
translation ($\mathcal{ST}$) of the inputs. The $SE(2)$-equivariant models
consider in this section include the Rotation Decomposed Convolutional Filters
network (RDCF) [9] and the Rotation Equivariant Steerable Network (RESN) [43],
which is shown to achieve best performance among all $SE(2)$-equivariant CNNs
in [41]. The $\mathcal{ST}$-equivariant models include the Scale Equivariant
Vector Field Network (SEVF) [32], Scale Equivariant Steerable Network (SESN)
[39], and Scale Decomposed Convolutional Filters network (SDCF) [51].

###  6.2 Equivariance Error

We first measure the $\mathcal{RST}$-equivariance error of our proposed model
with the presence of discretization and scale channel truncation. More
specifically, we construct a 5-layer $\mathcal{RST}$-CNN with randomly
initialized expansion coefficients $a_{\lambda^{\prime},\lambda}^{(l)}$
truncated to $K=5$ or $K=10$ low-frequency spatial (FB) modes
$\\{\psi_{k}\\}_{k=1}^{K}$. The scale channel is truncated to $[-1,1]$, which
is uniformly discretized into $N_{s}=9$ points; the rotation group $S^{1}$ is
sampled on a uniform grid of size $N_{r}=8$. The $\mathcal{RST}$ equivariance
error is computed on random RS-MNIST images, and measured in a relative
$L^{2}$ sense at the scale $\alpha=0$ and rotation $\theta=0$, with the
$\mathcal{RST}$-action corresponding to the group element
$(\eta,\beta,v)=(-\pi/2,-0.5,0)$, i.e.,

$\displaystyle\frac{\big{\|}\big{(}x^{(l)}[D_{\eta,\beta,v}x^{(0)}]-D_{\eta,\beta,v}x^{(l)}[x^{(0)}]\big{)}(\cdot,\alpha,\theta)\big{\|}_{L^{2}}}{\big{\|}D_{\eta,\beta,v}x^{(l)}[x^{(0)}](\cdot,\alpha,\theta)\big{\|}_{L^{2}}}$
(16)

We fix $L_{\theta}$, i.e., the number of the “inter-rotation” channels
corresponding to the “coarser” grid of $S^{1}$ for discrete $S^{1}$
integration, to $L_{\theta}=4$, and examine the equivariance error induced by
the “boundary leakage effect” with different numbers $L_{\alpha}$ of the
“inter-scale” channels [cf. Remark 4].

(a) Equivariance error, $K=5$.

(b) Equivariance error, $K=10$.

Figure 3: Layer-wise equivariance error (16) of the $\mathcal{RST}$-CNN.
Convolutional filters with varying number $L_{\alpha}$ of “inter-scale”
channels are constructed from $K$ low-frequency spatial modes
$\\{\psi_{k}\\}_{k=1}^{K}$. The error is smaller when $K$ decreases, verifying
our theoretical analysis Theorem 1.

Figure 3 displays the equivariance error (16) of the $\mathcal{RST}$-CNN at
different layers $l\in\\{1,\cdots,5\\}$ with varying
$L_{\alpha}\in\\{1,2,3\\}$. It can be observed that the equivariance error is
inevitable due to numerical discretization and truncation as the model goes
deeper. However, it can be mitigated by choosing a small $L_{\alpha}$, i.e.,
less “inter-scale” information transfer, to avoid the “boundary leakage
effect”, or expanding the filters with a small number $K$ of low-frequency
spatial components $\\{\psi_{k}\\}_{k=1}^{K}$, supporting our theoretical
analysis Theorem 1. Due to this finding, we will consider $\mathcal{RST}$-CNNs
with $L_{\alpha}=1$ in the following experiments, a practice that is adopted
also in [45, 39].

###  6.3 Image Classification

We next demonstrate the superior performance of the proposed
$\mathcal{RST}$-CNN in image classification under settings where a large
variation of rotation and scale is present in the test and/or the training
data.

####  6.3.1 RS-MNIST And RS-Fashion

| RS-MNIST test accuracy (%) | RS-MNIST+ test accuracy (%)
---|---|---
Models | $N_{\text{tr}}=2,000$ | $N_{\text{tr}}=5,000$ | $N_{\text{tr}}=2,000$ | $N_{\text{tr}}=5,000$
CNN | $80.63\pm 0.11$ | $87.41\pm 0.32$ | $89.52\pm 0.21$ | $93.08\pm 0.04$
SFCNN | $85.71\pm 0.18$ | $89.69\pm 0.40$ | $90.86\pm 0.20$ | $93.78\pm 0.35$
SFCNN+ | $87.96\pm 0.05$ | $92.29\pm 0.10$ | $92.20\pm 0.09$ | $95.32\pm 0.09$
RDCF | $86.79\pm 0.12$ | $90.46\pm 0.33$ | $90.40\pm 0.31$ | $94.84\pm 0.15$
RDCF+ | $87.68\pm 0.52$ | $91.74\pm 0.37$ | $92.49\pm 0.28$ | $95.81\pm 0.11$
SEVF | $85.76\pm 0.38$ | $90.29\pm 0.37$ | $89.97\pm 0.42$ | $93.47\pm 0.18$
SESN | $84.58\pm 0.29$ | $90.19\pm 0.39$ | $90.33\pm 0.30$ | $93.40\pm 0.29$
SDCF | $85.62\pm 0.51$ | $90.40\pm 0.09$ | $90.14\pm 0.16$ | $93.47\pm 0.05$
$\mathcal{RST}$-CNN(FB) | $89.16\pm 0.32$ | $93.19\pm 0.29$ | $92.58\pm 0.35$ | $96.33\pm 0.26$
$\mathcal{RST}$-CNN(SL) | $88.97\pm 0.17$ | $93.03\pm 0.20$ | $92.30\pm 0.19$ | $96.04\pm 0.11$
$\mathcal{RST}$-CNN+(FB) | $89.53\pm 0.27$ | $93.40\pm 0.26$ | $\textbf{93.99}\pm\textbf{0.07}$ | $96.53\pm 0.25$
$\mathcal{RST}$-CNN+(SL) | $\textbf{90.26}\pm\textbf{0.37}$ | $\textbf{93.59}\pm\textbf{0.06}$ | $93.82\pm 0.25$ | $\textbf{96.76}\pm\textbf{0.11}$

Table 1: Classification accuracy on the RS-MNIST data set. Models are trained
on $N_{\text{tr}}=$ 2K or 5K images with spatial resolution $56\times 56$. A
plus sign “+” on the data, i.e., RS-MNIST+, is used to denote the presence of
data augmentation during training. A plus sign “+” on the model, e.g., RDCF+,
denotes a larger network with more “inter-rotation” correlation $L_{\theta}=4$
[cf. Section 5]. The mean $\pm$ std of the test accuracy over five independent
trials are reported.

| RS-Fashion test accuracy (%) | RS-Fashion+ test accuracy (%)
---|---|---
Models | $N_{\text{tr}}=2,000$ | $N_{\text{tr}}=5,000$ | $N_{\text{tr}}=2,000$ | $N_{\text{tr}}=5,000$
CNN | $62.71\pm 0.37$ | $67.91\pm 0.28$ | $67.92\pm 0.12$ | $72.41\pm 0.46$
SFCNN | $70.80\pm 0.41$ | $75.80\pm 0.11$ | $74.10\pm 0.46$ | $77.76\pm 0.16$
SFCNN+ | $71.59\pm 0.71$ | $76.32\pm 0.26$ | $76.80\pm 0.55$ | $80.89\pm 0.41$
RDCF | $70.72\pm 0.10$ | $73.96\pm 0.19$ | $73.46\pm 0.10$ | $77.53\pm 0.11$
RDCF+ | $71.27\pm 0.34$ | $75.94\pm 0.35$ | $76.87\pm 0.57$ | $80.66\pm 0.48$
SEVF | $66.57\pm 0.32$ | $71.03\pm 0.31$ | $68.83\pm 0.52$ | $73.25\pm 0.22$
SESN | $66.28\pm 0.14$ | $72.19\pm 0.05$ | $69.43\pm 0.07$ | $75.85\pm 0.26$
SDCF | $66.29\pm 0.23$ | $72.24\pm 0.23$ | $68.40\pm 0.05$ | $75.11\pm 0.18$
$\mathcal{RST}$-CNN(FB) | $73.31\pm 0.16$ | $78.64\pm 0.60$ | $76.43\pm 0.59$ | $81.93\pm 0.04$
$\mathcal{RST}$-CNN(SL) | $72.90\pm 0.34$ | $78.37\pm 0.22$ | $76.06\pm 0.13$ | $80.81\pm 0.29$
$\mathcal{RST}$-CNN+(FB) | $74.37\pm 0.08$ | $79.19\pm 0.36$ | $\textbf{80.65}\pm\textbf{0.31}$ | $\textbf{84.37}\pm\textbf{0.19}$
$\mathcal{RST}$-CNN+(SL) | $\textbf{74.68}\pm\textbf{0.29}$ | $\textbf{79.81}\pm\textbf{0.06}$ | $80.65\pm 0.46$ | $84.09\pm 0.09$

Table 2: Classification accuracy on the RS-Fashion data set. Models are
trained on $N_{\text{tr}}=$ 2K or 5K images with spatial resolution $56\times
56$. A plus sign “+” on the data, i.e., RS-Fashion+, is used to denote the
presence of data augmentation during training. A plus sign “+” on the model,
e.g., RDCF+, denotes a larger network with more “inter-rotation” correlation
$L_{\theta}=4$ [cf. Section 5]. The mean $\pm$ std of the test accuracy over
five independent trials are reported.

We first benchmark the performance of different models on the RS-MNIST and RS-
Fashion data sets. We generate five independent realizations of the rotated
and rescaled data [cf. Secion 6.1], which are split into $N_{\text{tr}}=$
5,000 or 2,000 images for training, 2,000 images for validation, and 50,000
images for testing.

For fair comparison among different models, we use a benchmark CNN with three
convolutional and two fully-connected layers as a baseline. Each hidden layer
(i.e., the three convolutional and the first fully-connected layer) is
followed by a batch-normalization, and we set the number of output channels of
the hidden layers to $[32,63,95,256]$. The size of the convolutional filters
is set to $7\times 7$ for each layer in the CNN. All comparing models (those
in Table 1-3 without “+” after the name of the model) are built on the same
CNN baseline, and we keep the trainable parameters almost the same
($\sim$500K) by modifying the number of the unstructured channels. For models
that are equivariant to rotation (including SFCNN, RDCF, and
$\mathcal{RST}$-CNN), we set the number $N_{r}$ of rotation channels to
$N_{r}=8$ [cf. Section 5]; for scale-equivariant models (including SESN, SDCF,
and $\mathcal{RST}$-CNN), the number $N_{s}$ of scale channels is set to
$N_{s}=4$. In addition, for rotation-equivariant CNNs, we also construct
larger models (with the number of trainable parameters$\sim$1.6M) after
increasing the “inter-rotation” information transfer [cf. Section 5] by
setting $L_{\theta}=4$; we attach a “+” symbol to the end of the model name
(e.g., RDCF+) to denote such larger models with more “inter-rotation”. A group
max-pooling is performed only after the final convolutional layer to achieve
group-invariant representations for classification. Moreover, for
$\mathcal{RST}$-CNN, we consider two different spatial function basis for
filter expansion, namely the Fourier-Bessel (FB) basis, and Sturm-Liouville
(SL) basis [cf. Remark 1].

We use the Adam optimizer [22] to train all models for 60 epochs with the
batch size set to 128. We set the initial learning rate to 0.01, which is
scheduled to decrease tenfold after 30 epochs. We conduct the experiments in 4
different settings, where the number $N_{\text{tr}}$ of training samples is
either 2,000 or 5,000, and the models are trained with or without
$\mathcal{RST}$ data augmentation.

We report the mean $\pm$ std of the test accuracy after five independent
trials in Table 1 and Table 2, where, for example, RS-MNIST (or RS-MNIST+)
denotes models are trained on the RS-MNIST data set without (or with) data
augmentation. It is clear from Table 1 and Table 2 that $\mathcal{RST}$-CNN
has superior generalization capability compared to other models with
approximately the same trainable parameters, especially in the small data
regime. Furthermore, $\mathcal{RST}$-CNN+ with more inter-rotation correlation
(i.e., $L_{\theta}=4$) has further improved performance, achieving the best
accuracy among all methods. It can also be observed that neither of the two
spatial basis functions (i.e., FB and SL) has significant advantage over the
other. The preferred choice of the basis for filter expansion could depend on
experimental settings including sample size, the number of filters, original
data characteristics and the presence (or lack thereof) of data augmentation.

####  6.3.2 STL-10

Models | ID Accuracy (%) | OOD Accuracy (%)
---|---|---
ResNet-16 | $82.66\pm 0.53$ | $37.63\pm 1.95$
SFCNN | $83.84\pm 0.67$ | $51.28\pm 2.29$
RDCF | $83.66\pm 0.57$ | $51.12\pm 4.21$
SESN | $83.79\pm 0.24$ | $47.26\pm 0.63$
SDCF | $83.83\pm 0.41$ | $43.60\pm 0.87$
$\mathcal{RST}$-CNN | $\textbf{84.08}\pm\textbf{0.11}$ | $\textbf{58.31}\pm\textbf{3.62}$
Table 3: Test accuracy on the STL-10 data set for both in-distribution (ID)
and out-of-distribution (OOD) settings.

Finally, we use the STL-10 data set as an example to showcase both ID and OOD
generalization capacity of the proposed $\mathcal{RST}$-CNN. As explained in
Section 6.1, the training set remains unaltered, while the testing set is
either (a) unchanged for ID testing, or (b) randomly rotated and rescaled for
OOD testing. All models are built on a ResNet [18] with 16 layers as the
baseline, and we keep the number of trainable parameters almost the same for
comparing networks. A group max-pooling is performed after the final residual
block to achieve invariant representations.

Similar to the idea of [39, 51], the data set is augmented (without rotation
and rescaling) during training by randomly cropping a 12 pixel zero-padded
image. Furthermore, random horizontal flipping and Cutout [16] with 32 pixels
are applied to the cropped image. We train all models for 1000 epochs with a
batch size of 64, using an SGD optimizer with Nesterov momentum set to 0.9 and
weight decay set to $5\times 10^{-4}$. Learning rate starts at $0.1$, and is
scheduled to decrease tenfold after 300, 400, 600, and 800 epochs.

The mean $\pm$ std after three independent trials of both ID and OOD test
accuracy is displayed in Table 3. It can be observed that the proposed
$\mathcal{RST}$-CNN significantly outperforms all comparing models, especially
for OOD generalization, demonstrating further its advantage in computer vision
where both rotation and scale transformations are intrinsic symmetries of the
learning task.

##  Appendix A Proof of Proposition 1

###  A.1 Sufficient Part of the Proof of Proposition 1

###### Proof.

We first prove the sufficient part of Proposition 1. That is, given the layer-
wise definition of the $L$-layer feedforward neural network (6) and (7),
$\mathcal{RST}$-equivariance under the regular representation (2) (5) can be
achieved, i.e.,

$\displaystyle
x^{(l)}[D^{(l-1)}_{\eta,\beta,v}x^{(l-1)}]=D^{(l)}_{\eta,\beta,v}x^{(l)}[x^{(l-1)}],\quad\forall
l\geq 1.$ (17)

Indeed, for the first layer, i.e., $l=1$, we write the LHS of (17) as

$\displaystyle
x^{(1)}[D^{(0)}_{\eta,\beta,v}x^{(0)}](u,\theta,\alpha,\lambda)=\sigma\left(\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}2^{-2\alpha}D^{(0)}_{\eta,\beta,v}x^{(0)}(u+u^{\prime},\lambda^{\prime})W^{(1)}_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}u^{\prime})du^{\prime}+b^{(1)}(\lambda)\right)$
$\displaystyle=\sigma\left(\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}2^{-2\alpha}x^{(0)}\left(R_{-\eta}2^{-\beta}(u+u^{\prime}-v),\lambda^{\prime}\right)W^{(1)}_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}u^{\prime})du^{\prime}+b^{(1)}(\lambda)\right),$
(18)

The RHS of (17) when $l=1$ is

$\displaystyle
D_{\eta,\beta,v}^{(1)}x^{(1)}[x^{(0)}](u,\theta,\alpha,\lambda)=x^{(1)}[x^{(0)}]\left(R_{-\eta}2^{-\beta}(u-v),\theta-\eta,\alpha-\beta,\lambda\right)$
$\displaystyle=\sigma\left(\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}2^{-2(\alpha-\beta)}x^{(0)}\left(R_{-\eta}2^{-\beta}(u-v)+\tilde{u},\lambda^{\prime}\right)W^{(1)}_{\lambda^{\prime},\lambda}(2^{-(\alpha-\beta)}R_{\eta-\theta}\tilde{u})d\tilde{u}+b^{(1)}(\lambda)\right)$
$\displaystyle=\sigma\left(\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}2^{-2\alpha}2^{2\beta}x^{(0)}\left(R_{-\eta}2^{-\beta}(u-v)+2^{-\beta}R_{-\eta}u^{\prime},\lambda^{\prime}\right)W^{(1)}_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}u^{\prime})2^{-2\beta}du^{\prime}+b^{(1)}(\lambda)\right)$
(19)
$\displaystyle=\sigma\left(\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}2^{-2\alpha}x^{(0)}\left(R_{-\eta}2^{-\beta}(u+u^{\prime}-v),\lambda^{\prime}\right)W^{(1)}_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}u^{\prime})du^{\prime}+b^{(1)}(\lambda)\right),$
(20)

where (19) uses the change of variable
$u^{\prime}=2^{-\beta}R_{-\eta}\tilde{u}$. Thus (18) and (20) implies that
(17) holds valid for $l=1$.

For $l>1$, with the definition of the $l$-th layer operation (7), the LHS of
(17) becomes

$\displaystyle
x^{(l)}[D^{(l-1)}_{\eta,\beta,v}x^{(l-1)}](u,\theta,\alpha,\lambda)$
$\displaystyle=$ $\displaystyle\sigma\left(\sum_{\lambda^{\prime}}\int\int\int
2^{-2\alpha}D^{(l-1)}_{\eta,\beta,v}x^{(l-1)}\left(u+u^{\prime},\theta+\theta^{\prime},\alpha+\alpha^{\prime},\lambda^{\prime}\right)W^{(l)}_{\lambda^{\prime},\lambda}\left(2^{-\alpha}R_{-\theta}u^{\prime},\theta^{\prime},\alpha^{\prime}\right)d\alpha^{\prime}d\theta^{\prime}du^{\prime}+b^{(l)}(\lambda)\right)$
(21) $\displaystyle=$
$\displaystyle\sigma\bigg{(}\sum_{\lambda^{\prime}}\int\int\int
2^{-2\alpha}x^{(l-1)}\left(R_{-\eta}2^{-\beta}(u+u^{\prime}-v),\theta+\theta^{\prime}-\eta,\alpha+\alpha^{\prime}-\beta,\lambda^{\prime}\right)$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\cdot
W^{(l)}_{\lambda^{\prime},\lambda}\left(2^{-\alpha}R_{-\theta}u^{\prime},\theta^{\prime},\alpha^{\prime}\right)d\alpha^{\prime}d\theta^{\prime}du^{\prime}+b^{(l)}(\lambda)\bigg{)}.$
(22)

On the other hand, the RHS of (17) is

$\displaystyle
D^{(l)}_{\eta,\beta,v}x^{(l)}[x^{(l-1)}](u,\theta,\alpha,\lambda)$
$\displaystyle=$ $\displaystyle
x^{(l)}[x^{(l-1)}]\left(R_{-\eta}2^{-\beta}(u-v),\theta-\eta,\alpha-\beta,\lambda\right)$
(23) $\displaystyle=$
$\displaystyle\sigma\Bigg{(}\sum_{\lambda^{\prime}}\int\int\int
2^{-2(\alpha-\beta)}x^{(l-1)}\left(R_{-\eta}2^{-\beta}(u-v)+\tilde{u},\theta-\eta+\theta^{\prime},\alpha-\beta+\alpha^{\prime},\lambda^{\prime}\right)$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\cdot
W^{(l)}_{\lambda^{\prime}\lambda}\left(2^{-(\alpha-\beta)}R_{\eta-\theta}\tilde{u},\theta^{\prime},\alpha^{\prime}\right)d\alpha^{\prime}d\theta^{\prime}d\tilde{u}+b^{(l)}(\lambda)\Bigg{)}$
(24) $\displaystyle=$
$\displaystyle\sigma\Bigg{(}\sum_{\lambda^{\prime}}\int\int\int
2^{-2\alpha}2^{2\beta}x^{(l-1)}\left(R_{-\eta}2^{-\beta}(u-v)+R_{-\eta}2^{-\beta}u^{\prime},\theta-\eta+\theta^{\prime},\alpha-\beta+\alpha^{\prime},\lambda^{\prime}\right)$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\cdot
W^{(l)}_{\lambda^{\prime},\lambda}\left(2^{-\alpha}R_{-\theta}u^{\prime},\theta^{\prime},\alpha^{\prime}\right)2^{-2\beta}d\alpha^{\prime}d\theta^{\prime}d\tilde{u}+b^{(l)}(\lambda)\Bigg{)},$
(25)

where the last equation again uses the same change of variable. Equation (22)
combined with (25) implies that (17) holds true for all $l>1$. ∎

###  A.2 Necessary Part of the Proof of Proposition 1

###### Proof.

We first note that a general feedforward neural network propagating through
the feature spaces $\mathcal{X}^{(l)}$ has the following form: when $l=1$,

$\displaystyle
x^{(1)}[x^{(0)}](u,\theta,\alpha,\lambda)=\sigma\bigg{(}\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}x^{(0)}(u+u^{\prime})W^{(1)}(u^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)du^{\prime}+b^{(1)}(\lambda)\bigg{)},$
(26)

and, for $l>1$,

$\displaystyle
x^{(l)}[x^{(l-1)}](u,\theta,\alpha,\lambda)=\sigma\bigg{(}\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}\int_{S^{1}}\int_{\mathbb{R}}x^{(l-1)}(u+u^{\prime},\theta+\theta^{\prime},\alpha+\alpha^{\prime},\lambda^{\prime})$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\cdot
W^{(l)}(u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)d\alpha^{\prime}d\theta^{\prime}du^{\prime}+b^{(l)}(\lambda)\bigg{)}.$
(27)

To prove the necessary part of Proposition 1, we want to verify that
$\mathcal{RST}$-equivariance (17) implies that the weight matrices $W^{(l)}$
in (26) and (27) take the special convolutional form in (6) and (7).

Indeed, when $l=1$, the RHS of (17) under the operation (26) is

$\displaystyle
D^{(1)}_{\eta,\beta,v}x^{(1)}[x^{(0)}](u,\theta,\alpha,\lambda)$ (28)
$\displaystyle=$ $\displaystyle
x^{(1)}[x^{(0)}]\left(R_{-\eta}2^{-\beta}(u-v),\theta-\eta,\alpha-\beta,\lambda\right)$
(29) $\displaystyle=$ $\displaystyle\sigma\left(\sum_{\lambda^{\prime}}\int
x^{(0)}(R_{-\eta}2^{-\beta}(u-v)+\tilde{u},\lambda^{\prime})W^{(1)}(\tilde{u},\lambda^{\prime},R_{-\eta}2^{-\beta}(u-v),\theta-\eta,\alpha-\beta,\lambda)d\tilde{u}+b^{(1)}(\lambda)\right)$
(30) $\displaystyle=$ $\displaystyle\sigma\left(\sum_{\lambda^{\prime}}\int
2^{-2\beta}x^{(0)}(R_{-\eta}2^{-\beta}(u+u^{\prime}-v),\lambda^{\prime})\right.$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\cdot
W^{(1)}\left(R_{-\eta}2^{-\beta}u^{\prime},\lambda^{\prime},R_{-\eta}2^{-\beta}(u-v),\theta-\eta,\alpha-\beta,\lambda\right)du^{\prime}+b^{(1)}(\lambda)\bigg{)},$
(31)

with change of variable similar to the sufficient part. For the RHS, we have

$\displaystyle
x^{(1)}[D^{(0)}_{\eta,\beta,v}x^{(0)]}](u,\theta,\alpha,\lambda)$ (32)
$\displaystyle=$ $\displaystyle\sigma\left(\sum_{\lambda^{\prime}}\int
D^{(1)}_{\eta,\beta,v}x^{(0)}(u+u^{\prime},\lambda^{\prime})W^{(1)}(u^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)du^{\prime}+b^{(1)}(\lambda)\right)$
(33) $\displaystyle=$ $\displaystyle\sigma\left(\sum_{\lambda^{\prime}}\int
x^{(0)}(R_{-\eta}2^{-\beta}(u+u^{\prime}-v),\lambda^{\prime})W^{(1)}(u^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)du^{\prime}+b^{(1)}(\lambda)\right)$
(34)

Hence, for (17) to hold when $l=1$, we need

$\displaystyle
W^{(1)}(u^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)=2^{-2\beta}W^{(1)}(R_{-\eta}2^{-\beta}u^{\prime},\lambda^{\prime},R_{-\eta}2^{-\beta}(u-v),\theta-\eta,\alpha-\beta,\lambda),$
(35)

For all $u,\theta,\alpha,\lambda,u^{\prime},\lambda^{\prime},v,\eta,\beta$.
Keeping $u,\theta,\alpha,\lambda,u^{\prime},\lambda^{\prime},\eta,\beta$ fixed
while varying $v$ in (35), we deduce that
$W^{(1)}(u^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)$ does not depend
on the third variable $u$. Hence
$W^{(1)}(u^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)=W^{(1)}(u^{\prime},\lambda^{\prime},0,\theta,\alpha,\lambda),\forall
u\in\mathbb{R}^{2}$. Further define
$W_{\lambda,\lambda^{\prime}}^{(1)}(u^{\prime})$ as

$\displaystyle
W_{\lambda,\lambda^{\prime}}^{(1)}(u^{\prime})=W^{(1)}(u^{\prime},\lambda^{\prime},0,0,0,\lambda).$
(36)

Therefore, for any fixed
$u^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda$, setting $\beta=\alpha$,
$\eta=\theta$ in (35) yields

$\displaystyle
W^{(1)}(u^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)=W^{(1)}(R_{-\theta}2^{-\alpha}u^{\prime},\lambda^{\prime},R_{-\theta}2^{-\alpha}(u-v),\theta-\theta,\alpha-\alpha,\lambda)2^{-2\alpha}$
(37)
$\displaystyle=W^{(1)}(R_{-\theta}2^{-\alpha}u^{\prime},\lambda^{\prime},0,0,0,\lambda)2^{-2\alpha}=W^{(1)}_{\lambda,\lambda^{\prime}}(R_{-\theta}2^{-\alpha}u^{\prime})2^{-2\alpha}$
(38)

Hence (26) has the special form (6).

For the subsequent layers $l>1$, a similar argument yields

$\displaystyle
W^{(l)}(u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)=W^{(l)}(R_{-\eta}2^{-\beta}u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},R_{-\eta}2^{-\beta}(u-v),\theta-\eta,\alpha-\beta,\lambda)2^{-2\beta},$
(39)

for all
$u,\theta,\alpha,\lambda,u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},v,\eta,\beta$.
Similarly, we can keep
$u,\theta,\alpha,\lambda,u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},\eta,\beta$
fixed while varying $v$ in (39), which implies that
$W^{(l)}(u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)$
does not depend on the fifth variable $u$. Again, let us define

$\displaystyle
W_{\lambda,\lambda^{\prime}}^{(l)}(u^{\prime},\theta^{\prime},\alpha^{\prime})=W^{(l)}(u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},0,0,0,\lambda).$
(40)

For any given
$u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda$,
setting $\beta=\alpha$, $\eta=\theta$ leads us to

$\displaystyle
W^{(l)}(u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},u,\theta,\alpha,\lambda)=W^{(l)}(R_{-\theta}2^{-\alpha}u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},R_{-\theta}2^{-\alpha}(u-v),0,0,\lambda)2^{-2\alpha}$
$\displaystyle=$ $\displaystyle
W^{(l)}(R_{-\theta}2^{-\alpha}u^{\prime},\theta^{\prime},\alpha^{\prime},\lambda^{\prime},0,0,0,\lambda)2^{-2\alpha}=W^{(l)}_{\lambda^{\prime},\lambda}(2^{-\alpha}u^{\prime},\theta^{\prime}-\theta,\alpha^{\prime}-\alpha),$
(41)

which implies that (27) can be written in the form of (7). This concludes the
proof of Proposition 1. ∎

##  Appendix B Proof of Theorem 1

The proof of Theorem 1 follows the similar steps in [51]. More specifically,
we aim to establish the following two propositions that show the layer-wise
non-expansiveness of the model (Proposition 2) and quantify the perturbation
of equivariance with the presence of layer-wise spatial deformation $D_{\tau}$
(Proposition 3).

######  Proposition 2.

Under (A1) and (A2), an $\mathcal{RST}$-CNN satisfies:

  1. (a)

For any $l\geq 1$, the $l$-th layer mapping $x^{(l)}[\cdot]$ defined in (7) is
non-expansive, i.e.,

$\displaystyle\|x^{(l)}[x_{1}]-x^{(l)}[x_{2}]\|\leq\|x_{1}-x_{2}\|,\quad\forall
x_{1},x_{2}.$ (42)

  2. (b)

Let $x_{0}^{(l)}(u,\theta,\alpha,\lambda)$ be the $l$-th layer output given a
zero input $x^{(0)}(u,\lambda)=0$, then $x_{0}^{(l)}(u,\theta,\alpha,\lambda)$
depends only on $\lambda$, i.e.,
$x^{(l)}_{0}(u,\theta,\alpha,\lambda)=x_{0}^{(l)}(\lambda)$.

  3. (c)

Let $x_{c}^{(l)}$ be the centered version of $x^{(l)}$ after subtracting
$x_{0}^{(l)}$, i.e.,

$\displaystyle x_{c}^{(0)}(u,\lambda)\coloneqq
x^{(0)}(u,\lambda)-x_{0}^{(0)}(\lambda)=x^{(0)}(u,\lambda),~{}x_{c}^{(l)}(u,\theta,\alpha,\lambda)\coloneqq
x^{(l)}(u,\theta,\alpha,\lambda)-x_{0}^{(l)}(\lambda),~{}l\geq 1,$ (43)

then $\|x_{c}^{(l)}\|\leq\|x_{c}^{(l-1)}\|,~{}\forall l\geq 1$. As a result,
$\|x_{c}^{(l)}\|\leq\|x_{c}^{(0)}\|=\|x^{(0)}\|$.

######  Proposition 3.

In an $\mathcal{RST}$-CNN satisfying (A1) to (A3), the following statements
hold true.

  1. (a)

Given any $l\geq 1$,

$\displaystyle\left\|x^{(l)}[D_{\tau}x^{(l-1)}]-D_{\tau}x^{(l)}[x^{(l-1)}]\right\|\leq
8|\nabla\tau|_{\infty}\left\|x_{c}^{(l-1)}\right\|,$ (44)

where $x_{c}^{(l-1)}$ is defined in (43)

  2. (b)

Given any $l\geq 1$, we have

$\displaystyle\left\|D^{(l)}_{\eta,\beta,v}x^{(l)}\right\|=2^{\beta}\left\|x^{(l)}\right\|,$
(45)

and

$\displaystyle\left\|x^{(l)}[D_{\eta,\beta,v}^{(l-1)}\circ
D_{\tau}x^{(l-1)}]-D^{(l)}_{\eta,\beta,v}D_{\tau}x^{(l)}[x^{(l-1}]\right\|\leq
2^{\beta+3}|\nabla\tau|_{\infty}\left\|x_{c}^{(l-1)}\right\|.$ (46)

  3. (c)

For any $l\geq 1$,

$\displaystyle\left\|x^{(l)}[D_{\eta,\beta,v}^{(0)}\circ
D_{\tau}x^{(l-1)}]-D^{(l)}_{\eta,\beta,v}D_{\tau}x^{(l)}[x^{(0}]\right\|\leq
2^{\beta+3}l|\nabla\tau|_{\infty}\left\|x_{c}^{(0)}\right\|.$ (47)

  4. (d)

For any $l\geq 1$,

$\displaystyle\left\|D^{(l)}_{\eta,\beta,v}D_{\tau}x^{(l)}-D^{(l)}_{\eta,\beta,v}x^{(l)}\right\|\leq
2^{\beta+1-j_{l}}|\tau|_{\infty}\left\|x_{c}^{(l-1)}\right\|\leq
2^{\beta+1-j_{l}}|\tau|_{\infty}\left\|x^{(0)}\right\|.$ (48)

###  B.1 Proof of Proposition 2

Before proving Proposition 2, we present the following two lemmas that are
crucial to bound various norms of the convolutional filters using their
Fourier-Bessle (FB) norm

######  Lemma 1.

Let $\\{\psi_{k}\\}_{k}$ be the Fourier-Bessel basis on the unit disk
$D\subset\mathbb{R}^{2}$, and let $\\{\varphi_{m}\\}_{m}$ be the Fourier basis
on the unit circle $S^{1}$. Assume that

$\displaystyle F(u)=\sum_{k}a(k)\psi_{k}(u),\quad
G(u,\theta)=\sum_{k}\sum_{m}b(k,m)\psi_{k}(u)\varphi_{m}(\theta)$ (49)

are functions in $H^{1}_{0}(2^{j}D)$ and $H^{1}_{0}(2^{j}D)\times
L^{2}(S^{1})$, respectively. Then

$\displaystyle\quad\quad\quad\quad\quad\int|F(u)|du,~{}~{}\int|u||\nabla
F(u)|du,~{}~{}\int|\nabla
F(u)|du\leq\pi\|a\|_{\text{FB}}=\pi\left(\sum_{k}\mu_{k}a(k)^{2}\right)^{1/2},$
(50)
$\displaystyle\iint|G(u,\theta)|dud\theta,~{}~{}\iint|u||\nabla_{u}G(u,\theta)|dud\theta,~{}~{}\iint|\nabla_{u}G(u)|dud\theta\leq\pi\|b\|_{\text{FB}}=\pi\left(\sum_{k,m}\mu_{k}b(k,m)^{2}\right)^{1/2}.$
(51)

The proof of Lemma 1 can be found in Proposition A.1 of [9]. A direct
application of Lemma 1 leads to the following lemma.

######  Lemma 2.

Let $a_{\lambda^{\prime},\lambda}^{(l)}(k,m,n)$ be the coefficients of the
filter $W_{\lambda^{\prime},\lambda}^{(l)}(u,\theta,\alpha)$ (supported on
$2^{j_{l}}D\times S^{1}\times[-1,1]$) under the separable bases
$\left\\{\psi_{j_{l},k}(u)\right\\}_{k}$,
$\left\\{\varphi_{m}(\theta)\right\\}_{m}$ and
$\left\\{\xi_{n}(\alpha)\right\\}_{n}$ defined in the main paper, and define
$W_{\lambda^{\prime},\lambda,n}^{(l)}(u,\theta)$ as

$\displaystyle
W^{(l)}_{\lambda^{\prime},\lambda,n}(u,\theta)\coloneqq\sum_{k}\sum_{m}a_{\lambda^{\prime},\lambda}^{(l)}(k,m,n)\psi_{j_{l},k}(u)\varphi_{m}(\theta).$
(52)

We have

$\displaystyle
B^{(1)}_{\lambda^{\prime},\lambda},C^{(1)}_{\lambda^{\prime},\lambda},2^{j_{1}}D^{(1)}_{\lambda^{\prime},\lambda}\leq\pi\|a^{(1)}_{\lambda^{\prime},\lambda}\|_{\text{FB}},~{}~{}B^{(l)}_{\lambda^{\prime},\lambda,n},C^{(l)}_{\lambda^{\prime},\lambda,n},2^{j_{l}}D^{(l)}_{\lambda^{\prime},\lambda,n}\leq\pi\|a^{(l)}_{\lambda^{\prime},\lambda}(\cdot,n)\|_{\text{FB}},~{}\forall
l>1,$ (53)

where

$\displaystyle\left\\{\begin{aligned}
&B_{\lambda^{\prime},\lambda}^{(1)}\coloneqq\int\left|W_{\lambda^{\prime},\lambda}^{(1)}(u)\right|du,\quad&&B_{\lambda^{\prime},\lambda,n}^{(l)}\coloneqq\int_{S^{1}}\int_{\mathbb{R}^{2}}\left|W_{\lambda^{\prime},\lambda,n}^{(l)}(u,\theta)\right|dud\theta,\quad
l>1,\\\
&C_{\lambda^{\prime},\lambda}^{(1)}\coloneqq\int\left|u\right|\left|\nabla_{u}W_{\lambda^{\prime},\lambda}^{(1)}(u)\right|du,\quad&&C_{\lambda^{\prime},\lambda,n}^{(l)}\coloneqq\int_{S^{1}}\int_{\mathbb{R}^{2}}\left|u\right|\left|\nabla_{u}W_{\lambda^{\prime},\lambda,n}^{(l)}(u,\theta)\right|dud\theta,\quad
l>1,\\\
&D_{\lambda^{\prime},\lambda}^{(1)}\coloneqq\int\left|\nabla_{u}W_{\lambda^{\prime},\lambda}^{(1)}(u)\right|du,\quad&&D_{\lambda^{\prime},\lambda,n}^{(l)}\coloneqq\int_{S^{1}}\int_{\mathbb{R}^{2}}\left|\nabla_{u}W_{\lambda^{\prime},\lambda,n}^{(l)}(u,\theta)\right|dud\theta,\quad
l>1.\\\ \end{aligned}\right.$ (54)

Hence we have

$\displaystyle B_{l},C_{l},2^{j_{l}}D_{l}\leq A_{l},$ (55)

where

$\displaystyle\begin{aligned}
&B_{1}\coloneqq\max\left\\{\sup_{\lambda}\sum_{\lambda^{\prime}=1}^{M_{0}}B_{\lambda^{\prime},\lambda}^{(1)},~{}\frac{M_{0}}{M_{1}}\sup_{\lambda^{\prime}}\sum_{\lambda=1}^{M_{1}}B_{\lambda^{\prime},\lambda}^{(1)}\right\\},\\\
&C_{1}\coloneqq\max\left\\{\sup_{\lambda}\sum_{\lambda^{\prime}=1}^{M_{0}}C_{\lambda^{\prime},\lambda}^{(1)},~{}\frac{M_{0}}{M_{1}}\sup_{\lambda^{\prime}}\sum_{\lambda=1}^{M_{1}}C_{\lambda^{\prime},\lambda}^{(1)}\right\\},\\\
&D_{1}\coloneqq\max\left\\{\sup_{\lambda}\sum_{\lambda^{\prime}=1}^{M_{0}}D_{\lambda^{\prime},\lambda}^{(1)},~{}\frac{M_{0}}{M_{1}}\sup_{\lambda^{\prime}}\sum_{\lambda=1}^{M_{1}}D_{\lambda^{\prime},\lambda}^{(1)}\right\\},\end{aligned}$
(56)

and, for $l>1$,

$\displaystyle\begin{aligned}
&B_{l}\coloneqq\max\left\\{\sup_{\lambda}\sum_{\lambda^{\prime}=1}^{M_{l-1}}\sum_{n}B_{\lambda^{\prime},\lambda,n}^{(l)},~{}\frac{2M_{l-1}}{M_{l}}\sum_{n}B_{l,n}\right\\},\quad
B_{l,n}\coloneqq\sup_{\lambda^{\prime}}\sum_{\lambda=1}^{M_{l}}B_{\lambda^{\prime},\lambda,n}^{(l)},\\\
&C_{l}\coloneqq\max\left\\{\sup_{\lambda}\sum_{\lambda^{\prime}=1}^{M_{l-1}}\sum_{n}C_{\lambda^{\prime},\lambda,n}^{(l)},~{}\frac{2M_{l-1}}{M_{l}}\sum_{n}C_{l,n}\right\\},\quad
C_{l,n}\coloneqq\sup_{\lambda^{\prime}}\sum_{\lambda=1}^{M_{l}}C_{\lambda^{\prime},\lambda,n}^{(l)},\\\
&D_{l}\coloneqq\max\left\\{\sup_{\lambda}\sum_{\lambda^{\prime}=1}^{M_{l-1}}\sum_{n}D_{\lambda^{\prime},\lambda,n}^{(l)},~{}\frac{2M_{l-1}}{M_{l}}\sum_{n}D_{l,n}\right\\},\quad
D_{l,n}\coloneqq\sup_{\lambda^{\prime}}\sum_{\lambda=1}^{M_{l}}D_{\lambda^{\prime},\lambda,n}^{(l)}.\end{aligned}$
(57)

The bound on $A_{l},\forall l\geq 1$, i.e., (A2), therefore implies that

$\displaystyle B_{l},C_{l},2^{j_{l}}D_{l}\leq 1,\quad\forall l\geq 1.$ (58)

###### Proof of Lemma 2.

Applying Lemma 1 to the filters $W_{\lambda^{\prime},\lambda}^{(1)}(u)$ and
$W^{(l)}_{\lambda^{\prime},\lambda,n}(u,\theta)$ defined in (52) after
rescaling the spatial variable $u$ easily leads to the desired bounds (55).
The rest of the lemma follows from the assumption (A2). ∎

###### Proof of Proposition 2.

To avoid cumbersome notations, we drop the layer index $(l)$ in the filters
$W^{(l)}_{\lambda^{\prime},\lambda}$ and $b^{(l)}$, and let
$M=M_{l},M^{\prime}=M_{l-1}$ when the context is clear. The proof of (a) is
similar to that of Proposition 2(a) in [51] after a further integration on
$S^{1}$. More specifically, when $l=1$, the definition of $B_{1}$ in (54)
implies that

$\displaystyle\sup_{\lambda}\sum_{\lambda^{\prime}}B_{\lambda^{\prime},\lambda}^{(1)}\leq
B_{1},\quad\sup_{\lambda^{\prime}}\sum_{\lambda}B_{\lambda^{\prime},\lambda}^{(1)}\leq
B_{1}\frac{M}{M^{\prime}}$ (59)

Therefore, given two inputs $x_{1}$ and $x_{2}$, we have

$\displaystyle\left|\left(x^{(1)}[x_{1}]-x^{(1)}[x_{2}]\right)(u,\theta,\alpha,\lambda)\right|^{2}$
(60) $\displaystyle=$
$\displaystyle\left|\sigma\left(\sum_{\lambda^{\prime}}\int
x_{1}(u+u^{\prime},\lambda^{\prime})W_{\lambda^{\prime},\lambda}\left(2^{-\alpha}R_{-\theta}u^{\prime}\right)2^{-2\alpha}du^{\prime}+b(\lambda)\right)\right.$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\left.-\sigma\left(\sum_{\lambda^{\prime}}\int
x_{2}(u+u^{\prime},\lambda^{\prime})W_{\lambda^{\prime},\lambda}\left(2^{-\alpha}R_{-\theta}u^{\prime}\right)2^{-2\alpha}du^{\prime}+b(\lambda)\right)\right|^{2}$
(61) $\displaystyle\leq$ $\displaystyle\left|\sum_{\lambda^{\prime}}\int
x_{1}(u+u^{\prime},\lambda^{\prime})W_{\lambda^{\prime},\lambda}\left(2^{-\alpha}R_{-\theta}u^{\prime}\right)2^{-2\alpha}du^{\prime}-\sum_{\lambda^{\prime}}\int
x_{2}(u+u^{\prime},\lambda^{\prime})W_{\lambda^{\prime},\lambda}\left(2^{-\alpha}R_{-\theta}u^{\prime}\right)2^{-2\alpha}du^{\prime}\right|^{2}$
(62) $\displaystyle=$
$\displaystyle\left|\sum_{\lambda^{\prime}}\int(x_{1}-x_{2})(u+u^{\prime},\lambda^{\prime})W_{\lambda^{\prime},\lambda}\left(2^{-\alpha}R_{-\theta}u^{\prime}\right)2^{-2\alpha}du^{\prime}\right|^{2}$
(63) $\displaystyle\leq$
$\displaystyle\left(\sum_{\lambda^{\prime}}\int\left|(x_{1}-x_{2})(u+u^{\prime},\lambda^{\prime})\right|^{2}\left|W_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}u^{\prime})\right|2^{-2\alpha}du^{\prime}\right)\sum_{\lambda^{\prime}}\int\left|W_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}u^{\prime})\right|2^{-2\alpha}du^{\prime}$
(64) $\displaystyle=$
$\displaystyle\left(\sum_{\lambda^{\prime}}\int\left|(x_{1}-x_{2})(u+u^{\prime},\lambda^{\prime})\right|^{2}\left|W_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}u^{\prime})\right|2^{-2\alpha}du^{\prime}\right)\left(\sum_{\lambda^{\prime}}B_{\lambda^{\prime},\lambda}^{(1)}\right)$
(65) $\displaystyle\leq$ $\displaystyle
B_{1}\sum_{\lambda^{\prime}}\int\left|(x_{1}-x_{2})(\tilde{u},\lambda^{\prime})\right|^{2}\left|W_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}\left(\tilde{u}-u)\right)\right|2^{-2\alpha}d\tilde{u}$
(66)

Hence, given any $\alpha\in\mathbb{R}$, we have

$\displaystyle\sum_{\lambda}\int_{S^{1}}\int_{\mathbb{R}^{2}}\left|\left(x^{(1)}[x_{1}]-x^{(1)}[x_{2}]\right)(u,\alpha,\theta,\lambda)\right|^{2}dud\theta$
$\displaystyle\leq$
$\displaystyle\sum_{\lambda}\int_{S^{1}}\int_{\mathbb{R}^{2}}B_{1}\sum_{\lambda^{\prime}}\int\left|(x_{1}-x_{2})(\tilde{u},\lambda^{\prime})\right|^{2}\left|W_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}\left(\tilde{u}-u)\right)\right|2^{-2\alpha}d\tilde{u}dud\theta$
(67) $\displaystyle=$ $\displaystyle
B_{1}\sum_{\lambda^{\prime}}\int_{\mathbb{R}^{2}}\left|(x_{1}-x_{2})(\tilde{u},\lambda^{\prime})\right|^{2}\left(\sum_{\lambda}\int_{S^{1}}\int_{\mathbb{R}^{2}}\left|W_{\lambda^{\prime},\lambda}(2^{-\alpha}R_{-\theta}\left(\tilde{u}-u)\right)\right|2^{-2\alpha}dud\theta\right)d\tilde{u}$
(68) $\displaystyle=$ $\displaystyle
B_{1}\sum_{\lambda^{\prime}}\int\left|(x_{1}-x_{2})(\tilde{u},\lambda^{\prime})\right|^{2}\left(\sum_{\lambda}B_{\lambda^{\prime},\lambda}^{(1)}\right)d\tilde{u}$
(69) $\displaystyle\leq$ $\displaystyle
B_{1}^{2}\frac{M}{M^{\prime}}\sum_{\lambda^{\prime}}\int\left|(x_{1}-x_{2})(\tilde{u},\lambda^{\prime})\right|^{2}d\tilde{u}$
(70) $\displaystyle=$ $\displaystyle B_{1}^{2}M\|x_{1}-x_{2}\|^{2}$ (71)
$\displaystyle\leq$ $\displaystyle M\|x_{1}-x_{2}\|^{2},$ (72)

where the last inequality comes from Lemma 2, and (68) makes use of the fact
that

$\displaystyle\int_{\mathbb{R}^{2}}\left|W(2^{-\alpha}R_{-\theta}u)\right|2^{-2\alpha}du=\int_{\mathbb{R}^{2}}|W(u)|du,\quad\forall\alpha\in\mathbb{R},~{}\forall\theta\in
S^{1},$ (73)

and $\int_{S^{1}}d\theta=1$ due to the normalization factor $1/2\pi$ in the
definition. Therefore, we have

$\displaystyle\|x^{(1)}[x_{1}]-x^{(1)}[x_{2}]\|^{2}$
$\displaystyle=\sup_{\alpha}\frac{1}{M}\sum_{\lambda}\iint\left|\left(x^{(1)}[x_{1}]-x^{(1)}[x_{2}]\right)(u,\theta,\alpha,\lambda)\right|^{2}dud\theta\leq\|x_{1}-x_{2}\|^{2}.$
(74)

This concludes the proof of (a) for the case $l=1$. For the case $l>1$, the
same technique applies by considering the joint convolution
$\int_{S^{1}}\int_{\mathbb{R}^{2}}(\cdots)dud\theta$ while making use of (73),
and we omit the detail.

For part (b), we use mathematical induction. More specifically,
$x_{0}^{(0)}(u,\lambda)=0$ by definition. For $l=1$,
$x_{0}^{(1)}(u,\theta,\alpha,\lambda)=\sigma(b^{(1)}(\lambda))$. Assuming that
$x_{0}^{(l-1)}(u,\theta,\alpha,\lambda)=x_{0}^{(l-1)}(\lambda)$ for some
$l>1$, we have

$\displaystyle x_{0}^{(l)}(u,\alpha,\lambda)$ $\displaystyle=$
$\displaystyle\sigma\left(\sum_{\lambda^{\prime}}\int_{S^{1}}\int_{\mathbb{R}^{2}}\int_{\mathbb{R}}x_{0}^{(l-1)}(u+u^{\prime},\theta+\theta^{\prime},\alpha+\alpha^{\prime},\lambda^{\prime})W_{\lambda^{\prime},\lambda}^{(l)}\left(2^{-\alpha}R_{-\theta}u^{\prime},\theta^{\prime},\alpha^{\prime}\right)2^{-2\alpha}d\alpha^{\prime}du^{\prime}d\theta^{\prime}+b^{(l)}(\lambda)\right)$
(75) $\displaystyle=$
$\displaystyle\sigma\left(\sum_{\lambda^{\prime}}x_{0}^{(l-1)}(\lambda^{\prime})\int_{S^{1}}\int_{\mathbb{R}^{2}}\int_{\mathbb{R}}W_{\lambda^{\prime},\lambda}^{(l)}\left(2^{-\alpha}R_{-\theta}u^{\prime},\theta^{\prime},\alpha^{\prime}\right)2^{-2\alpha}d\alpha^{\prime}du^{\prime}d\theta^{\prime}+b^{(l)}(\lambda)\right)$
(76) $\displaystyle=$
$\displaystyle\sigma\left(\sum_{\lambda^{\prime}}x_{0}^{(l-1)}(\lambda^{\prime})\int_{S^{1}}\int_{\mathbb{R}^{2}}\int_{\mathbb{R}}W_{\lambda^{\prime},\lambda}^{(l)}\left(u^{\prime},\theta^{\prime},\alpha^{\prime}\right)d\alpha^{\prime}du^{\prime}d\theta^{\prime}+b^{(l)}(\lambda)\right)$
(77) $\displaystyle=$ $\displaystyle x_{0}^{(l)}(\lambda).$ (78)

To prove part (c): for any $l>1$, we have

$\displaystyle\|x_{c}^{(l)}\|=\|x^{(l)}-x_{0}^{(l)}\|=\|x^{(l)}[x^{(l-1)}]-x_{0}^{(l)}[x_{0}^{(l-1)}]\|\leq\|x^{(l-1)}-x_{0}^{(l-1)}\|=\|x_{c}^{(l-1)}\|,$
(79)

where the inequality comes from the layer-wise non-expansiveness (42) in part
(a). An easy induction leads to (c). ∎

###  B.2 Proof of Proposition 3

###### Proof.

Just like Proposition 2(a), the proofs of part (a) and part (d), respectively,
of Proposition 3 are similar to those of Proposition 3(a) and Proposition 4 of
[51]. More specifically, making use of (73) and $\int_{S^{1}}d\theta=1$ when
$l=1$, and further taking the integral
$\int_{S^{1}}\int_{\mathbb{R}^{2}}(\cdots)dud\theta$ over
$S^{1}\times\mathbb{R}^{2}$ instead of just $\mathbb{R}^{2}$ when $l>1$, we
arrive at the following

$\displaystyle\left\|x^{(l)}[D_{\tau}x^{(l-1)}]-D_{\tau}x^{(l)}[x^{(l-1)}]\right\|\leq
4(B_{l}+C_{l})|\nabla\tau|_{\infty}\left\|x_{c}^{(l-1)}\right\|\leq
4(B_{l}+C_{l})|\nabla\tau|_{\infty}\left\|x^{(0)}\right\|,\quad\forall l\geq
1,$ (80)
$\displaystyle\left\|D^{(l)}_{\eta,\beta,v}D_{\tau}x^{(l)}-D^{(l)}_{\eta,\beta,v}x^{(l)}\right\|\leq
2^{\beta+1}|\tau|_{\infty}D_{l}\left\|x_{c}^{(l-1)}\right\|\leq
2^{\beta+1}|\tau|_{\infty}D_{l}\left\|x^{(0)}\right\|,\quad\forall l\geq 1,$
(81)

where $B_{l},C_{l},D_{l}$ are defined in (56) and (57). Lemma 2 thus implies
that

$\displaystyle\left\|x^{(l)}[D_{\tau}x^{(l-1)}]-D_{\tau}x^{(l)}[x^{(l-1)}]\right\|\leq
8|\nabla\tau|_{\infty}\left\|x_{c}^{(l-1)}\right\|\leq
8|\nabla\tau|_{\infty}\left\|x^{(0)}\right\|,\quad\forall l\geq 1,$ (82)
$\displaystyle\left\|D^{(l)}_{\eta,\beta,v}D_{\tau}x^{(l)}-D^{(l)}_{\eta,\beta,v}x^{(l)}\right\|\leq
2^{\beta+1-j_{l}}|\tau|_{\infty}\left\|x_{c}^{(l-1)}\right\|\leq
2^{\beta+1-j_{l}}|\tau|_{\infty}\left\|x^{(0)}\right\|,\quad\forall l\geq 1.$
(83)

For part (b), given any $(\eta,\beta,v)\in
S^{1}\times\mathbb{R}\times\mathbb{R}^{2}$, we have

$\displaystyle\|D^{(l)}_{\eta,\beta,v}x^{(l)}\|^{2}$
$\displaystyle=\sup_{\alpha}\frac{1}{M_{l}}\sum_{\lambda}\int_{S^{1}}\int_{\mathbb{R}^{2}}\left|D^{(l)}_{\eta,\beta,v}x^{(l)}(u,\theta,\alpha,\lambda)\right|^{2}dud\theta$
(84)
$\displaystyle=\sup_{\alpha}\frac{1}{M_{l}}\sum_{\lambda}\int_{S^{1}}\int_{\mathbb{R}^{2}}\left|x^{(l)}(2^{-\beta}R_{-\theta}(u-v),\theta-\eta,\alpha-\beta,\lambda)\right|^{2}dud\theta$
(85)
$\displaystyle=\sup_{\alpha}\frac{1}{M_{l}}\sum_{\lambda}\int_{S^{1}}\int_{\mathbb{R}^{2}}\left|x^{(l)}(u^{\prime},\theta-\eta,\alpha-\beta,\lambda)\right|^{2}2^{2\beta}du^{\prime}d\theta$
(86) $\displaystyle=2^{2\beta}\|x^{(l)}\|^{2}$ (87)

Thus (45) is valid. The second half of part (b) holds since

$\displaystyle\left\|x^{(l)}[D^{(l)}_{\eta,\beta,v}\circ
D_{\tau}x^{(l-1)}]-D^{(l)}_{\eta,\beta,v}D_{\tau}x^{(l)}[x^{(l-1)}]\right\|$
$\displaystyle=$
$\displaystyle\left\|D^{(l)}_{\eta,\beta,v}x^{(l)}[D_{\tau}x^{(l-1)}]-D^{(l)}_{\eta,\beta,v}D_{\tau}x^{(l)}[x^{(l-1)}]\right\|$
(88) $\displaystyle=$ $\displaystyle
2^{\beta}\left\|x^{(l)}[D_{\tau}x^{(l-1)}]-D_{\tau}x^{(l)}[x^{(l-1)}]\right\|$
(89) $\displaystyle\leq$ $\displaystyle
2^{\beta+3}|\nabla\tau|_{\infty}\|x_{c}^{(l-1)}\|,$ (90)

where the first equality holds because of the $\mathcal{RST}$-equivariance,
i.e., Theorem 1, and the second equality follows from (45).

The proof of part (c) is exactly the same as that of Proposition 3(c) of [51].
Specifically, we telescope the inequality (46) while making use of the non-
expansiveness of the layer-wise features, i.e., Proposition 2(c). The detail
is omitted. ∎

###  B.3 Proof of Theorem 1

###### Proof.

Theorem 1 is a direct consequence of Proposition 3. More specifically,

$\displaystyle\left\|x^{(L)}[D^{(0)}_{\eta,\beta,v}\circ
D_{\tau}x^{(0)}]-D^{(L)}_{\eta,\beta,v}x^{(L)}[x^{(0)}]\right\|$
$\displaystyle\leq$ $\displaystyle\left\|x^{(L)}[D^{(0)}_{\eta,\beta,v}\circ
D_{\tau}x^{(0)}]-D^{(L)}_{\eta,\beta,v}D_{\tau}x^{(L)}[x^{(0)}]\right\|+\left\|D^{(L)}_{\eta,\beta,v}D_{\tau}x^{(L)}[x^{(0)}]-D^{(L)}_{\eta,\beta,v}x^{(L)}[x^{(0)}]\right\|$
(91) $\displaystyle\leq$ $\displaystyle
2^{\beta+3}L|\nabla\tau|_{\infty}\|x^{(0)}\|+2^{\beta+1-j_{L}}|\tau|_{\infty}\|x^{(0)}\|$
(92) $\displaystyle=$ $\displaystyle
2^{\beta+1}\left(4L|\nabla\tau|_{\infty}+2^{-j_{L}}|\tau|_{\infty}\right)\|x^{(0)}\|,$
(93)

where the second inequality comes from Proposition 3(c) and Proposition 3(d).
This concludes the proof of Theorem 1. ∎

## References

  * [1] M. Abramowitz and I. A. Stegun.  Handbook of mathematical functions: with formulas, graphs, and mathematical tables, volume 55.  Courier Corporation, 1965.
  * [2] B. Anderson, T. S. Hy, and R. Kondor.  Cormorant: Covariant molecular neural networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [3] E. J. Bekkers.  B-spline cnns on lie groups.  In International Conference on Learning Representations, 2020.
  * [4] A. Bietti and J. Mairal.  Invariance and stability of deep convolutional representations.  In NIPS 2017-31st Conference on Advances in Neural Information Processing Systems, pages 1622–1632, 2017.
  * [5] A. Bietti and J. Mairal.  Group invariance, stability to deformations, and complexity of deep convolutional representations.  The Journal of Machine Learning Research, 20(1):876–924, 2019.
  * [6] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al.  End to end learning for self-driving cars.  arXiv preprint arXiv:1604.07316, 2016.
  * [7] J. Bruna and S. Mallat.  Invariant scattering convolution networks.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1872–1886, 2013.
  * [8] H. Chen, S. Liu, W. Chen, H. Li, and R. Hill.  Equivariant point network for 3d point cloud analysis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14514–14523, 2021.
  * [9] X. Cheng, Q. Qiu, R. Calderbank, and G. Sapiro.  RotDCF: Decomposition of convolutional filters for rotation-equivariant deep networks.  In International Conference on Learning Representations, 2019.
  * [10] A. Coates, A. Ng, and H. Lee.  An analysis of single-layer networks in unsupervised feature learning.  In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223. JMLR Workshop and Conference Proceedings, 2011.
  * [11] T. Cohen and M. Welling.  Group equivariant convolutional networks.  In International conference on machine learning, pages 2990–2999, 2016.
  * [12] T. Cohen and M. Welling.  Steerable CNNs.  In International Conference on Learning Representations, 2017.
  * [13] T. S. Cohen, M. Geiger, J. Köhler, and M. Welling.  Spherical CNNs.  In International Conference on Learning Representations, 2018.
  * [14] T. S. Cohen, M. Geiger, and M. Weiler.  A general theory of equivariant cnns on homogeneous spaces.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [15] M. Defferrard, M. Milani, F. Gusset, and N. Perraudin.  Deepsphere: a graph-based spherical cnn.  In International Conference on Learning Representations, 2020.
  * [16] T. DeVries and G. W. Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [17] M. Finzi, S. Stanton, P. Izmailov, and A. G. Wilson.  Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data.  In International Conference on Machine Learning, pages 3165–3176. PMLR, 2020.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [19] E. Hoogeboom, J. W. Peters, T. S. Cohen, and M. Welling.  Hexaconv.  In International Conference on Learning Representations, 2018.
  * [20] A. Kanazawa, A. Sharma, and D. Jacobs.  Locally scale-invariant convolutional neural networks.  arXiv preprint arXiv:1412.5104, 2014.
  * [21] N. Keriven and G. Peyré.  Universal invariant and equivariant graph neural networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [22] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] R. Kondor.  N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials.  arXiv preprint arXiv:1803.01588, 2018.
  * [24] R. Kondor, Z. Lin, and S. Trivedi.  Clebsch–gordan nets: a fully fourier space spherical convolutional neural network.  Advances in Neural Information Processing Systems, 31:10117–10126, 2018.
  * [25] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [26] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [27] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [28] J. Mairal.  End-to-end kernel learning with supervised convolutional kernel networks.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [29] J. Mairal, P. Koniusz, Z. Harchaoui, and C. Schmid.  Convolutional kernel networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
  * [30] S. Mallat.  Recursive interferometric representation.  In Proc. of EUSICO conference, Danemark, 2010.
  * [31] S. Mallat.  Group invariant scattering.  Communications on Pure and Applied Mathematics, 65(10):1331–1398, 2012.
  * [32] D. Marcos, B. Kellenberger, S. Lobry, and D. Tuia.  Scale equivariance in CNNs with vector fields.  arXiv preprint arXiv:1807.11783, 2018.
  * [33] D. Marcos, M. Volpi, N. Komodakis, and D. Tuia.  Rotation equivariant vector field networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5048–5057, 2017.
  * [34] E. Oyallon and S. Mallat.  Deep roto-translation scattering for object classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2865–2873, 2015.
  * [35] Q. Qiu, X. Cheng, R. Calderbank, and G. Sapiro.  DCFNet: Deep neural network with decomposed convolutional filters.  In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4198–4207, Stockholmsmässan, Stockholm Sweden, 10–15 Jul 2018. PMLR.
  * [36] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 91–99. Curran Associates, Inc., 2015.
  * [37] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241, Cham, 2015. Springer International Publishing.
  * [38] L. Sifre and S. Mallat.  Rotation, scaling and deformation invariant scattering for texture discrimination.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1233–1240, 2013.
  * [39] I. Sosnovik, M. Szmaja, and A. Smeulders.  Scale-equivariant steerable networks.  In International Conference on Learning Representations, 2020.
  * [40] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley.  Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.  arXiv preprint arXiv:1802.08219, 2018.
  * [41] M. Weiler and G. Cesa.  General e(2)-equivariant steerable cnns.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [42] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. Cohen.  3d steerable cnns: Learning rotationally equivariant features in volumetric data.  In Advances in Neural Information Processing Systems, pages 10381–10392, 2018.
  * [43] M. Weiler, F. A. Hamprecht, and M. Storath.  Learning steerable filters for rotation equivariant cnns.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 849–858, 2018.
  * [44] D. Worrall and G. Brostow.  Cubenet: Equivariance to 3D rotation and translation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 567–584, 2018.
  * [45] D. Worrall and M. Welling.  Deep scale-spaces: Equivariance over scale.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [46] D. E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. J. Brostow.  Harmonic networks: Deep translation and rotation equivariance.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5028–5037, 2017.
  * [47] H. Xiao, K. Rasul, and R. Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
  * [48] Y. Xu, T. Xiao, J. Zhang, K. Yang, and Z. Zhang.  Scale-invariant convolutional neural networks.  arXiv preprint arXiv:1411.6369, 2014.
  * [49] Y. Zhao, T. Birdal, J. E. Lenssen, E. Menegatti, L. Guibas, and F. Tombari.  Quaternion equivariant capsule networks for 3d point clouds.  In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision – ECCV 2020, pages 1–19, Cham, 2020. Springer International Publishing.
  * [50] Y. Zhou, Q. Ye, Q. Qiu, and J. Jiao.  Oriented response networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 519–528, 2017.
  * [51] W. Zhu, Q. Qiu, R. Calderbank, G. Sapiro, and X. Cheng.  Scale-equivariant neural networks with decomposed convolutional filters.  arXiv preprint arXiv:1909.11193, 2019.