# Equivariance-aware Architectural Optimization of Neural Networks

Kaitlin Maile
IRIT, University of Toulouse
<EMAIL_ADDRESS>
&Dennis G. Wilson
ISAE-SUPAERO, University of Toulouse
<EMAIL_ADDRESS>
Patrick Forré
University of Amsterdam
<EMAIL_ADDRESS>
###### Abstract

Incorporating equivariance to symmetry groups as a constraint during neural
network training can improve performance and generalization for tasks
exhibiting those symmetries, but such symmetries are often not perfectly nor
explicitly present. This motivates algorithmically optimizing the
architectural constraints imposed by equivariance. We propose the equivariance
relaxation morphism, which preserves functionality while reparameterizing a
group equivariant layer to operate with equivariance constraints on a
subgroup, as well as the $[G]$-mixed equivariant layer, which mixes layers
constrained to different groups to enable within-layer equivariance
optimization. We further present evolutionary and differentiable neural
architecture search (NAS) algorithms that utilize these mechanisms
respectively for equivariance-aware architectural optimization. Experiments
across a variety of datasets show the benefit of dynamically constrained
equivariance to find effective architectures with approximate equivariance.

##  1 Introduction

Constraining neural networks to be equivariant to symmetry groups present in
the data can improve their task performance, efficiency, and generalization
capabilities (Bronstein et al., 2021), as shown by translation-equivariant
convolutional neural networks (Fukushima & Miyake, 1982; LeCun et al., 1989)
for image-based tasks (LeCun et al., 1998). Seminal works have developed
general theories and architectures for equivariance in neural networks,
providing a blueprint for equivariant operations on complex structured data
(Cohen & Welling, 2016; Ravanbakhsh et al., 2017; Kondor & Trivedi, 2018;
Weiler et al., 2021). However, these works design model constraints based on
an explicit equivariance property. Furthermore, their architectural assumption
of full equivariance in every layer may be overly constraining; e.g., in
handwritten digit recognition, full equivariance to $180^{\circ}$ rotation may
lead to misclassifying samples of “$6$” and “$9$”. Weiler & Cesa (2019) found
that local equivariance from a final subgroup convolutional layer improves
performance over full equivariance. If appropriate equivariance constraints
are instead learned, the benefits of equivariance could extend to applications
where the data may have unknown or imperfect symmetries.

Learning approximate equivariance has been recently approached through novel
layer operations (Wang et al., 2022; Finzi et al., 2021; Zhou et al., 2020;
Yeh et al., 2022; Basu et al., 2021). Separately, the field of neural
architecture search (NAS) aims to optimize full neural network architectures
(Zoph & Le, 2017; Real et al., 2017; Elsken et al., 2017; Liu et al., 2018; Lu
et al., 2019). Existing NAS methods have not yet been developed for explicitly
optimizing equivariance, although partial or soft equivariant approaches like
Romero & Lohit (2022) and van der Ouderaa et al. (2022) do allow for custom
equivariant architectures. An important aspect of NAS is network morphisms:
function-preserving architectural changes (Wei et al., 2016) which can be used
during training to change the loss landscape and gradient descent trajectory
while immediately maintaining the current functionality and loss value (Maile
et al., 2022). Developing tools for searching over a space of architectural
representations of equivariance would allow for existing NAS algorithms to be
applied towards architectural optimization of equivariance.

#### Contributions

First, we present two mechanisms towards equivariance-aware architectural
optimization. The equivariance relaxation morphism for group convolutional
layers partially expands the representation and parameters of the layer to
enable less constrained learning with a prior on symmetry. The $[G]$-mixed
equivariant layer parameterizes a layer as a weighted sum of layers
equivariant to different groups, permitting the learning of architectural
weighting parameters.

Second, we implement these concepts within two algorithms for architectural
optimization of partially-equivariant networks. Evolutionary Equivariance-
Aware NAS (EquiNASE) utilizes the equivariance relaxation morphism in a greedy
evolutionary algorithm, dynamically relaxing constraints throughout the
training process. Differentiable Equivariance-Aware NAS (EquiNASD) implements
$[G]$-mixed equivariant layers throughout a network to learn the appropriate
approximate equivariance of each layer, in addition to their optimized
weights, during training.

Finally, we analyze the proposed mechanisms via their respective NAS
approaches in multiple image classification tasks, investigating how the
dynamically learned approximate equivariance affects training and performance
over baseline models and other approaches.

###  1.1 Related works

#### Approximate equivariance

Although no other works on approximate equivariance explicitly study
architectural optimization, some approaches are architectural in nature. We
compare our contributions with the most conceptually similar works to our
knowledge.

The main contributions of Basu et al. (2021) and Agrawal & Ostrowski (2022)
are similar to our proposed equivariant relaxation morphism. Basu et al.
(2021) also utilizes subgroup decomposition but instead algorithmically builds
up equivariances from smaller groups, while our work focuses on relaxing
existing constraints. Agrawal & Ostrowski (2022) presents theoretical
contributions towards network morphisms for group-invariant shallow neural
networks: in comparison, our work focuses on deep group convolutional
architectures and implements the morphism in a NAS algorithm.

The main contributions of Wang et al. (2022) and Finzi et al. (2021) are
similar to our proposed $[G]$-mixed equivariant layer. Wang et al. (2022) also
uses a weighted sum of kernels, but uses the same group for each kernel and
defines the weights over the domain of group elements. Finzi et al. (2021)
uses an equivariant layer in parallel to a linear layer with weighted
regularization, thus only using two layers in parallel and weighting them
through regularization rather than parameterization.

In more diverse approaches, Zhou et al. (2020) and Yeh et al. (2022) represent
symmetry-inducing weight sharing through learnable matrices. Romero & Lohit
(2022) and van der Ouderaa et al. (2022) learn partial or soft equivariances
for each layer.

#### Neural architecture search

Neural architecture search (NAS) aims to optimize both the architecture and
its parameters for a given task. Liu et al. (2018) approaches this difficult
bi-level optimization by creating a large super-network containing all
possible elements and continuously relaxing the discrete architectural
parameters to enable search by gradient descent. Other NAS approaches include
evolutionary algorithms (Real et al., 2017; Lu et al., 2019; Elsken et al.,
2017) and reinforcement learning (Zoph & Le, 2017), which search over
discretely represented architectures.

##  2 Background

We assume familiarity with group theory (see Appendix A.1). Let $G$ be a
discrete group. The $l$th $G$-equivariant group convolutional layer (Cohen &
Welling, 2016) of a group convolutional neural network (G-CNN) convolves the
feature map $f\colon G\to\mathbb{R}^{C_{l-1}}$ output from the previous layer
with a filter with kernel size $k$ represented as learnable parameters
$\psi\colon G\to\mathbb{R}^{C_{l}\times C_{l-1}}$. For each output channel
$d\in[C_{l}]$, where $[C]:=\\{1,\dots,C\\}$, and group element $g\in G$, the
layer’s output is defined via the convolution operator111We identify the
correlation and convolution operators as they only differ where the inverse
group element is placed and refer to both as ”convolution” throughout this
work.:

$\displaystyle[f\star_{G}\psi]_{d}(g)$ $\displaystyle=\sum_{h\in
G}\sum_{c=1}^{C_{l-1}}f_{c}(h)\psi_{d,c}(g^{-1}h).$ (1)

The first layer is a special case: the input to the network needs to be lifted
via this operation such that the output feature map of this layer has a domain
of $G$. In the case of image data, an image $x$ with $C$ channels may be
interpreted as a function $x\colon\mathbb{Z}^{2}\to\mathbb{R}^{C}$ mapping
each pixel in coordinate space to a real number for each channel, where the
$c$th channel of $x$ is referred to as $x_{c}$. The input is
$x\colon\mathbb{Z}^{2}\to\mathbb{R}^{C_{0}}$, so the layer is instead a
lifting convolution:

$\displaystyle[x\star_{G}\psi]_{d}(g)$
$\displaystyle=\sum_{y\in\mathbb{Z}^{2}}\sum_{c=1}^{C_{0}}x_{c}(y)\psi_{d,c}(g^{-1}y).$
(2)

We present our contributions in the group convolutional layer case, although
similar claims apply for the lifting convolutional layer case.

##  3 Towards Architectural Optimization over Subgroups

We propose two mechanisms to enable search over subgroups: the equivariance
relaxation morphism and a $[G]$-mixed equivariant layer, shown in Figure 1.
The proposed morphism, described in Section 3.1, changes the equivariance
constraint from one group to another subgroup while preserving the learned
weights of the initial group convolutional operator. The $[G]$-mixed
equivariant layer, presented in Section 3.2, allows for a single layer to
represent equivariance to multiple subgroups through a weighted sum.

Figure 1: Visualizations of (A) the equivariance relaxation morphism and (B)
the $[G]$-mixed equivariant layer, using the $C_{4}$ group. In (A), the
learnable parameters of a $C_{4}$-equivariant convolutional layer are expanded
using the group element actions, such that the expanded filter can be used in
a standard convolutional layer. Applying the equivariance relaxation morphism
reparameterizes the layer to only be architecturally constrained to $C_{2}$
equivariance, initialized to be functionally $C_{4}$ equivariant. In (B),
convolutional operations equivariant to subgroups of $C_{4}$ are summed with
learnable architectural weighting parameters.

###  3.1 Equivariance Relaxation Morphism

The equivariance relaxation morphism reparameterizes a $G$-equivariant group
(or lifting) convolutional layer to operate over a subgroup of $G$, partially
removing weight-sharing constraints from the parameter space while maintaining
the functionality of the layer.

Let $G^{\prime}\leq G$ be a subgroup of $G$. Let $R$ be a system of
representatives of the left quotient (including the neutral element), so that
$G^{\prime}\setminus G=\left\\{G^{\prime}r\mid r\in R\right\\},$ where
$G^{\prime}r\coloneqq\left\\{g^{\prime}r\mid g^{\prime}\in
G^{\prime}\right\\}.$ Given a $G$-equivariant group convolutional layer with
feature map $f$ and kernel $\psi$, we define the relaxed feature map
$\tilde{f}\colon G^{\prime}\to\mathbb{R}^{(C_{l-1}\times|R|)}$ and relaxed
kernel $\tilde{\psi}\colon
G^{\prime}\to\mathbb{R}^{(C_{l}\times|R|)\times(C_{l-1}\times|R|)}$ as
follows. For $c\in[C_{l-1}]$, $s,t\in R$, $d\in[C_{l}]$:

$\displaystyle\tilde{f}_{(c,s)}(g^{\prime})$ $\displaystyle\coloneqq
f_{c}(g^{\prime}s),$ (3) $\displaystyle\tilde{\psi}_{(d,t),(c,s)}(g^{\prime})$
$\displaystyle\coloneqq\psi_{d,c}(t^{-1}g^{\prime}s).$ (4)

We define the equivariance relaxation morphism from $G$ to $G^{\prime}$ as the
reparameterization of $\psi$ as $\tilde{\psi}$ (Eq. 4) and reshaping of $f$ as
$\tilde{f}$ (Eq. 3). We will show that the new output layer,
$[\tilde{f}\star_{G^{\prime}}\tilde{\psi}]_{(d,t)}(g^{\prime})$, is equivalent
to $[f\star_{G}\psi]_{d}(g^{\prime}t)$ down to reshaping. Since the mapping
$G^{\prime}\times R\to G,\,(g^{\prime},t)\mapsto g^{\prime}t$, is bijective,
every $g$ can uniquely be written as $g=g^{\prime}t$ with $g^{\prime}\in
G^{\prime}$ and $t\in R$. For $g\in G$, $G^{\prime}g\in G^{\prime}\setminus G$
has a unique representative $t\in R$ with $G^{\prime}g=G^{\prime}t$, and
$g^{\prime}:=gt^{-1}\in G^{\prime}$. By the same argument, $h\in G$ may be
written as $h=h^{\prime}s$ with unique $h\in G^{\prime}$ and $s\in R$. With
these preliminaries, we get:

$\displaystyle[f\star_{G}\psi]_{d}(g^{\prime}t)$
$\displaystyle=[f\star_{G}\psi]_{d}(g)$ (5) $\displaystyle=\sum_{h\in
G}\sum_{c=1}^{C_{l-1}}f_{c}(h)\psi_{d,c}(g^{-1}h),$ (6)
$\displaystyle=\sum_{h^{\prime}\in G^{\prime}}\sum_{s\in
R}\sum_{c=1}^{C_{l-1}}f_{c}(h^{\prime}s)\psi_{d,c}(t^{-1}g^{\prime-1}h^{\prime}s),$
(7) $\displaystyle=\sum_{h^{\prime}\in
G^{\prime}}\sum_{c=1}^{C_{l-1}}\sum_{s\in
R}\tilde{f}_{(c,s)}(h^{\prime})\tilde{\psi}_{(d,t),(c,s)}(g^{\prime-1}h^{\prime}),$
(8)
$\displaystyle=\left[\tilde{f}\star_{G^{\prime}}\tilde{\psi}\right]_{(d,t)}(g^{\prime}),$
(9)

which shows the claim. Thus, the convolution of $\tilde{f}$ with
$\tilde{\psi}$ is equivariant to $G$ but parametrized as a
$G^{\prime}$-equivariant group convolutional layer, where the representatives
are expanded into independent channels. This morphism can be viewed as
initializing a $G^{\prime}$-equivariant layer with a pre-trained prior of
equivariance to $G$, maintaining any previous training.

Standard convolutional layers are a special case of group-equivariant layers,
where the group is translational symmetry over pixel space. Regular group
convolutions are often implemented by relaxation to the translational symmetry
group by expanding the kernel via the appropriate group actions, allowing a
standard convolution implementation from a deep learning library to be used.
The equivariance relaxation morphism generalizes this concept to any subgroup.
With the given preliminaries and the case of $G^{\prime}=T(2)$, $\tilde{f}$
and $\tilde{\psi}$ are computed such that
$\tilde{f}_{(c,s)}(g^{\prime})\coloneqq f_{c}(g^{\prime}s)$ and
$\tilde{\psi}_{(d,t),(c,s)}(g^{\prime})\coloneqq\psi_{d,c}(t^{-1}g^{\prime}s)$
for each $g^{\prime}\in T(2)$, $c\in[C_{l-1}]$, $s,t\in R$, and $d\in[C_{l}].$

Let $S_{G}:=|R|$. The learnable parameters of the $G_{l}$-equivariant $l$th
layer with $C_{l}$ output channels, corresponding to $\psi,$ are stored as a
tensor of size $C_{l}\times C_{l-1}\times S_{G_{l}}\times K_{l}\times K_{l}$.
The kernel transformation expands this kernel tensor by performing the action
of each $r\in R$ on another copy of the tensor to expand its shape along a new
dimension, resulting in a tensor of size $C_{l}\times S_{G_{l}}\times
C_{l-1}\times S_{G_{l}f}\times K_{l}\times K_{l}$, which is reshaped to
$C_{l}S_{G_{l}}\times C_{l-1}S_{G_{l}}\times K_{l}\times K_{l}$. The input
tensor to the $l$th layer, corresponding to $f,$ is in the shape of $B\times
C_{l-1}\times S_{G_{l}}\times H_{l-1}\times W_{l-1},$ which is reshaped to
$B\times C_{l-1}S_{G_{l}}\times H_{l-1}\times W_{l-1}$ and convolved with the
expanded kernel. The output of shape $B\times C_{l}S_{G_{l}}\times H_{l}\times
W_{l}$ is reshaped to $B\times C_{l}\times S_{G_{l}}\times H_{l}\times W_{l}.$

To implement the equivariance relaxation morphism, the new kernel tensor is
initialized by applying Equation 4 such that result of applying the preceding
kernel transformation is equivalent. Our implementation of group actions
relies on group channel indexing to represent the order of group elements: to
ensure this is consistent before and after the morphism, the appropriate
reordering of the output and input channels of the expanded filter are applied
upon expansion. The new kernel tensor has a shape of $C_{l}|R|\times
C_{l-1}|R|\times S_{G_{l}}/|R|\times K_{l}\times K_{l}$.

###  3.2 $[G]$-Mixed Equivariant Layer

Towards learning equivariance, we additionally propose partial equivariance
through a mixture of layers, each constrained to equivariance to different
groups and applied in parallel to the same input then all combined via a
weighted sum. The equivariance relaxation morphism provides a mapping of group
elements between pairs of groups where one is a subgroup of the other. For a
set of groups $[G]$ where each group is a subgroup or supergroup of all other
groups within the set, we define a $[G]$-mixed equivariant layer as:

$\displaystyle\left[f\hat{\star}_{[G]}[\psi]\right]_{(d,t)}(g)$
$\displaystyle=\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}}\right]_{(d,t)}(g)$
(10)
$\displaystyle=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G}}\right]_{(d,t)}(g),$
(11)

where each element $z_{G}$ of $[z]\coloneqq\left\\{z_{G}|G\in[G]\right\\}$ is
an architectural weighting parameter such that $\sum_{G\in[G]}z_{G}=1,$
$G^{\prime}$ is a subgroup of all groups in $[G]$, each element $\psi^{G}$ of
$[\psi]$ is a kernel with a domain of $G$, and $\widetilde{\psi^{G}}$ is the
transformation of $\psi^{G}$ from a domain of $G$ to $G^{\prime}$ as defined
in Equation 4. Thus, the layer is parametrized by $[\psi]$ and $[z]$,
computing a weighted sum of operations that are equivariant to different
groups of $[G]$. The layer may be equivalently computed by convolution of the
input with the weighted sum of transformed kernels, shown in Equation 11. The
implementation for the $[G]$-mixed equivariant layer can be built on top of
that of the equivariance relaxation morphism, also using proper input and
output channel reordering between layers to ensure correct mixing of group
channels.

##  4 Equivariance-Aware Neural Architecture Algorithms

We present two neural architecture search methods that utilize the presented
mechanisms for discovering appropriate equivariance during neural network
training: Evolutionary Equivariance-Aware NAS (EquiNASE) and Differentiable
Equivariance-Aware NAS (EquiNASD). Both methods optimize an architecture while
learning network weights, returning a final trained network adapted to the
equivariances present in the training data. However, they differ in NAS
paradigm and approximate equivariance representation: EqufiNASE, described in
Section 4.1, searches for networks composed of layers each fully equivariant
to possibly different groups, while EquiNASD, described in Section 4.2,
searches for smooth mixtures of equivariant layers.

###  4.1 Evolutionary Equivariance-aware NAS

Algorithm 1 Evolutionary equivariance-aware neural architecture search.

procedure EquiNASE(Initial symmetry group $G$)

Initialize population with a $G$-equivariant group convolutional network.

for each generation do

for each network in population do

Add children of network with relaxed equivariance constraints into population.

for each network in population do

Partially train network on dataset.

Select Pareto-efficient and high accuracy networks as new population. return
population

Towards finding the optimal full equivariance per layer, the equivariance
relaxation morphism presented in Section 3.1 is applied as the genetic
operator in an evolutionary hill-climbing algorithm. The Evolutionary
Equivariance-Aware NAS (EquiNASE) algorithm, given in Algorithm 1, is similar
to other evolutionary NAS methods such as Elsken et al. (2017) with pareto
selection as in Falanti et al. (2022). A population of networks, which starts
with an individual with all layers equivariant to the largest possible group,
undergoes mutation via equivariance relaxation and selection based on accuracy
and parameter count to optimize neural architecture while learning network
parameters. See Appendix A.2 for further background on evolutionary NAS.

In each generation, candidate networks are evaluated based on maximizing
validation accuracy and minimizing parameter count: the entire Pareto front is
kept, then additional high-accuracy individuals are added if necessary until
the desired parent population size is reached. Offspring are generated from
each parent separately through mutation using the relaxation morphism. This
preserves the weights of the parameterized equivariance during mutation,
allowing for the continuous training of networks over evolution through
inheritance from parent individuals. Specifically, mutation reduces a single
layer’s parameterized equivariance to a subgroup within the constraint that
each layer has parametrized equivariance to a subgroup of all preceding
layers. This constraint yields local equivariance properties for the network,
as shown in Weiler & Cesa (2019) and Elsayed et al. (2020) to be empirically
favorable in image classification tasks. The resulting individuals are each
trained independently for a given training time, and then this process
repeats.

The second objective of minimizing parameter count is intended to advance
efficient networks, such as those with large symmetry groups. Accuracy-based
selection alone would necessarily prefer larger networks as mutation via the
equivariance relaxation morphism results in two networks with identical
performance but different size, the relaxed network having more parameters,
until training; potentially short-term increases in validation accuracy after
training would then result in the selection of individuals with more
parameters. Thus, the proposed strategy of selecting both pareto-front and
high-accuracy individuals is intended to maintain a diverse yet efficient
population without succumbing to overly greedy selections too early.

###  4.2 Differentiable Equivariance-aware NAS

Algorithm 2 Differentiable equivariance-aware neural architecture search.

procedure EquiNASD(Set of groups $[G]$)

Initialize network with $[G]$-mixed equivariant layers, parameterized by
$\Psi$ and $Z$.

while not converged do

Update $Z$ by $\nabla_{Z}\mathcal{L}(\Psi,Z).$

Update $\Psi$ by $\nabla_{\Psi}\mathcal{L}(\Psi,Z).$ return trained network

In a contrasting paradigm, the $[G]$-mixed equivariant layer presented in
Section 3.2 allows for smoothly searching across a spectrum of equivariance
for each layer via a differentiable NAS algorithm. Our Differentiable
Equivariance-Aware NAS (EquiNASD) algorithm, defined in Algorithm 2, is
inspired by DARTS (Liu et al., 2018) with significant changes detailed in the
following paragraphs. EquiNASD simplifies the bilevel optimization of the
architecture weighting parameters $Z$ and kernel weights $\Psi$ into
alternating independent updates, computing the gradient update for $Z$ with
the current, rather than optimal, $\Psi$ for the current architecture encoded
by $Z$, to boost search efficiency with minimal performance loss compared to
higher order approximations (Liu et al., 2018).

In most differentiable NAS search spaces, the desired output architecture is
discretized to select a subset of architectural options within constraints,
then the weights are re-initialized and trained within the static
architecture. In our formulation, this is not necessary as any mixed operation
can be equivalently expressed as a single layer equivariant to any group
$G^{\prime}$ that is a common subgroup to all groups of the mixed operation
(Eq. 11): in our experimental case, this is a standard translation-equivariant
convolutional layer, so the final model can be equivalently expressed as a
standard convolutional model with encoded partial equivariance. Thus, the
final optimized architecture and trained weights are output from the single
search process. We explore the standard NAS paradigm, where only the
architecture is output and reused for evaluation such that the weights are
retrained in this static architecture, in further experiments.

In order to enforce that the scaling of each kernel does not confound the
architecture weighting parameters, we use the weight normalizing
reparameterization (Salimans & Kingma, 2016) and do not update the scalar norm
parameter of each kernel after initialization.

We do not use disjoint datasets for updating $\Psi$ and $Z$, but rather draw
one batch for $\Psi$ and another for $Z$ independently and randomly from the
same training split. This allows for a standard dataset split and to use the
validation set for hyperparameter tuning.

These two NAS approaches present adaptations of two standard types of NAS,
evolutionary and differentiable, to the search for optimal partial
equivariance. The equivariance relaxation morphism and the $[G]$-mixed
equivariant layer that enable the evolutionary and differentiable search
methods respectively are the main focus; the other characteristics of the NAS
methods are adapted from existing methods, but further study could advance
specialization of equivariance-aware NAS. We next study empirically the two
EquiNAS methods on three datasets, one with known rotational symmetry and two
with unknown but visually significant rotational and reflectional symmetry.

##  5 Experiments

We focus on the regular representation of groups and show experiments with
reflectional and up to 4-fold rotational symmetry groups applied to image
classification tasks. Examples of symmetry groups acting on pixel space, which
corresponds to $\mathbb{Z}^{2}$, include $T(2)$, which consists of discrete
translations in both dimensions; the cyclical groups $C_{n}$, which consist of
$n$-fold rotations; and the dihedral groups $D_{n}$, which consist of
reflections with $n$-fold rotations, where $n\in\\{1,2,4\\}$ for exact
symmetry without interpolation. The $p4$ group consists of discrete
translations and multiples of $90^{\circ}$ rotations and may be represented as
$T(2)\rtimes C_{4}.$ The $p4m$ group consists of discrete translations,
reflections, and multiples of $90^{\circ}$ rotations and may be represented as
$T(2)\rtimes D_{4}.$ As standard convolutional layers are already equivariant
to $T(2),$ we refer to layers also equivariant to $n$-fold rotations with or
without reflections as $D_{n}$ or $C_{n}$-equivariant, respectively. So, a
$C_{1}$ equivariant convolutional layer is a standard translation-equivariant
convolutional layer. We use $\\{C_{1},D_{1},C_{2},D_{2},C_{4},D_{4}\\}$ as the
set of potential groups for mutation in EquiNASE and as $[G]$ in EquiNASD.

We present experiments on image classification for a variety of datasets. The
Rotated MNIST dataset (Larochelle et al., 2007, rotMNIST) is a version of the
MNIST handwritten digit dataset but with the images rotated by any angle. This
task serves as a simple investigational study with known symmetry, while the
following two tasks are more realistic and complex. The Galaxy10 DECals
dataset (Leung & Bovy, 2019, Galaxy10) contains galaxy images in 10 broad
categories. The ISIC 2019 dataset (Codella et al., 2018; Tschandl et al.,
2018; Combalia et al., 2019, ISIC) contains dermascopic images of 8 types of
skin cancer plus a null class. For Galaxy10 and ISIC, we down-sample the
images to $64\times 64$ due to computational constraints, which adds notable
difficulty to the tasks. These tasks exhibit varying levels of rotational and
reflectional symmetry, motivating architectural optimization to determine the
most effective application of equivariance constraints.

Across all experiments, the architectures are designed to have consistent
channel dimensions once expanded to a standard translation-equivariant
convolutional layer for each layer across models. Thus, constrained
equivariance to a larger symmetry group results in fewer learnable parameters.
A layer constrained to $C_{4}$ equivariance has $|C_{4}\setminus D_{4}|=2$
times as many independent channels and as many parameters as a layer
constrained to $D_{4}$ equivariance. This is a notably different paradigm than
other works that equate parameter counts across architectures with different
equivariance properties.

As baseline comparisons, we train and test G-CNNs with static architectures.
In addition to the static baselines, we re-implement the residual pathway
priors (RPP) approach by Finzi et al. (2021) as a $C_{1}$ equivariant layer
with regularization in parallel with a $D_{4}$ equivariant convolutional
layer.

#### Architecture backbone

For both EquiNASE and EquiNASD experiments, we use the same backbone
architecture, such that the static baselines have the same architecture across
experiments. The architectures have a lifting layer followed by 7 group
convolutional layers, for a total of 8 convolutional layers. After 4 layers,
the channel count doubles, from 16 to 32 for a $D_{4}$ equivariant layer and
scaling up for smaller symmetry group equivariance constraints. An average
pooling layer is placed after every other layer for all architectures and
additionally after the fifth and seventh convolutional layers for Galaxy10 and
ISIC. After the final group convolutional layer is a group-dimension average
pooling followed by two linear layers to the output dimension. Every
convolutional and linear layer except the output layer is immediately followed
by a batchnorm then a ReLU.

#### Hyperparameters

The hyperparameters for each algorithm are selected such that baselines only
differ by training time and optimizers. The learning rates were selected by
grid search over baselines on rotMNIST. For all experiments in Section 5.1, we
use a simple SGD optimizer with learning rate $0.1$ to avoid confounding
effects such as momentum during the morphism. For EquiNASE, the parent
selection size is 5, the training time per generation is 0.5 epochs, and the
number of generations is 50 for all tasks. Baselines were trained for the
equivalent number of epochs. For all experiments in Section 5.2, we use
separate Adam optimizers for $\Psi$ and $Z$, each with a learning rate of
$0.01$ and otherwise default settings. The total training time is 100 epochs
for rotMNIST and 50 epochs for Galaxy10 and ISIC. For RPP, we use a
$C_{1}$-equivariant layer with an $L2$ regularization parameter of $110-6$ in
parallel with a $D_{4}$-equivariant layer without regularization.

For rotMNIST, we use the standard training and test split with a batch size of
64, reserving 10% of the training data as the validation set. For Galaxy10, we
set aside 10% of the dataset as the test set, reserving 10% of the remaining
training data as the validation set. For ISIC, we set aside 10% of the
available training dataset as the test set, reserving 10% of the remaining
training data as the validation. For the latter two datasets, we resize the
images to $64\times 64$ due to computational constraints and use a batchsize
of 32. The validation sets were previously used for hyperparameter tuning: for
experimental results, they are only used for the experiments in Section 5.1 as
necessary for the EquiNASE algorithm. No data augmentation is performed,
although the datasets are normalized.

#### Ablations and Random Search

We implement two kinds of random search for each NAS method. The first ablates
smart architecture search: EquiNASE Random Select works as described in
Algorithm 1 but with random parent selection (instead of pareto-front
selection) and EquiNASD Random $Z$ works as described in Algorithm 2 but with
random $Z$ updates (instead of gradient descent) by shuffling $Z$ gradients.
The second is more akin to standard NAS random search: for the evolutionary
paradigm, we train 30 randomly selected static architectures in the discrete
architecture search space for the same training time and selecting the top 5
by validation accuracy, and for the differentiable paradigm, we train 25
randomly selected static architectures in the continuous architecture search
space and selecting the top 5 by validation accuracy. 30 and 25 were
respectively calculated to be approximately the same compute cost as the
trials of EquiNASE and EquiNASD. These are labeled as “Random Static” for both
the evolutionary and differentiable paradigms. Since Random Static trains
static architectures while EquiNASE and EquiNASD dynamically search for both
architectures and parameters, we take the best 5 architectures for each and
retrain their parameters from scratch as in the standard NAS paradigm, labeled
as EquiNASE Retrain and EquiNASD Retrain, respectively, for fair comparison to
Random Static.

For each paradigm of experiments, we present their results in the following
subsections, with general discussion following in Section 6.

###  5.1 Evolutionary Equivariance-aware NAS

Method |  rotMNIST |  Galaxy10 |  ISIC
---|---|---|---
EquiNASE |  1.78 $\pm$ 0.04 |  20.3 $\pm$ 0.9 |  31.0 $\pm$ 0.4
RPP (Finzi et al., 2021) |  2.18 $\pm$ 0.04 |  24.3 $\pm$ 2.8 |  32.2 $\pm$ 1.7
$D_{4}$ baseline |  1.78 $\pm$ 0.11 |  50.8 $\pm$ 17.0 |  32.1 $\pm$ 2.4
$C_{4}$ baseline |  1.64 $\pm$ 0.22 |  29.6 $\pm$ 5.5 |  32.9 $\pm$ 1.0
$C_{1}$ baseline |  5.02 $\pm$ 1.15 |  31.6 $\pm$ 4.8 |  33.2 $\pm$ 1.5
$C_{4}$ (prior: $D_{4}$) |  1.93 $\pm$ 0.05 |  27.8 $\pm$ 5.3 |  31.9 $\pm$ 1.5
$C_{1}$ (prior: $D_{4}$) |  3.40 $\pm$ 0.07 |  25.9 $\pm$ 2.3 |  31.4 $\pm$ 2.6
$C_{1}$ (prior: $C_{4}$) |  2.96 $\pm$ 0.05 |  30.7 $\pm$ 7.1 |  32.5 $\pm$ 1.1
Table 1: Test error percentages (lower is better) across tasks and approaches.
Statistics are aggregated over the final selected population of 5 individuals
for EquiNASE and across 5 random seeds for all other methods. The best and
second best average errors for each task are highlighted. See Figure 2 for
individual results and additional experiments. Figure 2: Test errors for
experiments of Section 5.1, including ablation and random search experiments.
Figure 3: Test errors against stored parameter count for experiments of
Section 5.1. For EquiNASE, parameter counts of the final models are shown,
although training begins with the same parameter count as the $D_{4}$
baselines as shown in Figure 4. Although RPP trains with a higher parameter
count, the parameters may be stored in a single filter per layer using
Equation 11 for testing.

(a) RotMNIST

(b) Galaxy10

(c) ISIC

Figure 4: Historical parameter counts of each selected individual for EquiNASE
for each task. The architectures of the final selected population are labeled.
Each parameter count history is colored according to the final test accuracy,
which is measured of each individual upon removal.

The classification test errors are listed in Table 1 and visualized in Figure
2. The advantages of equivariance search methods are most apparent in the
Galaxy10 benchmark. While EquiNASE outperforms most baselines on rotMNIST and
all baselines on ISIC, it has similar performance on both tasks to the $D_{4}$
baseline, and some of the final architectures are very similar to the $D_{4}$
baseline architecture. However, the $D_{4}$ baseline fails at the Galaxy10
task, demonstrating that the same equivariant architecture can not be naively
applied to different tasks. Both search methods, EquiNASE and RPP, outperform
all baseline models on Galaxy10, and by a large margin for EquiNASE.

The evolutionary progress for the trial on each task is shown in Figure 4: the
selected population maintains a fully equivariant network in every generation,
except for the final generation in both Galaxy10 and ISIC. For RotMNIST, the
final selected population originates from two main lineages, one staying fully
equivariant until the last generations and the other diverging from the fully
equivariant network midway through, showing that training with dynamically
constrained parameterizations can produce performant models.

In addition to the normally initialized static baselines, we also train and
test baselines that are initialized with priors to larger symmetry groups.
These are implemented by initializing all layers to be constrained to the
prior symmetry group, then using the equivariance relaxation morphism on each
layer. EquiNASE searches for relaxation schedules that yield trained priors on
equivariance, while these additional baselines yield untrained priors. The
results in Table 1 show that the $C_{1}$-equivariant networks generally
improve with either equivariance prior, while the $C_{4}$ equivariant networks
perform better with $D_{4}$ equivariance initialization only when the $D_{4}$
constrained baselines also work well. The untrained prior methods do not
perform as well as EquiNASE on rotMNIST, showing the benefit of investing some
training time to the constrained equivariance. For the other tasks, the
baselines with priors have better performances than their constrained baseline
counterparts.

Shown in Figure 2, EquiNASE outperforms EquiNASE Random Select, showing the
benefit of using informed selection to guide the relaxation of equivariance
constraints over training. Additionally, EquiNASE Retrain outperforms the
Random Static baseline, showing that using compute in an informed search is
more beneficial than just randomly searching the space of static architecture
constraints.

###  5.2 Differentiable Equivariance-aware NAS

Method |  rotMNIST |  Galaxy10 |  ISIC
---|---|---|---
EquiNASD |  2.29 $\pm$ 0.27 |  21.8 $\pm$ 1.2 |  32.8 $\pm$ 0.6
RPP (Finzi et al., 2021) |  2.89 $\pm$ 0.27 |  22.0 $\pm$ 1.8 |  31.5 $\pm$ 0.9
$D_{4}$ Baseline |  2.97 $\pm$ 1.50 |  22.5 $\pm$ 2.0 |  32.0 $\pm$ 1.0
$C_{4}$ Baseline |  2.43 $\pm$ 0.54 |  22.2 $\pm$ 2.4 |  32.8 $\pm$ 1.0
$C_{1}$ Baseline |  3.97 $\pm$ 0.75 |  26.5 $\pm$ 1.5 |  32.9 $\pm$ 3.1
Table 2: Test error percentages (lower is better) in percent of incorrect
classifications across tasks and approaches. The best and second best average
errors for each task are highlighted. See Figure 5 for individual results and
additional experiments. Figure 5: Test errors for experiments of Section 5.2,
including ablation and random search experiments. Figure 6: Test errors
against stored parameter count for experiments of Section 5.2. Although
EquiNASD and RPP train with a higher parameter count, the parameters may be
stored in a single filter per layer using Equation 11 for testing. Figure 7:
Architecture weighting parameters by layer for all trials on rotMNIST. Figure
8: Architecture weighting parameters by layer for all trials on Galaxy10.
Figure 9: Architecture weighting parameters by layer for all trials on ISIC.

The classification test errors are listed in Table 2 and visualized in Figure
5. EquiNASD achieves better test accuracy than the other comparable methods on
rotMNIST and Galaxy10. Due to differences in training protocol, only
comparisons of relative rankings with Table 1 are possible: baseline methods
accuracies followed similar patterns to ranking between experimental
paradigms, suggesting the benefit of general $C_{4}$ equivariance for rotMNIST
and Galaxy10 and general $D_{4}$ equivariance, including RPP, for ISIC. In
this training protocol notably with adaptive optimizers, the results are more
consistent across methods and trials.

The architecture weighting parameter dynamics for each trial are shown in
Figure 7 for RotMNIST, Figure 8 for Galaxy10, and Figure 9 for ISIC. The
general trend of less constrained layers toward the end of the network
supports the conjecture of local equivariance being beneficial. However, this
effect is less consistent for ISIC with possibly less inherent symmetry:
trials on ISIC, the only task where EquiNASD did not exceed baselines, had the
most varied architectures. The final mixing of architectures for ISIC included
a high level of $C_{1}$, indicating that feature analysis outside of these
symmetry groups is important for this benchmark.

Previous differentiable NAS works often used regularization of network size or
even architecture weighting parameters themselves to encourage efficient
architectures with a single highly weighted choice for each layer. However,
our algorithm shows strong preference for a single, more equivariant and thus
more expressive layer, notably to $D_{4}$ or $C_{4}$ equivariance, without
such regularization. This may be due to the bilevel optimization dynamics:
more constrained layers may be able to make more effective updates to more
closely approximate the correct gradient computation that assumes optimal
weights and thus become favorable compared to the lagging larger layers. This
conjecture is shown particularly for trials on Galaxy10: increases in the
architectural weighting parameter towards $C_{1}$ equivariance often comes
after having strong weights for operations equivariant to larger groups.

EquiNASD finds competitive architectures on average and can find architectures
which outperform baseline choices like architectures fully equivariant to
$C_{1}$ or $D_{4}$. From comparing EquiNASD Retrain to EquiNASD results in
Figure 5, retraining a resulting architecture is not consistently better or
worse than using the final weights from EquiNASD, showing that there isn’t a
disadvantage to training weights during search and avoiding the additional
cost of a post-search training step.

The use of randomized loss information in Random Static in Figure 5 shows that
an informed search for architecture hyperparameters is generally useful.
However, experiments on the ISIC benchmark demonstrates that the architecture
search can be deceptive and that random loss information can outperform
informed loss. This motivates exploration into the use of noise during the
search process for architectural parameters.

A sampling of random continuous architectures in Random Static in Figure 5
shows that random architectures can perform well on problems where fully
equivariant architectures like the $D_{4}$ baseline already perform well.
However, on the Galaxy10 problem, the $D_{4}$ and $C_{4}$ baseline have high
variance, suggesting that a fully equivariant architecture is sub-optimal. On
this baseline, EquiNASD greatly outperforms a search of random architectures,
demonstrating that EquiNASD can discover the appropriate equivariance for a
specific dataset over fixed or randomly selected architectures.

The search space for EquiNASD is already well-formed for random architectures,
compared to the discrete search space of EquiNASE. This is enabled by the
$[G]$-mixed equivariant layer, which is a contribution of this work. Random
non-mixed equivariant architectures do worse on all three benchmarks compared
to random architectures which use the $[G]$-mixed equivariant layer. This can
explain why the EquiNASD results are closer to random baselines than the
EquiNASE results, as the search space permits easily finding the appropriate
mix of equivariances compared to a discretized search space.

Figure 10: EquiNASD on classification of six augmentations of MNIST, where
each row is a trial on MNIST augmented to exhibit symmetry to the labeled
group.

To explore the symmetry discovery of EquiNASD, we apply it to six
augmentations of the MNIST dataset (LeCun et al., 1998), where each
augmentation applies the group actions of each group in
$\\{C_{1},C_{2},C_{4},D_{1},D_{2},D_{4}\\}$ respectively. The resulting
architecture dynamics of this experiment are shown in Figure 10, showing that
less augmented versions still have some inherent symmetry, while more
augmented versions induce stronger architectural changes towards layers that
are equivariant to larger groups. Across all augmentations, earlier layers
tended towards more constraints to equivariance.

##  6 Discussion

To our knowledge, this is the first work which proposes search methods for
networks with dynamically constrained equivariance. Many NAS approaches
separately search for an architecture and then reinitialize and retrain the
weights, while our two proposed approaches find an optimal architecture with
trained weights in a single process, notably with dynamically constrained
weights. Gradient-based tuning (Maclaurin et al., 2015) has shown the benefit
not only of optimizing hyperparameters but also of dynamically adjusting them
during training (Lichtarge et al., 2022). There is an inherent trade-off
between accuracy and generalization capabilities with more constrained
equivariance: dynamically constrained weights can reap both over the course of
training.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNASE
searches for architectures composed of discretely equivariant layers, while
EquiNASD searches for continuous mixtures of equivariance within each layer.
The EquiNASD algorithm avoids many known problems in differentiable NAS such
as the discretization gap that occurs when searching over a continuous
relaxation of a discrete architectural search space (Xie et al., 2021), such
as that of EquiNASE. Towards searching for discretely equivariant layers using
the $[G]$-mixed equivariant layer, proximal NAS algorithms use techniques such
as projection (Yao et al., 2020) and straight-through estimation (Li et al.,
2022) to avoid the discretization gap and thus may be effective for this
application.

The EquiNASE algorithm is innately greedy. At each selection step, the
population is evaluated on the known current performance rather than the
unknown final performance, so this metric is biased to architectures that
train quickly. Networks constrained to higher symmetry group equivariance tend
to learn faster, but this could be confounded by the equivariance relaxation
causing large gradients for the newly unconstrained parameters and thus
potentially fast increases in performance. Further work could utilize other
metrics for final performance, such as proxies (White et al., 2022).

All of the theoretical and algorithmic contributions of this work are
applicable beyond the image classification experiments presented to
architectures with parametrized equivariance to any discrete group. We leave
the extension to other group representations and domains as future work, such
as the continuous case via careful analysis of the regular representation,
still given $G^{\prime}\setminus G$ is finite.

Our proposed equivariance-aware NAS problems can be practically applied to
find effective networks for datasets with hypothesizable symmetry. EquiNASE
may particularly work well on tasks that benefit from local equivariance,
which can be determined by analyzing the architecture weighting parameters
from first applying the more efficient EquiNASD, as well as for finding good
discrete architectures within which to retrain weights, based on the ablation
and random comparisons. For tasks where EquiNASD models have less consistent
equivariance patterns, EquiNASE could be adapted to propose candidates that
relax any layer in the network, removing our constraint of non-increasing
equivariance. We thus recommend EquiNASD for practical applications if the
final model is not restricted to discrete equivariance, in which case it can
be used to inform design decisions for applying EquiNASE.

Beyond NAS, the equivariance relaxation morphism could be used in other
applications such as fine-tuning and distillation. Layers of a pre-trained
equivariant network could be expanded via equivariance relaxation before fine-
tuning on the same or a new task. Similarly, a network could be distilled to a
wider architecture for additional performance benefits.

The proposed equivariance-aware NAS algorithms are intentionally simple to
focus on studying the architectural mechanisms presented in Section 3,
although we have already discussed potential improvements to these algorithms.
We include all valid results, even those that do not favor our own techniques,
for transparency. This work aims to build a foundation for the intersection of
equivariant architectures and NAS, rather than over-engineer the algorithms
and experiments for incremental performance gains on selected benchmarks.

#### Conclusion

We present two mechanisms towards equivariance-aware architectural
optimization, the equivariance relaxation morphism and the $[G]$-mixed
equivariant layer, as well as two NAS algorithms that implement these
mechanisms respectively in an evolutionary approach, EquiNASE, and a
differentiable approach, EquiNASD. We investigate how the dynamic equivariance
achieved by these algorithms affects the training and performance of networks
across multiple image classification tasks of varying complexity and assumed
symmetry, demonstrating that these techniques can search for performant
architectures and weights even on noisy tasks. The proposed mechanisms and
algorithms are extendable beyond vision tasks to any architecture with
parametrized equivariance to any discrete group.

## References

  * Agrawal & Ostrowski (2022) Devanshu Agrawal and James Ostrowski.  A classification of $G$-invariant shallow neural networks.  _Advances in Neural Information Processing Systems_ , 35, 2022.
  * Basu et al. (2021) Sourya Basu, Akshayaa Magesh, Harshit Yadav, and Lav R Varshney.  Autoequivariant network search via group decomposition.  _arXiv preprint arXiv:2104.04848_ , 2021.
  * Bronstein et al. (2021) Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković.  Geometric deep learning: grids, groups, graphs, geodesics, and gauges.  _arXiv preprint arXiv:2104.13478_ , 2021.
  * Codella et al. (2018) Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC).  In _International Symposium on Biomedical Imaging_ , pp. 168–172. IEEE, 2018.
  * Cohen & Welling (2016) Taco Cohen and Max Welling.  Group equivariant convolutional networks.  In _International Conference on Machine Learning_ , pp. 2990–2999. PMLR, 2016.
  * Combalia et al. (2019) Marc Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al.  BCN20000: Dermoscopic lesions in the wild.  _arXiv preprint arXiv:1908.02288_ , 2019.
  * Elsayed et al. (2020) Gamaleldin Elsayed, Prajit Ramachandran, Jonathon Shlens, and Simon Kornblith.  Revisiting spatial invariance with low-rank local connectivity.  In _International Conference on Machine Learning_ , pp. 2868–2879. PMLR, 2020.
  * Elsken et al. (2017) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Simple and efficient architecture search for convolutional neural networks.  In _NIPS Workshop on Meta-Learning_ , 2017.
  * Falanti et al. (2022) Andrea Falanti, Eugenio Lomurno, Stefano Samele, Danilo Ardagna, Matteo Matteucci, et al.  POPNASv2: An efficient multi-objective neural architecture search technique.  In _International Joint Conference on Neural Networks_ , 2022.
  * Finzi et al. (2021) Marc Finzi, Gregory Benton, and Andrew G Wilson.  Residual pathway priors for soft equivariance constraints.  _Advances in Neural Information Processing Systems_ , 34:30037–30049, 2021.
  * Fukushima & Miyake (1982) Kunihiko Fukushima and Sei Miyake.  Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition.  In _Competition and Cooperation in Neural Nets_ , pp. 267–285. Springer, 1982.
  * Herstein (2006) Israel N Herstein.  _Topics in algebra_.  John Wiley & Sons, 2006.
  * Kondor & Trivedi (2018) Risi Kondor and Shubhendu Trivedi.  On the generalization of equivariance and convolution in neural networks to the action of compact groups.  In _International Conference on Machine Learning_ , pp. 2747–2755. PMLR, 2018.
  * Larochelle et al. (2007) Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio.  An empirical evaluation of deep architectures on problems with many factors of variation.  In _International Conference on Machine Learning_ , pp. 473–480. PMLR, 2007.
  * LeCun et al. (1989) Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel.  Backpropagation applied to handwritten ZIP code recognition.  _Neural Computation_ , 1(4):541–551, 1989.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Leung & Bovy (2019) Henry W Leung and Jo Bovy.  Deep learning of multi-element abundances from high-resolution spectroscopic data.  _Monthly Notices of the Royal Astronomical Society_ , 483(3):3255–3277, 2019.
  * Li et al. (2022) Yanyu Li, Pu Zhao, Geng Yuan, Xue Lin, Yanzhi Wang, and Xin Chen.  Pruning-as-search: Efficient neural architecture search via channel pruning and structural reparameterization.  In _International Joint Conference on Artificial Intelligence_ , 2022\.
  * Lichtarge et al. (2022) Jared Lichtarge, Chris Alberti, and Shankar Kumar.  Simple and effective gradient-based tuning of sequence-to-sequence models.  _International Conference on Automated Machine Learning - Workshop Track_ , 2022.
  * Liu et al. (2018) Hanxiao Liu, Karen Simonyan, and Yiming Yang.  DARTS: Differentiable architecture search.  In _International Conference on Learning Representations_ , 2018.
  * Lu et al. (2019) Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf.  NSGA-Net: neural architecture search using multi-objective genetic algorithm.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , pp. 419–427, 2019.
  * Maclaurin et al. (2015) Dougal Maclaurin, David Duvenaud, and Ryan Adams.  Gradient-based hyperparameter optimization through reversible learning.  In _International Conference on Machine Learning_ , pp. 2113–2122. PMLR, 2015.
  * Maile et al. (2022) Kaitlin Maile, Hervé Luga, and Dennis G Wilson.  Structural learning in artificial neural networks: A neural operator perspective.  _Transactions on Machine Learning Research_ , 2022.
  * Ravanbakhsh et al. (2017) Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos.  Equivariance through parameter-sharing.  In _International Conference on Machine Learning_ , pp. 2892–2901. PMLR, 2017.
  * Real et al. (2017) Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin.  Large-scale evolution of image classifiers.  In _International Conference on Machine Learning_ , pp. 2902–2911. PMLR, 2017.
  * Romero & Lohit (2022) David W Romero and Suhas Lohit.  Learning partial equivariances from data.  _Advances in Neural Information Processing Systems_ , 35, 2022.
  * Salimans & Kingma (2016) Tim Salimans and Durk P Kingma.  Weight normalization: A simple reparameterization to accelerate training of deep neural networks.  _Advances in Neural Information Processing Systems_ , 29, 2016.
  * Tschandl et al. (2018) Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  _Scientific Data_ , 5(1):1–9, 2018.
  * van der Ouderaa et al. (2022) Tycho FA van der Ouderaa, David W Romero, and Mark van der Wilk.  Relaxing equivariance constraints with non-stationary continuous filters.  _Advances in Neural Information Processing Systems_ , 35, 2022.
  * Wang et al. (2022) Rui Wang, Robin Walters, and Rose Yu.  Approximately equivariant networks for imperfectly symmetric dynamics.  In _International Conference on Machine Learning_. PMLR, 2022.
  * Wei et al. (2016) Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen.  Network morphism.  In _International Conference on Machine Learning_ , pp. 564–572. PMLR, 2016.
  * Weiler & Cesa (2019) Maurice Weiler and Gabriele Cesa.  General E(2)-equivariant steerable CNNs.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Weiler et al. (2021) Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling.  Coordinate independent convolutional networks - isometry and gauge equivariant convolutions on riemannian manifolds.  _arXiv preprint arXiv:2106.06020_ , 2021.
  * White et al. (2022) Colin White, Mikhail Khodak, Renbo Tu, Shital Shah, Sébastien Bubeck, and Debadeepta Dey.  A deeper look at zero-cost proxies for lightweight NAS.  In _ICLR Blog Track_ , 2022.  URL https://iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/.
  * Xie et al. (2021) Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang, Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, et al.  Weight-sharing neural architecture search: A battle to shrink the optimization gap.  _ACM Computing Surveys (CSUR)_ , 54(9):1–37, 2021\.
  * Yao et al. (2020) Quanming Yao, Ju Xu, Wei-Wei Tu, and Zhanxing Zhu.  Efficient neural architecture search via proximal iterations.  In _AAAI Conference on Artificial Intelligence_ , pp. 6664–6671, 2020.
  * Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing.  Equivariance discovery by learned parameter-sharing.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 1527–1545. PMLR, 2022.
  * Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn.  Meta-learning symmetries by reparameterization.  In _International Conference on Learning Representations_ , 2020.
  * Zoph & Le (2017) Barret Zoph and Quoc Le.  Neural architecture search with reinforcement learning.  In _International Conference on Learning Representations_ , 2017.

##  Appendix A Additional Background

###  A.1 Symmetries in Neural Networks

A symmetry of an object is a mapping of the object onto itself such that
structure is preserved. A symmetry group $G$ is a set of such mappings along
with a binary operation $\cdot\colon G\times G\to G$, known as the group
product, that satisfies axioms for closure, associativity, the identity, and
the inverse (Herstein, 2006). A group $G$ acts on a set $\mathcal{X}$ via the
group action $.:G\times\mathcal{X}\to\mathcal{X},\ (g,x)\mapsto g.x$ that
satisfies axioms for identity and compatibility: $\mathcal{X}$ is called a
$G$-space.

Equivariance is the property of a mapping such that transformation of the
input results in equivalent transformation of the output. Formally, a mapping
$h\colon\mathcal{X}\to\mathcal{Z}$ between two $G$-spaces is $G$-equivariant
if for all $g\in G$ and $x\in\mathcal{X}$ we have: $h(g.x)=g.h(x)$. For
example, an image segmentation neural network should be $T(2)$-equivariant:
shifting the input should result in the same shift in the output.

Invariance is a special case of equivariance, where the output of the function
is completely independent of transformation of the input. Formally, a mapping
$h\colon\mathcal{X}\to\mathcal{Z}$ is $G$-invariant if for all $g\in G$ and
$x\in\mathcal{X}$ we have: $h(g.x)=h(x)$. For example, an image classification
network should be $T(2)$-invariant: shifting the input should not change the
output. Symmetries leave objects invariant.

For two groups $G$ and $H$ with group products $\cdot_{G}$ and $\cdot_{H}$
respectively where $H$ acts on $G$ with group action ., the (outer) semi-
direct product $G\rtimes H$ of $H$ acting on $G$ is a group composed of the
set of elements $G\times H$ with group product
$(g,h)\cdot(g^{\prime},h^{\prime})=(g\cdot_{G}(h.g^{\prime}),h\cdot_{H}h^{\prime})$
and inverse $(g,h)^{-1}=(h^{-1}.g^{-1},h^{-1}).$

A subgroup $H$ of $G$ is a nonempty subset with the same group product that
also fulfills the group axioms. Then, $gH=\\{g\cdot h|h\in H\\}$ and
$Hg=\\{h\cdot g|h\in H\\}$ denote the left coset and right coset,
respectively, of $H$ with representative $g$.

###  A.2 Neural Architecture Search

Evolutionary algorithms are optimization methods inspired by evolution in
biology, where individuals in a population compete with their phenotypic
traits in order to pass on their genotypic traits to offspring. The population
is the current collection of individuals. Each individual is an instance of
the object to be optimized and has a genotype that is decoded into a
phenotype. In this case, each individual is a neural network, with a genotype
that encodes the parametrized equivariance group of each convolutional layer,
represented as a vector of integers. The individual continues training on the
task before competing against other individuals to be selected as a parent to
mutate to generate the next population. Each parent itself is kept for the
next population, as well as each valid child that is generated via the
equivariance relaxation morphism, such that they are functionally equivalent
to their parent at initialization (although with a different architecture) and
thus have the same fitness before training. Each individual in this population
is partially trained, such that these children diverge from their siblings and
parent, so that the next set of parents may be selected and this process
repeats.

Pareto dominance can be used in multi-objective optimizations to select the
next parent population. For a population of individuals each scored in $n$
objectives, an individual is pareto-optimal if no individual has at least one
strictly better score for an objective without that of any other objectives
being strictly worse. The pareto front is the set formed by all pareto-optimal
individuals.