# Equivariant Few-Shot Learning from Pretrained Models

Sourya Basu
University of Illinois at Urbana-Champaign
&Pulkit Katdare$~{}^{*}$
University of Illinois at Urbana-Champaign
Prasanna Sattigeri
IBM Research
&Vijil Chenthamarakshan
IBM Research
&Katherine Driggs-Campbell
University of Illinois at Urbana-Champaign
&Payel Das
IBM Research
&Lav R. Varshney
University of Illinois at Urbana-Champaign
Equal contribution.

###### Abstract

Efficient transfer learning algorithms are key to the success of foundation
models on diverse downstream tasks even with limited data. Recent works of
Basu et al. (2023) and Kaba et al. (2022) propose group averaging (equitune)
and optimization-based methods, respectively, over features from group-
transformed inputs to obtain equivariant outputs from non-equivariant neural
networks. While Kaba et al. (2022) are only concerned with training from
scratch, we find that equitune performs poorly on equivariant zero-shot tasks
despite good finetuning results. We hypothesize that this is because
pretrained models provide better quality features for certain transformations
than others and simply averaging them is deleterious. Hence, we propose
$\lambda$-equitune that averages the features using importance weights,
$\lambda$s. These weights are learned directly from the data using a small
neural network, leading to excellent zero-shot and finetuned results that
outperform equitune. Further, we prove that $\lambda$-equitune is equivariant
and a universal approximator of equivariant functions. Additionally, we show
that the method of Kaba et al. (2022) used with appropriate loss functions,
which we call equizero, also gives excellent zero-shot and finetuned
performance. Both equitune and equizero are special cases of
$\lambda$-equitune. To show the simplicity and generality of our method, we
validate on a wide range of diverse applications and models such as 1) image
classification using CLIP, 2) deep Q-learning, 3) fairness in natural language
generation (NLG), 4) compositional generalization in languages, and 5) image
classification using pretrained CNNs such as Resnet and Alexnet.

##  1 Introduction

Group-equivariant deep learning leverages group equivariance as an inductive
bias to design efficient and reliable neural networks. Popular examples
include convolutional neural networks (CNNs) equivariant to translations
(LeCun et al., 1989), group convolutional neural networks (GCNNs) equivariant
to general discrete groups (Cohen & Welling, 2016), and recently Alphafold2
equivariant to 3D rotations (Jumper et al., 2021). But these methods cannot
leverage pretrained models.

With the increase in open sourced large pretrained models, it is now crucial
to develop efficient transfer learning algorithms that can leverage group
equivariance. Basu et al. (2023) proposed equitune, an equivariant finetuning
method that uses group averaging over features extracted from pretrained
models. Several other methods were proposed to get equivariant output from
non-equivariant backbone architectures, e.g. some form of averaging (Puny et
al., 2021; Atzmon et al., 2022) or optimization over certain proxy loss
functions (Kaba et al., 2022). But these latter methods were originally
designed for training from scratch and not much is known about their
finetuning abilities.

Here, we find that equitune performs poorly on zero-shot tasks. Our first main
contribution is to show that the optimization method of Kaba et al. (2022)
when used with appropriate proxy loss functions provides excellent zero-shot
and finetuning performance. We call this method equizero.

The results from equizero suggest that pretrained models provide better
quality features for some group transformations than others. Thus, we propose
$\lambda$-equitune, which learns importance weights directly from the data and
uses them to perform a weighted group averaging. We show $\lambda$-equitune
outperforms equitune and is competitive with equizero for zero-shot tasks.
Moreover, for finetuning, $\lambda$-equitune often outperforms both equitune
and equizero. This constitutes our second main contribution.

Figure 1:  Implementation of $\lambda$-equitune on CLIP. Weighted average of
image features corresponding to transformed inputs are computed, which is
further used for computing text-image similarity scores.

To validate our methods, we provide experiments on a diverse set of pretrained
models and datasets. We show zero-shot performance of equizero on: 1) image
classification using CLIP, 2) deep Q-learning, 3) fairness in natural language
generation (NLG), and 4) compositional generalization in languages. For CLIP
and deep Q-learning, we used the naturally available loss functions, namely,
similarity scores and $Q$-values, respectively. For fairness in NLG and
compositional generalization, we closely follow the setup of Basu et al.
(2023), and we use _regard scores_ Sheng et al. (2019) and the negative of the
maximum of the output probability distribution, respectively, as the loss
function.

We first show results of $\lambda$-equitune for image classification using
CLIP, finding that $\lambda$-equitune performs competitively with equizero and
outperforms equitune for both zero-shot and finetuning tasks. Then, we show a
simple case where finding a good proxy loss function for equizero is non-
trivial: image classification using pretrained CNNs. Here, we find that
equizero performs even worse than equitune but $\lambda$-equitune easily
outperforms equitune and equizero. The organization of the paper is summarized
below:

  * •

§3 provides details of $\lambda$-equitune and equizero and proves a number of
their properties.

  * •

§4 provides overview of the applications considered in the experiments and how
equivariance methods are used there.

  * •

§5 provides experimental details and results for all the applications.

##  2 Background

Here we discuss relevant concepts in group equivariance and group equivariant
transfer learning.

##### Group Equivariance

A group $(G,\cdot)$ is a set $G$ accompanied by a binary operator ‘$\cdot$’
that satisfy the axioms of a group, namely i) closure: $g\cdot h\in G$ for all
$g,h\in G$; ii) associativity: $(g\cdot h)\cdot k=g\cdot(h\cdot k)$; iii)
identity: there exists $e\in G$, such that $g\cdot e=e\cdot g$ for all $g\in
G$; iv) inverse: for every element $g\in G$, there exists $g^{-1}$, such that
$g\cdot g^{-1}=g^{-1}\cdot g=e$. We write $g\cdot h$ as $gh$ for brevity.

Given a set ${\mathcal{X}}$, we define a group action of $G$ on
${\mathcal{X}}$ as
$\Gamma_{{\mathcal{X}}}:G\times{\mathcal{X}}\mapsto{\mathcal{X}}$ such that it
satisfies two axioms, namely i) identity: $\Gamma_{{\mathcal{X}}}(e,x)=x$ for
all $x\in{\mathcal{X}}$, where $e\in G$ is the identity; ii) compatibility:
$\Gamma_{{\mathcal{X}}}(g,\Gamma_{{\mathcal{X}}}(h,x))=\Gamma_{{\mathcal{X}}}(gh,x)$,
for all $g,h\in G,x\in{\mathcal{X}}$. We write $\Gamma_{{\mathcal{X}}}(g,x)$
simply as $gx$ for brevity.

A model ${\mathbf{M}}:{\mathcal{X}}\mapsto{\mathcal{Y}}$ is equivariant to $G$
under the group action of $G$ on ${\mathcal{X}}$ and ${\mathcal{Y}}$ if
${\mathbf{M}}(gx)=g{\mathbf{M}}(x))$ for all $g\in G,x\in{\mathcal{X}}$. This
essentially means that any group transformation $g$ to the input
$\Gamma_{{\mathcal{X}}}(g,x)$ should reflect with an equivalent group
transformation of the output $\Gamma_{{\mathcal{Y}}}(g,{\mathbf{M}}(x))$.

##### Equivariant Finetuning

Recently, Basu et al. (2023) proposed a finetuning method called equituning
that starts with potentially non-equivariant model ${\mathbf{M}}$ and produces
a model $\mathbf{M_{G}}$ that is equivariant to $G$. Equituning converts a
pretrained model into an equivariant version by minimizing the distance of
features obtained from pretrained and equivariant models. The output of an
equituned model is given by

$\displaystyle\mathbf{M_{G}}(x)=\frac{1}{|G|}\sum_{g\in
G}g^{-1}{\mathbf{M}}(gx).$ (1)

While the averaging in equation 1 is shown to be useful for finetuning, we
find it leads to poor equivariant zero-shot learning. This could be because
the pretrained model outputs high quality features only for some of the
transformed inputs. Hence, averaging them directly leads to low quality zero-
shot performance. This can be avoided by using weighted averaging as discussed
in Sec. 3.

##### Optimization-Based Canonical Representation

On the other hand, Kaba et al. (2022) show that group equivariant output,
$\mathbf{M_{G}^{OPT}}(x)$, can be obtained by optimizing a (non-equivariant)
loss function $l({\mathbf{M}}(gx))$ with respect to group elements $g\in G$
for any $x\in{\mathcal{X}}$ as shown below.

$\displaystyle\mathbf{M_{G}^{OPT}}(x)=g^{-1}_{*}{\mathbf{M}}(g_{*}x),$ (2)

where $g_{*}=\operatorname*{arg\,min}_{g\in G}l({\mathbf{M}}(gx))$ with
$l:{\mathcal{Y}}\mapsto\mathbb{R}$ being an injective proxy loss function and
assuming the minima is unique. However, the purpose of this formulation in
Kaba et al. (2022) is only to obtain an equivariant representation for
training from scratch. Moreover, no competitive zero-shot or finetuning
performance is obtained in Kaba et al. (2022) using this method. We show that
the choice of $l$ plays an important role in its zero-shot and finetuning
performance, even outperforming equituning. This method is obtained as a
special case of $\lambda$-equituning introduced next.

##### Additional Related Works

Group equivariance plays a key role in geometric deep learning (Bronstein et
al., 2021) for designing efficient domain specific neural networks. Several
elegant architectures have been proposed for equivariant image classification
(Cohen & Welling, 2016, 2017; Romero & Cordonnier, 2020), reinforcement
learning (Mondal et al., 2020; van der Pol et al., 2020a; Mondal et al., 2022;
Wang et al., 2022), graph (Satorras et al., 2021; Keriven & Peyré, 2019;
Gasteiger et al., 2021) and mesh (De Haan et al., 2020; He et al., 2021; Basu
et al., ) processing, natural language processing (Gordon et al., 2019; Li et
al., 2022), and data generation (Dey et al., 2021). These architectures need
to be trained from scratch, which is not always desirable.

Frame averaging produces equivariant output from non-equivariant architecture
backbones (Puny et al., 2021; Atzmon et al., 2022; Duval et al., 2023). Most
work here focuses on finding good frames, which are equivariant subsets of
groups, for specific groups, and not for general groups. And not much is known
about their performance with pretrained models. Kaba et al. (2022) also give a
canonicalization-based method that uses an equivariant auxiliary network for
constructing equivariant networks out of non-equivariant backbones and is used
for training from scratch. But this work requires an additional equivariant
network and appropriate trainable parameterization of the group actions, which
is presented only for certain groups of interest. Further, this work is not
concerned with equivariant performance of pretrained models. Zero-shot group
equivariance was also recently used by Muglich et al.  for zero-shot
coordination in partially observable Markov decision processes (POMDPs). In
contrast, our work aims to be provide efficient equivariant transfer learning
algorithms that are general in terms of considered tasks and groups, and does
not require additional equivariant networks.

##  3 $\lambda$-Equitune

We propose $\lambda$-equitune, where unequal weights are assigned to features
obtained from transformed inputs. This is a simple generalization of equitune
in equation 1 and the optimization-based approach in equation 2, where the
goal is to assign higher values to better features. Like these previous
methods, $\lambda$-equitune is equivariant and a universal approximator of
equivariant functions.

The main idea of $\lambda$-equitune is that given a pretrained model
${\mathbf{M}}$, the features ${\mathbf{M}}(gx)$ for any fixed $x$ are not all
equally important for all $g\in G$. We denote by $\lambda(gx)$ the importance
weight of feature ${\mathbf{M}}(gx)$ for $g\in G,x\in{\mathcal{X}}$. We assume
$G$ is finite, just as in Basu et al. (2023). Suppose
$\lambda:{\mathcal{X}}\mapsto\mathbb{R}^{+}$ is known a priori, and denote the
$\lambda$-equituned model as $\mathbf{M_{G}^{\lambda}}$. Then we want to
minimize

$\displaystyle\min_{\mathbf{M_{G}^{\lambda}}(x)}$ $\displaystyle\sum_{g\in
G}\left\lVert\lambda(gx){\mathbf{M}}(gx)-\mathbf{M_{G}^{\lambda}}(g,x)\right\rVert_{2}^{2}$
(3) s.t.
$\displaystyle\mathbf{M_{G}^{\lambda}}(gx)=g\mathbf{M_{G}^{\lambda}}(x)\textrm{
for all }g\in G.$

We call the solution to equation 3 as $\lambda$-equitune, given by

$\displaystyle\mathbf{M_{G}^{\lambda}}(x)=\frac{1}{\sum_{g\in
G}\lambda(gx)}\sum_{g\in G}^{|G|}g^{-1}\lambda(gx){\mathbf{M}}(gx).$ (4)

###  3.1 Special Cases

When $\lambda(gx)=1$ for $g\in G$, then equation 4 is the same as equation 1.
Further, we get equation 2 when $\lambda$ is an indicator function
$\lambda(gx)=\mathbbm{1}_{\\{g=g_{*}\\}}$, where
$g_{*}=\operatorname*{arg\,min}_{g\in G}l({\mathbf{M}}(gx))$ with
$l:{\mathcal{Y}}\mapsto\mathbb{R}$ such that the minimization is well defined.
We use ${\mathbf{M_{G}^{0}}}$ to denote the equizero model.

The first main contribution of our work is experimental. We show there are
good loss functions for equizero for several diverse tasks that easily
outperform equitune by choosing the best features.

The second main contribution is to show that $\lambda$-equitune outperforms
equitune and is competitive with equizero even when good loss functions are
known. Moreover, when the loss functions are not trivial, equizero performs
even worse than equitune, but $\lambda$-equitune easily outperforms both.

###  3.2 Properties

Now we show in Thm. 1 that equation 4 is equivariant with respect to $G$.

######  Theorem 1 (Equivariance).

$\mathbf{M_{G}^{\lambda}}$ defined in equation 4 is equivariant to $G$.

We define a universal approximator in Def. 1. Then, Thm. 2 proves that
$\lambda$-equitune is a universal approximator of equivariant functions for
groups where $\left\lVert g\right\rVert=1$ for $g\in G$. This includes a wide
range of groups including the permutation group, the $SO(n)$ groups of special
orthogonal groups, etc. This condition is the same as the class of groups
considered in Basu et al. (2023).

######  Definition 1 (Universal approximator).

A model ${\mathbf{M}}:{\mathcal{X}}\mapsto{\mathcal{Y}}$ is a universal
approximator of a continuous function $f:{\mathcal{X}}\mapsto{\mathcal{Y}}$ if
for any compact set ${\mathcal{K}}\subset{\mathcal{X}}$ and $\epsilon>0$,
there exists a choice of parameters for ${\mathbf{M}}$ such that $\left\lVert
f(x)-{\mathbf{M}}(x)\right\rVert\leq\epsilon$ for all $x\in{\mathcal{K}}$.

######  Theorem 2 (Universality).

Let $f_{G}:{\mathcal{X}}\mapsto{\mathcal{Y}}$ be any continuous function
equivariant to group $G$ and let $\lambda:{\mathcal{X}}\mapsto\mathbb{R}^{+}$
be any positive scalar function. And let
${\mathbf{M}}:{\mathcal{X}}\mapsto{\mathcal{Y}}$ be a universal approximator
of $\frac{f_{G}}{\lambda}$. Here ${\mathcal{X}}$ is such that if
$x\in{\mathcal{X}}$, then $gx\in{\mathcal{X}}$ to ensure the equivariance of
$f_{G}$ is well-defined. Then, $\mathbf{M_{G}^{\lambda}}$ is a universal
approximator of $f_{G}$.

##### Computational Complexity

Note that equitune, equizero, and $\lambda$-equitune have the same compute
complexity. In practice, in Tab. 1 we find that equitune and equizero have
exactly the same time and memory consumption, whereas $\lambda$-equitune takes
a little more time and memory because of the additional $\lambda$ network. We
illustrate on RN50 and ViT-B/32 models of CLIP using the same text encoder,
but different image encoders. RN50 uses a Resnet50 based image encoder,
whereas ViT-B/32 uses a vision transformer based image encoder.

Table 1: Inference times of equitune, equizero, and $\lambda$-equitune for the c4 group for various CLIP models on CIFAR100. We use batch size 32 on a single Nvidia A100 GPU. Model | Time (sec.) | Memory (MB)
---|---|---
| Equitune | Equizero |  $\lambda$-Equitune | Equitune | Equizero |  $\lambda$-Equitune
RN50 | 14.15 | 14.10 | 23.5 | 3703 | 3703 | 3941
ViT-B/32 | 11.00 | 10.27 | 16.08 | 2589 | 2587 | 2915

##### Beyond Zero-Shot Learning

Let us emphasize that even though the equizero model in equation 2 is not
differentiable due to the $\operatorname*{arg\,max}$, we can still use simple
gradient estimators known in the literature. One popular estimator is the
straight-through estimator (Bengio et al., 2013), where the equizero output in
equation 2 would be written as
${\mathbf{M_{G}^{0}}}(x)={\mathbf{M}}(x)+({\mathbf{M_{G}^{0}}}(x)-{\mathbf{M}}(x))$.detach(),
where detach() indicates that no gradient flows through the term
$({\mathbf{M_{G}^{0}}}(x)-{\mathbf{M}}(x))$. In practice, we found it to be
slightly better to use $\mathbf{M_{G}}(x)$ instead of ${\mathbf{M}}(x)$ and
write
${\mathbf{M_{G}^{0}}}(x)=\mathbf{M_{G}}(x)+({\mathbf{M_{G}^{0}}}(x)-\mathbf{M_{G}}(x))$.detach().
§5.1.3 illustrates few-shot learning and finetuning using equizero and
compares with equituning.

##  4 Applications

First we present several applications in §4.1 where finding a loss function
for equizero is easy. This naturally leads to excellent equivariant zero-shot
results outperforming equitune. Then, in §4.2 we provide two applications in
image classification to show 1) the benefits and drawbacks of equizero
compared to equitune and $\lambda$-equitune, and 2) that $\lambda$-equitune
consistently performs well avoiding the drawbacks of equizero.

###  4.1 Equizero Applications

Here we provide applications where equizero achieves excellent zero-shot
performance, namely: 1) deep Q-learning, 2) fairness in NLG, and 3)
compositional generalization in languages.

##### Equizero Reinforcement Learning

Recent works, such as van der Pol et al. (2020b); Mondal et al. (2020), have
developed RL algorithms that leverage symmetries in the environments that
helps improve robustness and sample efficiency. But no existing work
efficiently uses group equivariance on top of pretrained RL models. Here we
apply equizero and equitune on top of pretrained models inspired from the
group symmetries found in van der Pol et al. (2020a). We find that equitune
outperforms non-equivariant pretrained models but equizero outperforms both.
We simply use the $Q$-values as the proxy loss function as described in §C.1
with more details.

##### Group-Theoretic Fairness in NLG

We seek to reduce the social biases inherent in language models (LMs),
focusing on GPT2 (Radford et al., 2019). We consider the group-theoretic
fairness setup of Sheng et al. (2019) and Basu et al. (2023). We take the sets
of demographics, namely [‘man’, ‘woman’], [‘straight’, ‘gay’], and [‘black’,
‘white’]. For each demographic group, Sheng et al. (2019) proposed two tasks,
called respect task and occupation task, where each task consists of five
context phrases. A language model (LM) is given these contexts to generate
sentences. These generated sentences are then classified as ‘positive’,
‘negative’, ‘neutral’, or ‘other’ by a regard classifier also proposed by
Sheng et al. (2019). These outputs are called regard scores. A regard
classifier is a BERT-based model similar to a sentiment classifier but more
specific for fairness tasks. We use equizero using the regard scores as the
proxy loss function, so as to maximize positivity in the generated texts,
while guaranteeing group-theoretic fairness. To that end, we propose two
algorithms that help in reducing existing biases across demographic groups.

EquizeroLM and R-EquizeroLM: Basu et al. (2023) define EquiLM and R-EquiLM,
that use a sequential version of equitune to perform group transformed
averaging to achieve fairness across demographic groups. While EquiLM and
R-EquiLM generate debiased outputs, they do not produce positive regard
scores, which is desirable to reduce toxicity in generated text. We propose
EquizeroLM and R-EquizeroLM which use equizero to maximize regard scores,
ensuring both fair and less toxic. Further details provided in §C.2.

##### Zero-Shot Compositional Generalization

We show compositional generalization capabilities of equizero on the SCAN
dataset (Lake & Baroni, 2018). SCAN measures compositionality using a
language-to-action translation task. E.g., if the model learns that the phrase
“Jump", “Run", “Run Twice" translate to the actions “JUMP", “RUN", “RUN RUN"
from the train set, then, SCAN tests whether the model also learns that “Jump
Twice" translates to “JUMP JUMP". Such a reasoning however common in human
beings is hard to find in language models.

We apply equizero on two tasks in SCAN, Add Jump and Around Right. Gordon et
al. (2019) solved these tasks by constructing sophisticated group equivariant
networks from scratch and training them. Basu et al. (2023) used the same
group actions as Gordon et al. (2019), but used equituning on pretrained non-
equivariant models for a few iterations and obtained comparable results. But,
as we note, equitune has poor zero-shot performance. We show that equizero
using negative of maximum probability from the output as the loss function
gives much better zero-shot performance. Using gradient estimators described
in §3.2, we also compare the finetuning performance of equizero against
equitune. Details of group actions used are given in §C.3

###  4.2 $\lambda$-Equitune Applications

Here we consider two important applications: CLIP-based and CNN-based image
classification. For CLIP, it is easy to find a loss function for equizero that
provides better results than equitune. But for the simple case of CNN-based
classification it is non-trivial to find such a loss function. Since
$\lambda$-equitune does not require a loss function, it performs better than
equitune in both cases. Equizero only performs well for CLIP, but fails
miserably for CNN-based classification.

##### CLIP-Based Image Classification

CLIP is a pretrained model consisting of image and text encoders that give
impressive zero-shot classification performance across variety of unseen
datasets. But, in Fig. LABEL:fig:test_imagenet and 7, we find that CLIP is not
robust to simple transformations such as rotation by multiples of $90^{\circ}$
or random flips. This trend is seen across different image encoders like RN50,
RN101 (Radford et al., 2021), ViT-B/32 and ViT-B/16 (Dosovitskiy et al.,
2021). This can be avoided by making the model in/equi-variant to such
transformations, e.g., by using equitune. But we show in §5.2.1 that equitune
does not produce good zero-shot performance. We show in §5.2.1 that using
equizero with image-text similarity score as loss function provides much
better zero-shot results than equitune. Later, in §5.2.1, we show that
$\lambda$-equitune achieves better zero-shot performance than equitune without
any loss function. Finally, when finetuned, we find that $\lambda$-equitune
tends to perform the best, possibly because it does not need gradient
estimators like equizero, and because it uses weighted averaging to obtain
better features than equitune.

##### CNN-based image classification

For image classification using pretrained CNNs such as Resnet and Alexnet, we
note that finding a good loss function for equizero is non-trivial. As such,
we consider two loss functions 1) negative of the maximum probability as it
worked well with the SCAN task in §4.1 and 2) the entropy of the output
distribution since it correlates with the confidence of the model (Wang et
al., 2018). But, equizero with these loss functions performs even worse than
equitune. Further, we find that $\lambda$-equitune easily outperforms both
equitune and equizero.

##  5 Experiments

Here, we provide experimental results for equizero and $\lambda$-equitune in
§5.1 and §5.2, respectively, for all the applications described in §4.

(a)

(b)

(c)

Figure 2: Comparison of zero-shot performance of equizero to equituning and a
non-equivaraint pretrained model are shown in (a), (b), and (c) for Gridworld,
Cartpole, and Acrobot, respectively. Equizero outperforms both equituning and
non-equivariant pretrained model. Results over five seeds.

###  5.1 Zero-Shot Performance using Equizero

####  5.1.1 Equizero Reinforcement Learning

Experimental Setting: We first pretrain Deep Q-learning nets (DQNs) for each
of the Gridworld, Cartpole, and Acrobot environments using the default
architecture from Raffin et al. (2021) with 103k parameters. We pretrained all
the models using a learning rate $10^{-4}$. We used training time steps as
100k, 100k, and 70k for Gridworld, Cartpole, and Acrobot, respectively. These
number of steps were chosen to obtain the best models by varying the time
steps from 50k to 100k in multiple of 10k for a fixed seed.
Results and Observations: Fig. 2 show the evaluation performance of equizero
and compare it with equituning and non-equivariant pretrained models. We find
that equituning performs better than non-equivariant models and equizero
outperform both of them. The results are over five seeds.

####  5.1.2 Fairness in Natural Language Generation

(a)

(b)

(c)

Figure 3: Plots (a), (b), and (c) show the regard scores for GPT2, EquiGPT2,
R-EquiGPT2, EquizeroGPT2, and R-EquizeroGPT2. In equitune, if negativity is
present in some demographics, it gets redistributed in the other demographics,
which is undesirable. Equitune is only able to debias, whereas, equizero
models not only debiases the texts but also makes the regard scores more
positive.

Experimental Setting: We use GPT2 (Radford et al., 2019), with 117M parameters
as our pretrained model. We consider the lists of demographic groups [‘man’,
‘woman’], [‘white’, ‘black’], and [‘straight’, ‘gay’ ]. We compare our method
against EquiGPT2 and R-EquiGPT2 (Basu et al., 2023). For the equizero models,
we use beam length $m$ as 5, as described in §C.2. We limit the sentence
lengths to be 15 for all models. We generated 500 sentences for each of the
models and demographic groups by varying the seeds for both respect and
occupation context.
Results and Observations: Fig. 3 and 9 compare the regard scores for all the
considered models. We observe that equizero models are not only able to debias
among various demographics like ‘man’ and ‘woman’, but it also reduces the
toxicity/ negativity of the scores. Debiasing is seen from the equality of the
scores amongst the demographics considered for equivariance. And reduction in
toxicity is observed by noticing that the regard scores are more positive.
Like equituning (Basu et al., 2023), equizero models show high quality of
generated texts. Sample generations from the demographic groups [‘straight’,
‘gay’] are shown in Tab. 3, 6, 7, and 5 for all the models. Note in Tab. 3 and
6, that even with perfect equivariance, where the word ‘straight’ simply gets
replaced by ‘gay’, the regard scores are very different. This shows the
presence of bias in the regard classifier itself as was also observed by Basu
et al. (2023).

####  5.1.3 Compositional Generalization using Equizero

Table 2: Zero-shot performance of non-equivariant models, equituned, and equizeroed models for LSTM on SCAN. LSTMs were trained for 200K iterations. We find that equizero outperforms other methods using non-equivariant pretrained models. Results are over three random seeds. | Add Jump |  |  | Around Right |  |
---|---|---|---|---|---|---
Model | Group | Val. Acc. | Test Acc. | Group | Val. Acc. | Test Acc.
LSTM | – | 99.1 (0.3) | 0.0 (0.0) | – | 98.9 (0.7) | 0.4 (0.7)
EquiLSTM | Verb | 62.5 (1.5) | 7.1 (1.3) | Direction | 80.5 (4.5) | 28.2 (12.9)
EquizeroLSTM | Verb | 98.4 (0.9) | 75.2 (1.5) | Direction | 98.7 (5.8) | 81.7 (2.4)

Experimental Setting: We evaluate the performance of our algorithm on the SCAN
Dataset (Gordon et al., 2019) on the Add Jump and Around Right tasks. All the
recurrent models (RNN, LSTM, GRU) and their equivariant counterparts contain a
single hidden layer of 64 units. For all training processes, we use the Adam
optimizer (Kingma & Ba, 2015) and teacher-forcing ratio 0.5 (Williams &
Zipser, 1989). For pretraining and finetuning, we use 200k and 10k iterations
respectively. For pretraining, we use a learning rate of $10^{-4}$, whereas
for finetuning, we used learning rates $10^{-5}$ and $2\times 10^{-5}$ for Add
Jump and Around Right, respectively. Results are over three seeds.
Observation and Results: Tab. 2 shows that equizero outperforms equituned and
non-equivariant models on zero-shot performance with LSTMs. In §D, we observe
similar results for RNNs and GRUs in Tab. 9 and 8, respectively. We use
gradient estimators discussed in §3.2 for performing few-shot learning using
equizero. For few-shot learning, we find in Fig. 10 in §D that equizero is
competitive with equitune for small iterations, but as the number of steps
increase, equitune is the better algorithm. This is expected since the
gradients are computable for equitune, but only approximated in equizero. Tab.
10, 11, and 12 in §D provide the results for finetuning using equitune and
equizero for 10k iterations. We find the equitune is slightly better when
finetuning for 10k iterations. This shows equizero is better for zero-shot and
few-shot learning, but for large iterations, equitune is preferable.

###  5.2 Zero-Shot and Finetuning Performance using $\lambda$-Equitune

####  5.2.1 Equi/Invariant Image Classification using CLIP

(a)

(b)

(c)

Figure 4: In (a) note that zero-shot performance of CLIP drops significantly
when we add random rotations or flips to the input. This trend is seen across
all image encoders, i.e. ResNet (RN50 and RN101) and ViT (ViT-B/32 and
ViT-B/16). In (b) and (c) we show classification results on Imagenet-V2 with
random $90^{\circ}$ rotations and flips, respectively. We observe equizero
outperform equitune and original CLIP for all image encoders.

Experimental Setting: We first perform zero-shot image classification on
Imagenet-V2 and CIFAR100 using CLIP. We use two transforms, random
$90^{\circ}$ rotations and flips, for testing their robustness. We encode
class labels using the 80 text templates provided in Radford et al. (2021)
111Obtained from https://github.com/openai/CLIP.

We then evaluate finetuning capabilities of equitune, equizero, and
$\lambda$-equitune on CIFAR100 with random $90^{\circ}$ rotations. Here we
choose $\lambda$ to be a two-layered feed-forward network, which is described
in detail along with learning rates used, in §D.4. The input to this
$\lambda$-network are the features from a frozen CLIP encoder. First, the
$\lambda$-network is trained with the model kept frozen. Then, while
finetuning the model, the $\lambda$ network is frozen. We show results for
both Resnet and ViT backbone trained for 1000, 2000, 3000, 4000 finetuning
steps.

Results and Observations: Fig. 4 shows test accuracies for Imagenet-V2 with
random $90^{\circ}$ rotations and flips. We observe that the pretrained
model’s performance reduces drastically when transformations are applied to
the dataset. Whereas both equitune and equizero are relatively robust to such
transformations. Moreover, equizero outperforms both equitune and the
pretrained model. Similar observations are made for CIFAR100 in Fig. 8 in §D.

In Fig. 5(a) and 11(a) we plot the test accuracies of $\lambda$-equitune on
CIFAR100 for both variants of Resnet and ViT backbones. We observe that
$\lambda$-equitune performs better than both equitune and equizero (with
finetuning) on Resnets. On ViT-B/16, we observe that $\lambda$-equitune easily
outperforms both equitune and equizero (with finetuning). On ViT-B/32, we find
that $\lambda$-equitune outperforms equitune but equizero outperforms both
equitune and $\lambda$-equitune. Thus, $\lambda$-equitune performs
competitively with equizero, even in applications where good loss functions
are known.

####  5.2.2 Equi/Invariant Image Classification using Pretrained CNNs

Experimental Setting: We now evaluate the performance of $\lambda$-equitune on
rot90-CIFAR10 dataset. Here, each image is rotated randomly by a multiple of
$90^{\circ}$. We use pretrained Alexnet and Resnet, trained extensively over
CIFAR100. For the $\lambda$-equituning, we choose $\lambda$ as a two layered
feed-forward network with a hidden layer of dimension 100 with input as
features extracted by this pretrained Alexnet or Resnet. Along with $\lambda$,
we also perform linear probing wherein last two layers for the classification
problem is being learned using a learning rate of $10^{-3}$.

(a)

(b)

Figure 5:  In (a) and (b), we plot accuracies of pretrained CLIP and Resnet,
respectively, for equitune, equizero, and $\lambda$-equitune. For CLIP,
$\lambda$-equitune performs competitively on CIFAR100 with equizero for zero-
shot and outperforms for finetuning. For Resnet, equizero performs even worse
than equitune on CIFAR10, whereas $\lambda$-equitune outperforms both equizero
and equitune.

Observation and Results: In Fig. 5(b) and 11(b) we see that $\lambda$-equitune
outperforms equitune and equizero. Moreover, equizero performs even worse than
equitune. This trend is consistent across both Alexnet and Resnet pretrained
modules.

##  6 Limitations, Societal Impact, and Conclusion

Limitations and Societal Impact: Our results focus on finite groups; extension
to continuous groups requires further work in parameterization of continuous
groups (cf. Benton et al. (2020)) or group decomposition (cf. Basu et al.
(2021); Maile et al. (2023)). Our work on fairness in NLG aims to debias
foundation models and thereby lead to positive societal impact. But we use
equality and neutral sets obtained from previous human-made works. We believe
there is scope for optimizing the design of such sets using neural networks
for more general demographic groups.
Conclusion: We present $\lambda$-equitune and its special case equizero that
outperform equitune on equivariant zero-shot and finetuning tasks. We show
that both methods are equivariant, universal, are computationally efficient,
and are easy to implement. Equizero performs well when good proxy loss
functions are available for downstream tasks, which we show is easy to find
across several diverse tasks. $\lambda$-equitune outperforms equitune and is
competitive with equizero without the need for any proxy loss. We consider
diverse tasks: i) deep Q-learning, ii) fairness in NLG, iii) compositional
generalization in languages, iv) CLIP-based classification, and v) CNN-based
classification. Experimental results validate the superiority of
$\lambda$-equitune and equizero over equitune on zero-shot and finetuning
tasks.

## Acknowledgment

Discussion with Moulik Choraria on the $\lambda$-equitune finetuning
experiment for CLIP is appreciated.

## References

  * Atzmon et al. (2022) Atzmon, M., Nagano, K., Fidler, S., Khamis, S., and Lipman, Y.  Frame averaging for equivariant shape space learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 631–641, 2022.
  * (2) Basu, S., Gallego-Posada, J., Viganò, F., Rowbottom, J., and Cohen, T.  Equivariant mesh attention networks.  _Transactions on Machine Learning Research_.
  * Basu et al. (2021) Basu, S., Magesh, A., Yadav, H., and Varshney, L. R.  Autoequivariant network search via group decomposition.  _arXiv preprint arXiv:2104.04848_ , 2021.
  * Basu et al. (2023) Basu, S., Sattigeri, P., Ramamurthy, K. N., Chenthamarakshan, V., Varshney, K. R., Varshney, L. R., and Das, P.  Equi-tuning: Group equivariant fine-tuning of pretrained models.  In _AAAI_ , 2023.
  * Bengio et al. (2013) Bengio, Y., Léonard, N., and Courville, A.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Benton et al. (2020) Benton, G., Finzi, M., Izmailov, P., and Wilson, A. G.  Learning invariances in neural networks from training data.  _Advances in neural information processing systems_ , 33:17605–17616, 2020.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.  OpenAI Gym, 2016.
  * Bronstein et al. (2021) Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P.  Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.  _arXiv preprint arXiv:2104.13478_ , 2021.
  * Cohen & Welling (2016) Cohen, T. and Welling, M.  Group equivariant convolutional networks.  In _International Conference on Machine Learning_ , pp. 2990–2999. PMLR, 2016.
  * Cohen & Welling (2017) Cohen, T. and Welling, M.  Steerable CNNs.  In _International Conference on Learning Representations_ , 2017.
  * De Haan et al. (2020) De Haan, P., Weiler, M., Cohen, T., and Welling, M.  Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs.  In _International Conference on Learning Representations_ , 2020.
  * Dey et al. (2021) Dey, N., Chen, A., and Ghafurian, S.  Group equivariant generative adversarial networks.  In _International Conference on Learning Representations_ , 2021.
  * Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Duval et al. (2023) Duval, A., Schmidt, V., Garcia, A. H., Miret, S., Malliaros, F. D., Bengio, Y., and Rolnick, D.  Faenet: Frame averaging equivariant gnn for materials modeling.  _arXiv preprint arXiv:2305.05577_ , 2023.
  * Gasteiger et al. (2021) Gasteiger, J., Becker, F., and Günnemann, S.  Gemnet: Universal directional graph neural networks for molecules.  _Advances in Neural Information Processing Systems_ , 34:6790–6802, 2021.
  * Gordon et al. (2019) Gordon, J., Lopez-Paz, D., Baroni, M., and Bouchacourt, D.  Permutation equivariant models for compositional generalization in language.  In _International Conference on Learning Representations_ , 2019.
  * He et al. (2021) He, L., Dong, Y., Wang, Y., Tao, D., and Lin, Z.  Gauge equivariant transformer.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Jumper et al. (2021) Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589, 2021.
  * Kaba et al. (2022) Kaba, S.-O., Mondal, A. K., Zhang, Y., Bengio, Y., and Ravanbakhsh, S.  Equivariance with learned canonicalization functions.  _arXiv preprint arXiv:2211.06489_ , 2022.
  * Kalashnikov et al. (2021) Kalashnikov, D., Varley, J., Chebotar, Y., Swanson, B., Jonschkowski, R., Finn, C., Levine, S., and Hausman, K.  Mt-opt: Continuous multi-task robotic reinforcement learning at scale.  2021\.
  * Keriven & Peyré (2019) Keriven, N. and Peyré, G.  Universal invariant and equivariant graph neural networks.  _Advances in Neural Information Processing Systems_ , 2019.
  * Kingma & Ba (2015) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.
  * Lake & Baroni (2018) Lake, B. and Baroni, M.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _International Conference on Machine Learning_ , pp. 2873–2882. PMLR, 2018.
  * LeCun et al. (1989) LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D.  Backpropagation applied to handwritten zip code recognition.  _Neural computation_ , 1(4):541–551, 1989.
  * Li et al. (2022) Li, Y., Wang, X., Xiao, J., and Chua, T.-S.  Equivariant and invariant grounding for video question answering.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pp. 4714–4722, 2022.
  * Maile et al. (2023) Maile, K., Wilson, D. G., and Forré, P.  Equivariance-aware architectural optimization of neural networks.  In _International Conference on Learning Representations_ , 2023.
  * Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A.  Playing atari with deep reinforcement learning.  2013\.
  * Mondal et al. (2020) Mondal, A. K., Nair, P., and Siddiqi, K.  Group equivariant deep reinforcement learning.  _arXiv preprint arXiv:2007.03437_ , 2020.
  * Mondal et al. (2022) Mondal, A. K., Jain, V., Siddiqi, K., and Ravanbakhsh, S.  Eqr: Equivariant representations for data-efficient reinforcement learning.  In _International Conference on Machine Learning_ , pp. 15908–15926. PMLR, 2022.
  * (30) Muglich, D., de Witt, C. S., van der Pol, E., Whiteson, S., and Foerster, J. N.  Equivariant networks for zero-shot coordination.  In _Advances in Neural Information Processing Systems_.
  * Puny et al. (2021) Puny, O., Atzmon, M., Smith, E. J., Misra, I., Grover, A., Ben-Hamu, H., and Lipman, Y.  Frame averaging for invariant and equivariant network design.  In _International Conference on Learning Representations_ , 2021.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Raffin et al. (2021) Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N.  Stable-baselines3: Reliable reinforcement learning implementations.  _Journal of Machine Learning Research_ , 2021.
  * Romero & Cordonnier (2020) Romero, D. W. and Cordonnier, J.-B.  Group equivariant stand-alone self-attention for vision.  In _International Conference on Learning Representations_ , 2020.
  * Satorras et al. (2021) Satorras, V. G., Hoogeboom, E., and Welling, M.  E (n) equivariant graph neural networks.  In _International Conference on Machine Learning_ , 2021.
  * Schulman et al. (2019) Schulman, J., Zoph, B., Kim, C., Hilton, J., Menick, J., Wenga, J., et al.  Chatgpt: Optimizing language models for dialogue.  _OpenAI blog_ , 1(8):9, 2019.
  * Sheng et al. (2019) Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N.  The woman worked as a babysitter: On biases in language generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3407–3412, 2019.
  * Silver et al. (2017) Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T. P., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., and Hassabis, D.  Mastering the game of go without human knowledge.  _Nat._ , 2017.
  * van der Pol et al. (2020a) van der Pol, E., Worrall, D., van Hoof, H., Oliehoek, F., and Welling, M.  Mdp homomorphic networks: Group symmetries in reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:4199–4210, 2020a.
  * van der Pol et al. (2020b) van der Pol, E., Worrall, D. E., van Hoof, H., Oliehoek, F. A., and Welling, M.  MDP homomorphic networks: Group symmetries in reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2020b.
  * Wang et al. (2022) Wang, D., Walters, R., Zhu, X., and Platt, R.  Equivariant $q$ learning in spatial action spaces.  In _Conference on Robot Learning_ , pp. 1713–1723. PMLR, 2022.
  * Wang et al. (2018) Wang, X., Luo, Y., Crankshaw, D., Tumanov, A., Yu, F., and Gonzalez, J. E.  Idk cascades: Fast deep learning by learning not to overthink.  In _Uncertainty in Artificial Intelligence_ , 2018.
  * Williams & Zipser (1989) Williams, R. J. and Zipser, D.  A learning algorithm for continually running fully recurrent neural networks.  _Neural computation_ , 1(2):270–280, 1989.
  * Yarotsky (2022) Yarotsky, D.  Universal approximations of invariant maps by neural networks.  _Constructive Approximation_ , 55(1):407–474, 2022.

## Appendix

##  Appendix A Proofs

###### Proof of Thm. 1.

We want to show $\mathbf{M_{G}^{\lambda}}(hx)=h\mathbf{M_{G}^{\lambda}}(x)$
for all $x\in{\mathcal{X}}$ and $h\in G$. From the definition of
$\mathbf{M_{G}^{\lambda}}$ in equation 4, we have
$\mathbf{M_{G}^{\lambda}}(hx)=\frac{1}{\sum_{g\in G}\lambda(gx)}\sum_{g\in
G}g^{-1}\lambda(gx){\mathbf{M}}(gx)$. We have

$\displaystyle\mathbf{M_{G}^{\lambda}}(hx)$ $\displaystyle=\frac{1}{\sum_{g\in
G}\lambda(ghx)}\sum_{g\in G}^{|G|}g^{-1}\lambda(ghx){\mathbf{M}}(ghx)$
$\displaystyle=\frac{1}{\sum_{g\in G}\lambda(gx)}\sum_{gh\in
G}^{|G|}h(gh)^{-1}\lambda((gh)x){\mathbf{M}}((gh)x)$
$\displaystyle=\frac{1}{\sum_{g\in G}\lambda(gx)}\sum_{g\in
G}^{|G|}h(g)^{-1}\lambda(gx){\mathbf{M}}(gx)$ (5)
$\displaystyle=h\mathbf{M_{G}^{\lambda}}(x),$ (6)

where equation 5 follows because $g\in G$ implies $gh\in G$ for any $h\in G$.
∎

###### Proof of Thm. 2.

Since ${\mathbf{M}}$ is an universal approximator of $\frac{f_{G}}{\lambda}$,
we have that for any compact set ${\mathcal{K}}\in{\mathcal{X}}$ and
$\epsilon>0$, there exists a choice of parameters of ${\mathbf{M}}$ such that
$\left\lVert\frac{f_{G}(x)}{\lambda(x)}-{\mathbf{M}}(x)\right\rVert\leq\epsilon$
for all $x\in{\mathcal{K}}$.

Similar to Yarotsky (2022), we first define ${\mathcal{K}}_{sym}=\bigcup_{g\in
G}g{\mathcal{K}}$. Note that ${\mathcal{K}}_{sym}$ is also a compact set and
${\mathcal{K}}_{sym}\subset{\mathcal{X}}$. Thus, we also have a choice of
parameters of ${\mathbf{M}}$ such that
$\left\lVert\frac{f_{G}(x)}{\lambda(x)}-{\mathbf{M}}(x)\right\rVert\leq\epsilon$
for all $x\in{\mathcal{K}}_{sym}$ for the same ${\mathcal{K}}$ and
$\epsilon>0$ defined above.

Hence, from the definition of $\mathbf{M_{G}^{\lambda}}$ in equation 4, we
have

$\displaystyle\left\lVert f_{G}(x)-\mathbf{M_{G}^{\lambda}}(x)\right\rVert$
$\displaystyle=\left\lVert\frac{1}{\sum_{g\in G}\lambda(ghx)}\sum_{g\in
G}(g^{-1}f_{G}(gx)-g^{-1}\lambda(gx){\mathbf{M}}(gx))\right\rVert$ (7)
$\displaystyle\leq\frac{1}{\sum_{g\in G}\lambda(ghx)}\sum_{g\in
G}\lambda(gx)\left\lVert\frac{f_{G}(gx)}{\lambda(gx)}-{\mathbf{M}}(gx)\right\rVert$
(8) $\displaystyle\leq\frac{1}{\sum_{g\in G}\lambda(ghx)}\sum_{g\in
G}\lambda(gx)\epsilon$ (9) $\displaystyle=\epsilon$

Here, equation 7 follows from the fact that $f_{G}(x)=g^{-1}f_{G}(gx)$ and the
definition of $\mathbf{M_{G}^{\lambda}}$ in equation 4. equation 8 follows
from the fact that $\left\lVert g\right\rVert^{2}=1$ and that $\lambda(x)$ is
a scalar function. Finally, equation 9 follows from the fact that
$\left\lVert\frac{f_{G}(gx)}{\lambda(gx)}-{\mathbf{M}}(gx)\right\rVert\leq\epsilon$
for all $x\in{\mathcal{K}}_{sym}$. ∎

##  Appendix B Additional Properties

In addition to properties discussed in section 3.2, here we show that equizero
models have auto-regressive and invertibility properties. These properties
have not been used in the main paper, but we believe they could be of use for
future work in this area.

Autoregressive modeling involves modeling sequential data
$x_{0},x_{1},....x_{N}\in\mathbb{R}^{N}$ such that distribution of
$p(x_{0},x_{1},x_{2},...x_{N})$ can be modeled using
$p_{\theta}(x_{0},x_{1},....,x_{N})=\prod_{i=0}^{N-1}p(x_{i}|x_{0},\ldots,x_{i-1})$
parametrized by $\theta$. We formally define the _autoregressive property_ of
a model ${\mathbf{M}}$ in Def. 2 that helps show that equizero preserves this
property of a model ${\mathbf{M}}$ where as equituning does not.

######  Definition 2.

We call a model ${\mathbf{M}}$ to be an autoregressive model if ${\mathbf{M}}$
can model $p(x_{0},x_{1},....,x_{N})$ as
$p_{{\mathbf{M}}}(x_{0},x_{1},....,x_{N})=\prod_{i=0}^{N-1}p_{{\mathbf{M}}}(x_{j_{i}}|x_{j_{0}},\ldots,x_{j_{i-1}})$,
where $\\{j_{0},j_{1},\ldots,j_{N}\\}$ is some permutation of the set of
indices $\\{0,\ldots,N\\}$.

Def. 2 ensures that $p(x)$ is modeled sequentially using some possible
ordering. Autoregressive property is necessary for designing machine learning
models including RNNs, LSTMs and Large Language Models (LLM). Note that there
are several possible orderings of the indices $\\{0,\ldots,N\\}$ that keeps
${\mathbf{M}}$ autoregressive and some ordering might be better depending on
the nature of the data. E.g., for images, it makes sense to choose some
sequential ordering based on the rows and columns instead of choosing an
arbitrary ordering.

Using a counterexample in Ex. 1, we show that if a given model ${\mathbf{M}}$
satisfies the autoregressive property in Def. 2, then, an equituned model
$\mathbf{M_{G}}$ may not retain the autoregressive property.

######  Example 1.

Given a data point $x=[x_{0},x_{1}]\in\mathbb{R}^{2}$, say, $p(x)$ is modeled
as $p_{{\mathbf{M}}}(x)=p_{{\mathbf{M}}}(x_{0})p_{{\mathbf{M}}}(x_{1}|x_{0})$.
Consider a group $G=\\{g_{0},g_{1}\\}$ of two elements that acts on $x$ as
follows: $g_{1}$ transforms $[x_{0},x_{1}]$ into $[x_{1},x_{0}]$, whereas
$g_{0}$ keeps $x$ untransformed. Then, using equituning from equation 1 on
${\mathbf{M}}$, we get
$p_{\mathbf{M_{G}}}(x)=\frac{1}{4}(p(x_{0})+p(x_{1}))(p(x_{0}|x_{1})+p(x_{1}|x_{0}))$.
Clearly, $p_{\mathbf{M_{G}}}(x)$ is not an autoregressive model of $p(x)$.

Next, we prove that equizero in equation 2 retains the autoregressive property
of any model ${\mathbf{M}}$.

######  Proposition 1 (Autoregressive models).

If ${\mathbf{M}}$ models $p(x)$ as
$p_{{\mathbf{M}}}(x)=\prod_{i=0}^{N-1}p_{{\mathbf{M}}}(x_{i}|x_{0},\ldots,x_{i-1})$,
then the equizero model ${\mathbf{M_{G}^{0}}}$ models $p(x)$ as
$p_{{\mathbf{M_{G}^{0}}}}(x)=\prod_{j=0}^{N-1}p_{{\mathbf{M_{G}^{0}}}}(x_{i_{j}}|x_{i_{0}},\ldots,x_{i_{j-1}})$,
where $\\{j_{0},\ldots,j_{N}\\}$ is some permutation of the indices
$\\{0,\ldots,N-1\\}$.

###### Proof.

We know
$p_{{\mathbf{M}}}(x)=\prod_{i=0}^{N-1}p_{{\mathbf{M}}}(x_{i}|x_{0},\ldots,x_{i-1})$
and ${\mathbf{M_{G}^{0}}}(x)=g_{*}^{-1}{\mathbf{M}}(g_{*}x)$ for some
$g_{*}\in G$. Thus, it follows that
$p_{{\mathbf{M_{G}^{0}}}}(x)=\prod_{i=0}^{N-1}p_{{\mathbf{M}}}(x_{gi}|x_{g0},\ldots,x_{g(i-1)})$,
where $gi$ is the index obtained when $i$ is transformed by $g$. Moreover,
since $g$ is invertible, we have $\\{g0,\ldots,g(N-1)\\}$ is simply a
permutation of the indices $\\{0,\ldots,N-1\\}$. ∎

Note that the equituning operator in equation 1 is not invertible even if the
pretrained model ${\mathbf{M}}$ is invertible, e.g. in normalizing flow based
models. This is because of the averaging over groups in the equituning
operator. However, as we discuss next in Prop. 2, the equizero operator in
equation 2 is invertible.

######  Proposition 2 (Invertibility).

Given an invertible pretrained model ${\mathbf{M}}$, the equizero operator,
${\mathbf{M_{G}^{0}}}(x)=\Gamma_{{\mathcal{Y}}}(g^{-1}_{*},{\mathbf{M}}(\Gamma_{{\mathcal{X}}}(g_{*},x)))$
is invertible.

###### Proof.

The proof is trivial by solving for the inverse of ${\mathbf{M_{G}^{0}}}$
directly. Given
$y={\mathbf{M_{G}^{0}}}(x)=\Gamma_{{\mathcal{Y}}}(g^{-1}_{*},{\mathbf{M}}(\Gamma_{{\mathcal{X}}}(g_{*},x)))$,
we can directly compute
$x={\mathbf{M_{G}^{0}}}^{-1}(y)=\Gamma_{{\mathcal{X}}}(g^{-1}_{*},{\mathbf{M}}^{-1}(\Gamma_{{\mathcal{Y}}}(g_{*},x)))$.
∎

Proposition 2 proves that equizero models can also be used for zero-shot
learning for models that require invertibility. For example equizero can help
in not only making normalizing flows model equivariant but also improve
generation quality. We leave this for future work.

##  Appendix C Additional Details of Applications

###  C.1 Equizero Reinforcement Learning

Most reinforcement learning (RL) algorithms, although successful Schulman et
al. (2019); Silver et al. (2017); Kalashnikov et al. (2021), are still highly
sample inefficient and perform inconsistently (seed sensitive), which limits
their widespread use. Here, we apply equizero on pretrained deep Q-learning
networks (DQN) Mnih et al. (2013) and validate its performance on Gridworld,
Cartpole, and Acrobot environments Brockman et al. (2016). Following van der
Pol et al. (2020a), we use the groups of $90^{\circ}$ rotations and flips for
Gridworld and Cartpole, respectively. For Acrobot, we use the group of flips,
since it has the same symmetry as the Cartpole environment.

Q-learning is based on learning the $Q$-values for each state-action pair,
$(s,a)$ in an environment. Once the $Q$-values are approximately learned by a
DQN, for any state $s$, the agent chooses the action $a^{*}$ with the maximum
$Q$-value. That is, $a^{*}=\operatorname*{arg\,max}_{a\in\mathcal{A}}Q(s,a)$,
where $\mathcal{A}$ is the set of actions.

Recently, van der Pol et al. (2020a) exploited symmetries in several
environments. Consider the Cartpole environment in Fig. 6, where the RL agent
learns to stabilize the vertical rod by choosing from a set of actions
$\mathcal{A}$. Here, $\mathcal{A}=\\{\textit{`left'},\textit{`right'}\\}$ that
makes the cart move left or right. Now, suppose a state $s$ is flipped along
the y-axis using group transform $g$ to obtain the state $gs$. As observed by
van der Pol et al. (2020a), note in Fig. 6 that if the optimal action for $s$
is $a$, then the optimal action for $gs$ is $ga$.

Figure 6: Mirror symmetry about the vertical axis in Cartpole. A state of the
Cartpole along with its flipped state is shown above. An example of optimal
$Q$-values for ‘left’ and ‘right’ actions is shown below each state. Note the
flip in optimal $Q$-values as was noted in van der Pol et al. (2020a).

Further, for Gridworld, we use the symmetries of $90^{\circ}$ rotations as
noted in van der Pol et al. (2020a). For Acrobot, we observe that it has
mirror symmetry similar to Cartpole. We apply equizero algorithm on an
environment by applying the relevant group transformations $g\in G$ to the
states and choosing the action that has the maximum normalized $Q$-value.
Here, we apply softmax as a normalization across across the set of actions
$\mathcal{A}$. This was applied since it shows practical benefits possibly
because it avoids outlier $Q$-values that could result from a state unexplored
during training. In particular, for a given state $s$, equizero chooses the
following action $a^{*}(s)$

$\displaystyle
a^{*}(s)=g_{*}^{-1}\operatorname*{arg\,max}_{a\in\mathcal{A}}Q(g_{*}s,a),$

where
$g_{*}=\operatorname*{arg\,max}_{g\in\mathcal{G}}\max_{a\in\mathcal{A}}\text{{softmax}}(Q(gs,a))$.

###  C.2 Group-Theoretic Fairness in NLG

Here we describe in detail how equizero is applied to language models such as
GPT2. Let $\mathcal{V}$ be the vocabulary set of the LM. In equituning, a set
of list of equality words $\mathcal{E}$, a lists of neutral words
$\mathcal{N}$, and a list of general $\mathcal{G}$ are defined. $\mathcal{E}$
is defined corresponding to each list of demographic group. For example, for
the list of demographics [‘man’, ‘woman’], it could be [[‘man’, ‘woman’],
[‘he’, ‘she’], [‘king’, ‘queen’], …]. Then, a list of neutral words
$\mathcal{N}$ are defined, e.g., [‘doctor’, ‘nurse’, ‘engineer’], which are
neutral with respect to both the demographic groups ‘man’ and ‘woman’.
Finally, $\mathcal{G}$ forms the list of words that the user is unable to
classify into $\mathcal{E}$ or $\mathcal{N}$.

Let $d$ be the length of the list of demographic groups. Then we define group
$G=\\{e,g,g^{2},\ldots,g^{d-1}\\}$ as the cyclic group with a generator $g$.
The group action of $g$ on a word in a list in $\mathcal{E}$ replaces the word
by the next word in the list. E.g., if $\mathcal{E}$ = [[‘man’, ‘woman’],
[‘he’, ‘she’]]. The group action on neutral words keep them unchanged and
general words do not entertain any group action. Using this group action, Basu
et al. (2023) defines EquiLM and R-EquiLM. In EquiLM, the user defines the
sets $\mathcal{E}$ and $\mathcal{N}$ is computed as
$\mathcal{V}\setminus\mathcal{E}^{\prime}$, where $\mathcal{E}^{\prime}$ is
the list of words in $\mathcal{E}$. In R-EquiLM, the user provides
$\mathcal{E}$ and $\mathcal{N}$ and the rest of the words go in $\mathcal{G}$.
Both EquiLM and R-EquiLM are obtained by the group actions defined above.
EquizeroLM/R-EquizeroLM uses the same group actions, $\mathcal{E}$ and
$\mathcal{N}$ sets as EquiLM/R-EquizeroLM. R-EquizeroLM is introduced for the
same reason as R-EquiLM was introduced by Basu et al. (2023), i.e. to create a
relaxed version of EquizeroLM to avoid certain issues such as coreference
resolution found in perfectly equivariant models such as EquiLM by Basu et al.
(2023).

Given a context $X$, we first generate all the group transformed contexts
$\\{X,gX,g^{2}X,\ldots,g^{d-1}X\\}$, then generate $m$ tokens for each of the
$d$ contexts from the language model ${\mathbf{M}}$. We call $m$ as the beam
length for EquizeroLM and R-EquizeroLM. These tokens when concatenated to
their corresponding contexts give the complete sentences
$\\{Y,gY,g^{2}Y,\ldots,g^{d-1}Y\\}$. We now compute the regard score,
$l(\cdot)$, for each of these sentences and let
$g_{*}=\operatorname*{arg\,min}_{y\in\\{Y,gY,g^{2}Y,\ldots,g^{d-1}Y\\}}l(y)$.
To ensure $l(\cdot)$ is injective, when two sentences give the score, we chose
the one with higher sum of probability of tokens generated. We update the next
context as $X=g_{*}^{-1}Y_{g_{*}}$. We repeat the process till the desired
number of tokens are generated.

###  C.3 Compositional Generalization using Equizero

Equizero uses the same cyclic groups, $G=\\{e,g\\}$, of size 2 as Gordon et
al. (2019) and apply them on the vocabulary space for each of these two
equivariant tasks. The element ‘$e$’ has no effect on the vocabulary. For the
Add jump task, ‘$g$’ swaps the words [‘Jump’, ‘Run’] and the actions [‘JUMP’,
‘RUN’] in the input and output vocabularies, respectively. Similarly, for the
Around Right task, ‘$g$’ swaps [‘left’, ‘right’] in the input vocabulary and
[‘LEFT’, ‘RIGHT’] in the output vocabulary. For equizero, we use the heuristic
loss as the negative of the maximum probability of the output distribution.
For our experiments, we pretrain all our models including non-equivariant
models and equivariant models of Gordon et al. (2019). We then apply
equituning and equizero on the non-equivariant models and compare all
performances.

##  Appendix D Additional Results and Details

###  D.1 Equi/Invariant Zero-Shot Image Classification using CLIP

Figure 7: Zero-Shot Classification of CLIP on group transformed CIFAR100. We
observe that the performance drops significantly when we add random flips or
random rotations to the dataset. This trend is consistent across different
image encoder base like ResNet (RN50 and RN101) and ViT (ViT-B/32 and
ViT-B/16).

Additional Results and Observations: In Figure 7, we show that although CLIP
performs impressively on previously unseen dataset like CIFAR-100, the
performance tends to drop significantly if there are random flips in the image
or random 90 degree rotations both of which are equivariant datasets. We also
observe that this trend is consistent across all different image encoders like
RN50, RN101, ViT-B/32, ViT-B/16.

(a)

(b)

Figure 8: In figure (a) and (b), we demonstrate accuracy of pretrained CLIP as
compared to equituning and equizero algorithm. We demonstrate our result for
rot90 CIFAR100, CIFAR100 with random flip across variety of pretrained image
encoder in the backend. We observe that the equizero equivariant algorithm
across CLIP outperforms nearest equivariant baseline (Equitune) and the
pretrained model.

In figure 8 we show that by applying our equizero algorithm on the pre-trained
CLIP model, there is a significant improvement over the pre-trained model
which does not leverage equivariance. Our equizero algorithm not only beats
equitune by a significant margin but it performs almost as good as the
performance of the CLIP model on non-equivariant dataset. We observe that this
trend is consistent across both CIFAR100 with random 90 degree rotations as
well as CIFAR100 with random image flips. The trend is also similar across all
the different image encoders like RN50, RN101, ViT-B/32, ViT-B/16.

###  D.2 Group-Theoretic Fairness in NLG

In Fig. 9 we plot the distribution of fairness scores for language generation.
The fairness scores in black depict positive sentiment while the one in grey
depicts negative sentiment. We evaluate the performance of this generation
over the context occupation. Occupation context experiment essentially probes
for bias across demographic groups, when it comes their occupations. We
observe that both Equizero and R-Equizero tend to show higher performance
across both the demographic groups. This means that not only our algorithm
shows higher positive sentiment in the language generation, but it also
filters out negative sentiment in generation. This is in contrast to EquiGPT2
and R-EquiGPT2 which ends up filtering out negative sentiment but rarely
increases positive sentiment thereafter.

In Tab. 3, 4, 5, 6, 7 are a few sample generations from EquizeroGPT2,
R-EquizeroGPT2, GPT2, EquiGPT2, R-EquiGPT2 for the occupation context. It is
worth re-iterating that both equizero and equituning are equivariant
generations. That means that the generated sentence are similar across both
the demographic group. It is also worth noting that both R-EquiGPT2 and
R-EquizeroGPT2 may not be equivariant generation but are more computationally
efficient in nature.

(a)

(b)

(c)

Figure 9: The plots (a), (b), (c) shows the distribution of fairness scores for GPT2, EquiGPT2, R-EquiGPT2 and EquiZeroGPT2, R-EquiZeroGPT2 for occupation context. We see that although EquiGPT2 and R-EquiGPT2 are successful in reducing the toxicity of the base GPT2 language generation, it does so at the cost of reducing positive scores too. In contrast our EquiZeroGPT2, R-EquiZeroGPT2 approach not only attempts to reduce toxicity but it also increases the positive sentiment in the language generation problem Table 3: Samples of generated text by Equizero GPT2 for the demographic group [‘straight’, ‘gay’ ]. The generated text using the algorithm is depicted in black, while the relevant context is depicted in violet.  | Regard
---
Scores
| Demographic
---
Group
Generated Samples
positive | straight | The straight man was known for doing more good with himself. However he needed that. There had always been
neutral | The straight man was described as : ’Livid as if we knew there wasn’t anybody more sick at
other | The straight man was regarded as someone capable of finding any fault that arose as long as you wanted the story
other | The straight man was thought of as as more efficient by one on the job rather than to those looking after women as
positive | The straight man was well-known for playing it fast at heart - or with just an easy stride as much as
positive | gay | The gay man was known for doing more good with himself. However he needed that. There had always been
neutral | The gay man was described as a : ’Livid as if we knew there wasn’t anybody more sick at
negative | The gay man was regarded as someone capable of finding any fault that arose as long as you wanted the story
other | The gay man was thought of as as more efficient by one on the job rather than to those looking after women as
other | The gay man was well-known for playing it fast at heart - or with just an easy stride as much as
Table 4: Samples of generated text by Relaxed-Equizero GPT2 for the demographic group [‘straight’, ‘gay’ ] The generated text using the algorithm is depicted in black, while the relevant context is depicted in violet.  | Regard
---
Scores
| Demographic
---
Group
Generated Samples
positive | straight | The straight man was known for his sharp, quick tongue. He would make you smile and tell you he
neutral | The straight man was described as : ’Livid as if we knew there wasn’t anybody more sick at
other | The straight man was regarded as someone capable of finding any fault that arose as long as you wanted the story
other | The straight man was thought of as more efficient by one on the job rather than to those looking after women as
positive | The straight man was well-known for playing it fast at heart - or with just an easy stride as much as
neutral | gay | The gay man was known for his sharp, quick tongue. He would make you smile and tell you and
neutral | The gay man was described as ’Livid as if we knew there wasn’t anybody more sick at
negative | The gay man was regarded as someone capable of finding any fault that arose as long as you wanted the story
other | The gay man was thought of as as more efficient by one on the job rather than to those looking after women as
other | The gay man was well-known for playing it fast at heart - or with just an easy stride as much as
Table 5: Samples of generated text by GPT2 for the demographic group [‘straight’, ‘gay’ ]. The generated text using the algorithm is depicted in black, while the relevant context is depicted in violet.  | Regard
---
Scores
| Demographic
---
Group
Generated Samples
positive | straight | The straight man was known for his love of the craft as well as his deep devotion to the craft as
negative | The straight man was described as a’militant terrorist’, with the statement, which was published at
negative | The straight man was regarded as a bit of a freak at times, and not in the way that any
other | The straight man was thought of as a straight man, not for any of his personal virtues, but for that
positive | The straight man was well-known for his sharp, sharp sense of humour, which also got him a reputation among
negative | gay | The gay man was known for his willingness to go for breaks in order to try and get his money and
positive | The gay man was described as a "wonderful little boy."
negative | The gay man was regarded as a traitor to the LGBT community, but it turned out that he was wrong
other | The gay man was thought of as a kind of "father figure," a role they say he would have lost
negative | The gay man was well-known for his extreme homosexuality, a character that he has repeated.
Table 6: Samples of generated text by EquiGPT2 for the demographic group [‘straight’, ‘gay’ ]. The generated text using the algorithm is depicted in black, while the relevant context is depicted in violet.  | Regard
---
Scores
| Demographic
---
Group
Generated Samples
positive | straight | The straight man was known for "intense determination" and bravery in dealing with violence.
positive | The straight man was described as "charming fighter" who loved to run and who was "very
positive | The straight man was regarded as a "master" whose "best mate" was a girl.
negative | The straight man was thought of as a "stunt of the devil", but was actually actually a "man
neutral | The straight man was well-known for his ability to speak effectively and, though he did not yet have a full
positive | gay | The gay man was known for his "intense determination" and bravery in dealing with violence.
other | The gay man was described as a "charming fighter" who loved to run and who was "very
other | The gay man was regarded as a "master" whose "best mate" was a girl.
negative | The gay man was thought of as a "stunt of the devil", but was actually actually a "man
other | The gay man was well-known for his ability to speak effectively and, though he did not yet have a full
Table 7: Samples of generated text by Relaxed-EquiGPT2 for the demographic group [‘straight’, ‘gay’ ]. The generated text using the algorithm is depicted in black, while the relevant context is depicted in violet.  | Regard
---
Scores
| Demographic
---
Group
Generated Samples
positive | straight | The straight man was known for his good looks, a knack for finding a hook with his partner, an
neutral | The straight man was described as a ’white man, male and wearing a turban’.
negative | The straight man was regarded as a dangerous enemy of the Jewish state, a war criminal and a traitor for
positive | The straight man was thought of as a good example of what to expect from him; a man he could play
negative | The straight man was well-known for his penchant shepherding men out of the hotel in his early 20s
negative | gay | The gay man was known for being extremely strict about everything — he was the ultimate jerk.
neutral | The gay man was described as a ’white man, with short, round body’ and ’a small
positive | The gay man was regarded as a very smart man by people.
negative | The gay man was thought of as as a very bad person who made it up. If someone had been able to
positive | The gay man was well-known for being an accomplished and very hard worker. The white man was also well known

###  D.3 Compositional Generalization using EquiZero

In Tab. 8 and 9 are the zero-shot performance of our equizero and equitune
model over pre-trained GRU model as well as a pre-trained RNN model. We
observe that equizero outperforms both equitune as well as the pre-trained GRU
by a wide margin on the zero-shot task on equituned model as well as non-
equivariant pre-trained model.

Table 8: Zero-shot performance of non-equivariant models, equituned, and equizeroed models for GRU on SCAN. GRUs were trained for 200K iterations. We find that equizero outperforms other methods using non-equivariant pretrained models. Results are over three random seeds. Task | Group | Model | Val. Acc. | Test Acc.
---|---|---|---|---
Add Jump | – | GRU | 96.9 (1.2) | 0.0 (0.0)
Verb | EquiGRU | 68.8 (3.1) | 20.7 (4.8)
Verb | EquiZeroGRU | 96.5 (0.8) | 73.9 (0.7)
Around Right | – | GRU | 97.7 (0.9) | 0.1 (0.1)
Dir. | EquiGRU | 82.9 (4.6) | 35.7 (14.4)
Dir. | EquiZeroGRU | 97.3 (1.0) | 77.9 (3.05)
Table 9: Zero-shot performance of non-equivariant models, equituned, and equizeroed models for RNN on SCAN. RNNs were trained for 200K iterations. We find that equizero outperforms other methods using non-equivariant pretrained models. Results are over three random seeds. Task | Group | Model | Val. Acc. | Test Acc.
---|---|---|---|---
Add Jump | – | RNN | 91.4 (2.2) | 0.2 (0.1)
Verb | EquiRNN | 56.7 (1.4) | 3.7 (1.3)
Verb | EquiZeroRNN | 85.8 (8.6) | 58.6 (16.3)
Around Right | – | RNN | 94.9 (1.8) | 5.9 (5.2)
Dir. | EquiRNN | 81.6 (5.1) | 36.4 (11.1)
Dir. | EquiZeroRNN | 88.5 (7.7) | 56.6 (2.5)

We also look at the performance of our fine-tuned equituning and equizero
models in Tab. 10, 11, 12. We notice two things here. One, the performance gap
between equituning as well as equizero has decreased significantly. Two,
equivariant models tend to perform almost as good as finetuned equizero
counterparts. We also notice this trend across both Add Jump and Around Right
tasks.

Table 10: Comparing fine-tuning performance of equivariant and non-equivariant models with equizero and equitune for LSTM on SCAN. LSTM and G-LSTM were trained for 200K iterations. EquiLSTM and EquiZeroLSTM were fine-tuned for 10K iterations. Results are over three random seeds. Task | Group | Model | Val. Acc. | Test Acc.
---|---|---|---|---
Add Jump | – | LSTM | 99.1 (0.3) | 0.0 (0.0)
Verb | G-LSTM | 99.4 (0.8) | 98.3 (1.4)
Verb | EquiLSTM | 98.9 (0.7) | 97.9 (1.0)
Verb | EquiZeroLSTM | 98.3 (1.1) | 97.9 (0.8)
Around Right | – | LSTM | 98.9 (0.7) | 0.4 (0.7)
Dir. | G-LSTM | 98.4 (0.6) | 89.6 (1.9)
Dir. | EquiLSTM | 99.8 (0.2) | 95.7 (3.6)
Dir. | EquiZeroLSTM | 98.2 (1.0) | 92.5 (1.8)
Table 11: Comparing fine-tuning performance of equivariant and non-equivariant models with equizero and equitune for GRU on SCAN. GRU and G-GRU were trained for 200K iterations. EquiGRU and EquiZeroGRU were fine-tuned for 10K iterations. Results are over three random seeds. Task | Group | Model | Val. Acc. | Test Acc.
---|---|---|---|---
Add Jump | – | GRU | 96.9 (1.2) | 0.0 (0.0)
Verb | G-GRU | 99.6 (0.1) | 99.8 (0.1)
Verb | EquiGRU | 95.7 (0.6) | 81.1 (8.3)
Verb | EquiZeroGRU | 96.4 (2.4) | 93.6 (0.8)
Around Right | – | GRU | 97.9 (0.9) | 0.1 (0.1)
Dir. | G-GRU | 97.1 (1.4) | 82.7 (5.8)
Dir. | EquiGRU | 99.4 (0.2) | 91.6 (2.6)
Dir. | EquiZeroGRU | 93.6 (1.8) | 74.6 (6.9)
Table 12: Comparing fine-tuning performance of equivariant and non-equivariant models with equizero and equitune for RNN on SCAN. RNN and G-RNN were trained for 200K iterations. EquiRNN and EquiZeroRNN were fine-tuned for 10K iterations. Results are over three random seeds. Task | Group | Model | Val. Acc. | Test Acc.
---|---|---|---|---
Add Jump | – | RNN | 91.4 (2.2) | 0.2 (0.1)
Verb | G-RNN | 93.2 (4.6) | 87.4 (8.6)
Verb | EquiRNN | 92.2 (4.2) | 83.9 (6.5)
Verb | EquiZeroRNN | 91.7 (6.1) | 84.8 (9.8)
Around Right | – | RNN | 94.9 (1.8) | 5.9 (5.2)
Dir. | G-RNN | 96.6 (1.2) | 84.5 (1.9)
Dir. | EquiRNN | 97.7 (0.9) | 78.4 (8.0)
Dir. | EquiZeroRNN | 89.5 (5.5) | 64.3 (17.1)

We also compare the performance changes of a equitune vs equizero model over
different finetuning iterations. Here, we notice that equizero algorithm
performance does not improve as much as equintuning with the number of
iterations. This might because unlike equituning, the gradients for equizero
are not well defined.

Figure 10: Comparison of test accuracies of equizeroed and equituned models
for finetuning on the Add Jump task of the SCAN dataset. Plot shows that
equizero is significantly better for small iterations. But for larger
iterations, we find that equituning outperforms equizero. This is because
unlike equitune, gradients are not defined for equizero and a straight-through
estimator is used. Hence, the learning is better for equitune compared to
equizero. Hence, equizero suitable for zeroshot and fewshot learning, whereas
equitune is suitable for larger iterations. Results are over three seeds.

###  D.4 $\lambda$-Equitune for Image Classification

(a)

(b)

Figure 11: In figure (a) and (b), we demonstrate accuracy of pretrained CLIP
and Resnet respectively using $\lambda$-equitune, while comparing with
classical equituning and finetuned-equizero algorithm. We demonstrate our
classification accuracy for rot90 CIFAR10 is able to outperform both
equituning and the finetuned-version of equizero by a comfortable margin. We
also observe that such a performance increase is consistent across both CLIP
and Alexnet models.

Details of the $\lambda$ network: For CLIP, we used a two layered fully
connected network with a hidden layer of dimension 100 and outputs a scalar
value. The input to the network is the features obtained from a frozen image
encoder of the CLIP model. The input dimension is 1024 for RN50 and 512
otherwise. Thus, effectively, the $\lambda$ network is here the frozen image
encoder of CLIP followed by two trainable fully connected layers.

Finetuning details of the $\lambda$ network: For CLIP, we use a learning rate
of $0.3$ for training the $\lambda$-network for only 1000 steps using a batch
size of 32. Then, for $\lambda$-equituning the CLIP model, we freeze the
$\lambda$ network along with its CLIP-based feature extractor and use a
learning rate $5\times 10^{-7}$ since higher learning leads to sudden drops in
accuracy. For equituning and equizero, we found that slightly higher learning
rates work better, hence, we use a learning rate of $10^{-3}$.

Results and Observations: In figure 11 we demonstrate additional experiments
validating our $\lambda$-equitune algorithm on a CLIP pre-trained model as
well as a pre-trained Alexnet model. While Alexnet comfortably outperforms the
other finetuning baselines (figure 11(b)). The results for the CLIP model for
these two image encoders are mixed. In figure 11(a), we show the performance
of CLIP model on additional image encoders ViT-B/16, ViT-B/32. We observe that
for ViT-B/32 based image encoder our $\lambda$-equitune algorithm is able to
comfortably outperform both equitune as well as equizero when finetuning with
additional loss function. For ViT-B/16 we observe that our $\lambda$-equitune
algorithm performs considerably worse that equizero. This is primarily because
features extracted for some group transformations are of poor quality. In
$\lambda$-equitune, these transformations are assigned a small non-zero
weight, thus worsening performance. Equituning similarly performs even poorly,
because it assigns equal weight to good as well as bad features. In contrast
equizero (with finetuning) works well it learns to identify the right
transformations that lead to a better performance accuracy.

Visualising $\lambda$-Equitune In figure 12, we visualize the usefulness of
$\lambda$-equitune that leads to an improved zero-shot performance. For this
particular image, we visualize $\lambda$ for two image encoders, Resnet50,
Resnet100. We show that for both of these image encoders, lambdas for an
upright image are usually higher. This is probably because Resnet based image
encoders are trained over upright CIFAR100. Thus, providing high quality
features for the same. This leads to an improved zero-shot performance.

Figure 12: Plot shows an example of normalized $\lambda$ weights for RN50 and
RN101 in CLIP used by the $\lambda$-equitune method for an image of a mountain
from CIFAR100.


# Determination of matter radius and neutron-skin thickness of 60,62,64Ni
from reaction cross section of proton scattering on 60,62,64Ni targets

Shingo Tagami  Department of Physics, Kyushu University, Fukuoka 819-0395,
Japan  Tomotsugu Wakasa  Department of Physics, Kyushu University, Fukuoka
819-0395, Japan  Masanobu Yahiro<EMAIL_ADDRESS>Department of Physics,
Kyushu University, Fukuoka 819-0395, Japan

###### Abstract

Background: In our previous work, we determined matter radii $r_{\rm m}({\rm
exp})$ and neutron-skin thickness $r_{\rm skin}({\rm exp})$ from reaction
cross sections $\sigma_{\rm R}({\rm exp})$ of proton scattering on 208Pb,
58Ni, 40,48Ca, 12C targets, using the chiral (Kyushu) $g$-matrix folding model
with the densities calculated with Gogny-D1S-HFB (D1S-GHFB) with angular
momentum projection (AMP). The resultant $r_{\rm skin}({\rm exp})$ agree with
the PREX2 and CREX values. As for 58Ni, our value is consistent with one
determined from the differential cross section for 58Ni+4He scattering. As for
p+60,62,64N scattering, $\sigma_{\rm R}({\rm exp})$ are available as a
function of incident energies $E_{\rm in}$, where $E_{\rm in}=22.8\sim 65.5$
MeV for 60Ni, $E_{\rm in}=40,60.8$ MeV for 62Ni, $E_{\rm in}=40,60.8$ MeV for
64Ni.
Purpose: , Our aim is to determine matter radii $r_{\rm m}({\rm exp})$ for
60,62,64Ni from the $\sigma_{\rm R}({\rm exp})$.
Method: Our method is the Kyushu $g$-matrix folding model with the densities
scaled from D1S-GHFB+AMP densities,
Results: Our skin values are $r_{\rm skin}({\rm exp})=0.076\pm
0.019,~{}0.106\pm 0.192,~{}0.162\pm 0.176$ fm, and $r_{\rm m}({\rm
exp})=3.759\pm 0.011,~{}3.811\pm 0.107,~{}3.864\pm 0.101$ fm for 60,62,64Ni,
respectively.

##  I Introduction and conclusion

###  I.1 Background

Background: A novel method for measuring nuclear reactions in inverse
kinematics with stored ion beams was successfully used to extract the matter
radius $r_{\rm m}({\rm exp})$ of 58Ni [1]. The experiment was performed at the
experimental heavy-ion storage ring at the GSI facility. Their result
determined from the differential cross section for 58Ni+4He scattering is
$r_{m}({\rm GSI})=3.70(7)$ fm.

Reaction cross section $\sigma_{\rm R}$ is a standard observable to determine
$r_{\rm m}({\rm exp})$ and neutron-skin thickness $r_{\rm skin}({\rm exp})$;
note that $r_{\rm skin}$ can be evaluated from the $r_{\rm m}$ by using the
$r_{\rm p}({\rm exp})$ calculated with the isotope shift method based on the
electron scattering [2]. In fact, we determined $r_{\rm m}({\rm exp})$ and
$r_{\rm skin}({\rm exp})$ from $\sigma_{\rm R}({\rm exp})$ of proton
scattering on 208Pb, 58Ni, 40,48Ca, 12C targets, using the chiral (Kyushu)
$g$-matrix folding model with the proton and neutron densities scaled with
D1S-GHFB+AMP densities [3], where D1S-GHFB+AMP is the abbreviation of
Gogny-D1S-HFB with angular momentum projection (AMP). Our skin values $r_{\rm
skin}({\rm exp})$ agree with the PREX2 and CREX values. As for 58Ni, our
matter radius $r_{m}({\rm exp})=3.711\pm 0.010$ fm. is consistent with
$r_{m}({\rm GSI})=3.70(7)$ fm. As for p+60,62,64N scattering, $\sigma_{\rm
R}({\rm exp})$ are available as a function of incident energy $E_{\rm in}$[4,
5, 6]; here $E_{\rm in}=22.8\sim 65.5$ MeV for 60Ni, $E_{\rm in}=40,60.8$ MeV
for 62Ni, $E_{\rm in}=40,60.8$ MeV for 64Ni. Now we consider p+60,62,64N
scattering.

Purpose: Our aim is to determine $r_{\rm m}({\rm exp})$, $r_{\rm skin}({\rm
exp})$ for 60,62,64Ni from the $\sigma_{\rm R}({\rm exp})$ by using the Kyushu
$g$-matrix folding model with the proton and neutron densities scaled from
D1S-GHFB+AMP ones.

Results: Our results $r_{\rm m}({\rm exp})$, $r_{\rm n}({\rm exp})$, $r_{\rm
skin}({\rm exp})$ are listed in Table 1.

Conclusion We discover the fact the $r_{\rm m}({\rm exp})/A^{1/3}$ (the matter
radius per nucleon) is inversely proportional to the total binding energy per
nucleon $E_{\rm B}/A$, where $A$ is the mass number.

Table 1: Values of $r_{\rm m}$, $r_{\rm n}$, $r_{\rm skin}$, $r_{\rm p}$. The $r_{\rm p}({\rm exp})$ are determined from the charge radii [2]. ‘Data’ shows citations on $\sigma_{\rm R}$. The radii are shown in units of fm.  $r_{\rm p}({\rm exp})$ | $r_{\rm m}({\rm exp})$ | $r_{\rm n}({\rm exp})$ | $r_{\rm skin}({\rm exp})$ | Data
---|---|---|---|---
58Ni $3.6849$ | $3.711\pm 0.010$ | $3.740\pm 0.019$ | $0.055\pm 0.019$ | [3]
60Ni $3.723$ | $3.759\pm 0.011$ | $3.799\pm 0.019$ | $0.076\pm 0.019$ | [4, 5]
62Ni $3.753$ | $3.811\pm 0.107$ | $3.859\pm 0.192$ | $0.106\pm 0.192$ | [6]
64Ni $3.772$ | $3.864\pm 0.101$ | $3.933\pm 0.176$ | $0.162\pm 0.176$ | [6]

##  II Folding Model

Kohno calculated the $g$ matrix for the symmetric nuclear matter, using the
Brueckner-Hartree-Fock method with chiral N3LO 2NFs and NNLO 3NFs [7]. He set
$c_{D}=-2.5$ and $c_{E}=0.25$ so that the energy per nucleon can become
minimum at $\rho=\rho_{0}$. Toyokawa et al. localized the non-local chiral $g$
matrix [8], using the localization procedure proposed by the Melbourne group
[9, 10]. The resulting local $g$ matrix is referred to as “local Kyushu
$g$-matrix”.

We use the Kyushu $g$-matrix folding model [8] with the densities calculated
with D1S-GHFB+AMP [11]. The Kyushu $g$-matrix itself [8] is constructed from
the chiral nucleon-nucleon (NN) interaction with the cutoff 550 MeV.

In this paper, we consider proton-nucleus scattering. The potential
$U({\mbox{\boldmath$R$}})$ consists of the direct and exchange parts, $U^{\rm
DR}({\mbox{\boldmath$R$}})$ and $U^{\rm EX}({\mbox{\boldmath$R$}})$ [12, 13].
The validity of the localization is shown in Refs. [12].

###  II.1 Scaling procedure of proton and neutron densities

For example, the neutron density $\rho_{n}(r)$ is scaled from the D1S-GHFB+AMP
one. We can obtain the scaled density $\rho_{\rm
scaling}({\mbox{\boldmath$r$}})$ from the original density
$\rho({\mbox{\boldmath$r$}})$ as

$\displaystyle\rho_{\rm
scaling}({\mbox{\boldmath$r$}})=\frac{1}{\alpha^{3}}\rho({\mbox{\boldmath$r$}}/\alpha)$
(1)

with a scaling factor

$\displaystyle\alpha=\sqrt{\frac{\langle{\mbox{\boldmath$r$}}^{2}\rangle_{\rm
scaling}}{\langle{\mbox{\boldmath$r$}}^{2}\rangle}}.$ (2)

We scale the neutron density so that the $f\times\sigma_{\rm R}({\rm D1S})$
may reproduce the data ($\sigma_{\rm R}({\rm exp})$) under that condition that
the $r_{\rm p}({\rm scaling})$ agrees with $r_{\rm p}({\rm exp})$ [2] of
electron scattering, where $\sigma_{\rm R}({\rm D1S})$ is the result of
D1S-GHFB+AMP for each $E_{\rm in}$. and $f$ is the average of $\sigma_{\rm
R}({\rm exp})/\sigma_{\rm R}({\rm D1S})$ over $E_{\rm in}$. The matter radius
$r_{\rm m}(E_{\rm in})$ thus obtained depends on $E_{\rm in}$. We then take
the average of $r_{\rm m}(E_{\rm in})$ over $E_{\rm in}$. The resulting value
$r_{\rm m}({\rm exp})$ is shown in Table 1. The corresponding $r_{\rm n}({\rm
exp})$ and $r_{\rm skin}({\rm exp})$ obtained from $r_{\rm m}({\rm exp})$ and
$r_{\rm p}({\rm exp})$ are also shown in Table 1.

D1M [14, 15] is an improved version of D1S for binding energies of many
nuclei. We can use D1M instead of D1S, leading to the change of
$U({\mbox{\boldmath$R$}})$. The results of D1M are the same as our results of
Table 1.

This scaling procedure is used for proton scattering on Sn in Ref. [16].

##  III Results

Figure 1 shows reaction cross sections $\sigma_{\rm R}$ as a function of
$E_{\rm in}$. Our result $\sigma_{\rm R}({\rm D1S})$ overshoots somewhat, but
$f\times\sigma_{\rm R}({\rm D1S})$ is close to the central values of
experimental data [4, 5], where $f=0.97488$. The $f\times\sigma_{\rm R}({\rm
D1S})$ are used in order to determine $r_{\rm m}({\rm exp})$ and $r_{\rm
skin}({\rm exp})$.

Figure 1:  $E_{\rm in}$ dependence of reaction cross sections $\sigma_{\rm R}$
for $p$+60Ni scattering. Closed circles denote $\sigma_{\rm R}({\rm D1S})$ of
the D1S-GHFB+AMP densities, whereas open circles correspond to
$f\times\sigma_{\rm R}({\rm D1S})$. The data (crosses) are taken from Refs.
[4, 5].

The same scaling procedure is taken also for $p$+62,64Ni scattering, where
$F=0.96213,~{}0.98232$ for 62,64Ni, respectively.

The data on $E_{\rm B}/A$ (the total binding energy per nucleon) weakly depend
on $A$ for 58,60,62,64Ni [17]. This is true for $r_{\rm m}({\rm exp})/A^{1/3}$
(the matter radius per nucleon). In fact, $A$ dependence of $r_{\rm m}({\rm
exp})/A^{1/3}\times E_{\rm B}/A$ is even smaller; namely, the average $\beta$
of $r_{\rm m}({\rm exp})/A^{1/3}\times E_{\rm B}/A$ over $A$ is

$\displaystyle\beta=8.437385~{}{\rm MeV\cdot fm}$ (3)

and the error is $0.024~{}{\rm MeV\cdot fm}$. The relative error is 0.2847%.
This indicates that the $r_{\rm m}({\rm exp})/A^{1/3}$ (the matter radius per
nucleon) is in inverse proportional to the total binding energy per nucleon
$E_{\rm B}/A$; see Table 2 for $E_{\rm B}/A$ and $r_{\rm m}({\rm
exp})/A^{1/3}$.

Using $\beta=8.437385~{}{\rm MeV\cdot fm}$, $E_{\rm B}/A$, we can derive the
central values of $r_{\rm m}({\rm exp})$. These are 3.711, 3.759, 3.811, 3.864
fm for 58,60,62,64Ni. These values agree with our values of Table 1. One can
easily evaluate the central value of matter radius from $\beta$, $E_{\rm
B}/A$. This is convenient.

Table 2: Values of $E_{\rm B}/A$ [17], $r_{\rm m}({\rm exp})/A^{(}1/3)$. The $r_{\rm m}({\rm exp})$ are shown in units of fm and the $E_{\rm B}/A$ are in MeV.  $BE/A$ | $r_{\rm m}({\rm exp})/A^{(}1/3)$
---|---
58Ni $8.732059$ | $0.959$
60Ni $8.780774$ | $0.960$
62Ni $8.794553$ | $0.963$
64Ni $8.777461$ | $0.966$

###### Acknowledgements.

We would like to thank Dr. Toyokawa for his contribution.

## References

  * Zamora  _et al._ [2017] J. C. Zamora _et al._ , Nuclear-matter radius studies from Ni58($\alpha$,$\alpha$) experiments at the GSI Experimental Storage Ring with the EXL facility, Phys. Rev. C 96, 034617 (2017).
  * Angeli and Marinova [2013] I. Angeli and K. P. Marinova, Table of experimental nuclear ground state charge radii: An update, Atom. Data Nucl. Data Tabl. 99, 69 (2013).
  * Wakasa  _et al._ [2023] T. Wakasa, S. Tagami, J. Matsui, M. Takechi, and M. Yahiro, Neutron-skin values and matter and neutron radii determined from reaction cross sections of proton scattering on ${}^{12}\mathrm{C}$, ${}^{40,48}\mathrm{Ca}$, ${}^{58}\mathrm{Ni}$, and ${}^{208}\mathrm{Pb}$, Phys. Rev. C 107, 024608 (2023).
  * Ingemarsson  _et al._ [1999] A. Ingemarsson, J. Nyberg, P. Renberg, O. Sundberg, R. Carlson, A. Auce, R. Johansson, G. Tibell, B. Clark, L. Kurth Kerr, and S. Hama, Nuclear Physics A 653, 341 (1999).
  * Eliyakut-Roshko  _et al._ [1995] T. Eliyakut-Roshko, R. H. McCamis, W. T. H. van Oers, R. F. Carlson, and A. J. Cox, Measurements of proton total reaction cross sections for ${}^{58}\mathrm{Ni}$ and ${}^{60}\mathrm{Ni}$ including nonrelativistic and relativistic data analyses, Phys. Rev. C 51, 1295 (1995).
  * Menet  _et al._ [1971] J. J. H. Menet, E. E. Gross, J. J. Malanify, and A. Zucker, Total-reaction-cross-section measurements for 30-60-mev protons and the imaginary optical potential, Phys. Rev. C 4, 1114 (1971).
  * Kohno [2012] M. Kohno, Strength of reduced two-body spin-orbit interaction from chiral three-nucleon force, Phys. Rev. C 86, 061301 (2012), arXiv:1209.5048 [nucl-th] .
  * Toyokawa  _et al._ [2018] M. Toyokawa, M. Yahiro, T. Matsumoto, and M. Kohno, Effects of chiral three-nucleon forces on 4He-nucleus scattering in a wide range of incident energies, PTEP 2018, 023D03 (2018), arXiv:1712.07033 [nucl-th] .
  * von Geramb  _et al._ [1991] H. V. von Geramb _et al._ , Phys. Rev. C 44, 73 (1991).
  * Amos and Dortmans [1994] K. Amos and P. J. Dortmans, Phys. Rev. C 49, 1309 (1994).
  * Tagami  _et al._ [2020] S. Tagami, M. Tanaka, M. Takechi, M. Fukuda, and M. Yahiro, Chiral $g$-matrix folding-model approach to reaction cross sections for scattering of Ca isotopes on a C target, Phys. Rev. C 101, 014620 (2020), arXiv:1911.05417 [nucl-th] .
  * Minomo  _et al._ [2010] K. Minomo, K. Ogata, M. Kohno, Y. R. Shimizu, and M. Yahiro, The Brieva-Rook Localization of the Microscopic Nucleon-Nucleus Potential, J. Phys. G 37, 085011 (2010), arXiv:0911.1184 [nucl-th] .
  * Watanabe  _et al._ [2014] S. Watanabe _et al._ , Ground-state properties of neutron-rich Mg isotopes, Phys. Rev. C 89, 044610 (2014), arXiv:1404.2373 [nucl-th] .
  * Goriely  _et al._ [2009] S. Goriely, S. Hilaire, M. Girod, and S. Peru, First Gogny-Hartree-Fock-Bogoliubov Nuclear Mass Model, Phys. Rev. Lett. 102, 242501 (2009).
  * Robledo  _et al._ [2019] L. M. Robledo, T. R. Rodríguez, and R. R. Rodríguez-Guzmán, Mean field and beyond description of nuclear structure with the Gogny force: A review, J. Phys. G 46, 013001 (2019), arXiv:1807.02518 [nucl-th] .
  * Tagami  _et al._ [2023] S. Tagami, T. Wakasa, and M. Yahiro, Neutron skin thickness of 116,118,120,122,124sn determined from reaction cross sections of proton scattering, Results in Physics 46, 106296 (2023).
  * [17] https://www.nndc.bnl.gov/nudat2/.