# U(1) Symmetry-breaking Observed in
Generic CNN Bottleneck Layers

Louis-François Bouchard Mohsen Ben Lazreg Matthew Toews

École de Technologie Supérieure ÉTS
Montréal, Canada
{bouchard.lf, mohsenbenlazreg<EMAIL_ADDRESS>
###### Abstract

We report on a novel model linking deep convolutional neural networks (CNN) to
biological vision and fundamental particle physics. Information propagation in
a CNN is modeled via an analogy to an optical system, where information is
concentrated near a bottleneck where the 2D spatial resolution collapses about
a focal point $1\times 1=1$. A 3D space $(x,y,t)$ is defined by $(x,y)$
coordinates in the image plane and CNN layer $t$, where a principal ray
$(0,0,t)$ runs in the direction of information propagation through both the
optical axis and the image center pixel located at $(x,y)=(0,0)$, about which
the sharpest possible spatial focus is limited to a circle of confusion in the
image plane. Our novel insight is to model the principal optical ray $(0,0,t)$
as geometrically equivalent to the medial vector in the positive orthant
$I(x,y)\in R^{N+}$ of a $N$-channel activation space, e.g. along the greyscale
(or luminance) vector $(t,t,t)$ in $RGB$ colour space. Information is thus
concentrated into an energy potential $E(x,y,t)=\|I(x,y,t)\|^{2}$, which,
particularly for bottleneck layers $t$ of generic CNNs, is highly concentrated
and symmetric about the spatial origin $(0,0,t)$ and exhibits the well-known
"Sombrero" potential of the boson particle. This symmetry is broken in
classification, where bottleneck layers of generic pre-trained CNN models
exhibit a consistent class-specific bias towards an angle $\theta\in U(1)$
defined simultaneously in the image plane and in activation feature space.
Initial observations validate our hypothesis from generic pre-trained CNN
activation maps and a bare-bones memory-based classification scheme, with no
training or tuning. Training from scratch using combined one-hot $+U(1)$ loss
improves classification for all tasks tested including ImageNet.

##  1 Introduction

A modern computer vision system involves image acquisition, computational
processing and memory indexing. Research in the last decade has focused
intensely on optimizing computational processing, i.e. deep neural network
classification of labelled image data [42, 74], and more rarely considers
aspects of acquisition. Nevertheless, both optical capture systems and deep
neural networks propagate visual information across time, and operate by
concentrating visual information in spatial bottlenecks. Might there be
unexplored analogies between the two systems leading to new insights and
improved architectures?

To answer this question, we propose to analyse information propagation through
an optical system equipped with a lens and followed by a deep neural network
encoder-decoder, as shown in Figure 1, noting several novel and remarkable
analogies between both systems. First, multi-spectral information in the
$(x,y)$ plane may be viewed as collapsing to a point $1\times 1=1$ resolution
at both a) an optical focal point and b) a neural network bottleneck. These
points are located on a ray $t$ perpendicular to the $(x,y)$ plane of both a)
the propagating optical wavefront and b) network image layer. Information
present in both $(x,y)$ spatial coordinates and activation channels $I(x,y)$
may thus be treated within a unified geometrical framework, centered about the
principal ray. We pay special attention to the circle of confusion, which
represents a minimum spatial radius within which information may be focused in
an optical system, given lens imperfections including diffraction, lens
aberrations, chromatic aberration of the input spectrum. Analogously, we
hypothesize a circle of confusion limiting the maximum classification accuracy
may be achieved in the spatial bottleneck of a deep neural network filtering
process. Both observations and training experiments support a model whereby
bottleneck activations for different classes are optimally distributed along a
unit circle angle $\theta\in U(1)$ defined simultaneous in the image plane and
in the activation feature space.

Figure 1: a) The propagation of light rays through an optical lens system. b)
the propagation of information through a deep neural network (encoder-
decoder). Shared aspects include a common principal ray, a spectrum and a
circle of confusion.

Our paper begins by presenting novel observations regarding the common
structure of activation maps in bottleneck layers [89, 44], shared across
diverse pre-trained architectures (ResNet [28], DenseNet [32], Inception [75],
VGG [69]) and datasets (textures, objects, fine-grained objects). In all
cases, the activation energy $E(x,y)=\|I(x,y)\|^{2}$ is highly symmetric and
concentrated near the image center $(x,y)$, highly reminiscent of the so-
called "sombrero" potential associated with boson particles [23]. This
symmetry is broken in classification trials via transfer learning from generic
pretrained networks, via classic nearest-neighbor lookup [18] of pre-trained
bottleneck activation information stored in memory. Lookup of individual
pixel-wise activations rather than typical representations such pooled vectors
or entire flattened layers both leads to superior classification accuracy and
allows symmetry-breaking to be observed. Most notably, while matches between
images of the same class are concentrated near the image center or origin
$(x,y)=(0,0)$, they exhibit a consistent class-specific bias towards an
angular direction $\theta\in U(1)$ in both the image plane and feature space.

These observations lead us to investigate the hypothesis of a unitary class
label $\theta\in U(1)$ parameter in network training from scratch. Training
networks with a single random angular class label leads to improved
classification across a wide variety of tasks including generic objects,
textures, animals, and highly specific categories including bird species or
aircrafts. We propose a combined $one-hot+U(1)$ loss function, introducing an
antisymmetric training signal via an angular class label $(x,y)\in U(1)$
constrained to the unit circle $x^{2}+y^{2}=1$, which leads to consistently
higher accuracy across a variety of CNN architextures and classification tasks
of varying specificity including ImageNet.

##  2 Related Work

Our work adopts an analogy between the propagation of light in an optical
system and layer-wise CNN processing, where layers are evaluated in discrete
time steps $t$. Deep neural networks may be used to simulate ordinary and
partial differential equations [20, 13], including solving forward and inverse
problems and simulating wave propagation via graphics processing unit
(GPU)-based CNN implementations [64, 60]. This is unsurprising given that GPUs
were designed initially for graphics algorithms such as ray tracing that
approximate Maxwell’s electromagnetic (EM) field equations for wavelengths
much smaller than objects imaged (e.g. the $[380,750]$nm range of the visible
RGB spectrum vs. human-sized objects). Neural networks may be computed via
physical optical filters, where the difficulty is in achieving non-linear
activations [72]. The key insight is that information may be modeled as
propagating through space as a field evolving in discrete time steps, in a
sequential, feed-forward manner with transitions defined by linear dot product
operators $I_{t,\bar{x}}=I_{t-1,\bar{x}}\cdot W_{t,\bar{x}}$ in addition to
non-linear activation, sub-sampling, pooling, etc. Our results derive from
basic geometrical notions, the $\bar{x}=(x,y)\in R^{2}$ image space centered
on $(x,y)=(0,0)$, a 3D spacetime $(t,\bar{x})\in R^{+}\times R^{2}$, and a
vector-valued activation image $I(t,\bar{x})\in R^{N}$. Our novel approach
considers a single unitary $U(1)$ variable, i.e. $(x,y)$ coordinates
constrained to the unit circle $x^{2}+y^{2}=1$ or equivalently an angle
$\theta=atan2(y,x)$. $U(1)$ may be also understood by considering that any
function $f(\bar{x})$ may be expressed as the sum
$f(\bar{x})=f_{s}(\bar{x})+f_{a}(\bar{x})$ of symmetric $f_{s}(\bar{x})$ and
anti-symmetric $f_{a}(\bar{x})$ components, i.e. the $cos(\bar{x})$ and
$sin(\bar{x})$ components of Euler’s equation
$e^{i\theta}=\cos(\theta)+i\sin(\theta)$. Unitary variables are well known in
mathematical analysis and increasingly used in machine learning formulations
[36, 78]. In the standard model of 3D particle physics, fundamental particles
are defined as fields or wave functions $\psi(\bar{x})$ referred to as bosons
$\psi_{s}(\bar{x})$ if the wave function is symmetric (eg. notably the photon
and the Higgs boson) and as fermions $\psi_{a}(\bar{x})$ if it is anti-
symmetric (e.g. quarks and leptons, notably the electron).

Symmetry-breaking is a process by which an energy potential transitions from a
symmetric to an asymmetric state. In deep neural networks, early
investigations included replica symmetry breaking [50] and non-monotonic
neural networks [9], more recently Euclidean neural networks [70] and
Noether’s notions of continuous symmetry from Lagrangian energy formulations
[77]. In biological networks it has been notably investigated as an
orientation selection mechanism within the cortex [6], ring attractor dynamics
projecting individual photons into the fruit fly brain and measuring the
angular response [37]. In particle physics, the Higgs mechanism lending mass
to other standard particles is based on symmetry breaking, first proposed in
1962 [30] and experimentally confirmed in 2012 [1] (Nobel prize 2013), the
same year AlexNet ushered in the wave of GPU-trained deep neural networks
[42]. Most recently, 2D quasi-particles called anyons were theorized in 1982
[87] and observed experimentally in 2020 as electrons constrained to 2D sheets
of gallium arsenide [5]. Anyon wavefunctions are not constrained to be
exclusively symmetric or anti-symmetric, as are standard 3D particles (e.g.
photons or electrons), and offer a potential mechanism for quantum computing
and memory formation via a braiding-like particle exchange process in the 2D
plane.

Our initial observations of $U(1)$ symmetry-breaking are based on nearest
neighbor indexing and classification [18], specifically using spatially-
localized activation vectors in pre-trained networks. This follows the
transfer learning approach, where networks pre-trained on large generic
datasets such as ImageNet [19] are used as general feature extractors for new
tasks [40, 4, 16]. Deep bottleneck activations tend to outperform specialized
shallower networks and meta-learning methods [14], particularly in the case of
few training data and a large domain shift between training and testing data
[25]. Various approaches seek to adapt ImageNet models to fine-grained tasks
by encoding activations at bottleneck layers, e.g. via descriptor information
(e.g. extracted off-the-shelf features [67], VLAD [3]), global average or max
pooling [61], generalized mean (GeM) [59], regional max pooling (R-MAC) [79]
in intermediate layers, modulated by attention operators [54]. Additional
training may consider joint loss between classification and instance retrieval
terms [7]. The mechanism of spatially localized activations (as opposed to
global descriptors) is closely linked to the attention mechanisms [33],
including non-local networks [83], squeeze-and-excitation networks [31],
transformer architectures [81, 11, 27] including hierarchically shifted
windows [45], thin bottleneck layers [65], self-attention mechanisms
considering locations and channels [88], intra-kernel correlations [26],
multi-layer perceptrons incorporating Euler’s angle [78], correspondence-based
transformers [34] and detectors [73], and geometrical embedding of spatial
information via graphs [39, 29]. Whereas these works typically seek end-to-end
learning solutions fitting within GPU memory constraints [24], we seek to
demonstrate the $U(1)$ theory via basic memory lookup.

Our final results training from scratch using $U(1)$ labels is similar in
spirit to work seeking to regularize the label and/or the activation feature
space, including using real-valued rather than one-hot training labels [62],
learning-based classifiers [84, 86], prototypical networks for few-shot
learning [52, 71], deep k-nearest neighbors [55], geometrical regularization
based on hyperspheres [49, 68], enforcing constant radial distance from the
feature space origin [91] or angular loss between prototypes [82]. We seek to
present our theory in the broadest, most general context. We consider a basic
classification and deliberately eschew architectural modifications that might
limit the generality of our analysis. We demonstrate our general theory with
results using a wide variety of pre-trained architectures including DenseNet
[32], Inception [75], ResNet [28], VGG [69] directly imported from TensorFlow
[2]. We consider a variety of testing datasets not used in ImageNet [19]
training, with including general categories (e.g. Caltech 101 [21]), and
specific instances (e.g. human brain MRIs of family members [80], faces [90]).

##  3 Information Propagation and $U(1)$ Symmetry-breaking

We propose modeling information propagation through a generic visual
information processing system, including both the optical capture apparatus
and discrete layer-wise processing in a deep convolutional neural network, as
shown in Figure 1. We develop an analogy between both, in order to develop the
notion of an information spectrum, unit circle of confusion and generic
classification as a $U(1)$ symmetry-breaking process.

The flow of information through spacetime in a computer vision system may be
viewed as the propagation of information quanta through spacetime. In an
optical system as in Figure 1 a), information is carried by photons radiating
from a 3D scene, where photons travel along rays in 3D space at a constant
speed in a vacuum. Light rays concentrated at a focus via a lens or a pinhole
may be accumulated into a spatially coherent 2D image at an image plane, e.g.
a RGB photosensor array. In a feedforward neural network system as in Figure 1
b), information is carried by activation vectors and propagating layer-wise
operations including linear dot product, non-linear activation functions such
as ReLu [51], pooling and subsampling. Information concentrated at a focus may
be used for prediction or propagated further to achieve a spatially coherent
2D image segmentation [46].

Several aspects common to both systems may be noted. Information is
distributed across a 2D spatial plane $(x,y)$, i.e. a propagating light
wavefront or a deep network layer, and across a spectrum $I(x,y)$, i.e. the
tri-chromatic RGB spectrum of visible light or the activation channels of a
network layer. We define the principal ray as perpendicular to the $(x,y)$
image plane, and central to both the 2D image space and information channel
space. In both systems, information is routed towards a focal point along the
principal ray, where the finest achievable focus is limited to a circle of
confusion, shown as a black ring in Figure 1. The circle of confusion is a
well-known optical phenomenon arising from diffraction or lens imperfections,
and compounded by dispersion, channel-specific refraction and chromatic
aberration. We hypothesize a similar phenomenon in deep neural networks, due
to the point spread function of filtering operations, and this motivates our
investigation into bottleneck disentanglement.

In an optical system, the electromagnetic (EM) spectrum encodes information as
photon particles, which may be modeled as propagating through 3D space via
Dirac delta operators $\delta(d\bar{x})$ (i.e. translation filters) defined in
3D space by the geometry of the optical apparatus. Photons are so-called boson
particles defined by a symmetric wave function
$\psi(x)=\psi_{s}(x)=\psi_{s}(-x)$, and may accumulate at a spatial focus
according to Bose-Einstein statistics [10] 111As opposed to so-called fermion
particles such as electrons defined by an anti-symmetric wave function
$\psi(x)=\psi_{a}(x)=-\psi_{a}(-x)$ which may only accumulate in (spin-up,
spin-down) pairs due to the Pauli exclusion principle.. In a deep neural
network, information is entangled in a limited number of activation channels
$I(x,y)\in R^{N+}$, and propagated via multi-channel filtering operations. An
important difference between the two systems is that in the case of deep CNNs
[43], the channel spectrum or gauge [17] is transformed at each layer in order
to encode and compress increasing amounts of visual information at reduced
spatial resolutions up to the network bottleneck, trading equivalent
representations for class-specific information the deeper we get into the
network [89, 44], whereas the EM gauge is preserved during light propagation.

The propagation of information in both systems may be modeled as a vector
field propagating through $(x,y,t)$ spacetime via local dot product operators
as follows. Let $I_{t,\bar{x}}$ be a vector of multi-channel information at
layer $t$ and spatial location $\bar{x}=(x,y)$, for example a 3-channel RGB
image or a N-channel activation layer. Information may be propagated from time
$t$ to $t+1$ by the dot product operation
$I_{t+1,\bar{x}}=\bar{I}_{t,\bar{x}}\cdot W_{t,\bar{x}}$, where
$\bar{I}_{t,\bar{x}}$ is a neighborhood tensor defined in the spatial window
surrounding $(t,\bar{x})$, and $W_{t,\bar{x}}$ are the weights of a linear
filter of the same size and shape as $\bar{I}_{t,\bar{x}}$. Note that in a
convolutional neural network (CNN) [43], filters are translation invariant and
constant across a layer $W_{t,\bar{x}}=W_{t}$ [89, 44], however they may
generally vary with spatial location as in multi-layer perceptrons [63] or
transformer networks [81] (hence transformers ’doing away with convolution’).

Our model is illustrated in Figure 2, where multi-channel activation
information is non-negative (e.g. following rectification (ReLu) [51]),
located in the positive orthant of activation space $I_{t,\bar{x}}\in R^{N+}$,
and may be expressed in polar coordinates $(r,\theta,t)$ centered about the
spatial origin $(x,y)=(0,0)$ and along the principal ray $(0,0,t)$. For
example, RGB color pixels may be represented by hue, saturation, intensity
(HSV) coordinates, where color perception is closely linked to a hue angle
$\theta\in U(1)$. We propose an analogy whereby multi-channel CNN activations
are also represented via a class angle $\theta\in U(1)$ according to dominant
intensity channels. We hypothesize that this is a primary mechanism by which
class information is encoded in image space, and that it may be identified and
used as a training signal in the $(x,y)$ image plane of arbitrary CNN layers,
most notably bottleneck layers with minimal spatial extent. We observe the
scalar energy $E_{t,\bar{x}}=\|I_{t,\bar{x}}\|^{2}\in R^{+}$ of activation
layers $I_{t,\bar{x}}$ from a variety of generic architectures pretrained on
the ImageNet dataset [19] in response to input images not used in training
(e.g. Caltech 101 [21]). As shown in Figure 2, the average activation energy
of pre-trained bottleneck layers is generally distributed symmetrically about
the origin in a "sombrero" shape reminiscent of a bosonic wave function. This
symmetry is broken in classification as shown in Figure 2 b), class-specific
activation information exhibits a consistent asymmetric bias towards an angle
$\theta\in U(1)$ shared across individuals of a class.

Figure 2: Our polar coordinate model for multi-channel data. a) The standard
3-channel red-green-blue color space with hue angle $\theta$. b) A 3-channel
subspace of the 1000-channel ImageNet [19] output, following a spatial
bottleneck layer, showing a class angle $\theta$ analogous to color hue and
defined by the input image content. (c) Information energy maps
$E_{t,\bar{x}}=\|I_{t,\bar{x}}\|^{2}$ computed from spatial bottleneck layers
of various ImageNet pre-trained networks and across Caltech101 [21] classes
not used in training. Note the highly symmetric structure and similarity to
the well-known bosonic "sombrero" energy potential [23]. b) shows how a
specific image (e.g. here a human brain MRI class not found in ImageNet)
exhibits asymmetric bias towards an angle $\theta$ in the direction of
visually similar ImageNet classes axes (here Nematode, Petri Dish and
Jellyfish).

##  4 Experiments

Our experiments validate our $U(1)$ theory in two sections. We first
demonstrate symmetry-breaking in basic nearest neighbor classification
experiments, for a variety generic ImageNet pre-trained CNNs. We then perform
training with $U(1)$ class labels, using a novel one-hot + $U(1)$ loss
function modeling symmetric + antisymmetric components.

###  4.1 Observing Symmetry-breaking in CNN Bottleneck Layers

We hypothesize that class information in trained networks is generally
concentrated into an angular argument $\theta\in U(1)$ both in the $(x,y)$
image plane and in activation feature space. We begin with observations
supporting this hypothesis, where classification via basic nearest-neighbor
indexing of pre-trained network activations is achieved via a symmetry-
breaking process in the image plane. Figure 3 illustrates our proposed
experimental retrieval architecture, where individual vectors $I_{t,\bar{x}}$
may translate across image space in order to match with similar vectors in
memory extracted from similar classes. As noted, the general mechanism is
closely linked to spatial attention approaches, and our implementation is
designed to maximize baseline classification performance. Note also that
minimizing the distance between vectors is equivalent to maximizing their dot
product.

Figure 3: (Left) Our proposed retrieval architecture where classification is
achieved from nearest neighbor matches between activation vectors derived from
an input image and labeled images stored in memory. Matches between images of
the same class (a) generally exhibit lower spatial confusion than (b) matches
to unrelated class images. (Right) Our hypothesis that image-to-memory
matching leads to symmetry-breaking of the layer energy function over the
$U(1)$ plane and class recognition.

Classification is achieved by maximising the likelihood function
$p(I|C,\\{I^{\prime}\\})$ of class $C$ associated with input image $I$ from
set of image examples $\\{I^{\prime}\\}$ stored in memory:

$\displaystyle C^{*}=\underset{C}{\mathrm{argmax}}~{}p(I|C,\\{I^{\prime}\\}),$
(1)

where $C^{*}$ is a maximum likelihood (ML) estimate of the image class. The
likelihood function is defined as follows. Let $i$ and $j$ be indices applying
respectively to query and memory images. Let $x_{i},x_{j}\in R^{2}$ be
discrete spatial point locations, $x_{i}\in\Omega_{I}$ in the query image and
$x_{j}\in\Omega_{I^{\prime}}$ in memory images. Indices $j\in NN_{i}$ belong
to a set of K nearest neighbors of associated with $i$ defined as
$NN_{i}:\\{j:\|I_{xi}-I^{\prime}_{xj}\|\leq\|I_{xi}-I^{\prime}_{xk}\|\\}$
identified via activation vector memory indexing, where $I^{\prime}_{xk}$ is
the $k^{th}$ nearest neighbor of $I_{xi}$ in memory. The likelihood may be
expressed as a kernel density

$\displaystyle p(I|C,\\{I^{\prime}\\})\propto\frac{\sum_{i}\sum_{j\in
N_{i}}f(I_{x_{i}}-I^{\prime}_{x_{j}})[C=C^{\prime}_{j}]}{\sum_{x_{j}}[C=C^{\prime}_{j}]},$
(2)

where $[C=C^{\prime}_{j}]$ is the Iverson bracket evaluating to 1 upon
equality and 0 otherwise, and the denominator normalizes for class frequency
across the entire memory set $\sum_{j}[C=C^{\prime}_{j}]$. The kernel function
in Equation (2) is based on activation vector (dis)similarity
$f(I_{x_{i}},I^{\prime}_{x_{j}})$ and is defined as:

$\displaystyle
f(I_{x_{i}},I^{\prime}_{x_{j}})=exp-\left\\{\frac{\|I_{x_{i}}-I^{\prime}_{x_{j}}\|^{2}}{\alpha_{i}^{2}+\epsilon}\right\\},$
(3)

where in Equation (3),
$\alpha_{i}=\underset{x_{j}\in\Omega_{I^{\prime}}}{\operatorname{min}}\|I_{x_{i}}-I^{\prime}_{x_{j}}\|$
is an adaptive kernel bandwidth parameter defined as the distance to nearest
activation vector $I_{x_{i}}$ in memory
$I^{\prime}_{x_{j}}\in\\{I^{\prime}\\}$, and $\epsilon$ is a small positive
constant ensuring a non-zero denominator. Note that the Euclidean and cosine
distances are equivalent
$\operatorname{argmin}\|I_{x_{i}}-I^{\prime}_{x_{j}}\|=\operatorname{argmin}\\{1-I_{x_{i}}\cdot
I^{\prime}_{x_{j}}\\}$ for magnitude-normalized descriptors
$\|I_{x_{i}}\|=\|I^{\prime}_{x_{j}}\|=1$, and may be used interchangeably
according to the computational architecture at hand.

Experiments are based on a variety of generic CNN architectures trained on the
ImageNet dataset [19], and tested in basic memory-based retrieval experiments
using various datasets specifically not used in training, including general
objects (Caltech 101 [21]), textures (DTD) [15] and specific objects [80, 90].
Figure 4 establishes baseline classification results across CNN architectures
and descriptors, showing that localized pixel vector matching lead to the
highest accuracy amongst alternatives, particularly for Densenet [32]. Note
that high accuracy for pixel vector matching is due to the trade-off between
memory and computation (e.g. 7x7=49 pixels vs. 1 global descriptor). As our
goal here is to observe symmetry-breaking via the most accurate spatially
localized information available, however, efficiency is not an immediate
concern and is beyond the focus of our work here. We achieve efficient
retrieval using the Approximate Nearest Neighbor library (Annoy) method [8]
indexing method, a rapid tree-based algorithm with $O(log~{}N)$ query
complexity for $N$ elements in memory. Further efficiency could be achieved
via compression (e.g PCA [22]) or specialized architectures [73, 34].

Examples of consistent $U(1)$ symmetry-breaking are shown in Figure 5. Note
how distributions of nearest neighbor pixel vector matches between images of
same class are consistently biased towards a similar angle $\theta$,
validating class-specific asymmetry as predicted by our model in Figure 2.
Visualizations in Figure 5 make use of the $7\times 7=49$ pixel DenseNet201
[32] architecture with $I_{t,\bar{x}}\in R^{1920}$ channel vectors, which led
to the highest accuracy in memory-based classification trials.

Figure 4: Baseline classification performance of architectures and descriptors
on six memory-based classification tasks with $(K=10)$ nearest neighbors.
Dashed lines indicate the superior performance of the Densenet [32]
architecture (max in 5 of 6 tasks) and pixel vector descriptors (max for all).
Figure 5: $U(1)$ symmetry-breaking observed in distributions of activation
vector matches for Caltech 101 [21] classes. a) Matches between images of the
same class are concentrated within a circle of confusion around the center
$(0,0)$, and exhibit consistent bias towards a similar angle $\theta$ for
specific classes and individuals of the same class (eg. A and B, blue arrows).
b) Matches between images of unrelated class are scattered about the
periphery.

###  4.2 Training with One-hot + $U(1)$ Loss

Given the pattern of regular angular symmetry-breaking observed in the case of
pre-trained network indexing, we hypothesize that orientation may be learned
from arbitrary $U(1)$ class labels in order to improve classification.
Training combines a standard one-hot loss function with an additional angular
$U(1)$ loss, and applies to generic networks and training from scratch with no
bells or whistles. $U(1)$ loss uses a single angular training label, selected
randomly for each class on a unit circle $\theta\in U(1)$ in the $(x,y)$
plane. $U(1)$ labels are parameterized as $(x,y)$ coordinates constrained to
$x^{2}+y^{2}=1$, and estimated via fully-connected layers immediately
following the bottleneck. The L2 distance between $U(1)$ labels and predicted
$(x,y)$ parameters is used as a loss function, and mixed in equal weighting
with one-hot difference to generate a combined cross-entropy loss for the
backward step. We train over a fixed 100 epochs, with original train,
validation, and test sets and 5-fold cross-validation on a single Titan RTX
GPU. We use the Adam optimizer [38] and CosineAnnealingLR scheduler [47] with
default parameters from PyTorch [57] in all experiments and basic data
augmentation in the form of horizontal flips and random crops. Training and
classification are evaluated in diverse few-shot learning tasks of varying
degrees of granularity, including the Describable Textures Dataset (DTD) [15],
Caltech-UCSD Birds [85], Stanford Dogs [35], Flowers [53], Pets [56], Indoor67
[58], FGVC-Aircraft [48], Cars [41], and Imagenet [19].

Training with one-hot + $U(1)$ loss improved classification for all tested
networks compared to one-hot alone. Table 1 reports results for the network
architectures leading to the highest overall accuracy (EfficientNet-B0) [76]
and the most improved (Resnet) [28]. As an additional experiment, we tested
$(x,y)$ labels generated from various 2-parameter distributions other than the
unit circle $U(1)$, including centered (one-hot alone), discrete binary
combinations $[\pm 1,\pm 1]$ and uniformly distribution over space. Figure 6
shows preliminary results for a single dataset, where $U(1)$ labels lead to
the highest accuracy amongst alternatives.

Model |  DTD [15] |  UCSD Birds [85] |  Stanford Dogs [35] |  Flowers [53] |  Pets [56] |  Indoor67 [58] |  FGVC-Aircraft [48] |  Cars [41] |  Imagenet [19]
---|---|---|---|---|---|---|---|---|---
ResNet-18 [28] |  |  |  |  |  |  |  |  |
One-hot |  0.4153 |  0.3875 |  0.4015 |  0.4863 |  0.4090 |  0.4866 |  0.8703 |  0.3575 |  0.6350
One-hot + $U(1)$ |  0.5230 |  0.5313 |  0.5754 |  0.6487 |  0.5919 |  0.6224 |  0.8920 |  0.8011 |  0.6621
EfficientNet-B0 [76] |  |  |  |  |  |  |  |  |
One-hot |  0.5310 |  0.4907 |  0.5655 |  0.7252 |  0.6689 |  0.5458 |  0.8685 |  0.7882 |  0.7144
One-hot + $U(1)$ |  0.5368 |  0.5569 |  0.5764 |  0.7438 |  0.6797 |  0.5477 |  0.8768 |  0.8079 |  0.7171

Table 1: Classification results using training based on $U(1)$ labels. Figure
6: Comparing classification accuracy for training from scratch with various
2-parameter $(x,y)$ label configurations: one-hot, discrete, uniform and unit
circle $U(1)$. Our proposed $U(1)$ leads to the highest accuracy (here using
the Caltech-UCSD Birds dataset [85]).

##  5 Discussion

We present a novel model of information propagation in deep CNN layers with an
analogy to light propagation in an optical system, including a focal point and
an inherent circle of confusion in both cases. We assume the equivalence of
the principal optical axis $(0,0,t)$ and the medial vector of positive
activation feature space. These are both central to the $(x,y)$ image plane
allow class to be characterized by a hue-like angle $\theta\in U(1)$, defined
both in the image plane and activation feature space. Our model provides an
’end-to-end’ view including also the optical capture system and memory
storage. Our most significant finding pertains to information regarding
$(x,y)$ image space, which in its most concentrated form in low-resolution
network layers just prior to spatial collapse $1\times 1=1$, is well-
represented by a single unitary variable $\theta\in U(1)$. Furthermore, the
process by which classification occurs involves $U(1)$ symmetry breaking of
the deep CNN bottleneck energy $E(x,y)=\|I_{t,\bar{x}}\|^{2}$, consistent with
the "sombrero" bosonic potential. Our observations of a regular angular
asymmetry consistent with image class are the first in the literature, to our
knowledge, and indicate that symmetry-breaking occurs naturally in generic
pre-trained CNNs, linking deep learning more closely to biological networks
and recent results in particle physics. Our angular observations are
consistent with biological models: specifically $\theta$ deviations in Figure
5 induced from image inputs, and $\theta$ deviations in orientation/heading
selection in the fruit fly visual system induced by individual photon inputs,
see [37] Figure 3. They are also reminiscent recently discovered anyon
quasiparticles [5], and may provide mechanism for memory formation whereby an
input sequence is woven into a "braid" of memory via the particle exchange
process in the 2D plane.

Our work brings together a broad range of fundamental concepts and state-of-
the-art research in optics, particle physics, biological and artificial neural
networks, which we attempt to present in the broadest and most understandable
description, avoiding as much as possible domain-specific jargon. Our
experiments make use of the most widely-known, generic neural network
architectures, data sets and NN classification [18] in order to best
demonstrate our $U(1)$ theory. One exception is the HCP brain MRI dataset
[80], a publicly available set of 1010 subject images acquired with ethics
review board approval and patient consent, designed to reflect the demographic
diversity of the USA population, including family members and siblings. Family
classification from brain MRI is a highly challenging few-shot learning task
not yet addressed via deep learning. Handcrafted descriptors have been used to
identify images of the same individuals or family members [12], note subjects
are labeled according to random codes with no personal identifiable
information. Our experimental pixel vector matching method is effective for
classification and demonstrating our theory with general pre-trained CNN
architectures, particularly lo-res bottleneck layers, however would require
optimisation for larger layers or applications with tight memory or timing
constraints. The classification accuracy for models training from scratch is
improved by incorporating an antisymmetric $U(1)$ loss function, a result
complementary to recent unitary optimization research [78, 36].

##  6 Appendices

###  6.1 U(1)-like Symmetry Breaking in the Drosophila Melanogaster Cortex

We noted that the U(1) symmetry breaking structure in deep CNN bottleneck
layers shown in Figure 5 (due to input image class) is similar to examples of
angular neural firing patterns observed in deep layers of biological neural
networks. Specifically, activations in deep/central brain layers of the
drosophila melanogaster cortex linked to the fly’s compass network determining
heading [37, 66], in response to dual-photon input stimuli. As central brain
regions are typically dominated by recurrent networks which often produce
complex patterns of neural activity, it is remarkable to observe a high-level,
abstract internal representation in the form of a topological ring
representing the heading direction in response to input light stimulus
patterns.

Figure 7: Left a) and b) show the $U(1)$ symmetry breaking observed in CNN
activations, i.e. NN matches of Densenet201 [32] activation vector layers in
response to input Caltech 101 [21] class images. Matches between same-class
images a) exhibit consistent angular deviations $\theta\in U(1)$ for specific
classes (e.g. class A, B, blue arrows) and b) unrelated class matches are
scattered about the periphery. Right shows the similar responses observed in
the central Drosophila cortex following two-photon input stimuli generated in
a specialized experimental apparatus [37] (c). The cortical response (d)
exhibits an "activation bump" consistent with a topological ring structure in
response to a specific ON stimulus (neural location ’5’ in the middle picture,
similar to our matches to the same class), and a dissipated peripheral
structure to OFF stimuli (first and third pictures, similar to our unrelated
class matches).

###  6.2 Pixel Vector Matching Distributions

Our experimental observations regarding symmetry are derived from a
rudimentary nearest neighbor (NN) classification algorithm (Section 4.1),
where activation vectors $I_{x}$ from individual pixel locations $x$ are
matched between query image locations $x_{i}$ to NN locations $x_{nn}$ of
vectors stored in memory. We justify the use of pixel vector indexing by the
observation that it generally leads to superior classification performance in
comparison to other representations including max pooling or flattening, see
Figure 3 and Figure 4, for a number of different networks and classification
tasks. Figure 5 shows distributions of nearest neighbor matching locations
$x_{nn}$, here Figure 8 shows distributions $p(x_{nn}|x_{i})$ conditioned
query pixel location $x_{i}$. Matching locations $x_{nn}$ are tightly
distributed around the query pixel locations $x_{i}$, indicating that in CNN
bottleneck layers, activation information $I_{x}$ is highly specific to
location $x$, and that unrelated class match distributions exhibit noticeably
higher variability in nearest neighbor location $x_{nn}$.

Figure 8: Distributions $p(x_{nn}|x_{i})$ of matching NN pixel locations
$x_{nn}$ conditioned on query locations $x_{i}$. Memory retrieval is based on
1920-dimensional activation vectors $I_{x}\in R^{1920+}$ from the $7\times
7=49$-pixel bottleneck layer of an Imagenet [19] pre-trained Densenet201 [32]
model and Caltech 101 [21] class images not used in training. Distributions
over $x_{nn}$ are shown for a selection of individual query pixel locations
$x_{i}$. These are tightly clustered around the original query pixel locations
$x_{i}$ (brightest pixels), and exhibit lower variance for matches to
instances of the same class (left) vs unrelated class (right). Note that
variations are stronger in tangential (as opposed to radial) directions,
indicating angular $\theta\in U(1)$ deviations about the unit circle.

## References

  * [1] Georges Aad, Tatevik Abajyan, B Abbott, J Abdallah, S Abdel Khalek, Ahmed Ali Abdelalim, R Aben, B Abi, M Abolins, OS AbouZeid, et al.  Observation of a new particle in the search for the standard model higgs boson with the atlas detector at the lhc.  Physics Letters B, 716(1):1–29, 2012.
  * [2] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  Software available from tensorflow.org.
  * [3] Relja Arandjelović, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition, 2016\.
  * [4] Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson.  Factors of transferability for a generic convnet representation, 2015\.
  * [5] Hugo Bartolomei, Manohar Kumar, Rémi Bisognin, Arthur Marguerite, J-M Berroir, Erwann Bocquillon, Bernard Placais, Antonella Cavanna, Q Dong, Ulf Gennser, et al.  Fractional statistics in anyon collisions.  Science, 368(6487):173–177, 2020.
  * [6] Rani Ben-Yishai, R Lev Bar-Or, and Haim Sompolinsky.  Theory of orientation tuning in visual cortex.  Proceedings of the National Academy of Sciences, 92(9):3844–3848, 1995.
  * [7] Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze.  Multigrain: a unified image embedding for classes and instances.  arXiv preprint arXiv:1902.05509, 2019.
  * [8] E. Bernhardsson.  Annoy on github.  https://github.com/spotify/annoy, 2015.
  * [9] Guido Boffetta, R Monasson, and R Zecchina.  Symmetry breaking in nonmonotonic neural networks.  Journal of Physics A: Mathematical and General, 26(12):L507, 1993\.
  * [10] Bose.  Plancks gesetz und lichtquantenhypothese.  Zeitschrift für Physik, 26:178–181, 1924.
  * [11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, pages 213–229. Springer, 2020.
  * [12] Laurent Chauvin, Kuldeep Kumar, Christian Desrosiers, William Wells III, and Matthew Toews.  Efficient pairwise neuroimage analysis using the soft jaccard index and 3d keypoint sets.  arXiv preprint arXiv:2103.06966, 2021.
  * [13] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  Advances in neural information processing systems, 31, 2018.
  * [14] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  arXiv preprint arXiv:1904.04232, 2019.
  * [15] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [16] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, and Andrea Vedaldi.  Deep filter banks for texture recognition, description, and segmentation.  International Journal of Computer Vision, 118(1):65–94, 2016.
  * [17] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling.  Gauge equivariant convolutional networks and the icosahedral cnn.  In International Conference on Machine Learning, pages 1321–1330. PMLR, 2019.
  * [18] Thomas Cover and Peter Hart.  Nearest neighbor pattern classification.  IEEE transactions on information theory, 13(1):21–27, 1967.
  * [19] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [20] MWMG Dissanayake and Nhan Phan-Thien.  Neural-network-based approximations for solving partial differential equations.  communications in Numerical Methods in Engineering, 10(3):195–201, 1994.
  * [21] Li Fei-Fei, Rob Fergus, and Pietro Perona.  One-shot learning of object categories.  IEEE Trans. Pattern Anal. Mach. Intell., 28(4):594–611, 2006.
  * [22] Karl Pearson F.R.S.  Liii. on lines and planes of closest fit to systems of points in space.  The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559–572, 1901.
  * [23] Jeffrey Goldstone.  Field theories with «superconductor» solutions.  Il Nuovo Cimento (1955-1965), 19(1):154–164, 1961.
  * [24] Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus.  Deep image retrieval: Learning global representations for image search.  In European conference on computer vision, pages 241–257. Springer, 2016.
  * [25] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris.  A broader study of cross-domain few-shot learning.  In European Conference on Computer Vision, pages 124–141. Springer, 2020.
  * [26] Daniel Haase and Manuel Amthor.  Rethinking depthwise separable convolutions: How intra-kernel correlations lead to improved mobilenets.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14600–14609, 2020.
  * [27] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.  A survey on visual transformer.  arXiv preprint arXiv:2012.12556, 2020.
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [29] Mikael Henaff, Joan Bruna, and Yann LeCun.  Deep convolutional networks on graph-structured data.  arXiv preprint arXiv:1506.05163, 2015.
  * [30] Peter Ware Higgs.  Broken symmetries, massless particles and gauge fields.  Phys. Lett., 12:132–133, 1964.
  * [31] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [32] Gao Huang, Zhuang Liu, and Kilian Q. Weinberger.  Densely connected convolutional networks.  CoRR, abs/1608.06993, 2016.
  * [33] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 603–612, 2019.
  * [34] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi.  Cotr: Correspondence transformer for matching across images.  arXiv preprint arXiv:2103.14167, 2021.
  * [35] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei.  Novel dataset for fine-grained image categorization.  In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
  * [36] Bobak Kiani, Randall Balestriero, Yann Lecun, and Seth Lloyd.  projunn: efficient method for training deep networks with unitary matrices.  arXiv preprint arXiv:2203.05483, 2022.
  * [37] Sung Soo Kim, Hervé Rouault, Shaul Druckmann, and Vivek Jayaraman.  Ring attractor dynamics in the drosophila central brain.  Science, 356(6340):849–853, 2017.
  * [38] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations, 12 2014.
  * [39] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * [40] Simon Kornblith, Jonathon Shlens, and Quoc V. Le.  Do better imagenet models transfer better?, 2019.
  * [41] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [42] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * [43] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural Computation, 1(4):541–551, 1989.
  * [44] Karel Lenc and Andrea Vedaldi.  Understanding image representations by measuring their equivariance and equivalence.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991–999, 2015.
  * [45] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [46] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  CoRR, abs/1411.4038, 2014.
  * [47] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with restarts.  CoRR, abs/1608.03983, 2016.
  * [48] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  CoRR, abs/1306.5151, 2013.
  * [49] Pascal Mettes, Elise van der Pol, and Cees Snoek.  Hyperspherical prototype networks.  Advances in neural information processing systems, 32, 2019.
  * [50] Rémi Monasson and Dominic O’Kane.  Domains of solutions and replica symmetry breaking in multilayer neural networks.  EPL (Europhysics Letters), 27(2):85, 1994.
  * [51] Vinod Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, page 807–814, Madison, WI, USA, 2010. Omnipress.
  * [52] Van Nhan Nguyen, Sigurd Løkse, Kristoffer Wickstrøm, Michael Kampffmeyer, Davide Roverso, and Robert Jenssen.  Sen: A novel feature normalization dissimilarity measure for prototypical few-shot learning networks.  In European Conference on Computer Vision, pages 118–134. Springer, 2020.
  * [53] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  * [54] Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han.  Large-scale image retrieval with attentive deep local features.  In Proceedings of the IEEE international conference on computer vision, pages 3456–3465, 2017.
  * [55] Nicolas Papernot and Patrick McDaniel.  Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning.  arXiv preprint arXiv:1803.04765, 2018.
  * [56] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In IEEE Conference on Computer Vision and Pattern Recognition, 2012\.
  * [57] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [58] Ariadna Quattoni and Antonio Torralba.  Recognizing indoor scenes.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 413–420, 2009.
  * [59] Filip Radenović, Giorgos Tolias, and Ondřej Chum.  Fine-tuning cnn image retrieval with no human annotation.  IEEE transactions on pattern analysis and machine intelligence, 41(7):1655–1668, 2018.
  * [60] Maziar Raissi, Paris Perdikaris, and George E Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics, 378:686–707, 2019.
  * [61] Ali Sharif Razavian, Josephine Sullivan, Stefan Carlsson, and Atsuto Maki.  Visual instance retrieval with deep convolutional networks, 2016.
  * [62] Pau Rodríguez, Miguel A Bautista, Jordi Gonzàlez, and Sergio Escalera.  Beyond one-hot encoding: Lower dimensional target embedding.  Image and Vision Computing, 75:21–31, 2018.
  * [63] F. Rosenblatt.  The perceptron: A probabilistic model for information storage and organization in the brain.  Psychological Review, pages 65–386, 1958.
  * [64] Tamás Roska, Leon O Chua, Dietrich Wolf, Tibor Kozek, Ronald Tetzlaff, and Frank Puffer.  Simulating nonlinear waves and partial differential equations via cnn. i. basic techniques.  IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 42(10):807–815, 1995.
  * [65] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [66] Johannes Seelig and Vivek Jayaraman.  Neural dynamics for landmark orientation and angular path integration.  Nature, 521:186–91, 05 2015.
  * [67] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson.  Cnn features off-the-shelf: an astounding baseline for recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813, 2014.
  * [68] Jiayi Shen, Zehao Xiao, Xiantong Zhen, and Lei Zhang.  Spherical zero-shot learning.  IEEE Transactions on Circuits and Systems for Video Technology, 32(2):634–645, 2021.
  * [69] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015\.
  * [70] Tess E Smidt, Mario Geiger, and Benjamin Kurt Miller.  Finding symmetry breaking order parameters with euclidean neural networks.  Physical Review Research, 3(1):L012002, 2021.
  * [71] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in neural information processing systems, 30, 2017.
  * [72] Xiubao Sui, Qiuhao Wu, Jia Liu, Qian Chen, and Guohua Gu.  A review of optical neural networks.  IEEE Access, 8:70773–70783, 2020.
  * [73] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  arXiv preprint arXiv:2104.00680, 2021.
  * [74] Shiliang Sun, Zehui Cao, Han Zhu, and Jing Zhao.  A survey of optimization methods from a machine learning perspective.  CoRR, abs/1906.06821, 2019.
  * [75] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [76] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  CoRR, abs/1905.11946, 2019.
  * [77] Hidenori Tanaka and Daniel Kunin.  Noether’s learning dynamics: Role of symmetry breaking in neural networks.  Advances in Neural Information Processing Systems, 34, 2021.
  * [78] Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang.  An image patch is a wave: Phase-aware vision mlp.  arXiv preprint arXiv:2111.12294, 2021.
  * [79] Giorgos Tolias, Ronan Sicre, and Hervé Jégou.  Particular object retrieval with integral max-pooling of cnn activations, 2016.
  * [80] David C. Van Essen, Stephen M. Smith, Deanna M. Barch, Timothy E.J. Behrens, Essa Yacoub, and Kamil Ugurbil.  The wu-minn human connectome project: An overview.  NeuroImage, 80:62–79, 2013.  Mapping the Connectome.
  * [81] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.
  * [82] Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin.  Deep metric learning with angular loss.  In Proceedings of the IEEE international conference on computer vision, pages 2593–2601, 2017.
  * [83] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [84] Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, and Laurens van der Maaten.  Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.  CoRR, abs/1911.04623, 2019.
  * [85] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona.  Caltech-ucsd birds 200.  Technical Report CNS-TR-201, Caltech, 2010.
  * [86] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao.  A discriminative feature learning approach for deep face recognition.  In European conference on computer vision, pages 499–515. Springer, 2016.
  * [87] Frank Wilczek.  Quantum mechanics of fractional-spin particles.  Physical review letters, 49(14):957, 1982.
  * [88] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [89] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.
  * [90] Zhifei Zhang, Yang Song, and Hairong Qi.  Age progression/regression by conditional adversarial autoencoder.  CoRR, abs/1702.08423, 2017.
  * [91] Yutong Zheng, Dipan K Pal, and Marios Savvides.  Ring loss: Convex feature normalization for face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5089–5097, 2018.