# Generative NeuroEvolution for Deep Learning

Phillip Verbancsics & Josh Harguess
Space and Naval Warfare Systems Center – Pacific
San Diego, CA 92101
<EMAIL_ADDRESS>

###### Abstract

An important goal for the machine learning (ML) community is to create
approaches that can learn solutions with human-level capability. One domain
where humans have held a significant advantage is visual processing. A
significant approach to addressing this gap has been machine learning
approaches that are inspired from the natural systems, such as artificial
neural networks (ANNs), evolutionary computation (EC), and generative and
developmental systems (GDS). Research into deep learning has demonstrated that
such architectures can achieve performance competitive with humans on some
visual tasks; however, these systems have been primarily trained through
supervised and unsupervised learning algorithms. Alternatively, research is
showing that evolution may have a significant role in the development of
visual systems. Thus this paper investigates the role neuro-evolution (NE) can
take in deep learning. In particular, the Hypercube-based NeuroEvolution of
Augmenting Topologies is a NE approach that can effectively learn large neural
structures by training an indirect encoding that compresses the ANN weight
pattern as a function of geometry. The results show that HyperNEAT struggles
with performing image classification by itself, but can be effective in
training a feature extractor that other ML approaches can learn from. Thus
NeuroEvolution combined with other ML methods provides an intriguing area of
research that can replicate the processes in nature.

##  1 Introduction

Evolution has a significant role in creating biological visual systems [1, 2],
thus a potential path for training artificial vision as effective as the
biological counterparts is neuro-evolution (NE; [3]) . However, NE has been
challenged because of the required structure sizes. While the human brain
includes trillions of connections [4, 5], traditional NE approaches produce
networks significantly smaller in size [3, 6]. Thus an active area of research
is evolutionary approaches that address this gap [6]. One significant area of
interest is generative and developmental systems (GDS), which are inspired by
principles that map genotype to phenotype in nature. Such principles are
enabled by reusing genetic information [4, 7]. For example, biological neural
networks exhibit several important organizational principles, such as
modularity, regularity, and hierarchy [8, 9], which can be exploited through
the indirect encoding and enable evolution of complex structures.

The GDS approach investigated in this paper is called the Hypercube-based
NeuroEvolution of Augmenting Topologies (HyperNEAT; [6, 10, 11]). HyperNEAT
trains an indirect encoding, called a _compositional pattern producing
network_ (CPPN; [12]) that represents artificial neural network (ANN)
connection weights. HyperNEAT has shown promise in simple visual
discrimination tasks [13, 14, 15, 16], but has been limited to two or fewer
hidden layers. This paper extends the investigation to deeper architectures by
applying HyperNEAT to the MNIST benchmark dataset. Interestingly, the results
show that HyperNEAT alone has difficulty in classifying the images, but
HyperNEAT is effective at training a feature extractor for backward
propagation learning. That is, HyperNEAT can learn the ‘deep’ parts of the
neural network, while backward propagation can perform the fine tuning to make
classifications from the generated features.

##  2 Background

The geometry-based methods that underlie the approach are described in this
section.

###  2.1 NeuroEvolution of Augmenting Topologies (NEAT)

_Neuro-evolution_ (NE; [3]) methods train artificial neural networks (ANNs)
through evolutionary algorithms. As opposed to updating weights according to a
learning rule, candidate ANNs are evaluated in a task and assigned fitness to
allow selection and creation of a new generation of ANNs by mutating and
recombining their selected genomes. The NeuroEvolution of Augmenting
Topologies (NEAT) algorithm [17] is a popular neuroevolutionary approach that
has been proven in a variety of challenging tasks, including particle physics
[18, 19], simulated car racing [20], RoboCup Keepaway [21], function
approximation [22], and real-time agent evolution [23], among others [17].

NEAT starts with a population of small, simple ANNs that increase their
complexity over generations by adding new nodes and connections through
mutation. That way, the topology of the network does not need to be known a
priori; NEAT searches through increasingly complex networks as it evolves
their connection weights to find a suitable level of complexity. The
techniques that facilitate evolving a population of diverse and increasingly
complex networks are described in detail in Stanley and Miikkulainen [17]; the
important concept for the approach in this paper is that NEAT is an
evolutionary method that discovers the right topology and weights of a network
to maximize performance on a task. The next section reviews the extension of
NEAT called HyperNEAT that allows it to effectively train large neural
structures.

###  2.2 CPPNs and HyperNEAT

Hypercube-based NEAT (HyperNEAT; [6, 11]) is a GDS extension of NEAT that
enables effective evolution of high-dimensional ANNs. The effectiveness of the
geometry-based learning in HyperNEAT has been demonstrated in multiple
domains, such as multi-agent predator prey [24, 25] and RoboCup Keepaway [26].
A full description of HyperNEAT is in Stanley et al. [6].

The main idea in HyperNEAT is that geometric relationships are learned though
an indirect encoding that describes how the _weights_ of the ANN can be
_generated_ as a function of geometry. Unlike a direct representation, wherein
every connection in the ANN is described individually, an indirect
representation describes a pattern of parameters without explicitly
enumerating each such parameter. That is, information is reused in such an
encoding, which is a major focus in the field of GDS from which HyperNEAT
originates [27, 28]. Such information reuse allows indirect encoding to search
a compressed space. HyperNEAT discovers the _regularities_ in the geometry and
learns from them.

The indirect encoding in HyperNEAT is called a _compositional pattern
producing network_ (CPPN; [12]), which encodes the _weight pattern_ of an ANN
[6, 10]. The idea behind CPPNs is that geometric patterns can be encoded by a
_composition of functions_ that are chosen to represent common regularities.
In this way, a set of simple functions can be composed into more elaborate
functions through hierarchical composition.Formally, CPPNs are _functions_ of
geometry (i.e. locations in space) that output connectivity patterns for nodes
situated in $n$ dimensions. Consider a CPPN that takes four inputs labeled
$x_{1}$, $y_{1}$, $x_{2}$, and $y_{2}$; this point in four-dimensional space
can _also_ denote the connection between the two-dimensional points
$(x_{1},y_{1})$ and $(x_{2},y_{2})$. The output of the CPPN for that input
thereby represents the weight of that connection (figure 1).

Figure 1: A CPPN Describes Connectivity. A grid of nodes, called the ANN
_substrate_ , is assigned coordinates. (1) Every connection between layers in
the substrate is queried by the CPPN to determine its weight; the line
connecting layers in the substrate represents a sample such connection. (2)
For each such query, the CPPN inputs the coordinates of the two endpoints. (3)
The weight between them is output by the CPPN. Thus, CPPNs can generate
regular patterns of connections.

Because the connection weights are produced as a function of their endpoints,
the final pattern is produced with _knowledge_ of the domain geometry, which
is literally depicted geometrically within the constellation of nodes. Weight
patterns produced by a CPPN in this way are called _substrates_ so that they
can be verbally distinguished from the CPPN itself. It is important to note
that the structure of the substrate is independent of the structure of the
CPPN. The substrate is an ANN whose nodes are situated in a coordinate system,
while the CPPN defines the connectivity among the nodes of the ANN. The
experimenter defines both the location and role (i.e. hidden, input, or
output) of each node in the substrate.

In summary, HyperNEAT evolves the topology and weights of the CPPN that
_encodes_ ANN weight patterns. An extension of HyperNEAT called HyperNEAT with
Link Expression Output (HyperNEAT-LEO) was introduced to constrain
connectivity with a bias towards modularity [15]. This extension separates the
decision of weight magnitude and expression into _two_ different CPPN outputs
and seeds the LEO with the concept of locality. The HyperNEAT-LEO variant is
shown in Algorithm 1. The next section reviews deep learning that has been
applied to image classification.

Input: Substrate Configuration

Output: Solution CPPN

Initialize population of minimal CPPNs with random weights;

while  _Stopping criteria is not met_ do

foreach  _CPPN in the population_ do

foreach  _Possible connection in the substrate_ do

Query the CPPN for weight $w$ of connection and LEO expression value $e$;

if  _$e >0.0$_ then

Create connection with a weight $w$;

Run the substrate as an ANN in the task domain to ascertain fitness;

Reproduce CPPNs according to the NEAT method to produce the next generation;

Output the champion CPPN.

Algorithm 1 HyperNEAT-LEO Algorithm

###  2.3 Deep Learning in Image Classification

Neural networks have experienced a resurgence thanks to breakthroughs in deep
learning that have led to state of the art results in a number of challenging
domains [29]. In particular, deep learning approaches have achieved remarkable
performance in a number of object recognition benchmarks, often achieving the
current best performance on these tasks. Such object recognition tasks where
deep learning has achieved the best results include the MNIST hand-written
digit dataset [30, 31], traffic sign recognition [32], and the ImageNet Large-
Scale Visual Recognition Challenge [33].

The challenge for deep learning is how to effectively train such large neural
structures. Traditional supervised learning approaches for neural networks,
such as backward propagation, face problems that include the “curse of
dimensionality” or vanishing gradient. Thus a popular alternative is to
perform pre-training on the deep architecture through unsupervised learning.
Restricted Boltzmann Machines [34] and auto-encoders [35] are among the common
approaches to performing this unsupervised training. This pre-training can
either act as an initial starting point for supervised learning of a deep
network or a feature extractor from which machine learning approaches can
learn [29].

Alternatively, supervised learning performance can be enhanced through the
selection of the deep architecture, such as _convolutional neural networks_
(CNNs). Beginning with the Neocognitron [36], these deep architectures have
been inspired by proposed models of the human visual cortex [37] and the
concept of local receptive fields that take advantage of the input topology.
That is, CNNs enforce a particular geometric knowledge by constructing an
architecture that learns features based upon locality. Through local receptive
fields, shared weights, and sub-sampling [38], CNNs enable backward
propagation (supervised learning) to effectively train deep architectures. The
next section introduces approaches to deep learning through HyperNEAT.

##  3 Approach: Deep Learning HyperNEAT

Although the HyperNEAT method succeeds in a number of challenging tasks [6,
10, 11, 26] by exploiting geometric regularities, it has not yet been applied
to tasks where deep learning is showing promise or to deep architectures.
Because HyperNEAT learns as a function of domain geometry, it is well-suited
towards architectures similar to CNNs. This approach introduces two
modifications to HyperNEAT training.

The first modification introduces the idea of HyperNEAT as a feature learner.
Conventional HyperNEAT trains a CPPN that defines an ANN that is the solution,
that is, the produced ANN is applied directly to the task and then the ANN’s
performance on that task determines the CPPN’s fitness score. However, this
HyperNEAT modification trains an ANN that transforms inputs into features
based upon domain geometry and then the features are given to another machine
learning approach to solve the problem, the performance of this learned
solution then defines the fitness score of the CPPN for HyperNEAT (figure 2).
In this way, HyperNEAT acts as a reinforcement learning approach that
determines the best features to extract for another machine learning approach
to maximize performance on the task.

Figure 2: HyperNEAT Feature Learning. To learn features, HyperNEAT trains
CPPNs (1) that generate the connectivity for a defined ANN substrate (2). The
ANN substrate processes the inputs from a data set to produced a set of
features (3). These features are given to another machine learning algorithm
(4) that learns to perform the task (e.g. image classification). Machine
learning then produces a solution that is evaluated on testing data (5). The
performance of the solution on data not seen during training provides the
fitness score of the CPPN for HyperNEAT. In this way, HyperNEAT not only
discovers better learning features, but also better features for
generalization.

The second modification introduces alternative architectures to HyperNEAT.
Traditionally, HyperNEAT produces the weight pattern for a ANN substrate that
is feed forward, fully connected, and containing only sigmoid activation
functions. However, the only information HyperNEAT sees about the substrate is
the geometric coordinates of the neurons, thus HyperNEAT could be applied to
any graph like structure wherein the nodes have coordinates associated with
them. Because CNNs have been demonstrably successful in a number of domains,
HyperNEAT is extended to include this alternative ANN architecture. Each of
these extensions to HyperNEAT is explored in experiments described in the next
section.

##  4 Experimental Setup

These investigations are conducted on the MNIST dataset, which is a popular
benchmark for machine learning because it is a challenging and relevant real
world problem. MNIST is a set of $28\times 28$ pixel images of handwritten
digits ($0-9$), separated into 60,000 training images and 10,000 testing
images. The goal for machine learning is to correctly classify the handwritten
digit contained within each image. To this end, we explore HyperNEAT with four
combinations of experimental settings on the benchmark MNIST dataset. These
settings are traditional HyperNEAT ANN architecture (feed-forward, fully-
connection, sigmoid activation functions) or CNN architecture and HyperNEAT
training the solution (an ANN for classification) or feature extractor (ANN
that transforms images into features).

The architecture for HyperNEAT in these experiments is a multi-layer neural
network wherein the layers travel along the $z$-axis, each layer consists of a
number of features ($f$-axis), and each feature has a constellation of neurons
on the $x,y$-plane corresponding to pixel locations. Thus the CPPN represents
points in an eight-dimensional Hyper-cube that correspond to connections in
the four-dimensional substrate and each neuron is located at a particular
($x,y,f,z$) coordinate. Each layer is presented by a triple ($X,Y,F$), wherein
$F$ is the number of features and $X,Y$ are the pixel dimensions. Thus the
input layer is ($28,28,1$), because the input image is $28\times 28$ and
contains only grayscale pixel values. The traditional HyperNEAT architecture
for these experiments is a seven layer neural network with one input, one
output, and five hidden layers, represented by the triples ($28,28,1$),
($16,16,3$), ($8,8,3$), ($6,6,8$), ($3,3,8$), ($1,1,100$), ($1,1,64$), and
($1,1,10$). Each layer in the traditional HyperNEAT ANN architecture is fully
connected to the adjacent layers and each neuron has a bipolar sigmoid
activation function. The CNN architecture to which HyperNEAT is applied
replicates the LeNet-5 architecture that was previously applied to the MNIST
dataset [38].

To operate as a feature extractor, the above architectures are modified such
that the ANN substrate is cut off before the last hidden layer, that is, for
the traditional ANN architecture the ($1,1,100$) layer becomes the new output
layer and for the CNN architecture the ($1,1,120$) layer becomes the outputs.
Thus each image is passed through the ANN substrate architecture to produce an
associated feature vector. These feature vectors are then given to backward
propagation to train an ANN with an architecture identical to the architecture
that was removed from the substrates. The next section presents the results of
these HyperNEAT variants.

##  5 Results

For each of these experiments, results are averaged over 30 independent runs
of 2500 generations with a HyperNEAT population size of 256. The fitness score
is the sum of the true positive rate, true negative rate, positive predictive
value, negative predictive value, and accuracy for each class plus the
fraction correctly classified overall and the inverse of the mean square error
from the correct label outputs. Each run randomly selects 300 images, evenly
spread across the classes, from the MNIST training set for training. For
regular HyperNEAT (i.e. not feature learning), fitness is determined by
applying the ANN substrate to the training images. For HyperNEAT feature
learning an additional 1000 images are randomly selected (again evenly spread
across classes) from the MNIST training set. Backward propagation training is
run for 250 epochs on the 300 selected images and then tested on the different
set of 1000 images. The testing performance of the backward propagation
trained ANN becomes the fitness of the CPPN for HyperNEAT. After evolution
completes, the generation champions are evaluated on the MNIST testing set. In
the HyperNEAT for feature learning case, the backward propagation trained ANN
learns from the full MNIST training set.

As seen in figure 3, HyperNEAT by itself quickly plateaus at a particular
performance level and only gradually learns, achieving an average fitness
score of $4.2$ by the end of training. The accuracy of these trained solutions
on the testing data is also not impressive, achieving only a $23.9\%$ correct
classifications. By applying HyperNEAT as a feature learner and allowing
backward propagation to train, both fitness score during training and
classification correctness during testing increase to $5.7$ and $58.4\%$,
respectively. It is interesting to note that in both cases, the fitness score
and the correct classifications are not completely correlated, that is,
improvements in the fitness score can lead to decreases in classification
accuracy.

Figure 3: HyperNEAT Performance with Traditional ANN Architecture. HyperNEAT
can learn to classify images by itself; however, learning quickly plateaus at
a low performance and then learning slows, reaching a fitness score of $4.2$
and testing performance of $23.9\%$ correct classifications. By acting as a
feature learner, HyperNEAT does not plateau and achieves improved performance
over HyperNEAT alone, finishing training with a fitness score of $5.7$ and a
testing score of $58.4\%$. Thus HyperNEAT as a feature learner is more
promising for than HyperNEAT alone.

Changing HyperNEAT to learn the weights of a CNN architecture, rather than the
normal ANN architecture of HyperNEAT, does change performance (figure 4). In
this case, HyperNEAT by itself plateaus faster at a lower fitness level,
achieving an average fitness score of $4.1$ by the end of training, but the
accuracy of these trained solutions on the testing data improves to $27.7\%$
correct classifications. On the other hand, feature learning with HyperNEAT on
a CNN architecture significantly improves both metrics, achieving a $7.0$
fitness score and a $92.1\%$ correct classifications. An example of the
feature maps generated by these HyperNEAT champions can be seen in figure 5.

Figure 4: HyperNEAT Performance with CNN Architecture. The CNN architecture
has a small effect on HyperNEAT only performance, lowering the fitness
performance plateau to $4.1$, but increasing testing performance of $27.7\%$
correctness. Shifting to CNN architecture significantly improves HyperNEAT’s
ability as a feature learner, allowing HyperNEAT to find features that achieve
a fitness score $7.0$ and a testing score of $92.1\%$. Thus HyperNEAT can
learn effective levels of performance given careful selection of the ANN
substrate.

(a) (b)

Figure 5: Visualization of Feature Maps Generated by HyperNEAT. Example
feature maps are shown for the digits nine (a) and three (b). Interestingly,
there is a distinct pattern along the feature dimension (top to bottom),
demonstrating patterns in the geometry of the feature maps.

##  6 Discussion and Future Work

Evolution is a significant factor in the creation of biological systems,
including the visual cortex [1, 2]. However, biological neural networks are
often deep architectures and conventional NeuroEvolution approaches have been
challenged in effectively training ANNs order of magnitude smaller than those
found in nature. Emerging research from generative and developmental systems
has provided an answer in the form of HyperNEAT. HyperNEAT can effectively
learn weight patterns for an ANN substrate by training CPPNs, an indirect
encoding that computes weights as a function of geometry. The challenge for
HyperNEAT is that, by training an indirect encoding, the ability to control
precise weights is diminished, thereby creating difficulties in making fine
adjustments for tasks such as classification. That is, a change in the
indirect encoding will cause changes across the entire weight pattern, when a
change to a single weight is needed. This challenge can be addressed by
applying HyperNEAT as a feature learner for another ML approach that then
makes the fine adjustments for the task, as shown in this paper. Indeed, the
results in this paper demonstrate that HyperNEAT successfully learn features
to train a backward propagation trained ANN.

Interestingly, changing to the convolutional neural network architecture had a
significant impact on HyperNEAT’s performance. Two interesting questions
arise: (1) Are there more effective architecture choices? and; (2) How do can
they be discovered? A path to answering these questions may be an extension of
HyperNEAT known as Evolvable Substrate HyperNEAT (ES-HyperNEAT; [39]). In ES-
HyperNEAT, the ANN substrate is not defined a priori (except for inputs and
outputs); instead, the pattern generated by the CPPN determines the placement
neurons in the hidden layers.

An exciting implication of this work is that evolution provides a means to
learn a representation that is well-suited for the target machine learning
approach. Through reinforcement learning, evolution can measure fitness as a
function of how well an approach performs on the generated representation.
Because the fitness measure does not depend on a particular error signal or
gradient, multiple metrics can be incorporated. For example, the results in
the paper incorporated several classification characteristics into the fitness
function, such as measures of true positives, true negatives, false positives,
and false negatives. By combining measures, evolution can explore multiple
paths through the different metrics. In addition, the many candidate solutions
that evolution generates will perform differently from each other and these
solutions may be combined to create an enhanced feature set. Finally, because
the fitness measure can be based upon performance of the trained solution on
data not seen by ML approach, the discovered features may encourage learning
that generalizes. Thus this approach provides anopportunity to encourage
representations that enhance generalizability.

##  7 Conclusion

This paper investigated deep learning through NeuroEvolution. While NE has
been limited in the size of ANNs that could be effectively trained in the
past, novel algorithms that operate on indirect encodings, such as HyperNEAT,
allow effective evolution of large ANNs. Prior work with HyperNEAT has shown
promise in simple vision tasks that operate on a raw visual field, but with
non-deep architectures. By itself, HyperNEAT struggles to find ANNs that
perform well in image classification; however, HyperNEAT demonstrates an
effective ability to act as a feature extractor by being combined with
backward propagation. Thus HyperNEAT provides a potentially interesting path
for combining reinforcement learning and supervised learning in image
classification, as evolution and lifetime learning combine to create the
capabilities in biological neural networks.

#### Acknowledgments

This work was supported and funded by the SSC Pacific Naval Innovative Science
and Engineering (NISE) Program.

## References

  * [1] Q. V. Le, L. A. Isbell, J. Matsumoto, M. Nguyen, E. Hori, R. S. Maior, C. Tomaz, A. H. Tran, T. Ono, and H. Nishijo, “Pulvinar neurons reveal neurobiological evidence of past selection for rapid detection of snakes,” Proceedings of the National Academy of Sciences, vol. 110, November 2013\.
  * [2] T. Gliga and G. Dehaene-Lambertz, “Structural encoding of body and face in human infants and adults,” Journal of Cognitive Neuroscience, vol. 17, August 2005.
  * [3] X. Yao, “Evolving artificial neural networks,” Proceedings of the IEEE, vol. 87, no. 9, pp. 1423–1447, 1999.
  * [4] E. R. Kandel, J. H. Schwartz, and T. M. Jessell, Principles of Neural Science.  New York: McGraw-Hill, fourth ed., 2000.
  * [5] M. J. Zigmond, F. E. Bloom, S. C. Landis, J. L. Roberts, and L. R. Squire, eds., Fundamental Neuroscience.  London: Academic Press, 1999.
  * [6] K. O. Stanley, D. B. D’Ambrosio, and J. Gauci, “A hypercube-based indirect encoding for evolving large-scale neural networks,” Artificial Life, vol. 15, 2009.
  * [7] P. J. Bentley and S. Kumar, “The ways to grow designs: A comparison of embryogenies for an evolutionary design problem,” in Proc. of the Genetic and Evo. Comp. Conf., pp. 35–43, ACM, 1999.
  * [8] L. Hartwell, J. Hopfield, S. Leibler, and A. Murray, “From molecular to modular cell biology,” Nature, vol. 402, 1999.
  * [9] G. Wagner and L. Altenberg, “Complex adapatations and the evolution of evolovability,” Evolution, vol. 50, 1996.
  * [10] J. Gauci and K. O. Stanley, “A case study on the critical role of geometric regularity in machine learning,” in Proc. of the 23rd AAAI Conf. on AI (AAAI-2008), (Menlo Park, CA), AAAI Press, 2008.
  * [11] J. Gauci and K. O. Stanley, “Autonomous evolution of topographic regularities in artificial neural networks,” Neural Computation, p. 38, 2010.
  * [12] K. O. Stanley, “Compositional pattern producing networks: A novel abstraction of development,” Genetic Programming and Evolvable Machines Special Issue on Dev. Sys., vol. 8, no. 2, pp. 131–162, 2007.
  * [13] J. Gauci and K. O. Stanley, “Generating large-scale neural networks through discovering geometric regularities,” in Proc. of the Genetic and Evo. Comp. Conf., (New York, NY), p. 8, ACM, 2007\.
  * [14] O. J. Coleman, Evolving Neural Networks for Visual Processing.  PhD thesis, The University of New South Wales, Kensingtion, Austrailia, 2010.
  * [15] P. Verbancsics and K. O. Stanley, “Constraining connectivity to encourage modularity in hyperneat,” in Proc. of the Genetic and Evo. Comp. Conf. , (New York, NY), ACM Press, 2011\.
  * [16] M. Hausknecht, P. Khandelwal, R. Miikkulainen, and P. Stone, “Hyperneat-gpp: A hyperneat-based atari general game player,” in Proceedings and the Genetic and Evolutionary Computation Conference, (Philadelphia, Pennsylvania), p. 8, ACM Press, July 2012.
  * [17] K. O. Stanley and R. Miikkulainen, “Competitive coevolution through evolutionary complexification,” Journal of Artificial Intelligence Research, vol. 21, pp. 63–100, 2004.
  * [18] T. Aaltonen et al., “Measurement of the top quark mass with dilepton events selected using neuroevolution at CDF,” Physical Review Letters, 2009.
  * [19] S. Whiteson and D. Whiteson, “Stochastic optimization for collision selection in high energy physics,” in IAAI 2007: Proceedings of the Nineteenth Annual Innovative Applications of Artificial Intelligence Conference, (Vancouver, British Columbia, Canada), AAAI Press, July 2007.
  * [20] L. Cardamone, D. Loiacono, and P. L. Lanzi, “On-line neuroevolution applied to the open racing car simulator,” in Proceedings of the 2009 IEEE Congress on Evolutionary Computation (IEEE CEC 2009), (Piscataway, NJ, USA), IEEE Press, 2009.
  * [21] M. E. Taylor, S. Whiteson, and P. Stone, “Comparing evolutionary and temporal difference methods in a reinforcement learning domain,” in Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2006), (New York, NY), pp. 1321–1328, ACM Press, July 2006.
  * [22] S. Whiteson, “Improving reinforcement learning function approximators via neuroevolution,” in AAMAS ’05: Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, (New York, NY, USA), pp. 1386–1386, ACM, 2005.
  * [23] K. O. Stanley, B. D. Bryant, and R. Miikkulainen, “Real-time neuroevolution in the NERO video game,” IEEE Transactions on Evolutionary Computation Special Issue on Evolutionary Computation and Games, vol. 9, no. 6, pp. 653–668, 2005.
  * [24] D. D’Ambroiso and K. O. Stanley, “Evolving policy geometry for scalable multiagent learning,” in Proceedings of the Ninth International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2010), (New York, NY, USA), p. 8, ACM Press, 2010.
  * [25] D. B. D’Ambrosio and K. O. Stanley, “Generative encoding for multiagent learning,” in Proc. of the Genetic and Evo. Comp. Conf., (New York, NY), ACM Press, 2008.
  * [26] P. Verbancsics and K. O. Stanley, “Evolving static representations for task transfer,” Journal of Machine Learning Research, vol. 11, 2010.
  * [27] K. O. Stanley and R. Miikkulainen, “A taxonomy for artificial embryogeny,” Artificial Life, vol. 9, no. 2, pp. 93–130, 2003.
  * [28] A. M. Turing, “The Chemical Basis of Morphogenesis,” Royal Society of London Philosophical Transactions Series B, vol. 237, pp. 37–72, Aug. 1952\.
  * [29] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, pp. 1798–1828, August 2013.
  * [30] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, pp. 1527–1554, 2006.
  * [31] S. Rifai, Y. N. Dauphin, P. Vincent, Y. Bengio, and X. Muller, “The manifold tangent classifer,” in Advances in Neural Information Processing Systems, 2011.
  * [32] D. Ciresan, U. Meier, J. Masci, and J. Schmidhuber, “Multi-column deep neural network for traffic sign classification,” Neural Networks, vol. 32, pp. 333–338, 2012.
  * [33] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classfication with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, 2012.
  * [34] R. Salakhutdinov and G. E. Hinton, “Deep boltzmann machines,” in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2009.
  * [35] G. E. Hinton and R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006.
  * [36] K. Fukushima and S. Miyake, “Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position,” Pattern recognition, vol. 15, no. 6, pp. 455–469, 1982.
  * [37] D. H. Hubel and T. N. Wiesel, “Receptive field of single neurones in the cat’s striate cortex,” The Journal of Physiology, vol. 148, no. 3, pp. 574–591, 1959.
  * [38] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to docment recognition,” IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  * [39] S. Risi and K. O. Stanley, “An enhanced hypercube-based encoding for evolving the placement, density and connectivity of neurons,” Artificial Life, vol. 18, no. 4, pp. 331–363, 2012.


# Exploring the Sharpened Cosine Similarity

Skyler Wu1,2,4, Fred Lu1,2,3, Edward Raff1,2,3, James Holt1
1Laboratory for Physical Sciences, 2Booz Allen Hamilton,
3 University of Maryland, Baltimore County, 4Harvard University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Convolutional layers have long served as the primary workhorse for image
classification. Recently, an alternative to convolution was proposed using the
Sharpened Cosine Similarity (SCS), which in theory may serve as a better
feature detector. While multiple sources report promising results, there has
not been to date a full-scale empirical analysis of neural network performance
using these new layers. In our work, we explore SCS’s parameter behavior and
potential as a drop-in replacement for convolutions in multiple CNN
architectures benchmarked on CIFAR-10. We find that while SCS may not yield
significant increases in accuracy, it may learn more interpretable
representations. We also find that, in some circumstances, SCS may confer a
slight increase in adversarial robustness.

##  1 Introduction

For decades, convolutional layers have served as the workhorses in neural
network architectures for image classification. Mathematically, a
convolutional layer slides across an input image and computes the dot product
between a signal $s$ and a kernel $k$ (see [1]). However, initial exploration
and discussion by [2] suggest that convolutional layers, while excellent image
filters, may not be very good feature detectors. In addition, [3] argue that
because the dot product operation is unbounded, neural networks using
convolutional layers may be vulnerable to increased model variance, over-
sensitivity, and a lack of generalizability. As such, [3] explored replacing
traditional convolutional layers with cosine similarity. Mathematically, the
cosine similarity (“CosSim") between $s$ and $k$ is defined as follows:

$\operatorname*{\textnormal{CosSim}}(s,k)=\frac{s\cdot k}{\lVert s\rVert\lVert
k\rVert}.$

Importantly, cosine similarity is bounded between $-1$ and $1$, potentially
reducing the variance concerns of standard CNNs. Traditionally, cosine
similarity has been used extensively in text-analysis to quantify the
“similarity" between two documents (see [4]), among other machine learning
tasks (see [5]). For deep-learning networks, [3] found that incorporating
cosine similarity in VGG-type CNNs outperformed standard convolution-based
alternatives using batch, weight, and layer normalization on CIFAR-10/100 and
SVHN.

Building on cosine similarity, [1] proposed that one can “sharpen" the
standard cosine similarity to better discern whether two vectors are similar
by raising the standard cosine similarity to a power $p$, while preserving the
sign of the original cosine value. To prevent numerical instability arising
from the signal $s$ potentially having a near-zero magnitude, [1] presents the
following modified mathematical formulation for Sharpened Cosine Similarity
(SCS), with a small positive scalar $q$ added to $||s||$:

$\operatorname*{\textnormal{SCS}}(s,k)=\textnormal{sign}(s\cdot
k)\Big{|}\frac{s\cdot k}{(\lVert s\rVert+q)\lVert k\rVert}\Big{|}^{p}$

In February 2020, [2] used a one-dimensional input vector to visually
demonstrate that an SCS kernel was a more accurate feature detector than a
standard convolution kernel. Specifically, Rohrer showed that the SCS kernel
more consistently output large values in the presence of a desired feature and
smaller values in the absence of such a feature, than a standard convolution
kernel. The math and examples provided by [2] are convincing that standard
convolutions are better filters than detectors, and the hypothesis that such a
change from convolutions to SCS would improve predictive accuracy is intuitive
and logical.

In this study, we more thoroughly investigate the SCS, exploring accuracy,
efficiency, interpretability, and adversarial robustness as hypotheses of what
may be improved by SCS. Our experiments are repeated using multiple different
current network architectures to see if any benefits of SCS improve with
smaller network size, as hypothesized by current small-scale results of
Rohrer. Our results find some measurable differences between SCS and
convolutional networks, but said differences do not yet rise to a level that
would meaningfully impact current practitioners.

##  2 Review of Related Work

Since the [2] Twitter thread, many have experimented with various
implementations of the SCS layer. Initial results suggest that SCS is very
parameter-efficient, which may be desirable in use-cases with limited compute
or wattage capacities. On MNIST, [6] reported achieving 99% accuracy using an
SCS-based architecture with less than 1.4K parameters. [6] also observed that
SCS networks tended to produce more interpretable weights, work well with
unscaled inputs, and require no normalization. It was also observed that SCS
kernels could extract not only exact matches of desired features, but also
their opposites (i.e., with opposite sign). As such, [6] recommended that SCS
be paired with MaxAbsPool2d (taking the maximum of the absolute values of an
input), in contrast to the traditional MaxPool2d, for optimal performance. On
CIFAR-10, Rohrer published an SCS-based model in his scs-gallery (see [7])
that claimed to achieve 80% accuracy with only 25.2K parameters, an order of
magnitude less than existing architectures with comparable accuracy. [8]
experimented with scaling up SCS networks to larger datasets and leveraging
TPU computations, but found that SCS was computationally significantly slower
to train than convolutional layers. This is primarily because the
exponentiation to the $p^{th}$ power in SCS does not parallelize well to GPUs
and TPUs.

In addition to the properties discussed above, members of the machine learning
community have found that the inclusion of SCS layers does not appear to
produce more accurate models than existing architectures (see [1]). Others
have also found that SCS architectures do not require nonlinear activations
(e.g., ReLU, sigmoid), dropout layers, nor normalization layers (e.g.,
BatchNorm2d) after SCS layers. Some researchers have also tried incorporating
SCS into existing architectures such as ResNet (see [9]), compact transformers
(see [10]), models for ASL classification (see [11]), and GANs (see [12]),
with limited or mixed reported results. However, to the best of our knowledge,
there do not exist any formally published manuscripts on SCS experiments and
behavior, especially in comparison to traditional convolutional layers.

##  3 Materials and Methods

###  3.1 Architectures

For initial exploratory experiments, we selected two network architectures to
serve as our test platforms: first, an SCS network with 100K parameters from
Rohrer’s scs-gallery, which we will call “Rohrer100K" (see [7]); second, the
ResNet18 network from PyTorch’s torchvision library with 11.6 million
parameters. A ResNet18/20 architecture specially designed for CIFAR-10 was
also briefly used when comparing the weight behaviors of SCS layers versus
their convolutional counterparts (see [13]). For follow-on experiments, we
selected four more originally SCS-based architectures from Rohrer’s scs-
gallery that we will call Rohrer25K, Rohrer47K, Rohrer68K, and Rohrer583K
(named after their parameter counts, see [7]). The first two of these were
reported to achieve 80%+ accuracy on CIFAR-10, while the last two were
reported to achieve 90%+ accuracy on standard CIFAR-10. These models were
selected to explore SCS behavior across a wide range of parameter-counts. For
the above architectures, our benchmark was the standard $32\times 32$ CIFAR-10
dataset. To explore the behavior of SCS on higher-resolution images, we also
explored SCS-based ResNet18 variants on CIFAR-10 resized to $224\times 224$.

For each starting architecture, we tested every variant combination of the
following settings: convolutional layer vs. SCS; MaxPool2d vs. MaxAbsPool2d;
ReLU vs. no activation. For our follow-up experiments on ResNet18 ($224\times
224$), in addition to the settings specified above, we also explored variants
with or without BatchNorm2d. When replacing one feature with another (e.g.,
convolutional with SCS), every attempt was made to carry over as many settings
as possible from the original starting architecture (e.g., kernel size, number
of filters, etc.). In addition, for each architecture family, all model
variants using that architecture were trained with the same initial weights.
We hope that such practices allow us to more accurately assess the direct
effects of transplanting SCS into existing architectures. Please see Appendix
A for additional experimental details.

For all model variants, train and test accuracies, losses, and compute times
(in seconds) were recorded per epoch. The norms of the weights and gradients
in each layer were also recorded per epoch for all variants. Vanilla gradient
saliency maps were generated for each model variant to probe model
interpretability. In addition, projected gradient descent (PGD) adversarial
attacks were performed against all model variants to further assess model
interpretability. For variants using SCS layers, we also recorded the $p$
exponentiation parameter values of each SCS layer per epoch.

##  4 Initial Exploratory Experiments on Rohrer100K and ResNet18

We begin by exploring the accuracy and training efficiency, the nature and
parameter behavior, and the interpretability and adversarial robustness of SCS
when used in the Rohrer100K and ResNet18 architectures on $32\times 32$
CIFAR-10. Initial testing suggested that SCS networks tended to require more
epochs of training to achieve their maximum test accuracy. As such, all models
using architectures originally from scs-gallery were trained for 800 epochs,
while all other models were trained for 200 epochs. Please see subsection A.1
for more experimental setting details. Additional initial exploratory
experiments regarding SCS parameter behavior can be found in Appendix C.

###  4.1 Accuracy and Efficiency

Table 1: Test accuracies of initial Rohrer100K model variants on $32\times 32$
CIFAR-10. See Appendix B for ResNet18 results.

Architecture Layer Activation Pooling Dropout Normalization Image Dim. Val.
Acc. Train Time (s) Eval Time (s) Rohrer100K Conv2d None MaxAbsPool2d None
None 32x32 0.4952 22.176 2.113 Rohrer100K Conv2d None MaxPool2d None None
32x32 0.7301 21.997 2.12 Rohrer100K Conv2d ReLU MaxAbsPool2d None None 32x32
0.6496 22.134 2.118 Rohrer100K Conv2d ReLU MaxPool2d None None 32x32 0.8032
22.118 2.126 Rohrer100K SCS None MaxAbsPool2d None None 32x32 0.7761 33.482
1.871 Rohrer100K SCS None MaxPool2d None None 32x32 0.787 33.341 1.791
Rohrer100K SCS ReLU MaxAbsPool2d None None 32x32 0.777 33.523 1.817 Rohrer100K
SCS ReLU MaxPool2d None None 32x32 0.8026 33.381 1.776

From Table 1 above and Table 2 (see Appendix B), we see that within the same
architecture family, the use of SCS layers does not yield a significant
increase in accuracy (approx. $78\%$ for Rohrer100K and $82\%$ for ResNet18)
on CIFAR-10. However, we do confirm that SCS is able to achieve comparable
accuracy performance to standard convolutional layers without the inclusion of
ReLU nonlinear activation functions, while convolutional layers must be paired
with such nonlinear activations for maximum performance. Yet, it appears that
the inclusion of ReLU activations in SCS networks may still provide a small
increase in accuracy performance. Interestingly, in Rohrer100K, we also find
that standard convolutional layers paired with MaxAbsPool2d (and ReLU) incur a
significant dip in accuracy performance. But, this dip does not seem to occur
with ResNet18.

We found that SCS-based networks require significantly more time (seconds per
epoch) to train than their corresponding convolution-based counterparts. This
corroborates best practices and observations that others have found. However,
while all SCS-based ResNet18 variants were slower in evaluation time than
their convolution-based counterparts, all SCS-based Rohrer100K variants were
noticeably faster in evaluation time than their convolution-based
counterparts. This suggests that SCS properties may not unilaterally hold
across all architectures.

###  4.2 Cosine Similarity (Unsharpened)

We hypothesize that SCS does not require nonlinear activations like ReLU to
achieve peak performance because the exponentiation by $p$ (sharpening) in SCS
is effectively an activation function in and of itself. In other words, we
postulate that SCS could be interpreted as a combination of output
normalization (via cosine similarity) and an exponential activation function.
To test this hypothesis, we fix $p=1$ in the SCS layers of Rohrer100K to test
whether (unsharpened) cosine similarity on its own would require an activation
function. The maximum accuracy of cosine similarity coupled with ReLU was
$76.04\%$, while the maximum accuracy of cosine similarity without ReLU was
$56.58\%$.

###  4.3 Weight Norms of SCS Layers

To explore how the normalization and exponentiation components of SCS would
affect the weights used in its convolution component (versus a standard
convolutional layer), we compare the weights of a fully-trained convolution-
based network versus those of a fully-trained SCS-based network. To ensure
that the weights of our fully-trained convolution-based network represented
the maximum performance (reasonably) possible, we decided to use a ResNet18
architecture specifically-optimized for CIFAR-10 by Idelbayev (see [13]), as
opposed to the torchvision variant optimized for ImageNet. We trained two SCS-
based variants, each only replacing the convolutional layers for SCS, and not
altering any other model components. However, one variant was trained using
randomly-initialized starting weights, while the other was trained using pre-
trained weights specifically designed for convolution-layers on CIFAR-10. We
recorded the L2 norms of each SCS layer’s convolution weights over each mini-
batch.

From Figure 2 in Appendix B, we see that for the variant trained from scratch,
the L2 norms of each SCS layer’s weights did not appear to noticeably change
over epoch time. However, for the variant trained from pre-trained weights
originally optimized for standard convolution layers, the L2 norms of the SCS
layer weights significantly monotonically increased over mini-batches. As
such, these results suggest that SCS-based architectures prefer larger weights
than their convolution-based counterparts. The L2 norms reported at mini-batch
0 in the pre-trained plot are precisely the norms found in the fully-trained
convolution-based network. It is possible that SCS may tolerate larger weights
because of the output normalization. Perhaps larger weights may also
contribute to more sensitive feature detection and differentiation.

###  4.4 Saliency Maps

To investigate Pisoni’s observations regarding SCS interpretability, vanilla
gradient saliency maps were generated for convolution and SCS variants of
Rohrer100K and ResNet18, presented as Figures 3 and 4 in Appendix B. In Figure
1 below, we provide one representative example of the saliency maps of a
convolution-based and an SCS-based Rohrer100K corresponding to an airplane
(class 0). We observe that the convolution-based variant yields a “scattered"
saliency map, while the SCS variant yields a more focused map that corresponds
to the triangular-shaped aircraft. Overall, across Rohrer100K and ResNet18,
our initial results suggest that SCS-based model variants tend to learn
representations that are more interpretable and focus on more critical parts
of the image than their convolution-based counterparts.

(a) With convolution

(b) With SCS

(c) Original image

Figure 1: Saliency maps of best convolutional and SCS initial Rohrer100K
variants on CIFAR-10. The left panel shows the convolutional variant with the
highest class-0-specific accuracy ($82\%$): convolutions, with ReLU and
MaxPool2d. The right panel shows the SCS variant with the highest
class-0-specific accuracy ($84\%$): SCS, also with ReLU and MaxPool2d.

###  4.5 Robustness to Adversarial Attacks

To further explore the interpretability of SCS-based networks, we performed
projected gradient descent (PGD) adversarial attacks on all Rohrer100K and
ResNet18 model variants, varying attack strength from $\epsilon=0.001$ to
$\epsilon=0.030$. We posit that architectures with more interpretable
representations (i.e., focus more on true features and signals, and less
noise) should be more robust to adversarial attacks. From Figures 3 and 4 in
Appendix B, we find that SCS-based Rohrer100K and ResNet18 variants are
noticeably more adversarially robust than their convolution-based
counterparts, in terms of experiencing a slower decay in accuracy with
increasing attack strength. The one outlier in the Rohrer100K trials was the
convolution-based variant with MaxAbsPool2d and ReLU, which was also an
outlier in terms of achieving only 64.96% accuracy (while other variants
achieved high 70%s or 80%s). Notably, for ResNet18, the best SCS-based
ResNet18 variant (incidentally, using the best practices suggested by Rohrer
and the ML community) was particularly more adversarially robust than all
convolution-based variants. As such, the initial adversarial experimental
evidence further suggests that the incorporation of SCS layers may produce
more interpretable networks.

##  5 Follow-up Experiments

To investigate the generalizability of our initial SCS findings across
different-sized architectures, we repeated our previous experiments on four
additional models from the scs-gallery: Rohrer25K, Rohrer47K, Rohrer68K,
Rohrer100K (with new randomized initial weights), and Rohrer583K. To further
discern which properties of SCS can be attributed to normalization and which
to exponentiation, we include a third potential layer option: an ablation of
convolutions and SCS that we will call SharpenedSDP. SharpenedSDP stands for
"sharpened strided dot product," and is mathematically equivalent to a
standard convolutional layer raised to a learned $p$ exponent, without the
normalization. From our initial testing, we recalled that using convolutional
layers without ReLU would almost always lead to poor performance. As such, we
no longer tested this combination in the following experiments. To investigate
SCS on higher-resolution images, we also repeated these experiments on
ResNet18 with CIFAR-10 resized to $224\times 224$. In addition to the settings
described above, we included variants with/without BatchNorm2d (present in the
original Resnet18 architecture). We also tried SharpenedSDP with $224\times
224$ ResNet18, but all of these variants encountered vanishing and/or
exploding gradients, so we omit them from discussion.

###  5.1 Accuracy and Efficiency

From Tables 3-7 in Appendix D, we find that our initial conclusions regarding
SCS accuracy and efficiency generally hold strong across Rohrer25K through
Rohrer583K. We also observe that the SharpenedSDP variants do not require
activation functions, further corroborating our hypothesis that the
exponentiation to the $p^{th}$ power is a substitute for nonlinear activation
functions. In addition, we found that SCS and SharpenedSDP variants appear to
consistently perform faster than their convolution-based counterparts on
evaluation time. This is potentially one major advantage of SCS: while SCS-
based networks might be slower to train, they are faster during evaluation.
Finally, across Rohrer 25K through Rohrer583K, we found that model variants
containing both ReLU and MaxAbsPool2d will almost always perform poorly,
regardless of whether we are using SCS, SharpenedSDP, or standard
convolutional layers. From Table 8 in Appendix D, we find that our initial
conclusions regarding SCS accuracy and efficiency also generally hold strong
on $224\times 224$ ResNet18. However, in contrast to Rohrer25K through
Rohrer583K, we do not see a dip in performance associated with the inclusion
of both MaxAbsPool2d and ReLU. We also observed that the inclusion of
BatchNorm2d, holding everything else constant, appears to grant a 3-4%
increase in accuracy.

###  5.2 Saliency Maps and Interpretability

From the saliency maps for Rohrer25K through 583K, as shown in Figures 9-13 in
Appendix D, we see that our initial findings regarding SCS-based model
variants learning sparser and more interpretable representations appear to
hold strong. In addition, we find significant evidence corroborating the
hypothesis that the $p$ exponentiation component in SCS is what creates such
“sparse" saliency maps. We conclude as such because the SharpenedSDP saliency
maps much more consistently resembled the SCS saliency maps than the
convolution ones. From Figure 14 in Appendix D, we find that our conclusions
regarding SCS versus convolutional variants’ saliency maps also extend to
higher-resolution $224\times 224$ images.

####  5.2.1 Adversarial Robustness

Repeating the PGD attack simulations on Rohrer25K - 583K, we found our results
differing significantly from our initial Rohrer100K results. This was
perplexing, because the underlying constructions of all the scs-gallery models
were all very similar. From Figure 15 in Appendix D, we find that the
SharpenedSDP/MaxAbsPool2d/ReLU and convolution/MaxAbsPool2d/ReLU variants
consistently outperformed the top SCS-based variant. However, these two
variants also consistently performed significantly lower in terms of accuracy,
across all four scs-gallery architectures (sometimes only hitting 10%
accuracy). To verify our observations, we simulated PGD attacks against
Rohrer100K again, this time using a different set of randomized initial
weights (see Figure 16 in Appendix D). Again, we found that
SharpenedSDP/MaxAbsPool2d/ReLU and convolution/MaxAbsPool2d/ReLU were the top
adversarially-robust variants (though their original accuracies were dismal).
There did not seem to be a significant difference between the remaining SCS,
SharpenedSDP, and convolution-based variants, but the top performers with
decent accuracy appear to be SharpenedSDP/MaxAbsPool2d/noReLU, followed by
SCS/MaxAbsPool2d/ReLU.

PGD attacks on $224\times 224$ ResNet18 variants also present a different
narrative than the initial PGD attacks on the $32\times 32$ ResNet18 variants.
As shown in Figure 17 in Appendix D, for variants without BatchNorm2d,
convolution-based variants were noticeably more robust to adversarial attack.
However, with BatchNorm2d, the best SCS-based variant maintains a small but
consistent advantage over the convolution-based variants. The differences are
not nearly as pronounced as in the $32\times 32$ CIFAR-10 experiments,
however. Upon closer inspection, it appears that the inclusion or exclusion of
BatchNorm2d simply does not affect SCS-based variants’ adversarial robustness,
but does significantly affect convolution-based variants’ robustness. The
causes of this behavior are not immediately obvious, and further exploration
is necessary.

##  6 Discussion and Future Work

In this manuscript, we find strong evidence supporting the following five
generalizations about SCS performance and behavior. First, SCS does not yield
significantly higher accuracy than standard convolution based layers. Second,
while SCS-based models are almost always slower to train (seconds per epoch)
than their convolution-based counterparts, they frequently benefit from
slightly faster evaluation times (seconds per epoch). Third, SCS can be
interpreted as a combination of convolution, normalization, and nonlinear
exponential activation. As such, they are capable of performing excellently
without additional normalization or nonlinear activations like ReLU. Fourth,
SCS tends to yield more interpretable learned representations than
convolutions. Fifth, in certain environments, SCS also yields potentially more
adversarially robust networks.

However, the above properties were not absolute. This suggests that
characterizing SCS performance may be more analytically and computationally-
involved than solely swapping out convolutional layers and associated
components for SCS layers. It appears that SCS performance and properties can
vary noticeably, depending on the existing architecture and other components
present in the system. Significant future work will be required to establish
accompanying best practices for SCS networks and potentially construct novel
architectures that best leverage SCS’s unique properties. For example,
creating a ResNet bottleneck module optimized for SCS may not be as
straightforward as swapping out convolutional layers for SCS and replacing
MaxPool2d with MaxAbsPool2d.

Given (unsharpened) cosine similarity’s established uses in text modeling, a
natural applied extension is to transplant SCS into existing text modeling
architectures. SCS’s interpretability properties also warrant its
transplanting to malware classification models that currently use standard
convolutions. Another direction is to explore the use of SCS in convolution-
based transformer models. We conclude that SCS is a promising alternative to
convolutional layers, with its own set of unique advantages and drawbacks.

## References

  * Rohrer [2022a] B. Rohrer, “Sharpened cosine similarity: An alternative to convolution in neural networks,” May 2022. [Online]. Available: https://github.com/brohrer/sharpened-cosine-similarity
  * Rohrer [2020] ——, “The thing that has surprised me the most about convolution is that it’s used in neural networks as a feature detector, but it’s pretty bad at detecting features.” Feb 2020. [Online]. Available: https://twitter.com/_brohrer_/status/1232063619657093120?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1232063619657093120%7Ctwgr%5E6c3a18acd4c451876f1acf93a5327d080a9cabc0%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Frpisoni.dev%2Fposts%2Fcossim-convolution%2F
  * Luo et al. [2018] C. Luo, J. Zhan, X. Xue, L. Wang, R. Ren, and Q. Yang, “Cosine normalization: Using cosine similarity instead of dot product in neural networks,” in _International Conference on Artificial Neural Networks_. Springer, 2018, pp. 382–391.
  * Singhal et al. [2001] A. Singhal _et al._ , “Modern information retrieval: A brief overview,” _IEEE Data Eng. Bull._ , vol. 24, no. 4, pp. 35–43, 2001.
  * Tan et al. [2013] P.-N. Tan, M. Steinbach, and V. Kumar, “Data mining cluster analysis: basic concepts and algorithms,” _Introduction to data mining_ , vol. 487, p. 533, 2013.
  * Pisoni [2022] R. Pisoni, “Sharpened cosine distance as an alternative for convolutions,” 2022, https://rpisoni.dev/posts/cossim-convolution/.
  * Rohrer [2022b] B. Rohrer, “Gallery for sharpened cosine similarity,” May 2022. [Online]. Available: https://github.com/brohrer/scs-gallery
  * Nestler [2022] L. Nestler, “To reproduce scs’ incredible results on large-scale datasets, i implemented a tpu-compatible version in pytorch.” Feb 2022. [Online]. Available: https://twitter.com/_clashluke/status/1497092150906941442
  * Batchelor [2022] O. Batchelor, “Training cifar10 with sharpened cosine similarity,” Feb 2022. [Online]. Available: https://github.com/oliver-batchelor/scs_cifar
  * Walton [2022] S. Walton, “Sharpened cosine similarity for compact transformers,” Apr 2022. [Online]. Available: https://github.com/stevenwalton/SCS-CCT
  * Wagner [2022] J. Wagner, “Kaggle notebooks,” Feb 2022. [Online]. Available: https://github.com/DrJohnWagner/Kaggle-Notebooks
  * Zimonitrome [2022] Zimonitrome, “Generative adversarial network using sharpened cosine similarity,” Feb 2022. [Online]. Available: https://github.com/zimonitrome/scs_gan
  * Idelbayev [2018] Y. Idelbayev, “Proper ResNet implementation for CIFAR10/CIFAR100 in PyTorch,” 2018. [Online]. Available: https://github.com/akamaster/pytorch_resnet_cifar10
  * Smith and Topin [2017] L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” 2017. [Online]. Available: https://arxiv.org/abs/1708.07120
  * Raff and Farris [2022] E. Raff and A. L. Farris, “A Siren Song of Open Source Reproducibility,” in _ML Evaluation Standards Workshop at ICLR 2022_ , 2022. [Online]. Available: https://arxiv.org/abs/2204.04372
  * Raff [2021] E. Raff, “Research Reproducibility as a Survival Analysis,” in _The Thirty-Fifth AAAI Conference on Artificial Intelligence_ , 2021, arXiv: 2012.09932. [Online]. Available: http://arxiv.org/abs/2012.09932
  * Raff [2019] ——, “A Step Toward Quantifying Independently Reproducible Machine Learning Research,” in _NeurIPS_ , 2019, arXiv: 1909.06674. [Online]. Available: http://arxiv.org/abs/1909.06674
  * Akiba et al. [2019] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” in _Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2019.

##  Appendix A Additional Experimental Details

To standardize model training, unless otherwise specified, we used an Adam
optimizer, coupled with a maximum learning rate of $0.01$ and a OneCycleLR
learning rate scheduler (see [14]). RandomCrop and RandomHorizontalFlip data
augmentations were also applied to the training set. Such settings were
directly borrowed from training scripts in Rohrer’s scs-gallery. Initial
experiments on Rohrer100K suggested that weight decay did not noticeably
affect SCS-based architectures’ performance, so for all results presented,
weight decay was set to $0$. For reproducibility [15, 16, 17] and to rule out
the effects of different weight initializations on model performance, we
ensured that all variants of the same base architecture (e.g., ResNet18 on
$32\times 32$) were trained with the same set of randomly-initialized weights.

All experiments were performed using PyTorch 1.11/1.12 on Linux. The hardware
available were 8 NVIDIA Tesla P100 GPUs with 16 GB of RAM each, and 8 NVIDIA
Tesla V100 GPUs with 32 GB of RAM each. We gratefully acknowledge the use of
the Sharpened Cosine Similarity and MaxAbsPool2d PyTorch implementations
borrowed from Rohrer’s public repository (see [1]).

###  A.1 Additional Details on Initial Exploratory Experiments

We only tested combinations of convolution or SCS, ReLU or no activation, and
MaxPool2d or MaxAbsPool2d. We retained the BatchNorm2d and AdaptiveAvgPool
layers in all tested ResNet18 variants in this section.

The accuracy results for ResNet18 in section 4 were obtained using a
torchvision ResNet18 originally designed for ImageNet, but we replaced the
final fully-connected layer to account for the 10 classes in CIFAR-10. During
initial experimentation, we also tried transfer learning with SCS networks by
starting ResNet18 training using pre-trained ResNet18 weights for ImageNet. We
found that there was no significant difference between model variants trained
from scratch or using pre-trained weights. As such, we only report results
using randomly-initialized starting weights.

##  Appendix B Tables and Additional Figures from Initial Exploratory
Experiments

Table 2: Test accuracies of initial ResNet18 model variants on $32\times 32$
CIFAR-10.

Architecture Layer Activation Pooling Dropout Normalization Image Dim. Val.
Acc. Train Time (s) Eval Time (s) ResNet18 Conv2d None MaxAbsPool2d None
BatchNorm2d 32x32 0.6027 28.816 2.217 ResNet18 Conv2d None MaxPool2d None
BatchNorm2d 32x32 0.7146 26.872 2.113 ResNet18 Conv2d ReLU MaxAbsPool2d None
BatchNorm2d 32x32 0.8225 28.067 2.245 ResNet18 Conv2d ReLU MaxPool2d None
BatchNorm2d 32x32 0.8219 26.478 2.141 ResNet18 SCS None MaxAbsPool2d None
BatchNorm2d 32x32 0.8134 86.227 3.176 ResNet18 SCS None MaxPool2d None
BatchNorm2d 32x32 0.8258 83.794 3.089 ResNet18 SCS ReLU MaxAbsPool2d None
BatchNorm2d 32x32 0.8306 85.369 3.075 ResNet18 SCS ReLU MaxPool2d None
BatchNorm2d 32x32 0.8241 83.711 2.908

###  B.1 Weight Norms of SCS Layers

(a) From scratch (b) From pre-trained weights for nn.Conv2d

Figure 2: Magnitudes of Idelbayev ResNet18 weights over mini-batches.

###  B.2 Saliency Maps and Interpretability

(a) Saliency Maps

(b) Original Image

Figure 3: Saliency maps of initial Rohrer100K variants on CIFAR-10.

(a) Saliency Maps

(b) Original Image

Figure 4: Saliency maps of initial ResNet18 variants on CIFAR-10.

###  B.3 Adversarial Robustness

Figure 5: PGD robustness of Rohrer100K variants on CIFAR-10 ($32\times 32$,
initial testing). Figure 6: PGD robustness of ResNet18 variants on CIFAR-10
($32\times 32$, initial testing).

##  Appendix C Additional Initial Experiments

###  C.1 P-values of SCS Layers

We also explored the behavior of the $p$ values in a SCS layer over epoch
time. Using a fixed learning rate of $1.0\times 10^{-3}$, coupled with a
weight decay of $1.0\times 10^{-5}$, we plotted the learned values of $p$ for
the kernels in the first SCS layer of Rohrer100K (with MaxAbsPool2d, without
ReLU). From Figure 7 below, we find that there are not many dynamics of note.
Monotonicity does not appear to hold.

Figure 7: Learned values of $p$ in first SCS layer for Rohrer100K on CIFAR-10
($32\times 32$, Initial Testing).

###  C.2 Optuna Hyperparameter Tuning of Fixed-p Hyperparameter

We also wished to investigate whether certain values of $p$ were more amenable
to SCS performance than others. As a proxy, we used a custom SCS layer with
$p$ fixed to a certain specified value. We swapped out the original layers in
both Rohrer100K and torchvision’s ResNet18 for these fixed-$p$ SCS layers, and
preserved all other existing components. For simplicity, we forced all SCS
layers in each network to share the same initialized and fixed value of $p$.
We used the Optuna Bayesian hyperparameter tuning utility to find the most
optimal values of $p$ (see [18]). Since multiple initial values of $p$ needed
to be tested, we restricted each Rohrer100K trial to 600 epochs (versus the
standard 800 epochs) and each ResNet18 trial to 100 epochs (versus the
standard 200 epochs). We also enabled pruning to terminate non-promising
$p$-value trials early.

(a) Rohrer100K

(b) ResNet18

Figure 8: Optuna parallel coordinate plots of $p$ value and accuracy.

Optuna results across both Rohrer100K and ResNet18 suggest that all tested
values of $p$ above or near $1.0$ appear to yield good performance. This
suggests that the specific value of $p$ may not be the most important –
rather, it is the mere presence of the (increasing) exponentiation operation
that enables successful SCS performance.

##  Appendix D Tables and Figures From Follow-up Experiments

###  D.1 Accuracy and Efficiency

Table 3: Test accuracies of Rohrer25K model variants on $32\times 32$
CIFAR-10.

Architecture Layer Activation Pooling Dropout Normalization Image Dim. Val.
Acc. Train Time (s) Eval Time (s) Rohrer25K Conv2d ReLU MaxAbsPool2d None None
32x32 0.1301 94.09 9.456 Rohrer25K Conv2d ReLU MaxPool2d None None 32x32
0.7609 92.674 9.379 Rohrer25K SCS None MaxAbsPool2d None None 32x32 0.7745
132.628 8.104 Rohrer25K SCS None MaxPool2d None None 32x32 0.7705 131.594
8.092 Rohrer25K SCS ReLU MaxAbsPool2d None None 32x32 0.2867 131.643 8.2
Rohrer25K SCS ReLU MaxPool2d None None 32x32 0.7682 131.877 8.132 Rohrer25K
SharpenedSDP None MaxAbsPool2d None None 32x32 0.7719 127.797 8.038 Rohrer25K
SharpenedSDP None MaxPool2d None None 32x32 0.7818 126.883 7.882 Rohrer25K
SharpenedSDP ReLU MaxAbsPool2d None None 32x32 0.1 128.575 8.056 Rohrer25K
SharpenedSDP ReLU MaxPool2d None None 32x32 0.7774 127.716 8.008

Table 4: Test accuracies of Rohrer47K model variants on $32\times 32$
CIFAR-10.

Architecture Layer Activation Pooling Dropout Normalization Image Dim. Val.
Acc. Train Time (s) Eval Time (s) Rohrer47K Conv2d ReLU MaxAbsPool2d None None
32x32 0.1 92.919 9.233 Rohrer47K Conv2d ReLU MaxPool2d None None 32x32 0.7978
92.469 9.274 Rohrer47K SCS None MaxAbsPool2d None None 32x32 0.7671 123.82
8.564 Rohrer47K SCS None MaxPool2d None None 32x32 0.7893 123.125 8.425
Rohrer47K SCS ReLU MaxAbsPool2d None None 32x32 0.2513 125.136 8.643 Rohrer47K
SCS ReLU MaxPool2d None None 32x32 0.7816 124.303 8.554 Rohrer47K SharpenedSDP
None MaxAbsPool2d None None 32x32 0.7866 121.93 8.155 Rohrer47K SharpenedSDP
None MaxPool2d None None 32x32 0.7919 121.683 8.017 Rohrer47K SharpenedSDP
ReLU MaxAbsPool2d None None 32x32 0.652 123.186 8.252 Rohrer47K SharpenedSDP
ReLU MaxPool2d None None 32x32 0.7972 121.878 8.154

Table 5: Test accuracies of Rohrer68K model variants on $32\times 32$
CIFAR-10.

Architecture Layer Activation Pooling Dropout Normalization Image Dim. Val.
Acc. Train Time (s) Eval Time (s) Rohrer68K Conv2d ReLU MaxAbsPool2d None None
32x32 0.6762 92.487 9.3 Rohrer68K Conv2d ReLU MaxPool2d None None 32x32 0.836
93.161 9.424 Rohrer68K SCS None MaxAbsPool2d None None 32x32 0.8032 129.316
8.253 Rohrer68K SCS None MaxPool2d None None 32x32 0.8103 128.442 7.998
Rohrer68K SCS ReLU MaxAbsPool2d None None 32x32 0.7594 128.578 8.031 Rohrer68K
SCS ReLU MaxPool2d None None 32x32 0.8258 129.175 8.093 Rohrer68K SharpenedSDP
None MaxAbsPool2d None None 32x32 0.8088 127.146 7.745 Rohrer68K SharpenedSDP
None MaxPool2d None None 32x32 0.8261 127.014 7.747 Rohrer68K SharpenedSDP
ReLU MaxAbsPool2d None None 32x32 0.7503 127.479 7.763 Rohrer68K SharpenedSDP
ReLU MaxPool2d None None 32x32 0.8303 128.185 7.667

Table 6: Test accuracies of Rohrer100K model variants on $32\times 32$
CIFAR-10 (new set of initial weights).

Architecture Layer Activation Pooling Dropout Normalization Image Dim. Val.
Acc. Train Time (s) Eval Time (s) Rohrer100K Conv2d ReLU MaxAbsPool2d None
None 32x32 0.7307 93.024 9.34 Rohrer100K Conv2d ReLU MaxPool2d None None 32x32
0.8347 91.761 9.243 Rohrer100K SCS None MaxAbsPool2d None None 32x32 0.818
135.934 7.738 Rohrer100K SCS None MaxPool2d None None 32x32 0.8178 134.931
7.655 Rohrer100K SCS ReLU MaxAbsPool2d None None 32x32 0.7848 134.43 7.803
Rohrer100K SCS ReLU MaxPool2d None None 32x32 0.8252 132.427 7.702 Rohrer100K
SharpenedSDP None MaxAbsPool2d None None 32x32 0.8206 130.617 7.374 Rohrer100K
SharpenedSDP None MaxPool2d None None 32x32 0.8376 132.427 7.503 Rohrer100K
SharpenedSDP ReLU MaxAbsPool2d None None 32x32 0.7605 132.941 7.481 Rohrer100K
SharpenedSDP ReLU MaxPool2d None None 32x32 0.8243 132.174 7.342

Table 7: Test accuracies of Rohrer583K model variants on $32\times 32$
CIFAR-10.

Architecture Layer Activation Pooling Dropout Normalization Image Dim. Val.
Acc. Train Time (s) Eval Time (s) Rohrer583K Conv2d ReLU MaxAbsPool2d None
None 32x32 0.731 95.824 9.544 Rohrer583K Conv2d ReLU MaxPool2d None None 32x32
0.8587 94.684 9.404 Rohrer583K SCS None MaxAbsPool2d None None 32x32 0.8543
133.319 6.823 Rohrer583K SCS None MaxPool2d None None 32x32 0.8634 132.682
6.559 Rohrer583K SCS ReLU MaxAbsPool2d None None 32x32 0.8227 130.912 6.631
Rohrer583K SCS ReLU MaxPool2d None None 32x32 0.8611 133.041 6.69 Rohrer583K
SharpenedSDP None MaxAbsPool2d None None 32x32 0.8319 131.443 6.271 Rohrer583K
SharpenedSDP None MaxPool2d None None 32x32 0.8585 131.557 6.16 Rohrer583K
SharpenedSDP ReLU MaxAbsPool2d None None 32x32 0.7679 131.32 6.32 Rohrer583K
SharpenedSDP ReLU MaxPool2d None None 32x32 0.8471 131.278 6.297

Table 8: Test accuracies of ResNet18 model variants on $224\times 224$
CIFAR-10.

Architecture Layer Activation Pooling Dropout Normalization Image Dim. Val.
Acc. Train Time (s) Eval Time (s) ResNet18 Conv2d ReLU MaxAbsPool2d None
BatchNorm2d 224x224 0.9287 156.556 26.798 ResNet18 Conv2d ReLU MaxAbsPool2d
None None 224x224 0.8835 160.957 23.954 ResNet18 Conv2d ReLU MaxPool2d None
BatchNorm2d 224x224 0.9255 153.939 26.491 ResNet18 Conv2d ReLU MaxPool2d None
None 224x224 0.8928 162.47 24.621 ResNet18 SCS None MaxAbsPool2d None
BatchNorm2d 224x224 0.9164 192.729 27.624 ResNet18 SCS None MaxAbsPool2d None
None 224x224 0.9006 191.534 27.098 ResNet18 SCS None MaxPool2d None
BatchNorm2d 224x224 0.905 192.184 25.025 ResNet18 SCS None MaxPool2d None None
224x224 0.8972 188.621 26.262 ResNet18 SCS ReLU MaxAbsPool2d None BatchNorm2d
224x224 0.9219 193.221 27.219 ResNet18 SCS ReLU MaxAbsPool2d None None 224x224
0.8934 193.492 25.573 ResNet18 SCS ReLU MaxPool2d None BatchNorm2d 224x224
0.9191 192.479 24.804 ResNet18 SCS ReLU MaxPool2d None None 224x224 0.8853
193.119 22.343

###  D.2 Saliency Maps and Interpretability

In this section, we only display saliency maps for model variants that
achieved an overall accuracy of $60\%$ or greater. Models achieving lower
accuracy than $60\%$ may not be representative of their layers’ overall
behaviors.

(a) Saliency Maps

(b) Original Image

Figure 9: Saliency maps of Rohrer25K variants on CIFAR-10. Note how both the
SCS and SharpenedSDP saliency maps tend to be significantly more “sparse"
compared to the noisy convolutional map. This suggests that the $p$
exponentiation is what decisively determines saliency map “sparsity."

(a) Saliency Maps

(b) Original Image

Figure 10: Saliency maps of Rohrer47K variants on CIFAR-10. We find that
SharpenedSDP may produce saliency maps that are noticeably sparser than SCS,
but not immediately interpretable. More thorough investigation is needed to
determine what causes the SharpenedSDP variants to focus on the bottom left of
the image of the ship presented. Regardless, it appears that the sharpening
process is the decisive component here.

(a) Saliency Maps

(b) Original Image

Figure 11: Saliency maps of Rohrer68K variants on CIFAR-10. It appears that
all of the model variants with sharpening (SCS and SharpenedSDP) seem to
capture the posterior of the deer and arguably some antlers (specifically,
SCS). The convolutional variant does not appear to capture any human-
interpretable features in the deer image. We note how some of the SCS and
SharpenedSDP saliency maps are almost indistinguishable. It is possible that
exponentiation by $p$ serves to emphasize the signal.

(a) Saliency Maps

(b) Original Image

Figure 12: Saliency maps of Rohrer100K variants on CIFAR-10. We see that 3 out
of the 5 SharpenedSDP model variants clearly captured the right wing and
stabilizer in the fighter jet image. All convolutional and SCS variants did
not appear to capture any meaningful, human-interpretable features in the
image. Such a result (and many similar results encountered) suggest that
SharpenedSDP itself may be a promising area of future exploration. Across
examples, it appears that SharpenedSDP coupled with no activation tends to be
a very promising combination.

(a) Saliency Maps

(b) Original Image

Figure 13: Saliency maps of Rohrer583K variants on CIFAR-10. The purpose of
this set of saliency maps is to show a case where all three layer-types
produce saliency maps that look very similar to each other. This suggests
that, at least theoretically, it is possible for various layer types to
converge on the same identifying features. It is clear that all model variants
(except the first) captured the silhouette of the bird.

(a) Saliency Maps

(b) Original Image

Figure 14: Saliency maps of ResNet18 variants on $224\times 224$ CIFAR-10
detecting a B-2 Spirit aircraft. Observe how the 4 convolution-based variants
do not seem to produce human-interpretable saliency maps for the B-2 stealth
bomber. However, all 8 SCS variants produced saliency maps that were extremely
similar to each other, all capturing the jagged bottom edge of the aircraft.
The SCS-based variants’ saliency maps were also significantly sparser than
that of the convolution-based variants.

###  D.3 Adversarial Robustness

(a) Rohrer25K

(b) Rohrer47K

(c) Rohrer68K

(d) Rohrer583K

Figure 15: PGD robustness of scs-gallery architecture variants on CIFAR-10
($32\times 32$). Figure 16: PGD robustness of Rohrer100K variants on CIFAR-10
($32\times 32$, additional testing).

(a) Without BatchNorm2d

(b) With BatchNorm2d

Figure 17: PGD robustness of ResNet18 variants on $224\times 224$ CIFAR-10.