# Neural Loss Function Evolution for Large-Scale
Image Classifier Convolutional Neural Networks

Brandon Morgan  Dean Hougen

###### Abstract

For classification, neural networks typically learn by minimizing cross-
entropy, but are evaluated and compared using accuracy. This disparity
suggests neural loss function search (NLFS), the search for a drop-in
replacement loss function of cross-entropy for neural networks. We apply NLFS
to image classifier convolutional neural networks. We propose a new search
space for NLFS that encourages more diverse loss functions to be explored, and
a surrogate function that accurately transfers to large-scale convolutional
neural networks. We search the space using regularized evolution, a mutation-
only aging genetic algorithm. After evolution and a proposed loss function
elimination protocol, we transferred the final loss functions across multiple
architectures, datasets, and image augmentation techniques to assess
generalization. In the end, we discovered three new loss functions, called
_NeuroLoss1_ , _NeuroLoss2_ , and _NeuroLoss3_ that were able to outperform
cross-entropy in terms of a higher mean test accuracy as a simple drop-in
replacement loss function across the majority of experiments.

Loss Function Evolution, Genetic Algorithm, AutoML, MetaLearning, Machine
Learning, ICML

##  1 Introduction

Convolutional neural networks (CNNs) trained for classification problems on
balanced datasets are typically evaluated using accuracy. However, CNNs learn
in these scenarios by minimizing cross-entropy, a loss metric from information
theory that measures how similar two probability distributions are concerning
a random variable. Minimizing cross-entropy reduces the sum of entropies
across each discrete class between the actual distribution and the prediction.
By minimizing this distributional difference, it is believed that the model’s
output will highly correlate with the evaluation metric, accuracy. In
practice, this typically holds true; however, a perfect linear mathematical
relationship between the two has not been shown. As a result, other possible
loss functions could exist that might yield better accuracy than cross-entropy
when minimizing. Searching for other candidate loss functions can be referred
to as _neural loss function search_ (NLFS).

In this work, the search for a drop-in replacement loss function for cross-
entropy on _large-scale CNNs_ (those containing millions of parameters or
more) for image classification was performed. Each candidate loss function was
evaluated on the CIFAR-10 dataset while using _regularized evolution_ (Real et
al., 2019) to explore the search space. The final loss functions were
transferred across multiple architectures, datasets, and image augmentation
techniques to assess generalization.

Our summarized contributions are: (1) we propose a new search space for NLFS,
allowing for greater exploration of possible loss functions for classification
problems; (2) we propose a surrogate function that correlates well to
transferable loss functions for large-scale CNNs, even when massive
regularization is present; and (3) we discover and present three new loss
functions, _NeuroLoss1_ , _NeuroLoss2_ , and _NeuroLoss3_ , that are capable
of surpassing cross-entropy as a drop-in replacement across multiple
architectures, datasets, and image augmentation techniques.

##  2 Related Work

NLFS is a new but rapidly growing domain (Chebotar et al., 2019; Gonzalez &
Miikkulainen, 2020, 2021; Liu et al., 2021; Gao et al., 2021, 2022; Gu et al.,
2022; Li et al., 2022; Raymond et al., 2022), that originated from the success
of neural architecture search (NAS) (Stanley & Miikkulainen, 2002; Zoph & Le,
2016; Zoph et al., 2018; Real et al., 2019; Elsken et al., 2019) and AutoML
(He et al., 2021). NLFS has been explored for various tasks using two primary
representations: computational graphs (Gonzalez & Miikkulainen, 2020; Liu et
al., 2021; Gu et al., 2022; Li et al., 2022; Raymond et al., 2022) and Taylor
polynomials (Gonzalez & Miikkulainen, 2021; Leng et al., 2022). Here, we focus
on computational-graph-based representations.

Gonzalez & Miikkulainen (2020) evolved a loss function for classification and
regression using genetic programming (GP); the function’s coefficients were
then fine-tuned using covariance-matrix adaptation evolutionary strategy (CMA-
ES). Liu et al. (2021) evolved a loss function specifically for object
detection. Raymond et al. (2022) found a loss function for classification
problems using both GP and a gradient-based local search algorithm. Instead of
using CMA-ES to evolve the coefficients of the loss function, they performed
meta-learning on the coefficients through gradient descent by comparing with
cross-entropy. Li et al. (2022) evolved a general loss function for generic
tasks in semantic segmentation, object detection, and instance segmentation
using GP. Gu et al. (2022) evolved margin-based softmax loss functions for the
specific task of person re-identification using GP.

In this work, we conduct NLFS for image classification, rather than object
detection (Liu et al., 2021), image segmentation (Li et al., 2022), or person
re-identification (Gu et al., 2022). We expand on previous image
classification NLFS work (Gonzalez & Miikkulainen, 2020; Raymond et al., 2022)
by (1) greatly increasing the allowed operations, (2) using a computational-
graph representation rather than grammar-based GP, (3) ensuring our surrogate
function transfers effectively to large-scale CNNs with state-of-the-art image
regularization techniques, and (4) not biasing our loss functions toward
cross-entropy through meta-learning.

##  3 Methodology

In this section, the methodology behind the proposed search space, loss
function initialization, loss function integrity check, proposed surrogate
function, initial population size de-couplization, loss function elimination
protocol, and software implementation details are explained.

###  3.1 Search Space

The proposed search space for the loss functions uses a computational graph,
as with previous work (Gonzalez & Miikkulainen, 2020; Liu et al., 2021; Gu et
al., 2022; Li et al., 2022; Raymond et al., 2022). However, instead of using
the grammar-based representation of those works, a derivative of the NASNet
search space (Zoph et al., 2018) is used.

The NASNet search space optimizes cell-like architectures containing free-
floating nodes called _hidden state nodes_. These nodes receive argument
connections, perform an operation, and output a value to be received by other
nodes. We propose a derivative of this format specifically for loss functions.
In addition, our proposed search space expands on previous research by greatly
expanding on the possible choices for operations, as previous research use
only a few, simple operations (Gonzalez & Miikkulainen, 2020; Liu et al.,
2021; Gu et al., 2022; Li et al., 2022; Raymond et al., 2022).

Tree-based grammars are known to suffer two primary issues: bloating (Luke &
Panait, 2006) and redundancy (Neshatian & Zhang, 2009). _Bloating_ occurs when
the average size of a tree within the population begins to grow
uncontrollably. The NASNet search space inherently prevents this as the
maximum number of nodes is prescribed prior to evolution, thus limiting the
maximum size of a cell. _Redundancy_ occurs in tree-based grammars when parts
of the tree contain redundant features, which can lead to excessively large
trees. Although redundancy is not inherently a problem, it signifies that a
particular expression is needed in multiple places, which tree-based grammars
must evolve independently. The NASNet search space inherently solves this
problem as hidden state nodes are abled to be reused multiple times within the
computational graph, thus allowing for redundant features without increasing
the number of nodes, thereby speeding up the evolution to acquire such
redundant features.

Unlike Gonzalez & Miikkulainen (2020) and Raymond et al. (2022), we do not
fine tune the coefficients of the final loss equation. Although fine tuning
can greatly increase convergence, this was not performed for two primary
reasons. First, to avoid over parameterizing the final loss function on a
particular dataset, ensuring simple and transferable loss functions. Second,
the computational overhead needed to perform a secondary search is greatly
increased when dealing with a continuous domain and an expensive surrogate
function. Although meta-learning can be faster to converge than CMA-ES, as it
uses a direction gradient to step toward, current meta-learning techniques
inherently bias losses towards cross-entropy (Chebotar et al., 2019; Raymond
et al., 2022), because they use entropy as the meta-learner when calculating
the gradients. In the end, we did not find not fine tuning the coefficients
problematic as evidenced by our results.

The derivative of the NASNet search space proposed in this work has three
components: (1) _input connections_ , which are analogous to leaf or terminal
nodes for tree-based grammars: $y$, $\hat{y}$, $1$, and $-1$, where $y$ is the
target output and $\hat{y}$ is the model’s prediction; (2) _hidden state
nodes_ , which are free-floating nodes able to use any other hidden node or
input connection as an argument; and (3) a specialized _output node_ , defined
to be the final operation before outputting the loss value. All nodes use
either a unary or binary function.

Table 1: The proposed set of unary operations. Unary Operations |
---|---
$-x$ | $\ln(|x|+\epsilon)$
$e^{x}$ | $\log_{10}(|x|+\epsilon)$
$\sigma(x)$ | $\frac{d}{dx}\sigma(x)$
$\text{softsign}(x)$ | $\frac{d}{dx}\text{softsign}(x)$
$\text{softplus}(x)$ | $\text{erf}(x)$
$\text{erfc}(x)$ | $\sin(x)$
$\sinh(x)$ | $\text{arcsinh}(x)$
$\tanh(x)$ | $\frac{d}{dx}\tanh(x)$
$\text{arctanh}(x)$ | $1/(x+\epsilon)$
$|x|$ | $x^{2}$
$\text{bessel}_{i0}(x)$ | $\sqrt{x}$
$\text{bessel}_{i1}(x)$ | $\max(x,0)$
$\text{bessel}_{i1e}(x)$ | $\min(x,0)$
$\text{bessel}_{i0e}(x)$ |
Table 2: The proposed set of binary operations. Binary Operations |
---|---
$x_{1}+x_{2}$ | $x_{1}*x_{2}$
$x_{1}-x_{2}$ | $x_{1}/(x_{2}+\epsilon)$
$x_{1}/\sqrt{1+x_{2}^{2}}$ | $\max(x_{1},x_{2})$
$\min(x_{1},x_{2})$ |

Figure 1: Example loss function with three active (blue) hidden state nodes,
one inactive hidden node (grey), and one root node (white). The final loss
equation is given next to the root node in the box.

The 27 proposed unary operations are listed in Table 1, where
$\text{bessel}_{ij}$ is the Bessel function of degree $j$; $e$ denotes
exponential; $\sigma$ is sigmoid activation; softsign is softsign activation;
softplus is softplus activation; erf is the Gauss error function; erfc is the
complementary error function; $\frac{d}{dx}$ refers to the derivative of the
respective function; and $\epsilon=10^{-7}$ for numerical stability. The seven
proposed binary operations are listed in Table 2. All operations were chosen
from previous related work in NLFS and available operations in Tensorflow. The
goal of increasing the number of possible operators is to create an expansive
search space, allowing for loss functions to be expressed in many possible
ways.

###  3.2 Loss Function Initialization

Each loss function contains four hidden state nodes and one output node. Each
node is initialized by sampling its operation uniformly from the set of all 34
operations. Each connection for each hidden state node is sampled uniformly
from all input connections and hidden state nodes. The connections for the
root node are determined such that at least one connection is from a hidden
state node to prevent the output from being a simple binary or unary operation
on the input. Figure 1 shows an example loss function after initialization.

###  3.3 Integrity Check

With the inclusion of many new operations in the proposed search space, and
the proposal of free-floating hidden state nodes, the total search space
contains a lower bound of $10^{10}$ possible combinations. Although the search
space allows for the exploration of many potential good loss functions, it
also allows for many degenerate loss functions. To prevent degenerate loss
functions from wasting valuable evaluation time, an integrity check was
implemented, similar to that by Liu et al. (2021). If a loss function failed
an integrity check, that function was re-initialized using the procedure
described in Section 3.2. The integrity check contains four components.

First, the computational graph containing the hidden state nodes for each loss
function is checked for cycles. If a cycle is found, the loss function is
rejected. In addition, if $y$ or $\hat{y}$ is not included at least once in
the graph, it is also rejected.

Second, if the first integrity test is passed, the phenotype of the loss
function is acquired for a simple binary case. The _phenotype of the loss
function_ refers to the function value of the loss for given prediction and
target arrays, specifically when $y=1$ in this case. The phenotype of cross-
entropy, normalized between 0 and 1, using this evaluation is plotted in
Figure 2. As one can see, the loss value for cross-entropy decreases
drastically as $\hat{y}\to 1$, signifying the global minimum, the place where
the input-target pairs match. A loss function for a binary case should contain
only one global minimum, as there should exist only one solution that
minimizes the loss function for all input. Cross-entropy exemplifies this as
it is a monotonically decreasing function, as one can see by the figure. To
eliminate multi-modal loss landscapes, all loss functions containing more than
one global optimum for a binary case are rejected. As a result, the remaining
loss functions are either monotonic or roughly parabolic. If the loss function
is roughly parabolic, it is rejected if its global optimum is within 1% of the
$\hat{y}$ value 0.5 to prevent global minimum values near $0.5$, the threshold
for accuracy.

Figure 2: Overview of the binary phenotypes of cross-entropy, NeuroLoss 1,
NeuroLoss 2, and NeuroLoss 3.

Figure 3: Zoomed in view (around $\hat{y}\to 1$) of the binary phenotypes of
cross-entropy, NeuroLoss 1, NeuroLoss 2, and NeuroLoss 3.

Third, the integrity check flips the sign of the loss function if it is either
monotonically increasing or an upside-down parabola. This was performed to
prevent _inverse learning_ —learning an output of 0 for a target output of 1,
and an output of 1 for a target output of 0. This simple adjustment inherently
speeds up evolution by not requiring loss functions to evolve the proper
signage for the root node.

Fourth, the phenotypes of all existing loss functions within the population,
and the current loss function, are normalized between 0 and 1. All normalized
phenotypes from the population are looped over and the the Euclidean norm of
the difference between each normalized phenotype and the current normalized
loss function are calculated. If a single normalized difference was below a
threshold of $0.01$ the loss function was rejected for being too similar to
another existing loss function. This was performed to allow for ample
exploration of non-explored regions of the search space without simply scaling
already existing loss functions. However, repeat loss functions were allowed
during evolution as long as the copy died off from within the population, a
benefit to the aging mechanism (see Section 3.4).

After passing all four components of the integrity check, each loss function
is evaluated using the proposed surrogate function. The final algorithm for
the integrity check is detailed in Appendix A.

###  3.4 Genetic Algorithm

A population-based genetic algorithm (GA) is used to explore the proposed
search space. Unlike grammar-based search spaces, our proposed NASNet
derivative does not allow for an easy implementation of crossover
(recombination), as the hidden state nodes can occur in any order and can
connect to any other node. As a result, a mutation-only GA is used. Given the
success of regularized evolution (Real et al., 2019) on the NASNet search
space, we use regularized evolution for our search space. _Regularized
evolution_ is a mutation-only aging GA that selects a parent through
tournament selection from the population. This parent is mutated to obtain a
child loss that is evaluated and added back to the population by replacing the
oldest. Because training neural networks is highly stochastic, and in order to
prevent pre-convergence to local minima and allow for ample exploration, aging
is incorporated for better exploration of the search space as each individual
has a maximum life span within the algorithm. This is inherently performed in
the GA as the new child replaces the oldest.

Mutation was performed by uniformly sampling either the root node or one of
the hidden state nodes, then changing either its operation or connection.
After mutation, an integrity check (Section 3.3) is performed to ensure a
successful mutation. If the integrity check fails, mutation is performed
again. If 2500 mutations fail in a row, the mutation does not take place, and
another iteration of the algorithm is performed using the procedure just
discussed. Further information on mutation can be found in Appendix B.

###  3.5 Surrogate Function

It is simple to find specific components of neural networks that perform
better than standard versions on a particular dataset or on smaller networks.
However, it can be very difficult to find components of neural networks that
perform better than standard choices across multiple datasets, on models
containing millions of parameters, or when massive amounts of regularization
are present during training. To address this issue, we choose to compare the
final evolved loss functions on EfficientNetV2Small (EffNetV2Small) (Tan & Le,
2021). EffNetV2Small is a state-of-the-art CNN model containing 20.3 million
parameters, that was able to achieve a Top-1% validation accuracy of 83.9% on
ImageNet ILSVRC2012 (Deng et al., 2009). Along with the EffNetV2Small model,
we use the same training strategy used by Tan & Le (2021): progressive RandAug
regularization (Cubuk et al., 2020). _RandAug_ is an image augmentation
strategy that randomly augments an image by applying $n$ augmentations from a
set of possible augmentations, with each augmentation strength capped at a
given maximum strength. In _progressive RandAug regularization_ , the training
strategy is split into four phases, where each hyper-parameter of augmentation
is increased throughout training along with image resolution size. We believe
that training loss functions using this state-of-the-art model and state-of-
the-art regularization technique will accurately compare our final proposed
loss functions against cross-entropy for large-scale CNNs.

Because it is prohibitively expensive to conduct NLFS directly using the full
EffNetV2Small training schedule, we use a reduced _surrogate function_ during
the evolutionary process, as is standard for learning transferable components
(Bello et al., 2017; Bingham et al., 2020; Gonzalez & Miikkulainen, 2020; Liu
et al., 2020). An issue that can plague learning transferable structures for
neural networks is the correlation between the full model and the surrogate
model. If there is no correlation between the smaller surrogate model and the
full version, then there is no logical reason to use the surrogate. Although
it is simple in concept, it can be very difficult to find an accurate
surrogate function for learning transferable structures for a particular
component. This problem will be further exemplified shortly with our initial
findings. Because we care about the relational ranking of loss functions
between the surrogate and the full model, Kendall’s Tau rank correlation
(Kendall, 1938) was used instead of Pearson’s Correlation.

As a cheap proxy dataset, all models were trained on the CIFAR-10 dataset
(Krizhevsky & Hinton, 2009). For promoting generalization, 5,000 of the
training images from CIFAR-10 were used as validation. The best validation
accuracy was used as the fitness value during evolution, and for calculating
rank correlation.

####  3.5.1 Chosen Surrogate Function

In the search for a suitable surrogate function, we looked at three possible
models: (1) a three-layer ConvNet, containing 14,000 parameters; (2) a ResNet9
version two (ResNet9V2) (He et al., 2016) with nine layers, containing 30,000
parameters; and (3) EffNetV2Small.

To generate potential loss functions, we performed two preliminary
evolutionary runs using the ConvNet and ResNet9 architectures as surrogate
functions, both trained on the native dataset image resolution size of 32 for
CIFAR-10 without RandAug regularization for 8,000 steps. The best 30 loss
functions from each run were then trained using the full EffNetV2Small
progressive RandAug training schedule for 64,000 steps. Their final Kendall’s
Tau rank correlation is shown in the first row of Table 3. Both performed
poorly in transferring their final best loss functions to a large-scale model,
both achieving a rank correlation of 0.1849, highlighting the issue of finding
transferable surrogate functions.

Table 3: Kendall’s Tau Rank Correlation between ConvNet and ResNet9V2 (ResNet), and EffNetV2Small (EffNet). The first row gives the rank correlation for both ConvNet and ResNet9V2 trained for 8,000 steps without RandAug regularization. The remaining rows give the rank correlation for the architectures trained using RandAug regularization at the given image size for 8,000 steps.  Image Size |  ConvNet | ResNet |  EffNet
---|---|---|---
- |  0.1849 | 0.1849 |  -
32 |  0.2352 | 0.1199 |  0.0344
48 |  0.1150 | 0.2649 |  0.1448
64 |  0.2768 | 0.1933 |  0.3149
80 |  0.1887 | 0.3498 |  0.2643
96 |  0.2926 | 0.1218 |  0.5080
112 |  0.2873 | 0.2715 |  0.3420

To find a surrogate function with better rank correlation, we took the final
30 loss functions from the ResNet9 evolutionary run (using their associated
full scale EffNetV2Small progressive RandAug training schedule validation
accuracy), and trained each one on the ConvNet, ResNet9, and EffNetV2Small
architectures again, except with progressive RandAug regularization at
different image resolution sizes for 8,000 steps. Table 3 shows the
correlation between each image resolution size and model trained using
progressive RandAug regularization and the full scale EffNetV2Small
progressive RandAug training schedule. As one can see, the correlations seem
almost random, as no apparent relationship can be derived from either the
image resolution or architecture. We believe this exemplifies the difficulty
of finding a successfully transferable surrogate function. However, a
configuration with an acceptable correlation appeared: EffNetV2Small trained
using 96$\times$96 image resolution, with a rank correlation of 0.5080.
Although EffNetV2Small is more expensive to train than ResNetV2 and ConvNet,
we decided to ensure a transferable surrogate function, and selected it as our
surrogate function.

###  3.6 Initial Evolutionary Run

In our initial evolutionary run using the ConvNet architecture as the
surrogate function, the best performing loss function was
$\frac{1}{n}\sum_{i}(\text{min}(\text{bessel}_{i0e}(\hat{y}_{i}),y_{i})+\text{bessel}_{i0e}(\hat{y}_{i}))$,
which we call the _Bessel Loss_. As a comparison, Table 4 shows the mean
validation accuracy on the ConvNet surrogate function after 30 independent
runs for _Bessel Loss_ , cross-entropy, and the final three _NeuroLoss_
functions. As one can see, _Bessel Loss_ outperformed all _NeuroLoss_
functions, and cross-entropy with a higher mean validation accuracy. In
addition, a two-sided statistical Welch’s $t$-test of the mean difference
between _Bessel Loss_ and the other loss functions being larger than zero
(indicating _Bessel Loss_ has a larger mean) was performed. The _Bessel Loss_
was shown to have a statistically larger mean _NeuroLoss2_ and cross-entropy.
However, when training _Bessel Loss_ on EffNetV2Small using the full training
schedule of progressive RandAug regularization on CIFAR-10, the loss struggles
to reach a mean test accuracy of 94.51%, compared to _NeuroLoss1_ ’s 95.95%,
_NeuroLoss2_ ’s 95.91%, and _NeuroLoss3_ ’s 95.82% test accuracy (From Table
9). This observation reveals that possibly using smaller architectures, with
little to no regularization, as a surrogate function for NLFS may not be
beneficial as their performance may not correlate well to large-scale CNNs
with regularization.

Table 4: Results for Bessel Loss, NeuroLoss1, NeuroLoss4, and cross-entropy
using the ConvNet surrogate function. (NL1 for NeuroLoss 1, etc.) The mean
test accuracy, and standard deviation, across 30 independent runs are reported
for each loss function along with the $p$-value for a one sided statistical
$t$-test for mean accuracy being larger than cross-entropy. The best mean test
accuracy is highlighted in bold. ConvNet Surrogate Function
---
Loss | Validation Accuracy |  $p$-value
BesselLoss |  58.839$\pm$0.572 |  -
NL1 |  58.711$\pm$0.455 |  $p=0.171$
NL2 |  58.579$\pm$0.612 |  $p=0.047$
NL3 |  58.660$\pm$0.557 |  $p=0.112$
CE |  58.474$\pm$0.569 |  $p=0.008$

###  3.7 Early Stopping

To avoid wasting time in training degenerate loss functions, an early stopping
method was implemented. Before training on EffNetV2Small 96$\times$96
resolution, a ConvNet was trained on the images, without regularization, for
800 steps. If the training accuracy was below a hand-tuned threshold of 37%,
the loss function was rejected and mutated again. If the training accuracy was
above this threshold, the loss was trained on the surrogate function, where a
secondary early stopping method was implemented. This secondary early stopping
method was different from the first. If the training accuracy had not reached
a hand-tuned threshold of 25% after 3,000 steps, training was stopped;
however, the best validation accuracy was returned as the fitness score to the
genetic algorithm.

###  3.8 Initial Population

In our initial shortened evolution we noted that the majority of loss
functions for the initial population of the algorithm were degenerate in that
the first early stopping mechanism implemented within the surrogate function
stopped training due to poor learning. Allowing for degenerate models within
the initial population caused three primary issues. First, it created weak
competition within the population, as working loss functions would have less
competition for mutation. Second, it allowed for early pre-convergence, as
only the smaller set of working functions were chosen for mutation. Third, for
tournaments containing only degenerate losses, the child loss of the
degenerate parent would inherently have higher probability of being degenerate
as well. We believe this allowance will degrade evolution quality in the long
run. As a possible solution, we decided to decouple the initial population
size from the population size. In doing so, we increased the initial
population size to allow for more diversity and to allow for more working loss
functions. The best from the initial population were then taken to form the
first generation, ensuring working initial loss functions.

###  3.9 Loss Function elimination Protocol

Due to the disparity of correlation between the surrogate function and full
EffNetV2Small training, we decided to spend the majority of our computational
resources on training for transferability rather than searching for loss
functions, as our integrity check and early stopping mechanisms eliminated
most degenerate solutions. To do so, we implemented a loss function
elimination protocol that progressively eliminated candidate loss functions,
after evolution, until a final set of winners remained by progressively
training them on the different phases of RandAug regularization for
EffNetV2Small. The best 150 loss functions discovered over the entire course
of evolution were trained on EffNetV2Small for phase 1 (16,000 steps) of
RandAug progressive regularization. The best 50 from that set were then
trained again, from scratch, for both phase 1 and 2 (32,000 steps) of RandAug
regularization. The best 25 from that set were then trained again until phase
3 (48,000 steps) of RandAug regularization. Finally, the best 10 were taken
and trained for the full RandAug progressive regularization training strategy
(64,000 steps). Note, all loss functions in our proposed loss function
elimination protocol were compared using the validation accuracy for CIFAR-10.

###  3.10 Implementation

For the GA, an initial population size of 1000 was used to allow for ample
initial diversity and working loss functions. The best 100 were then taken to
form the first generation. A tournament size of 20 was used for selection. The
GA was allowed to run for a time frame of 100 hours on an NVIDIA A100 GPU.
Each individual within the population was trained using the proposed surrogate
function, EffNetV2Small 96$\times$96 resolution for 8,000 steps of Phase 1 of
RandAug progressive regularization. Due to computational constraints, the
initial population was not trained using this surrogate function, but was
instead trained using the ConvNet architecture without RandAug regularization.
We did not find this problematic as the purpose of the enlarged initial
population was to find workable loss functions in a short amount of time.

##  4 Results and Discussion

In this section, the results from the evolutionary run are detailed, along
with the final best losses and their evaluation across multiple datasets and
architectures.

###  4.1 Evolution

After 100 hours of evolution on the CIFAR-10 dataset, a total of 2,000
iterations had completed, 29% of which did not fail the surrogate function.
The best, mean, and median validation scores over time are shown in Figure 4.
In addition, the max, mean, and min cross-entropy scores across 15 independent
training sessions for the surrogate function are plotted for comparison. The
algorithm was able to evolve 23 loss functions that were able to surpass the
best cross-entropy accuracy on the surrogate function, while 119 loss
functions were able to surpass the mean cross-entropy accuracy.

Figure 4: Results for the evolutionary algorithm evaluated using the proposed
surrogate function.

###  4.2 Best Losses

The final three remaining loss functions, which we call the _NeuroLoss_
functions, after running our loss function elimination protocol are listed in
Table 5. As one can see, the final three functions are very similar as all of
them follow the form $\log(y,\hat{y})+f(y,\hat{y})$, where $\log(y,\hat{y})$
was either $\ln$ or $\log_{10}$ and $f(y,\hat{y})$ was some function. In
addition, one can also see the repeated structure of $\hat{y}/(y+\epsilon)$
that is present twice in almost all functions, revealing the power of re-
usable hidden state nodes in our proposed search space. It seems that the
final loss functions heavily relied on scaling the prediction $\hat{y}$ by
some value, mainly $\sqrt{1+(\hat{y}/(y+\epsilon))^{2}}$ inside the
logarithmic operation. Honorable mentions after running the proposed loss
function elimination protocol are found in Table 6.

Table 5: The final three evolved loss functions (NL1 for NeuroLoss 1, etc.), along with cross-entropy (CE) for comparison. | Loss Equation
---|---
NL1 | $-\frac{1}{n}\sum_{i}\Bigl{(}\ln\Bigl{(}\Bigl{|}\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}\Bigr{|}+\epsilon\Bigr{)}+\hat{y}_{i}\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}\Bigr{)}$
NL2 | $-\frac{1}{n}\sum_{i}\Bigl{(}\log_{10}\Bigl{(}\Bigl{|}\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}\Bigr{|}+\epsilon\Bigr{)}+\sin\Bigl{(}\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}\Bigr{)}\Bigr{)}$
NL3 | $-\frac{1}{n}\sum_{i}\Bigl{(}\ln\Bigl{(}\Bigl{|}\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}\Bigr{|}+\epsilon\Bigr{)}+\min(y_{i},\hat{y}_{i}/(y_{i}+\epsilon))\Bigr{)}$
CE | $-\frac{1}{n}\sum_{i}y_{i}\ln\Bigl{(}|\hat{y}_{i}|+\epsilon\Bigr{)}$
Table 6: Honorable Mentions
Loss Equation
---
$-\frac{1}{n}\sum_{i}\Bigl{(}\ln\Bigl{(}\Bigl{|}\frac{\hat{y}_{i}}{\sqrt{1+(y_{i})^{2}}}\Bigr{|}+\epsilon\Bigr{)}\Bigr{)}\big{/}\Bigl{(}\sqrt{1+(\max(y_{i},(\hat{y}_{i}/(y+_{i}\epsilon)))^{2})}\Bigr{)}$
$-\frac{1}{n}\sum_{i}\max\Bigl{(}\log(|\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}|+\epsilon),\text{arcsinh}(\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}})\Bigr{)}$
$-\frac{1}{n}\sum_{i}\Bigl{(}\log\Bigl{(}|\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}|+\epsilon\Bigr{)}+(\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}})^{2}\Bigr{)}$
$-\frac{1}{n}\sum_{i}\Bigl{(}\ln\Bigl{(}|\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}|+\epsilon\Bigr{)}+\sinh(\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}})\Bigr{)}$
$-\frac{1}{n}\sum_{i}\Bigl{(}\ln\Bigl{(}|\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}|+\epsilon\Bigr{)}+\text{erf}\Bigl{(}\hat{y}_{i}/(y_{i}+\epsilon))\Bigr{)}\Bigr{)}$
$-\frac{1}{n}\sum_{i}\Bigl{(}\log\Bigl{(}|\frac{\hat{y}_{i}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}|+\epsilon\Bigr{)}+\tanh\Bigl{(}\hat{y}_{i}/(y_{i}+\epsilon)\Bigr{)}\Bigr{)}$
$-\frac{1}{n}\sum_{i}\frac{\ln\Bigl{(}|y_{i}*\hat{y}_{i}|+\epsilon\Bigr{)}}{\sqrt{1+(\hat{y}_{i}/(y_{i}+\epsilon))^{2}}}$

###  4.3 Evaluation

To evaluate the generalizability of the final three loss functions, they were
transferred across multiple data sets, architectures, image augmentation
techniques, and learning rate schedules and optimizers. Six primary datasets
were used in this work, CIFAR-10, CIFAR-100, Stanford-CARS196, Oxford-
Flowers102, Caltech101, and ImageNet. The meta-information for each of these
datasets are available in Table 7.

Table 7: The number of training and evaluation images, number of classes, and mean image resolution for the datasets used in this work. Dataset |  Train |  Test |  Classes |  Size
---|---|---|---|---
CIFAR10 |  50,000 |  10,000 |  10 |  32$\times$32
CIFAR100 |  50,000 |  10,000 |  100 |  32$\times$32
CARS196 |  8,144 |  8,041 |  196 |  482$\times$700
Flowers102 |  2,040 |  6,149 |  102 |  546$\times$624
Caltech101 |  3,060 |  6,084 |  102 |  386$\times$468
ImageNet |  1.28M |  50,000 |  1000 |  469$\times$387

Other than EffNetV2Small, the final loss functions were trained on three other
state-of-the art model architectures to assess architecture transferability:
WideResNet (depth=28, width=10, 36 million parameters) (Zagoruyko & Komodakis,
2016), SEResNeXt (4$\times$64d, 20 million parameters) (Hu et al., 2018), and
DavidNet (custom ResNet9, 6 million parameters, Appendix D). Three different
image augmentation techniques were used: progressive RandAug regularization
for EffNetV2Small; simple cutout with random cropping and horizontal flipping
for WideResNet, SEResNeXt, and DavidNet; and CutMix (Yun et al., 2019) for
DavidNet. CutMix was used as it mixes label distribution, testing for whether
or not the final loss functions could handle target values not equal to 0 or
1. To assess dataset transferability, the final loss functions were trained
using the previous architectures and image augmentation techniques on both
CIFAR-10 and CIFAR-100. Lastly, the final loss functions were transferred to
three large image resolution benchmark dataset: Cars196, Flowers102, and
Caltech101. Using EffNetV2Small, the final loss functions were trained both
from scratch and fine-tuning (using the official published ImageNet weights),
on the large image resolution datasets. Details on the training strategies
used for each architecture can be found in Appendix D.

###  4.4 Results

Table 8: Results for DavidNet using resize, cutout, and horizontal flipping, and CutMix for both CIFAR-10 and CIFAR-100. The mean test accuracy and standard deviation across 30 independent runs is reported for each loss function along with the $p$-value for a Welch’s two-sided statistical $t$-test for mean difference between the _NeuroLoss_ function and cross-entropy being larger than zero (indicating the function has a larger mean). The best mean test accuracy for each dataset is highlighted in bold.  | CIFAR-10 (cutout) | CIFAR-10 (cutmix)
---|---|---
Loss | Test Accuracy |  $p$-value | Test Accuracy |  $p$-value
NL1 |  94.042$\pm$0.126 |  $p<0.001$ |  91.621$\pm$0.184 |  $p<0.001$
NL2 |  93.787$\pm$0.138 |  $p=0.298$ |  91.409$\pm$0.178 |  $p=0.581$
NL3 |  94.006$\pm$0.134 |  $p<0.001$ |  91.601$\pm$0.169 |  $p<0.001$
CE |  93.761$\pm$0.134 |  - |  91.419$\pm$0.197 |  -
| CIFAR-100 (cutout) | CIFAR-100 (cutmix)
Loss | Test Accuracy |  $p$-value | Test Accuracy |  $p$-value
NL1 |  74.959$\pm$0.212 |  $p=0.003$ |  70.877$\pm$0.270 |  $p<0.001$
NL2 |  74.814$\pm$0.254 |  $p=0.421$ |  70.736$\pm$0.244 |  $p<0.001$
NL3 |  74.834$\pm$0.263 |  $p=0.302$ |  71.176$\pm$0.288 |  $p<0.001$
CE |  74.802$\pm$0.208 |  - |  69.534$\pm$0.228 |  -
Table 9: Results for EffNetV2Small using progressive RandAug regularization,
WideResNet (depth=10, width=28) and SEResNeXt (4x64d) using resize, cutout,
and horizontal flipping augmentation for both CIFAR-10 and CIFAR-100. The mean
test accuracy and standard deviation across 5 independent runs is reported for
each loss function. The best mean test accuracy for each combination of
dataset and architecture is highlighted in bold. CIFAR-10
---
Loss |  EffNetV2S |  WideResNet |  SEResNeXt
Bessel |  94.510$\pm$0.115 |  |
NL1 |  95.952$\pm$0.106 |  95.860$\pm$0.142 |  95.830$\pm$0.123
NL2 |  95.906$\pm$0.050 |  95.836$\pm$0.052 |  95.604$\pm$0.032
NL3 |  95.816$\pm$0.060 |  95.764$\pm$0.050 |  95.788$\pm$0.078
CE |  95.826$\pm$0.118 |  95.670$\pm$0.156 |  95.222$\pm$0.063
CIFAR-100
Loss |  EffNetV2S |  WideResNet |  SEResNeXt
NL1 |  78.172$\pm$0.249 |  79.848$\pm$0.061 |  79.964$\pm$0.167
NL2 |  77.926$\pm$0.189 |  79.628$\pm$0.242 |  80.152$\pm$0.263
NL3 |  78.230$\pm$0.154 |  80.038$\pm$0.271 |  79.804$\pm$0.263
CE |  77.130$\pm$0.228 |  79.640$\pm$0.213 |  80.526$\pm$0.228
Table 10: Results for EffNetV2Small using progressive RandAug regularization
when training from scratch for Stanford-CARS196, Oxford-Flowers102, and
Caltech101. The mean test accuracy and standard deviation across 5 independent
runs is reported for each loss function. The best mean test accuracy for each
dataset is highlighted in bold. CE0.10 refers to cross-entropy with a label
smoothing value of $0.10$. EffNetV2Small (From Scratch)
---
Loss |  CARS196 |  Flowers102 |  Caltech101
NL1 |  85.457$\pm$0.244 |  88.584$\pm$0.508 |  85.148$\pm$0.210
NL2 |  85.698$\pm$0.361 |  87.380$\pm$0.461 |  84.234$\pm$0.341
NL3 |  86.439$\pm$0.418 |  88.408$\pm$0.440 |  84.822$\pm$0.285
CE |  82.729$\pm$0.285 |  88.154$\pm$0.299 |  84.116$\pm$0.119
CE0.10 |  84.674$\pm$0.254 |  87.819$\pm$0.228 |  84.499$\pm$0.332
Table 11: Results for EffNetV2Small using progressive RandAug regularization
when fine-tuning for Stanford-CARS196, Oxford-Flowers102, and Caltech101. The
mean test accuracy and standard deviation across 5 independent runs is
reported for each loss function. The best mean test accuracy for each dataset
is highlighted in bold. CE0.10 refers to cross-entropy with a label smoothing
value of $0.10$. EffNetV2Small (Fine-Tune)
---
Loss |  CARS196 |  Flowers102 |  Caltech101
NL1 |  91.635$\pm$0.109 |  97.313$\pm$0.095 |  90.556$\pm$0.279
NL2 |  91.660$\pm$0.053 |  97.421$\pm$0.139 |  90.565$\pm$0.428
NL3 |  91.635$\pm$0.091 |  97.287$\pm$0.138 |  90.302$\pm$0.295
CE |  91.725$\pm$0.233 |  97.190$\pm$0.188 |  90.217$\pm$0.250
CE0.10 |  91.496$\pm$0.122 |  97.320$\pm$0.277 |  90.434$\pm$0.141

Results for DavidNet on CIFAR are given in Table 8. Results for EffNetV2Small,
WideResNet, and SEResNeXt on CIFAR are given in Table 9. Results for
EffNetV2Small on Cars196, Flowers102, and Caltech101 from scratch are given in
Table 10, while the results for fine-tuning are given in Table 11. As seen in
Table 8, multiple loss functions achieved significant $p$-values for having a
larger mean test accuracy than cross-entropy across both datasets and image
augmentation techniques. However, only one loss, NeuroLoss 1, obtained a
significant value for all datasets and augmentation techniques, revealing that
it can outperform and increase generalization better than cross-entropy in a
computationally stringent environment for DavidNet.

From Tables 8 and 9, only once was the mean test accuracy of cross-entropy
better than the best loss function (SEResNeXt for CIFAR-100). Two loss
functions dominated the results, NeuroLoss 1 and NeuroLoss 3. In terms of mean
test value, NeuroLoss 1 and NeuroLoss 3 surpassed cross-entropy across all
datasets and architectures (except NeuroLoss 3 for EffNetV2Small for CIFAR-10,
and both losses for SEResNeXt for CIFAR-100). However, NeuroLoss 1 was able to
surpass NeuroLoss 3 for all architectures trained upon CIFAR-10, but NeuroLoss
3 was able to surpass Loss 1 and cross-entropy for all architectures trained
on CIFAR-100 (except DavidNet using cutout and SEResNeXt).

From Tables Table 10, both NeuroLoss 1 or NeuroLoss 3 outperformed cross-
entropy, even with a label smoothing constant of $\alpha=0.10$, across all
datasets. However, when changing from training froms scratch to fine-tuning
using cross-entropy biased initial weights, Table 11 reports that NeuroLoss 2
outperforms all other NeuroLoss functions and cross-entropy, both with and
without label smoothing, on two of the three datasets. With NeuroLoss 2
trailing the other two NeuroLoss functions when training from scratch on
Flowers102 and Caltech101, but outperforming both when fine-tuning could
indicate that some loss functions might be better inclined to fine-tuning than
others.

###  4.5 What Makes _NeuroLoss_ Successful?

Figures 2 and 3 depict the phenotypes of all NeuroLoss functions and cross-
entropy in a binary example. From both of these figures, there are two
noticeable differences. First, all NeuroLoss functions have a less steep slope
compared to cross-entropy, resulting in higher normalized loss values. From
Figure 2, NeuroLoss 2 has the highest normalized loss values compared to all
other functions when $\hat{y}<0.4$; however, Figure 3 indicates that NeuroLoss
2 dips below the other NeuroLoss functions and even cross-entropy as
$\hat{y}\to 1$. NeuroLoss 1 and NeuroLoss 2 seem to hug each other for both
small and large $\hat{y}$ values. Perhaps this more gentle slope could
encourage more smoothness in the loss landscape than cross-entropy. However,
further investigation and study must be conducted to verify that hypothesis.
Second, the global minimum for all NeuroLoss functions do not occur at
$\hat{y}=1$, as with cross-entropy, but instead at $\hat{y}=0.99998$. This
small level of implicit regularization can be replicated in cross-entropy with
an extremely small label smoothing constant of $\alpha=0.00003$. Although the
global minimum can be shifted in cross-entropy via label smoothing to match
that of the NeuroLoss functions, the slope cannot be changed to equal them.
The exact reason for the _NeuroLoss_ functions success is unknown, but two
possible reasons arise. The implicit learning of an extremely small label
smoothing constant and a higher normalized loss values than cross-entropy over
the majority of $\hat{y}$ values.

##  5 Conclusions

In this work, we expanded on previous research in NLFS by proposing a new
search space containing more unary and binary operations, along with a new
computational graph representation that eliminated some of the downsides
associated with GP. In addition, we stressed the importance of ensuring
transferable surrogate functions to ensure good rank correlation. After
running our evolutionary algorithm and loss function elimination protocol,
three loss functions highly competitive with cross-entropy were discovered
through pure evolution. The best loss functions, termed _NeuroLoss_ 1,
_NeuroLoss_ 2, and _NeuroLoss_ 3, were able to surpass cross-entropy across
multiple architectures, datasets, and image augmentation techniques. The
proposed search space, integrity check, and elimination protocol can be used
for future NFLS endeavors, such as loss functions for image anomaly detection,
imbalanced image datasets, or traditional based machine learning algorithms,
such as logistic regression and decision trees.

## References

  * Abadi et al. (2015) Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.  URL https://www.tensorflow.org/.  Software available from tensorflow.org.
  * Anonymous (2023) Anonymous.  Neural loss function evolution for large scale image classifier convolutional neural networks.  https://github.com.invalid/Anon/NeuralLossFunctionEvolution, 2023.
  * Bello et al. (2017) Bello, I., Zoph, B., Vasudevan, V., and Le, Q. V.  Neural optimizer search with reinforcement learning.  In Precup, D. and Teh, Y. W. (eds.), _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 459–468. PMLR, 06–11 Aug 2017.  URL https://proceedings.mlr.press/v70/bello17a.html.
  * Bingham et al. (2020) Bingham, G., Macke, W., and Miikkulainen, R.  Evolutionary optimization of deep learning activation functions.  In _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_ , GECCO ’20, pp. 289–296, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450371285.  doi: 10.1145/3377930.3389841.  URL https://doi.org/10.1145/3377930.3389841.
  * Chebotar et al. (2019) Chebotar, Y., Molchanov, A., Bechtle, S., Righetti, L., Meier, F., and Sukhatme, G. S.  Meta learning via learned loss.  In _International Conference on Pattern Recognition (ICPR)_ , pp. 4161–4168, 2019.
  * Coleman et al. (2019) Coleman, C., Kang, D., Narayanan, D., Nardi, L., Zhao, T., Zhang, J., Bailis, P., Olukotun, K., Ré, C., and Zaharia, M.  Analysis of DAWNBench, a time-to-accuracy machine learning performance benchmark.  _SIGOPS Oper. Syst. Rev._ , 53(1):14–25, July 2019.  ISSN 0163-5980.  doi: 10.1145/3352020.3352024.  URL https://doi.org/10.1145/3352020.3352024.
  * Cubuk et al. (2020) Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 702–703, 2020.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Elsken et al. (2019) Elsken, T., Metzen, J. H., and Hutter, F.  Neural architecture search: A survey.  _The Journal of Machine Learning Research_ , 20(1):1997–2017, 2019.
  * Gao et al. (2021) Gao, B., Gouk, H., and Hospedales, T. M.  Searching for robustness: Loss learning for noisy classification tasks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 6670–6679, October 2021.
  * Gao et al. (2022) Gao, B., Gouk, H., Yang, Y., and Hospedales, T.  Loss function learning for domain generalization by implicit gradient.  In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 7002–7016. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/gao22b.html.
  * Gonzalez & Miikkulainen (2020) Gonzalez, S. and Miikkulainen, R.  Improved training speed, accuracy, and data utilization through loss function optimization.  In _2020 IEEE Congress on Evolutionary Computation (CEC)_ , pp. 1–8, 2020.  doi: 10.1109/CEC48606.2020.9185777.
  * Gonzalez & Miikkulainen (2021) Gonzalez, S. and Miikkulainen, R.  Optimizing loss functions through multi-variate Taylor polynomial parameterization.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , GECCO ’21, pp. 305–313, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450383509.  doi: 10.1145/3449639.3459277.  URL https://doi.org/10.1145/3449639.3459277.
  * Gu et al. (2022) Gu, H., Li, J., Fu, G., Wong, C., Chen, X., and Zhu, J.  AutoLoss-GMS: Searching generalized margin-based softmax loss function for person re-identification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 4744–4753, June 2022.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Identity mappings in deep residual networks.  In _European Conference on Computer Vision_ , pp. 630–645. Springer, 2016.
  * He et al. (2021) He, X., Zhao, K., and Chu, X.  AutoML: A survey of the state-of-the-art.  _Knowledge-Based Systems_ , 212:106622, 2021.
  * Hu et al. (2018) Hu, J., Shen, L., and Sun, G.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 7132–7141, 2018.
  * Kendall (1938) Kendall, M. G.  A new measure of rank correlation.  _Biometrika_ , 30(1/2):81–93, 1938.  ISSN 00063444.  URL http://www.jstor.org/stable/2332226.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Krizhevsky & Hinton (2009) Krizhevsky, A. and Hinton, G.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, Toronto, Ontario, 2009.
  * Leng et al. (2022) Leng, Z., Tan, M., Liu, C., Cubuk, E. D., Shi, X., Cheng, S., and Anguelov, D.  Polyloss: A polynomial expansion perspective of classification loss functions.  _arXiv preprint_ , 2022.  doi: 10.48550/ARXIV.2204.12511.  URL https://arxiv.org/abs/2204.12511.
  * Li et al. (2022) Li, H., Fu, T., Dai, J., Li, H., Huang, G., and Zhu, X.  Autoloss-zero: Searching loss functions from scratch for generic tasks.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 999–1008, 2022.  doi: 10.1109/CVPR52688.2022.00108.
  * Li (2017) Li, W.  cifar-10-cnn: Play deep learning with CIFAR datasets.  https://github.com/BIGBALLON/cifar-10-cnn, 2017.
  * Li (2022) Li, W.  Keras EfficientNetV2.  https://github.com/leondgarse/keras_efficientnet_v2, 2022.
  * Liu et al. (2020) Liu, H., Brock, A., Simonyan, K., and Le, Q.  Evolving normalization-activation layers.  _Advances in Neural Information Processing Systems_ , 33:13539–13550, 2020.
  * Liu et al. (2021) Liu, P., Zhang, G., Wang, B., Xu, H., Liang, X., Jiang, Y., and Li, Z.  Loss function discovery for object detection via convergence-simulation driven search.  _arXiv preprint arXiv:2102.04700_ , 2021.
  * Luke & Panait (2006) Luke, S. and Panait, L.  A comparison of bloat control methods for genetic programming.  _Evolutionary Computation_ , 14(3):309–344, 2006.
  * Neshatian & Zhang (2009) Neshatian, K. and Zhang, M.  Unsupervised elimination of redundant features using genetic programming.  In Nicholson, A. and Li, X. (eds.), _AI 2009: Advances in Artificial Intelligence_ , pp. 432–442, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg.  ISBN 978-3-642-10439-8.
  * Raymond et al. (2022) Raymond, C., Chen, Q., Xue, B., and Zhang, M.  Learning symbolic model-agnostic loss functions via meta-learning.  _arXiv preprint arXiv:2209.08907_ , 2022.
  * Real et al. (2019) Real, E., Aggarwal, A., Huang, Y., and Le, Q. V.  Regularized evolution for image classifier architecture search.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 4780–4789, 2019.
  * Stanley & Miikkulainen (2002) Stanley, K. O. and Miikkulainen, R.  Evolving neural networks through augmenting topologies.  _Evolutionary Computation_ , 10(2):99–127, 2002.
  * Tan & Le (2021) Tan, M. and Le, Q.  Efficientnetv2: Smaller models and faster training.  In _International Conference on Machine Learning_ , pp. 10096–10106. PMLR, 2021.
  * Yun et al. (2019) Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 6023–6032, 2019.
  * Zagoruyko & Komodakis (2016) Zagoruyko, S. and Komodakis, N.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ , 2016.
  * Zoph & Le (2016) Zoph, B. and Le, Q. V.  Neural architecture search with reinforcement learning.  _arXiv preprint_ , abs/1611.01578, 2016.  doi: 10.48550/ARXIV.1611.01578.  URL https://arxiv.org/abs/1611.01578.
  * Zoph et al. (2018) Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V.  Learning transferable architectures for scalable image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.

##  Appendix A Algorithms

Pseudocode for the integrity check and the genetic algorithm are given in
Algorithm 1 and Algorithm 2, respectively.

Algorithm 1 Integrity Check

1: Input: Candidate Loss $loss$, Population $population$

2: if ContainsCycle($loss$) == True then

3: Return False

4: end if

5: if ContainsYandYHat($loss$) == False then

6: Return False

7: end if

8: if ContainsOneGlobalMinOrMax($loss$) == False then

9: Return False

10: end if

11: CheckSign($loss$)

12: for $individual$ in $population$ do

13: if Norm($individual-loss$)$<0.01$ then

14: Return False

15: end if

16: end for

17: Return True

Algorithm 2 Genetic Algorithm

1: $initialPopulation$ = LossInitialization(1000)

2: $validationAcc$ = ConvNet($initialPopulation$)

3: $population$ = Best100($initialPopulation$, $validationAcc$)

4: repeat

5: $winner$ = Tournament($population$, $validationAcc$, 20)

6: $maxRedo$ = 7

7: while $maxRedo$ $>$ 0 do

8: $maxMutation$ = 2500

9: while $maxMutation$ $>$ 0 do

10: $child$ = Mutation($winner$)

11: if $child$.CheckIntegrity() then

12: break inner while loop

13: else

14: $maxMutation$ –= 1

15: end if

16: end while

17: $valAcc$ = TrainEffNetV2($child$)

18: if $valAcc$ == None then

19: $maxRedo$ –= 1

20: else

21: break while loop

22: end if

23: end while

24: if $maxRedo$ == 0 then

25: Pass

26: else

27: $population$ = ReplaceOldest($population$, $child$)

28: $validationAcc$ = Update($validationAcc$, $valAcc$)

29: end if

30: until Time Expired

##  Appendix B Further Mutation Details

Mutation is performed by selecting either the root node or one of the hidden
state nodes uniformly randomly. Once a node is chosen, either the operation or
connection is changed. With 70% probability, the operation is changed either
from unary to unary, or binary to binary, depending on the original operation.
With 15% probability, the operation was swapped from unary to binary, or
binary to unary, depending upon the original operation. If the original
operation was unary, the extra connection obtained from swapping to binary was
randomly selected from the set of input and hidden state nodes. If the
original operation was binary, then either the left or right connection was
eliminated with 50% probability. If the operation was not changed or swapped,
the remaining 15% probability went to changing the connection of the node. If
the operation was unary, then the connection was changed uniformly random from
the set of other possible connections. If the operation was binary, then with
20% probability the argument connections were swapped; else, either were
changed uniformly randomly.

The child loss was then trained using the surrogate function. Despite the
integrity check performed when creating the loss functions, losses can still
not learn during training. To prevent wasting computation, within the
surrogate function an early stopping mechanism was incorporated to stop the
learning process for degenerate losses. If this occurred, another child loss
was obtained through mutation. If seven mutated child loss functions in a row
failed integrity checks, the mutation procedure was considered to be failed,
and another iteration of the algorithm was performed. The entire genetic
algorithm used in this work is detailed in Algorithm 2.

##  Appendix C Further Implementation Details

All architectures, training strategies, and loss functions were implemented in
Tensorflow (Abadi et al., 2015). The code base for the EffNetV2Small model and
RandAug regularization strategy were derived from the repository (Li, 2022).
The code base for WideResNet and SEResNeXt were derived from the repository
(Li, 2017). All of our code is freely accessible through our repository
(Anonymous, 2023). For computational resources, we had access to a single
NVIDIA A100 GPU during evolution.

##  Appendix D Training Strategies

Table 12 shows the adapted progressive RandAug regularization training
strategy for CIFAR-10 and CIFAR-100 for EffNetV2Small. The surrogate function
was trained for only 8,000 steps of phase 1. The proposed loss function
elimination protocol trained on each phase incrementally, where each phase
contained 16,000 steps. EffNetV2Small was always trained using the Adam
optimizer (Kingma & Ba, 2014) with a maximum learning rate of 0.001 and a
batch size of 64. The first 6,400 steps were used to warm up the learning rate
linearly from 0 to the maximum, and then the learning rate was decayed using
cosine annealing back down to 0 during the rest of the training cycle. This
training strategy was only used for the surrogate function and loss function
elimination protocol.

Table 12: Adapted progressive RandAug regularization training strategy for CIFAR-10 and CIFAR-100. Phase |  1 |  2 |  3 |  4
---|---|---|---|---
Image Size |  128 |  160 |  192 |  224
Dropout |  0.10 |  0.20 |  0.30 |  0.40
RandAug |  5 |  8 |  12 |  15

When comparing the final loss functions after evolution and the elimination
protocol in Section 4, the training strategy was adjusted for EffNetV2Small to
allow for a longer training session. Each phase was increased to 24,000 steps,
allowing for 96,000 total steps over the entire course of training. The same
optimizer and learning rate strategy were used as before, except the learning
rate was held at its maximum of 0.001 for 12,400 steps before being decayed.

For WideResNet and SEResNeXt, SGD with Nesterov momentum of 0.9 was used with
a learning rate scheduler that warmed up to 0.1 after 1,000 steps, and then
was cosine decayed back down to 0. This same training strategy was used by
Zagoruyko & Komodakis (2016) for the WideResNet architecture, except they
decayed their learning rate with a piece-wise function rather than cosine.
With a batch size of 128, a total of 96,186 steps were performed. For image
augmentation, the images were resized to 40$\times$40, randomly cropped back
to 32$\times$32, cutout was applied with an 8$\times$8 mask, and then the
images were randomly flipped horizontally.

For DavidNet, two different image augmentation techniques were applied. For
both, SGD with Nesterov momentum of 0.9 was used with a learning rate
scheduler that warmed up to 0.4 after 1,176 steps, and then was linearly
decayed. For the first configuration using the random crop, cutout, and
horizontal flipping, a weight decay value of 0.512 was also applied to the
gradients. This training strategy was used by the original DavidNet proposal
for the DAWNBench competition (Coleman et al., 2019), an end-to-end deep
learning benchmark for training time, cost, and accuracy for CIFAR-10 and
ImageNet. With a batch size of 512, 98 steps were performed per epoch, for 30
epochs, totalling to a small budget of 2,940 steps. As a result, the loss
functions are evaluated for whether or not they can increase generalization
within a very computationally stringent budget. For the second image
augmentation technique, CutMix, the exact same parameters and setup were used
for DavidNet, except the weight decay was reduced to 0.0256, as CutMix is a
much harsher augmentation technique than simple cutout.

When training EffNetV2Small on Stanford-CARS196, Oxford-Flowers102, and
Caltech101 from scratch, a few minor changes were in-placed. The same
optimizer, learning rate, schedule, and total steps were used as before;
however, the progressive RandAug regularization strategy was reverted back to
its original form (Tan & Le, 2021) for the much larger image resolution
datasets. These changes are reflected in Table 13. Due to the increased image
resolution, the batch sizes for EffNetV2Small were decreased to 32.

When fine-tuning, the initial weights were set to the official ImageNet2012
weights, and the loss functions were trained for only 20 epochs, with 500
steps per epoch, totalling to only 10,000 steps. The images were set to
224x224, dropout to 0.5, RandAug magnitude to 15, batch size to 64, ptimizer
to Adam, and learning rate schedule that warmed up for one epoch to 0.001
before being decayed back down to zero after being held for three epochs.

Table 13: Original RandAug progressive regularization training strategy for EffNetV2Small for large resolution image datasets. Phase |  1 |  2 |  3 |  4
---|---|---|---|---
Image Size |  128 |  186 |  244 |  300
Dropout |  0.10 |  0.17 |  0.24 |  0.30
RandAug |  5 |  8 |  12 |  15