The NLC is continuous from layer to layer (section <ref>) According to the nonlinearity path equation, $\mathfrak{n}$ increases gradually and multiplicatively as we include more and more activation layers.

§ DEFINING AND EXPLAINING GAUSSIAN STABILITY

We have repeatedly encountered the notion of Gaussian stability and Gaussian instability throughout this work. (Gaussian instability simply denotes the absence of Gaussian stability.) In the context of our empirical studies, we defined Gaussian unstable architectures (GUAs) and Gaussian edge architectures (GEAs) in terms of activation functions and normalization methods as given at the end of section <ref> / <ref>, with a full list of GUAs and GEAs available in the appendix in chapter <ref>. GUAs have shown distinct empirical behavior in sections <ref>, <ref>, <ref>, <ref>, <ref>, <ref>, <ref> and <ref>. The aforementioned subsections of section <ref> demonstrate properties of the NLC that are explained by mean field theory in section <ref>. Hence, as long as an architecture's behavior in the limit of infinite width is predictive of its behavior at a practical width, we expect these properties of the NLC to hold. In section <ref> and <ref>, we found that the limit is often not predictive for GUAs. This explains why these properties of the NLC hold to a lesser degree for GUAs.

While Gaussian instability causes mean field theory to break down, it can nonetheless be explained using mean field theory, as we do below. The explanation will lead to the precise definition of the property of `mean field Gaussian stability', which then informs the more practical, “fuzzy” definition of Gaussian stability.

We begin by considering a setup like the one used in section <ref>. Let an architecture $f$ be composed of $M+1$ macro-layers which are themselves composed of an FC and an activation layer. Let the FC layers be LeCun Gaussian initialized and let all activation layers use activation function $\tau$. Then $\mathfrak{l}_L = \mathfrak{L}_\tau^{M+1}(\mathfrak{l}_0)$. Further, assume that $\tau$ is normalized to have $\mathfrak{L}_\tau(1) = 1$ and that $\mathbb{E}_ix[i]^2 = 1$ for all inputs $x$, so that $\mathfrak{l}_0=1$ and $\mathfrak{l}_L = 1$. Let's see what happens when we forward-propagate some input $x$. After the first macro-layer, we obtain $\mathfrak{l}_2=1$. (The 2 subscript arises because the macro-layer contains two layers.) If the layer is sufficiently wide, we also obtain $\mathbb{E}_if_2(x)[i]^2 = 1 + \epsilon$ for some small $\epsilon$. However, because the macro-layer still has finite width and thus cannot follow mean field theory exactly, we generally do not obtain $\epsilon = 0$. For the remainder of the architecture, we can now take two approaches for our mean field recursion. Either, we continue to recurse on $\mathfrak{l}_2=1$ to obtain $\mathfrak{l}_L=1$, or we recurse on $1+\epsilon$, which we denote by $\tilde{\mathfrak{l}}_2$, to obtain $\tilde{\mathfrak{l}}_L = \mathfrak{L}_\tau^M(1+\epsilon)$. For example, we can obtain $\tilde{\mathfrak{l}}_L$ by applying theorem <ref> with $f_3$ as the readin layer. If $\epsilon$ is sufficiently small, we further obtain $\tilde{\mathfrak{l}}_L\approx 1 + \epsilon\frac{d\mathfrak{L}_\tau^M(1)}{d\lambda} = 1 + \epsilon\mathfrak{L}_\tau'(1)^M$.

In section <ref>, we described the possible behaviors of $\mathfrak{L}_\tau$ upon iteration near a fixed point. If $\mathfrak{L}_\tau'(1) < 1$, the fixed point at 1 is stable and $\tilde{\mathfrak{l}}_l$ converges to 1 as $l$ increases. If $\mathfrak{L}_\tau'(1) = 1$, the fixed point is stationary and $\tilde{\mathfrak{l}}_l$ remains at $1 + \epsilon + O(\epsilon^2)$. If $\mathfrak{L}_\tau'(1) > 1$, the fixed point is unstable and $\tilde{\mathfrak{l}}_l$ diverges from 1 as $l$ increases. However, divergence also implies that $\tilde{\mathfrak{l}}_l$ diverges from $\mathfrak{l}_l$. Hence, if the activation function in $f$ has $\mathfrak{L}_\tau'(1) > 1$, mean field theory makes conflicting estimates for $\mathbb{E}_if_l(x)[i]^2$ for sufficiently large $l$, and hence mean field theory cannot be predictive for sufficiently deep $f$. In plain terms, mean field theory predicts that small deviations in the layer quadratic means (LQMs) of $f$ grow exponentially during forward propagation. However, since such deviations are always introduced in a practical network due to finite width, the LQMs of $f$ diverge from their mean field limit during forward propagation, and the mean field estimate becomes inaccurate. This constitutes a “breakdown” of the practical predictiveness of not just theorem <ref>, but also of mean field theory given in prior work as described in e.g. section <ref>. To our knowledge, this breakdown has not been described in literature.

Let's apply this reasoning to a more general setup. Let a network $f$ be composed of $M+1$ macro-layers composed of an FC and an activation layer, where the $\sigma_l^2$ and $\tau_l$ are arbitrary. Then $\tilde{\mathfrak{l}}_L \approx \mathfrak{l}_L + \epsilon\prod_{\tau_l}\mathfrak{L}_{\tau_l}'(\mathfrak{l}_k)$, where the product is taken over the last $M$ activation layers in $f$. We could argue that divergence happens if the value of the product is above a certain level. However, as is a recurring theme in this work, we must consider the deviation relative to the value from which it deviates. Say we have $\mathfrak{l}_L = 1$. Then we would consider $\tilde{\mathfrak{l}}_L = 1 + 1 = 2$ to be a significant deviation. However, if $\mathfrak{l}_L = 10^5$, then we would consider $\tilde{\mathfrak{l}}_L = 10^5 + 1$ to be a small deviation. Also consider that the value of both $\mathfrak{l}_L$ and $\tilde{\mathfrak{l}}_L$ can be arbitrarily multiplicatively scaled by appending an FC layer with an arbitrary initial weight variance to the architecture. Appending such an FC layer clearly should not affect our judgment of whether “mean field theory has broken down”. The same argument applies at layer $f_2$. If $\mathfrak{l}_2$ is small, then a given deviation $\epsilon$ is more significant than if $\mathfrak{l}_2$ is large, because our judgment of how severely an LQM deviates from the mean field estimate should not depend on e.g. whether $\tau_2$ is scaled by a constant.

Hence, we care about the relative deviation at the output $\frac{\epsilon\prod_{\tau_l}\mathfrak{L}_{\tau_l}'(\mathfrak{l}_k)}{\mathfrak{l}_L}$ relative to the relative deviation at the input $\frac{\epsilon}{\mathfrak{l}_2}$. Table <ref> yields that the ratio of those ratios is equal to the telescopic product $\prod_{\tau_l}\frac{\mathfrak{l}_k\mathfrak{L}_{\tau_l}'(\mathfrak{l}_k)}{\mathfrak{L}_{\tau_l}(\mathfrak{l}_k)}$. So, we can say that the deviation relatively grows at some activation layer $f_l$ if $\frac{\mathfrak{l}_k\mathfrak{L}_{\tau_l}'(\mathfrak{l}_k)}{\mathfrak{L}_{\tau_l}(\mathfrak{l}_k)} > 1$, and relatively shrinks if $\frac{\mathfrak{l}_k\mathfrak{L}_{\tau_l}'(\mathfrak{l}_k)}{\mathfrak{L}_{\tau_l}(\mathfrak{l}_k)} < 1$. This idea is now formalized.

We say an activation function $\tau$ exhibits `mean field Gaussian stability' at $\lambda > 0$ if $\frac{\lambda\mathfrak{L}_{\tau}'(\lambda)}{\mathfrak{L}_{\tau}(\lambda)} < 1$, `mean field Gaussian instability' if $\frac{\lambda\mathfrak{L}_{\tau}'(\lambda)}{\mathfrak{L}_{\tau}(\lambda)} > 1$ and is `mean field Gaussian edge' if $\frac{\lambda\mathfrak{L}_{\tau}'(\lambda)}{\mathfrak{L}_{\tau}(\lambda)} = 1$. (Note that $\frac{\lambda\mathfrak{L}_{\tau}'(\lambda)}{\mathfrak{L}_{\tau}(\lambda)} \le -1$ is excluded by theorem <ref>.)

If we substitute $\sqrt{q} = \lambda$ and $\sqrt{\mathfrak{C}_\tau(q,q)} = \mathfrak{L}_{\tau}(\lambda)$, we obtain $\frac{\lambda\mathfrak{L}_{\tau}'(\lambda)}{\mathfrak{L}_{\tau}(\lambda)} = \frac{q\mathfrak{C}_{\tau}'(q,q)}{\mathfrak{C}_{\tau}(q,q)}$, where the derivative is with respect to the double argument $q$. Hence, mean field Gaussian stability can be defined equivalently in terms of the length kernel or covariance kernel.

We already analyzed $\frac{\lambda\mathfrak{L}_{\tau}'(\lambda)}{\mathfrak{L}_{\tau}(\lambda)}$ as a function of $\lambda$ in section <ref> and plotted that function in tables <ref> through <ref>. We find that for activation functions used in our study A architectures at $\lambda=1$, stability occurs for SELU, tanh, sigmoid, even tanh and Gaussian. Instability occurs for square and odd square. ReLU is mean field Gaussian edge. The same thing holds for the debiased activation functions. This is the motivation behind which activation functions we include in our definition of GUAs and GEAs.

Mean field Gaussian instability does not only lead to the divergence of practical LQMs from their limit. By the same argument, it also leads to the divergence of practical LQMs corresponding to different inputs from each other. This observation provides the desired link to our empirical results. In section <ref>, we investigated the LCV metric, which measures the coefficient of variation of LQM values. Out of all the metrics we investigated in section <ref>, LCV provided the clearest signal. Especially before training, stable architectures all had extremely low LCV values, whereas many GUAs and GEAs had large ones. In proposition <ref>, we showed that layer values generated by a meta-Gaussian have small LCV. Hence, large LCVs imply that layers are not meta-Gaussian, and we found this to be the case for our GUAs throughout section <ref>. As stated at the beginning of this section, the breakdown of mean field theory implied by non-meta-Gaussian fully-connected layers then explains our observations in section <ref>. We stress that mean field Gaussian instability does not cause our theorems <ref> and <ref> to become false. However, it can cause the width that is required for a practical architecture to behave as its limit to become enormous.

It is also clear how a large LCV value would interfere with the accuracy of mean field predictions of e.g. the NLC. The strongest condition of theorem <ref> is that the input distribution is elem-like, which implies that all inputs have the same length. Mean field theory is an iterative theory, i.e. results for child layers are built from results for their parents. Hence, layer lengths remaining approximately constant throughout this recursion is a necessity.

While Gaussian instability can be reduced to the behavior of the length kernel in our setup above, the complexity of the phenomenon increases with the complexity of the architecture definition. Consider for example an architecture that also contains normalization layers between fully-connected and activation layers. If layer normalization is used, then LCV becomes 0 at the normalization layers, i.e. the deviation $\epsilon$ is eliminated. This prevents the exponential compounding of the instability at each activation layer, and hence the architecture as a whole behaves as if it contains mean field Gaussian stable activation functions. Hence, when a fully-connected architecture contains LN, we do not consider it a GUA or GEA in our studies. However, the story is different when batch normalization is used. BN only causes the layer quadratic mean across the entire batch to be 1, but not for individual inputs. This means the LQMs corresponding to different inputs in the same batch can still deviate, and the architecture can exhibit unstable behavior. In fact, this phenomenon is one of the key differences between LN and BN.

Providing an exhaustive analysis of Gaussian stability across architecture types goes beyond the scope of this work. Hence, we must settle for an imprecise definition of Gaussian stability in terms of the empirical observations we have made.

We say a neural network $f$ exhibits `Gaussian stability' if values of its fully-connected or convolutional layers are distributed as a draw from an elementwise meta-distribution generated by a meta-Gaussian when the data distribution is neural regular as defined in section <ref>. Gaussian stability can be measured by e.g. the NKURT, NGHI, CVNSTD, NCORR and especially the LCV metric given in section <ref>.

This definition makes no direct reference to mean field theory. We also include convolutional networks in the definition. The theoretical justification for meta-Gaussian convolutional layers is similar to that for meta-Gaussian fully-connected layers. We briefly discuss the mean field theory of convolutional architectures in the next section. Among activation functions used for our study B convolutional architectures, not only are square and odd square mean field Gaussian unstable at $\lambda=1$, but also Swish, softplus-debiased and Swish-debiased. However, their instability is much less severe than that of square and odd square. In practice, we find that only some architectures based on a linearized version of abs. val.-debiased exhibit clear Gaussian instability as captured by the above definition. We specify those architectures at the end of section <ref> / <ref> and in section <ref>. They are considered GUAs in our empirical studies. Further, layer normalization is not sufficient to prevent Gaussian instability in convolutional architectures, because it does not prevent the quadratic mean across some spatial regions of the layer value to diverge from the quadratic mean across other spatial regions. Hence, there are GUAs using LN in study B.

Gaussian instability can have extreme effects. Let's return one more time to our setup with alternating LeCun Gaussian initialized FC and activation layers. Let's say the square activation function is used, normalized to achieve $\mathfrak{L}_\tau(1) = 1$. When computing the recursive calculation rules of table <ref>, even if we compute the length kernel as accurately as possible under 32-bit floating-point precision, i.e. with a rounding error of around $10^{-7}$, at around macro-layer 30 this rounding error explodes and induces numerical underflow or overflow. Of course, this explosion happens much more quickly when we compute the length kernel inaccurately or during actual forward propagation in the finite-width network.

When we designed empirical studies A and B, we always observed that architectures based on square or odd square and no normalization layers exhibited numerical underflow or overflow. Hence, we decided to always combine square and odd square activation functions with normalization layers. We made this decision before we understood the phenomenon of Gaussian stability.

Gaussian stability plays an important role when computing metrics via statistical estimation, as we explained in sections <ref> and <ref>. Our metrics are often based on operators such as expectation and standard deviation applied to layer quantities.

Gaussian stability bears similarity to scale stability as given in section <ref>. We further distinguish the two in section <ref>.

§ MEAN FIELD THEORY OF CNNS: AN OUTLOOK

Due to time and space limitations, we do not extend our analysis in this chapter to architectures with convolutional layers. Yang, 2019, Xiao et al., 2018 show how to apply mean field theory to convolutional architectures. To apply mean field theory to A-architectures, which can contain a range of layer operations, we re-cast them as mean field architecture as defined in section <ref>, which can contain only two layer operations, one of which can be considered a “template”. For convolutional architectures, we can follow the same steps to derive recursive calculation rules for e.g. the square mean and co-mean of layers that use operations such as the convolutional and pooling operation in the style of table <ref>. However, there are a number of additional complexities.

Consider a tensor layer as defined in section <ref> with spatial dimensions $S_{l,1}, S_{l,1}, .., S_{l,T_l-1}$ and $C_l$ channels. When re-casting the convolutional architecture to a mean field architecture, in general, that tensor layer is cast not as one layer in the mean field architecture, but as $S_{l,1}S_{l,1}..S_{l,T_l-1}$ layers. For each tensor layer in the convolutional architecture, we require one layer in the mean field architecture per spatial location, each of which has width $C_ld_\text{MF}$. Consider, then, a convolutional layer with a weight tensor that has spatial dimensions $H_{l,1}, H_{l,2}, .., H_{l,T_l-1}$. In the mean field architecture, the convolutional layer corresponds to $H_{l,1}H_{l,1}..H_{l,T_l-1}S_{l,1}S_{l,1}..S_{l,T_l-1}$ fully-connected layers, each with a weight matrix of size $C_kd_\text{MF} \times C_ld_\text{MF}$. (Here, we assume the dependency is padded.) Each fully-connected layer is responsible for the contribution of one spatial location in the dependency to one spatial location in the convolutional layer. Finally, we add the contributions to each spatial location in the convolutional layer together with a single elementwise or addition layer per spatial location.

At the end of this process, we obtain for one tensor layer not one value of $\mathfrak{m}$, $\mathfrak{c}$, $\mathfrak{q}$ and $\mathfrak{g}$, but one for each spatial location. Of course, this is significantly more unwieldy than the fully-connected case. It is consequently somewhat more difficult to extract simple yet practically meaningful patterns from the recursion.

The technical conditions of mean field theory are also more difficult to (approximately) fulfill for convolutional architectures. If the input $x$ comes as a tensor, then in order to make theorems <ref> and <ref> work, we would require an elem-like condition that states that for all spatial locations and inputs, the square mean across channels is approximately equal to some globally fixed $q$. Since tensor-shaped inputs often have a small number of channels (e.g. 3 for color images), this condition is more severe than the equivalent condition for non-tensor inputs. Further, the co-mean for all pairs of inputs and spatial locations would have to be approximately equal to some globally fixed $c$.

In section <ref>, we showed that the covariance structure of the input is important for the value of the NLC in convolutional architectures, but not for fully-connected architectures. This is explained by the mean field theory of CNNs viewing different spatial locations in a single input as distinct inputs.

In future work, we plan to apply heuristics to simplify the mean field recursion for convolutional architectures to obtain practically relevant insights. Now, we present a first attempt at this which establishes that mean field theory is indeed applicable to the analysis of the nonlinearity of convolutional architectures. We greatly simplify the nonlinearity path equation to

$$\mathfrak{n} \approx  \prod_{\tau_l \in f} \mathfrak{n}_{\tau_l}(1,0)$$

Initial NLC vs the product of activation function NLCs with respect to the unit Gaussian, for non-residual study B architectures with debiased activation functions. GUAs are displayed in green and in the foreground. Conclusion: Despite the gross simplification of the mean field NLC, this estimate is predictive of the NLC.

We estimate the mean field NLC of the network as the product of activation function NLCs with respect to the unit Gaussian. In figure <ref>, we plot $\prod_{\tau_l \in f} \mathfrak{n}_{\tau_l}(1,0)$ vs the initial NLC for some study B architectures in the initial state. Note that all our study B architectures contain 19 instances of the same activation operation. We find that our rough estimate of $\mathfrak{n}$ is still strongly predictive of the NLC. This is remarkable as this estimate does not take into account varying $\mathfrak{q}_k$ and $\mathfrak{c}_k$ values, tensor structure, pooling, normalization layers or data augmentation, among other things. Note that we exclude residual architectures in figure <ref> as our estimate of $\mathfrak{n}$ cannot account for multiple paths in the layer graph. We also exclude architectures that do not use a debiased activation function so that our simplifying assumption $\mathfrak{c}_k=0$ has a greater chance of working.

Of course, the full nonlinearity path equation still holds for convolutional architectures under the proper technical conditions, though there is now one $\mathfrak{n}_\tau$ value per spatial location and activation layer. The number of paths is exponential in the number of sequentially composed convolutional layers. However, one key property of the nonlinearity path equation still holds. Replacing an activation function with another that has a higher $\mathfrak{n}_{\tau_l}(\mathfrak{q}_k,\mathfrak{c}_k)$ value increases $\mathfrak{n}$, assuming that there are no knock-on effects on downstream $\mathfrak{q}$ and $\mathfrak{c}$ values. We can eliminate knock-on effects by ensuring that $\mathfrak{q}_l$ and $\mathfrak{c}_l$ are not altered by the substitution. This is the insight behind the nonlinearity normalization algorithm which we present in chapter <ref>.

While convolutional mean field theory is more cumbersome, it also explains the great performance gains that a convolutional architecture can induce relative to a fully-connected architecture. In figure <ref>, we find that CNNs greatly outperform FCNs. In section <ref>, we explained how FCNs are determined by their covariance kernel, which is based on the input co-mean. CNNs can have much richer covariance kernels, which tend to map semantically similar images to large kernel values even in the initial state. CNNs encode information about images before training even begins. Novak et al., 2019 showed that fairly high performance can be attained by applying Bayesian inference to the Gaussian process to which a CNN converges. Saxe et al., 2011, Jarrett et al., 2009 showed that fairly high performance can be attained by training only the last linear layer of a CNN and retaining the random weights in other layers. Xiao et al., 2018, Ortiz-Jimenez et al., 2020 investigated the feature representations of CNNs in terms of their frequency spectrum. Tancik et al., 2020 showed that fully-connected architectures can learn high frequencies present in data much more easily when hard-coded Fourier features are added to the input.

CHAPTER: BEYOND THE NLC - EXPLAINING THE PERFORMANCE OF NEURAL ARCHITECTURES

In the previous two chapters, we have gone a long way towards establishing “$1 \le NLC \le 5$” (sections <ref>, <ref>) as a ZSAD guideline that fulfills the utility criteria of figure <ref>. We continue this task in this and later chapters. In this chapter, we focus on other ZSAD guidelines, and therefore on other things that can go wrong when designing an architecture. While a large fraction of this work explores the NLC in-depth, the purpose of this chapter is to add breadth.

How should we choose which guidelines to investigate? Of course, we do not simply want to present a disjointed list of minimally validated concepts. Instead, we use the following criteria to decide what to cover.

* The guideline has a significant explanatory influence on the test or training error of our study A and B architectures.

* The guideline influenced the design of the architectures in our studies.

* The guideline is explainable based on results from prior chapters. For example, we have derived insights from mean field theory to explain the NLC theoretically. In this chapter, we will apply these insights to other guidelines.

* The guideline is related to the NLC on a conceptual and / or quantitative level.

* The guideline can be used to explain popular building blocks and design strategies.

Many of our ZSAD guidelines fulfill multiple of these criteria.

Of course, we cannot investigate each guideline we cover in this chapter to the same depth as the NLC, due to space and time limitations. To mitigate the limitations of brevity, we draw heavily from analysis presented in prior chapters. To a significant degree, this chapter simply re-interprets prior analysis from the viewpoint of a different guideline. For example, robustness to input distribution and other properties follow from mean field theory like for the NLC in section <ref>. Nonetheless, some of the insights we generate here will have to be fleshed out in future work. Basing some of our guidelines on concrete, well-defined properties is also future work.

Each section from <ref> to <ref> is dedicated to one guideline which is named in the section title. In section <ref>, we document the importance of exhaustive learning rate tuning and present some intriguing results which might lead to fleshed-out learning rate selection strategies. In section <ref>, we bring the ZSAD guidelines of this chapter and the NLC together to explain the majority of the test and training error variation of our study A and B architectures. By then also relating those guidelines to popular building blocks and design strategies in chapter <ref>, we hope to make at least designing fully-connected architectures relatively foolproof. For a more detailed guideline-by-guideline overview of this chapter, see section <ref>.

Background from prior chapters Like sections <ref>, <ref> and <ref>, this chapter is based on the empirical studies detailed in chapter <ref>. We recommend reading at least summary section <ref> before proceeding. Throughout this chapter, we use the terminology, notation and conventions of section <ref>.

We continue to analyze the NLC in this chapter, which is motivated by chapter <ref>. A complete understanding of chapter <ref> beyond section <ref> is not required. Throughout this chapter, we make references to mean field theory as studied in chapter <ref>. A high-level understanding of that chapter and an awareness of its key definitions are required to follow such references.

Technical considerations As discussed in section <ref>, we implicitly assume things like integrability (section <ref>), differentiability (section <ref>) and non-zero denominators when necessary.

§ ENSURE GAUSSIAN STABILITY

Initial NLC vs test error and vs training error after training error minimization for study A and B architectures. Graphs are identical to graphs in figures <ref> and <ref>, except that GUAs are displayed in green and in the foreground. Note that the x-axis range differs significantly between graphs A-E and F-G. Keep in mind that throughout this work, axis ranges can vary when related but not identical metric values are depicted. However, we also strive to keep axis ranges the same for comparability when possible. Conclusion: In general, GUAs greatly underperform other architectures with a similar NLC.

Gaussian stability has been important throughout this work. We defined and discussed it in detail in section <ref>. In this section, we add the observation that Gaussian stability is also highly desirable for network performance. In figure <ref>A-G, we repeat the following graphs in order: <ref>A, <ref>B, <ref>C, <ref>A, <ref>B, <ref>D, <ref>B. These depict NLC before training vs test and training error after training for our study A and B architectures. The difference is that this time we display some points in green. Like in many previous figures and all figures throughout this chapter, green markers represent Gaussian unstable architectures (GUAs) as defined at the end of section <ref> / <ref>. Green markers are displayed in the foreground, and can therefore partially or fully occlude other markers. We find that a large fraction of architectures that perform significantly below what their initial NLC level would suggest are in fact GUAs. This holds especially for our convolutional architectures. Hence, Gaussian stability is not just necessary for accurate mean field estimates, but also for performance.

`Ensure Gaussian stability' requires that a network exhibits Gaussian stability as defined in section <ref>.

More work is needed to uncover the mechanisms behind the performance impact of Gaussian instability. A crude interpretation is as follows. Gaussian instability is associated with high LCV values (section <ref>). The gradient with respect to a weight tensor is the product (for FC layers) or convolution (for convolutional layers) of the layer value of the dependency with the gradient with respect to the linear layer. If the layer quadratic means of the dependency differ drastically from one datapoint to another for reasons that are “random”, i.e. not related to information present in the inputs, then the weight gradient of an entire batch will be dominated by the weight gradients of a small number of datapoints in that batch, leading the learning process to focus entirely on those few datapoints.

Gaussian stability, at least in simple architectures, stems from mean field Gaussian stability, as explained in section <ref>. Being a mean field property, it is robust to data distribution, random initialization and width change as is the NLC, as explained in section <ref>. Hence, the guideline of Gaussian stability is as data-agnostic as the NLC (sections <ref>, <ref>) and can be fundamentally regarded as an architecture property (section <ref>).

§ ENSURE SCALE STABILITY, AND SPECIFICALLY $LSCALE_L \APPROX 1$

We defined and outlined the ZSAD guideline of scale stability in section <ref>. In this section, we analyze it in detail. Note that scale instability denotes simply the absence of scale stability.

Scale stability and Gaussian stability are somewhat similar. Both properties ultimately deal with neuron values. The difference is that scale stability requires that the overall magnitude of neuron values across inputs and neurons is stable, which implies that layer quadratic mean (LQM; table <ref>) values are not too large or small overall. On the other hand, Gaussian stability requires that LQM values do not diverge relative to each other, as this would “break” meta-Gaussianity. Gaussian stability is (partially) captured by the LCV metric, whereas scale stability could be captured by e.g. the LSCALE metric (table <ref>). In mean field terms, scale instability is associated with small or large $\mathfrak{q}$ values but not necessarily non-Gaussian neuron distributions in linear layers. Conversely, Gaussian instability is associated with inaccurate $\mathfrak{q}$ values and non-Gaussian neuron distributions in linear layers, but $\mathfrak{q}$ itself does not necessarily become large or small. Of course, in the end, the distinction between both concepts is somewhat fuzzy and subjective.

Whereas Gaussian instability represents a breakdown of mean field theory, scale instability has a natural equivalent within mean field theory.

Initial $LSCALE_L$ and test error as the initial weight variance and the loss function varies, for depth-5 fully-connected ReLU architectures on waveform-noise. A weight variance scaling factor of 1 corresponds to He initialization. Results are obtained by averaging 100 independent runs corresponding to 100 random seeds. (Each of the 100 runs conducts 40 independent training runs with different starting learning rates.) $LSCALE_L$ is averaged in log space. Conclusion: Graph A: Softmax+cross-entropy performs optimally when network outputs have approximately unit quadratic mean, and then outperforms the other loss functions. Graph B: The introduction of bias layers causes basic gradient descent to fail for small initial weights.

We say an A-architecture $f$ exhibits `mean field scale stability' if at all layers $f_l$, $\mathfrak{l}_l = \sqrt{\mathfrak{q}_l}$ is close to 1. $\mathfrak{q}_l$ is defined according to table <ref>.

Because scale stability “has a mean field limit”, like Gaussian stability, it is relatively data-agnostic and can be fundamentally regarded as an architecture property, especially if we also assume that $\mathfrak{q}_0$ is close to 1.

The importance of scale stability stems from deep learning building blocks being designed to work best with LQMs around 1. Below, we discuss three different manifestations of this phenomenon. In section <ref>, we show how scale stability becomes irrelevant when the neuron value magnitude is deliberately taken into account during architecture design.

Activation functions tend to work best under scale stability Consider the $\mathfrak{n}_\tau(\lambda^2, 0)$ curves from tables <ref> through <ref>. The activation function NLC of many activation functions changes drastically as the $q$ / $\lambda^2$ parameter changes. Hence, the mean field NLC $\hat{\mathfrak{n}}_{l,k}$ of an activation layer can change drastically as the parameter and hence the magnitude of the input to the activation layer $\hat{\mathfrak{q}}_k$ grows. Of course, $\hat{\mathfrak{n}}_\tau\approx \mathfrak{n}_\tau$ is related to the mean field NLC of the architecture via the nonlinearity path equation. The mean field NLC is related to the NLC as shown in section <ref>. Therefore, uncontrolled growth or decay of layer values often corresponds to uncontrolled change in network NLC, and therefore to uncontrolled change in performance.

Many activation functions are designed with the assumption in mind that the LQMs of the dependency are around 1. For example, the sigmoid and tanh activation functions are meant to be continuous versions of the step function (section <ref>). However, if absolute neuron values in the dependency are much less than 1, they become equivalent to linear functions, i.e. they become `pseudo-linear' (section <ref>). If absolute neuron values in the dependency tend to be much greater than 1, they become equivalent to the discrete step function. Similarly, the softplus and Swish activation functions are smooth versions of ReLU. For large inputs, they become ReLU. For small inputs, they become pseudo-linear. Their $\mathfrak{n}_\tau$ values change accordingly, as discussed in section <ref>. It is worth noting that ReLU itself stands out in this regard. $\mathfrak{n}_\text{ReLU}(\lambda^2, 0)$ is independent of $\lambda$ because $\tau_\text{ReLU}(cs) = c\tau_\text{ReLU}(s)$ for all $c > 0$. In this way, ReLU networks are less reliant on scale stability and thus more forgiving of suboptimal design choices.

The softmax+cross-entropy loss function tends to work best under scale stability Consider the following two alternative loss functions, which we denote by $\ell_\text{large}$ and $\ell_\text{small}$.

\begin{eqnarray*}
\ell_\text{large}(f,y) &=& \max_{i_L}f[i_L] - f[y]\\
\ell_\text{small}(f,y) &=& \mathbb{E}_{i_L}f[i_L] - f[y]
\end{eqnarray*}

Denote the softmax+cross-entropy loss function by $\ell_\text{S+CE}(f,y)$ (section <ref>). It is easy to check that $\ell_\text{S+CE}(cf,y) \approx \ell_\text{large}(cf,y)$ for large $c$ and that $\ell_\text{S+CE}(cf,y) \approx \ell_\text{small}(cf,y)$ for small $c$. A priori, it is not clear whether $\ell_\text{S+CE}$, $\ell_\text{large}$ or $\ell_\text{small}$ is best for any given task or architecture. To gain some empirical understanding, we trained depth-5 ReLU architectures on waveform-noise. Specifically, we trained 7 architectures with different initial weight variances. The first architecture is He initialized, which leads to $LQM_L$ values around 1, as explained in section <ref>. We then created 6 additional architectures by multiplying the initial weight variances jointly by $10^{-3}$, $10^{-2}$, $10^{-1}$, $10^1$, $10^2$ and $10^3$ respectively. We ran our full training protocol for each architecture-loss function pair 100 times with different random seeds and averaged results across random seeds.

The results are given in figure <ref>. We find that for $\ell_\text{large}$ and $\ell_\text{small}$, test error is independent of the weight variance scaling factor, even though the magnitude of the output, as measured by $LSCALE_L$, differs by many orders of magnitude. Because $\ell_\text{large}(cf,y) = c\ell_\text{large}(f,y)$, $\ell_\text{small}(cf,y) = c\ell_\text{small}(f,y)$ and $\tau_\text{ReLU}(cs) = c\tau_\text{ReLU}(s)$ for $c > 0$, a ReLU network with one of our alternative loss functions is in some sense scale-invariant, which causes the learning dynamics to be preserved as long as the learning rate is adjusted to compensate for the scaling factor. We explore scale invariance further in section <ref>. The learning rate adjustment is performed automatically by our training protocol (section <ref>). In contrast, we find that for $\ell_\text{S+CE}$ the following happens. (i) For large scaling factors, the test error is equivalent to $\ell_\text{large}$. (ii) For small scaling factors, the test error is equivalent to $\ell_\text{small}$. (iii) The test error under $\ell_\text{S+CE}$ is minimized by $LSCALE_L\approx 1$. The first two findings are expected based on the construction of our loss functions. The third finding implies that $\ell_\text{S+CE}$ achieves the lowest test error out of the three loss functions. Again, while this is not clear a priori, it is somewhat expected given that $\ell_\text{S+CE}$ is universally popular but $\ell_\text{large}$ and $\ell_\text{small}$ are not. However, the gap in performance is surprising. The error under $\ell_\text{small}$ is more than twice as high as under $\ell_\text{S+CE}$! Hence, scale stability at the output layer can be necessary for high performance under $\ell_\text{S+CE}$.

Having a single learning rate for all layers tends to work best under scale stability One of the key properties of gradient descent is that the same learning rate is applied to the gradient of each parameter component for a given update. While this choice is in some sense natural, it is not necessarily optimal. Classic optimization algorithms such as Newton's algorithm and natural gradient descent can be viewed as modulating learning in different directions in parameter space based on the local shape of the objective function, which can greatly speed up learning. In the deep learning field, parameter sub-vectors of different layers can have very different meanings and properties. The Adam algorithm, which we used in section <ref> and defined in section <ref>, is the most popular way of adapting the learning rate of individual parameter components.

Scale instability can impede learning when a single learning rate is used for all parameter components. To show this empirically, again, we train depth-5 ReLU architectures on waveform-noise with different initial weight variances, using SGD as usual. This time, we use the augmented loss function from study A (section <ref>) which normalizes the network output before applying softmax+cross-entropy, thereby eliminating the influence of the weight variance scaling factor on performance via the loss function. Additionally, we now introduce bias layers into the architecture, so that a macro-layer is composed of a fully-connected layer, a bias layer and a ReLU layer.

In figure <ref>B, we give the results. Since the bias vectors are initialized to zero, we find that $LSCALE_L$ before training is the same as in graph A. Also, when straight He initialization is used, the test error is similar to graph A. In contrast to graph A, when the weight variance scaling factor is larger than 1, we find that the test error does not increase. This is because we use the augmented loss function. When the scaling factor is less than 1, we find that the error increases to random levels. This is because of the following effect. As the weight variance decreases, so does the length of the layer values. The gradient with respect to the weight matrices is multiplicatively dependent on layer values, whereas the gradient with respect to the bias vectors is not. Hence, as the scaling factor decreases, the gradient with respect to bias vectors increases relative to the gradient with respect to weight matrices. In fact, basic arithmetic yields that the gradient with respect to the bias vector in the last macro-layer is multiplied by $c^{-5}$ relative to the weight matrix gradients, where $c$ is the scaling factor. Hence, as $c$ decreases, only the bias vectors are trained. Since the majority of the architecture's modeling capacity is contained in the weight matrices, the error increases.

Again, using basic arithmetic, we find that it is trivial to circumvent this pathology by employing layer-specific learning rates. We formalize this in section <ref>. For now, we note that gradient descent can have a preference for layer values of non-extreme magnitude.

We end by formalizing the ZSAD guideline of scale stability by extending our definition from section <ref>. Based on (i) the analysis of this section, (ii) the mean field limit given in theorem <ref> and (iii) simplicity, we recommend that scale stability be measured by the LSCALE metric. Since scale stability is not as deep and meaningful a property as nonlinearity, there is likely no “single correct” metric for determining its presence.

`Ensure scale stability' requires that at each layer in a network $f$, the overall magnitude of neuron values across inputs and neurons is not excessively large or small. Specifically, we advocate measuring this via $LSCALE_l \approx 1$ at each layer $f_l$, where LSCALE is the quadratic expectation of layer quadratic means as defined in table <ref>.

§ ENSURE TRAINING STABILITY

In this work, we focus primarily on studying the impact of properties of the initial state on performance. However, we have shown that properties of the final state can, in a sense, be even more important. For example, in figure <ref>, we showed that the range of NLC values that an architecture can attain and still achieve high or even non-random performance is narrower after than before training. We show a similar property for LBIAS in the next section. This makes sense given that the ultimate goal is to obtain a network function that approximates the true input-label function, and therefore its properties, in the final state. If we ensure that desirable properties hold in the initial state, we need to ensure they do not disappear during training. This leads to the following ZSAD guideline.

`Ensure training stability' requires that the value of metrics or the presence of properties that either directly or indirectly influence architecture performance, such as LSCALE, LCV, NLC, LBIAS, Gaussian stability or noise stability, are not prone to change in an uncontrolled or harmful way during training for an architecture $f$.

Of course, the specificity of this guideline for architecture choice in particular is limited, because the extent of the change during training does depend greatly on training algorithm and learning rate. However, some architectures are indeed more resilient to such changes than others.

Consider again the impact of LQMs on the NLC. In section <ref>, we investigated how the mean field NLC $\hat{\mathfrak{n}}$ changes during training in study A architectures. Its change is dependent on the change of the parameter magnitude. We found that the parameter vector usually grows during training because of the updates that are added. This growth can be minuscule or enormous depending on the learning rate used among other factors. The $\mathfrak{n}_\tau(\lambda^2, 0)$ curves from tables <ref> through <ref> show the potential impact of parameter growth on the network NLC as discussed in the previous section. Again, we find that tanh and sigmoid are inherently susceptible whereas ReLU is not. From table <ref>, we also find that using BN before every activation layer eliminates the possibility of $\hat{\mathfrak{q}}_k$ or $\hat{\mathfrak{c}}_k$ changing, and therefore keeps the mean field NLC constant. Ioffe and Szegedy, 2015 advocated BN especially in the presence of tanh or sigmoid to eliminate what they term `covariate shift'. We further discuss this in section <ref>. Our concept of training stability certainly builds upon and is similar to covariate shift. The main difference is that while training stability requires the preservation of properties that have been shown to influence performance, we are not aware of a general argument as to why preserving the entire layer distribution is important, which is implied by covariate shift.

The harmful impact of parameter growth in sigmoid and tanh networks is widely known as the “exploding / vanishing gradient problem in sigmoid / tanh networks”, which we further discuss in section <ref>. Finally, we note that the most extreme cases of training instability, like the most extreme cases of scale instability and Gaussian instability, can induce numerical overflow and hence training failure.

§ AVOID NEURON BIAS, AND SPECIFICALLY ENSURE $LBIAS_L \APPROX 0$

We outlined the ZSAD guideline of avoiding neuron bias in section <ref>. In this section, we analyze it in detail. We measure it via the LBIAS metric. Like the NLC, LBIAS is highly predictive of performance. However, we show that in contrast to the NLC, it is possible to achieve significantly better-than-random test error with high LBIAS if the loss function, error function and training algorithm are somewhat modified. In section <ref>, we further show that NLC and LBIAS form a “dichotomy” when certain architecture design strategies are used. In chapter <ref>, we compare the NLC / LBIAS dichotomy to the exploding gradient / vanishing gradient and order / chaos dichotomies.

Like many other ZSAD guidelines, `avoid neuron bias' is not well-defined. Specifically, (i) it is not clear at what level neuron biases are pathological and (ii) it is not clear how to aggregate the biases of individual neurons to determine the “overall bias level” of a network.

With regards to (i), as usual, we must consider the bias relative to the overall magnitude of that which is biased, i.e. relative to the overall magnitude of neuron values. The meaning of having, say, $\mathbb{E}_xf_l[i_l] = 1$ for some $(l, i_l)$ pair is very different if we also have $\mathbb{S}_xf_l[i_l] = 0.001$ vs if we also have $\mathbb{S}_xf_l[i_l] = 1000$. ($\mathbb{S}$ denotes the standard deviation operator as defined in section <ref>.) The absolute value of neuron expectations can be manipulated via e.g. the initial weight variance of linear layers. In a vacuum, changing this weight variance clearly should not affect our judgment of whether a network is biased.

With regards to (ii), we believe that there are multiple ways of aggregating neuron biases that may highlight different underlying pathologies. One of the key explanations for the predictiveness of the NLC is that we need to learn a network that has a nonlinearity level comparable to that of the true input-label function. We argued in section <ref> that most practical true input-label functions have a relatively low nonlinearity level. If we apply the same reasoning to bias, then we should care about the biasedness of the network function, and therefore specifically about the biasedness of the output neurons as opposed to other neurons in the network.

These arguments lead to the following definitions.

The `layer bias' (LBIAS) is

$$LBIAS_l(f, \mathcal{D}) = \frac{\sqrt{\mathbb{E}_x||f_l(x)||_2^2}}{||\mathbb{S}_xf_l(x)||_2}$$

Further, we shorten $LBIAS_L$ to $LBIAS$. This is equivalent to how $NLC$ is the short form of $NLC_{L,0}$ defined in table <ref>.

Further, we refer to neuron biases at the output layer as `output bias'.

$LBIAS_l$ measures the magnitude of neuron values relative to the variation of neuron values in a layer. An alternative definition is $\frac{\sqrt{||\mathbb{E}_xf_l||_2^2 + ||\mathbb{S}_xf_l||_2^2}}{||\mathbb{S}_xf_l||_2}$. So if neurons in a layer are unbiased, i.e. $\mathbb{E}_xf_l=0$, then $LBIAS_l = 1$. In contrast, if neurons in a layer are highly biased, i.e. the coefficients of variation of neuron values are small, then $LBIAS_l$ is large. By definition, $LBIAS_l \ge 1$ always holds. Note that the denominators of LBIAS and NLC are identical. LBIAS is also the ratio of the LSCALE and QMNSTD metrics defined in table <ref>. Like the NLC, LBIAS does not require significant assumptions beyond those inherent in the functional-gradient paradigm, though, unlike the NLC, it does not even require a gradient or local linear approximation. See section <ref>.

LBIAS is predictive of performance In figure <ref>, we plot the LBIAS of our study A and B architectures before training vs test error and training error. The picture obtained is similar in many ways to the one from figures <ref> and <ref>, where we plotted the NLC vs error. We find that a large initial LBIAS leads architectures to attain a random test error in all cases. The cutoff point differs between datasets, but is always below 1000. To attain an optimal test error, an initial LBIAS of less than 10 is required. In contrast to the NLC, there are no high-LBIAS architectures that attain a better-than-random training error. In figure <ref>, we plot LBIAS after training vs test error and training error. All architectures that attain a better-than-random error attain an LBIAS that is very close to 1. In figure <ref> as throughout this work, whenever we plot values for metrics not based on error for architectures in the final state, we omit all architectures that did not attain a better-than-random error.

We can interpret this as follows. A high LBIAS indicates that all network outputs are relatively similar, i.e. close to some fixed non-zero expectation. If the network is randomly initialized, the output expectation would also be random. By our analysis from section <ref>, if the outputs are returned by a linear layer, the expectation would be approximately elementwise Gaussian. In general, we would expect the expectation to be non-constant across output neurons. Hence, the neuron with the highest activation is likely to be the same for all outputs, and so all inputs are assigned to the same class by the network. So at least in the initial state, networks with high LBIAS tend to assign all inputs to the same class. As with the NLC, in order for a network to have high performance, its LBIAS value should not differ too much from the true input-label function. All our datasets had roughly equal class frequency. Hence, to attain low test or training error, our architectures must assign a high value to each output neuron roughly equally often, and this roughly corresponds to low LBIAS. Hence, architectures must either “unlearn” a high initial LBIAS or will have random test and training error after training. This is what we find in figure <ref>. In figure <ref>, we find that this happens in practice only when the initial LBIAS is not too high. This behavior is very similar to the behavior found for the NLC in sections <ref> and <ref>.

The initial LBIAS, along with the test and training error, for our individual architectures is given in the appendix in chapter <ref>.

$LBIAS_l$ and mean field theory The mean field limit of $LBIAS_l$ according to theorem <ref> is $\sqrt{\frac{\mathfrak{q}_l}{\mathfrak{q}_l - \mathfrak{c}_l}}$. We have shown that the mean field estimate of both the numerator and denominator are accurate for non-GUAs in figures <ref> and <ref>. Hence, the estimate of the ratio is also accurate. Note that not only $LBIAS_l$ has such a simple mean field limit, but also other metrics that could be reasonably used to measure neuron bias. Hence, neuron bias is relatively data-agnostic and can be fundamentally regarded as an architecture property, especially if we also assume that $\frac{\mathfrak{c}_0}{\mathfrak{q}_0}$ is close to 0.

In section <ref>, we studied the mean field NLC of plain architectures as depth increases. We can do the same for the `mean field LBIAS'. Consider a plain stable$(q,1)$ architecture. There, $\mathfrak{c}_L = q\tilde{\mathfrak{C}}_\tau^M(\mathfrak{c}_0)$ and $\mathfrak{q}_L = q$. $\tilde{\mathfrak{C}}_\tau = \frac{\mathfrak{C}_\tau}{\mathfrak{C}_\tau(1)}$ is depicted in tables <ref> through <ref>. Hence, as $M$ increases, mean field LBIAS converges to $\sqrt{\frac{1}{1-c^\text{lim}}}$, where $c^\text{lim}$ is the stable fixed point of $\tilde{\mathfrak{C}}_\tau$. The convergence rate directly follows from the convergence rate of $\tilde{\mathfrak{C}}_\tau$ towards $c^\text{lim}$.

One interesting consequence of this is that at least in very simple architectures, if the input is processed to have expectation zero and hence $\mathfrak{c}_0=0$, mean field $LBIAS_l$ tends to either remain at one or increase from macro-layer to macro-layer. In figure <ref>, we plot $LBIAS=LBIAS_L$ vs the highest $LBIAS_l$ value across all layers in the initial state. Indeed, we find that across all our study A and B architectures, $LBIAS_l$ never exceeds LBIAS for any $l$ in the initial state by more than around an order of magnitude. Hence, LBIAS can, at least in simple architectures, be regarded as a meaningful “upper bound” of the biasedness of the whole initial state network. So for the purpose of this work, we will always use LBIAS as a measure for the `avoid neuron bias' guideline in general, not just for output bias. In figure <ref>, we make the same comparison in the final state. While LBIAS is always close to 1, $\max_l LBIAS_l$ can range up to 50.

Architectures with high $LBIAS_l$ values can be trained if we adjust the training protocol If the only meaningful difference between our architectures with output bias and our other architectures is that, well, their output is biased, then we should be able to eliminate the performance hit by simply normalizing the output expectation. For study A architectures, we used an augmented loss function that normalizes the magnitude of the network output. It evaluates $LSCALE_L$ before training on the training set and then divides all network outputs by that scalar value before applying softmax+cross-entropy, as described in section <ref>. We could apply the same strategy for the output expectation.

An architecture is trained and evaluated with `output debiasing' if the following augmented loss and error functions are used. Before training, $\mathbb{E}_{x \in D_\text{train}}f(\theta^{(0)},x)$ is evaluated. Then, loss and error function subtract that fixed value from the output before applying softmax+cross-entropy loss / classification error respectively.

Using our study A protocol plus output debiasing, we find that there is no significant improvement in the test or training error of biased architectures. We do not depict the results. This suggests that while LBIAS is predictive of performance, it does not tell the whole story. What if bias at intermediate layers disrupts the learning process? One way to test this hypothesis is to train our architectures by only applying gradient updates to the first fully-connected layer, which is not biased, and keeping all other parameter sub-vectors fixed. We trained our waveform-noise architectures with a modified study A protocol that includes this `first-layer-only training' and output debiasing. We give the results in figure <ref>. In graphs A-C, we plot results for test error. In graphs D-F, we plot results for training error after training error minimization. In graphs A and D, we plot LBIAS before training vs the error achieved. In graphs B and E, we plot LBIAS after training vs the error achieved. In all cases, we find that architectures with high LBIAS, and thus high $LBIAS_l$ values overall, can now achieve error levels that are somewhat comparable to other architectures. In graphs D and F, we plot LBIAS before training vs the reduction in test error achieved by training with our modified protocol vs our original protocol. We find that while the error of high-bias architectures is almost always reduced, there are some relatively unbiased architectures that experience a significant to large error increase. This is unsurprising when only one instead of all layers is trained.

Is there a way to train all layers despite bias? Let's consider a “unit” of two layers consisting of a linear layer followed by a bias layer: $f_l = f_kW_l + \beta_l$. This unit can be rewritten as follows: $f_l = (f_k - \bar{f}_k^{(0)})W_l + (\beta_l + \bar{f}_k^{(0)}W_l)$, where $\bar{f}_k^{(0)}$ is short for $\mathbb{E}_xf_k(\theta^{(0)},x)$. The first of the two terms has expectation zero, at least in the initial state. The idea is now to only train the first of the two terms while leaving the second term unchanged, thereby eliminating the influence of the bias on training. We can apply this idea to our study A architectures because all trainable parameter sub-vectors are contained in units of an FC and a bias layer. Let's see what this looks like in practice. We have $\frac{d\ell}{dW_l} = f_k^Tg_l$. If we write $f_l = (f_k - \bar{f}_k^{(0)})W_l' + (\beta_l + \bar{f}_k^{(0)}W_l'')$, then $\frac{d\ell}{dW'_l} = (f_k-\bar{f}_k^{(0)})^Tg_l$. If we apply an update based on this gradient to the single copy of the weight matrix stored in memory, this update would affect both the value of $W_l'$ and $W_l''$. However, we can compensate for this by applying a surrogate gradient $\bar{f}_k^{(0)}(f_k-\bar{f}_k^{(0)})^Tg_l$ to $\beta_l$. The resulting update to $\beta_l$ would cancel the change to $W_l''$, at least under SGD. Hence, we define the following algorithm.

`Debiased gradient descent' (DGD) is the training algorithm that performs the following updates for each unit $f_l = f_kW_l+\beta_l$ at each iteration.

\begin{eqnarray*}
\mathsf{grad} &=& \mathbb{E}_{(x,y)\in |B|}(f_k(\theta^{(t-1)},x)-\bar{f}_k^{(0)})^T\frac{d\ell(f(\theta^{(t-1)},x),y)}{df_l}\\
W_l^{(t)} &=& W_l^{(t-1)} - \alpha^{(t)}\mathsf{grad}\\
\beta_l^{(t)} &=& \beta_l^{(t-1)} + \alpha^{(t)}\bar{f}_k^{(0)}\mathsf{grad}\\
\end{eqnarray*}

$\alpha^{(t)}$ is the learning rate. DGD is not defined if non-empty parameter sub-vectors arise outside of FC-bias units.

We give the results of training with our study A protocol where SGD was swapped for DGD and output debiasing was used in figure <ref>. Individual graphs are equivalent to figure <ref>. For architectures with high LBIAS, we obtain similar results compared to first-layer-only training. However, for architectures with low LBIAS, we now very rarely see a large increase in error, as desired.

Why does DGD work? While investigating this point in detail goes beyond the scope of this work, we speculate that the negative impact of neuron bias arises because the gradient $f_k^Tg_l$ is dominated by the expectation of $f_k$ if it has greater magnitude than the variation of $f_k$. However, the expectation is the same for each input, and hence the learning itself is not responsive to the value of the input. In the DGD update, the term $(f_k^{(t)}-\bar{f}_k^{(0)})^T$ does differ significantly from input to input, at least in the initial state.

The effectiveness of DGD reveals a fundamental difference between NLC and $LBIAS_l$. While nonlinearity is an indicator of fundamental compatibility (or lack thereof) between the true input-label function and the network function, neuron bias is more of a nuisance factor that can be greatly mitigated by proper conditioning of the loss function, error function and training algorithm. In this sense, `avoid neuron bias' is similar to the `ensure scale stability' guideline, as we show in section <ref>. Of course, there is a caveat in that we cannot show that architectures with high NLC can never achieve high performance under any reasonable training protocol. However, it seems that such a protocol would have to at least be capable of drastically altering the network function. If this is the case, it might just be easier to choose an architecture with a good NLC to begin with.

Summary While DGD and output debiasing can mitigate the effects of large $LBIAS_l$, we still conclude that avoiding excessive $LBIAS_l$ values is desirable, at least in the context of current designs. Another reason for this is the relationship between neuron bias and pseudo-linearity discussed in section <ref>. However, more work is necessary to fully understand the nature of the pathologies involved. We formalize the ZSAD guideline below.

`Avoid neuron bias' requires that absolute neuron expectations are not excessively large. Specifically, we advocate measuring this via $LBIAS_l \approx 0$ at each layer $f_l$, where $LBIAS_l$ is the layer bias as defined above. We especially recommend ensuring $LBIAS_L \approx 0$.

This guideline contributes to the nonlinearity normalization algorithm in chapter <ref>.

Computing $LBIAS_l$ Since $LBIAS_l$ and $NLC_{l,0}$ have the same denominator, our discussion of the computation of the NLC denominator in section <ref> applies here as well. With a good implementation, the computation does not suffer from catastrophic rounding error unless the rounding error induced by the network evaluation exceeds the (mathematical) difference of outputs. Among our study A architectures, there were four architectures for which this happened. The computed LBIAS value was around $10^{15}$, which is around 1 over the spacing of the floating-point grid. While the true LBIAS values would have been somewhat larger, we decided that the computed value was an acceptable proxy and used it for plotting results in e.g. the figures of this section. Computing the $LBIAS_l$ numerator is trivial compared to the denominator.

In light of this section, we can say that architectures with initial $LBIAS_l$ values large enough to cause computational issues are likely to be suboptimal from a performance standpoint.

Initial LBIAS vs test and training error for study A and B architectures. Inset graphs in the bottom right are magnifications of the region $0.8 < LBIAS < 100$. Conclusion: The LBIAS of an architecture, when evaluated in the initial state before training, is a powerful predictor of test and training error after training and attaining a small LBIAS value is essential for attaining an optimal test or training error.

Final LBIAS vs test and training error for study A and B architectures. Conclusion: All architectures that attain a better-than-random test or training error also attain a near-1 LBIAS after training.

LBIAS vs the maximum $LBIAS_l$ value across all layers for study A and B architectures in the initial state. Correlations are close to 1. Conclusion: The two values are functionally equivalent for our purposes.

LBIAS vs the maximum $LBIAS_l$ value across all layers for study A and B architectures in the final state. Conclusion: While LBIAS is always close to 1 for architectures that achieve a better-than-random error, we observe $\max_l LBIAS_l$ values up to around 50.

LBIAS vs error for study A waveform-noise architectures trained with output debiasing and first-layer-only training. In graphs C and F, we plot LBIAS vs the error difference between the modified training protocol and training with the original study A protocol. The “original” error values are the same as in figure <ref>. Conclusion: Output debiasing and first-layer-only training enable training and generalization for architectures with high initial LBIAS values and do not force those architectures to attain low LBIAS values in the final state.

LBIAS vs error for study A waveform-noise architectures trained with output debiasing and DGD. Graphs are equivalent to figure <ref>. Conclusion: Output debiasing and DGD enable training and generalization for architectures with high initial LBIAS values and do not force those architectures to attain low LBIAS values in the final state. Additionally, they do not greatly harm the performance of unbiased architectures.

§ ENSURE NOISE STABILITY, AND SPECIFICALLY SUFFICIENT FLOATING-POINT PRECISION

Many deep learning building blocks purposefully make the prediction noisy. In the case of batch normalization, the prediction for an individual input depends on the other inputs with which it is forward-propagated. In the case of dropout, it depends on which neuron values are dropped. In the case of data augmentation, it depends on choices made by the data augmentation function. Some activation functions are explicitly noisy [Gulcehre et al., 2016]. All these choices are random and thus induce noise. While it is not the topic of this work to discuss the potential benefits of noise, we investigate one of the pitfalls in this section. In order to obtain a reliable prediction, any noise must not lead the output to vary too drastically. We can imagine that excessive noise caused by e.g. a high dropout rate or a small batch size would adversely impact performance. In this section, we confirm this hypothesis empirically for two examples: floating-point rounding error and batch normalization. Then we formulate the `ensure noise stability' guideline.

In section <ref>, we showed how the NLC is related to the degree to which random noise added to the network input affects the network output. In a nutshell, we found the NLC is proportional to the magnitude of the induced output noise relative to the diameter of the codomain, when adding a certain magnitude of input noise relative to the diameter of the domain. In that section, we empirically investigated the impact of artificial noise added to the input. In this section, we show that the impact of “natural” sources of noise is also modulated by the NLC, though not necessarily as straightforwardly.

Ensure that floating-point precision is sufficient Evaluating the mathematical functions represented by e.g. the network or the training algorithm using floating-point computation introduces error. This error is not technically random, as floating-point computation is generally implemented as a deterministic system. In practice, this error is often viewed as random noise, as the exact position of the floating-point grid relative to mathematical function values is viewed as random. This assumption is very practically reasonable for our purposes, as well as most other purposes. Note that floating-point rounding error is of prime importance in quantized, low-precision networks (e.g. Stock et al., 2020, Sun et al., 2020, Lin et al., 2020).

In the previous section, as well as section <ref>, we encountered the importance of floating-point precision for obtaining accurate NLC and LBIAS values. We begin our empirical investigation with this issue. In figure <ref>A, we plot the value of the NLC computed in the initial state when using our regular 64-bit setup vs the value computed using effectively the same program under 32-bit precision for our study A waveform-noise architectures. We find that for a large majority of architectures, the value is effectively unchanged. For some GUAs, the value changes somewhat but not drastically. This is explained by the instability of the estimator and has nothing directly to do with the change of precision. However, for some architectures, the NLC value does change drastically. These architectures have $LBIAS \gtrapprox 10^7$. For those architectures, the NLC denominator takes very small values. Under 32-bit, that value is effectively rounded up to the floating-point grid spacing, which leads to an overestimate of the denominator and thus to an underestimate of the NLC. For two waveform-noise architectures, which are in the lower left quadrant of figure <ref>A, even 64-bit precision is not enough for computing the NLC accurately, as mentioned in section <ref>.

Now let's look at architecture performance. We re-trained and re-evaluated our waveform-noise architectures using our study A protocol, except that all computation was conducted with 32-bit floating-point precision. The results are given in figure <ref>B. In figure <ref>C, we give the difference between the test error values obtained under 32- and 64-bit. We find that changing the precision does not have a significant impact on test error overall, though the test error of individual architectures does change to a small degree. These kinds of small changes are due to the sensitivity of the architecture's final state to minuscule levels of noise under gradient-based training [Metz et al., 2019], and are not inherently related to the change in precision. See also section <ref>.

In figure <ref>A and <ref>B, we give the results obtained from training error minimization under 32-bit. Here, there are three architectures where the training error increases massively relative to 64-bit. Specifically, we observe that these architectures all have an initial NLC larger than $10^7$. There is only one other architecture with such a large NLC that is still trainable under 32-bit, and it is exactly the architecture that changes its NLC significantly after the first gradient update, as described in section <ref>. Hence, as long as the NLC remains larger than around $10^7$, training fails under 32-bit precision.

This suggests that rounding error indeed behaves somewhat like uniformly random noise. In section <ref>, we showed that noise of relative magnitude around $\frac{1}{NLC}$ was needed to increase error. If we view the rounding error as noise of relative magnitude equal to the grid spacing, we would expect architectures to become untrainable when the NLC exceeds 1 over the grid spacing. This is exactly what we observe. Note that there can be other factors that create a need for high floating-point precision other than the NLC. If a network has, say, an intermediate layer that is extremely biased so that the first several significant digits are constant across inputs, the need for floating-point precision might be significantly higher than $\frac{1}{NLC}$.

For our study B architectures, unfortunately, we had to conduct all computation in 32-bit precision and were unable to try out 64-bit precision. Since some of our architectures had an initial NLC around $10^7$, it is possible that our experimental results were impacted, at least when it comes to evaluating the trainability of high-NLC architectures in e.g. figure <ref>.

Ensure sufficient batch size To investigate the impact of noise induced by batch selection under batch normalization, we re-trained our waveform-noise architectures with our study A protocol, except that we used a batch size of 3000, which corresponds to the entire training set. When computing validation and test error, we use batches of size 1000, which corresponds to the entire validation / test set. Therefore, the noise inherent in propagating the same input forward inside different batches is eliminated.

In figures <ref>E-F and <ref>C-D, we give the results of training and evaluating architectures in “fullbatch” mode with 64-bit precision. In figure <ref>H-I and <ref>E-F, we give the results of training and evaluating in fullbatch mode with 32-bit precision. We find that test error suffers for some architectures, and never improves significantly. Possible explanations for this are that larger batch sizes are inherently inferior, as is generally believed, or that training “overfits” to the single training batch and does not generalize as much to the test batch. For the training error, we obtain different results. A significant number of architectures with very high NLCs that were untrainable with small batches are trainable with large batches. The red color indicates that all those newly trainable architectures use BN. When restricted to 32-bit, fullbatch mode only reduces training error when the NLC is less than around $10^7$. The only exception to this, which is visible in the lower right quadrant of figure <ref>F, is an architecture that reduces its NLC massively after the first update, just as the architecture we discussed above. All of this is expected. Removing batch selection noise helps high-NLC architectures train, but introducing additional rounding error can negate that effect.

Based on these observations, we formulate the ZSAD guideline of noise stability.

`Ensure noise stability' requires that the random choices made when evaluating the network function do not induce output variation of a magnitude greater than the range of predictions that can be considered accurate for a given task. In this context, floating-point rounding error can also be viewed as noise.

We note that, while we did not observe a systematic increase in test error due to noise in this section, this is because our sources of noise were not large enough to impact architectures with an NLC small enough for generalization. If trainability were impacted for low-NLC architectures, then test error would be as well.

How much noise does BN induce? A rough estimate of the relative estimation error of the neuron mean and standard deviation from a sample of size $|B|$ is $\frac{1}{\sqrt{|B|}}$. If $|B| = 250$, then this value is around 0.06. However, in figure <ref>D, we find that the benefit of fullbatch training only kicks in for an NLC around 5000. Hence, noise from BN has a much lesser impact on trainability than the noise from floating-point rounding relative to its magnitude, and a much lesser impact than proposition <ref> would suggest. This fact is worth further study.

Finally, in figure <ref>D/G, we compare the “original” value of the NLC vs the NLC computed under fullbatch mode and 64-bit or 32-bit precision. We find that using full batches makes no difference to the NLC value for non-GUAs. This insight may confer flexibility when computing the NLC in practice in terms of the batch size used.

Computed metric values for study A waveform-noise architectures. In graphs A/B/C, we depict results from performing computation using 32-bit floating-point precision. In graphs D/E/F, we depict results from using the entire training / validation / test set as a single batch during training and metric computation. In graphs G/H/I, we depict results from using the entire training / validation / test set as a batch and 32-bit precision. In graphs A/D/G, we plot the initial NLC obtained from our original regime vs the initial NLC obtained in the modified regime. Note that these values are computed estimates and do not necessarily reflect the mathematical values of the NLC. GUAs are depicted in green in the foreground. In graphs B/E/H, we plot the initial NLC from the original regime vs the test error from the modified regime. In graphs C/F/I, we plot the initial NLC from the original regime vs the difference between the test error from the modified and original regime. Conclusion: Lowered precision compromises NLC computation for some architectures. Changing batch size from 250 to 3000 does not impact the NLC. The modified regimes do not yield drastically different test error values overall.

Metric values for study A waveform-noise architectures. In graphs A/B, we depict results from performing computation using 32-bit floating-point precision. In graphs C/D, we depict results from using the entire training / validation / test set as a single batch during training and error computation. In graphs E/F, we depict results from using the entire training / validation / test set as a batch and 32-bit precision. In graphs A/C/E, we plot the initial NLC obtained from our original regime vs the training error obtained in the modified regime after training error minimization. In graphs B/D/F, we plot the initial NLC from the original regime vs the difference between the training error from the modified and original regime. Red points correspond to architectures with BN. Conclusion: Lowered precision compromises trainability for high-NLC architectures. Using full batches enables trainability for high-NLC architectures with BN.

Count of the number of architectures for which the $n$'th element in the starting learning rate sequence was chosen, for study A and B architectures. In graphs A/B/C, the choice was based on validation error. In graph F, the choice was based on test error. In graphs D/E/F, the choice was based on training error for the purpose of training error minimization. Conclusion: We were largely successful in ensuring that, for each architecture, the best SLR considered lies within the wide valley of the SLR-to-error function.

§ PERFORM EXHAUSTIVE LEARNING RATE TUNING

This section differs from the previous sections of this chapter in that we do not discuss a specific guideline for zero-shot architecture design. Throughout this work, we investigate the question of how to build neural architectures to maximize performance. In order to speak meaningfully about the “performance of an architecture”, we need to ensure that our training protocol enables a fair comparison. As we explained in section <ref>, the most important aspect of this is to conduct an independent, exhaustive learning rate search for each architecture. In this section, we verify the importance of learning rate tuning for both scientific validity and practical performance maximization. Then, we go on to uncover patterns that might enable us to predict which learning rates work best for an architecture or task. This might then enable us to reduce the computational cost involved in learning rate tuning.

The SLR-to-error function has a characteristic shape, as depicted on page 425 of Goodfellow et al., 2016. There is almost always a single “wide valley” that has a relatively robust minimum which can be found by considering SLR grids of spacing factor around 3. This is the minimum we look for in this section. However, the SLR-to-error function also has an unknown number of valleys that can be extremely sharp, and can have minima far below the minimum of the wide valley. For example, if an SLR of 0.7465 induces an error of 0.103 and an SLR of 0.7466 induces an error of 0.104, an SLR of 0.74655 may induce an error of 0.07, for example. We term this the sharp valley problem, which we further discuss in section <ref>. Throughout this section and this work, we ignore the presence of these sharp valleys and focus on finding an SLR close to the minimum of the wide valley, as is done in practice.

Throughout this work, whenever we train an architecture, we select the best starting learning rate by conducting a large number of independent training runs. It is important to keep in mind that when we refer to the “best starting learning rate” in this work, we refer to the best SLR that is found via this procedure. The full list of starting learning rates chosen for each of our architectures for both validation / test error and training error minimization is given in the appendix in chapter <ref> along with key metrics.

The range of starting learning rates we considered contains the wide error valley for almost all architectures We selected the best SLR independently for each architecture by independently training with 40 different SLRs in study A and determining the best validation error. We re-trained study A architectures to minimize training error without early stopping based on validation error, where we used 60 different SLRs and selected based on best training error. In study B, we used 20 different SLRs and selected based on best test error or best training error for training error minimization. See sections <ref> and <ref> for details about the training protocol.

In study A, our SLRs formed a geometric sequence with spacing factor 3. The beginning of this sequence was different for each architecture. The reason behind this customization was to increase the chance that the range covered by the grid would contain the wide error valley and robust minimum described above. In a nutshell, the lowest SLR we considered for an architecture scales inversely with the magnitude of the parameter gradient. In figure <ref>A-C, we count the number of architectures for which the $n$'th member of the SLR sequence yielded the lowest validation error, for each $n$. We find that for almost all architectures, the best SLR lies somewhere in the middle of the sequence. Hence, the sequence positioning was successful. There were no architectures whose best SLR was either among the first 5 or last 5 elements of the sequence. We view this as confirmation that our ranges indeed cover the wide error valley. Note that in figure <ref> and figures <ref> through <ref>, architectures for which the error used to select SLR was not better than random for any SLR are not depicted in our graphs, as no meaningful best SLR could be chosen for them.

Initial NLC vs the smallest SLR considered for study A architectures. Conclusion: The smallest SLR decreases proportionally as the NLC increases.

In figure <ref>D/E, we depict how often each element of the sequence was chosen for training error minimization in study A. Again, most chosen SLRs lie in the middle of their sequence. However, the total range of sequence elements chosen is much larger compared to when we select based on validation error. This was the reason behind expanding our sequence to 60 elements. 40 elements would not have been enough to cover all architectures with the starting point heuristic we used. We will uncover the reason for this increased breadth later in this section. With 60 elements, we ensure that the first 5 or last 5 elements are never chosen, except for one architecture where the fourth largest element is chosen.

In study B, we considered 20 fixed starting learning rates. We were unable to adjust the sequence on a per-architecture basis, due to code base limitations as outlined in section <ref>. In figure <ref>F/G, we find that, while there is a buffer at the lower end of the sequence, the largest SLRs considered were chosen for some architectures. Hence, we cannot guarantee that considering even larger SLRs would not have increased performance further for some architectures. As we discussed in the previous section, the trainability of some high-NLC architectures might have been impacted by using only 32-bit floating-point precision. Under 64-bit precision, the range of optimal SLRs might have been larger. Because of this uncertainty, we also exclude results from training error minimization of study B architectures from figures <ref> through <ref>.

The smallest SLR scales inversely with the NLC In figure <ref>, we plot the initial NLC vs the least SLR considered, i.e. the smallest member of our SLR sequence. We find that there is an inversely proportional relationship. Across the architectures we consider, the NLC is roughly proportional to the magnitude of the parameter gradient, which in turn is inversely proportional to the lowest SLR we consider by construction. In general, the strength of the relationship between NLC and parameter gradient magnitude that is behind the trend of figure <ref> does depend on various architecture properties, which we do not investigate here. In general, we focus this work on the input and layer gradients rather than the parameter gradient. An analysis along the lines of section <ref>, chapter <ref> or Yang, 2020 can shed further light on the parameter gradient.

Initial NLC vs best SLR for study A and B architectures. Conclusion: The NLC is somewhat negatively related to the best SLR. However, only in the case of a large NLC and training error minimization is this relationship strong. In that case, the best SLR scales as the inverse square of the NLC.

As stated in section <ref>, the smallest SLR we consider for training error minimization of study A architectures is exactly $10^{-8}$ times the smallest SLR given in figure <ref>.

The best SLR has a wide range across architectures In figure <ref>A/B/C/F, we plot the initial NLC vs the best (=selected) SLR. We find that the range of best SLRs has width around $10^7$ in log space within each graph, and around $10^{10}$ across all graphs. In figure <ref>D/E, we plot the initial NLC vs the best SLR for training error minimization. Now, the range spans 30 orders of magnitude! This strongly underscores the need for exhaustive learning rate tuning in order to compare architectures fairly for the purpose of empirical study, and in order to maximize performance. Yet, very few studies conduct such tuning.

The NLC is not predictive of the best SLR when minimizing validation / test error but somewhat predictive when minimizing training error In figure <ref>A/B/C/F, we find that for any NLC level, the range of optimal SLRs has width around $10^4$ and $10^7$ in log space. This range seems to shift slightly as the NLC increases. Overall, the NLC is not much help in predicting the best SLR. In figure <ref>D/E, we find that there exists a small group of architectures with high NLCs for which the best SLR scales as the inverse square of the NLC. These are precisely the architectures that train but do not generalize, which we have investigated e.g. in the previous section. There are a few outliers to this trend. For waveform-noise, there is a single outlier with initial NLC around $10^{10}$. This is the same outlier we discussed before, which changes its NLC after the first update. For CIFAR10, there is an outlier with initial NLC around $10^7$ and one with initial NLC around $10^9$. These do not achieve a low final NLC. However, the magnitude of the parameter still changes drastically, which is confirmed by their PARMSHIFT values below.

Error vs best SLR for study A and B architectures. Conclusion: For CIFAR10 and MNIST, only a small range of SLRs can lead to close-to-optimal test error. For waveform-noise and training error, this range is much wider.

Only a narrow range of SLRs can lead to optimal test error for CIFAR10 and MNIST, but a wide range can lead to optimal error for waveform-noise In figure <ref>, we plot the test and training error achieved vs the best SLR. For CIFAR10 and MNIST, in order to attain a test error close to the optimal value for the dataset, we find that the SLR has to lie within a range of width around 10 in log space. Architectures whose best SLR lies outside that range do not attain a close-to-optimal test error. For waveform-noise, a much wider range of SLRs can lead to optimal or close-to-optimal test error. This is one of the few situations where we observe different behavior for our datasets. We hypothesize that waveform-noise is more forgiving because it is a smaller dataset and hence represents an easier task. Minimizing training error is also easier compared to minimizing test error.

The best parameter update size has a narrower range than the best SLR In addition to looking at the SLR directly, we can study it indirectly. As the learning rate increases, the update that is applied to the parameter during the first iteration also increases. In general, we would expect the magnitude of the sum of all updates to increase as well.

The `parameter shift' (PARMSHIFT) of a final parameter value $\theta^{(T)}$ relative to an initial parameter value $\theta^{(0)}$ is

$$PARMSHIFT(\theta^{(0)}, \theta^{(T)}) = \frac{||\theta^{(T)} - \theta^{(0)}||_2}{||\theta^{(0)}||_2}$$

We plot the initial NLC as well as error vs PARMSHIFT obtained from the best SLR for study A architectures in figures <ref> and <ref>. We find results similar to figures <ref> and <ref>. Again, for CIFAR10 and MNIST, only a narrow range of PARMSHIFT values can lead to close-to-optimal test error. Compared to SLR, the range of best PARMSHIFT values across architectures is narrower for at least, say, the top 50% of architectures. In figure <ref>D/E, we now find that when minimizing training error of our high-NLC architectures, PARMSHIFT scales inversely with the NLC. This can be interpreted in light of our earlier findings. In figure <ref>, we found that the NLC is roughly proportional to the parameter gradient magnitude. In figure <ref>, we found that the best SLR for training error minimization scales as the inverse square of the NLC. We might expect PARMSHIFT to be roughly proportional to the product of parameter gradient magnitude and SLR, thus inversely proportional to the NLC.

The best output update size lies in a very narrow range and is highly related to test error Finally, in addition to investigating the magnitude of the change to the parameter, we investigate the magnitude of the change to the network output. For the same reason as for PARMSHIFT, we expect it to be closely related to the SLR.

The `output shift' (OUTSHIFT) of an architecture $f$ during the first update is

$$OUTSHIFT(f,\mathcal{D}, \theta^{(0)}, \theta^{(1)}) = \sqrt{\frac{\mathbb{E}_x||f(\theta^{(0)},x) - f(\theta^{(1)},x)||_2^2}{\mathbb{E}_x||f(\theta^{(0)},x)||_2^2}}$$

Here, we consider specifically the change of the output during the first update, as we suspect that updates during consecutive iterations will cancel each other out to a significant degree. In figures <ref> and <ref>, we repeat figures <ref> and <ref> with the OUTSHIFT instead of PARMSHIFT metric, which is evaluated on the test set. The results are highly noteworthy. In graphs A/B/C, the majority of architectures have an OUTSHIFT value extremely close to a single fixed value, which turns out to be $\sqrt{2}$ for reasons described below. Even architectures that deviate from this value are generally not more than a factor of 10 away. The range of OUTSHIFT values is much narrower than the range of SLR values in figure <ref>. Further, in figure <ref>, we find that architectures with close-to-optimal test error all have $OUTSHIFT \approx \sqrt{2}$, even for waveform-noise. Note that the range covered by the y-axis in figures <ref> and <ref> is much smaller compared to that of figures <ref> and <ref>.

$OUTSHIFT \approx \sqrt{2}$ arises when (i) $\mathbb{E}_x||f(\theta^{(0)},x)||_2^2 \approx \mathbb{E}_x||f(\theta^{(1)},x)||_2^2$ and (ii) $f(\theta^{(0)},x)$ is approximately independent of $f(\theta^{(1)},x)$. In other words, we need the first update to reset the outputs completely without increasing their magnitude significantly. If the architecture uses neither BN nor LN, only a very small number of SLRs achieve this exactly. If BN or LN is used, all SLRs above a certain level achieve this, because OUTSHIFT is bounded above by the fact that $\mathbb{E}_x||f(\theta^{(0)},x)||_2^2 = \mathbb{E}_x||f(\theta^{(1)},x)||_2^2 = 1$ holds.

The OUTSHIFT requirement is reminiscent of line search. Because of their large capacity, neural networks are often able to move all outputs for the first batch close to the label in the first update. An “output reset” would correspond to doing this and therefore to minimizing loss on the batch.

In figure <ref>D/E, we find that OUTSHIFT does not decrease significantly for large NLC's under training error minimization, at least relative to SLR and PARMSHIFT. Again, there is a similar interpretation. If the parameter gradient magnitude is proportional to the NLC and the SLR scales with the inverse square of the NLC, then the magnitude of the first parameter update scales with the inverse of the NLC. Finally, OUTSHIFT is roughly the product of update size and parameter gradient magnitude.

Learning rate prediction: discussion We have presented some intriguing relationships between best SLR and other metrics. These findings are preliminary and more work is needed to develop fleshed-out learning rate selection strategies. Since we are not deeply familiar with the literature on learning rate selection, we are also not able to place these results in a wider context. However, given the breadth of our studies, we do believe there is value in mining them for insights about learning rate. We note that tying parameter to update length as in PARMSHIFT was recently done by Bernstein et al., 2020 and line search was recently used for deep networks by Mutschler and Zell, 2020.

While we found it difficult to predict the best SLR, or its surrogates PARMSHIFT and OUTSHIFT, for a given architecture, we found that the architectures that achieve close-to-optimal test error for a dataset have predictable behavior. The most significant pattern is $OUTSHIFT \approx \sqrt{2}$. This might enable us to greatly reduce our learning rate search while only attaining suboptimal error values for architectures that are suboptimal to begin with. This might be sufficient in some practical situations.

Finally, we note that using early stopping was crucial in reining in the computational expense.

Initial NLC vs PARMSHIFT corresponding to the best SLR, for study A architectures. Conclusion: PARMSHIFT values lie in a narrower range than SLR values. In the case of a large NLC and training error minimization, PARMSHIFT scales inversely with NLC.

Error vs PARMSHIFT corresponding to the best SLR, for study A architectures. Conclusion: PARMSHIFT values lie in a narrower range than SLR values. For CIFAR10 and MNIST, only a small range of PARMSHIFTs can lead to close-to-optimal test error.

Initial NLC vs OUTSHIFT corresponding to the best SLR, for study A architectures. Conclusion: OUTSHIFT values have a very narrow range relative to PARMSHIFT and SLR, especially for waveform-noise. When considering test error, most architectures have $OUTSHIFT \approx \sqrt{2}$.

Error vs OUTSHIFT corresponding to the best SLR, for study A architectures. Conclusion: OUTSHIFT values have a very narrow range relative to PARMSHIFT and SLR, especially for waveform-noise. All architectures that attain close-to-optimal test error have $OUTSHIFT \approx \sqrt{2}$. This is also necessary for close-to-optimal test error.

§ SUMMARY: EXPLAINING THE PERFORMANCE OF OUR ARCHITECTURES

In this chapter, we have introduced several novel ZSAD guidelines, and have analyzed them as well as previously existing ZSAD guidelines. We are now in a position to explain, to a large degree, the performance of our architectures in terms of these guidelines. In chapter <ref>, we further explain these guidelines in terms of the architecture definition itself.

In figure <ref>, we plot the initial NLC vs test and training error for our study A and B architectures. Graphs A/B/C/D/E/G/H are equivalent to the graphs in figure <ref>, except for colors. GUAs are displayed in green. Architectures with an initial LBIAS greater than 10 are displayed in red. In graphs D and H, architectures with an initial NLC greater than 1000 that use BN or data augmentation and are not GUAs are depicted in blue. Data augmentation is the third source of noise that occurs in study B (but not in study A), next to noise from BN and floating-point computation. While we did not address it in section <ref>, our data augmentation induces noise of significant magnitude via cropping and especially horizontal flipping (sections <ref>, <ref>).

In graphs A through C, we find that almost all low-NLC study A architectures that do not generalize are either GUAs or suffer from neuron bias. The relationship between the NLC and test error among the remaining architectures, depicted in grey, is very strong. In graph G, again, we find that among study B architectures with NLC less than around 1000, architectures that have the highest errors are GUAs or suffer from neuron bias.

In graphs D, E and H, we find that all untrainable architectures are either GUAs or suffer from neuron bias or noise instability. Unfortunately, study B did not end up containing architectures with an initial NLC greater than 3000 that did not also suffer from noise or Gaussian instability. Hence, our study did not verify that high-NLC convolutional architectures are trainable. (Note that we also would have been limited by 32-bit floating-point precision.) There are three CIFAR10 architectures with very high NLC, depicted in grey in graph D, which have elevated, though not random, training error. Two of them have $NLC > 10^{15}$. Hence, they likely suffer from noise instability induced by 64-bit floating-point precision. An even higher level of precision would have to be used to determine whether these architectures are untrainable in a mathematical sense. This leaves only a single high-NLC architecture with elevated training error, which still has training error less than 0.4. Hence, we consistently show that ultra-high complexity architectures are trainable. To our knowledge, we are the first to explicitly demonstrate this. For example, Schoenholz et al., 2017 and Xiao et al., 2018 previously argued this was impossible. We uncover that the requirements are a well-tuned small learning rate, noise stability and an absence of neuron bias. The latter requirement can be circumvented by e.g. debiased gradient descent. We further discuss this point in section <ref>.

In graphs G and H, we find that some architectures with NLC close to 1 attain a high, though not random, training and test error. This indicates underfitting as discussed in section <ref>. Some of our study B architectures are very close to linear functions due to the construction of our study B activation functions.

In graph F, we give results from conducting training and error computation using the entire training set as a single batch, as done in section <ref>. The graph is equivalent to figure <ref>C except for color. By eliminating the noise induced by small-batch BN, all architectures that are not GUAs and do not suffer from neuron bias attain better-than-random training error and all such architectures with $NLC > 10^5$ attain a training error close to zero.

Overall, an all-or-nothing pattern emerges. Even if there is a single ZSAD guideline that an architecture does not follow, the architecture is likely to attain suboptimal or even random performance. For low training error, we should avoid neuron bias and ensure Gaussian and noise stability. For low test error, we should also ensure $1 \le NLC \le 5$, or a different optimal NLC when the dataset is not neural regular. Scale stability was controlled in our empirical studies. Training stability and exhaustive learning rate tuning are prerequisites for our results in figure <ref>. Finally, we refer the reader again to the full list of architectures in the appendix in chapter <ref>, where it is possible to compare individual architecture definitions with metric values and properties.

This completes the explanation of the performance of our architectures. However, there is a larger lesson. The design of our empirical studies was essential for the discoveries we made. Without including the square and odd square activation functions, we would not have discovered Gaussian instability. Without studying very deep networks and activation functions with high $\mathfrak{n}_\tau$ values, we would not have observed such a wide range of NLC values. Without observing such a wide range of NLC values, we would not have discovered e.g. noise stability. Without conducting 64-bit precision computation, we would not be able to study high-NLC architectures in the same way. Without studying activation functions that are biased, i.e. that have $\mathbb{E}_{s\sim \mathcal{N}(0,1)}\tau(s) \neq 0$, we would not have been able to study neuron bias. Our ultimate results are dependent on choices made in study design. We were largely unaware of the outcomes of these choices when they were made. Of course, this obliviousness is essential for making scientifically and statistically sound discoveries. On the other hand, this means that our ZSAD guidelines are at least somewhat incidental and may not represent “the most important” guidelines, period. We also discuss this point in section <ref>. We would argue that any guideline that is of paramount importance in empirical studies as broad and careful as our studies, and that is explained in terms of general principles, is likely to have significance beyond a specific set of experiments.

We did make some deliberate choices when designing our studies in order to exclude some factors that are known to impact performance. The variance of weight tensor entries in the random initialization schemes we used never deviated significantly from the LeCun variance in order to ensure scale stability. For study A, we augmented the softmax+cross-entropy loss function to remove the influence of output magnitude on performance (section <ref>). In many architectures, we used activation function debiasing. We ensured that the parameter dimensionality was approximately constant across architectures within each study. Without these restrictions, the signals we found in this chapter might have been less clear. We believe that one of the key insights of this work is that, in order to make progress in understanding deep learning, we must control phenomena we already understand. There exist too many possible architectures to spend significant effort on known pathologies.

Initial NLC vs test and training error for study A and B architectures. In graph F, we plot initial NLC vs training error when the entire training set is used as a single batch for all computations. Architectures depicted in green are GUAs. Architectures depicted in red have $LBIAS > 10$. Architectures depicted in blue have $NLC > 1000$, use BN or data augmentation, and are not GUAs. Other architectures are depicted in grey. All colored markers are displayed in the foreground relative to grey markers. We verified that no grey markers in distinctive positions were occluded. Conclusion: Gaussian instability, neuron bias, noise instability and the NLC explain the majority of performance variation.

CHAPTER: NONLINEARITY NORMALIZATION (NLNORM)

Throughout this work, we have explored the importance of attaining a right-sized NLC, especially in the initial state. In chapter <ref>, we showed how to predict the NLC from the architecture definition. However, so far, all we can do with an architecture that has a suboptimal initial NLC is to discard it. In this chapter, we show how to control the NLC of an architecture by modifying the activation functions used, effectively turning the NLC into a tunable hyperparameter. This corresponds to utility criterion <ref>. Control is achieved via the simple nonlinearity normalization algorithm. Our analysis shows that the NLC can be considered not just predictive of performance, but causal for performance. Nonlinearity normalization can also eliminate neuron bias and scale instability. We find that applying nonlinearity normalization to an architecture often induces massive performance gains if that architecture's initial NLC or LBIAS is suboptimal.

We present evidence that nlnorm may significantly reduce the need for a range of building blocks like skip connections (section <ref>), normalization layers (section <ref>) or specific activation functions (section <ref>). While these building blocks have a large impact on performance when nonlinearity is not considered, this impact can significantly diminish when nlnorm is employed. This means that tuning the NLC when comparing deep learning pipelines and building blocks is essential to obtain a fair comparison, especially given that no single NLC value leads to optimal or even adequate performance for every type of architecture (section <ref>).

Background from prior chapters Like prior chapters, this chapter is based on the empirical studies detailed in chapter <ref>, specifically study B (section <ref>). We recommend reading at least summary section <ref> before proceeding. Throughout this chapter, we use the terminology, notation and conventions of section <ref>. We use results and definitions from sections <ref>, <ref>, <ref> and <ref> to motivate nonlinearity normalization. While we reference additional prior sections when relevant for discussion, a complete understanding of chapters <ref> through <ref> is not required.

Technical considerations As discussed in section <ref>, we implicitly assume things like integrability (section <ref>), differentiability (section <ref>) and non-zero denominators when necessary.

Limitations We designed study B specifically to validate nlnorm. However, we did not design study A in this way, as it was conducted before we understood nonlinearity. Hence, the empirical analysis of this chapter is restricted to convolutional architectures.

§ DEFINITION AND DISCUSSION

The idea of `nonlinearity normalization' (nlnorm) is based on the nonlinearity path equation from section <ref>. One of its implications is that when an activation function $\tau_l^{(1)}$ is replaced in an architecture with another activation function $\tau_l^{(2)}$ such that $\mathfrak{n}_{\tau_l^{(2)}}(\mathfrak{q}_k,\mathfrak{c}_k) < \mathfrak{n}_{\tau_l^{(1)}}(\mathfrak{q}_k,\mathfrak{c}_k)$, then the mean field NLC of the architecture decreases, as long as there are no knock-on effects on downstream $\mathfrak{q}$ and $\mathfrak{c}$ values. In plain words, the NPE suggests that we can control the nonlinearity of the network by controlling the nonlinearity of activation functions.

Assume for now that all activation layers in an architecture use the same activation function $\tau(s)$. nlnorm modifies this `base architecture' by replacing all instances of $\tau(s)$ with $c\ddot{\tau}(l,s) + b$, where $l$, $c$ and $b$ are parameters. (In the context of nlnorm, $l$ does not refer to a layer index.) The values of those parameters are shared between all occurrences of the activation function. We choose the three values to jointly achieve three objectives: right-sized NLC, low neuron bias, and scale stability. We begin by choosing the function $\ddot{\tau}(l,s)$ such that we obtain a range of $\mathfrak{n}_{\ddot{\tau}(l,s)}$ values, specifically $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ values, as the `linearization parameter' $l$ varies. Note that we have $\mathfrak{n}_{\ddot{\tau}(l,s)} = \mathfrak{n}_{c\ddot{\tau}(l,s) + b}$ for all $c \neq 0$ and $b$. For a given value of $l$, we jointly set $b$ and $c$ to attain $\mathbb{E}_{s \sim \mathcal{N}(0,1)}c\ddot{\tau}(l,s)+b=0$ and $\mathbb{E}_{s \sim \mathcal{N}(0,1)}(c\ddot{\tau}(l,s)+b)^2=1$. Hence, we ensure that $\mathfrak{C}_{c\ddot{\tau}(l,s) + b}(1,1) = 1$ and $\mathfrak{C}_{c\ddot{\tau}(l,s) + b}(1,0) = 0$. So, in the context of table <ref>, if $(\mathfrak{q}_k, \mathfrak{c}_k) = (1,0)$ holds for the dependency of an activation layer, $(\mathfrak{q}_l, \mathfrak{c}_l) = (1,0)$ holds for the activation layer itself. The idea here is to ensure $(\mathfrak{q}_l, \mathfrak{c}_l) = (1,0)$ throughout the network. If this “state of normalization” is preserved by activation layers as well as other layers by e.g. using LeCun initialization, then it holds at all layers and specifically at the output. If mean field theory is predictive, this ensures both near-zero neuron bias and scale stability. By ensuring $(\mathfrak{q}_k, \mathfrak{c}_k) = (1,0)$ for activation layers, we also ensure that the nonlinearity path equation is made up of $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ terms. Therefore, we can control the mean field NLC and, ultimately, the NLC of the architecture with $l$. The NLC indirectly becomes a tunable hyperparameter via $l$.  If mean field theory is predictive, values of $l$ that induce a high $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ value correspond to a large architecture NLC. Values of $l$ that induce a small $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ value correspond to a small architecture NLC. The last step is now to choose the desired NLC by choosing $l$. We summarize nlnorm for architectures with a single activation function in figure <ref>.

The nonlinearity normalization algorithm
for architectures with a single activation function.

* If the single activation function is $\tau(s)$, choose a parametrized version $\ddot{\tau}(l,s)$ such that varying $l$ yields a range of values for the activation function NLC (section <ref>):

$$\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0) = \sqrt{\frac{\mathbb{E}_{s\sim \mathcal{N}(0,1)}\tau'(s)^2}{\mathbb{E}_{s\sim \mathcal{N}(0,1)}\tau(s)^2 - (\mathbb{E}_{s\sim \mathcal{N}(0,1)}\tau(s))^2}}$$

See figure <ref> for examples, and section <ref> for an analysis of the universal choice $\ddot{\tau}(l,s) = \tau(s) + ls$.

* Replace each occurrence of $\tau(s)$ with $c\ddot{\tau}(l,s) + b$, where $c$ and $b$ are chosen so that $\mathbb{E}_{s\sim\mathcal{N}(0,1)}c\ddot{\tau}(l,s)+b=0$ and $\mathbb{E}_{s\sim\mathcal{N}(0,1)}(c\ddot{\tau}(l,s)+b)^2=1$ and $l$ is a free parameter.

* Choose a value of $l$ to obtain the desired network NLC in the initial state.

The nonlinearity normalization algorithm for architectures with a single activation function.

The simple form of nlnorm given in figure <ref> may not be sufficient for a given base architecture. A base architecture may not use only a single activation function. Weight tensors may not be LeCun initialized, which may lead to scale instability if the activation functions do not compensate. Layer operations that we have not discussed in this work may be present, which may induce different kinds of nonlinearity or partially invalidate table <ref>. With the understanding of the material presented in this work, especially figure <ref>, it should be somewhat straightforward to adapt nlnorm to more complex situations. For example, if an architecture contains multiple activation functions, we can simply choose a linearization method for each of them and then either choose a single $l$ value for all activation layers or separate $l$ values. Alternatively, we may choose not to linearize activation layers that play a “gating” role as they occur in e.g. LSTM [Hochreiter and Schmidhuber, 1997] or GRU [Cho et al., 2014]. Klambauer et al., 2017, Mishking and Matas, 2016, Arpit et al., 2016 also provide examples of how to conduct “normalization by recursion”, i.e. normalizing layers closer to the output based on assumptions derived from the normalization of layers closer to the input. For this work, the form in figure <ref> is sufficient. Providing explicit, well-defined generalizations is future work.

While $\mathfrak{n}_{c\ddot{\tau}(l,s) + b}$ is invariant to $b$ and $c$, replacing $\ddot{\tau}(l,s)$ with $c\ddot{\tau}(l,s)+b$ can and often does change the mean field NLC. If the replacement changes any $\mathfrak{q}$ or $\mathfrak{c}$ value, there can be knock-on effects throughout the architecture. For example, from our discussion in section <ref> we can see that debiasing an activation function can cause the mean field NLC to diverge exponentially from one macro-layer to the next when it was originally converging. In general, we would expect activation function debiasing not to decrease the mean field NLC, because $\mathfrak{n}_{\tau}(c)$ is a decreasing function of $c$, as discovered in section <ref>. We also find this to be true empirically in section <ref>.

As usual, and as described in section <ref>, we can compute the Gaussian expectations involved in the calculation of $b$ and $c$ accurately using standard numerical integration techniques.

§.§ Linearization methods

We term a method for converting an activation function $\tau(s)$ into a parametrized version $\ddot{\tau}(l,s)$ a `linearization method'. $\ddot{\tau}(l,s)$ itself becomes an (activation function, linearization method) pair or `AFLM'. We introduced these terms in section <ref>. See figure <ref> for examples of AFLMs used in this work. The simplest linearization method, which features most prominently, is `linear interpolation'.

$$\tau(s) \rightarrow \tau(s) + ls$$

What values of $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ can be obtained from this? Consider the theory of section <ref>. Let $a_\tau s + b_\tau$ be the least squares linear fit to $\tau$ under unit Gaussian input and let $\tilde{\tau} = \tau - a_\tau s - b_\tau$ be the nonlinear component of $\tau$. Then

$$\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0) = \sqrt{\frac{\mathfrak{n}_{\tilde{\tau}}(1,0)^2\mathbb{E}_{s\sim\mathcal{N}(1,0)}\tilde{\tau}(s)^2 + (a_\tau + l)^2}{\mathbb{E}_{s\sim\mathcal{N}(1,0)}\tilde{\tau}(s)^2 + (a_\tau + l)^2}}$$

As $l$ converges to infinity or minus infinity, $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ converges to 1. When $l = -a_\tau$, $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0) = \mathfrak{n}_{\tilde{\tau}}(1,0) \ge \sqrt{2}$ by theorem <ref> and proposition <ref>. Hence, as $l$ varies, $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ attains at least all values between 1 and $\sqrt{2}$. In table <ref>, we find that for popular activation functions $\tau$, $\mathfrak{n}_\tau(1,0)$ is significantly smaller than $\sqrt{2}$. So in practical situations, the desired $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ value should always be attainable by linear interpolation. For popular activation functions, we also have $a_\tau \ge 0$. Hence, setting $l \ge 0$ ensures that $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ varies monotonically between $\mathfrak{n}_\tau(1,0)$ and 1.

§ RELATED WORK

Mean and variance normalization is certainly a staple theme of deep learning. It is the foundation of normalization layers like BN and LN as well as methods that modify activation functions (e.g. normalization propagation [Arpit et al., 2016]), modify weight initialization (e.g. LeCun initialization, He initialization or LSUV [Mishking and Matas, 2016]) or normalize weights directly (e.g. weight normalization [Salimans and Kingma, 2016]), as well as of novel activation functions (e.g. SELU [Klambauer et al., 2017]). As mentioned above, several of these methods normalize layers recursively. The key innovation of nlnorm is that nonlinearity is normalized in addition to mean and variance. Of course, this is not as straightforward in the sense that we must also choose an $l$ value, which nlnorm deliberately leaves open. In section <ref>, we already found that there cannot be a universal best NLC across tasks. In section <ref>, we will show that there is also variation in terms of which NLC is best for a given base architecture. Predicting the exact best NLC based on the base architecture definition is future work.

In the context of mean field theory, several studies have controlled nonlinearity indirectly by controlling weight initialization (e.g. Pennington et al., 2017, Schoenholz et al., 2017, Xiao et al., 2018). Consider the $\mathfrak{n}_\tau(\lambda^2,0)$ curves from tables <ref> through <ref>. For many activation functions, the activation function NLC converges to 1 as $\lambda$ converges to zero. Specifically, this happens for activation functions that have a valid non-zero derivative at zero. If these activation functions are applied to small inputs, they mimic linear functions. The aforementioned studies generate small inputs by carefully reducing initial weight variance. This allows those studies to successfully train arbitrarily deep networks.

§ PROPERTIES OF NLNORM

In each subsection of this section, we establish a property of nlnorm empirically. This is similar to section <ref>, where we establish properties of the NLC. We will use study B for this. nlnorm is based on theory that we did not explicitly present for convolutional architectures. The fact that it still works for such architectures is an indicator of robustness.

The architecture design and training protocol for study B has been detailed in section <ref> and summarized in section <ref>. Study B contains a total of 552 architectures. Each of them uses a single activation function, which is based on one of 12 AFLMs given in table <ref>. In this work, we found that the square and odd square activation functions induce Gaussian instability, which is harmful to performance, as shown in section <ref>. Since nlnorm (unfortunately) does not eliminate Gaussian instability in general, we consider only architectures based on the other 10 AFLMs in this chapter, i.e. ReLU-interpolation through Gaussian. This leaves a total of 480 architectures. For each AFLM, $l$ has a default value that makes the AFLM revert to the basic activation function from table <ref>. Study B contains 12 different activation functions per AFLM, corresponding to 12 different $(l,c,b)$ triplets for the expression $c\ddot{\tau}(l,s) + b$. In the first triplet, $l$ is set to the default value, $b$ is set to zero and $c$ is set to achieve $\mathbb{E}_{s \in \mathcal{N}(0,1)}(c\ddot{\tau}(l_\text{default},s))^2 = 1$. This corresponds to ensuring scale stability only. We consider this to be the “base case” that an architecture without nlnorm would use. As in our design of study A, we treat scale stability as a pre-established ZSAD guideline that is respected even in “baseline architectures”. Hence, we consider architectures using this first triplet as our base architectures as defined above. Across the other 11 triplets, $l$ varies. Roughly, these 11 triplets correspond to 0% linearization, 10% linearization, etc. up to 100% linearization. $(b,c)$ are set jointly to achieve $\mathbb{E}_{s \in \mathcal{N}(0,1)}(c\ddot{\tau}(l,s) + b)^2 = 1$ and $\mathbb{E}_{s \in \mathcal{N}(0,1)}c\ddot{\tau}(l,s) + b = 0$. Hence, architectures using one of these 11 triplets use nlnorm and are thus termed `normed architectures'. For each AFLM we considered in this chapter, for each triplet, study B contains 4 architectures. One of them uses BN, one uses LN, one uses BN and skip connections, and one uses neither normalization layers nor skip connections. In this chapter, we refer to the last of the four specifically as a “vanilla architecture”. Each group of 12 architectures that share all properties except the $(l,c,b)$ triplet we term a `base group'. Finally, note that architectures vary in other properties, such as the presence of pooling or bias layers, between AFLMs. However, architectures that are based on the same AFLM do not vary except in the ways specified in this paragraph.

§.§ nlnorm controls the NLC

Comparing the initial NLCs of pairs of architectures. The initial NLC of one architecture is plotted on the x-axis and the initial NLC of the other architecture is plotted on the y-axis. Both architectures in a pair are from the same base group. Each graph contains one marker per base group. The labels denote which members of the base group are used for each axis. Conclusion: $l$ values that lead to larger / smaller $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ values generally lead to larger / smaller network NLCs in the initial state. $l_{10}$ induces an NLC close to 1. The base architecture NLC is approximately contained within the range of normed architecture NLCs.

A core requirement for nlnorm to work is that it is effective at controlling the NLC of practical, finite-width architectures. In figure <ref>, we verify this. As described above, we consider a total of 40 sequences of 11 normed architectures that differ only in the value of $l$ and the induced values of $c$ and $b$. For each sequence, let the 11 $l$ values be ordered so that $l_0$ is the $l$ value that induces the highest $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ value, and $l_{10}$ is the $l$ value that induces the lowest $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ value. Note that the actual values taken by $l_0$ through $l_{10}$ differ between AFLMs. See section <ref> for details.

For nlnorm to be effective, we require that for each base group, the architecture using $l_0$ has the highest NLC, the architecture using $l_1$ has the second highest, etc. The first 10 graphs of figure <ref> correspond to the 10 pairs $(l_0, l_1)$, $(l_1, l_2)$, .., $(l_9, l_{10})$. For each pair, we plot the initial NLC of the 40 architectures using the first $l$ value in the pair vs the NLC of the 40 architectures using the second $l$ value in the pair. In almost all cases, we find that the second architecture has a lower NLC than the first, as desired. We also find that architectures using $l_{10}$ have an NLC close to 1. This is also desired as we expect the ideal NLC to be somewhat close to 1 as well. We want the range of NLCs covered by nlnorm to contain the ideal NLC.

For nlnorm to be effective, we also require that the NLC of the base architecture is within the range of NLCs covered by the corresponding normed architectures. If this is not the case, then nlnorm may force us into an inferior NLC. In the bottom right of figure <ref>, we plot the initial NLCs of the 40 base architectures vs the initial NLCs of the corresponding normed architectures using $l_0$ and $l_{10}$ respectively. We find that the NLC of the base architecture is either between or very close to being between the other two NLCs, as desired. Each of our AFLMs allows us to choose $l$ values so that $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0)$ is either exactly or arbitrarily close to 1. With such an $l$ value, we can reduce the NLC relative to the base architecture. Note that we cannot always get the NLC exactly to 1, because some nonlinearity is contained in normalization and max pooling layers that arise in some architectures. Each of our AFLMs also allows us to choose the default $l$ value so that $\ddot{\tau}(l,s)$ reverts to the basic activation function from table <ref>. As mentioned above, we expect the introduction of debiasing not to decrease the NLC. Hence, by choosing $l_\text{default}$ within nlnorm, we expect an NLC approximately at least as large as the base NLC.

More generally, ensuring scale stability and low neuron bias in normed architectures causes the modeling of activation function inputs as unit Gaussians to be accurate, and therefore the simple version of nlnorm given in figure <ref> to be effective. We found that setting $b$ and $c$ always more or less ensures scale stability and low neuron bias in our architectures, with the exception of vanilla architectures based on abs. val. with a low degree of linearization. We found those architectures to also exhibit Gaussian instability, and they are considered GUAs in section <ref>. We also find in tables <ref> through <ref> that debiased abs. val. exhibits the highest degree of mean field Gaussian instability among activation functions that do not involve squaring, at least as soon as layer quadratic means grow slightly larger than 1.

§.§ nlnorm finds good nonlinearity levels

Initial NLC vs test error, broken down by base group. Blue markers correspond to vanilla architectures. Red markers correspond to BN architectures. Green markers correspond to LN architectures. Black markers correspond to BN-ResNet architectures. Squares correspond to base architectures. Lines arise from connecting points corresponding to the normed architectures within a base group with neighboring values of $l$. Conclusion: In the majority of cases, the lowest test error is reached at an intermediate NLC value, with the smallest and largest NLC values performing significantly worse.

nlnorm is based on the insight that an insufficient NLC leads to underfitting (see e.g. section <ref>) whereas an excessive NLC leads to overfitting (see e.g. section <ref>). We desire that the optimal tradeoff between both phenomena is obtained by an NLC that is within the range of NLCs obtained by varying $l$. In figure <ref>, we verify this.

We plot the initial NLC vs test error for our 480 architectures. The raw x- and y-axis values are identical to figure <ref>D, but the visual presentation is different. Each of the 10 graphs of figure <ref> depicts results for one AFLM, and each color in a graph corresponds to one base group. Blue corresponds to the vanilla base group, red to the BN base group, green to the LN base group and black to the BN-ResNet base group. The line is generated by connecting markers corresponding to normed architectures in the base group with neighboring values of $l$. The square represents the base architecture in the base group. Indeed, we find that for most base groups, the lowest test error is attained at a point that does not correspond to the smallest or largest NLC. Further, test error tends to increase in both directions when moving away from this optimal point.

There are some exceptions to this pattern. For the SELU AFLM, the optimal test error is reached with the largest NLC considered. We have $\ddot{\tau}_\text{SELU}(l,s) = \tau_\text{SELU}(s) + ls$. We only considered non-negative values of $l$. Hence, the largest $\mathfrak{n}_{\ddot{\tau}(l,s)}$ value we obtained was around 1.035 by table <ref>. A larger value might have been ideal, especially for SELU-BN-ResNet. This shows that it can be important to choose linearization methods that can make an activation function less linear, rather than only more linear. In contrast to SELU, we find that for softplus and abs. val., the least NLC considered attains the lowest or close to the lowest test error. This points to an important difference between linearization methods. Consider $\ddot{\tau}_\text{softplus}(l,s) = \frac{1}{l}\log(1 + e^{ls})$ and $\ddot{\tau}_\text{abs. val.}(l,s) = |s + l|$. When $l$ is small / large respectively, then $\mathfrak{n}_{\ddot{\tau}(l,s)}(1,0) \approx 1$. However, $\mathfrak{n}_{\ddot{\tau}(l,s)}(\lambda^2,0)$ can still be significantly larger than 1 for larger $q$. In fact, no matter the value of $l$, $\mathfrak{n}_{\ddot{\tau}(l,s)}(\lambda^2,0)$ converges to 1.21 for softplus and to 1.65 for abs. val. as $\lambda$ converges to infinity. Hence, growth of the parameter value during training, as studied in section <ref> and discussed in section <ref>, can cause the activation function NLC with respect to $\hat{\mathfrak{q}}_k$ to increase and hence counterbalance the effect of excessive nonlinearity normalization. This is not true to the same degree for activation functions linearized with linear interpolation.

In figure <ref>, we also find that for most base groups, the performance of the base architecture is close to the performance of normed architectures with comparable NLC. For three AFLMs, there is significant deviation: softplus, even tanh and Gaussian. For softplus and Gaussian, while the BN and BN-ResNet base architectures perform comparably to their normed counterparts, the vanilla and LN base architectures attain a random error of 0.9, and hence are not depicted in figure <ref>. These four base architectures are the ones that attain a very high initial LBIAS value, larger than $10^5$, as depicted in e.g. figure <ref>F/G. This demonstrates the importance of eliminating bias with nlnorm. For even tanh, similarly, the vanilla and LN base architectures have significant LBIAS. We have not investigated why the vanilla base architecture nevertheless outperforms normed architectures with comparable initial NLC. We note that the test error of that base architecture is still highly suboptimal overall. The even tanh-BN base architecture has an NLC greater $10^4$ and a test error greater than 0.5, and hence does not show up in the figure.

We end with two remarks. First, the non-smoothness of many curves in figure <ref> is most likely due to the fact that we only conducted a single training run per starting learning rate and architecture. If we had averaged over multiple runs corresponding to different random number sequences / initializations, we would expect results to be much less noisy. Our already high computational budget was not sufficient to do this for every architecture. Since searching for the best $l$ value and starting learning rate in a 2D grid is expensive, we expect that this will generally have to be done without multiple runs per grid point in practical situations. Hence, we were content to give results in figure <ref> that depict what one can expect from a single run. However, we would recommend that in practice, at least the same initial trainable parameter value is used for all values of $l$ and starting learning rates to reduce noise. We did not have the chance to do this. See section <ref>.

Second, we note that in figure <ref>, the lowest error value achieved differs significantly between AFLMs. Since architectures corresponding to different AFLMs differ in ways other than activation function, we cannot draw a conclusion about the relative strength of AFLMs “in a vacuum”.

§.§ nlnorm improves performance

Comparison of error, initial NLC and initial LBIAS between base and tuned architectures. All results are averaged over 10 re-training runs. Quantities depicted in log scale are averaged in log space. Conclusion: nlnorm drastically improves test error for many base architectures and never significantly increases test error. Larger NLC reduction corresponds to larger error reduction. Very large LBIAS reduction corresponds to very large error reduction.