A flipside of the theoretical power of the NTK is that it is not as predictive for practical, finite-width architectures trained with practical step sizes as e.g. background theorem <ref>. NTK theory predicts that in the infinite width limit, only the last linear layer learns. This phenomenon is termed `lazy training'. However, practical architectures often exceed the test error obtained from lazy training, even at very high width [Chizat et al., 2019, Goldblum et al., 2020, Liu et al., 2020, Novak et al., 2019, Lee et al., 2020]. While we investigate the NTK empirically in section <ref>, we do not further consider it in this chapter.

§ MEAN FIELD THEORY OF META-DISTRIBUTIONS

Throughout section <ref>, we studied the distribution of layer and network quantities where randomness was induced by the parameter $\theta$ and the surrogate input for the non-finite input layers generated by $\mathcal{G}$. In section <ref>, we studied layer means and co-means of mean field architectures. In section <ref>, we examined entire distributions of layer values. In section <ref>, we introduced a finite number of fixed inputs and examined the joint distribution of layer values across these inputs.

In this section, we extend mean field theory to cover not just distributions of layer values, but meta-distributions. Let $\mathcal{D}$ be an input distribution for a single finite input layer. For a fixed value of $\theta$ and surrogate input, $\mathcal{D}$ induces a distribution at each layer. If we now additionally allow $\theta$ and surrogate input to vary at random on top of this process, we obtain a `meta-distribution', i.e. a distribution over distributions over layer values. The first level of randomness stems from $x \sim \mathcal{D}$, and the second level of randomness stems from $\theta$ and surrogate input, which corresponds to the entire randomness from section <ref>. As given in section <ref>, we use the term `layer distribution' to refer to the distribution over layer values at some layer $f_l$ induced by $\mathcal{D}$ for a fixed $\theta$ and surrogate input, i.e. $f_l(\mathcal{D})$. We use e.g. `output distribution' and `neuron distribution' similarly.

There are at least three reasons why investigating the meta-distribution is interesting. First, we want to derive the limit of the NLC, which is defined in terms of $\mathcal{D}$ for a fixed $\theta$. Hence, we need to understand how e.g. $f(\mathcal{D})$ varies as a distribution as $\theta$ varies. Second, we want to investigate batch normalization when batch size converges to infinity. While BN with a fixed, finite batch size is covered by Yang, 2019, it is a rather cumbersome operation. When $|B| \rightarrow \infty$, we observe simple and nice behaviors for BN, similar to layer normalization when width converges to infinity. Third, it is instructive to understand the types of layer distributions we can expect for a fixed $\theta$. When we are dealing with a practical architecture in the initial state, we are dealing with a specific parameter value, not with parameter values that are drawn independently for each input. A practical layer distribution is a draw from the meta-distribution, and those draws can differ significantly from the distribution obtained when varying $x$ and $\theta$ jointly.

Our strategy for deriving the meta-distribution is to discretize $\mathcal{D}$ and then apply background corollary <ref>. Specifically, instead of considering $\mathcal{D}$ directly, we consider the uniform discrete distribution over an independent sample of size $N$ drawn from $\mathcal{D}$. A draw from the meta-distribution of any other layer is then also a uniform discrete distribution over $N$ fixed values. We then take the limit of that “discrete meta-distribution” as $d_\text{MF}$ converges to infinity. Finally, we “unify” these limits across all possible integer values of $N$. We formalize this process in a way that is intuitive, suitable for our analysis and similar to the notion of convergence in finite distribution used by Yang, 2019.

For any meta-distribution $\mathcal{M}$, let the `$N$-expansion' of $\mathcal{M}$ be the distribution over $N$ values that is equivalent to drawing a distribution from $\mathcal{M}$ and then drawing $N$ values from that distribution. Further, we say a sequence of meta-distributions $(\mathcal{M})$ `expansion-converges' to a meta-distribution $\mathcal{M}^\text{lim}$ if the sequence of $N$-expansions of the $\mathcal{M}$ converges in distribution to the $N$-expansion of $\mathcal{M}^\text{lim}$ for all $N\ge 1$.

For any $N$, $\mathcal{D}$ induces a distribution over $N \times N$ input co-mean kernel matrices $K_\text{in}$ when the $x^{(n)}$ are drawn from $\mathcal{D}$. In the setup of background corollary <ref>, as $d_\text{MF} \rightarrow \infty$, $K_\text{in}$ is mapped deterministically to the $N \times N$ output co-mean matrix $K_\text{out}$ via the covariance kernel function, which then generates the elementwise Gaussian output. So in the limit, we can say that $\mathcal{D}$ induces a distribution over $K_\text{out}$. To prove a theorem based on expansion-convergence, we need two things. We need (i) a $\mathcal{D}$ that induces a “nice and manageable” distribution over $K_\text{out}$ for each $N$ and (ii) a meta-distribution that yields elementwise $N$-expansions with generators that are (potentially infinite) mixtures of Gaussians with mean zero, where the covariance matrix is distributed as $K_\text{out}$ in (i). It turns out that what we call the `elem-like' distribution and the `meta-Gaussian' meta-distribution fulfill (i) and (ii) respectively. We now provide several definitions and then the main theorem of this section.

We say a meta-distribution $\mathsf{metadist}$ over distributions $\mathsf{dist}_\text{vec}$ over vectors of dimensionality $d$ is `elementwise' if there exists a meta-distribution $\mathsf{metadist}_\text{gen}$ over distributions $\mathsf{dist}_\text{sca}$ over scalars such that drawing a value of $\mathsf{dist}_\text{vec}$ from $\mathsf{metadist}$ is equivalent to drawing $d$ values of $\mathsf{dist}_\text{sca}$ from $\mathsf{metadist}_\text{gen}$ and then taking the product of those $d$ values. $\mathsf{metadist}_\text{gen}$ is called the `generator meta-distribution' or `generator' for short.

This mirrors the definition from section <ref>. It implies that any distribution drawn from an elementwise meta-distribution has independent components. It also implies that the distribution over vectors obtained by drawing from a draw from the meta-distribution is elementwise.

We say a meta-distribution over distributions over scalars is `meta-Gaussian' $\mathcal{MN}(q,c)$ with $q \ge c \ge 0$ if a distribution drawn from it is Gaussian with variance $q - c$ and mean drawn from another Gaussian with mean zero and variance $c$.

We say an input distribution $\mathcal{D}$ is `elem-like$(q,c)$' with $q \ge c \ge 0$ if $x,x'\sim \mathcal{D}$ implies $\mathbb{E}_i x[i]^2 = \mathbb{E}_i x'[i]^2 = q$ and $\mathbb{E}_i x[i]x'[i] = c$ with probability 1. We write $K_\text{in}(N,q,c)$ for the $N \times N$ matrix with diagonal entries equal to $q$ and off-diagonal entries equal to $c$, which is the co-mean kernel matrix for a sample of size $N$ drawn from an elem-like$(q,c)$ input distribution $\mathcal{D}$ with probability 1.

Let $f$ be a mean field architecture with a single finite input layer and a single readout layer. Let $\mathcal{G}$ be a generator distribution associated with the non-finite input layers of $f$. Let $\mathcal{D}$ be an input distribution associated with the finite input layer of $f$. Assume:

* $\mathcal{D}$ is elem-like$(q,c)$.

* $(\clipout(\clipin(f))^N, \mathcal{G}(K(N,q,c)) \times \mathcal{G}^N)$ has rank stability for all $N$.

* $\mathcal{G}$ is Gaussian.

* All multi-activation functions used in $f$ are parameter-controlled by $\mathcal{C}^\text{E2}$ at $(\mathfrak{m}_{\hat{k}[1]}, .., \mathfrak{m}_{\hat{k}[\hat{K}]})$.

Then as $d_\text{MF}$ converges to infinity, the meta-distribution of the readout layer expansion-converges to the elementwise meta-distribution with generator $\mathcal{MN}(\mathfrak{C}(q,q),\mathfrak{C}(q,c))$. $\mathfrak{C}$ and the $\mathfrak{m}_{\hat{k}[\hat{\kappa}]}$ are derived via table <ref> applied to $(\clipin(f)^N, \mathcal{G}(K(N,q,c)) \times \mathcal{G}^N)$. The first level of randomness is induced by $\mathcal{D}$. The second level of randomness is induced by $\theta$ and $\mathcal{G}$.

In practical terms, theorem <ref> states that neuron distributions in fully-connected layers are Gaussians with a standard deviation that is fixed across the layer and an expectation drawn from another zero mean Gaussian that is induced by the parameter. While we can glean this practical meaning, the statement of the theorem is still somewhat opaque. Specifically, it may not be clear how to cast a regular, practical architecture as a mean field architecture and generator $\mathcal{G}$. In a nutshell, we must re-cast segments of the layer graph between successive fully-connected layers of the regular architecture as a single elementwise layer in the mean field architecture. Then, we must re-cast the trainable parameter sub-vectors in non-fully-connected layers as a surrogate input generated by $\mathcal{G}$. Hence, $\theta$ in a regular context maps onto $\theta$ and $\mathcal{G}$ in a mean field context. (Hence, we usually skip over $\mathcal{G}$ in our non-technical discussions.) We demonstrate how to do this in section <ref> and in greater detail in section <ref>.

The major condition for theorem <ref> is that $\mathcal{D}$ is elem-like. We can immediately see that this condition cannot be achieved exactly by any input distribution over finite-dimensional vectors. For general $(q,c)$, for a set of $N$ vectors of dimensionality $d$ to have co-mean Kernel matrix $K(N,q,c)$, we must have $N \le d$! A strong requirement indeed. The elem-like condition is also not just a convenience. To see this, consider a $\mathcal{D}$ that is itself a discrete distribution over a finite set of values. If this is the case, the output distribution is clearly neither Gaussian nor elementwise, even in the limit $d_\text{MF} \rightarrow \infty$. In the limit, while the distribution of a neuron value in a fully-connected layer induced by $\theta$ for a fixed $x$ is Gaussian under very general conditions, the neuron distribution induced by $x \sim \mathcal{D}$ for a fixed $\theta$ is not. We further analyze the elem-like condition in section <ref>.

Finally, note that in our definition of `elem-like$(q,c)$', we require the $c$ parameter to be non-negative. This is because a negative $c$ would imply that every vector drawn from $\mathcal{D}$ is negatively correlated with every other vector drawn from $\mathcal{D}$, which doesn't even approximately make sense. $c$ also cannot be negative in proposition <ref>.

§.§ Empirical analysis

Like for most concepts in this work, the ultimate justification for theorem <ref> is its strong practical predictiveness, which we now demonstrate. Specifically, we investigate layer distributions of our fully-connected architectures from study A (section <ref>). We also investigate layer distributions of 40 “simple architectures” which have been randomly initialized 100 times using different random seeds (section <ref>). These architectures and initializations are the same as those used in e.g. section <ref>.

For each architecture, we investigate a single layer that is roughly halfway between input and output layer, which is the same layer used as the surrogate input layer for the “second half” of the network in sections <ref>, <ref> and <ref>. This layer is either a fully-connected layer in the case of non-residual architectures, or it is an addition layer in the case of residual architectures. Note that in our residual architectures, addition layers are effectively the sum of (possibly normalized) fully-connected layers. Therefore, we expect them to have the same meta-Gaussian properties as the layers they sum over. The reason for studying intermediate layers instead of output layers is that output layers in our architectures had width 10 or less, which makes it more difficult to assess e.g. whether neuron expectations in that layer appear to be Gaussian distributed across the layer. We denote the intermediate layer by $f_{\frac{1}{2}}$.

Our goal is to determine whether the layer meta-distribution is approximately the elementwise distribution that is generated by $\mathcal{MN}(\mathfrak{C}(q,q),\mathfrak{C}(q,c))$. We are not aware of a consensus protocol for empirically identifying a meta-distribution. However, it turns out that verifying that some $f_l$ indeed has the above meta-distribution is equivalent to verifying the following sub-properties.

* For each $\theta$ and $i$, $f_l(\mathcal{D})[i]$ is Gaussian.

* $\bar{f}_l$ is elementwise and generated by $\mathcal{N}(0,\mathfrak{C}(q,c))$ when $\theta$ is random.

* For each $\theta$ and $i$, $\mathbb{S}_xf_l(x)[i] = \mathfrak{C}(q,q) - \mathfrak{C}(q,c)$.

* For each $\theta$ and $i \neq i'$, $f_l(\mathcal{D})[i]$ is independent of $f_l(\mathcal{D})[i']$.

Of course, these properties, which hold exactly in the limit, can only hold approximately and with high probability over $\theta$ in practice. $\mathbb{S}$ here is the standard deviation operator as defined in section <ref>.

A convenient aspect of properties 1, 3 and 4 is that they are defined in terms of a fixed value of $\theta$. Hence, we can verify them on neural networks using fixed $\theta$. We do not have to compare different values of $\theta$. (In practice, we will use a trick to achieve the same thing for property 2.) Below, we investigate all four properties in turn. Note that we will defer some of the investigation to section <ref>, when we have derived rules for calculating $\mathfrak{C}(q,q)$ and $\mathfrak{C}(q,c)$ for study A architectures.

In this chapter, as in section <ref>, we will define metrics that capture the properties we investigate and plot the metric values for our architectures. Figures <ref> through <ref> are all laid out in the same way. In graph (A), we plot the metric values for our 40 simple architectures on CIFAR10 in the initial state. The interval depicted gives the range of the metric value across the 100 random seeds and the filled square depicts the mean. In graphs (B-D), we plot the metric values for our study A architectures in the initial state. In graphs (E-G), we plot the metric values for our study A architectures in the final state. While our theoretical framework applies directly only to the initial state, throughout this chapter, we also investigate how well it predicts the final state. In this chapter in general, as in section <ref>, we find that its predictiveness degrades significantly but not completely. As in section <ref>, whenever we quote the value of any metric not based on error in the final state, we exclude all architectures that did not attain a better-than-random validation error for any starting learning rate. If there is no meaningful way to select a starting learning rate, then the final state of the architecture is not meaningfully defined for the purpose of metric computation. Hence, the total number of architectures depicted in graphs E-G is somewhat lower than in graphs B-D in the following figures.

In this chapter, as in section <ref>, Gaussian unstable architectures (GUAs) play a prominent role as they again defy many trends that hold for other architectures. As always, they are depicted in graphs by green markers. (In figures <ref> and <ref>, they are depicted as blue points with green confidence intervals for visibility.) They were first discussed in section <ref>. As described at the end of section <ref>, we say one of our fully-connected architectures is a GUA if (i) it uses either the square or odd square activation function and (ii) it does not use layer normalization. In this and the next section, we add the color red for what we term `Gaussian edge architectures' (GEAs), which use ReLU and also do not use LN. We give a full explanation of these phenomena in section <ref>. For now, we notice that throughout this section, we find that GUAs and GEAs are “not (meta-) Gaussian”. Throughout this section and section <ref> only, we use the term `stable' to refer to architectures that are neither GUAs nor GEAs.

Now we turn to our four properties as given above.

Property 1: Gaussianity of individual neurons We begin by investigating whether $f_\frac{1}{2}(\mathcal{D})[i]$ is Gaussian for fixed $\theta$. A staple metric for measuring “degree of Gaussianity” is excess kurtosis.

The `neuron excess kurtosis' (NKURT) of a network $f$ with respect to an input distribution $\mathcal{D}$ at the $i$'th neuron in layer $l$ is

$$NKURT_l(f, \mathcal{D},i) = \mathbb{K}_xf_l(x)[i]$$

The excess kurtosis operator $\mathbb{K}$ as well as its estimator are given in section <ref>. The excess kurtosis of a Gaussian distribution is 0. Hence, we interpret NKURT being close to zero as Gaussian behavior. Specifically, in order to assess Gaussianity, excess kurtosis focuses on outliers, i.e. it measures whether a distribution is heavy-tailed or light-tailed relative to a Gaussian.

In figure <ref>, we plot the value of NKURT averaged over all neurons in $f_{\frac{1}{2}}$, i.e. $\mathbb{E}_iNKURT_{\frac{1}{2}}$. In figure <ref>(A-D), we find that for stable architectures, depicted in black, excess kurtosis is very close to zero, and actually more likely to be below than above zero, indicating Gaussianity. On the other hand, for GEAs and GUAs, depicted in red and green respectively, excess kurtosis is significant to enormous, indicating that neuron distributions are very non-Gaussian. GUAs have especially non-Gaussian neuron distributions. Specifically, before training, stable architectures tend to have neuron distributions in fully-connected (FC) layers that are as or more light-tailed than a Gaussian, whereas GEAs and especially GUAs have neuron distributions in FC layers that are heavy-tailed. In figure <ref>(E-G), we find that neuron distributions are often very light-tailed after training, but that there are also a few stable architectures that have large NKURT values.

In addition to $\mathbb{E}_iNKURT_{\frac{1}{2}}$, we investigated the standard deviation of excess kurtosis values across the layer, i.e. $\mathbb{S}_iNKURT_{\frac{1}{2}}$. We found that the standard deviation never significantly exceeded the absolute value of the mean and was often significantly lower. Hence, we do not display those values here as they provide little additional information. However, this does confirm that the neuron kurtosis is a characteristic value of the network and layer and does not vary too much from neuron to neuron in an FC layer.

In addition to capturing heavy-tailedness via NKURT, we wanted to examine the neuron distribution with another metric that focuses on whether the cumulative distribution function at each neuron is close to the cumulative distribution function of a Gaussian between, say, -2 and +2 standard deviations. For this purpose, we devised the `Gaussian histogram intersection' metric. We divide the real line into a small number of buckets and obtain a histogram from both the unit Gaussian and the expectation and variance normalized neuron distribution. We then sum the bucket-wise minimum of the two histograms to obtain the distribution overlap. A value of 1 then constitutes a perfect match, whereas a value close to 0 indicates dissimilarity.

The `neuron Gaussian histogram intersection' (NGHI) is

\begin{eqnarray*}
&=&\sum_{e=1}^{|E|-1} \min\Big(N(E[e+1]) - N(E[e]), \mathbb{E}_x\{E[e] \le \frac{f_l[i]-\bar{f}_l[i]}{\sqrt{\mathbb{E}_x(f_l[i]-\bar{f}_l[i])^2}} < E[e+1]\}\Big)
\end{eqnarray*}

where $N()$ is the unit Gaussian CDF, $E$ is the edge vector that defines the buckets and curly braces denote an indicator variable.

In figure <ref>, we plot $\mathbb{E}_iNGHI_{\frac{1}{2}}$ with

$$E=(-\infty,-1.8,-1.4,-1,-0.6,-0.2,0.2, 0.6, 1, 1.4, 1.8, \infty)$$

We obtain very similar results as for NKURT. Before training, stable architectures have an NGHI value above 0.9, and often above 0.95, indicating that their neuron distributions have CDFs comparable to a Gaussian. GUAs have NGHI values close to 0, with GEAs still having NGHI values mostly above 0.8. After training, NGHI is significantly lower overall and the architecture type is less predictive.

Property 2: Distribution of neuron expectations To show that $\bar{f}_\frac{1}{2}$ is elementwise and generated by $\mathcal{N}(0,\mathfrak{C}(q,c))$ as $\theta$ is random, we need to evaluate the covariance kernel. We calculate the kernel for study A architectures in the next section. For now, we will show that the neuron expectations across an FC layer for a given (typical) $\theta$ appear like a Gaussian sample. We use similar metrics as above.

$$KURTEX_l(f,\mathcal{D}) = \mathbb{K}_i\bar{f}[i]$$

\begin{eqnarray*}
&&GHIEX_l(f,\mathcal{D}, E)\\
&=& \sum_e \min\Big(N(E[e+1]) - N(E[e]), \mathbb{E}_i\{E[e] \le \frac{\bar{f}_l[i] - \mathbb{E}_{i'}\bar{f}_l[i']}{\sqrt{\mathbb{E}_{i'}(\bar{f}_l[i'] - \mathbb{E}_{i''}\bar{f}_l[i''])^2}} < E[e+1]\}\Big)
\end{eqnarray*}

where $N()$ is the unit Gaussian CDF, $E$ is the edge vector that defines the buckets and curly braces denote an indicator variable.

We give results in figures <ref> and <ref> with the same edge vector as before. Before training, KURTEX is close to zero and GHIEX is close to 1 for all architectures including GUAs and GEAs, indicating that neuron expectations appear highly Gaussian across the layer.

This is to be expected. If $f_l$ is an FC layer, we have $f_l = f_kW_l$, so we have $\bar{f}_l = \bar{f}_kW_l$. $\bar{f}_k$ can be viewed as a random vector that depends on the parameter sub-vectors of layers $f_1$ through $f_k$. Drawing $\bar{f}_l$ from the meta-distribution corresponds to drawing both $\bar{f}_k$ and $W_l$ independently. Because of the symmetry of $W_l$ when it is either Gaussian or orthogonally initialized, we have that the orientation of $\bar{f}_l$ is independent of $\bar{f}_k$. Further, since KURTEX and GHIEX are independent of the length of $\bar{f}_l$, both metrics are entirely independent of $\bar{f}_k$. But for a fixed $\bar{f}_k$, the distribution of $\bar{f}_l$ is either Gaussian when $W_l$ is Gaussian initialized or indistinguishably close to Gaussian when $W_l$ is orthogonally initialized. Hence, for any FC layer $f_l$ in any architecture with respect to any input distribution, the distribution of KURTEX and GHIEX induced by $\theta$ is the distribution those metrics attain on a Gaussian sample with size equal to the width of $f_l$. In plain terms, the distribution of neuron expectations across an FC layer appears as Gaussian as a sample from an actual Gaussian.

Because of this, any deviation we observe from $KURTEX=0$ and $GHIEX=1$ in figures <ref> and <ref> in the initial state can be regarded as sampling noise caused by the finite layer width, at least for non-residual architectures where $f_\frac{1}{2}$ is actually FC. In graph (A), architectures have width 100 and in graphs (B-D), architectures have width between 130 and 1000. This causes neuron expectations to appear more Gaussian for our study A architectures.
This above insight also provides further context for figures <ref> and <ref>. For our stable architectures, the individual neuron distributions in $f_\frac{1}{2}$ in the initial state appear significantly more Gaussian than samples from a Gaussian distribution with size between 100 and 1000.

After training, we find that neuron expectations across $f_\frac{1}{2}$ tend to become less Gaussian, though not much.

While we have shown that neuron expectations across an FC layer appear Gaussian for a fixed $\theta$, this does not necessarily imply that they are mean zero elementwise Gaussian as $\theta$ is random. However, we would obtain this from the above analysis if $\bar{f}_k$ has fixed square mean. In section <ref>, we show that this is true as $d_\text{MF} \rightarrow \infty$ under the conditions of theorem <ref> and approximately true empirically. Finally, in that section, we will calculate $\mathfrak{C}(q,c)$ and show that it empirically predicts the standard deviation of neuron expectations in FC layers. This completes the analysis of property 2.

Property 3: Standard deviation of individual neurons

Again, we will not calculate the value $\mathfrak{C}(q,q) - \mathfrak{C}(q,c)$ as it arises in property 3 for now. We will focus on demonstrating that the standard deviations of neuron distributions are approximately constant across the layer. Hence, we use the coefficient of variation of standard deviations.

The `coefficient of variation of neuron standard deviations' (CVNSTD) is

$$CVNSTD_l(f,\mathcal{D}) = \frac{\mathbb{S}_i(\mathbb{S}_xf_l[i])}{\mathbb{E}_i(\mathbb{S}_xf_l[i])}$$

A low value of CVNSTD indicates that the mean of standard deviations is much larger than the standard deviation of standard deviations, which indicates that the standard deviations are relatively constant across the layer as desired. This is what we find in figure <ref>. The largest CVNSTD values before training are around 0.5  and are largely observed on GUAs and GEAs. However, the “advantage” of stable architectures is not as drastic as e.g. in figures <ref> and <ref>. We also find that overall, CVNSTD values are much smaller for study A architectures in graphs (B-D) than for the simple architectures in graph (A). This is because study A uses orthogonal initialization for weight matrices, whereas the simple architectures use Gaussian initialization. Given some fixed value of $\Cov_{f_k}$, it is easy to see that minimizing $CVNSTD_l$ requires that the columns of $W_l$ have fixed and equal length, which happens under orthogonal initialization when $d_k \ge d_l$, but not under Gaussian initialization. Further, the spectrum of $\Cov_{f_k}$ itself tends to be better conditioned under orthogonal initialization [Pennington et al., 2017, Saxe et al., 2014].

After training, while CVNSTD values are much larger, standard deviations of neuron distributions are still relatively constant across the layer.

In section <ref>, we prove that  $\frac{1}{d_ld_\text{MF}}||\mathbb{S}_xf_l||_2^2$ converges to $\mathfrak{C}(q,q) - \mathfrak{C}(q,c)$ a.s. as $d_\text{MF} \rightarrow \infty$. Demonstrating this empirically for FC layers in section <ref> will complete the analysis of property 3.

Property 4: Neuron independence Finally, we examine to what degree the neuron distributions are independent. A staple metric for assessing independence is correlation.

The `neuron correlation' (NCORR) is

$$NCORR_l(f, \mathcal{D}) = \sqrt{\mathbb{E}_{i,i'}\Big(\frac{\mathbb{C}_x(f_l[i],f_l[i']))}{\mathbb{S}_xf_l[i]\mathbb{S}_xf_l[i']}\Big)^2}$$

The covariance operator $\mathbb{C}$ and its estimator are defined in section <ref>. It is worth noting that we consider the quadratic mean of pairwise correlations rather than the arithmetic mean. This is necessary because a positive and negative correlation with the same absolute value is equally likely due to the symmetry of $W_l$.

In figure <ref>, we plot $NCORR_{\frac{1}{2}}$. We find that even before training, most architectures exhibit some degree of correlation, and that the largest correlations tend to occur in GUAs and GEAs. Some amount of correlation can be explained by Gaussian initialization itself. Non-zero correlations between columns of $W_l$ lead to non-zero correlations between neurons. In fact, even if $\Cov_{f_k}$ is the identity matrix, we would expect $NCORR_\frac{1}{2} \approx \frac{1}{\sqrt{d_\frac{1}{2}}}$. Since architectures in figure <ref>(A) have $d_\frac{1}{2} = 100$, it is unsurprising that we do not observe $NCORR_{\frac{1}{2}}$ values below $\frac{1}{\sqrt{100}}$. As expected, neuron correlations are smaller for study A architectures than the simple architectures, as they are orthogonally initialized.

It is also interesting to note that $NCORR_0$ is 0.19 / 0.16 / 0.17 when evaluated for CIFAR10 / MNIST / waveform-noise respectively. Hence, the majority of stable architectures with orthogonal initialization actually reduce neuron correlation during forward propagation in the initial state. Since we would expect an orthogonal transformation to approximately preserve neuron correlation, this reduction is caused by the nonlinear layers. Pennington and Worah, 2017 studied the decorrelating effect of activation layers.

After training, NCORR rises significantly and can attain values close to 1.

If neuron distributions are independent across a layer, we expect the square mean of layer values to be approximately constant across $\mathcal{D}$. See e.g. proposition <ref> in the next subsection. Hence, another way to measure independence is with the metric below.

The `layer coefficient of variation' (LCV) is

$$LCV_l(f,\mathcal{D}) = \frac{\mathbb{S}_x||f_l||_2}{\mathbb{E}_x||f_l||_2}$$

In figure <ref>, we plot $LCV_\frac{1}{2}$. The results are more drastic than for the $NCORR$ metric. Stable architectures have an almost perfectly constant length, whereas many GUAs and GEAs exhibit severe variation. To appreciate the extent of this variation, consider an architecture with batch normalization. The largest possible value of LCV at a BN layer is attained when a single layer value in the batch has far greater length than the others, and then $LCV \approx \sqrt{|B|}$. Since we use a batch size of 250, many GUAs actually come close to this theoretical maximum. Length variation remains very small for stable architectures after training.

It turns out that studying the degree of variation among layer lengths is a key to explaining Gaussian instability in section <ref>.

NKURT averaged across an intermediate fully-connected or addition layer. In graph A, we depict the range across 100 random seeds and therefore 100 random initializations for 40 simple fully-connected architectures on CIFAR10 with a default depth of 51. The interval depicts the range of values across the random seeds and the filled square depicts the mean. In graphs B-G, we depict the value for study A architectures, before and after training. We place architectures on the x-axis in ascending order. Green markers correspond to GUAs and red markers correspond to GEAs. Some values fall outside the range of the y-axis. In graph A, we specify those values in the graph. Conclusion: Neuron distributions in FC layers of stable architectures in the initial state are approximately Gaussian with respect to excess kurtosis. This is not true for GUAs / GEAs.

NGHI averaged across an intermediate fully-connected or addition layer. Graphs are analogous to figure <ref>. Conclusion: Neuron distributions in FC layers of stable architectures in the initial state are approximately Gaussian in their cumulative distribution function. This is not necessarily true for GEAs and especially GUAs.

KURTEX at an intermediate fully-connected or addition layer. Graphs are analogous to previous figures. A very small number of values fall outside the range of the y-axis in graphs E- G. Conclusion: Neuron expectations appear very Gaussian across an FC layer, especially before training.

GHIEX at an intermediate fully-connected or addition layer. Graphs are analogous to previous figures. Conclusion: Neuron expectations appear very Gaussian across an FC layer, especially before training.

CVNSTD at an intermediate fully-connected or addition layer. Graphs are analogous to previous figures. A very small number of values fall outside the range of the y-axis in graphs E and G. Conclusion: Standard deviations are relatively constant across an FC layer, especially for stable architectures before training.

NCORR at an intermediate fully-connected or addition layer. Graphs are analogous to previous figures. Conclusion: Some correlation between neurons does arise, especially when Gaussian initialization (graph A) is used over orthogonal initialization (graphs B-F).

LCV at an intermediate fully-connected or addition layer. Graphs are analogous to previous figures. Conclusion: Stable architectures have near constant layer lengths, especially before training, whereas GUAs and GEAs often have wildly diverging lengths.

§.§ Neural regular data

Dataset   CIFAR10   MNIST   waveform-noise

INSQD density

INCMD density

INSQD and INCMD taken on the union of training and validation set for study A datasets. Note that INSQD for waveform-noise appears jittery because the sample size is smaller. Conclusion: Our datasets can be viewed as approximately `elem-like(1,0)'.

In theorem <ref>, we used the condition that the input distribution $\mathcal{D}$ is elem-like$(q,c)$. We explained that this condition cannot be fulfilled exactly. In this subsection, we further analyze it. We begin by simply examining the distribution of the input square mean and co-mean.

The `input square mean distribution' (INSQD) and `input co-mean distribution' (INCMD) are

\begin{eqnarray*}
INSQD(\mathcal{D}) &=& \mathbb{E}_ix[i]^2 \text{ where } x \sim \mathcal{D}\\
INCMD(\mathcal{D}) &=& \mathbb{E}_ix[i]x'[i] \text{ where } x,x' \sim \mathcal{D}\\
\end{eqnarray*}

We plot the density functions of INSQD and INCMD in table <ref>. They were taken on the union of training and validation set after data processing as always. We find that $INSQD \approx 1$ and $INCMD \approx 0$. Of course, for a hypothetical exact elem-like(1,0) distribution, INSQD would have all its probability mass concentrated at 1 and INCMD would have all its probability mass concentrated at 0. This suggests that our datasets are approximately elem-like$(1,0)$ and hence suitable for our mean field theory of meta-distributions.

Note that $INSQD=1$ can simply be achieved by normalizing the length of inputs and $\mathbb{E}_{x,x'}INCMD=0$ can simply be achieved by normalizing the expectation of input components. See pointwise and componentwise normalization in section <ref> respectively. $\mathbb{E}_xINSQD=1$ and $\mathbb{E}_{x,x'}INCMD=0$ indeed holds on the union of training and validation set of our datasets due to data processing according to section <ref>, and hence in figure <ref>. (Of course, this means that training and validation set are no longer strictly an independent sample for evaluating INSQD and INCMD.)

§.§.§ Elem-like $\approx$ elementwise

It turns out that an elementwise distribution becomes more and more elem-like as dimensionality increases. This explains our choice of the name `elem-like'.

Let $\mathcal{M}_1$ be a meta-distribution over distributions $\mathcal{X}_1$ over scalars $s$. For each $d > 0$, let $\mathcal{M}_d$ be the elementwise meta-distribution over distributions $\mathcal{X}_d$ over vectors $\chi_d$ of dimensionality $d$ that is generated by $\mathcal{M}_1$. Let $q = \mathbb{E}_{\mathcal{X}_1\sim\mathcal{M}_1,s\sim\mathcal{X}_1}s^2$ and $c =  \mathbb{E}_{\mathcal{X}_1\sim\mathcal{M}_1}(\mathbb{E}_{s\sim\mathcal{X}_1}s)^2$. Let $\chi_d$ and $\chi'_d$ be drawn from the same $\mathcal{X}_d$, which is drawn from $\mathcal{M}_d$. Then

$$\lim_{d\rightarrow \infty} \mathbb{E}_i\chi_d[i]^2 = q \text{ a.s.}$$

$$\lim_{d\rightarrow \infty} \mathbb{E}_i\chi_d[i]\chi'_d[i] = c \text{ a.s.}$$

Note that if $\mathcal{M}_1$ is $\mathcal{MN}(q,c)$ in the above proposition, then the $q$ and $c$ values we obtain are exactly the parameters of the meta-Gaussian. This was the reason behind parametrizing the meta-Gaussian in the way we did. Of course, the proposition also holds when $\mathcal{M}_1$ has all its probability mass on a single fixed distribution $\mathcal{X}_1$, which implies that each component of $\chi_d$ is drawn IID from $\mathcal{X}_1$.

In light of the proposition, the elem-like property can be interpreted as requiring that $\mathcal{D}$ mimics an elementwise distribution, which happens to be the type of distribution that allowed us to set up mean field theory in section <ref> in the first place. More generally, we can glean from proposition <ref> that the law of large numbers will lead to approximate elem-like-ness if input components are “relatively independent”, i.e. sufficiently independent that an average over all input components tends to exhibit a significant amount of convergence towards its mean. As mentioned in section <ref>, the $NCORR_0$ values obtained from our datasets are 0.19 for CIFAR10, 0.16 for MNIST and 0.17 for waveform-noise, i.e. they are low.

§.§.§ Elem-like $\approx$ use a small NLC

In section <ref>, we showed how the apparent nonlinearity of the true input-label function, as measured by the  PNLCD metric, is predictive of the ideal architecture NLC for a dataset. PNLCD is based on comparing the distance of labels relative to the size of the codomain to the distance of inputs relative to the size of the domain. Our evidence suggested that PNLCD concentrating around 1 would lead the dataset to also require a small initial NLC between 1 and 5. We described how inputs drawn from the unit Gaussian distribution, which is elementwise, would lead to PNLCD concentrating around 1. We showed that datasets must have “tight input clusters” to attain larger PNLCD sample values. We showed that our three study A datasets largely do not have such clusters. Now, we relate that analysis back to this section.

Let $\mathcal{D}$ be a data distribution using a finite set of labels in the form of one-hot vectors. Let $p_1, .., p_C > 0$ be the probabilities of each of the $C > 1$ classes occurring. Let the input of $\mathcal{D}$ be elem-like$(q,c)$ for some $q > c \ge 0$.

Then $PNLCD(\mathcal{D})=\frac{1}{\sqrt{1 - \sum_{\zeta=1}^Cp_\zeta^2}}$ with probability $1 - \sum_{\zeta=1}^Cp_\zeta^2$ and 0 with probability $\sum_{\zeta=1}^Cp_\zeta^2$.

In practical terms, if the inputs are elem-like and the class frequencies are not too unbalanced, PNLCD is concentrated at a value close to 1 for datapoints of different classes.

§.§.§ Discussion

We now have three different views of dataset regularity - (i) via its apparent nonlinearity, ideal NLC and the PNLCD metric, (ii) via the mean field theory of meta-distributions, elem-like-ness and the INSQD and INCMD metrics and (iii) via surrogate input distributions, elementwise-ness and the NCORR metric. These are all related. Below, we state a definition that unifies the three views.

A data distribution is `neural regular' if (i) datapoints with relatively different labels tend to have inputs that are relatively distant within their domain, (ii) input components are relatively independent and (iii) the distributions of input square means and co-means are relatively concentrated around fixed values.

Neural regularity, unfortunately, is not well-defined at this point. However, as also argued in section <ref>, we suspect a large fraction of practical deep learning datasets, especially after data processing, to be sufficiently neural regular to produce the results obtained in this work, as well as other important results. This can serve as a foundation for data-agnostic ZSAD, as outlined in section <ref>.

We note that the NLC is not just useful for neural regular datasets. In section <ref> we showed that the NLC is also predictive of test error when $PNLCD \approx 1$ is severely violated for datapoints of different classes. In that case, we have to estimate the ideal NLC range based on e.g. examining PNLCD.

Of course, we are not claiming that the precise requirements for the predictiveness of the NLC and the predictiveness of our mean field theory of meta-distributions will turn out to be exactly the same. There may be “imperfectly neural regular” datasets that are more suitable for one concept than the other. Investigating this beyond the analysis already performed is a task for future work. However, it is noteworthy that the predictiveness of the NLC, as modulated by the true input-label function, is closely related to the predictability of the NLC using mean field theory in the sense of figure <ref>.

§ MEAN FIELD THEORY OF PRACTICAL METRICS AND ARCHITECTURES

§.§ A-architectures

We now apply the abstract results from previous sections to derive even more practical insights. Specifically, we calculate the limit of metrics such as the NLC as width converges to infinity. In subsection <ref>, our analysis results in a simple and instructive formula for the limit of the NLC.

In section <ref>, we detailed the types of fully-connected architectures we used for study A, which are some of the simplest types of practical architectures. We will use our study A architectures as a blueprint in this section.

We say an architecture $f$ is an `A-architecture' (after study A) if it has the following properties. A-architectures follow our standard framework from section <ref>, except for a single property marked with (*).

* It is composed of layers using the following operations: fully-connected, activation, bias, layer normalization, batch normalization and addition. See section <ref> for how we define these operations.

* Activation functions used are non-constant, twice differentiable, and both they and their derivatives are controlled by $\mathcal{C}^\text{E2}$.

* Addition layers have fixed, non-zero addition weights.

* The input layer has fixed width $d_0=d_\text{in}$. Non-input, non-output layers $f_l$ have width $d_ld_\text{MF}$, where $d_l$ is fixed and $d_\text{MF}$ can vary. There is at least one non-input, non-output layer. The output layer may have variable width $d_Ld_\text{MF}$ or be a fully-connected `readout layer' of fixed width $d_L=d_\text{out}$. We also use $d_l^\text{MF}$ to denote the width of a layer, which can be $d_l$ or $d_ld_\text{MF}$ depending on the layer. (*)

* Components of weight matrices $W_l$ are initialized as independent Gaussians with mean zero and variance $\frac{\sigma_l^2}{d_k^\text{MF}}$ for some `variance parameter' $\sigma_l^2 > 0$ that is fixed as $d_\text{MF}$ varies.

* Components of bias vectors $\beta_l$ are initialized as independent Gaussians with mean zero and variance $\sigma_l^2$ for some fixed `variance parameter' $\sigma_l^2$.

* If $f_l$ is a normalization layer, the regularizer $\epsilon_l$ is positive.

* $f$ uses either batch normalization or layer normalization layers, but not both.

* In the layer graph, there do not exist two directed paths that begin at distinct activation layers, end at the same layer, and do not contain any fully-connected layers.

* In the layer graph, there does not exist a directed path that begins at an activation layer, ends at a different activation or normalization layer, and does not contain a fully-connected layer.

* In the layer graph, there do not exist two distinct directed paths that begin and end at the same layer and contain no fully-connected layer outside of their starting point.

A-architectures follow the framework of section <ref>, except that layers have variable width. Variable width stems from mean field architectures defined in section <ref>. In contrast to mean field architectures, A-architectures have a single input and output layer and there is no weight sharing.

Our architectures from study A are A-architectures when $d_\text{MF} = 1$, except (i) weight matrices are orthogonally initialized, (ii) some activation functions are only directionally differentiable instead of twice differentiable everywhere and (iii) the last addition layer in residual architectures multiplies the skip connection with a fixed orthogonally initialized matrix. These are minor differences in this context. Large orthogonally initialized matrices have approximately Gaussian entries with very mild cross-entry dependency. Our directionally differentiable activation functions are also very close to smooth activation functions. For example, ReLU is approximately $\frac{1}{c}\log(1+e^{cx})$ for some very large $c$. See section <ref> for further discussion on assuming differentiability in neural network analysis.

A-architectures turn out to be a specialization of mean field architectures. This insight is behind the results given below. While A-architectures allow more than the two layer operations used in mean field architectures, each possible combination of these operations can nonetheless be expressed in terms of those two. Yang, 2019 explained how to cast operations such as layer normalization and bias as elementwise layers. In this section, we cast entire architectures made up of up to five different operations as mean field architectures, which requires groups of layers to be re-cast jointly. This then enables the utilization of e.g. background theorem <ref> to prove the results below. Despite re-casting groups of layers as one, we show that we can nonetheless calculate a wide range of key metrics via recursion from one A-architecture layer to the next.

We note that many of the properties of A-architectures given in the definition, which are restrictive in nature, are not “strictly necessary”. One can derive results similar to the ones given below for more general classes of architectures, including those admitting convolutional layers, as we further discuss at the end of section <ref> and in section <ref>. Because of space and time limitations, we decided to focus on a class of architectures that would yield calculation rules that are as simple and instructive as possible. We hope that our presentation is itself instructive and will allow readers to generalize our results to whatever architecture class they are interested in with the techniques we employ.

§.§ Metric limits in A-architectures

We now show how to calculate the limits of metrics as width converges to infinity in A-architectures. After a definition, we state the main theorem of this section.

The `covariance kernel' $\mathfrak{C}_\tau$ of an activation function $\tau$ and scalars $q \ge c \ge -q$ is

$$\mathfrak{C}_\tau(q,c) = \mathbb{E}_{s,t\sim \mathcal{N}(\mu,\Sigma)} \tau(s)\tau(t) \text{ where } \mu = \begin{pmatrix} 0 \\ 0 \end{pmatrix} \text{, } \Sigma = \begin{pmatrix} q & c \\ c & q \end{pmatrix}$$

We also write $\mathfrak{C}_\tau(c)$ with $1 \ge c \ge -1$ short for $\mathfrak{C}_\tau(1,c)$.

This definition mirrors that of the covariance kernel of an architecture from section <ref>. When $c=0$, the definition simplifies to $\mathfrak{C}_\tau(q,0) = (\mathbb{E}_{s\sim\mathcal{N}(0,q)}\tau(s))^2$ and when $c=q$, the definition simplifies to $\mathfrak{C}_\tau(q,q) = \mathbb{E}_{s\sim\mathcal{N}(0,q)}\tau(s)^2$.

Let ...

* ... $f$ be an A-architecture with an output layer of variable width.

* ... $\mathcal{D}$ be an input distribution.

* ... $x^{(1)},..,x^{(N)}$ be a sample from $\mathcal{D}$ of size $N \ge 2$ and let $\mathcal{D}^{(N)}$ be the discrete uniform distribution over that sample.

* ... $\vec{\epsilon}$ be the vector of all regularizers used by normalization layers in $f$.

* $\mathcal{D}$ is elem-like$(q,c)$.

* $q > c$

If $f$ does not contain batch normalization but can contain layer normalization layers, we have

\begin{eqnarray}
\lim\mathbb{E}_if_l(x^{(1)})[i] &=& \mathfrak{m}_l\label{eqn5p0}\\
\lim\mathbb{E}_if_l(x^{(1)})[i]^2 &=& \mathfrak{q}_l\label{eqn5p1}\\
\lim\mathbb{E}_if_l(x^{(1)})[i]f_l(x^{(2)})[i] &=& \mathfrak{c}_l \label{eqn5p2}\\
\lim\frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m}(x^{(1)})||_F^2 &=& \frac{\mathfrak{g}_l}{\mathfrak{g}_m} \label{eqn5p4}\\
\lim\frac{1}{d_l^\text{MF}}\mathbb{E}_x||f_l(x)||^2_2 &=& \mathfrak{q}_l\label{eqn5p5}\\
\lim\frac{1}{d_l^\text{MF}}||\mathbb{E}_xf_l(x)||^2_2 &=& \mathfrak{c}_l \label{eqn5p6}\\
\lim\frac{1}{d_l^\text{MF}}||\mathbb{S}_xf_l(x)||^2_2 &=& \mathfrak{q}_l - \mathfrak{c}_l \label{eqn5p7}\\
\lim\frac{1}{d_l^\text{MF}}\mathbb{E}_x||\mathcal{J}_{l,m}(x)||_F^2 &=& \frac{\mathfrak{g}_l}{\mathfrak{g}_m} \label{eqn5p8}\\
\lim\frac{1}{d_l^\text{MF}}\mathbb{E}_x\Tr(\mathcal{J}_{l,m}(x)\Cov_{f_m}\mathcal{J}_{l,m}(x)^T) &=& \frac{\mathfrak{g}_l(\mathfrak{q}_m - \mathfrak{c}_m)}{\mathfrak{g}_m}\label{eqn5p9}\\
\lim NLC(f_l(f_m),f_m(\mathcal{D}^{(N)})) &=& \sqrt{\frac{\mathfrak{g}_l(\mathfrak{q}_m - \mathfrak{c}_m)}{\mathfrak{g}_m(\mathfrak{q}_l - \mathfrak{c}_l)}}\label{eqn5p10}
\end{eqnarray}

* ... $x \sim \mathcal{D}^{(N)}$

* ... $\lim$ stands for $\lim_{\vec{\epsilon}\rightarrow 0,N\rightarrow\infty}\big(\lim_{d_\text{MF}\rightarrow\infty} ... \text{a.s.}\big)$. The inner limit takes $d_\text{MF}$ to infinity and the outer limit takes $N$ to infinity as well as $\vec{\epsilon}$ to zero. The inner limit is an “almost sure” limit.

* ... $\mathfrak{m}_l$, $\mathfrak{q}_l$, $\mathfrak{c}_l$ and $\mathfrak{g}_l$ are calculated via table <ref>.

* ... $f_m$ is a bottleneck for $f_l$.

* ... $0 \le m \le l \le L$

* ... randomness is induced by $\theta$ and the $x^{(n)}$.

Further, if the output layer is instead a readout layer of fixed dimensionality $d_\text{out}$, the above limits do not necessarily hold when $l=L$. Instead, (a) as $d_\text{MF} \rightarrow \infty$, the meta-distribution of the output layer expansion-converges to the elementwise meta-distribution with generator $\mathcal{MN}(\mathsf{parm1}, \mathsf{parm2})$, where $\lim_{\vec{\epsilon}\rightarrow 0}\mathsf{parm1}=\mathfrak{q}_L$ and $\lim_{\vec{\epsilon}\rightarrow 0}\mathsf{parm2}=\mathfrak{c}_L$. The first level of randomness is induced by $\mathcal{D}$ and the second level by $\theta$. And, (b) when $N$ is fixed, for almost all samples, as $d_\text{MF} \rightarrow \infty$, $(f(x^{(1)}), .., f(x^{(N)}))$ converges in distribution to a Gaussian that is elementwise over output vectors where the generator has mean zero and covariance matrix $K(N, \mathsf{parm1}, \mathsf{parm2})$, where again $\lim_{\vec{\epsilon}\rightarrow 0}\mathsf{parm1}=\mathfrak{q}_L$ and $\lim_{\vec{\epsilon}\rightarrow 0}\mathsf{parm2}=\mathfrak{c}_L$. Randomness for each sample is induced by $\theta$.

If $f$ does not contain layer normalization but can contain batch normalization layers, analogous statements hold. See section <ref> for details.

Because this theorem is already very long, we decided not to give the BN case explicitly here, but instead defer it to section <ref>. Nonetheless, we include BN in table <ref> and in the discussion below. As in section <ref>, we generalize to the BN case by letting $f$ take batches of inputs instead of individual inputs.

§.§.§ Interpreting theorem <ref>

Assumptions As in theorem <ref>, the most severe assumption is that $\mathcal{D}$ is elem-like. See section <ref> for a discussion of elem-like-ness, which holds approximately for our study A datasets.

$q > c$ is a mild assumption. $q = c$ would effectively imply that all inputs to $f$ are equal, and therefore that each layer takes a single fixed value. This would e.g. make the NLC invalid and BN degenerate as $\vec{\epsilon}$ converges to zero. Since the definition of `elem-like$(q,c)$' also requires $c \ge 0$, we have $q > c \ge 0$.

Note that we no longer explicitly require rank stability or parameter-controlled multi-activation functions in theorem <ref> the way we did throughout sections <ref> and <ref>. Now, we obtain these things indirectly from our other assumptions and the definition of A-architectures.

Value     $f_l$     Formula
$\mathfrak{q}_l$   input   $q$ parameter of inputs

$\mathfrak{q}_l$   FC   $\sigma_l^2\mathfrak{q}_k$

$\mathfrak{q}_l$   activation   $\mathfrak{C}_{\tau_l}(\mathfrak{q}_k,\mathfrak{q}_k)$

$\mathfrak{q}_l$   bias   $\mathfrak{q}_k + \sigma_l^2$

$\mathfrak{q}_l$   LN   $1$

$\mathfrak{q}_l$   BN   $1$

$\mathfrak{q}_l$   addition   $\sum_{\kappa_l=1}^{K_l} w_{l,\kappa_l}^2\mathfrak{q}_{k_l[\kappa_l]}$

$\mathfrak{c}_l$   input   $c$ parameter of inputs

$\mathfrak{c}_l$   FC   $\sigma_l^2\mathfrak{c}_k$

$\mathfrak{c}_l$   activation   $\mathfrak{C}_{\tau_l}(\mathfrak{q}_k,\mathfrak{c}_k)$

$\mathfrak{c}_l$   bias   $\mathfrak{c}_k + \sigma_l^2$

$\mathfrak{c}_l$   LN   $\frac{\mathfrak{c}_k}{\mathfrak{q}_k}$

$\mathfrak{c}_l$   BN   $0$

$\mathfrak{c}_l$   addition   $\sum_{\kappa_l=1}^{K_l} w_{l,\kappa_l}^2\mathfrak{c}_{k_l[\kappa_l]}$

$\mathfrak{m}_l$   input   undetermined

$\mathfrak{m}_l$   FC   $0$

$\mathfrak{m}_l$   activation   $\mathbb{E}_{s \sim \mathcal{N}(0,\mathfrak{q}_k)}\tau_l(s)$

$\mathfrak{m}_l$   bias   $\mathfrak{m}_k$

$\mathfrak{m}_l$   LN   $0$

$\mathfrak{m}_l$   BN   $0$

$\mathfrak{m}_l$   addition   $\sum_{\kappa_l=1}^{K_l} w_{l,\kappa_l}\mathfrak{m}_{k_l[\kappa_l]}$

$\mathfrak{g}_l$   input   1

$\mathfrak{g}_l$   FC   $\sigma_l^2\mathfrak{g}_k$

$\mathfrak{g}_l$   activation   $\mathfrak{C}_{\tau_l'}(\mathfrak{q}_k,\mathfrak{q}_k)\mathfrak{g}_k$

$\mathfrak{g}_l$   bias   $\mathfrak{g}_k$

$\mathfrak{g}_l$   LN   $\frac{\mathfrak{g}_k}{\mathfrak{q}_k}$

$\mathfrak{g}_l$   BN   $\frac{\mathfrak{g}_k}{\mathfrak{q}_k-\mathfrak{c}_k}$

$\mathfrak{g}_l$   addition   $\sum_{\kappa_l=1}^{K_l} w_{l,\kappa_l}^2\mathfrak{g}_{k_l[\kappa_l]}$

Recursive rules for calculating mean field limits in an A-architecture $f$.

Limits It is important that we take the limit over $d_\text{MF}$ first, before we take the limit over $N$. Therefore, we are not technically making statements about metric values with respect to $\mathcal{D}$ as $d_\text{MF}$ converges to infinity, but only about the metric values with respect to arbitrarily large samples of $\mathcal{D}$ as $d_\text{MF}$ converges to infinity. This is not entirely satisfying from a mathematical standpoint. However, it is practically sufficient because we can only ever apply a neural network to a finite number of inputs. Yang, 2019 also focused on arbitrarily large, finite sets of inputs. The reason for this ordering of the limit is because we follow a proof strategy as outlined in section <ref>, where we propagate the sample forward using the $N$-duplex of $f$. We can then evaluate means and co-means of that $N$-duplex to obtain the metric values with respect to the sample of size $N$. Exchanging the limits is likely possible, but would require a more powerful version of background theorem <ref>, which goes beyond the scope of this work.

The limit over $\vec{\epsilon}$ is of minor importance. We need positive regularizers to eliminate the singularity of the normalization operations, which can be considered a “nuisance” for theoretical analysis. If $\vec{\epsilon}$ is very small, as it is in practice, it has no significant impact on any observed phenomena.

As mentioned at the beginning of section <ref>, batch normalization with a finite batch size is a cumbersome operation from a mean field perspective. When theorem <ref> is applied to architectures with BN, we take the limit of batch size to infinity while taking the limit of $N$ to infinity. When neural networks with BN are deployed after training, the batch moments are generally replaced by training set moments, as we described in section <ref>. This is similar to taking the batch size towards infinity. Unless very small batches are used during training, infinite batches capture the mean field behavior of the architecture adequately.

Limit quantities $\mathfrak{m}_l$, $\mathfrak{q}_l$ and $\mathfrak{c}_l$ can be easily interpreted as layer mean, square mean and co-mean based on statements (<ref>), (<ref>) and (<ref>). $\mathfrak{g}_l$ can be understood in light of statement (<ref>). This yields $\lim\frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,0}(x^{(1)})||_F^2 = \mathfrak{g}_l$. We further have $||\mathcal{J}_{l,0}(x^{(1)})||_F^2 = \mathbb{E}_u||\mathcal{J}_{l,0}(x^{(1)})u||_2^2$ when $u$ is unit Gaussian. Hence, $\mathfrak{g}_l$ can be regarded as the limit of the square mean of the gradient as it is forward-propagated from the input. Using forward-mode propagation instead of backpropagation for the gradient enables us to base theorem <ref> on background theorem <ref> instead of the master theorem of Yang, 2020. Yang, 2020 requires more complex mean field architectures and stronger assumptions to model gradient backpropagation. This strategy also allows us to bypass the gradient of the loss function $\frac{d\ell}{df}$ which is not part of the mean field framework. Yang, 2020 have to use a Gaussian approximation for the gradient of the loss function. We suspect that using forward-mode propagation, the tools and weaker assumptions from Yang, 2019 may actually be sufficient to obtain the results of Yang, 2020.

Statements We will interpret the statements one by one.

(<ref>) This is equivalent to the first statement of background theorem <ref>.

(<ref>) and (<ref>) This is equivalent to the second statement of background theorem <ref>. $\mathfrak{q}_l$ is the layer square mean and corresponds to $\mathfrak{c}_{l;l}$. $\mathfrak{c}_l$ is the layer co-mean and is obtained by propagating $x^{(1)}$ and $x^{(2)}$ jointly through the 2-duplex of $f$. $\mathfrak{c}_l$ corresponds to $\mathfrak{c}_{l;l'}$, where $f_l$ and $f_{l'}$ form a pair in the duplex.

(<ref>) As explained above, $\mathfrak{g}_l$ can be regarded as the square mean of the forward-propagated gradient. It ultimately derives from the second statement of background theorem <ref> when gradient propagation is cast as “regular” forward propagation in a mean field architecture (section <ref>). By definition, $\mathcal{J}_{l,0} = \mathcal{J}_{l,m}\mathcal{J}_{m,0}$ because $f_m$ is a bottleneck for $f_l$. Statement (<ref>) yields $||\mathcal{J}_{l,0}(x^{(1)})||_F^2 \approx \mathfrak{g}_l$, $||\mathcal{J}_{m,0}(x^{(1)})||_F^2 \approx \mathfrak{g}_m$ and $||\mathcal{J}_{l,m}(x^{(1)})||_F^2 \approx \frac{\mathfrak{g}_l}{\mathfrak{g}_m}$. So statement (<ref>) can be interpreted as saying that the Frobenius norm of the Jacobian is decomposable, similar to section <ref>.

(<ref>) This is similar to (<ref>), except that we take the expectation over $x$ in addition to the mean over $i$. Since $x$ is drawn from a finite sample, we simply take the mean over that sample, which works as expected.

(<ref>) This is an interesting statement that relates the co-mean of layer values to the magnitude of neuron expectations. It can be understood in light of proposition <ref>, which relates the co-mean of two vectors to the square mean of their componentwise expectations.

(<ref>) This follows directly from the previous two statements as $||\mathbb{S}_xf_l||_2^2 = \mathbb{E}_x||f_l||_2^2 - ||\mathbb{E}_xf_l||_2^2$.

(<ref>) This follows from (<ref>) by taking the mean over a finite sample.

(<ref>) First, we observe $\Tr(\Cov_{f_m}) = ||\mathbb{S}_xf_m(x)||_2^2$ and hence $\lim \frac{1}{d_m^\text{MF}}\Tr(\Cov_{f_m}) = (\mathfrak{q}_m - \mathfrak{c}_m)$, according to statement (<ref>). Second, we observe $\frac{1}{d_l^\text{MF}}\mathbb{E}_x\Tr(\mathcal{J}_{l,m}(x)\mathcal{J}_{l,m}(x)^T)=\frac{1}{d_l^\text{MF}}\mathbb{E}_x||\mathcal{J}_{l,m}||_F^2 \approx (\frac{1}{d_l^\text{MF}}\mathbb{E}_x||\mathcal{J}_{l,0}||_F^2)(\frac{1}{d_m^\text{MF}}\mathbb{E}_x||\mathcal{J}_{m,0}||_F^2)^{-1}$ if $\mathcal{J}_{m,0}$ is assumed to be uniformly random. Statement (<ref>) is the amalgamation of both observations.

(<ref>) This is the square root of the ratio of (<ref>) and (<ref>).

Expansion-convergence and convergence in distribution results These are straightforward applications of theorem <ref> and background corollary <ref> respectively, where the abstract $\mathfrak{C}$ based on table <ref> has been replaced by concrete values based on table <ref>.

Table <ref> We will begin with some high-level observations.

$\mathfrak{g}_l$ always depends multiplicatively on $\mathfrak{g}_k$. This is because gradient propagation multiplies the gradient of each layer to the in-flowing gradient. If the magnitude of the in-flowing gradient changes by some factor, so does the magnitude of the out-flowing gradient.

The recursion of $\mathfrak{q}$ does not depend on $\mathfrak{c}$. This is because $\mathfrak{q}$ is the limit of quantities that can be obtained by forward-propagating only a single input without regard to its relationship to other inputs, as long as $f$ does not use BN. And the BN operation sets $\mathfrak{q}$ to a fixed value regardless of $\mathfrak{c}$. Conversely, the recursion of $\mathfrak{c}$ does depend on $\mathfrak{q}$. This is because operations such as activation and LN depend upon the square mean of the dependency in a nonlinear fashion.

The recursion of $\mathfrak{q}$ and $\mathfrak{c}$ does not depend on $\mathfrak{m}$. This is because we chose the definition of A-architectures intentionally to eliminate this dependency. $\mathfrak{q}$ and $\mathfrak{c}$ are required to calculate the NLC limit, but $\mathfrak{m}$ is not. So A-architectures allow us to calculate the NLC with the simplest possible recursion. Both $\mathfrak{q}$ and $\mathfrak{c}$ would depend on $\mathfrak{m}$ if we allowed, say, activation layers followed by normalization layers or addition layers that depend on multiple activation layers.

In order for all statements of theorem <ref> to be valid, we require $\mathfrak{q}_l > \mathfrak{c}_l \ge 0$ and $\mathfrak{g}_l > 0$ for all $l$. Since we assume $q > c \ge 0$, we have $\mathfrak{q}_0 > \mathfrak{c}_0 \ge 0$ and table <ref> yields $\mathfrak{g}_0 =1 > 0$. It turns out that the recursion preserves this property from layer to layer. This is easy to see for non-activation layers. For activation layers, we have the following.

Assume $\tau$ is continuous, non-constant and $q > c > -q$. Then
$$\mathfrak{C}_{\tau}(q,q) > \mathfrak{C}_{\tau}(q,c)$$

For any $\tau$ and $q \ge c \ge 0$, we have
$$\mathfrak{C}_{\tau}(q,c) \ge 0$$

Assume $\tau$ is continuous, not the zero function and $q > 0$. Then
$$\mathfrak{C}_{\tau}(q,q) > 0$$

So, putting things together we have the following.

Let $f$ be an A-architecture. Let $\mathfrak{q}_l$, $\mathfrak{c}_l$ and $\mathfrak{g}_l$ be defined according to table <ref> where $q > c \ge 0$. Then $\mathfrak{q}_l > \mathfrak{c}_l \ge 0$ and $\mathfrak{g}_l > 0$ for all $l$.

In general, the rules for $\mathfrak{q}$, $\mathfrak{c}$ and $\mathfrak{g}$ look similar to the definition of the operation itself, and similar to each other. This will become more apparent below, where we discuss the rules for each individual operation.

input The conditions on $\mathcal{D}$ were chosen specifically to yield a fixed $\mathfrak{q}$ and $\mathfrak{c}$ value.

FC Both the operation and its gradient apply a multiplicative transformation to a vector, where the scale of that transformation is controlled by $\sigma_l^2$. This is reflected in the recursive calculation rules. Because each weight matrix entry has mean zero, we obtain an $\mathfrak{m}$ value of zero.

activation We effectively replace $\tau$ with $\mathfrak{C}_\tau$ and $\tau'$ with $\mathfrak{C}_{\tau'}$ during input / gradient propagation respectively. In this way, the recursion mimics the layer operation. $\mathfrak{C}_\tau$ is defined in terms of an expectation over a Gaussian distribution, similar to $\mathbb{E}_e$ in table <ref>. We will dive deeper into this in section <ref>.

bias Because the same bias vector is added to both $f_k(x^{(1)})$ and $f_k(x^{(2)})$, we experience the same increase in both the square mean and co-mean. Bias vector addition has a unit gradient, so there is no effect on $\mathfrak{g}_k$. Since bias vector components have mean zero, there is no effect on $\mathfrak{m}_k$.

LN Note that one property of A-architectures is that normalization layers cannot follow activation layers before another FC layer. Therefore, we have $\mathfrak{m}_k = 0$. So the mean subtraction part of LN has no effect on the limit and we obtain $\mathfrak{m}_l = 0$ regardless. Dividing the dependency by the standard deviation has the effect of dividing $\mathfrak{q}_k$, $\mathfrak{c}_k$ and $\mathfrak{g}_k$ by $\mathfrak{q}_k$.

BN The key difference with LN is that BN removes the neuron mean across inputs in a batch, whereas LN removes the layer mean for each input. So the subtraction part of BN affects $\mathfrak{c}_k$ instead of $\mathfrak{m}_k$, which may be nonzero. So we subtract $\mathfrak{c}_k$ from both $\mathfrak{q}_k$ and $\mathfrak{c}_k$ and then divide by $\mathfrak{q}_k - \mathfrak{c}_k$, which also affects the gradient.

addition We apply weighted addition to all limit quantities, where quantities that represent a square mean or co-mean require squaring the weights.

Scope As discussed in section <ref>, the proof of theorem <ref> requires us to re-cast segments of the layer graph of the A-architecture between successive fully-connected layers as a single elementwise layer in a mean field architecture. This process is highly dependent on the form that this segment takes. Properties <ref>, <ref>, <ref>, <ref> and <ref> of A-architectures limit the range of segments that can occur in an A-architecture. While the complexity of segments for which we prove mean field limits, to our knowledge, goes beyond that of related work, we need to draw the line somewhere. As stated, we orient ourselves on the architectures we use for study A. Limiting the scope of the class of A-architectures also keeps the calculation rules of table <ref> simple and enables e.g. proposition <ref>.

If we did not have property <ref>, it would be possible to have consecutive activation layers in an A-architecture. The first activation layers could induce non-Gaussian neuron distributions. Then it would no longer be possible to model the dependency of the second activation layer as Gaussian, as is done by the covariance kernel $\mathfrak{C}_\tau$. In table <ref>, the recursion for $\mathfrak{q}$, $\mathfrak{c}$, and $\mathfrak{g}$ do not depend on $\mathfrak{m}$. Without property <ref>, it would be possible to have an addition layer that adds together two activation layers, which can have non-zero $\mathfrak{m}$ values. Then, depending on whether those $\mathfrak{m}$ values e.g. have the same or different signs, they could cancel out or amplify in the addition layer, which would have a knock-on effect on e.g. $\mathfrak{q}$. The calculation rules for addition layers in table <ref> assume that the weighted sum is taken over essentially independent layers. Without property <ref>, this would not be the case as there could be an addition layer that e.g. adds a layer to itself.

Properties <ref>, <ref> and <ref> are basic requirements for applying mean field theory. Property <ref> is technical and ensures that normalization operations do not diverge for e.g. zero input vectors. We suspect property <ref> is unnecessary in that theorem <ref> as stated would hold without it. However, BN and LN layers require somewhat different proof strategies. We did not consider them jointly to keep the complexity and length of the proof at its current level. Property <ref> is a convenience. Any layer graph edge corresponding to a zero addition weight can simply be removed.

The aspect of property <ref> that requires differentiable activation functions is necessary to state theorem <ref> in terms of gradients. We use the second derivative of activation functions to prove parameter-control for layer graph segments containing LN and activation layers. A weaker assumption, such as a locally Lipschitz derivative, might also suffice. Finally, assuming control by $\mathcal{C}^\text{E2}$ is necessary to apply background theorem <ref> and its corollaries.

To fully appreciate the relevance of each of the properties of an A-architecture for theorem <ref>, it is necessary to study the proof (sections <ref>, <ref>).

The requirement that $f_m$ is a bottleneck for $f_l$ can likely be eliminated if we set all $\mathfrak{g}$ values corresponding to layers that are not descendants of $f_m$ to zero and apply all other calculation rules as normal.

§.§ Empirical analysis

Metric name     Notation     Definition     Statement     Limit
Layer quadratic mean     $LQM_l(f,x)$     $\sqrt{\mathbb{E}_if_l(x)[i]^2}$     (<ref>)    $\sqrt{\mathfrak{q}_l}$

Jacobian Frobenius norm     $JACF_{l,m}(f,x)$     $\frac{1}{\sqrt{d_l}}||\mathcal{J}_{l,m}(x)||_F$     (<ref>)    $\sqrt{\frac{\mathfrak{g}_l}{\mathfrak{g}_m}}$

Layer scale     $LSCALE_l(f,\mathcal{D})$     $\sqrt{\frac{1}{d_l}\mathbb{E}_x||f_l||^2_2}$     (<ref>)    $\sqrt{\mathfrak{q}_l}$

Quadratic mean of neuron expectations     $QMNEX_l(f,\mathcal{D})$     $\frac{1}{\sqrt{d_l}}||\mathbb{E}_xf_l||_2$     (<ref>)    $\sqrt{\mathfrak{c}_l}$

Quadratic mean of neuron standard deviations     $QMNSTD_l(f,\mathcal{D})$     $\frac{1}{\sqrt{d_l}}||\mathbb{S}_xf_l||_2$     (<ref>)    $\sqrt{\mathfrak{q}_l - \mathfrak{c}_l}$

Nonlinearity coefficient numerator     $NLCNUM_{l,m}(f,\mathcal{D})$     $\sqrt{\frac{1}{d_l}\mathbb{E}_x\Tr(\mathcal{J}_{l,m}\Cov_{f_m}\mathcal{J}_{l,m}^T)}$     (<ref>)    $\sqrt{\frac{\mathfrak{g}_l(\mathfrak{q}_m - \mathfrak{c}_m)}{\mathfrak{g}_m}}$

Nonlinearity coefficient     $NLC_{l,m}(f,\mathcal{D})$     $\sqrt{\frac{\mathbb{E}_x\Tr(\mathcal{J}_{l,m}\Cov_{f_m}\mathcal{J}_{l,m}^T)}{\Tr(\Cov_{f_l})}}$     (<ref>)    $\sqrt{\frac{\mathfrak{g}_l(\mathfrak{q}_m - \mathfrak{c}_m)}{\mathfrak{g}_m(\mathfrak{q}_l - \mathfrak{c}_l)}}$

Metrics used to determine the practical predictiveness of theorem <ref>. For each metric, we give the statement in theorem <ref> it corresponds to as well as its mean field limit.

In this subsection, we validate theorem <ref> empirically. Specifically, we focus on statements (<ref>), (<ref>) through (<ref>), (<ref>) and (<ref>). Combining those statements with the analysis done in section <ref> validates the expansion-covergence clause at the end of theorem <ref>. Each statement (<ref>) through (<ref>) is an equality statement with a left- and right-hand side. The left-hand side is a limit of a metric defined in terms of propagation through a network of variable but finite width. The right-hand side contains expressions of “limit quantities”, which are defined via table <ref> and do not require any network or layer evaluation. Below, we discuss how we compute a representative value for each side. Then, we present results.

Computing the finite width metrics To compute values for the left-hand side of the theorem statements, we simply ignore the limit. This requires choosing values for the quantities over which the limit is taken. We replace $\mathcal{D}^{(N)}$ with $\mathcal{D}$ and set $d_\text{MF} = 1$. We use a batch size of 250 and set $\vec{\epsilon}$ small enough so that its floating-point representation is zero, as usual. This turns the left-hand side into metrics of fixed width networks under the framework of section <ref> as we computed them throughout this work. The metrics we compute are given in table <ref>.

See table <ref>.

Each metric corresponds to one of the statements from theorem <ref>. In table <ref>, we include additional square roots. Taking the root on both sides of a theorem statement does not alter its validity, but makes the quantities a bit more intuitive. (In general, we prefer means over squares of means.)

Computing mean field metrics To compute values for the right-hand side of the theorem statements, we have to use table <ref>. A few issues arise. (i) Our datasets are not exactly elem-like and thus do not come with $q$ and $c$ parameters. Because our datasets are normalized to have component means of 0 and an average neuron variance of 1 across the union of training and validation set, we set $q=1$ and $c=0$ in table <ref>. See also section <ref>. (ii) As in section <ref>, we want to conduct experiments in the initial state, when weight matrices are not Gaussian initialized, and in the final state. Hence, we do not generally have explicit access to the variance parameters as strictly defined in section <ref>. Hence, we replace $\sigma_l^2$ in table <ref> with its finite-width estimate, i.e. $\frac{1}{d_l}||W_l||^2_F$ for FC layers and $\frac{1}{d_l}||\beta_l||^2_2$ for bias layers. For Gaussian initialized bias and especially FC layers, this estimate would be very close to $\sigma_l^2$ for practical widths, due to the law of large numbers.

The `mean field metrics' $\hat{\mathfrak{m}}_l(f,\theta,q,c)$, $\hat{\mathfrak{q}}_l(f,\theta,q,c)$, $\hat{\mathfrak{c}}_l(f,\theta,q,c)$ and $\hat{\mathfrak{g}}_l(f,\theta,q,c)$ of an architecture $f$, parameter value $\theta$ and scalars $q > c \ge 0$ are calculated via table <ref>, where we replace $\sigma_l^2$ with $\frac{1}{d_l}||W_l||^2_F$ for FC layers and $\frac{1}{d_l}||\beta_l||^2_2$ for bias layers. For all these metrics, the default value of $q$ is 1 and the default value of $c$ is zero. $f$ must be composed of the layer operations in table <ref> for the metrics to be valid. Finally, using the same function arguments on both sides, we write

$$\hat{\mathfrak{n}}_{l,m}(f,\theta,q,c) = \sqrt{\frac{\hat{\mathfrak{g}}_l(\hat{\mathfrak{q}}_m - \hat{\mathfrak{c}}_m)}{\hat{\mathfrak{g}}_m(\hat{\mathfrak{q}}_l - \hat{\mathfrak{c}}_l)}}$$

Mean field metrics are different from the metrics we defined previously in this work. They depend not on the network $f$ via function evaluation, but on the architecture $f$ via recursive calculation rules that utilize the architecture definition. Computing these metrics poses somewhat different challenges. Instead of statistical estimation, we have to use numerical integration for the Gaussian expectation inside $\mathfrak{C}_\tau$. While this works well enough for a single evaluation of $\mathfrak{C}_\tau$, it can create problems when we iterate many times for a deep network. Under Gaussian instability, even 64-bit floating-point rounding errors can grow to the point of overflow, though this does not happen in our experiments. We further discuss this in section <ref>. See also section <ref>.

Mean field theory predicts finite width metrics We plot the finite width metrics from table <ref> vs their estimate based on mean field metrics in figures <ref>, <ref>, <ref>, <ref>, <ref>, <ref> and <ref>. In all cases, we depict the estimate on the x-axis and the finite width metric on the y-axis. Each figure corresponds to one finite width metric. Note that we do not present the value of QMNEX explicitly, but via the ratio $\frac{QMNEX}{LSCALE}$. This is because computing the neuron expectation incurs an estimation error proportional to the neuron standard deviation. Therefore when $QMNEX << QMNSTD$ we cannot hope to compute QMNEX accurately in absolute terms, but only relative to QMNSTD or LSCALE.

We find that all mean field estimates are highly accurate when three conditions hold. (i) The architecture is in the initial state, (ii) we use CIFAR-10 or MNIST and (iii) the architecture is stable, i.e. depicted in black. Consider especially figure <ref>(A-B). There, we plot the network Jacobian Frobenius norm along with a confidence interval of +/-2 standard deviations. We find that even at +/-2 standard deviations, JACF is very close to the mean field estimate, even though the range of JACF values obtained for different architectures spans 30 orders of magnitude! Consider also figure <ref>(A-B).  We find that mean field theory predicts the NLC of stable architectures highly accurately in the initial state, and still reasonably accurately for GEAs. It only fails for GUAs,  which do not use popular activation functions. This means that we can use table <ref> as a neural architecture design guide. This is the central insight of this chapter.

Let's look at the three factors we mentioned above that cause the degradation of mean field predictiveness. (i) We find that estimates for waveform-noise are somewhat less accurate than estimates for the other datasets for most metrics, including the NLC. This is likely explained by the lower input and output dimensionality, which caused irregular behavior in e.g. section <ref>, and / or the lower dataset size which induced noise e.g. in section <ref>. (ii) As usual, GUAs and GEAs are less well-behaved. Of course, in table <ref> the limits of activation layers model the neurons in the dependency as Gaussian. Therefore, if they are less Gaussian in practice, as we showed in section <ref>, we can expect the theory to be less predictive. GEAs, depicted in red, are still estimated relatively accurately in general. Note, however, that GEAs have some of the biggest confidence intervals in figures <ref> and <ref>. (iii) As usual, architectures are not as well-behaved after training. However, it is worth noting that the accuracy of mean field estimates after training does differ significantly from metric to metric. Specifically, LQM (figure <ref>) and LSCALE (figure <ref>) are still estimated quite well. We conjecture that this is because the architecture does not have an incentive to modify layer quadratic means during training. The architecture does have an incentive to attain a low NLC after training for reasons discussed in sections <ref> and <ref>. This conjecture is supported by figure <ref>, where we find that mean field theory overestimates the NLC after training. More generally, we find that gradient-related metrics (figures <ref>, <ref>) are overestimated and QMNSTD (figure <ref>), which is the denominator of the NLC, is underestimated.

We include correlation values in the figures. As usual, the caveat applies that the exact correlation values are significantly influenced by the frequency of GUAs and GEAs among architectures, as well as by only considering architectures that attained a non-random validation error after training in the bottom row of each figure.

The mean field estimate of the NLC usually stays relatively constant during training, but sometimes changes drastically In figure <ref>(A-C), we plot the mean field estimate of the NLC before vs after training. We find that the value changes very little for most architectures. Of course, the only way in which training can impact the mean field estimate is via a change of weight matrix and / or bias vector magnitude.

The `parameter growth' (PARMGROWTH) of a final parameter value $\theta^{(T)}$ relative to an initial parameter value $\theta^{(0)}$ is

$$PARMGROWTH(\theta^{(0)}, \theta^{(T)}) = \frac{||\theta^{(T)}||_2}{||\theta^{(0)}||_2}$$

In figure <ref>(D-F), we plot PARMGROWTH. We find that the parameter length never decreases more than a tiny amount, and it stays approximately constant for a majority of architectures. However, for some architectures it increases by orders of magnitude. In figure <ref>(G-I), we confirm that significant changes to the mean field estimate of the NLC only happen when the parameter length undergoes a large relative change, except for a very small number of architectures. Even when the parameter length changes drastically, the mean field estimate often does not. It is worth noting that the mean field estimate of the NLC is actually independent of the parameter in study A architectures when batch normalization is used. Also, PARMGROWTH is highly dependent on the learning rate. While we chose the starting learning rate in a very systematic fashion (section <ref>), there were sometimes wide ranges of starting learning rates that yielded very similar performance, such that choosing from within this range was essentially random. However, this choice still affects PARMGROWTH enormously.

Overall, the inaccuracy of the mean field estimate of the NLC after training is not caused by a change in the value of the estimate, as it is approximately constant for most architectures.

LQM vs its mean field estimate for study A architectures. The metrics are evaluated at a fully-connected or addition layer halfway through the network, as in figures in section <ref>, because of the narrowness of the output layer. Vertical lines correspond to confidence intervals of 2 standard deviations. GUAs are depicted as blue points with green lines. GEAs are depicted as red points with red lines. Both GUAs and GEAs are displayed in the foreground as throughout chapter <ref>, i.e. their markers fully or partially occlude the markers of stable architectures when they overlap. Some graph diagonals are given in black as a visual aid. Note that all x- and y-axis ranges of graphs across this figure, as well as across each of the figures below, are identical to enable easier comparison. Conclusion: The mean field estimate is highly accurate for stable architectures and GEAs before training and still relatively accurate after training.

JACF vs its mean field estimate at an intermediate and at the input layer. Graphs are analogous to figure <ref>. Conclusion: The mean field estimate is highly accurate for stable architectures and GEAs before training, but not after training.

LSCALE vs its mean field estimate at an intermediate layer. GUAs are depicted in green. GEAs are depicted in red. Both are displayed in the foreground. There are no confidence intervals. Graphs are otherwise analogous to previous figures. Conclusion: The mean field estimate is highly accurate for stable architectures and almost all GEAs before training, and still relatively accurate after training.

$\frac{QMNEX}{LSCALE}$ vs its mean field estimate at an intermediate layer. Graphs are analogous to previous figures. Conclusion: The mean field estimate is highly accurate for stable architectures and fairly accurate for GEAs before training, but not after training.

QMNSTD vs its mean field estimate at an intermediate layer and at the output later. Graphs are analogous to previous figures. Conclusion: The mean field estimate is highly accurate for stable architectures and almost all GEAs before training. After training, the range of QMNSTD values is small, so it is difficult to assess accuracy.

NLCNUM vs its mean field estimate, for the whole network and for the second half of the network. Graphs are analogous to previous figures. Conclusion: The mean field estimate is highly accurate for stable architectures and GEAs before training. After training, the mean field estimate tends to overestimate NLCNUM, but is still highly correlated with it.

NLC vs its mean field estimate, for the whole network and for the second half of the network. Graphs are analogous to previous figures. Conclusion: The mean field estimate is highly accurate for stable architectures and GEAs before training. After training, the mean field estimate usually overestimates the NLC, but it is still highly correlated with it.

Metric values for study A architectures. In graphs A-C, we plot the initial vs final $\tilde{\mathfrak{n}}$. In graphs D-F, we plot PARMGROWTH for study A architectures, sorted on the x-axis in ascending order. In graphs G-I, we plot the ratio of initial over final $\tilde{\mathfrak{n}}$. Architectures are placed on the x-axis in the same order as in graphs D-F. Conclusion: $\tilde{\mathfrak{n}}$ is stable for most architectures, but changes drastically for some. This change is generally associated with significant PARMGROWTH.

§.§ The mean field NLC and nonlinearity path equation

While each recursion rule in table <ref> makes sense by itself, taken together they still appear somewhat “black-box”, i.e. it is not clear whether there is a simple relationship between specific qualitative architecture properties, such as depth, activation functions or skip connections, and the mean field limits of important metrics. In this section, we show that we can indeed express the mean field estimate of the NLC in an incredibly instructive way. This allows us to ultimately explain the nonlinearity of an architecture using mean field theory and the architecture definition.

The `mean field NLC' $\mathfrak{n}_{l,m}$ of an A-architecture $f$ and scalars $q > c \ge 0$ is

$$\mathfrak{n}_{l,m}(f,q,c) = \sqrt{\frac{\mathfrak{g}_l(\mathfrak{q}_m - \mathfrak{c}_m)}{\mathfrak{g}_m(\mathfrak{q}_l - \mathfrak{c}_l)}}$$

where the limit quantities on the right-hand side are calculated according to table <ref>. The variance parameters are considered part of the architecture definition as usual. We write $\mathfrak{n}$ short for $\mathfrak{n}_{L,0}$ for the “mean field NLC of $f$”.

Going forward, we will use the term `mean field NLC' interchangeably for $\mathfrak{n}_{l,m}$ and $\hat{\mathfrak{n}}_{l,m}$, as defined in the previous subsection, as both have nearly identical values for practical A-architectures in the initial state without severe Gaussian instability.

We can immediately observe that $\mathfrak{n}_{l,m}$ is decomposable in the sense of section <ref>, i.e. if we break down the section of the architecture between $f_m$ and $f_l$ into a series of segments where the endpoints of those segments are bottlenecks, $\mathfrak{n}_{l,m}$ is the product of mean field NLCs of these segments. Specifically, given a chain of layers in which each has a single dependency, the mean field NLC of that chain is equal to the product of mean field NLCs of each individual layer.

We also observe that the mean field NLC of individual layers has a simple form. For FC, bias, LN and BN layers, it is equal to 1. For addition layers with a single dependency, it is equal to 1. This is expected for operations that are linear functions of the dependency, like FC, bias and addition, because of proposition <ref>. Informally, we can say that LN and BN layers are approximately linear for large width and batch sizes. To see this, consider that subtracting the mean of a vector is linear. Dividing by the standard deviation projects the values of the dependency onto a lower-dimensional space which is curved in only a single dimension per neuron (BN) / input (LN). Therefore, the nonlinearity of that space becomes increasingly negligible as width / batch size increases.

If $f_l$ is an activation layer, we have

$$\mathfrak{n}_{l,k} = \sqrt{\frac{(\mathfrak{q}_k - \mathfrak{c}_k)\mathfrak{C}_{\tau_l'}(\mathfrak{q}_k,\mathfrak{q}_k)}{\mathfrak{C}_{\tau_l}(\mathfrak{q}_k,\mathfrak{q}_k) - \mathfrak{C}_{\tau_l}(\mathfrak{q}_k,\mathfrak{c}_k)}}$$

Hence, we define the following.

The `activation function NLC' of an activation function $\tau$ and scalars $q > c \ge 0$ is

$$\mathfrak{n}_\tau(q,c) =  \sqrt{\frac{(q - c)\mathfrak{C}_{\tau'}(q,q)}{\mathfrak{C}_{\tau}(q,q) - \mathfrak{C}_{\tau}(q,c)}}$$

We write $\mathfrak{n}_\tau(c)$ short for $\mathfrak{n}_\tau(1,c)$. When $c=0$, the formula simplifies to

$$\mathfrak{n}_\tau(q,0) =  \sqrt{\frac{q\mathbb{E}_{s\sim\mathcal{N}(0,q)}\tau'(s)^2}{\mathbb{E}_{s\sim\mathcal{N}(0,q)}\tau(s)^2 - (\mathbb{E}_{s\sim\mathcal{N}(0,q)}\tau(s))^2}}$$

We will investigate $\mathfrak{n}_\tau$ in detail in section <ref>. For now, note that $\mathfrak{n}_\tau$ is valid for any $q > c > -q$ for any differentiable and non-constant $\tau$ due to propositions <ref> and <ref>. Note also that the $q$ and $c$ arguments in $\mathfrak{n}_\tau(q,c)$ are somewhat different from those in $\mathfrak{n}_{m,l}(f,q,c)$ in that they capture the properties of the input to $\tau$, instead of the properties of $\mathcal{D}$.

In A-architectures, the only layer operation that can have multiple dependencies is the addition operation. In other words, if $p$ and $p'$ are two directed paths through the layer graph, then those paths can only join at an addition layer.

Define the `path-weight' $w(p)$ of a directed path in the layer graph of an A-architecture as follows. For each addition layer contained in the path, consider the ratio $\frac{(\mathfrak{q}_k-\mathfrak{c}_k)w^2}{\mathfrak{q}_l-\mathfrak{c}_l}$, where $f_l$ is the addition layer, $f_k$ is the dependency contained in the path, and $w$ is the addition weight corresponding to $f_k$. $w(p)$ is then the product of all such ratios for addition layers contained in the path that are not its starting point.

Considering the calculation rule for the $\mathfrak{q}$ and $\mathfrak{c}$ values of addition layers in table <ref>, it is easy to see that the sum of path weights over all directed paths from input to output layer in a given layer graph equals 1. Informally, if we consider $\mathfrak{q}_L - \mathfrak{c}_L$ as the variance of the network output, the path weight indicates the fraction of that variance that is contributed by that particular path. The path representation now enables the following proposition, which can be proven simply using induction.

Let $\mathfrak{q}_l$, $\mathfrak{c}_l$ and $\mathfrak{g}_l$ be defined according to table <ref>. Let $0 \le m \le l \le L$ and let $f_m$ be a bottleneck for $f_l$ in the A-architecture $f$. Then

$$\mathfrak{n}_{l,m}(f,q,c) = \sqrt{\sum_{p \in P_{m,l}} w(p) \prod_{f_{l'}\in p} \mathfrak{n}_{\tau_{l'}}(\mathfrak{q}_{k'},\mathfrak{c}_{k'})^2}$$

where $P_{m,l}$ is the set of directed paths from $f_m$ to $f_l$, the product is over all activation layers $f_{l'}$ on path $p$ excluding its starting point and $f_{k'}$ is the dependency of $f_{l'}$.

When $P_{m,l}$ only contains a single path / dependency chain $p_{m,l}$, this simplifies to

$$\mathfrak{n}_{l,m}(f,q,c) = \prod_{f_{l'}\in p_{m,l}} \mathfrak{n}_{\tau_{l'}}(\mathfrak{q}_{k'},\mathfrak{c}_{k'})$$

In plain words, the mean field NLC of an A-architecture is the weighted quadratic mean of the mean field NLC of each directed path in the layer graph, where each path is weighted according to the fraction of the output variance it contributes. The mean field NLC of a path is the product of the activation function NLCs on that path.

This is the instructive relationship between $\mathfrak{n}$ and the architecture definition we have been looking for. It allows us to make manual design decisions while controlling the initial NLC. We will use the nonlinearity path equation throughout the remainder of this work as a reference point for compiling design insights. For now, we will give some first observations.

* $\mathfrak{n}$ is independent of width as long as the $\sigma_l^2$ are fixed.

* $\mathfrak{n}$ decreases with the introduction of linear skip connections, proportionally with the square of the skip connection strength, assuming there are no knock-on effects on downstream $\mathfrak{q}$ and $\mathfrak{c}$ values.

* $\mathfrak{n}$ does not directly depend on normalization layers used.

* Replacing an activation function with another of higher $\mathfrak{n}_{\tau}$ value increases $\mathfrak{n}$, assuming there are no knock-on effects on downstream $\mathfrak{q}$ and $\mathfrak{c}$ values.

* Composition of activation functions corresponds to multiplying nonlinearity, a principle that is familiar from the chain rule of calculus, the exploding / vanishing gradient problem and the notion of neural network expressivity.

§ MEAN FIELD THEORY OF ACTIVATION FUNCTIONS

The key properties of an activation function from a mean field perspective are $\mathfrak{C}_\tau$, $\mathfrak{C}_{\tau'}$ and the derived quantity $\mathfrak{n}_\tau$. In this section, we study these constructs as activation function metrics. We will establish a plethora of useful properties that will be utilized later in this work. This effort culminates in subsection <ref>, where we show that the mean field NLC is the first-order approximation of the bandwidth of the covariance kernel. This is the most important of our results that establish the NLC as a primary measure of model complexity in deep learning.

$\mathfrak{C}_\tau(q,c)$ and $\mathfrak{n}_\tau(q,c)$ are defined as in the beginning of section <ref> and section <ref> respectively.

As stated in section <ref>, $\mathfrak{n}_\tau$ is valid for any $q > c > -q$ and differentiable, non-constant $\tau$. We will implicitly assume this in our discussion, but not in our theoretical results. In practical terms, activation function metrics are computed using numerical integration techniques as described in section <ref>. The discussion of section <ref> applies to the issue of non-differentiable activation functions such as ReLU. However, in this section we do also give results for $\tau$ that are only directionally differentiable, so that we have at least some theory in this work that explicitly includes this case.

§.§ The NLC of activation functions with meta-Gaussian input

We begin by showing that $\mathfrak{n}_\tau$ is not just an abstract quantity arising from table <ref>, but a direct measure of activation function nonlinearity.

Let the distribution $\mathcal{T}$ over vectors $\chi$ of dimensionality $d_\tau$ be drawn from an elementwise meta-distribution with generator $\mathcal{MN}(q,c)$, $q > c \ge 0$. Then

$$NLC(\tau,\mathcal{T}) = \sqrt{\frac{(q-c)\mathbb{E}_{i,s\sim\mathcal{N}(0,q-c)}\tau'(\bar{\chi}[i] + s)^2}{\mathbb{E}_{i,s\sim\mathcal{N}(0,q-c)}\tau(\bar{\chi}[i] + s)^2 - \mathbb{E}_i(\mathbb{E}_{s\sim\mathcal{N}(0,q-c)}\tau(\bar{\chi}[i] + s))^2}}$$

Here, the component means $\bar{\chi}[i]$ are drawn on the meta-level from a Gaussian with mean zero and variance $c$, and $\mathbb{E}_i$ is a finite mean over $d_\tau$ values. The term $NLC(\tau,\mathcal{T})$ implies that the scalar function $\tau$ is applied elementwise to vectors drawn from $\mathcal{T}$. Given that the sample mean converges almost surely to the distribution expectation, we obtain

$$\lim_{d_\tau \rightarrow \infty}NLC(\tau,\mathcal{T}) = \sqrt{\frac{(q-c)\mathbb{E}_{s\sim\mathcal{N}(0,q)}\tau'(s)^2}{\mathbb{E}_{s\sim\mathcal{N}(0,q)}\tau(s)^2 - \mathbb{E}_{t\sim\mathcal{N}(0,c)}(\mathbb{E}_{s\sim\mathcal{N}(0,q-c)}\tau(t + s))^2}} \text{ a.s.}$$

We can cast the Gaussian expectations in the above formula as a metric of $\tau$, $q$ and $c$, like we did with the covariance kernel.

The `bias kernel' $\mathfrak{B}_\tau$ of an activation function $\tau$ and scalars $q \ge c \ge 0$ is
$$\mathfrak{B}_\tau(q,c) = \mathbb{E}_{s\sim\mathcal{N}(0,c)}(\mathbb{E}_{t\sim\mathcal{N}(0,q-c)}\tau(s+t))^2$$

Then we obtain

$$\lim_{d_\tau \rightarrow \infty}NLC(\tau,\mathcal{T}) = \sqrt{\frac{(q-c)\mathfrak{B}_{\tau'}(q,q)}{\mathfrak{B}_{\tau}(q,q) - \mathfrak{B}_{\tau}(q,c)}} \text{ a.s.}$$

The right-hand side is exactly $\mathfrak{n}_\tau(q,c)$, except that the covariance kernel is replaced by the bias kernel. We close the loop with the following result.

For any $\tau$ and $q \ge c \ge 0$, we have
$$\mathfrak{B}_\tau(q,c) = \mathfrak{C}_\tau(q,c)$$

Hence, we have

$$\lim_{d_\tau \rightarrow \infty}NLC(\tau,\mathcal{T}) = \mathfrak{n}_\tau(q,c) \text{ a.s.}$$

So the activation function NLC is the limit of the NLC under meta-Gaussian input. Of course, this is not unexpected given theorem <ref>, which establishes that fully-connected layers are meta-Gaussian meta-distributed. Proposition <ref> also shows that these meta-Gaussians lead to the square means and co-means that arise in table <ref>. The above result also justifies the name `activation function NLC'. Finally, we obtain that the nonlinearity path equation relates the mean field NLC of the architecture to the actual nonlinearity of its activation functions.

§.§ The length kernel of activation functions

The term $\mathfrak{C}_\tau(q,q)$ has appeared repeatedly in this chapter. In this subsection, we investigate it further. We begin by giving it its own definition.

The `length kernel' $\mathfrak{L}_\tau$ of an activation function $\tau$ and scalar $\lambda \ge 0$ is
$$\mathfrak{L}_\tau(\lambda) = \sqrt{\mathfrak{C}_\tau(\lambda^2,\lambda^2)} = \sqrt{\mathbb{E}_{s\sim\mathcal{N}(0,\lambda^2)}\tau(s)^2}$$

The length kernel represents the infinite width limit of the quadratic mean of an activation layer when its dependency is distributed according to $\mathcal{N}(0,\lambda^2 I)$. We believe that the quadratic mean is a more intuitive quantity to deal with than the square mean. Hence, we define $\mathfrak{l}_l = \sqrt{\mathfrak{q}_l}$. Table <ref> then yields $\mathfrak{l}_l = \mathfrak{L}_{\tau_l}(\mathfrak{l}_k)$ for activation layers $f_l$. Since a deep network generally composes many activation layers, $\mathfrak{l}_L$ depends on $\mathfrak{l}_0$ via a sequence of length kernel applications.

We say a `plain architecture' is an architecture composed of $M$ macro-layers, which are themselves composed of an FC and an activation layer. Each activation layer uses the same activation function $\tau$ and each FC layer is Gaussian initialized with the same fixed multiple $\sigma^2$ of the LeCun variance.

A plain architecture is a fixed width A-architecture, except for the technical conditions on the activation function. Hence, for the purpose of discussion, we apply theorem <ref> and its corollaries, in addition to the results of section <ref>, to plain architectures.

In a plain architecture with $\sigma=1$, we have $\mathfrak{l}_L = \mathfrak{L}^M_\tau(\mathfrak{l}_0)$, where the exponent indicates composition. Of course, not all architectures we care about are this simple. However, this example motivates us to study the length kernel under the umbrella of function iteration. In a nutshell, because of the ZSAD guideline of scale stability given in section <ref>, we desire the iteration to converge to a non-zero value, which is ideally close to 1. We will further discuss this point in detail throughout chapter <ref>.

Below, we give the main theorem of this subsection, which gives a range of properties for the length kernel. We follow this up with an empirical study.

For any $\tau$ we have

* If $\mathfrak{L}_\tau(\lambda) \neq 0$ for some $\lambda > 0$, then for all $\lambda > 0$ we have

(a)$\mathfrak{L}_\tau(\lambda) > 0$

(b) $\mathfrak{L}_\tau$ is differentiable at $\lambda$ with $\mathfrak{L}_\tau'(\lambda) = \frac{1}{2}\mathfrak{L}_\tau(\lambda)^{-1}\mathbb{E}_{s \sim \mathcal{N}(0, \lambda^2)}\tau(s)^2(s^2\lambda^{-3}-\lambda^{-1})$

(c) $\frac{\mathfrak{L}_\tau'(\lambda)\lambda}{\mathfrak{L}_\tau(\lambda)} > -\frac{1}{2}$

(d) $(\sqrt{\lambda}\mathfrak{L}_\tau(\lambda))' > 0$

* If $\tau$ is continuous and not zero everywhere, then $\mathfrak{L}_\tau(\lambda) > 0$ for all $\lambda > 0$.

* If $\tau$ is continuous at zero, $\mathfrak{L}_\tau$ is continuous at zero.

* If $\tau$ is directionally differentiable at zero, then $\mathfrak{L}_\tau$ is differentiable at zero with $\mathfrak{L}_\tau'(0) = \sqrt{\frac{\tau^-(0)^2 + \tau^+(0)^2}{2}}$ if $\tau(0) = 0$ and $\mathfrak{L}_\tau'(0) = \frac{1}{\sqrt{2\pi}}\sign(\tau(0))(\tau^+(0)-\tau^-(0))$ if $\tau(0) \neq 0$. The + and - superscripts indicate the right and left derivative respectively.

* If $\mathfrak{L}_\tau(\lambda) \neq 0$ for some $\lambda > 0$, then for all $\lambda > 0$ the sequence $(\mathfrak{L}_\tau^n(\lambda))_n$ either (i) is strictly increasing and diverges to infinity or (ii) converges.

The highlights are that under very mild conditions $\mathfrak{L}_\tau$ is differentiable, iterating $\mathfrak{L}_\tau$ yields convergence or strictly increasing divergence, and statement 1(c) yields that $\mathfrak{L}_\tau$ has no unstable fixed points where it is decreasing. Going forward, we can think of the iteration convergence behavior of $\mathfrak{L}_\tau$ like that of a general differentiable function.

In tables <ref>, <ref>, <ref> and <ref>, we depict the length kernel for 24 activation functions: the 12 activation functions contained in tables <ref>, <ref> and <ref>, as well as the debiased versions of those activation functions. Debiasing in the context of activation functions, as described in e.g. section <ref>, corresponds to subtracting from the activation function its expectation under the unit Gaussian distribution, i.e. $\mathbb{E}_{s\sim \mathcal{N}(0,1)}\tau(s)$. This strategy was used for generating many architectures for both study A and study B. Note that tanh, odd square and sawtooth are equal to their debiased versions, and SeLU is almost equal to it.

We plot length kernels in three colors: blue, red and magenta. When we iterate $\mathfrak{L}_\tau$ starting from a point in a blue segment, the iteration converges to a non-zero point. When we iterate $\mathfrak{L}_\tau$ starting from a point in a red segment, the iteration either converges to zero or diverges to infinity. When we iterate $\mathfrak{L}_\tau$ starting from a point in a magenta segment, the iteration is stationary.

We find that we obtain a range of different convergence behaviors for different activation functions.

* For many activation functions, all starting values of $\lambda$ converge to the same non-zero value. Their length kernels are depicted in all blue and the limit is a bold blue point.

* Also for many activation functions, all starting values of $\lambda$ converge to zero. Their length kernels are depicted in all red and lie below the diagonal, which is depicted in black as a visual aid.

* For some activation functions, small values of $\lambda$ converge to zero whereas large values diverge. Their length kernels are depicted in red, and the curve crosses the diagonal.

* For one activation function (abs. val.), $\mathfrak{L}_\tau$ is the identity, so $\lambda$ is preserved. Its length kernel is depicted in magenta.

* Finally, for one activation function (square-debiased), all values of $\lambda$ diverge. Its length kernel is depicted in red and above the diagonal.

Similarly, we find a range of values for the convergence rate. The majority of the time, convergence is exponential. This occurs when the derivative of $\mathfrak{L}_\tau$ is neither 0, 1 or -1 at the limit point. If the derivative is 0, then the convergence is superexponential. This happens for the square and odd square activation functions. In fact, convergence is square exponential. If the derivative is either 1 or -1, convergence is sub-exponential. For tanh and even tanh, the rate is $O(\frac{1}{M^{0.5}})$, and for sawtooth the rate is sub-polynomial, i.e. slower than any $O(\frac{1}{M^\epsilon})$ with $\epsilon > 0$. In tables <ref> through <ref>, we specify convergence rates as precisely as we were able to determine them with simple methods.

In tables <ref> through <ref>, we also give the normalized length kernel $\frac{\mathfrak{L}_\tau(\lambda)}{\mathfrak{L}_\tau(1)}$. This is the length kernel we would obtain when normalizing the activation function by its quadratic mean under the unit Gaussian, i.e. $\sqrt{\mathbb{E}_{s\sim \mathcal{N}(0,1)}\tau(s)^2}$. This normalization is done for both study A and study B architectures. Hence, we are interested in normalized length kernels, which are themselves valid length kernels, as is any positive multiple of a valid length kernel. The range of convergence behaviors observed among our normalized length kernels is largely similar to the range we observed previously, except that there are now several activation functions that have both $\lambda$ values that converge to a non-zero value (depicted in blue) as well as $\lambda$ values that diverge (depicted in red).

While we have observed a wide range of convergence behaviors in tables <ref> through <ref>, through deliberate design of activation functions, we could generate an even more diverse set of behaviors, though this goes beyond the scope of this work. While length kernels depicted have at most one stable and one unstable fixed point, it is easy to generate activation functions that have an infinite number of them. When $\tau(s)^2$ oscillates between small and large values, so can $\mathfrak{L}_\tau$.

In addition to studying the length kernel normalized by its value at 1, i.e. $\frac{\mathfrak{L}_\tau(\lambda)}{\mathfrak{L}_\tau(1)}$, we can study the family of all normalizations of form $\frac{\mathfrak{L}_\tau(\lambda)\lambda'}{\mathfrak{L}_\tau(\lambda')}$ for some fixed $\lambda' > 0$. $\frac{\mathfrak{L}_\tau(\lambda)\lambda'}{\mathfrak{L}_\tau(\lambda')}$ has a fixed point at $\lambda = \lambda'$ with derivative $\frac{\mathfrak{L}_\tau'(\lambda)\lambda}{\mathfrak{L}_\tau(\lambda)}$. We plot $\frac{\mathfrak{L}_\tau'(\lambda)\lambda}{\mathfrak{L}_\tau(\lambda)}$ as a function of $\lambda$ in tables <ref> through <ref>. A value between -0.5 and 1 indicates stability (depicted in blue), a value above 1 indicates instability (depicted in red) and a value of 1 indicates stationary behavior (depicted in magenta). Note that statement 1(c) of theorem <ref> implies that this curve cannot dip below -0.5. For the majority of activation functions, we obtain the same regime for all values of $\lambda$, though for some we obtain stability specifically for small values of $\lambda$.

§.§ The covariance kernel of activation functions

When calculating $\mathfrak{c}_L$ according to table <ref>, we repeatedly apply $\mathfrak{C}_\tau(q,c)$. So just as with the length kernel, we are interested in function iteration.

We say an architecture is `plain stable$(q,q')$' if it is a plain architecture where $\sigma^2q=q'$ and $\mathfrak{L}_\tau(\sqrt{q'}) = \sqrt{q}$.

In a plain stable$(q,1)$ architecture, when the input distribution is elem-like$(q,c)$, we have $\mathfrak{c}_L = q\tilde{\mathfrak{C}}_\tau^M(1,c)$, where the exponent indicates composition and $\tilde{\mathfrak{C}}_\tau$ denotes $\frac{\mathfrak{C}_\tau}{\mathfrak{L}_\tau(1)^2}$. While this setup might appear specific, it describes networks fulfilling the scale stability ZSAD guideline from section <ref> in a mean field sense. This setup was used approximately for both study A and B and was the focus of prior work [Poole et al., 2016, Schoenholz et al., 2017, Jacot et al., 2018, Lee et al., 2019]. As previously mentioned, we shorten $\mathfrak{C}_\tau(1,c)$ to $\mathfrak{C}_\tau(c)$. While we mostly focus on the case $q'=1$ in this and the next subsection, we can easily obtain equivalent results for any other value of $q'$. Again, we will further use the results from this subsection in chapter <ref>.

It turns out that $\mathfrak{C}_\tau(c)$ is more regular than $\mathfrak{L}_\tau(\lambda)$. After a definition, we give our main theorem of this subsection, followed by an empirical study.

We say a function $F : \mathbb{R} \rightarrow \mathbb{R}$ is `piecewise $n$-differentiable' if there exists a partition of $\mathbb{R}$ into a finite set of intervals such that (i) $F$ is $n$ times differentiable in the interior of each interval and (ii) $F$ is continuous and $n$ times directionally differentiable everywhere. In particular, the left and right derivative at each interval endpoint does not have to be the same.

All of our activation functions in table <ref>, except sawtooth, are piecewise $n$-differentiable for any $n$. Sawtooth is still piecewise $n$-differentiable within any bounded interval.

Assume $\tau$ is piecewise 5-differentiable. Consider $\mathfrak{C}_\tau(c)$ defined on $[0,1]$. Let $a_\tau s + b_\tau$ be the least squares linear fit to $\tau$ under $\mathcal{N}(0,1)$ and let $\tilde{\tau} = \tau - a_\tau s - b_\tau$. Then

* $\mathfrak{C}_\tau$ is differentiable and $\mathfrak{C}_\tau' = \mathfrak{C}_{\tau'}$.

* $\mathfrak{C}_\tau$ is increasing, convex and for all $c > 0$, $\epsilon > 0$ with $c + 3\epsilon \le 1$ we have $\mathfrak{C}_\tau(c+3\epsilon) - 3\mathfrak{C}_\tau(c+2\epsilon) + 3\mathfrak{C}_\tau(c+\epsilon) - \mathfrak{C}_\tau(c)\ge 0$.

* (i) $\mathfrak{C}_\tau(c) = \mathfrak{C}_{\tilde{\tau}}(c) + a_\tau^2c + b_\tau^2$, (ii) $\mathfrak{C}_{\tilde{\tau}}(0) = 0$, (iii) $\mathfrak{C}_{\tau}(0) = b_\tau^2$, (iv) $\mathfrak{C}_{\tilde{\tau}}'(0) = 0$, (v) $\mathfrak{C}_{\tau}'(0) = a_\tau^2$, (vi) $CAR(\tau, \mathcal{N}(0,1)) = \frac{\mathfrak{C}_{\tau}(0)}{\mathfrak{C}_{\tau}(1)}$, (vii) $LAR(\tau, \mathcal{N}(0,1)) = \frac{\mathfrak{C}_{\tau}'(0)}{\mathfrak{C}_{\tau}(1)}$ and (viii) $\mathfrak{C}_{\tau}$ is linear if and only if $\tau$ is linear.

Now also assume $\tau$ is not linear. Then, for the normalized covariance kernel $\tilde{\mathfrak{C}}_\tau(c) = \frac{\mathfrak{C}_\tau(c)}{\mathfrak{C}_\tau(1)}$ exactly one of three cases hold on $[0,1]$.

* There is a stable fixed point at $c^\text{lim}=1$ with exponential convergence rate and no other fixed point. $0 < \tilde{\mathfrak{C}}_\tau'(1) < 1$ holds.

* There is a stable fixed point at $c^\text{lim}=1$ with sub-exponential convergence rate and no other fixed point. $\tilde{\mathfrak{C}}_\tau'(1) = 1$ holds.

* There is an unstable fixed point at 1 and there is exactly one other fixed point $c^\text{lim}$ in $[0,1)$, which is stable. $\tilde{\mathfrak{C}}_\tau'(1) > 1$ and $1 > \tilde{\mathfrak{C}}_\tau'(c^\text{lim}) \ge 0$ hold. The convergence rate is super-exponential or exponential depending on whether $\tilde{\mathfrak{C}}_\tau'(c^\text{lim}) = 0$ holds.

In particular, there exists exactly one stable fixed point $c^\text{lim}$ and iterating $\tilde{\mathfrak{C}}_\tau$ will lead to convergence towards $c^\text{lim}$ from any starting point.

The third part of statement (b) is similar to saying that $\mathfrak{C}_{\tau}(c)$ has a non-negative third derivative. Note that if $\tau$ is smooth, we can apply statement (a) repeatedly to obtain that the $n$'th derivative of $\mathfrak{C}_{\tau}(c)$ is the covariance kernel of the $n$'th derivative. Hence, we immediately have a wide range of properties for the $n$'th derivative, including those in the above theorem as well as propositions <ref>, <ref> and <ref>.

While we do not give a detailed interpretation of the theorem here, we make use of the different parts in various places throughout this work. We note that, to a large degree, the convergence case breakdown in the second half of the theorem was previously observed empirically by Poole et al., 2016 for various architectures based on the tanh activation function. However, to our knowledge, it was never proven for general $\tau$.

In tables <ref>, <ref>, <ref> and <ref>, we plot the normalized covariance kernel $\tilde{\mathfrak{C}}_\tau(c)$ for our 24 activation function. Note that $\tilde{\mathfrak{C}}_\tau(c)$ is itself a valid covariance kernel of the activation function $\frac{\tau}{\mathfrak{L}_\tau(1)}$. Indeed, each of the three convergence cases from the theorem is observed.

The LAR, CAR and NAR metrics were defined in section <ref>. In table <ref>, we gave LAR, CAR and NAR values for our activation functions as well as the $\tilde{\tau}$ corresponding to our $\tau$. More generally, the analysis from section <ref> applies to activation functions. For example, we have $\mathfrak{n}_{\tilde{\tau}}(0) \ge \sqrt{2}$ and $\mathfrak{n}_{\tau}(0)^2 = \frac{LAR}{LAR + NAR} + \frac{NAR}{LAR + NAR}\mathfrak{n}_{\tilde{\tau}}(0)^2$.

§.§ The activation function NLC

Properties of $\mathfrak{n}_\tau(c)$ Using theorem <ref>, we straightforwardly obtain the following about $\mathfrak{n}_\tau(c)$.

Assume $\tau$ is piecewise 5-differentiable. Consider $\mathfrak{n}_\tau(c)$ defined on $[0, 1)$. Then

$$\mathfrak{n}_\tau(c) = \sqrt{\frac{\mathfrak{C}_\tau'(1)(1-c)}{\mathfrak{C}_\tau(1)-\mathfrak{C}_\tau(c)}} = \sqrt{\frac{\tilde{\mathfrak{C}}_\tau'(1)(1-c)}{1-\tilde{\mathfrak{C}}_\tau(c)}}$$

* $\mathfrak{n}_\tau$ is decreasing

* $\lim_{c \rightarrow 1}\mathfrak{n}_\tau(c) = 1$

* $\mathfrak{n}_\tau(c) \ge 1$

Hence, $\mathfrak{n}_\tau(c)$ is determined entirely by the covariance kernel of $\tau$. Note the contrast of statement 1 to statement (c) from theorem <ref>. It turns out that the nonlinearity of an activation function as measured by the linear approximation error is related to the derivative of the covariance kernel at 0. The nonlinearity of an activation function as measured by the NLC is related to the derivative of the covariance kernel at 1. Statement 2 makes sense in light of proposition <ref>. The smaller the neuron variance under the meta-Gaussian, the more linear $\tau$ becomes across its effective domain.

Iterating $\mathfrak{n}_\tau(c)$ For plain stable$(q,1)$ A-architectures, the nonlinearity path equation yields

$$\mathfrak{n} = \prod_{m=1}^M\mathfrak{n}_\tau(\tilde{\mathfrak{C}}_\tau^m(\mathfrak{c}_0))$$

We are interested in the behavior of $\mathfrak{n}$ as $M$ converges to infinity. By theorem <ref>, $\tilde{\mathfrak{C}}_\tau^m(\mathfrak{c}_0)$ converges to some value $c^\text{lim}$ as $m$ increases, where $c^\text{lim}$ is the single stable fixed point of $\tilde{\mathfrak{C}}_\tau$. If $c^\text{lim} \neq 1$, then clearly $\mathfrak{n}$ diverges exponentially with rate $\mathfrak{n}_\tau(c^\text{lim})$. This corresponds to case 3 in theorem <ref>. Two cases remain.

Case 2 from theorem <ref>: $\mathfrak{n}$ is a telescopic product that equals $\sqrt{\frac{1 - \mathfrak{c}_0}{1 - \tilde{\mathfrak{C}}^M_\tau(\mathfrak{c}_0)}}$. This also diverges to infinity, where the rate is related to the convergence rate of $\tilde{\mathfrak{C}}^M_\tau$ to 1. For ReLU and abs. val., for example, that rate is $O(\frac{1}{M^2})$, and therefore $\mathfrak{n}$ grows linearly with depth. If $\tilde{\mathfrak{C}}_\tau$ is twice differentiable at 1, that rate is $O(\frac{1}{M})$ and so $\mathfrak{n}$ grows as the square root. By theorem <ref>, $\tilde{\mathfrak{C}}_\tau$ is twice differentiable if $\tau$ is differentiable and $\tau'$ is piecewise 5-differentiable. This is fulfilled by neither ReLU nor abs. val.. Because $\lim_{c \rightarrow 1}\mathfrak{n}_\tau(c) = 1$, divergence is always sub-exponential.

Case 1 from theorem <ref>: The limit of $\mathfrak{n}$ also depends on the analytical properties of $\tilde{\mathfrak{C}}_\tau$ at 1. If $\tilde{\mathfrak{C}}_\tau$ is twice differentiable at 1, then it is easy to show that $\mathfrak{n}$ converges exponentially with depth. By theorem <ref>, this holds, for example, for softplus, sigmoid and Gaussian. These are the three activation functions in tables <ref> to <ref> that fall under case 1. Unfortunately, we do not know what can happen when $\tilde{\mathfrak{C}}_\tau$ is not twice differentiable.

Properties of $\mathfrak{n}_\tau(q, 0)$ We are also interested in how $\mathfrak{n}_\tau$ responds to changes in $q$, though for the opposite reason that we are interested in $\mathfrak{n}_\tau(1, c)$. While we generally design architectures to achieve $\mathfrak{q}_k \approx 1$ for dependencies of activation layers, we also want to understand what happens if this condition fails.

In tables <ref>, <ref>, <ref> and <ref>, we plot $\mathfrak{n}_\tau(\lambda^2, 0)$, where $\lambda = \sqrt{q}$ as in section <ref>. We find that different activation functions exhibit drastically different behavior. Some activation functions, such as ReLU, abs. val. and square, have a stable $\mathfrak{n}_\tau$, whereas other activation functions, such as even tanh and sawtooth, have a rapidly increasing one. Hence, different activation functions respond very differently to e.g. changes in the variance of weight matrix entries. Here are some general patterns.

* $\mathfrak{n}_\tau(\lambda^2, 0)$ is invariant to debiasing.

* SeLU, softplus and Swish are all ReLU-like for large $\lambda$ and so $\lim_{\lambda \rightarrow \infty}\mathfrak{n}_\tau(\lambda^2, 0) = \mathfrak{n}_\text{ReLU}(1, 0) = 1.21$.

* ReLU, abs. val., square and odd square have the property $\tau(\lambda s) = \lambda\tau(s)$ for arbitrary $s$ and $\lambda \ge 0$, so $\mathfrak{n}_\tau(\lambda^2, 0)$ is constant.

* Tanh and sigmoid “converge to the step function” so $\mathfrak{n}_\tau(\lambda^2, 0)$ diverges as $O(\sqrt{\lambda})$.

* Even tanh and Gaussian “converge to the delta function” so $\mathfrak{n}_\tau(\lambda^2, 0)$ diverges as $O(\lambda)$.

* Sawtooth is periodic so $\mathfrak{n}_\tau(\lambda^2, 0)$ diverges as $O(\lambda)$.

§.§ The covariance kernel of A-architectures

Now we bring things back to the architecture level. The covariance kernel of a mean field architecture with a single finite input and readout layer with respect to finite sets of inputs was defined in section <ref> in the context of background corollary <ref>. In the context of A-architectures and elem-like input distributions, we define analogously.

For an A-architecture $f$ without batch normalization, the `covariance kernel' $\mathfrak{C}(q,c)$, $q \ge c \ge 0$, is the function that returns $\mathfrak{c}_L$ defined according to table <ref>.

It is easy to check that $\mathfrak{q}_L = \mathfrak{C}(q,q)$ holds. To see that $\mathfrak{C}$ is valid for the case $q=c$, which was excluded in e.g. theorem <ref> and proposition <ref>, we note that each operation other than BN preserves $\mathfrak{q} \ge \mathfrak{c} \ge 0$ in table <ref>. For the activation operation, see propositions <ref> and <ref>.

Let $f$ be an A-architecture where each activation function used is piecewise 5-differentiable and let $q > c \ge 0$. If $f$ does not contain batch normalization but can contain layer normalization layers, we have

$$\mathfrak{n}(f,q,c) = \sqrt{\frac{\frac{d}{dq'}\mathfrak{C}(q,q')|_{q'=q}(q - c)}{\mathfrak{C}(q,q) - \mathfrak{C}(q,c)}}$$

If $f$ does not contain layer normalization but can contain batch normalization layers, an analogous statement holds. See section <ref> for details.

Like the nonlinearity path equation, this can be proven simply with induction. Crucially, we also use statement (a) from theorem <ref>. Like for theorem <ref>, we defer the BN case to section <ref>. This proposition explicitly requires that the activation functions used by $f$ are piecewise 5-differentiable. Further, because $f$ is an A-architecture, its activation functions are also required to be controlled by $\mathcal{C}^\text{E2}$, non-constant and twice differentiable with a derivative controlled by $\mathcal{C}^\text{E2}$.

We can interpret proposition <ref> just as we interpreted the NLC in section <ref> and gradient-based local linear approximability in section <ref>. If we view $\mathcal{D}$ as an elementwise distribution in the spirit of proposition <ref>, then the dimensionality-normalized radius of the domain $\sqrt{\frac{1}{d_\text{in}}\Tr(\Cov_x)}$ becomes $\sqrt{q - c}$. Similarly, if the codomain is generated by $\mathcal{MN}(\mathfrak{C}(q,q), \mathfrak{C}(q,c))$, then $\sqrt{\frac{1}{d_\text{out}}\Tr(\Cov_f)}$ becomes $\sqrt{\mathfrak{C}(q,q) - \mathfrak{C}(q,c)}$. Finally, $\frac{d}{dq'}\mathfrak{C}(q,q')|_{q'=q}$ measures the sensitivity of the output co-mean with respect to small changes in input co-mean. So we can informally say the following. If $\mathbb{E}_ix[i]^2 = \mathbb{E}_ix'[i]^2 = q$ and $\mathbb{E}_ix[i]x'[i] = q - \epsilon$, then $\epsilon = \frac{1}{2d_\text{in}}||x - x'||_2^2$ and $\frac{d}{dq'}\mathfrak{C}(q,q')|_{q'=q}\epsilon + O(\epsilon^2) = \frac{1}{2d_\text{out}}||f(x) - f(x')||_2^2$. So $\sqrt{\frac{d_\text{out}}{d_\text{in}}\frac{d}{dq'}\mathfrak{C}(q,q')|_{q'=q}}$ measures the output change induced by a small input change as defined by Euclidean distance. But $\frac{1}{\sqrt{d_\text{in}}}||\mathcal{J}||_F$ also measures this change, so $\frac{d}{dq'}\mathfrak{C}(q,q')|_{q'=q}$ represents $\frac{1}{\sqrt{d_\text{out}}}||\mathcal{J}||_F^2$. Putting it all together, $\mathfrak{n}(f,q,c)$ resembles $NLCFROB$ from section <ref>, which has the same mean field limit as the NLC.

“Wide networks are Gaussian processes” [Yang, 2019], as we outlined in section <ref>. The associated kernel function for an A-architecture with elem-like input is $\mathfrak{C}(\mathbb{E}_ix[i]^2,\mathbb{E}_ix[i]x'[i])$. Therefore, we can view $\frac{d}{dq'}\mathfrak{C}(q,q')|_{q'=q}$ as the first-order approximation of the bandwidth of the kernel. $\mathfrak{n}(f,q,c)$ then becomes the (square root of the) bandwidth normalized by the radius of domain and codomain. This normalization is desirable for all the reasons it is desirable to employ this normalization in the NLC as discussed in e.g. sections <ref>, <ref> and <ref>. For a fixed unnormalized bandwidth value, a large domain implies that a small fraction of input pairs in that domain receive a significant kernel value, whereas a small domain implies that a large fraction of input pairs in that domain receive a significant kernel value. Similarly, a large codomain implies that a large kernel value, and hence a large bandwidth is required for outputs to be relatively close together, whereas a small codomain implies that a small bandwidth is sufficient. We will further study the importance of this normalization in chapter <ref>.

Via mean field theory, we therefore find that the NLC is a measure of model complexity for the same reason that kernel bandwidth is a measure of model complexity in the field of kernel methods. In the infinite width limit of the initial state, which this, as well as prior, work has shown is extremely practically predictive, the network function is determined by a kernel. To the degree to which an effectively normalized scalar bandwidth can be defined for that kernel, the mean field NLC captures it. In turn, the mean field NLC is highly practically predictive of the NLC. While proposition <ref> is unfortunately limited to A-architectures, we suspect an analogous result holds much more generally for mean field architectures under e.g. the setup of background corollary <ref>, or even for general Gaussian processes. Proving such an extension is future work.

Act. fun.    ReLU   SELU   softplus   Swish   abs. val.   tanh

Iter. limit   0   1.00   0.93   0   -   0

Conv. rate   $O(0.71^M)$   $O(0.78^M)$   $O(0.45^M)$   $O(0.5^M)$   -   $O(\frac{1}{M^{0.5}})$

Iter. limit   -   1   1   0   -   1

Conv. rate   -   $O(0.78^M)$   $O(0.49^M)$   $O(0.84^M)$   -   $O(0.46^M)$

Iter. limit    1   0   1   0.66   1   0

Conv. rate   $O(\frac{1}{M^2})$   $O(0.97^M)$   $O(0.32^M)$   $O(0.94^M)$   $O(\frac{1}{M^2})$   $O(0.93^M)$

$\mathfrak{n}_\tau(\lambda^2, 0)$

$\lambda \rightarrow \infty$   1.21   1.21   1.21   1.21   1.66   $O(\sqrt{\lambda})$

(This table is continued on the next three pages.) Activation function metrics for the activation functions used in this work, given in table <ref>, as well as their debiased versions. Segments of $\mathfrak{L}_\tau$ and $\frac{\mathfrak{L}_\tau(\lambda)}{\mathfrak{L}_\tau(1)}$ depicted in blue correspond to x-coordinate values that converge upon iteration to a non-zero value. Segments depicted in red correspond to x-coordinate values that diverge or converge to zero upon iteration. Segments depicted in magenta correspond to x-coordinate values that are stationary upon iteration. Segments of $\frac{\mathfrak{L}_\tau'(\lambda)\lambda}{\mathfrak{L}_\tau(\lambda)}$ are depicted in red if the y-coordinate is greater than 1, blue in $(-\frac{1}{2},1)$ and magenta at 1. Non-zero iteration limits are depicted as bold blue dots. Limits and convergence rates are given below the respective graph where applicable. Values of $\mathfrak{n}_\tau(\lambda^2,0)$ at $\lambda=0$, $\lambda=1$ and $\lambda=10$ are given in green. Diagonals are given as thin black lines for visual orientation. Each convergence rate is given as precisely as we were able to determine it with basic techniques. The values are discussed in sections <ref>, <ref> and <ref>. There is no single, overarching conclusion.

Act. fun.    sigmoid   even tanh   Gaussian   odd square   square   sawtooth

Iter. limit    0.51   0   0.37   0   0   0

Conv. rate   $O(0.05^M)$   $O(\frac{1}{M^{0.5}})$   $O(0.11^M)$   $\frac{1}{\sqrt{3}}(\sqrt{3}\lambda)^{2^M}$   $\frac{1}{\sqrt{3}}(\sqrt{3}\lambda)^{2^M}$   $\Omega(\frac{1}{M^\epsilon}) \forall \epsilon > 0$

Iter. limit    1   1   1   0   0   1

Conv. rate   $O(0.11^M)$   $O(0.46^M)$   $O(0.33^M)$   $\lambda^{2^M}$   $\lambda^{2^M}$   $\approx O(10^{-10M})$

Iter. limit    1   0.99   1   0   0.5   0

Conv. rate   $O(0.15^M)$   $O(0.91^M)$   $O(0.33^M)$   $O(0.85^M)$   $O(0.67^M)$   super-exp.

$\mathfrak{n}_\tau(\lambda^2, 0)$

$\lambda \rightarrow \infty$   $O(\sqrt{\lambda})$   $O(\lambda)$   $O(\lambda)$   1.15   1.41   $O(\lambda)$

(This table is a continuation of table <ref>.)

Act. fun.    ReLU-deb.   SELU-deb.   softpl.-deb.   Swish-deb.   abs. v.-deb.   tanh-deb.

Iter. limit    0.33   1.00   0.13   0.23   0.5   0

Conv. rate   $O(0.02^M)$   $O(0.78^M)$   $O(0.22^M)$   $O(0.17^M)$   $O(0.27^M)$   $O(\frac{1}{M^{0.5}})$

Iter. limit    1   1   0.63   0.67   1   1

Conv. rate   sub-exp.   $O(0.78^M)$   $O(0.93^M)$   $O(0.87^M)$   sub-exp.   $O(0.46^M)$

Iter. limit    0   0   0   0   0   0

Conv. rate   $O(0.73^M)$   $O(0.97^M)$   $O(0.92^M)$   $O(0.80^M)$   super-exp.   $O(0.93^M)$

$\mathfrak{n}_\tau(\lambda^2, 0)$

$\lambda \rightarrow \infty$   1.21   1.21   1.21   1.21   1.66   $O(\sqrt{\lambda})$

(This table is a continuation of table <ref>.)

Act. fun.    sigm.-deb.   even t.-deb.   Gauss.-deb.   odd sq.-deb.   sq.-deb.   sawtooth-deb.

Iter. limit    0   0.35   0.11   0   -   0

Conv. rate   $O(0.5^M)$   $O(0.36^M)$   $O(0.04^M)$   $\frac{1}{\sqrt{3}}(\sqrt{3}\lambda)^{2^M}$   -   $\Omega(\frac{1}{M^\epsilon}) \forall \epsilon > 0$

Iter. limit    1   1   1   0   0.58   1

Conv. rate   $O(0.72^M)$   $O(0.17^M)$   $O(0.74^M)$   $\lambda^{2^M}$   super-exp.   $\approx O(10^{-10M})$

Iter. limit    0   0   0   0   0   0

Conv. rate   $O(0.98^M)$   super-exp.   super-exp.   $O(0.85^M)$   $c^{2^M}$   super-exp.

$\mathfrak{n}_\tau(\lambda^2, 0)$

$\lambda \rightarrow \infty$   $O(\sqrt{\lambda})$   $O(\lambda)$   $O(\lambda)$   1.15   1.41   $O(\lambda)$

(This table is a continuation of table <ref>.)

§ EXPLAINING THE NLC'S PROPERTIES WITH MEAN FIELD THEORY

In section <ref>, we demonstrated that a range of properties holds for the NLC empirically in the architectures' initial state. It turns out that many of them can be explained via mean field theory as follows. (i) The property holds exactly for $\mathfrak{n}$. (ii) $\mathfrak{n}$ closely approximates the NLC as shown in section <ref>. Below, we go through the properties one by one. We list further properties of $\mathfrak{n}$ at the end of section <ref>.

The NLC is robust to data distribution (section <ref>) $\mathfrak{n}$ depends on the distribution only through the scalar $q$ and $c$ parameters. We showed in section <ref> that the three datasets we study in this work have characteristic $q$ and $c$ values which lead to accurate mean field estimates. $\mathfrak{n}$ will have the same value for any dataset with the same characteristic $q$ and $c$ values. In section <ref>, we also discussed the general class of such datasets. Note that when mean field theory is extended to cover convolutional layers, it utilizes the entire covariance structure of the input. See section <ref>.

The NLC is robust to random initialization (section <ref>) $\mathfrak{n}$ does not depend on $\theta$ directly, but only on the initialization scheme. Hence, it is invariant to the actual initial value of $\theta$ drawn from that scheme. When $\theta$ is random, the NLC converges almost surely as width converges to infinity.

The NLC can (sometimes) be proxied by even simpler metrics (section <ref>) Using statements (<ref>), (<ref>) and (<ref>) from theorem <ref> and the relation $||\mathbb{S}_xf_l||_2^2 = \Tr(\Cov_{f_l})$, we obtain that the NLC and NLCFROB as defined in section <ref> have the same mean field limit.

The NLC is related to kernel methods and model complexity (section <ref>) See section <ref>.

The NLC is decomposable into NLCs of individual layers (section <ref>) The mean field limits of the NLC and LNLC as defined in section <ref> are identical.

The NLC is robust to width change (section <ref>) The essence of mean field theory is convergence with increasing width. Hence, a change of width will not alter the NLC significantly as long as the width remains sufficiently large and the initial weight variance co-varies according to the LeCun initialization / variance parameter. In section <ref>, we show that even a width of 50, which is considered small in practice, is sufficient to obtain stability.

The initial NLC predicts the final NLC (section <ref>) In section <ref>, we measured our theoretical limit quantities with mean field metrics that use the magnitude of the actual parameter value. In that section, we also showed that the parameter magnitude, and hence the value of the mean field metrics, often does not change significantly during training.