Let's look at the square roots. Using the principle of the non-negative quadratic form on $(1, -1, 0 \times (NB-2))$, we obtain $\mathfrak{c}_{\lambda,1;\lambda,1}-\mathfrak{c}_{\lambda,1;\lambda,2} \ge 0$. Using it on $(1\times B, -1\times B, 0 \times B(N-2))$ yields $\frac{1}{B}\mathfrak{c}_{\lambda,1;\lambda,1}+\frac{B-1}{B}\mathfrak{c}_{\lambda,1;\lambda,2}-\mathfrak{c}^{\prime;\prime\prime}_{\lambda,1;\lambda,1} \ge 0$. Using it on $(1\times(NB))$ yields $\frac{1}{BN}\mathfrak{c}_{\lambda,1;\lambda,1}+\frac{B-1}{BN}\mathfrak{c}_{\lambda,1;\lambda,2}+\frac{N-1}{N}\mathfrak{c}^{\prime;\prime\prime}_{\lambda,1;\lambda,1} \ge 0$. The limits are then $\ddot{c}^{\{7\}}_\lambda=\sqrt{\ddot{\mathfrak{c}}_{\lambda,1;\lambda,2}-\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{\lambda,1;\lambda,1}}=0$ and $\ddot{c}^{\{8\}}_\lambda=\sqrt{\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{\lambda,1;\lambda,1}}=\sqrt{\ddot{\mathfrak{c}}_{\lambda,1;\lambda,2}}$ by stage 10.

Note that we consider the component index of $u$ vectors to be a batch index, and hence we consider the component mean $\overline{u}$ and batch mean $\widehat{u}$ interchangeable.

Recursive expansion. Next, we consider the recursive expansion of $\rho_{l,1}^-$. By expanding only $\rho_{l',b}^-$, $\rho_{b}^\text{MN}$ and $\rho_{\lambda,b}^{(n)}$ terms that do not occur inside a $\nu$, we obtain the following “first expansion”.

\begin{eqnarray*}
&&\rho_{l,1}^- \\
&=&\sum_{f_{l'}} c_{l'}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)(e_{l',1}^--\widehat{e_{l'}^-})\\
&&+\sum_{f_{l'}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{l',l}), Z=|\mathcal{L}|>0} c_{l'}(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
&&+\sum_{f_{\lambda}}\sum_{n=1}^N (\mathbbm{1}_{n=n^-}-\frac{1}{N})c_{\lambda}\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)\Bigg(\prod_{f_{l''}\in P_{\lambda,m}}\frac{1}{\sqrt{\nu_{l''}^{(n)}}}\Bigg)(e_{\lambda,1}^{(n)}-\widehat{e_{\lambda}^{(n)}})\\
&&+\sum_{f_{\lambda}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{m,l}), Z=|\mathcal{L}|>0}\sum_{n=1}^N(\mathbbm{1}_{n=n^-}-\frac{1}{N})c_{\lambda}(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
&&...\Bigg(\prod_{f_{l''}\in P_{\lambda,m}}\frac{1}{\sqrt{\nu_{l''}^{(n)}}}\Bigg)\frac{(\widehat{e_{\lambda}^{(n)}\rho_{k_{\mathcal{L}[1]}}^+}-\widehat{e_{\lambda}^{(n)}}\widehat{\rho_{k_{\mathcal{L}[1]}}^+})(\rho^+_{k_{\mathcal{L}[Z]},1}-\widehat{\rho^+_{k_{\mathcal{L}[Z]}}})}{\nu_{\mathcal{L}[Z]}^+}...\\
\end{eqnarray*}

Here, as in stage 9, $\sum_{f_{l'}}$ is over input and FC layers $f_{l'}$ from which there exists a directed path to $f_l$ that does not contain another FC layer, and $P_{l',l}$ refers to that path excluding its starting point $f_{l'}$. Now, $l'$ is restricted to $l > l' > m$. $\sum_{f_{\lambda}}$ is the equivalent sum for layer indices less than or equal to $m$. Here, as in stage 9, the $\prod_{f_{l''}\in P}$ are over BN layers in path $P$ and the $c_{l'}$ / $c_\lambda$ are the products of addition weights on $P$, which is 1 if there are no addition weights. Since $f_m$ is a bottleneck for $f_l$, $P_{\lambda,l}$ can be broken down into $P_{\lambda,m}$ and $P_{m,l}$. Since these paths do not contain their starting point, this represents a partitioning of the layers in $P_{\lambda,l}$.

Here, $\sum_{\mathcal{L} \in \mathsf{indpow}(P), Z=|\mathcal{L}|>0}$ is the sum over the powerset of the set of all layer indices of BN layers in $P$ except for the empty set. $\mathcal{L}$ is a vector of indices that is an element of the powerset where indices are sorted in decreasing order. For example, if path $P$ contains BN layers $f_4$, $f_7$ and $f_{13}$, then $\mathcal{L}$ takes values $(4)$, $(7)$, $(13)$, $(7,4)$, $(13,4)$, $(13,7)$, and $(13,7,4)$. $Z$ is the size of $\mathcal{L}$. $k_{\mathcal{L}[z]}$ is the index of the dependency of $f_{\mathcal{L}[z]}$. $\sum_{\mathcal{L} \in \mathsf{indpow}(P), Z=|\mathcal{L}|>0}$ arises by repeatedly applying the recursive definition of $\rho_{l,b}^-$ for BN layers, which is the sum of two parts. Passing through a BN layer causes any affected term in the recursive expansion to be replaced by two terms, which yields an exponential number of terms in the end. Each of those terms has a signature that determines whether the term corresponds to the left, simple part or the right, complex part of the recursive definition at any BN layer. $\mathcal{L}$ is that signature.

To better understand the first expansion, we recommend deriving it by hand for several example layer graphs.

We say the first expansion has four “super terms”. Super term 1 is on the first line, super term 2 is on the second and third line, super term 3 is on the fourth line and super term 4 is on the fifth, sixth and seventh line.

We continue the expansion process. Unfortunately, we cannot give further expansions explicitly as they are simply too complex. We must be content with describing them. The next step will be to replace each $\rho^+$ term (not inside a $\nu^+$) with its expansion in terms of $e^+$ and $\frac{1}{\sqrt{\nu^+}}$ terms. That expansion takes the form of the expansion of $\rho_{k,1}$ in stage 9, activation layer case. The replacement has the following effect. Super terms 1 and 3 are unaffected. Inside the $\sum_{\mathcal{L} \in \mathsf{indpow}(P_{l',l}), Z=|\mathcal{L}|>0}$ and $\sum_{\mathcal{L} \in \mathsf{indpow}(P_{m,l}), Z=|\mathcal{L}|>0}$ sums of super terms 2 and 4, we add $2Z$ additional sums, two per value of $z$, over the big layer indices that correspond to inputs of $\rho_{k_{\mathcal{L}[z]}}^+$. The numerator of the final part of the super term 2 becomes

where the $l_1$, .., $l_{2Z}$ are the indices of summation of the $2Z$ additional sums. For the super term 4, we obtain an analogous expression with $e_{\lambda}^{(n)}$ instead of $e_{l'}^-$. Also inside the additional sums, we obtain a product of addition weights / $\frac{1}{\sqrt{\nu^+}}$ terms stemming from addition layers / BN layers between the $\rho^+$ and their inputs respectively. Note that all $e_{l'}^{+\text{in}}$ and $e_{\lambda}^{+\text{in}}$ terms cancel out.

The next step in the expansion process is to replace each $e$ term (not inside a $\nu$) by the applicable $u^-$, $s^-$, $u^+$, $s^+$, $u^{(n)}$, $v$ and $s$. $e$ terms arise only inside of $(\mathsf{expr1} - \mathsf{expr2})$ constructs where $\mathsf{expr1}$ and $\mathsf{expr2}$ differ only in whether / how the batch mean is taken. Hence, the $s^-$, $s^+$, $v$ and $s$ cancel out, as they are constant across the batch. Also, $\mathsf{expr2}$ vanishes as the batch mean of individual $u$ vectors is zero. Hence super term 1 becomes

$$\Bigg(\sum_{f_{l'}} c_{l'}c^{\{5\}}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)u_{l'}^-[1]\Bigg)+ \Bigg(\sum_{f_{l'}} c_{l'}c^{\{3\}}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)u_{l'}^+[1]\Bigg)$$

We will refer to an instance of the expression inside the first $\sum_{f_{l'}}$ for one $f_{l'}$ as a “minus term” and to an instance of the expression inside the second $\sum_{f_{l'}}$ for one $f_{l'}$ as a “short plus term”. Similarly, we obtain two copies for super term 2. In the first copy, $e_{l'}^-$ becomes $c^{\{5\}}u_{l'}^-$ and in the second, it becomes $c^{\{3\}}u_{l'}^+$. We will refer to an instance of the expression inside the sums of the first copy as a “minus plus term” and to an instance of the expression inside the sums of the second copy as a “long plus term”. In both cases, the $e^+$ are replaced by $c^{\{1\}}u^+$. Note that going forward, we also use $u^+$ to refer to $u^{(n^+)}$ when the big layer index is less than or equal to $m$. Super term 3 becomes

$$\sum_{f_{\lambda}}\sum_{n=1}^N(\mathbbm{1}_{n=n^-}-\frac{1}{N}) c_{\lambda}c^{\{1\}}_\lambda\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)\Bigg(\prod_{f_{l''}\in P_{\lambda,m}}\frac{1}{\sqrt{\nu_{l''}^{(n)}}}\Bigg)u_{\lambda}^{(n)}[1]$$

We refer to an instance of the expression inside the $\sum_{f_{\lambda}}$ as an “n-term”. (Note that an n-term includes $\sum_{n=1}^N$.) Finally, in super term 4 the $e^{(n)}_\lambda$ become $c^{\{1\}}_\lambda u_{\lambda}^{(n)}$ and the $e^+$ become $c^{\{1\}} u^{+}$. We refer to an instance of the expression inside all sums except $\sum_{n=1}^N$ as an “n-plus term”. We use “pure term” to refer to either a minus term, short plus term, long plus term, minus plus term, n-term or n-plus term. The expansion of $\rho_{l,1}$ now consists of a sum over a number of pure terms that is fixed as $N$ and $B$ vary. We refer to that sum as the “second expansion”.

From the second expansion, it is already clear that (i) holds, i.e. that expectations involving $\rho_{l,b}^-$ and $\rho_{l,b}^+$ depend on $n^+$ and $n^-$ only through $\mathbbm{1}_{n^-=n^+}$ as the same holds for the $c^{\{.\}}$ by the induction hypothesis.

Each type of pure term has a similar form, which we call “standard form”. Specifically, it is a product of up to five things: an “N-sum”, an “N-frequency”, a “multiplier”, a “normalizer” and a “polynomial”. When present, the n-sum takes form $\sum_{n=1}^N$ and comes at the beginning of the pure term. The N-frequency is a function of $N$, $\mathbbm{1}_{n^-=n^+}$, $\mathbbm{1}_{n=n^+}$ and $\mathbbm{1}_{n=n^-}$ that converges as $N$ converges to infinity for any $n$, $n^-$, $n^+$. The multiplier is a product of addition weights, the $c^{\{.\}}$, and some number of $(-1)^{|Z|}$ terms. The normalizer is a product of $\frac{1}{\sqrt{\nu^+}}$ and possibly $\frac{1}{\sqrt{\nu^{(n)}}}$ terms with various subscripts. The polynomial is a product of terms which can include $u_{l'}^-[1]$ terms (e.g. minus term), $u^+[1]$ terms (e.g. short plus term, long plus term and n-plus term), $u^{(n)}_\lambda[1]$ terms (e.g. n-term), $\widehat{u_{l'}^-u^+}$ terms (e.g. minus plus term), $\widehat{u^{(n)}_\lambda u^+}$ terms (e.g. n-plus term), and  $\widehat{u^+u^+}$ terms (e.g. long plus term and n-plus term).

Expectation limits of standard form terms Let's look at the limit of the expectation of a standard form term as $B$ and $N$ converge to infinity. Consider first the N-sum, assuming it is present. Due to the symmetry of standard form terms, the expectation for a given value of $n$ depends on $n$ only via $\mathbbm{1}_{n=n^+}$ and $\mathbbm{1}_{n=n^-}$. (Remember that when $n=n^+$, we have $u^{(n)}_\lambda = u^+_\lambda$ for $\lambda \le m$.) Hence, we can eliminate the N-sum inside an expectation as follows. If $n^-=n^+$, we can duplicate the standard form term and replace the $(n)$ superscript with $(n^+)$ in one copy and some other fixed $(n^\circ)$ in the other copy, while replacing the N-frequency by its aggregate value. The aggregate N-frequency is obtained by summing all N-frequencies that belong to a given copy. For the first copy, this is only the N-frequency for $n=n^+$. For the second copy, this is all N-frequencies for $n \neq n^+$. For example, for n-terms, the aggregate N-frequency is $\frac{1}{N} - 1$ for the first copy and $1 - \frac{1}{N}$ for the second copy. If $n^-\neq n^+$, we can triplicate the standard form term and replace the $(n)$ superscript with $(n^+)$ in one copy, $(n^-)$ in the second copy and some other fixed $(n^\circ)$ in the third copy, while again replacing the N-frequency by its aggregate value. For example, for n-terms, the aggregate N-frequency is $-\frac{1}{N}$ for the first copy, $1 - \frac{1}{N}$ for the second copy and $-\frac{N-2}{N}$ for the third copy. Assuming the aggregate N-frequencies converge with $N$, all copies have themselves standard form.

Now consider the limit of the expectation of a standard form term without an N-sum. Consider the polynomial. We can multiply it out by eliminating all parentheses and batch mean operators. This yields what we call “chains” of individual $u$ components multiplied together, which are also multiplied by some power of $\frac{1}{B}$ stemming from the batch mean operators. $u$ components in these chains have component indices from 1 to B, which correspond to batch indices. However, since the multiplier, normalizer and N-frequency either do not contain or are symmetric with respect to batch indices, we can aggregate chains that are equivalent up to batch index reassignment inside the expectation. For example, $u^+[1](u^+[4])^2u^+[7]$ (times multiplier, normalizer and N-frequency) has the same expectation as $(u^+[1])^2u^+[2]u^+[3]$, and $u^-[1]u^-[B]u^+[1]u^+[B]$ has the same expectation as $u^-[1]u^-[2]u^+[1]u^+[2]$. This aggregation procedure allows us to eliminate all batch index values greater than the degree of the chains, which is fixed as $B$ and $N$ vary. Call this degree $D$. Hence, as $B$ and $N$ vary, there is a finite set of possible “aggregated chains”. The aggregation process causes each of these aggregated chains to be multiplied by the sum of powers of $\frac{1}{B}$ corresponding to the individual chains that were aggregated into it. This sum may depend on $B$ but converges as $B$ converges to infinity. Hence, we call it “B-frequency”. So we have that the standard form term is a sum over products of N-frequency, B-frequency, multiplier, normalizer and aggregated chain. By allowing the B-frequency to be zero, we can consider the sum to be over a fixed number of products and we can consider the set of aggregated chains fixed.

Both N-frequency and B-frequency converge in the limit of $N$ and $B$. Unless the multiplier contains a $c^{\{4\}}$ or $c^{\{6\}}$ term and $n^-=n^+$, the multiplier converges as addition weights and $(-1)^{|Z|}$ terms are constant and the $c^{\{.\}}$ converge as shown above. Now we apply lemma <ref> to each product of aggregated chain and normalizer, in a way that is similar to stage 9. $B$ maps onto $d$. Each different $u$ component from the aggregated chain maps onto a component of $\chi_d$. For example, if the aggregated chain is $(u^-[1])^2u^-[2](u^+[1])^3u^+[2]$, we obtain $d_G=4$. $\Sigma_d$ is then block-diagonal with blocks of sizes at most $D$ which have $\frac{d-1}{d}$ on the diagonal and $-\frac{1}{d}$ off the diagonal. So $\hat{\Sigma}$ is the identity matrix. $G(\mathsf{arg})$ is a product of $\mathsf{arg}$ components that mirrors the aggregated chain. The normalizer to the power of $\frac{1}{D}$ maps onto each component of $\omega_d^G$. $H$ and $\omega_d^H$ are equal to constant 1. It is easy to check that neither the distribution of an aggregated chain nor of the normalizer depends on $N$, so the limit with respect to $N$ can be ignored. The conditions of lemma <ref> hold as for the BN layer case in stage 9. The application of lemma <ref> yields that the product of aggregated chain and normalizer has a valid expectation limit as $B$ converges to infinity, and hence as $B$ and $N$ converge to infinity. This limit is obtained by endowing $u$ components with the unit Gaussian distribution and taking the almost sure limit of the normalizer. So finally we have that the valid limit of the expectation of the product of N-frequency, B-frequency, multiplier, normalizer and aggregated chain is equal to the product of the limit of the N-frequency, the limit of the B-frequency, the limit of the multiplier, the almost sure limit of the normalizer and the expectation of the aggregated chain with respect to the unit Gaussian, assuming the multiplier does not contain a $c^{\{4\}}$ or $c^{\{6\}}$ term.

Specific expectation limits  Now we are ready to investigate the $\limbn$ quantities: $\ddot{\mathfrak{m}}_{l,1}$, $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}$, $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}$, $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}$ and $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$. We will break these down into expectation limits of pure terms / standard form terms. Most of the time, the expectation limit of a standard form term turns out to be zero, for one of the following reasons: the N-frequency converges to zero; the B-frequency of aggregated chains converges to zero; the multiplier converges to zero; or the aggregated chains contain exactly one copy of some $u$ component. In the latter case, because individual $u$ components are independent from other components and have mean zero under the unit Gaussian, the expectation of the entire aggregated chain is zero.

We have $\tilde{\mathfrak{m}}_{l,1}=\widehat{\tilde{\mathfrak{m}}_l}=0$. The first half of this statement holds by symmetry. The second half uses that the batch mean of any $u$ vector is zero. Hence, $\ddot{\mathfrak{m}}_{l,1}=0$.

Let's look at $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}=\limbn \mathbb{E}_e \rho_{l,1}^-\rho_{l,1}^+$. $\rho_{l,1}^+$ recursively expands into a sum of terms containing a multiplier made up of addition weights and a $c^{\{1\}}$ term, a normalizer and a polynomial of form $u^+[1]$. We call those terms “co-terms”, and they have standard form. So $\rho_{l,1}^-\rho_{l,1}^+$ is the sum of products of one pure term and one co-term. Each product has standard form and does not contain a $c^{\{4\}}$ or $c^{\{6\}}$ term, so the limit of its expectation is valid if an N-sum is not present. An N-sum is only present in a product of an n-term and co-term and a product of an n-plus term and co-term. We remove the N-sum via duplication as described above. Note that because in an n-term and n-plus term only the N-frequency depends on $n^-$, we can aggregate all $n$ values unequal to $n^+$, no matter whether $n^-=n^+$. The aggregate N-frequency is $\frac{1}{N} - \mathbbm{1}_{n^-=n^+}$ for the copy corresponding to $n\neq n^+$ and $\mathbbm{1}_{n^-=n^+}-\frac{1}{N}$ for the copy corresponding to $n= n^+$. Both converge, so both copies have standard form, so the limits of their expectations are valid. Hence overall, $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}$ itself is valid.

Consider the value of the limit for the case $n^- \neq n^+$. We break this down by pure term type. For a product of minus term and co-term or minus plus term and co-term, each aggregated chain contains a single $u^-$ component. Hence, the expectation limit is zero. For a product of short plus term and co-term or long plus term and co-term, we have that the multiplier contains $c_{l'}^{\{3\}}$. Since $\ddot{c}^{\{3\}}_{l'}|_{n^-\neq n^+}=0$, the expectation limit is zero. For a product of n-term and co-term or n-plus term and co-term, we obtain an N-frequency of $\frac{1}{N} - \mathbbm{1}_{n^-=n^+}$ / $\mathbbm{1}_{n^-=n^+}-\frac{1}{N}$ as mentioned above. Both converge to zero when $n^- \neq n^+$. So the expectation limit is again zero. So, overall $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}|_{n^-\neq n^+}=0$ as required.

We have $\mathfrak{c}_{l,1;l,2}^{-;+}=\mathbb{E}_e\rho_{l,1}^-\rho_{l,2}^+=\mathbb{E}_e\rho_{l,1}^-(\frac{B}{B-1}\widehat{\rho_{l}^+} - \frac{1}{B-1}\rho_{l,1}^+)=- \frac{1}{B-1}\mathbb{E}_e\rho_{l,1}^-\rho_{l,1}^+=- \frac{1}{B-1}\mathfrak{c}_{l,1;l,1}^{-;+}$. Here we use symmetry and that the batch mean is zero at a big BN layer. So $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}=\limbn -\frac{1}{B-1}\mathfrak{c}_{l,1;l,1}^{-;+}=\limbn -\frac{1}{B-1}\limbn\mathfrak{c}_{l,1;l,1}^{-;+}=0$, so specifically $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}|_{n^-\neq n^+}=0$ as required.

Similarly, we have $\mathfrak{c}_{l,1;l,2}^{-;-}=- \frac{1}{B-1}\mathfrak{c}_{l,1;l,1}^{-;-}$, so if $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$ is valid as we show below, $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}=0$ and specifically $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}|_{n^-\neq n^+}=0$ as required.

Finally, let's look at $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}=\limbn \mathbb{E}_e \rho_{l,1}^-\rho_{l,1}^-$. $\rho_{l,1}^-\rho_{l,1}^-$ is the sum of products of two pure terms. Such a product has standard form, except it may contain two N-sums. We use the same replication trick as before when this is the case. Let the indices of summation corresponding to the two sums be $n$ and $n'$. Then we require one copy for $n=n'=n^+$, one copy for $n=n^+ \neq n'$, one copy for $n'=n^+\neq n$, one copy for $n=n'\neq n^+$, and one copy for when all three are different. The aggregate N-frequencies when $n^-=n^+$ are $\frac{(N-1)^2}{N^2}$, $-\frac{(N-1)^2}{N^2}$, $-\frac{(N-1)^2}{N^2}$, $\frac{(N-1)}{N^2}$ and $\frac{(N-1)(N-2)}{N^2}$ in that order. When $n^-\neq n^+$, they are $\frac{1}{N^2}$, $-\frac{1}{N^2}$, $-\frac{1}{N^2}$, $\frac{N^2-N-1}{N^2}$ and $\frac{-N^2+N+2}{N^2}$. All converge. Since the pure terms do not contain $c^{\{4\}}$ or $c^{\{6\}}$ terms, $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$ is valid.

Consider the value of the limit for the case $n^-\neq n^+$. Again, we go by pure term type.

Case: product of short plus term or long plus term and any pure term. The multiplier contains $c^{\{3\}}$, so the expectation limit is zero.

Case: product of minus term or minus plus term and n-term or n-plus term. Each product of an aggregated chain from the minus or minus plus term and an aggregated chain from a copy of the n-term or n-plus term contains a single $u^-$ component, so the expectation limit is zero.

Case: product of minus term or minus plus term and minus term or minus plus term where the big layer indices from the leading sum $\sum_{f_{l'}}$ are different. Each product of aggregated chains contains two single $u^-$ components, one each for the two big layer indices, so the expectation limit is zero.

Case: product of minus plus term and minus plus term where the big layer indices are the same. Then the polynomial of the product is composed as follows. It has a $\widehat{u^-_{l'}u^+_{l_1}}$ term; a $\widehat{u^-_{l'}u^+_{l_2}}$ term; a $u^+_{l_3}[1]$ term; a $u^+_{l_4}[1]$ term; and some number of $\widehat{u^+u^+}$ terms with various subscripts. The $l_1$ through $l_4$ are less than $l$ and not necessarily different. Let the product of the minus plus terms be written informally as $\mathsf{other} * \mathsf{poly}$. Then

\begin{eqnarray*}
&&\mathbb{E}_u \mathsf{other} * \mathsf{poly}\\
&=&\mathbb{E}_u \mathsf{other}\widehat{u^-_{l'}u^+_{l_1}}\widehat{u^-_{l'}u^+_{l_2}}u^+_{l_3}[1]u^+_{l_4}[1]\prod\widehat{u^+u^+}\\
&=&\mathbb{E}_u \mathsf{other}\widehat{u^-_{l'}u^+_{l_1}}\widehat{u^-_{l'}u^+_{l_2}}\widehat{u^+_{l_3}u^+_{l_4}}\prod\widehat{u^+u^+}\\
&=&\mathbb{E}_u \mathsf{other}(\frac{1}{B}u^-_{l'}[1]u^+_{l_1}[1]u^-_{l'}[1]u^+_{l_2}[1] \\
&&+ \frac{B-1}{B}u^-_{l'}[1]u^+_{l_1}[1]u^-_{l'}[2]u^+_{l_2}[2])\widehat{u^+_{l_3}u^+_{l_4}}\prod\widehat{u^+u^+}\\
\end{eqnarray*}

Here we use symmetry with respect to batch indices repeatedly. Now we form aggregated chains in a 2-step process. First, we aggregate the “partial chains” obtained from multiplying out $\widehat{u^+_{l_3}u^+_{l_4}}\prod\widehat{u^+u^+}$, where we do not replace batch indices 1 and 2 and replace no batch index with 1 or 2. (This can lead to batch indices up to $D+2$ being included in the aggregated partial chains. However, this does not affect our ability to apply lemma <ref> as discussed above.) Second, we multiply a copy of each aggregated partial chain with $\frac{1}{B}u^-_{l'}[1]u^+_{l_1}[1]u^-_{l'}[1]u^+_{l_2}[1]$ and a copy of each aggregated partial chain with $\frac{B-1}{B}u^-_{l'}[1]u^+_{l_1}[1]u^-_{l'}[2]u^+_{l_2}[2]$. Since the B-frequencies of the aggregated partial chains are convergent, the B-frequencies of aggregated chains stemming from $\frac{1}{B}u^-_{l'}[1]u^+_{l_1}[1]u^-_{l'}[1]u^+_{l_2}[1]$ converge to zero. Further, aggregated chains stemming from $\frac{B-1}{B}u^-_{l'}[1]u^+_{l_1}[1]u^-_{l'}[2]u^+_{l_2}[2]$ contain a single $u^-_{l'}[1]$ and a single $u^-_{l'}[2]$ component. So for either type of aggregated chain, the expectation limit is zero.

Case: product of minus plus term and minus term where the big layer indices are the same. The argument is as in the previous case where

\begin{eqnarray*}
&&\mathbb{E}_u \mathsf{other} * \mathsf{poly}\\
&=&\mathbb{E}_u \mathsf{other}\widehat{u^-_{l'}u^+_{l_1}}u^-_{l'}[1]u^+_{l_2}[1]\prod\widehat{u^+u^+}\\
&=&\mathbb{E}_u \mathsf{other}(\frac{1}{B}u^-_{l'}[1]u^+_{l_1}[1]u^-_{l'}[1]u^+_{l_2}[1] + \frac{B-1}{B}u^-_{l'}[1]u^+_{l_1}[1]u^-_{l'}[2]u^+_{l_2}[2])\prod\widehat{u^+u^+}\\
\end{eqnarray*}

Case: product of an n-term or n-plus term and an n-term or n-plus term where the big layer indices are different. We consider the 5 copies as described above. When $n=n^+$ and / or $n'=n^+$, the N-frequency converges to zero. Otherwise, each product of aggregated chains contains a single $u^{(n)}$ and a single $u^{(n')}$ component, which are unequal to any $u^+$ component because $n\neq n^+$ and $n'\neq n^+$. The expectation limit is zero.

Case: product of an n-term or n-plus term and n-plus term where the big layer indices are the same. We consider the 5 copies again. When $n=n^+$ and / or $n'=n^+$, the N-frequency converges to zero. Otherwise, when $n \neq n'$, there are single $u^{(n)}$ and $u^{(n')}$ components. When $n=n'$, the expectation equals

$$\mathbb{E}_u \mathsf{other}(\frac{1}{B}u^{(n)}_{\lambda}[1]u^+_{l_1}[1]u^{(n)}_{\lambda}[1]u^+_{l_2}[1] + \frac{B-1}{B}u^{(n)}_{\lambda}[1]u^+_{l_1}[1]u^{(n)}_{\lambda}[2]u^+_{l_2}[2])\prod\widehat{u^+u^+}$$

So the expectations based on individual aggregated chains have limit zero either based on B-frequency or a single $u$ component as above.

Case: product of an n-term and n-term where the big layer indices are the same. We consider for now only 4 of the 5 copies: those where at least one of $n$ and $n'$ equals $n^+$ and the one where all three are unequal. For the first three, the N-frequency converges to zero. When all three are unequal, there is a single $u^{(n)}$ and $u^{(n')}$ component.

So, in summary, we are left with one copy from the case above as well as products of minus terms where the big layer index is the same. Hence, letting $n^\circ$ be some value other than $n^+$ and endowing $u$ components with unit Gaussian distributions, we have for $n^-\neq n^+$

\begin{eqnarray*}
&=&\sum_{f_{l'}} \Bigg(\frac{\limbn c_{l'}^2c^{\{5\}}_{l'}c^{\{5\}}_{l'}}{\limbas \prod_{f_{l''}\in P_{l',l}}\nu_{l''}^+}\Bigg)\mathbb{E}_uu_{l'}^-[1]^2 \\
&&+ \sum_{f_{\lambda}} \Bigg(\frac{\limbn\frac{N^2-N-1}{N^2}c_{\lambda}^2c^{\{1\}}_{\lambda}c^{\{1\}}_{\lambda}}{\limbas\Big(\prod_{f_{l''}\in P_{m,l}}\nu_{l''}^+\Big)\Big(\prod_{f_{l''}\in P_{\lambda,m}}\nu_{l''}^{(n^\circ)}\Big)}\Bigg)\mathbb{E}_uu_{\lambda}^{(n^\circ)}[1]^2\\
&=&\sum_{f_{l'}} \frac{c_{l'}^2\ddot{c}^{\{5\}}_{l'} \ddot{c}^{\{5\}}_{l'}}{\prod_{f_{l''}\in P_{l',l}}\ddot{\nu}_{l''}} + \sum_{f_{\lambda}} \frac{c_{\lambda}^2\ddot{c}^{\{1\}}_{\lambda}\ddot{c}^{\{1\}}_{\lambda}}{\Big(\prod_{f_{l''}\in P_{m,l}}\ddot{\nu}_{l''}\Big)\Big(\prod_{f_{l''}\in P_{\lambda,m}}\ddot{\nu}_{l''}\Big)}\\
&=&\sum_{f_{l'}} \frac{c_{l'}^2\ddot{\mathfrak{c}}_{l',1;l',1}^{-;-}}{\prod_{f_{l''}\in P_{l',l}}\ddot{\nu}_{l''}} + \sum_{f_{\lambda}} \frac{c_{\lambda}^2(\ddot{\mathfrak{c}}_{\lambda,1;\lambda,1}-\ddot{\mathfrak{c}}_{\lambda,1;\lambda,2})}{\prod_{f_{l''}\in P_{\lambda,l}}\ddot{\nu}_{l''}}
\end{eqnarray*}

This is very reminiscent of the corresponding expression from the BN layer case of stage 9. The value of $\ddot{\nu}_{l''}$ carries over from stage 9. Again, using the recursive calculation rules for bias, addition and BN layers in this stage, for $\ddot{\mathfrak{c}}^{\text{MN};\text{MN}}_{1;1}$, and for bias, addition and BN layers in stage 9, it is easy to check that this is equivalent to $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} = \frac{\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}}{\ddot{\mathfrak{c}}_{k,1;k,1}-\ddot{\mathfrak{c}}_{k,1;k,2}+\epsilon_l}$. Hence $\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}|_{n^-\neq n^+} = \frac{\limep\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}|_{n^-\neq n^+}}{\limep\ddot{\mathfrak{c}}_{k,1;k,1}|_{n^-\neq n^+}-\limep\ddot{\mathfrak{c}}_{k,1;k,2}|_{n^-\neq n^+}+\limep\epsilon_l}=\frac{\mathfrak{g}_k(\mathfrak{q}_m-\mathfrak{c}_m)}{(\mathfrak{q}_k-\mathfrak{c}_k)\mathfrak{g}_m}=\frac{\mathfrak{g}_l(\mathfrak{q}_m-\mathfrak{c}_m)}{\mathfrak{g}_m}$ as required.

Case: $f_l$ is an addition layer.

* Control: Since $\mathcal{C}^\text{E2}$ is linearly closed, if the $\rho_{k[\kappa],1}^-$ are controlled by $\mathcal{C}^\text{E2}$, so is $\rho_{l,1}^-$.

* Limits: $\tilde{\mathfrak{m}}_{l,1}^- = \mathbb{E}_e \sum_{\kappa=1}^{K}w_{\kappa}\rho_{k[\kappa],1}^- = \sum_{\kappa=1}^Kw_{\kappa}\tilde{\mathfrak{m}}_{k[\kappa],1}^-$. We also have $\mathfrak{c}_{l,1;l,1}^{-;+}=\sum_{\kappa=1}^Kw_{\kappa}^2\mathfrak{c}_{k[\kappa],1;k[\kappa],1}^{-;+}$, $\mathfrak{c}_{l,1;l,2}^{-;+}=\sum_{\kappa=1}^Kw_{\kappa}^2\mathfrak{c}_{k[\kappa],1;k[\kappa],2}^{-;+}$, $\mathfrak{c}_{l,1;l,1}^{-;-}=\sum_{\kappa=1}^Kw_{\kappa}^2\mathfrak{c}_{k[\kappa],1;k[\kappa],1}^{-;-}$ and $\mathfrak{c}_{l,1;l,2}^{-;-}=\sum_{\kappa=1}^Kw_{\kappa}^2\mathfrak{c}_{k[\kappa],1;k[\kappa],2}^{-;-}$, . We use the same recursion argument as in e.g. stage 9. The expansion itself is a bit more complex than in that stage. As it flows backward through the small layer graph of $f$, each $f_{l',b}$ / $f_{\lambda,b}$ is represented in $F$ by $F^-_{l',b}$ and $F^+_{l',b}$ / the $F_{\lambda,b}^{(n)}$ respectively. Hence, we must apply the recursive definitions corresponding to those layers. If the expansion touches $f_m$, we must also apply the recursive definition of the $\rho^\text{MN}_b$. However, the argument still goes through with a wider range of recursion steps.

So, $\ddot{\mathfrak{m}}_{l,1}^- = \sum_{\kappa=1}^Kw_{\kappa}\ddot{\mathfrak{m}}_{k[\kappa],1}^-$, $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}=\sum_{\kappa=1}^Kw_{\kappa}^2\ddot{\mathfrak{c}}_{k[\kappa],1;k[\kappa],1}^{-;-}$ etc. so (i) through (iii) carry over from the dependencies to $f_l$. Finally, we have $\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}|_{n^-\neq n^+}=\sum_{\kappa=1}^Kw_{\kappa}^2\limep\ddot{\mathfrak{c}}_{k[\kappa],1;k[\kappa],1}^{-;-}|_{n^-\neq n^+}=\sum_{\kappa=1}^Kw_{\kappa}^2\frac{\mathfrak{g}_{k[\kappa]}(\mathfrak{q}_m-\mathfrak{c}_m)}{\mathfrak{g}_m}=\frac{\mathfrak{g}_{l}(\mathfrak{q}_m-\mathfrak{c}_m)}{\mathfrak{g}_m}$ and so (iv).

Case: $f_l$ is an activation layer.

* Control: We have $\rho_{l,1}^-=\rho_{k,1}^-\tau_l'(\rho_{k,1}^+)$. $\tau_l'(\rho_{k,1}^+)$ is controlled by $\mathcal{C}^\text{E2}$ as $\tau_l(\rho_{k,1}^+)$ is controlled by $\mathcal{C}^\text{E2}$ in stage 9. The induction hypothesis yields that $\rho_{k,1}^-$ is controlled by $\mathcal{C}^\text{E2}$. The product of two functions controlled by $\mathcal{C}^\text{E2}$ is itself controlled by $\mathcal{C}^\text{E2}$.

* Limits: We follow the same steps as for the BN layer case of this stage.

Joint distribution of inputs The joint distribution of the inputs of $\rho_{l,1}$ is as for the BN layer case, except that $e^{+\text{in}}_{l',1}$ terms for $l > l' > m$ and $e^{+\text{in}}_{\lambda,1}$ terms for $\lambda \le m$ can arise inside of $\tau_l'()$. (Again, we restrict $l'$ to $l > l' > m$.) The $e^{+\text{in}}_{l',1}$ / $e^{+\text{in}}_{\lambda,1}$ are Gaussian scalars independent of all other inputs of $\rho_{l,1}$ with mean zero and variance $\sigma_{l'}^2$ / $\sigma_{\lambda}^2$. Hence, we set $e^{+\text{in}}_{l',1}=\sigma_{l'}^2s^\text{in}_{l'}$ and $e^{+\text{in}}_{\lambda,1}=\sigma_\lambda^2s^\text{in}_{\lambda}$ where the $s^\text{in}_{l'}$ and $s^\text{in}_{\lambda}$ are unit Gaussian scalars. Because the bias vector is shared across individual inputs, those scalars depend neither on the batch index nor on the duplex index $n$.

Recursive expansion The “first expansion” is

\begin{eqnarray*}
&&\rho_{l,1}^- \\
&=&\sum_{f_{l'}\text{ normed}} c_{l'}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)(e_{l',1}^--\widehat{e_{l'}^-})\tau_l'(\rho_{k,1}^+)\\
&&+\sum_{f_{l'}\text{ normed}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{l',l}), Z=|\mathcal{L}|>0} c_{l'}(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
&&+\sum_{f_{\lambda}\text{ normed}}\sum_{n=1}^N (\mathbbm{1}_{n=n^-}-\frac{1}{N})c_{\lambda}\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)\Bigg(\prod_{f_{l''}\in P_{\lambda,m}}\frac{1}{\sqrt{\nu_{l''}^{(n)}}}\Bigg)...\\
&&+\sum_{f_{\lambda}\text{ normed}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{m,l}), Z=|\mathcal{L}|>0}\sum_{n=1}^N(\mathbbm{1}_{n=n^-}-\frac{1}{N})c_{\lambda}(-1)^{|Z|}...\\
&&...\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)\Bigg(\prod_{f_{l''}\in P_{\lambda,m}}\frac{1}{\sqrt{\nu_{l''}^{(n)}}}\Bigg)...\\
&&+\sum_{f_{l'}\text{ unnormed}} c_{l'}e_{l',1}^-\tau_l'(\rho_{k,1}^+)+\sum_{f_{\lambda}\text{ unnormed}}\sum_{n=1}^N (\mathbbm{1}_{n=n^-}-\frac{1}{N})c_{\lambda}e_{\lambda,1}^{(n)}\tau_l'(\rho_{k,1}^+)\\
\end{eqnarray*}

Here, as in the activation layer case of stage 9, the leading sums are either over input and FC big layers from which the directed path to $f_l$ not containing another FC layer does contain a BN layer, or does not contain a BN layer. The part of the first expansion corresponding to normed $f_{l'}$ and $f_\lambda$ is as for the BN layer case. The part of the first expansion corresponding to unnormed $f_{l'}$ and $f_\lambda$ is much simpler because the recursive definitions of multi-activation functions derived from BN layers are not applied.

The first expansion now has two additional super terms 5 and 6, and super terms 1 through 4 differ in that they now also contain a $\tau_l'(\rho_{k,1}^+)$ term. We continue the expansion process. Replacing $\rho^+$ terms not in a $\tau_l'()$ or $\nu^+$ with their expansion in terms of $e^+$ and $\frac{1}{\sqrt{\nu^+}}$ terms has the same effect as in the BN layer case, i.e. it adds $2Z$ additional sums and some other components to the second and fourth super term. Expanding the $\rho^+_{k,1}$ inside the $\tau_l'()$ replaces them with

$$\sum_{f_{l'} \text{ normed}}c_{l'}\Big(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Big)(e_{l',1}^+-\widehat{e_{l'}^+}) + \sum_{f_{l'}\text{ unnormed}}c_{l'} e_{l',1}^+ + \sum_{f_{l'}\text{ bias}}c_{l'} e_{l',1}^{+\text{in}}$$

$$+\sum_{f_{\lambda} \text{ normed}}c_{\lambda'}\Big(\prod_{f_{l''}\in P_{\lambda,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Big)(e_{\lambda,1}^+ -\widehat{e_{\lambda}^+}) + \sum_{f_{\lambda}\text{ unnormed}}c_{\lambda} e_{\lambda,1}^+ + \sum_{f_{\lambda}\text{ bias}}c_{\lambda} e_{\lambda,1}^{+\text{in}}$$

This is equivalent to the expansion of $\rho_{k,1}$ in stage 9, activation layer case, except that we break down each sum into big layers with index larger than $m$ and big layers with index at most $m$.

Now let's look at replacing the $e$ terms with $u$ and $v$ components and $s$ terms. The expression inside $\tau_l'()$ becomes $\mathsf{exp}_{l,1}$ where

\begin{eqnarray*}
\mathsf{exp}_{l,b}&=&\sum_{f_{l'} \text{ normed}}c_{l'}c_{l'}^{\{1\}}\Big(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Big)u_{l'}^+[b] + \sum_{f_{l'}\text{ unnormed}}c_{l'}c_{l'}^{\{1\}}u_{l'}^+[b] \\
&& + \sum_{f_{l'}\text{ unnormed}}c_{l'}c_{l'}^{\{2\}}s_{l'}^+ + \sum_{f_{l'}\text{ bias}}c_{l'}\sigma_{l'}^2 s_{l'}^\text{in}\\
&&+\sum_{f_{\lambda} \text{ normed}}c_{\lambda'}c_{\lambda}^{\{1\}}\Big(\prod_{f_{l''}\in P_{\lambda,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Big)u_{\lambda}^+[b]  \\
&&+ \sum_{f_{\lambda}\text{ unnormed}}c_{\lambda} c_{\lambda}^{\{1\}}u_{\lambda}^+[b] + \sum_{f_{\lambda}\text{ unnormed}}c_{\lambda}c_{\lambda}^{\{7\}} v_{\lambda}[n^+] \\
&&+ \sum_{f_{\lambda}\text{ unnormed}}c_{\lambda}c_{\lambda}^{\{8\}} s_{\lambda}+ \sum_{f_{\lambda}\text{ bias}}c_{\lambda} \sigma_{\lambda}^2s_{\lambda}^\text{in}
\end{eqnarray*}

Note that each $u$ or $v$ component or $s$ term that arises in this expression arises in only one sum, and only once in that sum. What is multiplied to it we term its “coefficient”. For super terms 1 through 4, introducing $u$, $v$ and $s$ outside of the $\tau_l'(\mathsf{exp}_{l,1})$ works exactly as in the BN layer case. We obtain minus terms, minus plus terms, short plus terms, long plus terms, n-terms and n-plus terms as before, except that these terms now also contain $\tau_l'(\mathsf{exp}_{l,1})$. Our two new super terms 5 and 6 become

$$\sum_{f_{l'}\text{ unnormed}} c_{l'}c_{l'}^{\{3\}}u_{l'}^+[1]\tau_l'(\mathsf{exp}_{l,1}) + \sum_{f_{l'}\text{ unnormed}} c_{l'}c_{l'}^{\{4\}}s_{l'}^+\tau_l'(\mathsf{exp}_{l,1})$$

$$ + \sum_{f_{l'}\text{ unnormed}} c_{l'}c_{l'}^{\{5\}}u_{l'}^-[1]\tau_l'(\mathsf{exp}_{l,1}) + \sum_{f_{l'}\text{ unnormed}} c_{l'}c_{l'}^{\{6\}}s_{l'}^-\tau_l'(\mathsf{exp}_{l,1})$$

$$\sum_{f_{\lambda}\text{ unnormed}}\sum_{n=1}^N (\mathbbm{1}_{n=n^-}-\frac{1}{N})c_{\lambda}c_{\lambda}^{\{1\}}u_{\lambda}^{(n)}[1]\tau_l'(\mathsf{exp}_{l,1})$$

$$ + \sum_{f_{\lambda}\text{ unnormed}}\sum_{n=1}^N (\mathbbm{1}_{n=n^-}-\frac{1}{N})c_{\lambda}c_{\lambda}^{\{7\}}v_{\lambda}[n]\tau_l'(\mathsf{exp}_{l,1})$$

respectively. $s_\lambda$ cancels out. This yields 6 new kinds of pure term, which we call 3-term, 4-term, 5-term, 6-term, 1-term and 7-term, respectively, for lack of better names, based on the superscript of $c$. Each of these new pure terms refers to an instance of the expression inside the leading sum. We thus obtain the “second expansion” as a sum over a fixed number of 12 kinds of pure terms, instead of a sum over a fixed number of 6 kinds of pure terms as for the BN layer case. Based on the second expansion, it is clear that (i) holds.

We need to expand our definition of the “standard form” from the BN layer case to cover the pure terms in this second expansion. In addition to N-sum, N-frequency, multiplier, normalizer and polynomial, standard form terms now also contain a “tau-deriv” of form $\tau_l'(\mathsf{exp}_{l,1})$. Polynomials are now also allowed to have $s_{l'}^+$ terms (e.g. 4-term),  $s_{l'}^-$ terms (e.g. 6-term) and $v_\lambda[n]$ terms (e.g. 7-term).

Expectation limits of standard form terms Let's look at the limit of the expectation of a “new” standard form term as $N$ and $B$ converge to infinity. N-sums are eliminated and the polynomial is broken down into aggregated chains as for the BN layer case, except that batch index value 1, which arises in $\mathsf{exp}_{l,1}$, is excluded from aggregation. We then apply lemma <ref> to each product of normalizer, aggregated chain and tau-deriv. As in the BN layer case, the distribution of normalizer and aggregated chain do not depend on $N$. While the distribution of some terms in $\mathsf{exp}_{l,1}$ such as $v_\lambda[n^+]$ terms do depend on $N$, the distribution of $\mathsf{exp}_{l,1}$ itself does not. Hence, we can map both $B$ and $N$ onto $d$ for lemma <ref>, thereby setting $B=N$, and still get the general limit with respect to $B$ and $N$ out of the lemma, as $N$ does not affect the value of the expectation. Each different $u$ component, $v$ component or $s$ term that arises in the aggregated chain or tau-deriv maps onto a component of $\chi_d$. As for the BN layer case, $\hat{\Sigma}$ is the identity matrix, $G(\mathsf{arg})$ is a product of $\mathsf{arg}$ components that mirrors the aggregated chain, and the normalizer to the power of $\frac{1}{D}$ maps onto each component of $\omega_d^G$. Now, the coefficient of the $u$ or $v$ component or $s$ term in $\mathsf{exp}_{l,1}$ corresponding to $\chi_d[i]$ maps onto $\omega^H_d[i]$ for each $i$. $\omega^H_d[i]$ is zero if the component or term does not appear. Finally, $H(\mathsf{arg}) = \tau_l'(\sum_i\mathsf{arg}[i])$. Let's check the conditions. Differentiability holds by property <ref> and integrability by assumption <ref>. $\hat{\Sigma}$ is positive definite. The components of $\omega_d^G$ are bounded and converge a.s. as for the BN layer case above and in stage 9. The components of $\omega_d^H$ are bounded and converge a.s. because $\frac{1}{\sqrt{\nu^+}}$ terms are bounded and converge a.s. and the $c^{\{.\}}$ represent deterministic, convergent sequences in $B=N$. Note that $c^{\{4\}}$ or $c^{\{6\}}$ terms do not appear in the tau-deriv.

So, analogously to the BN layer case above, we have that the valid limit of the expectation of the product of N-frequency, B-frequency, multiplier, normalizer, aggregated chain and tau-deriv is equal to the product of the limit of the N-frequency, the limit of the B-frequency, the limit of the multiplier, the almost sure limit of the normalizer, and the expectation of the product of aggregated chain and tau-deriv with respect to the unit Gaussian, where the coefficients of the Gaussian variables in the tau-deriv are also replaced by their almost sure limit. As discussed under the BN layer case, the limit of the multiplier may not exist when $n^- = n^+$ and it contains $c^{\{4\}}$ or $c^{\{6\}}$ terms. We consider this specific case separately below.

Specific expectation limits Again, we are ready to investigate the $\limbn$ quantities. Again, we will break these down into the expectation limits of pure terms / standard form terms, which are most often zero.

Let's look at $\ddot{\mathfrak{m}}_{l,1}^-$. $\rho_{l,1}^-$ is the sum of “new” standard form terms. We eliminate each N-sum and obtain aggregate N-frequencies $\frac{1}{N} - \mathbbm{1}_{n^-=n^+}$ and $\mathbbm{1}_{n^-=n^+} - \frac{1}{N}$ for the two copies respectively as for the BN layer case. Expectation limits of all standard form terms, which are now N-sum-free, are valid except possibly those of 4-terms and 6-terms. 6-terms contain a single $s^-_{l'}$ term. (Going forward, “single” implies that exactly one copy of the component or term appears in the polynomial and no copy appears in the tau-deriv.) So the expectation limit of the product of normalizer, polynomial and tau-deriv is zero. Since $c^{\{6\}}_{l'}$ is bounded for sufficiently large $N$ and $B$, the multiplier is eventually bounded. So multiplying the zero limit with the eventually bounded multiplier and bounded frequencies still yields a zero limit for the entire 6-term, and hence a valid limit. For 4-terms, we similarly have that $c^{\{4\}}_{l'}$ is eventually bounded. By non-singularity from stage 9, $\ddot{\mathfrak{c}}_{l',1;l',2} \ge 0$. If $\ddot{\mathfrak{c}}_{l',1;l',2} > 0$, then $\ddot{c}^{\{4\}}_{l'}$ is simply the limit of its numerator over the limit of its denominator. A valid $\ddot{c}^{\{4\}}_{l'}$ yields a valid limit of the expectation of the 4-term. On the other hand, if $\ddot{\mathfrak{c}}_{l',1;l',2} = 0$, we have $\ddot{c}^{\{2\}}_{l'}=0$. This implies that $s_{l'}^+$ has a coefficient in $\mathsf{exp}_{l,1}$ with limit zero. Hence, the $s_{l'}^+$ in the polynomial / aggregated chain of the 4-term becomes effectively a single term and we obtain a zero expectation limit as for 6-terms above. So in any case, the expectation limit of the 4-term is valid. So,  overall, $\ddot{\mathfrak{m}}_{l,1}^-$ is valid.

Consider the value of $\ddot{\mathfrak{m}}_{l,1}^-$ for $n^- \neq n^+$. Minus, minus plus terms and 5-terms become zero due to a single $u^-$ component. Short plus terms / long plus terms / 3-terms / 4-terms / 6-terms become zero as $\ddot{c}^{\{3\}}$ / $\ddot{c}^{\{3\}}$ / $\ddot{c}^{\{3\}}$ / $\ddot{c}^{\{4\}}$ / $\ddot{c}^{\{6\}}$ is zero respectively. For n-terms, n-plus terms, 1-terms and 7-terms, the N-frequency of both copies becomes zero. So $\ddot{\mathfrak{m}}_{l,1}^-|_{n^-\neq n^+} = 0$.

Let's look at $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}$ and $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}$. Multiplying $\rho_{l,1}^+$ to $\rho_{l,1}^-$ corresponds to additionally multiplying each pure term that already contains a $\tau_l'(\mathsf{exp}_{l,1})$ with a $\tau_l(\mathsf{exp}_{l,1})$ as well. Compared to a standard form term, in our application of lemma <ref>, we can simply replace $H(\mathsf{arg}) = \tau_l'(\sum_i\mathsf{arg}[i])$ with $H(\mathsf{arg}) = \tau_l'(\sum_i\mathsf{arg}[i])\tau_l(\sum_i\mathsf{arg}[i])$. We obtain that $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}$ is valid and $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}|_{n^-\neq n^+}=0$ in the same manner as for $\ddot{\mathfrak{m}}_{l,1}^-$ above. Similarly, multiplying $\rho_{l,2}^+$ to $\rho_{l,1}^-$ corresponds to additionally multiplying each pure term that already contains a $\tau_l'(\mathsf{exp}_{l,1})$ with a $\tau_l(\mathsf{exp}_{l,2})$ as well. Every $u$ or $v$ component or $s$ term that arises in both $\mathsf{exp}_{l,1}$ and $\mathsf{exp}_{l,2}$ is paired with the same coefficient in both. Hence, compared to a standard form term, in our application of lemma <ref>, we can simply let $H(\mathsf{arg})$ be the product of $\tau_l'$ applied to the sum of components of $\mathsf{arg}$ present in $\mathsf{exp}_{l,1}$ and $\tau_l$ applied to the sum of components of $\mathsf{arg}$ present in $\mathsf{exp}_{l,2}$. We also exclude batch index value 2 from aggregation. We obtain that $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}$ is valid and $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}|_{n^-\neq n^+}=0$ in the same manner as for $\ddot{\mathfrak{m}}_{l,1}^-$ above.

Let's look at $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$. $\rho_{l,1}^-\rho_{l,1}^-$ is the sum of products of two pure terms. Such products have standard form except that $\tau_l'(\mathsf{exp}_{l,1})^2$ is present instead of $\tau_l'(\mathsf{exp}_{l,1})$, which is a trivial change, and that two N-sums can arise, which are eliminated by creating five copies as for the BN layer case. So any product of two pure terms not containing a 4-term or 6-term has a valid expectation limit. For a product of a 4/6-term and a non-4/6-term, and for a product of two 4/6-terms where the big layer indices from the leading sum $\sum_{f_{l'}}$ are different, we obtain validity as for an individual 4/6-term above. Thus we are left for each unnormed $f_{l'}$ to confirm the validity of

$$\limbn c_{l'}c_{l'}c_{l'}^{\{4\}}c_{l'}^{\{4\}}\mathbb{E}_{u,s,v}s_{l'}^+s_{l'}^+\tau_l'(\mathsf{exp}_{l,1})^2 + 2c_{l'}c_{l'}c_{l'}^{\{4\}}c_{l'}^{\{6\}}\mathbb{E}_{u,s,v}s_{l'}^+s_{l'}^-\tau_l'(\mathsf{exp}_{l,1})^2$$

$$ + c_{l'}c_{l'}c_{l'}^{\{6\}}c_{l'}^{\{6\}}\mathbb{E}_{u,s,v}s_{l'}^-s_{l'}^-\tau_l'(\mathsf{exp}_{l,1})^2$$

If $\ddot{\mathfrak{c}}_{l',1;l',2} > 0$, then $\ddot{c}^{\{4\}}_{l'}$ and $\ddot{c}^{\{6\}}_{l'}$ are simply the limit of their respective numerators over the limit of their respective denominators, and so the above limit is valid. On the other hand, if $\ddot{\mathfrak{c}}_{l',1;l',2} = 0$, we have $\ddot{c}^{\{2\}}_{l'}=0$. This implies that $s_{l'}^+$ has a coefficient in $\mathsf{exp}_{l,1}$ with limit zero. Since $s_{l'}^-$ does not appear in $\mathsf{exp}_{l,1}$ at all, we have

and also

due to a single $s_{l'}^+$ term and a single $s_{l'}^-$ term and both $c_{l'}^{\{4\}}$ and $c_{l'}^{\{6\}}$ being eventually bounded. So we obtain

\begin{eqnarray*}
&&\limbn c_{l'}c_{l'}c_{l'}^{\{4\}}c_{l'}^{\{4\}}\mathbb{E}_{u,s,v}s_{l'}^+s_{l'}^+\tau_l'(\mathsf{exp}_{l,1})^2 + 2c_{l'}c_{l'}c_{l'}^{\{4\}}c_{l'}^{\{6\}}\mathbb{E}_{u,s,v}s_{l'}^+s_{l'}^-\tau_l'(\mathsf{exp}_{l,1})^2\\
&& + c_{l'}c_{l'}c_{l'}^{\{6\}}c_{l'}^{\{6\}}\mathbb{E}_{u,s,v}s_{l'}^-s_{l'}^-\tau_l'(\mathsf{exp}_{l,1})^2\\
&=&c_{l'}c_{l'}\Big(\limbn c_{l'}^{\{4\}}c_{l'}^{\{4\}}+c_{l'}^{\{6\}}c_{l'}^{\{6\}}\Big)\Big(\limbn\mathbb{E}_{u,s,v}s_{l'}^-s_{l'}^-\tau_l'(\mathsf{exp}_{l,1})^2\Big)\\
\end{eqnarray*}

The expression on the last line is valid, so the expression on the first line is valid, so overall $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$ is valid.

Consider the value of $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$ for $n^- \neq n^+$. Products of two pure terms where at least one is a short plus term, long plus term, 3-term, 4-term, 6-term or 7-term become zero due to the $c^{\{3\}}$, $c^{\{4\}}$, $c^{\{6\}}$ and $c^{\{7\}}$ becoming zero. So we are left with minus terms, minus plus terms, n-terms, n-plus terms, 1-terms and 5-terms. Any product of an n-term, n-plus term or 1-term on the one hand, and a minus term, minus plus term or 5-term on the other becomes zero as the aggregate N-frequencies are $\frac{1}{N} - \mathbbm{1}_{n^-=n^+}$ and $\mathbbm{1}_{n^-=n^+} - \frac{1}{N}$ as before. So we are left with two kinds of products. The first kind has only n-terms, n-plus terms and 1-terms and the second kind has only minus terms, minus plus terms and 5-terms.

Let's consider the first kind. For a product of an n-term, n-plus term or 1-term and another such term, we create 5 copies to eliminate the double N-sum as for the BN layer case. The aggregate N-frequency becomes zero when $n$ or $n'$ are equal to $n^+$. Further, for the copy where $n$, $n'$ and $n^+$ are all unequal, aggregated chains contain a single $u^{(n)}$ and $u^{(n')}$ component. So we are left with only the copy corresponding to $n=n'\neq n^+$. When the big layer indices from the leading sum are different, that copy also has aggregated chains with a single $u^{(n)}$ component. So we are left with one copy stemming from products where the big layer index of both pure terms is the same. In that situation, the product cannot contain a 1-term on the one hand and an n-term or n-plus term on the other as 1-terms correspond to unnormed big layers, and n-terms and n-plus terms correspond to normed big layers. For a product of an n-plus term and an n-term or n-plus term, we use the same argument as for the BN layer case to show it becomes zero due to either a single $u^{(n)}$ component or the B-frequency becoming zero, depending on the aggregated chain. The tau-deriv is immaterial to this argument. So we are left with one copy from products of 1-terms that share big layer index and with one copy from products of n-terms that share big layer index.

Let's now consider the second kind. Differing big layer indices lead to single $u^-$ components. So we are left with products of 5-terms for unnormed big layers, and minus terms and / or minus plus terms for normed big layers. For a product of a minus plus term and a minus or minus plus term, we use the same argument as for the BN layer case to show it becomes zero due to either a single $u^-$ components or the B-frequency becoming zero, depending on the aggregated chain. So we are left with products of minus terms that share big layer index and with products of 5-terms that share big layer index.

So analogously to the BN layer case, endowing $u$ and $v$ components and $s$ terms with unit Gaussian distributions, we obtain for $n^- \neq n^+$

\begin{eqnarray*}
&=&\sum_{f_{l'}} \Bigg(\frac{\limbn c_{l'}^2c^{\{5\}}_{l'}c^{\{5\}}_{l'}}{\limbas\prod_{f_{l''}\in P_{l',l}}\nu_{l''}^+}\Bigg)\mathbb{E}_{u,s,v}u_{l'}^-[1]^2\tau_l'(\ddot{\mathsf{exp}}_{l,1})^2 \\
&&\text{\scalebox{0.99}{$+\sum_{f_{\lambda}} \Bigg(\frac{\limbn\frac{N^2-N-1}{N^2}c_{\lambda}^2c^{\{1\}}_{\lambda}c^{\{1\}}_{\lambda}}{\limbas \Big(\prod_{f_{l''}\in P_{m,l}}\nu_{l''}^+\Big)\Big(\prod_{f_{l''}\in P_{\lambda,m}}\nu_{l''}^{(n^\circ)}\Big)}\Bigg)\mathbb{E}_{u,s,v}u_{\lambda}^{(n^\circ)}[1]^2\tau_l'(\ddot{\mathsf{exp}}_{l,1})^2$}}\\
&=&\Bigg(\sum_{f_{l'}} \frac{c_{l'}^2\ddot{\mathfrak{c}}_{l',1;l',1}^{-;-}}{\prod_{f_{l''}\in P_{l',l}}\ddot{\nu}_{l''}} + \sum_{f_{\lambda}} \frac{c_{\lambda}^2(\ddot{\mathfrak{c}}_{\lambda,1;\lambda,1}-\ddot{\mathfrak{c}}_{\lambda,1;\lambda,2})}{\prod_{f_{l''}\in P_{\lambda,l}}\ddot{\nu}_{l''}}\Bigg)\mathbb{E}_{u,s,v}\tau_l'(\ddot{\mathsf{exp}}_{l,1})^2
\end{eqnarray*}

Here, we aggregate minus terms and 5-terms as well as n-terms and 1-terms into a single leading sum each. $\ddot{\mathsf{exp}}_{l,1}$ is obtained by replacing the coefficients in $\mathsf{exp}_{l,1}$ with their limit. $\ddot{\mathsf{exp}}_{l,1}$ is a linear combination of unit Gaussians, and hence is a mean zero Gaussian itself. By stage 9, activation layer case, the variance of that Gaussian is $\ddot{\mathfrak{c}}_{k,1;k,1}$. Using the recursive calculation rules for bias, addition and BN layers in this stage, for $\ddot{\mathfrak{c}}^{\text{MN};\text{MN}}_{1;1}$, and for the bias, addition and BN layers in stage 9, it is easy to check that $\Big(\sum_{f_{l'}} \frac{c_{l'}^2\ddot{\mathfrak{c}}_{l',1;l',1}^{-;-}}{\prod_{f_{l''}\in P_{l',l}}\ddot{\nu}_{l''}} + \sum_{f_{\lambda}} \frac{c_{\lambda}^2(\ddot{\mathfrak{c}}_{\lambda,1;\lambda,1}-\ddot{\mathfrak{c}}_{\lambda,1;\lambda,2})}{\prod_{f_{l''}\in P_{\lambda,l}}\ddot{\nu}_{l''}}\Big)=\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}$. So finally, $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} = \ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}\mathbb{E}_{s\sim\mathcal{N}(0,\ddot{\mathfrak{c}}_{k,1;k,1})}\tau_l'(s)^2$ and one last application of lemma <ref> yields $\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}|_{n^-\neq n^+} =  \frac{\mathfrak{g}_{k}(\mathfrak{q}_{m}-\mathfrak{c}_{m})}{\mathfrak{g}_{m}}\mathbb{E}_{s\sim\mathcal{N}(0,\mathfrak{q}_{k})}\tau_l'(s)^2=\frac{\mathfrak{g}_{k}(\mathfrak{q}_{m}-\mathfrak{c}_{m})}{\mathfrak{g}_{m}}\mathfrak{C}_{\tau_l'}(\mathfrak{q}_{k},\mathfrak{q}_{k})=\frac{\mathfrak{g}_{l}(\mathfrak{q}_{m}-\mathfrak{c}_{m})}{\mathfrak{g}_{m}}$ as required.

Finally, let's look at $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}$. We obtain validity as for $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$. In the context of lemma <ref>, $\tau_l'(\mathsf{exp}_{l,1})\tau_l'(\mathsf{exp}_{l,2})$ is handled  just as $\tau_l'(\mathsf{exp}_{l,1})\tau_l(\mathsf{exp}_{l,2})$ is handled for $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}$ above. To obtain the value of the limit for $n^- \neq n^+$, we also proceed along the lines of $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$. Again, we show that expectation limits of (copies of) products of pure terms become zero. We use the same arguments as for $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$, except for products of two 5-terms; products of one minus or minus plus term and another minus or minus plus term; the $n=n'\neq n^+$ copy of a product of two 1-terms; and for the $n=n'\neq n^+$ copy of a product of one n-term or n-plus term and another n-term or n-plus term. For these products, when the big layer index differs, we obtain a single $u^-$ or $u^{(n)}$ component as usual. But even when the big layer index is the same for both pure terms, there is a single $u_{l'}^-[1]$ and $u_{l'}^-[2]$ component, or a single $u_{\lambda}^{(n)}[1]$ and $u_{\lambda}^{(n)}[2]$ component. So, overall we obtain $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}|_{n^-\neq n^+}=0$ as required.

So as in stage 4, letting $n^-$ and $n^+$ vary again, we have

\begin{eqnarray*}
&&\limep \limbn \limd \frac{1}{d_l^\text{MF}}\mathbb{E}_X\Tr(\mathcal{J}_{l,m,1}\Cov_{f_m}\mathcal{J}_{l,m,1}^T)\\
&=&\limep \limbn \frac{1}{N^2}\sum_{n^-,n^+=1}^N\limd\overline{p_{l,1}^{(n^-,n^+)}.p_{l,1}^{(n^-,n^+)}}\\
&=&\limep \limbn \frac{1}{N}\mathfrak{c}_{l,1;l,1}^{-;-}|_{n^-=n^+} + \frac{N-1}{N}\mathfrak{c}_{l,1;l,1}^{-;-}|_{n^-\neq n^+}\\
&=&\limep \limbn \mathfrak{c}_{l;l}^{-;-}|_{n^-\neq n^+}\\
\end{eqnarray*}

as required.

Stage 13: statement (<ref>)

By symmetry and writing $\mathfrak{c}_{0,1;0,1}$ / $\mathfrak{c}_{0,1;0,1}^{\prime;\prime\prime}$ for $q$ / $c$ respectively, we have

\begin{eqnarray*}
&&\limep \limbn \limd NLC(f_l(f_m),f_m(\mathcal{D}^{(N,B)}))\\
&=&\limep \limbn\limd \sqrt{\frac{\mathbb{E}_X\Tr(\mathcal{J}_{l,m}(X)\Cov_{f_m}\mathcal{J}_{l,m}^T(X))}{\Tr(\Cov_{f_l})}}\\
&=&\limep \limbn \limd \sqrt{\frac{\sum_{b=1}^B\mathbb{E}_X\Tr(\mathcal{J}_{l,m,b}(X)\Cov_{f_m}\mathcal{J}_{l,m,b}^T(X))}{\sum_{b=1}^B\mathbb{E}_X||f_{l,b}(X)||_2^2 - ||\mathbb{E}_Xf_{l,b}(X)||_2^2}}\\
&=&\limep \limbn \sqrt{\frac{\sum_{b=1}^B\frac{1}{N}\mathfrak{c}_{l,b;l,b}^{-;-}|_{n^-=n^+} + \frac{N-1}{N}\mathfrak{c}_{l,b;l,b}^{-;-}|_{n^-\neq n^+}}{\sum_{b=1}^B\frac{N-1}{N}(\mathfrak{c}_{l,b;l,b}-\mathfrak{c}^{\prime;\prime\prime}_{l,b;l,b})}}\\
&=&\limep \limbn \sqrt{\frac{\frac{1}{N}\mathfrak{c}_{l,1;l,1}^{-;-}|_{n^-=n^+} + \frac{N-1}{N}\mathfrak{c}_{l,1;l,1}^{-;-}|_{n^-\neq n^+}}{\frac{N-1}{N}(\mathfrak{c}_{l,1;l,1}-\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,1})}}\\
\end{eqnarray*}

Stage 14: statement (<ref>)

This stage resembles stage 6, where references to stage 4 are replaced with references to stage 12. The case $l=m$ is again trivial. For $l > m$, as in stage 12, we will treat the cases $m > 0$ and $m = 0$ separately. However, now, the differences are more extensive and so there is no overlap in our presentation of the two cases.

Case: $m > 0$. We will show $\limep \limbn \limd \frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(1)})u||_2^2 = \frac{\mathfrak{g}_l}{\mathfrak{g}_m}$ where randomness is induced by $\theta$, $X^{(1)}$ and the unit Gaussian vector $u$, which has dimensionality $Bd_m^\text{MF}$. From lemma <ref>, we then obtain $\limd \frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(1)})u||_2^2=\limd \frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(1)})||_F^2$, and hence $\limep \limbn \limd \frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(1)})||_F^2 = \frac{\mathfrak{g}_l}{\mathfrak{g}_m}$ as required.

We have $\limd \frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(1)})u||_2^2 = \limd \overline{(u\mathcal{J}_{l,m,1}(X^{(1)})^T)^{.2}}$. This is the same expression we investigated in stage 12 as $p_{l,1}^{(n^-,n^+)}$ with $n^+=1$, except that $f_m(X^{(n^-)})-\frac{1}{N}\sum_{n=1}^Nf_m(X^{(n)})$ is replaced by $u$. In fact, if we let $p^{(n^-,1)}_{l,1}$ refer to $u\mathcal{J}_{l,m,1}(X^{(1)})^T$, we can proceed analogously to stage 12, which eventually yields

\begin{eqnarray*}
&&\limep\limbn\limd \frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(1)})||_F^2\\
&=&\limep\limbn\limd \overline{p^{(n^-,1)}_{l,1}.p^{(n^-,1)}_{l,1}}\\
&=&\limep\limbn\limd \overline{F^-_{l,1}.F^-_{l,1}}\\
&=&\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}\\
\end{eqnarray*}

The major difference is the form of the recursive expansion as it arises in the BN and activation layer cases of the induction, which we dive into below. The following minor differences also arise.

* The $F^\text{MN}_b$ are input layers. Each of them corresponds to a segment of $u$ of dimensionality $d_m^\text{MF}$. Because $u$ is unit Gaussian, we add a diagonal entry of 1 and off-diagonal entries of zero to the covariance matrix of $\mathcal{G}$ for each $F^\text{MN}_b$. Distribution equality is obvious. The $\limd$ quantities become $\tilde{\mathfrak{m}}^\text{MN}_1 = 0$, $\mathfrak{c}^\text{MN;MN}_{1;1} = 1$ and $\mathfrak{c}^\text{MN;MN}_{1;2}=\mathfrak{c}^\text{MN;(n)}_{1;m,1} = \mathfrak{c}^\text{MN;(n)}_{1;m,2}=0$. The same holds for the $\limbn$ quantities. Finally, $\limep \ddot{\mathfrak{c}}^\text{MN;MN}_{1;1}=1=\frac{\mathfrak{g}_m}{\mathfrak{g}_m}$. This is required to start off the induction.

* $n^-$ does not arise in the construction of $F$ and all limit quantities are independent of $n^-$ and $N$. The $n^-\neq n^+$ case becomes the general case.

* The case where $f_l$ is a readin layer does not appear in the induction step as $l > m > 0$ and $f_m$ is a bottleneck for $f_l$. We can ignore all associated arguments.

* When $f_l$ is an addition layer, the recursive expansion of $\rho_l^-$ halts at the $\rho^\text{MN}_b$, as these are now dummy multi-activation functions. The same arguments apply as the $e^\text{MN}_b$ are independent of all other $e$ terms for input and FC layers in $F$ and as $\tilde{\mathfrak{m}}^\text{MN}_1=0$.

* If $f_l$ is an FC, bias or addition layer, we obtain $\limep \ddot{\mathfrak{c}}^{-;-}_{l,1;l,1}=\frac{\mathfrak{g}_l}{\mathfrak{g}_m}$ as required, as the induction hypothesis now yields $\limep \ddot{\mathfrak{c}}^{-;-}_{k,1;k,1}=\frac{\mathfrak{g}_k}{\mathfrak{g}_m}$.

Let's look at the BN layer case. The “first expansion” becomes

\begin{eqnarray*}
&&\rho_{l,1}^- \\
&=&\sum_{f_{l'}} c_{l'}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)(e_{l',1}^--\widehat{e_{l'}^-})\\
&&+\sum_{f_{l'}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{l',l}), Z=|\mathcal{L}|>0} c_{l'}(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
&&+\sum_{f_m}c_{m}\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)(e_1^\text{MN}-\widehat{e^\text{MN}})\\
&&+\sum_{f_m}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{m,l}), Z=|\mathcal{L}|>0}c_{m}(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
\end{eqnarray*}

Here, for consistency of notation, we write $\sum_{f_m}$ for the sum over only layer $f_m$ if that layer is touched by the recursive expansion and a sum over no layers otherwise. (Note again that the recursive expansion halts at the $\rho^\text{MN}_b$.) The difference between this first expansion and that for the BN layer case of stage 12 is that super terms 3 and 4 have been replaced by copies of super terms 1 and 2 where $l'$ is replaced by $m$. Control holds as in stage 12.

We can decompose $(e_1^\text{MN}, .., e_B^\text{MN})$ as $u^\text{MN} + c^{\{9\}}s^\text{MN}$ where $u^\text{MN}$ is a Gaussian vector with mean zero and covariance matrix as our other $u$ vectors, $s^\text{MN}$ is a unit Gaussian scalar and $c^{\{9\}}=\frac{1}{B}$. We use this to derive the “second expansion”. It contains minus terms, minus plus terms, short plus terms and long plus terms as in the BN layer case of stage 12. We no longer obtain n-terms and n-plus terms, but we obtain similar terms where some $u^{(n)}$ components are replaced by $u^\text{MN}$ components, and the N-sum, N-frequency and part of the normalizer are removed. For example, an n-term becomes

$$c_{m}\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)u^\text{MN}[1]$$

Note that no such term is present if $\sum_{f_m}$ is the empty sum. Again, $\rho_{l,1}$ is a sum over a fixed number of six kinds of pure terms of standard form. Compared to the BN layer case of stage 12, standard form terms now do not contain N-sums and N-frequencies, but we allow them to contain $u^\text{MN}$ components. The application of lemma <ref> is as in the BN layer case of stage 12. Since no N-sums are present, each product of a pure term and a pure or co-term has standard form. Since now no $c^{\{4\}}$ terms or $c^{\{6\}}$ terms are present, we immediately obtain that $\ddot{\mathfrak{m}}_{l,1}$, $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}$, $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}$, $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}$ and $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$ are valid. $\ddot{\mathfrak{m}}_{l,1}=\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}=\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}=0$ holds again by symmetry. To determine the value of $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}$ and $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$, we again consider individual products of pure terms and pure terms or co-terms and find that most of them become zero because of the multiplier converging to zero, the presence of a single $u$ component, or the B-frequency converging to zero. We obtain $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}=0$ and when we endow $u$ components with unit Gaussian distributions, we have

\begin{eqnarray*}
&=&\sum_{f_{l'}} \Bigg(\frac{\limbn c_{l'}^2c^{\{5\}}_{l'}c^{\{5\}}_{l'}}{\limbas \prod_{f_{l''}\in P_{l',l}}\nu_{l''}^+}\Bigg)\mathbb{E}_uu_{l'}^-[1]^2 + \sum_{f_{m}} \Bigg(\frac{\limbn c_{m}^2}{\limbas \prod_{f_{l''}\in P_{m,l}}\nu_{l''}^+}\Bigg)\mathbb{E}_uu^\text{MN}[1]^2\\
&=&\sum_{f_{l'}} \frac{c_{l'}^2\ddot{\mathfrak{c}}_{l',1;l',1}^{-;-}}{\prod_{f_{l''}\in P_{l',l}}\ddot{\nu}_{l''}} + \sum_{f_{m}} \frac{c_m^2}{\prod_{f_{l''}\in P_{m,l}}\ddot{\nu}_{l''}}
\end{eqnarray*}

Again, using the recursive calculation rules for bias, addition and BN layers, it is easy to check that this is equivalent to $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} = \frac{\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}}{\ddot{\mathfrak{c}}_{k,1;k,1}-\ddot{\mathfrak{c}}_{k,1;k,2}+\epsilon_l}$. Hence $\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} = \frac{\limep\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}}{\limep\ddot{\mathfrak{c}}_{k,1;k,1}-\limep\ddot{\mathfrak{c}}_{k,1;k,2}+\limep\epsilon_l}=\frac{\mathfrak{g}_k}{(\mathfrak{q}_k-\mathfrak{c}_k)\mathfrak{g}_m}=\frac{\mathfrak{g}_l}{\mathfrak{g}_m}$ as required.

Finally, let's turn to the activation layer case. The first recursive expansion becomes

\begin{eqnarray*}
&&\rho_{l,1}^- \\
&=&\sum_{f_{l'}\text{ normed}} c_{l'}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)(e_{l',1}^--\widehat{e_{l'}^-})\tau_l'(\rho_{k,1}^+)\\
&&+\sum_{f_{l'}\text{ normed}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{l',l}), Z=|\mathcal{L}|>0} c_{l'}(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
&&+\sum_{f_m\text{ normed}}c_m\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)(e_1^\text{MN}-\widehat{e^\text{MN}})\tau_l'(\rho_{k,1}^+)\\
&&+\sum_{f_m \text{ normed}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{m,l}), Z=|\mathcal{L}|>0}c_m(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
&&+\sum_{f_{l'}\text{ unnormed}} c_{l'}e_{l',1}^-\tau_l'(\rho_{k,1}^+)+\sum_{f_m\text{ unnormed}}c_m e_1^\text{MN}\tau_l'(\rho_{k,1}^+)\\
\end{eqnarray*}

The difference between this first expansion and that for the activation layer case of stage 12 is that super terms 3, 4 and 6 have been replaced by copies of super terms 1, 2 and 5 where $l'$ is replaced by $m$. Control holds as in stage 12. Again, we derive the second expansion. It contains minus terms, minus plus terms, short plus terms and long plus terms as in the activation layer case of stage 12. We no longer obtain n-terms and n-plus terms, but we obtain similar terms where some $u^{(n)}$ components are replaced by $u^\text{MN}$ components, and the N-sum, N-frequency and part of the normalizer are removed. For example, an n-term becomes

$$c_{m}\Bigg(\prod_{f_{l''}\in P_{m,l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)u^\text{MN}[1]\tau_l'(\mathsf{exp}_{l,1})$$

Note that no such term is present if $\sum_{f_m}$ is the empty sum. We obtain 3-terms, 4-terms, 5-terms and 6-terms as before. Instead of 1-terms we now have

$$c_m u^\text{MN}[1]\tau_l'(\mathsf{exp}_{l,1})$$

and instead of 7-terms we now have

Again, $\rho_{l,1}$ is a sum over a fixed number of 12 kinds of pure terms of standard form. Compared to the activation layer case of stage 12, standard form terms now do not contain N-sums and N-frequencies but we allow them to contain $u^\text{MN}$ components and $s^\text{MN}$ terms. The application of lemma <ref> is as in the activation layer case of stage 12 and depends on whether $\tau_l'(\mathsf{exp}_{l,1})$, $\tau_l'(\mathsf{exp}_{l,1})^2$, $\tau_l'(\mathsf{exp}_{l,1})\tau_l(\mathsf{exp}_{l,1})$, $\tau_l'(\mathsf{exp}_{l,1})\tau_l'(\mathsf{exp}_{l,2})$ or $\tau_l'(\mathsf{exp}_{l,1})\tau_l(\mathsf{exp}_{l,2})$ is present. Pure terms containing $c^{\{4\}}$ and $c^{\{6\}}$ are as they were in the activation layer case of stage 12, and so the arguments for the validity of the expectation limits of those terms and products involving those terms are unchanged. Since also now no N-sums are present that need to be eliminated, we obtain the validity of the $\limbn$ quantities. To determine the value of these quantities, we again consider individual pure terms and their products and find that most of them become zero because of the multiplier converging to zero, the presence of a single $u$ component, or the B-frequency converging to zero. We obtain $\ddot{\mathfrak{m}}_{l,1}^-=\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}=\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}=\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}=0$ and, when we endow $u$ and $v$ components and $s$ terms with unit Gaussian distributions, we have

\begin{eqnarray*}
&=&\sum_{f_{l'}} \Bigg(\frac{\limbn c_{l'}^2c^{\{5\}}_{l'}c^{\{5\}}_{l'}}{\limbas\prod_{f_{l''}\in P_{l',l}}\nu_{l''}^+}\Bigg)\mathbb{E}_{u,s,v}u_{l'}^-[1]^2\tau_l'(\ddot{\mathsf{exp}}_{l,1})^2\\
&& + \sum_{f_{m}} \Bigg(\frac{\limbn c_{m}^2}{\limbas\prod_{f_{l''}\in P_{m,l}}\nu_{l''}^+}\Bigg)\mathbb{E}_{u,s,v}u^\text{MN}[1]^2\tau_l'(\ddot{\mathsf{exp}}_{l,1})^2\\
&=&\Big(\sum_{f_{l'}} \frac{c_{l'}^2\ddot{\mathfrak{c}}_{l',1;l',1}^{-;-}}{\prod_{f_{l''}\in P_{l',l}}\ddot{\nu}_{l''}} + \sum_{f_m} \frac{c_m^2}{\prod_{f_{l''}\in P_{m,l}}\ddot{\nu}_{l''}}\Big)\mathbb{E}_{u,s,v}\tau_l'(\ddot{\mathsf{exp}}_{l,1})^2
\end{eqnarray*}

Again, using the recursive calculation rules for bias, addition and BN layers, it is easy to check that we have $\Big(\sum_{f_{l'}} \frac{c_{l'}^2\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}}{\prod_{f_{l''}\in P_{l',l}}\ddot{\nu}_{l''}} + \sum_{f_m} \frac{c_m^2}{\prod_{f_{l''}\in P_{m,l}}\ddot{\nu}_{l''}}\Big)=\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}$. So as in the activation layer case of stage 12, $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} = \ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}\mathbb{E}_{s\sim\mathcal{N}(0,\ddot{\mathfrak{c}}_{k,1;k,1})}\tau_l'(s)^2$ and one last application of lemma <ref> yields $\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}=  \frac{\mathfrak{g}_{k}}{\mathfrak{g}_{m}}\mathbb{E}_{s\sim\mathcal{N}(0,\mathfrak{q}_{k})}\tau_l'(s)^2=\frac{\mathfrak{g}_{k}}{\mathfrak{g}_{m}}\mathfrak{C}_{\tau_l'}(\mathfrak{q}_{k},\mathfrak{q}_{k})=\frac{\mathfrak{g}_{l}}{\mathfrak{g}_{m}}$ as required.

Case: $m=0$. We have

\begin{eqnarray*}
&&\limep \limbn \limd \frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(1)})||_F^2\\
&=&\limep \limbn \limd \frac{1}{d_l^\text{MF}}\Big|\Big|\sum_{f_{l'} \text{ readin}}\mathcal{J}_{l',0}(X^{(1)})^T\mathcal{J}_{l,l',1}(X^{(1)})^T\Big|\Big|_F^2\\
&=&\limep \limbn \limd \frac{1}{d_l^\text{MF}}\Big|\Big|\sum_{f_{l'} \text{ readin}}W_{l'}^B\mathcal{J}_{l,l',1}(X^{(1)})^T\Big|\Big|_F^2\\
&=&\limep \limbn \limd \frac{1}{d_l^\text{MF}}\sum_{i=0}^{d_\text{in}-1}\sum_{b=0}^{B-1}\Big|\Big|\sum_{f_{l'} \text{ readin}}W_{l'}^B[i+d_\text{in}b, :]\mathcal{J}_{l,l',1}(X^{(1)})^T\Big|\Big|_2^2\\
&=&\limep \limbn \limd \frac{1}{d_l^\text{MF}}\frac{1}{B}\sum_{i=0}^{d_\text{in}-1}\sum_{b=0}^{B-1}\sum_{b=1}^{B}\Big|\Big|\sum_{f_{l'} \text{ readin}}W_{l'}^B[i+d_\text{in}b, :]\mathcal{J}_{l,l',b}(X^{(1)})^T\Big|\Big|_2^2\\
&=&\limep \limbn \limd \frac{1}{d_l^\text{MF}}\frac{1}{B}\sum_{i=0}^{d_\text{in}-1}\sum_{b=0}^{B-1}\Big|\Big|\sum_{f_{l'} \text{ readin}}W_{l'}^B[i+d_\text{in}b, :]\mathcal{J}_{l,l'}(X^{(1)})^T\Big|\Big|_2^2\\
&=&\limep \limbn \limd \frac{1}{d_l^\text{MF}}d_\text{in}\Big|\Big|\sum_{f_{l'} \text{ readin}}W_{l'}^B[0, :]\mathcal{J}_{l,l'}(X^{(1)})^T\Big|\Big|_2^2\\
&=&\limep \limbn \limd \overline{\Big(\sum_{f_{l'} \text{ readin}}\sqrt{d_\text{in}}W_{l'}^B[0, :]\mathcal{J}_{l,l'}(X^{(1)})^T\Big)^{.2}}\\
\end{eqnarray*}

Here, $\sum_{f_{l'} \text{ readin}}$ is over all readin layers. $W_{l'}^B$ is the block-diagonal matrix of size $Bd_\text{in} \times Bd_{l'}^\text{MF}$ with $B$ blocks of $W_{l'}$. $W_{l'}^B[i+d_\text{in}b, :]$ indicates a row of that matrix. In the derivation, we use symmetry repeatedly.

Again, we proceed along the lines of stage 12, where we now let $p_{l,1}^{(n^-, 1)}$ refer to $\sum_{f_{l'} \text{ readin}}\sqrt{d_\text{in}}W_{l'}^B[0, :]\mathcal{J}_{l,l',1}(X^{(1)})^T$. However, we now no longer have $\frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(1)})||_F^2 = \overline{p_{l,1}^{(n^-, 1)}.p_{l,1}^{(n^-, 1)}}$ but

\begin{eqnarray*}
&=&\limep\limbn\limd\sum_{b=1}^B \overline{p^{(n^-,1)}_{l,b}.p^{(n^-,1)}_{l,b}}\\
&=&\limep\limbn\limd\sum_{b=1}^B \overline{F^-_{l,b}.F^-_{l,b}}\\
&=&\limep\limbn\sum_{b=1}^B \mathfrak{c}_{l,b;l,b}^{-;-}
\end{eqnarray*}

We will have to show that the expression on the last line is $\frac{\mathfrak{g}_l}{\mathfrak{g}_m}$.

Let's start by looking at the readin layers. If $f_l$ if a readin big layer, the distribution of the $p_{l,b}^{(n^-, 1)}$ is the distribution of the segments of $\sqrt{d_\text{in}}W_l^B[0, :]$. Hence, components of $p_{l,1}^{(n^-, 1)}$ are unit Gaussian and the components of $p_{l,b}^{(n^-, 1)}$ with $b \neq 1$ are equal to zero. The unit Gaussians are independent of those corresponding to other readin big layers as there is no weight sharing in $f$. Hence, the entries added to the covariance matrix of $\mathcal{G}$ for the $F_{l,b}^-$ are zero, except the diagonal entry added for $F_{l,1}^-$, which is $\sigma_l^2$. While distribution equality is clear, the major consequence of this is that there is no symmetry between the batch index value 1 and other batch index values for any quantity that depends directly or indirectly on the distribution of the $F^-_{l,b}$ or the $\rho^-_{l,b}(e)$. Thus, we obtain the following distinct $\limd$ quantities: $\tilde{\mathfrak{m}}_{l,1}^-$, $\tilde{\mathfrak{m}}_{l,2}^-$, $\tilde{\mathfrak{m}}_{l,1}^+=\tilde{\mathfrak{m}}_{l,1}$, $\mathfrak{c}_{l,1;l,1}^{-;-}$, $\mathfrak{c}_{l,1;l,2}^{-;-}$, $\mathfrak{c}_{l,2;l,2}^{-;-}$, $\mathfrak{c}_{l,2;l,3}^{-;-}$, $\mathfrak{c}_{l,1;l,1}^{-;+}$, $\mathfrak{c}_{l,1;l,2}^{-;+}$, $\mathfrak{c}_{l,2;l,1}^{-;+}$, $\mathfrak{c}_{l,2;l,2}^{-;+}$, $\mathfrak{c}_{l,2;l,3}^{-;+}$, $\mathfrak{c}_{l,1;l,1}^{+;+}=\mathfrak{c}_{l,1;l,1}$ and $\mathfrak{c}_{l,1;l,2}^{+;+}=\mathfrak{c}_{l,1;l,2}$. The differences arise based on which batch indices are equal to 1 and based on whether they are equal. Because the distribution of the layers in $F^+$ is not affected by the distribution of the layers in $F^-$, $\tilde{\mathfrak{m}}_{l,1}$, $\mathfrak{c}_{l,1;l,1}$ and $\mathfrak{c}_{l,1;l,2}$ continue to represent all batch index combinations for $\limd$ quantities that depend only on $F^+$.

This wider range of limit quantities will unfortunately complicate our limit analysis for BN and activation big layers. Before we go ahead with this analysis, we again document “minor” differences to the analysis in stage 12, just as for the case $m > 0$ above.

* The $F^\text{MN}_b$ do not arise, as they pertain only to the case $m > 0$. Further, we can ignore all arguments that are associated specifically with $m > 0$.

* $n^-$ does not arise in the construction of $F$ and all limit quantities are independent of $n^-$ and $N$. The $n^-\neq n^+$ case becomes the general case.

* The induction is now used to show the following with regards to limits: (i) $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}=\ddot{\mathfrak{c}}_{l,2;l,2}^{-;-}=\ddot{\mathfrak{c}}_{l,2;l,3}^{-;-}=\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}=\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}=\ddot{\mathfrak{c}}_{l,2;l,1}^{-;+}=\ddot{\mathfrak{c}}_{l,2;l,2}^{-;+}=\ddot{\mathfrak{c}}_{l,2;l,3}^{-;+}=0$; (ii) $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} > 0$; (iii) $\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}=\frac{\mathfrak{g}_l}{\mathfrak{g}_m}$; and (iv) $\limbn B\ddot{\mathfrak{c}}_{l,2;l,2}^{-;-} = 0$. From this we obtain $ \limep\limbn\sum_{b=1}^B \mathfrak{c}_{l,b;l,b}^{-;-} = \limep\limbn \mathfrak{c}_{l,1;l,1}^{-;-} + \limep\limbn\frac{B-1}{B} (B\mathfrak{c}_{l,2;l,2}^{-;-}) = \limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}=\frac{\mathfrak{g}_l}{\mathfrak{g}_m}$ as required. When $f_l$ is a readin, FC, bias or addition layer, (i) through (iv) carry over easily from the dependencies or the generator, as in stage 12. For an addition layer, as we have $m=0$, the recursive expansion cannot touch $f_m$. This makes the expansion simpler than in stage 12, as there are no recursive definitions of the $\rho^{(n)}_{\lambda,b}$ to apply. The argument still goes through.

Case: $f_l$ is a BN layer.

The first expansion becomes

\begin{eqnarray*}
&&\rho_{l,1}^- \\
&=&\sum_{f_{l'}} c_{l'}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)(e_{l',1}^--\widehat{e_{l'}^-})\\
&&+\sum_{f_{l'}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{l',l}), Z=|\mathcal{L}|>0} c_{l'}(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
\end{eqnarray*}

That is, we are left with only super terms 1 and 2. Control holds as in the BN layer case of stage 12. Note that the definitions of the multi-activation functions corresponding to $F^-$ are, of course, still fully symmetric with respect to the batch index even if the distribution induced by $e$ is not.

Joint distribution of inputs Compared to the BN layer case of stage 12, the inputs of $\rho_{l,1}^-$ now come only in Gaussian vectors $(e_{l',1}^-, .., e_{l',B}^-, e_{l',1}^+, .., e_{l',B}^+)$, which are independent of each other. Again, the mean is zero. Call the covariance matrix $Q_{l'}$. We decompose the Gaussian vector as follows.

\begin{eqnarray*}
e_{l',1}^+ &=& c^{\{1\}}_{l'}z_{l'}^+\\
e_{l',1}^- &=& c^{\{2\}}_{l'}z_{l'}^+ + c^{\{3\}}_{l'}z_{l'}^-\\
e_{l',b}^+ &=& c^{\{4\}}_{l'}z_{l'}^+ + c^{\{5\}}_{l'}z_{l'}^- + c^{\{6\}}_{l'}s_{l'}^+ + c^{\{11\}}_{l'}u_{l'}^+[b] \text{ for } b \ge 2\\
e_{l',b}^- &=& c^{\{7\}}_{l'}z_{l'}^+ + c^{\{8\}}_{l'}z_{l'}^- + c^{\{9\}}_{l'}s_{l'}^+  + c^{\{10\}}_{l'}s_{l'}^- + c^{\{12\}}_{l'}u_{l'}^+[b] + c^{\{13\}}_{l'}u_{l'}^-[b] \text{ for } b \ge 2\\
\end{eqnarray*}

We override the definition of the $u$, $s$ and $c^{\{.\}}$ from stage 12. $z_{l'}^+$, $z_{l'}^-$, $s_{l'}^+$ and $s_{l'}^-$ are unit Gaussian scalars. $u_{l'}^+$ and $u_{l'}^-$ are $B-1$-dimensional Gaussian vectors with mean zero and a covariance matrix that has diagonal entries $\frac{B-2}{B-1}$ and off-diagonal entries $-\frac{1}{B-1}$. For convenience, their component index ranges from 2 through $B$. An assignment to the 13 $c^{\{.\}}_{l'}$ terms is valid if it satisfies the following 13 constraints: $\mathfrak{c}_{l',1;l',1}^{-;-}=\mathbb{E}_ee_{l',1}^-e_{l',1}^-$, $\mathfrak{c}_{l',1;l',2}^{-;-}=\mathbb{E}_ee_{l',1}^-e_{l',2}^-$, $\mathfrak{c}_{l',2;l',2}^{-;-}=\mathbb{E}_ee_{l',2}^-e_{l',2}^-$, $\mathfrak{c}_{l',2;l',3}^{-;-}=\mathbb{E}_ee_{l',2}^-e_{l',3}^-$, $\mathfrak{c}_{l',1;l',1}^{-;+}=\mathbb{E}_ee_{l',1}^-e_{l',1}^+$, $\mathfrak{c}_{l',1;l',2}^{-;+}=\mathbb{E}_ee_{l',1}^-e_{l',2}^+$, $\mathfrak{c}_{l',2;l',1}^{-;+}=\mathbb{E}_ee_{l',2}^-e_{l',1}^+$, $\mathfrak{c}_{l',2;l',2}^{-;+}=\mathbb{E}_ee_{l',2}^-e_{l',2}^+$, $\mathfrak{c}_{l',2;l',3}^{-;+}=\mathbb{E}_ee_{l',2}^-e_{l',3}^+$, $\mathfrak{c}_{l',1;l',1}=\mathbb{E}_ee_{l',1}^+e_{l',1}^+$, $\mathfrak{c}_{l',1;l',2}=\mathbb{E}_ee_{l',1}^+e_{l',2}^+$, $\mathfrak{c}_{l',1;l',1}=\mathbb{E}_ee_{l',2}^+e_{l',2}^+$ and $\mathfrak{c}_{l',1;l',2}=\mathbb{E}_ee_{l',2}^+e_{l',3}^+$. To start with, we obtain

$$c_{l'}^{\{11\}} = \sqrt{\mathbb{E}_ee_{l',2}^+e_{l',2}^+ - \mathbb{E}_ee_{l',2}^+e_{l',3}^+} = \sqrt{\mathfrak{c}_{l',1;l',1} - \mathfrak{c}_{l',1;l',2}}$$

$$c_{l'}^{\{12\}} = \frac{\mathbb{E}_ee_{l',2}^-e_{l',2}^+ - \mathbb{E}_ee_{l',2}^-e_{l',3}^+}{c^{\{11\}}_{l'}} = \frac{\mathfrak{c}_{l',2;l',2}^{-;+} - \mathfrak{c}_{l',2;l',3}^{-;+}}{\sqrt{\mathfrak{c}_{l',1;l',1} - \mathfrak{c}_{l',1;l',2}}}$$

$$c_{l'}^{\{13\}} = \sqrt{\mathbb{E}_ee_{l',2}^-e_{l',2}^- - \mathbb{E}_ee_{l',2}^-e_{l',3}^- - c^{\{12\}}_{l'}c^{\{12\}}_{l'}}$$

$$= \sqrt{\frac{(\mathfrak{c}_{l',2;l',2}^{-;-} - \mathfrak{c}_{l',2;l',3}^{-;-})(\mathfrak{c}_{l',1;l',1} - \mathfrak{c}_{l',1;l',2}) - (\mathfrak{c}_{l',2;l',2}^{-;+} - \mathfrak{c}_{l',2;l',3}^{-;+})^2}{\mathfrak{c}_{l',1;l',1} - \mathfrak{c}_{l',1;l',2}}}$$

This is reminiscent of $c_{l'}^{\{1\}}$, $c_{l'}^{\{3\}}$ and $c_{l'}^{\{5\}}$ from the BN layer case of stage 12. Let's look at the square roots that arise in the formulas above. Because $Q_{l'}$ is positive semi-definite, the quadratic form defined by $Q_{l'}$ applied to any vector is non-negative. Using this principle on $(0 \times B, 1, -1, 0 \times (B-2))$ yields $\mathfrak{c}_{l',1;l',1}-\mathfrak{c}_{l',1;l',2} \ge 0$. Here, $\times$ refers to the repetition of a vector component. Going forward, we use $\widetilde{\mathsf{expr}}$ to refer to the mean of $\mathsf{expr}$ over batch index values 2 through $B$, i.e. $\widetilde{\mathsf{expr}}=\frac{1}{B-1}\sum_{b'=2}^B \mathsf{expr}|_{b=b'}$. The Cauchy-Schwartz inequality yields $\frac{(B-1)^2}{(B-2)^2}\Big(\mathbb{E}_e(e_{l',2}^- - \widetilde{e_{l'}^-})^2\mathbb{E}_e(e_{l',2}^+ - \widetilde{e_{l'}^+})^2 - (\mathbb{E}_e(e_{l',2}^- - \widetilde{e_{l'}^-})(e_{l',2}^+ - \widetilde{e_{l'}^+}))^2\Big) = (\mathfrak{c}_{l',2;l',2}^{-;-}-\mathfrak{c}_{l',2;l',3}^{-;-})(\mathfrak{c}_{l',1;l',1}-\mathfrak{c}_{l',1;l',2})-(\mathfrak{c}_{l',2;l',3}^{+;-}-\mathfrak{c}_{l',2;l',3}^{+;-})^2 \ge 0$. So expressions to which square roots are applied are indeed non-negative.

Let's look at denominators that arise in the formulas above. When $\mathfrak{c}_{l',1;l',1}-\mathfrak{c}_{l',1;l',2}=0$, the Cauchy-Schwartz-derived relationship above yields $\mathfrak{c}_{l',2;l',2}^{+;-}-\mathfrak{c}_{l',2;l',3}^{+;-}=0$, and hence we can remove $u_{l'}^+$ from the decomposition, which corresponds to setting $c_{l'}^{\{11\}}=c_{l'}^{\{12\}}=0$ and $c_{l'}^{\{13\}}=\sqrt{\mathfrak{c}_{l,2;l,2}^{-;-} - \mathfrak{c}_{l,2;l,3}^{-;-}}$. So we never have to divide by zero.

Let's look at the limits of $c^{\{11\}}_{l'}$, $c^{\{12\}}_{l'}$ and $c^{\{13\}}_{l'}$ with respect to $N$ and $B$. From the induction hypothesis, we have that the individual $\mathfrak{c}^{-;-}_{l',.;l',.}$ and $\mathfrak{c}^{+;-}_{l',.;l',.}$ have valid limits. By non-singularity from stage 9, we have $\ddot{\mathfrak{c}}_{l',1;l',1}-\ddot{\mathfrak{c}}_{l',1;l',2} > 0$. So we obtain that $\ddot{c}^{\{11\}}_{l'}$, $\ddot{c}^{\{12\}}_{l'}$ and $\ddot{c}^{\{13\}}_{l'}$ have identical formulas to $c^{\{11\}}_{l'}$, $c^{\{12\}}_{l'}$ and $c^{\{13\}}_{l'}$ above except that two dots are placed above each $\mathfrak{c}$ term. So, by the induction hypothesis, we further have $\ddot{c}_{l'}^{\{11\}} = \sqrt{\ddot{\mathfrak{c}}_{l',1;l',1} - \ddot{\mathfrak{c}}_{l',1;l',2}}$ and $\ddot{c}_{l'}^{\{12\}} = \ddot{c}_{l'}^{\{13\}} = 0$. Using the principle of the non-negative quadratic form on $(0, 1, -1, 0 \times (2B-3))$ yields $\mathfrak{c}_{l',2;l',2}^{-;-}-\mathfrak{c}_{l',2;l',3}^{-;-} \ge 0$.  Using the principle of the non-negative quadratic form on $(0, 1, 1, 0 \times (2B-3))$ yields $\mathfrak{c}_{l',2;l',2}^{-;-}+\mathfrak{c}_{l',2;l',3}^{-;-} \ge 0$. Combining those results yields $|\mathfrak{c}_{l',2;l',2}^{-;-}|\ge |\mathfrak{c}_{l',2;l',3}^{-;-}| \ge 0$. Since the induction hypothesis yields $\limbn B\mathfrak{c}_{l',2;l',2}^{-;-}=0$, we also have $\limbn B\mathfrak{c}_{l',2;l',3}^{-;-}=0$. Finally, since $c^{\{12\}}_{l'}c^{\{12\}}_{l'}+c^{\{13\}}_{l'}c^{\{13\}}_{l'} = \mathfrak{c}_{l',2;l',2}^{-;-}-\mathfrak{c}_{l',2;l',3}^{-;-}$, we have $\limbn \sqrt{B}c^{\{12\}}_{l'}=\limbn \sqrt{B}c^{\{13\}}_{l'}=0$.

Knowing the value of $c^{\{11\}}_{l'}$, $c^{\{12\}}_{l'}$ and $c^{\{13\}}_{l'}$, an assignment to the other 10 $c^{\{.\}}_{l'}$ terms is valid if they satisfy the following system.

\mathfrak{c}_{l',1;l',1} & \mathfrak{c}_{l',1;l',1}^{-;+} & \mathfrak{c}_{l',1;l',2}  & \mathfrak{c}_{l',2;l',1}^{-;+}\\
\mathfrak{c}_{l',1;l',1}^{-;+} & \mathfrak{c}_{l',1;l',1}^{-;-} & \mathfrak{c}_{l',1;l',2}^{-;+} & \mathfrak{c}_{l',1;l',2}^{-;-} \\
\mathfrak{c}_{l',1;l',2} & \mathfrak{c}_{l',1;l',2}^{-;+} & \frac{B-2}{B-1}\mathfrak{c}_{l',1;l',2}+\frac{1}{B-1}\mathfrak{c}_{l',1;l',1} & \frac{B-2}{B-1}\mathfrak{c}_{l',2;l',3}^{-;+}+\frac{1}{B-1}\mathfrak{c}_{l',2;l',2}^{-;+}\\
\mathfrak{c}_{l',2;l',1}^{-;+} & \mathfrak{c}_{l',1;l',2}^{-;-} & \frac{B-2}{B-1}\mathfrak{c}_{l',2;l',3}^{-;+}+\frac{1}{B-1}\mathfrak{c}_{l',2;l',2}^{-;+} & \frac{B-2}{B-1}\mathfrak{c}_{l',2;l',3}^{-;-}+\frac{1}{B-1}\mathfrak{c}_{l',2;l',2}^{-;-}
\end{pmatrix} = A_{l'}A_{l'}^T$$

$$A_{l'} = \begin{pmatrix}
c^{\{1\}}_{l'} & 0 & 0 & 0 \\
c^{\{2\}}_{l'} & c^{\{3\}}_{l'} & 0 & 0 \\
c^{\{4\}}_{l'} & c^{\{5\}}_{l'} & c^{\{6\}}_{l'} & 0 \\
c^{\{7\}}_{l'} & c^{\{8\}}_{l'} & c^{\{9\}}_{l'} & c^{\{10\}}_{l'} \\
\end{pmatrix}$$

In other words, $A_{l'}A_{l'}^T$ must be the Cholesky decomposition of a matrix. Call it $R_{l'}$ . Let $v$ be an arbitrary 4-dimensional row vector. Then

$$vR_{l'}v^T=v^*Q_{l'}(v^*)^T\ge 0 \text{ where } v^*=(v[2], \frac{v[4]}{B-1}\times (B-1), v[1], \frac{v[3]}{B-1}\times (B-1))$$

as $Q_{l'}$ is positive semi-definite. So $R_{l'}$ is itself positive semi-definite. So we can set unique valid values for $c^{\{1\}}_{l'}$ through $c^{\{10\}}_{l'}$ by letting $A_{l'}A_{l'}^T$ be the Cholesky decomposition where the number of columns that are not all zero equals the rank of $R_{l'}$.

What remains is to investigate the limits of $c_{l'}^{\{1\}}$ through $c_{l'}^{\{10\}}$. We have $c_{l'}^{\{7\}}c_{l'}^{\{7\}} + c_{l'}^{\{8\}}c_{l'}^{\{8\}} + c_{l'}^{\{9\}}c_{l'}^{\{9\}} + c_{l'}^{\{10\}}c_{l'}^{\{10\}} =  \frac{B-2}{B-1}\mathfrak{c}_{l',2;l',3}^{-;-}+\frac{1}{B-1}\mathfrak{c}_{l',2;l',2}^{-;-}$. $\ddot{\mathfrak{c}}_{l',2;l',3}^{-;-}=\ddot{\mathfrak{c}}_{l',2;l',2}^{-;-}=0$ so $\ddot{c}_{l'}^{\{7\}}=\ddot{c}_{l'}^{\{8\}}=\ddot{c}_{l'}^{\{9\}}=\ddot{c}_{l'}^{\{10\}}=0$. Since $\limbn B\ddot{\mathfrak{c}}_{l',2;l',2}^{-;-}=\limbn B\ddot{\mathfrak{c}}_{l',2;l',3}^{-;-}=0$, we also have $\limbn \sqrt{B}c_{l'}^{\{7\}}=\limbn \sqrt{B}c_{l'}^{\{8\}}=\limbn \sqrt{B}c_{l'}^{\{9\}}=\limbn \sqrt{B}c_{l'}^{\{10\}}=0$. For $c_{l'}^{\{1\}}$ through $c_{l'}^{\{6\}}$, we need to use explicit formulas. We obtain

\begin{eqnarray*}
c^{\{3\}}_{l'}&=& \sqrt{\mathfrak{c}_{l',1;l',1}^{-;-} - \frac{\mathfrak{c}_{l',1;l',1}^{-;+}\mathfrak{c}_{l',1;l',1}^{-;+}}{\mathfrak{c}_{l',1;l',1}}}\\
c^{\{4\}}_{l'}&=& \frac{\mathfrak{c}_{l',1;l',2}}{\sqrt{\mathfrak{c}_{l',1;l',1}}}\\
c^{\{5\}}_{l'}&=& \frac{\mathfrak{c}_{l',1;l',2}^{-;+}\mathfrak{c}_{l',1;l',1}-\mathfrak{c}_{l',1;l',1}^{-;+}\mathfrak{c}_{l',1;l',2}}{\mathfrak{c}_{l',1;l',1}c^{\{3\}}_{l'}}\\
c^{\{6\}}_{l'}&=&\sqrt{\frac{B-2}{B-1}\mathfrak{c}_{l',1;l',2}+\frac{1}{B-1}\mathfrak{c}_{l',1;l',1} - c^{\{4\}}_{l'}c^{\{4\}}_{l'} - c^{\{5\}}_{l'}c^{\{5\}}_{l'}}\\
\end{eqnarray*}

Stage 9 yields $\ddot{\mathfrak{c}}_{l',1;l',1} > 0$ and the induction hypothesis of this stage yields $\ddot{\mathfrak{c}}_{l',1;l',1}^{-;-} > 0$ as well as $\ddot{\mathfrak{c}}_{l',1;l',1}^{-;+}=0$. So $\ddot{c}^{\{3\}}_{l'} = \sqrt{\ddot{\mathfrak{c}}_{l',1;l',1}^{-;-}} > 0$, so all denominators in the above formulas have positive limits, so all limits are straightforward and we obtain

\begin{eqnarray*}
\ddot{c}^{\{1\}}_{l'}&=&\sqrt{\ddot{\mathfrak{c}}_{l',1;l',1}}\\
\ddot{c}^{\{2\}}_{l'}&=&0\\
\ddot{c}^{\{3\}}_{l'}&=&\sqrt{\ddot{\mathfrak{c}}_{l',1;l',1}^{-;-}}\\
\ddot{c}^{\{4\}}_{l'}&=& \frac{\ddot{\mathfrak{c}}_{l',1;l',2}}{\sqrt{\ddot{\mathfrak{c}}_{l',1;l',1}}}\\
\ddot{c}^{\{5\}}_{l'}&=& 0\\
\ddot{c}^{\{6\}}_{l'}&=&\sqrt{\frac{\ddot{\mathfrak{c}}_{l',1;l',2}(\ddot{\mathfrak{c}}_{l',1;l',1}-\ddot{\mathfrak{c}}_{l',1;l',2})}{\ddot{\mathfrak{c}}_{l',1;l',1}}}
\end{eqnarray*}

Recursive expansion The first expansion is given above. Replacing the remaining $\rho^+$ terms (not inside a $\nu^+$) with $e^+$ and $\frac{1}{\sqrt{\nu^+}}$ terms is as for super terms 1 and 2 in the BN layer case of stage 12. However, the second expansion in terms of $u$ components and $s$ and $z$ terms is more complex. For now, we simply replace the $e$ terms (not inside a $\nu^+$) with their decomposition in terms of $u$ components and $s$ and $z$ terms. We do not further simplify the resulting expression for now. We thus obtain two kinds of pure terms. “Short terms” originate from super term 1 and “long terms” originate from super term 2. Pure terms are an instance of the expression inside the leading sum(s) as in stage 12. So the second expansion is a sum over a fixed number of 2 kinds of pure terms. Again, they contain a multiplier, normalizer and polynomial. N-sums and N-frequencies are not present. The polynomial can now also contain $s$, $z$ and $c^{\{.\}}$ terms that stem from the replacement of the $e$ terms by their decomposition. In this context, we consider a standard form term to be defined by those conditions.

Expectation limits of standard form terms Relative to the BN layer case of stage 12, multiplying out the polynomial results in a sum of “chains” that can now contain $s$, $z$ and $c^{\{.\}}$ terms in addition to $u$ components. Nonetheless, aggregation works as before except that chains are only considered equivalent if the same $s$, $z$ and $c^{\{.\}}$ terms are present, of which there is a bounded number. The B-frequencies converge. After aggregation, the $c^{\{.\}}$ terms in the aggregated chain can be absorbed into the multiplier, which causes the multiplier to differ from aggregated chain to aggregated chain. Nonetheless, applying lemma <ref> works as before. $\chi_d$ has one component for each distinct $u$ component or $s$ or $z$ term that arises in the aggregated chain. $\hat{\Sigma}$ is still the identity. The valid limit of the expectation of the product of B-frequency, (chain-specific) multiplier, normalizer and ($c^{\{.\}}$-free) aggregated chain is equal to the product of the limit of the B-frequency, the limit of the multiplier, the almost sure limit of the normalizer and the expectation of the aggregated chain with respect to the unit Gaussian.

When investigating limit quantities such as $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+} =\limbn \mathbb{E}_e\rho_{l,1}^-\rho_{l,1}^+$, the recursive expansion of the $\rho_{l,b}^+$ is as in the BN layer case of stage 9 / 12, except that the decomposition of the $e$ terms differs, which yields co-terms that can contain $s$, $z$ and $c^{\{.\}}$ terms in their polynomial in addition to $u$ components. Nonetheless, they are standard form terms in this context, and products of a pure term and a pure term or co-term are standard form terms. This yields that all $\limbn$ quantities of interest are valid as long as the additional $B$ factor in $\limbn B\mathfrak{c}_{l,2;l,2}^{-;-}$ does not cause divergence, which we show below.

Specific expectation limits Let's look at $\ddot{\mathfrak{m}}_{l,1}^-$, $\ddot{\mathfrak{m}}_{l,2}^-$, $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;+}$, $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;+}$, $\ddot{\mathfrak{c}}_{l,2;l,1}^{-;+}$, $\ddot{\mathfrak{c}}_{l,2;l,2}^{-;+}$ and $\ddot{\mathfrak{c}}_{l,2;l,3}^{-;+}$. For each of these, each aggregated chain contains exactly one copy of exactly one of $c^{\{2\}}_{l_1}z_{l_1}^+$, $c^{\{3\}}_{l_1}z_{l_1}^-$, $c^{\{7\}}_{l_1}z_{l_1}^+$, $c^{\{8\}}_{l_1}z_{l_1}^-$, $c_{l_1}^{\{9\}}s_{l_1}^+$, $c^{\{10\}}_{l_1}s_{l_1}^-$, $c^{\{12\}}_{l_1}u_{l_1}^+[b]$ or $c^{\{13\}}_{l_1}u_{l_1}^-[b]$ for some $l_1 < l$. The expectation limit of a product of multiplier, normalizer and aggregated chain is zero if the chain contains any of the above except $c^{\{3\}}_{l_1}z_{l_1}^-$, due to the chain-specific multiplier becoming zero. If the aggregated chain contains $c^{\{3\}}_{l_1}z_{l_1}^-$ and it does not contain another $z_{l_1}^-$ term, the expectation limit is zero due to the single $z_{l_1}^-$ term. Finally, if the aggregated chain contains $c^{\{3\}}_{l_1}z_{l_1}^-$ and it does contain another $z_{l_1}^-$ term, it must also contain a $c^{\{5\}}_{l_1}$ term, which has limit zero. So overall we have $\ddot{\mathfrak{m}}_{l,1}^- = \ddot{\mathfrak{m}}_{l,2}^- = \ddot{\mathfrak{c}}_{l,1;l,1}^{-;+} = \ddot{\mathfrak{c}}_{l,1;l,2}^{-;+} = \ddot{\mathfrak{c}}_{l,2;l,1}^{-;+} = \ddot{\mathfrak{c}}_{l,2;l,2}^{-;+} = \ddot{\mathfrak{c}}_{l,2;l,3}^{-;+} = 0$ as required.

Let's look at $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$, $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}$, $\ddot{\mathfrak{c}}_{l,2;l,2}^{-;-}$ and $\ddot{\mathfrak{c}}_{l,2;l,3}^{-;-}$. Each of $\rho_{l,1}^-\rho_{l,1}^-$, $\rho_{l,1}^-\rho_{l,2}^-$, $\rho_{l,2}^-\rho_{l,2}^-$ and $\rho_{l,2}^-\rho_{l,3}^-$ is a sum of products of two pure terms. As usual, we consider the expectation limit of individual products by pure term type. A product of a short term and a long term as well as a product of a long term and another long term contains in its polynomial a factor stemming from $\widehat{e_{l_1}^-e_{l_2}^+} - \widehat{e_{l_1}^-}\widehat{e_{l_2}^+}$, where $l_1 < l$ and $l_2 < l$ are not necessarily different. We have

\begin{eqnarray*}
&&\widehat{e_{l_1}^-e_{l_2}^+} - \widehat{e_{l_1}^-}\widehat{e_{l_2}^+}\\
&=&\frac{1}{B^2}\Big(B(c^{\{2\}}_{l_1}z_{l_1}^+ + c^{\{3\}}_{l_1}z_{l_1}^-)c^{\{1\}}_{l_2}z_{l_2}^+ + B\sum_{b=2}^B(c^{\{7\}}_{l_1}z_{l_1}^+ + c^{\{8\}}_{l_1}z_{l_1}^- + c^{\{9\}}_{l_1}s_{l_1}^+ + c^{\{10\}}_{l_1}s_{l_1}^- \\
&&+ c^{\{12\}}_{l_1}u_{l_1}^+[b] + c^{\{13\}}_{l_1}u_{l_1}^-[b])(c^{\{4\}}_{l_2}z_{l_2}^+ + c^{\{5\}}_{l_2}z_{l_2}^- + c^{\{6\}}_{l_2}s_{l_2}^+ + c^{\{11\}}_{l_2}u_{l_2}^+[b])\\
&&-(c^{\{2\}}_{l_1}z_{l_1}^+ + c^{\{3\}}_{l_1}z_{l_1}^- + \sum_{b=2}^B(c^{\{7\}}_{l_1}z_{l_1}^+ + c^{\{8\}}_{l_1}z_{l_1}^- + c^{\{9\}}_{l_1}s_{l_1}^+ + c^{\{10\}}_{l_1}s_{l_1}^-\\
&&+ c^{\{12\}}_{l_1}u_{l_1}^+[b] + c^{\{13\}}_{l_1}u_{l_1}^-[b]))(c^{\{1\}}_{l_2}z_{l_2}^+ + \sum_{b=2}^B(c^{\{4\}}_{l_2}z_{l_2}^+ + c^{\{5\}}_{l_2}z_{l_2}^- + c^{\{6\}}_{l_2}s_{l_2}^+ + c^{\{11\}}_{l_2}u_{l_2}^+[b]))\Big)\\
&=&\frac{B-1}{B^2}(c^{\{2\}}_{l_1}z_{l_1}^+ + c^{\{3\}}_{l_1}z_{l_1}^- - c^{\{7\}}_{l_1}z_{l_1}^+ - c^{\{8\}}_{l_1}z_{l_1}^- - c^{\{9\}}_{l_1}s_{l_1}^+ - c^{\{10\}}_{l_1}s_{l_1}^-)(c^{\{1\}}_{l_2}z_{l_2}^+ - c^{\{4\}}_{l_2}z_{l_2}^+\\
&&- c^{\{5\}}_{l_2}z_{l_2}^- - c^{\{6\}}_{l_2}s_{l_2}^+) + \frac{B-1}{B}c^{\{12\}}_{l_1}c^{\{11\}}_{l_2}\widetilde{u_{l_1}^+u_{l_2}^+} + \frac{B-1}{B}c^{\{13\}}_{l_1}c^{\{11\}}_{l_2}\widetilde{u_{l_1}^-u_{l_2}^+}
\end{eqnarray*}

Here we use that $\widetilde{u} = 0$ for all $u$ vectors. Now we form aggregated chains in a 3-step process. First, we aggregate the “partial chains” obtained from multiplying out all of the polynomial except for the factor stemming from $\widehat{e_{l_1}^-e_{l_2}^+} - \widehat{e_{l_1}^-}\widehat{e_{l_2}^+}$. Second, we multiply a copy of each aggregated partial chain with $\frac{B-1}{B^2}(c^{\{2\}}_{l_1}z_{l_1}^+ + c^{\{3\}}_{l_1}z_{l_1}^- - c^{\{7\}}_{l_1}z_{l_1}^+ - c^{\{8\}}_{l_1}z_{l_1}^- - c^{\{9\}}_{l_1}s_{l_1}^+ - c^{\{10\}}_{l_1}s_{l_1}^-)(c^{\{1\}}_{l_2}z_{l_2}^+ - c^{\{4\}}_{l_2}z_{l_2}^+ - c^{\{5\}}_{l_2}z_{l_2}^- - c^{\{6\}}_{l_2}s_{l_2}^+)$, a copy of each aggregated partial chain with $\frac{B-1}{B}c^{\{12\}}_{l_1}c^{\{11\}}_{l_2}\widetilde{u_{l_1}^+u_{l_2}^+}$ and a copy of each aggregated partial chain with $\frac{B-1}{B}c^{\{13\}}_{l_1}c^{\{11\}}_{l_2}\widetilde{u_{l_1}^-u_{l_2}^+}$. Third, we form aggregated chains from each copy. Since the B-frequencies of aggregated partial chains  are convergent, the B-frequencies of aggregated chains stemming from the first copy converge to zero. Aggregated chains stemming from the second and third copy contain $c^{\{12\}}_{l_1}$ and $c^{\{13\}}_{l_1}$ respectively, which converge to zero. So for any type of aggregated chain, the expectation limit is zero.

This leaves products of a short term and another short term. If the big layer indices from the leading sum are different, all aggregated chains contain two single $u$ components or $s$ or $z$ terms, one each for the two big layer indices. The expectation limit is zero. If the big layer indices are both equal to some $l'$, the polynomial stems from $(e_{l',b} - \widehat{e_{l'}})(e_{l',b'} - \widehat{e_{l'}})$, where $(b, b')$ is one of $(1,1)$, $(1,2)$, $(2,2)$ and $(2,3)$ depending on the limit quantity considered. We have

\begin{eqnarray*}
&&e_{l',b} - \widehat{e_{l'}}\\
&=&(\mathbbm{1}_{b=1} - \frac{1}{B})c^{\{2\}}_{l'}z_{l'}^+ + (\mathbbm{1}_{b=1} - \frac{1}{B})c^{\{3\}}_{l'}z_{l'}^- + (\mathbbm{1}_{b\neq 1} - \frac{B-1}{B})c^{\{7\}}_{l'}z_{l'}^+\\
&& + (\mathbbm{1}_{b\neq 1} - \frac{B-1}{B})c^{\{8\}}_{l'}z_{l'}^- + (\mathbbm{1}_{b\neq 1} - \frac{B-1}{B})c^{\{9\}}_{l'}s_{l'}^+ + (\mathbbm{1}_{b\neq 1} - \frac{B-1}{B})c^{\{10\}}_{l'}s_{l'}^-\\
&&+ \mathbbm{1}_{b\neq 1}c^{\{12\}}_{l'}u_{l'}^+[b] + \mathbbm{1}_{b\neq 1}c^{\{13\}}_{l'}u_{l'}^-[b]
\end{eqnarray*}

There are a total of eight terms in this expression. An aggregated chain stems from the product of two of those terms. If one of those terms contains $c^{\{2\}}_{l'}$, $c^{\{7\}}_{l'}$, $c^{\{8\}}_{l'}$, $c^{\{9\}}_{l'}$, $c^{\{10\}}_{l'}$, $c^{\{12\}}_{l'}$ or $c^{\{13\}}_{l'}$, the chain-specific multiplier converges to zero. Otherwise, the B-frequency converges to zero unless $b=b'=1$. So overall we have $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-} = \ddot{\mathfrak{c}}_{l,2;l,2}^{-;-} = \ddot{\mathfrak{c}}_{l,2;l,3}^{-;-} = 0$, and endowing $u$ components and $s$ and $z$ terms with unit Gaussian distributions we have

\begin{eqnarray*}
&=&\sum_{f_{l'}}\Bigg(\frac{\limbn c_{l'}^2c_{l'}^{\{3\}}c_{l'}^{\{3\}}(1 - \frac{1}{B})^2}{\limbas \prod_{f_{l''}\in P_{l',l}} \nu_{l''}^+}\Bigg)\mathbb{E}_zz_{l'}^-z_{l'}^-\\
&=&\sum_{f_{l'}}\frac{c_{l'}^2\ddot{\mathfrak{c}}_{l',1;l',1}}{\prod_{f_{l''}\in P_{l',l}} \ddot{\nu}_{l''}}
\end{eqnarray*}

This is the same expression as for the case $m > 0$ at the beginning of stage 14 except that the recursive expansion cannot hit $f_m$, so again we have $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} = \frac{\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}}{\ddot{\mathfrak{c}}_{k,1;k,1}-\ddot{\mathfrak{c}}_{k,1;k,2}+\epsilon_l}$ and $\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} =\frac{\mathfrak{g}_l}{\mathfrak{g}_m}$ as required. By the induction hypothesis, we have $\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-} > 0$ and by non-singularity from stage 9 we have $\ddot{\mathfrak{c}}_{k,1;k,1}-\ddot{\mathfrak{c}}_{k,1;k,2} \ge 0$, so $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} > 0$ as required.

Finally, let's look at $\limbn B\mathfrak{c}_{l,2;l,2}^{-;-}$. We obtain this limit just as we obtained $\ddot{\mathfrak{c}}_{l,2;l,2}^{-;-} = \limbn \mathfrak{c}_{l,2;l,2}^{-;-}$, except that we absorb the additional $B$ factor either into the B-frequency or chain-specific multiplier before taking their limit, as detailed below. We proceed by pure term type as usual.

A product of a long term and another long term contains in its polynomial two factors stemming from $\widehat{e_{l_1}^-e_{l_2}^+} - \widehat{e_{l_1}^-}\widehat{e_{l_2}^+}$ and $\widehat{e_{l_3}^-e_{l_4}^+} - \widehat{e_{l_3}^-}\widehat{e_{l_4}^+}$ respectively. Both factors expand as shown above. As above, we form aggregated chains in a 3-step process. First, we aggregate “partial chains” from multiplying out the remaining factors. Second, we multiply a copy of each aggregated partial chain with

$$\frac{B-1}{B^2}(c^{\{2\}}_{l_1}z_{l_1}^+ + c^{\{3\}}_{l_1}z_{l_1}^- - c^{\{7\}}_{l_1}z_{l_1}^+ - c^{\{8\}}_{l_1}z_{l_1}^- - c^{\{9\}}_{l_1}s_{l_1}^+ - c^{\{10\}}_{l_1}s_{l_1}^-)(c^{\{1\}}_{l_2}z_{l_2}^+ - c^{\{4\}}_{l_2}z_{l_2}^+ - c^{\{5\}}_{l_2}z_{l_2}^- - c^{\{6\}}_{l_2}s_{l_2}^+)...$$

$$...\frac{B-1}{B^2}(c^{\{2\}}_{l_3}z_{l_3}^+ + c^{\{3\}}_{l_3}z_{l_3}^- - c^{\{7\}}_{l_3}z_{l_3}^+ - c^{\{8\}}_{l_3}z_{l_3}^- - c^{\{9\}}_{l_3}s_{l_3}^+ - c^{\{10\}}_{l_3}s_{l_3}^-)(c^{\{1\}}_{l_4}z_{l_4}^+ - c^{\{4\}}_{l_4}z_{l_4}^+ - c^{\{5\}}_{l_4}z_{l_4}^- - c^{\{6\}}_{l_4}s_{l_4}^+)$$

and a copy of each aggregated partial chain with

$$\frac{B-1}{B^2}(c^{\{2\}}_{l_1}z_{l_1}^+ + c^{\{3\}}_{l_1}z_{l_1}^- - c^{\{7\}}_{l_1}z_{l_1}^+ - c^{\{8\}}_{l_1}z_{l_1}^- - c^{\{9\}}_{l_1}s_{l_1}^+ - c^{\{10\}}_{l_1}s_{l_1}^-)...$$

$$...(c^{\{1\}}_{l_2}z_{l_2}^+ - c^{\{4\}}_{l_2}z_{l_2}^+ - c^{\{5\}}_{l_2}z_{l_2}^- - c^{\{6\}}_{l_2}s_{l_2}^+)(\frac{B-1}{B}c^{\{12\}}_{l_3}c^{\{11\}}_{l_4}\widetilde{u_{l_3}^+u_{l_4}^+} + \frac{B-1}{B}c^{\{13\}}_{l_3}c^{\{11\}}_{l_4}\widetilde{u_{l_3}^-u_{l_4}^+})$$

and a copy of each aggregated partial chain with

$$(\frac{B-1}{B}c^{\{12\}}_{l_1}c^{\{11\}}_{l_2}\widetilde{u_{l_1}^+u_{l_2}^+} + \frac{B-1}{B}c^{\{13\}}_{l_1}c^{\{11\}}_{l_2}\widetilde{u_{l_1}^-u_{l_2}^+})\frac{B-1}{B^2}...$$

$$...(c^{\{2\}}_{l_3}z_{l_3}^+ + c^{\{3\}}_{l_3}z_{l_3}^- - c^{\{7\}}_{l_3}z_{l_3}^+ - c^{\{8\}}_{l_3}z_{l_3}^- - c^{\{9\}}_{l_3}s_{l_3}^+ - c^{\{10\}}_{l_3}s_{l_3}^-)(c^{\{1\}}_{l_4}z_{l_4}^+ - c^{\{4\}}_{l_4}z_{l_4}^+ - c^{\{5\}}_{l_4}z_{l_4}^- - c^{\{6\}}_{l_4}s_{l_4}^+)$$

and a copy of each aggregated partial chain with

$$(\frac{B-1}{B}c^{\{12\}}_{l_1}c^{\{11\}}_{l_2}\widetilde{u_{l_1}^+u_{l_2}^+} + \frac{B-1}{B}c^{\{13\}}_{l_1}c^{\{11\}}_{l_2}\widetilde{u_{l_1}^-u_{l_2}^+})...$$

$$...(\frac{B-1}{B}c^{\{12\}}_{l_3}c^{\{11\}}_{l_4}\widetilde{u_{l_3}^+u_{l_4}^+} + \frac{B-1}{B}c^{\{13\}}_{l_3}c^{\{11\}}_{l_4}\widetilde{u_{l_3}^-u_{l_4}^+})$$

Third, we form aggregated chains from each copy. Since aggregated partial chains have convergent B-frequency, aggregated chains stemming from the first copy have a B-frequency that when multiplied with $B$ converges to zero, because of the two $\frac{B-1}{B^2}$ factors. Aggregated chains stemming from the second or third copy have a B-frequency that when multiplied with $B$ converges, because of the $\frac{B-1}{B^2}$ factor. Further, the chain-specific multiplier for those chains contains either a $c^{\{12\}}$ or $c^{\{13\}}$ term, so it converges to zero. Aggregated chains stemming from the fourth copy contain two $c^{\{12\}}$ terms, two $c^{\{13\}}$ terms or both terms. Hence, the multiplier for those chains times $B$ converges to zero. So no matter the copy, the expectation limit of the product of B-frequency, multiplier, normalizer and aggregated chain is zero.

A product of a short term and a long term contains in its polynomial a factor stemming from $\widehat{e_{l_1}^-e_{l_2}^+} - \widehat{e_{l_1}^-}\widehat{e_{l_2}^+}$ and a factor stemming from $e_{l_3,2}^- - \widehat{e_{l_3}^-}$. The first factor expands as above. The second factor expands as

$$\frac{1}{B}(c^{\{7\}}_{l_3}z_{l_3}^+ + c^{\{8\}}_{l_3}z_{l_3}^- + c^{\{9\}}_{l_3}s_{l_3}^+ + c^{\{10\}}_{l_3}s_{l_3}^- - c^{\{2\}}_{l_3}z_{l_3}^+ - c^{\{3\}}_{l_3}z_{l_3}^-) + c^{\{12\}}_{l_3}u_{l_3}^+[2] + c^{\{13\}}_{l_3}u_{l_3}^-[2]$$

We create aggregated partial chains as above, except that we do not replace batch index values of 2 and replace no batch index value with 2, as the expansion of $e_{l_3,2}^- - \widehat{e_{l_3}^-}$ lacks symmetry with respect to that batch index value. We create four copies of aggregated partial chains as above. For each copy, the argument that the resulting products of B-frequency, multiplier, normalizer and aggregated chain have expectation limit zero is also as above.

This leaves products of a short term and a short term. The polynomial stems from $(e_{l_1,2} - \widehat{e_{l_1}})(e_{l_2,2'} - \widehat{e_{l_2}})$. Each factor expands as above. Again, we create four copies and obtain expectation limits of zero. So overall $\limbn B\mathfrak{c}_{l,2;l,2}^{-;-}=0$ as required.

Case: $f_l$ is an activation layer.

The first expansion becomes

\begin{eqnarray*}
&&\rho_{l,1}^- \\
&=&\sum_{f_{l'}\text{ normed}} c_{l'}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)(e_{l',1}^--\widehat{e_{l'}^-})\tau_l'(\rho_{k,1}^+)\\
&&+\sum_{f_{l'}\text{ normed}}\sum_{\mathcal{L} \in \mathsf{indpow}(P_{l',l}), Z=|\mathcal{L}|>0} c_{l'}(-1)^{|Z|}\Bigg(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Bigg)...\\
&&+\sum_{f_{l'}\text{ unnormed}} c_{l'}e_{l',1}^-\tau_l'(\rho_{k,1}^+)\\
\end{eqnarray*}

That is, we are left with only super terms 1, 2 and 5. Control holds as in the activation layer case of stage 12. The joint distribution of inputs is as in the BN layer case above except that $e_{l',1}^{+\text{in}}$ terms can arise inside of $\tau_l'()$ that are decomposed as $\sigma_{l'}^2s_{l'}^\text{in}$ as in stage 12, where $s_{l'}^\text{in}$ is a unit Gaussian scalar. The second expansion contains short terms and long terms as for the BN layer case above, which now also contain a $\tau_l'(\mathsf{exp}_{l,1})$ term where now

\begin{eqnarray*}
\mathsf{exp}_{l,b}&=&\sum_{f_{l'} \text{ normed}}c_{l'}\Big(\prod_{f_{l''}\in P_{l',l}}\frac{1}{\sqrt{\nu_{l''}^+}}\Big)\Big((\mathbbm{1}_{b=1} - \frac{1}{B})c^{\{1\}}_{l'}z_{l'}^+  + (\mathbbm{1}_{b\neq 1} - \frac{B-1}{B})c^{\{4\}}_{l'}z_{l'}^+ \\
&& + (\mathbbm{1}_{b\neq 1} - \frac{B-1}{B})c^{\{5\}}_{l'}z_{l'}^- + (\mathbbm{1}_{b\neq 1} - \frac{B-1}{B})c^{\{6\}}_{l'}s_{l'}^+ + \mathbbm{1}_{b\neq 1}c^{\{11\}}_{l'}u_{l'}^+[b]\Big)\\
&&\sum_{f_{l'} \text{ unnormed}}c_{l'}\Big(\mathbbm{1}_{b=1}c^{\{1\}}_{l'}z_{l'}^+  + \mathbbm{1}_{b\neq 1}c^{\{4\}}_{l'}z_{l'}^+ + \mathbbm{1}_{b\neq 1} c^{\{5\}}_{l'}z_{l'}^-\\
&& + \mathbbm{1}_{b\neq 1}c^{\{6\}}_{l'}s_{l'}^+ + \mathbbm{1}_{b\neq 1}c^{\{11\}}_{l'}u_{l'}^+[b]\Big) + \sum_{f_{l'}\text{ bias}}c_{l'}\sigma_{l'}^2 s_{l'}^\text{in}
\end{eqnarray*}

Note that each $u$ component or $s$ or $z$ term that arises in this expression for a specific $b$ arises only once. In addition to short terms and long terms, the second expansion now also contains “unnormed terms” that are instances of the expression inside $\sum_{f_{l'}\text{ unnormed}}$. An unnormed term in $\rho_{l,b}^-$ for some $f_{l'}$ is

$$c_{l'}\Big(\mathbbm{1}_{b=1}c^{\{2\}}_{l'}z_{l'}^+ + \mathbbm{1}_{b=1}c^{\{3\}}_{l'}z_{l'}^- + \mathbbm{1}_{b\neq 1}c^{\{7\}}_{l'}z_{l'}^+ + \mathbbm{1}_{b\neq 1} c^{\{8\}}_{l'}z_{l'}^- + \mathbbm{1}_{b\neq 1}c^{\{9\}}_{l'}s_{l'}^+$$

$$ + \mathbbm{1}_{b\neq 1}c^{\{10\}}_{l'}s_{l'}^- + \mathbbm{1}_{b\neq 1}c^{\{12\}}_{l'}u_{l'}^+[b] + \mathbbm{1}_{b\neq 1}c^{\{13\}}_{l'}u_{l'}^-[b]\Big)\tau_l'(\mathsf{exp}_{l,b})$$

Standard form terms can again contain $s$, $z$ and $c^{\{.\}}$ terms in the polynomial, but now they can also contain a tau-deriv $\tau_l'(\mathsf{exp}_{l,1})$. The application of lemma <ref> is as in the activation layer case of stage 12 except (i) different Gaussian variables are present, (ii) no dependency of terms in $\mathsf{exp}_{l,b}$ on $N$ needs to be considered, (iii) chain aggregation works as in the BN layer case above and (iv) $c^{\{.\}}$ terms from the aggregated chain are absorbed into the multiplier. So the valid limit of the expectation of a standard form term is the sum of products of the limit of a B-frequency, the limit of a chain-specific multiplier, the almost sure limit of a normalizer, and the expectation of a product of $c^{\{.\}}$-free aggregated chain and tau-deriv with respect to the unit Gaussian, where the coefficients of the Gaussian variables in the tau-deriv are also replaced by their almost sure limit.

When investigating our limit quantities, as in the activation layer case of stage 12, we consider the expectation limit of products of two pure terms as well as products of a pure term with $\tau_l(\mathsf{exp}_{l,b})$. These products can differ from standard form terms in that they contain $\tau_l'(\mathsf{exp}_{l,b})\tau_l'(\mathsf{exp}_{l,b'})$ or $\tau_l'(\mathsf{exp}_{l,b})\tau_l(\mathsf{exp}_{l,b'})$ instead of just $\tau_l'(\mathsf{exp}_{l,b})$. $b$ and $b'$ depend on the limit quantity considered. In contrast to stage 12, if one of $b$ and $b'$ is equal to 1 and the other is unequal to 1, the coefficient of some Gaussian variables (namely $z^+$ terms) differs between the two $\tau$ terms. This means that we require a generalization of lemma <ref> where we do not just have two functions $G$ and $H$, but a third function as well that is multiplied to the first two inside the expectation that has its own $\omega$ vector. Obtaining such a generalization is straightforward. The proof essentially requires replacing products of two functions with products of three functions and replacing statements about the components of two $\omega$ vectors with statements about the components of three $\omega$ vectors. In the statement of the generalization, we need to extend the list of functions that are assumed to be integrable with respect to any Gaussian measure, but this is still covered by assumption <ref>. The three $\omega$ vectors can then represent the normalizer, the coefficients in the first $\tau$ term and the coefficients in the second $\tau$ term respectively. $H(\mathsf{arg})$ then remains $\tau_l'(\sum_i\mathsf{arg}[i])$ and the third function becomes either $\tau_l'(\sum_i\mathsf{arg}[i])$ or $\tau_l(\sum_i\mathsf{arg}[i])$. In the end, we obtain that all $\limbn$ quantities of interest are valid as long as the additional $B$ factor in $\limbn B\mathfrak{c}_{l,2;l,2}^{-;-}$ does not cause divergence, which we show below.

$\ddot{\mathfrak{m}}_{l,1}^- = \ddot{\mathfrak{m}}_{l,2}^- = \ddot{\mathfrak{c}}_{l,1;l,1}^{-;+} = \ddot{\mathfrak{c}}_{l,1;l,2}^{-;+} = \ddot{\mathfrak{c}}_{l,2;l,1}^{-;+} = \ddot{\mathfrak{c}}_{l,2;l,2}^{-;+} = \ddot{\mathfrak{c}}_{l,2;l,3}^{-;+} = 0$ follows as for the BN layer case above when we also note that $z_{l_1}^-$ cannot have a non-zero coefficient in $\ddot{\mathsf{exp}_{l,b}}$ as $\ddot{c}^{\{5\}}_{l_1}=0$.

Let's look at $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$, $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-}$, $\ddot{\mathfrak{c}}_{l,2;l,2}^{-;-}$, $\ddot{\mathfrak{c}}_{l,2;l,3}^{-;-}$. As in the BN layer case above, products involving only short and long terms become zero due to B-frequency or chain-specific multiplier becoming zero, except for the product of two short terms when $b=b'=1$. This leaves products involving an unnormed term. For a product of an unnormed term and a short or long term, the big layer indices from the leading sum differ. If $l'$ is the big layer index corresponding to the unnormed term, each aggregated chain contains exactly one copy of exactly one of $c^{\{2\}}_{l'}z_{l'}^+$, $c^{\{3\}}_{l'}z_{l'}^-$, $c^{\{7\}}_{l'}z_{l'}^+$, $c^{\{8\}}_{l'}z_{l'}^-$, $c^{\{9\}}_{l'}s_{l'}^+$, $c^{\{10\}}_{l'}s_{l'}^-$, $c^{\{12\}}_{l'}u_{l'}^+[b]$ or $c^{\{13\}}_{l'}u_{l'}^-[b]$. The expectation limit is zero if the chain contains any of the above except $c^{\{3\}}_{l'}z_{l'}^-$, due to the chain-specific multiplier becoming zero. If the aggregated chain contains $c^{\{3\}}_{l'}z_{l'}^-$ and it does not contain another $z_{l'}^-$ term, the expectation limit is zero due to the single $z_{l'}^-$ term, as $z_{l'}^-$ cannot have a non-zero coefficient in $\ddot{\mathsf{exp}_{l,b}}$ as $\ddot{c}^{\{5\}}_{l'}=0$.  Finally, if the aggregated chain contains $c^{\{3\}}_{l'}z_{l'}^-$ and it does contain another $z_{l'}^-$ term, it must also contain another $c^{\{5\}}_{l'}$ term, which has limit zero.

For a product of an unnormed term and an unnormed term, the polynomial stems from $e_{l_1,b}\tau_l(\rho_{k,b}^+)e_{l_2,b'}\tau_l(\rho_{k,b'}^+)$, where $(b, b')$ is one of $(1,1)$, $(1,2)$, $(2,2)$ and $(2,3)$ depending on the limit quantity considered. Expanding and multiplying out $e_{l_1,b}e_{l_2,b'}$ yields $\mathbbm{1}_{b=1}c^{\{3\}}_{l_1}z_{l_1}^-\tau_l'(\mathsf{exp}_{l,b})\mathbbm{1}_{b'=1}c^{\{3\}}_{l_2}z_{l_2}^-\tau_l'(\mathsf{exp}_{l,b'})$ as the only term that does not become zero due to its multiplier. If $(b,b') \neq (1,1)$, it is zero. If $l_1 \neq l_2$, it becomes zero due to the single $z_{l_1}^-$ and $z_{l_2}^-$ terms which cannot have a non-zero coefficient in $\ddot{\mathsf{exp}}_{l,b}$ or $\ddot{\mathsf{exp}}_{l,b'}$. So, overall we have $\ddot{\mathfrak{c}}_{l,1;l,2}^{-;-} = \ddot{\mathfrak{c}}_{l,2;l,2}^{-;-} = \ddot{\mathfrak{c}}_{l,2;l,3}^{-;-} = 0$. For $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}$, combining the non-zero “leftover” from products of two short terms and products of two unnormed terms, endowing $u$ components and $s$ and $z$ terms with unit Gaussian distributions, we obtain

\begin{eqnarray*}
&=&\sum_{f_{l'}}\Bigg(\frac{c_{l'}^2\ddot{c}^{\{3\}}_{l'}\ddot{c}^{\{3\}}_{l'}}{\limbas \prod_{f_{l''}\in P_{l',l}} \nu_{l''}^+}\Bigg)\mathbb{E}_{u,s,z}z_{l'}^-z_{l'}^-\tau_l'(\ddot{\mathsf{exp}}_{l,1})^2\\
&=&\sum_{f_{l'}}\frac{c_{l'}^2\ddot{\mathfrak{c}}_{l',1;l',1}}{\prod_{f_{l''}\in P_{l',l}} \ddot{\nu}_{l''}}\mathbb{E}_{u,s,z}\tau_l'(\ddot{\mathsf{exp}}_{l,1})^2
\end{eqnarray*}

Here, we use again that $z_{l'}^-$ cannot have a non-zero coefficient in $\ddot{\mathsf{exp}}_{l,1}$. This is an analogous expression as for the case $m > 0$ at the beginning of stage 14 except that the recursive expansion cannot hit $f_m$. While the recursive expansion of the content of $\tau_l'()$ differs from the case $m > 0$, its limiting distribution is still $\mathcal{N}(0,\ddot{\mathfrak{c}}_{k,1;k,1})$. So again we have $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} = \ddot{\mathfrak{c}}_{k,1;k,1}^{-;-}\mathbb{E}_{s\sim\mathcal{N}(0,\ddot{\mathfrak{c}}_{k,1;k,1})}\tau_l'(s)^2$ and $\limep \ddot{\mathfrak{c}}_{l,1;l,1}^{-;-}=\frac{\mathfrak{g}_{l}}{\mathfrak{g}_{m}}$ as required. By the induction hypothesis, we have $\ddot{\mathfrak{c}}_{k,1;k,1}^{-;-} > 0$, and by non-singularity from stage 9, we have $\ddot{\mathfrak{c}}_{k,1;k,1} > 0$. Along with proposition <ref> and condition <ref>, we obtain $\ddot{\mathfrak{c}}_{l,1;l,1}^{-;-} > 0$ as required.

Finally, let's look at $\limbn B\mathfrak{c}_{l,2;l,2}^{-;-}$. A product of two pure terms contains in its polynomial two factors that, depending on pure term type, stem from $e_{l_1,2}^-$, $\widehat{e_{l_1}^-e_{l_2}^+} - \widehat{e_{l_1}^-}\widehat{e_{l_2}^+}$ or $e_{l_1,2}^- - \widehat{e_{l_1}^-}$. Expanding and multiplying out those two factors yields a sum of terms where each term contains two of the following: $\frac{B-1}{B^2}$ factor, $\frac{1}{B}$ factor, $c^{\{7\}}$ term, $c^{\{8\}}$ term, $c^{\{9\}}$ term, $c^{\{10\}}$ term, $c^{\{12\}}$ term, $c^{\{13\}}$ term. (This includes the possibility of having two copies of the same factor or term.) The same 3-step process for forming aggregated chains as in the BN layer case above yields $\limbn B\mathfrak{c}_{l,2;l,2}^{-;-} = 0$ as required.

Stage 15: statement (<ref>)

Defining $\mathfrak{c}_{l,1;l,1}^{-;-}$ as in stage 14, we have

\begin{eqnarray*}
&&\limep \limbn \limd \frac{1}{d_l^\text{MF}}\mathbb{E}_X||\mathcal{J}_{l,m,1}(X)||^2_F\\
&=&\limep \limbn \limd\frac{1}{N}\sum_{n=1}^N\frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(n)})||^2_F\\
&=&\limep \limbn \frac{1}{N}\sum_{n=1}^N\limd\frac{1}{d_l^\text{MF}}||\mathcal{J}_{l,m,1}(X^{(n)})||^2_F\\
&=&\limep \limbn \frac{1}{N}\sum_{n=1}^N\begin{cases}\mathfrak{c}_{l,1;l,1}^{-;-} \text{ if } l>m\\1 \text{ else}\end{cases}\\
&=&\limep \limbn\begin{cases}\mathfrak{c}_{l,1;l,1}^{-;-} \text{ if } l>m\\1 \text{ else}\end{cases}\\
\end{eqnarray*}

Stage 16: statements (c) and (d)

Assume now that $f_{L,1}$ is a readout layer. Fix $N$. As in stage 8, we construct a mean field architecture $F$ with a single readout layer. The difference is that now we require multiple finite input layers. We proceed as in stage 9 with the following differences.

* Like $f_{L,1}$, $F_{L,1}$ is now a readout layer of dimensionality $d_L$. The $F_{L,b}$ for $b \ge 2$ are removed from $F$.

* $F$ now has finite input layers $F_{0,b}$ which are jointly associated with $\mathcal{D}^B$ and the batch inputs $X^{(n)}$.

* For a readin layer $f_l$ in $f$, instead of adding input layers $F_{l,b}$ to $F$ and extending $\mathcal{G}$, we add readin layers $F_{l,b}$ with the same width and variance parameter as the $f_{l,b}$ that share weights with each other but not other readin layers in $F$ and have dependency $F_{0,b}$ respectively.

* $\mathcal{G}$ now only covers layers derived from bias big layers.

It is easy to see that, as in stage 9, we have distribution equality. For (c), this means that $f_{L,1}$ and $F_{L,1}$ have the same meta-distribution. For (b), this means that $(f_{L,1}(X^{(1)}), .., f_{L,1}(X^{(N)}))$ has the same distribution as $(F_{L,1}(X^{(1)}), .., F_{L,1}(X^{(N)}))$ for any fixed sample $X^{(1)}, .., X^{(N)}$. Note that as in the statement of background corollary <ref>, the surrogate input generated by $\mathcal{G}$ does not vary between batch inputs $X^{(n)}$, as it represents the random initial bias vectors.

In stage 8, we used theorem <ref> and background corollary <ref> for statements (a) and (b) respectively. Now we require generalized versions that handle the case where there is more than one finite input layer. It is easy to obtain such generalizations. For background corollary <ref>, we simply need to replace $\mathcal{G}(K_\text{in})$ with the proper generator for the wider set of readin layers. The value of the covariance kernel for a pair $(X^{(n)},X^{(n')})$ then depends on all the square means and co-means of the individual inputs in $X^{(n)}$ and $X^{(n')}$. In the context of this stage, $\clipout(\clipin(F))^N$ has $NBI$ input layers that stem from readin layers in $F$, where $I$ is the number of readin big layers in $f$. The proper generator that yields distribution equality is $\mathcal{G}(K(NB, q, c))$, i.e. a mean zero Gaussian where the covariance matrix entry corresponding to $F_{l,b}^{(n)}$ and $F_{l',b'}^{(n')}$ is

$$\sigma_l\sigma_{l'}(\mathbbm{1}_{b=b' \text{ and } n=n'}q + \mathbbm{1}_{b\neq b' \text{ or } n\neq n'}c)\mathbbm{1}_{\text{$l=m$ or $f_l$ and $f_m$ share weights}}$$

Here, we use $n$ and $n'$ to represent duplex indices. As (d) only requires considering almost all samples, we can assume that all square means of individual inputs are $q$ and all co-means of individual inputs are $c$. As opposed to the general case, we can thus define a covariance kernel over only those two parameters.

We base the generalized theorem <ref> on the generalized background corollary. We associate the finite input layers with $\mathcal{D}^B$ and replace $\mathcal{G}(K(N,q,c))$ with $\mathcal{G}(K(NB,q,c))$. The proof goes through as before. The conditions of the generalizations hold because $(\clipout(\clipin(F))^N, \mathcal{G}(K(NB,q,c)) \times \mathcal{G}^N)$ is the mean field architecture and generator we constructed early in stage 12 denoted as $(F^{(1)}, .., F^{(N)})$ for the case $m=L-1$. We proved the conditions in that stage.

So we have that (i) the meta-distribution of $F_{L,1}$ expansion-converges as $d_\text{MF} \rightarrow \infty$ to the elementwise meta-distribution with generator $\mathcal{MN}(\mathfrak{c}_{L,1;L,1}, \mathfrak{c}_{L,1;L,1}^{\prime;\prime\prime})$ and (ii) for almost all samples $(F_{L,1}(X^{(1)}), .., F_{L,1}(X^{(N)}))$ converges in distribution as $d_\text{MF} \rightarrow \infty$ to a Gaussian that is elementwise over output vectors where the generator has mean zero and covariance matrix $K(N, \mathfrak{c}_{L,1;L,1}, \mathfrak{c}_{L,1;L,1}^{\prime;\prime\prime})$. By distribution equality, the same holds for $f_{L,1}$. And, as we showed in stage 9 and 10, $\limep \limb \mathfrak{c}_{L,1;L,1} = \mathfrak{q}_L$ and $\limep \limb \mathfrak{c}_{L,1;L,1}^{\prime;\prime\prime} = \mathfrak{c}_L$. (The limit over $N$ was not used in stages 9 and 10.)

Finally, we note that when $f_{L,1}$ is a readout layer, statements (<ref>) through (<ref>) still hold when $l < L$ as $(f_0, .., f_{L-1})$ is a valid A-architecture without finite output layer.

§.§ Proposition <ref> part 2

mfntCNLC (part 2; see section <ref> for part 1)

Let $f$ be an A-architecture and let $X^{(1)} = (x^{(1,1)}, .., x^{(1,B)})$ and  $X^{(2)} = (x^{(2,1)}, .., x^{(2,B)})$ be two batch inputs. Assume:

* $\mathbb{E}_ix^{(n,b)}[i]x^{(n,b)}[i]=q$ for some $q$ and all $1 \le n \le 2$ and $1 \le b \le B$

* $\mathbb{E}_ix^{(n,b)}[i]x^{(n',b')}[i]=c$ for some $c$ and all $1 \le n, n', \le 2$, $1 \le b, b' \le B$,  except when $(n,b) = (n',b')$ or $(n,n',b,b')=(1,2,1,1)$

* $\mathbb{E}_ix^{(1,1)}[i]x^{(2,1)}[i]=r$ for some $r$

* $q > r \ge c \ge 0$

* The activation functions used in $f$ are piecewise 5-differentiable.

If $f$ does not contain layer normalization but can contain batch normalization layers, we have

$$\lim_{\vec{\epsilon}\rightarrow 0}\Big(\lim_{B\rightarrow\infty}\big(\lim_{d_\text{MF}\rightarrow\infty}\mathbb{E}_if_{l,1}(X^{(1)})[i]f_{l,1}(X^{(2)})[i]=\mathfrak{r}_l  \text{ a.s.}\big)\Big)$$

* ... $\vec{\epsilon}$ is the vector of all regularizers used by normalization layers in $f$.

* ... $0 \le l \le L$

* ... $\mathfrak{r}_l = \begin{cases} r \text{ parameter of inputs if $f_l$ is an input layer} \\  \sigma_l^2\mathfrak{r}_k \text{ if $f_l$ is an FC layer} \\ \mathfrak{C}_{\tau_l}(\mathfrak{q}_k,\mathfrak{r}_k) \text{ if $f_l$ is an activation layer} \\  \mathfrak{r}_k + \sigma_l^2 \text{ if $f_l$ is a bias layer} \\ \frac{\mathfrak{r}_k - \mathfrak{c}_k}{\mathfrak{q}_k - \mathfrak{c}_k} \text{ if $f_l$ is a BN layer} \\  \sum_{\kappa_l=1}^{K_l} w_{l,\kappa_l}^2\mathfrak{r}_{k_l[\kappa_l]} \text{ if $f_l$ is an addition layer} \end{cases}$

* ... randomness is induced by $\theta$ and the $X^{(n)}$.

Define the covariance kernel $\mathfrak{C}(q,c,r)$ as the function that returns $\mathfrak{r}_L$ given $q$, $c$ and $r$. Then we further have

$$\mathfrak{n}(f,q,c) = \sqrt{\frac{\frac{d}{dq'}\mathfrak{C}(q,c,q')|_{q'=q}(q - c)}{\mathfrak{C}(q,c,q) - \mathfrak{C}(q,c,c)}}$$

Remark While we phrase part 2 of this proposition entirely in terms of the three-argument covariance kernel $\mathfrak{C}(q,c,r)$ for simplicity, it is worth noting that $\mathfrak{C}(q,c,q)$ and $\mathfrak{C}(q,c,c)$ are identical to $\mathfrak{C}(q,q)$ and $\mathfrak{C}(q,c)$ respectively, as used in part 1 of the proposition.

It is worth noting that the co-means stipulated by the proposition are achievable. Let $v^\text{all}$, $v^\text{1}$, $v^\text{(1,1)}$, .., $v^\text{(1,B)}$, $v^\text{(2,1)}$, .., $v^\text{(2,B)}$ be a set of orthonormal vectors. Then letting $x^{(n,b)} = \sqrt{c}v^\text{all} + \sqrt{r-c}v^\text{1} + \sqrt{q-r}v^{(n,b)}$ when $b=1$ and $x^{(n,b)} = \sqrt{c}v^\text{all} +\sqrt{q-c}v^{(n,b)}$ otherwise, we obtain the desired co-means.

Like the statement of the proposition, the proof has two parts. The first part is a continuation of the proof of theorem <ref> for the BN case given in section <ref>, i.e. it can be viewed as another stage. The second part of the proof proceeds along the lines of the proof of the first part of this proposition given in section <ref>, and is based only on the calculation rules for the $\mathfrak{r}_l$, $\mathfrak{q}_l$, $\mathfrak{c}_l$ and $\mathfrak{g}_l$ given above and in table <ref>. Because activation functions used are piecewise 5-differentiable, we can use theorem <ref> in this proof.

Part 1: We proceed along the lines of stage 10. The statement trivially holds for $l=0$. $F$ is constructed as in that stage, where again $F'_{l,b}$ corresponds to $f_{l,b}(X^{(1)})$ and $F''_{l,b}$ corresponds to $f_{l,b}(X^{(2)})$. $\mathcal{G}$ is similar. It is a Gaussian with mean zero and covariance matrix as follows. An entry corresponding to layers $F_{l,b}^\mathsf{prime1}$ and $F_{l',b'}^\mathsf{prime2}$ deriving from readin layers is 0 if $l \neq l'$ and is otherwise $q\sigma_l^2$, $r\sigma_l^2$ or $c\sigma_l^2$ depending on how $(\mathsf{prime1},\mathsf{prime2},b,b')$ matches up with the conditions of this proposition. An entry corresponding to $F_{l,b}^{\mathsf{prime1}\text{ in}}$ and $F_{l',b'}^{\mathsf{prime2}\text{ in}}$ deriving from bias layers is 0 if $l \neq l'$ and $\sigma_l^2$ otherwise. An entry corresponding to $F_{l,b}^\mathsf{prime1}$ and $F_{l',b'}^{\mathsf{prime2} \text{ in}}$ is zero. As in stage 10, we have distribution equality, control, $\mathfrak{c}^{\prime;\prime}_{l,1;l,1}=\mathfrak{c}^{\prime\prime;\prime\prime}_{l,1;l,1} = \mathfrak{c}_{l,1;l,1}$ and $\mathfrak{c}^{\prime;\prime}_{l,1;l,2}=\mathfrak{c}^{\prime\prime;\prime\prime}_{l,1;l,2} = \mathfrak{c}_{l,1;l,2}$ where $\mathfrak{c}_{l,1;l,1}$ and $\mathfrak{c}_{l,1;l,2}$ stem from stage 9. A difference to stage 10 is that now there is less symmetry between $F'$ and $F''$. Hence, we now have to consider $\mathbb{E}_e\rho'_{l,1}\rho''_{l,1}$, $\mathbb{E}_e\rho'_{l,1}\rho''_{l,2}$ and $\mathbb{E}_e\rho'_{l,2}\rho''_{l,2}$ separately. The second expression accounts for the case where one batch index is equal to 1 and the third expression accounts for the case where neither batch index is equal to 1. As part of the induction, we show (i) $\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,2}=\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,2;l,2}=\ddot{\mathfrak{c}}_{l,1;l,2}$, (ii) $\limep\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,1} = \mathfrak{r}_l$,  (iii) $\ddot{\mathfrak{c}}_{l,1;l,1} > \ddot{\mathfrak{c}}_{l,1;l,1}^{\prime;\prime\prime} \ge \ddot{\mathfrak{c}}_{l,1;l,2}$ and (iv) $\mathfrak{q}_l > \mathfrak{r}_l \ge \mathfrak{c}_l$. We also use proposition <ref> and non-singularity from stage 9.

* Case: $f_l$ is a readin layer. From table <ref>, we obtain $\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,1}=r\sigma_l^2$ and $\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,2}=\mathfrak{c}^{\prime;\prime\prime}_{l,2;l,2}=c\sigma_l^2=\mathfrak{c}_{l,1;l,2}$. Hence, the same relationships hold for the $\limbn$ and $\limep \limbn$ quantities, which yields (i). Also $\limep \ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,1}=r\sigma_l^2=\mathfrak{r}_l$, so (ii). (iii) follows from $q > r \ge c$. (iv) follows from table <ref>.

* Case: $f_l$ is a fully-connected, non-readin layer. We have $\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,1}=\sigma_l^2\mathfrak{c}^{\prime;\prime\prime}_{k,1;k,1}$, $\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,2}=\sigma_l^2\mathfrak{c}^{\prime;\prime\prime}_{k,1;k,2}$ and $\mathfrak{c}^{\prime;\prime\prime}_{l,2;l,2}=\sigma_l^2\mathfrak{c}^{\prime;\prime\prime}_{k,2;k,2}$. The same relationships hold for the $\limbn$ and $\limep \limbn$ quantities, so (i) and (iii) carry over from $f_k$. Also $\limep \ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,1}=\sigma_l^2\mathfrak{r}_k=\mathfrak{r}_l$, so (ii). (iv) carries over from $f_k$ using table <ref>.

* Case: $f_l$ is a bias layer. We have $\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,1} = \mathbb{E}_e(e_{l,1}^{\prime\text{in}} + \rho_{k,1}')(e_{l,1}^{\prime\prime\text{in}} + \rho_{k,1}'') = \mathfrak{c}^{\prime;\prime\prime}_{k,1;k,1} + \sigma_l^2$ and similarly $\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,2} = \mathfrak{c}^{\prime;\prime\prime}_{k,1;k,2} + \sigma_l^2$ and $\mathfrak{c}^{\prime;\prime\prime}_{l,2;l,2} = \mathfrak{c}^{\prime;\prime\prime}_{k,2;k,2} + \sigma_l^2$. The same relationships hold for the $\limbn$ and $\limep \limbn$ quantities, so (i) and (iii) carry over from $f_k$. Also $\limep \ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,1}=\mathfrak{r}_k + \sigma_l^2=\mathfrak{r}_l$, so (ii). (iv) carries over using table <ref>.

* Case: $f_l$ is an BN layer. $\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,1}$, $\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,2}$ and $\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,2;l,2}$ are obtained analogously to $\ddot{\mathfrak{c}}_{l,1;l,2}$ in stage 9 via lemma <ref>, except that the components of $\chi_d$ are now pairs of $e'_{l',1}-\widehat{e'_{l'}}$ and $e''_{l',1}-\widehat{e''_{l'}}$, $e'_{l',1}-\widehat{e'_{l'}}$ and $e''_{l',2}-\widehat{e''_{l'}}$ and $e'_{l',2}-\widehat{e'_{l'}}$ and $e''_{l',2}-\widehat{e''_{l'}}$ respectively. In the latter two cases, this again yields a diagonal $\hat{\Sigma}$ with entries $\ddot{\mathfrak{c}}_{l',1;l',1} - \ddot{\mathfrak{c}}_{l',1;l',2}$, and hence $\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,2}=\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,2;l,2}=0=\ddot{\mathfrak{c}}_{l,1;l,2}$, so (i). But in the first case it yields a block-diagonal $\hat{\Sigma}$ with off-diagonal entries $\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l',1;l',1} - \ddot{\mathfrak{c}}_{l',1;l',2}$ in each block. Because $\ddot{\mathfrak{c}}_{l',1;l',1} > \ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l',1;l',1} \ge \ddot{\mathfrak{c}}_{l',1;l',2}$, this is positive definite. Hence

$$\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,1}=\sum_{f_{l'}} \frac{c_{l'}^2(\ddot{\mathfrak{c}}_{l',1;l',1}^{\prime;\prime\prime}-\ddot{\mathfrak{c}}_{l',1;l',2})}{\prod_{f_{l''}\in P_{l',l}}\ddot{\mathfrak{c}}_{k'',1;k'',1}-\ddot{\mathfrak{c}}_{k'',1;k'',2}+\epsilon_{l''}}$$

Using the calculation rules or BN, bias and addition layers, it is easy to check that this is equivalent to $\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,1} = \frac{\ddot{\mathfrak{c}}^{\prime;\prime\prime}_{k,1;k,1}-\ddot{\mathfrak{c}}_{k,1;k,2}}{\ddot{\mathfrak{c}}_{k,1;k,1}-\ddot{\mathfrak{c}}_{k,1;k,2}+\epsilon_l}$. So $\limep \ddot{\mathfrak{c}}^{\prime;\prime\prime}_{l,1;l,1} = \frac{\mathfrak{r}_k - \mathfrak{c}_k}{\mathfrak{q}_k - \mathfrak{c}_k} = \mathfrak{r}_l$, so (ii). (iii) carries over from $f_k$ using also $\ddot{\mathfrak{c}}_{l,1;l,2}=0$. (iv) carries over using table <ref>.

* Case: $f_l$ is an addition layer. Analogously to stage 10, $\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,1} = \sum_{\kappa=1}^Kw_{\kappa}^2\mathfrak{c}_{k[\kappa],1;k[\kappa],1}^{\prime;\prime\prime}$, $\mathfrak{c}^{\prime;\prime\prime}_{l,1;l,2} = \sum_{\kappa=1}^Kw_{\kappa}^2\mathfrak{c}_{k[\kappa],1;k[\kappa],2}^{\prime;\prime\prime}$ and $\mathfrak{c}^{\prime;\prime\prime}_{l,2;l,2} = \sum_{\kappa=1}^Kw_{\kappa}^2\mathfrak{c}_{k[\kappa],2;k[\kappa],2}^{\prime;\prime\prime}$. The relationships hold for $\limbn$ and $\limep \limbn$ quantities. (i) and (iii) carry over from the dependencies and (ii) and (iv) carry over using table <ref>.