LR ($\alpha=1/2$, $\gamma=0$), we observe the optimal learning rate shifts
significantly. For standard parametrization without any depth scaling
($\alpha=\gamma=0$), the optimal learning rate exhibits a more significant
shift as depth grows. Moreover, even if one picks the optimal learning rate
for each depth, the performance still degrades when the depth is very large,
suggesting that standard parametrization is not suitable for depth scaling.
Additional figures with multiple time slices are provided in Appendix G.

##### Is feature learning sufficient for HP transfer?

In Section 5, we explained when and why hyperparameter transfer occurs.
Precisely, to obtain HP transfer, one needs to classify all feature learning
limits and choose the optimal one. We introduced the notion of feature
diversity and showed that Depth-$\mu$P is optimal in the sense that it
maximizes feature diversity. To show that optimality is needed for HP
transfer, we train a resnet with $(\alpha,\gamma)=(1,0)$ which is also a
feature learning limit. Figure 11 shows that in this case the learning rate
exhibits a significant shift with depth. Interestingly, the constant $\eta$ in
this case seems to increase with depth, suggesting that the network is trying
to break from the _ODE_ limit, which is sub-optimal. Note that in Figure 10,
with Depth-$\mu$P we obtain better training loss compared to the ODE
parametrization in Figure 11.

Figure 11: Same setup as fig. 10 for the parametrization
$(\alpha,\gamma)=(1,0)$ (the ODE limit).

##### Do we still have transfer with LayerNorm (LN)?

Our theory considers only Mean Substraction (MS), and Figure 10 shows the
results with MS. To see wether LN affects HP transfer, we train resnets with
the same setup as Figure 10 with absolute value non-linearity and LN applied
to $x^{l-1}$ before matrix multiplication with $W^{l}$ (preLN). We keep MS
after non-linearity although it can be removed since LN is applied in the next
layer. Our results, reported in Figure 12 suggest that Depth-$\mu$P guarantees
learning rate transfer with LN as well.

Figure 12: Same setup as Figure 10 with Abs non-linearity instead of ReLU and
LayerNorm applied to $x^{l-1}$ before matrix multiplication with $W^{l}$. We
show the logloss versus the learning rate of the hidden layers (input/output
layers fixed) for two parametrizations: Depth-$\mu$P (Left) and scaling only
the blocks without LR scaling ($(\alpha,\gamma)=(1/2,0)$) (Right). The results
show that Depth-$\mu$P preserves the optimal learning rate while consistently
improving the training loss as depth increases. If we only scale the blocks
without scaling the LR ($\alpha=1/2,\gamma=0$) when training with Adam, the
optimal learning rate shifts significantly with depth.

##### Block multiplier transfer ($a$).

In Figure 13, we investigate the stability of the hyperparameter $a$ in
Depth-$\mu$P as depth increases. The results suggest that the optimal value of
this constant converges as depth grows, which suggest transferability.
Additional experiments with multiple time slices are provided in Appendix G.

Figure 13: Train logloss versus block multiplier $a$ for varying depths. Same
training setup as in fig. 10. The results suggest that Depth-$\mu$P stabilizes
the hyperparameter $a$ as depth increases.

###  10.3 What Happens in a Transformer?

Figure 14: Modern transformers are insensitive to block multiplier $a$.

Figure 15: In (Megatron) Transformer trained on Common Crawl, deeper does
worse initially (Left) but eventually does better (Right).

Figure 16: In the middle of (Megatron) transformer training, optimal learning
rate is approximately invariant (Left), while at the end of training, it
approximately scales like $1/\sqrt{L}$. However, the $1/\sqrt{L}$ scaling
transfers the maximum viable learning rate better in either case.

Because transformers have block depth 2, as discussed in section 9, we have
plenty of reasons to suspect that no parametrization of (learning rate, block
multiplier) will be able to robustly transfer hyperparameters across depth for
transformers.

Here we do a large scale experiment using Megatron trained on Common Crawl and
catalogue our observations.252525We train the models for 3900 steps, using
cosine decay schedule with 500 warmup steps. We use a sequence length of 4096,
batch size 256, resulting in approximately 4B tokens per training run. In
summary, in our particular setup (which should be close to most large language
model pretraining), we see that the $1/\sqrt{L}$ scaling seems to transfer
hyperparameters at the end of training (LABEL:{fig:megatron-scaling-
shifts}(Right)). However, we also see that 1) deeper does worse in initial
training (LABEL:{fig:megatron-deeper-worse}(Left)), and 2) optimal
hyperparameters scale like $\Theta(1)$ in the middle of training (Figure
16(Left)). Combined with the theoretical insights of Section 9, this leads us
to conclude that while the $1/\sqrt{L}$ scaling can potentially be practically
useful in transformer training, it is likely to be brittle to architectural
and algorithmic changes, or even simple things like training time.

In fact, we observe that transformers are insensitive to the block multiplier
$a$ (Figure 14), so that the only relevant hyperparameter is really just
learning rate. Thus, empirically measuring the scaling trend of the optimal
learning rate, as done in modern large scale pretraining, can be a practically
more robust way to transfer hyperparameters.

Here $L$ is the number of transformer layers, each of which consists of an
attention layer and an MLP layer (each of which has depth 2).

###  10.4 Feature Diversity

In this section, we empirically verify our claims about feature diversity
exponent (Claims 7.4 and 7.5). We use the same setup as in the last section,
i.e., we train deep residual networks of width $n=256$ on CIFAR-10 dataset
with Adam and batch size $64$. In Figure 17, we compare two parametrizations,
Depth-$\mu$P ($\alpha=\gamma=1/2$) and the ODE parametrization
$(\alpha,\gamma)=(1,0)$. We measure
$\left\|\bm{x}^{\lfloor(\lambda+\epsilon)L\rfloor}_{t}-\bm{x}_{t}^{\lfloor\lambda
L\rfloor}\right\|\mathbin{\overset{\text{{def}}}{=}}d(\epsilon)$ at $t=1000$
for the two parametrizations and varying depth. For each parametrization and
depth $L$, we rescale function $d$ by multiplying a constant $c$ such that
$c\cdot d(1/256)=1$, and then plot the rescaled function $c\cdot d$ for a
clean presentation. One can observe clearly that Depth-$\mu$P has feature
diversity exponent (almost) $1/2$ for any $L$, while the curves for ODE
parametrization move from $\epsilon^{1/2}$ to $\epsilon$ when $L$ grows. This
exactly fits our theory that Depth-$\mu$P maximizes the feature diversity,
while other parametrizations (even with feature learning) have smaller feature
diversity exponents that should go to $0$ in the infinite depth limit.

##### Growth along with $L$ and $t$.

In Figure 18, we measure $d(\epsilon)$ at $t=100,500,1000$, and rescale it by
_dividing_ _additional_ $\epsilon^{0.5}$ and a constant $c$ such that
$\frac{d(1/256)}{c\cdot\epsilon^{0.5}}=1$, and then plot the rescaled function
$d/(c\cdot\epsilon^{0.5})$ for a clean comparison between $d$ and
$\epsilon^{0.5}$. We observe that for both Depth-$\mu$P and ODE
parametrization, the slopes of the curves grow along with $L$ and $t$. The
growth along $t$ can be explained by the cumulative correlation between
layers. The growth along $L$ for ODE parametrization is because the
independent components between nearby layers decrease when $L$ grows. We do
not have a clear understanding for the growth along $L$ for Depth-$\mu$P and
we leave it as a future work.

##### Absolute value activation increases feature diversity.

In Figure 19, we plot the same curves as in Figure 18 but comparing ReLU
activation and absolute value activation under Depth-$\mu$P. We observe that
the slope of the curves for absolute value activation is smaller than ReLU
activation. It matches our theory that absolute value activation increases
feature diversity.

Figure 17: Difference between feature at layer $\lfloor\lambda L\rfloor$ and
feature at layer $\lfloor(\lambda+\epsilon)L\rfloor$ as a curve of $\epsilon$
for width $n=256$ and varying depths. For a clean presentation, each curve is
scaled by a constant so it always passes $(1/256,1)$. The feature diversity
exponent $\kappa$ depends on the growth of the curve when $L\to\infty$. For
Depth-$\mu$P (left), the curve is always close to $\epsilon^{1/2}$, meaning
$\kappa=1/2$. For ODE parametrization (right), the curve shifts from
$\epsilon^{1/2}$ to $\epsilon$ when $L$ grows, indicating its $\kappa$ goes to
$0$ in the infinite depth limit.  Figure 18: Same setup as Figure 17 but at
step $t=100,500,1000$, and each curve is scaled by dividing a constant and
_additional_ $\epsilon^{1/2}$ so it always passes $(1/256,1)$. The curve
indicating feature diversity exponent $\kappa$ exactly $1/2$ should be a
horizontal line at $1$. For Depth-$\mu$P ($\alpha=0.5$), the curves are almost
horizontal. For ODE parametrization ($\alpha=1$), slopes of the curves are
larger with larger $L$ and larger $t$. Figure 19: Same setup as Figure 18, but
comparing Depth-$\mu$P with ReLU activation and absolute value activation.
Each curve is scaled by dividing a constant and $\epsilon^{1/2}$ so it always
passes $(1/256,1)$. The curve indicating feature diversity exponent $\kappa$
exactly $1/2$ should be a horizontal line at $1$. For both activations, slopes
of curves are small, but growing along with $L$ and $t$. The slopes with
absolute value activation ($\phi=\mathrm{Abs}$) are slower than the slopes
with ReLU activation ($\phi=\mathrm{ReLU}$), indicating feature diversity is
higher with absolute value activation.

## Acknowledgement

We thank Huishuai Zhang, Jeremy Bernstein, Edward Hu, Michael Santacroce,
Lucas Liu for their helpful comments and discussion. D. Yu was supported by
NSF and ONR. Part of this work was done during D. Yu’s internship at
Microsoft.

## Author Contributions

GY developed the core theory and ran experiments in early part of the
exploratory stage and most experiments in the final draft. DY worked on and
proved key claims for linear resnets (including the limiting equations,
convergence, and classification of parametrization), drafted the very first
version of the paper, and ran experiments verifying the theoretical claims
(including the convergence of linear case and feature diversity separation).
CZ ran experiments in later part of the exploratory stage. They revealed the
viability of Depth-$\mu$P in the block depth 1 case, in contrast to the
general block depth case. CZ also ran the Megatron experiments in the final
version of the paper. SH contributed to brainstorming since the beginning of
the project, wrote the warm-up section on linear networks, formalized the
notion of feature diversity exponent, and helped transforming experimental
results into plots and visualizations.

## References

  * Allen-Zhu et al. [2019] Z. Allen-Zhu, Y. Li, and Z. Song.  A convergence theory for deep learning via over-parameterization, 2019.
  * Chizat and Bach [2018] L. Chizat and F. Bach.  On the global convergence of gradient descent for over-parameterized models using optimal transport, 2018.
  * Chizat et al. [2020] L. Chizat, E. Oyallon, and F. Bach.  On lazy training in differentiable programming, 2020.
  * Hanin and Rolnick [2018] B. Hanin and D. Rolnick.  How to start training: The effect of initialization and architecture, 2018.
  * Hayou [2023] S. Hayou.  On the infinite-depth limit of finite-width neural networks.  _Transactions on Machine Learning Research_ , 2023.
  * Hayou and Yang [2023] S. Hayou and G. Yang.  Width and depth limits commute in residual networks.  In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 12700–12723. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/hayou23a.html.
  * Hayou et al. [2021] S. Hayou, E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau.  Stable resnet.  In A. Banerjee and K. Fukumizu, editors, _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics_ , volume 130 of _Proceedings of Machine Learning Research_ , pages 1324–1332. PMLR, 13–15 Apr 2021.  URL https://proceedings.mlr.press/v130/hayou21a.html.
  * He et al. [2016a] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016a.
  * He et al. [2016b] K. He, X. Zhang, S. Ren, and J. Sun.  Identity mappings in deep residual networks.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14_ , pages 630–645. Springer, 2016b.
  * Jacot et al. [2020] A. Jacot, F. Gabriel, and C. Hongler.  Neural tangent kernel: Convergence and generalization in neural networks, 2020.
  * Jelassi et al. [2023] S. Jelassi, B. Hanin, Z. Ji, S. J. Reddi, S. Bhojanapalli, and S. Kumar.  Depth dependence of $\mu$p learning rates in relu mlps, 2023.
  * Liu et al. [2020] L. Liu, X. Liu, J. Gao, W. Chen, and J. Han.  Understanding the difficulty of training transformers.  _arXiv preprint arXiv:2004.08249_ , 2020.
  * Noci et al. [2022] L. Noci, S. Anagnostidis, L. Biggio, A. Orvieto, S. P. Singh, and A. Lucchi.  Signal propagation in transformers: Theoretical perspectives and the role of rank collapse, 2022.
  * Noci et al. [2023] L. Noci, C. Li, M. B. Li, B. He, T. Hofmann, C. Maddison, and D. M. Roy.  The shaped transformer: Attention models in the infinite depth-and-width limit, 2023.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Shoeybi et al. [2019] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  _arXiv preprint arXiv:1909.08053_ , 2019.
  * Silver et al. [2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis.  Mastering the game of go with deep neural networks and tree search.  _Nature_ , 529:484–489, 2016.
  * Srivastava et al. [2015] R. K. Srivastava, K. Greff, and J. Schmidhuber.  Highway networks, 2015.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need, 2017.
  * Yang [2020a] G. Yang.  Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation, 2020a.
  * Yang [2020b] G. Yang.  Tensor programs ii: Neural tangent kernel for any architecture, 2020b.
  * Yang [2021] G. Yang.  Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes, 2021.
  * Yang and Hu [2021] G. Yang and E. J. Hu.  Tensor programs iv: Feature learning in infinite-width neural networks.  In _International Conference on Machine Learning_ , pages 11727–11737. PMLR, 2021.
  * Yang and Littwin [2023] G. Yang and E. Littwin.  Tensor programs ivb: Adaptive optimization in the infinite-width limit, 2023.
  * Yang et al. [2022] G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao.  Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.  _arXiv preprint arXiv:2203.03466_ , 2022.
  * Zhang et al. [2019] H. Zhang, Y. N. Dauphin, and T. Ma.  Fixup initialization: Residual learning without normalization, 2019.
  * Zhang et al. [2023] H. Zhang, D. Yu, M. Yi, W. Chen, and T.-Y. Liu.  Stabilize deep resnet with a sharp scaling factor $\tau$, 2023.
  * Zou et al. [2018] D. Zou, Y. Cao, D. Zhou, and Q. Gu.  Stochastic gradient descent optimizes over-parameterized deep relu networks, 2018.

##  Appendix A Notations

This section provides an introduction to the new TP notations from [24]. We
only require the definition of the inner and outer products in this paper.

##### Averaging over $n$

When $x\in\mathbb{R}^{n}$, we always use greek subscript
$\alpha,\beta,\ldots\in[n]$ to index its entries. Then $\langle
x_{\alpha}\rangle_{\alpha}$ denotes its average entry. This notation will only
be used to average over $n$-dimensions, but not over constant dimensions.

###  A.1 The Tensor Program Ansatz: Representing Vectors via Random Variables

From the Tensor Programs framework [25], we know that as width becomes large,
the entries of the (pre-)activation vectors and their gradients will become
roughly iid, both at initialization and training. Hence any such vector’s
behavior can be tracked via a random variable that reflects the distribution
of its entries. While we call this the “Tensor Program Ansatz”, it is a
completely rigorous calculus.

####  A.1.1 Ket Notation

Concretely, if $x\in\mathbb{R}^{n}$ is one such vector, then we write
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle\in\mathbb{R}$
(called a _ket_) for such a random variable, such that $x$’s entries look like
iid samples from
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle$.For
any two such vectors $x,y\in\mathbb{R}^{n}$,
$(x_{\alpha},y_{\alpha})\in\mathbb{R}^{2}$ for each $\alpha$ will look like
iid samples from the random vector
$(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle)$,
such that, for example,
$\lim_{n\to\infty}\frac{x^{\top}y}{n}=\operatorname*{\mathbb{E}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle\cdot\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle$,
which we write succinctly as just $\langle
x\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle$.
Here $\langle
x\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
is called a _bra_ , interpreted as a sort of “transpose” to
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle$.
In our convention,
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle$
is always a random variable independent of $n$ and $x$ always has $\Theta(1)$
typical entry size.262626i.e., $\|x\|^{2}/n=\Theta(1)$ as $n\to\infty$.

This notation can be generalized to the case where
$\bm{x}\in\mathbb{R}^{n\times k},\bm{y}\in\mathbb{R}^{n\times j}$. In this
case, we can think of
$\langle\bm{x}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{y}\rangle$
as the $k\times j$ matrix given by $(\langle
x_{a}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y_{b}\rangle)_{\begin{subarray}{c}1\leq
a\leq k\\\ 1\leq b\leq j\end{subarray}}$.

Because we will often need to multiply a ket with a diagonal matrix, we
introduce a shorthand:

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle_{\bm{\chi}}=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\mathrm{Diag}(\bm{\chi}),$
(7)

if $\bm{x}$ is $n\times k$ and $\bm{\chi}$ is a $k$-dimensional vector.

####  A.1.2 Outer Product

Likewise, if both $\bm{x}$ and $\bm{y}$ have shape $n\times k$, the expression

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\
\text{ represents the limit of $\bm{x}\bm{y}^{\top}\in\mathbb{R}^{n\times
n}$.}$

More formally,
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
is defined as an operator that takes a ket
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle\in\mathbb{R}^{j}$
and return the ket

$(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}})\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle(\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle)\in\mathbb{R}^{j}$

i.e., it returns the random vector
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\in\mathbb{R}^{k}$
multiplied by the deterministic matrix
$\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle\in\mathbb{R}^{k\times
j}$ on the right. This corresponds to the limit of
$\bm{x}\bm{y}^{\top}\bm{z}/n$. Likewise,
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
acts on a bra
$\langle\bm{w}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\in\mathbb{R}^{j}$
by

$\langle\bm{w}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}})=(\langle\bm{w}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle)\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\in\mathbb{R}^{j}.$

which corresponds to the limit of
$\frac{1}{n}\bm{w}^{\top}\bm{x}\bm{y}^{\top}$. This definition of
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
makes the expressions

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle,\quad\langle\bm{w}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}},\quad\langle\bm{w}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle$

unambiguous (since any way of ordering the operations give the same answer).

######  Remark A.1 (Potential Confusion).

One should _not_ interpret
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
as the scalar random variable
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\cdot\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{y}\rangle=\sum_{i=1}^{k}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x^{i}\rangle\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y^{i}\rangle$,
which would act on a ket
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle$
to produce
$(\langle\bm{x}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\cdot\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}})\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle=\operatorname*{\mathbb{E}}(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\cdot\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{y}\rangle)\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle$,
which is deterministic. On the other hand,
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle$
is always a linear combination of
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle$,
a nondeterministic random variable in general. In particular, any correlation
between
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle$
and
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{y}\rangle$
does not directly play a role in their outer product
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$:
we always have
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern 0.5pt}}}$, where
$(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{y}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}},\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern 0.5pt}}})$ is an iid copy of
$(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{y}\rangle,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle)$
independent from
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle$.

##### Outer Product with Diagonal Inserted

Finally, if $\bm{\chi}\in\mathbb{R}^{k}$ is deterministic, then (consistent
with eq. 7) we define
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle{}_{\bm{\chi}}\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
as the operator that acts on kets
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle\in\mathbb{R}^{j}$
by

$(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle{}_{\bm{\chi}}\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}})\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle{}_{\bm{\chi}}\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}{\bm{z}}\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\mathrm{Diag}(\bm{\chi})(\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle)\in\mathbb{R}^{j}.$

Morally,
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle{}_{\bm{\chi}}\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
is just a shorter way of writing
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\mathrm{Diag}(\bm{\chi})\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
and represents the limit of $\bm{x}\mathrm{Diag}(\bm{\chi})\bm{y}^{\top}$. In
particular,
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle{}_{\bm{1}}\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$.

####  A.1.3 Nonlinear Outer Product

If $xy^{\top}\in\mathbb{R}^{n\times n}$ is the (linear) outer product of two
vectors $x\in\mathbb{R}^{n}$ and $y\in\mathbb{R}^{n}$, then $\phi(xy^{\top})$,
the entrywise application of nonlinear $\phi:\mathbb{R}\to\mathbb{R}$ to
$xy^{\top}$, is a kind of _nonlinear outer product_.Passing to the ket
notation, in general we define
$\phi\left(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\right)$
as the operator that acts on kets as

$\phi\left(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\right)\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle\mathrel{\raisebox{-1.29167pt}{$\mathbin{\overset{\text{{def}}}{=}}$}}\operatorname*{\mathbb{E}}_{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}}\phi\left(\sum_{i=1}^{k}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x^{i}\rangle\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y^{i}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}}\right)\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern 0.5pt}}}$

where
$\left(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y^{1}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}},\ldots,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y^{k}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}},\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern 0.5pt}}}\right)$ is an iid copy of
$\left(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y^{1}\rangle,\ldots,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y^{k}\rangle,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle\right)$
independent from
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle$
and the expectation is taken only over the former. This is just like, in the
finite $n$ case,

$\phi\left(\bm{x}\bm{y}^{\top}\right)\bm{z}/n=\phi\left(\sum_{i=1}^{k}x^{i}{y^{i\top}}\right){\bm{z}}/n.$

Moreover, if
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{w}\rangle\in\mathbb{R}^{j},\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle\in\mathbb{R}^{k}$,
then

$\displaystyle\langle\bm{w}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\phi\left(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\right)\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle$
$\displaystyle=\langle\bm{w}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\phi\left(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}}\right)\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern 0.5pt}}}\in\mathbb{R}^{j\times k}$
$\displaystyle=\operatorname*{\mathbb{E}}\phi\left(\sum_{i=1}^{k}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x^{i}\rangle\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y^{i}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}}\right)\left(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{w}\rangle\otimes\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern 0.5pt}}}\right)$

where $\otimes$ denotes outer product of vectors and expectation is taken over
everything.

More generally, if $\phi:\mathbb{R}^{t}\to\mathbb{R}$, then
$\phi\Big{(}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}_{1}\rangle\langle\bm{y}_{1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}},\ldots,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}_{t}\rangle\langle\bm{y}_{t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\Big{)}$
is an operator taking kets to kets, defined by

$\displaystyle\phi\Big{(}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}_{1}\rangle\langle\bm{y}_{1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}},\ldots,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}_{t}\rangle\langle\bm{y}_{t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\Big{)}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle\mathrel{\raisebox{-1.29167pt}{$\mathbin{\overset{\text{{def}}}{=}}$}}\operatorname*{\mathbb{E}}_{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}}\phi\left(\sum_{i=1}^{k}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{1}^{i}\rangle\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y_{1}^{i}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}},\ldots,\sum_{i=1}^{k}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{i}\rangle\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y_{t}^{i}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern
0.5pt}}}\right)\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{z}\rangle^{\mathord{\mathchoice{\kern
1.0pt\framebox{$\displaystyle 1$}\kern 1.0pt}{\kern 1.0pt\framebox{$\textstyle
1$}\kern 1.0pt}{\kern 0.7pt\framebox{$\scriptstyle 1$}\kern 0.7pt}{\kern
0.5pt\framebox{$\scriptscriptstyle 1$}\kern 0.5pt}}}$

######  Remark A.2 (Potential Confusion).

Note
$\phi(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}})$
is not the image of the operator
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\langle\bm{y}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
under $\phi$ in the continuous function calculus of operators, but rather a
“coordinatewise application” of $\phi$. For example, if $\phi(t)=t^{2}$, then
$\phi(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle\langle
y\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}})$
_is not_
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle\langle
y\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle\langle
y\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$,
the latter being what typically “squaring an operator” means, but rather
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle^{2}\langle
y\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}^{2}=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\odot
x\rangle\langle y\odot
y\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$.

####  A.1.4 Comparison with Previous $Z^{\bullet}$ Notation

For readers familiar with the _Tensor Programs_ papers, this new “bra-ket”
notation (aka Dirac notation) relates to the old $Z^{\bullet}$ notation by

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle=Z^{x},\quad\langle
x\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle=\operatorname*{\mathbb{E}}Z^{x}Z^{y}.$

The new notation’s succinctness of expectation inner product should already be
apparent. Furthermore, the old notation is not very compatible with multi-
vectors whereas
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle$
makes it clear that $\rangle$ represents the constant dimension side.
Consequently, (nonlinear) outer product is awkward to express in it,
especially when its contraction with random variables requires an explicit
expectation symbol $\operatorname*{\mathbb{E}}$.

##  Appendix B Infinite-Width Limit with the Bra-ket notation

As before, when the width $n$ of the program goes to infinity, one can infer
how the program behaves via a calculus of random variables. We define them
below via the new ket notation instead of the earlier $Z$ notation.

##### Ket Construction.

We recursively define the random variable
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle$
(called a _ket_) for each vector $x$ and deterministic number
$\mathring{\theta}$ for each scalar $\theta$ in the program. For a vector $Wx$
in the program, we also define random variables
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu$ and
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\dot{\rangle\mkern-3.0mu}\mkern
3.0mu$ (called _hat-ket_ and _dot-ket_ respectively) such that
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu+\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\dot{\rangle\mkern-3.0mu}\mkern
3.0mu$. These are the same as $\widehat{Z}$ and $\dot{Z}$ in the old TP
notation [25] and they satisfy

  * Hat

All hat-kets are jointly Gaussian with zero-mean and covariance272727In eq. 8,
$\mathbb{I}(W=U)$ is the deterministic number that is 1 iff $W$ and $U$ are
the same matrix (as symbols in the program) and 0 otherwise. This should _not_
be interpreted as a random variable that is 1 precisely when $W$ and $U$ take
the same values.

$\displaystyle\mathrm{Cov}(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Uy\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu)$ $\displaystyle=\mathbb{I}(W=U)\langle
x\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle$
(8)

  * Dot

Every dot-ket is a linear combination of previous kets, expressed by the
following equation

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\dot{\rangle\mkern-3.0mu}\mkern
3.0mu\mathrel{\raisebox{-1.29167pt}{$\mathbin{\overset{\text{{def}}}{=}}$}}\sum_{y\in\bm{x}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle\operatorname*{\mathbb{E}}\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W^{\top}y\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}$ (9)

eq. 9 is the same equation as in[25, Zdot] but formulated much more succinctly
in the bra-ket notation:

$\displaystyle\text{\cite[citep]{[\@@bibref{Number}{yang2022tensor}{}{},
Zdot]}},\quad\dot{Z}^{Wx}$
$\displaystyle=\sum_{y\in\bm{x}}Z^{y}\operatorname*{\mathbb{E}}\frac{\partial
Z^{x}}{\partial\widehat{Z}^{W^{\top}y}}.$

There is an alternative notion for
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\dot{\rangle\mkern-3.0mu}\mkern
3.0mu$ in Yang and Littwin [24] that write

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\dot{\rangle\mkern-3.0mu}\mkern
3.0mu=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bm{x}\rangle\check{\langle}W^{\top}\bm{x}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle.$

This is more convenient to write as we introduce the operator view.

We can see the ket
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\rangle$
as the result of the action of an operator on the ket
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle$.

######  Definition B.1.

Let $W$ be an initial matrix in a Tensor Program. We define
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}},\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}},\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}$
to be the linear operators on kets 282828 To be rigorous, we need to specify
the “Hilbert space” of kets. This is somewhat pedantic and not crucial to the
key points of this paper, but the Hilbert space can be constructed as follows:
Let $\sigma(\pi)$ be the $\sigma$-algebra generated by the kets of the program
$\pi$. Let
$\Sigma(\pi)\mathrel{\raisebox{-1.29167pt}{$\mathbin{\overset{\text{{def}}}{=}}$}}\bigcup_{\pi^{\prime}\supseteq\pi}\sigma(\pi)$
be the union (more precisely, the direct limit) of $\sigma(\pi^{\prime})$ over
all programs $\pi^{\prime}$ extending $\pi$. Then the Hilbert space in
question is the $L^{2}$ space of random variables over the $\Sigma$ of our
program.  that act by

$\displaystyle\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}x\rangle$
$\displaystyle\mathrel{\raisebox{-1.29167pt}{$\mathbin{\overset{\text{{def}}}{=}}$}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu$
$\displaystyle\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}x\rangle$
$\displaystyle\mathrel{\raisebox{-1.29167pt}{$\mathbin{\overset{\text{{def}}}{=}}$}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}Wx\dot{\rangle\mkern-3.0mu}\mkern
3.0mu$
$\displaystyle\scalebox{0.7}[1.0]{$\talloblong$}W\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x\rangle$
$\displaystyle\mathrel{\raisebox{-1.29167pt}{$\mathbin{\overset{\text{{def}}}{=}}$}}\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}x\rangle+\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}x\rangle.$

Any linear operator that is equal to
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
for some initial matrix $W$ is called an _initial operator_.

We also define the adjoint relations between the operators:

$\displaystyle\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}^{\dagger}$
$\displaystyle=\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W^{\top}\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}},$
$\displaystyle\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W\dot{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}^{\dagger}$
$\displaystyle=\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}W^{\top}\widehat{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}},$
$\displaystyle\scalebox{0.7}[1.0]{$\talloblong$}W\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}^{\dagger}$
$\displaystyle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W^{\top}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}.$

##### Parameter Update

In the SGD case, the parameter update of $W^{l}$ is simple. With the operator
notation and outer product notation, we can write

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{t+1}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{t}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}-\eta\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}h^{l}_{t}\rangle{}_{\chi_{t}}\langle
x^{l-1}_{t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}.$

In this work, $\Delta$ denotes change for one step, i.e.,

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\Delta
W_{t+1}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}=-\eta\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}h^{l}_{t}\rangle{}_{\chi_{t}}\langle
x^{l-1}_{t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}};$

$\bar{\Delta}$ denotes total change, i.e.,

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bar{\Delta}W_{t}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}=-\sum_{\tau=0}^{t-1}\eta\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}h^{l}_{\tau}\rangle{}_{\chi_{\tau}}\langle
x^{l-1}_{\tau}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}},$

which we write succinctly
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bar{\Delta}W_{t}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}=-\eta\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}\bm{h}^{l}_{<t}\rangle{}_{\bm{\chi}}\langle\bm{x}^{l-1}_{<t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$.
(Compared to Yang and Littwin [24], $\Delta$ and $\bar{\Delta}$ are changed
from $\delta$ and $\Delta$ because we want to use $\delta$ for gradients
instead of $d$, which is now used for depth differentiation).

Note in the general case,

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\Delta
W_{t+1}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}=-\eta\overline{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}\bm{h}^{l}_{\leq
t}\rangle{}_{\bm{\chi}_{\leq t}}\langle\bm{x}^{l-1}_{\leq
t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}$

where

$\overline{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}\bm{h}^{l}_{\leq
t}\rangle{}_{\bm{\chi}_{\leq t}}\langle\bm{x}^{l-1}_{\leq
t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}\mathbin{\overset{\text{{def}}}{=}}Q_{t}^{l}(\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}h^{l}_{0}\rangle{}_{\chi_{0}}\langle
x^{l-1}_{0}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}},\ldots,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}h^{l}_{t}\rangle{}_{\chi_{t}}\langle
x^{l-1}_{t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}).$

So

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bar{\Delta}W_{t}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}=-\eta\sum_{\tau=0}^{t-1}\overline{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}\bm{h}^{l}_{\leq\tau}\rangle{}_{\bm{\chi}_{\leq\tau}}\langle\bm{x}^{l-1}_{\leq\tau}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}}.$
(10)

For the rest of the paper, we write
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\bar{\Delta}W_{t}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}=-\eta\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}\bm{h}^{l}_{<t}\rangle{}_{\bm{\chi}}\langle\bm{x}^{l-1}_{<t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}$
for convenience. The generalization to eq. 10 follows Yang and Littwin [24].

##  Appendix C Details of the linear case

###  C.1 Proof sketch of 4.2

Here we provide a proof sketch of 4.2, the formal prove is implied by the
existence of $\Gamma$ and $C$ in the infinite depth limit.

###### Proof sketch.

The claims can be reasoned by induction on $t$ and $l$. Let us take
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle$
as an example, since
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{t}^{l-1}\rangle$
is symmetric with
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle$.
By expanding the definition of
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle$,
we have

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l-1}\rangle+\frac{1}{\sqrt{L}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{l}x_{t}^{l-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu+\frac{1}{\sqrt{L}}\sum_{s=1}^{t-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s}^{l}\rangle\left(\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l-1}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{l\top}\widetilde{\delta}x_{s}^{l}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}-\frac{1}{\sqrt{L}}\langle
x_{s}^{l-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l-1}\rangle\right).$

Note by induction, $\langle
x_{s}^{l-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l-1}\rangle=\mathcal{O}(1)$
and $\frac{\partial
x_{t}^{l-1}}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{l\top}\widetilde{\delta}x_{s}^{l}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}=\mathcal{O}(1/\sqrt{L})$, so

$\displaystyle\scalebox{0.7}[1.0]{$\talloblong$}x_{t}^{l}\rangle$
$\displaystyle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l-1}\rangle+\frac{1}{\sqrt{L}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{l}x_{t}^{l-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu+\mathcal{O}\left(\frac{1}{L}\right)\sum_{s=1}^{t-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s}^{l}\rangle$
$\displaystyle=\xi_{t}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}U\rangle+\sum_{m=1}^{l}\frac{1}{\sqrt{L}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m}x_{t}^{m-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu+\mathcal{O}\left(\frac{1}{L}\right)\sum_{m^{\prime}=1}^{l}\sum_{s^{\prime}=1}^{t-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s^{\prime}}^{m^{\prime}}\rangle.$

Then by unwinding
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s^{\prime}}^{m^{\prime}}\rangle$
and noting that by induction, $\forall s<t$,
$\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s^{\prime}}^{m^{\prime}}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m}x_{s}^{m-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}=\mathcal{O}\left(\frac{1}{\sqrt{L}}\right)$,
$\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s^{\prime}}^{m^{\prime}}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m\top}\widetilde{\delta}x_{s}^{m}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}=\mathcal{O}\left(\frac{1}{\sqrt{L}}\right)$,
$\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s^{\prime}}^{m^{\prime}}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}U\rangle}=\mathcal{O}\left(1\right)$,
$\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s^{\prime}}^{m^{\prime}}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}nV\rangle}=\mathcal{O}\left(1\right)$,
we have

$\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m}x_{s}^{m-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}=\mathcal{O}\left(\frac{1}{\sqrt{L}}\right),\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m\top}\widetilde{\delta}x_{s}^{m}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}=\mathcal{O}\left(\frac{1}{\sqrt{L}}\right),\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}U\rangle}=\mathcal{O}\left(1\right),\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}nV\rangle}=\mathcal{O}\left(1\right).$

Also by unwinding,
$\forall\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle\in\\{\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{s}^{m}\rangle,\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s}^{m}\rangle\\}$,

$\displaystyle\langle
y\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle=$
$\displaystyle\sum_{m^{\prime}}\sum_{s^{\prime}}\sum_{t^{\prime}}\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m^{\prime}}x_{t^{\prime}}^{m^{\prime}-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}\cdot\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m^{\prime}}x_{s^{\prime}}^{m^{\prime}-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}\cdot\langle
x_{t^{\prime}}^{m^{\prime}-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{s^{\prime}}^{m^{\prime}-1}\rangle$
$\displaystyle+\sum_{m^{\prime}}\sum_{s^{\prime}}\sum_{t^{\prime}}\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m^{\prime}\top}\widetilde{\delta}x_{t^{\prime}}^{m^{\prime}}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}\cdot\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{m^{\prime}\top}\widetilde{\delta}x_{s^{\prime}}^{m^{\prime}}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu}\cdot\langle\widetilde{\delta}x_{t^{\prime}}^{m^{\prime}}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}\widetilde{\delta}x_{s^{\prime}}^{m^{\prime}}\rangle$
$\displaystyle+\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}U\rangle}\cdot\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}U\rangle}+\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}nV\rangle}\frac{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}y\rangle}{\partial\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}nV\rangle}$
$\displaystyle=$ $\displaystyle\mathcal{O}(1).\qed$

###  C.2 Formal recursive formula of $\Gamma$ and $C$

By the same way of expanding
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle$
and $\langle
y\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{t}^{l}\rangle$,
we formally derive the recursive formula for $\Gamma$ and $C$ below.

######  Lemma C.1 (Finite depth recursive formula for $\Gamma$ and $C$).

$\Gamma$ can be computed recursively as follows:

For $t=0,\ldots,T-1$,

  * •

$\forall q\in(0,1],\Gamma_{t,-1,0,q}(0,q)=\xi_{t}$,

  * •

For $l=1,\ldots,L$, $\forall r\leq t$, $\forall
p\in\left(\frac{l-1}{L},\frac{l}{L}\right]$, $\forall q\in(0,1]$, $\forall
b\in\\{0,1\\}$,

$\displaystyle C_{t,s,0}(p)=$ $\displaystyle\leavevmode\nobreak\
\sum_{t^{\prime}=-1}^{t}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}\Gamma_{t,t^{\prime},0,b}\left(\frac{l-1}{L},q\right)C_{t^{\prime},s^{\prime},b}(q)\Gamma_{s,s^{\prime},0,b}\left(\frac{l-1}{L},q\right)\mathrm{d}q;$
$\displaystyle\Gamma_{t,r,0,b}\left(p,q\right)=$
$\displaystyle\leavevmode\nobreak\
\Gamma_{t,r,0,b}\left(\frac{l-1}{L},q\right)+\mathbb{I}_{[(t=r)\wedge(b=0)\wedge(l=\lceil
Lq\rceil)]}$
$\displaystyle+\frac{1}{L}\sum_{s=0}^{t-1}\Gamma_{s,r,1,b}\left(\frac{l}{L},q\right)\left(\Gamma_{t,s,0,1}\left(\frac{l-1}{L},\frac{l}{L}\right)-C_{t,s,0}\left(\frac{l}{L}\right)\right).$

  * •

$\mathring{f}_{t}=\Gamma_{t,-1,0,1}(1,1)$,

  * •

$\mathring{\chi}_{t}=\ell^{\prime}_{t}(\mathring{f}_{t})$,

  * •

$\forall q\in(0,1],\Gamma_{t,-1,1,1}(1,q)=\mathring{\chi}_{t}$,

  * •

For $l=L,\ldots,1$, $\forall r\leq t$, $\forall
p\in\left(\frac{l-2}{L},\frac{l-1}{L}\right]$, $\forall q\in(0,1]$, $\forall
b\in\\{0,1\\}$,

$\displaystyle C_{t,s,1}\left(p+\frac{1}{L}\right)=$
$\displaystyle\leavevmode\nobreak\
\sum_{t^{\prime}=-1}^{t}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}\Gamma_{t,t^{\prime},1,b}(l/L,q)C_{t^{\prime},s^{\prime},b}(q)\Gamma_{s,s^{\prime},1,b}(l/L,q)\mathrm{d}q;$
$\displaystyle\Gamma_{t,r,1,b}\left(p,q\right)=$
$\displaystyle\leavevmode\nobreak\
\Gamma_{t,r,1,b}\left(\frac{l}{L},q\right)+\mathbb{I}_{[(t=r)\wedge(b=1)\wedge(l=\lceil
Lq\rceil)]}$
$\displaystyle+\frac{1}{L}\sum_{s=0}^{t-1}\Gamma_{s,r,0,b}\left(\frac{l-1}{L},q\right)\left(\Gamma_{t,s,1,0}\left(\frac{l}{L},\frac{l}{L}\right)-C_{t,s,1}\left(\frac{l}{L}\right)\right).$

The proof is straightforward from Program 1. The recursive nature of $\Gamma$
and $C$ yields the following infinite-depth behavior.

######  Proposition C.2 (Infinite depth limit of $\Gamma$ and $C$).

In the limit $L\to\infty$, we have $\forall
p\in[0,1],q\in(0,1],b\in\\{0,1\\}$:

$\displaystyle\Gamma_{t,-1,0,0}(0,q)=\xi_{t};$
$\displaystyle\Gamma_{t,r,0,b}(p,q)=\mathbb{I}_{[(t=r)\wedge(b=0)\wedge(p\geq
q)]}+\int_{0}^{p}\sum_{s=0}^{t-1}\Gamma_{s,r,1,b}(p^{\prime},q)\cdot(\Gamma_{t,s,0,1}(p^{\prime},p^{\prime})-C_{t,s,0}(p^{\prime}))\mathrm{d}p^{\prime};$
$\displaystyle\mathring{f}_{t}=\Gamma_{t,-1,0,1}(1,1);$
$\displaystyle\mathring{\chi}_{t}=\ell^{\prime}_{t}(\mathring{f}_{t});$
$\displaystyle\Gamma_{t,-1,1,1}(1,q)=\mathring{\chi}_{t};$
$\displaystyle\Gamma_{t,r,1,b}(p,q)=\mathbb{I}_{[(t=r)\wedge(b=1)\wedge(p\leq
q)]}+\int_{p}^{1}\sum_{s=0}^{t-1}\Gamma_{s,r,0,b}(p^{\prime},q)\cdot(\Gamma_{t,s,1,0}(p^{\prime},p^{\prime})-C_{t,s,1}(p^{\prime}))\mathrm{d}p^{\prime};$
$\displaystyle
C_{t,s,a}(p)=\sum_{t^{\prime}=-1}^{t}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}\Gamma_{t,t^{\prime},a,b}(p,q)C_{t^{\prime},s^{\prime},b}(q)\Gamma_{s,s^{\prime},a,b}(p,q)\mathrm{d}q.$

###  C.3 Convergence of $\Gamma$ and $C$ when $L=2^{k}$

In this section, we prove $\Gamma$ and $C$ will converge when $L\to\infty$.
For convenience, we will only consider the case when $L=2^{k}$ for some
integer $k$. To distinguish $\Gamma$ and $C$ corresponding to different $L$,
we add the depth as the superscript, i.e., $\Gamma^{L}$ and $C^{L}$.

######  Theorem C.3.

$\forall t\leq T,s<t,a\in\\{0,1\\},b\in\\{0,1\\}$, $\forall
p\in[0,1],q\in(0,1]$,

  * •

$\\{\Gamma^{2^{k}}_{t,s,a,b}(p,q)\\}_{k\in\mathbb{N}}$ is a Cauchy sequence,

  * •

$\\{C^{2^{k}}_{t,s,a}(p)\\}_{k\in\mathbb{N}}$ is a Cauchy sequence.

The proof is by induction on $t$. We will prove the following claims (A) (B)
(C) (D) on $t>0$ given they are satisfied for any $s<t$. For $t=0$, (A) (B)
(C) (D) are trivial.

##### Assumption on $s<t$

Assume $\exists c>1$ such that $\forall L>L^{\prime}$ and $L=2^{k}$ for
$k\in\mathbb{N}$, $\forall s<t$, $\forall r<s$,

  1. (A)

$\forall p\in\\{0,\frac{1}{L},\ldots,1\\},q\in(0,1]$,

$|\Gamma^{L/2}_{s,r,a,b}(p,q)-\Gamma^{L}_{s,r,a,b}(p,q)|\leq
c/L,\qquad|C^{L/2}_{s,r,a}(p,q)-C^{L}_{s,r,a}(p,q)|\leq c/L.$

  2. (B)

$|\Gamma^{L}_{s,r,a,b}(p,q)|\leq c$, $|C^{L}_{s,r,a}(p)|\leq c$.

  3. (C)

$C^{L}_{s,r,a}(p)$ is $c$-Lipschitz w.r.t. $p$, and
$\Gamma^{L}_{s,r,a,b}(p,q)$ is $c$-Lipschitz w.r.t. $p$.

  4. (D)

$|\Gamma^{L}_{s,r,0,1}(p-\frac{1}{L},p+\frac{1}{L})-\Gamma^{L}_{s,r,0,1}(p-\frac{1}{L},p)|\leq
c/L$, $|\Gamma^{L}_{s,r,1,0}(p,p)-\Gamma^{L}_{s,r,1,0}(p,p-\frac{1}{L})|\leq
c/L$.

##### Remark

(A) indicates that $\\{\Gamma^{2^{k}}_{s,r,a,b}\\}_{k}$ and
$\\{C^{2^{k}}_{s,r,a}\\}_{k}$ converge. We only care about $r<s$ because
$C^{L}_{s,s,a}$ will never be used, and $\Gamma^{L}_{s,s,a,b}$ is known: for
$p\in\\{0,\frac{1}{L},\ldots,1\\}$,

$\Gamma^{L}_{s,s,a,b}(p,q)=\mathbb{I}[(a=0)\wedge(b=0)\wedge(p\geq
q)]+\mathbb{I}[(a=1)\wedge(b=1)\wedge(p+1/L\leq q)].$

##### Proof for $t$-th step (the forward pass)

In the following subsections, we will prove inductively on increasing order of
all $L>L^{\prime}$ and $L=2^{k}$, and increasing order of
$p\in\\{0,1/L,\ldots,1\\}$ that $\forall s<t$,

  1. (D0)

$|\Gamma^{L}_{t,s,0,1}(p,p+\frac{2}{L})-\Gamma^{L}_{t,s,0,1}(p,p+\frac{1}{L})|\leq
c_{2}\exp(c_{1}p)/L$;

  2. (C0)

For $s<t$,
$|\Gamma^{L}_{t,s,0,b}(p,q)-\Gamma^{L}_{t,s,0,b}(p-\frac{1}{L},q)|\leq
tcc_{2}\exp(c_{1}(p-\frac{1}{L}))/L$;

  3. (B0)

$|\Gamma^{L}_{t,s,0,b}(p,q)|\leq c_{2}\exp(c_{1}(p-\frac{1}{2L}))$;

  4. (A0)

$|\Gamma^{L/2}_{t,s,0,b}(p,q)-\Gamma^{L}_{t,s,0,b}(p,q)|\leq
c_{3}c_{2}\exp(c_{1}(p-\frac{1}{2L}))/L$;

  5. (C1)

$|C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L}_{t,s,0}(p)|\leq
c_{4}c_{2}\exp(c_{1}(p-\frac{1}{L}))/L$;

  6. (B1)

$|C^{L}_{t,s,0}(p+\frac{1}{L})|\leq c_{2}\exp(c_{1}p)$;

  7. (A1)

$|C^{L/2}_{t,s,0}(p+\frac{1}{L})-C^{L}_{t,s,0}(p+\frac{1}{L})|\leq
c_{5}c_{2}\exp(c_{1}p)/L,$

where $c_{2}=\max\\{\xi_{t}^{2},|\xi_{t}|\\}\exp(c_{1}/{2L^{\prime}})$,
$c_{3}=3ct$, $c_{4}=4t(t+1)c^{2}+2tc$, $c_{5}=c_{4}+1$,
$c_{1}=c^{3}t(4ct+2c_{4}+29)+tc(3c_{4}+14)+c(2c_{4}+2c)$.

##### Proof for $t$-th step (the backward pass)

Similar bounds also apply to $\Gamma_{t,s,1,b}$ and $C_{t,s,1}$ by induction
on decreasing order of $p$.

##### Conclusion

Combining both backward pass and forward pass at time $t$ shows (A)(B)(C)(D)
also hold for $s=t$ with a larger (but constant) $c$. Thus, (A)(B)(C)(D) hold
for any constant $s$ by induction on training steps.

####  C.3.1 $\Gamma^{L}_{t,s,0,b}(p,q)$ in forward pass (Proof for D0, C0, B0,
A0)

We first consider

$\displaystyle\Gamma^{L}_{t,r,0,b}\left(p,q\right)=$
$\displaystyle\leavevmode\nobreak\
\Gamma^{L}_{t,r,0,b}\left(p-\frac{1}{L},q\right)+\mathbb{I}[(t=r)\wedge(b=0)\wedge(Lp=\lceil
Lq\rceil)]$
$\displaystyle+\frac{1}{L}\sum_{s=0}^{t-1}\Gamma^{L}_{s,r,1,b}\left(p,q\right)\left(\Gamma^{L}_{t,s,0,1}\left(p-\frac{1}{L},p\right)-C^{L}_{t,s,0}\left(p\right)\right).$

##### (D0) Difference between $\Gamma^{L}_{t,s,0,1}(p,p+\frac{2}{L})$ and
$\Gamma^{L}_{t,s,0,1}(p,p+\frac{1}{L})$

Assume $p\geq 1/L$ ($p=0$ is trivial), let $q=p+1/L,q^{\prime}=p+2/L$, note
that $\Gamma_{s,s,1,b}^{L}(p,q)=\Gamma_{s,s,1,b}^{L}(p,q^{\prime})$ since
$p+1/L\leq q\leq q^{\prime}$, so for $r<t$,

$\displaystyle|\Gamma^{L}_{t,r,0,b}\left(p,q\right)-\Gamma^{L}_{t,r,0,b}\left(p,q^{\prime}\right)|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
|\Gamma^{L}_{t,r,0,b}\left(p-\frac{1}{L},q\right)-\Gamma^{L}_{t,r,0,b}\left(p-\frac{1}{L},q^{\prime}\right)|$
$\displaystyle+\frac{1}{L}\sum_{s=0}^{t-1}|\Gamma^{L}_{s,r,1,b}\left(p,q\right)-\Gamma^{L}_{s,r,1,b}\left(p,q^{\prime}\right)|\left|\Gamma^{L}_{t,s,0,1}\left(p-\frac{1}{L},p\right)-C^{L}_{t,s,0}\left(p\right)\right|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
c_{2}\exp(c_{1}(p-\frac{1}{L}))/L+\frac{1}{L}\cdot t\cdot c/L\cdot
2c_{2}\exp(c_{1}(p-\frac{1}{L}))$ $\displaystyle=$
$\displaystyle\leavevmode\nobreak\
(1+2ct/L)c_{2}\exp(c_{1}(p-\frac{1}{L}))/L\leq c_{2}\exp(c_{1}p)/L,$

as $c_{1}\geq 2ct$.

##### (C0) Lipschitz w.r.t. $p$

For $r<t$,

$\displaystyle|\Gamma^{L}_{t,r,0,b}(p,q)-\Gamma^{L}_{t,r,0,b}(p-\frac{1}{L},q)|$
$\displaystyle=$ $\displaystyle\leavevmode\nobreak\
|\frac{1}{L}\sum_{s=0}^{t-1}\Gamma^{L}_{s,r,1,b}\left(p,q\right)\left(\Gamma^{L}_{t,s,0,1}\left(p-\frac{1}{L},p\right)-C^{L}_{t,s,0}\left(p\right)\right)|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{1}{L}\sum_{s=0}^{t-1}c\left(c_{2}\exp(c_{1}(p-\frac{1}{L}))+c_{2}\exp(c_{1}(p-\frac{1}{L}))\right)$
$\displaystyle=$ $\displaystyle\leavevmode\nobreak\
ctc_{2}\exp(c_{1}(p-\frac{1}{L}))/L.$

##### (B0) Bounded

Again assume $p\geq 1/L$ ($p=0$ is trivial because
$c_{2}\geq|\xi_{t}|\exp(c_{1}/2L)$), since
$|\Gamma^{L}_{t,r,0,b}(p-\frac{1}{L},q)|\leq c_{2}\exp(c_{1}(p-\frac{1}{L}))$,
we can bound $|\Gamma^{L}_{t,r,0,b}(p,q)|$:

$\displaystyle|\Gamma^{L}_{t,r,0,b}(p,q)|\leq$
$\displaystyle\leavevmode\nobreak\
c_{2}\exp(c_{1}(p-\frac{1}{L}))+ctc_{2}\exp(c_{1}(p-\frac{1}{L}))/L$
$\displaystyle=$ $\displaystyle\leavevmode\nobreak\
c_{2}\exp(c_{1}(p-\frac{1}{L}))(1+ct/L)$ $\displaystyle\leq$
$\displaystyle\leavevmode\nobreak\ c_{2}\exp(c_{1}(p-\frac{1}{2L})),$

as long as $c_{1}\geq 2ct$.

##### (A0) Difference between $L$ and $L/2$ bounded

When $p=0$, it is trivial. When $p=1/L$, it is also trivial by Lipschitz
w.r.t. $p$, which results

$|\Gamma_{t,r,0,b}^{L/2}\left(p,q\right)-\Gamma_{t,r,0,b}^{L}\left(p,q\right)|\leq
3ctc_{2}/L\leq c_{3}c_{2}\exp(c_{1}/2L)/L.$

When $p\geq 2/L$, since

$\displaystyle\Gamma_{t,r,0,b}^{L/2}\left(p,q\right)=\Gamma_{t,r,0,b}^{L/2}\left(p-\frac{2}{L},q\right)+\frac{2}{L}\sum_{s=0}^{t-1}\Gamma_{s,r,1,b}^{L/2}\left(p,q\right)\left(\Gamma_{t,s,0,1}^{L/2}\left(p-\frac{2}{L},p\right)-C_{t,s,0}^{L/2}(p)\right),$

we compare it with $\Gamma_{t,r,0,b}^{L}\left(p,q\right)$ expanded based on
its previous two steps

$\displaystyle\Gamma^{L}_{t,r,0,b}\left(p,q\right)=$
$\displaystyle\leavevmode\nobreak\
\Gamma^{L}_{t,r,0,b}\left(p-\frac{2}{L},q\right)+\frac{1}{L}\sum_{s=0}^{t-1}\Gamma^{L}_{s,r,1,b}\left(p,q\right)\left(\Gamma^{L}_{t,s,0,1}\left(p-\frac{1}{L},p\right)-C^{L}_{t,s,0}\left(p\right)\right)$
$\displaystyle+\frac{1}{L}\sum_{s=0}^{t-1}\Gamma^{L}_{s,r,1,b}\left(p-\frac{1}{L},q\right)\left(\Gamma^{L}_{t,s,0,1}\left(p-\frac{2}{L},p-\frac{1}{L}\right)-C^{L}_{t,s,0}\left(p-\frac{1}{L}\right)\right).$

In order to bridge the two above, namely matching the inputs for $\Gamma$ and
$C$, we need a middle term

$\displaystyle\widetilde{\Gamma}^{L}_{t,r,0,b}\left(p,q\right)=$
$\displaystyle\leavevmode\nobreak\
\Gamma^{L}_{t,r,0,b}\left(p-\frac{2}{L},q\right)+\frac{2}{L}\sum_{s=0}^{t-1}\Gamma^{L}_{s,r,1,b}\left(p,q\right)\left(\Gamma^{L}_{t,s,0,1}\left(p-\frac{2}{L},p\right)-C^{L}_{t,s,0}\left(p\right)\right).$

Now we can bound
$|\Gamma^{L}_{t,r,0,b}\left(p,q\right)-\widetilde{\Gamma}^{L}_{t,r,0,b}\left(p,q\right)|$,
and
$|\widetilde{\Gamma}^{L}_{t,r,0,b}\left(p,q\right)-\Gamma^{L/2}_{t,r,0,b}\left(p,q\right)|$
separately, which add up to be the bound for
$|\Gamma^{L}_{t,r,0,b}\left(p,q\right)-\Gamma^{L/2}_{t,r,0,b}\left(p,q\right)|$.

$\displaystyle\leavevmode\nobreak\
|\Gamma^{L}_{t,r,0,b}\left(p,q\right)-\widetilde{\Gamma}^{L}_{t,r,0,b}\left(p,q\right)|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{1}{L}\sum_{s=0}^{t-1}|\Gamma^{L}_{s,r,1,b}\left(p,q\right)|\left|\Gamma^{L}_{t,s,0,1}\left(p-\frac{1}{L},p\right)-\Gamma^{L}_{t,s,0,1}\left(p-\frac{2}{L},p\right)\right|$
$\displaystyle+\frac{1}{L}\sum_{s=0}^{t-1}\left|\Gamma^{L}_{s,r,1,b}\left(p-\frac{1}{L},q\right)\left(\Gamma^{L}_{t,s,0,1}\left(p-\frac{2}{L},p-\frac{1}{L}\right)-C^{L}_{t,s,0}\left(p-\frac{1}{L}\right)\right)\right.$
$\displaystyle\qquad\qquad\left.-\Gamma^{L}_{s,r,1,b}\left(p,q\right)\left(\Gamma^{L}_{t,s,0,1}\left(p-\frac{2}{L},p\right)-C^{L}_{t,s,0}\left(p\right)\right)\right|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\ \frac{1}{L}\cdot
ct\cdot ctc_{2}\exp(c_{1}(p-\frac{2}{L}))/L+\frac{1}{L}\cdot 2t\cdot c/L\cdot
c_{2}\exp(c_{1}(p-\frac{2}{L}))$ $\displaystyle+\frac{1}{L}\cdot ct\cdot
c_{2}\exp(c_{1}(p-\frac{2}{L}))/L+\frac{1}{L}\cdot ct\cdot
c_{4}c_{2}\exp(c_{1}(p-\frac{2}{L}))/L$ $\displaystyle=$
$\displaystyle\leavevmode\nobreak\ \frac{c^{2}t^{2}+3ct+c_{4}ct}{L^{2}}\cdot
c_{2}\exp(c_{1}(p-\frac{2}{L})).$

and

$\displaystyle\leavevmode\nobreak\
|\Gamma_{t,r,0,b}^{L/2}\left(p,q\right)-\widetilde{\Gamma}_{t,r,0,b}^{L}\left(p,q\right)|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
|\Gamma_{t,r,0,b}^{L/2}\left(p-\frac{2}{L},q\right)-\Gamma_{t,r,0,b}^{L}\left(p-\frac{2}{L},q\right)|$
$\displaystyle+\frac{1}{L}\sum_{s=0}^{t-1}c\left|\Gamma_{t,s,0,1}^{L/2}\left(p-\frac{2}{L},p\right)-\Gamma_{t,s,0,1}^{L}\left(p-\frac{2}{L},p\right)-C_{t,s,0}^{L/2}\left(p\right)+C_{t,s,0}^{L}\left(p\right)\right|$
$\displaystyle+\frac{1}{L}\sum_{s=0}^{t-1}\frac{c}{L}\left(|\Gamma^{L}_{t,s,0,1}\left(p-\frac{2}{L},p\right)|+|C^{L}_{t,s,0}\left(p\right)|\right)$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{1}{L}\cdot(c_{3}c_{2}\exp(c_{1}(p-\frac{1}{L}))+ct(c_{3}+c_{5})c_{2}\exp(c_{1}(p-\frac{1}{L}))/L+2t\cdot\frac{c}{L}\cdot
c_{2}\exp(c_{1}(p-\frac{1}{L})))$ $\displaystyle\leq$
$\displaystyle\leavevmode\nobreak\ \frac{c_{3}+ct(c_{3}+c_{5}+2)/L}{L}\cdot
c_{2}\exp(c_{1}(p-\frac{1}{L})).$

In sum, as $c_{1}\geq\frac{2(c_{3}+c_{5}+ct+c_{4}+5)}{3}$,

$\displaystyle|\Gamma_{t,r,0,b}^{L/2}\left(p,q\right)-\Gamma_{t,r,0,b}^{L}\left(p,q\right)|\leq$
$\displaystyle\leavevmode\nobreak\
\frac{c_{3}+ct(c_{3}+c_{5}+ct+c_{4}+5)/L}{L}c_{2}\exp(c_{1}(p-\frac{1}{L}))$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
c_{2}\exp(c_{1}(p-\frac{1}{2L}))/L.$

####  C.3.2 $C_{t,s,0}(p+\frac{1}{L})$ in forward pass (Proof for C1, B1, A1)

Now consider $C^{L}_{t,s,0}$. By expanding

$C^{L}_{t,s,0}(p+\frac{1}{L})=\sum_{t^{\prime}=-1}^{t}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}\Gamma^{L}_{t,t^{\prime},0,b}(p,q)C^{L}_{t^{\prime},s^{\prime},b}(q)\Gamma^{L}_{s,s^{\prime},0,b}(p,q)\mathrm{d}q,$

we will have

$\displaystyle C^{L}_{t,s,0}(p+\frac{1}{L})=$
$\displaystyle\sum_{t^{\prime}=-1}^{t-1}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}\Gamma^{L}_{t,t^{\prime},0,b}(p,q)C^{L}_{t^{\prime},s^{\prime},b}(q)\Gamma^{L}_{s,s^{\prime},0,b}(p,q)\mathrm{d}q$
$\displaystyle+\sum_{s^{\prime}=0}^{s}\int_{0}^{p}C^{L}_{t,s^{\prime},0}(q)\Gamma^{L}_{s,s^{\prime},0,0}(p,q)\mathrm{d}q.$

##### (C1) Lipschitz

Since $C^{L}_{t^{\prime},s^{\prime},b}$ and $\Gamma^{L}_{s,s^{\prime},0,b}$
are bounded and Lipschitz,

$\displaystyle|C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L}_{t,s,0}(p)|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\sum_{t^{\prime}=-1}^{t-1}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}|\Gamma^{L}_{t,t^{\prime},0,b}(p,q)-\Gamma^{L}_{t,t^{\prime},0,b}(p-\frac{1}{L},q)|\cdot
c^{2}\mathrm{d}q$
$\displaystyle+\sum_{t^{\prime}=-1}^{t-1}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}|\Gamma^{L}_{t,t^{\prime},0,b}(p-\frac{1}{L},q)|\cdot
c\cdot\frac{c}{L}\mathrm{d}q$
$\displaystyle+\sum_{s^{\prime}=0}^{s}\frac{1}{L}\cdot|C^{L}_{t,s^{\prime},0}(p)\Gamma^{L}_{s,s^{\prime},0,0}(p,p)|$
$\displaystyle+\sum_{s^{\prime}=0}^{s}\int_{0}^{p-\frac{1}{L}}|C^{L}_{t,s^{\prime},0}(q)|\cdot\frac{c}{L}\mathrm{d}q.$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\ 1/L\cdot(2t(s+1)\cdot
ctc_{2}\exp(c_{1}(p-\frac{1}{L}))\cdot
c^{2}+2t(s+1)c_{2}\exp(c_{1}(p-\frac{1}{L}))\cdot c^{2}$ $\displaystyle+s\cdot
c_{2}\exp(c_{1}(p-\frac{1}{L}))\cdot c+s\cdot
c_{2}\exp(c_{1}(p-\frac{1}{L}))\cdot c)$ $\displaystyle=$
$\displaystyle\leavevmode\nobreak\ (4t(s+1)c^{2}+2sc)/L\cdot
c_{2}\exp(c_{1}(p-\frac{1}{L}))$ $\displaystyle\leq$
$\displaystyle\leavevmode\nobreak\ c_{4}c_{2}\exp(c_{1}(p-\frac{1}{L}))/L.$

##### (B1) Bounded

Since $|C^{L}_{t,s,0}(p)|\leq c_{2}\exp(c_{1}(p-\frac{1}{L}))$, we bound
$C^{L}_{t,s,0}(p+\frac{1}{L})$ as:

$|C^{L}_{t,s,0}(p+\frac{1}{L})|\leq
c_{2}\exp(c_{1}(p-\frac{1}{L}))\cdot(1+c_{4}/L)\leq c_{2}\exp(c_{1}p),$

as long as $c_{1}\geq c_{4}$.

##### (A1) Difference between $L$ and $L/2$ bounded

It is easy to see that for $p=0$,

$C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L/2}_{t,s,0}(p+\frac{1}{L})=0,$

we will prove that for $p\in\\{2/L,4/L,\ldots,1\\}$,

$|C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L/2}_{t,s,0}(p+\frac{1}{L})|\leq
c_{2}\exp(c_{1}p)/L.$

Then by (C1), for $p\in\\{1/L,3/L,\ldots,1-1/L\\}$,

$\displaystyle|C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L/2}_{t,s,0}(p+\frac{1}{L})|\leq$
$\displaystyle\leavevmode\nobreak\
c_{2}\exp(c_{1}(p-\frac{1}{L}))/L+c_{4}c_{2}\exp(c_{1}(p-\frac{1}{L}))/L$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
(c_{4}+1)c_{2}\exp(c_{1}p)/L$ $\displaystyle=$
$\displaystyle\leavevmode\nobreak\ c_{5}c_{2}\exp(c_{1}p)/L.$

Suppose $p\in\\{2/L,4/L,\ldots,1\\}$, we compare
$C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L}_{t,s,0}(p-\frac{1}{L})$ and
$C^{L/2}_{t,s,0}(p+\frac{1}{L})-C^{L/2}_{t,s,0}(p-\frac{1}{L})$. Intuitively,
both of them are $\mathcal{O}(1/L)$, and their difference is
$\mathcal{O}(1/L^{2})$. In particular, both of them can be written into four
parts:

$\displaystyle\leavevmode\nobreak\
C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L}_{t,s,0}(p-\frac{1}{L})$ $\displaystyle=$
$\displaystyle\leavevmode\nobreak\
\sum_{t^{\prime}=-1}^{t-1}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}\left(\Gamma^{L}_{t,t^{\prime},0,b}(p,q)-\Gamma^{L}_{t,t^{\prime},0,b}(p-\frac{2}{L},q)\right)C^{L}_{t^{\prime},s^{\prime},b}(q)\Gamma^{L}_{s,s^{\prime},0,b}(p,q)\mathrm{d}q$
$\displaystyle(\mathcal{E}_{1}^{L})$
$\displaystyle+\sum_{t^{\prime}=-1}^{t-1}\sum_{s^{\prime}=-1}^{s}\sum_{b\in\\{0,1\\}}\int_{0}^{1}\Gamma^{L}_{t,t^{\prime},0,b}(p-\frac{2}{L},q)C^{L}_{t^{\prime},s^{\prime},b}(q)\left(\Gamma^{L}_{s,s^{\prime},0,b}(p,q)-\Gamma^{L}_{s,s^{\prime},0,b}(p-\frac{2}{L},q)\right)\mathrm{d}q$
$\displaystyle(\mathcal{E}_{2}^{L})$
$\displaystyle+\sum_{s^{\prime}=0}^{s}\int_{p-\frac{2}{L}}^{p}C^{L}_{t,s^{\prime},0}(q)\Gamma_{s,s^{\prime},0,0}^{L}(p,q)\mathrm{d}q$
$\displaystyle(\mathcal{E}_{3}^{L})$
$\displaystyle+\sum_{s^{\prime}=0}^{s}\int_{0}^{p-\frac{2}{L}}C^{L}_{t,s^{\prime},0}(q)(\Gamma_{s,s^{\prime},0,0}^{L}(p,q)-\Gamma_{s,s^{\prime},0,0}^{L}(p-\frac{2}{L},q))\mathrm{d}q$
$\displaystyle(\mathcal{E}_{4}^{L})$

and
$C^{L/2}_{t,s,0}(p+\frac{1}{L})-C^{L/2}_{t,s,0}(p-\frac{1}{L})=\mathcal{E}_{1}^{L/2}+\mathcal{E}_{2}^{L/2}+\mathcal{E}_{3}^{L/2}+\mathcal{E}_{4}^{L/2}$
where $\mathcal{E}_{i}^{L/2}$ is defined in the same way as
$\mathcal{E}_{i}^{L}$ but with $C^{L/2}$ and $\Gamma^{L/2}$ instead of $C^{L}$
and $\Gamma^{L}$. Next we bound $|\mathcal{E}_{i}^{L}-\mathcal{E}_{i}^{L/2}|$
one by one:

  1. 1.

The only hard part to bound in $|\mathcal{E}_{i}^{L}-\mathcal{E}_{i}^{L/2}|$
is

$|\Gamma^{L}_{t,t^{\prime},0,b}(p,q)-\Gamma^{L}_{t,t^{\prime},0,b}(p-\frac{2}{L},q)-(\Gamma^{L/2}_{t,t^{\prime},0,b}(p,q)-\Gamma^{L/2}_{t,t^{\prime},0,b}(p-\frac{2}{L},q))|.$

By almost the same proof of (A0),

$\displaystyle|\Gamma^{L}_{t,t^{\prime},0,b}(p,q)-\Gamma^{L}_{t,t^{\prime},0,b}(p-\frac{2}{L},q)-(\Gamma^{L/2}_{t,t^{\prime},0,b}(p,q)-\Gamma^{L/2}_{t,t^{\prime},0,b}(p-\frac{2}{L},q))|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{ct(c_{3}+c_{5}+ct+c_{4}+5)}{L^{2}}c_{2}\exp(c_{1}(p-\frac{1}{L})).$

Then we have

$\displaystyle|\mathcal{E}_{1}^{L}-\mathcal{E}_{1}^{L/2}|/(2t(s+1))$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{ct(c_{3}+c_{5}+ct+c_{4}+5)}{L^{2}}c_{2}\exp(c_{1}(p-\frac{1}{L}))\cdot
c\cdot c$ $\displaystyle+4ctc_{2}\exp(c_{1}(p-\frac{1}{L}))/L\cdot c/L\cdot c$
$\displaystyle+4ctc_{2}\exp(c_{1}(p-\frac{1}{L}))/L\cdot c\cdot c/L$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{c^{3}t(c_{3}+c_{5}+ct+c_{4}+13)}{L^{2}}c_{2}\exp(c_{1}(p-\frac{1}{L}))$

  2. 2.

Bounding $|\mathcal{E}_{2}^{L}-\mathcal{E}_{2}^{L/2}|$ is similar to
$|\mathcal{E}_{1}^{L}-\mathcal{E}_{1}^{L/2}|$, where we first bound

$|\Gamma^{L}_{s,s^{\prime},0,b}(p,q)-\Gamma^{L}_{s,s^{\prime},0,b}(p-\frac{2}{L},q)-(\Gamma^{L/2}_{s,s^{\prime},0,b}(p,q)-\Gamma^{L/2}_{s,s^{\prime},0,b}(p-\frac{2}{L},q))|\leq
9c^{2}t/L^{2}.$

Then we have

$\displaystyle|\mathcal{E}_{2}^{L}-\mathcal{E}_{2}^{L/2}|/(2t(s+1))$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
c_{3}c_{2}\exp(c_{1}(p-\frac{2}{L}))/L\cdot c\cdot 2c/L$
$\displaystyle+c_{2}\exp(c_{1}(p-\frac{2}{L}))\cdot c/L\cdot 2c/L$
$\displaystyle+c_{2}\exp(c_{1}(p-\frac{2}{L}))\cdot c\cdot 9c^{2}t/L^{2}$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{c^{2}(2c_{3}+2+9ct)}{L^{2}}c_{2}\exp(c_{1}(p-\frac{2}{L})).$

  3. 3.

For $|\mathcal{E}_{3}^{L}-\mathcal{E}_{3}^{L/2}|$, we first simplify

$\mathcal{E}_{3}^{L/2}=\frac{2}{L}\sum_{s^{\prime}=0}^{s}C_{t,s^{\prime},0}^{L/2}(p)\Gamma_{s,s^{\prime},0,0}^{L/2}(p,p),$

and

$\mathcal{E}_{3}^{L}=\frac{1}{L}\sum_{s^{\prime}=0}^{s}C_{t,s^{\prime},0}^{L}(p)\Gamma_{s,s^{\prime},0,0}^{L}(p,p)+C_{t,s^{\prime},0}^{L}(p-\frac{1}{L})\Gamma_{s,s^{\prime},0,0}^{L}(p,p-\frac{1}{L}).$

Again, we introduce an intermediate term

$\widetilde{\mathcal{E}}_{3}^{L}=\frac{2}{L}\sum_{s^{\prime}=0}^{s}C_{t,s^{\prime},0}^{L}(p)\Gamma_{s,s^{\prime},0,0}^{L}(p,p).$

Then we can bound

$\displaystyle|\mathcal{E}_{3}^{L}-\mathcal{E}_{3}^{L/2}|$ $\displaystyle\leq$
$\displaystyle\leavevmode\nobreak\
|\mathcal{E}_{3}^{L}-\widetilde{\mathcal{E}}_{3}^{L}|+|\widetilde{\mathcal{E}}_{3}^{L}-\mathcal{E}_{3}^{L/2}|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{t}{L}(c_{4}c_{2}\exp(c_{1}(p-\frac{2}{L}))/L\cdot
c+c_{2}\exp(c_{1}(p-\frac{2}{L}))\cdot c/L)$
$\displaystyle+\frac{2t}{L}(c_{5}c_{2}\exp(c_{1}(p-\frac{1}{L}))/Lc+c_{2}\exp(c_{1}(p-\frac{1}{L}))\cdot
c/L)$ $\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{tc(c_{4}+1+2c_{5}+2)}{L^{2}}c_{2}\exp(c_{1}(p-\frac{1}{L})).$

  4. 4.

For $|\mathcal{E}_{4}^{L}-\mathcal{E}_{4}^{L/2}|$, we use

$|\Gamma^{L}_{s,s^{\prime},0,b}(p,q)-\Gamma^{L}_{s,s^{\prime},0,b}(p-\frac{2}{L},q)-(\Gamma^{L/2}_{s,s^{\prime},0,b}(p,q)-\Gamma^{L/2}_{s,s^{\prime},0,b}(p-\frac{2}{L},q))|\leq
9c^{2}t/L^{2},$

which is used in $|\mathcal{E}_{2}^{L}-\mathcal{E}_{2}^{L/2}|$. Finally,

$\displaystyle|\mathcal{E}_{4}^{L}-\mathcal{E}_{4}^{L/2}|/t$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
c_{4}c_{2}\exp(c_{1}(p-\frac{2}{L}))/L\cdot 2c/L$
$\displaystyle+c_{2}\exp(c_{1}(p-\frac{2}{L}))\cdot 9c^{2}t/L^{2}$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{c(2c_{4}+9ct)}{L^{2}}c_{2}\exp(c_{1}(p-\frac{2}{L})).$

In sum,

$\displaystyle|C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L}_{t,s,0}(p-\frac{1}{L})-C^{L/2}_{t,s,0}(p+\frac{1}{L})+C^{L/2}_{t,s,0}(p-\frac{1}{L})|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
\frac{c^{3}t(4ct+2c_{4}+14)+c^{2}(2+15ct)+tc(3c_{4}+5)+c(2c_{4}+9ct)}{L^{2}}\cdot
c_{2}\exp(c_{1}(p-\frac{1}{L}))$ $\displaystyle=$
$\displaystyle\leavevmode\nobreak\
\frac{c^{3}t(4ct+2c_{4}+29)+tc(3c_{4}+14)+c(2c_{4}+2c)}{L^{2}}\cdot
c_{2}\exp(c_{1}(p-\frac{1}{L})).$

Therefore, since $c_{1}=c^{3}t(4ct+2c_{4}+29)+tc(3c_{4}+14)+c(2c_{4}+2c)$,

$\displaystyle|C^{L}_{t,s,0}(p+\frac{1}{L})-C^{L/2}_{t,s,0}(p+\frac{1}{L})|$
$\displaystyle\leq$ $\displaystyle\leavevmode\nobreak\
|C^{L}_{t,s,0}(p-\frac{1}{L})-C^{L/2}_{t,s,0}(p-\frac{1}{L})|+c_{1}/L^{2}\cdot
c_{2}\exp(c_{1}(p-\frac{1}{L}))$ $\displaystyle\leq$
$\displaystyle\leavevmode\nobreak\
(1+c_{1}/L)c_{2}\exp(c_{1}(p-\frac{1}{L}))/L$ $\displaystyle\leq$
$\displaystyle\leavevmode\nobreak\ c_{2}\exp(c_{1}p)/L.$

##  Appendix D Classification of Depthwise Parametrizations in Linear Case

We discuss the classification results on the linear residual networks with SGD
training and give rigorous proofs for the claims in this simplified setting.
Recall the linear residual networks:

$\forall l\in[L],x^{l}=x^{l-1}+aL^{-\alpha}h^{l},$

where $h^{l}=W^{l}x^{l}$, and the effective learning rate of $W^{l}$ is $\eta
n^{-1}L^{-\gamma}$. Without loss of generality, we assume $\eta=a=1$.

###  D.1 Initialization

At initialization, we have

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{0}^{l}\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{0}^{l-1}\rangle+L^{-\alpha}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}h^{l}_{0}\rangle,$

where

$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}h_{0}^{l}\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{l}x_{0}^{l-1}\rangle=\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{l}x_{0}^{l-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu.$

Since
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{0}^{l-1}\rangle$
is independent from
$\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}W_{0}^{l}x_{0}^{l-1}\widehat{\rangle\mkern-3.0mu}\mkern
3.0mu$, we have

$\langle
x_{0}^{l}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{0}^{l}\rangle=\langle
x_{0}^{l-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{0}^{l-1}\rangle+L^{-2\alpha}\langle
h^{l}_{0}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}h^{l}_{0}\rangle=\langle
x_{0}^{l-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{0}^{l-1}\rangle+L^{-2\alpha}\langle
x_{0}^{l-1}\mathchoice{\scalebox{0.7}[1.0]{$\displaystyle\talloblong$}}{\scalebox{0.7}[1.0]{$\textstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptstyle\talloblong$}}{\scalebox{0.7}[1.0]{$\scriptscriptstyle\talloblong$}}x_{0}^{l-1}\rangle=(1+L^{-2\alpha})\langle