# Rethinking Skip Connection with Layer Normalization
in Transformers and ResNets

Fenglin Liu1, Xuancheng Ren211footnotemark: 1, Zhiyuan Zhang2, Xu Sun2,
Yuexian Zou1
1ADSPLAB, School of ECE, Peking University, China
2MOE Key Laboratory of Computational Linguistics, School of EECS, Peking
University
{fenglinliu98, renxc, zzy1210, xusun<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Skip connection, is a widely-used technique to improve the performance and the
convergence of deep neural networks, which is believed to relieve the
difficulty in optimization due to non-linearity by propagating a linear
component through the neural network layers. However, from another point of
view, it can also be seen as a modulating mechanism between the input and the
output, with the input scaled by a pre-defined value one. In this work, we
investigate how the scale factors in the effectiveness of the skip connection
and reveal that a trivial adjustment of the scale will lead to spurious
gradient exploding or vanishing in line with the deepness of the models, which
could be addressed by normalization, in particular, layer normalization, which
induces consistent improvements over the plain skip connection. Inspired by
the findings, we further propose to adaptively adjust the scale of the input
by recursively applying skip connection with layer normalization, which
promotes the performance substantially and generalizes well across diverse
tasks including both machine translation and image classification datasets.

##  1 Introduction

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/.

There is a surge of research interest in solving the optimization problem of
deep neural networks, which presents a considerable number of novel
difficulties that are in need of practical solutions. Batch Normalization
[Ioffe and Szegedy (2015], Layer Normalization [Ba et al. (2016] and Skip
Connection [He et al. (2016a] are widely-used techniques to facilitate the
optimization of deep neural networks, which prove to be effective in multiple
contexts [Szegedy et al. (2016, Vaswani et al. (2017].

Among them, Batch Normalization (BN) and Layer Normalization (LN) make
progress by normalizing and rescaling the intermediate hidden representations
to deal with the so-called internal co-variate shift in mini-batch training
and collective optimization of multiple correlated features. Skip Connection
bypasses the gradient exploding or vanishing problem and tries to solve the
model optimization problem from the perspective of information transfer. It
enables the delivery and integration of information by adding an identity
mapping from the input of the neural network to the output, which may ease the
optimization and allow the error signal to pass through the non-linearities.

Recently, several efforts have been made to deal with the optimization problem
by further exploiting the existing solutions, such as exploring the
implementation of skip connection [Srivastava et al. (2015] and combining the
skip connection with other normalization methods [He et al. (2016b, Vaswani et
al. (2017]. Some of the proposals are illustrated in Figure 1. Most of these
variations could be formulated as or coarsely equivalent to

$\displaystyle\boldsymbol{y}=\mathcal{G}(\lambda\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})),$
(1)

where $\boldsymbol{x}$ denotes the input of the block, i.e., the skip
connection or the shortcut, $\mathcal{F}$ denotes the non-linear
transformation induced by the neural network parameterized by $\mathbf{W}$,
$\mathcal{G}$ denotes the normalization function, $\boldsymbol{y}$ denotes the
output of the residual block, and $\lambda$ denotes the modulating factor that
controls the relative importance of the skip connection or the shortcut.111The
terms skip connection and shortcut are used interchangeably in this work. A
residual model consists of a stack of such blocks.

Figure 1: Illustrations of commonly-used skip connections: (a) Conventional
Residual Unit [He et al. (2016a]; (b) Residual Unit in Highway Net [Srivastava
et al. (2015]; (c) Residual Unit in PreAct-ResNet [He et al. (2016b], where BN
represents batch normalization [Ioffe and Szegedy (2015]; (d) Residual Unit in
Transformer [Vaswani et al. (2017], where LN represents layer normalization
[Ba et al. (2016].

A missing piece from the existing work is how would the residual block perform
if $\mathcal{G}$ is realized as layer normalization and $\lambda$ does not
equal one. Although Transformer [Vaswani et al. (2017] has shown the
effectiveness of the combination of layer normalization and skip connection,
it is intuitive that the modulating factor $\lambda$ may not always be one,
especially when $\mathcal{F}$ is not well trained and not sufficient in
representation ability. In addition, the intuition is supported by ?) who
found that if using a gate mechanism to determine the balance between the two
components, the skip component wins in almost all situations. However, in the
study of ?)222They move the batch normalization applied to
$\mathcal{F}(\boldsymbol{x},\mathcal{W})$ forward and apply it to
$\boldsymbol{x}$ instead. From another point of view, it is similar to
applying the batch normalization to $\boldsymbol{y}$ of the previous block.,
which discussed the combination of batch normalization, learned $\lambda$, and
convolutional neural networks, they found empirically that the best
performance is achieved when $\lambda$ is fixed as one, suggesting the
shortcut and the transformed input should be of equal importance.

Therefore, in order to explore whether the optimal ratio of $\boldsymbol{x}$
to $\mathcal{F}(\boldsymbol{x},\mathcal{W})$ is one in the residual block, and
to extend the understandings of the residual block formed as
$\boldsymbol{y}=\mathcal{G}(\lambda\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$,
we propose three different implementations of the residual block in
combination with layer normalization. The first implementation, which we name
Expanded Skip Connection, simply expands the skip connection and sets
$\lambda$ larger than one, following the findings of ?). The second variation,
named as Expanded Skip Connection with Layer Normalization, includes the layer
normalization after the expanded skip connection, since layer normalization is
observed to be helpful in facilitating the optimization of skip connection as
in ?). The final proposal, Recursive Skip Connection with Layer Normalization,
is a novel combination that does not fit in the general form of the residual
block, which takes the advantages of skip connection and layer normalization
in a recursive manner, so that the input signal is amplified yet with layer
normalization stabilizing the optimization. The illustrations of those
combinations are shown in Figure 2. Our experiments on diverse tasks,
including machine translation and image classification with strong baselines,
i.e., Transformer and ResNet, conclude the following:

  * •

The expanded skip connection without any normalization indeed causes serve
gradient malformation, leading to unsatisfactory learning of the neural
networks. Layer normalization helps to deal with the optimization problem
introduced by the expanded skip connection to a certain extent.

  * •

The proposed recursive skip connection with layer normalization further
facilities the optimization by separating the expanded skip connection into
multiple stages to better incorporate the effect of the transformed input. It
achieves comfortable margin on datasets from different domains in extensive
experiments to determine the proper granularity of the skip connection.

  * •

The experimental results on the WMT-2014 EN-DE machine translation dataset
using Transformer further proves the effectiveness and the efficiency of the
recursive architecture, which helps a model to perform even better than the
model three times larger.

##  2 Related Work

In recent years, a large number of skip connection methods have been used for
neural networks [He et al. (2016a, Srivastava et al. (2015, He et al. (2016b,
Vaswani et al. (2017, Szegedy et al. (2017, Wang et al. (2019, Nguyen and
Salazar (2019, Liu et al. (2020a, Xu et al. (2019, Bachlechner et al. (2020,
Liu et al. (2020b, Shen et al. (2020, Xiong et al. (2020]. For the skip
connection, there are two main types of problems. The first is how should the
skip connection be incorporated into the existing neural network structures so
that the best improvements can be achieved. The second is how should the
neural network with skip connections be optimized so that its representation
capability could be fully mined. Therefore, we categorized the related work
into On the Connection Problem and On the Optimization Problem.

Figure 2: Various combination of skip connection and layer normalization: (a)
Original skip connection, (b) Expanded skip connection, (c) Expanded skip
connection with layer normalization, and (d) Recursive skip connection with
layer normalization ($\lambda=2$ as an example), where LN represents layer
normalization.

##### On the connection problem

Highway network [Srivastava et al. (2015] built a highway connection from the
input to the output, similar to the skip connection. However, a transform gate
was proposed to control the balance of the input $\boldsymbol{x}$ and the
transformed input $\mathcal{F}(\boldsymbol{x},\mathcal{W})$, instead of using
identity mapping to combine the input and the transformed input. ?) designed
five types of skip connections and discussed the possible skip connections in
detail. Based on their theory and experiments, they argued that it is likely
for a model to perform the best when the skip connection is directly combined
into the output without any additional scaling such that the skip and the
transformed input have equal contribution to the output. The reason is that
with scaling, the gradient of the skip suffers from the gradient exploding or
vanishing problem, which hinders the deep neural network from efficient
optimization.

##### On the optimization problem

Since the early work has established that it is sufficient to directly combine
skip connection into forward propagation of neural networks without any
scaling [He et al. (2016b], the succeeding studies on the optimization problem
mostly followed the simple yet effective architecture of the skip connection.
In computer vision task, PreAct-ResNet [He et al. (2016b] demonstrated that it
is helpful to move the batch normalization applied to
$\mathcal{F}(\boldsymbol{x},\mathcal{W})$ forward and apply it to
$\boldsymbol{x}$ instead. From another point of view, it is similar to
applying the batch normalization to $\boldsymbol{y}$ of the previous block. In
natural language processing tasks, Transformer [Vaswani et al. (2017] made use
of skip connection and layer normalization extensively in its architecture,
and from our experiments, the layer normalization seems essential to the
overall performance of the model and can help the optimization of the non-
linear transformation to some extent.

In all, we propose to use the form
$\boldsymbol{y}=\mathcal{G}(\lambda\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$
to summarize the combination of normalization and skip connection in existing
work. The form is in accordance with the findings that the normalization
$\mathcal{G}$ should be placed after the addition of the skip and it could be
beneficial to elevate the portion of the input information with $\lambda$.
Moreover, this work distinguishes itself in that (a) we mainly focus on the
combination of layer normalization and skip connection, which is less
investigated but deems promising, (b) we rethink the common practice that the
skip need not scale in consideration of the help of layer normalization, (c)
we conduct experiments on both representative computer vision and natural
language processing tasks, while existing work only provides result in a
single modality, and most importantly (d) we escape from the general form of
the residual block that generalizes all previous work and propose a novel
recursive construction of the residual block with layer normalization that
outperforms all the variants of the general form that are examined in this
work.

##  3 Architecture

In this section, we describe three skip connection constructions, which are
illustrated in Figure 2. They are based on the integration of more input
information and application of the layer normalization.

##### E _x_ panded _Skip_ Connection (xSkip)

The expanded skip connection is defined similar to the constant scaling method
in ?) as

$\boldsymbol{y}=\lambda\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}),$
(2)

where $\boldsymbol{x}$ and $\boldsymbol{y}$ are the input and the output of
the residual block, respectively. $\mathcal{F}$ denotes the weighted neural
network layer, and $\lambda$ is a modulating scalar. Naturally, this
construction adjusts the importance of the skip considering that the neural
network layer may come with different representation ability and optimization
difficulty. However, it should be noted that in this work $\lambda$ is fixed
to isolate the effect of scaling. While it is possible that a learned
$\lambda$ may better capture the balance between the two components, the
learning of $\lambda$ becomes another variable. Following ?), we mainly
experiment with $\lambda>1$.

##### E _x_ panded _Skip_ Connection with Layer Normalization (xSkip+LN)

Motivated by Transformer, which combines skip connection with layer
normalization, we further examine the effect of layer normalization on
expanded skip connection, which takes the form of

$\boldsymbol{y}=\text{LN}(\lambda\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})).$
(3)

Layer normalization may help mitigate the gradient malformation in
optimization caused by the modulating factor. We use the parameterized layer
normalization as suggested in ?), because different input features naturally
have different importance. Different from batch normalization which acts in
“sample space”, layer normalization acts in “feature space”. Considering the
special case that $\mathcal{F}(\boldsymbol{x},\mathcal{W})$ is ill-learned and
outputs random values $\boldsymbol{\epsilon}$ and the examples
$\mathbf{X}=\\{\boldsymbol{x}_{i}\\}$ in the mini-batch all contain the same
input feature $f$ of the same value $v$, batch normalization simply ignores
the input feature $f$ which results in uneventful optimization according to
mini-batch statistics, while layer normalization still recognizes the input
feature $f$ and its value can be properly adjusted according to the other
input features. The same may also apply to the situation where
$\lambda\boldsymbol{x}$ is dominant compared to
$\mathcal{F}(\boldsymbol{x},\mathcal{W})$ and the transformed input falls into
the range comparable to random noise. In such situations, which may happen
when the neural network is hard to optimize, layer normalization could still
help the learning of the shortcut, while batch normalization might fail.

#####  _R_ ecursive _Skip_ Connection with _L_ ayer _N_ ormalization
(rSkip+LN)

Another way to stabilize the gradient is to keep $\lambda=1$ each time but
repeatedly add the shortcut with layer normalization, such that more input
information is also modeled. It is defined recursively as

$\displaystyle\boldsymbol{y}_{\lambda}=\text{LN}(\boldsymbol{x}+\boldsymbol{y}_{\lambda-1})\quad\text{and}\quad\boldsymbol{y}_{1}=\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})),$
(4)

where $\lambda$ should be an integer no smaller than one. For example, when
$\lambda=1$, it regresses to the block used in Transformer and conforms to the
findings that the skip need not scale. A further example is when $\lambda=2$,
$\boldsymbol{y}_{2}=\text{LN}(\boldsymbol{x}+\boldsymbol{y}_{1})=\text{LN}(\boldsymbol{x}+\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})))$,
which is shown in Figure 2. Through recursive skip connection with layer
normalization, the model is encouraged to use layer normalization multiple
times to improve optimization, and can incorporate more information of
$\boldsymbol{x}$ by the skip connection. Moreover, the model may gain more
expressive power than simply incorporating the scaled skip at once, since each
recursive step essentially constructs a different feature distribution and the
recursive structure can learn the adaptive ratio of $\boldsymbol{x}$ to
$\mathcal{F}(\boldsymbol{x},\mathcal{W})$. The ratio between the input
$\boldsymbol{x}$ and the residual $\mathcal{F}(\boldsymbol{x},\mathcal{W})$ is
$\nicefrac{{\boldsymbol{x}}}{{\mathcal{F}(\boldsymbol{x},\mathcal{W})}}=\nicefrac{{\sigma_{1}}}{{\boldsymbol{w}_{1}}}+1$,
where $\sigma_{1}$ is the standard deviation of
$\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})$ and
$\boldsymbol{w}_{1}$ is the gain parameters of the inner layer normalization,
which penetrate through the normalization and effectively act as the learnable
ratio. It is also important to note that the ratio is also dependent on the
current layer distribution, which could serve as a regularization component.
For any $\lambda$ larger than 2, the general form of the ratio is
$1+\sum_{i=1}^{\lambda}\prod_{j=1}^{i}\nicefrac{{\sigma_{j}}}{{\boldsymbol{w}_{j}}}$.
This derivation, which is detailed in Appendix A, also indicates that setting
$\lambda=2$ should be sufficient for learning the shortcut-residual ratio and
adjusting the ratio based on instant data distribution. Yet with larger
$\lambda$, the distribution of the higher order combination of
$\boldsymbol{x}$ and $\mathcal{F}(\boldsymbol{x})$ will be considered. The
recursive definition introduces gradual and smooth combination of the input
and the residual up to the specified order $\lambda$ which could deal with
different distributions better.

##  4 Empirical Study

In this section, we examine the effect of the discussed combinations on two
representative baselines through empirical studies. First, we conduct
exploratory experiments on datasets that are relatively small to gain insights
of the combinations, such as the choice of $\lambda$ and the effectiveness of
the constructions. Then, we discuss the observed results and conduct further
analysis. Finally, we validate our findings on larger and more difficult
datasets.

###  4.1 Tasks and Settings

For ease of introduction, brief descriptions of the tasks and corresponding
models are summarized together in the following. As the combinations to be
examined only relates to the skip connection and we keep the inner structure
of the residual block untouched, please refer to the cited work for more
detailed description of the baselines.

####  4.1.1 Image Classification

##### Datasets

CIFAR-10 and CIFAR-100 [Krizhevsky (2009] are two datasets for colored image
classification. Both contain 50K images for training and 10K images for
testing. The former has 10 classes, while the latter has 100 classes and is
considered more difficult. We report the classification error rates as the
evaluation of the performance.

##### Baselines and settings

We conduct experiments using the ResNet with pre-activation (PreAct-ResNet) as
proposed in ?). For experiments on CIFAR-10, we use ResNet-110 for exploration
because of its deep structures. It contains 54 two-layer residual blocks,
which makes it a challenging baseline for optimization. We validate the
proposal on CIFAR-100, using a variety of ResNet structures, such as
ResNet-20, ResNet-32, ResNet-44, ResNet-56, and ResNet-110, to ensure the
generality of the results. We mostly adopt the hyperparameters from ?) to keep
the number of variables controlled and comparable, except that we use a weight
decay rate of 0.0002.

####  4.1.2 Machine Translation

##### Datasets

IWSLT 2015 English-Vietnamese dataset (EN-VI) [Cettolo et al. (2015]
consisting of 133K sentence pairs, WMT 2014 English-German dataset (EN-DE)
consisting of 4.5M sentence pairs and WMT 2014 English-French dataset (EN-FR)
consisting of 36M sentence pairs are for machine translation. Following
previous work [Luong and Manning (2015], we use tst2012 and tst2013 as
development set and test set, respectively for EN-VI dataset. For EN-DE, we
use newstest2013 for development and newstest2014 for testing. For EN-FR, we
validate on newstest2012+2013 and test on newstest2014. The evaluation metric
is BLEU [Papineni et al. (2002] against the reference translations.

##### Baselines and settings

Transformer [Vaswani et al. (2017] is a representative model for machine
translation, which includes skip connection and layer normalization in the
model structure. Transformer follows the encoder-decoder framework, except
that it substitutes the conventional recurrent layers with the self-attention
based blocks, i.e., $\mathcal{F}(\boldsymbol{x},\mathcal{W})$ consists of a
dot-product based attention layer and two feed-forward layers. Although
Transformer is not as deep as ResNet-110, it is heavily parameterized and the
commonly-used six-layer Transformer-Base structure embodies 65M learnable
parameters, which also poses challenge for optimization. We base our
experiments on the tensor2tensor [Vaswani et al. (2018] code. For experiments
on IWSLT-2015, we set batch size to 4096 and train on single GPU, as it is
relatively small. For experiments on WMT-2014 that is much larger, due to the
complication caused by the special batching algorithm and the distributed
learning, we find setting batch size to 16,384 tokens in total on 3 GPUs can
reproduce the reported results on 8 GPUs using batch size of 32,768 tokens in
total [Vaswani et al. (2017]. However, the training steps required are
increased from 100,000 steps to 550,000 steps. The rest of hyperparameters are
the same as ?).

Method | Architecture | $\lambda$ | $\mathcal{G}$ | Error (%)
---|---|---|---|---
Baseline | $\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})$ | 1 | - | 6.37
1xSkip | $\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})$ | 1 | - | 6.31
2xSkip | $2\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})$ | 2 | - | 8.41
3xSkip | $3\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})$ | 3 | - | 9.23
1xSkip+LN | $\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 1 | LN | 7.72
2xSkip+LN | $\text{LN}(2\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 2 | LN | 6.29
3xSkip+LN | $\text{LN}(3\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 3 | LN | 6.07
1rSkip+LN | $\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 1 | LN | 7.72
2rSkip+LN | $\text{LN}(\boldsymbol{x}+\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})))$ | 2 | LN | 6.02
3rSkip+LN | $\text{LN}(\boldsymbol{x}+\text{LN}(\boldsymbol{x}+\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))))$ | 3 | LN | 6.55
Table 1: Results on CIFAR-10 using PreAct-ResNet-110. Average of 5 runs. The
lower the better. The underlined numbers denote the best results under the
same comparable settings. $\lambda$xSkip, $\lambda$xSkip+LN, and
$\lambda$rSkip+LN stand for expanded skip connection, expanded skip connection
with LN, and recursive skip connection with LN, respectively. Please note that
1xSkip is equivalent to Baseline and 1rSkip+LN is equivalent to 1xSkip+LN. It
is clear that simply expanding the skip connection hurts the performance and
using layer normalization can mitigate the problem.

###  4.2 Exploratory Results

In this section, we evaluate the effect of the proposed constructions and
report the exploratory results on CIFAR-10 using ResNet and IWSLT-2015 using
Transformer.

The main results on CIFAR-10 using PreAct-ResNet-110 are summarized in Table
1. We experiment with different constructions and different $\lambda$s. 1xSkip
is our reimplementation of the ResNet-110 with pre-activation and outperforms
the reported result in ?) by 0.06. This margin could approximately serve as
the boundary deciding whether the improvement is caused by random variations
or experimental settings. As we can see, for skip connection without
normalization, it is indeed the best to add the input without scaling and
expanding the skip results in significant loss in accuracy. However, with the
application of layer normalization, the expanded skip can also be sufficiently
optimized and surpass the baseline, which is suggested by ?) by extensive
structure search. Moreover, with the recursive design, the performance can
also be maintained and yet further improved, which reaches the best result in
our experiments when using $\lambda=2$. It is also interesting to see that in
combinations with layer normalization, the $\lambda$ that results in the best
accuracy is also different. It is intuitive that the recursive structure
adjusts the effective ratio of the skip and the transformed input more
sophisticatedly than the plain expanding structure so that different $\lambda$
may be required.

Table 2 shows the results on IWSLT-2015 using Transformer and the results
about layer normalization are similar. With the increasing of the weight of
the skip and layer normalization, the performance of the model improves. Since
the models incorporated expanded skip connection with layer normalization have
the same number of parameters, regardless of $\lambda$, thus embodying the
same representation ability, the improved performance should mainly be
attributed to the enhanced shortcut information. Moreover, the recursive
version brings significant improvements upon the simply expanding version,
pushing the BLEU score past 31 and reaching a margin of 1.1 BLEU scores at
most, demonstrating the advantage of gradually adding the expanded skip to
ease the optimization. A notable difference from the results on PreAct-
ResNet-110 is that the plain skip connection version does not perform better
than the one with layer normalization (30.31 BLEU vs. 28.98 BLEU on
Transformer and 92.28 accuracy vs. 93.69 accuracy on PreAct-ResNet-110). We
find that the reason is likely to be that in the residual block of PreAct-
ResNet-110, there are already two instances of batch normalization before each
convolutional layer to facilitate its optimization, while in the Transformer
block no other normalization technique exists.

In summary, our experiments show that feeding more input signals to the layer
output should be helpful to learning of the overall model. Previous concerns
on the optimization difficulty introduced by the expanded skip connection
could be alleviated by the usage of layer normalization. Especially, the
proposed recursive skip connection with layer normalization brings the most
improvements by dividing the expanded skip and iteratively adding the pieces
to the output, which can further stabilize the gradient on the input and help
the training of the weighted layer.

Method | Architecture | $\lambda$ | $\mathcal{G}$ | BLEU
---|---|---|---|---
Baseline | $\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 1 | - | 30.31
1xSkip | $\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})$ | 1 | - | 28.98
1xSkip+LN | $\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 1 | LN | 30.31
2xSkip+LN | $\text{LN}(2\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 2 | LN | 30.99
3xSkip+LN | $\text{LN}(3\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 3 | LN | 30.92
1rSkip+LN | $\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 1 | LN | 30.31
2rSkip+LN | $\text{LN}(\boldsymbol{x}+\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})))$ | 2 | LN | 31.45
3rSkip+LN | $\text{LN}(\boldsymbol{x}+\text{LN}(\boldsymbol{x}+\text{LN}(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))))$ | 3 | LN | 31.10
Table 2: Results on EN-VI machine translation dataset using Transformer. The
higher the better. $\lambda$xSkip, $\lambda$xSkip+LN, and $\lambda$rSkip+LN
are defined similarly. Please note that Baseline, 1xSkip+LN, and 1rSkip+LN are
equivalent. It is clear that without layer normalization, Transformer
experiences significantly performance degradation. By expanding the shortcut,
substantially better results can be achieved. The best improvements of over
1.1 BLEU scores are brought by the proposed recursive skip connection.

###  4.3 Discussions

However, questions remain whether the improvements can also be achieved by
contracting skip connection and whether batch normalization can achieve the
same effect which is more commonly in convolutional neural networks. We
conduct the following analysis to answer these questions. Furthermore, we
demonstrate that layer normalization adaptively adjusts the gradients of the
scaled skip, solving the gradient malformation caused by the modulating factor
$\lambda$.

##### Contracting skip connection

We experiment two different ways to relatively tune down the skip ratio, i.e.,
using smaller $\lambda$ and expanding
$\mathcal{F}(\boldsymbol{x},\mathcal{W})$. The results are shown in Table 5.
As we can see, if the shortcut is forced to play the minor role, the deep
neural network fails to be optimized properly and shows catastrophic
performance. In contrast, the expanded skip causes less harm to the overall
performance (cf. Table 1), indicating the importance of linear component in
the optimization of deep neural networks. It is consistent with the findings
of ?) and ?). Moreover, the results also show that applying layer
normalization under this situation further deteriorates the performance.

##### Using batch normalization

Since batch normalization has been widely used in CNNs, we replace layer
normalization with batch normalization to see the difference. However,
comparing Table 1 and Table 5, it can be seen that batch normalization is less
effective than layer normalization in combination with skip connection. The
proposed recursive structure can help batch normalization deal with the
expanded skip as each time the ratio is still kept as one. As we explained in
Section 3, batch normalization makes use of mini-batch statistics which may
prevent the plainly expanded skip from coming into effect.

Figure 3: Gradient norm of the output of residual blocks of various
constructions on CIFAR-10 using PreAct-ResNet-110. We show the gradient norm
of the output $\boldsymbol{y}$ of the residual blocks of middle depth in the
model averaged over 10,000 random examples. By definition, $\boldsymbol{y}$
equals $\boldsymbol{x}$ of the following block.

Architecture | Error (%)
---|---
$\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})$ |  06.31
$0.5\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W})$ | 18.61
$\text{LN}(0.5\boldsymbol{x}+\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 29.88
$\text{LN}(\boldsymbol{x}+2*\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 24.98
$\text{LN}(\boldsymbol{x}+3*\mathcal{F}(\boldsymbol{x},\mathcal{W}))$ | 61.02
Table 3: Results on CIFAR-10 with contracted skip connection. Lowering the
skip ratio leads to catastrophic results and LN further aggravates the
performance.

ResNet-110 | $\lambda$ | $\mathcal{G}$ | Error (%)
---|---|---|---
1xSkip | 1 | - | 6.31
1xSkip+BN | 1 | BN | 8.23
2xSkip+BN | 2 | BN | 6.43
3xSkip+BN | 3 | BN | 6.50
1rSkip+BN | 1 | BN | 8.23
2rSkip+BN | 2 | BN | 6.77
3rSkip+BN | 3 | BN | 6.68
Table 4: Results on CIFAR-10 with BN. Compared to LN, BN is less effective
dealing with the expanded skip connection.

Transformer | # Blocks | BLEU
---|---|---
1xSkip+LN | 6 | 30.31
2rSkip+LN | 6 | 31.45
1xSkip+LN | 2 | 30.04
2rSkip+LN | 2 | 30.49
Table 5: Results on Transformer of fewer blocks. As we can see, with the
proposed combination of recursive skip connection, the same performance could
be achieved by fewer parameters.

##### Parameter efficiency

A byproduct of the proposed structure is that it could be possible to use
fewer parameters to reach the same performance, especially for Transformer,
which is heavily parameterized, as the expanded skip connection and layer
normalization both helps to ease the optimization. Fewer parameter may
translate into lower time cost. We conduct experiments on the IWSLT-2015 EN-VI
machine translation dataset using Transformer with two blocks instead of six
blocks used in the main experiments. The results are shown in Table 5.
Surprisingly, a Transformer of two blocks does not perform substantially
worse, and with the proposed rSkip+LN, the reduction could be easily made up.
A interesting phenomenon is that the proposed method gains larger margin on
deeper models, which also attests that proposed combination helps the
optimization of the model, so that the performance of deeper model could be
further exploited.

##### Better Optimization

To see how the proposed combinations help the optimization of the deep neural
models, in Figure 3, we show the averaged gradient norm of the output of each
residual block in pretrained ResNet-110 on CIFAR-10 based on 10,000 random
training examples. Due to limited space, we only show the representative
blocks at the “middle” of the model, meaning those blocks have the same filter
size but different from the blocks lower and the blocks higher. As we can see,
the original construction 1xSkip shows slight gradient “explosion”, while
simply scaling the skip causes severe gradient malformation (e.g., 2xSkip and
0.5xSkip). With normalization techniques such as layer normalization and batch
normalization, the gradient malformation is generally mitigated. However, the
magnitude of the gradient is different, with layer normalization more
accurately reflecting the modulating factor $\lambda$ (e.g. 1xSkip and
1xSkip+LN have similar values, 2xSkip+LN is approximately twice 1xSkip, etc.),
which is intuitive since batch normalization is based on mini-batch statistics
unaware of the feature space. The recursive structure 2rSkip+LN is similar to
2xSkip+LN, except that its norm is larger. Especially, note the gradient at
the Block 36, which receives gradient from the higher block that have
different filter size; xSkip+LN and xSkip+BN seem to fail normalize its
gradient properly, while rSkip+LN successfully elevates its scale, suggesting
it could mitigate the difference between blocks of different nature as well.
The last but not the least, layer normalization does not adjust the effective
scale of the skip connection in forward propagation such that the effective
scale becomes 1 and regresses to the plain residual structure. The learned
effective scale is 2.82 and 2.64 for 2xSkip+LN and 2rSkip+LN, respectively,
averaged over the shown blocks.

Figure 4: Learning curve for 2rSkip+LN and $w$Skip initialized as
$\boldsymbol{1}$ on CIFAR-10 using ResNet-110. It can be observed that $w$Skip
has difficulty in start training.

##### Learning $\boldsymbol{\lambda}$

To automatically adjust the ratio between the skip and the residual branch, a
trivial extension is to replace the fixed scalar $\lambda$ with a learnable
scalar or vector. We also try this method on CIFAR-10 using ResNet-110.
Concretely, we adopt the form of
$\boldsymbol{y}=\text{LN}(\boldsymbol{w}\cdot\boldsymbol{x}+\mathcal{F}(\boldsymbol{x}))$
($\boldsymbol{w}$Skip+LN), as using a parameter vector should be more general
and a vector covers all the possible solutions of a scalar. However, the
findings are not in favor of this approach over the fixed scalar version. When
the $\boldsymbol{w}$ is initialized as $\boldsymbol{1}$, the model experiences
significantly performance degradation. The averaged error rate over 5 runs is
8.66 and the final norm of a sampled $\boldsymbol{w}\in\mathbb{R}^{32}$ at
middle layers is 1.34, suggesting very small scale for each dimension of the
skip connection (0.24 on average). The small scale of the skip connection
indicates that the model relies more on the residual branch that is
parameterized and optimized for the training data, entailing a higher
overfitting risk. It can be confirmed from Figure 4, where we show the
learning curve of $w$Skip compared to 2rSkip (+LN is omitted for conciseness).
$w$Skip and 2rSkip both achieve low training loss, but $w$Skip has difficulty
in starting learning and has higher validation loss. We also try different
initialization strategy for $w$Skip, e.g., initializing $\boldsymbol{w}$ as
$\boldsymbol{2}$ since we find 2 is a good empirical value for rSkip+LN. We
observe a performance of 6.45 in terms of error rate, which is indeed better
but still worse than rSkip+LN. Moreover, such strategy would make $w$Skip no
better than using pre-determined fixed scalar values, as a proper
initialization value still needs to be searched.

###  4.4 Validation Results

We verify the proposed combination of recursive skip connection with layer
normalization on more difficult datasets, i.e., CIFAR-100, WMT-2014 EN-DE and
EN-FR datasets. Considering the training budget, we only validate the
performance of 2rSkip+LN based on the exploratory results.

The results are shown in Table 6 and Table 7, respectively, which demonstrate
the effectiveness of the proposed recursive combination that brings consistent
improvements regardless of the model and the dataset. The results are in
accordance with our findings in exploratory experiments as well. For
CIFAR-100, the best improvement of 1.02 absolute error rates is achieved upon
the baseline that is the deepest and should be the most difficult to optimize.
The relative improvements across models of different depth also suggest that
the representation ability of the deep models still needs to be developed and
transferred into real performance elevation, while the proposal shows
potential. The results on EN-VI, EN-DE and EN-FR datasets are also
encouraging. With approximately one third of the parameters, the Transformer-
Base model could be improved to compete with the Transformer-Big model on EN-
VI and EN-DE, with the help of the refined skip connection architecture. It
again demonstrates the potential of the proposal in realizing the capability
of the models by inducing more efficient optimization.

Dataset | Method | ResNet-20 | ResNet-32 | ResNet-44 | ResNet-56 | ResNet-110
---|---|---|---|---|---|---
CIFAR-100 | Baseline | 33.38 | 32.00 | 30.50 | 29.81 | 28.71
\+ proposal | 32.99 (+0.39) | 31.39 (+0.61) | 29.85 (+0.65) | 28.98 (+0.83) | 27.69 (+1.02)
Table 6: Results on CIFAR-100 using ResNet and the proposed 2rSkip+LN skip connection construction. Average of 5 runs. The numbers in parentheses denote the improvements. The proposal brings consistent improvements. The improvements are more evident for deeper and complex models. Method | # Parameters | EN-VI | EN-DE | EN-FR
---|---|---|---|---
Transformer-Big [Vaswani et al. (2017] | 213M | - | 28.4 | 41.0
Deep Representations [Dou et al. (2018] | 356M | - | 29.2 | -
Evolved Transformer [So et al. (2019] | 218M | - | 29.8 | 41.3
Fixup [Zhang et al. (2019] | - | - | 29.3 | -
AdaNorm [Xu et al. (2019] | - | 31.4 | 28.5 | -
ScaleNorm [Nguyen and Salazar (2019] | - | 32.8 | 27.6 | -
Admin [Liu et al. (2020b] | - | - | 29.0 | 43.8
MUSE [Zhao et al. (2019] | - | 31.3 | 29.9 | 43.5
Transformer-Base | 65M | 30.3 | 27.9 | 38.2
\+ proposal |  $+$0.03M | 31.5 (+1.2) | 28.6 (+0.7) | 39.0 (+0.8)
Transformer-Big | 213M | 31.4 | 28.5 | 41.0
\+ proposal |  $+$0.06M | 32.2 (+0.8) | 29.0 (+0.5) | 41.6 (+0.6)
Table 7: Results on EN-VI, EN-DE and EN-FR using Transformer and the proposed
2rSkip+LN skip connection construction. The numbers of parameters are reported
for the EN-DE models. As we can see, the proposal brings consistent
substantial improvements.

##  5 Conclusion and Future Work

In this work, we focus on exploring the combination of skip connection and
layer normalization to build a novel skip connection architecture able to
incorporate reasonable skip information at output level and solve the gradient
explosion problem in practice. According to the empirical studies, we find
that expanding skip information should be helpful to the optimization of the
model, which is hindered by the gradient malformation in practice, and layer
normalization does better than batch normalization in mitigating such problem.
The proposed Recursive Skip Connection with Layer Normalization guarantees the
stability of the shorcut gradient by recursively dividing the expanded skip
connection and adding it to the output iteratively with layer normalization.
The refined skip connection architecture may enable more efficient
optimization. Validation on CIFAR-100, WMT-2014 EN-DE and EN-FR proves the
findings and the effectiveness of the proposed recursive skip connection
construction with best improvements over 1.0 absolute point, demonstrating its
generalization ability to a wide range of existing systems.

Unfortunately, a limitation of our investigation is that $\lambda$ is selected
as integer and only a small number of values are used, which may not be of the
best granularity for expanding the skip connection. A more desirable solution
would be to adaptively determine the value of $\lambda$, maybe in terms of the
gradient distribution during the learning so that more efficient optimization
could be realized. We will further seek to address this limitation in future
work.

## Acknowledgments

This work is partly supported by AOTO-PKUSZ Joint Research Center for
Artificial Intelligence on Scene Cognition Technology Innovation, and Beijing
Academy of Artificial Intelligence (BAAI). We thank all the anonymous
reviewers for their constructive comments and suggestions.

## References

  * [Ba et al. (2016]  Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton.  2016\.  Layer normalization.  CoRR, abs/1607.06450.
  * [Bachlechner et al. (2020]  Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian J. McAuley.  2020\.  Rezero is all you need: Fast convergence at large depth.  CoRR, abs/2003.04887.
  * [Cettolo et al. (2015]  Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico.  2015\.  The iwslt 2015 evaluation campaign.  In International Workshop on Spoken Language Translation 2015, IWSLT 2015.
  * [Dou et al. (2018]  Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang.  2018\.  Exploiting deep representations for neural machine translation.  In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4253–4262. Association for Computational Linguistics.
  * [He et al. (2016a]  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  2016a.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society.
  * [He et al. (2016b]  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  2016b.  Identity mappings in deep residual networks.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 630–645. Springer.
  * [Ioffe and Szegedy (2015]  Sergey Ioffe and Christian Szegedy.  2015\.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 448–456. JMLR.org.
  * [Krizhevsky (2009]  Alex Krizhevsky.  2009\.  Learning multiple layers of features from tiny images.  Technical report, Computer Science Department, University of Toronto.
  * [Liu et al. (2020a]  Fenglin Liu, Xuancheng Ren, Guangxiang Zhao, and Xu Sun.  2020a.  Layer-wise cross-view decoding for sequence-to-sequence learning.  CoRR, abs/2005.08081.
  * [Liu et al. (2020b]  Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han.  2020b.  Understanding the difficulty of training transformers.  CoRR, abs/2004.08249.
  * [Luong and Manning (2015]  Minh-Thang Luong and Christopher D. Manning.  2015\.  Stanford neural machine translation systems for spoken language domain.  In International Workshop on Spoken Language Translation 2015, IWSLT 2015.
  * [Nguyen and Salazar (2019]  Toan Q. Nguyen and Julian Salazar.  2019\.  Transformers without tears: Improving the normalization of self-attention.  In International Workshop on Spoken Language Translation 2019, IWSLT 2019.
  * [Papineni et al. (2002]  Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  2002\.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA., pages 311–318. ACL.
  * [Shen et al. (2020]  Sheng Shen, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer.  2020\.  Powernorm: Rethinking batch normalization in transformers.  CoRR, abs/2003.07845.
  * [So et al. (2019]  David R. So, Quoc V. Le, and Chen Liang.  2019\.  The evolved transformer.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5877–5886. PMLR.
  * [Srivastava et al. (2015]  Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber.  2015\.  Highway networks.  In ICML 2015 Deep Learning Workshop.
  * [Szegedy et al. (2016]  Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  2016\.  Rethinking the inception architecture for computer vision.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818–2826. IEEE Computer Society.
  * [Szegedy et al. (2017]  Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi.  2017\.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In Satinder P. Singh and Shaul Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA., pages 4278–4284. AAAI Press.
  * [Vaswani et al. (2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6000–6010.
  * [Vaswani et al. (2018]  Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit.  2018\.  Tensor2tensor for neural machine translation.  In Colin Cherry and Graham Neubig, editors, Proceedings of the 13th Conference of the Association for Machine Translation in the Americas, AMTA 2018, Boston, MA, USA, March 17-21, 2018 - Volume 1: Research Papers, pages 193–199. Association for Machine Translation in the Americas.
  * [Wang et al. (2019]  Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao.  2019\.  Learning deep transformer models for machine translation.  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1810–1822. Association for Computational Linguistics.
  * [Xiong et al. (2020]  Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu.  2020\.  On layer normalization in the transformer architecture.  CoRR, abs/2002.04745.
  * [Xu et al. (2019]  Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin.  2019\.  Understanding and improving layer normalization.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 4383–4393.
  * [Zhang et al. (2019]  Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma.  2019\.  Fixup initialization: Residual learning without normalization.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  * [Zhao et al. (2019]  Guangxiang Zhao, Xu Sun, Jingjing Xu, Zhiyuan Zhang, and Liangchen Luo.  2019\.  MUSE: parallel multi-scale attention for sequence to sequence learning.  CoRR, abs/1911.09483.

##  Appendix A On Adaptive Ratio of Recursive Skip Connection with Layer
Normalization

To see how the recursive structure can learn the ratio, we take $\lambda=2$ as
an example and unroll the formula:

$\displaystyle\boldsymbol{y}_{2}$
$\displaystyle=\text{LN}\left(\boldsymbol{x}+\text{LN}\left(\boldsymbol{x}+\mathcal{F}(\boldsymbol{x})\right)\right)$
$\displaystyle=\boldsymbol{w}_{2}\cdot\left(\frac{\boldsymbol{x}+\left(\boldsymbol{w}_{1}\cdot\left(\frac{\boldsymbol{x}+(\mathcal{F}(\boldsymbol{x}))-\mu_{1}}{\sigma_{1}}\right)+\boldsymbol{b}_{1}\right)-\mu_{2}}{\sigma_{2}}\right)+\boldsymbol{b}_{2}$
$\displaystyle=\left(\frac{\boldsymbol{w}_{2}}{\sigma_{2}}+\frac{\boldsymbol{w}_{2}\cdot\boldsymbol{w}_{1}}{\sigma_{2}\sigma_{1}}\right)\boldsymbol{x}+\frac{\boldsymbol{w}_{2}\cdot\boldsymbol{w}_{1}}{\sigma_{2}\sigma_{1}}\mathcal{F}(\boldsymbol{x})+C,$
(5)

where $\cdot$ denotes entrywise-product,
$\boldsymbol{w}_{1},\boldsymbol{w}_{2},\boldsymbol{b}_{1},\boldsymbol{b}_{2}$
are the gain and the bias parameters of the affine transformation in layer
normalization, and $\sigma_{1},\sigma_{2},\mu_{1},\mu_{2}$ are the standard
deviation and the mean of $\boldsymbol{x}+\mathcal{F}(\boldsymbol{x})$ and
$\boldsymbol{x}+\boldsymbol{y}_{1}$, respectively. As we can see, the ratio
between the input and the residual is

$\displaystyle\boldsymbol{x}:\mathcal{F}(\boldsymbol{x})\coloneqq\frac{\boldsymbol{w}_{2}}{\sigma_{2}}+\frac{\boldsymbol{w}_{2}\cdot\boldsymbol{w}_{1}}{\sigma_{2}\sigma_{1}}:\frac{\boldsymbol{w}_{2}\cdot\boldsymbol{w}_{1}}{\sigma_{2}\sigma_{1}}=\frac{\sigma_{1}}{\boldsymbol{w}_{1}}+1.$
(6)

Because the recursive definition, the inner gain parameters of layer
normalization could penetrate through the normalization and effectively act as
the learnable ratio. It is also important to note that the ratio is also
dependent on the current layer distribution, which could serve as a
regularization component. For any $\lambda$ larger than 2, the general form of
the ratio is
$1+\sum_{i=1}^{\lambda}\prod_{j=1}^{i}\nicefrac{{\sigma_{j}}}{{\boldsymbol{w}_{j}}}$.
It should be reminded that setting $\lambda=2$ is sufficient for automatically
learning the shortcut-residual ratio and dynamically adjusting the ratio based
on instant data distribution. Yet with larger $\lambda$, the distribution of
the higher order combination of $\boldsymbol{x}$ and
$\mathcal{F}(\boldsymbol{x})$ will be considered. The recursive definition
introduces gradual and smooth combination of the input and the residual up to
the specified order $\lambda$ which could deal with different distributions
better. Our experimental results also demonstrate that the proposal performs
better than the scaled shortcut.


# The $L^{2}$-completion of the space of Riemannian metrics is CAT$(0)$: a
shorter proof

Nicola Cavallucci  Nicola Cavallucci, Karlsruhe Institute of Technology,
Engelstrasse 2, D-76128 Karlsruhe<EMAIL_ADDRESS>
###### Abstract.

We reprove in an easier way a result of Brian Clarke: the completion of the
space of Riemannian metrics of a compact, orientable smooth manifold with
respect to the $L^{2}$-distance is CAT$(0)$. In particular we show that this
completion is isometric to the space of $L^{2}$-maps from a standard
probability space to a fixed CAT$(0)$ space.

Ebin ([Ebi70]) introduced the so-called $L^{2}$-metric on the space of all
Riemannian metrics on a given compact, orientable manifold. This metric has
been extensively studied: its geodesics and curvature has been computed
explicitely by Freed and Groisser ([FG89]) and the properties of the
exponential map has been studied by Gil-Merano and Michor ([GMM91]). Recently
Clarke ([Cla13b]) showed that the completion of the $L^{2}$-metric is a
CAT$(0)$ space. In Clarke’s proof the thinness of geodesic triangles is
verified by direct computations. The scope of this paper is to provide a
shorter and more conceptual proof of the same result. As a by-product of our
proof we deduce the

######  Theorem 1.

The completion of the space of Riemannian metrics on a compact, orientable
$n$-dimensional, smooth manifold $M$ with respect to the $L^{2}$-metric
depends only on the dimension of $M$.

In theorem 3 we will describe explicitly this completion as the space of
$L^{2}$ maps from $M$ to a fixed CAT$(0)$ metric space: that is why it does
not depend on $M$. The CAT$(0)$ property of the target will be detected as a
special case of the warped product theorem of [AB04]. It is exactly this
identification with a space of $L^{2}$ maps with target a CAT$(0)$ space that
proves the CAT$(0)$ property of the completion.

Let $M$ be a orientable, compact, $n$-dimensional, smooth manifold $M$. We
equip $M$ with a smooth Riemannian metric $g_{0}$ of total volume $1$, with
corresponding volume form $\mu_{g_{0}}$. We denote by $S^{2}T_{p}^{*}M$ the
vector space of $(0,2)$-symmetric tensors at $p\in M$ and by
$S^{2}_{+}T_{p}^{*}M$ the subspace of positive-definite ones. The disjoint
union of the spaces $S^{2}_{+}T_{p}^{*}M$ defines naturally a fiber bundle $E$
over $M$ with fiber $P(n)$: the space of $n\times n$ real positive definite
symmetric matrices. By definition a global smooth section of $E$ is a
Riemannian metric on $M$. The set of global smooth sections will be denoted by
$\mathcal{E}_{C^{\infty}}$. On $S^{2}_{+}T_{p}^{*}M$ we consider the following
Riemannian metric. For $h\in S^{2}_{+}T_{p}^{*}M$ and $a,b\in
T_{h}(S^{2}_{+}T_{p}^{*}M)\cong S^{2}T_{p}^{*}M$ we set

(1) $\langle
a,b\rangle_{h,p}=\text{tr}(h^{-1}ah^{-1}b)\cdot\sqrt{\det(g_{0}(p)^{-1}a)}.$

We denote by $d_{p}$ the metric induced by this Riemannian structure on
$S^{2}_{+}T_{p}^{*}M$. Let $\mathcal{E}_{m}$ be the space of all measurable
global sections $\sigma\colon M\to E$, where $M$ and $E$ are equipped with
their Borel $\sigma$-algebras. A section $\sigma\in\mathcal{E}_{m}$ is said to
be square integrable if

$\int_{M}d_{p}^{2}(\sigma(p),g_{0}(p))d\mu_{g_{0}}(p)<\infty.$

We remark that the integral above is meaningful because the distances $d_{p}$
vary continuously with $p$, and so the integrand function is measurable.
Moreover since $M$ is compact the definition above does not depend on the
choice of $g_{0}$. We denote the space of square integrable sections by
$\mathcal{E}_{\mathcal{L}^{2}}$. On $\mathcal{E}_{\mathcal{L}^{2}}$ we define
the equivalence relation $\sigma\sim\sigma^{\prime}$ if and only if
$\mu_{g_{0}}(\\{p\in M\text{ s.t. }\sigma(p)\neq\sigma^{\prime}(p)\\})=0$. The
quotient space is denoted by $\mathcal{E}_{L^{2}}$. For
$\sigma,\sigma^{\prime}\in\mathcal{E}_{\mathcal{L}^{2}}$ we set

$d_{L^{2}}(\sigma,\sigma^{\prime})=\left(\int_{M}d_{p}^{2}(\sigma(p),\sigma^{\prime}(p))d\mu_{g_{0}}(p)\right)^{\frac{1}{2}}.$

An application of the triangle inequality for $d_{p}$ and the classical
inequalities for maps from $M$ to $\mathbb{R}$ shows that $d_{L^{2}}$ is a
finite pseudodistance on $\mathcal{E}_{\mathcal{L}^{2}}$. Moreover
$d_{L^{2}}(\sigma,\sigma^{\prime})=0$ if and only if
$\sigma\sim\sigma^{\prime}$, so $d_{L^{2}}$ defines a distance on
$\mathcal{E}_{L^{2}}$. Of course every continuous section
$\sigma\in\mathcal{E}_{C^{0}}$ is square integrable. Moreover if we restrict
the distance $d_{L^{2}}$ to the space of smooth sections
$\mathcal{E}_{C^{\infty}}$ we obtain the classical $L^{2}$-distance ([Cla13b],
Theorem 3.8). The metric space $(\mathcal{E}_{C^{\infty}},d_{L^{2}})$ is not
complete because the fibers $(S^{2}_{+}T_{p}^{*}M,d_{p})$ are not complete and
because $C^{\infty}$-sections are not complete in the $L^{2}$-sense. Its
completion has been described in [Cla13a], and here we are going to present it
in a very natural way.
Observe that if $(U,\varphi)$ is a chart of $M$ sending the volume form
$\mu_{g_{0}}$ to the standard volume form of $\mathbb{R}^{n}$ then the push-
forward of the scalar product (1) at $p$ on $P(n)$ reads as

(2) $\langle A,B\rangle_{H}=\text{tr}(H^{-1}AH^{-1}B)\cdot\sqrt{\det A},$

for $H\in P(n)$ and $A,B\in T_{H}P(n)$. The distance metric induced on $P(n)$
by this Riemannian metric is denoted by $d_{P(n)}$. Let
$(\bar{P}(n),d_{P(n)})$ be the completion of the metric space
$(P(n),d_{P(n)})$. We cover $M$ by charts sending $\mu_{g_{0}}$ to the
Euclidean volume form. The change of coordinate maps between these charts act
naturally as isometries of $(P(n),d_{P(n)})$. These isometries can be extended
to isometries of $(\bar{P}(n),d_{P(n)})$ and they naturally define a new fiber
bundle on $M$ modeled on $\bar{P}(n)$. We denote this fiber bundle by
$\bar{E}$. Each fiber $\bar{E}_{p}$ is naturally equipped with the metric
$d_{p}$ induced by the metric $d_{P(n)}$ on $\bar{P}(n)$. Moreover the fiber
bundle $E$ has a natural inclusion map of fiber bundles in $\bar{E}$ and the
restriction of this map to each fiber $(E_{p},d_{p})$ is an isometric
embedding in $(\bar{E}_{p},d_{p})$ with dense image. In other words
$(\bar{E}_{p},d_{p})$ is the completion of $(E_{p},d_{p})$. We define as above
the space $\bar{\mathcal{E}}_{L^{2}}$, with the corresponding distance
$d_{L^{2}}$. We have the obvious embeddings

$\mathcal{E}_{C^{\infty}}\subset\mathcal{E}_{C^{0}}\subset\mathcal{E}_{L^{2}}\subset\bar{\mathcal{E}}_{L^{2}}.$

The following result describes the completion of the space of Riemannian
metrics with respect to $d_{L^{2}}$.

######  Theorem 2.

The set $\mathcal{E}_{C^{\infty}}$ is dense in $\bar{\mathcal{E}}_{L^{2}}$
with respect to $d_{L^{2}}$.

The proof is standard, we just sketch it leaving the details to the reader.

###### Proof.

We fix a triangulation of $M$ and we denote by $V_{1},\ldots,V_{k}$ the
interior of the maximal dimensional simplices. We can suppose that each
$V_{i}$ is the support of a trivializing chart for the bundles $\bar{E}$ and
$E$ sending $\mu_{g_{0}}$ to the Euclidean volume form. In order to define a
$L^{2}$-section it is enough to define it on the sets $V_{i}$ since the
complementary of their union has null measure in $M$. A $\bar{E}$-section
(resp. $E$-section) on the set $V_{i}$ can be seen as a map from $V_{i}$ to
$\bar{P}(n)$ (resp. $P(n)$). Let $\sigma\in\bar{\mathcal{E}}_{L^{2}}$ and
$\varepsilon>0$. The restriction $\sigma_{i}$ of $\sigma$ to $V_{i}$ can be
seen as a map from $V_{i}$ to $\bar{P}(n)$. For every $p\in V_{i}$ we can
choose a point $\sigma_{i}^{\varepsilon}(p)$ of $P(n)$ such that
$d_{P(n)}(\sigma_{i}^{\varepsilon}(p),\sigma_{i}(p))<\varepsilon$. Gluing
together the $\sigma_{i}^{\varepsilon}$ we define a section
$\sigma^{\varepsilon}$ of the fiber bundle $E$ such that
$\sigma^{\varepsilon}\in\mathcal{E}_{L^{2}}$ and
$d_{L^{2}}(\sigma,\sigma^{\varepsilon})<\varepsilon\cdot\sqrt{\mu_{g_{0}}(M)}$.
This shows that $\mathcal{E}_{L^{2}}$ is dense in $\bar{\mathcal{E}}_{L^{2}}$.
Now we pass to continuous sections. We fix $\sigma\in\mathcal{E}_{L^{2}}$ and
$\varepsilon>0$. By Lusin’s Theorem we find a subset
$W_{i,\sigma}^{\varepsilon}$ of $V_{i}$ such that $\mu_{g_{0}}(V_{i}\setminus
W_{i,\sigma}^{\varepsilon})<\varepsilon$ and $\sigma_{i}$ restricted to
$W_{i,\sigma}^{\varepsilon}$ is continuous. Since $P(n)$ is a convex subset of
the vector space $S(n)$ of all $n\times n$ symmetric matrices we can apply
Theorem 4.1 of [Dug51] to find a continuous map $\sigma_{i}^{\varepsilon}$
defined on $V_{i}$ such that its restriction to $W_{i,\sigma}^{\varepsilon}$
coincides with $\sigma_{i}$ and with image contained in the convex hull of the
image of $\sigma_{i}$, in particular it is contained in $P(n)$. In order to
glue the pieces together we fix a partition of unity
$\\{\rho_{i}\\}_{i=1,\ldots,k}$ such that $\rho_{i}\equiv 1$ on
$W_{i,\sigma}^{\varepsilon}$ and $\rho_{i}\equiv 0$ outside $V_{i}$. We set
$\sigma^{\varepsilon}(p)=\sum_{i=1}^{k}\rho_{i}(p)\cdot\sigma_{i}^{\varepsilon}(p)$.
By construction $\sigma^{\varepsilon}\in\mathcal{E}_{C^{0}}$ and
$d_{L^{2}}(\sigma^{\varepsilon},\sigma)\to 0$ when $\varepsilon$ goes to $0$.
Finally we check that smooth sections are dense in the continuous ones. Use
Nash’s Theorem to find a Riemannian isometric embedding of $(P(n),d_{P(n)})$
into some Euclidean space. The continuous sections $\sigma_{i}^{\varepsilon}$
found above can be approximated by using standard convolution techniques and
tubular neighbourhood projections. This, together with partitions of unity as
above, allow us to define global smooth sections approximating continuous
ones.
∎

Arguing as at the beginning of the previous proof and using the fact that
$L^{2}$-sections are defined almost everywhere we can simplify the description
of the space $\bar{\mathcal{E}}_{L^{2}}$ as follows.

######  Theorem 3.

The metric space $(\bar{\mathcal{E}}_{L^{2}},d_{L^{2}})$ is isometric to
$L^{2}(M,\bar{P}(n))$.

We recall the definition of $L^{2}(M,\bar{P}(n))$ and we refer to [KS93], §1.1
for more details. A function $f\colon M\to\bar{P}(n)$ is said to be square
integrable if $\int_{M}d_{P(n)}^{2}(f(p),A)d\mu_{g_{0}}(p)<\infty$ for some
(hence any) $A\in\bar{P}(n)$. As usual we define the equivalence relation
$f\sim g$ if $f(p)=g(p)$ for $\mu_{g_{0}}$-a.e.$(p)$ and we denote by
$L^{2}(M,\bar{P}(n))$ the quotient of the set of square integrable functions
modulo this equivalence relation. The formula
$d_{L^{2}}(f,g):=\int_{M}d_{P(n)}^{2}(f(p),g(p))d\mu_{g_{0}}(p)$ defines a
distance on $L^{2}(M,\bar{P}(n))$. This is the distance we consider on
$L^{2}(M,\bar{P}(n))$ in the statement of Theorem 3.

###### Proof of Theorem 3.

We fix a triangulation of $M$ and we denote by $V_{1},\ldots,V_{k}$ the
interior of the maximal dimensional simplices. We can suppose that each
$V_{i}$ is the support of a trivializing chart for the bundle $\bar{E}$
sending $\mu_{g_{0}}$ to the Euclidean volume form. In other words we have the
following diagram

${\pi^{-1}(V_{i})}$${V_{i}\times\bar{P}(n)}$${\bar{P}(n)}$${V_{i}}$$\scriptstyle{\varphi_{i}}$$\scriptstyle{\pi}$$\scriptstyle{\pi_{2}}$

where $\pi$ is the bundle map and $\pi_{2}$ is the projection on the second
factor of the product. Let $\sigma\in\bar{\mathcal{E}}_{L^{2}}$ and
$i\in\\{1,\ldots,k\\}$. On $V_{i}$ we use the trivialization above to define
$f_{\sigma,i}\colon V_{i}\to\bar{P}(n)=\pi_{2}\circ\varphi_{i}\circ\sigma$.
Now define $f_{\sigma}\colon M\to\bar{P}(n)$ as
$f_{\sigma}(p)=f_{\sigma,i}(p)$ if $p\in V_{i}$ and
$f_{\sigma}(p)=\text{Id}\in P(n)$ if $p\notin\bigcup_{i}V_{i}$. The choice of
the value of $\Phi$ outside $\bigcup_{i}V_{i}$ does not matter because this
set has measure $0$. By definitions of the distances on the spaces the map
$\Phi\colon\bar{\mathcal{E}}_{L^{2}}\to L^{2}(M,\bar{P}(n))$, $\sigma\mapsto
f_{\sigma}$ is an isometry. ∎

The immediate consequence is the

###### Proof of Theorem 1.

The space $L^{2}(M,\bar{P}(n))$ does not depend on $M$. Indeed the measure
space $(M,\mathcal{B},\mu_{g_{0}})$, where $\mathcal{B}$ is the Borel
$\sigma$-algebra of $M$, is a standard probability space, i.e. it is
isomorphic as measure space to the interval $[0,1]$ with the standard Lebesgue
measure (cp. [dLR93], Theorem 4.3). It is easy to see that the definition of
the space $L^{2}(M,\bar{P}(n))$ depends only on the measure space class of
$M$, so it is isometric to $L^{2}([0,1],\bar{P}(n))$ and in particular it
depends only on $n$. ∎

Finally we have the shorter proof of Clarke’s Theorem, as promised.

######  Theorem 4.

The space $(\bar{\mathcal{E}}_{L^{2}},d_{L^{2}})$ is CAT$(0)$ and complete. In
particular it is the completion of $(\mathcal{E}_{C^{\infty}},d_{L^{2}})$.

###### Proof.

By Theorem 3 we know that $\bar{\mathcal{E}}_{L^{2}}$ is isometric to
$L^{2}(M,\bar{P}(n))$. Therefore it is enough to show that the completion
$(\bar{P}(n),d_{P(n)})$ of $(P(n),d_{P(n)})$ is CAT$(0)$ by [KS93], page 615
or [GN21], §4.1.1. There is another symmetric, complete, nonpositively curved
metric $d_{0}$ on $P(n)$ induced by the Riemannian product

$\langle A,B\rangle_{H}=\text{tr}(H^{-1}AH^{-1}B),$

where $H\in P(n)$ and $A,B\in T_{H}P(n)$. In other words the space
$(P(n),d_{0})$ is CAT$(0)$. The metric $d_{P(n)}$ is obtained by a conformal
change of the metric $d_{0}$ with conformal factor
$H\mapsto\varphi(H)=\sqrt{\det H}$. We denote by $P_{1}(n)$ the subset of
$P(n)$ of matrices with determinant $1$: it is a convex subset (cp. [BH13],
Lemma II.10.52). Moreover the map $P_{1}(n)\times\mathbb{R}\to P(n)$,
$(H,t)\mapsto e^{\frac{t}{\sqrt{n}}}H$ is an isometry by [BH13], Proposition
II.10.53. If $H$ is seen as a couple $(H_{1},t)$ through this identification
then clearly $\varphi$ depends only on $t$:
$\varphi(H_{1},t)=e^{\frac{t\sqrt{n}}{2}}$. Therefore the metric $d_{P(n)}$
can be seen as the warped product metric
$d_{0}\times_{e^{\frac{t\sqrt{n}}{2}}}e^{\frac{t\sqrt{n}}{2}}dt$ on
$P_{1}(n)\times\mathbb{R}$. The metric space $(-\infty,+\infty)$ with the
metric $e^{\frac{t\sqrt{n}}{2}}dt$ is a ray, whose completion is the space
$[-\infty,+\infty)$ with the metric $e^{\frac{t\sqrt{n}}{2}}dt$ obtained
extending the function $e^{\frac{t\sqrt{n}}{2}}$ by sending $-\infty$ to $0$.
Therefore we can construct the new, complete warped product metric space
$(P_{1}(n)\times[-\infty,+\infty),d_{0}\times_{e^{\frac{t\sqrt{n}}{2}}}e^{\frac{t\sqrt{n}}{2}}dt)$.
The natural isometric embedding of
$(P(n)\times\mathbb{R},d_{0}\times_{e^{\frac{t\sqrt{n}}{2}}}e^{\frac{t\sqrt{n}}{2}}dt)$
inside the space above has dense image, so
$(P_{1}(n)\times[-\infty,+\infty),d_{0}\times_{e^{\frac{t\sqrt{n}}{2}}}e^{\frac{t\sqrt{n}}{2}}dt)$
is the completion $\bar{P}(n)$ of $(P(n),d_{P(n)})$ and it is CAT$(0)$ because
of [AB04], Theorem 1.1. ∎

The explicit description of the completion of the space of Riemannian metrics
with respect to the $L^{2}$-distance given in Theorem 3 as
$L^{2}(M,\bar{P}(n))$ tells us that its isometry group is larger than
expected. Indeed not only it contains the group of volume-preserving
diffeomorphisms of any compact, orientable, $n$-dimensional smooth manifold,
but also the much larger group of all automorphisms of the standard
probability space $[0,1]$ endowed with the Lebesgue measure. It could be
interesting to determine explicitly the whole isometry group of this space. An
additional question could be to find the description of the boundary at
infinity of $L^{2}(M,\bar{P}(n))$.

Acknowledgments. I thank A. Lytchak for introducing me to the problem studied
in this paper and for teaching me the general philosophy behind this kind of
questions.

## References

  * [AB04] S.B. Alexander and R.L. Bishop.  Curvature bounds for warped products of metric spaces.  Geometric & Functional Analysis GAFA, 14(6):1143–1181, 2004.
  * [BH13] M. Bridson and A. Haefliger.  Metric spaces of non-positive curvature, volume 319.  Springer Science & Business Media, 2013.
  * [Cla13a] B. Clarke.  The completion of the manifold of riemannian metrics.  Journal of Differential Geometry, 93(2):203–268, 2013.
  * [Cla13b] B. Clarke.  Geodesics, distance, and the cat (0) property for the manifold of riemannian metrics.  Mathematische Zeitschrift, 273(1):55–93, 2013.
  * [dLR93] T. de La Rue.  Espaces de lebesgue.  Séminaire de probabilités de Strasbourg, 27:15–21, 1993\.
  * [Dug51] J. Dugundji.  An extension of tietze’s theorem.  Pacific Journal of Mathematics, 1(3):353–367, 1951.
  * [Ebi70] D.G. Ebin.  The manifold of riemannian metrics, in: Global analysis, berkeley, calif., 1968.  In Proc. Sympos. Pure Math., volume 15, pages 11–40, 1970.
  * [FG89] D.S. Freed and D. Groisser.  The basic geometry of the manifold of riemannian metrics and of its quotient by the diffeomorphism group.  Michigan Mathematical Journal, 36(3):323–344, 1989.
  * [GMM91] O. Gil-Medrano and P.W. Michor.  The Riemannian manifold of all Riemannian metrics.  The Quarterly Journal of Mathematics, 42(1):183–202, 01 1991.
  * [GN21] N. Gigli and F. Nobili.  A differential perspective on gradient flows on cat(k)-spaces and applications.  The Journal of Geometric Analysis, 31(12):11780–11818, 2021.
  * [KS93] N. Korevaar and R. Schoen.  Sobolev spaces and harmonic maps for metric space targets.  Communications in Analysis and Geometry, 1(4):561–659, 1993.