# Bearing and Distance Formation Control of Rigid Bodies in $SE(3)$ with
Bearing and Distance Constraints

Sara Mansourinasab1, Mahdi Sojoodi2 and S. Reza Moghadasi3 1Sara Mansourinasab
is with the Department of Electrical and Computer Engineering, Tarbiat Modares
University, Tehran, Iran<EMAIL_ADDRESS>Sojoodi is with the
Department of Electrical and Computer Engineering, Tarbiat Modares University,
Tehran, Iran<EMAIL_ADDRESS>Reza Moghadasi is with the Department of
Mathematical Sciences, Sharif University of Technology, Tehran, Iran
<EMAIL_ADDRESS>
###### Abstract

Rigidity of the interaction graph is a fundamental condition for achieving the
desired formation which can be defined in terms of distance or bearing
constraints between agents. In this paper, for reaching a unique formation
with the same scaling and orientation as the target formation, both distance
and bearing constraints are considered for defining the desired formation.
Besides, both distance and bearing measurements are also available. Each agent
is able to gather the measurements with respect to other agents in its own
body frame. So, the agents are coordinated-free concerning a global reference
frame. On the other hand, the framework is embedded in $SE(3)$. The control
signal is designed based on a gradient descent method by introducing a cost
function. Firstly, the formation problem is considered for bearing-only
constraints in $SE(3)$ configuration. Then, the formation control is expressed
for the general case of both bearing and distance constraints. Furthermore,
the essential conditions that guarantee reaching the desired formation is
discussed. Finally, the validity of the proposed formation control is verified
by numerical simulations.

##  I Introduction

Formation control is an actively studied strategy in analyzing multi agents
systems [1, 2]. In formation control, the type of available data that agents
have access plays an important role in designing the control strategy. Based
on sensing capabilities for measuring the relative positions between agents,
formation control methods are categorized into three types of position-based,
displacement-based, and distance-based. The two former methods are based on
global distance measuring which necessitates the use of global positioning
system (GPS), for example. Then, each agent shares the information with other
agents through wireless communications [3]. Using GPS is not always reliable
since limitations such as high dependence of measurements accuracy on the
number of available satellites, environmental challenges like indoors, deep
urban canyons, dense vegetation and cloud cover obstruct line-of-sight to the
satellite [4]. In distance-based method, the distances between agents are
controlled to achieve the desired formation. In this method, the sensing
capability is only defined with respect to local coordinate systems or body
frame of agents and no GPS is required. Local coordinates formation control
procedures include distance-based and bearing-based methods depending on the
type of measuring and constraint parameters. Formation rigidity theory helps
to achieve the desired formation up to scaling, translation and coordinated
rotation factors by its inter-agent bearings and distances. According to the
definitions, in a given system connected by flexible linkages and hinges,
rigidity has addressed the amount of stiffness of the given framework to an
induced deformation [5]. In a rigid formation, converging to the desired
formation can be achieved by distance and/or bearing measurements. Besides,
the desired formation can be defined based on distance and/or bearing
constraints between any pair of agents. These types of controllers are
classified as distance/bearing-based controller. In distance-based formation
control, each agent has its own body frame which does not need to be aligned
with other agents body frame orientation.

Distance-based rigidity problem has been studied in many investigations [6, 7,
8]. In most of them, the target formation constraints are only defined as
inter-agent distances. In bearing-based theory, the desired formation
constraints are defined in terms of the direction of the neighboring agents.
In other words, the direction of the unit vector aligned the edge that
connecting two agents is considered as inter-agent bearing between those two
agents. [9, 10] have extensively studied different multi-agent strategies
based on bearing rigidity theory for converging to the desired formation,
generally assume each agent is able to measure bearings and distances of the
nearest neighbors. Approaches with only bearing measurements capability are
proposed as bearing-only formation control problems. Vision-based devices are
suitable for bearing measurements. Optical cameras are one of low-cost and
light-weight onboard sensors that are bearing-only sensors. This yields the
resulting formation to be with a different scale of the desired formation.

In this paper, formation control for a multi-rigid body system is proposed
considering bearing measurements and constraints. Since the motion behavior of
a rigid body is a combination of translation and rotation movements, the
notion of bearing rigidity comes into play for synchronized target formations.
As an illustrative example, multi-satellite systems that are orbiting around
the Earth, should have a specific orientation to cover a specific region on
Earth. As a result, in multi-rigid body problems, using bearing rigidity
concept is beneficial, more accurate and low-cost. Geometrically speaking,
roto-translational motion of a rigid body represents an $SE(3)$ state space
configuration which is a Lie group. Using bearing measurements for analyzing
such systems is compatible with geometry of the problem [11].

In bearing rigidity problems, the only bearing preserving motions subtend
translation and scaling of the entire system [12]. However, distance
preserving motions in distance-based problems include roto-translational
motions. It is clear that the rotation of the resulting formation in bearing-
based constraints problem will change the inter-agent bearings. On the other
hand, the configuration space of the problem is embedded in Special Euclidean
space $SE(3)$. Considering the geometry and the nature of the $SE(3)$ manifold
enables the rotation of the entire framework. This result in a synchronous
rotation of all agents around their body attached framework with the same
angular velocity that is coupled with the entire graph[5].
It can be concluded that in analysing bearing rigidity problem on
$\mathbb{R}^{3}$, the only admissible infinitesimally rigid motions include
translation and scaling of the framework. However, for analysing the problem
on $SE(3)$, the only admissible infinitesimally rigid motions include
translation, scaling and coordinate rotation. besides, in distance preserving
problems, the only admissible infinitesimally rigid motions include
translation and coordinate rotation of the framework. Due to this reason, in
bearing-only formation problem on $\mathbb{R}^{3}$, the graph is able to
converge to the target formation but with a different scale. Furthermore, in
distance-based problems, the final graph may be a rotated form of the target
graph. Since in some problems which the final formation should be exactly the
same as the desired formation, the combination of bearing and distance
rigidity is applied. As a result, in this paper, the formation control problem
on $SE(3)$ is considered in two cases of bearing constraints and bearing-
distance constraints in order that the framework uniquely reach to the desired
formation.
Since implementing bearing measurements in local frame is compatible with more
realistic and applicable scenarios, [13] have addressed the position and
orientation of a graph of rigid bodies with $SE(2)$ architecture, which
restricts the results to the plane. Another local body frame scheme is found
in [2] with 3 dimensional extension. However, the agents attitude rotation is
limited to just one axis. The aforementioned problem is eliminated in [5] by
considering fully-actuated multi-agents formations with six controllable
degrees of freedom.
The bearing rigidity problem has been mostly studied in 2-dimensional spaces.
The extension of this problem to 3D is formulated in [12]. The relation and
comparison between bearing and distance rigidity is described in detail. This
paper defines the bearing based formation problem by bearing-only measurements
and provides the bearing-only control in order to reach formation. Besides, it
demonstrates under what conditions a framework can be achieved by bearing
measurements with translation and scaling factors. However, this paper is
confined to the Euclidean $\mathbb{R}^{3}$ space. [14] is a comprehensive
reference aims at expressing bearing rigidity theory concept of a framework in
$\mathbb{R}^{n}$ spaces. It redefines the concept of bearing rigidity to the
more complicated manifold spaces than n-dimensional Euclidean space. Although
the paper demonstrates the main notions of bearing rigidity theory in $SO(n)$
and $SE(n)$ which has been cited by the current paper, it is deprived of any
controlling interpretations to steer the framework to the target formation.
Therefore, in the presented paper, referring to the results of [14], we
present the required control strategy to convey the agents to the desired
formation in $SE(3)$ space. This strategy is investigated on two cases of
bearing-only and mixed bearing-distance desired constraints.

The paper is organized as follows. Section II is dedicated to review the
required elements of graph and bearing rigidity theory. The initial problem
formulation for rigid body agents is defined in the second part of this
section. In section III, the desired formation is considered as bearing-only
constraints and the control law is designed based on relative bearing
measurements in local body frame to achieve the desired formation. In section
IV, the formation is introduced as bearing-distance measurements in order to
provide the exact shape as the desired one with no scale and rotation. In
section V, the results are supported with computer simulations.

##  II Preliminaries and Problem Formulation

###  II-A Graph Theory

In this section, some elements of graph theory and bearing rigidity theory are
briefly reviewed. A graph of $n$ agents with $m$ edges is defined as the pair
$\mathcal{G}=(\mathcal{V},\mathcal{E})$ while
$\mathcal{V}=\\{v_{1},...,v_{n}\\}$ is the vertex set and
$\mathcal{E}=\\{e_{1},...,e_{m}\\}\subseteq\mathcal{V}\times\mathcal{V}$ is
the edge set. An edge $e_{k}=(v_{i},v_{j})\in\mathcal{E}$, while an
information exchange such as distance measurements or bearing measurements
exists between two vertices $(i,j)$, and $k$ relates to the $k$-th directed
edge in the graph. Each vertex $v_{i}\in\mathcal{V}$ in the graph is
associated to the point $p_{i}\in\mathbb{R}^{3}$ in the configuration, so the
framework $(\mathcal{G},p)$ in $\mathbb{R}^{3}$ consists of the graph
$\mathcal{G}=(\mathcal{V},\mathcal{E})$ and the configuration
$p=[p_{1}^{T}...p_{n}^{T}]^{T}\in\mathbb{R}^{3n}$. The neighborhood of the
vertex $i$ is denoted by
$\mathcal{N}_{i}=\\{j\in\mathcal{V}|(i,j)\in\mathcal{E}\\}$. In a directed
framework $\mathcal{G}=(\mathcal{V},\mathcal{E})$, the incidence matrix
$E\in\mathbb{R}^{n\times m}$ is defined as

$\displaystyle[E]_{ik}=\left\\{\begin{array}[]{rl}-1&\text{if}\
e_{k}=(v_{i},v_{j})\in\mathcal{E}\ (\text{outgoing edge})\\\ 1&\text{if}\
e_{k}=(v_{j},v_{i})\in\mathcal{E}\ (\text{ingoing edge})\\\
0&\text{otherwise}\end{array}\right.$ (4)

and the matrix $E_{o}\in\mathbb{R}^{n\times m}$ is introduced by

$\displaystyle[E_{o}]_{ik}=\left\\{\begin{array}[]{rl}-1&\text{if}\
e_{k}=(v_{i},v_{j})\in\mathcal{E}\ (\text{outgoing edge})\\\
0&\text{otherwise}\end{array}\right.$ (7)

while $\bar{E}=E\otimes I_{3}\in\mathbb{R}^{3n\times 3m}$,
$\bar{E}_{o}=E_{o}\otimes I_{3}\in\mathbb{R}^{3n\times 3m}$, $I_{3}$ is the
3-dimensional identity matrix and $\otimes$ is the Kronecker product. The
orthogonal projection operator $P:\mathbb{R}^{3}\to\mathbb{R}^{3\times 3}$ is
defined as

$\displaystyle P(v)=I_{3}-\frac{v}{\|v\|}\frac{v^{T}}{\|v\|}.$ (8)

This operator projects any non-zero vector $v\in\mathbb{R}^{3}$ to its
orthogonal complement. $\nabla_{x}V$ denotes the partial differentiation of
$V$ with respect to $x$. For each agent $v_{i}\in\mathcal{V}$, the state
$R_{i}\in SO(3)$ is devoted such that the matrix group
$SO(3)=\\{R\in\mathbb{R}^{3\times 3}:R^{T}R=I_{3\times 3},det(R)=1\\}$ is the
space of 3D rotations, where $I_{3\times 3}$ indicates the three dimensional
identity matrix. The group identity indicated by $e$, equals the identity
matrix $I_{3\times 3}$. The tangent space of $SO(3)$ in the identity element
of the group, is the Lie algebra of $SO(3)$ written as
$\mathfrak{so}(3)=T_{e}SO(3)=\\{\Omega\in\mathbb{R}^{3\times
3}:\Omega^{T}=-\Omega\\}$, which is the space of all $3\times 3$ skew
symmetric matrices. The tangent space of $SO(3)$ at any group member $R$ is
$T_{R}SO(3)=\\{RV:V\in\mathfrak{so}(3)\\}$. There is a map between the tangent
vector in Lie group $\Omega=\hat{\omega}\in\mathfrak{so}(3)$ with a vector in
$\omega\in\mathbb{R}^{3}$ using the following $hat(\hat{.})$ and
$vee(\check{.})$ maps

$\displaystyle\omega=\left(\begin{array}[]{c}\omega_{1}\cr\omega_{2}\cr\omega_{3}\end{array}\right)\in\mathbb{R}^{3}\stackrel{{\scriptstyle[}}{{(}}\check{.})]{(\hat{.})}{\rightleftharpoons}\hat{\omega}=\left(\begin{array}[]{ccc}0&-\omega_{3}&\omega_{2}\cr\omega_{3}&0&-\omega_{1}\cr-\omega_{2}&\omega_{1}&0\end{array}\right)\in\mathfrak{so}(3)$
(15)

###  II-B Problem Formulation

Consider a team of $n$ agents in $SE(3)$. The kinematics of the agent $i$,
$i\in\\{1\cdots,n\\}$ is expressed as

$\displaystyle\left\\{\begin{array}[]{ccc}\dot{R}_{i}&=&R_{i}\hat{\omega}_{i}\\\
\dot{p}_{i}&=&R_{i}v_{i}\end{array}\right.$ (18)

where $p_{i}=[p_{xi},p_{yi},p_{zi}]^{T}\in\mathbb{R}^{3}$ is the position of
the $i$-th agent and $R_{i}\in SO(3)$ is the rotation matrix associated with
the orientation of it. Furthermore, $v_{i},\omega_{i}\in\mathbb{R}^{3}$ are
respectively the linear and angular velocity of agent $i$ with respect to the
body frame. For notation convenience, we use the position and attitude of the
complete configuration in the stacked form as
$\chi(i)=(\chi_{p}(i),\chi_{r}(i))=(p_{i},R_{i})\in SE(3)$.

The relative bearing measurement between agents $i$ and $j$ associated to the
edge $e_{k}=(v_{i},v_{j})$ is defined as

$\displaystyle
b_{k}=b_{ij}=R_{i}^{T}\frac{p_{i}-p_{j}}{\|p_{i}-p_{j}\|}=R_{i}^{T}\bar{p}_{ij}$
(19)

and

$\displaystyle\bar{p}_{ij}=d_{ij}p_{ij},\ \ p_{ij}=p_{i}-p_{j},\ \
d_{ij}=\frac{1}{\|p_{i}-p_{j}\|}.$

Definition 1. The bearing rigidity function for a framework
$(\mathcal{G},\chi)$ of n-agent is expressed as the map

$\displaystyle b_{\mathcal{G}}:SE(3)^{n}\to\mathbb{S}^{2m}$ $\displaystyle
b_{\mathcal{G}}(\chi)=[b_{1}^{T}\dots b_{m}^{T}]^{T}$ (20)

that can be written in the following compact form

$\displaystyle
b_{\mathcal{G}}(\chi)=\text{diag}(d_{ij}R_{i}^{T})\bar{E}^{T}p.$ (21)

The bearing rigidity matrix in $SE(3)$ is defined as the gradient of the
rigidity function as
$\text{B}_{\mathcal{G}}(\chi)=\nabla_{\chi}b_{\mathcal{G}}(\chi)\in\mathbb{R}^{3m\times
6n}$ that can be expressed as

$\displaystyle\text{B}_{\mathcal{G}}(\chi)$
$\displaystyle=\left[\frac{\partial b_{ij}}{\partial p_{i}}\quad\frac{\partial
b_{ij}}{\partial R_{i}}\right]$
$\displaystyle=\left[-\text{diag}(d_{ij}R_{i}^{T}P(\bar{p}_{ij}))\bar{E}^{T}\quad-\text{diag}(R_{i}^{T}\widehat{\bar{p}_{ij}})\bar{E}_{o}^{T}\right]$
(22)

such that $\widehat{\bar{p}_{ij}}$ is defined as (15).

##  III Bearing Formation Control in SE(3) with bearing-only constraints

In this section, the formation control problem in $SE(3)$ with no common
reference frame is studied. The desired formation is proposed as bearing only
constraints. A network of n-agent is considered with dynamics presented in
(18). The control strategy to transmit the agents to the desired formation is
based on the gradient field of a potential function $\phi$. This function is
defined to describe the bearing and/or distance constraints. For converging to
the desired formation that is addressed in this section, the potential
function is only introduced in terms of the bearing only constraints as

$\displaystyle\phi(\chi)=\frac{1}{2}\|b_{\mathcal{G}}-b_{\mathcal{G}}^{*}\|^{2}$
(23)

while $b_{\mathcal{G}}^{*}$ is the desired bearing rigidity function. The
control input is considered as the negated gradient of this potential function
as

$\displaystyle\binom{v_{i}}{\omega_{i}}=-k\nabla_{\chi}\phi$ (24)

while

$\displaystyle\nabla_{\chi}\phi=\binom{\nabla_{p_{i}}\phi}{\nabla_{R_{i}}\phi}.$

Whereas

$\displaystyle\nabla_{\chi}\phi$
$\displaystyle=\nabla_{\chi}b_{\mathcal{G}}^{T}\frac{\partial\phi}{\partial
b_{\mathcal{G}}}$
$\displaystyle=\text{B}_{\mathcal{G}}^{T}(\chi)\frac{b_{\mathcal{G}}-b_{\mathcal{G}}^{*}}{\|b_{\mathcal{G}}-b_{\mathcal{G}}^{*}\|}=-\text{B}_{\mathcal{G}}^{T}(\chi)\frac{b_{\mathcal{G}}^{*}}{\|b_{\mathcal{G}}-b_{\mathcal{G}}^{*}\|}$
$\displaystyle=\left[\text{diag}(d_{ij}R_{i}^{T}P(\bar{p}_{ij}))\bar{E}^{T}\quad\text{diag}(R_{i}^{T}\widehat{\bar{p}_{ij}})\bar{E}_{o}^{T}\right]^{T}$
$\displaystyle\ \ \ \ \
\frac{b_{\mathcal{G}}^{*}}{\|b_{\mathcal{G}}-b_{\mathcal{G}}^{*}\|}.$

As a result, the local body frame control signal is expressed as

$\displaystyle\left\\{\begin{array}[]{ccc}v_{i}&=\sum_{(i,j)\in\mathcal{E}}-d_{ij}R_{i}^{T}P(\bar{p}_{ij})b_{ij}^{*}\\\
\omega_{i}&=\sum_{(i,j)\in\mathcal{E}}-R_{i}^{T}\widehat{\bar{p}_{ij}}b_{ij}^{*}\\\
\end{array}\right.$ (27)

In the same way as [12], the following theorem is used for guaranteeing the
stability of the formation.

######  Lemma 1.

The control law (27) keeps the centroid $\bar{p}$ and the scale $s$ invariant

###### Proof.

Due to lack of space the proof is omitted. Because of the same procedure, for
more information, please refer to [12]. ∎

(a) Behavior of agents from initial value to the final formation (b) Error
signal norm between agents bearing and the desired bearing

Figure 1: Bearing-only formation and error for a network of 8 agents

##  IV Bearing-distance based Formation Control in $SE(3)$

###  IV-A Bearing-Distance Rigidity Theory in $SE(3)$

In this section, the desired formation is considered in terms of both distance
and bearing constraints. Besides, the control strategy requires mixed distance
and bearing information.
Again, consider a network of $n$ rigid bodies as a multi-agent system modeled
as a framework $(\mathcal{G},\chi)$ with kinematics defined as (18). We have
the following definitions.
Definition 2: Suppose the edge set $\mathcal{E}$ in the graph
$\mathcal{G}(\mathcal{V},\mathcal{E})$ consists of two edge sets of
$\mathcal{E}_{B}$ and $\mathcal{E}_{D}$ which are bearing and distance
measurements respectively, such that
$\mathcal{E}_{B}\cup\mathcal{E}_{D}=\mathcal{E}$, while none of
$\mathcal{E}_{D}$ and $\mathcal{E}_{B}$ are not zero. So the network graph is
defined by $\mathcal{G}(\mathcal{V},\mathcal{E}_{B},\mathcal{E}_{D})$. In the
case that $(i,j)\in\mathcal{E}_{B}$ and $(i,j)\in\mathcal{E}_{D}$, it means
there exists two edges between agents $i$ and $j$, namely, both bearing and
distance measurements are available between these agents. Besides, $m_{b}$
indicates the number of available bearing measurements between agents, and
$m_{d}$ is the number of available distance measurements between agents.
Definition 3: Rigidity Function equals the map $F_{\mathcal{G}}$ defined as

$\displaystyle F_{\mathcal{G}}:\mathbb{R}^{3n}$
$\displaystyle\to\mathbb{R}^{(m_{d}+m_{b})}$ $\displaystyle\chi$
$\displaystyle\mapsto F_{\mathcal{G}}(\chi)$

while
$\chi=(p,R)=\\{(p_{1},R_{1}),\dots(p_{n},R_{n})\\}\in(SE(3))^{n}$
$p=[p_{1}^{T}\dots p_{n}^{T}]^{T}\in\mathbb{R}^{3n}$
$R=[\ R_{1}^{T}\dots R_{n}^{T}]^{T}\in(SO(3))^{n}$.
Inspiring by [15], the vector $F_{\mathcal{G}}$ dedicates all the available
bearing and distance measurements on stack form as

$\displaystyle F_{\mathcal{G}}(\chi)=\left[\begin{array}[]{c}F_{1}\\\
\vdots\\\ F_{m_{d}}\\\ \hdashline[2pt/2pt]F_{m_{d+1}}\\\ \vdots\\\
F_{m_{d}+m_{b}}\end{array}\right]\triangleq\left[\begin{array}[]{c}\tfrac{1}{2}z_{1}^{2}\\\
\vdots\\\ \tfrac{1}{2}z_{m_{d}}^{2}\\\ \hdashline[2pt/2pt]b_{1}\\\ \vdots\\\
b_{m_{b}}\end{array}\right]=\left[\begin{array}[]{c}{\color[rgb]{1,1,1}z}\\\
d_{\mathcal{G}}\\\ {\color[rgb]{1,1,1}z}\\\ \hdashline[2pt/2pt]\\\
b_{\mathcal{G}}\\\ \\\ \end{array}\right]$ (46)

with the following parameter definitions

$\displaystyle b_{k}$
$\displaystyle=b_{ij}=R_{i}^{T}\frac{p_{i}-p_{j}}{\|p_{i}-p_{j}\|}=R_{i}^{T}\bar{p}_{ij}$
$\displaystyle e_{k}$ $\displaystyle=p_{i}-p_{j}$ $\displaystyle z_{k}$
$\displaystyle=z_{ij}=\|e_{k}\|=\|p_{i}-p_{j}\|,k=1,...,m_{d}$ $\displaystyle
d_{\mathcal{G}}$
$\displaystyle=[\tfrac{1}{2}z_{1}^{2}\dots\tfrac{1}{2}z_{m_{d}}^{2}]^{T}\in\mathbb{R}^{m_{d}}$
$\displaystyle b_{\mathcal{G}}$ $\displaystyle=[b_{1}\dots
b_{m_{b}}]^{T}\in\mathbb{R}^{3m_{b}}.$

To clarify, all parameters with $\mathcal{G}$ indices indicate that parmeter
in stack form which related to the whole graph.
As a result, the $SE(3)$-Bearing Rigidity Matrix
$R_{\mathcal{G}}(\chi)\in\mathbb{R}^{(m_{d}+3m_{b})\times 6n}$ for the
framework $(\mathcal{G},\chi)$ is defined as follows

$\displaystyle R_{\mathcal{G}}(\chi)=\nabla_{\chi}F_{\mathcal{G}(\chi)}$
$\displaystyle=\left[\begin{array}[]{ccc;{2pt/2pt}ccc}\frac{\partial
F_{1}}{\partial p_{1}}&\ldots&\frac{\partial F_{1}}{\partial
p_{n}}&\frac{\partial F_{1}}{\partial R_{1}}&\ldots&\frac{\partial
F_{1}}{\partial R_{1}}\\\ \vdots&\ddots&\vdots&\vdots&\ddots&\vdots\\\
\frac{\partial F_{m_{d}}}{\partial p_{1}}&\ldots&\frac{\partial
F_{m_{d}}}{\partial p_{n}}&\frac{\partial F_{m_{d}}}{\partial
R_{1}}&\ldots&\frac{\partial F_{m_{d}}}{\partial R_{n}}\\\
\hdashline[2pt/2pt]\frac{\partial F_{m_{d+1}}}{\partial
p_{1}}&\ldots&\frac{\partial F_{m_{d+1}}}{\partial p_{n}}&\frac{\partial
F_{m_{d+1}}}{\partial R_{1}}&\ldots&\frac{\partial F_{m_{d+1}}}{\partial
R_{n}}\\\ \vdots&\ddots&\vdots&\vdots&\ddots&\vdots\\\ \frac{\partial
F_{m_{d}+m_{b}}}{\partial p_{1}}&\ldots&\frac{\partial
F_{m_{d}+m_{b}}}{\partial p_{n}}&\frac{\partial F_{m_{d}+m_{b}}}{\partial
R_{1}}&\ldots&\frac{\partial F_{m_{d}+m_{b}}}{\partial R_{n}}\\\
\end{array}\right]$ (53)
$\displaystyle=\left[\begin{array}[]{c;{2pt/2pt}c}D_{\mathcal{G}}&0\\\
\hdashline[2pt/2pt]G_{\mathcal{G}}&K_\mathcal{G}\end{array}\right].$ (56)

Figure 2: Behavior of agents from initial value to the final
formation

Figure 3: Formation for the case with 3-bearing and 4-distance constraints

It can be observed that the rigidity matrix (56) equals the Jacobian of the
$SE(3)$-rigidity function. It has been indicated that the null-space of the
$SE(3)$-rigidity matrix $R_{\mathcal{G}}(\chi)$ specifies the set of
infinitesimal motions of the framework [5]. The above rigidity matrix is
interpreted as the bearing part as $B_{\mathcal{G}}=[G_{\mathcal{G}}\
K_{\mathcal{G}}]$ and distance part $D_{\mathcal{G}}$. Calculating all
parameters of the rigidity matrix $R_{\mathcal{G}}$ yields

$\displaystyle D_{\mathcal{G}}$ $\displaystyle=J(e)\bar{E};$ (57)
$\displaystyle G_{\mathcal{G}}$
$\displaystyle=-diag(d_{ij}R_{i}^{T}P(\bar{p}_{ij}))\bar{E}^{T}$ (58)
$\displaystyle K_{\mathcal{G}}$
$\displaystyle=-diag(R_{i}^{T}\hat{\bar{p}}_{ij})\bar{E}_{o}^{T}$ (59)

while

$\displaystyle J(e)=diag(e_{1}^{T},...,e_{m_{d}}^{T})$

and

$\displaystyle R_{\mathcal{G}}=\left[\begin{array}[]{c;{2pt/2pt}c}\\\
J(e)\bar{E}&0\\\ \\\ \hdashline[2pt/2pt]\\\
-diag(d_{ij}R_{i}^{T}P(\bar{p}_{ij}))\bar{E}^{T}&-diag(R_i^T
\hat{\bar{p}}_{ij}) \bar{E}_o^T\end{array}\right].$ (65)

###  IV-B Formation Control in $SE(3)$

The formation control problem in SE(3) is formulated in this section to steer
a network of $n$ rigid bodies with kinematics equation as (18) to a desired
formation consists of bearing and distance constraints. The cost function is
introduced based on both bearing and distance constraints as follows

$\displaystyle\phi(\chi)=\frac{1}{2}\|b_{\mathcal{G}}-b_{\mathcal{G}}^{*}\|^{2}+\frac{1}{2}\|d_{\mathcal{G}}-d_{\mathcal{G}}^{*}\|^{2}$
(66)

while $b_{\mathcal{G}}^{*}$ is the desired bearing rigidity function and
$d_{\mathcal{G}}^{*}$ is the desired distance function. The control input is
similarly expressed as (24). As a result, the gradient of the cost function
$\phi$ can be computed using the chain rule as

$\displaystyle\left[\begin{array}[]{c}v_{i}\\\
\omega_{i}\end{array}\right]=-\nabla_{\chi}\phi=\left[\begin{array}[]{c}-\nabla_{p_{i}}\phi\\\
-\nabla_{R_{i}}\phi\end{array}\right]$ (71)

such that

$\displaystyle\nabla_{p_{i}}\phi$
$\displaystyle=\nabla_{p_{i}}b_{\mathcal{G}}(b_{\mathcal{G}}-b_{\mathcal{G}}^{*})+\nabla_{p_{i}}d_{\mathcal{G}}(d_{\mathcal{G}}-d_{\mathcal{G}}^{*})$
$\displaystyle=G_{\mathcal{G}}^{T}(b_{\mathcal{G}}-b_{\mathcal{G}}^{*})+D_{\mathcal{G}}^{T}(d_{\mathcal{G}}-d_{\mathcal{G}}^{*})$
$\displaystyle=-(diag(d_{ij}R_{i}^{T}P(\bar{p}_{ij}))\bar{E}^{T})^{T}(b_{\mathcal{G}}-b_{\mathcal{G}}^{*})$
$\displaystyle\ \ \ \
+\bar{E}^{T}J(e)^{T}(d_{\mathcal{G}}-d_{\mathcal{G}}^{*})$
$\displaystyle=(diag(d_{ij}R_{i}^{T}P(\bar{p}_{ij}))\bar{E}^{T})^{T}b_{\mathcal{G}}^{*}+\bar{E}^{T}J(e)^{T}(d_{\mathcal{G}}-d_{\mathcal{G}}^{*})$
(72) $\displaystyle\nabla_{R_{i}}\phi$
$\displaystyle=\nabla_{R_{i}}b_{\mathcal{G}}(b_{\mathcal{G}}-b_{\mathcal{G}}^{*})=K_{\mathcal{G}}^{T}(b_{\mathcal{G}}-_{\mathcal{G}}^{*})$
$\displaystyle=-\bar{E}_{o}(diag(R_{i}^{T}\hat{\bar{p}}_{ij}))^{T}(b_{\mathcal{G}}-b_{\mathcal{G}}^{*})$
$\displaystyle=\bar{E}_{o}(diag(R_{i}^{T}\hat{\bar{p}}_{ij}))^{T}b_{\mathcal{G}}^{*}.$
(73)

It should be noted that $P(\bar{p}_{ij})R_{i}b_{g}=0$. Therefore, the
resulting control law navigate the agent to the desired formation is given by
(71) as

$\displaystyle\left[\begin{array}[]{c}v_{i}\\\
\omega_{i}\end{array}\right]=\left[\begin{array}[]{c}\\!\\!\\!\\!\\!\\!-(diag(d_{ij}R_{i}^{T}P(\bar{p}_{ij}))\bar{E}^{T})^{T}b_{\mathcal{G}}^{*}\\!+\bar{E}^{T}J(e)^{T}(d_{\mathcal{G}}-d_{\mathcal{G}}^{*})\\!\\!\\!\\!\\!\\!\\\
\vspace*{-0.25cm}\hfil\\\
-\bar{E}_{o}(diag(R_{i}^{T}\hat{\bar{p}}_{ij}))^{T}b_{\mathcal{G}}^{*}\end{array}\right].$
(79)

######  Theorem 1.

Using the control input (79), the centroid $\bar{p}$ and the scale $s$ are
invariant. while

$\displaystyle\bar{p}=1/n\sum_{1=1}^{n}p_{i}\ \ \ ,\ \ \
s=\sqrt{\frac{1}{n}\sum_{i=1}^{n}\|p_{i}-\bar{p}\|^{2}}$ (80)

This theorem represents the target formation and the initial formation have
the same scale and centroid.

###### Proof.

Centroid: It should be concluded that $\bar{p}^{*}=\bar{p}(0)$.

$\displaystyle\dot{\bar{p}}=$ $\displaystyle\frac{1}{n}(1_{n}^{T}\otimes
I_{3})diag(d_{ij}R_{i}^{T}P(\bar{p}_{ij}))\bar{E}^{T}b_{g}^{*}$
$\displaystyle+\frac{1}{n}(1_{n}^{T}\otimes
I_{3})J(e)\bar{E}(d_{\mathcal{G}}-d_{\mathcal{G}}^{*}).$

For the first term, as it is indicated in [14] that $(1_{n}^{T}\otimes
I_{3})\in\mathcal{S}_{t}=ker(G_{g})$, so $(1_{n}^{T}\otimes
I_{3})(R_{i}^{T}P(\bar{p}_{ij}))^{T}b_{g}^{*}=0$. For the second term, in
distance rigidity problem with rigidity matrix $D_{\mathcal{G}}$ using the
fact that $D_{\mathcal{G}}(I_{n}\otimes I_{3})=0$ results in $(I_{n}\otimes
I_{3})J(e)\bar{E}=0$. So, $\dot{\bar{p}}=0$ yields $\bar{p}^{*}=\bar{p}(0)$.
Scaling: Since (80) can be written as the centroid term such that
$s=\|p-(1_{n}\otimes\bar{p})\|/\sqrt{n}$, so

$\displaystyle\dot{s}=\frac{1}{\sqrt{n}}\frac{(p-(1_{n}\otimes\bar{p}))^{T}}{\|p-(1_{n}\otimes\bar{p})\|}\dot{p}.$

It is obtained from $\dot{p}\perp p$ that $p^{T}\dot{p}=0$, and from the
previous part $\dot{p}\perp(1_{n}\otimes\bar{p})$, thus $\dot{s}=0$ and
$s^{*}=s(0)$. ∎

Figure 4: Formation for the case 3-bearing and 3-distance
constraints

Figure 5: Formation for the case 5-bearing and 1-distance constraints

##  V simulation results

Numerical simulations of the proposed method is presented in this section to
verify the validity of the investigated control law algorithm. In the first
simulation, a network of 8 agents evolves under the bearing-based control law
(27). The next example illustrates the mixed bearing and distance final
constraints formation.
Example 1. Eight agents with bearing edges are considered as a network under
(18) dynamical equations that should be converged to a cubical desired
formation which is defined based on bearing-only constraints, namely with the
cost function considered as (23). Applying the control law (27) to this system
results in a fast convergence to the desired formation. Fig. (1) shows the
trajectories agents travel from initial to final orientation. Gray dots are
the indicator of the initial configuration of agents and gray lines are the
paths they pass until reach the final formation that is marked with blue
signs. Red signs depict the desired formation. It is obvious from the figure
that agents reach the target bearing formation but in different scaling. It
means bearing constraints are completely satisfied that is confirmed by Fig.
(1)-b showing the bearing error between the desired and final formation.

Example 2. Four agents are evolved under the control law (27). As it is
conceived from Fig. (3), the network converges to the target bearing
formation; however, as the control law is only based on bearing constraints,
the final formation is in a different scaling with the desired formation. So,
in this example, mixed bearing and distance constraints implemented to the
system which results in an exact final formation with the same scale as the
desired one. The control law (79) is applied to the system of four agents with
different desired mixed constraints. First of all, the target formation
includes 3 bearing and 4 distance constraints. As it is seen from Fig. (3),
bearing constraints are not totally satisfied. Fig. (5) shows the problem with
3 bearing and 3 distance constraints. The graph has not been succeeded in
achieving convergence to the desired formation. Finally, the combination of 5
bearing, 1 distance constraints leads to the exact formation as the desired
one. From Fig. (5), it is seen that the final formation is the same as the
desired one and both scaling and coordinated rotation properties of the
formation are strongly satisfied. Due to the lack of space, the minimum
quantities of the required bearing and distance constraints under which the
desired formation is obtained will be investigated in the next paper.

##  VI Conclusion

In this paper, bearing formation control problem is implemented in $SE(3)$
manifold. The control law is constructed based on cost function and gradient
descent method. Then the problem is extended to the mix distance and bearing
formation constraints, and conditions under which the problem has solution is
considered.

## References

  * [1] Kwang Kyo Oh, Myoung Chul Park, and Hyo Sung Ahn,  A survey of multi-agent formation control, Automatica, Vol. 53, pp. 424- 440, 2015.
  * [2] H. Liu, T. Ma, F. L. Lewis and Y. Wan, Robust Formation Control for Multiple Quadrotors With Nonlinearities and Disturbances, IEEE Transactions on Cybernetics, vol. 50, no. 4, pp. 1362-1371, April 2020.
  * [3] Florian Dörfler and Bruce Francis,  Formation control of autonomous robots based on cooperative behavior, in Proceedings of the 2009 European Control Conference, Budapest, Hungary, pp. 2432-2437, 2009.
  * [4] Sonya Ragothaman, Mahdi Maaref, and Zaher M. Kassa,  Autonomous Ground Vehicle Path Planning in Urban Environments using GNSS and Cellular Signals Reliability Maps: Simulation and Experimental Results, IEEE Transactions on Aerospace and Electronic Systems, vol. 57, Issue 4, Aug. 2021.
  * [5] Giulia Michieletto, Angelo Cenedese, and Antonio Franchi,  Bearing rigidity theory in SE(3), 2016 IEEE 55th Conference on Decision and Control (CDC), December 12-14, Las Vegas, USA, 2016.
  * [6] B. D. O. Anderson, C. Yu, B. Fidan and J. M. Hendrickx, Rigid graph control architectures for autonomous formations, in IEEE Control Systems Magazine, vol. 28, no. 6, pp. 48-63, Dec. 2008.
  * [7] Kwang-Kyo Oh, Hyo-Sung Ahn,  Formation control of mobile agents based on inter-agent distance dynamics, Automatica, Volume 47, Issue 10, Pages 2306-2312, 2011.
  * [8] Feng Liang, Fuqiang Wang, Xue Zhao and Yun Hou,  Distance-based control of formations with hybrid communication topology: Three-agent case, 2016 IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), 2016.
  * [9] Peng Sun, Bing Zhu, Zongyu Zuo, Michael V. Basin,  Vision-based finite-time uncooperative target tracking for UAV subject to actuator saturation, Automatica, Volume 130, 2021.
  * [10] S. Zhao and D. Zelazo,  Bearing rigidity theory and its applications for control and estimation of network systems: Life beyond distance rigidity, IEEE Control Systems Magazine, vol. 39, no. 2, April 2019.
  * [11] D. Zelazo, A. Franchi and P. R. Giordano, Rigidity theory in SE(2) for unscaled relative position estimation using only bearing measurements, 2014 European Control Conference (ECC), pp. 2703-2708, 2014.
  * [12] S. Zhao and D. Zelazo, Bearing Rigidity and Almost Global Bearing-Only Formation Stabilization, in IEEE Transactions on Automatic Control, vol. 61, no. 5, pp. 1255-1268, May 2016.
  * [13] D. Zelazo, P. R. Giordano and A. Franchi,  Bearing-only formation control using an SE(2) rigidity theory, 2015 54th IEEE Conference on Decision and Control (CDC), pp. 6121-6126,2015.
  * [14] Giulia Michieletto, Angelo Cenedese, Zelazo D.,  A unified dissertation on bearing rigidity theory, IEEE Transactions on Control of Network Systems. 2021 May 5.
  * [15] A. N. Bishop, T. H. Summers and B. D. O. Anderson,  Stabilization of stiff formations with a mix of direction and distance constraints, IEEE International Conference on Control Applications(CCA), 2013.


# DrBERT: Unveiling the Potential of Masked Language Modeling Decoder in BERT
pretraining

Wen Liang
Google Research
Mountain View, CA 94043
<EMAIL_ADDRESS>
&Youzhi Liang
Department of Computer Science
Stanford University
Stanford, CA 94305
<EMAIL_ADDRESS>

###### Abstract

BERT (Bidirectional Encoder Representations from Transformers) has
revolutionized the field of natural language processing through its
exceptional performance on numerous tasks. Yet, the majority of researchers
have mainly concentrated on enhancements related to the model structure, such
as relative position embedding and more efficient attention mechanisms. Others
have delved into pretraining tricks associated with Masked Language Modeling,
including whole word masking. DeBERTa introduced an enhanced decoder adapted
for BERT’s encoder model for pretraining, proving to be highly effective. We
argue that the design and research around enhanced masked language modeling
decoders have been underappreciated. In this paper, we propose several designs
of enhanced decoders and introduce DrBERT (Decoder-refined BERT), a novel
method for modeling training. Typically, a pretrained BERT model is fine-tuned
for specific Natural Language Understanding (NLU) tasks. In our approach, we
utilize the original BERT model as the encoder, making only changes to the
decoder without altering the encoder. This approach does not necessitate
extensive modifications to the model’s architecture and can be seamlessly
integrated into existing fine-tuning pipelines and services, offering an
efficient and effective enhancement strategy. Compared to other methods, while
we also incur a moderate training cost for the decoder during the pretraining
process, our approach does not introduce additional training costs during the
fine-tuning phase. We test multiple enhanced decoder structures after
pretraining and evaluate their performance on the GLUE benchmark. Our results
demonstrate that DrBERT, having only undergone subtle refinements to the model
structure during pretraining, significantly enhances model performance without
escalating the inference time and serving budget.

##  1 Introduction

The journey of deep learning is marked by significant innovations with
propelling advancements that redefine our understanding and capabilities in
solving complex problems [1, 2, 3],. ResNet revolutionized a wide range of
computer vision tasks with its deep residual learning framework, enabling the
training of substantially deeper networks by addressing the vanishing gradient
problem [2]. This breakthrough laid the groundwork for subsequent works in
model architectures. The transformer model shifted the paradigm, which
eschewed conventional recurrent layers in favor of attention mechanisms, thus
setting a new precedent in handling sequential data [4]. Building on these
foundations, BERT model marked another milestone in natural language
processing. By pretraining on a large corpus and adopting a bidirectional
approach, BERT achieved unprecedented performance across a range of language
tasks. These landmark contributions collectively have not only advanced the
field of deep learning but also have set the stage for our current exploration
into refining the BERT architecture through an pretraining and finetuning
approach.

Following the groundbreaking inception of the Transformer architecture, there
has been a continuous evolution in model design and methodology, further
enhancing its applicability and performance in diverse tasks. TransformerXL
extended the Transformer’s capability to process longer sequences, overcoming
the limitations of the standard architecture in handling extended context [5].
This was achieved through the innovative use of relative positional encodings,
allowing the model to capture dependencies over extended sequences, far beyond
the absolute positional encoding that was applied in Transformer and BERT
models. This enhancement not only addressed the limitations in handling
extended context but also paved the way for more dynamic and contextually
aware representations in sequence modeling. Complementing this, XLNet
integrated the strengths of both autoregressive and autoencoding techniques
[6]. It further leveraged the concept of relative positional embeddings,
enhancing the model’s ability to understand and predict the relationships
between elements in a sequence. This approach enabled XLNet to surpass BERT in
numerous benchmarks, This development was complemented by XLNet, a model by
Yang et al., which combined the strengths of autoregressive and autoencoding
techniques to outperform BERT in various benchmarks. The Funnel-Transformer,
another significant innovation, introduces a methodology that gradually
compresses the sequence of hidden states into a shorter format, effectively
reducing computation costs without compromising the model’s performance across
a variety of NLP tasks [7]. This trend underscores a growing emphasis in the
field not only on achieving high accuracy but also on improving computational
efficiency.

The original Transformer architecture was distinct in its separation of the
encoder and decoder components, each serving a unique function in the model’s
processing of sequences. BERT focused exclusively on the encoder part of this
architecture. By harnessing the power of the encoder, BERT excelled in
creating deep, bidirectional representations with Masked Language Modeling
(MLM), but it lose the sequential generation capabilities of the decoder. On
the other hand, T5 (Text-to-Text Transfer Transformer) took a holistic
approach by further developing both the encoder and decoder components [8]. T5
not only embraced the full encoder-decoder structure but also innovated by
treating every NLP task as a text-to-text problem and using a multi-tasking
pretraining strategy. This model incorporated varied methods such as MLM with
different lengths and formats, thereby expanding the versatility and
applicability of the Transformer architecture. Such developments in models
like T5 demonstrate the continuous evolution and adaptation of the Transformer
framework, exploring new potentials in both its separate and combined encoder-
decoder capabilities. These advancements reflect that, in addition to
improvements in model architecture, changes in strategies during the
pretraining phase are also very important.

Not surprisingly, alongside these architectural advancements, there has been a
significant emphasis on refining strategies and methods in optimization,
hyperparameter tuning, and model training. For instance, RoBERTa, an iteration
of BERT, has unveiled its potential by utilizing more extensive and diverse
datasets, dynamic masking in Masked Language Modeling (MLM), and a refined set
of hyperparameters [9]. This work underlines the importance of not just model
design, but also the quality and variety of data and training techniques.
Furthermore, some studies have delved into the optimal ratio of masking in
MLM, a key factor that significantly influences the efficiency and
effectiveness of the pretraining process [10]. Another intriguing area of
research is the adaptability and flexibility of training structures.
Innovations like BERT-of-Theseus demonstrate how a compressing strategy can
lead to more efficient models without compromising performance [11]. These
strategies involve progressively replacing parts of the network with more
efficient modules, thereby refining the model’s structure and training
process. Such developments suggest a growing recognition of the intricate
interplay between various aspects of model training, extending beyond mere
architectural changes to include strategic modifications in data handling,
training methodologies, and structural efficiency.

DeBERTa represents a significant milestone in the ongoing evolution of
language models, introducing several novel methods that have achieved state-
of-the-art performance. This model has innovatively combined disentangled
attention with enhanced mask decoders. However, DeBERTa’s introduction of the
disentangled attention mechanism represents a significant deviation from
vanilla multi-head attention, particularly in terms of its complexity and
resource demands. Compared to a BERT model with an equivalent number of layers
and parameters, DeBERTa incurs a higher computational cost for both training
and serving. Furthermore, for developers considering transitioning from BERT
to DeBERTa, there is an added layer of development cost, which may not always
be feasible or cost-effective for every application or organization.

In this paper, we propose DrBERT, a novel architecture that harnesses the
potential of the Masked Language Modeling (MLM) Decoder. Our primary
contributions are as follows:

1\. Introduction of DrBERT: Its encoder is identical to the vanilla BERT, but
it includes a redesigned MLM Decoder. The main modifications are:

  * •

Addition of multiple transformer block layers post-encoder, serving as the MLM
decoder.

  * •

In the MLM decoder, we remove the restriction present in the encoder’s
attention mechanism, which prevents attending to masked positions.

  * •

Integration of a degree of randomness before output, where the final output is
a randomly combined result of the encoder output and the MLM decoder output.

2\. Guidance on Hyperparameter Tuning for MLM Decoder: To further assist
readers, we have incorporated a dedicated section in our paper to guide the
tuning process of the MLM decoder’s hyperparameters.

To assess the performance of our proposed DrBERT method, we conducted a series
of rigorous evaluations across a variety of tasks. These tasks include, but
are not limited to, MNLI (Multi-Genre Natural Language Inference), SQuAD
(Stanford Question Answering Dataset), RACE (Reading Comprehension from
Examinations), among others. In addition, an extensive ablation study was
performed to validate the effectiveness of each modification made in DrBERT.
This study methodically deconstructed the model to isolate and evaluate the
impact of individual changes, thereby providing empirical evidence of their
contribution to the model’s overall performance. Furthermore, comparative
analyses were conducted against baseline models, including the original BERT
and other state-of-the-art architectures. These comparisons not only highlight
DrBERT’s advancements but also offer a comprehensive understanding of its
positioning in the current NLP landscape.

##  2 Related Works

###  2.1 Masked Language Modeling

In the context of traditional Language Modeling (LM), the objective is to
maximize the probability of a given sequence of tokens. This can be
mathematically defined as:

$\max_{\theta}\log p_{\theta}(X|\tilde{X})=\max_{\theta}\sum_{i\in C}\log
p_{\theta}(x_{i}|\tilde{X})$ (1)

Masked Language Modeling (MLM) has become a fundamental technique in the field
of natural language processing, particularly since the introduction of the
BERT model [12, 9]. BERT represents a significant shift in model architecture
by employing an encoder-only framework [1]. For autoregressive language
modeling, as seen in models like GPT, the objective differs in its sequential
approach to predicting each token. In autoregressive language modeling, each
token $X_{i}$ is predicted based on the sequence of preceding tokens.
Mathematically, this can be defined as:

$\max_{\theta}\log p_{\theta}(X|\tilde{X})=\max_{\theta}\sum_{i=1}^{N}\log
p_{\theta}(X_{i}|X_{1},X_{2},...,X_{i-1})$ (2)

The goal is to optimize the model parameters $\theta$ such that the
probability of each token $X_{i}$, given its preceding context, is maximized.
The encoder-only design of BERT enables bidirectional context understanding
and full-text attention, unlike autoregressive decoders that are limited to
unidirectional context. BERT’s training methodology evolves to bidirectional
MLM. This approach can be mathematically represented as follows:

$\max_{\theta}\log p_{\theta}(X_{masked}|\tilde{X})=\max_{\theta}\sum_{i\in
C_{\text{masked}}}\log p_{\theta}(X_{i}|X_{\text{unmasked}})$ (3)

Here, $X_{masked}$ represents the set of tokens that have been masked, and
$X_{unmasked}$ is the set of tokens that remain visible to the model. The
model parameters $\theta$ are optimized to maximize the log probability of
correctly predicting the masked tokens given the context of unmasked tokens.
BERT’s MLM approach randomly masks 15% of the tokens in each sequence. Within
this subset, 80% of the tokens are substituted with a [MASK] token, 10% remain
as is, and the remaining 10% are replaced with a random token from the
vocabulary. The model is trained to predict these tokens based on the context.
This training method allows BERT to effectively grasp the bidirectional
context, making it highly effective across a wide range of NLP tasks after
finetuning.

###  2.2 Finetuning

Following the pretraining phase, BERT and similar model will undergo a process
known as fine-tuning, which tailors them to specific tasks. In this phase, the
embedding corresponding to the [CLS] token, typically at the final layer, is
often employed as a representative summary of the entire input sequence and is
finetuned along with other model parameters to suit various downstream tasks.
This fine-tuning step is essential for adapting the generalized pretraining to
more specialized applications, thereby enhancing the model’s performance on
task-specific datasets.

There are variations in the finetuning process. Some approaches suggest the
possibility of dropping certain layers during fine-tuning for efficiency [13].
This efficiency-focused approaches suggest the feasibility of omitting
pretrained layers during fine-tuning. Notable findings in this area highlight
that:

  1. 1.

Up to 50% of pretrained transformer blocks can be pruned while retaining
approximately 98% efficiency in specific downstream tasks.

  2. 2.

The retention of lower layers is essential for maintaining performance in
downstream applications.

  3. 3.

For certain tasks, a minimal subset of layers, as few as 3 out of 12, can
sustain performance within a 1% variance of the full model’s capacity.

Besides, The T5 model, can be adapted for similar fine-tuning tasks by
utilizing only its encoder part [8]. DeBERTa introduces another approach,
where it is pretrained with an additional MLM decoder and then this decoder is
dropped during the fine-tuning phase [12]. In the following sections, we will
focus on a detailed introduction of the MLM Decoder as implemented in DeBERTa.

###  2.3 Enhanced Masked Language Modeling Decoder in DeBERTa

DeBERTa introduces an innovative approach known as the Enhanced Masked
Language Modeling Decoder (EMD). While BERT incorporates absolute positional
encodings at the input layer, DeBERTa adopts a different strategy. In DeBERTa,
absolute positions are incorporated after all Transformer layers, just before
the softmax layer dedicated to masked token prediction. This design choice
allows the model to capture the relative positions within all Transformer
layers and utilize absolute positions as complementary information during the
decoding phase. The placement of EMD enables the potential introduction of
additional useful information during the pre-training phase, apart from
positional data.

##  3 Methods

In considering the structural modifications of DeBERTa and their impact on the
overall computational cost of the model, we find the concept of Enhanced Mask
Decoders (EMD) to be particularly intriguing. The MLM decoder offers a
solution that it is utilized exclusively during the pretraining phase, thereby
avoiding additional computational burdens during the fine-tuning and serving
stages. This aspect caught our attention and led us to explore the potential
it holds for BERT pretraining. While the primary intent of introducing the
Enhanced Mask Decoder (EMD) in the DeBERTa model, as highlighted by the
authors, was to integrate absolute position information before the output
layer, insights from other studies suggest an alternative perspective [13]. It
is conceivable that incorporating additional MLM-only layers particularly
suited to tasks involving Masked Language Modeling. However, the ability to
understand the input text and the ability to distill language into specific
embeddings already present sufficiently in the earlier layers, even without
the need for this additional layer. Thus, I believe that isolating and
studying the MLM-specific decoder, a focus not yet extensively explored, is
both novel and highly necessary in the current research landscape. Therefore,
we introduce our method – the MLM Decoder foR BERT pretraining, termed DrBERT.

###  3.1 MLM Decoder

Our approach with DrBERT is to retain the core encoder structure of the
original, or vanilla BERT, while augmenting it with a specially designed
decoder that is active only during the pretraining stage. This additional
decoder is crafted to enhance the model’s understanding and processing of
language, drawing upon the strengths of the EMD approach. By incorporating
this decoder, we aim to improve the overall capabilities of the BERT model
without imposing significant computational costs during the subsequent phases
of fine-tuning and deployment. The design of DrBERT’s decoder component is
strategically aligned with the objectives of pretraining. This decoder is
adeptly positioned right after the BERT encoder, and just before the final
dense layer and softmax layer dedicated to masked token prediction. It focuses
on refining the model’s ability to predict masked tokens, thereby expanding
the depth and versatility of the encoded representations.

###  3.2 Decoder Layers without Attention Mask

Figure 1: Examples of attention heads with and without attention masks.

Another key aspect of our proposed methodology in DrBERT involves modifying
the restrictions present in the standard multi-head attention mechanism of
BERT, particularly in relation to masked positions. Traditionally, the BERT
architecture avoids attending to embeddings at the 15% masked positions during
the multi-head attention by setting a mask to the it during pretraining. The
embeddings at masked positions can be less informative and less accurate than
those are not masked. The mechanism helps ensure that the model is not unduly
influenced by potentially misleading or noisy inputs.

However, in DrBERT MLM Decoder, we remove the restriction in the last or last
a few decoder layers. A comparative visualization is depicted in Figure 1.
This modification is rooted in the understanding that as the layers in the
model become deeper, the output features become increasingly abstract and,
conceivably, become sufficiently enough to guess the masked or noised
information. Such information, though not fully revealed, can provide valuable
context to the model. It encourages the model to formulate hypotheses and
fosters a deeper alignment and understanding of the input based on a mix of
noise and these formulated assumptions. This approach not only enriches the
model’s interpretative capacity but also allows for a more nuanced and
comprehensive processing of the linguistic information, leading to enhanced
overall performance as shown in the experiments.

###  3.3 Random Mix of Encoder and Decoder Outputs

Figure 2: Random encoder-decoder output mixing.

In our approach, we introduce a modest blending of the encoder’s output into
the decoder’s output as shown in Figure 2. This technique serves a dual
purpose. Firstly, it encourages the MLM task to not depend exclusively on the
abstract representations formed in the deeper layers of the decoder. Instead,
it also takes into account the more direct and less processed information from
the encoder stage. Drawing inspiration from the NEFTUNE method, which
emphasizes the benefits of introducing a certain degree of randomness in
Language Model training, we understand that this can potentially enhance
performance [14]. Additionally, echoing the Theseus Compression Method
utilized in BERT-of-Theseus, the authors recognize that random layer
replacement is an effective approach to model compression [11]. In our case,
randomly skipping decoder layers can be seen as a form of compression as well.
Our experimental results show that while this mixing occurs across batch and
sequence dimensions, preserving the structure within the hidden embedding
dimension is crucial to maintain the integrity and quality of the output.
Experiments also reveal that using 80% of the decoder output mixed with 20% of
the encoder output can significantly improve performance on downstream tasks.

##  4 Results

In order to examine the benefits of DrBERT, we conduct experiments on various
NLP tasks with a few models as baselines.

###  4.1 Pretraining

Pretraining was conducted on both DrBERT-base and DrBERT-large models. The
encoder architecture, pretraining data, pretraining tasks, and mask settings
align precisely with those of the original BERT-base and BERT-large models.
The distinctive aspect of our DrBERT lies in the addition of a decoder
following the encoder. Details about the parameters of this decoder can be
found in Table 1. This strategic augmentation aims to enhance the model’s
capabilities while maintaining the foundational strengths of BERT, offering a
nuanced approach to language modeling that leverages the best of both encoder
and decoder functionalities.

Hyperparam | DrBERT-base | DrBERT-large
---|---|---
Number of decoder layers | 2 | 4
decoder layers with attention mask | [1] | [1, 2]
decoder layers without attention mask | [2] | [3, 4]
% of mix with encoder output | 20 | 20
Hidden size | 768 | 1024
FFN inner hidden size | 3072 | 4096
Number of attention heads | 12 | 16
Attention head size | 64 | 64
Table 1: Hyperparameters for decoders used in DrBERT-base and DrBERT-large
pretraining. The architectural hyperparameters, including hidden size and the
number of attention heads, are consistent with those of the corresponding BERT
encoder. These aligned parameters that remain unchanged are highlighted with a
gray background in the table for easy reference.

In our study, we conducted a comparative analysis using the BERT model and
BERT model with pre-LN [15] trained under identical settings as a baseline,
and the DeBERTa model as a point of comparison. Here, we trained our own
baseline models, utilizing the same hyperparameters, tokenizer, and
pretraining dataset to isolate the impact of other factors on model
performance. Our focus was solely on determining the effectiveness of the
architectural change. For instance, with RoBERTa [9], authors significantly
extended the pretraining duration, increasing the number of steps from 100K to
to 500K. This extended training led to substantial improvements in downstream
task performance, with the 500K step models outperforming other models in most
tasks. Authors emphasized that the longest-trained model still haven’t show
signs of overfitting. This indicates that simply increasing training steps can
continuously improve the model. Without a fully aligned comparison, we might
not arrive at the most insightful conclusions regarding the true impact of
architectural modifications on model performance.

Moreover, it is important to note that although the DeBERTa model was trained
for the same number of steps as other models, its disentangled attention
mechanism introduced a significant overhead comparing with original BERT
model. This added complexity affected both the pretraining and fine-tuning
phases, but ours DrBERT models only have extra ovearhead during pretraining.
The implications of this increased overhead are critical, especially when
considering the balance between model performance and computational
efficiency. Our analysis aims to provide insights into how these different
architectures impact training and operational efficiency, while also
maintaining or enhancing language processing capabilities.

###  4.2 Performance on GLUE Tasks

We summarize the results on seven Natural Language Understanding (NLU) tasks
from the GLUE benchmark. These tasks and their corresponding metrics are as
follows:

  * •

AX (GLUE Diagnostic Task): Tests ability to understand linguistic phenomena
such as logic, predicate-argument structure, and lexical semantics. The
evaluation metric is typically overall accuracy across all diagnostic
categories.

  * •

CoLA (Corpus of Linguistic Acceptability): Measures grammatical correctness.
The metric used is Matthew’s correlation.

  * •

MNLI-m/mm (Multi-Genre NLI - Matched/Mismatched): Assesses the ability to
predict textual entailment across different genres. Accuracy is the metric for
both the matched (MNLI-m) and mismatched (MNLI-mm) sections.

  * •

MRPC (Microsoft Research Paraphrase Corpus): Focuses on identifying whether
two sentences are paraphrases of each other. Accuracy is used as the main
evaluation metric.

  * •

QNLI (Question NLI): Involves determining whether a context sentence contains
the answer to a question. Accuracy is the evaluation metric.

  * •

QQP (Quora Question Pairs): Focuses on determining if two questions asked on
Quora are semantically equivalent. The metrics for evaluation are accuracy and
F1 score.

  * •

RTE (Recognizing Textual Entailment): Sentence entailment task. Accuracy is
used as the metric.

  * •

STS (Semantic Textual Similarity Benchmark): Evaluates the degree of semantic
similarity between two sentences. The primary metric is Pearson and Spearman
correlation coefficients.

These tasks provide a broad evaluation of the model’s performance across
various dimensions of NLU. The experiment results are shown in Table 2

Model | AX | COLA | MNLI-m/mm | MRPC | QNLI | QQP | RTE | SST | Avg
---|---|---|---|---|---|---|---|---|---
BERT-base [1] | 85.05 | 83.13 | 84.67/84.85 | 88.48 | 91.84 | 90.85 | 71.92 | 93.58 | 86.04
BERT-PreLN-base [15] | 85.03 | 83.89 | 85.10/85.45 | 88.24 | 91.96 | 90.78 | 71.56 | 92.89 | 86.10
DeBERTa-base [12] | 85.43 | 82.74 |  85.51/85.66 | 88.48 | 92.17 | 91.18 | 71.79 | 93.58 | 86.28
DrBERT-base | 85.86 | 83.13 |  85.66/85.59 | 88.73 | 92.39 | 90.98 | 71.12 | 93.81 | 86.36
BERT-large [1] | 87.21 | 85.43 | 87.32/87.49 | 87.50 | 93.47 | 91.44 | 72.92 | 94.04 | 87.42
BERT-PreLN-large [15] | 87.11 | 84.99 | 86.91/87.41 | 88.24 | 93.70 | 91.24 | 75.81 | 93.69 | 87.68
DeBERTa-large [12] | 87.23 | 85.62 |  87.01/87.54 | 87.75 | 93.14 | 91.17 | 76.53 | 94.84 | 87.87
DrBERT-large | 87.12 | 85.81 |  87.36/87.28 | 88.24 | 93.28 | 91.42 | 76.90 | 94.27 | 87.96
Table 2: Results on 8 GLUE benchmark tasks. Note that the results presented
here are not based on publicly available checkpoints but are derived from
models we trained ourselves with aligned settings. Given that DeBERTa incurs
higher costs in finetuning and has more parameters, its corresponding results
are highlighted in gray for comparative analysis.

In our finetuning experiments, each model was trained on the designated
training set and subsequently evaluated on the development set. Table 2
illustrates that under identical conditions and with the same number of
pretraining samples, DrBERT consistently outperforms both BERT and DeBERTa
models in terms of the averaged score. Notably, MNLI, the largest dataset
among these tasks, serves as a robust indicator of model performance. Our
model competes closely with the similarly sized DeBERTa model, achieving
slightly higher scores in the matched condition and comparable results in the
mismatched one. However, it’s crucial to consider that the DeBERTa model’s
encoder is approximately 17% larger in parameter size than DrBERT’s, and it
incurs a 36% higher cost in finetuning time and a 30% increase in inference
time. Despite these disparities, DrBERT matches or even surpasses DeBERTa in
various benchmarks in both base and large level sizes. Moreover, DrBERT
achieves significant improvement over the original BERT at the same finetuning
and serving costs, underlining its efficiency and effectiveness. The results
tell us that this decoder side only enhancement in DrBERT significantly boosts
performance without incurring further additional costs. Building on this, we
extended our testing to question answering tasks, further solidifying the
viewpoint that our work can yield substantial performance gains.

###  4.3 Performance on SQuAD Tasks

| SQuAD v1 | SQuAD v2
---|---|---
Model | EM | F1 | EM | F1
BERT-base [1] | 83.70 | 90.63 | 76.83 | 79.78
BERT-PreLN-base [15] | 83.65 | 90.58 | 77.03 | 79.87
DeBERTa-base [12] | 83.75 | 90.78 | 77.71 | 80.77
DrBERT-base | 84.20 | 91.12 | 77.87 | 80.94
BERT-large [1] | 86.58 | 92.92 | 82.06 | 85.22
BERT-PreLN-large [15] | 85.94 | 92.56 | 81.19 | 84.42
DeBERTa-large [12] | 85.91 | 92.62 | 82.09 | 85.12
DrBERT-large | 86.11 | 92.74 | 82.50 | 85.29
Table 3: Results on SQuAD v1 and v2 tasks. Given that DeBERTa incurs higher
costs in finetuning and has more parameters, its corresponding results are
highlighted in gray for comparative analysis.

The Stanford Question Answering Dataset (SQuAD) serves as a benchmark for
evaluating question answering systems in NLP. It comprises two versions: SQuAD
v1, focused on answerable questions from given passages, and SQuAD v2, which
includes both answerable and unanswerable questions. Performance is measured
using Exact Match (EM) and F1 Score, reflecting the precision and accuracy of
the model’s responses.

The experiments conducted on both the v1 and v2 tasks, as presented in Table
3, followed a similar trend to those performed on the GLUE tasks: all models
were trained on the training set and evaluated on the development set. While
the results for DrBERT on SQuAD v1 did not achieve the highest scores in the
table, its performance on SQuAD v2, particularly for the base size model, was
notably stronger than the original BERT. Especially, when considering that
DrBERT competes closely with, or even surpasses, the DeBERTa model, the
improvement is significant. These results underscore DrBERT’s efficiency and
efficacy, showcasing its ability to deliver robust performance in question
answering tasks, a key area in NLP, without additional overhead.

###  4.4 Ablation Study

The DrBERT’s MLM Decoder involves multiple modifications of architecture and
randomization which introduce many parameters in the process. It is essential
to clearly articulate whether each change is necessary and how much it
contributes to the final result. Furthermore, understanding the influence of
each major parameter on the outcome is also crucial. To further demonstrate
the effectiveness of our method, we conducted several ablation experiments on
selected GLUE tasks and SQuAD tasks with base-level sized models.

####  4.4.1 Number of MLM Decoder Layers

Model | AX | COLA | MNLI-m/mm | MRPC | QNLI | QQP | SQuAD v2 EM/F1
---|---|---|---|---|---|---|---
BERT-base | 85.05 | 83.13 | 84.67/84.85 | 88.48 | 91.84 | 90.85 | 76.83/79.78
\+ 1 decoder layers | 84.89 | 83.28 |  85.30/85.06 | 88.24 | 91.67 | 90.78 | 76.92/80.12
\+ 2 decoder layers | 85.10 | 83.41 | 84.82/85.64 | 88.97 | 91.31 | 90.99 |  77.67/81.13
\+ 3 decoder layers | 84.91 | 83.41 | 84.48/84.53 | 87.75 | 91.62 | 90.78 | 77.14/80.19
\+ 4 decoder layers | 84.74 | 81.21 | 84.19/84.64 | 88.24 | 91.01 | 91.11 | 75.89/78.72
\+ 6 decoder layers | 83.49 | 80.35 | 83.98/84.19 | 86.52 | 91.01 | 90.81 | 74.21/77.03
Table 4: Comparative results on various tasks with different numbers of
decoder layers added to BERT-base.

In our study, we initially sought to determine the optimal number of MLM
decoder layers for improving model performance. To achieve this, we conducted
an ablation study focusing solely on increasing the number of decoder layers
during pretraining, without implementing other changes. During finetuning, we
use the same number of layers as the BERT-base model.

Our findings in Table 4 revealed that adding two decoder layers to the BERT-
base model yielded the best results. Parallel experiments were conducted on
the larger model as well, where adding four decoder layers to the BERT-large
model proved most effective. These results led us to conclude that augmenting
the model with additional decoder layers equivalent to one-sixth of the number
of encoder layers seems to be the optimal strategy.

####  4.4.2 Decoder Layers without Attention Masks

Layers Without Attention Masks | AX | COLA | MNLI-m/mm | MRPC | QNLI | QQP | SQuAD v2 EM/F1
---|---|---|---|---|---|---|---
$[]$ | 85.10 | 83.41 |  85.30/85.06 | 88.24 | 91.67 | 90.78 | 76.92/80.12
decoder $[1,2]$ | 84.77 | 82.84 | 84.80/85.08 | 87.50 | 91.32 | 90.86 | 78.23/81.26
decoder $[2]$ | 85.08 | 83.13 | 85.28/85.40 | 89.46 | 91.62 | 90.99 |  78.83/81.73
encoder $[11,12]$ decoder $[1,2]$ | 84.95 | 83.70 | 84.49/85.02 | 87.75 | 91.43 | 90.86 | 77.68/80.61
Table 5: Comparative results on various tasks for a 12-layer encoder + 2-layer
decoder DrBERT model with different configurations regarding the application
of attention mask restrictions. The configurations are denoted by lists, where
each list specifies the layers operating without attention mask restrictions.

In our ablation study shown in Table 5, we examined the effects of releasing
attention mask restrictions across various configurations of the DrBERT model
during LM pretraining. Initially, this approach was applied to the encoder
layers, where we observed no positive impact on performance. However, a
notable improvement emerged when the same restriction release was applied to
selected layers of the DrBERT MLM Decoder. Additional experiments were carried
out on the DrBERT-large, revealing that the best results were achieved by
removing the attention mask restrictions on the final two layers of a four-
layer decoder. Here, we found that the optimal configuration involved
releasing the attention mask restrictions in the latter 50% of the decoder
layers. This specific adjustment yielded the most significant performance
enhancements. These results highlight the nuanced interplay between model
architecture and attention mechanisms, emphasizing the potential of targeted
modifications in MLM decoder to achieve better outcomes in complex language
processing tasks.

####  4.4.3 Effectiveness of Output Random Mix

Mix Axis | AX | COLA | MNLI-m/mm | MRPC | QNLI | QQP | SQuAD v2 EM/F1
---|---|---|---|---|---|---|---
(batch, seq, hidden) | 84.68 | 83.03 | 84.78/84.91 | 88.73 | 91.16 | 90.80 | 77.83/80.84
(batch, seq) | 85.86 | 83.13 |  85.66/85.59 | 88.73 | 92.39 | 90.98 |  77.87/80.94
(batch) | 85.17 | 83.60 | 85.11/85.23 | 88.97 | 92.46 | 90.88 | 78.37/81.34
Table 6: Comparative results on various tasks with different mixing axis configurations on DrBERT-base model during pretraining. We employ the decoder output values 80% of the time, while for the remaining 20%, we utilize the values from the encoder. The configurations specify the dimensions across which mixing is applied, impacting the model’s performance on a range of tasks. Ratio of Decoder | AX | COLA | MNLI-m/mm | MRPC | QNLI | QQP | SQuAD v2 EM/F1
---|---|---|---|---|---|---|---
0.6 | 84.65 | 82.45 | 85.04/85.05 | 87.75 | 91.93 | 90.99 | 77.43/80.45
0.7 | 84.81 | 83.89 | 85.08/85.15 | 87.99 | 92.37 | 91.08 | 78.31/81.49
0.8 | 85.86 | 83.13 |  85.66/85.59 | 88.73 | 92.39 | 90.98 |  77.87/80.94
0.9 | 84.64 | 83.22 | 84.51/85.48 | 87.99 | 92.17 | 91.09 | 76.22/79.29
Table 7: Comparative results on various tasks with different ratios of decoder
output in a DrBERT-base model during pretraining.

In our ablation studies, we explored the integration of randomness into the
model output before the Softmax layer with Masked Language Modeling Task. A
particularly effective method was found is to randomly alternate between using
outputs from the decoder and the encoder. Some results of these experiments
are detailed in Table 6. We observed that a complete mix of the two outputs
did not yield optimal results, possibly due to the disruption of the output
embeddings. To address this, we adjusted our randomization strategy to only
apply randomness along the (batch, seq) dimensions, maintaining the
cohesiveness of individual output embeddings. This implies that an output
embedding is sourced entirely from either the decoder or the encoder.

Furthermore, we experimented with varying mix ratios of decoder and encoder
outputs. As shown in Table 7, the most effective ratio during pretraining was
found to be 80% decoder output and 20% encoder output. This specific mix ratio
led to the best performance, underlining the importance of fine-tuning the
balance between the two types of outputs to achieve optimal results in the
finetuning phase. The 80% ratio for utilizing decoder output emerged as a
reasonable choice. If this ratio were lower, it might result in insufficient
training of the decoder, diminishing its potential benefits. Conversely, if
the ratio were higher, it could undermine the intended effect of randomness.
Therefore, this balanced approach not only ensures effective training of the
decoder but also maintains the necessary level of randomness to optimize the
model’s overall performance.

##  5 Conclusion

In this paper, we introduced DrBERT, an enhanced version of the original BERT
model, focusing on improvements with the Masked Language Modeling (MLM)
Decoder used in pretraining. Key innovations include adding transformer block
layers as MLM decoder, removing restrictions on attending to masked positions
in the MLM decoder, and introducing a degree of output randomness by combining
encoder and decoder outputs. Through rigorous evaluations on tasks such as
MNLI, SQuAD, and RACE, DrBERT demonstrated superior performance over the
original BERT and other state-of-the-art models, without any increasing
computational complexity in further finetuning and inference. The ablation
study confirmed the effectiveness of each modification, highlighting DrBERT’s
potential in advancing NLP efficiency and effectiveness. Furthermore, we
provided optimal hyperparameter settings for DrBERT’s decoder, aiding in
enhancing efficiency, ensuring consistent performance, while also reducing
energy consumption and emissions. This work not only presents a significant
step forward in language model optimization but also offers valuable insights
for future research in the field.

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [3] Wen Liang, Youzhi Liang, and Jianguo Jia.  Miamix: Enhancing image classification through a multi-stage augmented mixed sample data augmentation method.  Processes, 11(12), 2023.
  * [4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context, 2019.
  * [6] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  Xlnet: Generalized autoregressive pretraining for language understanding, 2020.
  * [7] Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le.  Funnel-transformer: Filtering out sequential redundancy for efficient language processing.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4271–4282. Curran Associates, Inc., 2020.
  * [8] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
  * [9] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * [10] Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen.  Should you mask 15
  * [11] Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou.  Bert-of-theseus: Compressing BERT by progressive module replacing.  CoRR, abs/2002.02925, 2020.
  * [12] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  Deberta: Decoding-enhanced bert with disentangled attention, 2021.
  * [13] Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov.  Poor man’s BERT: smaller and faster transformer models.  CoRR, abs/2004.03844, 2020.
  * [14] Neel Jain, Ping yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Neftune: Noisy embeddings improve instruction finetuning, 2023.
  * [15] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu.  On layer normalization in the transformer architecture, 2020.