# General Null Lagrangians and Their Novel Role in Classical Dynamics

R. Das1 and Z.E. Musielak2 1Department of Physics and Earth Space Science, The
University of Indianapolis, Indianapolis, IN 46227, USA  2Department of
Physics, The University of Texas at Arlington, Arlington, TX 76019, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A method for constructing general null Lagrangians and their higher harmonics
is presented for dynamical systems with one degree of freedom. It is shown
that these Lagrangians can be used to obtain non-standard Lagrangians, which
give equations of motion for the law of inertia and some dissipative dynamical
systems. The necessary condition for deriving equations of motion by using
null Lagrangians is presented, and it is demonstrated that this condition
plays the same role for null Lagrangians as the Euler-Lagrange equation plays
for standard and non-standard Lagrangians. The obtained results and their
applications establish a novel role of null Lagrangians in classical dynamics.

##  1 Introduction

There are three families of Lagrangians that can be classified as standard,
non-standard and null Lagrangians. The main characteristic of standard
Lagrangians is the presence of the kinetic and potential energy-like terms,
and these Lagrangians are commonly used in Classical Mechanics (CM) to derive
equations of motion for different dynamical systems (e.g., [1-5]). On the
other hand, in non-standard Lagrangians, neither the kinetic nor the potential
energy-like terms can be identified. Different methods have been developed to
derive the standard and non-standard Lagrangians for given ordinary
differential equations (ODEs) [6-12].

The third family of null Lagrangians (NLs) has the following two main
equivalent characteristics: (i) they must satisfy identically the Euler-
Lagrange (E-L) equation: the null condition, and (ii) they must be expressed
as the total derivative of any scalar function known as gauge function: the
gauge condition. The properties and applications of these null Lagrangians
have been extensively explored in different fields of mathematics (e.g.,
13-20]) as well as in some physical applications (e.g., [21,22]) that include
restoring Galilean invariance of Lagrangians in Newtonian dynamics [23,24],
and introducing forces to CM [25-27].

Several methods of constructing the NLs have been proposed and most of them
rely on specifying a gauge function and using it to obtain the resulting null
Lagrangian [13,14,26,27]. However, in this paper, we develop a new method that
is based on a generating function that differs from the gauge function. It is
shown that using this generating function a general null Lagrangian can be
obtained. Depending on the form of the generating function, either standard or
non-standard null Lagrangians can be constructed [28]. Moreover, the method
also allows us to derive higher harmonics of these null Lagrangians, and the
harmonics are new mathematical objects in calculus of variations; therefore,
their role and possible applications are discussed.

Since the NLs satisfy identically the E-L equation, they can be added to any
standard or non-standard Lagrangian, and this addition does not affect the
resulting equation of motion [13-18]. Despite this well-known property of the
NLs, we demonstrate that they may be used to construct non-standard
Lagrangians, which then give equations of motion when susbtituted into the E-L
equations. Using this approach, the NLs for some equations of motion can be
found and compared to other Lagrangians for these equations, which may give
new insights into symmetry of the Lagrangians and the resulting equations of
motion [29-33]. The fact that equations of motion can be derived by using the
NLs demonstrates a novel role of these Lagrangians in classical dynamics.

Applications of the obtained results show that the law of inertia can be
derived from its null Lagrangian, and that other constructed NLs give
equations of motion for some dissipative systems with linear and quadratic
dissipative terms. There have been many attempts to establish Lagrangian
formalism for dissipative systems [7-9,27,34-47]. However, none of the
previous work on this problem has involved null Lagrangians and shown that
they can be used to obtain equations of motion. The results of this paper
demonstrate the validity of this approach and presents the necessary condition
that null Lagrangians must obey in order to give equations of motion. The
condition plays the same role for null Lagrangians as the Euler-Lagrange
equation plays for standard and non-standard Lagrangians.

The paper is organized as follows. In Section 2, Lagrangian formalism and its
Lagrangians are briefly described; construction of null Lagrangians and non-
standard Lagrangians resulting form them is presented in Sections 3 and 4,
respectively; applications of the results to dynamical systems are considered
and discussed in Section 5; and conclusions are given in Section 6.

##  2 Lagrangian formalism and its Lagrangians

###  2.1 Standard and non-standard Lagrangians

Dynamical systems with holonomic constraints can be analyzed using the
Lagrangian formalism. The foundation of this formalism is the smooth
configuration manifold $\mathcal{Q}$ constructed from the generalized
coordinates of the system of interest with holonomic constraints and its
tangent bundle $T\mathcal{Q}$. A Lagrangian $\mathcal{L}$ for this system is
commonly defined as a $C^{\infty}$ map from $T\mathcal{Q}$ to the real line
$\mathbf{R}$, i.e., $\mathcal{L}:T\mathcal{Q}\rightarrow\mathbf{R}$ [3].

In the case of one-dimensional dynamical systems, a Lagrangian can be
symbolized as $\mathcal{L}=\mathcal{L}(x,\dot{x})$ in an open neighborhood
$\mathcal{U}\subset T\mathcal{Q}$ containing a point $p\in T\mathcal{Q}$ in a
local coordinate system defined by the coordinate function
$x:\mathcal{Q}\rightarrow\mathbf{R}$ and its associated tangent vector
$\dot{x}$ at the point $p$ using a diffeomorphism
$\phi:p\in\mathcal{U}\subset{T{\mathcal{Q}}}\rightarrow\mathcal{Q}\times\mathbf{R}$
known as a local trivialization. Similarly, the explicit dependence of a
Lagrangian on time can be defined as
$\mathcal{L}:T\mathcal{Q}\times\mathbf{R}\rightarrow\mathbf{R}$ and thus
expressing $\mathcal{L}=\mathcal{L}(x,\dot{x},t)$. After a suitable Lagrangian
for an interested system is constructed or discovered, the time evolution of
the system is analyzed by applying the Euler-Lagrange operator on the chosen
Lagrangian, i.e., $\hat{EL}[\mathcal{L}(\dot{x},x,t)]=0$, with $\hat{EL}$
being the Euler-Lagrange (E-L) operator [2-4].

The Euler-Lagrange (EL) operator in each degree of freedom for a system is a
linear operator acting on the vector space of $C^{\infty}$ functions defined
on $T\mathcal{Q}$ and thus $\mathcal{L}\in\mathcal{F}(T\mathcal{Q})$ with
$\mathcal{F}$ denoting the space of $C^{\infty}$ functions. Most commonly, a
standard strategy for the construction of the most common Lagrangians (thus
known as standard Lagrangians) are carried out from the difference of the
kinetic and potential energy of the respective system under consideration,
i.e., $\mathcal{L}(x,\dot{x})=\frac{1}{2}\dot{x}^{2}-V(x)$, with the potential
energy $V(x)$ for a system with only one degree of freedom [1-3].

On the other hand, non-standard Lagrangians are dependent on functions of time
in addition to the usual dynamical variables such as $x$ and $\dot{x}$ for
systems with only one degree of freedom [7-11]. In general, the construction
of these non-standard Lagrangians for the set of desired equations of motion
requires solving nonlinear (Riccati-type) equations [9,12]. However, the
problem can be simplified by replacing functions to be determined by
constants, for instance, we may consider the following form of the non-
standard Lagrangian

$\mathcal{L}_{ns}(\dot{x},x,t)={1\over{a_{1}\dot{x}+a_{2}t+a_{3}}}\ ,$ (1)

with constants $a_{1}$, $a_{2}$, and $a_{3}$ describes a system undergoing a
constant acceleration. It should be emphasized that the above method for
constructing non-standard Lagrangians $C^{\infty}$ functions demands an extra
constraint that the denominators do not vanish, i.e.,
${a_{1}\dot{x}+a_{2}t+a_{3}}\neq 0$ in the domain
$\mathcal{U}\subset{T\mathcal{Q}}$. Therefore, non-standard Lagrangians in
general represent dynamical systems with more constraints than the usual
accompanying holonomic constraints. The construction of these Lagrangians can
be carried out with the assumption that the corresponding Lagrangians are
well-behaved on the domain $\mathcal{U}\subset{T\mathcal{Q}}$.

###  2.2 Null Lagrangians

Since the null space of the Euler-Lagrange (EL) operator is itself a vector
subspace, the members of this null space, rightly known as the null
Lagrangians, are nullified by the E-L operator. More explicitly, an action
functional defined in terms of null Lagrangians as

$A[x;t_{e},t_{o}]=\int^{t_{e}}_{t_{o}}L_{\rm null}(t,x,\dot{x})dt\ ,$ (2)

satisfies the property $A[x;t_{e},t_{o}]=A[x+\eta;t_{e},t_{o}]$ for continuous
functions $x(t)$ and $\eta(t)$ of time on the same domain. This implies
through approximations [13,14] that $A[x;t_{e},t_{o}]=A[\eta;t_{e},t_{o}]$ for
all permissible continuous functions $x(t)$ and $\eta(t)$ on the domain such
that $x(t_{e})=\eta(t_{e})$ and $x(t_{0})=\eta(t_{o})$. In other words, the
above action integral produces the same real number for all admissible
trajectories $x(t)$’s between $t_{e}$ and $t_{o}$ in the same domain, which in
turn implies that the variation of the action integral is rendered identically
to zero. This also implies that no preferred trajectory that extremizes the
action integral exists for the relevant null Lagrangians. Therefore, the E-L
operator renders all the null Lagrangians defined on the same domain
identically to zero, i.e., no equation of motion can be obtained directly from
the application of the E-L operator on null Lagrangians.

However, the study of null space of the E-L operator has been of particular
interest due to its applicability to the exploration of symmetries in field
theories. An effort to construct these null Lagrangians has been an active
practice for a while [13,14]. The most prevalent constructions centered around
the observation that the E-L operator nullifies the total time derivative of
an arbitrary $C^{1}$ (continuous first derivatives) function $\Phi(x,t)$,
known as the gauge function, defined on the domain of interest
$\mathcal{U}\subset{T\mathcal{Q}}$. More explicitly, a Lagrangian is null if,
and only if, it is the total time derivative of a scalar function $\Phi(x,t)$
[13] that can be expressed as

$L_{null}(\dot{x},x,t)=\frac{d\Phi(x,t)}{dt}=\frac{\partial\Phi}{\partial
x}\dot{x}+\frac{\partial\Phi}{\partial t}.$ (3)

Again, it should be emphasized that the above form of a null Lagrangian is
rendered identically to zero by the E-L operator, i.e.,
$\hat{EL}[L_{null}:=\frac{d\Phi(x,t)}{dt}]=0$. Then a natural strategy to
construct a standard null Lagrangian is to cast it in the following form:

$L_{null}(\dot{x},x,t)=B(x,t)\dot{x}+C(x,t)$ (4)

such that $B(x,t):=\frac{\partial\Phi}{\partial x}$ and
$C(x,t):=\frac{\partial\Phi}{\partial t}$. A similar construction has been
presented in [14]. A simple example of a standard null Lagrangian can be
obtained from a given gauge function $\Phi(x,t):=f_{1}(t)x^{2}+f_{2}(t)$ as
$L_{null}=2f_{1}(t)x\dot{x}+\dot{f}_{1}(t)x^{2}+\dot{f}_{2}(t)$ with the
obvious identifications $B(x,t)=2f_{1}(t)x$ and
$C(x,t)=\dot{f}_{1}(t)x^{2}+\dot{f}_{2}(t)$. It is also possible to construct
non-standard null Lagrangians following the same method as above. For example,
the following non-standard null Lagrangian

$L_{ns,test1}(\dot{x},x,t)={{a_{1}\dot{x}}\over{a_{2}x+a_{4}}}\ .$ (5)

can be obtained from $\Phi(x,t)={{a_{1}}\over{a_{2}}}\ln|a_{2}x+a_{4}|$ [28].
Therefore, it is always possible to construct a standard as well as a non-
standard null Lagrangian in the above form in Eq.(4) if a corresponding gauge
function $\Phi(x,t)$ is known.

On the contrary, it is also possible to construct some standard null
Lagrangians in the above form in Eq. (4) without any reference to the
corresponding gauge functions. In the following section, an earnest humble
effort is made to construct a set of both standard and non-standard null
Lagrangians and their higher harmonics in the similar form as in Eq. (4)
without explicitly resorting to any gauge function.

##  3 Method to construct null Lagrangians

###  3.1 Standard null Lagrangians

The construction of null Lagrangians is motivated by the recognition of the
null-condition $\hat{EL}[\mathcal{L}_{null}(\dot{x},x,t)]=0$, i.e., the action
of the E-L operator on these smooth functions nullifies them identically to
zero. In other words, these null Lagrangians must satisfy the following
relation:

$\frac{dp_{null}}{dt}=\frac{\partial L_{null}}{\partial x}\ ,$ (6)

with the generalized momentum $p_{null}=\partial{L_{null}}/\partial{\dot{x}}$.

Proposition 1: Let $B(x,t)$, $C(x,t)$ and $f(t)$ be locally $C^{1}$ functions
on $\mathcal{U}\subset{T\mathcal{Q}}$ such that they satisfy the following
condition:

$\frac{dB(x,t)}{dt}=\left[\frac{\partial B(x,t)}{\partial
x}\right]\dot{x}+\frac{\partial[xC(x,t)]}{\partial x}.$ (7)

Then $B(x,t)$ generates a family of null Lagrangians $L_{null}(\dot{x},x,t)$
of the form

$L_{null}(\dot{x},x,t)=B(x,t)\dot{x}+C(x,t)x+f(t).$ (8)

Proof: The proof follows from the null-condition
$\hat{EL}[{L}_{null}(\dot{x},x,t)]=0$ and Eq. (6).

It is interesting to note that the only restriction on $B(x,t)$ and $C(x,t)$
is that they be continuously differentiable functions on
$\mathcal{U}\subset{T\mathcal{Q}}$ and they satisfy the null condition, i.e.,
Eq. (6). A similar construction is presented in [14] albeit with a different
constraint.

However, the application of the null condition, Eq. (7), on a Lagrangian of
the form in Eq. (8) implies that there exists a gauge function $\Phi(x,t)$
such that the gauge condition, $L_{\rm null}(\dot{x},x,t)=d\Phi(x,t)/dt$, is
automatically satisfied. Therefore, the null condition on the function
$B(x,t)$ is equivalent to the gauge condition on the gauge function
$\Phi(x,t)$ related to the same null Lagrangian. In this paper, no direct
reference to a gauge function $\Phi(x,t)$, i.e., no a priori assumption of the
gauge condition on the gauge function is made to construct the corresponding
null Lagrangian. Instead, a general function $B(x,t)$, called a generating
function, is chosen along with its null condition to construct the appropriate
null Lagrangian. Now, some choices of $B(x,t)$ generating different null
Lagrangians would clarify the applications of the above proposition.

  1. 1.

For the simplest choice of a constant function $B(x,t)$, Eq. (7) forces both
$xC(x,t)$ and $L_{null}(\dot{x},x,t)$ to be functions of $t$ only, implying
that $\hat{EL}[{L}_{null}(\dot{x},x,t)]=0$.

  2. 2.

If $B(x,t)$ is a linear function of $x$, then the null-Lagrangian presented in
[27] is reproduced. Let $B(x,t)=f_{1}(t)x+f_{2}(t)t+f_{3}(t)$ such that
$f_{1}(t)$, $f_{2}(t)$, $f_{3}(t)$ and $f_{4}(t)$ are assumed to be arbitrary
but at least twice differentiable functions of the independent variable, then
the following null Lagrangian [26] is obtained

${L}_{null1}(\dot{x},x,t)=\left[f_{1}(t)x+f_{2}(t)t+f_{3}(t)\right]\dot{x}$
$\hskip
18.06749pt+\left[\frac{1}{2}\dot{f}_{1}(t)x+\dot{f}_{2}(t)t+f_{2}(t)+\dot{f}_{3}(t)\right]x+f_{4}(t)\
.$ (9)

  3. 3.

Now, let $B(x,t)=f_{1}(t)x^{2}+f_{2}(t)t+f_{3}(t)$. Then the corresponding
null Lagrangian turns out to be

${L}_{null2}(\dot{x},x,t)=\left[f_{1}(t)x^{2}+f_{2}(t)t+f_{3}(t)\right]\dot{x}$
$\hskip
18.06749pt+\left[\frac{1}{3}\dot{f}_{1}(t)x^{2}+\dot{f}_{2}(t)t+f_{2}(t)+\dot{f}_{3}(t)\right]x+f_{4}(t)\
.$ (10)

  4. 4.

It follows from the above special choices of $B(x,t)$ that it can be any
polynomial of $x$. However, in general, $B(x,t)$ can be any continuously
differentiable function of any order such as a $C^{k}$ function to obtain a
null Lagrangian. To show that, let
$B(x,t)=f_{1}(t)\sin(x)+f_{2}(t)e^{x}t+f_{3}(t)$. Then, the corresponding null
Lagrangian turns out to be

${L}_{null3}(\dot{x},x,t)=\left[f_{1}(t)sin(x)+f_{2}(t)e^{x}t+f_{3}(t)\right]\dot{x}$
$\hskip
18.06749pt-\dot{f}_{1}(t)cos(x)+\dot{f}_{2}(t)e^{x}t+f_{2}(t)e^{x}+\dot{f}_{3}(t)x+f_{4}(t)\
.$ (11)

Furthermore, since the addition of a total time derivative of a function whose
variation vanishes at the both ends of the trajectory does not affect the
dynamics of the system, the total time derivative of $B(x,t)$ can be added to
the null-Lagrangian in Eq. (8) to generate its higher harmonics, which is
presented below in Proposition 2.

Proposition 2: Let $B(x,t)$, and $C(x,t)$ be locally $C^{\infty}$ functions on
$\mathcal{U}\subset{T\mathcal{Q}}$ such that they are differentiable up to
order $n$ to satisfy the following condition:

$\frac{dB_{n}(x,t)}{dt}=\left[\frac{\partial B_{n}(x,t)}{\partial
x}\right]\dot{x}+\frac{\partial[xC(x,t)]_{n}}{\partial x},$ (12)

with

$B_{n}(x,t):=\Sigma^{n}_{i=0}\mathcal{C}^{n}_{n-i}B^{(i)}(x,t)$ (13)

and

$[xC(x,t)]_{n}:=\Sigma^{n}_{i=0}\mathcal{C}^{n}_{n-i}\left[xC(x,t)\right]^{(i)}.$
(14)

Here, $(i)$ represents the $i$-th spatial derivative and
$\mathcal{C}^{n}_{n-i}=n!/[(n-i)!i!]$. Then $B(x,t)$ generates a family of
higher harmonics of null Lagrangians $L_{null}(\dot{x},x,t)$ of the form

$L^{(n)}_{null}(\dot{x},x,t)=B_{n}(x,t)\dot{x}+\left[xC(x,t)\right]_{n}+f(t).$
(15)

Proof: Starting from the base case, i.e., the first harmonic,
$L^{(1)}_{null}=L_{null}+\frac{dB(x,t)}{dt}$, it follows by induction that
$L^{(n)}_{null}=L^{(n-1)}_{null}+\frac{dB_{n-1}(x,t)}{dt}$. Then the
application of the null condition for the higher harmonics, i.e., Eq. (12),
proves equation Eq. (15).

Again, if $B(x,t)$ is a linear function of $x$, then the null-Lagrangian
presented in [26,27] is reproduced. Let $B(x,t)=f_{1}(t)x+f_{2}(t)t+f_{3}(t)$
and the following harmonics of the null-Lagrangian in Eq. (9) are obtained

${L}^{(1)}_{null1}(\dot{x},x,t)=\left[f_{1}(t)(x+1)+f_{2}(t)t+f_{3}(t)\right]\dot{x}$
$\hskip
18.06749pt+\dot{f}_{1}(t)(\frac{x}{2}+1)x+\left[\dot{f}_{2}(t)t+f_{2}(t)+\dot{f}_{3}(t)\right](x+1)+f_{4}(t)\
,$ (16)

and the highest harmonics

${L}^{(2)}_{null1}(\dot{x},x,t)=\left[f_{1}(t)(x+2)+f_{2}(t)t+f_{3}(t)\right]\dot{x}$
$\hskip
18.06749pt+\dot{f}_{1}(t)(\frac{x^{2}}{2}+2x+1)+\left[\dot{f}_{2}(t)t+f_{2}(t)+\dot{f}_{3}(t)\right](x+2)+f_{4}(t).$
(17)

Let $B(x,t)=f_{1}(t)sin(x)+f_{2}(t)e^{x}t+f_{3}(t)$. Then, the corresponding
null Lagrangian turns out to be

${L}^{(1)}_{null3}(\dot{x},x,t)=\left[f_{1}(t)[sin(x)+cos(x)]+2f_{2}(t)e^{x}t+f_{3}(t)\right]\dot{x}$
$\hskip
18.06749pt\dot{f}_{1}(t)[sin(x)-cos(x)]+2\dot{f}_{2}(t)e^{x}t+2f_{2}(t)e^{x}+\dot{f}_{3}(t)(x+1)+f_{4}(t),\
$ (18)

and the higher harmonics can be computed according to Eq. (15). It turns out
that the above strategy can also be employed to generate a family of non-
standard null Lagrangians as presented below.

###  3.2 Non-standard null Lagrangians

The key idea to generate a non-standard null-Lagrangian is to make a judicious
choice of $B(x,t)$ as a fraction defined as $B(x,t)=f(t)/h(x,t)$ with $h(t)$
being a non-vanishing differentiable and integrable function. Once a choice on
$B(x,t)$ is made, the corresponding non-standard null Lagrangian can be
obtained by following the procedure described above in Proposition 1. Let a
generating function $B(x,t)$ be expressed as

$B(x,t)=\frac{f_{1}(t)}{f_{2}(t)x+f_{3}(t)t+f_{4}(t)},$ (19)

where $f(t)$’s are differential functions of time and
$f_{2}(t)x+f_{3}(t)t+f_{4}(t)\neq 0$. Then, the corresponding non-standard
null Lagrangian can be obtained by using Eqs. (7) and (8) as

$L_{nsn}(\dot{x},x,t)=\frac{f_{1}(t)\dot{x}}{f_{2}(t)x+f_{3}(t)t+f_{4}(t)}$
$\hskip
18.06749pt+\frac{h_{2}(t)}{f_{2}^{2}}\left[ln|f_{2}x+f_{3}t+f_{4}|+\frac{f_{3}t+f_{4}}{{f_{3}(t)x+f_{3}(t)t+f_{4}(t)}}\right]$
$\hskip
18.06749pt-\frac{h_{3}(t)t+h_{4}(t)}{f_{2}\left[f_{2}(t)x+f_{3}(t)t+f_{4}(t)\right]}+f(t),$
(20)

where $h_{2}(t):=[\dot{f}_{1}(t)f_{2}(t)-f_{1}(t)\dot{f}_{2}(t)]$,
$h_{3}(t):=[\dot{f}_{1}(t)f_{3}(t)-f_{1}(t)\dot{f}_{3}(t)]-f_{1}(t)f_{3}(t)$,
and $h_{4}(t):=[\dot{f}_{1}(t)f_{4}(t)-f_{1}(t)\dot{f}_{4}(t)]$. It is worth
mentioning that any other suitably chosen differentiable and integrable
function $B(x,t)$ of the form in Eq. (19) can be used as a generating function
to construct its corrensponding non-standard null Lagrangian.

Again, the harmonics of any non-standard null-Lagrangian can be obtained using
the procedure presented in Proposition 2. For instance, the first harmonics of
the above non-standard null Lagrangian in Eq. (20) can be calculated as

$L^{(1)}_{nsn}(\dot{x},x,t)=\left[\frac{f_{1}(t)}{f_{2}(t)x+f_{3}(t)t+f_{4}(t)}-\frac{f_{1}(t)f_{2}(t)}{[f_{2}(t)x+f_{3}(t)t+f_{4}(t)]^{2}}\right]\dot{x}$
$\hskip
18.06749pt+\frac{h_{2}(t)}{f_{2}^{2}}\left[ln|f_{2}x+f_{3}t+f_{4}|+\frac{f_{3}t+f_{4}}{{f_{3}(t)x+f_{3}(t)t+f_{4}(t)}}\right]$
$\hskip
18.06749pt-\frac{h_{3}(t)t+h_{4}(t)}{f_{2}\left[f_{2}(t)x+f_{3}(t)t+f_{4}(t)\right]}+\frac{h_{2}(t)x+h_{3}(t)t+h_{4}(t)}{\left[f_{2}(t)x+f_{3}(t)t+f_{4}(t)\right]^{2}}+f(t).$
(21)

After the exploration of the above strategy for constructing both standard and
non-standard null Lagrangians, it is natural to seek for similar strategy for
constructing non-standard Lagrangians. In the following section, it is shown
that indeed a simple strategy for constructing non-standard Lagrangians from
the above null-Lagrangians can be adopted.

##  4 From null to non-standard Lagrangians

The construction of non-standard Lagrangians follows from the availability of
the smooth and invertible Lagrangians defined as $C^{\infty}$ functions on the
domain $\mathcal{U}\subset{T\mathcal{Q}}$. After a desirable Lagrangian,
$L(\dot{x},x,t)$, is identified, a non-standard Lagrangian,
$\mathcal{L}(\dot{x},x,t)$ may be constructed by function composition as
$\mathcal{L}(\dot{x},x,t)=(F\circ L)(\dot{x},x,t)=F(L(\dot{x},x,t))$ over the
same domain $\mathcal{U}\subset{T\mathcal{Q}}$. The following proposition
encodes this construction of a non-standard Lagrangian and the derivation of
its corresponding evolution equation.

Proposition 3: Let
$L(\dot{x},x,t):\mathcal{U}\subset{T\mathcal{Q}}\rightarrow\mathbf{R}$ be an
invertible and $C^{\infty}$ function. Then a non-standard Lagrangian
$\mathcal{L}:\mathcal{U}\subset{T\mathcal{Q}}\rightarrow\mathbf{R}$ can be
defined by using at least a $C^{2}$ function
$F:\mathbf{R}\rightarrow\mathbf{R}$ as $\mathcal{L}(\dot{x},x,t)=(F\circ
L)(\dot{x},x,t))=F(L(\dot{x},x,t))$ and the corresponding equation of motion
is given by

$p_{L}\left[\frac{d^{2}F}{dL^{2}}\right]\frac{dL}{dt}+\left(\dot{p}_{L}-f_{L}\right)\frac{dF}{dL}=0,$
(22)

where $p_{L}:=\partial L/\partial\dot{x}$, $\dot{p}_{L}=dp_{L}/dt$ and
$f_{L}:={\partial{L}}/{\partial{{x}}}$.

Proof: First of all, the composition is well defined since
$L(\dot{x},x,t):\mathcal{U}\subset{T\mathcal{Q}}\rightarrow\mathbf{R}$ is
invertible and $F:\mathbf{R}\rightarrow\mathbf{R}$ is at least a $C^{2}$
function and thereby implying $\mathcal{L}:=(F\circ
L):\mathcal{U}\subset{T\mathcal{Q}}\rightarrow\mathbf{R}$ is at least a
$C^{2}$ function, which can therefore be used to define the action functional
such as Eq. (2) for extremization. Therefore, the derivation of Eq. (22)
follows directly from $\hat{EL}[\mathcal{L}=F(L(\dot{x},x,t))]=0$.

It should be noted that no assumption on Lagrangians
$L(\dot{x},x,t)\in\mathcal{F}(\mathcal{U}\subset{T\mathcal{Q}})$ being null is
made in the above proposition. As a simple check, the equation of motion for
$\mathcal{L}=F(L(\dot{x},x,t))=L(\dot{x},x,t)$ reproduces the expected
$\dot{p}_{L}-f_{L}=0$. Some interesting forms of
$\mathcal{L}=F(L(\dot{x},x,t))$ are $F(L(\dot{x},x,t))=exp{(L)}$ [11],
$F(L(\dot{x},x,t))=ln{(L)}$ and $F(L(\dot{x},x,t))={L}^{-1}$. Upon inserting
these forms in Eq. (22), the respective equations of motion are obtained as
${dL}/{dt}+({\dot{p}_{L}-f_{L}})/{p_{L}}=0$,
${dL}/{dt}-[({\dot{p}_{L}-f_{L}})/{p_{L}}]L=0$ and
${dL}/{dt}-[({\dot{p}_{L}-f_{L}})/{2p_{L}}]L=0$.

Next, it is interesting to note that Proposition 3 simplifies considerably for
functions of null Lagrangians. The use of null Lagrangians in Proposition 3
eliminates the second term in Eq. (22), for $\dot{p}_{L_{null}}=f_{L_{null}}$
for all null Lagrangians. In particular, inserting
$\mathcal{L}=F(L_{null}(\dot{x},x,t))={L}_{null}(\dot{x},x,t)$ in Eq. (22)
gives the expected $\hat{EL}[\mathcal{L}=L_{null}]=0$ validating that no
equation of motion is obtained from $\mathcal{L}=L_{null}$. However, most
importantly, the final equation of motion derived from a function of a null
Lagrangian turns out to be independent of the form of the function provided
that $p_{L}\left[\frac{d^{2}F}{dL_{null}^{2}}\right]\neq 0$ for a given
function $L_{null}$, which is encoded in the following Corollary.

Corollary 1: Let $\mathcal{L}:=(F\circ
L_{null}):\mathcal{U}\subset{T\mathcal{Q}}\rightarrow\mathbf{R}$ be a set of
$C^{\infty}$ functions composed from
${L_{null}}:\mathcal{U}\subset{T\mathcal{Q}}\rightarrow\mathbf{R}$ and
$F:\mathbf{R}\rightarrow\mathbf{R}$ Ṫhen, for any permissible form of function
$\mathcal{L}=(F\circ L_{null})(\dot{x},x,t)$ satisfying
$p_{L_{null}}\left[\frac{d^{2}F}{dL_{null}^{2}}\right]\neq 0$, the
corresponding equation of motion is given by

$\frac{d}{dt}\left[L_{null}\right]=0.$ (23)

The simplicity of the derivation of an equation of motion from Eq. (23) is
rather remarkable, for it generates an equation of motion describing a
dynamical system simply from the total time derivative of a permissible null
Lagrangian defined over a suitable domain. In other words, there exists a
dynamical system corresponding to every permissible null Lagrangian over a
domain of interest. Furthermore, the form of the corresponding equation of
motion is independent of the functional form defined over different domain for
each functional. For example, the simplified equation of motion for
$F[L_{null}(\dot{x},x,t)]=exp{(L_{null})}$ hinges upon the assumption that
$p_{L_{null}}\neq 0$ and $L_{null}$ is finite (but can be zero at some points)
over the chosen domain. On the contrary, both $p_{L_{null}}\neq 0$ and
${L_{null}}\neq 0$ must be satisfied over the same domain for
$F[L_{null}(\dot{x},x,t)]=ln{(L_{null})}$.

At this point, it is desirable to explicitly express the equation of motion
for a null Lagrangian in the form presented in the previous sections.
Therefore, an invertible null Lagrangian,
$L(\dot{x},x,t)=B(x,t)\dot{x}+C(x,t)x+f(t)$ with its null condition Eq. (7)
over a domain $\mathcal{U}\subset{T\mathcal{Q}}$, generates the following
equation of motion:

$B(x,t)\ddot{x}+\left[B^{{}^{\prime}}(x,t)\dot{x}+2\dot{B}(x,t)\right]\dot{x}+\dot{C}(x,t)x+\dot{f}(t)=0,$
(24)

where primes and dots are defined respectively as $B^{{}^{\prime}}:=\partial
B(x,t)/\partial x$ and $\dot{B}:=\partial B(x,t)/\partial t$.

Similarly, the equation of motion for $n$-th harmonics of an invertible null
Lagrangian, $L_{null}(\dot{x},x,t)=B(x,t)\dot{x}+C(x,t)x+f(t)$ over some
domain $\mathcal{U}\subset{T\mathcal{Q}}$, can be computed from

$\frac{dL^{(n)}_{null}}{dt}=\frac{dL^{(n-1)}_{null}}{dt}+\frac{d^{2}B_{n-1}(x,t)}{dt^{2}}=0\
,$ (25)

where again $L^{(n-1)}_{null}$ and $B_{n-1}(x,t)$ are defined in Eqs. (15) and
(13) respectively.

Now, after constructing the non-standard Lagrangians from null Lagrangians and
the simplified method for obtaining the corresponding equation of motion, it
is imperative that the above formulation be applied to explore the range of
allowed physical systems. Therefore, a few specific examples are presented
below to apply these discoveries for a range of allowed dynamical systems.

##  5 Applications to dynamical systems

###  5.1 Newton’s law of inertia

The general form of the equation of motion (see Eq. 24) resulting from Eq.
(23) for the null Lagrangian
$L_{null}(\dot{x},x,t)=B(x,t)\dot{x}+C(x,t)x+f(t)$ may imply that the equation
represents a broad variety of dynamical systems. However, it must be kept in
mind that there is also the null condition that can be written as

$\left(\frac{\partial B(x,t)}{\partial
t}\right)=\frac{\partial[xC(x,t)]}{\partial x}\ ,$ (26)

and that this condition imposes stringent constraints on admissible functions
$B(x,t)$ and $C(x,t)$. As a result, only certain dynamical systems may have
null Lagrangians that allow deriving their equations of motion directly from
Eq. (23), which is a necessary condition for obtaining the equation of motion
directly from a given null Lagrangian. The existence of this condition is a
new phenomenon in the calculus of variations.

In the simplest case when $B(x,t)=c_{1}$ = const, $C(x,t)=c_{2}$ = const, and
$f(t)=c_{3}$ = const, Eq. (24) reduces to $\ddot{x}=0$, which is the Newton
law of inertia. Then, $L_{null}(\dot{x})=c_{1}\dot{x}$, which is the simplest
null Lagrangian. It is easy to see that substitution of this Lagrangian into
Eq. (23) gives the required equation of motion. Thus, the law of inertia can
be derived by using the null Lagrangian, which is a new result of this paper.

###  5.2 Dissipative systems with constant coefficients

To indentify dynamical systems, whose equations of motion can be obtained from
the null Lgrangians by using Eq. (23), we consider the following equation

$\ddot{x}+\alpha_{o}\dot{x}^{2}+\beta_{o}\dot{x}+\gamma_{o}x=0\ ,$ (27)

where $\alpha_{o}$ = const, $\beta_{o}$ = const and $\gamma_{o}$ = const.
Comparison of this equation to Eq. (24) allows finding the functions $B(x,t)$
and $C(x,t)$ and obtaining the corresponding null Lagrangian.

However, if the coefficients in Eq. (27) are constant, we obtain

$B(x,t)=B_{o}e^{\alpha_{o}x+\beta_{o}t/2}\ ,$ (28)

and

$C(x,t)=2B_{o}\left(\frac{\gamma_{o}}{\beta_{o}}\right)e^{\alpha_{o}x+\beta_{o}t/2}\
.$ (29)

Moreover, the null condition gives
$(1+\alpha_{o}x)\gamma_{o}=\beta_{o}^{2}/4$. Since the coefficients are
constant, the dependence on $x$ must be eliminated by either $\alpha_{o}=0$ or
$\alpha_{o}\neq 0$ but $\beta_{o}=\gamma_{o}=0$. This shows that there is no
null Lagrangian for the equation of motion $\ddot{x}+\gamma_{o}x=0$, which for
$\gamma_{o}$ being a spring constant represents a harmonic oscillator. Our
method of deriving equations of motion from null Lagrangians seems to be
restricted to dissipative systems, with the law of inertia being an exception.

We now present the resulting ODEs for the cases when null Lagrangians exist
and can be used used to derived them. With $\alpha_{o}=0$ and
$\gamma_{o}=\beta_{o}^{2}/4$, Eq. (27) reduces to

$\ddot{x}+\beta_{o}\dot{x}+\frac{1}{4}\beta_{o}^{2}x=0\ ,$ (30)

and its null Lagrangian is

$L_{null}(\dot{x},x,t)=\left(\dot{x}+\frac{1}{2}x\right)B_{o}e^{\beta_{o}t/2}\
.$ (31)

It is easy to verify that $L_{null}(\dot{x},x,t)$ is the null Lagrangian and
that this Lagrangian gives the equation of motion (see Eq. 29) after it is
substituted into Eq. (23). The obtained equation of motion describes a damped
harmonic oscillator in which the damping term and the natural frequency of
this system are related to each other.

In case $\alpha_{o}\neq 0$ and $\beta_{o}=\gamma_{o}=0$, the resulting
equation of motion is

$\ddot{x}+\alpha_{o}\dot{x}^{2}=0\ ,$ (32)

and the null Lagrangian that gives this equation is

$L_{null}(\dot{x},x,t)=B_{o}\dot{x}e^{\alpha_{o}x}\ .$ (33)

In this case the equation of motion describes a dynamical system with the
quadratic damping and its coefficient $\alpha_{o}$ being arbitrary.

###  5.3 Dissipative systems with time-dependent coefficients

Let us generalize Eq. (27) and consider the following equation

$\ddot{x}+\alpha_{1}(t)\dot{x}^{2}+\beta_{1}(t)\dot{x}+\gamma_{1}(t)x=0\ ,$
(34)

where $\alpha_{1}(t)$, $\beta_{1}(t)$ and $\gamma_{1}(t)$ are given functions
of time. By comparing this equation to Eq. (24), the functions $B(x,t)$ and
$C(x,t)$ are obtained

$B(x,t)=B_{o}e^{\alpha_{1}(t)x+I_{\beta}(t)}\ ,$ (35)

where

$I_{\beta}(t)=\frac{1}{2}\int^{t}\beta_{1}({\tilde{t}})d\tilde{t}\ ,$ (36)

and

$C(x,t)=B_{o}\int^{t}\gamma_{1}({\tilde{t}})e^{\alpha_{1}({\tilde{t}})x+I_{\beta}({\tilde{t}})}d\tilde{t}\
.$ (37)

Then, the null condition is

$\left[\dot{\alpha}_{1}(t)x+\frac{1}{2}\beta_{1}(t)\right]e^{\alpha_{1}(t)x+I_{\beta}(t)}=\int^{t}\gamma_{1}({\tilde{t}})e^{\alpha_{1}({\tilde{t}})x+I_{\beta}({\tilde{t}})}d\tilde{t}$
$\hskip 148.15372pt\
+x\int^{t}\alpha_{1}({\tilde{t}})\gamma_{1}({\tilde{t}})e^{\alpha_{1}({\tilde{t}})x+I_{\beta}({\tilde{t}})}d\tilde{t}\
.$ (38)

Since the coefficients are only functions of $t$, the dependence on $x$ must
be eliminated. This can be achieved by taking either $\alpha_{1}(t)=0$ or
$\alpha_{1}(t)\neq 0$ but $\beta_{1}(t)=\gamma_{1}(t)=0$. Both cases are
similar to those considered in Section 5.1 but the resulting equations of
motion and the null Lagrangians are more general, thus, we present them now.

Having obtained $B(x,t)$ and $C(x,t)$, we take $\alpha_{1}(t)=0$ and find the
following null Lagrangian

$L_{null}(\dot{x},x,t)=B_{o}\left[\dot{x}e^{I_{\beta}(t)}+x\int^{t}\gamma_{1}({\tilde{t}})e^{I_{\beta}({\tilde{t}})}d\tilde{t}\right]\
.$ (39)

Substituting this Lagrangian into Eq. (23), we obtain

$\ddot{x}+\beta_{1}(t)\dot{x}+\gamma_{1}(t)x=0\ ,$ (40)

which is the correct equation of motion. However, the functions $\beta_{1}(t)$
and $\gamma_{1}(t)$ are related to each other through the null condition (see
Eq. 38) given by

$\frac{1}{2}\beta_{1}(t)e^{I_{\beta}(t)}=\int^{t}\gamma_{1}({\tilde{t}})e^{I_{\beta}({\tilde{t}})}d\tilde{t}\
.$ (41)

This equation generalizes that given by Eq. (30) and describes a damped
harmonic oscillator, whose coefficients are functions of $t$ and they are
dependent.

Now, let us consider the second case of $\beta_{1}(t)=\gamma_{1}(t)=0$, which
reduces the null condition given by Eq. (38) to $\dot{\alpha}_{1}(t)=0$ or
$\alpha_{1}(t)=\alpha_{o}$ = const. This is the same case as considered in
Section 5.1 and the resulting equation of motion and its null Lagrangian are
given by Eqs (32) and (33), respectively. The obtained result shows that the
null Lagrangian exists only for the equation of motion with the quadratic
damping when the damping coefficient is constant.

###  5.4 Dissipative systems with displacement-dependent coefficients

We may also generalize Eq. (27) and consider the following equation

$\ddot{x}+\alpha_{2}(x)\dot{x}^{2}+\beta_{2}(x)\dot{x}+\gamma_{2}(x)x=0\ ,$
(42)

where $\alpha_{2}(x)$, $\beta_{2}(x)$ and $\gamma_{2}(x)$ are given functions
of displacement. To find $B(x,t)$ and $C(x,t)$, we compare the above equation
to Eq. (24), and obtain

$B(x,t)=B_{o}e^{I_{\alpha}(x)+\beta_{2}(x)t/2}\ ,$ (43)

where

$I_{\alpha}(x)=\int^{t}\alpha_{2}({\tilde{x}})d\tilde{x}\ ,$ (44)

and

$C(x,t)=2B_{o}\frac{\gamma_{2}(x)}{\beta_{2}(x)}e^{I_{\alpha}(x)+\beta_{2}(x)t/2}\
.$ (45)

Then, the null condition can be written as

$\frac{1}{4}\frac{\beta_{2}^{2}(x)}{\gamma_{2}(x)}=1+\left[\frac{\gamma^{\prime}_{2}(x)}{\gamma_{2}(x)}-\frac{\beta^{\prime}_{2}(x)}{\beta_{2}(x)}+\alpha_{2}(x)+\frac{1}{2}\beta^{\prime}_{2}(x)t\right]\
.$ (46)

Since the coefficients depend only on $x$, the dependence on $t$ must be
eliminated by taking $\beta^{\prime}_{2}(x)=0$, which gives
$\beta_{2}(x)=\beta_{o}$ = const. As a result, the null condition reduces to

$\frac{1}{4}\beta_{o}^{2}=x\gamma^{\prime}_{2}(x)+\gamma_{2}(x)\left[1+\alpha_{2}(x)x\right]\
.$ (47)

The most general case when the above null condition is satisfied requires
$\alpha_{2}(x)$, $\beta_{2}(x)=\beta_{o}$ and $\gamma_{2}(x)$. Then, the
resulting null Lagrangian is

$L_{null}(\dot{x},x,t)=B_{o}\left[\dot{x}+2x\frac{\gamma_{2}(x)}{\beta_{o}}\right]e^{I_{\alpha}(x)+\beta_{o}t/2}\
,$ (48)

which gives the following equation of motion

$\ddot{x}+\alpha_{2}(x)\dot{x}^{2}+\beta_{o}\dot{x}+\gamma_{2}(x)x=0\ .$ (49)

In this equation of motion, the functions $\alpha_{2}(x)$ and $\gamma_{2}(x)$
are related to each other by the null condition (see Eq. 47), which also
depends on the damping coefficient $\beta_{o}$.

There are also three special cases: $\alpha_{2}(x)=\alpha_{o}/x$,
$\alpha_{2}(x)=0$ and $\beta_{o}=0$. In the first case, we find

$\gamma_{2}(x)=\frac{1}{4}\frac{\beta_{o}^{2}}{1+\alpha_{o}}+\tilde{c}_{1}x^{-(1+\alpha_{o})}\
,$ (50)

where $\tilde{c}_{1}$ is an integration constant. The null Lagrangian is the
same as that given by Eq. (48) but with the replacement
$\alpha_{2}(x)=\alpha_{o}/x$. The same replacement in Eq. (49) gives the
required equation of motion. In the second case, $\alpha_{2}(x)=0$, and the
solution to the null condition can be found and written as

$\gamma_{2}(x)=\frac{\tilde{c}_{2}}{x}+\frac{1}{4}\beta_{o}^{2}\ .$ (51)

where $\tilde{c}_{2}$ is an integration constant. It is seen that
$\gamma_{2}(x)$ must have a special dependence on $x$ and on the coefiicient
$\beta_{o}$. The null Lagrangian and the resulting equation of motion are
given by Eqs (48) and (49), respectively, by taking $\alpha_{2}(x)=0$.

The third case requires that $\beta_{o}=0$, which gives

$B(x,t)=B_{o}e^{I_{\alpha}(x)}\ ,$ (52)

and

$C(x,t)=B_{o}t\gamma_{2}(x)e^{I_{\alpha}(x)}\ .$ (53)

Then, the null Lagrangian becomes

$L_{null}(\dot{x},x,t)=B_{o}\left[\dot{x}+xt\gamma_{2}(x)\right]e^{I_{\alpha}(x)}\
,$ (54)

and the equation of motion is

$\ddot{x}+\alpha_{2}(x)\dot{x}^{2}+\gamma_{2}(x)x=0\ .$ (55)

Again, the functions $\alpha_{2}(x)$ and $\gamma_{2}(x)$ are not arbitrary but
instead they must satisfy the following condition

$\gamma_{2}(x)=\frac{\tilde{c}_{3}}{x}e^{I_{\alpha}(x)}\ ,$ (56)

where $\tilde{c}_{3}$ is an integration constant.

###  5.5 Comparison to other Lagrangians

The presented results show that null Lagrangians can be obtained for the law
of inertia and some dynamical systems. An interesting result is that, except
the first Newton law, these systems must be dissipative, which suggests a
novel role of null Lagrangians in classical dynamics of dissipative systems.
For some systems considered in this paper, standard and nonstandard
Lagrangians were already obtained, so now we compare those Lagrangians to the
null Lagrangians derived in this paper.

For the law of inertia, $\ddot{x}=0$, the standard [1-4] and non-standard [28]
Lagrangians are given by

$L_{sd}(\dot{x})=\frac{1}{2}\dot{x}^{2}\ ,$ (57)

and

$L_{nsd}[\dot{x},x,t]=\frac{1}{C_{1}(a_{o}t+v_{o})^{2}[(a_{o}t+v_{o})\dot{x}-a_{o}x+C_{2}]}\
,$ (58)

where $C_{1}$ and $C_{2}$ being constants of integration, and $v_{o}$ and
$a_{o}$ being specified by the initial conditions for solving the auxiliary
differential equation [28]. Both Lagrangians, when substituted into the E-L
equation, give the law of inertia.

A new result of this paper is that the null Lagrangian

$L_{null}(\dot{x})=c_{1}\dot{x}\ ,$ (59)

where $c_{1}$ is a constant, gives also the law of inertia after it is
substituted into Eq. (23). Thus, the law of inertia can be derived by using
either the standard or non-standard or null Lagrangian. Among these three
Lagrangians, the simplest one is the null Lagrangian and the most complicated
one is the non-standard Lagrangian, yet all of them give the same law of
inertia. The advantage of the Lagrangian formulation presented in this paper
is that the law of inertia is derived from the null Lagrangian by simply
taking its time derivative.

The equation of motion with the quadratic damping term,
$\ddot{x}+\alpha_{o}\dot{x}^{2}=0$, was also previously studied [7,8], and the
following standard and non-standard Lagrangians were obtained

$L_{sd}(\dot{x})=\frac{1}{2}\dot{x}^{2}e^{2\alpha_{o}x}\ ,$ (60)

and

$L_{nsd}[\dot{x},x,t]=\frac{1}{\dot{x}e^{\alpha_{o}x}+1}\ .$ (61)

Both Lagrangians, when substituted into the E-L equation, give the equation of
motion. The same equation of motion is obtained when the null Lagrangian

$L_{null}(\dot{x})=c_{2}\dot{x}e^{\alpha_{o}x}\ ,$ (62)

where $c_{2}$ is a constant, is substituted into Eq. (23). Comparison of the
above Lagrangians shows that each one of them contains $\dot{x}$ and
$e^{\alpha_{o}x}$, and that their forms are similar.

Finally, we consider the equation of motion for a damped oscillator,
$\ddot{x}+\beta_{o}\dot{x}+\frac{1}{4}\beta^{2}_{o}x=0$, whose standard and
non-standard Lagrangians [7,8] are

$L_{sd}(\dot{x},t)=\frac{1}{2}\left(\dot{x}^{2}-\frac{1}{4}\beta_{o}^{2}x^{2}\right)e^{\beta_{o}t}\
,$ (63)

and

$L_{nsd}[\dot{x},x,t]=\frac{e^{-\beta_{o}x/2}}{\dot{x}+\frac{1}{2}\beta_{o}x}\
.$ (64)

The null Lagrangian for the equation of motion is

$L_{null}(\dot{x},x,t)=c_{3}\left(\dot{x}+\frac{1}{2}\beta_{o}x\right)e^{\beta_{o}t/2}\
.$ (65)

where $c_{2}$ is a constant. Comparison of $L_{nsd}(\dot{x},x,t)$ and
$L_{null}(\dot{x},x,t)$ shows that the non-standard Lagrangian is exactly the
inverse of the null Lagrangian, if $c_{3}=1$.

##  6 Conclusions

General null Lagrangians and their higher harmonics are constructed for
dynamical systems with one degree of freedom. Since the null Lagrangians
satisfy identically the E-L equation, they cannot be directly used to obtain
equations of motion. However, as the results of this paper show, the derived
null Lagrangians can be used to construct non-standard Lagrangians that do
give equations of motion. The main result of this paper is the following
condition

$\frac{d}{dt}\left[L_{null}\right]=0,$ (66)

which gives an equation of motion for any null Lagrangian. Thus, the condition
plays the same role for null Lagrangians as the E-L equation plays for
standard or non-standard Lagrangians.

The presented results show how to determine a null Lagrangian for a given
dynamical system and then how to use this Lagrangian to obtain the resulting
equation of motion. Specific applications of the method to general dynamical
systems show that the method preferentially works for dissipative systems but
it also can be used to some conservative systems such as the law of inertia.
The results of this paper demonstrate a novel role of null Lagrangians in
classical dynamics.

##  7 References

## References

  * [1] J.L. Lagrange, Analytical Mechanics (Springer, Netherlands, 1997).
  * [2] H. Goldstein, C.P. Poole, J.L. Safko, Classical Mechanics (3rd Edition) (Addison-Wesley, San Francisco, CA, 2002)
  * [3] J.V. José, E.J. Saletan, Classical Dynamics, A Contemporary Approach (Cambridge Univ. Press, Cambridge, 2002)
  * [4] N.A. Daughty, Lagrangian Interactions (Addison-Wesley Publ. Comp., Inc., Sydney, 1990)
  * [5] Lopuszanski, J., The Inverse Variational Problems in Classical Mechanics (World Scientific, Singapore, 1999).
  * [6] M.C. Nucci and P.G.L. Leach, J. Math. Phys., 48, 123510, 2007
  * [7] Z.E. Musielak, J. Phys. A Math. Theor., 41, 055205, 2008
  * [8] J.L. Cieśliński and T. Nikiciuk, J. Phys. A Math. Theor., 43, 175205, 2010
  * [9] Z.E. Musielak, Chaos, Solitons Fractals, 42, 2640, 2009
  * [10] A. Saha and B. Talukdar, Rep. Math. Phys., 73, 299–309, 2014
  * [11] R.A. El-Nabulsi, Int. J. Theor. Phys., 56, 1159, 2017
  * [12] N. Davachi and Z.E. Musielak, J. Undergrad. Rep. Phys., 29, 100004, 2019
  * [13] P.J. Olver, Applications of Lie Groups to Differential Equations (Springer-Verlag, New York, 1993)
  * [14] P.J. Olver and J. Sivaloganathan, Nonlinearity 1, 389, 1989
  * [15] M. Crampin and D.J. Saunders, Diff. Geom. Appl., 22, 131–146, 2005
  * [16] D.J. Saunders, Math. Slovaca, 65, 1063–1078, 2015
  * [17] D. Krupka, O. Krupkova and D. Saunders, Int. J. Geom. Meth. Mod. Phys., 7, 631, 2010
  * [18] R. Vitolo, Diff. Geom. Appl., 10, 293-305, 1999
  * [19] D. Krupka and J. Musilova, Diff. Geom. Appl., 9, 225, 1998
  * [20] D.R. Grigore, J. Phys. A 28, 2921, 1995
  * [21] D.R. Anderson, D.E. Carlson and J. Fried, Elasticity, 56, 35, 1999
  * [22] G. Saccomandi and R. Vitolo, J. Math. Sciences, 136, 4470, 2006
  * [23] J.-M. Levy-Leblond, Comm. Math. Phys. 12, 64, 1969
  * [24] Z. E. Musielak and T. B. Watson, Phys. Let. A 384, 126642, 2020
  * [25] Z.E. Musielak, T.B. Watson, Phys. Let. A, 384, 126838, 2020
  * [26] Z.E. Musielak, L.C. Vestal, B.D. Tran, T.B. Watson, Physics, 2, 425, 2020
  * [27] L.C. Vestal, Z.E. Musielak, Physics, 3, 449, 2021
  * [28] Z.E. Musielak, Physics, 3, 903, 2021
  * [29] A.K. Halder, A. Palithanasis and P.G.L. Leach, Symmetry, 10, 744 (21 pages), 2018
  * [30] S. Hojman, J. Phys. A: Math. Gen., 17, 2399, 1984
  * [31] S.A. Hojman, J. Phys. A: Math. Gen., 25, L291, 1992
  * [32] W. Sarlet, J. Phys. A: Math. Gen., 16, L229 – L233, 1983
  * [33] G.F. Torres del Castillo, Rev. Mex. Fisica, 60, 129, 2014
  * [34] H. Bateman, Phys. Rev., 38, 815, 1931
  * [35] P.S. Bauer, Proc. Natl. Acad. Sci. USA, 17, 311, 1931
  * [36] P. Caldirola, Nuovo Cim., 18, 393, 1941
  * [37] E. Kanai, Prog. Theor. Phys., 3, 44, 1948
  * [38] H. Dekker, Phys. Rep., 80, 1, 1981
  * [39] B.D. Vujanovic, S.E. Jones, (Eds.), Variational Methods in Nonconservative Phenomena (Academic Press, New York, 1989)
  * [40] F. Riewe, Phys. Rev. E, 53, 1890, 1996
  * [41] F. Riewe, Phys. Rev. E, 55, 3581, 1997
  * [42] D.W. Dreisigmeyer, and P.M. Young, J. Phys. A: Math. Gen., 36, 8297, 2003
  * [43] E.M. Rabei, T.S. Alhalholy, and A.A. Taani, Turk. J. Phys., 28, 213, 2004
  * [44] R.A. El-Nabulsi, Fiz., A14, 289, 2005
  * [45] R.A. El-Nabulsi, App. Math. Lett., 24, 1647, 2011
  * [46] R.A. El-Nabulsi, Anal. Theor. Appl., 30, 1, 2014
  * [47] F.E. Udwadia, H. Cho, J. Appl. Mech., 80, 041023, 2013


#  All Roads Lead to Rome?
Exploring the Invariance of Transformers’ Representations

Yuxin Ren1, Qipeng Guo2 Zhijing Jin3,4 Shauli Ravfogel5
Mrinmaya Sachan4 Bernhard Schölkopf3,4 and Ryan Cotterell4
1Tsinghua University 2Fudan University
3Max Planck Institute for Intelligent Systems, Tübingen, Germany
4ETH Zürich 5Bar-Ilan University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Done during the research internship with Max Planck Institute & ETH Zürich.

###### Abstract

Transformer models bring propelling advances in various NLP tasks, thus
inducing lots of interpretability research on the learned representations of
the models. However, we raise a fundamental question regarding the reliability
of the representations. Specifically, we investigate whether transformers
learn essentially isomorphic representation spaces, or those that are
sensitive to the random seeds in their pretraining process. In this work, we
formulate the Bijection Hypothesis, which suggests the use of bijective
methods to align different models’ representation spaces. We propose a model
based on invertible neural networks, BERT-INN, to learn the bijection more
effectively than other existing bijective methods such as the canonical
correlation analysis (CCA). We show the advantage of BERT-INN both
theoretically and through extensive experiments, and apply it to align the
reproduced BERT embeddings to draw insights that are meaningful to the
interpretability research.111Our code is at
https://github.com/twinkle0331/BERT-similarity.

##  1 Introduction

Transformer models Vaswani et al. (2017) have emerged as a powerful
architecture for natural language processing (NLP) tasks, achieving state-of-
the-art results on a wide range of benchmarks (Devlin et al., 2019; Brown et
al., 2020; OpenAI, 2023, inter alia). However, one issue that has received
less attention is the robustness of these models to random initializations.
Specifically, it is unclear whether Transformer models learn essentially
_isomorphic representation spaces_ , or if they learn representations that are
sensitive to the random seeds in their pretraining process.

Recent studies has shown that small variations in the random seed used for
training can result in significant performance differences between models with
the same architecture (Sellam et al., 2022; Dodge et al., 2020). Understanding
the degree to which the learned representation space is sensitive to random
initialization is important because it has implications for interpretability
research, i.e, understanding _what is encoded_ in those models. Such efforts
can include an analysis of the learned representations of LLMs by probing the
information contained within these representations (Tenney et al., 2019;
Rogers et al., 2020), or by observing the change of neuron activations under
intervention (Ravfogel et al., 2021; Meng et al., 2022; Lasri et al., 2022).
As we illustrate in Figure 1, if a model’s learned representation space is
highly sensitive to initialization, then it may be necessary to run probing
tasks multiple times with different random initializations to obtain more
reliable results. Alternatively, techniques to improve the robustness of the
learned representation space can be applied to make the models more suitable
for probing tasks.

Figure 1: Overview of this work. Specifically, we investigate the similarity
of reproduced BERT representations. Given multiple BERT models with different
random seeds, we align their representation space according to our Bijection
Hypothesis, and show clusters of BERT embeddings.

The representation spaces of models trained with different random
initializations may be isomorphic due to two main reasons: the properties of
the training data or inherent symmetries in the models. For instance,
permutation symmetry, which permits the swapping of neurons in the hidden
layer without altering the loss function’s value, is a typical symmetry
observed in neural networks (Hecht-Nielsen, 1990). However, prior research has
suggested that the permutation symmetry in neural networks may not entirely
capture the underlying invariances in their training dynamics (Ainsworth et
al., 2022).

This paper introduces the Bijection Hypothesis, which posits the existence of
a bijection mapping between the representation spaces of different models. To
investigate this hypothesis, we propose BERT-INN, an invertible neural network
(INN) to align the representation spaces of two BERT models by a set of affine
coupling layers. We show the advantage of our BERT-INN over four other
bijective methods both theoretically and through three different experiments.

Finally, we apply our BERT-INN method to align the embedding spaces of the
reproduced BERTs Sellam et al. (2022), and draw several key findings.
Specifically, we identify that (1) the shallower layers are, in general, more
invariant than the deeper layers; (2) the interactions of neurons (i.e.,
attention weights) are more invariant than the representations; and (3)
finetuning breaks the variance.

In conclusion, our contributions are as follows:

  1. 1.

We are, to the best of our knowledge, the first to highlight the consistency
check of different transformer models’ representation spaces.

  2. 2.

We pose the Bijection Hypothesis, for which we train a BERT-INN model to learn
the bijective function mapping and recover the non-linear transformation.

  3. 3.

We show the effectiveness our BERT-INN model over four other bijective methods
in terms of both theoretical properties and three different empiricial
experiments.

  4. 4.

We align the BERT representations using our BERT-INN method, and reveal
several insights that have meaningful implications for future work on
interpretability research of transformer models.

##  2 Background of LLM Analysis

###  2.1 Large Language Models (LLMs)

The recent success of NLP is driven by LLMs (Radford et al., 2018; Devlin et
al., 2019; Liu et al., 2019; Brown et al., 2020; Ouyang et al., 2022). These
models are pre-trained on massive amount of text data, allowing them to learn
general language representations that can be fine-tuned for specific NLP
tasks. Scaling up the size of language models has been shown to confer a range
of benefits, such as improved performance and sample efficiency. LLMs show
strong performance in few-shot or even zero-shot tasks and a better
generalization of out of distribution tasks.

###  2.2 Interpretability of LLMs

The empirical success of BERT on various NLP domains, encourages researchers
to analyse and understand these architectures (Rogers et al., 2020). Kovaleva
et al. (2019) visualize self-attention modules to probe how tokens in a
sequence interact with each other. Clark et al. (2019) provide in-depth
analysis on the self-attention mechanism, showing that some attention heads
correspond to linguistic notions of syntax and correference. In addition, Zhao
et al. (2020); Merchant et al. (2020) investigate the impacts of downstream
task finetuning for BERT representations. Hewitt and Liang (2019); Pimentel et
al. (2020) stress that controlling for important factors, e.g., probing
classifier’s capacity, is crucial to obtain meaningful probing results.

###  2.3 The Effect of Random Seeds

Sellam et al. (2022) conduct experiments on pre-trained models with different
random seeds for pretraining, and find that the instance-level agreement on
downstream tasks varies by the random seeds, and the gap on out-of-
distribution tasks is more pronounced. Dodge et al. (2020) shows that varying
only the random seed of fine-tuning process leads to substantial performance
gap. The two factors influenced by random seed, training order and weight
initalization, contribute equally to the variance of performance. Zhong et al.
(2021) argue that model predictions are noisy at the instance level. On the
MNLI dataset, even the same architecture with different finetuning seeds will
lead to different predictions on $\approx 8\%$ of the instances, due to under-
specification (D’Amour et al., 2020).

##  3 The Bijection Hypothesis

###  3.1 Representation Spaces of BERTs

Given two BERT models $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ with identical
architecture, but different random initializations, the goal is to evaluate
their similarity through their representation spaces.

We denote the sets of BERT parameters after the pre-training as
$\bm{\theta}_{1}$ and $\bm{\theta}_{2}$, respectively. Their representation
transformation functions are thus $f_{\bm{\theta}_{1}}$ and
$f_{\bm{\theta}_{2}}$, respectively. Since a BERT has 12 transformer layers
Vaswani et al. (2017), it is a hyperparameter to specify the representation
space of which transformer layer we want to analyze. Let us denote the layer
of interest as $l$. Then, the sequence transformation function works as
follows. For any text sequence ${\bm{t}}$ with $k$ tokens, we first get its
token-wise embedding ${\bm{T}}\in\mathbb{R}^{k\times 768}$ at Layer $l$ by
${\bm{T}}=\mathrm{Transformer}_{1,\dots,l}({\bm{t}})=\mathrm{FFN}_{l}(\mathrm{Att}_{l}(\dots\mathrm{FFN}_{1}(\mathrm{Att}_{1}(\mathrm{Emb}({\bm{t}})))))$,
where $\mathrm{FFN}_{i}$ and $\mathrm{Att}_{i}$ denotes the feedforward layer
and the self-attention network at Layer $i$, respectively, and $\mathrm{Emb}$
denotes the embedding layer. Then we follow the practice in Del and Fishel
(2021); Wu et al. (2019); Zhelezniak et al. (2019) to do a mean-pooling
operation to obtain the overall sentence embedding
$\frac{1}{k}\sum_{i=1}^{k}{\bm{T}}_{i,*}$.

For the two transformation functions $f_{\bm{\theta}_{1}}$ and
$f_{\bm{\theta}_{2}}$, we feed a large list of $n$ arbitrary text inputs,
$\bm{D}:=[\bm{t}_{1},\bm{t}_{2},\dots,\bm{t}_{n}]$ to obtain the two sentence
representation spaces:

$\displaystyle\bm{X}$
$\displaystyle:=[f_{\bm{\theta}_{1}}(\bm{t}_{1}),\cdots,f_{\bm{\theta}_{1}}(\bm{t}_{n})]\text{
for }\mathcal{M}_{1}$ (1) $\displaystyle\bm{Y}$
$\displaystyle:=[f_{\bm{\theta}_{2}}(\bm{t}_{1}),\cdots,f_{\bm{\theta}_{2}}(\bm{t}_{n})]\text{
for }\mathcal{M}_{2}~{}.$ (2)

Here, ${\bm{X}},{\bm{Y}}\in\mathbb{R}^{n\times d}$, where $d$ is the dimension
of these embeddings, or the number of features.

###  3.2 Perfect Similarity $=$ Diffeomorphism

We propose the Bijection Hypothesis, which states that two models are
perfectly similar if and only if there exists a diffeomorphism in their
representation spaces. The mapping can be discovered through techniques such
as linear or non-linear projections.

Namely, for the two spaces ${\bm{X}}$ and ${\bm{Y}}$, we learn an invertible
function $g$, such that we minimize the distance metric $\delta$ between the
transformed $g({\bm{X}})$ and the original ${\bm{Y}}$. The similarity of the
two spaces can be quantified by the distance metric, and if
$\delta(g({\bm{X}}),{\bm{Y}})$ is zero, then the two spaces are perfectly
similar.

###  3.3  Reformulating Existing Similarity Indices into Linear Bijective
Functions

We briefly introduce existing methods to align the two representation spaces
before measuring their similarity.

#### Linear Regression.

An intuitive way to learn a bijective function for aligning two representation
spaces is linear regression, which can capture all linear transformations by
minimizing the L2 norm. Namely, we learn a matrix ${\bm{W}}$ which satisfies
the following objective:

$\min_{{\bm{W}}}\delta({\bm{X}}{\bm{W}},{\bm{Y}})=\min_{{\bm{W}}}||{\bm{X}}{\bm{W}}-{\bm{Y}}||_{2}~{}.$

#### Canonical Correlation Analysis (CCA).

CCA projects the two matrices ${\bm{X}}$ and ${\bm{Y}}$ into the subspaces
which maximize the canonical correlation between their projections. The
formulation is as follows:

$\displaystyle\rho_{i}=\max_{\bm{w}_{X}^{i},\bm{w}_{Y}^{i}}$
$\displaystyle\mathrm{corr}({\bm{X}}\bm{w}_{X}^{i},{\bm{Y}}\bm{w}_{Y}^{i})$
(3) for $\displaystyle\
~{}~{}\forall_{j<i}~{}~{}{\bm{X}}\bm{w}_{X}^{i}\perp{\bm{X}}\bm{w}_{X}^{j}~{},$
(4) $\displaystyle\
~{}~{}\forall_{j<i}~{}~{}{\bm{Y}}\bm{w}_{Y}^{i}\perp{\bm{Y}}\bm{w}_{Y}^{j}.$
(5)

Here, the objective is to maximize the correlation between all pairs of
projected subspaces of ${\bm{X}}$ and ${\bm{Y}}$, and the constraint is to
keep the orthogonality among all the subspaces of each original space.

Since the dimension of CCA is equal to $d$, by concating the projection
vectors into matrices ${\bm{W}}_{X}$ and ${\bm{W}}_{Y}$, both ${\bm{X}}$ and
${\bm{Y}}$ are invertible matrices. The transformation matrix from ${\bm{X}}$
to ${\bm{Y}}$ is then equivalent to the product of ${\bm{W}}_{X}$ and the
inverse of ${\bm{W}}_{Y}$:

$\displaystyle{\bm{W}}_{X}{\bm{W}}_{Y}^{-1}~{}.$ (6)

However, since we have to pre-determine the number of canonical correlations
$c$, CCA suffers when the guessed $c$ differs from the optimal one.

#### Noise-Robust Improvements of CCA.

As CCA might suffer from outliers in the two spaces, there are two types of
improvements that are more robust towards outliers or noises in the spaces.

The first variant is singular vector CCA (SVCCA), which performs singular
value decomposition (SVD) to extract directions which capture most of the
variance of the original space, thus removing the noisy dimensions before
running the CCA.

The other method is projection-weighted CCA (PWCCA), which computes a weighted
average over the subspaces learned by CCA:

$\displaystyle\rho_{\mathrm{PWCCA}}$
$\displaystyle=\frac{\sum_{i=1}^{c}\alpha_{i}\rho_{i}}{\sum_{i=1}^{c}\alpha_{i}}$
(7) $\displaystyle\alpha_{i}$
$\displaystyle=\sum_{j}|\langle\bm{p}_{i},\bm{x}_{j}\rangle|$ (8)
$\displaystyle{\bm{p}}_{i}$ $\displaystyle={\bm{X}}\bm{w}_{X}^{i}~{},$ (9)

where ${\bm{x}}_{j}$ is the $j$-th feature of the ${\bm{X}}$ space, namely,
the $j$-th column. The weight $\alpha_{i}$ for each correlation measure
$\rho_{i}$ is calculated as how similar the $i$-th subspace projection
$\bm{p}_{i}$ is to all the features ${\bm{x}}_{j}$ of the original space
${\bm{X}}$.

However, SVCCA and PWCCA are still linear methods and unable to capture non-
linear relationship between ${\bm{X}}$ and ${\bm{Y}}$. Further, Csiszárik et
al. (2021) shows that the low-rank transformation learned by SVD is not as
effective as learning a matching under a task loss and the effectiveness of
learning a matching stems from the non-linear function in a network.

###  3.4 Non-Linear Bijection by INNs

All the linear bijective methods introduced previously do not necessarily suit
BERTs, as BERT learns non-linear feature transformation through the non-linear
activation layers GELU Hendrycks and Gimpel (2016). Between two BERT models,
there might be a complex, non-linear correspondence between their feature
spaces. Hence, we propose the use of INNs to discover bijective mappings
between the two BERTs.

INNs learn a neural network-based _invertible_ transformation from the latent
space $\mathcal{U}$ to the observed space $\mathcal{Z}$. Using it as our
bijective method, we learn an invertible transformation $g(\cdot)$ from
${\bm{X}}$ to ${\bm{Y}}$. Specifically, we optimize the parameters
$\bm{\theta}_{g}$ of the INN function by minimizing the L2 distance $\delta$
between the transformed $g_{\bm{\theta}}({\bm{X}})$ and ${\bm{Y}}$:

$\min_{\bm{\theta}_{g}}\delta(g_{\bm{\theta}_{g}}({\bm{X}}),{\bm{Y}})~{}.$
(10)

Intuitively, the invertibility constraint verifies that the mapping between
the two representation spaces is _isomorphic_ : there is no loss of
information when mapping from ${\bm{X}}$ to ${\bm{Y}}$.

For the implementation of the INN, we draw inspirations from RealNVP (Dinh et
al., 2016) and adapt it into BERT-INN, which stacks $L$ affine coupling layers
to learn the bijective function $g$ to map from the embedding of the text
${\bm{t}}$ by the first BERT model $f_{\bm{\theta}_{1}}(\bm{t})$, to that of
the second BERT model $f_{\bm{\theta}_{2}}(\bm{t})$.

In our BERT-INN, each affine coupling layer $\ell$ takes in an input embedding
${\bm{e}}^{\ell-1}$ of dimension $K$, sets a smaller dimension $k<K$, and
produces the output embedding ${\bm{e}}^{\ell}$ in the following way:

$\displaystyle{\bm{e}}^{0}$ $\displaystyle=f_{\bm{\theta}_{1}}(\bm{t})$ (11)
$\displaystyle{\bm{e}}^{\ell}_{1:k}$ $\displaystyle={\bm{e}}^{\ell-1}_{1:k}$
(12) $\displaystyle{\bm{e}}^{\ell}_{k+1:K}$
$\displaystyle={\bm{e}}^{\ell-1}_{k+1:K}\odot\exp(s^{\ell}({\bm{e}}^{\ell-1}_{1:k}))+t^{\ell}({\bm{e}}^{\ell-1}_{1:k})~{}.$
(13)

In each layer, it keeps the first $k$ dimensions of the input as in Eq. 12,
and transforms the remaining $K-k$ dimensions by a scaling layer $s^{\ell}$
and a translation layer $t^{\ell}$ as in Eq. 13. And the first layer takes in
the embedding ${\bm{e}}^{0}$ from the representations of the first BERT model
as in Eq. 11. The main spirit of these affine coupling layers is that they
form an INN simple enough to invert, but also relies on the remainder of the
input vector in a non-trivial manner.

Finally, the loss objective for the last layer is as follows:

$\displaystyle\mathcal{L}=||{\bm{e}}^{L}-f_{\bm{\theta}_{2}}(\bm{t})||_{2}~{},$
(14)

where we optimize for the L2 distance $||\cdot||_{2}$ between the last layer’s
embedding ${\bm{e}}^{L}$ and the second BERT’s representations
$f_{\bm{\theta}_{2}}(\bm{t})$.

##  4 Comparing the Bijective Methods

In this section, we show the effectiveness of our proposed INN method against
all other bijective methods in terms of the theoretical properties, and
empirical performance across synthetic data and real NLP datasets.

###  4.1 Experimental Setup

#### Reproduced BERT Models

We conduct the experiments using the 25 reproduced BERT models released by
Sellam et al. (2022). The BERT models vary only by random seeds, but adopt the
same training procedure as in the original BERT model (Devlin et al., 2019).

#### Datasets

To get the BERT representations, we run the 25 trained BERT models in the
inference mode, and plug in a variety of datasets and tasks. Specifically, we
choose three diverse tasks from the commonly used GLUE benchmark Wang et al.
(2019): the natural language inference dataset MNLI (Williams et al., 2018),
the sentiment classification dataset SST-2 (Socher et al., 2013), and the
paraphrase detection dataset MRPC (Dolan and Brockett, 2005). These datasets
also cover a variety of domains from news, social media, and movie reviews,
which are the common domains for many NLP datasets. The dataset statistics are
in Table 1.

Dataset | Task | Train | Test | Domain
---|---|---|---|---
MNLI | Inference | 393K | 20K | Mixed
SST-2 | Sentiment | 67K | 1.8K | Movie reviews
MRPC | Paraphrase | 3.7K | 1.7K | News
Table 1: Dataset statistics from the GLUE benchmark Wang et al. (2019).

#### Baselines

We compare against four commonly used existing bijective methods introduced in
Section 3.4, namely linear regression, CCA, SVCCA, and PWCCA. Note that PWCCA
sometimes generates results same as or similar to CCA, because it learns the
mapping using the same principles as the CCA, and then computes a weighted
average over the coefficients, which could generate similar results to CCA
under some conditions.

#### Evaluation Setup

For each method $g$, we evaluate their alignment performance by measuring the
distance between two spaces: the ground-truth target space $\bm{Y}$, versus
the learned space $g(\bm{X})$ transformed from the original $\bm{X}$ with the
alignment method $g(\cdot)$. We adopt the a widely used distance function,
Euclidean distance, also known as the L2 distance, to measure the distance
between the two spaces, namely
$\delta(g(\bm{X}),\bm{Y}):=||\bm{Y}-g(\bm{X})||_{2}$. Since our goal is to
align the two spaces, the smaller the L2 distance, the better, with the best
value being zero. For SVCCA, since it reduces the dimension of the original
spaces, we compute the L2 norms between the dimension-reduced matrices. To
show the effectiveness of our INN method, we randomly select eight model pairs
to report the average of their pair-wise L2 distances using each of the
bijective methods. And after we verify the effectiveness of our INN method, we
apply it on all 25 models to draw overall insights on the similarity of the
model representations.

###  4.2 Advantage of INN in Theoretical Properties

| Transformation Type | Non-Lin. | Noise-Robust
---|---|---|---
Lin. Reg. | $||Q_{Y}^{\text{T}}X||_{\text{F}}^{2}/||X||_{\text{F}}^{2}$ | ✗ | ✗
CCA | $||Q_{Y}^{\text{T}}Q_{X}||_{\text{F}}^{2}/d$ | ✗ | ✗
SVCCA | $||(U_{Y}T_{Y})^{\text{T}}U_{X}T_{X}||_{\text{F}}^{2}/\text{min}(||T_{X}||^{2}_{\text{F}},||T_{Y}||^{2}_{\text{F}})$ | ✗ | ✓
PWCCA | $\sum_{i=1}^{d}\alpha_{i}\rho_{i}/||\mathbf{\alpha}||_{1}$ | ✗ | ✓
INN (Ours) | Any invertible function | ✓ | ✓

Table 2: Theoretical properties of all the alignment methods. We focus on
properties such as whether the method can handle non-linear mapping between
the two spaces (“Non-Lin.”), and whether the method is robust against noise in
the transformation (“Allow Noise”).

We compare the theoretical properties of all the alignment methods in Table 2.
As a result of the nature of the transformation types, we can see that all
previous methods can only handle linear transformations, whereas our INN can
model non-linear, invertible mappings. Moreover, linear regression and CCA are
sensitive towards noise in the embedding space; SVCCA and PWCCA improves upon
CCA by filtering out the noise; and our model is also robust against noises in
that the training objective of INN is to minimize the overall loss across all
data points.

###  4.3 Advantage of INN on Synthetic Data

We also show empirical evidence of the effectiveness of our method. For our
empirical experiment, we want to understand whether INNs are capable of
identifying the correspondence between the two representation spaces. We
cannot do that with raw BERT representations, since we do not know how
invariant the BERT representations are, and also we do not know its form of
bijection, if such invariance exists. Therefore we generate synthetic data
where the correspondence is known, and we measure the ability of INNs to
identify it.

|  L2 Distance $\delta(g(\bm{X}),\bm{Y})$ ($\downarrow$)
---|---
Linear Regression |  1.0882$\pm$0.3867
CCA |  0.3060$\pm$0.1948
SVCCA |  0.1396$\pm$0.1246
PWCCA |  0.3060$\pm$0.1948
INN (Ours) |  0.0059$\pm$0.0037
Table 3:  Empirical performance of all the alignment methods to recover a non-linear, invertible transformation. For L2 distance ($\downarrow$), the smaller, the better, and the best possible L2 distance is zero, which means the learned mapping recovers the ground-truth transformation between the two spaces.  | MNLI | SST2 | MRPC
---|---|---|---
| Train | Test | Train | Test | Train | Test
CCA |  0.1484$\pm$0.0763 |  0.1534$\pm$0.0802 |  0.3064$\pm$0.1791 |  0.2535$\pm$0.1784 |  0.1455$\pm$ 0.0663 |  0.1502$\pm$0.0668
SVCCA |  0.1216$\pm$0.1011 |  0.1180$\pm$0.0989 |  0.2399$\pm$0.2322 |  0.1997$\pm$0.2375 |  0.1033$\pm$0.1136 |  0.0852$\pm$0.0730
PWCCA |  0.0546$\pm$0.0441 |  0.0529$\pm$0.0418 |  0.1095$\pm$0.1214 |  0.0877$\pm$0.1080 |  0.0424$\pm$0.0344 |  0.0391$\pm$0.0288
Lin. Reg. |  0.1003$\pm$0.0799 |  0.1024$\pm$0.0822 |  0.0884$\pm$0.0533 |  0.0896$\pm$0.0623 |  0.0618$\pm$0.0392 |  0.0641$\pm$0.0424
INN (Ours) |  0.0520$\pm$0.0388 |  0.0461$\pm$0.0336 |  0.0563$\pm$0.0315 |  0.0523$\pm$0.0401 |  0.0299$\pm$0.0165 |  0.0310$\pm$0.0180
Sanity Check: |  |  |  |  |  |
Non-Bij. NN |  0.0360$\pm$0.0256 |  0.0371$\pm$0.0272 |  0.0479$\pm$0.0279 |  0.0449$\pm$0.0352 |  0.0247$\pm$0.0145 |  0.0263$\pm$0.0165
Table 4: Alignment performance on MNLI, SST2, and MRPC datasets using CCA,
SVCCA, PWCCA, linear regression (Lin. Reg.), and our INN method. We report the
L2 norm $\delta$ ($\downarrow$) after aligning the sentence space on the
training set (just for a reference) and test set (as the main result). The
training and test samples were randomly selected from the respective datasets,
with a sample size of 1600. For sanity check, we also report the performance
of a non-bijective neural network (Non-Bij. NN), which should lead to the
smallest L2 norm because it has less constraint than INN, although at the
expense of violating the Bijection Hypothesis.

Specifically, we make the first space $\bm{X}$ a real BERT representation
space generated by Sellam et al. (2022). For the second subspace $\bm{Y}$, we
generate it by applying a randomly initialized INN $f$ to the first space
$\bm{X}$, namely $\bm{Y}=f(\bm{X})$ as the ground truth.

Then, without any knowledge of the transformation $f$, we apply all methods to
obtain their corresponding mapping function $g$ given the two spaces $\bm{X}$
and $\bm{Y}$. For each method, we evaluate their alignment performance by
measuring the distance between the learned $g(\bm{Y})$ and ground-truth
$\bm{Y}$ using L2 distance, namely
$\delta(g(\bm{Y}),\bm{X}):=||g(\bm{Y})-\bm{X}||_{2}$. Note that since the
ground truth mapping $f$ gives an L2 distance of zero, when interpreting the
results, the lower L2 distance, the better an alignment method is.

We show the performance of our method compared with all the baselines in Table
3. All baselines (linear regression, CCA, SVCCA, and PWCCA) struggle to learn
a good mapping between the two spaces, with an L2 distance always larger than
0.13, which is quite large. In contrast, our INN method gives a very small L2
distance, 0.0059, which almost recovers the ground-truth transformation. The
results of this experiment echoes with the theoretical properties analyzed
previously in Section 4.2.

###  4.4 Advantage of INN on Real Data’s Representation Similarity

We now move from synthetic data to real NLP datasets. As introduce in Section
4.1, we use three diverse datasets: MNLI (Williams et al., 2018), SST-2
(Socher et al., 2013), and MRPC (Dolan and Brockett, 2005).

We report the alignment results in Table 4. For each method, we use the splits
in the original data, learn the mapping using the training set, (if
applicable,) tune the hyperparameters on the validation set, and finally
report the performance on the test set. For some additional references, we
also report the training set performance (to check how well the method fits
the data), in addition to the test set performance (as the final alignment
performance).

Among all the bijective methods in Table 4, our INN method leads to the
smallest L2 distance on all datasets by a clear margin. We can also see that
the vanilla CCA is usually the weakest, followed by SVCCA. PWCCA and linear
regression learns the bijective mapping relatively well, but still falling
behind our proposed INN method.

###  4.5 Advantage of INN on Real Data’s Behavioral Similarity

The results presented thus far showcase the effective mapping capability of
INNs between the representation spaces of two models. However, it remains
unclear how applying this mapping would impact the actual _behavior_ of the
models. If an accurate bijection between the representation spaces of two
models is established, it should be possible to _intervene_ during the forward
pass of one model by substituting its representation with the corresponding
representation from the other model, without causing significant loss. In this
section, we conduct a behavioral analysis to investigate this hypothesis.

We design the following “layer injection” experiment: For each pair of BERTs
$\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ finetuned on a downstream
classification task (e.g., MNLI, SST2, or MRPC), we denote the embedding
spaces of the last layer before the classification as $\bm{X}$ for model
$\mathcal{M}_{1}$, and $\bm{Y}$ for model $\mathcal{M}_{2}$. We learn a
transformation function $g$ to map $\bm{X}$ to $\bm{Y}$. , we generate the
transformed embedding $g(\bm{X})$ and inject it into the second model
$\mathcal{M}_{2}$. With this direct replacement, we keep all the other weights
in $\mathcal{M}_{2}$ unchanged, and run it in the inference mode to get the
performance on all three datasets. Our experiment design carries a similar
spirit to the existing work in computer vision on convolutional neural
networks (Csiszárik et al., 2021).

| MNLI | SST2 | MRPC
---|---|---|---
Random Emb. |  33.12$\pm$14.63 |  53.97$\pm$26.83 |  42.49$\pm$23.89
Random Label |  33.55$\pm$0.43 |  50.44$\pm$1.20 |  52.9$\pm$2.51
Lin. Reg. |  37.15$\pm$24.26 |  54.14$\pm$31.27 |  45.93$\pm$28.48
CCA |  30.37$\pm$9.30 |  45.46$\pm$12.36 |  51.72$\pm$18.60
PWCCA |  39.70$\pm$4.57 |  50.24$\pm$0.90 |  50.00$\pm$18.38
Non-Bij. NN |  29.61$\pm$15.37 |  64.25$\pm$19.78 |  53.58$\pm$25.76
INN (Ours) |  75.75$\pm$8.97 |  74.01$\pm$15.61 |  86.34$\pm$1.07
Table 5: Behavioral check (by accuracy on the validation set) of various
bijective mapping methods across three datasets, MNLI, SST2, and MRPC. Apart
from linear regression (Lin. Reg.), CCA, PWCCA, non-bijective neural network
(Non-Bij. NN), and our INN method, we also include two random baselines, one
using random labels, and the other using random embeddings (Random Emb.),
where we apply an untrained, randomly initialized classifier on top of the
pretrained BERT encoder.

As we can see in Table 5, our model generates the best performance across all
datasets, which is quite impressive compared with the close-to-random
performance of many other methods. Moreover, our performance is even better
than that of the non-bijective neural networks, which might indicate that the
non-bijective transformation is not the proper way to go. It might distort
some features in the embedding space, making it incompatible with the
following layers of the injected BERT. Previous work also shows a discrepancy
between representational similarity and behavior similarity.

##  5 Findings: Variance and Invariance across BERTs

In this section, we utilize the INN alignment method on the 25 reproduced
BERTs to address the following question: How similar are these BERTs (which
only differ by random seeds)?

This question holds significance for the NLP community because both answers
matter for future work directions: If these BERTs are similar, then all the
work built on one single BERT will generalize to other BERTs. And if these
BERTs differ, then follow-up work on BERT should also test their findings on
the other BERTs varied by random seeds, to make sure the findings are robust.

To answer this overall question, we ask three subquestions from different
perspectives: (1) Across the 12 representation layers, which ones are similar,
and which ones are different? (2) How does the way how hidden states interact
with each other (i.e., attention weights) vary? (3) How does finetuning
enlarge or reduce the differences?

###  5.1 Deep vs. Shallow Layers

Figure 2:  Similarity of the hidden states learned in the 12 layers of BERTs.
For each layer, we align the 25 BERTs and plot the distribution of their L2
distances after alignment. The shallow layers (Layer 1 – 6) are similar, with
a small L2 distance of around 0.02, whereas the deep layers (Layer 7 – 12)
start to be more different, all the way to a large L2 distance of over 0.06.

We plot in Figure 2 the similarity of the representation spaces across the 12
layers of the BERTs. We show here the experimental results on the largest
dataset in our previous experiments, MNLI, and show in appendix the
experiments on other datasets.

As we can see, the shallow layers (Layer 1 – 6) are relatively similar, all
having an L2 distance close to 0.02, which is very small. However, the deeper
layers (Layer 7 – 12) start to see an increase in inconsistency, with a larger
L2 norm which rises from 0.03 to over 0.06, tripling the distance of shallow
layers.

Figure 3:  Similarity of the 12 layers of BERTs. We can see that the
transformation in different layers are more similar for layers next to each
other, and differ a lot when the layers are far away from each other.  Figure
4:  Similarity of the attention weights in the 12 layers of BERTs. For each
layer, we align the 25 BERTs and plot the distribution of their L2 distances
after alignment.

One potential explanation for the observed results is that the shallower
layers of the model tend to learn more low-level features that are shared
across various texts, whereas the deeper layers are capable of capturing more
complex semantic features that can have multiple plausible interpretations. To
support this claim, we present the results of our INN method, which measure
the similarity across all 12 layers, in Figure 3.

There are three notable findings from our analysis: (1) Two areas of high
similarity, represented by whiteness, are present in the plot, suggesting that
the representations of the shallow layers are similar among themselves, as are
the representations of the deep layers. (2) The last shows similarity to the
7th layer, since it needs both low-level features and high-level features in
order to predict the word correctly. (3) The difference in the representation
space changes gradually across the layers, without any abrupt changes.

###  5.2 Representation vs. Interaction

The transformer architecture of BERT learns not only the hidden
representations, but also the way how the hidden states should interact, which
are expressed by the weight matrices in the attention layers. Hence, we
inspect the similarity of the attention weights in Figure 4. Surprisingly, the
attention weights are far more invariant than the representations, with L2
distances lower by two orders of magnitude, ranging from 0.0002 to 0.0004.
This indicates that the way the hidden states interact tend to be very similar
across all BERTs.

###  5.3 Before vs. After Finetuning

(a) Similarity of the hidden states after fintuning. We can see that the
shallow layers stay invariant, whereas the deep layers vary even more than
those before finetuning in Figure 2.

(b) Similarity of the attention weights after fintuning. We can see that the
invariance is mostly kept across all layers.

Figure 5:  Similarity across the 12 layers of BERTs after finetuning. The deep
representations tend to vary more, whereas the shallow representations and
attention mechanisms tend to stay similar.

Finally, we inspect how finetuning affects the similarity among BERTs. In
Figure 5(a), we can see that, although the shallow layers (1 – 6) remain
similar after finetuning, the deep layers (7 – 12) vary drastically, reaching
a high L2 distance of up to 0.12, which doubles its original maximum L2
distance of 0.06 in Figure 2. On the contrary, the way how the hidden states
interact stays more invariant, as we can see that the attention weights stay
similar across the BERTs, with L2 distances still mostly around 0.0002 to
0.0004 in Figure 5(b).

##  6 Conclusion

In this work, we explored the question of how invariant are transformer models
that differ by random seeds. We proposed the Bijection Hypothesis, for which
we investigated a set of bijective methods to align different BERT models.
Through theoretical and empirical analyses, we identified that our proposed
INN is the most effective bijective method to align BERT embeddings. Applying
the INN method, we identify that shallow layers of BERTs are more consistent,
whereas deeper layers encode more variance, and finetuning brings a
substantial variance to the models, which imply that the finetuning process
very possibly leads the model to different alternative local minima. Our work
brings a unique insight for future work interpreting the internal
representations of transformer models.

## Ethical Considerations

This work aims to measure the similarity of the learned representation space
of the commonly used NLP model, BERT. This is a step towards safer model
deployment, and more transparency of the black-box LLMs. As far as we are
concerned, there seem to be no obvious ethical concerns for this work. All
datasets used in this work are commonly used benchmark datasets in the
community, and there is no user data involved.

## Author Contributions

This project originated from a discussion between Zhijing and Qipeng in 2020.
Inspired by Bernhard’s causal representation learning Schölkopf et al. (2021)
and promising results in computer vision, they did some preliminary
exploration into running ICA on BERT embeddings as a first step of causal
representation learning for NLP. When Yuxin joins MPI and ETH as an intern
with Zhijing, he discovers that ICA shows some results that could vary a lot
by the random seed, which inspires this work on inspecting the what random
seeds bring to the BERT models. Yuxin then did extensive analysis into the
model representations, and, then, with the insights of Ryan, we together
formulate this Bijection Hypothesis. Yuxin completed all the analyses over 300
model pairs and many iterations of experiments. Shauli added insights when we
change from CCA methods to INN, as well as many additional experiments such as
exploring the behavioral similarity. Mrinmaya and Bernhard co-supervised the
project. And all of the coauthors contributed to the writing.

## Acknowledgment

This material is based in part upon works supported by the German Federal
Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ:
01IS18039B; by the Machine Learning Cluster of Excellence, EXC number 2064/1 –
Project number 390727645; by the John Templeton Foundation (grant #61156); by
a Responsible AI grant by the Haslerstiftung; and an ETH Grant (ETH-19 21-1).
Zhijing Jin is supported by PhD fellowships from the Future of Life Institute
and Open Philanthropy.

## References

  * Ainsworth et al. (2022) Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2022.  Git re-basin: Merging models modulo permutation symmetries.  _arXiv preprint arXiv:2209.04836_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019.  What does bert look at? an analysis of bert’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286.
  * Csiszárik et al. (2021) Adrián Csiszárik, Péter Kőrösi-Szabó, Ákos Matszangosz, Gergely Papp, and Dániel Varga. 2021.  Similarity and matching of neural network representations.  _Advances in Neural Information Processing Systems_ , 34.
  * Del and Fishel (2021) Maksym Del and Mark Fishel. 2021.  Establishing interlingua in multilingual language models.  _arXiv preprint arXiv:2109.01207_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Association for Computational Linguistics (ACL)_ , pages 4171–4186.
  * Dinh et al. (2016) Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2016.  Density estimation using real nvp.  _arXiv preprint arXiv:1605.08803_.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020.  Finetuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv_.
  * Dolan and Brockett (2005) Bill Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_.
  * D’Amour et al. (2020) Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. 2020.  Underspecification presents challenges for credibility in modern machine learning.  _Journal of Machine Learning Research_.
  * Hecht-Nielsen (1990) Robert Hecht-Nielsen. 1990.  On the algebraic structure of feedforward network weight spaces.  In _Advanced Neural Computers_ , pages 129–135. Elsevier.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_.
  * Hewitt and Liang (2019) John Hewitt and Percy Liang. 2019.  Designing and interpreting probes with control tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2733–2743, Hong Kong, China. Association for Computational Linguistics.
  * Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019.  Revealing the dark secrets of bert.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4365–4374.
  * Lasri et al. (2022) Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022.  Probing for the usage of grammatical number.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8818–8831.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in gpt.  _arXiv preprint arXiv:2202.05262_.
  * Merchant et al. (2020) Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020.  What happens to bert embeddings during fine-tuning?  _arXiv preprint arXiv:2004.14448_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  _CoRR_ , abs/2203.02155.
  * Pimentel et al. (2020) Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020.  Pareto probing: Trading off accuracy for complexity.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3138–3153, Online. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  Technical report, OpenAI.
  * Ravfogel et al. (2021) Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021.  Counterfactual interventions reveal the causal effect of relative clause representations on agreement prediction.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 194–209.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020.  A primer in bertology: What we know about how bert works.  _Transactions of the Association for Computational Linguistics (TACL)_ , 8:842–866.
  * Schölkopf et al. (2021) Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021.  Towards causal representation learning.  _CoRR_ , abs/2102.11107.
  * Sellam et al. (2022) Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, et al. 2022.  The multiberts: Bert reproductions for robustness analysis.  In _ICLR_. OpenReview.net.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  Bert rediscovers the classical nlp pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_.
  * Wang et al. (2019) Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _International Conference on Learning Representations (ICLR)_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Association for Computational Linguistics (ACL)_ , pages 1112–1122.
  * Wu et al. (2019) Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2019\.  Emerging cross-lingual structure in pretrained language models.  _arXiv preprint arXiv:1911.01464_.
  * Zhao et al. (2020) Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, and Hinrich Schütze. 2020\.  Quantifying the contextualization of word representations with semantic class probing.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1219–1234, Online. Association for Computational Linguistics.
  * Zhelezniak et al. (2019) Vitalii Zhelezniak, April Shen, Daniel Busbridge, Aleksandar Savkov, and Nils Hammerla. 2019.  Correlations between word vector sets.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 77–87.
  * Zhong et al. (2021) Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. 2021.  Are larger pretrained language models uniformly better? comparing performance at the instance level.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3813–3827, Online. Association for Computational Linguistics.