11footnotetext: Ru-Hao Liu and Chen-Xin Qin contributed equivalently to this
work.22footnotetext<EMAIL_ADDRESS>Corresponding
author.33footnotetext<EMAIL_ADDRESS>Corresponding author.

# An Effective and Efficient Method to Solve the High-Order and the Non-Linear
Ordinary Differential Equations: the Ratio Net

Chen-Xin Qin  Ru-Hao Liu  Mao-Cai Li  Chi-Chun Zhou  and Yi-Liu

###### Abstract

An effective and efficient method that solves the high-order and the non-
linear ordinary differential equations is provided. The method is based on the
ratio net. By comparing the method with existing methods such as the
polynomial based method and the multilayer perceptron network based method, we
show that the ratio net gives good results and has higher efficiency.

##  1 Introduction

Many problems encountered in the fields of engineering, physics, mathematics,
and e.t.c. can be described by differential equations. For example,
differential equations are used in the simulation of oscillation phenomena
[1], the social sciences [2], the treatment of tumors [3], and e.t.c.. In many
cases, problems are expressed in the form of the higher-order and the non-
linear ordinary differential equations. For example, a high-order differential
equation filter can effectively remove the noise in a signal [4]. However, to
analytically solve the ordinary differential equations (ODEs), especially the
higher-order and the non-linear ODEs, is difficult. For example, the
analytical methods such as the series method [5, 6] and the constant-transform
method [5] sometimes fail at the higher-order and the non-linear cases.

Among many numerical methods, the neural network based methods are prominent
and give good results universally to various kinds of ODEs. For example,
neural network based methods are used to solve the first-order linear [7] and
nonlinear ODEs [8]. S. Chakraverty and S. Mall provide a regression-based
weight generation algorithm [9] and a Legendre neural network [10] to solve
high-order ODEs. YunLei Yang et al. prove the effectiveness of using neural
networks to solve ODEs [11];

In order to construct an effective neural network structure, several methods
are provided and these methods can be divided into two categories: (1) the
neural network based method with the trial functions based on the multilayer
perceptron network (MLP) [12, 13, 14, 15], the radial basis function network
(RBF) [16], and e.t.c.. (2) The polynomial based method with the trial
function based on the Legendre polynomials [10, 11], the Chebyshev polynomials
[17, 18, 19, 20], and e.t.c..

Beyond the construction of the trial functions, the boundary condition of ODEs
brings other difficulties. Usually, researchers consider the boundary
conditions as an additional constraint by transforming it into an extra term
in the loss function [11]. In that approach, the parameters of the model are
trained to not only fit the shape of the target function but also meet the
boundary condition. At cases, the boundary condition prevent the neural
network from finding the target function. Another approach introduces a trial
function by constructing a network that automatically satisfies the boundary
conditions. For example, a trial function in the form
$y^{trial}\left(x\right)=Net\left(x\right)B\left(x\right)+g\left(x\right)$,
where $B\left(x\right)$ and $g\left(x\right)$ are designed to satisfy the
boundary conditions and $Net\left(x\right)$ are designed to search the target
function [21].

There are two key issues in solving the ODEs by using the neural network based
methods, namely effectiveness and efficiency. (1) Effectiveness is about how
to construct an effective neural network structure that is able to find the
target function, or the solution of the ODE. (2) Efficiency is about how to
accelerate the training process. For the first issue, researchers provide
various kinds of neural network based methods and have proved the
effectiveness. Beyond the effectiveness, the efficiency need to be considered
too. That is, methods that can solves the ODEs efficiently are preferred.

In this paper, an effective and efficient method that solves the high-order
and the non-linear ODEs is provided. The method is based on the ratio net,
which is proposed in the previous works [22, 23]. Based on the previous work
[22, 23], we show that the ratio net is both efficiency and effective in
searching the target function. We solve the illustrative examples from Refs.
[11, 24, 25, 26, 27]. By comparing the method with existing methods such as
the polynomial based method and the MLP based method, we show that the ratio
net is both efficient and effective.

This paper is organized as follows: in Sec. 2, we give the description of the
method. In Sec. 3, we show the advantage of the method by applying it on
various illustrative examples. In Sec. 4, conclusions are given.

##  2 Method description

In this section, we introduce the main method used in the present paper,
including the structure of ratio net, the trial function under boundary
conditions, the loss function, and the updating algorithm for the weights.

###  2.1 The ratio net: a brief review

The method in the present paper are based on the ratio net [22]. The ratio net
is proved to be more efficient than the conventional neural networks such as
the MLP and the RBF [22]. In this section, we give a brief review on the ratio
net.

A neural network is good at searching the mode of the relation between the
outputs and the inputs. To search the mode, the nonlinearity between the
outputs and the inputs is the main difficulty. Instead of the nonlinear
activation function and the nonlinear kernel function, the ratio net uses
fractional forms to solve the difficulty of nonlinearity, see Fig. (1). In the
ratio net, the relation between the outputs and the inputs are

$y^{Ratio}_{i}(x)=\frac{\left(\sum_{j=1}^{n}w^{\prime}_{ij}x_{j}+b^{\prime}_{i}\right)\left(\sum_{j=1}^{n}w^{\prime\prime}_{ij}x_{j}+b^{\prime\prime}_{i}\right)....}{\left(\sum_{j=1}^{n}w^{\prime\prime\prime}_{ij}x_{j}+b^{\prime\prime\prime}_{i}\right)\left(\sum_{j=1}^{n}w^{\prime\prime\prime\prime}_{ij}x_{j}+b^{\prime\prime\prime\prime}_{i}\right)....}$
(2.1)

Where $w$, $b$, and e.t.c. are parameters or weights. $j$ runs from $1$ to the
dimension of the input $x$ and $i$ runs from $1$ to the dimension of the
output $y$. Here, in order to solve the ODEs, the dimension of input and
output are both $1$.

Figure 1: A diagram of Ratio neural networks

###  2.2 The trial function under the boundary condition

In solving an ODE, the boundary condition brings another difficulty. We find
that to construct a network as the trial function that meet the constraint of
the boundary condition automatically is important. In this section, based on
the ratio net, we construct a trial function that can automatically meet the
constraint of the boundary condition.

Here, we consider ODEs in the interval $[a,b]$ with boundary conditions
$y^{(n)}(a)=A_{n}$ and $y^{(m)}(b)=B_{m}$, where $y^{(n)}(a)$ and $y^{(m)}(b)$
are the $n$-th and $m$-th derivative at $a$ and $b$ respectively. $A_{n}$ and
$B_{m}$ are real numbers representing the boundary values. For example, a set
of boundary conditions are $y^{(0)}(a)=A_{0}$, $y^{(0)}(b)=B_{0}$,
$y^{(2)}(a)=A_{2}$, and $y^{(2)}(b)=B_{2}$. The trial function in the present
paper is in the form:

$y^{trial}_{ratio}(x)=y^{ratio}(x)(x-a)^{M_{a}}(b-x)^{M_{b}}+g(x),$ (2.2)

where $M_{a}$ and $M_{b}$ are the maximum order of derivative in the boundary
condition at $a$ and $b$ respectively. $y_{ratio}(x)$ is the ratio net giving
in Eq. (2.1) with the dimension of input and output both $1$. $g(x)$ is a
series of highest order $L$ with $L$ the number of equations in the boundary
conditions. $g(x)$ meet the boundary conditions and thus is decided by the
boundary conditions. For example, if the boundary conditions read
$y^{(0)}(a)=A$ and $y^{(0)}(b)=B$, $g(x)=(xB-xA+bA-aB)/(b-a)$.

Figure 2: An example of the trial function. By adjusting the weights, the
trial function is capable of searching the target function that meet the
boundary conditions.

###  2.3 Other existing methods: a brief review

In order to show the efficiency of the new method in solving the ODEs, we
compare the proposed method with other existing methods as well. Without loss
of generality, we consider two typical conventional neural networks that are
used to solve the ODEs. They are the Legendre polynomial based method and the
MLP based method. In this section, we give a brief review on these methods.

The polynomial based methods: the Legendre polynomials. The Legendre
polynomial is a typical class of orthogonal polynomials. The linear
combination of Legendre polynomials up to order $L$ can be used as a effective
function approximant. The Legendre polynomials read:

$P_{0}(x)=1,$ (2.3) $P_{1}(x)=x,$ (2.4)

and

$P_{n+1}(x)=\frac{2n+1}{n+1}xP_{n}(x)-\frac{n}{n+1}P_{n-1}(x),$ (2.5)

where $P_{l}(x)$ is Legendre polynomials of order $l$. The trial function
based on the Legendre polynomials is

$y^{trial}_{Legendre}(x)=y^{Legendre}(x)(x-a)^{M_{a}}(b-x)^{M_{b}}+N(x),$
(2.6)

where

$y^{Legendre}(x)=\sum_{j=0}^{L}\omega_{j}P_{j}(x)$ (2.7)

with $\omega_{j}$ the weight to be decided.

The neural network based methods: the MLP. The MLP is a classical neural
network that consists of an input layer, several hidden layers, and an output
layer. To obtain the next layer, a nonlinear activation function is applied to
the linear combination of the previous layer. For example, a one-hidden layer
MLP with the dimension of input layer and output layer $1$ is given as

$y^{MLP}(x)=\sum_{j=1}^{n}\omega_{ij}\sigma\left(\sum_{i=1}^{L}\omega_{i}x+b_{1}\right)+b_{2},$
(2.8)

where $L$ is the size of hidden layer and $\sigma(x)$ is the activation
function. Usually, the activation function can be
$\tanh\left(x\right)=\left(e^{x}-e^{-x}\right)/\left(e^{x}+e^{-x}\right)$,
$sigmod\left(x\right)=1/\left(1+e^{-x}\right)$, and $relu\left(x\right)=x$ for
$x>0$ and $0$ for $x<0$. The trial function of neural network based methods
read:

$y^{trial}_{MLP}(x)=y^{MLP}(x)(x-a)^{M_{a}}(b-x)^{M_{b}}+N(x),$ (2.9)

In this work, we consider trial functions as follows:

$y^{trial}=\left\\{\begin{array}[c]{c}y^{ratio}\left(x\right)\left(x-a\right)^{M_{a}}\left(b-x\right)^{M_{b}}+g\left(x\right)\\\
y^{Legendre}\left(x\right)\left(x-a\right)^{M_{a}}\left(b-x\right)^{M_{b}}+g\left(x\right)\\\
y^{MLP}\left(x\right)\left(x-a\right)^{M_{a}}\left(b-x\right)^{M_{b}}+g\left(x\right)\end{array}\right.$

###  2.4 The loss function and the update of the weights

The general form of differential equations with boundary conditions can be
expressed as

$F[x,y(x),y^{(1)}(x),.....,y^{(k)}(x)]=0,x\in[a,b]$ (2.10)

and

$C[x,y(x),y^{(1)}(x),.....,y^{(l)}(x)]=0,x=a,b,$ (2.11)

where $n$ is the order of the ODE and $C(x)$ is the boundary condition of the
ODE.

Since the trial function, Eq. (2.2), meet the boundary conditions
automatically, in solving such ODEs, we only need to minimize the following
loss function

$loss=\begin{Bmatrix}F[x,y^{trial}(x),\frac{dy^{trial}}{dx},.....,\frac{d^{n}y^{trial}}{dx^{n}}]\end{Bmatrix}^{2}$
(2.12)

by training the ratio net.

The wrights in the ratio net are trained to minimize the loss function Eq.
(2.12) through the gradient descent method:

$\omega^{\prime}=\omega+k\frac{\partial loss}{\omega}.$ (2.13)

The iteration of weight in the ratio net is implemented through tensorflow.
The details of the implementation of the algorithm will be given in the github
for each illustrative example.

##  3 Illustrative examples

In this section, we apply the ratio net on various illustrative examples.
Illustrative examples from Refs. [11, 25, 24, 26, 27] are included. We compare
the ratio net with other representative methods. It shows that the ratio net
is with higher efficiency.

###  3.1 Non-linear ordinary differential equations.

Example 1. We consider the ODE

$y^{\prime}\left(x\right)=2y\left(x\right)-y^{2}\left(x\right)+1$ (3.1)

with boundary conditions

$y\left(0\right)=1+\sqrt{2}\tanh\left[\frac{1}{2}\log\left(\frac{\sqrt{2}-1}{\sqrt{2}+1}\right)\right]$
(3.2)

and

$y\left(1\right)=1+\sqrt{2}\tanh\left[\sqrt{2}+\frac{1}{2}\log\left(\frac{\sqrt{2}-1}{\sqrt{2}+1}\right)\right].$
(3.3)

The exact solution of example 1 is

$y\left(x\right)=1+\sqrt{2}\tanh\left[\sqrt{2}x+\frac{1}{2}\log\left(\frac{\sqrt{2}-1}{\sqrt{2}+1}\right)\right].$
(3.4)

Fig. (3) shows the results given by three neural networks. The effectiveness
is characterized by the fitting diagram and the relative error between the
numerical solution and the analytical solution. The decreasing trend of the
loss function shows the efficiency of the method.

Figure 3: Comparison of the results given by the three neural networks of
example 1. The learning rates are all $0.01$.

Example 2. We consider the ODE

$y^{\prime\prime}\left(x\right)=\frac{y^{3}\left(x\right)-2y^{2}\left(x\right)}{2x^{2}}$
(3.5)

with boundary conditions

$y\left(0\right)=0\text{ and }y\left(2\right)=\frac{4}{3},$ (3.6)

which has the exact solution $y\left(x\right)=2x/\left(x+1\right)$. Fig. (4)
shows the results.

Figure 4: Comparison of the results given by the three neural networks of
example 2. The learning rates are all $0.1$.

Example 3. We consider the ODE

$y^{\prime\prime\prime}\left(x\right)=-y^{2}\left(x\right)-\cos\left(x\right)+\sin^{2}\left(x\right)$
(3.7)

with boundary conditions

$y\left(0\right)=0\text{, }y^{\prime}\left(0\right)=1\text{, and
}y\left(\pi\right)=0,$ (3.8)

which has the exact solution $y\left(x\right)=\sin\left(x\right)$. Fig. (5)
shows the results.

Figure 5: Comparison of the results given by the three neural networks of
example 3. The learning rates are all $0.01$.

###  3.2 High-order ordinary differential equations.

Example 4. We consider the ODE

$y^{\left(4\right)}\left(x\right)=120x$ (3.9)

with boundary conditions

$y\left(-1\right)=1\text{, }y^{\prime}\left(-1\right)=5\text{,
}y\left(1\right)=3\text{, and }y^{\prime}\left(1\right)=5,$ (3.10)

which has the exact solution $y\left(x\right)=x^{5}+2$. Fig. (6) shows the
results.

Figure 6: Comparison of the results given by the three neural networks of
example 4. The learning rates are all $0.0001$.

Example 5. We consider the ODE

$y^{(4)}\left(x\right)=-\frac{x^{2}}{1+y^{2}\left(x\right)}-72\left(1-5x+5x^{2}\right)+\frac{x^{2}}{1+{\left(x-x^{2}\right)}^{6}}$
(3.11)

with boundary conditions

$y\left(0\right)=0\text{, }y^{\prime}\left(0\right)=0\text{,
}y\left(1\right)=0\text{, and }y^{\prime}\left(1\right)=0,$ (3.12)

which has the exact solution $y\left(x\right)=x^{3}{\left(1-x\right)}^{3}$.
Fig. (7) shows the results.

Figure 7: Comparison of the results given by the three neural networks of
example 5. The learning rates are all $0.0001$.

Example 6. We consider the ODE

$y^{(4)}\left(x\right)+y\left(x\right)=\left[{\left(\frac{\pi}{2}\right)}^{4}+1\right]\cos\left(\frac{\pi}{2}x\right)$
(3.13)

with boundary conditions

$y\left(-1\right)=0\text{, }y^{\prime}\left(-1\right)=\frac{\pi}{2}\text{,
}y\left(1\right)=0\text{, and }y^{\prime}\left(1\right)=-\frac{\pi}{2},$
(3.14)

which has the exact solution
$y\left(x\right)=\cos\left(\pi\frac{x}{2}\right)$. Fig. (8) shows the results.

Figure 8: Comparison of the results given by the three neural networks of
example 6. The learning rates are all $0.0001$.

Example 7. We consider the ODE

$y^{(4)}\left(x\right)+y^{\prime}\left(x\right)=4x^{3}+24$ (3.15)

with boundary conditions

$y\left(0\right)=0\text{, }y^{\prime}\left(0\right)=0\text{,
}y\left(1\right)=1\text{, and }y^{\prime}\left(1\right)=4,$ (3.16)

which has the exact solution $y\left(x\right)=x^{4}$. Fig. (9) shows the
results.

Figure 9: Comparison of the results given by the three neural networks of
example 7. The learning rates are all $0.0001$.

###  3.3 Non-linear and high-order ordinary differential equations.

Example 8. We consider the ODE

$y^{(5)}\left(x\right)=e^{-x}y^{2}\left(x\right)$ (3.17)

with boundary conditions

$y\left(0\right)=1\text{, }y^{\prime}\left(0\right)=1\text{,
}y^{\prime\prime}\left(0\right)=1\text{, }y\left(1\right)=e\text{, and
}y^{\prime}\left(1\right)=e,$ (3.18)

which has the exact solution $y\left(x\right)=e^{x}$. Fig. (10) shows the
results.

Figure 10: Comparison of the results given by the three neural networks of
example 8. The learning rates are all $0.01$.

Example 9. We consider the ODE

$y^{(6)}\left(x\right)+y\left(x\right)y^{\prime\prime}\left(x\right)+y^{\prime}\left(x\right)y^{(5)}\left(x\right)-\pi^{2}\sin\left(\pi
x\right)y^{\prime\prime\prime}\left(x\right)+\pi^{1}y^{2}\left(x\right)=-\pi^{6}\cos\left(\pi
x\right)$ (3.19)

with boundary conditions

$y\left(-1\right)=\cos\left(-\pi\right)\text{,
}y^{\prime}\left(-1\right)=-\pi\sin\left(-\pi\right)\text{,
}y^{\prime\prime}\left(-1\right)=-\pi^{2}\cos\left(-\pi\right),$ (3.20)
$y\left(1\right)=\cos\left(\pi\right)\text{,
}y^{\prime}\left(1\right)=-\pi\sin\left(\pi\right)\text{, and
}y^{\prime\prime}\left(1\right)=-\pi^{2}\cos\left(\pi\right),$ (3.21)

which has the exact solution $y\left(x\right)=\cos\left(\pi x\right)$. Fig.
(11) shows the results.

Figure 11: Comparison of the results given by the three neural networks of
example 9. The learning rates are all $0.001$.

###  3.4 Linear ordinary differential equations.

In this section, we consider several linear ODEs.

Example 10,

$y^{\prime\prime}\left(x\right)-y^{\prime}\left(x\right)=-2\sin\left(x\right)$
(3.22)

with boundary conditions

$y\left(0\right)=-1\text{ and }y\left(\frac{\pi}{2}\right)=1,$ (3.23)

which has the exact solution
$y\left(x\right)=\sin\left(x\right)-\cos\left(x\right)$.

Example 11,

$y^{\prime}\left(x\right)+\frac{1}{5}y=e^{-\frac{x}{5}}\cos\left(x\right)$
(3.24)

with boundary conditions

$y\left(0\right)=0\text{ and
}y\left(1\right)=\frac{\sin\left(1\right)}{e^{1/5}},$ (3.25)

which has the exact solution $y\left(x\right)=e^{-x/5}\sin\left(x\right)$.

Example 12,

$y^{\prime}\left(x\right)+\left(x+\frac{1+3x^{2}}{1+x+x^{3}}\right)y\left(x\right)=x^{3}+2x+\frac{x^{2}+3x^{4}}{1+x+x^{3}}$
(3.26)

with boundary conditions

$y\left(0\right)=-1\text{ and }y\left(1\right)=1+\frac{1}{3e^{1/2}},$ (3.27)

which has the exact solution $y(x)=e^{-x^{2}/2}/\left(1+x+x^{3}\right)+x^{2}$.

Example 13,

$y^{\prime}\left(x\right)-\sin\left(x\right)y\left(x\right)=2x-x^{2}\sin\left(x\right)$
(3.28)

with boundary conditions

$y\left(-1\right)=1\text{ and }y\left(1\right)=1,$ (3.29)

which has the exact solution $y\left(x\right)=x^{2}$.

Example 14,

$y^{\prime\prime}\left(x\right)+xy^{\prime}\left(x\right)-4y\left(x\right)=12x^{2}-3x$
(3.30)

with boundary conditions

$y\left(0\right)=0\text{ and }y\left(2\right)=18,$ (3.31)

which has the exact solution $y\left(x\right)=x^{4}+x$.

Example 15,

$y^{\prime\prime}\left(x\right)-y^{\prime}\left(x\right)=-2\sin\left(x\right)$
(3.32)

with boundary conditions

$y\left(0\right)=-1\text{ and }y\left(\frac{\pi}{2}\right)=1,$ (3.33)

which has the exact solution
$y\left(x\right)=\sin\left(x\right)-\cos\left(x\right)$.

Example 16,

$y^{\prime\prime}\left(x\right)+2y^{\prime}\left(x\right)+y\left(x\right)=x^{2}+3x+1$
(3.34)

with boundary conditions

$y\left(0\right)=0\text{ and }y\left(1\right)=-e^{-1}+1,$ (3.35)

which has the exact solution $y\left(x\right)=-e^{-x}+x^{2}-x+1$.

Example 17,

$y^{\prime\prime}\left(x\right)+\frac{1}{5}y^{\prime}\left(x\right)+y\left(x\right)=-\frac{1}{5}e^{-\frac{x}{5}}\cos\left(x\right)$
(3.36)

with boundary conditions

$y\left(0\right)=0\text{ and
}y\left(2\right)=\sin\left(2\right)e^{-\frac{2}{5}},$ (3.37)

which has the exact solution $y\left(x\right)=\sin\left(x\right)e^{-x/5}$.

Example 18,

$y^{\prime}\left(x\right)=y\left(x\right)-x^{2}+1$ (3.38)

with boundary conditions

$y\left(0\right)=0.5\text{ and }y\left(1\right)=4-\frac{\pi}{2},$ (3.39)

which has the exact solution
$y\left(x\right)={\left(x+1\right)}^{2}-0.5e^{x}$. Fig. (12) shows the results
of examples 11-18 given by the three neural networks. The effectiveness is
characterized by the fitting diagram and the relative error between the
analytical solution and the numerical solution. The decreasing trend of the
loss function shows the efficiency of the method.

Figure 12: The loss versus steps of examples 10 to 18. The learning rates of each examples may be different, but for the same examples, the learning rates of different models are same. Table 1: The relative errors of examples 10–18 Relative error | Ratio | MLP | Legendre |
---|---|---|---|---
Example 10 | $3.119e^{-06}$ | $3.496e^{-04}$ | $2.061e^{-05}$ |
Example 11 | $1.592e^{-07}$ | $1.247e^{-06}$ | $1.202e^{-06}$ |
Example 12 | $1.799e^{-02}$ | $2.715e^{-02}$ | $1.832e^{-02}$ |
Example 13 | $5.591e^{-06}$ | $2.236e^{-05}$ | $3.136e^{-07}$ |
Example 14 | $1.066e^{-05}$ | $4.515e^{-06}$ | $9.550e^{-05}$ |
Example 15 | $5.041e^{-07}$ | $9.846e^{-04}$ | $9.964e^{-05}$ |
Example 16 | $4.967e^{-06}$ | $6.803e^{-06}$ | $2.391e^{-06}$ |
Example 17 | $3.863e^{-06}$ | $8.451e^{-03}$ | $7.467e^{-06}$ |
Example 18 | $8.191e^{-07}$ | $2.684e^{-05}$ | $1.055e^{-05}$ |

Example 19,

$y^{\prime\prime}\left(x\right)+y\left(x\right)=2$ (3.40)

with boundary conditions

$y\left(0\right)=1\text{ and }y\left(1\right)=0,$ (3.41)

which has the exact solution
$y\left(x\right)=\left[(\cos\left(1\right)-2)/(\sin\left(1\right))\right]\sin\left(x\right)-\cos\left(x\right)+2$.

Example 20,

$y^{\prime\prime}\left(x\right)+\frac{2}{x}y^{\prime}\left(x\right)+2y\left(x\right)=0$
(3.42)

with boundary conditions

$y\left(0.001\right)=1\text{ and
}y\left(1\right)=\frac{\sin\left(\sqrt{2}\right)}{\sqrt{2}},$ (3.43)

which has the exact solution
$y\left(x\right)=\sin\left(\sqrt{2}x\right)/\sqrt{2}x$.

Example 21,

$y^{\prime}\left(x\right)+\frac{\cos\left(x\right)}{\sin\left(x\right)}y\left(x\right)=\frac{1}{\sin\left(x\right)}$
(3.44)

with boundary conditions

$y\left(1\right)=\frac{3}{\sin\left(1\right)}\text{ and
}y\left(2\right)=\frac{4}{\sin\left(2\right)},$ (3.45)

which has the exact solution
$y\left(x\right)=\left(x+2\right)/\sin\left(x\right)$.

Example 22,

$y^{\prime\prime}\left(x\right)-\frac{1}{1+e^{x}}y^{\prime}\left(x\right)-\frac{15e^{2x}}{{\left(1+e^{x}\right)}^{2}}y\left(x\right)=\frac{e^{2x}}{{\left(1+e^{x}\right)}^{6}}$
(3.46)

with boundary conditions

$y\left(-1\right)=\frac{1}{{\left(1+e^{-1}\right)}^{4}}\text{ and
}y\left(0\right)=\frac{1}{2^{4}},$ (3.47)

which has the exact solution $y\left(x\right)={\left(1+e^{x}\right)}^{-4}$.

Example 23,

$y^{\prime}\left(x\right)=4x^{3}-3x^{2}+2$ (3.48)

with boundary conditions

$y\left(0\right)=0\text{ and }y\left(1\right)=2,$ (3.49)

which has the exact solution $y\left(x\right)=x^{4}-x^{3}+2x$.

Example 24,

$y^{\prime}\left(x\right)=y\left(x\right)$ (3.50)

with boundary conditions

$y\left(0\right)=1\text{ and }y\left(1\right)=e,$ (3.51)

which has the exact solution $y\left(x\right)=e^{x}$.

Example 25,

$y^{\prime}\left(x\right)=3\cos\left(x\right)\sin^{2}\left(x\right)+6\sin\left(3x\right)$
(3.52)

with boundary conditions

$y\left(0\right)=-3\text{ and
}y\left(1\right)=-2\cos\left(3\right)+\sin^{3}\left(1\right)-1,$ (3.53)

which has the exact solution
$y\left(x\right)=-2\cos\left(3x\right)+\sin^{3}x-1$

Example 26,

$y^{\prime\prime}\left(x\right)=\frac{3}{x}\left(1-y^{\prime}\left(x\right)\right)-\frac{27}{x^{3}}$
(3.54)

with boundary conditions

$y\left(1\right)=2\text{ and }y\left(2\right)=\frac{27}{4},$ (3.55)

which has the exact solution $y\left(x\right)=x-3+27/x-23/x^{2}$. Fig. (13)
shows the results of examples 19-26.

Figure 13: The loss versus steps of examples 19 to 26. The learning rates of each examples may be different, but for the same examples, the learning rates of different models are same. Table 2: The relative errors of examples 19–26 Relative error | Ratio | MLP | Legendre |
---|---|---|---|---
Example 19 | $2.625e^{-05}$ | $4.880e^{-05}$ | $2.376e^{-06}$ |
Example 20 | $2.092e^{-07}$ | $5.598e^{-05}$ | $6.479e^{-06}$ |
Example 21 | $4.484e^{-05}$ | $1.578e^{-04}$ | $3.319e^{-04}$ |
Example 22 | $4.916e^{-05}$ | $1.053e^{-05}$ | $1.497e^{-05}$ |
Example 23 | $2.534e^{-05}$ | $5.515e^{-05}$ | $9.775e^{-06}$ |
Example 24 | $1.008e^{-05}$ | $1.264e^{-07}$ | $6.382e^{-06}$ |
Example 25 | $5.541e^{-05}$ | $1.110e^{-04}$ | $2.172e^{-04}$ |
Example 26 | $2.730e^{-04}$ | $1.473e^{-02}$ | $1.742e^{-02}$ |

##  4 Conclusions

In this paper, an effective and efficient method that solves the high-order
and the non-linear ordinary differential equations is provided. The method is
based on the ratio net. By comparing the method with existing methods such as
the polynomial based method and the MLP based method, we show that the ratio
net is both effective and efficient. In the following research, we use the
ratio net to solve partial differential equations.

##  5 Acknowledgments

We are very indebted to Prof. Wu-Sheng Dai for his enlightenment and
encouragement. We are very indebted to Prof. Guan-Wen Fang and Yong-Xie for
their encouragements. This work is supported by Yunnan Youth Basic Research
Projects (202001AU070020) and Doctoral Programs of Dali University
(KYBS201910).

## References

  * [1] J. M. Franco, Runge–kutta–nystrm methods adapted to the numerical integration of perturbed oscillators, Computer Physics Communications (2002).
  * [2] Ledder, Differential equations : a modeling approach.  McGraw-Hill Higher Education, 2005.
  * [3] R. Duclous, B. Dubroca, and M. Frank, A deterministic partial differential equation model for dose calculation in electron radiotherapy, Physics in Medicine & Biology 55 (2010), no. 13 3843.
  * [4] L. Zhao and A. Yin, High-order partial differential equation de-noising method for vibration signal, Mathematical Methods in the Applied Sciences (2015).
  * [5] V. F. Zaitsev and A. D. Polyanin, Handbook of exact solutions for ordinary differential equations.  CRC press, 2002.
  * [6] G. Corliss and Y. Chang, Solving ordinary differential equations using taylor series, ACM Transactions on Mathematical Software (TOMS) 8 (1982), no. 2 114–144.
  * [7] H. Lee and I. S. Kang, Neural algorithm for solving differential equations, Journal of Computational Physics 91 (1990), no. 1 110–131.
  * [8] S. Mall and S. Chakraverty, Regression-based neural network training for the solution of ordinary differential equations, International Journal of Mathematical Modelling and Numerical Optimisation 4 (2013), no. 2 136–149.
  * [9] S. Chakraverty and S. Mall, Regression-based weight generation algorithm in neural network for solution of initial and boundary value problems, Neural Computing and Applications 25 (2014), no. 3 585–594.
  * [10] S. Mall and S. Chakraverty, Application of legendre neural network for solving ordinary differential equations, Applied Soft Computing (2016) 347–356.
  * [11] Y. Yang, M. Hou, and J. Luo, A novel improved extreme learning machine algorithm in solving ordinary differential equations by legendre neural network methods, Advances in Difference Equations 2018 (2018), no. 1.
  * [12] M. Kumar and N. Yadav, Multilayer perceptrons and radial basis function neural network methods for the solution of differential equations: a survey, Computers & Mathematics with Applications 62 (2011), no. 10 3796–3811.
  * [13] H. Alli, A. U?ar, and Y. Demir, The solutions of vibration control problems using artificial neural networks, Journal of the Franklin Institute 340 (2003), no. 5 307–325.
  * [14] Y. Shirvany, M. Hayati, and R. Moradian, Multilayer perceptron neural networks with novel unsupervised training method for numerical solution of the partial differential equations, Applied Soft Computing 9 (2009), no. 1 20–29.
  * [15] T. Schneidereit and M. Breuß, Solving ordinary differential equations using artificial neural networks-a study on the solution variance, in Proceedings of the Conference Algoritmy, pp. 21–30, 2020.
  * [16] F. B. Rizaner and A. Rizaner, Approximate solutions of initial value problems for ordinary differential equations using radial basis function networks, Neural Processing Letters (2017).
  * [17] S. Mall and S. Chakraverty, Single layer chebyshev neural network model for solving elliptic partial differential equations, Neural Processing Letters (2016).
  * [18] S. Mall and S. Chakraverty, Chebyshev neural network based model for solving lane–emden type equations, Applied Mathematics and Computation 247 (2014) 100–114.
  * [19] S. Chakraverty and S. Mall, Single layer chebyshev neural network model with regression-based weights for solving nonlinear ordinary differential equations, Evolutionary Intelligence 13 (2020), no. 4.
  * [20] M. Juhola, J. C. Patra, P. K. Meher, J. Patra, and P. Meher, Intelligent sensors using computationally efficient chebyshev neural networks, .
  * [21] A. Malek and R. S. Beidokhti, Numerical solution for high order differential equations using a hybrid neural network—optimization method, Applied Mathematics and Computation 183 (2006), no. 1 260–271.
  * [22] C.-C. Zhou, H.-L. Tu, Y. Liu, and J. Hua, Activation functions are not needed: the ratio net, arXiv preprint arXiv:2005.06678 (2020).
  * [23] C.-C. Zhou and Y. Liu, The pade approximant based network for variational problems, arXiv preprint arXiv:2004.00711 (2020).
  * [24] S. Ezadi and N. Parandin, An application of neural networks to solve ordinary differential equations, .
  * [25] M. A. Ramadan, K. R. Raslan, T. S. El Danaf, and M. A. Abd, El Salam, An exponential chebyshev second kind approximation for solving high-order ordinary differential equations in unbounded domains, with application to dawson’s integral, Journal of the Egyptian Mathematical Society 25 (2016), no. 2.
  * [26] A. Akyuz-Dacolu and H. erdik Yaslan, The solution of high-order nonlinear ordinary differential equations by chebyshev series, Applied Mathematics Computation 217 (2011), no. 12 5658–5666.
  * [27] H. F. Parapari and M. B. Menhaj, Solving nonlinear ordinary differential equations using neural networks, in International Conference on Control, 2016.


# Efficient transfer learning for NLP with ELECTRA

François Mercier
Mila, Université de Montréal
<EMAIL_ADDRESS>

## Reproducibility Summary

### Scope of Reproducibility

Clark et al. (2020) claims that the ELECTRA approach is highly efficient in
NLP performances relative to computation budget. As such, this study focus on
this claim, summarized by the following question: Can we use ELECTRA to
achieve close to SOTA performances for NLP in low-resource settings, in term
of compute cost?

### Methodology

This replication study has been conducted by fully reimplementing the small
variant of the original ELECTRA model (Clark et al. (2020)). All experiments
are performed on single GPU computers. GLUE benchmark dev set (Wang et al.
(2018)) is used for models evaluation and compared with the original paper.

### Results

My results are similar to the original ELECTRA’s implementation (Clark et al.
(2020)), despite minor differences compared to the original paper for both
implementations. With only 14M parameters, ELECTRA-Small outperforms, in
absolute performances, concurrent pretraining approaches from some previous
SOTA, such as GPT, or alternative efficient approaches using knowledge
distillation, such as DistilBERT. By taking into account compute cost, ELECTRA
is clearly outperforming all compared approaches, including BERT and TinyBERT.
Therefore, this work supports the claim that ELECTRA achieves high level of
performances in low-resource settings, in term of compute cost.

Furthermore, with an increased generator capacity than recommended by Clark et
al. (2020), the discriminant can collapses by being unable to distinguish if
inputs are fake or not. Thus, while ELECTRA is easier to train than GAN
(Goodfellow et al. (2014)), it appears to be sensitive to capacity allocation
between generator and discriminator.

The code and a pretrained model are available via this link
https://github.com/cccwam/rc2020_electra

### What was easy

Information provided by the authors of the original paper (Clark et al.
(2020)), either from the paper; within the source code: or from the official
Github repository, is very rich and exhaustive to understand the proposed
approach. In addition, as stated with their main claim, ELECTRA can be easily
run on a single GPU.

### What was difficult

By being an aggregation of several tasks and the variance from results, GLUE
benchmark requires significant amount of effort, in term of implementation and
computation. For models comparison, several tricks can also influence the
results, which is even more amplified by the different aggregation formulas
and the lack of measure of dispersion for published results. As such,
confirming the correctness of this reimplementation was harder than expected.

### Communication with original authors

Kevin Clark, one of the original authors, has been helpful by answering some
questions. Unfortunately, breakdown of GLUE score per tasks have not yet been
provided to fully compare this implementation with the original one.
Otherwise, most of questions that I had were already answered though the
Github repository or by inspecting the source code.

##  1 Introduction

Over the recent years, the combination of pretraining task with large
unlabelled text corpus has shown a lot of success for Natural Language
Processing (NLP) tasks (Howard and Ruder (2018), Peters et al. (2018), Devlin
et al. (2019), Liu et al. (2019), Radford et al. (2019)). However, the main
drawback is the computation requirements, which limits the adoption of these
approaches for more specialized domains for some potential use cases.

Unlike the recent trend of increasing computing requirements, in the original
paper ELECTRA (Clark et al. (2020)), the authors are taking the opposite
research direction to have more efficient approach for transfer learning for
NLP tasks. Thus, the authors provide a way to democratize the use of deep
learning for more actors and more use cases.

##  2 Scope of reproducibility

The scope for this reproducibility work is related to the following questions,
which are part of the original paper (Clark et al. (2020)).

Can we use ELECTRA pretraining task to achieve close to SOTA performances on
low-resource settings, in term of compute cost?
To formalize this claim, we use the GLUE benchmark (Wang et al. (2018)) as
measure of NLP performances and compare it with large pretrained models, such
as GPT (Radford et al. (2018)). All experiments are executed on single GPU
computers.

How does the training process behave with different generator size ?
While ELECTRA shares some aspects from GAN (Goodfellow et al. (2014)), with
the use of discriminator and generator networks, the training is not
adversarial. Nevertheless, the capacity of the generator compared to the
discriminator remains an important hyper-parameter in the architecture.

##  3 Methodology

The approach for this work is to fully reimplement in PyTorch the small
variant of ELECTRA (Clark et al. (2020)). For this, the different libraries
from HuggingFace (Wolf et al. (2020)), namely Transformers; Tokenizers; and
Datasets, have been used respectively for the Transformer (Vaswani et al.
(2017)) implementation and training loop logic; tokenization models; and to
ease the access to relevant datasets.

Due to constraint of compute resources availability initially, some changes
have been implemented in this reimplementation, such as the ability to use
gradient accumulation. Furthermore, to ease further explorations, as the
preprocessing steps are time consuming, approximately 3 hours, the
reimplementation’s preprocessing has been adapted to be less dependant on the
sequence length, see section 3.2 for the details.

###  3.1 Model descriptions

ELECTRA-Small is a complete reimplementation in PyTorch of ELECTRA-Small*,
original implementation in TensorFlow from Clark et al. (2020). The
architecture consists in two Transformer based neural networks:

  * •

The generator network transforms a sequence of tokens, containing masked
tokens, into a new sequence with the original tokens predictions for these
masked tokens.

  * •

The discriminator network transforms a sequence of tokens, containing tokens
replaced by a generator, into a new sequence of binary predictions, true if
the token has been replaced, otherwise false.

Similar to ELECTRA-Small* (Clark et al. (2020)), input embeddings are shared
between the generator and the discriminator models. Moreover, input token
embeddings and output token embeddings are shared for the generator model.
Position embeddings are also used similar to BERT (Devlin et al. (2019)) in
both ELECTRA implementations.

For the downstream tasks only, both implementations use a 2 layer MLP heads.
The original implementation (Clark et al. (2020)) uses the first
contextualized embeddings for the downstream head. However, this
reimplementation uses average pooling across all contextualized embeddings.

For a visual representation, please refer to the figure 1.

Figure 1: Model architecture.
Generator is only used at pretraining. Average pooling and MLP are only used
for downstream tasks.
Note: the original implementation (Clark et al. (2020)) uses first embedding
for pooling layer instead of average pooling.
Image inspired by the original paper (Clark et al. (2020)).

###  3.2 Datasets

Pretraining task:  The original paper (Clark et al. (2020)) uses the same
datasets and preprocessing steps as BERT (Devlin et al. (2018)), namely
Wikipedia and BookCorpus (Zhu et al. (2015)), also referred as Wikibooks. In
the official GitHub repository111https://github.com/google-research/electra,
the authors use also OpenWebText, (Gokaslan and Cohen (2019)), referred as
OWT. The preprocessing steps consist to extract text segments satisfying the
maximum sequence length from large text corpus. Segments are then tokenized
using WordPiece (Wu et al. (2016)) with a vocabulary size 30,522. For each
training input, two tokenized segments are concatenated with the special token
<SEP> and the special token <CLS> is added in the first position. In addition
of this input, a sequence id is provided to identify the segment, either 0 or
1, and a position id is also provided to give spacial information into inputs.

For this reproducibility study, the dataset for the pretraining task is
OpenWebText (Gokaslan and Cohen (2019)). For the preprocessing, the
reimplementation uses a slight variation from the original paper. Indeed, all
documents are tokenized offline using Byte-level Byte Pair Encoding (Radford
et al. (2019)) with a vocabulary size 30,522. During training, for each
document, a random segment with the maximum sequence length allowed by the
model is selected. In other words, this reimplementation uses a dynamic
segmentation instead of precomputed static segments in the original
implementation.

Downstream task:  Similar to the original paper (Clark et al. (2020)), the
downstream tasks are the GLUE benchmark (Wang et al. (2018)). The
preprocessing steps are mostly similar to the original paper. For single
sentence tasks, inputs consist of the single sentence for each training input.
For double sentences tasks, both sentences are concatenated with the special
token <SEP>. Finally, inputs are potentially truncated to ensure the maximum
sequence length.

###  3.3 Hyperparameters

The selected hyperparameters are the same as the original paper, except for
the optimization algorithm and the use of gradient accumulation. For the
optimizer, AdamW (Loshchilov and Hutter (2019)) has been used instead of Adam
(Kingma and Ba (2014)). In addition, the original implementation is actually
using a lower layer-wise learning rate than described in Clark et al. (2020).

For more information, please refer to the tables 5 and 6.

###  3.4 Experimental setup

Pretrained models were trained on a single GPU computer, with Nvidia RTX 3090.
Regarding the downstream tasks, experiments were either run on Google Colab
Pro instances, with Nvidia P100 or V100. Due to the heterogeneous
infrastructure, a key element of the experimental setup is the centralization
of results with Weights & Biases (Biewald (2020)). For further details, please
refer to the Github repository222https://github.com/cccwam/rc2020_electra.

###  3.5 Computational requirements

The pretraining task and the downstream tasks can be performed with a relative
limited GPU memory budget and a single GPU, thanks to gradient accumulation
and mixed precision. For example, pretraining can be done with a single GTX
1060 6GB in 21 days; or on GTX 1080 Ti 11GB in 13 days instead of 3.75 days on
RTX 3090 24GB.

Interestingly, by constraining the GPU memory usages to 16GB, minimum amount
for the GPU used in the original paper (Clark et al. (2020)), the duration for
the pretraining task is similar to the original implementation (Clark et al.
(2020)), meaning 3.75 days on RTX 3090 instead of 4 days on V100. Using all
GPU memory for the RTX 3090 would have decrease the computation requirements,
but it would have complicated the comparison between reimplementations.

Computational requirements, per experiments and overall, are summarized in the
table 1.

Table 1: Compute costs for the different experiments in GPU.days, relative to
Nvidia RTX 3090.
The overall total is less than the sum of individual experiments due to overlap. Experiment | Task | Run cost |  | #
---
config.
| #
---
seeds
Total
Reproducing ELECTRA on GLUE benchmark | Pretraining | $5$ | $1$ | $1$ | $5$
(section 4.1) | Downstream | $0.2$ | $6$ | $5$ | $6$
Sensitivity to generator size | Pretraining | $25\%\times 5$ | $4$ | $1$ | $5$
(section 4.2) | Downstream | $0.2$ | $4$ | $5$ | $4$
Overall |  |  |  |  | $17.5$
Overall USD cost assuming 1 V100 GPU at USD2.26 hourly rate |  |  |  | $962

##  4 Experiments

###  4.1 Reproducing ELECTRA on GLUE benchmark

In this experiment, one model is pretrained with 1 million steps on
OpenWebText (Gokaslan and Cohen (2019)) with the similar approach as the
original paper (Clark et al. (2020)). With the weights initialization from
this pretrained model, the downstream task, the GLUE benchmark (Wang et al.
(2018)), is executed 5 times with different seeds. The results are summarized
in the table 2.

Reimplementation compared to the original: This reimplementation provides
results, which are fairly similar to the original implementation (Clark et al.
(2020)) on OpenWebText dataset for individual GLUE tasks. Furthermore, the
training dynamics, captured by the different metrics from the generator and
the discriminator, are very similar to the one provided in the official
ELECTRA’s Github repository, see figure 3 in appendix. Thus, this
reimplementation behaves similarly to the original implementation.

Discrepancies for the aggregated GLUE score: By comparing results in table 2
and table 1 from Clark et al. (2020), this study reports lower GLUE scores for
each baseline and ELECTRA models. For baseline models, I collected the
different information from different sources, mainly from the original papers
(see notes from table 2 for exact sources). Then, I recomputed the GLUE score
on dev set as per Wang et al. (2018) from the individual task results. The
discrepancy appears to be related to a difference of reporting methodology,
surely related to the treatment of the WNLI task. Therefore, table 1 from
Clark et al. (2020) is not reporting the actual GLUE score on dev set but a
meta score, from which I cannot fully reproduce. While this discrepancy
involves an overstatement for all models in Clark et al. (2020), it doesn’t
change the main claim about the high performance of ELECTRA relative to other
approaches.

Table 2: Results on GLUE dev set. For this reimplementation, results are
reported as mean and standard deviation, using 5 different seeds per task.
(mc: Matthews correlation, acc: Accuracy, spc: Spearman correlation, AVG:
Average of individual metrics as Clark et al. (2020))

Model |  | CoLA
---
(mc)
| SST-2
---
(acc)
| MRPC
---
(acc)
| STS-B
---
(spc)
| QQP
---
(acc)
| MNLI
---
(acc)
| QNLI
---
(acc)
| RTE
---
(acc)
AVG | GLUE*
Baselines ($\gg$14M parameters)
ELMo2 | $15.6$ | $84.9$ | $80.6$ | $64.4$ | $82.2$ | $69.4$ | $73.0$ | $50.9$ | $65.1$ | $63.8$
ELMo frozen1 | $44.1$ | $91.5$ | $82.3$ | $70.5$ | $84.3$ | $68.6$ | $71.2$ | $53.4$ | $70.7$ | $68.7$
GPT2 | $50.2$ | $93.2$ | $85.9$ | $86.5$ | $85.9$ | $81.2$ | $82.4$ | $58.1$ | $77.9$ | $75.4$
DistilBERT63 | $51.3$ | $91.3$ |  $87.5$† |  $86.9$† |  $88.5$† | $82.2$ | $89.2$ | $59.9$ |  | $77.0$
TinyBERT64 | $54.0$ | $93.0$ | $86.3$ | $89.6$ | $91.1$ | $84.5$ | $91.1$ | $73.4$ | $82.9$ | $80.0$
BERT-Base3 | $56.3$ | $92.7$ |  $88.6$† |  $89.0$† |  $89.6$† | $86.7$ | $91.8$ | $69.3$ |  | $80.0$
ELECTRA-Small OWT - Original ($\approx$14M parameters)
$100\%$ trained5 | $56.8$ | $88.3$ | $87.4$ | $86.8$ | $88.3$ | $78.9$ | $87.9$ | $68.5$ | $80.4$ |
ELECTRA-Small OWT - Mine ($\approx$14M parameters)
6% trained |  | $44.4$
---
$\pm$ $1.87$
| $81.9$
---
$\pm$ $0.83$
| $83.4$
---
$\pm$ $1.11$
| $80.2$
---
$\pm$ $0.37$
| $83.6$
---
$\pm$ $0.16$
| $74.9$
---
$\pm$ $0.21$
| $84.1$
---
$\pm$ $0.27$
| $57.6$
---
$\pm$ $2.76$
| $74.3$
---
$\pm$ $0.64$
| $72.3$
---
$\pm$ $0.61$
12% trained |  | $46.1$
---
$\pm$ $1.34$
| $84.2$
---
$\pm$ $0.46$
| $83.0$
---
$\pm$ $1.75$
| $82.0$
---
$\pm$ $0.35$
| $84.2$
---
$\pm$ $0.08$
| $76.2$
---
$\pm$ $0.12$
| $84.7$
---
$\pm$ $0.37$
| $59.4$
---
$\pm$ $2.61$
| $75.4$
---
$\pm$ $0.53$
| $73.4$
---
$\pm$ $0.45$
$25\%$ trained |  | $50.2$
---
$\pm$ $1.38$
| $86.6$
---
$\pm$ $0.66$
| $85.3$
---
$\pm$ $0.74$
| $83.9$
---
$\pm$ $0.61$
| $85.0$
---
$\pm$ $0.09$
| $77.9$
---
$\pm$ $0.19$
| $86.0$
---
$\pm$ $0.14$
| $58.2$
---
$\pm$ $2.52$
| $77.1$
---
$\pm$ $0.33$
| $74.8$
---
$\pm$ $0.29$
50% trained |  | $51.5$
---
$\pm$ $0.98$
| $88.0$
---
$\pm$ $0.66$
| $86.7$
---
$\pm$ $1.47$
| $84.3$
---
$\pm$ $0.60$
| $85.5$
---
$\pm$ $0.08$
| $79.2$
---
$\pm$ $0.23$
| $86.8$
---
$\pm$ $0.36$
| $60.8$
---
$\pm$ $1.54$
| $78.3$
---
$\pm$ $0.40$
| $75.9$
---
$\pm$ $0.34$
75% trained |  | $53.8$
---
$\pm$ $1.52$
| $89.2$
---
$\pm$ $0.67$
| $87.0$
---
$\pm$ $0.47$
| $84.8$
---
$\pm$ $0.47$
| $85.9$
---
$\pm$ $0.08$
| $79.8$
---
$\pm$ $0.11$
| $87.2$
---
$\pm$ $0.50$
| $61.6$
---
$\pm$ $1.23$
| $79.1$
---
$\pm$ $0.26$
| $76.6$
---
$\pm$ $0.24$
100% trained |  | $53.5$
---
$\pm$ $2.47$
| $88.7$
---
$\pm$ $0.53$
| $87.6$
---
$\pm$ $1.58$
| $85.2$
---
$\pm$ $0.36$
| $86.1$
---
$\pm$ $0.18$
| $80.2$
---
$\pm$ $0.13$
| $87.5$
---
$\pm$ $0.34$
| $61.5$
---
$\pm$ $0.97$
| $79.2$
---
$\pm$ $0.30$
| $76.7$
---
$\pm$ $0.25$

  * 1

Figures from Wang et al. (2018).

  * 2

Figures from Phang et al. (2019).

  * 3

Figures from Sanh et al. (2020).

  * 4

Figures from Jiao et al. (2020).

  * 5

Figures from the official ELECTRA’s Github repository. The associated GLUE
score cannot be computed as F1 scores and Person correlation are unavailable.

  * †

Figures are the average of the 2 metrics (F1 and accuracy for MRPC and QQP;
and Person and Spearman correlation for STS) and are not comparable.

  * *

GLUE scores are recomputed with the required metrics for each GLUE tasks. For
WNLI. as the majority class predictor beats all models, the GLUE scores use
56.34% accuracy for this specific task.

Robustness: Despite the reported bug333https://github.com/google-
research/electra/issues/51 from the original implementation regarding layer-
wise learning rate decay; the specific data augmentation procedure for STS and
MRPC task444https://github.com/google-research/electra/issues/98, which is not
used in this reimplementation; and the minor differences in preprocessing
steps, see section 3.2, the results are very close between this
reimplementation and the original one (Clark et al. (2020)). Thus, we can
conclude the ELECTRA approach is robust and generalizable as results are
similar despite different implementations and different datasets (Wikibooks
and OWT).

Efficiency: I also compared results from this reimplementation to the
literature with the hardware requirements. Findings are summarized in table 3.
By looking at the computation requirements, estimated by heuristics (see
OpenAI blog on AI and Compute), relative to GLUE scores, ELECTRA (Clark et al.
(2020)) is outperforming all models. Interestingly, this observation is also
valid for models using knowledge distillation such as DistilBERT (Sanh et al.
(2020)) or TinyBERT (Jiao et al. (2020), even if we don’t take into account
the initial cost for the teacher model. A similar conclusion can be made by
looking only at number of parameters, which is somehow an approximate proxy
for inference time. Therefore, ELECTRA is outperforming all compared
approaches in term of efficiency.

Table 3: Efficiency metrics for results on GLUE dev set. Peta-flops are
estimated using heuristic from theoritical FLOPS per GPU; number of GPUs; and
their utilization rate (see OpenAI blog on AI and Compute). Lower pfs-days is
better. The compute cost is overstated for RTX 3090 due to the constraint to
use only 16GB of GPU memory instead of the full 24GB.
(Pfs-days: peta-flops day; AVG: Average of individual metrics as ELECTRA
(Clark et al. (2020))

Model | Parameters |  | Train time +
---
hardware
Pfs-days† | AVG‡ | GLUE‡ |  | Pfs-day
---
per AVG
%
| Pfs-day
---
per GLUE
%
Baselines
ELMo |  $93$Ma | 14d on 3 GTX 1080b | $\approx 0.12$ | $65.1$ | $63.8$ |  $\approx 0.19$ ($3.5$x) |  $\approx 0.19$ ($3.4$x)
ELMo frozen |  $93$Ma | 14d on 3 GTX 1080b | $\approx 0.12$ | $70.7$ | $68.7$ |  $\approx 0.17$ ($3.2$x) |  $\approx 0.18$ ($3.2$x)
GPT |  $110$Mc | 30d on 8 P600d | $\approx 0.95$ | $77.9$ | $75.4$ |  $\approx 1.22$ ($22$x) |  $\approx 1.22$ ($22$x)
DistilBERT6 |  $67$Me |  $90$h on 8 V100f | $\approx 0.16$ |  | $77.0$ |  |  $\approx 0.21$ ($3.6$x)
TinyBERT6 |  $67$Me |  |  | $82.9$ | $80.0$ |  |
BERT-Base |  $110$Mg |  $4$d on 16 TPUg | $\approx 0.95$ |  | $80.0$ |  |  $\approx 1.19$ ($21$x)
ELECTRA-Small OWT - Original
$100\%$ trained |  $14$Mh |  $4$d on 1 V100h | $\approx 0.02$ | $80.4$ |  |  $\approx 0.03$ ($0.5$x) |
ELECTRA-Small OWT - Mine
100% trained |  $14$M |  $3.75$d on 1 RTX 3090 | $\approx 0.04$ | $79.2$ | $76.7$ |  $\approx 0.05$ ($1.0$x) |  $\approx 0.06$ ($1.0$x)

  * a

Information from Allen AI website .

  * b

Information from ELMo github repository .

  * c

Information from HuggingFace.

  * d

Information from OpenAI blog on AI and Compute.

  * e

Information from Jiao et al. (2020).

  * f

Information from Sanh et al. (2020).

  * g

Information from Devlin et al. (2019).

  * h

Information from Clark et al. (2020).

  * †

Formula from OpenAI blog on AI and Compute.
Assumptions: 8.9TFLOPS GTX 1080; 12TFLOPS for P600; 16TFLOPS for V100;
45TFLOPS for TPU; 35TFLOPS for RTX 3090; 0.33 utilization rate.

  * ‡

See table 2.

###  4.2 Sensitivity to generator size

In this experiment, different models, with different generator capacity and
with the same discriminator capacity, are pretrained like in section 4.1 with
that the exception the training was stopped at 250k steps ($25\%$). The
results are summarized in the table 4.

Training stability: Interestingly, in case of larger generator model than
recommended in the original paper (Clark et al. (2020)), in this
reimplementation, the discriminator network can collapse by not being able to
distinguish which tokens are fake or not. Therefore, even if ELECTRA is not
using an adversarial setting (Goodfellow et al. (2014)), the training
procedure may collapse if we allocate too much capacity to the generator
compared to the discriminator. Please refer to figure 2 for a visual
representation.

Table 4: Results on GLUE dev set for different generator capacities trained
with 250k steps. Results are reported as mean and standard deviation, using 5
different seeds per task. Gen.Size represents the multiplier for hidden-size,
FFN-size and num-attention-heads for the generator network compared to the
discriminator network.
(mc: Matthews correlation, acc: Accuracy, spc: Spearman correlation, AVG: Average of individual metrics as Clark et al. (2020)) Model |  | CoLA
---
(mc)
| SST-2
---
(acc)
| MRPC
---
(acc)
| STS-B
---
(spc)
| QQP
---
(acc)
| MNLI
---
(acc)
| QNLI
---
(acc)
| RTE
---
(acc)
AVG | GLUE*
ELECTRA-Small |  |  |  |  |  |  |  |  |  |
12.5% Gen.Size |  | $35.5$
---
$\pm$ $3.51$
| $85.9$
---
$\pm$ $0.58$
| $77.6$
---
$\pm$ $0.75$
| $82.0$
---
$\pm$ $0.37$
| $83.7$
---
$\pm$ $0.12$
| $76.1$
---
$\pm$ $0.17$
| $84.9$
---
$\pm$ $0.45$
| $55.0$
---
$\pm$ $4.65$
| $73.1$
---
$\pm$ $0.90$
| $71.4$
---
$\pm$ $0.80$
25% Gen.Size |  | $50.2$
---
$\pm$ $1.38$
| $86.6$
---
$\pm$ $0.66$
| $85.3$
---
$\pm$ $0.74$
| $83.9$
---
$\pm$ $0.61$
| $85.0$
---
$\pm$ $0.09$
| $77.9$
---
$\pm$ $0.19$
| $86.0$
---
$\pm$ $0.14$
| $58.2$
---
$\pm$ $2.52$
| $77.1$
---
$\pm$ $0.33$
| $74.8$
---
$\pm$ $0.29$
50% Gen.Size | Discriminator collapses at 10K steps
75% Gen.Size | Discriminator collapses at 10K steps
100% Gen.Size | Discriminator collapses at 10K steps

Figure 2: Generator (left for the accuracy metric) and discriminator (right
for the AUC metric) behaviours for different generator size relative to the
discriminator. For larger generator size than 25%, the discriminator may be
unable to distinguish if inputs are fake or not.

##  5 Discussion

###  5.1 What was easy

Ability to run on small GPU: One claim of ELECTRA (Clark et al. (2020)) is the
relative low computation requirements. In this work, only runs with a single
GPU have been executed. Furthermore, the GPU memory requirement, roughly 15GB,
can be easily decreased with the use of gradient accumulation. As example,
with 2 gradient accumulation steps, the pretraining can be performed on a 11GB
GPU.

Clarity of the original paper: The original paper (Clark et al. (2020))
contains a lot of detailed information, including the hyperparameters used for
the pretraining and downstream tasks. The proposed pretraining task is also
well explained and the replication was fairly easy as such. The only exception
was the unavailability of all GLUE metrics, such as F1 score or Person
correlation. The section about negative results is also very beneficial to
better understand some early explorations which has lead the Authors to
propose the ELECTRA approach. It would be beneficial for all if such openness
about negative results would be more systematically present in research
papers, along the positive results.

Original source code and documentation: Authors of the original paper (Clark
et al. (2020)) have released their source code in their Github
repository555https://github.com/google-research/electra. While the source code
was developed in Tensorflow instead of the target framework, PyTorch, for this
reimplementation, the original source code was relatively easy to understand.
Plenty of documentation and answers were already present in this Github
repository.

###  5.2 What was difficult

GLUE benchmark: The GLUE benchmark is composed on 9 different tasks. While
each task is fairly simple to implement, implementing all of them to use the
same pipeline requires some software engineering effort, at least further than
expected. More importantly for comparison, results are subject to high
variance and therefore, most papers use some statistics over several runs,
usually mean or median. The multitude of runs increase significantly the
computation budget. It would also be beneficial to also report the variance of
the results to have a better sense of this variance, and to ease comparisons
between models. Finally, papers use different way to report results, such as
meta score like in Devlin et al. (2019), or different metrics for MRPC, STS or
QQP, for example Clark et al. (2020) and Sanh et al. (2020). More practically,
the validation of this ELECTRA reimplemention, compared to the original one,
was harder due these reporting differences.

Additional tricks for GLUE benchmark: Submissions for the GLUE benchmark have
been using additional tricks to improve performances, such as ensembling,
further pretraining, multi-task training vs single task training, different
hyperparameters between tasks. For example, while this has been mentioned in
the original paper, see appendix from Clark et al. (2020), one specific task
data augmentation procedure is also used for the results submissions for MRPC
and STSB tasks. All these tricks make harder to compare different models, as
described in Aßenmacher and Heumann (2020).

### Communication with original authors

Kevin Clark, one of the original authors (Clark et al. (2020)), has been
helpful by answering some questions. Unfortunately, breakdown of GLUE score
per tasks have not yet been provided to fully compare this implementation with
the original one. Otherwise, most of questions that I had were already
answered though the Github repository or by inspecting the source code.

## References

  * Aßenmacher and Heumann [2020] Matthias Aßenmacher and Christian Heumann.  On the comparability of pre-trained language models, 2020.
  * Biewald [2020] Lukas Biewald.  Experiment tracking with weights and biases, 2020.  URL https://www.wandb.com/.  Software available from wandb.com.
  * Clark et al. [2020] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  Electra: Pre-training text encoders as discriminators rather than generators, 2020.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805, 2018.  URL http://arxiv.org/abs/1810.04805.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Gokaslan and Cohen [2019] Aaron Gokaslan and Vanya Cohen.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus, 2019.
  * Goodfellow et al. [2014] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.
  * Howard and Ruder [2018] Jeremy Howard and Sebastian Ruder.  Universal language model fine-tuning for text classification, 2018.
  * Jiao et al. [2020] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  Tinybert: Distilling bert for natural language understanding, 2020.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization, 2019.
  * Peters et al. [2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations, 2018.
  * Phang et al. [2019] Jason Phang, Thibault Févry, and Samuel R. Bowman.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks, 2019.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training, 2018.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners, 2019.
  * Sanh et al. [2020] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-5446.  URL https://www.aclweb.org/anthology/W18-5446.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In Qun Liu and David Schlangen, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020_ , pages 38–45. Association for Computational Linguistics, 2020.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.6/.
  * Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s neural machine translation system: Bridging the gap between human and machine translation, 2016.
  * Zhu et al. [2015] Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, 2015.

## Appendix

Table 5: Hyper-pameters for the pretraining task.  Hyperparameter |  | ELECTRA-Small
---
Original
| ELECTRA-Small
---
Mine
Underlying base architecture | Transformer | Transformer
Number of layers at token level | 12 | 12
Number of layers at sentence level | N/A | N/A
Hidden size | 256 | 256
FFN inner hidden size | 1024 | 1024
Attention heads | 4 | 4
Attention head size | 64 | 64
Embedding size | 128 | 128
Token embeddings | Yes | Yes
Position embeddings | Yes | Yes
| Generator size
---
(multiplier for hidden-size, FFN-size, and num-attention-heads)
0.25 | 0.25
| Generator layer size
---
(multiplier for number of layers)
1.0 | 1.0
Mask percent | 15 | 15
Learning rate | $5\mathrm{e}{-4}$ | $5\mathrm{e}{-4}$
Learning rate decay | Linear | Linear
Warmup steps | 10000 | 10000
Optimizer | Adam | AdamW
Adam $\epsilon$ | $1\mathrm{e}{-6}$ | $1\mathrm{e}{-6}$
Adam $\beta_{1}$ | 0.9 | 0.9
Adam $\beta_{2}$ | 0.9999 | 0.9999
Attention dropout | 0.1 | 0.1
Dropout | 0.1 | 0.1
Weight decay | 0.01 | 0.01
Batch size | 128 | 128
Gradient accumulation steps | 1 |  |  2 on 11GB GPU
---
1 on 24GB GPU
Mixed precision | No | Yes
Dataset | Wikibooks | OpenWebText
Tokenizer | WordPiece | BBPE
Vocab size | 30522 | 30522
Train steps | 1M | 1M

Table 6: Hyper-parameters for the GLUE tasks. Hyperparameter |  | ELECTRA-Small
---
Original
| ELECTRA-Small
---
Mine
Learning rate | $3\mathrm{e}{-4}$ | $3\mathrm{e}{-4}$
Layerwise LR decay | 0.8666The original implementation has got a bug which involves using a lower layerwise learning rate decay, see https://github.com/google-research/electra/issues/51 | 0.8
Learning rate decay | Linear | Linear
Warmup steps | 10000 | 10000
Optimizer | Adam | AdamW
Adam $\epsilon$ | $1\mathrm{e}{-6}$ | $1\mathrm{e}{-6}$
Adam $\beta_{1}$ | 0.9 | 0.9
Adam $\beta_{2}$ | 0.9999 | 0.9999
Attention dropout | 0.1 | 0.1
Dropout | 0.1 | 0.1
Weight Decay | 0 | 0
Batch size | 32 | 32
Gradient accumulation steps | 1 | 1
Mixed precision | No | Yes
Epochs |  | 10 for RTE and STS
---
3 for other GLUE benchmarks
| 10 for RTE and STS
---
3 for other GLUE benchmarks
Pooling | First contextualized embedding | Average pooling

(a) Original implementation (Clark et al. [2020])888Figures from
https://github.com/google-research/electra/issues/3

(b) This reimplementation

Figure 3: Metrics and losses from generator and discriminator networks