# Experimental Validation of Linear and Nonlinear MPC on an Articulated
Unmanned Ground Vehicle

Erkan Kayacan, , Wouter Saeys, Herman Ramon, Calin Belta, and Joshua Peschel
This work was supported by the IWT-SBO 80032 (LeCoPro) project funded by the
Institute for the Promotion of Innovation through Science and Technology in
Flanders (IWT-Vlaanderen).E. Kayacan is with Senseable City Laboratory and
Computer Science $\&$ Artificial Intelligence Laboratory, Massachusetts
Institute of Technology, Cambridge, Massachusetts 02139, USA. e-mail:
<EMAIL_ADDRESS>H. Ramon and W. Saeys are with the Division of Mechatronics,
Biostatistics and Sensors, Department of Biosystems, University of Leuven (KU
Leuven), 3001 Leuven, Belgium. e-mail: {herman.ramon,
<EMAIL_ADDRESS>C. Belta is with the Division of Systems
Engineering, Boston University, Boston, MA, USA<EMAIL_ADDRESS>e-mail:
<EMAIL_ADDRESS>J. Peschel is with the Department of Agricultural and Biosystems
Engineering, Iowa State University, Ames, Iowa 50011-3270, USA. e-mail:
<EMAIL_ADDRESS>
###### Abstract

This paper focuses on the trajectory tracking control problem for an
articulated unmanned ground vehicle. We propose and compare two approaches in
terms of performance and computational complexity. The first uses a nonlinear
mathematical model derived from first principles and combines a nonlinear
model predictive controller (NMPC) with a nonlinear moving horizon estimator
(NMHE) to produce a control strategy. The second is based on an input-state
linearization (ISL) of the original model followed by linear model predictive
control (LMPC). A fast real-time iteration scheme is proposed, implemented for
the NMHE-NMPC framework and benchmarked against the ISL-LMPC framework, which
is a traditional and cheap method. The experimental results for a time-based
trajectory show that the NMHE-NMPC framework with the proposed real-time
iteration scheme gives better trajectory tracking performance than the ISL-
LMPC framework and the required computation time is feasible for real-time
applications. Moreover, the ISL-LMPC produces results of a quality comparable
to the NMHE-NMPC framework at a significantly reduced computational cost.

###### Index Terms:

Articulated unmanned vehicle, autonomous system, input-state linearization,
model predictive control.

##  I Introduction

The size of arable farmland on the earth has been decreasing while human
population has been increasing outstandingly. It is expected that the
population of the world will reach $9.1$ billion by 2050. Therefore,
agricultural production will have to be double in order to feed a larger
population and provide increasing demands for bioenergy [1]. To meet the
demand for agricultural products, one possible solution is the automation of
agricultural machines to get higher efficiencies and better precisions.
Moreover, multitasking operations are needed in agricultural applications. For
instance, a human operator simultaneously has to drive the agricultural
vehicle with high precision, and adjust the position of a trailer and/or
further parameters of several agricultural apparatus during tillage and
planting. In this instance, a sophisticated and versatile control algorithm
for the navigation of unmanned agricultural ground vehicles is a necessity to
lead to an additional increment in the performance of the human operator.

The current implementations for automatic guidance of autonomous ground
vehicles are based on either local positioning systems (vision or laser-based
sensors) or global positioning systems (GPSs). Local positioning systems have
been used in autonomous applications since the 70s [2, 3]. It has been
reported that their main disadvantage is the sensitivity to light conditions
in outdoor environments although they are cheap to implement [4]. Recent
developments in satellite technologies have led to an increase in the use of
the latter, which has gradually replaced the former prevalent in the 90s [5,
6]. Real-time kinematic (RTK) GPS yielding centimeter precision [7] has
enabled intensive research on agricultural vehicles. Automated agricultural
vehicles with GPSs have many advantages, such as extricating the driver from
tiresome tasks of accurately steering the vehicle, increasing trajectory
tracking accuracy, and being able to operate at night or in foggy weather.

Various control techniques have been used to solve the trajectory tracking
problem for tractors with and without trailers [8, 9]. An adaptive controller
was employed for a tractor assembled with dissimilar trailers in order to
track straight lines [10]. Moreover, a linear optimal control method was
proposed for a tractor-trailer system [11]. These controllers have been
contingent on linearized dynamic and kinematic models, which are only valid
for small yaw deviations around a fixed value and small steering angles, such
that they are restricted to linear trajectories.

Model predictive control (MPC) is a popular technique in the process industry
for multi-input-multi-output applications [12, 13]. Forasmuch as the tractor-
trailer system can be described by variable set points for following
curvilinear trajectories, this involves a merger between MPC structure and a
nonlinear model as known nonlinear MPC (NMPC). NMPC was designed for a
tractor-trailer system along with a curvilinear trajectory in [14] while an
extended Kalman filter (EKF) was designed to estimate of the yaw angles of the
tractor and trailer. In [15], the states of an agricultural vehicle including
slips parameters were estimated with nonlinear moving horizon estimation
(NMHE) and forwarded to an NMPC. This concept has been extended for the
tracking of a space-based trajectory by a tractor-trailer system in a
centralized control structure and accomplished results have been reported in
[16]. Moreover, decentralized and distributed NMPC approaches have been
recommended to reduce the computational burden with minimal loss of tracking
performance [17, 18] .

Although trajectory tracking performance obtained in the aforementioned
studies is quite good, traditional NMPC implementations are computationally
expensive. Therefore, the aim of this study was to design fast frameworks for
trajectory tracking problem and compare their performance with regard to
computational burden and tracking error. Firstly, a fast NMHE-NMPC framework
is designed for tracking a time-based trajectory. In the NMHE-NMPC framework,
the NMHE learns traction parameters using onboard sensors online, and the NMPC
enables high accurate trajectory tracking. Thus, we provide robust tracking
performance when uncertainty is high as uncertainty is reduced through
learning whereas traditional NMPC approaches do not typically account for
model uncertainty. Moreover, a real-time iteration scheme is proposed to solve
NMHE and NMPC problems efficiently. Secondly, it is shown that the nonlinear
model is input-state linearizable and an LMPC is proposed for the linear
transformation of the system. Both the frameworks are then implemented on a
real-time system and benchmarked against each other.

This paper is organized as follows: The system is described in Section II. The
formulations and implementations of NMHE and NMPC are given in Section III.
The input-state linearization approach and LMPC control structure are
explained in Section IV. The real-time experimental results are given in
Section V. Finally, the study is summarized in Section VI.

##  II Unmanned Tractor-Trailer System

The goal of this paper is to obtain a precise trajectory tracking performance
to ensure constant distances between rows to prevent from crop damage while
variable soil conditions are subjected to an uneven, rough and wet grass
field. The small tractor-trailer system, the actuators, and the sensors are
shown in Fig. 1.

Figure 1: On the top left side: the electro-mechanical valve, on the bottom
left side: the electro-hydraulic valve, on the top right side: the
potentiometer, on the bottom right side: the articulated unmanned ground
vehicle.

In order to measure the global position of the system, a real-time kinematic
global positioning system (AsteRx2eH, Septentrio Satellite Navigation NV,
Belgium) with the aid of the Flepos network is used with two antennas
installed on the tractor and the trailer. There are three actuators to control
the system: two electro-hydraulic valves (OSPC50-LS/EH-20, Dan-foss, Nordborg,
Denmark) for the steering mechanisms and an electromechanical valve (LA12,
Linak, Nordborg, Denmark) for the hydrostat system. In addition, a
potentiometer (533-540- J00A3X0-0, Mobil Elektronik, Langenbeutingen,
Germany), an inductive sensor and two encoders mounted on the rear wheels are
used to measure respectively the angle of the front wheels of the tractor, the
steering angle of the trailer, and the speed of the system. A real time
operating system equipped with a 2.26 GHz Intel $Core^{TM}$2 Quad Q9100 quad-
core processor is used to implement the control algorithms that have been
executed and updated at a rate of 200 milliseconds in $LabVIEW^{TM}$.

Figure 2: A schematic representation of tricycle model for an unmanned
tractor-trailer system

The adaptive kinematic model for the unmanned tractor-trailer system is an
extended version of the one presented in [11, 19]. In order to make the system
model adaptive, three traction parameters ($\mu$, $\kappa$, $\eta$) are
inserted. The tractor and trailer rigid bodies are mechanically coupled by the
drawbar so that there exist two revolute joints (RJs) that interconnect the
drawbar to the tractor at $RJ^{1}$ and the drawbar to the trailer at $RJ^{2}$
as illustrated in Fig. 2 . The centers of gravity of the tractor and trailer
are respectively represented by $CG^{t}$ and $CG^{i}$. The equations for the
system, which are a combination of the kinematic model in [20] and the speed
model in [21], are written:

$\displaystyle\dot{x}^{t}$ $\displaystyle=$ $\displaystyle\mu
v\cos{(\psi^{t})}$ $\displaystyle\dot{y}^{t}$ $\displaystyle=$
$\displaystyle\mu v\sin{(\psi^{t})}$ $\displaystyle\dot{\psi}^{t}$
$\displaystyle=$ $\displaystyle\frac{\mu v\tan{(\kappa\delta^{t})}}{L^{t}}$
$\displaystyle\dot{x}^{i}$ $\displaystyle=$ $\displaystyle\mu
v\cos{(\psi^{i})}$ $\displaystyle\dot{y}^{i}$ $\displaystyle=$
$\displaystyle\mu v\sin{(\psi^{i})}$ $\displaystyle\dot{\psi}^{i}$
$\displaystyle=$ $\displaystyle\frac{\mu
v}{L^{i}}\Big{(}\sin{(\eta\delta^{i}+\beta)}+\frac{L^{d}}{L^{t}}\tan{(\kappa\delta^{t})}\cos{(\eta\delta^{i}+\beta)}\Big{)}$
$\displaystyle\dot{v}$ $\displaystyle=$
$\displaystyle-\frac{v}{\tau}+\frac{K}{\tau}HP$ (1)

where ${x}^{t}$ $(m)$, $y^{t}$ $(m)$, ${x}^{i}$ $(m)$, $y^{i}$ $(m)$,
$\psi^{t}$ $(rad)$ and $\psi^{i}$ $(rad)$ denote respectively the positions
and yaw angles of the tractor and trailer while $v$ $(m/s)$ denotes
circumferential speed of the wheels. The steering angles are respectively
denoted by $\delta^{t}$ $(rad)$ and $\delta^{i}$ $(rad)$ for the tractor and
trailer while the angle at $RJ^{1}$ and the hydrostat position are
respectively denoted by $\beta$ $(rad)$ and $HP$ $(\%)$. Moreover, $\mu$,
$\kappa$ and $\eta$ denote for the traction coefficients for the longitudinal
and side slips. It should be noted that the traction parameters can merely
acquire values between zero and one on asphalt roads and soil surfaces. If the
traction parameter for the longitudinal slip is equal to one, all the rotary
motion of the wheels is transformed into the linear motion of the vehicle.
Moreover, a proportion of zero expresses that tires are rotating and the
ground wheel speed is equal to zero so that the system is not completely
controllable. In other words, the definition of the traction parameter $\mu$
for longitudinal slip allows to convert the circumferential wheel speed $\nu$
into a _ground wheel speed $\mu\nu$_. The definition of the two traction
parameters for side-slips maintains a manner to determine _effective steering
angles_ , i.e., $\kappa\delta^{t}$ and $\eta\delta^{i}$. It is assumed that
each fraction for each steering angle ensures the actual tractor and trailer
turning motions. Furthermore, the difference between yaw angles is equal to
the summation of the angle at $RJ^{1}$ and the steering angle of the trailer,
i.e., $\phi^{i}-\phi^{t}=\beta+\delta^{i}$. Since there are constraints on
$\beta$ and $\delta^{i}$ defined respectively in (11) and (16), it is to be
noted that the yaw angle difference cannot become larger than
$\beta+\delta^{i}=45$ degrees.

The equations in (II) are formulated in the following form;

$\displaystyle\dot{x}=f\big{(}x,u,p\big{)}\quad and\quad
y=h\big{(}x,u,p\big{)}$ (2)

with

$\displaystyle x$ $\displaystyle=$
$\displaystyle\left[\begin{array}[]{ccccccc}x^{t}&y^{t}&\psi^{t}&x^{i}&y^{i}&\psi^{i}&v\end{array}\right]^{T}$
(4) $\displaystyle u$ $\displaystyle=$
$\displaystyle\left[\begin{array}[]{ccc}\delta^{t}&\delta^{i}&HP\end{array}\right]^{T}$
(6) $\displaystyle p$ $\displaystyle=$
$\displaystyle\left[\begin{array}[]{cccc}\mu&\kappa&\eta&\beta\end{array}\right]^{T}$
(8) $\displaystyle y$ $\displaystyle=$
$\displaystyle\left[\begin{array}[]{ccccccccc}x^{t}&y^{t}&x^{i}&y^{i}&v&\delta^{t}&\delta^{i}&HP&\beta\end{array}\right]^{T}$
(10)

where x, u, p and y denote respectively the vectors of state, input, varying
parameter and output of the system. The measured, fixed physical parameters
are the distance between the front and rear wheels of the tractor
$L^{t}=1.4m$, the distance between the $RJ^{2}$ and the rear wheel of the
trailer $L^{i}=1.3m$ and the distance between the rear wheel of the tractor
and $RJ^{2}$ $L^{d}=1.1m$. The identified, fixed parameters are [21]: the
time-constant $\tau=2.05$ and the gain $K=0.016$ for the wheel speed model
while the engine speed is at $2500$ RPM. The angle between the tractor and
drawbar $\beta$ is measured and the traction parameters $\mu,\kappa,\eta$ are
estimated online so that the parameters in (8) can vary over time.

##  III NMHE-NMPC Framework

NMHE-NMPC framework was develoepd and implemented for the space-based
trajectory approach in [13, 16, 17]. In this section, we will develop this
framework for the time-based trajectory approach.

###  III-A Nonlinear Moving Horizon Estimation

Although values of all system states must be gathered for NMPC, it is
impossible to measure all of them in practice. For this reason, it is a
requirement to estimate unmeasured states or unknown model parameters online.
The traditional method as a state estimator is the Extended Kalman filter
(EKF). However, the major drawback of the EKF is that it cannot take the
bounds on the states into account. To accomplish this restraint of the EKF in
this paper, NMHE has been employed inasmuch as it takes the state and
parameter estimation regarding bounds into account within the same problem
[15, 22, Kuhl2011].

In this paper, we consider an NMHE formulation in the following form at each
sampling time t:

$\displaystyle\underset{x(.),p,u(.)}{\text{min}}$
$\displaystyle\left\|\begin{array}[]{c}\hat{x}(t_{k}-t_{h})-x(t_{k}-t_{h})\\\
\hat{p}-p\end{array}\right\|^{2}_{P}+\int^{t_{k}}_{t_{k}-t_{h}}\|y_{m}-y(t)\|^{2}_{H}dt$
(11) s. t. $\displaystyle\dot{x}(t)=f\big{(}x(t),u(t),p\big{)}$ $\displaystyle
y(t)=h\big{(}x(t),u(t),p\big{)}$ $\displaystyle-20\deg\leq\beta\leq 20\deg$
$\displaystyle 0\leq\mu,\kappa,\eta\leq 1\quad\forall t\in[t_{k}-t_{h},t_{k}]$

where $y_{m}$ and $y$ denote respectively the measured output and the output
function of the system model. The deviations in the estimates for the states
and parameters before the estimation horizon $\hat{x}(t_{k}-t_{h})$ and
$\hat{p}(t_{k}-t_{h})$ are minimized by a symmetric positive semi-definite
weighting matrix $P$, while the deviations of the measured and system outputs
in the estimation horizon are minimized by a symmetric positive semi-definite
weighting matrix $H$ [23]. The first part of the cost function in (11) is
named the arrival cost and must be bounded as a requirement. If not, it may go
to infinity. Therefore, the impact of the old measurements on P is reduced by
a weighting matrix $D_{update}$ in (14) [Kuhl2011].

The NMHE method can estimate the immeasurable states and parameters of the
system model simultaneously. The parameters have been assumed to be time-
invariant and not subject to process noise over the estimation horizon.
However, it is assumed that the parameters are time-varying Gaussian random
variables in the arrival cost. Therefore, additional weighting factors must be
added as the variance of the parameters noise and the parameters appears only
in the arrival cost. The extended weighting matrix $D_{update}\in
R^{(n_{x}+n_{p})\times(n_{x}+n_{p})}$ can be written as follows:

$\displaystyle D_{update}=\left[\begin{array}[]{cc}D^{n_{x}}&0\\\
0&D^{n_{p}}\\\ \end{array}\right]$ (14)

where $D^{n_{x}}\in R^{n_{x}\times n_{x}}$ and $D^{n_{p}}\in R^{n_{p}\times
n_{p}}$ represent the weighting matrix for the state noise covariance and the
weighting matrix for the parameter pseudo-variance. The weighting matrix
$D_{update}$ is chosen based on the objective. Low gain in the process noise
results in better estimation accuracy; however, it causes time-lag between
true and estimated values. Therefore, the weighting coefficients for the
measured states and parameters (e.g., $x^{t}$, $y^{t}$, $x^{i}$, $y^{i}$, $v$
and $\beta$) are selected large while the weighting coefficients for the
immeasurable states and parameters (e.g., $\psi^{t}$, $\psi^{i}$, $\mu$,
$\kappa$ and $\eta$) are selected small in this paper. Thus, the input to the
NMHE algorithm becomes the output of the system in (10) while the output of
NMHE becomes the full states in (4) and full varying parameters in (8).
Moreover, the standard deviations of the measurements have been set to
$\sigma_{x^{t}}=\sigma_{y^{t}}=\sigma_{x^{i}}=\sigma_{y^{i}}=0.03$ m,
$\sigma_{\beta}=0.0175$ rad, $\sigma_{v}=0.1$ m/s,
$\sigma_{\delta^{t}}=0.0175$ rad, $\sigma_{\delta^{i}}=0.0175$ rad and
$\sigma_{HP}=3$ based on the information obtained from the real- time
experiments. The following weighting matrices $H$ and $D_{update}$ have been
used in NMHE:

$\displaystyle H$ $\displaystyle=$ $\displaystyle
diag(\sigma^{2}_{x^{t}},\sigma^{2}_{y^{t}},\sigma^{2}_{x^{i}},\sigma^{2}_{y^{i}},\sigma^{2}_{v},\sigma^{2}_{\delta^{t}},\sigma^{2}_{\delta^{i}},\sigma^{2}_{HP},\sigma^{2}_{\beta})^{-1}$
$\displaystyle D_{update}$ $\displaystyle=$ $\displaystyle
diag(x^{t},y^{t},\psi^{t},x^{i},y^{i},\psi^{i},\mu,\kappa,\eta,\beta,v)$ (15)
$\displaystyle=$ $\displaystyle diag(10.0,10.0,0.1,10.0,10.0,0.1,$
$\displaystyle 0.25,0.25,0.25,1,1)$

The estimation horizon $t_{h}$ has been set to 3 seconds.

###  III-B Nonlinear Model Predictive Control

A nonlinear model represented in (2)
$f(\cdot,\cdot,\cdot):\mathbb{R}^{n_{x}}\times\mathbb{R}^{n_{u}}\longrightarrow\mathbb{R}^{n_{x}}$
is the continuously state update function and $f(0,0,p)=0\quad\forall t$ in
which $x$ $\in$ $\mathbb{R}^{n_{x}}$ and $u$ $\in$ $\mathbb{R}^{n_{u}}$ are
the state and input vectors. The states and inputs have to fulfill
$x\in\mathbb{X},\;\;\;u\in\mathbb{U}$ where
$\mathbb{X}\subseteq\mathbb{R}^{n_{x}}$ is closed,
$\mathbb{U}\subseteq\mathbb{R}^{n_{u}}$ is compact and each set contains the
origin in its interior point.

In this study, we consider an NMPC formulation at each sampling time $t$ in
the following form:

$\displaystyle\underset{x(.),u(.)}{\text{min}}$
$\displaystyle\int^{t_{k}+t_{h}}_{t_{k}}\Big{(}\|x_{r}(t)-x(t)\|^{2}_{Q}+\|\Delta
u(t)\|^{2}_{R}\Big{)}dt$ (16)
$\displaystyle+\|x_{r}(t_{k}+t_{h})-x(t_{k}+t_{h})\|^{2}_{S}$ s.t.
$\displaystyle x(t_{k})=\hat{x}(t_{k})$
$\displaystyle\dot{x}(t)=f\big{(}x(t),u(t),p\big{)}$
$\displaystyle-35\deg\leq\delta^{t}(t)\leq 35\deg\ $
$\displaystyle-25\deg\leq\delta^{i}(t)\leq 25\deg$ $\displaystyle 0\%\leq
HP(t)\leq 100\%\qquad\forall t\in[t_{k},t_{k}+t_{h}]$

where the first and last parts are called the stage cost and the terminal
penalty enforced the stability of NMPC in [24] in which
$Q\in\mathbb{R}^{n_{x}\times n_{x}}$, $R\in\mathbb{R}^{n_{u}\times n_{u}}$ and
$S\in\mathbb{R}^{n_{x}\times n_{x}}$ are symmetric positive definite weighting
matrices, $x_{r}$ denotes respectively the references for the states, $x$ and
$\Delta u$ denote respectively the states and the change of the inputs,
$t_{k}$ denotes the current time, $t_{h}$ denotes the prediction horizon.
$\hat{x}(t_{k})$ denotes the estimated state vector by the NMHE. The first
sample of $u(t)$, $u(t,x(t))=u^{*}(t_{k})$, is applied to the system and the
NMPC problem is solved again over a moving horizon for the subsequent sampling
time [13].

The references for the state are written:

$\displaystyle x_{r}$ $\displaystyle=$
$\displaystyle(x^{t}_{r},y^{t}_{r},\psi^{t}_{r},x^{i}_{r},y^{i}_{r},\psi^{i}_{r},v_{r})^{T}$
(17)

The weighting matrices Q, R and S have been written:

$\displaystyle Q$ $\displaystyle=$ $\displaystyle diag(2,2,0,4,4,0,0),\quad
S=10\times Q$ $\displaystyle R$ $\displaystyle=$ $\displaystyle diag(7,7,7)$
(18)

The weighting matrix R is selected larger than the weighting matrix Q so as to
obtain well damped closed-loop system response. The other justification is
that the system dynamics are slow so that it is not able to give a rapid
reaction. Inasmuch as the last state error value in the prediction horizon is
so crucial for the stability issues, the weighting matrix S is adjusted to 10
times larger than the weighting matrix Q.

If the prediction and control horizons are selected large, the computation
burden for NMPC will increase unreasonably so that solving the optimization
problem will be infeasible. Moreover, if the prediction and control horizons
are selected too small, the stabilization of the system may not be achieved.
As reported in [25], the prediction and control horizons of the NMPC must be
large enough for a stable performance taking the velocity of the vehicle into
consideration. Since the velocity of the tractor-trailer system is quite low,
the prediction and control horizons do not have to be very large in this
study. Therefore, the prediction and control horizons $t_{h}$ have been set to
3 seconds.

###  III-C Implementation

The optimization problems in NMHE (11) and NMPC (16) are very similar so that
using the same solution method for both of them makes sense [15]. Inasmuch as
they are nonlinear and non-convex optimization problems, the computational
burden for solving these problems is quite large, and depends on the order of
the system, the non-linearity of the system, the horizon length and used
nonlinear optimization solver.

In this study, the multiple shooting method has been consolidated with a
generalized Gauss-Newton method [26]. The significant benefit is that the
second derivatives that are arduous computing are not necessary. However, the
drawback is that it is troublesome to foreknow the required number of
iterations to attain a desired accuracy [27]. A simple solution that limits
the number of iterations to 1 was proposed in [28]. Moreover, the Gauss-Newton
iteration is divided into two parts: preparation and feedback parts. The
preparation part is executed prior to the feedback part, and the feedback part
is executed after measurements for NMHE and estimates for NMPC are available.
In the preparation part, the system dynamics are integrated with the previous
solution, and objectives, constraints, and corresponding sensitives are
evaluated. In the feedback part, a single quadratic programming is solved with
the current measurements for the NMHE and the current estimates for the NMPC.
Thus, the new estimates for the NMHE and a new control signal for the NMPC are
obtained. Compared to the classical method, this method minimizes feedback
delay and produces similar results with higher computational efficiency [28].
Furthermore, the NMPC and NMHE are run in parallel, on separate processor
cores, the NMHE preparation step is triggered at the same time as the NMPC
feedback step. Therefore, this solution method reduces the overall required
time for the preparation steps of the NMHE-NMPC. The ACADO code generation
tool has been used to solve the constrained nonlinear optimization problems in
the NMPC and NMHE [27]. Moreover, qpOASES software package, which is an open-
source C++ implementation of online active set strategy, has been used as a QP
solver [29].

##  IV ISL-LMPC Framework

###  IV-A Input-State Linearization Transformation

In the ISL-LMPC framework, an EKF has been used as an estimator. Since an EKF
is not able to deal with the bounds on the states and parameters, we exclude
the traction parameters for this framework.

The input-state linearization is a useful method to compensate the non-
linearity of a system. In this section, a nonlinear model for the tractor-
trailer system excluding the traction parameters in (II) is transformed into a
virtual linear model by using an input-state linearization method. Once a
virtual linear model has been obtained, linear control techniques are used to
design a controller for the overall system.

By taking a nonlinear system in (2) into account in which $f(x(t),u(t),p)$ is
input-state linearizable if there exists a diffeomorphism such that the new
state variables $z=T_{x}(x)$ transform the nonlinear system in (2) into the
following linear time-invariant system [30]:

$\dot{z}=Az+Bu_{z}$ (19)

where the pair (A,B) is controllable. The transformation between the real and
virtual control inputs resulting in the compensation of the system
nonlinearities and a controllable linear system can be written as follows:

$u=\phi(x)+T_{u}(x)u_{z}$ (20)

where $T_{u}(x)$ is assumed to be non-singular [30].

The new states, the positions and velocities of the tractor and trailer, are
defined as follows:

$\displaystyle z$ $\displaystyle=$
$\displaystyle\left[\begin{array}[]{cccccccc}x^{t}&y^{t}&v\cos{\psi^{t}}&v\sin{\psi^{t}}&x^{i}&y^{i}&v\cos{\psi^{i}}&v\sin{\psi^{i}}\end{array}\right]^{T}$
(22) $\displaystyle=$
$\displaystyle\left[\begin{array}[]{cccccccc}z_{1}&z_{2}&z_{3}&z_{4}&z_{5}&z_{6}&z_{7}&z_{8}\end{array}\right]^{T}$
(24)

By combing the time derivative of (22) with the equations for the yaw angles
and longitudinal speed model in (II), the state-space model can be written as
follows:

$\displaystyle\dot{z}$ $\displaystyle=$ $\displaystyle Az+Bu_{z}$ (25)
$\displaystyle y_{z}$ $\displaystyle=$ $\displaystyle Cz$ (26)

where
$\dot{z}_{1}=\dot{z}_{3},\dot{z}_{2}=\dot{z}_{4},\dot{z}_{3}=-z_{3}/\tau+u_{z_{1}},\dot{z}_{4}=-z_{4}/\tau+u_{z_{2}},\dot{z}_{5}=\dot{z}_{7},\dot{z}_{6}=\dot{z}_{8},\dot{z}_{7}=-z_{7}/\tau+u_{z_{3}},\dot{z}_{8}=-z_{8}/\tau+u_{z_{4}}$
and
$y_{z}=\left[\begin{array}[]{cccc}z_{1}&z_{2}&z_{5}&z_{6}\end{array}\right]^{T}$

As can be seen from the formulation above, there are 4 inputs for the virtual
linear system even though the number of inputs for the real-time system is
equal to 3. This results in 2 input transformations for the hydrostat position
HP. One of these transformations is based on the position of the tractor while
the other is calculated with respect to the information coming from the
trailer. Since the hydrostat position HP is the input for the speed measured
by encoders mounted on the tractor rear wheels, the transformation obtained
from the equations of the tractor is used for the hydrostat position
transformation. Moreover, the steering angle of the trailer is not input-
linearizable for the transformation. Therefore, we have to rely on the small
steering angle assumption so that the term $\cos(\delta^{i}+\beta)$ is assumed
to be equal 1. Thus, the total input transformation can be written as follows:

$\displaystyle\delta^{t}$ $\displaystyle=$
$\displaystyle\arctan{\Big{(}\frac{L^{t}(-u_{z_{1}}\sin{\psi^{t}}+u_{z_{2}}\cos{\psi^{t}})}{v^{2}}\Big{)}}$
$\displaystyle\delta^{i}$ $\displaystyle=$
$\displaystyle\arcsin\Big{(}\frac{L^{i}(-u_{z_{3}}\sin{\psi^{i}}+u_{z_{4}}\cos{\psi^{i}})}{{v^{2}}}$
$\displaystyle-\frac{L^{d}}{L^{t}}(-u_{z_{1}}\sin{\psi^{t}}+u_{z_{2}}\cos{\psi^{t}})\Big{)}-\beta$
$\displaystyle HP$ $\displaystyle=$
$\displaystyle\frac{\tau}{K}\Big{(}u_{z_{1}}\cos{\psi^{t}}+u_{z_{2}}\sin{\psi^{t}}+\frac{v}{\tau}\Big{)}$
(27)

###  IV-B Linear Model Predictive Control

In this study, we considered the following LMPC formulation at each sampling
time $t$:

$\displaystyle\underset{x(.),u(.)}{\text{min}}$
$\displaystyle\int^{t_{k}+t_{h}}_{t_{k}}(\|z_{r}(t)-z(t)\|^{2}_{Q}+\|\triangle
u_{z}(t)\|^{2}_{R})dt$ (28)
$\displaystyle+\|z_{r}(t_{k}+t_{h})-z(t_{k}+t_{h})\|^{2}_{S}$ s. t.
$\displaystyle\dot{z}(t)=Az(t)+Bu_{z}(t)$ $\displaystyle-2\leq
z_{3}(t),z_{4}(t),z_{7}(t),z_{8}(t)\leq 2\quad\forall t\in[t_{k},t_{k}+t_{h}]$
$\displaystyle-2\leq
z_{3}(t_{k}+t_{h}),z_{4}(t_{k}+t_{h}),z_{7}(t_{k}+t_{h}),z_{8}(t_{k}+t_{h})\leq
2$

where $z_{r}$ is the references for the system states and $\triangle u_{z}$ is
the change of the input. The maximum speed of the system is $2$ $m/s$;
therefore, the constraints on $z_{3}$, $z_{4}$, $z_{7}$ and $z_{8}$ are
defined in the formulation of the LMPC.

The prediction horizon $t_{h}$ has been set to $3$ second. As motivated in
Section III-B, the prediction horizon must not be very large due to the fact
that the velocity of the system is too low. Moreover, the weighting matrices
$Q$, $R$ and $S$ have been defined as follows:

$\displaystyle Q$ $\displaystyle=$ $\displaystyle
diag(1,1,0,0,0.01,0.0,0,0),\quad S=10\times Q$ $\displaystyle R$
$\displaystyle=$ $\displaystyle diag(1,1,0.01,0.01)$ (29)

Figure 3: Control scheme for the LMPC by using the input-state linearization

As can be seen from (28), the LMPC formulation is a convex optimization
problem while the formulation for NMPC in (16) is the constrained nonlinear
optimization problem that is non-convex. Therefore, it should be noted that
the required computational time for LMPC is significantly less than the one
for NMPC. The LMPC was implemented by using the Model Predictive Control
toolbox in LabVIEW, which is a traditional method.

The block diagram of the control scheme for the IST-LMPC framework is shown in
Fig. 3. As can be seen in Fig. 3, the generated inputs for the linear system
are fed to the input linearization transformation to find proper inputs for
the real-time system. Similarly, the outputs of the real-time system are fed
to the state linearization transformation to calculate the states of the
linear system.

##  V Experimental Results

For an autonomous ground vehicle application, there are two types of reference
definitions: one is a time-based trajectory and the second is a space-based
trajectory. Whereas the longitudinal speed of the ground vehicle is constant
in the latter, it is controlled in the former [31, 20]. The space-based
trajectory approach is convenient in case of one vehicle in agricultural
operations. If several vehicles are operating cooperatively, some of them need
to be in a specific position in a specific time instant. For example, if a
combine harvester and multiple tractor-trailer combinations are operating
together, the tractor-trailer systems have to align with and follow the
combine harvester which may vary its speed to maximally use its capacity.
Therefore, the tractor-trailer systems should change their speed to get in
line with and keep track of the combine harvester. This cannot be obtained
with a space-based trajectory but requires the tracking of a time-based
trajectory approach. Another example is the tracking of path with variable
speed to adapt the machine to variable crop density. Therefore, a time-based
trajectory consisting of an 8-shaped trajectory has been used as a reference
signal. The 8-shaped trajectory consists of two smooth curvilinear lines and
two straight lines.

Throughout the experiments, the articulated unmanned ground vehicle has faced
with uneven terrain and the sampling time of the frameworks is $0.2$ second in
real-time. The autonomous tractor-trailer system has succeeded in staying on-
track for the NMHE-NMPC and ISL-LMPC frameworks as shown respectively in Figs.
4 and 4(a).

The Euclidean errors for the tractor and trailer are respectively shown for
the nonlinear and linear controllers in Figs. 4(b) and 4(c). By using the
NMHE-NMPC framework, the mean values of the Euclidean errors of the tractor
and trailer are obtained respectively $16.65$ cm and $10.32$ cm for the
straight lines while $33.09$ cm and $25.01$ cm for the curvilinear lines. it
is pointed out that that the trajectory tracking error for straight lines has
been less than the one for the curvilinear lines as shown in Fig. 4(b). The
same framework was implemented for the space-based trajectory method in [16]
while the time-based one has been used in this paper. The trajectory error to
the space-based trajectory was less than the one to the time-based trajectory
for straight lines, while it was more than the one to the time-based
trajectory for curvilinear lines. Therefore, it can be concluded that the
preferred approach depends on the shape of the trajectory. Moreover, an NMHE-
NMPC framework for the time-based approach was designed for agricultural
vehicles in [15]. It was reported that the Euclidean error was around $1$ m.
This shows the superiority of our frameworks.

(a)

(b)

(c)

(d)

(e)

Figure 4:  Experimental results (a) Reference and actual trajectories for NMPC
(b) Reference and actual trajectories for LMPC (c) Euclidean error to the
reference trajectory for NMPC (d) Euclidean error to the reference trajectory
for LMPC (e) Control signals (f) Traction parameters

By using the ISL-LMPC framework, the mean values of the Euclidean errors of
the tractor and the trailer have been obtained respectively $19.26$ cm and
$15.27$ cm for the straight lines while $37.01$ cm and $33.33$ cm for the
curvilinear lines as shown in Fig. 4(c). As reported in [32], the linear
control techniques are invalid for curvilinear trajectories. However, thanks
to the ISL transformation, the ISL-LMPC framework is capable of staying on-
track. When these two frameworks have been compared, it is seen that the ISL-
LMPC framework has performed worse than the NMHE-NMPC framework for both the
tracking of the straight and the curvilinear lines. In the ISL-LMPC framework,
the traction parameters are excluded from the model and the linearization of
the system is executed at every time-step. The aforementioned factors are the
reasons for the degraded performance.

The outputs of the controllers, which are the steering angles references for
the tractor and trailer ($\delta^{t}$, $\delta^{i}$), and the hydrostat
position (HP) reference, are illustrated in Fig. 4(d). As seen in these
figures, the control signals stay within the bounds and the control signals
generated by NMPC are more smooth than the ones generated by LMPC. The reason
for this difference is the high non-linearity of the input transformation for
the ISL-LMPC framework. Moreover, the estimated traction parameters by the
NMHE are shown in Fig. 4(e). The estimates stay within the bounds.

The execution times for NMHE, NMPC and LMPC are summarized in Table I.
Preparation time denotes required computation time to evaluate objective,
constraints and condensing procedure till all measurements are received, while
feedback time is the required computation time to compute linear term and
constraints bounds in condensed QP, and to send generated signals to
actuators. As seen from this table, the average computation times for NMHE and
NMPC were respectively equal to $6.8575$ ms and $5.3904$ ms. Thus, the overall
computation time for the NMHE-NMPC framework was equal to $12.2479$ ms. While
the maximal computation time for the NMPC has been still reasonable for in
real-time, the mean value of the computation time for LMPC has been $8$ times
lower with $1.2330$ ms. Moreover, it is required to monitor the Karush-Kuhn-
Tucker tolerances to check the optimality of the optimization problems for the
NMHE and NMPC. The mean values of the KKT tolerances were respectively
$7.8641$ $10^{-4}$ and $4.264$ $10^{-3}$. They are low enough to claim the
optimality.

TABLE I: Execution times of the NMHE, NMPC, and LMPC. | NMHE | NMPC | LMPC
---|---|---|---
Preparation step | 6.0637 | 5.1745 | 1.2000
Feedback step | 0.7938 | 0.2159 | 0.0330
Overall | 6.8575 | 5.3904 | 1.2330

##  VI Conclusions

The NMHE-NMPC and ISL-LMPC frameworks have been developed for the time-based
trajectory tracking problem of an articulated unmanned ground vehicle and
implemented on a real-time system. The experimental results have shown that
both frameworks are capable of keeping the system on-track. Thanks to the
1-step Gauss-Newton iteration principle, the computationally efficient NMHE-
NMPC framework requires a computation time of around $12$ ms, while the
computation time for the ISL-LMPC framework is less than $2$ ms. This
reduction on computational burden came at the price of a worse tracking error;
however, the ISL-LMPC framework can be used in case of limited computation
power in real-time.

Recent developments in microprocessors technology and fast solution tools for
NMPC have changed the well-known paradigm in a way that the belief of using
NMPC for only relatively slow dynamic systems is no longer true. The
comparative results presented in this paper also show that NMPC
implementations for fast robotic systems do not require enormous computation
power anymore.

## References

  * [1] FAO, “How to feed the world in 2050,” Food and Agriculture Organization of the United Nations (FAO), Tech. Rep., October 2009.
  * [2] A. Julian, “Design and performance of a steering control system for agricultural tractors,” _Journal of Agricultural Engineering Research_ , vol. 16, no. 3, pp. 324–336, 1971.
  * [3] J. Reid and S. Searcy, “Vision-based guidance of an agriculture tractor,” _Control Systems Magazine, IEEE_ , vol. 7, no. 2, pp. 39–43, 1987.
  * [4] S. A. Hiremath, G. W. Van Der Heijden, F. K. Van Evert, A. Stein, and C. J. Ter Braak, “Laser range finder model for autonomous navigation of a robot in a maize field using a particle filter,” _Computers and Electronics in Agriculture_ , vol. 100, pp. 41–50, 2014.
  * [5] P. W. Bradford, J. Spilker, and P. Enge, “Global positioning system: theory and applications,” _AIAA Washington DC_ , vol. 109, 1996.
  * [6] T. Bell, “Automatic tractor guidance using carrier-phase differential GPS,” _Computers and electronics in agriculture_ , vol. 25, no. 1-2, pp. 53–66, 2000.
  * [7] D. Bevly and B. Parkinson, “Cascaded kalman filters for accurate estimation of multiple biases, dead-reckoning navigation, and full state feedback control of ground vehicles,” _IEEE Transactions on Control Systems Technology_ , vol. 15, no. 2, pp. 199–208, March 2007.
  * [8] A. Khalaji and S. Moosavian, “Robust adaptive controller for a tractor-trailer mobile robot,” _Mechatronics, IEEE/ASME Transactions on_ , vol. 19, no. 3, pp. 943–953, June 2014.
  * [9] M. Michalek and M. Kielczewski, “The concept of passive control assistance for docking maneuvers with n-trailer vehicles,” _Mechatronics, IEEE/ASME Transactions on_ , vol. 20, no. 5, pp. 2075–2084, Oct 2015.
  * [10] J. B. Derrick and D. M. Bevly, “Adaptive steering control of a farm tractor with varying yaw rate properties,” _Journal of Field Robotics_ , vol. 26, no. 6-7, pp. 519–536, 2009.
  * [11] M. Karkee and B. L. Steward, “Study of the open and closed loop characteristics of a tractor and a single axle towed implement system,” _Journal of Terramechanics_ , vol. 47, no. 6, pp. 379 – 393, 2010.
  * [12] H. Li and W. Yan, “Model predictive stabilization of constrained underactuated autonomous underwater vehicles with guaranteed feasibility and stability,” _IEEE/ASME Transactions on Mechatronics_ , vol. 22, no. 3, pp. 1185–1194, June 2017.
  * [13] E. Kayacan, E. Kayacan, I.-M. Chen, H. Ramon, and W. Saeys, _On the Comparison of Model-Based and Model-Free Controllers in Guidance, Navigation and Control of Agricultural Vehicles_. Cham: Springer International Publishing, 2018, pp. 49–73.
  * [14] J. Backman, T. Oksanen, and A. Visala, “Navigation system for agricultural machines: Nonlinear model predictive path tracking,” _Computers and Electronics in Agriculture_ , vol. 82, pp. 32 – 43, 2012.
  * [15] T. Kraus, H. Ferreau, E. Kayacan, H. Ramon, J. D. Baerdemaeker, M. Diehl, and W. Saeys, “Moving horizon estimation and nonlinear model predictive control for autonomous agricultural vehicles,” _Computers and Electronics in Agriculture_ , vol. 98, pp. 25 – 33, 2013.
  * [16] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Learning in centralized nonlinear model predictive control: Application to an autonomous tractor-trailer system,” _Control Systems Technology, IEEE Transactions on_ , vol. 23, no. 1, pp. 197–205, Jan 2015.
  * [17] ——, “Robust tube-based decentralized nonlinear model predictive control of an autonomous tractor-trailer system,” _Mechatronics, IEEE/ASME Transactions on_ , vol. 20, no. 1, pp. 447–456, Feb 2015.
  * [18] ——, “Distributed nonlinear model predictive control of an autonomous tractor–trailer system,” _Mechatronics_ , vol. 24, no. 8, pp. 926 – 933, 2014.
  * [19] M. Karkee, “Modeling, identification and analysis of tractor and single axle towed implement system,” Ph.D. dissertation, Iowa State University, 2009.
  * [20] E. Kayacan, H. Ramon, and W. Saeys, “Robust trajectory tracking error model-based predictive control for unmanned ground vehicles,” _IEEE/ASME Transactions on Mechatronics_ , vol. 21, no. 2, pp. 806–814, April 2016.
  * [21] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Nonlinear modeling and identification of an autonomous tractor–trailer system,” _Computers and Electronics in Agriculture_ , vol. 106, no. 0, pp. 1 – 10, 2014.
  * [22] C. Rao, J. Rawlings, and D. Mayne, “Constrained state estimation for nonlinear discrete-time systems: stability and moving horizon approximations,” _Automatic Control, IEEE Transactions on_ , vol. 48, no. 2, pp. 246–258, 2003\.
  * [23] H. Ferreau, T. Kraus, M. Vukov, W. Saeys, and M. Diehl, “High-speed moving horizon estimation based on automatic code generation,” in _Decision and Control (CDC), 2012 IEEE 51st Annual Conference on_ , 2012, pp. 687–692.
  * [24] D. Mayne, J. Rawlings, C. Rao, and P. Scokaert, “Constrained model predictive control: Stability and optimality,” _Automatica_ , vol. 36, no. 6, pp. 789 – 814, 2000.
  * [25] P. Falcone, F. Borrelli, J. Asgari, H. Tseng, and D. Hrovat, “Predictive active steering control for autonomous vehicle systems,” _Control Systems Technology, IEEE Transactions on_ , vol. 15, no. 3, pp. 566–580, May 2007\.
  * [26] M. Diehl, H. G. Bock, and J. P. Schlöder, “A real-time iteration scheme for nonlinear optimization in optimal feedback control,” _SIAM Journal on Control and Optimization_ , vol. 43, no. 5, pp. 1714–1736, 2005.
  * [27] B. Houska, H. J. Ferreau, and M. Diehl, “Acado toolkit—an open-source framework for automatic control and dynamic optimization,” _Optimal Control Applications and Methods_ , vol. 32, no. 3, pp. 298 – 312, 2011.
  * [28] M. Diehl, H. Bock, J. P. Schlöder, R. Findeisen, Z. Nagy, and F. Allgower, “Real-time optimization and nonlinear model predictive control of processes governed by differential-algebraic equations,” _Journal of Process Control_ , vol. 12, no. 4, pp. 577 – 585, 2002.
  * [29] H. J. Ferreau, C. Kirches, A. Potschka, H. G. Bock, and M. Diehl, “qpoases: a parametric active-set algorithm for quadratic programming,” _Mathematical Programming Computation_ , vol. 6, no. 4, pp. 327–363, Dec 2014\.
  * [30] J.-J. E. Slotine, W. Li _et al._ , _Applied nonlinear control_. Prentice-Hall Englewood Cliffs, NJ, 1991, vol. 199, no. 1.
  * [31] E. Kayacan, E. Kayacan, H. Ramon, and W. Saeys, “Towards agrobots: Identification of the yaw dynamics and trajectory tracking of an autonomous tractor,” _Computers and Electronics in Agriculture_ , vol. 115, pp. 78 – 87, 2015.
  * [32] J. M. Snider, “Automatic steering methods for autonomous automobile path tracking,” _Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RITR-09-08_ , 2009.

|  Erkan Kayacan (S’12 M’16) was born in Istanbul, Turkey, on April 17, 1985.
He received the B.Sc. degree in mechanical engineering and the M.Sc. degree in
system dynamics and control from Istanbul Technical University, Istanbul,
Turkey in 2008 and 2010, respectively. He received the Ph.D. degree in
Mechatronics, Biostatistics and Sensors from University of Leuven (KU Leuven),
Leuven, Belgium in 2014. He is currently a Postdoctoral Associate with
Senseable City Laboratory and Computer Science and Artificial Intelligence
Laboratory, Massachusetts Institute of Technology, MA, USA. Prior to MIT, he
was a Postdoctoral Researcher with Delft Center for Systems and Control
(DCSC), Delft University of Technology, The Netherlands and Distributed
Autonomous Systems lab, University of Illinois at Urbana-Champaign, IL, USA.
His research interests include real-time optimization-based control and
estimation methods, nonlinear control theory, learning algorithms and machine
learning with a heavy emphasis on applications to autonomous systems and field
robotics. Dr. Kayacan is a recipient of the Best Systems Paper Award at
Robotics: Science and Systems (RSS) in 2018.
---|---
|  Wouter Saeys received the M.Sc degree in Bioscience Engineering from
University of Leuven (KU Leuven), Leuven, Belgium in 2002. On the basis of his
Master’s thesis, he was awarded the engineering prize by the Royal Flemish
Society of Engineers (KVIV). In 2006, he received the Ph.D. in Bioscience
Engineering from KU Leuven, Leuven, Belgium. Since 2017, he is an Associate
Professor at the Biosystems Department of KU Leuven, where he leads a group
focusing on Agrofood Mechatronics. His main research interests include
agricultural automation and robotics, chemometrics, light transport modelling
and optical characterization of biological materials. He has supervised 20
PhDs and is (co-)author of over 145 peer reviewed journal articles.
---|---
|  Herman Ramon received the M.Sc. degree in bioscience engineering from Gent
University, Gent, Belgium and the Ph.D. degree in biological sciences from the
University of Leuven (KU Leuven), Leuven, Belgium, in 1993. He is currently a
Professor with the Faculty of Bioscience Engineering, KU Leuven, lecturing on
field robotics, system dynamics, applied mechanics and mathematical biology.
His current research interests include mechatronic design on agricultural
machinery, theoretical and experimental study of particulate systems from
fundamental research in single-cell and tissue dynamics to granular mechanics
and liquid-like flow in agricultural and industrial processes. He has
(co-)authored more than 200 peer reviewed journal articles.
---|---
|  Calin Belta (M’03-SM’11-F’17) received the B.Sc. and M.Sc. degrees in
control and computer science from the Technical University of Iasi, Iasi,
Romania, in 1995 and 1996, respectively, the M.Sc. degree in electrical
engineering from Louisiana State University, Baton Rouge, LA, USA, in 1999,
and the M.Sc. and Ph.D. degrees in mechanical engineering from the University
of Pennsylvania, Philadelphia, PA, USA, in 2001 and 2003, respectively. He is
currently Professor with the Department of Mechanical Engineering, Boston
University (BU), Boston, MA, USA, where he holds the Tegan family
Distinguished Faculty Fellowship. He is the Director of the BU Robotics
Laboratory and of the Center for Autonomous and Robotic Systems. He is also
affiliated with the Department of Electrical and Computer Engineering, the
Division of Systems Engineering, the Center for Information and Systems
Engineering, and the Bioinformatics Program. His research interests include
dynamics and control theory, with particular emphasis on hybrid and cyber-
physical systems, formal synthesis and verification, and applications in
robotics and systems biology. Prof. Belta received the Air Force Office of
Scientific Research Young Investigator Award and the National Science
Foundation CAREER Award.
---|---
|  Joshua M. Peschel (IEEE Member) is an Assistant Professor of Agricultural
and Biosystems Engineering, Black & Veatch Faculty Fellow, and Director of the
Human-Infrastructure Interaction Lab at Iowa State University. He received the
B.S. in Biological Systems Engineering, M.S. in Biological and Agricultural
Engineering, and Ph.D. in Computer Science in 2001, 2004, and 2012, all from
Texas A&M University. His research interests focus on innovative sensing and
sense-making for smart agricultural, natural, and urban systems.
---|---


# Parameter-Efficient Tuning with Special Token Adaptation

Xiaoocong Yang†, James Y. Huang‡, Wenxuan Zhou‡ Work done when visiting USC.
Muhao Chen‡
†Tsinghua University; ‡University of Southern California
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Parameter-efficient tuning aims at updating only a small subset of parameters
when adapting a pretrained model to downstream tasks. In this work, we
introduce PaSTA, in which we only modify the special token representations
(e.g., [SEP] and [CLS] in BERT) before the self-attention module at each layer
in Transformer-based models. PaSTA achieves comparable performance to fine-
tuning in natural language understanding tasks including text classification
and NER with up to only 0.029% of total parameters trained. Our work not only
provides a simple yet effective way of parameter-efficient tuning, which has a
wide range of practical applications when deploying finetuned models for
multiple tasks, but also demonstrates the pivotal role of special tokens in
pretrained language models.111Our implementation is available at
https://github.com/luka-group/PASTA.

##  1 Introduction

Built upon a pretrained language model (PLM; Devlin et al. 2019; Liu et al.
2019; Yang et al. 2019; Chowdhery et al. 2022), many of the recent NLP systems
are developed based on task-specific finetuning. In this way, the PLM
effectively leverages the task-agnostic knowledge captured during self-
supervised pretraining and adapts itself to downstream tasks. However, full
finetuning poses a challenge to model deployment under multi-task, memory-
limited scenarios, where we need to train and store a separate full-sized
model for each substantially distinct task. As an alternative, parameter-
efficient tuning Ding et al. (2022) aims at only updating a small number of
parameters when adapting PLMs to downstream tasks while making most of the
model parameters fixed and shared among tasks, thus reducing memory usage.

L5-H3

L5-H5

L5-H6

L5-H7

L20-H3

L20-H5

L20-H8

L20-H15

Figure 1: Examples of vertical attention heads in the 5-th and 19-th layer of
BERT-large with a random sample from CoLA Warstadt et al. (2019) as input.
Heads in the first row and second row assign most of maximal attention weights
to [CLS] and [SEP] respectively. See Appendix D for the full attention map.

In this paper, we propose Parameter-efficient tuning with Special Token
Adaptation (PaSTA), where we only add trainable vectors to hidden
representations of _special tokens_ 222WLOG, we use the notation of special
tokens [CLS] and [SEP] in BERT for the convenience of expression, while the
method applies in the same way to other paradigms such as <S> and </S> in
RoBERTa Liu et al. (2019). at each layer before the multi-head attention
module in Transformer-based PLMs. Our work is motivated by the role of special
tokens in PLMs. First, special tokens such as [CLS] collect information from
the whole input sequence and are typically regarded as the global text
representation Devlin et al. (2019). For sentence-level tasks such as GLUE
Wang et al. (2018), a common practice is to add a new classifier head based on
the [CLS] representation in the last model layer. Thus, if trained properly,
by updating the [CLS] representations, we can approximate the result of the
information collection process in PLMs. Second, many attention heads in PLMs
follow a vertical pattern333Following Voita et al. (2019) and Yao et al.
(2021), an attention head is regarded as vertical if at least 90% tokens
assign maximal attention scores to either [CLS] or [SEP]., where the attention
scores are mostly allocated to either the [CLS] or [SEP] token Clark et al.
(2019); Kovaleva et al. (2019), as illustrated in Figure 1. Therefore, updates
to special tokens can also be disseminated to other tokens during the forward
pass through the vertical attention heads Elhage et al. (2021), enabling the
PLMs to adapt to both sentential and lexical tasks.

By tuning as few as up to 0.029% of the total parameters, PaSTA achieves
competitive performance on par with full finetuning and BitFit Zaken et al.
(2022) on GLUE (Section 4.2). It also outperforms P-tuning v2 Liu et al.
(2022) by 0.6% on CoNLL2003 with 20$\times$ fewer additional parameters
(Section 4.3). The ablation study shows that we can further reduce trainable
parameters to 0.009% with only a slight performance drop (Section 4.4),
showing the merit of adapting special token representations.

##  2 Related Work

A recent survey Ding et al. (2022) categorizes three types of _parameter-
efficient tuning_ methods. _Addition methods_ Houlsby et al. (2019); Lester et
al. (2021); Liu et al. (2022) introduce a small number of additional trainable
parameters while keeping those in the PLM unchanged. _Specification methods_
Zaken et al. (2022); Guo et al. (2021); Zhao et al. (2020) update a portion of
parameters in the PLM while keeping others frozen. _Reparameterization
methods_ Aghajanyan et al. (2021); Hu et al. (2021); Qin et al. (2021) modify
PLMs’ structures to parameter-efficient forms. Our method belongs to the
addition-based methods and follows the basic settings of P-tuning v2 Liu et
al. (2022), where newly initialized hidden representations of tokens are
inserted into each Transformer layer. Different from most prompt tuning
methods that introduce new tokens, we add the introduced vectors to the hidden
states of special tokens and keep the sequence length unchanged.

Previous works use probing tasks Wu et al. (2020) and pruning methods Prasanna
et al. (2020) to study the roles of different modules inside BERT. It has been
shown that functional specialization exists in BERT self-attention heads Clark
et al. (2019), and vertical attention heads3 take up a large portion Yao et
al. (2021). Kovaleva et al. (2019) find that vertical attention heads are
almost exclusively associated with attention to [SEP] or [CLS] tokens, and
Clark et al. (2019) conclude that heads in early layers often attend to [CLS]
while in middle layers attend to [SEP]. In this work, we demonstrate that
adapting hidden representations of special tokens is sufficient to bring the
performance of PLMs to the level of full finetuning.

##  3 PASTA

Figure 2: Architecture of PaSTA layer in Transformer. Skip-connections in
Transformers are not shown for brevity. At layer $l$ we add a trainable vector
$\mathbf{e}(\mathbf{v}_{p}^{l})\in\mathbb{R}^{d}$ to the hidden representation
of the $p$-th special token in the input sequence, and freeze the weights of
the PLM.

Given a large PLM, our goal is to develop a parameter-efficient tuning method
that only updates a small set of parameters when adapting to a downstream
task. To this end, we propose a simple yet effective method called PaSTA, in
which we train a hidden vector for every special token at each Transformer
layer, along with a task-specific classifier, while freezing the parameters of
the PLM.

###  3.1 Special Token Adaptation

The special token adaption is illustrated in Figure 2. Although these
adaptations are not directly applied to non-special tokens, changes in special
token hidden states can be effectively disseminated to other tokens via self-
attention, thanks to the prevalence of vertical attention heads3 in PLMs.

Specifically, denote the inputs to the $l$-th Transformer layer as
$\mathbf{H}^{l}=\\{\mathbf{h}_{i}^{l}\\}_{i=1}^{N},\mathbf{h}_{i}^{l}\in\mathbb{R}^{d}$,
where $N$ is the number of input tokens, $d$ is the hidden size, PaSTA
modifies the inputs as follows:

$\displaystyle\mathbf{H}^{l}_{\text{mod}}$
$\displaystyle=\\{\mathbf{h}_{i}^{l}+\mathbf{m}_{i}^{l}\\}_{i=1}^{N},$
$\displaystyle\mathbf{H}^{l+1}$
$\displaystyle=\text{Trm}^{l}(\mathbf{H}^{l}_{\text{mod}}),$

where $\text{Trm}^{l}$ is the $l$-th Transformer layer,
$\mathbf{m}_{i}^{l}\in\mathbb{R}^{d}$ is our special token adaptation defined
as follows:

$\mathbf{m}^{l}_{i}=\left\\{\begin{aligned} \mathbf{0}\qquad\text{if token $i$
is not a special token}\\\ \mathbf{e}(\mathbf{v}_{p}^{l})\quad\text{if token
$i$ is the $p$-th special token}\end{aligned}\right.\vspace{-0.5em}$

where $\mathbf{e}(\mathbf{v}_{p}^{l})\in\mathbb{R}^{d}$ is the trainable
vector added to the hidden representation of the $p$-th special token in the
input sequence. During downstream task training, only those introduced hidden
vectors for special tokens and the task-specific classifier are optimized, and
the rest of model parameters are frozen.

###  3.2 Parameter Efficiency and Consistency

As shown in Table 1, PaSTA achieves $\mathcal{O}(L\times d)$ parameter
complexity444The number of special tokens are invariant to the scale of
models, and are usually very small. and updates as few as 0.015%-0.029% of the
parameters compared to a full PLM when using BERT-large or RoBERTa-large as
backbone. Unlike Adapter Houlsby et al. (2019) that learns the transformation
of all input tokens using a shared FFN, PaSTA only learns the task-specific
update of special token representations as a bias term, which significantly
reduces the parameter capacity needed for adaptation on downstream tasks.

Meanwhile, the set of parameters introduced by PaSTA is consistent across
different tasks, making it efficient for hardware-based deployment Zaken et
al. (2022). On the contrary, in Diff-Prune, the parameter update is considered
as a term of the loss function Guo et al. (2021), resulting in different sets
of updated parameters in distinct tasks.

| # Param | Parameter Consistency
---|---|---
Adapter | $\mathcal{O}(L\times d\times r)$ | ✓
P-tuning v2 | $\mathcal{O}(L\times d\times T)$ | ✓
BitFit | $\mathcal{O}(L\times(d+m))$ | ✓
Diff-Prune | - | ✗
PaSTA | $\mathcal{O}(L\times d)$ | ✓
Table 1: Parameter complexity of PaSTA and baselines. Here $L$ and $d$ refer
to the number of layers and hidden size of the PLM. $m$ and $r$ refer to the
intermediate size of FFN modules in Transformers and Adapters, respectively.
$T$ is the prompt length. Parameter consistency refers to whether the set of
trainable parameters is consistent across different tasks Zaken et al. (2022).

##  4 Experiments and Results

We hereby study the downstream performance of PaSTA and analyze the properties
of introduced hidden vectors.

| %Param | RTE | CoLA | STS-B | MRPC | SST-2 | QNLI | MNLI(m/mm) | QQP | Avg.
---|---|---|---|---|---|---|---|---|---|---
|  | acc. | mcc. | Spearman | F1 | acc. | acc. | acc. | F1 |
Full Finetuning∗ | 100% | 70.1 | 60.5 | 86.5 | 89.3 | 94.9 | 92.7 | 86.7/85.9 | 72.1 | 81.6
Adapter∗∗ | 3.6% | 71.5 | 59.5 | 86.9 | 89.5 | 94.0 | 90.7 | 84.9/85.1 | 71.8 | 81.1
Diff-Prune† | 0.5% | 70.6 | 61.1 | 86.0 | 89.7 | 94.1 | 93.3 | 86.4/86.0 | 71.1 | 81.5
P-tuning v2 | 0.29% | 70.1 | 60.1 | 86.8 | 88.0 | 94.6 | 92.3 | 85.3/84.9 | 70.6 | 81.0
BitFit‡ | 0.08% | 72.0 | 59.7 | 85.5 | 88.9 | 94.2 | 92.0 | 84.5/84.8 | 70.5 | 80.9
PaSTA | 0.015%-0.022% | 70.8 | 62.3 | 86.6 | 87.9 | 94.4 | 92.8 | 83.4/83.4 | 68.6 | 80.9
Table 2: BERT-large model performance on GLUE benchmark test set. Lines with ∗ and ∗∗ are results from Devlin et al. (2019) and Houlsby et al. (2019), and lines with † and ‡ are from Guo et al. (2021) and Zaken et al. (2022) respectively. We reimplement experiments of P-tuning v2 on GLUE benchmark with a prompt length of 20. | %Param | RTE | CoLA | STS-B | MRPC | SST-2 | QNLI | MNLI(overall) | QQP | Avg.
---|---|---|---|---|---|---|---|---|---|---
|  | acc. | mcc. | Pearson | acc. | acc. | acc. | acc. | acc. |
Full Finetuning∗ | 100% | 86.6 | 68.0 | 92.4 | 90.9 | 96.4 | 94.7 | 90.2 | 92.2 | 88.9
LoRA† | 0.24% | 87.4 | 68.2 | 92.6 | 90.9 | 96.2 | 94.9 | 90.6 | 91.6 | 89.0
PaSTA | 0.015%-0.029% | 86.6 | 69.7 | 91.8 | 90.9 | 96.8 | 95.1 | 90.4 | 89.9 | 88.9
Table 3: RoBERTa-large model performance on GLUE benchmark. Lines with ∗ are results from Liu et al. (2019), and lines with † are from Hu et al. (2021). We follow the metric settings of baselines and also report results on GLUE development set for the convenience of direct comparison. | CoLA | RTE | MRPC | STS-B | CoNLL2003
---|---|---|---|---|---
PaSTA | 65.4 | 76.2 | 89.7 | 90.8 | 94.0
\- w/o [CLS] | 58.8 | 72.6 | 91.4 | 90.2 | 93.7
\- w/o [SEP] | 64.5 | 71.1 | 91.9 | 90.3 | 93.7
\- shared vector | 64.7 | 74.7 | 92.1 | 90.0 | 93.9
\- classifier only | 36.5 | 54.2 | 81.5 | 64.9 | 77.4
Table 4: Performance of ablation study with BERT-large on GLUE and CoNLL2003
development sets.

###  4.1 Experimental Setup

Baseline Methods. We compare PaSTA with the following parameter-efficient
tuning methods in prior studies. Adapter Houlsby et al. (2019) introduces new
feed-forward modules in Transformer layers while keeping original parameters
of the PLM frozen. BitFit Zaken et al. (2022) updates all bias terms in the
PLM during finetuning. Diff-Prune Guo et al. (2021) introduces $L_{0}$-norm
penalty on the updated parameters to encourage sparsity of tuned parameters.
P-tuning v2 Liu et al. (2022) prepends trainable hidden vectors before the
input sequence at each layer while keeping the original PLM parameters frozen.
LoRA Hu et al. (2021) uses low-rank decomposition matrices to model the
parameter updates.

Model Configuration. We conduct our experiments using BERT-large and RoBERTa-
large (We also report experiments with BERT-base in Appendix A). To facilitate
comparison with baseline works, we take most of the experimental results from
their original papers which are reported with either BERT-large or RoBERTa-
large. Note that multiple [SEP] tokens in a single sequence (e.g., in sentence
pair tasks like MNLI) are treated as different special tokens and have
separate sets of trainable parameters, and the number of trainable parameters
varies among downstream tasks according to the number of special tokens added.
Details of training and hyperparameters settings are shown in Appendix B.

###  4.2 GLUE Tasks

Task Setup. We evaluate PaSTA on the widely used GLUE benchmark Wang et al.
(2018). Following the same setting as previous work Zaken et al. (2022);
Houlsby et al. (2019), WNLI task is excluded, and we add a linear prediction
head on top of the final [CLS] token representation to predict the labels.
More task setup details are in Appendix C.

Results. Tables 2 and 3 report the performance of PaSTA on GLUE benchmark with
BERT-large and RoBERTa-large respectively. PaSTA with RoBERTa-large achieves
the same average score as that of full finetuning over GLUE tasks. PaSTA with
BERT-large achieves an average score on par with BitFit using over 3$\times$
fewer trainable parameters and comparable results to other higher parameter
complexity baselines. The results demonstrate that by leveraging the pivotal
role of special tokens in PLMs, PaSTA is able to effectively adapt the model
to sentence-level tasks with significantly fewer parameters tuned than
previous methods.

###  4.3 Named Entity Recognition

Task Setup. We experiment with the NER task on CoNLL2003 Tjong Kim Sang and De
Meulder (2003). Following Devlin et al. (2019), we formulate NER as a token
classification problem.

Results. PaSTA with BERT-large achieves an F1 score of 90.8% on CoNLL2003 test
set, outperforming P-tuning v2 Liu et al. (2022) by 0.6% with 20$\times$ fewer
trainable parameters, while falling behind full finetuning by 2.0%.
Nevertheless, the strong performance of PaSTA compared to P-tuning v2
indicates that even though PaSTA only directly adapts special tokens, the
representations of all input tokens can still be properly tuned, supporting
our hypothesis that vertical attention heads are able to disseminate
adaptations in special token hidden states to other tokens.

###  4.4 Analysis

Ablation on choices of special tokens. To understand the effect of tuning
different combinations of special tokens on downstream tasks, we further limit
the additional parameter capacity of PaSTA by only adapting either [CLS] or
[SEP], or share a common vector across all special tokens. Table 4 shows the
performance of three ablated variants and a baseline that only tunes the
classification head on top of a fixed BERT-large. In general, we observe a
decrease in performance on most tasks except on MRPC for three PaSTA variants,
and performance degrades significantly if we do not adapt any special tokens.
These results demonstrate the vital role of introduced hidden vectors for
special tokens in PaSTA, while the best choice of special tokens to be adapted
may vary depending on the task.

Norm distribution of introduced hidden vectors. The introduced hidden vectors
learn the difference of special tokens between pretrained and adapted models,
and thus norms of those vectors indicate the magnitude of parameter change at
different layers. Similar to the pattern of parameter change during full
finetuning Kovaleva et al. (2019), PaSTA generally has larger norms of hidden
vectors at layers closer to the output (see Figure 4).

##  5 Conclusion

We present PaSTA, a parameter-efficient tuning method that only modifies
special token representations at each Transformer layer when adapting to
downstream tasks. Our approach is motivated by the observation that PLMs have
a large amount of vertical attention heads that heavily attend to special
tokens. Experiments show that PaSTA achieves strong performance comparable to
full finetuning on sentential and lexical tasks with high parameter
efficiency. Our work not only provides an effective solution for parameter-
efficient tuning, but also demonstrates the pivotal role of special tokens in
PLMs.

## Limitations

In this work we hypothesize that the vertical attention heads could play a
role as “information disseminator” based on the theoretical analysis of
Transformers Elhage et al. (2021). However, we still have no direct approaches
such as probing tasks and reverse engineering to prove this assumption. And
since PaSTA relies on adaptation of special tokens, it cannot be applied to
language models which do not pad special tokens to input sequences such as
GPT-2 Radford et al. (2019). For the empirical results, we choose GLUE
benchmark and CoNLL2003 to study the performance on language understanding
tasks. The effectiveness of PaSTA on language generation tasks has not been
tested in this work due to limited bandwidth.

## References

  * Aghajanyan et al. (2021) Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7319–7328, Online. Association for Computational Linguistics.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2022) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. 2022.  Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models.
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_.  Https://transformer-circuits.pub/2021/framework/index.html.
  * Guo et al. (2021) Demi Guo, Alexander Rush, and Yoon Kim. 2021.  Parameter-efficient transfer learning with diff pruning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4884–4896, Online. Association for Computational Linguistics.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019.  Revealing the dark secrets of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4365–4374, Hong Kong, China. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2022) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022.  P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 61–68, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.
  * Mao et al. (2021) Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. 2021.  Unipelt: A unified framework for parameter-efficient language model tuning.
  * Prasanna et al. (2020) Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020.  When BERT Plays the Lottery, All Tickets Are Winning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3208–3229, Online. Association for Computational Linguistics.
  * Qin et al. (2021) Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and Jie Zhou. 2021\.  Exploring universal intrinsic task subspace via prompt tuning.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5797–5808, Florence, Italy. Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _International Conference on Learning Representations_.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wu et al. (2020) Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020.  Perturbed masking: Parameter-free probing for analyzing and interpreting BERT.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4166–4176, Online. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_.
  * Yao et al. (2021) Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. 2021.  Nlp from scratch without large-scale pretraining: A simple and efficient framework.
  * Zaken et al. (2022) Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–9, Dublin, Ireland. Association for Computational Linguistics.
  * Zhao et al. (2020) Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020.  Masking as an efficient alternative to finetuning for pretrained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2226–2241, Online. Association for Computational Linguistics.

##  Appendix A Performance of BERT-base

Table 5 reports the performance of PaSTA and a part of baselines with BERT-
base as backbones on GLUE development sets. We do not put the result table in
the main body of this work because most baselines did not report GLUE test set
scores using BERT-base, and only three of them reported GLUE development set
scores with BERT-base in their works. PaSTA slightly underperforms baselines
on average, while it outperforms other models on small datasets such as RTE
and MRPC.

| %Param | RTE | CoLA | STS-B | MRPC | SST-2 | QNLI | MNLI(m/mm) | QQP | Avg.
---|---|---|---|---|---|---|---|---|---|---
Full Finetuning ∗ | 100% | 66.4 | 62.1 | 89.8 | 90.9 | 91.6 | 90.0 | 83.2/ - | 87.4 | 82.7
Adapter∗ | 0.81% | 71.8 | 61.5 | 88.6 | 89.9 | 91.9 | 90.6 | 83.1/ - | 86.8 | 83.0
BitFit${\dagger}$ | 0.8% | 72.3 | 58.8 | 89.2 | 90.4 | 92.1 | 90.2 | 81.4/ - | 84.0 | 82.3
PaSTA | 0.015%-0.022% | 73.6 | 57.9 | 88.7 | 91.5 | 91.2 | 89.7 | 77.8/78.8 | 80.8 | 81.4
Table 5: PASTA with BERT-base model performance on GLUE benchmark development
set. Lines with ∗ and † refer to results from Mao et al. (2021) and Zaken et
al. (2022) respectively.

##  Appendix B Implementation Details

| RTE | CoLA | STS-B | MRPC | SST-2 | QNLI | MNLI | QQP | CoNLL2003
---|---|---|---|---|---|---|---|---|---
Learning rate | 4.5e-3 | 5e-3 | 2e-3 | 2.5e-3 | 7e-3 | 2e-3 | 5e-4 | 5e-3 | 3e-3
Batch size | 32$\times$4 | 32$\times$1 | 32$\times$3 | 32$\times$4 | 64$\times$3 | 32$\times$4 | 32$\times$1 | 32$\times$4 | 16$\times$1
Number of adapted tokens | 3 | 2 | 3 | 3 | 2 | 3 | 3 | 3 | 2
Training epochs | 150 | 100 | 150 | 150 | 100 | 80 | 50 | 100 | 100
Best dev performance | 76.2 | 65.4 | 90.8 | 89.7 | 93.9 | 92.2 | 83.7 | 87.9 | 94.1
Best epochs | 121 | 63 | 109 | 136 | 99 | 42 | 49 | 97 | 89
Table 6: PaSTA with BERT-large training details for GLUE and CoNLL2003 tasks.
Distributed training on multiple GPUs is used when avaiable for less training
time.

Our model is implemented based on Huggingface’s Transformers. We optimize
models with AdamW Loshchilov and Hutter (2017). We set the maximum input
length to 128 and use a fixed random seed of 42 for all tasks. Experiments are
done on NVIDIA RTX A5000 for averagely 3 hours per task, and distributed
training is used for most tasks. Table 6 reports the best hyperparameters for
model training. For hyperparameter search, we select learning rate from {5e-4,
1e-3, 2e-3, 2.5e-3, 3e-3, 4.5e-3, 5e-3, 7e-3} and the number of epochs from
{50, 80, 100, 150}.

##  Appendix C GLUE Benchmark

We test the performance of our models on 8 out of 9 tasks in the GLUE
benchmark: the Corpus of Linguistic Acceptability (CoLA), the Stanford
Sentiment Treebank (SST-2), the Microsoft Research Paraphrase Corpus (MRPC),
the Quora Question Pairs (QQP), the Semantic Textual Similarity Benchmark
(STS-B), the Multi-Genre Natural Language Inference Corpus (MNLI), the
Stanford Question Answering Dataset (QNLI) and the Recognizing Textual
Entailment (RTE). We exclude The Winograd Language Inference Corpus (WNLI)
task because previous studies show that BERT underperforms the majority class
baseline Zaken et al. (2022). Following previous works Zaken et al. (2022);
Houlsby et al. (2019); Devlin et al. (2019), MRPC and QQP are evaluated using
F1 score, STS-B is evaluated using Spearman’s correlation coefficient, CoLA is
evaluated using Matthew’s Correlation, and the other tasks are evaluated using
accuracy.

##  Appendix D Full Attention Map

Figure 3: Full attention map of BERT-large pretrained with a random sample
from CoLA as input. Rows and columns represent model layers and heads
respectively, and darker color indicates larger weights. Vertical attention
heads are highlighted in orange.

Figure 3 illustrates the attention maps of all heads in BERT-large. With a
random sample from CoLA dataset as input ("Fred watered the plants flat."),
there are 112 heads out of 384 in total being vertical heads3.

##  Appendix E Distribution of Vector Norms

Figure 4: The distribution map of norms of introduced hidden vectors.

Figure 4 shows the norms of introduced vectors, which measure the magnitude of
parameter change at different layers. Consistent with studies Kovaleva et al.
(2019) on patterns of parameter change of full finetuning, PaSTA also
generally undergoes the largest parameter update at layers close to the
output.