# Towards Optimal Head-to-head Autonomous Racing with Curriculum Reinforcement
Learning

Dvij Kalaria1, Qin Lin2, and John M. Dolan1 1The authors are with the Robotics
Institute, Carnegie Mellon University
dkalaria<EMAIL_ADDRESS>Lin is with the Electrical
Engineering and Computer Science Department, Cleveland State University
<EMAIL_ADDRESS>
###### Abstract

Head-to-head autonomous racing is a challenging problem, as the vehicle needs
to operate at the friction or handling limits in order to achieve minimum lap
times while also actively looking for strategies to overtake/stay ahead of the
opponent. In this work we propose a head-to-head racing environment for
reinforcement learning which accurately models vehicle dynamics. Some previous
works have tried learning a policy directly in the complex vehicle dynamics
environment but have failed to learn an optimal policy. In this work, we
propose a curriculum learning-based framework by transitioning from a simpler
vehicle model to a more complex real environment to teach the reinforcement
learning agent a policy closer to the optimal policy. We also propose a
control barrier function-based safe reinforcement learning algorithm to
enforce the safety of the agent in a more effective way while not compromising
on optimality.

###### Index Terms:

Reinforcement learning-based control, head-to-head autonomous racing, game
theory

##  I Introduction

There has been a growing interest in autonomous racing research in recent
years [1] also accelerated by competitions such as RoboRace [2], F1Tenth [3],
and the Indy Autonomous Challenge [4]. Professional human race drivers operate
to follow racing lines to achieve optimal performance and tend to outperform
opponents while adhering to the racing rules. Prior works in autonomous racing
tend to ignore the latter and only consider collision avoidance. It is
difficult to inculcate these complex rules and design a classical rule-based
controller which takes care of all scenarios and tackles a wide range of
opponent behaviors. Most previous reinforcement learning (RL)-based works
tackling this do not include racing line information in the framework [5] [6],
due to which it becomes difficult for the learning agent to learn an optimal
policy which can be generalized to other tracks. Also, training an agent
directly on a complex racing environment with complex vehicle dynamics debars
the learning agent from learning complex behaviors like skidding. In this
work, we propose a complex racing environment which can be used to train head-
to-head RL agents to learn an optimal policy for competing with the opponents.
Then, we propose a curriculum learning-based framework which transitions the
vehicle model from a simple to a more complex one to tackle the problem of
this sub-optimality. Some works [7] also use Control Barrier Functions (CBFs)
as a shield while learning, which has also been proven to boost the safety
performance. However, these safety constraints may debar the agent from
learning a high-performant policy, especially in competitive environments like
ours where performance is also a primary concern alongside safety. In this
work, we also propose a curriculum learning-based CBF framework to enforce
safety during learning while not compromising on the optimality of the final
learned policy. We gradually remove the CBF interference with the policy as
the agent progressively learns to be more safe, hence focusing on improving
performance next. We test our controllers against baselines by performing
head-to-head races with other baselines. We briefly summarize our contribution
as follows: 1) We propose a head-to-head racing environment which models
complex vehicle dynamics and collision among agents or with walls; 2) We
design an effective hierarchical controller which includes racing line
information inside the hierarchical controller to train an optimal policy; 3)
We design a curriculum learning-based framework which effectively enables
learning an optimal policy for the agent.

The rest of the paper is organized as follows: Section II briefly discusses
the previous related works. Section III presents the problem formulation.
Section IV elaborates on the proposed framework. The simulation results are
presented in Section V. The concluding remarks and future work can be found in
Section VI.

##  II Related Works

Autonomous racing has received a lot of interest from the research community
recently at all levels of the stack including the perception, localization,
path planning and control, as discussed in this literature review paper [1].
We specifically focus on the path planning and control level of the stack.
Most recent works focus on optimizing lap times for a single agent, with very
recent works addressing multi-agent planning and control.

For single-agent, most works propose calculating an optimal racing line
offline and using a control algorithm to run on it as a reference online. [8]
propose IPOPT optimization to compute the racing line while [9] proposes
Bayesian optimization to obtain the racing line. [10] calculates a minimum-
curvature path which is very close to the optimal line. [11] proposes using an
LQR controller to track the racing line. [12] [13] proposes a discrete MPC
controller, while [14] proposes a model predictive contouring controller
(MPCC). There also have been some recent works to account for model and
environment uncertainty [15] and some [16] [17] to account for model changes
online. Several works proposed to use reinforcement learning [6] [18] and
imitation learning [19] [20] to control a vehicle around the track with the
objective of minimal race times.

For multi-agent racing control, there are works [21] [22] on using rule-based
strategy selection and using high-level path planner/low-level control to
execute strategies like overtaking, blocking, collision avoidance, etc.
However, these works rely on a lot of parameters and it always difficult to
find an optimal set of parameters to work on all track maps and environments.
Some works use game theoretic planning followed by classical control [23].
Some recent works like [5] [24] use reinforcement learning to effectively
learn an end-to-end controller that learns to win the race and thus learns
certain strategies to do so. The environments used to train the RL agent(s)
can be varied with different maps, surfaces, etc. to learn a widely
generalizable policy. However, these works still struggle in learning an
optimal policy. Some works like [5] use a simple kinematic vehicle model to
train and test the RL policy with other classical approaches. In this work, we
propose a complex racing environment with a dynamic vehicle model with
challenging parameters close to the actual racing car to train and test the RL
controller. We also propose a curriculum-based course to train the RL agent
that helps in learning a better policy.

##  III Problem Formulation

We first present our dynamic game formulation. Let there be 2 players $i$ and
$j$ racing against each other over $T$ time steps. The track is defined by a
sequence of $k$ checkpoints along the center, $\\{c_{i}\\}_{i=1}^{T}$. The
objective for each player is to minimize the time difference between it and
its opponent in completing the final lap defined as $c_{k}$. Let $\gamma_{i}$
be the earliest time step when a player $i$ reaches a checkpoint. Let the
state of the vehicle be $x_{i}\in X\subset R^{6}$ and control action at each
time step be $u_{i}\in U\subset R^{2}$. Let $r_{i}\in{1,2...k}$ be the index
of the last checkpoint passed by the player. Let $p:S\rightarrow C$ be a
function mapping a state $s$ to a checkpoint. Also, we must ensure the state
$s_{t}$ is always within the track boundaries, i.e., $q(s_{t})\leq w$ where
$q$ is a function for the closest distance to the center line and $w$ is the
track width, assumed to be constant. For collision avoidance, let
$d:X*X\rightarrow R$ be a function that returns the shortest distance between
bodies given the $2$ vehicle states as $d(s_{t}^{i},s_{t}^{j})$. Based on
these variables, the objective for agent $i$ is given as:

$\displaystyle\mathop{\min}\limits_{u_{0}^{i},u_{1}^{i}..u_{T-1}^{i}}$
$\displaystyle\gamma_{i}-\gamma_{j}$ (1) $\displaystyle s.t.$ $\displaystyle
x_{t+1}^{k}=f(x_{t}^{k},u_{t}^{k}),\forall t\in{0,1,..,T-1}k\in{i,j}$
$\displaystyle x_{t+1}^{k}=f(x_{t}^{k},u_{t}^{k}),\forall
t\in{0,1,..,T-1}\forall k\in{i,j}$ $\displaystyle q(x_{t}^{k})\leq w$
$\displaystyle d(x_{t}^{i},x_{t}^{j})\forall k\in{i,j}\geq 0$

This formulation is similar to [5] except for the transition model $f$. For
more details, readers are referred to it. The dynamic bicycle model is used to
define the model transition $f$. The dynamic model state $s_{t}$ is defined
with global coordinates $x$, $y$, and yaw rotation $\phi$ in the global frame;
longitudinal velocity $v_{x}$, lateral velocity $v_{y}$, and yaw angular
velocity $\omega$ in the vehicle’s body frame. Throttle $d$ and steering
$\delta$ define the action space of the model. $F_{r,x}$ is the longitudinal
force on the rear tire in the tire frame assuming a rear-driven vehicle,
$F_{f,y}$ and $F_{r,y}$ are the forces on the front and rear tires,
respectively, and $\alpha_{f}$ and $\alpha_{r}$ are the corresponding slip
angles. We denote the mass of the vehicle $m$, the moment of inertia in the
vertical direction about the center of mass of the vehicle $I_{z}$, the length
of the vehicle from the COM (center of mass) to the front wheel $l_{f}$, and
the length from the COM to the rear wheel $l_{r}$. $B_{f/r}$, $C_{f/r}$,
$D_{f/r}$ are the Pacejka tire model parameters specific to the tire and track
surface. For longitudinal force, $C_{m1},C_{m2}$ are known constants obtained
from the gear model and $C_{r},C_{d}$ are aerodynamic force constants which
are learned from vehicle interactions. Mathematically, the vehicle model $f$
is defined as follows:

$\begin{split}&\begin{bmatrix}\dot{x}\\\ \dot{y}\\\ \dot{\phi}\\\
\dot{v}_{x}\\\ \dot{v}_{y}\\\ \dot{\omega}\\\
\end{bmatrix}=\begin{bmatrix}v_{x}\cos(\phi)-v_{y}\sin(\phi)\\\
v_{x}\sin(\phi)+v_{y}\cos(\phi)\\\ \omega\\\
\frac{1}{m}(F_{r,x}-F_{f,y}\sin(\delta)+mv_{y}\omega-mg\sin(p))\\\
\frac{1}{m}(F_{r,y}+F_{f,y}\cos(\delta)-mv_{x}\omega+mg\sin(r))\\\
\frac{1}{I_{z}}(F_{f,y}l_{f}\cos(\delta)-F_{r,y}l_{r})\\\ \end{bmatrix}\\\
\end{split}$ (2)

where $F_{r,x}=(C_{m1}-C_{m2}v_{x})d-C_{r}-C_{d}v_{x}^{2}$,
$F_{f,y}=D_{f}\sin(C_{f}\tan^{-1}(B_{f}\alpha_{f})),\alpha_{f}=\delta-\tan^{-1}\left(\frac{\omega
l_{f}+v_{y}}{v_{x}}\right)$, and
$F_{r,y}=D_{r}\sin(C_{r}\tan^{-1}(B_{r}\alpha_{r})),\alpha_{r}=\tan^{-1}\left(\frac{\omega
l_{r}-v_{y}}{v_{x}}\right)$.

##  IV Hierarchical control design

Similar to [5], we also propose a hierarchical design with a high-level
planner returning a discrete checkpoint plan followed by a low-level
controller to track the planned checkpoints. Having a decoupled planning
approach helps to achieve long-term plans like overtaking. As discussed in
detail in [5], directly executing reinforcement learning strategies as a
single controller may not allow reliably meeting all the constraints or is not
strategically optimal in the long run.

###  IV-A High level planner

The high-level tactical planner approximates the general game formulation
discussed earlier into a simpler discrete form. This discrete game formulation
requires $2$ components.

####  IV-A1 State space model

We first transform the continuous state of the vehicle into a discrete state.
We convert the position of the vehicle into a pair of discrete variables,
i.e., lane ID and last passed checkpoint velocity, the latter converted to a
range with suitable window size. Tire wear is also contained within a range.
An example of a continuous state conversion to discrete state is shown in Fig
1.

Figure 1: Converting a continuous state to discrete state

####  IV-A2 Dynamics transition model

We need a transition model to define the transition between $2$ states. If the
transition is possible such that all the continuous states from the current
state are able to reach at least one state on the next state, then the
transition is deemed to be feasible. If all the boundary conditions are not
satisfied, we rule out the transition. In our update implementation, we use
simple one-dimensional equations of motion to determine current time state by
taking the mean of the velocities in the range of the initial and final state
[25]. For every state transition, the longitudinal segment is incremented
strictly by $1$. The number of lane changes can be used to formulate a penalty
on changing lanes too frequently on straights, which we will discuss later.

The game is played with both players starting at an initial checkpoint and
progressively updating each player’s choices with the smallest time state at
each point. A lower time state value implies that the player in question
reached the particular checkpoint before the other player and hence gets to
choose the next action. This gives the other player a chance to make a
resultant strategic action, like whether and how to overtake, etc. A collision
avoidance rule is incorporated in the high-level planner by restricting
actions that result in the same checkpoint and the same lateral lane with time
difference less than $mT$.

####  IV-A3 Solution

The high-level problem of minimizing time w.r.t. the other agent is solved
using monte-carlo tree search (MCTS). The optimal performance is defined as
follows:

$C_{X_{a},X_{a+1},..,X_{b-1}}=\sum_{i=a}^{i=b-1}(o_{i}-X_{i,\text{lane}})^{2}$
(3)

where $o_{i}$ is the optimal lane and $X_{i}$ is the high-level state at the
$i^{\text{th}}$ checkpoint. The optimal lane is obtained from the optimal
racing line. The optimal racing line is obtained from [8] by computing the
time-optimal trajectory with the given vehicle model parameters. The optimal
lane at each checkpoint is obtained by finding the (segment,lane) pair at
which the racing line passes. The solution from MCTS is a series of discrete
states both for the player and the adversarial opponent. Note that we assume
that the opponent is optimal here, i.e., it too tries to achieve optimal
performance. The formulation for MCTS is similar to [5] except that the
optimal solution is defined by staying closer to the racing line rather than
minimizing time. We believe this leads to a better optimal solution, as
explained in Section V. Figure 2 shows path planned from the 2 approaches. On
choosing minimum time difference as the criterion for a solution, the
trajectory comes out to be Figure 2(a), which is closer to the inner boundary,
as it covers less distance, but would take a longer time in the long run.
Figure 2(b), however, shows a new optimal trajectory along the racing line
which yields shorter lap times in the long run.

(a)

(b)

Figure 2: High-level plans for (a) min. time cost (b) min. distance to
raceline cost

###  IV-B Low level controller

The aim of the low-level controller is to execute the high-level trajectory
plan. The low-level controller is typically an RL controller which takes the
current state as an input and outputs the control command.

####  IV-B1 Reward design

The reward for the low-level controller is defined by the distance to capture
the following objectives:

1\. Reward for passing through a checkpoint and additional reward for passing
through the target lane and through the target speed :
$k_{\text{target}}e^{-d_{tc}}$ where $d_{tc}$ is the distance from the target
checkpoint

2\. Reward for minimizing time between passing $2$ checkpoints :
$-k_{\text{time}}\Delta t$ where $\Delta t$ is the time difference between
passing $2$ checkpoints

3\. Negative reward for swerving too frequently on straights :
$-k_{\text{swerve}}\mathbb{1}_{(x,y)\in S}$ where $S$ is the set of straight
section checkpoints

4\. Negative reward for colliding with the wall. We use an indicator function
$\mathbb{1}_{I_{j}\leq h\cup I_{j}\text{ hit wall}}$ that determines if the
$j^{th}$ LIDAR reading is less than $h$ and if LIDAR bounced off the wall :
$-\sum_{j=1}^{9}k_{\text{wall-hit}}$ $\mathbb{1}_{I_{j}\leq h\cup I_{j}\text{
hit wall}}$

5\. Negative reward for collision with other players. We use the indicator
function $\mathbb{1}_{I_{j}\leq h\cup I_{j}\text{ opponent}}$ to check if the
$j^{th}$ LIDAR reading reads hitting the opponent and we have a set $\phi$
containing all LIDAR rays that point to the front of the car for which we
impose additional penalty :
$-\sum_{j=1}^{9}k_{\text{opp,1}}\mathbb{1}_{I_{j}\leq h\cup I_{j}\text{
opponent}}+k_{\text{opp,2}}\mathbb{1}_{I_{j}\leq h\cup I_{j}\text{
opponent}\cup j\in\phi}$

6\. Negative reward for braking unnecessarily, i.e., when speed is lower than
the target window, high lateral slips : $k_{\text{brake}}\textbb{1}_{v\leq
v_{\text{target}}\cup d\leq 0}+k_{\text{slip}}(\alpha_{f}^{2}+\alpha_{r}^{2})$

####  IV-B2 Network architecture

PPO RL algorithm is used to obtain the optimal policy. The neural network used
for estimating the value function and the policy is a simple feedforward
neural network with $8$ layers and $128$ neurons on each layer, as shown in
Fig. 3. There is a Tanh layer at the end to restrict the output steering and
throttle to their ranges. Both the steering and throttle are obtained from the
output by scaling them by their ranges. The input consists of the following
values: 1\. The dynamic state of the vehicle consisting of ($v_{x}$, $v_{y}$,
$w$); 2\. The Frenet frame state w.r.t. the racing line reference, i.e.,
signed lateral distance from the racing line $e_{1}$, relative angle w.r.t.
the closest point on the racing line $e_{2}$; 3\. Relative position of the
opponent vehicle; 4\. Discrete high-level target state. All the values within
the range are passed by the average of the lower and upper limits 5\. Raw
Lidar data consisting of distances at $32$ rays cast from the extreme left of
the car to the right.

####  IV-B3 Training environment

Training is conducted on $16$ parallel tracks ($8$ clock-wise and $8$ counter-
clockwise) so that the agent does not overfit to one side. Also, for each
side, $4$ environments have steeper turns and $4$ moderate turns, as shown in
Fig. 3.

Figure 3: Training environment

###  IV-C Curriculum learning

With the problem formulation and the hierarchical control design in place, we
now define the proposed curriculum learning framework to progressively teach
the RL agent an optimal policy. Let us define a parameter $t_{s}$ denoting the
time scale. We vary $t_{s}$ as:

$t_{s}=\max\left(0,\min\left(1,\frac{t-t_{\text{start}}}{t_{\text{end}}-t_{\text{start}}}\right)\right)$
(4)

####  IV-C1 Vehicle model transition

The dynamic model defined in 2 makes it very difficult for the RL agent to
learn an effective policy, as it is very difficult to learn to move at optimal
speeds while respecting the friction limits and skidding caused due to high
lateral slips at higher speeds. Hence, we define a transition from a
relatively simpler dynamic model which is close to the kinematic model (very
few slips at the same speeds and much higher friction limits) to the complex
model with the actual parameters. The tire model Pacejka parameter changes are
defined as follows:

$\displaystyle D_{ft_{s}}=2^{1-t_{s}}D_{f0}$ (5) $\displaystyle
C_{ft_{s}}=C_{f0}^{2^{t_{s}-1}}$ $\displaystyle
B_{ft_{s}}=2^{1-t_{s}}\frac{D_{f0}C_{f0}B_{f0}}{D_{ft_{s}}B_{ft_{s}}}$

Figure 4: Tire curve variation for curriculum learning

####  IV-C2 Safety CBF transition

We also define a safety Control Barrier Function (CBF) to shield the agent
while learning similar to [26]. We observed that the RL agent struggles a lot
at the beginning hitting the walls as it tries to understand the environment.
Hitting the wall once debars the agent from learning meaningful behavior later
in the episode, as it gets stuck at the wall. To get rid of this, we define a
safety CBF for the wall boundary constraints which overrides the RL controller
by the minimum amount required to avoid hitting the wall. For more rigorous
details on the CBF, readers are referred to [27]. We define the CBF function
$h$ and the $2^{\text{nd}}$ order CBF as follows:

$\begin{split}&h_{\text{right}}(x)=-e_{\text{center}}+w\\\
&\dot{h}_{\text{right}}(x)=-v_{x}\sin(\theta-\theta_{\text{ref}})-v_{y}\cos(\theta-\theta_{\text{ref}})\\\
&\ddot{h}_{\text{right}}(x,u)=-\dot{v}_{x}\sin(\theta-\theta_{\text{ref}})-\dot{v}_{y}\cos(\theta-\theta_{\text{ref}})\\\
&\ \ \ \ \ \ \ \ \ \ \
-\omega.(v_{x}\cos(\theta-\theta_{\text{ref}})-v_{y}\sin(\theta-\theta_{\text{ref}}))\\\
&h_{\text{left}}(x)=e_{\text{center}}+w\\\
&\dot{h}_{\text{left}}(x)=v_{x}\sin(\theta-\theta_{\text{ref}})+v_{y}\cos(\theta-\theta_{\text{ref}})\\\
&\ddot{h}_{\text{left}}(x,u)=\dot{v}_{x}\sin(\theta-\theta_{\text{ref}})+\dot{v}_{y}\cos(\theta-\theta_{\text{ref}})\\\
&\ \ \ \ \ \ \ \ \ \ \
+\omega.(v_{x}\cos(\theta-\theta_{\text{ref}})-v_{y}\sin(\theta-\theta_{\text{ref}}))\\\
&\text{Finally, }\\\
&C_{\text{right}}(x,u)=\max(0,\lambda_{1}\lambda_{2}\ddot{h}_{\text{right}}+(\lambda_{1}+\lambda_{2})\dot{h}_{\text{right}}+h_{\text{right}})\\\
&C_{\text{left}}(x,u)=\max(0,\lambda_{1}\lambda_{2}\ddot{h}_{\text{left}}+(\lambda_{1}+\lambda_{2})\dot{h}_{\text{left}}+h_{\text{left}})\\\
&\lambda_{1,t_{s}}=\lambda_{1,0}(1-t_{s})\\\
&\lambda_{2,t_{s}}=\lambda_{2,0}(1-t_{s})\\\ \end{split}$ (6)

The updated command is obtained via the following optimization process where
$K_{\text{viol}}$ is typically set to a very high value and $u_{\text{ref}}$
is the reference control command before the change:

$\min_{u}(K_{\text{viol}}(C_{\text{right}}^{2}+C_{\text{left}}^{2})+|u-u_{\text{ref}}|^{2})$
(7)

We also add a negative reward for constraint violation as follows:

$R_{\text{constraint}}=k_{\text{constraint}}(C_{\text{right}}^{2}+C_{\text{left}}^{2})$
(8)

Higher values of $\lambda_{1}$ and $\lambda_{2}$ imply higher interference
from the CBF, as the constraints get activated even when the agent is far from
the wall, while lower values imply less interference. As the RL agent is more
prone to make collisions at the beginning, higher values of $\lambda_{1}$ and
$\lambda_{2}$ enable the agent to learn quickly to move along the safer
centerline so as to avoid any violations. We vary the parameters of the CBF as
follows:

$\begin{split}&\lambda_{1,t_{s}}=\lambda_{1,0}(1-t_{s})\\\
&\lambda_{2,t_{s}}=\lambda_{2,0}(1-t_{s})\\\ \end{split}$ (9)

##  V Results

Our framework is implemented in the Unity Game Engine, with an example
representative image shown in Fig. 6. We test our controllers on $2$ track
maps, as shown in Figure 6. We conduct test races on this track against $2$
agents to compare with them. The races are conducted with initial position set
randomly either on the left or the right at the starting line with both
starting at the same longitudinal level. One position may be at an advantage
if it is closer to the racing line. Hence, we randomly choose the positions
with an equal chance of getting either position. We first give training
rewards obtained by curriculum learning to show the advantage of using
curriculum learning in Fig. 5(b). As can be observed, using curriculum
learning clearly with only model changes clearly beats the rewards without
using it. $t_{\text{start}}$ and $t_{\text{end}}$ are chosen to be $500000$
and $1500000$. It is unfair to compare till $1500000$ steps, as the first
controller runs on a simple RL environment, but after $1500000$ steps both
environments are the same and our controller clearly beats the non-curriculum-
based RL controller in reward. Also, the number of wins with $4$ races each
between $3$ pairs of agents (so effectively $12$ races) conducted every
$250000$ steps clearly shows our controller wins most races at all times. With
the CBF-based curriculum added, due to negative reward for each CBF constraint
violation, the reward is less at the beginning but it eventually improves at
the end, achieving an even larger reward. Also, it is slightly better than
only using model-based curriculum learning in terms of the number of races
won.

(a)

(b)

Figure 5: (a) Training rewards (b) No of wins across training steps Figure 6:
The race setup

Finally, we conduct races among other baselines for comparison. All races
consist of $3$ laps, with the car which crosses the finish line first after
$3$ laps winning the race. In total $20$ races are conducted with each pair
for comparison.

Figure 7: The race win statistics

###  V-A Metrics

We compare the runs against the following metrics: 1. No. of wins; 2. Average
lap time; 3\. Average lateral distance from race line; 4. No of collisions
with wall; 5. No. of collisions with opponents from behind.

|  Avg. lap time |  Avg. lateral distance from | No of collisions
---|---|---|---
Wall | Opponent
| (in s) | raceline (in m) |  |
Ours | 28.8 | 1.03 | 250 | 85
Ours - CBF | 29.2 | 1.05 | 360 | 92
| Ours - raceline,
---
curriculum learning
29.5 | 1.80 | 589 | 78
MCTS + LQR | 29.6 | 0.53 | 212 | 64
End to end | 30.3 | 1.67 | 670 | 56
TABLE I: Race statistics of different methods

###  V-B Baseline methods

We compare against the following baselines: 1\. Ours 2\. Ours - CBF 3\. [5] :
Ours - raceline, curriculum learning 4\. MCTS + LQR : To compare against a
classical rule-based controller 5\. End-to-end : Not using any hierarchical
controller

Fig.6 contains the win statistics for all methods. As can be observed, Our
Method beats all other methods in most races. Especially with races against
[5], we observe that using racing line information and our proposed curriculum
learning approach is beneficial as compared to training without them. Table I
contains all the other statistics. As can be seen, we achieve better lap times
as compared to other methods. Also, we stay much closer to raceline as
compared to [5] after adding explicit raceline state as input. However, using
classical control like LQR is able to achieve better raceline tracking but may
not be able to achieve optimal speeds as the RL controller does. Also,
clearly, using the hierarchical controller yields better results than direct
End-to-end approach as it looks from the average lap time. In terms of the no
of collisions, it suggests that using CBF based curriculum helps the agent in
learning more robust safety as it has fewer collision with wall as compared to
not using CBF.

##  VI Conclusion and Future Work

In this work we propose a more realistic head-to-head racing environment to
race against with more closer to actual dynamics as compared to [5]. We then
propose a hierarchical control design with a high-level controller planning a
sequence of checkpoints as close to the racing line as possible and avoiding
collision with other agents. We then propose a curriculum-based learning
method to effectively learn an optimal policy. We compare the results with
other baseline methods. It is important to note here that this is a work in
progress and we admit that the experiments and inferences are incomplete (for
example, comparison with an RL controller trained with constant CBF parameters
should have been added to compare with [5]). This is because we were not able
to complete all experiments before the deadline. In the near work, we aim to
also use trajectory prediction for the opponent agent instead of MCTS. We also
aim to test against more complex environments where the agents would be
allowed to take a pit stop for tire changes due to wear and more agents can be
added into the game.

## References

  * [1] J. Betz, H. Zheng, A. Liniger, U. Rosolia, P. Karle, M. Behl, V. N. Krovi, and R. Mangharam, “Autonomous vehicles on the edge: A survey on autonomous vehicle racing,” _IEEE Open Journal of Intelligent Transportation Systems_ , vol. 3, pp. 458–488, 2022.
  * [2] J. M. Rieber, H. Wehlan, and F. Allgower, “The roborace contest,” _IEEE Control Systems_ , vol. 24, pp. 57–60, 2004.
  * [3] M. O’Kelly, H. Zheng, D. Karthik, and R. Mangharam, “F1tenth: An open-source evaluation environment for continuous control and reinforcement learning,” in _Neural Information Processing Systems_ , 2019.
  * [4] A. Wischnewski, M. Geisslinger, J. Betz, T. Betz, F. Fent, A. Heilmeier, L. Hermansdorfer, T. Herrmann, S. Huch, P. Karle, F. Nobis, L. Ogretmen, M. Rowold, F. Sauerbeck, T. Stahl, R. Trauth, M. Lienkamp, and B. Lohmann, “Indy autonomous challenge - autonomous race cars at the handling limits,” _ArXiv_ , vol. abs/2202.03807, 2022.
  * [5] R. S. Thakkar, A. S. Samyal, D. Fridovich-Keil, Z. Xu, and U. Topcu, “Hierarchical control for multi-agent autonomous racing,” _ArXiv_ , vol. abs/2202.12861, 2022.
  * [6] J. Herman, J. M. Francis, S. Ganju, B. Chen, A. Koul, A. K. Gupta, A. Skabelkin, I. Zhukov, M. Kumskoy, and E. Nyberg, “Learn-to-race: A multimodal control environment for autonomous racing,” _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 9773–9782, 2021.
  * [7] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick, “End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks,” in _AAAI Conference on Artificial Intelligence_ , 2019.
  * [8] F. Christ, A. Wischnewski, A. Heilmeier, and B. Lohmann, “Time-optimal trajectory planning for a race car considering variable tyre-road friction coefficients,” _Vehicle System Dynamics_ , vol. 59, pp. 588 – 612, 2021\.
  * [9] A. Jain and M. Morari, “Computing the racing line using Bayesian optimization,” _2020 59th IEEE Conference on Decision and Control (CDC)_ , pp. 6192–6197, 2020.
  * [10] A. Heilmeier, A. Wischnewski, L. Hermansdorfer, J. Betz, M. Lienkamp, and B. Lohmann, “Minimum curvature trajectory planning and control for an autonomous race car,” _Vehicle System Dynamics_ , vol. 58, pp. 1497 – 1527, 2020.
  * [11] C. Piao, X. Liu, and C. Lu, “Lateral control using parameter self-tuning LQR on autonomous vehicle,” in _2019 International Conference on Intelligent Computing, Automation and Systems (ICICAS)_ , 2019, pp. 913–917.
  * [12] F. Curinga, “Autonomous racing using model predictive control,” 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:46852961
  * [13] E. Alcalá, V. Puig, J. Quevedo, and U. Rosolia, “Autonomous racing using linear parameter varying-model predictive control (LPV-MPC),” _Control Engineering Practice_ , vol. 95, p. 104270, 2020.
  * [14] A. Liniger, A. Domahidi, and M. Morari, “Optimization‐based autonomous racing of 1:43 scale RC cars,” _Optimal Control Applications and Methods_ , vol. 36, pp. 628 – 647, 2015.
  * [15] D. Kalaria, Q. Lin, and J. M. Dolan, “Delay-aware robust control for safe autonomous driving and racing,” _ArXiv_ , vol. abs/2208.13856, 2022.
  * [16] U. Rosolia and F. Borrelli, “Learning how to autonomously race a car: A predictive control approach,” _IEEE Transactions on Control Systems Technology_ , vol. 28, pp. 2713–2719, 2019.
  * [17] D. Kalaria, Q. Lin, and J. M. Dolan, “Adaptive planning and control with time-varying tire models for autonomous racing using extreme learning machine,” _ArXiv_ , vol. abs/2303.08235, 2023.
  * [18] M. Jaritz, R. de Charette, M. Toromanoff, E. Perot, and F. Nashashibi, “End-to-end race driving with deep reinforcement learning,” _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 2070–2075, 2018\.
  * [19] D. Kalaria, Q. Lin, and J. M. Dolan, “Towards safety assured end-to-end vision-based control for autonomous racing,” _ArXiv_ , vol. abs/2303.02267, 2023.
  * [20] X. Sun, M. Zhou, Z. Zhuang, S. Yang, J. Betz, and R. Mangharam, “A benchmark comparison of imitation learning-based control policies for autonomous racing,” _2023 IEEE Intelligent Vehicles Symposium (IV)_ , pp. 1–5, 2022\.
  * [21] A. Raji, A. Liniger, A. Giove, A. Toschi, N. Musiu, D. Morra, M. Verucchi, D. Caporale, and M. Bertogna, “Motion planning and control for multi vehicle autonomous racing at high speeds,” _2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)_ , pp. 2775–2782, 2022\.
  * [22] A. Liniger, “Path planning and control for autonomous racing,” 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:70032582
  * [23] C. Jung, S. Lee, H. Seong, A. Finazzi, and D. H. Shim, “Game-theoretic model predictive control with data-driven identification of vehicle model for head-to-head autonomous racing,” _ArXiv_ , vol. abs/2106.04094, 2021.
  * [24] J. M. Francis, B. Chen, S. Ganju, S. Kathpal, J. Poonganam, A. Shivani, S. Genc, I. Zhukov, M. Kumskoy, A. Koul, J. Oh, and E. Nyberg, “Learn-to-race challenge 2022: Benchmarking safe learning and cross-domain generalisation in autonomous racing,” _ArXiv_ , vol. abs/2205.02953, 2022\.
  * [25] D. W. Thompson, “Galileo and the principle of similitude,” _Nature_ , vol. 95, pp. 426–427, 1915. [Online]. Available: https://api.semanticscholar.org/CorpusID:3980913
  * [26] Y. Emam, G. Notomista, P. Glotfelter, Z. Kira, and M. Egerstedt, “Safe reinforcement learning using robust control barrier functions,” _IEEE Robotics and Automation Letters_ , 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:249954121
  * [27] A. Ames, S. D. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada, “Control barrier functions: Theory and applications,” _2019 18th European Control Conference (ECC)_ , pp. 3420–3431, 2019.


# Boosting Span-based Joint Entity and Relation Extraction via Squence Tagging
Mechanism

Bin Ji, Shasha Li, Jie Yu, Jun Ma, Huijun Liu
College of Computer, National University of Defense Technology
<EMAIL_ADDRESS>

###### Abstract

Span-based joint extraction simultaneously conducts named entity recognition
(NER) and relation extraction (RE) in text span form. Recent studies have
shown that token labels can convey crucial task-specific information and
enrich token semantics. However, as far as we know, due to completely abstain
from sequence tagging mechanism, all prior span-based work fails to use token
label information. To solve this problem, we propose Sequence Tagging enhanced
Span-based Network (STSN), a span-based joint extraction network that is
enhanced by token BIO label information derived from sequence tagging based
NER. By stacking multiple attention layers in depth, we design a deep neural
architecture to build STSN, and each attention layer consists of three basic
attention units. The deep neural architecture first learns semantic
representations for token labels and span-based joint extraction, and then
constructs information interactions between them, which also realizes
bidirectional information interactions between span-based NER and RE.
Furthermore, we extend the BIO tagging scheme to make STSN can extract
overlapping entity. Experiments on three benchmark datasets show that our
model consistently outperforms previous optimal models by a large margin,
creating new state-of-the-art results111Training code and models will be
available at xxx..

##  1 Introduction

Joint entity and relation extraction aims to simultaneously detect entities
and semantic relations among these entities from unstructured texts. It serves
as a step stone for many downstream tasks such as question answering,
knowledge base population, and has recently become a major focus of NLP
researches.

Generally, joint entity and relation extraction can be classified into two
categories: sequence tagging based mode Li and Ji (2014); Miwa and Bansal
(2016); Katiyar and Cardie (2017); Ye et al. (2019); Lin et al. (2020); Zhao
et al. (2020) and span-based mode. Recently, extensive span-based approaches
have been investigated due to its good performance Luan et al. (2018); Dixit
and Al-Onaizan (2019); Eberts and Ulges (2019); Ji et al. (2020). Typically,
span-based approaches first split text into text spans as candidate entities;
then form span pairs as candidate relation tuples; finally, jointly classify
spans and span pairs. For example, in Figure 1, “Jack”, “Jack taught”…are text
spans; $<$“Jack”, “Jack taught”$>$, $<$“Jack”, “Harvard University”$>$…are
span pairs; and “Jack” is classified into PER, $<$“Jack”, “Harvard
University”$>$ is classified into Work-for. Although these approaches have
achieved promising results, they still suffer from insufficient semantics of
text spans, caused by excessively relying on the encoding ability of pre-
trained language models, e.g., ELMo Peters et al. (2018), BERT Devlin et al.
(2019).

Figure 1: A span-based joint extraction example from CoNLL04, which is tagged
by BIO tagging scheme for sequence tagging based NER.

Recent span-based work further enriches semantics by introducing other NLP
tasks, e.g., coreference resolution Luan et al. (2019), event detection Wadden
et al. (2019), but these work needs additional data annotations besides
annotations for NER and RE, which do not exist in many datasets. Actually,
prior sequence tagging based work Miwa and Bansal (2016); Bekoulis et al.
(2018); Ye et al. (2019); Zhao et al. (2020) has shown that token BIO labels
for sequence tagging based NER can convey crucial task-specific information
and greatly enrich token semantics, which turns out to deliver significant
performance improvements. For example, if the model knows that “Jack” is
tagged with PER label and “Harvard University” is tagged with ORG label
beforehand, it can easily infer that there may exist a Work-for relation
between them. However, to the best of our knowledge, all prior span-based work
fails to use token label information, due to that they completely abstain from
sequence tagging mechanism. Besides, by leveraging token label information in
RE, prior work Miwa and Bansal (2016); Bekoulis et al. (2018); Ye et al.
(2019); Zhao et al. (2020) constructs an information flow from NER to RE, but
fails to construct an information flow from RE to NER.

To address above issues, we propose Sequence Tagging enhanced Span-based
Network (STSN), a span-based joint extraction network that can leverage token
BIO label information derived from sequence tagging based NER. To build STSN,
we design a deep neural architecture by stacking multiple attention layers in
depth. The deep neural architecture first learns semantic representations of
token BIO labels (“label semantic representations” for short) for sequence
tagging based NER, and token semantic representations for span-based NER and
RE, and then constructs information interactions among the learned semantic
representations. Subsequently, STSN decodes these fully interactive semantic
representations.

Each of the deep stacked attention layers consists of three basic attention
units: (1) Entity&Relation to Label Attention (E&R-L-A), which can feed back
task information of span-based NER and RE to label semantic representations,
aiming to make label semantic representations can better capture task-specific
information; (2) Label to Entity Attention (L-E-A), which can inject NER-
specific information derived from label semantic representations into token
semantic representations for span-based NER; (3) Label to Relation Attention
(L-R-A), which can inject RE-specific information derived from label semantic
representations into token semantic representations for span-based RE.

It is worth noting that by taking E&R-L-A as medium, STSN builds bidirectional
information interactions between span-based NER and RE, which will be proven
to be of great significance in $\S 4.4$. Besides, L-E-A makes STSN can utilize
token label information in span-based NER, which is important but all prior
joint extraction work fails to do, as far as we know. For example, a model can
easily classify “Harvard University” to ORG type, if it knows the span is
tagged with ORG label beforehand. Moreover, to make STSN can extract
overlapping entity, we extend the BIO tagging scheme to tag overlapping
entity, details will be described in §4.1. Furthermore, STSN can leverage
fixed-size token label embeddings by concatenation manner.

We conduct extensive experiments on ACE05, CoNLL04 and ADE to evaluate STSN.
Experimental results show that STSN overwhelmingly outperforms previous best
performed models on above three benchmark datasets, creating new state-of-the-
art results. To the best of our knowledge, STSN is the first span-based joint
extraction network that incorporates sequence tagging mechanism, making use of
advantages of both span-based and sequence tagging based joint extraction.

##  2 Related Works

##### Span-based joint extraction.

Span-based joint entity and relation extraction has been widely investigated.
Luan et al. (2018) propose the first published span-based model, obtaining
span representations the same as Lee et al. (2017), and sharing them in both
NER and RE. Dixit and Al-Onaizan (2019) realize span-based extraction by
obtaining span representations through a BiLSTM over concatenation of ELMo,
word and character embeddings. Eberts and Ulges (2019) propose SpERT, which
takes BERT as backbone and dramatically reduces training complexity by
adopting negative sampling. Ji et al. (2020) further improve SpERT by
enriching semantic representations with local and global features. Some
approaches further promote system performance by incorporating other NLP
tasks. Luan et al. (2019) propose DyGIE, a span-based joint extraction model
that incorporates coreference resolution. Wadden et al. (2019) introduce event
extraction to DyGIE and propose DyGIE++. Compared to these work, our span-
based model incorporates sequence tagging mechanism, and consists of cascaded
attention layers to fully leverage token label information, which does not
need additional data annotations.

Figure 2: Model architecture of STSN. STSN first generates BERT embeddings for
a input sentence and projects them to initial label semantic representations,
token representations for span-based NER and RE, i.e.,
$H_{L}^{0},H_{E}^{0},H_{R}^{0}$; then the deep stacked attention layers fully
inject (1) task information derived from $H_{E}$ and $H_{R}$ into $H_{L}$, and
(2) task-specific information derived from $H_{L}$ into $H_{E}$ and $H_{R}$,
in a recursive way, obtaining the final outputs, i.e.,
$H_{L}^{N},H_{E}^{N},H_{R}^{N}$; finally, three linear decoders are designed
to decode token BIO labels for sequence tagging based NER, span-based NER and
RE with $H_{L}^{N},H_{E}^{N},H_{R}^{N}$. Meanwhile, STSN constructs
bidirectional information interactions between span-based NER and RE, by
taking E&R-L-A as medium.

##### Token label information.

Recent work has proven that token BIO labels for sequence tagging based NER
can convey crucial task-specific information for NLP tasks Wang et al. (2018);
Cui and Zhang (2019). However, this crucial information has not been fully
studied in joint entity and relation extraction. Generally, prior sequence
tagging based work Miwa and Bansal (2016); Bekoulis et al. (2018); Ye et al.
(2019) explicitly concatenates fixed-size embeddings for token BIO labels to
token semantic representations for RE. Zhao et al. (2020) further propose a
deep fusion manner. Limited by the sequence tagging mechanism, these work only
leverages token label information in RE, and constructs a unidirectional
information flow from NER to RE. In contrast, our model is in span-based mode,
making it possible to use token label information in both NER and RE. Besides,
our model constructs bidirectional information interactions between NER and
RE.

##  3 Model

In this section, we will describe the Sequence Tagging enhanced Span-based
Network (STSN) in detail, as Figure 2 shows. For a given sentence
${\mathcal{S}=(t_{1},t_{2},t_{3},…,t_{n})}$, where $t_{i}$ denotes the
$i$-$th$ token in the sentence, we first generate its semantic representations
using pre-trained language models ($\S$3.1); then, we construct a deep neural
architecture by stacking multiple attention layers in depth and each layer
consists of three basic attention units -- Entity&Relation to Label Attention
(E&R-L-A), Label to Entity Attention (L-E-A) and Label to Relation Attention
(L-R-A), which are used to build information interactions between token labels
derived from sequence tagging based NER and span-based joint extraction
($\S$3.2); finally, we design three linear decoders for sequence tagging based
NER, span-based NER and RE respectively ($\S$3.3).

###  3.1 Embedding Layer

We use BERT Devlin et al. (2019) as the default word embedding generator. For
$\mathcal{S}$, BERT first tokenizes it with the WordPiece vocabulary Wu et al.
(2016) to get the input sequence. For each sequence element, its input
representation is the element-wise addition of WordPiece embedding, positional
embedding, and segment embedding. Then, a list of input embeddings
$\mathbf{H}\in\mathbb{R}^{len\ast dim}$ are obtained, where len is the
sequence length and dim is the hidden size. A series of pre-trained
Transformer blocks Vaswani et al. (2017) are then used to project $\mathbf{H}$
into BERT embedding sequence.

${\mathbf{E}_{\mathcal{S}}=\\{X_{1},X_{2},X_{3},...,X_{len}\\}}$

Where $X_{1}$ and $X_{len}$ are the BERT embeddings for the specific [CLS] and
[SEP] tokens.

BERT may tokenize a token into several sub-tokens to alleviate out of
vocabulary problem. In STSN, we apply $max$-$pooling$ to BERT embeddings for
the tokenized sub-tokens of one token to obtain token semantic representation
(“token representation” for short), aiming to align token sequence and token
representation sequence. We remove the BERT embeddings for [CLS] and [SEP],
and denote the representation sequence for $\mathcal{S}$ as:

${\hat{\mathbf{E}}_{\mathcal{S}}=\\{\hat{X}_{1},\hat{X}_{2},\hat{X}_{3},...,\hat{X}_{n}\\}}$

Where ${\hat{\mathbf{E}}_{\mathcal{S}}\in\mathbb{R}^{n*d}}$ and $d$ is the
BERT embedding dimension.

###  3.2 Attention Layer

####  3.2.1 Deep Stacked Neural Architecture

We deep stack multiple attention layers to construct a deep neural
architecture for STSN, and each of the stacked attention layers consists of
three basic attention units, i.e. E&R-L-A, L-E-A and L-R-A, as shown in Figure
2.

The deep stacked neural architecture learns three semantic representations,
i.e., semantic representations of token BIO labels for sequence tagging based
NER (denoted as $H_{L}$), token representations for span-based NER (denoted as
$H_{E}$) and token representations for span-based RE (denoted as $H_{R}$), of
which the dimensions keep the same. Besides, we denote $H_{C}$ as the
concatenation of $H_{E}$ and $H_{R}$, which keeps the same dimension to
$H_{E}$ and $H_{R}$ via a FFN.

$H_{C}=[H_{E};H_{R}]W^{C}+b^{C}$

Where $W^{C}$, $b^{C}$ are trainable FFN parameters.

For the first stacked attention layer, $\hat{E_{\mathcal{S}}}$ is projected
into $H_{L}^{0}$, $H_{E}^{0}$ and $H_{R}^{0}$ respectively, which can be
formulated as:

$[Layer]^{1}=\left\\{\begin{array}[]{lrc}H_{C}^{0}=[H_{E}^{0};H_{R}^{0}]W_{C}^{0}+b_{C}^{0}\\\
H_{L}^{1}=E\&R$-$L$-$A(H_{L}^{0},H_{C}^{0},H_{C}^{0})\\\
H_{E}^{1}=L$-$E$-$A(H_{E}^{0},H_{L}^{1},H_{L}^{1})\\\
H_{R}^{1}=L$-$R$-$A(H_{R}^{0},H_{L}^{1},H_{L}^{1})\end{array}\right.$

Figure 3: Model architectures of E&R-L-A, L-E-A and L-R-A. The three units
interact with each other through model inputs.

Where $W_{C}^{0}\in\mathbb{R}^{2d*d}$ and $b_{C}^{0}\in\mathbb{R}^{d}$ are
trainable FFN parameters. We then pass the layer outputs, i.e., $H_{L}^{1}$,
$H_{E}^{1}$, $H_{R}^{1}$, to the next layer and repeat this procedure in a
recursive manner, until we obtain the outputs of the $N$-$th$ layer, i.e.,
$H_{L}^{N}$, $H_{E}^{N}$ and $H_{R}^{N}$, which are taken as the final
semantic representations. Now, $H_{E}^{N}$ and $H_{R}^{N}$ fully contain task-
specific information and are suitable for span-based NER and RE.

As shown in Figure 2, in each stacked attention layer, we construct
bidirectional information interactions among the three basic attention units.
Specifically, E&R-L-A and L-E-A can directly interact with each other, as well
E&R-L-A and L-R-A. Therefore, by taking E&R-L-A as medium, L-E-A and L-R-A can
interact with each other too, which constructs bidirectional information
interactions between span-based NER and RE in essence. Besides, L-E-A makes
STSN can utilize token label information in span-based NER.

####  3.2.2 Three basic Attention Units

As Figure 3 shows, the three basic attention units share a general neural
architecture, but differ in model inputs. We first describe the general
architecture, then introduce their implementation details.

##### General architecture.

The general architecture has two sub-layers: the first is a multi-head
attention, the second is a position-wise Feed Forward Network (FFN). A
residual connection is adopted around each of the two sub-layers, followed by
layer normalization. The general architecture is similar to Transformer
encoder layer, but they differ in model inputs.

Multi-head attention has proven to be effective for capturing long-range
dependencies by explicitly attending to all positions. Therefore, we apply the
multi-head attention to capture task information and task-specific
information. The multi-head attention owns a serious of $h$ parallel heads and
requires three inputs, i.e., query, key and value.

$head_{i}=softmax(\frac{(QW_{i}^{Q})(KW_{i}^{K})^{T}}{\sqrt{d/h}}(VW_{i}^{V}))$
$I=Concat(head_{1},….head_{h})W^{o}$

Where $\\{Q,K,V\\}\in\mathbb{R}^{n*d}$ are query, key and value;
$\\{W_{i}^{Q},W_{i}^{K},W_{i}^{V}\\}\in{\mathbb{R}^{d*d/h}}$,
$W^{o}\in\mathbb{R}^{d*d}$ are trainable model parameters; and
$I\in\mathbb{R}^{n*d}$ is the output. The multi-head attention learns the
pairwise relationship between the query and key, and finally outputs the
captured information by weighted summation across all instances. Then the
residual connection conducts element-wise addition of the captured information
and query, accomplishing information injection in a shallow manner.

The position-wise FFN contains two linear transformations with a ReLU
activation in between.

$FFN(I)=max(0,IW_{1}+b_{1})W_{2}+b_{2}$

Where $\\{W_{1},W_{2}\\}\in\mathbb{R}^{d*d}$ and
$\\{b_{1},b_{2}\\}\in\mathbb{R}^{d}$ are trainable FFN parameters. By
conducting linear transformation on the outputs of multi-head attention, FFN
projects the captured information to the representation space of $I$, which
further accomplishes information injection in a deep manner.

##### E&R-L-A.

E&R-L-A (see Figure 3) takes $H_{L}$ as query, $H_{C}$ as key and value
respectively, and can feed back task information of span-based NER and RE,
which is captured from $H_{C}$, to $H_{L}$, aiming to make $H_{L}$ can better
learn task-specific information.

##### L-E-A and L-R-A.

L-E-A (see Figure 3) takes $H_{E}$ as query, $H_{L}$ as key and value
respectively, and is capable of injecting task-specific information of span-
based NER, which is captured from $H_{L}$, into $H_{E}$. Similarly, L-R-A (see
Figure 3) takes $H_{R}$ as query, $H_{L}$ as key and value respectively, and
can inject RE-specific information into $H_{R}$.

###  3.3 Decoding Layer

We propose three linear decoders for sequence tagging based NER, span-based
NER and RE respectively.

##### Decoder for sequence tagging based NER.

This decoder first utilizes a FFN to reduce label semantic representation,
i.e., $H_{L}^{N}$, to space of token BIO label size; then utilizes $softmax$
to calculate probability distributions on the label size space.

$\hat{y}^{L}=softmax(H_{L}^{N}W^{L}+b^{L})$

Where $W^{L}\in\mathbb{R}^{d*l}$ and $b^{L}\in\mathbb{R}^{l}$ are trainable
FFN parameters; $l$ is the token BIO label size. The training objective is to
minimize the cross-entropy loss.

${\mathcal{L}_{L}=-\frac{1}{M_{L}}\sum\limits_{i=1}^{M_{L}}{y}_{i}^{{L}}\log\hat{y}_{i}^{{L}}}$

Where ${y}^{{L}}$ is the one-hot vector of gold token BIO label for sequence
tagging based NER; ${M_{L}}$ is the number of token instances. For each token
BIO label, we maintain a label embedding with fixed size (denoted as
$H^{N}_{L^{\prime}}$) for it, which is trained during model training. We use
gold token BIO labels during model training and predicted labels during
inference.

##### Decoder for span-based NER.

We first add NoneEntity type to the pre-defined entity types. Spans will be
decoded into NoneEntity space if they are not predicted as entities. We
formulate an arbitrary span from $\mathcal{S}$ to help introduce the rest:
${\mathbf{s}=(t_{i},t_{i+1},t_{i+2}...,t_{i+j})}$.

We obtain span representation by concatenating token representations of span
head and tail, and a span width embedding, following Luan et al. (2018).

$E_{\mathbf{s}}=[H_{E^{\prime},i}^{N};H_{E^{\prime},i+j}^{N};W_{j+1}]$

Where $H_{E^{\prime},i}^{N}$ and $H_{E^{\prime},i+j}^{N}$ are from
[$H_{E}^{N};H^{N}_{L^{\prime}}$], which is the concatenation of $H_{E}^{N}$
and $H^{N}_{L^{\prime}}$; $W_{j+1}$ is a span width embedding with fixed size,
which is trained during model training.

$E_{\mathbf{s}}$ first passes through a FFN, then is fed into ${softmax}$,
which yields a posterior on the space of pre-defined entity type size.

${\hat{y}^{\mathbf{s}}=softmax({E}_{\mathbf{s}}W^{\mathbf{s}}+b^{\mathbf{s}})}$

Where $W^{\mathbf{s}}$ and $b^{\mathbf{s}}$ are trainable FFN parameters. The
training objective is to minimize the cross-entropy loss.

${\mathcal{L}_{E}=-\frac{1}{M_{E}}\sum\limits_{i=1}^{M_{E}}{y}_{i}^{\mathbf{s}}\log\hat{y}_{i}^{\mathbf{s}}}$

Where ${y}^{\mathbf{s}}$ is the one-hot vector of gold span type; ${M_{E}}$ is
the number of span instances.

##### Decoder for span-based RE.

We first formulate an arbitrary span pair from $\mathcal{S}$:
$\mathbf{r}=<\textbf{s}_{1},\textbf{s}_{2}>$, where $\mathbf{s}_{1}$ and
$\mathbf{s}_{2}$ are two text spans. Then we obtain relation representation by
concatenation manner.

$E_{\mathbf{r}}=[E_{\mathbf{s}_{1}};E_{\mathbf{s}_{2}};C_{\textbf{r}}]$

Where $E_{\mathbf{s}_{1}}$ and $E_{\mathbf{s}_{2}}$ are semantic
representations of $\textbf{s}_{1}$ and $\textbf{s}_{2}$. It is worth noting
that we obtain $E_{\mathbf{s}_{1}}$ and $E_{\mathbf{s}_{2}}$ with
[$H_{R}^{N};H^{N}_{L’}$] in the same way as in the NER decoder. Following
Eberts and Ulges (2019), $C_{\textbf{r}}$ is the semantic representation of
relation context, which is obtained with [$H_{R}^{N};H^{N}_{L’}$] as well.

$E_{\mathbf{r}}$ first passes through a FFN, then is fed into ${sigmoid}$,
which yields probability distributions on the space of pre-defined relation
type size.

${\hat{y}^{\mathbf{r}}=\sigma({E}_{\mathbf{r}}W^{\mathbf{r}}+b^{\mathbf{r}})}$

Where $W^{\mathbf{r}}$ and $b^{\mathbf{r}}$ are trainable FFN parameters.
Given a confidence threshold $\alpha$, any relation with a $score\geq\alpha$
is considered activated. If none is activated, the model assumes that the span
pair holds no pre-defined relation. The training objective is to minimize the
binary cross-entropy loss.

${\mathcal{L}_{R}=-\frac{1}{M_{R}}\sum\limits_{i=1}^{M_{R}}(y_{i}^{\mathbf{r}}\log\hat{y}_{i}^{\mathbf{r}}+(1-y_{i}^{\mathbf{r}})\log(1-\hat{y}_{i}^{\mathbf{r}}))}$

Where ${y}^{\mathbf{r}}$ is the one-hot vector of gold relation type for span
pair; ${M_{R}}$ is the number of span pair instances.

Finally, we optimize the following joint objective function during model
training.

$\mathcal{L}_{joint}(W;\theta)=\mathcal{L}_{L}+\mathcal{L}_{E}+\mathcal{L}_{R}$

Figure 4: An example of overlapping entities from ADE, tagged by the extended
BIO tagging scheme.

##  4 Experiments

###  4.1 Experimental Setup

##### Datasets.

We evaluate our model on ACE05 Doddington et al. (2004), CoNLL04 Roth and Yih
(2004) and ADE Gurulingappa et al. (2012). For ACE05, we use the same entity
and relation types, data splits, and pre-processing as Luan et al. (2019);
Wang and Lu (2020); for CoNLL04, we use the same train-test split as Zhao et
al. (2020); Wang and Lu (2020); for ADE, we maintain two dataset versions
following Eberts and Ulges (2019); Wang and Lu (2020), one is the full ADE
dataset with 119 instances that contain overlapping entity, the other without
these instances.

##### Extended BIO tagging scheme.

Aiming to make STSN can extract the overlapping entities from the full ADE
dataset, we extend the BIO tagging scheme. We first give two term definitions:

  * •

two-fold overlapping entities is a pair of overlapping entities that the
overlapping tokens are only contained in the two entities.

  * •

preceding entity is distinguished by entity head location, while if two
entities have the same head location, the entity with longer length is chosen
as the preceding entity.

As Figure 4 shows, “Codeine” and “Codeine intoxication” are two-fold
overlapping entities, and “Codeine intoxication” is the preceding entity.

Detailed tagging rule is that we first tag the preceding entity with BIO
tagging scheme; then for the overlapping entity, we append its BIO labels to
existing labels, separated by “/”. For example, we tag the overlapping
“Codeine” with “B-AE/B-DRUG”.

As all overlapping entities in the full ADE dataset are two-fold, we tag the
dataset with the extended BIO tagging scheme. For other datasets, we tag them
with BIO tagging scheme.

##### Evaluation measures.

We use the F1 measure to evaluate model performance. For NER, a predicted
entity is considered correct if both its type and boundaries (span head for
ACE05) match ground truth. For RE, we adopt two evaluation measures: a
predicted relation is considered correct if its relation type and boundaries
of the two entities match ground truth (denoted as RE); while RE+ needs both
relation type and the two entities all match ground truth. More discussion of
the evaluation settings can be found in Wang and Lu (2020).

##### Implementation details.

For all datasets, we build STSN by stacking 4 attention layers in depth, and
evaluate it upon bert-base-cased Devlin et al. (2019) and albert-xxlarge-v1
Lan et al. (2019) with a single NVIDIA RTX 3090 GPU. We optimize STSN using
AdamW for 100 epochs with a learning rate of 5e-5, a linear scheduler with a
warmup ratio of 0.1 and a weight decay of 10-2. We set the training batch size
to 4, dimensions of $W_{j+1},H_{L^{\prime}}^{N}$ to 150, $h$ for multi-head
attention to 8, span width threshold to 10 and relation threshold $\alpha$ to
0.4. We adopt negative sampling strategy following Eberts and Ulges (2019).
For ACE05 and CoNLL04, we run STSN for 20 runs and report the averaged F1 of
the top-5 runs. For ADE, we adopt 10-fold cross validation, run each fold for
20 runs and report the averaged F1 of the top-5 runs.

###  4.2 Main Results

Table 1 and Table 2 compare performances of STSN with previous optimal
results. We report the F1 scores for a fair comparison with prior work. Our
BERT-based model significantly outperforms all previous work including models
built upon ALBERT, where our model performances are further improved by using
ALBERT. For NER, our best model delivers absolute F1 gains of +1.7,
+1.5(micro)/+2.7(macro), +1.9 on ACE05, CoNLL04 and ADE(without overlap)
respectively, while better ones for RE, by +3.5(RE)/+3.7(RE+),
+2.6(micro)/+2.9(macro) and +4.2 respectively. All these overwhelming gains
demonstrate the effectiveness of leveraging token label information in span-
based NER and RE, as well as the bidirectional information interactions
between the two tasks.

Model | NER | RE | RE+
---|---|---|---
Li and Ji Li and Ji (2014) $\vartriangle$ | 80.8 | 52.1 | 49.5
Miwa et al. Miwa and Bansal (2016) | 83.4 | - | 55.6
Katiyar et al. Katiyar and Cardie (2017) | 82.6 | 55.9 | 53.6
Sun et al. Sun et al. (2018) | 83.6 | - | 59.6
Li et al. Li et al. (2019) † | 84.8 | - | 60.2
Dixit and Al Dixit and Al-Onaizan (2019) $\perp$ | 86.0 | 62.8 | -
Luan et al. Luan et al. (2019) | 88.4 | 63.2 | -
Wadden et al. Wadden et al. (2019) † | 88.6 | 63.4 | -
Wang et al. Wang et al. (2018) † | 87.2 | 66.7 | -
Lin et al. Lin et al. (2020) † | 88.8 | 67.5 |
Wang and Lu Wang and Lu (2020) ‡ | 89.5 | 67.6 | 64.3
Ji et al. Ji et al. (2020) † | 89.6 | - | 65.2
ours † | 90.2 | 68.1 | 66.9
ours ‡ | 91.3 | 71.1 | 68.9
Table 1: Main results on ACE05 using micro-averaged F1. $\perp$: LSTM + ELMo; †: bert-base-cased (or -uncased); ‡: albert-xxlarge-v1. | Model | NER | RE+
---|---|---|---
CoNLL04 | Miwa et al. Miwa and Sasaki (2014) $\vartriangle$ | 80.7 | 61.0
Bekoulis et al. Bekoulis et al. (2018) $\blacktriangle$ | 83.9 | 62.0
Nguyen et al.Nguyen and Verspoor (2019) $\blacktriangle$ | 86.2 | 64.4
Zhang et al. Zhang et al. (2017) $\vartriangle$ | 85.6 | 67.8
Li et al. Li et al. (2019) $\vartriangle$ † | 87.8 | 68.9
Eberts et al. Eberts and Ulges (2019) $\vartriangle$ † | 88.9 | 71.5
Eberts et al. Eberts and Ulges (2019) $\blacktriangle$ † | 86.3 | 72.9
Zhao et al. Zhao et al. (2020) $\vartriangle$ $\perp$ | 90.6 | 73.0
Wang and Lu Wang and Lu (2020) $\vartriangle$ ‡ | 90.1 | 73.6
Wang and Lu Wang and Lu (2020) $\blacktriangle$ ‡ | 86.9 | 75.4
Ji et al. Ji et al. (2020) $\vartriangle$ † | 90.2 | 74.3
ours $\vartriangle$ † | 90.8 | 75.2
ours $\vartriangle$ ‡ | 91.7 | 76.9
ours $\blacktriangle$ † | 88.3 | 77.1
| ours $\blacktriangle$ ‡ | 89.6 | 78.3
ADE | Li et al. Li et al. (2016) $\blacktriangle$ | 79.5 | 63.4
Li et al. Li et al. (2017) $\blacktriangle$ | 84.6 | 71.4
Bekoulis et al. Bekoulis et al. (2018) $\blacktriangle$ | 86.4 | 74.6
Eberts et al. Eberts and Ulges (2019) $\blacktriangle$ † | 89.3 | 79.2
Eberts et al. Eberts and Ulges (2019) $\blacktriangle\spadesuit$ † | 89.3 | 78.8
Wang and Lu Wang and Lu (2020) $\blacktriangle$ ‡ | 89.7 | 80.1
Zhao et al. Zhao et al. (2020) $\blacktriangle$ $\perp$ | 89.4 | 81.1
Ji et al. Ji et al. (2020) $\blacktriangle$ † | 90.6 | 80.7
ours $\blacktriangle$ † | 91.4 | 83.2
ours $\blacktriangle$ ‡ | 92.5 | 84.9
| ours $\blacktriangle\spadesuit$ † | 91.0 | 83.4
| ours $\blacktriangle\spadesuit$ ‡ | 91.9 | 84.5
Table 2: Main results on CoNLL04 and ADE. $\vartriangle$: micro-averaged F1;
$\blacktriangle$: macro-averaged F1; $\spadesuit$: with overlap; $\perp$: LSTM
+ ELMo; †: bert-base-cased (or -uncased); ‡: albert-xxlarge-v1.

We evaluate STSN for extracting overlapping entity on the full ADE dataset.
Experimental results are shown in Table 2, marked by $\spadesuit$. Compared to
Eberts and Ulges (2019), our model delivers overwhelming performance gains.
Specifically, on NER and RE, our BERT-based model delivers absolute F1 gains
of +1.7 and +4.6, while our ALBERT-based model delivers +2.6 and +5.7. These
gains validate the effectiveness of both the proposed extended BIO tagging
scheme and STSN.

###  4.3 Analysis

Following established line of work Eberts and Ulges (2019); Zhao et al.
(2020), we inspect detailed model performance in two aspects by taking SpERT
Eberts and Ulges (2019) as baseline. For fair comparison, we utilize the BERT-
based STSN.

Figure 5: NER performance comparisons of the baseline and STSN under different
grouped entity lengths on the ACE05 dev set. Figure 6: NER and RE performances
of the baseline and STSN under different grouped sentence lengths on the ACE05
dev set.

####  4.3.1 Performance against Entity Length

Figure 5 shows NER performance comparisons of the baseline and STSN under
different entity distances on the ACE05 dev set. We partition entity lengths,
which is restricted by span width threshold, into 1-2, 3-4, 5-6, 7-8, 9-10. We
can observe that STSN consistently outperforms the baseline under all length
intervals. Moreover, absolute F1 gains achieved by STSN are further enhanced
when entity length increases. In particular, STSN obtains 13.7 and 20.5
absolute F1 gains when entity length is 7-8 and 9-10 respectively,
demonstrating that STSN is more effective in terms of long entities.

####  4.3.2 Performance against Sentence Length

We analyze performances of the baseline and STSN under different grouped
sentence lengths on the ACE05 dev set. As Figure 6 shows, we partition the
sentence length into 0-19, 20-34, 35-49, $\geq$50\. We can observe that STSN
performs way better than the baseline under all sentence lengths. Moreover,
the gains achieved by STSN are further enhanced when the sentence length
increases. In particular, STSN outperforms the baseline by 11.1 and 16.8
absolute F1 scores on NER and RE, when the sentence length is greater than or
equal to 50. These results demonstrate that STSN is more effective in terms of
long sentences.

Method |  | Entity
---
(F1)
| Relation
---
(F1)
STSN + |  |
1 AttentionLayer | 87.8 | 59.3
2 AttentionLayers | 88.6 | 59.7
3 AttentionLayers | 89.3 | 62.1
4 AttentionLayers | 89.2 | 62.8
5 AttentionLayers | 89.0 | 62.6
6 AttentionLayers | 89.2 | 61.9
Table 3: Ablations of attention layers on ACE05 dev set. Method |  | Entity
---
(F1)
| Relation
---
(F1)
STSN | 89.2 | 62.8
-LabelEmbedding | 88.8 | 62.3
-E&R-L-A | 88.4 | 60.6
-AttentionLayer | 87.2 | 58.6
Table 4: Ablations of model components on ACE05 dev set.

###  4.4 Ablation Study

We conduct ablation studies on the BERT-based STSN, and use the ACE05 dev set
to report ablation results.

##### Ablations of attention layers.

We conduct ablations of attention layers by deep stacking different amounts of
attention layers in STSN. Table 3 shows ablation results, from which we can
conclude that: (1) the model with four stacked layers performs the best; (2)
the model consists of one attention layer performs the worst, due to that one
layer cannot fully inject token label information into semantic
representations for span-based joint extraction; (3) as the model depth
increases, model performance first significantly increases and then slightly
decreases. This is because that deep models make it easier to fully inject
information, while much deeper models tend to inject more noise, which imposes
negative impacts on model performance.

##### Ablations of model components.

Table 4 reports ablation results, where (1) -LabelEmbedding denotes ablating
token label embeddings, which is realized by removing $H^{N}_{L^{\prime}}$
from decoders for span-based NER and RE; (2) -E&R-L-A denotes ablating the
bidirectional information interactions between span-based NER and RE, which is
realized by making E&R-L-A take $H_{L}$ as query, key and value respectively
in all stacked attention layers; (3) -AttentionLayer denotes ablating all
stacked attention layers, where ${\hat{\mathbf{E}}_{\mathcal{S}}}$ is directly
projected into $H_{L}^{N}$, $H_{E}^{N}$ and $H_{R}^{N}$.

Ablation results show that: (1) token label embeddings impose minor positive
impacts on model performance, by delivering +0.4 and +0.5 absolute F1 gains on
NER and RE respectively; (2) bidirectional information interactions
significantly improve model performance, by delivering +0.8 and +2.2 absolute
F1 gains on NER and RE respectively; (3) stacked attention layers boost model
performance, delivering +2.0 and +4.2 absolute F1 gains on NER and RE
respectively.

Reasons can be summarized as that for span-based NER and RE: (1) the deep
stacked attention layers can fully inject token label information into
semantic representations for them, which greatly enriches their semantics.
While compared to above deep fusion way, shallow concatenation way achieved by
concatenating token label embeddings plays a negligible role in enriching
their semantics; (2) the bidirectional information interactions between the
two tasks can greatly enrich their semantics, by fully leveraging task-
specific information of each other.

##  5 Conclusion

In this paper, we propose the Sequence Tagging enhanced Span-based Network
(STSN) for joint entity and relation extraction. By adopting a deep stacked
neural architecture composed of three basic attention units, STSN can leverage
token BIO label information in span-based joint entity and relation
extraction. Moreover, the deep neural architecture realizes bidirectional
information interactions between span-based NER and RE, which greatly promotes
information interactions. Furthermore, we extend the BIO tagging scheme to
make STSN can extract overlapping entity. Experiments on three benchmark
datasets show that our model overwhelmingly outperforms other competing
approaches, creating new state-of-the-art results.

## References

  * Bekoulis et al. (2018) Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018.  Joint entity recognition and relation extraction as a multi-head selection problem.  _CoRR_ , abs/1804.07847.
  * Cui and Zhang (2019) Leyang Cui and Yue Zhang. 2019.  Hierarchically-refined label attention network for sequence labeling.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4115–4128, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dixit and Al-Onaizan (2019) Kalpit Dixit and Yaser Al-Onaizan. 2019.  Span-level model for relation extraction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5308–5314, Florence, Italy. Association for Computational Linguistics.
  * Doddington et al. (2004) George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Eberts and Ulges (2019) Markus Eberts and Adrian Ulges. 2019.  Span-based joint entity and relation extraction with transformer pre-training.  _arXiv preprint arXiv:1909.07755_.
  * Gurulingappa et al. (2012) Harsha Gurulingappa, Abdul Mateen Rajput, and Angus Roberts. 2012.  Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports.  _J. Biomed. Informatics_ , 45(5):885–892.
  * Ji et al. (2020) Bin Ji, Jie Yu, Shasha Li, Jun Ma, Qingbo Wu, Yusong Tan, and Huijun Liu. 2020.  Span-based joint entity and relation extraction with attention-based span-specific and contextual semantic representations.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 88–99.
  * Katiyar and Cardie (2017) Arzoo Katiyar and Claire Cardie. 2017.  Going out on a limb: Joint extraction of entity mentions and relations without dependency trees.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 917–928, Vancouver, Canada. Association for Computational Linguistics.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  ALBERT: A lite BERT for self-supervised learning of language representations.  _CoRR_ , abs/1909.11942.
  * Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.  End-to-end neural coreference resolution.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics.
  * Li et al. (2017) Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji. 2017.  A neural joint model for entity and relation extraction from biomedical text.  _BMC bioinformatics_ , 18(1):1–11.
  * Li et al. (2016) Fei Li, Yue Zhang, Meishan Zhang, and Donghong Ji. 2016.  Joint models for extracting adverse drug events from biomedical text.  In _IJCAI_ , volume 2016, pages 2838–2844.
  * Li and Ji (2014) Qi Li and Heng Ji. 2014.  Incremental joint extraction of entity mentions and relations.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 402–412, Baltimore, Maryland. Association for Computational Linguistics.
  * Li et al. (2019) Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019.  Entity-relation extraction as multi-turn question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1340–1350, Florence, Italy. Association for Computational Linguistics.
  * Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020.  A joint neural model for information extraction with global features.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7999–8009, Online. Association for Computational Linguistics.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3219–3232, Brussels, Belgium. Association for Computational Linguistics.
  * Luan et al. (2019) Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019.  A general framework for information extraction using dynamic span graphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3036–3046, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016.  End-to-end relation extraction using LSTMs on sequences and tree structures.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1105–1116, Berlin, Germany. Association for Computational Linguistics.
  * Miwa and Sasaki (2014) Makoto Miwa and Yutaka Sasaki. 2014.  Modeling joint entity and relation extraction with table representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1858–1869, Doha, Qatar. Association for Computational Linguistics.
  * Nguyen and Verspoor (2019) Dat Quoc Nguyen and Karin Verspoor. 2019.  End-to-end neural relation extraction using deep biaffine attention.  In _European Conference on Information Retrieval_ , pages 729–738. Springer.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Roth and Yih (2004) Dan Roth and Wen-tau Yih. 2004.  A linear programming formulation for global inference in natural language tasks.  In _Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004_ , pages 1–8, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Sun et al. (2018) Changzhi Sun, Yuanbin Wu, Man Lan, Shiliang Sun, Wenting Wang, Kuang-Chih Lee, and Kewen Wu. 2018.  Extracting entities and relations with joint minimum risk training.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2256–2265, Brussels, Belgium. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wadden et al. (2019) David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019.  Entity, relation, and event extraction with contextualized span representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5784–5789, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2018) Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018.  Joint embedding of words and labels for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2321–2331, Melbourne, Australia. Association for Computational Linguistics.
  * Wang and Lu (2020) Jue Wang and Wei Lu. 2020.  Two are better than one: Joint entity and relation extraction with table-sequence encoders.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1706–1721, Online. Association for Computational Linguistics.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, and Mohammad Norouzi. 2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _CoRR_ , abs/1609.08144.
  * Ye et al. (2019) Wei Ye, Bo Li, Rui Xie, Zhonghao Sheng, Long Chen, and Shikun Zhang. 2019.  Exploiting entity BIO tag embeddings and multi-task learning for relation extraction with imbalanced data.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1351–1360, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2017) Meishan Zhang, Yue Zhang, and Guohong Fu. 2017.  End-to-end neural relation extraction with global optimization.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1730–1740, Copenhagen, Denmark. Association for Computational Linguistics.
  * Zhao et al. (2020) Shan Zhao, Minghao Hu, Zhiping Cai, and Fang Liu. 2020.  Modeling dense cross-modal interactions for joint entity-relation extraction.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 4032–4038. International Joint Conferences on Artificial Intelligence Organization.  Main track.