# Deep Reinforcement Learning for Cyber System Defense under
Dynamic Adversarial Uncertainties

Ashutosh Dutta1, Samrat Chatterjee1, Arnab Bhattacharya1, Mahantesh
Halappanavar1

###### Abstract

Development of autonomous cyber system defense strategies and action
recommendations in the real-world is challenging and includes characterizing
system state uncertainties and attack-defense dynamics. We propose a data-
driven deep reinforcement learning (DRL) framework to learn proactive,
context-aware, defense countermeasures that dynamically adapt to evolving
adversarial behaviors while minimizing loss of cyber system operations. A
dynamic defense optimization problem is formulated with multiple protective
postures against different types of adversaries with varying levels of skill
and persistence. A custom simulation environment was developed and experiments
were devised to systematically evaluate the performance of four model-free DRL
algorithms against realistic, multi-stage attack sequences. Our results
suggest the efficacy of DRL algorithms for proactive cyber defense under
multi-stage attack profiles and system uncertainties.

## Introduction

Decision support for cyber system defense in real-world dynamic environments
is a challenging research problem that includes the dynamical characterization
of uncertainties in the system state and the incorporation of dynamics between
attackers and defenders. Cyber defenders typically operate under resource
constraints (e.g., labor, time, cost) and must update their strategies and
tactics dynamically as the system evolves with/without attack influence from
adaptive adversaries. Often, the defender may be unaware of system compromise
and must adopt proactive strategies to maintain mission-critical operations.
The past decade has seen a growing body of literature focused on the
application of game-theoretic approaches for cybersecurity (Roy et al. 2010;
Liang and Xiao 2012). These approaches typically involve resource allocation
optimization with Markov decision process (MDP) models in non-cooperative
settings. Cyber system attack and defense modeling methods also include: 1)
probabilistic approaches for system reliability and attack outcome dependency
(Hu et al. 2017; Chen, Xu, and Shi 2018); 2) Bayesian networks (Frigault et
al. 2008; Shin et al. 2015); and 3) fault/decision trees (Fovino, Masera, and
De Cian 2009). Although attack graph-based methods (Poolsappasit, Dewri, and
Ray 2011) for identifying system vulnerabilities and known exploits are
valuable, within realistic settings, insufficient and imperfect information
about system properties and attack goals are typically available to the
defender.

Recent advances in reinforcement learning (RL) approaches have led to the
development of partially observable stochastic games (POSG) in partial
information settings (MacDermed, Isbell, and Weiss 2011; Sutton and Barto
2018). Also, cyber system state-space modeling has generated interest in
potential use of POSGs for cybersecurity problems (Ramuhalli et al. 2013;
Chatterjee et al. 2016; Tipireddy et al. 2017). However, POSGs tend to be
general formulations and are often intractable problems. In addition, the
state-of-the-art in cyber decision-support also includes the use of partially
observable Markov decision process (POMDP) models, as well as distributed
POMDPs, for solving a variety of problems such as: 1) cyber risk assessment
(Carin, Cybenko, and Hughes 2008), 2) uncertainty in penetration testing
(Sarraute, Buffet, and Hoffmann 2012), 3) network task time allocation
(Caulfield and Fielder 2015), 4) data exfiltration detection (McCarthy et al.
2016), and 5) effective deception resource allocation (Islam et al. 2021).
Most of these problems either focus only on specific attacks (or attack
types), consider a limited defender action space, and assume that a system
model is available. Recent work in deep RL (DRL) methods for cybersecurity
primarily focus either on optimizing network operations or cyber defense
against different threat types (Nguyen and Reddi 2021; Dutta, Al-Shaer, and
Chatterjee 2021). While progress mostly includes focus on model-free DRL
methods, there is a critical need for investigating further the role of
different DRL algorithms for cyber defense trained under diverse adversarial
settings.

This paper focuses on the applicability of DRL in optimizing cybersecurity
defense planning against strategic multi-stage adversaries. Specifically, the
objective of a DRL defense agent is to compute context-aware defense plans by
learning network and multi-stage attack behaviors while minimizing impacts to
benign system operations. Generally, the dynamics of cyber systems depend on
many correlated factors (e.g., traffic volume, network utilization) which may
exhibit uncertain behaviors across time due to unanticipated physical link
failures, sensor errors, and others. On the other hand, a stealthy cyber
adversary may adapt their strategies (i.e., tactics and techniques) based on
current network conditions and ongoing attack impact. For example, multi-stage
attacks can propagate through the deployment of multiple software processes
(e.g., Application Programming Interface (API) calls) to execute attack
actions without inducing suspicion, instead of relying on a single process.
However, a defender may not always be able to block a process due to its role
in maintaining critical operations governed by interdependent processes. Thus,
defense planning needs to consider the aggregated impact of different attack
actions at different stages of its propagation. Figure 1 presents a multi-
stage attack propagation template based on MITRE ATT&CK framework (MITRE 2022)
that was leveraged in our study for evaluating multiple DRL-based defense
mechanisms. Within an attack-defense interaction, an adversary can start from
any technique of the Reconnaissance/Initial Access tactic, and wins if they
reach any technique of Impact/Exfiltration tactic. Based on defense actions,
the attacker may abort (i.e., move to the Attack Terminated state or
defender’s win) or persist to move on to the next stage.

Figure 1: Multi-stage attack propagation represented with MITRE ATT&CK Tactics
and Techniques. (Note: A directed edge between an attack tactic and technique
specifies that the attacker may try to implement that technique next after
achieving the objective of the attack tactic. Bidirectional arrow represents
that Defense Evasion can come before Persistence.)

The main contribution of this paper is to systematically evaluate the
performance of multiple DRL algorithms for cyber defense that were trained
under diverse adversarial uncertainties. The next section briefly describes
DRL algorithms used in this study. Next, we describe the proposed autonomous
cyber-defense framework and custom simulation environment. This is followed by
experimental results and discussion, and some concluding remarks.

## Background

Cyber defense DRL agents compute a policy that recommends optimal action at
the current cyber-network state. At time-sequence $t$, the agent executes an
action $a_{t}$ at current state $s_{t}$ and receives reward $R(s_{t},a_{t})$
that is used to update the policy. One of the objective of DRL agent is to
balance the trade-off between exploration (i.e., executing random actions to
understand consequences) and exploitation (i.e., executing optimal actions
based on previous exploration knowledge). This research uses $\epsilon$-greedy
approaches, where it executes random actions with probability $\epsilon$.
Notably, our $\epsilon$ decays with the passage of time. In this research, we
evaluate four different DRL approaches:

#### Deep Q-Network (DQN)

Deep Q-Network is a model-free DRL approach that maximizes the Bellman
equation:

$\begin{split}Q_{t}(s_{t},a_{t})&=Q_{t}(s_{t},a_{t})+\alpha(R_{t}(s_{t},a_{t})+\\\
&\gamma\max_{a}Q_{t}(s_{t+1},a)-Q_{t}(s_{t},a_{t}))\end{split}$ (1)

where, $Q_{t}(s_{t},a_{t})$ is the approximate Q-value (i.e., expected
accumulated reward), $R_{t}(s_{t},a_{t})$ is the reward ($r_{t}$) at $t$, and
$\alpha$ is the learning rate. At each $t$, the agent executes an action and
stores the $(s_{t},a_{t},r_{t},s_{t+1})$ in a replay buffer. After every $n$
steps, the agent calculates the loss using the Eqn. 1 for random batches from
the buffer. Here, we use a neural network as a universal function approximator
of the Q-function $Q_{t}(s_{t},a_{t})$ (Riedmiller 2005; Sutton and Barto
2018). Consequently, the neural network parameters are updated by minimizing
the loss in Eqn. 2 via stochastic gradient descent. Extensions include the
Double DQN method that predicts $Q_{t}(s_{t+1},a)$ and $Q_{t}(s_{t},a)$ with
two different neural networks for more stable Q-function updates.

$Loss=R_{t}(s_{t},a_{t})+\gamma\max_{a}Q_{t}(s_{t+1},a)-Q_{t}(s_{t},a_{t}).$
(2)

#### Actor-Critic

Actor-critic approaches combine both the policy and value iteration methods
using the following components:

  * •

Critic is responsible for policy evaluation and uses a deep neural network
(DNN) to estimate the Q-value. Based on the loss in Eqn. 2, the critic updates
the parameters of the DNN and sends the computed gradients to the actor.

  * •

Actor recommends the optimal action for the current state of a critic using a
DNN, whose parameters are updated based on gradients received from the critic.
Specifically, the actor searches for the optimal parameters of the DNN,
$\theta^{*}$ (i.e., weights of DNN), that maximize the expected accumulated
reward (Sutton and Barto 2018):

$J(\pi_{\theta})=E_{\pi_{\theta}}[\sum_{t=1}^{T}G(s_{t},a_{t})]$ (3)

where, $T$ is the number of decision epochs, and $G(s_{t},a_{t})$ is the total
accumulated reward. The DNN parameters are updated using the following
equation:

$\theta=\theta+\alpha\nabla J(\theta)$ (4)

where, $\alpha$ is the learning rate. In Eqn. 4, $\nabla J(\theta)$ is the
gradient descent derived as follows (Sutton and Barto 2018):

$\begin{split}\nabla J(\pi_{\theta})&=\nabla
E_{\pi_{\theta}}[\sum_{t=1}^{T}G(s_{t},a_{t})]\\\ &=\nabla
E_{\pi_{\theta}}(\sum_{t=1}^{T}\nabla_{\theta}log\pi_{\theta}(a_{t}|s_{t})G(s_{t},a_{t}))\end{split}$
(5)

where, $\pi(a_{t}|s_{t})$ is the probability of taking action $a_{t}$ in state
$s_{t}$ subject to the current policy parameters $\theta$. We implement three
different variants of actor-critic algorithms described as follows.

    * –

Advantage Actor-Critic Approach (A2C) This method uses an advantage function,
$A(s_{t},a_{t})$, defined as

$A_{(}s_{t},a_{t})=G(s_{t},a_{t})-V(s_{t})$ (6)

to replace the $G(s_{t},a_{t})$ of Eqn. 5. The advantage function reduces the
high variance to make the policy network more stable. Note that $V(s_{t})$ in
Eqn. 6 is the baseline value achieved at $s_{t}$.

    * –

Asynchronous Advantage Actor Critic Approach (A3C): A3C is different from A2C
in that the actor updates the policy parameters asynchronously as soon as any
gradient update from any critic (and does not wait for all critics to finish).

    * –

Proximal Policy Optimization (PPO) This method clips the divergence of new
policy when it is out of the region $[1-e,1+e]$ ($e$ is called clip
parameter), in order to avoid drastic deviation from an older evaluated
policy. This may be beneficial against sensor noises and errors. PPO optimizes
the following clipped surrogate objective function, $L^{CLIP}(\theta)$:

$\begin{split}L_{t}^{CLIP}(\theta)&=\mathbb{E}[\min\\{r_{t}(\theta)A_{t},\\\
&\textrm{clip}\left(\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{old}(a_{t}|s_{t})},1-e,1+e\right)A_{t}\\}]\end{split}$
(7)

where, $\pi_{\theta}$ and $\pi_{old}$ represents new and old stochastic
policies respectively.

## Autonomous Cyber Defense Framework

Figure 2 presents our proposed autonomous cyber defense framework. The core
element of our framework is a custom OpenAI Gym (Brockman et al. 2016)
simulation environment that we developed, where at each time-sequence, a DRL
defense agent executes a Defense Action and observes Attack Position and
Reward as feedback. Each episode (i.e., multiple time-sequences) considers an
attack path (per Figure 1), where the red node in figure 2 represents the
current Attack Position. Next, we describe adversary and defense models in our
framework.

Figure 2: Autonomus cyber defense framework.

#### Adversary Model

The objective of an adversary is to move from the Reconnaissance/Initial
Access tactic to the goal tactic of Impact/Exfiltration sequentially (per Fig.
1). The adversary uses a logic-based model to achieve the objective of a
specific tactic if they successfully execute one of their techniques. For
example, from attack initiation an adversary may reach the
Reconnaissance/Initial Access tactic by successfully executing either Active
Scanning (e.g., scans to find vulnerable machines or services) or Phishing
(e.g., sends malicious links to users). At any time, the attack position is
the last successfully executed attack technique. The attack graph in Fig. 1
satisfies the monotonicity property, which means that an adversary never
discards a specific attack position once they successfully execute it.

There are many possible chains/paths to move from Reconnaissance/Initial
Access position to Impact/Exfiltration, from which, the attacker can choose
any path based on the attack strategy. We assume that an adversary can change
their strategy according to observations of previous attack impacts and
network/system conditions. As a result, they can evade or defeat any static
defense policy by discovering the deployed countermeasures.

To execute a technique successfully, the adversary needs to exploit at least
one of the existing vulnerabilities that are still not discovered or cannot be
patched due to usability, safety, or stability requirements. To do so, the
adversary may follow diverse procedures based on the following logic. An
attack attempt fails if they cannot exploit any relevant vulnerability due to
countermeasures or lack of expertise. From any tactic/technique, the adversary
moves to Attack Terminated position if their multiple attempts to execute next
attack technique fails due to successful defense actions. Hence, there are two
attack consequences: (i) adversary wins if they can successfully execute any
technique under the Impact/Exfiltration position; (ii) adversary loses if they
move to the Attack Terminated position. It is important to note that the
adversary’s strategy is implicitly captured in the description of the system’s
state evolution.

#### Defense Model

The main objective of the defender (DRL agent) is to proactively prevent the
adversary to reach the Impact/Exfiltration tactic phase, while minimizing loss
due to interrupting benign operations. To determine the optimal defense, the
agent needs to infer the current attack position and predict the next attack
action. However, lack of domain-specific information on system complexities
and adversarial behavior results in the defender facing the following
uncertainties that prevents the creation of an apriori system dynamics model:

  * •

Uncertain Next Attack Technique: The defense agent cannot predict the next
attack technique due to two reasons. First, the defense agent has no prior
knowledge of the attack graph in Fig. 1 as it requires domain-specific real-
world attack sequences that are hard to obtain. Second, an adversary does not
always follow the same strategy (attack sequence). For example, after User
Execution, an adversary can either try to Modify Registry or Modify System
Process.

  * •

Uncertain Next Attack Procedure: The defense agent does not know the next
attack procedure (i.e., attack actions required to implement an attack
technique) or its likelihood due to lack of domain data.

  * •

Imperfect and Incomplete Observations: Deployed alert mechanisms may not prove
the current position of the adversary due to two reasons: (1) limited
observability of processes, and (2) uncertain mapping from observations to
attack techniques.

For our DRL experimentation, we assume that an adversary is at an initial
access position at the start of each episode, and the defender leverages alert
systems that monitor API calls to understand the current attack position. As
mentioned before, this alert information is not only incomplete due to not
observing all API calls but also imperfect due to probability of errors in
mapping aggregated API calls to MITRE tactic and technique. Moreover, the
defender only partially knows attack position but does not know which process
is malicious, and therefore, seeks to learn the adversary’s dynamic attack
strategy governed by attack action sequences.

## Defense Optimization Problem

We formulate the cyber defender’s optimization problem using a Sequential
Decision Process (SDP) (Braziunas 2003) construct, where the next attack
technique depends on the current attack position and defense action. Note here
that the adversary’s optimal strategy is not learned in this work; instead, we
train the defense agent against different adversarial strategies corresponding
to distinct attack behaviors. Our SDP model is a tuple with four parameters:
$(S,A,R,\gamma)$, where $S$ is the cyber system state space, $A$ is the
defense action space, $R$ is the reward function, and $\gamma$ is the discount
factor. The agent only knows about $S$, $A$, and $\gamma$ which define its
interaction with the environment. Though the agent is unaware of the
mathematical form of $R$. The objective in the SDP model is to compute an
optimal action, $a^{*}\in A$, for any current state, $s\in S$ that maximizes
the cumulative reward over a finite attack horizon.

At the start of each time $t$, the agent executes a defense action, $a_{t}\in
A$, based on current environment state, $s_{t}\in S$, and receives feedback
from the environment. Such feedback consists of current attack position,
$s_{t+1}\in S$, as observation and defense payoff, $r_{t}$, as a reward. The
payoff of $a_{t}$ depends on its effectiveness on whether $a_{t}$ prevented
the next attack technique or not. To clarify, $a_{t}$ is an effective defense
action at $s_{t}$ if it forces the attacker to stay at the previous position
($s_{t+1}=s_{t}$) or move to Attack Terminated node. Thus, this model also
implicitly integrates the expected attack behavior into decision-model through
learning effectiveness of defense actions at particular environment condition.
Next, we describe the optimization model elements in detail.

#### State Space ($S$)

Our state space, $S$, consists of 17 states, where each state, $s\in S$, is a
sparse vector that represents an unique attack position. Hence, the current
state at any time $t$ specifies the position of adversary at $t$, based on
which, the defense agent aims to choose the optimal defense action for that
time-sequence. We consider three types of states: (1) 15 attack-technique
states corresponding to each attack technique in Fig. 1; (2) Attack Initiated
state, and (3) Attack Terminated state. The initial state is the Attack
Initiated state, and the adversary moves to Attack Terminated state if they
abort the attack due to failure or detection. Moreover, the adversary moves to
a new state by successfully executing the associated attack technique. We
consider two types of goal states:

  * •

Attack Goal State: Adversary wins if they reach (1) Automated Exfiltration,
(2) Data Encryption/Obstruction, or (3) Endpoint Denial of Service (DoS) state
of Impact/Exfiltration tactic;

  * •

Defense Goal State: Defender wins if the adversary moves to Attack Terminated
state.

#### Defense Action Space ($A$)

The defender’s action space, $A$, has three different modes of operation:

  1. 1.

Inactive: The defender remains silent and does nothing;

  2. 2.

Reactive: The defender removes all processes that called or executed actions
related to current attack position or last attack action;

  3. 3.

Proactive: The defender blocks a specific set of API calls or operations to
prevent the next attack action.

The defender remains Inactive for strategic reasoning or to avoid termination
of critical benign processes. Whereas, for both reactive and proactive defense
approaches, benign operations may be interrupted. We consider only one
reactive action assuming that it can remove all processes associated with last
attack action (whether successful or not). For proactive defense, we consider
21 distinct defense actions that block unique sets of API calls and operations
or adopt particular methods/measurements to prevent specific attack
techniques. For instance, a proactive defense action: Restrict Registry
Permission blocks the ability to change certain keys to prevent adversary’s
autostart execution (Persistence tactic) or registry modification (Defense
Evasion tactic). Another proactive action such as Restrict File and Directory
Permission (only certain set of sensitive files) can also stop autostart
execution. However, success likelihood of different defense actions vary in
both defense effectiveness (i.e., preventing attack action) and expected false
positive rate (i.e., terminating benign operations). Hence, the defense agent
must choose the best action considering defense effectiveness and expected
false positive rate. Moreover, this research also assumes that the defender
can execute one action at a time-sequence. Hence, for proactive defense
approach, the defender’s decision-model needs to understand what the adversary
may do next to execute an effective mitigation action. The defender wins if
the adversary moves to Attack Terminated position due to failure to execute
new attack techniques or being detected and removed.

#### Reward Function ($R$)

We consider the following reward function for the defense agent:

$R=-p_{g}(s)\times I_{g}-\mathcal{I}_{v}\times I_{g}-C_{f}$ (8)

where, $\mathcal{I}_{v}=-1$ if the defender wins, and 0 otherwise. Note that
$p_{g}(s)$ is the probability of the adversary reaching the
Impact/Exfiltration tactic from state $s$, and $I_{g}$ is the impact/loss of a
successful attack execution. Equation 8 comprise of three terms: (1)
$p_{g}(s)\times I_{g}$ quantifies the risk at state $s$ due to the probability
($p_{g}(s)$) of attacker’s reaching to goal; (2) $I_{v}\times I_{g}$
quantifies the penalty or incentive to the defense agent when the adversary
wins or loses, respectively; and (3) $C_{f}$ is the cost of executing any
defense action in $A$. The cost $C_{f}$ depends on the aggregated loss due to
interrupting benign operations, and defense implementation or operational
cost. We assume that the defense implementation cost is same for all
mitigation actions and zero cost for Inactive action.

## Experiments

To implement our framework and solve the cyber defense optimization problem,
an experimental plan was established to evaluate the performance of four DRL
approaches (i.e., DQN, A2C, A3C, and PPO). The experimental setup, training
and testing scenarios, adversary types, and simulation environment are
described next.

### Experimental Setup

We designed our experiments using Python 3.7 (Van Rossum and Drake Jr 1995),
and used RLlib library (Liang et al. 2018) for implementing DRL algorithms. We
have simulated our experiments using a Dell Alienware machine with 16-core
3GHz Intel Core i7-5960X processor, 64GB RAM, and three 4GB NVIDIA GM200
graphics cards. Although MITRE ATT&CK framework contains 11 tactics and many
more techniques, we consider 7 tactics and 15 techniques for our experiments.
Our defense action space $A$ consists of 23 mitigation actions, including 21
proactive actions. Table 1 presents learning parameters that we used for all
our experiments.

Parameter Name | Value
---|---
Entropy Coefficient | 0.05
Initial exploration probability | 1.0
Final exploration probability | 0.04
Exploration delay period | 300,000
Number of workers | 4
Rollout fragment length | 12
Batch size | 48
PPO clip value | 0.4
Training epochs | 100
Steps per training epoch | 25000

Table 1: Parameters used for training and testing simulation experiments.
(Note: Number of workers refers to the number of parallel processes, and
Exploration delay period refers to the number of training steps that decays
exploration probability from 1.0 to 0.04.)

### Training and Testing Scenarios

We generated all possible distinct attack paths from initial attack position
(i.e., initial attack state) to any state of the last attack tactic (i.e.,
Impact/Exfiltration tactic). We used 80% attack paths for training and 20%
attack paths for testing, where each attack propagation path is a unique
sequence of attack techniques executed by an adversary to achieve their
objective.

During training, each unique episode contains one attack propagation path,
which ends if the defender/adversary wins or loses. The adversary wins or the
defender loses in an episode if the adversary successfully executes all attack
techniques across the attack path to satisfy Impact/Exfiltration tactic.
However, an adversary loses or defender wins if they move to Attack Terminated
state due to failing at least a number of times, $n$, across a path.

### Adversary Types

For our experiments, we determine adversary types based on two attack
parameters: (1) skill, and (2) persistence. With greater attack skill, success
rate in executing next attack technique is higher due to increased capability
in exploiting vulnerabilities. We define $\rho$ as an attack skill parameter,
representing attack success rate in exploiting a vulnerability. We assume that
all vulnerabilities have same exploitability (i.e., impact and complexity). On
the other hand, greater attack persistence indicates that the adversary does
not abort their objective or cannot be readily detected despite their failed
attempts. We define $\tau$ as an attack persistence parameter, representing
the number of failed attempts before moving to attack terminated state (i.e.,
defense win). By tuning $\rho$ and $\tau$, we consider three different attack
profiles/strategy: (1) Attack profile 1 ($Av_{1}$): $\rho=0.75$ and $\tau=4$,
(2) Attack profile 2 ($Av_{2}$): $\rho=0.85$ and $\tau=5$, and (3) Attack
profile 3 ($Av_{3}$): $\rho=0.95$ and $\tau=7$. For example,
$Av_{2}(\rho=0.85,\tau=5)$ indicates that the adversary with profile 2 has 85%
success likelihood in exploiting a specific vulnerability and tolerates
maximum 5 failed attempts in an episode. Thus, attack profile 3 represents the
most sophisticated adversary, and attack profile 1 represents a naïve
adversary. It is important to note that the adversary may change attack
procedures if their previous attempt in exploiting a specific attack technique
fails.

### Simulation Environment

We developed a custom OpenAI Gym simulation environment (not in the public
domain at this time) for the autonomous cyber defense framework illustrated in
Figure 2. The DRL defense agent has no knowledge of the environment and attack
behavior. The defense agent receives observations about next attack position
and reward as feedback. The agent then determines the current attack position
based on recent observation. We assume that system alerts can be translated to
a specific attack position with 85%, 75%, and 65% accuracy against $Av_{1}$,
$Av_{2}$, and $Av_{3}$, which indicates that the adversary is more stealthy
with increased level of sophistication.

Within the simulation environment, at the start of each episode, an attack
propagation path is selected, and the environment is set to the initial attack
position. At the start of each timestep, a Defense Success module determines
whether the current defense action stopped the current attack technique or
not, based on the correlations among attack procedures and defense actions.
Another module, Benign Process Interrupter, sends the list of benign
processes/operations interrupted due to the recent defense action to the
Reward Generation module. We assume a power-law distribution function to
generate the number of interrupted benign operations, which gradually becomes
lower towards the last attack tactic. If the defense action is successful, the
adversary stays at their current position. Otherwise, if the defense action
fails, Attack Success determines the success of the current attack technique
based on attack skill ($\rho$) of attack profile. If the attack is successful,
the adversary moves to next position; otherwise, they remain at the current
position or move to Attack Terminated state (end of episode). The reward is
generated based on attack position, interrupted benign operations, and others
using Eqn. 8. At the end of the timestep, reward and next observation are sent
to the DRL defense agent that is used to update and refine their policy.

## Evaluation

(a) Convergence for different values of $\gamma$ ($\alpha$ set to 0.005).

(b) Convergence for different values of $\alpha$ (with $\gamma$ set to
optimized value of 0.8).

Figure 3: Sensitivity of A2C for different values of $\gamma$ and $\alpha$
against attack profiles $Av_{1}$, $Av_{2}$, and $Av_{3}$ (left to right).

We assess the performance of different DRL algorithms using the Defense-Win
Ratio (DWR) metric, which evaluates the fraction of episodes where the
defender won in a single batch, where each batch comprises of 200 episodes.
Using the DWR ratio, we analyze (i) the convergence of the DRL algorithms
during training, and (ii) the performance of the trained defense models
against unseen attack sequences during testing. We evaluate all four DRL
algorithms against three attack profiles (as described in Section Adversary
Types).

### Hyperparameter Optimization and Training

For brevity, here we only illustrate the training performance of DQN and A2C
using the DWR metric by varying two hyper-parameters: discount factor
($\gamma$) and learning rate ($\alpha$). In the following sections, each
figure has two rows and three columns; the columns refer to the three attack
profiles, while the rows correspond to the hyperparameters (the first row is
for $\gamma$ and the second row is for $\alpha$). In this research, for all
DRL algorithms, we have used fully connected neural network with 2 hidden
layers where each of them has 256 neurons. We have used tanh as activation
function in all the cases.

#### A2C

Figure 3(a) illustrates the sensitivity of A2C to different values of
$\gamma$, keeping $\alpha$ fixed to 0.005. Against attack profile 1, all A2C
instances converge at DWR value of 0.95 within 40 iterations ($40\times 200$
episodes). Against attack profile 2, the A2C instance with $\gamma=0.8$
achieves the highest DWR ratio, while all other discount factors induce poor
performance. For attack profile 3, $\gamma=0.8$ reaches the highest DWR of 0.8
while requiring 200 iterations. Thus, attack sophistication not only increases
the convergence time but also reduces the defense agent’s success rate.
Averaging across all attack profiles, A2C performs best with $\gamma=0.8$,
which shows that the defense agent must balance the trade-off between the
current reward and possible future payoff. Fig. 3(b) exhibits the sensitivity
of A2C (with optimal $\gamma$ set to 0.8) for different values of $\alpha$. We
observed that the algorithm performance does not change significantly against
attack profile 1. Interestingly, against both attack profiles 2 and 3, A2C
with $\alpha=0.0005$ shows the best performance; this is possibly because the
algorithms get stuck to local minima for larger values of $\alpha$. Therefore,
for A2C we set $\alpha=0.005$ and $\gamma=0.8$ for the test experiments.

#### DQN

(a) Convergence for different values of $\gamma$ ($\alpha$ set to 0.005).

(b) Convergence for different values of $\alpha$ (with $\gamma$ set to
optimized value of 0.8).

Figure 4: Sensitivity of DQN for different values of $\gamma$ and $\alpha$
against attack profiles $Av_{1}$, $Av_{2}$, and $Av_{3}$ (left to right)

We followed the same training approach for DQN as that of A2C. From Fig. 4(a),
we observe that DQN converges to optimal policy within 10 iterations, which is
much faster compared to A2C; in fact, it was better than all of the DRL
algorithms we tested. Moreover, DWR was higher than other DRL methods for each
attack profile, which showed the superior performance of DQN for environments
with discrete states and actions. Moreover, Fig. 4(b) illustrated that
DQNperformance does not change significantly with change in $\alpha$. For
testing, we set $\gamma=0.8$ and $\alpha=0.01$ for DQN.

### Testing Results

Figure 5: DWR performance during testing against the three attack profiles $Av_{1}$, $Av_{2}$, and $Av_{3}$ (from left to right). | DRL
---
Algorithm
| Attack Profile 1
---
(in %)
| Attack Profile 2
---
(in %)
| Attack Profile 3
---
(in %)
|  | Tactic
---
ID: 3
| Tactic
---
ID: 6
| Mean
---
Reward
| Tactic
---
ID: 3
| Tactic
---
ID: 6
| Mean
---
Reward
| Tactic
---
ID: 3
| Tactic
---
ID: 6
| Mean
---
Reward
A2C | 51.3 | 76.9 | 56.7 | 60.5 | 80 | 59.8 | 26.2 | 51.9 | 27.11
PPO | 71.3 | 87.9 | 70.3 | 32.9 | 58.5 | 36.8 | 41.9 | 63.2 | 40
A3C | 68.8 | 89.4 | 72 | 53.5 | 66.7 | 48 | 18.5 | 42.8 | 18.5
DQN | 79.6 | 93.3 | 77 | 82 | 95 | 77.2 | 57.3 | 84.1 | 60.7
Table 2: Testing performance comparison among different DRL algorithms.

We discuss the testing performance of the DRL algorithms against the three
attack profiles with unseen attack sequences not used during training. Fig. 5
illustrates the cumulative result in defending against the adversary at
different phases. Here, each phase corresponds to a distinct adversary tactic.
Tactic ID: 0 specifies the initial attack position before Reconnaissance,
Tactic ID: 6 specifies the Collection, and all other tactics in Fig. 1 are
numbered sequentially from Tactic ID: 1 to Tactic ID: 5. Note that
Impact/Exfiltration (tactic ID 7) is not shown in Fig. 5 as the defender loses
if the attacker reaches that state. Table 2 reports how many attack sequences
were stopped at the corresponding attack tactic. For example, the column
corresponding to Tactic ID: 3 specifies how many attacks were stopped at
Defense Evasion; similarly for the other columns. The Mean Reward column
specifies how much reward is achieved compared to the best reward (i.e., when
stopping all attacks at Tactic ID: 0).

The objective of the defense DRL agent is not only to stop the adversary from
moving to Impact/Exfiltration state but also to stop the attack progression as
early as possible. Against all attack profiles, we observed that DQN exhibited
the best performance in stopping the adversary as soon as possible. As can be
seen in Table 2 and Fig. 5, DQN stops the three attack profiles within the
Defense Evasion (tactic ID 3) in 79.6%, 82%, and 57.3% of the cases
respectively, while achieving corresponding defense success rates of 93.3%,
95%, and 84.1%. Against $Av_{3}$, DQN has a lower success rate than other
attack profiles, as the sophisticated adversary hardly fails in exploiting a
vulnerability. Besides, $Av_{3}$ is persistent and does not give up easily in
spite of failed attempts, which leads to lower DWR values for the defense
agent. Note that other DRL algorithms did not consistently perform well
against all three attack profiles. In fact, the next-best performing algorithm
against $Av_{3}$ had approximately 50% success rate. This is possibly due to
the fact that actor-critic methods typically require more training samples
than DQN to show better test accuracy.

## Conclusion

Application of DRL methods for cyber system defense are promising, especially
under dynamic adversarial uncertainties and limited system state information.
Evaluating multiple DRL algorithms trained under diverse adversarial settings
is an important step toward practical autonomous cyber defense solutions. Our
experiments suggest that model-free DRL algorithms can be effectively trained
under multi-stage attack profiles with different skill and persistence levels,
yielding favorable defense outcomes in contested settings. However, some
practical challenges that need to be addressed further in using model-free DRL
include (Dulac-Arnold, Mankowitz, and Hester 2019): (i) explainability of the
black-box DRL policies, (ii) vulnerability to adversarial noise and data
poisoning, and (iii) convergence for large state-action spaces. Future work
will include developing DRL-based transfer learning approaches within dynamic
environments for distributed multi-agent defense systems.

## Acknowledgments

This research was supported by the U.S. Department of Energy, through the
Office of Advanced Scientific Computing Research’s “Data-Driven Decision
Control for Complex Systems (DnC2S)” project. Part of this research was
supported by the Mathematics for Artificial Reasoning in Science (MARS)
initiative at Pacific Northwest National Laboratory (PNNL) under the
Laboratory Directed Research and Development (LDRD) program. PNNL is
multiprogram national laboratory operated by Battelle for the U.S. Department
of Energy under contract DE-AC05-76RL01830.

## References

  * Braziunas (2003) Braziunas, D. 2003.  POMDP solution methods.  https://www.techfak.uni-bielefeld.de/~skopp/Lehre/STdKI˙SS10/POMDP˙solution.pdf.  (Date last accessed 09-November-2022).
  * Brockman et al. (2016) Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016.  OpenAI Gym.  _arXiv preprint arXiv:1606.01540_.
  * Carin, Cybenko, and Hughes (2008) Carin, L.; Cybenko, G.; and Hughes, J. 2008.  Cybersecurity strategies: The queries methodology.  _Computer_ , 41(8): 20–26.
  * Caulfield and Fielder (2015) Caulfield, T.; and Fielder, A. 2015.  Optimizing time allocation for network defence.  _Journal of Cybersecurity_ , 1(1): 37–51.
  * Chatterjee et al. (2016) Chatterjee, S.; Halappanavar, M.; Tipireddy, R.; and Oster, M. 2016.  Game theory and uncertainty quantification for cyber defense applications.  _SIAM News_ , 49(6).
  * Chen, Xu, and Shi (2018) Chen, D.; Xu, M.; and Shi, W. 2018.  Defending a cyber system with early warning mechanism.  _Reliability Engineering & System Safety_, 169: 224–234.
  * Dulac-Arnold, Mankowitz, and Hester (2019) Dulac-Arnold, G.; Mankowitz, D.; and Hester, T. 2019.  Challenges of Real-World Reinforcement Learning.  In _Proceedings of 36th International Conference on Machine Learning (ICML)_.  arXiv:1904.12901.
  * Dutta, Al-Shaer, and Chatterjee (2021) Dutta, A.; Al-Shaer, E.; and Chatterjee, S. 2021.  Constraints Satisfiability Driven Reinforcement Learning for Autonomous Cyber Defense.  In _Proceedings of 1st International Conference on Autonomous Intelligent Cyber-Defence Agents (AICA)_.  arXiv:2104.08994.
  * Fovino, Masera, and De Cian (2009) Fovino, I. N.; Masera, M.; and De Cian, A. 2009.  Integrating cyber attacks within fault trees.  _Reliability Engineering & System Safety_, 94(9): 1394–1402.
  * Frigault et al. (2008) Frigault, M.; Wang, L.; Singhal, A.; and Jajodia, S. 2008.  Measuring network security using dynamic bayesian network.  In _Proceedings of the 4th ACM workshop on Quality of protection_ , 23–30.
  * Hu et al. (2017) Hu, X.; Xu, M.; Xu, S.; and Zhao, P. 2017.  Multiple cyber attacks against a target with observation errors and dependent outcomes: Characterization and optimization.  _Reliability Engineering & System Safety_, 159: 119–133.
  * Islam et al. (2021) Islam, M. M.; Dutta, A.; Sajid, M. S. I.; Al-Shaer, E.; Wei, J.; and Farhang, S. 2021.  CHIMERA: Autonomous Planning and Orchestration for Malware Deception.  In _2021 IEEE Conference on Communications and Network Security (CNS)_ , 173–181. IEEE.
  * Liang et al. (2018) Liang, E.; Liaw, R.; Nishihara, R.; Moritz, P.; Fox, R.; Goldberg, K.; Gonzalez, J.; Jordan, M.; and Stoica, I. 2018.  RLlib: Abstractions for distributed reinforcement learning.  In _International Conference on Machine Learning_ , 3053–3062. PMLR.
  * Liang and Xiao (2012) Liang, X.; and Xiao, Y. 2012.  Game theory for network security.  _IEEE Communications Surveys & Tutorials_, 15(1): 472–486.
  * MacDermed, Isbell, and Weiss (2011) MacDermed, L.; Isbell, C.; and Weiss, L. 2011.  Markov games of incomplete information for multi-agent reinforcement learning.  In _Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence_.
  * McCarthy et al. (2016) McCarthy, S. M.; Sinha, A.; Tambe, M.; and Manadhata, P. 2016.  Data exfiltration detection and prevention: Virtually distributed pomdps for practically safer networks.  In _International Conference on Decision and Game Theory for Security_ , 39–61. Springer.
  * MITRE (2022) MITRE. 2022.  MITRE ATT&CK.  https://attack.mitre.org/.  (Date last accessed 09-November-2022).
  * Nguyen and Reddi (2021) Nguyen, T. T.; and Reddi, V. J. 2021.  Deep Reinforcement Learning for Cyber Security.  _IEEE Transactions on Neural Networks and Learning Systems_ , 1–17.
  * Poolsappasit, Dewri, and Ray (2011) Poolsappasit, N.; Dewri, R.; and Ray, I. 2011.  Dynamic security risk management using bayesian attack graphs.  _IEEE Transactions on Dependable and Secure Computing_ , 9(1): 61–74.
  * Ramuhalli et al. (2013) Ramuhalli, P.; Halappanavar, M.; Coble, J.; and Dixit, M. 2013.  Towards a theory of autonomous reconstitution of compromised cyber-systems.  In _2013 IEEE International Conference on Technologies for Homeland Security (HST)_ , 577–583. IEEE.
  * Riedmiller (2005) Riedmiller, M. 2005.  Neural fitted Q iteration–First experiences with a data efficient neural reinforcement learning method.  In _Proceedings of the 16th European Conference on Machine Learning (ECML), Porto, Portugal, October 3-7, 2005. Proceedings 16_ , 317–328. Springer.
  * Roy et al. (2010) Roy, S.; Ellis, C.; Shiva, S.; Dasgupta, D.; Shandilya, V.; and Wu, Q. 2010.  A survey of game theory as applied to network security.  In _2010 43rd Hawaii International Conference on System Sciences_ , 1–10. IEEE.
  * Sarraute, Buffet, and Hoffmann (2012) Sarraute, C.; Buffet, O.; and Hoffmann, J. 2012.  POMDPs make better hackers: Accounting for uncertainty in penetration testing.  In _Twenty-Sixth AAAI Conference on Artificial Intelligence_.
  * Shin et al. (2015) Shin, J.; Son, H.; Heo, G.; et al. 2015.  Development of a cyber security risk model using Bayesian networks.  _Reliability Engineering & System Safety_, 134: 208–217.
  * Sutton and Barto (2018) Sutton, R. S.; and Barto, A. G. 2018.  _Reinforcement Learning: An Introduction_.  MIT press.
  * Tipireddy et al. (2017) Tipireddy, R.; Chatterjee, S.; Paulson, P.; Oster, M.; and Halappanavar, M. 2017\.  Agent-centric approach for cybersecurity decision-support with partial observability.  In _2017 IEEE International Symposium on Technologies for Homeland Security (HST)_ , 1–6. IEEE.
  * Van Rossum and Drake Jr (1995) Van Rossum, G.; and Drake Jr, F. L. 1995.  _Python reference manual_.  Centrum voor Wiskunde en Informatica Amsterdam.


# CD2CR: Co-reference Resolution Across Documents and Domains

James Ravenscroft Centre for Scientific Computing, University of Warwick, CV4
7AL, United Kingdom  Alan Turing Institute, 96 Euston Rd, London, NW1 2DB,
United Kingdom  Filament AI, 1 King William St, London, EC4N 7BJ, United
Kingdom  Arie Cattan Computer Science Department, Bar-Ilan University, Ramat-
Gan, Israel  Amanda Clare Department of Computer Science, Aberystwyth
University, SY23 3DB, United Kingdom  Ido Dagan Computer Science Department,
Bar-Ilan University, Ramat-Gan, Israel  Maria Liakata Centre for Scientific
Computing, University of Warwick, CV4 7AL, United Kingdom  Alan Turing
Institute, 96 Euston Rd, London, NW1 2DB, United Kingdom

###### Abstract

Cross-document co-reference resolution (CDCR) is the task of identifying and
linking mentions to entities and concepts across many text documents. Current
state-of-the-art models for this task assume that all documents are of the
same type (e.g. news articles) or fall under the same theme. However, it is
also desirable to perform CDCR across different domains (type or theme). A
particular use case we focus on in this paper is the resolution of entities
mentioned across scientific work and newspaper articles that discuss them.
Identifying the same entities and corresponding concepts in both scientific
articles and news can help scientists understand how their work is represented
in mainstream media. We propose a new task and English language dataset for
cross-document cross-domain co-reference resolution (CD2CR). The task aims to
identify links between entities across heterogeneous document types. We show
that in this cross-domain, cross-document setting, existing CDCR models do not
perform well and we provide a baseline model that outperforms current state-
of-the-art CDCR models on CD2CR. Our data set, annotation tool and guidelines
as well as our model for cross-document cross-domain co-reference are all
supplied as open access open source resources.

##  1 Introduction

Cross-document co-reference resolution (CDCR) is the task of recognising when
multiple documents mention and refer to the same real-world entity or concept.
CDCR is a useful NLP process that has many downstream applications. For
example, CDCR carried out on separate news articles that refer to the same
politician can facilitate inter-document sentence alignment required for
stance detection and natural language inference models. Furthermore, CDCR can
improve information retrieval and multi-document summarisation by grouping
documents based on the entities that are mentioned within them.

Recent CDCR work Dutta and Weikum (2015); Barhom et al. (2019); Cattan et al.
(2020) has primarily focused on resolution of entity mentions across news
articles. Despite differences in tone and political alignment, most news
articles are relatively similar in terms of grammatical and lexical structure.
Work based on modern transformer networks such as BERT Devlin et al. (2019)
and ElMo Peters et al. (2018) have been pre-trained on large news corpora and
are therefore well suited to news-based CDCR Barhom et al. (2019).

However, there are cases where CDCR across documents from different domains
(i.e. that differ much more significantly in style, vocabulary and structure)
is useful. One such example is the task of resolving references to concepts
across scientific papers and related news articles. This can help scientists
understand how their work is being presented to the public by mainstream media
or facilitate fact checking of journalists’ work Wadden et al. (2020). A
chatbot or recommender that is able to resolve references to current affairs
in both news articles and user input could be more effective at suggesting
topics that interest the user. Finally, it may be helpful for e-commerce
companies to know when product reviews gathered from third party websites
refer to one of their own listings. The work we present here focuses on the
first cross-document, cross-domain co-reference-resolution (CD2CR) use case,
namely co-reference resolution between news articles and scientific papers.

The objective of CD2CR is to identify co-referring entities from documents
belonging to different domains. In this case co-reference resolution is made
more challenging by the differences in language use (lexical but also
syntactic) across the different domains. Specifically, authors of scientific
papers aim to communicate novel scientific work in an accurate and unambiguous
way by using precise scientific terminology. Whilst scientific journalists
also aim to accurately communicate novel scientific work, their work is
primarily funded by newspaper sales and thus they also aim to captivate as
large an audience as possible. Therefore journalists tend to use simplified
vocabulary and structure, creative and unexpected writing style, slang,
simile, metaphor and exaggeration to make their work accessible, informative
and entertaining in order to maximise readership Louis and Nenkova (2013).

Success at the CD2CR task in this setting is dependent on context sensitive
understanding of how the accessible but imprecise writing of journalists maps
on to precise terminology used in scientific writing. For example, a recent
study has found that “convalescent plasma derived from donors who have
recovered from COVID-19 can be used to treat patients sick with the disease”
111DOI: 10.1101/2020.03.16.20036145. A news
article222https://tinyurl.com/ycnq9xg7 discussing this work says that “…blood
from recovered Covid-19 patients in the hope that transfusions…[can help to
treat severely ill patients]” . In this example the task is to link ‘blood’ to
‘convalescent plasma’ and ‘recovered Covid-19 patients’ to ‘donors’. These
cross-document, cross-domain co-reference chains can be used as contextual
anchors for downstream analysis of the two document settings via tasks such as
natural language inference, stance detection and frame analysis.

The contributions in this paper are the following:

  * •

A novel task setting for CDCR that is more challenging than those that already
exist due to linguistic variation between different domains and document types
(we call this CD2CR).

  * •

An open source English language CD2CR dataset with 7602 co-reference pair
annotations over 528 documents and detailed 11 page annotation guidelines
(section 3.1).

  * •

A novel annotation tool to support ongoing data collection and annotation for
CD2CR including a novel sampling mechanism for calculating inter-annotator
agreement (Section 3.4).

  * •

A series of experiments on our dataset using different baseline models and an
in-depth capability-based evaluation of the best-performing baseline (Section
5)

##  2 Related Work

###  2.1 Co-reference Resolution

Intra-document co-reference resolution is a well understood task with mature
training data sets Weischedel et al. (2013) and academic tasks Recasens et al.
(2010). The current state of the art model by Joshi et al. (2020) is based on
Lee et al. (2017, 2018) and uses a modern BERT-based Devlin et al. (2019)
architecture. Comparatively, CDCR, which involves co-reference resolution
across multiple documents, has received less attention in recent years Bagga
and Baldwin (1998); Rao et al. (2010); Dutta and Weikum (2015); Barhom et al.
(2019). Cattan et al. (2020) jointly learns both entity and event co-reference
tasks, achieving current state of the art performance for CDCR, and as such
provides a strong baseline for experiments in CD2CR. Both Cattan et al. (2020)
and Barhom et al. (2019) models are trained and evaluated using the ECB+
corpus Cybulska and Vossen (2014) which contains news articles annotated with
both entity and event mentions.

###  2.2 Entity Linking

Entity Linking (EL) focuses on alignment of mentions in documents to resources
in an external knowledge resource Ji et al. (2010) such as SNOMED
CT333https://tinyurl.com/yy7g4ttz or DBPedia444https://wiki.dbpedia.org/. EL
is challenging due to the large number of pairwise comparisons between
document mentions and knowledge resource entities that may need to be carried
out. Raiman and Raiman (2018) provide state of the art performance by building
on Ling et al. (2015)’s work in which an entity type system is used to limit
the number of required pairwise comparisons to related types. Yin et al.
(2019) achieved comparable results using a graph-traversal method to similarly
constrain the problem space to candidates within a similar graph
neighbourhood. EL can be considered a narrow sub-task of CDCR since it cannot
resolve novel and rare entities or pronouns Shen et al. (2015). Moreover EL’s
dependency on expensive-to-maintain external knowledge graphs is also
problematic when limited human expertise is available. Given these
limitations, EL is inappropriate within our task setting, hence our CDCR-based
approach.

###  2.3 Semantic Specialisation

Like earlier static vector language models, contextual language models such as
BERT Devlin et al. (2019) and ElMo Peters et al. (2018) use distributional
knowledge Harris (1954) inherent in large text corpora to learn context-aware
word embeddings that can be used for downstream NLP tasks. However, these
models do not learn about formal lexical constraints, often conflating
different types of semantic relatedness Ponti et al. (2018); Lauscher et al.
(2020). This is a weakness of all distributional language models that is
particularly problematic in the context of CD2CR for entity mentions that are
related but not co-referent (e.g. “Mars” and “Jupiter”) as shown in section 5.

A number of solutions have been proposed for adding lexical knowledge to
static word embeddings Yu and Dredze (2014); Wieting et al. (2015); Ponti et
al. (2018) but contextual language models have received comparatively less
attention. Lauscher et al Lauscher et al. (2020) propose adding a lexical
relation classification step to BERT’s language model pre-training phase to
allow the model to integrate both lexical and distributional knowledge. Their
model, LIBERT, has been shown to facilitate statistically-significant
performance boosts on a variety of downstream NLP tasks.

##  3 Dataset creation

Our dataset is composed of pairs of news articles and scientific papers
gathered automatically (Section 3.1). Our annotation process begins by
obtaining summaries of the news and science document pairs (extractive news
summaries and scientific abstracts, respectively) (Section 3.2). Candidate co-
reference pairs from each summary-abstract pair are identified and scored
automatically. (Section 3.3). Candidate co-reference pairs are then presented
to human annotators via a bespoke annotation interface for scoring (Section
3.4). Annotation quality is measured on an ongoing basis as new candidates are
added to the system (Section 3.5).

###  3.1 Data Collection

We have developed a novel data set that allows us to train and evaluate a
CD2CR model. The corpus is approximately 50% the size of the ECB+ corpus (918
documents) Cybulska and Vossen (2014) and is split into training, development
and test sets (statistics for each subset are provided in Table 1). Each pair
of documents consists of a scientific paper and a newspaper article that
discusses the scientific work. In order to detect pairs of documents, we
follow the approach of Ravenscroft et al. (2018), using approximate matching
of author name and affiliation metadata, date of publishing and exact DOI
matching where available to connect news articles to scientific publications.

Subset | Documents | Mentions | Clusters
---|---|---|---
Train | 300 | 4,604 | 426
Dev | 142 | 1,821 | 199
Test | 86 | 1,177 | 101
Table 1: Total individual documents, mentions, co-reference clusters of each
subset excluding singletons.

We built a web scraper that scans for new articles from the ‘Science’ and
‘Technology’ sections of 3 well-known online news outlets (BBC, The Guardian,
New York Times) and press releases from Eurekalert, a widely popular
scientific press release aggregator. Once a newspaper article and related
scientific paper are detected, the full text from the news article and the
scientific paper abstract and metadata are stored. Where available the full
scientific paper content is also collected. We ran the scraper between April
and June 2020 collecting news articles and scientific papers including
preprints discussing a range of topics such as astronomy, computer science and
biology (incl. coverage of COVID-19). New relevant content is downloaded and
ingested into our annotation tool (see Section 3.4) on an ongoing basis as it
becomes available.

###  3.2 Article Summarisation

Newspaper articles and scientific papers are long and often complex documents,
usually spanning multiple pages, particularly the latter. Moreover the two
document types differ significantly in length. Comparing documents of such
uneven length is a difficult task for human annotators. We also assume that
asking human annotators to read the documents in their entirety to identify
co-references would be particularly hard with a very low chance for good
inter-annotator agreement (IAA). We therefore decided to simplify the task by
asking annotators to compare summaries of the newspaper article (5-10
sentences long) and the scientific paper (abstract).

For each document pair, we ask the annotators to identify co-referent mentions
between the scientific paper abstract and a summary of the news article that
is of similar length (e.g. 5-10 sentences). Scientific paper abstracts act as
a natural summary of a scientific work and have been used as a strong baseline
or even a gold-standard in scientific summarisation tasks Liakata et al.
(2013). Furthermore, abstracts are almost always available rather than behind
paywalls like full text articles. For news summarisation, we used a state-of-
the-art extractive model Grenander et al. (2019) to extract sentences forming
a summary of the original text. This model provides a summary de-biasing
mechanism preventing it from focusing on specific parts of the full article,
preserving the summary’s informational authenticity as much as possible.

The difference in style between the two documents is preserved by both types
of summary since abstracts are written in the same scientific style as full
papers and the extractive summaries use verbatim excerpts of the original news
articles.

###  3.3 Generation of pairs for annotation

Figure 1: Illustration of the generation process for pairs of potentially co-
referring expressions, left boxes represent related news summary (top) and
abstract (bottom), co-referent entity pairs in middle boxes shown with same
formatting (underline,italic).

To populate our annotation tool, we generate pairs of candidate cross-document
mentions to be evaluated by the user. Candidate mentions are identified by
using spaCy Honnibal and Montani (2017) for the recognition of noun phrases
and named entities from each input document pair (abstract-news summary). For
each pair of documents, pairs of all possible mention combinations are
generated and stored for annotation.

In any given pair of documents, the majority of mention pairs ($M_{0}$,
$M_{1}$) generated automatically in this way will not co-refer thus resulting
in a vastly imbalanced dataset and also running the risk of demotivating
annotators. To ensure that annotators are exposed to both positive and
negative examples, we use a similarity score to rank examples based on how
likely they are to co-refer. The first step in generating a similarity score
$s$ is to concatenate each abstract-news-summary pair together: “summary [SEP]
abstract” into a pre-trained $\text{BERT}_{\text{large}}$ model. Then we take
the mean of the word vectors that correspond to the mention spans within the
documents and calculate the cosine similarity of these vectors. We find that
this BERT-based similarity score performs well in practice. We also use it in
combination with a thresholding policy as one of our baseline models in
Section 4.

###  3.4 Annotation Tool & Interface

We developed an open source annotation
tool555https://github.com/ravenscroftj/cdcrtool that allows humans to identify
cross-document co-reference between each pair of related documents. Whilst
designing this tool, we made a number of decisions to simplify the task and
provide clear instructions for the human annotators in order to encourage
consistent annotation behaviour.

To maximise the quality and consistency of annotations in our corpus, we
simplified the task as much as possible for the end user. Annotation tasks
were framed as a single yes or no question: “Are x and y mentions of the same
entity?”. Mentions in context were shown in bold font whereas mentions already
flagged as co-referent were shown in green. This enabled annotators to
understand the implications for existing co-reference chains before responding
(see Figure 3). Questions were generated and ranked via our task generation
pipeline (see Section 3.3 above).

We added two additional features to our annotation interface to improve
annotators’ experience and to speed up the annotation process. Firstly, if the
candidate pair is marked as co-referent, the user is allowed to add more
mentions to the coreference cluster at once. Secondly, inspired by Li et al.
(2020), if the automatically shown mention pair is not co-referent, the user
can select a different mention that is co-referent.

The upstream automated mention detection mechanism can sometimes introduce
incomplete or erroneous mentions, leading to comparisons that don’t make sense
or that are particularly difficult. Therefore, annotators can also move or
resize the mention spans they are annotating.

We use string offsets of mention span pairs to tokens to check that they do
not overlap with each other in order to prevent the creation of duplicates.
Figure 1 shows an illustrated example of the generation pipeline for mention
pairs.

###  3.5 Annotation Protocol

We recruited three university-educated human annotators and provided them with
detailed annotation guidelines for the resolution of yes/no questions on
potentially co-referring entities in pairs from the ordered queue described
above. By default each entity pair resolution is carried out once, allowing us
to quickly expand our data set. However, we pseudo-randomly sample 5% of
mention pairs in order to calculate inter-annotator-agreement (IAA) and make
sure that data collected from the tool is consistent and suitable for
modelling. New entity pairs for IAA are continually sampled as new document
pairs and mention tuples are added to the corpus by the web scraper (Section
3.1). The annotation system puts mention pairs flagged for IAA first in the
annotation queue. Thus, all annotators are required to complete IAA
comparisons before moving on to novel mention pairs. This allows us to ensure
that all annotators are well represented in the IAA exercise. To avoid
annotators being faced with a huge backlog of IAA comparisons before being
able to proceed with novel annotations, we also limited the number of
comparisons for IAA required by each user to a maximum of 150 per week.

###  3.6 Task Difficulty and Annotator Agreement

We anticipated that annotation of the CD2CR corpus would be difficult in
nature due to its dependencies on context and lexical style. We invited users
to provide feedback regularly to help us refine and clarify our guidelines and
annotation tool in an iterative fashion. Users could alert us to examples they
found challenging by flagging them as difficult in the tool. Qualitative
analysis of the subset of ‘difficult’ cases showed that the resolution of
mention pairs is often perceived by annotators as difficult when:

  * •

Deep subject-matter-expertise is required to understand the mentions, e.g. is
“jasmonic acid” the same as “regulator cis ‐(+)‐12‐oxophytodienoic acid”.

  * •

Mentions involve non-commutable set membership ambiguity e.g. “Diplodocidae”
and “the dinosaurs”

  * •

Mentions are context dependent e.g. “the struggling insect” and “the monarch
butterfly”.

This feedback prompted the introduction of highlighting for existing co-
reference chains in the user interface (as described in section 3.4 above) to
make it easier to tell when non-commutable set membership would likely
introduce inconsistencies into the dataset. For mention pairs requiring
subject-matter-expertise, annotators were encouraged to research the terms
online. For context sensitive mention pairs, annotators were encouraged to
read the full news article and full scientific paper in order to make a
decision.

In our 11 page annotation guidelines document (appendix) we describe the use
of our annotation tool and illustrate some challenging CD2CR tasks and
resolution strategies. For example precise entities mentioned in the
scientific document may be referenced using ambiguous exophoric mentions in
the news article (e.g. ‘a mountain breed of sheep’ vs ‘eight ovis aries’). Our
guidelines require resolving these cases based on the journalist’s intent
(e.g. ‘a mountain breed’ refers to the ‘ovis aries’ sheep involved in the
experiment).

We evaluated the final pairwise agreement between annotators using Cohen’s
Kappa Cohen (1960) ($\kappa_{\text{cohen}}$) and an aggregate ‘n-way’
agreement score using Fleiss’ Kappa Fleiss (1971) ($\kappa_{\text{fleiss}}$).
Pairwise $\kappa_{\text{cohen}}$ is shown in table 2 along with the total
number of tasks each annotator completed. Annotator 3 (A3) shows the most
consistent agreement with the other two annotators. Our Fleiss’ Kappa analysis
of tasks common across the three annotators gave
$\kappa_{\text{fleiss}}=0.554$. We note that Fleiss’ Kappa is a relatively
harsh metric and values, like ours, between 0.41 and 0.60 are considered to
demonstrate ’moderate agreement’Landis and Koch (1977). We also carried out
Fleiss’ Kappa analysis on the subset of mention pairs that were completed by
all annotators and were also marked as difficult by at least one user (180
mention pairs in total). We found that for this subset of pairs,
$\kappa_{\text{fleiss}}=0.399$ which is considered to be fair agreementLandis
and Koch (1977).

| # Annotations | A1 | A2 | A3
---|---|---|---|---
A1 | 10,685 | - | 0.492 | 0.600
A2 | 3,051 | 0.492 | - | 0.500
A3 | 9,847 | 0.600 | 0.500 | -
Table 2: Number of Annotations and Pairwise Cohen’s Kappa scores
$\kappa_{\text{cohen}}$ between annotators.

##  4 Model

Below we describe several baseline models including state of the art CDCR
models that we used to evaluate how well current approaches can be used in our
CD2CR task setting.

###  4.1 BERT Cosine Similarity (BCOS) Baseline

In this model we calculate the cosine-similarity between embeddings of the two
mentions in context ($M_{0},M_{1}$) encoded using a pre-trained BERT model as
discussed above in section 3.3. We define a thresholding function $f$ to
decide if $M_{0}$ and $M_{1}$ are co-referent ($f(x)=1$) or not ($f(x)=0$):

$f(x)=\begin{dcases}1,&\text{if }\text{COSSIM}(M_{0},M_{1})\geq t\\\
0,&\text{otherwise}\end{dcases}$

During inference, we pass this function over all pairs $M_{0},M_{1}$ and infer
missing links such that if $f(A,B)=1$ and $f(B,C)=1$ then $f(A,C)=1$.

Based on Figure 2, we test values in increments of 0.01 between 0.3 and 0.8
inclusive for threshold cut off $t$. We evaluated the baseline by measuring
its accuracy at predicting co-reference in each mention pair in the $CD^{2}CR$
development set. The best performance was attained when $t=0.65$. A
visualisation of the BERT Cosine Similarity distributions of co-referent and
non co-referent annotated mention pairs can be seen in Figure 2.

Figure 2: BERT Cosine Similarity frequency distribution for co-referent (Yes)
and non-co-referent (No) mention pairs in the CD2CR corpus.

Co-referent mention pairs tend to have a slightly higher BERT cosine
similarity than non co-referent mention pairs but there is significant overlap
of the two distributions suggesting that in many cases BERT similarity is too
simplistic a measure.

Figure 3: An example of a cross-document co-reference task presented within
our annotation tool.

###  4.2 Entities Only Baseline (CA)

We use a state-of-the-art model Cattan et al. (2020) (CA) for cross-document
co-reference resolution. In this model, each document is separately encoded
using a RoBERTa encoder (without fine-tuning) to get contextualized
representations for each token. Then, similarly to the within-document co-
reference model by Lee et al. (2017), the mention spans are represented by the
concatenation of four vectors: the vectors of the first and last token in the
span, an attention-weighted sum of the span token vectors, and a feature
vector to encode the span width. Two mention representations are then
concatenated and fed to a feed-forward network to learn a likelihood score for
whether two mentions co-refer. At inference time, agglomerative clustering is
used on the pairwise scores to form coreference clusters.

The CA model is trained to perform both event and entity recognition on the
ECB+ corpus Cybulska and Vossen (2014) In our setting there is no event
detection subtask so, for fair comparison, we pre-train the CA model on ECB+
entity annotations only and evaluate it on our new CD2CR task to see how well
it generalises to our task setting.

###  4.3 CA + Fine-Tuned (CA-FT) Baseline

Here we aim to evaluate whether fine tuning the CA model from section 4.2
using the CD2CR corpus can improve its performance in the new task setting.
The CA model is first trained on the ECB+ corpus in the manner described
above. We then further fine-tune the feed-forward model (without affecting the
RoBERTa encoder) on the CD2CR corpus for 10 epochs with early stopping.
Pseudo-random sub-sampling is carried out on the training set to ensure a
balance of co-referent and non-co-referent mention pairs.

###  4.4 CA - Vanilla (CA-V) Baseline

Here we aim to evaluate whether training the CA model on the CD2CR dataset
from the RoBERTa baseline without first training on the ECB+ corpus allows it
to fit well to the new task setting. We re-initialise the CA encoder (Section
4.2) using weights from RoBERTa Liu et al. (2019) and randomly initialise the
remaining model parameters. We then train the model on the CD2CR corpus for up
to 20 epochs with early stopping with pseudo-random sub-sampling as above.

###  4.5 CA - SciBERT (CA-S) Baseline

This model is the same as CA-V but we replace the RoBERTa encoder with SciBERT
Beltagy et al. (2019), a version of BERT pre-trained on scientific literature
in order to test whether the scientific terms and context captured by SciBERT
improve performance at the CD2CR task compared to RoBERTa. Similarly to CA-V
in section 4.4, we initialise the BERT model with weights from
$\text{SciBERT}_{\text{scivocab-uncased}}$ Beltagy et al. (2019) and randomly
initialise the remaining model parameters, training on the CD2CR corpus for up
to 20 epochs with early stopping.

##  5 Results and Discussion

Model | MUC | $\text{B}^{3}$
---|---|---
P | R |  F1 |  P |  R |  F1
BCOS |  0.42 | 0.94 |  0.58 |  0.01 |  0.45 |  0.00
CA |  0.41 | 0.51 |  0.46 |  0.39 |  0.33 |  0.35
CA-V |  0.50 | 0.69 |  0.58 |  0.35 |  0.57 |  0.44
CA-FT |  0.47 | 0.71 |  0.52 |  0.30 |  0.62 |  0.41
CA-S |  0.58 | 0.46 |  0.51 |  0.32 |  0.53 |  0.39
Table 3: MUC and $B^{3}$ results from running baseline models on CD2CR test
subset, BCOS threshold=0.65

We evaluate each of the model baselines described in section 4 above on the
test subset of our CD2CR corpus. Results are shown in table 3.

For the purposes of evaluation, we use named entity spans from the manually
annotated CD2CR as the “gold standard” in all experiments rather than using
the end-to-end Named Entity Recognition capabilities provided by some of the
models. We evaluate the models using the metrics described by Vilain et al.
(1995) (henceforth MUC) and Bagga and Baldwin (1998) (henceforth $B^{3}$). MUC
F1, precision and recall are defined in terms of pairwise co-reference
relationships between each mention. $B^{3}$, F1, precision and recall are
defined in terms of presence or absence of specific entities in the cluster.
When measuring $B^{3}$, we remove entities with no co-references (singletons)
from the evaluation to avoid inflation of results Cattan et al. (2020).

Test Type |  Co-referent? |  Pass Rate & Total Tests |  Example test case and outcome for test case
---|---|---|---
Anaphora and Exophora resolution |  Yes |  47.1% (16/34) |  M1: …to boost the struggling insect’s numbers… [PASS] M2: the annual migration of the monarch butterfly…
No |  76.5% (26/34) |  M1: …monarchs raised in captivity… [FAIL] M2: …rearing wild-caught monarchs in an indoor environment…
Subset relationship resolution |  Yes |  24.3% (9/37) |  M1: …it was in fact a hive of human activity… [FAIL] M2: …this region for Pre-Columbian cultural developments…
No |  60.0% (18/30) |  M1: … the carnivore’s skull… [FAIL] M2: … the gigantic extinct Agriotherium africanum
Para-phrase resolution |  Yes |  33.3% (13/39) |  M1: …a giant short-faced bear… [PASS] M2: …the gigantic extinct Agriotherium africanum…
No |  80.5% (29/36) |  M1: …half the energy that existing techniques require [FAIL] M2: …the lack of efficient catalysts for ammonia synthesis
Table 4: A breakdown of specific tests carried out on CA-V model against three
challenging types of relationships found in the $CD^{2}CR$ corpus. [PASS] or
[FAIL] indicates CA-V model correctness. Pass Rate is mathematically
equivalent to Recall for test sets.

The threshold baseline (BCOS) gives the highest MUC recall but also poor MUC
precision and poorest $B^{3}$ precision. The $B^{3}$ metric is highly specific
with respect to false-positive entity mentions and strongly penalises BCOS for
linking all non-coreferent pairs with $COSSIM(M_{0},M_{1})\geq 0.65$.
Furthermore, Fig. 2 shows that a thresholding strategy is clearly sub-optimal
given that there is a significant overlap of co-referent and non-co-referent
pairs with only a small minority of pairs at the top and bottom of the
distribution that do not overlap. Therefore, despite its promising MUC F1
score, it is clear that BCOS is not useful in practical terms.

Whilst our thresholding baseline above uses BERT, RoBERTa is used by Cattan et
al. (2020) as the basis for their state-of-the-art model and thus for our
models based on their work. Although the two models have the same
architecture, RoBERTa has been shown to outperform BERT at a range of tasks
Liu et al. (2019). However, as shown in Figure 4, the cosine similarity
distribution of mention pair embeddings produced by RoBERTa is compressed to
use a smaller area of the potential distribution space compared to that of
BERT (Figure 2). This compression of similarities may imply a reduction in
RoBERTa’s ability to discriminate in our task setting. Liu et al. (2019)
explain that their byte-pair-encoding (BPE) mechanism, which expands RoBERTa’s
sub-word vocabulary and simplifies pre-processing, can reduce model
performance for some tasks, although this is not further explored in their
work. We leave further exploration of RoBERTa’s BPE scheme and its effects on
the CD2CR task setting to future work.

Figure 4: RoBERTa Cosine Similarity frequency distribution for co-referent
(Yes) and non-co-referent (No) mention pairs in the CD2CR corpus. Distribution
is compressed between 0.8 and 1.0.

All of the models specifically trained on the CD2CR corpus (CA-V, CA-FT, CA-S)
outperform the CA model by a large margin. Furthermore, the CA-V model
(without pre-training on ECB+ corpus) outperforms the CA-FT model (with ECB+
pre-training) by 6% MUC and 3% B3. These results suggest that the CD2CR task
setting is distinct from the CDCR and ECB+ task setting and that this
distinction is not solvable with fine-tuning.

In terms of both MUC and B3, CA-S performs much worse than CA-V suggesting
that SciBERT embeddings are less effective than RoBERTa embeddings in this
task setting. We hypothesise that SciBERT’s specialisation towards scientific
embeddings may come at the cost of significantly worse news summary embeddings
when compared to those produced by RoBERTa.

We next evaluate our best performing CD2CR baseline model (CA-V) at the entity
resolution CDCR task using the ECB+ test corpus, to see how well it
generalises to the original CDCR task. Results are presented in 5 along-side
Cattan et al’s original model results (CA). The CA-V model still shows good
performance, despite a small drop, when compared to the original CA model. The
drop in $B^{3}$ F1 is more pronounced than MUC but is still broadly in line
with other contemporary CDCR systems Cattan et al. (2020). The CA-V model
demonstrates a promising ability to generalise beyond our corpus to other
tasks and reveals an interesting correspondence between CDCR and CD2CR
settings.

Model | MUC | $B^{3}$
---|---|---
P | R |  F1 |  P |  R |  F1
CA |  0.86 | 0.82 |  0.84 |  0.63 |  0.68 |  0.65
CA-V |  0.82 | 0.81 |  0.81 |  0.56 |  0.53 |  0.55
Table 5: MUC and $B^{3}$ results from running the CD2CR baseline model (CA-V)
on ECB+ dataset compared with original Cattan et al. (2020) (CA).

Finally, the best model (CA-V) is analysed using a series of challenging test
cases inspired by Ribeiro et al Ribeiro et al. (2020). These test cases were
created using 210 manually annotated mention-pairs found in the test subset of
the $CD^{2}CR$ corpus according to the type of relationship illustrated
(Anaphora & Exophora, Subset relationships, paraphrases). We collected a
balanced set of 30-40 examples of both co-referent and non-coreferent-but-
challenging pairs for each type of relationship (exact numbers in Table 4). We
then recorded whether the model correctly predicted co-reference for these
pairs. The results along with illustrative examples of each relationship type
are shown in Table 4. The results suggest that the model is better at
identifying non-co-referent pairs than co-referent pairs and that it struggles
with positive co-referent mentions for all three types of relationship. The
model struggles to relate general reader-friendly descriptions of entities
from news articles to precise and clinical descriptions found in scientific
papers. The model often successfully identifies related concepts such as ‘the
carnivore’s skull’ and ‘Agriotherium africanum’. However it is unable to deal
with the complexity of these relationships and appears to conflate ‘related’
with ‘co-referent’, which is likely due to lack of lexical knowledge as we
discussed in section 2.3. Figure 5 shows significant overlap between co-
referent and non-co-referent RoBERTa-based cosine similarities, which can also
be observed for the wider corpus in Figure 4, but is especially bad for these
test examples. This overlap suggests that disentangling these pairs is likely
to be a challenging task for the downstream classification layer in the CA-V
model. These challenges are less likely to occur in homogeneous corpora like
ECB+ where descriptions and relationships remain consistent in detail and
complexity.

Figure 5: RoBERTa-based mention pair similarity frequency distribution for
test examples from Table 4. ’yes’ and ’no’ for ’co-referent’ and ’not-co-
referent’ respectively

##  6 Conclusion

We have defined cross-document, cross-domain co-reference resolution (CD2CR),
a special and challenging case of cross-document co-reference resolution for
comparing mentions across documents of different types and/or themes. We have
constructed a specialised CD2CR annotated dataset, available, along with our
annotation guidelines and tool, as a free and open resource for future
research. We have shown that state-of-the-art CDCR models do not perform well
on the CD2CR dataset without specific training. Furthermore, even with task-
specific training, models perform modestly and leave room for further research
and improvement. Finally, we show that the understanding of semantic
relatedness offered by current generation transformer-based language models
may not be precise enough to reliably resolve complex linguistic relationships
such as those found in CD2CR as well as other types of co-reference resolution
and relationship extraction tasks. The use of semantic enrichment techniques
(such as those discussed in Section 2.3) to improve model performance in the
CD2CR task should be investigated as future work.

## Acknowledgments

This work was supported by The Alan Turing Institute under the EPSRC grant
EP/N510129/1 and the University of Warwick’s CDT in Urban Science under EPSRC
grant EP/L016400/1.

## References

  * Bagga and Baldwin (1998) Amit Bagga and Breck Baldwin. 1998.  Entity-based cross-document coreferencing using the vector space model.  In _Proceedings of ACL ’98/COLING ’98_ , page 79–85, USA. Association for Computational Linguistics.
  * Barhom et al. (2019) Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019.  Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolution.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4179–4189.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620.
  * Cattan et al. (2020) Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2020.  Streamlining cross-document coreference resolution: Evaluation and modeling.  arXiv:2009.11032.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and Psychological Measurement_ , 20:37–46.
  * Cybulska and Vossen (2014) Agata Cybulska and Piek Vossen. 2014.  Using a sledgehammer to crack a nut? Lexical diversity and event coreference resolution.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 4545–4552.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186. Association for Computational Linguistics.
  * Dutta and Weikum (2015) Sourav Dutta and Gerhard Weikum. 2015.  Cross-Document Co-Reference Resolution using Sample-Based Clustering with Knowledge Enrichment.  _Transactions of the Association for Computational Linguistics_ , 3:15–28.
  * Fleiss (1971) Joseph L. Fleiss. 1971.  Measuring nominal scale agreement among many raters. _Psychological Bulletin_ , 76(5):378–382.
  * Grenander et al. (2019) Matt Grenander, Yue Dong, Jackie Chi Kit Cheung, and Annie Louis. 2019.  Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6019–6024.
  * Harris (1954) Zellig S. Harris. 1954.  Distributional structure.  _WORD_ , 10(2-3):146–162.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Ji et al. (2010) Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffitt, and Joe Ellis. 2010.  Overview of the TAC 2010 knowledge base population track.  In _Proceedings of the 2010 Text Analysis Conference_.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving Pre-training by Representing and Predicting Spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Landis and Koch (1977) JR Landis and GG Koch. 1977.  The measurement of observer agreement for categorical data.  _Biometrics_ , 33(1):159—174.
  * Lauscher et al. (2020) Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2020.  Specializing unsupervised pretraining models for word-level semantic similarity.  _Computing Research Repository_ , arXiv:1909.02339.
  * Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.  End-to-end neural coreference resolution.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics.
  * Lee et al. (2018) Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018.  Higher-order coreference resolution with coarse-to-fine inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics.
  * Li et al. (2020) Belinda Z. Li, Gabriel Stanovsky, and Luke Zettlemoyer. 2020.  Active learning for coreference resolution using discrete annotation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8320–8331.
  * Liakata et al. (2013) Maria Liakata, Simon Dobnik, Shyamasree Saha, Colin Batchelor, and Dietrich Rebholz-Schuhmann. 2013.  A discourse-driven content model for summarising scientific articles evaluated in a complex question answering task.  In _Proceedings of Conference on Empirical Methods in Natural Language Processing, EMNLP 2013_ , pages 747–757. Association for Computational Linguistics.
  * Ling et al. (2015) Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015.  Design challenges for entity linking.  _Transactions of the Association for Computational Linguistics_ , 3:315–328.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _Computing Research Repository_ , arXiv:1907.11692.
  * Louis and Nenkova (2013) Annie Louis and Ani Nenkova. 2013.  A corpus of science journalism for analyzing writing quality.  _Dialogue and Discourse_ , 4(2):87–117.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2227–2237.
  * Ponti et al. (2018) Edoardo Maria Ponti, Ivan Vulić, Goran Glavaš, Nikola Mrkšić, and Anna Korhonen. 2018.  Adversarial propagation and zero-shot cross-lingual transfer of word vector specialization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 282–293.
  * Raiman and Raiman (2018) Jonathan Raiman and Olivier Raiman. 2018.  DeepType: Multilingual entity linking by neural type system evolution.  In _AAAI Conference on Artificial Intelligence_.
  * Rao et al. (2010) Delip Rao, Paul McNamee, and Mark Dredze. 2010.  Streaming cross document entity coreference resolution.  In _Coling 2010: Posters_ , pages 1050–1058, Beijing, China. Coling 2010 Organizing Committee.
  * Ravenscroft et al. (2018) James Ravenscroft, Amanda Clare, and Maria Liakata. 2018.  HarriGT: A tool for linking news to science.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 19–24.
  * Recasens et al. (2010) Marta Recasens, Lluís Màrquez, Emili Sapena, M. Antònia Martí, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010.  SemEval-2010 task 1: Coreference resolution in multiple languages.  In _Proceedings of the 5th International Workshop on Semantic Evaluation_ , pages 1–8. Association for Computational Linguistics.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Shen et al. (2015) W. Shen, J. Wang, and J. Han. 2015.  Entity linking with a knowledge base: Issues, techniques, and solutions.  _IEEE Transactions on Knowledge and Data Engineering_ , 27(2):443–460.
  * Vilain et al. (1995) Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995.  A model-theoretic coreference scoring scheme.  In _MUC6 ’95: Proceedings of the 6th conference on Message understanding_ , pages 45–52.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  _Computing Research Repository_ , arXiv:2004.14974.
  * Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013.  Ontonotes release 5.0 LDC2013T19.  Linguistic Data Consortium, Philadelphia, PA.
  * Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015.  From paraphrase database to compositional paraphrase model and back.  _Transactions of the Association for Computational Linguistics_ , 3:345–358.
  * Yin et al. (2019) Xiaoyao Yin, Yangchen Huang, Bin Zhou, Aiping Li, Long Lan, and Yan Jia. 2019.  Deep entity linking via eliminating semantic ambiguity with BERT.  _IEEE Access_ , 7:169434–169445.
  * Yu and Dredze (2014) Mo Yu and Mark Dredze. 2014.  Improving lexical embeddings with semantic knowledge.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_ , pages 545–550.