# Distributed Kalman filtering with event-triggered communication:
a robust approach

Davide Ghion, Mattia Zorzi D. Ghion is with Serenissima Informatica Spa,
(e-mail: davide.ghion@serinf.it)M. Zorzi is with the Department of Information
Engineering, University of Padova, Padova, Italy; email<EMAIL_ADDRESS>
###### Abstract

We consider the problem of distributed Kalman filtering for sensor networks in
the case there is a limit in data transmission and there is model uncertainty.
More precisely, we propose a distributed filtering strategy with event-
triggered communication in which the state estimators are computed according
to the least favorable model. The latter belongs to a ball (in Kullback-
Leibler topology) about the nominal model. We also present a preliminary
numerical example in order to test the performance of the proposed strategy.

##  I Introduction

Nowadays, sensor networks play an important role in various fields such as
security, monitoring, data analysis and so on. In these applications, the
sensors collect the measurements and from them it is required to estimate the
state of a dynamic system. This task is performed in a distributed fashion and
it can be accomplished in different ways, see [1, 2, 3, 4, 5]; for instance,
each sensor can update its estimate and then share the latter with its
neighbors.

The device at each node of the sensor network is typically low-cost and
battery-supplied. The latter feature, indeed, guarantees a certain ability of
the device to adapt to the surrounding environment. On the other hand, the
data transmission represents the most energy consuming node task. Thus, it is
fundamental to keep under control data transmission which can be done by means
of data-driven (or event-triggered) strategies for scheduling data
communication, see [6, 7, 8], [9, 10, 11]. In this paper we focus our
attention to the distributed scheme proposed in [12]: each node updates its
estimate with the new measurement (if available); then, the latter is compared
with the one just propagated in time through the state space model (i.e. the
estimate that can be computed also by the neighbors of that node in the case
there is no transmission). If the discrepancy between them is large, then the
node sends out the updated estimate to its neighbors. Finally, each node
performs a fusion of its estimate and the ones from its neighbors. The
appealing property of this transmission rule is that it allows to obtain a
distributed algorithm which enjoys nice stability properties (i.e., mean-
square boundedness of the state estimation error in all nodes) under minimal
requirements.

In many situations the actual model is different from the nominal one (i.e.
the one used in the estimation algorithm), [13]. As a consequence, the
resulting performance will be poor. One possible way to address this issue is
to consider the robust Kalman filter proposed in [14]: the idea is to consider
a dynamic minimax game where one player is the estimator minimizing the state
prediction error, while the other one is the hostile player which selects the
least favorable model in a set of plausible models called ambiguity set. The
latter is a ball which is formed by placing an upper bound on the Kullback-
Leibler divergence between the nominal state space model and the models inside
in. The radius of this ball is called tolerance and defines the magnitude of
the uncertainty in the nominal model.

Distributed strategies which are robust to model uncertainty have been already
proposed in the literature, see for instance [15, 16, 17, 18], however, to the
best of the authors’ knowledge, none of them consider the case with event-
triggered communication.

The contribution of this paper is to propose a new distributed Kalman filter
with event-triggered communication under model uncertainty. More precisely,
our approach represents a robust version of the distributed strategy proposed
by [12]. Moreover, we also present a preliminary numerical example in order to
test the performance of the proposed strategy.

The outline of the paper is as follows. In Section II we formulate the
distributed state estimation problem characterized by data transmission
constraints and model uncertainty. In Section III we introduce the proposed
robust distributed approach with event-triggered communication. In Section IV
we consider a numerical example showing the strength of our method. Finally,
in Section V we draw the conclusions.

##  II Problem formulation

Consider a network of nodes described by the digraph
$(\mathcal{N},\mathcal{A},\mathcal{S})$ where: $\mathcal{N}=\\{1,...,N\\}$ is
the set of the nodes, $\mathcal{A}\subseteq\mathcal{N}\times\mathcal{N}$ is
the set of edges and $\mathcal{S}\subseteq\mathcal{N}$ is the subset of
sensors node. The nodes in $\mathcal{S}$ are the only one that have the
capabilities to perform measurements, while the simple nodes in
$\mathcal{N}\setminus\mathcal{S}$ are used to increase the connectivity of the
network. If $(j,i)\in\mathcal{A}$, then it means that node $j$ can transmit
data to node $i$; moreover, all the possible self-loops belong to
$\mathcal{A}$. For each node $i\in\mathcal{N}$, the subset
$\mathcal{N}_{i}:=\\{j:(j,i)\in\mathcal{A},\;j\neq i\\}\subseteq\mathcal{N}$
denotes the set of its in-neighbors, i.e. the set composed by the nodes that
send information to node $i$.

We attach to this network the following nominal state space model

$\displaystyle x_{t+1}$ $\displaystyle=Ax_{t}+B\omega_{t}$ (1) $\displaystyle
y^{i}_{t}$
$\displaystyle=C^{i}x_{t}+D^{i}v^{i}_{t}\;,\;\;\;\;\;i\in\mathcal{S}$ (2)

where $x_{t}\in\mathbb{R}^{n}$ is the state, $y^{i}_{t}\in\mathbb{R}^{p_{i}}$
is the output at the sensor node $i$. Furthermore, matrix $B$ and $D^{i}$ are
full row rank matrices, $\omega_{t}$ and $v^{i}_{t}$ are zero-mean normalized
Gaussian white noises. The initial state $x_{0}$ is with mean $\mu_{0}$ and
with covariance matrix $P_{0}$. Finally, we assume that $\omega_{k}$’s,
$v^{i}_{k}$’s and $x_{0}$ are independent. Notice that, the model (1)-(2) can
be written as

$\displaystyle x_{t+1}$ $\displaystyle=Ax_{t}+B\omega_{t}$ $\displaystyle
y_{t}$ $\displaystyle=Cx_{t}+D^{i}v_{t}$

where $C:=\mathrm{col}(C^{i},i\in\mathcal{S})$, i.e. it is the matrix obtained
by stacking $C^{i}$’s, $D:=\mathrm{diag}(D^{i},i\in\mathcal{S})$ , i.e. it is
the block diagonal matrix whose main blocks are $D^{i}$’s,
$y_{t}:=\mathrm{col}(y_{t}^{i},i\in\mathcal{S})\in\mathbb{R}^{p}$ and
$v_{t}:=\mathrm{col}(v_{t}^{i},i\in\mathcal{S})\in\mathbb{R}^{p}$.

In this paper we face a distributed state estimation problem under model
uncertainty and data transmission constraints. More precisely, each node
$i\in\mathcal{N}$ must estimate the state $x_{t}$ at each time instant
$t\in\mathbb{Z}_{+}=\\{1,...,T\\}$ given $Y_{t}:=\\{y_{0}\ldots y_{t}\\}$ and
taking into account that: i) the actual model does not coincide with the
nominal one (1)-(2); ii) each node $i$ can selectively transmits only the most
relevant data, without compromising stability properties.

Regarding model uncertainty, we consider the framework proposed in [14, 19,
20] which is briefly reviewed below. The nominal model (1)-(2) over the time
interval $t=1\ldots N$ can be equivalently described by the conditional
probability densities $\phi_{t}(z_{t}|x_{t})$, where
$z_{t}=[\,x_{t+1}\;y_{t}\,]^{T}$, $t=1\ldots N$, and $f(x_{0})$. We assume
that the actual model is described by the conditional probability densities
$\tilde{\phi}_{t}(z_{t}|x_{t})$, $t=1\ldots N$, and $f(x_{0})$. Moreover, we
assume that $\tilde{\phi}_{t}$ belongs to the ambiguity set
$\mathcal{B}_{t}=\\{\,\tilde{\phi}_{t}\hbox{ s.t.
}\tilde{\mathbb{E}}[\log(\tilde{\phi}_{t}/\phi_{t})|Y_{t-1}]\leq b\,\\}$,

$\displaystyle\tilde{\mathbb{E}}[\log(\tilde{\phi}_{t}/\phi_{t})|Y_{t-1}]:=$
$\displaystyle\int_{\mathbb{R}^{n}}\int_{\mathbb{R}^{n+p}}\tilde{\phi}_{t}(z_{t}|x_{t})\tilde{f}_{t}(x_{t}|Y_{t-1})\log\left(\frac{\tilde{\phi}_{t}(z_{t}|x_{t})}{\phi_{t}(z_{t}|x_{t})}\right)\mathrm{d}z_{t}\mathrm{d}x_{t},$
(3)

$\tilde{f}_{t}$ is the actual probability density of $x_{t}$ given $Y_{t-1}$
and $b>0$ is called tolerance which accounts for model uncertainty. In plain
words, $\mathcal{B}_{t}$ is a ball about the nominal density with with radius
$b$.

[14] proposed as robust state estimator the one solving the following minimax
game:

$\displaystyle
x_{t+1|t}=\underset{g_{t}\in\mathcal{G}}{\operatornamewithlimits{argmin}}\underset{\tilde{\phi}_{t}\in\mathcal{B}_{t}}{\max}\,\tilde{\mathbb{E}}[\|x_{t+1}-g_{t}(y_{t})\|^{2}|Y_{t-1}]$
(4)

where $x_{t+1|t}$ is the estimator of $x_{t+1}$ given $Y_{t}$; $\mathcal{G}$
is the set of estimators having finite second order moments for any
$\tilde{\phi}_{t}\in\mathcal{B}_{t}$;

$\displaystyle\tilde{\mathbb{E}}$
$\displaystyle\left[\|x_{t+1}-g_{t}(y_{t})\|^{2}|Y_{t-1}\right]=$
$\displaystyle\int_{\mathbb{R}^{n}}\int_{\mathbb{R}^{n+p}}\|x_{t+1}-g_{t}(y_{t})\|^{2}\tilde{\phi}_{t}(z_{t}|x_{t})\tilde{f}_{t}(x_{t}|Y_{t-1})dz_{t}dx_{t}$
(5)

and it is assumed that
$\tilde{f}_{t}(x_{t}|Y_{t-1})\sim\mathcal{N}(\hat{x}_{t|t-1},V_{t|t-1})$. The
basic idea behind this paradigm is that when we are looking for an estimator
that minimize properly the selected loss function, a hostile player called
“nature” conspires to select the worst possible model in the ambiguity set
$\mathcal{B}_{t}$. In [14] it has been shown that the (centralized) robust
estimator solution to (4) admits a Kalman-like structure. It is not difficult
to show that such a filter can be written in the information form as follows.
Let $P_{t|t-1}$ and $V_{t|t-1}$ denote the pseudo-nominal and least favorable,
respectively, covariance matrix of the prediction error at time $t$; let
$x_{t|t}$ denote the estimator of $x_{t}$ given $Y_{t}$ and $P_{t|t}$ denotes
the covariance matrices of the corresponding estimation error at time $t$. We
define the corresponding matrices in the information form as
$\Omega_{t|t-1}=P_{t|t-1}^{-1}$, $\Psi_{t|t-1}=V_{t|t-1}^{-1}$,
$\Omega_{t|t}=P_{t|t}^{-1}$ and the information states as

$q_{t|t-1}=\Psi_{t|t-1}{x}_{t|t-1},\quad q_{t|t}=\Omega_{t|t}{x}_{t|t}.$

Then, it is not difficult to see that the robust estimator obeys to

Correction step:
$\displaystyle\begin{cases}\Omega_{t|t}=\Psi_{t|t-1}+C^{T}R^{-1}C\\\
q_{t|t}=q_{t|t-1}+C^{T}R^{-1}y_{t},\end{cases}$ (6) Prediction step:
$\displaystyle\begin{cases}\Omega_{t+1|t}=\\\ \hskip
5.69046ptQ^{-1}-Q^{-1}A(A^{T}Q^{-1}A+\Omega_{t|t})A^{T}Q^{-1}\\\ \text{Find
}\theta_{t}>0\text{ s.t. }\gamma(\Omega_{t+1|t},\theta_{t})=b\\\
\Psi_{t+1|t}=\Omega_{t+1|t}-\theta_{t}\mathbf{I}_{n}\\\
q_{t+1|t}=\Psi_{t+1|t}A\Omega^{-1}_{t|t}q_{t|t},\end{cases}$ (7)

where $R:=DD^{T}$, $Q:=BB^{T}$ are positive definite matrices and

$\displaystyle\gamma$ $\displaystyle(\Omega,\theta)$
$\displaystyle:=\frac{1}{2}\left\\{tr[(\mathbf{I}_{n}-\theta\Omega^{-1})^{-1}-\mathbf{I}_{n}]+\log\det(\mathbf{I}_{n}-\theta\Omega^{-1})\right\\}.$

The parameter $\theta_{t}>0$ is called risk sensitivity parameter. It is worth
noting that given $\Omega>0$ and $b>0$, the equation $\gamma(\Omega,\theta)=b$
always admits a unique solution $\theta>0$ such that
$\Omega-\theta\mathbf{I}_{n}>0$. Furthermore, in the special case where $b=0$,
i.e. there is no model uncertainty, then $\gamma(\Omega,\theta)=0$ implies
that $\theta_{t}=0$ and thus the above equations degenerates in the usual
Kalman equations in the information form.

##  III Proposed algorithm

Before to introduce our robust distributed estimation paradigm we consider the
following quite simple scenario. We assume that $\mathcal{N}=\mathcal{S}$,
i.e. all the nodes are sensor nodes, and $\mathcal{A}=\emptyset$, i.e. the
nodes do not communicate. In the presence of model uncertainty, at node $i$ we
can consider the robust Kalman filter in the information form based on the
local model

$\displaystyle x_{t+1}$ $\displaystyle=Ax_{t}+B\omega_{t}$ (8) $\displaystyle
y^{i}_{t}$ $\displaystyle=C^{i}x_{t}+D^{i}v^{i}_{t}\;;$ (9)

thus, we obtain the following algorithm:

Correction step:
$\displaystyle\begin{cases}\Omega_{t|t}^{i}=\Psi_{t|t-1}^{i}+(C^{i})^{T}(R^{i})^{-1}C^{i}\\\
q_{t|t}^{i}=q_{t|t-1}^{i}+(C^{i})^{T}(R^{i})^{-1}y_{t}^{i}\end{cases}$ (10)
Prediction step: $\displaystyle\begin{cases}\Omega_{t+1|t}^{i}=\\\ \hskip
5.69046ptQ^{-1}-Q^{-1}A(A^{T}Q^{-1}A+\Omega_{t|t}^{i})A^{T}Q^{-1}\\\
\text{Find }\theta_{t}^{i}>0\text{ s.t.
}\gamma(\Omega_{t+1|t}^{i},\theta_{t}^{i})=b\\\
\Psi_{t+1|t}^{i}=\Omega_{t+1|t}^{i}-\theta_{t}^{i}\mathbf{I}_{n}\\\
q_{t+1|t}^{i}=\Psi_{t+1|t}^{i}A(\Omega^{i}_{t|t})^{-1}q_{t|t}^{i}\end{cases}$
(11)

where $R^{i}:=D^{i}(D^{i})^{T}$. It is worth noting that each node has its own
risk sensitivity parameter $\theta_{t}^{i}$.

Next, we consider the scenario in which: i) $\mathcal{S}\subseteq\mathcal{N}$;
ii) each node $i$ can transmit its local estimate ${q}^{i}_{t|t}$ and
information matrix $\Omega^{i}_{t|t}$ to all its out-neighbors
$\\{\,j\,:\,(i,j)\in\mathcal{A},\,i\neq j\,\\}$ if necessary. In plain words,
each node $i$ can decide at any time step whether to transmit or not its data,
i.e. $(q^{i}_{t|t},\Omega^{i}_{t|t})$, without compromising the stability
properties of the algorithm, i.e. $\|x_{t}-x^{i}_{t|t}\|$ and
$\|P^{i}_{t|t}\|$ do not diverge for any $i\in\mathcal{N}$ as $t$ approaches
infinity.

The estimation paradigm that we now present is a robust extension of the
distributed state estimation algorithm with event-triggered communication
proposed in [12] and it is composed by four steps described below.

Correction. At time $t$, the predicted pair $(q^{i}_{t|t-1},\Psi^{i}_{t|t-1})$
is available at node $i\in\mathcal{N}$. If $i\in\mathcal{S}$, i.e. it is a
sensor node, then also the measurement $y_{t}^{i}$ is available and thus the
correction step coincides with (10). If $i\notin\mathcal{S}$, no measurement
is available at the node, then we can only propagate the prediction couple.
Therefore, the so called information pair is obtained as

$(q^{i}_{t|t},\Omega^{i}_{t|t})=\begin{cases}\text{use equations
(\ref{eq:RKFfilteringloc}),}&\text{if $i\in\mathcal{S}$}\\\
(q^{i}_{t|t-1},\Psi^{i}_{t|t-1}),&\text{if $i\in\mathcal{N\setminus
S}$}.\end{cases}$ (12)

Information exchange. Each node $i\in\mathcal{N}$ sends its information couple
$(q^{i}_{t|t},\Omega^{i}_{t|t})$ to all its out-neighbors according to the
binary variable $c^{i}_{t}$:

  * •

if $c^{i}_{t}=1$, then node $i$ transmits the information couple to all its
out-neighbors at time $t$;

  * •

if $c^{i}_{t}=0$, then node $i$ does not transmit the information couple to
all its out-neighbors at time $t$.

It remains to define the binary variable $c^{i}_{t}$. Let
$n^{i}_{t}\in\mathbb{N}$ be the number of time instants elapsed from the most
recent transmission of node $i$, i.e. the most recently transmitted data is
$(q^{i}_{t-n^{i}_{r}|r-n^{i}_{t}},\Omega^{i}_{t-n^{i}_{t}|t-n^{i}_{t}})$.
Then, all the out-neighbors of node $i$ propagate
$(q^{i}_{t-n^{i}_{r}|t-n^{i}_{t}},\Omega^{i}_{t-n^{i}_{t}|t-n^{i}_{t}})$ in
time through a prediction step which takes into account the fact that the
actual model does not coincide with the nominal one (see (26) in the
prediction step below). Let $(\bar{q}^{i}_{t},\bar{\Omega}^{i}_{t})$ denote
this propagated pair at time $t$. Then, the transmission rule $c_{t}^{i}$
computed at node $i$ is defined as in [12]:

$c^{i}_{t}=\begin{cases}0,&\quad\text{if
}\|{x}^{i}_{t|t}-\bar{x}^{i}_{t}\|^{2}_{\Omega^{i}_{t|t}}\leq\alpha\\\ &\text{
and
}\frac{1}{1+\beta}\Omega^{i}_{t|t}\leq\bar{\Omega}^{i}_{t}\leq(1+\delta)\Omega^{i}_{t|t}\\\
1,&\quad\text{otherwise}\end{cases}$ (13)

where $\bar{x}^{i}_{t}=(\bar{\Omega}^{i}_{t})^{-1}\bar{q}^{i}_{t}$ represents
the state prediction based on the propagation of the most recent transmitted
pair $(q^{i}_{t-n^{i}_{r}|t-n^{i}_{t}},\Omega^{i}_{t-n^{i}_{t}|t-n^{i}_{t}})$;
$x^{i}_{t|t}=({\Omega}^{i}_{t|t})^{-1}q^{i}_{t|t}$ is the state estimate at
node $i$. In plain words, the transmission rule in (13) checks the discrepancy
between $(q^{i}_{t|t},\Omega^{i}_{t|t})$ and
$(\bar{q}^{i}_{t},\bar{\Omega}^{i}_{t})$. If the latter is large, then it
means that the out-neighbors own a prediction corresponding to node $i$ which
is bad and thus node $i$ must transmit the data. The positive scalars
$\alpha$, $\beta$ and $\delta$ can be tuned by the user in order to reach a
desired behavior in terms of transmission rate and performance. More
precisely, $\alpha$ tunes the bound on the discrepancy between $x^{i}_{t|t}$
and $\bar{x}^{i}_{t}$, while $\beta$ and $\delta$ tunes the allowed mismatch
between the covariance matrices ${\Omega}^{i}_{t|t}$ and
$\bar{\Omega}^{i}_{t}$. In [21] it has been shown that the transmission
strategy in (13) guarantees the following upper bound. If we model the
propagated and the information pairs as
$\mathcal{N}(\bar{q}^{i}_{t},\bar{\Omega}^{i}_{t})$ and
$\mathcal{N}(q^{i}_{t|t},\Omega^{i}_{t|t})$, respectively, then

$D_{KL}(\mathcal{N}(q^{i}_{t|t},\Omega^{i}_{t|t}),\mathcal{N}(\bar{q}^{i}_{t},\bar{\Omega}^{i}_{t}))\leq\frac{1}{2}[\alpha+\beta
n+n\log(1+\delta)]$

where $n$ is the state dimension and $D_{KL}$ denotes the Kullback-Leibler
divergence.

Information fusion. In this step, any node merges its information with the
ones regarding its in-neighbors. Let $\Pi\in\mathbb{R}^{N\times N}$ denote the
consensus matrix whose element in position $(i,j)$ is defined as:

$\pi_{i,j}=\begin{cases}(d_{i}+1)^{-1},&\text{ if }(j,i)\in\mathcal{A}\\\
0,&\text{otherwise}\end{cases}$

where $d_{i}$ denotes the degree of node $i$; in this way we have that
$\pi_{i,j}$ with $j\in\mathcal{N}_{i}$ represents the coefficients of a convex
combination. Then, the fusion step is performed through the following convex
combination of the pairs:

$q^{i,F}_{t|t}=\pi_{i,i}q^{i}_{t|t}+\sum_{j\in\mathcal{N}_{i}}\pi_{i,j}\left[c^{j}_{t}q^{j}_{t|t}+(1-c^{j}_{t})\tilde{q}^{j}_{t}\right]$
(14)
$\Omega^{i,F}_{t|t}=\pi_{i,i}\Omega^{i}_{t|t}+\sum_{j\in\mathcal{N}_{i}}\pi_{i,j}\left[c^{j}_{t}\Omega^{j}_{t|t}+(1-c^{j}_{t})\tilde{\Omega}^{j}_{t}\right]$
(15)

where

$\tilde{q}^{j}_{t}=\frac{1}{1+\delta}\bar{q}^{j}_{t}\;\;\;,\;\;\;\tilde{\Omega}^{j}_{t}=\frac{1}{1+\delta}\bar{\Omega}^{j}_{t}.$

In view of (14)-(15), we can see that in the fusion step we consider
$(q^{j}_{t|t},\Omega^{j}_{t|t})$, if node $j$ transmitted its information pair
at time $t$. If node $j$ does not transmit, then the aforementioned pair is
not available at node $i$. To account for this lack, given that at each
iteration the nodes can calculate the pair
$(\bar{q}^{i}_{t},\bar{\Omega}^{i}_{t})$, which is certainly less informative
than $(q^{i}_{t|t},\Omega^{i}_{t|t})$, it becomes convenient to shrink
$(\bar{q}^{i}_{t},\bar{\Omega}^{i}_{t})$ in (14)-(15) by the factor
$(1+\delta)^{-1}$ in order to decrease its importance in the fusion step, see
[12] for more details.

Prediction. Once each node $i\in\mathcal{N}$ has computed the fused
information couple $(q^{i,F}_{t|t}\Omega^{i,F}_{t|t})$, the latter is
propagated in time with the robust prediction step in (11) where
$(q^{i}_{t|t},\Omega_{t|t}^{i})$ is now replaced by
$(q^{i,F}_{t|t},\Omega_{t|t}^{i,F})$:

$\displaystyle\left.\begin{array}[]{ll}\Omega_{t+1|t}^{i}&=Q^{-1}-Q^{-1}A(A^{T}Q^{-1}A+\Omega_{t|t}^{i,F})A^{T}Q^{-1}\\\
\text{Find }&\theta_{t}^{i}>0\text{ s.t.
}\gamma(\Omega_{t+1|t}^{i},\theta_{t}^{i})=b\\\
\Psi_{t+1|t}^{i}&=\Omega_{t+1|t}^{i}-\theta_{t}^{i}\mathbf{I}_{n}\\\
q_{t+1|t}^{i}&=\Psi_{t+1|t}^{i}A(\Omega^{i,F}_{t|t})^{-1}q_{t|t}^{i,F}.\end{array}\right.$
(20)

In this step we also need to propagate in time the pair
$(\bar{q}^{i}_{t},\bar{\Omega}^{i}_{t})$, i.e. the one used in the case node
$i$ does not transmit its information pair. Notice this operation is performed
by both node $i$ and its out-neighbors and it can be summarized as follows. At
nodes $i\,\cup\,\mathcal{N}_{i}$ we have the pair
$(\breve{q}^{i}_{t},\breve{\Omega}^{i}_{t})$ defined as

$\displaystyle\begin{cases}\breve{q}^{i}_{t}&=c^{i}_{t}q^{i}_{t|t}+(1-c^{i}_{t})\bar{q}^{i}_{t}\\\
\breve{\Omega}^{i}_{t}&=c^{i}_{t}\Omega^{i}_{t|t}+(1-c^{i}_{t})\bar{\Psi}^{i}_{t};\end{cases}$
(21)

then, it is propagated in time with the robust prediction step in (11) where
$(q^{i}_{t|t},\Omega_{t|t}^{i})$ is now replaced by
$(\breve{q}^{i}_{t},\breve{\Omega}_{t}^{i})$:

$\displaystyle\left.\begin{array}[]{ll}\bar{\Omega}_{t+1}^{i}&=Q^{-1}-Q^{-1}A(A^{T}Q^{-1}A+\breve{\Omega}_{t}^{i})A^{T}Q^{-1}\\\
\text{Find }&\bar{\theta}_{t}^{i}>0\text{ s.t.
}\gamma(\bar{\Omega}_{t+1}^{i},\bar{\theta}_{t}^{i})=b\\\
\bar{\Psi}_{t+1}^{i}&=\bar{\Omega}_{t+1}^{i}-\bar{\theta}_{t}^{i}\mathbf{I}_{n}\\\
\bar{q}_{t+1}^{i}&=\bar{\Psi}_{t+1}^{i}A(\breve{\Omega}^{i}_{t})^{-1}\breve{q}_{t}^{i}.\end{array}\right.$
(26)

The procedure is summarized in Algorithm 1 below. It is interesting to note
that each node $i$ is characterized by two risk sensitivity parameters, i.e.
$\theta^{i}_{t}$ and $\bar{\theta}^{i}_{t}$. In the case that $b=0$, i.e.
there is no model uncertainty, in Algorithm 1 we have: $\theta^{i}_{t}=0$,
$\bar{\theta}^{i}_{t}=0$ and thus $\Psi_{t}^{i}=\Omega_{t}^{i}$,
$\bar{\Psi}_{t}^{i}=\bar{\Omega}_{t}^{i}$, i.e. we recover the distributed
Kalman algorithm with event-triggered communication proposed in [12].

Algorithm 1 RDKF with event-triggered communication

Initialization: Set $(q^{i}_{0|-1},\Psi^{i}_{0|-1})$ for any $i\in\mathcal{N}$
For each $t=0,1,\ldots$
For each node $i\in\mathcal{N}$

Correction:

$(q^{i}_{t|t},\Omega^{i}_{t|t})=\begin{cases}\text{use
(\ref{eq:RKFfilteringloc}),}&\text{if $i\in\mathcal{S}$}\\\
(q^{i}_{t|t-1},\Psi^{i}_{t|t-1}),&\text{if $i\in\mathcal{N\setminus
S}$}\end{cases}$

Information exchange:

-

if $t=0$ set $c^{i}_{t}=1$, otherwise determine $c_{t}^{i}$ according to (13)

-

if $c^{i}_{t}=1$ transmit $(q^{i}_{t|t},\Omega^{i}_{t|t})$ to the out-
neighbors

-

receive $(q^{j}_{t|t},\Omega^{j}_{t|t})$ from all the in-neighbors
$j\in\mathcal{N}_{i}$ for which $c_{t}^{j}=1$

Information fusion:

$\tilde{q}^{j}_{t}=\frac{1}{1+\delta}\bar{q}^{j}_{t}\;\;\;,\;\;\;\tilde{\Omega}^{j}_{t}=\frac{1}{1+\delta}\bar{\Omega}^{j}_{t}\text{
with }j\in\mathcal{N}_{i}$ $\displaystyle q^{i,F}_{t|t}$
$\displaystyle=\pi_{i,i}q^{i}_{t|t}+\sum_{j\in\mathcal{N}_{i}}\pi_{i,j}\left[c^{j}_{t}q^{j}_{t|t}+(1-c^{j}_{t})\tilde{q}^{j}_{t}\right]$
$\displaystyle\Omega^{i,F}_{t|t}$
$\displaystyle=\pi_{i,i}\Omega^{i}_{t|t}+\sum_{j\in\mathcal{N}_{i}}\pi_{i,j}\left[c^{j}_{t}\Omega^{j}_{t|t}+(1-c^{j}_{t})\tilde{\Omega}^{j}_{t}\right]$

Prediction step:

-

Compute $(q^{i}_{t+1|t},\Omega^{i}_{t+1|t})$ using (20)

-

Compute $(\bar{q}^{i}_{t+1|t},\bar{\Psi}^{i}_{t+1|t})$ using (21) and (26)

##  IV Numerical example

Figure 1: Average root mean square error across the network. Figure 2: Average
root mean square error at each node over the time horizon $[1,300]$. Figure 3:
Transmission rate across the network. Figure 4: Average risk sensitivity
parameters across the network.

In this section we evaluate the performance of the proposed robust distributed
Kalman algorithm with event-triggered communication. We consider the problem
in [1] of tracking the position of a projectile by using noisy position
measurements obtained by a network of $N=100$ nodes where $20$ of them are
sensor nodes. The possible connections among the nodes has been randomly
generated in such a way the network is strongly connected. The model for the
projectile motion is

$\displaystyle\dot{x}^{c}_{t}=\Phi x^{c}_{t}+u^{c}_{t}$ (27)

where

$\displaystyle\Phi=\left[\begin{array}[]{cc}0&0\\\
\mathbf{I}_{3}&0\end{array}\right],$ (30)

$u^{c}_{t}=[\,0\;0\;-g\;0\;0\;0\,]^{T}$, with $g=-10$, and
$x^{c}_{t}=[\,v_{x,t}\;v_{y,t}\;v_{z,t}\;p_{x,t}\;p_{y,t}\;p_{z,t}\;\,]^{T}$
with $v$ denoting the velocity, $p$ the position and the subscripts $x,y,z$
denoting the three spatial dimensions. We discretize (27) with sampling time
equal to $0.1$. The corresponding discrete time model is
$x_{t+1}=Ax_{t}+u_{t}$ where $x_{t}$ is the sampled version of $x^{c}_{t}$,
$A=\mathbf{I}_{6}+0.1\Phi$ and
$u_{t}=(0.1\mathbf{I}_{6}+0.1^{2}\Phi/2)u^{c}_{t}$. We assume that every
sensor measures the position of the projectile in either two horizontal
dimensions, or a combination of one horizontal dimension and the vertical
dimension; in plain words, one sensor does not have measurements in all the
three dimensions. Therefore, we obtain the nominal discrete state-space model
(1)-(2) where $C^{i}=[\,0\;0\;0\;\mathrm{diag(1,1,0)}\,]$, in the case that
the sensor measures only the horizontal positions, or
$C^{i}=[\,0\;0\;0\;\mathrm{diag(1,0,1)}\,]$,
$C^{i}=[\,0\;0\;0\;\mathrm{diag(0,1,1)}\,]$, in the case that the sensor
measures one horizontal position and the vertical position. Moreover, we
choose $B=\sqrt{0.001}I$, $R^{i}=D^{i}(D^{i})^{T}=\sqrt{k}PR_{0}P^{T}$ where
$R_{0}=0.5\cdot\mathrm{diag}(1,4,7)$ and $P$ is a permutation matrix randomly
chosen for every node. Finally, the initial state $x_{0}$ is a Gaussian random
vector with covariance matrix $P_{0}=I$. Since the previous model is just an
idealization of the underlying physical system, we assume that the actual
state-space model belongs to the ambiguity set $\mathcal{B}_{t}$ about the
aforementioned nominal model and with tolerance $b=0.05$. More precisely, we
assume that the actual model is the least favorable model solution to (i.e.
the maximizer of) the centralized problem (4).

In the following we consider the following distributed filters:

  * •

RDKF – the distributed robust Kalman filter with event-triggered communication
in Algorithm 1 with $b=0.05$. Here, the transmission rule (13) is with
$\alpha=30$, $\beta=0.2$ and $\delta=0.1$;

  * •

DKF1 – the distributed Kalman filter with event-triggered communication
proposed in [12] and the transmission rule is with $\alpha=30$, $\beta=0.2$
and $\delta=0.1$;

  * •

DKF2 – is the same as DKF1 but the transmission rule is with $\alpha=0.01$,
$\beta=0.2$ and $\delta=0.1$.

It is worth noting that RDKF and DKF1 have the same parameters for the
transmission rule. However, taking the parameter $\alpha$ in (13) the same for
both RDKF and DKF1 provides a transmission rate for DKF1 which is smaller than
the one of RDKF (see below). For this reason, we also consider DKF2 where the
parameter $\alpha$ has been decreased in oder to increase the transmission
rate. We also tried to increase the transmission rate by keeping fixed
$\alpha=0.01$ and changing $\beta,\delta$; however, we did not notice a
significant growth in terms of transmission rate.

We consider the following performance indexes:

  * •

The average root mean square error across the network at time $t$:

$\mathrm{RMSE}_{t}=\frac{1}{N}\sum_{i=1}^{N}\|x_{t}-x_{t|t}^{i}\|^{2}.$

  * •

The average root mean square error at node $i$ over the time horizon
$[1,300]$:

$\mathrm{RMSE}_{i}=\frac{1}{300}\sum_{t=1}^{300}\|x_{t}-x_{t|t}^{i}\|^{2}.$

  * •

The transmission rate across the network at time $t$, i.e. the faction of
nodes that transmit their data at time $t$.

Figure 1 and Figure 2 show the two aforementioned root mean square errors,
while Figure 3 shows the transmission rate. As we can see RDKF outperforms
DKF1. On the other hand, the transmission rate of DKF1 is smaller than the one
of DKF. However, even in the case we increase the transmission rate, i.e. we
consider DKF2, RDKF is the best estimator.

Finally, Figure 4 shows the average risk sensitivity parameters across the
network:

$\displaystyle\theta_{t}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\theta_{t}^{i},\quad\bar{\theta}_{t}=\frac{1}{N}\sum_{i=1}^{N}\bar{\theta}_{t}^{i}.$

We can notice that $\theta_{t}\geq\bar{\theta}_{t}$, moreover we have checked
that $\theta_{t}^{i}\geq\bar{\theta}_{t}^{i}$ for many nodes. Since the
mapping $\theta\mapsto\gamma(\Omega,\theta)$, with $\Omega>0$, is monotone
increasing, [22, 23, 24], and in view of the fact that

$\gamma(\Omega_{t+1|t}^{i},\theta_{t}^{i})=\gamma(\bar{\Omega}^{i}_{t+1},\bar{\theta}_{t}^{i})=b,$

it follows that $\Omega_{t+1|t}^{i}\geq\bar{\Omega}^{i}_{t+1}$ for many nodes.

##  V Conclusion

We have proposed a robust distributed Kalman filter with event-triggered
communication. More precisely, each sensor designs its predictor according to
the least favorable model in a prescribed ambiguity set. The latter is ball
about the nominal model. Moreover, each node transmits its data to its
neighbors only in the case the discrepancy between the information pair, i.e.
the one if we transmit, and the propagated pair, i.e. the one obtained without
transmission, is not negligible. A numerical example showed the effectiveness
of the proposed method in the case the actual model is different from the
nominal one.

It remains to investigate the stability of the proposed distributed filter. We
conjecture that, under the assumptions made in [12], that is

  * A1.

the system matrix A is invertible,

  * A2.

the system is collectively observable, i.e., the pair (A,C) is observable
where $C=\mathrm{col}((C^{i};\;i\in\mathcal{S}))$,

  * A3.

the network is strongly connected, i.e. there exists a directed path between
any pair of nodes $i,j\in\mathcal{N}$,

the estimation error $x_{t}-x_{t|t}^{i}$ is bounded in mean-square in each
node.

## References

  * [1] F. S. Cattivelli and A. H. Sayed, “Diffusion strategies for distributed Kalman filtering and smoothing,” IEEE Transactions on Automatic Control, vol. 55, no. 9, pp. 2069–2084, 2010.
  * [2] D. P. Spanos, R. Olfati-Saber, and R. M. Murray, “Approximate distributed Kalman filtering in sensor networks with quantifiable performance,” in Fourth International Symposium on Information Processing in Sensor Networks, pp. 133–139, April 2005.
  * [3] R. Olfati-Saber, “Distributed Kalman filtering for sensor networks,” in 2007 46th IEEE Conference on Decision and Control, pp. 5492–5498, 2007\.
  * [4] G. Battistelli and L. Chisci, “Kullback-Leibler average, consensus on probability densities, and distributed state estimation with guaranteed stability,” Automatica, vol. 50, no. 3, pp. 707–718, 2014.
  * [5] A. Kamal, J. Farrell, and A. Roy-Chowdhury, “Information weighted consensus filters and their application in distributed camera networks,” IEEE Transactions on Automatic Control, vol. 58, no. 12, pp. 3112–3125, 2013.
  * [6] G. Battistelli, A. Benavoli, and L. Chisci, “Data-driven communication for state estimation with sensor networks,” Automatica, vol. 48, no. 5, pp. 926–935, 2012.
  * [7] D. Han, Y. Mo, J. Wu, S. Weerakkody, B. Sinopoli, and L. Shi, “Stochastic event-triggered sensor schedule for remote state estimation,” IEEE Transactions on Automatic Control, vol. 60, no. 10, pp. 2661–2675, 2015.
  * [8] D. Shi, L. Shi, and T. Chen, Event-based state estimation.  Switzerland: Springer, 2016.
  * [9] Q. Liu, Z. Wang, X. He, and D. Zhou, “Event-based recursive distributed filtering over wireless sensor networks,” IEEE Transactions on Automatic Control, vol. 60, no. 9, pp. 2470–2475, 2015.
  * [10] W. Li, Y. Jia, and J. Du, “Event-triggered Kalman consensus filter over sensor networks,” IET Control Theory & Applications, vol. 10, no. 1, pp. 103–110, 2015.
  * [11] L. Yan, X. Zhang, Z. Zhang, and Y. Yang, “Distributed state estimation in sensor networks with event-triggered communication,” Nonlinear Dynamics, vol. 76, no. 1, pp. 169–181, 2014.
  * [12] G. Battistelli, L. Chisci, and D. Selvi, “A distributed Kalman filter with event-triggered communication and guaranteed stability,” Automatica, vol. 93, pp. 75–82, 2018.
  * [13] A. Zenere and M. Zorzi, “On the coupling of model predictive control and robust Kalman filtering,” IET Control Theory Applications, vol. 12, no. 13, pp. 1873–1881, 2018.
  * [14] B. Levy and R. Nikoukhah, “Robust state-space filtering under incremental model perturbations subject to a relative entropy tolerance,” IEEE Trans. Automat. Control, vol. 58, pp. 682–695, Mar. 2013.
  * [15] B. Shen, Z. Wang, and Y. Hung, “Distributed H∞-consensus filtering in sensor networks with multiple missing measurements: The finite-horizon case,” Automatica, vol. 46, no. 10, pp. 1682–1688, 2010.
  * [16] Y. Luo, Y. Zhu, D. Luo, J. Zhou, E. Song, and D. Wang, “Globally optimal multisensor distributed random parameter matrices Kalman filtering fusion with applications,” Sensors, vol. 8, no. 12, pp. 8086–8103, 2008.
  * [17] M. Zorzi, “Distributed Kalman filtering under model uncertainty,” IEEE Transactions on Control of Network Systems, vol. 7, no. 2, pp. 990–1001, 2020\.
  * [18] A. Emanuele, F. Gasparotto, G. Guerra, and M. Zorzi, “Robust distributed kalman filtering: On the choice of the local tolerance,” Sensors, vol. 20, no. 11, 2020.
  * [19] M. Zorzi, “Robust Kalman filtering under model perturbations,” IEEE Transactions on Automatic Control, vol. 62, June 2017.
  * [20] M. Zorzi, “On the robustness of the Bayes and Wiener estimators under model uncertainty,” Automatica, vol. 83, pp. 133–140, 2017.
  * [21] G. Battistelli, L. Chisci, and D. Selvi, “Distributed kalman filtering with data-driven communication,” in 19th International Conference on Information Fusion (FUSION), pp. 1042–1048, 2016.
  * [22] M. Zorzi, “Convergence analysis of a family of robust Kalman filters based on the contraction principle,” SIAM Journal on Control and Optimization, vol. 55, no. 5, pp. 3116–3131, 2017.
  * [23] M. Zorzi and B. Levy, “On the convergence of a risk sensitive like filter,” in 54th IEEE Conference on Decision and Control (CDC), pp. 4990–4995, Dec 2015.
  * [24] B. C. Levy and M. Zorzi, “A contraction analysis of the convergence of risk-sensitive filters,” SIAM Journal on Control and Optimization, vol. 54, no. 4, pp. 2154–2173, 2016.


11institutetext:  Dept. of Computer Science, University of Konstanz, Germany
11email<EMAIL_ADDRESS>22institutetext: Heidelberg Academy of
Sciences and Humanities, Germany 33institutetext: Dept. of Political Science,
University of Zurich, Switzerland 44institutetext: Data and Knowledge
Engineering, University of Wuppertal, Germany

# Towards Target-dependent Sentiment Classification in News Articles

Felix Hamborg 1122 0000-0003-2444-8056 Karsten Donnay 2233 0000-0002-9080-6539
Bela Gipp 2244 0000-0001-6522-3019

###### Abstract

Extensive research on target-dependent sentiment classification (TSC) has led
to strong classification performances in domains where authors tend to
explicitly express sentiment about specific entities or topics, such as in
reviews or on social media. We investigate TSC in news articles, a much less
researched domain, despite the importance of news as an essential information
source in individual and societal decision making. This article introduces
NewsTSC, a manually annotated dataset to explore TSC on news articles.
Investigating characteristics of sentiment in news and contrasting them to
popular TSC domains, we find that sentiment in the news is expressed less
explicitly, is more dependent on context and readership, and requires a
greater degree of interpretation. In an extensive evaluation, we find that the
current state-of-the-art in TSC performs worse on news articles than on other
domains (average recall $AvgRec=69.8$ on NewsTSC compared to
$AvgRev=\left[75.6,82.2\right]$ on established TSC datasets). Reasons include
incorrectly resolved relation of target and sentiment-bearing phrases and off-
context dependence. As a major improvement over previous news TSC, we find
that BERT’s natural language understanding capabilities capture the less
explicit sentiment used in news articles.

###### Keywords:

sentiment classification stance detection news bias media bias.

(0cm,1cm) This is a preprint. The accepted manuscript can be found at:
https://doi.org/10.1007/978-3-030-71305-8_12

##  1 Introduction

Target-dependent sentiment classification (TSC) is a sub-task of sentiment
analysis that aims to identify the sentiment of a text, usually on sentence-
level, towards a given target, such as named entities (NEs) or other semantic
concepts [19]. Aspect-based sentiment classification (ABSC) [29], a closely
related task, defines such targets as aspects of a given topic, e.g.,
“service” and “food” may be aspects of the topic “restaurant.” Previous
research on TSC and ABSC (due to their technical similarity we will refer to
both as TSC) has focused mostly on domains in which authors tend to express
their opinions explicitly, such as reviews, surveys, and social media [7, 26,
24, 29].

In this paper, we investigate TSC in the domain of news articles – a much less
researched domain that is of critical relevance, especially in times of “fake
news,” echo chambers, and news owner centralization [15]. How persons and
other entities are portrayed in articles on political topics is, e.g., very
relevant for individual and societal opinion formation [3, 17, 14].

The main contributions of this paper are: (1) We introduce _NewsTSC_ , a
manually annotated dataset for the exploration of TSC in political news
articles. (2) We discuss similarities and differences between political news
and established TSC domains. (3) We perform an extensive evaluation of state-
of-the-art TSC approaches on NewsTSC. To improve classification performance,
we also fine-tune a BERT language model [6] on a large news dataset, thereby
establishing the current state-of-the-art in TSC on political news.

We provide the dataset including code book, code to reproduce our experiments,
and the fine-tuned BERT at: https://github.com/fhamborg/newstsc

##  2 Related Work

Most TSC-related research uses three annotated datasets: _Restaurant_ and
_Laptop_ , containing reviews on restaurants and laptops [26], and _Twitter_ ,
consisting of tweets [7]. Each example in these datasets consists of a target,
context (often a single sentence or tweet), and the target’s sentiment within
its context.

The advent of word embeddings and deep learning including neural language
models, such as BERT [6], has led to a performance leap in many natural
language processing (NLP) disciplines including TSC, where, e.g., macro F1
gained from $F1_{m}=63.3$ [21] to $F1_{m}=75.8$ on the Twitter set [38].
Whereas traditional TSC research focused on careful feature engineering and
dictionary creation (cf. [21]), researchers now focus on designing neural
architectures suited to catch the relation between target and context [39, 32,
38]. By fine-tuning the underlying language model for the particular
classification domain, performance can be improved further [28].

Text in news articles differs from reviews and social media in that news
authors typically do not express sentiment towards a target explicitly
(exceptions include opinion pieces and columns). Instead, they implicitly or
indirectly express sentiment because language in news is expected to be
neutral and journalists to be objective [1, 10, 15]. For example, news texts
express sentiment by describing actions performed by a target, or by including
and highlighting information in favor or against a target (or omitting and
downplaying such information, respectively) [31]. Adding to the difficulty of
news TSC, different readers may assess an article’s sentiment towards a target
differently [1], depending on their own political or ideological views (we
discuss real-world examples in Section 3.3). Previous news TSC approaches
mostly employ manually created [1] or semi-automatically extended [10]
sentiment dictionaries. To our knowledge, there exist one dataset for
evaluation of news TSC methods [34], which – perhaps due to its small size
($N=1274$) – has not been used or tested in recent TSC literature. Another
dataset contains quotes extracted from news articles, since quotes more likely
contain explicit sentiment ($N=1592$) [1].

To our knowledge, no suitable datasets for news TSC exist nor have news TSC
approaches been proposed that exploit recent advances in NLP.

##  3 Dataset

We describe how we create the news TSC dataset, including the collection of
articles and the annotation procedure. Afterward, we discuss the
characteristics of the dataset.

###  3.1 Data Collection and Example Extraction

We create a base set of articles of high diversity in topics covered and
writing styles, e.g., whether emotional or factual words are used (cf. [8]).
Using a news extractor [16], we collect news articles from the Common Crawl
news crawl (CCNC, also known as CC-NEWS), consisting of over 250M articles
until August 2019 [23]. To ensure diversity in writing styles, we select 14 US
news outlets,111BBC, Breitbart, Chicago Tribune, CNN, LA Daily News, Fox News,
HuffPost, LA Times, NBC, NY Times, Reuters, USA Today, Washington Post, and
Wall Street Journal. which are mostly major outlets that represent the
political spectrum from left to right, based on selections by [4, 13, 2]. We
cannot simply select the whole corpus, because CCNC lacks articles for some
outlets and time frames. By selecting articles published between August 2017
and July 2019, we minimize such gaps while covering a time frame of two years,
which is sufficiently large to include many diverse news topics. To facilitate
the balanced contribution of each outlet and time-range we perform binning: we
create 336 bins, one for each outlet and month, and randomly draw 10 articles
reporting on politics for each bin, resulting in 3360 articles in total.222To
classify whether an article reports on politics, we use a DistilBERT-based
[30] classifier with a single dense layer and softmax trained on the HuffPost
[22] and BBC datasets [12]. During the subsequent manual annotation, coders
discard remaining, non-political articles. During binning, we remove any
article duplicates by text equivalence.

To create examples for annotation, we select all mentions of NEs recognized as
PERSON, NROP, or ORG for each article [37].333For this task, we use spaCy
v2.1. We discard NE mentions in sentences shorter than 50 characters. For each
NE mention, we create an example by using the mention as the target and its
surrounding sentence as its context. We remove any example duplicates.
Afterward, to ensure diversity in writing styles and topics, we use the
outlet-month binning described previously and randomly draw examples from each
bin.

Different means may be used to address expected class imbalance, e.g., for the
Twitter set, only examples that contained at least one word from a sentiment
dictionary were annotated [25, 24]. While doing so yields high frequencies of
classes that are infrequent in real-world distribution, it also causes dataset
shift and selection bias [27]. Thus, we instead investigate the effectiveness
of different means to address class imbalance during training and evaluation
(see Section 4).

###  3.2 Annotation

We set up an annotation process following best practices from the TSC
literature [26, 24, 29, 34]. For each example, we asked three coders to read
the context, in which we visually highlighted the target and assess the
target’s sentiment. Examples were shown in random order to each coder. Coders
could choose from positive, neutral, and negative polarity, whereby they were
allowed to choose positive and negative polarity at the same time. Coders were
asked to reject an example, e.g., if it was not political or a meaningless
text fragment. Before, coders read a code book that included instructions on
how to code and already annotated examples. Five coders, students, aged
between 24 and 32, participated in the process.

In total, 3288 examples were annotated, from which we discard 125 (3.8%) that
were rejected by at least one coder, resulting in 3163 non-rejected examples.
From these, we discard 3.3% that lacked a majority class, i.e., examples where
each coder assigned a different sentiment class, and 1.8% that were annotated
as positive and negative sentiment at the same time, to allow for better
comparison with previous TSC datasets and methods (see Section 2). Lastly, we
split the remaining 3002 examples into 2301 training and 701 test examples.
Table 1 shows class frequencies of the sets.

We use the full set of 3163 non-rejected examples to illustrate the degree of
agreement between coders: 3.3% lack a majority class, for 62.7%, two coders
assigned the same sentiment, and for 33.9% all coders agreed. On average, the
accuracy of individual coders is $acc_{h}=72.9\%$. We calculate two intercoder
reliability (ICR) measures. For completeness, Cohen’s Kappa is $\kappa=25.1$,
but it is unreliable in our case due to Kappa’s sensitivity to class imbalance
[5]. The mean pairwise observed agreement over all coders is $72.5$.

| negative | neutral | positive | total
---|---|---|---|---
training | 530 | 1600 | 171 | 2301
test | 167 | 487 | 47 | 701
total | 697 | 2087 | 218 | 3002
Table 1:  Class frequencies of NewsTSC sets.

###  3.3 Characteristics of Sentiment in News Articles

In a manual, qualitative analysis of NewsTSC, we find two key differences of
news compared to established domains: first, we confirm that news contains
mostly implicit and indirect sentiment (see Section 2). Second, determining
the sentiment in news articles typically requires a greater degree of
interpretation (cf. [34]). The second difference is caused by multiple
factors, particularly the implicitness of sentiment (mentioned as the first
difference) and that sentiment in news articles is more often dependent on
non-local, i.e., off-sentence, context. In the following, we discuss annotated
examples (part of the dataset and discarded examples) to understand the
characteristics of target-dependent sentiment in news texts.

We find that in news articles, a key means to express targeted sentiment is to
describe actions performed by the target. This is in contrast, e.g., to
product reviews where more often a target’s feature, e.g., “high resolution”,
or the mention of the target itself, e.g., “the camera is awesome,” express
sentiment. For example, in “The Trump administration has worked tirelessly to
impede a transition to a green economy with actions ranging from opening the
long-protected Arctic National Wildlife Refuge to drilling, […].” the target
(underlined) was assigned negative sentiment due to its actions.

We find sentiment in $\approx 3\%$ of the examples to be strongly reader-
dependent (cf. [1]).444We drew a random sample of 300 examples and concluded
in a two-person discussion that the sentiment in 8 examples could be perceived
differently. In the previous example, the perceived sentiment may, in part,
depend on the reader’s own ideological or political stance, e.g., readers
focusing on economic growth could perceive the described action positively
whereas those concerned with environmental issues would perceive it
negatively.

In some examples, targeted sentiment expressions can be interpreted
differently due to ambiguity. As a consequence, we mostly find such examples
in the discarded examples and thus they are not contained in NewsTSC. While
this can be true for any domain (cf. “polarity ambiguity” in [26]), we think
it is especially characteristic for news articles, which are lengthier than
tweets and reviews, giving authors more ways to refer to non-local statements
and to embed their arguments in larger argumentative structures. For instance,
in “And it is true that even when using similar tactics, President Trump and
President Obama have expressed very different attitudes toward immigration and
espoused different goals.” the target was assigned neutral sentiment. However,
when considering this sentence in the context of its article [36], the
target’s sentiment may be shifted (slightly) negatively.

From a practical perspective, considering more context than only the current
sentence seems to be an effective means to determine otherwise ambiguous
sentiment expressions. By considering a broader context, e.g., the current
sentence and previous sentences, annotators can get a more comprehensive
understanding of the author’s intention and the sentiment the author may have
wanted to communicate. The greater degree of interpretation required to
determine non-explicit sentiment expressions may naturally lead to a higher
degree of subjectivity. Due to our majority-based consolidation method (see
Section 3.2), examples with non-explicit or apparently ambiguous sentiment
expressions are not contained in NewsTSC.

##  4 Experiments and Discussion

We evaluate three TSC methods that define the state-of-the-art on the
established TSC datasets Laptop, Restaurant, and Twitter: AEN-BERT [32], BERT-
SPC [6], and LCF-BERT [38]. Additionally, we test the methods using a domain-
adapted language model, which we created by fine-tuning BERT (base, uncased)
for 3 epochs on 10M English sentences sampled from CCNC (cf. [28]). For all
methods, we test hyperparameter ranges suggested by their respective
authors.555Epochs $\in\\{3,4\\}$; batch size $\in\\{16,32\\}$; learning rate
$\in\\{2e-5,3e-5,5e-5\\}$; label smoothing regularization (LSR) [35]:
$\epsilon\in\\{0,0.2$ }; dropout rate: $0.1$; $\mathcal{L}_{2}$
regularization: $\lambda=10^{-5}$. We use Adam optimization [20], Xaviar
uniform initialization [9], and cross-entropy loss [11]. Where multiple values
for a hyperparameter are given, we test all their combinations in an
exhaustive search. Additionally, we investigate the effects of two common
measures to address class imbalance: weighted cross-entropy loss (using
inverse class frequencies as weights) and oversampling of the training set. Of
the training set, we use 2001 examples for training and 300 for validation.

We use average recall ($AvgRec$) as our primary measure, which was also chosen
as the primary measure in the TSC task of the latest SemEval series, due to
its robustness against class imbalance [29]. We also measure accuracy ($acc$),
macro F1 ($F1_{m}$), and average F1 on positive and negative classes
($F1_{pn}$) to allow comparison to previous works [24].

Table 2 shows that LCF-BERT performs best ($AvgRec=67.3$ using BERT and $69.8$
using our news-adapted language model).666Each row in Table 2 shows the
results of the hyperparameters that performed best on the validation set.
Class-weighted cross-entropy loss helps best to address class imbalance
($AvgRec=69.8$ compared to $67.2$ using oversampling and $64.6$ without any
measure).

LM | Method | AvgRec | acc | $\mathbf{F1_{m}}$ | $\mathbf{F1_{pn}}$
---|---|---|---|---|---
| AEN-BERT | 59.7 | 62.9 | 55.0 | 47.3
base | BERT-SPC | 62.1 | 62.1 | 53.3 | 44.9
| LCF-BERT | 67.3 | 61.3 | 54.4 | 46.5
| AEN-BERT | 59.8 | 62.9 | 54.5 | 46.2
news | BERT-SPC | 66.7 | 63.5 | 55.0 | 45.8
| LCF-BERT | 69.8 | 66.0 | 58.8 | 51.4
Table 2: Experiment results. LM refers to the language model used, where base
is BERT (base, uncased) and news is our fine-tuned BERT model.

Performance in news articles is significantly lower than in established
domains, where the top model (LCF-BERT) yields in our experiments
$AvgRev=78.0$ (Laptop), $82.2$ (Restaurant), and $75.6$ (Twitter). For Laptop
and Restaurant, we used domain-adapted language models [28]. News TSC accuracy
$acc=66.0$ is lower than single-human-level $acc_{h}=72.9$ (see Section 3.3).

We carry out a manual error analysis (up to 30 randomly sampled examples for
each true class). We find target misassociation as the most common error
cause: in 40%, sentences express the predicted sentiment towards a different
target. In 30%, we cannot find any apparent cause. The remaining cases contain
various potential causes, including usage of euphemisms or sayings (12% of
examples with negative sentiment). Infrequently, we find that sentiment is
expressed by rare words or figurative speech, or is reader-dependent (the
latter in 2%, approximately matching the 3% of reader-dependent examples
reported in Section 3.3).

Previous news TSC approaches, mostly dictionary-based, could not reliably
classify implicit or indirect sentiment expressions (see Section 2). In
contrast, our experiments indicate that BERT’s language understanding suffices
to interpret implicitly expressed sentiment correctly (cf. [6, 1, 10]).
NewsTSC does not contain instances in which the broader context defines
sentiment, since human coders could not classify them correctly in the first
place. Our experiments therefore cannot elucidate this particular
characteristic discussed in Section 3.3.

##  5 Conclusion and Future Work

We explore how target-dependent sentiment classification (TSC) can be applied
to political news articles. Our main contributions are as follows: first, we
introduce NewsTSC, a dataset to explore target-dependent sentiment
classification (TSC) in political news articles, consisting of over 3000
manually annotated sentences.

Second, in a qualitative analysis, we find notable differences concerning how
authors express sentiment towards targets as compared to other well-researched
domains of TSC, such as product review or posts on social media. In these
domains, authors tend to explicitly express their opinions. In contrast, in
news articles, we find dominant use of implicit or indirect sentiment
expressions, e.g., by describing actions, which were performed by a given
target, and their consequences. Thus, sentiment expressions may be more
ambiguous, and determining their polarity requires a greater degree of
interpretation.

Third, in a quantitative evaluation, we find that state-of-the-art TSC methods
perform lower on the news domain (average recall $AvgRec=69.8$ using our news-
adapted BERT model, $AvgRec=67.3$ without) than on popular TSC domains
($AvgRec=\left[75.6,82.2\right]$).

We identify multiple future research directions for news TSC. While NewsTSC
contains clear sentiment expressions, it lacks other sentiment types that
occur in real-world news coverage. For example, sentences that express
sentiment more implicitly or ambiguously. To create a labeled TSC dataset that
better reflects real-world news coverage, we suggest to adjust annotation
instructions to raise annotators’ awareness of these sentiment types and
clearly define how they should be labeled. Technically, apparently ambiguous
sentiment expressions might be easier to label when considering a broader
context, e.g., not only the current sentence but also previous sentences.
Considering more context might also help to improve a classifier’s
performance.

We envision to integrate TSC methods into a system that identifies slanted
news coverage [18, 33]. For example, given a set of articles reporting on the
same topic, a system could identify articles that similarly frame the actors
involved in the event. To do so, the system would analyze frequently mentioned
persons’ polarities in each article. Then, it would group articles that
similarly portray these persons.

#### Acknowledgements

The work described in this paper is partially funded by the WIN program of the
Heidelberg Academy of Sciences and Humanities, financed by the Ministry of
Science, Research and the Arts of the State of Baden-Wurttemberg, Germany. The
authors thank the students who participated in the manual annotation as well
as the anonymous reviewers for their valuable comments.

## References

  * [1] Balahur, A., Steinberger, R., Kabadjov, M., Zavarella, V., Van Der Goot, E., Halkia, M., Pouliquen, B., Belyaeva, J.: Sentiment analysis in the news. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), Valletta, Malta (2010)
  * [2] Baum, M.A., Groeling, T.: New Media and the Polarization of American Political Discourse. Political Communication 25(4), 345–365 (2008). https://doi.org/10.1080/10584600802426965
  * [3] Bernhardt, D., Krasa, S., Polborn, M.: Political polarization and the electoral effects of media bias. Journal of Public Economics 92(5), 1092–1104 (2008)
  * [4] Budak, C., Goel, S., Rao, J.M.: Fair and balanced? Quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly 80(S1), 250–271 (2016). https://doi.org/10.1093/poq/nfw007
  * [5] Cicchetti, D.V., Feinstein, A.R.: High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology 43(6), 551–558 (1 1990). https://doi.org/10.1016/0895-4356(90)90159-M, https://linkinghub.elsevier.com/retrieve/pii/089543569090159M
  * [6] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North. pp. 4171–4186. Association for Computational Linguistics, Stroudsburg, PA, USA (2019). https://doi.org/10.18653/v1/N19-1423, http://aclweb.org/anthology/N19-1423
  * [7] Dong, L., Wei, F., Tan, C., Tang, D., Zhou, M., Xu, K.: Adaptive Recursive Neural Network for target-dependent Twitter sentiment classification. In: 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014\. pp. 49–54. Baltimore, MD, USA (2014). https://doi.org/10.3115/v1/p14-2009
  * [8] Gebhard, L., Hamborg, F.: The POLUSA Dataset: 0.9M Political News Articles Balanced by Time and Outlet Popularity. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020. pp. 467–468. ACM, New York, NY, USA (8 2020). https://doi.org/10.1145/3383583.3398567, https://dl.acm.org/doi/10.1145/3383583.3398567
  * [9] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Journal of Machine Learning Research. pp. 249–256 (2010)
  * [10] Godbole, N., Srinivasaiah, M., Skiena, S.: Large-Scale Sentiment Analysis for News and Blogs. In: Proceedings of the International Conference on Weblogs and Social Media (ICWSM). vol. 7, pp. 219–222. Boulder, CO, USA (2007)
  * [11] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016), http://www.deeplearningbook.org
  * [12] Greene, D., Cunningham, P.: Practical Solutions to the Problem of Diagonal Dominance in Kernel Document Clustering. In: Proceedings of the 23rd international conference on Machine learning. pp. 377–384. ACM Press (2006)
  * [13] Groseclose, T., Milyo, J.: A measure of media bias. The Quarterly Journal of Economics 120(4), 1191–1237 (2005). https://doi.org/10.1162/003355305775097542, http://dx.doi.org/10.1162/003355305775097542
  * [14] Hamborg, F.: Media Bias, the Social Sciences, and NLP: Automating Frame Analyses to Identify Bias by Word Choice and Labeling. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop. pp. 79–87. Association for Computational Linguistics, Stroudsburg, PA, USA (2020). https://doi.org/10.18653/v1/2020.acl-srw.12, https://www.aclweb.org/anthology/2020.acl-srw.12
  * [15] Hamborg, F., Donnay, K., Gipp, B.: Automated identification of media bias in news articles: an interdisciplinary literature review. International Journal on Digital Libraries 20(4), 391–415 (12 2019). https://doi.org/10.1007/s00799-018-0261-y, http://link.springer.com/10.1007/s00799-018-0261-y
  * [16] Hamborg, F., Meuschke, N., Breitinger, C., Gipp, B.: news-please: A Generic News Crawler and Extractor. In: Proceedings of the 15th International Symposium of Information Science. pp. 218–223. Verlag Werner Hülsbusch (2017)
  * [17] Hamborg, F., Zhukova, A., Donnay, K., Gipp, B.: Newsalyze: Enabling News Consumers to Understand Media Bias. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020. pp. 455–456. ACM, New York, NY, USA (8 2020). https://doi.org/10.1145/3383583.3398561, https://dl.acm.org/doi/10.1145/3383583.3398561
  * [18] Hamborg, F., Zhukova, A., Gipp, B.: Automated Identification of Media Bias by Word Choice and Labeling in News Articles. In: 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL). pp. 196–205. IEEE, Champaign, IL, USA (6 2019). https://doi.org/10.1109/JCDL.2019.00036, https://ieeexplore.ieee.org/document/8791197/
  * [19] Jiang, L., Yu, M., Zhou, M., Liu, X., Zhao, T.: Target-dependent Twitter sentiment classification. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics. pp. 151–160. Portland, OR, USA (2011)
  * [20] Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: 1412.6980 (12 2014), http://arxiv.org/abs/1412.6980
  * [21] Kiritchenko, S., Zhu, X., Cherry, C., Mohammad, S.: NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). pp. 437–442. Association for Computational Linguistics, Dublin, Ireland (2014). https://doi.org/10.3115/v1/s14-2076
  * [22] Misra, R.: News Category Dataset (2018), https://www.kaggle.com/rmisra/news-category-dataset
  * [23] Nagel, S.: Common Crawl: News Crawl (2016), https://web.archive.org/web/20191118111519/https://commoncrawl.org/2016/10/news-dataset-available/
  * [24] Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., Stoyanov, V.: SemEval-2016 Task 4: Sentiment Analysis in Twitter. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). pp. 1–18. Association for Computational Linguistics, San Diego, CA, USA (2016). https://doi.org/10.18653/v1/S16-1001
  * [25] Nakov, P., Rosenthal, S., Kozareva, Z., Stoyanov, V., Ritter, A., Wilson, T.: SemEval-2013 Task 2: Sentiment Analysis in Twitter. In: Second Joint Conference on Lexical and Computational Semantics (SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). pp. 312–320. Association for Computational Linguistics, Atlanta, GA, USA (2013)
  * [26] Pontiki, M., Galanis, D., Papageorgiou, H., Manandhar, S., Androutsopoulos, I.: SemEval-2015 Task 12: Aspect Based Sentiment Analysis. In: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). pp. 486–495. Association for Computational Linguistics, Stroudsburg, PA, USA (2015). https://doi.org/10.18653/v1/S15-2082, http://aclweb.org/anthology/S15-2082
  * [27] Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.D.: Dataset Shift in Machine Learning. The MIT Press (2009)
  * [28] Rietzler, A., Stabinger, S., Opitz, P., Engl, S.: Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification. arXiv preprint arXiv:1908.11860 (8 2019), http://arxiv.org/abs/1908.11860
  * [29] Rosenthal, S., Farra, N., Nakov, P.: SemEval-2017 Task 4: Sentiment Analysis in Twitter. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pp. 502–518. Association for Computational Linguistics, Vancouver, Canada (2017). https://doi.org/10.18653/v1/s17-2088
  * [30] Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01108 (10 2019), http://arxiv.org/abs/1910.01108
  * [31] Schreier, M.: Qualitative content analysis in practice. SAGE Publications (2012)
  * [32] Song, Y., Wang, J., Jiang, T., Liu, Z., Rao, Y.: Targeted Sentiment Classification with Attentional Encoder Network. In: Artificial Neural Networks and Machine Learning - ICANN 2019: Text and Time Series. pp. 93–103. Springer International Publishing, Cham, US (2019). https://doi.org/10.1007/978-3-030-30490-4_9, http://link.springer.com/10.1007/978-3-030-30490-4_9
  * [33] Spinde, T., Hamborg, F., Donnay, K., Becerra, A., Gipp, B.: Enabling News Consumers to View and Understand Biased News Coverage: A Study on the Perception and Visualization of Media Bias. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020. pp. 389–392. ACM, New York, NY, USA (8 2020). https://doi.org/10.1145/3383583.3398619, https://dl.acm.org/doi/10.1145/3383583.3398619
  * [34] Steinberger, R., Hegele, S., Tanev, H., Della Rocca, L.: Large-scale news entity sentiment analysis. In: RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning. pp. 707–715. Incoma Ltd. Shoumen, Bulgaria (11 2017). https://doi.org/10.26615/978-954-452-049-6_091, http://www.acl-bg.org/proceedings/2017/RANLP2017/pdf/RANLP091.pdf
  * [35] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the Inception Architecture for Computer Vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826. IEEE (2016). https://doi.org/10.1109/CVPR.2016.308, http://arxiv.org/abs/1512.00567
  * [36] Taub, A.: How Liberals Got Lost on the Story of Missing Children at the Border (2018), https://web.archive.org/web/20191120151037/https://www.nytimes.com/2018/05/31/upshot/liberals-immigration-children-border-misinformation.html
  * [37] Weischedel, R., Palmer, M., Marcus, M., Hovy, E., Pradhan, S., Ramshaw, L., Xue, N., Taylor, A., Kaufman, J., Franchini, M., El-Bachouti, M., Belvin, R., Houston, A.: OntoNotes 5.0 (2013), https://web.archive.org/web/20190705173013/https://catalog.ldc.upenn.edu/LDC2013T19
  * [38] Zeng, B., Yang, H., Xu, R., Zhou, W., Han, X.: LCF: A Local Context Focus Mechanism for Aspect-Based Sentiment Classification. Applied Sciences 9(16), 1–22 (8 2019). https://doi.org/10.3390/app9163389, https://www.mdpi.com/2076-3417/9/16/3389
  * [39] Zhaoa, P., Houb, L., Wua, O.: Modeling Sentiment Dependencies with Graph Convolutional Networks for Aspect-level Sentiment Classification. arXiv preprint arXiv:1906.04501 (6 2019), http://arxiv.org/abs/1906.04501


11institutetext: Center for Information and Language Processing, Ludwig
Maximilian University, Munich, Germany
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://www.cis.uni-muenchen.de/ 22institutetext: Digital Philology, Research
Group Data Mining and Machine Learning,
University of Vienna, Austria 33institutetext: Bayerische Staatsbibliothek
München, Digital Library/Munich Digitization Center, Munich, Germany
44institutetext: NLP Expert Center, Data:Lab, Volkswagen AG, Munich, Germany

# Data Centric Domain Adaptation for Historical Text with OCR Errors

Luisa März 112244 Stefan Schweter 33 Nina Poerner 11 Benjamin Roth 22 Hinrich
Schütze 11

###### Abstract

We propose new methods for in-domain and cross-domain Named Entity Recognition
(NER) on historical data for Dutch and French. For the cross-domain case, we
address domain shift by integrating unsupervised in-domain data via
contextualized string embeddings; and OCR errors by injecting synthetic OCR
errors into the source domain and address data centric domain adaptation. We
propose a general approach to imitate OCR errors in arbitrary input data. Our
cross-domain as well as our in-domain results outperform several strong
baselines and establish state-of-the-art results. We publish preprocessed
versions of the French and Dutch Europeana NER corpora.

###### Keywords:

Named Entity Recognition Historical data Flair.

##  1 Introduction

Neural networks achieve good NER accuracy on high-resource domains such as
modern news text or Twitter [2, 4]. But on historical text, NER often performs
poorly. This is due to several challenges: i) Domain shift: Entities in
historical texts can be different from contemporary entities, this makes it
difficult for modern taggers to work with historical data. ii) OCR errors:
historical texts – usually digitized by OCR – contain systematic errors not
found in non-OCR text [14]. In addition these errors can change the surface
form of entities. iii) Lack of annotation: Some historical text is now
available in digitized form, but without labels, and methods are required for
beneficial use of such data [16].

In this paper, we address data centric domain adaptation for NER tagging on
historical French and Dutch data. Following Ramponi and Plank [20], data
centric approaches do not adapt the model but the training data in order to
improve generalization across domains. We address both in-domain and cross-
domain NER. In the cross-domain setup, we use supervised contemporary data and
integrate unsupervised historical data via contextualized embeddings. We
introduce artificial OCR errors into supervised modern data and find a way to
perturb corpora in a general and robust way – independent of language or
linguistic properties.

In the cross-domain setup as well as in-domain, our system outperforms neural
and statistical state-of-the-art methods, achieving 69.3% $F_{1}$ for French
and 63.4% for Dutch. With the in-domain setup, we achieve 77.9% for French and
84.2% for Dutch. If we only consider named entities that contain OCR errors,
our domain-adapted cross-domain tagger even performs better (83.5% French/
46.2% Dutch) than in-domain training (77.1% French/ 43.8% Dutch). Our main
contributions are:

  * $\bullet$

Release of the preprocessed French and Dutch NER
corpora111https://github.com/stefan-it/historic-domain-adaptation-icdar;

  * $\bullet$

Developing synOCR to mimic historical data while exploiting the annotation of
modern data;

  * $\bullet$

Training historical embeddings on a large amount of unlabeled historical data;

  * $\bullet$

Ensembling a NER system that establishes SOTA results for both languages and
scenarios.

##  2 Methods

###  2.1 Architecture

We use the Flair NLP framework [1]. Flair taggers achieve SOTA results on
various benchmarks and are well suited for NER. Secondly, there are powerful
Flair embeddings. They are trained without explicit notion of words and model
words as character sequences depending on their context. These two properties
contribute to making atypical entities - even those with distorted surface -
easier to recognize. In all our experiments, word embeddings are generated by
a character-level RNN and passed to a word-level bidirectional LSTM with a CRF
as the final layer. Depending on the experiment, we concatenate ($\odot$)
additional embeddings and refer to that as ensembling process.

###  2.2 Noise methods

Since digitizing by OCR introduces a lot of noise into the data, we recreate
some of those phenomena in the modern corpora that we use for training. Our
goal is to increase the similarity of historical (OCR’d) and modern (clean)
data. An example drawn from the dutch training corpora can be found in Figure
1. Words that are different from the original text are indicated in bold font.

Generation of synthetic OCR (synOCR) errors This method processes every
sentence by assigning a randomly selected font and a font size between 6 and
11 pt. Batches of 150 sentences are printed to PDF documents and then
converted to PNG images. The images are perturbed using
imgaug222https://github.com/aleju/imgaug with the following steps: (i)
rotation, (ii) Gaussian blur and (iii) white or black pixel dropout. The
resulting image is recognized using tesseract version
0.2.6.333https://github.com/tesseract-ocr/. We re-align the recognized
sentences with the clean annotated corpus to transfer the NER tags. For the
alignment between original and degraded text we select a window of the bitext
and calculate a character-based alignment cost. We then use the Wagner-Fisher
algorithm [27] to obtain the best alignment path through the window and the
lowest possible cost. If the cost is below a threshold, we shift the window to
the mid-point of the discovered path. Otherwise, we iteratively increase the
window size and re-align, until the threshold criterion is met. This procedure
allows us to find an alignment with reasonable time and space resources,
without risking to lose the optimal path in low-quality areas. Finally this
results in an OCR-error enhanced annotated corpus with a range of recognition
quality, from perfectly recognized to fully illegible. We refer to OCR-
corrupted data as _synOCR’d data_.

Generation of synthetic corruptions This method is applied to our modern
corpora, again to introduce noise as we find it in historical data. Similar to
[21], we randomly corrupt 20% of all words by (i) inserting a character or
(ii) removing a character or (iii) transposing two characters. Therefore, we
use the standard alphabet of French/ Dutch. We re-align the corrupted tokens
with the clean annotated tokens while maintaining the sentence boundaries to
transfer the NER tags. Since the corruption method does not break the word
boundaries we can simply map each corrupted word to the original one and
retrieve the corresponding NER tag. We refer to synthetically corrupted data
as _corrupted data_.

Figure 1: Example from the dutch train set. Text in its originial, the
synOCR’d and the corrputed form.

###  2.3 Embeddings

We experiment with various common embeddings and integrate them in our neural
system. Some of them are available in the community and some others we did
train on data described in section 3.1.

Flair embeddings [3] present contextual string embeddings which can be
extracted from a neural language model. Flair embeddings use the internal
states of a trained character language model at token boundaries. They are
contextualized because a word can have different embeddings depending on its
context. These embeddings are also less sensitive to misspellings and rare
words and can be learned on unlabeled corpora. We also use multilingual Flair
embeddings. They were trained on a mix of corpora from different domains (Web,
Wikipedia, Subtitles, News) and languages.

Historical embeddings We train Flair embeddings on large unlabeled historical
corpora from a comparable time period (see Section 3.1.3) and refer to them as
historical embeddings.

BERT embeddings Since BERT embeddings [9] produce state-of-the-art results for
a wide range of NLP tasks, we also experiment with multilingual BERT
embeddings444We use the cased variant from https://huggingface.co/bert-base-
multilingual-cased. BERT embeddings are subword embeddings based on a
bidirectional transformer architecture and can model the context of a word.
For NER on CoNLL-03 [25], BERT embeddings do not perform as well as on other
tasks [9] and we want to examine if this observation holds for a cross-domain
scenario with different data.

FastText embeddings We do also use FastText embeddings [6] which are widely
used in NLP. They can be efficiently trained and address character-level
phenomena. Subwords are used to represent the target word (as a sum of all its
subword embeddings). We use pre-trained FastText embeddings for
French/Dutch555https://fasttext.cc/docs/en/crawl-vectors.html.

Character-level embeddings Due to the OCR errors out-of-vocabulary problems
occur. Lample et al. [15] create character embeddings, passing all characters
in a sentence to a bidirectional LSTM. To obtain word representations, the
forward and backward representations of all the characters of the word from
this LSTM are concatenated. Having the character embedding, every single words
vector can be formed even if it is out-of-vocabulary. Therefore, we do also
compute these embeddings for our experiments.

##  3 Experiments

In the cross-domain setup, we train on modern data (clean or synOCR’d) and
test on historical data (OCR’d). In the in-domain setup, we train and test on
a set of historical data (OCR’d). We do use different combinations of
embeddings and also use our noise methods in the experiments.

###  3.1 Data

We use different data sources for our experiments from which some are openly
available and some historical data come from an in-house project. For an
overview of different properties (domain, labeling, size, language) see Table
1.

####  3.1.1 Annotated historical data

Our annotated historical data comes from the Europeana Newspapers
collection666http://www.europeana-newspapers.eu/, which contains historical
news articles in 12 languages published between 1618 and 1990. Parts of the
German, Dutch and French data were manually annotated with NER tags in IO/IOB
format for PER (person), LOC (location), ORG (organization) by Neudecker [17].
Each NER corpus contains 100 scanned pages (with OCR accuracy over 80%),
amounting to 207K tokens for French and 182K tokens for Dutch.

We preprocess the data as follows. We perform sentence splitting, filter out
metadata, re-tokenize punctuation and convert all annotations to IOB1 format.
We split the data 80/10/10 into train/dev/test. We will make this preprocessed
version available in CoNLL format.

####  3.1.2 Annotated modern data

For the French cross-domain experiments, we use the French WikiNER corpus
[18]. WikiNER is tagged in IOB format with an additional MISC (miscellaneous)
category; we convert the tags to our Europeana format. For better
comparability we downsample (sentence-wise) the corpus from 3.5M to 525K
tokens. Therefore, entire sentences were sampled uniformly at random without
replacement. For Dutch, we use the CoNLL-02 corpus [24], which consists of
four editions of the Belgian Dutch newspaper “De Morgen” from the year 2000.
The data comprises 309K tokens and is annotated for PER, ORG, LOC and MISC. We
convert the tags to our Europeana format.

####  3.1.3 Unlabeled historical data

For historical French, we use “Le Temps”, a journal published between 1861 and
1942 (initially under a different name), a similar time period as the
Europeana Newspapers. The corpus contains 977M tokens and is available from
the National Library of France.777https://www.bnf.fr/fr For historical Dutch,
we use data from an in-house OCR project. The data is from the 19th century
and it consists of 444M tokens. We use the unlabeled historical data to pre-
train historical embeddings (see Section 2.3).

Table 1: Number of tokens per dataset in our experiments. | domain | data | labeled | size
---|---|---|---|---
French | historical | Europeana NER | + | 207K
| modern | WikiNER | + | 525K
| historical | “Le Temps” | - | 977M
Dutch | historical | Europeana NER | + | 182K
| modern | CoNLL-02 | + | 309K
| historical | in-house OCR | - | 444M

###  3.2 Baselines

We experiment with three baselines. (i) The Java
implementation888https://nlp.stanford.edu/software/CRF-NER.html of the
Stanford NER tagger [12]. (ii) A version of Stanford NER published by
Neudecker [17]999https://lab.kb.nl/dataset/europeana-newspapers-ner that was
trained on Europeana. In contrast to our system they trained theirs on the
entire amount of the labeled Europeana corpora with 4-fold cross validation.
(iii) NN base. The neural network (see Section 2.1) with FastText, character
and multilingual Flair embeddings, as recommended in Akbik et al. [1]. For
French, we also list the result reported by Çavdar [8]. Since we do not have
access to their implementation and could not confirm that their data splits
conform to ours, we could not compute the combined $F_{1}$ score or test for
significance.

Table 2: Results ($F_{1}$ scores on French/Dutch Europeana test set) of
training on Europeana French/Dutch training set. _Hist. Embs._ are historical
embeddings. Scores marked with * are significantly lower than _NN base $\odot$
hist. Es_.

French Models | overall | PER | ORG | LOC
---|---|---|---|---
Çavdar |  | 0.68 | 0.37 | 0.68
Stanford NER tagger | 0.662* | 0.569* | 0.335* | 0.753*
Stanford Neudecker | 0.750* | 0.750* | 0.505 | 0.826*
NN base | 0.741* | 0.703* | 0.320* | 0.813*
NN base $\odot$ hist. Embs. | 0.779 | 0.759 | 0.498 | 0.832

Dutch Models | overall | PER | ORG | LOC
---|---|---|---|---
Stanford NER tagger | 0.696* | 0.640* | 0.333* | 0.794*
Stanford Neudecker | 0.623* | 0.700* | 0.253* | 0.702*
NN base | 0.818* | 0.809* | 0.442* | 0.871*
NN base $\odot$ hist. Embs. | 0.842 | 0.833 | 0.480 | 0.891

##  4 Results and Discussion

We evaluate our systems using the CoNLL-2000 evaluation
script101010https://www.clips.uantwerpen.be/conll2000/chunking/conlleval.txt,
with $F_{1}$ score. To check statistical significance we use randomized
testing [28] and results are considered significant if $p<0.05$.

###  4.1 In-domain setup

For both languages we achieve the best results with NN base $\odot$ historical
embeddings. With this setup we can produce $F_{1}$ scores of around 80% for
both languages, which outperforms all three baselines in the overall
performance significantly. The results are presented in Table 2. For French,
the overall $F_{1}$ score as well as the $F_{1}$ for LOC and ORG is best with
NN base $\odot$ historical embeddings. For ORG the pre-trained tagger of
Neudecker [17] works best, which could be due to the gazetteer information
they included and of course due to the fact that they train with the entire
Europeana data. We hypothesize that the category with the most structural
changes over time is ORG. In the military or ecclesiastical context in
particular, there are a number of names that no longer exist (in this form).
For Dutch we observe the best overall performance with NN base $\odot$
historical embeddings except for all entity types.

###  4.2 Cross-domain setup

As shown in Table 3, NN base performs better than the statistical Stanford NER
baseline, which is in line with the observations for the in-domain training.
We experimented with concatenating BERT embeddings to NN base. For both
languages this increases the performance (Table 3, NN base $\odot$ BERT). The
usage of the historical embeddings is also very beneficial for both languages.
We can achieve our best results by using BERT for Dutch and by using
historical embeddings for French. We conclude that the usage of modern pre-
trained language models is crucial for the performance of NER taggers.

We generated synthetic corruptions for the WikiNER/CoNLL corpus. This could
not outperform NN base for both languages. The training on synOCR’d
WikiNER/CoNLL gives slightly worse results than NN base too. The corruption of
the training data without the usage of any embeddings seems to harm
performance drastically, what is in line with the observation of Hamdi et al.
[13]. It is striking that the training on corrupted/synOCR’d Dutch gives
especially bad results for PER compared to French. A look at the Dutch test
set shows that many entities are abbreviated first names (e.g. in A J van
Roozendal) and are often misrecognized what leads to a performance decrease.

Table 3: Results of training on WikiNER/CoNLL corpus. Scores marked with * are
significantly lower than _NN ensemble_.

French Models | overall | PER | ORG | LOC
---|---|---|---|---
Çavdar |  | 0.48 | 0.11 | 0.56
Stanford NER tagger | 0.536* | 0.451* | 0.059* | 0.618*
NN base | 0.646* | 0.636* | 0.096* | 0.721*
NN base $\odot$ BERT | 0.660 | 0.639* | 0.163 | 0.725*
NN base $\odot$ hist.Embs. | 0.672 | 0.661* | 0.015* | 0.748
corrupted WikiNER | 0.627* | 0.635* | 0.085* | 0.710*
synOCR’d WikiNER | 0.619* | 0.590* | 0.078 | 0.710
NN ensemble corrupted | 0.693 | 0.624 | 0.063 | 0.783
NN ensemble synOCR | 0.684 | 0.710 | 0.111 | 0.744

Dutch Models | overall | PER | ORG | LOC
---|---|---|---|---
Stanford NER tagger | 0.371* | 0.217* | 0.083* | 0.564*
NN base | 0.567* | 0.493* | 0.085* | 0.700*
NN base $\odot$ BERT | 0.634 | 0.572 | 0.250 | 0.771
NN base $\odot$ hist. Embs. | 0.632 | 0.568 | 0.084 | 0.738*
corrupted CoNLL | 0.535* | 0.376* | 0.155* | 0.717*
synOCR’d CoNLL | 0.521* | 0.327* | 0.061* | 0.721*
NN ensemble corrupted | 0.606* | 0.439* | 0.158 | 0.799
NN ensemble synOCR | 0.614 | 0.481 | 0.157 | 0.775

For French the combination of NN base and historical embeddings, trained on
corrupted data or on synOCR’d (NN ensemble corrupted/ NN ensemble synOCR’d)
gives the best results and outperforms all other systems. For Dutch NN
ensemble corrupted and NN ensemble synOCR give slightly worse results than NN
base $\odot$ BERT and NN base $\odot$ historical embeddings, but performs
better than the tagger trained on synOCR’d or corrupted data only (Table 3, NN
ensemble).

####  4.2.1 Ablation study

We analyze our results and examine the composition of NN ensemble synOCR more
closely (since the results for NN ensemble corrupted are very similar we
perform the analysis for NN ensemble synOCR as a representative for both NN
ensemble).

The ablation study (see Table 4) shows that NN ensemble benefits from
different information in combination. For French NN ensemble gives the best
results only for PER. The overall performance increases if we do not use
character level embeddings. There is a big performance loss if we omit the
historical embeddings. If we do not train on synOCR’d data the performance
decreases.

Table 4: Ablation study. Results of training on the clean and the synOCR’d
WikiNER/CoNLL corpus.

French Models | overall | PER | ORG | LOC
---|---|---|---|---
NN ensemble synOCR | 0.684 | 0.710 | 0.011 | 0.744
\- char | 0.693 | 0.681 | 0.078 | 0.758
\- word | 0.686 | 0.664 | 0.080 | 0.756
\- hist. Embs. | 0.619 | 0.590 | 0.078 | 0.710
\- synOCR’d data | 0.672 | 0.661 | 0.015 | 0.748

Dutch Models | overall | PER | ORG | LOC
---|---|---|---|---
NN ensemble synOCR | 0.614 | 0.382 | 0.157 | 0.775
\- char | 0.600 | 0.404 | 0.119 | 0.780
\- word | 0.584 | 0.430 | 0.102 | 0.745
\- hist. Embs. | 0.521 | 0.327 | 0.061 | 0.721
\- synOCR’d data | 0.632 | 0.568 | 0.084 | 0.738

For Dutch we can observe these facts even more clearly. If we do not train on
synOCR’d data the $F_{1}$ score even increases. If omitting the historical
embeddings we loose performance as well.

To find out why our implementation of the assumption – synOCR increases the
similarity of the data and improves results – does not have the expected
effect, we analyze the test sets. It shows, that only 10% of the French and 6%
of the Dutch entities contain OCR errors. Therefore the wrong predictions are
mostly not due to the OCR errors, but due to the inherent difficulty of
recognizing entities cross-domain. This also explains why synthetic
noisyfication does not consistently improve the system. In addition there are
some illegible lines in the synOCR’d corpora consisting of dashes and
metasymbols, what is not similar to real OCR errors.

Figure 2: Example sentence from the French test set.

To verify our assumption we also compare the different systems only on the
entities with OCR errors. Here NN ensemble outperforms both of the cross-
domain baselines (Table 5, Stanford NER tagger, NN base cross-domain).
Compared to the French results Dutch is a lot worse. A look at the entities
shows that in the Dutch test set there are many hyphenated words where both
word parts are labeled. However, if looking at the parts of the word
individually, a clear assignment to an entity type cannot be made, which leads
to difficulties with tagging. Though it is plausible that NN ensemble can
capture specific phenomena in the historical data better, since the difference
between the domains is reduced by the synthetic noisyfication and the
historical embeddings. The example in Figure 2 drawn from the test set shows,
that NN ensemble can handle noisy entities well in contrast to e.g. the
Stanford NER tagger. Thus in a scenario with many OCR errors the NN ensemble
performs well.

Table 5: Results on entities with OCR errors in the French/Dutch test set. Scores marked with * are significantly lower than _NN ensemble_. Models | French | Dutch
---|---|---
Stanford NER tagger | 0.661* | 0.207*
NN base in-domain | 0.771 | 0.438
NN base cross-domain | 0.783 | 0.200*
NN ensemble synOCR | 0.835 | 0.462

##  5 Related Work

There is some research on using natural language processing for improving OCR
for historical documents [5, 26] and also on NER for historical documents
[11]. In the latter - a shared task for Named Entity Processing in historical
documents - Ehrmann et al. find that OCR noise drastically harms systems
performance. Like us several participants (e.g. [7], [23]) also use language
models that were trained on historical data to boost the performance of NER
taggers. Schweter and Baiter [22] explore NER for historical German data in a
cross-domain setting. Like us, they train a language model on unannotated in-
domain data and integrate it into a NER tagger. In addition to the above
mentioned work, we employ “OCR noisyfication” (Section 2.2) and examine the
influence of different pretrained embeddings systematically. Çavdar [8]
addresses NER and relation extraction on the French Europeana Newspaper
corpus. Ehrmann et al. [10] investigate the performance of NER systems on
Swiss historical Newspapers and show that historical texts are a great
challenge compared to contemporary texts. They find that the LOC class
entities causes the most difficulties in the recognition of named entities.
The recent work of Hamdi et al. [13] investigates the impact of OCR errors on
NER. To do so, they also perturb modern corpora synthetically with different
degrees of error rates. They experiment with Spanish, Dutch and English. Like
us they perturb the Dutch CoNLL corpus and train NER taggers on that data.
Unlike us they do also train on a subset of the perturbed corpus. We test on a
subset of the Dutch Europeana corpus. Hamdi et al. [13] show that neural
taggers perform better compared to other taggers like the Stanford NER tagger
and they also prove that performance decreases drastically if the OCR error
rate increases. Piktus et al. [19] learn misspelling-oblivious FastText
embeddings from synthetic misspellings generated by an error model for part-
of-speech tagging. We use a similar corruption method, but we also use synOCR
and historical embeddings for NER.

##  6 Conclusion

We proposed new methods for in-domain and cross-domain Named Entity
Recognition (NER) on historical data and addressed data centric domain
adaptation. For the cross-domain case, we handle domain shift by integrating
non-annotated historical data via contextualized string embeddings; and OCR
errors by injecting synthetic OCR errors into the modern data. This allowed us
to get good results when labeled historical data is not available and the
historical data is noisy. For training on contemporary corpora and testing on
historical corpora we achieve new state-of-the-art results of 69.3% on French
and 63.4% on Dutch. For the in-domain case we obtain state-of-the-art results
of 77.9% for French and 84.2% for Dutch. There is an increasing demand for
advancing the digitization of the world’s cultural heritage. High quality
digitized historical data, with reliable meta information, will facilitate
convenient access and search capabilities, and allow for extensive analysis,
for example of historical linguistic or social phenomena. Since named entity
recognition is one of the most fundamental labeling tasks, it would be
desirable that advances in this area translate to other labeling tasks in
processing of historical data as well.

##  7 Acknowledgement

This work was funded by the European Research Council (ERC #740516).

## References

  * [1] Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: FLAIR: An easy-to-use framework for state-of-the-art NLP. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). pp. 54–59. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019), https://www.aclweb.org/anthology/N19-4010
  * [2] Akbik, A., Bergmann, T., Vollgraf, R.: Pooled contextualized embeddings for named entity recognition. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 724–728. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019)
  * [3] Akbik, A., Blythe, D., Vollgraf, R.: Contextual string embeddings for sequence labeling. In: COLING 2018, 27th International Conference on Computational Linguistics. pp. 1638–1649 (2018)
  * [4] Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., Auli, M.: Cloze-driven pretraining of self-attention networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 5359–5368. Association for Computational Linguistics, Hong Kong, China (Nov 2019), https://www.aclweb.org/anthology/D19-1539
  * [5] Berg-Kirkpatrick, T., Durrett, G., Klein, D.: Unsupervised transcription of historical documents. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 207–217. Association for Computational Linguistics, Sofia, Bulgaria (Aug 2013), https://www.aclweb.org/anthology/P13-1021
  * [6] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, 135–146 (2017), https://www.aclweb.org/anthology/Q17-1010
  * [7] Boros, E., Linhares Pontes, E., Cabrera-Diego, L.A., Hamdi, A., Moreno, J.G., Sidère, N., Doucet, A.: Robust Named Entity Recognition and Linking on Historical Multilingual Documents. In: Conference and Labs of the Evaluation Forum (CLEF 2020). Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, vol. 2696, pp. 1–17. CEUR-WS Working Notes, Thessaloniki, Greece (Sep 2020), https://hal.archives-ouvertes.fr/hal-03026969
  * [8] Çavdar, M.: Distant supervision for french relation extraction (2017)
  * [9] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019), https://www.aclweb.org/anthology/N19-1423
  * [10] Ehrmann, M., Colavizza, G., Rochat, Y., Kaplan, F.: Diachronic evaluation of ner systems on old newspapers. In: KONVENS (2016)
  * [11] Ehrmann, M., Romanello, M., Flückiger, A., Clematide, S.: Extended Overview of CLEF HIPE 2020: Named Entity Processing on Historical Newspapers. Zenodo (Oct 2020)
  * [12] Finkel, J.R., Grenager, T., Manning, C.: Incorporating non-local information into information extraction systems by Gibbs sampling. In: Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05). pp. 363–370. Association for Computational Linguistics, Ann Arbor, Michigan (Jun 2005), https://www.aclweb.org/anthology/P05-1045
  * [13] Hamdi, A., Jean-Caurant, A., Sidère, N., Coustaty, M., Doucet, A.: Assessing and minimizing the impact of ocr quality on named entity recognition. In: Hall, M., Merčun, T., Risse, T., Duchateau, F. (eds.) Digital Libraries for Open Knowledge. pp. 87–101. Springer International Publishing, Cham (2020)
  * [14] Jean-Caurant, A., Tamani, N., Courboulay, V., Burie, J.: Lexicographical-based order for post-ocr correction of named entities. In: 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan, November 9-15, 2017. pp. 1192–1197. IEEE (2017). https://doi.org/10.1109/ICDAR.2017.197, https://doi.org/10.1109/ICDAR.2017.197
  * [15] Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 260–270. Association for Computational Linguistics, San Diego, California (Jun 2016), https://www.aclweb.org/anthology/N16-1030
  * [16] Martinek, J., Lenc, L., Král, P., Nicolaou, A., Christlein, V.: Hybrid training data for historical text ocr. pp. 565–570 (09 2019). https://doi.org/10.1109/ICDAR.2019.00096
  * [17] Neudecker, C.: An open corpus for named entity recognition in historic newspapers. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). pp. 4348–4352. European Language Resources Association (ELRA), Portorož, Slovenia (May 2016), https://www.aclweb.org/anthology/L16-1689
  * [18] Nothman, J., Ringland, N., Radford, W., Murphy, T., R. Curran, J.: Learning multilingual named entity recognition from wikipedia. Artificial Intelligence 194, 151–175 (01 2013)
  * [19] Piktus, A., Edizel, N.B., Bojanowski, P., Grave, E., Ferreira, R., Silvestri, F.: Misspelling oblivious word embeddings. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 3226–3234. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019), https://www.aclweb.org/anthology/N19-1326
  * [20] Ramponi, A., Plank, B.: Neural unsupervised domain adaptation in nlp—a survey (2020)
  * [21] Schick, T., Schütze, H.: Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. CoRR abs/1904.06707 (2019), http://arxiv.org/abs/1904.06707
  * [22] Schweter, S., Baiter, J.: Towards robust named entity recognition for historic German. In: Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019). pp. 96–103. Association for Computational Linguistics, Florence, Italy (Aug 2019), https://www.aclweb.org/anthology/W19-4312
  * [23] Schweter, S., März, L.: Triple E - effective ensembling of embeddings and language models for NER of historical german. In: Cappellato, L., Eickhoff, C., Ferro, N., Névéol, A. (eds.) Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020. CEUR Workshop Proceedings, vol. 2696. CEUR-WS.org (2020), http://ceur-ws.org/Vol-2696/paper_173.pdf
  * [24] Tjong Kim Sang, E.F.: Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In: COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002) (2002), https://www.aclweb.org/anthology/W02-2024
  * [25] Tjong Kim Sang, E.F., De Meulder, F.: Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003. pp. 142–147 (2003), https://www.aclweb.org/anthology/W03-0419
  * [26] Vobl, T., Gotscharek, A., Reffle, U., Ringlstetter, C., Schulz, K.U.: Pocoto - an open source system for efficient interactive postcorrection of ocred historical texts. In: Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage. pp. 57–61. DATeCH ’14, ACM, New York, NY, USA (2014), http://doi.acm.org/10.1145/2595188.2595197
  * [27] Wagner, R.A., Fischer, M.J.: The string-to-string correction problem. J. ACM 21(1), 168–173 (Jan 1974), https://doi.org/10.1145/321796.321811
  * [28] Yeh, A.: More accurate tests for the statistical significance of result differences. In: COLING 2000 Volume 2: The 18th International Conference on Computational Linguistics (2000), https://www.aclweb.org/anthology/C00-2137

##  Appendix 0.A Appendix

Detailed information about experiments and data

The computing infrastructure we use for all our experiments is one GeForce GTX
1080Ti GPU with an average runtime of 12 hours per experiment. For the French
and Dutch baseline model NN base we count 15,895,683 parameters each. For the
French NN ensemble model there are 88,264,777 parameters and 96,895,161
parameters for the Dutch NN ensemble.
The Europeana Newspaper Corpus is split 80/10/10 into train/dev/test. The
downsampled French WikiNER corpus is split 70/15/15 into train/dev/test and
the Dutch CoNLL-02 corpus is already split in its original version. A link to
the downloadable version of the French and Dutch Europeana data will be
included in the camera-ready version of the paper.

Table 6: Number of tokens for each datasplit. Dataset | train | dev | test
---|---|---|---
French Europeana | 167,723 | 18,841 | 20,346
Dutch Europeana | 147,822 | 16,391 | 18,218
French WikiNER | 411,687 | 88,410 | 88,509
Dutch ConNLL-02 | 202,930 | 68,994 | 37,761