# Neural Modeling for Named Entities and Morphology (NEMO2)

Dan Bareket1,2 Reut Tsarfaty1
1Bar Ilan University, Ramat-Gan, Israel
2Open Media and Information Lab (OMILab), The Open University of Israel,
Israel
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Named Entity Recognition (NER) is a fundamental NLP task, commonly formulated
as classification over a sequence of tokens. Morphologically-Rich Languages
(MRLs) pose a challenge to this basic formulation, as the boundaries of Named
Entities do not necessarily coincide with token boundaries, rather, they
respect morphological boundaries. To address NER in MRLs we then need to
answer two fundamental questions, namely, what are the basic units to be
labeled, and how can these units be detected and classified in realistic
settings, i.e., where no gold morphology is available. We empirically
investigate these questions on a novel NER benchmark, with parallel token-
level and morpheme-level NER annotations, which we develop for Modern Hebrew,
a morphologically rich-and-ambiguous language. Our results show that
explicitly modeling morphological boundaries leads to improved NER
performance, and that a novel hybrid architecture, in which NER precedes and
prunes morphological decomposition, greatly outperforms the standard pipeline,
where morphological decomposition strictly precedes NER, setting a new
performance bar for both Hebrew NER and Hebrew morphological decomposition
tasks.

##  1 Introduction

Named Entity Recognition (NER) is a fundamental task in the area of
Information Extraction (IE), in which mentions of Named Entities (NE) are
extracted and classified in naturally-occurring texts. This task is most
commonly formulated as a sequence labeling task, where extraction takes the
form of assigning each input token with a label that marks the boundaries of
the NE (e.g., B,I,O), and classification takes the form of assigning labels to
indicate entity type (Per, Org, Loc, etc.).

Despite a common initial impression from latest NER performance, brought about
by neural models on the main English NER benchmarks — CoNLL 2003 Tjong Kim
Sang (2003) and OntoNotes Weischedel et al. (2013) — the NER task in real-
world settings is far from solved. Specifically, NER performance is shown to
greatly diminish when moving to other domains Luan et al. (2018); Song et al.
(2018), when addressing the long tail of rare, unseen, and new user-generated
entities Derczynski et al. (2017), and when handling languages with
fundamentally different structure than English. In particular, there is no
readily available and empirically verified neural modeling strategy for Neural
NER in those languages with complex word-internal structure, also known as
morphologically-rich languages.

Morphologically-rich languages (MRL) Tsarfaty et al. (2010); Seddah et al.
(2013); Tsarfaty et al. (2020) are languages in which substantial information
concerning the arrangement of words into phrases and the relations between
them is expressed at word level, rather than in a fixed word-order or a rigid
structure. The extended amount of information expressed at word-level and the
morpho-phonological processes creating these words result in high token-
internal complexity, which poses serious challenges to the basic formulation
of NER as classification of raw, space-delimited, tokens. Specifically, while
NER in English is formulated as the sequence labeling of space-delimited
tokens, in MRLs a single token may include multiple meaning-bearing units,
henceforth morphemes, only some of which are relevant for the entity mention
at hand.

In this paper we formulate two questions concerning neural modelling
strategies for NER in MRLs, namely: (i) what should be the granularity of the
units to be labeled? Space-delimited tokens or finer-grain morphological
segments? and, (ii) how can we effectively encode, and accurately detect, the
morphological segments that are relevant to NER, and specifically in realistic
settings, when gold morphological boundaries are not available?

To empirically investigate these questions we develop a novel parallel
benchmark, containing parallel token-level and morpheme-level NER annotations
for texts in Modern Hebrew — a morphologically rich and morphologically
ambiguous language, which is known to be notoriously hard to parse More et al.
(2019); Tsarfaty et al. (2019).

Our results show that morpheme-based NER is superior to token-based NER, which
encourages a segmentation-first pipeline. At the same time, we demonstrate
that token-based NER improves morphological segmentation in realistic
scenarios, encouraging a NER-first pipeline. While these two findings may
appear contradictory, we aim here to offer a climax; a hybrid architecture
where the token-based NER predictions precede and prune the space of
morphological decomposition options, while the actual morpheme-based NER takes
place only after the morphological decomposition. We empirically show that the
hybrid architecture we propose outperforms all token-based and morpheme-based
model variants of Hebrew NER on our benchmark, and it further outperforms all
previously reported results on Hebrew NER and morphological decomposition. Our
error analysis further demonstrates that morpheme-based models better
generalize, that is, they contribute to recognizing the long tail of entities
unseen during training (out-of-vocabulary, OOV), in particular those unseen
entities that turn out to be composed of previously seen morphemes.

The contribution of this paper is thus manifold. First, we define key
architectural questions for Neural NER modeling in MRLs and chart the space of
modeling options. Second, we deliver a new and novel parallel benchmark that
allows one to empirically compare and contrast the morpheme vs. token modeling
strategies. Third, we show consistent advantages for morpheme-based NER,
demonstrating the importance of morphologically-aware modeling. Next we
present a novel hybrid architecture which demonstrates an even further
improved performance on both NER and morphological decomposition tasks. Our
results for Hebrew present a new bar on these tasks, outperforming the
reported state-of-the-art results on various benchmarks.111Data & code:
https://github.com/OnlpLab/NEMO

##  2 Research Questions: NER for MRLs

In MRLs, words are internally complex, and word boundaries do not generally
coincide with the boundaries of more basic meaning-bearing units. This fact
has critical ramifications for sequence labeling tasks in MRLs in general, and
for NER in MRLs in particular. Consider, for instance, the three-token Hebrew
phrase in (1):222Glossing conventions are in accord with the Leipzig Glossing
Rules Comrie et al. (2008).

  * (1)

.tsnw mt’ylnd lsyn
tasnu mithailand lesin
flew.1PL from-Thailand to-China
‘we flew from Thailand to China’

It is clear that t’ylnd/thailand (_Thailand_) and syn/sin (_China_) are NEs,
and in English, each NE is its own token. In the Hebrew phrase though, neither
NE constitutes a single token. In either case, the NE occupies only one of two
morphemes in the token, the other being a case-assigning preposition. This
simple example demonstrates an extremely frequent phenomenon in MRLs such as
Hebrew, Arabic or Turkish, that the adequate boundaries for NEs do not
coincide with token boundaries, and tokens must be segmented in order to
obtain accurate NE boundaries.333We use the term morphological segmentation
(or segmentation) to refer to splitting raw tokens into morphological
segments, each carrying a single Part-Of-Speech tag. That is, we segment away
prepositions, determiners, subordination markers and multiple kinds of
pronominal clitics, that attach to their hosts via complex morpho-phonological
processes. Throughout this work, we use the terms morphological segment,
morpheme, or segment interchangeably.

The segmentation of tokens and the identification of adequate NE boundaries is
however far from trivial, due to complex morpho-phonological and orthographic
processes in some MRLs Vania et al. (2018); Klein and Tsarfaty (2020). This
means that the morphemes that compose NEs are not necessarily transparent in
the character sequence of the raw tokens. Consider for example phrase (2):

  * (2)

hmrw.S lbyt hlbn
hamerotz labayit halavan
the-race to-house.DEF the-white
‘the race to the White House’

Here, the full form of the NE hbyt hlbn / _habayit halavan_ (_the White
House_), is not present in the utterances, only the sub-string byt hlbn /
_bayit halavan_ (_(the) White House_) is present in (2) — due to phonetic and
orthographic processes suppressing the definite article h/_ha_ in certain
environments. In this and many other cases, it is not only that NE boundaries
do not coincide with token boundaries, they do not coincide with characters or
sub-strings of the token either. This calls for accessing the more basic
meaning-bearing units of the token, that is, to decompose the tokens into
_morphemes_.

Unfortunately though, the morphological decomposition of surface tokens may be
very challenging due to extreme morphological ambiguity. The sequence of
morphemes composing a token is not always directly recoverable from its
character sequence, and is not known in advance.444This ambiguity gets
magnified by the fact that Semitic languages that use abjads, like Hebrew and
Arabic, lack capitalization altogether and suppress all vowels (diacritics).
This means that for every raw space-delimited token, there are many
conceivable readings which impose different segmentations, yielding different
sets of potential NE boundaries. Consider for example the token lbny (_lbny_)
in different contexts:

  * (3)

(a) h/srh lbny
_hasara livni_
the-minister [Livni]PER
‘Minister [Livni]PER’
(b) lbny gn.S
le-beny gantz
for-[Benny Gantz]PER
‘for [Benny Gantz]PER’
(c) lbny hyqr
li-bni hayakar
for-son.POSS.1SG the-dear
‘for my dear son’
(d) lbny xymr
livney kheymar
brick.CS clay
‘clay bricks’

In (3a) the token lbny is completely consumed as a labeled NE. In (3b) lbny is
only partly consumed by an NE, and in (3c) and (3d) the token is entirely out
of an NE context. In (3c) the token is composed of several morphemes, and in
(3d) it consists of a single morpheme. These are only some of the possible
decompositions of this surface token, other alternatives may still be
available. As shown by Goldberg and Tsarfaty (2008); Green and Manning (2010);
Seeker and Çetinoğlu (2015); Habash and Rambow (2005); More et al. (2019), and
others, the correct morphological decomposition becomes apparent only in the
larger (syntactic or semantic) context. The challenge, in a nutshell, is as
follows: in order to detect accurately NE boundaries, we need to segment the
raw token first, however, in order to segment tokens correctly, we need to
know the greater semantic content, including, e.g., the participating
entities. How can we break out of this apparent loop?

Finally, MRLs are often characterized by an extremely sparse lexicon,
consisting of a long-tail of out-of-vocabulary (OOV) entities unseen during
training Czarnowska et al. (2019). Even in cases where all morphemes are
present in the training data, morphological compositions of seen morphemes may
yield tokens and entities which were unseen during training. Take for example
the utterance in (4), which the reader may inspect as familiar:

  * (4)

.tsnw msyn lt’ylnd
tasnu misin lethailand
flew.1PL from-China to-Thailand
’we flew from China to Thailand’

Example (4) is in fact example (1) with a switched flight direction. This
subtle change creates two new surface tokens msyn, lt’ylnd which might not
have been seen during training, even if example (1) had been observed.
Morphological compositions of an entity with prepositions, conjunctions,
definite markers, possessive clitics and more, cause mentions of seen entities
to have unfamiliar surface forms, which often fail to be accurately detected
and analyzed.

Given the aforementioned complexities, in order to solve NER for MRLs we ought
to answer the following fundamental modeling questions:
Q1. Units: What are the discrete units upon which we need to set NE boundaries
in MRLs? Are they tokens? characters? morphemes? a representation containing
multiple levels of granularity?
Q2. Architecture: When employing morphemes in NER, the classical approach is
“segmentation-first". However, segmentation errors are detrimental and
downstream NER cannot recover from them. How is it best to set up the pipeline
so that segmentation and NER could interact?
Q3. Generalization: How do the different modeling choices affect NER
generalization in MRLs? How can we address the long tail of OOV NEs in MRLs?
Which modeling strategy best handles pseudo-OOV entities that result from a
previously unseen composition of already seen morphemes?

##  3 Formalizing NER for MRLs

To answer the aforementioned questions, we chart and formalize the space of
modeling options for neural NER in MRLs. We cast NER as a Sequence Labelling
task and formalize it as $f:\mathcal{X}\rightarrow\mathcal{Y}$, where
$x\in\mathcal{X}$ is a sequence $x_{1},...,x_{n}$ of $n$ discrete strings from
some vocabulary $x_{i}\in\Sigma$, and $y\in\mathcal{Y}$ is a sequence
$y_{1},..,y_{n}$ of the same length, where $y_{i}\in Labels$, and $Labels$ is
a finite set of labels composed of the BIOSE tags (a.k.a., BIOLU as described
in Ratinov and Roth (2009)). Every non-O label is also enriched with an entity
type label. Our list of types is presented in Table 2.

###  3.1 Token-Based or Morpheme-Based?

Our first modeling question concerns the discrete units upon which to set the
NE boundaries. That is, what is the formal definition of the input vocabulary
$\Sigma$ for the sequence labeling task?

The simplest scenario, adopted in most NER studies, assumes token-based input,
where each token admits a single label — hence token-single:

$\textit{NER}_{\textit{token-single}}:\mathcal{W}\rightarrow\mathcal{L}$

Here, $\mathcal{W}=\\{w^{*}|w\in\Sigma\\}$ is the set of all possible token
sequences in the language and $\mathcal{L}=\\{l^{*}|l\in Labels\\}$ is the set
of all possible label sequences over the label set defined above. Each token
gets assigned a single label, so the input and output sequences are of the
same length. The drawback of this scenario is that since the input for token-
single incorporates no morphological boundaries, the exact boundaries of the
NEs remain underspecified. This case is exemplified at the top row of Table 1.

There is another conceivable scenario, where the input is again the sequence
of space-delimited tokens, and the output consists of complex labels
(henceforth multi-labels) reflecting, for each token, the labels of its
constituent morphemes; henceforth, a token-multi scenario:

$\textit{NER}_{\textit{token-multi}}:\mathcal{W}\rightarrow\mathcal{L}^{*}$

Here, $\mathcal{W}=\\{w^{*}|w\in\Sigma\\}$ is the set of sequences of tokens
as in token-single. Each token is assigned a multi-label, i.e., a sequence
($l^{*}\in\mathcal{L}$) which indicates the labels of the token’s morphemes in
order. The output is a sequence of such multi-labels, one multi-label per
token. This variant incorporates morphological information concerning the
number and order of labeled morphemes, but lacks the precise morphological
boundaries. This is illustrated at the middle of Table 1. A downstream
application may require (possibly noisy) heuristics to determine the precise
NE boundaries of each individual label in the multi-label for an input token.

Another possible scenario is a morpheme-based scenario, assigning a label
$l\in L$ for each segment:

$\textit{NER}_{morph}:\mathcal{M}\rightarrow\mathcal{L}$

Here, $\mathcal{M}=\\{m^{*}|m\in\textit{Morphemes}\\}$ is the set of sequences
of morphological segments in the language, and
$\mathcal{L}=\\{l^{*}|l\in\textit{Labels}\\}$ is the set of label sequences as
defined above. The upshot of this scenario is that NE boundaries are precise.
An example is given in the bottom row of Table 1. But, since each token may
contain many meaningful morphological segments, the length of the token
sequence is not the same as the length of morphological segments to be
labeled, and the model assumes prior morphological segmentation — which in
realistic scenarios is not necessarily available.

Nickname | Input | Lit | Output
---|---|---|---
token-single |  hmrw.S | the-race | O
|  lbyt | to-house.DEF | B_ORG
|  hlbn | the-white | E_ORG
token-multi |  hmrw.S | the-race | O + O
|  lbyt | to-house.DEF | O + B_ORG + I_ORG
|  hlbn | the-white | I_ORG + E_ORG
morpheme |  h | the | O
|  mrw.S | race | O
|  l | to | O
|  h | the | B_ORG
|  byt | house | I_ORG
|  h | the | I_ORG
|  lbn | white | E-ORG

Table 1:  Input/output for token-single, token-multi and morpheme models for
example (2) in Sec. 2.

###  3.2 Realistic Morphological Decomposition

A major caveat with morpheme-based modeling strategies is that they often
assume an ideal scenario of gold morphological decomposition of the space-
delimited tokens into morphological segments (cf. Nivre et al. (2007); Pradhan
et al. (2012)). But in reality, gold morphological decomposition is not known
in advance, it has to be predicted automatically, and prediction errors may
propagate to contaminate the downstream task.

Our second modeling question therefore concerns the interaction between the
morphological decomposition and the NER tasks: how would it be best to set up
the pipeline so that the prediction of the two tasks can interact?

Figure 1:  Lattice for a partial list of analyses of the Hebrew tokens lbyt
hlbn corresponding to Table 1. Bold nodes are token boundaries. Light nodes
are segment boundaries. Every path through the lattice is a single
morphological analysis. The bold path is a single NE.

To answer this, we define _morphological decomposition_ as consisting of two
subtasks: morphological analysis (MA) and morphological disambiguation (MD).
We view sentence-based MA as:

$MA:\mathcal{W}\rightarrow\mathcal{P(M)}$

Here $\mathcal{W}=\\{w^{*}|w\in\Sigma\\}$ is the set of possible token
sequences as before, $\mathcal{M}=\\{m^{*}|m\in Morphemes\\}$ is the set of
possible morpheme sequences, and $\mathcal{P(M)}$ is the set of subsets of
$\mathcal{M}$. The role of $MA$ is then to assign a token sequence
$w\in\mathcal{W}$ with all of its possible morphological decomposition
options. We represent this set of alternatives in a dense structure that we
call a _lattice_ (exemplified in Figure 1). MD is the task of picking the
single correct morphological path $M\in\mathcal{M}$ through the MA lattice of
a given sentence:

$MD:\mathcal{P(M)}\rightarrow\mathcal{M}$

Now, assume $x\in\mathcal{W}$ is a surface sentence in the language, with its
morphological decomposition initially unknown and underspecified. In a
Standard pipeline, MA strictly precedes MD:

$MD_{\textit{Standard}}:M=MD(MA(x))$

The main problem here is that MD errors may propagate to contaminate the NER
output.

We propose a novel Hybrid alternative, in which we inject a task-specific
signal, in this case NER,555We can do this for any sequence labeling task in
MRLs. to constrain the search for $M$ through the lattice:

$MD_{\textit{Hybrid}}:M=MD(MA(x)\restriction\textit{NER}_{\textit{token}}(x))$

Here, the restriction $MA(x)\restriction NER(x)$ indicates pruning the lattice
structure $MA(x)$ to contain only MD options that are compatible with the
token-based NER predictions, and only then apply $MD$ to the pruned lattice.

Both $MD_{\textit{Standard}}$ and $MD_{\textit{Hybrid}}$ are disambiguation
architectures that result in a morpheme sequence $M\in\mathcal{M}$. The latter
benefits from the NER signal, while the former doesn’t. The sequence
$M\in\mathcal{M}$ can be used in one of two ways. We can use $M$ as _input_ to
a morpheme model to output morpheme labels. Or, we can rely on the output of
the token-multi model and align the token’s multi-label with the segments in
$M$.

In what follows, we want to empirically assess the effect of different
modeling choices (token-single, token-multi, morpheme) and disambiguation
architectures (Standard, Hybrid) on the performance of NER in MRLs. To this
end, we need a corpus that allows training and evaluating NER at both token
and morpheme-level granularity.

##  4 The Data: A Novel NER Corpus

This work empirically investigates NER modeling strategies in Hebrew, a
Semitic language known for its complex and highly ambiguous morphology. Ben-
Mordecai (2005), the only previous work on Hebrew NER to date, annotated
space-delimited tokens, basing their guidelines on the CoNLL 2003 shared task
Chinchor et al. (1999).

Popular Arabic NER corpora also label space-delimited tokens (ANERcorp
Benajiba et al. (2007), AQMAR Mohit et al. (2012), TWEETS Darwish (2013)),
with the exception of the Arabic portion of OntoNotes Weischedel et al. (2013)
and ACE LDC (2008) which annotate NER labels on gold morphologically pre-
segmented texts. However, these works do not provide a comprehensive analysis
on the performance gaps between morpheme-based and token-based scenarios.

In agglutinative languages as Turkish, token segmentation is always performed
before NER (Tür et al. (2003); Küçük and Can (2019), re-enforcing the need to
contrast the token-based scenario, widely adopted for Semitic languages, with
the morpheme-based scenarios in other MRLs.

Our first contribution is thus a parallel corpus for Hebrew NER, one version
consists of gold-labeled tokens and the other consists of gold-labeled
morphemes, for the same text. For this, we performed gold NE annotation of the
Hebrew Treebank Sima’an et al. (2001), based on the 6,143 morpho-syntactically
analyzed sentences of the HAARETZ corpus, to create both token-level and
morpheme-level variants, as illustrated at the topmost and lowest rows of
Table 1, respectively.

#### Annotation Scheme

We started off with the guidelines of Ben-Mordecai (2005), from which we
deviate in three main ways. First, we label NE boundaries and their types on
sequences of morphemes, in addition to the space-delimited token
annotations.666A single NE is always continuous. Token-morpheme discrepancies
do not lead to discontinuous NEs. Secondly, we use the finer-grained entity
categories list of ACE LDC (2008).777Entity categories are listed in Table 2.
We dropped the NORP category, since it introduced complexity concerning the
distinction between adjectives and group names. Law did not appear in our
corpus. Finally, we allow nested entity mentions, as in Finkel and Manning
(2009); Benikova et al. (2014).888Nested labels are are not modeled in this
paper, but they are published with the corpus, to allow for further research.

#### Annotation Cycle

As Fort et al. (2009) put it, examples and rules would never cover all
possible cases because of the specificity of natural language and the
ambiguity of formulation. To address this we employed the cyclic approach of
agile annotation as offered by Alex et al. (2010). Every cycle consisted of:
annotation, evaluation and curation, clarification and refinements. We used
WebAnno Yimam et al. (2013) as our annotation interface.

The Initial Annotation Cycle was a two-stage pilot with 12 participants,
divided into 2 teams of 6. The teams received the same guidelines, with the
exception of the specifications of entity boundaries. One team was guided to
annotate the minimal string that designates the entity. The other was guided
to tag the maximal string which can still be considered as the entity. Our
agreement analysis showed that the minimal guideline generally led to more
consistent annotations. Based on this result (as well as low-level
refinements) from the pilot, we devised the full version of the
guidelines.999The complete annotation guide is publicly available at
https://github.com/OnlpLab/NEMO-Corpus.

Annotation, Evaluation and Curation: Every annotation cycle was performed by
two annotators (_A_ , _B_) and an annotation manager/curator (_C_). We
annotated the full corpus in 7 cycles. We evaluated the annotation in two
ways, manual curation and automatic evaluation. After each annotation step,
the curator manually reviewed every sentence in which disagreements arose, as
well as specific points of difficulty pointed out by the annotators. The
inter-annotator agreement metric described below was also used to
quantitatively gauge the progress and quality of the annotation.

Clarifications and Refinements: In the end of each cycle we held a
clarification talk between A, B and C, in which issues that came up during the
cycle were discussed. Following that talk we refined the guidelines and
updated the annotators, which went on to the next cycle. In the end we
performed a final curation run to make sentences from earlier cycles comply
with later refinements.101010A, B and C annotations are published to enable
research on learning with disagreements Plank et al. (2014).

Inter-Annotator Agreement (IAA) IAA is commonly measured using
$\kappa$-statistic. However, Pyysalo et al. (2007) show that it is not
suitable for evaluating inter-annotator agreement in NER. Instead, an $F_{1}$
metric on entity mentions has in recent years been adopted for this purpose
Zhang (2013). This metric allows for computing pair-wise IAA using standard
$F_{1}$ score by treating one annotator as gold and the other as the
prediction.

Our full corpus pair-wise $F_{1}$ scores are: IAA(_A_ ,_B_)=89, IAA(_B_
,_C_)=92, IAA(_A_ ,_C_)=96. Table 2 presents final corpus statistics.

Annotation Costs The annotation took on average about 35 seconds per sentence,
and thus a total of 60 hours for all sentences in the corpus for each
annotator. Six clarification talks were held between the cycles, which lasted
from thirty minutes to an hour. Giving a total of about 130 work hours of
expert annotators.111111The corpus is available at
https://github.com/OnlpLab/NEMO-Corpus.

| train | dev | test
---|---|---|---
Sentences | $4,937$ | $500$ | $706$
Tokens | $93,504$ | $8,531$ | $12,619$
Morphemes | $127,031$ | $11,301$ | $16,828$
All mentions | $6,282$ | $499$ | $932$
Type: Person (Per) | $2,128$ | $193$ | $267$
Type: Organization (Org) | $2,043$ | $119$ | $408$
Type: Geo-Political (Gpe) | $1,377$ | $121$ | $195$
Type: Location (Loc) | $331$ | $28$ | $41$
Type: Facility (Fac) | $163$ | $12$ | $11$
Type: Work-of-Art (Woa) | $114$ | $9$ | $6$
Type: Event (Eve) | $57$ | $12$ | $0$
Type: Product (Duc) | $36$ | $2$ | $3$
Type: Language (Ang) | $33$ | $3$ | $1$

Table 2:  Basic Corpus Statistics. Standard HTB Splits.

##  5 Experimental Settings

#### Goal

We set out to empirically evaluate the representation alternatives for the
input/output sequences (token-single, token-multi, morpheme) and the effect of
different architectures (Standard, Hybrid) on the performance of NER for
Hebrew.

#### Modeling Variants

All experiments use the corpus we just described and employ a standard Bi-
LSTM-CRF architecture for implementing the neural sequence labeling task Huang
et al. (2015). Our basic architecture121212Using the NCRF++ suite of Yang and
Zhang (2018). is composed of an embedding layer for the input and a 2-layer
Bi-LSTM followed by a CRF inference layer — for which we test three modeling
variants.

Figure 2:  The token-single and token-multi Models. The input and output
correspond to rows 1,2 in Tab. 1. Triangles indicate string embeddings.
Circles indicate char-based encoding. Figure 3:  The morpheme Model. The input
and output correspond to row 3 in Tab. 1. Triangles indicate string
embeddings. Circles indicate char-based encoding.

Figures 2–3 present the variants we employ. Figure 2 shows the token-based
variants, token-single and token-multi. The former outputs a single BIOSE
label per token, and the latter outputs a multi-label per token — a
concatenation of BIOSE labels of the morphemes composing the token. Figure 3
shows the morpheme-based variant for the same input phrase. It has the same
basic architecture, but now the input consists of morphological segments
instead of tokens. The model outputs a single BIOSE label for each
morphological segment in the input.

In all modeling variants, the input may be encoded in two ways: (a) String-
level embeddings (token-based or morpheme-based) optionally initialized with
pre-trained embeddings. (b) Char-level embeddings, trained simultaneously with
the main task (cf. Ma and Hovy (2016); Chiu and Nichols (2015); Lample et al.
(2016)). For char-based encoding (of either tokens or morphemes) we experiment
with CharLSTM, CharCNN or NoChar, that is, no character embedding at all.

We pre-trained all token-based or morpheme-based embeddings on the Hebrew
Wikipedia dump of Goldberg (2014). For morpheme-based embeddings, we decompose
the input using More et al. (2019), and use the morphological segments as the
embedding units.131313Embeddings and Wikipedia corpus also available in:
https://github.com/OnlpLab/NEMO We compare GloVe Pennington et al. (2014) and
fastText Bojanowski et al. (2017). We hypothesize that since FastText uses
sub-string information, it will be more useful for analyzing OOVs.

#### Hyper parameters

Following Reimers and Gurevych (2017); Yang et al. (2018), we performed hyper-
parameter tuning for each of our model variants. We performed hyper-parameter
tuning on the dev set in a number of rounds of random search, independently on
every input/output and char-embedding architecture. Table 3 shows our selected
hyper-parameters.141414A few interesting empirical observations diverging from
those of Reimers and Gurevych (2017); Yang et al. (2018) are worth mentioning.
We found that a lower Learning Rate than the one recommended by Yang et al.
(2018) (0.015), led to better results and less occurrences of divergence. We
further found that raising the number of Epochs from 100 to 200 did not result
in over-fitting, and significantly improved NER results. We used for
evaluation the weights from the best epoch. The Char CNN window size is
particularly interesting as it was not treated as a hyper-parameter in Reimers
and Gurevych (2017), Yang et al. (2018). However, given the token-internal
complexity in MRLs we conjecture that the window size over characters might
make a crucial effect. In our experiments we found that a larger window (7)
increased the performance. For MRLs, further research into this hyper-
parameter might be of interest.

Parameter | Value | Parameter | Value
---|---|---|---
Optimizer | SGD | *LR (token-single) | 0.01
Batch Size | 8 | *LR (token-multi) | 0.005
LR decay | 0.05 | *LR (morpheme) | 0.01
Epochs | 200 | Dropout | 0.5
Bi-LSTM layers | 2 | *CharCNN window | 7
Word Emb Dim | 300 | Char Emb dim | 30
Word Hidden Dim | 200 | *Char Hidden Dim | 70

Table 3:  Summary of Hyper-Parameter Tuning. The * indicates divergence from
the NCRF++ proposed setup and empirical findings Yang and Zhang (2018).

#### Evaluation

Standard NER studies typically invoke the CoNLL evaluation script that anchors
NEs in token positions Tjong Kim Sang (2003). However, it is inadequate for
our purposes because we want to compare entities across token-based vs.
morpheme-based settings. To this end, we use a revised evaluation procedure,
which anchors the entity in its form rather than its index. Specifically, we
report $F_{1}$ scores on strict, exact-match of the surface forms of the
entity mentions. I.e., the gold and predicted NE spans must exactly match in
their form, boundaries, and entity type. In all experiments, we report both
token-level F-scores and morpheme-level F-scores, for all models.

  * •

Token-Level evaluation. For the sake of backwards compatibility with previous
work on Hebrew NER, we first define token-level evaluation. For token-single
this is a straightforward calculation of $F_{1}$ against gold spans. For
token-multi and morpheme, we need to map the predicted label sequence of that
token to a single label, and we do so using linguistically-informed rules we
devise (as elaborated in Appendix A).151515In the morpheme case we might
encounter “illegal” label sequences in case of a prediction error. We employ
similar linguistically-informed heuristics to recover from that (See Appendix
A).

  * •

Morpheme-Level evaluation. Our ultimate goal is to obtain precise boundaries
of the NEs. Thus, our main metric evaluates NEs against the gold morphological
boundaries. For morpheme and token-single models, this is a straightforward
$F_{1}$ calculation against gold spans. Note for token-single we are expected
to pay a price for boundary mismatch. For token-multi, we know the number and
order of labels, so we align the labels in the multi-label of the token with
the morphemes in its morphological decomposition.161616In case of a
misalignment (in the number of morphemes and labels) we match the label-
morpheme pairs from the final one backwards, and pad unpaired morphemes with O
labels.

For all experiments and metrics, we report mean and confidence interval (0.95)
over ten runs.

#### Input-Output Scenarios

We experiment with two kinds of input settings: token-based, where the input
consists of the sequence of space-delimited tokens, and morpheme-based, where
the input consists of morphological segments. For the morpheme input, there
are three input variants:

  * (i) Morph-gold: where the morphological sequence is produced by an expert (idealistic).
(ii) Morph-standard: where the morphological sequence is produced by a
standard segmentation-first pipeline (realistic).
(iii) Morph-hybrid: where the morphological sequence is produced by the hybrid
architecture we propose (realistic).

In the token-multi case we can perform morpheme-based evaluation by aligning
individual labels in the multi-label with the morpheme sequence of the
respective token. Again we have three options as to which morphemes to use:

  * (i) Tok-multi-gold: The multi-label is aligned with morphemes produced by an expert (idealistic).
(ii) Tok-multi-standard: The multi-label is aligned with morphemes produced by
a standard pipeline (realistic).
(iii) Tok-multi-hybrid: The multi-label is aligned with morphemes produced by
the hybrid architecture we propose (realistic).

#### Pipeline Scenarios

Assume an input sentence $x$. In the Standard pipeline we use YAP,171717For
other languages this may be done using models for canonical segmentation as in
Kann et al. (2016). the current state-of-the-art morpho-syntactic parser for
Hebrew More et al. (2019), for the predicted segmentation $M=MD(MA(x))$. In
the Hybrid pipeline, we use YAP to first generate complete morphological
lattices $MA(x)$. Then, to obtain $MA(x)\restriction NER(x)$ we omit lattice
paths where the number of morphemes in the token decomposition does not
conform with the number of labels in the multi-label of
$\textit{NER}_{\textit{{\em token-multi}}}(x)$. Then, we apply YAP to obtain
$MD(MA(x)\restriction NER(x))$ on the constrained lattice. In predicted
morphology scenarios (either Standard or Hybrid), we use the same model
weights as trained on the gold segments, but feed predicted morphemes as
input.181818We do not re-train the morpheme models with predicted
segmentation, which might achieve better performance (e.g.. jackknifing). We
leave this for future work.

##  6 Results

###  6.1 The Units: Tokens vs. Morphemes

Figure 4:  Token-level Eval. on Dev w/ Gold Segmentation. CharCNN for morph,
CharLSTM for tok.

Figure 5:  Morph-Level Eval. on Dev w/ Gold Segmentation. CharCNN for morph,
CharLSTM for tok.

Figure 6:  Token-Level Evaluation in Realistic Scenarios on Dev, comparing
_Gold_ , Standard and Hybrid Morphology. CharCNN for morph, CharLSTM for tok.
Results for Gold, token-single and token-multi are taken from Fig 4.

Figure 7:  Morph-Level Evaluation in Realistic Scenarios on Dev, comparing
_Gold_ , Standard and Hybrid Morphology. CharCNN for morph, CharLSTM for tok.
Results for Gold, token-single and token-multi are taken from Fig 5.

Figure 4 shows the token-level evaluation for the different model variants we
defined. We see that morpheme models perform significantly better than the
token-single and token-multi variants. Interestingly, explicit modeling of
morphemes leads to better NER performance even when evaluated against token-
level boundaries. As expected, the performance gaps between variants are
smaller with fastText than they are with embeddings that are unaware of
characters (GloVe) or with no pre-training at all. We further pursue this in
Sec. 6.3.

Figure 5 shows the morpheme-level evaluation for the same model variants as in
Table 4. The most obvious trend here is the drop in the performance of the
token-single model. This is expected, reflecting the inadequacy of token
boundaries for identifying accurate boundaries for NER. Interestingly,
morpheme and token-multi models keep a similar level of performance as in
token-level evaluation, only slightly lower. Their performance gap is also
maintained, with morpheme performing better than token-multi. An obvious
caveat is that these results are obtained with gold morphology. What happens
in realistic scenarios?

###  6.2 The Architecture: Pipeline vs. Hybrid

Figure 8:  Entity Mention Counts and Ratio by Category and OOTV Category, for
Dev Set.

Figure 6 shows the token-level evaluation results in realistic scenarios. We
first observe a significant drop for morpheme models when Standard predicted
segmentation is introduced instead of gold. This means that MD errors are
indeed detrimental for the downstream task, in a non-negligible rate. Second,
we observe that much of this performance gap is recovered with the Hybrid
pipeline. It is noteworthy that while morph hybrid lags behind morph gold, it
is still consistently better than token-based models, token-single and token-
multi.

Figure 7 shows morpheme-level evaluation results for the same scenarios as in
Table 6. All trends from the token-level evaluation persist, including a drop
for all models with predicted segmentation relative to gold, with the hybrid
variant recovering much of the gap. Again morph gold outperforms token-multi,
but morph hybrid shows great advantages over all tok-multi variants. This
performance gap between morph (gold or hybrid) and tok-multi indicates that
explicit morphological modeling is indeed crucial for accurate NER.

Figure 9:  Token-Level Eval on Dev by OOTV Category. Using fastText and
CharLSTM.

###  6.3 Morphologically-Aware OOV Evaluation

As discussed in Section 2, morphological composition introduces an extremely
sparse word-level “long-tail” in MRLs. In order to gauge this phenomenon and
its effects on NER performance, we categorize unseen, out-of-training-
vocabulary (OOTV) mentions into 3 categories:

  * •

_Lexical_ : Unknown mentions caused by an unknown token which consists of a
single morpheme. This is a strictly lexical unknown with no morphological
composition (most English unknowns are in this category).

  * •

_Compositional_ : Unknown mentions caused by an unknown token which consists
of multiple _known_ morphemes. These are unknowns introduced strictly by
morphological composition, with no lexical unknowns.

  * •

_LexComp_ : Unknown mentions caused by an unknown token consisting of multiple
morphemes, of which (at least) one morpheme was not seen during training. In
such cases, both unknown morphological composition _and_ lexical unknowns are
involved.

We group NEs based on these categories, and evaluate each group separately. We
consider mentions that do not fall into any category as _Known_.

Figure 8 shows the distributions of entity mentions in the dev set by entity
type and OOTV category. OOTV categories that involve composition (Comp and
LexComp) are spread across all categories but one, and in some they even make
up more than half of all mentions.

Figure 9 shows token-level evaluation191919This section focuses on token-level
evaluation, which is a permissive evaluation metric, allowing us to compare
the models on a more level playing field, where all models (including token-
single) have an equal opportunity to perform. with fastText embeddings,
grouped by OOTV type. We first observe that indeed unknown NEs that are due to
morphological composition (_Comp_ and _LexComp_) proved the most challenging
for all models. We also find that in strictly _Compositional_ OOTV mentions,
morpheme-based models exhibit their most significant performance advantage,
supporting the hypothesis that explicit morphology helps to generalize. We
finally observe that token-multi models perform better than token-single
models for these NEs (in contrast with the trend for non-compositional NEs).
This corroborates the hypothesis that even partial modeling of morphology (as
in token-multi compared to token-single) is better than none, leading to
better generalization.

Figure 10:  Token-Level Eval. on Dev for Different OOTV Types, Char- and Word-
Embeddings.

#### String-level vs. Character-level Embeddings

To further understand the generalization capacity of different modeling
alternatives in MRLs, we probe into the interplay of string-based and char-
based embeddings in treating OOTV NEs.

Figure 10 presents 12 plots, each of which presents the level of performance
(y-axes) for all models (x-axes). Token-based models are on the left of each
x-axes, morpheme-based are on the right. We plot results with and without
character embeddings,202020For brevity we only show char LSTM (vs. no char
representation), there was no significant difference with CNN. in orange and
blue respectively. The plots are organized in a large grid, with the type of
NE on the y-axes (Known, Lex, Comp, LexComp), and the type of pre-training on
the x-axes (No pre-training, GloVe, fastText) .

At the top-most row, plotting the accuracy for Known NEs, we see a high level
of performance for all pre-training methods, with not much differences between
the type of pre-training, with or without the character embeddings. Moving
further down to the row of Lexical unseen NEs, char-based representations lead
to significant advantages when we assume no pre-training, but with GloVe pre-
training the performance substantially increases, and with fastText the
differences in performance with/without char-embeddings almost entirely
diminish, indicating the char-based embeddings are somewhat redundant in this
case.

The two lower rows in the large grid show the performance for Comp and LexComp
unseen NEs, which are ubiquitous in MRLs. For Compositional NEs, pre-training
closes only part of the gap between token-based and morpheme-based models.
Adding char-based representations indeed helps the token-based models, but
crucially does not close the gap with the morpheme-based variants.

Finally, for LexComp NEs at the lowest row, we again see that adding GloVe
pre-training and char-based embeddings does not close the gap with morpheme-
based models, indicating that not all morphological information is captured by
these vectors. For fastText with char-based embeddings the gap between token-
multi and morpheme greatly diminishes, but is still well above token-single.
This suggests biasing the model to learn about morphology (either via multi-
labels or by incorporating morphological boundaries) has advantages for
analysing OOTV entities, beyond the contribution of char-based embeddings
alone.

All in all, the biggest advantage of morpheme-based models over token-based
models is their ability to generalize from observed tokens to composition-
related OOTV (Comp/LexComp). While character-based embeddings do help token-
based models generalize, the contribution of modeling morphology is
indispensable, above and beyond the contribution of char-based embeddings.

Eval | Model | dev | test
---|---|---|---
Morph- | morph gold | $\textit{80.03}\pm\textit{0.4}$ | $\textit{79.10}\pm\textit{0.6}$
Level | morph hybrid | $\textbf{78.51}\pm 0.5$ | $\textbf{77.11}\pm 0.7$
| morph standard | $72.79\pm 0.5$ | $69.52\pm 0.6$
| token-multi hybrid | $75.70\pm 0.5$ | $74.64\pm 0.3$
Token- | morph gold | $\textit{80.30}\pm\textit{0.5}$ | $\textit{79.28}\pm\textit{0.6}$
Level | morph hybrid | $\textbf{79.04}\pm 0.5$ | ${77.64}\pm 0.7$
| morph standard | 74.52 $\pm$ 0.7 | $73.53\pm 0.8$
| token-multi | $77.59\pm 0.4$ | $\textbf{77.75}\pm 0.3$
| token-single | $78.15\pm 0.3$ | $77.15\pm 0.6$

Table 4:  Test vs. Dev: Results with fastText for all Models. morph-gold
presents an ideal upper-bound.

###  6.4 Setting in the Greater Context

#### Test Set Results

Table 4 confirms our best results on the Test set. The trends are kept, though
results on Test are lower than on Dev. The morph gold scenario still provides
an upperbound of the performance, but it is not realistic. For the realistic
scenarios, morph hybrid generally outperforms all other alternatives. The only
divergence is that in token-level evaluation, token-multi performs on a par
with morph hybrid on the Test set.

#### Results on MD Tasks.

While the Hybrid pipeline achieves superior performance on NER, it also
improves the state-of-the-art on other tasks in the pipeline. Table 5 shows
the Seg+POS results of our Hybrid pipeline scenario, compared with the
Standard pipeline which replicates the pipeline of More et al. (2019). We use
the metrics defined by More et al. (2019). We show substantial improvements
for the Hybrid pipeline over the results of More et al. (2019), and also
outperforming the Test results of Seker and Tsarfaty (2020).

|  | Seg+POS
---|---|---
dev |  Standard More et al. (2019) | $92.36$
| Ptr-Network Seker and Tsarfaty (2020) | 93.90
|  Hybrid (This work) | $93.12$
test |  Standard More et al. (2019) | $89.08$
| Ptr-Network Seker and Tsarfaty (2020) | $90.49$
|  Hybrid (This work) | 90.89

Table 5:  Morphological Segmentation & POS scores.

| Precision | Recall | F1
---|---|---|---
Ben-Mordecai (2005) | $84.54$ | $74.31$ | $79.10$
MEMM+HMM+REGEX |  |  |
This work | 86.84 | 82.6 | 84.71
token-single +FT+CharLSTM | $\pm 0.5$ | $\pm 0.9$ | $\pm 0.5$
This work | 86.93 | 83.59 | 85.22
_morph_ -Hybrid +FT+CharLSTM | $\pm 0.6$ | $\pm 0.8$ | $\pm 0.5$

Table 6:  NER Comparison with Ben-Mordecai (2005).

#### Comparison with Prior Art.

Table 6 presents our results on the Hebrew NER corpus of Ben-Mordecai (2005)
compared to their model, which uses a hand-crafted feature-engineered MEMM
with regular-expression rule-based enhancements and an entity lexicon. Like
Ben-Mordecai (2005) we performed three 75%-25% random train/test splits, and
used the same seven NE categories (Per,Loc,Org,Time,Date,Percent,Money). We
trained a token-single model on the original space-delimited tokens and a
morpheme model on automatically segmented morphemes we obtained using our best
segmentation model (Hybrid MD on our trained token-multi model, as in Table
5). Since their annotation includes only token-level boundaries, all of the
results we report conform with token-level evaluation.

Table 6 presents the results of these experiments. Both models significantly
outperform the previous state-of-the-art by Ben-Mordecai (2005), setting a new
performance bar on this earlier benchmark. Moreover, we again observe an
empirical advantage when explicitly modeling morphemes, even with the
automatic noisy segmentation that is used for the morpheme-based training.

##  7 Discussion: Joint Modeling Alternatives and Future Work

The present study provides the motivation and the necessary foundations for
comparing morpheme-based and token-based modeling for NER. While our findings
clearly demonstrate the advantages of morpheme-based modeling for NER in a
morphologically rich language, it is clear that our proposed Hybrid
architecture is not the only modeling alternative for linking NER and
morphology.

For example, a previous study by Güngör et al. (2018) addresses joint neural
modeling of morphological segmentation and NER labeling, proposing a multi-
task learning (MTL) approach for joint MD and NER in Turkish. They employ
separate Bi-LSTM networks for the MD and NER tasks, with a shared loss to
allow for joint learning. Their results indicate improved NER performance,
with no improvement in the MD results. Contrary to our proposal, they view MD
and NER as distinct tasks, assuming a single NER label per token, and not
providing disambiguated morpheme-level boundaries for the NER task. More
generally, they test only token-based NER labeling and do not attend to the
question of input/output granularity in their models.

A different approach for joint NER and morphology is jointly predicting the
segmentation and labels for each token in the input stream. This is the
approach taken, for instance, by the lattice-based Pointer-Network of Seker
and Tsarfaty (2020). As shown in Table 5, their results for morphological
segmentation and POS tagging are on a par with our reported results and, at
least in principle, it should be possible to extend the Seker and Tsarfaty
(2020) approach to yield also NER predictions.

However, our preliminary experiments with a lattice-based Pointer-network for
token segmentation and NER labeling shows that this is not a straightforward
task. Contrary to POS tags, which are constrained by the MA, every NER label
can potentially go with any segment, and this leads to a combinatorial
explosion of the search space represented by the lattice. As a result, the NER
predictions are brittle to learn, and the complexity of the resulting model is
computationally prohibitive.

A different approach to joint sequence segmentation and labeling can be
applying the neural model directly on the character-sequence of the input
stream. Such an approach is for instance the char-based labeling as
segmentation setup proposed by Shao et al. (2017). Shao et al. use a
character-based Bi-RNN-CRF to output a single label-per-char which indicates
both word boundary (using BIES sequence labels) and the POS tags. This method
is also used in their universal segmentation paper, Shao et al. (2018).
However, as seen in the results of Shao et al. (2018), char-based labeling for
segmenting Semitic languages lags far behind all other languages, precisely
because morphological boundaries are not explicit in the character sequences.

Additional proposals are those of Kong et al. (2015); Kemos et al. (2019).
First, Kong et al. (2015) proposed to solve e.g. Chinese segmentation and POS
tagging using dynamic programming with neural encoding, by using a Bi-LSTM to
encode the character input, and then feed it to a semi-markov CRF to obtain
probabilities for the different segmentation options. Kemos et al. (2019)
propose an approach similar to Kong et al. (2015) for joint segmentation and
tagging but add convolution layers on top of the Bi-LSTM encodings to obtain
segment features hierarchically and then feed them to the semi-markov CRF.

Preliminary experiments we conducted confirm that char-based joint
segmentation and NER labeling for Hebrew, either using char-based labeling or
a seq2seq architecture, still lags behind our reported results. We conjecture
that this is due to the complex morpho-phonological and orthographic processed
in Semitic languages. Going into char-based modeling nuances and offering a
sound joint solution for a language like Hebrew is an important matter that
merits its own investigation. Such work is feasible now given the new corpus,
however, it is out of the scope of the current study.

All in all, the design of sophisticated joint modeling strategies for
morpheme-based NER poses fascinating questions — for which our work provides a
solid foundation (data, protocols, metrics, strong baselines). More work is
needed for investigating joint modeling of NER and morphology, in the
directions portrayed in this Section, yet it is beyond the scope of this
paper, and we leave this investigation for future work.

Finally, while the joint approach is appealing, we argue that the elegance of
our Hybrid solution is precisely in providing a clear and well-defined
interface between MD and NER through which the two tasks can interact, while
still keeping the distinct models simple, robust, and efficiently trainable.
It also has the advantage of allowing us to seamlessly integrate sequence
labelling with any lattice-based MA, in a plug-and-play language-agnostic
fashion, towards obtaining further advantages on both of these tasks.

##  8 Conclusion

This work addresses the modeling challenges of Neural NER in MRLs. We deliver
a parallel token-vs-morpheme NER corpus for Modern Hebrew, that allows one to
assess NER modeling strategies in morphologically rich-and-ambiguous
environments. Our experiments show that while NER benefits from morphological
decomposition, downstream results are sensitive to segmentation errors. We
thus propose a Hybrid architecture in which NER precedes and prunes the
morphological decomposition. This approach greatly outperforms a Standard
pipeline in realistic (non-gold) scenarios. Our analysis further shows that
morpheme-based models better recognize OOVs that result from morphological
composition. All in all we deliver new state-of-the-art results for Hebrew NER
and MD, along with a novel benchmark, to encourage further investigation into
the interaction between NER and morphology.

## Acknowledgments

We are grateful to the BIU-NLP lab members as well as 6 anonymous reviewers
for their insightful remarks. We further thank Daphna Amit and Zef Segal for
their meticulous annotation and profound discussions. This research is funded
by an ISF Individual Grant (1739/26) and an ERC Starting Grant (677352), for
which we are grateful.

## References

  * Alex et al. (2010) Bea Alex, Claire Grover, Rongzhou Shen, and Mijail Kabadjov. 2010.  Agile corpus annotation in practice: An overview of manual and automatic annotation of CVs.  In _Proceedings of the Fourth Linguistic Annotation Workshop_ , pages 29–37, Uppsala, Sweden. Association for Computational Linguistics.
  * Ben-Mordecai (2005) Naama Ben-Mordecai. 2005.  Hebrew named entity recognition.  Master’s thesis, Department of Computer Science, Ben-Gurion University.
  * Benajiba et al. (2007) Yassine Benajiba, Paolo Rosso, and José Miguel BenedíRuiz. 2007.  Anersys: An arabic named entity recognition system based on maximum entropy.  In _Computational Linguistics and Intelligent Text Processing_ , pages 143–153, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Benikova et al. (2014) Darina Benikova, Chris Biemann, and Marc Reznicek. 2014.  NoSta-D named entity annotation for German: Guidelines and dataset.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014)_ , pages 2524–2531, Reykjavik, Iceland. European Languages Resources Association (ELRA).
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Chinchor et al. (1999) N. Chinchor, E. Brown, L. Ferro, and P. Robinson. 1999.  Named entity recognition task definition.  Technical Report Version 1.4, The MITRE Corporation and SAIC.
  * Chiu and Nichols (2015) Jason P. C. Chiu and Eric Nichols. 2015.  Named entity recognition with bidirectional lstm-cnns.  _CoRR_ , abs/1511.08308.
  * Comrie et al. (2008) Bernard Comrie, Martin Haspelmath, and Balthasar Bickel. 2008.  The leipzig glossing rules: Conventions for interlinear morpheme-by-morpheme glosses.  _Department of Linguistics of the Max Planck Institute for Evolutionary Anthropology & the Department of Linguistics of the University of Leipzig_.
  * Czarnowska et al. (2019) Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, and Ann Copestake. 2019.  Don’t forget the long tail! a comprehensive analysis of morphological generalization in bilingual lexicon induction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 974–983, Hong Kong, China. Association for Computational Linguistics.
  * Darwish (2013) Kareem Darwish. 2013.  Named entity recognition using cross-lingual resources: Arabic as an example.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1558–1567, Sofia, Bulgaria. Association for Computational Linguistics.
  * Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017.  Results of the WNUT2017 shared task on novel and emerging entity recognition.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text_ , pages 140–147, Copenhagen, Denmark. Association for Computational Linguistics.
  * Finkel and Manning (2009) Jenny Rose Finkel and Christopher D. Manning. 2009.  Nested named entity recognition.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1_ , EMNLP ’09, pages 141–150, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Fort et al. (2009) Karën Fort, Maud Ehrmann, and Adeline Nazarenko. 2009.  Towards a methodology for named entities annotation.  In _Proceedings of the Third Linguistic Annotation Workshop (LAW III)_ , pages 142–145, Suntec, Singapore. Association for Computational Linguistics.
  * Goldberg (2014) Yoav Goldberg. 2014.  Hebrew wikipedia dependency parsed corpus, v.1.0.  Technical Report V.1.0.
  * Goldberg and Tsarfaty (2008) Yoav Goldberg and Reut Tsarfaty. 2008.  A single generative model for joint morphological segmentation and syntactic parsing.  In _Proceedings of ACL-08: HLT_ , pages 371–379, Columbus, Ohio. Association for Computational Linguistics.
  * Green and Manning (2010) Spence Green and Christopher D. Manning. 2010.  Better Arabic parsing: Baselines, evaluations, and analysis.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_ , pages 394–402, Beijing, China. Coling 2010 Organizing Committee.
  * Güngör et al. (2018) Onur Güngör, Suzan Üsküdarli, and Tunga Güngör. 2018.  Improving named entity recognition by jointly learning to disambiguate morphological tags.  _CoRR_ , abs/1807.06683.
  * Habash and Rambow (2005) Nizar Habash and Owen Rambow. 2005.  Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 573–580, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF models for sequence tagging.  _CoRR_ , abs/1508.01991.
  * Kann et al. (2016) Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2016.  Neural morphological analysis: Encoding-decoding canonical segments.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 961–967, Austin, Texas. Association for Computational Linguistics.
  * Kemos et al. (2019) Apostolos Kemos, Heike Adel, and Hinrich Schütze. 2019.  Neural semi-Markov conditional random fields for robust character-based part-of-speech tagging.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2736–2743, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Klein and Tsarfaty (2020) Stav Klein and Reut Tsarfaty. 2020.  Getting the ##life out of living: How adequate are word-pieces for modelling complex morphology? In _Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_ , pages 204–209, Online. Association for Computational Linguistics.
  * Kong et al. (2015) Lingpeng Kong, Chris Dyer, and Noah A Smith. 2015.  Segmental recurrent neural networks.  _arXiv preprint arXiv:1511.06018_.
  * Küçük and Can (2019) Dilek Küçük and Fazli Can. 2019.  A tweet dataset annotated for named entity recognition and stance detection.  _CoRR_ , abs/1901.04787.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  _CoRR_ , abs/1603.01360.
  * LDC (2008) LDC. 2008.  ACE (automatic content extraction) english annotation guidelines for entities version 6.6.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3219–3232, Brussels, Belgium. Association for Computational Linguistics.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard H. Hovy. 2016.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  _CoRR_ , abs/1603.01354.
  * Mohit et al. (2012) Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A. Smith. 2012.  Recall-oriented learning of named entities in Arabic Wikipedia.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 162–173, Avignon, France. Association for Computational Linguistics.
  * More et al. (2019) Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019.  Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from modern Hebrew.  _Transactions of the Association for Computational Linguistics_ , 7:33–48.
  * Nivre et al. (2007) Joakim Nivre, Johan Hall, Sandra Kübler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007.  The CoNLL 2007 shared task on dependency parsing.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 915–932, Prague, Czech Republic. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Plank et al. (2014) Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014.  Learning part-of-speech taggers with inter-annotator agreement loss.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 742–751, Gothenburg, Sweden. Association for Computational Linguistics.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes.  In _Joint Conference on EMNLP and CoNLL - Shared Task_ , pages 1–40, Jeju Island, Korea. Association for Computational Linguistics.
  * Pyysalo et al. (2007) Sampo Pyysalo, Filip Ginter, Juho Heimonen, Jari Björne, Jorma Boberg, Jouni Järvinen, and Tapio Salakoski. 2007.  Bioinfer: a corpus for information extraction in the biomedical domain.  _BMC bioinformatics_ , 8(1):50.
  * Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009.  Design challenges and misconceptions in named entity recognition.  In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009)_ , pages 147–155, Boulder, Colorado. Association for Computational Linguistics.
  * Reimers and Gurevych (2017) Nils Reimers and Iryna Gurevych. 2017.  Optimal hyperparameters for deep lstm-networks for sequence labeling tasks.  _CoRR_ , abs/1707.06799.
  * Seddah et al. (2013) Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013.  Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages.  In _Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages_ , pages 146–182, Seattle, Washington, USA. Association for Computational Linguistics.
  * Seeker and Çetinoğlu (2015) Wolfgang Seeker and Özlem Çetinoğlu. 2015.  A graph-based lattice dependency parser for joint morphological segmentation and syntactic analysis.  _Transactions of the Association for Computational Linguistics_ , 3:359–373.
  * Seker and Tsarfaty (2020) Amit Seker and Reut Tsarfaty. 2020.  A pointer network architecture for joint morphological segmentation and tagging.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4368–4378, Online. Association for Computational Linguistics.
  * Shao et al. (2018) Yan Shao, Christian Hardmeier, and Joakim Nivre. 2018.  Universal word segmentation: Implementation and interpretation.  _Transactions of the Association for Computational Linguistics_ , 6:421–435.
  * Shao et al. (2017) Yan Shao, Christian Hardmeier, Jörg Tiedemann, and Joakim Nivre. 2017.  Character-based joint segmentation and POS tagging for Chinese using bidirectional RNN-CRF.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 173–183, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Sima’an et al. (2001) Khalil Sima’an, Alon Itai, Yoad Winter, Alon Altman, and Noa Nativ. 2001.  Building a tree-bank of modern hebrew text.
  * Song et al. (2018) Hye-Jeong Song, Byeong-Cheol Jo, Chan-Young Park, Jong-Dae Kim, and Yu-Seop Kim. 2018.  Comparison of named entity recognition methodologies in biomedical documents.  _Biomedical engineering online_ , 17(2):158.
  * Tjong Kim Sang (2003) Erik F. Tjong Kim Sang. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 - Volume 4_ , CONLL ’03, pages 142–147, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Tsarfaty et al. (2020) Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020.  From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7396–7408, Online. Association for Computational Linguistics.
  * Tsarfaty et al. (2019) Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019.  What’s wrong with Hebrew NLP? and how to make it right.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_ , pages 259–264, Hong Kong, China. Association for Computational Linguistics.
  * Tsarfaty et al. (2010) Reut Tsarfaty, Djamé Seddah, Yoav Goldberg, Sandra Kuebler, Yannick Versley, Marie Candito, Jennifer Foster, Ines Rehbein, and Lamia Tounsi. 2010\.  Statistical parsing of morphologically rich languages (SPMRL) what, how and whither.  In _Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages_ , pages 1–12, Los Angeles, CA, USA. Association for Computational Linguistics.
  * Tür et al. (2003) Gökhan Tür, Dilek Hakkani-tür, and Kemal Oflazer. 2003.  A statistical information extraction system for turkish.  _Nat. Lang. Eng._ , 9(2):181–210.
  * Vania et al. (2018) Clara Vania, Andreas Grivas, and Adam Lopez. 2018.  What do character-level models learn about morphology? the case of dependency parsing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2573–2583, Brussels, Belgium. Association for Computational Linguistics.
  * Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013.  Ontonotes release 5.0.  _Linguistic Data Consortium, Philadelphia, PA_.
  * Yang et al. (2018) Jie Yang, Shuailong Liang, and Yue Zhang. 2018.  Design challenges and misconceptions in neural sequence labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics (COLING)_.
  * Yang and Zhang (2018) Jie Yang and Yue Zhang. 2018.  NCRF++: An open-source neural sequence labeling toolkit.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_.
  * Yimam et al. (2013) Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013.  WebAnno: A flexible, web-based and visually supported system for distributed annotations.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 1–6, Sofia, Bulgaria. Association for Computational Linguistics.
  * Zhang (2013) Ziqi Zhang. 2013.  _Named entity recognition: challenges in document annotation, gazetteer construction and disambiguation_.  Ph.D. thesis, University of Sheffield.

##  Appendix A Alignment Heuristics

Aligning Multi-labels to Single Labels. In order to evaluate morpheme-based
labels (morph or token-multi) in token-based settings, we introduce a
deterministic procedure to extend the morphological labels to token
boundaries. Specifically, we use regular expressions to map the multiple
sequence labels to a single label by choosing the first non-O entity category
(BIES) as the single category. In case the sequence of labels is not valid
(e.g., B comes after E, or there is an O between two I labels), we use a
relaxed mapping that does not take the order of the labels into consideration:
if there is an S or both B and E in the sequence, return an S. Otherwise, if
there is an E, return an E; if there is a B, return a B; if there is an I
return an I (Figure 11).

Aligning Multi-labels to Morphemes. In order to obtain morpheme boundary
labels from _token-multi_ , we introduce a deterministic procedure to align
the token’s predicted multi-label with the list of morphemes predicted for it
by the MD. Specifically, we align the multi-labels to morphemes in the order
that they are both provided. In case of a mismatch between the number of
labels and morphemes predicted for the token, we match label-morpheme pairs
from the final one backwards. If the number of morphemes exceeds the number of
labels, we pad unpaired morphemes with O labels. If the number of labels
exceeds the morphemes, we drop unmatched labels (Figure 12).

def extend_morph_to_tok(token_labels):
_# gets list of morpheme NER labels for a token_
_# returns single NER label_
biose_labels = ”
categories = list()
for label in token_labels:
if label==’O’:
biose_labels += ’O’
categories += None
else:
bio, cat = label.split(’-’)
biose_labels += bio
categories += cat
single_biose = get_single_biose(biose_labels)
single_cat = get_single_token_categ(categories)
if single_biose == ’O’:
return ’O’
else: return single_biose + ’-’ +single_cat

def get_single_token_categ(categ_labels):
_# gets list of NER categories for a token_
_# returns the first category which is not None_
[…]

def get_single_biose(biose_labels):
_# gets string of ’BIOSE’ for a token_
_# returns single character from ’BIOSE’_
_# regex matches valid labels for: O|B|S|I|E|S_ valid_seq =
’O+|O*BI*|O*BI*EO*|I+|I*EO*|O*SO*’
if valid_seq.match(biose_labels):
_# split valid_seq by ’|’ and return_
_# label based on the corresponding re match_
[…]
else: _# treat as set_
if ’S’ in biose_labels:
return ’S’
elif ’B’ and ’E’ in biose_labels:
return ’S’
elif ’E’ in biose_labels:
return ’E’
elif ’B’ in biose_labels:
return ’B’
elif ’I’ in biose_labels:
return ’I’
return ’O’

Figure 11: Multi-Label to Single Label Alignment

def align_multi_to_morph(labels, forms):
_# gets list of labels and list of morph forms_
_# returns aligned list of tuples (morph, label)_
if len(labels)==len(forms):
_# if same length, return zipped by order_
return zip(forms, labels)
elif len(labels)>len(forms):
_# if extra labels, trim labels from the beginning_
_# return zipped by order_
[ … ]
elif len(labels)<len(forms):
_# if extra forms, pad beginning with ’O’_
_# return zipped by order_
[ … ]

Figure 12: Multi-Label to Morpheme Alignment