# ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language
Generation

Kaushal Kumar Maurya
Indian Institute of Technology Hyderabad
Hyderabad, India
<EMAIL_ADDRESS>

Yoshinobu Kano
Shizuoka University, Japan
<EMAIL_ADDRESS>&Maunendra Sankar Desarkar
Indian Institute of Technology Hyderabad
Hyderabad, India
<EMAIL_ADDRESS>

Kumari Deepshikha
NVIDIA, India
<EMAIL_ADDRESS>
###### Abstract

Despite the recent advancement in NLP research, cross-lingual transfer for
natural language generation is relatively understudied. In this work, we
transfer supervision from high resource language (HRL) to multiple low-
resource languages (LRLs) for natural language generation (NLG). We consider
four NLG tasks (text summarization, question generation, news headline
generation, and distractor generation) and three syntactically diverse
languages, i.e., English, Hindi, and Japanese. We propose an unsupervised
cross-lingual language generation framework (called ZmBART) that does not use
any parallel or pseudo-parallel/back-translated data. In this framework, we
further pre-train mBART sequence-to-sequence denoising auto-encoder model with
an auxiliary task using monolingual data of three languages. The objective
function of the auxiliary task is close to the target tasks which enriches the
multi-lingual latent representation of mBART and provides good initialization
for target tasks. Then, this model is fine-tuned with task-specific supervised
English data and directly evaluated with low-resource languages in the Zero-
shot setting. To overcome catastrophic forgetting and spurious correlation
issues, we applied freezing model component and data argumentation approaches
respectively. This simple modeling approach gave us promising results. We
experimented with few-shot training (with 1000 supervised data-points) which
boosted the model performance further. We performed several ablations and
cross-lingual transferability analysis to demonstrate the robustness of
ZmBART.

##  1 Introduction

Figure 1: Zero-shot news headline generation from ZmBART in Hindi language

Recent advancement in natural language generation (NLG) is heavily oriented
towards large annotated training data. Such large task-specific annotated data
is available for high resource language (HRL) like English. The tasks become
challenging when limited training data is available. This is often observed
for low-resource languages (LRLs) like Hindi, Japanese, etc. Manually
annotating large data is time-consuming, expensive and uninteresting. This
limits the model development and product deployment for LRLs. Moreover,
despite large active research in cross-lingual representation learning Hu et
al. (2020); Conneau et al. (2020); Lewis et al. (2020b), the area of cross-
lingual transfer and generation is relatively under-explored. Motivated by
these factors, we propose a novel framework to transfer supervision from HRL
to LRLs where model is trained on one language and directly evaluated for
unseen languages. This enables cross-lingual transfer and generation for low
resource languages in zero and few-shot settings for different tasks. The
framework can be easily extended to other tasks and languages.

We carefully selected four challenging NLG tasks i.e., news headline-
generation (NHG), question generation (QG), abstractive text summarization
(ATS) and distractor generation (DG) to validate the framework’s performance.
NHG and ATS require understanding of input passage to generate meaningful
headline and summary respectively. QG task should accumulate information from
a passage and answer to generate high-quality questions. Distractor generation
is the task of generating incorrect options from reading comprehension MCQ. It
is challenging because generated distractors should be in the context with
question but should not be semantically equivalent to the answer. We consider
two LRLs i.e., Hindi and Japanese from two different language families.
English is selected as the HRL from which the learning would be transferred to
the LRLs. All three selected languages are different in their syntactic
structures and typologically diverse. As there is no established publicly
available dataset for DG in Hindi, we also create a new DG dataset for Hindi
called as HiDG111HiDG dataset download link:
https://github.com/kaushal0494/ZmBART.

Our proposed framework to achieve this transfer of supervision from HRL to LRL
under multiple languages and multiple tasks is named as ZmBART. ZmBART is
based on mBART Liu et al. (2020), a pre-trained model for cross-lingual
natural language generation (NLG). We further pre-train mBART with a novel
auxiliary task. Then the trained model is fine-tuned on large task-specific
supervised data in English and evaluated directly on Hindi and Japanese
languages in zero/few-shot setting for the tasks under consideration. We
observe that the auxiliary task plays a critical role on the model’s
performance and needs to be carefully designed. This framework can be directly
applied to multiple cross-lingual generation tasks without even the need to
modify model hyper-parameters. Figure-1 shows a zero-shot NHG sample output
generated by the ZmBART model. Our main contributions in this work can be
summarized as:

  1. 1.

We propose a novel zero-shot cross-lingual generation framework called ZmBART
without parallel data and without back-translation. The framework can be
directly applied across multiple tasks without even modifications in hyper-
parameter values.

  2. 2.

We demonstrate the effectiveness of ZmBART on four cross-lingual generation
tasks across three typologically diverse languages.

  3. 3.

We have created HiDG, a high-quality distractor generation dataset for the
Hindi language.

##  2 Related Work

Early works on cross-lingual generation rely on machine translation (MT). In
the very first work, Wan et al. (2010) leveraged the MT pipeline for cross-
language document summarization. They first translate the non-English test
instances to English. This translated text is fed through the supervised model
(trained with document summarization data in English) to generate English
summaries. Finally, these summaries are translated back to the target
language. Shen et al. (2018) and Duan et al. (2019) used MT systems to
generate pseudo training data for cross-lingual summarization and news
headline generation respectively. However these MT based models are not
suitable for low resource languages as they do not share parameters across-
languages and generated translations are error-prone.

Recently there are a few works in the direction of supervision transfer from
HRL(s) to LRL(s) for language generation. Kumar et al. (2019) used back-
translation (needs MT system) and annotated supervised data for cross-lingual
question generation. Chi et al. (2020) used parallel data to train a sequence-
to-sequence model for zero-shot cross-lingual abstractive text summarization
and question generation. Lewis et al. (2020a) proposed a pre-training based on
mono-lingual paragraphs. Then this pre-trained model is used for zero-shot
abstractive text summarization (ATS) in multiple languages. They trained a
model on the ATS dataset on all the languages except the test language. This
approach needs annotated data in multiple languages. Existing supervision
transfer methods require parallel data for the cross-lingual tasks. Either
they use available parallel corpora directly, or they translate/ back-
translate data to generate pseudo-parallel corpora. Both these approaches pose
significant challenges, as task-specific parallel data for multiple languages
is difficult to obtain, and MT are far from perfect, especially for low
resource languages.

Unlike the previous approaches, we did not use any parallel data or back-
translation in our proposed framework. We did not pre-train any model from
scratch. Instead, we leveraged the existing pre-trained model mBART. We
included four challenging generation tasks across three syntactically diverse
languages. Even we did not modify any hyper-parameters across the tasks and
languages. All these considerations make the framework simple and easy to use.
Further, it enables the addition of different other languages and NLG tasks in
the proposed framework a simple extension exercise.

##  3 Methodology

Figure 2 shows an outline of our proposed ZmBART framework. ZmBART is based on
pre-trained mBART Liu et al. (2020) model. In our framework, we take the mBART
model and further pre-train it on an auxiliary task. The auxiliary task is
designed in such a way that the objective function of auxiliary task is close
to fine-tuning tasks and only utilizes the mono-lingual data from the selected
languages. Similar to mBART model we use language identifier tag with slight
modification. We concatenate $<fxx><2xx>$ tags in input data instance where
$xx$ indicates the language tag. Given an input sentence and the language tag
the model encodes the sentence in multi-lingual space. By conditioning on the
encoded representation and language tag the decoder generates output text in
target language.

Figure 2: Architecture diagram of ZmBART

###  3.1 Multilingual BART (mBART)

Multilingual BART Liu et al. (2020) is an extension of BART model Lewis et al.
(2020c) to multiple languages. It is a transformer-based sequence-to-sequence
pre-trained model. The model is trained on monolingual data in many languages
from Wikipedia Common Crawl corpus with BART language model objective.
Particularly, The training data is concatenation of data from $K$ languages
i.e., $\mathcal{D}=\\{\mathcal{D}_{1},\mathcal{D}_{2}\dots\mathcal{D}_{K}\\}$
where $\mathcal{D}_{i}$ is a collection of monolingual documents in language
$i$. They introduced two types of noises to corrupt the text: (1) random token
span masking and (2) sentence order permutation. mBART is trained as denoising
autoencoder. During training, the model has to predict text $X$ from it’s
corrupted version $g(X)$, where $g$ is noise function. The aim is to maximize
the following objective function

$\mathcal{L}_{\theta}=\sum_{\mathcal{D}_{i}\in\mathcal{D}}\sum_{x\in\mathcal{D}_{i}}logP(x|g(x);\theta),$
(1)

where $x$ is a data instance of language $i$. Probability distribution $P$ is
defined by the sequence-to-sequence model. mBART gave state-of-the-art results
in sentence and document level machine translations tasks. Details about mBART
model can be found in Liu et al. (2020).

###  3.2 Unsupervised Auxiliary Task

Although the mBART pre-trained model encodes a multi-lingual latent space, it
can not be used directly for cross-lingual generation. This is because the
model is jointly trained on denoising objectives which do not directly follow
auto-regressive decoding, thereby causing mismatch between pre-training and
fine-tuning objectives. To overcome this problem, an unsupervised auxiliary
task is introduced. We design the auxiliary task with the following desiderata
in mind. It (1) should only utilize mono-lingual data from selected languages,
(2) should enrich the mBART latent representations for selected languages and
(3) train the decoder in pure auto-regressive manner with a training objective
which is close to multiple fine-tuning tasks.

The auxiliary task in ZmBART is an additional pre-training step for better
warm-start to downstream auto-regressive NLG tasks - although the final task
(Distractor/Question/Summary generation) can be different from the auxiliary
task. Additionally, this step allows the model to have a closer look at the
languages under consideration and enrich/adjust the representations and
parameters accordingly.

Outputs of the NLG tasks considered in this work are expected to contain words
from different parts of the input. Generation of the output tokens are handled
by the framework using an encoder-decoder setup. Hence we decide to have an
auxiliary task that also encodes the input, and attends to this encoded
representation to generate the output words in auto-regressive manner. This
way, a single auxiliary task can help to enrich the token representations,
warm up the encoder-decoder weights for fine tuning, and also caters to the
multiple final output tasks. We define the auxiliary task as: Given an input
passage, generate few random sentences (called rand-summary) from the passage.
After experimentation we found that randomly generating 20% sentences from
passage works the best. Particularly, the input passage has length between
$5\mathchar 45\relax 25$ sentences and output is $1\mathchar 45\relax 5$
random sentences from the passage. We do not assume any relations among
sentences of the passage. We sample equal proportion of monolingual data from
three languages. Data preparation steps for the auxiliary task are given
below:

  1. 1.

Generate a random number $k\in\\{5.\cdots,25\\}$. $k$ denotes the size of
input passage

  2. 2.

passage: Append $k$ continuous sentences, starting from a random index of
monolingual corpus $D_{i}$ of the $i^{th}$ language

  3. 3.

rand-summary: Randomly select 20% sentences from the passage

  4. 4.

Repeat steps $1$ to $3$ for $p$ languages

  5. 5.

Repeat steps $1$ to $4$ for $N$ times, to collect $Np$ $<$passage, rand-
summary$>$ pairs

###  3.3 Fine-Tuning on Downstream NLG Tasks

The proposed pre-trained model is directly fine-tuned on four downstream
tasks: Question Generation (QG), News Headline Generation (NHG), Abstractive
Text Summarization (ATS) and Distractor Generation (DG). First, the model is
fine-tuned on large task-specific English supervised data and then this
trained model is directly evaluated on Hindi and Japanese evaluation datasets
in zero-shot setting. To validate the hypothesis that the ZmBART framework is
robust across multiple tasks and languages, we did not modify any hyper-
parameters during fine-tuning. It is often observed that including a few
instances from LRL to supervised data boosts the model performance. To
validate this point we further fine-tuned ZmBART with 1000 task-specific
supervised data-points in Hindi and Japanese languages in few-shot setting
which boosts the model performance.

###  3.4 Dealing with Catastrophic Forgetting and Spurious Correlation

During experimentation with the zero shot setup, it is observed that the model
always generates the output text in English irrespective of input and language
tag. We suspect this to be due to catastrophic forgetting problem Van de Ven
and Tolias (2019). The supervised training completely overrides/erases the
pre-trained learning. The generator (decoder) becomes biased towards English
due to the explicit supervision learned from large task-specific English data.
To overcome this problem, we freeze all word embeddings and all the parameters
of decoder layers during fine-tuning with English data. Although this resolves
the problem for NHG, QG and DG, the problem did not get completely resolved
for the ATS task. We noticed that the zero-shot ATS output now is not
completely in English, but it became of code-mix nature. In other words, the
number of English words in the output reduced, but still lot many English
words remained. The code-mixed outputs were logical and meaningful. We assume
this to be due to spurious correlation issue, also reported in Gu et al.
(2019). To resolve this issue, we added a few examples (25 in number) of the
auxiliary-task data during the fine-tuning step. This augmentation was helpful
to address the spurious correlation issue for ATS. It is to be noted that the
non-English data used for this augmentation is still of unsupervised and
monolingual nature.

##  4 Experimental Setup and Results

We conduct experiments over four NLG tasks in three languages. We compare the
performance of ZmBART with strong and MT pipeline based baseline models. We
use both automated and manual evaluation metrics to evaluate model
performances.

###  4.1 Baselines

Prior results are not available in literature for selected languages and
datasets. Hence, for performance comparison, we developed several strong
baselines based on recent models and architectures. Details of these baselines
are mentioned below:

  * •

MT Pipeline (mBART): Here, we fine-tune mBART on task-specific English data.
Non-English test data instances are first translated into English and passed
to the fine-tuned model. The output is translated back to the input language.
Google Translator is used for translations.

  * •

mBART+MADMo: This is an mBART based baseline where the auxiliary task has
Masking And Denoising objective with Mono-lingual data in three languages. The
aim is to enrich the cross-lingual latent representation space of mBART for
English, Hindi and Japanese.

  * •

mBART+MADPd: Inspired from Chi et al. (2020), we took Parallel Data (English-
Hindi and English-Japanese) and concatenate each parallel instances of two
languages. Then we used this data with Masking And Doising objective to
further train mBART. Including parallel data provides explicit supervision
while generating Hindi and Japanese text.

###  4.2 Evaluation

We use both automated and manual evaluation metrics for performance
comparison. Multiple metrics are used in literature for NLG tasks. Since we
are considering multiple tasks, for brevity, against each task we only report
values of the metrics commonly used by the community for that particular task.
For automatic evaluation we used both lexical match (BLEU Papineni et al.
(2002) and ROUGE Lin (2004)) as well as embedding based evaluation metrics
(BERTScore Zhang et al. (2020)). To evaluate question generation and
distractor generation tasks we use case-mix BLEU-4 (BL) score from sacreBLEU
implementation, ROUGE-L (R-L) and BERTScore (BS). For ATS and NHG tasks
ROUGE-1, ROUGE-2 and ROUGE-L are used.

We follow a similar approach for manual evaluation as Chi et al. (2020). We
sampled 50 generated data points each for QG, ATS and NHG tasks in both Hindi
and Japanese languages. We use three metrics: Fluency (Flu), Relatedness (Rel)
and Correctness (Corr). Fluency measures how fluent the generated text is.
Relatedness indicates how much the generated outputs are in the context with
input(s), Correctness measures semantics and meaningfulness. For DG, we use an
additional metric called Distractibility that measures the degree of confusion
for generated incorrect options. For DG task, there can be large number of
good distractors for given input, in such situation the manual evaluation is
more reliable. We sample 100 generated outputs for DG task. We employed large
pool of evaluators from native Hindi and Japanese speakers to evaluate Hindi
and Japanese output texts respectively. We asked each annotator to rate the
generated texts on a scale of 1-5 (1 is very bad and 5 is very good) for all
the metrics. We intentionally selected outputs of ZmBART and two best
baselines to reduce the evaluators workload.

Model | News Headline Generation | Question Generation | Abstractive TS | Distractor Generation
---|---|---|---|---
Metrics |  R-1 R-2 R-L |  BL R-L BS |  R-1 R-2 R-L |  BL R-L BS
Cross-lingual zero-shot generation results
MT Pipeline(mBART) | 16.61 4.91 15.83 | 2.6 21.31 71.53 | 11.15 3.11 10.93 | 1.6 9.66 67.35
mBART+MADMo | 29.32 16.36 27.52 | 3.9 23.70 73.76 | 18.25 4.92 16.10 | 2.8 15.86 72.26
mBART+MADPd | 24.02 13.41 23.29 | 4.3 25.29 73.74 | 10.47 2.55 12.30 | 2.9 15.43 72.89
ZmBART |  34.94 19.38 32.74 |  4.4 26.51 74.19 |  21.27 5.30 17.64 |  4.1 21.05 73.39
Cross-lingual few-shot generation results (with 1000 supervised data points)
ZmBART | 52.37 35.52 50.50 | 7.6 34.11 78.29 | 36.29 14.21 27.22 | 6.5 26.58 78.27

Table 1:  Zero and few-shot cross-lingual generation results for Hindi
Language

Model | News Headline Generation | Question Generation | Abstractive TS
---|---|---|---
Metrics |  R-1 R-2 R-L |  BL R-L BS |  R-1 R-2 R-L
Cross-lingual zero-shot generation results
MT Pipeline(mBART) | 13.82 0.38 7.92 | 8.9 26.92 71.93 | 17.90 3.98 18.46
mBART+MADMo | 33.75 8.12 17.78 | 16.6 34.80 74.01 | 28.74 9.01 23.63
mBART+MADPd | 31.58 6.98 18.95 | 18.2 36.22 74.99 | 19.17 4.89 18.22
ZmBART |  35.25 9.24 19.92 |  18.8 38.74 75.91 |  36.60 15.26 29.85
Cross-lingual few-shot generation results (with 1000 supervised data points)
ZmBART | 47.06 22.36 31.55 | 30.4 53.98 82.66 | 41.65 20.33 33.49

Table 2:  Zero and few-shot cross-lingual generation results for Japanese
Language

Model | News Headline Generation | Question Generation | Abstractive TS | Distractor Generation
---|---|---|---|---
Metrics |  Flu Rel Corr |  Flu Rel Corr |  Flu Rel Corr |  Flu Rel Dist
Annotator set-01
mBART+MADMo | 3.86 4.34 3.94 | 2.66 3.38 3.52 | 3.56 3.58 3.22 | 3.61 4.08 2.89
mBART+MADPd | 2.54 2.96 2.28 | 3.1 3.4 3.78 | 2.26 2.62 1.92 | 2.42 3.72 3.08
ZmBART |  4.14 4.22 4.04 |  3.24 3.44 3.9 |  4.02 4.12 3.54 |  4.12 4.19 3.83
Annotator set-02
mBART+MADMo | 3.84 4.18 3.8 | 3.83 4.63 3.96 | 3.38 3.96 3.4 | 3.38 3.00 2.24
mBART+MADPd | 2.96 3.02 2.7 |  3.98 4.70 3.98 | 2.96 3.16 2.84 | 2.97 3.11 2.46
ZmBART |  4.12 4.38 4.16 | 3.95 4.80 4.27 |  4.24 4.52 4.38 |  3.56 3.18 2.36
Annotator set-03
mBART+MADMo | 3.56 3.74 3.78 | 2.68 3.76 3.32 | 2.9 3.34 2.9 | 3.96 3.74 3.12
mBART+MADPd | 3.1 3.42 2.91 | 2.80 3.88 3.56 | 2.64 2.34 2.46 | 4.13 3.74 2.94
ZmBART |  3.70 3.84 3.76 |  2.86 4.04 3.76 |  4.06 3.56 3.56 |  4.44 4.12 3.12

Table 3:  Manual evaluation results of Zero-shot generated outputs for Hindi
language

Model | News Headline Generation | Question Generation | Abstractive TS
---|---|---|---
Metrics |  Flu Rel Corr |  Flu Rel Corr |  Flu Rel Corr
Annotator set-01
mBART+MADMo | 2.66 2.98 2.50 | 1.98 3.70 3.18 | 3.04 3.55 3.44
mBART+MADPd | 2.26 2.70 2.04 | 2.00 3.38 2.82 | 1.44 2.22 2.20
ZmBART |  3.60 4.02 3.50 |  2.12 3.30 2.94 |  4.24 3.90 3.90
Annotator set-02
mBART+MADMo | 2.1 2.58 1.98 | 1.24 1.70 1.33 | 2.56 3.40 2.62
mBART+MADPd | 1.58 1.78 1.46 |  1.46 1.72 1.78 | 1.00 1.00 1.00
ZmBART |  3.78 4.16 3.86 | 1.26 1.76 1.88 |  4.04 4.26 3.84
Annotator set-03
mBART+MADMo | 2.24 2.72 2.24 |  2.34 2.46 2.39 | 2.82 3.18 3.52
mBART+MADPd | 1.9 2.14 1.82 | 2.10 2.66 2.28 | 1.16 1.84 1.44
ZmBART |  2.88 3.22 2.92 | 2.10 2.70 2.46 |  3.32 3.52 3.04

Table 4:  Manual evaluation results of Zero-shot generated outputs for
Japanese language

###  4.3 News Headline Generation (NHG)

In this task, given a news article, we generate grammatically coherent,
semantically correct and abstractive headline. We use 500k/30k/30k
(train/validation/test) English NHG data splits from Gigaword headline
generation corpus222https://github.com/harvardnlp/sent-summary. For Hindi and
Japanese we use 1k/1k/5k spilt from
Kaggle333https://www.kaggle.com/disisbig/hindi-text-short-summarization-corpus
(we manually filtered high-quality news and headlines) and Iwama and Kano
(2019) respectively.

In a zero-shot setting we fine-tune ZmBART model on supervised data and
directly evaluate results on Hindi and Japanese test datasets. Automated
evaluation results are included in Tables 1 and 2. We observe that, quality of
generated headlines in Hindi is better compared to Japanese. The possible
reasoning can be the input size. ZmBART outperforms the baseline with an
absolute difference of 5.22 ROUGE-L score. mBART+MADMo is best among others
which shows that masking and denoising with monolingual data indeed enrich the
multi-lingual latent space for selected three languages. mBART+MADMo generates
code mixed (Hindi-English or Hindi-Japanese) output which degrades the model
performance. Few-shot training fills the mistakes of zero-shot models and
generates better quality output. Manual evaluation scores (Tables 3 and 4) and
automated scores correlate well validating ZmBART’s performance on NHG task.

###  4.4 Question Generation (QG)

In the Question Generation (QG) task, given an input passage and an answer,
the aim is to generate semantically and syntactically correct questions that
can produce the answer. We use SQuAD 1.1 Rajpurkar et al. (2016) English data
for supervised training. SQuAD is popular question answering dataset
consisting of 100k+ $<$passage, question, answer$>$ tuples. Following Zhao et
al. (2018), we combine the train and validation sets of SQuAD and then spilt
it as 80k/8k/10k training/validation/test tuples. For Hindi we use 1k/5.5k
(train/test) from MLQA Lewis et al. (2020d) and TyDiQA-GoldP Clark et al.
(2020) datasets. We use 1k/1k/5k for Japanese data from Takahashi et al.
(2019). Hindi and Japanese data are available in SQuAD data format which
maintains consistency in terms of passage size, question and number of
answers. For given passage and question we randomly selected one answer to
form the dataset. We combine answer and passage as single input sequence
separated by special token $<$s$>$.

Even without any parallel data, ZmBART outperformed all the baselines
consistently across all automated evaluation metrics for zero-shot setting.
Regarding manual evaluations, we see that Hindi questions received good score
from the annotators, whereas the questions generated for the Japanese language
inputs were considered as poor. Upon closer inspection of the generated text
we find that several generated questions start with English wh-words. This
mixing of English ’code’ in the output happened somewhat seamlessly for the
Hindi data as tokens in both languages are written in left-to-right manner.
Moreover, Hindi-English code-mixed data is now getting very common and the
annotators mostly accepted the mixing of the wh-words with the Hindi texts.
Such mixing is not very common with Japanese text. As a result, the annotators
assigned lower scores to such texts.

We then tried to understand the reason for getting the wh-words at the
beginning of the output. English interrogative sentences often introduce wh-
words at the beginning even though they are not present in the original data.
The model gets exposed to such special characteristics of the English
interrogative sentences during the fine tuning. The output from other
languages get impacted due to this in zero-shot settings. However, the
semantics of the text is captured well for the model as demonstrated by the
high BERTScore, indicating good cross-lingual transfer of semantic knowledge.

###  4.5 Abstractive Text Summarization (ATS)

In Abstractive Text Summarization (ATS), we aim to generate grammatically
coherent, semantically correct and abstractive summary given an input
document. We use recently released WikiLingua Ladhak et al. (2020) cross-
lingual abstractive summarization dataset containing data in 18 languages.
Prior splits are not available for this dataset. We use 131k/5k/5k
(train/validation/test) splits for English, and 1k/1k/5k splits for Hindi and
Japanese.

By skimming through data in Hindi we observe that many input documents consist
of technical instructions on usage of softwares/tools. Summarizing these
instructions are challenging. Zero-shot ZmBART performed better as compared to
baselines as shown in human evaluation (Tables 3 and 4 for Hindi and Japanese
respectively). The human evaluation results correlate with automated
evaluation as shown in Tables 1 and 2. Ladhak et al. (2020) reported cross-
lingual ATS score with same data for four different languages. The R-L score
for four languages are 34.06, 37.09, 31.67 and 32.33. We obtain R-L scores of
27.22 and 33.49 for Hindi and Japanese respectively, which shows that the few-
shot performance of ZmBART is acceptable.

###  4.6 Distractor Generation (DG)

The final task to judge ZmBART’s performance is Distractor Generation (DG). It
is the task of generating incorrect options (also known as distractors) from
reading comprehension MCQ. The generated distractors should be in the context
with the question but shouldn’t be semantically equivalent to the answer.
Formally, for given passage, question and answer triplet, generate a long,
coherent, and grammatically correct wrong option. Considering the fact that
for a given triplet there can be many incorrect options that are completely
different from each other, the problem is even more challenging. We use
English DG dataset from Maurya and Desarkar (2020) which consists of approx
135k/17k/17k (train/validation/test) split. We were unable to find a suitable
dataset in Japanese language. For Hindi language we created a dataset called
HiDG444Implementation, dataset, pre-trained checkpoints and ZmBART generated
text are available at https://github.com/kaushal0494/ZmBART of 1k/1k/5k split.
Similar to QG, to create input for ZmBART we concatenate the answer, question
and passage in the same order and separate them with special token $<$s$>$.

To generate HiDG, we first extracted $<$passage, question, answer$>$ triplets
from English SQuAD 1.1 with atleast 150 tokens in the triplet. We generate
distractors for these examples using model proposed by Maurya and Desarkar
(2020). The distractors were translated to Hindi using Google Translator
service. The translated distractors were manually verified or corrected (if
necessary) by human annotators.

The evaluation of the task is challenging because: 1) there can be more then
one correct distractors. Automated evaluation metrics may not able to capture
this aspect as only one ground truth distractor is available and 2) it may
possible that the generated distractor is semantically similar to answer with
high lexical overlap with reference distractor in those situation lexical
match based metrics are not suitable. To evaluate the DG task we mainly rely
on BERTScore and manual evaluation. Towards this effort we consider higher
number of DG samples for manual evaluation. Results from Tables 1 and 3
indicate the superiority of ZmBART over the baseline models for this task.

To summarize, we have performed experiments for 14 different task-setup
combinations involving low resource languages. With four tasks in Hindi and
three tasks in Japanese, and each task in zero shot and few shot setup, we
provide detailed comparative evaluation for the tasks. The tasks are of
different natures, and each task offers its own unique challenge. We
critically analyze the performances to show the robustness and the range of
applicability for the proposed ZmBART framework. We use fairseq library Ott et
al. (2019) for all the implementation and experiments. The implementation
details are included in supplementary.

##  5 Results Analysis and Ablation Study

In this section, we provide further analysis of the experimental results. We
also perform ablation studies to understand the impacts of the different
modeling decisions made in designing the framework.

•Supervised Training Results:  Table 5 shows the comparative results of fine-
tuned mBART with and without auxiliary task on task-specific supervised
English data. We observe that there is no significant performance degradation
of ZmBART over original mBART model with pure supervised training. Even, the
auxiliary task helps in achieving slight improvement over the original mBART
performance in most setups. This concludes that ZmBART can be adopted as
replacement of original mBART model with additional functionalities.

Task | Setting | BL | R-1 | R-2 | R-L | BS
---|---|---|---|---|---|---
NHG | W/ Aux-Task | 15.9 | 43.22 | 21.33 | 40.88 | 90.13
| W/O Aux-Task | 15.9 | 43.15 | 21.25 | 40.77 | 90.13
QG | W/ Aux-Task | 20.6 | 53.20 | 26.53 | 51.37 | 92.18
| W/O Aux-Task | 21.4 | 52.66 | 26.63 | 51.25 | 92.41
ATS | W/ Aux-Task | 16.0 | 40.01 | 18.11 | 38.29 | 90.20
| W/O Aux-Task | 15.8 | 39.52 | 18.00 | 37.91 | 90.10
DG | W/ Aux-Task | 10.3 | 31.76 | 14.89 | 31.18 | 89.33
| W/O Aux-Task | 10.0 | 31.87 | 14.59 | 31.30 | 89.42

Table 5:  Automated evaluation results of mBART on task-specific supervised
English dataset (with and without Auxiliary Task)

•Effect of Auxiliary Task:  Table 6 includes the results with and without
auxiliary task of ZmBART for ATS and QG tasks in zero-shot setting. It can be
inferred that without the auxiliary task, lexical match based scores are poor
because the decoder generates code-mixed outputs. We see that the BERTScore is
still reasonable without auxiliary task owing to the multilingual mBART
embedding. However, generation of the data in appropriate language is enabled
only after inclusion of the auxiliary task. The auxiliary task contributes in
two ways: it enables zero-shot generation and improves the mBART multilingual
latent space even more as indicated by the improved BERTScore.

Model | Abstractive TS | Question Generation
---|---|---
Metrics |  R-1 R-2 R-3 |  BL R-L BS
Hindi Language
ZmBART w/o Aux | 4.34 0.10 3.19 | 0.9 16.64 70.72
ZmBART with Aux | 21.27 5.30 17.64 | 4.4 26.51 74.19
Japanese Language
ZmBART w/o Aux | 6.80 0.11 5.30 | 6.7 33.07 70.35
ZmBART with Aux | 36.60 15.26 29.89 | 18.8 38.74 75.91

Table 6:  Zero-shot results of ZmBART with and without auxiliary task for
Hindi and Japanese

With these results we now want to understand _whether the auxiliary task is
able to generalize across multiple tasks, or favors specific tasks._ Among the
tasks considered in this work, we see that generation of meaningful
summaries/headlines require understanding/abstracting of input text which is
unlikely to be obtained by repeating sentences from input passages, as done in
the auxiliary task. ZmBART achieves good zero-shot/few-shot/supervised results
(Tables 1-5) on ATS and NHG over strong baselines. The generated headlines and
summaries were found to be mostly abstractive, they don’t contain large
continuous sequences from input text. As described in Sections 4.4 and 4.6,
Question Generation and Distractor Generation are more challenging tasks and
have objectives vastly different from the auxiliary task’s objective. Even for
these tasks, decent evaluation scores (Tables 1-5) and improvements over the
baselines across the languages considered indicate that the solutions are not
spurious. Incorporation of auxiliary task improves the performance of diverse
downstream tasks on real benchmark datasets, and does not favor any specific
task or dataset.

• Approaches to avoid Catastrophic Forgetting: We use two approaches to
address the catastrophic forgetting problem, (a) Freezing model components and
(b) optimized regularization Van de Ven and Tolias (2019). Tables 7 and 8 show
the automated evaluation results with different approaches used to deal with
the catastrophic forgetting problem. It can be noted that the proposed
modelling setup (i.e., ZmBART) gives best results.

Setup | Setting-Details | BL(hi/ja) | R-L(hi/ja) | BS(hi/ja)
---|---|---|---|---
Model Components | Freeze word embedding (WE) | 2.5/13.6 | 21.55/31.99 | 72.02/73.18
| Freeze WE + subset of Encoder & Decoder layers | 2.9/15.3 | 22.62/36.60 | 72.24/72.98
| Freeze WE + Encoder layers | 2.2/13.8 | 19.69/36.91 | 69.73/72.97
| Freeze WE + Decoder layers (ZmBART) | 4.4/18.8 | 26.51/38.74 | 74.19/75.91
Regularized Optimization | Elastic Weight Consolidation (EWC) | 2.1/11.6 | 18.21/29.47 | 68.36/72.91

Table 7:  Evaluation scores for different modeling approaches to avoid
catastrophic-forgetting for QG Task

Setup | Setting-Details | R-1(hi/ja) | R-2(hi/ja) | R-L(hi/ja)
---|---|---|---|---
Model Components | Freeze word embedding (WE) | 13.02/26.07 | 05.67/03.96 | 12.45/17.62
| Freeze WE + subset of Encoder & Decoder layers | 14.27/25.72 | 06.70/03.21 | 13.76/18.28
| Freeze WE + Encoder layers | 09.81/22.67 | 04.10/02.38 | 09.66/13.68
| Freeze WE + Decoder layers (ZmBART) | 34.94/35.25 | 19.38/09.24 | 32.74/19.92
Regularized Optimization | Elastic Weight Consolidation (EWC) | 12.01/22.16 | 05.43/03.11 | 11.22/16.31

Table 8:  Evaluation scores for different modeling approaches to avoid
catastrophic-forgetting for NHG Task

• Effect of Architecture on Few-shot Training:  In this set-up we experiment
with few-shot training with mBART (directly fine-tuned on task-specific
supervised English data) and ZmBART (trained with auxiliary task and fine-
tuned with English data). The results are presented in Table 9. We find that
ZmBART does better than mBART in corresponding setups. Moreover, although
freezing the decoder layer and word embeddings helps in zero-shot setting, it
is natural and useful to unfreeze them during few shot training.

Model | NHG | QG
---|---|---
Metrics |  R-1 R-2 R-3 |  BL R-L BS
mBART+WE | 50.61 34.32 49.01 | 6.1 31.20 77.01
mBART | 51.49 35.04 49.64 | 7.1 32.96 77.61
ZmBART+WE | 51.81 35.04 50.07 | 6.9 32.82 77.40
ZmBART |  52.37 35.52 50.50 |  7.9 34.49 78.39

Table 9:  Hindi language few-shot results for different architectural setups.
WE indicates that word embeddings and decoder layer parameters are frozen

• Few-shot performance with Supervised data:  Figures 4 and 4 show the trends
of few-shot training of ZmBART with respect to supervised Hindi and Japanese
training data for ATS and QG tasks respectively. We observe that with a small
number of supervised examples (e.g. 100) the model achieves decent few-shot
performance. We found the trends for different tasks to be similar. The
improvement in model performance tends to be minimal after 1000 examples.

Figure 3: ZmBART model few-shot performance with supervised Hindi/Japanese
data for ATS task

Figure 4: ZmBART model few-shot performance with supervised Hindi/Japanese
data for QG task

##  6 Conclusion

In this paper, we propose a novel unsupervised framework (ZmBART) for cross-
lingual transfer and generation. The framework transfers supervision from HRL
to LRLs which enables zero-shot language generation. The framework does not
use any direct or pseudo-parallel data. ZmBART is directly applied to multiple
generation tasks and languages. The model includes a carefully designed
auxiliary task that further improved the multilingual embedding space, and
helped to initialize encoder-decoder weights to enable zero shot language
generation. We performed experiments in three languages and 18 task-setup
combinations: four supervised tasks in English, four tasks in Hindi (each with
zero-shot and few-shot), and three tasks in Japanese (each with zero-shot and
few-shot). Except zero-shot question generation tasks, for all other tasks
involving LRLs, the proposed model generated good quality results as validated
by automated and manual evaluation measures. In future we want to extend this
work by adding multiple other languages and tasks, and also explore other
choices of auxiliary tasks for better model transfer.

## Acknowledgments

We thank the support from Nvidia AI Technology Center (NVAITC) towards the
requirements of computing power and compute infrastructure. We thank the human
annotators for human evaluation and the anonymous reviewers for their
constructive feedback.

## References

  * Chi et al. (2020) Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, and Heyan Huang. 2020\.  Cross-lingual natural language generation via pre-training.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):7570–7577.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages.  In _Transactions of the Association of Computational Linguistics_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Duan et al. (2019) Xiangyu Duan, Mingming Yin, Min Zhang, Boxing Chen, and Weihua Luo. 2019.  Zero-shot cross-lingual abstractive sentence summarization through teaching generation and attention.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3162–3172, Florence, Italy. Association for Computational Linguistics.
  * Gu et al. (2019) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O.K. Li. 2019.  Improved zero-shot neural machine translation via ignoring spurious correlations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1258–1268, Florence, Italy. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _International Conference on Machine Learning_ , pages 4411–4421. PMLR.
  * Iwama and Kano (2019) Kango Iwama and Yoshinobu Kano. 2019.  Multiple news headlines generation using page metadata.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 101–105, Tokyo, Japan. Association for Computational Linguistics.
  * Kumar et al. (2019) Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, and Preethi Jyothi. 2019.  Cross-lingual training for automatic question generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4863–4872, Florence, Italy. Association for Computational Linguistics.
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4034–4048, Online. Association for Computational Linguistics.
  * Lewis et al. (2020a) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida I. Wang, and Luke Zettlemoyer. 2020a.  Pre-training via paraphrasing.  _CoRR_ , abs/2006.15020.
  * Lewis et al. (2020b) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lewis et al. (2020c) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020c.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lewis et al. (2020d) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020d.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Maurya and Desarkar (2020) Kaushal Kumar Maurya and Maunendra Sankar Desarkar. 2020.  Learning to distract: A hierarchical multi-decoder network for automated generation of long distractors for multiple-choice questions for reading comprehension.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 1115–1124.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Shen et al. (2018) Shi-qi Shen, Yun Chen, Cheng Yang, Zhi-yuan Liu, and Mao-song Sun. 2018.  Zero-shot cross-lingual neural headline generation.  _IEEE/ACM Trans. Audio, Speech and Lang. Proc._ , 26(12):2319–2327.
  * Takahashi et al. (2019) Norio Takahashi, Tomohide Shibata, Daisuke Kawahara, and Sadao Kurohashi. 2019.  Machine comprehension improves domain-specific Japanese predicate-argument structure analysis.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 98–104, Hong Kong, China. Association for Computational Linguistics.
  * Van de Ven and Tolias (2019) Gido M Van de Ven and Andreas S Tolias. 2019.  Three scenarios for continual learning.  _arXiv preprint arXiv:1904.07734_.
  * Wan et al. (2010) Xiaojun Wan, Huiying Li, and Jianguo Xiao. 2010.  Cross-language document summarization based on machine translation quality prediction.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 917–926, Uppsala, Sweden. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhao et al. (2018) Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018.  Paragraph-level neural question generation with maxout pointer and gated self-attention networks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3901–3910, Brussels, Belgium. Association for Computational Linguistics.

##  7 Supplementary Materials

###  7.1 Implementation Details:

We use a standard sequence-to-sequence Transformer architecture with 12 layers
(each 16 heads) for encoder and decoder. The model has a dimension of 1024
(approx 680M parameters). Additional layer-normalization was used with both
the encoder and decoder. We found FP16 precision stabilized the training. We
trained all the models on 4 Nvidia V100 GPUs (32GB). Similar to mBART we use
the Adam optimizer ($\epsilon=1e\mathchar 45\relax 6$, $\beta_{2}=0.98$) and
linear learning rate decay scheduling. The training started with a dropout
value 0.3 and was later reduced to 0.2 after 20k steps and 0 after 40k steps.
The loss function was cross-entropy label smoothing loss. 2500 warm-up steps
and $3e\mathchar 45\relax 5$ learning rate were used. The model selection was
done based on validation data likelihood. We use beam-search with beam size 5
in the decoding for all the tasks. We loaded mBARTCC25 pre-trained checkpoint
weights and further pre-train/fine-tune model on task-specific data with
teacher forcing method.

The above set of parameters are used for all the target tasks as well as the
auxiliary task. We process different batch sizes of input for different tasks.
We use 2048, 3000, 4096, 2048, and 5000 tokens per GPU for ATS, DG, QG,
auxiliary, and NHG tasks, respectively. We use shared Byte Pair Encoding (BPE)
vocabulary from sentence-piece tokenizer of size 250k. We use 34k/1k/1k
(train/validation/test) data-points for auxiliary language (approx 11333 from
each languages). We train the mBART model with the auxiliary task around 10k
steps. Training time for the auxiliary task is around 2-3 hours. The fine-
tuning times for TS, QG, NHG, and DG were around 4-5, 1-2, 1-2, and 2-3 hours.
We observe a longer fine-tuning time for ATS because of long passages. We
selected the best model based on loss and perplexity on the validation
datasets. We checked with early-stopping and other checkpoints, which resulted
in poor performance.

###  7.2 Evaluation Metric and Tokenizer Details:

For Automated evaluation, we use sacreBLEU implementation, ROUGE-L, and
BERTScore. For ATS and NHG tasks, ROUGE-1, ROUGE-2, and ROUGE-L are used. We
explicitly use community-adopted language specific-tokenizers. Links for
language-specific tokenizers are given below:

  * •

English: Default sacreBLEU tokenizer i.e, https://github.com/mjpost/sacrebleu

  * •

Hindi: https://anoopkunchukuttan.github.io/indic_nlp_library/

  * •

Japanese: http://www.phontron.com/kytea/

Links of publicly available implementations of automated evaluation metrics
which we use directly in this work:

  * •

BLEU: https://github.com/mjpost/sacrebleu

  * •

ROUGE: https://github.com/pltrdy/files2rouge

  * •

BERTScore: https://github.com/Tiiiger/bert_score

###  7.3 Few Zero-shot Generated outputs from ZmBART:

In the next few figures, we present sample outputs generated by the model in
zero-shot setups, for Hindi and Japanese languages.

Figure 5: Sample outputs for zero-short NHG, ATS, QG and DG in Hindi language
Figure 6: Sample outputs for zero-shot NHG, ATS and QG in Japanese language


# Locally-Contextual Nonlinear CRFs for Sequence Labeling

Harshil Shah  Tim Z. Xiao  David Barber

###### Abstract

Linear chain conditional random fields (CRFs) combined with contextual word
embeddings have achieved state of the art performance on sequence labeling
tasks. In many of these tasks, the identity of the neighboring words is often
the most useful contextual information when predicting the label of a given
word. However, contextual embeddings are usually trained in a task-agnostic
manner. This means that although they may encode information about the
neighboring words, it is not guaranteed. It can therefore be beneficial to
design the sequence labeling architecture to directly extract this information
from the embeddings. We propose locally-contextual nonlinear CRFs for sequence
labeling. Our approach directly incorporates information from the neighboring
embeddings when predicting the label for a given word, and parametrizes the
potential functions using deep neural networks. Our model serves as a drop-in
replacement for the linear chain CRF, consistently outperforming it in our
ablation study. On a variety of tasks, our results are competitive with those
of the best published methods. In particular, we outperform the previous state
of the art on chunking on CoNLL 2000 and named entity recognition on OntoNotes
5.0 English.

Machine Learning, ICML

##  1 Introduction

In natural language processing, sequence labeling tasks involve labeling every
word in a sequence of text with a linguistic tag. These tasks were
traditionally performed using shallow linear models such as hidden Markov
models (HMMs) (Kupiec, 1992; Bikel et al., 1999) and linear chain conditional
random fields (CRFs) (Lafferty et al., 2001; McCallum & Li, 2003; Sha &
Pereira, 2003). These approaches model the dependencies between adjacent word-
level labels. However when predicting the label for a given word, they do not
directly incorporate information from the surrounding words in the sentence
(known as ‘context’). As a result, linear chain CRFs combined with deeper
models which do use such contextual information (e.g. convolutional and
recurrent networks) gained popularity (Collobert et al., 2011; Graves, 2012;
Huang et al., 2015; Ma & Hovy, 2016).

Recently, contextual word embeddings such as those provided by pre-trained
language models have become more prominent (Peters et al., 2018; Radford et
al., 2018; Akbik et al., 2018; Devlin et al., 2019). Contextual embeddings
incorporate sentence-level information, therefore they can be used directly
with linear chain CRFs to achieve state of the art performance on sequence
labeling tasks (Yamada et al., 2020; Li et al., 2020b).

Contextual word embeddings are typically trained using a generic language
modeling objective. This means that the embeddings encode contextual
information which can generally be useful for a variety of tasks. However,
because these embeddings are not trained for any specific downstream task,
there is no guarantee that they will encode the most useful information for
that task. Certain tasks such as sentiment analysis and textual entailment
typically require global, semantic information about a sentence. In contrast,
for sequence labeling tasks it is often the neighboring words in a sentence
which are most informative when predicting the label for a given word.
Although the contextual embedding of a given word may encode information about
its neighboring words, this is not guaranteed. It can therefore be beneficial
to design the sequence labeling architecture to directly extract this
information from the embeddings (Bhattacharjee et al., 2020).

We therefore propose locally-contextual nonlinear CRFs for sequence labeling.
Our approach extends the linear chain CRF in two straightforward ways.
Firstly, we directly incorporate information from the neighboring embeddings
when predicting the label for a given word. This means that we no longer rely
on each contextual embedding to have encoded information about the neighboring
words in the sentence. Secondly, we replace the linear potential functions
with deep neural networks, resulting in greater modeling flexibility. Locally-
contextual nonlinear CRFs can serve as a drop-in replacement for linear chain
CRFs, and they have the same computational complexity for training and
inference.

We evaluate our approach on chunking, part-of-speech tagging and named entity
recognition. On all tasks, our results are competitive with those of the best
published methods. In particular, we outperform the previous state of the art
on chunking on CoNLL 2000 and named entity recognition on OntoNotes 5.0
English. We also perform an ablation study which shows that both the local
context and nonlinear potentials consistently provide improvements compared to
linear chain CRFs.

##  2 Linear chain CRFs

$\mathbf{h}_{1}$$\mathbf{h}_{2}$$\mathbf{h}_{3}$$\cdots$$\mathbf{h}_{T}$$y_{1}$$y_{2}$$y_{3}$$\cdots$$y_{T}$
Figure 1: The graphical model of the linear chain CRF.

Linear chain CRFs (Lafferty et al., 2001) are popular for various sequence
labeling tasks. These include chunking, part-of-speech tagging and named
entity recognition, all of which involve labeling every word in a sequence
according to a predefined set of labels.

We denote the sequence of words $\mathbf{x}=x_{1},\ldots,x_{T}$ and the
corresponding sequence of labels $\mathbf{y}=y_{1},\ldots,y_{T}$. We assume
that the words have been embedded using either a non-contextual or contextual
embedding model; we denote the sequence of embeddings
$\mathbf{h}=\mathbf{h}_{1},\ldots,\mathbf{h}_{T}$. If non-contextual
embeddings are used, each embedding is a function only of the word at that
time step, i.e. $\mathbf{h}_{t}=\mathbf{f}^{\mathrm{emb-NC}}(x_{t})$. If
contextual embeddings are used, each embedding is a function of the entire
sentence, i.e. $\mathbf{h}_{t}=\mathbf{f}^{\mathrm{emb-C}}_{t}(\mathbf{x})$.

The linear chain CRF is shown graphically in Figure 1; ‘linear’ here refers to
the graphical structure. The conditional distribution of the sequence of
labels $\mathbf{y}$ given the sequence of words $\mathbf{x}$ is parametrized
as:

$\displaystyle
p(\mathbf{y}|\mathbf{x},\theta)=\frac{\prod_{t}\psi_{t}\eta_{t}}{\sum_{\mathbf{y}}\prod_{t}\psi_{t}\eta_{t}}$
(1)

where $\theta$ denotes the set of model parameters. The potentials $\psi_{t}$
and $\eta_{t}$ are defined as:

$\displaystyle\psi_{t}$ $\displaystyle=\psi(y_{t-1},y_{t};\theta)$ (2)
$\displaystyle\eta_{t}$ $\displaystyle=\eta(y_{t},\mathbf{h}_{t};\theta)$ (3)

The potentials are constrained to be positive and so are parametrized in
$\log$-space. Linear chain CRFs typically use $\log$-linear potential
functions:

$\displaystyle\log\psi_{t}$
$\displaystyle=\mathbf{i}(y_{t-1})^{\mathsf{T}}\mathbf{A}\cdot\mathbf{i}(y_{t})$
(4) $\displaystyle\log\eta_{t}$
$\displaystyle=\mathbf{i}(y_{t})^{\mathsf{T}}\mathbf{B}\cdot\mathbf{h}_{t}$
(5)

where $\mathbf{i}(y)$ denotes the one-ht encoding of $y$, and $\mathbf{A}$ and
$\mathbf{B}$ are parameters of the model. Note that ‘$\log$-linear’ here
refers to linearity in the parameters. Henceforth, we refer to linear chain
CRFs simply as CRFs.

###  2.1 Incorporating contextual information

When predicting the label at a given time step $y_{t}$, it is often necessary
to use information from words in the sentence other than only $x_{t}$; we
refer to this as contextual information. For example, below are two sentences
from the CoNLL 2003 named entity recognition training set (Tjong Kim Sang & De
Meulder, 2003):

One had threatened to blow it up unless it was refuelled and they were taken
to [LOC London] where they intended to surrender and seek political asylum.
---
[ORG St Helens] have now set their sights on taking the treble by winning the
end-of-season premiership which begins with next Sunday’s semifinal against
[ORG London].

In each example, the word “London” ($x_{t}$) has a different label ($y_{t}$)
and so it is necessary to use contextual information to make the correct
prediction.

In Figure 1, we see that there is no direct path from any $\mathbf{h}_{j}$
with $j\neq t$ to $y_{t}$. This means that in order to use contextual
information, CRFs rely on either/both of the following:

  * •

The transition potentials $\psi(y_{t-1},y_{t};\theta)$ and
$\psi(y_{t},y_{t+1};\theta)$ carrying this information from the labels at
positions $t-1$ and $t+1$.

  * •

The (contextual) embedding $\mathbf{h}_{t}$ encoding information about words
$x_{j}$ with $j\neq t$.

Relying on the transition potentials may not always be effective because
knowing the previous/next label is often not sufficient when labeling a given
word. In the above examples, the previous/next labels indicate that they are
not part of a named entity (i.e. $y_{t-1}=\mathrm{O}$, $y_{t+1}=\mathrm{O}$).
Knowing this does not help to identify whether “London” refers to a location,
an organization, or even a person’s name.

In the first example, knowing that the previous word ($x_{t-1}$) is “to” and
the next word ($x_{t+1}$) is “where” indicates that “London” is a location. In
the second example, knowing that the previous word ($x_{t-1}$) is “against”
indicates that “London” is a sports organization. Therefore, to label these
sentences correctly, a CRF relies on the embedding $\mathbf{h}_{t}$ to encode
the identities of the neighboring words.

However, contextual embedding models are usually trained in a manner that is
agnostic to the downstream tasks they will be used for. Tasks such as
sentiment analysis and textual entailment typically require global sentence-
level context whereas sequence labeling tasks usually require local contextual
information from the neighboring words in the sentence, as shown in the
example above. Because different downstream tasks require different types of
contextual information, it is not guaranteed that task-agnostic contextual
embeddings will encode the most useful contextual information for any specific
task. It can therefore be beneficial to design the architecture for the
downstream task to directly extract the most useful information from the
embeddings.

##  3 The CRF⊗ model

$\mathbf{h}_{1}$$\mathbf{h}_{2}$$\mathbf{h}_{3}$$\cdots$$\mathbf{h}_{T}$$y_{1}$$y_{2}$$y_{3}$$\cdots$$y_{T}$
Figure 2: The graphical model of the CRF⊗.

We introduce the CRF⊗, which extends the CRF by directly using the neighboring
embeddings when predicting the label for a given word. We also replace the
$\log$-linear potential functions with deep neural networks to provide greater
modeling flexibility. We call our model CRF⊗, where $\times$ refers to the
locally-contextual structure and $\ocircle$ refers to the nonlinear
potentials.

The graphical model is shown in Figure 2. The conditional distribution of the
labels $\mathbf{y}$ given the sentence $\mathbf{x}$ is parametrized as
follows:

$\displaystyle
p(\mathbf{y}|\mathbf{x},\theta)=\frac{\prod_{t}\psi_{t}\phi_{t}\eta_{t}\xi_{t}}{\sum_{\mathbf{y}}\prod_{t}\psi_{t}\phi_{t}\eta_{t}\xi_{t}}$
(6)

where the additional potentials $\phi_{t}$ and $\xi_{t}$ are defined as:

$\displaystyle\phi_{t}$ $\displaystyle=\phi(y_{t},\mathbf{h}_{t-1};\theta)$
(7) $\displaystyle\xi_{t}$ $\displaystyle=\xi(y_{t},\mathbf{h}_{t+1};\theta)$
(8)

With this structure, the embeddings $\mathbf{h}_{t-1}$ and $\mathbf{h}_{t+1}$,
in addition to $\mathbf{h}_{t}$, are directly used when modeling the label
$y_{t}$. This means that the model no longer relies on the embedding
$\mathbf{h}_{t}$ to have encoded information about the neighboring words in
the sentence. We evaluate a more general parametrization in Appendix B but
find its empirical performance to be worse. We also evaluate a wider
contextual window in Appendix C but find that it does not provide improved
results.

Since the labels $y_{t}$ are discrete, the parametrization of $\log\psi_{t}$
remains the same as in Equation (4). However unlike the $\log$-linear
parametrization in Equation (5), $\log\eta_{t}$ along with $\log\phi_{t}$ and
$\log\xi_{t}$ are each parametrized by feedforward networks which take as
input the embeddings $\mathbf{h}_{t-1}$, $\mathbf{h}_{t}$ and
$\mathbf{h}_{t+1}$ respectively (Peng et al., 2009):

$\displaystyle\log\phi_{t}=\mathbf{i}(y_{t})^{\mathsf{T}}\mathbf{f}^{\phi}(\mathbf{h}_{t-1})$
(9)
$\displaystyle\log\eta_{t}=\mathbf{i}(y_{t})^{\mathsf{T}}\mathbf{f}^{\eta}(\mathbf{h}_{t})$
(10)
$\displaystyle\log\xi_{t}=\mathbf{i}(y_{t})^{\mathsf{T}}\mathbf{f}^{\xi}(\mathbf{h}_{t+1})$
(11)

where $\mathbf{f}^{\phi}$, $\mathbf{f}^{\eta}$ and $\mathbf{f}^{\xi}$ are the
feedforward networks.

$\mathbf{h}_{1}$$\mathbf{h}_{2}$$\mathbf{h}_{3}$$\cdots$$\mathbf{h}_{T}$$\mathbf{g}_{1}$$\mathbf{g}_{2}$$\mathbf{g}_{3}$$\cdots$$\mathbf{g}_{T}$$y_{1}$$y_{2}$$y_{3}$$\cdots$$y_{T}$
Figure 3: The graphical model of the CRF⊗ combined with a bidirectional LSTM.

### Training

We train the model to maximize $\log p(\mathbf{y}|\mathbf{x},\theta)$ with
respect to the set of parameters $\theta=\\{\mathbf{A},\phi,\eta,\xi\\}$. The
objective can be expressed as:

$\displaystyle\log p(\mathbf{y}|\mathbf{x},\theta)=$
$\displaystyle\sum_{t}\left[\log\psi_{t}+\log\phi_{t}+\log\eta_{t}+\log\xi_{t}\right]$
$\displaystyle-\log\sum_{\mathbf{y}}\prod_{t}\psi_{t}\phi_{t}\eta_{t}\xi_{t}$
(12)

The first term is straightforward to compute. The second term can be computed
using dynamic programming, analogous to computing the likelihood in HMMs
(Rabiner, 1989). We initialize:

$\displaystyle\alpha(y_{1})=\psi_{1}\eta_{1}$ (13)

Then, for $t=2,\ldots,T$:

$\displaystyle\alpha(y_{t})=\sum_{y_{t-1}}\alpha(y_{t-1})\psi_{t}\phi_{t}\eta_{t}\xi_{t-1}$
(14)

Finally:

$\displaystyle\sum_{\mathbf{y}}\prod_{t}\psi_{t}\phi_{t}\eta_{t}\xi_{t}=\sum_{y_{T}}\alpha(y_{T})$
(15)

Denoting $\mathcal{Y}$ as the set of possible labels, the time complexity of
this recursion is $O(T|\mathcal{Y}|^{2})$, the same as for the CRF.

### Inference

During inference, we want to find $\mathbf{y}^{*}$ such that:

$\displaystyle\mathbf{y}^{*}=\operatorname*{arg\,max}_{\mathbf{y}}\ \log
p(\mathbf{y}|\mathbf{x})$ (16)

This can be done using the Viterbi algorithm (Viterbi, 1967), which simply
replaces the sum in Equation (14) with a $\max$ operation.

The key advantage of the CRF⊗ is that it uses enhanced local contextual
information while retaining the computational tractability and parsimony of
the CRF. As discussed in Section 2.1, sequence labeling tasks will benefit
from this enhanced local context, as demonstrated empirically in the
experiments and ablation study in Sections 6 and 7 respectively.

###  3.1 Combination with bi-LSTMs

Huang et al. (2015) augment the CRF with a bidirectional LSTM layer in order
to use global, sentence-level contextual information when predicting the label
for a given word. The CRF⊗ can similarly be combined with a bidirectional LSTM
layer.

Figure 2 shows that with the CRF⊗, there is still no direct path from any
$\mathbf{h}_{t\pm j}$ with $j\geq 2$ to $y_{t}$. Therefore, particularly when
using non-contextual embeddings, encoding wider contextual information by
using a bidirectional LSTM can be very useful.

The graphical model of the CRF⊗ combined with a bidirectional LSTM is shown in
Figure 3. The sequence of embeddings $\mathbf{h}_{1},\ldots,\mathbf{h}_{T}$ is
fed to a bidirectional LSTM to produce a sequence of states
$\mathbf{g}_{1},\ldots,\mathbf{g}_{T}$. These states are then used as the
inputs to the feedforward networks in Equations (9) to (11) instead of the
embeddings.

##  4 Related work

There have been several approaches to directly incorporate contextual
information into sequence labeling architectures. The Conv-CRF (Collobert et
al., 2011), biLSTM-CRF (Huang et al., 2015), and biLSTM-CNN-CRF (Ma & Hovy,
2016) each augment the CRF with either a convolutional network, a
bidirectional LSTM, or both in order to use contextual features when labeling
a given word. More recently, Zhang et al. (2018) propose an alternative LSTM
structure for encoding text which consists of a parallel state for each word,
achieving improved results over the biLSTM-CRF for sequence labeling tasks.
GCDT (Liu et al., 2019) improves an RNN-based architecture by augmenting it
with a sentence-level representation which captures wider contextual
information. Luo et al. (2020) learn representations encoding both sentence-
level and document-level features for named entity recognition using a
hierarchical bidirectional LSTM. Luoma & Pyysalo (2020) leverage the fact that
BERT can represent inputs consisting of several sentences in order to use
cross-sentence context when performing named entity recognition.

In work done concurrently to ours, Hu et al. (2020) also evaluate locally-
contextual parametrisations of the CRF and find that the local context
consistently improves results.

We compare our approach against the best of these methods in Section 6.

##  5 Datasets

Dataset | Task | Labels | Train | Validation | Test
---|---|---|---|---|---
CoNLL 2000 | Chunking | 11 | 7,936* | 1,000* | 2,012
Penn Treebank | POS | 45 | 38,219 | 5,527 | 5,462
CoNLL 2003 | NER | 4 | 14,987 | 3,466 | 3,684
OntoNotes | NER | 18 | 59,924 | 8,528 | 8,262

Table 1: Statistics of each of the datasets used. We use the standard splits
for all datasets.

*The CoNLL 2000 dataset does not include a validation set. We therefore randomly sample 1,000 sentences from the training set to use for validation.

Model | F1
---|---
CRF⊗(GloVe) | 96.12
CRF⊗(GloVe, biLSTM) | 96.14
CRF⊗(BERT) | 97.40
CRF⊗(Flair) | 97.52
Liu et al. (2019) | 97.30
Clark et al. (2018) | 97.00
Akbik et al. (2018) | 96.72
Table 2: Chunking results on the test set of the CoNLL 2000 dataset. Model | Accuracy
---|---
CRF⊗(GloVe) | 97.15
CRF⊗(GloVe, biLSTM) | 97.15
CRF⊗(BERT) | 97.24
CRF⊗(Flair) | 97.56
Bohnet et al. (2018) | 97.96
Akbik et al. (2018) | 97.85
Ling et al. (2015) | 97.78
Table 3: Part-of-speech tagging results on the test set of the Penn Treebank dataset. Model | F1
---|---
CRF⊗(GloVe) | 91.81
CRF⊗(GloVe, biLSTM) | 91.34
CRF⊗(BERT) | 93.82
CRF⊗(Flair) | 94.22
Yamada et al. (2020) | 94.30
Luoma & Pyysalo (2020) | 93.74
Baevski et al. (2019) | 93.50
Table 4: Named entity recognition results on the test set of the CoNLL 2003 dataset. Model | F1
---|---
CRF⊗(GloVe) | 88.70
CRF⊗(GloVe, biLSTM) | 89.04
CRF⊗(BERT) | 92.17
CRF⊗(Flair) | 91.54
Li et al. (2020b) | 92.07
Yu et al. (2020) | 91.30
Li et al. (2020a) | 91.11
Table 5: Named entity recognition results on the test set of the OntoNotes 5.0
dataset.

Figure 4: Results of the ablation study on each of the test sets. We compare
the results when augmenting the CRF with locally-contextual connections,
nonlinear potential functions, and both.

We evaluate the CRF⊗ on the classic sequence labeling tasks of chunking, part-
of-speech tagging, and named entity recognition. We use the following datasets
(summary statistics for each are shown in Table 1):

#### Chunking

Chunking consists of dividing sentences into syntactically correlated parts of
words according to a set of predefined chunk types. This task is evaluated
using the span F1 score.

We use the CoNLL 2000 dataset (Tjong Kim Sang & Buchholz, 2000). An example
sentence from the training set is shown below:

[NP He] [VP reckons] [NP the current account deficit] [VP will narrow] [PP to]
[NP only # 1.8 billion] [PP in] [NP September].
---

#### Part-of-speech tagging

Part-of-speech tagging consists of labeling each word in a sentence according
to its part-of-speech. This task is evaluated using accuracy.

We use the Wall Street Journal portion of the Penn Treebank dataset (Marcus et
al., 1993). An example sentence from the training set is shown below:

[NN Compound] [NNS yields] [VBP assume] [NN reinvestment] [IN of] [NNS
dividends] [CC and] [IN that] [DT the] [JJ current] [NN yield] [VBZ continues]
[IN for] [DT a] [NN year].
---

#### Named entity recognition

Named entity recognition consists of locating and classifying named entities
in sentences. The entities are classified under a pre-defined set of entity
categories.

We use the CoNLL 2003 English (Tjong Kim Sang & De Meulder, 2003) and
OntoNotes 5.0 English (Pradhan et al., 2013) datasets. The CoNLL 2003 dataset
consists of 4 high-level entity types while the OntoNotes 5.0 dataset involves
18 fine-grained entity types. An example from the OntoNotes 5.0 training set
is shown below:

[NORP Japanese] Prime Minister [PERSON Junichiro Koizumi] also arrived in [GPE
Pusan] [TIME this afternoon], beginning his [ORG APEC] trip this time.
---

These are all important tasks in natural language processing and play vital
roles in downstream tasks such as dependency parsing, question answering and
relation extraction. Even small improvements on sequence labeling tasks can
provide significant benefits for these downstream tasks (Nguyen & Verspoor,
2018; Park et al., 2015; Liu et al., 2017; Yamada et al., 2020).

##  6 Experiments

Dataset |  Model |  Prediction
---|---|---
CoNLL 2003 |  Ground truth & CRF⊗(BERT) & CRF⊗(Flair) |  [PER Mike Cito], 17, was expelled from [ORG St Pius X High School] in [LOC Albuquerque] after an October game in which he used the sharpened chin strap buckles to injure two opposing players and the referee.
CRF(BERT) |  [PER Mike Cito], 17, was expelled from [ORG St Pius X] High School in [LOC Albuquerque] after an October game in which he used the sharpened chin strap buckles to injure two opposing players and the referee.
CRF(Flair) |  [PER Mike Cito], 17, was expelled from [LOC St Pius X High School] in [LOC Albuquerque] after an October game in which he used the sharpened chin strap buckles to injure two opposing players and the referee.
OntoNotes 5.0 |  Ground truth & CRF⊗(BERT) & CRF⊗(Flair) |  It is [TIME three hours] by car to [GPE Hong Kong] and [TIME one and a half hours] by boat to [GPE Humen].
CRF(BERT) |  It is [CARDINAL three] hours by car to [GPE Hong Kong] and [TIME one and a half hours] by boat to [GPE Humen].
CRF(Flair) |  It is [TIME three hours] by car to [GPE Hong Kong] and [CARDINAL one] and [TIME a half hours] by boat to [GPE Humen].
Table 6: Example sentences from the named entity recognition test sets where the CRF⊗(BERT) and CRF⊗(Flair) models make the correct predictions but the CRF(BERT) and CRF(Flair) models make the incorrect predictions. Dataset |  Model |  Prediction
---|---|---
CoNLL 2003 |  Ground truth |  [ORG Bre-X], [ORG Barrick] said to continue [LOC Busang] talks.
CRF⊗(BERT) |  [ORG Bre-X], [PER Barrick] said to continue [ORG Busang] talks.
CRF⊗(Flair) |  [LOC Bre-X], [ORG Barrick] said to continue [MISC Busang] talks.
OntoNotes 5.0 |  Ground truth |  In the near future, [EVENT the Russian Tumen River Region Negotiation Conference] will also be held in [GPE Vladivostok].
CRF⊗(BERT) |  In the near future, the [NORP Russian] [LOC Tumen River] [ORG Region Negotiation Conference] will also be held in [GPE Vladivostok].
CRF⊗(Flair) |  In the near future, the [NORP Russian] [LOC Tumen River Region] [EVENT Negotiation Conference] will also be held in [GPE Vladivostok].
Table 7: Example sentences from the named entity recognition test sets where
the CRF⊗(BERT) and CRF⊗(Flair) models make the incorrect predictions.

###  6.1 Model architectures and training

We train four different versions of the CRF⊗ and compare against the best
published results on each dataset. The variants are as follows:

  * •

CRF⊗(GloVe):

    * –

This uses 300-dimensional GloVe embeddings (Pennington et al., 2014) combined
with the CRF⊗.

  * •

CRF⊗(GloVe, biLSTM)

    * –

This uses 300-dimensional GloVe embeddings combined with a biLSTM and the
CRF⊗. The forward and backward LSTM states each have 300 units.

  * •

CRF⊗(BERT)

    * –

This uses the final (768-dimensional) layer of the BERT-Base model (Devlin et
al., 2019) with the CRF⊗. BERT embeddings are defined on sub-word tokens
rather than on words. Therefore if the BERT tokenizer splits a word into
multiple sub-word tokens, we take the mean of the sub-word embeddings as the
word embedding.

  * •

CRF⊗(Flair)

    * –

This uses Flair embeddings (Akbik et al., 2018) with the CRF⊗. Akbik et al.
(2018) recommend concatenating 100-dimensional GloVe embeddings to their
forward and backward language model embeddings. The resulting Flair embeddings
are 4196-dimensional.

Note that the GloVe embeddings are non-contextual whereas the BERT and Flair
embeddings are contextual. Throughout training, we update the embeddings in
the GloVe versions but do not update the BERT or Flair embedding models.

In our initial experiments, we also trained CRF⊗(BERT, biLSTM) and CRF⊗(Flair,
biLSTM) models but found that these did not improve performance compared to
using the CRF⊗ without the bidirectional LSTM. This is unsurprising because
the BERT and Flair embeddings are already trained to encode sentence-level
context.

In each model, for the feedforward networks $\mathbf{f}^{\phi}$,
$\mathbf{f}^{\eta}$ and $\mathbf{f}^{\xi}$ referred to in Equations (9), (10)
and (11), we use 2 layers with 600 units, ReLU activations and a skip
connection. We train each model using stochastic gradient descent with a
learning rate of 0.001 and Nesterov momentum of 0.9 (Nesterov, 1983). We train
for a maximum of 100,000 iterations, using early stopping on the validation
set.

###  6.2 Results

The results for the four datasets are shown in Tables 2, 3, 4, and 5. Across
all of the tasks, our results are competitive with the best published methods.
We find that the previous state of the art is outperformed by both CRF⊗(BERT)
and CRF⊗(Flair) on chunking on CoNLL 2000 and by CRF⊗(BERT) on named entity
recognition on OntoNotes 5.0.

We find that on all of the tasks, the BERT and Flair versions outperform both
of the GloVe versions. With CRF⊗(Glove, biLSTM), the bidirectional LSTM does
encode sentence-level contextual information. However, it is trained on a much
smaller amount of data than the BERT and Flair embedding models, likely
resulting in its lower scores.

In general, there is little difference in performance between CRF⊗(Glove) and
CRF⊗(Glove, biLSTM). This may suggest that the local context used by the CRF⊗
is sufficient for these tasks and that the wider context provided by the
bidirectional LSTM does not add significant value when using the CRF⊗. This is
consistent with the results provided in Appendix C.

##  7 Ablation study

In this section, we attempt to understand the effects of the local context and
nonlinear potentials when compared to the CRF.

To this end, we train three additional variants of each of the model versions
described in Section 6.1. In each case, we replace the CRF⊗ component with one
of the following:

  * •

CRF×

    * –

This is the same as the CRF⊗, but with linear (instead of nonlinear) potential
functions.

  * •

CRF○

    * –

This removes the local context, i.e. it is the same as the CRF⊗ but with the
$\phi_{t}$ and $\xi_{t}$ potentials removed. The $\eta_{t}$ potential function
is still nonlinear.

  * •

CRF

    * –

This is the linear chain CRF. It is the same as the CRF○, but the $\eta_{t}$
potential function is linear instead of nonlinear.

###  7.1 Results

Figure 4 shows the results with each of these variants. In all cases, we see
that the local context is helpful: for each set of embeddings, the results for
the CRF⊗ are higher than for the CRF○ and the results for the CRF× are higher
than for the CRF. The story is similar for the nonlinear potentials; in almost
all cases, the CRF⊗ is better than the CRF× and the CRF○ is better than the
CRF.

Table 6 shows example sentences from the two named entity recognition test
sets where CRF⊗(BERT) and CRF⊗(Flair) make correct predictions but CRF(BERT)
and CRF(Flair) do not. In the first example, CRF(BERT) mistakes the length of
the organization “St Pius X High School”. This does not happen with
CRF⊗(BERT), most likely because the model is aware that the word after “High”
is “School” and therefore that this is a single organization entity.
Similarly, in the second example both CRF(BERT) and CRF(Flair) struggle to
identify whether or not the quantities are cardinals or if they are related to
the time measurements. In contrast, CRF⊗(BERT) and CRF⊗(Flair) do not suffer
from the same mistakes.

While the CRF⊗ models work well, it is interesting to examine where they still
make errors. Table 7 shows example sentences from the named entity recognition
test sets where CRF⊗(BERT) and CRF⊗(Flair) make incorrect predictions. We see
that in both examples, both models correctly identify the positions of the
named entities but struggle with their types. The first example is one that
even a human may struggle to label without any external knowledge beyond the
sentence itself. In the second example, both models fail to recognize the one
long entity and instead break it down into several smaller, plausible
entities.

###  7.2 Computational efficiency

Table 8 in Appendix A shows the detailed computational efficiency statistics
of each of the models we train. In summary, we find that the CRF⊗ takes
approximately 1.5 to 2 times longer for each training and inference iteration
than the CRF. The CRF× and CRF⊗ models have approximately 3 times as many
parameters as the CRF and CRF○ respectively.

##  8 Discussion and future work

We propose locally-contextual nonlinear CRFs for sequence labeling. Our
approach directly incorporates information from the neighboring embeddings
when predicting the label for a given word, and parametrizes the potential
functions using deep neural networks. Our model serves as a drop-in
replacement for the linear chain CRF, consistently outperforming it in our
ablation study. On a variety of tasks, our results are competitive with those
of the best published methods. In particular, we outperform the previous state
of the art on chunking on CoNLL 2000 and named entity recognition on OntoNotes
5.0 English.

Compared to the CRF, the CRF⊗ makes it easier to use locally-contextual
information when predicting the label for a given word in a sentence. However
if contextual embedding models such as BERT and Flair were jointly trained
with sequence labeling tasks, this may eliminate the need for the additional
locally-contextual connections of the CRF⊗; investigating this would be a
particularly interesting avenue for future work.

The example sentences discussed in Section 7 showed that the CRF⊗ model could
benefit from being combined with a source of external knowledge. Therefore
another fruitful direction for future work could be to augment the CRF⊗ with a
knowledge base. This has been shown to improve performance on named entity
recognition in particular (Kazama & Torisawa, 2007; Seyler et al., 2017; He et
al., 2020).

## References

  * Akbik et al. (2018) Akbik, A., Blythe, D., and Vollgraf, R.  Contextual String Embeddings for Sequence Labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , 2018.
  * Baevski et al. (2019) Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., and Auli, M.  Cloze-driven Pretraining of Self-attention Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ , 2019.
  * Bhattacharjee et al. (2020) Bhattacharjee, K., Ballesteros, M., Anubhai, R., Muresan, S., Ma, J., Ladhak, F., and Al-Onaizan, Y.  To BERT or Not to BERT: Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing EMNLP_ , 2020.
  * Bikel et al. (1999) Bikel, D. M., Schwartz, R., and Weischedel, R. M.  An Algorithm That Learns What‘s in a Name.  _Machine Learning_ , 34, 1999.
  * Bohnet et al. (2018) Bohnet, B., McDonald, R., Simões, G., Andor, D., Pitler, E., and Maynez, J.  Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , 2018.
  * Clark et al. (2018) Clark, K., Luong, M.-T., Manning, C. D., and Le, Q.  Semi-Supervised Sequence Modeling with Cross-View Training.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018.
  * Collobert et al. (2011) Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P.  Natural Language Processing (Almost) from Scratch.  _Journal of Machine Learning Research_ , 12, 2011.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_ , 2019.
  * Graves (2012) Graves, A.  _Supervised Sequence Labelling with Recurrent Neural Networks_.  Springer, 2012.
  * He et al. (2020) He, Q., Wu, L., Yin, Y., and Cai, H.  Knowledge-Graph Augmented Word Representations for Named Entity Recognition.  In _Proceedings of the Thrity-Fourth AAAI Conference on Artificial Intelligence_ , 2020.
  * Hu et al. (2020) Hu, Z., Jiang, Y., Bach, N., Wang, T., Huang, Z., Huang, F., and Tu, K.  An Investigation of Potential Function Designs for Neural CRF.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020.
  * Huang et al. (2015) Huang, Z., Xu, W., and Yu, K.  Bidirectional LSTM-CRF Models for Sequence Tagging.  In _arXiv:1508.01991_ , 2015.
  * Kazama & Torisawa (2007) Kazama, J. and Torisawa, K.  Exploiting Wikipedia as External Knowledge for Named Entity Recognition.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , 2007.
  * Kupiec (1992) Kupiec, J.  Robust Part-of-Speech Tagging Using a Hidden Markov Model.  _Computer Speech & Language_, 6, 1992.
  * Lafferty et al. (2001) Lafferty, J. D., McCallum, A., and Pereira, F. C. N.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data.  In _Proceedings of the 18th International Conference on Machine Learning_ , 2001.
  * Li et al. (2020a) Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., and Li, J.  A Unified MRC Framework for Named Entity Recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020a.
  * Li et al. (2020b) Li, X., Sun, X., Meng, Y., Liang, J., Wu, F., and Li, J.  Dice Loss for Data-imbalanced NLP Tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020b.
  * Ling et al. (2015) Ling, W., Dyer, C., Black, A. W., Trancoso, I., Fermandez, R., Amir, S., Marujo, L., and Luís, T.  Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2015.
  * Liu et al. (2017) Liu, L., Ren, X., Zhu, Q., Zhi, S., Gui, H., Ji, H., and Han, J.  Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2017.
  * Liu et al. (2019) Liu, Y., Meng, F., Zhang, J., Xu, J., Chen, Y., and Zhou, J.  GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 2019.
  * Luo et al. (2020) Luo, Y., Xiao, F., and Zhao, H.  Hierarchical Contextualized Representation for Named Entity Recognition.  In _Proceedings of the Thrity-Fourth AAAI Conference on Artificial Intelligence_ , 2020.
  * Luoma & Pyysalo (2020) Luoma, J. and Pyysalo, S.  Exploring Cross-sentence Contexts for Named Entity Recognition with BERT.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 2020.
  * Ma & Hovy (2016) Ma, X. and Hovy, E.  End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ , 2016.
  * Marcus et al. (1993) Marcus, M. P., Marcinkiewicz, M. A., and Santorini, B.  Building a Large Annotated Corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19, 1993.
  * McCallum & Li (2003) McCallum, A. and Li, W.  Early results for Named Entity Recognition with Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , 2003.
  * Nesterov (1983) Nesterov, Y. E.  A method for solving the convex programming problem with convergence rate $O(1/k^{2})$.  _Dokl. Akad. Nauk SSSR_ , 269, 1983.
  * Nguyen & Verspoor (2018) Nguyen, D. Q. and Verspoor, K.  An Improved Neural Network Model for Joint POS Tagging and Dependency Parsing.  In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ , 2018.
  * Park et al. (2015) Park, S., Kwon, S., Kim, B., Han, S., Shim, H., and Lee, G. G.  Question Answering System using Multiple Information Source and Open Type Answer Merge.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics_ , 2015.
  * Peng et al. (2009) Peng, J., Bo, L., and Xu, J.  Conditional Neural Fields.  In _Advances in Neural Information Processing Systems_ , 2009.
  * Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C.  GloVe: Global Vectors for Word Representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_ , 2014.
  * Peters et al. (2018) Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L.  Deep Contextualized Word Representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics_ , 2018.
  * Pradhan et al. (2013) Pradhan, S., Moschitti, A., Xue, N., Ng, H. T., Björkelund, A., Uryupina, O., Zhang, Y., and Zhong, Z.  Towards Robust Linguistic Analysis using OntoNotes.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_ , 2013.
  * Rabiner (1989) Rabiner, L. R.  A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition.  _Proceedings of the IEEE_ , 77, 1989.
  * Radford et al. (2018) Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I.  Improving Language Understanding by Generative Pre-Training.  Technical report, OpenAI, 2018.
  * Seyler et al. (2017) Seyler, D., Dembelova, T., Corro, L. D., Hoffart, J., and Weikum, G.  KnowNER: Incremental Multilingual Knowledge in Named Entity Recognition.  _CoRR_ , abs/1709.03544, 2017.
  * Sha & Pereira (2003) Sha, F. and Pereira, F.  Shallow Parsing with Conditional Random Fields.  In _Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , 2003.
  * Tjong Kim Sang & Buchholz (2000) Tjong Kim Sang, E. F. and Buchholz, S.  Introduction to the CoNLL-2000 Shared Task Chunking.  In _Fourth Conference on Computational Natural Language Learning and the Second Learning Language in Logic Workshop_ , 2000.
  * Tjong Kim Sang & De Meulder (2003) Tjong Kim Sang, E. F. and De Meulder, F.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL_ , 2003.
  * Viterbi (1967) Viterbi, A. J.  Error bounds for convolutional codes and an asymptotically optimum decoding algorithm.  _IEEE Transactions on Information Theory_ , 13, 1967.
  * Yamada et al. (2020) Yamada, I., Asai, A., Shindo, H., Takeda, H., and Matsumoto, Y.  LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , 2020.
  * Yu et al. (2020) Yu, J., Bohnet, B., and Poesio, M.  Named Entity Recognition as Dependency Parsing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020.
  * Zhang et al. (2018) Zhang, Y., Liu, Q., and Song, L.  Sentence-State LSTM for Text Representation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , 2018.

##  Appendix A Computational efficiency

Model | CoNLL 2000 | Penn Treebank | CoNLL 2003 | OntoNotes 5.0
---|---|---|---|---
| Prms | Trn | Inf | Prms | Trn | Inf | Prms | Trn | Inf | Prms | Trn | Inf
CRF(GloVe) |  $<$0.1 | 0.05 | 0.08 |  $<$0.1 | 0.07 | 0.10 |  $<$0.1 | 0.07 | 0.11 |  $<$0.1 | 0.10 | 0.13
CRF×(GloVe) |  $<$0.1 | 0.07 | 0.10 |  $<$0.1 | 0.10 | 0.14 |  $<$0.1 | 0.11 | 0.16 |  $<$0.1 | 0.14 | 0.18
CRF○(GloVe) | 0.7 | 0.05 | 0.09 | 0.8 | 0.08 | 0.13 | 0.7 | 0.09 | 0.14 | 0.8 | 0.10 | 0.15
CRF⊗(GloVe) | 2.2 | 0.10 | 0.15 | 2.3 | 0.13 | 0.18 | 2.2 | 0.16 | 0.20 | 2.3 | 0.17 | 0.22
CRF(GloVe, biLSTM) | 1.6 | 0.07 | 0.10 | 1.6 | 0.09 | 0.13 | 1.6 | 0.11 | 0.16 | 1.6 | 0.13 | 0.17
CRF×(GloVe, biLSTM) | 1.6 | 0.11 | 0.15 | 1.7 | 0.13 | 0.18 | 1.6 | 0.14 | 0.19 | 1.7 | 0.16 | 0.22
CRF○(GloVe, biLSTM) | 2.4 | 0.07 | 0.12 | 2.4 | 0.11 | 0.16 | 2.4 | 0.13 | 0.17 | 2.4 | 0.13 | 0.18
CRF⊗(GloVe, biLSTM) | 3.8 | 0.12 | 0.18 | 3.9 | 0.16 | 0.22 | 3.8 | 0.16 | 0.23 | 3.9 | 0.20 | 0.28
CRF(BERT) |  $<$0.1 | 0.05 | 0.09 |  $<$0.1 | 0.12 | 0.17 |  $<$0.1 | 0.13 | 0.19 |  $<$0.1 | 0.18 | 0.23
CRF×(BERT) | 0.1 | 0.09 | 0.14 | 0.1 | 0.14 | 0.21 |  $<$0.1 | 0.16 | 0.23 | 0.1 | 0.18 | 0.26
CRF○(BERT) | 1.3 | 0.07 | 0.11 | 1.3 | 0.15 | 0.20 | 1.3 | 0.15 | 0.21 | 1.3 | 0.18 | 0.25
CRF⊗(BERT) | 3.9 | 0.11 | 0.18 | 4.0 | 0.22 | 0.29 | 3.9 | 0.22 | 0.31 | 4.0 | 0.23 | 0.33
CRF(Flair) | 0.1 | 0.20 | 0.27 | 0.2 | 0.36 | 0.42 |  $<$0.1 | 0.46 | 0.58 | 0.2 | 0.50 | 0.64
CRF×(Flair) | 0.3 | 0.29 | 0.39 | 0.6 | 0.38 | 0.43 | 0.1 | 0.55 | 0.65 | 0.5 | 0.56 | 0.67
CRF○(Flair) | 5.5 | 0.25 | 0.35 | 5.6 | 0.36 | 0.45 | 5.4 | 0.46 | 0.60 | 5.6 | 0.54 | 0.66
CRF⊗(Flair) | 16.5 | 0.35 | 0.48 | 16.9 | 0.52 | 0.63 | 16.3 | 0.62 | 0.74 | 16.7 | 0.68 | 0.81
Table 8: Computational efficiency statistics of each the models trained. The
Prms column shows the number of parameters, in millions. Trn shows the average
time taken, in seconds, for a training iteration with a batch of 128
sentences. Inf shows the average time taken, in seconds, to infer the most
likely labels for a batch of 256 sentences. All values were computed using the
same GPU.

Table 8 shows the detailed computational efficiency statistics of each of the
models we train. For a given embedding model, we find that the CRF⊗ takes
approximately 1.5 to 2 times longer for each training and inference iteration
than the CRF. When not using the bidirectional LSTM, the CRF× and CRF⊗ have
approximately 3 times as many parameters as the CRF and CRF○ respectively.
However when using the bidirectional LSTM, this component dominates the number
of parameters. This means that the CRF×(GloVe, biLSTM) has approximately the
same number of parameters as the CRF(GloVe, biLSTM) and the CRF⊗(GloVe,
biLSTM) only has approximately 1.6 times as many parameters as the CRF○(GloVe,
biLSTM).

##  Appendix B A more general parametrization

$\mathbf{h}_{1}$$\mathbf{h}_{2}$$\mathbf{h}_{3}$$\cdots$$\mathbf{h}_{T}$$y_{1}$$y_{2}$$y_{3}$$\cdots$$y_{T}$ Figure 5: The graphical model of the CRF⊗-Concat. Model | CoNLL 2000 | Penn Treebank | CoNLL 2003 | OntoNotes 5.0
---|---|---|---|---
| F1 | Accuracy | F1 | F1
CRF⊗(BERT) | 97.40 | 97.24 | 93.82 | 92.17
CRF⊗-Concat(BERT) | 97.23 | 97.10 | 93.10 | 91.80
CRF⊗(Flair) | 97.52 | 97.56 | 94.22 | 91.54
CRF⊗-Concat(Flair) | 97.48 | 97.44 | 93.90 | 91.30
Table 9: Test set results with a more general parametrization of the local
context.

A more general parametrization of the local context would, at each time step,
have a single potential for $\mathbf{h}_{t-1}$, $\mathbf{h}_{t}$,
$\mathbf{h}_{t+1}$ and $y_{t}$ instead of three separate ones as in Section 3.

The graph of this model, which we refer to as CRF⊗-Concat, is shown in Figure
5. We define the following potential:

$\displaystyle\sigma_{t}=\sigma(y_{t},\mathbf{h}_{t-1},\mathbf{h}_{t},\mathbf{h}_{t+1};\theta)$
(17)

Then, the model is parametrized as follows:

$\displaystyle p(\mathbf{y}|\mathbf{x},\theta)$
$\displaystyle=\frac{\prod_{t}\psi_{t}\sigma_{t}}{\sum_{\mathbf{y}}\prod_{t}\psi_{t}\sigma_{t}}$
(18)

The parametrization of $\log\psi_{t}$ remains the same as in Equation (4). We
parametrize $\log\sigma_{t}$ as follows:

$\displaystyle\log\sigma_{t}=\mathbf{i}(y_{t})^{\mathsf{T}}\mathbf{f}^{\sigma}([\mathbf{h}_{t-1};\mathbf{h}_{t};\mathbf{h}_{t+1}])$
(19)

$\mathbf{f}^{\sigma}$ is a feedforward network which takes as input the
concatenated embeddings $\mathbf{h}_{t-1}$, $\mathbf{h}_{t}$, and
$\mathbf{h}_{t+1}$.

We train this model with BERT and Flair embeddings. As per Section 6.1,
$\mathbf{f}^{\sigma}$ has 2 layers with 600 units, ReLU activations and a skip
connection. This means that for any choice of word embeddings, CRF⊗ and
CRF⊗-Concat have approximately the same number of parameters.

The results are shown in Table 9. We see that in all cases, CRF⊗ performs
better than CRF⊗-Concat. This result is likely due to CRF⊗ being easier to
train than CRF⊗-Concat: as well as having better test set scores, we find that
CRF⊗ consistently achieves a higher value for the training objective than
CRF⊗-Concat, which suggests that it finds a better local optimum.

##  Appendix C Widening the contextual window

$\mathbf{h}_{1}$$\mathbf{h}_{2}$$\mathbf{h}_{3}$$\cdots$$\mathbf{h}_{T}$$y_{1}$$y_{2}$$y_{3}$$\cdots$$y_{T}$ Figure 6: The graphical model of the CRF⊗-Wide. Model | CoNLL 2000 | Penn Treebank | CoNLL 2003 | OntoNotes 5.0
---|---|---|---|---
| F1 | Accuracy | F1 | F1
CRF⊗(BERT) | 97.40 | 97.24 | 93.82 | 92.17
CRF⊗-Wide(BERT) | 97.39 | 97.19 | 93.81 | 92.15
CRF⊗(Flair) | 97.52 | 97.56 | 94.22 | 91.54
CRF⊗-Wide(Flair) | 97.50 | 97.53 | 94.23 | 91.44
Table 10: Test set results with a wider local context.

The CRF⊗ as described in Section 3 has a local context of width 3. That is,
the embeddings $\mathbf{h}_{t-1}$, $\mathbf{h}_{t}$ and $\mathbf{h}_{t+1}$ are
directly used when modeling the label $y_{t}$. One may consider that the
performance of the model would be better with a wider context.

The graph of this model, which we refer to as CRF⊗-Wide, is shown in Figure 6.
We define the following potentials, in addition to those in Equations (2),
(3), (7) and (8):

$\displaystyle\pi_{t}$ $\displaystyle=\pi(y_{t},\mathbf{h}_{t-2};\theta)$ (20)
$\displaystyle\zeta_{t}$ $\displaystyle=\zeta(y_{t},\mathbf{h}_{t+2};\theta)$
(21)

The model is then parametrized as:

$\displaystyle p(\mathbf{y}|\mathbf{x},\theta)$
$\displaystyle=\frac{\prod_{t}\psi_{t}\pi_{t}\phi_{t}\eta_{t}\xi_{t}\zeta_{t}}{\sum_{\mathbf{y}}\prod_{t}\psi_{t}\pi_{t}\phi_{t}\eta_{t}\xi_{t}\zeta_{t}}$
(22)

As with the parametrization of the potentials $\phi_{t}$, $\eta_{t}$ and
$\xi_{t}$ in Equations (9) to (11), $\log\pi_{t}$ and $\log\zeta_{t}$ are
parametrized using feedforward networks $\mathbf{f}^{\pi}$ and
$\mathbf{f}^{\zeta}$ whose inputs are the embeddings $\mathbf{h}_{t-2}$ and
$\mathbf{h}_{t+2}$ respectively:

$\displaystyle\log\pi_{t}=\mathbf{i}(y_{t})^{\mathsf{T}}\mathbf{f}^{\pi}(\mathbf{h}_{t-2})$
(23)
$\displaystyle\log\zeta_{t}=\mathbf{i}(y_{t})^{\mathsf{T}}\mathbf{f}^{\zeta}(\mathbf{h}_{t+2})$
(24)

We train this model with BERT and Flair embeddings. As per Section 6.1,
$\mathbf{f}^{\pi}$ and $\mathbf{f}^{\zeta}$ each have 2 layers with 600 units,
ReLU activations and a skip connection.

The results are shown in Table 10. In general, we see that there is very
little difference in performance between CRF⊗ and CRF⊗-Wide. This suggests
that the local context used by the CRF⊗ is sufficiently wide for the tasks
evaluated.