#  EASE: An Easily-Customized Annotation System Powered by
Efficiency Enhancement Mechanisms

Naihao Deng∗♢ Yikai Liu∗♢ Mingye Chen∗♢ Winston Wu♢
Siyang Liu♢ Yulong Chen† Yue Zhang† Rada Mihalcea♢
♢ University of Michigan †Westlake University
<EMAIL_ADDRESS>
###### Abstract

**footnotetext: Naihao Deng, Yikai Liu, and Mingye Chen contributed equally to
the manuscript.

The performance of current supervised AI systems is tightly connected to the
availability of annotated datasets. Annotations are usually collected through
annotation tools, which are often designed for specific tasks and are
difficult to customize. Moreover, existing annotation tools with an active
learning mechanism often only support limited use cases. To address these
limitations, we present EASE, an Easily-Customized Annotation System Powered
by Efficiency Enhancement Mechanisms. EASE provides modular annotation units
for building customized annotation interfaces and also provides multiple back-
end options that suggest annotations using (1) multi-task active learning; (2)
demographic feature based active learning; (3) a prompt system that can query
the API of large language models. We conduct multiple experiments and user
studies to evaluate our system’s flexibility and effectiveness. Our results
show that our system can meet the diverse needs of NLP researchers and
significantly accelerate the annotation process.

##  1 Introduction

The surge of data-driven methods has accentuated the demand for faster data
annotation, with various tasks and applications requiring customized data
collection solutions. Recent years have witnessed a significant increase in
the number of proposed benchmarks (Zhang et al., 2021), and a myriad of tasks
and datasets have been emerging to address the resource bottleneck of AI
applications (Deng et al., 2009; Bowman et al., 2015; Rajpurkar et al., 2016;
Swayamdipta et al., 2020; Mazumder et al., 2022), suggesting the necessity for
data annotation systems that allow fast and economic data construction under
diverse needs. Many existing annotation systems either are designed for
specific tasks, or do not adequately assist the annotator in efficiently
providing annotations. In this paper, we propose EASE, an Easily-Customized
Annotation System Powered by Efficiency Enhancement Mechanisms, aiming to
assist researchers with high-volume data annotation in more effective ways.
EASE has two main advantages over existing annotation interfaces:

(1) EASE is flexible, supporting a wide range of customization options.
Existing annotation tools mostly support annotations for specific tasks (Li et
al., 2021) or have fixed templates that are difficult to customize to meet the
specific needs of researchers. In contrast, we define a series of minimum
front-end modular units, which researchers can specify using JSON. EASE will
then render the corresponding interface. This allows researchers to build the
annotation interface according to their intents while exempting them from the
tedious work of coding the interface from scratch.

Figure 1: System workflow for EASE. We provide a customizable front end
interface (Section 2) with 3 back-end options of multi-task active learning
(Section 4), active learning with demographic features (Section 5), prompt
query API (Section 6).

Task Family Task Dataset Input Form Annotation Form Text Classification
Ontology Classification DBpedia-14 Zhang et al. (2015) Question Multi-choice
Fact Checking FEVER Thorne et al. (2018) Text Multi-choice Sentiment Analysis
Restaurants Pontiki et al. (2016) Question Scores Textual Entailment SNLI
Bowman et al. (2015) Statements Multi-choice Syntactic Analysis CoNLL 2003
Sang and De Meulder (2003) Question Tag sequence Word Segmentation MSR Gao et
al. (2005) Question Segmentation Text Generation Semantic Parsing WikiSQL
Zhong et al. (2017) Question, Table SQL Summarization CNN/Daily Mails See et
al. (2017) Text Text

Table 1: Some existing NLP datasets with their input forms and annotation
forms.

Component Type How to Use Corresponding Components in Tasks Example Input Form
Text Present questions Questions in QA or passage in Summarization. Figure 5
(a) Table Present tables Tables in Text-to-SQL or TableQA task. Figure 5 (d)
Annotation Form Text Type in the text Summaries in Summarization or
translations in Machine Translation. Figure 5 (a) Multi-Choice Select an
option Inference choices for NLI. Figure 5 (b) Segmentation Use cursors to
highlight text span Answer span for Reading Comprehension. Figure 5 (c)
Dropdown Select an option in the dropdown menu, then use cursors to highlight
the corresponding text span Entities and their pos-tags in POS Tagging. Figure
5 (e) Slider Move the Slider Scores in Textual Semantic Similarity. Figure 5
(f)

Table 2: Components provided in EASE to support the construction of customized
annotation interfaces.

(2) EASE boosts annotation efficiency with diverse options for annotation
suggestion, as shown in Figure 1. First, the flexibility of our annotation
interface allows researchers to collect data for multiple sub-tasks at the
same time. Correspondingly, EASE includes a multi-task active learning (MTAL)
system for annotation suggestions in such a setting. Hence, researchers can
adopt a single versatile active learning model which reuses parameters, rather
than having multiple models each dedicated to a single task. Second, EASE can
collect certain annotator demographic features while collecting annotations.
We construct another active learning system that leverages these demographic
features, such as the age of the annotators, to make suggestions better
aligned with the annotators’ intentions. Third, given the recent advances in
large language models (LLMs) (Brown et al., 2020), our system also provides
the ability to prompt LLMs using an API.

We conduct multiple experiments to demonstrate the effectiveness of our
interface and the diverse annotation options it provides. Additionally,
through user studies from the perspective of both researchers and annotators,
we demonstrate that EASE is flexible and can meet the various needs of NLP
researchers, as well as accelerate the annotation process for annotators. The
video link to our demo system is at
https://www.youtube.com/watch?v=0trF6zpIax0.

##  2 Front-End

Many tasks in NLP can be decomposed into a small set of basic actions. For
example, given a piece of text, an annotator should select from a fixed set of
labels (e.g. NLI) or provide a numeric score (e.g. sentiment analysis). We
surveyed several existing NLP datasets (in Table 1) to identify common basic
actions. We then developed several modular front-end components (in Table 2)
which researchers can use to construct custom interfaces for presenting
examples to annotate and collecting annotations. These components cover many
comprehensive NLP annotation tasks: Figure 5 in Appendix A shows several
interfaces for common NLP tasks constructed using these components. These
interfaces are also provided in EASE.

Figure 2: A custom EASE annotation interface consisting of the Segmentation
and Text components. This interface requires the annotator to complete two
tasks, evidence selection and free-form question answering, at the same time
for one instance.

#### Building Custom Interfaces

In addition, researchers can build their own custom interfaces by simply
specifying the desired components in a JSON file. These components can be
added in any order or repeatedly to meet researchers’ needs. Figure 2 provides
a simple example of a custom interface, consisting of the Segmentation and
Text components. The corresponding JSON to construct this interface is
presented in LABEL:appendix:_customized-json. Appendix A presents more details
about annotation procedures and our implementation.

#### Survey of Front-End Effectiveness

To further motivate and validate the effectiveness of our modular front-end
components and custom interface building, we surveyed ten NLP researchers,
asking them to design a project where they need to collect data, and to
describe in detail the types of data they want to collect. The responses from
this survey (in Appendix G) demonstrate that EASE can fulfill researchers’
needs on a variety of tasks.

##  3 Back-End

To match the flexibility of the front-end interface, we provide several back-
end options to assist the annotator. In the Multi-Task Active Learning back-
end (Section 4), annotators annotate sub-tasks at the same time for the same
instance, and we train a single back-end model on those sub-tasks
simultaneously. The Active Learning with Demographic Features back-end
(Section 5) provides another option for active learning, which leverages
demographic information collected from the annotator to improve the accuracy
of annotation suggestions. With the surge of large language models (LLMs), we
also introduce a Prompt Query System as an alternative back-end (Section 6),
which queries the API of certain LLMs to provide annotation suggestions.

##  4 Back-End Option 1: Multi-Task Active Learning

Some annotation tasks in NLP may require annotating multiple types of
information for the same input, e.g. providing part of speech tags and
labeling named entities in a sentence. In the process of providing answers to
multiple sub-tasks for the same input, annotators may be able to annotate
these sub-tasks more quickly when done together on the same input, rather than
separately, due to their growing familiarity with the sub-tasks through the
annotation process. To test this hypothesis, we conduct a user study with six
annotators, who were tasked with annotating chunk parsing and named entity
recognition for 20 sentences from the CoNLL2003 shared task (Tjong Kim Sang
and De Meulder, 2003). We measured the time it took for them to annotate the
two sub-tasks separately versus together for the same input.

We find that annotators tend to spend less time as they become more familiar
with the tasks, as shown in Figure 3. In addition, annotators become more
efficient when annotating the sub-tasks together instead of annotating them
individually once the annotator gets more familiar with the task. Appendix B
provides more details of the study.

Leveraging this result, we build a multi-task active learning back-end system
to assist annotators on the front-end. We employ a single versatile model that
can suggest annotations for various tasks, which has benefits over training
multiple models each dedicated to an individual task.

Figure 3: Average annotation time for user studies on CONLL2003 datasets. For
“Single task”, annotators annotate chunk parsing and named entity recognition
sub-tasks separately, and we calculate the sum of their annotation time. For
“Multi task”, annotators annotate the two sub-tasks together for the same
instance. The time for each range of 10 annotations is the sum of time for 5
chunk parsing tasks and 5 NER tasks.

###  4.1 Method

Given a model with a linear prediction layer, we define the probability of the
model generating an answer to be the softmax of logits from the linear layer.
That is, given the logits $\\{h_{1},h_{2},\cdots,h_{n}\\}$, where $n$ is the
number of possible answers, the probability of selecting answer $i$ is:

$P_{i}=\frac{e^{h_{i}}}{\sum_{j=1}^{n}e^{h_{j}}}$

Following the uncertainty sampling method from Lewis (1995); Settles (2009),
we query the instance whose best labeling (the predicted label in our case) is
the least confident. The flexibility of the annotation interface builder
allows for collecting data for different tasks for the same input at the same
time. Therefore, we calculate a joint loss $L$ over multiple tasks:

$L=\sum_{i=1}^{m}\alpha_{i}L_{i},$

where $L_{i}$ is the loss for the $i$th task, $\alpha_{i}$ is a weight
hyperparameter.

###  4.2 Experiments

We conduct experiments to investigate the performance of single-task vs.
multi-task active learning using EASE. The data for these experiments are from
the CoNLL 2003 shared task (Tjong Kim Sang and De Meulder, 2003), which
includes chunk parsing and named entity recognition. Appendix C provides more
details about the models we use for our experiments. We train a single model
jointly on both tasks and compare with models trained separately on each task.

#### Results

Table 3 shows models’ performance on the original CoNLL 2003 test set. Active
learning on each task separately yields slightly better results on named
entity recognition, while multi-task active learning with EASE performs better
on chunk parsing. In terms of performance, the difference is only about or
within 1 percent. However, we see substantial gains in terms of training time:
EASE trains in around half of the time than training on each task separately.
Thus, multi-task active learning from EASE saves GPU hours while still
performing comparably to active learning on each task separately. In Appendix
D, Figures 11, 12, 13 and 14 show the model performance on the validation set
as the number of instances increases. EASE can achieve an accuracy of 95% and
80% on chunk parsing and named entity recognition after tuning on 1,000
examples, respectively. After this point, EASE can start suggesting
annotations with high accuracy and f1 scores.

Setting | Task | Acc | F1 | Time (h)
---|---|---|---|---
AL | CP | 94.07 0.07 | 93.57 0.07 | 7.61 0.16
NER | 97.70 0.04 | 90.18 0.23
EASE ${}_{\text{o1}}$ | CP | 94.20 0.07 | 93.73 0.08 | 4.28 0.03
NER | 97.33 0.23 | 88.77 0.78

Table 3: Model performance on chunk parsing (“CP”) and named entity
recognition (“NER”). “AL” refers to active learning, EASE ${}_{\text{o1}}$
refers to the first option of multi-task active learning in Section 4. “Acc”,
“F1” refer to accuracy and f1 scores, respectively. The experiments are on a
single NVIDIA Tesla V100 GPU. We run each experiment 3 times and report the
average scores and the standard deviation as the footnote.

##  5 Back End Option 2: Active Learning With Demographic Features

Prior work has shown that annotators may have different opinions in tasks such
as sentiment analysis or natural language inference (Plank et al., 2014; Aroyo
and Welty, 2015; Pavlick and Kwiatkowski, 2019). In those cases, annotators’
demographic information has been shown to be correlated with their annotations
(Al Kuwatly et al., 2020). Thus, we built EASE’s customizable interface with
the ability to collect basic demographic features of the annotators, such as
age, race, or gender. On the back-end, we develop an active learning method
that incorporates annotators’ demographic features to make personalized
annotation suggestions.

###  5.1 Method

Following Geva et al. (2019), we concatenate the demographic features
$(d_{1},d_{2},\cdots,d_{m})$ to the sentence tokens
$(w_{1},w_{2},\cdots,w_{n})$ and feed the concatenated sequence
$(d_{1},d_{2},\cdots,d_{m},w_{1},w_{2},\cdots,w_{n})$ to the model to make a
prediction, where $d_{j}$ is a single value for one type of demographic
feature (e.g. age), $w_{i}$ is a token in the input instance. We follow the
same active learning paradigm as in multi-task active learning (Section 4).

###  5.2 Experiments

We conduct experiments to investigate the performance of active learning when
augmenting an input passage with the annotators’ age as a demographic feature.
We use the sentiment analysis dataset of (Díaz et al., 2018), a 5-way
classification dataset. Appendix E gives more details about the datasets and
our model.

#### Results

There has been a recent trend in our community to collect different
annotations from different annotators on the same instance Plank (2022). Our
experiments show that under such a setting, providing annotation suggestions
is difficult. If we only provide the input sentence to the active learning
model, the model performs poorly, close to random performance (Table 4), as it
might get confused with different labels given by different annotators on the
same instance. Moreover, datasets such as the Sentiment Analysis dataset
involve over 7k annotators; it is unrealistic to train an annotation
suggestion model for each annotator. However, our active learning model
performs substantially better when the demographic feature of annotator age is
fed along with the input sentence (Table 4). Therefore, by leveraging some
non-essential demographic features such as age, EASE is able to make
annotation suggestions that better align with annotators’ intents.

| Acc
---|---
Random | 26.80 0.86
Statement-Only | 29.61 17.69
EASE ${}_{\text{o2}}$ | 38.04 8.08
Table 4: Model performance on the Sentiment Analysis dataset (Díaz et al.,
2018) (5-way classification). For “Random”, the model randomly chooses from
the 5 labels; The inputs are: “Statement-Only”: only the statement; In EASE
${}_{\text{o2}}$, we concatenate the statement with age as the demographic
feature as discussed in Section 5.

##  6 Back End Option 3: Prompt Query System

In recent years, LLMs have demonstrated an undeniable capability to perform
few-shot tasks and data augmentation Brown et al. (2020); Yoo et al. (2021);
Huang et al. (2022). To this end, EASE supports a third back-end option:
leveraging LLMs to suggest annotations by querying via API calls.

###  6.1 Method

We devise a few-shot prompt consisting of example training annotations,
followed by an instance where the model must provide the annotations. The
prompt structure we use is:

Given the sentence [Sentence1] the [Task Name] are [Answer1]\n\n Given the
sentence [Sentence2] the [Task Name] are [Answer2]\n\n $\cdots$ Given the
sentence [Sentence2] the [Task Name] are,

An example prompt is shown in Appendix F. EASE allows for specifying the
number of few-shot examples as well as multiple methods for selecting these
examples. One way of constructing the prompt is by choosing examples randomly.
Another way is to select similar examples from the training data to construct
the prompt Su et al. (2022). Specifically, to select similar examples, we
provide the input sentence from the training set and test set to the BERT
model. We then take the $N$ training examples that have the highest cosine
similarity scores to the example in the test set.

###  6.2 Experiments

To determine the effectiveness of these prompts, we conduct experiments where
we vary the number of few-shot examples (5 or 10) and the methods for
selecting examples for the prompt (random or similar). In these experiments,
we again use the CoNLL 2003 shared task data, and we query the API of the GPT3
(Brown et al., 2020) text-curie-001 model (6.7B).

#### Results

Performance of the LLMs with various prompt settings is shown in Table 5.
First, we find that the performance of LLMs on the POS tagging and NER tasks
is much worse than the performance of a tuned active learning model (Table 3).
This aligns with other observations that prompt-based methods are not
appropriate for sequence labeling tasks (Hou et al., 2022). The reason could
be that sequence labeling tasks differ from text completion, and the LLMs lack
the knowledge of the syntactic structure of the sentence in order to make
correct predictions. In addition, accuracy on named entity recognition is
unproportionally high, because the model “cheats” by predicting many “O”s (in
BIO tagging) instead of the actual entities. However, due to the input length
limit to the API of GPT3, we cannot feed an unlimited number of examples,
which limits the power of utilizing prompts for annotation suggestions. Our
results show that the prompt might not be suitable to be used as an annotation
suggestion for tasks such as sequence labeling, but researchers might find
prompts useful in other tasks, especially text generation (Chen et al., 2022a,
b).

| Setting | Acc | R | P | F1
---|---|---|---|---|---
CP | Random5 | 18.58 | 18.37 | 5.26 | 8.17
Random10 | 31.34 | 25.84 | 6.71 | 10.66
Sim5 | 24.45 | 26.07 | 4.62 | 7.85
Sim10 | 25.54 | 26.22 | 4.48 | 7.65
NER | Random5 | 55.24 | 1.10 | 1.49 | 1.27
Random10 | 64.66 | 4.41 | 6.24 | 5.17
Sim5 | 59.26 | 8.85 | 0.49 | 0.92
Sim10 | 53.09 | 8.68 | 0.52 | 0.97

Table 5: Prompt performance on chunk parsing (CP) and named entity recognition
(NER) for settings of random selection (Random) and selecting the most similar
examples (Sim) from the training set. Footnote 5 and 10 denote the number of
examples selected to construct the prompt query.

.

##  7 Related Work

As EASE is designed to be accessible, flexible, and efficient for annotators,
we briefly compare our tool with existing annotation tools in these aspects.
First, regarding accessibility, popular existing paid annotation tools such as
Prodigy111prodi.gy are hundreds of dollars for a single license, which may be
cost-prohibitive for small research labs or hobbyists. In contrast, our tool
is free and open-source, which is ideal for researchers to use and extend for
their own annotation purposes.

Second, regarding flexibility, many existing tools have been developed for
individual tasks, such as coreference resolution (Bornstein et al., 2020) or
sentiment analysis (Yimam et al., 2020). Recently, there has been work
developing tools that support multiple tasks within the same framework,
including Doccano (Nakayama et al., 2018) and Crowdaq (Ning et al., 2020).
Others have extended this concept to allow the annotation of multiple tasks at
the same time, including WebAnno (Yimam et al., 2013) and LexiClean (Bikaun et
al., 2021). Similar to these tools, EASE provides several modular front-end
components which researchers can use to construct annotation interfaces for
their own needs. Furthermore, EASE does not need technical knowledge, as
custom interfaces can simply be defined in JSON.

One of EASE’s main contributions is its support of multi-task active learning,
which is not supported by the above tools. For an overview of active learning,
we refer the reader to the survey of Settles (2009). Recent work has focused
on active learning in annotation tools to save annotators’ time and effort.
These include INCEpTION (Klie et al., 2018), which used uncertainty sampling,
and DART (Chang et al., 2020), which proposed a retrieval-based method to
suggest labels to the annotator. Ein-Dor et al. (2020) also presents an
analysis of different example selection methods under data imbalance
scenarios. EASE supports multi-task active learning, where a model can learn
from different task annotations of the same sentence or document. Most
comparable to our tool are ActiveAnno (Wiechmann et al., 2021) and POTATO (Pei
et al., 2022), which also support modular components and multi-task active
learning. However, these papers do not present experiments validating their
multi-task active learning setup. In contrast, we perform extensive
experiments on multi-task active learning with human annotators.

##  8 Conclusion

We present our annotation system EASE, which is designed to be flexible,
customizable, and support efficient annotation collection. EASE consists of a
customizable front-end interface and multiple back-ends for suggesting
annotations. Through user studies, we demonstrate that our modular front-end
components can be used to construct custom interfaces that support various
needs of data collection by NLP researchers. We also conducted several
experiments that showcase EASE’s backends for multi-task active learning,
demographic feature based active learning, and LLM prompting. Our work
continues a recent trend of building customizable front-end annotation
interfaces. Moreover, we design efficient back-end systems that can take
advantage of the custom interface as well as the recent advances of LLMs to
more efficiently gather annotations. Future work might further consider how to
leverage the power of LLMs to ease the annotation process for annotators.

##  9 Limitations

We demonstrated that the front-end components we provide in EASE can cover a
wide range of data collection. But further investigation might be needed to
study annotation for certain custom tasks, especially tasks involving
languages other than Chinese and English. We welcome additional contributions
to EASE.

Due to budget, we only tested the LLM prompts with EASE using the GPT3 text-
curie-001 model, but our framework can be easily adapted to query other LLMs.
With advances in LLMs, researchers might get better performance on tasks such
as sequence labeling by using more powerful models such as ChatGPT.

##  10 Ethics

#### Large Language Models (LLMs)

We use one GPT-3.5 model, text-curies-001, to provide annotation options in
EASE. We only use text-curies-001 for experiments and analysis, which are
presented in § 6. Other usage of GPT-3 or its family (e.g., ChatGPT) is not
included.

#### Human Evaluation

We surveyed ten NLP researchers, who are currently holding research positions
in NLP and have published articles in *ACL venues, to demonstrate that EASE
can cover diverse NLP tasks. The annotation study is conducted by six college
students voluntarily, who are to complete or have completed courses from
universities in the U.S.

## References

  * Al Kuwatly et al. (2020) Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020.  Identifying and measuring annotator bias based on annotators’ demographic characteristics.  In _Proceedings of the Fourth Workshop on Online Abuse and Harms_ , pages 184–190, Online. Association for Computational Linguistics.
  * Aroyo and Welty (2015) Lora Aroyo and Chris Welty. 2015.  Truth is a lie: Crowd truth and the seven myths of human annotation.  _AI Magazine_ , 36(1):15–24.
  * Bikaun et al. (2021) Tyler Bikaun, Tim French, Melinda Hodkiewicz, Michael Stewart, and Wei Liu. 2021\.  LexiClean: An annotation tool for rapid multi-task lexical normalisation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 212–219, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Bornstein et al. (2020) Ari Bornstein, Arie Cattan, and Ido Dagan. 2020.  CoRefi: A crowd sourcing suite for coreference annotation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 205–215, Online. Association for Computational Linguistics.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chang et al. (2020) Ernie Chang, Jeriah Caplinger, Alex Marin, Xiaoyu Shen, and Vera Demberg. 2020.  DART: A lightweight quality-suggestive data-to-text annotation tool.  In _Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations_ , pages 12–17, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).
  * Chen et al. (2022a) Guandan Chen, Jiashu Pu, Yadong Xi, and Rongsheng Zhang. 2022a.  Coherent long text generation by contrastive soft prompt.  In _Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)_ , pages 445–455, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Chen et al. (2022b) Yulong Chen, Yang Liu, Ruochen Xu, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Yue Zhang. 2022b.  Unisumm: Unified few-shot summarization with multi-task pre-training and prefix-tuning.  _arXiv preprint arXiv:2211.09783_.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. IEEE.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Díaz et al. (2018) Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018.  Addressing age-related bias in sentiment analysis.  In _Proceedings of the 2018 chi conference on human factors in computing systems_ , pages 1–14.
  * Ein-Dor et al. (2020) Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020.  Active Learning for BERT: An Empirical Study.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7949–7962, Online. Association for Computational Linguistics.
  * Gao et al. (2005) Jianfeng Gao, Mu Li, Andi Wu, and Chang-Ning Huang. 2005.  Chinese word segmentation and named entity recognition: A pragmatic approach.  _Computational Linguistics_ , 31(4):531–574.
  * Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019.  Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1161–1166, Hong Kong, China. Association for Computational Linguistics.
  * Hou et al. (2022) Yutai Hou, Cheng Chen, Xianzhen Luo, Bohan Li, and Wanxiang Che. 2022.  Inverse is better! fast and accurate prompt for few-shot slot tagging.  _arXiv preprint arXiv:2204.00885_.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022.  Large language models can self-improve.  _arXiv preprint arXiv:2210.11610_.
  * Klie et al. (2018) Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018.  The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation.  In _Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations_ , pages 5–9, Santa Fe, New Mexico. Association for Computational Linguistics.
  * Lewis (1995) David D Lewis. 1995.  A sequential algorithm for training text classifiers: Corrigendum and additional data.  In _Acm Sigir Forum_ , volume 29, pages 13–19. ACM New York, NY, USA.
  * Li et al. (2021) Yanzeng Li, Bowen Yu, Li Quangang, and Tingwen Liu. 2021.  Fitannotator: A flexible and intelligent text annotation system.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations_ , pages 35–41.
  * Mazumder et al. (2022) Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Douwe Kiela, David Jurado, et al. 2022\.  Dataperf: Benchmarks for data-centric ai development.  _arXiv preprint arXiv:2207.10062_.
  * Nakayama et al. (2018) Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018\.  doccano: Text annotation tool for human.  Software available from https://github.com/doccano/doccano.
  * Ning et al. (2020) Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, Ana Marasović, and Zhen Nie. 2020.  Easy, reproducible and quality-controlled data collection with CROWDAQ.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 127–134, Online. Association for Computational Linguistics.
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent disagreements in human textual inferences.  _Transactions of the Association for Computational Linguistics_ , 7:677–694.
  * Pei et al. (2022) Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022.  POTATO: The portable text annotation tool.  In _Proceedings of the The 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 327–337, Abu Dhabi, UAE. Association for Computational Linguistics.
  * Plank (2022) Barbara Plank. 2022.  The “problem” of human label variation: On ground truth in data, modeling and evaluation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Plank et al. (2014) Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014.  Linguistically debatable or just plain wrong? In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 507–511, Baltimore, Maryland. Association for Computational Linguistics.
  * Pontiki et al. (2016) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammed AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. 2016.  Semeval-2016 task 5: Aspect based sentiment analysis.  In _ProWorkshop on Semantic Evaluation (SemEval-2016)_ , pages 19–30. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Reichart et al. (2008) Roi Reichart, Katrin Tomanek, Udo Hahn, and Ari Rappoport. 2008.  Multi-task active learning for linguistic annotations.  In _Proceedings of ACL-08: HLT_ , pages 861–869.
  * Rotman and Reichart (2022) Guy Rotman and Roi Reichart. 2022.  Multi-task active learning for pre-trained transformer-based models.  _arXiv preprint arXiv:2208.05379_.
  * Sang and De Meulder (2003) Erik F Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  _arXiv preprint cs/0306050_.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Settles (2009) Burr Settles. 2009.  Active learning literature survey.
  * Su et al. (2022) Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2022.  Selective annotation makes language models better few-shot learners.  _ArXiv_.
  * Swayamdipta et al. (2020) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9275–9293, Online. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Wiechmann et al. (2021) Max Wiechmann, Seid Muhie Yimam, and Chris Biemann. 2021.  ActiveAnno: General-purpose document-level annotation tool with active learning integration.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations_ , pages 99–105, Online. Association for Computational Linguistics.
  * Yimam et al. (2020) Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele, and Chris Biemann. 2020\.  Exploring Amharic sentiment analysis from social media texts: Building annotation tools and classification models.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1048–1060, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Yimam et al. (2013) Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013.  WebAnno: A flexible, web-based and visually supported system for distributed annotations.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 1–6, Sofia, Bulgaria. Association for Computational Linguistics.
  * Yoo et al. (2021) Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. 2021\.  Gpt3mix: Leveraging large-scale language models for text augmentation.  _arXiv preprint arXiv:2104.08826_.
  * Zhang et al. (2021) Daniel Zhang, Saurabh Mishra, Erik Brynjolfsson, John Etchemendy, Deep Ganguli, Barbara Grosz, Terah Lyons, James Manyika, Juan Carlos Niebles, Michael Sellitto, et al. 2021.  The ai index 2021 annual report.  _arXiv preprint arXiv:2103.06312_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  _Advances in neural information processing systems_ , 28.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017.  Seq2sql: Generating structured queries from natural language using reinforcement learning.  _CoRR_ , abs/1709.00103.

##  Appendix A More Details about Our System

Figure 4: The administrator uploads the data to be annotated, selects the
“annotation type” by choosing pre-defined (“Text Summarization” in this
figure) or his customized version of the interface, and assigns to the
annotator “bob”.

Figure 5: Annotation interfaces constructed by EASE for (a) text
summarization; (b) fake news detection; (c) Chinese word segmentation; (d)
text-to-SQL; (e) text chunking; (f) semantic similarity.

#### Annotation Interface Examples

Figure 5 shows some basic annotation interface examples constructed by EASE,
where the basic components correspond to those described in Table 2.

#### Annotation Procedure

We define two types of roles: administrator and annotator in the system. The
administrator is the person who composes the annotation interface, while
annotators are the people who annotate the data. A typical annotation process
is:

  1. 1.

The administrator designs the task format and uploads the dataset to be
annotated.

  2. 2.

The administrator creates and assigns tasks to specific users, as shown in
Figure 4.

  3. 3.

The annotators annotate the data with annotations suggested by our back-end
system (discussed in Section 3).

  4. 4.

The administrator exports the annotated data in JSON format.

#### Implementation and Logistics

The annotation interface is built using the React.js framework and various
open-source web components. The interface is responsive and can adapt to many
screen sizes. The server-side back-end system uses the Flask framework in
Python to serve REST APIs for interaction with the front-end interface. We use
a SQLite database to store and manage annotation data. For tasks with active
learning support, we define a REST API interface for interaction with the
back-end active learning model using sockets, threads, and JSON
(LABEL:appendix:_customized-json shows an example JSON file for the customized
interface). With this new REST API interface, new models can be connected
easily.

##  Appendix B Multi-Task User Studies

Figure 6: Accuracy scores of annotators’ performance on 30 chunk parsing
examples. “Single task” corresponds to the time they spend on the examples
when they annotate the 2 tasks separately, while “Multi task” corresponds to
the time they spend when they annotate the two tasks (chunk parsing and named
entity recognition) for the same instance at the same time . Figure 7:
Accuracy scores of annotators’ performance on 30 named entity recognition
examples. Figure 8: F1 scores of annotators’ performance on 30 chunk parsing
for user studies. Figure 9: F1 scores of annotators’ performance on 30 named
entity recognition examples.

Figures 7, 6, 9 and 8 show the annotators’ performance on the tasks of chunk
parsing and named entity recognition. We compare the annotations we collect to
the ground truth labels from the original CoNLL2003 to calculate the
annotation performance. In general, annotators perform about the same when
annotating the two tasks separately v.s. annotating tasks for the same
instance at the same time in terms of accuracy and f1 scores. Note that there
is a big fluctuation in the f1 score for named entity recognition because of
the way we calculate the score. The f1 score only considers tags that are not
“O” into account, small variations in annotations might lead to a big f1 score
difference then.

##  Appendix C Experiment Set-ups for Back-End Option 1: Multi-Task Active
Learning

Figure 10: Model structure for multi-task active learning of the BERT base
model. “layer${}_{\text{1},\cdots,\text{12}}$” correspond to encoder layers in
BERT-base encoder. We adopt the same structure following Rotman and Reichart
(2022).

Following Reichart et al. (2008); Rotman and Reichart (2022), we build upon
BERT-base (Devlin et al., 2019) for all experiments. For each task, we add a
linear layer on top of the BERT encoder. For multi-task training, parameters
in the BERT encoder are shared while parameters in the linear layers are not.
Figure 10 shows an example of the model structure for the multi-task active
learning setting.

##  Appendix D EASE Results on CoNLL2003

(a) Named entity recognition (b) Chunk parsing

Figure 11: Validation accuracy v.s. the number of training instances for (a)
named entity recognition and (b) chunk parsing for CoNLL2003 dataset.

(a) Named entity recognition (b) Chunk parsing

Figure 12: Validation recall v.s. the number of training instances for (a)
named entity recognition and (b) chunk parsing for CoNLL2003 dataset.

(a) Named entity recognition (b) Chunk parsing

Figure 13: Validation precision v.s. the number of training instances for (a)
named entity recognition and (b) chunk parsing on the CoNLL 2003 dataset.

(a) Named entity recognition (b) Chunk parsing

Figure 14: Validation f1 v.s. the number of training instances for (a) named
entity recognition and (b) chunk parsing for CoNLL 2003 dataset.

Figures 11, 12, 13 and 14 show the model performance on the validation set as
the training goes. In general EASE ${}_{\text{o1}}$ (multi-task active
learning) performs comparably with active learning.

##  Appendix E Experiment Set-ups for Back End Option 2: Active Learning With
Demographic Features

The Sentiment Analysis dataset Díaz et al. (2018) provides abundant
demographic features such as age, race, gender, etc. For each statement,
annotators choose one of five sentiments: negative, somewhat negative,
neutral, somewhat positive, or positive. We use the BERT-base model with a
linear classification layer in the experiments. We experiment on various
combinations of the demographic features in our preliminary experiments and
find that the model trained with the age concatenated to the input performs
the best.

##  Appendix F Prompt Examples

Figure 15 gives an example of the prompt for the task of named entity
recognition.

Given the sentence ‘‘ 1860 Munich 3 1 0 2 3 5 3 ’’ the entities-recognition
are ‘‘ B-ORG I-ORG O O O O O O O ’’\n\n Given the sentence ‘‘ Production of
cold laminates was 120,500 tonnes , 4.2 percent higher than the same month
last year and 11 percent higher than in June . ’’ the entities-recognition are
‘‘ O O O O O O O O O O O O O O O O O O O O O O O O O ’’\n\n Given the sentence
‘‘ Jul-18.Jul ’’ the entities-recognition are ‘‘ O ’’\n\n Given the sentence
‘‘ 5/8 - Meluawati ( Indonesia ) beat Chan Chia Fong ( Malaysia ) 11-6 ’’ the
entities-recognition are ‘‘ O O B-PER O B-LOC O O B-PER I-PER I-PER O B-LOC O
O ’’\n\n Given the sentence ‘‘ The poll , taken on Sunday and Monday as the
president engaged in a whistle-stop train trip to the Democratic Convention in
Chicago , put Clinton at 51 percent , Dole at 36 percent and Ross Perot of the
Reform Party at 8 percent . ’’ the entities-recognition are ‘‘ O O O O O O O O
O O O O O O O O O O O B-MISC I-MISC O B-LOC O O B-PER O O O O B-PER O O O O
B-PER I-PER O O B-ORG I-ORG O O O O ’’\n\n Given the sentence ‘‘ 1. Ilke
Wyludda ( Germany ) 66.60 metres ’’ the entities-recognition are ‘‘ O B-PER
I-PER O B-LOC O O O ’’\n\n Given the sentence ‘‘ -DOCSTART- ’’ the entities-
recognition are ‘‘ O ’’\n\n Given the sentence ‘‘ Dynamo Kiev 5 Kremin
Kremenchuk 0 ’’ the entities-recognition are ‘‘ B-ORG I-ORG O B-ORG I-ORG O
’’\n\n Given the sentence ‘‘ MANCHESTER , England 1996-08-27 ’’ the entities-
recognition are ‘‘ B-LOC O B-LOC O ’’\n\n Given the sentence ‘‘ Sokol Tychy 5
Lech Poznan 3 ’’ the entities-recognition are ‘‘ B-ORG I-ORG O B-ORG I-ORG O
’’\n\n Given the sentence ‘‘ Longyear is a town in mourning , a close-knit
community that has been shattered . ’’ the entities-recognition are

Figure 15: Sample prompt given to the language model.

##  Appendix G Front-End User Study

###  G.1 Online Survey Content

Here are the questions we asked in the online survey:

  1. 1.

Describe one of the NLP tasks that you want to work on (e.g. text-to-SQL,
summarization, etc)

  2. 2.

Describe the data that you wish to gather for a specific task.

Here are some examples for your reference:

    1. (a)

To investigate individual biases in summarization, data that can be collected
includes individual summaries of a passage and the annotator’s identification
who created it.

    2. (b)

For analyzing models’ performance on various question types in the Natural
Language Inference task, the data collected could include text data in the
form of questions along with their corresponding inferences in a text box.
Additionally, annotators can be asked to categorize the question types by
providing multiple-choice options for classification.

###  G.2 Collected Answers

We collected 10 answers from NLP researchers and list them in the form of NLP
task and their detailed descriptions as follows:

#### Sequence editing

Given the source sentence, apply a set of actions to change the sentence. So
the data will be (source sentence, set of actions, target sentence). Humans
will need to apply several actions and the tool should be able to record those
actions.

#### Counselor response editing

Provide an example of rewriting, then provide a response and ask for a
rewrite. Importantly, I will also provide examples of varying degrees of
content preservation, and ask the annotator to conform to the specified level.

#### Response generation

To investigate the cultural factors behind the conversations among people from
different backgrounds, data that can be collected would include a conversation
on a controversial topic and annotations on the speaker’s background
information and the utterance aligned to those background annotations.

#### Commonsense inference

To study how annotators’ cultural background affects the commonsense
inference, the data can be collected by writing the tail entity given the head
entity and relations or identifying inferences in the existing knowledge base
that are incorrect in some cultures.

#### Text to text generation

To investigate perspective differences of Tweets written by different
languages, Tweets regarding a certain topic and their language labels can be
collected.

#### Chinese sentiment analysis

I want the annotated Chinese texts.

Generally, annotators should be asked to do multiple choices for different
classifications like Positive, Neutral, and Negative. They may also be asked
to label a range of sentiment levels in -3, -2, -1, 0, 1, 2, 3. Other
necessary information includes AnnotatorID, Language background, Education
background (CS, Linguistics, Chinese Studies, High Schools, etc.), and age.

In addition, the task of annotation could be a Chinese sentence. It could also
be a Chinese paragraph. These two methods may result in two results.

If the research needs to compare Chinese sentiment annotations and English
sentiment annotations, it would be better to shuffle the same meaning of texts
in two languages. If the same meaning of texts in two languages appears at the
same time, most annotators may choose the same evaluation.

#### Machine translation

I would want to have paired data for the source and target language, from a
diverse range of sources to get good representations. Therefore, I want to
collect translations for colloquial terms in daily dialogues.

#### Commonsense reasoning QA in cultural contexts

Data could be collected in form of questions about events or actions together
with a set of possible answers. Among those, one of the answers is the most
plausible in the context described in the question.

#### Summarization

For evaluating existing models as well as formulating new ones to summarize
dialogue, data can be collected by obtaining meeting transcripts paired with
meeting summaries authored by participants of that meeting.

#### Dialogue evaluation

Given a dialogue, annotate the level of empathy across multiple dimensions.

###  G.3 Constructed Script

EASE can support all answers we collect in Section G.2. The JSON scripts to
construct these interfaces are provided in our GitHub repo.

Figure 16 provides the interface constructed for Chinese sentiment analysis.

Figure 16: Example interface constructed for Chinese sentiment analysis by
EASE. For display purposes, we translate the Chinese poem into English.

Here we provide an example JSON script to render the annotation interface
corresponding to Figure 16 in Code Snippet LABEL:lst:json-chinese-sentiment-
analysis as the following:

⬇

{

"data": {

"source": [

"Goose, goose, goose,You bend your neck towards the sky and sing. Your white
feathers float on the emerald water,Your red feet push the clear waves.",

"Moonlight reflects off the front of my bed. Could it actually be the frost on
the ground? I look up to view the bright moon, And look down to reminisce
about my hometown.",

"Green hills skirt the northern border, White waters gird the eastern town;
Here we part with each other, And you set out like a lonesome wisp of grass,
Floating across the miles, farther and farther away. You’ve longed to travel
like roaming clouds, But our friendship, unwilling to wane as the sun is to
set, Let it be here to stay. As we wave each other good-bye, Our horses neigh,
as if for us they sigh."

],

"question": [

[

"What is the sentiment of this paragraph?",

"Please level your sentiment (-3 to 3)"

],

[

"What is the sentiment of this paragraph?",

"Please level your sentiment (-3 to 3)"

],

[

"What is the sentiment of this paragraph?",

"Please level your sentiment (-3 to 3)"

]

],

"result": [

[

{

"result": 0

},

{

"result": 0

}

],

[

{

"result": 0

},

{

"result": 0

}

],

[

{

"result": 0

},

{

"result": 0

}

]

],

"done": [

0,

0,

0

]

},

"format": [

{

"type": "button",

"properties": {

"contents": [

"Positive",

"Neutral",

"Negative"

]

}

},

{

"type": "button",

"properties": {

"contents": [

"-3",

"-2",

"-1",

"0",

"1",

"2",

"3"

]

}

}

]

}

% \end{lstlisting}

\twocolumn

\section{Example of JSON file}

\label{appendix: customized-json}

Code Snippet~\ref{lst:json-custom-interface} shows the JSON file that
corresponds to the custom interface as follows:

\onecolumn

\begin{lstlisting}[language=json,firstnumber=1,title = {JSON file for the
customized interface}, captionpos=b, label=lst:json-custom-interface]

{

"data": {

"source": [

"The heat required for boiling the water and supplying the steam can be
derived from various sources, most commonly from burning combustible materials
with an appropriate supply of air in a closed space (called variously
combustion chamber, firebox). In some cases the heat source is a nuclear
reactor, geothermal energy, solar energy or waste heat from an internal
combustion engine or industrial process. In the case of model or toy steam
engines, the heat source can be an electric heating element.",

"The most useful instrument for analyzing the performance of steam engines is
the steam engine indicator. Early versions were in use by 1851, but the most
successful indicator was developed for the high speed engine inventor and
manufacturer Charles Porter by Charles Richard and exhibited at London
Exhibition in 1862. The steam engine indicator traces on paper the pressure in
the cylinder throughout the cycle, which can be used to spot various problems
and calculate developed horsepower. It was routinely used by engineers,
mechanics and insurance inspectors. The engine indicator can also be used on
internal combustion engines. See image of indicator diagram below (in Types of
motor units section)."

],

"question": [

[

"What is the usual source of heat for boiling water in the steam engine?",

"Aside from firebox, what is another name for the space in which combustible
material is burned in the engine?",

"What is the sentiment of this paragraph?",

"Please do the part of speech tagging on this paragraph.",

"Translate the paragraph into Chinese."

],

[

"What instrument is used to examine steam engine performance?",

"What year saw the earliest recorded use of the steam engine indicator?",

"What is the sentiment of this paragraph?",

"Please do the part of speech tagging on this paragraph.",

"Translate the paragraph into Chinese."

]

],

"result": [

[

{

"result": []

},

{

"result": ""

},

{

"result": 0

},

{

"result": []

},

{

"result": ""

}

],

[

{

"result": []

},

{

"result": ""

},

{

"result": 0

},

{

"result": []

},

{

"result": ""

}

]

],

"done": [

0,

0

]

},

"format": [

{

"type": "selection",

"properties": {

"contents": []

}

},

{

"type": "textbox",

"properties": {}

},

{

"type": "button",

"properties": {

"contents": [

"positive",

"negative",

"neutral"

]

}

},

{

"type": "selection",

"properties": {

"contents": [

"NP",

"PP",

"VP"

]

}

},

{

"type": "textbox",

"properties": {}

}

]

}

JSON file for the requested interface of Chinese sentiment analysis. For
display purposes, we translate Chinese poems into English.


# Discriminating Between Similar Nordic Languages

René Haas
IT University of Copenhagen
<EMAIL_ADDRESS>
&Leon Derczynski
IT University of Copenhagen
<EMAIL_ADDRESS>
###### Abstract

Automatic language identification is a challenging problem. Discriminating
between closely related languages is especially difficult. This paper presents
a machine learning approach for automatic language identification for the
Nordic languages, which often suffer miscategorisation by existing state-of-
the-art tools. Concretely we will focus on discrimination between six Nordic
languages: Danish, Swedish, Norwegian (Nynorsk), Norwegian (Bokmål), Faroese
and Icelandic.

##  1 Introduction

Automatic language identification is a challenging problem and especially
discriminating between closely related languages is one of the main
bottlenecks of state-of-the-art language identification systems Zampieri et
al. (2014).

Language technology for Scandinavian languages is in a nascent phase (e.g.
Kirkedal et al. (2019)). One problem is acquiring enough text with which to
train e.g. large language models. Good quality language ID is critical to this
data sourcing, though leading models often confuse similar Nordic languages.

This paper presents a machine learning approach for automatic language
identification between six closely-related Nordic languages: Danish, Swedish,
Norwegian (Nynorsk), Norwegian (Bokmål), Faroese and Icelandic.

This paper explores different ways of extracting features from a corpus of raw
text data consisting of Wikipedia summaries in respective languages and
evaluates the performance of a selection of machine learning models.

Concretely we will compare the performance of classic machine learning models
such as Logistic Regression, Naive Bayes, Support vector machine, and K
nearest Neighbors with more contemporary neural network approaches such as
Multilayer Perceptrons (MLP) and Convolutional Neural Networks (CNNs).

After evaluating these models on the Wikipedia data set we will continue to
evaluate the best models on a data set from a different domain in order to
investigate how well the models generalize when classifying sentences from a
different domain.

##  2 Related Work

The problem of discriminating between similar languages has been investigated
in recent work Goutte et al. (2016); Zampieri et al. (2015) which discuss the
results from two editions of the “Discriminating between Similar Languages
(DSL) shared task". Over the two editions of the DSL shared task different
teams competed to develop the best machine learning algorithms to discriminate
between the languages in a corpus consisting of 20K sentences in each of the
languages: Bosnian, Croatian, Serbian, Indonesian, Malaysian, Czech, Slovak,
Brazil Portuguese, European Portuguese, Argentine Spanish, Peninsular Spanish,
Bulgarian and Macedonian.

##  3 The Nordic DSL Dataset

This section describes the construction of the Nordic DSL (Distinguishing
Similar Lanugages) dataset.

Data was scraped from Wikipedia. We downloaded summaries for randomly chosen
Wikipedia articles in each of the languages, saved as raw text to six .txt
files of about 10MB each. While Bornholmsk would be a welcome addition
Derczynski and Kjeldsen (2019), exhibiting some similarity to Faroese and
Danish, there is not yet enough digital text.

After the initial cleaning (described in the next section) the dataset
contained just over 50K sentences in each of the language categories. From
this, two datasets with exactly 10K and 50K sentences respectively were drawn
from the raw dataset. In this way the datasets are stratified, containing the
same number of sentences for each language.

We split these datasets, reserving 80% for the training set and 20% for the
test set.

###  3.1 Data Cleaning

This section describes how the dataset is initially cleaned and how sentences
are extracted from the raw data.

#### Extracting Sentences

The first pass in sentence tokenisation is splitting by line breaks. We then
extract shorter sentences with the sentence tokenizer (sent_tokenize) function
from the NLTKLoper and Bird (2002) python package. This does a better job than
just splitting by ’.’ due to the fact that abbreviations, which can appear in
a legitimate sentence, typically include a period symbol.

#### Cleaning characters

The initial dataset has many characters that do not belong to the alphabets of
the languages we work with. Often the Wikipedia pages for people or places
contain names in foreign languages. For example a summary might contain
Chinese or Russian characters which are not strong signals for the purpose of
discriminating between the target languages.

Further, it can be that some characters in the target languages are mis-
encoded. These mis-encodings are also not likely to be intrinsically strong or
stable signals.

To simplify feature extraction, and to reduce the size of the vocabulary, the
raw data is converted to lowercase and stripped of all characters with are not
part of the standard alphabet of the six languages.

In this way we only accept the following character set

    ’abcdefghijklmnopqr
    stuvwxyz ’

and replace everything else with white space before continuing to extract the
features. For example the raw sentence

    ’Hesbjerg er dannet ved
    sammenlgning af de 2 grde
    Store Hesbjerg
    og Lille Hesbjerg i 1822.’

will be reduced to

    ’hesbjerg er dannet ved sammenlgning
     af de grde store hesbjerg
     og lille hesbjerg i ’,

We thus make the assumption that capitalisation, numbers and characters
outside this character set do not contribute much information relevant for
language classification.

##  4 Baselines

###  4.1 Baseline With langid.py

As a baseline to compare the performance of the models in we compare with an
off-the-shelf language identification system. “langid.py: An Off-the-shelf
Language Identification Tool." Lui and Baldwin (2012) is such a tool.

langid.py comes with with a pretrained model which covers 97 languages. The
data for langid.py comes from from five different domains: government
documents, software documentation, newswire, online encyclopedia and an
internet crawl. Features are selected for cross-domain stability using the LD
heuristic Lui and Baldwin (2011).

We evaluated how well langid.py performed on the Nordic DSL data set. It is a
peculiar feature of the Norwegian language that there exist two different
written languages but three different language codes. Since langid.py also
returned the language id “no" (Norwegian) on some of the data points we
restrict langid.py to only be able to return either “nn" (Nynorsk) or “nb"
(Bokmål) as predictions.

Figure 1: Confusion matrix with results from langid.py on the full dataset,
300K instances.

Figure 1 shows the confusion matrix for the langid.py classifier. The largest
confusions were between Danish and Bokmål, and between Faroese and Icelandic.
We see that langid.py was able to correctly classify most of the Danish
instances; however, approximately a quarter of the instance in Bokmål were
incorrectly classified as Danish and just under an eighth was misclassified as
Nynorsk.

Furthermore, langid.py correctly classified most of the Icelandic data points;
however, over half of the data points in Faroese were incorrectly classified
as Icelandic.

Model | Encoding | Accuracy
---|---|---
Knn | cbow | 0.780
Log-Reg | cbow | 0.819
Naive Bayes | cbow | 0.660
SVM | cbow | 0.843
Knn | skipgram | 0.918
Log-Reg | skipgram | 0.929
Naive Bayes | skipgram | 0.840
SVM | skipgram | 0.928
Knn | char bi-gram | 0.745
Log-Reg | char bi-gram | 0.907
Naive Bayes | char bi-gram | 0.653
SVM | char bi-gram | 0.905
Knn | char uni-gram | 0.620
Log-Reg | char uni-gram | 0.755
Naive Bayes | char uni-gram | 0.614
SVM | char uni-gram | 0.707
Table 1: Overview of results for the data set with 10K data points in each
language.

###  4.2 Baseline with linear models

Table 1 shows results for running the models on a data set with 10K sentences
in each language category. We see that the models tend to perform better if we
use character bi-grams instead of single characters.

Also we see that logistic regression and support vector machines outperform
Naive Bayes and K-nearest neighbors in all cases. Furthermore, for all models,
we get the best performance if we use the skip-gram model from FastText.

Comparing the CBOW mode from FastText with character bi-grams we see that the
CBOW model is on par with bi-grams for the KNN and Naive Bayes classifiers,
while bi-grams outperform CBOW for Logistic Regression and support vector
machines.

##  5 Our Approach

###  5.1 Using FastText

The methods described above are quite simple. We also compared the above
method with FastText, which is a library for creating word embeddings
developed by Facebook Joulin et al. (2016).

Bojanowski et al. (2016) explain how FastText extracts feature vectors from
raw text data. FastText makes word embeddings using one of two model
architectures: continuous bag of words (CBOW) or the continuous skip-gram
model.

The skip-gram and CBOW models are first proposed in Mikolov et al. (2013)
which is the paper introducing the word2vec model for word embeddings.
FastText builds upon this work by proposing an extension to the skip-gram
model which takes into account sub-word information.

Both models use a neural network to learn word embedding from using a context
windows consisting of the words surrounding the current target word. The CBOW
architecture predicts the current word based on the context, and the skip-gram
predicts surrounding words given the current word Mikolov et al. (2013).

Figure 2: Diagram of Convolutional Neural network.

###  5.2 Using A Convolutional Neural Network

While every layer in a classic multilayer perceptron is densely connected,
such that each of the nodes in a layer are connected to all nodes in the next
layer, in a convolutional neural network we use one or more convolutional
layers. Convolutional Neural Networks have an established use for text
classification Jacovi et al. (2018).

The basic premise of a convolutional layer is illustrated in Figure
2.111Source: https://realpython.com/python-keras-text-classification/ In a CNN
a filter “slides" over the input. The CNN then takes e.g. the dot product of
the weights of the filter and the corresponding input features, before
applying a further function.

Model | Encoding | Accuracy
---|---|---
MLP | char bi-gram | 0.898
CNN | char bi-gram | 0.956
MLP | char uni-gram | 0.697
CNN | char uni-gram | 0.942
Table 2: Overview of results for the neural network models for the data set
with 10K data points in each language.

##  6 Results

###  6.1 Results with neural networks

Results for the neural network architectures are in Table 2. Here we compare
the result of doing character level uni- and bi-grams using Multilayer
Perceptron and Convolutional neural networks. We see that the CNN performs the
best, achieving an accuracy of 95.6% when using character bi-grams. Both
models perform better using bi-grams than individual characters as features
while the relative increase in performance is greater for the MLP model.

###  6.2 Increasing the size of the data set

Often the performance of supervised classification models increases with more
training data. To measure this effect we increase the amount of training data
to 50K sentences in each of the language categories. Due to longer training
times only the baseline models were included, with the skip-gram encoding from
FastText which we saw achieved the highest accuracy.

Model | Encoding | Accuracy
---|---|---
Knn | skipgram | 0.931
Logistic Regression | skipgram | 0.933
Naive Bayes | skipgram | 0.806
SVM | skipgram | 0.925
Table 3: Overview of results for the data set with 50K data points in each language. Model | Encoding | Accuracy
---|---|---
MLP | char bi-gram | 0.918
CNN | char bi-gram | 0.970
Table 4: Overview of results for the data set with 50K data points in each
language. Figure 3: Confusion matrix with results from the convolutional
neural network on the full data set with 50K data points in each language.

Table 3 shows that the performance of the logistic regression model and the
K-nearest-neighbors algorithm improved slightly by including more data.
Unexpectedly, performance of the support vector machine and Naïve Bayes
dropped slightly with extra data.

Even when including five times the amount of data, the best result, logistic
regression with an accuracy of 93.3%, is still worse than for the
Convolutional Neural Network trained on 10K data points in each language.

In Table 4 we see the results for running the neural networks on the larger
data set. Both models improve by increasing the amount of data and the
Convolutional Neural Network reached an accuracy of 97% which is the best so
far.

In Figure 3 we see the confusion matrix for the convolutional Neural Network
trained on the full Wikipedia data set with 300K data points pr language. We
see that the largest classification errors still happens between Danish,
Bokmål and Nynorsk as well as between Icelandic and Faroese.

Figure 4: Confusion matrix with results from a supervised FastText model on
the full data set with 300K data points.

###  6.3 Using FastText supervised

FastText can also be used for be supervised classification. In Joulin et al.
(2016) the authors show that FastText can obtain performance on par with
methods inspired by deep learning, while being much faster on a selection of
different tasks, e.g. tag prediction and sentiment analysis. We apply FastText
classification to the Nordic DSL task. The confusion matrix from running the
FastText supervised classifier can be seen in Figure 4. We see that FastText
performance is similar to that of the CNN.

###  6.4 Cross-domain evaluation

Training on single-domain data can lead to classifiers that only work well on
a single domain. To see how the two best performing models generalize, we
tested on a non-Wikipedia data set.

For this, we used Tatoeba,222tatoeba.org/ a large database of user-provided
sentences and translations.

The language style used in the Tatoeba data set is different from the language
used in Wikipedia. The Tatoeba data set mainly consists of sentences written
in everyday language. Below we see some examples from the Danish part of the
Tatoeba data set.

> Hvordan har du det? (How are you?)
>
> På trods af al sin rigdom og berømmelse, er han ulykkelig. (Despite all his
> riches and renown, he is unlucky.)
>
> Vi fløj over Atlanterhavet. (We flew over the Atlantic Ocean.)
>
> Jeg kan ikke lide æg. (I don’t like eggs.)
>
> Folk som ikke synes at latin er det smukkeste sprog, har intet forstået.
> (People who don’t think Latin is the most beautiful language have understood
> nothing.)

(a) Distribution of the number of sentences in each language in the Tatoeba
data set.

(b) Distribution of the length of sentences in the Tatoeba data set.

Figure 5: Distribution of the lengths and language classes of Tatoeba
sentences.

In Figure 5(a) we see the number of sentences in each language for all
sentences in the Tatoeba data set. Observe that we have very few samples in
Nynorsk and Faroese.

We see that the performance drops when shifting to Tatoeba conversations. For
reference the accuracy of langid.py on this data set is 80.9% so FastText
actually performs worse than the baseline with an accuracy of 75.5% while the
CNN is better than the baseline with an accuracy of 83.8 %.

Figure 6: Confusion matrix for FastText trained using only character level
n-grams on the Wikipedia data set and evaluated on the Tatoeba data set.

One explanation for the drop in performance is that the sentences in the
Tatoeba data are significantly shorter than the sentences in the Wikipedia
data set as seen in Figure 5(b). As we saw in the previous section, both
models tend to mis-classify shorter sentences more often than longer
sentences. This and the fact that the text genre is different might explain
why the models trained on the Wikipedia data set does not generalise to the
Tatoeba data set without a drop on performance.

The CNN uses character bi-grams as features while, with the standard settings,
FastText uses only individual words to train. The better performance of the
CNN might indicate that character level n-grams are more useful features for
language identification than words alone.

To test this we changed the setting of FastText to train using only character
level n-grams in the range 1-5 instead of individual words. In Figure 6 we see
the confusion matrix for this version of the FastText model. This version
still achieved 97.8% on the Wikipedia test set while improving the accuracy on
the Tatoeba data set from 75.4% to 85.8% which is a substantial increase.

Thus, using character-level features seems to improve the FastText models’
ability to generalize to sentences belonging to a domain different from the
one they have been trained on.

Figure 7: Results for FastText trained w. char n-grams on Wikipedia+Tatoeba
and evaluated on Tatoeba. Figure 8: Distribution of sentence lengths Tatoeba
test set along with the mis-classified sentences.

###  6.5 Retraining on the combined data set

To improve the accuracy over the Tatoeba data set, we retrained the FastText
model on a combined data set consisting of data points from both the Wikipedia
and Tatoeba data set.

The FastText model achieved an accuracy of 97.2% on this combined data set and
an accuracy of 93.2% when evaluating this model on the Tatoeba test set alone
- the confusion matrix can be seen in Figure 7.

As was the case with the Wikipedia data set the mis-classified sentences tend
to be shorter than the average sentence in the data set. In Figure 8 we see
the distribution of sentence lengths for the Tatoeba test set along with the
mis-classified sentences.

In the Tatoeba test set the mean length of sentences is 37.66 characters with
a standard deviation of 17.91 while the mean length is only 29.70 characters
for the mis-classified sentences with a standard deviation of 9.65. This again
supports the conclusion that shorter sentences are harder to classify.

##  7 Analysis

###  7.1 Principal Component analysis and t-SNE

To gain additional insight on how the different word embedding capture
important information about each of the language classes, we visualized the
embeddings using two different techniques for dimensionality reduction.

We used two different methods: Principal Component Analysis (PCA) and
T-distributed Stochastic Neighbor Embedding (t-SNE). We begin with a brief
explanation of the two techniques and proceed with an analysis of the results.

#### Principal Component Analysis

The first step is to calculate the covariance matrix of the dataset. The
components of the covariance matrix is given by

$K_{X_{i},X_{j}}=E[(X_{i}-\mu_{i})(X_{j}-\mu_{j})]$ (1)

where $X_{i}$ is the $i$th component of the feature vector and $\mu_{i}$ is
the mean of that component.

In matrix form we can thus write the covariance matrix as

$K(\mathbf{x},\mathbf{z})=\begin{bmatrix}\text{cov}(x_{1},z_{1})&\dots&\text{cov}(x_{1},z_{n})\\\
\vdots&\ddots&\vdots\\\
\text{cov}(x_{n},z_{1})&\dots&\text{cov}(x_{n},z_{n})\\\ \end{bmatrix}$ (2)

The next step is to calculate the eigenvectors and eigenvalues of the
covariance matrix by solving the eigenvalue equation.

$\det(Kv-\lambda v)=0$ (3)

The eigenvalues are the variances along the direction of the eigenvectors or
“Principal Components". To project our data set onto 2D space we select the
two eigenvectors’ largest associated eigenvalue and project our data set onto
this subspace.

In Figure 9 we see the result of running PCA on the wikipedia data set where
we have used character level bi-grams as features, as well as the CBOW and
skipgram models from FastText.

(a) Character bigram

(b) Fasttext cbow

(c) Fasttext skipgram

Figure 9: Dimensionality reduction using PCA

In the figure for encoding with character level bi-grams, the PCA algorithm
resulted in two elongated clusters. Without giving any prior information about
the language of each sentences, PCA is apparently able to discriminate between
Danish, Swedish, Nynorsk and Bokmål on one side, and Faroese and Icelandic on
the other, since the majority of the sentences in each language belong to
either of these two clusters.

With the FastText implementations we observe three clusters. For both CBOW and
skipgram we see a distinct cluster of Swedish sentences. When comparing the
two FastText models we see that the t-SNE algorithm with skipgrams seems to be
able to separate Faroese and Icelandic data points to a high degree compared
with the CBOW model. Also for the cluster identified with the sentences with
Danish, Bokmål and Nynorsk the skipgram models seem seem to give a better
separation, however to a lesser degree than with the two former languages.

#### t-SNE

The T-distributed Stochastic Neighbor Embedding method was first proposed in
2008 van der Maaten and Hinton (2008), which favours retaining local spatial
relationships over remote ones.

In t-SNE, for a given data point $x_{i}$, the probability of picking another
data point $x_{j}$ as a neighbor to $x_{i}$ is given by:

$p_{ji}=\frac{\exp(||x_{i}-x_{j}||^{2}/2\sigma_{i}^{2})}{\sum_{k\neq
i}\exp(||x_{i}-x_{k}||^{2}/2\sigma_{i}^{2})}$ (4)

Given this probability distribution the goal is to find the low-dimensional
mapping of the data points $x_{i}$ which we denote $y_{i}$ follow a similar
distribution. To solve what is referred to as the “crowding problem", t-SNE
uses the Student t-distribution which is given by:

$q_{ij}=\frac{(1+||y_{i}-y_{j}||^{2})^{-1}}{\sum_{k\neq
l}(1+||y_{k}-y_{l}||^{2})^{-1}}$ (5)

Optimization of this distribution is done using gradient decent on the
Kullback-Leibler divergence which is given by:

$\frac{\delta C}{\delta
y_{i}}=4\sum_{j}(p_{ij}-q_{ij})(y_{i}-y_{j})(1+||y_{i}-y_{j}||^{2})^{-1}$ (6)

The result from running the t-SNE algorithm on the Wikipedia data set can be
seen in Figure 10. As was the case with PCA, it appears that the encoding with
FastText seem to capture the most relevant information to discriminate between
the languages; especially the skip-gram mode does well at capturing
information relevant to this task.

(a) Character bi-gram

(b) FastText CBOW

(c) FastText skip-gram

Figure 10: Dimensionality reduction using t-SNE

Here we recover some interesting information about the similarity of the
languages. The data points in Bokmål lie in between those in Danish and
Nynorsk, while Icelandic and Faroese have their own two clusters which are
separated from the three former languages.

This is in good agreement with what we already know about the languages.
Interestingly the Swedish data points and quite scattered and the t-SNE is not
able to make a coherent Swedish cluster.

This does not however mean that the Swedish data points are not close in the
original space. Some care is needed when interpreting the plot since t-SNE
groups together data points such that neighboring points in the input space
will tend to be neighbors in the low dimensional space.

If points are separated in input space, t-SNE would like to separate them in
the low dimensional space however it does not care how far they are separated.
So clusters that are far away in the low dimensional space are not necessarily
far away in the input space.

###  7.2 Discussion

We used the dimensionality reduction techniques PCA and t-SNE to make
visualizations of feature vectors obtained by making a one-hot encoding with
character bi-grams and with the two modes from FastText.

These unsupervised techniques was able to separate the sentences from
Wikipedia into different clusters. Without any prior knowledge about the
actual language of each sentence these techniques indicated that the six
languages can be divided into three main language categories: (1) Danish
Nynorsk Bokmål (2) Faroese Icelandic and (3) Swedish.

Generally the supervised models had the largest errors when discriminating
between languages belonging to either of the language groups mentioned above.

For the “classical" models we saw that Logistic Regression and support vector
machines achieved better performance than KNN and Naive Bayes, where the
latter performed the worst. This was true in all cases irrespective of the
method of feature extraction.

Additionally we saw that when we used feature vectors from the FastText skip-
gram model the classification models achieved better results than when using
either FastText CBOW or character n-grams.

Generally we saw that increasing the number of data points lead to better
performance. When comparing the CNN with the “classical" models however the
CNN performed better than any of the other models even when trained on less
data points. In this way it seems that the CNN achieves higher sample
efficiency compared to the other models.

##  8 Conclusion

This paper presented research on the difficult task of distinguishing similar
languages applied for the first time to the Scandinavian context. We describe
and release a dataset and detail baseline approaches and problem analysis.

The dataset and code are available at https://github.com/renhaa/NordicDSL.

We compared four different classical models: K nearest Neighbors, Logistic
regression, Naive Bayes and a linear support vector machines with two neural
network architectures: Multilayer perceptron and a convolutional neural
network. The two best performing models, FastText supervised and CNN, saw low
performance when going off-domain. Using character n-grams as features instead
of words increased the performance for the FastText supervised classifier. By
also training FastText on the Tatoeba data set as well as the Wikipedia data
set resulted in an additional increase in performance.

##  9 References

## References

  * Bojanowski et al. (2016) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016.  Enriching word vectors with subword information.
  * Derczynski and Kjeldsen (2019) Leon Derczynski and Alex Speed Kjeldsen. 2019.  Bornholmsk natural language processing: Resources and tools.  In _Proceedings of the 22nd Nordic Conference on Computational Linguistics_ , pages 338–344.
  * Goutte et al. (2016) Cyril Goutte, Serge Léger, Shervin Malmasi, and Marcos Zampieri. 2016.  Discriminating similar languages: Evaluations and explorations.
  * Jacovi et al. (2018) Alon Jacovi, Oren Sar Shalom, and Yoav Goldberg. 2018.  Understanding convolutional neural networks for text classification.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 56–65, Brussels, Belgium. Association for Computational Linguistics.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016.  Bag of tricks for efficient text classification.
  * Kirkedal et al. (2019) Andreas Kirkedal, Barbara Plank, Leon Derczynski, and Natalie Schluter. 2019.  The lacunae of danish natural language processing.  In _Proceedings of the 22nd Nordic Conference on Computational Linguistics_ , pages 356–362.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  Nltk: The natural language toolkit.  In _In Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. Philadelphia: Association for Computational Linguistics_.
  * Lui and Baldwin (2011) Marco Lui and Timothy Baldwin. 2011.  Cross-domain feature selection for language identification.  In _Proceedings of 5th international joint conference on natural language processing_ , pages 553–561.
  * Lui and Baldwin (2012) Marco Lui and Timothy Baldwin. 2012.  Langid.py: An off-the-shelf language identification tool.  In _Proceedings of the ACL 2012 System Demonstrations_ , ACL ’12, pages 25–30, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ , 9:2579–2605.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.
  * Zampieri et al. (2014) Marcos Zampieri, Liling Tan, Nikola Ljubesic, and Jörg Tiedemann. 2014.  A report on the dsl shared task 2014.  In _VarDial@COLING_.
  * Zampieri et al. (2015) Marcos Zampieri, Liling Tan, Nikola Ljubešic, Jörg Tiedemann, and Preslav Nakov. 2015.  Overview of the dsl shared task 2015.  In _Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects_ , page 1.