resolution have been proposed (Jakob and Gurevych, 2010; Ding and Liu, 2010;
Le et al., 2016).

###  3.9 Summary

Anaphora resolution has been explored extensively by theoretical linguists,
psycholinguists as well as computational linguistics. It is the manifest of
structural semantics because the meaning of an anaphor elucidates the
syntactic relationship between the anaphor and its antecedent. Early anaphora
resolution models were inspired by theories and findings in linguistics, such
as the theory of syntactic and semantic constraints from theoretical
linguistics and the findings about factors that influence the choice of
referential form from psycholinguists. Later on, by marrying these theories
with computational models, linguists also gained insights regarding the
comprehension and production of anaphora from anaphora resolution systems. For
instance, we could understand better how each salience factor contributes to
the use of anaphora through the importance analysis of a computational model
that considers the factor. Most recently, though most computational works
focus on building End2End anaphora resolution systems based on deep learning
techniques, linguistic theories about anaphora are still proven to play vital
roles (Chai and Strube, 2022). Dataset is core for either practical or
theoretical anaphora resolution research. Though many annotation schemes and
datasets have been introduced, we found that they share two limitations: one
is that due to the fact that anaphora is a complex concept, annotations of
anaphora resolution datasets are always imperfect (Deemter and Kibble, 2000).
The other is the lack of wide-coverage datasets that covers all kinds of
anaphora. Finally, we found that anaphora resolution is useful in many
downstream tasks, including major tasks of both natural language understanding
and natural language generation. It is always utilized as a producer of
additional features for downstream tasks. Different from other tasks in this
survey, we rarely see how anaphora resolution techniques help boost the
explainability of downstream models, apart from the work of Saunders et al.
(2020). We also have not observed that anaphora resolution techniques are used
for constructing datasets for downstream tasks.

####  3.9.1 Technical Trends

Task | Reference | Feature | Framework | Dataset | Score | Metric
---|---|---|---|---|---|---
Rule-based | Carter (1987) | Salience | Logic rules | self-collected dataset | 93.00% | Acc
Lappin and Leass (1994b) | Salience | Logic rules | self-collected dataset | 85.00% | Acc
Brennan et al. (1987) | Semantic constraints | Centering theory | New York Times | 59.40% | Acc
Tetreault (2001) | Semantic constraints | Centering theory | New York Times | 80.40% | Acc
Baldwin (1997) |  | Syntactic, Semantic,
---
Discourse
Logic rules | self-collected dataset | 77.90% | Acc
Liang and Wu (2003) | WordNet | Logic rules | Brown Corpus | 77.00% | Acc
Haghighi and Klein (2009) | Syntactic, Semantic | Logic rules | ACE | 79.60% | MUC-F
Stat.-based | Soon et al. (2001) |  | Syntactic, Semantic,
---
WordNet
Mention-pair | MUC-6 | 62.60% | MUC-F
Cardie and Wagstaff (1999) |  | Lexical, Syntactic,
---
Semantic
Entity-Mention | MUC-6 | 64.90% | MUC-F
Denis and Baldridge (2008) | Linguistic & Positional | Mention-ranking | ACE | 67.00% | CEAF-F
Rahman and Ng (2011) |  | Lexical, Syntactic,
---
Semantic
Mention-ranking | ACE | 60.80% | CEAF-F
DL-based | Wiseman et al. (2015) | Syntactic, Semantic | Mention-rank., DNN | OntoNotes | 82.86% | Acc
Wiseman et al. (2016) |  | Syntactic, Semantic,
---
Global Feature
Mention-rank., RNN | OntoNotes | 64.21% | CoNLL-F
Clark and Manning (2016a) | Syntactic, Semantic | DRL | OntoNotes | 65.73% | CoNLL-F
Clark and Manning (2016b) |  | Syntactic, Semantic,
---
Global Feature
Mention-ranking, DNN | OntoNotes | 65.52% | CoNLL-F
Lee et al. (2017b) | Word & Cha. Emb. | End2End, LSTM, DNN | OntoNotes | 68.80% | CoNLL-F
Lee et al. (2018) | ELMo | End2End, LSTM, DNN | OntoNotes | 73.00% | CoNLL-F
Zhang et al. (2018) | Glove & Cha. Emb. | BiLSTM, Joint Learning | OntoNotes | 69.20% | CoNLL-F
Joshi et al. (2019) | BERT | Lee et al. (2018) | OntoNotes | 76.90% | CoNLL-F
Joshi et al. (2020) | SpanBERT | Lee et al. (2018) | OntoNotes | 79.60% | CoNLL-F
Wu et al. (2020b) | SpanBERT | QA | OntoNotes | 83.10% | CoNLL-F
Kocijan et al. (2019) | BERT_WikiCREM | DNN | DPR | 84.80% | Acc
Liu et al. (2023b) | BERT | Transformer, MTL | DPR | 84.58% | Acc
AZPR | Okumura and Tamura (1996) | Salience | Center Theory | self-collected dataset | 78.30% | Acc
Sasano et al. (2008) | Salience | Probalistic | self-collected dataset | 39.10% | F1
Chen and Ng (2016) | Syntactic, Lexical | DNN | OntoNotes | 52.20% | F1
Yin et al. (2017a) | Word2Vec, Global | RNN | OntoNotes | 53.60% | F1
Yin et al. (2018b) | Word Embedding | DRL | OntoNotes | 57.20% | F1
Song et al. (2020) | BERT | DNN, MTL | OntoNotes | 58.49% | F1
Table 8: A summary of representative anaphora resolution techniques. Note that
Rahman and Ng (2011) reported that the performance of Denis and Baldridge
(2008) was 57.7% CEAF-F and that CoNLL-F is the average of MUC, B3, and CEAF
scores. Stat. denotes statistics. DL denotes deep learning. AZPR denotes
Anaphoric Zero Pronoun Resolution. Cha. Emb. denotes character embedding. ACE
denotes automatic content extraction. MTL denotes multi-task learning. DRL
denotes deep reinforcement learning

As seen in Table 8, there are two clear technical trends. One is that the
research interest in the realm of anaphora resolution has shifted from machine
learning-based or rule-based anaphora resolution to neural approaches,
especially the End2End neural anaphora resolution, which does mention
identification and linking simultaneously. Another one is that, as previously
elucidated in Section 3.7, there exist distinct shortcomings associated with
each of the task formulations such as mention pair, entity mention, and
mention ranking. Consequently, a recent tendency is to employ higher-order
inferences (Lee et al., 2018) to directly rank clusters or entities, which
allows for the incorporation of benefits from all the formulations. To sum up,
the SOTA anaphora resolution models are often _End2End cluster ranking
models_.

Most recent advances tended to further improve this paradigm from two angles,
namely reducing the search space as an End2End anaphora resolution searches
across all possible spans in its inputs for antecedents (Wu et al., 2020b);
and equipping anaphora resolution systems with knowledge (which, recently,
often large-scale pre-trained language models) to boost their ability of
reasoning (Joshi et al., 2019, 2020). Furthermore, recent investigations on
anaphora resolution have also led to advancements in various deep learning
paradigms. Deep reinforcement learning and multi-task learning were employed
for obviating the need for language-orientated hyperparameter tuning (Clark
and Manning, 2016a), investigating the enduring impact of pronoun-candidate
antecedent pairs Yin et al. (2018b), and enhancing the dependency learning of
mention pairs (Liu et al., 2023b).

Meanwhile, there were also certain efforts that concentrated on resolving
“hard” cases and multi-linguality in anaphora resolution. As for the former
one, people were aware of the models’ capacity to resolve ambiguous pronouns
and biases (especially, gender bias) learned by anaphora resolution models
(Levesque et al., 2012; Rudinger et al., 2018). The SOTA models of this line
of work are often assembled with knowledge bases (Emami et al., 2018) or pre-
trained language models (Kocijan et al., 2019). As for the latter one, multi-
lingual anaphora resolution systems were developed in order to either,
theoretically, unify the theory of reference for different languages
(Nedoluzhko et al., 2022), or, practically, enrich the datasets for low-
resource anaphora resolution languages or tasks (e.g., AZPR; Aloraini and
Poesio (2020)).

In addition to these two trends for developing practical anaphora resolution
systems, there is also a long tradition of studying how human beings
understand and use anaphors with the algorithms introduced in this section
from the age of rule-based methods (Sidner, 1979; Carter, 1987) to the most
recent deep learning based methods (Chai and Strube, 2022; Same et al., 2022).

####  3.9.2 Application Trends

Reference | Downstream Task | Feature | Explain.
---|---|---|---
Le Nagard and Koehn (2010) | Machine Translation | ✓ |
Hardmeier and Federico (2010) | Machine Translation | ✓ |
Miculicich and Popescu-Belis (2017) | Machine Translation | ✓ |
Saunders et al. (2020) | Machine Translation | ✓ | ✓
Steinberger et al. (2007) | Summarization Evaluation | ✓ |
Bergler et al. (2003) | Summarization | ✓ |
Liu et al. (2021b) | Summarization | ✓ |
Agichtein et al. (2008) | Textual Entailment | ✓ |
Jakob and Gurevych (2010) | Sentiment Computing | ✓ |
Ding and Liu (2010) | Sentiment Computing | ✓ |
Table 9: A summary of the representative applications of anaphora resolution
in downstream tasks. ✓denotes the role of anaphora resolution in a downstream
task.

Many demonstrations were carried out approximately 15 years ago to validate
the necessity of anaphora resolution for both language generation and
understanding downstream tasks (Steinberger et al., 2007; Mirkin et al., 2010;
Nicolov et al., 2008; Li et al., 2021; He et al., 2022a). Nevertheless,
practically, at that moment, anaphora resolution often had negative effects
(Bar-Haim et al., 2008; Chambers et al., 2007; Orasan, 2007; Mitkov et al.,
2007). This is mainly because anaphora resolution systems were not powerful
enough and errors they made may propagate to their downstream tasks.

Recently, with significant advancements in the capabilities of anaphora
resolution systems, more and more anaphora resolution systems have been used
for providing anaphora information for downstream tasks (see Table 9). In
short, anaphora resolution helps its downstream applications mainly in two
ways. It links noun phrases in different sentences. As a consequence, these
applications have better performance in comprehending discourse-level
information. On the other hand, linking noun phrases helps downstream
applications to do higher-level reasoning, e.g., extracting global entities
(Sukthanker et al., 2020) and recovering the ellipses (Aralikatte et al.,
2021).

Most downstream task models utilize anaphora resolution as an additional
feature to improve task performance. However, we did not see how anaphora
resolution techniques help to explain how and why anaphora is used in a
certain context.

####  3.9.3 Future Works

Developing robust annotation schemes. Current annotation schemes for anaphora
practically work fine but are theoretically problematic as there is no unified
rule of what is remarkable, and no clear cut between co-reference and anaphora
(though there is a clear boundary between them in linguistic theory).
Annotation schemes so far are imperfect to improve the practicality so that
large anaphora/co-reference resolution datasets (that can be used for training
and assessing data-driven anaphora resolution systems) could be constructed.
In exchange, the resulting corpora are imperfect in terms of both quality
(i.e., some annotated relations might not be anaphoras) and coverage (i.e.,
some kinds of anaphora are not covered). On a different note, anaphora
resolution, which can also be seen as a pragmatics task, disagreement on how
an anaphora is interpreted happens across different readers (Uma et al.,
2022). Nonetheless, many datasets resolve disagreements through majority
voting, while only a few works explicitly annotated ambiguities, which are the
causes of the disagreements (e.g., Poesio and Artstein (2008b)). In aggregate,
it is plausible to design a scheme (probably by extending MATE) that not only
handles disagreements but also balances quality, practicality, and coverage.
Furthermore, it is important to empirically investigate how the errors and
limitations inherent in the annotation scheme can impact the performance of
anaphora resolution systems.

Anaphora resolution evaluation. Analogue to the disagreements in the anaphora
annotation, one can expect that, for a single mismatch between an output and a
reference answer, it might be an error for some readers but not an error for
the rest. For different mismatches, they might have different severity. The
impact of severity of errors has been studied for the production of reference
(see van Miltenburg et al. (2020); e.g., saying “a woman is a man” is more
serious than saying “a red coat is pink”), but it has never been explored in
the realm of anaphora resolution. This said, roughly computing the overlaps
between model outputs and reference outputs might be problematic. On the one
hand, due to discrepancies and varying degrees of errors in anaphora
resolution, human evaluation (Martschat and Strube, 2014) is necessary to
improve the analysis and evaluation of anaphora resolution models, as well as
to establish benchmarks for developing more accurate evaluation metrics. On
the other hand, when designing new evaluation metrics, disagreements, and
error severity should be considered by data-driven methods.

Model development. Regarding future advancements in anaphora resolution
models, a significant area of focus should be on computational studies of
anaphora resolution tasks that are firmly grounded in theory but have yet to
be extensively explored. Examples of such tasks include but are not restricted
to (1) bridging, deictic, and plural references, which are crucial aspects of
referential language, yet their computational treatment has been limited,
possibly due to a shortage of relevant annotated datasets; and (2)
disagreement resolution, which involves learning from discrepancies in human
interpretations of anaphoric expressions to better capture the pragmatic
nuances of such references, and should be incorporated into future models (Uma
et al., 2021); and (3) cross-document anaphora resolution, which is critical
for downstream applications such as knowledge graph construction and cross-
document information extraction, yet has received insufficient attention in
terms of data, methods, and evaluation metrics, particularly in relation to
event resolution.

##  4 Named Entity Recognition

Name Entity Recognition (NER) is a critical component of Information
Extraction, which involves identifying entity mentions in text, defining their
boundaries, and assigning them entity types. The most commonly recognized
entity types by NER systems are Location, Person, and Organization, and tokens
referring to these entities are classified as entity mentions. In the
following example: . Steve Jobs is the founder of Apple.

an NER system would recognize the entities that “Steve Jobs” is Person;
“Apple” is Organization. NER systems use pre-defined entity types, which may
vary across different implementations. For example, Stanford’s widely used NER
software (Finkel et al., 2005) provides three versions that recognize three
classes (Location, Person, Organization), four classes (Location, Person,
Organization, Misc), and seven classes (Location, Person, Organization, Money,
Percent, Date, Time), respectively. NER is a critical component in the field
of NLP (Jinjie et al., 2022; Gao et al., 2022; He et al., 2021) and is often
combined with other tasks, such as Relation Extraction (RE), to serve as a
foundation for various NLP applications. Besides, NER is also used in various
data mining tasks to recognize keywords, topics, and attributes (He et al.,
2019a; Li et al., 2021, 2019).

NER can be traced back to the third Message Understanding Conference (MUC-3)
(Chinchor et al., 1993). The task for MUC-3 was designed to extract relevant
information from the text and convert it into a structured format based on a
predefined template, e.g., incident, the targets, perpetrators, date,
location, and effects. Early NER systems that participated in MUC-3 primarily
relied on rule-based approaches, which involved the manual creation of rules
to identify named entities based on their linguistic and contextual features.
However, with the dominance of deep learning in the NLP community, most NER
tasks are now performed using neural networks. One of the first neural
networks for NER was proposed by Collobert and Weston (2008), which used a
single convolutional neural network with manually constructed feature vectors.
Later, this approach was replaced with high-dimensional continuous vectors,
which were learned from large amounts of unlabeled data in an unsupervised
manner (Collobert et al., 2011). With stronger models, now, the research in
NER has been largely extended to nested NER (Su et al., 2022), few-shot NER
(Huang et al., 2022a), joint entity and relation extraction (JERE) (Zhong and
Chen, 2021; Mao et al., 2022a).

Compared to standard NER whose entity relationship is absent, entities in
nested NER have a hierarchical or nested structure, where one entity is
embedded within another entity. For example, given . The Ontario Supreme Court
said …

“Ontario” is a state entity that is embedded under the government entity of
“Ontario Supreme Court” (Ringland et al., 2019a). Given the very expensive
annotation costs, few-shot NER is also a very important research trend. It
learns NER with a limited amount of labeled data. JERE tasks are established
based on the needs of downstream applications. In many cases, people not only
need to know what an entity is but also need to know the relationship between
entities. Thus, JERE needs to identify named entities in text as well as
extract the relationships that exist between them. In the following example .
Greg Christie has been one of the greatest engineers at Apple.

For standard NER, “Greg Christie” should be identified as Person; “Apple”
should be identified as Company. However, for JERE, besides the above entity
recognition, an additional relationship label, “work_at” should also be
predicted. Compared to identifying entities that are hierarchically structured
within each other in nested NER tasks, the outcomes of JERE deliver another
relationship dimension to connect entities. Both tasks are helpful in
developing a comprehensive knowledge graph.

Due to the wide range of applications of NER, there have been several surveys
conducted on this typical NLP task (Li et al., 2020a; Yadav and Bethard,
2018). One recent study (Song et al., 2021) focused specifically on NER in the
biomedical field, also known as Bio-NER. In this domain, the presence of
meaningless characters in biomedical data presents a significant challenge,
particularly with regards to inconsistent word distribution. Similarly, Liu et
al. (2022a) summarized and discussed the challenges specific to Chinese NER,
rather than the more general English NER tasks. Meanwhile, Nasar et al. (2021)
explored both NER and RE tasks, as they are closely linked and are typically
composed of pipeline tasks. The aforementioned surveys focus on the technical
perspective of NER, based on deep learning technology, while this section
broadens the horizon of NER from theoretical foundations to applications.

###  4.1 Theoretical Research

####  4.1.1 Prototype Theory

Rosch (1973) argued that our classification system, which includes the
classification of named entities, is based on a central or prototype example.
A prototype is a typical example of a category that represents the most common
features or characteristics associated with the category. For example, the
prototype of “bird” must associate the features, such as wings, feathers, and
the ability to fly. Birds such as ostriches or penguins, which do not
perfectly possess these characteristics, may be viewed as less typical
examples. Rosch and Mervis (1975) discovered that individuals can identify
typical category examples faster and with greater precision than atypical
examples. Thus, learning from prototypes can help to quickly grasp the
important features of a named entity with a few examples.

####  4.1.2 Graded Membership

Rosch et al. (1976) argued that the classification of categories is frequently
determined not by strict boundaries, but by various degrees of membership. We
can use this theory for NER because the NER task also categorizes entities by
predefined classes. The idea of Graded Membership implies how humans perceive
and categorize the world around us. Some categories, e.g., “vegetable”, may be
viewed as less distinct and vaguer. The theory suggests that the borders
between categories may not be well-defined in some cases, leading to
ambiguities when attempting to classify certain items, such as tomatoes or
mushrooms. The ambiguity can be further compounded by cultural or regional
differences in how categories are defined or classified.

####  4.1.3 Conceptual Blending

According to Fauconnier and Turner (2008), the act of blending different
elements and their corresponding relationships is an unconscious process that
is believed to be ubiquitous in everyday thought and language. This process
involves the combination of various mental spaces or cognitive domains that
are drawn from different scenarios and experiences. These scenarios may be
derived from personal experiences, cultural practices, or societal norms,
among others. Concept blending allows us to create a new concept by combining
existing ones in novel ways. For example “SpaceX” may be mapped to mental
spaces related to “aerospace” and “technology”; “Tesla” may be mapped to
mental spaces related to “car” and “clean energy”. Conceptual blending
provides an explanation for the recognition and comprehension of newly named
entities by mapping them onto existing mental spaces or concepts.

####  4.1.4 Grammatical Category

From the aspect of computational linguistics, the core issue of NER is how to
define a named entity. Marrero et al. (2013) group the criteria of a named
entity as grammatical category, rigid designation, unique identification, and
the domain of applications. However, many of the entity definitions in the NER
domain are imperfect. From the view of grammatical category, a named entity is
traditionally defined as a proper noun or a common name for a proper noun.
Previous work has described NER as the recognition of proper nouns in general.
However, as pointed out by Borrega et al. (2007), the classic grammatical
approach to proper noun analysis is insufficient to deal with the challenges
posed by NER applications. For instance, in a toy question-answering task such
as . Do crocodiles live in the sea or on land?

“crocodiles”, “sea”, and “land” are not proper nouns, while they are commonly
recognized as the essential entities for a proper understanding of the
question. Consequently, a proper noun is no longer considered a criterion for
identifying named entities in current NER research.

####  4.1.5 Rigid Designation

The rigid designation is a concept in the philosophy of language which
suggests that certain names or labels are inherently linked to the things they
represent, e.g., “Barack Obama” rigidly designates the person who is the 44th
President of the US, and it cannot be used to refer to any other person or
entity. NER can be viewed as a form of rigid designation as it assigns labels
to entities based on their intrinsic identity (Kripke, 1972), rather than on
their usage in the text. However, LaPorte (2006) noted that not all
expressions that appear to designate rigidly can be analyzed as directly
referring to an object in every possible world. This highlights the difficulty
of defining entities with complex concepts in real-world applications. As a
result, annotators likely make subjective judgments when labeling complex
entities, which may be affected by entity descriptions and annotators’
understanding.

####  4.1.6 Unique Identification

From the view of unique identification, the MUC conferences require that NER
tasks annotate the “unique identification” of entities for all expressions
(Grishman and Sundheim, 1996). However, determining what is unique depends on
contextual elements, and can be a subjective process. While this “unique
identification” is typically considered to be the reference being referred to,
the definition itself poses a challenge in terms of defining what is truly
unique.

####  4.1.7 Domain of Applications

The definition of named entities was frequently grounded in the domain of
applications. Entity definitions can be different between different NER tasks.
For instance, in drug-drug interaction tasks (Deng et al., 2020), diseases may
not be considered entities, whereas they are entities in adverse drug events
(Demner-Fushman et al., 2019). Inconsistent entity definitions create
challenges for machine learning. Because inconsistent entity definitions mean
that for the same semantic unit, the machine has to summarize different entity
representations to distinguish their labels under different tasks. This is
also not conducive to training an all-around NER classifier on different
application domains.

###  4.2 Annotation Schemes

Tokens: | West | African | Crocodile | are | semiaquatic | reptiles | that | live | in | Africa
---|---|---|---|---|---|---|---|---|---|---
IO | I | I | I | O | I | I | O | O | O | I
BIO | B | I | I | O | B | I | O | O | O | B
BIOES | B | I | E | O | B | E | O | O | O | S
Table 10: The three common annotation schemes for NER.

NER is typically approached as a sequence labeling task, where each token in a
sentence is assigned a label. Three common annotation schemes are shown in
Table 10. The IO scheme is a classification task that distinguishes between
two classes, namely “Inner” and “Other”, to determine whether a token belongs
to an entity or not. On the other hand, the BIO scheme employs three labels,
namely “Beginning”, “Inner”, and “Other”, to identify tokens that represent
the start of an entity, tokens that belong to an entity, and tokens that do
not belong to any entity. The BIOES scheme expands on the BIO scheme by
incorporating two additional labels, namely “Single” and “End”, to more
precisely define the boundaries of entities.

By employing the IO scheme, the binary classification of tokens is simplified,
as each token is labeled as either belonging to an entity or not. This
straightforward labeling system makes it easier to identify entities in a
text, but it fails to specify the position of the entities within the text. In
contrast, the BIO scheme provides more precise annotations by identifying the
beginning and continuation of an entity in the text. This labeling system
allows for more accurate recognition of entities in a text and better
classification of individual tokens. The BIOES scheme further extends the BIO
scheme by providing more precise boundaries for entities, thereby allowing for
better recognition of entity boundaries in a text. The “Single” label is used
to denote an entity that consists of a single token, whereas the “End” label
is used to indicate the final token of an entity. By incorporating these
additional labels, the BIOES scheme provides a more nuanced approach to entity
recognition and annotation.

###  4.3 Datasets

Dataset | Source | # Sample | Reference
---|---|---|---
MUC-6 | Newswire | 318 articles | Grishman and Sundheim (1996)
ACE-05 | Social media | 12,548 sentences | Walker et al. (2006)
TACRED | Newswire | 106,264 instances | Zhang et al. (2017b)
CoNLL-2003 |  Reuters212121www.reuters.com/researchandstandards/ | 1,499 articles | Sang and De Meulder (2003)
I2B2 |  ECI Corpus222222http://www.ldc.upenn.edu/ | 1,600 patient records | Stubbs and Uzuner (2015)
ADE |  MEDLINE 232323 http://www.nlm.nih.gov/bsd/indexing/training/PUB_050.htm | 2,972 document | Gurulingappa et al. (2012)
DDI |  DrugBank242424https://go.drugbank.com/ | 1,025 document | Herrero-Zazo et al. (2013)
WNUT-17 | Social media | 2,295 documents | Derczynski et al. (2017)
OntoNote 5.0 | Social media | - | Weischedel et al. (2013)
CPR | MEDLINE | - | Krallinger et al. (2017)
MultiNERD | Wikipedia | 10 languages | Tedeschi and Navigli (2022)
HIPE-2020 | Newspapers | 17,553 mentions | Ehrmann et al. (2022)
NNE | Newswire | 49,208 sentences | Ringland et al. (2019a)
GENIA | MEDLINE | 18,546 sentences | Kim et al. (2003)
Table 11: NER datasets and statistics.

The surveyed popular NER datasets and their statistics can be viewed in Table
11. The first NER-focused dataset was published in the 6th MUC Conference
(Grishman and Sundheim, 1996). This task consists of three sub-tasks,
including entity names, temporal expressions, and number expressions. The
defined entities include organizations, persons, and locations; The defined
time expressions include dates and times; The defined quantities include
monetary values and percentages. More details can be seen in the office
website252525https://cs.nyu.edu/g̃rishman/NEtask20.book_2.html. The example of
this dataset is shown as follows.

text: "Taga Co.",
type: "ORGANIZATION".

The MUC conference was replaced by Automatic Content Extraction (ACE) after
1997. ACE05 (Walker et al., 2006) is another popular NER dataset published at
ACE Conference. ACE05 is a multi-lingual dataset, which contains English,
Arabic, and Chinese data. The corpus consists of data of various types
annotated for entities, relations, and events. Its data source includes
broadcast conversation, broadcast news, newsgroups, telephone conversations,
and weblogs. More details can be seen on the office
website262626https://catalog.ldc.upenn.edu/LDC2006T06. The example of this
dataset is shown as follows.

entity id: "NN_ENG_20030630_085848.18-E1",
type: "GPE",
subtype: "State-or-Province",
class: "SPC",
start: "82",
end: "91",
name: "california".

After MUC, the Text Analysis Conference (TAC) published the Knowledge Base
Population challenge. In this challenge, the Stanford NLP Group developed TAC
Relation Extraction Dataset (TACRED) (Zhang et al., 2017b), which contains
106,264 instances with annotated entities, relations and some other NLP tasks.
More details can be seen on the office
website272727https://nlp.stanford.edu/projects/tacred/#intro. The example of
this dataset is shown as follows.

id: "e7798fb926b9403cfcd2",
docid: "APW_ENG_20101103.0539",
relation: "per:title",
token: "[‘At’, ‘the’, ‘same’, ‘time’, ‘,’, ‘Chief’, ...]",
subj_start: "8",
subj_end: "9",
obj_start: "12",
obj_end: "12",
subj_type: "PERSON",
obj_type: "TITLE",
stanford_pos: "[‘IN’, ‘DT’, ‘JJ’, ‘NN’, ‘,’, ‘NNP’, ‘NNP’, ...]",
stanford_ner: "[‘O’, ‘O’, ‘O’, ‘O’, ‘O’, ‘O’, ‘O’, ‘O’, ...]"
stanford_head: "[4, 4, 4, 12, 12, 10, 10, 10, 10, 12, ...]",
stanford_deprel: "[‘case’, ‘det’, ‘amod’, ‘nmod’, ‘punct’, ....]".

CoNLL-2003 (Sang and De Meulder, 2003) is another widely used NER dataset.
This task concerned language-independent named entity recognition, which
concentrates on four kinds of named entities: locations, persons,
organizations, and names of miscellaneous entities that do not belong to the
previous three kinds. The related data files are available in English and
German. More details can be seen on the office
website282828https://www.clips.uantwerpen.be/conll2003/ner/. The example of
this dataset is shown as follows.

text: "[‘U.N.’, ‘official’, ‘Ekeus’, ‘heads’, ...], ",
pos: "[‘NNP’, ‘NN’, ‘NNP’, ‘VBZ’, ...], ",
syntactic chunk: "[‘I-NP’, ‘I-NP’, ‘I-NP’, ‘I-VP’, ...], ",
named entity tag: "[‘I-ORG’, ‘O’, ‘I-PER’, ‘O’, ...]".

Besides the above famous datasets, MultiNERD (Tedeschi and Navigli, 2022),
HIPE-2020 (Ehrmann et al., 2022), and NNE (Ringland et al., 2019b) are also
popular NER datasets in general domain. NER tasks have garnered considerable
attention in numerous specialized domains. Informatics for Integrating Biology
and the Bedside (I2B2) (Stubbs and Uzuner, 2015) is a national biomedical
computing project sponsored by the National Institutes of Health (NIH) from
2004 to 2014. I2B2 actively advocates mining medical value from clinical data
and has organized a series of evaluation tasks and workshops for unstructured
medical record data, and these evaluation tasks and open datasets have gained
wide influence in the medical NLP community. I2B2 is maintained in the
Department of Biomedical Information at Harvard Medical School and continues
to conduct assessment tasks and workshops, and the project has been renamed
National NLP Clinical Challenges (N2C2). More details can be seen on the
office website292929https://www.i2b2.org/. Besides, there also exist many
other biomedical datasets for specific medical NER tasks, including Adverse
Drug Events (ADE) Gurulingappa et al. (2012); Alvaro et al. (2017), Drug-Drug
Interaction Herrero-Zazo et al. (2013), and Chemical Protein Reaction (CPR)
Krallinger et al. (2017), and GENIA (Shibuya and Hovy, 2020).

###  4.4 Knowledge Bases

Name | Knowledge | # Entities | structure
---|---|---|---
Wikipedia | World | 13,489,694 | unstructured
Wikidata | World | 100,905,254 | graph
DrugBank | Medical | over 500,000 | structured
UMLS | Medical | 16,857,345 | structured
BioModels | Medical | unclear | structured
SNOMED CT | Medical | over 350,000 | structured
ICD-10 | Medical | unclear | structured
MIMIC-III | Medical | unclear | structured
MeSH | Medical | over 28,000 | structured
GeoNames | Geographical | over 25,000,000 | structured
EDGAR | Financial | unclear | structured
EduKG | Educational | 5,452 | structured
Table 12: Useful knowledge bases for NER.

Table 12 illustrates useful knowledge bases for NER. The biggest ones are
Wikidata303030https://www.wikidata.org/ and
Wikipedia313131https://en.wikipedia.org/, which are multi-lingual free online
encyclopedias maintained by worldwide volunteers.

There are also knowledge bases in a specific field. SNOMED CT (Systematized
Nomenclature of Medicine - Clinical Terms) (Donnelly et al., 2006) is a
systematically organized collection of medical terms that provides a
standardized representation of clinical information, which is often used in
NER tasks involving clinical data. MeSH (Medical Subject Headings) (Lipscomb,
2000) is another controlled vocabulary, developed by the U.S. National Library
of Medicine. It is used for indexing and organizing biomedical literature.
Other medical knowledge bases include UMLS (Unified Medical Language System)
(Wheeler et al., 2007; Bodenreider, 2004), ICD-10 (Hirsch et al., 2016),
MIMIC-III (Johnson et al., 2016), DrugBank (Wishart et al., 2018), and
bioinformatics knowledge base BioModels (Li et al., 2010). GeoNames (Ahlers,
2013) is a comprehensive geographic knowledge repository that encompasses over
25 million geographical names and comprises over 11 million distinctive
features, including cities, countries, and landmarks. EDGAR (Electronic Data
Gathering, Analysis, and Retrieval) (Branahl, 1998) is a database maintained
by the U.S. Securities and Exchange Commission (SEC), containing financial
filings and reports from publicly traded companies. EduKG (Hu et al., 2016) is
an educational knowledge base.

###  4.5 Evaluation Metrics

In the process of named entity recognition task evaluation, the main
evaluation metrics are also Precision, Recall, and F-value.

###  4.6 Annotation Tools

One AI323232https://docs.oneai.com/docs is an online platform that offers NLP-
as-a-service. The utilization of APIs enables developers to effectively
analyze, manipulate, and transform natural language inputs within their
programming code without requiring any specialized knowledge of NLP. One AI
facilitates the interpretation of both the meaning and information conveyed in
textual data, and can produce structured data in context via language
processing.

GATE Teamware333333https://gate.ac.uk/teamware/ (Bontcheva et al., 2013) is an
integrated annotation tool for comprehensive language processing tasks,
especially for Information Extraction systems. The University of Sheffield
developed GATE Teamware that enables collaborative semantic annotation
projects through a shared annotation environment. The software comprises
several beneficial attributes such as the ability to load document
collections, create project templates that can be used multiple times,
initiate projects based on templates, assign project roles to individual
users, monitor progress and obtain various project statistics in real-time,
report project status, annotator activity, and statistics, and apply automatic
annotations or post-annotation processing via GATE-based processing routines.

MAE343434https://keighrim.github.io/mae-annotation/ (Rim, 2016) (Multi-
document Annotation Environment) is a general-purpose and lightweight natural
language annotation tool. The tool enables users to specify and create their
customized annotation tasks, annotate any text spans of their choice, utilize
non-consuming tags, effortlessly establish links between annotations, and
produce annotations in stand-off XML format. It also provides a simple
adjudication process with a visualization feature that displays the extent
tags, link tags, and non-consuming tags of any XML standoff annotated
documents.

UIMA353535https://uima.apache.org/sandbox.html (Ferrucci and Lally, 2004)
(Unstructured Information Management Applications) is a framework that falls
under the purview of the Apache Software Foundation. It serves as a
comprehensive platform for managing language processing projects and is
licensed under Apache’s open-source license. With its versatile capabilities,
UIMA can effectively handle a diverse array of language processing tasks and
extract various types of information. The UIMA’s Regular Expression Annotator
is capable of identifying entities such as email addresses, phone numbers,
URLs, zip codes, or any other entities based on the utilization of regular
expressions and concepts. The tool can generate an annotation for each
detected entity or update an existing annotation with relevant feature values.

Brat363636https://brat.nlplab.org/ (Browser-based Rapid Annotation Tool) is a
free data labeling tool that offers a seamless browser-based interface for
annotating text. It streamlines numerous annotation tasks related to natural
language processing. With a thriving support community, Brat is a well-known
and widely used tool in NER. It also offers the option of integrating with
external resources, such as Wikipedia. Moreover, Brat enables organizations to
establish servers that allow multiple users to collaborate on annotation
tasks. However, implementing this feature does necessitate some technical
proficiency and server management skills.

###  4.7 Methods

####  4.7.1 Nested NER

A. Multi-label Method

Due to the fact that nested named entities can have multiple labels for a
single token, traditional sequence labeling methods are not directly
applicable to the recognition of nested named entities. To address this issue,
researchers have attempted to convert the multi-label problem into a single-
label problem or adjust the decoder to assign multiple labels to the same
entity.

Katiyar and Cardie (2018) proposed a method to address nested named entity
recognition by modifying the label representation in the training set. Instead
of using one-hot encoding, they used a uniform distribution over the specified
classes as the label. During inference, a hard threshold is set and any class
with probability above this threshold is predicted for the token. However,
this approach has two limitations: it is difficult to determine the objective
for model learning; the method is sensitive to the manually chosen threshold
value.

Straková et al. (2019) changed nested NER from multi-label to single-label
tasks by modifying the annotation schema. They combined any two categories
that may co-occur to produce a new label (e.g., combine B-Location with
B-Organization to construct a new label $B\\_Loc\\_Org$). One benefit of this
approach is that the final classification task is still a single category
because all possible classification targets had been covered in the schema.
Nonetheless, this method brought about a proliferation of label categories in
an exponential manner, leading to sparsely annotated labels that proved
difficult to learn, particularly in the context of entities nested across
multiple layers.

In order to address the issue of label sparsity, Shibuya and Hovy (2020)
proposed a hierarchical approach. If the classification of nested entities
cannot be resolved in a single pass, the classification is continued
iteratively until either the maximum number of iterations is reached or no new
entities can be generated. Nevertheless, this approach is susceptible to error
propagation, whereby an erroneous classification in a preceding iteration
could impact subsequent iterations.

B. Generation-based Method

Li et al. (2020c) proposed a unified framework to accomplish flat and nested
NER tasks by formulating NER as a machine reading comprehension (MRC) task
(Liu et al., 2023a). In this approach, the extraction of each entity type
corresponds to specific questions. For instance, when the model is given the
question “which location is mentioned in the sentence?” along with the
original sentences, it generates an answer such as “Washington”. This approach
is similar to Prompt Tuning (Liu et al., 2021a), which avoids the labor-
intensive process of constructing manual questions. However, in this method,
the generated tokens must be mapped to pre-defined named entity types.

Yan et al. (2021a) proposed a novel pointer generation network. Given an input
sentence, the model generates the entity indexes in this sentence that belong
to entities. In such a way, flat, nested, and discontinuous entities can be
recognized in a unified framework. Skylaki et al. (2020); Fei et al. (2021);
Yang and Tu (2022); Su et al. (2022) are also following the idea of generating
indexes of a sentence to recognize nested entities.

C. Hypergraph-based Method

A hypergraph is a generalized variant of a normal graph, which is
characterized by an edge that can connect an arbitrary number of vertices
(Feng et al., 2019). It is widely used in the NLP community for the tasks of
syntactic parsing, semantic parsing, and machine translation because it can
accurately describe the relationship between objects with multiple
associations. A set of objects with only binary relations can be described by
a normal graph. However, when the objects are often related to each other in a
more complex one-to-many or many-to-many, e.g., nested named entities,
hypergraphs become a more appropriate data structure. A typical example of
nested NER with a hypergraph solution is shown in Figure 4.

(a) Instance of a nested label result (b) Corresponding hypergraph structure

Figure 4: A typical example for nested NER with hypergraph solution

Finkel and Manning (2009) firstly introduced hypergraphs into nested NER
tasks, named Mention Hypergraph. In their model, Mention Hypergraph utilized
nodes and directed hyper-edges to jointly represent named entities and their
combinations. To compute the training loss, the proportion of accurate
structures was calculated and divided by a normalized term. This term was
obtained using a dynamic programming algorithm that aggregated feasible nested
sub-graphs for NER. However, the normalized terms obtained from this algorithm
included fractions of pseudo-structures, which led to errors.

To deal with the problem of pseudo-structures, Muis and Lu (2017) proposed a
gap-based marker model to identify nested entity structures by combining
mention separators with features. In this method, the authors manually
designed 8 types of mention separators for various scenarios. Based on the
mention separators’ states for any two consecutive tokens, they defined
accurate and novel graph structures. However, since this approach only
utilized local information to construct the graph structures, it may not be
unambiguous for long-nested named entities. For instance, when presented with
the nested entity “a West African Crocodile”, which includes two separate
entities, “West African” and “a West African Crocodile”, their approach may
also recognize “a West African” as a named entity.

This ambiguous problem was solved by Wang and Lu (2018), which proposes a
segmental hypergraphs method. The method used an unambiguous ambiguity-free
compact hypergraph representation to encode all possible combinations of
nested named entities. Upon Mention Hypergraph Finkel and Manning (2009),
segmental hypergraphs employed an inside-outside message-passing algorithm
that can summarize the features of child nodes to the parent node and achieve
efficient interference.

Besides the above work, Wan et al. (2021) introduced the concept of regional
hypernodes and a combination method of graph convolutional network (GCN) and
BiLSTM to generate hypernodes for each region. Yan and Song (2022) employed
start token candidates and generated corresponding queries with related
contexts, then used a query-based sequence labeling module to form a local
hypergraph for each candidate.

####  4.7.2 Few-shot NER

A. Metric Learning

Metric Learning is a common technology in various few-shot tasks. Prototypical
Networks (Snell et al., 2017) is a milestone in few-shot metric learning.
Prototypical Networks compute the centroid of each category based on the
support set. They determine the distance between the samples in the query set
and the prototype center, followed by updating the model by optimizing this
distance. Upon completion of the training phase, the embedding of each sample
will be situated in closer proximity to the centroid of the corresponding
category. Such an idea was largely inspired by Prototype Theory (see Section
4.1.1).

Fritzler et al. (2019) adopted the prototypical network into few-shot NER
tasks. They argued that words in a sentence are interdependent and, therefore,
the labeling of adjacent words should be taken into account. To address this
issue, they substituted the conventional token input of Prototypical Networks
with complete sentences. However, this method ignores the problem of the
Outside (O) class in NER tasks, which actually represent different semantic
meanings. This problem would significantly affect the model’s performance
under few-shot settings.

To avoid the above issues, Yang and Katiyar (2020) followed the nearest
neighbor inference (Wiseman and Stratos, 2019) to assign labels to tokens. In
contrast to Prototypical Networks, which learn a prototype for each entity
class, this study characterized each token by its labeled instances in the
support set alongside its context. The approach determined the nearest labeled
token in the support set, followed by assigning labels to the tokens in the
query set that require prediction.

Das et al. (2022) proposed CONTaiNER, which optimized the inter-token
distribution distance. CONTaiNER employed generalized objectives to different
token categories based on their Gaussian-distributed feature vectors. Such a
method has the potential to mitigate overfitting problems that arise from the
training domains.

B. Prompt Tuning

Recently, prompt tuning has shown great potential on few-shot tasks by
reformulating other tasks as mask language tasks (He et al., 2023; Mao et al.,
2022c; Schick and Schütze, 2021). Prompt tuning-based methods need construct
prompts to obtain masked word predictions and then map predicted works into
pre-defined labels, as shown in Figure 5.

Figure 5: A typical prompt tuning example for NER tasks.

Cui et al. (2021) proposed a template-based method for NER, which first
applied the prompt tuning to NER tasks. However, their method had to enumerate
all possible spans of sentences combined with all entity types to predict
labels, which suffered serious redundancy when entity types or sentence
lengths increased.

Manually defined prompts were labor-intensive and made the algorithm sensitive
to these prompts. To avoid the manual prompt constructions, Ma et al. (2022a)
tried to explore a prompt-free method for few-shot NER. The present study
introduced an entity-oriented language model that decodes input tokens into
their corresponding label words if they belong to entities. In cases where the
tokens are not entities, the entity-oriented language model decodes the
original tokens. Nevertheless, this approach encounters difficulties in
labeling word engineering. While this study proposed an automated label
selection technique, the associated experiments revealed some degree of
instability.

COPNER (Huang et al., 2022b) introduced class-specific words to construct
prompt tuning. By comparing each token with manually selected class-specific
words, this method needed neither manual prompts nor label words engineering.
The selected class-specific words (a representative word corresponding to a
class) were directly concatenated with original sentences as prompts. However,
the manual selection of class-specific words is subjective, and a single word
may not entirely capture the semantics of an entity category.

####  4.7.3 Joint NER and Relation Extraction

A. Parameter Sharing-based Multi-tasks Learning

Considering that NER is usually combined with relation extraction tasks
applied in various downstream tasks, jointly recognizing named entities and
classifying relations is a hot topic in related fields. Multi-task learning is
the most common solution in joint NER and relation extraction. Miwa and Bansal
(2016) firstly employed a shared Bi-LSTM encoder to obtain token
representations, and then fed encoded representations into NER and relation
extraction classifiers, respectively. Sun et al. (2020) utilized a GCN as a
shared encoder to enable joint inference of both entity and relation types.
The core idea of the above study is that multi-task models can enhance the
interactions between the learning of NER and relation extraction, and further
alleviate the error propagation by sharing common parameters (He et al.,
2021). However, this work cannot ensure that the sharing of information is
useful and proper. NER and relation extraction might need different features
to result in precise predictions.

To deal with such a problem, Yan et al. (2021b) proposed an information
filtering mechanism to provide valid features for NER and relation extraction.
Their method used an entity and relation gate to divide cell neurons into
different parts and established a two-way interaction between NER and relation
extraction. In the final employed network, each neuron contained a shared
partition and two task-specific partitions.

B. Table Filling

While multi-task learning can improve the interdependence between NER and
relation extraction, the relation extraction process still requires the
pairing of all entities from the NER tasks to classify relations, making it
impossible to completely eliminate error propagation. To solve the problem,
Miwa and Sasaki (2014) proposed a table-filling strategy to achieve joint NER
and relation extraction by labeling input tokens in a table. The method
utilized token lists of sentences to form rows and columns. Then, they
extracted entities using the diagonal elements and classified relations with a
lower/upper triangular matrix of the table. This basic table-filling strategy
can be seen in Figure 6. Nonetheless, this approach involved the explicit
integration of entity-relation label interdependence, which necessitated the
use of intricate features and search heuristics.

Figure 6: The illustration of the table-filling strategy.

Gupta et al. (2016) incorporated neural networks with a table-filling strategy
via a unified multi-task recurrent neural network. This method detected both
entity pairs and the related relations with an entity-relation table, which
alleviated the need for search heuristics and explicit entity-relation label
dependencies. Zhang et al. (2017a) further integrated global optimization and
syntax information into the table-filling strategy to combine NER and relation
extraction tasks. Ren et al. (2021) argued that the above table-filling-based
studies only focus on utilizing local features without the global associations
between relations and pairs. Ren et al. (2021) first produced a table feature
for every relation, followed by extracting two types of global associations
from the generated table features. Finally, the table feature for each
relation was integrated with the global associations. Such a process is
performed iteratively to enhance the final features for joint learning of NER
and relation extraction tasks.

C. Tagging Scheme

The table-filling approach can mitigate issues related to error propagation.
However, these techniques require the pairing of all sentence elements to
assign labels, resulting in significant redundancy. To address the redundancy
and avoid error propagation, Zheng et al. (2017) proposed a novel tagging
scheme that converted joint NER and relation extraction into a united task.
The idea was similar to the solution for nested entities (Straková et al.,
2019), which combined NER labels with relation extraction labels by modifying
the annotation schema. For example, given the sentence “The United States
president Biden will visit …”, by allocating the customized labels “Country-
President_B_1”, “Country-President_E_1” for tokens “United”, “States”, and
“Country-President_E_2” for token “Biden”, the proposed method can directly
obtain the triplet (United State, Country-President, Biden).

Yu et al. (2020); Wei et al. (2019) proposed two similar methods. In contrast
to conventional joint approaches for NER and relation extraction, which
involve recognizing entities followed by relation classification, the two
methods first identified all head entities. Next, for each identified head
entity, they simultaneously predicted corresponding tail-entities and
relations, achieving cascade frameworks combined with a customized tagging
scheme. The typical joint NER and relation extraction tasks learn to model the
conditional probability:

$P(h,r,t)=P(s)P(t\mid h)P(r\mid h,t),$ (7)

where $h$ represent head entity; $r$ represent relation; $h$ represent tail
entity. The above methods combined the last two parts in Eq. 7, yielding

$P(h,r,t)=P(s)P(t,r\mid s).$ (8)

###  4.8 Downstream Applications

####  4.8.1 Knowledge Graph Construction

Knowledge graphs are structured semantic knowledge bases for rapidly
describing concepts and their interrelationships in the physical world,
aggregating large amounts of knowledge by reducing the data granularity from
the document level to the instance level (Yao et al., 2022). Thus, knowledge
graphs enable rapid response and reasoning about knowledge. At present, the
application of knowledge graphs has become prevalent in industrial domains,
such as Google search. Generally, the construction of Knowledge Graphs
consists of three main parts: information extraction, information fusion, and
information processing. The task of information extraction involves the
identification of nodes through NER and the establishment of edges via
relation extraction. The task of information fusion is utilized for
normalizing nodes and edges. The normalized nodes and edges need to go through
a quality assessment with the task of information processing to be added to
knowledge graphs.

He et al. (2021) proposed a multi-task learning-based method for the
construction of genealogical knowledge graphs. At first, He et al. (2019b)
collected unstructured online obituary data. Then, they extracted named
entities as nodes and classified family relationships for these recognized
people as edges to construct genealogical knowledge graphs. Similarly, Jiang
et al. (2020) utilized NER and relation extraction for obtaining the nodes and
edge in biomedical knowledge graphs. They proposed a customized tagging schema
to convert the construction of biomedical knowledge graphs into a sequence
labeling task with multiple inputs and multiple outputs. Li et al. (2020b)
proposed a systematic approach for constructing a medical knowledge graph,
which involves extracting entities such as diseases and symptoms, as well as
related relationships, from electronic medical records. Silvestri et al.
(2022), Peng et al. (2019), and Shafqat et al. (2022) aimed to collect and
utilize medical knowledge for NER. Further, constructing knowledge graphs
requires the task of Entity Linking (Tedeschi et al., 2021a) to normalize
entities with different names. Entity Linking and NER are typically performed
as pipeline tasks to yield more nodes and edges for the constructed graphs.
Additionally, Entity Linking can be seen as a downstream task for NER, as it
further refines the identified entities by linking them to a specific
reference entity in a knowledge graph.

####  4.8.2 Recommendation Systems

Recommendation systems can be classified into two primary categories based on
their solutions, namely content-based recommenders and collaborative
filtering-based recommenders (Batmaz et al., 2019). For both of these groups,
gathering data on users and products is a crucial step in the entire process.
In this regard, NER modules play a pivotal role. For example, Kim et al.
(2012) introduced the 5W1H model, which utilizes NER to extract contextual
information, specifically Who, Why, Where, What, When, and How, to generate
contextual recommendations.

Zhou et al. (2020) argued that recommendation systems currently in use suffer
from a deficiency of contextual information in conversational data, as well as
a semantic gap between natural language expressions and the preferences of
individual users for specific items. To overcome these challenges, word- and
entity-oriented knowledge graphs were incorporated to enhance the data
representations. Mutual Information Maximization was adopted to align the
word-level and entity-level semantic spaces. The aligned semantic
representations were used to develop a knowledge graph-enhanced recommender
component to make accurate recommendations, and a knowledge graph-enhanced
dialog component that can generate informative keywords or entities in the
response text. A NER module is a crucial component in creating such a
knowledge graph-enhanced system (Wu et al., 2023).

Iovine et al. (2020) proposed a domain-independent, configurable
recommendation system framework, named ConveRSE (Conversational Recommender
System framEwork). ConveRSE utilized various interaction mechanisms, including
natural language, buttons, and a combination of the two. The framework
comprised a dialog manager, an intent recognizer, a sentiment analyzer, an
entity recognizer, and a set of recommendation services. The entity recognizer
component specifically focused on identifying relevant entities that were
mentioned in the user’s input, and linking them to an appropriate concept in
the knowledge base. The ConveRSE framework’s success is heavily reliant on the
performance of the NER component, as it plays a crucial role in enhancing the
system’s overall performance.

Wang et al. (2019a) proposed RippleNet, an end-to-end framework that
incorporates the knowledge graph into a recommender system. RippleNet overcame
the limitations of previous embedding-based and path-based approaches to
knowledge graph-aware recommendation by incorporating the knowledge graph as a
form of supplementary information. RippleNet included both inward aggregation
and outward propagation models. The inward aggregation version aggregated and
incorporated neighborhood information when computing the representation of a
given entity. By extending the neighborhood to multiple hops away, it was
possible to model high-order proximity, thereby capturing users’ long-distance
interests. On the other hand, the outward propagation model propagated users’
potential preferences and explored their hierarchical interests in knowledge
graph entities.

Upadhyay et al. (2021) proposed an explainable job recommendation system by
matching users with the most pertinent jobs, based on their profiles. The
system also provided a human-readable explanation for each recommendation. The
NER module was customized to extract pertinent details from both job postings
and user profiles. These details were utilized to create comprehensible
explanations for each recommendation. By identifying and categorizing
entities, the NER module enhanced the accuracy and understandability of the
textual explanations, providing a clear representation of the reasoning behind
the recommendation system.

####  4.8.3 Dialogue Systems

Commonly, dialogue systems can be categorized into three main types, namely
task-oriented, question-answering, and open-domain (Ni et al., 2022). NER
plays a role in enhancing the natural language understanding of the three
types of dialogue systems, organizing original user messages into semantic
slots, and classifying data domain and user intention (Li et al., 2017b). Abro
et al. (2022) proposed an argumentative dialogue system with NER and other
natural language understanding tasks. The approach can enhance comprehension
of user intent by comprehending injected entities and relationships. For the
question-answering Dimitrakis et al. (2020) and open-domain dialogue systems,
NER also plays a crucial role in the part of intent recognition and knowledge
retrieval. For example, Zhang et al. (2021b) developed a sequence of sub-goals
with external knowledge to improve generation performance. External knowledge
refers to a range of named entities and relationships that are associated with
a conceptual entity. Leveraging external knowledge allows the dialogue system
to deliver a more cohesive small talk from the open domain.

###  4.9 Summary

NER is a very important semantic processing technique for information
retrieval. It is the manifest of cognitive semantics, because named entities
are not simply categorized by their semantics. The classified named entities
also reflect their inherent attributes in people’s cognition. According to
Prototype Theory (see Section 4.1.1), the inherent attributes of named
entities can be represented by prototypes. It is gratifying to observe that a
theory has had a significant influence on research related to few-shot NER. On
the other hand, the ambiguity of named entity classification argued by Graded
Membership (see Section 4.1.2) and Grammatical Category (see Section 4.1.4)
was rarely analyzed from computational linguistic aspects. We also do not see
explainable NER studies that explain why an entity is classified into a
particular category from the perspective of conceptual blending (see Section
4.1.3). The NER research on these aspects is helpful for achieving human-like
intelligence in categorizing named entities.

The availability of numerous named entity recognition (NER) datasets, both in
general and medical domains, has significantly enhanced computational research
in this area. This may be attributed to the great application value of NER, as
well as a wide range of data annotation tools. Encyclopedias knowledge and
domain-specific knowledge also provide external information to help NER models
better understand the context and commonsense. Now, NER has developed many
practical task setups to the need of technical applications, e.g., nested NER,
few-shot NER, joint NER and relation extraction, and downstream tasks, e.g.,
knowledge graph construction, recommendation systems, and dialogue systems.

####  4.9.1 Technical Trends

Task | Reference | Tech | Feature and KB. | Framework | Dataset | Score | Metric
---|---|---|---|---|---|---|---
Nested NER | Katiyar and Cardie (2018) | DL | Emb. | Bi-LSTM | ACE-05 | 70.2% | F1
Straková et al. (2019) | DL | Emb. | LSTM-CRF | ACE-05 | 84.3% | F1
Shibuya and Hovy (2020) | DL | Emb. | LSTM-CRF | ACE-05 | 84.3% | F1
Li et al. (2020c) | DL | BERT, Wikipedia | Unified Framework | ACE-05 | 86.9% | F1
Yan et al. (2021a) | DL | BERT | Pointer Networks | ACE-05 | 84.7% | F1
Finkel and Manning (2009) | Graph | Emb., Constituency Parsing | Hypergraph | GENIA | 72.0% | F1
Muis and Lu (2017) | Graph | Emb., Multigraph Representation | Hypergraph | GENIA | 70.8% | F1
Wang and Lu (2018) | Graph | Emb., Segmental Hypergraphs | Hypergraph | GENIA | 75.1% | F1
| Yang and Tu (2022) | DL | BERT | Pointer Networks | ACE-05 | 85.0% | F1
| Su et al. (2022) | DL | BERT | Pointer Networks | CONLL04 | 88.6% | F1
Few-shot NER (5 shot) |  Fritzler et al. (2019)∗ | DL | Prototypical network | RNN+ CRF | Ontonotes | - | F1
Yang and Katiyar (2020) | DL | BERT | Nearest Neighbor | I2B2 | 22.1% | F1
Das et al. (2022) | DL | BERT | Contrastive Learning | I2B2 | 31.8% | F1
Cui et al. (2021) | DL | BERT | Prompt Tuning | I2B2 | 36.7% | F1
Huang et al. (2022a) | DL | BERT | Prompt Tuning | I2B2 | 43.7% | F1
Joint NER and RE | Miwa and Bansal (2016) | DL | Emb. | Bi-LSTM | ACE-05 | 55.6% | F1
Sun et al. (2020) | Graph | Emb., | Bipartite Graph | ACE-05 | 59.1% | F1
Yan et al. (2021b) | DL | BERT | Partition Filter | ACE-05 | 66.8% | F1
Gupta et al. (2016) | ML | Emb., | Table filling | CoNLL04 | 72.1% | F1
Zhang et al. (2017a) | DL | Emb., | Table filling | ACE-05 | 57.5% | F1
Zheng et al. (2017) | DL | Emb., | Tagging scheme | NYT | 49.5% | F1
| Yu et al. (2020) | DL | Emb., | Tagging scheme | NYT | 59.0% | F1
Task-driven NER |  Shafqat et al. (2022)∗ | DL | Emb., ICD-10 | Bi-LSTM | no public | - | F1
Hirsch et al. (2016)∗ | DL | Emb., UMLS | Bi-LSTM | no public | - | F1
Peng et al. (2019)∗ | DL | BERT, MIMIC-III | Fine Tuning | no public | - | F1
Table 13: A summary of representative NER techniques. The study with ∗ means
it cannot be compared with other studies since it did not report 5-shot
results.

Due to the extensive research conducted on typical NER methods over the years,
researchers are shifting their focus towards NER techniques that are more
applicable to practical scenarios, for example, nested NER, few-shot NER, and
joint NER and relation extraction. Recent technological trends for the
aforementioned NER tasks are summarized in Table 13.

Overall, nested NER can be addressed by multi-label, generation-based, and
hypergraph-based methods. Among them, multi-label methods are straightforward
and easy to implement. However, there are several limitations in the surveyed
multi-label methods. For example, thresholds for multi-label selection are
hard to decide empirically (Katiyar and Cardie, 2018); multiple labels are
suffering sparsity (Straková et al., 2019) or error propagation (Shibuya and
Hovy, 2020), which can lower model performance. Generation-based methods are
flexible. By reformulating NER tasks as question-answering, they can generate
any results which satisfied the pre-defined requirements (Shibuya and Hovy,
2020; Li et al., 2020c). These methods are used for handling Flat NER Skylaki
et al. (2020), nested NER (Yan et al., 2021a), and discontinuous NER Fei et
al. (2021). However, a generation-based method is hard to control what is
generated, even if some studies (Skylaki et al., 2020; Fei et al., 2021; Yang
and Tu, 2022; Su et al., 2022) have attempted to restrict the outputs of
generation-based methods to a specific set of indexes (pointer network). The
core point of the hypergraph-based method is about how to establish a
hypergraph data structure to better represent interaction among all tokens in
a sentence. These methods are good at modeling the interactions among all
tokens in a sentence. It is important to note that the majority of hypergraph-
based methods exhibit a task-specific nature, indicating a limited scope of
applicability. These methods may not be universally applicable, and their
effectiveness may be constrained by the specific task they are designed for.

Few-shot NER is usually achieved by metric learning and prompt tuning. Metric
learning has demonstrated its effectiveness in various few-shot tasks (Kaya
and Bilge, 2019; Fritzler et al., 2019). For few-shot NER tasks, some works
predict the final labels by comparing token-to-token distance (Yang and
Katiyar, 2020; Das et al., 2022) or token-to-prototype distance (Huang et al.,
2022b). These methods have to decide different distance calculation functions
according to different task (Kulis et al., 2013) and suffer instability
introduced by insufficient data. By exploiting the full potential of language
models, prompt tuning is proposed and demonstrated as a promising technology
for few-shot tasks (Liu et al., 2021a; He et al., 2023; Liu et al., 2022b).
Prompt tuning reformulate NER as a mask language model task to reduce the gap
between NER and employed pre-training LMs. The backward is that prompt tuning
needs extra template construction and label word mappings and some studies
have tried to deal with such problems (Huang et al., 2022b).

For Joint NER and RE tasks, we summarize related studies into three groups,
including parameter sharing-based multi-task learning, table-filling strategy,
and customized tagging scheme. Parameter sharing is the basic idea in multi-
task learning, which can be used to enhance the interaction between NER and RE
(Li et al., 2017a; Bekoulis et al., 2018). This method can provide some relief
from error propagation, but it cannot completely eliminate the issue. Also,
this method has to pair every two entities for relation extraction, which
introduces unnecessary redundancy. Table filling-based joint NER and relation
extraction can completely eliminate error propagation by converting NER and
relation extraction into a whole sequence-tagging task (Gupta et al., 2016;
Ren et al., 2021; Ma et al., 2022b). However, these methods have to label
every two token pairs in an input sentence in an enumerable fashion. If
relation extraction is defined as an unidirectional task, the half of
calculations are wasted. Following the idea of the table filling strategy,
tagging scheme-based methods also model the NER and relation extraction as an
integrated task. The fundamental concept of the tagging scheme is to merge the
labels assigned for NER with those assigned for relation extraction into a
unified label (Zheng et al., 2017; Straková et al., 2019). Such a method has
the potential to circumvent issues related to both error propagation and
redundancy; however, it may also lead to a sparsity of labels.

####  4.9.2 Application Trends

Reference | Downstream Task | Feature | Parser | Explain.
---|---|---|---|---
Yao et al. (2022) | Knowledge Graph Construction |  | ✓ |
He et al. (2021) | Knowledge Graph Construction |  | ✓ |
Jiang et al. (2020) | Knowledge Graph Construction |  | ✓ |
Li et al. (2020b) | Knowledge Graph Construction |  | ✓ |
Kim et al. (2012) | Recommendation Systems |  | ✓ | ✓
Adomavicius and Tuzhilin (2011) | Recommendation Systems | ✓ |  |
Zhou et al. (2020) | Recommendation Systems | ✓ |  |
Iovine et al. (2020) | Recommendation Systems | ✓ |  |
Wang et al. (2019a) | Recommendation Systems |  | ✓ |
Li et al. (2017b) | Dialogue Systems |  | ✓ |
Abro et al. (2022) | Dialogue Systems |  | ✓ |
Dimitrakis et al. (2020) | Dialogue Systems |  | ✓ |
Zhang et al. (2021b) | Dialogue System |  | ✓ | ✓
Table 14: A summary of the representative applications of NER in downstream
tasks. ✓denotes the role of NER in a downstream task.

We have discussed three main downstream applications of NER, including
knowledge graph construction, dialogue systems, and recommendation systems.
Table 14 illustrate related studies. Usually, NER is the basic module for
providing recognized entities for further utilization. In this case, a NER
model works as a parser to mine knowledge from unstructured text. The
recognized entities and relations can be used as nodes and edges for knowledge
graph construction. The entities can also serve as intent recognition methods
in recommendation systems, and slot-filling methods in dialogue systems. For
example, Wu et al. (2020a) proposed a pre-trained task-oriented dialogue BERT,
which significantly boosts the performance of a dialogue system by improving
the intent detection sub-task. Wang et al. (2020) proposed a method for
recognizing related spans and value normalization with slot attention to
improve the dialogue system. Besides, we also observe that using the
identified named entities as features can also improve the performance of
recommendation systems, because NER can help identify important entities that
could be useful for making recommendations.

The most common problem is error propagation between NER and other components
in a downstream system. Kim et al. (2018) employed a two-step neural dialog
state tracker to alleviate the impact of the original error. With the
development of PLMs and LLMs, many downstream tasks are organized as end-to-
end processing tasks to achieve higher accuracy and mitigate error propagation
issues. However, we can still observe that NER can improve the explainability
in recommendation and dialogue systems (Kim et al., 2012; Zhang et al.,
2021b), which is also an important aspect of AI research. There is still a
considerable untapped potential for integrating NER with other downstream
tasks, e.g., explaining how concepts blend each other between different
entities; what the inherent attribute of a group of entities the selected
prototypes represent; how robust an identified named entity is.

####  4.9.3 Future Works

Open-domain NER. Compared with typical single-domain NER, open-domain NER has
more categories. Besides, the entity classes are hardly defined in advance.
For such reason, open-domain NER is more capable of handling rapidly expanding
data, and mining more potential knowledge which is hidden in massive
unstructured text data (Hohenecker et al., 2020; Kolluru et al., 2020). Open-
domain NER is significant because it discovers and connects world knowledge
via automatic text mining. Many manually developed lexical resources, e.g.,
WordNet can only cover limited concepts. When the concepts come to multi-word
expressions, manually mining, structuring and updating those concepts can
result in the exponential growth of human efforts. Open-domain NER is helpful
for mitigating human efforts and delivering a knowledge base that connects
entities from different domains.

Multi-lingual NER. In light of the fact that a significant number of languages
in existence lack sufficient annotated data, knowledge transfer from high-
resource languages to low-resource languages can serve as a viable solution to
compensate for the paucity of data (Rahimi et al., 2019; Tedeschi et al.,
2021b). Developing robust multi-lingual NER systems that can perform across
multiple languages will achieve more comprehensive knowledge graphs, linking
entities from different languages. It is valuable because it may lead to a
united concept representation system covering different languages. On the
other hand, the task of developing multi-lingual NER systems is fraught with
difficulties, primarily due to the inherent dissimilarities in entity types
and language structures across different languages. As a result, aligning
entities and transferring knowledge learned from one language to another can
present significant challenges for multi-lingual NER systems.

Unified framework for NER. In the real-world scenario, there exist flat-named
entities, nested entities, and discontinuous entities. Most NER-related
studies only focus on the combination of flat with nested entities or flat
discontinuous entities. Both of them cannot recognize all kinds of entities.
Developing a unified framework to simultaneously handle such a problem becomes
an urgent need for NER (Fei et al., 2021). Hierarchical concept representation
knowledge bases may provide a preliminary ontology that can be used for
organizing entities and their relationships. However, most of the ontology
systems were manually developed by experts. This manually constructed
knowledge may be invalid in specific application scenarios. A potential avenue
for future research in NER is the development of a unified and robust
framework for organizing entities. Such a framework could facilitate the
creation of comprehensive knowledge graphs that capture the relationships
between entities and can better support downstream tasks.

Continual-learning for NER. Humans exhibit a remarkable aptitude for
transferring acquired knowledge from one task to another and retain their
ability to perform the former task even after learning the latter. This
ability is called continuous learning or life-long learning, which a regarded
as an important characteristic of an intelligent system. Also, such ability
can help us continue to use already deployed models when a new class of entity
to be identified appears, rather than developed a new model from scratch (De
Lange et al., 2021). There are some exploratory studies started to pay
attention to such a problem. However, a satisfactory solution has not been
found yet and existing methods still suffer the severe Catastrophic Forgetting
(Monaikul et al., 2021; Xia et al., 2022; Vijay and Priyanshu, 2022).
Continual learning is a critical skill for NER because NER is corpus-
dependent. It is very important to update entity collections and the
associated label sets, when a new corpus arrives (He et al., 2022b). In this
case, detecting new entities and new labels with a former trained NER model
represents a challenging yet highly promising research avenue.

##  5 Concept Extraction

Concept extraction is a process to extract concepts of interest from the text.
To our best knowledge, the task of computational concept extraction was first
proposed by Montgomery (1982), which analyzed the next 5 years of evolutionary
progress in contemporary military message routing systems, with a focus on
their transition towards becoming more advanced and knowledge-based systems.
They argue that taxonomic hierarchies could be constructed to allow property
inheritance of concepts, and therefore to perform rudimentary inference and
analogic reasoning based on the taxonomies. Montgomery (1982) also highlighted
two important sub-tasks of concept extraction for the next-generation
knowledge-based systems from the perspective of 1982, namely lexicon
development and conceptual structure construction.

Recent research on concept extraction has been conducted in various fields of
AI research, including natural language processing (NLP) and data mining
(Miner et al., 2012). Keyphrase generation (Alami Merrouni et al., 2020) is
one of the most common concept extraction tasks. It is a summarization task
focusing on extracting keyphrases from a full passage to help readers quickly
understand the passage, where keyphrases can be understood as the important
concepts within a passage. Methods for keyphrase extraction can be both
extractive (copying from existing words) and abstractive (not copying but
summarizing and abstracting from existing texts). The process of generating
keyphrases facilitates the creation of a lexicon that corresponds to a
specific set of concepts. Another stream of concept extraction aims at the
development of ontological knowledge bases to represent, e.g., commonsense
knowledge (Havasi and Speer, 2007), hypernym and synonym knowledge (Snow et
al., 2006), sentic knowledge (Cambria et al., 2022a). These tasks tried to
extract concepts to fit into pre-defined knowledge structures. Then, the
structured knowledge can be directly used in downstream tasks.

Current concept extraction research is also grounded on related application
scenarios, such as clinical concept extraction (Fu et al., 2020), course
concept extraction (Pan et al., 2017a), and patent concept extraction (Liu et
al., 2020). Clinical concept extraction is to transform massive unstructured
electronic health records data into structured data; Course concept extraction
is to extract important phrases in course captions to help to understand.
Among them, clinical concept extraction is very similar to the information
extraction task in NLP which aims at extracting most of the details in the
unstructured text. Course and patent concept extraction are more similar to
summarization tasks in NLP that target extracting important phrases.

The main difference between concept extraction and NER tasks is that the
extracted concepts or keyphrases are not identified by pre-defined entity
classes. In contrast, they reflect the general idea of their contexts or
target domain whose concepts are being discussed, while the goal of NER is to
extract important factual information from the text. However, there are
overlaps between NER and concept extraction when some concepts of interest,
e.g., proper nouns can be also defined as named entities. Many domain-specific
concept extraction tasks, e.g., clinical concept extraction, course concept
extraction, and patent concept extraction can also be categorized as NER tasks
because they aim at extracting concepts that are related to specific events.
These events are also factual information. We review them in this section
because they define themselves as concept extraction tasks in their original
works. It also has become a trend of domain-specific concept extraction.

Another related field is relation extraction, which is a sub-field of
information extraction. Relation extraction extracts information from raw text
and represents it in the form of a semantic relation between entities (Kartik
Detroja, 2023). The main difference is that, relation extraction targets at
extracting relations between entities, while concept extraction targets at
extracting noun entities. In knowledge graph development, relation extraction
can help to connect nodes of concepts with purposeful relationships.

Concept extraction has also accelerated and contributed to multiple downstream
applications, such as sentiment computing (Cambria et al., 2022a), information
retrieval (Xiong et al., 2017), commonsense explanation generation (Fang and
Zhang, 2022). These applications mostly leverage explicitly extracted
concepts.

Previous survey on concept extraction on focuses on clinical concept
extraction (Fu et al., 2020), which is a particular application field of
concept extraction. In this section, we provide a more comprehensive review on
concept extraction.

###  5.1 Theoretical Research

####  5.1.1 Exemplar Theory

Medin and Schaffer (1978) argued that concepts are represented by a collection
of particular exemplars or individual instances that are linked to the
category. When we categorize an instance, we compare it with multiple specific
exemplars of the category. This is different from Prototype Theory where a new
instance is categorized by comparing the instance to the abstract prototype of
the category (see Section 4.1.1). Medin and Schaffer (1978) formed the task of
concept categorization as a classification task, and conducted experiments
with 32 participants. The experiments showed that the classification judgments
made by participants were impacted by various factors. These factors included
the extent of resemblance between the probe item and exemplars previously
acquired, the number of prior exemplars that shared resemblances with the
probe item, and the similarity present both within and between the categories
of the previously learned exemplars. For concept extraction and
categorization, Exemplar Theory may suggest that models may take categorized
instances into account when categorizing a new instance.

####  5.1.2 Semantic Primitives

Wierzbicka (1972) believed that it is possible to describe every human
language by using a limited number of universal semantic primitives. These
primitives are representative of fundamental concepts that form the basis of
human communication and thinking. Wierzbicka (1972) established 64 universal
semantic primes, which consist of basic words or ideas that cannot be defined
in relation to more elementary concepts. However, these primes can be utilized
to describe all other concepts present within a language. Semantic Primitives
suggest that concepts should be organized as multiple layers from the concrete
to abstract ones. Decision-making that runs on concrete concepts can be
completed through the upper-level abstract concepts that contain those
concrete concepts. Thus, it is critical to represent the hierarchical and
linking relationships between concepts. There are other theories mentioned
before, e.g., Frame Semantics (see Section 2.1.4), that may guide concept
structure development. Frame Semantics highlights the connection of related
concepts, while Semantic Primitives suggest the hierarchical relationships
between concepts and the distinction between primitive concepts and others.

####  5.1.3 Conceptual Spaces

Gardenfors (2004) defined concept as the “theoretical entities that can be
used to explain and predict various empirical phenomena concerning concept
formation”. The author believed that concept representations are multi-
dimensional, where each dimension is indicative of a different characteristic
or property associated with the concept. For example, one could represent the
concept of a car within a conceptual space that includes dimensions such as
size, speed, color, and shape. This is very similar to current vectorial
representations of words or entities in NLP, while the dimensionality of
Conceptual Spaces is explainable by concept properties. Gardenfors (2004) also
placed significant emphasis on the role of context in understanding and
representing concepts. This is due to the fact that different contexts may
emphasize different features or dimensions of concepts. Then, the connections
between concepts are determined by the relationships between their property
similarity in the conceptual space. For example, “dog” and “cat” are similar
in the animal concept space, because their properties are similar; “mammals”
can be separated from “reptiles” by a property difference boundary, although
both are in the animal space. This may encourage concept extraction tasks to
extract both concept entities and properties associated in contexts. This is
because properties define how concepts are connected from the view of
Gardenfors (2004).

###  5.2 Annotation Schemes

From the goal of the keyphrase annotation aspect, there are in general two
types of annotation schemes for keyphrase extraction-liked concept extraction.
The first is to precisely select existing keyphrases from input text, but not
to create semantically-equivalent phrases. The second is to both select
existing keyphrases and create “absent keyphrases” that are necessary but do
not exist in the input text (Hulth, 2003).

From the format of assigned annotations aspect, there are in general two
annotation schemes as well. The first scheme is to directly give the
keyphrases existing in the source text. The second scheme treats the keyphrase
extraction task as a sequence labeling task, and assigns a label to each of
the tokens in source text (Hulth, 2003). The assigned labels in the current
dataset follow a BIO scheme defined in table 10. Specifically, three labels
are used: B (Beginning), I (Inner), and O (Other).

###  5.3 Datasets

Dataset | Task | Source | # Samples | Reference
---|---|---|---|---
Inspec | KE | Inspec database | 2,000 | Hulth (2003)
NUS | KE | Google SOAP API | 211 | Nguyen and Kan (2007)
Krapivin | KE | ACM Digital Library | 2,304 | Krapivin et al. (2009)
SemEval2010 | KE | ACM Digital Library | 244 | Kim et al. (2010)
Twitter | KE | Twitter | 1,000 | Zhang et al. (2016)
KP-20K | KE |  | ACM Digital Library,
---
ScienceDirect,
and Web of Science
567,830 | Meng et al. (2017)
CCF | KE | China Computer Federation | 13,449 | Wang et al. (2018b)
MLDBMD | KE | Academic Conferences | 128.1k | Li et al. (2018)
TempEval | ClCE | Mayo Clinic | 600 | Bethard et al. (2016)
i2b2-2010 | ClCE | Clinical Records | 826 | Uzuner et al. (2011)
n2c2-2018 | ClCE | Clinical Records | 505 | Henry et al. (2020)
MIMIC | ClCE | MIMIC-III Database | 1,610 | Gehrmann et al. (2018)
MOOCs | CoCE | Coursera and XuetangX | 4375 videos | Pan et al. (2017b)
EMRCM | CoCE | Chinese Textbooks | 3,730 pages | Huang et al. (2019b)
USPTO | PCE | USPTO Database | 94,000 | Liu et al. (2020)
Table 15: Concept extraction datasets and statistics. KE denotes Keyphrase
Extraction. ClCE denotes Clinical Concept Extraction. CoCE denotes Course
Concept Extraction. PCE denotes Patent Concept Extraction.

The surveyed popular concept extraction datasets and their statistics can be
viewed in Table 15. Overall the main thread of dataset development is (1)
larger scale of datasets; (2) attending to both extractive keyphrases and
abstractive keyphrases; (3) more fine-grained annotations for tags; (4) more
application domains. Hulth (2003) proposed one of the first keyphrase
extraction datasets, termed Inspec. Their dataset is based on the scientific
papers under Computers and Control, and Information Technology disciplines in
the Inspec database. The keywords used in the scientific papers are selected
as the keyphrases. Abstracts are used as the keyphrase extraction context.
Keywords in scientific papers are used as keyphrases. Each abstract has two
sets of keywords: a set of controlled terms, i.e., terms restricted to the
Inspec thesaurus; and a set of uncontrolled terms that can be any suitable
terms that may or may not be present in the abstracts. They collected 1000
abstracts as a train set, 500 as a validation set, and 500 as a test set.

abstract: "[ ‘A’, ‘scalable’, ‘model’, ‘of’, ‘cerebellar’, ‘adaptive’,
‘timing’, ‘and’, ‘sequencing’, ‘:’, ...]"
doc bio tags: "[ ‘O’, ‘B’, ‘I’, ‘O’, ‘B’, ‘I’, ‘I’, ‘O’, ‘O’, ‘O’, ...]"
extractive keyphrases: "[ ‘scalable model’, ‘cerebellar adaptive timing’, ...
]"
abstractive keyphrase: "[ ‘cerebellar sequencing’, ...]"

Nguyen and Kan (2007) proposed the NUS dataset with the motivation that
keyphrase extraction requires multiple judgments and cannot rely merely on the
single set of author-provided keyphrases. They first used Google Search API to
retrieve scientific publications, and then recruited student volunteers to
participate in manual keyphrase assignments. They finally collect 211
documents, each with two sets of keyphrases: one is given by the original
authors of the paper, and the other is given by student volunteers. The data
format of NUS is the same as Inspec (Hulth, 2003).

Krapivin et al. (2009) proposed the Krapivin dataset, consisting of around
2,000 scientific papers as well as their keywords assigned by the original
authors. The scientific papers were published by ACM in the period from 2003
to 2005, and were written in English. One of the novelties of this dataset is
that the text data in the scientific papers were collected with three distinct
categories: title, abstract, and main body. They finally collect 460 test data
and 1.84k validation data. The data format is similar to Inspec(Hulth, 2003)
but has a title and body in addition to the abstract.

SemEval-2010 Task 5 (Kim et al., 2010) is on automatic keyphrase extraction
from scientific articles. Input for this task is a document from either of the
four domains: distributed systems, information search, and retrieval,
distributed artificial intelligence, and social and behavioral sciences.
Outputs are manually annotated keyphrases for the document. This dataset
contains 144 documents as a train set, and 100 documents as a test set. It
also selects 40 documents from the train set to compose a trial set. For each
set, documents are evenly distributed from the four topics. The annotation
follows the first scheme in Section 5.2. The data format is the same as Inspec
(Hulth, 2003).

Zhang et al. (2016) constructed a keyphrase extract dataset from Twitter using
an automatic text mining method. Their core assumption is that hashtags in a
tweet can be used as keyphrases for the tweet. To construct the dataset, they
first collected 41 million tweets, and then filtered them which contain non-
Latin tokens. URL links, and reply tweets were removed. Thus, the remaining
text only contains tweets and a hashtag. They finally kept 110K tweets. To
evaluate the quality of the collected tweets, they sampled 1000 tweets and
chose three volunteers to score them. As a result, 90.2% tweets are suitable,
and 66.1% are perfectly suitable. The annotation follows the first scheme in
Section 5.2.

tweet: "Hard to believe it but these are REAL state alternatives to taking
Obamacare
funds from the gov’t (via @Upworthy)"
keyphrase: "obamacare"

Pan et al. (2017b) proposed a keyphrase extraction dataset, where data were
sourced from online course captions. Labels are existing phrases in the
captions. The courses are computer science and economics courses, selected
from two famous MOOC platforms — Coursera and XuetangX. Labels were first
filtered from captions using automatic methods and then annotated by two human
annotators. A candidate concept was only labeled as a course concept if the
two annotators were in agreement. As a result, they collected captions from
4375 videos, and 16720 labeled concepts.

course caption: "You might learn how to write a bubble sort and learn why a
bubble sort is not as good as a heapsort."
keyphrase: "[ ‘bubble sort’, ‘heapsort’ ]"

KP-20K (Meng et al., 2017) is a testing dataset, where the input texts are
titles and abstracts of computer science research papers collected from ACM
Digital Library. The labeled keyphrases are the keyphrases shown in the
research papers. The annotation follows the second scheme in Section 5.2,
since the keyphrases given by authors were not necessarily existing keyphrases
in the papers. KP-20K has the same data format as Inspec.

Huang et al. (2019b) were motivated to automatically construct an educational
concept map. The educational concept map shows concepts that will be learned
in courses, as well as the temporal relation between the concepts (e.g., to
learn concept A, it is a prerequisite to learn concept B; Concept A and
concept B can help with the understanding of each other). To construct the
dataset written in Chinese, they first used OCR to obtain the text from
textbooks, then manually labeled key concepts for each textbook (as “key
concept” or “not key concept”) and finally manually annotated the
relationships among the labeled key concepts (as “$w_{i}$ is $w_{j}$’s
prerequisite”, “$w_{i}$ and $w_{j}$ has collaboration relationship”, or “no
relationship”). As a result, they collected 3730 pages in textbooks, 1092 key
concepts, 818 prerequisite relations, and 916 collaboration relations.
However, in their GitHub repo, only keyphrases and relations between
keyphrases can be found, while the text cannot be found.

keyword: "[ ‘average’, ‘weighted average’, ... ]"
relation: "[ ‘average : weighted average’, ... ]"

There are concept extraction datasets focused on a specific domain, e.g.,
clinical concepts (TempEval (Bethard et al., 2016), i2b2-2010 (Uzuner et al.,
2011), n2c2-2018 (Henry et al., 2020), and MIMIC (Gehrmann et al., 2018)),
course concepts (MOOCs (Pan et al., 2017b), and EMRCM (Huang et al., 2019b)),
and patent concepts (USPTO (Liu et al., 2020)). They also followed keyphrase
extraction setups, whereas the targets are to extract concepts of interest.

###  5.4 Knowledge Bases

Besides classical lexicon resources such as WordNet, encyclopedias (including
Baidu Encyclopedias and Wikipedia) can also be used to provide external
knowledge for concepts (Pan et al., 2017b). Methods for extracting concepts
based on embedding techniques may encounter issues with low frequency, where
some of the concepts have infrequent occurrences. Pan et al. (2017b) utilize
word embeddings (Mikolov et al., 2013), which is trained on encyclopedias, to
obtain the semantic embedding for each concept. Inspec database is a
scientific and technical database storing scientific papers. The papers of
this database have been used to construct a keyphrase extraction dataset.

Name | Knowledge | #Entities | Structure
---|---|---|---
WordNet | Lexical | 155,327 | Tree
Baidu Encyclopedia | World | 6,223,649 | Unstructured
Wikipedia | World | 9,834,664 | Unstructured
Inspec | Science | 20,000,000 | Unstructured
Table 16: Useful knowledge bases for concept extraction.

###  5.5 Evaluation Metrics

The field of concept extraction also uses Precision, Recall, and F1-score as
evaluation metrics. Some keyphrase extraction research considered the task as
an information retrieval task. Then, the information retrieval metric, e.g.,
mean average precision (MAP) was also used for keyphrase extraction as the
main measure. It is calculated by taking the average of the average precision
scores for each query in a dataset.

$MAP=\frac{1}{n}\sum_{i=1}^{n}Avg\\_Precision_{i},$ (9)

where n is the total number of queries. $Avg\\_Precision_{i}$ denotes the
averaged precision of query $i$. In the context of keyphrase extraction, the
MAP score is determined by comparing the generated list of keyphrases with a
predefined gold standard set, and evaluating the average precision of the top
$n$ keyphrases, where $n$ corresponds to the total number of keyphrases in the
gold standard set. Each generated keyphrase is considered as a query; The gold
standard set serves as the relevant document.

###  5.6 Annotation Tools

Since the annotation schemes of concept extraction are similar to that of NER.
The aforementioned NER annotation tools can also be used for annotating
concept extraction data. Numerous studies have investigated the utilization of
pre-existing keywords in scientific publications (Hulth, 2003; Nguyen and Kan,
2007; Krapivin et al., 2009; Kim et al., 2010; Chen et al., 2018c) or hashtags
in tweets (Zhang et al., 2016), whereby such in-context information can serve
as labels without requiring additional annotation efforts, provided that the
labels align with the research objectives.

###  5.7 Methods

####  5.7.1 Keyphrase Extraction

The task of keyphrase extraction is to obtain keyphrases from a document to
represent and summarize the document with the keyphrases. There are generally
two trends of methods, namely extractive keyphrase extraction and generative
keyphrase extraction.

Extractive methods appear first but have a systematic disadvantage in that
they can only extract existing phrases in the documents. For example, Meng et
al. (2017) argued that in addition to present keyphrases, there are also
absent keyphrases, which can better summarize a document but do not explicitly
present in the document. Generative methods, however, can generate every
possible word. Therefore generative methods can alleviate the disadvantage of
extractive methods, but might be more difficult because it requires a model to
accurately catch the “semantic meaning” of a document to precisely generate a
keyphrase.

A. Extractive Keyphrase Extraction

Zhang et al. (2016) focused on the task of keyphrase generation on Twitter
data, and framed this task as a sequence labeling task. They proposed a joint-
layer RNN model. For each input token, the joint-layer RNN model outputs two
indicators ($\hat{y}_{1}$ and $\hat{y}_{2}$), where $\hat{y}_{1}$ has two
values $True$ and $False$, indicating whether the current word is a keyword.
$\hat{y}_{2}$ has 5 values $Single$, $Begin$, $Middle$, $End$ and $Not$
indicating the current word is a single keyword, the beginning of a keyphrase,
the middle of a keyphrase, the ending of a keyphrase, or not a part of a
keyphrase, respectively. Their experiments show that the joint-layer RNN model
outperforms both the vanilla RNN model and the LSTM model. However, when
$\hat{y}_{1}$ and $\hat{y}_{2}$ have contradictions, it might be hard to find
an optimal strategy to determine which indicator to refer to. In addition,
joint-layer RNN can only extract an existing sequence as a keyphrase, but
cannot abstractively obtain a (non-existing but better) keyphrase.

Wang et al. (2018b) hypothesized that the performance of keyphrase extraction
could be improved in the unlabeled or insufficiently labeled target domain by
transferring knowledge from a resource-rich domain. They accordingly proposed
a topic-based adversarial neural network (called TANN) that can learn
transferable knowledge across domains efficiently by performing adversarial
training. The experiment section shows that TANN largely outperforms joint-
layer RNN (Zhang et al., 2016).

Li et al. (2018) proposed an unsupervised method for concept mining, which was
motivated by the fact that supervised methods might be hard to generalize to
unseen domains. They assumed that the quality of an extracted concept can be
measured by its occurrence contexts and proposed a pipeline method for concept
mining. The method first populates many raw concepts extracted from text, and
then evaluates the concepts by comparing the embedding of concepts against the
current local context.

Al-Zaidy et al. (2019) identified two limitations of previous supervised
approaches: 1) They classify the labels of each candidate phrase independently
without considering potential dependencies between candidate phrases. 2) They
do not incorporate hidden semantics in the input text. Correspondingly, Al-
Zaidy et al. (2019) addressed keyphrase extraction as a sequence labeling
task, and proposed a model named Bi-LSTM-CRF that unite both the advantages of
LSTM (capturing semantics) and CRF (Conditional Random Field, capturing
dependencies,(Lafferty et al., 2001)). Their results show that Bi-LSTM-CRF
outperforms CopyRNN (Meng et al., 2017) by a large margin.

Fang et al. (2021) hypothesized that previous extractive methods ignore
structured information in the raw textual data (title, topic, and clue words),
which might lead to worse performance. They accordingly proposed a model named
GACEN that can utilize the title, topic, and clue words as additional
supervision to provide guidance. GACEN also utilized CRF to model dependencies
in the output. The experiment section shows that GACEN outperforms Joint-
layer-RNN (Zhang et al., 2016) and CopyRNN (Meng et al., 2017).

B. Generative Keyphrase Extraction

Meng et al. (2017) were motivated that classic keyphrase generation methods
can only extract the keyphrases that appear in the source text. Those methods
are unable to reveal and leverage the full semantics for keyphrase ranking.
Consequently, they proposed an RNN-based generative model incorporating a
copying mechanism (Gu et al., 2016) (named with CopyRNN), which can generate
absent keyphrases. Their method uses an encoder-decoder architecture to catch
the semantics of the input text.

Previous methods such as Meng et al. (2017) suffered from both coverage (not
all keyphrases are extracted) and repetition (similar keyphrases are
extracted) problems. For the coverage issue, Chen et al. (2018c) integrated a
coverage mechanism (Tu et al., 2016) into their approach, which enhances the
attention distributions of multiple keyphrases in order to cover a wider range
of information within the source document and effectively summarize it into
keyphrases. For the repetition issue, they constructed a target side review
context set that contains contextual information of generated phrases.

Ye and Wang (2018) believed that although sequence-to-sequence (seq2seq)
models have achieved good performance, model training still relies on large
amounts of labeled data. Correspondingly, they leveraged unsupervised learning
methods such as TF-IDF and self-learning algorithms to create keyphrase labels
for large amounts of unlabeled data. Then, they train their model with a
mixture of self-labeled and labeled data together for training. They also used
multi-task learning to train their model. Experiments show that their
performance outperforms previous works.

Chen et al. (2019) argued that prior research on keyphrase generation has
treated the document title and main body in the same manner, overlooking the
significant role that the title plays in shaping the overall document. They
accordingly proposed a Title-Guided Network (TG-Net) where the title is
additionally employed as a query-like input to particularly assign attention
to the title. The performance of TG-Net outperforms CopyRNN (Meng et al.,
2017). Their ablation study also shows the importance of additional attention
to the title.

####  5.7.2 Structured Concept Extraction

Compared with keyphrase extraction-liked concept extraction, structured
concept extraction aimed to develop an ontology where concepts are connected
with each other by certain relationships. Here, we introduce three knowledge
bases resulting from concept extraction: WordNet, ConceptNet, and SenticNet.
Out of them, WordNet focuses more on a word-level ontology, ConceptNet focuses
more on a concept-level ontology (e.g., also including phrases for concepts),
and SenticNet is a concept-level ontology focusing on contributing to
sentiment analysis tasks.

WordNet is a manually developed knowledge base, where words and concepts are
hierarchically organized. Snow et al. (2006) proposed a taxonomy induction
method to expand WordNet 2.1 concepts by automatic noun hyponym acquisition,
achieving 10,000 novel synsets with 84% precision. Compared to previous
methods that relied on individual classifiers to uncover new relationships
based on pre-designed or automatically extracted textual patterns, the
proposed approach considers input from multiple classifiers to enhance the
overall structure of the taxonomy and prioritizes the optimization of the
entire taxonomy structure with a probabilistic architecture. Snow et al.
(2006) also proposed an (m,n)-cousin classification-based model to learn
coordinate terms, which allows it to integrate heterogeneous evidence from
different classifiers and choose the correct word sense to which to attach a
new hypernym. The evaluation of the inferred taxonomies produced by the
algorithm was conducted by directly comparing them with the WordNet 2.1
taxonomy. This was achieved by testing each taxonomy using a set of human
judgments of noun pairs sampled from newswire text, to determine the hypernym
and non-hypernym relationships.

ConceptNet (Havasi and Speer, 2007) grew out of Open Mind Common Sense project
that aimed at commonsense acquisition. Contributors delivered knowledge by
fulfilling blanks within a sentence, For example, given “[ ] can be used to [
]”, the concepts, e.g., “ink” and “print” and the associated relationship
“UsedFor” can be obtained. ConceptNet aimed to obtain and structure concepts
automatically from natural language. It obtained concepts (the nodes) in the
form of noun phrases, verb phrases, adjective phrases, prepositional phrases,
or complete verb phrases (Havasi and Speer, 2007). The edges of ConcepNet are
predicates that represent the relationships between two concept nodes, such as
“IsA”, “PartOf”, UsedFor, and more. Havasi and Speer (2007) defined 21 basic
relation types. In the latest ConceptNet 5.5 (Speer et al., 2017), the
relations are increased to 36. Concepts and predicates were obtained via
pattern matching. Each collected sentence is compared with pre-defined regular
expressions, e.g., “NP is used for VP”(UsedFor), “NP is a kind of NP”(IsA),
“NP can VP” (CapableOf). NP (noun phrases) and VP (verb phrases) are concepts,
while “UsedFor”, “IsA”, and “CapableOf” are predicates. In the case of a
complex sentence that contains several clauses, the patterns are employed to
extract a simpler sentence from it, which can then be subjected to the concept
and predicate extraction process. To evaluate ConceptNet, its assertions were
compared with those in similar lexical resources to determine their alignment.

SenticNet is a commonsense knowledge base that is used for affective
computing. The concepts were extracted by a graph-based semantic parsing
method (Cambria et al., 2014) and assigned with sentiment polarity labels.
Sentences are divided into chunks, e.g., “go walk”, first. Then, verb-noun
chunks are normalized by stemming, and included in the concept set. The PoS-
based bigram algorithm is used to extract object concepts. To capture event
concepts, the approach explores matches between object concepts and normalized
verb-noun chunks. Finally, single-word concepts, e.g., “house” that have
appeared in the clause as multi-word concepts “beautiful house” are deemed
redundant and are therefore excluded. In the following version of SenticNet
(Cambria et al., 2016), the authors proposed an automatic method to discover
primitives from the SenticNet concepts, based on hierarchical clustering and
dimensionality reduction. Thus, the “animal” concept can be identified as the
primitive of “cat”, “dog”, or “pet”. Later, Cambria et al. (2022b) proposed a
pipeline method for concept extraction, which is used for expanding SenticNet
with multi-word expressions. They first deconstructed text using sentence
chunking, semantic parser, and PoS tagging. Then, verb and noun chunks are
extracted and normalized as concepts. The proposed method offers novel
contributions in utilizing morphology for syntactic normalization and
employing primitives for semantic normalization. The method was evaluated on a
sentiment analysis task, achieving explainable and primitive- and concept-
level sentiment analysis via algebra operations. The latest version of
SenticNet (Cambria et al., 2022a) offers the function that sentiment
predictions can be effectively conducted on the primitive level, mitigating
symbol grounding problems.

An important task of concept extraction is to abstract concept representations
from entities. Unlike SenticNet which obtains abstract concepts (primitives)
by selecting the most typical entities from a group of extracted similar
entities (Cambria et al., 2016), Ge et al. (2022) proposed a conceptualization
method that can directly abstract concepts from input text. The task is
realized in the metaphor identification and interpretation domain. The authors
aimed to generate concept mappings from metaphorical word pairs to explain the
metaphoricity of the word pairs. For example, given “blind alley”, “street is
adult” can be automatically generated. This work is the realization of
conceptual metaphor theory (Lakoff and Johnson, 1980) (see Section 6.1.3) that
the generated concept mapping explains the mapping of source (e.g., adult) and
target (e.g., street) concepts of a metaphor. The conceptualization (e.g.,
from “alley” to “street”) was achieved by selecting the most appropriate
hypernym on the chain from the leaf node of “alley” to the root node “entity”
in WordNet. The most appropriate hypernym is defined as the node that can
cover the major senses of the leaf, meantime, keeping it as concrete as
possible373737Intuitively, “entity” can cover all possible senses of the
“alley” in WordNet, while it is not the ideal concept representation of
“alley”, because it is too abstract. Thus, the authors aimed at a concrete
concept representation that can cover the majority senses of a word.. The
conceptualization and concept mapping method was evaluated on a metaphor
identification task, yielding better performance and explainability on the
task. Subsequently, within MetaPro Online (Mao et al., 2023b), the
conceptualization algorithm is synergistically integrated with sequential
metaphor identification and interpretation techniques, culminating in the
attainment of end-to-end concept mapping generation from full sentences.

####  5.7.3 Domain-specific Concept Extraction

A. Clinical Concept Extraction

The task of clinical concept extraction is to extract structural information
from unstructured clinical narratives (Fu et al., 2020). Li and Huang (2016)
constructed a dataset for a seminal task called “UTA-DLNLP at SemEval-2016
Task 12” for clinical concept extraction. A system developed for this task
should task raw clinical notes or pathology reports as input, and identify
event expressions consisting of the “the spans of the expression in the raw
text”, “contextual modality”, “degree”, “polarity”, and “type”. As a baseline
for this task, they propose a convolutional neural network to learn hidden
feature representations for predictions, taking text and part-of-speech tags
as input.

Liu et al. (2017b) adopted BiLSTM to recognize the entity in clinical text.
They found that BiLSTM outperforms the CRF baselines. Gehrmann et al. (2018)
compared CNN with classic rule-based methods, bag of words, n-grams, and
embedding-based logistic regression. They found that CNN is a valid
alternative to rule-based and classic NLP methods, and should be further
investigated. Yang et al. (2020) comprehensively explored 4 widely used
transformer-based architectures, including BERT (Devlin et al., 2019), RoBERTa
(Liu et al., 2019b), ALBERT (Lan et al., 2020), and ELECTRA (Clark et al.,
2020). They compared the 4 models to long short-term memory conditional random
fields (LSTM-CRFs) (Huang et al., 2015) baselines and found that transformer-
based models are effective for clinical concept extraction tasks. Lange et al.
(2020) proposed a joint model for both clinical concept extraction and de-
identification tasks. De-identification is important since in some clinical
concept extraction scenarios, the privacy of patients should be protected.
They hypothesized that jointly modeling the two tasks can be beneficial, and
proposed two end-to-end models. One is a multitask model where the tasks share
the input representation across tasks; the other is a stacked model, which
used the privacy token predictions to mask the corresponding embeddings in the
input layer and only use the masked embeddings for concept extraction. They
found that the performance of the concept extraction model can be improved by
training and evaluating it on anonymized data, thereby confirming their
initial hypothesis.

B. Course Concept Extraction

In tasks involving the extraction of course concepts, the concepts are
typically defined as the knowledge concepts that are taught in the course
videos, as well as the related topics that aid in the students’ comprehension
of the course videos (Pan et al., 2017b). Identifying course concepts at a
fine level is very important, as students with different backgrounds need
different concepts to quickly understand the main content of a course (Pan et
al., 2017b).

Pan et al. (2017b) contributed the first attempt to systematically investigate
the problem of course concept extraction in MOOCs. in the past, course
concepts were presented by instructors at a general level, with only a few
concepts being covered in an entire course video. However, they emphasized the
significance of identifying course concepts at a granular level, i.e.,
automatically identifying all course concepts from each video clip, to
facilitate easier comprehension. They identified a challenge for the task that
the course concept appears at a low frequency mainly because the different
courses have different concepts. They accordingly proposed to utilize word
embedding to catch the semantic relations between words and incorporate online
encyclopedias to learn the latent representations for candidate course
concepts. They also proposed a graph-based propagation algorithm to rank the
candidates based on learned representations.

Wang et al. (2018a) argued that external knowledge must be involved to solve
the concept extraction problem and proposed to utilize both the structured and
unstructured data in Wikipedia to provide external knowledge to concept
extraction. Their results show that their method outperforms prior works (Pan
et al., 2017b).

C. Patent Concept Extraction

Liu et al. (2020) developed a framework to extract technical concepts from
patents. Patent documents have different structures than other documents. For
instance, they have “title”, “abstract”, and “claim”, which exhibit a multi-
level of information. Motivated by this, the authors proposed a framework
named UMTPE, which can effectively leverage multi-level information to extract
concepts.

###  5.8 Downstream Applications

####  5.8.1 Sentiment Computing

SenticNet 7 (Cambria et al., 2022a) is a neuro-symbolic sentiment analysis
system, based on SenticNet knowledge base. It assumes that concepts that share
the same primitive would have similar sentiments. One can use algebra
operations to achieve sentiment analysis with the symbolic and structural
knowledge base. Incorporating a symbolic knowledge base and a transparent
algorithm provides SenticNet’s reasoning process with the benefit of
interpretability and accuracy.

Li et al. (2023) proposed a neuro-symbolic system for conversational emotion
recognition. ConceptNet was used as a knowledge base to acquire commonsense
knowledge out of context. For example, if a person mentions that he will “chop
all onions we have and cry”, another conversation participant expresses
“disgust” emotion. This is because “onion IsA lacrimator” is a commonsense in
ConceptNet. Such a commonsense cannot be obtained from the dictionary meanings
of “onion” and the context, while ConceptNet commonsense knowledge provides
the evidence and explainability to infer such an emotional status from the
context. The authors used an utterance dependency parser and a neural network
to learn symbolic knowledge to enhance the explainability and accuracy of
their method.

By using the concept mapping method from the work of Ge et al. (2022), Han et
al. (2022) used concept mappings to support depression detection and
explanation. The hypothesis is that depression patients may have similar
cognition patterns that are reflected in their metaphorical expressions. Thus,
they used concept mappings as additional features besides tweets. The concept
mappings were generated from tweets that contained metaphors. They also
proposed an explainable encoder that can identify significant concept mappings
that contribute to depression detection. The concept mappings also improve the
accuracy of depression detection, besides explaining the common concept
mapping patterns.

####  5.8.2 Information Retrieval

Xiong et al. (2017) manually analyzed the potential problems of a literature
search website SemanticScholar.org, and found that the issue of “Concept Not
Understood” represents one of the most significant challenges. The reason is
that previous methods measure similarity based on text, but not on their
semantic embeddings. As a result, they proposed an embedding-based similarity
matching method, which extracts the concepts in both query and documents and
measures the similarity between these concepts to obtain the similarity
between a query and a document. Liu et al. (2018b) used extracted knowledge
concepts as one of the inputs to obtain a unified semantic representation for
educational excises. The representation is further used to retrieve similar
excises based on similarity with other representations.

####  5.8.3 Dialogue Systems

Young et al. (2018) integrated commonsense knowledge from ConceptNet in their
dialogue system. They believed that in human dialogues, individuals responding
to each other is not dependent on the most recent utterance only, but also on
recollecting pertinent information related to the concepts addressed within
the dialogue, e.g., commonsense. Thus, in retrieval-based dialogue generation,
the model considers both the message content and relevant commonsense
knowledge to effectively choose a suitable response.

Huang et al. (2020) proposed a new dialogue coherence evaluation matric,
termed Graph-enhanced Representations for Automatic Dialogue Evaluation
(GRADE). Liu et al. (2016) argued that traditional BLEU-liked statistic-based
metrics are biased in response coherence. Thus, Huang et al. (2020) were
motivated to propose a metric that measures the coherence by the topics of
utterances. They believe that a cohesive exchange of dialogues is
characterized by a seamless transition between topics. Thus, they used a
ConceptNet-based method to construct topic-level dialogue graphs. The topic-
level dialogue graphs were constructed by connecting the concepts that are
extracted from utterances. The edge was weighted and undirected, which was
derived from the shortest path between two nodes in the ConceptNet. Such an
evaluation metric can better represent the coherence of topics between
utterances because it measures the relatedness of concepts from different
utterances.

####  5.8.4 Commonsense Explanation Generation

Fang and Zhang (2022) grounded concept extraction in the context of
commonsense explanation generation. Commonsense explanation generation aims to
generate an explanation in natural language to explain the reason why a
statement is anti-commonsense. For example, given “he took a nap in the sink”,
the model aimed to generate “a sink is too small and dirty to take a nap in”.
The concepts, “small” and “dirty” (bridge concepts), are obtained via a
prompt-tuning method. The authors developed a masked word prediction template
to query the bridge concepts that are most likely to appear in the “mask”
position. Then, they use a generator to generate the explanation with the
concatenation of the original statement and the discrete bridge concepts. This
method improves the explainability in explaining why a statement is anti-
commonsense.

###  5.9 Summary

A concept is an abstract idea that is reflected in the mind. Concept
extraction is the foundation of detecting the main idea of a context and
developing conceptual knowledge bases. Related theoretical research showed
that concepts may be abstracted from multiple specific exemplars (Medin and
Schaffer, 1978) or prototypes (Rosch, 1973). There are limited primitives that
construct human cognition and reasoning, which are the foundation of complex
concepts (Wierzbicka, 1972). According to Gardenfors (2004), conceptual space
is multi-dimensional. The similarity between concepts can be measured by the
similarity between concept properties. These theoretical research works frame
the tasks of concept extraction from the perspectives of lexicon development
and conceptual structure construction. On the other hand, current
computational concept extraction methods divide this task into three
categories, namely keyphrase extraction, structured concept extraction, and
domain-specific concept extraction. We found that the existing computational
approaches inadequately address the tasks that have been put forth by the
academic community’s theoretical research. Although current concept extraction
methods are limited, this task has greatly improved the explainability of
downstream tasks such as sentiment computing, information extraction, and
counter-commonsense recognition.

####  5.9.1 Technical Trends

Task | Reference | Techniques | Feature and KB | Framework | Dataset | Score | Metric
---|---|---|---|---|---|---|---
Extractive KE | Zhang et al. (2016) | DL | word2vec | Joint-layer RNN | Twitter | 86.40% | F1
Wang et al. (2018b) | DL | word2vec | BiLSTM, adversarial loss | CCF | 29.60% | F1
Li et al. (2018) | Pipeline | word2vec | Similarity matching | MLDBMD | 97.00% | MAP
Al-Zaidy et al. (2019) | DL | word2vec | BiLSTM-CRF | KP-20K | 35.63% | F1
Fang et al. (2021) | DL | word2vec | Attention; CRF | KP-20K | 45.69% | F1
Generative KE | Meng et al. (2017) | DL | word2vec | RNN | KP-20K | 32.80% | F1@5
Chen et al. (2018c) | DL | word2vec | Seq2seq | Krapivin | 31.80% | F1@5
Ye and Wang (2018) | DL | word2vec | Seq2seq, semi-supervised | KP-20K | 30.80% | F1@5
Chen et al. (2019) | DL | word2vec |  | Seq2seq, additional
---
Title input
KP-20K | 37.20% | F1@5
Structured CE | Havasi and Speer (2007) | Knwl. eng. | textual patterns | Pattern matching | - | - | -
Snow et al. (2006) | SL | feature vectors, WN | Probabilistic | - | - | -
SenticNet |  | chunking,
---
sem. pars.,
PoS tag.
syntactic patterns | Syntactic parsing | - | - | -
Ge et al. (2022) | SL | statistics, WN | Elbow algorithm | - | - | -
Clinical CE | Li and Huang (2016) | DL |  | token mention, pos
---
tag, word shape
CNN | TempEval | 78.80% | F1
Liu et al. (2017b) | DL | word2vec, character2vec | BiLSTM | i2b2-2010 | 85.78% | F1
Gehrmann et al. (2018) | DL | word2vec | CNN | MIMIC | 76.00% | F1
Yang et al. (2020) | DL | word2vec | Transformer | n2c2-2018 | 88.36% | F1
Lange et al. (2020) | DL | word2vec | Multitask-biLSTM | i2b2-2010 | 88.90% | F1
Course CE | Pan et al. (2017b) | Graph | word2vec; Encyclopedia | Graph-based propagation | MOOCs | 41.60% | MAP
Wang et al. (2018a) | Graph | word2vec, Wikipedia | Graph-based propagation | MOOCs | 47.50% | MAP
Patent CE | Liu et al. (2020) | ML |  | self pretrained
---
word2vec, DBpedia
Clustering | USPTO | 43.37% | F1
Table 17: A summary of representative concept extraction techniques. KE
denotes keyphrase extraction. CE denotes concept extraction. SL denotes
statistical learning. Knwl. eng. denotes knowledge engineering. SenticNet
denotes the works of Cambria et al. (2014, 2016, 2022a, 2022b). We do not show
the evaluation results for structured concept extraction methods, because they
all used very task-specific evaluation methods and datasets, where the results
are not comparable.

Within the domain of keyphrase extraction, generative keyphrase extraction
takes advantage of generating “absent keyphrases”, compared to extractive
keyphrase extraction. Both tasks followed the general development of the NLP
fields. They likely considered the task as a sequence labeling task
(extractive keyphrase extraction) or a generation task (generative keyphrase
extraction), and used typical NLP frameworks, e.g., sequence labeling and
sequence to sequence frameworks. However, it is unclear if these general NLP
frameworks have really learned how to summarize the main idea of context or
just have learned by label distributions. There were no task-specific
mechanisms proposed to explicitly learn the keyphrase extraction task on the
concept level, with an explainable decision-making process. On the other hand,
keyphrase extraction-based concept extraction is helpful for obtaining concept
lexicons. However, compared to structured concept extraction, keyphrase
extraction cannot learn the relationships between concepts. The theoretical
research of Conceptual Spaces from Gardenfors (2004) suggested that the
similarity between concepts can be measured by their properties. It suggests
that keyphrase extraction-based concept extraction should consider extracting
properties together with keyphrases. Thus, the later works can use keyphrases
and the associated properties to structure concepts by similarities.

In contrast, structured concept extraction research likely utilized
statistical learning and syntactic parsing methods. This is because the aim of
structured concept extraction is to develop a large knowledge base or detect
structured relationships between concepts. Labeled data are insufficient in
these areas. Thus, unsupervised methods are preferred. However, the concept
knowledge base development is task-specific. As a result, the concepts in
different knowledge bases share different relationships. For example,
ConceptNet aimed to parse concepts sharing 36 commonsense relationships;
Stanford WordNet (Snow et al., 2006) was expended in synonyms and hypernyms
relationships; SenticNet grouped concepts and extract primitives for sentiment
computing; Ge et al. (2022) abstracted concepts for concept mappings. Then,
the evaluation of different concept extraction methods is different. Most of