# Universal Information Extraction as Unified Semantic Matching

Jie Lou1, Yaojie Lu211footnotemark: 1, Dai Dai1, Wei Jia1, Hongyu Lin2,
Xianpei Han2,322footnotemark: 2, Le Sun2,3, Hua Wu1 Equally contribution.
Corresponding authors.

###### Abstract

The challenge of information extraction (IE) lies in the diversity of label
schemas and the heterogeneity of structures. Traditional methods require task-
specific model design and rely heavily on expensive supervision, making them
difficult to generalize to new schemas. In this paper, we decouple IE into two
basic abilities, structuring and conceptualizing, which are shared by
different tasks and schemas. Based on this paradigm, we propose to universally
model various IE tasks with Unified Semantic Matching (USM) framework, which
introduces three unified token linking operations to model the abilities of
structuring and conceptualizing. In this way, USM can jointly encode schema
and input text, uniformly extract substructures in parallel, and controllably
decode target structures on demand. Empirical evaluation on 4 IE tasks shows
that the proposed method achieves state-of-the-art performance under the
supervised experiments and shows strong generalization ability in zero/few-
shot transfer settings.

## Introduction

Information extraction aims to extract various information structures from
texts (Andersen et al. 1992; Grishman 2019). For example, given the sentence
“Monet was born in Paris, the capital of France”, an IE system needs to
extract various task structures such as entities, relations, events, or
sentiments in the sentence. It is challenging because the target structures
have diversified label schemas (person, work for, positive sentiment, etc.)
and heterogeneous structures (span, triplet, etc.).

Traditional IE model leverages task- and schema-specialized architecture,
which is commonly specific to different target structures and label schemas.
The expensive annotation leads to limited predefined categories and small data
size in general domains for information extraction tasks. From another
perspective, task-specific model design makes it challenging to migrate
learned knowledge between different tasks and extraction frameworks. The above
problems lead to the poor performance of IE models in low-resource settings or
facing new label schema, which greatly restricts the application of IE in real
scenarios.

Very recently, Lu et al. (2022) proposed the concept of universal information
extraction (UIE), which aims to resolve multiple IE tasks using one universal
model. To this end, they proposed a sequence-to-sequence generation model,
which takes flattened schema and text as input, and directly generates
diversified target information structures. Unfortunately, all associations
between information pieces and schemas are implicitly formulated due to the
black-box nature of sequence-to-sequence models (Alvarez-Melis and Jaakkola
2017). Consequently, it is difficult to identify what kind of abilities and
knowledge are learned to transfer across different tasks and schemas.
Therefore we have no way of diagnosing under what circumstances such transfer
learning across tasks or schemas would fail. For the above reasons, it is
necessary to explicitly model and learn transferable knowledge to obtain
effective, robust, and explainable transferability.

Figure 1: The USM framework for UIE. USM takes label schema and text as input
and directly outputs the target structure through the Structuring and
Conceptualizing operations. Figure 2: The overall framework of Unified
Semantic Matching.

We find that, as shown in Figure 1, even with diversified tasks and extraction
targets, all IE tasks can be fundamentally decoupled into the following two
critical operations: 1) Structuring, which proposes label-agnostic basic
substructures of the target structure from the text. For example, proposing
the utterance structure “Monet” for entity mention and “born in” for event
mention, the associated pair structure (“Monet”, “Paris”) for relation
mention, and (“born in”, “Paris”) for event argument mention. 2)
Conceptualizing, which generalizes utterance and paired substructures to
corresponding target semantic concepts. More importantly, these two operations
can be explicitly reformulated using a semantic matching paradigm when given a
target extraction schema. Specifically, structuring operations can be viewed
as building specific kinds of semantic associations between utterances in the
input text, while conceptualizing operations can be regarded as matching
between target semantic labels and the given utterances or substructures.
Consequently, if we universally transform information extraction into
combinations of a series of structuring and conceptualizing, reformulate all
these operations with the semantic matching between structures and schemas,
and jointly learn all IE tasks under the same paradigm, we can easily conduct
various kinds of IE tasks with one universal architecture and share knowledge
across different tasks and schemas.

Unfortunately, directly conducting semantic matching between structures and
schemas is impractical for universal information extraction. First, sentences
have many substructures, resulting in a large number of potential matching
candidates and a large scale of matching, which makes the computational
efficiency of the model unacceptable. Second, the schema of IE is structural
and hard to match with the plain text. In this paper, we propose directed
token linking for universal IE. The main idea is to transform the structuring
and conceptualizing into a series of directed token linking operations, which
can be reverted to semantic matching between utterances and schema.

Based on the above observation, we propose USM, a unified semantic matching
framework for universal information extraction (UIE), which decomposes
structures and verbalizes label types for sharing structuring and
conceptualizing abilities. Specifically, we design a set of directed token
linking operations (token-token linking, label-token linking, and token-label
linking) to decouple task-specific IE tasks into two extraction abilities. To
learn the common extraction abilities, we pre-train USM by leveraging
heterogeneous supervision from linguistic resources. Compared to previous
works, USM is a new transferable, controllable, efficient end-to-end framework
for UIE, which jointly encodes extraction schema and input text, uniformly
extracts substructures, and controllably decodes target structures on demand.

We conduct experiments on four main IE tasks under the supervised, multi-task,
and zero/few-shot transfer settings. The proposed USM framework achieves
state-of-the-art results in all settings and solves massive tasks using a
single multi-task model. Under the zero/few-shot transfer settings, USM shows
a strong cross-type transfer ability due to the shared structuring and
conceptualizing obtained by pre-training.

In summary, the main contributions of this paper are:

  1. 1.

We propose an end-to-end framework for universal information extraction – USM,
which can jointly model schema and text, uniformly extract substructures, and
controllably generate the target structure on demand.

  2. 2.

We design three unified token linking operations to decouple various IE tasks,
sharing extraction capabilities across different target structures and
semantic schemas and achieving “one model for solving all tasks” by multi-task
learning.

  3. 3.

We pre-train a universal foundation model with large-scale heterogeneous
supervisions, which can benefit future research on IE.

Figure 3:  Illustrations of Directed Token Linking. Token-Token Linking
structures utterance and association pair substructures from the text, Label-
Token Linking conceptualizes the utterance, and Token-Label Linking
conceptualizes the association pair. In practice, we employ different label
symbols “[L]” for utterance conceptualizing: “[LM]” for the label of single
mention, such as entity types and event trigger types; “[LP]” for the
predicate of association pair, such as relation types and event argument
types.

## Unified Semantic Matching via Directed Token Linking

Information extraction is structuring the text’s information and elevating it
into specific semantic categories. As shown in Figure 2, USM takes the
arbitrary extraction label schema $l$ and the raw text $t$ as input and
directly outputs the structure according to the given schema. For example,
given the text “Monet was born in Paris, the capital of France”, USM needs to
extract (“France”, capital, “Paris”) for the relation type capital and
(person, “Monet”)/(country, “France”) for the entity type person and country.
The main challenges here are: 1) how to unifiedly extract heterogeneous
structures using the shared structuring ability; 2) how to uniformly represent
different extraction tasks under diversified label schemas to share the common
conceptualizing ability.

In this section, we describe how to end-to-end extract the information
structures from the text using USM. Specifically, as shown in Figure 3, USM
first verbalizes all label schemas (Levy et al. 2017; Li et al. 2020; Lu et
al. 2022) and learns the schema-text joint embedding to build a shared label
text semantic space. Then we describe three basic token linking operations and
how to structure and conceptualize information from text using these three
operations. Finally, we introduce how to decode the final results using
schema-constraint decoding.

### Schema-Text Joint Embedding

To capture the interaction between label schema and text, USM first learns the
joint contextualized embeddings of schema labels and text tokens. Concretely,
USM first verbalizes the extraction schema $s$ as token sequence
$l=\\{l_{1},l_{2},...,l_{|l|}\\}$ following the structural schema instructor
(Lu et al. 2022), then concatenates schema sequence $l$ and text tokens
$t=\\{t_{1},t_{2},...,t_{|t|}\\}$ as input, and finally computes the joint
label-text embeddings
$\mathbf{H}=[\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{|l|+|t|}]$ as
follow:

$\mathbf{H}=\text{Encoder}(l_{1},l_{2},...,l_{|l|},t_{1},t_{2},...,t_{|t|},\mathbf{M})$
(1)

where $\text{Encoder}(\cdot)$ is a transformer encoder, and
$\mathbf{M}\in\mathbb{R}^{(|l|+|t|)\times(|l|+|t|)}$ is the mask matrix that
determines whether a pair of tokens can be attended to each other.

### Token-Token Linking for Structuring

After obtaining the joint label-text embeddings
$\mathbf{H}=[\mathbf{h}_{1}^{l},...,\mathbf{h}_{|l|}^{l},\mathbf{h}_{1}^{t},...,\mathbf{h}_{|t|}^{t}]$,
USM structures all valid substructures using Token-Token Linking (TTL)
operations:

  1. 1.

Utterance: a continuous token sequence in the input text, e.g., entity mention
“Monet” or event trigger “born in”. We extract a single utterance with inner
span head-to-tail (H2T) linking, as shown in Figure 3. For example, to extract
the span “Monet” and “born in” as valid substructures, USM utilizes H2T to
link “Monet” to itself and link “born” to “in”.

  2. 2.

Association pair: a basic related pair unit extracted from the text, e.g.,
relation subject-object pair (“Monet”, “Paris”) or event trigger-argument
(“born in”, “Paris”). We extract span pairs with head-to-head (H2H) and tail-
to-tail (T2T) linking operations. For example, to extract the subject-object
pair “Monet” and “Paris” as a valid substructure, USM links “Monet” and
“Paris” using H2H as well as links “Monet” and “Paris” using T2T.

For the above three token-to-token linking (H2T, H2H, T2T) operations, USM
respectively calculates the token-to-token linking score
$\mathbf{s}_{\text{TTL}}(t_{i},t_{j})$ over all valid token pair candidates
$\langle t_{i},t_{j}\rangle$. For each token pair $\langle
t_{i},t_{j}\rangle$, the linking score $\mathbf{s}_{\text{TTL}}(t_{i},t_{j})$
is calculated as:

$\begin{gathered}\mathbf{s}_{\text{TTL}}(t_{i},t_{j})=\text{FFNN}^{l}_{\text{TTL}}(\mathbf{h}_{t}^{i})^{T}\mathbf{R}_{j-i}\text{FFNN}^{r}_{\text{TTL}}(\mathbf{h}_{t}^{j})\end{gathered}$
(2)

where $\text{FFNN}^{l/r}$ are feed-forward layers with output size $d$.
$\mathbf{R}_{j-i}\in\mathbb{R}^{d\times d}$ is the rotary position embedding
(Su et al. 2021, 2022) that can effectively inject relative position
information into the valid structure mentioned above.

### Label-Token Linking for Utterance Conceptualizing

Given label token embeddings $\mathbf{h}_{1}^{l},...,\mathbf{h}_{|l|}^{l}$ and
text token embeddings $\mathbf{h}_{1}^{t},...,\mathbf{h}_{|t|}^{t}$, USM
conceptualizes valid utterance structures with label-token linking (LTL)
operations. The output of LTL is a pair of label name and text mention, e,g.,
(person, “Monet”), (country, “France”), and (born, “born in”). There are two
types of utterance conceptualizing: the first one is the type of mention,
which indicates assigning the label types to every single mention, such as
entity type person for entity mention “Monet”; the second one is the predicate
of object, which assigns the predicate type to each object candidate, such as
relation type birth place for “Paris” and event argument type place for
“Paris”.

We conceptualize the type of mention and the predicate of object with the same
label-to-token linking operation, thus enabling the two label semantics to
reinforce each other. Following the head-tail span extraction style, we name
each substructure with label-to-head (L2H) and label-to-tail (L2T) linking
operations. For the pair of label name birth place and text span Paris, USM
links the head of the label birth with the head of text span “Paris” and links
the tail of label place with the tail of text span “Paris”.

For the above two label-to-token linking (L2H, L2T) operations, USM
respectively calculates the label-to-token linking score
$\mathbf{s}_{\text{LTL}}(l_{i},t_{j})$ over all valid label and text token
pair candidates $\langle l_{i},t_{j}\rangle$:

$\begin{gathered}\mathbf{s}_{\text{LTL}}(l_{i},t_{j})=\text{FFNN}^{\text{label}}_{\text{LTL}}(\mathbf{h}^{l}_{i})^{T}\mathbf{R}_{j-i}\text{FFNN}_{\text{LTL}}^{\text{text}}(\mathbf{h}_{j}^{t})\end{gathered}$
(3)

### Token-Label Linking for Pairing Conceptualizing

To conceptualize the association pair, USM links the subject of the
association pair to the label name using Token-Label Linking (TLL). Precisely,
TLL operation links the subject of triplet and the predicate type with head-
to-label (H2L) and tail-to-label (T2L) operations. For instance, TLL links the
head of text span “Monet” and the head of the label birth with H2L and links
the tail of text span “Monet” and the tail of the label place with T2L
following the head-tail span extraction style. For the above two token-label
linking (H2L, T2L) operations, the linking score
$\mathbf{s}_{\text{TLL}}(t_{i},l_{j})$ is computed as:

$\begin{gathered}\mathbf{s}_{\text{TLL}}(t_{i},l_{j})=\text{FFNN}^{\text{text}}_{\text{TLL}}(\mathbf{h}^{l}_{i})^{T}\mathbf{R}_{j-i}\text{FFNN}_{\text{TLL}}^{\text{label}}(\mathbf{h}_{j}^{t})\end{gathered}$
(4)

### Schema-constraint Decoding for Structure Composing

USM decodes the final structures using a schema-constraint decoding algorithm,
given substructures extracted by unified token linking operations. During the
decoding stage, we separate types for different tasks according to the schema
definition. For instance, in the joint entity and relation extraction task, we
uniformly encode entity types and relation types as labels to utilize the
common structuring and conceptualizing ability but compose the final result by
separating the entity or relation types from input types.

As shown in Figure 3, USM 1) first decodes mentions and subject-object unit
extracted by token-token linking operation: {“Monet”, “Paris”, “France”,
(“Monet”, “Pairs”), (“France”, “Pairs”)}; 2) and then decodes label-mention
pairs by label-token linking operation: {(person, “Monet”), (country,
“France”), (birth place, “Paris”), (capital, “Paris”)}; 3) and finally decodes
label-association pairs using token-label linking operation: (“Monet”, birth
place), (“France”, capital). The above three token linking operations do not
affect each other; hence the extraction operations are fully non-
autoregressive and highly parallel.

Finally, we separate the entity types country and person, relation types birth
place, and capital from input types according to the schema definition. Based
on the result from token-label linking (“Monet”, birth place), (“France”,
capital), we can consistently obtain the full structure (“Monet”, birth place,
“Paris”) and (“France”, capital, “Paris”).

## Learning from Heterogeneous Supervision

This section introduces how to leverage heterogeneous supervised resources to
learn the common structuring and conceptualizing abilities for unified token
linking. Specifically, with the help of verbalized label representation and
unified token linking, we unify heterogeneous supervision signals into
$<$text, token pairs$>$ for pre-training. We first pre-train the USM on the
heterogeneous resources, which contain three different supervised signals,
including task annotation signals (e.g., IE datasets), distant signals (e.g.,
distant supervision datasets), and indirect signals (e.g., question answering
datasets), then adopt the pre-trained USM model to specific downstream
information extraction tasks.

### Pre-training

USM uniformly encodes label schema and text in the shared semantic
representation and employs unified token linking to structure and
conceptualize information from text. To help USM to learn the common
structuring and conceptualizing abilities, we collect three different
supervised signals from existing linguistic sources for the pre-training of
USM:

$\mathcal{D}_{\text{task}}$ is the task annotation dataset, where each
instance has a gold annotation for information extraction. We use Ontonotes
(Pradhan et al. 2013), widely used in the field of information extraction as
gold annotation, which contains 18 entity types. $\mathcal{D}_{\text{task}}$
is used as in-task supervision signals to learn task-specific structuring and
conceptualizing abilities.

Dataset | Metric | UIE | Task-specific SOTA Methods | USM${}_{\text{Roberta}}$ | USM | USM${}_{\text{Unify}}$
---|---|---|---|---|---|---
ACE04 | Entity F1 | 86.89 | (Lou, Yang, and Tu 2022) | 87.90 | 87.79 | 87.62 | 87.34
ACE05-Ent | Entity F1 | 85.78 | (Lou, Yang, and Tu 2022) | 86.91 | 86.98 | 87.14 | -
CoNLL03 | Entity F1 | 92.99 | (Wang et al. 2021b) | 93.21 | 92.76 | 93.16 | 92.97
ACE05-Rel | Relation Strict F1 | 66.06 | (Yan et al. 2021) | 66.80 | 66.54 | 67.88 | -
CoNLL04 | Relation Strict F1 | 75.00 | (Huguet Cabot and Navigli 2021) | 75.40 | 75.86 | 78.84 | 77.12
NYT | Relation Boundary F1 | 93.54 | (Huguet Cabot and Navigli 2021) | 93.40 | 93.96 | 94.07 | 94.01
SciERC | Relation Strict F1 | 36.53 | (Yan et al. 2021) | 38.40 | 37.05 | 37.36 | 37.42
ACE05-Evt | Event Trigger F1 | 73.36 | (Wang et al. 2022b) | 73.60 | 71.68 | 72.41 | 72.31
ACE05-Evt | Event Argument F1 | 54.79 | (Wang et al. 2022b) | 55.10 | 55.37 | 55.83 | 53.57
CASIE | Event Trigger F1 | 69.33 | (Lu et al. 2021) | 68.98 | 70.77 | 71.73 | 71.56
CASIE | Event Argument F1 | 61.30 | (Lu et al. 2021) | 60.37 | 63.05 | 63.26 | 63.00
14-res | Sentiment Triplet F1 | 74.52 | (Lu et al. 2022) | 74.52 | 76.35 | 77.26 | 77.29
14-lap | Sentiment Triplet F1 | 63.88 | (Lu et al. 2022) | 63.88 | 65.46 | 65.51 | 66.60
15-res | Sentiment Triplet F1 | 67.15 | (Lu et al. 2022) | 67.15 | 68.80 | 69.86 | -
16-res | Sentiment Triplet F1 | 75.07 | (Lu et al. 2022) | 75.07 | 76.73 | 78.25 | -
AVE-unify | - | 71.10 | - | 71.34 | 71.83 | 72.46 | 72.11
AVE-total | - | 71.75 | - | 72.05 | 72.61 | 73.35 | -

Table 1:  Overall results of USM on different datasets. AVE-unify indicates
the average performance of non-overlapped datasets (except ACE05-Rel/Evt and
15/16-res), and AVE-total indicates the average performance of all datasets.

$\mathcal{D}_{\text{distant}}$ is the distant supervision dataset, where each
instance is aligned by text and knowledge base. Distant supervision is a
common practice to obtain large-scale training data for information extraction
(Mintz et al. 2009; Riedel et al. 2013). We employ NYT (Riedel et al. 2013)
and Rebel (Huguet Cabot and Navigli 2021) as our distant supervision datasets,
which are obtained by aligning text with Freebase and Wikidata, respectively.
Rebel dataset has a large label schema, and all verbalized schemas are too
long to be concatenated with input text and fed to the pre-trained transformer
encoder. We sample negative label schema to construct meta schema (Lu et al.
2022) as label schema for pre-training.

$\mathcal{D}_{\text{indirect}}$ is the indirect supervision dataset, where
each instance is derived from other related NLP tasks (Wang, Ning, and Roth
2020; Chen et al. 2022b). We utilize reading comprehension datasets from MRQA
(Fisch et al. 2019) as our indirect supervision datasets: HotpotQA (Yang et
al. 2018), Natural Questions (Kwiatkowski et al. 2019), NewsQA (Trischler et
al. 2017), SQuAD (Rajpurkar et al. 2016) and TriviaQA (Joshi et al. 2017).
Compared with limited entity types in $\mathcal{D}_{\text{task}}$ and relation
types $\mathcal{D}_{\text{distant}}$, diversified question expressions can
provide richer label semantic information for learning conceptualizing. For
each (question, context, answer) instance in $\mathcal{D}_{\text{indirect}}$,
we take the question as label schema, the context as input text, and the
answer as mention. It captures structuring and conceptualizing ability in the
pre-training stage by learning token-token and label-token linking operations.

### Learning function

For pre-training, fine-tuning and multi-task learning, we unify all datasets
as $\\{(x_{i},y_{i})\\}$, where $x_{i}$ is text and $y_{i}$ is linking
annotation of each token linking pair (TTM, LTM, TLM). We use the same
learning function for all settings with the homogenized data format.

The main challenge of USM learning is the sparsity of linked token pairs. The
linked ratio only occupies less than 1% of all valid token pair candidates. To
overcome the extreme sparsity of linking instances, we optimize class
imbalance loss (Su et al. 2022) for each instance as follows:

$\displaystyle\mathcal{L}=\sum_{m\in\mathcal{M}}$
$\displaystyle\log\left(1+\sum_{(i,j)\in m^{+}}e^{-s_{m}(i,j)}\right)$ (5)
$\displaystyle+\log\left(1+\sum_{(i,j)\in m^{-}}e^{s_{m}(i,j)}\right)$

where $\mathcal{M}$ denotes linking types of USM, $m^{+}$ indicates the linked
pairs, $m^{-}$ indicates the non-linked pairs, and $s_{m}(i,j)$ is the
predicate linking score for the linking operation $m$.

## Experiments

This section conducts massive experiments under supervised settings and
transfer settings to demonstrate the effectiveness of the proposed unified
semantic matching framework.

| Movie | Restaurant | Social | AI | Literature | Music | Politics | Science | Ave
---|---|---|---|---|---|---|---|---|---
| Performance on Unseen Label Subset of $\mathcal{D}_{t}$ and
$\mathcal{D}_{i}$
#Unseen/#All | 12/12 | 7/8 | 7/10 | 10/14 | 8/12 | 9/13 | 5/9 | 13/17 | -
$\mathcal{D}_{\text{task}}$ | 25.07 | 2.50 | 22.54 | 10.82 | 50.74 | 44.11 | 9.75 | 13.98 | 22.44
$\mathcal{D}_{\text{task}}$ \+ $\mathcal{D}_{\text{indirect}}$ | 37.73 | 14.73 | 29.34 | 28.18 | 56.00 | 44.93 | 36.10 | 44.09 | 36.39
| Performance on Unseen Label Subset of Pre-training Dataset
#Unseen/#All | 10/12 | 7/8 | 6/10 | 8/14 | 7/12 | 8/13 | 4/9 | 12/17 | -
$\mathcal{D}_{\text{task}}$ | 32.1 | 2.50 | 1.64 | 10.68 | 52.42 | 45.93 | 11.16 | 14.12 | 21.32
$\mathcal{D}_{\text{task}}$ \+ $\mathcal{D}_{\text{indirect}}$ | 39.76 | 14.73 | 20.62 | 24.12 | 56.24 | 44.21 | 32.92 | 44.25 | 34.61
$\mathcal{D}_{\text{task}}$ \+ $\mathcal{D}_{\text{distant}}$ | 35.35 | 21.10 | 40.64 | 27.57 | 56.97 | 49.29 | 43.72 | 44.05 | 39.84
$\mathcal{D}_{\text{task}}$ \+ $\mathcal{D}_{\text{distant}}$ \+ $\mathcal{D}_{\text{indirect}}$ | 42.11 | 26.01 | 44.37 | 34.91 | 65.69 | 60.07 | 56.65 | 55.26 | 48.13
$\mathbf{\Delta}$ | 10.01 | 23.51 | 42.73 | 24.23 | 13.27 | 14.14 | 45.49 | 41.14 | 26.82

Table 2:  Performance of Zero-shot transfer settings on unseen entity label
subset with different supervision signals. Unseen indicates label types that
do not appear in the pre-training dataset. $\mathbf{\Delta}$ indicates the
improvement of pre-training using extra supervision signals
($\mathcal{D}_{\text{distant}}$ and $\mathcal{D}_{\text{indirect}}$).

| CoNLL04 | Model Size
---|---|---
GPT-3 | 18.10 | 137B
DeepStruct | 25.80 | 10B
USM | 25.95 | 356M

Table 3:  Performance of Zero-shot transfer settings on relation extraction. *
GPT-3 175B indicates formulating the extraction task as a question answering
problem through prompting, and DeepStruct 10B is a pre-trained language model
for structure prediction (Wang et al. 2022a)

| Model | 1-Shot | 5-Shot | 10-Shot | AVE-S
---|---|---|---|---|---
Entity CoNLL03 | UIE-Large* | 57.53 | 75.32 | 79.12 | 70.66
USM${}_{\text{Roberta}}$ | 9.69 | 40.66 | 62.87 | 37.74
USM${}_{\text{Symbolic}}$ | 60.56 | 81.87 | 83.87 | 75.43
USM | 71.11 | 83.25 | 84.58 | 79.65
Relation CoNLL04 | UIE-Large* | 34.88 | 51.64 | 58.98 | 48.50
USM${}_{\text{Roberta}}$ | 0.00 | 12.81 | 31.02 | 14.61
USM${}_{\text{Symbolic}}$ | 13.45 | 48.31 | 58.91 | 40.22
USM | 36.17 | 53.20 | 60.99 | 50.12
Event Trigger ACE05-Evt | UIE-Large* | 42.37 | 53.07 | 54.35 | 49.93
USM${}_{\text{Roberta}}$ | 26.39 | 47.10 | 51.46 | 41.65
USM${}_{\text{Symbolic}}$ | 1.97 | 30.77 | 52.30 | 28.35
USM | 40.86 | 55.61 | 58.79 | 51.75
Event Argument ACE05-Evt | UIE-Large* | 14.56 | 31.20 | 35.19 | 26.98
USM${}_{\text{Roberta}}$ | 6.47 | 27.00 | 34.20 | 22.56
USM${}_{\text{Symbolic}}$ | 0.08 | 13.71 | 33.52 | 15.77
USM | 19.01 | 36.69 | 42.48 | 32.73
Sentiment 16res | UIE-Large* | 23.04 | 42.67 | 53.28 | 39.66
USM${}_{\text{Roberta}}$ | 2.68 | 35.71 | 48.56 | 28.98
USM${}_{\text{Symbolic}}$ | 20.08 | 41.25 | 50.90 | 37.41
USM | 30.81 | 52.06 | 58.29 | 47.05

Table 4: Few-shot results on end-to-end IE tasks. For a fair comparison, we
conduct text-structure pre-training from T5-v1.1-large using the same pre-
training corpus of USM, refer to UIE-Large*.

### Experiments on Supervised Settings

We conduct supervised experiments on extensive information extraction tasks,
including 4 tasks and 13 datasets (entity extraction, relation extraction,
event extraction, sentiment extraction) and their combinations (e.g., joint
entity-relation extraction). The used datasets includes ACE04 (Mitchell et al.
2005), ACE05 (Walker et al. 2006); CoNLL03 (Tjong Kim Sang and De Meulder
2003), CoNLL04 (Roth and Yih 2004), SciERC (Luan et al. 2018), NYT (Riedel,
Yao, and McCallum 2010), CASIE (Satyapanich, Ferraro, and Finin 2020),
SemEval-14/15/16 (Pontiki et al. 2014, 2015, 2016). We employ the same end-to-
end settings and evaluation metrics as Lu et al. (2022).

We compare the proposed USM framework with the task-specific state-of-the-art
methods and the unified structure generation method – UIE (Lu et al. 2022).
For our approach, we show three different settings:

  * •

USM is the pre-trained model which learned unified token linking ability from
heterogeneous supervision;

  * •

USM${}_{\text{Roberta}}$ is the initial model of the pre-trained USM, which
employs RoBERTa-Large (Liu et al. 2019) as the pre-trained transformer
encoder;

  * •

USM${}_{\text{Unify}}$ is initialized by the pre-trained USM and conducts
multi-task learning with all datasets but ignores overlapped datasets:
ACE05-Ent/Rel and 15/16-res.

For the USM${}_{\text{Roberta}}$ and USM settings, we fine-tune them on each
specific task separately. We run each experiment with three seeds and report
their average performance.

Table 1 shows the overall performance of USM and other baselines on the 13
datasets, where AVE-unify indicates the average performance of non-overlapped
datasets, and AVE-total indicates the average performance of all datasets. We
can observe that: 1) By verbalizing labels and modeling all IE tasks as
unified token linking, USM provides a novel and effective framework for IE.
USM achieves state-of-the-art performance and outperforms the strong task-
specific methods by 1.30 in AVE-total. Even without pre-training,
USM${}_{\text{Roberta}}$ also shows strong performance, which indicates the
strong portability and generalization ability of unified token linking. 2)
Heterogeneous supervision provides a better foundation for structuring and
conceptualizing information extraction. Compared to the initial model
USM${}_{\text{Roberta}}$ and the pre-trained model USM, the heterogeneous pre-
training achieved an average 0.74 improvement across all datasets. 3) By
homogenizing diversified label schemas and heterogeneous target structures
into the unified token sequence, USM${}_{\text{Unify}}$ can solve massive IE
tasks with a single multi-task model. USM${}_{\text{Unify}}$ outperforms task-
specific state-of-the-art methods with different model architectures and
encoder backbones in average, providing an efficient solution for application
and deployment.

### Experiments on Zero-shot Transfer Settings

We conduct zero-shot cross-type transfer experiments on 9 datasets across
various domains to verify the transferable conceptualization learned by USM.
In this setting, we directly employ the pre-trained USM to conduct extraction
on new datasets.

For entity extraction, the cross-type extraction datasets include Movie (MIT-
Movie), Restaurant (MIT-Restaurant) (Liu et al. 2013), Social (WNUT-16)
(Strauss et al. 2016), and AI/Literature/Music/Politics/Science from CrossNER
(Liu et al. 2021). We investigate the effect of different supervised signals
in the zero-shot entity extraction setting. $\mathcal{D}_{task}$ indicates we
first train USM on the common entity extraction dataset – Ontonotes, then
directly conduct extraction on the new types, which emulates the most common
label transfer method used in real-world scenarios. To be consistent with the
real scenario, we select the best checkpoint according to the F1 score on the
dev set of $\mathcal{D}_{task}$.

For zero-shot relation extraction, we compare USM with the following strong
baselines:

  * •

GPT-3 175B (Brown et al. 2020) is a large-scale, generative pre-trained model,
which can extract entity and relation by formulating the task as a question
answering problem through prompting (Wang et al. 2022a).

  * •

DeepStruct 10B is a structured prediction model pre-trained on six large-scale
entity, relation, and triple datasets (Wang et al. 2022a).

Table 2 shows the entity extraction performance on the unseen label subset, in
which types are not appearing in the pre-training dataset. And Table 3 shows
the performance of zero-shot relation extraction on CoNLL04. From Table 2 and
Table 3, we can see that: 1) USM has a strong zero-shot transferability across
labels. USM shows good migration performance on Movie, Literature, and Music
domains even when learning from $\mathcal{D}_{\text{task}}$ with limited
entity types. For relation extraction, USM (356M) outperforms the strong zero-
shot baseline GPT-3 (175B) and DeepStructure (10B) with a smaller model size.
2) Heterogeneous supervision boosts USM with unified label semantics and
outperforms the task annotation baseline by a large margin. Compared to the
task annotation baseline ($\mathcal{D}_{\text{task}}$), USM significantly and
consistently improves the performance on all datasets.

### Experiments on Few-shot Transfer Settings

To further investigate the effects of verbalized label semantics, we conduct
few-shot transfer experiments on four IE tasks and compare USM with the
following baselines:

  * •

UIE-large* is the pre-trained sequence-to-structure model for effective low-
resource IE tasks, which injects label semantics by generating labels and
words in structured extraction language synchronously and guiding the
generation with a structural schema instructor.

  * •

USM${}_{\text{Roberta}}$ is the initial model of USM, which directly use
Roberta-large as the pre-trained encoder;

  * •

USM${}_{\text{Symbolic}}$ replaces the names of labels with symbolic
representation (meaning-less labels, e.g., label1, label2, …) during the fine-
tuning stage of USM, which is used to verify the effect of verbalized label
semantics.

For few-shot transfer experiments, we follow the data splits and settings with
the previous work (Lu et al. 2022) and repeat each experiment 10 times to
avoid the influence of random sampling (Huang et al. 2021). Table 4 shows the
performance on 4 IE tasks under the few-shot settings, where AVE-S is the
average performance of 1/5/10-shot experiments. We can see that: 1) By
modeling IE tasks via unified semantic matching, USM exceeds the few-shot
state-of-the-art UIE-large 5.11 on average. Although UIE also adopts
verbalized label representation, this structure generation method needs to
learn to generate the novel schema word in the target structure during
transfer learning. In contrast, USM only needs to learn to match them,
providing a better inductive bias and leading to a much smaller decoding
search space. The pre-trained unified token linking ability boosts the USM in
all settings. 2) It is crucial to verbalize label schemas rather than
meaningless symbols, especially for complex extraction tasks.
USM${}_{\text{Symbolic}}$, which uses symbolic labels instead of verbalized
labels, drastically reduces performance on all tasks. For tasks with more
semantic types, such as event extraction with 33 types, the performance drops
significantly, even lower than that of USM${}_{\text{Roberta}}$ initialized
directly with Roberta-large.

## Related Work

In the past decade, due to powerful representation ability, deep learning
methods (Bengio et al. 2003; Collobert et al. 2011) have made amazing
achievements in information extraction tasks. Most of these methods decompose
extraction into multiple sub-tasks and follow the classical neural classifier
method (Krizhevsky, Sutskever, and Hinton 2012) to model each sub-task, such
as entity extraction, relation classification, event trigger detection, event
argument classification, etc. And several architectures are proposed to model
the extraction, such as sequence tagging (Lample et al. 2016; Zheng et al.
2017), span classification (Sohrab and Miwa 2018; Song et al. 2019; Wadden et
al. 2019), table filling (Gupta, Schütze, and Andrassy 2016; Wang and Lu
2020), question answering (Levy et al. 2017; Li et al. 2020), and token pair
(Wang et al. 2020; Yu et al. 2021).

Recently, to solve various IE tasks with a single architecture, UIE employs
unified structure generation, models the various IE tasks with structured
extraction language, and pre-trains the ability of structure generation using
distant text-structure supervision (Lu et al. 2022). Unlike the generation-
based approach, we model universal information extraction as unified token
linking, which reduces the search space during decoding and leads to better
generalization performance. Beyond distant supervision, we further introduce
indirect supervision from related NLP tasks to learn the unified token linking
ability.

Similar to USM in this paper, matching-based IE approaches aim to verbalize
the label schema and structure candidate to achieve better generalization (Liu
et al. 2022). Such methods usually use pre-extracted syntactic structures
(Wang et al. 2021a) and semantic structures (Huang et al. 2018) as candidate
structures, then model the extraction as text entailment (Obamuyide and
Vlachos 2018; Sainz et al. 2021; Lyu et al. 2021; Sainz et al. 2022) and
semantic structure mapping (Chen and Li 2021; Dong, Pan, and Luo 2021).
Different from the pre-extraction and matching style, this paper decouples
various IE tasks to unified token linking operations and designs a one-pass
end-to-end information extraction framework for modeling all tasks.

## Conclusion

In this paper, we propose a unified semantic matching framework – USM, which
jointly encodes extraction schema and input text, uniformly extracts
substructures in parallel, and controllably decodes target structures on
demand. Experimental results show that USM achieves state-of-the-art
performance under the supervised experiments and shows strong generalization
ability under zero/few-shot transfer settings, which verifies USM is a novel,
transferable, controllable, and efficient framework. For future work, we want
to extend USM to NLU tasks, e.g., text classification, and investigate more
indirect supervision signals for IE, e.g., text entailment.

## Acknowledgments

We sincerely thank the reviewers for their insightful comments and valuable
suggestions. This work is supported by the National Key Research and
Development Program of China (No.2020AAA0109400) and the Natural Science
Foundation of China (No.62122077, 61876223, and 62106251). Hongyu Lin is
sponsored by CCF-Baidu Open Fund.

##  Appendix A Appendix: Experiment Details

This section describes the details of the experiments, including
implementation details and extra experiments analysis.

### Implementation Details

For all experiments, we optimize our model using AdamW (Loshchilov and Hutter
2019) with the constant learning rate. For single-task fine-tuning, we tune
the learning rate from {1e-5, 2e-5, 3e-5} with three seeds and select the best
hyper-parameter setting according to the performance of the development set.
For multi-task learning of USM${}_{\text{Unify}}$, we select the best
checkpoint according to the average performance of all datasets. We conducted
each experiment on NVIDIA A100 GPUs, and detailed hyper-parameters are shown
in Table 5.

| Learning Rate | Global Batch | Epoch
---|---|---|---
Pre-training | 2e-5 | 96 | 5
Fine-tuning |  |  |
Entity | 1e-5, 2e-5, 3e-5 | 64 | 100
Relation | 1e-5, 2e-5, 3e-5 | 64 | 200
Event | 1e-5, 2e-5, 3e-5 | 96 | 200
Sentiment | 1e-5, 2e-5, 3e-5 | 32 | 100
Low-resource | 2e-5 | 32 | 200
Multi-task | 2e-5 | 96 | 200

Table 5: Hyper-parameters of USM experiments.

### Pre-train Datasets

We collect three types of supervision signals for model pre-training: named
entity annotation in Ontonotes for task annotation
$\mathcal{D}_{\text{task}}$; NYT (Riedel, Yao, and McCallum 2010) and Rebel
(Huguet Cabot and Navigli 2021) for distant supervision
$\mathcal{D}_{\text{distant}}$; machine reading comprehension from MRQA (Fisch
et al. 2019) for indirect supervision $\mathcal{D}_{\text{indirect}}$. For the
Rebel data, we only keep the 230 most frequently occurring relation types and
randomly sample 300K instances for pre-training. For the reading comprehension
data, we reserve a maximum of 5 questions for each instance and filter out
instances where the total tokenized length of question and context exceeds
500. The final statistics are shown in Table 6.

| Dataset | #instance
---|---|---
$\mathcal{D}_{\text{task}}$ | Ontonote | 60K
$\mathcal{D}_{\text{distant}}$ | NYT + Rebel | 356K
$\mathcal{D}_{\text{indirect}}$ | MRQA | 195K
Table 6: Detailed statistics of pre-training datasets.

### Ablation Analysis of Label-Text Interaction

To investigate the effect of label-text interaction and accelerate the
extraction process, we propose an approximate shallow label-text interaction
model to reuse the computation of label embedding during the inference stage.
Motivated by Dong et al. (2019), we design attention mask strategies to
control the interaction between label and text, as illustrated in Figure 4. In
the full mask setting (Label $\Leftrightarrow$ Text, Figure 4(a)), label and
text can attend to each other to obtain deep interaction; in the partial mask
setting (Label $\times$ Text, Figure 4(b)), label and text only attend to
themselves. For the partial mask setting, USM can cache and reuse the
calculation of label embedding to reduce the computation cost in a dual
encoder way during the inference stage.

((a))

((b))

Figure 4: Different attention masks for text-schema joint embedding.

| Entity | Relation | Event | Sentiment
---|---|---|---|---
Full-shot
Label $\Leftrightarrow$ Text | 97.03 | 81.91 | 63.51 | 81.22
Label $\times$ Text | 96.99 | 81.18 | 62.03 | 80.92
Few-shot (AVE-S)
Label $\Leftrightarrow$ Text | 82.12 | 52.23 | 37.52 | 51.51
Label $\times$ Text | 82.37 | 45.75 | 24.70 | 26.65

Table 7: Experiment results on the development set of entity (CoNLL03),
relation (CoNLL04), event (ACE05-Evt argument) and sentiment (16res) of USM
with different label-text interaction.

Table 7 shows the performance of two different label-text interactions, and we
can see that: 1) Deep interaction ($\Leftrightarrow$) can effectively improve
the ability of unified token linking, especially in low-resource settings. 2)
In resource-rich scenarios, shallow interaction ($\times$) can replace deep
interaction between label-text linking. This dynamic and variable scalability
enables USM to have better application scenarios in practice: for common rich
resource extraction tasks, USM can pre-compute the representation of label and
text separately in a dual encoder fashion, speeding up the inference process
without the need for other deployments; for low-resource extraction tasks, USM
can use deep-level interactive information to improve transfer ability and
retain high parallelism.

### Effects of Controllable Ability

To investigate the controllable ability of USM, we conduct partial extraction
experiments on the CoNLL04 (Joint Entity and Relation Extraction), ACE05-Evt
(Event Trigger and Argument), and 14lap (Sentiment Extraction). We employ two
kinds of partial extraction settings: 1) partial task extraction: we train an
end-to-end joint entity and relation extraction model using the full schema of
CoNLL04 (entity and relation) but feed the partial schema (entity) to USM. 2)
partial label extraction: we train an extraction model on the full label set
(positive, neutral, negative of sentiment), and only extract part of the label
set (positive) from the text. Table 8 shows the performance of three different
partial extraction experiments. We can see that USM achieves almost the same
performance in both settings and has highly controllable extraction ability.

| Full | Partial | Partial Details
---|---|---|---
CoNLL04 Entity | 90.74 | 90.50 | Only Entity
ACE05-Evt Trigger | 70.40 | 70.99 | Only 16 Types of 33 Types
ACE05-Evt Argument | 60.87 | 60.24 | Only 16 Types of 33 Types
14lap Sentiment | 75.00 | 74.78 | Only Positive of 3 Types

Table 8:  Experiment results of partial extraction schema on the development
set of different datasets. Partial indicates feeding part of the whole schema
to USM, such as only extracting positive sentiment rather than extracting all
types (positive, neutral, negative) from the text. All results are evaluated
on the partial extraction schema. For instance, the performances of ACE05-Evt
Trigger under the full and partial settings result from 16 types in the
partial extraction schema.

| 14res | 14lap | 15res | 16res
---|---|---|---|---
System using BERT-base |  |  |  |
(Xu et al. 2020) | 62.40 | 51.04 | 57.53 | 63.83
(Xu, Chia, and Bing 2021) | 71.85 | 59.38 | 63.27 | 70.26
(Yu Bai Jian et al. 2021) | 69.61 | 59.50 | 62.72 | 68.41
(Chen et al. 2022a) | 71.78 | 58.81 | 61.93 | 68.33
USM${}_{\text{BERT-base}}$ | 71.87 | 58.63 | 63.41 | 72.68

Table 9: Experiment results of USM${}_{\text{BERT-base}}$ on aspect based sentiment triplet extraction tasks. | P | R | F
---|---|---|---
System using BERT-base |  |  |
(Wang et al. 2020) | 91.4 | 92.6 | 92.0
(Sui et al. 2020) | 92.5 | 92.2 | 92.3
(Zheng et al. 2021) | 93.5 | 91.9 | 92.7
USM${}_{\text{BERT-base}}$ | 93.7 | 91.9 | 92.8
Table 10:  Experiment results of USM${}_{\text{BERT-base}}$ on the NYT.

### Comparison of BERT-base

This section compares USM with other BERT-base based state-of-the-art systems.
USM${}_{\text{BERT-base}}$ indicates USM uses BERT-base (Devlin et al. 2019)
as a pre-trained transformer encoder. Table 9 shows the performance of USM and
the state-of-the-art systems on the four aspect-based sentiment analysis
datasets, and Table 10 shows the performance of USM and the state-of-the-art
joint entity relation extraction systems on the NYT dataset. We can see that
USM${}_{\text{BERT-base}}$ achieves competitive performance on above datasets,
which verifies the effectiveness of the proposed unified semantic matching
framework.

### Effect of Token-Label Linking

This section investigates the effect of the token-label linking operation.
Table 11 shows results of different decoding strategies with golden token
links: 1) Full employs all three types of token linking operations to decode
the final structures; 2) w/o TLL indicates decoding without the token-label
links for pairing conceptualizing.

Dataset | Metric | F1 with golden links
---|---|---
|  |  w/o TLL | Full
ACE05-Rel | Relation Strict F1 | 98.54 | 99.96
CoNLL04 | Relation Strict F1 | 100.00 | 100.00
NYT | Relation Boundary F1 | 72.74 | 100.00
SciERC | Relation Strict F1 | 92.06 | 99.74
ACE05-Evt | Event Argument F1 | 98.75 | 100.00
CASIE | Event Argument F1 | 99.98 | 99.99
14-res | Sentiment Triplet F1 | 99.10 | 100.00
14-lap | Sentiment Triplet F1 | 98.54 | 100.00

Table 11:  Performance of different decoding strategies using golden links.

## References

  * Alvarez-Melis and Jaakkola (2017) Alvarez-Melis, D.; and Jaakkola, T. 2017.  A causal framework for explaining the predictions of black-box sequence-to-sequence models.  In _Proc. of EMNLP_.
  * Andersen et al. (1992) Andersen, P. M.; Hayes, P. J.; Weinstein, S. P.; Huettner, A. K.; Schmandt, L. M.; and Nirenburg, I. B. 1992.  Automatic Extraction of Facts from Press Releases to Generate News Stories.  In _Proc. of ANLP_.
  * Bengio et al. (2003) Bengio, Y.; Ducharme, R.; Vincent, P.; and Janvin, C. 2003.  A Neural Probabilistic Language Model.  _J. Mach. Learn. Res._
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020.  Language Models are Few-Shot Learners.  In _Proc. of NeurIPS_.
  * Chen and Li (2021) Chen, C.-Y.; and Li, C.-T. 2021.  ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning.  In _Proc. of NAACL_.
  * Chen et al. (2022a) Chen, H.; Zhai, Z.; Feng, F.; Li, R.; and Wang, X. 2022a.  Enhanced Multi-Channel Graph Convolutional Network for Aspect Sentiment Triplet Extraction.  In _Proc. of ACL_.
  * Chen et al. (2022b) Chen, M.; Huang, L.; Li, M.; Zhou, B.; Ji, H.; and Roth, D. 2022b.  New Frontiers of Information Extraction.  In _Proc. of NAACL_.
  * Collobert et al. (2011) Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. 2011.  Natural Language Processing (Almost) from Scratch.  _J. Mach. Learn. Res._
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proc. of NAACL_.
  * Dong et al. (2019) Dong, L.; Yang, N.; Wang, W.; Wei, F.; Liu, X.; Wang, Y.; Gao, J.; Zhou, M.; and Hon, H.-W. 2019.  Unified Language Model Pre-training for Natural Language Understanding and Generation.  In _Proc. of NeurIPS_.
  * Dong, Pan, and Luo (2021) Dong, M.; Pan, C.; and Luo, Z. 2021.  MapRE: An Effective Semantic Mapping Approach for Low-resource Relation Extraction.  In _Proc. of EMNLP_.
  * Fisch et al. (2019) Fisch, A.; Talmor, A.; Jia, R.; Seo, M.; Choi, E.; and Chen, D. 2019.  MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension.  In _Proc. of MRQA_.
  * Grishman (2019) Grishman, R. 2019.  Twenty-five years of information extraction.  _Natural Language Engineering_.
  * Gupta, Schütze, and Andrassy (2016) Gupta, P.; Schütze, H.; and Andrassy, B. 2016.  Table Filling Multi-Task Recurrent Neural Network for Joint Entity and Relation Extraction.  In _Proc. of COLING_.
  * Huang et al. (2021) Huang, J.; Li, C.; Subudhi, K.; Jose, D.; Balakrishnan, S.; Chen, W.; Peng, B.; Gao, J.; and Han, J. 2021.  Few-Shot Named Entity Recognition: An Empirical Baseline Study.  In _Proc. of EMNLP_.
  * Huang et al. (2018) Huang, L.; Ji, H.; Cho, K.; Dagan, I.; Riedel, S.; and Voss, C. 2018.  Zero-Shot Transfer Learning for Event Extraction.  In _Proc. of ACL_.
  * Huguet Cabot and Navigli (2021) Huguet Cabot, P.-L.; and Navigli, R. 2021.  REBEL: Relation Extraction By End-to-end Language generation.  In _Proc. of EMNLP Findings_.
  * Joshi et al. (2017) Joshi, M.; Choi, E.; Weld, D.; and Zettlemoyer, L. 2017.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.  In _Proc. of ACL_.
  * Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.  ImageNet Classification with Deep Convolutional Neural Networks.  In _Proc. of NeurIPS_.
  * Kwiatkowski et al. (2019) Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; Toutanova, K.; Jones, L.; Kelcey, M.; Chang, M.-W.; Dai, A. M.; Uszkoreit, J.; Le, Q.; and Petrov, S. 2019.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the Association for Computational Linguistics_.
  * Lample et al. (2016) Lample, G.; Ballesteros, M.; Subramanian, S.; Kawakami, K.; and Dyer, C. 2016.  Neural Architectures for Named Entity Recognition.  In _Proc. of NAACL_.
  * Levy et al. (2017) Levy, O.; Seo, M.; Choi, E.; and Zettlemoyer, L. 2017.  Zero-Shot Relation Extraction via Reading Comprehension.  In _Proc. of CoNLL_.
  * Li et al. (2020) Li, X.; Feng, J.; Meng, Y.; Han, Q.; Wu, F.; and Li, J. 2020.  A Unified MRC Framework for Named Entity Recognition.  In _Proc. of ACL_.
  * Liu et al. (2022) Liu, F.; Lin, H.; Han, X.; Cao, B.; and Sun, L. 2022.  Pre-training to Match for Unified Low-shot Relation Extraction.  In _Proc. of ACL_.
  * Liu et al. (2013) Liu, J.; Pasupat, P.; Cyphers, S.; and Glass, J. 2013.  Asgard: A portable architecture for multilingual dialogue systems.  In _Proc. of ICASSP_.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_.
  * Liu et al. (2021) Liu, Z.; Xu, Y.; Yu, T.; Dai, W.; Ji, Z.; Cahyawijaya, S.; Madotto, A.; and Fung, P. 2021.  CrossNER: Evaluating Cross-Domain Named Entity Recognition.  _Proc. of AAAI_.
  * Loshchilov and Hutter (2019) Loshchilov, I.; and Hutter, F. 2019.  Decoupled Weight Decay Regularization.  In _Proc. of ICLR_.
  * Lou, Yang, and Tu (2022) Lou, C.; Yang, S.; and Tu, K. 2022.  Nested Named Entity Recognition as Latent Lexicalized Constituency Parsing.  In _Proc. of ACL_.
  * Lu et al. (2021) Lu, Y.; Lin, H.; Xu, J.; Han, X.; Tang, J.; Li, A.; Sun, L.; Liao, M.; and Chen, S. 2021.  Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction.  In _Proc. of ACL_.
  * Lu et al. (2022) Lu, Y.; Liu, Q.; Dai, D.; Xiao, X.; Lin, H.; Han, X.; Sun, L.; and Wu, H. 2022.  Unified Structure Generation for Universal Information Extraction.  In _Proc. of ACL_.
  * Luan et al. (2018) Luan, Y.; He, L.; Ostendorf, M.; and Hajishirzi, H. 2018.  Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction.  In _Proc. of EMNLP_.
  * Lyu et al. (2021) Lyu, Q.; Zhang, H.; Sulem, E.; and Roth, D. 2021.  Zero-shot Event Extraction via Transfer Learning: Challenges and Insights.  In _Proc. of ACL_.
  * Mintz et al. (2009) Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009.  Distant supervision for relation extraction without labeled data.  In _Proc. of ACL_.
  * Mitchell et al. (2005) Mitchell, A.; Strassel, S.; Huang, S.; and Zakhary, R. 2005.  ACE 2004 Multilingual Training Corpus.
  * Obamuyide and Vlachos (2018) Obamuyide, A.; and Vlachos, A. 2018.  Zero-shot Relation Classification as Textual Entailment.  In _Proc. of FEVER_.
  * Pontiki et al. (2016) Pontiki, M.; Galanis, D.; Papageorgiou, H.; Androutsopoulos, I.; Manandhar, S.; AL-Smadi, M.; Al-Ayyoub, M.; Zhao, Y.; Qin, B.; De Clercq, O.; Hoste, V.; Apidianaki, M.; Tannier, X.; Loukachevitch, N.; Kotelnikov, E.; Bel, N.; Jiménez-Zafra, S. M.; and Eryiğit, G. 2016.  SemEval-2016 Task 5: Aspect Based Sentiment Analysis.  In _Proc. of SemEval_.
  * Pontiki et al. (2015) Pontiki, M.; Galanis, D.; Papageorgiou, H.; Manandhar, S.; and Androutsopoulos, I. 2015.  SemEval-2015 Task 12: Aspect Based Sentiment Analysis.  In _Proc. of SemEval_.
  * Pontiki et al. (2014) Pontiki, M.; Galanis, D.; Pavlopoulos, J.; Papageorgiou, H.; Androutsopoulos, I.; and Manandhar, S. 2014.  SemEval-2014 Task 4: Aspect Based Sentiment Analysis.  In _Proc. of SemEval_.
  * Pradhan et al. (2013) Pradhan, S.; Moschitti, A.; Xue, N.; Ng, H. T.; Björkelund, A.; Uryupina, O.; Zhang, Y.; and Zhong, Z. 2013.  Towards Robust Linguistic Analysis using OntoNotes.  In _Proc. of CoNLL_.
  * Rajpurkar et al. (2016) Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _Proc. of EMNLP_.
  * Riedel, Yao, and McCallum (2010) Riedel, S.; Yao, L.; and McCallum, A. 2010.  Modeling Relations and Their Mentions without Labeled Text.  In _Machine Learning and Knowledge Discovery in Databases_.
  * Riedel et al. (2013) Riedel, S.; Yao, L.; McCallum, A.; and Marlin, B. M. 2013.  Relation Extraction with Matrix Factorization and Universal Schemas.  In _Proc. of NAACL_.
  * Roth and Yih (2004) Roth, D.; and Yih, W.-t. 2004.  A Linear Programming Formulation for Global Inference in Natural Language Tasks.  In _Proc. of CoNLL_.
  * Sainz et al. (2022) Sainz, O.; Gonzalez-Dios, I.; Lopez de Lacalle, O.; Min, B.; and Agirre, E. 2022\.  Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning.  In _Proc. of ACL Findings_.
  * Sainz et al. (2021) Sainz, O.; Lopez de Lacalle, O.; Labaka, G.; Barrena, A.; and Agirre, E. 2021.  Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction.  In _Proc. of EMNLP_.
  * Satyapanich, Ferraro, and Finin (2020) Satyapanich, T.; Ferraro, F.; and Finin, T. 2020.  CASIE: Extracting Cybersecurity Event Information from Text.  In _Proc. of AAAI_.
  * Sohrab and Miwa (2018) Sohrab, M. G.; and Miwa, M. 2018.  Deep Exhaustive Model for Nested Named Entity Recognition.  In _Proc. of EMNLP_.
  * Song et al. (2019) Song, L.; Zhang, Y.; Gildea, D.; Yu, M.; Wang, Z.; and Su, J. 2019.  Leveraging Dependency Forest for Neural Medical Relation Extraction.  In _Proc. of EMNLP-IJCNLP_.
  * Strauss et al. (2016) Strauss, B.; Toma, B.; Ritter, A.; de Marneffe, M.-C.; and Xu, W. 2016.  Results of the WNUT16 Named Entity Recognition Shared Task.  In _Proc. of WNUT_.
  * Su et al. (2021) Su, J.; Lu, Y.; Pan, S.; Murta, A.; Wen, B.; and Liu, Y. 2021.  RoFormer: Enhanced Transformer with Rotary Position Embedding.
  * Su et al. (2022) Su, J.; Murtadha, A.; Pan, S.; Hou, J.; Sun, J.; Huang, W.; Wen, B.; and Liu, Y. 2022.  Global Pointer: Novel Efficient Span-based Approach for Named Entity Recognition.
  * Sui et al. (2020) Sui, D.; Chen, Y.; Liu, K.; Zhao, J.; Zeng, X.; and Liu, S. 2020.  Joint Entity and Relation Extraction with Set Prediction Networks.  _CoRR_.
  * Tjong Kim Sang and De Meulder (2003) Tjong Kim Sang, E. F.; and De Meulder, F. 2003.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition.
  * Trischler et al. (2017) Trischler, A.; Wang, T.; Yuan, X.; Harris, J.; Sordoni, A.; Bachman, P.; and Suleman, K. 2017.  NewsQA: A Machine Comprehension Dataset.  In _Proc. of RepL4NLP_.
  * Wadden et al. (2019) Wadden, D.; Wennberg, U.; Luan, Y.; and Hajishirzi, H. 2019.  Entity, Relation, and Event Extraction with Contextualized Span Representations.  In _Proc. of EMNLP_.
  * Walker et al. (2006) Walker, C.; Strassel, S.; Medero, J.; and Maeda, K. 2006.  ACE 2005 Multilingual Training Corpus.
  * Wang et al. (2021a) Wang, C.; Liu, X.; Chen, Z.; Hong, H.; Tang, J.; and Song, D. 2021a.  Zero-Shot Information Extraction as a Unified Text-to-Triple Translation.  In _Proc. of EMNLP_.
  * Wang et al. (2022a) Wang, C.; Liu, X.; Chen, Z.; Hong, H.; Tang, J.; and Song, D. 2022a.  DeepStruct: Pretraining of Language Models for Structure Prediction.  In _Proc. of ACL Findings_.
  * Wang and Lu (2020) Wang, J.; and Lu, W. 2020.  Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders.  In _Proc. of EMNLP_.
  * Wang, Ning, and Roth (2020) Wang, K.; Ning, Q.; and Roth, D. 2020.  Learnability with Indirect Supervision Signals.  In _Proc. of NeurIPS_.
  * Wang et al. (2022b) Wang, S.; Yu, M.; Chang, S.; Sun, L.; and Huang, L. 2022b.  Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding.  In _Proc. of ACL Findings_.
  * Wang et al. (2021b) Wang, X.; Jiang, Y.; Bach, N.; Wang, T.; Huang, Z.; Huang, F.; and Tu, K. 2021b.  Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning.  In _Proc. of ACL_.
  * Wang et al. (2020) Wang, Y.; Yu, B.; Zhang, Y.; Liu, T.; Zhu, H.; and Sun, L. 2020.  TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking.  In _Proc. of COLING_.
  * Xu, Chia, and Bing (2021) Xu, L.; Chia, Y. K.; and Bing, L. 2021.  Learning Span-Level Interactions for Aspect Sentiment Triplet Extraction.  In _Proc. of ACL_.
  * Xu et al. (2020) Xu, L.; Li, H.; Lu, W.; and Bing, L. 2020.  Position-Aware Tagging for Aspect Sentiment Triplet Extraction.  In _Proc. of EMNLP_.
  * Yan et al. (2021) Yan, Z.; Zhang, C.; Fu, J.; Zhang, Q.; and Wei, Z. 2021.  A Partition Filter Network for Joint Entity and Relation Extraction.  In _Proc. of EMNLP_.
  * Yang et al. (2018) Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W.; Salakhutdinov, R.; and Manning, C. D. 2018.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.  In _Proc. of EMNLP_.
  * Yu et al. (2021) Yu, B.; Wang, Y.; Liu, T.; Zhu, H.; Sun, L.; and Wang, B. 2021.  Maximal Clique Based Non-Autoregressive Open Information Extraction.  In _Proc. of EMNLP_.
  * Yu Bai Jian et al. (2021) Yu Bai Jian, S.; Nayak, T.; Majumder, N.; and Poria, S. 2021.  Aspect Sentiment Triplet Extraction Using Reinforcement Learning.  In _Proc. of CIKM_.
  * Zheng et al. (2021) Zheng, H.; Wen, R.; Chen, X.; Yang, Y.; Zhang, Y.; Zhang, Z.; Zhang, N.; Qin, B.; Ming, X.; and Zheng, Y. 2021.  PRGC: Potential Relation and Global Correspondence Based Joint Relational Triple Extraction.  In _Proc. of ACL_.
  * Zheng et al. (2017) Zheng, S.; Wang, F.; Bao, H.; Hao, Y.; Zhou, P.; and Xu, B. 2017.  Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme.  In _Proc. of ACL_.