# When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE
Systems for Downstream Applications

Kevin Pei1, Ishan Jindal2, Kevin Chen-Chuan Chang1, Chengxiang Zhai1, Yunyao
Li3
1 Grainger College of Engineering, University of Illinois at Urbana-Champaign
2 IBM Research 3 Apple Knowledge Platform
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Open Information Extraction (OpenIE) has been used in the pipelines of various
NLP tasks. Unfortunately, there is no clear consensus on which models to use
in which tasks. Muddying things further is the lack of comparisons that take
differing training sets into account. In this paper, we present an
application-focused empirical survey of neural OpenIE models, training sets,
and benchmarks in an effort to help users choose the most suitable OpenIE
systems for their applications. We find that the different assumptions made by
different models and datasets have a statistically significant effect on
performance, making it important to choose the most appropriate model for
one’s applications. We demonstrate the applicability of our recommendations on
a downstream Complex QA application.

##  1 Introduction

Open Information Extraction (OpenIE) is the task of extracting relation tuples
from plain text (Angeli, Premkumar, and Manning 2015). In its simplest form,
OpenIE extracts information in the form of tuples consisting of subject _(S)_
, predicate _(P)_ , object _(O)_ , and any additional arguments _(A)_. OpenIE
is open domain, intended to be easy to deploy in different domains without
fine-tuning. The tuples extracted by OpenIE consist of tokens from the
original text, allowing for the extraction of all relations regardless of
type. Models largely fall into two categories: models with hand-crafted
extraction patterns and models with automatically learned extraction patterns
(Niklaus et al. 2018). The increasing availability of semi-automatically
generated training datasets using previous OpenIE methods (Cui, Wei, and Zhou
2018), as well as significant advances in deep learning modeling techniques
such as LSTM have led to the development of state-of-the-art neural models
that automatically learn extraction patterns (Cui, Wei, and Zhou 2018; Garg
and Kalai 2018).

Since its introduction in Etzioni et al. (2008), OpenIE has attracted a large
amount of attention by the research community for use as a tool for a wide
range of downstream NLP tasks such as Slot Filling (Soderland et al. 2013;
Angeli, Premkumar, and Manning 2015), Question Answering (QA) (Fader,
Zettlemoyer, and Etzioni 2013; Khot, Sabharwal, and Clark 2017), Summarization
(Cao et al. 2018; Ponza, Del Corro, and Weikum 2018), and Event Schema
Induction (Balasubramanian et al. 2013; Romadhony, Widyantoro, and Purwarianti
2019). However, there is no real consensus on which OpenIE model is best for
which application. We can observe this lack of consensus in summarization,
where different papers use OLLIE (Christensen et al. 2014), MinIE (Ponza, Del
Corro, and Weikum 2018), and Stanford CoreNLP (Cao et al. 2018; Zhang et al.
2021) as their OpenIE models. Different applications may also have different
best OpenIE models. As an example, choosing a model that assumes all relations
only have a subject and object may not be suitable for event schema induction
since that excludes any event schemas with more than two entities. The papers
that introduce new OpenIE models and datasets do not specify how downstream
applications would be impacted by the different assumptions those papers make
about which relations to extract.

We find that prior OpenIE surveys are also insufficient to find the best
OpenIE model for a given application. The only previous application-focused
OpenIE survey we found was Mausam (2016). However, this survey does not
identify the desired properties of OpenIE for those applications or provide an
empirical comparison of OpenIE systems. Niklaus et al. (2018) provide a
taxonomy of OpenIE models, but does not include any comparison of datasets or
an empirical study. Glauber and Claro (2018) and Claro et al. (2019) also do
not provide an empirical application-focused survey.

Another problem is the lack of apples-to-apples comparisons between OpenIE
models. Comparisons should keep the training set, benchmark, and evaluation
metric constant when comparing models to eliminate confounders. Unfortunately,
the papers that introduce new OpenIE models and datasets often do not provide
this apples-to-apples comparison. For example, CopyAttention (Cui, Wei, and
Zhou 2018), SpanOIE (Zhan and Zhao 2020), IMoJIE (Kolluru et al. 2020b), and
OpenIE6 (Kolluru et al. 2020a) all compare their model to models trained on
different training sets. OpenIE6 reports performance on WiRE57 which Multi2OIE
(Ro, Lee, and Kang 2020) does not, but Multi2OIE reports performance on
ReOIE2016 which OpenIE6 does not. Because the training set can greatly affect
the performance of a neural model, we focus on selecting both the appropriate
OpenIE model and training set, which we refer to as an OpenIE System.

To resolve our lack of understanding, we focus on the particular question: How
do I choose a particular OpenIE system for a given application? Different
implicit assumptions about OpenIE may have a significant impact on the
performance of downstream applications such as the assumptions that all
relations are verb-based (Zhan and Zhao 2020) or that all relations have only
a subject and object (Kolluru et al. 2020b). To answer this question an
apples-to-apples comparison must be conducted for different application
settings, keeping the training data, benchmark, and evaluation criteria
constant while comparing models.

Because it is impractical to find the best model for every application given
the many possible applications of OpenIE, we instead characterize applications
based on what properties they desire from OpenIE. For example, the desire for
N-ary relation extraction by event schema induction. We use these properties
to characterize OpenIE models and datasets and then evaluate whether those
properties result in meaningful differences in performance. In the process of
answering these questions we provide an extensive apples-to-apples comparison
of existing neural OpenIE models such that a practitioner can utilize our
practical observations to effectively select a neural OpenIE model and
training set for their downstream application. Finally, we apply our
recommendations to a downstream Complex QA task. We hope our survey provides
insight into how to select OpenIE models and datasets for future
practitioners’ applications. In summary, our contributions are as follows:

  * •

We propose a comprehensive taxonomy that covers OpenIE training sets,
benchmarks, evaluation metrics, and neural models.

  * •

We present an extensive empirical comparison of different models on different
datasets with recommendations based on the results.

  * •

We perform a case study on Complex QA to show the efficacy of our
recommendations.

To the best of our knowledge, our survey is the only application-focused
empirical survey on OpenIE datasets and recent neural OpenIE methods.

|  | Question
---
Answering
Slot Filling |  | Event Schema
---
Induction
Summarization |  | Knowledge Base
---
Population
HR: Higher Recall | ✓ | ✓ | ✓ | ✓ | ✓
HP: Higher Precision |  | ✓ |  | ✓ |
N-ary: N-ary Relation Extraction | ✓ | ✓ | ✓ |  |
IN: Inferred Relation Extraction | ✓ | ✓ | ✓ | ✓ |
FE: Fast Extraction |  |  |  |  | ✓

Table 1: Properties explicitly mentioned in application papers as motivation
for choosing a particular OpenIE model or as a way to improve performance
within a case study. There are additional desired properties we omit that no
existing models or datasets have, such as the canonicalization of extracted
relations and extracting relations from imperative sentences (Fader,
Zettlemoyer, and Etzioni 2013; Khot, Sabharwal, and Clark 2017; Zhang et al.
2021).

##  2 Motivating Applications

In this section, we identify the properties of OpenIE systems desired by 5
downstream applications: Slot Filling, Question Answering (QA), Summarization,
Event Schema Induction, and Knowledge Base Population. We survey how OpenIE is
used in each of the applications and the properties explicitly desired based
on the corresponding papers, either as motivation for choosing a given OpenIE
model or within a case study as a property that would improve performance.

The desired properties we observe are Higher Recall, Higher Precision, N-ary
Relation Extraction, Inferred Relation Extraction, and Fast Extraction. We
define an ”Inferred Relation” (IN) to be a relation where the predicate
contains words that are not in the original sentence. For example, given the
sentence ”Bill Gates, former CEO of Microsoft, is a Harvard dropout”, the
relation (Bill Gates, was, former CEO of Microsoft) can be inferred even
though ”was” is not in the original sentence. We define an ”N-ary Relation”
(N-ary) to be a relation with more arguments than just (subject, predicate,
object). For instance, the relation (Alice, baked, Bob, a pie) has an
additional argument.

Slot Filling Slot filling is a task where an incomplete tuple must be
completed using information from a given corpus (Chen, Zhuo, and Wang 2019).
For example, given the incomplete tuple (Obama, born in, ?), extract (Obama,
was born in, Honolulu) using information from the corpus. In this task, OpenIE
is used to extract complete tuples which are used to fill slots by linking
those tuples to tuples missing slots using entity linking methods. OpenIE can
be used by extracting complete tuples which are linked to the incomplete tuple
using entity linking methods. The incomplete tuple can then be completed by
using the complete tuple extracted using OpenIE. Because OpenIE is not
constrained to a predefined schema, it is useful for extracting different
surface forms of relations. Soderland et al. (2013), Angeli, Premkumar, and
Manning (2015), Soderland et al. (2015b), and Soderland et al. (2015a) take
advantage of how correct relations often appear multiple times in text to
match empty slots to the highest precision OpenIE tuple. Soderland et al.
(2013), Angeli, Premkumar, and Manning (2015), Soderland et al. (2015b), and
Soderland et al. (2015a) all state in their case studies they would benefit
from the ability to extract inferred relations (IN), and Soderland et al.
(2015b) and Soderland et al. (2015a) state they would benefit from the ability
to extract n-ary relations (N-ary). These two properties allow more surface
forms of relations to be extracted, which allows for more slots to be filled.

Question Answering We focus on 2 subtasks of QA that utilize OpenIE: Open-
domain Question Answering (OpenQA) and Complex QA. Fader, Zettlemoyer, and
Etzioni (2013, 2014a); Yin et al. (2015), and Clark et al. (2018) are OpenQA
methods that use retrieval-based methods to match OpenIE extractions to
questions. OpenQA involves answering questions given a large database (Fader,
Zettlemoyer, and Etzioni 2014b). By rewriting queries into incomplete tuples,
it is possible to use relations extracted from the database to answer queries
by filling in the missing slots in the query. For example, rewriting the query
”Where was Obama born?” into slot filling the tuple (Obama, born in, ?) and
answering using the relation (Obama, was born in, Honolulu). Fader,
Zettlemoyer, and Etzioni (2014a) and Yin et al. (2015) obtain those relations
from a knowledge base of OpenIE extractions.

Complex questions must use information from multiple sentences to find answers
and require inferring relationships between multiple entities(Chali, Joty, and
Hasan 2009). Khot, Sabharwal, and Clark (2017) and Lu et al. (2019) generate
graphs from extracted relation tuples, then reason over these graphs to answer
the questions.

In all QA applications surveyed, high recall (HR) is desired, with Lu et al.
(2019) using a custom OpenIE method specifically to obtain a higher recall.
Yin et al. (2015)’s case studies state that N-ary in particular would benefit
performance while Lu et al. (2019) uses a custom OpenIE method that supports
IN. Several methods already paraphrase questions so that the surface forms of
extracted relations match at least one of the question paraphrases, indicating
that extracting more surface forms of a relation would answer more questions
(Fader, Zettlemoyer, and Etzioni 2013, 2014a; Yin et al. 2015).

Summarization OpenIE addresses the problems of redundancy and fact fabrication
in summarization. To combat redundancy in hierarchical and abstractive
summarization, OpenIE is used to ensure that the generated summary does not
have repeated relations or more relations than the gold standard summary
(Christensen et al. 2014; Zhang et al. 2021). To combat fact fabrication in
abstractive summarization, OpenIE is used to ensure that the generated summary
only contains relations from the original text (Cao et al. 2018; Zhang et al.
2021). In summarization tasks, HR is useful to ensure summaries contain all
information, with Ponza, Del Corro, and Weikum (2018) citing greater diversity
of extractions as a way to improve performance. The exception is Zhang et al.
(2021), where high precision (HP) is desired in order to reduce redundant
extractions.

Event Schema Induction Event Schema Induction is the automatic discovery of
patterns that indicate events and the agents and their roles within that
event. Extracted relation tuples can be used to find surface forms of events,
with repeated extracted tuples being used to induce event schemas. The open
nature of OpenIE allows for events to be found regardless of the domain or
surface form of the event. Predicates of extracted tuples are generally mapped
to events and arguments to agents of the event (Balasubramanian et al. 2013;
Romadhony, Widyantoro, and Purwarianti 2019; Sahnoun, Elloumi, and Ben Yahia
2020). HR is useful for Event Schema Induction for the same reason it is
useful for Slot Filling: finding more surface forms of an event allows for
more event schemas to be induced. Sahnoun, Elloumi, and Ben Yahia (2020) also
specifically desire IN so that more event schemas can be learned, while
Balasubramanian et al. (2013) state that N-ary would improve performance.

Knowledge Base Population OpenIE’s open-domain nature has led to its use in
automatically populating knowledge bases (KBs). The relations extracted by
OpenIE can be used to create new nodes and edges in KBs. Muhammad et al.
(2020) and Kroll, Pirklbauer, and Balke (2021) use learning-based OpenIE
models because of their ability to generalize to unseen relations and achieve
HR. Kroll, Pirklbauer, and Balke (2021) also explicitly chooses Stanford
CoreNLP and OpenIE6 because of their fast extraction times (FE).

We use these desired properties to formulate research questions in Section 5.
Table 1 provides a summary of applications and their explicitly desired
properties.

| Dataset |  | Creation Method
---
Source | #Extractions | #IN | #N-ary
Training Sets | SpanOIE |  | Weak Labeling
---
Wikipedia | 2,175K | 2K | 231K
OIE4 |  | Weak Labeling
---
Wikipedia | 181K | 3K | 34K
IMoJIE |  | Weak Labeling
---
Wikipedia | 215K | 3K | 0
LSOIE |  | Weak Labeling
---
| QA-SRL 2.0 Wikipedia, Science
---
101K | 0 | 32K
Test Sets | OIE2016 |  | Weak Labeling
---
QA-SRL | 1,730 | 359 | 708
WiRe57 |  | Manual Annotation
---
| Wikipedia and Newswire
---
343 | 173 | 79
ReOIE2016 |  | Manual Annotation
---
OIE2016 | 1,508 | 155 | 611
CaRB |  | Crowdsourced Annotation
---
OIE2016 | 5,263 | 736 | 683
LSOIE |  | Weak Labeling
---
| QA-SRL 2.0 Wikipedia, Science
---
22,376 | 0 | 4,920

Table 2: Comparison of the attributes of different datasets. $\\#$Extractions:
Number of Extractions, $\\#$IN : Number of inferred relations, $\\#$N-ary:
Number of N-ary Relations.

##  3 Datasets

In this section, we discuss the differences between different OpenIE training
sets and benchmarks and what properties they possess. We also briefly touch on
different evaluation metrics. The common principles that guide the creation of
OpenIE training sets and benchmarks are Assertedness, Minimal
Propositions/Atomicity, and Completeness and Open Lexicon (Stanovsky and Dagan
2016; Léchelle, Gotti, and Langlais 2018; Bhardwaj, Aggarwal, and Mausam
2019). Assertedness means the relation must be implied by the original
sentence alone. Minimal Propositions means each relation should include as few
words as possible while retaining meaning. Completeness and Open Lexicon means
all relations should be extracted, without a predefined domain or scope.

Despite these common guiding principles, OpenIE datasets still differ in their
creation method and subsequently have differing properties. We provide
statistics about different datasets in table 2.

### Training Datasets

Given how data-hungry deep learning models are and how costly it is to
manually label OpenIE datasets, most OpenIE training sets are weakly labeled
using high confidence extractions from prior OpenIE models to get ”silver-
standard” labels. CopyAttention (Cui, Wei, and Zhou 2018), SpanOIE (Zhan and
Zhao 2020), and OIE4 (Kolluru et al. 2020b) are training sets consisting of
high confidence OpenIE4 extractions from Wikipedia. Unlike CopyAttention and
OIE4, SpanOIE includes low-quality extractions with pronoun arguments. The
IMoJIE dataset (Kolluru et al. 2020b) attempts to get higher quality labels by
combining Wikipedia extractions from OpenIE4, ClausIE, and RNNOIE, using a
common scoring metric to combine extractions and filtering out repeated
extractions. The LSOIE training set (Solawetz and Larson 2021) is composed of
automatically converted Semantic Role Labeling (SRL) extractions with high
inter-annotator agreement from the Wikipedia and Science domain of the
crowdsourced QA-SRL Bank 2.0 dataset. Because this dataset is derived from
SRL, all relations are assumed to be verb-based and only contain words in the
original sentence.

### Benchmarks

OIE2016 (Stanovsky and Dagan 2016) is a benchmark for OpenIE automatically
derived from the crowdsourced QA-SRL dataset annotated on PropBank and
Wikipedia sentences. WiRe57 (Léchelle, Gotti, and Langlais 2018) consists of
expert annotations for 57 sentences. CaRB (Bhardwaj, Aggarwal, and Mausam
2019) uses crowdsourcing to re-annotate the sentences in the OIE2016
benchmark. In contrast to other OpenIE datasets, each extraction contains as
much information as possible in the arguments, meaning prepositions are part
of the arguments and not the predicate. ReOIE2016 (Zhan and Zhao 2020) uses
manual annotation to re-annotate OIE2016 to attempt to resolve problems
arising from incorrect extraction. The LSOIE (Solawetz and Larson 2021)
training set has a corresponding benchmark derived using the same source and
rules. BenchIE (Gashteovski et al. 2021) is derived from CaRB and is based on
the idea that extracted relations need to exactly match at least one gold
standard out of a set of equivalent manually annotated relations to be useful
for downstream applications.

Some of these benchmarks introduce new evaluation metrics. OIE2016 introduces
lexical matching, which treats extraction as a binary classification task. A
predicted relation is matched to a gold standard relation if the heads of the
predicate and all arguments are the same. WiRe57 and CaRB use word-level
matching, which calculate recall and precision based on the proportion of
matching tokens in the predicted and gold standard relations. The difference
between the two is that there is a greater penalty to recall for WiRe57 if
there are fewer predicted relations than relations in the gold standard.

BenchIE uses sentence-level matching. Sentence-level matching requires an
exact match of the predicate and arguments instead of just the heads like
lexical matching. Instead of a set of gold tuples to represent the gold
standard relations, fact sets represent the gold standard relations. Each fact
set consists of a set of equivalent relations and a predicted relation is
considered a true positive for a given relation if it matches a relation
within the fact set exactly.

Because of BenchIE’s reliance on fact sets which other benchmarks lack, the
BenchIE metric is only compatible with BenchIE and no other metrics can be
used with the BenchIE dataset. As a result, an apples-to-apples comparison of
the BenchIE dataset and metric are not possible like with other datasets and
metrics, so we do not report performance on BenchIE.

Model Problem Formulation N-ary IN SpanOIE Labeling ✓ IMoJIE Generation
Multi2OIE Labeling ✓ IGL-OIE Labeling ✓ CIGL-OIE Labeling ✓ OpenIE6 Labeling ✓

Table 3: Comparison of neural OpenIE models.

##  4 Models

In this section, we present recent neural OpenIE models and the properties
that set them apart. OpenIE models can be categorized based on how they
formulate the OpenIE problem: as a text generation or labeling problem.
Different models also possess different properties depending on what
assumptions they make about relations. We provide overviews of the models in
table 3.

### Generative Problem Formulation

Generative OpenIE models cast OpenIE as a sequence-to-sequence problem, taking
the sentence as input and attempting to generate all relations in the sentence
as output. CopyAttention (Cui, Wei, and Zhou 2018) generates extractions using
GloVe embeddings and a 3-layer stacked Long Short-Term Memory (LSTM) as the
encoder and decoder. IMoJIE (Kolluru et al. 2020b) builds upon CopyAttention
by using BERT embeddings and introducing iterative extraction to combat
repeated extractions. Iterative extraction is the appending of extractions to
the end of the sentence before being used as input so the model can identify
what relations have previously been extracted at the cost of significantly
reduced extraction speed. Adversarial-OIE (Han and Wang 2021) uses a
Generative Adversarial Network (GAN) to generate extractions. The model
consists of a sequence generator model that takes BERT and position embeddings
as input and generates extractions, and an adversary model that tries to
distinguish between relations in the gold standard and relations extracted by
the sequence generator model.

Generative models rely on a copy mechanism to copy vocabulary from the
original sentence, meaning they can not generate tokens that are not in the
original sentence and subsequently can not extract IN relations.

### Labeling Problem Formulation

Labeling OpenIE models cast OpenIE as a sequence labeling problem, taking the
sentence as input and labeling each token in the sentence with its role in
each relation, usually using a BIO tagging scheme. Labeling models can be
further subdivided into Piecewise and Holistic Labeling.

Piecewise Labeling models label predicates and arguments in different stages.
RnnOIE (Stanovsky et al. 2018) is a bi-directional LSTM (BiLSTM) transducer
inspired by SRL that firsts labels predicates and then labels arguments with
BIO tags for each extracted predicate. SpanOIE (Zhan and Zhao 2020) is also
based on SRL, using a BiLSTM to perform span classification instead of BIO
tagging. Span classification enables the use of span features, which can be
richer than word-level features. Multi2OIE’s (Ro, Lee, and Kang 2020) novelty
is multi-head attention and BERT embeddings. After labeling the predicates,
multi-head attention is used between the predicate and the rest of the
sentence to label the arguments. MILIE (Kotnis et al. 2021) introduces its own
iterative prediction, the process of extracting one component of the relation
tuple at a time, for multilingual OpenIE. Extraction can be performed
predicate first, subject first, or object first, followed by any of the
remaining components. The intention was to make multilingual extraction
easier, in case other languages benefited from different extraction orders.
DetIE (Vasilkovsky et al. 2022) uses ideas from single-shot object detection
to make predictions more quickly than previous methods.

Uniquely, piecewise labeling models label all predicates in a sentence
simultaneously and assume that for each predicate, there is only one set of
arguments. This means that they can not extract multiple relations that share
the same predicate, unlike generative and holistic labeling models.

Holistic Labeling models label predicates and arguments simultaneously OpenIE6
(Kolluru et al. 2020a) introduces grid labeling, constraint rules, and
conjunction rules to improve the labeling process. Grid labeling is
simultaneous extraction of multiple relations from a sentence. Constraint
rules are used to penalize certain things like repeated extractions or not
extracting a relation for a head verb. Conjunction rules are used to split
relations containing conjunctions into two separate relations. IGL-OIE is the
first stage, using only grid labeling; CIGL-OIE is the second stage, adding in
constraint rules; OpenIE6 is the final stage, using conjunction rules to
handle relations that contain conjunctions.

Labeling models generally can not label tokens that are not in the original
sentence, meaning they can not extract IN relations. The exceptions are IGL-
OIE, CIGL-OIE, and OpenIE6, which explicitly allows for the extraction of ”be”
relations even if they are not in the original sentence.

##  5 Experiments

In this section, we describe how we compare OpenIE models and datasets for the
sake of recommendation. In our experiments, we focus on the reported state-of-
the art models for English OpenIE, the most widely-used English benchmarks
with associated papers, and the 4 training sets used by those models. We then
use OpenIE in a downstream Complex QA task to demonstrate the applicability of
our recommendations.

When comparing OpenIE systems, we place a greater emphasis on F1 score than
AUC. The original implementations of CaRB, OIE2016, and WiRe57 use the
trapezoidal rule to calculate the area under the PR curve. It is assumed that
all PR curves have data points at recall 0, precision 1. This means that
methods without low recall points on the PR curve will have inflated AUC
values. For example, for the CaRB test set using the CaRB metric, CIGL-OIE
trained on OIE4 has a minimum recall of 0.214 with precision 0.678 at
confidence 1.0, while Multi2OIE trained on SpanOIE has a minimum recall of
0.0003 with precision 0.5 at confidence 1.0. Considering just these lowest
recall data points, CIGL-OIE has an AUC of 0.180, while Multi2OIE has an AUC
of 0.0002, meaning CIGL-OIE already has a large inherent advantage in AUC
without considering the higher confidence data points. As a result, we
consider the highest F1 score on the PR curve to be a better metric than AUC
when evaluating overall model performance.

### Research Questions

To find the best system for different applications, we test whether the
properties of OpenIE models and datasets have a statistically significant
effect on accuracy in benchmarks with properties that correspond to the
desires of downstream applications. If there is no significant difference
between the performance of OpenIE systems with different properties, then the
chosen system should be the system with the best overall empirical
performance. In addition to how model and dataset properties affect accuracy,
we are also interested in how the choice of model affects efficiency in order
to satisfy the fast extraction property (FE). Generally, generative models are
much slower than labeling models (Kolluru et al. 2020a). Subsequently, we
investigate whether the efficiency difference between model types is
significant.

  1. 1.

R1: How does whether a model supports N-ary relation (N-ary) extraction and
whether the training set contains N-ary affect the F1 score of a model on test
sets with N-ary?

  2. 2.

R2: How does whether a model supports inferred relation (IN) extraction and
whether the training set contains IN affect the F1 score of a model on test
sets with or without IN?

  3. 3.

R3: How does the model type affect efficiency as measured by the number of
sentences relations are extracted from per second (Sen./Sec)?

Model Sen./Sec. CaRB WiRE 57 P R F1 P R F1 SpanOIE 13.40 0.474 0.464 0.433
0.474 0.374 0.375 IMoJIE 2.07 0.598 0.431 0.488 0.598 0.355 0.428 Multi2OIE
29.22 0.626 0.501 0.552 0.624 0.419 0.488 IGL-OIE 84.07 0.574 0.442 0.497
0.574 0.365 0.434 CIGL-OIE 68.80 0.490 0.531 0.503 0.489 0.429 0.442 OpenIE6
28.36 0.394 0.518 0.438 0.394 0.463 0.413

Table 4: Performances of different models with different evaluation metrics
averaged across training and test data.

Training Set Test Set CaRB WiRE 57 P R F1 P R F1 SpanOIE OIE2016 0.485 0.502
0.478 0.484 0.420 0.434 OIE4 OIE2016 0.537 0.493 0.511 0.536 0.410 0.461 LSOIE
OIE2016 0.620 0.538 0.564 0.620 0.447 0.508 IMoJIE OIE2016 0.446 0.449 0.423
0.445 0.378 0.382 SpanOIE WiRe57 0.411 0.371 0.381 0.415 0.199 0.261 OIE4
WiRe57 0.470 0.374 0.416 0.469 0.211 0.289 LSOIE WiRe57 0.347 0.208 0.257
0.347 0.127 0.183 IMoJIE WiRe57 0.421 0.370 0.373 0.419 0.223 0.266 SpanOIE
ReOIE2016 0.636 0.635 0.615 0.636 0.622 0.610 OIE4 ReOIE2016 0.720 0.565 0.597
0.720 0.553 0.590 LSOIE ReOIE2016 0.622 0.527 0.558 0.622 0.516 0.552 IMoJIE
ReOIE2016 0.586 0.588 0.554 0.584 0.569 0.544 SpanOIE CaRB 0.529 0.446 0.471
0.526 0.313 0.378 OIE4 CaRB 0.604 0.449 0.513 0.604 0.317 0.412 LSOIE CaRB
0.532 0.344 0.412 0.532 0.255 0.337 IMoJIE CaRB 0.517 0.429 0.447 0.514 0.315
0.360 SpanOIE LSOIE 0.461 0.573 0.501 0.461 0.525 0.477 OIE4 LSOIE 0.505 0.566
0.532 0.505 0.517 0.506 LSOIE LSOIE 0.653 0.686 0.660 0.653 0.629 0.629 IMoJIE
LSOIE 0.419 0.508 0.440 0.419 0.473 0.427

Table 5: Performances of different training and test sets averaged across
models.

### Experimental Setup

Models: We compare SpanOIE, IMoJIE, Multi2OIE, and the 3 stages of OpenIE6:
IGL-OIE, CIGL-OIE, and OpenIE6. These models are publicly available, with
Multi2OIE and OpenIE6 being state-of-the-art for labeling models and IMoJIE
being state-of-the-art for generative models. For each model, we train them
with their original dev set and their original hyperparameters. We run all
experiments using a Quadro RTX 5000 GPU.

Training Datasets: We train the models on the SpanOIE, OIE4, IMoJIE, and LSOIE
training sets. The LSOIE training set contains both the Science and Wikipedia
domain sentences to increase the amount of training data. Due to the input
structure of SpanOIE and Multi2OIE models, they can not be trained on training
datasets with inferred relations. We remove any inferred relations from the
training sets of those models. Similarly, as IMoJIE and OpenIE6 can not
extract N-ary relations, we convert all N-ary relations in the training set
into binary relations by moving arguments beyond the subject and object into
the object. For instance, the relation (Alice, baked, Bob, a pie) was
converted into (Alice, baked, Bob a pie). None of these limitations apply to
the test sets.

Benchmarks: We evaluate all the models on the publicly available English
benchmarks OIE2016, WiRE57, ReOIE2016, CaRB, and LSOIE.

Evaluation Metrics: We use OIE2016’s, WiRE57’s, and CaRB’s metrics for
evaluation. We compare performance using primarily F1 score to address HR and
HP and sentences extracted per second to address FE. We perform student’s
t-test between OpenIE system, test set, and evaluation metric configurations
to answer R1, R2, and R3. For R1 and R2 the t-scores are computed using the
per-sentence F1 scores of each method. For R3 the t-scores are computed using
the mean sentences per second for each training set and test set combination
for a given model, resulting in 24 data points per model. We do not measure
the time required to extract from each individual sentence because it can be
difficult to measure given the parallel execution of each mode.

##  6 Results

### Overall Evaluation

In this section, we perform an apples-to-apples comparison among different
OpenIE systems to first determine the SoTA OpenIE model and then to determine
the best general-purpose OpenIE training dataset.

Best OpenIE Model We compare the different models on different evaluation
metrics averaged across different training and test sets in Table 4. We
observe that across all evaluation metrics Multi2OIE and the CIGL-OIE have the
highest or second highest F1 score. This means that independent of training
and test sets the Labeling OpenIE models are better than the Span
Classification and Generative models. We also observe that Labeling OpenIE
models are more efficient than the Span Classification and Generative models,
extracting from more sentences per second.

Best OpenIE Training Set Because performance on a test set is also greatly
dependent on the training set depending on the domain and generation method of
the training and test sets, we determine the best training set for each test
set. In Table 5, we compare different training and test set combinations with
different evaluation metrics averaged across models. We observe that the
models trained on LSOIE training set perform best on the OIE2016 and LSOIE
test sets. This is because the LSOIE training set and the OIE2016 and LSOIE
test sets are derived from different versions of QA-SRL and generated using
the same rules. On the WiRe57, ReOIE2016, and CaRB test sets, we observe that
the models trained on the OIE4 and SpanOIE training sets generally perform the
best. It is likely because the OIE4 and SpanOIE training sets contain N-ary
and IN relations like the WiRe57, ReOIE2016, and CaRB test sets. Other
training sets lack one or both of these properties.

Sentence A short distance to the east, NC 111 diverges on Greenwood Boulevard.
Multi2OIE (NC 111, diverges, on Greenwood Boulevard) CIGL-OIE (NC 111,
diverges, A short distance to the east on Greenwood Boulevard)

Table 6: A demonstration that CIGL-OIE tends to extract longer arguments than
Multi2OIE. Multi2OIE is trained on SpanOIE and CIGL-OIE is trained on OIE4.
The sentence is from the CaRB test set.

Of the two models with the highest average CaRB F1 scores, Multi2OIE and CIGL-
OIE, Multi2OIE has higher average precision while CIGL-OIE has higher average
recall. CIGL-OIE tends to extract longer objects than Multi2OIE as seen in
table 6, which may explain this difference.

### Research Questions

To answer our research questions, we perform student’s t-test using the CaRB
F1 scores of the highest scoring model, training set, and test set
combinations for each setting. We perform comparisons of OpenIE systems, where
one aspect (model or training set) is changed and the other aspects are kept
constant. Then, we choose the test set and evaluation metric for the two
settings that results in the highest t-score between methods, which is used to
answer R1 and R2.

For R1, we conclude (1) regardless of training set, the best N-ary models
perform better than the best non-N-ary models; (2) regardless of the model,
training on the best N-ary training sets results in higher performance than
training on the best non-N-ary training sets. Therefore if an application
benefits from N-ary, then the best OpenIE system should include either a N-ary
model, N-ary training set, or both, with both being preferred if both are
available.

For R2, we infer that IN models are better than non-IN models when there is
either a IN training and IN test set, or when there is a non-IN training and
non-IN test set. IN training sets are better than non-IN training sets when
there is an IN model and IN test set. In the case of non-IN and IN test set,
it is unclear whether IN or non-IN training sets are superior. Therefore if an
application benefits from IN, then the chosen training set and model should
either both be IN or both be non-IN. If an application benefits from non-IN,
then the chosen training set should be non-IN, and subsequently the chosen
model should be IN.

For R3, we compare the efficiency of the sole generative model, IMoJIE, to the
efficiency of every other model. From our results, we observe that every other
model is faster than IMoJIE and the difference is statistically significant.
This makes intuitive sense, since generative models tend to have longer
execution times than labeling models and it has been previously shown that
IMoJIE is exceptionally slow compared to other OpenIE methods (Kolluru et al.
2020a). Therefore if an application is concerned about efficiency, then the
chosen OpenIE model should not be a generative model.

##  7 A Case Study: Complex QA

To verify our recommendations, we perform a case study using QUEST (Lu et al.
2019), a Complex QA method that uses OpenIE to extract entities and predicates
from the question and from answer documents to generate knowledge graphs. The
nodes are entities derived from the subjects and objects, while the edges are
predicates. The knowledge graph is matched to the entities in the question and
traversed to find potential answers. Because more extractions result in a
larger knowledge graph, QUEST benefits from HR which the authors use their own
rule-based OpenIE method to achieve.

### Experimental Setup

To test our recommendations, we replace the OpenIE method used by the authors
with Multi2OIE trained on SpanOIE, CIGL-OIE trained on OIE4, and OpenIE6
trained on OIE4. We chose these models and training sets because they are the
ones with the overall highest CaRB recall and F1 scores that match the
properties desired by complex QA, namely N-ary and IN.

One caveat is that in order for QUEST to connect entities from multiple
sentences, they must have the same surface form. Because OpenIE methods often
extract long subjects and objects that include adjectives and modifiers, if
the subject or object of an extraction contains entities extracted by QUEST,
we add additional relations using those entities. For example, in the sentence
”Hector Elizondo was nominated for a Golden Globe for his role in Pretty
Woman,” QUEST may extract the entities ”Hector Elizondo,” ”Golden Globe,” and
”Pretty Woman.” If an OpenIE method were to extract the triple (”Hector
Elizondo”, ”was nominated”, ”for a Golden Globe for his role in Pretty
Woman”), we would add the additional extractions (”Hector Elizondo”, ”was
nominated”, ”Golden Globe”) and (”Hector Elizondo”, ”was nominated”, ”Pretty
Woman”). QUEST also performs preprocessing before running its rule-based
OpenIE method, most prominently replacing pronouns with the entities they
refer to. This is because nodes in the knowledge graph can not be made using
pronouns. We replace pronouns using the same method QUEST does before running
any OpenIE method.

We run QUEST using the CQ-W question set and search for answers in the Top-10
Google document set used in their paper. Because CIGL-OIE has the highest CaRB
recall and OpenIE6 has the highest WiRe57 recall, we expect that using either
of them will result in higher downstream performance than using Multi2OIE.

OpenIE Questions Documents MRR P@1 Hit@5 QUEST CQ-W Top 10 0.132 0.080 0.167
CIGL-OIE CQ-W Top 10 0.111 0.060 0.167 OpenIE6 CQ-W Top 10 0.104 0.060 0.147
Multi2OIE CQ-W Top 10 0.094 0.053 0.140

Table 7: Performance of QUEST using different OpenIE methods on the CQ-W
dataset using the Top 10 Google documents.

### Evaluation

We compare the Mean Reciprocal Rank (MRR), Precision@1 (P@1), and Hit@5 for
each OpenIE model. The results of our case study are summarized in table 7. We
observe higher performance of CIGL-OIE and OpenIE6 than Multi2OIE on QUEST,
which matches our expectations based on the higher recall of CIGL-OIE and
OpenIE6. Our case study demonstrates the applicability of our empirical study
to the use of OpenIE methods in downstream applications.

An important note is that oftentimes a great deal of pre- and post-processing
is necessary to adapt OpenIE for different downstream applications. Removing
pronouns and adding additional entity-based extractions was necessary to
achieve reasonable performance with different OpenIE methods in QUEST. Even
after modifying Multi2OIE, CIGL-OIE, and OpenIE6 in this way, their
performance is less than the original performance of QUEST. As a result, it is
important for practitioners to not just consider the performance of OpenIE
models on test sets matching their application, but to also consider how to
adapt OpenIE to the specific needs of their application.

##  8 Conclusion

In this paper, we presented an application-focused empirical comparison of
recent neural OpenIE models, training sets, and benchmarks. Our experiments
showed that the different properties of OpenIE models and datasets affect the
performance, meaning it is important to choose the appropriate system for a
given application and not just choose whatever model is state-of-the-art. We
hope that this survey helps users identify the best OpenIE system for their
downstream applications and inspires new OpenIE research into addressing the
properties desired by downstream applications.

## References

  * Angeli, Premkumar, and Manning (2015) Angeli, G.; Premkumar, M. J. J.; and Manning, C. D. 2015.  Leveraging linguistic structure for open domain information extraction.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 344–354.
  * Balasubramanian et al. (2013) Balasubramanian, N.; Soderland, S.; Etzioni, O.; et al. 2013.  Generating coherent event schemas at scale.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , 1721–1731.
  * Bhardwaj, Aggarwal, and Mausam (2019) Bhardwaj, S.; Aggarwal, S.; and Mausam, M. 2019.  CaRB: A crowdsourced benchmark for open IE.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 6262–6267.
  * Cao et al. (2018) Cao, Z.; Wei, F.; Li, W.; and Li, S. 2018.  Faithful to the original: Fact aware neural abstractive summarization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Chali, Joty, and Hasan (2009) Chali, Y.; Joty, S. R.; and Hasan, S. A. 2009.  Complex question answering: unsupervised learning approaches and experiments.  _Journal of Artificial Intelligence Research_ , 35: 1–47.
  * Chen, Zhuo, and Wang (2019) Chen, Q.; Zhuo, Z.; and Wang, W. 2019.  Bert for joint intent classification and slot filling.  _arXiv preprint arXiv:1902.10909_.
  * Christensen et al. (2014) Christensen, J.; Soderland, S.; Bansal, G.; et al. 2014.  Hierarchical summarization: Scaling up multi-document summarization.  In _Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers)_ , 902–912.
  * Clark et al. (2018) Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Claro et al. (2019) Claro, D. B.; Souza, M.; Castellã Xavier, C.; and Oliveira, L. 2019.  Multilingual open information extraction: Challenges and opportunities.  _Information_ , 10(7): 228.
  * Cui, Wei, and Zhou (2018) Cui, L.; Wei, F.; and Zhou, M. 2018.  Neural open information extraction.  _arXiv preprint arXiv:1805.04270_.
  * Etzioni et al. (2008) Etzioni, O.; Banko, M.; Soderland, S.; and Weld, D. S. 2008.  Open information extraction from the web.  _Communications of the ACM_ , 51(12): 68–74.
  * Fader, Zettlemoyer, and Etzioni (2013) Fader, A.; Zettlemoyer, L.; and Etzioni, O. 2013.  Paraphrase-driven learning for open question answering.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1608–1618.
  * Fader, Zettlemoyer, and Etzioni (2014a) Fader, A.; Zettlemoyer, L.; and Etzioni, O. 2014a.  Open question answering over curated and extracted knowledge bases.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 1156–1165.
  * Fader, Zettlemoyer, and Etzioni (2014b) Fader, A.; Zettlemoyer, L.; and Etzioni, O. 2014b.  Open question answering over curated and extracted knowledge bases.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 1156–1165.
  * Garg and Kalai (2018) Garg, V.; and Kalai, A. T. 2018.  Supervising unsupervised learning.  _Advances in Neural Information Processing Systems_ , 31.
  * Gashteovski et al. (2021) Gashteovski, K.; Yu, M.; Kotnis, B.; Lawrence, C.; Glavas, G.; and Niepert, M. 2021\.  BenchIE: Open Information Extraction Evaluation Based on Facts, Not Tokens.  _arXiv preprint arXiv:2109.06850_.
  * Glauber and Claro (2018) Glauber, R.; and Claro, D. B. 2018.  A systematic mapping study on open information extraction.  _Expert Systems with Applications_ , 112: 372–387.
  * Han and Wang (2021) Han, J.; and Wang, H. 2021.  Generative adversarial networks for open information extraction.  _Advances in Computational Intelligence_ , 1(4): 1–11.
  * Khot, Sabharwal, and Clark (2017) Khot, T.; Sabharwal, A.; and Clark, P. 2017.  Answering complex questions using open information extraction.  _arXiv preprint arXiv:1704.05572_.
  * Kolluru et al. (2020a) Kolluru, K.; Adlakha, V.; Aggarwal, S.; Chakrabarti, S.; et al. 2020a.  Openie6: Iterative grid labeling and coordination analysis for open information extraction.  _arXiv preprint arXiv:2010.03147_.
  * Kolluru et al. (2020b) Kolluru, K.; Aggarwal, S.; Rathore, V.; Chakrabarti, S.; et al. 2020b.  IMoJIE: Iterative Memory-Based Joint Open Information Extraction.  _arXiv preprint arXiv:2005.08178_.
  * Kotnis et al. (2021) Kotnis, B.; Gashteovski, K.; Lawrence, C.; Rubio, D. O.; Rodriguez-Tembras, V.; Takamoto, M.; and Niepert, M. 2021.  Integrating diverse extraction pathways using iterative predictions for Multilingual Open Information Extraction.  _arXiv preprint arXiv:2110.08144_.
  * Kroll, Pirklbauer, and Balke (2021) Kroll, H.; Pirklbauer, J.; and Balke, W.-T. 2021.  A toolbox for the nearly-unsupervised construction of digital library knowledge graphs.  In _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in_.
  * Léchelle, Gotti, and Langlais (2018) Léchelle, W.; Gotti, F.; and Langlais, P. 2018.  Wire57: A fine-grained benchmark for open information extraction.  _arXiv preprint arXiv:1809.08962_.
  * Lu et al. (2019) Lu, X.; Pramanik, S.; Saha Roy, R.; Abujabal, A.; Wang, Y.; and Weikum, G. 2019\.  Answering complex questions by joining multi-document evidence with quasi knowledge graphs.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 105–114.
  * Mausam (2016) Mausam, M. 2016.  Open information extraction systems and downstream applications.  In _Proceedings of the twenty-fifth international joint conference on artificial intelligence_ , 4074–4077.
  * Muhammad et al. (2020) Muhammad, I.; Kearney, A.; Gamble, C.; Coenen, F.; and Williamson, P. 2020.  Open Information Extraction for Knowledge Graph Construction.  In _International Conference on Database and Expert Systems Applications_ , 103–113. Springer.
  * Niklaus et al. (2018) Niklaus, C.; Cetto, M.; Freitas, A.; and Handschuh, S. 2018.  A survey on open information extraction.  _arXiv preprint arXiv:1806.05599_.
  * Ponza, Del Corro, and Weikum (2018) Ponza, M.; Del Corro, L.; and Weikum, G. 2018.  Facts that matter.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 1043–1048.
  * Ro, Lee, and Kang (2020) Ro, Y.; Lee, Y.; and Kang, P. 2020.  Multi2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT.  _arXiv preprint arXiv:2009.08128_.
  * Romadhony, Widyantoro, and Purwarianti (2019) Romadhony, A.; Widyantoro, D. H.; and Purwarianti, A. 2019.  Utilizing structured knowledge bases in open IE based event template extraction.  _Applied Intelligence_ , 49(1): 206–219.
  * Sahnoun, Elloumi, and Ben Yahia (2020) Sahnoun, S.; Elloumi, S.; and Ben Yahia, S. 2020.  Event detection based on open information extraction and ontology.  _Journal of Information and Telecommunication_ , 4(3): 383–403.
  * Soderland et al. (2013) Soderland, S.; Gilmer, J.; Bart, R.; Etzioni, O.; and Weld, D. S. 2013.  Open Information Extraction to KBP Relations in 3 Hours.  In _TAC_.
  * Soderland et al. (2015a) Soderland, S.; Hawkins, N.; Gilmer, J.; and Weld, D. S. 2015a.  Combining Open IE and Distant Supervision for KBP Slot Filling.  In _TAC_.
  * Soderland et al. (2015b) Soderland, S.; Hawkins, N.; Kim, G. L.; and Weld, D. S. 2015b.  University of Washington system for 2015 KBP cold start slot filling.  _Proceedings of TAC-KBP_ , 2015.
  * Solawetz and Larson (2021) Solawetz, J.; and Larson, S. 2021.  LSOIE: A Large-Scale Dataset for Supervised Open Information Extraction.  _arXiv preprint arXiv:2101.11177_.
  * Stanovsky and Dagan (2016) Stanovsky, G.; and Dagan, I. 2016.  Creating a large benchmark for open information extraction.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , 2300–2305.
  * Stanovsky et al. (2018) Stanovsky, G.; Michael, J.; Zettlemoyer, L.; and Dagan, I. 2018.  Supervised open information extraction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , 885–895.
  * Vasilkovsky et al. (2022) Vasilkovsky, M.; Alekseev, A.; Malykh, V.; Shenbin, I.; Tutubalina, E.; Salikhov, D.; Stepnov, M.; Chertok, A.; and Nikolenko, S. 2022.  DetIE: Multilingual Open Information Extraction Inspired by Object Detection.  In _Proceedings of the 36th AAAI Conference on Artificial Intelligence_.
  * Yin et al. (2015) Yin, P.; Duan, N.; Kao, B.; Bao, J.; and Zhou, M. 2015.  Answering questions with complex semantic constraints on open knowledge bases.  In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_ , 1301–1310.
  * Zhan and Zhao (2020) Zhan, J.; and Zhao, H. 2020.  Span model for open information extraction on accurate corpus.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 9523–9530.
  * Zhang et al. (2021) Zhang, M.; Zhou, G.; Yu, W.; and Liu, W. 2021.  FAR-ASS: Fact-aware reinforced abstractive sentence summarization.  _Information Processing & Management_, 58(3): 102478.

##  Appendix A Appendix A: Empirical Results

In this section, we show the empirical results of training each model on a
variety of training sets and evaluating them on a variety of test sets with
different evaluation metrics. We also show the empirical results of our
student’s t-tests comparing different OpenIE systems.

Model | Training set | Test set | Sen./Sec | OIE2016 | WiRe57 | CaRB
---|---|---|---|---|---|---
P | R | F1 | AUC | P | R | F1 | AUC | P | R | F1 | AUC
SpanOIE | SpanOIE | OIE2016 | 16.65 | 0.704 | 0.792 | 0.745 | 0.675 | 0.576 | 0.376 | 0.455 | 0.296 | 0.576 | 0.459 | 0.511 | 0.362
IMoJIE | SpanOIE | OIE2016 | 2.61 | 0.755 | 0.851 | 0.8 | 0.614 | 0.575 | 0.389 | 0.464 | 0.212 | 0.575 | 0.466 | 0.515 | 0.253
Multi2OIE | SpanOIE | OIE2016 | 28.21 | 0.724 | 0.915 | 0.809 | 0.719 | 0.558 | 0.439 | 0.491 | 0.29 | 0.566 | 0.521 | 0.542 | 0.348
IGL-OIE | SpanOIE | OIE2016 | 67.55 | 0.733 | 0.768 | 0.75 | 0.585 | 0.551 | 0.347 | 0.426 | 0.211 | 0.551 | 0.419 | 0.476 | 0.253
CIGL-OIE | SpanOIE | OIE2016 | 50.61 | 0.711 | 0.981 | 0.824 | 0.737 | 0.375 | 0.474 | 0.419 | 0.212 | 0.375 | 0.592 | 0.459 | 0.263
OpenIE6 | SpanOIE | OIE2016 | 23.32 | 0.519 | 0.975 | 0.678 | 0.532 | 0.269 | 0.492 | 0.348 | 0.177 | 0.269 | 0.556 | 0.362 | 0.2
SpanOIE | OIE4 | OIE2016 | 16.19 | 0.584 | 0.454 | 0.511 | 0.36 | 0.584 | 0.37 | 0.453 | 0.293 | 0.703 | 0.813 | 0.754 | 0.692
IMoJIE | OIE4 | OIE2016 | 3.44 | 0.553 | 0.474 | 0.51 | 0.231 | 0.553 | 0.399 | 0.464 | 0.196 | 0.695 | 0.824 | 0.754 | 0.495
Multi2OIE | OIE4 | OIE2016 | 31.14 | 0.597 | 0.491 | 0.539 | 0.32 | 0.595 | 0.4 | 0.478 | 0.261 | 0.747 | 0.864 | 0.801 | 0.72
IGL-OIE | OIE4 | OIE2016 | 70.02 | 0.544 | 0.48 | 0.51 | 0.313 | 0.544 | 0.39 | 0.455 | 0.257 | 0.718 | 0.84 | 0.774 | 0.661
CIGL-OIE | OIE4 | OIE2016 | 49.26 | 0.529 | 0.537 | 0.533 | 0.356 | 0.529 | 0.436 | 0.478 | 0.289 | 0.718 | 0.92 | 0.806 | 0.726
OpenIE6 | OIE4 | OIE2016 | 24.20 | 0.415 | 0.523 | 0.463 | 0.314 | 0.413 | 0.467 | 0.438 | 0.278 | 0.557 | 0.922 | 0.694 | 0.615
SpanOIE | LSOIE | OIE2016 | 15.36 | 0.657 | 0.521 | 0.581 | 0.432 | 0.657 | 0.432 | 0.521 | 0.358 | 0.657 | 0.804 | 0.723 | 0.666
IMoJIE | LSOIE | OIE2016 | 1.00 | 0.719 | 0.411 | 0.523 | 0.261 | 0.719 | 0.339 | 0.461 | 0.216 | 0.852 | 0.766 | 0.807 | 0.577
Multi2OIE | LSOIE | OIE2016 | 31.00 | 0.728 | 0.585 | 0.649 | 0.483 | 0.728 | 0.484 | 0.582 | 0.401 | 0.758 | 0.894 | 0.821 | 0.767
IGL-OIE | LSOIE | OIE2016 | 68.27 | 0.636 | 0.485 | 0.551 | 0.331 | 0.636 | 0.394 | 0.487 | 0.27 | 0.762 | 0.823 | 0.791 | 0.634
CIGL-OIE | LSOIE | OIE2016 | 52.40 | 0.568 | 0.618 | 0.592 | 0.391 | 0.568 | 0.494 | 0.528 | 0.314 | 0.74 | 0.947 | 0.831 | 0.738
OpenIE6 | LSOIE | OIE2016 | 24.56 | 0.41 | 0.609 | 0.49 | 0.315 | 0.41 | 0.541 | 0.466 | 0.279 | 0.542 | 0.924 | 0.683 | 0.563
SpanOIE | IMoJIE | OIE2016 | 7.16 | 0.084 | 0.428 | 0.14 | 0.232 | 0.084 | 0.394 | 0.138 | 0.213 | 0.188 | 0.975 | 0.316 | 0.579
IMoJIE | IMoJIE | OIE2016 | 1.68 | 0.551 | 0.451 | 0.496 | 0.225 | 0.551 | 0.381 | 0.451 | 0.191 | 0.779 | 0.905 | 0.837 | 0.607
Multi2OIE | IMoJIE | OIE2016 | 31.58 | 0.599 | 0.453 | 0.516 | 0.302 | 0.596 | 0.378 | 0.463 | 0.252 | 0.764 | 0.842 | 0.801 | 0.739
IGL-OIE | IMoJIE | OIE2016 | 63.00 | 0.545 | 0.396 | 0.459 | 0.238 | 0.545 | 0.323 | 0.406 | 0.194 | 0.775 | 0.797 | 0.786 | 0.592
CIGL-OIE | IMoJIE | OIE2016 | 49.62 | 0.509 | 0.482 | 0.495 | 0.269 | 0.509 | 0.375 | 0.432 | 0.21 | 0.775 | 0.928 | 0.845 | 0.69
OpenIE6 | IMoJIE | OIE2016 | 23.59 | 0.386 | 0.484 | 0.43 | 0.215 | 0.386 | 0.416 | 0.4 | 0.184 | 0.582 | 0.91 | 0.71 | 0.511

Table 8: A table that lists performances of different OpenIE systems on the
OIE2016 benchmark.

Model | Training set | Test set | Sen./Sec | OIE2016 | WiRe57 | CaRB
---|---|---|---|---|---|---
P | R | F1 | AUC | P | R | F1 | AUC | P | R | F1 | AUC
SpanOIE | SpanOIE | WiRe57 | 9.10 | 0.87 | 0.72 | 0.788 | 0.673 | 0.464 | 0.194 | 0.274 | 0.142 | 0.464 | 0.372 | 0.413 | 0.272
IMoJIE | SpanOIE | WiRe57 | 0.91 | 0.863 | 0.644 | 0.738 | 0.465 | 0.461 | 0.154 | 0.231 | 0.061 | 0.461 | 0.313 | 0.373 | 0.123
Multi2OIE | SpanOIE | WiRe57 | 23.17 | 0.9 | 0.758 | 0.823 | 0.698 | 0.498 | 0.203 | 0.288 | 0.097 | 0.498 | 0.391 | 0.438 | 0.186
IGL-OIE | SpanOIE | WiRe57 | 9.34 | 0.916 | 0.638 | 0.753 | 0.604 | 0.482 | 0.167 | 0.248 | 0.097 | 0.482 | 0.333 | 0.394 | 0.189
CIGL-OIE | SpanOIE | WiRe57 | 7.75 | 0.889 | 0.84 | 0.864 | 0.77 | 0.281 | 0.195 | 0.231 | 0.069 | 0.283 | 0.406 | 0.333 | 0.145
OpenIE6 | SpanOIE | WiRe57 | 3.50 | 0.74 | 0.831 | 0.783 | 0.641 | 0.304 | 0.28 | 0.291 | 0.127 | 0.28 | 0.408 | 0.332 | 0.167
SpanOIE | OIE4 | WiRe57 | 9.07 | 0.526 | 0.397 | 0.453 | 0.303 | 0.526 | 0.217 | 0.307 | 0.166 | 0.895 | 0.743 | 0.812 | 0.704
IMoJIE | OIE4 | WiRe57 | 1.19 | 0.414 | 0.35 | 0.379 | 0.109 | 0.414 | 0.189 | 0.26 | 0.059 | 0.823 | 0.665 | 0.735 | 0.433
Multi2OIE | OIE4 | WiRe57 | 19.65 | 0.537 | 0.37 | 0.439 | 0.194 | 0.537 | 0.197 | 0.289 | 0.104 | 0.921 | 0.717 | 0.807 | 0.67
IGL-OIE | OIE4 | WiRe57 | 8.19 | 0.457 | 0.337 | 0.388 | 0.22 | 0.452 | 0.174 | 0.251 | 0.111 | 0.931 | 0.673 | 0.782 | 0.653
CIGL-OIE | OIE4 | WiRe57 | 6.82 | 0.436 | 0.391 | 0.413 | 0.247 | 0.436 | 0.196 | 0.27 | 0.123 | 0.9 | 0.787 | 0.84 | 0.742
OpenIE6 | OIE4 | WiRe57 | 3.47 | 0.451 | 0.397 | 0.423 | 0.261 | 0.451 | 0.295 | 0.357 | 0.192 | 0.799 | 0.755 | 0.777 | 0.662
SpanOIE | LSOIE | WiRe57 | 8.52 | 0.357 | 0.209 | 0.263 | 0.142 | 0.357 | 0.135 | 0.196 | 0.092 | 0.759 | 0.534 | 0.627 | 0.469
IMoJIE | LSOIE | WiRe57 | 0.46 | 0.351 | 0.182 | 0.24 | 0.052 | 0.351 | 0.094 | 0.148 | 0.026 | 0.961 | 0.574 | 0.719 | 0.534
Multi2OIE | LSOIE | WiRe57 | 18.31 | 0.44 | 0.202 | 0.276 | 0.106 | 0.44 | 0.128 | 0.198 | 0.067 | 0.851 | 0.534 | 0.656 | 0.485
IGL-OIE | LSOIE | WiRe57 | 9.54 | 0.32 | 0.183 | 0.233 | 0.063 | 0.32 | 0.099 | 0.151 | 0.034 | 0.92 | 0.571 | 0.705 | 0.549
CIGL-OIE | LSOIE | WiRe57 | 7.65 | 0.301 | 0.223 | 0.256 | 0.082 | 0.301 | 0.114 | 0.165 | 0.044 | 0.933 | 0.694 | 0.796 | 0.671
OpenIE6 | LSOIE | WiRe57 | 3.81 | 0.311 | 0.247 | 0.275 | 0.114 | 0.311 | 0.194 | 0.239 | 0.086 | 0.766 | 0.688 | 0.725 | 0.554
SpanOIE | IMoJIE | WiRe57 | 7.33 | 0.087 | 0.364 | 0.141 | 0.198 | 0.087 | 0.274 | 0.133 | 0.149 | 0.303 | 0.898 | 0.454 | 0.585
IMoJIE | IMoJIE | WiRe57 | 1.17 | 0.517 | 0.404 | 0.454 | 0.207 | 0.517 | 0.224 | 0.313 | 0.116 | 0.911 | 0.778 | 0.84 | 0.622
Multi2OIE | IMoJIE | WiRe57 | 24.83 | 0.539 | 0.373 | 0.44 | 0.228 | 0.539 | 0.195 | 0.287 | 0.12 | 0.9 | 0.706 | 0.791 | 0.692
IGL-OIE | IMoJIE | WiRe57 | 10.36 | 0.485 | 0.291 | 0.364 | 0.144 | 0.48 | 0.157 | 0.236 | 0.08 | 0.934 | 0.7 | 0.8 | 0.65
CIGL-OIE | IMoJIE | WiRe57 | 7.83 | 0.44 | 0.395 | 0.417 | 0.197 | 0.44 | 0.196 | 0.271 | 0.099 | 0.926 | 0.799 | 0.858 | 0.744
OpenIE6 | IMoJIE | WiRe57 | 4.24 | 0.459 | 0.393 | 0.424 | 0.2 | 0.452 | 0.292 | 0.355 | 0.144 | 0.802 | 0.781 | 0.792 | 0.648

Table 9: A table that lists performances of different OpenIE systems on the
WiRe57 benchmark.

Model | Training set | Test set | Sen./Sec | OIE2016 | WiRe57 | CaRB
---|---|---|---|---|---|---
P | R | F1 | AUC | P | R | F1 | AUC | P | R | F1 | AUC
SpanOIE | SpanOIE | ReOIE2016 | 16.87 | 0.741 | 0.842 | 0.788 | 0.733 | 0.772 | 0.595 | 0.672 | 0.527 | 0.772 | 0.61 | 0.681 | 0.54
IMoJIE | SpanOIE | ReOIE2016 | 2.71 | 0.773 | 0.84 | 0.805 | 0.627 | 0.785 | 0.601 | 0.681 | 0.456 | 0.785 | 0.607 | 0.684 | 0.46
Multi2OIE | SpanOIE | ReOIE2016 | 27.70 | 0.737 | 0.932 | 0.823 | 0.753 | 0.749 | 0.688 | 0.717 | 0.586 | 0.749 | 0.698 | 0.723 | 0.596
IGL-OIE | SpanOIE | ReOIE2016 | 67.16 | 0.762 | 0.784 | 0.773 | 0.653 | 0.756 | 0.557 | 0.641 | 0.455 | 0.756 | 0.569 | 0.649 | 0.465
CIGL-OIE | SpanOIE | ReOIE2016 | 49.33 | 0.688 | 0.991 | 0.812 | 0.733 | 0.437 | 0.663 | 0.527 | 0.35 | 0.437 | 0.69 | 0.535 | 0.365
OpenIE6 | SpanOIE | ReOIE2016 | 22.84 | 0.498 | 0.988 | 0.662 | 0.532 | 0.314 | 0.628 | 0.419 | 0.268 | 0.314 | 0.636 | 0.42 | 0.272
SpanOIE | OIE4 | ReOIE2016 | 16.72 | 0.815 | 0.617 | 0.702 | 0.56 | 0.815 | 0.604 | 0.694 | 0.548 | 0.729 | 0.839 | 0.78 | 0.726
IMoJIE | OIE4 | ReOIE2016 | 3.00 | 0.756 | 0.119 | 0.206 | 0.075 | 0.756 | 0.119 | 0.205 | 0.075 | 0.75 | 0.155 | 0.257 | 0.095
Multi2OIE | OIE4 | ReOIE2016 | 27.74 | 0.813 | 0.647 | 0.72 | 0.561 | 0.813 | 0.635 | 0.713 | 0.55 | 0.773 | 0.869 | 0.818 | 0.746
IGL-OIE | OIE4 | ReOIE2016 | 64.23 | 0.732 | 0.629 | 0.677 | 0.531 | 0.732 | 0.615 | 0.668 | 0.52 | 0.751 | 0.877 | 0.809 | 0.72
CIGL-OIE | OIE4 | ReOIE2016 | 51.78 | 0.698 | 0.697 | 0.698 | 0.582 | 0.698 | 0.675 | 0.686 | 0.564 | 0.74 | 0.948 | 0.831 | 0.776
OpenIE6 | OIE4 | ReOIE2016 | 23.30 | 0.506 | 0.679 | 0.58 | 0.472 | 0.506 | 0.671 | 0.577 | 0.467 | 0.559 | 0.938 | 0.701 | 0.642
SpanOIE | LSOIE | ReOIE2016 | 16.33 | 0.69 | 0.536 | 0.603 | 0.453 | 0.69 | 0.53 | 0.6 | 0.448 | 0.65 | 0.814 | 0.723 | 0.672
IMoJIE | LSOIE | ReOIE2016 | 1.03 | 0.747 | 0.414 | 0.533 | 0.283 | 0.747 | 0.409 | 0.529 | 0.279 | 0.836 | 0.726 | 0.778 | 0.525
Multi2OIE | LSOIE | ReOIE2016 | 31.24 | 0.746 | 0.586 | 0.657 | 0.495 | 0.746 | 0.582 | 0.654 | 0.49 | 0.759 | 0.845 | 0.8 | 0.736
IGL-OIE | LSOIE | ReOIE2016 | 69.48 | 0.626 | 0.472 | 0.538 | 0.325 | 0.626 | 0.453 | 0.525 | 0.312 | 0.742 | 0.786 | 0.763 | 0.602
CIGL-OIE | LSOIE | ReOIE2016 | 53.49 | 0.548 | 0.582 | 0.564 | 0.365 | 0.548 | 0.559 | 0.553 | 0.351 | 0.715 | 0.93 | 0.808 | 0.716
OpenIE6 | LSOIE | ReOIE2016 | 24.94 | 0.374 | 0.574 | 0.453 | 0.281 | 0.374 | 0.562 | 0.45 | 0.275 | 0.518 | 0.924 | 0.664 | 0.53
SpanOIE | IMoJIE | ReOIE2016 | 7.36 | 0.099 | 0.535 | 0.167 | 0.294 | 0.099 | 0.527 | 0.166 | 0.289 | 0.175 | 0.993 | 0.298 | 0.584
IMoJIE | IMoJIE | ReOIE2016 | 1.84 | 0.713 | 0.603 | 0.653 | 0.395 | 0.713 | 0.592 | 0.647 | 0.388 | 0.802 | 0.947 | 0.868 | 0.65
Multi2OIE | IMoJIE | ReOIE2016 | 30.72 | 0.817 | 0.614 | 0.701 | 0.542 | 0.812 | 0.606 | 0.694 | 0.534 | 0.794 | 0.863 | 0.827 | 0.793
IGL-OIE | IMoJIE | ReOIE2016 | 68.80 | 0.728 | 0.53 | 0.614 | 0.42 | 0.728 | 0.508 | 0.599 | 0.403 | 0.799 | 0.817 | 0.808 | 0.644
CIGL-OIE | IMoJIE | ReOIE2016 | 49.48 | 0.674 | 0.622 | 0.647 | 0.464 | 0.671 | 0.579 | 0.621 | 0.431 | 0.796 | 0.919 | 0.853 | 0.723
OpenIE6 | IMoJIE | ReOIE2016 | 26.42 | 0.483 | 0.623 | 0.544 | 0.342 | 0.483 | 0.601 | 0.535 | 0.33 | 0.584 | 0.925 | 0.716 | 0.514

Table 10: A table that lists performances of different OpenIE systems on the
ReOIE2016 benchmark.

Model | Training set | Test set | Sen./Sec | OIE2016 | WiRe57 | CaRB
---|---|---|---|---|---|---
P | R | F1 | AUC | P | R | F1 | AUC | P | R | F1 | AUC
SpanOIE | SpanOIE | CaRB | 17.14 | 0.81 | 0.778 | 0.794 | 0.704 | 0.609 | 0.273 | 0.377 | 0.219 | 0.609 | 0.403 | 0.485 | 0.324
IMoJIE | SpanOIE | CaRB | 3.12 | 0.836 | 0.794 | 0.814 | 0.639 | 0.629 | 0.283 | 0.39 | 0.17 | 0.629 | 0.416 | 0.5 | 0.25
Multi2OIE | SpanOIE | CaRB | 22.39 | 0.826 | 0.878 | 0.851 | 0.793 | 0.59 | 0.315 | 0.411 | 0.22 | 0.609 | 0.458 | 0.523 | 0.326
IGL-OIE | SpanOIE | CaRB | 69.67 | 0.831 | 0.771 | 0.8 | 0.672 | 0.611 | 0.267 | 0.371 | 0.184 | 0.611 | 0.399 | 0.483 | 0.275
CIGL-OIE | SpanOIE | CaRB | 52.62 | 0.789 | 0.986 | 0.876 | 0.818 | 0.379 | 0.331 | 0.354 | 0.148 | 0.379 | 0.508 | 0.434 | 0.228
OpenIE6 | SpanOIE | CaRB | 24.17 | 0.643 | 0.981 | 0.777 | 0.671 | 0.335 | 0.406 | 0.367 | 0.181 | 0.338 | 0.489 | 0.399 | 0.223
SpanOIE | OIE4 | CaRB | 16.92 | 0.646 | 0.413 | 0.503 | 0.339 | 0.646 | 0.28 | 0.39 | 0.23 | 0.804 | 0.777 | 0.79 | 0.701
IMoJIE | OIE4 | CaRB | 3.83 | 0.624 | 0.442 | 0.517 | 0.247 | 0.624 | 0.304 | 0.408 | 0.17 | 0.804 | 0.816 | 0.81 | 0.572
Multi2OIE | OIE4 | CaRB | 33.37 | 0.647 | 0.442 | 0.525 | 0.317 | 0.647 | 0.298 | 0.408 | 0.213 | 0.838 | 0.831 | 0.835 | 0.761
IGL-OIE | OIE4 | CaRB | 72.82 | 0.607 | 0.438 | 0.509 | 0.323 | 0.607 | 0.298 | 0.399 | 0.219 | 0.82 | 0.834 | 0.827 | 0.734
CIGL-OIE | OIE4 | CaRB | 58.49 | 0.584 | 0.479 | 0.526 | 0.35 | 0.584 | 0.326 | 0.418 | 0.237 | 0.814 | 0.908 | 0.858 | 0.796
OpenIE6 | OIE4 | CaRB | 24.93 | 0.518 | 0.482 | 0.499 | 0.346 | 0.518 | 0.395 | 0.448 | 0.281 | 0.685 | 0.903 | 0.779 | 0.716
SpanOIE | LSOIE | CaRB | 16.59 | 0.561 | 0.334 | 0.418 | 0.26 | 0.561 | 0.244 | 0.34 | 0.191 | 0.741 | 0.731 | 0.736 | 0.636
IMoJIE | LSOIE | CaRB | 1.05 | 0.615 | 0.281 | 0.386 | 0.157 | 0.615 | 0.195 | 0.296 | 0.109 | 0.896 | 0.702 | 0.788 | 0.569
Multi2OIE | LSOIE | CaRB | 33.89 | 0.611 | 0.369 | 0.461 | 0.262 | 0.611 | 0.267 | 0.372 | 0.189 | 0.818 | 0.81 | 0.814 | 0.738
IGL-OIE | LSOIE | CaRB | 67.65 | 0.529 | 0.304 | 0.386 | 0.178 | 0.529 | 0.215 | 0.305 | 0.127 | 0.825 | 0.743 | 0.782 | 0.616
CIGL-OIE | LSOIE | CaRB | 49.70 | 0.475 | 0.386 | 0.426 | 0.21 | 0.475 | 0.273 | 0.346 | 0.149 | 0.814 | 0.897 | 0.853 | 0.753
OpenIE6 | LSOIE | CaRB | 28.14 | 0.403 | 0.389 | 0.396 | 0.198 | 0.403 | 0.333 | 0.365 | 0.168 | 0.667 | 0.898 | 0.766 | 0.627
SpanOIE | IMoJIE | CaRB | 7.41 | 0.131 | 0.438 | 0.202 | 0.248 | 0.131 | 0.4 | 0.198 | 0.226 | 0.265 | 0.979 | 0.417 | 0.619
IMoJIE | IMoJIE | CaRB | 1.77 | 0.633 | 0.457 | 0.531 | 0.266 | 0.633 | 0.306 | 0.413 | 0.179 | 0.863 | 0.914 | 0.888 | 0.696
Multi2OIE | IMoJIE | CaRB | 31.22 | 0.648 | 0.418 | 0.508 | 0.301 | 0.645 | 0.28 | 0.39 | 0.201 | 0.848 | 0.813 | 0.83 | 0.771
IGL-OIE | IMoJIE | CaRB | 73.88 | 0.615 | 0.384 | 0.473 | 0.252 | 0.615 | 0.252 | 0.357 | 0.165 | 0.865 | 0.803 | 0.833 | 0.681
CIGL-OIE | IMoJIE | CaRB | 55.01 | 0.574 | 0.437 | 0.496 | 0.274 | 0.563 | 0.286 | 0.379 | 0.178 | 0.855 | 0.909 | 0.881 | 0.768
OpenIE6 | IMoJIE | CaRB | 25.35 | 0.503 | 0.44 | 0.47 | 0.252 | 0.498 | 0.365 | 0.421 | 0.204 | 0.715 | 0.898 | 0.796 | 0.633

Table 11: A table that lists performances of different OpenIE systems on the
CaRB benchmark.

Model | Training set | Test set | Sen./Sec | OIE2016 | WiRe57 | CaRB
---|---|---|---|---|---|---
P | R | F1 | AUC | P | R | F1 | AUC | P | R | F1 | AUC
SpanOIE | SpanOIE | LSOIE | 18.56 | 0.745 | 0.851 | 0.794 | 0.742 | 0.537 | 0.388 | 0.451 | 0.298 | 0.537 | 0.551 | 0.544 | 0.423
IMoJIE | SpanOIE | LSOIE | 2.92 | 0.631 | 0.866 | 0.73 | 0.499 | 0.53 | 0.516 | 0.523 | 0.244 | 0.53 | 0.537 | 0.534 | 0.253
Multi2OIE | SpanOIE | LSOIE | 27.55 | 0.618 | 0.909 | 0.736 | 0.646 | 0.525 | 0.596 | 0.558 | 0.364 | 0.525 | 0.628 | 0.571 | 0.383
IGL-OIE | SpanOIE | LSOIE | 205.07 | 0.636 | 0.815 | 0.714 | 0.582 | 0.529 | 0.484 | 0.505 | 0.295 | 0.529 | 0.506 | 0.517 | 0.308
CIGL-OIE | SpanOIE | LSOIE | 159.43 | 0.634 | 0.975 | 0.769 | 0.653 | 0.379 | 0.601 | 0.464 | 0.284 | 0.379 | 0.633 | 0.474 | 0.3
OpenIE6 | SpanOIE | LSOIE | 59.69 | 0.458 | 0.965 | 0.622 | 0.468 | 0.268 | 0.562 | 0.363 | 0.215 | 0.268 | 0.58 | 0.366 | 0.222
SpanOIE | OIE4 | LSOIE | 19.48 | 0.541 | 0.541 | 0.541 | 0.416 | 0.541 | 0.382 | 0.447 | 0.294 | 0.737 | 0.848 | 0.788 | 0.736
IMoJIE | OIE4 | LSOIE | 3.62 | 0.52 | 0.564 | 0.541 | 0.248 | 0.52 | 0.541 | 0.53 | 0.239 | 0.61 | 0.89 | 0.724 | 0.442
Multi2OIE | OIE4 | LSOIE | 39.12 | 0.547 | 0.547 | 0.547 | 0.327 | 0.547 | 0.517 | 0.532 | 0.309 | 0.642 | 0.877 | 0.742 | 0.637
IGL-OIE | OIE4 | LSOIE | 196.72 | 0.521 | 0.566 | 0.543 | 0.378 | 0.521 | 0.54 | 0.53 | 0.361 | 0.628 | 0.896 | 0.738 | 0.659
CIGL-OIE | OIE4 | LSOIE | 191.90 | 0.505 | 0.621 | 0.557 | 0.414 | 0.505 | 0.587 | 0.543 | 0.392 | 0.617 | 0.945 | 0.747 | 0.692
OpenIE6 | OIE4 | LSOIE | 64.24 | 0.394 | 0.557 | 0.462 | 0.354 | 0.394 | 0.537 | 0.455 | 0.342 | 0.47 | 0.924 | 0.623 | 0.587
SpanOIE | LSOIE | LSOIE | 18.09 | 0.666 | 0.65 | 0.658 | 0.541 | 0.666 | 0.474 | 0.554 | 0.394 | 0.715 | 0.888 | 0.792 | 0.762
IMoJIE | LSOIE | LSOIE | 1.09 | 0.748 | 0.597 | 0.664 | 0.395 | 0.748 | 0.571 | 0.648 | 0.379 | 0.741 | 0.891 | 0.809 | 0.563
Multi2OIE | LSOIE | LSOIE | 37.98 | 0.745 | 0.703 | 0.723 | 0.579 | 0.745 | 0.676 | 0.709 | 0.557 | 0.662 | 0.935 | 0.775 | 0.707
IGL-OIE | LSOIE | LSOIE | 201.64 | 0.697 | 0.65 | 0.673 | 0.515 | 0.697 | 0.611 | 0.652 | 0.485 | 0.679 | 0.891 | 0.771 | 0.651
CIGL-OIE | LSOIE | LSOIE | 183.46 | 0.621 | 0.767 | 0.686 | 0.566 | 0.621 | 0.717 | 0.666 | 0.529 | 0.643 | 0.978 | 0.776 | 0.705
OpenIE6 | LSOIE | LSOIE | 65.63 | 0.438 | 0.75 | 0.553 | 0.447 | 0.438 | 0.723 | 0.546 | 0.428 | 0.473 | 0.954 | 0.633 | 0.529
SpanOIE | IMoJIE | LSOIE | 7.19 | 0.085 | 0.439 | 0.142 | 0.238 | 0.085 | 0.389 | 0.139 | 0.211 | 0.226 | 0.996 | 0.368 | 0.61
IMoJIE | IMoJIE | LSOIE | 2.98 | 0.517 | 0.523 | 0.52 | 0.236 | 0.517 | 0.497 | 0.507 | 0.225 | 0.681 | 0.945 | 0.792 | 0.532
Multi2OIE | IMoJIE | LSOIE | 33.67 | 0.554 | 0.527 | 0.54 | 0.348 | 0.554 | 0.502 | 0.527 | 0.333 | 0.651 | 0.882 | 0.749 | 0.703
IGL-OIE | IMoJIE | LSOIE | 218.05 | 0.517 | 0.472 | 0.493 | 0.256 | 0.517 | 0.443 | 0.477 | 0.241 | 0.691 | 0.863 | 0.767 | 0.567
CIGL-OIE | IMoJIE | LSOIE | 189.39 | 0.489 | 0.551 | 0.518 | 0.286 | 0.489 | 0.503 | 0.496 | 0.262 | 0.678 | 0.934 | 0.785 | 0.6
OpenIE6 | IMoJIE | LSOIE | 66.80 | 0.353 | 0.534 | 0.425 | 0.219 | 0.353 | 0.506 | 0.416 | 0.207 | 0.502 | 0.924 | 0.651 | 0.452

Table 12: A table that lists performances of different OpenIE systems on the
LSOIE benchmark.

Independent Var. | Constants | p-value $\leq 0.05$ | p-value $>0.05$
---|---|---|---
t-score $>0$ | t-score $<0$ | t-score $>0$ | t-score $<0$
non-N-ary model vs. N-ary model | non-N-ary train, N-ary test | 2 | 5 | 3 | 5
N-ary train, N-ary test | 3 | 5 | 1 | 6
non-N-ary train vs. N-ary train | non-N-ary model, N-ary test | 0 | 11 | 0 | 4
N-ary model, N-ary test | 4 | 9 | 0 | 2

Table 13: Statistical significance tests to answer R1. Each number represents
the number of test set and evaluation metric combinations with the
corresponding t-score and p-value. When t-score is greater than 0, non-N-ary
outperforms N-ary, and when t-score is less than 0, N-ary outperforms non-N-
ary.

Independent Var. | Constants | p-value $\leq 0.05$ | p-value $>0.05$
---|---|---|---
t-score $>0$ | t-score $<0$ | t-score $>0$ | t-score $<0$
non-IN model vs. IN model | non-IN train, IN test | 9 | 0 | 2 | 1
IN train, IN test | 0 | 4 | 7 | 1
non-IN train, non-IN test | 0 | 1 | 2 | 0
IN train, non-IN test | 3 | 0 | 0 | 0
non-IN train vs. IN train | non-IN model, IN test | 6 | 6 | 0 | 0
IN model, IN test | 2 | 7 | 0 | 3
non-IN model, non-IN test | 2 | 1 | 0 | 0
IN model, non-IN test | 2 | 0 | 1 | 0

Table 14: Statistical significance tests to answer R2. Each number represents
the number of test set and evaluation metric combinations with the
corresponding t-score and p-value. When t-score is greater than 0, non-IN
outperforms IN, and when t-score is less than 0, IN outperforms non-IN.

Configuration 1 Configuration 2 t-Score p-value Model Sen./Sec Model Sen./Sec
IMoJIE 2.070 Multi2OIE 29.225 -21.621 1.50E-15 IMoJIE 2.070 IGL-OIE 84.072
-5.501 2.63E-05 IMoJIE 2.070 CIGL-OIE 68.800 -4.929 9.31E-05 IMoJIE 2.070
OpenIE6 28.357 -5.813 1.31E-05

Table 15: Statistical significance tests to answer R3 with Generative Model
vs. Non-generative Model independent variable . Sentences per second is
averaged over all training and test sets.