# Recall Them All: Retrieval-Augmented Language Models for
Long Object List Extraction from Long Documents

Sneha Singhania
MPI for Informatics
<EMAIL_ADDRESS>
&Simon Razniewski
MPI for Informatics
<EMAIL_ADDRESS>
&Gerhard Weikum
MPI for Informatics
<EMAIL_ADDRESS>

###### Abstract

Methods for relation extraction from text mostly focus on high precision, at
the cost of limited recall. High recall is crucial, though, to populate long
lists of object entities that stand in a specific relation with a given
subject. Cues for relevant objects can be spread across many passages in long
texts. This poses the challenge of extracting long lists from long texts. We
present the L3X method which tackles the problem in two stages: (1) recall-
oriented generation using a large language model (LLM) with judicious
techniques for retrieval augmentation, and (2) precision-oriented
scrutinization to validate or prune candidates. Our L3X method outperforms
LLM-only generations by a substantial margin.

##  1 Introduction

Motivation and Problem.

Information extraction (IE, for short) is the methodology for distilling
structured information out of unstructured texts. Specifically, relation
extraction aims to yield subject-predicate-object (SPO) triples where S and O
are named entities that stand in a certain relation P. State-of-the-art
methods are based on neural learning, and perform well in terms of precision
but with limited recall Han et al. (2020). Recently, large language models
(LLM) have been studied for this task, with emphasis on long-tail facts, but
they exhibit similar deficits in recall Kandpal et al. (2023); Veseli et al.
(2023); Sun et al. (2023). Moreover, most methods are designed to operate only
on single passages, as classifiers or sequence taggers.

Figure 1: Example for extracting long lists from long texts. For the subject
“Harry Potter”, we aim to extract all 57 friends, appearing throughout the
book series.

In this work, we address the underexplored and unsolved problem that IE faces
with two “longs”: extracting a long list of object entities that stand in a
certain relation to a subject, appearing in long text, such as entire books or
websites with many pages. Examples for this open challenge would be extracting
a complete list of (nearly) all acquisitions and subsidiaries of Alphabet
Inc., identifying all artists who have covered Bob Dylan songs, or finding all
friends of Harry Potter in the Harry Potter book series. Figure 1 illustrates
the task.

This work focuses on extracting long object lists from books, as a
particularly challenging use case, but is also applicable to extraction from
websites.

Approach and Contributions. We devise a novel methodology to address this
“doubly long” challenge. Our method, called L3X (for LM-based Long List
eXtraction), works in two stages:

  * $\bullet$

Stage 1: Recall-oriented Generation
An LLM is prompted with the subject and relation at hand, and asked to
generate a full list of objects, using a variety of prompt formulations. In
addition, we use information retrieval (IR) systems to find promising
candidate passages from long texts and feed them into the LLM prompts. In
contrast to prior works on retrieval-augmented LLMs, we retrieve a large
number of such passages (e.g., 500 for a given SP pair) and judiciously select
the best ones for prompting. Moreover, our method iteratively re-ranks the
passages and re-prompts the LLM, to improve on the initial generation of
objects.

  * $\bullet$

Stage 2: Precision-oriented Scrutinization
Given a high-recall list of object candidates from the first stage, the second
stage uses conservative techniques to corroborate or prune objects. We employ
novel techniques to identify high-confidence objects and their best support
passages, and re-assess lower-confidence candidates against these.

By the inherent trade-off between precision and recall, neither of the two is
a suitable metric for our task, and F1 would merely be a generic compromise.
The task really requires maximizing recall, for an effect on KG population,
with sufficiently high precision, to keep downstream curation efforts
manageable. Therefore, the metric that we aim to optimize is Recall@PrecisionX
(R@Px) where $X$ is the precision target under which we must not fall (e.g.,
$X$ being 50% or, ideally, 80%).

Since we are solving a new task, we constructed a dataset consisting of 10
books or book series, and 8 relations. In experiments with GPT-3.5-turbo Brown
et al. (2020); Ouyang et al. (2022) as underlying LLM, we reach nearly 80%
recall using our passage re-ranking and batching technique and ca. 48% R@P50
and 30% R@P80 through our scrutinizing technique. We also ran GPT-4 OpenAI
(2023) as LLM, but did not observe significant gains; hence our focus on
GPT-3.5, which has much lower computing and energy costs.

Salient contributions of this work are:

  * $\bullet$

the new task of extracting a long list of objects for given subject and
relation, from long documents such as books;

  * $\bullet$

a methodology for this task, based on retrieval-augmented LLMs and combining
IR techniques with LLM generation;

  * $\bullet$

experiments with a new benchmark, showing that L3X outperforms LLM-only
baselines that rely on parametric memory from their pre-traing, and providing
an in-depth analysis of strengths and limitations of different methods.

The new dataset, code, and additional experimental results, will be released
upon publication.

##  2 Related Work

#### Relation Extraction.

A common task in IE is to extract a relation/predicate P that holds between
two given entities S and O, where P comes from a pre-specified set of possible
predicates. State-of-the-art methods (e.g., Han et al. (2020); Wang et al.
(2020); Cabot and Navigli (2021); Xie et al. (2022); Josifoski et al. (2022);
Ma et al. (2023)) are based on deep neural networks, often Transformers. They
typically operate on one passage at a time, as input to a multi-label
classifier or sequence tagger. Aggregating cues from many passages is out of
scope for these methods. The popular benchmark DocRED Yao et al. (2019) solely
requires processing a single paragraph from Wikipedia.

The retrieval component is designed to find only passages with known O values,
as the extractor/classifier requires S and O as inputs. In contrast, our long-
list task takes S and P as input and seeks previously unseen O values as
output. This changes the overarching goal from high-precision classification
to high-recall extraction.

#### Open IE.

Open IE Mausam (2016); Stanovsky et al. (2018); Kolluru et al. (2022) is a
variant where S, P and O are simply surface phrases (without linkage to a
knowledge base). Open IE may provide broader coverage across different
relations, but it is not suitable for populating lists of crisp object
entities for a given relation.

#### LLMs as Knowledge Bases.

Petroni et al. (2019) showed that simple prompts can generate facts of
knowledge-base style. The approach has been expanded, refined, and evaluated
in various ways: incorporating entity identifiers Poerner et al. (2020);
Heinzerling and Inui (2021), automatic learning of prompts Jiang et al.
(2020); Shin et al. (2020); Roberts et al. (2020), prompt tuning Zhong et al.
(2021); Qin and Eisner (2021); Chen et al. (2022), and editing model
parameters De Cao et al. (2021). All these aim at precision, mostly for
popular entities, disregarding recall and the long tail. The LM-KBC challenge
Razniewski et al. (2023) is a benchmark for this setting. Recent studies
indicate that LLMs have major problems with the long tail Kandpal et al.
(2023); Veseli et al. (2023); Sun et al. (2023).

#### Retrieval-Augmented Generation (RAG).

The accuracy of LLM generations can be improved by judiciously retrieving
relevant text snippets that are fed into the LLM as part of in-context prompts
Lewis et al. (2020); Guu et al. (2020). Recent works such as Ram et al.
(2023); Shi et al. (2023) showed improvements in output accuracy. Iteratively
retrieving more text, with interleaved output generation, has been proposed
for QA (e.g., Jiang et al. (2023)). The surveys Cai et al. (2022); Asai et al.
(2023); Wang et al. (2023); Gao et al. (2023) discuss RAG architectures for
improving factuality.

#### Evidence and Factuality.

LLMs can also be harnessed to construct evidence for certain claims, and for
checking factuality with confidence scoring (e.g., Manakul et al. (2023); Min
et al. (2023); Chern et al. (2023); Wang et al. (2023)). It has been observed
that, for some use cases, such factuality checks are more reliable than
generating the underlying facts (e.g., Tian et al. (2023); Guan et al.
(2023)). These techniques tap into external sources, such as Wikipedia
articles, which is not feasible in our setting with long-tail entities in
fiction books.

#### IE from Books.

LLM-supported extraction of information about characters in fiction books has
been pursued by Bamman et al. (2019); Stammbach et al. (2022); Chang et al.
(2023). However, these methods focus on generating a single name from a single
passage.

Figure 2: Overview of the L3X methodology.

##  3 Recall-oriented Generation

Our first, recall-oriented stage comprises several steps; Figure 2 (left side)
gives an overview on the flow between components.

  * 0.

Direct prompting of an LLM, with book title, S and P as input, yields an
initial list of O values.

  * 1.

Retrieval of a large pool of passages from the long text, using a sparse or
dense retriever by searching with S and a set of paraphrases of P.

  * 2.

Re-ranking the passages by various criteria. This uses three complementary
techniques: (a) number of named-entity mentions in a passage, to leverage co-
occurrences of multiple O values for the same predicate (e.g., a passage about
several friends); (b) clustering passages and round-robin selection from each
cluster to diversify the next prompts; (c) pseudo-relevance feedback Zhai
(2008) to amplify the best passages in a refined prompt for the next round.

  * 3.

Batching passages with the same or similar entity references or narratives, so
as to provide semantically coherent inputs to the LLM.

  * 4.

Prompting the LLM again in retrieval-augmented mode with the top-ranked
passages. For recall, this is an ensemble over different choices of passage
batches, and we collect the union of all objects generated by the LLM.

  * 5.

Analyzing the pool of passages, re-prioritizing them by the re-ranking methods
(item 2 above), and iterating the preceding steps.

###  3.1 Passage Retriever

Long texts, like entire books, are chunked into short passages of 15
sentences, up to 1000 characters. This way, sentences with coreferences are
implicitly connected to named entities in their proximity. To fully utilize
this potential, we consider all sentence-overlapping passages. On this large
pool of passages, indexed for efficient retrieval, we experiment with the
following three options.

#### BM25.

We employ the classic yet effective sparse BM25 Robertson and Zaragoza (2009)
scoring model using the Pyserini library Lin et al. (2021).

The query consists of the subject S, the predicate P, and a small set of hand-
crafted paraphrases (2-35) and specific cues for P. For example, for friends,
we expand the basic query to include “companion, supporter, pal, buddy, mate,
…”.

#### Contriever.

This dense retriever Izacard et al. (2022) uses BERT to cast passages and
queries into dense vectors. The query is in the form of a question mentioning
just the S and P labels; paraphrasing is automatically taken into account by
the generated embeddings. Passages are ranked by cosine similarity to the
query vector. We directly use the fine-tuned model from
HuggingFace111huggingface.co/facebook/contriever-msmarco.

#### Embeddings.

As Contriever has a tight limit on the number of tokens in its input (512
tokens at a time), we consider an alternative dense retriever by generating
embeddings with the OpenAI text-embedding-3-large
model222platform.openai.com/docs/guides/embeddings, which takes up to ca. 8000
tokens at a time.

###  3.2 Prompt-based Object Generation

The retrieved top-$k$ passages mention the subject in some form (first name,
last name, or alias) and potentially contain other named entities. There are
many ways to feed the retrieved passages into an LLM Borgeaud et al. (2022);
Lewis et al. (2020); Guu et al. (2020); Ram et al. (2023); Shi et al. (2023).
We incorporate passages in a simple and intuitive way, by appending them into
the prompt context, along with other task details333All the details will be
released upon publication, to perform retrieval-augmented list generation
(L3X-RAG).

#### Basic.

To obtain a long list of objects, we should ideally add all the relevant
passages into the prompt at once; however, LLMs have a limited prompt length.
Hence, we divide the top-$k$ passages (with the highest retriever scores) into
batches of $b$ passages each (e.g., k=20 and b=4 results in 5 batches). We
combine the generated O values from this batch-wise processing by their union,
for high recall.

#### Conv.

We use LLM-only generated objects as a starting point and then the series of
batches to extract more objects for a given SP by synthesizing a follow-up
prompt: for example, a prompt like “list friends of S” is followed by: “list
more friends of S …”, to obtain more O values.

#### Ensemble.

Given the sensitivity of LLM to the choice of in-context prompts Zhao et al.
(2021); Lu et al. (2022); Sclar et al. (2023), we experimented with several
prompt templates and settled on the best one. Since we use the retrievers in
an off-the-shelf manner, we constructed an ensemble of five different query
templates and took the union of the generated O values.

###  3.3 Passage Re-ranking

We explore three re-ranking heuristics to improve the richness and aptness of
top-$k$ passages.

#### Entity Mention Frequency (num):

ordering passages by their counts of named-entity mentions. We detect mentions
of entities (without disambiguation) of the proper type (usually person,
place, or org) using spaCy and a hand-crafted dictionary of alias names for S
and O and paraphrases for P (incl. both nominal and verbal phrases). Passages
with many mentions are prioritized because they could potentially yield
multiple O candidates.

#### Diversification (div):

clustering similar passages, then picking round-robin by rank from all
clusters. The rationale here is that a diverse set of passages may yield
higher recall, whereas highly similar passages might give redundant cues. Our
clustering technique uses the following steps:

  * 1.

Passage embeddings of dimension $d$ are stacked into matrix
$A\in\mathbb{R}^{k\times d}$, and we compute the eigenvalues of $S$ =
$AA^{T}$.

  * 2.

On the eigenvalues of $S$, the Knee Detection algorithm Satopaa et al. (2011)
finds the best number of clusters $c$.

  * 3.

The top-$k$ embeddings are clustered by the K-means algorithm with #clusters =
max(5, $c$).

  * 4.

To re-order the top-$k$ passages, we choose $b$ (batch size) passages from
each cluster, moving round-robin across clusters.

#### Amplification (amp):

query refinement by pseudo-revelance feedback. The best passages likely
contain good cues, expressing the predicate P (in whatsoever surface form).
Thus, by the rationale of pseudo-relevance feedback Zhai (2008), identifying
textually similar passages could be useful for the next round of retrieval.
However, if we prioritize passages that are similar also in terms of named
entities, using these may merely re-produce the same popular objects already
collected. To reconcile the two objectives - similar in cues for P, but
different in entity mentions, we devise the following technique for query
refinement:

  * 1.

We consider the previously generated objects and their support passages:
passages that led the LLM to produce at least one object that appears in the
passage.

  * 2.

The query for further iterations is refined by combining the original query
with the support passages. This re-ranks the passages in the pool, from which
the batch is chosen for the next LLM call.

For BM25, we perform query expansion by using keywords extracted Campos et al.
(2020) from the support passages. We pick $5$ keywords that are most related
to P via embedding similarity, excluding entity mentions, and add them to the
original query. For dense retrievers, we mask out all entity mentions in the
support passages, and then combine the original query embedding and sum of the
support passages’ vectors by a convex combination:
$~{}~{}~{}~{}~{}~{}~{}~{}~{}\textbf{E}(Q^{\prime})=\alpha{\textbf{E}(Q)}+(1-\alpha){\sum_{i=1}^{l}{\textbf{E}(S_{i})}}$
with embeddings E( ) and hyper-parameter $\alpha$.

###  3.4 Passage Batching

When feeding groups of passages into LLM, the default is to combine successive
ranks into small batches, as determined by the (re-)ranker. Alternatively, we
can group passages that form a coherent story so as to aid the LLM in
extracting O values. We devise two criteria for this purpose and batch:

  * $\bullet$

Named Entity Overlap (neo): passages with a large overlap in named entity
mentions;

  * $\bullet$

Passage Similarity (sim): passages whose textual embeddings have a high cosine
similarity.

We compute min-hash sketches for passages with Jaccard similarity for neo and
OpenAI text-embedding model in sim.

Based on these criteria, the batching processes a given priority queue of
passages as follows: for each rank $r$, find the $b-1$ most related passages
on lower ranks, and give this batch to the LLM. Mark all these passages as
“done” and proceed with the next lower rank $r^{\prime}>r$, which is not yet
“done”.

##  4 Precision-oriented Scrutinization

To scrutinize the candidate objects O for a given SP and prune the false
positives, we devise several techniques. The key idea is to identify passages
that clearly reflect SPO triples, and utilize these support passages to rank
(and prune) O values and learn embeddings for the P predicates. Figure 2
(right side) gives a pictorial overview. As a basic ingredient from the first
stage, each batch of passages yields an LLM-generated score for the output
list of O candidates. The total score for this O (for a given SP) can be
computed as a weighted occurrence frequency:
$\texttt{score}(O)=\sum_{\text{batch}_{i}}\exp{(\text{score}_{\text{LLM}}(L_{i}))}\times\textbf{I}_{i}(O)$
where $\textbf{I}_{i}(O)$ is an indicator variable set to 1 if O occurs in the
output list $L_{i}$ for the ith batch of passages and zero otherwise, and
$\text{score}_{\text{LLM}}$ is the log probability. This scoring serves as a
simple baseline for pruning doubtful O values.

###  4.1 Evidence Retrieval

While stage 1 needs to start the retrieval with S and P only, stage 2 has O
candidates at its disposal. So, we can now search the entire book for textual
snippets that indicate SPO together. We do this for each SPO candidate and
call the retrieved top-$s$ results the support passages of SPO. For the search
itself, we build on OpenAI embeddings, comparing the embeddings for passages
against embeddings of the concatenated strings for SPO (along with their alias
names or paraphrases).

The retrieved passages per SPO are also re-ranked by comparing their
embeddings to a vector that encodes all paraphrases for P alone. This aims to
favor cues that indicate relation P for the given SO pair. The embedding of
book passages has two variants:

  * $\bullet$

full: all of SPO and their aliases/paraphrases are considered;

  * $\bullet$

masked: all named-entity mentions except S and O (and their aliases) are
masked out from the passages, so as to focus the embeddings on the language
cues that indicate the predicate P.

###  4.2 Classifiers

For the final scrutinization of O values, we devise several classifiers.

#### Score-based Thresholding (thr):

The O candidates for a given SP are ranked using $\texttt{score}(O)$ and we
accept those that fall within the $t^{th}$ quantile of the cumulative score
distribution, with hyper-parameter $t$ (e.g., set to 0.8).

#### Confidence Elicitation (conf):

We ask the LLM itself for its confidence in the generated O values. For each
[SPO], we include the top-$p$ support passages in their full form (with all
named entities incl. S and O) into an LLM prompt for in-context inference:
“Given this information, is [SPO] a correct statement?”. Note that this is
different from the passage-based extraction of the recall-oriented stage, as
the support passages can be very different from the ones considered in stage
1.

#### Predicate-specific Classifier (pred):

The collection of support passages, for all SO with the same predicate P, can
be utilized to learn an embedding for P cues, sort of a “mini-LM” for each
predicate.

To this end, we identify the top-ranked O values with $\texttt{score}(O)$
above a threshold $\omega$, and retrieve their top-$p$ evidence support
passages. For each O, the top-$p$ passages are cast into embedding vectors.
Then, we combine these per-O vectors via a weighted sum, with
$\texttt{score}(O)$ as weights, to obtain a single P-vector. Each SO pair
under scrutiny is tested by comparing the vector of the top-$p$ support
passages for SPO against the P-vector. This self-supervised classifier accepts
SO if the cosine between the embeddings is above a threshold $\theta$.

#### Discriminative Classifier (dis):

Another way of harnessing the SPO support passages is to train a
discriminative classifier, again in a self-supervised manner. We consider the
ranked list of [SPO] candidates for a given SP and pick:

  * $\bullet$

the top-$q$ high-confidence O candidates

  * $\bullet$

the bottom-$q$ low-confidence O candidates

with $q$ being a hyper-parameter. For each top-$q$ and bottom-$q$ candidate O,
we retrieve top-$p$ support passages, forming one passage pool for the high-
confidence Os and a second pool for the low-confidence Os. In each of these
pools, the passages are cast into embedding vectors using the OpenAI embedding
model (see Section 3.1), and weighted averaged with $\texttt{score}(O)$ to
form $SP_{high}$ and $SP_{low}$ vectors.

Finally, each candidate O for a given SP is classified by whether its own
support-passage vector is closer to the $SP_{high}$ or the $SP_{low}$ vector,
in terms of cosine distance, leading to acceptance or rejection, respectively.

##  5 Experimental Setup

Relation | Type | #S | #O per S
---|---|---|---
range | avg (stddev)
parent | pers$\rightarrow$pers | 84 | 1–4 | 1.9 (0.5)
child | pers$\rightarrow$pers | 44 | 1–9 | 3.2 (2.3)
sibling | pers$\rightarrow$pers | 65 | 1–8 | 3.0 (1.8)
hasPlace | pers$\rightarrow$loc | 68 | 1–50 | 4.6 (7.0)
family | pers$\rightarrow$pers | 72 | 1–48 | 13.3 (10.1)
friend | pers$\rightarrow$pers | 85 | 1–86 | 12.4 (17.7)
opponent | pers$\rightarrow$pers | 77 | 1–61 | 10.0 (11.8)
hasMember | org$\rightarrow$pers | 85 | 1–147 | 15.4 (25.9)

Table 1: Dataset Statistics. #S denotes the no. of unique subjects and #O is
the no. of objects per subject.

#### Dataset.

The task of extracting long O lists from long texts is new, and there is no
suitable benchmark data yet. Book collections like Project Gutenberg Rae et
al. (2020) could be considered, but there is no (easy way of curating) ground-
truth on SPO triples. Therefore, we constructed a new dataset of books and
ground-truth lists of O values associated with SP pairs.

We chose ten popular novels and entire book series (with a total of ca. 16,000
pages)444A Song of Ice and Fire Series, Godfather Series, Harry Potter Series,
Outlander Series, Little Women, Malibu Rising, Pride and Prejudice, Steve
Jobs, The Girl with the Dragon Tattoo, Wuthering Heights, enthusiastically
discussed on community websites like Fandom555https://www.fandom.com,
Cliffnotes666https://www.cliffsnotes.com and
Bookcompanion777https://www.bookcompanion.com.

These fan communities feature extensive lists and infoboxes from which we
derive SPO ground-truth with high confidence. As the same entity appears under
different surface forms in the books, we manually constructed an entity name
dictionary that groups alias names for each distinct entity. We utilize, on a
per book basis, that certain first names, last names, or nicknames are unique.
For example, “Daenerys” is unique, whereas “Targaryen” is highly ambiguous. So
for this entity, aliases include “Daenerys”, “Dany”, “Daenerys Targaryen”,
“Daenerys Stormborn”, but not “Targaryen”. If the LLM generates “Targaryen”
only, this counts as false. This construction was aided by additional
community sources888incl. https://potterdb.com for Harry Potter series,
https://www.reddit.com/r/asoiaf/ for Song of Ice and Fire, and others.

The dataset comprises 580 distinct SP pairs for 8 predicates P. In total, it
covers ca. 4000 entities that appear under ca. 7300 alias names. The S
entities are typically prominent characters in the books, but they are
associated with long O lists mostly consisting of rarely mentioned long-tail
entities.

#### Relation Difficulty.

The chosen 8 predicates P cover 4 easier relations with a relatively limited
number of O values (parent, child, sibling, and hasPlace (i.e., people being
at a place) and 4 harder relations with potentially long O lists (family,
friend, opponent, and hasMember (i.e., members of org. or events). Table 1
gives statistics for our new dataset.

#### Entity Popularity.

For drill-down analysis, we divide the ground-truth O entities into head and
tail groups. There are 4909 unique O, with 25% occurring more than 400 times
in their respective books, and 75% with lower frequency. We refer to these
groups as head and tail entities, respectively.

#### Hyper-Parameters.

Our L3X framework comes with a number of tunable hyper-parameters; Table 2
lists the default values for most experiments. We widely varied these
settings, reporting the most interesting ones in Section 6.3 on sensitivity
studies.

Parameter | Default
---|---
$l$: passage length (#char) | 1000
passage overlap (#char) | 200
#retrieved pool of passages | 500
$k$: top-k passages | 20
$b$: #passages per batch | 2
#clusters for div | 5
#keywords for amp (sparse retriever) | 5
$\alpha$: feedback weight for amp | 0.7
$t$: percentile retained for thr | 0.8
$\omega$: cut-off score for selecting O values for pred | 5
$q$: top-$q$ O candidates for dis | 6
$p$: top-$p$ support passages for conf | 2
$p$: top/bottom-$p$ support passages for pred | 1
$p$: top/bottom-$p$ support passages for dis | 3
$\theta$: acceptance bound for pred | 0.85
Table 2: Hyper-Parameters (for Retriever, Extractor, Scrutinizer)

#### Evaluation Metrics.

To obtain insights into the precision-recall trade-off and to assess the end-
to-end goal of populating a knowledge base with high-quality SPO triples, the
primary metric of interest is recall@precision (R@P), reporting on the most
interesting cases of R@P50 and R@P80 (50% and 80% correctness). Additionally,
we give numbers for absolute precision and recall (before and after
scrutinization). All numbers are micro-averaged (%) over the extracted O
values. Section 6.2 will further drill-down by various criteria, including
results per predicate P and by entity popularity.

Method | Stage 1 | Stage 2
---|---|---
P | R | P | R | R@P50 | R@P80
zero-shot | 45.2 | 33.3 | 43.4 | 31.0 | 19.6 | 15.3
instr | 45.9 | 36.4 | 43.6 | 25.2 | 22.5 | 17.0
instr_conv | 40.2 | 40.5 | 39.3 | 31.2 | 26.6 | 20.4
instr_ensemble | 34.4 | 51.6 | 39.4 | 46.0 | 37.8 | 30.2
instr_conv_ensemble | 27.4 | 48.7 | 30.8 | 43.2 | 36.3 | 26.5

Table 3: Results (%) for LLM Baselines (with 80-percentile thresholding) L3X | Stage 1 | Stage 2
---|---|---
Recall (R) | P | R | R@P50 | R@P80
plain | 77.8 | 9.3 | 75.5 | 45.6 | 30.1
+num | 76.3 | 7.4 | 72.8 | 42.6 | 30.1
+div | 78.8 | 9.3 | 76.2 | 45.4 | 30.2
+amp | 79.8 | 10.0 | 76.3 | 46.5 | 29.8
+neo | 77.7 | 9.6 | 75.7 | 45.8 | 30.1
+sim | 76.2 | 9.9 | 74.2 | 44.3 | 29.4
+num +neo | 75.4 | 7.7 | 72.2 | 42.8 | 29.3
+div +neo | 78.4 | 9.5 | 76.2 | 45.2 | 30.0
+amp +neo | 78.2 | 10.7 | 74.9 | 44.4 | 29.8
+num +sim | 75.2 | 7.9 | 72.7 | 43.1 | 29.6
+div +sim | 76.7 | 16.3 | 66.8 | 44.0 | 29.9
+amp +sim | 75.5 | 10.8 | 71.7 | 42.0 | 27.2
Table 4: Results (%) for L3X Stage 1 Configurations (ranking:{num, div, amp}; batching:{neo, sim}; top-$k$=20 passages; 80-percentile thresholding for stage 2) L3X | metric | thr80 | conf | pred | thr50 + pred | dis | thr50 + dis
---|---|---|---|---|---|---|---
plain | P |  9.3 | 17.4 | 14.4 | 13.7 | 14.0 | 11.9
R | 75.5 | 49.6 | 64.9 | 67.8 | 63.6 | 74.0
R@P30 | 53.8 | 49.6 | 51.6 | 52.1 | 51.0 | 54.8
R@P50 | 45.6 | 33.3 | 44.9 | 45.4 | 44.0 | 45.9
R@P80 | 30.1 | 20.5 | 30.8 | 30.8 | 30.3 | 30.9
amp | P | 10.0 | 21.8 | 15.0 | 14.2 | 14.4 | 13.7
R | 76.3 | 51.5 | 70.5 | 71.6 | 63.9 | 64.1
R@P30 | 53.8 | 28.7 | 53.9 | 53.9 | 51.1 | 56.1
R@P50 | 46.5 | 21.5 | 46.5 | 46.4 | 44.6 | 47.5
R@P80 | 29.8 | 10.6 | 30.3 | 30.3 | 29.9 | 30.5
div+neo | P |  9.5 | 18.1 | 15.8 | 14.0 | 14.2 | 14.2
R | 76.2 | 49.1 | 64.6 | 66.7 | 70.3 | 64.0
R@P30 | 53.9 | 38.7 | 51.3 | 52.0 | 52.9 | 55.3
R@P50 | 45.2 | 32.6 | 44.3 | 44.7 | 45.4 | 46.3
R@P80 | 30.0 | 19.2 | 30.2 | 30.3 | 30.3 | 30.8
Table 5: Results (%) for L3X Stage 2 Configurations.

##  6 Results

We present our findings on L3X with Contriever for passage retrieval and
configurations with conv-ensemble prompting. These are the most insightful
cases.

###  6.1 Main Findings

#### Stage 1: Recall-oriented Extractor.

As a reference point, Table 3 gives results for different configurations of
LLM-only generations/extractions. We observe recall up to ca. 50%, with
precision between 30% and 40%. After scrutinization by thresholding on the
LLM-provided aggregated scores for each O, these baselines reach about 38% for
R@P80 and 30% for R@P50. This shows the limits of parametric memory from the
LLM training.

Table 4 shows the results for various L3X configurations, with different
techniques for prompting, passage ranking, and batching. Again, the
scrutinization uses default 80-percentile thresholding on the LLM-score-based
ranking of O values. We make the following key observations:

  * $\bullet$

With increasing number of passages $k$, stage-1 recall increases strongly,
reaching up to ca. 75% with the plain L3X configuration (no ranking, no
batching), compared to a maximum of ca. 50% for LLM baselines without
passages.

  * $\bullet$

The different L3X configurations are close to each other, with the +amp
technique achieving the highest recall, ca. 2% points better than plain L3X.

  * $\bullet$

All numbers reflect the inherent precision-recall trade-off: gains in recall
are paid by lower precision. Nevertheless, the +amp configuration also stands
out by the highest R@P50 value, ca. 1% point better than plain L3X.

  * $\bullet$

To get insight into the quality of the retrieved passages, we inspected all
passages for samples of SP pairs, and estimated the ideal recall that a smart
human would obtain by reading all passages. On average, this would be around
90%. So the L3X stage-1 recall of ca. 80% is actually reasonably close to the
best possible.

A closer inspection of the generated O lists and their underlying passages
reveals that the LLM is still prone to hallucinations. It sometimes generates
O values that are not present in the input passages at all, obviously relying
on its parametric memory. Most of the books in our dataset have been heavily
discussed in online media, and some have been made into movies or TV series.
When the LLM generates O values it implicitly taps into this content, at the
risk of latent mash-up with incorrect outputs. For recall it is actually
useful, but puts more pressure on the scrutinization stage.

In the following, the presentation is restricted to a subset of stage-1
configurations, namely, plain L3X, L3X +amp (the best performing) and L3X
+div+neo (the most promising one with a combination of ranking and batching).

#### Stage 2: Precision-oriented Scrutinizer.

Table 5 shows the stage-2 results with different scrutinization methods. For
the classifiers that use support passages, we report only the mode with named-
entity masking, as it consistently performed better. In addition to the base
techniques thr, conf, pred and dis, we also studied combinations of thr and
the others. This works by accepting the top-ranked O values from thr50, where
we keep the highest-scoring O values up to the 50-percentile in score mass,
and running the other classifiers on the remaining uncertain candidates.

We highlight the following key findings:

  * $\bullet$

Thresholding (thr) works amazingly well, with 80-percentile cut-off. Note that
this is the percentile for the score-mass distribution; the simpler techniques
with a cut-off at a certain rank or an absolute score are far inferior.

  * $\bullet$

Confidence elicitation (conf) from the LLM itself performs very poorly. When
fed with support passages about SPO candidates, the LLM becomes rather
conservative and rejects too many candidates.

  * $\bullet$

The classifiers that harness support passages about SPO candidates from
evidence retrieval, pred and dis, are viable options, but do not achieve
substantial gains over thr80.

  * $\bullet$

When combining pred or dis with keeping the O values from thresholding at the
50-percentile (i.e., keeping only the better part of what thr80 yields),
particularly thr50+dis provides notable gains, in the order of 2% points in
R@P.

Overall, we observe notable gains by the smarter variants of L3X
scrutinization, but the gains over thresholding on the score-mass distribution
are not as high as we expected. One hypothesis about this is that global
settings for the hyper-parameters $q$, $p$ and $\theta$ (see Table 2) are not
sufficient to reflect the big differences between the score-mass distributions
of the different predicates. This would call for P-specific tuning of hyper-
parameters, which would need withheld training data or human guidance—a breach
with the L3X rationale of being entirely unsupervised (or self-supervised).

###  6.2 Analysis by Drill-Down

The reported results so far are micro-averages across all SPO outputs.
However, some relations P are easier to deal with than others (see Section 5).
Table 6 shows results with drill-down by predicate. We observe that L3X+amp
wins over plain L3X on practically all relations. A striking effect is that in
recall alone (stage 1 columns), L3X+amp substantially gains for some P,
particularly friend, but substantially loses on other P, particularly
opponent. After scrutinization with thr80, these differences become smaller,
though. We believe the stage-1 effect comes from the different complexity of
paraphrases and other cues for the various predicates. Friend has a crisp set
of strongly related words (e.g., companion, buddy, mate etc.), whereas books
often express the opponent relation in a very implicit form, such as hostile
or offensive conversations between characters.

Another refined view is by breaking down the results by head entities vs. tail
entities (see Section 5). Table 7 shows these results in combination with easy
vs. hard predicates. We see that the L3X+amp configuration generates 3% more
tail entities in both predicate groups, and L3X+div+neo has the highest yield
on head entities for easy predicates.

###  6.3 Sensitivity w.r.t. Hyper-Parameters

We performed extensive experiments by varying the hyper-parameters. We limit
ourselves here to reporting numbers for the sensitivity of the stage-1
parameters for $k$ (no. top-$k$ passages), $l$ (passage length), and $b$ (no.
passages per batch). Figure 3 plots curves for selected choices of $l/b$
against varying $k$. We observe massive gains in recall when increasing $k$,
up to a certain saturation level. The L3X+amp configuration slightly
outperforms the others. Increasing $l$ beyond 1000 characters is not
beneficial, as overly long passages become too diverse in content. Likewise,
batching more than 2 passages together does not pay off, as it is often
difficult to identify 4 or more passages that are coherent in their story
about the same SO pair.

Figure 3: Sensitivity w.r.t. Hyper-parameters $k$, $l$, $b$. L3X | stage 1: recall | plain | amp | div+neo
---|---|---|---|---
plain | amp | div+neo | R@P50 | R@P80 | R@P50 | R@P80 | R@P50 | R@P80
parent | 86.2 | 87.4 | 86.9 | 66.0 | 54.7 | 69.5 | 59.3 | 65.7 | 54.4
child | 78.2 | 82.4 | 82.2 | 53.1 | 36.6 | 56.3 | 39.4 | 54.3 | 37.4
sibling | 85.1 | 84.5 | 87.4 | 70.9 | 52.4 | 70.7 | 54.2 | 73.2 | 55.0
hasPlace | 79.8 | 84.7 | 80.8 | 39.2 | 25.2 | 39.0 | 20.9 | 37.6 | 22.1
avg. Easy P | 82.3 | 84.8 | 84.3 | 57.3 | 42.2 | 58.9 | 43.5 | 57.7 | 42.2
family | 80.9 | 83 | 79.7 | 45.6 | 28.5 | 48.4 | 29.2 | 46.9 | 29.6
friend | 74.1 | 79.3 | 73.3 | 26.0 | 14.5 | 26.1 | 12.9 | 25.0 | 12.7
opponent | 68.8 | 64.1 | 67.2 | 30.1 | 17.8 | 32.4 | 15.6 | 33.4 | 17.8
hasMember | 69.5 | 73 | 70.2 | 36.2 | 17.5 | 37.9 | 12.6 | 34.3 | 17.3
avg. Hard P | 73.3 | 74.8 | 72.6 | 34.5 | 19.6 | 36.2 | 17.6 | 34.9 | 19.4
avg. All P | 77.8 | 79.8 | 78.4 | 45.9 | 30.9 | 47.5 | 30.5 | 46.3 | 30.8
Table 6: Drill-Down Results (Stage 1 and after Stage 2) by Predicate Popularity | plain | amp | div+neo
---|---|---|---
easy P | head O | 96.2 | 95.1 | 97.6
#0: 171
tail O | 78.1 | 81.6 | 79.9
#0: 642
hard P | head O | 90.9 | 90.7 | 90.8
#O: 969
tail O | 65.1 | 68.3 | 64.7
#O: 3127
Table 7: Recall Results on Head-vs-Tail Objects

###  6.4 Discussion

#### Anecdotal Error Analysis.

If the book is widely discussed on the web, one would expect LLMs to
accurately generate objects for easy predicates and head entities. However,
social media content does not always align with ground truth, and LLMs are
still prone to hallucinations. Even in the simple example of asking for family
of Harry Potter, the LLM repeatedly returns “Albus Dumbledore” (his teacher,
not family), despite prompts clearly defining what constitutes a family
member.

On the other hand, the LLM can generate complete names for person-type
objects, even without mentions in the input passages, by associating with the
family name of the subject. This may be seen as an unfaithful output, but it
often includes true positives, and does contribute to higher recall.

Another failure case with LLM hallucination is the mash-up of the first and
last names of two prominent characters. For example, the leading character
“Lisbeth Salander” (from “Girl with the Dragon Tattoo”) was generated as
“Lisbeth Blomkvist”, the child of “Mikael Blomkvist” (the other protagonist).

The object lists generated by L3X are often long, mostly with true positives
(TPs) in the higher ranks and many false positives (FPs) in the lower ranks.
However, there are also cases where a good number of TPs are spread in the
tail of the ranking. E.g., for subject “Stannis Baratheon” (from “A Song of
Ice and Fire”) and opponent relation, the ground truth has 60 objects. L3X+amp
generates a much longer list of 140 objects. For good R@P, it’s essential to
identify the spread-out TPs in the lower ranks, while still pruning the
interleaving FPs. L3X achieves some of this, but there is much room for
improvement.

#### Comparison of Retrievers.

We solely presented results with Contriever for passage search. Additional
experiments with BM25 and OpenAI embeddings showed that all three retrievers
gave similar results, with the two dense retrievers being slightly superior
and Contriever being best.

#### Comparison to Relational IE.

We ran the full task with two state-of-the-art methods for relational IE from
documents: GenIE Josifoski et al. (2022) and DREEAM Ma et al. (2023). Both
performed very poorly, reaching less than 5% recall and precision 10% at best
(with various configurations). Clearly, our task is outside their comfort
zone. We find O for a given SP, whereas prior methods label given SO pairs
with P. We perform unsupervised extraction, whereas prior methods rely on
extensive training from Wikipedia paragraphs. This underlines the uniqueness
and challenging nature of the proposed long-lists-from-long-documents task.

#### Cost/Benefit Ratio.

All our methods harness repeated LLM prompts in an ensemble manner, with
different prompts and different batches of passages. This incurs costs per API
call and per input token—computational, environmental, and monetary costs. Our
hyper-parameter settings for $k$ and $l$ were partly driven by this aspect. A
small number of $k$ passages is clearly insufficient, but we did not exercise
$k$ larger than 20 (other than for the sensitivity study). A deeper look into
cost/benefit optimization is the subject of future work.

##  7 Conclusion

We introduced the task of extracting long lists of objects from long
documents, and developed the L3X methodology, comprising LLM prompting,
retrieval augmentation, passage ranking and batching, and classifiers to
scrutinize candidates and prune false positives. Extensive experiments with a
range of L3X configurations over a dataset of full-length books provide key
insights. First, L3X substantially outperforms LLM-only extraction in recall
and R@P. Second, techniques for ranking, batching, evidence retrieval and
scrutinizing show promising trends, but are not (yet) major game-changers.
Third, this underlines the challenge of the new task: passages scattered
across long books give cues to a smart human, but are still very hard to
pinpoint and extract for AI systems (incl. LLMs).

##  8 Limitations

This work aims to reconcile high recall with good precision. However, this is
a fundamental trade-off. Our approach, like all others, is a judicious
compromise. L3X hinges on and augments LLM technology. Our choice of GPT-3.5
is for convenience; reliable open LLMs without per-token payment are a
desirable route to explore.

In the absence of suitable datasets, we constructed a new benchmark
resource—limited in scale, though. Expanding the dataset would be useful, and
additional data from web pages could be pursued as well.

## References

  * Asai et al. (2023) Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023.  Retrieval-based language models and applications.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts)_ , pages 41–46, Toronto, Canada. Association for Computational Linguistics.  Tutorial materials at "http://acl2023-retrieval-lm.github.io/".
  * Bamman et al. (2019) David Bamman, Sejal Popat, and Sheng Shen. 2019.  An annotated dataset of literary entities.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2138–2144, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2206–2240. PMLR.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165.
  * Cabot and Navigli (2021) Pere-Lluís Huguet Cabot and Roberto Navigli. 2021.  REBEL: relation extraction by end-to-end language generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , pages 2370–2381. Association for Computational Linguistics.
  * Cai et al. (2022) Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. 2022.  Recent advances in retrieval-augmented text generation.  In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022_ , pages 3417–3419. ACM.  Tutorial materials at "https://jcyk.github.io/RetGenTutorial/".
  * Campos et al. (2020) Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. 2020.  Yake! keyword extraction from single documents using multiple local features.  _Information Sciences_ , 509:257–289.
  * Chang et al. (2023) Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023.  Speak, memory: An archaeology of books known to ChatGPT/GPT-4.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7312–7327, Singapore. Association for Computational Linguistics.
  * Chen et al. (2022) Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022.  Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction.  In _WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022_ , pages 2778–2788. ACM.
  * Chern et al. (2023) I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. 2023.  Factool: Factuality detection in generative AI - A tool augmented framework for multi-task and multi-domain scenarios.  _CoRR_ , abs/2307.13528.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6491–6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023.  Retrieval-augmented generation for large language models: A survey.  _CoRR_ , abs/2312.10997.
  * Guan et al. (2023) Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. 2023.  Language models hallucinate, but may excel at fact verification.  _CoRR_ , abs/2310.14564.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning_ , ICML’20. JMLR.org.
  * Han et al. (2020) Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Jie Zhou, and Maosong Sun. 2020.  More data, more relations, more context and more openness: A review and outlook for relation extraction.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, AACL/IJCNLP 2020, Suzhou, China, December 4-7, 2020_ , pages 745–758. Association for Computational Linguistics.
  * Heinzerling and Inui (2021) Benjamin Heinzerling and Kentaro Inui. 2021.  Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1772–1791, Online. Association for Computational Linguistics.
  * Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.  Unsupervised dense information retrieval with contrastive learning.  _Transactions on Machine Learning Research_.
  * Jiang et al. (2023) Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.  Active retrieval augmented generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7969–7992, Singapore. Association for Computational Linguistics.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know.  _Trans. Assoc. Comput. Linguistics_ , 8:423–438.
  * Josifoski et al. (2022) Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, and Robert West. 2022.  GenIE: Generative information extraction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4626–4643, Seattle, United States. Association for Computational Linguistics.
  * Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023.  Large language models struggle to learn long-tail knowledge.  In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 15696–15707. PMLR.
  * Kolluru et al. (2022) Keshav Kolluru, Muqeeth Mohammed, Shubham Mittal, Soumen Chakrabarti, and Mausam. 2022.  Alignment-augmented consistent translation for multilingual open information extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 2502–2517. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Lin et al. (2021) Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021.  Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 2356–2362, New York, NY, USA. Association for Computing Machinery.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Ma et al. (2023) Youmi Ma, An Wang, and Naoaki Okazaki. 2023.  DREEAM: Guiding attention with evidence for improving document-level relation extraction.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1971–1983, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pages 9004–9017. Association for Computational Linguistics.
  * Mausam (2016) Mausam. 2016.  Open information extraction systems and downstream applications.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016_ , pages 4074–4077. IJCAI/AAAI Press.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pages 12076–12100. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Poerner et al. (2020) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020.  E-BERT: Efficient-yet-effective entity embeddings for BERT.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 803–818, Online. Association for Computational Linguistics.
  * Qin and Eisner (2021) Guanghui Qin and Jason Eisner. 2021.  Learning how to ask: Querying lms with mixtures of soft prompts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 5203–5212. Association for Computational Linguistics.
  * Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020.  Compressive transformers for long-range sequence modelling.  In _International Conference on Learning Representations_.
  * Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023.  In-context retrieval-augmented language models.  _Transactions of the Association for Computational Linguistics_ , 11:1316–1331.
  * Razniewski et al. (2023) Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, and Jeff Z. Pan, editors. 2023.  _Joint proceedings of the 1st workshop on Knowledge Base Construction from Pre-Trained Language Models (KBC-LM) and the 2nd challenge on Language Models for Knowledge Base Construction (LM-KBC) co-located with the 22nd International Semantic Web Conference (ISWC 2023), Athens, Greece, November 6, 2023_, volume 3577 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Satopaa et al. (2011) Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. 2011.  Finding a "kneedle" in a haystack: Detecting knee points in system behavior.  In _2011 31st International Conference on Distributed Computing Systems Workshops_ , pages 166–171.
  * Sclar et al. (2023) Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023.  Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.
  * Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2023.  Replug: Retrieval-augmented black-box language models.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online. Association for Computational Linguistics.
  * Stammbach et al. (2022) Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022.  Heroes, villains, and victims, and GPT-3: Automated extraction of character roles without training data.  In _Proceedings of the 4th Workshop of Narrative Understanding (WNU2022)_ , pages 47–56, Seattle, United States. Association for Computational Linguistics.
  * Stanovsky et al. (2018) Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018.  Supervised open information extraction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , pages 885–895. Association for Computational Linguistics.
  * Sun et al. (2023) Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2023.  Head-to-tail: How knowledgeable are large language models (llm)? A.K.A. will llms replace knowledge graphs? _CoRR_ , abs/2308.10168.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. 2023.  Fine-tuning language models for factuality.  _CoRR_ , abs/2311.08401.
  * Veseli et al. (2023) Blerta Veseli, Simon Razniewski, Jan-Christoph Kalo, and Gerhard Weikum. 2023.  Evaluating the knowledge base completion potential of GPT.  In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_ , pages 6432–6443. Association for Computational Linguistics.
  * Wang et al. (2023) Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Jiayang Cheng, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. 2023.  Survey on factuality in large language models: Knowledge, retrieval and domain-specificity.  _CoRR_ , abs/2310.07521.
  * Wang et al. (2020) Difeng Wang, Wei Hu, Ermei Cao, and Weijian Sun. 2020.  Global-to-local neural networks for document-level relation extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 3711–3721. Association for Computational Linguistics.
  * Xie et al. (2022) Yiqing Xie, Jiaming Shen, Sha Li, Yuning Mao, and Jiawei Han. 2022.  Eider: Empowering document-level relation extraction with efficient evidence extraction and inference-stage fusion.  In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 257–268. Association for Computational Linguistics.
  * Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019.  Docred: A large-scale document-level relation extraction dataset.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 764–777. Association for Computational Linguistics.
  * Zhai (2008) ChengXiang Zhai. 2008.  Statistical language models for information retrieval: A critical review.  _Found. Trends Inf. Retr._ , 2(3):137–213.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 12697–12706. PMLR.
  * Zhong et al. (2021) Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021.  Factual probing is [MASK]: learning vs. learning to recall.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 5017–5033. Association for Computational Linguistics.

## Appendix A: Query and Prompt Templates

Prompt: in-context
---
Task Description:
You are a knowledgebase. Generate the list of OBJECTS (people or persons,
essentially PERSON-type named entities) in comma-separated form for the given
BOOK, SUBJECT and RELATION.
Examples:
BOOK: Percy Jackson & The Olympians, SUBJECT: Percy Jackson, RELATION: friend,
OBJECTS: [Blackjack, Mrs. O’Leary, Grover Underwood, Jason Grace, Thalia
Grace, Nico di Angelo, Chiron, Clarisse La Rue, Zoë Nightshade, Charles
Beckendorf, Silena Beauregard, Bianca di Angelo, Hazel Levesque, Reyna
Ramírez-Arellano, Dakota, Leo Valdez, Piper McLean, Gleeson Hedge]
BOOK: To Kill a Mockingbird, SUBJECT: Scout Finch, RELATION: friend, OBJECTS:
[Jem Finch, Dill Harris, Atticus Finch, Calpurnia]
BOOK: ##book##, SUBJECT: ##subject##, RELATION: friend, OBJECTS:
Table 8: LLM-only prompt templates with in-context examples. For zero-shot,
only the task description is given. Prompt
---
Task Description:
You are a knowledgebase. Generate the list of OBJECTS (people or persons,
essentially PERSON-type named entities) in comma-separated form for the given
BOOK, SUBJECT and RELATION.
Examples:
BOOK: Percy Jackson & The Olympians, SUBJECT: Percy Jackson, RELATION: friend,
OBJECTS: [Blackjack, Mrs. O’Leary, Grover Underwood, Jason Grace, Thalia
Grace, Nico di Angelo, Chiron, Clarisse La Rue, Zoë Nightshade, Charles
Beckendorf, Silena Beauregard, Bianca di Angelo, Hazel Levesque, Reyna
Ramírez-Arellano, Dakota, Leo Valdez, Piper McLean, Gleeson Hedge]
BOOK: To Kill a Mockingbird, SUBJECT: Scout Finch, RELATION: friend, OBJECTS:
[Jem Finch, Dill Harris, Atticus Finch, Calpurnia]
BOOK: ##book##, SUBJECT: ##subject##, RELATION: friend, OBJECTS:
Assistant:
##output##
Follow-up prompt
Generate more correct OBJECTS using the SUPPORTING PASSAGES from the book
(some of which might be irrelevant).
SUPPORTING PASSAGES:
##passages##
BOOK: ##book##, SUBJECT: ##subject##, RELATION: friend, OBJECTS:
Table 9: L3X-RAG conv template for friend relation. The placeholders are given
with ##. Dense Retriever (contriever-msmarco)
---
Relation |  Template
parent |  Who are the parents of ##subj##?
parents of ##subj##
father and mother of ##subj##
Who are the father and mother of ##subj##?
Who are the parents, including adopted parents and foster parents, of
##subj##?
children |  Who are the children of ##subj##?
Who are the children, including son and daughter, of ##subj##?
##subj## children child son daughter
##subj## children child baby kid
##subj## son daughter
sibling |  Who are the siblings of ##subj##?
siblings of ##subj##
Who are the brothers and sisters of ##subj##?
brothers and sisters of ##subj##
Who are the siblings, including stepbrother, stepsister, half-brother and
half-sister, of ##subj##?
has-place |  Where did ##subj## live or travel or go for a visit?
Where did ##subj## travel or go for a vacation?
Where did ##subj## travel?
Where is the residence of ##subj##?
residence or place of stay of ##subj##
family |  Who are the family members of ##subj##?
family members of ##subj##
Who are the family members of ##subj##, including parent, father, mother, son,
daughter, sibling, brother, sister, child, aunt, uncle, auntie, cousin,
nephew, niece, grandfather, grandmother, grandson, granddaughter, father-in-
law, mother-in-law, son-in-law, daughter-in-law, brother-in-law, sister-in-
law, stepfather, stepmother, stepson, stepdaughter, stepbrother, stepsister,
half brother, half sister, godfather, godmother?
Who are the relatives of ##subj##, including parent, father, mother, son,
daughter, sibling, brother, sister, child, aunt, uncle, auntie, cousin,
nephew, niece, grandfather, grandmother, grandson, granddaughter, father-in-
law, mother-in-law, son-in-law, daughter-in-law, brother-in-law, sister-in-
law, stepfather, stepmother, stepson, stepdaughter, stepbrother, stepsister,
half brother, half sister, godfather, godmother?
Who are the relatives of ##subj##?
friend |  Who are the friends of ##subj##?
friends of ##subj##
Who are the friends or supporters of ##subj##?
friends or supporters of ##subj##
Who are the friends, including companion, pal, supporter, buddy, ally, bestie,
and follower, of ##subj##?
opponent |  Who are the enemies of ##subj##?
enemies of ##subj##
Who are the enemies or rivals of ##subj##?
enemies or rivals of ##subj##
Who are the enemies, including rival, attacker, opponent, foe, nemesis and
critic, of ##subj##?
has-member |  Who are the members of ##subj##?
Who are part of ##subj##?
##subj## member teammate group part of champions followers belongs member
##subj## member group family part of
##subj## member teammate group
Table 10: Retriever query templates.