# Query Expansion by Prompting Large Language Models

Rolf Jagerman  0000-0002-5169-495X Google Research<EMAIL_ADDRESS>,
Honglei Zhuang  0000-0001-8134-1509 Google Research<EMAIL_ADDRESS>,  Zhen Qin
0000-0001-6739-134X Google Research<EMAIL_ADDRESS>,  Xuanhui Wang
0009-0000-1388-1423 Google Research<EMAIL_ADDRESS>and  Michael Bendersky
0000-0002-2941-6240 Google Research<EMAIL_ADDRESS>
(2023)

###### Abstract.

Query expansion is a widely used technique to improve the recall of search
systems. In this paper, we propose an approach to query expansion that
leverages the generative abilities of Large Language Models (LLMs). Unlike
traditional query expansion approaches such as Pseudo-Relevance Feedback (PRF)
that relies on retrieving a good set of pseudo-relevant documents to expand
queries, we rely on the generative and creative abilities of an Large Language
Model (LLM) and leverage the knowledge inherent in the model. We study a
variety of different prompts, including zero-shot, few-shot and Chain-of-
Thought (CoT). We find that CoT prompts are especially useful for query
expansion as these prompts instruct the model to break queries down step-by-
step and can provide a large number of terms related to the original query.
Experimental results on MS-MARCO and BEIR demonstrate that query expansions
generated by LLMs can be more powerful than traditional query expansion
methods.

††journalyear: 2023††copyright: acmlicensed

##  1\. Introduction

Query expansion is a widely used technique that improves the recall of search
systems by adding additional terms to the original query. The expanded query
may be able to recover relevant documents that had no lexical overlap with the
original query. Traditional query expansion approaches are typically based on
Pseudo-Relevance Feedback (PRF) (Rocchio Jr, 1971; Robertson and Jones, 1976;
Robertson, 1990; Amati and Van Rijsbergen, 2002), which treats the set of
retrieved documents from the original query as “pseudo-relevant” and uses
those documents’ contents to extract new query terms. However, PRF-based
approaches assume that the top retrieved documents are relevant to the query.
In practice the initial retrieved documents may not be perfectly aligned with
the original query, especially if the query is short or ambiguous. As a
result, PRF-based approaches may fail if the initial set of retrieved
documents is not good enough.

In this paper we propose the use of Large Language Models (LLMs) (Devlin et
al., 2018; Raffel et al., 2020; Brown et al., 2020) to aid in query expansion.
LLMs have seen a growing interest in the Information Retrieval (IR) community
in recent years. They exhibit several properties, including the ability to
answer questions and generate text, that make them powerful tools. We propose
using those generative abilities to generate useful query expansions. In
particular we investigate ways to prompt an LLM and have it generate a variety
of alternative and new terms for the original query. This means that, instead
of relying on the knowledge within PRF documents or lexical knowledge bases,
we rely on the knowledge inherent in the LLM. An example of the proposed
methodology is presented in Figure 1.

Our main contributions in this work are as follows: First, we formulate
various prompts to perform query expansion (zero-shot, few-shot and CoT) with
and without PRF to study their relative performance. Second, we find that
Chain-of-Thought (CoT) prompts perform best and hypothesize that this is
because CoT prompts instruct the model to break its answer down step-by-step
which includes many keywords that can aid in query expansion. Finally, we
study the performance across various model sizes to better understand the
practical capabilities and limitations of an LLM approach to query expansion.

Answer the following question: {query}
Give the rationale before answering Large Language Model (LLM) Concat({query},
{model output}) Retrieval System (BM25)

Figure 1. High-level overview of using a zero-shot Chain-of-Thought (CoT)
prompt to generate query expansion terms.

[A high-level overview of using a CoT prompt to generate query
expansions.]Four boxes on top of each other connected by arrows. The first box
contains the prompt ”Answer the following question: question Give the
rationale before answering”. The second box represents a Large Language Model
(LLM). The third boxes contains the text ’Concat(query, model output).
Finally, the fourth box contains the text ”Retrieval System (BM25)”.

##  2\. Related Work

Query expansion is widely studied (Efthimiadis, 1996; Carpineto and Romano,
2012). At its core, query expansion helps retrieval systems by expanding query
terms into new terms that express the same concept or information need,
increasing the likelihood of a lexical match with documents in the corpus.
Early works on query expansion focused on either using lexical knowledge bases
(Bhogal et al., 2007; Voorhees, 1994; Qiu and Frei, 1993) or Pseudo-Relevance
Feedback (PRF) (Robertson, 1990; Amati and Van Rijsbergen, 2002; Rocchio Jr,
1971). PRF-based approaches are particularly useful in practice because they
do not need to construct a domain-specific knowledge base and can be applied
to any corpus. Orthogonal to query expansion is _document expansion_ (Tao et
al., 2006; Efron et al., 2012; Nogueira et al., 2019; Zheng et al., 2020)
which applies similar techniques but expands document terms during indexing
instead of query terms during retrieval.

Recent works on query expansion have leveraged neural networks to generate or
select expansion terms (Imani et al., 2019; Roy et al., 2016; Zheng et al.,
2020, 2021), generally by either training or fine-tuning a model. In contrast,
our work leverages the abilities inherent in _general-purpose_ LLMs without
needing to train or fine-tune the model.

We note that our work is similar to the recent works of (Claveau, 2021) and
(Wang et al., 2023): leveraging an LLM to expand a query. However, we
differentiate our work in several important ways: First, we study a number of
different prompts whereas (Wang et al., 2023) focuses on a single few-shot
prompt and (Claveau, 2021) does not study prompts. Second, unlike (Wang et
al., 2023) and (Claveau, 2021), we focus on generating _query expansion terms_
instead of entire pseudo documents. To this end, we demonstrate the
performance of our prompts on a variety of _smaller_ model sizes which helps
understand both the limitations and the practical capabilities of an LLM
approach to query expansion. Finally, we experiment with entirely open-source
models, inviting reproducibility and openness of research, while (Wang et al.,
2023) experiments with a single type of model which is only accessible through
a third-party API.

##  3\. Methodology

We formulate the query expansion problem as follows: given a query $q$ we wish
to generate an _expanded query_ $q^{\prime}$ that contains additional query
terms that may help in retrieving relevant documents. In particular we study
the use of an LLM to expand the query terms and generate a new query
$q^{\prime}$. Since the LLM output may be verbose, we repeat the original
query terms 5 times to upweigh their relative importance. This is the same as
the trick employed by (Wang et al., 2023). More formally:

(1)
$q^{\prime}=\operatorname{Concat}(q,q,q,q,q,\operatorname{LLM}(\textit{prompt}_{q})),$

where $\operatorname{Concat}$ is the string concatenation operator, $q$ is the
original query, $\operatorname{LLM}$ is a Large Language Model and
$\textit{prompt}_{q}$ is the generated prompt based on the query (and
potentially side information like few-shot examples or PRF documents).

In this paper we study eight different prompts:

  * Q2D

The Query2Doc (Wang et al., 2023) few-shot prompt, asking the model to write a
passage that answers the query.

  * Q2D/ZS

A zero-shot version of Q2D.

  * Q2D/PRF

A zero-shot prompt like Q2D/ZS but which also contains extra context in the
form of top-3 retrieved PRF documents for the query.

  * Q2E

Similar to the Query2Doc few-shot prompt but with examples of query _expansion
terms_ instead of _documents_.

  * Q2E/ZS

A zero-shot version of Q2E.

  * Q2E/PRF

A zero-shot prompt like Q2E/ZS but with extra context in the form of PRF
documents like Q2D/PRF.

  * CoT

A zero-shot Chain-of-Thought prompt which instructs the model to provide
rationale for its answer.

  * CoT/PRF

A prompt like CoT but which also contains extra context in the form of top-3
retrieved PRF documents for the query.

Zero-shot prompts (Q2D/ZS and Q2E/ZS) are the simplest as they consist of a
simple plaintext instruction and the input query. Few-shot prompts (Q2D and
Q2E) additionally contain several examples to support in-context learning, for
example they contain queries and corresponding expansions. Chain-of-Thought
(CoT) prompts formulate their instruction to obtain a more verbose output from
the model by asking it to break its response down step-by-step. Finally,
Pseudo-Relevance Feedback ($\cdot$/PRF) variations of prompts use the top-3
retrieved documents as additional context for the model. See Appendix A for
the exact prompts that are used in the experiments.

##  4\. Experiments

To validate the effectiveness of the LLM-based query expansion we run
experiments on two retrieval tasks: MS-MARCO (Nguyen et al., 2016) passage
retrieval and BEIR (Thakur et al., 2021). For the retrieval system we use BM25
(Robertson and Jones, 1976; Robertson et al., 1995) as implemented by Terrier
(Ounis et al., 2005)111http://terrier.org/. We use the default BM25 parameters
($b=0.75$, $k_{1}=1.2$, $k_{3}=8.0$) provided by Terrier.

###  4.1. Baselines

To analyze the LLM-based query expansion methods we compare against several
classical PRF-based query expansion methods (Amati and Van Rijsbergen, 2002):

  * •

Bo1: Bose-Einstein 1 weighting

  * •

Bo2: Bose-Einstein 2 weighting

  * •

KL: Kullback-Leibler weighting

The implementations for these are provided by Terrier. In all cases we use the
default Terrier settings for query expansion: 3 PRF docs and 10 expansion
terms.

Furthermore, we include the prompt from Query2Doc (Wang et al., 2023) as a
baseline. However, we do not compare against their exact setup since they use
a significantly larger model than the models we study in this paper. The
comparisons in this paper are focused on prompts and not on the exact numbers
produced by different, potentially much larger, models. Furthermore, for
models with a small receptive field (specifically the Flan-T5 models) we only
use a 3-shot Q2D prompt instead of the standard 4-shot prompt to prevent the
prompt from being truncated.

###  4.2. Language Models

We compare the prompts on two types of models, Flan-T5 (Raffel et al., 2020;
Chung et al., 2022) and Flan-UL2 (Tay et al., 2022), at various model sizes:

  * •

Flan-T5-Small (60M parameters)

  * •

Flan-T5-Base (220M parameters)

  * •

Flan-T5-Large (770M parameters)

  * •

Flan-T5-XL (3B parameters)

  * •

Flan-T5-XXL (11B parameters)

  * •

Flan-UL2 (20B parameters)

We choose to use the Flan (Wei et al., 2021; Chung et al., 2022) versions of
the T5 (Raffel et al., 2020) and UL2 (Tay et al., 2022) models as they are
fine-tuned to follow instructions which is critical when using prompt-based
approaches. Furthermore, all of these models are available as open-
source222Models are available at
https://huggingface.co/docs/transformers/model_doc/flan-t5 and
https://huggingface.co/google/flan-ul2.

###  4.3. Metrics

Since we are interested in query expansion, which is largely focussed on
improving the recall of first-stage retrieval, we use Recall@1K as our core
evaluation metric. We also report top-heavy ranking metrics using MRR@10
(Voorhees et al., 1999) and NDCG@10 (Järvelin and Kekäläinen, 2002) to better
understand how the models change the top retrieved results. We report all our
results with significance testing using a paired $t$-test and consider a
result significant at $p<0.01$.

##  5\. Results

###  5.1. MS-MARCO Passage Ranking

Table 1 presents the results on the MS-MARCO passage ranking task. The
classical query expansion baselines (Bo1, Bo2 and KL), already provide a
useful gain in terms of Recall@1K over the standard BM25 retrieval. In line
with the results of (Harman, [n.d.]), we observe that this increase in recall
comes at the cost of top-heavy ranking metrics such as MRR@10 and NDCG@10.

Next, we see the results of LLM-based query expansion depend heavily on the
type of prompts used. Similar to the findings of (Wang et al., 2023), the
Query2Doc prompt (Q2D) can provide a substantial gain in terms of Recall@1K
over the classical approaches. Interestingly, Query2Doc does not only improve
recall, but also improves the top-heavy ranking metrics such as MRR@10 and
NDCG@10, providing a good improvement across metrics. This contrasts with
classical query expansion methods which typically sacrifice top-heavy ranking
metrics in order to improve recall.

Finally, the best performance is obtained by CoT (and the corresponding PRF-
enhanced prompt CoT/PRF). This particular prompt instructs the model to
generate a verbose explanation by breaking its answer down into steps. We
hypothesize that this verbosity may lead to many potential keywords that are
useful for query expansion. Finally, we find that adding PRF documents to the
prompt helps significantly in top-heavy ranking metrics like MRR@10 and
NDCG@10 across models and prompts. A possible explanation for this is that
LLMs are effective in distilling the PRF documents, which may already contain
relevant passages, by attending over the most promising keywords and using
them in the output. We provide a more concrete example of the prompt output in
Appendix B.

Table 1. LLM-based query expansion on the MS-MARCO passage ranking dev set. ▲ indicates a statistically significant (paired $t$-test, $p<0.01$) improvement relative to the Q2D Flan-UL2 method. The best result per metric is bolded. | Recall@1K | MRR@10 | NDCG@10
---|---|---|---
BM25 | 87.82 | 18.77 | 23.44
BM25 + Bo1 | 88.68 | 17.75 | 22.48
BM25 + Bo2 | 88.32 | 17.58 | 22.30
BM25 + KL | 88.62 | 17.71 | 22.44
Flan-T5-XXL (11B) |  |  |
Q2D | 88.76 | 19.07 | 23.76
Q2D/ZS | 88.88 | 18.55 | 23.13
Q2D/PRF | 89.31 |  22.13▲ |  26.43▲
Q2E | 87.74 | 18.74 | 23.37
Q2E/ZS | 87.93 | 18.79 | 23.45
Q2E/PRF | 88.20 | 19.20 | 23.83
CoT | 89.86 | 19.16 | 23.82
CoT/PRF | 89.02 |  22.08▲ |  26.32▲
Flan-UL2 (20B) |  |  |
Q2D | 89.87 | 19.22 | 23.96
Q2D/ZS | 86.60 | 15.56 | 19.54
Q2D/PRF | 89.28 |  21.42▲ |  25.82▲
Q2E | 88.04 | 18.84 | 23.52
Q2E/ZS | 88.11 | 18.87 | 23.56
Q2E/PRF | 88.43 | 19.24 | 23.90
CoT | 90.61▲ |  20.05▲ |  24.85▲
CoT/PRF | 89.30 | 22.62▲ | 26.89▲

###  5.2. BEIR

The BEIR datasets comprise many different zero-shot information retrieval
tasks from a variety of domains. We compare the performance of the different
prompts on the BEIR datasets in Table 2. The first thing to observe here is
that the classical PRF-based query expansion baselines still work very well,
especially on domain-specific datasets such as trec-covid, scidocs and
touche2020. These datasets are largely academic and scientific in nature, and
the PRF documents may provide useful query terms in these cases. In contrast,
the general purpose LLMs may not have sufficient domain knowledge to be useful
for these datasets. Second, we note that the question-answering style datasets
(fiqa, hotpotqa, msmarco and nq) seem to benefit the most from an LLM approach
to query expansion. It is likely that the language model is producing relevant
answers towards the query which helps retrieve the relevant passages more
effectively. Across all datasets, the Q2D/PRF prompt produces the highest
average Recall@1K, with the CoT prompt as a close second.

Table 2. Recall@1K of various prompts on BEIR using Flan-UL2. ▲ indicates a statistically significant (paired $t$-test, $p<0.01$) improvement relative to the best classical QE method. The best result per dataset is highlighted in bold. |  | Classical QE | LLM-based QE
---|---|---|---
Dataset | BM25 | Bo1 | Bo2 | KL | Q2D | Q2D/ZS | Q2D/PRF | Q2E | Q2E/ZS | Q2E/PRF | CoT | CoT/PRF
arguana | 98.93 | 99.00 | 99.00 | 99.00 | 98.86 | 98.93 | 98.93 | 98.93 | 98.93 | 98.93 | 98.93 | 98.86
climate-fever | 46.60 | 45.69 | 45.38 | 45.65 | 47.62 | 47.66 | 47.94 | 46.08 | 46.44 | 46.44 | 47.42 | 46.81
cqadupstack | 65.55 | 66.82 | 66.57 | 66.70 | 65.51 | 64.19 | 65.01 | 65.69 | 65.71 | 65.90 | 66.39 | 66.12
dbpedia | 63.72 | 64.77 | 64.55 | 64.60 | 65.89 | 65.47 | 65.78 | 63.55 | 63.92 | 63.93 | 65.77 | 65.06
fever | 75.73 | 76.28 | 75.83 | 76.32 | 79.06▲ |  78.87▲ | 77.29 | 75.78 | 75.79 | 76.27 |  78.21▲ | 77.25
fiqa | 77.42 | 79.18 | 79.06 | 78.84 | 78.34 | 78.26 | 78.69 | 77.33 | 77.31 | 77.68 | 80.08 | 79.03
hotpotqa | 85.78 | 84.84 | 81.71 | 84.65 |  86.90▲ | 85.71 |  87.58▲ | 85.60 | 85.54 |  87.25▲ |  87.54▲ | 88.79▲
msmarco | 73.61 | 75.08 | 75.14 | 74.66 | 76.77 | 75.73 | 78.75 | 73.87 | 73.79 | 74.14 | 79.58 | 78.36
nfcorpus | 38.70 | 57.30 | 57.67 | 56.46 | 55.34 | 59.81 | 59.68 | 43.38 | 44.12 | 47.06 | 52.63 | 53.32
nq | 78.96 | 81.09 | 80.64 | 80.82 |  85.18▲ |  84.71▲ |  83.53▲ | 79.30 | 79.11 | 80.35 | 85.46▲ |  83.11▲
quora | 99.26 | 99.20 | 99.12 | 99.20 | 99.00 | 98.84 | 98.92 | 99.25 | 99.29 | 99.26 | 99.17 | 99.21
scidocs | 57.46 | 59.78 | 61.03 | 59.86 | 59.09 | 59.78 | 60.10 | 57.88 | 57.70 | 58.32 | 58.51 | 59.69
scifact | 97.17 | 97.57 | 97.57 | 97.57 | 97.57 | 97.57 | 97.57 | 97.17 | 97.17 | 97.17 | 97.57 | 97.17
touche2020 | 84.96 | 85.94 | 86.38 | 86.01 | 83.61 | 83.44 | 84.54 | 85.21 | 85.02 | 86.04 | 85.51 | 84.58
trec-covid | 42.58 | 45.21 | 45.58 | 45.39 | 43.52 | 38.05 | 44.17 | 43.16 | 43.12 | 43.85 | 43.43 | 44.02
Average | 72.43 | 74.52 | 74.35 | 74.38 | 74.82 | 74.47 | 75.23 | 72.81 | 72.86 | 73.50 | 75.08 | 74.76

###  5.3. The Impact of Model Size

60M220M770M3B11B20B$85$$86$$87$$88$$89$$90$$91$Model size (#
parameters)Recall@1KCoTQ2DCoT/PRFBM25 + Bo1

Figure 2. Performance on MS-MARCO passage ranking dev set across different
model sizes. The shaded areas indicate a 99% confidence interval.

[A graph showing the performance of different query expansion techniques
across various model sizes.]The x-axis contains the model size (60M up to 20B)
and the y-axis contains the Recall@1K ranging from 85 to 91. The BM25+Bo1
baseline is plotted as a horizontal line at 88.68 Recall@1K as it is not
affected by model size. The three lines represent three different prompt
techniques (labeled CoT, CoT/PRF, and, Q2D) and show that query expansion
performance improves as the model size gets larger. At the largest model size,
CoT is the best. At a medium-sized model (770M) CoT/PRF is best. CoT is
consistently better than Q2D across all model sizes.

To understand the practical capabilities and limitations of an LLM-based query
expander, we compare different model sizes in Figure 2. We range the model
size from 60M parameters (Flan-T5-small) up to 11B (Flan-T5-XXL) and also try
a 20B parameter model (Flan-UL2) but note that the latter also has a different
pre-training objective. In general we observe the expected trend that larger
models tend to perform better. The Q2D approach requires at least an 11B
parameter model to reach parity with the BM25+Bo1 baseline. In contrast, the
CoT approach only needs a 3B parameter model to reach parity. Furthermore,
adding PRF documents to the CoT prompt seems to help stabilize the performance
for smaller model sizes but does inhibit its performance at larger capacities.
A possible explanation for this behavior is that the PRF documents decreases
the creativity of the model, as it may focus too much on the provided
documents. Although this helps prevent the model from making errors at smaller
model sizes, it also inhibits the creative abilities that we wish to leverage
at larger model sizes. The CoT/PRF prompt is able to outperform the other
prompts at the 770M parameter model size, making it a good candidate for
possible deployment in realistic search settings where serving a larger model
may be impossible. Overall, it is clear that large models are able to provide
significant gains which may limit the practical application of an LLM approach
to query expansion. Distillation has been shown to be an effective way to
transfer the ability of a large model to a smaller one. We leave the study of
distillation of these models for query expansion as future work.

##  6\. Limitations & Future Work

There are a number of limitations in our work: First, we only study sparse
retrieval (BM25) which is where query expansion is important. Dense retrieval
systems (e.g. dual encoders) are less prone to the vocabulary gap and, as a
result, are less likely to benefit from a query expansion. Wang et al. (2023)
has already studied this setting in more detail and we leave the analysis of
our prompts for a dense retrieval setting as future work. Second, our work
focuses on Flan (Wei et al., 2021) instruction-finetuned language models. We
chose these models due to their ability to follow instructions and the fact
that these models are open-source. Our work can naturally be extended to other
language models (Brown et al., 2020; Chowdhery et al., 2022; Du et al., 2022;
Thoppilan et al., 2022) and we leave the study of such models as a topic for
future research. Third, we study specific prompt templates (see Appendix A)
and there may be other ways to formulate the different prompts. Finally, the
computational cost of LLMs may be prohibitive to deploy LLM-based query
expansions in practice. It may be possible to distill the output of the large
model into a smaller servable model. How to productionize LLM-based query
expansions is left as an open problem.

##  7\. Conclusion

In this paper we study LLM-based query expansions. In contrast to traditional
PRF-based query expansion, LLMs are not restricted to the initial retrieved
set of documents and may be able to generate expansion terms not covered by
traditional methods. Our proposed method is simple: we prompt a large language
model and provide it a query, then we use the model’s output to expand the
original query with new terms that help during document retrieval.

Our results show that Chain-of-Thought prompts are especially promising for
query expansion, since they instruct the model to generate verbose
explanations that can cover a wide variety of new keywords. Furthermore, our
results indicate that including PRF documents in various prompts can improve
top-heavy ranking metric performance during the retrieval stage _and_ is more
robust when used with smaller model sizes, which can help practical deployment
of LLM-based query expansion.

As demonstrated in this paper, IR tasks like query expansion can benefit from
LLMs. As the capabilities of LLMs continue to improve, it is promising to see
their capabilities translate to various IR tasks. Furthermore, as LLMs become
more widely available, they will be easier to use and deploy as core parts of
IR systems which is exciting for both practitioners and researchers of such
systems.

## References

  * (1)
  * Amati and Van Rijsbergen (2002) Gianni Amati and Cornelis Joost Van Rijsbergen. 2002\.  Probabilistic models of information retrieval based on measuring the divergence from randomness.  _ACM Transactions on Information Systems (TOIS)_ 20, 4 (2002), 357–389.
  * Bhogal et al. (2007) Jagdev Bhogal, Andrew MacFarlane, and Peter Smith. 2007\.  A review of ontology based query expansion.  _Information processing & management_ 43, 4 (2007), 866–886.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Carpineto and Romano (2012) Claudio Carpineto and Giovanni Romano. 2012.  A survey of automatic query expansion in information retrieval.  _Acm Computing Surveys (CSUR)_ 44, 1 (2012), 1–50.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022\.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022\.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ (2022).
  * Claveau (2021) Vincent Claveau. 2021\.  Neural text generation for query expansion in information retrieval. In _IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology_. 202–209.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Du et al. (2022) Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022\.  Glam: Efficient scaling of language models with mixture-of-experts. In _International Conference on Machine Learning_. PMLR, 5547–5569.
  * Efron et al. (2012) Miles Efron, Peter Organisciak, and Katrina Fenlon. 2012\.  Improving retrieval of short texts through document expansion. In _Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval_. 911–920.
  * Efthimiadis (1996) Efthimis N Efthimiadis. 1996\.  Query Expansion.  _Annual review of information science and technology (ARIST)_ 31 (1996), 121–87.
  * Harman ([n.d.]) D Harman. [n.d.].  Relevance feedback and other query modification techniques.  _Information Retrieval: Data Structures & Algorithms_ ([n. d.]), 241–263.
  * Imani et al. (2019) Ayyoob Imani, Amir Vakili, Ali Montazer, and Azadeh Shakery. 2019.  Deep neural networks for query expansion using word embeddings. In _Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Part II 41_. Springer, 203–210.
  * Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002.  Cumulated gain-based evaluation of IR techniques.  _ACM Transactions on Information Systems (TOIS)_ 20, 4 (2002), 422–446.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  _choice_ 2640 (2016), 660.
  * Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019\.  Document expansion by query prediction.  _arXiv preprint arXiv:1904.08375_ (2019).
  * Ounis et al. (2005) Iadh Ounis, Gianni Amati, Vassilis Plachouras, Ben He, Craig Macdonald, and Douglas Johnson. 2005\.  Terrier information retrieval platform. In _Advances in Information Retrieval: 27th European Conference on IR Research, ECIR 2005, Santiago de Compostela, Spain, March 21-23, 2005. Proceedings 27_. Springer, 517–519.
  * Qiu and Frei (1993) Yonggang Qiu and Hans-Peter Frei. 1993.  Concept based query expansion. In _Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval_. 160–169.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5485–5551.
  * Robertson (1990) Stephen E Robertson. 1990\.  On term selection for query expansion.  _Journal of documentation_ 46, 4 (1990), 359–364.
  * Robertson and Jones (1976) Stephen E Robertson and K Sparck Jones. 1976.  Relevance weighting of search terms.  _Journal of the American Society for Information science_ 27, 3 (1976), 129–146.
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995\.  Okapi at TREC-3.  _Nist Special Publication Sp_ 109 (1995), 109.
  * Rocchio Jr (1971) Joseph John Rocchio Jr. 1971\.  Relevance feedback in information retrieval.  _The SMART retrieval system: experiments in automatic document processing_ (1971).
  * Roy et al. (2016) Dwaipayan Roy, Debjyoti Paul, Mandar Mitra, and Utpal Garain. 2016\.  Using word embeddings for automatic query expansion.  _arXiv preprint arXiv:1606.07608_ (2016).
  * Tao et al. (2006) Tao Tao, Xuanhui Wang, Qiaozhu Mei, and ChengXiang Zhai. 2006\.  Language model information retrieval with document expansion. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_. 407–414.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022.  Unifying language learning paradigms.  _arXiv preprint arXiv:2205.05131_ (2022).
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021\.  BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.  https://openreview.net/forum?id=wCu6T5xFjeJ
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022\.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ (2022).
  * Voorhees (1994) Ellen M Voorhees. 1994\.  Query expansion using lexical-semantic relations. In _SIGIR’94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University_. Springer, 61–69.
  * Voorhees et al. (1999) Ellen M Voorhees et al. 1999\.  The trec-8 question answering track report.. In _Trec_ , Vol. 99. 77–82.
  * Wang et al. (2023) Liang Wang, Nan Yang, and Furu Wei. 2023.  Query2doc: Query Expansion with Large Language Models.  _arXiv preprint arXiv:2303.07678_ (2023).
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ (2021).
  * Zheng et al. (2020) Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. 2020.  BERT-QE: contextualized query expansion for document re-ranking.  _arXiv preprint arXiv:2009.07258_ (2020).
  * Zheng et al. (2021) Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. 2021.  Contextualized query expansion via unsupervised chunk selection for text retrieval.  _Information Processing & Management_ 58, 5 (2021), 102672\.

##  Appendix A Prompts

Table 3 contains all the prompts tried in this paper. In each prompt {query}
denotes the query for which we want to generate a query expansion. We denote
with {query 1}, $\ldots$, {query 4} the sample queries from the MS-MARCO train
set. Similarly, {doc 1}, $\ldots$, {doc 4} represent relevant passages
corresponding to the sampled queries, and, {expansion 1}, $\ldots$, {expansion
4} represent corresponding expansions generated with Terrier KL method (at
most 20 terms) from those relevant passages. Finally we denote with {PRF doc
1}, $\ldots$, {PRF doc 3} the top 3 retrieved documents using the original
query, acting as Pseudo-Relevance Feedback documents. For the CoT prompt, we
note that the model tends to output “The final answer:” or “So the final
answer is:” towards the end and we filter those two sentences out prior to
concatenating the model output with the query.

Table 3. The various query expansion prompts. ID | Prompt
---|---
Q2D (Wang et al., 2023) |  Write a passage that answers the given query: Query: {query 1} Passage: {doc 1} Query: {query 2} Passage: {doc 2} Query: {query 3} Passage: {doc 3} Query: {query 4} Passage: {doc 4} Query: {query} Passage:
Q2D/ZS |  Write a passage that answers the following query: {query}
Q2D/PRF |  Write a passage that answers the given query based on the context: Context: {PRF doc 1} {PRF doc 2} {PRF doc 3} Query: {query} Passage:
Q2E |  Write a list of keywords for the given query: Query: {query 1} Keywords: {expansion 1} Query: {query 2} Keywords: {expansion 2} Query: {query 3} Keywords: {expansion 3} Query: {query 4} Keywords: {expansion 4} Query: {query} Keywords:
Q2E/ZS |  Write a list of keywords for the following query: {query}
Q2E/PRF |  Write a list of keywords for the given query based on the context: Context: {PRF doc 1} {PRF doc 2} {PRF doc 3} Query: {query} Keywords:
CoT |  Answer the following query: {query} Give the rationale before answering
CoT/PRF |  Answer the following query based on the context: Context: {PRF doc 1} {PRF doc 2} {PRF doc 3} Query: {query} Give the rationale before answering

##  Appendix B Example Output

Table 4 shows the results of a query expansion for both the Flan-T5-Large
(770M) model size and the Flan-UL2 (20B) model size. First, note that at the
smaller model size, the CoT and Q2D prompts are not producing the correct
answer which is harmful for retrieval performance. The CoT/PRF prompt, being
more grounded in its PRF documents, avoids this problem and correctly produces
the answer “Tata Motors” which helps retrieve the relevant passage. At the
larger model size (Flan-UL2), all prompts Q2D, CoT and CoT/PRF produce the
correct answer “Tata Motors”. However, the CoT prompt provides the most
verbose explanation towards its answer and has many term overlaps with the
relevant passage, improving its overall retrieval performance.

Table 4. Example of query expansions with Flan-UL2. Green highlights the correct answer while blue indicate specific term overlaps that indicate potentially useful query expansions. | Query |  who owns jaguar motors? (ID=1045405)
---|---|---
| Passage |  Jaguar is the luxury vehicle brand of Jaguar Land Rover, a British multinational car manufacturer with its headquarters in Whitley, Coventry England, owned by the Indian company Tata Motors since 2008.
Flan-T5-Large (770M) | Q2D |  Jaguar Motors is a British luxury car manufacturer based in the United Kingdom.
CoT |  Jaguar Land Rover is the owner of Jaguar. The answer: Jaguar Land Rover.
CoT/PRF |  The relevant information is: Jaguar is owned by the Indian automobile manufacturer Tata Motors Ltd. So the final answer is the Indian automobile manufacturer Tata Motors Ltd.
Flan-UL2 (20B) | Q2D |  Jaguar is a division of Tata Motors, a company owned by the Tata family.
CoT |  Jaguar Land Rover is a British multinational car manufacturer, founded by William Lyons in 1931. Its headquarters are in Whitley, Coventry, United Kingdom and is a constituent of the FTSE 250 Index. The company is a wholly owned subsidiary of Tata Motors of India. So the final answer is Tata Motors.
CoT/PRF |  Jaguar is owned by the Indian automobile manufacturer Tata Motors Ltd. The final answer: Tata Motors Ltd.
  *[LLMs]: Large Language Model
  *[PRF]: Pseudo-Relevance Feedback
  *[LLM]: Large Language Model
  *[CoT]: Chain-of-Thought
  *[IR]: Information Retrieval


# Improved Semantic Role Labeling using Parameterized Neighborhood Memory
Adaptation

Ishan Jindala, Ranit Aharonova, Siddhartha Brahmab, Huaiyu Zhua, Yunyao Lia
aIBM Research, Almaden Research Center, CA 95120
bGoogle Research, Berlin, Germany
<EMAIL_ADDRESS>
{huaiyu<EMAIL_ADDRESS>
<EMAIL_ADDRESS>b Work done while at IBM Research

###### Abstract

Deep neural models achieve some of the best results for semantic role
labeling. Inspired by instance-based learning that utilizes nearest neighbors
to handle low-frequency context-specific training samples, we investigate the
use of memory adaptation techniques in deep neural models. We propose a
parameterized neighborhood memory adaptive (PNMA) method that uses a
parameterized representation of the nearest neighbors of tokens in a memory of
activations and makes predictions based on the most similar samples in the
training data. We empirically show that PNMA consistently improves the SRL
performance of the base model irrespective of types of word embeddings.
Coupled with contextualized word embeddings derived from BERT, PNMA improves
over existing models for both span and dependency semantic parsing datasets,
especially on out-of-domain text, reaching F1 scores of $80.2\%$, and
$84.97\%$ on CoNLL2005, and CoNLL2009 datasets, respectively.

##  1 Introduction

Semantic role labeling (SRL) is the task of identifying predicate-argument
structures from a given sentence based on semantic frames and their roles
Jurafsky and Martin (2008). It is a fundamental task for natural language
understanding.

As shown in Fig.1, SRL is typically decomposed into four sub-tasks: 1)
_predicate identification (e.g. issue)_ ; 2) _sense disambiguation (e.g.
issue.01)_ ; 3) _argument identification_ for each predicate _(e.g. a special
edition)_ ; and 4) _role classification_ of identified arguments (_e.g.
ARG1_). A popular approach for SRL Li et al. (2018) is to assume predicates
are given and convert the argument prediction problem into a sequence tagging
problem both for span and dependency type arguments.

Deep neural models achieve some of the best results in predicting semantic
roles over standard benchmark datasets He et al. (2017b); Tan et al. (2018a);
Ouchi et al. (2018); Li et al. (2019); Kasai et al. (2019).Meanwhile, a very
simple instance-based learning method, K-SRL Akbik and Li (2016a) has been
shown strong performance using a distance function based on syntactic
features. The semantic role of a candidate test word is determined by taking a
majority vote of the labels of closest words present in a memory populated by
words from the training set.

Figure 1: Semantic Role Labeling with BIO tags

A natural question is whether one can apply such memory based methods with
features derived from deep neural models. Memory based and memory adaptive
learning methods have been successfully applied to language modeling and other
problems e.g. the continuous cache model Grave et al. (2016, 2017). The memory
contains activations (e.g. from a RNN) observed during training and the
associated labels. Sprechmann et al. (2018) propose an inference time memory
based parameter adaptation method for language modeling. At each point during
inference, they use its contextual representation to compute $K$ nearest
neighbors in the memory. The labels of the neighbors are then used to update
the classification layer for final prediction.

In this paper, we propose a _parameterized neighborhood memory adaptive_
(PNMA) method to exploit the information contained in the memory of
activations. Our method proceeds in two phases.

  * •

Memory Generation: We populate a memory consists of each token representations
from the training set and compute the $K$ nearest neighbors for each token
using Euclidean distance. We hypothesize and show empirically that the nearest
neighbors contain valuable information about the correct label of $w$ even
when the base model prediction itself is wrong.

  * •

PNMA: We then exploit the memory constructed earlier by computing a
_parameterized_ single vector representation
$\mathbf{n}_{K}(w)\in\mathbb{R}^{d}$ of the $K$ nearest neighbors, which is
then used to retrain the classification layers of the base model. Instead of a
single representation $\mathbf{h}(w)$, the nearest neighbors define an
ensemble of $K$ representations close to $\mathbf{h}(w)$, which we combine
using learned parameters into $\mathbf{n}_{K}(w)$.

We show empirically that PNMA improves the performance of the base SRL model,
both without and with the use of pretrained word embeddings. We further
demonstrate that PNMA addresses the low-frequency predicates in the training
data and improves the base model prediction on low-frequency exceptions. Our
best model combining PNMA and BERT achieves new state-of-the-art F1 scores on
both datasets (for a single model), with a 2.0 point increase over existing
methods on the out-of-domain Brown test set in CoNLL2005 and 5.0 point
increase on CoNLL2009.

In the rest of the paper, we describe the PNMA model in Sec. 2 and demonstrate
its efficacy with extensive empirical evaluation in Sec. 3 and analysis in
Sec. 4. We then review the existing literature in Sec. 5 and conclude in Sec.
6.

=

Figure 2: (a) Base model architecture for SRL. (b) Learning the parametrized
neighborhood representation, $\mathbf{n}_{K}(w)$. When learning
$\mathbf{n}_{K}(w)$, the word representation and LSTM + connection layers are
kept fixed.

##  2 Model

We formulate SRL as a sequence tagging problem both for span and dependency
type arguments Ouchi et al. (2018). For datasets with span style arguments, we
convert the semantic role to BIO tags. We assume that the input to the model
is a sentence $S$ with $n$ tokens. Knowing the predicate position in a
sentence is previously known to improve the performance of argument
classification Li et al. (2018). Since all benchmark datasets have the marked
predicate position, we use this information and assume that each token is
tagged with a 0/1 bit indicating the predicate word. Our proposed method takes
the base model representation for populating the memory.

###  2.1 Base Model for SRL

We use an Alternating LSTM He et al. (2017b); Ouchi et al. (2018) that has
been successfully used for SRL as the base model. As shown in Fig. 2, the base
model has the following main components.

Word Embeddings: Word embeddings are the numerical vector representations of
the textual words. Contextualized word embeddings derived from pretrained
language models such as ELMo Peters et al. (2018) and BERT Devlin et al.
(2018) have been shown to give superior performance in many NLP tasks. Here,
we denote the embeddings of the $n$ tokens by
$\mathbf{e}_{1},\cdots,\mathbf{e}_{n}$.

Predicate Embeddings: We use the marked predicate position to obtain a
predicate-specific word representation. We map the predicate indicator bits to
dense embeddings $\mathbf{p}_{1},\cdots,\mathbf{p}_{n}$. The combined
embedding for token $i$ is the concatenation
$\mathbf{x}_{1}^{i}=[\mathbf{e}_{i};\mathbf{p}_{i}]$, which is the input to
the LSTM.

LSTM Layers: To sequentially process the word representations from the
previous step, we use an Alternating LSTM architecture, where forward and
backward LSTM layers alternate. We use a total of $L=4$ layers. The inputs
$\mathbf{x}_{l}^{1},\cdots,\mathbf{x}_{l}^{n}$ to the $l$-th layer produces
the output $\mathbf{h}_{l}^{1},\cdots,\mathbf{h}_{l}^{n}$.

Connection Layers. To facilitate the flow of gradients between LSTM layers,
the input to the $(l+1)$th LSTM layer is
$\mathbf{x}_{l+1}^{i}~{}=~{}\text{ReLU}(\mathbf{W}_{l}[\mathbf{h}_{l}^{i};\mathbf{x}_{l}^{i}])\in~{}\mathbb{R}^{d}$,
where $\mathbf{W}_{l}$ is a parameter matrix.

CRF Layer: The representations
$\mathbf{h}_{L}^{1},\cdots,\mathbf{h}_{L}^{n}~{}\in~{}\mathbb{R}^{d}$ from the
LSTM layers are passed onto a CRF layer Zhou and Xu (2015a), which produces
predictions for each token. The CRF layer is trained end-to-end with the other
layers.

###  2.2 Parameterized Neighborhood Memory Adaptive (PNMA) Model

After we train a strong multilayer LSTM base model on the SRL dataset, in the
first phase we generate the memory of activations and in the second phase we
compute the parameterized vector representations of each token and retrain the
classification layers of base model. Here we describe the two phases of our
proposed model in detail.

####  2.2.1 Phase 1: Memory Generation

$1$$4$$8$$16$$32$$64$$0$$0.1$$0.2$RankNormalized freqeuncy Figure 3:
Distribution of rank of nearest token with correct label among $K=64$ nearest
neighbors.

Intuition: We hypothesize that the $K$ nearest neighbors contain valuable
information about the correct label of $w$ even when $\mathbf{h}_{L}(w)$
itself leads to an incorrect prediction. To test this hypothesis, we train a
base SRL model on the CoNLL2005 training set (see details in Section 3) and
compute the predicted labels for the sentences in the validation set. For each
token $w^{\prime}$ with a wrong predicted label, we compute its $K=64$ nearest
neighbors and find the rank (in order of increasing distance) of the first
token with the correct label (of $w^{\prime}$) among the neighbors. The
distribution of these ranks, as shown in Fig. 3, is highly left skewed. Almost
all tokens labeled incorrectly by the base model have a close neighbor in
memory that has the correct label. The corresponding distribution for tokens
correctly labeled by the base model is even more left skewed. However, it is
not clear a-priori which neighbor of $w^{\prime}$ should we trust. This
observation motivates the second phase of training.

Memory Generation: We use the trained base model to populate a memory $M$ with
activations $\mathbf{h}_{L}(w)$ produced by the final LSTM layer on words $w$
in the training set, where

$M=\\{\mathbf{h}_{L}(w)\text{ where }w\text{ is in the training
set.}\\}\vspace{-1em}$

The size of $M$ can be varied and our particular choice is discussed in
Section 3. Now, for each token $w$ in the training set, we compute its $K$
nearest neighbors using the Euclidean distance between $\mathbf{h}_{L}(w)$ and
$\mathbf{m}\in M$. We denote the representation of the $i$-th nearest neighbor
of $w$ by $\mathbf{m}_{i}(w)\in\mathbb{R}^{d}$ and the set of nearest neighbor
vectors by $N_{K}(w)$.

Parametrized Representation: To exploit the information contained in the
neighborhood, we compute a _parameterized_ compact representation of
$N_{K}(w)$. Let $\\{~{}\mathbf{n}_{i}~{}\in~{}\mathbb{R}^{d},\text{ for }1\leq
i\leq K\\}$ denote a set of _neighborhood_ parameters learned during training.
Consider the following quantities.

$\\{\eta_{i}(w)\\}_{i=1}^{K}=\text{Softmax}\\{\mathbf{n}_{i}\cdot|\mathbf{m}_{i}(w)-\mathbf{h}_{L}(w)|\\}_{i=1}^{K}\vspace{-0.25em}$
(1)

Intuitively, $|\mathbf{m}_{i}(w)-\mathbf{h}_{L}(w)|$ (the element-wise
absolute difference) is a dense representation of the separation between $w$
and its $i$-th nearest neighbor. The inner product with $\mathbf{n}_{i}$
followed by the softmax computes a distribution over the neighbors
$\\{\eta_{i}\\}_{i=1}^{K}$, which is then used to compute a compact weighted
representation of $N_{K}(w)$.

$\mathbf{n}_{K}(w)=\sum_{i=1}^{K}\eta_{i}(w)\mathbf{m}_{i}(w)\in\mathbb{R}^{d}$
(2)

It is worthwhile to contrast $\mathbf{h}_{L}(w)$ with $\mathbf{n}_{K}(w)$. The
former is the representation of $w$ produced by the base model, while the
latter is derived from an ensemble of $K$ representations that are close to
$\mathbf{h}_{L}(w)$ in $M$. By making $\mathbf{n}_{K}(w)$ a function of
$|\mathbf{m}_{i}(w)-\mathbf{h}_{L}(w)|$ and learnable parameters, we aim to
extract the most relevant information in $N_{K}(w)$. We note that we observed
consistently better performance by using a distinct $\mathbf{n}_{i}$ for each
neighbor instead of a single vector for all neighbors.

###  2.3 PNMA Training

In the second training phase, we freeze the parameters associated with the
LSTM, connection and embedding layers in the base model and update the
neighborhood parameters $\\{\mathbf{n}_{i}\\}_{i=1}^{K}$ and the parameters in
the classification and CRF layers using $\mathbf{n}_{K}(w)$ and the label of
$w$ in the training set. We did not observe benefits by using
$\mathbf{h}_{L}(w)$ itself in the second training phase. At test time, we
obtain the SRL label predictions by computing $\mathbf{n}_{K}(w)$ for each
token $w$ in the test sentence, which itself requires the computation of
$N_{K}(w)$ using the LSTM representations of $w$ and $M$.

| CoNLL2005 | CoNLL2012
---|---|---
| Validation | Test WSJ | Test Brown | Validation | Test
Model | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Syntax Aware Models |  |  |  |  |  |
Zhou et al. (2019)+ELMo | - | - | - | 87.76 | 88.29 | 88.02 | 79.59 | 78.64 | 79.11 | - | - | - | - | - | -
Zhou et al. (2019)+BERT | - | - | - | 89.04 | 88.79 | 88.91 | 81.89 | 80.98 | 81.43 | - | - | - | - | - | -
Li et al. (2019)+ELMo | - | - | - | 89.6 | 90.1 | 89.8 | 82.4 | 82.6 | 82.5 | - | - | - | - | - | -
Li et al. (2019)+BERT | - | - | - | 90.2 | 91.8 | 91.0 | 83.2 | 84.0 | 83.6 | - | - | - | - | - | -
Pred-Span-ELMo | - | - | - | - | - | 87.4 | - | - | 80.4 | - | - | - | - | - | 85.5
Syntax Agnostic Models |  |  |  |  |  |
ALC-Span-ELMo | 87.4 | 86.3 | 86.9 | 88.2 | 87.0 | 87.6 | 79.9 | 77.5 | 78.7 | 87.2 | 85.5 | 86.3 | 87.1 | 85.3 | 86.2
ALC-BIO-ELMo | 86.6 | 86.8 | 86.7 | 87.4 | 87.3 | 87.3 | 78.5 | 78.3 | 78.4 | 86.1 | 85.8 | 85.9 | 86.0 | 85.7 | 85.9
Ours (All use BIO) |  |  |  |  |  |
Base-Rand | 79.4 | 79.0 | 79.2 | 80.9 | 80.3 | 80.6 | 69.6 | 67.3 | 68.5 | 79.7 | 77.7 | 78.7 | 77.5 | 77.8 | 78.7
Base-Rand+PNMA | 80.2 | 79.3 | 79.7 | 81.8 | 80.6 | 81.2 | 70.8 | 68.1 | 69.4 | 80.5 | 78.7 | 79.6 | 78.3 | 78.6 | 79.5
Base-ELMo | 86.4 | 86.4 | 86.4 | 87.6 | 87.2 | 87.4 | 79.1 | 78.1 | 78.6 | 84.1 | 84.9 | 84.5 | 83.7 | 84.9 | 84.3
Base-ELMo+PNMA | 86.9 | 87.0 | 86.9 | 87.6 | 87.3 | 87.5 | 79.8 | 78.9 | 79.3 | 85.0 | 84.9 | 84.9 | 84.9 | 85.1 | 85.0
Base-BERT | 86.9 | 86.9 | 86.9 | 87.6 | 87.7 | 87.7 | 79.8 | 80.0 | 79.9 | 85.5 | 86.2 | 85.9 | 85.4 | 86.3 | 85.9
Base-BERT+PNMA | 87.0 | 87.2 | 87.1 | 88.7 | 88.0 | 87.9 | 80.3 | 80.1 | 80.2 | 86.4 | 86.8 | 86.6 | 86.3 | 86.8 | 86.6

Table 1: Results on the CoNLL2005 and CoNLL2012 datasets. ALC-{Span,BIO}-ELMo
are from Ouchi et al. (2018) and Pred-Span-ELMo is from He et al. (2018b). WSJ
(Wall Street Journal) is in-domain and Brown is out-of-domain.
(ALC=Alternating LSTM+CRF).

The updated base model after the second phase of training is the final SRL
model, which we call the Parameterized Neighborhood Memory Adaptive (PNMA)
model. By parameterizing the representation of $N_{K}(w)$ and retraining, we
allow our model to make optimal use of the nearest neighbors of $w$ in $M$. In
contrast to Akbik and Li (2016a), we do not use handcrafted features or
distance functions. Compared to Sprechmann et al. (2018), we do not update the
classification and CRF layers for each token or sentence separately.

##  3 Experiments

###  3.1 Experimental Setup

Datasets: We evaluate PNMA on both span style and dependency style Propbank
semantic parsing datasets. For span style SRL evaluation we present results on
the CoNLL2005 shared task Carreras and Màrquez (2005) and the CoNLL2012
Pradhan et al. (2012) datasets. We evaluate PNMA on the English subset of
CoNLL2009 Hajič et al. (2009) shared task dataset to present the applicability
of PNMA to dependency style SRL datasets. Key statistics of these datasets is
in Appendix A.

For the dependency SRL task, we follow the work of Marcheggiani and Titov
(2017a) and use the off-the-shelf disambiguator for predicate sense
disambiguation Roth and Lapata (2016).

Word Embeddings: We experiment with randomly initialized embeddings and
publicly available pretrained contextualized embeddings from ELMo and BERT
(the large cased model). In both cases, we use a scaled convex combination of
embeddings $\mathbf{e}_{i}^{\text{CE}}(w)$ produced by layer $i$ of the
underlying language model.

Training Specifics: We use embeddings of size 50 to map the 0/1 predicate
indicators. Each LSTM layer has a hidden dimension of 300. We largely follow
the training procedure outlined in Ouchi et al. (2018). We optimize the loss
function with the Adam optimizer Kingma and Ba (2015) for 100 epochs, with an
initial learning rate of 1e-3, which is decreased by half after epoch 50 and
75. A $L2$ weight decay of 1e-4 is used for regularization.

We apply a dropout of $\delta_{l}\in\\{0.05,0.1,0.15\\}$ after each LSTM layer
and a dropout of $\delta_{e}\in\\{0.45,0.5,0.55\\}$ after the word embedding
layer. We use 1024 dimensions for randomly initialized word embeddings, the
same as ELMo and BERT. For the memory adaptive training phase, we compute
$K=64$ neighbors for each token and train for 20 epochs with a constant
learning rate of 4e-4. For both the datasets, we populate a memory $M$ with
$15\%$ of the tokens in the respective training sets. We did not observe
significant performance differences by increasing memory size beyond this.

###  3.2 Results

We evaluate our models using the official CoNLL SRL evaluation scripts. For a
span style dataset we use the CoNLL 2005 SRL evaluation script and for a
dependency SRL dataset we use the CoNLL 2009 SRL evaluation script. The
results are averaged over five runs of model training with random seeds, as
summarized in Table 1. We show results for the base models trained with
randomly initialized word embeddings (Base-Rand in the table) and those with
ELMo (Base-ELMo) and BERT (Base-BERT) embeddings and the gains obtained by
using PNMA in each case.

Span SRL Result: For CoNLL2005, the use of PNMA improves validation and in-
domain WSJ test F1 scores for all cases, albeit by varying margins, ranging
from 0.1 to 0.6 F1 points. In particular, we observe the highest gains of 0.9,
0.7 and 0.3 F1 points on the out-of-domain Brown test set for the three
respective models. The consistent improvements confirm the effectiveness of
our proposed PNMA model by exploiting information in the nearest neighbors of
tokens in $M$. Coupled with the strong performance of Base-BERT, the Base-
BERT+PNMA model improves upon syntax agnostic state-of-art results for
CoNLL2005 by margins of 0.2, 0.3 and 1.8 F1 points on the validation, WSJ and
Brown test sets, providing the best results among syntax agnostic SRL model.

Although our model works independent of any kind of syntactic information, we
also compare PNMA with models that are syntax aware and find that it is very
competitive with syntax aware SRL models. This observation is important since
syntax aware models assume the availability of a state-of-the-art syntactic
parser which is not always available, and may be challenging to obtain for new
domains or languages.

The results for the CoNLL2012 dataset are also shown in Table 1. Here again,
PNMA results in gains across the board with the best results achieved by Base-
BERT+PNMA, improving upon the current state of the art by 0.3 and 0.4 F1
points for the validation and test sets.

Dependency SRL Results: In Table 2 we present the PNMA results on the
CoNLL2009 dataset and compare it with the state-of-the-art syntax aware and
syntax agnostic SRL models. When compared to syntax agnostic models we observe
a significant performance gain of $5.2$ absolute F1 points on the out-of-
domain Brown set. The results also show that PNMA is very competitive when
compared with syntax aware models. Irrespective of whether the model is syntax
agnostic or syntax aware, we obtain new state-of-the-art on the out-of-domain
Brown set. Here again, we observe the performance gain with the PNMA over the
base models.

| CoNLL2009 | Predicate disambiguation not evaluated
---|---|---
| Test WSJ | Test Brown | Test WSJ | Test Brown
Model | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Syntax Aware Models |  |  |
Akbik and Li (2016b) | 91.2 | 87.4 | 89.3 | 82.1 | 77.8 | 79.9 | - | - | - | - | - | -
Li et al. (2018)+ELmo | 90.8 | 88.6 | 89.7 | 81.0 | 78.2 | 79.6 | - | - | - | - | - | -
Kasai et al. (2019) | 89.0 | 88.2 | 88.6 | 78.0 | 77.2 | 77.6 | - | - | - | - | - | -
Kasai et al. (2019)+ELMo | 90.3 | 90.0 | 90.2 | 81.0 | 80.5 | 80.8 | - | - | - | - | - | -
Zhou et al. (2019)+ELMo | 89.7 | 90.9 | 90.3 | 83.9 | 85.0 | 84.4 | - | - | - | - | - | -
Li et al. (2019)+ELMo | 90.8 | 93.5 | 92.2 | 82.0 | 83.4 | 82.7 | - | - | - | - | - | -
Syntax Agnostic Models |  |  |
Marcheggiani et al. (2017) | 88.7 | 86.8 | 87.7 | 79.4 | 76.2 | 77.7 | - | - | - | - | - | -
He et al. (2018a) | 89.5 | 87.9 | 88.7 | 81.7 | 76.1 | 78.8 | - | - | - | - | - | -
Cai et al. (2018) | 89.9 | 89.2 | 89.6 | 79.8 | 78.3 | 79.0 | - | - | - | - | - | -
Guan et al. (2019)+ELMo | 90.0 | 89.2 | 89.6 | 80.0 | 79.4 | 79.7 | - | - | - | - | - | -
Ours |  |  |
Base-Rand | 86.13 | 84.45 | 85.28 | 75.60 | 74.58 | 75.09 | 81.35 | 77.24 | 79.24 | 70.35 | 64.61 | 67.36
Base-Rand+PNMA | 86.05 | 84.56 | 85.30 | 75.66 | 74.99 | 75.33 | 81.26 | 77.41 | 79.29 | 70.48 | 65.18 | 67.73
Base-ELMo | 89.73 | 90.13 | 89.93 | 82.38 | 83.77 | 83.07 | 86.91 | 85.55 | 86.23 | 80.58 | 77.40 | 78.96
Base-ELMo+PNMA | 89.91 | 90.13 | 90.02 | 82.51 | 83.93 | 83.21 | 87.17 | 85.56 | 86.36 | 80.78 | 77.62 | 79.17
Base-BERT | 90.51 | 90.88 | 90.70 | 82.99 | 86.42 | 84.67 | 88.06 | 86.66 | 87.36 | 81.53 | 81.09 | 81.31
Base-BERT+PNMA | 90.03 | 91.52 | 90.77 | 83.54 | 86.45 | 84.97 | 87.40 | 87.59 | 87.50 | 82.32 | 81.13 | 81.72

Table 2: Results on English subset of CoNLL2009 shared task dataset. In each
column we bold the corresponding best results and underlined bold numbers
represents the best score overall.

Since we use an off-the-shelf sense disambiguator for all the experiments, we
also present the results on argument classification alone in Table 2 (that is
we do not evaluate the predicate sense disambiguation following Marcheggiani
and Titov (2017b)) to show the actual performance gain by PNMA. In all the
regimes PNMA outperforms the corresponding base model predictions.

Computation Overhead: Depending on the size of $M$, the time required to
compute $N_{K}(w)$ may vary. We utilize GPUs to do fast batched computation of
distances which results in less than $10\%$ overhead compared to training the
base model with BERT. Since the LSTM layers are frozen, the second phase is
significantly faster than the first.

##  4 Analysis

We analyze factors that may impact the effectiveness of PNMA models. We focus
on the models where PNMA alters the base model predictions the most: the Base-
ELMo model for CoNLL2005 111On average Base-ELMo+PNMA alters about 40 samples
whereas Base-Bert+PNMA alters only 20 samples and the Base-Bert model for
CoNLL2009 222On average Base-Bert+PNMA alters about 37 samples whereas Base-
ELMo+PNMA alters only 30 samples.

Figure 4: Heatmap of the differences between the confusion matrices of Base-
ELMo+PNMA and Base-ELMo on the Brown test set in CoNLL2005 for the top 10
argument labels. Greener values on diagonal (more correct labels) and redder
values off-diagonal (fewer errors) signal better performance of Base-
ELMo+PNMA.

###  4.1 Role Level Analysis

We first analyze how the effectiveness of PNMA differ across individual
argument role labels. Fig. 4 shows a heatmap produced by plotting the
difference between the confusion matrices of Base-ELMo+PNMA and Base-ELMo on
the Brown test set in CoNLL2005. As can be seen, PNMA improves the prediction
of role labels for almost all types in various degrees, except for LOC. In
particular, PNMA considerably improves the prediction of the core role A2,
mainly by reducing misclassification to A1 and LOC. Similarly, it evidently
improves the prediction of A1 by reducing misclassification to A0, A2 and LOC.
However, its somewhat poorer performance on the prediction for LOC indicts
slight over-correction of misclassification to LOC. Among adjunct roles, ADV
benefits the most from PNMA thanks to reduced confusion with DIR, MOD and A2.
We lists example instances from the CoNLL2005 datasets where PNMA corrects the
base model’s predictions in Appendix B.

We further analyze how PNMA leads to better prediction of argument role
labels. Taking the core role A2 as an example. As pointed out in He et al.
(2017b), A2 shows semantic relations with the adjuncts such as location and
direction, often leading to confusion in base model. For instance, for the
sentence “His father would come upstairs and stand at the foot of the bed and
look at his son”, the span _at the foot of the bed_ is classified as LOC by
our base model. However, PNMA is able to correctly classify this span with the
help of the nearest neighbors (Table 3), which contains the valuable
information about the correct label and hence improves this confusion over
base model.

Memory Label | Nearest Neighbor
---|---
A2 | ..stood next to his car…
LOC | ..concentrated in the northwest..
A2 | ..taken it for granted..
A2 | ..reaped an extra $ on the seventh game..
A2 | ..take skeptics on a tour of its new Gaylord..
LOC | .. rising much more in the Northeast..
A2 | ..fixed on the stock market..
TMP | ..dies at 49..
A2 | ..come to a halt..
LOC | .. in some cases become..

Table 3: Top 10 nearest neighbors of the token span _at the foot of the bed_
which is corrected by PNMA from LOC to A2. In each sentence blue marks a
predicate and orange marks an argument span.

###  4.2 Instance Level Analysis

For all the datatsets, it is almost four times more likely for PNMA to change
a wrong prediction of the base model into a correct prediction than it is for
it to change a correct prediction of the base model into a wrong prediction.
Furthermore, focusing on the the samples where PNMA’s prediction differs from
that of the base model, we find that in most cases PNMA’s prediction is
correct: for the CoNLL2005(CoNLL2009) datasets, out of the cases where the
models disagree, PNMA is correct in $68\%(48\%)$ whereas the base model is
correct in only $20\%(12\%)$ (both are wrong for the rest of these
disagreement cases). This shows that the nearest neighbors indeed contain
valuable information about the correct label of the test token even when the
base model prediction itself is wrong.

Fig. 5 plots the two dimensional representation of a test sample for each
scenario, before and after applying PNMA along with the computed nearest
neighbors: 1) the base model argument role prediction is wrong; PNMA corrects
it in Fig. 5; 2) the base model argument role prediction is wrong; PNMA does
not alter it in Fig. 5; 3) the base model argument role prediction is correct;
PNMA fails to alter it in Fig. 5; 4) the base model argument role prediction
is correct; PNMA alters it to a wrong prediction in Fig. 5.

As can be seen in Figs.5 and 5, PNMA effectively moves the base model’s
representation of the test sample to an area that is denser in examples with
the correct label. In this area, the likelihood of the classifier to predict
the correct label is higher. Thus, PNMA enables to correct wrong labels. In
Fig. 5 the correct label is B-A4, but all neighbors are of other labels and
hence both models are wrong. There are around $10\%$ of the total samples that
lie in this category. Our investigation reveals that most of these samples are
not well represented in the memory similar to the situation shown for the
example in column (d). That is, for these instances there are a very small
number of samples in the neighborhood which have the same label as that of the
true label of the current instance.

(a) Scenario 1

(b) Scenario 2

(c) Scenario 3

(d) Scenario 4

Figure 5: Examples of the base model representation of test samples. Each
column depicts one test sample. Row 1 depicts the representation of the test
sample and the corresponding prediction in the base model; Row 2 depicts the
sample representation after PNMA translation and the corresponding change in
argument label prediction. The correct labels are: (a) I-A2 (b)B-A4 (c) B-A2
(d) B-AM-LOC.

(a) CoNLL2005

(b) CoNLL2009

Figure 6: Model predictions as a function of predicate frequency and
neighborhood representation. Neighborhood samples: the number of samples in
the neighborhood of the test sample of the same role as the correct role of
the test sample.

Low-Frequency Exceptions: As noted in Akbik and Li (2016a), low-frequency
exceptions are important in many real use cases, in which argument roles may
be context specific and hence difficult to learn by generalization from the
entire training data. Fig. 6 shows how the number of samples on which PNMA
disagrees with the base model varies by predicate frequency in the training
data. As can be seen, the disagreement between PNMA and the base model is
higher in low-frequency predicates. As noted above, when the two models
disagree, PNMA is correct most of the time for both CoNLL2005 and CoNLL2009.
This observation confirms the effectiveness of PNMA in addressing low-
frequency exceptions, similar to instance-learning based K-SRL Akbik and Li
(2016a).

###  4.3 Error Analysis

For PNMA to work, two conditions must be met. First, there should be samples
in the neighborhood of the test sample whose gold label matches that of the
test sample. Second, enough of those neighboring samples should have
representations that results in correct label predictions so as to help PNMA
translate the test sample representation to an area which results in a correct
label.

To investigate further, we analyze the predictions of the model as a function
of the number of neighborhood samples and the predicate frequency, as shown in
Fig. 6. These plots show that the PNMA model mostly improves the samples
associated with low-frequency predicates and which have moderate
representation in the memory, i.e. intermediate number of neighborhood samples
(10-40). However, there are some samples that lie in moderate predicate
frequency and moderate neighborhood sample regime which are predicted wrong by
both models. Although these samples do have a good amount of neighbors that
belong to the same category as that of the true label, they are still wrongly
predicted by the PNMA model. This is because some of these neighbors have a
wrong prediction by the base model, which means that the base model
representation in the memory is such that it leads to a wrong prediction. When
many neighbors have such representations, PNMA will translate the test sample
representation into an area which will result in a wrong prediction. This
possibly cause the PNMA to misclassify the arguments roles which are correctly
classified by the base model. Therefore, to further improve our method, better
memory generation techniques are required to cover such cases. This will be
explored in future work.

##  5 Related Work

Recently, a great attention is paid on the use of deep neural network for the
semantic role labeling task Tan et al. (2018b); He et al. (2018a). These
networks largely fall into two broad categories: syntax-agnostic and syntax-
aware. Syntax-aware models are known to perform better on SRL task Roth and
Lapata (2016); He et al. (2017a); Strubell et al. (2018a) and for a long time
syntax was considered a prerequisite for better SRL performance Punyakanok et
al. (2008); Gildea and Jurafsky (2002). These performance gains are due to the
availability of high-quality syntax parser Marcheggiani and Titov (2017a); Li
et al. (2018). However, several works Zhou and Xu (2015b); Marcheggiani et al.
(2017); Tan et al. (2018b); He et al. (2018a) show that the deep networks can
extract useful discriminatory features even without the syntactic information.

Span based SRL, formulated as a sequence tagging problem Villodre et al.
(2005), lends itself to end-to-end deep learning models Zhou and Xu (2015a).
Using more sophisticated encoders such as LSTMs and Transformers has lead to
improved accuracy He et al. (2017b); Tan et al. (2018a); Zhao et al. (2018);
Strubell et al. (2018b). An alternate method of directly predicting the spans
also achieves comparable results He et al. (2018b); Ouchi et al. (2018). In
general, context sensitive word embeddings such as ELMo and BERT leads to
better performance.

Our model using memory adaptation is inspired by the success of similar
techniques in neural models for language modeling e.g. neural cache Grave et
al. (2016, 2017) and memory based methods for few-shot learning Sprechmann et
al. (2018); Rae et al. (2018). These methods shown to capture the more long
range dependencies. Recently, Guan et al. (2019) demonstrate the use of memory
network in SRL task. Our model is different from Guan et al. (2019) in the
sense that in PNMA nearest neighbors don’t need to be labeled and hence can be
applied to large unlabeled corpora.

##  6 Conclusion

To conclude, we propose a Parameterized Neighborhood Memory Adaptive (PNMA)
method to exploit information contained in the nearest neighbors of token
representations derived from deep LSTM models for SRL. Combined with
contextualized word embeddings from BERT, we achieve new state-of-the-art
results for single models on both span and dependency style datasets. Our
experimental results indicate that PNMA improves over various argument roles
and correct most of the sample where PNMA disagree with the base model. For
the samples predicted wrongly by both models, we plan to investigate in the
future better form of memory or multi-layered representations in the base
model to compute better neighborhoods to better handle such cases.

## References

  * Akbik and Li (2016a) Alan Akbik and Yunyao Li. 2016a.  K-srl: Instance-based learning for semantic role labeling.  In _COLING_.
  * Akbik and Li (2016b) Alan Akbik and Yunyao Li. 2016b.  K-srl: Instance-based learning for semantic role labeling.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 599–608.
  * Cai et al. (2018) Jiaxun Cai, Shexia He, Zuchao Li, and Hai Zhao. 2018.  A full end-to-end semantic role labeler, syntactic-agnostic over syntactic-aware?  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2753–2765.
  * Carreras and Màrquez (2005) Xavier Carreras and Lluís Màrquez. 2005.  Introduction to the conll-2005 shared task: Semantic role labeling.  In _Proceedings of the Ninth Conference on Computational Natural Language Learning_ , CONLL ’05, pages 152–164, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002.  Automatic labeling of semantic roles.  _Computational linguistics_ , 28(3):245–288.
  * Grave et al. (2017) Edouard Grave, Moustapha Cissé, and Armand Joulin. 2017.  Unbounded cache model for online language modeling with open vocabulary.  In _NIPS_.
  * Grave et al. (2016) Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016.  Improving neural language models with a continuous cache.  _CoRR_ , abs/1612.04426.
  * Guan et al. (2019) Chaoyu Guan, Yuhao Cheng, and Hai Zhao. 2019.  Semantic role labeling with associated memory network.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3361–3371, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hajič et al. (2009) Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martí, Lluís Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, et al. 2009.  The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages.  In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 1–18. Association for Computational Linguistics.
  * He et al. (2018a) Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018a.  Jointly predicting predicates and arguments in neural semantic role labeling.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 364–369.
  * He et al. (2018b) Luheng He, Kenton Lee, Omer Levy, and Luke S. Zettlemoyer. 2018b.  Jointly predicting predicates and arguments in neural semantic role labeling.  In _ACL_.
  * He et al. (2017a) Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017a.  Deep semantic role labeling: What works and what’s next.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 473–483.
  * He et al. (2017b) Luheng He, Kenton Lee, Mike Lewis, and Luke S. Zettlemoyer. 2017b.  Deep semantic role labeling: What works and what’s next.  In _ACL_.
  * Jurafsky and Martin (2008) Daniel Jurafsky and James H. Martin. 2008.  Speech and language processing: an introduction to natural language processing.
  * Kasai et al. (2019) Jungo Kasai, Dan Friedman, Robert Frank, Dragomir Radev, and Owen Rambow. 2019.  Syntax-aware neural semantic role labeling with supertags.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 701–709.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980.
  * Li et al. (2018) Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018.  A unified syntax-aware framework for semantic role labeling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2401–2411.
  * Li et al. (2019) Zuchao Li, Shexia He, Junru Zhou, Hai Zhao, Kevin Parnow, and Rui Wang. 2019.  Dependency and span, cross-style semantic role labeling on propbank and nombank.  _arXiv preprint arXiv:1911.02851_.
  * Marcheggiani et al. (2017) Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017.  A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 411–420.
  * Marcheggiani and Titov (2017a) Diego Marcheggiani and Ivan Titov. 2017a.  Encoding sentences with graph convolutional networks for semantic role labeling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1506–1515.
  * Marcheggiani and Titov (2017b) Diego Marcheggiani and Ivan Titov. 2017b.  Encoding sentences with graph convolutional networks for semantic role labeling.  In _EMNLP_.
  * Ouchi et al. (2018) Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018.  A span selection model for semantic role labeling.  In _EMNLP_.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018.  Deep contextualized word representations.  In _NAACL-HLT_.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes.  In _EMNLP-CoNLL Shared Task_.
  * Punyakanok et al. (2008) Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008.  The importance of syntactic parsing and inference in semantic role labeling.  _Computational Linguistics_ , 34(2):257–287.
  * Rae et al. (2018) Jack W. Rae, Chris Dyer, Peter Dayan, and Timothy P. Lillicrap. 2018.  Fast parametric learning with activation memorization.  In _ICML_.
  * Roth and Lapata (2016) Michael Roth and Mirella Lapata. 2016.  Neural semantic role labeling with dependency path embeddings.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1192–1202.
  * Sprechmann et al. (2018) Pablo Sprechmann, Siddhant M. Jayakumar, Jack W. Rae, Alexander Pritzel, Adrià Puigdomènech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. 2018.  Memory-based parameter adaptation.  _CoRR_ , abs/1802.10542.
  * Strubell et al. (2018a) Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018a.  Linguistically-informed self-attention for semantic role labeling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5027–5038.
  * Strubell et al. (2018b) Emma Strubell, Patrick Verga, Daniel Andor, David I Weiss, and Andrew McCallum. 2018b.  Linguistically-informed self-attention for semantic role labeling.  In _EMNLP_.
  * Tan et al. (2018a) Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018a.  Deep semantic role labeling with self-attention.  In _AAAI_.
  * Tan et al. (2018b) Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018b.  Deep semantic role labeling with self-attention.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Villodre et al. (2005) Lluís Màrquez Villodre, Pere Comas, Jesús Giménez, and Neus Català. 2005.  Semantic role labeling as sequential tagging.  In _CoNLL_.
  * Zhao et al. (2018) Hai Zhao, Shexia He, Zuchao Li, and Hongxiao Bai. 2018.  Syntax for semantic role labeling, to be, or not to be.  In _ACL_.
  * Zhou and Xu (2015a) Jie Zhou and Wei Xu. 2015a.  End-to-end learning of semantic role labeling using recurrent neural networks.  In _ACL_.
  * Zhou and Xu (2015b) Jie Zhou and Wei Xu. 2015b.  End-to-end learning of semantic role labeling using recurrent neural networks.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1127–1137.
  * Zhou et al. (2019) Junru Zhou, Zuchao Li, and Hai Zhao. 2019.  Parsing all: Syntax and semantics, dependencies and spans.  _arXiv preprint arXiv:1908.11522_.

Dataset | Sentences | Tokens | Labels
---|---|---|---
CoNLL2005 | 90.8K$|$3.3K$|$5.3K$|$0.8K | 2.6M$|$95K$|$149K$|$19K | 106
CoNLL2012 | 253K$|$35.3K$|$24.5K | 6.6M$|$935K$|$640K | 129
CoNLL2009 | 179K$|$6K$|$10K$|$1.2K | 4.5M$|$169K$|$267K$|$26K | 55

Table 4: Summary of datasets. CoNLL2005 and CoNLL2009 contain an extra out-of-
domain test set in addition to the train$|$valid$|$test sets.

##  Appendix A Dataset

The key statistic of the datasets is shown in Table 4. For both the span style
datasets we convert the semantic predicates and arguments roles to BIO
boundary-encoded tags. There are 28 and 42 distinct role types in CoNLL 2005
and CoNLL2012 datasets which were converted to 126 and 129 distinct BIO tags,
respectively.

|  | Base $\rightarrow$ PNMA
---|---|---
S1 |  I can fix him something later in the afternoon when we get home . |  A1 $\rightarrow$ A2
I can fix him something later in the afternoon when we get home . |  A2 $\rightarrow$ A1
S2 |  The patrol snaked around in back of the cave , approached it from above and dropped in suddenly with wild howls . |  A1 $\rightarrow$ A2
S3 |  “ Will you please wait in here ” . |  AM-MOD $\rightarrow$ AM-ADV
S4 |  The fear of punishment just did not bother him . |  AM-DIS $\rightarrow$ AM-ADV
Table 5: Sample corrections made by PNMA in role classification. In each
sentence blue marks a predicate and red marks an argument for role
classification.

##  Appendix B PNMA Corrections

Here we lists the example instances from the CoNLL2005 datasets where PNMA
corrects the base model’s predictions. PNMA improves the prediction of role
labels for almost all types in various degrees. In particular, PNMA
considerably improves the prediction of the core roles.