#  ZeroTOP: Zero-Shot Task-Oriented Semantic Parsing using
Large Language Models

Dheeraj Mekala1 Jason Wolfe2 Subhro Roy2
1UC San Diego 2Microsoft Semantic Machines
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done during an internship at
Microsoft Semantic Machines

###### Abstract

We explore the use of large language models (LLMs) for zero-shot semantic
parsing. Semantic parsing involves mapping natural language utterances to
task-specific meaning representations. Language models are generally trained
on the publicly available text and code and cannot be expected to directly
generalize to domain-specific parsing tasks in a zero-shot setting. In this
work, we propose ZeroTOP, a zero-shot task-oriented parsing method that
decomposes a semantic parsing problem into a set of abstractive and extractive
question-answering (QA) problems, enabling us to leverage the ability of LLMs
to zero-shot answer reading comprehension questions. For each utterance, we
prompt the LLM with questions corresponding to its top-level intent and a set
of slots and use the LLM generations to construct the target meaning
representation. We observe that current LLMs fail to detect unanswerable
questions; and as a result cannot handle questions corresponding to missing
slots. To address this problem, we fine-tune a language model on public QA
datasets using synthetic negative samples. Experimental results show that our
QA-based decomposition paired with the fine-tuned LLM can correctly parse
$\approx 16$% of utterances in the MTOP dataset without requiring any
annotated data.

##  1 Introduction

Large pre-trained language models (LLMs) Sanh et al. (2021); Brown et al.
(2020); Chen et al. (2021); He et al. (2021) have been shown to attain
reasonable zero-shot generalization on a diverse set of NLP tasks including
text classification, question answering and text summarization Wang et al.
(2019). These models when prompted with natural language description or a
common template for the task along with the input, can directly generate the
correct output with reasonable accuracy for many natural language processing
problems.

LLMs are generally trained on publicly available text Liu et al. (2019);
Raffel et al. (2020); Brown et al. (2020) and code Chen et al. (2021), and
they are not expected to generalize to domain-specific semantic parsing tasks
in a similar way. The output of semantic parsing comprises user-defined
domain-specific functions, argument names, and syntax where the inductive bias
from pre-training is less helpful. Instead, we leverage the LLM’s ability to
zero-shot answer reading comprehension questions. In this work, we propose
ZeroTOP that decomposes the semantic parsing task into one of answering a
series of extractive and abstractive questions, corresponding to its top-level
intent and a set of relevant slots. For example, as shown in Figure 1, the
utterance is decomposed into multiple questions related to the intent (alarm
creation) and slots (parameters of the alarm creation). The answers to these
questions can be combined to infer the target meaning representation.

Figure 1:  Example of decomposing semantic parsing task into QA task.  Figure
2:  ZeroTOP decomposes semantic parsing into multiple QA tasks. First, we
identify top-level intent by casting it as an abstractive QA task. Next, we
prompt for each slot using its corresponding question to extract their
respective slot values. If a slot can accommodate nested intents (in red), we
hierarchically prompt for nested slots. Finally, we combine all the responses
to create the meaning representation.

As illustrated in Figure 2, we cast top-level intent classification as an
abstractive QA task. One intuitive way to predict intent is to constrain
autoregressive LMs to search over only valid intent labels. However, LLMs are
known to be biased towards predicting labels common in the pretraining data
Zhao et al. (2021). Therefore, we propose to generate an intent description in
an unconstrained manner from an LLM and infer the intent label most similar to
the generated description.

We view slot value prediction as an extractive QA problem. Most utterances do
not mention all the slots. For example, in the MTOP dataset, on average, only
one-third of possible slots are mentioned per utterance. It is therefore
essential for the model to abstain from prediction when corresponding slots
are not mentioned. Through our analyses, we observe that most LLMs frequently
hallucinate text for missing slots with high confidence, resulting in poor
performance in slot value prediction. To address this, we fine-tune a language
model on a collection of public QA datasets augmented with synthetic
unanswerable samples. We call our trained model Abstainer, as it is capable of
identifying unanswerable questions and abstaining from prediction. We
hierarchically prompt for nested slots using the Abstainer, and infer nested
intents if their corresponding slots are detected.

Our contributions are summarized as follows:

  * •

We propose ZeroTOP that leverages LLMs for zero-shot semantic parsing by
decomposing the task into answering a series of questions.

  * •

To address the LM’s bias towards specific intent labels, we propose intent
description generation in an unconstrained manner and then selecting most
similar intent label to the description.

  * •

We show that current pre-trained LLMs fail to abstain from prediction for
unanswerable questions. We address this by finetuning a language model called
Abstainer on public QA datasets augmented with synthetically generated
unanswerable QA pairs. Abstainer can be successively applied to generate
nested meaning representations.

  * •

Our method significantly outperforms pre-trained LLMs on the MTOP dataset.

##  2 Related Work

Large pre-trained language models are increasingly used to support semantic
parsing in low-data scenarios. Shin et al. (2021) constrain GPT-3 and BART so
that the LMs generate valid canonical representations that can be mapped back
to meaning representations. Schucher et al. (2022); Drozdov et al. (2022)
explore prompt tuning for semantic parsing with LLMs. Yang et al. (2022)
decomposes canonical utterance generation into sub-clause generation and
augments the generated sub-clauses into a canonical utterance. Rubino et al.
(2022) presents a cross-schema parser for several tasks in a given vertical by
augmenting schema-specific context to the input along with the utterance. The
closest work to ours is Zhao et al. (2022) where they decompose parsing into
abstractive QA tasks and aggregate answers to construct a meaning
representation. These methods assume access to some data either from the same
domain, in the same format from a different domain or synthetically generated
from a synchronous grammar. In contrast, we focus on a strict zero-shot
setting where only the schema information is available along with some natural
language prompts for schema entities.

Our work is also related to approaches towards zero-shot dialog state tracking
using language models Gao et al. (2020); Lin et al. (2021a, b). Specifically,
Lin et al. (2021a) uses an Abstainer similar to ours to handle missing slots.
Our method differs in that, we focus on semantic parsing where the Abstainer
needs to be applied multiple times along with intent detection to create
nested meaning representations. Our approach does not require enumerating
choices in the prompt allowing us to handle a large number of intents and
slots as found in semantic parsing datasets like MTOP.

##  3 ZeroTOP: Zero-Shot Task-Oriented Semantic Parsing

###  3.1 Problem Formulation

We focus on task-oriented parsing with hierarchical intent-slot schema. Let
$\mathcal{I}=\\{\mathcal{I}_{1},\mathcal{I}_{2},\ldots,\mathcal{I}_{n}\\}$ be
the set of all possible top-level intents and
$\mathcal{S}=\\{\mathcal{S}_{1},\mathcal{S}_{2},\ldots,\mathcal{S}_{m}\\}$ be
the set of all possible slots. Each intent $\mathcal{I}_{j}$ has a set of
slots
$\mathcal{S}^{j}=\\{\mathcal{S}^{j}_{1},\mathcal{S}^{j}_{2},\ldots,\mathcal{S}^{j}_{n}\\}$
that can be filled. Possible slots in an intent are represented by the intent-
to-slot mapping $\mathrm{I2S}$: $\mathcal{I}$ $\rightarrow$
$\mathcal{P}(\mathcal{S})$, where $\mathcal{P}(\cdot)$ is the powerset
operator that generates the set of all subsets. Similarly, the inverse slot-
to-intent mapping is represented by $\mathrm{S2I}$: $\mathcal{S}$
$\rightarrow$ $\mathcal{I}$. The input in a zero-shot setting contains no
training data, however, we assume access to the intent-slot, slot-intent
mappings $\mathrm{I2S}$, $\mathrm{S2I}$. Our method requires users to provide
a question per slot, that is representative of their purpose. Let
$\mathcal{Q}=\\{\mathcal{Q}_{\mathcal{S}_{1}},\mathcal{Q}_{\mathcal{S}_{2}},\ldots,\mathcal{Q}_{\mathcal{S}_{k}}\\}$
represent respective questions for slots. In a real-life setting, this can be
easily obtained from the domain developer who designed the schema.

###  3.2 Unconstrained Generation for Zero-Shot Intent Classification

We view zero-shot intent classification as an abstractive question-answering
problem and use pre-trained LLMs to answer them. One intuitive way to predict
the intent is to prime the LLM with a QA prompt and then constrain the
generation to search over only valid intent labels Shin et al. (2021).
However, LLMs are known to be biased towards certain text sequences Zhao et
al. (2021) more common in pretraining data. For example, in the MTOP dataset,
the constrained T0-3B model predicts CREATE$\\_$CALL (make call) as intent for
$92\%$ of the data in the call domain. Therefore, we propose to first generate
intent description in an unconstrained fashion and then choose the label that
is most similar to the generated answer. Specifically, we prime the LM with
the below prompt and let the LM generate without any constraints. Then, we
choose the most similar label based on cosine similarity between intent label
and generated text using RoBERTa sentence similarity Reimers and Gurevych
(2019). We find our proposed approach does not exhibit bias towards certain
labels as noticed with constrained generation.

Answer the following question depending on the context. context: A user said,
{utterance}. question: What did the user intend to do? answer:

Figure 3:  An example demonstrating unanswerable sample generation. The first
unanswerable sample is generated by removing the sentence containing the
answer (highlighted with dashed lines) from the context. The second
unanswerable sample is generated by swapping the context of two original
samples.

Input: Set of intents $\mathcal{I}$, Set of slots $\mathcal{S}$, Slot
questions $\mathcal{Q}$, intent-to-slot mapping $\mathrm{I2S}$, slot-to-intent
mapping $\mathrm{S2I}$, slot-to-candidate-nested-intent mapping
$\mathrm{S2NI}$, Intent-model $\mathcal{M}_{I}$, Abstainer
$\mathcal{M}_{abs}$, and Utterance $u$

Output: Predicted meaning representations $\mathrm{MR}$

$\mathrm{intent}$ = $\mathcal{M}_{I}(u)$

$\mathrm{slotValues}$ = {}

for  _slot $\mathcal{S}_{i}\in\mathrm{I2S}(\mathrm{intent})$ _ do

$\mathrm{slotValues}[\mathcal{S}_{i}]$ =
$\mathcal{M}_{abs}(u,\mathcal{Q}_{\mathcal{S}_{i}})$

for  _candidate N.intent $\mathcal{I}_{j}\in\mathrm{S2NI}(\mathcal{S}_{i})$ _
do

for  _slot $\mathcal{S}_{j}\in\mathrm{I2S}(\mathrm{\mathcal{I}_{j}})$ _ do

if
_$\mathcal{M}_{abs}(\mathrm{slotValues}[\mathcal{S}_{i}],\mathcal{Q}_{\mathcal{S}_{j}})$
is not NONE_ then

Update $\mathrm{slotValues}[\mathcal{S}_{i}]$ with nested intent
$\mathcal{I}_{j}$,
$\mathcal{M}_{abs}(\mathrm{slotValues}[\mathcal{S}_{i}],\mathcal{Q}_{\mathcal{S}_{j}})$

$\mathrm{MR}$ = Construct representation with $\mathrm{intent}$,
$\mathrm{slotValues}$

Return $\mathrm{MR}$

Algorithm 1 ZeroTOP: Our proposed Zero-shot semantic parsing method.

###  3.3 Leveraging QA datasets for Slot Value Prediction

Slot value prediction involves extracting phrases for a slot from the user
utterance. Thus, we cast this as an extractive QA problem. All slots might not
be mentioned in an input utterance. The QA model needs to abstain from
prediction for the corresponding questions for the missing slots. To analyze
the abstaining capability of pre-trained QA models, we consider some of the
top-performing zero-shot models T0-3B Sanh et al. (2021), GPT-3 Brown et al.
(2020), and Codex Chen et al. (2021) and experiment on a 500 sample subset of
unanswerable questions from the SQuAD dataset Rajpurkar et al. (2018). We
constrain the models to generate either from the context or from a set of
manually created phrases indicating that the question cannot be answered. We
use the prompts and phrases mentioned in Sanh et al. (2021); Brown et al.
(2020); Chen et al. (2021) and observe the accuracy of all models to be
$<5\%$. We notice that the LLMs frequently hallucinate and generate answers
for unanswerable questions. In our experiments section 4.5, we also consider a
log-likelihood-based threshold for abstaining and we show that this threshold
is difficult to tune using public QA datasets.

To address this challenge, we leverage multiple publicly available extractive
and abstractive QA datasets111The QA datasets details are mentioned in
Appendix A.1 to train Abstainer, a QA model capable of abstaining from
prediction. Specifically, we generate synthetic unanswerable training samples
by modifying existing QA data, and train a QA model jointly on the existing
datasets and the synthetic unanswerable questions. For every (question,
answer, context) triplet, we generate synthetic unanswerable questions by
either (1) removing the sentence containing the answer span from the context,
or (2) randomly sampling a context that doesn’t have the same question. For
example, as shown in Figure 3, we generate the first unanswerable sample by
removing the sentence “Tagore was born in Calcutta” containing the answer
Calcutta from the context and second unanswerable sample by switching the
context of two original answerable samples. We augment these synthetic
unanswerable training samples to the existing datasets and train our
Abstainer. After training the Abstainer, we prompt it for each slot with its
corresponding question for slot value prediction. The prompt has the following
format:

Answer the following question depending on the context. context: A user said,
{utterance}. question: {slot question} answer:

#### Nested Intents

To identify nested intents, we assume knowledge of the candidate nested
intents that can be accommodated by each slot, represented by the slot-to-
candidate-nested-intent mapping $\mathrm{S2NI}$: $\mathcal{S}$ $\rightarrow$
$\mathcal{P}(\mathcal{I})$. Our method assumes that the depth of output
representations is at most 4 i.e. nested intents cannot further have more
nested intents.

Since we assume access to slots that accommodate nested intents, we query for
possible nested intents for all such slots from their respective slot values.
One intuitive way is to prompt the LLM for nested intent with the intent
prediction prompt. However, our unconstrained generation-based intent model
would predict many false positive nested intents. We instead use Abstainer to
prompt for their respective slots. If any slot value is identified, we
consider its corresponding intent via slot-to-intent mapping $\mathrm{S2I}$ to
be present as well.

###  3.4 ZeroTOP: Putting it all together

The pseudo-code of the proposed pipeline is mentioned in Algorithm-1. ZeroTOP
employs a top-down, greedy prompting strategy, where we first prompt for
intent and then, their respective slots. First, we obtain the top-level intent
using the intent model. Based on the predicted intent, we prime the Abstainer
for corresponding slots using their respective questions as prompts, to
identify their slot values. For each identified slot value, we prompt the
Abstainer for slots of candidate nested intents. We use the same prompt format
for this step with the identified slot value now considered as the input
utterance. Finally, we combine predicted intent, identified slot values, and
nested intents to create the meaning representation. We illustrate the
pipeline with an example in Figure 2.

##  4 Experiments

In this section, we evaluate ZeroTOP against multiple pre-trained LLMs under
zero-shot setting.

Dataset | # Samples | # Intents | # Slots
---|---|---|---
MTOP-En | 4386 | 113 | 74
Table 1: Relevant statistics of the dataset used in our experiments.

###  4.1 Datasets

We experiment on the English language subset of MTOP Li et al. (2021) dataset.
MTOP is a multilingual task-oriented semantic parsing dataset comprising data
from 6 languages and 11 domains. The dataset details are mentioned in Table 1.
On average, each intent has 3.6 slots and only one-third of possible slots are
filled for an utterance.

###  4.2 Baselines

We compare ZeroTOP with high-performing zero-shot LLMs as both intent and slot
models. We consider the following models for zero-shot intent prediction:

  1. 1.

T0-3B, GPT-3, Codex constrained are pre-trained T0-3B, GPT-3, Codex models
that are primed with intent generation prompt and the output is constrained to
search over valid intent labels.

  2. 2.

T0-3B constrained calibrated is same as above T0-3B constrained that is
calibrated for intent labels following Zhao et al. (2021). Specifically, we
estimate the model’s bias towards each intent label by prompting with a
content-free test input such as "N/A" and fit calibration parameters
accordingly to cause the prediction for this input to be uniform across
answers.

  3. 3.

RoBERTa-base similarity is an ablated version of ZeroTOP where we assign
intent labels based only on their similarity with user utterance using the
RoBERTa sentence transformer (`stsb-roberta-base`).

  4. 4.

ZeroTOP-Intent is our proposed intent prediction method that generates intents
in an unconstrained fashion by priming T0-3B with intent generation prompt and
assigns labels based on their similarity with intent labels using the RoBERTa
sentence transformer (`stsb-roberta-base`).

We consider the following models for slot prediction:

  1. 1.

GPT-3, Codex, T0-3B constrained are pre-trained GPT-3, Codex, T0-3B models
that are primed with the question corresponding to slot and the output is
constrained to be either from the user utterance or from a set of phrases
indicating that the question cannot be answered. We use their corresponding
phrases mentioned in Brown et al. (2020); Chen et al. (2021); Sanh et al.
(2021). We use the OpenAI API text-davinci-001 for GPT-3 and code-davinci-002
for Codex.

  2. 2.

Abstainer is our finetuned T0-3B model that abstains from prediction. It is
primed with the question corresponding to the slot and the output is
constrained to be either from the user utterance or from the set of phrases
from Sanh et al. (2021) indicating that the question is unanswerable.

Intent Model | Accuracy(%)
---|---
T0-3B constrained | $34.02$
T0-3B constrained calibrated | $36.64$
GPT-3 constrained | $40.44$
Codex constrained | $48.02$
RoBERTa-base similarity | $47.14$
ZeroTOP-Intent | $\mathbf{49.58}$
Table 2: Top-level intent classification results. Accuracy is used as the
evaluation metric.

Intent Model | Slot Model | Acc(%)
---|---|---
GPT-3 constrained | GPT-3 constrained | $3.00^{\textbf{*}}$
Codex constrained | Codex constrained | $5.40^{\textbf{*}}$
T0-3B constrained | T0-3B constrained | $2.42$
Abstainer | $11.81$
RoBERTa-base similarity | T0-3B constrained | $3.88$
Abstainer | $12.90$
ZeroTOP-Intent | T0-3B constrained | $4.10$
Abstainer | $\mathbf{15.89}$

Table 3: Complete meaning representation match evaluation. Accuracy is used as
the evaluation metric. To limit API cost, we limit GPT-3 and Codex evaluation
on a 500-example subset, and hence their results are not directly comparable.

###  4.3 Experiment Settings

We evaluate on the zero-shot setting, therefore we have no training data. We
manually create questions for slots $\mathcal{Q}$ by looking at one example
per slot. For training Abstainer, we fine-tune T0-3B on the extractive and
abstractive QA datasets for $1$ epoch with a constant learning rate of
$10^{-4}$. The Abstainer is fine-tuned on $411732$ answerable and $435898$
unanswerable samples. The batch size is $32$ and each batch contains an equal
number of answerable and unanswerable samples. We used 8 $\times$ NVIDIA Tesla
V100 for our experiments. We use complete meaning representation match
accuracy as the performance metric.

###  4.4 Results and Discussion

We present intent classification results in Table 2 and complete meaning
representation evaluation results in Table 3.

From Table 2, we observe that ZeroTOP-Intent performs significantly better
than constrained T0-3B, GPT-3, and Codex for zero-shot intent classification.
We found that constrained T0-3B is biased towards certain labels. For example,
it predicts CREATE$\\_$CALL (make call), SEND$\\_$Message (send message), and
CREATE$\\_$REMINDER (create reminder), as intent for more than $90\%$ of the
data in call, message, reminder domains. Our proposed unconstrained
formulation lets the model freely express the intent and, computing similarity
later with the intent labels addresses this bias.

As shown in Table 3, the combination of ZeroTOP-Intent model and Abstainer
demonstrates superior performance than alternative combinations. We observe
that T0-3B, GPT-3 and Codex all fail to abstain frequently. The T0-3B model
abstains only for $38\%$ of unanswerable slot questions whereas our Abstainer
is able to abstain for $89\%$ of the unanswerable questions. As a result, we
observe a notable performance gain by plugging in Abstainer as the slot model
for each intent model baseline. This abstaining capability compounded with
unconstrained generation-based intent prediction results in a significant
performance gain.

###  4.5 Confidence score-based Abstainer study

(a) F1 score vs NLL threshold on MTOP dataset

(b) F1 score vs NLL threshold on SQuAD dataset

Figure 4: We consider negative log-likelihood (NLL) as a confidence score and
vary the threshold to abstain from prediction and plot F1 scores on the MTOP
dataset. We show that this NLL threshold is difficult to tune using public QA
datasets such as SQuAD as performance on answerable and unanswerable subsets
is mutually exclusive.

We can alternatively have LLMs abstain from prediction based on a confidence
score based threshold. We consider negative log likelihood (NLL) of the
predicted slot value as the confidence score and abstain from prediction if it
is greater than the threshold. We experiment on slot value prediction task
with T0-3B, Codex, and GPT3 as LLMs and plot macro F1 scores for multiple NLL
thresholds on a randomly sampled subset of 500 samples from MTOP dataset in
Figure 4(a). Specifically, we consider the gold intent of each sample and
prime LLM for extracting slot values for each slot of the gold intent. We
consider F1 score as the metric due to the label imbalance across possible
slot values. We present the F1-score of the Abstainer for reference. First, we
observe that Abstainer is significantly better than T0-3B and GPT3 for all
confidence thresholds. Second, we notice that there is no threshold that
consistently results in good performance for all LLMs, which implies that this
has to be individually tuned for each LLM. Finally, we observe Codex performs
better than Abstainer for some thresholds. As our problem setting includes no
annotated data, we investigate whether we can infer the optimal threshold for
Codex using public QA datasets. Specifically, we consider 500 answerable and
500 unanswerable QA pairs from SQuAD dataset and plot F1 scores with a range
of confidence thresholds in Figure 4(b). We can observe that the performance
on answerable and unanswerable subsets is mutually exclusive i.e. there is no
threshold where the performance on both answerable and unanswerable subsets is
high. The range of thresholds that result in the best performance on the whole
set (highlighted in green) does not transfer to MTOP and is achieved at the
cost of unanswerable set where the F1 score is less than 5%. Given the
difficulty in tuning threshold and the API costs of Codex, we believe using
Abstainer as the slot model to be a better choice.

###  4.6 Annotation Effort Analysis

Model | Accuracy(%)
---|---
T5-3B parser | $8.19$
ZeroTOP | $\mathbf{15.89}$
Table 4: Annotation effort analysis. ZeroTOP outperforms T5-3B parser trained
on samples used for annotating questions $\mathcal{Q}$, justifying our effort
for creating questions for ZeroTOP.

We use 74 samples i.e. one example per slot to design questions for slots. To
analyze the annotation effort, we train an end-to-end parser using these 74
samples and compare it against our method. We fine-tune T5-3B model Raffel et
al. (2020) on these samples with user utterance as input and its corresponding
meaning representation as output. The complete match accuracy on the MTOP
dataset is shown in Table 4. We observe that ZeroTOP performs significantly
better than the parser trained on these 74 samples, justifying the annotation
effort to create questions for ZeroTOP.

Strategy | Accuracy(%)
---|---
Greedy | $15.89$
Beam search (k=3) | $16.86$
Table 5: Greedy and Beam search prompting strategies are compared. We observe
that beam search-based prompting can improve performance when validation data
is provided.

###  4.7 Greedy vs Beam search

In our proposed method, we employ a greedy strategy where we hierarchically
prompt for top-level intent and for its corresponding slots. We compare it
with the beam search strategy with beam size $k$ for every prediction.
Specifically, we consider $k$ top-level intents and prompt for their
corresponding slots, consider top-$k$ slot values for every slot and finally
compute the best meaning representation based on their aggregated NLL scores.
The NLL score of intent $\mathcal{I}_{m}$, its slots
$\mathcal{S}_{j}\in\mathrm{I2S}(\mathcal{I}_{m})$, and their corresponding
slot values $\mathrm{slotValues}[\mathcal{S}_{j}]$ is aggregated as follows:

$\displaystyle\alpha\log
p(\mathcal{I}_{m})+(1-\alpha)\sum_{\mathcal{S}_{j}\in\mathrm{I2S}(\mathcal{I}_{m})}\log
p(\mathrm{slotValues}[\mathcal{S}_{j}]|\mathcal{I}_{m})$

where $\alpha$ is tuned on a held-out validation set. Note that
$p(\mathrm{slotValues}|\mathcal{I}_{m})$ is computed recursively for its
nested intents. The performance comparison is reported in Table 5. We observe
that beam search can further improve the performance when validation data is
provided.

##  5 Conclusion

In this paper, we explore zero-shot semantic parsing using large language
models. We propose ZeroTOP that decomposes semantic parsing task into
abstractive and extractive QA tasks and leverage LLMs to answer them. For
identifying top-level intent, we view it as an abstractive QA task and propose
to generate an answer in an unconstrained fashion and infer the intent label
most similar to the generated description. We demonstrate that current LLMs
fail to abstain from prediction for unanswerable questions, which leads to
poor performance. To address this challenge, we train Abstainer using public
QA datasets, that is capable of identifying unanswerable questions and
abstaining from prediction. Extensive experiments on the MTOP dataset show
significant improvements over pre-trained LLMs.

## References

  * Bartolo et al. (2020) Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020.  Beat the ai: Investigating adversarial human annotation for reading comprehension.  _Transactions of the Association for Computational Linguistics_ , 8:662–678.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Dasigi et al. (2019) Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019.  Quoref: A reading comprehension dataset with questions requiring coreferential reasoning.  In _EMNLP_.
  * Drozdov et al. (2022) Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022.  Compositional semantic parsing with large language models.  _arXiv preprint arXiv:2209.15003_.
  * Gao et al. (2020) Shuyang Gao, Sanchit Agarwal, Tagyoung Chung, Di Jin, and Dilek Hakkani-Tur. 2020\.  From machine reading comprehension to dialogue state tracking: bridging the gap.  In _ACL 2020 Workshop on NLP for Conversational AI_.
  * He et al. (2015) Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015.  Question-answer driven semantic role labeling: Using natural language to annotate natural language.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 643–653, Lisbon, Portugal. Association for Computational Linguistics.
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _arXiv preprint arXiv:2111.09543_.
  * Li et al. (2021) Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021.  MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2950–2962, Online. Association for Computational Linguistics.
  * Lin et al. (2019) Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019.  Reasoning over paragraph effects in situations.  _ArXiv_ , abs/1908.05852.
  * Lin et al. (2021a) Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul A Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, et al. 2021a.  Zero-shot dialogue state tracking via cross-task transfer.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7890–7900.
  * Lin et al. (2021b) Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul A Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. 2021b.  Leveraging slot descriptions for zero-shot cross-domain dialogue statetracking.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5640–5648.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Rubino et al. (2022) Melanie Rubino, Nicolas Guenon des Mesnards, Uday Shah, Nanjiang Jiang, Weiqi Sun, and Konstantine Arkoudas. 2022.  Cross-TOP: Zero-shot cross-schema task-oriented parsing.  In _Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing_ , pages 48–60, Hybrid. Association for Computational Linguistics.
  * Saha et al. (2018) Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. 2018\.  DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension.  In _Meeting of the Association for Computational Linguistics (ACL)_.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_.
  * Schucher et al. (2022) Nathan Schucher, Siva Reddy, and Harm de Vries. 2022.  The power of prompt tuning for low-resource semantic parsing.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 148–156, Dublin, Ireland. Association for Computational Linguistics.
  * Shin et al. (2021) Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021.  Constrained language models yield few-shot semantic parsers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7699–7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sun et al. (2019) Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019.  Dream: A challenge data set and models for dialogue-based reading comprehension.  _Transactions of the Association for Computational Linguistics_ , 7:217–231.
  * Tafjord et al. (2019) Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019.  Quartz: An open-domain dataset of qualitative relationship questions.  _arXiv preprint arXiv:1909.03553_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32.
  * Xiong et al. (2019) Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019.  TWEETQA: A social media focused question answering dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5020–5031, Florence, Italy. Association for Computational Linguistics.
  * Yang et al. (2022) Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin, and Diyi Yang. 2022.  SEQZERO: Few-shot compositional semantic parsing with sequential prompts and zero-shot models.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 49–60, Seattle, United States. Association for Computational Linguistics.
  * Zhao et al. (2022) Wenting Zhao, Konstantine Arkoudas, Weiqi Sun, and Claire Cardie. 2022.  Compositional task-oriented parsing as abstractive question answering.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4418–4427, Seattle, United States. Association for Computational Linguistics.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_ , pages 12697–12706. PMLR.

##  Appendix A Appendix

###  A.1 QA Datasets for Training Abstainer

We use multiple extractive and abstractive QA datasets to generate synthetic
unanswerable samples and train Abstainer. The details about datasets are
mentioned in Table 6.

Type | Dataset | # Samples
---|---|---
Extractive | Adversarial QA Bartolo et al. (2020) | $36000$
QA-SRL He et al. (2015) | $8597$
DuoRC Saha et al. (2018) | $186089$
ROPES Lin et al. (2019) | $14000$
SQuADv2 Rajpurkar et al. (2018) | $150000$
Quoref Dasigi et al. (2019) | $24000$
Abstractive | ReCoRD Wang et al. (2019) | $121000$
DREAM Sun et al. (2019) | $10197$
QuaRTz Tafjord et al. (2019) | $3864$
Tweet-QA Xiong et al. (2019) | $10692$

Table 6: Relevant statistics of the QA dataset used to train Abstainer.