#  Can Your Model Tell a Negation from an Implicature?
Unravelling Challenges With Intent Encoders

Yuwei Zhang† Siffi Singh‡ Sailik Sengupta‡ Igor Shalyminov‡
Hang Su‡ Hwanjun Song¶∗ Saab Mansour‡
†University of California, San Diego ‡ WS AI Labs ¶KAIST, Republic of Korea
[regular]envelope {siffis<EMAIL_ADDRESS>Work conducted as in an
intern/employee of Amazon.

###### Abstract

Conversational systems often rely on embedding models for intent
classification and intent clustering tasks. The advent of Large Language
Models (LLMs), which enable instructional embeddings allowing one to adjust
semantics over the embedding space using prompts, are being viewed as a
panacea for these downstream conversational tasks. However, traditional
evaluation benchmarks rely solely on task metrics that don’t particularly
measure gaps related to semantic understanding. Thus, we propose an intent
semantic toolkit that gives a more holistic view of intent embedding models by
considering three tasks– (1) intent classification, (2) intent clustering, and
(3) a novel triplet task. The triplet task gauges the model’s understanding of
two semantic concepts paramount in real-world conversational systems– negation
and implicature. We observe that current embedding models fare poorly in
semantic understanding of these concepts. To address this, we propose a pre-
training approach to improve the embedding model by leveraging augmentation
with data generated by an auto-regressive model and a contrastive loss term.
Our approach improves the semantic understanding of the intent embedding model
on the aforementioned linguistic dimensions while slightly effecting their
performance on downstream task metrics.

Can Your Model Tell a Negation from an Implicature?
Unravelling Challenges With Intent Encoders

Yuwei Zhang†††thanks:  Work conducted as in an intern/employee of Amazon.
Siffi Singh‡ Sailik Sengupta‡ Igor Shalyminov‡ Hang Su‡ Hwanjun Song¶∗ Saab
Mansour‡ †University of California, San Diego ‡ WS AI Labs ¶KAIST, Republic of
Korea [regular]envelope {siffis<EMAIL_ADDRESS>

##  1 Introduction

Conversational systems use intent embedding models to encode input utterances
into vectors that are used to understand intent semantics for few-shot intent
classification and/or intent discovery Zhang et al. (2021); Ma et al. (2022);
Sung et al. (2023). The intent classification task leverages a pre-defined
distance metric to find the nearest intent class/instances to the test
utterance in the embedding space Vinyals et al. (2016); Snell et al. (2017);
Dopierre et al. (2021), while intent discovery considers a clustering
algorithm on top of multiple utterance embeddings to detects novel intent
clusters. While these applications have been studied separately, a common
underlying assumption is the embedding space encodes semantics (and a distance
metric) between utterances and intents. To achieve this, various approaches
have been proposed such as supervised pre-training on labeled utterance data
belonging to a wide range of intents Zhang et al. (2021, 2022a) or by
combining pseudo intent names Sung et al. (2023); Mueller et al. (2022).

However, the evaluation of these embedding models only consider existing
conversational benchmarks (and task metrics) that lack dedicated test data
necessary to evaluate gaps in semantic understanding. In this paper, we
consider a complementary evaluation approach that tries to understand how well
these embedding models capture the semantics of two common linguistic
phenomenon seen in real-world conversational systems: negation – negation
semantics alongside an explicitly mentioned intent utterance (e.g. No, I don’t
want you to play music!) and implicature – utterances indirectly hinting at an
intent that might require some reasoning steps (e.g. I feel like dancing
$\implies$ play some music). For this purpose, we propose an Intent Semantics
Toolkit that includes challenging test splits for existing classification &
clustering tasks and a novel triplet task. The triplet task consider an
utterance triplet $\langle$original, implicature, negation$\rangle$ and
evaluates if the implicature utterance is closer to the original utterance in
the embedding space as compared to negation. The implicature and negation
utterances for the test data are generated using two novel prompt designs for
ChatGPT followed by human-in-the-loop quality control mechanisms.

Figure 1: Intent Semantics Toolkit (top) to benchmark embedding model semantic
concepts capabilities and training pipeline (bottom) to synthesize training
data for improved semantic concepts understanding. For benchmarking data, we
prompt the LLM to generate negation and implicature from original utterances,
which are validated by both automatic and manual quality control. For training
data, we first extract intents from unlabeled utterances, then, we generate
hard examples using the LLM, which will be combined with retrieved utterances
for fine-tuning.

The recent popularity of embeddings derived from Large Language Models and the
possibility of prompt-based encoding give an impression of semantic
understanding, making them seem like an ideal candidate for the aforementioned
intent identification tasks. Our proposed Intent Semantics Toolkit indicates
that current representation of the negation and implicature utterances are far
from perfect (see Figure 2).

To improve an embedding model’s semantic understanding on the aforementioned
linguistic phenomena, we consider a fine-tuning approach that leverages LLM-
generated positive (related to an intent) and negative (unrelated to an
intent) utterances for augmentation alongside a contrastive loss objective.
Our ablations highlight the need for both kinds of utterances for augmentation
and our best model consistently outperforms the original embedding models on
the triplet task. Observing different magnitudes of improvement on the
different tasks, we dive deep to understand correlation between the various
tasks and highlight some negative interactions. This highlights that
approaches to improve embedding models on semantic understanding and
downstream task might need to consider trade-offs; this is in line with
several previous works Marelli et al. (2014); Jeretic et al. (2020); Sengupta
et al. (2021); Cooper Stickland et al. (2023). To summarize, our contributions
are three-fold: (1) We identify two linguistic challenges that are commonly
observed in real-world intent detection systems, but are overlooked in the
literature: negation and implicature.111We note that these phenomenon also
have equivalence in the context of reasoning. (2) We devise Intent Semantics
Toolkit that includes a novel triplet task and exposes the shortcomings of
intent embedding models on semantic understanding of negation and implicature.
We propose prompting strategies to generate evaluation data with ChatGPT and
consider human-in-the-loop quality control. (3) We explore fine-tuning
approaches with automatically generated utterances for data augmentation and
interpret the semantic dimensions of implicature and negation as positive and
negative examples in the contrastive learning loss. The results show that
generated utterances can help improve the performance on the triplet task.

Figure 2:  The instructor-large embeds  original utterances further away from
semantically similar  implicature utterances and closer to semantically
dissimilar  negation utterances, a failure mode for the triplet task (as seen
in the tSNE projection space).

##  2 Related Work

Sentence embedding evaluation. Sentence embeddings are used in various
downstream applications, especially where efficiency is important Reimers and
Gurevych (2019); Su et al. (2023). For instance, semantic search of relevant
document from a large vector database based on a similarity metric Nguyen et
al. (2016); Thakur et al. (2021), clustering a set of ‘unorganized’ documents
based on the semantics MacQueen et al. (1967); Ester et al. (1996), and/or
classification of an input at test time to the semantically closest class seen
during training Vinyals et al. (2016); Snell et al. (2017); Conneau and Kiela
(2018). Recent works have also proposed a benchmark that unifies the
evaluation of sentence embeddings across the various tasks Muennighoff et al.
(2023). Along these lines, Liu et al. (2023) proposes a triplet evaluation
task that examines similarity of triplets sampled from existing class labels.
In our work, we also propose a triplet evaluation task, but consider examining
triplets based on dimensions related to language semantics such as implication
and negation.

Conversational implicature. In conversation, an agent can often imply an
intent via an utterance that doesn’t explicitly specify the intent, but rather
hints at it Grice (1975); Recanati (1989); Zalta et al. (1995). For instance,
the utterance I have not had breakfast today can imply that the person is
hungry and could hint at a particular intent (eg. order food) depending on the
context (eg. saying this to a hotel operator). Previous works have shown that
current language models do not have enough understanding of implicature
Jeretic et al. (2020); Ruis et al. (2022). In this work, we consider
implicature for intent detection scenarios and are especially interested in
the capabilities of embedding models.

##  3 Preliminary

A model ($f$) for embedding encodes input sentences ($u_{i}$) onto a
continuous vector space ($f(u_{i})$). For a good embedding model, a distance
metric $D$ over embeddings (say $f(u_{i})$ and $f(u_{j})$) should be able to
capture some notion of semantic similarity that in turn empowers downstream
applications such as clustering and classification. In this paper, we will
consider $D$ to be the widely popular cosine distance, i.e.
$D(f(u_{i}),f(u_{j}))=1-\frac{f(u_{i})^{T}f(u_{j})}{||f(u_{i})||||f(u_{j})||}$.

While these embedding models are common in several applications, we will
concentrate on intent embedding models, where input sentences are user
utterances (e.g. I would like to order food) and intents (e.g. order food,
cancel order, etc.) comprise of semantic clusters represented by a set of
similar user utterances. In such contexts, embedding models have been used to
perform few/zero-shot classification of test-time utterances to a set-of
predefined intents Snell et al. (2017); Dopierre et al. (2021); Sung et al.
(2023) or to discover novel intent classes given a set of unlabelled user
utterances (Zhang et al., 2022b; Gung et al., 2023).

Despite its success, we observe that these intent embedding models often fail
to capture nuanced language semantics that are common in real-world
conversational agents. While previous work has shown such failures are common
in the context of real-world noise Sengupta et al. (2021); Cooper Stickland et
al. (2023), we focus on two semantic aspects– (1) Negation: utterances that
explicitly express no interest towards a particular intent (e.g. I don’t want
to order a drink.), and (2) Implicature: utterances that do not explicitly
convey an intent but imply it (e.g. I am hungry $\rightarrow$ order_food).
Figure 2 shows that intent embedding models embed negations (that clearly
disregards an intent) closer to intent utterances than implicatures (that
implies an intent). In the real-world, we observe negation often occurs when a
system incorrectly directs a customer towards an intent (and the customer has
to explicitly mention they didn’t intend it), while implicature is common due
to a customer’s incomplete knowledge about the potential functionality
supported by a chatbot.

##  4 Intent Semantics Toolkit

In this section, we first introduce the four evaluation tasks in our toolkit
that analyze the semantic understanding capabilities of intent embedding
models (§4.1). Then, we consider the challenge of obtaining data related to
negation and implicature (§3) and quality assurance procedures to ensure a
high-quality test set (§4.3). Eventually, we show evaluation of SOTA models on
our dataset (§4.4).

###  4.1 Evaluation Tasks

#### Triplet Task

A triplet task is composed of three utterances
$\\{u_{i},u_{i}^{p},u_{i}^{n}\\}$, where $u_{i}$ (I want to order a pizza) is
an utterance belonging to intent $i$ (order_food), $u_{i}^{n}$ is the negation
of $u_{i}$ (I don’t want to order pizza), and $u_{i}^{p}$ is either another
utterance of the same intent (I need pizza) or an implicature one (I am really
hungry). For each triplet, we expect the negated utterance $u_{i}^{n}$ to be
embedded further away from $u_{i}$ than $u_{i}^{p}$, i.e.
$D(f(u_{i}),f(u_{i}^{p}))<D(f(u_{i}),f(u_{i}^{n}))$. As we show later, this is
often difficult for embedding models that may focus on other surface-form
aspects of utterances rather than nuanced semantic understanding of intents.
We calculate success among a set of $N_{T}$ triplets as,

$\frac{1}{N_{T}}\sum_{i=1}^{N_{T}}\mathbb{I}(D(f(u_{i}),f(u_{i}^{p}))<D(f(u_{i}),f(u_{i}^{n})))$
(1)

In principle, we can also define another success rate by interchanging $u_{i}$
and $u_{i}^{p}$ as follows,

$\frac{1}{N_{T}}\sum_{i=1}^{N_{T}}\mathbb{I}(D(f(u_{i}^{p}),f(u_{i}))<D(f(u_{i}^{p}),f(u_{i}^{n})))$
(2)

Given $u_{i}^{p}$ and $u_{i}^{n}$ are not direct negations of one another, the
value of $D(f(u_{i}^{p}),f(u_{i}^{n}))$ in (2) is expected to be higher than
$D(f(u_{i}),f(u_{i}^{n})))$ in (1) above. Hence, (2) is a more relaxed success
criterion. Thus, we denote the (1) as $T_{\text{hard}}$ and (2) as
$T_{\text{easy}}$ hereafter. We will show two cases “Ori-Ori” or “Ori-Imp”
where $u^{p}_{i}$ is from either the original test set or the implicature set.

#### Binary Classification

In the binary classification task, an utterance $u_{i}$ needs to be classified
into either original intent $i$ or a negated intent class$\neg i$. Thus,
success implies $D(f(u_{i}),f(i))<D(f(u_{i}),f(\neg i))$. And the success rate
is calculated among a set of $N_{B}$ utterances,

$\frac{1}{N_{B}}\sum_{i=1}^{N_{B}}\mathbb{I}(D(f(u_{i}),f(i))<D(f(u_{i}),f(\neg
i)))$ (3)

We compute success rates for three different sets of utterances– original and
implicature (closer to $f(i)$) and negation (closer to $f(\neg i)$).

#### Clustering

The input to the task is a set of unlabeled utterances $\\{u_{i}\\}$. A
clustering algorithm $\mathcal{C}$ then takes the embeddings $\\{f(u_{i})\\}$
as inputs and outputs clustering indices $\\{q_{i}\\}$ where number of
clusters $k$ is first specified. Since the permutation of $\\{q_{i}\\}$ might
be different from labels, we measure normalized mutual information (NMI)
Estévez et al. (2009) between them as our metric. We show results on both
original and implicature sets with two clustering algorithms: $k$-means
MacQueen et al. (1967) and agglomerative clustering Nielsen and Nielsen
(2016).

#### Multi-class Classification

We adopt ProtoNet Snell et al. (2017) as the classifier for few-/zero- shot
classification aligning with previous works Dopierre et al. (2021); Sung et
al. (2023). It takes as inputs a training set
$\\{\tilde{u}_{j},\tilde{y}_{j}\\}$ and label names $\\{l_{c}\\}$. A class
prototype is first calculated for each class by averaging the embeddings of
utterances of that intent class and the label name,

$p_{c}=\frac{1}{N_{c}+1}\\{[\sum_{\tilde{y}_{j}=c}f(\tilde{u}_{j})]+f(l_{c})\\}$
(4)

During test time, we simply find the prototype that is closest to the test
utterance $\operatorname*{argmin}_{c}D(p_{c},f(u_{i}))$. Different from
previous works, we discard episodic training and always use full categories
within the dataset instead of random $M$ ways. We argue that such a setting is
more realistic and show results on both original and implicature sets with
$0/10$-shots.

###  4.2 Data Generation

In this section, we show data generation workflows with ChatGPT222Both
gpt-4-0613 and gpt-3.5-turbo-0613 are used. models (in order to achieve high
quality) on test splits of $3$ intent-classification benchmarks: BANKING77
Casanueva et al. (2020), HWU64 Liu et al. (2021) and CLINC150 Larson et al.
(2019). Across all datasets, we needed to make the intent names more
informative (e.g. “mpg” to “check car mpg”) and focused on four dimensions
when optimizing prompts for the LLM-generation: (R1) faithful: The generated
utterances correctly conveys/negates the target intent, (R2) realistic: The
generated utterances can be said by a real-world customer, (R3) diverse: The
generated utterances are diverse enough for evaluation on a large scale, and
(R4) reproducible: The temperature was set to $0$ to guarantee
reproducibility.

Negation. Directly negating an utterance without a target intent has lower
guarantee on generation faithfulness. For instance, consider the utterance
“are there restrictions for carry-ons on Delta” that belongs to the intent
“carry on”. The generated negation utterance is “are there no restrictions for
carry-ons on delta” with gpt-4-0613 and temperature set to $0$, which still
belongs to the same intent. In order to reduce the task complexity for LLMs,
we first manually write a few negated intents for each original intent. And
then we instruct the LLM to directly modify original utterances according to a
randomly sampled negated intent corresponding to the original one. To further
increase the generation quality, we provide $6$ in-context examples in the
prompt manually written by humans. We choose gpt-4-0613 for negation.

Implicature. Implicature utterances are completely generated by LLMs without
original utterances provided. In order to achieve diversity and realisticness,
we first ask an LLM to brain-storm $10$ scenarios that a customer may
encounter scenarios for a certain intent, where the scenario may contain
various roles and situations. We choose gpt-3.5-turbo-0613 for this part since
the preliminary results show sufficient quality. Then, we use another LLM to
generate $3$ utterances for each scenario. In addition, we provide the
definitions of implicature and $3$ manually written in-context examples for
higher quality. We choose gpt-4-0613 for the latter part.

###  4.3 Data Quality Control

We first employ automatic metrics to understand the generated data. To achieve
this, we use training set as reference and calculate BLEU Papineni et al.
(2002), ROUGE-L Lin (2004), METEOR Banerjee and Lavie (2005) and BertScore
Zhang et al. (2020). We show the measures on three sets separately in Figure
3. As expected, original set has the largest vocabulary overlap with training
set, probably because the original utterances are generated in the same style
with lack of variability. Negation set contains fewer vocabulary overlap but
still significantly higher than implicature set, which demonstrates they are
similar to original ones in surface form.

Figure 3: Similarity metrics between the training data and the original, and
(generated) negation and implicature test splits. Averaged results show the
negation utterances are closer to the original ones on surface form than the
implicature ones.

However, merely measuring vocabulary overlap does not necessarily guarantee
the faithfulness of generated data. Hence, we conduct human evaluation
centered around two questions: (1) “Can the utterance imply the intent?” (2)
“If yes, is it conveyed explicitly?” We first sample $180$ utterances
containing $60$ from original, $60$ from negation and $60$ from implicature. 8
human annotators from our internal team (with fluent English skills) provided
binary answers for these two questions in a sequential manner333We also allow
them to annotate “unsure”.. We show the annotation guidelines in Figure 6. For
each utterance, $3$ annotations are gathered, and those without $3$
annotations are filtered out. We take the majority as the final decision. In
Table 1, we show the results for all three sets. Both negation and implicature
generations are indeed faithful. Furthermore, $88.46\%$ of original set are
annotated as explicit while $62.50\%$ of implicature set are annotated as not
explicit. Finally, the inter-annotator agreement (i.e. the proportion of
queries with $3$ consistent answers and with those that have less than $3$
answers filtered) for Q1 is $79.10\%$ and Q2 is $62.11\%$, which shows
consensus in Q1 while some conflicts in Q2 probably due to its subjectivity.

Q1: convey intent?
---
original | negation | implicature
59 / 59 (100%) | 9 / 59 (15.25%) | 52 / 59 (88.14%)

Q2: explicitly conveyed?
---
original | negation | implicature
46 / 52 (88.46%) | - | 15 / 40 (37.50%)

Table 1: Human evaluation of the quality of the automatically generated
negation and implicature utterances. We note that negation mostly do not
convey the original intent while implicature do (as expected). In addition,
from Q2, we note that the implicature utterances are challenging (mostly
implicit).

| Original | Intent Semantic Toolkit
---|---|---
Model | Clustering | Multi-class | Triplet (Ori-Ori) | Triplet (Ori-Imp) | Binary Classification | Clustering | Multi-class
| KM | Agg | 0-shot | 10-shot | $T_{hard}$ | $T_{easy}$ | $T_{hard}$ | $T_{easy}$ | Ori | Imp | Neg | KM | Agg | 0-shot | 10-shot
paraphrase | 81.7 | 83.5 | 61.1 | 83.3 | 22.6 | 84.8 | 3.9 | 68.9 | 77.6 | 57.3 | 82.4 | 57.2 | 58.8 | 22.2 | 28.2
IAE | 83.4 | 84.7 | 66.6 | 84.7 | 24.0 | 84.3 | 3.2 | 67.3 | 86.6 | 70.1 | 79.6 | 58.3 | 59.9 | 25.4 | 30.1
instructor-base | 83.8 | 84.9 | 67.5 | 85.8 | 19.1 | 86.1 | 2.0 | 68.0 | 89.1 | 67.3 | 78.4 | 57.9 | 59.2 | 26.2 | 30.9
instructor-large | 84.3 | 86.0 | 67.6 | 86.2 | 23.4 | 87.5 | 3.6 | 71.0 | 89.6 | 73.5 | 87.4 | 59.1 | 61.4 | 28.8 | 34.3

Table 2: Model performance on the original tasks and our proposed Intent
Semantic Toolkit with $4$ popular intent encoders. Despite promising
performance on the originaldatasets, our toolkit reveals a lack of
understanding on negation and implicature.

###  4.4 Evaluation Results

We report results on $3$ intent encoders. (1) paraphrase is a vanilla
Sentence-BERT Reimers and Gurevych (2019)
model444https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2
which shows strong performance on intent detection (used as an initialization
in Sung et al. (2023)). (2) IAE Sung et al. (2023) first employs an intent-
role labeler to extract pseudo intent names and then optimizes contrastive
objective leveraging utterances, pseudo intent names and golden intent names.
(3) Instructor Su et al. (2023) pre-trains on multi-task dataset with
instructions (“Represent the purpose for retrieval: ”) pretended on each input
text. We experiment with both the base and large versions.

The evaluation results in Table 2 shows (1) success rates from $T_{hard}$ are
consistently low ($<25\%$) indicating positive utterances (i.e. utterances
with the same intent) are further away that negations. In contrast, success
rates for $T_{easy}$ are much higher, highlighting implicature and negation
utterances are far away from one another; (2) for binary classification,
performances on the implicature split are lower than those on the original set
showcasing the former is set of utterances are further awar for the expected
intent than the latter; (3) for clustering and classification, performances on
implicature sets are consistently lower than the original sets, however, this
might be due to the multi-labeled nature of the implicature utterances. In
Appendix A, we verify the upper bound with gpt-4-0613 on 10-shot
classification task and demonstrates there is still a large room for
improvement; (4) instructor-large consistently improves upon its base model,
and instructor-base that possesses a similar parameter size with IAE seems to
outperform the latter on most tasks. Given this, we use instructor-large as
the baseline embedding model for improvement.

##  5 Model Improvement

We seek to improve the semantic understanding of embeddings models to negation
and implicature. To answer this, we first introduce a data curation procedure
where new utterances are generated based on an unlabeled dialogue corpus
(§5.1) and then investigate continued fine-tuning approaches with a
contrastive learning loss objective adapted for the triplet task (§5.2). The
lower part of Figure 1 given a diagrammatic overview.

| Original | Intent Semantic Toolkit
---|---|---
Model | Clustering | Multi-class | Triplet (Ori-Ori) | Triplet (Ori-Imp) | Binary Classification | Clustering | Multi-class
| KM | Agg | 0-shot | 10-shot | $T_{hard}$ | $T_{easy}$ | $T_{hard}$ | $T_{easy}$ | Ori | Imp | Neg | KM | Agg | 0-shot | 10-shot
Baseline | 84.3 | 86.0 | 67.6 | 86.2 | 23.4 | 87.5 | 3.3 | 70.6 | 89.6 | 73.9 | 79.4 | 62.2 | 64.4 | 29.2 | 34.1
Disable LLM | 84.1 | 85.0 | 71.7 | 85.5 | 39.3 | 86.2 | 8.0 | 50.8 | 89.2 | 53.0 | 84.8 | 65.5 | 66.9 | 31.7 | 34.3
Ours best | 84.6 | 86.8 | 73.4 | 87.2 | 51.1 | 93.7 | 20.4 | 77.6 | 94.0 | 73.6 | 83.1 | 65.9 | 68.2 | 33.9 | 37.2

Table 3: Main results for instructor-large on the test set. “Ours best”
corresponds to “$-P^{4},N^{1,3}$” model in Table 6. Our model achieves better
performance on both original and our proposed toolkit.

|  | Original | Intent Semantic Toolkit
---|---|---|---
Model | Rank | Clustering | Multi-class | Triplet (Ori-Ori) | Triplet (Ori-Imp) | Binary Classification | Clustering | Multi-class
|  | KM | Agg | 0-shot | 10-shot | $T_{hard}$ | $T_{easy}$ | $T_{hard}$ | $T_{easy}$ | Ori | Imp | Neg | KM | Agg | 0-shot | 10-shot
Baseline | 14 | 84.0 | 85.4 | 67.3 | 86.1 | 22.5 | 87.9 | 4.1 | 72.1 | 90.1 | 73.2 | 78.9 | 61.6 | 63.1 | 25.9 | 28.3
All $\\{P^{*},N^{*}\\}$ | 7 | 84.6 | 85.8 | 69.9 | 86.5 | 36.9 | 83.7 | 12.6 | 56.0 | 73.9 | 35.8 | 90.7 | 64.2 | 66.1 | 27.7 | 33.2
$-P^{4},N^{1,3}$ | 1 | 84.7 | 86.7 | 73.5 | 87.8 | 53.3 | 94.9 | 20.8 | 78.4 | 93.9 | 73.9 | 82.3 | 66.2 | 68.3 | 34.2 | 37.6
$-N^{*}$ | 6 | 85.8 | 87.2 | 70.3 | 87.5 | 16.7 | 72.2 | 3.1 | 40.9 | 58.9 | 27.7 | 89.7 | 63.6 | 65.7 | 28.1 | 33.4
$-P^{*}$ | 12 | 80.6 | 81.0 | 66.4 | 83.0 | 56.7 | 91.2 | 22.8 | 64.8 | 90.7 | 62.0 | 85.1 | 62.6 | 64.7 | 25.8 | 30.3
$-LLM$ | 15 | 83.6 | 84.4 | 69.2 | 85.5 | 41.8 | 87.2 | 8.9 | 51.6 | 89.1 | 53.5 | 84.0 | 63.4 | 65.4 | 27.3 | 32.1

Table 4: Ablation study for various prompts with instructor-large on dev set.
Negative sign represents disabling. For instance, “$-P^{*}$” means disabling
all hard positive prompts and only using retrieved positive utterances.
“$-LLM$” means disabling data generated from LLM and only using retrieved
utterances. “$-P^{4}-N^{1,3}$” means disabling prompts $P^{4}$, $N^{1}$ and
$N^{3}$. Numbers in the brackets of first column are the rankings Colombo et
al. (2022).  green
---
represents increased score compared with “Baseline” (vanilla instructor-large)
and  red
---
vice versa.

###  5.1 Fine-tune Data Curation

Similar to Sung et al. (2023), we first collect a set $252,744$ unique and
unlabelled utterances from general domain dialogue datasets: MultiWOZ Zang et
al. (2020), SGD Rastogi et al. (2020), TOP Gupta et al. (2018) and TOPv2 Chen
et al. (2020). We then use a performant LLM, namely falcon-40b-instruct
Almazrouei et al. (2023) for the sub-modules in Algorithm 1.555We opt out of
using OpenAI models due to their restrictions on distilling data for model
training, see §Limitations for details. Another rationale for these design
choices is that they ensure a separation between training and evaluation data,
thereby demonstrating the model’s generative capabilities.

As highlighted in Algorithm 1, for each unlabelled utterance, we first use the
LLM to generate a user goal using the prompt what does a customer want by
saying ‘$u_{i}$’? Then, we use a dependency parser666https://spacy.io/ to
extract the action (ROOT) or object (dobj) tokens given the output goal
phrase. If the object is not found, we prompt the LLM to summarize an object
token. The final tuple is then used as a prefix for the utterance generation
(see Figure 5 for an example) that is eventually used for finetuning.

Input: A set of unlabeled utterances $\\{u_{i}\\}_{i=1}^{N}$.

1 $s\leftarrow\\{\\}$

2 for  _i=1:N_ do

3 $g_{i}\leftarrow GoalGeneration(u_{i})$

4 $(a_{i},o_{i})\leftarrow DependencyParser(g_{i})$

5 if  _ $o_{i}$ is None _ then

6 $o_{i}=SummarizeObject(u_{i},g_{i},a_{i})$

7 $s\leftarrow s\cup(u_{i},g_{i},a_{i},o_{i})$

Output: s

Algorithm 1 Intent Extraction Pipeline

To further generate hard positive/negative utterances, we propose to utilize a
“zoo” of prompts written by experts that asks the LLM to imagine itself as the
customer and generate hard positives or negatives. And then at each time, we
sample two prompts, one for generating positive and one for generating
negative. Table 9 in Appendix gives an overview of all the prompts we used for
fine-tuning in this work. The generated utterances will be diversified before
training by switching common phrases such as “want to”. Apart from that, we
also “retrieve” one positive utterance that has the same action-object pairs
and one negative utterance that has different action-object pairs but close in
cosine distance777We always use the same embedding model for utterance
retrieving and fine-tuning. See more details in Appendix B.

###  5.2 Fine-tuning Objective

We adopt fine-tuning objective from Su et al. (2023); Zhang et al. (2023)
where our input is a batch $\mathcal{B}$ of triplets $\\{u,u^{p},u^{n}\\}$
generated in the previous section (§5.1) and placed within the same
instruction template used in the triplet task evaluation (in §4.4). More
precisely, if we let $i$ and $j$ denote indices of $\mathcal{B}$, the
objective function pulls together the original utterance $u_{i}$ and it
corresponding positive example $u_{i}^{p}$ while pushing away all in-batch
negative $u_{j}^{n}\forall j\in\mathcal{B}$. Precisely, we consider the
following loss function:

$\begin{split}l_{i}&=\frac{\exp(s(f(u_{i}),f(u_{i}^{p}))/\gamma)}{\sum_{j\in\mathcal{B}}\exp(s(f(u_{i}),f(u_{j}^{n}))/\gamma)}\\\
&+\frac{\exp(s(f(u_{i}^{p}),f(u_{i}))/\gamma)}{\sum_{j\in\mathcal{B}}\exp(s(f(u_{i}^{p}),f(u_{j}^{n}))/\gamma)}\end{split}$
(5)

where $\gamma$ is a temperature parameter. The second term swaps the positive
and anchor.

| Original | Intent Semantic Toolkit
---|---|---
Model | Clustering | Multi-class | Triplet (Ori-Ori) | Triplet (Ori-Imp) | Binary Classification | Clustering | Multi-class
| KM | Agg | 0-shot | 10-shot | $T_{hard}$ | $T_{easy}$ | $T_{hard}$ | $T_{easy}$ | Ori | Imp | Neg | KM | Agg | 0-shot | 10-shot
IAE | 83.4 | 84.6 | 66.3 | 84.5 | 25.3 | 84.9 | 3.5 | 68.1 | 87.3 | 70.5 | 80.0 | 61.9 | 64.2 | 25.5 | 30.4
IAE-pos | 83.6 | 85.0 | 66.1 | 84.4 | 27.7 | 85.8 | 4.3 | 68.9 | 87.8 | 70.9 | 80.9 | 62.1 | 64.5 | 26.2 | 30.9
IAE-pos-neg | 83.0 | 84.2 | 65.6 | 84.4 | 40.6 | 90.8 | 13.7 | 78.4 | 91.3 | 76.1 | 81.5 | 62.1 | 65.1 | 25.6 | 30.6

Table 5: Adding LLM generated hard positive/negative to Intent-Aware-Encoder
(IAE) Sung et al. (2023) pre-training. Results averaged over $3$ random seeds.

##  6 Experimental Results

In this section, we first evaluate our fine-tuned models in §5 on our proposed
Intent Semantics Toolkit. We then select the best prompt combination for
training models on the validation set, and then evaluate on the test set.
Finally, we shed lights on the correlations between tasks in §6.3 and the
effectiveness of LLM-augmented utterances when combining intent-aware encoder
training Sung et al. (2023) in §6.4.

###  6.1 Experimental Setup

We follow fine-tuning parameters in Su et al. (2023), except that we choose
smaller learning rate $4\times 10^{-6}$, batch size $8$, maximum sequence
length $128$ and train with one epoch. These parameters are consistent across
all experiments in this paper.

###  6.2 Main Results

The fine-tuned models are evaluated on our proposed Intent Semantics Toolkit
in Section 4. We split the evaluation data into $50\%$ for dev and $50\%$ for
test to select proper prompt combinations. We report results based on the test
set using instructor-large as the baseline in Table 3 and highlight some
ablations study with various prompts in Table 4 (see Appendix for experiments
with other prompts and instructor-base). As It is difficult to exhaust all
possible combinations of prompts, we consider a subset of experiments in our
ablations and rank the various trained models using Colombo et al. (2022). We
observe that (1) disabling all LLM-generated data achieves lower performance
than Baseline, (2) disabling LLM-generated hard positive degrades performances
on clustering and classification tasks, while disabling hard negative degrades
performances on triplet tasks and binary classification tasks, and (3) the
best performance is achieved by “$-P^{4}-N^{1,3}$”, which outperforms the
Baseline on most tasks. As seen in Table 3, while we notice consistent
improvement with our model across original and our proposed tasks, the
magnitude of improvements is not uniform (eg. a $\uparrow 27.7$ on $T_{hard}$
vs. $\downarrow 0.3$ on the binary task with implicatures). To understand this
better, we consider the correlation between tasks.

###  6.3 Correlation between tasks

Figure 4: Pearson correlations between tasks using performances on dev set as
a feature vector for each task.

We plot the pearson correlation matrix between pairs of tasks across
instructor-large (Table 4) and instructor-base (Table 8) dev set performances.
Figure 4 clearly highlights the negative correlation between the first two
tasks (triplet tasks and binary classification) and the last two (clustering
and classification). Thus, it may be important to consider trade-offs in
improving an embedding model across all tasks that we leave as future work.

###  6.4 Augment Intent-Aware-Encoder

We further show the performances by adding LLM-generated data into Intent-
Aware-Encoder (IAE) pre-training data in order to properly compare
performances. “IAE-pos” replaces part of the IAE loss function that uses
pseudo label names with LLM generated data augmentation (for positive labels),
while “IAE-pos-neg” uses both positive and negative examples (either retrieved
or LLM-generated) in the contrastive loss term.

##  7 Conclusion and Future Works

In this paper, we propose a new evaluation toolkit for intent embedding models
that measure their semantic understanding on two lingusitic phenomenon common
in conversational systems– negation and implicature. For this we propose a
novel triplet task, a binary classification task and challenge test splits
that evaluate the model’s semantic understanding on downstream intent
recognition (classification and clustering) tasks. Our study shows that
current intent embedding models do not have sufficient understanding of these
two real world phenomenon, i.e. negation and implicature. We then propose to
integrate hard positives and negatives generated from an LLM with a “zoo” of
prompts to fine-tune the model. The fine-tuning is conducted on a set of
unlabeled utterances from general domain and is evaluated on our proposed
toolkit. Our best model demonstrates improvements across most tasks over
baseline models. We further combine LLM-generated data with Intent-Aware-
Encoder training and show performance improvements across all original test
datasets and most newly introduced evaluation datasets in the Intent Semantic
Toolkit. Finally, the correlation analysis between pairs of tasks indicates
that a better balance between negation-related tasks and regular benchmark
tasks need to be achieved. Our work also inspires future works to develop and
evaluate an instruction-following embedding model that can improve performance
via prompting without the need for further fine-tuning of the model.

## Limitations

In this paper, we used ChatGPT for evaluation and a smaller LLM
falcon-40b-instruct for training. This is mainly due to the legal concern. Our
use of OpenAI for this publication, is to the best our knowledge, in
compliance with applicable OpenAI’s terms and conditions as of March 14, 2023.
Our results show that Falcon-based data augmentation improves performance
significantly. However, we argue that using a larger and more capable LLM can
potentially improve the quality of data generated and thus further improve the
model.

## Ethical Considerations

This paper uses various open-source datasets and models for evaluation and
training, which are reproducible. Our evaluation toolkit uses OpenAI models
which might impact the reproducibility. However, we set the temperature to $0$
in order to reduce the variances.

## Acknowledgements

We would like to thank James Gung for insightful discussions. We also want to
thank Hossein Aboutalebi and Dennis Ulmer for their efforts in contributing to
our human evaluation study.

## References

  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  Falcon-40B: an open large language model with state-of-the-art performance.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 38–45, Online. Association for Computational Linguistics.
  * Chen et al. (2020) Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. 2020.  Low-resource domain adaptation for compositional task-oriented semantic parsing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5090–5100, Online. Association for Computational Linguistics.
  * Colombo et al. (2022) Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stephan Clemencon. 2022.  What are the best systems? new perspectives on nlp benchmarking.  _arXiv preprint arXiv:2202.03799_.
  * Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018.  SentEval: An evaluation toolkit for universal sentence representations.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Cooper Stickland et al. (2023) Asa Cooper Stickland, Sailik Sengupta, Jason Krone, Saab Mansour, and He He. 2023.  Robustification of multilingual language models to real-world noise in crosslingual zero-shot settings with robust contrastive pretraining.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1375–1391, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Dopierre et al. (2021) Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021.  PROTAUGMENT: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2454–2466, Online. Association for Computational Linguistics.
  * Ester et al. (1996) Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996.  A density-based algorithm for discovering clusters in large spatial databases with noise.  In _kdd_ , volume 96, pages 226–231.
  * Estévez et al. (2009) Pablo A Estévez, Michel Tesmer, Claudio A Perez, and Jacek M Zurada. 2009.  Normalized mutual information feature selection.  _IEEE Transactions on neural networks_ , 20(2):189–201.
  * Grice (1975) Herbert P Grice. 1975.  Logic and conversation.  In _Speech acts_ , pages 41–58. Brill.
  * Gung et al. (2023) James Gung, Raphael Shu, Emily Moeng, Wesley Rose, Salvatore Romeo, Yassine Benajiba, Arshit Gupta, Saab Mansour, and Yi Zhang. 2023.  Intent induction from conversations for task-oriented dialogue track at dstc 11.
  * Gupta et al. (2018) Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018.  Semantic parsing for task oriented dialog using hierarchical representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2787–2792, Brussels, Belgium. Association for Computational Linguistics.
  * Jeretic et al. (2020) Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020.  Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8690–8705, Online. Association for Computational Linguistics.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2023) Xiaoming Liu, Hongyu Lin, Xianpei Han, and Le Sun. 2023.  Sentbench: Comprehensive evaluation of self-supervised sentence representation with benchmark construction.  In _Chinese Computational Linguistics_ , pages 449–463, Singapore. Springer Nature Singapore.
  * Liu et al. (2021) Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2021.  _Benchmarking Natural Language Understanding Services for Building Conversational Agents_, pages 165–183. Springer Singapore, Singapore.
  * Ma et al. (2022) Tingting Ma, Qianhui Wu, Zhiwei Yu, Tiejun Zhao, and Chin-Yew Lin. 2022.  On the effectiveness of sentence encoding for intent detection meta-learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3806–3818, Seattle, United States. Association for Computational Linguistics.
  * MacQueen et al. (1967) James MacQueen et al. 1967.  Some methods for classification and analysis of multivariate observations.  In _Proceedings of the fifth Berkeley symposium on mathematical statistics and probability_ , volume 1, pages 281–297. Oakland, CA, USA.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Mueller et al. (2022) Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, and Dan Roth. 2022.  Label semantic aware pre-training for few-shot text classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8318–8334, Dublin, Ireland. Association for Computational Linguistics.
  * Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023.  MTEB: Massive text embedding benchmark.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  _CoRR_ , abs/1611.09268.
  * Nielsen and Nielsen (2016) Frank Nielsen and Frank Nielsen. 2016.  Hierarchical clustering.  _Introduction to HPC with MPI for Data Science_ , pages 195–211.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8689–8696.
  * Recanati (1989) François Recanati. 1989.  The pragmatics of what is said.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Ruis et al. (2022) Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022.  Large language models are not zero-shot communicators.
  * Sengupta et al. (2021) Sailik Sengupta, Jason Krone, and Saab Mansour. 2021.  On the robustness of intent classification and slot labeling in goal-oriented dialog systems to real-world noise.  In _Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI_ , pages 68–79, Online. Association for Computational Linguistics.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017.  Prototypical networks for few-shot learning.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Su et al. (2023) Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023.  One embedder, any task: Instruction-finetuned text embeddings.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1102–1121, Toronto, Canada. Association for Computational Linguistics.
  * Sung et al. (2023) Mujeen Sung, James Gung, Elman Mansimov, Nikolaos Pappas, Raphael Shu, Salvatore Romeo, Yi Zhang, and Vittorio Castelli. 2023.  Pre-training intent-aware encoders for zero- and few-shot intent classification.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29.
  * Zalta et al. (1995) Edward N Zalta, Uri Nodelman, Colin Allen, and John Perry. 1995.  Stanford encyclopedia of philosophy.
  * Zang et al. (2020) Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020.  MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 109–117, Online. Association for Computational Linguistics.
  * Zhang et al. (2022a) Haode Zhang, Haowen Liang, Yuwei Zhang, Li-Ming Zhan, Xiao-Ming Wu, Xiaolei Lu, and Albert Lam. 2022a.  Fine-tuning pre-trained language models for few-shot intent detection: Supervised pre-training and isotropization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 532–542, Seattle, United States. Association for Computational Linguistics.
  * Zhang et al. (2021) Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao-Ming Wu, and Albert Y.S. Lam. 2021.  Effectiveness of pre-training for few-shot intent classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1114–1120, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2023) Yuwei Zhang, Zihan Wang, and Jingbo Shang. 2023.  ClusterLLM: Large language models as a guide for text clustering.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 13903–13920, Singapore. Association for Computational Linguistics.
  * Zhang et al. (2022b) Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, and Albert Lam. 2022b.  New intent discovery with pre-training and contrastive learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 256–269, Dublin, Ireland. Association for Computational Linguistics.

Figure 5: Intent extraction pipeline. We first prompt the LLM for generating
the user goal from the utterance. We then find the action-object pair from the
generated goal. For those that can not find objects during second step, we
will further summarize the object with LLM. And finally, we will save the goal
and action-object pair for further generation.

Figure 6: Human annotation guidelines for quality control. Our annotators are
from diverse cultures and ethical groups including Asians, Europeans and
Americans.

##  Appendix A Classification Upper-Bound with GPT-4

In Section 4, we discussed the multi-label nature of implicature utterances
because of the inherent ambiguity of these utterances. For instance, “set the
mood please” could mean both “INTENT:change_light_hue” and
“INTENT:play_music”. Such a phenomena could be one of the reasons that the
evaluated embedding models show degraded performances compared with original
set. In order to decompose the effects of multi-label, we present a
classification upper-bound on 10-shot multi-class classification with
gpt-4-0613. We present gpt-4-0613 with the utterance and the top-$5$
predictions from instructor-large plus one ground truth label. We then ask the
model to classify it into one of the class. The calculated accuracy is
comparable with $10$-shot classification performance in Table 2. The obtained
results from gpt-4-0613 is $55.2\%$ which is significantly higher than
$34.3\%$. This further illustrates that we still have a lot of rooms for
improvement for embedding models.

##  Appendix B More Details on Fine-tuning: Retrieving and Diversifying

Retrieving Apart from LLM-generated data, we also use the retrieved utterances
which is considered as a standard data collecting procedure (and thus a
baseline method) in contrastive learning. We retrieve one positive and one
negative data for each of the utterances in the unlabeled corpus. For positive
one, we simply uniformly choose one utterance from the same-intent utterance
group. Notice that although there is no guarantee a positive utterance can be
retrieved, in practice we found most of utterances in the training corpus can
find positive pairs, i.e. $95.39\%$ ($241,080$ utterances). And we simply
filter out the rest of utterances that can not retrieve at least one positive.
For negative one, we first encode all the negative utterances and the original
utterance into the embedding space with the same embedding model that is going
to be trained. And then we calculate cosine distance between original
embedding and negative embeddings, and sort them from smaller to larger. We
then acquire the one in the middle of the list as we empirically found that
these utterances are both similar to the original utterances in terms of
surface forms and possess different intents.

Diversifying We empirically observe that the generated utterances usually
contain similar surface forms, e.g. most utterances from $P^{4}$ starts with
“i do not want to”. This is due to the question forms being used in the
specific prompts, and may potentially harm the diversity of training data. In
order to diversify them, we manually create more patterns such “i try not to”,
“i prefer not to” for $P^{4}$, and then modify the original utterance with it.
In addition, we also remove those responses that reject to produce answers by
identifying keywords “ai language model”.

Computing Budget All our models can be trained on a single 48GB GPU with less
than 3 hours of training.

##  Appendix C Qualitative Analysis on Errors

Note that we have some qualitative examples in Figure 2 that are sampled from
our dataset to demonstrate the characteristics of implicature and negation. In
order to better showcase some examples, we provide more qualitative analysis
here on CLINC150, especially failure cases during binary classification tasks
(classify to one of original label or negated label) using instructor-large.

Implicature errors:

> i’ve recently retired and i’m trying to cut back on my spending
> (credit_limit_change)
> the passenger will be stopping in london, is the time difference going to be
> a problem? (timezone)
> i think my bags missed the connection (lost_luggage)
> i’m having a hard time understanding the app because it’s not in my native
> language (change_language)
> i can’t decide what to have for dinner (flip_coin)

Negation errors:

> i didn’t mean to ask for the spanish word for pasta
> (no_need_to_request_translate)
> please don’t change my name, it was a mistake (no_need_to_change_user_name)
> my visa was damaged but it has been replaced (card_not_damaged)
> i like this song, don’t skip it (stay_at_current_song)
> i mistakenly set an alarm (no_need_to_set_alarm)

Furthermore, we performed clustering on these failure cases with instructor-
large embeddings and then perform $k$-means clustering. We then run Tf-idf
with each cluster as a document to acquire feature vectors for them, where the
high value entries indicate keywords for the cluster. We show some example
keywords for both negation and implicature here:

Implicature: ’my’, ’to’, ’the’, ’you’, ’your’, ’can’, ’not’ ’my’, ’card’,
’credit’, ’to’, ’the’, ’wallet’, ’bank’ ’to’, ’the’, ’for’, ’recipe’, ’salad’,
’lunch’, ’dinner’ ’to’, ’the’, ’my’, ’need’, ’meeting’, ’have’, ’for’ ’my’,
’to’, ’meter’, ’the’, ’utility’, ’reading’, ’bill’ ’the’, ’song’, ’to’, ’my’,
’this’, ’quiet’, ’is’ ’to’, ’the’, ’car’, ’my’, ’need’, ’get’, ’long’ ’to’,
’in’, ’thinking’, ’the’, ’for’, ’trip’, ’my’ ’language’, ’in’, ’my’, ’the’,
’to’, ’app’, ’english’ ’my’, ’luggage’, ’the’, ’suitcase’, ’bags’, ’bag’,
’plane’

Negation: ’reservations’, ’list’, ’to’, ’for’, ’the’ ’phone’, ’to’, ’know’,
’my’, ’you’ ’account’, ’my’, ’bank’, ’401k’, ’paid’ ’happened’, ’song’,
’what’, ’the’, ’is’ ’whisper’, ’mode’, ’settings’, ’changed’, ’unexpectedly’
’card’, ’credit’, ’my’, ’the’, ’limit’ ’meeting’, ’booked’, ’my’, ’flight’,
’to’ ’thanks’, ’for’, ’not’, ’nothing’, ’thank’ ’my’, ’engine’, ’light’, ’to’,
’check’ ’jump’, ’car’, ’start’, ’my’, ’to’

The results did not indicate strong bias towards a specific topic, and thus
showing that such failures are ubiquitous across domains and intent classes.

|  | Original | Intent Semantic Toolkit
---|---|---|---
Model | Rank | Clustering | Multi-class | Triplet (Ori-Ori) | Triplet (Ori-Imp) | Binary Classification | Clustering | Multi-class
|  | KM | Agg | 0-shot | 10-shot | $T_{hard}$ | $T_{easy}$ | $T_{hard}$ | $T_{easy}$ | Ori | Imp | Neg | KM | Agg | 0-shot | 10-shot
Baseline | 14 | 84.0 | 85.4 | 67.3 | 86.1 | 22.5 | 87.9 | 4.1 | 72.1 | 90.1 | 73.2 | 78.9 | 61.6 | 63.1 | 25.9 | 28.3
All $\\{P^{*},N^{*}\\}$ | 7 | 84.6 | 85.8 | 69.9 | 86.5 | 36.9 | 83.7 | 12.6 | 56.0 | 73.9 | 35.8 | 90.7 | 64.2 | 66.1 | 27.7 | 33.2
$-P^{4},N^{1,3}$ | 1 | 84.7 | 86.7 | 73.5 | 87.8 | 53.3 | 94.9 | 20.8 | 78.4 | 93.9 | 73.9 | 82.3 | 66.2 | 68.3 | 34.2 | 37.6
$-P^{1,4},N^{1,3}$ | 2 | 85.2 | 86.0 | 73.6 | 87.2 | 54.4 | 94.5 | 20.5 | 76.4 | 93.3 | 71.4 | 83.3 | 66.1 | 68.0 | 33.8 | 37.5
$-P^{4},N^{1}$ | 3 | 84.8 | 85.5 | 72.7 | 87.0 | 55.2 | 94.7 | 21.1 | 79.3 | 94.2 | 77.2 | 82.5 | 66.5 | 67.8 | 33.9 | 36.9
$-P^{1,2,4},N^{1,3}$ | 4 | 84.9 | 85.4 | 73.0 | 86.9 | 56.6 | 94.2 | 21.8 | 75.3 | 93.2 | 71.5 | 83.5 | 65.9 | 67.7 | 33.0 | 36.8
$-P^{4},N^{3}$ | 5 | 85.2 | 86.0 | 73.2 | 86.8 | 52.4 | 94.3 | 18.8 | 75.8 | 92.9 | 71.1 | 83.6 | 65.7 | 67.7 | 33.1 | 36.9
$-N^{*}$ | 6 | 85.8 | 87.2 | 70.3 | 87.5 | 16.7 | 72.2 | 3.1 | 40.9 | 58.9 | 27.7 | 89.7 | 63.6 | 65.7 | 28.1 | 33.4
$-P^{4}$ | 8 | 84.2 | 84.8 | 72.2 | 86.2 | 54.8 | 94.0 | 20.7 | 77.7 | 93.1 | 72.2 | 83.3 | 65.9 | 67.2 | 33.7 | 36.7
$-P^{2}$ | 9 | 84.5 | 85.1 | 72.7 | 86.4 | 37.4 | 82.8 | 13.1 | 53.4 | 70.8 | 32.7 | 90.9 | 65.8 | 67.8 | 34.0 | 36.6
$-P^{1}$ | 10 | 84.9 | 85.3 | 73.4 | 87.1 | 36.0 | 82.9 | 12.0 | 53.0 | 68.3 | 31.5 | 92.0 | 65.7 | 67.6 | 33.9 | 36.7
$-P^{4},N^{2}$ | 11 | 84.3 | 85.2 | 72.6 | 86.6 | 32.4 | 86.0 | 6.3 | 55.0 | 86.6 | 51.3 | 86.6 | 65.7 | 67.5 | 33.4 | 36.2
$-P^{*}$ | 12 | 80.6 | 81.0 | 66.4 | 83.0 | 56.7 | 91.2 | 22.8 | 64.8 | 90.7 | 62.0 | 85.1 | 62.6 | 64.7 | 25.8 | 30.3
$-P^{3}$ | 13 | 83.4 | 84.5 | 72.5 | 86.3 | 39.0 | 82.7 | 13.5 | 53.2 | 68.9 | 32.9 | 91.3 | 65.2 | 66.6 | 33.3 | 35.4
$-LLM$ | 15 | 83.6 | 84.4 | 69.2 | 85.5 | 41.8 | 87.2 | 8.9 | 51.6 | 89.1 | 53.5 | 84.0 | 63.4 | 65.4 | 27.3 | 32.1

Table 6: This is the complete version of Table 4 that includes all the prompt
variants.

| Original | Intent Semantic Toolkit
---|---|---
Model | Clustering | Multi-class | Triplet (Ori-Ori) | Triplet (Ori-Imp) | Binary Classification | Clustering | Multi-class
| KM | Agg | 0-shot | 10-shot | $T_{hard}$ | $T_{easy}$ | $T_{hard}$ | $T_{easy}$ | Ori | Imp | Neg | KM | Agg | 0-shot | 10-shot
Baseline | 83.8 | 84.9 | 67.5 | 85.8 | 19.1 | 86.1 | 2.0 | 67.6 | 89.1 | 67.0 | 78.4 | 61.1 | 62.9 | 26.4 | 31.1
Disable LLM | 82.6 | 84.2 | 69.5 | 85.3 | 24.5 | 80.5 | 3.0 | 43.4 | 83.6 | 43.1 | 83.6 | 63.1 | 65.1 | 27.0 | 31.7
Ours best | 83.3 | 85.2 | 69.3 | 86.4 | 46.9 | 93.3 | 17.8 | 80.9 | 94.1 | 81.0 | 78.9 | 63.5 | 65.5 | 28.3 | 33.8

Table 7: Main results for instructor-base on test set. “Ours best” corresponds
to “-$P^{4},N^{1,3}$” model in Table 8. Best results for each task are bolded.

|  | Original | Intent Semantic Toolkit
---|---|---|---
Model | Rank | Clustering | Multi-class | Triplet (Ori-Ori) | Triplet (Ori-Imp) | Binary Classification | Clustering | Multi-class
|  | KM | Agg | 0-shot | 10-shot | $T_{hard}$ | $T_{easy}$ | $T_{hard}$ | $T_{easy}$ | Ori | Imp | Neg | KM | Agg | 0-shot | 10-shot
Baseline | 14 | 83.5 | 84.5 | 67.3 | 86.1 | 18.9 | 86.5 | 2.6 | 68.2 | 89.0 | 67.6 | 78.0 | 61.6 | 63.1 | 25.9 | 28.3
All $\\{P^{*}$, $N^{*}\\}$ | 5 | 83.8 | 84.4 | 69.9 | 86.5 | 31.1 | 82.1 | 9.4 | 55.9 | 67.0 | 32.8 | 87.6 | 64.2 | 66.1 | 27.7 | 33.2
$-P^{4},N^{1,3}$ | 1 | 83.5 | 84.8 | 69.8 | 86.5 | 48.6 | 94.0 | 18.2 | 81.4 | 94.1 | 81.8 | 78.2 | 64.0 | 66.1 | 28.2 | 33.9
$-P^{4},N^{1}$ | 2 | 82.8 | 83.9 | 69.2 | 86.3 | 49.5 | 93.5 | 19.5 | 79.3 | 92.9 | 77.5 | 79.1 | 64.5 | 66.3 | 28.0 | 33.5
$-P^{1,4},N^{1,3}$ | 3 | 83.1 | 84.3 | 70.0 | 86.4 | 48.1 | 93.6 | 18.0 | 79.0 | 93.6 | 77.2 | 78.7 | 64.1 | 65.9 | 27.5 | 33.2
$-P^{4},N^{3}$ | 4 | 83.8 | 84.2 | 69.4 | 85.9 | 48.1 | 93.9 | 17.5 | 80.2 | 93.4 | 79.4 | 78.7 | 63.5 | 65.9 | 27.6 | 33.2
$-P^{1,2,4},N^{1,3}$ | 6 | 83.2 | 84.2 | 69.7 | 86.1 | 47.8 | 92.7 | 16.5 | 75.9 | 92.7 | 73.0 | 79.6 | 63.5 | 66.4 | 27.7 | 33.1
$-N^{*}$ | 7 | 84.0 | 85.6 | 70.3 | 87.5 | 10.7 | 64.5 | 1.2 | 32.8 | 49.3 | 19.8 | 91.0 | 63.6 | 65.7 | 28.1 | 33.4
$-P^{4}$ | 8 | 81.9 | 83.3 | 69.0 | 85.9 | 48.2 | 93.2 | 17.8 | 78.6 | 92.3 | 74.7 | 79.6 | 64.0 | 65.8 | 27.4 | 32.8
$-P^{2}$ | 9 | 83.1 | 83.8 | 69.6 | 85.9 | 32.5 | 81.9 | 9.7 | 53.7 | 68.0 | 32.6 | 87.5 | 64.0 | 66.0 | 27.6 | 32.7
$-P^{1}$ | 10 | 82.9 | 84.0 | 69.8 | 86.2 | 31.9 | 81.2 | 9.4 | 54.3 | 64.8 | 30.3 | 88.2 | 63.6 | 65.8 | 27.6 | 32.6
$-P^{4},N^{2}$ | 11 | 83.4 | 83.7 | 69.8 | 86.1 | 19.6 | 78.6 | 2.3 | 43.9 | 79.0 | 38.4 | 85.6 | 63.1 | 66.4 | 27.4 | 32.7
$-P^{3}$ | 12 | 83.3 | 83.7 | 69.8 | 85.9 | 31.4 | 79.7 | 9.4 | 51.6 | 61.0 | 27.0 | 88.7 | 64.0 | 65.9 | 27.4 | 32.6
$-P^{*}$ | 13 | 80.0 | 80.0 | 66.4 | 83.0 | 49.5 | 90.6 | 18.1 | 67.6 | 90.0 | 64.5 | 80.3 | 62.6 | 64.7 | 25.8 | 30.3
$-LLM$ | 15 | 82.8 | 83.3 | 69.2 | 85.5 | 26.0 | 81.3 | 3.4 | 44.9 | 83.9 | 44.7 | 83.2 | 63.4 | 65.4 | 27.3 | 32.1

Table 8: Ablation study for various prompts with instructor-base on dev set.
See Table 6 for descriptions.

Prompt |  Example
---|---
$P^{1}$: … Tell me 3 other kinds of [OBJECT] that you want to [ACTION] … |  what are the bike lanes like around the orlando area (know about,transportation) $\rightarrow$ interested in knowing about the public transportation system in orlando.
$P^{2}$: … Give me 1 other ways to express what you want … |  can you still just recommend one please?? (be recommended,product) $\rightarrow$ can you please suggest a specific product or service that you think would be a good fit for me?
$P^{3}$: … Give me 1 reasons that you want to do this … |  is highways 72 icy this morning? (know,weather) $\rightarrow$ could you help me know if the roads are icy so that i can take necessary precautions while driving.
$P^{4}$: … Give me 2 things that you do not want to do in this scenario … |  remind me tomorrow to pick up charlie, pickup new glasses, grab groceries (be reminded,reminder) $\rightarrow$ i’d rather avoid forgetting to pick up charlie.
$N^{1}$: … Give me 2 other things you want to [ACTION] rather than [OBJECT] … |  remind me tomorrow to pick up charlie, pickup new glasses, grab groceries (be reminded,reminder) $\rightarrow$ i want to be reminded to call my mom and wish her a happy birthday.
$N^{2}$: .. Now you no longer need to [ACTION] [OBJECT]. Give me 2 reasons for that … |  is highways 72 icy this morning? (know,weather) $\rightarrow$ i can check the weather forecast on my phone or computer.
$N^{3}$: … Now you do not want to [ACTION] [OBJECT], give me 2 other things you want to do … |  when is my wakeup alarm on thursday (know,time) $\rightarrow$ want to know if there are any important events happening on thursday.

Table 9: Hard positive/negative utterance generation prompts. The extracted
intents are shown in the brackets (action,object) on the right. The arrows
indicate before and after prompting.