# Zero-Shot Learning for Joint Intent and Slot Labeling

Rashmi Gangadharaiah  AWS AI Labs<EMAIL_ADDRESS>and  Balakrishnan
Narayanaswamy  AWS AI Labs<EMAIL_ADDRESS>
###### Abstract.

It is expensive and difficult to obtain the large number of sentence-level
intent and token-level slot label annotations required to train neural network
(NN)-based Natural Language Understanding (NLU) components of task-oriented
dialog systems, especially for the many real world tasks that have a large and
growing number of intents and slot types. While zero shot learning approaches
that require no labeled examples - only features and auxiliary information -
have been proposed only for slot labeling, we show that one can profitably
perform joint zero-shot intent classification and slot labeling. We
demonstrate the value of capturing dependencies between intents and slots, and
between different slots in an utterance in the zero shot setting. We describe
NN architectures that translate between word and sentence embedding spaces,
and demonstrate that these modifications are required to enable zero shot
learning for this task. We show a substantial improvement over strong
baselines and explain the intuition behind each architectural modification
through visualizations and ablation studies.

Intent Detection, Slot Labeling, Zero-shot Learning, Constrained Beam Search
decoding, Task-Oriented Dialog Systems, Conversational AI, Neural Networks

##  1\. Introduction

The NLU component in task oriented dialog systems (e.g., Alexa, Google
Assistant, Cortana or Siri), is responsible for identifying the user’s intent
and for extracting relevant constraints from the user’s utterance or input
sentence. Consider an example in Figure 1 (see Traditional Setup). Given a
user’s utterance, Play music from 2014, the NLU (Obuchowski, 2020; Goo et al.,
2018; Liu and Lane, 2016b; Han et al., 2019; Zhang et al., 2018; Qin et al.,
2019; E et al., 2019; Wang et al., 2018; Liu and Lane, 2016a) detects the
PlayMusic intent and extracts slot labels relevant to this intent, such as,
$\\{$year=2014$\\}$. These NLU components have recently been constructed using
sophisticated NN architectures (Bellegarda, 2014; Mesnil et al., 2015; Kurata
et al., 2016; Gangadharaiah and Narayanaswamy, 2019). NN-based NLUs have shown
remarkable performance but have millions of parameters. They are data hungry,
requiring thousands of labeled examples as training data in order to achieve
reasonable accuracy (Bellegarda, 2014; Mesnil et al., 2015; Kurata et al.,
2016). This hinders the training and deployment of models for new domains,
especially production domains where intents/slots are added over time.

Figure 1. Traditional vs. Zero-Shot learning setup. Note: previous approaches
considered either slot labeling or intent detection in zero-shot learning
settings and not both.

Zero-shot learning (Xian et al., 2018) enables models to output class labels
which have not been seen during training (Bapna et al., 2017). Zero-shot slot
labeling can be performed as a sequential labeling task - see how slot labels
are defined in the Traditional vs. Zero-shot setup in Figure 1. In the zero-
shot setting, the space of output labels is Begin(B), Inside(I, to handle
multi-word phrases) and Other/Outside(O, for words that are not relevant) for
each specific slot - as opposed to the space of all possible slot types (such
as, city, cuisine, etc.) in the traditional setting. Prediction is performed
by feeding in pairs of (the user’s sentence, slot description) for each slot
type as input. In the example, the slot description for year is passed as
input to the model along with the user’s sentence. The model then predicts
B/I/O labels for individual words/tokens in the user’s sentence that may
correspond to the year slot type. The token, 2014, is predicted as belonging
to the slot, year (as it received the label, B) while all other tokens (Play,
music, from) in the sentence were assigned to O (implying that they do not
belong to year). This process is repeated for every slot type on the same
user’s sentence. The individual I/O/B predictions for each slot type in the
user’s sentence are then aggregated to obtain the final prediction to match
the output in the traditional setting. Lee et. al., (2018) use slot
descriptions to bootstrap to new slots. Shah et. al., (2019) utilize both the
slot description and a few exemplar slot values as opposed to just using slot
descriptions.

Previous zero-shot approaches have either been applied for intent detection
(Liu et al., 2019; Xia et al., 2018; Siddique et al., 2021; Williams, 2019) or
slot labeling (Bapna et al., 2017; Shah et al., 2019) but not both. As a
result, these approaches do not consider dependencies between slot labels and
intents. Additionally, previously proposed zero-shot slot-labeling approaches
(Bapna et al., 2017; Shah et al., 2019) do not capture dependencies between
slot labels. To understand the implications of these, consider an example in
Figure 2 from the popular SNIPS dataset (Coucke et al., 2018).

The presence of condition_description and state slot labels are strong
indicators that the numbers (13, 2038) provided correspond to timeRange.
However, prior approaches tend to predict out of context slots, in this case
party_size_number is predicted even though the input sentence does not belong
to the BookRestaurant intent.Such inconsistencies can be avoided by capturing
(1) dependencies between intents and slot labels and (2) dependencies between
slot labels. We incorporate such dependencies using global constraints in our
model and a form of beam search decoding. The predicted GetWeather intent can
provide additional support for timeRange.

Figure 2. Predicted slot labels using the baseline.

In this paper, we propose an architecture (Zero-Shot learning setup in Figure
1) and novel training and inference schemes to jointly predict both intents as
well as slot labels, all of which extend to the zero-shot setting for new
intents and slots. The contributions of this paper are as follows:

  * •

We develop zero-shot learning approaches to jointly predict intents and slot
labels (Section 4). The approaches capture dependencies (1) between intents
and slot labels and (2) between slot labels, using learned global constraints.

  * •

We show that sentence-level representations of input utterances and
descriptions of short intents from pre-trained language models (e.g. BERT
(Devlin et al., 2018)), do not lie in a comparable space (Section 3). This
leads to inaccurate predictions on unseen intents (Section 5). We show how to
learn translations between these spaces within our joint intent-slot model to
fix this.

  * •

We show that fine-tuning BERT improves traditional intent and slot
classification, but hurts zero shot performance. To overcome this limitation,
we describe a sequential training procedure that first trains a translation
model for sentence and word embeddings and then fine tunes the language model.

  * •

Along the way we also mention important tricks (e.g. how to use negative
sampling) that seem to be required to get improvements.

We now describe the experimental set up and datasets used to evaluate our
proposed approach and baselines.

##  2\. Experimental Setup

We demonstrate results on the SNIPS dataset (Coucke et al., 2018), commonly
used for evaluating NLU and is especially appropriate for evaluating zero-shot
learning as the data covers multiple domains. The SNIPS data has 7 intent
types and 72 slot labels. The training set contains 13,084 utterances, 700
utterances in the test set and development set. Similar to test settings used
in (Shah et al., 2019; Bapna et al., 2017), for every target intent, we train
a model on utterances from the remaining intents and evaluate on the test set
that includes the ‘missing’ or ‘target’ intent. In contrast to the previously
proposed approaches that only predict presence of slot labels, we also predict
the presence of intents. We used the BertAdam optimizer (Kingma and Ba, 2017)
with warmup ratio of 0.1, with maximum number of epochs set to 20. For
ablation models M0-M5 below, the optimal learning rates were varied between
5e-03 to 5e-06, and final value chosen based on the performance of the models
on a validation set. The reported scores are averaged over 3 random seeds.
Preliminary experiments had unsatisfactory performance. To enable learning,
how we sample negative and positive examples for intents and slots is
critical. To enable learning from different combinations of intents and slots,
we sample one example from each of the following (1) an intent is present and
a slot label is present (2) an intent is present but a slot label is absent
(3) an intent is absent but a slot label is present (4) both intent and slot
label are absent. We report accuracy for intent and f1 for slot labeling
performance using CONLL’s evaluation script (Bansal and Villavicencio, 2019).

(a)

(b)

(c)

Figure 3. (a) S-level-reps of intent descriptions and input sentences (b)
S-level-reps of input sentences and W-level-reps of intent descriptions (c)
proposed representations for input sentence ($g^{a}$) and intent descriptions

##  3\. Issues with sentence representations

Following prior work in slot labeling in zero-shot settings (Shah et al.,
2019; Bapna et al., 2017; Lee and Jha, 2018) we leverage slot labels to create
descriptions, e.g., GetWeather is converted to get weather and
condition_temperature is converted to condition temperature. This is
reasonable since intent and slot names typically convey information on what
they represent. We leverage BERT (Devlin et al., 2018) to obtain sentence-
level ($\mathbf{CLS}$/S-level-reps) and token-level (W-level-reps)
representations, due to its superior performance on many NLP tasks, especially
on traditional joint intent and slot labeling, where sufficient number of
examples are available for all intents and slot types (Chen et al., 2019).
Since zero-shot settings rely on an encoding that is a function of the input
sentence and the descriptions, we further analyzed the representation spaces
of both the sentence and token-level representations obtained via BERT.

We found that it is not useful to compare S-level-reps of input sentences and
S-level-reps of very short intent descriptions (Figure 3(a), we consider only
3 intents for this plot for comprehensible visualization). Another option
would be to compare the word(W)-level-reps of these short intent descriptions
with S-level-reps of the input sentences. However, as Figure 3(b) illustrates,
W-level-reps and S-level-reps do not lie in a comparable space. To overcome
this problem, we propose using attention mechanisms on the W-level-reps of the
intent descriptions, together with translation layers that can map between
spaces (Figure 3(c)).

##  4\. Proposed Method

Consider a user’s sentence with $\mathbf{T}$ tokens. The corresponding BERT
S-level-reps and W-level-reps (of size $dim$) are passed through translation
layers, $Z_{S}$ and $Z_{W}$ respectively, to obtain,
$\\{x_{\mathbf{CLS}},x_{i}\in\textbf{R}^{dim},i\in[1,\mathbf{T}]\\}$.
$x_{\mathbf{CLS}}$ stands for the sentence representation and $x_{i}$ is the
token-level representation for the token $i$. $Z_{S}$ and $Z_{W}$ are composed
of two fully connected feed forward layers. Similarly, we obtain translated
representations $\\{a_{\mathbf{CLS}},a_{q}\in R^{dim},q\in[1,\textbf{Q}]\\}$
for Q intent description tokens,
$\\{b_{\mathbf{CLS}},b_{s}\in\textbf{R}^{dim},s\in[1,\textbf{S}]$} for S slot
description tokens,
$\\{e^{k}_{\mathbf{CLS}},e_{k}^{n}\in\textbf{R}^{dim},n\in[1,\textbf{N}_{k}]\\}$
for each of K examples per slot, with $\textbf{N}_{k}$ tokens.

###  4.1. Slot prediction

We perform mean pooling of the translated representations and compute an
attention weighted representation of all the $\mathbf{K}$ slot examples for
each token in the user’s utterance $u$, resulting in a $\mathbf{R}^{dim}$
representation.

$\displaystyle
e_{k}=\frac{1}{\mathbf{N}_{k}}\sum_{n=1}^{\mathbf{N}_{k}}e^{n}_{k},\ \
k\in[1,\mathbf{K}]$
$\displaystyle\alpha^{i}_{k}=\mathbf{softmax}(x_{i}W_{s}e_{k})$ (1)
$\displaystyle e^{s}_{i}=\sum_{k=1}^{\mathbf{K}}\alpha^{i}_{k}\odot e_{k}$

$\odot$ denotes dot product and $\oplus$ denotes concatenation. The resulting
representation at each token position $i$, $e^{s}_{i}$ from (4.1), is then
concatenated with $x_{i}$, $b_{\mathbf{CLS}}$ and $a_{\mathbf{CLS}}$ and sent
through a $\mathbf{biLSTM}$ (Hochreiter and Schmidhuber, 1997) layer, followed
by a $\mathbf{softmax}$ layer to obtain predictions for the slot labels.

$\displaystyle d_{i}=\mathbf{biLSTM}(\\{x_{i}\oplus b_{\mathbf{CLS}}\oplus
a_{\mathbf{CLS}}\oplus e^{s}_{i}\\})$ (2) $\displaystyle
y_{i}=\mathbf{softmax}(W_{y}d_{i}+\textrm{bias}_{y})$

###  4.2. Intent prediction

We then perform max pooling to obtain $g^{a}\in\mathbf{R}^{dim}$, followed by
a dense and $\mathbf{sigmoid}$ layer to obtain, $z^{a}$, corresponding to the
likelihood that intent $a$ is present (we overload $a$).

(3) $\displaystyle\beta^{i}_{q}=\mathbf{softmax}(\\{x_{i}W_{p}a_{q}\\})$
$\displaystyle g_{i}^{a}=\sum_{q=1}^{\mathbf{Q}}\beta^{i}_{q}\odot a_{q}$
$\displaystyle g^{a}=\mathbf{max}_{i}(g^{a}_{i})$ $\displaystyle
z^{a}=\mathbf{sigmoid}\ (\ \mathbf{Dense}(a_{\mathbf{CLS}}\oplus g^{a})\ )$

We also compare this with a variation that uses sentence representations of
the input sentence and the intent description where,

(4) $g^{a}=x_{\mathbf{CLS}}\odot a_{\mathbf{CLS}}$

We are essentially interested in the distance between learned embeddings of
intents, slots and utterances. The euclidean distance between two vectors $x$
and $y$ ((Snell et al., 2017)), can often be well approximated by networks
with inner products, but often not by networks where the inputs are
concatenated. In particular, if $|x|_{2}$ approximately proportional
$(\mathrel{\raisebox{3.01385pt}{\mbox{$\underset{\vbox{\hbox{$\sim$}}}{\vbox{\hbox{$\propto$}}}$}}}$)
$|y|_{2}$,
$|x-y|^{2}\mathrel{\raisebox{3.01385pt}{\mbox{$\underset{\vbox{\hbox{$\sim$}}}{\vbox{\hbox{$\propto$}}}$}}}-(x\odot
y)$. In our experiments, we found that element-wise multiplication layers
outperformed concatenation layers.

###  4.3. Global Slot Constraint:

We learn a global constraint vector that captures dependencies between slot
labels- dependencies previously ignored by vanilla zero-shot slot label
prediction. We consider descriptions of all slot labels of the intent. Say,
$\\{b_{1},..b_{\mathbf{L}}\\}$ represent translated vectors of all slot label
descriptions that belong to the intent (e.g., artist, album, genre, etc. for
the intent PlayMusic), with each intent containing $\mathbf{L}$ slot labels
and each slot description containing $\mathbf{M}_{l}$ tokens.

$\displaystyle
b_{l}=\frac{1}{\mathbf{M}_{l}}\sum_{m=1}^{\mathbf{M}_{l}}b_{m}^{l},l\in[1,\mathbf{L}]$
$\displaystyle\gamma^{i}_{l}=\mathbf{softmax}(\\{x_{i}W_{r}b_{l}\\})$ (5)
$\displaystyle\gamma_{l}=\textbf{max}_{i}(\gamma^{i}_{l})$

$\gamma\in\mathbf{R}^{\mathbf{L}}$ is concatenated in (4.1) and (3), resulting
in:

$\displaystyle d_{i}=\mathbf{biLSTM}(\\{x_{i}\oplus b_{\mathbf{CLS}}\oplus
a_{\mathbf{CLS}}\oplus\gamma\oplus e_{i}^{s}\\})$ (6) $\displaystyle
z^{a}=\mathbf{sigmoid}(\ \mathbf{Dense}(a_{\mathbf{CLS}}\oplus\gamma\oplus
g^{a})\ )$

We use categorical cross entropy loss for slots and binary cross entropy loss
for predicting the presence of intents. The losses are combined using a weight
that is learned along with the rest of the parameters of the model (Kendall et
al., 2018).

Metric$\rightarrow$ | Intent Accuracy | Slot F1 score
---|---|---
Model$\rightarrow$ | M1 | M2 | M3 | M4 | M5 | CT | ZT | XT | M0 | M1 | M2 | M3 | M4 | M5
Target Intent$\downarrow$ |  |  |  |  |  |  |  |  |  |  |  |  |  |
GetWeather | 83.0 | 83.4 | 84.4 | 88.0 | 88.6 | 63.5 | 60.7 | 66.0 | 85.8 | 86.0 | 85.9 | 81.3 | 87.7 | 90.9
BookRestaurant | 85.0 | 86.0 | 87.6 | 91.9 | 93.8 | 45.7 | 46.6 | 48.6 | 79.4 | 82.3 | 87.2 | 79.2 | 83.3 | 84.8
PlayMusic | 86.5 | 86.3 | 86.3 | 87.9 | 89.3 | 28.7 | 30.1 | 33.8 | 83.4 | 80.4 | 82.0 | 85.1 | 86.0 | 88.3
AddToPlaylist | 79.6 | 80.6 | 79.0 | 80.6 | 81.0 | 53.3 | 46.8 | 55.2 | 83.7 | 85.8 | 78.8 | 83.0 | 83.4 | 84.7
SearchCreativeWork | 79.5 | 85.0 | 89.0 | 87.1 | 87.1 | 24.7 | 26.7 | 26.2 | 85.3 | 85.8 | 84.6 | 83.6 | 83.5 | 85.8
SearchScreeningEvent | 82.7 | 84.5 | 85.3 | 86.7 | 87.7 | 23.7 | 19.7 | 25.5 | 82.7 | 86.6 | 84.0 | 81.5 | 85.0 | 86.6
RateBook | 88.2 | 88.3 | 88.0 | 88.3 | 90.7 | 24.5 | 31.0 | 28.5 | 76.4 | 78.2 | 81.8 | 75.1 | 79.0 | 83.3
Table 1. Comparing baselines (CT (Bapna et al., 2017), ZT (Lee and Jha, 2018),
XT (Shah et al., 2019)) with models M0-5, in terms of intent accuracy and slot
F1 scores.

##  5\. Results

Table 1 shows results for the following:

  * •

M0: slot labeling only, using eqn (4.1),

  * •

M1: joint intent and slot labeling using W-level-reps in eqn (4),

  * •

M2: joint intent and slot labeling with S-level-reps as in eqn (3),

  * •

M3: Joint intent and slot labeling with eqn (3) but without fine tuning the
BERT layer,

  * •

M4: A model initialized using M3, with a second training stage with the
translation parameters ($Z_{S}$ and $Z_{W}$) fixed,

  * •

M5: beam search decoding on M4’s predictions. M0, M1 and M2 do not use the
translation layers $Z_{S}$ and $Z_{W}$.

  * •

We also report scores from prior work on zero shot learning for slot labeling
CT (Bapna et al., 2017), ZT (Lee and Jha, 2018), XT (Shah et al., 2019).

Comparing M0 with M1-M5, it is clear that slot labeling can be improved by
incorporating information about intents. Hence, a joint model of intents and
slot labels is better than predicting slot labels independent of intents.
Further analysis of the intent predictions of M1 revealed that most of the
unseen target intents had zero (at most 2) matches with the ground truth
intent. This implies that the S-level-reps of BERT are not suitable for
predicting unseen intents as also seen with the t-SNE plot in Figure 3(a).

Intent$\rightarrow$ | GW | BR | PM | ATP | SCW | SSE | RB
---|---|---|---|---|---|---|---
%TPR | 51.0 | 81.5 | 20.9 | 33.0 | 30.0 | 36.4 | 45.0
%FDR | 0.0 | 13.8 | 0.0 | 0.0 | 13.9 | 12.3 | 2.7
Table 2. %TPR and %FDR using M5. GW: GetWeather, BR: BookRestaurant, PM:
PlayMusic, ATP: AddToPlaylist, SCW: SearchCreativeWork, SSE:
SearchScreeningEvent, RB: RateBook.

Comparing M2 and M3, we notice a drop in performance for some intents and
slots even though M3 was able to find a substantial number of matches of the
target intent, perhaps due to the absence of BERT fine-tuning (Peters et al.,
2019) in M3. To take advantage of BERT fine-tuning and yet obtain the
performance of M2 on target intents, we perform two stage training where the
model (M4) is initialized using M3 and trained with $Z_{S}$ and $Z_{W}$
parameters fixed. The resulting model, M4, performs significantly better than
M2 on intent prediction and in most cases on slot labeling.

###  5.1. Beam Search Decoding with Constraints

Dependencies between intents and slot labels are already captured in the
outputs of our model, since we jointly model intents and slot labels and have
a global slot constraint $\gamma$ that captures dependencies between slot
labels. To further enforce dependencies between slots and intents at inference
time, we perform a variant of beam search (Russell and Norvig, 2009) as a post
processing step that finds tags for the input sequence by only considering
legal paths. At every word position, the algorithm keeps track of top
$\textbf{V}=3$ (beam width) legal paths that respect intent and slot
constraints. The decoded path in Figure 2 would be considered illegal as
party_size_number and condition_description cannot co-occur. Each utterance in
the SNIPS dataset belongs to a single intent only. Beam search on multiple
intents (Gangadharaiah and Narayanaswamy, 2019) will be explored as future
work.

The top $3$ highest scoring intents are considered as the initial paths and
extended via beam search. A beam search matrix of size T rows and
$2*\textbf{S}+1$ columns (where $2*\textbf{S}+1$ is the total number of slot
labels including $\text{B}-$ and $\text{I}-$ for every slot label and O) is
created by averaging the slot probabilities $p(\text{slot}_{j}|w_{i})$
assigned by the model to each token $w_{i}$ across all intents.

(7)
$p\text{(O}|w_{i})=\max(\epsilon,1-\sum_{j=1}^{2S}p(\text{slot}_{j}|w_{i}))$

We use $\epsilon=10^{-7}$. Beam search on M4’s predictions (shown as M5) shows
further improvements in slot labeling and intent detection.

We also report True Positive Rate (%TPR, predicted target intent is the same
as the ground-truth target intent) and False Discovery Rate (%FDR, a target
intent is predicted when ground-truth has a different intent) on target
intents for M5 in Table 2. We see that M5 found a significant number of target
intent matches, with room for improvement for certain intents.

##  6\. Conclusion and Future work

We explored strategies to predict unseen intents and slots in a zero-shot
learning setup where we jointly modeled both slot labeling and intent
prediction. We showed the importance of capturing dependencies between intents
and slot labels using learned global constraints. Through experimentation, we
showed how fine-tuning BERT hurt zero shot performance. To overcome this
limitation, we proposed a sequential training procedure to first train a
translation model for sentence and word embeddings and then fine tune the
language model. As seen in Table 2, although the proposed models found target
intent matches, the performance is still low. We will continue to explore
strategies to further improve the performance of NLUs on unseen intents.

## References

  * (1)
  * Bansal and Villavicencio (2019) Mohit Bansal and Aline Villavicencio (Eds.). 2019.  _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_. Association for Computational Linguistics, Hong Kong, China.  https://www.aclweb.org/anthology/K19-1000
  * Bapna et al. (2017) Ankur Bapna, Gokhan Tür, Dilek Hakkani-Tür, and Larry Heck. 2017\.  Towards Zero-Shot Frame Semantic Parsing for Domain Scaling. In _Proc. Interspeech 2017_. 2476–2480.  https://doi.org/10.21437/Interspeech.2017-518
  * Bellegarda (2014) Jerome R. Bellegarda. 2014\.  In _Natural Interaction with Robots, Knowbots and Smartphones_. New York, Chapter 1, 3–14.
  * Chen et al. (2019) Qian Chen, Zhu Zhuo, and Wen Wang. 2019.  BERT for Joint Intent Classification and Slot Filling.  arXiv:1902.10909 [cs.CL]
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018.  Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces.  _CoRR_ abs/1805.10190 (2018).  arXiv:1805.10190 http://arxiv.org/abs/1805.10190
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * E et al. (2019) Haihong E, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019\.  A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling.  _CoRR_ abs/1907.00390 (2019).  arXiv:1907.00390 http://arxiv.org/abs/1907.00390
  * Gangadharaiah and Narayanaswamy (2019) Rashmi Gangadharaiah and Balakrishnan Narayanaswamy. 2019\.  Joint Multiple Intent Detection and Slot Labeling for Goal-Oriented Dialog. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 564–569.  https://doi.org/10.18653/v1/N19-1055
  * Goo et al. (2018) Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018.  Slot-Gated Modeling for Joint Slot Filling and Intent Prediction. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 753–757.  https://doi.org/10.18653/v1/N18-2118
  * Han et al. (2019) Kun Han, Junwen Chen, Hui Zhang, Haiyang Xu, Yiping Peng, Yun Wang, Ning Ding, Hui Deng, Yonghu Gao, Tingwei Guo, Yi Zhang, Yahao He, Baochang Ma, Yulong Zhou, Kangli Zhang, Chao Liu, Ying Lyu, Chenxi Wang, Cheng Gong, Yunbo Wang, Wei Zou, Hui Song, and Xiangang Li. 2019.  DELTA: A DEep learning based Language Technology plAtform.  arXiv:1908.01853 [cs.CL]
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Comput._ 9, 8 (Nov. 1997), 1735–1780.  https://doi.org/10.1162/neco.1997.9.8.1735
  * Kendall et al. (2018) Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018.  Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics.  arXiv:1705.07115 [cs.CV]
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017.  Adam: A Method for Stochastic Optimization.  arXiv:1412.6980 [cs.LG]
  * Kurata et al. (2016) Gakuto Kurata, Bing Xiang, Bowen Zhou, and Mo Yu. 2016\.  Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, 2077–2083.  https://doi.org/10.18653/v1/D16-1223
  * Lee and Jha (2018) Sungjin Lee and Rahul Jha. 2018.  Zero-Shot Adaptive Transfer for Conversational Language Understanding.  arXiv:1808.10059 [cs.CL]
  * Liu and Lane (2016a) Bing Liu and Ian R. Lane. 2016a.  Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling.  _CoRR_ abs/1609.01454 (2016).  arXiv:1609.01454 http://arxiv.org/abs/1609.01454
  * Liu and Lane (2016b) Bing Liu and Ian R. Lane. 2016b.  Joint Online Spoken Language Understanding and Language Modeling with Recurrent Neural Networks.  _CoRR_ abs/1609.01462 (2016).  arXiv:1609.01462 http://arxiv.org/abs/1609.01462
  * Liu et al. (2019) Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao-Ming Wu, and Albert Y.S. Lam. 2019.  Reconstructing Capsule Networks for Zero-shot Intent Classification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 4799–4809.  https://doi.org/10.18653/v1/D19-1486
  * Mesnil et al. (2015) G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L. Deng, D. Hakkani-Tur, X. He, L. Heck, G. Tur, D. Yu, and G. Zweig. 2015\.  Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 23, 3 (2015), 530–539.  https://doi.org/10.1109/TASLP.2014.2383614
  * Obuchowski (2020) Aleksander Obuchowski. 2020\.  Transformer-Capsule Model for Intent Detection.  (2020).
  * Peters et al. (2019) Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019\.  To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks.  _CoRR_ abs/1903.05987 (2019).  arXiv:1903.05987 http://arxiv.org/abs/1903.05987
  * Qin et al. (2019) Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019.  A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding.  _CoRR_ abs/1909.02188 (2019).  arXiv:1909.02188 http://arxiv.org/abs/1909.02188
  * Russell and Norvig (2009) Stuart Russell and Peter Norvig. 2009.  _Artificial Intelligence: A Modern Approach_ (3rd ed.).  Prentice Hall Press, USA.
  * Shah et al. (2019) Darsh J Shah, Raghav Gupta, Amir A Fayazi, and Dilek Hakkani-Tur. 2019.  Robust zero-shot cross-domain slot filling with example values.  _arXiv preprint arXiv:1906.06870_ (2019).
  * Siddique et al. (2021) A. B. Siddique, Fuad T. Jamour, Luxun Xu, and Vagelis Hristidis. 2021.  Generalized Zero-shot Intent Detection via Commonsense Knowledge.  _CoRR_ abs/2102.02925 (2021).  arXiv:2102.02925 https://arxiv.org/abs/2102.02925
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017\.  Prototypical networks for few-shot learning. In _Advances in neural information processing systems_. 4077–4087.
  * Wang et al. (2018) Yu Wang, Yilin Shen, and Hongxia Jin. 2018.  A Bi-Model Based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 309–314.  https://doi.org/10.18653/v1/N18-2050
  * Williams (2019) Kyle Williams. 2019\.  Zero Shot Intent Classification Using Long-Short Term Memory Networks.. In _INTERSPEECH_. 844–848.
  * Xia et al. (2018) Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip S. Yu. 2018.  Zero-shot User Intent Detection via Capsule Neural Networks.  arXiv:1809.00385 [cs.CL]
  * Xian et al. (2018) Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. 2018.  Zero-shot learning—A comprehensive evaluation of the good, the bad and the ugly.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 9 (2018), 2251–2265.
  * Zhang et al. (2018) Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip S. Yu. 2018.  Joint Slot Filling and Intent Detection via Capsule Neural Networks.  _CoRR_ abs/1812.09471 (2018).  arXiv:1812.09471 http://arxiv.org/abs/1812.09471