# A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems

Zihao Yi<EMAIL_ADDRESS>Sun Yat-Sen UniversityShenzhenChina ,
Jiarui Ouyang<EMAIL_ADDRESS>Sun Yat-Sen UniversityShenzhenChina ,
Yuwen Liu<EMAIL_ADDRESS>Sun Yat-Sen UniversityShenzhenChina ,
Tianhao Liao<EMAIL_ADDRESS>Sun Yat-Sen UniversityShenzhenChina ,
Zhe Xu<EMAIL_ADDRESS>Sun Yat-Sen UniversityShenzhenChina and
Ying Shen<EMAIL_ADDRESS>Sun Yat-Sen UniversityShenzhenChina

(2024; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

This survey provides a comprehensive review of research on multi-turn dialogue
systems, with a particular focus on multi-turn dialogue systems based on large
language models (LLMs). This paper aims to (a) give a summary of existing LLMs
and approaches for adapting LLMs to downstream tasks; (b) elaborate recent
advances in multi-turn dialogue systems, covering both LLM-based open-domain
dialogue (ODD) and task-oriented dialogue (TOD) systems, along with datasets
and evaluation metrics; (c) discuss some future emphasis and recent research
problems arising from the development of LLMs and the increasing demands on
multi-turn dialogue systems.

large language models, fine-tuning, prompt engineering, task-oriented dialogue
systems, open-domain dialogue systems

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††journal:
JACM††journalvolume: 37††journalnumber: 4††article: 111††publicationmonth:
2††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Discourse,
dialogue and pragmatics††ccs: General and reference Surveys and overviews

##  1\. INTRODUCTION

###  1.1. What is Multi-turn Dialogue System?

Multi-turn dialogue systems that can generate natural and meaningful responses
to communicate with humans are a long-term goal of artificial intelligence
(AI). Such human-computer interaction task has attracted increasing attention
from academia and industry due to their potential impact and attractive
commercial value. The multi-turn dialogue task can be regarded as a sequence-
to-sequence task, which generates the system response
$\mathcal{S}=(s_{1},s_{2},...s_{t})$ from the user message
$\mathcal{U}=(u_{1},u_{2},...u_{t})$, where $u_{t}$ and $s_{t}$ are the user
message and system response in the $t$-th round, respectively.

Multi-turn dialogue systems can be divided into TOD systems and ODD systems.
TOD systems assist users in addressing tasks within a specific domain such as
hotel booking, restaurant recommendation, etc., while ODD systems chat with
users without domain restrictions. TOD tasks and ODD tasks are not entirely
independent, an ODD task can be converted into a TOD task once the dialogue
system detects specific user requirement.

Conventional dialogue systems primarily rely on rule-based approaches and
retrieval-based methods. Rule-based dialogue systems (weizenbaum1966eliza, ;
colby1971artificial, ; goddeau1996form, ) generate responses by predefining
conversation flows for specific scenarios. Retrieval-based dialogue systems
(wu2016sequential, ; zhao2016towards, ; ma2019triplenet, ) rely on predefined
templates, making them more flexible than rule-based systems. However, the
application scope of retrieval-based dialogue systems remains constrained, as
the generated responses are based on predefined template. With the development
of deep-learning methods, many multi-turn dialogue systems
(serban2016building, ; he2020amalgamating, ; qiu2019training, ) based on deep
neural networks are proposed. Recently, the performance of multi-turn dialogue
systems is significantly enhanced by the emergence of pre-trained LLMs.

###  1.2. Why a Survey on LLM-based Multi-turn Dialogue System?

Arora et al. (arora2013dialogue, ) provided an overview of dialogue systems
and introduced various dialogue system frameworks. However, this survey
treated the dialogue system as a generic system rather than categorizing it
into TOD and ODD systems and did not encompass deep learning models. Chen et
al. (chen2017survey, ) categorized dialogue systems into TOD and ODD systems,
discussing the application of deep learning techniques in both types of
dialogue systems. Nevertheless, this survey did not dive into multi-turn
dialogue systems based on pre-trained LLMs. The review written by Ni et al.
(ni2023recent, ) covered multi-turn dialogue systems based on pre-trained
LLMs, but this study did not provide detailed insights into LLMs and methods
for adapting them to downstream sub-tasks. In contrast, Qin et al.
(qin2023end, ) provided a more comprehensive exploration of the application of
pre-trained LLMs in specific-target dialogue scenarios. However, the focus of
this paper primarily centered on end-to-end task-oriented multi-turn dialogue
systems.

Our paper aims to provide a state-of-the-art overview of LLM-based multi-turn
dialogue systems, before which we will provide a comprehensive exposition of
existing pre-trained LLMs and methodologies employed to adapt these models for
downstream tasks. This investigation is anticipated to appeal to broad
audiences in both academia and industry, encompassing researchers and
practitioners alike.

###  1.3. Contribution of this Survey

In this paper, we provide a comprehensive review of methods, evaluation
metrics and datasets for LLM-based multi-turn dialogue. The contribution of
our paper can be summarized as follows:

(1) To give a thorough review of LLMs and methods adapting LLMs to different
subtasks, as well as up-to-date LLM-based muti-turn dialogue systems;

(2) To provide a detailed exposition on state-of-the-art multi-turn dialogue
datasets and evaluation metrics.

(3) To discuss some future emphasis and recent research problems arising from
the increasing demands on dialogue systems and the development of LLMs.

The rest of this survey is organized as follows. In Sec.2, we provide a
detailed exposition of prevalent LLMs. From Sec.3 to Sec.4, we will give a
comprehensive introduction to methods for adapting LLMs to downstream tasks.
In Sec.5, we present important methods of TOD, including pipeline-based
methods and end-to-end methods. State-of-the-art methods of ODD are proposed
in Sec.6. In Sec.7 and Sec.8, we introduce some relevant datasets and
evaluation metrics. Besides, some problems and challenges of LLM based multi-
turn dialogue are proposed in Sec.9. Finally, we conclude our survey in
Sec.10.

##  2\. GENERAL METHODS

LLMs are a class of extensive artificial intelligence models characterized by
their massive scale with billions of parameters (kaplan2020scaling, ). Scaling
up LLMs allows them to learn more intricate and accurate language
representations, resulting in improved performance across diverse downstream
Natural Language Processing (NLP) tasks, particularly excelling in Natural
Language Generation (NLG) challenges (wei2022emergent, ; qiu2020pre, ). The
brief comparison of different structures of the LLMs mentioned can be seen in
Table 1.

The vanilla Transformer architecture (vaswani2017attention, ), a sequence-to-
sequence model, has emerged as a foundational framework for diverse LLMs,
utilizing encoders and decoders with self-attention mechanisms as its core
components, thanks to its exceptional parallelism and capacity. Based on the
masking methods utilized by various attention mechanisms in the model, the
current LLMs can be divided into three categories, i.e., Encoder-Decoder,
Decoder-only, and Encoder-only. The decoder-only category further includes
distinctions such as causal decoders and prefix decoders, illustrated in
Figure 1.

In the following subsection, we shall introduce different types of LLMs based
on various Transformer architectures.

Table 1. The Comparison of Different Model Structures

Model | Model Name | Decoder | Encoder | Attention Mechanisms
---|---|---|---|---
Causal | Prefix
GPT series | GPT-1 | ✓ | - | - | Masked unidirectional multi-head self-attention
GPT-2 | ✓ | - | - | Masked unidirectional multi-head self-attention
GPT-3 | ✓ | - | - | Sparse unidirectional attention (Factorized attention)
GPT-3.5 | ✓ | - | - | Sparse unidirectional attention (Factorized attention)
GPT-4 | ✓ | - | - | Multi-query unidirectional Attention
LLaMA series | LLaMA | ✓ | - | - | Causal unidirectional multi-head attention
LLaMA2 | ✓ | - | - | Grouped-query unidirectional attention
GLM series | GLM | - | ✓ | - | bidirectional self-attention
BERT series | BERT | - | - | ✓ | bidirectional self-attention
UNILM series | UNILM | - | - | ✓ | bidirectional self-attention
BART series | BERT | ✓ | - | ✓ | Masked multi-head self-attention & Cross-attention between encoder and decoder
T5 series | T5 | ✓ | - | ✓ | Masked multi-head self-attention & Cross-attention between encoder and decoder

Figure 1. The matrix comparison of attention mask patterns between decoder-
only and encoder-decoder architectures. The matrix uses dark cells to allow
for self-attention of input elements $j$ at the output time step $i$, while
light cells restrict this attention. The left panel represents the full input
attention, the middle panel refers to preventing future input reliance, and
the right panel combines causal masking with a prefix for partial input
sequence fully-visible masking. (raffel2020exploring, )

###  2.1. Decoder-only Transformer Architecture

The decoder-only model (raffel2020exploring, ), functioning independently
without an encoder, can act as a language model designed primarily for next-
step prediction (liu2018generating, ; radford2018improving, ; al2019character,
). During the training of language models, decoder-only model takes on the
task of generating the target sequence.

####  2.1.1. Causal Decoder

The causal decoder architecture employs unidirectional attention masking to
ensure that each input token can only attend to past tokens and itself. Both
input and output tokens are processed in a similar manner within the decoder.
The schematic for this architecture is depicted in the middle panel of Figure
1.

##### GPT Series

The Generative Pre-trained Transformer (GPT) model has garnered significant
attention and interest in the field of NLP (ye2023comprehensive, ), and the
technical evolution of the GPT series models is illustrated in Figure 2.
Positioned as a cutting-edge technology built upon the Transformer
architecture, the versatility and robust performance of the GPT model
establish it as a universal solution for various NLP tasks.

Figure 2. A brief illustration for the technical evolution of GPT-series
models. We created this flowchart primarily relying on information from
research papers, blog articles, and official APIs provided by OpenAI. The
solid line represents explicit evidence of the evolutionary path between the
two models (e.g., the official statement that a new model is developed based
on a base model), while the dashed line denotes a relatively weaker
evolutionary relationship.

###### GPT-1.

Proposed by the OpenAI team, GPT-1 (radford2018improving, ), short for
Generative Pre-trained Transformer 1 (vaswani2017attention, ), serves as the
foundational model for the GPT-series, establishing the key architecture and
the fundamental principle of modeling natural language text, specifically
predicting the next word. GPT-1 adopts a decoder-only Transformer
architecture, implementing a semi-supervised approach for language
understanding through a combination of unsupervised pre-training and
supervised fine-tuning.

###### GPT-2.

GPT-2 (radford2019language, ), is an extension of the GPT-1 architecture
(radford2018improving, ), where the parameter scale is increased to 1.5
billion. Trained on the extensive WebText dataset, GPT-2 aims to perform
diverse tasks through unsupervised language modeling, eliminating the need for
explicit fine-tuning with labeled data. GPT-2 utilizes a probabilistic form
for multitask solving, denoted as $p(output|input,task)$, predicting the
output based on input and task information. This approach is reminiscent of
similar methods found in (mccann2018natural, ). Leveraging a multi-layer self-
attentive mechanism, GPT-2 achieves fully-connected cross-attention to the
entire context in a computationally efficient manner.

###### GPT-3.

GPT-3 (brown2020language, ) employs attention mechanisms, allowing the model
to selectively focus on segments of input text it deems most relevant and
adopts an autoregressive approach, leveraging the transformative power of the
Transformer architecture.

###### GPT-3.5.

The GPT-3.5 models, also known as InstructGPT (ouyang2022training, ), are
developed based on the code-davinci-002 underscoring the efficacy of training
on code data to improve the reasoning capacity of GPT models (zhao2023survey,
). InstructGPT, is a language model fine-tuned on GPT-3 (brown2020language, )
using a combination of supervised learning and reinforcement learning based on
human feedback (RLHF). GPT-3.5 is utilized to develop the chatbot product
ChatGPT (openai_chatgpt, ; lock2022ai, ). The gpt-3.5-turbo is the most
capable and cost-effective GPT-3.5 model. GPT-3.5 shows improvements in
maintaining truthfulness and reducing toxic outputs, demonstrating that fine-
tuning with human feedback is an effective approach to enhance the alignment
of language models with human intent.

###### GPT-4.

GPT-4 (openai2023gpt4, ), is a multimodal LLM, accepting both image and text
inputs and produces text outputs. Despite being less capable than humans in
various real-world scenarios, GPT-4 showcases human-level performance across a
range of professional and academic benchmarks. Notably, GPT-4 has improved
calibration compared to GPT-3.5 (ouyang2022training, ), exhibiting enhanced
accuracy in predicting the correctness of its answers. GPT-4’s versatility
extends to ChatGPT (openai_chatgpt, ), where it can process images as inputs.

###### ChatGPT.

ChatGPT (openai_chatgpt, ), short for Chat Generative Pre-trained Transformer,
represents a chatbot innovation. Leveraging a robust LLM, it empowers users to
shape and guide conversations according to specific criteria such as length,
format, style, level of detail, and language (lock2022chatgpt, ). ChatGPT is
built upon either GPT-3.5 (ouyang2022training, ) or GPT-4 (openai2023gpt4, ),
and is fine-tuned for conversational applications using a combination of
supervised learning and reinforcement learning (gertner2023wikipedia, ).
ChatGPT Plus (chatgpt-openai, )is a version of ChatGPT backed by GPT-4. Users
of ChatGPT Plus have the ability to upload images, and mobile app users can
engage in conversations with the chatbot (roose2023chatgpt, ).

###### GPTs.

GPTs (openaigpts, ) represent an innovative feature enabling users to
customize versions of ChatGPT (openai_chatgpt, ) for specific purposes. These
personalized GPTs assist users in enhancing daily efficiency, optimizing
performance in specific tasks, and simplifying work or home activities. Users
can share their creations with others to promote the utility of these purpose-
built GPTs. GPTs prioritize privacy and safety by ensuring user control over
data within the ChatGPT environment. Interactions with GPTs remain
confidential, with no sharing of user conversations with the creators.

##### LLAMA Series

LLaMA (touvron2023llama, ), short for Large Language Model Meta AI, is a
family of LLMs, released by Meta AI (llama2023, ). Due to its openness and
effectiveness, LLaMA attracts significant attention from the research
community, and many efforts are devoted to fine-tuning or continually pre-
training its different model versions for implementing new models or tools.
The technical evolution of the LLaMA series models is illustrated in Figure 3.

Figure 3. A brief illustration for LLaMA series.

###### LLAMA.

LLaMA (touvron2023llama, ), a collection of foundation language models ranging
from 7B to 65B parameters, is trained on trillions of tokens, and show that it
is possible to train state-of-the-art models using publicly available datasets
exclusively, without resorting to proprietary and inaccessible datasets. In
particular, LLaMA-13B outperforms GPT-3 (brown2020language, ) (175B) on most
benchmarks, and LLaMA-65B is competitive with the best models at that time,
Chinchilla-70B (hoffmann2022training, ) and PaLM-540B (rae2021scaling, ).

###### LLAMA2.

LLaMA2 (llama2announcement, ), the next iteration in the LLaMA series, has
been unveiled through a collaborative effort between Meta and Microsoft. Llama
2 is a collection of pre-trained and fine-tuned LLMs ranging in scale from 7
billion to 70 billion parameters. The Llama 2 pre-trained models, trained on 2
trillion tokens, feature double the context length compared to Llama. The
release of LLaMA2 includes model weights and starting code for pre-trained and
fine-tuned LLaMA language models (Llama Chat, Code Llama).

###### LLAMA2 CHAT.

LLaMA2 includes both foundational models and models fine-tuned for dialog,
called LLaMA 2-Chat (touvron2023llama2, ). Llama 2-Chat, are optimized for
dialogue use cases. The models outperform open-source chat models on most
benchmarks tested, and, based on human evaluations for helpfulness and safety,
may be a suitable substitute for closed-source models.

###### CODE LLAMA.

Code Llama (rozière2023code, ) contains three versions with different
parameter amounts, i.e., 700 million parameter version, 1.3 billion parameter
version and 34 billion parameter version. When training the basic model, first
initialize the weights with the Llama 2 model (llama2announcement, ) with the
same number of parameters, and then train on the code dataset of 500 billion
words. Meta has also fine-tuned the trained basic model in two different
styles: Python expert version (plus 100 billion additional words), and
instruction fine-tuning version which can understand natural language
instructions.

####  2.1.2. Prefix Decoder

The prefix decoder structure (raffel2020exploring, ) modifies the causal
encoder’s masking mechanism to enable bidirectional attention on prefix tokens
while maintaining unidirectional attention on generated tokens. Similar to the
encoder-decoder paradigm, this allows bidirectional encoding of prefix
sequences and auto-regressive generation of output tokens, sharing the same
parameters during both encoding and decoding phases. Prominent models
utilizing the prefix decoder architecture include U-PaLM, GLM-130B, and others
in large-scale prefix encoders. The schematic for this architecture is
depicted in the right panel of Figure 1.

##### GLM

GLM (du2021glm, ), short for General Language Model, is a comprehensive pre-
training framework. The technical evolution of the GLM series models is
illustrated in Figure 4. The architecture modifications in GLM include
rearranging the order of layer normalization and the residual connection,
using a single linear layer for output token prediction, and replacing ReLU
activation functions with GeLUs. GLM unifies pre-training objectives for
various tasks under the common framework of autoregressive blank infilling,
employing mixed attention masks and novel 2D position encodings.

Figure 4. A brief illustration for GLM series.

##### ChatGLM series

ChatGLM (zeng2022glm, ) is a bilingual model incorporating question-answering,
multi-turn dialogue, and code generation. Built upon GLM-130B (zeng2022glm, ),
ChatGLM follows the design principles of ChatGPT (openai_chatgpt, ).

###### ChatGLM-6B.

ChatGLM-6B (zeng2022glm, ), as the inaugural ChatGLM conversational model,
builds upon the training insights from GLM-130B. It resolves issues in the
implementation of 2D RoPE positional encoding and adopts a conventional
FeedForward Network(FFN) structure.

###### ChatGLM2-6B.

ChatGLM2-6B (thudm2022chatglm2-6b, ), incorporates a blend of target functions
within the GLM framework, undergoing pre-training with 1.4 trillion Chinese
and English identifiers and aligning with human preferences. Leveraging
FlashAttention technology, ChatGLM2-6B extends the base model’s context length
from 2K (ChatGLM-6B) to 32K, employing an 8K context length during dialogue
training. With the integration of Multi-Query Attention technology,
ChatGLM2-6B achieves more efficient inference speed and lower GPU memory
usage, boasting a 42% improvement in inference speed compared to its
predecessor in the official model implementation.

###### ChatGLM3-6B.

ChatGLM3-6B (THUDM2023chatglm3, ) introduces a newly designed Prompt format,
accommodating normal multi-turn conversations as well as native support for
complex scenarios such as tool invocation (Function Call), code execution
(Code Interpreter), and Agent tasks.

###  2.2. Encoder-only Transformer Architecture

Unlike decoder-only and encoder-decoder LLMs that utilize autoregressive
regression, the encoder-only LLMs emphasize comprehension of input content and
generation of task-specific outputs.

####  2.2.1. BERT Series

BERT (devlin2018bert, ), short for Bidirectional Encoder Representations from
Transformers, is a language model based on the transformer architecture, known
for its significant improvement over previous state-of-the-art models. The
brief illustration for BERT series is shown in Figure 5.

Figure 5. A brief illustration for BERT series.

##### BERT

BERT (devlin2018bert, ) namely a bidirectional Transformer-based encoder, is a
LLM introduced by the Google AI Language team (bertopensourcing, ). It
incorporates masked language modeling, allowing pre-training to capture
interactions between left and right context words. Recent advancements, such
as extended training duration, parameter tying across layers, and span masking
instead of individual words, have demonstrated improved performance. Notably,
BERT’s auto-regressive predictions limit its effectiveness for generation
tasks.

##### RoBERTa

RoBERTa (liu2019roberta, ), namely Robustly optimized BERT (devlin2018bert, )
approach, improves upon BERT (devlin2018bert, ) through straightforward
modifications, such as longer training with larger batches, removal of the
next sentence prediction objective, and dynamic changes to the masking
pattern, and employs a more extensive byte-level Byte Pair Encoding (BPE)
vocabulary. However, unlike BERT, RoBERTa streamlines its training process by
omitting the Next Sentence Prediction (NSP) task and focusing solely on
optimizing the Masked Language Model (MLM) task. This approach enhances the
model’s ability to learn bidirectional contextual information in language.

####  2.2.2. UNiLM

UNILM (dong2019unified, ) is a unified pre-training model jointly optimized
for multiple language modeling objectives with shared parameters, covering
bidirectional, unidirectional, and sequence-to-sequence language models. The
model undergoes pre-training using three types of language modeling tasks:
unidirectional, bidirectional, and sequence-to-sequence prediction. Unified
modeling is achieved through a shared Transformer network, incorporating
specific self-attention masks to control the contextual conditions for
predictions.

###  2.3. Encoder-decoder Transformer Architecture

The traditional Transformer model is built on an encoder-decoder architecture
(raffel2020exploring, ), consisting of two Transformer blocks serving as the
encoder and decoder, respectively. The encoder utilizes stacked multi-head
self-attention layers to encode the input sequence and generate its latent
representation. Meanwhile, the decoder performs cross-attention on these
representations and autoregressively generates the target sequence. The
schematic for this architecture is depicted in the left panel of Figure 1.

####  2.3.1. BART

BART (lewis2019bart, ) is a denoising autoencoder designed for pre-training
sequence-to-sequence models, combining Bidirectional and Auto-Regressive
Transformers. It employs a sequence-to-sequence model applicable to diverse
end tasks, utilizing a two-stage pre-training process involving text
corruption and reconstruction. BART’s architecture, based on the standard
Transformer for neural machine translation, can be seen as a generalization of
BERT, GPT, and other recent pre-training schemes.

####  2.3.2. Text-to-Text Transfer Transformer

T5 (raffel2020exploring, ), known as the ”Text-to-Text Transfer Transformer,”
adopts the Transformer encoder-decoder structure, treating each text
processing challenge as a ”text-to-text” task—generating new text from given
input. This framework ensures consistent application of the same model,
objective, training procedure, and decoding process across all tasks. The
model employs an unsupervised span-corruption objective and a multi-task pre-
training strategy. T5 establishes a unified model framework for various NLP
tasks, simplifying the evaluation of different structures, pre-training
objectives, and datasets across reading comprehension, summarization, and text
classification tasks.

##  3\. Fine-Tuning

###  3.1. Full Fine-Tuning

Full Fine-Tuning (FFT) is a cornerstone technique in the domain of neural
network adaptation, it involves optimizing all model parameters to integrate
task-specific knowledge into the foundational architecture acquired during
pre-training. FFT is instrumental in tailoring these models for specialized
applications, ranging from nuanced language understanding to domain-specific
tasks.

FFT is based on optimizing a pre-trained model’s parameters. For a model $M$
characterized by parameters $\theta$, and a dataset $D$ designed for a
specific task, the objective of FFT is to find the optimal parameter
configuration $\theta^{*}$ that minimizes a defined loss function
$\mathcal{L}$. This can be formally expressed as:

(1) $\theta^{*}=\underset{\theta}{\mathrm{argmin}}\ \mathcal{L}(M(\theta),D),$

where $\mathcal{L}(M(\theta),D)$ represents the loss function that quantifies
the disparity between the model’s predictions and the true outcomes in $D$.

Selecting a pre-trained model that has learned from a wide variety of data
begins the process. The next step involves preparing a dataset tailored to the
specific task at hand. The core of FFT lies in thoroughly optimizing all model
parameters to reduce the loss associated with the task. Additionally, the
process includes enhancing the model’s learning through techniques such as
data augmentation, advanced regularization, and optimizing the learning rate.

FFT is valued for its capacity to incorporate detailed, task-specific features
into a model, thereby boosting its accuracy and effectiveness. This approach
has contributed to significant advancements in fields like natural language
processing, computer vision, and predictive analytics.

Despite FFT’s effectiveness in deeply adapting models, the development of
neural network technologies has led to more focused fine-tuning methods that
conserve resources. Parameter-Efficient Fine-Tuning (PEFT) methods, for
example, adjust only a portion of the parameters, achieving a balance between
thorough adaptation and computational demand, making model adaptation feasible
even with limited resources.

###  3.2. Parameter-efficient Fine-Tuning

Parameter-Efficient Fine-Tuning (PEFT) methods have gained popularity due to
their ability to fine-tune pre-trained models without altering all the model
parameters(houlsby2019parameter, ; pfeiffer2020Adapter, ; liu2021ptuning, ;
hu2021lora, ). This section provides an overview of several PEFT techniques
that have been developed, highlighting their key concepts and contributions to
the field.

####  3.2.1. Adapters

Adapters have emerged as an innovative approach within the domain of
Parameter-Efficient Fine-Tuning, particularly for adapting large pre-trained
models to specific tasks. Initially conceptualized by Houlsby et al.
(houlsby2019parameter, ), Adapters are strategically inserted between the
layers of a pre-trained model, allowing for the original model parameters to
remain unaltered while the adapters learn the nuances of the task-specific
features.

The architecture of an Adapter is characterized by a down-projection layer, a
non-linear activation function, and an up-projection layer. The down-
projection layer compresses the input to a lower dimension, the activation
function introduces non-linearity to enable complex mappings, and the up-
projection layer expands the transformed representation back to the original
dimensionality. It can be mathematically depicted as:

(2)
$\text{Adapter}(\mathbf{x})=\mathbf{U}(\text{Activation}(\mathbf{D}\mathbf{x}+\mathbf{b}_{d}))+\mathbf{b}_{u},$

where $\mathbf{x}$ is the input, $\mathbf{D}$ and $\mathbf{U}$ denote the
down-projection and up-projection matrices, and $\mathbf{b}_{d}$ and
$\mathbf{b}_{u}$ are their corresponding bias vectors.

Adapters are integrated into pre-trained models by inserting them after the
feedforward networks of each layer. The output of a layer, after incorporating
an Adapter, is a summation of the original layer output and the Adapter’s
processed output. This is represented as:

(3)
$\text{Layer}_{\text{output}}^{\text{mod}}=\text{Layer}_{\text{output}}+\text{Adapter}(\text{Layer}_{\text{output}}),$

where $\text{Layer}_{\text{output}}$ is the initial output of a layer in the
model.

In the Adapter framework, the training phase is exclusively focused on the
parameters of the Adapters, with the remaining model parameters being kept
static. This selective training targets the minimization of a task-specific
loss function $\mathcal{L}_{\text{task}}$, formulated as:

(4)
$\theta_{\text{adapter}}^{*}=\underset{\theta_{\text{adapter}}}{\mathrm{argmin}}\
\mathcal{L}_{\text{task}}(M_{\text{adapter}}(\theta_{\text{adapter}}),D_{\text{task}}),$

with $\theta_{\text{adapter}}$ representing the Adapter parameters,
$M_{\text{adapter}}$ the model including Adapters, and $D_{\text{task}}$ the
specific dataset for the task.

Adapters offer distinct advantages in fine-tuning scenarios. They require
training significantly fewer parameters compared to full model fine-tuning,
leading to a more resource-efficient training process. Additionally, adapters’
modular nature allows for easy insertion and removal from models, enabling
swift adaptation to various tasks. Importantly, by keeping the original model
parameters frozen, Adapters preserve the foundational knowledge and
representations learned during pre-training, ensuring that the integrity and
robustness of the pre-trained model are maintained.

####  3.2.2. LoRA

LoRA (Low-Rank Adaptation) (hu2021lora, ) is a parameter-efficient fine-tuning
method that modifies a pre-trained model by introducing low-rank updates to
specific weight matrices. It allows for significant changes in the model’s
behavior while only training a small number of additional parameters. The idea
behind LoRA is to update the weights of the model using low-rank matrices,
which significantly reduces the number of parameters to be fine-tuned. For a
weight matrix $\mathbf{W}\in\mathbb{R}^{m\times n}$, the low-rank update is
given by:

(5) $\Delta\mathbf{W}=\mathbf{B}\mathbf{A},$

where $\mathbf{B}\in\mathbb{R}^{m\times r}$ and
$\mathbf{A}\in\mathbb{R}^{r\times n}$ are the low-rank matrices, and $r$ is
the rank which is much smaller than $m$ and $n$.

In practice, LoRA is applied to specific layers of a neural network, such as
the attention and feedforward layers in transformer models
(vaswani2017attention, ). The updated weight matrix is:

(6) $\mathbf{W}^{\prime}=\mathbf{W}+\Delta\mathbf{W},$

where $\mathbf{W}^{\prime}$ is the new weight matrix used during fine-tuning
and inference.

LoRA offers several advantages: By updating only a small number of parameters,
LoRA reduces computational and memory requirements. It can be applied to
various layers of a network, allowing for targeted modifications. Since the
original weights are not discarded, LoRA maintains the rich representations
learned during pre-training.

Several extensions of LoRA have been proposed to further enhance its
efficiency. For instance, Quantized LoRA (QLoRA) (dettmers2023qlora, ) is an
advancement in the efficient fine-tuning. It combines the principles of LoRA
with 4-bit quantization to reduce the memory footprint significantly. QLoRA
enables the fine-tuning of extremely large models on limited hardware
resources while maintaining task performance.

####  3.2.3. Instruction Fine-Tuning

Instruction Fine-Tuning (IFT) (wei2021finetuned, ) is an approach to enhance
the capabilities of PLMs by leveraging task-specific instructions. This
technique adapts PLMs to better understand and execute instructions, thus
improving their performance on a wide range of tasks.

The core idea of IFT involves fine-tuning a pre-trained LM on a dataset where
each data point includes a specific instruction and its associated input-
output pair. The goal is to enable the LM to comprehend and follow the
instructions for generating the desired output.

IFT is particularly beneficial for tasks requiring nuanced understanding and
execution of complex instructions. By preserving the original model
architecture, it offers an efficient way to extend the applicability of PLMs
to new tasks and domains without extensive architectural modifications.

###  3.3. Mitigating Fine-Tuning Instabilities

Fine-tuning pre-trained models, particularly in deep learning applications,
often encounters instabilities leading to suboptimal
performance(zhang2020revisiting, ; mosbach2021on, ). Instabilities manifest as
erratic loss landscapes, difficulty in convergence, and sensitivity to
hyperparameter settings. A common issue is representational collapse, where
the model’s representations become less expressive over the course of fine-
tuning (aghajanyan2020better, ). Researchers have developed various strategies
to stabilize the fine-tuning process and enhance the robustness of fine-tuned
models (dodge2020fine; mosbach2021on, ).

Aghajanyan et al. (aghajanyan2020better, ) present methods to mitigate
representational collapse during fine-tuning of pre-trained models. They
introduced robust representation through Regularization Fine-tuning (R3F) and
Regularization and Reparameterization Fine-tuning (R4F), with the former
adding regularization terms to the loss function and the latter extending R3F
by introducing reparameterization. The approaches are formulated as:

(7) R3F Loss:
$\displaystyle\mathcal{L}_{\text{R3F}}=\mathcal{L}_{\text{original}}+\lambda\cdot\mathcal{R}(\theta),$
(8) R4F Loss:
$\displaystyle\mathcal{L}_{\text{R4F}}=\mathcal{L}_{\text{original}}+\lambda\cdot(\mathcal{R}(\theta)+\mathcal{R}(\text{Reparam}(\theta))),$

where $\mathcal{L}_{\text{original}}$ is the original loss function, $\lambda$
is the regularization strength, $\mathcal{R}$ is the regularization term, and
Reparam is the reparameterization function.

Jiang et al. (jiang2019smart, ) propose the SMART (Smoothness-inducing
Adversarial Regularization for Multitask Training) to enhance fine-tuning of
pre-trained models through regularized optimization. It controls model
complexity using Smoothness-inducing Adversarial Regularization and stabilizes
updates with Bregman Proximal Point Optimization. The method is designed to
improve robustness and efficiency in fine-tuning PLMs, addressing issues like
overfitting and improving generalization in NLP tasks.

(9)
$\min_{\theta}\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\max_{\delta}\mathcal{L}(f(x+\delta;\theta),y)-\rho\cdot\|\delta\|_{2}^{2}\right],$

where $\delta$ represents adversarial perturbations, $\rho$ balances the
adversarial objective with regularization, and $f$ is the predictive function
of the model.

Zhu et al. (zhu2019freelb, ) propose Free Large-Batch Adversarial
Training(FreeLB), which is an adversarial training algorithm for improving the
robustness and generalization of PLMs (zhu2019freelb, ). It enhances training
by adding adversarial perturbations to word embeddings and minimizing
adversarial risk.

(10)
$\min_{\theta}\frac{1}{N}\sum_{i=1}^{N}\max_{\delta_{i}}\mathcal{L}(f(x_{i}+\delta_{i};\theta),y_{i})-\rho\cdot\|\delta_{i}\|_{2}^{2},$

where $N$ is the batch size, $\delta_{i}$ are the adversarial perturbations,
and $\rho$ balances the original loss with the adversarial term.

##  4\. Prompt Engineering

Prompt engineering have gained significant attention in the field of helping
PLMs understand a given task. Prompt engineering can be divided into two
methods: Prompt Tuning and Tuning-free Prompting, which will be discussed in
this section.

###  4.1. Prompt Tuning

Prompt tuning involves modifying the parameters of the pre-trained model or
adjusting additional prompt-related parameters to enhance the adaptation of
the pre-trained model to downstream tasks. A prominent approach in this regard
is Pattern-Verbalizer-Pair (PVP) structure, which is initially proposed by
(schick2020exploiting, ). Subsequent Prompt Tuning methods largely build upon
the PVP framework. As shown in Figure 6, Gao et al. (gao2020making, ) has
proved that the choice of templates and labels can lead to substantial
differences in final accuracy. Therefore, current research primarily focuses
on how to select or construct appropriate Patterns and Verbalizers. The design
of prompts can be categorized into discrete prompts and continuous prompts.

Figure 6. The impact of templates and label words on prompt tuning.

####  4.1.1. Discrete Prompts

Previous efforts have been devoted to the exploration of discrete prompts,
also referred to as hard prompts, which usually correspond to readable
language phrases. The word embeddings of discrete template remain unchanged
during training and won’t introduce any new parameter to the LLMs. In the
following sections, we provide a detailed overview of several methods proposed
for this purpose:

##### D1:Pattern-Exploiting Training

Pattern-Exploiting Training (PET) (schick-schutze-2021-exploiting, )
constructs a prompt set by manually generating PVP content p = (_P_ , _v_),
where $P$ is a function that takes $x$ as input and outputs a phrase or
sentence $P(x)\in V^{*}$ that contains exactly one mask token. $v$ is an
injective function $\mathcal{L}\rightarrow V$ that maps each label to a word
in the masked language model vocabulary $M$. The PET can be formulated by:

(11) $\displaystyle s_{p}(l\ |\ x)=M(v(l)\ |\ P(x)),$ (12) $\displaystyle
q_{p}(l\ |\
x)=\frac{e^{s_{p}(l|x)}}{{\textstyle\sum_{l^{\prime}\in\mathcal{L}}e^{s_{p}(l^{\prime}|x)}}},$

where $s_{p}$ is the score for label $l\in\mathcal{L}$, $q_{p}$ is the
probability distribution over labels using softmax. Then, the cross-entropy
between $q_{p}(l\ |\ x)$ and the true (one-hot) is used as the loss function
for fine-tuning $M$ in p.

##### D2:LM-BFF

Finding the right prompts is cumbersome and fallible, which requires both
domain expertise and understanding of NLP. Even if significant effort is
invested, manual prompts are likely to be suboptimal. Therefore, LM-BFF
(gao2020making, ) automatically generates prompts, including pruned brute-
force searches to identify the best working label words, and utilizes the T5
model to automatically generate templates (raffel2020exploring, ). Given a
fixed template $T(x)$, constructing a pruned set $V_{c}\subset V$ of the top
$k$ vocabulary words based on their conditional likelihood using the initial
$L$ can be formulated as:

(13) $\displaystyle{}_{v\in V}^{{Top-k}}\left\\{\sum_{x_{in}\in
D_{train}^{c}}\log{P_{\mathcal{L}}([MASK]=v\ |\ T(x_{in}))}\right\\},$

where $P_{\mathcal{L}}$ denotes the output probability distribution of
$\mathcal{L}$.

##### D3:Prompt Tuning with Rules

Prompt Tuning with Rules (PTR) (han2022ptr, ) can apply logical rules to
automatically compose a small number of manually created sub-prompts into
final task-specific prompts. Take relation classification as an example, given
a sentence $x=\\{...e_{s}...e_{o}...\\}$, where $e_{s}$ and $e_{o}$ are the
subject entity and the object entity respectively, the sub-prompt template and
label word set of $e_{s}$ and $e_{o}$ can be formalized as:

(14) $\displaystyle T_{e_{s}/e_{o}}=``x\ the\ [MASK]\ e_{s}/e_{o}",$
$\displaystyle V_{e_{s}/e_{o}}=\\{``person",``organization",...\\},$

and the sub-prompt of relationship between $e_{s}$ and $e_{o}$ can be
formalized as:

(15) $\displaystyle T_{e_{s}/e_{o}}=``x\ e_{s}\ [MASK]\ e_{o}",$
$\displaystyle V_{e_{s}/e_{o}}=\\{``^{\prime}s\ parent\ was",``was\ born\
in",...\\},$

by aggregating the sub-prompts, the complete prompt is as follows,

(16) $\displaystyle T=``x\ the\ [MASK]_{1}\ e_{s}\ [MASK]_{2}\ the\
[MASK]_{3}\ e_{o}",$ $\displaystyle
V_{[MASK]_{1}}=\\{``person",``organization",...\\},$ $\displaystyle
V_{[MASK]_{2}}=\\{``^{\prime}s\ parent\ was,``was\ born\ in",...\\},$
$\displaystyle V_{[MASK]_{3}}=\\{``person",``organization",...\\},$

the final learning objective of PTR is to maximize

(17) $\displaystyle\frac{1}{|X|}\sum_{x\in
X}\log{\prod_{j=1}^{n}}p\left([MASK]_{j}=\phi_{j}(y)|T(x)\right),$

where $\phi_{j}(y)$ is to map the class $y$ to the set of label words
$V_{[MASK]_{j}}$ for the $j$-th masked position $[MASK]_{j}$.

##### D4:Knowledgeable Prompt-tuning

Knowledgeable prompt-tuning (KPT) (hu2021knowledgeable, ) generate a set of
label words for each label using external knowledge bases (KBs). The expanded
label words are not simply synonyms of each other, but cover different
granularities and perspectives, thus are more comprehensive and unbiased than
the class name. Then, refinement methods are proposed to cope with the noise
in the generated label words. This preserves high-quality words that can be
used to fine tune PLMs and demonstrate the effectiveness of in-context
learning (ICL).

####  4.1.2. Continuous Prompts

Instead of creating readable language phrases, continuous prompts convert the
prompts into continuous vectors. Continuous prompts have their own parameters
that can be tuned based on training data from downstream tasks.

##### C1:P-tuning

P-tuning (liu2023gpt, ) is an early attempt to achieve prompt tuning by
employing continuous prompts. Instead of readable templates in discrete
prompts, P-tuning uses continuous prompt embedding $p_{i}$ to build the prompt
template:

(18) $\displaystyle T=\\{[P_{0:i}],x,[p_{(i+q:j)}],y,[P_{j+1}:k]\\},$

and then leverages an extra embedding function $f:[p_{i}]\to h_{i}$ to map the
template to:

(19)
$\displaystyle\\{h_{0},...,h_{i},e(x),h_{(i+1)},...h_{j},e(y),h_{j+1},...j_{k}\\},$

with the function $f$, task loss function can be optimized by updating the
embedding $\\{P_{i}\\}_{i=1}^{k}$.

##### C2:Prefix Tuning

Prefix tuning(li2021prefix, ) freezes the parameters of PLMs and only
optimizes the prefix parameters. Consequently, we only need to store the
prefix for each task, making prefix-tuning modular and space-efficient. Given
a trainable prefix matrix $M_{\phi}$ and a fixed pre-trained LM parameterized
by $\theta$, the training objective is the same as that of full fine-tuning:

(20) $\displaystyle\max_{\phi}{\log{P_{\phi}(y|x)}=\max_{\phi}\sum_{i\in
Y_{idx}}\log{P_{\phi}(z_{i}|h_{<i})}},$

where $h_{<i}$ is the concatenation of all neural network layers at time step
$i$. If $i\in P_{idx}$, $h_{i}=P_{\theta}[i,:]$, otherwise,
$h_{i}=LM_{\phi}(z_{i},h_{<i})$.

Lester et al. propose prompt tuning (lester2021power, ), which can be regarded
as a simplification of prefix tuning. It greatly reduces the number of
parameters and for the first time demonstrates that using only prompt tuning
is also competitive.

Figure 7. From P-tuning to P-tuning v2. (liu2021p, )

##### C3: P-tuning V2

P-tuning v2 (liu2021p, ) is an implementation of prefix-tuning (li2021prefix,
) and mixture of soft prompts (qin2021learning, ). As shown in Figure 7, when
compared to P-tuning, it not only has more tunable task-specific parameters
(from 0.01% to 0.1%-3%) to allow more per-task capacity while being parameter-
efficient, but also adds prompts to deeper layers have more direct impact on
model predictions. P-tuning v2 is always comparable to fine-tuning at all
scales but with only 0.1% task-specific parameters needed comparing to fine-
tuning, proving that prompt-tuning can effectively assist PLMs in adapting to
downstream tasks.

###  4.2. Tuning-free Prompting

Tuning-free Prompting directly generate answers without modifying the
parameters of the PLMs. These can optionally leverage response prompts to
enhance inputs, previous studies have explored the impact of prompts on the
generation effectiveness of PLMs and have provided numerous tricks for
creating prompts. The mainstream tuning-free prompting methods include ICL and
Chain-of-thought (CoT).

####  4.2.1. In-Context Learning

ICL uses multiple input-output demonstration pairs to PLMs to generate the
desired response, which is first proposed along with GPT-3. ICL is an
effective method as no parameter is tuned, an example prompt of ICL is shown
in the Figure 8.

Previous researches have demonstrated that ICL can acquire knowledge of the
target tasks’ label space, the distribution of the input text and the input-
label correspondence from in-context examples. The similarity between in-
context prompts and the target task also significantly influences the
performance of ICL. Generally, the performance tends to improve when in-
context prompts are closer to the test samples in the embedding space
(liu2021makes, ). The arrangement of in-context prompts itself also exerts a
substantial impact on the performance of ICL, which becomes particularly
pronounced in smaller-scale models (lu2021fantastically, ). Therefore, many
researchers are devoted to exploring methodologies for constructing well-
performing in-context prompts and many methods (chen2022improving, ;
chen2021meta, ; min2021metaicl, ) based on ICL are proposed to better design
in-context prompts.

Figure 8. Examples of ICL and CoT.

####  4.2.2. Chain-of-Thought

CoT (wei2022chain, ) improves performance on a range of arithmetic,
commonsense, and symbolic reasoning tasks by mimics a step-by-step thought
process for arriving at the answer. An example prompt of CoT is shown in the
Figure 8. Zero-shot CoT (kojima2022large, ) is a classical CoT method that
guides the model in reasoning and generating results by employing the same
prompt ”Let’s think step by step,” across different tasks. Another classical
approach is Least-to-Most (zhou2022least, ), which decomposes the target
problem into a series of simpler sub-problems, solving them gradually.

A more widespread approach involves feeding the model a set of CoT prompts
designed for step-by-step reasoning to guide its thinking. Similar to ICL, the
selection of prompts significantly influences the generated results, prompting
extensive efforts to identify optimal prompts through methods such as voting
or automatically generating prompts (zhang2022automatic, ; shum2023automatic,
). Additionally, recent researches have focused on exploring the application
of CoT in the context of multimodal (zhang2023multimodal, ) and multilingual
(huang2023not, ) scenarios.

##  5\. LLM Based Task-oriented Dialogue Systems

The TOD system assists users in achieving specific domain-related goals, such
as hotel reservations or restaurant queries, through interactive
conversations. Due to its pronounced utility, this technology has garnered
increasing attention from researchers in recent years. Generally speaking, TOD
can be divided into pipeline-based TOD and end-to-end TOD. In this section, we
will give a comprehensive introduction to LLM-based TOD.

As shown in Figure 9, the pipeline-based TOD system comprises four connected
modules: (1) NLU, employed for extracting the user intent, and filling slots;
(2) Dialogue State Tracking (DST), a pivotal module in the Pipeline-based TOD,
utilized to track the dialogue state for the current turn based on the output
of the NLU module and historical inputs of the dialogue; (3) Policy Learning
(PL), determining the subsequent action based on the dialogue state generated
from the DST module; (4) NLG, the final module in the Pipeline-based TOD
system, transforming dialogue actions generated by the PL module into
comprehensible natural language. Dialogue Manager (DM) is the central
controller of a pipeline-based TOD system, which is comprised of DST module
and PL module.

###  5.1. Pipeline-based Methods

Figure 9. Pipeline-based task-oriented dialogue framework.

As each module in pipeline-based TOD systems is trained independently, any
module’s failure in adapting to sub-tasks can result in terrible performance
of the entire system. Simultaneously, as the Pipeline-based TOD system
sequentially solve all the sub-tasks, errors accumulate between modules,
leading to the error propagation problem. However, since each module in the
Pipeline-based TOD system operates separately, ensuring consistent input and
output, it becomes convenient to interchange individual modules within the
Pipeline-based TOD system. With the development of PLMs, large-scale language
modules fine-tuned through different approaches can be easily accessed and
seamlessly integrated to replace modules within the TOD system, which enables
users to easily adapt the system to sub-tasks in the target domain.

####  5.1.1. Natural Language Understanding

The NLU module identifies and extracts information such as user intent and
slot values from the natural language input provided by the user. Generally,
the output of the NLU module is as follows: $Un=(In,Zn)$, where $In$ is the
detected user intent and $Zn$ is the slot-value pair. For example, in
restaurant recommendation tasks, the user intent is “find-restaurant” and the
domain is “restaurant”. Slot filling can be regarded as a sequence
classification task. For example, the user inputs a message: “I am looking for
a place to eat in the east that is expensive,” and the NLU module reads this
input and generates the following slot-value pairs: {restaurant-area: east,
restaurant-pricerange: expensive}.

##### Intent detection

Deep-learning based methods (deng2012use, ; tur2012towards, ) are widely used
in solving intent detecting tasks. In particular, many methods based on neural
networks achieve promising performances. However, with the development of
LLMs, numerous researchers apply LLMs to solving TOD tasks and many methods
achieve great performance. Comi et al. propose a pipeline-based intent
detection method (comi2023zero, ) based on pre-trained BERT model. They first
extract a set of potential intents as candidate classes for the utterance
intent classification problem using a zero-shot approach, which is based on a
fine-tuned BERT model. GPT-3 and Flan-T5-XXL models using prompts are utilized
for intent classification tasks by Parikh et al. (parikh2023exploring, ). They
also use PEFT methods to fine-tune LLMs and demonstrate outstanding
performance in intent classification tasks. To solve the problem that
augmentation via in-context prompting of LLMs alone does not improve
performance, Lin et al. (lin2023selective, ) introduce a novel approach based
on pointwise V-information and successfully improve the performance of intent
detection tasks based on LLMs.

Slot filling task tags each word subsequence with different labels. Therefore,
slot filling tasks can be regarded as sequence classification tasks. Coope et
al. proposed Span-ConveRT (coope2020span, ), a model for dialog slot-filling
tasks, which demonstrate great performance in few-shot scenarios by
integrating conversational knowledge coded in large pre-trained conversational
models. Siddique et al (siddique2021linguistically, ). propose a zero-shot
slot filling model, LEONA, which employs the pre-trained LLMs to provide
contextualized word representations that capture complex syntactic and
semantic features of words based on the context of their usage and uses these
word representations to produce slot-specific predictions for each word.

##### Joint intent detection and slot filling

Some researches combine intent detection and slot filling into a joint intent
detection and slot filling module, which promote two-way information sharing
between intent detection tasks and slot filling tasks. Chen et al.
(chen2019bert, ) fine-tune the LLM based on NLU datasets and their experiment
results indicate that joint NLU module based on fine-tuned LLMs outperforms
both separated NLU module and NLU module based on untuned LLMs. Nguyen et al
(nguyen2023cof, ). propose CoF-CoT approach, which breaks down NLU tasks into
multiple reasoning steps. LLMs can enhance their capability in solving NLU
tasks from different granularities by learning to acquire and leverage
essential concepts.

####  5.1.2. Dialogue State Tracking

As shown in Figure 9, DST and PL constitute the dialogue manager (DM) module,
the central controller of a pipeline-cased TOD system. As the first module of
a DM module, DST involves tracking the current dialogue state by predicting
the slot-value pairs at current turn $t$. In TOD tasks, a dialogue state
$\mathcal{B}_{t}$ records the entire dialogue history until turn t. DST
modules record user’s objectives in the form of slot-value pairs, for example,
in hotel reservation tasks, the dialogue state at turn $t$ is
$\mathcal{B}_{t}={(hotel-bookstay,5),(hotel-bookday,Friday),(hotel-
bookname,Hiltion)}$.

DST methods can be divided into static ontology DST models and dynamic
ontology DST models. Static ontology DST models predict dialogue state from
predefined slot-value pairs and dynamic ontology DST models predict dialogue
state from an unfixed set of slot-values. Many static ontology DST models
(balaraman2019scalable, ; zhong2018global, ; lee2019sumbt, ) have been
proposed. However, most LLM-based DST methods are based on dynamic ontology
DST models, which tracks the dialogue state from unfixed slot-value pairs. For
example, SAVN and MinTL (wang2020slot, ; lin2020mintl, ) focus on creating
methods or frameworks where LLMs can be effectively applied. These methods
achieve competitive results and allow users to plug-and-play pre-trained
sequence-to-sequence models for solving DST tasks. Hu et al (hu2022context, ).
proposed IC-DST, a zero-shot and few-shot DST framework based on ICL. IC-DST
retrieves a few most similar turns from the labeled dialogues as prompts,
which are subsequently fed into the LLMs to produce dialogue state changes of
the current turn. Feng et al. proposed LDST (feng2023towards, ), a DST
framework that leverage LLaMa model. LDST initially create an instruction-
tuning dataset and fine-tune the LLaMa model on this dataset. Subsequently,
LDST guided the LLaMa in generating accurate responses by constructing and
inputting an output prompt.

####  5.1.3. Policy Learning

As the second part of DM module, PL module takes the responsibility for
generating the appropriate next system action based on the dialogue state
$\mathcal{B}_{t}$ at current turn t from the DST module. Therefore, the task
of PL module can be formulated as learn a mapping function:

$f:\mathcal{B}_{t}\to a_{i}\in\mathcal{A},$

where $\mathcal{A}$ is the action set $\mathcal{A}=\left\\{a_{1},\ \dots,\
a_{n}\right\\}$.

The PL module in TOD systems can be approached at two levels: the dialogue act
(DA)-level and the word-level dialogue policy. The goal of DA-level dialogue
policy is to generate dialogue acts such as ‘Inform’: (‘people’, ‘area’),
which are then transferred into readable output in the NLG module.
Reinforcement learning methods (takanobu2020multi, ; wang2020task, ;
gordon2020learning, ) are widely used in DA-level PL tasks. Word-level
dialogue PL module combines PL and NLG module since it conducts a sequence of
actions by selecting a string of words as a readable sentence. In this way,
the word-level dialogue PL task can be regarded as a sequence-to-sequence
generation task. Since LLMs have an outstanding performance in solving
sequence-to-sequence tasks, numerous LLM-based word-level dialogue PL methods
are proposed. Chen et al. (chen2019semantically, ) use the BERT model as a
decoder. Li et al. (li2021retrieve, ) utilize BERT model as a context-aware
retrieval module. Numerous researchers (budzianowski2019hello, ;
hosseini2020simple, ; jang2022gpt, ) fine-tune GPT-2 model and apply the fine-
tuned model to address word-level dialogue PL tasks. Ramachandran et al.
(ramachandran2021causal, ) fine-tune BART and He et al. (he2022galaxy, ) fine-
tune UniLM. Yu et al. (yu2023prompt, ) propose a prompt-based method, which
solves PL tasks by prompting LLMs to act as a policy prior.

####  5.1.4. Natural Language Generation

NLG, which comprises data-to-text generation and text-to-text generation, is
the process of generating natural language text for specific purposes. In
pipeline-based TOD systems, NLG is the last module, which is responsible for
transforming the dialogue actions generated by the PL module into readable
natural language. For example, given the dialogue active: “Inform:
(‘people’)”, NLG module converts it into readable sentence “How many people
are planning to check in?” Conventional NLG modules are based on pipeline
structure, which can be divided into Text Planner module, Sentence Planner
module and Linguistic Planner module (REITER_DALE_1997, ). With the
development of deep learning methods, researchers introduce end-to-end NLG
methods (wen2015stochastic, ; wen2015semantically, ; zhou2016context, ) based
on neural network to solve NLG tasks in recent works. Many recent works are
proposed to solve NLG tasks with LLMs since NLG tasks in pipeline-based TOD
systems are sequence-to-sequence tasks, which can be efficiently addressed by
LLMs. For example, Peng et al. propose SC-GPT (peng2020few, ) model, which is
pre-trained on a large set of annotated NLG corpus and fine-tuned on datasets
with limited domain labels to adapt to new domains. Chen et al. (chen2019few,
) fine-tune other parameters of the PLMs and keep the pre-trained word
embeddings fixed to enhance the model’s generalization ability. Baheti et al.
(baheti2020fluent, ) incorporate a BERT-based classifier into end-to-end NLG
system to identify the best answer from candidate responses. Qian et al.
(qian2022controllable, ) enhance the performance of GPT-2 in addressing NLG
tasks by utilizing prefix tuning method.

###  5.2. End-to-End Methods

Figure 10. Modularly end-to-end task-oriented dialogue system (a) and fully
end-to-end task-oriented dialogue system (b).

As shown in Figure 10, end-to-end TOD systems can be divided into modularly
end-to-end TOD systems and fully end-to-end TOD systems. Although modularly
end-to-end TOD systems generate response through separated modules, which is
similar to pipeline-based TOD systems, modularly end-to-end TOD systems
simultaneously train all the modules and optimize the parameters of all
modules. End-to-end TOD system generates dialogue system response
$\mathcal{S}$ base on the corresponding knowledge base $\mathcal{KB}$ and the
dialogue history
$\mathcal{H}=\left(u_{1},s_{1}\right),\left(u_{2},s_{2}\right),\ldots,\left(u_{n-1},s_{n-1}\right)$,
where $u$ is the user input and $s$ is the system answer:

(21) $\mathcal{S}=\text{ End-to-end TOD }(\mathcal{H},\mathcal{KB}).$

LLMs achieve significant success in ODD tasks. However, researches related to
training fully LLM-based end-to-end TOD models remains relatively limited due
to the lack of large amount of training datasets for TOD tasks. Therefore,
most existing LLM-based end-to-end TOD methods are based on modularly end-to-
end TOD systems.

Simple Task-Oriented Dialogue (SimpleTOD) (hosseini2020simple, ) is an end-to-
end TOD method based on a single, causal language model trained on all sub-
tasks. The methodology SimpleTOD employed in training LLMs, serves as a
successful example of leveraging such models for solving TOD tasks. Given the
concatenation
$x^{t}=[\mathcal{H}_{t},\mathcal{B}_{t},\mathcal{D}_{t},\mathcal{A}_{t},\mathcal{S}_{t}]$,
where $\mathcal{H}_{t}$, $\mathcal{B}_{t}$, $\mathcal{D}_{t}$,
$\mathcal{A}_{t}$ and $\mathcal{S}_{t}$ are the values of dialogue history
$\mathcal{H}$, belief state $\mathcal{B}$, database query result
$\mathcal{D}$, dialogue action $\mathcal{A}$ and system answer $\mathcal{S}$
in turn $t$. The joint probability $p(x)$ and negative log-likelihood
$\mathcal{L}(D)$ over a dataset $D=\left\\{x^{1},\ldots,x^{|D|}\right\\}$ can
be formulated as:

(22) $p(x)=\prod_{i=1}^{n}p\left(x_{i}\mid x_{<i}\right),$ (23)
$\mathcal{L}(D)=-\sum_{t=1}^{|D|}\sum_{i=1}^{n_{t}}\log
p_{\theta}\left(x_{i}^{t}\mid x_{<i}^{t}\right),$

where $n_{t}$ is the length of $x^{t}$ and $\theta$ is the parameters in a
neural network, which is trained to minimize $\mathcal{L}(D)$.

Peng et al. proposed Soloist (peng2021soloist, ), an approach that uses
transfer learning and machine teaching to construct the end-to-end TOD system.
The training process of Soloist is pretty similar to that of SimpleTOD.
However, Soloist refines the data format for each dialogue turn, dialogue
action $\mathcal{A}$ is no longer required. Each dialog turn in the training
data can be represented as
$x=[\mathcal{H},\mathcal{B},\mathcal{D},\mathcal{S}]$. The full pre-training
objective of Soloist is divided into three sub-tasks: belief prediction,
grounded response generation and grounded response generation. Given the
length of belief state sequence $T_{\mathcal{B}}$ and the tokens before turn
$t$ $\mathcal{B}_{<t}$, the objective of predicting the belief state is
defined as:

(24) $\mathcal{L}_{\mathrm{B}}=\log
p(\mathcal{B}\mid\mathcal{H})=\sum_{t=1}^{T_{\mathcal{B}}}\log
p_{\boldsymbol{\theta}}\left(\mathcal{B}_{t}\mid\mathcal{B}_{<t},\mathcal{H}\right),$

where $p(x)$ is the joint probability and $\theta$ is the parameters to be
learned.

Similarly, given the length $T_{\mathcal{S}}$ of delexicalized response
$\mathcal{S}=\left[\mathcal{S}_{1},\cdots,\mathcal{S}_{T_{\mathcal{S}}}\right]$,
the corresponding training objective can be formulated as:

(25) $\displaystyle\mathcal{L}_{\mathrm{R}}$ $\displaystyle=\log
p(\mathcal{S}\mid\mathcal{D},\mathcal{B},\mathcal{H})$
$\displaystyle=\sum_{t=1}^{T_{\mathcal{S}}}\log
p_{\boldsymbol{\theta}}\left(\mathcal{S}_{t}\mid\mathcal{S}_{<t},\mathcal{D},\mathcal{B},\mathcal{H}\right).$

Let $x$ be the positive samples and let $x^{\prime}$ be the negative samples,
Soloist utilizes a binary classifier applied to the features to forecast
whether the items within the sequence correspond ($y$ = 1) or do not
correspond ($y$ = 0).The contrastive object is cross-entropy defined as:

(26)
$\mathcal{L}_{\mathrm{C}}=y\log\left(p_{\boldsymbol{\theta}}(\boldsymbol{x})\right)+(1-y)\log\left(1-p_{\boldsymbol{\theta}}\left(\boldsymbol{x}^{\prime}\right)\right).$

Then, the fully training objective can be formulated as:

(27)
$\mathcal{L}_{\boldsymbol{\theta}}(D)=\sum_{t=1}^{|D|}\left(\mathcal{L}_{\mathrm{B}}\left(\boldsymbol{x}_{t}\right)+\mathcal{L}_{\mathrm{R}}\left(\boldsymbol{x}_{t}\right)+\mathcal{L}_{\mathrm{C}}\left(\boldsymbol{x}_{t}\right)\right).$

For the UBAR (yang2021ubar, ), previous modularly end-to-end TOD methods that
are trained and evaluated in turn-level sequences where they are based on
dialog
history$\mathcal{H}_{t}=\left(u_{1},s_{1}\right),\left(u_{2},s_{2}\right),\ldots\left(u_{t-1},s_{t-1}\right),\left(u_{t}\right)$
to generate response in turn t. While UBAR integrates the intermediate
information $\mathcal{B}$, $\mathcal{D}$, and $\mathcal{A}$ within the
context. Therefore, the training sequence of UBAR in turn $t$ is de fined as
$[\mathcal{H}_{0},\mathcal{B}_{0},\mathcal{D}_{0},\mathcal{A}_{0},\mathcal{S}_{0},\dots\mathcal{H}_{t},\mathcal{B}_{t},\mathcal{D}_{t},\mathcal{A}_{t},\mathcal{S}_{t}]$,
which is then used to fine-tune the large pre-trained model GPT-2.

Su et al. propose a plug-and-play model for task-oriented dialogue (PPTOD)
(su2021multi, ), which is a modularly end-to-end TOD model. PPTOD is pre-
trained with four TOD-related tasks and prompts are used to enhance the
performance of language model. It’s worth mentioning that the learning
framework of PPTOD allows it to be trained with partially annotated data,
which significantly reduce the cost of manually creating datasets.

Semi-supervised Pre-trAined Conversation ModEl (SPACE) comprises a serious of
PLMs (he2022galaxy, ; he2022space, ; he2022unified, ) proposed by
Conversational AI Team, Alibaba DAMO Academy. GALAXY (SPACE-1) (he2022galaxy,
) is a modularly end-to-end TOD model that explicitly acquires dialog policy
from a combination of limited labeled dialogues and extensive unlabeled
dialogue corpora through the application of semi-supervised learning. Previous
works predominantly focused on enhancing the performance of NLU and NLG
modules, while GALAXY optimizes the performance of PL module by introducing a
new dialogue action prediction task during pre-training. These methodologies
enhance the performance of GALAXY in solving TOD tasks and empower GALAXY with
superior few-shot capabilities compared to other models.

SPACE-2 (he2022space, ) is a tree-structured conversation model pre-trained on
limited labeled dialogs and large-scale unlabeled dialog corpora. In
conventional methods, positive samples are exclusively defined as examples
with identical annotations, while all other instances are categorized as
negative samples. This classification overlooks the possibility that diverse
examples may exhibit shared semantic similarities to some extent. Therefore,
the SPACE-2 framework establishes tree structures, which is called semantic
tree structure (STS), for diverse datasets based on their respective data
structures. Then, SPACE-2 measures the similarity among different labeled
dialogues and aggregate the output multiple scores. In this approach, all
annotated data is considered as positive instances with soft scores, as
opposed to the binary scores (0 or 1) commonly employed in previous methods.

SPACE-3 (he2022unified, ) is one of the most state-of-art pre-trained
modularly end-to-end TOD models. The SPACE-3 framework consolidates the
efforts of SPACE-1 and SPACE-2, incorporating STS to unify the inconsistent
annotation schema across different datasets and devising a dedicated pre-
training objective for each component. SPACE-3 uses
$p^{u}=\left\\{p_{1}^{u},p_{2}^{u},\ldots,p_{A}^{u}\right\\}$ and
$p^{o}=\left\\{p_{1}^{o},p_{2}^{o},\ldots,p_{B}^{o}\right\\}$ to represent the
dialog understanding prompt sequence and policy prompt sequence, where $A$ and
$B$ are the length of prompt sequences. Then, $p^{u}$ and $p^{o}$ are
leveraged to extract semantics and help pass the task-flow in a TOD system.

##  6\. LLM Based Open-Domain Dialogue Systems

ODD systems are designed to converse on a wide range of topics without a
specific task or goal. While TOD are focused on achieving specific tasks, ODD
aim to provide coherent and contextually relevant responses across any topic
presented by the user. ODD is primarily categorized into three methods:
Retrieval-based Methods, which select responses from a predefined set;
Generation-based Methods, which generate responses dynamically; and Hybrid
Methods, which combine both retrieval and generation to optimize dialogue
outcomes. Table 2. shows the latest advancements in these three approaches
within the domain of ODD systems.

Table 2. Recent Advances in Open-Domain Dialogue Systems.

Task | Methods | Description
---|---|---
Retrieval-based Methods | Dense Retriever (karpukhin2020dense, ) | Dense vector representations for improved accuracy
MSN (yuan2019multi, ) | Context management via multi-hop mechanism
IoI Network (tao2019one, ) | Multi-turn response selection enhancement
Generation-based Methods | PLATO-LTM (xu2022long, ) | Persona coherence with long-term memory
PAML (madotto2019personalizing, ) | Personalization via meta-learning
Persona-Consistent Generation (chen2023learning, ) | Coherence with latent variables for consistency
PHMN (li2021dialogue, ) | Personalized matching with user history
DHAP (ma2021one, ) | Dynamic user profile learning for personalization
MSP Model (zhong2022less, ) | Dialogue history refinement for personalization
GDR Framework (song2020generate, ) | Persona-consistent dialogue generation
CLV Model (tang2023enhancing, ) | Dual persona data utilization for personalized responses
Hybrid Methods | Retro (borgeaud2022improving, ) | Retrieval-augmented auto-regressive LM
FiD (izacard2020leveraging, ) | Passage retrieval and decoding fusion
K2R (adolphs2021reason, ) | Knowledge-first approach for factual accuracy
EMDR2 (singh2021end, ) | T5 integration with Top-k MIPS retrieval
Latent Retrieval (lee2019latent, ) | MIPS for efficient evidence retrieval
IAG (komeili2021internet, ) | Real-time Internet search integration

###  6.1. Retrieval-based Methods

In ODD systems, early foundational works have set the stage for subsequent
advancements. Bordes et al. (bordes2016learning, ) were shifting the focus
towards end-to-end learning approaches in goal-oriented dialog systems,
challenging the traditional reliance on domain-specific handcrafting.
Furthering this paradigm, Tao et al. (tao2019one, ) introduced a nuanced
approach to interaction depth in multi-turn response selection, demonstrating
that a more profound interaction could significantly enhance context-response
matching. Concurrently, Henderson et al.(henderson2017efficient, ) showcased
the practical application of these concepts in a large-scale commercial
environment. These seminal contributions have laid the groundwork for a new
wave of innovations in retrieval-based ODD systems.

Karpukhin et al.(karpukhin2020dense, ) introduce an approach in the field of
information retrieval, centering around the dense retriever model. This model
represents a departure from traditional retrieval methods like Lucene-BM25,
primarily through its use of dense vector representations as opposed to
conventional sparse vector models such as TF-IDF and BM25. These dense
representations are obtained from embeddings learned from a carefully curated
set of questions and passages, leading to improvements in top-20 passage
retrieval accuracy. Additionally, the model utilizes a dual-encoder framework
based on BERT, with one encoder processing questions and the other focusing on
passages, each mapping its input into a low-dimensional vector space for
efficient retrieval.The training process of the model focuses on the
optimization of embeddings to effectively align question and passage vectors.
The objective function for this optimization is defined as follows:

(28)
$L(q_{i},p_{i}^{+},p_{i,1}^{-},\ldots,p_{i,n}^{-})=-\log\frac{e^{\text{sim}(q_{i},p_{i}^{+})}}{e^{\text{sim}(q_{i},p_{i}^{+})}+\sum_{j=1}^{n}e^{\text{sim}(q_{i},p_{i,j}^{-})}},$

where $q_{i}$ is the question vector, $p_{i}^{+}$ the positively aligned
passage vector, and $p_{i,j}^{-}$ the negative passage vectors. The similarity
function $\text{sim}(q,p)$ computes the dot product of question and passage
embeddings.

With the groundwork laid by dense retrieval models, subsequent efforts focused
on refining the interaction depth within dialogue systems. Tao et
al.(tao2019one, ) introduce the Interaction-over-Interaction (IoI) Network.
IoI enhances multi-turn response selection in retrieval-based chatbots. This
model deepens the interaction between context and response by leveraging
multiple rounds of interaction. It introduces multiple layers of interaction
between the utterance and the response, allowing the network to capture
complex semantic relationships more effectively. The IoI model’s interaction
blocks process the utterance-response pairs sequentially, with self-attention
mechanisms and iterative refinement enhancing the depth of interaction.
Additionally, the model incorporates a mechanism for aggregating matching
signals across various interaction layers.

###  6.2. Generation-based Methods

Generation-based methods in ODD systems have evolved significantly, offering
flexibility and creativity in response synthesis. Early works like Vinyals and
Le (vinyals2015neural, ) and Sutskever et al. (sutskever2014sequence, )
pioneered the use of sequence-to-sequence models for coherent dialogue
generation. Further advancements by Shang et al. (shang2015neural, )
introduced attention mechanisms, improving response relevance and quality,
while Serban et al. (serban2016building, ) developed the Hierarchical
Recurrent Encoder-Decoder for more complex dialogues. Recent progress, marked
by Radford et al. (radford2019language, ), has seen the integration of large-
scale transformer models like GPT-2, pushing the boundaries of fluency and
contextual awareness in generated responses.

####  6.2.1. Knowledge-Enhanced Generation

Building on these foundational advancements in sequence-to-sequence models and
attention mechanisms, Zhao et al. (zhao2020knowledge, ) propose an enhancement
for dialogue generation through the integration of external knowledge with
PLMs such as GPT-2. Their approach involves a BERT-based Knowledge Selection
Module for selecting relevant documents $D^{\prime}$ from a set $D$ based on
the dialogue context $U$. The selected knowledge is utilized in a GPT-2-based
Response Generation Model for generating responses:

(29)
$P(r|U,D^{\prime};\theta)=\prod_{t=1}^{lr}P(r_{t}|g(U,D^{\prime}),r_{1:t-1};\theta),$

where $g(U,D^{\prime})$ represents the integrated user dialogue context and
selected knowledge, and $\theta$ denotes the GPT-2 parameters. This method
allows the generation of responses that are contextually relevant and informed
by external knowledge.

Moreover, while Zhao et al. focused on enhancing dialogue generation with
external knowledge, Xu et al. (xu2022long, ) introduce the PLATO-LTM model,
featuring a Long-Term Memory (LTM) mechanism to consistently maintain persona
information. The model utilizes a Persona Extractor (PE) to classify persona
labels based on user input $U_{i}$ using an ERNIE-CNN architecture. The LTM
module in PLATO-LTM is responsible for matching the context with the relevant
persona, enabling the system to retrieve appropriate persona-specific
information. The Generation Module is represented as:

(30) $L_{\text{NLL}}=-\mathbb{E}\left[\sum_{t=1}^{T}\log
p(r_{t}|c,\rho_{u},\rho_{s},r_{<t})\right].$

This equation represents the negative log-likelihood loss function, where
$r_{t}$ is the generated response at time $t$, $c$ denotes the current
context, $\rho_{u}$ and $\rho_{s}$ are user and system persona embeddings,
respectively, and $r_{<t}$ symbolizes the responses generated up to the
previous time step. PLATO-LTM’s design focuses on enhancing the coherence and
engagement of conversations by dynamically managing persona information, thus
facilitating more natural and contextually relevant dialogue generation.

Further expanding on the theme of personalization, Song et al.
(song2020generate, ) develop the Generate-Delete-Rewrite (GDR) framework to
create persona-consistent dialogues. The GDR framework operates in three
stages: initially generating a response prototype, then identifying and
masking elements that are inconsistent with the established persona, and
finally refining the output. The process starts with the creation of an
initial response vector, followed by calculating attention weights to identify
inconsistencies, and concludes with a refinement stage where the initial
output is adjusted to ensure persona consistency. Building on the idea of
tailored dialogue generation, Tang et al. (tang2023enhancing, ) introduce the
Contrastive Latent Variable (CLV)-based model. This model enhances dialogue
personalization by using both sparse and dense persona resources. It begins by
encoding persona information and user queries, and then generates personalized
responses by combining these encoded inputs. The CLV model distinguishes
itself by clustering dense persona descriptions into sparse categories,
effectively utilizing varied persona data to inform and personalize dialogue
responses. The CLV model is not only consistent with user personas but also
enriched by a deeper understanding of individual user characteristics.

####  6.2.2. Personalization and Consistency

Transitioning from the focus on dialogue personalization and consistency,
Madotto et al. (madotto2019personalizing, ) use Model-Agnostic Meta-Learning
(MAML) for personalizing dialogue learning. Their Persona-Agnostic Meta-
Learning (PAML) framework treats different personas as separate tasks in meta-
learning. PAML adapts dialogue models to new personas through dialogue
samples, contrasting traditional persona-specific descriptions. This method is
effective in generating fluent and consistent dialogues.

As the field advances towards more nuanced methods of personalization, Li et
al. (li2021dialogue, ) have developed a Personalized Hybrid Matching Network
(PHMN) that incorporates user-specific dialogue history into response
selection. The PHMN model operates on two principal fronts: firstly, it
extracts personalized wording behaviors from the user’s dialogue history.
Secondly, it employs hybrid representation learning on context-response
utterances, integrating a customized attention mechanism to extract essential
information from context-response interactions. It improves the accuracy of
matching responses to the user’s conversational style and preferences.

Building upon the concept of utilizing dialogue history for personalized
response generation, Zhong et al. (zhong2022less, ) introduce the MSP model to
refine user dialogue history. The MSP model includes User Refiner, Topic
Refiner, Token Refiner, and Response Generator. The dialogue generation
process integrates these components:

(31) $\hat{y}=\text{TRMdec}(x,u_{\text{sim}},t,A),$

where $\hat{y}$ is the response, $x$ the dialogue input, $u_{\text{sim}}$ the
user similarity output, $t$ the topic information, and $A$ the token-level
attention. This model enhances personalization in response generation by
refining user dialogue history.

###  6.3. Hybrid Methods

Hybrid methods in ODD systems represent the integration of retrieval-based and
generation-based approaches, combining the strengths of both. Early research
in this area includes the work of Sordoni et al. (sordoni2015neural, ), who
explored the use of context-sensitive generation models conditioned on
traditional retrieval methods. This approach laid the groundwork for
subsequent developments in the field. Another contribution by Yan et al.
(yan2016docchat, ), introduced a model that dynamically selects between
generating a response and retrieving one, depending on the conversation’s
context. This blend allows for more flexible and contextually appropriate
responses.

####  6.3.1. Integrating Retrieval and Generation

Building upon these initial explorations into hybrid methodologies, Borgeaud
et al. (borgeaud2022improving, ) introduce the Retrieval-Enhanced Transformer
(Retro) model, which combines a large-scale retrieval mechanism with an auto-
regressive language model. This integration allows the language model to
access contextually relevant information from a retrieval database. The model
retrieval-enhanced sequence log-likelihood defined as:

(32)
$\mathcal{L}(\mathbf{X}|\theta,\mathcal{D})=\sum_{u=1}^{l}\sum_{i=1}^{m}\log
p_{\theta}(x^{(u-1)m+i}|\mathbf{x}{<(u-1)m+i},\text{Ret}\mathcal{D}(\mathcal{C}^{u_{0}}{<u})),$

where $\mathbf{X}$ represents the input sequence, $\theta$ denotes the model
parameters, $\mathcal{D}$ is the retrieval database, and
$\text{Ret}\mathcal{D}$ refers to the retrieval operation.

####  6.3.2. Enhancing Dialogue with External Knowledge

Adolphs et al. ((adolphs2021reason, )) developed the Knowledge to Response
(K2R) model, focusing on factual accuracy. K2R first generates knowledge
sequences relevant to the dialogue context and then integrates this knowledge
to synthesize the final response. Izacard and Grave ((izacard2020leveraging,
)) developed the Fusion-in-Decoder (FiD) model, integrating generative models
with passage retrieval. FiD encodes a question and multiple retrieved passages
independently, concatenating each passage with the question for decoding,
which facilitates the synthesis of information from various passages.

Expanding on the concept of retrieval augmentation, Xu et al. ((xu2021beyond,
)) focus on the dynamics of long-term conversation. They explore Retrieval-
Augmented Generative Models, characterized by their approach to integrating
conversation context and model mechanics:

(33) $p(y|x)=\sum_{z\in\text{Retrieve}(x)}p(z|x)\cdot p(y|x,z).$

Additionally, they explored Memory-Based Models with Summarization, where the
model summarizes past dialogues for response generation:

(34) $P(y|x,S)=\text{SummaryGeneration}(x,\text{Past Dialogues}).$

Singh et al. (singh2021end, ) develop $\mathrm{EMDR}^{2}$, integrating T5
encoding and decoding with a Top-k MIPS retrieval mechanism, enhanced by a
scale evidence document encoder. The system aims to optimize multi-document
reading and retrieval. $\mathrm{EMDR}^{2}$ implements an Expectation-
Maximization (EM) algorithm for latent variable model training. The training
process involves computing the posterior of the latent variable $Z$, and the
training objective is given by:

(35) $\displaystyle L=\log
p(a|q,Z_{\text{top-K}};\Theta)+\log\sum_{k=1}^{K}\text{SG}(p(a|q,z_{k};\Theta))p(z_{k}|q,Z_{\text{top-K}};\lambda),$

where SG is the stop-gradient operator, $\Theta$ are the reader’s parameters,
and $\lambda$ are the retriever’s parameters.

Komeili et al. (komeili2021internet, ) introduce an ”Internet-Augmented
Dialogue Generation” system, integrating real-time Internet. It consists of a
search query generator and a response generation module. The search query
generator, a transformer-based encoder-decoder model, generates internet
search queries from the dialogue context. The response module uses the
retrieved information to construct responses. The system’s function is
encapsulated in:

(36) $R=\text{FiD}\left(\text{Encoder-
Decoder}(C),\,\text{InternetSearch}(\text{Encoder-Decoder}(C))\right),$

where $C$ is the dialogue context and $R$ the generated response. The Encoder-
Decoder function processes $C$, and InternetSearch retrieves information based
on the query. The FiD method integrates this information to generate $R$.

##  7\. EVALUATION APPROACHES

The effective evaluation approaches for models of kinds of tasks have always
been the focus of attention in the research field. In this section we
introduce the automatic evaluation and human evaluation approaches that have
been widely used.

###  7.1. Automatic Evaluation

####  7.1.1. Automatic Evaluation Approaches for Task-oriented Dialogue
Systems

The evaluation for TOD systems is mainly performed using automatic approaches,
namely joint goal accuracy, slot accuracy, average goal accuracy, requested
slot F1, BLEU, and Entity F1. In the following, a brief description of each
approach is provided.

##### Joint Goal Accuracy

Joint goal accuracy (JGA), developed from Henderson et al. (henderson2014word,
) and Zhong et al. (zhong2018global, ) , is the most widely used evaluation
approach for DST. The joint goal is a set of accumulated turn level goals up
to a given turn in the dialogue. It compares the predicted dialogue states to
the ground truth which includes slot values for all possible pairs. The output
is considered as a correct prediction if and only if all the predicted values
match its ground truth values at each turn. JGA can be expressed as:

(37) $JGA=\begin{cases}1&\text{if predicted state = gold state},\\\
0&\text{otherwise}.\end{cases}$

##### Slot Accuracy

Slot accuracy (SA) (wu2019transferable, ) is also a widely used automatic
evaluation approach. Unlike joint goal accuracy, it only compares each value
to the corresponding ground truth individually without seeing other turns. SA
can be expressed as:

(38) $SA=\frac{T-M-W}{T},$

where $T$ indicates the total number of predefined slots for all the domains,
$M$ denotes the number of missed slots that the model does not accurately
predict among the slots included in the gold state, and $W$ represents the
number of wrongly predicted slots among the slots that do not exist in the
gold state.

##### Average Goal Accuracy

Average goal accuracy (AGA) (rastogi2020towards, ) is the average accuracy of
predicting the correct value for an active slot in each turn. A slot becomes
active if its value is mentioned in the current turn and is not inherited from
previous turns. AGA can be expressed as:

(39) $AGA=\frac{|N_{t}\cap B_{t}^{\prime}|}{|N_{t}|},$

where $B_{t}$ and $B_{t}^{\prime}$ are the set of ground-truth and predicted
belief state for turn $t$ respectively. Then let $N_{t}\subseteq B_{t}$ be the
set of ground-truth triplets having non-empty slot values.

##### Requested Slot F1

Requested slot F1 indicates the model performance in correctly predicting if a
requested slot is requested by the user, estimated as the macro-averaged F1
score over for all requested slot. The macro-averaged of F1 score is computed
over the individual slot-type and slot-value for every turn. To define the
macro-averaged F1 score($ma\,F_{1}$), first consider the following
precision($P_{i}$) and recall ($R_{i}$) within each class, $i\,=\,1,\ \dots,\
2$:

(40) $\displaystyle P_{i}$
$\displaystyle=\frac{TP_{i}}{(TP_{i}+FP_{i})}=\frac{p_{ii}}{p_{i-}},$ (41)
$\displaystyle R_{i}$
$\displaystyle=\frac{TP_{i}}{(TP_{i}+FN_{i})}=\frac{p_{ii}}{p_{-i}},$

and F1 score within each class ($F_{1i}$) is defined as the harmonic mean of
$P_{i}$ and $R_{i}$, that is:

(42) $F_{1i}=2\frac{P_{i}\times
R_{i}}{P_{i}+R_{i}}=2\frac{p_{ii}}{p_{i-}+p_{-i}}.$

The macro-average F1 score is defined as the simple arithmetic mean of
$F_{1i}$:

(43)
$ma\,F_{1}=\frac{1}{r}\sum_{i=1}^{r}F_{1i}=\frac{2}{r}\sum_{i=1}^{r}\frac{p_{ii}}{p_{i-}+p_{-i}}.$

##### BLEU

BLEU (papineni2002bleu, ) is used to calculate the co-occurrence frequency of
two sentences based on the weighted average of matched n-gram phrases. BLEU
was originally used to evaluate machine translation, and has been used for
evaluating TOD and ODD systems.

##### Entity F1

Entity F1 is used to evaluated the model’s ability to generate relevant
entities from the underlying knowledge base and to capture the semantics of
the user-initiated dialogue flow. To compute an entity F1, one need to micro-
average over the entire set of system dialogue responses and use the entities
in their canonicalized forms.

####  7.1.2. Automatic Evaluation Approaches for Open-domain Dialogue Systems

The evaluation for ODD system is mainly performed using automated approaches,
namely perplexity, BLEU, DIST-n, and recall@K. In the following, a brief
description of each approach is provided.

##### Perplexity

Perplexity (vinyals2015neural, ) was originally conceived as an information-
theoretic measure to assess how much a given language model is suited to
predict a text sequence or, equivalently, how much a word sequence fits into a
specific language model. It is now used as an analytic approach with potential
application to support early diagnosis of symptoms of mental disorder.
Perplexity can be expressed as:

(44) $PP(W)=P(w_{1}w_{2}...w_{N})^{-1/N}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}\dots
w_{N})}},$

where $W$ is a word sequence of length $N$, $P(w_{1}w_{2}\dots w_{N})$ is the
probability of that word sequence.

##### DIST-n

DIST-n (li2015diversity, ) is used to measure the diversity of response
sequence for dialogue generation by calculating the number of distinct
unigrams and bigrams in generated responses. The value is scaled by total
number of generated tokens to avoid favoring long sentences. DIST-1 and DIST-2
are respectively the number of the distinct unigrams and bigrams divided by
total number of generated words.

##### Recall@K

Recall@K is one of the standard approaches to evaluate. For query $q$, it is
defined as a radio of the number of relevant(positive) examples within the
top-k ranked examples to the total number of relevant examples for $q$ given
by $|P_{q}|$. It is denoted by $R^{k}_{\Omega}(q)$ when compute for query $q$
and database $\Omega$ and function $H(.)$ is the Heaviside step function.
Therefore, it can be expressed as:

(45) $R^{k}_{\Omega}(q)=\frac{H(k-1-\sum_{z\in\Omega,z\neq
x}H(S_{qz}-S_{qx}))}{|P_{q}|}.$

###  7.2. Human Evaluation

Human evaluation is also used as an evaluation approach in different tasks.
Human evaluation focuses on the explanation of two matters: diversity and
creativity, i.e., the capacity of varying their texts in form and emphasis to
fit an enormous range of speaking situations, and the potential to express any
object or relation as a natural language text. Furthermore, human evaluation
scrutinizes three key aspects: Grammar (whether a generated sentence is
grammatically correct and fluent), Faithful (whether the output accurately
reflects the input), and Coherence (ensuring that a sentence is logically
consistent and follows the natural flow of human writing).

##  8\. DATASETS

In this section we introduce the datasets that have been widely used in TOD
and ODD systems in recent years. Table 3. and Table 4. show some information
for TOD and ODD datasets respectively.

###  8.1. Datasets for Task-oriented Dialogue Systems

##### MultiWOZ

MultiWOZ (budzianowski2018multiwoz, ) is a fully-labeled collection of human-
human written conversations, containing 10,438 dialogues. The dataset is
collected using Wizard-of-Oz and contains dialogues in 7 domains and the
dialogues cover between 1 and 5 domains per dialogue thus greatly varying in
length and complexity. MultiWOZ has undergone various versions, with several
error corrections. MultiWOZ 2.1 (eric2019multiwoz, ) provides 2-3 descriptions
for each slot in the dataset. MultiWOZ 2.2 (zang2020multiwoz, ) further
provides descriptions of domain and slot as well as possible values for
categorical slot. MultiWOZ 2.3 (han2021multiwoz, ) differentiates incorrect
annotation in dialogue acts from dialogue states, identifying a lack of co-
reference. MultiWOZ 2.4 (ye2021multiwoz, ) is the latest version and fixes the
incorrect and inconsistent annotations.

##### RiSAWOZ

RiSAWOZ (quan2020risawoz, ) is a large-scale multi-domain Chinese Wizard-of-Oz
dataset with rich semantic annotations. It contains 11.2 thousand human-to-
human multi-turn semantically annotated dialogues, with more than 150 thousand
utterances spanning over 12 domains. Each dialogue is labeled with
comprehensive dialogue annotations, including dialogue goal, domain, dialogue
states and acts at both the users and system side.

##### CrossWOZ

CrossWOZ (zhu2020crosswoz, ) is a large-scale Chinese cross-domain Wizard-of-
Oz task-oriented dataset. It contains 6 thousand dialogue sessions and 102
thousand utterances for 5 domains. It contains rich annotation of dialogue
states and acts at both the users and system side and about 60% of the
dialogues have cross-domain user goals.

##### PersuasionForGood

PersuasionForGood (P4G) (wang2019persuasion, ) is a dataset with 1,017
dialogues and annotated emerging persuasion strategies from a subset. The
dataset was collected using an online persuasion task where one participant
was asked to persuade the other to donate to a specific charity. It is a rich
human-human persuasion dialogue dataset with comprehensive user psychological
study and persuasion strategy annotation.

##### WOZ 2.0

WOZ 2.0 (mrkvsic2016neural, ) is a dataset updated from the CamRest
(wen2016network, ) dataset which has 676 dialogues. The dataset is collected
using Wizard-of-Oz and contains 1,200 dialogues. Each turn in a dialogue was
contributed by different users, who had to review all previous turns in that
dialogue.

##### Stanford Multi-Domain

Stanford Multi-Domain (SMD) (eric2017key, ) is a Wizard-of-Oz dataset. It
contains 3,031 dialogues in 3 distinct domains. The dialogues are grounded
through underlying knowledge bases and a knowledge snippet is attached with
each dialogue as a piece of simplified database information.

Table 3. Overview of datasets for task-oriented dialogue. *SMD only provides
statistics of avg of utterances per dialogue and avg of tokens per utterance.
Single domain and multi domains show whether the dataset has dialogues that
contain single domain or multi domains respectively.

Dataset | Dialogues | Avg. turns / dial. | Avg. tokens / turn | Domains | Single Domain | Multi Domains
---|---|---|---|---|---|---
MultiWOZ | 10,438 | 13.70 | 13.18 | 7 | ✓ | ✓
RiSAWOZ | 11,200 | 13.57 | 10.91 | 12 | ✓ | ✓
CrossWOZ | 6,012 | 16.90 | 16.25 | 5 | ✓ | ✓
P4G | 1,017 | 10.43 | - | 1 | ✓ | ✗
WOZ 2.0 | 1,200 | 7.35 | 11.27 | 1 | ✓ | ✗
SMD | 3,031 | 5.29* | 9* | 3 | ✓ | ✗

###  8.2. Datasets for Open-domain Dialogue Systems

##### PersonaChat

PersonaChat (zhang2018personalizing, ) consists of chats and persons which are
collections of five or more sentences that describe a personality. The dataset
consists crowd-source dialogues where each participant plays the part of an
assigned persona; and each person has a word distinct paraphrase. It paired
human generated profiles and conversations aiding the construction of agents
that have consistent personalities and viewpoints.

##### MMdialog

MMdialgo (feng2022mmdialog, ) is a large-scale multi-turn dialogue dataset
towards multi-model open domain conversations. It is composed of a curated set
of 1.08 million real-world dialogues with 1.53 million unique images across
4148 topics. It contains massive topics to generalize the open domain and it
is the largest multi-model conversation dataset by the number of dialogues by
88x.

##### Dailydialog

Dailydialog (li2017dailydialog, ) is a multi-turn dialogue dataset with 13118
dialogues. It is created by scarping text from conversations held on an
English learning website. The dialogues cover 10 topics and conform common
dialog flows. Besides the dataset contains unique multi-turn dialog flow
patterns, which reflect our realistic communication way. Each utterance is
labeled with a dialogue act and an emotion.

##### Pchatbot

Pchatbot (qian2021pchatbot, ) is a large-scale dialogue dataset that contains
two subsets collected from Weibo and Judicial forums respectively. It is
composed of almost 200 million dialogue pairs. The dataset is elaborately
normalized via process such as anonymization, deduplication, segmentation, and
filtering. It provides anonymized user IDs and timestamps for both posts and
responses. This enables the development of models to directly learn implicit
user personality from user’s dialogue history.

##### PersonalDialogue

PersonalDialogue (zheng2019personalized, ) is a large-scale multi-turn
dialogue dataset containing various traits from a large number of people. The
dataset consists of 20.83 million sessions and 56.25 million utterances from
8.47 million speakers. Each utterance is connected with a speaker who is
marked with traits like age, gender, location, interest tags, etc. The dataset
facilitates the study of personalized dialogue generation.

##### Douban

Douban (wu2016sequential, ) is the first human-labeled dataset for multi-turn
response selection. It crawled 1.1 million dyadic dialogues longer than 2
turns from Douban group and randomly sampled 0.5 million dialogues for
creating a training set, 25 thousand dialogues for creating a validation set,
and 1,000 dialogues for creating a test set. Conversations in this dataset
come from the open domain, and response candidates in this dataset are
collected from a retrieval engine.

Table 4. Overview of datasets for open-domain dialogue. Human–Human denotes
datasets where two people converse with each other. Scraped marks datasets
which are gathered from an existing online resource.

Dataset | Dialogues | Method | Source | Language
---|---|---|---|---
PersonaChat | 164,356 | Human-Human | Crowdsourcing | en
MMdialog | 1,079,117 | Scraped | Social Media | en
Dailydialog | 13,118 | Scraped | - | en
Pchatbot | 198,875,796 | Scraped | Weibo, Judicial | zh
PersonalDialogue |  $\approx$20.83 million | Scaped | Weibo | zh
Douban | 526,000 | Scaped | Douban | zh

##  9\. DISCUSSION

More and more researchers are investigating on applying LLMs to different
components of a multi-turn TOD system or a muti-turn ODD system. An
influential factor driving the popularity of multi-turn dialogue system tasks
is the growing demand for chatbots in both industrial settings and everyday
activities. Industry representatives such as OpenAI’s ChatGPT, Anthropic’s
Claude 2, Meta’s Meta AI, and Google’s Gemini Ultra have greatly improved the
convenience of people’s lives. Another reason is that a significant portion of
natural language data exists in the form of dialogues, and multi-turn dialogue
systems are more aligned with real-world scenarios, thus fostering the
development of large model-based multi-turn dialogue systems. Meanwhile, LLMs
provide humans with a powerful toolkit.

This section presents a discussion on challenges faced by LLM based multi-turn
dialogue systems that deserved to be tackled and investigated future.

Deep understanding and Large-scale open-retrieval. By employing LLMs in multi-
turn conversations, it requires a more proficient understanding and retention
of longer contextual information to generate responses that are more coherent
and relevant. In contrast to open-domain QA tasks, open-retrieval dialogues
pose new challenges primarily arising from the features of human-machine
interaction, impacting both efficiency and effectiveness.

Emotionalization and Personalization. Emotionalizing dialogue systems and
enhancing logical consistency can make responses more aligned with the needs
of the queries, understand the emotions behind expressions, thus better
understanding the semantic and contextual connections, and providing more
appropriate answers.

Multi-task dialogue systems. Recent research in end-to-end TOD systems and
knowledge-grounded open-domain systems has opened up the prospect of merging
these two distinct paradigms within a cohesive framework, or perhaps even a
unified model. These hybrid dialogue systems are designed to operate
concurrently as both assistants, efficiently executing specific tasks, and as
chatbots, engaging in conversational interactions.

Multi-model dialogue systems. The world we inhabit is inherently multi-modal,
with humans utilizing a multitude of senses such as sight, hearing, smell,
taste, and touch to perceive it. Hence, it is imperative that chatbots possess
the proficiency to blend information from disparate modalities. While
contemporary large chat models exhibit admirable effectiveness in processing
text, audio, and pictures, they still encounter significant limitations and
challenges when it comes to video processing.

Bias identification and Privacy protection. LLMs have the potential to
generate content that is harmful, offensive, or biased due to their training
on publicly available online datasets. While researchers have endeavored to
address this through fine-tuning, challenges may persist, particularly in
languages other than English where publicly available datasets are limited. It
is important to prioritize user privacy alongside the advancement of more
sophisticated dialogue systems.

##  10\. CONCLUSION

In recent years, the rapid advancement of LLMs has propelled multi-turn
dialogue tasks to the forefront of natural language processing research. This
paper delves into the study of LLM-based multi-turn dialogue systems. It
begins by categorizing common LLMs based on their model structures and
introduces methods for adapting LLMs to various subtasks, including fine-
tuning and prompt engineering. Subsequently, it discusses two main categories
of LLM-based multi-turn dialogue systems: LLM-Based TOD systems and LLM-Based
ODD systems. Following this, the paper outlines evaluation metrics derived
from the outputs of multi-turn dialogue systems, which aid in assessing and
understanding the conversational abilities of LLMs. Additionally, it
highlights datasets that have been widely used in TOD and OOD systems in
recent years. Finally, the paper suggests some open problems to indicate the
major challenges and future research directions for LLM-based multi-turn
dialogue systems.

## References

  * (1) Joseph Weizenbaum.  Eliza—a computer program for the study of natural language communication between man and machine.  Communications of the ACM, 9(1):36–45, 1966.
  * (2) Kenneth Mark Colby, Sylvia Weber, and Franklin Dennis Hilf.  Artificial paranoia.  Artificial intelligence, 2(1):1–25, 1971.
  * (3) David Goddeau et al.  A form-based dialogue manager for spoken language applications.  In Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP’96, volume 2, pages 701–704. IEEE, 1996.
  * (4) Yu Wu et al.  Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 496–505, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * (5) Tiancheng Zhao and Maxine Eskenazi.  Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning.  In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 1–10, Los Angeles, September 2016. Association for Computational Linguistics.
  * (6) Wentao Ma et al.  TripleNet: Triple attention network for multi-turn response selection in retrieval-based chatbots.  In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 737–746, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * (7) Iulian Serban et al.  Building end-to-end dialogue systems using generative hierarchical neural network models.  In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  * (8) Wanwei He et al.  Amalgamating knowledge from two teachers for task-oriented dialogue system with adversarial training.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3498–3507, 2020.
  * (9) Lisong Qiu et al.  Are training samples correlated? learning to generate dialogue responses with multiple references.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3826–3835, 2019.
  * (10) Suket Arora, Kamaljeet Batra, and Sarabjit Singh.  Dialogue system: A brief review.  arXiv preprint arXiv:1306.4134, 2013.
  * (11) Hongshen Chen et al.  A survey on dialogue systems: Recent advances and new frontiers.  Acm Sigkdd Explorations Newsletter, 19(2):25–35, 2017.
  * (12) Jinjie Ni et al.  Recent advances in deep learning based dialogue systems: A systematic survey.  Artificial intelligence review, 56(4):3055–3155, 2023.
  * (13) Libo et al. Qin.  End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5925–5941, Singapore, December 2023. Association for Computational Linguistics.
  * (14) Jared Kaplan et al.  Scaling laws for neural language models.  arXiv preprint arXiv:2001.08361, 2020.
  * (15) Jason Wei et al.  Emergent abilities of large language models.  arXiv preprint arXiv:2206.07682, 2022.
  * (16) Xipeng Qiu et al.  Pre-trained models for natural language processing: A survey.  Science China Technological Sciences, 63(10):1872–1897, 2020.
  * (17) Ashish Vaswani et al.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * (18) Colin Raffel et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  * (19) Peter J. Liu et al.  Generating wikipedia by summarizing long sequences.  arXiv preprint arXiv:1801.10198, 2018.
  * (20) Alec Radford et al.  Improving language understanding by generative pre-training.  2018\.
  * (21) Rami Al-Rfou et al.  Character-level language modeling with deeper self-attention.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3159–3166, 2019.
  * (22) Junjie Ye et al.  A comprehensive capability analysis of gpt-3 and gpt-3.5 series models.  arXiv preprint arXiv:2303.10420, 2023.
  * (23) Alec Radford et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * (24) Bryan McCann et al.  The natural language decathlon: Multitask learning as question answering.  arXiv preprint arXiv:1806.08730, 2018.
  * (25) Tom Brown et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * (26) Long Ouyang et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * (27) Wayne Zhao Xin et al.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.
  * (28) OpenAI.  Introducing chatgpt, November 2022.
  * (29) Samantha Lock.  What is ai chatbot phenomenon chatgpt and could it replace humans.  The Guardian, 5, 2022.
  * (30) OpenAI.  Gpt-4 technical report, 2023.
  * (31) Samantha Lock.  What is ai chatbot phenomenon chatgpt and could it replace humans?  The Guardian.
  * (32) Jon Gertner.  Wikipedia’s moment of truth.  The New York Times Magazine.
  * (33) Chatgpt can now see, hear, and speak.  https://www.openai.com, 2023.  Retrieved October 16, 2023.
  * (34) Kevin Roose.  The new chatgpt can ’see’ and ’talk.’ here’s what it’s like.  The New York Times, September 27 2023.  Retrieved October 16, 2023.
  * (35) OpenAI.  Introducing gpts, November 2023.
  * (36) Hugo Touvron et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * (37) Meta AI.  Introducing llama: A foundational, 65-billion-parameter large language model.  February 2023.  URL-of-the-Article (Accessed: ).
  * (38) Jordan Hoffmann et al.  Training compute-optimal large language models.  arXiv preprint arXiv:2203.15556, 2022.
  * (39) Jack W. Rae et al.  Scaling language models: Methods, analysis & insights from training gopher.  arXiv preprint arXiv:2112.11446, 2021.
  * (40) Meta.  Meta and microsoft introduce the next generation of llama.  July 2023.  Retrieved 21 July 2023, from https://about.fb.com/news/2023/07/llama-2/.
  * (41) Hugo Touvron et al.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * (42) Baptiste Rozière et al.  Code llama: Open foundation models for code, 2023.
  * (43) Zhengxiao Du et al.  Glm: General language model pretraining with autoregressive blank infilling.  arXiv preprint arXiv:2103.10360, 2021.
  * (44) Aohan Zeng et al.  Glm-130b: An open bilingual pre-trained model.  arXiv preprint arXiv:2210.02414, 2022.
  * (45) THUDM.  Chatglm2-6b.  https://github.com/thudm/chatglm2-6b, 2022.
  * (46) THUDM.  Chatglm3.  https://github.com/THUDM/ChatGLM3, 2023.
  * (47) Jacob Devlin et al.  BERT: Pre-training of deep bidirectional transformers for language understanding.  pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * (48) Google AI Blog.  Open sourcing bert: State-of-the-art pre-training for natural language processing.  November 2018.  Retrieved November 27, 2019.
  * (49) Yinhan Liu et al.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * (50) Li Dong et al.  Unified language model pre-training for natural language understanding and generation.  Advances in neural information processing systems, 32, 2019.
  * (51) Mike Lewis et al.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  arXiv preprint arXiv:1910.13461, 2019.
  * (52) Neil Houlsby et al.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * (53) Jonas Pfeiffer et al.  AdapterHub: A framework for adapting transformers.  pages 46–54, Online, October 2020. Association for Computational Linguistics.
  * (54) Xiao Liu et al.  Gpt understands, too.  AI Open, 2023.
  * (55) Edward Hu et al.  Lora: Low-rank adaptation of large language models.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4395–4409, 2021.
  * (56) Tim Dettmers et al.  Qlora: Efficient finetuning of quantized llms.  arXiv preprint arXiv:2305.14314, 2023.
  * (57) Jason Wei et al.  Finetuned language models are zero-shot learners.  arXiv preprint arXiv:2109.01652, 2021.
  * (58) Tianyi Zhang et al.  Revisiting few-sample bert fine-tuning.  arXiv preprint arXiv:2006.05987, 2020.
  * (59) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow.  On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines.  arXiv preprint arXiv:2006.04884, 2021.
  * (60) Armen Aghajanyan et al.  Better fine-tuning by reducing representational collapse.  arXiv preprint arXiv:2008.03156, 2020.
  * (61) Haoming Jiang et al.  SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177–2190, Online, July 2020. Association for Computational Linguistics.
  * (62) Chen Zhu et al.  Freelb: Enhanced adversarial training for natural language understanding.  arXiv preprint arXiv:1909.11764, 2019.
  * (63) Timo Schick and Hinrich Schütze.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online, April 2021. Association for Computational Linguistics.
  * (64) Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816–3830, Online, August 2021. Association for Computational Linguistics.
  * (65) Timo Schick and Hinrich Schütze.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online, April 2021. Association for Computational Linguistics.
  * (66) Xu Han et al.  Ptr: Prompt tuning with rules for text classification.  AI Open, 3:182–192, 2022.
  * (67) Shengding Hu et al.  Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2225–2240, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * (68) Xiao Liu et al.  Gpt understands, too.  AI Open, 2023.
  * (69) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, Online, August 2021. Association for Computational Linguistics.
  * (70) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * (71) Xiao Liu et al.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  arXiv preprint arXiv:2110.07602, 2021.
  * (72) Guanghui Qin and Jason Eisner.  Learning how to ask: Querying LMs with mixtures of soft prompts.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212, Online, June 2021. Association for Computational Linguistics.
  * (73) Jiachang Liu et al.  What makes good in-context examples for GPT-3?  In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics.
  * (74) Yao Lu et al.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * (75) Mingda Chen et al.  Improving in-context few-shot learning via self-supervised training.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3558–3573, Seattle, United States, July 2022. Association for Computational Linguistics.
  * (76) Yanda Chen et al.  Meta-learning via language model in-context tuning.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 719–730, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * (77) Sewon Min et al.  MetaICL: Learning to learn in context.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809, Seattle, United States, July 2022. Association for Computational Linguistics.
  * (78) Jason Wei et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  * (79) Takeshi Kojima et al.  Large language models are zero-shot reasoners.  Advances in neural information processing systems, 35:22199–22213, 2022.
  * (80) Denny Zhou et al.  Least-to-most prompting enables complex reasoning in large language models.  arXiv preprint arXiv:2205.10625, 2022.
  * (81) Zhuosheng Zhang et al.  Automatic chain of thought prompting in large language models.  arXiv preprint arXiv:2210.03493, 2022.
  * (82) Kashun Shum, Shizhe Diao, and Tong Zhang.  Automatic prompt augmentation and selection with chain-of-thought from labeled data.  In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12113–12139, Singapore, December 2023. Association for Computational Linguistics.
  * (83) Zhuosheng Zhang et al.  Multimodal chain-of-thought reasoning in language models.  arXiv preprint arXiv:2302.00923, 2023.
  * (84) Haoyang Huang et al.  Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting.  In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365–12394, Singapore, December 2023. Association for Computational Linguistics.
  * (85) Li Deng et al.  Use of kernel deep convex networks and end-to-end learning for spoken language understanding.  In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 210–215. IEEE, 2012.
  * (86) Gokhan Tur et al.  Towards deeper understanding: Deep convex networks for semantic utterance classification.  In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5045–5048. IEEE, 2012.
  * (87) Daniele Comi et al.  Zero-shot-bert-adapters: a zero-shot pipeline for unknown intent detection.  In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 650–663, 2023.
  * (88) Soham Parikh et al.  Exploring zero and few-shot techniques for intent classification.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 744–751, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * (89) Yen-Ting Lin et al.  Selective in-context data augmentation for intent detection using pointwise V-information.  In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1463–1476, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.
  * (90) Samuel Coope et al.  Span-ConveRT: Few-shot span extraction for dialog with pretrained conversational representations.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 107–121, Online, July 2020. Association for Computational Linguistics.
  * (91) AB Siddique, Fuad Jamour, and Vagelis Hristidis.  Linguistically-enriched and context-awarezero-shot slot filling.  In Proceedings of the Web Conference 2021, pages 3279–3290, 2021.
  * (92) Qian Chen, Zhu Zhuo, and Wen Wang.  Bert for joint intent classification and slot filling. arxiv.  arXiv preprint arXiv:1902.10909, 2019.
  * (93) Hoang Nguyen et al.  CoF-CoT: Enhancing large language models with coarse-to-fine chain-of-thought prompting for multi-domain NLU tasks.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12109–12119, Singapore, December 2023. Association for Computational Linguistics.
  * (94) Vevake Balaraman and Bernardo Magnini.  Scalable neural dialogue state tracking.  In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 830–837. IEEE, 2019.
  * (95) Victor Zhong, Caiming Xiong, and Richard Socher.  Global-locally self-attentive encoder for dialogue state tracking.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1458–1467, 2018.
  * (96) Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim.  SUMBT: Slot-utterance matching for universal and scalable belief tracking.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5478–5483, Florence, Italy, July 2019. Association for Computational Linguistics.
  * (97) Yexiang Wang, Yi Guo, and Siqi Zhu.  Slot attention with value normalization for multi-domain dialogue state tracking.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3019–3028, 2020.
  * (98) Zhaojiang Lin et al.  MinTL: Minimalist transfer learning for task-oriented dialogue systems.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3391–3405, Online, November 2020. Association for Computational Linguistics.
  * (99) Yushi Hu et al.  In-context learning for few-shot dialogue state tracking.  In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627–2643, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  * (100) Yujie Feng et al.  Towards LLM-driven dialogue state tracking.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 739–755, Singapore, December 2023. Association for Computational Linguistics.
  * (101) Ryuichi Takanobu, Runze Liang, and Minlie Huang.  Multi-agent task-oriented dialog policy learning with role-aware reward decomposition.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 625–638, Online, July 2020. Association for Computational Linguistics.
  * (102) Sihan Wang et al.  Task-completion dialogue policy learning via monte carlo tree search with dueling network.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3461–3471, 2020.
  * (103) Gabriel Gordon-Hall, Philip John Gorinski, and Shay B. Cohen.  Learning Dialog Policies from Weak Demonstrations.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1394–1405, Online, July 2020. Association for Computational Linguistics.
  * (104) Wenhu Chen et al.  Semantically conditioned dialog response generation via hierarchical disentangled self-attention.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3696–3709, Florence, Italy, July 2019. Association for Computational Linguistics.
  * (105) YunHao Li et al.  Retrieve & memorize: Dialog policy learning with multi-action memory.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 447–459, Online, August 2021. Association for Computational Linguistics.
  * (106) Paweł Budzianowski and Ivan Vulić.  Hello, it’s GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems.  In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 15–22, Hong Kong, November 2019. Association for Computational Linguistics.
  * (107) Ehsan Hosseini-Asl et al.  A simple language model for task-oriented dialogue.  Advances in Neural Information Processing Systems, 33:20179–20191, 2020.
  * (108) Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim.  Gpt-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems.  In 10th International Conference on Learning Representations, ICLR 2022. International Conference on Learning Representations, ICLR, 2022.
  * (109) Govardana Sachithanandam Ramachandran, Kazuma Hashimoto, and Caiming Xiong.  [CASPI] causal-aware safe policy improvement for task-oriented dialogue.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 92–102, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * (110) Wanwei He et al.  Galaxy: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection.  In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 10749–10757, 2022.
  * (111) Xiao Yu, Maximillian Chen, and Zhou Yu.  Prompt-based Monte-Carlo tree search for goal-oriented dialogue policy planning.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7101–7125, Singapore, December 2023. Association for Computational Linguistics.
  * (112) EHUD REITER and ROBERT DALE.  Building applied natural language generation systems.  Natural Language Engineering, 3(1):57–87, 1997.
  * (113) Tsung-Hsien Wen et al.  Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking.  In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 275–284, Prague, Czech Republic, September 2015. Association for Computational Linguistics.
  * (114) Tsung-Hsien Wen et al.  Semantically conditioned LSTM-based natural language generation for spoken dialogue systems.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
  * (115) Hao Zhou, Minlie Huang, and Xiaoyan Zhu.  Context-aware natural language generation for spoken dialogue systems.  In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2032–2041, 2016.
  * (116) Baolin Peng et al.  Few-shot natural language generation for task-oriented dialog.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 172–182, Online, November 2020. Association for Computational Linguistics.
  * (117) Zhiyu Chen et al.  Few-shot NLG with pre-trained language model.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 183–190, Online, July 2020. Association for Computational Linguistics.
  * (118) Ashutosh Baheti, Alan Ritter, and Kevin Small.  Fluent response generation for conversational question answering.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 191–207, Online, July 2020. Association for Computational Linguistics.
  * (119) Jing Qian et al.  Controllable natural language generation with contrastive prefixes.  In Findings of the Association for Computational Linguistics: ACL 2022, pages 2912–2924, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * (120) Baolin Peng et al.  Soloist: Building task bots at scale with transfer learning and machine teaching.  Transactions of the Association for Computational Linguistics, 9:807–824, 2021.
  * (121) Yunyi Yang, Yunhao Li, and Xiaojun Quan.  Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14230–14238, 2021.
  * (122) Yixuan Su et al.  Multi-task pre-training for plug-and-play task-oriented dialogue system.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4661–4676, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * (123) Wanwei He et al.  SPACE-2: Tree-structured semi-supervised contrastive pre-training for task-oriented dialog understanding.  In Proceedings of the 29th International Conference on Computational Linguistics, pages 553–569, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
  * (124) Wanwei He et al.  Unified dialog model pre-training for task-oriented dialog understanding and generation.  In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 187–200, 2022.
  * (125) Vladimir Karpukhin et al.  Dense passage retrieval for open-domain question answering.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online, November 2020. Association for Computational Linguistics.
  * (126) Chunyuan Yuan et al.  Multi-hop selector network for multi-turn response selection in retrieval-based chatbots.  In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 111–120, 2019.
  * (127) Chongyang Tao et al.  One time of interaction may not be enough: Go deep with an interaction-over-interaction network for response selection in dialogues.  In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1–11, 2019.
  * (128) Xinchao Xu et al.  Long time no see! open-domain conversation with long-term persona memory.  In Findings of the Association for Computational Linguistics: ACL 2022, pages 2639–2650, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * (129) Andrea Madotto et al.  Personalizing dialogue agents via meta-learning.  In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 5454–5459, 2019.
  * (130) Ruijun Chen et al.  Learning to memorize entailment and discourse relations for persona-consistent dialogues.  arXiv preprint arXiv:2301.04871, 2023.
  * (131) Juntao Li et al.  Dialogue history matters! personalized response selection in multi-turn retrieval-based chatbots.  ACM Transactions on Information Systems (TOIS), 39(4):1–25, 2021.