# Self-seeding and Multi-intent Self-instructing LLMs for Generating Intent-
aware Information-Seeking dialogs

Arian Askari1, Roxana Petcu2, Chuan Meng2, Mohammad Aliannejadi2
Amin Abolghasemi1, Evangelos Kanoulas2, Suzan Verberne1
1Leiden University<EMAIL_ADDRESS>
2University of Amsterdam
{r.m.petcu<EMAIL_ADDRESS>

###### Abstract

Identifying user intents in information-seeking dialogs is crucial for a
system to meet user’s information needs. Intent prediction (IP) is challenging
and demands sufficient dialogs with human-labeled intents for training.
However, manually annotating intents is resource-intensive. While large
language models (LLMs) have been shown to be effective in generating synthetic
data, there is no study on using LLMs to generate intent-aware information-
seeking dialogs. In this paper, we focus on leveraging LLMs for zero-shot
generation of large-scale, open-domain, and intent-aware information-seeking
dialogs. We propose SOLID, which has novel self-seeding and multi-intent self-
instructing schemes. The former improves the generation quality by using the
LLM’s own knowledge scope to initiate dialog generation; the latter prompts
the LLM to generate utterances sequentially, and mitigates the need for manual
prompt design by asking the LLM to autonomously adapt its prompt instruction
when generating complex multi-intent utterances. Furthermore, we propose
SOLID-RL, which is further trained to generate a dialog in one step on the
data generated by SOLID. We propose a length-based quality estimation
mechanism to assign varying weights to SOLID-generated dialogs based on their
quality during the training process of SOLID-RL. We use SOLID and SOLID-RL to
generate more than 300k intent-aware dialogs, surpassing the size of existing
datasets. Experiments show that IP methods trained on dialogs generated by
SOLID and SOLID-RL achieve better IP quality than ones trained on human-
generated dialogs.

##  1 Introduction

Intent prediction111Intent prediction and intent classification are used
interchangeably in this work. (IP) plays a vital role in information-seeking
dialogs (Zamani et al., 2023; Qu et al., 2019b, a). It aims to identify the
intent of the speaker at each utterance in the dialog (Colombo et al., 2020).
Figure 1 illustrates an example of intents in an information-seeking
conversation. The list of unique intents used in this work is presented in
Table 7 in the Appendix. Leveraging predicted intents has been shown to
benefit various IR and NLP tasks, e.g., response selection (Yang et al., 2020;
Kumar et al., 2019), answer selection (Deng et al., 2023) and user
satisfaction estimation (Deng et al., 2022).

Figure 1: Example dialog with sequence of intents.

Motivation. IP has been shown to be a challenging task (Qu et al., 2019b).
Recent work (Zhao et al., 2023a) reveals that prompting ChatGPT in a few-shot
manner achieves limited IP performance and that ChatGPT performs worse than
supervised IP methods. These supervised methods require a large number of
information-seeking dialogs with human-annotated intents to get a satisfactory
performance. However, manually annotating intents is resource-intensive,
resulting in limited information-seeking dialog training data for IP (Qu et
al., 2019a; Penha et al., 2019). For example, (Qu et al., 2019b) shows that
labeling intents for 2,199 dialogs costs $1,700.

Figure 2: Example of a seed generated through self-seeding via SOLID.

To address this issue, we focus on data augmentation for intent-aware
information-seeking dialogs, where each dialog utterance has one or more
associated intents. We investigate LLMs’ potential Abbasiantaeb et al. (2023);
Ding et al. (2023) in zero-shot intent-aware information-seeking dialog
generation. Specifically, we focus on the scenario where we initially provide
a sequence of intents and then instruct LLMs to generate dialogs following
those intent sequences. While one could argue that LLMs can be used to label
intents for information-seeking dialogs without human intent labels, both our
experiments and previous research indicate that LLMs have limited
effectiveness in accurately labeling intents Zhao et al. (2023b); Hu et al.
(2023); Zhao et al. (2023a). To the best of our knowledge, no prior research
has been done in this direction. We focus on open-source LLMs because
commercial ones, such as ChatGPT or GPT-4, bring limitations like non-
reproducibility, non-deterministic outputs, and potential data leakage between
pre-training and evaluation, impeding their utility in scientific research
(Pradeep et al., 2023; Zhang et al., 2023).

Challenges. We identify the following challenges when using LLMs for
generating intent-aware information-seeking dialogs:  (i) it is important to
ground the generation of an intent-aware dialog on a “seed” (Kulkarni et al.,
2024), i.e., user’s information need (entity information), user’s initial
query (conversation starter), intent sequence (the flow of intents in the
dialog to be generated), etc; see Figure 2; although the common practice is to
extract information from external resources (e.g., Wikipedia) and regard it as
seeds Ding et al. (2023), feeding such seeds (we define them as external
seeds) into an LLM has the risk of resulting in a low-quality generation.
Recent work has shown that LLMs tend to perform badly if the input information
was not observed during LLM pre-training (Sun et al., 2023), and it is
challenging to verify the LLM’s familiarity with specific information;  (ii)
our preliminary results show that prompting an open-source LLM to directly
generate the entire dialog performs badly, i.e., the LLM often generates too
many or few utterances that do not follow the intent sequence in the given
seed; although generating utterances one by one (generate the next one based
on the previous one) has the potential to improve generation quality. Also, as
one utterance can be associated with multiple intent labels (see Figure 1), it
requires us to adjust prompt instruction for all possible combinations of
intent labels, leading to a substantial amount of manual prompt engineering.

Figure 3: Illustrating SOLID and SOLID-RL: Starting with a seed generated by
SOLID and with sequences of real-world intents, SOLID generates utterances
guided by a specific intent or intents. It creates utterances sequentially,
each building on the previous ones. During phase 3, each utterance generation
fits under one of two cases: either the intent corresponds to one definition
from taxonomy 7 (we call this a single intent), or it combines multiple
definitions (we call this multiple intents). If it’s the second case, SOLID
employs multi-intent self-instruction to generate one merged instruction. This
allows SOLID to generate an utterance that encompasses any combination of
multiple intents.

SOLID. To overcome these challenges, we propose SOLID, a Self-seeding and
multi-intent self-instructing method for generating Large-scale Intent-aware
Information-Seeking Dialogues. SOLID is featured by novel self-seeding and
multi-intent self-instructing schemes:  (i) self-seedingasks an LLM first to
use its own knowledge scope to generate a dialog seed, and the generated seed
would be fed to the same LLM to generate a dialog; it ensures the LLM is
familiar with the input seed and has the potential to increase dialog
generation quality; and  (ii) based on the LLM-generated seeds, SOLID
generates utterances one-by-one within a dialog; during utterance generation,
multi-intent self-instructing first asks the LLM to automatically generate a
prompt instruction based on the intent label combination for an utterance, and
the generated prompt instruction is used to prompt the same LLM to generate
the corresponding utterance; it significantly decreases the need for manual
prompt engineering across various intent combinations.

SOLID-RL. SOLID incurs a significant computational cost as it generates
utterances one by one, requiring feeding the dialog history for each utterance
generation. While prompting an LLM to generate an entire dialog in one step
may enhance efficiency, it adversely hurts generation quality. To enhance
SOLID’s efficiency without compromising its effectiveness, we propose SOLID-
RL, which is trained to generate entire dialogs in a single step using
synthetic dialogs generated by SOLID; see Figure 3. We identify the new
challenge of training SOLID-RL: training SOLID-RL on all synthetic dialogs
equally leads to suboptimal generation quality because some low-quality
synthetic dialogs may hurt SOLID-RL’s effectiveness. To address this new
challenge, we propose a mechanism of Length-based quality estimation for
Mixed-Quality training (LMQ). It treats the length of a dialog as a proxy for
estimating the quality of SOLID-generated dialogs and then assigns weights to
these synthetic dialogs based on their estimated quality during the training
process of SOLID-RL. Note that SOLID-RL remains zero-shot, as we do not fine-
tune it on any human-labeled intent-aware dialogs. We release our data and
code at https://github.com/arian-askari/solid.

Experiments. Our analysis indicates that IP methods trained on synthetic
dialogs based on self-seeding achieve significantly higher effectiveness than
methods trained on dialogs based on external seeds; e.g., a T5-based IP model
trained on an equal number of training instances achieves 0.725 F1 score for
self-seeding versus 0.663 with external seeding. Furthermore, we show that
SOLID-RL can generate dialogs from a single prompt with an 11 times increase
in efficiency over SOLID, and yields even better dialog generation quality
with the help of LMQ.

Contributions. Our contributions are as follows:

  * •

SOLID: We propose a novel Self-seeding and multi-intent self-instructing
method for generating Large-scale Intent-aware Information-Seeking Dialogues.

  * •

We introduce SOLID-RL, which incorporates a novel mechanism of Length-based
quality estimation for Mixed-Quality training (LMQ), making it 11 times more
efficient than SOLID while improving data generation quality.

  * •

We provide two large-scale synthetic intent-aware information-seeking dialog
datasets: SOLISpeak, consisting of 316,697 dialogs generated by SOLID; and
SOLITurbo, which is built by SOLID-RL.

  * •

We showcase that IP methods trained on a combination of limited available
human data and a limited portion of SOLISpeak and SOLITurbo show significantly
better effectiveness than those purely trained on human-labeled datasets.

##  2 Related work

Figure 4: The illustration of seed generation through self-seeding via SOLID.

Data augmentation with LLMs. Augmenting training data using LLMs such as GPT-4
has become prominent with advancements in text generation Tunstall et al.
(2023). The Self-Instruct method Wang et al. (2022) allows LLMs to augment
their instruction-tuning training datasets; this being used in many LLMs Taori
et al. (2023); Chiang et al. (2023); Ding et al. (2023); Zheng et al. (2023).
However, none of the previously generated datasets provide associated user
intents to each dialog utterance, such that they can be used for downstream
tasks. There are studies that augment training data for intent classification
tasks using LLMs where utterances are not associated with any dialog Sahu et
al. (2022), which is completely different from intent-aware conversation
generation. Furthermore, unlike Kim et al. (2022), which generates dialogs
limited to "yes" and "no" responses, our work supports 12 unique intents where
one utterance can contain multiple intents.

Intent Prediction in Information-seeking dialogs. IP of utterances in dialog
systems is a well-studied task that aims to identify the intent of a speaker
at each utterance in a dialog (Colombo et al., 2020; Zamani et al., 2023; Qu
et al., 2019b). MSdialog-Intent Qu et al. (2018a) and MANtIS Gustavo Penha and
Hauff (2019) are the largest intent-aware information-seeking dataset;
statistics of them are represented in Table 1. The reason behind the small
size of these datasets is that building an intent-aware conversation dataset
with human annotators is very costly; as stated by Qu et al. (2019a)
annotating utterances’ intents of 2,100 information-seeking conversations
costs about 1,700 dollars using Amazon MTurk. In this paper, we focus on open-
domain information-seeking dialogs, while the research that studies IP on
other types of dialogs are out of the scope of our study, such as chit-chat
dialogs (Deng et al., 2022; Ghazarian et al., 2022; Qu et al., 2023), dialogs
with evidence from single or multi knowledge sources Wu et al. (2023), and
task-oriented dialogs Yan et al. (2017); Rastogi et al. (2020); Shah et al.
(2018).

Dataset | # Dialogs |  Avg. # Turns |  Avg. Length (by token) |  Avg. Utt. Length (by token)
---|---|---|---|---
MSDialog | 2,199 | 4.56 | 297.12 | 65.16
MANtIS | 1,356 | 5.90 | 492.65 | 131.48
SOLID-LM (Ours) | 316,697 | 4.57 | 1064.7 | 236.6

Table 1: Statistics of intent-aware information-seeking dialog datasets. ‘T’,
‘Dial’, ‘Utt’, and ‘L’ refer to Turns, Dialogs, Utterances, and Length
respectively.

##  3 Intent-Aware Dialog Generation

###  3.1 SOLID

SOLID generates dialog seeds that it uses later as the initial point of dialog
generation. See Figure 2 as an example of a seed that can be generated by
SOLID. It starts by first generating entity types (e.g., ’Person’), names
(e.g., ’Albert Einstein’), and attributes (e.g., ’Occupation’). We prompt the
LLM to generate entity background documents (e.g., "Albert Einstein was born
in…") together with questions about the entity and its attributes, as having
all this information in the seed helps the LLM to generate deeper and more
diverse dialogs.

Below, we first describe the steps we take to generate the dialog seed and
starter, followed by background document, intent sequence, and dialog
generation, as illustrated in Figure 4 and 10.

Step 1: seed generation. As SOLID’s dialogs are pivoted around specific entity
names, we follow 3 steps to ensure the diversity of the seeds, as well as
LLM’s familiarity with them.

  * •

Entity type generation: We first prompt the LLM to generate 100 entity types
(e.g., ‘Person’), as well as 10 attributes associated with each entity type
(e.g., ‘Occupation’).

  * •

Entity name generation: for each entity type, we prompt the LLM to generate a
name (e.g., ‘Albert Einstein’). As a proxy for diversity, we constraint the
LLM to generate 100 names per English alphabet and entity type ($26\times
100\times 100$), resulting in 50K entity names after automatic post-
processing.222Details of the prompts are available in the appendix in section
8.9.

Step 2: ice breaker. Given the output of Step 1, we aim to generate an ice
breaker for the LLM to start the conversation, following the steps below:

  * •

Background document generation: inspired by Yu et al. (2022), to enhance the
quality of the generated dialog, we prompt the LLM to generate a document
about the given entity name (e.g., “Albert Einstein was a …”).

  * •

Conversation starter: given the entity type, attributes, and name, as well as
the background document, we prompt the LLM to generate the first utterance in
a conversation (e.g., “Can you delve …”).

Step 3: intent-aware dialogs. We use the output of Step 1 & 2 and follow
Algorithm 1 presented in the Appendix to generate the data, prompting the LLM
to generate the next utterance for a given intent at every step (see Section
8.8 for more details). Finally, we post-process the generated dialogs and
filter out the noisy ones.

###  3.2 RL-guided generation

While we generated large-scale intent-aware conversations in the zero-shot
setup, there are two drawbacks: (1) we need to generate the dialog utterance-
by-utterance which is an inefficient process; (2) we need to truncate each
utterance to a maximum number of new tokens if the LLM does not stop
generation by itself. For some utterances, this makes them look less natural.
If we could assign the task of generating the whole dialog in one step to the
LLM, it could ensure the quality of finishing each utterance naturally.
However, our studied LLMs cannot generate a whole dialog in the desired format
in a zero-shot setup. In this section, we propose SOLID-RL, which trains on
the generated data in a zero-shot setup to build more natural conversations in
one pass.

To improve efficiency while maintaining effectiveness and making the dialogs
more natural, we propose a novel length-based mixed-quality (LMQ) training
method and apply it to the recently proposed recipe of training for LLMs
proven to be effective consisting of two steps of supervised fine-tuning
followed (SFT) by direct preference optimization (DPO) Tunstall et al. (2023);
Rafailov et al. (2023). We use SOLISpeak data generated by SOLID as the ideal
output during this optimization to be independent of any human data. Our
intuition in proposing LMQ is to avoid treating all training samples because
some low-quality dialogs could affect the training optimization. The closest
work to this setup is OpenChat, where the authors make use of training data
from GPT3.5 and GPT4, aware that the quality of GPT4 is higher in contrast to
our setup Wang et al. (2024).

Specifically, we propose a simple yet effective heuristic to classify our data
that comes from the same source (LLM) – in contrast to Wang et al. (2024) –
but can have different qualities. We implement this by conditioning each
example based on its length during training, where we look at different ranges
of length as indicators of different class qualities. By conditioning, we mean
adding a textual input before the dialog that represents a meaningful
categorical phrase that estimates the quality of the data. This is achieved by
appending one of the following five length-based conditioned inputs before the
dialog, corresponding to the number of utterances: “excellent quality dialog:”
for 1-3, “good quality dialog:” for 4, “average quality dialog:” for 5-10,
“poor quality dialog:” for 11-15, and “very poor quality dialog:” for 16-20.
Below is an example of distinct initial prompt tokens for the highest quality:
[for short dialog] Seed<|assisstant|> Excellent quality dialog:
[for lengthiest dialog] Seed<|assisstant|> Poor quality dialog:
Our motivation is based on the finding that lengthier dialogs tend to have
more mistakes than shorter ones. During the inference phase, we assume that
our SOLID-RL has learned to distinguish higher-quality data distributions.
Considering that we aim to exclusively generate high-quality responses by
prompting them with:
[for any dialog] Seed<|assisstant|> Excellent quality dialog:

###  3.3 Utility analysis

We evaluate the utility, i.e., usefulness, of our SOLISpeak and SOLITurbo
datasets by training IP models on them and evaluating the trained models on
human-labeled datasets. We use the following list of IP models:  (i) cnn-
context-rep, a top-performing model on MSDialog;  (ii) BERT, which has shown
state-of-the-art results on MANtIS;  (iii) We also fine-tune T5 for intent
prediction, adapting it for intent generation;  (iv) zephyr-7b-betain a few-
shot intent prediction setup.

To maximize the efficiency and effectiveness of our dataset, we explore three
distinct scenarios, where different sampling strategies are applicable. By
sampling, we mean utilizing part of the SOLISpeak or SOLITurbo dataset which
is more relevant to the target human-written dialog dataset where the trained
IP models on our data are going to be applied:

  * •

Scenario 1: No data We suppose that we do not have access to any dialog data
in the target human-written dialog dataset. We train IP models solely on our
synthetic datasets and evaluate them on each human-annotated dataset.

  * •

Scenario 2: Limited unlabeled data. Here, we suppose we have access to human-
written dialogs where there is no available information regarding the intent
of each utterance. In this scenario, we use the human-written dialogs as
queries in a retrieval setup where we retrieve the top-k most relevant dialogs
from SOLISpeak or SOLITurbo as a collection of dialogs. This way, we sample
the most textually similar text of our dataset to the target human data. We
utilize two retrieval-based approaches: BM25 Robertson and Zaragoza (2009);
Elasticsearch (2023), and a bi-encoder BERT-based model Transformers (2023),
and experiment with k values from 5 to 20, incrementing by 5, to identify the
most effective setup.

  * •

Scenario 3: Limited labeled dialog. This is the more realistic scenario where
some dialogs are annotated but the size of the annotation is quite small. For
such a scenario, we propose two methods—intent balancing (Int-Bal) and
sequence of intents balancing (SeqInt-Bal)—to sample dialogs from the human-
written dialogs dataset. These methods aim to equalize the frequency of each
intent label or intent sequence (sequence of intents) in the final training
dataset. The final training dataset in this scenario is a combination of both
small available human-labeled data and our synthetic data, ensuring diverse
training examples. Additionally, we explore random sampling to supplement
human-labeled data, selecting an equal number of dialogs from SOLID to match
the human-labeled set, we call this method “Random-EQ”.

##  4 Experimental Setup

LLM selection and Training. We select Zephyr-7b-beta as our LLM during this
study as it is one of the competitive and strong 7B models at the time of this
work. We use Huggingface’s trainers for supervised fine-tuning of SOLID-RL and
direct preference optimization (DPO). We use QLora 4bit (Dettmers et al.,
2023) for efficient fine-tuning. We employ a batch size and gradient
accumulation of 4, and train for 4000 steps for both SFT and DPO, which takes
about 7 hours to be trained on a single NVIDIA GPU P6000 40GB GPU memory.

Dataset and Metrics. We evaluate the SOLID-generated dataset based on the
human annotations provided in the MSDialog-Intent Qu et al. (2019a) and MANtIS
Gustavo Penha and Hauff (2019) datasets. During the seed generation process
for SOLID, we select intent sequences from the real-world dataset MSDialog-
Intent to ensure a realistic order and distribution of intents, and randomly
assign each real-world intent sequence to a seed. Furthermore, we evaluate the
power of our RL-trained LLM on the SOLID-generated data using the benchmark
datasets from the leaderboard of Huggingface Huggingface (2024). Although this
leaderboard does not directly measure the conversational quality of chat
models, it does provide a useful signal to validate whether fine-tuning has
introduced regressions on the base model’s reasoning and truthfulness
capabilities.

Table 2:  The results of the ‘cnn-context-rep’ model Qu et al. (2018b) in
different scenarios, utilizing various sampling methods. The terms
‘SOLISpeak%’ and ‘MS%’ denote the proportion of dialogs sampled from our
synthetically generated dataset (SOLISpeak) and the human-annotated MSDialog-
intent dataset, respectively.

| Sampling | SOLISpeak% | MS% | P | F1
---|---|---|---|---|---
| Micro | Macro
Scenario 3
| — | 0% | 100% | .7321 | .7562 | .6713
| SeqInt-Bal | 35.3 % | 100% | .7641 | .8076 | .7246
| Int-Bal | 6.0% | 100% | .7613 | .7857 | .7211
| Random-Eq | 0.8 % | 100% | .7402 | .7640 | .6940
Scenario 2
| Bi-Encoder | 8.1% | 0% | .7626 | .7989 | .7228
| BM25 | 8.1% | 0% | .7600 | .7745 | .7198
Scenario 1
| — | 100% | 0% | .7167 | .7197 | .6406

Table 3: The Results of intent predictors on the most effective sampling
method (SeqInt-Bal) and the most simple while effective (RandomEq) in addition
to the using only human as training data.

Model | Train source | MSDialog | MANtIS
---|---|---|---
P | F1-Micro | F1-Macro | P | F1-Micro | F1-Macro
cnn-context-rep | SeqInt-Bal | .764 | .808 | .725 | .801 | .786 | .625
Random-Eq | .740 | .764 | .694 | .784 | .744 | .595
Only human | .732 | .756 | .671 | .777 | .736 | .572
BERT | SeqInt-Bal | .810 | .831 | .743 | .855 | .814 | .644
Random-Eq | .762 | .792 | .711 | .814 | .772 | .617
Only human | .734 | .758 | .674 | .779 | .738 | .591
T5 | SeqInt-Bal | .839 | .868 | .771 | .877 | .833 | .669
Random-Eq | .791 | .828 | .737 | .824 | .812 | .650
Only human | .750 | .772 | .685 | .780 | .745 | .587
T5 on SOLID-RL | SeqInt-Bal | .846 | .879 | .786 | .881 | .839 | .677
Random-Eq | .807 | .835 | .746 | .829 | .814 | .656
Zephyr-fewshot | 3-shots | .416 | .479 | .405 | .486 | .437 | .344

##  5 Results

We address the following research questions:

  * •

(RQ1) How effective is SOLISpeak for being used to train intent predictors
across different scenarios with either zero or partial availability of human
data, and to what extent can the use of SOLITurbo dataset enhance the
effectiveness of these intent predictors compared to SOLISpeak?

  * •

(RQ2) What is the impact of self-seeding on SOLID compared to alternative
external-seeding methods?

  * •

(RQ3) How does SOLID-RL’s capability to generate a full intent-aware dialog
from a single zero-shot instruction impact its computational efficiency
compared to SOLID’s utterance-by-utterance dialog generation?

Table 4: Comparative effectiveness of dialog generation using self-seeding vs.
random selection of popular entities. The utterance classification model is T5
evaluated on MSDialog.

| P | F1
---|---|---
| Micro | Macro
Self-seeding (ours) | .7085 | .7251 | .624
External-seeding methods
Top-most-frequent entities | .6457 | .6604 | .5748
Sampling frequent entities | .6460 | .6633 | .5769
Sampling random entities | .5839 | .6101 | .4976

Main results (RQ1). Table 2 shows the impact of different sampling methods on
MSDialog-intent. We use the state-of-the-art IP method on this dataset for
this analysis, as the goal is to compare the usefulness of generated data in
different scenarios per designed sampling methods and provide insights on that
rather than analyzing the performance across different IP methods. We find
that the most effective sampling method is the sequence of intents balancing
(SeqInt-Bal), which uses a higher proportion of SOLID data, and intent
balancing is the second-best method by using about 6 times smaller amount of
SOLISpeak data. We find that for all sampling methods, training the IP method
on the combination of human and our dataset (SOLISpeak) leads to higher
effectiveness than only using human data, proving the usefulness of SOLISpeak.
Furthermore, we find using a bi-encoder for sampling to be the second most
effective sampling method. An interesting observation is that Random-Eq
outperforms the cases where only the human-labeled data or only SOLID-
generated data is used. We also find that using only the SOLISpeak leads to
only slightly lower effectiveness than using human data, which shows the great
utility of our SOLISpeak dataset in scenarios where no human data is available
or shared with IP methods to be trained on.

Next, to address R1 in more depth, we investigate the utility of SOLISpeak and
SOLITurbo datasets for an additional dataset and across several IP methods. We
focus on scenario 3, where a small human-generated dataset is available, and
use the most effective (SeqInt-Bal) and simplest but effective (Random-EQ)
method. We also consider Zephyr in a few-shot setup for intent prediction to
assess if the LLM in a few-shot setup could be a slow but effective IP method.
We have three main observations based on Table 3: (1) utilizing our synthetic
dataset improves all three of the supervised IP methods compared to only using
human data by a large margin; this proves the usefulness of our dataset in
downstream tasks; (2) SOLID-RL generates more useful data for the IP method as
the T5 model trained on a combination of human and SOLID-RL data achieves the
highest effectiveness compared to other IP methods; (3) Zephyr cannot predict
the intents effectively in a few-shot setup, showing the challenge of
prediction for LLMs while they can follow instructions to generate content
with specific intents in them. This aligns with previous studies’ findings
Zhao et al. (2023b); Hu et al. (2023) and can be due to the fact that
following instructions is a more natural task for LLMs than acting as an IP
method in a few-shot setup.

Table 5: Analyzing the impact of self-instructing and length-based quality
estimation for mixed-quality training in SOLID-RL.

Model | P | F1
---|---|---
Micro | Macro
SOLID (ours) | .708 | .725 | .624
W/o self-instructing | .515 | .537 | .428
SOLID-RL (ours) | .741 | .778 | .68
W/o L-based MQ | .532 | .554 | .457

Table 6: The evaluation results in terms of accuracy on the Huggingface Open
LLM Leaderboard (Huggingface, 2024), which measures the performance of LMs
across four multiclass classification tasks: ARC (Clark et al., 2018),
HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful
QA (Lin et al., 2022).

| Truthful QA | Hella Swag | MMLU | ARC
---|---|---|---|---
UltraLM-13B Touvron et al. (2023) | 52.00 | 61.32 | 50.45 | 57.25
Zephyr-7B Tunstall et al. (2023) | 57.44 | 84.52 | 61.44 | 62.03
SOLID-RL-7B (Ours) | 58.13 | 83.82 | 61.68 | 60.78

Analyzing self-seeding impact (RQ2). We analyze self-seeding alongside three
alternative external seeding methods. We employ different sampling methods for
external seeding. Specifically, we prompt an LLM to generate 10,000 intent-
aware dialogs about 1,000 entities as the external seed. This is achieved by
initially generating ten different questions for each entity to serve as the
starting point of the conversation. The alternative external-seeding methods
are as follows: Top-most-popular entities, we prompt the LLM with the top
1,000 most popular entities from Wikipedia; in Sampling from popular entities,
we prompt the LLM with 1,000 entities randomly sampled from the top 10,000
most popular entities on Wikipedia; and for Sampling random entities, we
randomly sample entities from Wikipedia. It is crucial to note that the
sequence of intents remains consistent across this analysis. Table 4 shows
that the IP models trained on data generated by the external seeding methods
achieve lower effectiveness than those trained with self-seeding. This
confirms the intuition behind self-seeding, which shows that it could improve
the quality of data generated by the LLM.

SOLID-RL efficiency (RQ3). We analyze the efficiency of SOLID-RL, with which
we can generate the entire dialog given a single instruction, in contrast to
the utterance-by-utterance generation process of SOLID. We assess the time it
takes to generate 1K dialogs using both our SOLID and SOLID-RL methods. We use
a batch size of 16 during inference. Our experiments show that this generation
takes 23K seconds for SOLID compared to 2K seconds for SOLID-RL, demonstrating
that SOLID-RL is nearly 12 times more efficient than SOLID for intent-aware
dialog generation.

##  6 Discussion

Multi-intent Self-instruction Analysis. We assess the impact of multi-intent
self-instructing on IP models, using 10,000 seeds for dialog generation with
SOLID, comparing self-instruction to an alternative rule-based approach that
creates multi-intent utterances by inserting “an” between instructions. The
top section of Table 5 reveals a significant performance discrepancy between
models trained on SOLID with self-instruction versus those using the rule-
based method, indicating multi-intent self-instructing as a promising
direction.

Length-based mixed-quality training (LMQ). Utilizing the same 10,000 seeds as
above, we explore LMQ’s impact on intent-aware dialog generation quality. By
excluding LMQ’s class-conditioned input and training SOLID-RL from scratch, we
assess IP methods on SOLID-RL with and without LMQ. Results in the bottom
section of Table 5 reveal a significant advantage for SOLID-RL trained with
LMQ, which shows LMQ optimizes the LLM effectively and avoids low-quality
dialogs impacting the LLM’s performance. It also shows that conditioning the
model with LMQ for high-quality output during inference aids in generating
higher-quality data.

LLM benchmarks. We analyze the effect of SOLID-RL on the performance of its
base model, Zephyr-7b-beta, on multiple LLM benchmarks on the Open LLM
Leaderboard (Huggingface, 2024) to gain insight into whether SOLID-RL
introduces regressions on the base model’s reasoning and truthfulness
capabilities. Table 6 shows that SOLID-RL can enhance Zephyr on MMLU
(Hendrycks et al., 2021), and Truthful QA (Lin et al., 2022) benchmarks. Our
finding is relatively aligned with previous studies that show LLMs can enhance
themselves when trained on high-quality self-generated data (Huang et al.,
2022; Yuan et al., 2024; Chen and Li, 2024; Chen et al., 2024). More details
are presented in Table 9 in the appendix alongside an in-depth analysis of the
impact of each part of SOLID-RL.

##  7 Conclusion

This work introduces a novel approach to automatically generating intent-aware
information-seeking dialog datasets through two zero-shot generation methods,
SOLID and SOLID-RL. SOLID utilizes a novel zero-shot, multi-step process that
initiates with self-seeding and progresses to self-instructing the LLM. SOLID-
RL, optimized with supervised fine-tuning followed by DPO optimization, uses
SOLID’s output as its training set. SOLID-RL can generate dialogs given a
single instruction, resulting in an efficiency that is 11 times higher than
the utterance-by-utterance generation process of SOLID. This is achieved by
employing a novel length-based mixed-quality training method that dynamically
assesses dialog quality. Using these methods, we developed two synthetic
datasets that can significantly enhance intent prediction accuracy. Despite
LLMs’ limitations in intent prediction, our findings show their potential to
enrich training datasets. Our generated datasets offer valuable resources for
other tasks such as next intent prediction, and conversational search. Our
self-seeding and self-instructing approach is adaptable to various
conversation types and languages with minimal modifications.

## Limitations

In this paper, we demonstrated that LLMs can generate data to effectively
train intent predictor models within the context of information-seeking
dialogs. It is important to emphasize that this paper did not explore all
aspects of evaluation, particularly the impact of biased information in the
generated dialogs on the biases within the LLM’s conversational abilities. It
is well-known that LLMs exhibit biases in their outputs. In our study, we did
not examine how these biases affect the performance of intent predictors,
which could lead to a tendency for certain utterances to be predicted with
biased intents. Factually incorrect information in the generated data (as a
result of LLM’s hallucination) is not likely to be harmful in the intent
prediction models context, because only information contained in human dialog
will serve as input for the intent predictors in the test and real-world
scenarios. Moreover, these predictors are limited to only the defined intents
and cannot hallucinate new intents. We did not systematically investigate or
quantify the impact of hallucinated data on the performance of intent
predictor models and the potential bias they could introduce in the
prediction. In the Appendix, we provide more information on this and estimate
a generated entity that does not exist in sources like Wikipedia as a
hallucination, but this is susceptible to errors.

## References

  * Abbasiantaeb et al. (2023) Zahra Abbasiantaeb, Yifei Yuan, Evangelos Kanoulas, and Mohammad Aliannejadi. 2023.  Let the llms talk: Simulating human-to-human conversational qa via zero-shot llm-to-llm interactions.  _arXiv preprint arXiv:2312.02913_.
  * Chen and Li (2024) Weixin Chen and Bo Li. 2024.  Grath: Gradual self-truthifying for large language models.  _arXiv preprint arXiv:2401.12292_.
  * Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024.  Self-play fine-tuning converts weak language models to strong language models.  _arXiv preprint arXiv:2401.01335_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try ARC, the AI2 reasoning challenge.
  * Colombo et al. (2020) Pierre Colombo, Emile Chapuis, Matteo Manica, Emmanuel Vignon, Giovanna Varni, and Chloe Clavel. 2020.  Guiding attention in sequence-to-sequence models for dialogue act prediction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7594–7601.
  * Deng et al. (2023) Wentao Deng, Jiahuan Pei, Zhaochun Ren, Zhumin Chen, and Pengjie Ren. 2023.  Intent-calibrated self-training for answer selection in open-domain dialogues.  _Transactions of the Association for Computational Linguistics_ , 11:1232–1249.
  * Deng et al. (2022) Yang Deng, Wenxuan Zhang, Wai Lam, Hong Cheng, and Helen Meng. 2022.  User satisfaction estimation with sequential dialogue act modeling in goal-oriented conversational systems.  In _WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022_ , pages 2998–3008. ACM.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023.  Enhancing chat language models by scaling high-quality instructional conversations.
  * Elasticsearch (2023) Elasticsearch. 2023.  Elasticsearch: The official distributed search & analytics engine.  https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html#bm25h.
  * Ghazarian et al. (2022) Sarik Ghazarian, Behnam Hedayatnia, Alexandros Papangelis, Yang Liu, and Dilek Hakkani-Tur. 2022.  What is wrong with you?: Leveraging user sentiment for automatic dialog evaluation.  In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 4194–4204. Association for Computational Linguistics.
  * Gustavo Penha and Hauff (2019) Alexandru Balan Gustavo Penha and Claudia Hauff. 2019.  Introducing mantis: a novel multi-domain information seeking dialogues dataset.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.
  * Hu et al. (2023) Zhiyuan Hu, Yue Feng, Anh Tuan Luu, Bryan Hooi, and Aldo Lipani. 2023.  Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system.  In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , CIKM ’23, page 3953–3957, New York, NY, USA. Association for Computing Machinery.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022.  Large language models can self-improve.  _arXiv preprint arXiv:2210.11610_.
  * Huggingface (2024) Huggingface. 2024.  Huggingface open llm leaderboard: The huggingface open llm leaderboard aims to track, rank and evaluate open llms and chatbots.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Kim et al. (2022) Gangwoo Kim, Sungdong Kim, Kang Min Yoo, and Jaewoo Kang. 2022.  Generating information-seeking conversations from unlabeled documents.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2362–2378.
  * Kulkarni et al. (2024) Atharva Kulkarni, Bo-Hsiang Tseng, Joel Ruben Antony Moniz, Dhivya Piraviperumal, Hong Yu, and Shruti Bhargava. 2024.  Synthdst: Synthetic data is all you need for few-shot dialog state tracking.  _arXiv preprint arXiv:2402.02285_.
  * Kumar et al. (2019) Harshit Kumar, Arvind Agarwal, and Sachindra Joshi. 2019.  A practical dialogue-act-driven conversation model for multi-turn response selection.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1980–1989.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  TruthfulQA: Measuring how models mimic human falsehoods.
  * Mediawiki (2023) Mediawiki. 2023.  Wikipedia api search: Get request to perform an advanced search for wiki pages by title or content text match.  https://www.mediawiki.org/wiki/API:Search.
  * Penha et al. (2019) Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019.  Introducing mantis: a novel multi-domain information seeking dialogues dataset.  _arXiv preprint arXiv:1912.04639_.
  * Pradeep et al. (2023) Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023.  Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze!  _arXiv preprint arXiv:2312.02724_.
  * Qu et al. (2019a) C. Qu, L. Yang, W. B. Croft, Y. Zhang, J. Trippas, and M. Qiu. 2019a.  User intent prediction in information-seeking conversations.  In _CHIIR ’19_.
  * Qu et al. (2018a) Chen Qu, Liu Yang, W. Bruce Croft, Johanne R. Trippas, Yongfeng Zhang, and Minghui Qiu. 2018a.  Analyzing and characterizing user intent in information-seeking conversations.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, pages 989–992. ACM.
  * Qu et al. (2018b) Chen Qu, Liu Yang, W. Bruce Croft, Johanne R. Trippas, Yongfeng Zhang, and Minghui Qiu. 2018b.  Analyzing and characterizing user intent in information-seeking conversations.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, pages 989–992. ACM.
  * Qu et al. (2019b) Chen Qu, Liu Yang, W. Bruce Croft, Yongfeng Zhang, Johanne R. Trippas, and Minghui Qiu. 2019b.  User intent prediction in information-seeking conversations.  In _Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, CHIIR 2019, Glasgow, Scotland, UK, March 10-14, 2019_ , pages 25–33. ACM.
  * Qu et al. (2023) Renyi Qu, Lyle H. Ungar, and João Sedoc. 2023.  Conditioning on dialog acts improves empathy style transfer.  In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_ , pages 13254–13271. Association for Computational Linguistics.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8689–8696. AAAI Press.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends in Information Retrieval_ , 3(4):333–389.
  * Sahu et al. (2022) Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bahdanau. 2022.  Data augmentation for intent classification with off-the-shelf large language models.  In _Proceedings of the 4th Workshop on NLP for Conversational AI_ , pages 47–57, Dublin, Ireland. Association for Computational Linguistics.
  * Shah et al. (2018) Pararth Shah, Dilek Hakkani-Tür, Gökhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry P. Heck. 2018.  Building a conversational agent overnight with dialogue self-play.  _CoRR_ , abs/1801.04871.
  * Sun et al. (2023) Jiuding Sun, Chantal Shaib, and Byron C Wallace. 2023.  Evaluating the zero-shot robustness of instruction-tuned language models.  _arXiv preprint arXiv:2306.11270_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Transformers (2023) Sentence Transformers. 2023.  multi-qa-mpnet-base-dot-v1: a sentence-transformers model.  https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023.  Zephyr: Direct distillation of lm alignment.
  * Wang et al. (2024) Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2024.  Openchat: Advancing open-source language models with mixed-quality data.  In _The Twelfth International Conference on Learning Representations_.
  * Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  Self-instruct: Aligning language model with self generated instructions.
  * Wu et al. (2023) Zeqiu Wu, Ryu Parish, Hao Cheng, Sewon Min, Prithviraj Ammanabrolu, Mari Ostendorf, and Hannaneh Hajishirzi. 2023.  Inscit: Information-seeking conversations with mixed-initiative interactions.  _Transactions of the Association for Computational Linguistics_ , 11:453–468.
  * Yan et al. (2017) Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017.  Building task-oriented dialogue systems for online shopping.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_ , pages 4618–4626. AAAI Press.
  * Yang et al. (2020) Liu Yang, Minghui Qiu, Chen Qu, Cen Chen, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, and Haiqing Chen. 2020.  Iart: Intent-aware response ranking with transformers in information-seeking conversation systems.  In _Proceedings of The Web Conference 2020_ , WWW ’20, page 2592–2598, New York, NY, USA. Association for Computing Machinery.
  * Yu et al. (2022) Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022.  Generate rather than retrieve: Large language models are strong context generators.  _arXiv preprint arXiv:2209.10063_.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024.  Self-rewarding language models.  _arXiv preprint arXiv:2401.10020_.
  * Zamani et al. (2023) Hamed Zamani, Johanne R Trippas, Jeff Dalton, Filip Radlinski, et al. 2023.  Conversational information seeking.  _Foundations and Trends® in Information Retrieval_ , 17(3-4):244–456.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence?
  * Zhang et al. (2023) Xinyu Zhang, Sebastian Hofstätter, Patrick Lewis, Raphael Tang, and Jimmy Lin. 2023.  Rank-without-gpt: Building gpt-independent listwise rerankers on open-source large language models.  _arXiv preprint arXiv:2312.02969_.
  * Zhao et al. (2023a) Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, and Bing Qin. 2023a.  Is chatgpt equipped with emotional dialogue capabilities?  _arXiv preprint arXiv:2304.09582_.
  * Zhao et al. (2023b) Yukun Zhao, Lingyong Yan, Weiwei Sun, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2023b.  DiQAD: A benchmark dataset for open-domain dialogue quality assessment.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15128–15145, Singapore. Association for Computational Linguistics.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2023.  Lmsys-chat-1m: A large-scale real-world llm conversation dataset.

##  8 Appendix

###  8.1 Intents

In the context of an information-seeking conversation, we refer to intent as
the role or function that an utterance plays in the conversation. It can
include acts like requesting information, giving positive/negative feedback,
requesting further details, or repeating the question. Table 7 shows the list
of 12 unique utterances that exist in the SOLID (ours) and MSDialog datasets.
Intents can be used to help the chatbot understand the context and purpose of
a user’s message and to determine how to respond appropriately.

###  8.2 Details of intent predictors

For implementing the intent predictor model, we fine-tune BERT in an intent
prediction setup with a cross-entropy loss function. For T5, we fine-tune it
for a seq-to-seq generation task with cross-entropy loss with the prefix of
"multilabel classification: " prompted to the input. For cnn-context-rep Qu et
al. (2019a), we utilize the released code by the original paper and used the
most effective reported hyperparameter.

###  8.3 Impact of hallucination

We categorize an entity as likely hallucinated if it cannot be located within
Wikipedia. To facilitate this identification process, we leverage the
Wikipedia search API Mediawiki (2023). It is important to acknowledge that our
methodology, being automated, would generate hallucinated content inevitably.
In Table 8, we examine the influence of hallucinated entities on subsequent
tasks. We assume that an entity is to be hallucinated if it does not exist on
Wikipedia. Surprisingly, the results show that including hallucinated entities
leads to higher effectiveness for intent prediction. We suspect that the task
of intent prediction is mostly about understanding the intent of the text
rather than ensuring factually correctness of the text.

Table 7: Intent taxonomy and distribution in MSDialog

Code | Label | Description | %
---|---|---|---
OQ | Original Question | The first question from the user to initiate the dialog. | 13
RQ | Repeat Question | Other users repeat a previous question. | 3
CQ | Clarifying Question | User or agent asks for clarifications. | 4
FD | Further Details | User or agent provides more details. | 14
FQ | Follow Up Question | User asks for follow up questions about relevant issues. | 5
IR | Information Request | Agent asks for information from users. | 6
PA | Potential Answer | A potential answer or solution provided by agents. | 22
PF | Positive Feedback | User provides positive feedback for working solutions. | 6
NF | Negative Feedback | User provides negative feedback for useless solutions. | 4
GG | Greetings/Gratitude | Greetings or expressing gratitude. | 22
JK | Junk | No useful information in the utterance. | 1
O | Others | Utterances cannot be categorized using other classes. | 1

Table 8: Analyzing impact of hallucination in SOLISpeak as training source on
T5 intent predictor evaluated on MSDialog dataset.

| MSDialog
---|---
| P | F1
|  | Micro | Macro
Only hallucinated | .7295 | .7498 | .6692
Only non-hallucinated | .7289 | .7502 | .6684
Full | .7321 | .7580 | .6710

Table 9: The evaluation results on the Open LLM Leaderboard (Huggingface,
2024), which measures the performance of LMs across four multiclass
classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al.,
2019), MMLU (Hendrycks et al., 2021), and Truthful QA (Lin et al., 2022).
There are 57 tasks for MMLU and a detailed report on the effectiveness of
SOLID-RL on these tasks is shown in Figure 9. The bottom section of the tables
provides various analyses on SOLID-RL including: an ablation study, the impact
of our length-oriented mixed-quality training, and an analysis of whether
multi-task training is useful in this setup. The base model of SOLID-RL-7B is
Zephyr-7B.

Model | Accuracy
---|---
| Truthful QA | Hella Swag | MMLU | ARC
UltraLM-13B Touvron et al. (2023) | 52.00 | 61.32 | 50.45 | 57.25
Zephyr-7B Tunstall et al. (2023) | 57.44 | 84.52 | 61.44 | 62.03
SOLID-RL-7B (Ours) | 58.13 | 83.82 | 61.68 | 60.78
Analysis on SOLID-RL-7B:
W/o ours length-oriented MixedQ training | 55.25 | 80.69 | 58.40 | 59.24
W/o DPO (i.e., only SFT) | 53.77 | 81.72 | 60.47 | 59.73
Reverse chosen/rejected for DPO | 52.36 | 79.82 | 60.14 | 58.96
W/ multi-task training | 57.58 | 82.53 | 59.60 | 59.56

###  8.4 SOLID-RL on LLM benchmarks

We provide an in-depth analysis of the impact of different training strategies
for SOLID-RL on its performance over LLM benchmarks at Table 9. Our results
indicate that excluding length-based mixed-quality training can significantly
harm the general capability of SOLID-RL. Additionally, we highlight the
positive impact of direct preference optimization (DPO) after supervised fine-
tuning (SFT). We further investigated the effect of reversing chosen/rejected
outputs in DPO training by using the output generated by SOLID as the rejected
output during DPO optimization. We find that this setup results in lower
performance for the LLM, confirming the chosen setup in SOLID-RL. We also
experimented with training SOLID-RL alongside Zephyr’s training dataset in a
multi-task setup; however, we found that this led to a decrease in the
effectiveness of the LLM and lower-quality dialogs. Our analysis suggests that
SOLID-RL is the most robust methodology compared to the variations
investigated. Furthermore, to make a deeper analysis, we demonstrate the
performance of SOLID-RL on 57 tasks of the MMLU benchmark in comparison with
Zephyr-7b, as illustrated in Figure 9. To give insight into the MMLU dataset,
an example of a machine learning task question from the MMLU dataset is
presented in Figure 8.

###  8.5 Generalizability of SOLID across various contexts

While we have analyzed the utilization of our method, SOLID, in the English
language, we believe that our method is versatile and can be adapted for
diverse scenarios with minimal modifications: (1) For other languages, by
substituting the LLM with a multilingual version and translating instructions;
(2) Within the same language but for varied intents, by crafting specific
instructions for each new intent; (3) For the same language and intents across
different domains, through adjustments to instructions to cover domain-
specific intents; (4) For different languages and intents, by developing new
instructions for each intent and translating existing ones.

###  8.6 Safeguard Data Leakage

In order to be sure that there is no data leakage between the evaluation
datasets in our experiments and the training set of the Zephyr, we assess in
the Zephyr and Mistral training data documentation Tunstall et al. (2023) if
any of the BEIR collection datasets of MSDialog or MANtIS dataset are involved
in the training dataset of the Mistralai or Zephyr. We made sure that there is
no mutual dataset within the used LLM training data, X Tunstall et al. (2023),
and our evaluation datasets.

###  8.7 Manually crafted instructions per intent given user/agent

There are 24 manually written instructions per intent given user/agent that
can be seen in Tables 11 and Table 10.

Table 10: The last part of the intent-based LLM-instruction. Actor type: Agent

Intent |  Instruction
---|---
CQ |  Reply with one follow-up response in conversation style.
FD |  Reply with further details in conversation style.
GG |  Continue the conversation by expressing gratitude for the user’s questions.
PA |  Provide a potential solution or answer in conversation style.
IR |  Ask the user to provide relevant information needed for their previous question.
OQ |  Formulate an original question posed by an agent.
FQ |  Formulate a follow-up question from an agent, seeking further clarification or information.
RQ |  Now you are talking from the point of view of a third participant in the conversation. Repeat Question:
PF |  Express satisfaction and appreciation for the conversation.
NF |  Convey dissatisfaction for the previous response.
JK |  Reply with gibberish information. It can contain emojis.
O |  Reply with a system error. Return N/A

Table 11: The last part of the intent-based LLM-instruction. Actor type: User

Intent |  Instruction
---|---
CQ |  Reply with one question asking for clarification in conversation style.
FD |  Reply with more details in conversation style.
GG |  Continue the conversation by expressing gratitude for the agent’s help.
PA |  Provide a potential solution or answer in conversation style.
IR |  Reply with relevant information.
OQ |  Formulate the first question posed by a user that initiates a QA dialog.
FQ |  Formulate a follow-up question from a user, seeking further clarification or information.
RQ |  Now you are talking from the point of view of a third participant in the conversation. Repeat Question:
PF |  Express satisfaction and appreciation for a working solution.
NF |  Convey dissatisfaction for the previous response.
JK |  Reply with gibberish information. It can contain emojis.
O |  Reply with a system error. Return N/A

Algorithm 1 Generate dialog

1:Global $globGenInstr$, $dTurn$, $inst\\_dict$

2:$contx$, $convStart$, $intentSeq$

3:$dialog\leftarrow convStart$;

4:for $intnt\in intentSeq$:

5: if $is\\_multiple(intnt)$ then

6: $inst\leftarrow$ GenSelfInstruct($intnt$)

7: else

8: $inst\leftarrow inst\\_dict[intnt]$

9: $tupple\leftarrow$ $(globGenInstr,contx,dialog,inst,dTurn)$

10: $message\leftarrow$ concat($tupple$)

11: $turn\leftarrow$ GenUttr($message$);

12: $turn\leftarrow$ PostProcess($turn$);

13: $dialog\leftarrow$ InsertTurn($dialog,turn$);

14:return $dialog$, $intentSeq$

###  8.8 Generating intent-aware dialogs.

We design one instruction for each intent, which is used during prompting to
guide the LLM generation into conveying an utterance around the intent. We
generate one dialog for each sampled sequence of intents. The dialog
generation process is illustrated in Algorithm 1. The algorithm returns a
multi-turn intent-driven dialog generated by the LLM, given three object-
specific attributes: the entity description $contx$, the conversation starter
$convStart$, and sampled sequence of intents $intentSeq$. Moreover, there are
three dialog-agnostic variables: $globGenInstr$ is a general instruction that
introduces the task to the LLM; $dTurn$ keeps track of the current speaker;
and $inst\\_dict$ maps each intent to its designed instruction, as presented
in Table 11. The values of the first two variables are given below:

  * •

General Instruction: "I will give you an entity, its type, a background
document, and a conversation history that ends in a «question»"

  * •

Dialog Turn: This is used to guide the dialog generation by indicating the
role of the LLM as either User or Agent.

We initialize the dialog with the conversation starter $convStart$, enforcing
each dialog to start with an "Original Question". We prompt the LLM to
generate one dialog turn for each $intnt\in intentSeq$. An intent can be
either simple or complex: simple is an intent $intnt\in$ $keys(inst\\_dict)$;
a complex intent is a combination of two or more intents
$intnt=\\{i_{1},i_{2},...i_{n}\\}\text{, where}\quad
i_{k}\in\text{keys}(inst\\_dict)\text{ and}\quad k\sim\text{Uniform}(1,12)$.
The taxonomy of the 12 simple intents is shown in table 7. For each intent in
the sequence, the algorithm follows these steps:

  * Line 1: Initialize the dialog with the conversation starter.

  * Line 2: Iterate through the sequence of intents.

  * Lines 3-6: Check if intent is simple or complex. If simple, retrieve its instruction from $intnt\\_dict$; Otherwise, merge the instructions for all $i_{k}\in intnt$, and prompt the LLM to self-instruction generation.

  * Lines 7-8: Concatenate the general instruction, entity description, dialog, extracted or generated instruction, and current dialog turn.

  * Line 9: Pass the message to the LLM to generate a new utterance.

  * Line 10: Apply post-processing in three steps: 1) trimming to the last punctuation, as LLMs can reach their max token capacity without ending the last utterance, 2) remove empty lines, and 3) remove the dialog turn keywords.

  * Line 11: Insert the new turn into the dialog. This is a meticulous process, as further processing must be done to remove $globGenInstr$, and the current $instr$ from $dialog$; Afterwards we append $turn\\_post$ to the conversation history.

  * Line 12: Return both the dialog and sequence of intents.

###  8.9 Prompts

####  8.9.1 Prompt for building self-instruct instruction

Where an utterance needed to have more than one intent, we ask llm itself to
generate an instruction for that considering instruction of each single
intent. The prompt is shown in Figure 5

####  8.9.2 Prompt for generating entity type

We utilize a simple prompt asking ‘Provide a list of 100 entity types.’ to
generate entity types.

###  8.10 Prompts for generating entities

We empirically figured out that this task cannot be addressed effectively
without providing a one-shot example to the LLM. Therefore, we provide one
example in the prompt. In our example, we show 10 examples of generated
entities, but we ask LLM to generate 100 entities. We figured out that in this
way the LLM follows to generalize to generate more entities in the same
format. However, we also automatically exclude lines with more than 20
characters and also remove special characters. The used prompt is shown in
Figure 6.

####  8.10.1 Prompt for generating conversation starters

The prompt for generating a conversation starter is shown in Figure 7.

####  8.10.2 Prompt for generating intent-aware conversation

We combine the base prompt with each designed instruction per intent. The main
prompt is. ‘I will give you an entity, its type, and a background document,
and a conversation history that ends in a’. Each time, we have the
conversation history, entity and its type, and prompt LLM to generate the next
utterance based on that specific utterance. We only do one-way forward pass
for generating dialog. We believe we could increase the quality of the data by
going one backward-pass from the end to the first of dialogs and now ask LLM
to rewrite each utterance in a way that the next utterance of them needs to
have the intent that is assigned to it already.

Figure 5: Prompt for generating self-instruction to generate utterance with
multiple intent Figure 6: Prompt for generating entity type Figure 7: Prompt
for generating conversation starter

A 6-sided die is rolled 15 times and the results are: side 1 comes up 0 times;
side 2: 1 time; side 3: 2 times; side 4: 3 times; side 5: 4 times; side 6: 5
times. Based on these results, what is the probability of side 3 coming up
when using Add-1 Smoothing? (A) 2/15 (B) 1/7 (C) 3/16 (D) 1/5

Figure 8: A Machine Learning example of MMLU benchmark Hendrycks et al.
(2020). Table 12: Diverse set of 98 entity types generated via SOLID.

Person |  Historical Event |  Beverage |  Galaxy |  Author |  Deity
---|---|---|---|---|---
Book |  Constellation |  Artist |  Mythology |  Song |  Astronomical Object
Politician |  Mythical Creature |  Animal |  Natural Disaster |  Actor |  Supernatural Being
Game |  Weather Phenomenon |  Musician |  Character (Fictional) |  Gadget |  Disease
Celebrity |  Organization |  Software |  Medication |  Actress |  Company
App |  Medical Procedure |  Athlete |  Government Agency |  Website |  Law
Historical Figure |  Nonprofit Organization |  Social Media Platform |  Legal Case |  Entrepreneur |  Hospital
Device |  Political Ideology |  Inventor |  University |  Instrument |  Social Movement
Scientist |  Event |  Tool |  Philosophy |  Mathematician |  School
Furniture |  Religion |  Philosopher |  Place |  Clothing |  Folklore
Explorer |  Country |  Artwork |  Language |  Poet |  City
Painting |  Brand |  Photographer |  Architectural Structure |  Sculpture |  Fashion Brand
Journalist |  Festival |  Cuisine |  Style (Fashion, Art) |  Activist |  Conference
Recipe |  Genre (Music, Film) |  Sportsperson |  Service |  Scientific Concept |  Technology
Sport |  Product |  Chemical Element |  Programming Language |  Team |  Food
Particle |  Movie |  League |  Vehicle |  Planet |  Historical Period
Plant |  Star |  |  |  |

Figure 9: Effectiveness of SOLID-RL-7B vs. Zephyr-7B in terms of accuracy over
tasks of Measuring Massive Multitask Language Understanding (MMLU) benchmark
Hendrycks et al. (2020).

###  8.11 Self-seed generation process

Figure 10 presents a detailed illustration of the self-seed generation
process. Additionally, Table 12 provides insights on how including 98 distinct
entity types ensures the diversity of our dataset.

Figure 10: Comprehensive depiction of the self-seed generation process in
SOLID, illustrated through examples and prompts, alongside a summary of
intent-aware dialog generation leveraging these self-seeds.