# Prompting and Evaluating Large Language Models for Proactive Dialogues:
Clarification, Target-guided, and Non-collaboration

Yang Deng1, Wenqiang Lei2, Lizi Liao3, Tat-Seng Chua1
1National University of Singapore, 2Sichuan University,
3Singapore Management University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Conversational systems based on Large Language Models (LLMs), such as ChatGPT,
show exceptional proficiency in context understanding and response generation.
However, despite their impressive capabilities, they still possess
limitations, such as providing randomly-guessed answers to ambiguous queries
or failing to refuse users’ requests, both of which are considered aspects of
a conversational agent’s proactivity. This raises the question of whether LLM-
based conversational systems are equipped to handle proactive dialogue
problems. In this work, we conduct a comprehensive analysis of LLM-based
conversational systems, specifically focusing on three aspects of proactive
dialogue systems: clarification, target-guided, and non-collaborative
dialogues. To trigger the proactivity of LLMs, we propose the Proactive Chain-
of-Thought prompting scheme, which augments LLMs with the goal planning
capability over descriptive reasoning chains. Empirical findings are discussed
to promote future studies on LLM-based proactive dialogue systems.

##  1 Introduction

Conversational systems are envisioned to provide social support or functional
service to human users via natural language interactions. Conversation
researches typically center around a system’s response capabilities, such as
understanding the context of dialogue Wu et al. (2020); Chen et al. (2022) and
generating appropriate responses to user requests Zhang et al. (2020b); Roller
et al. (2021). The popularity of conversational systems has grown
unprecedentedly with the advent of ChatGPT, which showcases exceptional
proficiency in the capabilities of context understanding and response
generation with large language models (LLMs). Recent studies observe that,
compared with current fine-tuned methods, ChatGPT can still achieve
competitive performance under zero-shot setting on different dialogue
problems, such as knowledge-grounded dialogues Bang et al. (2023), task-
oriented dialogues Zhang et al. (2023), emotion-aware dialogues Zhao et al.
(2023).

Despite the powerfulness of ChatGPT, there are still several limitations111as
stated in its official blog https://openai.com/blog/chatgpt/., such as
providing randomly-guessed answers to ambiguous user queries or failing to
refuse problematic user requests. These kinds of capabilities are typically
regarded as the proactivity of the conversational system Deng et al. (2023),
where the system can create or control the conversation to achieve the
conversational goals by taking initiative and anticipating impacts on
themselves or human users. Thus, it raises the question: Are these LLM-based
conversational systems equipped to manage proactive dialogue problems? In this
work, we conduct the first comprehensive analysis of LLM-based conversational
systems regarding three common aspects of proactive dialogue systems,
including clarification Guo et al. (2021); Deng et al. (2022), target-guided
Tang et al. (2019); Wu et al. (2019), and non-collaborative dialogues Zhan et
al. (2022).

Figure 1: Examples of three kinds of prompting schemes for proactive
dialogues. In the example of non-collaborative dialogue, the system plays the
role of "Buyer", and the sale-to-list (SL) ratio shows the effectiveness of
negotiation, which is calculated by ($\text{listed price}-\text{bargain
price})/(\text{listed price}-\text{buyer target price})$. The higher ratio
means the current bargain price is closer to the target.

Motivated by the emergent capabilities of LLMs Wei et al. (2022a, b) on
reasoning over texts, some recent studies investigate in-context learning or
chain-of-thought prompting schemes on planning Huang et al. (2022) or taking
actions Yao et al. (2022) in interactive environments. Similarly, strategy
learning and goal planning attach great importance in proactive dialogue
systems. In order to enhance the proactivity of LLM-based conversational
systems, we design the proactive chain-of-thought prompting (ProCoT) scheme.
As shown in Figure 1, with standard prompting, LLM-based systems directly
provide a randomly-guessed answer to the ambiguous user question (left), and
generate a general bargain response without any negotiation strategy (right).
When providing the system with options to take different dialogue acts
(proactive prompting), the generated responses are unaware of the
conversational goal, such as generating under-specified clarification
questions (left) and conservative negotiation responses (right). To this end,
ProCoT first instructs the system to generate descriptive thoughts about
intermediate steps of reasoning and planning for reaching the conversational
goal, and then make the decision of the next action to take. Finally, the
system generates an appropriate response based on the decided action.

We conduct extensive experiments with two LLM-based conversational systems,
including ChatGPT and an open-sourced model, Vicuna Chiang et al. (2023). With
the aforementioned three types of prompting schemes, we compare these LLM-
based conversational systems with fine-tuned SOTA dialogue models. The main
contributions of this work can be summarized as follows:

  * •

This work presents the first comprehensive evaluation on the proactivity of
LLM-based dialogue systems222https://github.com/dengyang17/LLM-Proactive,
including handling clarification, target-guided, and non-collaborative
dialogues.

  * •

We design the proactive chain-of-thought prompting scheme to endow LLM-based
dialogue systems with the capability of planning and taking the initiative
towards the conversational goal.

  * •

Specifically, the main findings of the evaluation on LLM-based dialogue
systems include 1) Barely ask clarification questions when encountering
ambiguous queries. ProCoT largely overcomes this issue, but the performance is
still unsatisfactory in domain-specific applications ($\S\ref{sec:clari}$). 2)
Proficient at performing topic shifting towards the designated target, but
tend to make aggressive topic transition. ProCoT further improves this
capability by planning a more smooth transition. ($\S\ref{sec:target}$) 3)
Fail to make strategic decision for non-collaborative dialogues, even with
ProCoT prompting ($\S\ref{sec:non-colla}$).

##  2 Related Works

##### Proactive Dialogues

Recent years have witnessed many advanced designs on developing proactive
dialogue systems Liao et al. (2023) for various applications. For example,
target-guided dialogues aim to proactively lead the conversation to either a
designated target topic Tang et al. (2019) or a pre-defined knowledge entity
Wu et al. (2019). Existing studies typically adopt keyword transition Qin et
al. (2020); Zhong et al. (2021) or knowledge graph reasoning Yang et al.
(2022); Lei et al. (2022) techniques to proactively plan the topic thread
towards the target. Besides, in information-seeking dialogues, proactive
dialogue systems can ask clarification questions for clarifying the ambiguity
of the query or question in conversational search and recommendation
Aliannejadi et al. (2021); Deng et al. (2021) and conversation question
answering Guo et al. (2021); Deng et al. (2022). In addition, under non-
collaborative setting, the system and the user have competing goals towards
the task completion but the system aims to proactively reach an agreement
favorable to itself Zhou et al. (2020b), such as negotiating a product price
He et al. (2018) and persuading users to make a donation Wang et al. (2019).

##### Large Language Models for Dialogues

Previous dialogue systems, such as DialoGPT Zhang et al. (2020b), Meena
Adiwardana et al. (2020), BlenderBot Roller et al. (2021), LaMDA Thoppilan et
al. (2022), typically fine-tune pre-trained language models on public dialogue
data. Inspired by the success of ChatGPT, recent practices of building
dialogue systems through conducting supervised fine-tuning on open-source
large language models, such as LLaMA Touvron et al. (2023), with either
constructed instruction-following examples or distilled conversation data from
ChatGPT. For example, Alpaca Taori et al. (2023) adopts Self-Instruct Wang et
al. (2022) to collect instruction-following examples from GPT-3.5 Ouyang et
al. (2022), while Dolly333https://www.databricks.com/blog/2023/04/12/dolly-
first-open-commercially-viable-instruction-tuned-llm relies on a large-scale
human-annotated instruction-following examples. Vicuna Chiang et al. (2023)
and Baize Xu et al. (2023) leverage use-shared or self-chat conversation data
that is generated by ChatGPT. As all these LLM-based dialogue systems are
trained to follow the user’s instruction, it remains the question on whether
these systems can take the initiative for handling proactive dialogues.

##### Prompting in Dialogue Systems

To induce the knowledge from PLMs, various prompting methods are designed for
zero-shot or few-shot learning for dialogue applications, such as task-
oriented dialogues Lee et al. (2021); Mi et al. (2022), knowledge-grounded
dialogues Shuster et al. (2022); Liu et al. (2022), and open-domain dialogues
Chen et al. (2023a); Lee et al. (2023). Chen et al. (2023b) propose to prompt
LLMs for controllable response generation in emotional support and persuasion
dialogues, conditioned on the ground-truth dialogue strategies. In this work,
we aim at prompting LLMs to proactively interact with users.

##  3 Prompting LLMs to be Proactive

We describe the prompting schemes in a general form, including the standard
prompting, the proactive prompting, and the proactive chain-of-thought
(ProCoT) prompting. Two specific examples are presented in Figure 1.

##### Standard Prompting

In order to instruct LLMs to perform specific dialogue tasks, the typical
prompting scheme can formulated as

$p(r|\mathcal{D},\mathcal{C}).$ (1)

Given the task background $\mathcal{D}$ and the conversation history
$\mathcal{C}$, instruct the LLM to generate the response $r$. In specific, the
task background can be the grounded document in clarification dialogues or the
target description in target-guided dialogues.

##### Proactive Prompting

Proactive prompting aims to provide alternative options for LLMs to decide
what kinds of actions should be taken in the response, instead of simply
responding to the instruction, which can be formulated as:

$p(a,r|\mathcal{D},\mathcal{C},\mathcal{A}).$ (2)

Given the task background $\mathcal{D}$, the conversation history
$\mathcal{C}$, and a set of possible dialogue acts $\mathcal{A}$, instruct the
LLM to select the most appropriate dialogue act $a\in\mathcal{A}$ and then
generate the response $r$. For example, the dialogue act can be Ask a
Clarification Question or Directly Answer the Question in clarification
dialogues. Besides, it can also be different negotiation strategies in non-
collaborative dialogues or different conversation topics in target-guided
dialogues.

##### Proactive Chain-of-Thought Prompting

In order to endow LLMs with the capability of planning and taking the
initiative towards the ultimate goal, we develop the proactive chain-of-
thought prompting scheme. ProCoT first analyse the next action to take by
performing dynamic reasoning and planning for reaching the conversational
goal. Then the response is generated based on the decided action. ProCoT can
be formulated as:

$p(t,a,r|\mathcal{D},\mathcal{C},\mathcal{A}),$ (3)

where $t$ is the thought description for the decision-making process of the
next action. For example, $t$ can be the ambiguity analysis of the user
question at the current turn in clarification dialogues or the topic
transition analysis of the current topic in target-guided dialogues.

Abg-CoQA∗ PACIFIC∗∗ Method Shot Prompt P R F1 BLEU-1 Human P R F1 ROUGE-2
Human Baseline - - 19.0 26.6 22.1 36.5 30.0 78.7 79.2 79.0 69.2 38.2 SOTA - -
30.0 19.5 23.6 38.2 56.0 87.4 86.6 86.9 90.7 80.1 Vicuna-13B 0 Standard - - -
11.3 0.0 - - - 1.2 0.0 1 Standard - - - 11.4 0.0 - - - 2.5 0.0 0 Proactive
13.0 2.4 4.1 13.2 0.0 13.8 1.3 2.3 2.3 0.0 1 Proactive 16.0 9.8 12.1 13.2 4.5
0.0 0.0 0.0 3.3 0.0 0 ProCoT 6.7 0.8 1.4 21.3 9.1 26.8 5.9 9.7 3.8 10.5 1
ProCoT 14.4 25.2 18.3 23.7 22.7 20.2 40.9 27.0 41.3 33.1 ChatGPT 0 Standard -
- - 12.1 0.0 - - - 2.2 0.0 1 Standard - - - 12.3 0.0 - - - 2.0 0.0 0 Proactive
15.1 50.7 22.0 13.7 17.6 18.2 20.9 19.4 2.9 0.0 1 Proactive 27.4 16.3 20.4
23.4 23.5 19.1 16.6 17.7 14.0 12.5 0 ProCoT 13.8 87.8 23.8 21.6 32.4 17.9 63.8
28.0 21.5 26.7 1 ProCoT 17.6 66.7 27.9 18.4 45.9 18.7 54.1 27.7 16.2 35.8

Table 1: Experimental results on Abg-CoQA and PACIFIC datasets. ∗Baseline and
SOTA results are adopting from Guo et al. (2021). ∗∗The baseline method is
fine-tuned T5, while the SOTA method is UniPCQA Deng et al. (2022).

##  4 Evaluation

We evaluate the proactivity of LLM-based conversational systems from three
perspectives, including the capability of asking clarification questions ($\S$
4.1), guiding the conversation towards the designated target ($\S$ 4.2), and
strategically handling conflicting goals ($\S$ 4.3).

###  4.1 Clarification Dialogues

Clarification in information-seeking dialogues Zamani et al. (2022) refers to
the process of seeking further information or details to better understand the
topic or question at hand. In this context, clarification is an important part
of the dialogue as it helps to ensure that the information being shared is
accurate and complete.

####  4.1.1 Problem Definition

Following previous studies Aliannejadi et al. (2021); Guo et al. (2021); Deng
et al. (2022), the problem of asking clarification questions can be decomposed
into two subtasks: 1) Clarification Need Prediction to identify the necessity
of clarification in the current turn and 2) Clarification Question Generation
to produce an appropriate clarifying question if needed. Given the grounded
document $d$ and the dialogue context
$\mathcal{C}=\\{q_{1},a_{1},...,q_{t-1},a_{t-1},q_{t}\\}$, the dialogue system
aims to first predict the binary ambiguity label $y$ on whether the current
question $q_{t}$ needs to be clarified. If so, a corresponding clarification
question should be generated as the response $a_{t}$ for clarifying the
ambiguity.

####  4.1.2 Experimental Setups

##### Datasets

Two datasets are adopted for evaluating the capability of asking clarification
questions in LLM-based dialogue systems: 1) Abg-CoQA Guo et al. (2021) and 2)
PACIFIC Deng et al. (2022). Detailed descriptions can be found in Appendix A.

##### Evaluation Metrics

Following previous studies Guo et al. (2021); Deng et al. (2022), we use
Precision, Recall, and F1 for the evaluation of Clarification Need Prediction,
and BLEU-2 and ROUGE-2 (F1) for the evaluation of Clarification Question
Generation. In addition, since the automatic lexical matching metrics may fail
to actually estimate the clarification capability of the generated clarifying
questions Guo et al. (2021), we also adopt human evaluation to score whether
the generated question is helpful for clarifying the existing ambiguity.

##### Usage of LLMs

In order to facilitate the reproducibility, we adopt a static version of
ChatGPT, i.e., gpt-3.5-turbo-0301, and set the temperature to 0 for generating
deterministic outputs under the same inputs. In addition, we also adopt an
open-source LLM-based conversational system, i.e.,
Vicuna-13B-delta-v1.1444https://github.com/lm-sys/FastChat, for evaluation.
The maximum number of new tokens is set to be 128 for generation.

##### Prompting Schemes

We evaluate three prompting schemes introduced in Section 3, including
standard prompting, proactive prompting and proactive chain-of-thought
prompting. In addition, we report their results under both zero-shot and few-
shot settings. Due to the limitation of the maximum sequence length in Vicuna
(2,048 tokens), we only apply one-shot in-context learning for comparisons.
The complete prompts adopted for evaluation is presented in Appendix D.

####  4.1.3 Experimental Results

Table 1 summarizes the evaluation results on Abg-CoQA and PACIFIC datasets.
There are several notable observations as follows:

  * •

Standard Prompting: 1) ChatGPT never asks clarification questions when
encountering ambiguous queries. 2) One-shot in-context learning (ICL) cannot
provide them with such ability.

  * •

Proactive Prompting: 1) Given the option of clarification, Vicuna still barely
take this action, while ChatGPT becomes capable of asking clarification
questions. 2) One-shot ICL further improves the performance of proactive
prompting.

  * •

ProCoT prompting: 1) ChatGPT achieves competitive performance with SOTA fine-
tuned methods on the open-domain problem, i.e., Abg-CoQA. 2) The performance
on the domain-specific task, i.e., PACIFIC (finance), is still far behind the
fine-tuned method. 3) Zero-shot ProCoT is not working in Vicuna, but one-shot
ICL can largely improve the performance.

Overall, LLM-based conversational systems fail to ask clarification questions
if there is no related instruction, even with demonstration. ProCoT
effectively endows LLM-based conversational systems the capability of asking
clarification questions so that they can achieve competitive performance as
fine-tuned SOTA methods on the task in general domain. However, as for domain-
specific problem, there is still a noticeable gap from the fine-tuned methods.

####  4.1.4 Error Analysis

In order to find out the reason why LLM-based dialogue systems with ProCoT
prompting fall short of handling domain-specific clarification dialogues,
i.e., PACIFIC in finance domain, we randomly sample 100 error cases from each
dataset for analysis (all cases are generated by ChatGPT with one-shot
ProCoT). We categorize these failure cases into five groups, including Wrong
Clarification Need Prediction, Wrong Aspect, Under-specified Clarification,
Over-specified Clarification, and Generation Error. The details and examples
can be found in the Appendix B. The statistics of error analysis is presented
in Table 2. It can be observed that there are more failure cases attribute to
the wrong aspect and under-specified clarification in PACIFIC. This indicates
that ChatGPT may lack of some domain knowledge required for asking precise and
specific clarification questions.

Abg-CoQA PACIFIC Wrong Clari. Need Pred. 52% 40% Wrong Aspect 10% 18% Under-
spec. Clari. 8% 14% Over-spec. Clari. 7% 3% Generation Error 23% 26%

Table 2: Statistics of error analysis.

###  4.2 Target-guided Dialogues

Instead of making consistent responses to the user-oriented topics, the
dialogue system for target-guided dialogues is required to proactively lead
the conversation topics towards a designated target Tang et al. (2019).
According to different applications, the target can be topical keywords Zhong
et al. (2021), knowledge entities Wu et al. (2019), items to recommended Zhou
et al. (2020a), etc.

####  4.2.1 Problem Definition

Given a target $t$ that is only presented to the agent and is unknown to the
user, the dialogue begins from an arbitrary initial topic, and the system
needs to produce multiple turns of responses $\\{u_{n}\\}$ to lead the
conversation towards the target in the end. The produced responses should
satisfy (i) transition smoothness, natural and appropriate content under the
given dialogue history, and (ii) target achievement, driving the conversation
to reach the designated target. The problem is typically decomposed into two
subtasks Tang et al. (2019); Zhong et al. (2021); Yang et al. (2022): next
topic selection and transition response generation.

####  4.2.2 Experimental Setups

##### Datasets

We first conduct turn-level evaluation of the target-guided capability on a
next-turn target-oriented dataset OTTers Sevegnani et al. (2021), which
requires the dialogue system to pro-actively bridge the current conversation
topic to approach the target. Furthermore, we adopt TGConv Yang et al. (2022)
to testify the ability to guide the user to the target topic in multi-turn
conversations as the dialogue-level evaluation. Detailed descriptions can be
found in Appendix A.

##### Evaluation Metrics

Following previous studies Sevegnani et al. (2021); Yang et al. (2022), we
adopt the hits@$k$ ($k\in[1,3]$) for evaluating next topic prediction, and
ROUGE-L, METEOR, CIDEr scores for the evaluation of response generation on the
Otters dataset. As for the dialogue-level evaluation on the TGConv dataset, we
follow existing studies Yang et al. (2022); Wang et al. (2023) to simulate
multi-turn conversations via self-play Tang et al. (2019), where the simulated
user is unaware of the target topic. Three aspects are evaluated: 1) Succ. is
the success rate of generating the target word within 8 turns of
conversations; 2) Turns is the average turns of all dialogues that
successfully reach the target word; and 3) Coh. is the contextual semantic
similarity between the last utterance and the generated response, which is
measured by MiniLM Wang et al. (2020).

##### Baselines

We report the results of several strong fine-tuned baseline methods for
target-guided dialogues, including GPT-2 Radford et al. (2019), MultiGen Ji et
al. (2020), DKRN Qin et al. (2020), CKC Zhong et al. (2021), TopKG Yang et al.
(2022), and Color Wang et al. (2023).

####  4.2.3 Turn-level Evaluation

Response Generation Next Topic Prediction Method Shot Prompt BLEU METEOR R-L
hits@1 hits@3 GPT2 - - 11.58 10.26 17.67 4.39 15.79 MultiGen - - 13.57 12.51
26.27 6.58 20.51 DKRN - - 12.86 11.90 21.52 4.91 17.72 CKC - - 13.34 11.65
24.77 6.87 21.89 TopKG - - 15.35 13.41 27.16 7.78 22.06 Vicuna-13B 0 Standard
10.01 13.27 16.00 12.01 19.03 1 Standard 10.63 14.81 17.53 12.10 16.13 0
Proactive 1.41 18.45 15.45 9.41 19.89 1 Proactive 13.87 20.96 21.36 12.90
22.31 0 ProCoT 5.27 16.59 15.96 11.56 18.01 1 ProCoT 13.38 19.70 20.62 15.05
20.70 ChatGPT 0 Standard 11.34 20.62 18.26 13.44 27.69 1 Standard 14.41 19.29
17.73 15.86 26.34 0 Proactive 14.09 21.06 15.56 7.53 22.58 1 Proactive 14.74
19.59 16.29 8.60 21.23 0 ProCoT 10.20 19.57 15.97 12.63 23.92 1 ProCoT 9.63
19.82 17.19 17.74 29.57

Table 3: Turn-level evaluation results on Next Topic Prediction and Transition
Response Generation.

Table 3 summarizes the turn-level evaluation results on the OTTers dataset.
There are several notable observations as follows:

  * •

Standard Prompting: 1) As for the next-topic prediction, ChatGPT has already
achieved better performance than fine-tuned methods with a noticeable margin.
2) As for the transition response generation, automatic evaluation metrics
indicate close performance with fine-tuned methods on the lexical similarity
with the reference response.

  * •

Proactive Prompting: It is effective in smaller-size LLM, i.e., Vicuna, but
not in ChatGPT.

  * •

ProCoT Prompting: 1) ProCoT is effective in both Vicuna and ChatGPT. 2) One-
shot ICL further improves the performance of ProCoT prompting on target-guided
topic shifting.

####  4.2.4 Dialogue-level Evaluation

Easy Target Hard Target Method Shot Prompt Succ.(%) Turns Coh. Succ.(%) Turns
Coh. GPT2 - - 22.3 2.86 0.23 17.3 2.94 0.21 MultiGen - - 26.7 2.55 0.21 19.6
7.31 0.24 DKRN - - 38.6 4.24 0.33 21.7 7.19 0.31 CKC - - 41.9 4.08 0.35 24.8
6.88 0.33 TopKG - - 48.9 3.95 0.31 27.3 4.96 0.33 Color - - 66.3 - 0.36 30.1 -
0.35 Vicuna-13B 0 Standard 63.0 2.63 0.43 62.5 2.45 0.39 1 Standard 62.7 2.83
0.45 65.0 2.90 0.43 0 Proactive 37.8 2.71 0.48 35.6 2.56 0.55 1 Proactive 48.3
2.71 0.50 34.6 2.95 0.51 0 ProCoT 65.2 4.22 0.49 54.9 4.17 0.45 1 ProCoT 72.3
3.55 0.52 59.8 3.81 0.48 ChatGPT 0 Standard 97.5 2.26 0.38 96.3 2.30 0.41 1
Standard 96.3 2.42 0.42 93.5 2.28 0.38 0 Proactive 85.9 3.20 0.47 83.0 2.83
0.43 1 Proactive 90.7 2.86 0.36 86.2 2.94 0.31 0 ProCoT 96.3 2.47 0.41 92.0
2.29 0.34 1 ProCoT 95.9 2.63 0.45 92.1 2.47 0.39

Table 4: Dialogue-level evaluation results on Target-guided Open-domain
Dialogues. Note that "Turns" and "Coh." not the higher/lower the better.

Negotiation Strategies Dialogue Acts Response Generation F1 ROC AUC F1 ROC AUC
BERTScore Method Shot Prompt Macro Micro Weighted Macro Micro Weighted Macro
Micro Weighted Macro Weighted BLEU P R F1 FeHED - - 17.6 25.6 36.3 55.8 61.7
54.7 20.6 37.4 30.6 76.9 79.2 23.7 27.1 26.8 27.0 HED+RNN - - 23.2 26.7 42.4
65.3 65.3 60.4 33.0 46.2 42.8 83.1 84.2 22.5 22.9 22.7 22.8 HED+TFM - - 26.3
32.1 43.3 68.2 71.8 61.8 32.5 44.6 42.0 85.6 85.1 24.4 27.4 28.1 27.7
DialoGraph - - 26.1 34.1 43.5 68.1 73.0 61.8 33.4 45.8 43.7 85.6 85.4 24.7
27.8 28.3 28.1 Vicuna-13B 0 Standard - - - - - - - - - - - 1.7 -28.9 1.7 -14.0
1 Standard - - - - - - - - - - - 1.9 -3.1 -2.0 -2.8 0 Proactive 20.6 25.2 39.6
51.1 48.2 49.8 4.2 18.5 8.4 50.3 49.8 2.3 -6.1 -7.0 -7.0 1 Proactive 15.2 21.0
26.0 50.0 48.8 49.5 6.7 12.0 11.4 50.8 51.3 2.6 -10.3 8.9 -0.9 0 ProCoT 19.0
24.0 38.5 49.7 47.4 49.3 3.6 13.5 7.0 50.3 49.4 2.6 -7.5 -4.1 -6.2 1 ProCoT
17.8 23.8 31.9 48.9 50.0 49.0 7.7 14.0 13.9 52.5 52.2 2.6 -9.0 7.6 -0.9
ChatGPT 0 Standard - - - - - - - - - - - 2.3 -16.4 8.3 -4.3 1 Standard - - - -
- - - - - - - 3.1 -3.4 6.9 0.7 0 Proactive 12.8 19.2 19.6 51.3 49.3 50.3 13.3
29.7 19.5 56.3 60.0 4.2 -4.3 7.3 1.3 1 Proactive 13.7 22.4 20.8 50.9 51.6 51.2
12.0 26.1 17.6 54.9 58.0 3.9 -4.3 10.4 2.9 0 ProCoT 10.8 17.5 16.0 50.4 47.5
50.6 10.1 26.2 16.8 54.2 57.7 3.7 -0.2 -0.9 -0.9 1 ProCoT 15.1 22.8 22.9 55.5
52.5 53.1 16.3 33.3 24.4 58.2 62.8 3.9 -7.1 10.5 1.6

Table 5: Evaluation results on Negotiation Strategy Prediction, Dialogue Act
Prediction, and Response Generation.

Table 4 summarizes the dialogue-level evaluation results on the TGConv
dataset. Results show an overwhelming capability of ChatGPT for controllable
response generation, since the target topics are almost achieved in two turns,
which means that ChatGPT will aggressively generate the response with the
target topic without planning a smooth multi-turn conversation. As for Vicuna,
we draw the following observations:

  * •

Standard Prompting: 1) LLM-based dialogue systems can achieve a high success
rate of reaching the designated target using standard prompting. 2) Similar to
ChatGPT, the target is reached averagely in two turns, which means that the
system tends to directly generate the response with the target topic. 3) The
coherence score is relatively low, indicating the topic transition is
aggressive.

  * •

Proactive Prompting: Although it improves the coherence of the generated
responses, the success rate is quite low and the systems still tend to make a
direct topic transition to the designated target.

  * •

ProCoT Prompting: With ProCoT, one-shot Vicuna effectively outperform all the
fine-tuned SOTA methods for successfully guiding the conversation towards the
designated target with a more smooth and interactive conversation.

Overall, LLM-based dialogue systems are proficient at performing topic
shifting towards the designated target. However, when using standard
prompting, these systems tend to make aggressive topic transition, as they
possess powerful capability of controllable generation. ProCoT prompting
enables a more smooth topic transition of target-guided dialogues with LLMs.

###  4.3 Non-collaborative Dialogues

Unlike collaborative dialogue settings, where the user and the system work
together to reach a common goal (e.g., booking hotels), in non-collaborative
dialogues, the user and the system have a conflict of interest but aim to
strategically communicate to reach an agreement (e.g., negotiation) Zhan et
al. (2022). Therefore, the system is required to leverage a series of
proactive dialogue strategies to reach an agreement favorable to itself,
instead of passively following the user’s intents.

####  4.3.1 Problem Definition

Given the dialogue history $\mathcal{C}=\\{u_{1},...,u_{t-1}\\}$ and the
dialogue background $d$, the goal is to generate a response $u_{t}$ with
appropriate dialogue strategy $s_{t}$ that can lead to a consensus state
between the system’s and the user’s goal. A set of dialogue strategies
$\mathcal{S}$ is pre-defined for prediction. Based on different applications,
the dialogue strategy can be coarse dialogue act labels or fine-grained
strategy labels. The dialogue background includes the system’s goal and the
related grounded information that can be like item descriptions in bargain
negotiation He et al. (2018) or user profile in persuasion dialogues Wang et
al. (2019).

####  4.3.2 Experimental Setups

##### Datasets

We use the CraigslistBargain dataset He et al. (2018) for evaluating the
capability of strategically handling non-collaboration in LLM-based dialogue
systems. The dataset was created under the bargain negotiation setting where
the buyer and the seller are negotiating the price of an item on sale.
Detailed descriptions can be found in Appendix A.

Figure 2: Heatmaps on the relationships between target and predicted dialogue
acts. As no dialogue act is predicted in standard prompting, a dialogue act
classifier is trained to identify the dialogue act of the generated response.
Figure 3: Distribution of selected negotiation strategies. Similarly, a
negotiation strategy classifier is trained to identify the negotiation
strategies of the generated response in standard prompting.

##### Evaluation Metrics

Following the previous study Joshi et al. (2021), we conduct a comprehensive
evaluation over three subtasks, including negotiation strategy prediction,
dialogue act prediction, and response generation. We report the F1 and ROC AUC
scores for strategy prediction and dialogue act prediction, where the former
one is a multi-label prediction problem. For the response generation, we adopt
BLEU score and BERTScore Zhang et al. (2020a) for evaluation.

##### Usage of LLMs & Prompting Schemes

The adopted LLMs are the same, but the maximum number of new tokens is set to
be 256, as there are more information needed to be generated, including
negotiation strategies and dialogue acts.

##### Baselines

We compare several fine-tuned SOTA baselines for negotiation dialogues,
including FeHED Zhou et al. (2020b), HED+RNN/TFM, and DialoGraph Joshi et al.
(2021).

####  4.3.3 Experimental Results

Table 5 summarizes the experimental results on the CraigslistBargain dataset.
Experimental results show that LLM-based dialogue systems fail to predict
appropriate negotiation strategies and dialogue acts in non-collaborative
dialogues, further resulting in a low performance of response generation. Chen
et al. (2023b) empirically show that, given the optimal planned strategy,
ChatGPT achieves strong performance on controllable response generation in
strategy-based dialogues. Drawing upon these findings, the key challenge of
LLMs in handling non-collaborative dialogues is how to effectively optimize
the strategy planning.

####  4.3.4 Analysis of Strategy Learning

Figure 2 presents the analysis of the relationships between target and
predicted dialogue acts by ChatGPT with three types of prompting schemes. As
for the standard prompting, we observe three typical mistakes: 1) The system
tends to propose the initial bargain price (init-price) instead of greetings
(intro). 2) The system often directly accepts the buyer’s offer (accept) when
it is supposed to offer another price for negotiation (offer). On the other
hand, Proactive and ProCoT prompting share similar patterns of mistakes, where
ChatGPT tends to propose a counter price (counter-price) to negotiate with the
buyer.

Figure 3 presents the analysis of the distribution of selected strategies by
ChatGPT. In the reference responses, the seller often shows positive/negative
sentiment to negotiate with the buyer. However, ChatGPT inclines to adopt
conservative or concessionary strategies, such as using hedge words, show
gratitude, or propose a counter price.

Overall, we conclude that ChatGPT tends to make compromise with the buyer
during the negotiation, rather than strategically taking actions to maximize
its own benefit.

##  5 Conclusion

In this work, we conduct the first comprehensive evaluation on the capability
of LLM-based dialogue systems in handling proactive dialogues, including
clarification, target-guided, and non-collaborative dialogues. To enhance the
proactivity of LLM-based dialogue systems, we propose a proactive chain-of-
thought prompting scheme that triggers the reasoning and planning capability
of LLMs. The empirical analysis sheds light on the potentials of LLM-based
dialogue systems for proactive dialogues: 1) ProCoT largely enhances the
originally poor performance of asking clarification questions, but still
limits in handling domain-specific applications. 2) LLM-based dialogue systems
perform aggressive topic shifting towards the designated target, while ProCoT
enables the topic planning to be more smooth. 3) Despite the powerfulness on
controllable response generation, the capability of strategy learning and
planning is the key challenge for LLM-based dialogue systems handling non-
collaborative dialogues.

## References

  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020.  Towards a human-like open-domain chatbot.  _CoRR_ , abs/2001.09977.
  * Aliannejadi et al. (2021) Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail S. Burtsev. 2021.  Building and evaluating open-domain dialogue corpora with clarifying questions.  In _EMNLP_.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  _CoRR_ , abs/2302.04023.
  * Chen et al. (2023a) Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani-Tur. 2023a.  PLACES: prompting language models for social conversation synthesis.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 814–838.
  * Chen et al. (2023b) Maximillian Chen, Xiao Yu, Weiyan Shi, Urvi Awasthi, and Zhou Yu. 2023b.  Controllable mixed-initiative dialogue generation through prompting.  _CoRR_ , abs/2305.04147.
  * Chen et al. (2022) Zhi Chen, Lu Chen, Bei Chen, Libo Qin, Yuncong Liu, Su Zhu, Jian-Guang Lou, and Kai Yu. 2022.  Unidu: Towards A unified generative dialogue understanding framework.  In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2022_ , pages 442–455.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Deng et al. (2023) Yang Deng, Wenqiang Lei, Wai Lam, and Tat-Seng Chua. 2023.  A survey on proactive dialogue systems: Problems, methods, and prospects.  _CoRR_ , abs/2305.02750.
  * Deng et al. (2022) Yang Deng, Wenqiang Lei, Wenxuan Zhang, Wai Lam, and Tat-Seng Chua. 2022.  PACIFIC: towards proactive conversational question answering over tabular and textual data in finance.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022_ , pages 6970–6984.
  * Deng et al. (2021) Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021.  Unified conversational recommendation policy learning via graph-based reinforcement learning.  In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1431–1441.
  * Dinan et al. (2019) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander I. Rudnicky, Jason D. Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason Weston. 2019.  The second conversational intelligence challenge (convai2).  _CoRR_ , abs/1902.00098.
  * Guo et al. (2021) Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. 2021.  Abg-coqa: Clarifying ambiguity in conversational question answering.  In _3rd Conference on Automated Knowledge Base Construction, AKBC 2021_.
  * He et al. (2018) He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018.  Decoupling strategy and generation in negotiation dialogues.  In _EMNLP_.
  * Huang et al. (2022) Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In _International Conference on Machine Learning, ICML_ , volume 162, pages 9118–9147.
  * Ji et al. (2020) Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, and Minlie Huang. 2020\.  Language generation with multi-hop reasoning on commonsense knowledge graph.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020_ , pages 725–736.
  * Joshi et al. (2021) Rishabh Joshi, Vidhisha Balachandran, Shikhar Vashishth, Alan W. Black, and Yulia Tsvetkov. 2021.  Dialograph: Incorporating interpretable strategy-graph networks into negotiation dialogues.  In _ICLR_.
  * Lee et al. (2021) Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021.  Dialogue state tracking with a language model using schema-driven prompting.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021_ , pages 4937–4949.
  * Lee et al. (2023) Gibbeum Lee, Volker Hartmann, Jongho Park, Dimitris Papailiopoulos, and Kangwook Lee. 2023.  Prompted llms as chatbot modules for long open-domain conversation.  _CoRR_ , abs/2305.04533.
  * Lei et al. (2022) Wenqiang Lei, Yao Zhang, Feifan Song, Hongru Liang, Jiaxin Mao, J. Lv, Z. Yang, and Tat-Seng Chua. 2022.  Interacting with non-cooperative user: A new paradigm for proactive dialogue policy.  In _SIGIR_.
  * Liao et al. (2023) Lizi Liao, Grace Hui Yang, and Chirag Shah. 2023.  Proactive conversational agents.  In _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023_ , pages 1244–1247. ACM.
  * Liu et al. (2022) Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. 2022.  Multi-stage prompting for knowledgeable dialogue generation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1317–1337.
  * Mi et al. (2022) Fei Mi, Yasheng Wang, and Yitong Li. 2022.  CINS: comprehensive instruction for few-shot learning in task-oriented dialog systems.  In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022_ , pages 11076–11084.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  _CoRR_ , abs/2203.02155.
  * Qin et al. (2020) Jinghui Qin, Zheng Ye, Jianheng Tang, and Xiaodan Liang. 2020.  Dynamic knowledge routing network for target-guided open-domain conversation.  In _AAAI_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  Coqa: A conversational question answering challenge.  _Trans. Assoc. Comput. Linguistics_ , 7:249–266.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021\.  Recipes for building an open-domain chatbot.  In _EACL_.
  * Sevegnani et al. (2021) Karin Sevegnani, David M. Howcroft, Ioannis Konstas, and Verena Rieser. 2021.  Otters: One-turn topic transitions for open-domain dialogue.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021_ , pages 2492–2504.
  * Shuster et al. (2022) Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022.  Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 373–393.
  * Tang et al. (2019) Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019.  Target-guided open-domain conversation.  In _ACL_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  _GitHub repository_.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.  _CoRR_ , abs/2201.08239.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971.
  * Wang et al. (2023) Jian Wang, Dongding Lin, and Wenjie Li. 2023.  Dialogue planning via brownian bridge stochastic process for goal-directed proactive dialogue.  _CoRR_ , abs/2305.05290.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020_.
  * Wang et al. (2019) Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019.  Persuasion for good: Towards a personalized persuasive dialogue system for social good.  In _ACL_.
  * Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  Self-instruct: Aligning language model with self generated instructions.  _CoRR_ , abs/2212.10560.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a.  Emergent abilities of large language models.  _CoRR_ , abs/2206.07682.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022b.  Chain of thought prompting elicits reasoning in large language models.  _CoRR_ , abs/2201.11903.
  * Wu et al. (2020) Chien-Sheng Wu, Steven C. H. Hoi, Richard Socher, and Caiming Xiong. 2020.  TOD-BERT: pre-trained natural language understanding for task-oriented dialogue.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020_ , pages 917–929.
  * Wu et al. (2019) Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019.  Proactive human-machine conversation with explicit conversation goal.  In _ACL_.
  * Xu et al. (2023) Canwen Xu, Daya Guo, Nan Duan, and Julian J. McAuley. 2023.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  _CoRR_ , abs/2304.01196.
  * Yang et al. (2022) Zhitong Yang, Bo Wang, Jinfeng Zhou, Yue Tan, Dongming Zhao, Kun Huang, Ruifang He, and Yuexian Hou. 2022.  Topkg: Target-oriented dialog via global planning on knowledge graph.  In _COLING_.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.  React: Synergizing reasoning and acting in language models.  _CoRR_ , abs/2210.03629.
  * Zamani et al. (2022) Hamed Zamani, Johanne R. Trippas, Jeff Dalton, and Filip Radlinski. 2022.  Conversational information seeking.  _CoRR_.
  * Zhan et al. (2022) Haolan Zhan, Yufei Wang, Tao Feng, Yuncheng Hua, Suraj Sharma, Zhuang Li, Lizhen Qu, and Gholamreza Haffari. 2022.  Let’s negotiate! A survey of negotiation dialogue systems.  _CoRR_.
  * Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020_.
  * Zhang et al. (2023) Xiaoying Zhang, Baolin Peng, Kun Li, Jingyan Zhou, and Helen Meng. 2023.  Sgp-tod: Building task bots effortlessly via schema-guided llm prompting.
  * Zhang et al. (2020b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020_ , pages 270–278.
  * Zhao et al. (2023) Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang, Yanpeng Tong, and Bing Qin. 2023\.  Is chatgpt equipped with emotional dialogue capabilities? _CoRR_ , abs/2304.09582.
  * Zhong et al. (2021) Peixiang Zhong, Yong Liu, Hao Wang, and Chunyan Miao. 2021.  Keyword-guided neural conversational model.  In _AAAI_.
  * Zhou et al. (2020a) Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. 2020a.  Towards topic-guided conversational recommender system.  In _COLING_.
  * Zhou et al. (2020b) Yiheng Zhou, Yulia Tsvetkov, Alan W. Black, and Zhou Yu. 2020b.  Augmenting non-collaborative dialog systems with explicit semantic and strategic dialog history.  In _ICLR_.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021.  TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021_ , pages 3277–3287.

##  Appendix A Details of Datasets

In the experiment, we adopt the test sets from five datasets for evaluation,
including Abg-CoQA Guo et al. (2021), PACIFIC Deng et al. (2022), Otters
Sevegnani et al. (2021), TGConv Yang et al. (2022), and CraigslistBargain He
et al. (2018). Detailed descriptions of each dataset are as follows:

  * •

Abg-CoQA555https://github.com/MeiqiGuo/AKBC2021-Abg-CoQA/tree/main/abg-coqa is
constructed based on the CoQA dataset Reddy et al. (2019) by truncating a
partial conversation from the full conversation and selecting ambiguous
questions.

  * •

PACIFIC666https://github.com/dengyang17/PACIFIC/tree/main/data/pacific. Since
the labels in the test set is not publicly released, we adopt the validation
set for evaluation. is constructed based on the TAT-QA dataset Zhu et al.
(2021), an question answering dataset in the financial domain, whose contexts
contain a hybrid of tables and texts. Deng et al. (2022) rewrite the questions
to be ambiguous for introducing clarification turns in the conversation.

  * •

Otters is a next-turn target-oriented dialogue dataset, which requires the
agent proactively generate a transition utterance to approach the designated
target. We adopt the processed version777https://github.com/yyyyyyzt/topkgchat
by Yang et al. (2022) for evaluation.

  * •

TGConv is constructed based on ConvAI2 Dinan et al. (2019) and is split to two
settings, including "easy-to-reach" and "hard-to-reach".

  * •

CraigslistBargain was created in a negotiation setting where two crowdsourced
workers play the roles of the buyer and the seller to bargain the price of an
item. We adopt the processed
version888https://github.com/rishabhjoshi/DialoGraph_ICLR21/tree/main by Joshi
et al. (2021) for evaluation, which assigns 10 dialogue acts and 21
negotiation strategies to the utterances.

Wrong Q: What was the operating loss carryforward Clarification amount? Need
G: Which year are you asking about? Prediction P: The operating loss
carryforward amount was $73,260 as of December 31, 2019. Under-specified Q:
What is the 2019 average defined schemes? Clarification G: What kind of
defined schemes are you asking about? P: Could you please clarify what you
mean by "average defined schemes"? Wrong Aspect Q: What was the increase /
(decrease) in the cost? G: Which period are you asking about? P: Which cost
are you referring to? Over-specified Q: How much amount of goodwill
acquisitions was done in 2019? Clarification G: What kind of goodwill
acquisitions are you asking about? P: Are you asking about the total amount of
goodwill activity or specifically about acquisitions in 2019? Generation Q:
How much was the average operating Error income? G: Which period are you
asking about? P: The document does not provide information on the average
operating income. Therefore, the question cannot be answered.

Table 6: Error Analysis (Q: User Query, G: Ground-truth, P: Prediction).

##  Appendix B Error Analysis Details

As shown in Table 6, we categorize these failure cases into the following five
groups:

  * •

Wrong Clarification Need Prediction: The model makes a wrong prediction on
whether the user query requires clarification.

  * •

Wrong Aspect: The model generates a question for clarifying a wrong aspect of
the user query.

  * •

Under-specified Clarification: The model generates an under-specified
clarification question, where the requested information is too general so that
it would be difficult for the user to provide feedbacks.

  * •

Over-specified Clarification: The model generates an over-specified
clarification question, where the requested information is already clear in
the user query.

  * •

Generation Error: The model doesn’t generate the output as the required format
so that there is no clarification question.

##  Appendix C Case Study

In order to intuitively compare the three prompting schemes, we conduct case
studies on the generated responses regarding three proactive dialogue
problems.

###  C.1 Clarification Dialogues

Figure 4: Case study from the Abg-CoQA dataset.

Figure 4 presents a case study from the Abg-CoQA dataset. There are two
possible answers for the ambiguous user question in this case. When using
standard prompting, ChatGPT generates a response based on a random guess,
potentially resulting in an answer that may not align with the user’s desired
outcome. When using proactive prompting, although ChatGPT decides to ask a
clarification question, the concerned aspect of clarification is not the focus
of this conversation. ProCoT prompting first provides a thorough analysis to
identify the underlying reasons for the ambiguity in the user’s question. This
analysis serves as the foundation for generating an effective clarifying
question, which helps to clarify and disambiguate the user’s intended meaning.

###  C.2 Target-guided Dialogues

Figure 5: Case study from the TGConv dataset.

Figure 5 presents a case study from the TGConv dataset, where the hard-to-
reach target of this case is "Gaullist", and the system is required to reach
this target within 8 turns of conversations under the self-play simulation. As
for Vicuna, it is struggled to lead the conversation towards this hard-to-
reach target, when using standard and proactive prompting. ProCoT prompting
enables Vicuna to effectively and smoothly drive the conversation towards the
designated target. In addition, owing to the powerful capability of
controllable text generation, ChatGPT directly responds with the target topic
to the initial user utterance. However, the topic transition is relatively
aggressive, which might downgrade the user engagement or experience during the
conversation.

Figure 6: Case study from the CraigslistBargain dataset.

###  C.3 Non-collaborative Dialogues

Figure 6 presents a case study from the CraigslistBargain dataset, where the
system plays the seller role to bargain with the buyer. At turn 3, even though
the buyer just inquires about the item information without showing the bargain
intention, ChatGPT with standard prompting tends to initiate the negotiation,
which may put the seller in a disadvantageous position. Proactive and ProCoT
prompting enable the dialogue act and strategy prediction of the next
response. Especially for the analysis of the current negotiation status,
ProCoT points out that the negotiation has not yet started.

At turn 9, we observe that the seller has already lowered down the bargain
price to $40 in a previous turn. Without the reasoning and planning process,
ChatGPT with standard and proactive prompting generates the response with
contradictory statement, i.e., propose a higher counter price ($45) for
bargain, which is unreasonable in negotiation dialogues. With proactive CoTs,
ChatGPT effectively summarizes the current negotiation progress and makes a
better decision on the next negotiation goal.

##  Appendix D Examples of Prompts

Table 7, 8, and 9 present the example of prompts for clarification, target-
guided, non-collaborative dialogues, respectively. As for the zero-shot
setting, the overall prompt is composed by the task instruction and the
sample. As for the few-shot setting, the overall prompt is composed by the
task instruction, a number of samples with demonstrations, and the test
sample.

Clarification Dialogues
---
Standard Prompting: Given the document and the conversation history, generate
the response.
Proactive Prompting: Given the document and the conversation history, answer
the question or ask a clarifying question. The response should start with "The
answer is" or "The clarifying question is".
ProCoT Prompting: Given the document and the conversation history, first
identify whether the question is ambiguous or not. If it is ambiguous, ask a
clarifying question. If it is not ambiguous, answer the question. The response
should start with the ambiguity analysis of the question and then follow by
"Therefore, the question is not ambiguous. The answer is" or "Therefore, the
question is ambiguous. The clarifying question is".
Sample: Document: "Angie went to the library with her mother. First she had to
turn in the books she was returning at the return desk. They said hello to the
man there. He took their books. Then they went into the adult reading room.
Angie sat in a brown chair at the table. She made a drawing of her mother. Her
mother found a large red book. Then they went to the Mystery section. Angie
sat in a blue chair. She drew a picture of her brother. Her mother found the
book. It was a green book. Finally it was time to go to the children’s room.
It was Story Hour. Miss Hudson was there to read to all the children. She read
a book about friendship. After the story Angie sat in the red chair and began
drawing. They were drawing pictures of friends. Angie drew a picture of her
best friend Lilly. Miss Hudson hung the pictures on the wall. Then Angie and
her mother picked out 8 books to read at home. They checked the books out and
went home." Conversation history: [“User”: “What did she draw?”, “System”:
“Her mother”, “User”: “What did her mother find?”, “System”: “The book”]
Question: “What color was it?”
Demonstration (Standard): Do you mean the first book?
Demonstration (Proactive): The clarifying question is "Do you mean the first
book?"
Demonstration (ProCoT): There are two books that book that Angie’s mother
found. It is uncertain which book is referred to. Therefore, the question is
ambiguous. The clarifying question is "Do you mean the first book?"
Table 7: Examples of prompting LLMs for clarification dialogues. Target-guided
Dialogues
---
Standard Prompting: Given the target topic and the conversation history,
generate the response.
Proactive Prompting: Given the target topic and the conversation history,
predict the appropriate next topics that can bridge the current conversation
topics to approach the target topics smoothly. Then based on the predicted
next topics, generate the response. Please reply by completing the output
template "The next topics are []. The response is".
Proactive Prompting w/ CoT: Given the target topic and the conversation
history, consider the relationship between the current conversation topics and
the target topics, and then predict the appropriate next topics that can
bridge the current conversation topics to approach the target topics smoothly.
Then based on the predicted next topics, generate the response. Please reply
by completing the output template "The current topics are []. To bridge the
current topics with the target topics, the next topics are []. Based on the
predicted next topics, the response is".
Sample:
Target topic: "Chicken" Conversation history: ["User": "I also remodel homes
when I am not out bow hunting.", "System": "That’s neat. When I was in high
school I placed 6th in 100m dash!", "User": "That’s awesome. Do you have a
favorite season or time of year?"]
Demonstration (Standard): I do not. But I do have a favorite meat since that
is all I eat exclusively.
Demonstration (Proactive): The next topics are ["eat", "meat"]. The response
is "I do not. But I do have a favorite meat since that is all I eat
exclusively."
Demonstration (ProCoT): The current topics are ["season", "time", "year"]. To
bridge the current topics with the target topics, the next topics are ["eat",
"meat"]. Based on the predicted next topics, the response is "I do not. But I
do have a favorite meat since that is all I eat exclusively."
Table 8: Examples of prompting LLMs for target-guided dialogues. Non-
collaborative Dialogues
---
Standard Prompting: Assume you are the seller. Given the item description, the
target selling price, and the conversation history, generate the response.
Proactive Prompting: Assume you are the seller. Given the item description,
the target selling price, and the conversation history, in order to reach a
better deal with the buyer, first select the most appropriate set of
negotiation strategies and the most appropriate dialogue act to reach the
bargain price. Based on the selected negotiation strategies and dialogue act,
generate the response. The reply should be in the form "The most appropriate
set of negotiation strategies is [] and the most appropriate dialogue act is
[]. Based on the selected negotiation strategies and dialogue act, the
response is"
ProCoT Prompting: Assume you are the seller. Given the item description, the
target selling price, and the conversation history, in order to reach a better
deal with the buyer, first analyse the current negotiation progress and
consider an appropriate negotiation goal, then select the most appropriate set
of negotiation strategies and the most appropriate dialogue act to reach the
bargain price. Based on the selected negotiation strategies and dialogue act,
generate a response. The reply should start with the analysis of the current
negotiation progress and an appropriate goal, and then follow by "To reach
this goal, the most appropriate set of negotiation strategies is [] and the
most appropriate dialogue act is []. Based on the selected negotiation
strategies and dialogue act, the response is"
Pre-defined Dialogue Acts: [Answer a question, Proposing a counter price,
Accept the offer, …]
Pre-defined Negotiation Strategies: [Communicate politely, Build rapport, Show
dominance, …]
Sample:
Item description: "Phone charge two devices simultaneously on the go. This
vehicle charger with an additional USB port delivers enough power to charge
two devices at once. The push button activated led connector light means no
more fumbling in the dark trying to connect your device. Auto detect IC
technology automatically detects the device type and its specific charging
needs for improved compatibility. And the built in indicator light illuminates
red to let you know the charger is receiving power and the power socket is
working properly. Verizon car charger with dual output micro USB and led
light." Target selling price: 10. Conversation history: ["Buyer": "Hi, not
sure if the charger would work for my car. can you sell it to me for $5?",
"Seller": "It will work, I have never seen a car without a cigarette lighter
port.", "Buyer": "Still, can i buy it for $5? I’m on a tight budget."]
Demonstration (Standard): I think the lowest I would want to go is 8.
Demonstration (Proactive): The most appropriate set of negotiation strategies
is ["Propose price", "Show dominance", ’Certainty words’] and the most
appropriate dialogue act is ["Proposing a counter price"]. Based on the
selected negotiation strategies and dialogue act, the response is "I think the
lowest I would want to go is 8."
Demonstration (ProCoT): The buyer proposes a low price, which is unacceptable.
The next step should assertively raise the bargain price. To reach this goal,
the most appropriate set of negotiation strategies is ["Propose price", "Show
dominance", ’Certainty words’] and the most appropriate dialogue act is
["Proposing a counter price"]. Based on the selected negotiation strategies
and dialogue act, the response is "I think the lowest I would want to go is
8."
Table 9: Examples of prompting LLMs for non-collaborative dialogues.


Proceedings of the CTD 2023 PROC-CTD2023-28

High Pileup Particle Tracking with Object Condensation

Kilian Lieret1,2, Gage DeZoort1, Devdoot Chatterjee3,
Jian Park4, Siqi Miao5, Pan Li5

1Princeton University

2Institute for Research and Innovation in Software for High Energy Physics
(IRIS-HEP)

3Delhi Technological University

4University of Chicago

5Georgia Tech

> ABSTRACT
>
>
>
>
> Recent work has demonstrated that graph neural networks (GNNs) can match the
> performance of traditional algorithms for charged particle tracking while
> improving scalability to meet the computing challenges posed by the HL-LHC.
> Most GNN tracking algorithms are based on edge classification and identify
> tracks as connected components from an initial graph containing spurious
> connections. In this talk, we consider an alternative based on object
> condensation (OC), a multi-objective learning framework designed to cluster
> points (hits) belonging to an arbitrary number of objects (tracks) and
> regress the properties of each object. Building on our previous results, we
> present a streamlined model and show progress toward a one-shot OC tracking
> algorithm in a high-pileup environment.

> PRESENTED AT
>
>
>
>
> Connecting the Dots Workshop (CTD 2023)
>
> October 10-13, 2023

##  1 Introduction

Traditional charged particle tracking algorithms at the Large Hadron Collider
(LHC) are based on the combinatorial Kalman filter. However, this class of
algorithms exhibits sub-optimal scaling with respect to pileup, rendering
tracking a bottleneck for future experiments such as the High Luminosity LHC
(HL-LHC) [1]. This has prompted research into tracking algorithms leveraging
graph neural networks (GNNs) or similar machine learning (ML) architectures
demonstrating improved computational scaling. Recent results have confirmed
that GNN-based algorithms can indeed achieve linear scaling with pileup [2,
3].

The majority of GNN approaches adopt an edge classification (EC) approach to
tackle the tracking problem. Given an initial graph that connects all hits
that potentially belong to the same particle, a GNN is trained to remove edges
that connect hits belonging to different particles. Tracks can then be
identified as connected components of the graph***This is simplified: Most
state-of-the-art EC approaches use iterative “graph walking” algorithms to
determine tracks based on EC scores rather than simply applying a threshold
and using connected components., and subsequent steps assess track quality or
fit track parameters.

This work considers an alternative approach based on clustering track hits in
a learned latent space. Specifically, we employ object condensation (OC) [4],
a multi-objective training scheme designed to cluster points (hits) matched to
an arbitrary number of objects (tracks) and regress the properties of each
object. Besides a general need for broad exploration of ML architectures in
tracking, this class of approaches is motivated by several observations:

  1. 1.

Most state-of-the-art EC approaches use a learned embedding space to build the
initial graph edges between clusters of hits (metric learning, see Ref. [2]).
In an EC approach, this initial embedding space is discarded, whereas OC
algorithms learn to refine it.

  2. 2.

Edges in the EC approach serve two competing purposes: 1) facilitating message
passing across the graph and 2) representing individual tracks in the graph.
Though having more edges might facilitate a broader scope of message passing,
the ultimate goal of an EC algorithm is to produce few edges, representing
only real particle trajectories. In contrast, edges are _only_ used for
message passing in the OC approach, and tracks are constructed based on the
node embedding space.

This has important consequences: for example, any missing edge in an EC
approach degrades performance (a track that is “broken up” cannot be fixed).
However, we have shown [5] that OC is not subject to the same limitation and
can (to some extent) deal with missing edges.

  3. 3.

Because the OC approach directly addresses the relationship between hits and
tracks, it can be trained to regress track properties; therefore, it is more
suitable for one-shot architectures with little-to-no post-processing.
Training to regress track properties such as $p_{\text{T}}$ might also
increase model robustness by imparting track physics onto the model.

In this paper, we build on our previous results and show a new streamlined
model that outperforms our previous architecture. Instead of relying on a
geometric graph construction, this model uses the learned clustering strategy
of [2].

We also discuss several other new insights and ongoing developments: The use
of the Modified Differential Multiplier Method to balance different training
objectives in object condensation, ongoing development of a model with GravNet
layers, and ongoing development of a different approach using sparse
transformers instead of GNNs.

##  2 Dataset and metrics

All results are produced using the TrackML dataset [6, 7] that simulates a
generic tracking detector geometry in the worst-case HL-LHC pileup conditions
($\langle\mu\rangle=200$). We only consider the pixel detector layers (4
barrel layers and 7 layers in each endcap). Each tracker event is represented
as a graph by embedding track hits as nodes; a detailed summary of the
features associated with each node is available in Ref. [5].

We use several different metrics to quantify the performance of our pipeline:

  * •

Perfect match efficiency ($\epsilon^{\text{perfect}}$): The number of
reconstructed tracks that include all hits of the matched particle and no
other hits, normalized to the number of particles.

  * •

LHC-style match efficiency ($\epsilon^{\text{LHC}}$): The fraction of
reconstructed tracks in which 75% of the hits belong to the same particle,
normalized to the number of reconstructed tracks.

  * •

Double majority match efficiency ($\epsilon^{\text{DM}}$): The fraction of
reconstructed tracks in which at least 50% of the hits belong to one particle
and this particle has less than 50% of its hits outside of the reconstructed
track, normalized to the number of particles.

  * •

Double majority match fake rate ($f^{\text{DM}}$): The fraction of
reconstructed tracks that does not satisfy the double majority match
criterion, normalized to the number of reconstructed tracks.

Throughout most of the paper, we consider these metrics for _particles of
interest_ , that is, particles with $p_{\text{T}}>0.9\,\mathrm{GeV}$,
$|\eta|<4.0$ that have at least three hits. These metrics are denoted as
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$,
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}$,
$\epsilon^{\text{LHC}}_{p_{\text{T}}>0.9}$, and
$f^{\text{DM}}_{p_{\text{T}}>0.9}$. For a more verbose definition of these
metrics, see Ref. [5].

##  3 A streamlined OC architecture

The pipeline outlined in this section comprises three stages: graph
construction in a learned clustering space, object condensation, and
postprocessing/track rendering.

###  3.1 Graph construction

The main improvement compared to the pipeline presented in Ref. [5] is the use
of a learned clustering approach to graph construction (GC). This approach is
very similar to that used in the Exa.TrkX EC pipeline in Ref. [2].

We use a fully connected neural network (FCNN) to produce learned clustering
coordinates for each track hit. The FCNN takes the node features
$z^{(0)}_{i}\in\mathbb{R}^{14}$ (enumerated in Ref. [5]) as inputs and embeds
them into a 256-dimensional space by a fully connected layer:
$z^{(1)}_{i}\coloneqq W^{(1)}z^{(0)}_{i}$, with learnable weights
$W^{(1)}\in\mathbb{R}^{256\times 14}$. Five subsequent layers with width 256,
$\operatorname{ReLU}$ activations, and residual connections of the form
$z^{(\ell+1)}_{i}\coloneqq\sqrt{\beta}\,W^{(\ell+1)}\operatorname{ReLU}\bigl{(}z^{(\ell)}_{i}\bigr{)}+\sqrt{1-\beta}\,z^{(\ell)}_{i}$
(where $l=1,\dots,5$ and $\beta=0.4$) are applied. A final layer is applied to
map the hidden representations $z_{i}^{(6)}$ down to an 8-dimensional space
$h^{\text{GC}}_{i}\coloneqq W^{(7)}\operatorname{ReLU}(z_{i}^{(6)})$.

The network is trained with an attractive loss $L^{\text{att}}$ and a
repulsive hinge loss $L^{\text{rep}}$:

$\displaystyle L^{\text{att}}$
$\displaystyle\coloneqq\frac{1}{|I_{\text{att}}|}\sum_{(i,j)\in
I_{\text{att}}}\|h^{\text{GC}}_{i}-h^{\text{GC}}_{j}\|^{2},\qquad
I_{\text{att}}\coloneqq\\{(i,j)\,|\,1\leq i,j\leq N_{\text{hits}},\
\pi_{i}=\pi_{j},\ \pi_{i}\ \text{of interest}\\},$ (1) $\displaystyle
L^{\text{rep}}$
$\displaystyle\coloneqq\frac{1}{|I_{\text{att}}|}\sum_{(i,j)\in
I_{\text{rep}}}\operatorname{ReLU}\bigl{(}1-\|h^{\text{GC}}_{i}-h^{\text{GC}}_{j}\|^{2}\bigr{)},\qquad
I_{\text{rep}}\coloneqq\\{(i,j)\,|\,1\leq i,j\leq N_{\text{hits}},\
\pi_{i}\neq\pi_{j}\ \text{or noise}\\},$

where $\pi_{i}$ denotes the particle of hit $i$ and “of interest” is defined
as in section 2. Note that these loss functions differ from those of Ref. [2]
in how they deal with low-$p_{\text{T}}$ hits. In addition, Ref. [2] includes
all pairs of hits in non-consecutive detector layers in the repulsive
potential.

Balancing the attractive and repulsive loss strongly influences the
performance of the graph construction. We find that linear scalarization of
the two objectives, $L\coloneqq L^{\text{att}}+s_{\text{rep}}L^{\text{rep}}$
(we use $s_{\text{rep}}=0.06$), leads to overall good convergence with a
corresponding convex Pareto front (Figure 1). In preparation for dealing with
the many objective functions of the OC approach, we have furthermore
investigated the use of the modified method of differential modifiers (MDMM)
[8] and optimized $L^{\text{att}}$ subject to a constraint on $L^{\text{rep}}$
(Figure 1). This achieved identical results but did not significantly simplify
the optimization process (note that this might become more relevant when track
parameter prediction is incorporated as an additional objective).

Figure 1: Identifying the Pareto front. Each colored line corresponds to the
training trajectory of one model through $(L_{\text{att}},L_{\text{rep}})$
space. Models with a specified value of $s_{\text{rep}}$ use linear
scalarization of the two objectives, while the model with
$\epsilon_{\text{rep}}$ uses MDMM (with $\epsilon_{\text{rep}}$ denoting the
constraint on $L_{\text{rep}}$).

A visual representation of track hits embedded in the learned clustering space
is shown in Figure 2. While the embedding looks near-perfect to the eye,
Figure 4a (discussed in the next section) confirms that this initial
clustering space is still insufficient to directly reconstruct tracks from.

The graph is then built from this latent space using $k$-nearest neighbors
(kNN) while also limiting the maximal edge length to $1$ (arbitrary units). To
quantify the quality of the constructed graph, we call an edge _true_ , if it
connects two hits of the same particle (any edge that connects to a noise hits
is _false_). We call an edge _of interest_ if at least one of the two hits
belongs to a particle of interest. The quality of the constructed graph vs $k$
is quantified in Figure 3 in terms of four metrics:

  * •

Efficiency: The number of true edges of interest normalized to the number of
possible true edges of interest ($\sum_{\pi\ \text{of
interest}}{N_{\text{hits}}^{\pi}\choose 2}$, where $N_{\text{hits}}^{\pi}$ is
the number of hits for a particle $\pi$)

  * •

Purity: The number of true edges of interest normalized to the number of edges
of interest.

  * •

Upper bounds on figures of merit: As introduced in Ref. [5], we define upper
bounds of an EC pipeline for $\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$ and
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}$ by calculating both metrics for
tracks given by the connected components of the edge-subgraph that only
contains true edges. It was shown that these upper bounds do not necessarily
hold for an OC pipeline, but they are still useful in quantifying the
connectivity of the graph in a way that relates to the key tracking metrics.

While the overall efficiency of the GC step flattens out slowly, the upper
bounds on $\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$ and
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}$ show only very limited
increases for $k>10$. We therefore choose $k=10$ for the remainder of this
section. At this point, an average of $468\times 10^{3}$ edges are built with
an efficiency of $77\%$, a purity of $44\%$, and EC upper bounds on
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$ and
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}$ of $98\%$ and $92\%$.

Figure 2: t-SNE projection of the input space (left) and the graph
construction embedding space $h^{\text{GC}}$ (right). The hits of seven
randomly selected particles of interest have been colored. Figure 3: Choosing
the number of nearest neighbors ($k$) for graph construction in our 2.0
pipeline. Note that efficiency and purity are relative to a fully connected
graph of all hits per particle (rather than a path-graph of layer-to-layer
connections).

###  3.2 Object condensation

The GNN used to perform OC is identical to that in Ref. [5] except for
different input shapes and an additional residual connection that connects the
8-dimensional embedding space ($h_{i}^{\mathrm{GC}}$) that was used for GC
with the final OC output coordinates.

The model is built from _interaction network_ layers [9] with residual
connections in the node updates. The node features $x_{i}^{(0)}$ are the input
features $z_{i}^{(0)}\in\mathbb{R}^{14}$ described in section 2 concatenated
with the GC embedding $h^{\text{GC}}_{i}\in\mathbb{R}^{8}$, i.e.,
$x_{i}^{(0)}\coloneqq[z_{i}^{(0)},h^{\text{GC}}_{i}]$. The edge features
$e_{ij}^{(0)}$ ($(i,j)\in I$, where $I$ represents the edges of the graph) are
given by
$e_{ij}^{(0)}\coloneqq[x_{i}^{(0)}-x_{j}^{(0)},x_{i}^{(0)}+x_{j}^{(0)}]\in\mathbb{R}^{44}$.

Node and edge features are first encoded, $x_{i}^{(1)}\coloneqq
W^{\text{enc}}_{\text{node}}x_{i}^{(0)}$, $e_{ij}^{(1)}\coloneqq
W^{\text{enc}}_{\text{edge}}e_{ij}^{(0)}$,
$W^{\text{enc}}_{\text{node}}\in\mathbb{R}^{192\times 22}$, and
$W^{\text{enc}}_{\text{edge}}\in\mathbb{R}^{192\times 44}$. Then, five
iterations of message passing are performed with

$\displaystyle\begin{split}e_{ij}^{(k+1)}&\coloneqq\bigl{(}\Phi^{(k+1)}\circ\operatorname{ReLU}\bigr{)}\Big{(}\bigl{[}x_{i}^{(k)},x_{j}^{(k)},e_{ij}^{(k)}\bigr{]}\Big{)},\\\
x_{i}^{(k+1)}&\coloneqq\sqrt{\beta}\,\Psi^{(k+1)}\Bigl{(}\bigl{[}\operatorname{ReLU}\bigl{(}x_{i}^{(k)}\bigr{)},{\textstyle\sum\nolimits_{j\in\mathcal{N}_{i}}}e_{ij}^{(k+1)}\bigr{]}\Bigr{)}+\sqrt{1-\beta}\,x_{i}^{(k)}.\end{split}$
(2)

Here, $(i,j)\in I$, $k=1,\dots,5$, and $\mathcal{N}_{i}$ denotes the
neighborhood of $i$. $\Phi$ and $\Psi$ are FCNNs with $\operatorname{ReLU}$
activations, a layer width of 192, and one hidden layer. $\beta$ has been
chosen to be $0.5$. Finally, the clustering coordinates are decoded (and added
to a residual from the GC space) as

$\displaystyle
c_{i}\coloneqq\sqrt{\beta^{\prime}}\,W^{\text{dec}}_{\text{c}}\operatorname{ReLU}(x_{i}^{(6)})+\sqrt{1-\beta^{\prime}}\,[h_{i}^{\text{GC}},0,\dots,0],$
(3)

where we chose $\beta^{\prime}=0.5$,
$W_{\text{c}}^{\text{dec}}\in\mathbb{R}^{24\times 192}$, and
$h_{i}^{\text{GC}}$ is zero-padded to $\mathbb{R}^{24}$. The condensation
likelihoods are decoded as
$\beta_{i}\coloneqq\sigma\bigl{(}W^{\text{dec}}_{\beta}\operatorname{ReLU}(x_{i}^{(6)})\bigr{)}$,
where $\sigma$ is the logistic function, and
$W^{\text{dec}}_{\beta}\in\mathbb{R}^{1\times 192}$. The total number of
parameters of this model is $1.9\times 10^{6}$.

To train the model, we use the object condensation loss functions [4] as
described in Ref. [5] with the following hyperparameters: $s_{\beta}=0$,
$s_{\text{rep}}=0.74$, and $q_{\text{min}}=0.01$.

###  3.3 Post-processing and results

We use Density-Based Spatial Clustering of Applications with Noise (DBSCAN)
[10] to determine clusters in the OC clustering space $c$. DBSCAN is an
iterative clustering algorithm with two parameters, $\epsilon$ (the size of
the neighborhood of a point that is considered when merging clusters), and $k$
(minimum number of points within a neighborhood for the points to be
considered a _core point_). In our case, $k=1,2,3$ produces equal results
(this is related to the fact that clusters with less than three hits are
discarded as track candidates) and $k\geq 4$ degrades performance (see Figure
4b). Therefore, all results use $k=1$. Based on Figure 4b, we choose
$\epsilon=0.53$. The broad plateau in $\epsilon$ vs
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}$ shows that the clusters are well
separated from each other and from noise. We obtain
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}=96.4\%$,
$\epsilon^{\text{LHC}}_{p_{\text{T}}>0.9}=98.0\%$,
$\epsilon^{\text{perfect}}_{p_{\text{T}}>0.9}=85.8\%$ and
$f_{p_{\text{T}}>0.9}=0.9\%$. These metrics uniformly improve our previous
results in Ref. [5]. All metrics are presented vs $p_{\text{T}}$ and vs $\eta$
in Figure 5.

(a)

(b)

Figure 4:  Rendering tracks from the learned clustering space with DBSCAN. The
left plot shows track reconstruction using the embedding space $h^{\text{GC}}$
used for graph construction, and the right plot shows the significantly
improved tracking performance using the OC embedding space. Dashed lines
correspond to DBSCAN with $k=4$.

Figure 5: Tracking performance in bins of $p_{\text{T}}$ and $\eta$.

##  4 Progress on other architectures

While the model presented in the main part of this paper is built on
interaction networks, many other architectures are left to be explored in
detail. For example, _GravNet_ [11] layers have been successfully used with an
OC approach for multi-particle reconstruction in high occupancy imaging
calorimeters [12, 13]. A similar architecture is also being investigated for
the Belle II outer tracker, motivating studies for its suitability for an HL-
LHC application. In our first experiments, we confirmed that an architecture
of four (slightly modified) GravNet layers can achieve
$\epsilon^{\text{DM}}_{p_{\text{T}}>0.9}>90\%$ using $k=4$ kNN with
8-dimensional latent-spaces. However, a careful optimization of
hyperparameters is needed for an apples-to-apples comparison with the pipeline
described in section 3.

Shifting from a very edge-oriented perspective on tracking models (e.g., EC
approaches) to one that emphasizes embeddings (e.g. OC) also motivates to
consider non-GNN architectures. The success of transformer architectures in
encoding complex relationships between tokens and a large body of work related
to hardware optimization makes this class of models attractive for the
tracking problem. A key challenge for the application to tracking is to find
efficient realizations of the attention mechanism in order to overcome the
quadratic scaling with the attention window size (that would need to encompass
all hits of an event in the naive scaled dot product attention). In our
approach, this is achieved by using relative positional encoding and Exact
Euclidean Locality Sensitive Hashing [14] (E2LSH) with an $\mathcal{O}(n\log
n)$ scaling. The resulting model features more regular and parallelizable
computations than GNN models and avoids the performance bottleneck of GPU-
implementations of kNN graph building. The model is trained with contrastive
learning loss functions and hard negative mining. While we are still working
on detailed evaluations in terms of tracking performance, we observe up to
hundred-fold inference speedups on a Quadro RTX 6000 compared to the pipeline
presented in section 3.

##  5 Conclusions

This paper presents an object condensation approach to charged particle
tracking at the worst-case pileup conditions expected at the HL-LHC. Replacing
the geometric graph construction stage of our previous pipeline [5] with an
adaptation of the metric learning approach from Ref. [2] allowed us to make
our pipeline more streamlined while significantly improving tracking
performance on the pixel detector of the TrackML dataset. Ongoing work to
lower memory consumption will make it possible to apply this pipeline to the
full detector. Our pipeline is part of our open-source project [15] that
implements various OC tracking architectures in a modular and extensible
Python package.

In addition, we discussed ongoing research into other architectures, such as
GravNet and sparse transformers. The first results with sparse transformers
show major improvements in inference time due to hardware-friendly
implementations.

ACKNOWLEDGEMENTS

We would like to thank Javier Duarte, Peter Elmer, Philip Chang, Jonathan
Guiang, David Lange, and Savannah Thais for helpful discussions and support.

## References

  * [1] Giuseppe Cerati et al.  “Kalman Filter Tracking on Parallel Architectures”  In _EPJ Web of Conferences_ 127 EDP Sciences, 2016, pp. 00010  DOI: 10.1051/epjconf/201612700010
  * [2] Xiangyang Ju et al.  “Performance of a geometric deep learning pipeline for HL-LHC particle tracking”  In _The European Physical Journal C_ 81.10, 2021, pp. 876  URL: https://link.springer.com/10.1140/epjc/s10052-021-09675-8
  * [3] Alina Lazar et al.  “Accelerating the Inference of the Exa.TrkX Pipeline”  In _Journal of Physics: Conference Series_ 2438.1 IOP Publishing, 2023, pp. 012008  DOI: 10.1088/1742-6596/2438/1/012008
  * [4] Jan Kieseler  “Object condensation: one-stage grid-free multi-object reconstruction in physics detectors, graph, and image data”  In _The European Physical Journal C_ 80.9 Springer ScienceBusiness Media LLC, 2020  DOI: 10.1140/epjc/s10052-020-08461-2
  * [5] Kilian Lieret and Gage DeZoort  “An Object Condensation Pipeline for Charged Particle Tracking at the High Luminosity LHC”, 2023  arXiv:2309.16754 [physics.data-an]
  * [6] Sabrina Amrouche et al.  “The Tracking Machine Learning challenge : Accuracy phase”  In _The NeurIPS ’18 Competition_ Cham: Springer International Publishing, 2020, pp. 231–264  eprint: https://doi.org/10.1007/978-3-030-29135-8–“˙˝9
  * [7] Sabrina Amrouche et al.  “The Tracking Machine Learning challenge : Throughput phase” submitted to Computing and Software for Big Science  In _arXiv:2105.01160 [cs.LG]_ , 2021  URL: http://arxiv.org/abs/2105.01160
  * [8] John Platt and Alan Barr  “Constrained Differential Optimization”  In _Neural Information Processing Systems_ 0 American Institute of Physics, 1987  URL: https://proceedings.neurips.cc/paper_files/paper/1987/file/a87ff679a2f3e71d9181a67b7542122c-Paper.pdf
  * [9] Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende and Koray Kavukcuoglu  “Interaction Networks for Learning about Objects, Relations and Physics” Preprint at https://arxiv.org/abs/1612.00222, 2016  arXiv:1612.00222 [cs.AI]
  * [10] Martin Ester, Hans-Peter Kriegel, Jorg Sander and Xiaowei Xu  “A density-based algorithm for discovering clusters in large spatial databases with noise.”  In _kdd_ 96.34, 1996, pp. 226–231
  * [11] Shah Rukh Qasim, Jan Kieseler, Yutaro Iiyama and Maurizio Pierini  “Learning representations of irregular particle-detector geometry with distance-weighted graph networks”  In _Eur. Phys. J. C_ 79.7, 2019, pp. 608  DOI: 10.1140/epjc/s10052-019-7113-9
  * [12] Shah Rukh Qasim, Kenneth Long, Jan Kieseler, Maurizio Pierini and Raheel Nawaz  “Multi-particle reconstruction in the High Granularity Calorimeter using object condensation and graph neural networks” https://doi.org/10.48550/arXiv.2106.01832 In _arXiv:2106.01832 [physics.ins-det]_ , 2021  URL: http://arxiv.org/abs/2106.01832
  * [13] Shah Rukh Qasim, Nadezda Chernyavskaya, Jan Kieseler, Kenneth Long, Oleksandr Viazlo, Maurizio Pierini and Raheel Nawaz  “End-to-end multi-particle reconstruction in high occupancy imaging calorimeters with graph neural networks”  In _Eur. Phys. J. C_ 82.8, 2022, pp. 753  arXiv:https://doi.org/10.1140/epjc/s10052-022-10665-7 [physics.ins-det]
  * [14] Mayur Datar, Nicole Immorlica, Piotr Indyk and Vahab S. Mirrokni  “Locality-Sensitive Hashing Scheme Based on p-Stable Distributions”  In _Proceedings of the Twentieth Annual Symposium on Computational Geometry_ , SCG ’04  Brooklyn, New York, USA: Association for Computing Machinery, 2004, pp. 253–262  DOI: 10.1145/997817.997857
  * [15] Kilian Lieret and Gage DeZoort  “gnn_tracking: An open-source GNN tracking project”, https://github.com/gnn-tracking/gnn_tracking