# Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support
Conversations

Yang Deng1, Wenxuan Zhang2,†, Yifei Yuan1, Wai Lam1
1 The Chinese University of Hong Kong, 2 DAMO Academy, Alibaba Group
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Unlike empathetic dialogues, the system in emotional support conversations
(ESC) is expected to not only convey empathy for comforting the help-seeker,
but also proactively assist in exploring and addressing their problems during
the conversation. In this work, we study the problem of mixed-initiative ESC
where the user and system can both take the initiative in leading the
conversation. Specifically, we conduct a novel analysis on mixed-initiative
ESC systems with a tailor-designed schema that divides utterances into
different types with speaker roles and initiative types. Four emotional
support metrics are proposed to evaluate the mixed-initiative interactions.
The analysis reveals the necessity and challenges of building mixed-initiative
ESC systems. In the light of this, we propose a knowledge-enhanced mixed-
initiative framework (KEMI) for ESC, which retrieves actual case knowledge
from a large-scale mental health knowledge graph for generating mixed-
initiative responses. Experimental results on two ESC datasets show the
superiority of KEMI in both content-preserving evaluation and mixed initiative
related analyses. ††∗ The work described in this paper is substantially
supported by a grant from the Research Grant Council of the Hong Kong Special
Administrative Region, China (Project Code: 14200620). ††† Corresponding
author.

##  1 Introduction

As the world is making efforts to recover from Covid-19 and plans for future
construction, emotional support is of great importance in resolving the
widespread emotional distress and increased risk for psychiatric illness
associated with the pandemic Pfefferbaum and North (2020); Suh et al. (2021).
A wide range of emotional support conversation (ESC) systems are emerging to
provide prompt and convenient emotional support for help-seekers, including
mental health support Sharma et al. (2021); Lokala et al. (2022), counseling
Althoff et al. (2016); Shen et al. (2020, 2022) or motivational interviewing
Pérez-Rosas et al. (2016); Saha et al. (2021, 2022). Generally, the ESC system
aims at reducing the user’s emotional distress as well as assisting the user
to identify and overcome the problem via conversations Liu et al. (2021).

Figure 1: Examples from EmpatheticDialogues and ESConv datasets with a similar
job loss problem.

Mixed initiative is commonly defined as an intrinsic feature of human-AI
interactions where the user and the system can both take the initiative in
leading the interaction directions Allen et al. (1999); Kraus et al. (2020).
For example, mixed-initiative conversational information-seeking (CIS) systems
Aliannejadi et al. (2019); Deng et al. (2023) can proactively initiate
clarification interactions for resolving the ambiguity in the user query,
instead of only reacting to the query. Accordingly, a mixed-initiative ESC
system can proactively switch the initiative to provide an empathetic response
or initiate a problem-solving discussion when appropriate. Many efforts have
been made on the emotion reasoning for generating empathetic responses Shen et
al. (2020); Zhang and Danescu-Niculescu-Mizil (2020); Cheng et al. (2022);
Peng et al. (2022). Another line of work focuses on identifying the dialogue
acts of the utterances Welivita and Pu (2020); Malhotra et al. (2022);
Svikhnushina et al. (2022) or predicting the next conversational strategies
Pérez-Rosas et al. (2017); Liu et al. (2021); Tu et al. (2022) in ESC systems.
However, the feature of mixed initiative has not been investigated in existing
ESC studies.

To facilitate the analysis on mixed-initiative ESC systems, we first propose
an EAFR schema to annotate the utterances into different types with speaker
roles and initiative types, named Expression (User-initiative), Action
(Support-initiative), Feedback (User Non-initiative), and Reflection (System
Non-initiative). Besides, four emotional support metrics are designed to
measure the characteristics of initiative and non-initiative interactions in
ESC, including Proactivity, Information, Repetition, and Relaxation.

To analyze the necessity of considering mixed initiative in ESC systems, we
conduct a preliminary analysis on the different interaction patterns between
ESC and empathetic dialogues (ED). Firstly, the dialogue flow analysis shows
that the system in ED generally serves as a passive role, while the system in
ESC proactively switches the initiative role during the conversation. As shown
in Figure 1, the system in ED solely targets at comforting the user by
reflecting their feelings or echoing their situations, i.e., Non-Initiative.
Differently, ESC systems are further expected to proactively explore the
user’s problem by asking clarifying questions and help the user overcome the
problem by providing useful information or supportive suggestions, i.e.,
Initiative. Furthermore, the analysis of the conversation progress and the
emotional support metrics reveal three challenges in building a mixed-
initiative ESC system: 1) When should the system take the initiative during
the conversation? 2) What kind of information is required for the system to
initiate a subdialogue? 3) How could the system facilitate the mixed-
initiative interactions?

According to these challenges, we define the problem of mixed-initiative ESC,
which includes three sub-tasks: 1) Strategy Prediction to determine the mixed-
initiative strategy in the next turn, 2) Knowledge Selection to collect the
necessary knowledge for the next turn, and 3) Response Generation to produce
emotional support responses with appropriate mixed-initiative strategy and
knowledge. To tackle this problem, we propose a novel framework, named
Knowledge Enhanced Mixed-Initiative model (KEMI), to build a mixed-initiative
dialogue system for emotional support conversations with external domain-
specific knowledge. In detail, KEMI first employs a knowledge acquisition
module to acquire emotional support knowledge from a large-scale knowledge
graph on mental health dialogues. Specifically, we expand the user utterance
with generated commonsense knowledge as a query graph and then perform
subgraph retrieval over the knowledge graph. Secondly, a response generation
module conducts multi-task learning of strategy prediction and response
generation in a sequence-to-sequence manner to generate mixed-initiative
responses with external knowledge.

The main contributions of this work are summarized as follows: (1) To measure
the mixed-initiative interactions in ESC systems, we propose an innovative
analysis method, including an EAFR annotation schema and corresponding
emotional support metrics. (2) We propose a novel knowledge-enhanced mixed-
initiative framework for ESC, which retrieves external knowledge from mental
health knowledge graph by subgraph retrieval using the query graph expanded
with commonsense knowledge. (3) Experimental results show that the mixed
initiative is of great importance in ESC, and the proposed method effectively
outperforms existing methods on both content-preserving evaluation and mixed
initiative analyses.

##  2 Related Works

Emotional Support Conversation Similar to fine-grained sentiment analysis
Zhang et al. (2022, 2021c, 2021b) in conversations Li et al. (2022a); Zhang et
al. (2021a), early works on emotional chatting mainly investigate approaches
to detecting user emotions Li et al. (2017); Zhou et al. (2018) or
incorporating emotional signals into response generation Wei et al. (2019);
Song et al. (2019). As for empathetic dialogue systems Rashkin et al. (2019);
Welivita et al. (2021), evolving from emotion-aware response generation Lin et
al. (2019); Majumder et al. (2020) and emotional style transfer Sharma et al.
(2021), more efforts have been made on emotional reasoning techniques Li et
al. (2021); Kim et al. (2021); Gao et al. (2021); Cheng et al. (2022). Some
latest studies explore the utilization of external knowledge for enhancing the
model capability of emotion reasoning, including commonsense knowledge graph
Zhong et al. (2021); Li et al. (2022b), generative commonsense model Sabour et
al. (2021), and domain-specific knowledge Shen et al. (2020, 2022). Shen et
al. (2022) collectively exploit three kinds of external knowledge. Likewise,
many ESC systems also leverage commonsense knowledge for response generation
Tu et al. (2022); Peng et al. (2022). However, the commonsense knowledge is
rather abstractive without detailed information, so that it is less helpful
for the ESC system to generate meaningful and informative responses. In this
work, we employ the generative commonsense model for query expansion to
retrieve actual case knowledge from an external knowledge graph.

Role Type EAFR Definition Sample Utterances User Initiative Expression The
user describes details or expresses feelings My school was closed due to the
pandemic. about the situation. I feel so frustrated. The system requests for
information related to How are your feelings at that time? System Initiative
Action the problem or provides suggestions and infor- Deep breaths can help
people calm down. mation for helping the user solve the problem. Some
researches has found that … User Non-Initiative Feedback The user responds to
the system’s request or Okay, this makes me feel better. delivers opinions on
the system’s statement. No, I haven’t. The system conveys the empathy to the
user’s I understand you. I would also have been System Non-Initiative
Reflection emotion or shares similar experiences and really frustrated if that
happened to me. feelings to comfort the user. I’m sorry to hear about that.

Table 1: Definition and Examples for EAFR Schema Reflecting Patterns of
Initiative Switch between Dialogue Participants in Emotional Support
Conversations.

Mixed-initiative Dialogue Recent years have witnessed many efforts on
developing mixed-initiative conversational systems for various dialogues, such
as information-seeking dialogues Zamani et al. (2020); Aliannejadi et al.
(2019), open-domain dialogues Wu et al. (2019); Rachna et al. (2021); Lei et
al. (2022), recommendation dialogues Deng et al. (2021), conversational
question answering Deng et al. (2022a). Despite the importance of mixed
initiative in ESC systems, this area has not been investigated. One closely
related research scope is to recognize the conversation strategies Liu et al.
(2021); Pérez-Rosas et al. (2017) or the dialogue acts Malhotra et al. (2022);
Welivita and Pu (2020); Svikhnushina et al. (2022); Deng et al. (2022b) of the
utterances in ESC systems. However, these studies only focus on predicting the
support strategies, instead of actually involving mixed-initiative
interactions in ESC.

In addition, measuring mixed initiative is also regarded as an essential
perspective for assessing dialogue quality Vakulenko et al. (2021, 2020,
2019). Due to the high expenses in human evaluation, Sekulic et al. (2022) and
Zhang and Balog (2020) investigate user simulation for evaluating the mixed-
initiative interactions in conversational systems. In this work, we
investigate several metrics for measuring the characteristics of the mixed
initiative in ESC systems.

##  3 Preliminary Analysis

###  3.1 EAFR Schema & Metrics

Inspired by the ConversationShape Vakulenko et al. (2021) for the analysis of
mixed-initiative CIS systems, we first propose an EAFR annotation schema to
study the mixed initiative in ESC systems. The EAFR annotation schema
classifies the utterance in ESC into four categories w.r.t the role of
speakers and the type of initiative, including Expression (User-initiative),
Action (System-initiative), Feedback (User Non-Initiative), and Reflection
(System Non-Initiative). Definitions and examples of each type are presented
in Table 1.

Then, each utterance $i$ in a dialogue is annotated as a tuple
$(r_{i},t_{i},\bm{v}_{i},e_{i})$ for analysis.
$r_{i}\in\\{\text{User}(U),\text{System}(S)\\}$ denotes the speaker role.
$t_{i}\in\\{\text{Initiative}(I),\text{Non-Initiative}(N)\\}$ denotes the
initiative type. $\bm{v}_{i}\in\\{0,1\\}^{|V|}$ denotes the one-hot vocabulary
embeddings. $e_{i}\in[1,5]$ denotes the level of emotion intensity111A
decrease from the intensity reflects emotion improvement Liu et al. (2021)..
We further design four emotional support metrics for investigating patterns of
mixed initiative in ESC systems as follows:

  * •

Proactivity: how proactive is the system in the emotional support
conversation?

$\text{Pro}=\frac{1}{\sum_{i=1}^{n}\mathcal{I}(r_{i}=S)}\sum\nolimits_{i=1}^{n}\mathcal{I}(r_{i}=S,t_{i}=I)$
(1)

denotes the ratio of system-initiative interactions.

  * •

Information: how much information does the system contribute to the dialogue?

$\text{Inf}=\frac{\sum_{i=1}^{n}\sum_{k=1}^{|V|}\mathcal{I}(r_{i}=S,v_{ik}=1,\sum_{j=1}^{i-1}v_{jk}=0)}{\sum_{i=1}^{n}\mathcal{I}(r_{i}=S)}$
(2)

represents the average number of new frequent terms222We only consider
frequent terms that appear in the dialogue more than once. Standard pre-
processing pipeline is adopted: remove punctuation, tokenization, lowercase,
remove stopwords, and apply the English Snowball stemmer. that are introduced
by the system.

  * •

Repetition: how often does the system follow up on the topic introduced by the
user?

$\text{Rep}=\frac{\sum\limits_{i=1}^{n}\sum\limits_{k=1}^{|V|}\mathcal{I}(r_{i}=S,v_{ik}=1,\sum\limits_{j=1}^{i-1}v_{jk}[r_{j}=U]>0)}{\sum_{i=1}^{n}\mathcal{I}(r_{i}=S)}$
(3)

represents the average number of repeated frequent terms that are introduced
by the user and mentioned by the system.

  * •

Relaxation: how well does the system relax the emotional intensity of the
user?

$\text{Rel}_{i}[r_{i}=S]=e_{<i}[r_{<i}=U]-e_{>i}[r_{>i}=U]$ (4)
$\text{Rel}=\frac{1}{\sum_{i=1}^{n}\mathcal{I}(r_{i}=S)}\sum\nolimits_{i=1}^{n}\text{Rel}_{i}[r_{i}=S]$
(5)

represents the change of the user’s emotion intensity. $e_{<i}[r_{<i}=U]$ and
$e_{>i}[r_{>i}=U]$ denote the emotion intensity of the first user utterance
before and after the utterance $i$, respectively.

###  3.2 Analysis of Mixed Initiative in ESC

To reveal the necessity of incorporating mixed initiative into ESC systems, we
analyze the different interaction patterns between empathetic dialogues (ED)
and emotional support conversations (ESC): (i) EmpatheticDialogues Rashkin et
al. (2019), a dataset for ED that aims to provide empathetic responses for
comforting the help-seeker, and (ii) ESConv Liu et al. (2021), a dataset for
ESC that aims to not only reduce users’ emotional distress, but also help them
understand and overcome the issues they face.

Due to the space limitation, we present the detailed analysis in Appendix A,
including (i) the visualization of dialogue flow that indicates the initiative
patterns between the user and system (A.2); (ii) the visualization of
conversation progress that shows the phased change of the user’s emotion
intensity (A.3); and (iii) the evaluation of emotional support metrics that
quantifies different aspects of mixed-initiative interactions (A.4).

###  3.3 Challenges of Mixed Initiative in ESC

The preliminary analysis reveals the importance of mixed-initiative
interactions in ESC systems. Meanwhile, it is also challenging to balance the
mixed-initiative interactions, as overacting in one way or taking the
initiative inappropriately can be harmful to the emotional support
conversations. Based on these analyses, we identify three key challenges in
building a mixed-initiative ESC system:

1) When should the system take the initiative during the conversation? The
analysis of conversation progress (A.3) shows that taking initiative at
different phases of the conversation may lead to different impacts on the
user’s emotional state. In particular, support strategies or dialogue acts
attach great importance to conversational effectiveness in ESC Zhang and
Danescu-Niculescu-Mizil (2020); Tu et al. (2022). Therefore, it is a crucial
capability for the ESC system to determine whether to take the initiative at
each conversation turn.

2) What kind of information is required for the system to initiate a
subdialogue? The analysis of mixed initiative metrics (A.4) show that the
initiative system utterances are much informative than the non-initiative
ones. Therefore, it is of great importance to discover necessary information
and knowledge to make an appropriate mixed-initiative interaction. Researchers
Burleson (2003) in communication and sociology states that the helpfulness of
supportive statement is contingent on the following knowledge: (i) Affective
Knowledge, the emotion recognition of the user’s affective state, (ii) Causal
Knowledge, the emotional reasoning of stressors that cause the current
affective state of the user, and (iii) Cognitive Knowledge, the cognitive
analysis of coping processes to solve the core problematic situation that the
user faces.

3) How could the system facilitate the mixed-initiative interactions?  Since
the system in ESC ultimately provides a natural language utterance to interact
with the user, this challenge can be defined as a function that generates an
initiative-aware utterance based on the given information.

###  3.4 Problem Definition

Similar to the ED problem, the ESC problem is typically defined as: given the
dialogue context $\mathcal{C}=\\{u_{1},u_{2},...,u_{t}\\}$ and the description
of the user’s problematic situation $s$, the goal is to estimate a function
$p(r|\mathcal{C},s)$ that generates the target response $r$. In the light of
the challenges discussed in Section 3.3, we further define the mixed-
initiative emotion support conversation problem with the following three sub-
tasks, corresponding to the above three challenges:

1) Strategy Prediction predicts the support strategy $y$ that can be regarded
as the fine-grained initiative.

2) Knowledge Selection selects appropriate knowledge $k$ from the available
resources $\mathcal{K}$.

3) Response Generation generates the mixed-initiative response $r$ based on
the predicted strategy and the selected knowledge.

##  4 Method

Motivated by the analysis in the last section, we propose the KEMI framework
that aims to generate mixed-initiative responses with external knowledge. As
illustrated in Figure 2, KEMI contains two parts: 1) Knowledge Acquisition,
and 2) Mixed-initiative Response Generation.

Figure 2: Overview of KEMI. Each expanded query is represented as a graph to
retrieve subgraphs from HEAL, and each subgraph in HEAL can be regarded as an
actual case of emotional support conversations.

###  4.1 Knowledge Acquisition

Commonsense knowledge is widely adopted to enhance the emotion reasoning in
ESC systems. Despite the wide usage of commonsense knowledge in ESC systems,
it is usually succinct and lacks specific context information. We propose an
approach to retrieve relevant actual cases of ESC from a large-scale mental
health knowledge graph, namely HEAL Welivita and Pu (2022), for compensating
the deficiency of commonsense knowledge.

####  4.1.1 Query Expansion with COMET

Given the user utterance $u_{t}$ at the current turn $t$, a straight-forward
knowledge acquisition approach is to use $u_{t}$ as the query to directly
retrieve actual cases from the HEAL KG. However, there is limited information
provided by the user utterance, which may hinder the preciseness and
explainability of the knowledge retrieval. To this end, we exploit COMET
Bosselut et al. (2019), a commonsense knowledge generator, to expand the query
with multi-perspective additional information regarding the user’s affective
and cognitive state.

Specifically, the current user utterance $u_{t}$ is fed into COMET with five
special relation tokens,
$p\in\\{\texttt{[xReact]},\texttt{[xIntent]},\texttt{[xWant]},\texttt{[xNeed]}\\\
,\texttt{[xEffect]}\\}$, to generate commonsense inference $c_{p}$ for the
relation $p$, i.e., $c_{p}=\texttt{COMET}(p,u_{t})$. Definitions of each
commonsense relation can be found in Appendix B. Then the original user
utterance $u_{t}$ can be expanded with commonsense knowledge $\\{c_{p}\\}$.

####  4.1.2 Query Graph Construction

The actual case in HEAL Welivita and Pu (2022) is represented as a graph
structure. Specifically, we consider 4 out of 5 types of nodes in HEAL that
are related to response generation: 1) expectation: commonly asked questions
by the user in an emotional support conversation; 2) affective state:
emotional states associated with each speaker; 3) stressor: the cause of
emotional issues; and 4) response: frequent types of responses by the system
to address the user’s problems. Edges are constructed to build the connections
between nodes according to actual emotional support conversations. More
details of HEAL can be found in Appendix C.

In accordance with the HEAL knowledge graph, the relation [xReact], which
reveals the user’s emotional state, provides the same information as nodes in
HEAL with the type of affective state. The relation [xIntent], which reveals
the causes of the user’s current situation, also shares the same information
as nodes in HEAL with the type of stressor. The rest of relations, including
[xWant], [xNeed], and [xEffect], which reveal the user’s cognitive state, are
relevant to the responses for addressing the user’s problem. Therefore, the
expanded query $\hat{u}_{t}=\\{u_{t},\\{c_{p}\\}\\}$ can be represented as a
graph with abstractive entity descriptions, as shown in Figure 2.

####  4.1.3 Subgraph Retrieval

To avoid enumerating all the subgraphs in HEAL, which is a densely-connected
graph (over 2 million subgraphs), we propose a subgraph retrieval approach to
select the top relevant subgraphs to form a candidate set. We first retrieve
top-$K$ entities relevant to each abstractive entity description in the
expanded query graph $\hat{u}_{t}$. Specifically, we use sentence-BERT Reimers
and Gurevych (2019) to be an embedding-based retriever $f_{r}(\cdot)$ for
modeling the semantic similarity between the entities in the query and HEAL.
With the retrieved top-$K$ entities for each type of nodes, we merge them
based on the edge connections in the knowledge graph to induce candidate
subgraphs. Finally, we adopt top-$N$ candidate subgraphs as the retrieved
knowledge $\mathcal{K}$. The subgraphs are ranked by the sum of similarity
scores of each node in the subgraph
$E=\\{e_{\text{exp}},e_{\text{aff}},e_{\text{str}},e_{\text{resp}}\\}$:

$\displaystyle\textbf{Sim}(\hat{u}_{t},E)=$ $\displaystyle
f_{r}(u_{t},e_{\text{exp}})+f_{r}(c_{\text{xR}},e_{\text{aff}})+f_{r}(c_{\text{xI}},e_{\text{str}})$
(6)
$\displaystyle+f_{r}([c_{\text{xW}},c_{\text{xN}},c_{\text{xE}}],e_{\text{resp}}).$

###  4.2 Mixed-initiative Response Generation

Given the dialogue context $\mathcal{C}$ and the retrieved knowledge
$\mathcal{K}$, we first encode them into distributed representations with
contextualized encoders. Specifically, we add special tokens to differentiate
the roles of user and system as well as different types of knowledge as:

$\texttt{<context>}=\texttt{[situ.]},s,\texttt{[usr]},u_{1},\texttt{[sys]},u_{2},...$
$\texttt{<know.>}=\texttt{[xR.]},c_{\text{xR}},\texttt{[xI.]},...,\texttt{[Aff.]},e_{\text{aff}},...$

Pretrained language models (PLMs), e.g., GPT2 Radford et al. (2019), have
shown superior capability of generating high-quality responses in many
dialogue systems, especially those PLMs pretrained on dialogue corpus, e.g.,
BlenderBot Roller et al. (2021). To leverage the advantages of these
generative PLMs, we reformulate the mixed-initiative emotional support
conversation problem as a Seq2Seq problem, which linearizes the input and
output as a sequence of tokens as follows:

$X=\texttt{[CLS]},\texttt{<context>},\texttt{[know.]},\texttt{<know.>}_{i},...$
$Y=\texttt{[strategy]},y,\texttt{[response]},r$

where $X$ and $Y$ are the linearized input and output sequences for Seq2Seq
learning. Then the model is trained to maximize the negative log likelihood:

$\mathcal{L}=-\frac{1}{L}\sum\nolimits^{L}_{l=1}\log P(Y_{l}|Y_{<l};X).$ (7)

##  5 Experiment

###  5.1 Experimental Setups

Datasets We adopt the following two datasets for the evaluation: (i) ESConv
Liu et al. (2021), an emotional support conversation dataset, contains 1,300
dialogues with 38,365 utterances and 8 types of support strategies. We adopt
the original train/dev/test split; and (ii) MI Pérez-Rosas et al. (2016), a
motivational interviewing dataset, contains 284 counseling sessions with
22,719 utterances and 10 types of behavior strategies. We randomly split the
dataset for train/dev/test by 8:1:1333Since there is no speaker label in the
MI dataset, it is only adopted for response generation evaluation while the
analysis of mixed initiative is not applicable..

Evaluation Metrics As for automatic evaluation, we adopt Macro F1 as the
strategy prediction metric. Following previous studies Liu et al. (2021); Tu
et al. (2022), Perplexity (PPL), BLEU-$n$ (B-$n$), and ROUGE-L (R-L) are
included for the evaluation of response generation.

Baselines We provide extensive comparisons with both non-PLM and PLM-based
methods, including three Transformer-based methods (Transformer Vaswani et al.
(2017), MoEL Lin et al. (2019), and MIME Majumder et al. (2020)) and four
BlenderBot-based methods (BlenderBot Roller et al. (2021), BlenderBot-Joint
Liu et al. (2021), GLHG Peng et al. (2022)444Since GLHG leverages the problem
type as an additional label, we also report the ablation result for a fair
comparison, i.e., GLHG w/o $\mathcal{L}_{2}$ Loss. , and MISC Tu et al.
(2022)555Due to a different train/test split adopted in Tu et al. (2022), we
reproduce the performance of MISC on the standard split of ESConv Liu et al.
(2021).). Details about these baselines can be found in Appendix D.

Implementation Details KEMI is based on the BlenderBot model Roller et al.
(2021). Following previous BlenderBot-based models Liu et al. (2021); Peng et
al. (2022); Tu et al. (2022), we adopt the small
version666https://huggingface.co/facebook/blenderbot_small-90M of BlenderBot
in experiments. The learning rate and the warmup step are set to be 3e-5 and
100, respectively. The max input sequence length and the max target sequence
length are 160 and 40, respectively. We retrieve the top-$1$ subgraph from
HEAL as the knowledge. The training epoch is set to 5 and the best model is
saved according to the PPL score in the dev
set.777https://github.com/dengyang17/KEMI

###  5.2 Overall Performance

Table 2 and Table 3 summarize the experimental results on the ESConv and MI
dataset, respectively. Among the baselines, BlenderBot-based methods largely
outperform Transformer-based methods by leveraging the valuable pretrained
knowledge. GLHG and MISC effectively exploit the commonsense knowledge to
improve the performance of response generation. Besides, the joint learning
with strategy prediction task is beneficial to the performance of response
generation. Finally, KEMI substantially outperforms other methods with a
noticeable margin. This indicates the domain-specific actual case knowledge
from HEAL can alleviate the reliance on large-scale PLMs. Compared with
commonsense knowledge, the knowledge from HEAL is much more effective in
predicting support strategies, as this relevant knowledge can serve as an real
example for guiding the system to respond.

Model F1$\uparrow$ PPL$\downarrow$ B-2$\uparrow$ B-4$\uparrow$ R-L$\uparrow$
Transformer∗ Vaswani et al. (2017) - 81.55 5.66 1.31 14.68 MoEL∗ Lin et al.
(2019) - 62.93 5.02 1.14 14.21 MIME∗ Majumder et al. (2020) - 43.27 4.82 1.03
14.83 BlenderBot∗∗ Roller et al. (2021) - 16.23 5.45 - 15.43 GLHG∗ Peng et al.
(2022) - 15.67 7.57 2.13 16.37 GLHG w/o $\mathcal{L}_{2}$ Loss∗ Peng et al.
(2022) - - 6.15 1.75 15.87 BlenderBot-Joint Liu et al. (2021) 19.23 16.15 5.52
1.29 15.51 MISC Tu et al. (2022) 19.89 16.08 7.62 2.19 16.40 KEMI 24.66† 15.92
8.31† 2.51† 17.05†

Table 2: Experimental results on ESConv. ∗ and ∗∗ indicate the results
reported in Peng et al. (2022) and Liu et al. (2021) respectively. Other
results are reproduced. † indicates statistically significant improvement
($p$<0.05) over the best baseline.

Model F1$\uparrow$ PPL$\downarrow$ B-2$\uparrow$ B-4$\uparrow$ R-L$\uparrow$
Transformer Vaswani et al. (2017) - 65.52 6.23 1.52 15.04 BlenderBot Roller et
al. (2021) - 16.06 6.57 1.66 15.64 BlenderBot-Joint Liu et al. (2021) 22.66
14.74 7.28 2.18 16.41 MISC Tu et al. (2022) 22.68 14.33 7.75 2.30 17.11 KEMI
25.91† 13.84† 8.52† 2.72† 18.00†

Table 3: Experimental results on MI Counseling.

###  5.3 Human Evaluation

Following previous studies Liu et al. (2021); Peng et al. (2022), we conduct
human evaluation to compare the generated responses from two given models on
five aspects: 1) Fluency: which model’s response is more fluent? 2)
Identification: which model’s response is more skillful in identifying the
user’s problem? 3) Comforting: which model’s response is better at comforting
the user? 4) Suggestion: which model can give more helpful and informative
suggestions? 5) Overall: which model’s response is generally better? We
randomly sample 100 dialogues from ESConv and three annotators are asked to
determine the Win/Tie/Lose for each comparison.

vs. | BlenderBot-Joint | MISC
---|---|---
Win | Tie | Loss | Win | Tie | Loss
Flu. | 26% | 51% | 23% | 37% | 47% | 16%
Ide. | 50% | 38% | 12% | 46% | 30% | 24%
Com. | 46% | 40% | 14% | 44% | 30% | 26%
Sug. | 52% | 22% | 26% | 52% | 16% | 28%
Ove. | 62% | 20% | 18% | 70% | 12% | 18%
Table 4: Human evaluation results (KEMI vs.).

Table 4 presents the human evaluation results. We compare the generated
responses from KEMI with those produced by other two baselines, BlenderBot-
Joint and MISC. The results show that KEMI achieves remarkable improvement on
initiative interactions, including Identification and Suggestion.
Consequently, KEMI can generate more satisfactory and helpful responses than
other methods, according to the Overall metric.

###  5.4 Ablation Study

In order to investigate the effect of each sub-task and each type of knowledge
on the final performance, we report the experimental results of the ablation
study in Table 5. In general, both the strategy prediction and the knowledge
selection tasks as well as all types of knowledge contribute to the final
performance more or less. There are several notable observations in detailed
comparisons: (i) The knowledge from HEAL is the key to the improvement on the
strategy prediction task, since the actual case knowledge can provide a good
guidance for the next support strategy. (ii) Different from discarding the
actual case knowledge (w/o HEAL), discarding the commonsense knowledge (w/o
COMET) brings a positive effect on the fluency metrics (PPL), as the
commonsense knowledge is not a natural sentence. However, the COMET
contributes more on the content-preserving metrics (BLEU and ROUGE) than the
HEAL, indicating that the succinct commonsense knowledge can be more precise.
(iii) Among the three types of knowledge, cognitive knowledge is the most
effective one for both strategy prediction and response generation tasks. (iv)
Using Oracle strategy and Oracle knowledge substantially improves the overall
performance, which demonstrates the effectiveness of considering these two
sub-tasks in ESC systems. The performance gap between KEMI and Oracle also
shows that the knowledge selection is very challenging and there is still much
room for improvement.

Strategy Knowledge F1$\uparrow$ PPL$\downarrow$ B-2$\uparrow$ R-L$\uparrow$ -
- - 16.23 5.45 15.43 - KEMI - 16.16 6.54 16.21 Joint KEMI 24.66 15.92 8.31
17.05 Joint w/o COMET 23.26 15.74 7.60 16.47 Joint w/o HEAL 19.99 16.08 7.98
16.92 Joint w/o Affective 22.68 16.08 8.22 16.98 Joint w/o Causal 23.14 15.94
8.16 16.92 Joint w/o Cognitive 20.24 16.22 7.62 16.64 Joint Oracle 32.38 12.79
18.45 28.01 Oracle KEMI - 15.92 9.75 18.81 Oracle Oracle - 12.78 19.11 28.88

Table 5: Ablation study. Oracle knowledge is obtained by the lexical match
between the reference response and the candidate knowledge from HEAL.

Proactivity Information Repetition Relaxation Init. Non. Init. Non. All Init.
Non. All Init. Non. All BB 0.36 0.64 1.79 1.32 1.48 1.00 1.11 1.07 -0.01 0.11
0.07 BB-J 0.68 0.32 1.89 1.18 1.66 1.18 1.09 1.15 0.01 0.07 0.03 MISC 0.61
0.39 1.91 1.25 1.65 1.16 1.12 1.14 0.00 0.04 0.02 KEMI 0.45 0.55 2.04 1.40
1.68 1.18 1.09 1.13 0.09 0.13 0.11 REF 0.51 0.49 3.09 3.01 3.05 1.12 1.06 1.09
0.10 0.13 0.11

Table 6: Emotional support metrics. BB and BB-J denote BlenderBot and
BlenderBot-Joint.

###  5.5 Analysis of Mixed Initiative

We conduct the mixed initiative analysis introduced in Section 3.2 over the
proposed KEMI method and other baselines. Since the calculation of the
Relaxation metric in Eq.(4) requires the emotion intensity score of the user
feedback, we adopt a model-based user simulator for automatic evaluation,
which is described in Appendix A.1.3.

####  5.5.1 Emotional Support Metrics

Figure 3: The distribution of system utterance initiative (the stack plot) and
the user’s emotion intensity change (the bar chart) at different conversation
progress. Higher scores of the emotion intensity change represent better
emotion improvement of users. Figure 4: Case study. Bold terms denote new
(red) and repeated (blue) frequent terms respectively.

Table 6 summarizes the results of the four emotional support metrics for the
generated responses from four BlenderBot-based methods and the reference
responses in the test set. Note that, for a fair comparison, we also adopt
Eq.(9) to calculate the Relaxation metric for the reference responses in the
test set (i.e., REF). It can be observed that (i) As for the Proactivity
metric, BlenderBot tends to act passively in ESC. While BlenderBot-Joint and
MISC overly take the initiative after simply taking into account the support
strategies. KEMI effectively balances the proportion of initiative and non-
initiative interactions in ESC. (ii) With the actual case knowledge, KEMI can
generate much informative responses than other baselines w.r.t the Information
metric. However, there is still a large gap to reach the reference responses.
(iii) Indeed, it is relatively easier to generate responses that repeat the
previous information w.r.t the Repetition metric. (iv) KEMI outperforms other
baselines in terms of the Relaxation metric on the initiative interactions
with a large margin, which shows the superiority of KEMI on taking the
initiative role for helping the user to solve emotional problems.

####  5.5.2 Conversation Progress

We conduct the conversation progress analysis by dividing the whole
conversation into five equal length intervals and observing the change of
users’ emotion intensity levels at each conversation phase. As the results
shown in Figure 3, we observe that BlenderBot and MISC have a clear
inclination to take non-initiative and initiative interactions in all stages
of the conversation, respectively. Our KEMI method shares a more similar
progress as the reference conversation with a balanced interaction pattern.
More importantly, the initiative responses generated by KEMI has a more
positive impact on the user’s emotional intensity than other baselines,
especially in the last two stages of the conversation. This result indicates
that KEMI effectively takes the initiative to generate responses that can
provide suggestions or information for relaxing the help-seekers by solving
their emotional problems.

###  5.6 Case Study

To intuitively show the superiority of KEMI over other baselines, Figure 4
presents a case study of generated responses with the scores of mixed-
initiative metrics. In the reference response, the system takes the initiative
to provide useful suggestions to the user for solving her/his problem, which
effectively reduce the user’s emotional intensity. Among the generated
responses, BlenderBot and BlenderBot-Joint decide to convey empathy to the
user by paraphrasing the previous information, while MISC and KEMI proactively
initiate a discussion about potential solutions to the problem. Based on the
Relaxation metric, two initiative responses can better comfort the emotional
intensity of the user than two non-initiative responses. Furthermore, KEMI can
generate more informative and specific responses with actual case knowledge.

##  6 Conclusions

In this paper, we design a novel analysis framework for analyzing the feature
of mixed initiative in ESC. The analysis demonstrates the necessity and
importance of mixed-initiative interactions in ESC systems. To this end, we
propose the KEMI framework to tackle the problem of mixed-initiative ESC. KEMI
first retrieves actual case knowledge from a large-scale mental health
knowledge graph with query expansion and subgraph retrieval. Then KEMI
performs multi-task learning of strategy prediction and response generation
with the retrieved knowledge. Extensive experiments show that KEMI outperforms
existing methods on both automatic and human evaluation. The analysis also
shows the effectiveness of incorporating actual case knowledge and the
superiority of KEMI on the mixed-initiative interactions.

## Limitations

In this section, we analyze the limitations of this work:

  * •

As it is the first attempt to analyze the mixed-initiative interactions in
emotional support conversations, the proposed metrics can be further improved
for more robust evaluation.

  * •

Since the knowledge retrieval is not the focus of this work, we did not spend
much space on discussing the choice of different retrieval methods. As shown
in Table 5, there is still much room for improving the knowledge retrieval
from a large scale knowledge graph. It is also worth studying more efficient
retrieval methods for retrieving knowledge from a densely connected KG.

  * •

The proposed method requires an additional mental health related knowledge
graph constructed by experts or knowledgeable workers, which is probably
difficult to obtain in some applications. However, different from other
knowledge-intensive tasks that can be benefited from open-domain knowledge
(e.g., Wikipedia), it attaches great importance in the professionals of the
knowledge for building a helpful and safe ESC system.

## Ethical Considerations

The datasets adopted are publicly available and widely studied benchmarks
collected from professionals or well-trained annotators. All personally
identifiable and sensitive information, e.g., user and platform identifiers,
in these dataset has been filtered out. We do not make any treatment
recommendations or diagnostic claims. Compared with existing methods for
emotional support conversations, the proposed method can be regarded as one
step further to a more safer ESC system. The proposed method retrieves
knowledge from a well-established mental health knowledge graph, which can be
maintained by filtering out harmful information when applying into
applications. Then the knowledge-enhanced approach can alleviate the
randomness during the response generation and provide the guidance towards
more positive responses. In order to prevent the happening of unsafe cases,
the analysis of emotion intensity prediction can also serve as an alarming
mechanism that calls for handoffs to an actual psychologist.

## References

  * Aliannejadi et al. (2019) Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019.  Asking clarifying questions in open-domain information-seeking conversations.  In _SIGIR 2019_ , pages 475–484.
  * Allen et al. (1999) James E Allen, Curry I Guinn, and Eric Horvtz. 1999.  Mixed-initiative interaction.  _IEEE Intelligent Systems and their Applications_ , 14(5):14–23.
  * Althoff et al. (2016) Tim Althoff, Kevin Clark, and Jure Leskovec. 2016.  Large-scale analysis of counseling conversations: An application of natural language processing to mental health.  _Trans. Assoc. Comput. Linguistics_ , 4:463–476.
  * Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019.  COMET: commonsense transformers for automatic knowledge graph construction.  In _ACL 2019_ , pages 4762–4779.
  * Burleson (2003) Brant R Burleson. 2003.  Emotional support skills.  In _Handbook of communication and social interaction skills_ , pages 569–612. Routledge.
  * Cheng et al. (2022) Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. 2022.  Improving multi-turn emotional support dialogue generation with lookahead strategy planning.  _CoRR_ , abs/2210.04242.
  * Deng et al. (2023) Yang Deng, Wenqiang Lei, Wai Lam, and Tat-Seng Chua. 2023.  A survey on proactive dialogue systems: Problems, methods, and prospects.  _CoRR_ , abs/2305.02750.
  * Deng et al. (2022a) Yang Deng, Wenqiang Lei, Wenxuan Zhang, Wai Lam, and Tat-Seng Chua. 2022a.  PACIFIC: towards proactive conversational question answering over tabular and textual data in finance.  In _EMNLP 2022_ , pages 6970–6984.
  * Deng et al. (2021) Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021.  Unified conversational recommendation policy learning via graph-based reinforcement learning.  In _SIGIR 2021_ , pages 1431–1441.
  * Deng et al. (2022b) Yang Deng, Wenxuan Zhang, Wai Lam, Hong Cheng, and Helen Meng. 2022b.  User satisfaction estimation with sequential dialogue act modeling in goal-oriented conversational systems.  In _WWW ’22: The ACM Web Conference 2022_ , pages 2998–3008.
  * Gao et al. (2021) Jun Gao, Yuhan Liu, Haolin Deng, Wei Wang, Yu Cao, Jiachen Du, and Ruifeng Xu. 2021\.  Improving empathetic response generation by recognizing emotion cause in conversations.  In _Findings of ACL: EMNLP 2021_ , pages 807–819.
  * Kim et al. (2021) Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2021.  Perspective-taking and pragmatics for generating empathetic responses focused on emotion causes.  In _EMNLP 2021_ , pages 2227–2240.
  * Kraus et al. (2020) Matthias Kraus, Nicolas Wagner, and Wolfgang Minker. 2020.  Effects of proactive dialogue strategies on human-computer trust.  In _UMAP 2020_ , pages 107–116.
  * Lei et al. (2022) Wenqiang Lei, Yao Zhang, Feifan Song, Hongru Liang, Jiaxin Mao, Jiancheng Lv, Zhenglu Yang, and Tat-Seng Chua. 2022.  Interacting with non-cooperative user: A new paradigm for proactive dialogue policy.  In _SIGIR 2022_.
  * Li et al. (2022a) Bobo Li, Hao Fei, Fei Li, Yuhan Wu, Jinsong Zhang, Shengqiong Wu, Jingye Li, Yijiang Liu, Lizi Liao, Tat-Seng Chua, and Donghong Ji. 2022a.  Diaasq : A benchmark of conversational aspect-based sentiment quadruple analysis.  _CoRR_ , abs/2211.05705.
  * Li et al. (2022b) Qintong Li, Piji Li, Zhumin Chen, and Zhaochun Ren. 2022b.  Knowledge bridging for empathetic dialogue generation.  In _AAAI 2022_.
  * Li et al. (2021) Yanran Li, Ke Li, Hongke Ning, Xiaoqiang Xia, Yalong Guo, Chen Wei, Jianwei Cui, and Bin Wang. 2021.  Towards an online empathetic chatbot with emotion causes.  In _SIGIR 2021_ , pages 2041–2045.
  * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017.  Dailydialog: A manually labelled multi-turn dialogue dataset.  In _IJCNLP_.
  * Lin et al. (2019) Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, and Pascale Fung. 2019.  Moel: Mixture of empathetic listeners.  In _EMNLP-IJCNLP 2019_ , pages 121–132.
  * Liu et al. (2021) Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021.  Towards emotional support dialog systems.  In _ACL/IJCNLP 2021_ , pages 3469–3483.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Lokala et al. (2022) Usha Lokala, Aseem Srivastava, Triyasha Ghosh Dastidar, Tanmoy Chakraborty, Md Shad Akthar, Maryam Panahiazar, and Amit P. Sheth. 2022.  A computational approach to understand mental health from reddit: Knowledge-aware multitask learning framework.  In _ICWSM 2022_.
  * Majumder et al. (2020) Navonil Majumder, Pengfei Hong, Shanshan Peng, Jiankun Lu, Deepanway Ghosal, Alexander F. Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020.  MIME: mimicking emotions for empathetic response generation.  In _EMNLP 2020_.
  * Malhotra et al. (2022) Ganeshan Malhotra, Abdul Waheed, Aseem Srivastava, Md. Shad Akhtar, and Tanmoy Chakraborty. 2022.  Speaker and time-aware joint contextual learning for dialogue-act classification in counselling conversations.  In _WSDM 2022_.
  * Peng et al. (2022) Wei Peng, Yue Hu, Luxi Xing, Yuqiang Xie, Yajing Sun, and Yunpeng Li. 2022.  Control globally, understand locally: A global-to-local hierarchical graph network for emotional support conversation.  _CoRR_ , abs/2204.12749.
  * Pérez-Rosas et al. (2016) Verónica Pérez-Rosas, Rada Mihalcea, Kenneth Resnicow, Satinder Singh, and Lawrence C. An. 2016.  Building a motivational interviewing dataset.  In _CLPsych@NAACL-HLT 2016_ , pages 42–51.
  * Pérez-Rosas et al. (2017) Verónica Pérez-Rosas, Rada Mihalcea, Kenneth Resnicow, Satinder Singh, Lawrence C. An, Kathy J. Goggin, and Delwyn Catley. 2017.  Predicting counselor behaviors in motivational interviewing encounters.  In _EACL 2017_ , pages 1128–1137.
  * Pfefferbaum and North (2020) Betty Pfefferbaum and Carol S North. 2020.  Mental health and the covid-19 pandemic.  _New England Journal of Medicine_ , 383(6):510–512.
  * Rachna et al. (2021) Konigari Rachna, Saurabh Ramola, Vijay Vardhan Alluri, and Manish Shrivastava. 2021\.  Topic shift detection for mixed initiative response.  In _SIGdial 2021_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In _ACL 2019_ , pages 5370–5381.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _EMNLP-IJCNLP 2019_ , pages 3980–3990.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021\.  Recipes for building an open-domain chatbot.  In _EACL 2021_ , pages 300–325.
  * Sabour et al. (2021) Sahand Sabour, Chujie Zheng, and Minlie Huang. 2021.  CEM: commonsense-aware empathetic response generation.  _CoRR_ , abs/2109.05739.
  * Saha et al. (2021) Tulika Saha, Saraansh Chopra, Sriparna Saha, Pushpak Bhattacharyya, and Pankaj Kumar. 2021.  A large-scale dataset for motivational dialogue system: An application of natural language generation to mental health.  In _IJCNN 2021_ , pages 1–8.
  * Saha et al. (2022) Tulika Saha, Saichethan Miriyala Reddy, Sriparna Saha, and Pushpak Bhattacharyya. 2022.  Mental health disorder identification from motivational conversations.  _IEEE Transactions on Computational Social Systems_.
  * Sekulic et al. (2022) Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. 2022.  Evaluating mixed-initiative conversational search systems via user simulation.  In _WSDM 2022_ , pages 888–896.
  * Sharma et al. (2021) Ashish Sharma, Inna W. Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2021\.  Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach.  In _WWW 2021_ , pages 194–205.
  * Shen et al. (2022) Siqi Shen, Verónica Pérez-Rosas, Charles Welch, Soujanya Poria, and Rada Mihalcea. 2022.  Knowledge enhanced reflection generation for counseling dialogues.  In _ACL 2022_ , pages 3096–3107.
  * Shen et al. (2020) Siqi Shen, Charles Welch, Rada Mihalcea, and Verónica Pérez-Rosas. 2020.  Counseling-style reflection generation using generative pretrained transformers with augmented context.  In _SIGdial 2020_ , pages 10–20.
  * Song et al. (2019) Zhenqiao Song, Xiaoqing Zheng, Lu Liu, Mu Xu, and Xuanjing Huang. 2019.  Generating responses with a specific emotion in dialog.  In _ACL 2019_ , pages 3685–3695.
  * Suh et al. (2021) Jina Suh, Eric Horvitz, Ryen W. White, and Tim Althoff. 2021.  Population-scale study of human needs during the COVID-19 pandemic: Analysis and implications.  In _WSDM 2021_ , pages 4–12.
  * Svikhnushina et al. (2022) Ekaterina Svikhnushina, Iuliana Voinea, Anuradha Welivita, and Pearl Pu. 2022.  A taxonomy of empathetic questions in social dialogs.  In _ACL 2022_ , pages 2952–2973.
  * Tu et al. (2022) Quan Tu, Yanran Li, Jianwei Cui, Bin Wang, Ji-Rong Wen, and Rui Yan. 2022.  MISC: A mixed strategy-aware model integrating COMET for emotional support conversation.  In _ACL 2022_ , pages 308–319.
  * Vakulenko et al. (2020) Svitlana Vakulenko, Evangelos Kanoulas, and Maarten de Rijke. 2020.  An analysis of mixed initiative and collaboration in information-seeking dialogues.  In _SIGIR 2020_ , pages 2085–2088.
  * Vakulenko et al. (2021) Svitlana Vakulenko, Evangelos Kanoulas, and Maarten de Rijke. 2021.  A large-scale analysis of mixed initiative in information-seeking dialogues for conversational search.  _ACM Trans. Inf. Syst._ , 39(4):49:1–49:32.
  * Vakulenko et al. (2019) Svitlana Vakulenko, Kate Revoredo, Claudio Di Ciccio, and Maarten de Rijke. 2019\.  QRFA: A data-driven model of information-seeking dialogues.  In _ECIR 2019_ , pages 541–557.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS 2017_ , pages 5998–6008.
  * Wei et al. (2019) Wei Wei, Jiayi Liu, Xianling Mao, Guibing Guo, Feida Zhu, Pan Zhou, and Yuchong Hu. 2019.  Emotion-aware chat machine: Automatic emotional response generation for human-like emotional interaction.  In _CIKM 2019_ , pages 1401–1410.
  * Welivita and Pu (2020) Anuradha Welivita and Pearl Pu. 2020.  A taxonomy of empathetic response intents in human social conversations.  In _COLING 2020_ , pages 4886–4899.
  * Welivita and Pu (2022) Anuradha Welivita and Pearl Pu. 2022.  Heal: A knowledge graph for distress management conversations.  In _AAAI 2022_.
  * Welivita et al. (2021) Anuradha Welivita, Yubo Xie, and Pearl Pu. 2021.  A large-scale dataset for empathetic response generation.  In _EMNLP 2021_ , pages 1251–1264.
  * Wu et al. (2019) Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019.  Proactive human-machine conversation with explicit conversation goal.  In _ACL 2019_.
  * Zamani et al. (2020) Hamed Zamani, Susan T. Dumais, Nick Craswell, Paul N. Bennett, and Gord Lueck. 2020\.  Generating clarifying questions for information retrieval.  In _WWW 2020_ , pages 418–428.
  * Zhang and Danescu-Niculescu-Mizil (2020) Justine Zhang and Cristian Danescu-Niculescu-Mizil. 2020.  Balancing objectives in counseling conversations: Advancing forwards or looking backwards.  In _ACL 2020_ , pages 5276–5289.
  * Zhang and Balog (2020) Shuo Zhang and Krisztian Balog. 2020.  Evaluating conversational recommender systems via user simulation.  In _KDD 2020_ , pages 1512–1520.
  * Zhang et al. (2021a) Wenxuan Zhang, Yang Deng, Xin Li, Lidong Bing, and Wai Lam. 2021a.  Aspect-based sentiment analysis in question answering forums.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4582–4591.
  * Zhang et al. (2021b) Wenxuan Zhang, Yang Deng, Xin Li, Yifei Yuan, Lidong Bing, and Wai Lam. 2021b.  Aspect sentiment quad prediction as paraphrase generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021_ , pages 9209–9219.
  * Zhang et al. (2021c) Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021c.  Towards generative aspect-based sentiment analysis.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021_ , pages 504–510.
  * Zhang et al. (2022) Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2022.  A survey on aspect-based sentiment analysis: Tasks, methods, and challenges.  _CoRR_ , abs/2203.01054.
  * Zhong et al. (2021) Peixiang Zhong, Di Wang, Pengfei Li, Chen Zhang, Hao Wang, and Chunyan Miao. 2021\.  CARE: commonsense-aware emotional response generation with latent concepts.  In _AAAI 2021_ , pages 14577–14585.
  * Zhou et al. (2018) Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018.  Emotional chatting machine: Emotional conversation generation with internal and external memory.  In _AAAI 2018_ , pages 730–739.

## Appendix

##  Appendix A Analysis of Mixed Initiative

###  A.1 Tools for Analysis

We introduce two models that are adopted as off-the-shelf tools for the
analysis of mixed initiative.

####  A.1.1 Utterance Initiative Classification

To facilitate automatic analysis of utterance initiative, we train two
utterance classification models by fine-tuning the pre-trained
RoBERTa${}_{\text{large}}$ model Liu et al. (2019) on the ESConv Liu et al.
(2021) dataset, one for system utterance classification and the other for user
utterance classification. We concatenate the previous utterance from another
participant and the current utterance as the input for the binary initiative
classification, either Initiative (I) or Non-Initiative (N). However, there is
no initiative label in ESConv. Therefore, we manually annotate the initiative
labels, I or N, for each utterance according to the EAFR schema. The resulting
dataset contains $\sim$38K utterance-label pairs (E: 13K, A: 9K, F: 7K, R:
10K).

####  A.1.2 Emotion Intensity Prediction

Similarly, we also fine-tune an emotion intensity prediction model
$f_{e}(\cdot)$ based on the pre-trained RoBERTa${}_{\text{large}}$ model Liu
et al. (2019) on the ESConv Liu et al. (2021) dataset. Given a user utterance,
the model aims to predict the negative emotion intensity level
$e_{ij}=f_{e}(u_{ij})$, ranging from 1 to 5, which indicates the user’s
emotional state. In ESConv, the initial and final emotion intensity levels of
the user have already been annotated. Therefore, we regard the first user
utterance after greetings to match with the initial emotion intensity, while
the last user utterance before goodbyes to match with the final emotion
intensity. The resulting dataset contains 2,450 utterance-label pairs (1: 331,
2: 506, 3: 557, 4: 629, 5: 427).

####  A.1.3 User Simulator

Inspired by the evaluation of mixed-initiative CIS Sekulic et al. (2022), we
simulate a user based on a large-scale generative language model, namely
BlenderBot Roller et al. (2021). In our case, we fine-tune a semantically-
conditioned generation model $g(\cdot)$, guided by the underlying problematic
situation:

$p_{g}(\bm{a}|s,\mathcal{C},r)=\prod\nolimits^{L}_{l=1}p_{g}(a_{l}|a_{<l};s,\mathcal{C},r),$
(8)

where $a$ is the user’s feedback to the generated response $r$. The generation
model is fine-tuned on the whole dataset, including the test set. If the ESC
system generates a perfect response, the user simulator should give the
ground-truth feedback as the real user.

We adopt the same utterance initiative classification model and emotion
intensity prediction model $f_{e}(\cdot)$ described in Appendix A.1.2 to
annotate the generated response. The annotation results are used for
calculating the emotional support metrics. In particular, the calculation of
Relaxation metric involves the user’s emotion intensity after receiving the
generated response. The user simulator $g(\cdot)$ is employed to simulate the
user’s feedback. Then the calculation of the Relaxation metric in Eq.(4)
becomes:

$\text{Rel}_{i}[r_{i}=S]=f_{e}(u_{<i}[r_{<i}=U])-f_{e}(g(s,\mathcal{C},r)).$
(9)

Figure 5: Dialogue flow. H, B, E, A, F, and R denote Hi, Bye, Expression,
Action, Feedback, and Reflection. The color intensity denotes the proportions
of the utterance labels and the initiative transitions.

###  A.2 Dialogue Flow

Following previous studies on mixed-initiative CIS systems Vakulenko et al.
(2019, 2021), we draw the dialogue flow diagram to observe the initiative
switch patterns between dialogue participants in ED and ESC. As shown in
Figure 5, the circles represent the beginning and ending of the dialogue,
while the boxes represent the EAFR utterance labels. The color intensity
denotes the proportions of the utterance labels and the initiative
transitions. There are several notable observations: (i) As for the proportion
of EAFR labels, Expression and Reflection constitute the majority of the
utterances in ED, while four labels are more equally distributed in ESC. (ii)
As for the beginning and ending of dialogues, users are more often to take the
initiative to start a conversation in ED, and the dialogue will be ended by
the system. Differently, in ESC, the conversation is usually started by the
system. (iii) As for the initiative switches, most of cases in ED are that
users express their feelings and then the system tries to comfort them with
empathy. However, the proportion of each type of initiative transitions in ESC
is relatively equal. Therefore, we conclude that the system in ED generally
serves as a passive role, while the system in ESC needs to switch the
initiative role during the conversation.

###  A.3 Conversation Progress

We analyze the conversation progress by dividing the whole conversation into
five equal length intervals. To alleviate the noise from greeting (Hi) and
farewell (Bye) utterances, we heuristically identify these utterances by
rules, e.g., containing “Hi/Hello” at the beginning or “Bye/Goodbye” at the
end of the conversation. Specifically, we compute the distribution of
initiative labels for system utterances and the average change of emotion
intensity levels at each conversation phase. As shown in Figure 6, under both
cases, the system tends to take the initiative at the beginning of the
conversation for exploring the user’s problem, while acting passively at the
latter stage of the conversation. Interestingly, at the early phase of
conversations, compared with non-initiative utterances, system-initiative ones
fail to relax the emotion intensity of help-seekers in ESC. This is because
the request for information from users to understand their problems is likely
to raise users’ negative emotions. However, at the latter stage of the
conversation, initiative utterances can better lower down users’ intensity
levels, leading to a higher emotional intensity change rate than non-
initiative ones. This indicates that (i) the timing for system-initiative
interactions is important, and (ii) it is more helpful to provide suggestions
or information for users to solve the problem when the emotion of users has
been eased.

Figure 6: The distribution of utterance initiative (the stack plot) and the
emotion intensity change (the bar chart) at different conversation progress.

Proactivity Information Repetition Relaxation Init. Non. Init. Non. All Init.
Non. All Init. Non. All ED 0.28 0.72 2.14 2.69 2.46 0.42 0.44 0.43 0.83 0.82
0.83 ESC 0.48 0.52 3.32 3.06 3.19 1.06 1.18 1.12 0.16 0.20 0.18

Table 7: Comparisons on emotional support metrics.

###  A.4 Emotional Support Metrics

Table 7 summarizes the scores of the emotional support metrics introduced in
Section 3.1 for two datasets. Firstly, the proportion of system-initiative
interactions in ESC is much higher than that in ED, showing the importance of
mixed initiative in ESC systems. Secondly, system-initiative utterances
provide more information than non-initiative utterances in ESC, while an
opposite result is observed in ED. This shows that the ESC system provides
informative responses when taking the initiative. Thirdly, in both datasets,
there is more repetitive information in non-initiative system utterances than
initiative ones, indicating that reflection is more important in non-
initiative interactions. Last but not least, the average relaxation score in
ESC is much lower than that in ED. We attribute this to two reasons: (i)
Empathetic responses have more positive effects on the user’s emotions. (ii)
The system-initiative interactions sometimes may increase the user’s emotion
intensity, as discussed in Appendix A.3.

##  Appendix B Definition of COMET Relations

We adopt five types of commonsense relations in COMET Bosselut et al. (2019),
whose original definitions are as follows:

  * •

xEffect: The effect that the event would have on Person X.

  * •

xIntent: The reason why X would cause the event.

  * •

xNeed: What Person X might need to do before the event.

  * •

xReact: The reaction that Person X would have to the event.

  * •

xWant: What Person X may want to do after the event.

##  Appendix C Details of HEAL

Stressor Expectation Response Affect. State Stressor 4,363 9,801 - -
Expectation 9,801 3,050 26,628 3,050 Response - 26,628 13,416 - Affect. State
- 3,050 - 41

Table 8: Statistics of HEAL adopted in our experiments.

HEAL Welivita and Pu (2022) is a knowledge graph developed upon 1M distress
discussions and their corresponding consoling responses curated from mental
health support conversations. It consists of 22K nodes with five different
types: stressors, expectations, responses, feedback, and affective states
associated with distress dialogues, and forms 104K connections between
different types of nodes. The statistics of the adopted HEAL are presented in
Table 8.

##  Appendix D Baselines

We provide extensive comparisons with the following strong baselines,
including both non-PLM and PLM-based methods:

  * •

Transformer Vaswani et al. (2017) for Seq2Seq response generation.

  * •

MoEL Lin et al. (2019) is a Transformer-based model that involves multi-
decoders to enhance the empathy for different emotions.

  * •

MIME Majumder et al. (2020) is a Transformer-based model that mimics the
emotion of the speaker for empathetic response generation.

  * •

BlenderBot Roller et al. (2021) is an open-domain dialogue model pretrained
with multiple skills, including empathetic responding. BlenderBot-Joint Liu et
al. (2021) jointly predicts strategies and generates responses.

  * •

GLHG Peng et al. (2022) is a BlenderBot-based model, which employs a
hierarchical graph network to encode multi-source information.

  * •

MISC Tu et al. (2022) is a BlenderBot-based model, which incorporates
commonsense knowledge and mixed support strategy to jointly predicts support
strategies and generates responses.


[affiliation=†]JiwonSuh [affiliation=†]InjaeNa [affiliation=]WoohwanJung

# Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions

###### Abstract

End-to-end automatic speech recognition (E2E ASR) systems have significantly
improved speech recognition through training on extensive datasets. Despite
these advancements, they still struggle to accurately recognize domain
specific words, such as proper nouns and technical terminologies. To address
this problem, we propose a method to utilize the state-of-the-art Whisper
without modifying its architecture, preserving its generalization performance
while enabling it to leverage descriptions effectively. Moreover, we propose
two additional training techniques to improve the domain specific ASR: decoder
fine-tuning, and context perturbation. We also propose a method to use a Large
Language Model (LLM) to generate descriptions with simple metadata, when
descriptions are unavailable. Our experiments demonstrate that proposed
methods notably enhance domain-specific ASR accuracy on real-life datasets,
with LLM-generated descriptions outperforming human-crafted ones in
effectiveness.

###### keywords:

automatic speech recognition, contextual biasing, large language model

††footnotetext: Major in Bio Artificial Intelligence

##  1 Introduction

Recent advancements in end-to-end (E2E) automatic speech recognition (ASR)
systems, such as Wav2Vec 2.0 [1] and Whisper [2], have significantly improved
the capabilities of speech recognition through extensive training on large
datasets. However, these systems often encounter difficulties in accurately
identifying domain specific terms, such as proper nouns and technical jargon.
Consider a scenario where an ASR system is tasked with transcribing a lecture
on mycology. The pronunciation of `morel' ([mrel] ) and `moral' ([mrl]) can be
very similar, especially in rapid speech, although these words have different
meanings. Since `moral' appears much more frequently in general speech than
`morel', general-domain ASRs may misinterpret `morel' as `moral'.

Contextual biasing [3] is widely used in previous studies [4, 5, 6, 7] to
improve the performance of speech recognition for domain-specific words. In
addition to the audio input, this method provides a biasing list that consists
of words that do not appear frequently throughout the dataset or with a high
error rate. However, a notable limitation arises from the reliance on a single
biasing list tailored for each specific dataset, such as Mathematics, Finance,
and Chemistry. Creating a comprehensive list that covers all potential domain-
specific terms is inherently challenging. Thus, it often fails to recognize
highly specific words in the current audio input (e.g., Creating a
comprehensive list that covers all product names across various companies for
earnings calls.).

To alleviate the problem, recent works [8, 9, 10, 11] introduce methods to
incorporate more specific contextual information. Some of the works [9, 10]
utilize the preceding utterance as the contextual information. However, this
approach can suffer from the error propagation problem: an incorrect
recognition in an utterance causes incorrect recognition in the next
utterances. In addition, they require an additional text encoder which should
be trained on the domain specific dataset, demanding a substantial amount of
training data that is frequently unavailable in domain-specific ASR. The most
relevant work to ours is [11], which utilizes human-written descriptions.
However, it also employs an additional model LLaMa [12] to utilize the textual
description. Since LLaMa (7B$\sim$65B parameters) is much larger than typical
ASR models like Whisper (0.04B $\sim$ 1.6B parameters), it introduces a
significant computational overhead. Moreover, the human-written descriptions
in real-world datasets often lack details and even are unavailable in many
cases.

In response to these limitations, primarily due to the integration of
additional models with pretrained ASR models, We utilize the existing Whisper
[2] framework without any additional module. Since Whisper is already trained
for ASR tasks, our approach minimizes the requirement for domain-specific
training data. We also introduce several techniques to efficiently fine-tune
Whisper for domain-specific ASR with small domain specific data. Moreover, for
the cases without human-created descriptions, we propose a method to generate
a description for each audio by using a Large Language Model (LLM). The cost
of utilizing an LLM is relatively small by generating a single description for
each speech rather than for each utterance. Notably, we empirically
demonstrate that the LLM-generated description can surpass human-created ones
in improving the accuracy of domain-specific ASRs, thanks to the detailed
explanations of LLMs.

##  2 Method

In this section, we propose a
method111https://github.com/nickjw0205/Improving-ASR-with-LLM-Description to
improve domain-specific ASR by incorporating textual descriptions.

Figure 1: An overview of our method

###  2.1 Overview

Previous works [8, 9, 10, 11] have explored improving ASR models with textual
context typically by combining an acoustic model and a language model. For
instance, a recent work [11] leverages HuBERT [13] as the audio encoder and
LLaMa [12] as the language model. Despite the effective utilization of textual
information, this approach requires extensive domain-specific data for model
training. To address this challenge, we decide to utilize descriptions without
modifying the architecture of an existing ASR model.

Conventional ASR models usually do not have a text encoder since they
primarily focus on converting speech to text. Thus, we cannot use such a model
without additional modules. However, Whisper [2], a state-of-the-art ASR
model, intrinsically supports textual input. Whisper is built on Transformer
[14] architecture with an encoder and a decoder. The encoder processes audio
inputs while the decoder generates transcripts. Given that the decoder is
based on a Transformer, it can accept a prompt prior to transcript generation.
Thus, we input textual descriptions as prompts.

Figure 1 shows an overview of our method which utilizes a pretrained Whisper
model without modifying its original architecture. The audio input is
processed by the encoder, consistent with the original Whisper. Note that the
encoder remains frozen during fine-tuning, a topic further explored in Section
2.2. We give textual descriptions as prompts to the decoder. Whisper has two
special tokens <SOP> (Start of previous) and <SOT> (Start of transcript) which
is to take the transcription of the preceding 30 seconds of audio segment as
contextual information. We adapt these tokens to input our textual
descriptions in the format <SOP> description <SOT>. This strategy enables the
model to utilize descriptions without an additional text encoder.

###  2.2 Training the Proposed Model

Our model can be fine-tuned in the same way as the original Whisper was
trained since it uses the architecture of Whisper without any modification.
However, to improve the performance of the domain-specific ASR tasks with
textual description, we need to consider its unique characteristics. First,
domain-specific datasets are generally much smaller than general-domain
datasets. Second, not every utterance directly relates to the provided textual
description. We propose two methods to address the issues: decoder fine-tuning
and context perturbation.

Decoder Fine-tuning. Domain-specific ASR models suffer from the scarcity of
training data with speech-text pairs. To avoid overfitting and catastrophic
forgetting, it is necessary to minimize the number of parameters to be
trained. The decoder of our model process descriptions as additional inputs, a
feature not present during the pretraining phase of Whisper, mandating fine-
tuning. Conversely, fine-tuning the encoder is not essential since it solely
processes audio, just as during pretraining. The pretrained Whisper encoder
has an excellent capability in featuring audio information even without fine-
tuning. According to a recent study [15], Whisper with a frozen encoder
exhibits superior performance in various tasks, including intent
classification, keyword spotting, and ASR, compared to the model where the
encoder has been fine-tuned. It implies that retraining the encoder may
require a significant amount of data and risk losing its generalization
performance. Therefore, we opt to freeze the encoder and fine-tune the decoder
to effectively leverage the textual description.

Context Perturbation. Although textual descriptions aid in recognizing domain
specific words, not all utterances contain domain specific words. For example,
situations like greetings or humorous remarks for ice-breaking may not include
domain-specific words. In these instances, ASR models should be able to ignore
the provided description. Inspired by [8], we occasionally input a randomly
sampled description instead of the actual one with a probability of 5% in our
experiments. This approach trains the model to selectively utilize the
description.

###  2.3 Generating Descriptions using LLM

Utilizing textual descriptions can significantly improve ASR performance.
However, there are numerous situations where textual descriptions may not be
available, such as personal recordings and live streams. Moreover, even when
textual descriptions are available, they often do not include necessary
detail. For example, consider the task of transcribing a university lecture
through ASR. While the syllabus may provide an overview of the course, it
typically lacks specific details about individual lecture topics. To overcome
this challenge, we propose generating detailed descriptions through LLM.

We generate descriptions by using LLM with simple metadata about audio files.
To validate the effectiveness of LLM-generated descriptions with very simple
metadata, we use two datasets in our experiments: Earnings Call [16], which
contains discussions about financial results of particular public companies,
and OCW (MIT OpenCourseWare), which includes video lectures. For the Earnings
Call dataset, the description of conference call is created by using the
company name with a prompt “Explain about [company name] in 2 sentences.”. Due
to the weakness of Whisper with processing long descriptions [17], we limit
the length of the description with the instruction "in 2 sentences". For the
OCW dataset, we use the prompt “Today’s lecture title is [lecture title].
Please explain the academic field and content in 2 lines.”. This technique,
relying solely on simple metadata (lecture titles and company names), enables
the extraction of essence-capturing descriptions, drawing on the extensive
knowledge of LLM. It is worth noting that the cost of using LLM is not high.
For example, the OCW dataset has 508 audio segments per lecture on average,
indicating that the execution of ASR model is 508 times more frequent than
that of LLM in the OCW dataset.

##  3 Experiments

###  3.1 Datasets

To evaluate the domain-specific ASR models with descriptions, we use the
following two datasets.

Earnings Call [16]. It consists of quarterly earning conference calls from S&P
500 companies in 2017. The Earnings Call dataset contains various domain-
specific information, such as company and product names. After removing
duplicate calls from the same companies, this dataset consists of 169
conference calls. We split the dataset into 113, 28, and 28 calls for
training, validation, and testing, respectively. The total lengths of the
audio in train/valid/test sets are approximately 40/10/10 hours, respectively.
The audio quality of Earnings Call is much lower than professionally recorded
datasets due to factors like multiple speakers and fluctuating recording
environments.

OCW222https://github.com/nickjw0205/Improving-ASR-with-LLM-Description . We
collected a new dataset to validate the effectiveness of our model with
academic words. This dataset is collected from a MIT OpenCourseWare
website333https://ocw.mit.edu/, CC BY-NC-SA 4.0 license and includes the
videos, English transcripts, and titles of 65 lectures on various academic
fields such as linear algebra, biological chemistry, and cryptocurrency. We
split the dataset into train (44 lectures), valid (12 lectures), and test (9
lectures) sets to make their lengths are approximately 40 hours, 10 hours, and
10 hours, respectively. We excluded all words from the transcript that were
not actual utterances but were added to aid understanding. Examples of such
words include [No audio], [Not AUDIBLE], [Laugh], [DOOR CLOSES], etc.
Additionally, we performed preprocessing to ensure proper alignment between
the audio and transcript by removing the speaker identifiers, such as
`Speaker:', used to denote the speaker.

###  3.2 Experimental Setup

We utilize the Whisper (base.en), a Transformer encoder-decoder model with 74
million parameters. Notably, our method fine-tunes only the decoder which has
52 million parameters. To ensure compatibility with Whisper, all audio files
are resampled to 16kHz and transcriptions are normalized by converting to
lowercase and removing punctuations. To train Whisper, we use the HuggingFace
Transformers library444https://github.com/huggingface/transformers. Following
the Whisper's training, we use AdamW [18] optimizer with a linear learning
rate decay scheduler. The training process was conducted with a batch size of
32, initiating learning rate decay following a 100-step warm-up period. We
explored initial learning rates of 1e-4, 1e-5, and 1e-6, ultimately selecting
1e-4 for Earnings Call and 1e-6 for OCW based on performance. Training
durations are set at 10 epochs for Earnings Call and 15 epochs for OCW. We use
the cross entropy loss following Whisper. Other settings follow the default
configuration of the HuggingFace trainer. All models are trained on an NVIDIA
RTX 4090 GPU. Unless otherwise specified, we use the descriptions generated by
GPT-3.5 Turbo555https://openai.com/product as proposed in Section 2.3. The
evaluation metric is the Word Error Rate (WER). Note that the training time
was about 6 hours for Earnings Call, and 8 hours for OCW.

Table 1: Main results

| Earnings Call | OCW
---|---|---
Models | 20 h | 40 h | 20 h | 40 h
Whisper (Frozen) | 16.39% | 16.39% | 11.98% | 11.98%
\+ Full Fine-tuning | 17.38% | 16.64% | 10.41% | 9.94%
\+ Description | 20.63% | 17.70% | 9.81% | 9.72%
\+ Decoder Fine-tuning | 16.61% | 15.70% | 9.79% | 9.67%
\+ Context Perturbation | 16.24% |  15.15% | 9.79% | 9.68%

Table 2: Examples of outputs with and without descriptions. | Edwards Lifesciences (Earnings Call) | Assembly Language & Computer Architecture (OCW)
---|---|---
Description |  Edwards Lifesciences is a global medical technology company that specializes in the development and manufacturing of heart valves … improve the lives of patients with cardiovascular diseases |  Assembly language is a low-level programming language that communicates directly with the hardware of a computer system. Computer architecture is the study of how computers …
Reference |  New treatments and therapies for cardiovascular care and heart muscle recovery |  It's just a bunch of bytes
Output w/ description |  New treatments and therapies for cardiovascular care and heart muscle recovery |  It's just a bunch of bytes
Output w/o description |  New treatments and therapies for cardiovasular care and heart missile recovery |  It's just a bunch of bites

###  3.3 Main Results

To demonstrate the effectiveness of our proposed method we show the word error
rates of ASR with different train data sizes (20 and 40 hours) in Table 1. We
systematically investigate the effectiveness of the proposed components by
incrementally introducing the following elements to the frozen Whisper: fine-
tuning, utilization of descriptions, decoder fine-tuning (freezing encoder),
and context perturbation. Applying all proposed techniques result in the best
ASR model performance, demonstrating the effectiveness of our methods.

The experimental results on Earnings Call (20 hours) are quite interesting.
When the Whisper model was fully fine-tuned with textual descriptions, there
was a significant increase in WER by more than 4 percentage points, indicating
a substantial performance decline. This can be interpreted as a result of
overfitting and catastrophic forgetting, which occurs when training on a small
amount of data for domain-specific ASR tasks, as we previously mentioned.
However, even in such cases, our method, employing decoder fine-tuning and
context perturbation, demonstrated its ability to overcome these issues. This
suggests that it is essential to perform decoder fine-tuning along with
context perturbation to achieve robust results with a small amount of data and
in conditions of low audio quality. While the results from the OCW dataset are
not as dramatic as those from the Earnings Call (20h), they still confirm the
stable performance improvements by our methods.

Table 3: Examples of collected and generated descriptions. | Description
---|---
Collected |  This course is an introduction to mathematical modeling of computational problems, as well as common algorithms, algorithmic paradigms, and data structures used to solve these problems. It emphasizes the relationship between algorithms and programming, and introduces basic performance measures and analysis techniques for these problems.
LLM Generated |  hashing is a fundamental concept in computer science and cryptography that involves creating a unique, fixed-length representation of data using a mathematical algorithm. It is commonly used in data retrieval, password security, and ensuring data integrity.

###  3.4 Qualitative Analysis

Table 2 shows examples of transcription, both with and without using
descriptions. The first example provides a description about Edward
Lifesciences, a medical technology company, and transcription results for a
sentence from an Edward Lifesciences earnings call. The description
incorporates domain-specific terminology such as cardiovascular and includes
comprehensive information about the company. By using this description, our
method accurately transcribes the medical term `cardiovascular'. However, the
conventional speech recognition method struggles to accurately transcribe the
domain-specific word `cardiovascular'. In the OCW dataset, the description
offers insights into "Assembly Language and Computer Architecture,"
facilitating the accurate identification of computer-related terms such as
byte' based on this contextual information. Conversely, without utilizing the
description, the term `byte' is misrecognized as its homophone `bite', because
there is no computer-related context information during the transcription
process. These indicate that descriptions provide not only domain-specific
terms but also important contextual information to the ASR model. Therefore,
incorporating descriptions into the ASR model can enhance the recognition
accuracy of domain-specific words.

###  3.5 Comparison of LLM-Generated and Collected Descriptions

Collection of human-written descriptions. To evaluate the effectiveness of the
LLM-generated descriptions, we also collect human-written descriptions for our
dataset as a comparison. We collect descriptions from Wikipedia for company
information in the Earnings Call dataset, which are often overly detailed
because they aim to provide comprehensive explanations. For the OCW dataset,
we collect descriptions of the respective courses that provide a broad course
overview, rather than specifics of individual lectures, reflecting their
purpose of providing a general course summary.

Description Quality. Table 3 presents a LLM-generated description and using
the lecture title ``Hashing" and a human-written one collected from the course
description of ``Introduction to Algorithms". The collected description offers
an overview of the course, outlining the content to be learned at a
comprehensive level. In contrast, the description generated by LLM offers
detailed information about the specific lecture `Hashing'. This indicates that
LLM can generate more specific and relevant descriptions than collected
descriptions in real world applications.

Table 4: Results on generated and collected descriptions. Description | Earnings Call | OCW
---|---|---
Collected | 15.33% | 10.30%
LLM Generated |  15.15% |  9.68%

ASR Performance. In Table 4, we compare ASR performances with collected
descriptions and LLM-generated descriptions. For the Earnings Call,
transcription results using LLM-generated descriptions slightly outperform
those using collected descriptions by 0.18%p. The LLM-generated descriptions
contribute to this improvement by omitting unnecessary details and adopting
more concise and central information. In the OCW, transcriptions utilizing
LLM-generated descriptions demonstrated a lower WER compared to those with
collected descriptions by 0.62%p. This difference is due to the clear
disparity in content between the LLM-generated description and the collected
one, as detailed in Table 3. Collected descriptions provide only general
information about the courses, failing to capture the specific details
essential for each lecture audio. However, the LLM-generated description is
tailored to the lecture title ``hashing", providing detailed information
relevant to the specific lecture.

##  4 Conclusion

We propose a domain-specific speech recognition model which leverages
descriptions of speeches. To address the data scarcity problem in domain-
specific ASR, we propose a method to utilize the state-of-the-art Whisper
without modifying its architecture. Moreover, we propose decoder fine-tuning,
and context perturbation to preserve the Whisper's generalization performance
while utilizing descriptions. The empirical studies show the effectiveness of
our methods with two real datasets. Furthermore, we find that LLM-generated
descriptions can outperform human-written ones in domain-specific ASR with
description.

## References

  * [1] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [2] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in International Conference on Machine Learning, pp. 28492–28518, PMLR, 2023.
  * [3] P. Aleksic, M. Ghodsi, A. Michaely, C. Allauzen, K. Hall, B. Roark, D. Rybach, and P. Moreno, ``Bringing contextual information to google speech recognition,'' in Proc. Interspeech 2015, pp. 468–472, 2015.
  * [4] D. Le, M. Jain, G. Keren, S. Kim, Y. Shi, J. Mahadeokar, J. Chan, Y. Shangguan, C. Fuegen, O. Kalinli, Y. Saraf, and M. L. Seltzer, ``Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion,'' in Proc. Interspeech 2021, pp. 1772–1776, 2021.
  * [5] K. Huang, A. Zhang, Z. Yang, P. Guo, B. Mu, T. Xu, and L. Xie, ``Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network,'' in Proc. INTERSPEECH 2023, pp. 4933–4937, 2023.
  * [6] Y. Xu, B. Liu, Q. Huang, X. Song, Z. Wu, S. Kang, and H. Meng, ``Cb-conformer: Contextual biasing conformer for biased word recognition,'' in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, 2023.
  * [7] G. Sun, X. Zheng, C. Zhang, and P. C. Woodland, ``Can Contextual Biasing Remain Effective with Whisper and GPT-2?,'' in Proc. INTERSPEECH 2023, pp. 1289–1293, 2023.
  * [8] X. Yang, W. Kang, Z. Yao, Y. Yang, L. Guo, F. Kuang, L. Lin, and D. Povey, ``Promptasr for contextualized asr with controllable style,'' arXiv preprint arXiv:2309.07414, 2023.
  * [9] S.-Y. Chang, C. Zhang, T. N. Sainath, B. Li, and T. Strohman, ``Context-aware end-to-end asr using self-attentive embedding and tensor fusion,'' in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, 2023.
  * [10] S. Duarte-Torres, A. Sen, A. Rana, L. Drude, A. Gomez-Alanis, A. Schwarz, L. Rädel, and V. Leutnant, ``Promptformer: Prompted conformer transducer for asr,'' arXiv preprint arXiv:2401.07360, 2024.
  * [11] Y. Li, Y. Wu, J. Li, and S. Liu, ``Prompting large language models for zero-shot domain adaptation in speech recognition,'' in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1–8, 2023.
  * [12] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., ``Llama: Open and efficient foundation language models,'' arXiv preprint arXiv:2302.13971, 2023.
  * [13] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, ``Attention is all you need,'' Advances in neural information processing systems, vol. 30, 2017.
  * [15] H. Yang, J. Zhao, G. Haffari, and E. Shareghi, ``Investigating Pre-trained Audio Encoders in the Low-Resource Condition,'' in Proc. INTERSPEECH 2023, pp. 1498–1502, 2023.
  * [16] Y. Qin and Y. Yang, ``What you say and how you say it matters: Predicting stock volatility using verbal and vocal cues,'' in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 390–401, 2019.
  * [17] L. Zhuo, R. Yuan, J. Pan, Y. Ma, Y. LI, G. Zhang, S. Liu, R. Dannenberg, J. Fu, C. Lin, et al., ``Lyricwhiz: Robust multilingual zero-shot lyrics transcription by whispering to chatgpt,'' arXiv preprint arXiv:2306.17103, 2023.
  * [18] L. Ilya and F. Hutter, ``Decoupled weight decay regularization,'' arXiv preprint arXiv:1711.05101, 2017.