# Speaker and Time-aware Joint Contextual Learning for Dialogue-act
Classification in Counselling Conversations

Ganeshan Malhotra1, Abdul Waheed2, Aseem Srivastava3 and  Md Shad Akhtar3,
Tanmoy Chakraborty3 1BITS Pilani, Goa, India, 2Maharaja Agrasen Institute of
Technology, New Delhi, India, 3IIIT-Delhi, India  ganeshanmalhotra,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(2022)

###### Abstract.

The onset of the COVID-19 pandemic has brought the mental health of people
under risk. Social counselling has gained remarkable significance in this
environment. Unlike general goal-oriented dialogues, a conversation between a
patient and a therapist is considerably implicit, though the objective of the
conversation is quite apparent. In such a case, understanding the intent of
the patient is imperative in providing effective counselling in therapy
sessions, and the same applies to a dialogue system as well. In this work, we
take forward a small but an important step in the development of an automated
dialogue system for mental-health counselling. We develop a novel dataset,
named HOPE, to provide a platform for the dialogue-act classification in
counselling conversations. We identify the requirement of such conversation
and propose twelve domain-specific dialogue-act (DAC) labels. We collect $\sim
12.9K$ utterances from publicly-available counselling session videos on
YouTube, extract their transcripts, clean, and annotate them with DAC labels.
Further, we propose SPARTA, a transformer-based architecture with a novel
speaker- and time-aware contextual learning for the dialogue-act
classification. Our evaluation shows convincing performance over several
baselines, achieving state-of-the-art on HOPE. We also supplement our
experiments with extensive empirical and qualitative analyses of SPARTA.

††copyright: acmcopyright††journalyear: 2022††copyright:
acmcopyright††conference: Proceedings of the Fifteenth ACM International
Conference on Web Search and Data Mining; February 21–25, 2022; Tempe, AZ,
USA.††booktitle: Proceedings of the Fifteenth ACM International Conference on
Web Search and Data Mining (WSDM ’22), February 21–25, 2022, Tempe, AZ,
USA††price: 15.00††isbn: 978-1-4503-9132-0/22/02

##  1\. Introduction

Mental illness remains an alarming global health issue today. Due to the
COVID-19 pandemic, there has been a significant growth in mental health
disorders such as depression, attention deficit hyperactivity disorder (ADHD)
and hypertension (Moreno et al., 2020). A recent study shows an unprecedented
$20$% increase in patients with mental health
illness111https://cutt.ly/WbEziBF/. Similar study discusses the adverse impact
on the mental health of US college students due to the pandemic (Wang et al.,
[n.d.]).

Table 1. Example of a sample conversation session between a patient and a
therapist. Each utterance has an associated dialogue-act classification (DAC)
label.

Utterance |  DAC labels
---|---
Therapist: Jackie, how are you? |  Greeting
Patient: Okay, How are you? |  Greeting
Therapist: Thanks for asking. I see that you have signed a release so I could talk to your mother and that she brought you in today. What’s going on there? |  Information Request
Patient: They think I have a drinking problem. My family.. |  Information Delivery
Therapist: Your family thinks you have a drinking problem? |  Clarification Request
Patient: Yeah. So we really started this was this past weekend. They came to pick me up for my bridal shower. And I was I was drunk when they came to get me so I couldn’t go and now everybody’s pretty pissed at me. |  Clarification Delivery
Therapist: So they asked you to come into the agency? |  Clarification Request
Patient: Yeah, you know, I don’t want them to hate me or anything. So I agreed to come. |  Clarification Delivery

Counselling therapy can benefit many people under risk by providing them
emotional support. Amidst the surge in the number of patients, it has become a
challenge for the therapists to diagnose too many patients. On the other hand,
patients have found it difficult to access the services of the therapist amid
lockdown.

Counseling therapy is a sophisticated procedure that deals with the expression
of emotion and intent of patients with different personalities. To build a
strong therapeutic relationship with the patient, it is essential for a
therapist to develop a better understanding of the implicit intents of the
patients. The nature of conversations in a social counselling setting is
particularly distinct as compared to a conventional chit-chat or goal-oriented
conversations. It follows a pattern which is different from both goal-oriented
and general chit-chat based conversations. Usually these conversations begin
with greetings followed by the therapist inquiring for problems faced by the
patient. The therapist usually delves deeper into a particular problem
acquiring as much context and fine-grained information before advising a
remedy. These conversations also heavily utilise the contextual information of
the entire conversation history. Moreover, the prime objective of the
conversation is to understand the explicit and implicit requirements of the
patients and suggest potential solutions accordingly. In comparison, a
traditional goal-oriented dialogue system does not regard any implicit
requirements, whereas a chit-chat based system lacks a target and does not
care about the final solution. Another major difference is the length of
utterances and conversations in a counseling session. These are particularly
lengthy as patients describe their difficulties and issues, while the
therapists list out possible causes and preventive solutions.

The task of Dialogue-act classification (DAC) is cardinal in a dialogue system
and even more so in counselling based conversations. It deals with
understanding the intended requirements of the utterances, which essentially
act as one of the precursors for the dialogue response generation. For
instance, we present an example of a therapy session in Table 1. For each
utterance in Table 1, a corresponding label defines its dialogue-act. As we
can observe from the first two utterances, they are a part of the
complementary greetings that usually occur at the beginning of a natural
conversation. Subsequently, in the third utterance, the therapist leads the
conversation and requests for information. In response, the patient delivers
the requested information.

Earlier studies like (Ahmadvand et al., 2019; Raheja and Tetreault, 2019a)
tackle the task of dialogue-act classification on chit-chat based conversation
datasets such as Switchboard corpus (Godfrey et al., 1992). Their proposed
architectures take into account the contextual dependency of an utterance that
aids in efficient dialogue-act classification. For example, an utterance
tagged as having a dialogue-act ‘question’ has a high probability of being
followed by an utterance with tag ‘answer’. In another work, Shang et al.
(2020a) argued that the information of speaker change is a critical feature in
the dialogue-act classification task.

Considering the severity of the issue and the complexity of the task,
designing an automated system can facilitate the counselling sessions or
assist the therapist, thus allowing them to cater to more patients. Literature
in the natural language processing domain suggests a significant effort in
understanding and building models for conversational dialogue (Li et al.,
2017; Cerisara et al., 2018). However, there are hardly any models that
support mental-health counseling as a dialogue system; this is primarily due
to lack of data. In this paper, we aim to address these limitations by
creating the HOPE222Mental Health cOunselling of PatiEnts dataset which
consists of therapy conversations covering cognitive-behavioral therapy (CBT),
child therapy, family therapy, etc. The HOPE dataset contains $\sim 12.9K$
utterances across $212$ mental-health counseling sessions. Each utterance in
the dataset is tagged with one of the $12$ counseling-aligned dialogue-act
labels (c.f. Section 3).

We also propose SPARTA333SPeaker and time-AwaRe conTextual trAnsformer, a
novel speaker- and time-aware contextual transformer model for dialogue-act
classification. SPARTA exploits both the local and global contexts along with
the speaker-dynamics in the dialogue. We model the problem as a dialogue-level
sequence classification task, where the aim is to predict an appropriate
dialogue-act for each utterance in a dialogue. To incorporate the global
context, we employ a Gated Recurrent Unit (GRU) (Cho et al., 2014) that takes
an utterance representation at each step of the dialogue. In addition, we
introduce a novel time-aware attention mechanism to capture the local context
– a sliding-window based memory unit is maintained, and subsequently, a cross-
attention between the current utterance and the memory unit is computed. Our
evaluation shows substantial improvement in performance in comparison to the
recent state-of-the-art systems. Furthermore, we provide empirical evidences
for each module of SPARTA using an extensive ablation study and detailed
analyses.

#### Major contributions:

We summarize the main contributions of our current work as follows:

  * •

We present HOPE, a novel and large-scale manually annotated, counselling-based
conversation data for dialogue-act classification. To the best of our
knowledge, the current study is one of the first efforts in compiling a
dataset related to the mental-health counseling dialogue system.

  * •

To cater to the requirements of counseling conversations, we define a novel
hierarchical annotation scheme for the dialogue-act annotation. We propose
twelve dialogue-act labels that are aligned with mental-health counseling
session.

  * •

We propose SPARTA, a novel dialogue-act classification system that combines
speaker-dynamics and local context through a time-aware attention mechanism,
along with long-term global context.

  * •

We perform extensive ablation study to establish the efficacy of each module
of SPARTA. Furthermore, the comparative analysis shows that it attains state-
of-the-art performance on HOPE.

#### Societal Impact:

A significant increase in the number of mental health issues has been observed
in the last few years. The lack of therapists is a stumbling block to the
mental health of society. Therapist-Bots (mental health chatbots) could bridge
the gap by effectively interacting with patients and understanding them.
Conversely, end-to-end chatbots in the mental health domain are delicate,
where every aspect of the therapy is needed to be perceived precisely. Our
research aims at the dialogue-understanding module in the mental health
conversational system. The ongoing research in mental health domain could
exploit this work and benefit the chatbots to understand the therapy
conversation in a better way.

#### Reproducibility:

We have made the code and (a subset of) the dataset available at
github444https://github.com/LCS2-IIITD/SPARTA_WSDM2022.

##  2\. Related Work

The current work is connected to the existing literature in at least two
dimensions – first, the dialogue-act classification, and second, text
processing for mental-health counseling. We present our literature survey for
both dimensions.

Mental Health and the role of text processing. The impact of Natural language
processing in the study of mental-health is substantial. Though the field of
therapeutic discourse analysis has been around since 1960s (Van Dijk, 2011),
research on dialogue systems in mental-health domain is in nascent stage.
Previous research on mental health intervention systems primarily focused on
the problems related to the suicide ideation prevention (Mishra et al., 2019)
or generating empathetic responses to users (Naous et al., 2020). Levis et al.
(2020) studied the mental-health notes to detect suicide ideation, whereas,
Fine et al. (2020) employed text processing to detect symptoms of anxiety and
depression using social media text. In another work, Pennebaker et al. (2003)
showed the importance of several keywords in revealing users’ social and
psychological behaviours. Wei et al. (2019) proposed an emotion-aware model
for human-like emotional conversations.

Gratch et al. (2014) presented the Distress Analysis Interview Corpus (DAIC)
to identify verbal and non-verbal cues of distress during an interview. Among
other methods, the data collection was done using the automated agent Ellie
based on the work of (Morbini et al., 2014). Recently, Kretzschmar et al.
(2019) presented a survey of the chat-bots in the mental-health domain. The
authors compared the strengths and weaknesses of three existing conversational
agents, namely Wysa555https://www.wysa.io/,
Woebot666https://woebothealth.com/, and Joy777
https://www.hellojoy.ai/support. The drawbacks of these systems are that some
of these are rule-based, while others are primarily data collection module for
an offline counselling. In comparison, ours is the effort in the development
of an online counselling system.

Dialogue-act Classification. Studies on dialogue systems have always
fascinated researchers ever since ELIZA (Weizenbaum, 1966), the first rule-
based system was developed. The dialogue-act classification module is one of
the most critical components of a dialogue agent which caters to the
requirement of the dialogue system by serving at the natural language
understanding helm of the dialogue system.

Previous research treats the problem of dialogue-act classification either as
a standalone text classification task or a sequence labelling task. Recently,
Colombo et al. (2020) suggested a sequence-to-sequence text generation
approach for the dialogue-act classification. Earlier studies like Reithinger
and Klesen (1997) and Grau et al. (2004) focused on lexical, syntactical, and
prosodic features for classification. In another work, Ortega et al. (2019)
employed CNNs (LeCun et al., 1989) and CRFs (Lafferty et al., 2001). Lee and
Dernoncourt (2016) proposed a method based on CNNs and RNNs (Rumelhart et al.,
1986) that used the previous contextual utterances to predict the dialogue-act
of the current utterance.

Ahmadvand et al. (2019) proposed a contextual dialogue-act classifier (CDAC)
and used transfer-learning to train their model on human-machine
conversations. The model proposed by Raheja and Tetreault (2019b) uses self-
attention mechanism on RNNs to achieve impressive results on benchmark
datasets. However, these works do not take into account the speaker-level
information which is imperative in social-counseling based conversations. Hua
et al. (2020) proposed a method to detect relevant context in retrieval-based
dialogue systems. Chen et al. (2018) proposed a CRF-attentive structured
network to capture the long-range contextual dependencies using structured
attention mechanism. Yu et al. (2019) proposed to classify concurrent
dialogue-acts of an utterance by modelling the contextual features. Recently,
Qin et al. (2020) used co-interactive relation networks to jointly capture
sentiment and the associated dialogue-acts with an utterance. Their model
achieved significant results on Mastodon (Cerisara et al., 2018) and
DailyDialog (Li et al., 2017) datasets. Similarly, Saha et al. (2020) jointly
learns the dialogue-act classification and emotion recognition tasks in a
multi-modal setup.

To the best of our knowledge, the model by Shang et al. (2020b) is the first
one which takes speaker transitions for DAC into account. It uses a modified
version of CRFs to capture the speaker-change and achieves state-of-the-art
results on SwitchBoard dataset (Godfrey et al., 1992). Similar to earlier
works, we also treat DAC as a dialogue-level sequence labelling task. We
jointly take the global and local contexts of the conversation and the speaker
of the utterance as the key factors for the classification. We hypothesize
that such information offers crucial clues at different stages of the model.
In contrast, the existing systems had incorporated the role of global context
and speakers dynamic independently.

##  3\. Dataset

In this section, we present our dialogue-act classification dataset, called
HOPE. In total, we annotated $\sim 12.9K$ utterances with $12$ dialogue-act
labels carefully designed to cater to the requirements of a counseling
session. The remaining section furnishes the details of data collection,
annotation schemes, dialogue-act labels and necessary statistics.

#### Data Collection

One of major hurdles we faced in the process of data collection was the
unavailability of public counseling sessions, mainly due to the fact that they
usually contain sensitive personal information. To curate this data, we
carefully explored the web and collected publicly-available pre-recorded
counselling videos on YouTube. To ensure confidentiality, we randomly assign
synthetic names to all patients and therapists in all examples.

In the next step, we extract the transcriptions of each video using OTTER
(https://www.otter.ai/), an automatic speech recognition tool. Subsequently,
we correct transcription errors to remove any noise (i.e., spelling or
grammatical mistakes). The data collection process provides us $12.9K$
utterances from 212 counseling therapy sessions – all of them are dyadic
conversations only.

Table 2. Dialogue-act distribution in HOPE. The train, test, and validation
splits are 70:20:10.

|  |  | Dialogue-act labels |  |
---|---|---|---|---|---
Split |  | Speaker | ID | IRQ | GT | GC | CRQ | YNQ | CD | ACK | PA | NA | OD | ORQ | Total
Train |  | Patient | 1886 | 23 | 155 | 494 | 77 | 12 | 666 | 607 | 269 | 271 | 205 | 3 | 4668
Therapist | 393 | 1474 | 212 | 829 | 710 | 692 | 83 | 125 | 6 | 7 | 40 | 180 | 4751
1-1 3-16 Test |  | Patient | 466 | 7 | 32 | 122 | 13 | 6 | 189 | 97 | 73 | 66 | 48 | - | 1119
Therapist | 66 | 376 | 44 | 209 | 199 | 134 | 15 | 31 | 6 | - | 8 | 34 | 1122
1-1 3-16 Val |  | Patient | 246 | 3 | 17 | 83 | 8 | - | 97 | 63 | 34 | 27 | 16 | 1 | 595
Therapist | 41 | 194 | 22 | 113 | 101 | 85 | 10 | 16 | - | - | 3 | 14 | 599
Total |  | Patient | 2598 | 33 | 204 | 699 | 98 | 18 | 952 | 767 | 376 | 364 | 269 | 4 | 6382
Therapist | 500 | 2044 | 278 | 1151 | 1010 | 911 | 108 | 172 | 12 | 7 | 51 | 228 | 6472

Figure 1. Annotation tree for HOPE.

#### Dataset Annotation Scheme

Since the counseling conversations have inherent differences with the standard
conversations (such as SwitchBoard dataset conversation), it demands a
carefully designed set of dialogue-act labels capable of catering the
requirements of counselling conversations. Hence, we, in consultation with
therapist and counselling experts, design a set of 12 dialog-act labels that
are arranged in a hierarchy. These labels are designed to capture the intents
of both the patient and therapist, and also be easily comprehensible to assist
in the development of a conversational dialogue system. A high-level
annotation hierarchy is shown in Figure 1. Each utterance in the dialogue
belongs to one of the three categories888Sometimes, an utterance can have
multiple dialog-acts; however, they are rare in the annotated dataset. Hence,
for simplicity, we consider only one (primary) label for each utterance. –
speaker initiative, speaker responsive, and general or mixed initiative. Our
annotation scheme assigns three distinct dialogue-act labels to the first two
categories, while the remaining four labels belong to the general category.

  * •

Speaker initiative labels: When the speaker drives the conversation for the
next few utterances.

    * –

Information Request (IRQ): This label is used as a request for some
information, e.g., ‘Tell me your name.’.

    * –

Yes/No Question (YNQ): The YNQ label is similar to IRQ; however, the expected
response is a trivial yes or no answer. For example, the utterance, ‘Did you
complete your work?’ shows how a query is raised with an expected answer of
yes or no.

    * –

Clarification Request (CRQ): This label is assigned to those utterances in
which a speaker usually asks the therapist for further clarification about
topic that is being currently discussed. The distinction between IRQ and CRQ
is the continuation of topics – IRQ is used whenever a discussion about a new
topic or entity is started, and CRQ is used when the speaker wants to gather
more information and delves deeper into the current topic at hand. For
instance, if the therapist asks  You’re in a situation where there is alcohol?
and follows it with another utterance And what sort of situations are you in?,
the later utterance is an example of clarification request as the therapist
delves deeper to seek causes of distress for the patient.

    * –

Opinion Request (ORQ): The ORQ label is used when the speaker seeks opinions
of the listener. For instance, the utterance How does it feel to tell me about
this? is labelled as Opinion Request.

  * •

Speaker responsive labels: The dialogue-act labels under this category are
used in response to the speaker initiative labels.

    * –

Clarification Delivery (CD): This label is used when the speaker provides
further clarifications about a topic/entity under discussion. For example,
while discussing about charges that have been levied upon the patient, the
therapist asks ‘You mean get out of the charge?’ and in response, the patient
responds with the utterance Yeah. I mean, I feel like they’re probably gonna
make me like go to like rehab or something like that. And I don’t want to do
that.. The patient response is an example of a clarification delivery.

    * –

Information Delivery (ID): When the speaker provides some factual objective
information about herself. For instance, the utterances ‘In my last job I
worked in a paper factory.’ and ‘I do not like my present job, I am fed up
with my boss.’ are the examples of information delivery.

    * –

Positive Answer (PA): These labels are used when the utterance is an answer in
the form of a simple yes to a question that was previously uttered, e.g., ‘Are
you alright’ or ‘Yes’.

    * –

Negative Answer(NA): It is used when the utterance is an answer of the form of
a simple No to a question asked earlier.

    * –

Opinion Delivery (OD): When the speaker explicitly lists out her/his opinions.
For example, ‘You don’t seem real certain.’

  * •

General: The utterances under this category can be uttered by either of the
speakers under varying circumstances.

    * –

Greeting (GT): Each session usually starts with greeting by one speaker and an
appropriate response from the other, e.g., ‘Hello, how are you?’ and ‘I am
fine, thank you.’. We tag each of these utterances as GT.

    * –

Acknowledgment (ACK): In normal conversation, very often, we utter (e.g.,
‘Yeah! You are right.’) to acknowledge the other speaker or to show our
agreement without an explicit information request, question, or command. We
also observe such cases in our collected dataset; hence, we tag them as ACK.

    * –

General Chit-Chat (GC): Other utterances that do not belong to any of the
above labels are tagged as GC, possibly because of the vagueness and the lack
of sense in the context of the conversation. For example, the utterance ’It’s
a beautiful day today!’ is an example of GC.

#### Annotation Process

We employed three annotators999Annotators are in the age group of 25-35, with
2-10 years of professional experience. who are experts in linguistics. To
ensure the understanding of the tasks and annotation scheme, we took a sample
of the dataset and asked each annotator to annotate them as per the prepared
set of guidelines. Following this, every annotation was discussed in the
presence of the annotators and an expert therapist as moderator to ensure
consistency. After a couple of annotation and discussion rounds, the whole
dataset was made available for the annotation.

After the annotation process, we compute Cohen’s Kappa score (Cohen, 1960) to
measure the agreement among annotators. We obtain the inter-rater agreement
score of 0.7234 – which falls under the ‘substantial’ (Cohen, 1960) category.

Figure 2. Relationship among the dialogue-act labels. The directed connection
$U_{t}^{x}\rightarrow U_{t+1}^{y}$ shows the co-occurance counts of two
sequential utterances with $x$ & $y$ dialogue-acts, respectively.

#### Dataset Statistics

The broad definitions of speaker-initiative and speaker-responsive dialogue-
act label pairs, IRQ & ID; ORQ & OD, and CRQ & CD, seem complementary in
nature. However, the dataset does not support the above view entirely. To show
the relationship between the respective speaker-initiative and speaker-
responsive dialogue-act label pairs, we present a chord diagram in Figure 2.
For any two consecutive utterances $U_{t}^{x}$ and $U_{t+1}^{y}$ with
corresponding dialogue-act labels $x$ and $y$, each directed link
($U_{t}^{x}\rightarrow U_{t+1}^{y}$) between the two labels reflects their co-
occurrence, and the strength of the link signifies their co-occurrence counts.
Though a significant number of IRQ utterances are followed by ID utterance, in
a few cases, they are followed by other dialogue-act utterances as well (e.g.,
ACK, GC, etc.). Similarly, ID utterances are not always preceded by IRQ
utterances. We observe similar behaviour for YNQ & PA, NA, and CRQ & CD
dialogue-act pairs as well. Table 2 provides the statistics of the HOPE
dataset. In total, HOPE has transcripts for $12.9k$ utterances which are
annotated with $12$ dialogue-act labels. These utterances are evenly
distributed between the patients and therapists with $6.38K$ and $6.47K$
utterances, respectively. We split the dataset into 70:20:10 ratio as the
train, test, and validation sets, respectively.

In contrast to the regular patient-doctor conversations (e.g., SOAP), the
dialogue sessions in HOPE are usually lengthy ($\sim 59$ utterances per
session). Moreover, the utterances in these sessions are themselves
significantly longer as compared to other conversational datasets, with the
average length of utterance of a patient being $103$ words, whereas for
therapist, it is $\sim 124$ words.

Figure 3. Architecture of SPARTA. For each utterance $U_{t}$, SPARTA computes
the local-context $L_{t}$, through a time-aware attention (TAA) mechanism on
the sliding-window memory unit and the current utterance. The dialogue-level
global context is maintained using a GRU $G_{t}$. Finally, the speaker-aware
and invariant local and global contexts are fused for the task.

##  4\. Methodology

We represent a therapy session as a conversation dialogue $D$ consisting of a
sequence of utterances $U[1:m]=\langle U_{1},U_{2},...,U_{m}\rangle$, where
$m$ is the number of utterances in a dialogue. These utterances are uttered by
the therapists and patients alternatively in a session. The objective of
SPARTA is to assign a correct dialogue-act label $y_{t}$ to every utterance
$U_{t}$ in the dialogue.

SPARTA is a transformer-based architecture that incorporates the speaker-aware
contextual information for the dialogue-act classification. In our analysis of
the HOPE dataset, we observed that a few of the dialogue-act labels are
majorly associated with the patient, while a few others are related to the
therapist. To model the speaker dynamics within a conversation dialogue, we
consolidate a speaker-aware (SA) module in addition to the speaker-invariant
(SI) module. The latter does not consider the distinction between a therapist
and a patient utterance, while the former module distinguishes between the two
through a pre-trained speaker identification module. Moreover, we also propose
a novel time-aware attention (TAA) mechanism that considers the positions of
contextual utterances during the attention computation. We hypothesize that
the recent past contextual utterances have higher significance than the
distant past utterances; hence, as opposed to the standard attention
mechanism, TAA focuses more on the nearby (local) utterances.

For each utterance $U_{t}$ in a conversation dialogue, we extract the semantic
representation through pre-trained RoBERTa model (Liu et al., 2019), which is
subsequently utilized to leverage the local ($L_{t}$) and global ($G_{t}$)
contextual information within the dialogue. We incorporate a sliding-window
based dynamic memory unit to compute local context through a time-aware
attention mechanism. In parallel, we employ a GRU layer to capture the
dialogue history as global context. Our analysis reveals that a few dialogues
contain utterances which discuss a topic (or an entity) that has occurred at
the initial stage of the dialogue, and to correctly exploit the semantic of
the utterance, the global information is desirable. We repeat the process of
local and global context extraction for both speaker-aware and speaker-
invariant setups. Finally, we combine these two representations with the
residual connections for the classification. Figure 3 shows a high-level
architecture diagram of SPARTA.

#### Utterance Representations

As mentioned above, SPARTA maintains two separate modules for capturing the
speaker-aware and speaker-invariant information. For speaker-invariant
representations, we employ a pre-trained RoBERTa language model which is
further fine-tuned on DAC task. The speaker aware module is a RoBERTa model
fine-tuned on Speaker Classification task.

$H^{SI}_{t}=RoBERTa^{SI}(U^{t});\ H^{SA}_{t}=RoBERTa^{SA}(U^{t})$.

#### Local Context and Time-Aware Attention

At every point in a dialogue, the nearby utterances provide important clues in
the prediction of a dialogue-act label for the current utterance. For example,
if the previous utterance is an information request (IRQ), then there is a
good chance that the next label should be either information delivery (ID) or
yes-no-answer (YNA). Therefore, we exploit the local context maintained in a
memory $M_{[t-k:t]}$ for each utterance $U_{t}$ in the dialogue, where $k$ is
the fixed local-context size.

(1) $\displaystyle
M^{SA}_{[t-k:t]}=M^{SA}_{[(t-1)-k:(t-1)]}-M^{SA}_{[k:k+1]}+M^{SA}_{[t-1:t]}\oplus
H^{SA}_{t}$ (2) $\displaystyle
M^{SI}_{[t-k:t]}=M^{SI}_{[(t-1)-k:(t-1)]}-M^{SI}_{[k:k+1]}+M^{SI}_{[t-1:t]}\oplus
H^{SI}_{t}$

We utilize TAA to learn the importance of contextual utterances based on their
distance from the current position. At first, we pass the utterance
representation ($H_{t}$), as computed by RoBERTa in the previous step, through
a tanh activation layer to obtain the pooler output, and subsequently project
it as the query ($q\in\mathbb{R}^{1\times d}$) vector in the attention
computation. On the other hand, the contextual memory $M_{[t-k:t]}$ is
projected as the key ($K\in\mathbb{R}^{k\times d}$) and value
($V\in\mathbb{R}^{k\times d}$) matrices. Next, we encode the local context as
follows:

$\displaystyle L_{t}=$ $\displaystyle Softmax\left(\frac{qK^{T}}{D}\right)V;\
D\in\mathbb{R}^{k},\ D_{i}=\frac{1}{i};\ \forall k:\ 1\leq i\leq k$

To extract the time-aware feature, we scale-down the dot product between the
query and the key by a monotonically decreasing function of time. The inverse
function was chosen based on its empirical advantage as shown in (Baytas et
al., 2017). The hypothesis for scaling-down the dot-product is due to the fact
that as we move deeper into the dialogue history their influence on the
dialogue-act reduces accordingly. Similar interaction dynamics was used in
(Zhang et al., 2021); but to our knowledge, we introduce an inverse function
for the first time to compute the fixed window size attention for local
context. Following the above procedure, we compute local contexts $L_{t}^{SA}$
and $L_{t}^{SI}$ for both speaker-aware and speaker-invariant modules,
respectively.

#### Global Context

As the dialogue progresses, we maintain the global context of the dialogue
through a GRU layer on top of the RoBERTa hidden representations.

#### Fusion and Final classification

Finally, we fuse the local and global contexts of speaker-aware and speaker-
invariant modules for the final classification. We also add residual
connections for better gradient flow during backpropagation. Our validation
results supplement the choice of concatenation as the fusion operation to be
better than other operations such as global max-pooling, global mean-pooling,
etc.

##  5\. Experiments, Results and Analysis

In this section, we report our experimental results, comparative study, and
other analyses.

#### Baselines

We choose the following existing systems as baselines. $\blacktriangleright$
CASA (Raheja and Tetreault, 2019a): It is a context-aware attention-based
system for the dialogue-act classification. It uses RNNs at dialogue and
utterance levels and computes context-aware self attention before the final
classification. $\blacktriangleright$ SA-CRF (Shang et al., 2020a): This
recent baseline incorporates a CRF layer for the classification. Moreover, it
consolidates the speaker-change information using a Bi-LSTM encoder.

In addition to these recent baselines on dialogue-act classification, we also
include other sequence-labelling classification systems. $\blacktriangleright$
DRNN (Wang, 2018): It is a novel Disconnected RNNs architecture which
incorporates the position-invariant features for modelling.
$\blacktriangleright$ ProSeqo (Kozareva and Ravi, 2019): This proposed to
efficiently handle the short and long texts using dynamic recurrent
projections. ProSeqo avoids the use of store-and-lookup in pre-trained word-
embeddings model through the context and locality-sensitive projections (Ravi,
2017). $\blacktriangleright$ TextVDCNN (Conneau et al., 2017): This is a deep
convolutional network with residual connections for text classification. The
convolutional layer works at character-level, and k-max pooling is used to
down-sample the output of convolutional layers for classification.
$\blacktriangleright$ TextRNN (Liu et al., 2016): This was the first work to
integrate RNNs into the multi-task learning framework. We use the uniform
layer architecture as described in the paper. $\blacktriangleright$ RoBERTa
(Liu et al., 2019): We use RoBERTa as a baseline in this work due to its
superioirity on various benchamarks. RoBERTa is similar to (Devlin et al.,
2019). It is an encoder-only language model trained with masked language
modelling objective on vast amounts of unlabelled data in unsupervised manner.

#### Experimental Results

For the experiments, we randomly split the HOPE dataset into $70:20:10$ ratio
for the train, test, and validation sets. To measure the performances of
SPARTA and other baseline systems, we compute macro-F1, weighted-F1, and
accuracy scores. We implemented our system in PyTorch (Paszke et al., 2019)
and utilized the pre-trained models from Huggingface Transformers library.

Table 3. Table showcasing the performance of baseline models as compared with
our SPARTA model. $U^{t}$ represents the use of Utterance representation, GC
represents Global Context, LC represents Local Context and SA means the
presence of Speaker Aware representations. The dagger symbol (${\dagger}$)
represents statistically significant results compared to the best baseline,
CASA.

Model | Type of Modelling | Precision | Recall | F1 | Accuracy
---|---|---|---|---|---
Macro | Weighted | Macro | Weighted | Macro | Weighted
TextVDCNN (Conneau et al., 2017) | $U^{t}$ | 11.01 | 21.02 | 19.53 | 38.53 | 13.37 | 36.81 | 41.77
ProSeqo (Kozareva and Ravi, 2019) | $U^{t}$ | 9.77 | 17.60 | 11.20 | 27.90 | 7.11 | 14.29 | 27.35
RoBERTa (Liu et al., 2019) | $U^{t}$ | 51.01 | 58.12 | 47.14 | 52.97 | 43.97 | 49.13 | 52.97
1-9 TextRNN (Liu et al., 2016) |  $U^{t}$ \+ GC | 30.27 | 37.92 | 27.9 | 41.76 | 25.55 | 36.81 | 41.77
DRNN (Wang, 2018) |  $U^{t}$ \+ GC | 28.39 | 36.72 | 31.87 | 44.32 | 28.12 | 37.82 | 44.32
CASA (Raheja and Tetreault, 2019b) |  $U^{t}$ \+ GC | 59.78 | 62.56 | 51.22 | 58.46 | 51.65 | 55.95 | 58.46
SPARTA-BS |  $U^{t}$ \+ GC | 58.94 | 62.31 | 52.02 | 57.70 | 51.83 | 54.98 | 57.70
1-9 SA-CRF (Shang et al., 2020a) |  $U^{t}$ \+ LC +SA | 33.30 | 38.97 | 26.18 | 45.07 | 35.97 | 24.20 | 45.07
SPARTA-BS |  $U^{t}$ \+ GC + SA | 58.87 | 63.02 | 53.28 | 58.41 | 52.22 | 55.57 | 58.41
SPARTA-MHA (3-fold CV) | $U^{t}$ \+ LC + GC + SA | 69.60 | 71.77 | 59.45 | 62.67 | 59.00 | 62.12 | 62.67
SPARTA-TAA (3-fold CV) | 71.01 | 72.36 | 60.49 | 63.82 | 60.74 | 63.38 | 63.82
1-9 SPARTA-MHA | $U^{t}$ \+ LC + GC + SA | 60.24 | 66.53 | 59.64 | 63.45 | 58.16 | 63.26 | 63.45
SPARTA-TAA | 62.15 | 67.36 | 61.13 | 64.75 | 60.29† | 64.53† | 64.75†
Significance T-test† (p-value) |  |  |  |  | 0.009 | 0.014 | 0.048

Table 4. Label-wise classification report for SPARTA-TAA.

| ACK | CD | CRQ | GC | GT | ID | IRQ | OD | NA | PA | ORQ | YNQ
---|---|---|---|---|---|---|---|---|---|---|---|---
Pre | 44.08 | 89.91 | 50.72 | 87.38 | 60.24 | 63.76 | 70.95 | 53.19 | 58.75 | 57.33 | 48.5 | 61.36
Rec | 52.34 | 48.04 | 49.53 | 54.38 | 65.79 | 81.02 | 77.81 | 44.64 | 71.21 | 54.43 | 76.47 | 57.86
F1 | 47.86 | 62.62 | 50.12 | 67.04 | 62.89 | 71.36 | 74.22 | 48.54 | 64.38 | 55.84 | 59.09 | 59.56

The hyperparameters are listed in Table 6. The experimental results of SPARTA
is presented in Table 7 in appendix section. Since SPARTA incorporates three
major components – local context, global context, and speaker-aware modules.
The SPARTA-TAA system obtains the best scores of $60.29$ macro-F1, $64.53$
weighted-F1, and $64.75\%$ accuracy, as reported in the last row of Table
2(appendix). We have also shown our observations on the ablation study on all
three key-factors: Contextual information, Time-aware attention, and Speaker
dynamics in ablation section in the appendix.

We also present the label-wise performance of SPARTA in Table 4. We can
observe that SPARTA consistently yields good scores for the majority of the
dialogue-acts, except for the Acknowledgement (ACK) where it records F1-score
of merely 47.86%. Even for the under-represented labels (ORQ,NA and PA) in
HOPE, SPARTA reports good F1-scores of 59.09%, 64.38% and 55.84%, respectively
for these three labels.

#### Comparative Analysis:

We compare SPARTA with various existing systems and other baselines. The
comparative analysis is reported in Table 3. Based on the type of modelling,
we categorize the baselines into three groups – utterance-driven ($U_{t}$),
utterance+global context driven ($U_{t}$ \+ GC), and utterance + global
context + speaker-aware driven ($U_{t}$ \+ GC + SA). Comparatively, SPARTA
incorporates local context in addition to utterance, global context, and
speaker dynamics ($U_{t}$ \+ LC + GC + SA). In the first category, the
standard RoBERTa model attains the best macro-F1, weighted-F1 and accuracy of
$43.97$, $49.13$, and $52.97\%$, respectively. In comparison, CASA (Raheja and
Tetreault, 2019a) yields the improved weighted-F1 and accuracy scores at
$55.95\%$ ($+6.82\%$) and $58.46\%$ ($+5.49\%$), respectively, with the global
context as an additional information. Finally, we experiment with SA-CRF
(Shang et al., 2020a) which also includes the speaker-dynamics for the
dialogue-act classification; however, its performance on HOPE is not at par
with CASA (Raheja and Tetreault, 2019a). It reports $35.97$, $24.20$, and
$45.07\%$ macro-F1, weighted-F1, and accuracy, respectively. In comparison,
SPARTA-TAA obtains significant improvements over all baselines. It reports
improvements of $+8.64\%$, $+8.58\%$, and $+6.29\%$ in macro-F1 ($60.29$),
weighted-F1 ($64.53$), and accuracy ($64.75\%$), respectively, as compared to
CASA suggesting the incorporation of local context extremely effective. Note
that ProSeqo (Kozareva and Ravi, 2019) and CASA (Raheja and Tetreault, 2019a)
are currently the state-of-the-art on switchboard dialogue-act corpus
benchmark101010https://paperswithcode.com/sota/dialogue-act-classification-on-
switchboard; yet they report inferior scores on HOPE compared to SPARTA.
Moreover, we also report the mean of the 3-fold cross-validation results for
both SPARTA-MHA and SPARTA-TAA, and the results are consistent with the train-
val-test split case. We also perform statistical significance T-test comparing
SPARTA-TAA and the best performing baseline (CASA). We observe that our
results are significant with $>95\%$ confidence across macro-F1
(p-value$=0.009$), weighted-F1 (p-value$=0.014$), and accuracy
(p-value$=0.048$) values.

Figure 4. Confusion matrix for SPARTA-TAA. Table 5. An example therapy session
showcasing the differences in predictions between the baseline and SPARTA. We
truncate the dialogue due to its length (total 151 utterances).

Utterances | Ground Truth | Prediction
---|---|---
CASA |  SPARTA-TAA |  SPARTA-MHA
Therapist: Hi Angela, how are you doing today ? | GT | GT | GT | GT
Patient: I am all right, thanks | GT | GT | GT | GT
… …. | .. | .. | .. | ..
Therapist: How’s your mood overall? | IRQ | IRQ | IRQ | IRQ
Patient: I mean, what can I say I lost my job. I’m seeing you because I have a drinking problem. So things are not that great. | ID | ID | ID | CD
… …. | .. | .. | .. | ..
Therapist: When you’re drinking does it still feel good? | IRQ | IRQ | IRQ | IRQ
Patient: Nothing really feels good. | ID | ID | ID | ID
Therapist: Just gets back to normal? | CRQ | CRQ | CRQ | CRQ
Patient: Yeah, pretty much. | CD | ID | CD | ID
Therapist: When was the last time you drank ? | IRQ | IRQ | IRQ | IRQ
Patient: This morning. | ID | ID | ID | GC
… …. | .. | .. | .. | ..
Patient: Oh, I should stop. I mean, like, I’m not, not unrealistic. | OD | ID | OD | ID
Therapist: Yeah | ACK | OD | ACK | ACK

#### Error Analysis:

In this section, we present two-way error analyses of SPARTA in terms of
quantitative and qualitative evaluations. Quantitative analysis: We report the
confusion matrix for SPARTA-TAA in Figure 4. We observe three pairs with
significant error-rates ($\geq 25\%$) – YNQ:IRQ ($26\%$), OD:ID ($43\%$),
ID:ACK ($28\%$). For the prediction of information delivery (ID), SPARTA is
confused most of the time with other classes – $19\%$ with PA, $20\%$ with NA,
$43\%$ with OD, $13\%$ with GT, $17\%$ with GC, $14\%$ with CRQ, $28\%$ with
ACK, and $22\%$ with CD.. We can relate this behaviour to the diversity of the
utterances with ID tag, i.e., the discussion in these utterances generally
contains a fair segments of utterances from other dialogue-acts (e.g., ‘Yeah,
That’s something I always do.’ could be easily be confused with PA). The other
prominent error-case is found in IRQ:YNQ pair, we observe a confusion of
$26\%$ between IRQ and YNQ utterances because of the versatile questioning
behaviour. For the remaining cases, error-rates are nominal. Thus, we
articulate that SPARTA can be further improved with more balanced dataset.

Qualitative analysis:  Table 5 shows a sample session along with actual and
the predicted dialogue-act labels for SPARTA and the best baseline model
(i.e., CASA). Due to the length of the conversation, we truncate some of the
utterances in between; however, the gist of the conversation is that the
patient is stressed of losing her job and having drinking issues and the
therapist is trying to understand the core problem. The conversation has
mostly information request and information delivery types of utterances with a
few other dialogue-acts in between (e.g., CRQ GT, etc.).

We can observe that for the first three utterances, SPARTA and the baseline
are consistent with the actual labels. For the fourth utterance, the SPARTA-
MHA model misclassifies the utterance as CD when the patient is clearly
providing objective information which is necessary for further conversation.
In the seventh utterance, the therapist wants more clarification about the
drinking habits, and the patient provides the clarification but CASA and
SPARTA-TAA models wrongly classify this utterance as ID when there is no
objective information being provided here. Next, we notice that when the
patient talks about stopping this and provides her opinion that she is not
unrealistic, the SPARTA-MHA and CASA models predict the wrong label for this
utterance as ID. In the next utterance, the therapist acknowledges this
opinion, but the CASA model predicts the wrong label. So, not only the SPARTA-
TAA model is able to capture the semantics of the utterances better, it also
utilizes the contextual information in a better way by relating the past
information about the speaker with the current utterances.

General Discussion:  The work presented in this paper was motivated solely by
the dire need of understanding conversations that occur in counselling
sessions and to design solutions that would help the therapists to better
understand the intents of their patients. However, the proposed model can be
easily adaptable to other domain (such as normal chit-chat-based conversation)
as well. To ensure we do not deviate from the prime objective of this work, we
restrict ourselves to explore the dialog-act classification in the counselling
conversations only.

##  6\. Limitations and Ethical Considerations

We aim to tackle a very sensitive and a pervasive public-health crisis. We
transcribe the data from publicly available counselling videos. The automatic
transfer of utterance from the speech modality to the text causes some
information loss, though we tried our best to recover them through manual
intervention. Moreover, we consulted with mental-health professionals and
linguists in preparing the annotation guideline. However, the annotator’s bias
cannot be ruled out completely. The names of the patients and therapists
involved in these sessions have been systematically masked. Another important
aspect of the current work is that the majority of the sessions in HOPE belong
to the mental health professionals and patients based in United States. Hence,
the effectiveness of SPARTA on data from other geographical or demographical
regions may vary. We understand that the building computational models in
mental-health avenues has high-stakes associated with it and ethical
considerations, therefore, become necessary. No technology will work perfectly
in solving the problems related to mental health (Miner et al., 2020). It is
important to note that we do not make any diagnostic claims. Further, the
deployment of any such technology will be done keeping in mind the safety-
risks and mitigating any sources of bias that may arise.

##  7\. Conclusion and Future Work

Paying heed to the consequences of the COVID-19 pandemic on mental health, in
this paper, we raised the attention on the much deserved research on dialogue
system for mental-health counselling. To this end, we collected and developed
the HOPE dataset for the dialogue-act classification in dyadic counselling
conversations. We defined twelve dialogue-act labels to cater to the
requirement of counselling sessions. In total, we annotated $\sim 12.9k$
utterances across $212$ sessions. We also proposed SPARTA, a novel
transformer-based speaker and time-aware joint contextual learning model for
dialogue-act classification. SPARTA utilizes the global and local context in
the speaker-aware and speaker-invariant setups while also using a novel
memory-driven time-aware attention mechanism to leverage the local context.
Our extensive ablation study and the comparative analysis established the
superiority of SPARTA over several existing models. In future, we would like
to extend our effort in the development of dialogue-systems for mental-health
counselling by including other crucial tasks such as emotion recognition,
dialogue summary generation, dialogue-state tracking, empathetic response
generation, etc. Another dimension of the future work is to include other
languages and demographic diversities to cater the requirements of a larger
population.

## References

  * (1)
  * Ahmadvand et al. (2019) Ali Ahmadvand, Jason Ingyu Choi, and Eugene Agichtein. 2019\.  Contextual dialogue act classification for open-domain conversational agents. In _SIGIR_. 1273–1276.
  * Baytas et al. (2017) Inci M. Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K. Jain, and Jiayu Zhou. 2017\.  Patient Subtyping via Time-Aware LSTM Networks. In _SIGKDD_. 65–74.
  * Cerisara et al. (2018) Christophe Cerisara, Somayeh Jafaritazehjani, Adedayo Oluokun, and Hoa T Le. 2018.  Multi-task dialog act and sentiment recognition on Mastodon. In _COLING_. 745–754.
  * Chen et al. (2018) Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018.  Dialogue act recognition via crf-attentive structured network. In _SIGIR_. 225–234.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In _EMNLP_. 1724–1734.
  * Cohen (1960) Jacob Cohen. 1960\.  A Coefficient of Agreement for Nominal Scales.  _Educational and Psychological Measurement_ 20, 1 (1960), 37–46.
  * Colombo et al. (2020) Pierre Colombo, Emile Chapuis, Matteo Manica, Emmanuel Vignon, Giovanna Varni, and C. Clavel. 2020\.  Guiding attention in Sequence-to-sequence models for Dialogue Act prediction. In _AAAI_.
  * Conneau et al. (2017) Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. 2017.  Very Deep Convolutional Networks for Text Classification. In _EACL_. 1107–1116.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL_. 4171–4186.
  * Fine et al. (2020) Alex Fine, Patrick Crutchley, Jenny Blase, Joshua Carroll, and Glen Coppersmith. 2020.  Assessing population-level symptoms of anxiety, depression, and suicide risk in real time using NLP applied to social media data. In _Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science_. 50–54.
  * Godfrey et al. (1992) J. J. Godfrey, E. C. Holliman, and J. McDaniel. 1992\.  SWITCHBOARD: telephone speech corpus for research and development. In _[Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing_ , Vol. 1. 517–520 vol.1.  https://doi.org/10.1109/ICASSP.1992.225858
  * Gratch et al. (2014) Jonathan Gratch, Ron Artstein, Gale M Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, et al. 2014\.  The distress analysis interview corpus of human and computer interviews.. In _LREC_. 3123–3128.
  * Grau et al. (2004) Sergio Grau, Emilio Sanchis, Maria Jose Castro, and David Vilar. 2004.  Dialogue act classification using a Bayesian approach. In _9th Conference Speech and Computer_.
  * Hua et al. (2020) Kai Hua, Zhiyuan Feng, Chongyang Tao, Rui Yan, and Lu Zhang. 2020.  Learning to Detect Relevant Contexts and Knowledge for Response Selection in Retrieval-Based Dialogue Systems. In _CIKM_. 525–534.
  * Kozareva and Ravi (2019) Zornitsa Kozareva and Sujith Ravi. 2019.  ProSeqo: Projection Sequence Networks for On-Device Text Classification. In _EMNLP-IJCNLP_. 3894–3903.
  * Kretzschmar et al. (2019) K. Kretzschmar, H. Tyroll, G. Pavarini, A. Manzini, and I. Singh. 2019.  Can Your Phone Be Your Therapist? Young People’s Ethical Perspectives on the Use of Fully Automated Conversational Agents (Chatbots) in Mental Health Support.  _Biomed Inform Insights_ 11 (2019), 1178222619829083\.
  * Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001\.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In _ICML_. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 282–289.
  * LeCun et al. (1989) Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989.  Backpropagation Applied to Handwritten Zip Code Recognition.  _Neural Comput._ (Dec. 1989), 541–551.
  * Lee and Dernoncourt (2016) Ji Young Lee and Franck Dernoncourt. 2016.  Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks. In _NAACL_. Association for Computational Linguistics, San Diego, California, 515–520.
  * Levis et al. (2020) Maxwell Levis, Christine Leonard Westgate, Jiang Gui, Bradley V. Watts, and Brian Shiner. 2020\.  Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models.  _Psychological Medicine_ (2020), 1–10.
  * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017\.  DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. In _IJCNLP_.
  * Liu et al. (2016) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016.  Recurrent Neural Network for Text Classification with Multi-Task Learning. In _IJCAI_. AAAI Press, 2873–2879.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ abs/1907.11692 (2019).
  * Miner et al. (2020) Adam S. Miner, Albert Haque, Jason Alan Fries, S. Fleming, D. Wilfley, G. Terence Wilson, A. Milstein, D. Jurafsky, B. Arnow, W. Stewart Agras, L. Fei-Fei, and N. Shah. 2020\.  Assessing the accuracy of automatic speech recognition for psychotherapy.  _NPJ Digital Medicine_ 3 (2020).
  * Mishra et al. (2019) Rohan Mishra, Pradyumn Prakhar Sinha, Ramit Sawhney, Debanjan Mahata, Puneet Mathur, and Rajiv Ratn Shah. 2019.  SNAP-BATNET: Cascading author profiling and social network graphs for suicide ideation detection on social media. In _NAACL: Student Research Workshop_. 147–156.
  * Morbini et al. (2014) Fabrizio Morbini, David DeVault, Kallirroi Georgila, Ron Artstein, David Traum, and Louis-Philippe Morency. 2014.  A Demonstration of Dialogue Processing in SimSensei Kiosk. In _SIGDIAL_. 254\.
  * Moreno et al. (2020) Spain C Moreno, C Arango, Carmen Moreno, Til Wykes, Silvana Galderisi, Merete Nordentoft, Nicolas Crossley, Nev Jones, Mary Cannon, Christoph U Correll, Louise Byrne, Sarah Carr, Eric Y H Chen, Philip Gorwood, Sonia Johnson, Hilkka Kärkkäinen, John H Krystal, Jimmy Lee, Jeffrey Lieberman, Carlos López-Jaramillo, Miia Männikkö, Michael R Phillips, Hiroyuki Uchida, Eduard Vieta, Antonio Vita, and Celso Arango. 2020\.  Position Paper How mental health care should change as a consequence of the COVID-19 pandemic.  _The Lancet Psychiatry_ 7 (2020), 813–824.  https://doi.org/10.1016/S2215-0366(20)30307-2
  * Naous et al. (2020) Tarek Naous, Christian Hokayem, and Hazem Hajj. 2020\.  Empathy-driven Arabic Conversational Chatbot. In _Proceedings of the Fifth Arabic Natural Language Processing Workshop_. 58–68.
  * Ortega et al. (2019) Daniel Ortega, Chia-Yu Li, Gisela Vallejo, Pavel Denisov, and Ngoc Thang Vu. 2019.  Context-aware neural-based dialog act classification on automatically generated transcriptions. In _ICASSP_. 7265–7269.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _NeurIPS_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). 8024–8035.
  * Pennebaker et al. (2003) J. Pennebaker, M. Mehl, and Kate Niederhoffer. 2003.  Psychological aspects of natural language. use: our words, our selves.  _Annual review of psychology_ 54 (2003), 547–77.
  * Qin et al. (2020) Libo Qin, Wanxiang Che, Yangming Li, Minheng Ni, and Ting Liu. 2020.  DCR-Net: A Deep Co-Interactive Relation Network for Joint Dialog Act Recognition and Sentiment Classification. In _AAAI_. 8665–8672.
  * Raheja and Tetreault (2019a) Vipul Raheja and Joel Tetreault. 2019a.  Dialogue Act Classification with Context-Aware Self-Attention. In _NAACL_. 3727–3733.
  * Raheja and Tetreault (2019b) Vipul Raheja and Joel Tetreault. 2019b.  Dialogue Act Classification with Context-Aware Self-Attention. In _NAACL_. 3727–3733.
  * Ravi (2017) Sujith Ravi. 2017\.  ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections.  _CoRR_ abs/1708.00630 (2017).  arXiv:1708.00630 http://arxiv.org/abs/1708.00630
  * Reithinger and Klesen (1997) Norbert Reithinger and Martin Klesen. 1997.  Dialogue act classification using language models. In _Fifth European Conference on Speech Communication and Technology_.
  * Rumelhart et al. (1986) D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986\.  _Learning Internal Representations by Error Propagation_.  MIT Press, Cambridge, MA, USA, 318–362.
  * Saha et al. (2020) Tulika Saha, Dhawal Gupta, Sriparna Saha, and P. Bhattacharyya. 2020.  Emotion Aided Dialogue Act Classification for Task-Independent Conversations in a Multi-modal Framework.  _Cognitive Computation_ (2020), 1–13.
  * Shang et al. (2020a) Guokan Shang, Antoine Tixier, Michalis Vazirgiannis, and Jean-Pierre Lorré. 2020a.  Speaker-change Aware CRF for Dialogue Act Classification. In _COLING_. 450–464.
  * Shang et al. (2020b) Guokan Shang, Antoine Jean-Pierre Tixier, M. Vazirgiannis, and J. Lorré. 2020b.  Speaker-change Aware CRF for Dialogue Act Classification. In _COLING_.
  * Van Dijk (2011) Teun A Van Dijk. 2011\.  _Discourse studies: A multidisciplinary introduction_.  Sage.
  * Wang (2018) Baoxin Wang. 2018\.  Disconnected Recurrent Neural Networks for Text Categorization. In _ACL_. 2311–2320.
  * Wang et al. ([n.d.]) Xiaomei Wang, Sudeep Hegde, Changwon Son, Bruce Keller, Alec Smith, and Farzan Sasangohar. [n.d.].  Investigating Mental Health of US College Students During the COVID-19 Pandemic: Cross-Sectional Survey Study.  _JMIR_ 22, 9 (Sept. [n. d.]), e22817.
  * Wei et al. (2019) Wei Wei, Jiayi Liu, Xianling Mao, Guibing Guo, Feida Zhu, Pan Zhou, and Yuchong Hu. 2019.  Emotion-Aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction. In _CIKM_. 1401–1410.
  * Weizenbaum (1966) Joseph Weizenbaum. 1966\.  ELIZA – A Computer Program For the Study of Natural Language Communication Between Man and Machine.  _CACM_ 9 (January 1966), 36–45.  Issue 1.
  * Yu et al. (2019) Yue Yu, Siyao Peng, and Grace Hui Yang. 2019.  Modeling Long-Range Context for Concurrent Dialogue Acts Recognition. In _CIKM_. 2277–2280.
  * Zhang et al. (2021) Yazhou Zhang, Prayag Tiwari, Dawei Song, Xiaoliu Mao, Panpan Wang, Xiang Li, and Hari Mohan Pandey. 2021.  Learning interaction dynamics with an interactive LSTM for conversational sentiment analysis.  _Neural Networks_ 133 (2021), 40 – 56.

##  Appendix A Speaker Aware and Speaker Invariant Representations

Furthermore, in the quest of explanation, we visualize the speaker-aware
($H_{t}^{SA}$) and speaker-invariant ($H_{t}^{SA}$) utterance representation
in Figure 5. We employ principal component analysis to project the
representations into 2D. We can observe that the speaker-aware representation
distinguishes between the patient’s and therapist’s utterances pretty well. In
contrast, the patient and therapist utterances are mixed in the speaker-
invariant representation. Considering the underlying task, where the therapist
and the patient have the major contributions in speaker-initiative and
speaker-responsive dialogue-acts, speaker-aware representation provides
crucial assistance to SPARTA.

Figure 5. Speaker-aware (left) and speaker-invariant (right) utterance
representations for the two speakers in a 2D space.

##  Appendix B Reproducibility

We use PyTorch 111111https://pytorch.org/ and Pytorch Lightning
121212https://www.pytorchlightning.ai/ frameworks for all our experiments. We
extensively used Hugging Face Library for implementation of transformer based
NLP models.

####  B.0.1. Hardware Utilization:

The complete model training was done on a linux machine with following
specifications:

  * •

GPU: TeslaV100

  * •

Memory: 32GB

  * •

Linux 64 Bit: Ubuntu 18.04 LTS

####  B.0.2. Output Logs:

We use WandB131313https://wandb.ai/ library to log all our results.

####  B.0.3. Experiment Details

The categorical cross-entropy loss is optimized using the Adam optimizer. We
use dropout$=0.15$ and train the models for 50 epochs in a batch of 8 samples.
We use EarlyStopping to prevent our model from overfiting. A comprehensive
list of hyperparameters is given in 6.

We use pretrained RoBERTa available from the
Hugginface141414https://huggingface.co/ transformers for speaker aware as well
as speaker invariant utterance representations. We create separate classes for
each sub-module in our model. Built-in dataset and dataloader classes are used
to load the data in the correct format and a total of 6 workers are used to
load the data. For the GRU, we use input size and hidden dimension of 768 with
1 layer. For the classifier sub-module, we use a dropout of 0.1 a linear layer
of 768 neurons and LeakyReLU as the activation. Our attention blocks comprise
of 3 Linear Layers for keys,queries and values and one linear layer to project
the keys, queries and values.

####  B.0.4. Requirements

  * •

pytorch==1.7.1

  * •

pytorch-lightning==1.10

  * •

transformers==4.1.0

  * •

scikit-learn==0.24.0

  * •

numpy==1.19.4

  * •

pandas==1.1.5

  * •

wandb==0.10.12

  * •

datasets==1.1.3

Table 6. Training hyperparameters Hyperparameter | Value
---|---
Max Length | 512
Batch Size | 8
Hidden Size | 768
LayersGRU | 1
LayersRoBERTa | 12
Dropout | 0.15
Window Size | 6
Learning Rate | 1e-5
Precision | 32
Min Delta | 0.001
Patience | 5
Epochs | 50
Table 7. Experimental results of SPARTA on HOPE Dialogue-act Corpus. TAA:
Time-aware attention; MHA: Multihead attention; BS: Baseline - without
attention; LC: Local context; GC: Global context; SA: Speaker-aware. Best
results are highlighted in bold.

Model | LC | GC | SA | Precision | Recall | F1 | Accuracy
---|---|---|---|---|---|---|---
Macro | Weighted | Macro | Weighted | Macro | Weighted |
SPARTA-BS | - | ✓ | - | 58.94 | 62.31 | 52.02 | 57.70 | 51.83 | 54.98 | 57.70
- | ✓ | ✓ | 58.87 | 63.02 | 53.28 | 58.41 | 52.22 | 55.57 | 58.41
1-11 SPARTA-MHA | ✓ | - | - | 51.57 | 61.11 | 50.01 | 53.10 | 46.51 | 53.25 | 53.10
✓ | ✓ | - | 58.99 | 63.06 | 53.00 | 58.23 | 52.24 | 55.49 | 58.23
✓ | ✓ | ✓ | 60.24 | 66.53 | 59.64 | 63.45 | 58.16 | 63.26 | 63.45
1-11 SPARTA-TAA | ✓ | - | - | 52.90 | 60.29 | 49.87 | 54.84 | 47.63 | 53.72 | 54.84
✓ | ✓ | - | 60.25 | 66.25 | 57.23 | 59.08 | 53.73 | 55.07 | 59.08
✓ | ✓ | ✓ | 62.15 | 67.36 | 61.13 | 64.75 | 60.29 | 64.53 | 64.75

####  B.0.5. Training and Evaluation

  * •

Create a python environment and install the above libraries.

  * •

Prepare the data and store it in csv files and change the data directory in
the config.py file in ’data_dir’ and ’data_files’ variables.

  * •

The models/ directory contains the code of all the sub-modules.

    * –

The file Classifier.py contains the code of a generic classifier that
classifies data into two classes.

    * –

SpeakerClassifier.py contains the code of the speaker classifier. It has been
isolated so that ablation studies can be performed easily.

    * –

MHA.py contains the code of our Multi-Head attention module whereas
Relevance.py contains the code of our Time Aware Attention module.

    * –

Roberta.py and GRU.py contains the code to initialize RoBERTa and GRU models
respectively.

    * –

DAC.py contains the code of our complete architecture.

  * •

The file dataset.py in dataset/ directory contains the code to create a custom
dataset class and load data using the torch.utils.data.Dataset and
torch.utils.data.DataLoader wrappers.

  * •

Execution

    * –

Trainer.py contains the code to create pytorch-lightning trainer with wandb
logger to log the results.

    * –

Use config.py to change the configuration of models or to change the
hyperparameters used in the experiments.

    * –

Run python main.py to run the code and get results.

##  Appendix C Ablation Study

To verify the effectiveness of the time-aware attention to compute the local
context, we also report the results with standard multi-head attention (MHA)
mechanism. The first set of results (i.e., SPARTA-BS or the baseline version
of SPARTA) in Table 7 represents the absence of local context in SPARTA. It
yields macro and weighted F1-scores of $51.83$ and $54.98$ respectively with
global context only. We obtain $57.70\%$ accuracy for the same setup. On the
other hand, incorporating the speaker-aware information into the system
improves the trio of accuracy, macro, and weighted F1-scores by $[1-2]$
points, suggesting a positive impact of speaker-dynamics in SPARTA.
Subsequently, we introduce the local context with multi-head attention in
SPARTA-MHA. In comparison with SPARTA-BS, SPARTA-MHA reports $52.24$ macro-F1
and $55.49\%$ weighted-F1 with both the local and global contexts.
Furthermore, we obtain $58.16$ $(+5.92\%)$ and $63.26$ $(+7.77\%)$ macro and
weighted F1-scores, respectively, with the inclusion of speaker-aware module
in the system as compared to SPARTA-MHA without the Speaker-Aware module.
Moreover, it yields better accuracy at $63.45\%$ $(+5.22\%)$. These results
clearly indicate the importance of local context and speaker-aware modules for
the dialogue-act classification. In the final stage of our experiments, we
replace the multi-headed attention with the proposed time-aware attention
mechanism (TAA).

###  C.1. Observations from Ablation Study

We summarize our observation as follows:

  * •

Contextual information: Dialogue-act labels in a counseling based conversation
not only depend on the abstract information of the dialogue but also on the
utterances in the immediate vicinity of the current utterance. As can be
observed from Table 7, the presence of both the global and local contextual
information plays an importance role in SPARTA. Moreover, the absence of
either of the component degrades the overall performance. This corroborates
that they carry distinct and diverse contextual information.

  * •

Time-aware attention: The comparison between the standard multi-head attention
and the novel time-aware attention highlights the importance of attending over
the relative positions of utterances. As stated earlier, extensive
experimental results show that TAA yields better performance compare to MHA
for all possible configurations.

  * •

Speakers dynamics: For all combinations, we observe performance drop of
$[3-4]\%$ without the speaker information. This is particularly apparent as a
majority portion of the counseling conversation is driven by the therapist,
thus the speaker-initiative dialogue-acts have higher relevance with the
therapist utterances. Therefore, our intuition of incorporating the speaker-
aware module as a critical component in SPARTA also corroborates the empirical
evidences.

##  Appendix D Role of Emotion

In earlier phases of experiments, we also considered the role of emotion in
dialogue act classification. We annotated a subset of our dataset with three
emotion classes consisting of ‘positive’, ‘negative’ and ‘neutral’. We found
that around $\sim 70\%$ of utterances by the patients belonged to ‘negative’
class whereas $\sim 90\%$ of therapists’ utterances belonged to ‘neutral’
class. Due to such imbalance, this data was not used in the final version of
our proposed architecture.