# Counseling Summarization using Mental Health Knowledge Guided Utterance
Filtering

Aseem Srivastava$1$, Tharun Suresh$1$, Sarah Peregrine (Grin) Lord$2,3$,
Md. Shad Akhtar$1$, Tanmoy Chakraborty$1$ $1$IIIT-Delhi, India; $2$University
of Washington; $3$Mpathic.ai  aseems, tharun20119, shad.akhtar,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
(2022)

###### Abstract.

The psychotherapy intervention technique is a multifaceted conversation
between a therapist and a patient. Unlike general clinical discussions,
psychotherapy’s core components (viz. symptoms) are hard to distinguish, thus
becoming a complex problem to summarize later. A structured counseling
conversation may contain discussions about symptoms, history of mental health
issues, or the discovery of the patient’s behavior. It may also contain
discussion filler words irrelevant to a clinical summary. We refer to these
elements of structured psychotherapy as counseling components. In this paper,
the aim is mental health counseling summarization to build upon domain
knowledge and to help clinicians quickly glean meaning. We create a new
dataset after annotating $12.9K$ utterances of counseling components and
reference summaries for each dialogue. Further, we propose ConSum, a novel
counseling-component guided summarization model. ConSum undergoes three
independent modules. First, to assess the presence of depressive symptoms, it
filters utterances utilizing the Patient Health Questionnaire (PHQ-9), while
the second and third modules aim to classify counseling components. At last,
we propose a problem-specific Mental Health Information Capture (MHIC)
evaluation metric for counseling summaries. Our comparative study shows that
we improve on performance and generate cohesive, semantic, and coherent
summaries. We comprehensively analyze the generated summaries to investigate
the capturing of psychotherapy elements. Human and clinical evaluations on the
summary show that ConSum generates quality summary. Further, mental health
experts validate the clinical acceptability of the ConSum. Lastly, we discuss
the uniqueness in mental health counseling summarization in the real world and
show evidences of its deployment on an online application with the support of
mpathic.ai.

Dialogue Summarization, Natural Language Processing

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2022; Washington, DC, USA††booktitle: Proceedings of the 28th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22), August
14–18, 2022, Washington, DC, USA††price: 15.00††doi:
10.1145/3534678.3539187††isbn: 978-1-4503-9385-0/22/08††ccs: Computing
methodologies Discourse, dialogue and pragmatics††ccs: Computing methodologies
Natural language generation Figure 1. A sample counseling session instance
from the MEMO dataset. Symptom and History, Patient Discovery, Reflecting, and
Discussion Filler are psychotherapy elements. The summaries pertaining to this
truncated snippet of conversation is shown at the bottom. _Note: The summary
and conversation are truncated for brevity._

##  1\. Introduction

Mental health counseling is one of the front-line defenses against mental
health illness. In medical and primary care settings, the doctor follows a
highly-structured assessment approach that elicits specific information about
the patient’s medical problems to rule out different diagnoses. In
psychotherapy counseling sessions, patients take center stage in elucidating
their situation with subtle details. The therapist introduces diverse
auxiliary context in conversations to put the patient at ease, discuss events
happening in the patient’s recent past including the feelings, reflections and
emotions that the patient experiences, and other relevant topics. Follow-up
conversations with the patient are also vital for a successful treatment. The
points of the counseling session that are crucial for continuity of care in
follow-up and treatment planning include: (a) the patient’s presenting
problem, (b) symptoms and diagnosis, (c) treatments (current and prior), (d)
mental status and risk assessment, and (e) other varied context and history.
An efficient way to present this information to a therapist is to summarize
the therapist-patient conversation. The process highlighted earlier,
differentiates a mental health counseling session from other medical
conversations, and extraction of relevant utterances is key to avoid noise in
generated summaries.

In this paper, we aim for the task of counseling conversation summarization.
Our work incorporates domain knowledge to generate summaries that inherit
essential psychotherapy elements. Earlier studies (See et al., 2017; Nallapati
et al., 2016) show attention based architectures to generate summary. Recent
studies (Afzal et al., 2020; Song et al., 2020) use segmentation of the
conversation based on topic similarities in summarizing dialogue. Extending
the ideas by incorporating mental health domain knowledge, we propose
‘counseling components’ for each utterance to include the understanding
associated with counseling conversation. Comprehensive analyses of counseling
conversations show that the conversation aims to discuss the reasons for the
problem, discover the patient’s insights, and reflect on their past. Hence, we
create the dataset, MEMO111MEMO: Mental hEalth suMmarizatOn dataset where we
annotate 12.9K utterances with the following four labels - _symptom and
history, patient discovery, reflecting, discussion filler_. Also, we annotate
and validate the dataset with the help of mental health experts.

The state-of-the-art models are capable of generating semantically rich text;
however, to pick the fitting symptoms and essence of the counseling problem is
still a challenge. In this paper, we incorporated mental health domain
knowledge using the Patient Health Questionnaire (PHQ-9) (Kroenke et al.,
2001) to represent an established set of questions to assess the patient’s
mental health symptoms. The PHQ-9 specifically assesses depressive symptoms
but represents an example of symptom-specific domain knowledge. This approach
could be generalized to other frequently-used assessments focusing on other
symptom profiles like anxiety (e.g., GAD-7). Taking advantage of domain
knowledge, we identify utterances categorized as discussion filler or
irrelevant and will not be prioritized. Labeling across different counseling
conversations aids the model to map similarities and serve as the supervision
signal to attend to salient utterances and extract appropriate information to
be a part of the summaries.

We propose ConSum222ConSum: Counseling Summarization, a mental health
counseling summarization architecture. The complete pipeline of the model
works in three different modules: (a) The first module (MH-Know) exploits
domain knowledge to filter utterances irrelevant to the counseling
summarization task. (b) Secondly, we propose a discussion filler
classification module that filters irrelevant utterances like ‘yeah’, ‘ummm’,
etc. (c) The third module predicts the counseling components to utilize the
structured therapy knowledge for summarization task.

We run our experiments on the MEMO dataset. Evaluations show improvements
across all baselines and discuss the effect of counseling components and
effective use of domain knowledge in our ConSum model. Finally, we discuss the
uniqueness of counseling summarization and explore the generalizability of
ConSum. Significant contributions of our work are as follows:

  * •

We propose ConSum, a summarization model that exploits mental health domain
knowledge and counseling components.

  * •

We propose a novel counseling summarization dataset - MEMO and present a novel
annotation scheme for psychotherapy elements viz. symptom and history, patient
discovery, reflecting aspects, and discussion filler in utterances of
counseling dialogue.

  * •

We propose a new problem specific metric to evaluate summaries i.e., Mental
Health Information Capture (MHIC) metric which reasonably evaluates summaries
that are most useful from a counseling’s perspective.

  * •

Mental health experts in mpathic.ai meticulously analyzed MEMO. They studied
the results on various clinical and linguistic parameters to evaluate the
acceptability of ConSum’s performance for commercial and clinical application
at scale.

##  2\. Related Work

Given the critical nature of the medical domain, the pace of adoption of
modern deep learning models did not gain enough traction for impactful change.
Instead, it was used to address mundane medical procedures. On the other hand,
summarization is a long-studied problem in text processing. Earlier, most of
the focus and improvements were on extractive summarization, with a drastic
shift to abstractive summarization recently. Some of the work done in the
medical health domain is discussed below, followed by a section on technical
progress in summarization methods.

###  2.1. Mental Health and NLP

One of the early works by Chen and Verma (2006) explored a simple information
retrieval system to fetch information from medical documents using document
summaries. They employed medical ontology to help fine-tune the user query;
however, simple summarization methods limited its effectiveness. Later, an
NLP-based framework was introduced by Konovalov et al. (2010) to identify a
personnel’s physical exposure (in a battle) and their emotional reaction to
it. Their work highlighted the importance of subtle indicators of mental
illness and the advantages of diagnosing “early components” in counseling.
Strauss et al. (2013) analyzed clinical forms in mental health to cut down on
the arduous task of manual analysis using machine learning. Their work paved
the way to explore machine learning as a tool to automate tasks dependably, to
a certain extent.

Secondly, certain efforts helped address mental health problems specifically.
Kennedy et al. (2016) showed a comprehensive study about the management of
adults with Major Depressive Disorder (MDD). Their analysis identifies
different stages of depression and discusses an appropriate level of
antidepressant to choose based on the patient’s age, anxiety levels, and long
episode duration. Subsequently, the prescribed antidepressants are monitored
for satisfactory patient response. This notion motivates the identification of
key indicators of the patient’s conversation with the therapist and follow-up
sessions that rely on the patient’s past interactions (and prescriptions).
Tran and Kavuluru (2017) experimented with neural network models to predict
the mental illness condition from neuropsychiatric notes. These notes
comprised 300 words on average about the patient’s present illness and events
associated with it, followed by a psychiatric review system that mentions
mental illness associated with the patient. Despite using various models such
as RNN, BiRNN, and LSTM, it is a straightforward classification problem among
the 13 predefined mental illnesses.

In deep learning based summarization models, extractive summarization (Chen et
al., 2020) addressed the problem of doctors going through elaborate discharge
diagnoses by summarizing them using the BERT model. They used character-level
tokens to reduce the parameter size of the BERT model to deploy in low-
resource setups. However, there were no sizeable efforts made in explicitly
capturing medical information in summaries.

Some of the contemporary approaches include the work by Ive et al. (2020)
where they generated artificial datasets from genuine mental health records.
Besides, Afzal et al. (2020) focused on identifying PICO (Patient/Problem,
Intervention, Comparison, and Outcome) on a well-structured sequence of
sentences from medical documents to present a concise extractive summary only.
Recently, to filter relevant utterances, Gaur et al. (2021) used domain
knowledge from patient health care questionnaire (PHQ-9) to build knowledge
graphs. They used an unsupervised approach to build abstractive summaries for
the counseling session. Zhang et al. (2021) showed the generation of summary
in two steps, one for building chunks of partial summaries and then second for
fusing those chunks to generate final summary. Moreover, Zafari and Zulkernine
(2021) showed an online application developed based on medical domain-specific
annotation and information extraction.

###  2.2. Dialogue Summarization

Broadly, the two categories of dialogue summarization tasks are extractive
(utterance filtering) and abstractive (semantically rich). Summarization has
been long studied under various settings (Atri et al., 2021; Dey et al.,
2020). Earlier, Chen et al. (2011) proposed a graph-based approach for
extractive summarization wherein topical similarities between each sentence
are identified using Probabilistic Latent Semantic Analysis (PLSA). Saggion
and Poibeau (2012) highlighted two broad extractive summarization techniques,
which are _superficial techniques_ including statistical-based approach and
_knowledge-based_ methods for pre-training from a large corpora.

An observation on conversation summaries reveals that capturing salient points
with a coherent narration (abstractive) is the most succinct format to
express. Hence, abstractive summarization gained traction with early work by
Nallapati et al. (2016) exploring encoder-decoder architecture which was
further extended with suitable attention mechanism by Vaswani et al. (2017)
for abstractive summarization. Then, a mix of extractive and abstractive
summarization with a suitable copy model was introduced by See et al. (2017).
They used a generator model for abstractive summarization, a copy model for
extractive summarization, and a coverage vector to limit repetition in
summarization. In the sequence, Chen and Bansal (2018) proposed a
reinforcement learning-based approach to extract the salient sentences and
concised them to summary sentences with an abstractive network. They employed
a specific emphasis to reduce redundancy in the extracted utterances from the
conversation.

Further nuanced observation of the relation between salient sentence
extraction and conversation utterances shows the dependence on certain groups
of utterances. On this basis, recent work by Narayan et al. (2018) introduced
topic distribution based on Latent Dirichlet Allocation (Blei et al., 2003).
Extending on the idea of topic segregation, Song et al. (2020) used three
labels to tag each utterance with - Problem Description, Diagnosis, and Other.
They showed that adding utterance labels aids in summarization performance.
Nevertheless, since their reference summaries are extractive, the complexities
of their models are limited. Further, Zhang et al. (2020b) and Raﬀel et al.
(2020) pushed the boundaries of abstractive summarization by generating
semantically rich sentences, diversely adapted to different generative tasks.
In the clinical conversation domain, Quiroz et al. (2020) and Krishna et al.
(2021) followed the approach of important utterance selection for medical
conversation summarization.

##  3\. MEMO: Mental Health Summarization Dataset

We create a novel dataset for the task of counseling conversation
summarization in which we extend data collected from the publicly available
counseling conversation dataset – HOPE (Malhotra et al., 2021). The dataset
contains $12.9K$ utterances from $212$ counseling conversations between
therapist and patient. The dataset is gathered from multiple counseling videos
on publicly available platforms like YouTube. It belongs to diverse
demographic groups with distinct mental health matters and several therapists,
helping researchers propose a generalized approach. Further, the authors
extracted transcriptions from the videos and pre-processed them. Collected
dialogues are dyadic, in which patients and therapists are the only
interlocutors. Since our task aims for counseling summarization built upon
domain knowledge, we extend the HOPE dataset to annotate psychotherapy
elements and counseling summary. We consulted with a team of the leading
mental health experts and proposed a hierarchical labeling structure for each
utterance. We discuss the details related to annotation in Section 3.1. The
same experts helped us develop annotation guidelines and validate the
annotations. We now discuss the extended dataset, which we call MEMO.

Next, the task is to add psychotherapy elements to these dyadic conversations.
We observe that the conversations can contain three essential counseling
components, namely symptom and history, patient discovery, and reflecting
utterances as shown in Figure 2. The rest of the conversation between
therapist and patient falls under the category of discussion filler.
Furthermore, to ensure the reference summaries represent rich knowledge
considering the clinical and linguistic perspectives and contained
psychotherapy elements, we follow the annotation guidelines designed by the
team of mental health experts.

###  3.1. Annotation Process

Counseling conversations are typically challenging as the patients are
reluctant to express themselves in front of therapists. As a consequence,
therapist articulately involves the patient into discussion. In support of
this, we meticulously analyze counseling conversations to symptom and history
of the mental health problem, the discovery of the patient’s behavior, and
insights into the past story reflecting on the patient’s current situation.
The rest of the counseling conversation is generally enclosed with discussion
filler phrases. To better understand the counseling and support summarization
task, focusing only on a subset of knowledge-enhanced utterances, we annotate
counseling components and discussion filler for all utterances. With the help
of a comprehensive annotation process in place, the techniques used by
therapists are highlighted with each utterance. Therapists can be
significantly helped with AI based tools adhering to such domain-specific
annotation guidelines333https://familytherapybasics.com/blog/therapy-case-
summary. With the help of guidelines, the annotations keep track of the
therapy techniques; therefore, the summaries generated are succinct in their
information. We discuss more about annotation process of psychotherapy
elements in Appendix A.

Figure 2. Hierarchical labeling for each utterance. Counseling components
including _SH, PD_ and RT essentially contribute in summary generation.

Psychotherapy elements: Mental health therapy sessions are composed of
counseling components that effectively contribute towards successful
interventions and discussion filler which barely add any relevance to the
summary generation. We labeled utterances with four fine-grained labels:

  * •

Symptom and History (SH): It captures the utterances that comprise the most
insightful information for the therapist to assess the patient’s situation. A
few examples are mentioned below:

Speaker |  Examples
---|---
Patient |  _“Ever since I moved into the apartment I have been feeling anxious”_
Patient |  _’This has been the first time I have experienced mood swings”_
Patient |  _“I would be so stressed afterward. And worried I would make another mistake”_

  * •

Patient Discovery (PD): The patients coming in for a counseling session arrive
with complex thoughts. The therapist tries to build therapeutic relationships
to calm patients to unravel their thoughts. Examples are as follows:

Speaker |  Examples
---|---
Therapist |  _“Don’t worry if it lasts for several days”_
Therapist |  _“I believe sleep is not an issue for you anymore now that you are getting eight hours of sleep everyday”_
Therapist |  _“Any drug or alcohol use?”_ , _“Has it ever happened to you?”_

  * •

Reflecting (RT): Therapist utterances are concise most of the time, which is
to ensure there is sufficient space for the patient to express themselves.
Patients are encouraged to share their stories and events of their lives. On
the other hand, the therapist understands the patient by giving them an
imaginary situation to assess their actions. Some utterances are shown below:

Speaker |  Examples
---|---
Therapist |  _“It’s been bothering you for a few weeks, something new, something a little frightening for you …”_
Patient |  _“By the time it it’s my turn to present I start. I just, I’m almost like just …”_

  * •

Discussion Filler (DF): When a therapist and a patient engage in conversation,
they articulate some utterances that are peripheral to the session. In this
work, we tag them as discussion filler. These include pleasantries (‘Good
morning!’), non-lexical fillers (‘Ummm’), acknowledgments (‘Right’), and
restatement of affirmations in subsequent utterances (‘Yeah. Yeah’), etc.
Discussion Filler utterances carry little to no relevance in the summary
generation.

###  3.2. Data Analysis

Table 1 shows data analysis on MEMO. The dataset consists of an almost equal
number of patient and therapist utterances. Therapist utterances are tagged
more with discussion filler labels showing that therapists converse to put the
patients at ease and agree with them. PD is the most prominent label in the
dataset. Patients tend to talk about several auxiliary topics, not their
mental health issue, when asked to share their experiences, whereas RT is the
least tagged label.

###  3.3. Ethical Considerations

Three domain experts reviewed the annotations at mpathic.ai, a seed-stage
corporation specializing in conversation analysis in commercial applications,
including evaluation of psychotherapy, coaching, and customer services. All
experts are independently licensed in clinical practice and have backgrounds
in automating fidelity of therapy at scale. They included a board-certified,
licensed psychologist (SPL) with over 15 years of experience in applying
machine learning to psychotherapy evaluation, a licensed clinical social
worker specializing in the evaluation of human labeling for machine learning,
and doctoral-level counselor, director of Clinical AI at mpathic.ai.

Following Sekhon et al.’s framework (Sekhon et al., 2017), they reviewed the
summaries for clinical acceptability in the areas of affective attitude,
burden (i.e., the cognitive load of interpreting the summaries), ethicality,
coherence (i.e., how well the summaries were understood), opportunity costs
(i.e., pros and cons of utilizing the summaries), and perceived effectiveness
(i.e., how well they might perform in a clinical setting). The acceptability
parameters were independently evaluated and given a ranking of 0-2 on each
parameter with 0 being not acceptable, 1, being acceptable with modification,
and 2 being acceptable for commercialization. After independent ranking,
discrepancies were discussed, and an average gold-standard acceptability the
ranking was determined for each summary. Additionally, the experts provided
feedback for commercialization considerations in applications like electronic
health care records and quality assurance.

##  4\. Proposed Methodology

In this section, we describe our proposed method, called ConSum. It encodes
dialogue’s utterances and uses three modules in the pipeline to extract
complementary information. These three modules operate independently on the
selection and filtering of utterances. We use mental health’s Patient Health
Questionnaire (PHQ-9) (Kroenke et al., 2001) knowledge similarity (MH-Know)
and discussion filler classification to filter utterances from each counseling
dialogue. Further, we classify utterances from dialogue for counseling
component labels and use these label information to generate abstractive
summaries. Figure 3 shows a high-level architecture of ConSum.

Consider a dialogue containing $n$ utterances, ${D=\langle u_{1},u_{2},\dots
u_{n}\rangle}$. We fine-tune DistilBERT (Sanh et al., 2020) embeddings to
create utterance representation of $d$ dimension. With knowledge-infused
utterance selection (MH-Know) module and discussion filler classifier, we
generate binary mask arrays from each module, which is fused to filter the
utterances. On the other hand, the classification from counseling components
classifier augments each utterance with mental health aspects. We describe
each module below.

Table 1. Distribution of psychotherapy elements in MEMO dataset. Here, #D:
Number of dialogues; #U: Number of utterances; U/D: Utterances per dialogue;
Sp: speaker; Th: Therapist; and Pt:. Patient. The train, test, and validation
splits are 70:20:10.

|  |  |  |  | Counseling Components |
---|---|---|---|---|---|---
|  |  | Discussion Filler | SH | RT | PD |
Split | #D | Sp. | U/D | #U | U/D | #U | U/D | #U | U/D | #U | Total
Train | 152 | Pt | $5.02$ | $764$ | $5.67$ | $862$ | $2.52$ | $383$ | $18.84$ | $2863$ | $4766$
Th | $8.10$ | $1232$ | $8.17$ | $1243$ | $4.22$ | $642$ | $10.80$ | $1643$ | $4877$
1-12 Test | 39 | Pt | $3.43$ | $134$ | $2.95$ | $115$ | $1.00$ | $39$ | $18.38$ | $717$ | $1004$
Th | $5.46$ | $213$ | $4.51$ | $176$ | $4.43$ | $173$ | $11.28$ | $440$ | $1006$
1-12 Val | 21 | Pt | $8.09$ | $170$ | $4.23$ | $89$ | $2.28$ | $48$ | $13.80$ | $290$ | $594$
Th | $10.48$ | $220$ | $5.57$ | $117$ | $4.66$ | $98$ | $7.38$ | $155$ | $597$
Total | 212 | Pt | $5.51$ | $1068$ | $4.28$ | $1066$ | $1.60$ | $470$ | $17.00$ | $3870$ | $6364$
Th | $8.01$ | $1665$ | $6.08$ | $1536$ | $4.44$ | $913$ | $9.82$ | $2238$ | $6480$

  * •

Discussion Filler Classifier (DFC): Discussion filler classification is a
binary classification task. We use a feed-forward network with $2$ hidden
layers to classify each utterance $u_{i}$ into ‘counseling’ or ‘discussion
filler’ from the obtained input representations. It compresses representations
to $100$ dimensional hidden representation and then to $2$ at the output
layer. A dropout of $30\%$ is applied between the linear layers to regularize
the model’s performance. The model is trained with adam optimizer to minimize
the cross-entropy loss. The output is a mask array labeled $\tau$, where
$\tau_{i}$ represents mask array from dialogue $D_{i}$. Using independent
networks to classify the discussion filler label and the counseling components
helps smooth errors better than having a single unified classifier.

  * •

Mental Health Knowledge Infused Utterance Selection (MH-Know): Earlier efforts
in mental health counseling summarization paid less attention to the domain
knowledge. Patient Health Questionnaire (PHQ-9) represents an established set
of questionings to assess the patient’s mental health condition (Kroenke et
al., 2001). We use PHQ-9 lexicons (Yazdavar et al., 2017) to compute
similarity between nine questions and input utterance. The intuition is to
obtain the most relevant utterances corresponding to knowledge infused
questions. We use BERTScore (Zhang et al., 2020a) to compute the similarity
between the utterances $u_{i}$ and the PHQ-9 lexicons $PHQ=\langle
phq_{1},phq_{2},\dots phq_{9}\rangle$. The nine similarity scores, $s=\langle
s_{1},s_{2},\dots s_{9}\rangle$ corresponding to each utterance ranges between
$0$ to $1$ each. The summation of similarity score denoted by $\psi$ is
compared with the hyperparameter tuned value of threshold denoted by $\phi$.

$s_{i}=bertscore(u_{i},phq_{i})\Rightarrow\psi_{i}=\sum_{m=1}^{m=9}s_{m}$

The value of $\phi$ is experimentally chosen to retain maximum domain
knowledge and intervention components without compromising other pipelines’
impact. This creates a mask-array,
$\sigma_{i}=\langle\sigma_{1},\sigma_{2}\dots\sigma_{n}\rangle$ containing $1$
for cases where intervention similarity score is less than the threshold and
$0$ otherwise. Here, $i$ in $\sigma_{i}$ represents $i\textsuperscript{th}$
utterance.

$\sigma_{i}=\begin{cases}1,&\text{if $\psi_{i}\leq\phi$}.\\\
0,&\text{otherwise}\end{cases}$

ConSum prioritizes the utterances fetched from the MH-Know module along with
corresponding counseling tags to generate summary. During experimentation, we
use the threshold value of $6$ to extract utterances relevant to the task.

Figure 3.  Model diagram of ConSum. The Utterance embeddings are reused across
three modules in the pipeline.

  * •

Counseling Components Classifier (CCC):  As discussed earlier, the four
components are SH, RT, PD, and DF (c.f. Section 3.1). Considering SH, RT and
PD as the key components contributing to the summary generation, we frame the
problem as a four-class classification task. Hence, for each utterance
representation of dimension $d$, we first fetch the ‘counseling context’
across utterances. As the counseling progresses, we maintain the context of
the dialogue through a GRU layer on top of the utterance representations.
Further, we apply multi-head self-attention following Vaswani et al. (2017).
Representations fused with attention weights are passed through two dense
layers to learn hidden representations from $100$ dimensional and then further
to $3$. We apply a softmax classifier to predict probabilities corresponding
to each class label.

Identification of discussion filler is a domain-independent binary
classification task. Therefore, rather than following an end-to-end model for
psychotherapy element classification, we opt for a hierarchical model to
predict discussion filler and counseling component. In our case, the standard
model performs well for DFC. However, we sense the requirement of a model that
can learn contextual counseling labels in the summary generation process. The
remaining section shows the direct use of counseling labels in the summary
generation module.

  * •

Summary decoder: We obtain two mask-arrays from the MH-Know module and DFC.
Discussion Filler mask $\tau$ and knowledge-infused mask $\sigma$ are merged
as follows. $F$ denotes the resulting mask array. Further, utterances $U_{i}$
from the dialogue $D_{i}$ are concatenated with predicted counseling
components (c.f. Section • ‣ 4) and filtered using the final mask array $F$.
The final subset of utterances with domain knowledge and counseling components
($CL_{i}$) are concatenated, resulting in $G$. It is used as an input to fine-
tune the pre-trained T5 model (Raﬀel et al., 2020) for abstractive summary
generation.

$F=(\sigma\cap\tau)^{\prime};G=(U_{i}\oplus CL_{i})\otimes F$

##  5\. Experiments and Results

This section reports our experimental results, comparative study, and other
analyses. To evaluate the summaries, we use ROUGE (Lin, 2004) metrics, namely
Rouge-1 (R-1), Rouge-2 (R-2), and Rouge-L (R-L). In addition, we show
QuestEval (QAE) score and Bleurt Score (BS) to capture the contextual
advantage in abstractive summarization. QuestEval (Scialom et al., 2021)
comprehensively judges the reference and predictions on four aspects –
consistency, coherence, fluency, and relevance by generating question-answer
pairs on the given source document, reference, and generated summaries. Bleurt
(Sellam et al., 2020) utilizes pre-training on the BERT model using
unsupervised techniques with millions of synthetic examples to generalize all
possible data distributions. At last, we show human and clinical evaluations
to validate the quality of generated summaries. We show hyper-parameters and
other experiment details Appendix B.

Table 2. Results obtained on MEMO. We report Rouge-1 (R-1), Rouge-2 (R-2), Rouge-L (R-L), Bleurt Score (BS) and QuestEval Score (QAE). Model | R-1 | R-2 | R-L | QAE | BS
---|---|---|---|---|---
PLM | 34.24 | 11.19 | 33.35 | 24.34 | -0.8678
RankAE | 25.57 | 3.43 | 24.16 | 29.98 | -1.063
SM | 20.46 | 3.80 | 18.87 | 20.22 | -0.9454
Pegasus | 29.71 | 7.77 | 27.57 | 36.80 | -0.6130
T5 | 31.44 | 5.63 | 27.38 | 33.55 | -0.5655
1-6 ConSum | 45.36 | 15.71 | 24.75 | 25.42 | 0.3407

###  5.1. Baselines

We choose following systems as our baselines - Pretrained Language Model
(PLM): Feng et al. (2021) used DialoGPT to segment topics and then generated
summaries with BART. RankAE: Zou et al. (2021) used BERT to group utterances
into segments and further generated summaries using denoising auto-encoder
(Vincent et al., 2008). Segmented Modeling (SM): Goo and Chen (2018) leveraged
dialogue act information on standard BiLSTM (Nallapati et al., 2016)
architecture. Transfer Learning with unified Text-to-Text Transformer (T5):
Raﬀel et al. (2020) used a shared framework on transformer architecture to
pre-train on huge custom crawled corpus (C4). Pegasus: Zhang et al. (2020b)
introduced a novel pre-training objective of using gap-sentences-generation
and then generated summaries.

###  5.2. Results and Ablation Study

The experimental results on MEMO are shown in Table 2. ConSum performs better
than all other baselines with the highest R-1 score of $45.36$ and R-2 score
of $15.71$ – significant improvements of $11.12$ and $4.52$ points over the
best baseline, respectively. The PLM model turns out to be the second ranked
model in both cases with an R-1 score of $34.24$ and an R-2 score of $11.19$.
It shows the importance of topic segmentation in a conversational dataset,
which aids in the summarization outputs. In contrast, PLM scores the highest
R-L score of $33.35$; however, its summaries have a higher structural
similarity instead of contextual similarity between its output and the
reference summaries. On the other hand, ConSum produces a dominant performance
on Bleurt score with a difference of $0.90$ points over the best baseline
($-0.5655$ of T5 vs $0.3407$ of ConSum). This score indicates contextually
superior performance concerning the reference summary.

Table 3. Ablation study on the effect of different modules of ConSum. SH, PD,
DF and RT are psychotherapy elements. MH-Know and CCC are knowledge infused
module and counseling component classifier module respectively. We present our
analysis on five scores to understand all aspects of summary generation.

Counselling Label | R-1 | R-2 | R-L | QAE | BS
---|---|---|---|---|---
ConSum $-$ SH $-$ PD | $20.92$ | $5.00$ | $7.44$ | $20.71$ | $0.0019$
ConSum $-$ PD $-$ RT | $36.00$ | $9.00$ | $9.14$ | $20.47$ | $0.2032$
ConSum $-$ RT $-$ SH | $28.63$ | $8.06$ | $9.55$ | $23.02$ | $-0.0209$
ConSum $-$ SH | $39.77$ | $9.55$ | $8.98$ | $24.11$ | $0.1908$
ConSum $-$ PD | $36.87$ | $10.02$ | $11.22$ | 33.38 | $0.2420$
ConSum $-$ RT | $42.01$ | $9.83$ | $16.50$ | $18.03$ | $0.2060$
ConSum $-$ MH-Know $-$ CCC | $39.67$ | $9.95$ | $12.69$ | $21.19$ | $0.2003$
ConSum $-$ MH-Know | $40.42$ | $10.09$ | $11.00$ | $23.97$ | $0.2429$
1-6 ConSum | 45.36 | 15.71 | 24.75 | 25.42 | 0.3407

Table 4. Comparative analysis on generated summaries between T5 and ConSum
model. Green denotes primary medical condition information, Cyan denotes SH
information, Brown denotes RT information and Red denotes errors/irrelevance
in generated summaries. Comparison shows ConSum captures better contextual
information compared to T5.

Model |  Summaries
---|---
T5 |  Patient is asked a series of questions to test their housing. Patients do not want to take drugs, police arrest them and others in their area. They secretly talk about the incident. The patient lives with friends from different places including the city, where they are arrested or taken into custody. The patients feel they have no medical condition such as diabetes or stroke. These occured recently. The patient has had problems with housing due augensis. Patients feel they have been victimized by drug abusers since they were 12 years old. They know people who don’t use marijuana to steal
1-2 ConSum |  Whenever there is a basketball or or football, the patient does not really hear what to take. Patient feels overwhelming and feels like having this issue since childhood. Therapist made him realise his problems and patient was curing gradually. Therapist suggested him use. The patient was raised Catholic and they feel guilty about spending more money on drugs and parties. The patient has switched 12 houses in the past year and feel it must have affected their kids. The patient occasionally borrows money and gets less sleep in general
T5 |  The patient is tired when they have to go to work and it’s a bit frustrating for them. They feel tired throughout the day without any food, no panic attacks, no medical condition such as diabetes or stroke. Patients are in a position where they can focus on anything. They do not want pills to reduce stress hence their life is limited by diet. The patient has lost three pounds in recent weeks due to this fatigue
1-2 ConSum |  Whenever the patient goes to work. The patient is worried that they might have ADHD. The patient does not suffer from depression, anxiety nor use drugs or call it a metaphysical stuff. The patient wishes to get better and needs something to hold. The patient feel they sway at things, and they have two options. The patient was sent in by a counselor fearing they might hurt themselves. The patient’s dad had committed suicide 15 years ago and their sister had attempted once. The patient has been diagnosed with depression and anxiety. The patient lives alone

We also show the effect of counseling components and different modules in the
pipeline and analyze the results. We demonstrate the importance of our domain-
specific modules by evaluating various combinations of counseling components.
We show our ablation study in Table 3. We observe that masking SH and PD
utterances have a drastic effect on R-1 score as it reports a reduction of
over 25 points. Similarly, we observe reductions in other metric as well – 10
points in R-2, 17 points in R-L, 5 points in QAE, and 33 points in BS.
Moreover, we observe similar phenomena in masking any two components (PD and
RT; SH and RT) as well. We also perform experiments with masking one component
at a time and observe inferior results on average. Overall, we observe that
removing the information of any counseling components degrades the performance
of ConSum. Moreover, the importance of CCC module in capturing mental health
information is presented in Section 6. Further, we perform ablation with the
MH-Know module and observe similar phenomenon with the omission of the MH-Know
module as well.

###  5.3. Error Analysis

Table 4 shows example summaries generated by T5 as well as ConSum. Sticking to
our primary aim of covering mental health-related aspects in summaries, we
observe that T5 generates rich semantics; however, lacks in attending critical
domain-related phrases in many cases. On the other hand, ConSum encapsulates
more domain knowledge and captures critical information inherited from three
counseling components. For instance, in the first example in Table 4, T5
covers only the high-level gist of the conversation, viz. ‘They secretly talk
about incident.’, whereas ConSum covers essential phrases keeping the
semantics intact viz. ‘Patient feels overwhelming and feels like having this
issue since childhood’. Moreover, we observe that T5 also commits a few
mistakes and generates irrelevant text in both examples (as highlighted in
red).

###  5.4. Clinical and Human Evaluation

A team of mental health experts use clinical acceptability framework (Sekhon
et al., 2017) to evaluate generated summaries. The clinical acceptability
framework involves six parameters – affective attitude, burden, ethicality,
coherence, opportunity costs, and perceived effectiveness (c.f. Appendix C for
more details). Experts rate each summary considering the acceptability
parameters on a continuous scale of $0$ to $2$, where higher rating signifies
better acceptability. Through expert evaluation, we obtain an average rating
of $0.94$ out of $2.00$ – which falls under the standard acceptability range
of $0.70$ to $1.40$ by mental health experts (Sekhon et al., 2017). Hence, the
domain experts conclude the summaries to be acceptable for applications in
therapy settings. However, in some cases, they pointed out issues with ConSum
in comprehending metaphors and subtle transcription errors, which affected the
semantic. Tables 6 and 7 in Appendix show the acceptability ratings and
definition of acceptability parameters, respectively. Overall, mpathic.ai
recommend deploying ConSum with their customers and a commercial pilot with a
health-coaching therapy company.

Furthermore, results of human evaluation on four standard linguistic
parameters namely, relevance (selection of relevant content), consistency
(factual alignment between the summary and the source), fluency (linguistic
quality of each sentence), and coherence (structure and organization of
summary) are shown in Table 5. Each parameter is rated on the scale of $1$ to
$5$ and found to be qualitatively better than the best baselines.

Table 5. Human evaluation on the summaries generated from ConSum model. Model | Relevance | Consistency | Fluency | Coherence
---|---|---|---|---
RankAE | 2.80 | 2.91 | 3.02 | 2.98
T5 | 2.99 | 3.05 | 3.04 | 2.95
1-5 ConSum | 3.37 | 3.22 | 3.11 | 3.13

##  6\. Mental Health Information Capture (MHIC) metric

Our primary aim is to capture most critical components of mental health
counseling leading to successful interventions. Since most of the deep
learning methodologies are able to generate contextually and semantically rich
text, we observe their incompetence in attaining essential phrases relevant to
mental health aspects such as symptom, history of the illness, or other
reflecting aspects. Consequently, it becomes extremely challenging to
qualitatively assess the summaries and discriminate between the state-of-the-
art and domain-specific models. Hence, we propose a new problem-specific
metric, MHIC, to mitigate the issue of disregarding such aspects. Mental
Health Information Capture (MHIC) attains to the domain-specific counseling
components in generated summaries. It incorporates the prediction of
counseling components classifier and Rouge 1 score (R-1) to evaluate the
generated summaries qualitatively. We calculate MHIC for each counseling
component $CL_{i}\in\\{SH,PD,RT\\}$ as follows:

$MHIC(CL_{i})={R1\\_Score}(S_{g},H_{i})$

where $H_{i}$ represents concatenation of all utterances predicted with
$CL_{i}$ and $S_{g}$ denotes the generated summary. Figure 4 shows the
performance of our model and the best baseline, T5. We also show plots with
the inclusion and exclusion of counseling labels for each component. We
observe that ConSum’s summaries are more aligned towards the three counseling
components. Moreover, we observe that utterances tagged with the SH label have
significant representation in generated summaries.

Figure 4. Comparison between ConSum and T5 summaries and condensed dialogue
utterances based on counseling components. The y-axis represents Rouge-1 F1
scores. $D^{\prime}$ denotes concatenation of utterances tagged with
particular counseling component, and $D$ \- $D^{\prime}$ denotes concatenation
of utterances excluding corresponding counseling component.

##  7\. Uniqueness of Mental-health counseling and ConSum

Though dialogue summarization in clinical domain (Song et al., 2020; Joshi et
al., 2020; Molenaar et al., 2020) is a well-explored research area, it remains
unexplored in the mental health counseling domain. Considering this, in this
paper, we raise the research question as “Why mental health counseling
summarization is different than other medical conversation summarization?”. To
answer this question, first we draw significant differences between the two
domains and subsequently, show that our proposed model, ConSum, is
specifically designed to handle crucial counseling components of the mental
health conversations. Clinical practitioners directly aim for the core problem
and the conversation about disease and symptoms are up-front – usually
patients do not shy away discussing their concerns. On the other hand,
patients suffering from mental health issues are very much reluctant to speak
out their issues in a blunt fashion.

In our proposed model, we leverage these insights and specifically design
various domain-specific sub-modules (DFC, MH-Know, and CCC) to account for
such attributes. As a result, ConSum is unique to the mental-health counseling
domain instead of other domains.

To demonstrate this, we exploit an existing summarization dataset for clinical
conversations in Chinese (ChiCCo) (Song et al., 2020) and extend it for our
purpose. Following Song et al. (2020), we translate the dataset into English
using Google Translate. Since ConSum requires counseling psychotherapy
elements to operate, we obtain these from the pretrained counseling component
and discussion filler classifiers. Subsequently, we train ConSum and other
comparative systems on the ChiCCo dataset.

We show detailed results of the comparative study in Appendix Section D. In
comparison with the encouraging performance of ConSum on MEMO (c.f. Table 2),
we argue that ConSum is competent in handling crucial components of the mental
health counseling pretty efficiently, as in the absence of such components in
other domains, it underperforms.

##  8\. Commercialization of ConSum

In this section, we discuss deployment aspects of ConSum as a commercial tool
for mental health experts. We lay down two key challenges and resolutions in
this respect.

Challenges:

  * •

Ethical concern: The first critical challenge in the commercialization of
ConSum is the usage of publicly available data for training – we extend HOPE
to develop MEMO. It would be unethical to use it for commercialization.

  * •

Data sparsity: Even though ConSum performs well on MEMO covering many aspects
of mental health components, our experts suggest to include diverse and
sophisticated data in the training process, prior to commercialization.

Resolutions: We resolve these challenges in the following way and hope to
deploy ConSum as a real-world tool soon.

  * •

We recently signed an MoU with mpathic.ai to commercially utilize a large-
scale conversational dataset to extend the training of ConSum.

  * •

We conduct the experiments and evaluate outputs in two different research labs
and an industrial lab at mpathic.ai to cross-check the consistency and quality
of ConSum.

  * •

Further, a team of mental health experts from the University of Washington,
Wake Forest University, and mpathic.ai validate the feasibility of ConSum as a
commercial application in the real world.

##  9\. Conclusion

Considering the situation of mental health therapies amid the COVID-19
pandemic, in this paper, we attempted to empower therapists through our
research in mental health counseling summarization. To this end, we developed
a novel summarization dataset, MEMO, in a dyadic counseling setup. Moreover,
to model the counseling summarization in an abstractive fashion, we proposed a
domain-enrich transformer architecture, ConSum. It utilized three domain-
centric sub-modules viz. a discussion filler classifier, a counseling
component classifier, and a PHQ-9 driven mental health knowledge-infused
utterance selection module. Our extensive ablation study and the comparative
analysis established the superiority of ConSum over several existing models.
Furthermore, ConSum is being considered for a large-scale deployment by
mpathic.ai.

###### Acknowledgements.

The authors acknowledge the support of the Ramanujan Fellowship, ihub-
Anubhuti-iiitd Foundation set up under the NM-ICPS scheme of the DST, and CAI
at IIIT-Delhi. We thank Taichi Shinohara, LICSW, mpathic.ai and Amber Jolley-
Paige, PhD, LCPC, mpathic.ai - Wake Forest University - Department of
Counseling for providing additional clinical validation.

## References

  * (1)
  * Afzal et al. (2020) Muhammad Afzal, Fakhare Alam, Khalid Mahmood Malik, and Ghaus M Malik. 2020.  Clinical Context–Aware Biomedical Text Summarization Using Deep Neural Network: Model Development and Validation.  22, 10 (2020).
  * Atri et al. (2021) Yash Kumar Atri, Shraman Pramanick, Vikram Goyal, and Tanmoy Chakraborty. 2021.  See, hear, read: Leveraging multimodality with guided attention for abstractive text summarization.  _Knowledge-Based Systems_ 227 (2021), 107152.  https://doi.org/10.1016/j.knosys.2021.107152
  * Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003\.  Latent Dirichlet Allocation.  3 (2003).  Issue Jan.
  * Chen and Verma (2006) Ping Chen and R. Verma. 2006.  A Query-Based Medical Information Summarization System Using Ontology Knowledge. In _19th IEEE Symposium on Computer-Based Medical Systems (CBMS’06)_.  ISSN: 1063-7125.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting.  _arXiv:1805.11080 [cs]_ (May 2018).
  * Chen et al. (2011) Yun-Nung Chen, Yu Huang, Ching-feng Yeh, and Lin-Shan Lee. 2011\.  Spoken Lecture Summarization by Random Walk over a Graph Constructed with Automatically Extracted Key Terms. 933–936.
  * Chen et al. (2020) Yen-Pin Chen, Yi-Ying Chen, Jr-Jiun Lin, Chien-Hua Huang, and Feipei Lai. 2020.  Modified Bidirectional Encoder Representations From Transformers Extractive Summarization Model for Hospital Information Systems Based on Character-Level Tokens (AlphaBERT): Development and Performance Evaluation.  _JMIR Medical Informatics_ 8, 4 (April 2020), e17787.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  (2019).  arXiv:1810.04805
  * Dey et al. (2020) Alvin Dey, Tanya Chowdhury, Yash Kumar, and Tanmoy Chakraborty. 2020.  Corpora Evaluation and System Bias Detection in Multi-document Summarization. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 2830–2840.  https://doi.org/10.18653/v1/2020.findings-emnlp.254
  * Feng et al. (2021) Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021.  Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization. In _Proceedings of the 59th ACL and the 11th IJCNLP_. Online.
  * Gaur et al. (2021) Manas Gaur, Vamsi Aribandi, Ugur Kursuncu, Amanuel Alambo, Valerie L Shalin, Krishnaprasad Thirunarayan, Jonathan Beich, Meera Narasimhan, and Amit Sheth. 2021\.  Knowledge-Infused Abstractive Summarization of Clinical Diagnostic Interviews: Framework Development Study.  _JMIR_ 8, 5 (10 May 2021).
  * Goo and Chen (2018) Chih-Wen Goo and Yun-Nung Chen. 2018.  Abstractive Dialogue Summarization with Sentence-Gated Modeling Optimized by Dialogue Acts.  (2018).  arXiv:1809.05715
  * Ive et al. (2020) Julia Ive, Natalia Viani, Joyce Kam, Lucia Yin, Somain Verma, Stephen Puntis, Rudolf N. Cardinal, Angus Roberts, Robert Stewart, and Sumithra Velupillai. 2020.  Generation and evaluation of artificial mental health records for Natural Language Processing.  3 (2020).
  * Joshi et al. (2020) Anirudh Joshi, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2020.  Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures.. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online.
  * Kennedy et al. (2016) Sidney H. Kennedy, Raymond W. Lam, Roger S. McIntyre, S. Valérie Tourjman, Venkat Bhat, Pierre Blier, Mehrul Hasnain, Fabrice Jollant, Anthony J. Levitt, Glenda M. MacQueen, Shane J. McInerney, Diane McIntosh, Roumen V. Milev, Daniel J. Müller, Sagar V. Parikh, Norma L. Pearson, Arun V. Ravindran, and Rudolf Uher. 2016.  Canadian Network for Mood and Anxiety Treatments (CANMAT) 2016 Clinical Guidelines for the Management of Adults with Major Depressive Disorder.  _Canadian Journal of Psychiatry_ 61, 9 (Sept. 2016).
  * Konovalov et al. (2010) Sergiy Konovalov, Matthew Scotch, Lori Post, and Cynthia Brandt. 2010.  Biomedical informatics techniques for processing and analyzing web blogs of military service members.  _Journal of Medical Internet Research_ 12, 4 (Oct. 2010).
  * Krishna et al. (2021) Kundan Krishna, Sopan Khosla, Jeffrey Bigham, and Zachary C. Lipton. 2021.  Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques. In _Proceedings of the 59th ACL and the 11th IJCNLP_.
  * Kroenke et al. (2001) Kurt Kroenke, Robert L Spitzer, and Janet B W Williams. 2001\.  The PHQ-9.  _Journal of General Internal Medicine_ 16, 9 (Sept. 2001).
  * Lin (2004) Chin-Yew Lin. 2004\.  ROUGE: A Package for Automatic Evaluation of Summaries.  (2004).
  * Malhotra et al. (2021) Ganeshan Malhotra, Abdul Waheed, Aseem Srivastava, Md Shad Akhtar, and Tanmoy Chakraborty. 2021\.  Speaker and Time-aware Joint Contextual Learning for Dialogue-act Classification in Counselling Conversations. In _15th ACM International WSDM Conference (WSDM-22)_. ”Arizona, USA”.  arXiv:2111.06647
  * Molenaar et al. (2020) Sabine Molenaar, Lientje Maas, Verónica Burriel, Fabiano Dalpiaz, and Sjaak Brinkkemper. 2020\.  _Medical Dialogue Summarization for Automated Reporting in Healthcare_.  https://doi.org/10.1007/978-3-030-49165-9_7
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulcehre, and Bing Xiang. 2016\.  Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. In _The 20th CoNLL_ (Berlin, Germany, 2016-08).
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018\.  Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ (Brussels, Belgium, 2018-10). ACL.
  * Quiroz et al. (2020) Juan C Quiroz, Liliana Laranjo, Ahmet Baki Kocaballi, Agustina Briatore, Shlomo Berkovsky, Dana Rezazadegan, and Enrico Coiera. 2020\.  Identifying relevant information in medical conversations to summarize a clinician-patient encounter.  _Health Informatics Journal_ 26, 4 (2020).  PMID: 32865113.
  * Raﬀel et al. (2020) Colin Raﬀel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the Limits of Transfer Learning with a Uniﬁed Text-to-Text Transformer.  (2020), 67.
  * Saggion and Poibeau (2012) Horacio Saggion and Thierry Poibeau. 2012.  Automatic Text Summarization: Past, Present and Future.  In _Multi-source, Multilingual Information Extraction and Summarization_ , R. Yangarber T. Poibeau; H. Saggion. J. Piskorski (Ed.). Springer.
  * Sanh et al. (2020) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv:1910.01108 [cs]_ (Feb. 2020).  arXiv: 1910.01108.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021\.  QuestEval: Summarization Asks for Fact-based Evaluation.  _arXiv:2103.12693 [cs]_ (April 2021).
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017\.  Get To The Point: Summarization with Pointer-Generator Networks. In _Proceedings of the 55th ACL_.
  * Sekhon et al. (2017) Mandeep Sekhon, Martin Cartwright, and Jill J. Francis. 2017\.  Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework.  _BMC Health Services Research_ 17, 1 (26 Jan 2017).
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning Robust Metrics for Text Generation. In _Proceedings of the 58th ACL_. Online.
  * Song et al. (2020) Yan Song, Yuanhe Tian, Nan Wang, and Fei Xia. 2020\.  Summarizing Medical Conversations via Identifying Important Utterances. In _Proceedings of the 28th COLING_ (Barcelona, Spain (Online), 2020).
  * Strauss et al. (2013) John Strauss, Arturo Martinez Peguero, and Graeme Hirst. 2013.  Machine learning methods for clinical forms analysis in mental health.  _Studies in Health Technology and Informatics_ 192 (2013), 1024\.
  * Tran and Kavuluru (2017) Tung Tran and Ramakanth Kavuluru. 2017.  Predicting mental conditions based on “history of present illness” in psychiatric notes with deep neural networks.  75 (2017).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention Is All You Need.  _arXiv:1706.03762 [cs]_ (Dec. 2017).  arXiv: 1706.03762.
  * Vincent et al. (2008) Pascal Vincent, Hugo Larochelle, Y. Bengio, and Pierre-Antoine Manzagol. 2008.  Extracting and composing robust features with denoising autoencoders.
  * Yazdavar et al. (2017) Amir Hossein Yazdavar, Hussein S Al-Olimat, Monireh Ebrahimi, Goonmeet Bajaj, Tanvi Banerjee, Krishnaprasad Thirunarayan, Jyotishman Pathak, and Amit Sheth. 2017\.  Semi-Supervised Approach to Monitoring Clinical Depressive Symptoms in Social Media.  _IEEE/ACM International Conference on Advances in Social Network Analysis and Mining_ 2017 (2017).
  * Zafari and Zulkernine (2021) Hasan Zafari and Farhana Zulkernine. 2021.  Chatsum: An Intelligent Medical Chat Summarization Tool. In _2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI)_.
  * Zhang et al. (2020b) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020b.  PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization.  _arXiv:1912.08777_ (July 2020).
  * Zhang et al. (2021) Longxiang Zhang, Renato Negrinho, Arindam Ghosh, Vasudevan Jagannathan, Hamid Reza Hassanzadeh, Thomas Schaaf, and Matthew R. Gormley. 2021\.  Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations. In _Findings of the ACL: EMNLP 2021_. ACL.
  * Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a.  BERTScore: Evaluating Text Generation with BERT.  arXiv:1904.09675 [cs.CL]
  * Zou et al. (2021) Yicheng Zou, Jun Lin, Lujun Zhao, Yangyang Kang, Zhuoren Jiang, Changlong Sun, Qi Zhang, Xuanjing Huang, and Xiaozhong Liu. 2021.  Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders.  _Proceedings of the AAAI Conference_ 35, 16 (May 2021).

Rater | Affective Attitude | Burden | Ethicality | Intervention Coherence | Opportunity Cost | Perceived Effectiveness
---|---|---|---|---|---|---
Rater-1 | 0.875 | 1 | 1 | 1.125 | 0.875 | 0.875
Rater-2 | 0.75 | 0.875 | 1 | 0.875 | 0.875 | 1
1-7 Average | 0.813 | 0.936 | 1 | 1 | 0.875 | 0.936
Table 6. Mental health experts rate the summaries generated by ConSum on
mental-health aspects.

##  Appendix A Annotation Process

Augmenting dialogue utterances with additional information can aid the model
to learn the representations better given the limited dataset. The therapists
exhibit the three most essential skills, which are - active listening,
observing non-verbal actions, and building rapport
444https://counsellingtutor.com/basic-counselling-skills/. Elaborating on the
techniques followed by therapists in any of their counseling sessions can be
categorized as follows: Attending is being in company of a person and giving
them your full attention, Silence is to let the patient take complete control
of the context, language and pace of their interaction, Paraphrasing lets the
patient know that the therapist is actively listening to them by presenting a
concise information of context discussed, Clarifying questions makes the
therapist understand the patient’s issue while avoiding any leading questions
which puts the patient at discomfort, Focusing enables the therapist to make
decisions with the patient about the issues the patient wants to focus on and
disregard other topics, Building Rapport connects the therapist with the
patient in an amicable manner, Summarizing is paraphrasing the context
discussed till then in a more extended fashion and Immediacy reveals how the
therapist feels towards the patient after listening to their events which is
regarded as the most necessary skill of a therapist. Finally, Homeworks are
given by therapists to patients to follow a particular routine until the next
follow-up session, which helps in the patient’s path to recovery.

To incorporate the information of the techniques discussed above in the
dialogue set, we introduce a set of 4 annotation labels considering the
structure of therapy counseling sessions carefully and with the help of
extensive discussion with leading mental health experts. The annotations are
arranged hierarchically in two levels. The first level is binary
classification among two labels, namely - Counseling components and Discussion
Filler, followed by a sub-classification on counseling components to the
following - Symptom and History (SH), Patient Discovery (PD), and Reflecting
(RT).

Construct |  Definition |  Application
---|---|---
Affective attitude |  How an individual feels about an intervention |  What are your perceptions of the intervention based upon your clinical knowledge
Burden |  Perceived amount of effort required to participate |  How much effort is required to understand the intervention, consider: spelling, grammar, overall interpretation
Ethicality |  Extent to which this is a good fit with your organizations’ value system |  How does this align with your respective code of ethics? Are there concerns?
Intervention Coherence |  Extent to which the intervention is understood |  How well the summaries are understood
Opportunity Cost |  The extent to which one would benefit from using this intervention |  Pros and cons of using this intervention in your respective setting
Perceived Effectiveness |  Extent to whcih this intervention will perform in the intended setting |  How well this will perform in your clinical setting
Table 7. Individual metrics as reported in Table 6 are explained with the
application setting.

##  Appendix B Implementation Details

We used Hugging Face555https://huggingface.com to use pre-trained transformers
and further fine-tune it for the downstream tasks. All the experiments are
carried out on V100 GPUs with GPU RAM of 32GB and 512GB RAM. We discuss the
implementation of models below.

###  B.1. Hyperparameters

T5: We use _t5-base_ offered by Simple
Tranformers666https://simpletransformers.ai/ library. For the hyperparameters,
namely max_length, repetition_penalty, num_beams, train_batch_size,
max_epochs, we set values of 150, 5.0, 8 and 200 respectively.
SM: We use the standard framework of this model, which consists of BiLSTM
layer for dialogue history encoder and two LSTM networks each for dialogue act
labeling and summary decoder. We set only one hidden layer with a size of 256
units for each LSTM network. We experiment using summary attention with a
train batch size of 16 and number of epochs as 100.
PLM: As followed by the standard pipeline of this model, we use GPT-2 byte
pair encoding (bpe) and binarizer provided by
Fairseq777https://github.com/pytorch/fairseq. We experiment with BART model
for the summarizer. The parameters, namely beam_size, min_len, max_len,
block_ngram, are set to 4, 10, 150 and 3, respectively. We train with a batch
size of 4 and number of epochs as 200.
RankAE:  We implement the standard BERT (Devlin et al., 2019) encoding for the
utterances, which includes the speaker information at the beginning of each
utterance. For the encoding, we set shard_size as 2000, min_tgt_tokens as 10
and max_tgt_tokens as 150. Following this, the hyperparameters for the main
model, namely max_pos, dec_dropout, dec_layers, dec_hidden_size, lr_bert, are
set to 512, 0.2, 6, 768 and $1e-3$, respectively. The decoder and encoder
follow the same parameters. The train batch size is 4, and number of epochs is
100.
Pegasus: From the standard set of pretrained models, we use _pegasus-xsum_.
The tokenizer is set to the use the _longest_ padding. For the model, we set
warmup_steps, weight_decay, max_length as 500, 0.01 and 150, respectively. The
train batch size is 4, and num_train_epochs is 200, respectively.

####  B.1.1. ConSum

We first generate embeddings from $distil-bert-uncased$. Further, we implement
the model in 3 sub-modular tracks namely, counseling component classifier,
discussion filler classifier and MH-Know. Let’s discuss the implementation
step by step:
Counseling component classifier The embeddings are passed through a linear
layer to bring down the embedding size from $768$ to $100$. We pass the new
input to GRU with size set to $100$ and hidden output size is also $100$.
Further linear layers bring down the embedding of size $100$ to $4$. We train
the sub-module for $20$ epochs with a learning rate of $0.001$ and batch size
$4$. We optimize the Adam optimizer using cross-entropy loss.
MH-Know The embeddings are used to calculate the bert score. We use the
$0.1.0$ version of bert score to calculate the similarity. The scores obtained
are ranged between $0$ to $1$. Here, a bert score of $1$ represents highly
similar and $0$ represents highly dissimilar. The summation of all bert score
calculated across nine set of lexicons is further used for masking array
formation. The mask array is a binary array equivalent to the batch size or
dialogue size.
DFC: The distil-bert embedding are fine-tuned for binary classification task.
On top of encoder, we use a linear layer to condense embedding space from 768
to 100 and further from $100$ to $2$ with a dropout value of $0.20$. We use
adam optimizer and cross-entropy loss to perform the training.

##  Appendix C Clinical Validation

We design an annotation scheme to create a novel dataset in the domain of
mental health counseling. The dataset belongs to a critical domain where
mental health expert’s validation is a must. A team of mental health experts
from the University of Washington, Wake Forest University, and mpathic.ai
validated the feasibility of ConSum as an application in the real world. We
conducted clinical evaluations from two different experts for clinical
acceptability measures. All experts are independently licensed in clinical
practice and have backgrounds in automating fidelity of therapy at scale. They
included a board-certified, licensed psychologist (SPL) with over 15 years of
experience in applying machine learning to psychotherapy evaluation, a
licensed clinical social worker specializing in the evaluation of human
labeling for machine learning, and doctoral-level counselor and director of
clinical AI at mpathic.ai.

We show the detailed results in Table 6. The ratings recorded in Table 6 are
on six different clinical parameters namely, construct, affective attitude,
burden, ethicality, intervention coherence, opportunity cost, and perceived
effectiveness. The definitions and application use cases of the preceding
parameters are mentioned in Table 7.

##  Appendix D Counseling vs Clinical Conversation Summarization

In this section we discuss the critical difference between the clinical and
counseling conversations. Unlike mental health counseling, clinical
conversations are more direct towards the problem. In mental health
counseling, it is the job of the therapist to frame the conversation flow and
build therapeutic alliance to gain the trust of patients. Another crucial
difference is the direction of information (viz issues, symptoms, etc.) flow
between the patient and the medical personnel – in clinical conversations,
patients usually convey their issues and symptoms without any prompts from the
doctor; whereas, most often than not, the therapist has to nudge and encourage
patients to express their issues in counseling sessions. To support this, we
conduct a comparative analysis on ChiCCo and MEMO datasets. Table 8 shows the
results. As expected, ConSum does not perform well due to its highly domain-
centric modules. Furthermore, we also observe that T5, which is essentially
ConSum without the there domain-specific modules, obtains the overall best
performance among all models.

Table 8. Results on the clinical conversation ChiCCo dataset. Domain-centric ConSum underperforms in the non-mental health conversation setup. Together with the encouraging performance on MEMO, it demonstrates the high association of ConSum for the mental health domain. Model | R-1 | R-2 | R-L | QAE | BS
---|---|---|---|---|---
PLM | $11.82$ | $0.17$ | $14.84$ | $12.37$ | $-1.0890$
RankAE | $24.00$ | $6.37$ | $20.15$ | $30.25$ | $-0.7466$
SM | $21.65$ | $7.94$ | $20.37$ | $14.14$ | $-1.001$
Pegasus | $57.54$ | $55.86$ | $27.57$ | $41.10$ | $-0.1253$
T5 | 71.67 | 63.62 | 70.75 | $47.45$ | 0.2028
ConSum | $37.13$ | $22.12$ | $17.24$ | 83.21 | $0.0210$