# Reasoning in Dialog:
Improving Response Generation by Context Reading Comprehension

Xiuying Chen1,2, Zhi Cui3, Jiayi Zhang3, Chen Wei3,
Jianwei Cui3, Bin Wang3, Dongyan Zhao1,2, and Rui Yan4,5
Corresponding author (ruiyan@ruc.edu.cn).

###### Abstract

In multi-turn dialog, utterances do not always take the full form of sentences
(Carbonell 1983), which naturally makes understanding the dialog context more
difficult. However, it is essential to fully grasp the dialog context to
generate a reasonable response. Hence, in this paper, we propose to improve
the response generation performance by examining the model’s ability to answer
a reading comprehension question, where the question is focused on the omitted
information in the dialog. Enlightened by the multi-task learning scheme, we
propose a joint framework that unifies these two tasks, sharing the same
encoder to extract the common and task-invariant features with different
decoders to learn task-specific features. To better fusing information from
the question and the dialog history in the encoding part, we propose to
augment the Transformer architecture with a memory updater, which is designed
to selectively store and update the history dialog information so as to
support downstream tasks. For the experiment, we employ human annotators to
write and examine a large-scale dialog reading comprehension dataset.
Extensive experiments are conducted on this dataset, and the results show that
the proposed model brings substantial improvements over several strong
baselines on both tasks. In this way, we demonstrate that reasoning can indeed
help better response generation and vice versa. We release our large-scale
dataset for further research111https://github.com/yingtaomj/Reasoning-in-
Dialog.

## Introduction

In recent years, text generation has made impressive progress (Chen et al.
2019; Li et al. 2020; Yu et al. 2020; Liu et al. 2020; Zhang et al. 2021), and
open-domain dialogue generation has become a research hotspot in Natural
Language Processing due to its broad application prospect, including chatbots,
virtual personal assistants (Qiu et al. 2019; Debnath, Sengupta, and
Wabgaonkar 2018; Li et al. 2019), etc. However, studies (Carbonell 1983) show
that users of dialogue systems tend to use succinct language which often omits
entities or concepts made in previous utterances. To make appropriate
responses, dialogue systems must be equipped with the ability to understand
these incomplete utterances. This naturally leads to the reading comprehension
task, where correctly answering questions about the context requires
understanding of natural language of the dialog context (Rajpurkar et al.
2016).

| Example 1 | Example 2 | Example 3
---|---|---|---
$A_{1}$ |  | 求帮忙取名字姓程，俩男娃
---
Please help me decide how to name
my two kids whose last name is Cheng
| 我最喜欢的歌手是MJ
---
My favorite singer is MJ
| 那么我们即使不死,也在天堂
---
Then we are in heaven even
if we don’t die
$B_{1}$ |  | 程饭和程菜
---
Cheng fan and Cheng cai
| 你最喜欢他的什么歌呢？
---
What’s your favorite song?
| 这话哪抄的
---
Where did you copy that
$A_{2}$ |  | 哈哈哈哈哈
---
LOL
| Thriller
---
Thriller
| 三毛
---
Sanmao
$B_{2}$ |  | 请务必接受我的建议
---
Please accept my advice
| 我没听过呢，有这首歌的mv吗
---
I haven’t heard of it.
Is there an MV of this song?
| 想起以前豆瓣有个帅哥叫东门
---
Remember that there was a
handsome man named Dongmen
in Douban
$A_{3}$ |  | 咱俩一起生我就接受
---
（取名程饭和程菜）
I’ll accept that (name as Cheng
fan and Cheng cai) if
they are our children
| 有(这首歌的MV)，我发给你看
---
Yes (I have the MV), I’ll send it to you
| 那我（豆瓣）叫个南亭算了
---
Then my name (in Douban)
will be Nanting.
Question |  | 如果一起生娃那孩子叫什么
---
If the children are ours,
how to call them?
| 准备发什么？
---
What is going to be sent?
| 南亭是什么的ID？
---
Nanting is ID of what
Answer |  | 程饭和程菜
---
Cheng fan and Cheng cai
| 发thriller的的mv
---
Thriller MV
| 豆瓣
---
Douban
$B_{3}$ |  | 我觉得这名字很好听啊
---
I think it’s a nice name
| 好啊，我一直想看他的MV呢
---
Good, I’ve always
wanted to see his MV
| 豆瓣就差你这个ID了
---
Douban is waiting for your ID
| Reasoning
---
Type
Paraphrasing (49.0%) |  Lexical match (28.5%) |  Pragmatics (22.5%)
Table 1: Examples from the dataset. Questions are concentrated on the omitted
information of $A_{3}$ (which is shown in brackets), and reasoning type is the
type of ability that is needed to answer the question.

Take Example 2 in Table 1 for example, contents in parentheses are information
omitted in the utterance. Humans are capable of comprehending such missing
utterances dependent based on previous utterances and commonsense. For
instance, $A_{3}$ means sending an MV to B instead of a gift. However, though
of high importance, it is difficult for models to capture the implicit
dependency between utterances without specific design, and that is why the
reading comprehension task is proposed (Rajpurkar et al. 2016; Reddy, Chen,
and Manning 2019). In this case, by reasoning and correctly answering the
question with keyword “MV”, the model learns that the dialog is focused on MV,
which leads to a proper response that is also concentrated on music. Such
cases that require dependency on the previous context to fully comprehend
current utterance takes up about 60% according to a survey in Pan et al.
(2019). This inspires us to come up with a multi-task framework that generates
the response and answers reading comprehension question at the same time,
which can boost the performance of each task.

Our Multi-task Response Generator (MRG) augments the previously proposed
Transformer architecture (Vaswani et al. 2017) with the ability to encode
multiple utterances in a question-aware fashion. The proposed model first uses
a cross-attention mechanism (Vaswani et al. 2017) between the question and
dialog words to identify representative words in dialog with the help of
question. Concretely, we propose a memory updater, which updates its memory
state using both the current inputs and previous memory state. The memory
state can be interpreted as a container of the highly summarized dialog
history information. During the cross-attention process, the current dialog
representation is enhanced with the memory state from the previous step. MRG
then uses a hierarchical inner attention, first over different words in each
utterance, and then over all utterances in dialog history, to successively
learn the utterance-level features. Finally, MRG utilizes the utterance-level
and question features to select the answer to the question while generating
the response words.

Since there lacks large-scale dialog reading comprehension datasets, we hire
an annotation team to construct a dialog reading comprehension dataset (DRCD).
Concretely, based on the Restoration-200K dataset proposed by Pan et al.
(2019), where the omitted word span is annotated by humans, we ask the
annotators to write question where the answer is the missing phrase. We
manually construct 10k cases, based on which we train a question generator and
leverage the model to construct questions for the rest of the dataset. We
benchmark several classic dialog generation and reading comprehension
baselines on DRCD. We also conduct experiments to show that the proposed model
brings substantial improvements over these baselines on both tasks. In this
way, we demonstrate that reasoning can indeed help better response generation
and vice versa.

Our contributions can be summarized as follows:

$\bullet$ We propose the multi-task learning framework, which jointly answers
reading comprehension questions and generates a proper response in multi-turn
dialog scenario.

$\bullet$ We augment the Transformer architecture with a memory updater, which
helps selectively store and update history dialog information.

$\bullet$ We release a large scale dialog reading comprehension dataset.
Experimental results on this dataset demonstrate the effectiveness of our
proposed framework.

## Related Work

#### Multi-turn Dialog.

In recent years, text generation has made impressive progress (Li et al. 2018;
Chan et al. 2019; Gao et al. 2020b; Xie et al. 2020), and multi-turn dialog
model aims to take a message and utterances in previous turns as input and
generates a response (Tao et al. 2019; Gao et al. 2020a). Several works (Zhang
et al. 2019; Adiwardana et al. 2020; Chan et al. 2020) simplify the multi-turn
dialog into single-turn problem by simply concatenating multiple sentences
into one sentence, and utilized the basic Seq2seq based on RNN or Transformer
to model long sequence. To make better use of multi-turn utterances, Xing et
al. (2017) apply hierarchical attention on word-level and utterance-level
information. There also various dialog datasets (Lowe et al. 2015; Zhang et
al. 2018; Welleck et al. 2018; Reddy, Chen, and Manning 2019). However, these
datasets do not contain reading comprehension question-answering pairs.

#### Machine Reading Comprehension.

Machine reading comprehension (MRC) focuses on modeling semantic matching
between a question and a reference document, which read the full text to
select relevant text spans and then infer answers. Choi et al. (2017) propose
hierarchical coarse-to-fine methods in order to mimic the reading mode of
human. Huang et al. (2017) come up with a fully-aware fusion attention
mechanism and apply it on MRC tasks. Large-scale datasets for MRC have also
been proposed in parallel. CommonsenseQA (Talmor et al. 2018) is a dataset for
commonsense question answering extracted from CONCEPTNET (Speer, Chin, and
Havasi 2016). DROP (Dua et al. 2019) and COSMOS (Huang et al. 2019) focus on
factual understanding and commonsense comprehension, respectively. In this
paper, we propose another MRC dataset focused on machine comprehension on
dialog corpus.

#### Multi-task Learning.

Multi-task learning (MTL) is a learning paradigm in machine learning and it
aims to leverage useful information contained in multiple related tasks to
help improve the generalization performance of all the tasks(Caruana 1997).
There are a large quantity of natural language processing tasks based on
multi-task learning, such as word segmentation, POS tagging, dependency
parsing, and text classification (Bohnet and Nivre 2012; Hatori et al. 2012;
Li et al. 2013; Liu, Qiu, and Huang 2016). Collobert and Weston (2008)
describe a single convolutional network that jointly trained several NLP
tasks, such as part-of-speech tags, chunks, named entity tags, semantic roles.
Liu et al. (2015) develop a multi-task deep neural network combining tasks of
multiple-domain classification and information retrieval to learn
representations across multiple tasks. In this work, we apply multi-task
learning on response generation and reading comprehension on dialog.

## Problem Formulation

Before presenting our approach for the dialog reading comprehension multi-
task, we first introduce our notations and key concepts.

We assume that a conversation is conducted between two users. Suppose there
are already $N^{u}$ turns in a dialogue, so we have historical utterances as
$X=(X_{1},X_{2},\ldots,X_{N^{u}})$, where each utterance $X_{j}$ is depicted
as $X_{j}=(x^{j}_{1},x^{j}_{2},\ldots,x^{j}_{N_{j}})$ and $x^{j}_{i}$ denotes
a word. Accordingly, MRG aims to predict the ($N^{u}$+$1$)-th utterance ,i.e.,
the response, $Y=(y_{1},y_{2},\ldots,y_{N^{y}})$, according to the historical
utterances $X$:

$p(Y|X)=\textstyle\prod_{i=1}^{N^{y}}p\left(y_{i}|X,y_{1},\ldots,y_{i-1}\right)$
(1)

Apart from the reponse generation, we also design a question-answering task
for the model. That is, targeted at the $N^{u}$-th utterance, where some
keywords are missing, there is a question $Q=(q_{1},q_{2},...,q_{N^{q}})$ that
asks about such missing information, and the answer is a score vector
$A=(a_{1},a_{2},...a_{N^{a}})$ that extracts the missing keywords from
previous utterances. $N^{a}=\textstyle\sum_{i=1}^{N^{u}}N_{i}$. Each score
$a_{i}\in\\{0,1\\}$ denotes whether the $i$-th word is selected (1) or not
(0). The objective is to maximize the likelihood of all word labels $A$ given
the input:

$p\left(A|X\right)=\textstyle\prod_{i=1}^{N^{a}}p\left(a_{i}|X\right)$ (2)

## The Proposed MRG Model

### Overview

In this section, we propose the _Multi-task Response Generator_ , abbreviated
as MRG. An overview of MRG is shown in Figure 1, which can be split into three
main parts:

Figure 1: Overview of MRG. We divide our model into three parts: (1) Cross-
hierarchical Encoder (which consists of memory-augment cross attention and two
hierarchical self attentions); (2) Answer Selecter; (3) Response Generator.

$\bullet$ Cross-hierarchical encoder first uses a memory-augmented cross-
attention mechanism (Vaswani et al. 2017) between the question and dialog
words to identify representative words in dialog with the help of question. It
then uses a hierarchical inner attention, first over different words in an
utterance, and then over all utterances in dialog history, to successively
learn the utterance-level features.

$\bullet$ Answer selecter takes the question representation and utterance-
level dialog features as input to predict the answer.

$\bullet$ Response generator produces the response by attending to the
utterance-level features.

### Cross-hierarchical Encoder

To begin with, we use an embedding matrix $e$ to map a one-hot representation
of each word in $X$, $Q$, into a high-dimensional vector space. We then employ
a bi-directional recurrent neural network (Bi-RNN) to model the temporal
interactions between words:

$\displaystyle h_{i}^{x,j}$ $\displaystyle=\text{Bi-
RNN}_{x}\left(e(x_{i}^{j}),h_{i-1}^{x,j}\right),$ (3) $\displaystyle
h_{i}^{q}$ $\displaystyle=\text{Bi-RNN}_{y}\left(e(q_{i}),h_{i-1}^{q}\right),$

where $h_{i}^{x,j}$ and $h_{i}^{q}$ denote the hidden state of $i$-th step in
Bi-RNN for $X_{j}$ and $Q$, respectively. Following (Zhao, Zhao, and Eskénazi
2017; Chen et al. 2018), we choose long short-term memory (LSTM) as the cell
for Bi-RNN.

#### Memory-augmented Cross Attention.

This module grounds the conversation context by the question and fuses the
information of the question into the dialog representation. Concretely, it has
a stack of $L$ identical layers. In each layer, we iteratively fuse the
information from question words to the dialog words by Memory-augmented Cross
Attention Module (MCAM). For convenience, we denote the output of $l$-th
encoder layer as $m^{l,j}_{i}$ and the input for the first layer $m^{0,j}_{i}$
is initialized as $h^{x,j}_{i}$. Concretely, MCAM is based on the traditional
Cross Attention Module (CAM) Transformer architecture (Vaswani et al. 2017).
We first introduce the original CAM, and then introduce our modification.

The first input for CAM is for query $Q$ and the second input is for keys $K$
and values $V$ for attention, which we denote as $x_{i}$ and $h^{q}_{*}$
respectively:

$m^{l,j}_{i}=\text{CAM}(m^{l-1,j}_{i},h^{q}_{*}).$ (4)

Each output element, $m^{l,j}_{i}$, is computed as weighted sum of a linearly
transformed input values:

$m^{l,j}_{i}=\textstyle\sum_{k=1}^{N_{j}}\alpha_{i,k}^{l,j}\left(h^{q}_{k}W^{V}\right).$
(5)

Each weight coefficient, $\alpha_{i,k}^{l,j}$ , is computed using a softmax
function:

$\alpha_{i,k}^{l,j}=\frac{\exp\left(\beta_{i,k}^{l,j}\right)}{\sum_{k=1}^{N_{j}}\exp\left(\beta_{i,k}^{l,j}\right)}.$
(6)

And $\beta_{i,k}^{l,j}$ is computed using a compatibility function that
compares two input elements:

$\beta_{i,k}^{l,j}=\frac{\left(m^{l-1,j}_{i}W^{Q}\right)\left(h^{q}_{k}W^{K}\right)^{T}}{\sqrt{d}},$
(7)

where $d$ stands for hidden dimension.
$W^{Q},W^{K},W^{V}\in\mathbb{R}^{N_{j}\times N_{j}}$ are parameter matrices.

While the aforementioned vanilla CAM is a powerful method, it is less suitable
for multi-turn dialog due to its inability to fully utilize dialog history
information. Thus, we augment it with an external memory module, which helps
to remember and update history dialog information in a multi-slot way as
illustrated in Figure 1. The input for query, i.e., $m^{l-1,j}_{i}$ is updated
to $n^{l-1,j}_{i}$ through a memory updator, which will then be fed into CAM
in Equation 4. Concretely, the memory updator aggregates the information from
both its intermediate hidden states $\hat{m}^{l-1}_{i}$
($\hat{m}^{l-1}\in\mathbb{R}^{N_{j}\times d}$) and the utterance (memory)
states $m^{l-1,j}_{i}$ from the last layer, using a multi-head attention.
Specifically, the input for query $Q$ is $m^{l-1,j}_{i}$, and input for key
$K$ and value $V$ is $[\hat{m}^{l-1}_{i};m^{l-1,j}_{i}]$. The memory augmented
hidden states are further encoded using a feed-forward layer and then merged
with the intermediate hidden states $\hat{m}^{l-1}$ using a residual
connection and layer norm. We summarize the procedure below:

$\displaystyle s^{l-1,j}_{i}$
$\displaystyle=\textrm{CAM}(m^{l-1,j}_{i},\hat{m}^{l-1}_{i}),$ $\displaystyle
c^{l-1,j}_{i}$
$\displaystyle=\textrm{tanh}(W_{a}^{l-1}m^{l-1,j}_{i}+W_{b}^{l-1}s^{l-1,j}_{i}),$
$\displaystyle z^{l-1}_{i}$
$\displaystyle=\textrm{sigmoid}(W_{c}^{l-1,j}m^{l-1,j}_{i}+W_{d}^{l-1}s_{i}^{l-1}),$
$\displaystyle n^{l-1,j}_{i}$ $\displaystyle=(1-z_{i}^{l-1,j})\odot
c_{t}^{l-1,j}+z^{l-1,j}_{i}\odot m^{l-1,j}_{i},$

where $\odot$ denotes Hadamard product, $W_{a}^{l-1}$, $W_{b}^{l-1}$,
$W_{c}^{l-1}$, and $W_{d}^{l-1}$ are trainable weights, $c_{i}^{l-1}$ is the
internal cell state. $z_{i}^{l-1}$ is the update gate that controls which
information to retain from the previous memory state.

#### Hierarchical Self Attention.

After utilizing question information to emphasize important keywords,
$m^{L,j}_{i}$ (the output of last MCAM layer) is then processed by a
hierarchical attentive module to encode long-term dependency among words into
the representations. The first level in our hierarchical attention encodes
each utterance independently from other utterances at word-level, resulting in
a fixed-dimensional representation of each utterance. Concretely, the word-
level attentive module simplifies the Multi-head Attention Module (MAM) in
Transformer, which is similar to CAM, but takes the same input for query, key
and value:

$h^{w,j}_{i}=\text{MAM}(m^{L,j}_{i},m^{L,j}_{*}).$ (8)

A mean-pooling operation is then used over word vectors in each utterance to
obtain a fixed-length utterance-level representation:

$h^{u^{\prime},j}=\operatorname{meanpool}\left(\left\\{h^{w,j}_{1},\cdots,h^{w,j}_{N_{j}}\right\\}\right).$
(9)

Similar to word-level attention, an utterance-level MAM is applied on these
representations to fuse information between different utterances:

$h^{u,j}=\text{MAM}(h^{u^{\prime},j},h^{u^{\prime},*}).$ (10)

From the utterance representation, we can also obtain the overall dialog
history representation, which will be used in the response decoder part:

$h^{d}=\operatorname{meanpool}\left(\left\\{h^{u,1},\cdots,h^{u,N^{u}}\right\\}\right).$
(11)

### Answer Selector

After fusing information from question and dialog context, it is time to
select words from context as the answer to the question. Since we have several
utterance representations, and either taking the average or summing them
together by specific weights is inappropriate and inelegant. Hence, we
concatenate all utterance and question representations together and apply a
multi-layer perceptron to them to generate the word extracting probabilities:

$\displaystyle h^{q}$
$\displaystyle=\operatorname{meanpool}\left(\left\\{h^{q}_{1},\cdots,h^{q}_{N^{q}}\right\\}\right),$
$\displaystyle\hat{A}$
$\displaystyle=W_{f}\tanh\left(W_{e}\left[h^{u,1};\ldots;h^{u,N^{u}};h^{q}\right]+b^{e}\right)+b^{f},$

where $[;]$ denotes concatenation operation.

### Response Generator

To generate a consistent and informative response, we propose an RNN-based
decoder that incorporates outputs of utterance representations as illustrated
in Figure 1.

We first apply a linear transform layer on the input document vector
representation $h^{d}$ and use the output of this layer as the initial state
of decoder LSTM, shown in Equation 12. In order to reduce the burden of
compressing document information into the initial state $s_{0}$, we use the
attention mechanism (Bahdanau, Cho, and Bengio 2015a) to summarize the
utterance representations into context vector $f_{t-1}$ dynamically and we
will show the detail of these in this section later. We then concatenate the
context vector $f_{t-1}$ with the embedding of previous step output
$e(y_{t-1})$ and feed this into decoder LSTM, shown in Equation 13:

$\displaystyle s_{0}$ $\displaystyle=W_{g}h^{d}+b_{g},$ (12) $\displaystyle
s_{t}$ $\displaystyle=\text{LSTM}\left(s_{t-1},[f_{t-1};e(y_{t-1})]\right).$
(13)

Context vector $f_{t-1}$ is the vector that stores the dialog context
information at $t$-th step. Concretely, we use the decoder state $s_{t-1}$ to
attend to each utterance states $h^{u,i}$ and results in the attention
distribution $\gamma_{t}$, shown in Equation 15. Then we use the attention
distribution $\gamma_{t}$ to weighted sum the document states as the context
vector $f_{t-1}$.

$\displaystyle\gamma^{{}^{\prime}}_{t-1,i}$
$\displaystyle=W_{n}^{\intercal}\tanh\left(W_{s}s_{t-1}+W_{h}h^{u,i}\right),$
(14) $\displaystyle\gamma_{t-1,i}$
$\displaystyle=\exp\left(\gamma^{{}^{\prime}}_{t-1,i}\right)/\textstyle\sum^{N_{u}}_{j=1}\exp\left(\gamma^{{}^{\prime}}_{t-1,j}\right),$
(15) $\displaystyle f_{t-1}$
$\displaystyle=\textstyle\sum_{i=1}^{N_{u}}\gamma_{t-1,i}h^{u,i}.$ (16)

Finally, an output projection layer is applied to get the final generating
distribution $P_{t}^{v}$ over vocabulary, as shown in Equation 17. We
concatenate utterance context vector and the output of decoder LSTM $s_{t}$ as
the input of the output projection layer:

$P^{v}_{t}=\text{softmax}\left(W_{v}[s_{t};f_{t}]+b_{v}\right),$ (17)

We use the negative log-likelihood as the loss function:

$\displaystyle\mathcal{L}_{g}$
$\displaystyle=-\textstyle\sum^{N^{y}}_{t=1}\log P_{t}^{v}(y_{t}).$ (18)

## Experimental Setup

### Dataset

To our best knowledge, no exsiting works consider MRC in response generation
task. Hence, we first propose a dialog reading comprehension dataset (DRCD).
DRCD is based on the Restoration-200k dataset proposed by Pan et al. (2019),
where the utterance with omitted information is manually annotated. Such
omitted information leads to a difficulty in fully understanding the dialog
context and requires reasoning ability to for a model. Hence, we hire an
annotation team to write questions that are focused on the missing
information.

Since it is time-consuming to write questions for the whole dataset, and based
on the labeled answer it is rather easy to construct the question, we ask the
team to write questions for 10k cases, and then automatically generate
questions for the rest of the dataset. Concretely, we utilize PG (See, Liu,
and Manning 2017) to generate questions due to its good performance in many
tasks including summarization and dialog completion (Pan et al. 2019; Chen et
al. 2019). We then conduct a human evaluation to examine the generation
quality. Concretely, we randomly sample 200 cases and asked three annotators
to state how well they agree with the following two statements, on a scale of
one to five (strongly disagree, disagree, neutral, agree, or strongly agree):
1) The generated question asks about the omitted phrase. 2) The generated
question is written in fluent Chinese. The result shows that generated
questions that score over 3 takes up 76.5%, showing that most of the generated
questions are of good quality. The kappa statistics indicate the moderate
agreement between annotators.

We randomly split the dataset with question-answer pair to 113,116 training,
3,000 validation, and 3,000 test cases. The average character-level context
length and utterance length of the dataset is and 43.4 and 9.05. Note that in
the validation and test datasets the questions are all written by human,
ensuring that the testing results are convincing.

Model | BLEU1 | BLEU2 | BLEU3 | BLEU4 | Average | Extrema | Greedy
---|---|---|---|---|---|---|---
Seq2Seq | 0.2260 | 0.1566 | 0.0876 | 0.0671 | 0.4341 | 0.6695 | 0.7759
HRED | 0.2273 | 0.1559 | 0.0871 | 0.0667 | 0.4320 | 0.6601 | 0.7885
VAE | 0.2316 | 0.1586 | 0.0886 | 0.0680 | 0.4350 | 0.6396 | 0.7808
Transfomer | 0.2181 | 0.1482 | 0.0825 | 0.0631 | 0.4407 | 0.6500 | 0.7920
PAC | 0.2413 | 0.1624 | 0.0902 | 0.0689 | 0.4396 | 0.6447 | 0.7909
MRG | 0.2632 | 0.1735 | 0.0968 | 0.0741 | 0.4513 | 0.6769 | 0.8025
MRG w/o MCAM | 0.2224 | 0.1533 | 0.0857 | 0.0656 | 0.4436 | 0.6630 | 0.7837
MRG w/o MAM | 0.2404 | 0.1616 | 0.0946 | 0.0665 | 0.4343 | 0.6740 | 0.7798
MRG w/o MemUpd | 0.2498 | 0.1585 | 0.0894 | 0.0747 | 0.4419 | 0.6551 | 0.7884
MRG w/o MT | 0.2231 | 0.1541 | 0.0862 | 0.0661 | 0.4343 | 0.6734 | 0.7645
Table 2: Automatic evaluation results on response generation task. The best results are bold. Model | Accuracy(%)
---|---
Mean | Best
MemN2N | 37.85 | 38.22
DMN | 40.83 | 42.21
QRN | 40.80 | 43.71
DMN+ | 43.97 | 45.02
MRG | 45.43 | 47.17
MRG w/o MT | 44.89 | 46.34
Table 3: Automatic evaluation results on MRC task. Best accuracy over 10 runs.

### Comparison Methods

To evaluate the performance of our proposed model, we compare it with the
following response generation and MRC baselines:

Response Generation baselines:

Seq2Seq (Bahdanau, Cho, and Bengio 2015b): the vanilla schema of the sequence
to sequence model with attention mechanism.

HRED (Serban et al. 2016): extends the hierarchical recurrent encoder-decoder
neural network to the dialogue domain.

VAE (Zhao, Zhao, and Eskénazi 2017): uses latent variables to learn a
distribution over potential conversational intents and generates diverse
responses.

Transformer (Vaswani et al. 2017): is based solely on attention mechanisms.

PAC (Pan et al. 2019): is a “pick-and-combine” model to restore the incomplete
utterance from its context, and then use the restored utterance to generate
the next response.

MRC baselines:

MemN2N (Sukhbaatar et al. 2015): is an extension of RNNsearch to the case with
multiple computational hops.

DMN (Kumar et al. 2016): processes input sequences and questions, forms
episodic memories, and generates relevant answers.

DMN+ (Xiong, Merity, and Socher 2016): proposes several improvements to memory
and input modules of DMN.

QRN (Seo et al. 2017): is a variant of RNN that effectively handles both
short-term (local) and long-term (global) sequential dependencies to reason
over multiple facts.

### Implementation Details

We implement our experiments in TensorFlow (Abadi et al. 2016) on an NVIDIA
GTX 1080 Ti GPU. We truncate input dialog to 100 words, with 20 words in each
utterance. We chose 100 as our truncation size as we did not find significant
improvement when increasing input length from 100 to 200 tokens. The minimum
decoding step is 10, and the maximum step is 20. The word embedding dimension
is set to 128 and the number of hidden units is 256. We initialize all of the
parameters randomly using a Gaussian distribution. The batch size is set to
16, and we limit the vocabulary size to 50K. We use Adagrad optimizer (Duchi,
Hazan, and Singer 2010) as our optimizing algorithm. We also apply gradient
clipping (Pascanu, Mikolov, and Bengio 2013) with a range of $[-2,2]$ during
training. During the inference stage, the checkpoint with smallest validation
loss is chosen and the beam-search size is set to 4 for all methods. Note that
when evaluating the response generation performance, we use the generated
questions as input instead of the ground truth human-written questions for the
sake of fairness.

### Evaluation Metrics

To evaluate the results of the generated responses, we adopt the following
metrics widely used in existing research.

Overlap-based Metric. Following Li et al. (2021); Xu et al. (2020), we utilize
BLEU score (Papineni et al. 2002), an algorithm which has been widely used in
machine translation and dialogue system, to measure n-grams overlaps between
ground-truth and generated response. Specifically, we follow the conventional
setting in previous work (Gu et al. 2019) to compute BLEU scores using
smoothing techniques (smoothing 7).

| Example 1 | Example 2 | Example 3
---|---|---|---
$A_{1}$ |  | 有木有人带妹子吃喝玩乐在杭州
---
Is there anyone to take girls to eat,
drink and have fun in Hangzhou
| 天蝎座不自恋真的就能死
---
Scorpio will die without narcissism
| lz女坐标杭州心情不nice
---
The author is a girl, located in
Hangzhou, has a bad mood
$B_{1}$ |  | 没真相不敢带
---
I dare not bring a girl without a photo
| 挺准最近就是被一个天蝎虐
---
That’s right. I’ve been
abused by a Scorpio recently.
| 那怎么办
---
Then what to do
$A_{2}$ |  | 有了真相更不敢带哈
---
With a photo, you will
dare not bring her more
| 嗯
---
Yes
| 睡觉么
---
What about sleep
$B_{2}$ |  | 犀利
---
Sharp
| 你是摩羯啊
---
So you are a Capricorn
| 也可以不过不是长久之计
---
That’s good, but not a long-term solution
$A_{3}$ |  | 一般(犀利)啦
---
Generally (sharp)
| (摩羯被天蝎虐)这就是宿命
---
This (Capricorn abused
by Scorpio) is fate.
| 哈哈那(心情不nice)怎么办
---
Ha-ha, then what to do (if in bad mood)
Question |  | 什么很一般
---
What is general
|  什么是宿命
---
What is fate
| 什么出问题了
---
What’s wrong
Answer |  | 犀利程度一般
---
The degree of sharpness
| 摩羯被天蝎虐
---
Capricorn abused by Scorpio
| 心情
---
Mood
Reference |  | 一般都这么犀利男人咋办
---
If that is general sharp then
what to do with man
| 所以我喜欢上天蝎就是个错误
---
So it was a mistake for me
to like Scorpio
| 找个心灵的驿站
---
Find a post station for the soul
HRED |  | 是你的错觉
---
It’s your delusion
| 洗洗睡了
---
Wash and sleep
| 生命健康更加可贵
---
Life and health are more valuable
VAE |  | 好吧,改天可以一起去嗨皮了
---
Well, we can play the other day
| 我帮同事发帖子勾搭上了
---
I helped my colleagues
post and hook up
| 年龄大了就想睡
---
When you get older, you want to sleep
PAC |  | 你也要去对不啦
---
So you will go there, aren’t you?
| 那你还不如玩手机
---
You might as well
play with your cell phone
| 我闲了就读书
---
I read when I’m free
MRG |  |  是过于犀利啦
---
It’s too sharp
| 还是要看人，怎么就宿命了哈哈哈
---
It still depends on people.
You should not believe in fate lol.
| 出门旅游，逛逛豆瓣
---
Travel outside, visit Douban
Table 4: Responses generated by baselines and our model along with the QA
pairs.

Embedding Metrics. To capture the semantic matching degrees between generated
responses and ground-truth, we perform evaluations on embedding space. In
consistent with previous study (Gu et al. 2019), we compute the similarity
between the bag-of-words (BOW) embeddings representations of generated results
and reference. In particular, we calculate three metrics:1) Greedy (BOW-
Greedy), i.e., greedily matching words in two utterances based on the cosine
similarities; 2) Average (BOW-Average), cosine similarity between the averaged
word embeddings in the two utterances (Mitchell and Lapata 2008); 3) Extrema
(BOW-Extrema), cosine similarity between the largest extreme values among the
word embeddings in the two utterances (Forgues et al. 2014).

Human Metrics. We also employ human evaluation to assess the responses
generated by our model and the baselines. Three well-educated annotators are
hired to evaluate the quality of generated responses, where the evaluation is
conducted in a double-blind fashion. Totally, 200 randomly sampled responses
generated by each model are rated by each annotator with two different
aspects, i.e., readability (Is the response grammatically formed and smooth?),
informativeness (Does the response contains informative words?). Criteria are
scored from 1 to 3, i.e., bad, normal, and good.

## Experimental Results

### Overall Performance

Automatic evaluation. We first examine whether our MRG outperforms generative
baselines as listed in Table 2. Our model outperforms baselines on all
automatic metrics. This demonstrates that our model generates more appropriate
responses by reading comprehension, and understands the dialog context better
by predicting response. Especially, our model improves approximately 16.46%
over seq2seq on BLEU1, and outperforms PAC by 9.07%. We also list the results
of ablation study in Table 2, aiming to assess the contribution of individual
components. Our experiments confirmed that interacting between dialog and
question by Memory-augmented Cross Attention Module is beneficial (see row w/o
MCAM), as well as self-attention module (see row w/o MAM) memory updator (see
row w/o MemUpd).

We next examine whether our MRG outperforms MRC baselines in Table 3.
Generally, these baselines perform similar to the experiment on bAbI dataset
(Bordes and Weston 2017). Specifically, DMN+ is the strongest baseline, which
achieves 43.97% accuracy on average. QRN, however, does not perform as well as
it does on bAbI dataset, obtaining lower accuracy than DMN+. Our model obtains
highest mean accuracy and best accuracy over 10 runs among all baselins.

Human evaluation. The results of human evaluation are listed in Table 5. Our
model significantly outperforms most of the baselines in terms of all the
metrics. Particularly, our model increases informativeness approximately 4.1%
over PAC. This demonstrates that trying to answer reading comprehension
question about dialog history if beneficial for improving and enriching the
responses.

### Analysis of Multi-task learning

Our model aims to generate response as well as answering MRC questions, which
can be regarded as a multi-task. Hence, in this subsection, we examine whether
these two tasks can complement each other. We list the performance on two
single tasks by ’MRG w/o MT’ in Table 2 and Table 3, which solely generates
response and answers MRC question, respectively. It can be seen that by
answering reading comprehension question, the performance of dialog generation
increases by 12.1% in terms of BLEU4 score, and by generating responses at the
same time, MRC accuracy increases by 1.2%.

Model | Readability | Informativeness
---|---|---
HRED | 1.43 | 1.46
VAE | 1.60 | 1.58
PAC | 1.56 | 1.72
MRG | 1.63 | 1.75
Table 5: Human evaluation on two aspects: Readability and informativeness.

## Conclusion

In this paper we propose the multi-task framework to generate response and
answer reading comprehension questions about multi-turn dialog. Concretely,
the two tasks share the same encoder to extract the common and task-invariant
features with different decoders to learn specific features. To better fusing
information from the question and the dialog history in the encoding part, we
propose to augment the Transformer architecture with a memory updater, which
is designed to selectively store and update the history dialog information.
Experimental results show that our proposed model outperforms classic
baselines. In the future we would like to apply our model to other multi-task
scenarios.

## Acknowledgments

This work was supported by the National Key Research and Development Program
of China (No. 2020YFB1406702), the National Science Foundation of China (NSFC
No. 61876196 and No. 61672058), Beijing Outstanding Young Scientist Program
No. BJJWZYJH012019100020098. Rui Yan is supported as a young fellow at Beijing
Academy of Artificial Intelligence (BAAI).

## References

  * Abadi et al. (2016) Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; Kudlur, M.; Levenberg, J.; Monga, R.; Moore, S.; Murray, D. G.; Steiner, B.; Tucker, P. A.; Vasudevan, V.; Warden, P.; Wicke, M.; Yu, Y.; and Zhang, X. 2016.  TensorFlow: A System for Large-Scale Machine Learning.  In _OSDI_.
  * Adiwardana et al. (2020) Adiwardana, D.; Luong, M.-T.; So, D. R.; Hall, J.; Fiedel, N.; Thoppilan, R.; Yang, Z.; Kulshreshtha, A.; Nemade, G.; Lu, Y.; et al. 2020.  Towards a human-like open-domain chatbot.  _arXiv preprint arXiv:2001.09977_ .
  * Bahdanau, Cho, and Bengio (2015a) Bahdanau, D.; Cho, K.; and Bengio, Y. 2015a.  Neural Machine Translation by Jointly Learning to Align and Translate.  In _ICLR_.
  * Bahdanau, Cho, and Bengio (2015b) Bahdanau, D.; Cho, K.; and Bengio, Y. 2015b.  Neural Machine Translation by Jointly Learning to Align and Translate.  _ICLR_ .
  * Bohnet and Nivre (2012) Bohnet, B.; and Nivre, J. 2012.  A transition-based system for joint part-of-speech tagging and labeled non-projective dependency parsing.  In _EMNLP_ , 1455–1465.
  * Bordes and Weston (2017) Bordes, A.; and Weston, J. 2017.  Learning End-to-End Goal-Oriented Dialog.  _ICLR_ .
  * Carbonell (1983) Carbonell, J. 1983.  Discourse Pragmatics and Ellipsis Resolution in Task-Oriented Natural Language Interfaces.  In _ACL_.
  * Caruana (1997) Caruana, R. 1997.  Multitask learning.  _Machine learning_ 28(1): 41–75.
  * Chan et al. (2019) Chan, Z.; Chen, X.; Wang, Y.; Li, J.; Zhang, Z.; Gai, K.; Zhao, D.; and Yan, R. 2019\.  Stick to the Facts: Learning towards a Fidelity-oriented E-Commerce Product Description Generation.  In _EMNLP_ , 4960–4969.
  * Chan et al. (2020) Chan, Z.; Zhang, Y.; Chen, X.; Gao, S.; Zhang, Z.; Zhao, D.; and Yan, R. 2020.  Selection and Generation: Learning towards Multi-Product Advertisement Post Generation.  In _EMNLP_ , 3818–3829.
  * Chen et al. (2019) Chen, X.; Chan, Z.; Gao, S.; Yu, M.-H.; Zhao, D.; and Yan, R. 2019.  Learning towards Abstractive Timeline Summarization.  In _IJCAI_ , 4939–4945.
  * Chen et al. (2018) Chen, X.; Gao, S.; Tao, C.; Song, Y.; Zhao, D.; and Yan, R. 2018.  Iterative Document Representation Learning Towards Summarization with Polishing.  _ArXiv_ abs/1809.10324.
  * Choi et al. (2017) Choi, E.; Hewlett, D.; Uszkoreit, J.; Polosukhin, I.; Lacoste, A.; and Berant, J. 2017.  Coarse-to-fine question answering for long documents.  In _ACL_ , 209–220.
  * Collobert and Weston (2008) Collobert, R.; and Weston, J. 2008.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _ICML_ , 160–167.
  * Debnath, Sengupta, and Wabgaonkar (2018) Debnath, P.; Sengupta, S.; and Wabgaonkar, H. M. 2018.  Identifying , Classifying and Resolving Non-Sentential Utterances in Customer Support Systems.
  * Dua et al. (2019) Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  _arXiv preprint arXiv:1903.00161_ .
  * Duchi, Hazan, and Singer (2010) Duchi, J. C.; Hazan, E.; and Singer, Y. 2010.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.  _J. Mach. Learn. Res._ 12: 2121–2159.
  * Forgues et al. (2014) Forgues, G.; Pineau, J.; Larchevêque, J.-M.; and Tremblay, R. 2014.  Bootstrapping dialog systems with word embeddings.  In _Nips, workshop_ , volume 2.
  * Gao et al. (2020a) Gao, S.; Chen, X.; Liu, C.; Liu, L.; Zhao, D.; and Yan, R. 2020a.  Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog.  In _Proceedings of The Web Conference 2020_ , 1138–1148.
  * Gao et al. (2020b) Gao, S.; Chen, X.; Ren, Z.; Zhao, D.; and Yan, R. 2020b.  Meaningful Answer Generation of E-Commerce Question-Answering.  _ACM Trans. Inf. Syst._ .
  * Gu et al. (2019) Gu, X.; Cho, K.; Ha, J.-W.; and Kim, S. 2019.  DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder.  In _International Conference on Learning Representations_.  URL https://openreview.net/forum?id=BkgBvsC9FQ.
  * Hatori et al. (2012) Hatori, J.; Matsuzaki, T.; Miyao, Y.; and Tsujii, J. 2012.  Incremental joint approach to word segmentation, POS tagging, and dependency parsing in Chinese.  In _ACL_ , 1045–1053.
  * Huang et al. (2017) Huang, H.-Y.; Zhu, C.; Shen, Y.; and Chen, W. 2017.  Fusionnet: Fusing via fully-aware attention with application to machine comprehension.  _arXiv preprint arXiv:1711.07341_ .
  * Huang et al. (2019) Huang, L.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019.  Cosmos qa: Machine reading comprehension with contextual commonsense reasoning.  _arXiv preprint arXiv:1909.00277_ .
  * Kumar et al. (2016) Kumar, A.; Irsoy, O.; Ondruska, P.; Iyyer, M.; Bradbury, J.; Gulrajani, I.; Zhong, V.; Paulus, R.; and Socher, R. 2016.  Ask Me Anything: Dynamic Memory Networks for Natural Language Processing.  In _ICML_.
  * Li et al. (2019) Li, J.; Qiu, L.; Tang, B.; Chen, D.; Zhao, D.; and Yan, R. 2019.  Insufficient data can also rock! learning to converse using smaller data with augmentation.  In _AAAI_ , volume 33, 6698–6705.
  * Li et al. (2018) Li, J.; Song, Y.; Zhang, H.; Chen, D.; Shi, S.; Zhao, D.; and Yan, R. 2018.  Generating classical chinese poems via conditional variational autoencoder and adversarial training.  In _EMNLP_ , 3890–3900.
  * Li et al. (2020) Li, M.; Chen, X.; Gao, S.; Chan, Z.; Zhao, D.; and Yan, R. 2020.  VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles.  In _EMNLP_ , 9360–9369.
  * Li et al. (2021) Li, M.; Chen, X.; Yang, M.; Gao, S.; Zhao, D.; and Yan, R. 2021.  The Style-Content Duality of Attractiveness: Learning to Write Eye-Catching Headlines via Disentanglement.  In _AAAI_.
  * Li et al. (2013) Li, Z.; Zhang, M.; Che, W.; Liu, T.; and Chen, W. 2013.  Joint optimization for Chinese pos tagging and dependency parsing.  _IEEE/ACM transactions on audio, speech, and language processing_ 22(1): 274–286.
  * Liu et al. (2020) Liu, D.; Li, J.; Yu, M.-H.; Huang, Z.; Liu, G.; Zhao, D.; and Yan, R. 2020.  A Character-Centric Neural Model for Automated Story Generation.  In _AAAI_ , 1725–1732.
  * Liu, Qiu, and Huang (2016) Liu, P.; Qiu, X.; and Huang, X. 2016.  Recurrent neural network for text classification with multi-task learning.  _arXiv preprint arXiv:1605.05101_ .
  * Liu et al. (2015) Liu, X.; Gao, J.; He, X.; Deng, L.; Duh, K.; and Wang, Y.-Y. 2015.  Representation learning using multi-task deep neural networks for semantic classification and information retrieval .
  * Lowe et al. (2015) Lowe, R.; Pow, N.; Serban, I.; and Pineau, J. 2015.  The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems.  _arXiv preprint arXiv:1506.08909_ .
  * Mitchell and Lapata (2008) Mitchell, J.; and Lapata, M. 2008.  Vector-based models of semantic composition.  _NAACL-HLT_ 236–244.
  * Pan et al. (2019) Pan, Z.; Wang, Y.; Bai, K.; Zhou, L.; and Liu, X. 2019.  Improving Open-Domain Dialogue Systems via Multi-Turn Incomplete Utterance Restoration.  In _EMNLP_.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  BLEU: a method for automatic evaluation of machine translation.  In _ACL_ , 311–318. ACL.
  * Pascanu, Mikolov, and Bengio (2013) Pascanu, R.; Mikolov, T.; and Bengio, Y. 2013.  On the difficulty of training recurrent neural networks.  In _ICML_.
  * Qiu et al. (2019) Qiu, L.; Li, J.; Bi, W.; Zhao, D.; and Yan, R. 2019.  Are training samples correlated? learning to generate dialogue responses with multiple references.  In _ACL_ , 3826–3835.
  * Rajpurkar et al. (2016) Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.  SQuAD: 100, 000+ Questions for Machine Comprehension of Text.  _EMNLP_ .
  * Reddy, Chen, and Manning (2019) Reddy, S.; Chen, D.; and Manning, C. D. 2019.  Coqa: A conversational question answering challenge.  _TACL_ 7: 249–266.
  * See, Liu, and Manning (2017) See, A.; Liu, P. J.; and Manning, C. D. 2017.  Get To The Point: Summarization with Pointer-Generator Networks.  In _ACL_.
  * Seo et al. (2017) Seo, M.; Min, S.; Farhadi, A.; and Hajishirzi, H. 2017.  Query-Reduction Networks for Question Answering.  _arXiv: Computation and Language_ .
  * Serban et al. (2016) Serban, I.; Sordoni, A.; Bengio, Y.; Courville, A. C.; and Pineau, J. 2016.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models.  In _AAAI_.
  * Speer, Chin, and Havasi (2016) Speer, R.; Chin, J.; and Havasi, C. 2016.  Conceptnet 5.5: An open multilingual graph of general knowledge.  _arXiv preprint arXiv:1612.03975_ .
  * Sukhbaatar et al. (2015) Sukhbaatar, S.; Szlam, A.; Weston, J.; and Fergus, R. 2015.  End-To-End Memory Networks.  In _NIPS_.
  * Talmor et al. (2018) Talmor, A.; Herzig, J.; Lourie, N.; and Berant, J. 2018.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_ .
  * Tao et al. (2019) Tao, C.; Wu, W.; Xu, C.; Hu, W.; Zhao, D.; and Yan, R. 2019.  One time of interaction may not be enough: Go deep with an interaction-over-interaction network for response selection in dialogues.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 1–11.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NIPS_ , 5998–6008.
  * Welleck et al. (2018) Welleck, S.; Weston, J.; Szlam, A.; and Cho, K. 2018.  Dialogue natural language inference.  _arXiv preprint arXiv:1811.00671_ .
  * Xie et al. (2020) Xie, P.; Cui, Z.; Chen, X.; Hu, X.; Cui, J.; and Wang, B. 2020.  Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism.  In _COLING_ , 15–25.
  * Xing et al. (2017) Xing, C.; Wu, W.; Wu, Y.; Zhou, M.; Huang, Y.; and Ma, W.-Y. 2017.  Hierarchical recurrent attention network for response generation.  _arXiv preprint arXiv:1701.07149_ .
  * Xiong, Merity, and Socher (2016) Xiong, C.; Merity, S.; and Socher, R. 2016.  Dynamic Memory Networks for Visual and Textual Question Answering.  In _ICML_.
  * Xu et al. (2020) Xu, R.; Tao, C.; Jiang, D.; Zhao, X.; Zhao, D.; and Yan, R. 2020.  Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues.
  * Yu et al. (2020) Yu, M.-H.; Li, J.; Liu, D.; Zhao, D.; Yan, R.; Tang, B.; and Zhang, H. 2020.  Draft and edit: Automatic storytelling through multi-pass hierarchical conditional variational autoencoder.  In _AAAI_ , volume 34, 1741–1748.
  * Zhang et al. (2021) Zhang, J.; Cui, Z.; Xia, X.; Guo, Y.; Li, Y.; Wei, C.; and Cui, J. 2021.  Writing Polishment with Simile: Task, Dataset and A Neural Approach.  In _AAAI_.
  * Zhang et al. (2018) Zhang, S.; Dinan, E.; Urbanek, J.; Szlam, A.; Kiela, D.; and Weston, J. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too?  _arXiv preprint arXiv:1801.07243_ .
  * Zhang et al. (2019) Zhang, Y.; Sun, S.; Galley, M.; Chen, Y.-C.; Brockett, C.; Gao, X.; Gao, J.; Liu, J.; and Dolan, B. 2019.  Dialogpt: Large-scale generative pre-training for conversational response generation.  _arXiv preprint arXiv:1911.00536_ .
  * Zhao, Zhao, and Eskénazi (2017) Zhao, T.; Zhao, R.; and Eskénazi, M. 2017.  Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders.  In _ACL_.


# Understanding Telecom Language Through Large Language Models

Lina Bariah  Technology Innovation Institute, 9639 Masdar City, Abu Dhabi, UAE
Email<EMAIL_ADDRESS>Hang Zou  Technology Innovation Institute,
9639 Masdar City, Abu Dhabi, UAE
Email<EMAIL_ADDRESS>Qiyang Zhao  Technology Innovation
Institute, 9639 Masdar City, Abu Dhabi, UAE
Email<EMAIL_ADDRESS>Belkacem Mouhouche  Technology Innovation
Institute, 9639 Masdar City, Abu Dhabi, UAE
Email<EMAIL_ADDRESS>Faouzi Bader  Technology Innovation
Institute, 9639 Masdar City, Abu Dhabi, UAE
Email<EMAIL_ADDRESS>and Merouane Debbah  Technology Innovation
Institute, 9639 Masdar City, Abu Dhabi, UAE
Email<EMAIL_ADDRESS>
###### Abstract

The recent progress of artificial intelligence (AI) opens up new frontiers in
the possibility of automating many tasks involved in Telecom networks design,
implementation, and deployment. This has been further pushed forward with the
evolution of generative artificial intelligence (AI), including the emergence
of large language models (LLMs), which is believed to be the cornerstone
toward realizing self-governed, interactive AI agents. Motivated by this, in
this paper, we aim to adapt the paradigm of LLMs to the Telecom domain. In
particular, we fine-tune several LLMs including BERT, distilled BERT, RoBERTa
and GPT-2, to the Telecom domain languages, and demonstrate a use case for
identifying the 3rd Generation Partnership Project (3GPP) standard working
groups. We consider training the selected models on 3GPP technical documents
(Tdoc) pertinent to years 2009-2019 and predict the Tdoc categories in years
2020-2023. The results demonstrate that fine-tuning BERT and RoBERTa model
achieves 84.6% accuracy, while GPT-2 model achieves 83% in identifying 3GPP
working groups. The distilled BERT model with around 50% less parameters
achieves similar performance as others. This corroborates that fine-tuning
pretrained LLM can effectively identify the categories of Telecom language.
The developed framework shows a stepping stone towards realizing intent-driven
and self-evolving wireless networks from Telecom languages, and paves the way
for the implementation of generative AI in the Telecom domain.

###### Index Terms:

Generative AI, Large Language Models, Pre-trained Transformer, Telecom
Language, 3GPP

TABLE I: BERT fine-tuning: Domains and Data Ref. | Domain |  Data
---|---|---
[1] | General Science |  Scientific publication
[2] | Multi-domain Patents |  Google Patents Public dataset
[3] | General domain Q&A |  Social commenting platforms
[4] | General domain |  Wikipedia articles
[5] | Cross-domain sentiment analysis |  Amazon reviews dataset
[6] | Customers delivery and pickup services (open-domain) |  AG’s News Corpus & TREC dataset
[7] | Authorship attribute |  Public datasets, e.g., IMDb
[8] | Chinese language |  Chinese web news
[9] | Russian language |  OpenCorpora (Russian online media and Russian Wikipedia)
[10] | Arabic language |  Arabic media & news (Modern Standard Arabic (MSA) & Dialectal Arabic (DA))
[11] | Healthcare |  Wikipedia articles
[12] | Telecom Q&A |  TeleQuAD
Our work | Telecom Language Understanding |  3GPP technical documents and specifications

##  I Introduction

In the last couple of decades, considerable efforts have been devoted to push
the frontiers of wireless technologies in order to achieve key performance
indicators pertinent to latency, reliability, spectral and energy
efficiencies, to name a few, through the exploitation of artificial
intelligence (AI) as a network orchestrator. Recently, parallel initiatives
have been focused on advancing the paradigm of self-evolving networks (under
several names including autonomous networks, zero-touch networks, self-
optimizing/configuring/healing networks, etc.), through the evolution of
native intelligent network architecture [13]. However, recent developments are
revolving around realizing adaptivity, in which wireless networks
functionalities can be autonomously adjusted to fit within a particular
scenario. The ultimate vision of self-evolving networks goes way beyond
adaptivity and automation. In particular, it expands toward realizing
perpetual sustainability of network performance and the flexibility to
accommodate highly complex, and sometimes unfamiliar, network scenarios, and
hence, this necessitates generalized, inclusive, and multi-functional schemes
that are capable of handling diverse network conditions. Accordingly,
conventional AI algorithms are highly probable to fall behind in fulfilling
the required performance, and therefore, a radical departure to more
innovative AI-driven approaches is anticipated to shape the future of next
generation wireless networks.

fm was coined by Stanford Center for Research on Foundation Models (CRFM) in
2021 and have attracted a considerable attention as generalized models that
are capable of handling a wide range of downstream tasks [14]. In particular,
foundation models are extremely large neural networks that are trained over
massive unlabeled datasets, in a self-supervised fashion, allowing several
opportunities to be reaped with reduced time and cost (that would be
unbearable in case of human labeling). Rapidly after being developed, FMs have
found their applications in several domains, including text classification and
summarizing, sentiment analysis, information extraction, and image captioning.
While FMs were not aimed to follow a particular model or application,
language-related models, i.e., large language models, are currently one of the
most common subfield of FMs, which rely on the principle of pretraining large
models over a large-scale corpus. Such pretrained large models, e.g.,
bidirectional encoder representations from transformers (BERT) [15] and
generative pre-trained transformers (GPT) [16], can be further fine-tuned in
various downstream tasks, and hence, avoid the cost of retraining large models
from scratch in the new domains.

###  I-A Related Work

Focusing on text generation-related tasks, language models trained on large
corpus can successfully understand the natural language, and create human-like
language responses according to the specific tasks. Several domain-specific
variations of well-known pretrained language models were presented in the
literature to demonstrate the opportunities that can be obtained from domain-
specific fine-tuning and retraining. In [1], the authors proposed SCIBERT, a
BERT-based language model that is fine-tuned to the scientific domain, where
it was trained over corpus from scientific publications. The authors in [2]
have considered fine-tuning BERT model using Google Patents Public datasets to
perform patents classification. Furthermore, generative language model, based
on multiple choice question answering, is fine-tuned using social commenting
platforms in [3], in order to realize zero-shot text classification. From a
different perspective, the authors in [4] proposed a Universal Language Model
Fine-tuning (ULMFiT) approach for fine-tuning generative large models for
enhanced text classification. The proposed scheme in [4] demonstrated reduced
error by 18-24% up to six-class text classification, when tested over general
Wikipedia articles. Cross-domain sentiment analysis through fine-tuning BERT
and XLNet models is proposed in [5], in which the fine-tuned model showed
promising results with less amount of data. The authors in [6] explored
several active learning strategies to adapt BERT model into customers
transactions application to classify transactions to different market-related
categories, for improved market demands understanding. Targeting different
domain, the work in [7] presents BertAA, a framework for BERT fine-tuning for
authorship classification purposes, in which public datasets, e.g., IMDb, are
utilized to refine the BERT model and enable it to extract the characteristics
of authors’ identities from the provided text. The proposed work in [7] showed
5.3% improved in the authorship attribute task. From a language perspective,
multi-lingual and single-lingual frameworks were presented in the literature
to fine-tune/retrain a pre-trained BERT model in order to allow the model to
deal with different languages, e.g. Chinese [8], Russian [9], Arabic [10]. The
presented results in [8]-[10] demonstrated the robustness of BERT as a large
model for different languages. For healthcare applications, the authors in
[11] provided a framework for disease name recognition, where a BERT model,
fine-tuned using data pertinent to disease knowledge, demonstrated an enhanced
performance compared to the literature.

###  I-B Contributions

While the field of domain-specific fine-tuning of large generative models is
very active and several contributions were presented for different domains,
the telecom domain is still almost untouched. We strongly believe that
adapting various large generative models to the telecom domain is a key
building block in the development of self-evolving networks, where such models
can play an essential role through the different stages of designing,
building, and operating wireless networks. The advantages of large Telecom
language models are envisioned to be particularly important with the rise of
generative agents paradigm [17], in which LLM implemented at Telecom networks
will require a comprehensive understanding of the Telecom terminologies, and
their relationship with different network operational and configuration
functions, in order to enable them to communicate meaningfully and to perform
Telecom-specific downstream tasks when implemented in future wireless
networks.

Within this context, in [12], the authors have focused on adapting BERT-like
model to the telecom domain, where the considered model is pretrained/fine-
tuned in order to perform a question answering downstream task within the
telecom domain. Note that the work in [12] is constrained by the small dataset
used (few hundreds of technical documents and web articles), which was
prepared in a manual manner as follows. The data were acquired from technology
specification files of the 3GPP, and it was collected from 347 telecom-related
documents, resulting in 2,021 question-answer pairs only. It is worthy to note
that the dataset used in [12] is not publicly available. For enabling a
holistic understanding of Telecom language, a comprehensive dataset comprising
a wide-range of technical discussion pertinent to different network
operational, configuration, and design parameters need to be generated and
used in the pretraining/fine-tuning process. Motivated by this, in this paper,
we develop a framework for adapting pretrained generative models, including
BERT, DistilBERT, RoBERTa, and GPT-2 models, to the Telecom domain, through
exploiting a huge number of technical documents that consist of technical
specification from 3GPP standard. Among different language models, the
selection of considered models are motivated by the fact that it generates a
contextual representation for each word, while considering previous and
following words, rendering it a well-suited model for technical text
classification.

The main contributions of our work are summarized as follows:

  1. 1.

Create an annotated large Telecom datasets from 3GPP technical specification
of various working group (WG), including technical pertinent to radio
frequency (RF) spectrum usage, network architecture, radio interface
protocols, signaling procedures, and mobility management, network
architecture, system interfaces, security, quality of service (QoS), network
management, routing, switching, and control functions.

  2. 2.

Adapt the pre-trained BERT, DistilBERT, RoBERTa, and GPT-2 model into the
Telecom domain, through fine-tuning the models for 3GPP Tdocs text
classification. The fine-tuned models allow to identify a particular technical
text, related to 3GPP cellular architecture category, i.e., radio access
network (RAN), system architecture (SA), or core network and terminals (CT),
with characterizing the WG corresponding to each category.

The remaining of the paper is organized as follows. In Sec. II we detail the
developed approach to adapt the pre-trained models to Telecom domain,
including solutions on data collection, data pre-processing, and model fine-
tuning. Experimental results with performance analysis are discussed in Sec.
III. Finally, the paper is concluded in Sec. IV.

##  II Method

###  II-A LLMs for Telecom Language Classification

In this work, we use BERT, DistilBERT, RoBERTa, GPT-2 language models, which
are trained on large amounts of unlabeled textual data using self-supervised
or contrastive learning [15]. These models can be adapted to various
downstream tasks via fine-tuning. Specifically, the architecture of BERT and
its variants allows it to understand the context and meaning of words in a
sentence by taking into account the surrounding words on both sides of the
target word. This bidirectional approach helps the pre-trained model to
capture more complex relationships between words and their contextual meaning,
making it a powerful tool for text classification.

The following models are implemented in our work: 1) Pretrained BERT-Base
(uncased): contains 12 layers, 768 hidden units, 12 self-attention heads, and
110M parameters; 2) DistilBERT: a lighter version of BERT-Base (uncased) with
40% less parameters, which is particularly useful for wireless network with
constrained resources; 3) RoBERTa: contains the same architecture as BERT,
with byte-level byte pair encoding (BPE) as a tokenizer is used, which
operates at the byte level instead of the traditional character or subword
levels; 4) GPT-2: the smallest version with 6 layers, 36 hidden units, 48
self-attention heads, and 124M parameters. A linear classification layer with
SoftMax function is added to the pre-trained models to produce the WGs.

In order to adapt the selected models into the desired Telecom domain for the
downstream task of text classification, we consider the single-task single-
label fine-tuning approach [18] for 3GPP Tdoc classification. A cross entropy
loss function is used to update the pre-trained model weights. For efficient
fine-tuning, we employed a batch size of 32, ensuring a balance between
computational efficiency and memory requirements. The learning rate was set to
2e-5, enabling gradual convergence to an optimal solution. To prevent
overfitting, we applied L2 regularization with a rate of 0.01. Also, F1 score
is considered to evaluate the performance of the tuned models.

###  II-B 3GPP Technical Document Dataset

3GPP is the main Standard Developing Organization (SDO) in the area of
Telecommunication. The universal standards for 3G, 4G and 5G have been
developed by 3GPP since 1999. 3GPP works with Tdoc contributed by companies
during the development phase and produces technical specifications as a final
output. The specification work is carried out in Technical Specification
Groups (TSGs). There are three Technical Specifications Groups: RAN, SA, and
CT. Each TSG consists of multiple WG focused on specific areas, ranging from
radio access network specifications, core network specifications, service
requirements and specifications, and architecture and protocols for mobile
communication systems, to QoS and performance requirements, security and
privacy in mobile communication systems, interoperability and compatibility
requirements, network management and operation, and testing and certification
procedures. These topics are further divided into specific subtopics, and each
Tdoc file may focus on one or more of these areas. The content of Tdoc files
is typically technical and detailed, intended for experts and engineers
involved in the development and implementation of mobile communication
systems. Thus, The ability to classify a text into one of the WG requires a
deep understanding of the functions and scope of each group.

In this paper, the technical documents are acquired from the 3GPP website. The
collected files belong to years 2009-2023 and include technical specifications
put by different WGs, including, RAN1, RAN2, RAN3, RAN4, RAN5, SA1, SA2, SA3,
SA4, SA5, SA6, CT1, CT3, CT4, CT6. The Tdoc files are available as ZIP files,
and accordingly Apache Tika application [19] is used to unzip and extract the
information from the files. Table II demonstrates the size of the dataset
acquired from 3GPP WGs, where documents belonging to years 2009-2019 are used
for training, while documents related to years 2020-2023 are exploited for
testing.

Figure 1: Pipeline for Adapting LLM to 3GPP Technical Language TABLE II: 3GPP Tdoc files in different WGs and years WG |  Train (’09-’19) |  Train (’15-’19) |  Test (’20-’23)
---|---|---|---
RAN1 | 83905 | 56969 | 39230
RAN2 | 82853 | 55189 | 38744
RAN3 | 36651 | 22981 | 20219
RAN4 | 43845 | 28953 | 43482
RAN5 | 50812 | 31706 | 32456
SA1 | 19497 | 11282 | 10086
SA2 | 64065 | 42931 | 43860
SA3 | 19546 | 13903 | 13815
SA4 | 6583 | 1776 | 5128
SA5 | 28044 | 15031 | 13040
SA6 | 9010 | 9010 | 10360
CT1 | 30990 | 20840 | 18910
CT3 | 22269 | 12734 | 12584
CT4 | 28245 | 14731 | 13109
CT6 | 4446 | 2213 | 1571
Total | 520761 | 340244 | 316594

###  II-C Data Pre-Processing

We pre-process the 3GPP Tdoc files via following steps:

  1. 1.

Parse the HTML tags in the text and return the text content without any HTML
tags using BeautifulSoup.

  2. 2.

Remove any URLs (web links) from the text: identify the regex pattern that
matches URLs starting with either ”http” or ”https” and may include
alphanumeric characters, special characters, and encoded characters.

  3. 3.

Remove tables from the parsed HTML document using BeautifulSoup.

  4. 4.

Divide each document into multiple text segments with different number of
words extracted from natural language toolkit (NLTK). This allows us to
evaluate the model’s capability of understanding technical descriptions in
different lengths.

  5. 5.

Remove headers, footers, captions, and pseudo codes, while ensuring each
paragraph doesn’t exceed a particular maximum length. Also, we eliminate the
references section and all the text afterward.

  6. 6.

Remove change requests, drafts, templates due to their limited technical
information.

##  III Experiment Results and Discussions

In this section, we present experimental results to demonstrate the accuracy
of the fine-tuned LLMs in understanding and classifying technical text within
the Telecom domain. We split 3GPP Tdoc into training, validation, and test
datasets. Specifically, the test set contains textual segments of Tdocs from
2020 to 2023 (April). The training and validation sets contains: 1) Tdocs from
2010 to 2019 (’10-’19); and 2) Tdocs from 2015 to 2019 (’15-’19). The
proportion of these two datasets is 80% and 20%. The number of words within a
textual segment in training, validation and test set is $200$ in what follows
without further mentioning.

We start by comparing the performance of different LLMs fine-tuned with 3GPP
files from 2015 to 2019 in terms of classification accuracy as illustrated in
Fig. 2. The selected models have the following sizes, BERT (117M), RoBERTa
(125M), GPT-2 (124M) and DistilBERT (66M). Considering 100% of the files,
while it can be observed that all models experience relatively close accuracy,
GPT-2 model encounters the weakest performance. This is motivated by the fact
that for text classification, it is important for the large model to have
concise and interpretable predictions features rather than generative
capabilities, where the latter is the key element of GPT-2. Meanwhile,
RoBERTa, the optimized version of BERT, demonstrates the strongest
performance. It can be noticed further that the performance gap increases as
the number of Tdoc files decreases.

Figure 2: Classification accuracy of different LLMs vs. portion of 3GPP files
used for fine-tuning.

In Fig. 3, we evaluate the accuracy of prediction and the receiver operating
characteristic - area under the curve (ROC-AUC) as a function of the portion
of textual segments used for fine-tuning a BERT model. We can observe that a
BERT model fine-tuned to 3GPP files from 2015 to 2019 can achieve an accuracy
performance around $80\%$ even if only $20\%$ of the text segments are used.
Furthermore, although fine-tuning to Telecom language is essential, it can be
demonstrated that Tdoc files from recent years are sufficient to provide the
needed accuracy. On the other hand, when the number of files is relatively
small (below $10\%$), data 2010-2015 produce better understanding of the
Telecom technical language.

Figure 3: Classificaiton accuracy and ROC-AUC vs. portion of 3GPP files used
for fine-tuning BERT (’10-’19 and ’15-’19).

The role of 3GPP WGs vary from one to another. Therefore the structure of
files and especially the number of files available differs distinctly. For
examples, RAN1 and RAN2 contains much more files than other WGs given that
they are main categories in RAN (specifying PHY, MAC, RLC, PDCP layers), and
hence, more activities pertinent to these layers are conducted within these
two groups. To show the impact of different WGs on the performance of the
classification, the accuracy of a BERT model fine-tuned by 3GPP files is
illustrated in Table III for different combinations of WGs. It can be noticed
that the fine-tuned model can achieve better classification accuracy for
textual segment among RAN1, SA1 and CT1 than the combination of RAN1, RAN2 and
RAN3. This is stemmed from the fact that Tdoc files belonging to different
category number but fall within the same TSG are highly correlated, and
therefore, the probability of error is higher. In contrast, technical files
within different TSGs comprises relatively uncorrelated topics, and therefore,
the model has a higher capability to distinguish between these different TSGs.
The presented results in Table III reveal that the test accuracy is determined
mainly by the documents of RAN1, RAN2 and RAN3.

TABLE III: Classification accuracy on different WG combinations from BERT RAN | SA | CT | Accuracy ($\%$)
---|---|---|---
1 | 1 | 1 | 98.05
1,2,3 | None | None | 88.90
1,2,3 | 1 | 1 | 88.26
1,2,3,4 | 2,5 | None | 87.42
1,2,3 | 1,2,3 | 1,3,4 | 86.61
1,2,3,4 | 1,2,3,4 | 1,3,4,6 | 85.57
1,2,3,4,5 | 1,2,3,4,5,6 | 1,3,4,6 | 84.35

In Fig. 4 we evaluate the impact of the length of technical text segments to
the accuracy of classification. This is a critical aspect to be studied, as it
is important to know the minimum amount of text required by the tuned LLMs to
realize accurate identification of the technical groups. We set the maximum
number of words for training and validation to $200$ and we vary the number of
words during the testing process. We can observe that the accuracy increases
as the number of words grows. However, performance enhancement difference
starts to decrease as well as the number of words increases, indicating the
significant role of selecting the optimum size for a LLM, in order to strike a
balance between performance and computing complexity.

Figure 4: Classification accuracy vs. maximum text segment length in words
produced from fine-tuned BERT

##  IV Conclusion

Motivated by the promising potentials of LLMs, in this paper, we proposed a
framework for 3GPP technical documents identification, where we leveraged pre-
trained language models, fine-tuned using 3GPP data, in order to allow the
model to identify the 3GPP specification categories with the corresponding
working group. In more details, we have considered BERT, DistilBERT, RoBERTa,
and GPT-2 models, in which they are fine-tuned using 3GPP Tdocs belonging to
TSGs, namely RAN, SA, and CT. The obtained results demonstrate the
applicability of adapting a pre-trained language model into the Telecom
domain, where all fine-tuned models showed accurate classification performance
under different scenarios. It is important to emphasize the significance of
developing LLMs that are capable of understanding the Telecom language, as a
cornerstone to enable autonomous networks driven by intelligent generative
agents.

## References

  * [1] I. Beltagy, K. Lo, and A. Cohan, “SciBERT: A pretrained language model for scientific text,” _arXiv preprint arXiv:1903.10676_ , 2019.
  * [2] J.-S. Lee and J. Hsiang, “Patent classification by fine-tuning BERT language model,” _World Patent Information_ , vol. 61, p. 101965, 2020.
  * [3] R. Puri and B. Catanzaro, “Zero-shot text classification with generative language models,” _arXiv preprint arXiv:1912.10165_ , 2019.
  * [4] J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” _arXiv preprint arXiv:1801.06146_ , 2018.
  * [5] B. Myagmar, J. Li, and S. Kimura, “Cross-domain sentiment classification with bidirectional contextualized transformer language models,” _IEEE Access_ , vol. 7, pp. 163 219–163 230, 2019.
  * [6] S. Prabhu, M. Mohamed, and H. Misra, “Multi-class text classification using bert-based active learning,” _arXiv preprint arXiv:2104.14289_ , 2021.
  * [7] M. Fabien, E. Villatoro-Tello, P. Motlicek, and S. Parida, “BertAA: BERT fine-tuning for authorship attribution,” in _Proceedings of the 17th International Conference on Natural Language Processing (ICON)_ , 2020, pp. 127–137.
  * [8] X. Chen, P. Cong, and S. Lv, “A long-text classification method of Chinese news based on BERT and CNN,” _IEEE Access_ , vol. 10, pp. 34 046–34 057, 2022.
  * [9] K. Lagutina, “Topical text classification of Russian news: a comparison of BERT and standard models,” in _Conference of Open Innovations Association (FRUCT)_. IEEE, 2022, pp. 160–166.
  * [10] W. Antoun, F. Baly, and H. Hajj, “Arabert: Transformer-based model for Arabic language understanding,” _arXiv preprint arXiv:2003.00104_ , 2020\.
  * [11] Y. He, Z. Zhu, Y. Zhang, Q. Chen, and J. Caverlee, “Infusing disease knowledge into BERT for health question answering, medical inference and disease name recognition,” _arXiv preprint arXiv:2010.03746_ , 2020.
  * [12] H. Holm, “Bidirectional encoder representations from transformers (BERT) for question answering in the telecom domain.: Adapting a BERT-like language model to the telecom domain using the ELECTRA pre-training approach,” 2021\.
  * [13] C.-X. Wang, X. You, X. Gao, X. Zhu, Z. Li, C. Zhang, H. Wang, Y. Huang, Y. Chen, H. Haas _et al._ , “On the road to 6G: Visions, requirements, key technologies and testbeds,” _IEEE Commun. Surveys Tuts._ , 2023.
  * [14] R. Bommasani _et al._ , “On the opportunities and risks of foundation models,” _arXiv preprint arXiv:2108.07258_ , 2021.
  * [15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [16] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [17] J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” _arXiv preprint arXiv:2304.03442_ , 2023.
  * [18] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?” in _Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18_. Springer, 2019, pp. 194–206.
  * [19] “Apache tika - a content analysis toolkit,” https://tika.apache.org, accessed: 2023-05-04.

  *[3GPP]: 3rd Generation Partnership Project
  *[Tdoc]: technical documents
  *[AI]: artificial intelligence
  *[FMs]: foundation model
  *[BERT]: bidirectional encoder representations from transformers
  *[GPT]: generative pre-trained transformers
  *[LLM]: large language model
  *[WG]: working group
  *[RF]: radio frequency
  *[QoS]: quality of service
  *[Tdocs]: technical documents
  *[RAN]: radio access network
  *[SA]: system architecture
  *[CT]: core network and terminals
  *[LLMs]: large language model
  *[BPE]: byte pair encoding
  *[WGs]: working group