# Curricular Transfer Learning for Sentence Encoded Tasks

Jader Martins Camboim de Sá1,2,*, Matheus Ferraroni Sanches2,
Rafael Roque de Souza2, Júlio Cesar dos Reis2, Leandro Aparecido Villas2
1 Luxembourg Institute of Science and Technology (LIST)
2 Institute of Computing - University of Campinas (Unicamp)
<EMAIL_ADDRESS>
###### Abstract

Fine-tuning language models in a downstream task is the standard approach for
many state-of-the-art methodologies in the field of NLP. However, when the
distribution between the source task and target task drifts, e.g.,
conversational environments, these gains tend to be diminished. This article
proposes a sequence of pre-training steps (a curriculum) guided by “data
hacking” and grammar analysis that allows further gradual adaptation between
pre-training distributions. In our experiments, we acquire a considerable
improvement from our method compared to other known pre-training approaches
for the MultiWoZ task.

##  1 Introduction

Traditional Machine Learning (ML) technology is influenced by the assumption
that a difference in data distribution between training and real-world data
can result in a degradation of the predictive learner Shimodaira (2000). To
overcome the effects of data divergence, studies have developed several
algorithms that have some regularizing hyper-parameter to mitigate the
variance in the hypothesis space and learn computable functions that have
great generality to out-of-distribution data Sarker (2021).

These traditional methods have been successfully applied in many practical
applications, but they still present limitations for specific real-world
scenarios where we have complex data like images or text Bengio et al. (2007).
Deep learning overcame a few of these limitations, acquiring near-human
performance in various tasks. Whereas traditional ML has an explicit bias to
control over-fitting, deep models need to acquire its bias from data, which
generally could be expensive or even unattainable in some scenarios.

Transfer learning was introduced to achieve high performance in low-resource
domains by artificially increasing available data from a different domain to
overcome these limitations Ruder (2019). This technique consists in pre-
training a model with high variance in domains with several available
examples, so those learning models can acquire suitable biases that generalize
for other tasks, including tasks where little to no examples are available
Howard and Ruder (2018).

In Natural Language Processing (NLP), the standard approach for transfer
learning is language modeling, which consists in pre-training a model to
denoise a sentence, and then fine-tuning it to the target task where the model
is applied Devlin et al. (2018). This self-supervised sequential transfer
learning enables the model to learn high-level linguistic features and
statistical properties of language that help the model to generalize for many
downstream tasks. Exploiting these capabilities, recent studies encode tasks
as a pure sequence-to-sequence problem Raffel et al. (2020) to directly solve
it with auto-regressive language modeling task Brown et al. (2020).

Task-Oriented Dialog Systems (TODS) are conversational agents designed for
solving business-related tasks by interacting with customers through a
conversation in a natural language interface. Its task is to solve sub-
problems related to business attendance — identify intentions, and entities,
decide which action to take, and generate a system response. As a natural
language process, TODS could benefit significantly from the general language
capabilities of Language Models Fellows et al. (2021). The academy has
explored encoding TODS as a sequence-to-sequence problem Lin et al. (2020);
Hosseini-Asl et al. (2020); Yang et al. (2020) to improve significantly the
effectiveness of agents in unrestricted domains. Figure 1 illustrates an
example of how this process occurs. The general task is composed of four
steps; first, it receives a user utterance (denoted by su tag); then it
identifies intents and entities in the utterance; the belief state (denoted by
sb tag); then it decides which action to take (sa tag); finally it generates a
response for this action to return to the user.

Figure 1: Operational diagram of TODS based on LMs. The task is encoded as a
sequence-to-sequence problem to later be decoded and parsed for system
communication.

In Figure 1, the interactions occur in the following manner: a) we first
represent every step in textual symbols, and each field is marked by the start
and end of tags; (b) we concatenate the current utterance with the fields from
previous turns in the dialog; (c) we feed the whole dialog history to the
auto-regressive model to generate the following steps (belief state, action,
and response). (d) the generated fields are parsed, then used for querying
information; later we return the response to the user as shown in (e).

The TODS task encoded as sequence-to-sequence presents an unusual structure
that is not present in the text used for pre-training language models. The
encoded sentence presents a regular grammar, for the meta-structure encoded by
unique tag tokens. Utterances and responses have a conversation distribution,
so it is not the same distribution of words of a general text domain Zhang et
al. (2020b). We have a classification task, where the model aims to infer user
intent given his utterance. Then, a Named Entity Recognition (NER) and parsing
task, where the model has to extract and format slot-values, and it has a
policy that is trying to infer that will shape the response Yang et al.
(2020); Hosseini-Asl et al. (2020).

General language modeling is well suited for many NLP tasks. However, it still
requires fine-tuning on large datasets for tasks with complex behavior, like
textual distributions that diverge a lot from source training Liu et al.
(2021b); Ouyang et al. (2022); Nakano et al. (2021), or domain-specific
applications Wu et al. (2020); Zhang et al. (2020b); Thoppilan et al. (2022).
For TODS, given the sequential nature of decisions, a high error is not
permissible, as it propagates to the following interactions in a dialog, and
many examples in the downstream task are needed for proper behavior of
conversational agents Fellows et al. (2021).

Although we could build conversational datasets for our TODS application, this
specialized data is expensive Zhang et al. (2020c), and small businesses could
not afford to build massive datasets. Facing the problem of adaptation with
low-resource in the downstream task, we should learn a proper bias to allow
the rapid adaptation of our learning model to the drift in downstream data, so
the final model should require only a fraction of the usual needs.

Inspired by psycho-pedagogical and optimization literature, a curriculum
approach presents examples to a learner that gradually increases in complexity
or difficulty Bengio et al. (2009). This means the learner could focus on
acquiring the basic skills in the initial steps that help to learn later posed
examples. By the same principle, we could guide the learner through this
learning process by presenting not only examples, but new tasks with
increasing complexity that are somehow related Green (2005).

In this study, we propose to have intermediate tasks in that we can more
easily acquire training data, and this data simulates one or more properties
present in the final task. By having another pre-training step to acquire
intermediate skills, the model would require fewer fine-tuning steps for the
final task, being less prone to over-fit, and acquiring a better
generalization. In our approach, we present a transfer learning “curriculum”,
where we gradually adapt our LM by posing it to intermediate pre-training
tasks. In initial tasks, we have an infinite number of examples; as the
complexity of the task increases, the available data decreases exponentially.
However, our initial tasks adjust the bias for the LM such that it can learn
more easily those more complex examples.

We use forum structure to simulate the regular grammar and the word
distribution in utterances and responses, and the topic being discussed to
simulate the intent classification task. This approach significantly improves
BLEU, SUCCESS, and INFORM, compared to other pre-training approaches for TODS.
In this article, we contribute with the following:

  * •

We propose a new curriculum learning method where the complexity of learning
varies at a sentence task level instead of an instance level. This approach
allows for gradually fine-tuning scaled LMs to small datasets without over-
fitting.

  * •

As a medium for intermediate training steps, we propose a method for
constructing pseudo-supervised data in the context of conversational agents
that encompass the main problems in the target task, forcing the LM to meta-
learn behaviors that help to solve the final target task.

The remaining of this article is organized as follows: Section 2 presents
background studies. Section 3 formalizes our Curricular Transfer Learning
proposal and presents our approach to solving the TODS task. Section 4
presents our conducted experimental evaluation. Section 5 discusses the
results and Section 6 draws conclusions from our study.

##  2 Background

Our research work is heavily inspired by two key studies addresing curriculum
learning and transfer learning. We review studies applied to TODS, but also in
general machine learning literature.

Curriculum Learning. Curriculum learning was initially proposed for next word
prediction Bengio et al. (2009), and later extended for many NLP tasks Soviany
et al. (2021). In the majority of the studies the curriculum is applied in an
instance level Liu et al. (2021a); Kim et al. (2021); Dai et al. (2021); Zhu
et al. (2021b). Other set of studies, the task is adjusted with a gradual
change in the task, where the complexity increases based on length Foglino and
Leonetti (2019) or modeLiu et al. (2020).

In the context of TODS, some studies apply a curriculum technique in
Reinforcement Learning (RL) context, e.g., sort dialogue instance based on
number os slots to track Saito (2018), dialog reward information as a measure
of complexity Zhu et al. (2021a); Liu et al. (2021a), or dialog metrics are
used to sort instances in the curriculum Zhao et al. (2022); Dai et al.
(2021).

Transfer Learning. Literature of end-to-end TODS recently presented studies
focusing on rapid adaptation or learning with low resources. Most
investigations focus on heuristics for rapid adaptation to data, like
variational inference Zhang et al. (2020a), optimizing editing distance Lin et
al. (2020), multi-tasking Kulhánek et al. (2021); Su et al. (2021), or
denoising Sun et al. (2022).

For the conversational literature, there is a set of studies that explore
exclusive “conversation-biased” data for pre-training language models. Some
works explore self-supervision on raw text from Reddit Wu et al. (2020); Zhang
et al. (2020b), while other present the text as a simple question-answer pair
Adiwardana et al. (2020); Thoppilan et al. (2022).

We summarize our contribution as follows: In the curriculum learning
literature for NLP, no study employs a curriculum at a dataset level. For the
transfer learning literature, some studies apply pre-training on previous
general conversation-oriented data, but do not exploit massive pseudo-
supervised data as one of the steps of pre-training.

Our solution aims to reduce the requirements for annotated data in training
sequence-to-sequence tasks. We manipulate the existing structure in web-data
to create intermediate tasks that simulate the same sequence structure present
in the target task, which we call “data hacking”. This approach helps the
language model to acquire a better bias for the final task. We create data
that resembles a simplified version of the target grammar we want our model to
acquire, so it can learn basic properties of the grammar and later generalize
to more complex instances. In the next section, we present our proposal for
complex grammar acquisition by grammar decomposition and instantiate the
solution for the problem of TODS.

##  3 Grammar Acquisition with Curricular Transfer Learning

The classical transfer learning framework, using sequential induction, allows
us to use out-of-distribution data from a source task to leverage the
knowledge in this data for another correlated target task. By learning the
proper bias for the problem to be modeled, the learner model can rapidly adapt
to the target hypothesis, demanding fewer examples to acquire a good
generalization point in the target domain. In the context of NLP, transfer
learning is achieved by pre-training an LM, with self-supervision, in texts
sampled from indexed pages on the web Devlin et al. (2019); Radford et al.
(2019); Raffel et al. (2020), generally with diverse contents.

Although evidence shows that language modeling acquires high generalization
capabilities for the out-of-distribution data Hendrycks et al. (2020), some
specialized and complex tasks that diverge from the general distribution of
random web text are less benefited Mielke et al. (2019); Cotterell et al.
(2018); Zellers et al. (2019); Delétang et al. (2022); Thoppilan et al.
(2022). In low-resource contexts, those pre-trained models tend to overfit or
underfit the target task, failing to arrive at a desirable optimum and
presenting poor generalization.

In the case of TODS, the sequence we want to model has a conversation-biased
distribution of words, many unique marking tokens for the classification,
named entity recognition, and parsing. This drift between pre-training and
fine-tuning makes it difficult for our agent to learn and acquire optimal
effectiveness Zhang et al. (2020b). Exploring other pre-training data that
resemble the same properties in the TODS task could significantly improve the
learner.

We observe the problem of training a model from two perspectives: 1) the
learning perspective, where we aim to obtain the model to acquire the pattern
recognition skills; 2) the optimization aspect, where we expect that the model
minimizes some loss function. As posed by (Bengio et al., 2009), a curriculum
approach has a beautiful interpretation and practical application for both
perspectives. In our approach, we teach a model in the same way humans learn
by gradually increasing the task’s difficulty, so it first acquires the basic
skills to thrive in more complex scenarios. Also, relying on this curriculum
approach, we might minimize a less noisy version of the original problem to
arrive at the global optima, a continuation method.

For human language acquisition, the generative grammar theory assumes that a
learner has an innate universal grammar that restricts what kind of grammar a
learner could acquire White (2003). The process starts by recognizing simple
structures in a grammar; instead of memorizing, the learner identifies
syntactic structures they encounter and evaluates the feedback in an
environment to determine precisely the grammar being used Guasti (2004).

Whereas most of a child’s primary language is obtained culturally, without
directed control, for second language acquisition the process is almost
entirely performed in a controlled environment. In the initial stages, the
learner recognizes simple grammatical structures; which is crucial for
complete grammar acquisition, as many possible candidate grammars are filtered
in this search step Komarova et al. (2001). By exploiting this universal
grammar, the learner could quickly acquire the grammar of the language it is
inserted.

Several approaches in Artificial Intelligence (AI) explore the advantages of
posing easier instances of the problem or addressing sequentially surrogate
objectives to achieve more complex goals. In RL, robotic hand manipulation is
an example. Instead of directly training the robot to put a red box over a
blue box, it is easier first to teach the arm to recognize which color to
pick, then how to hold the box correctly, and later how to place it over the
blue box Manela and Biess (2022). This decomposition of complexity allows
agents to learn the final task faster than directly attempting to perform the
final task.

In the context of sequence-to-sequence modeling, we can view the problem as a
grammar acquisition process, starting from basic construction to then
acquiring more complex ones. In our developed solution, we first present the
learner with basic grammar with a more simple composition, to later extend for
more complex elements Davies (1980); Guasti (2004).

We present a formal definition for the problem of grammar acquisition. We
extend the model of (Pan and Yang, 2009) for the multi-sequential case which
is an instance of the curriculum learning model.

First, given a domain $\mathcal{D}=\\{\mathcal{X},P(X)\\}$ consisting in a
feature space $\mathcal{X}$ and a probability distribution $P(X)$, where
$X=\\{x_{1},x_{2},...,x_{n}\\}\in\mathcal{X}$. Consider a task
$\mathcal{T}=\\{\mathcal{Y},f(\cdot)\\}$, where $\mathcal{Y}$ is the label
space, and $f(\cdot)$ is the objective predictive function, which is not
observed but could be learned from the data, where
$f:X\rightarrow\mathcal{Y}$.

We define transfer learning as, given a source domain $\mathcal{D}_{S}$, a
source task $\mathcal{T}_{S}$, and target domain $\mathcal{D}_{T}$ with a
target task $\mathcal{T}_{T}$. Transfer learning aims to help improve the
learning of the target predictive function $f(\cdot)_{T}$ in $\mathcal{D}_{T}$
using the knowledge in $\mathcal{D}_{S}$ and $\mathcal{T}_{S}$, where
$\mathcal{D}_{S}\neq\mathcal{D}_{T}$ or $\mathcal{T}_{S}\neq\mathcal{T}_{T}$.

Although most studies in NLP literature apply a single step of sequential
transfer learning, we propose many steps of transference in the Curricular
Transfer Learning (CTL). It is a curriculum of transference, e.g.,
$\boldsymbol{\mathcal{T}}=\\{\mathcal{T}_{S_{0}},\mathcal{T}_{S_{1}},...,\mathcal{T}_{S_{n}},\mathcal{T}_{T}\\}$,
where we have $n$ pre-training source tasks $\mathcal{T}_{S}$. By this
approach, the $\mathcal{T}_{S_{k}}$ source task helps optimizing for
$\mathcal{T}_{S_{k+1}}$ task, acquiring a better generalization for the
following task and, consequently, the general curriculum as an accumulation of
improvements.

Similar to a curriculum learning approach, consider the instances from
$\mathcal{T}_{S_{k}}$ as smoothed objectives of $\mathcal{T}_{T}$. The cost
function $C_{S_{0}}$ is easier to optimize as it does not compass the entire
TODS objective, and we have more available instances in this set. After
optimizing for this objective, we gradually increase the task complexity by
passing tasks with more linguistic features to extract from $C_{S_{0}}$ to
$C_{S_{n}}$; and finally $C_{T}$, fewer instances are presented for more
complex sets.

In the language modeling problem, the cost function we aim to minimize is the
cross-entropy for the next token. When the $\lambda$ family is composed of
languages with increasingly complex grammar for a cost function $C_{0}$ that
resembles some other target cost function $C_{1}$ for some family
$C_{\lambda}(\theta)$ of cost functions, the $\lambda$ factor varies between
instances according to each data set.

Now, suppose a serialized decoding is learned to a specific pattern. In that
case, for instance, the regular grammar of starting and ending fields,
adapting to a new grammar that shares the same pattern, will be more
straightforward. So the probability of the token to come after <eos_u> is
previously learned in a more manageable task, so the optimization in this
phase should only focus on learning the classification task.

Considering that our final task is composed of formal grammar. Suppose another
data source could simulate some derivation nodes in the target grammar without
loss in the global structure, e.g., the first production rule. In that case,
we pose this simpler version of the grammar as a step in the pre-training
curriculum, and recursively, append more simplified grammars that simulate
previous ones.

We formally call it a curricular transfer learning if, given an ordered set of
sequence-to-sequence tasks
$\boldsymbol{\mathcal{T}}=\\{\mathcal{T}_{S},\mathcal{T}_{I_{1}},...,\mathcal{T}_{I_{n}},\mathcal{T}_{T}\\}$,
some complexity ordering $<_{\mathcal{C}}$, and some grammatical similarity
$\sim_{G}$. Every task in $\boldsymbol{\mathcal{T}}$ respects the order

$\mathcal{T}_{a}<_{\mathcal{C}}\mathcal{T}_{b}\hskip 30.00005pt\forall
a,b:a<b$ (1)

where $a$ and $b$ are indexes for tasks in $\boldsymbol{\mathcal{T}}$ and

$\mathcal{T}_{a}\sim_{G}\mathcal{T}_{b}\hskip 30.00005pt\forall a,b.$ (2)

The main appeal of this method is that more accessible instances of the
problem have a derisive cost to obtain, whereas final instances have a
considerable cost. The artificially created intermediate tasks should help the
LM to meta-learn the final task, like learning unique tokens to classify,
perform a name entity recognition, parsing, or generate a response. We
describe the method for complex grammar acquisition in the following manner:

  1. 1.

Describe the desired grammar formally to be learned, e.g. the generation rule
for the tokens, listing the expected types of text distribution and regular
structure.

  2. 2.

Find text and structures that can be morphed to simulate this behavior. For
the same kind of word distribution in the environment (formal, conversational,
etc.), some NLP tasks are present in the problem, like classification or NER,
etc.

  3. 3.

Recursively proposes (1) that address the subsequent grammar as another pre-
training step.

##  4 Experimental Evaluation

We present how we developed our proposed curricular transfer learning steps
for the case of MultiWoZ. A general intuition for creating intermediate tasks
is to find textual corpora or data sets that we transform into a form that
resembles properties from final grammar. We demonstrate how we applied our
proposal to improve the optimization and overall effectiveness in MultiWoZ, a
widely adopted framework for study and evaluation of TODS. We present a
curriculum decomposition with a meta-structure and a single additional task
and we compare it with previous pre-training approaches.

###  4.1 Protocol

For the conversation-oriented curriculum, we consider a diverse corpus in the
first stage to meta-learn general language skills; a corpus that simulates the
conversational patterns and distributions of words; and finally, the data set
for the target task. We use the following text data sets as pre-training
steps:

  1. 1.

Common Crawl (General Language Modeling) This dataset was proposed as the
unsupervised auto-regressive pre-train for the proposed architecture Radford
et al. (2019). Although we do not train the GPT-2 from scratch in this
dataset, we describe it here to illustrate the CTL approach.

  2. 2.

Tripadvisor (Dialog Modeling) We built this data set by crawling major cities
discussed in Tripadvisor (Paris, Rome, Istanbul, Barcelona, Madrid, Amsterdam,
Lisbon, and London), for each thread we encoded the thread creator with user
utterance tokens, the city as the belief tokens, and thread replies as system
response. We replicate each pseudo-user utterance for each pseudo-system
response. The crawler is available in
Github111https://github.com/jadermcs/tripadvisor-dialogues.

  3. 3.

Multi-Domain Wizard-of-Oz (Task-Oriented Modeling) The MultiWoZ is a general
domain task-oriented dialogues data set collected in a Wizard-of-Oz setting.
It comprises 10000 dialogues in domains like ‘restaurant,’ ‘train,’ and ‘hotel
booking.’ We selected the latest official data set for our evaluation, version
2.2.

To our LM learn how to solve this problem, we need to encode it as a sequence-
to-sequence prediction problem. To this end, we use the special <sos_X>, where
$X\in{u,b,a,r}$ for the start and end of the string in utterance, belief,
action, and response tokens. The example below represents the tokens dialog
from MultiWoZ, encoded as a sequence-to-sequence problem:

<sos_u> I am looking for [...] <eos_u> <sos_b> [restaurant] pricerange cheap
area centre <eos_b> <sos_a> [inform] choice [request] food <eos_a> <sos_r>
There are [value_count] restaurants [...] <eos_r> <sos_u> ...

We first formalize the target grammar to enlist known properties of this
sequence. We notice that we have a regular grammar for demarking field tokens,
in long sequences if our model is unable to predict this simple pattern we
might fail to parse the decoded sequence.

As the training example is the concatenation of the entire dialog session, the
language of this grammar should be any form in
$\mathcal{L}=\\{\epsilon,UBAR,UBARUBAR,...\\}$, cycling infinitely in the
regular pattern.

For each field in our language $\mathcal{L}$, we have a grounded theory for
linguistic patterns that are present. Fields $u$ and $r$ follow a dialog act
distribution Traum and Hinkelman (1992); Bunt (2006). We have that string $w$
in $u$ is sampled from $u(w)$ an utterance distribution. The string $w$ in $R$
is sampled from a response distribution of words $r(w)$. <se*> are delimiter
tokens for each segment.

In the $b$ field, we have a classification task, and a NER with parsing as a
sequence problem so $k\subset K_{b};i\subset I_{b}$. In $a$, our model aims to
learn a policy optimization Williams et al. (2016), $k\subset K_{a};i\subset
I_{a}$. Table 1 illustrates the grammar our model aims to acquire.

Table 1: Regular grammar for TODS encoding in a language modeling problem. $S$ |  $\rightarrow UBARS$ | $\epsilon$
---|---
$U$ |  $\rightarrow$ <su>$w_{u}\sim u(w_{u})$<eu>
$B$ |  $\rightarrow$ <sb>$k\subset K_{b};i\subset I_{b}$<eb>
$A$ |  $\rightarrow$ <sa>$k\subset K_{a};i\subset I_{a}$<ea>
$R$ |  $\rightarrow$ <sr>$w_{r}\sim r(w_{r})$<er>

If we simulate nodes $U$, $B$, and $R$ we obtain a candidate grammar for an
intermediate task. To obtain this data, we crawled conversations from online
forums where a user has a centered topic discussion and starts a thread by
posing a question. Later, users can join the discussion by replying to the
thread creator or other messages.

We manipulate these data to resemble our grammar in the following manner: we
set the message that creates the thread as an utterance and the topic of this
conversation as the classification problem. We put an empty string for action,
and for the response, we pick an answer in the thread. We replicate this
pattern for every message replying to the original topic creator. Finally, we
append the unique tokens to encode it. This process is fully automated by a
script using the HTML annotation from scraped data. We present the template
below:

<sos_u> i’ll be in amsterdam for a week [...] <eos_u> <sos_b> amsterdam
<eos_b> <sos_a> <eos_a> <sos_r> take your pick [...] <eos_r>

Table 2 illustrates the formal version of our fabricated grammar that
simulates an intermediate transfer step.

Table 2: Regular grammar for simulating the distribution of the target grammar. $S$ |  $\rightarrow UBARS$ | $\epsilon$
---|---
$U$ |  $\rightarrow$ <su>$w_{u}\sim u(w_{u})$<eu>
$B$ |  $\rightarrow$ <sb>$\boldsymbol{k\subset K_{c}}{\color[rgb]{.5,.5,.5};i\subset I_{b}}$<eb>
$A$ |  $\rightarrow$ <sa>${\color[rgb]{.5,.5,.5}k\subset K_{a};i\subset I_{a}}$<ea>
$R$ |  $\rightarrow$ <sr>$w_{r}\sim r(w_{r})$<er>

In gray, we show the removed elements, and in bold pseudo-elements. The final
step is to randomly concatenate the above template with random messages from
different discussion topics, so it can generalize the regular pattern.

To better evaluate our transference curriculum, we proposed four different
training settings. The first is the used in (Yang et al., 2020). The second
approach is to pre-train it in a conversational distribution without any
special encoding. Finally, the last curriculum encodes all the grammar in
Table 2. We specify the curriculum details in the following:

  1. 1.

The first curriculum named “gpt-*/multiwoz,” uses pre-trained weights from a
GPT-2 from HuggingFace.

  2. 2.

The second curriculum named “gpt-*/noencode/multiwoz,” starts with GPT-2
weights, then we train it on TripAdvisor data, with the random ordering of the
texts and no special encoding.

  3. 3.

The third curriculum named “gpt-*/encode/multiwoz,” is the fully encoded task
modeling utterance, response, and classification task for the pre-training.

For all curriculums that include TripAdvisor, we pre-process the text using
the same script for MultiWoZ. We train this architecture in the same strategy
as Yang et al. (2020), where all the dialog session is presented to the model
in a single sequence. The training sequence should contain at most 256 tokens,
sessions with more than this maximum length we split into the following
sequence. We train the models with early stopping with a plateau of 5 steps.

For the inference stage, we decode tokens with a sampling strategy normalizing
with $\tau=0.7$ until we reach an end-of-response token. We give belief state
and action from the oracle and just decode the response with the slot-values
tokens.

We evaluate the proposed curriculum under two perspectives: (i) the running
loss for each curriculum variation; and (ii) the standard metrics for the
MultiWoZ data sets with oracle values for database and action, computed by
Nekvinda and Dušek (2021) library.

In the first analysis, we investigate how the initialization proposed by each
curriculum helps in the optimization process for minimizing the empirical loss
for the target grammar. The second analysis measure, given each curriculum,
how this improved minimization translates into better agent performance for
the TODS task. In this step, we compute using BLUE, INFORM, and SUCCESS.

###  4.2 Results

Our evaluation consists of two experiments, curriculum convergence, and
optimality. For convergence, we vary the model size and evaluate the
scalability and time required to stop training in the curriculum.

Figure 2 presents the loss for the validation split over time. We observe that
the curriculum with the pseudo-tods task has a significantly better starting
point for the MultiWoZ task and converges to a lower loss than the other
curriculums. For the small architecture, it converges faster.

Figure 2: Loss during training on validation split of MultiWoZ. Each line
represents one curriculum configuration. The bottom figure is a zoom in the
final steps of early stopping.

Table 3 presents the TODS metrics computed for the test split. We observe that
the pseudo-tods curriculum performs better than other curricula. The COMBINED
score is given by $(\texttt{INFORM}+\texttt{SUCCESS})*0.5+\texttt{BLEU}$.

Table 3: Standard metrics for the test split in MultiWoZ. Each row contain a
curriculum that is evaluate in different model sizes.

Curriculum | Model Size | BLEU | INFORM | SUCCESS | COMBINED
---|---|---|---|---|---
No |  | 31.3 | 93.4 | 90.4 | 123.2
No-encode | Small | 31.3 | 93.3 | 90.4 | 123.2
CTL (ours) |  | 31.3 | 93.6 | 90.4 | 123.3
No |  | 31.7 | 93.2 | 90.3 | 123.5
No-encode | Medium | 31.8 | 93.3 | 90.2 | 123.6
CTL (ours) |  | 31.7 | 93.5 | 90.5 | 123.7
No |  | 31.5 | 93.3 | 89.6 | 123.0
No-encode | Large | 31.0 | 93.0 | 89.3 | 122.2
CTL (ours) |  | 31.6 | 93.3 | 89.9 | 123.2

We found that the faster adaptation and overall loss observed during training
also translated to the high-level metrics; and the larger the model, the more
consistent the gain.

##  5 Discussion

Differing from traditional language modeling, where the task is to predict
sequences of words given a context, a task-oriented dialog presents a
structured sequence prediction problem with three NLP sub-tasks: (i) a
classification task to recognize intents; (ii) a NER to recognize entities;
and (iii) a natural language generation task to predict the adequate response
for a given utterance and system actions. LMs are general multi-task learners
for NLP, which overfit on specific contexts with a nonstandard distribution of
tokens.

In this research, we defended that having a general language modeling task as
a starting point is not ideal for the problem of TODS. As our problem has a
strong grammar dependency, it is, the generated text should be parseable, and
the model should always predict the same tokens even if the dialog session has
a hundred interactions. This requires constructing gradual datasets to better
initialize the LM and guarantee it does not degenerate in predicting the
grammar for long sequences.

Our proposed solution, CTL, a sequence of transfer learning steps, helps to
balance the trade-off between the right bias in the data and annotation costs.
Our proposal shows how this approach is not only a multi-step sequential
transfer learning, and it can be viewed as a form of curriculum learning when
considering the overall ordering within all tasks. We optimized order through
the same data-generating process in the original curriculum learning approach.
Our CTL approach allows us to use out-of-distribution data, which is central
to learning with scalability and generality for modern NLP systems.

In our experiments we could verify that our proposed pre-training approach
signicantly improve the initialization for training on MultiWoZ data,
resulting in an overall improvement in standard TODS metrics when compared to
previous approaches.

Our approach explored the CTL on a narrow case study, given that sequence-to-
sequence task encoding is a relatively new proposal Hosseini-Asl et al.
(2020). However, prompt-based methods are a recent trending topic in NLP.
These methods allow a pre-trained LM to perform out-of-the-box several tasks
they were not explicitly trained for (also called zero-shot learning).
Although those models perform pretty well in classic NLP tasks, those with a
complex morphological structure still need additional learning. We could
extend those models with CTL to other complex applications by creating pseudo-
labeled datasets in a super-scale.

##  6 Conclusion

In this research, we developed a solution to train LMs for complex grammar
acquisition. Our proposal was accomplished by training the model in
intermediate datasets following the grammar simplification method. This allows
external data sources to encode as an intermediate sequence task to improve
the general optima. Unlike earlier research attempts to use additional data
depending on annotated data, which is expensive and not feasible for many
business models, our approach used unsupervised data to structure pseudo-
supervised data and model sharing to significantly reduce the costs of
maintaining this kind of system.

## References

  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020.  Towards a human-like open-domain chatbot.
  * Bengio et al. (2007) Yoshua Bengio, Yann LeCun, et al. 2007.  Scaling learning algorithms towards ai.  _Large-scale kernel machines_ , 34(5):1–41.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009\.  Curriculum learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 41–48.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165.
  * Bunt (2006) Harry Bunt. 2006.  Dimensions in dialogue act annotation.  In _LREC_.
  * Cotterell et al. (2018) Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018.  Are all languages equally hard to language-model? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 536–541, New Orleans, Louisiana. Association for Computational Linguistics.
  * Dai et al. (2021) Yinpei Dai, Hangyu Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, and Xiaodan Zhu. 2021.  Preview, attend and review: Schema-aware curriculum learning for multi-domain dialogue state tracking.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 879–885, Online. Association for Computational Linguistics.
  * Davies (1980) Norman F Davies. 1980.  Language acquisition, language learning and the school curriculum.  _System_ , 8(2):97–102.
  * Delétang et al. (2022) Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Marcus Hutter, Shane Legg, and Pedro A. Ortega. 2022.  Neural networks and the chomsky hierarchy.
  * Devlin et al. (2019) J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Fellows et al. (2021) Ryan Fellows, Hisham Ihshaish, Steve Battle, Ciaran Haines, Peter Mayhew, and J Ignacio Deza. 2021.  Task-oriented dialogue systems: performance vs. quality-optima, a review.  _arXiv preprint arXiv:2112.11176_.
  * Foglino and Leonetti (2019) Francesco Foglino and Matteo Leonetti. 2019.  An optimization framework for task sequencing in curriculum learning.  _2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)_ , pages 207–214.
  * Green (2005) Christopher Green. 2005.  Integrating extensive reading in the task-based curriculum.  _ELT journal_ , 59(4):306–311.
  * Guasti (2004) Maria Teresa Guasti. 2004.  _Language acquisition: The growth of grammar_.  Cambridge University Press.
  * Hendrycks et al. (2020) Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Xiaodong Song. 2020.  Pretrained transformers improve out-of-distribution robustness.  In _ACL_.
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020.  A simple language model for task-oriented dialogue.  _arXiv preprint arXiv:2005.00796_.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _ACL_.
  * Kim et al. (2021) Boeun Kim, Dohaeng Lee, Sihyung Kim, Yejin Lee, Jin-Xia Huang, Oh-Woog Kwon, and Harksoo Kim. 2021.  Document-grounded goal-oriented dialogue systems on pre-trained language model with diverse input representation.  In _Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021)_ , pages 98–102, Online. Association for Computational Linguistics.
  * Komarova et al. (2001) Natalia L Komarova, Partha Niyogi, and Martin A Nowak. 2001.  The evolutionary dynamics of grammar acquisition.  _Journal of theoretical biology_ , 209(1):43–59.
  * Kulhánek et al. (2021) Jonáš Kulhánek, Vojtěch Hudeček, Tomáš Nekvinda, and Ondřej Dušek. 2021.  AuGPT: Auxiliary tasks and data augmentation for end-to-end dialogue with pre-trained language models.  In _Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI_ , pages 198–210, Online. Association for Computational Linguistics.
  * Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020.  MinTL: Minimalist transfer learning for task-oriented dialogue systems.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3391–3405, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Jinglin Liu, Yi Ren, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2020\.  Task-level curriculum learning for non-autoregressive neural machine translation.  _ArXiv_ , abs/2007.08772.
  * Liu et al. (2021a) Sihong Liu, Jinchao Zhang, Keqing He, Weiran Xu, and Jie Zhou. 2021a.  Scheduled dialog policy learning: An automatic curriculum learning framework for task-oriented dialog system.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1091–1102, Online. Association for Computational Linguistics.
  * Liu et al. (2021b) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b.  Gpt understands, too.  _ArXiv_ , abs/2103.10385.
  * Manela and Biess (2022) Binyamin Manela and Armin Biess. 2022.  Curriculum learning with hindsight experience replay for sequential object manipulation tasks.  _Neural Networks_ , 145:260–270.
  * Mielke et al. (2019) Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019\.  What kind of language is hard to language-model? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4975–4989, Florence, Italy. Association for Computational Linguistics.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _ArXiv_ , abs/2112.09332.
  * Nekvinda and Dušek (2021) Tomáš Nekvinda and Ondřej Dušek. 2021.  Shades of BLEU, flavours of success: The case of MultiWOZ.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 34–46, Online. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022\.  Training language models to follow instructions with human feedback.  _ArXiv_ , abs/2203.02155.
  * Pan and Yang (2009) Sinno Jialin Pan and Qiang Yang. 2009.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ , 22(10):1345–1359.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _ArXiv_ , abs/1910.10683.
  * Ruder (2019) Sebastian Ruder. 2019.  _Neural transfer learning for natural language processing_.  Ph.D. thesis, NUI Galway.
  * Saito (2018) Atsushi Saito. 2018.  Curriculum learning based on reward sparseness for deep reinforcement learning of task completion dialogue management.  In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ , pages 46–51, Brussels, Belgium. Association for Computational Linguistics.
  * Sarker (2021) Iqbal H Sarker. 2021.  Machine learning: Algorithms, real-world applications and research directions.  _SN Computer Science_ , 2(3):1–21.
  * Shimodaira (2000) Hidetoshi Shimodaira. 2000.  Improving predictive inference under covariate shift by weighting the log-likelihood function.  _Journal of statistical planning and inference_ , 90(2):227–244.
  * Soviany et al. (2021) Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2021.  Curriculum learning: A survey.  _arXiv preprint arXiv:2101.10382_.
  * Su et al. (2021) Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2021.  Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System.  _ArXiv_.
  * Sun et al. (2022) Haipeng Sun, Junwei Bao, Youzheng Wu, and Xiaodong He. 2022.  BORT: Back and denoising reconstruction for end-to-end task-oriented dialog.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2156–2170, Seattle, United States. Association for Computational Linguistics.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.
  * Traum and Hinkelman (1992) David R Traum and Elizabeth A Hinkelman. 1992.  Conversation acts in task-oriented spoken dialogue.  _Computational intelligence_ , 8(3):575–599.
  * White (2003) L. White. 2003.  _Second Language Acquisition and Universal Grammar_.  Cambridge Textbooks in Linguistics. Cambridge University Press.
  * Williams et al. (2016) Jason D Williams, Antoine Raux, and Matthew Henderson. 2016.  The dialog state tracking challenge series: A review.  _Dialogue & Discourse_, 7(3):4–33.
  * Wu et al. (2020) Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020.  TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 917–929, Online. Association for Computational Linguistics.
  * Yang et al. (2020) Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2020.  Ubar: Towards fully end-to-end task-oriented dialog systems with gpt-2.  In _AAAI Conference on Artificial Intelligence_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2020a) Yichi Zhang, Zhijian Ou, Min Hu, and Junlan Feng. 2020a.  A probabilistic end-to-end task-oriented dialog model with latent belief states towards semi-supervised learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9207–9219, Online. Association for Computational Linguistics.
  * Zhang et al. (2020b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 270–278, Online. Association for Computational Linguistics.
  * Zhang et al. (2020c) Zheng Zhang, Ryuichi Takanobu, Qi Zhu, Minlie Huang, and Xiaoyan Zhu. 2020c.  Recent advances and challenges in task-oriented dialog systems.  _Science China Technological Sciences_ , pages 1–17.
  * Zhao et al. (2022) Yang Zhao, Hua Qin, Wang Zhenyu, Changxi Zhu, and Shihan Wang. 2022.  A versatile adaptive curriculum learning framework for task-oriented dialogue policy learning.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 711–723, Seattle, United States. Association for Computational Linguistics.
  * Zhu et al. (2021a) Hui Zhu, Yangyang Zhao, and Hua Qin. 2021a.  Cold-started curriculum learning for task-oriented dialogue policy.  _2021 IEEE International Conference on e-Business Engineering (ICEBE)_ , pages 100–105.
  * Zhu et al. (2021b) Qingqing Zhu, Xiuying Chen, Pengfei Wu, JunFei Liu, and Dongyan Zhao. 2021b.  Combining curriculum learning and knowledge distillation for dialogue generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1284–1295, Punta Cana, Dominican Republic. Association for Computational Linguistics.

  *[ML]: Machine Learning
  *[NLP]: Natural Language Processing
  *[TODS]: Task-Oriented Dialog Systems
  *[LMs]: Language Model
  *[NER]: Named Entity Recognition
  *[LM]: Language Model
  *[RL]: Reinforcement Learning
  *[AI]: Artificial Intelligence
  *[CTL]: Curricular Transfer Learning