# Reinforcement Learning for Few-Shot Text Generation Adaptation

Pengsen Cheng Corresponding Author<EMAIL_ADDRESS>Sichuan University
Jinqiao Dai<EMAIL_ADDRESS>Sichuan University  Jiamiao Liu
<EMAIL_ADDRESS>Sichuan University  Jiayong Liu<EMAIL_ADDRESS>Sichuan
University  Peng Jia<EMAIL_ADDRESS>Sichuan University

###### Abstract

Controlling the generative model to adapt a new domain with limited samples is
a difficult challenge and it is receiving increasing attention. Recently,
methods based on meta-learning have shown promising results for few-shot
domain adaptation. However, meta-learning-based methods usually suffer from
the problem of overfitting, which results in a lack of diversity in the
generated texts. To avoid this problem, in this study, a novel framework based
on reinforcement learning (RL) is proposed. In this framework, to increase the
sample utilization of RL and decrease its sample requirement, maximum
likelihood estimation learning is incorporated into the RL process. When there
are only a few in-domain samples available, experimental results on five
target domains in two few-shot configurations show that this framework
performs better than baselines.

##  1 Introduction

Category text generation is an extension of emotional and conditional text
generation [1], which generates coherent and meaningful text in different
categories [2]. Due to its widespread use, it has attracted a lot more
attention lately. For instance, techniques for category text generation have
been used to successfully write and publish large-scale articles on sports,
finance, and other single-domain topics. Each text generation domain task is
different, due to the specific domain traits and expertise. For training, a
trainable text generation system needs thousands of texts. However, the
availability of the training data is usually limited. Therefore, a crucial
task in the study of category text generation is to make use of existing rich-
resource data for new domains with limited resources.

Few-shot learning [3] is introduced in solving such data scarcity problems in
machine learning. But it is extremely challenging that training large models
with small datasets often leads to overfitting [4]. Overfitting in text
generation shows a lack of randomization in the model sampling due to sample
over-simulation, which results in a lack of diversity in generated texts. As a
result, it is challenging to generate texts with domain traits while
preserving expressive diversity.

In recent years, the most popular framework for few-shot learning has been
meta-learning [5]. The main idea of meta-learning is to build an internal
representation of multiple tasks and maximize the loss function’s sensitivity.
A minor adjustment to the meta-learning-based models’ parameters could
significantly reduce the loss of a new task [4]. It means models can fit a
distribution quickly with limited information. A distribution fitted by small
samples is incomplete [6], and the result is that the generated texts are
highly similar to training samples. More significantly, different domain
descriptions of the same language share the same grammar, which is the
fundamental rule of language. Different word combinations are used to express
the differences between various domain descriptions. A typical phenomenon is
that a sentence expresses different domain descriptions by changing just one
word. A domain description of a book would be “This is one of my favorite
books”, and a domain description of a movie would be “This is one of my
favorite films”. Sentences constructed from the same words in different orders
can also be used to describe various domains. A domain description of a book
would be “The novel has increased the attention of this film adaptation”, and
a domain description of a movie would be “This film adaptation has increased
the attention of the novel”. This study aims to achieve a diversity of
expressions in the generated texts by replacing words in sentences while
preserving the sentence structure to the greatest extent possible. The words
that can affect the expression of the domain should be replaced with the words
that can express the semantics of new domains.

Reinforcement learning (RL) is used in this study to generate domain texts by
adjusting the distribution of words that can represent new domains in
sentences. Following studies [7, 8], text generation is considered as a
sequential decision making process. As an RL agent, a generative model must
learn how to select the appropriate word. This is RL’s benefit, but RL can’t
completely solve the few-shot problem [9]. On the one hand, RL is an
inefficient sampling. To maximize rewards, the agent must continually choose
the best from states according to the exploit-exploration of RL. On the other
hand, in text generation, the number of states in RL is determined by the size
of the dictionary which typically has thousands of words. The issue of
inefficient sampling in RL is exacerbated by a large number of states, and RL
needs more samples to learn a policy [10, 11]. In addition, the training of a
text classifier that is usually chosen as the reward function is also stuck in
the limited samples.

In this study, a framework DARL (Domain-Adaption-via-Reinforcement-Learning)
based on RL is proposed to address the issue. In addition to indirect transfer
of knowledge via reward signals, the process of active knowledge learning is
added by merging maximum likelihood estimation (MLE) training with RL. And a
text classifier based on few-shot learning is employed as reward function to
evaluate the generated texts and guide the learning of the generative model.

The major contributions of this paper are summarized as follows:

  * •

A novel framework DARL based on RL is proposed to generate diversified and
high-quality texts with low-resource domains. During RL, a typical non-
likelihood estimation learning process, MLE is randomly inserted to enhance
the active learning capability of the framework.

  * •

A new metric is proposed to evaluate the domain relevance of the generated
texts. The metric can exclude the impact of generated duplicate texts on
results.

  * •

The effectiveness and superiority of the proposed framework are shown by
extensive experiments on five target domains in two few-shot configurations.

The remainder of the paper is organized as follows. In Section 2, the
literature on both models and on tasks related to the research work are
provided. In Section 3, the constraints of meta-learning to explain the
problem is discussed and the problem is setup by RL. In Section 4, DARL is
described in detail. In Section 5, the effectiveness of the proposed model is
validated. In Section 6, the impact of classifiers trained using different
techniques on the results is discussed. Concluding remarks are presented in
Section 7.

##  2 Related Work

Few-shot-learning-based approaches are increasingly able to train powerful
neural networks on small datasets in many nature language processing (NLP)
problems [3]. Meta-learning has gained more attention than other few-shot
learning approaches due to its more efficient learning process [12]. Meta-
learning aims at learning new tasks with few steps and little data based on
well-known tasks [4]. Several optimization-based methods for the tasks of
nature language generation (NLG) have been proposed. These methods learn a
good point of parameter initialization for a neural model from which a few
steps of gradient descent, given a few examples, can reach the optimal point
for a new task. One way is to learn an optimal initialization that could be
adapted to a new task accurately and quickly with little data [13, 15, 16].
Another way to learn the learning progress is to train a meta-learner to
optimize the optimizer of original network for updating parameters [17, 18,
19, 20, 21]. A few model-based learning methods [22, 23, 24], like external
memory storage, rely on the design of the model’s internal structure to
facilitate the learning process. There are also a few methods based on
optimization and model combinations [14].

Besides, other researches have tried different attempts. A zero-shot learning
method was proposed to adapt models learned from multiple source domains to a
new target domain only using its domain description [25]. A transfer learning
method for NLG based on data augmentation was proposed to address the
scalability of large-scale conversational systems [26]. A selection strategy
on training instances in few-shot neural text generation was proposed to
choose the few-shot training instances that should be diverse and
representative of the entire data distribution [27].

The challenge of few-shot-learning-based domain adaptation in NLG is
concentrated on the dialog system in terms of application; in terms of
implementation methodologies, part of the research focuses on meta-learning.
As long as the expression is accurate, participants in the dialog system are
willing to accept questions asked and responses given in a fixed expression
style. But in category text generation, diversity expression is a form of
creativity. The existing studies have not focused on the problem of generating
text diversity based on few-shot learning. The application of RL on top of the
model to achieve domain adaptation will increase the diversity of text
generation based on few-shot learning.

##  3 Problem Setup

In this section, the meta-learning problem in domain adaptation of text
generation is defined and formulated, and the limitation is discussed. Then
the problem is defined and formulated by RL. For the convenience of
discussion, the basic problem of text generation is defined as follow:

$p(x)=\prod_{i=1}^{n}p(x_{i}|x_{<i})$ (1)

A discrete sequence of text tokens $x=(x_{1},x_{2},...,x_{n})$ is given as
input where each $x_{i}$ is drawn from a fixed set of symbols. The goal of
text generation is to learn the unconditional probability distribution $p(x)$
of the sequence $x$.

When $p(x)$ is modeled by a neural network with parameters $\theta$, the
neural network is trained to minimize the negative log-likelihood over a
collection of samples $X={x^{1},x^{2},...,x^{|X|}}$:

$\mathcal{L}(X)=-\prod_{k=1}^{|X|}p_{\theta}(x_{i}^{k}|x_{<i}^{k})$ (2)

###  3.1 Problem Setup by Meta-Learning

The problem of domain adaptation of text generation is defined and formulated
using the model-agnostic meta-learning algorithm (MAML) [28] which is the
state-of-art work in meta-learning.

In meta learning, it learns a $\theta$ by repeatedly sampling from the task
distribution $\mathcal{D}$ over a family of tasks. The $\theta$ can be easily
fine-tuned on new tasks. Every domain is viewed as a task in domain
adaptation, and $\mathcal{D}$ is the set of all recognized domains. Each task
$\mathcal{T}_{i}$ is consisted by a support set $X_{i}^{(t)}$ and a query set
$X_{i}^{(v)}$, $X_{i}=\\{x_{ij}\\}_{j=1}^{M}$. The aim is to fit
$X_{i}^{(v)}\sim p(X_{i}^{(t)})$ of the query input $x_{ij}^{(v)}$ given the
small support set of task $\mathcal{T}_{i}$ (e.g. $M\leq 10$). The objective
function of meta-learning is to find a meta-learner, parameterized by
$\theta$, across tasks from $\mathcal{D}$, as follows:

$\theta^{*}=\mathop{\arg\min}_{\theta}-\frac{1}{T}\prod_{i=1}^{T}\ln{p(X_{i}^{(v)}|X_{i}^{(t)}),\theta)},$
(3)

where $T$ denotes the number of tasks.

In meta testing, it learns a $\theta^{{}^{\prime}}$ based on the $\theta$ and
a new task by minimizing the negative log-likelihood, as follows:

$\mathcal{L}(X)=-\prod_{k=1}^{M}p_{{\theta}^{{}^{\prime}}}(x_{i}^{k}|x_{<i}^{k},\theta).$
(4)

Therefore, in the domain adaptation of text generation task, the final output
of meta-learning-based methods is $p(X)$ modeled by the
$\theta^{{}^{\prime}}$, where $X$ are samples of new domains. As a model
generates another model, the Bayes-based MAML model [29, 30] is more
understandable in this regard. Directly fitting the probability to the limited
samples can easily lead to the problem of insufficient diversity of generated
texts. The issue becomes more noticeable when the algorithm’s generalization
ability is poor. However, improving generalization has always been the focus
and difficulty of meta-learning [3, 4, 12].

###  3.2 Problem Setup by Reinforcement Learning

The task of domain adaptation of text generation can be described by RL as
follows. An agent that has memorized the necessary vocabulary and grammar is
known as a generative model. It could generate texts that described the
environment. However, the environment has since changed, and texts generated
by the agent no longer accurately reflect the environment. As a result, the
environment provides negative feedback to the agent. The agent must try a new
vocabulary organization to be recognized by the environment.

The above process is formulated. First, the Markov decision process (MDP) can
be used to describe the text generation process. In each timestep $t$, the
model generates $x_{t}$ according a policy $\pi_{\theta}(a_{t}|s_{t})$, where
$s_{t}$ is the current state of the previous prediction $x_{1:t}$ and $a_{t}$
is the action to select the next word $x_{t+1}$. Secondly, the reward function
explains the expert behavior by determining whether the sequence belongs to
the target domain. Given the extensive state-of-the-art research on few-shot
learning-based text classifiers [31, 32] and the prevalence of text
classifiers as reward functions in RL [7, 33], the problem of few-shot
learning is shared by the policy model and the reward function.

Compared to fine-tuning meta-learner parameters to fit a distribution, a
distribution is fine-tuned with domain-insensitive samples by RL.

##  4 Proposed Methods

A detailed description of DARL is described in this section. First, the
feasibility of using a text classification model based on few-shot learning as
the reward function is analyzed. Then, the approach that can increase the
sample utilization for RL is described, which can improve the limitation that
the agent can only be passively guided by the reward signal. Finally, the
whole processing flow of the DARL framework is described completely.

###  4.1 Reward Function

In this task, the reward function must be able to distinguish between the
differences in generated target domain text and real target domain text. In
NLG frameworks based on RL, the text classification model is frequently used
as a reward function. But since the task’s underlying presumption is that the
sample is limited, conventional text classification models typically require
extensive training to perform adequately. Because of this, a text
classification based on few-shot learning is adopted as a reward function.
First, it has been demonstrated that few-shot learning-based text
classification models are very successful [34]. Theoretically, few-shot
learning-based text classification models are possible when the target
domain’s sample size is insufficient. Second, model generated texts are
sampled from a specific distribution and are more regular than human-written
texts, and more easily recognized by text classification models [35]. As a
result, a few-shot learning-based text classification model can be used as the
reward function.

Furthermore, if the reward function fails to accurately determine whether the
texts are generated or human written, the RL agent has a tendency to follow a
particular generative pattern. In this generative pattern, the agent generates
special texts that can trick the reward function into yielding higher reward
signals, but these texts are typically unreadable to humans. For this reason,
adversarial training is introduced to strengthen the text classification
model’s robustness and ensure that the agent is appropriately rewarded. RL is
combined with a generative adversarial network (GAN) after adversarial
training. In the following description, the reward function or text
classification model will be referred to as a discriminator, and the agent or
generative model will be referred to as a generator.

###  4.2 Active Learning

During RL, agents are not actively informed of new knowledge about the
environment or what the best action is. The environment is the only source of
direct instruction for agents, and even then, it only provides rewards rather
than actual knowledge. By gaining limited new knowledge directly, humans can
quicken their learning. This is especially important for NLP, where new
knowledge refers to new words, expressions, or word interpretations
(ambiguities in language). Words cannot be generated by memory, so knowledge
based on new vocabulary cannot be learned indirectly. Additionally, indirect
learning takes a lot of time because agents continuously try out new actions
to maximize reward.

This study integrates MLE into RL to ensure that agents can directly acquire
samples to address this problem. The ability to assign a high probability to
the tokens updated by a likelihood is inspired by the work of unlikelihood
training [36]. However, this research differs from unlikelihood training. The
aim of unlikelihood training is to improve neural text degeneration [37]. To
reduce the likelihood of previously generated tokens, it regulated likelihood
loss by unlikelihood loss [36, 37]. This study aims to provide agents with
direct access to samples of the target domain and increase the probability of
tokens that are more likely to be used in the target domain.

###  4.3 Process of Learning

Following the work of SeqGAN [7], the same policy gradient is adopted and
Monte Carlo (MC) search is employed to approximate the state-action value. The
objective of the generator model $G_{\theta}(y_{t}|Y_{1:t-1})$ is to generate
a text from the start state $s_{0}$ to maximize its reward:

$J(\theta)=\mathbb{E}[R_{T}|s_{0},\theta]=\sum_{y\in
Y}G_{\theta}(y|s_{0})\cdot Q_{D_{\phi}}^{G_{\theta}}(s_{0},y),$ (5)

where $Q_{D_{\phi}}^{G_{\theta}}(s,a)$ is the action-value function of a text.
$R_{T}$ is the reward for a complete sequence and the reward from
discriminator $D_{\phi}$.

However, the discriminator only provides a reward value for a finished
sequence and every action should be rewarded. Thus, MC search with generator
policy $G_{\theta}$ is applied to sample the unknown last tokens. $N$-time MC
search be represented as

$\\{Y_{1:T}^{1},...,Y_{1:T}^{N}\\}=MC^{G_{\theta}}(Y_{1:t};N),$ (6)

where $Y_{1:t}^{n}$ and $Y_{t+1:T}^{n}$ is sampled based on roll-out policy
$G_{\theta}$ and the current state. Thus, it has:

$Q_{D_{\phi}}^{G_{\theta}}(a=y_{t},s=Y_{1:t-1})=\left\\{\begin{array}[]{lcr}\frac{1}{N}\sum_{n=1}^{N}D_{\phi}(Y_{1:T}^{n})&for\quad
t<T\\\ D_{\phi}(Y_{1:t})&for\quad t=T,\end{array}\right.$ (7)

where $Y_{1:T}^{n}\in MC^{G_{\theta}}(Y_{1:t};N)$.

The generator’s parameters are updated in RL as:

$\theta\leftarrow\theta+\alpha\nabla_{\theta}J(\theta),$ (8)

where $\alpha$ is the learning rate. And $\nabla_{\theta}J(\theta)$ is an
unbiased estimation as:

$\nabla_{\theta}J(\theta)=\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{y_{t}\sim
G_{\theta}(y_{t}|Y_{1:t-1})}[\nabla_{\theta}\log
G_{\theta}(y_{t}|Y_{1:t-1})\cdot Q_{D_{\phi}}^{G_{\theta}}(Y_{1:t-1},y_{t})].$
(9)

Algorithm 1 The training process in DARL

1:generator policy $G_{\theta}$; discriminator $D_{\phi}$

2:update rate $R$

3:a source dataset $S=\\{X_{1:m}^{1},...,X_{1:m}^{N}\\}$ includes $N$ domains;
a target domain dataset $S^{{}^{\prime}}=\\{X_{1:k}\\}$

4:Initialize $G_{\theta}$ and $D_{\phi}$ with random weights $\theta$ and
$\phi$

5:Pre-train $G_{\theta}$ using MLE on $S$

6:Pre-train $D_{\phi}$ using few-shot learning on $S$

7:Generate negative samples using $G_{\theta}$ for training $D_{\phi}$

8:train $D_{\phi}$ using few-shot learning on negative samples and
$S^{{}^{\prime}}$

9:while not done do

10: sample $r\sim U[0,1]$

11: if $r<R$ then

12: Generate a sequence $Y_{1:T}=(y_{1},...,y_{T})\sim G_{\theta}$

13: for $t$ in $1:T$ do

14: Compute $Q(a=y_{t};S=Y_{1:t-1})$ by Eq.7

15: Update generator parameters via a policy gradient Eq.8

16: else

17: Update generator parameters via MLE on $S^{{}^{\prime}}$

18: Use current $G_{\theta}$ to generate negative examples and combine with
$S^{{}^{\prime}}$

19: Train discriminator $D_{\phi}$ by few-shot learning

In summary, algorithm 1 shows full details of the proposed DARL. At the
beginning of the training, $G_{\theta}$ is pre-trained by MLE on source
dataset $S$. Despite containing $N$ domains, $S$ is considered as a whole. The
category information from the domain is ignored during $G_{\theta}$’s pre-
training. And $D_{\phi}$ is also pre-trained using few-shot learning on $S$.
Each domain in $S$ represents a specific task. The category information for
the domain is taken into account during $D_{\phi}$’s pre-training.

After pre-training, the generator and discriminator are trained alternatively.
The ratio of RL training to MLE training is determined by the hyperparameter
$R$, and the generator is randomly trained by either method. The discriminator
needs to be trained periodically to keep up with the generator as it advances
through training. Positive examples for the discriminator’s training come from
the provided dataset $S^{{}^{\prime}}$, whereas negative examples come from
$G_{\theta}$. And the number of negative examples is the same as the number of
positive examples.

The RL training of DARL is similar to SeqGAN. However, they are fundamentally
distinct. SeqGAN adopts RL and MC search to generate sequences of discrete
tokens because GANs are designed for generating real valued and continuous
data. SeqGAN extents the application fields of GANs. DARL adopts RL and MC
search to find phrases with target domain features and increase the likelihood
of certain tokens. And it adopts the adversarial ideal to improve robustness
of the reward function. DARL achieves domain adaptation for text generation
using RL.

##  5 Experiment

The proposed framework was evaluated by conducting experiments on the dataset.
According to the common experiment configuration of few-shot learning, a
5-shot learning dataset and a 10-shot learning dataset were created,
respectively.

###  5.1 Baselines

To evaluate its effectiveness, DARL was compared with the following five
baselines:

  * $\bullet$

Fine-tune fine-tunes the pre-trained base model on the support sets of each
target domain. It can be viewed as a special case of DARL where the
hyperparameter $R$ of DARL is set to 0 and no RL is done.

  * $\bullet$

MetaNLG [13] is a generalized optimization-based approach based on the well-
recognized MAML algorithm. It learns a better initialization of model
parameters that facilitates fast adaptation to new low-resource domains.

  * $\bullet$

DAML [14] is an end-to-end trainable generation model based on meta-learning
that learns from multiple rich-resource tasks and then adapts to new domains.
It combines multiple tasks in training to learn general and transferable
information that is applicable to new domains.

  * $\bullet$

MemIML [24] is a memory imitation meta-learning method that enhances the
model’s reliance on support sets for task adaptation. It introduces a task-
specific memory module to address memorization overfitting issue.

  * $\bullet$

SeqGAN [7] is a sequence generation framework based on GAN to generate
sequences of discrete tokens. It can be viewed as a special case of DARL where
the hyperparameter $R$ is set to 1 and no MLE is done.

The benchmark was measured using fine-tuning; the advantages and disadvantages
of meta-learning were demonstrated using three meta-learning-based baselines;
the effectiveness of DRAL for RL improvement was illustrated using a GAN-based
baseline.

###  5.2 Datasets

The dataset must include several domains because the meta-learning-based
baselines need to sample across a family of domains. The following dataset,
which is frequently used in text generation tasks, was used to satisfy this
requirement.

Amazon Reviews [38] contains product reviews and metadata from Amazon,
including 24 products reviews. Five products were picked randomly as source
domains, and the other 5 products were chosen randomly as target domains. The
sentences with words between 15 and 30 were selected randomly from each
domain, resulting in a dataset of 2000 sentences per domain. Toys and games
(toys), pet supplies (pet), beauty, grocery and gourmet food (food), and baby
were included by source domains. Cell phones and accessories (phones), tools
and home improvement (tools), office products (office), automotive and digital
music (music) were included by target domains. In the $k$-shot configuration,
$k$ samples of the per target domain were selected randomly, and all methods
shared same samples.

###  5.3 Experiment Setup

A RNN was set as a single-layer LSTM with a hidden dimension size of 256 and a
maximum length of 30 words. Word embeddings were randomly initialized to a
dimension of 300, and the $R$ of DARL was set to 0.5. The induction networks
[31] which were proposed for few-shot learning text classification was
selected as discriminator. DARL was pre-trained for 100 epochs and trained for
150 epochs. And it was implemented based on Pytorch111The repeatable
experiment code is made publicly available on https://github.com/cps11/DARL..

To account for the instability of few-shot learning, each method was tested
ten times, and the model that resulted from each test was sampled ten times
with 5000 samples per domain. The mean of the aforementioned results was used
to calculate the final experimental results.

###  5.4 Quality of Generated Texts

Bilingual evaluation understudy (BLEU) [39] and Self-BLEU [40] were adopted to
measure the quality of the generated texts in terms of fluency and diversity.

  * $\bullet$

BLEU uses source domains set as a reference and evaluates each generated text
using the BLEU 5-gram score. It measures the fluency of the generated texts.

  * $\bullet$

Self-BLEU uses generated texts as a reference and evaluates each generated
text using Self-BLEU 5-gram score. It measures the diversity of the generated
texts.

The results are displayed in Table 1 and Table 2. Comparing with baselines,
DARL improves not only the diversity of the generated text (lower Self-BLEU),
but also the fluency of the generated text (higher BLEU).

Some methods used to generate texts had a high degree of similarity to samples
when they were manually examined. To quantify this phenomenon, Eq.10 was used
to identify the sample with which each generated text was most similar. And
the similarity was evaluated using $BLEU(s^{*},y)$.

$s^{*}=\arg\max_{s\in S^{{}^{\prime}}}BLEU(s,y),y\in Y$ (10)

Table 1: Quality of the generated texts on the 5-shot dataset

Metrics | Methods | Automotive | Music | Office | Phone | Tools | Average
---|---|---|---|---|---|---|---
BLEU$\uparrow$ | Fine-tune | 0.328 | 0.263 | 0.376 | 0.446 | 0.407 | 0.364
MetaNLG | 0.354 | 0.275 | 0.386 | 0.459 | 0.421 | 0.379
DAML | 0.390 | 0.273 | 0.400 | 0.439 | 0.433 | 0.387
MemIML | 0.136 | 0.112 | 0.170 | 0.117 | 0.123 | 0.132
SeqGAN | 0.199 | 0.142 | 0.302 | 0.258 | 0.313 | 0.243
DARL | 0.391 | 0.340 | 0.418 | 0.472 | 0.405 | 0.405
Self-BLEU$\downarrow$ | Fine-tune | 0.932 | 0.932 | 0.943 | 0.949 | 0.946 | 0.940
MetaNLG | 0.852 | 0.829 | 0.866 | 0.935 | 0.927 | 0.882
DAML | 0.902 | 0.825 | 0.896 | 0.947 | 0.938 | 0.902
MemIML | 0.995 | 0.993 | 0.995 | 0.994 | 0.994 | 0.994
SeqGAN | 0.813 | 0.899 | 0.936 | 0.910 | 0.949 | 0.901
DARL | 0.812 | 0.822 | 0.842 | 0.875 | 0.860 | 0.842

Table 2: Quality of the generated texts on the 10-shot dataset

Metrics | Methods | Automotive | Music | Office | Phone | Tools | Average
---|---|---|---|---|---|---|---
BLEU$\uparrow$ | Fine-tune | 0.341 | 0.324 | 0.418 | 0.322 | 0.355 | 0.352
MetaNLG | 0.326 | 0.300 | 0.419 | 0.307 | 0.347 | 0.340
DAML | 0.314 | 0.302 | 0.429 | 0.314 | 0.351 | 0.342
MemIML | 0.132 | 0.133 | 0.077 | 0.147 | 0.067 | 0.111
SeqGAN | 0.197 | 0.278 | 0.249 | 0.199 | 0.233 | 0.231
DARL | 0.378 | 0.397 | 0.419 | 0.365 | 0.393 | 0.391
Self-BLEU$\downarrow$ | Fine-tune | 0.865 | 0.875 | 0.888 | 0.863 | 0.873 | 0.873
MetaNLG | 0.789 | 0.767 | 0.849 | 0.868 | 0.805 | 0.815
DAML | 0.843 | 0.854 | 0.896 | 0.897 | 0.874 | 0.873
MemIML | 0.992 | 0.994 | 0.995 | 0.993 | 0.992 | 0.993
SeqGAN | 0.882 | 0.905 | 0.901 | 0.877 | 0.944 | 0.902
DARL | 0.689 | 0.691 | 0.726 | 0.673 | 0.737 | 0.703

As shown in Figure 1 and Figure 2, the methods can be split into two parts
using Fine-tune method as a benchmark. METANLG and DAML, which are based on
meta-learning, have higher similarity than Fine-tune method. This suggests
that while these two methods are more capable of learning, they are also more
prone to overfitting. MemIML, DARL and SeqGAN have lower similarity than Fine-
tune method. The main objective of MemIML, which is also based on meta-
learning, is to address the issue of memorization overfitting in meta-
learning. Although MemIML’s similarity is lower than that of the other two
meta-learning-based methods, it is still higher than that of DARL and SeqGAN.
SeqGAN achieves the lowest similarity because it is not directly trained on
samples. Its higher Self-BLEU value, however, suggests that a pattern collapse
had taken place. Pattern collapse causes generated texts to follow only a few
fixed patterns, which results in a high degree of similarity between generated
texts even though they differ from samples. Because DARL can be trained
directly with samples through MLE, the similarity between its generated texts
and samples is higher than that between SeqGAN generated texts and samples.
However, the phenomenon that the similarity between texts generated by DARL is
lower than the similarity between texts generated by SeqGAN illustrates that
adding MLE to RL can improve the problem of GAN pattern collapse.

(a) automotive (b) music (c) office (d) phone

(e) tools (f) average

Figure 1: The similarity between generated texts and samples on the 5-shot
database. As the bar is raised, generated texts and samples become more
similar. The mean value of results across all target domains is shown in the
subfigure titled average.

(a) automotive (b) music (c) office (d) phone

(e) tools (f) average

Figure 2: The similarity between generated texts and samples on the 10-shot
database. As the bar is raised, generated texts and samples become more
similar. The mean value of results across all target domains is shown in the
subfigure titled average.

The analysis that was just mentioned explains the rationale for each method’s
diversity indicator (Self-BLEU). Texts generated based on meta-learning
methods are similar to each other because they are highly similar to samples.
Texts generated by these methods are less diverse. The Fine-tune method is the
same as the meta-learning based methods. Due to pattern collapse, texts
generated by SeqGAN have a few fixed patterns. They are also similar to each
other and less diverse. The diversity of the texts generated by DARL
outperforms the baselines because they are less similar to the samples or do
not experience to pattern collapse like SeqGAN.

In conclusion, the fluency and diversity of the texts generated by DARL are
higher than the baselines. In the meantime, the pattern collapse issue can be
improved by implementing MLE in RL.

###  5.5 Domain relevance of the generated text

The accuracy of text classification was adopted to measure the domain
relevance of generated texts. A state-of-art text classifier [41] based on
convolutional neural network (CNN) was trained independently to automatically
evaluate the semantic relevance of generated texts. And a larger dataset than
the one utilized for the generation task was used to train the classifier.

As mentioned in Table 1 and Table 2, all baselines have higher Self-BLEU
metrics which measure repetition within generated texts. Therefore, some
models generate texts that will contain exactly the same texts. The accuracy
obtained by measuring the generated texts containing duplicate texts is
falsely high. When there is a high similarity between a generated text and the
sample, more duplicate texts imply an increased likelihood of similarity
between generated texts and samples. As a result, it is unfair to compare
various methods based on inflated results.

To objectively and fairly reflect the performance of various methods, concepts
of gross profit margin and net profit margin in business accounting are
introduced to define the following concepts.

  * $\bullet$

Sales are the total amount of generated texts.

  * $\bullet$

Cost is the number of texts generated in non-target domain as determined by
the classifier.

  * $\bullet$

Tax is the number of generated texts that are duplicated after non-target
domain texts are excluded.

Gross accuracy and net accuracy were calculated using Eq.11 and Eq.12,
respectively. The duplicate generated texts are treated as a tax penalty in
Eq.12, just like the net profit margin is. The tax should be higher as a
penalty because generating more duplicate texts results in more waste.

$Gross\ Accuracy=\frac{Sales-Cost}{Sales}$ (11)

$Net\ Accuracy=\frac{Sales-Cost-Tax}{Sales}$ (12)

The results are shown in Figure 3 and Figure 4. In terms of gross accuracy,
Fine-tune, MetaNLG, and DAML all have very notable advantages. But in terms of
net accuracy, DARL and Fine-tune have significant advantages. Additionally,
the data displays how different methods’ variations in gross accuracy and net
accuracy. The top three methods (Fine-tune, MetaNLG, and DAML) with the
highest gross accuracy also suffered the greatest net accuracy declines. This
implies that these

(a) automotive (b) music (c) office (d) phone

(e) tools (f) average

Figure 3: The accuracy of generated texts on the 5-shot dataset. The mean
value of the results across all target domains is shown in the subfigure
titled average.

(a) automotive (b) music (c) office (d) phone

(e) tools (f) average

Figure 4: The accuracy of generated texts on the 10-shot dataset. The mean
value of the results across all target domains is shown in the subfigure
titled average.

methods generate more duplicate texts and consequently involve more tax
penalties. The remaining three methods (DAML, MetaNLG, SeqGAN) only experience
a slight decline in net accuracy, indicating that fewer duplicate texts are
generated and that the effective output rate is higher. The phenomenon of high
similarity between generated texts is consistent with the one mentioned in
section 5.4. And it highlights shortcomings of the meta-learning-based method.
Although overfitting can be reduced, this could also result in a lack of
learning about the target task. For instance, MemIML performs better than the
other two meta-learning-based methods in terms of overfitting, but it falls
far short of DARL and Fine-tune in terms of representing the target domain.

In summary, the effective domain relevance of texts generated by DARL is
higher than baselines. Additionally, DARL is no issue with overfitting.

###  5.6 Analysis of hyperparameter $R$

To test the effect of $R$ on results, $R$ was set to be 0, 0.25, 0.5, 0.75,
and 1, respectively. During the training process, as $R$ increases, fewer MLE
is executed while more RL is executed. The results are shown in Figure 5 and
Figure 6.

(a) average (b) automotive (c) music (d) office

(e) phone (f) tools

Figure 5: The impact of $R$ on findings on the 5-shot dataset. The mean value
of the results across all target domains is shown in the subfigure titled
average.

In terms of text quality, the fluency and diversity of the generated texts are
two metrics that can be improved with more RL training, but maintaining some
MLE can stop these two metrics from getting worse. The fluency and diversity
metrics are at their worst when R is set to 1, no MLE is run, and DARL
experiences severe pattern collapse. This once more demonstrates that the
pattern collapse issue can be improved by including MLE in RL.

In terms of text domain relevance, gross accuracy decreases as RL training
progresses, while net accuracy trends downward after a brief uptrend. When R
is set to be 1, the fact that the accuracy of DARL is not zero supports the
viability of changing the current probabilities to describe the target domain
probabilities using RL. However, RL based on trial-and-error exploration is
ineffective. Guided by reward signals rather than samples, the model is only
capable of generating a very small amount of target domain texts. The addition
of MLE significantly reduces this issue. With the introduction of MLE, DARL is
now able to directly learn the target domain knowledge from samples. The
difference between gross and net accuracy shows that DARL generates more
duplicate texts when R is set to be a low value, proving that more MLE is not
necessarily better.

(a) average (b) automotive (c) music (d) office

(e) phone (f) tools

Figure 6: The impact of $R$ on findings on the 10-shot dataset. The mean value
of the results across all target domains is shown in the subfigure titled
average.

As a result, the value of $R$ shouldn’t be excessively high or low. When $R$
is low, although the generated texts’ domain relevance is high, the quality is
poor, and there are a lot of duplicate texts; When R is high, although the
generated texts include fewer duplicate texts, the domain relevance is low,
and there is a chance of pattern collapse, which could lead to poor text
quality.

##  6 Discussion

In adversarial training, the discriminator is essential. The text classifier
is used as a discriminator in text generation tasks based on adversarial
training. It is the same as DARL. How will classifiers with different training
methods affect the results? The comparison experiments were conducted to
clarify this issue. In the experiments, the same CNN [41] was trained by
following three methods.

  * $\bullet$

Classic: All samples of a target domain (size is 2000) were positive samples
and texts generated by DARL with same size were negative samples. The
processes of pre-training and training were followed the work [7] with
$d-step=1$.

  * $\bullet$

Transition: $k$ samples of a target domain were positive samples and texts
generated by DARL with $k$ size were negative samples. The processes of pre-
training and training were followed the classic method. And positive samples
were kept same in each training.

  * $\bullet$

Maml: The processes of pre-training and training were followed the work of
MAML [28]. In pre-training, CNN was trained with source domains. At the last
iteration of pre-training, it was trained with the $k$ samples of target
domain and the $k$ samples of DARL generated texts.

The benchmark used to compare the efficacy of training methods is the classic
method that is successful and widely used in practice. As the results shown in
Table 3 and Table 4, the metrics of the classic method are measured values,
and the metrics for the other two methods are the increase/decrease values
compared to the classic method. It can be found that the measurements of the
transition and maml methods do not differ significantly from those of the
classic method, respectively. It is anticipated that using a small sample set
to train the classifier using the classic method over several epochs would
result in overfitting [4]. However, in this task, half of the generated data
in the sample set is changing. The classifier is motivated to continuously
learn new information and new features as a result of the dataset’s frequent
updates, which prevents overfitting. This explains why the final results are
not significantly affected by the selection of different training methods to
update the discriminator parameters in the adversarial process.

In conclusion, when the sample size is small, it makes sense to use a
classifier based on few-shot learning as a discriminator.

Table 3: Impact evaluation of discriminator training methods using the classic method as a benchmark on the 5-shot dataset Domains | Methods | BLEU |  | Self
---
BLEU
| Gross
---
Accuracy
| Net
---
Accuracy
Automotive | Classic | 0.340 | 0.769 | 0.796 | 0.795
Transition | 0.004 | 0.017 | 0.014 | 0.013
Maml | 0.015 | 0.017 | 0.016 | 0.015
Music | Classic | 0.306 | 0.789 | 0.689 | 0.687
Transition | 0.023 | 0.024 | 0.016 | 0.013
Maml | 0.029 | 0.016 | -0.004 | -0.005
Office | Classic | 0.368 | 0.810 | 0.543 | 0.541
Transition | 0.029 | 0.019 | 0.006 | 0.006
Maml | -0.014 | 0.010 | 0.006 | 0.005
Phone | Classic | 0.440 | 0.859 | 0.846 | 0.841
Transition | -0.009 | -0.001 | 0.021 | 0.023
Maml | -0.012 | 0.003 | 0.012 | 0.009
Tools | Classic | 0.395 | 0.840 | 0.748 | 0.743
Transition | 0.007 | 0.016 | 0.014 | 0.014
Maml | 0.022 | 0.016 | -0.004 | -0.003
Average | Classic | 0.370 | 0.814 | 0.724 | 0.721
Transition | 0.011 | 0.014 | 0.015 | 0.014
Maml | 0.003 | 0.017 | 0.005 | 0.005
Table 4: Impact evaluation of discriminator training methods using the classic method as a benchmark on the 10-shot dataset Domains | Methods | BLEU |  | Self
---
BLEU
| Gross
---
Accuracy
| Net
---
Accuracy
Automotive | Classic | 0.351 | 0.652 | 0.573 | 0.573
Transition | 0.010 | 0.013 | -0.001 | -0.001
Maml | -0.010 | 0.004 | 0.009 | 0.009
Music | Classic | 0.356 | 0.645 | 0.838 | 0.836
Transition | -0.001 | 0.013 | 0.007 | 0.007
Maml | -0.026 | -0.002 | 0.014 | 0.014
Office | Classic | 0.391 | 0.684 | 0.477 | 0.477
Transition | -0.007 | 0.016 | 0.000 | -0.001
Maml | 0.007 | 0.023 | 0.027 | 0.027
Phone | Classic | 0.364 | 0.668 | 0.756 | 0.755
Transition | 0.021 | 0.021 | 0.007 | 0.007
Maml | -0.024 | -0.003 | 0.025 | 0.025
Tools | Classic | 0.352 | 0.702 | 0.811 | 0.811
Transition | 0.013 | 0.018 | 0.016 | 0.015
Maml | 0.000 | 0.008 | 0.010 | 0.010
Average | Classic | 0.363 | 0.670 | 0.691 | 0.690
Transition | 0.007 | 0.016 | 0.006 | 0.006
Maml | -0.011 | 0.006 | 0.017 | 0.017

##  7 Conclusion

To generate domain texts with few samples, an RL-based framework called DARL
is proposed in this paper. The fundamental concept behind this study is to use
RL to change the distribution of words that can convey the target domain.
Additionally, MLE training is incorporated into RL to improve sample
utilization, which also alleviates the pattern collapse problem. Due to the
limited size of the dataset, a classifier based on few-shot learning is used
as the discriminator. Extensive experiments demonstrate the efficacy of DARL.

However, it is discovered in some experiments that DARL tends to learn local
knowledge of particular samples. The explanation of this phenomenon helps to
improve the learning ability of DARL by constructing special samples. In
future work, this question will continue to be investigated by related
studies.

## Acknowledgments

This work was supported by the Key Research and Development Program of Sichuan
Province (Grant no.2021YFG0156).

## References

  * [1] B. Guo, H. Wang, Y. Ding, W. Wu, S. Hao, Y. Sun, Z. Yu, Conditional Text Generation for Harmonious Human-Machine Interaction, ACM Transactions on Intelligent Systems and Technology 12 (2) (mar 2021).  arXiv:1909.03409, doi:10.1145/3439816.
  * [2] Z. Liu, J. Wang, Z. Liang, CatGAN: Category-aware generative adversarial networks with hierarchical evolutionary learning for category text generation, in: AAAI 2020 - 34th AAAI Conference on Artificial Intelligence, Vol. 34, AAAI, 2020, pp. 8425–8432.  arXiv:1911.06641, doi:10.1609/aaai.v34i05.6361.
  * [3] Y. Wang, Q. Yao, J. T. Kwok, L. M. Ni, Generalizing from a Few Examples: A Survey on Few-shot Learning, ACM Computing Surveys (CSUR) 53 (3) (jun 2020).  arXiv:1904.05046, doi:10.1145/3386252.
URL https://dl.acm.org/doi/abs/10.1145/3386252

  * [4] T. Hospedales, A. Antoniou, P. Micaelli, A. Storkey, Meta-Learning in Neural Networks: A Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence (apr 2020).  arXiv:2004.05439.
URL https://arxiv.org/abs/2004.05439v2

  * [5] Y. Chen, Z. Liu, H. Xu, T. Darrell, X. Wang, Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning, in: Proceedings of the IEEE International Conference on Computer Vision, 2021, pp. 9042–9051.  arXiv:2003.04390, doi:10.1109/ICCV48922.2021.00893.
URL https://github.com/yinboc/

  * [6] C. Sun, A. Shrivastava, S. Singh, A. Gupta, Revisiting Unreasonable Effectiveness of Data in Deep Learning Era, in: Proceedings of the IEEE International Conference on Computer Vision, Vol. 2017-Octob, IEEE, 2017, pp. 843–852.  arXiv:1707.02968, doi:10.1109/ICCV.2017.97.
URL https://arxiv.org/abs/1707.02968v2

  * [7] L. Yu, W. Zhang, J. Wang, Y. Yu, SeqGAN: Sequence generative adversarial nets with policy gradient, in: 31st AAAI Conference on Artificial Intelligence, AAAI 2017, AAAI, 2017, pp. 2852–2858.  arXiv:1609.05473.
URL https://arxiv.org/abs/1609.05473

  * [8] K. Wang, X. Wan, Sentigan: Generating sentimental texts via mixture adversarial networks, in: IJCAI International Joint Conference on Artificial Intelligence, AAAI, 2018, pp. 4446–4452.  doi:10.24963/ijcai.2018/618.
  * [9] V. Uc-Cetina, N. Navarro-Guerrero, A. Martin-Gonzalez, C. Weber, S. Wermter, Survey on reinforcement learning for language processing, Artificial Intelligence Review (2022).  arXiv:2104.05565, doi:10.1007/s10462-022-10205-5.
URL https://doi.org/10.1007/s10462-022-10205-5

  * [10] J. Luketina, N. Nardelli, G. Farquhar, J. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, T. Rocktäschel, A survey of reinforcement learning informed by natural language, in: IJCAI International Joint Conference on Artificial Intelligence, Vol. 2019-Augus, International Joint Conferences on Artificial Intelligence, 2019, pp. 6309–6317.  arXiv:1906.03926, doi:10.24963/ijcai.2019/880.
URL https://arxiv.org/abs/1906.03926v1

  * [11] Y. Yu, Towards sample efficient reinforcement learning, in: IJCAI International Joint Conference on Artificial Intelligence, Vol. 2018-July, 2018, pp. 5739–5743.  doi:10.24963/ijcai.2018/820.
  * [12] L. FanZhang, L. Yang, W. PengXiang, D. Fang, C. Qi, W. Zhe, A Survey on Recent Advances in Meta-learning, Chinese Journal of Computers 44 (02) (2020) 422–446.
  * [13] F. Mi, M. Huang, J. Zhang, B. Faltings, Meta-learning for low-resource natural language generation in task-oriented dialogue systems, in: IJCAI International Joint Conference on Artificial Intelligence, Vol. 2019-Augus, AAAI, 2019, pp. 3151–3157.  arXiv:1905.05644, doi:10.24963/ijcai.2019/437.
  * [14] K. Qian, Z. Yu, Domain adaptive dialog generation via meta learning, in: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, ACL, 2019, pp. 2639–2649.  arXiv:1906.03520, doi:10.18653/v1/p19-1253.
URL https://aclanthology.org/P19-1253

  * [15] A. Madotto, Z. Lin, C. S. Wu, P. Fung, Personalizing dialogue agents via meta-learning, in: ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, ACL, 2019, pp. 5454–5459.  arXiv:1905.10033, doi:10.18653/v1/p19-1542.
  * [16] J. Zeng, Y. Yin, Y. Liu, Y. Ge, J. Su, Domain Adaptive Meta-Learning for Dialogue State Tracking, IEEE/ACM Transactions on Audio Speech and Language Processing 29 (2021) 2493–2501.  doi:10.1109/TASLP.2021.3080182.
  * [17] R. Li, X. Wang, H. Yu, MetaMT, a meta learning method leveraging multiple domain data for low resource machine translation, in: AAAI 2020 - 34th AAAI Conference on Artificial Intelligence, Vol. 34, AAAI, 2020, pp. 8245–8252.  arXiv:1912.05467, doi:10.1609/aaai.v34i05.6339.
URL https://ojs.aaai.org/index.php/AAAI/article/view/6339

  * [18] R. Zhan, X. Liu, D. F. Wong, L. S. Chao, Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation, in: AAAI 2021 - 35th AAAI Conference on Artificial Intelligence, AAAI, 2021.  arXiv:2103.02262.
URL https://arxiv.org/abs/2103.02262v1 http://arxiv.org/abs/2103.02262

  * [19] S. Indurthi, H. Han, N. K. Lakumarapu, B. Lee, I. Chung, S. Kim, C. Kim, End-end Speech-to-Text Translation with Modality Agnostic Meta-Learning, in: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, Vol. 2020-May, IEEE, 2020, pp. 7904–7908.  doi:10.1109/ICASSP40776.2020.9054759.
  * [20] C. Park, Y. Tae, T. Kim, S. Yang, M. A. Khan, L. Park, J. Choo, Unsupervised Neural Machine Translation for Low-Resource Domains via Meta-Learning, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL, 2021, pp. 2888–2901.  arXiv:2010.09046, doi:10.18653/v1/2021.acl-long.225.
  * [21] K. Qian, W. Wei, Z. Yu, A Student-Teacher Architecture for Dialog Domain Adaptation under the Meta-Learning Setting, in: 35st AAAI Conference on Artificial Intelligence, AAAI 2021, AAAI, 2021.  arXiv:2104.02689.
URL https://arxiv.org/abs/2104.02689v1 http://arxiv.org/abs/2104.02689

  * [22] Y. SONG, Z. Liu, W. Bi, R. Yan, M. Zhang, Learning to Customize Model Structures for Few-shot Dialogue Generation Tasks, in: Proceedings of the 58th Annual Meeting ofthe Association for Computational Linguistics, ACL, 2020, pp. 5832–5841.  arXiv:1910.14326, doi:10.18653/v1/2020.acl-main.517.
  * [23] S. Lin, P. Zhou, X. Liang, J. Tang, R. Zhao, Z. Chen, L. Lin, Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation, in: 35st AAAI Conference on Artificial Intelligence, AAAI 2021, AAAI, 2021.  arXiv:2012.11988.
URL https://arxiv.org/abs/2012.11988v1 http://arxiv.org/abs/2012.11988

  * [24] Y. Zhao, Z. Tian, H. Yao, Y. Zheng, D. Lee, Y. Song, J. Sun, N. Zhang, Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Vol. 1, Long Papers, 2022, pp. 583–595.  arXiv:2203.11670, doi:10.18653/v1/2022.acl-long.44.
  * [25] I. Shalyminov, S. Lee, A. Eshghi, O. Lemon, Few-shot dialogue generation without annotated data: A transfer learning approach, in: SIGDIAL 2019 - 20th Annual Meeting of the Special Interest Group Discourse Dialogue - Proceedings of the Conference, ACL, 2019, pp. 32–39.  arXiv:1908.05854, doi:10.18653/v1/w19-5904.
URL https://arxiv.org/abs/1908.05854v1

  * [26] X. Xu, G. Wang, Y.-B. Kim, S. Lee, AugNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL, 2021, pp. 1183–1195.  arXiv:2106.05589, doi:10.18653/v1/2021.acl-long.95.
URL https://aclanthology.org/2021.acl-long.95

  * [27] E. Chang, X. Shen, H.-S. Yeh, V. Demberg, On Training Instance Selection for Few-Shot Neural Text Generation, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL, 2021, pp. 8–13.  arXiv:2107.03176, doi:10.18653/v1/2021.acl-short.2.
URL https://aclanthology.org/2021.acl-short.2

  * [28] C. Finn, P. Abbeel, S. Levine, Model-agnostic meta-learning for fast adaptation of deep networks, in: 34th International Conference on Machine Learning, ICML 2017, Vol. 3, IMLS, 2017, pp. 1856–1868.  arXiv:1703.03400, doi:10.5555/3305381.3305498.
  * [29] C. Finn, K. Xu, S. Levine, Probabilistic model-agnostic meta-learning, in: Advances in Neural Information Processing Systems, Vol. 2018-Decem, Curran Associates, Inc., 2018, pp. 9516–9527.  arXiv:1806.02817.
URL https://arxiv.org/abs/1806.02817v2

  * [30] J. Yoon, T. Kim, O. Dia, S. Kim, Y. Bengio, S. Ahn, Bayesian Model-Agnostic Meta-Learning, in: Proceedings of the 32nd International Conference on Neural Information Processing System, Curran Associates, Inc., 2018.
  * [31] R. Geng, B. Li, Y. Li, X. Zhu, P. Jian, J. Sun, Induction networks for few-shot text classification, in: EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, ACL, 2019, pp. 3904–3913.  arXiv:1902.10482, doi:10.18653/v1/d19-1403.
  * [32] J. Xu, Q. Du, Learning transferable features in meta-learning for few-shot text classification, Pattern Recognition Letters 135 (2020) 271–278.  doi:10.1016/J.PATREC.2020.05.007.
  * [33] Z. Shi, X. Chen, X. Qiu, X. Huang, Toward diverse text generation with inverse reinforcement learning, in: IJCAI International Joint Conference on Artificial Intelligence, Vol. 2018-July, AAAI, 2018, pp. 4361–4367.  arXiv:1804.11258, doi:10.24963/ijcai.2018/606.
  * [34] A. Holtzman, J. Buys, L. Du, M. Forbes, Y. Choi, The curious case of neural text degeneration, in: International Conference on Learning Representations, ICLR, 2020.  arXiv:1904.09751.
URL https://arxiv.org/abs/1904.09751v2

  * [35] D. Ippolito, D. Duckworth, C. Callison-Burch, D. Eck, Automatic Detection of Generated Text is Easiest when Humans are Fooled, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, 2020, pp. 1808–1822.  arXiv:1911.00650, doi:10.18653/v1/2020.acl-main.164.
  * [36] E. Lagutin, D. Gavrilov, P. Kalaidin, Implicit unlikelihood training: Improving neural text generation with reinforcement learning, in: EACL 2021 \- 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, ACL, 2021, pp. 1432–1441.  arXiv:2101.04229, doi:10.18653/v1/2021.eacl-main.123.
  * [37] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, J. Weston, Neural Text Generation with Unlikelihood Training, in: 8th International Conference on Learning Representations, ICLR 2020, ICLR, 2020.  arXiv:1908.04319.
URL https://arxiv.org/abs/1908.04319v2 http://arxiv.org/abs/1908.04319

  * [38] J. McAuley, C. Targett, Q. Shi, A. Van Den Hengel, Image-based recommendations on styles and substitutes, in: SIGIR 2015 - Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2015, pp. 43–52.  arXiv:1506.04757, doi:10.1145/2766462.2767755.
URL http://dx.doi.org/10.1145/2766462.2767755.

  * [39] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, BLEU: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL, 2002, pp. 311–318.  doi:10.3115/1073083.1073135.
  * [40] D. Alihosseini, E. Montahaei, M. Soleymani Baghshah, Jointly Measuring Diversity and Quality in Text Generation Models, in: Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, ACL, 2019, pp. 90–98.  arXiv:1904.03971, doi:10.18653/v1/w19-2311.
URL https://aclanthology.org/W19-2311

  * [41] Y. Kim, Convolutional neural networks for sentence classification, in: EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, ACL, 2014, pp. 1746–1751.  arXiv:1408.5882, doi:10.3115/v1/d14-1181.