#  Inference-Time Policy Adapters (IPA):
Tailoring Extreme-Scale LMs without Fine-tuning

Ximing Lu♡♣ Faeze Brahman♡♣ Peter West ♡♣ Jaehun Jang ♡
Khyathi Chandu ♣ Abhilasha Ravichander ♣ Lianhui Qin ♡
Prithviraj Ammanabrolu ♡♣ Liwei Jiang ♡♣ Sahana Ramnath ♢
Nouha Dziri ♣ Jillian Fisher ♡ Bill Yuchen Lin ♣ Skyler Hallinan ♡
Xiang Ren ♢♣ Sean Welleck ♡♣ Yejin Choi♡♣
♣Allen Institute for Artificial Intelligence
♡University of Washington ♢University of Southern California

###### Abstract

Large language models excel at a variety of language tasks when prompted with
examples or instructions. Yet controlling these models through prompting alone
is limited. Tailoring language models through fine-tuning (e.g., via
reinforcement learning) can be effective, but it is expensive and requires
model access.

We propose Inference-time Policy Adapters (IPA), which efficiently tailors a
language model such as GPT-3 without fine-tuning it. IPA guides a large base
model during decoding time through a lightweight policy adaptor trained to
optimize an arbitrary user objective with reinforcement learning.

On five challenging text generation tasks, such as toxicity reduction and
open-domain generation, IPA consistently brings significant improvements over
off-the-shelf language models. It outperforms competitive baseline methods,
sometimes even including expensive fine-tuning. In particular, tailoring GPT-2
with IPA can outperform GPT-3, while tailoring GPT-3 with IPA brings a major
performance boost over GPT-3 (and sometimes even over GPT-4). Our promising
results highlight the potential of IPA as a lightweight alternative to
tailoring extreme-scale language models.

##  1 Introduction

Large language models (LLMs) excel at many text generation tasks when prompted
with examples or instructions (Ouyang et al., 2022; Brown et al., 2020).
However, ensuring that desired objectives (e.g., avoiding harmful language)
are met via prompting alone is still limited Lou et al. (2023); Webson and
Pavlick (2021); Liu et al. (2022a); Zong and Krishnamachari (2022)), and often
brittle (Perez et al., 2021; Lu et al., 2022c).

Resource-intensive fine-tuning, through supervised learning, and more recently
reinforcement learning (RL) have shown promise in tailoring language models to
arbitrary objectives. However, they require accessing and updating models’
parameters, which are extremely large or inaccessible in state-of-the-art
models (e.g., GPT-4 OpenAI (2023b)). This makes fine-tuning infeasible for the
broader community.

Figure 1: Inference-time Policy Adapters (IPA) efficiently steer a large-scale
language model (such as GPT-3) during decoding-time through a lightweight
policy adapter trained to reflect any arbitrary user objective with
reinforcement learning.

Alternatively, inference-time algorithms can tailor a language model without
having access to its parameters. These algorithms adjust the model’s output
distribution based on task-specific heuristics, while leaving the underlying
model untouched. However, inference-time heuristics are traditionally hand-
designed for specific tasks (e.g., Lu et al. (2021, 2020); Liu et al. (2021a);
Yang and Klein (2021); Qin et al. (2022a)) and inference-time methods are
often less effective than fine-tuning a model with reinforcement learning Lu
et al. (2022a).

Drawing inspiration from RL and inference-time techniques, we propose
Inference-time Policy Adapters ( IPA), which tailors a base language model at
inference-time toward arbitrary task-specific objectives without the need to
fine-tune it. To do so, IPA combines a large base language model’s output
distribution with that of a smaller-sized model (an adapter policy), and
optimizes the combined distribution towards a given objective with
reinforcement learning. IPA uses two key ideas to make learning efficient.
First, IPA only updates the adapter’s parameters, avoiding the need to update
the base LM. Second, IPA replaces the base model with an approximate policy–a
smaller model that approximates the base model’s distribution. The approximate
policy is either a smaller model from the same language model family or a
distilled version of the base model. At inference time, we use the combined
distribution of the base model and the trained policy adapter.

Experiments across five challenging text generation tasks show that IPA brings
consistent improvements over off-the-shelf language models and competitive
baselines, sometimes even including expensive fine-tuning. In particular,
tailoring GPT-2 with IPA can outperform GPT-3, while tailoring GPT-3 with IPA
brings a major performance boost over GPT-3 (and sometimes even over GPT-4).
Our results highlight IPA as a lightweight alternative for tailoring large
language models to a wide range of objectives. IPA opens new ways to augment
or customize LLMs using only academic-level resources.

##  2 Background

In this section, we introduce our text generation setting (§2.1) and a brief
background on tailoring language models with reinforcement learning (§2.2). We
then introduce our IPA algorithm for tailoring large language models without
fine-tuning (§3).

###  2.1 Problem Setting

Text generation is the task of generating an output sequence y given an input
sequence x. We consider standard autoregressive language models, which
decompose a sequence’s probability as
$p_{\theta}(\textbf{y}|\textbf{x})=\prod^{|\textbf{y}|}_{t=1}p_{\theta}(\textbf{y}_{t}|\textbf{y}_{<t},\textbf{x})$,
where $p_{\theta}$ is a neural network with parameters $\theta$. Intuitively,
our goal is to ‘tailor’ a pretrained model $p_{\theta}$ towards a user-
specified objective (e.g., safety). Concretely, we assume that the objective
is quantified by a reward function $\mathcal{R}(\textbf{y})\in\mathbb{R}$. We
then aim to adjust $p_{\theta}$ so that its generated sequences have high
reward and reasonable language quality (e.g., fluency).

###  2.2 Preliminary: Tailoring LMs with RL

Online policy-based reinforcement learning has emerged as an effective way to
adjust a language model towards a reward function. Formally, these algorithms
(e.g., PPO Stiennon et al. (2022), Quark Lu et al. (2022b), or NLPO
Ramamurthy* et al. (2023)) optimize a language model $p_{\theta}$ towards
generating outputs y that maximize a given reward $\mathcal{R}$:

$\displaystyle\theta^{\star}=\arg\max\mathbb{E}_{\textbf{y}\sim
p_{\theta}(\cdot|\textbf{x})}\mathcal{R}(\textbf{y}),$

often along with regularization to maintain language quality. At a high-level,
these algorithms use a policy $p_{\theta}$ to collect input-output examples,
score the outputs with a reward function $\mathcal{R}$, and produce an
optimized policy $p_{\theta^{\star}}$. Formally,

$\displaystyle
f_{\textnormal{RL}}:(p_{\theta},\mathcal{R};\theta^{\prime})\rightarrow\theta^{\star}.$
(1)

Here $\theta^{\prime}\subseteq\theta$ denotes the subset of $p_{\theta}$’s
parameters that are updated by the algorithm. The key idea behind IPA is to
use a full model $p_{\theta}$ to collect examples, but update a small set of
parameters $\theta^{\prime}$.

##  3 Inference-time Policy Adapters (IPA)

We introduce Inference-time Policy Adapters (IPA), a lightweight approach to
tailoring language models toward a user-specified objective. IPA trains a
small adapter policy that adjusts the outputs of a (larger) base model at
inference-time in order to maximize a reward. In doing so, IPA avoids the cost
of updating the large base model, without the need to hand-design inference-
time heuristics.

###  3.1 Policy Adaptation

First, we introduce the notion of ‘tailoring’ used by IPA, which mainly
involves three policies. First, IPA starts with a base policy $p_{\theta}$,
which is the language model to tailor. Second, IPA introduces an adapter
policy $p_{\phi}$, which is a language model with the same output space as the
base policy (i.e., vocabulary), but different parameters $\phi$. Finally, IPA
combines the base and adapter policies into a tailored policy:

######  Definition 1 (Tailored policy).

The tailored policy $p_{\theta\leftarrow\phi}$ combines the distributions of
the base policy $p_{\theta}$ and the adapter policy $p_{\phi}$,

$\displaystyle
p_{\theta\leftarrow\phi}(\textbf{y}_{t}|\textbf{y}_{<t})=\frac{1}{Z}p_{\theta}(\textbf{y}_{t}|\textbf{y}_{<t})p_{\phi}(\textbf{y}_{t}|\textbf{y}_{<t}),$

where $Z$ is a normalization factor.

The tailored policy is a product-of-experts Hinton (2002), which amounts to
multiplying the next-token probabilities from the base and adapter policies,
then normalizing the result. IPA’s tailored policy has two key properties.
First, it allows for adjusting the base policy’s output without direct access
to the base policy’s parameters. This is critical for tailoring modern LLMs
that provide access to the model’s output distribution but not the model’s
parameters. Second, the policy adapter can use a much smaller model (i.e.,
$\phi\ll\theta)$. This provides an efficient way to tailor a large base model.

###  3.2 Adapter Training with RL

Our goal is to adjust the tailored policy towards a user-specified objective.
The key idea in IPA is to train the tailored policy to optimize a reward with
reinforcement learning, while only updating the parameters of the adapter
policy.

Concretely, we use a reinforcement learning algorithm $f_{\textnormal{RL}}$
(Eqn. 1) to optimize the tailored policy $p_{\theta\leftarrow\phi}$ with a
reward function $\mathcal{R}$. Notably, we keep the base policy’s parameters
($\theta$) frozen, and only update the adapter policy’s parameters ($\phi$).
That is,

$\displaystyle\phi^{\star}=f_{\textnormal{RL}}\>(p_{\theta\leftarrow\phi},\mathcal{R};\phi).$

Intuitively, the adapter policy $p_{\phi}$ learns to rescale the frozen base
policy $p_{\theta}$, yielding a tailored policy that is ‘tailored to’ the
reward. Notice that our framework does not depend on a specific RL algorithm.
As we will demonstrate later, IPA proves to be effective when paired with
three different RL algorithms Lu et al. (2022b); Schulman et al. (2017);
Ramamurthy et al. (2023), and in principle, it can easily integrate with
others.

#### Approximate Policy.

When the base model is extremely large (e.g., GPT-3), its forward pass is too
costly to be used in the RL training loop. To overcome this, we propose using
an approximate policy in IPA.

######  Definition 2 (Approximate policy).

The approximate policy is defined as a smaller-sized neural model
parameterized by $\hat{\theta}$ that approximates the distribution of the base
policy and is used to replace the base policy in the RL-based adapter
training:

$\displaystyle\phi^{\star}=f_{\textnormal{RL}}\>(p_{\hat{\theta}\leftarrow\phi},\mathcal{R};\phi).$

In practice, we can obtain an approximate policy in two different ways. First,
we can use a smaller pre-trained language model from the same model family. We
do this if the smaller model has similar conditional generation behavior as
the base policy. For instance, we use an off-the-shelf GPT2-XL as the
approximate policy to tailor GPT-3 in an open-ended generation. Alternatively,
we can use a distilled base policy as the approximate policy. A distilled base
policy is a language model trained on generations from the base policy,
$\hat{\theta}=\arg\max\mathbb{E}_{\textbf{y}\sim
p_{\theta}(\cdot|\textbf{x})}\big{[}\log P_{\hat{\theta}}(\textbf{y})\big{]}$,
known as sequence-level knowledge distillation Kim and Rush (2016); West et
al. (2022). For example, to tailor GPT-3 for lexically constrained generation,
we tune GPT2-XL on prompt-generation pairs from GPT-3 to get a distilled base
policy.

#### IPA at Inference Time.

At inference time, IPA uses the tailored policy $p_{\theta\leftarrow\phi}$ for
decoding. Namely, at each time-step we obtain the next-token distribution from
the tailored policy
$p_{\theta\leftarrow\phi}(\textbf{y}_{t}|\textbf{y}_{<t})$, which can then be
used with a standard decoding algorithm (e.g. nucleus sampling).

##  4 Experiments

We evaluate IPA on a diverse range of tasks: toxicity reduction (§4.1),
lexically constrained generation (§4.2), open-ended generation (§4.3),
dialogue safety control (§4.4), and knowledge-grounded dialogue (§4.5). Across
all benchmarks, IPA brings significant improvement over off-the-shelf large-
scale language models such as GPT-3 (text-davinci-003), and outperforms all
existing competitive baselines, sometimes even including expensive fine-tuned
versions of GPT-3, while costing only a fraction of the price.

###  4.1 Toxicity Reduction

Language models are susceptible to generating toxic completions, even when
prompted with seemingly innocuous text Gehman et al. (2020). Here, we
investigate the extent to which IPA is effective in reducing the toxicity of
language models in an open-ended setting.

#### Datasets and Metrics.

The task is to generate a fluent continuation $y$ while avoiding offensive
content for a given prompt $x$. We evaluate on RealToxicityPrompts benchmark
Gehman et al. (2020), which contains 100k prompts designed to elicit toxic
generations. Following the experimental setup of Liu et al. (2021b), we use
Perspective API to measure maximum toxicity, defined as the average maximum
toxicity over 25 sampled generations, and the (empirical) toxicity probability
of at least 1 out of 25 generations being toxic. In addition, we report
fluency as the perplexity of generated output according to an off-the-shelf
GPT2-XL model, and diversity as the count of unique n-grams normalized by the
length of text.

#### Setup and Baselines

We apply IPA to tailor two types of base policy: off-the-shelf GPT-2 and
GPT-3. To tailor GPT-2, we directly apply the base policy in the adaptor
training, referred to as IPA(GPT-2). For tailoring GPT-3, we use an off-the-
shelf GPT-2 and a distilled GPT-3 as the approximate policy for the adaptor
training, labeled as IPA-(GPT-3) and IPA*(GPT-3) respectively. Notice that
IPA-(GPT-3) is equivalent to directly applying the policy adaptor trained to
tailor GPT-2 on top of GPT-3. In all these scenarios, we initialize the policy
adaptor with a pre-trained GPT2-large model.

We use Quark as the RL algorithm to optimize the policy adaptor, and conduct
an ablation study to assess the effects of using different RL algorithms in
the adaptor training. As the reward function, we use the Perspective API score
which measures the toxicity of the completed sequence. During inference, we
use nucleus sampling with p = 0.9 to generate 25 samples for all baselines.

For tailoring GPT-2, we compare IPA with previously reported baselines from Lu
et al. (2022a), including decoding-based methods: PPLM (Dathathri et al.,
2020a), GeDi (Krause et al., 2021), DExpert (Liu et al., 2021a), and learning-
based methods: DAPT (Gururangan et al., 2020), PPO (Schulman et al., 2017),
and Quark (Lu et al., 2022a). For tailoring GPT-3, we compare IPA to the
baselines described above that are compatible with GPT-3’s limited
accessibility: DExpert (Liu et al., 2021a) and DAPT (Gururangan et al., 2020).

#### Results

Models | Toxicity | Fluency | Diversity
---|---|---|---
Avg Max. | Prob. | Pl. | Dist-2. | Dist-3.
GPT-2 | 0.527 | 0.520 | 11.31 | 0.85 | 0.85
PPLM | 0.520 | 0.518 | 32.58 | 0.86 | 0.86
GeDi | 0.363 | 0.217 | 60.03 | 0.84 | 0.83
Dexperts | 0.314 | 0.128 | 32.41 | 0.84 | 0.84
DAPT | 0.428 | 0.360 | 31.21 | 0.84 | 0.84
PPO | 0.218 | 0.044 | 14.27 | 0.80 | 0.84
Quark | 0.196 | 0.035 | 12.47 | 0.80 | 0.84
IPA (GPT-2) | 0.138 | 0.031 | 11.94 | 0.80 | 0.84
GPT-3 | 0.275 | 0.197 | 10.65 | 0.78 | 0.81
Dexperts | 0.223 | 0.112 | 23.41 | 0.79 | 0.82
DAPT | 0.254 | 0.176 | 20.19 | 0.80 | 0.83
IPA- (GPT3) | 0.150 | 0.056 | 10.34 | 0.79 | 0.81
IPA* (GPT3) | 0.101 | 0.028 | 12.68 | 0.79 | 0.83

Table 1: Automatic evaluation for Toxicity Reduction with off-the-shelf
GPT2-large (top) and GPT-3 (bottom) as the base policy to tailor.

RL Algo. | Toxicity | Fluency | Diversity
---|---|---|---
Avg Max. | Prob. | Pl. | Dist-2. | Dist-3.
Quark | 0.138 | 0.031 | 11.94 | 0.80 | 0.84
PPO | 0.125 | 0.029 | 12.47 | 0.80 | 0.84
NLPO | 0.136 | 0.032 | 12.13 | 0.80 | 0.85

Table 2: Comparison of using different RL algorithm for training IPA for
Toxicity Reduction with off-the-shelf GPT2-large as the base policy to tailor.

Table 1 shows the results of the toxicity reduction task. In terms of
tailoring GPT-2, IPA outperforms all learning-based and decoding-based methods
on the toxicity score while maintaining the language quality (i.e., fluency
and diversity). It is worth noting that IPA shows improvements compared to
previous methods on keeping the fluency metric untouched from the base policy,
suggesting that the adapter policy effectively injects desirable quality into
the base policy while keeping their existing good quality.

As for tailoring GPT-3, we found that applying the policy adapter optimized
for GPT-2 directly on top of GPT-3 (denoted as IPA-) leads to a noticeable
reduction in toxicity while maintaining language quality, demonstrating the
adaptability and reusability of IPA. We observed further improvement when
using distilled GPT-3 as the approximate policy in adapter training to tailor
GPT-3 (denoted as IPA*). In both cases, our method significantly outperforms
all previous baselines for controlling GPT-3, especially the costly domain
adaptive training (DAPT) which exhaustively fine-tune GPT-3 on a non-toxic
corpus. Our results highlight the promise of IPA as a cost-efficient way to
align large language models with user-specified objectives without the need
for fine-tuning.

Finally, as shown in Table 2, ablation studies on using different RL
algorithms for optimizing the policy adaptor show that IPA can smoothly
accommodate different RL algorithms, all leading to better performance
compared to other baselines.

###  4.2 Lexically Constrained generation

Next, we evaluate the effectiveness of IPA in lexically constrained
generation, where given a set of constraint words, the model needs to generate
a sentence that includes all the given constraints. While prior works on this
task evaluated the order-invariant satisfaction of the constraints – i.e. the
generation was deemed to be correct when it simply includes all the keywords
Lin et al. (2020); Lu et al. (2020), we consider a more challenging setup of
ordered lexical constraints, where the generation is considered correct when
it includes all the keywords with the correct order specified in the input
prompt.

#### Datasets and Metrics.

We use CommonGen Lin et al. (2020), a dataset for generative commonsense
reasoning where the task is to generate a coherent sentence given a set of
concept words. To evaluate both the inclusion and order of the given
constraints, we deliberately instruct the models to generate a sentence with
the given keywords while following the order they appear in the input prompt.
For automatic evaluation, we gauge the constraint satisfaction with coverage,
a binary metric that evaluates a generation to be correct only when it
includes all the given words and also matches the order in which they appear.
We also measure the fluency of each generation using a critic model fine-tuned
on CoLA Warstadt et al. (2019). For human evaluation, we assess the quality
and plausibility of model generations for 100 randomly sampled test examples
based on a 3-point Likert Scale; see details Appendix Figure 5.

Models | Automatic | Human
---|---|---
Cov. | Fl. | Qu. | Pl. | Overall
GPT-3 | 37.01 | 94.89 | 2.84 | 2.81 | 2.60
GPT-3.5 | 65.17 | 95.89 | 2.93 | 2.88 | 2.90
GPT-4 | 84.81 | 95.49 | 2.95 | 2.97 | 2.96
GPT-3${}_{\text{sft}}$ | 72.89 | 73.96 | 2.56 | 2.60 | 2.50
$\text{IPA}^{*}$ (GPT-3) | 88.54 | 92.58 | 2.90 | 2.87 | 2.88

Table 3: Automatic and human evaluation results for Lexically Constrained
Generation. Human evaluation scores are on a 3-point Likert Scale.

#### Setup and Baselines.

As we’ll demonstrate later, GPT-3 (text-davinci-003) is surprisingly poor at
faithfully following ordered lexical constraints, even after being given
explicit instructions. Our goal is to tailor GPT-3 to be more reliable for
constraint satisfaction in zero-shot setting. We use distilled GPT3, a
GPT-2-XL fine-tuned with prompt-output pairs from GPT-3’s zero-shot
generations on CommonGen train set as the approximate policy for adaptor
training, since an off-the-shelf GPT-2 cannot perform lexically constrained
generation out of the box. We initialize the policy adaptor with a pre-trained
GPT2-large model. We use Quark as the RL algorithm for adaptor optimization
and choose our reward to be the multiplication of the coverage score of the
lexical constraints and the fluency score inferred by the CoLA model,
promoting the fluency of the generation while satisfying the ordered lexical
constraints.

We compare IPA with its base policy GPT-3, as well as the more advanced LLMs:
GPT-3.5 and GPT-4 OpenAI (2023a). As a strong supervised baseline, we also
fine-tune GPT-3 on the CommonGen train set, which contains human-written
outputs with the correct lexical order, denoted as GPT-3${}_{\text{sft}}$.

#### Results.

We present the experimental results in Table 3. Surprisingly, powerful
language models such as GPT-3 struggle at faithfully following ordered lexical
constraints even after being given explicit instructions. IPA leads to
remarkable improvement top of of GPT-3 and surpasses more advanced models such
as GPT-3.5 and GPT-4 in terms of constraint coverage, while achieving better
or comparable generation quality. Noticeably, IPA outperform fine-tuned GPT-3
in both constraint coverage and generation quality at a fraction of the cost:
while fine-tuning GPT-3 costs $156.82, training a distilled GPT-3 as the
approximate policy requires only $28.59 for generating outputs from GPT-3. Our
results highlight the potential of the IPA as a cost-efficient way to enhance
the capability of large language models without the need for fine-tuning.

###  4.3 Open-ended generation

To evaluate the effectiveness of IPA in a general setting, we conduct
experiments on an open-ended generation task; following the experimental setup
in Li et al. (2022b). The goal is to make machine-generated content more
fluent, coherent, and human-like.

#### Datasets and Metrics.

We experiment on the news domain using XSum dataset Narayan et al. (2018).
Following Li et al. (2022b), we filter out news articles with fewer than 160
tokens. We use the first 32 words as our input prompt, and generate 84 tokens
as continuations. We evaluate using both automatic and pairwise human
evaluation. For automatic evaluation, we use aggregate n-gram diversity and
coherence scores Li et al. (2022b) as well as MAUVE Pillutla et al. (2021)
which measures the distribution similarity between the set of human-written
gold and machine-generated texts. To measure the human-likeness of generated
texts, we employ OpenAI detector111https://github.com/promptslab/openai-
detector, a classifier for distinguishing AI vs. human-written text. We use
the classifier’s probability assigned to ‘human’ text to serve as an
additional metric, denoted as Critic.

For human evaluation, we randomly sample 100 test examples and generate
continuations using different models. We conduct pairwise comparisons (of our
method against baselines) on coherence and fluency using Amazon Mechanical
Turk; see details Appendix Figure 4.

#### Setup and Baselines.

We apply IPA to tailor two types of base policy: off-the-shelf GPT2-XL and
GPT-3. To tailor GPT-2, we directly apply the base policy in the adaptor
training, referred to as IPA(GPT-2). For tailoring GPT-3, we use an off-the-
shelf GPT2-XL and a distilled GPT-3 as the approximate policy for the adaptor
training, denoted as IPA-(GPT-3) and IPA*(GPT-3) respectively. Notice that
IPA-(GPT-3) is equivalent to directly applying the policy adaptor trained to
tailor GPT-2 on top of GPT-3. In all these scenarios, we initialize the policy
adaptor with a pre-trained GPT2-large model, and use Quark as the RL algorithm
to optimize the policy adaptor. As the reward function, we use the
multiplication of diversity, coherence, and critic scores described above.

| (decoding)
---
Method
Diversity | Coherence | Critic | Mauve
base policy: GPT2-XL
greedy | 55.05 | 49.57 | 7.88 | 15.32
top-k (k=50) | 92.60 | 48.53 | 10.72 | 53.13
top-p (p=0.95) | 95.85 | 47.61 | 13.24 | 56.42
typical ($\tau$=0.95) | 95.80 | 46.08 | 23.49 | 63.92
SimCTG | 95.67 | 46.12 | 23.67 | 62.21
Contrastive | 95.99 | 49.42 | 36.73 | 61.95
IPA (GPT2-XL) | 96.12 | 51.81 | 50.93 | 84.18
base policy: GPT-3
top-p (p=0.95) | 95.63 | 56.16 | 18.58 | 63.73
IPA- (GPT-3) | 95.35 | 57.26 | 22.62 | 71.40
IPA* (GPT-3) | 96.26 | 61.94 | 32.84 | 73.17
Table 4: Automatic evaluation results for open-domain generations on XSum with
off-the-shelf GPT2-XL (top) and GPT-3 (bottom) as the base policy to tailor.
Critic scores refer to human-likeness according to OpenAI detector.  Figure 2:
Pairwise human evaluation in terms of overall quality for open-ended
generation on XSum with off-the-shelf GPT2-XL (top) and GPT-3 (bottom) as the
base policy to tailor.

For tailoring GPT-2, we compare decoding with IPA with six different decoding
strategies: greedy, top-k sampling ($k=50$), nucleus sampling ($p=0.95$),
typical sampling ($\tau=0.95$) Meister et al. (2023), SimCTG Su et al. (2022),
and Contrastive decoding Li et al. (2022b). The latter three methods are
designed to improve the coherence and naturalness of the generated text. For
all decoding baselines, we use the recommended hyper-parameters in the papers.
For tailoring GPT-3, we compare decoding with IPA with GPT-3’s default
decoding strategy: nucleus sampling ($p=0.95$). Other decoding methods are not
applicable to GPT-3 due to its limited access through API interface.

#### Results.

As shown in the top section of Table 4, when tailoring GPT-2, IPA
significantly outperforms all other decoding baselines across all automatic
metrics. Notably, it achieves an absolute improvement of 20.26% over the best-
performing baseline in the Mauve score which has shown a high correlation with
the human judgment of quality. Our pairwise human evaluation in Figure 2 also
verify the results. IPA generates significantly more coherent and fluent texts
compared to other decoding baselines. Overall, on average, human evaluators
preferred IPA 1.8$\times$ more than other decoding baselines.

Regarding tailoring GPT-3, as shown in the bottom section of Table 4, directly
applying the policy adapter optimized for GPT-2 on top of GPT-3 (denoted as
IPA-) results in a remarkable improvement in generation quality across all
automatic metrics, demonstrating the adaptability and reusability of IPA. We
observed further improvement when using distilled GPT-3 as the approximate
policy in adapter training to tailor GPT-3 (denoted as IPA*). Our findings are
verified by the results of human evaluation in Figure 2, which shows a
noticeable preference by human evaluators for outputs from the GPT-3 tailored
by IPA over the off-the-shelf GPT-3.

###  4.4 Dialogue Safety Control

Models | Automatic |  | Human
---|---|---|---
Safety |  | Safety | Coherence
DialoGPT | 0.46 |  | 1.34 | 2.45
Godel | 0.49 |  | 1.40 | 2.53
Blenderbot | 0.53 |  | 1.43 | 2.60
ChatGPT | 0.74 |  | 1.60 | 2.68
IPA- (BlenderBot-3B) | 0.78 |  | 1.57 | 2.75
Table 5: Automatic and human evaluation results for Dialogue Safety Control.
Human evaluation scores are on a 3-point Likert Scale.

Existing dialogue systems often fail to respond safely to potentially unsafe
user utterances Kim et al. (2022), limiting their deployment in real-world
applications. Here, we aim to evaluate IPA for controlling the safety of a
dialogue model.

#### Datasets and Metrics.

We experiment on DiaSafety Sun et al. (2022), a challenging dataset containing
54K context-sensitive unsafe examples. The task is to generate a coherent
response to a potentially unsafe utterance while avoiding offensive, harmful,
toxic or biased language. DiaSafety contains human-written safe and unsafe
responses which we use to train a dialogue safety classifier. We use the
classifier score as an automatic measure of safety. In addition, we conduct a
human evaluation of safety and coherence (3-point Likert scale) for 200
randomly sampled test examples on Amazon Mechanical Turk; see Appendix A
Figure 3 for details.

#### Setup and Baselines.

We apply IPA to tailor the Blenderbot family models Roller et al. (2021),
which are pretrained dialogue agents. Specifically, we use
Blenderbot-3B-distill as the frozen base policy, a samller
Blenderbot-1B-distill as the approximate policy and initialize the policy
adaptor with a Blenderbot-1B-distill model.222BlenderBot checkpoints are
available at: https://huggingface.co/models We use Quark as the RL algorithm
to optimize the policy adaptor. To preserve coherence and engagingness while
controlling the safety of a dialogue response, we choose our reward to be the
multiplication of the safety score from our trained dialogue safety
classifier, as well as coherence and engagingness scores from UniEval-Dialogue
Zhong et al. (2022).333https://github.com/maszhongming/UniEval During
inference, we use Nucleus sampling Holtzman et al. (2020) with $p=0.6$ and
temperature 1.0.

We compare IPA with its base policy, i.e., Blenderbot-3B-distill, and other
off-the-shelf dialogue models including DialoGPT Zhang et al. (2020), GODEL
Peng et al. (2022) as well as ChatGPT OpenAI (2022). ChatGPT is known to have
safeguards through content filtering and is considered a strong baseline.

#### Results.

As shown in Table 5, IPA significantly improves the dialogue safety and
coherence over its base policy Blenderbot-3B-distill, surpassing other
dialogue models including DialoGPT and GODEL. In comparison with ChatGPT, IPA
achieves comparable performance on safety based on both automatic and human
evaluation while showcasing improved coherence. Upon further investigation, we
found that ChatGPT often generates canned responses like "I’m a language
model; I’m not allowed…" as hard safeguards, which hurts the coherence and
naturalness of the dialogue flow. On the other hand, Blenderbot tailored by
IPA can generate safe responses that are coherent, natural, and human-like.
Our results demonstrate the potential of IPA to enhance controllability in
various NLP applications beyond conditional text generation.

###  4.5 Knowledge-grounded Dialogue

Ideally, knowledge-grounded dialogue systems should generate responses that
are faithful to the given knowledge $K$. However, models tend to generate
hallucinated responses that contain unverifiable information Dziri et al.
(2022a); Rashkin et al. (2021a); Dziri et al. (2022c). To address this
undesirable behavior, we use IPA to tailor dialogue model towards generating
more faithful content. Given the knowledge $K$ and the conversation history
$H$, the task is to generate a response $r$ that’s faithful to $K$ and
coherent with $H$.

#### Dataset and Metrics

We evaluate on the Wizard of Wikipedia (WoW) data. WoW Dinan et al.  involves
a Wizard and an Apprentice engaging in a conversation. The Wizard’s role is to
provide information on a specific topic, while the Apprentice’s task is to
seek further details. WoW has been shown to suffer from hallucinations Dziri
et al. (2022b), in more than 60% of the turns, making it a valuable dataset
for studying and addressing hallucination issues. To evaluate the faithfulness
of responses and rate them against the knowledge snippets and gold responses,
we use the FaithDial test data Dziri et al. (2022a) at test time. FaithDial is
a hallucination-free benchmark created by modifying the hallucinated responses
within the WoW dataset.

To measure faithfulness, we use the critic model Dziri et al. (2022a), which
returns the percentage of utterances identified as faithful. Additionally, we
use BERTScore to measure the semantic similarity between the generated
response $r$ and the knowledge $K$. We compare the token-level F1 score to
rate the lexical overlap between $r$ and $K$. To measure coherence and
engagingness, we use UniEval model Zhong et al. (2022).

| Dialogue
---
Model
Critic | BERTScore | F1 | Coherence | Engaging
supervised baseline
GPT-2 | 39.9 | 0.29 | 47.7 | 0.77 | 1.26
DialoGPT | 40.6 | 0.34 | 53.5 | 0.83 | 1.32
DOHA | 46.8 | 0.32 | 56.1 | 0.88 | 1.33
T5 | 53.5 | 0.41 | 61.7 | 0.86 | 1.28
T5-CTRL | 54.8 | 0.45 | 65.2 | 0.83 | 1.21
T5-LT | 58.6 | 0.43 | 65.0 | 0.83 | 1.21
off-the-shelf dialogue model
BlenderBot | 10.3 | 0.12 | 9.8 | 0.92 | 1.21
IPA-(BlenderBot) | 76.6 | 0.68 | 80.1 | 0.91 | 1.34

Table 6: Evaluation results for knowledge-grouded dialogue generations on
Faithdial. We use off-the-shelf Blenderbot as the base policy to tailor.

#### Setup and Baselines

Similar to the dialogue safety experiment, we use the Blenderbot-{3,
1}B-distill model Roller et al. (2021) as our base policy and approximate
policy respectively, and initialize the policy adaptor with a
Blenderbot-1B-distill model. We use Quark as the RL algorithm to optimize the
policy adaptor. To preserve coherence and engagingness while ensuring the
faithfulness of a dialogue response, we choose our reward to be the
multiplication of the faithfulness score from the critic model described
above, as well as coherence and engagingness scores from UniEval-Dialogue
Zhong et al. (2022).444https://github.com/maszhongming/UniEval During
inference, we use Nucleus sampling Holtzman et al. (2020) with $p=0.6$ and
temperature 1.0.

We compare to previously reported baselines from Dziri et al. (2022a),
supervised models fine-tuned on WoW, including GPT2, DialoGPT Zhang et al.
(2020), DoHA Prabhumoye et al. (2021) T5 Raffel et al. (2020), T5-CTRL Rashkin
et al. (2021b), and T5-LossTruncation Kang and Hashimoto (2020). We also
compare against the base policy, off-the-shelf BlenderBot model Roller et al.
(2021).

#### Results

As shown in Table 6, supervised models struggle to generate faithful dialogue
response grounded on the given knowledge. This is mainly because of the poor
data quality of their supervision dataset: WoW has been shown to suffer from
hallucinations in more than 60% of the turns Dziri et al. (2022a). Moreover,
pre-trained dialogue models like Blenderbot demonstrate even worse performance
at generating faithful response, despite being trained on WoW and other
knowledge-grounded dialogue datasets in their pre-training stage. IPA
significantly improves the faithfulness of the generated dialogue response
over its base policy Blenderbot while preserving the dialogue quality (i.e.,
coherence and engagingness), outperforming all other baselines. Our results
showcases the potential of IPA to improve reliability and trustworthiness of
NLP systems in various downstream applications.

##  5 Related Work

#### Controlled Decoding

Recent studies have explored controlled generation at inference time by
designing new decoding algorithms Keskar et al. (2019); Mireshghallah et al.
(2022); Li et al. (2022a); Chen et al. (2022); Zhang et al. (2022). For
example, to ensure the inclusion of given keywords, Neurologic decoding Lu et
al. (2020), and GBS Hokamp and Liu (2017) generalize beam search for lexically
constrained decoding, by constraining the decoding space with keyword-related
penalties. DExperts Liu et al. (2021b) modifies the output distribution during
decoding with attribute-specific expert models. However, these decoding
methods are designed for particular control types only. Another line of
research develops gradient-based decoding for more general control Qin et al.
(2020, 2022b); Sha (2020); Dathathri et al. (2020b); Kumar et al. (2021). For
example, COLD Decoding Qin et al. (2022b) introduces energy-based modeling to
impose arbitrary constraints on text and samples with Langevin dynamics. PPLM
Dathathri et al. (2020b) steers generation towards desirable attributes via
the gradients from a small fine-tuned model. Despite their progress, these
approaches rely on computationally expensive gradient computations.

#### Reinforcement Learning for Natural Language Generation

RL has historically been used in multiple NLG tasks such as machine
translation (Wu et al., 2016; Nguyen et al., 2017), summarization (Paulus et
al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017), text games
(Narasimhan et al., 2015; Hausknecht et al., 2020), etc. to ensure that the
generated text is optimized for an arbitrary non-differentiable reward. This
was often done using online policy gradient methods such as REINFORCE (Sutton
and Barto, 2018), leading to documented issues with reward hacking—where a
model has great metrics but fails to solve the spirit of the task and produces
inarticulate sounding text (Choshen et al., 2020; Kiegeland and Kreutzer,
2021). Recent advances use the success of LMs in modeling language fluency to
introduce a KL reward penalty which significantly increases the naturalness of
generated text (Ouyang et al., 2022; Korbak et al., 2022). This method has
been used extensively to tune a base LM via online on-policy (Ramamurthy* et
al., 2023), off-policy (Guo et al., 2022; Lu et al., 2022b), and offline
(Snell et al., 2023; Korbak et al., 2023) RL. Such methods quickly become
computationally infeasible for extreme-scale LMs with billions of parameters
and thus IPA tunes a separate smaller policy designed to adapt an existing
larger LM.

#### Refinement.

Recent work controls (L)LMs by refining a generated sequence into an improved
one with a refinement module Yasunaga and Liang (2020); Saunders et al.
(2022); Schick et al. (2022); Yang et al. (2022); Welleck et al. (2023);
Madaan et al. (2023). These methods operate in the sequence space, while IPA’s
adaptor policy makes fine-grained ‘refinements’ in the simplex (i.e., on next-
token distributions). Typically the refiner is large (e.g., Saunders et al.
(2022); Madaan et al. (2023)), or depends on specialized training data Schick
et al. (2022) or learning algorithms Welleck et al. (2023). IPA’s adaptor
policy is lightweight, and is directly optimized with standard RL algorithms.

#### Parameter-Efficient Fine-Tuning

Prompting and prefix-tuning Li and Liang (2021) adapt a very large model to a
specific task. However, they are affected by sensitivity based on order of
words or examples Zhao et al. (2021); Webson and Pavlick (2022), lack
associative clarity Min et al. (2022) and tuning prompts work for only very
large models Mahabadi et al. (2021); Liu et al. (2022b). These methods compose
the input to the model. In contrast, parameter-efficient finetuning offers a
clean way to compose parameters directly by adding or updating a smaller
subset of model parameters. A common strategy is to prune the model parameters
and introduce sparsity Han et al. (2017); Frankle and Carbin (2019); Frankle
et al. (2020). The effectiveness of this approach is also substantiated with
the use of RL Yu et al. (2020). Instead of pruning individual units,
structured-pruning prunes an entire group, such as attention heads in
pretrained models Michel et al. (2019); Voita et al. (2019). Additionally, Li
et al. (2018) demonstrate the effectiveness of optimizing a model in a low-
dimensional randomly oriented subspace. Later studies Aghajanyan et al. (2021)
have also shown that the intrinsic dimensionality decreases with pretraining
larger models. Hu et al. (2022) learns a low-rank factorization via projection
matrix and applies them to the self-attention weights. Recently, adding a
small subset of parameters called adapters Rebuffi et al. (2017) and compact
adapters Mahabadi et al. (2021) which are model-specific Stickland and Murray
(2019). Pfeiffer et al. (2020) introduced a continuously evolving Adapter-Hub
that stitches different pre-trained adapters for languages and tasks inspired
from routing networks Rosenbaum et al. (2019) optimized through reinforcement
learning Kirsch et al. (2018); Chang et al. (2019). Though these methods are
efficient, they require access to the internal representation for model and
gradient, which is not feasible for large models like GPT3 with limited
access.

##  6 Conclusion

In this work, we introduce IPA, a lightweight Inference-time Policy Adapter
that tailor a frozen large language model towards desirable properties (e.g.,
safety, coherence, faithfulness) in an efficient, generalizable, and flexible
way. IPA inherits the generalizability of the RL approach to enable tailoring
models with arbitrary objectives while incorporating plug-and-play flexibility
of the inference-time techniques. IPA can effectively customize the base
policy models to specified desirable objectives without sacrificing the
existing favorable attributes of the base models. Crucially, IPA enables
tailoring massive-size large language models that are generally untunable due
to computational cost or restricted API access. As IPA operates independently
of the choice of the online learning RL algorithm, it is flexible to
accommodate future alternative optimization techniques. Extensive experiments
across five challenging and diverse text generation tasks demonstrate that IPA
consistently outperforms competitive baselines, for both small and massive
model sizes, sometimes even beating expensive fine-tuning. We hope our work
sheds light on creative and efficient algorithmic innovations to complement
the pursuit of model scales leveraging academic-level resources.

####

## References

  * Aghajanyan et al. (2021) Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 7319–7328. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chang et al. (2019) Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. 2019.  Automatically composing representation transformations as a means for generalization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Chen et al. (2022) Howard Chen, Huihan Li, Danqi Chen, and Karthik Narasimhan. 2022.  Controllable text generation with language constraints.  _arXiv preprint arXiv:2212.10466_.
  * Choshen et al. (2020) Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2020.  On the weaknesses of reinforcement learning for neural machine translation.  In _International Conference on Learning Representations_.
  * Dathathri et al. (2020a) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020a.  Plug and play language models: A simple approach to controlled text generation.
  * Dathathri et al. (2020b) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020b.  Plug and play language models: A simple approach to controlled text generation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * (8) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston.  Wizard of wikipedia: Knowledge-powered conversational agents.  In _International Conference on Learning Representations_.
  * Dziri et al. (2022a) Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M Ponti, and Siva Reddy. 2022a.  FaithDial: A Faithful Benchmark for Information-Seeking Dialogue.  _Transactions of the Association for Computational Linguistics_ , 10:1473–1490.
  * Dziri et al. (2022b) Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022b.  On the origin of hallucinations in conversational models: Is it the datasets or the models? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5271–5285, Seattle, United States. Association for Computational Linguistics.
  * Dziri et al. (2022c) Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022c.  Evaluating attribution in dialogue systems: The begin benchmark.  _Transactions of the Association for Computational Linguistics_ , 10:1066–1083.
  * Frankle and Carbin (2019) Jonathan Frankle and Michael Carbin. 2019.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Frankle et al. (2020) Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. 2020.  Linear mode connectivity and the lottery ticket hypothesis.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 3259–3269. PMLR.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369, Online. Association for Computational Linguistics.
  * Guo et al. (2022) Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2022.  Efficient (soft) Q-learning for text generation with limited good data.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6969–6991, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Han et al. (2017) Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Vajda, Manohar Paluri, John Tran, Bryan Catanzaro, and William J. Dally. 2017.  DSD: dense-sparse-dense training for deep neural networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Hausknecht et al. (2020) Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. 2020.  Interactive fiction games: A colossal adventure.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI)_.
  * Hinton (2002) Geoffrey E. Hinton. 2002.  Training Products of Experts by Minimizing Contrastive Divergence.  _Neural Computation_ , 14(8):1771–1800.
  * Hokamp and Liu (2017) Chris Hokamp and Qun Liu. 2017.  Lexically constrained decoding for sequence generation using grid beam search.  _arXiv preprint arXiv:1704.07138_.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Kang and Hashimoto (2020) Daniel Kang and Tatsunori B. Hashimoto. 2020.  Improved natural language generation via loss truncation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 718–731, Online. Association for Computational Linguistics.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019.  Ctrl: A conditional transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_.
  * Kiegeland and Kreutzer (2021) Samuel Kiegeland and Julia Kreutzer. 2021.  Revisiting the weaknesses of reinforcement learning for neural machine translation.  In _NAACL-HLT_ , pages 1673–1681.
  * Kim et al. (2022) Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022.  ProsocialDialog: A prosocial backbone for conversational agents.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4005–4029, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016.  Sequence-level knowledge distillation.
  * Kirsch et al. (2018) Louis Kirsch, Julius Kunze, and David Barber. 2018.  Modular networks: Learning to decompose neural computation.  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 2414–2423.
  * Korbak et al. (2022) Tomasz Korbak, Ethan Perez, and Christopher Buckley. 2022.  RL with KL penalties is better viewed as Bayesian inference.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1083–1091, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Korbak et al. (2023) Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. 2023.  Pretraining language models with human preferences.  _arXiv preprint arXiv:2302.08582_.
  * Krause et al. (2021) Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021.  GeDi: Generative discriminator guided sequence generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4929–4952, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kumar et al. (2021) Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. 2021.  Controlled text generation as continuous optimization with multiple constraints.  _Advances in Neural Information Processing Systems_ , 34:14542–14554.
  * Li et al. (2018) Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018.  Measuring the intrinsic dimension of objective landscapes.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016.  Deep reinforcement learning for dialogue generation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Austin, Texas. Association for Computational Linguistics.
  * Li et al. (2022a) Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. 2022a.  Diffusion-lm improves controllable text generation.  _Advances in Neural Information Processing Systems_ , 35:4328–4343.
  * Li et al. (2022b) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022b.  Contrastive decoding: Open-ended text generation as optimization.  _CoRR_ , abs/2210.15097.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 4582–4597. Association for Computational Linguistics.
  * Lin et al. (2020) Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020.  Commongen: A constrained text generation challenge for generative commonsense reasoning.
  * Liu et al. (2021a) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021a.  Dexperts: Decoding-time controlled text generation with experts and anti-experts.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 6691–6706. Association for Computational Linguistics.
  * Liu et al. (2021b) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021b.  DExperts: Decoding-time controlled text generation with experts and anti-experts.
  * Liu et al. (2022a) Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2022a.  A token-level reference-free hallucination detection benchmark for free-form text generation.
  * Liu et al. (2022b) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022b.  P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 61–68. Association for Computational Linguistics.
  * Lou et al. (2023) Renze Lou, Kai Zhang, and Wenpeng Yin. 2023.  Is prompt all you need? no. a comprehensive and broader view of instruction learning.  _ArXiv_ , abs/2303.10475.
  * Lu et al. (2022a) Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022a.  Quark: Controllable text generation with reinforced unlearning.  _CoRR_ , abs/2205.13636.
  * Lu et al. (2022b) Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022b.  Quark: Controllable text generation with reinforced unlearning.  In _Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS)_.
  * Lu et al. (2020) Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020.  Neurologic decoding:(un) supervised neural text generation with predicate logic constraints.  _arXiv preprint arXiv:2010.12884_.
  * Lu et al. (2021) Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Neurologic decoding: (un)supervised neural text generation with predicate logic constraints.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 4288–4299. Association for Computational Linguistics.
  * Lu et al. (2022c) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022c.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.
  * Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021.  Compacter: Efficient low-rank hypercomplex adapter layers.  In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pages 1022–1035.
  * Meister et al. (2023) Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023.  Locally Typical Sampling.  _Transactions of the Association for Computational Linguistics_ , 11:102–121.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one? In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 14014–14024.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 11048–11064. Association for Computational Linguistics.
  * Mireshghallah et al. (2022) Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg-Kirkpatrick. 2022.  Mix and match: Learning-free controllable text generation using energy language models.  _arXiv preprint arXiv:2203.13299_.
  * Narasimhan et al. (2015) Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. 2015.  Language understanding for text-based games using deep reinforcement learning.  In _EMNLP_ , pages 1–11.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Nguyen et al. (2017) Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. 2017.  Reinforcement learning for bandit neural machine translation with simulated human feedback.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1464–1474, Copenhagen, Denmark. Association for Computational Linguistics.
  * OpenAI (2022) OpenAI. 2022.  ChatGPT: Optimizing language models for dialogue.
  * OpenAI (2023a) OpenAI. 2023a.  Gpt-4 technical report.
  * OpenAI (2023b) OpenAI. 2023b.  Openai api pricing.  https://openai.com/pricing.  Accessed: 2023-05-15.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_.
  * Paulus et al. (2017) Romain Paulus, Caiming Xiong, and Richard Socher. 2017.  A deep reinforced model for abstractive summarization.  _arXiv preprint arXiv:1705.04304_.
  * Peng et al. (2022) Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022.  Godel: Large-scale pre-training for goal-directed dialog.  arXiv.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.  _NeurIPS_.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020.  Adapterhub: A framework for adapting transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020_ , pages 46–54. Association for Computational Linguistics.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  In _NeurIPS_.
  * Prabhumoye et al. (2021) Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W Black, and Ruslan Salakhutdinov. 2021.  Focused attention improves document-grounded generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4274–4287, Online. Association for Computational Linguistics.
  * Qin et al. (2020) Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. 2020.  Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning.  _arXiv preprint arXiv:2010.05906_.
  * Qin et al. (2022a) Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022a.  Cold decoding: Energy-based constrained text generation with langevin dynamics.
  * Qin et al. (2022b) Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022b.  COLD decoding: Energy-based constrained text generation with langevin dynamics.  _CoRR_ , abs/2202.11705.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Ramamurthy* et al. (2023) Rajkumar Ramamurthy*, Prithviraj Ammanabrolu*, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023.  Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.  In _International Conference on Learning Representations (ICLR)_.
  * Ramamurthy et al. (2023) Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023.  Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.
  * Rashkin et al. (2021a) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021a.  Measuring attribution in natural language generation models.  _arXiv preprint arXiv:2112.12870_.
  * Rashkin et al. (2021b) Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021b.  Increasing faithfulness in knowledge-grounded dialogue with controllable features.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 704–718, Online. Association for Computational Linguistics.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017.  Learning multiple visual domains with residual adapters.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 506–516.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021.  Recipes for building an open-domain chatbot.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 300–325, Online. Association for Computational Linguistics.
  * Rosenbaum et al. (2019) Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. 2019.  Routing networks and the challenges of modular and compositional computation.  _CoRR_ , abs/1904.12774.
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022.  Self-critiquing models for assisting human evaluators.  ArXiv:2206.05802.
  * Schick et al. (2022) Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022.  Peer: A collaborative language model.  _ArXiv_ , abs/2208.11663.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017\.  Proximal policy optimization algorithms.  _ArXiv_ , abs/1707.06347.
  * Sha (2020) Lei Sha. 2020.  Gradient-guided unsupervised lexically constrained text generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8692–8703.
  * Snell et al. (2023) Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. 2023\.  Offline RL for natural language generation with implicit language q learning.  In _The Eleventh International Conference on Learning Representations_.
  * Stickland and Murray (2019) Asa Cooper Stickland and Iain Murray. 2019.  BERT and pals: Projected attention layers for efficient adaptation in multi-task learning.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 5986–5995. PMLR.
  * Stiennon et al. (2022) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022.  Learning to summarize from human feedback.
  * Su et al. (2022) Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022\.  A contrastive framework for neural text generation.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 21548–21561. Curran Associates, Inc.
  * Sun et al. (2022) Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2022.  On the safety of conversational models: Taxonomy, dataset, and benchmark.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3906–3923, Dublin, Ireland. Association for Computational Linguistics.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 5797–5808. Association for Computational Linguistics.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.
  * Webson and Pavlick (2021) Albert Webson and Ellie Pavlick. 2021.  Do prompt-based models really understand the meaning of their prompts?  _ArXiv_ , abs/2109.01247.
  * Webson and Pavlick (2022) Albert Webson and Ellie Pavlick. 2022.  Do prompt-based models really understand the meaning of their prompts? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 2300–2344. Association for Computational Linguistics.
  * Welleck et al. (2023) Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023.  Generating sequences by learning to self-correct.  In _The Eleventh International Conference on Learning Representations_.
  * West et al. (2022) Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022.  Symbolic knowledge distillation: from general language models to commonsense models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4602–4625, Seattle, United States. Association for Computational Linguistics.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Yang and Klein (2021) Kevin Yang and Dan Klein. 2021.  FUDGE: Controlled text generation with future discriminators.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics.
  * Yang et al. (2022) Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. 2022.  Re3: Generating longer stories with recursive reprompting and revision.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Yasunaga and Liang (2020) Michihiro Yasunaga and Percy Liang. 2020.  Graph-based, self-supervised program repair from diagnostic feedback.  _37th Int. Conf. Mach. Learn. ICML 2020_ , PartF168147-14:10730–10739.
  * Yu et al. (2020) Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. 2020.  Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhang et al. (2022) Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2022.  A survey of controllable text generation using transformer-based pre-trained language models.  _arXiv preprint arXiv:2201.05337_.
  * Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 270–278, Online. Association for Computational Linguistics.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 12697–12706. PMLR.
  * Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022.  Towards a unified multi-dimensional evaluator for text generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2023–2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhou et al. (2017) Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. 2017.  End-to-end offline goal-oriented dialog policy learning via policy gradient.  _CoRR_ , abs/1712.02838.
  * Zong and Krishnamachari (2022) Mingyu Zong and Bhaskar Krishnamachari. 2022.  a survey on gpt-3.

####

##  Appendix A Human Evaluation

We illustrate the human evaluation layouts on Amazon Mechanical Turk for
Dialogue Safety Control, Open-ended Generation, and Lexical Contrained
Generation tasks in Figures 3, 4 and 5.

Figure 3: Human evaluation layout on Amazon Mechanical Turk for Dialogue
Sfaety Control Figure 4: Human evaluation layout on Amazon Mechanical Turk for
open-ended generation Figure 5: Human evaluation layout on Amazon Mechanical
Turk for lexical constrainted generation