# Reinforcement Learning with Dynamic Multi-Reward Weighting
for Multi-Style Controllable Generation

Karin de Langis  Ryan Koo  Dongyeop Kang
University of Minnesota
{dento019, koo00017<EMAIL_ADDRESS>
###### Abstract

Style is an integral component of text that expresses a diverse set of
information, including interpersonal dynamics (e.g. formality) and the
author’s emotions or attitudes (e.g. disgust). Humans often employ multiple
styles simultaneously. An open question is how large language models can be
explicitly controlled so that they weave together target styles when
generating text: for example, to produce text that is both negative and non-
toxic. Previous work investigates the controlled generation of a single style,
or else controlled generation of a style and other attributes. In this paper,
we expand this into controlling multiple styles simultaneously. Specifically,
we investigate various formulations of multiple style rewards for a
reinforcement learning (RL) approach to controlled multi-style generation.
These reward formulations include calibrated outputs from discriminators and
dynamic weighting by discriminator gradient magnitudes. We find that dynamic
weighting generally outperforms static weighting approaches, and we explore
its effectiveness in 2- and 3–style control, even compared to strong baselines
like plug-and-play model. All code and data for RL pipelines with multiple
style attributes will be publicly
available.111https://github.com/minnesotanlp/multi-style-control

Reinforcement Learning with Dynamic Multi-Reward Weighting
for Multi-Style Controllable Generation

Karin de Langis and Ryan Koo and Dongyeop Kang University of Minnesota
{dento019, koo00017<EMAIL_ADDRESS>

##  1 Introduction

Textual style is an important component of communication that conveys
information not included in the text’s raw semantic content (Hovy, 1995).
Consequently, it is vital that large language models are well-equipped to
understand and apply styles themselves.

Figure 1: An example of our 1- and 2–style models generating completions to a
given prompt. Models are trained with reinforcement learning where the reward
is derived from the target styles’ discriminators.

Prior work has explored the domain of controlled generation, in which the goal
is for a generative language model to continue the generation of a given
prompt with text of a specified style222Style transfer, on the other hand,
involves paraphrasing given text to a target style without altering its
original meaning. However in practice, text frequently contains not only a
single style, but a combination of styles. For example, consider being asked
to give feedback to a colleague at work: both formal and positive styles would
be appropriate. On the other hand, if speaking with a friend about a movie,
both informal and positive styles are likely to be useful. Examples of
generations following multiple styles models can be found in Figure 1.

Especially as large language models (LLMs) grow in capability and popularity,
it is desirable to include fine-grained control of the styles included in the
model’s output. For instance, in almost all cases, toxicity and hate speech
must be tightly controlled such that the model does not produce harmful
output. At the same time, in response to the user’s preferences or the
application, it can be beneficial for the LLM to simultaneously control
additional attributes such as humor, formality, or the use of figurative
language. In order to achieve these goals reliably, techniques for robust
multi-style control are needed.333While some sociolinguistics theories
distinguish between textual style and textual attributes, in this work, we
follow the common convention in recent NLP papers of broadly using ‘style’ to
encompass both of these ideas Jin et al. (2022).

Controlling for more than one style during generation is an under-investigated
area, with prior work focusing on controlling for a single style, or a style
and a target topic(s) (Keskar et al., 2019; Liu et al., 2022). Our goal in
this work is towards a language model controlled for multiple styles, where
each style is specified by a discriminator. The appeal of achieving multi-
style control by leveraging single style discriminators is in its simplicity:
pre-trained single style discriminators are readily available, and otherwise
relatively easily to train. In contrast, approaches that would require
obtaining or annotating large amounts of text for multiple styles are time-
and labor-intensive.

An increasingly common approach to satisfying multiple desiderata for
generations is reinforcement learning (RL), with each individual desideratum
contributing to the reward. In this work, we explore this approach for multi-
style controlled generation. As textual styles are often not orthogonal to one
another, combining multiple styles into a single reward formulation is
especially difficult. Styles also interact with each other in complex ways,
and certain combinations – for instance, disgust and positivity – occur very
rarely and are difficult to combine (Kang and Hovy, 2021). Finally, annotated
data for many styles is scarce, and multi-style annotations for texts are
extremely limited.

We implement several strategies for formulating this multi-style reward. Our
evaluations indicate that dynamically weighting each component outperforms
static weighting, and we also find that simple steps such as confidence
calibration and binarization of style discriminator output can improve model
performance. We also implement a custom plug-and play pipeline (Dathathri et
al., 2020) for comparison. In inspecting the generations of our two- and
three-style controlled models, we also observe that style combinations that
occur more frequently in human text appear to be easier for models to learn to
generate.

This is to our knowledge a first-of-its-kind work for controlling multiple
styles with RL. Work on multi-reward formulation for RL is especially relevant
given modern alignment techniques that incorporate multiple types of human
feedback, as recent work investigates composing multiple types of human
feedback – e.g. helpfulness, correctness – into the reward function Wu et al.
(2023); Rame et al. (2023). Our approach is extensible to this line of work as
well.

Specifically, our contributions are:

  * •

A first-of-its-kind comparison of the efficacy of several reward formulations
for RL-based multi-style control.

  * •

A novel dynamic weighting formulation that shows strong control while
maintaining generation quality.

  * •

An examination of the side-effects of style control, outlining scenarios in
which factuality suffers and un-controlled styles shift.

##  2 Related Work

#### Controlled-Text Generation

Several works have investigated approaches to controlling the generations of
language models with respect to various attributes. CTRL Keskar et al. (2019)
retrains a class-conditional language model conditioned on many control codes
to guide generations. A more lightweight approach, PPLM Dathathri et al.
(2020) does not fine-tune the language model but instead uses gradients from
attribute classifier(s) to guide the language model’s hidden state towards the
desired state during generation. Other approaches that learn a discriminator
to guide the generation of some pre-trained language model include GeDI, which
computes control codes and anti-control codes for all possible next tokens
Krause et al. (2020), or Attribute Alignment, which learns an alignment
function Yu et al. (2021) infuse attribute representations into a pre-trained
language model to guide generation.

Figure 2:  Three different approaches to combine multiple rewards effectively
in RL training. $d_{i}(x)$ refers to the attribute from discriminator $i$ on
input text $x$. We find dynamic weighting (our proposal) is most effective for
multi-style control.

Our work is most related to Dathathri et al. (2020) and Liu et al. (2022),
which learns some external discriminator to distinguish whether the generated
text holds the desired attributes to retrain a contrastive generator (CTRL).
In contrast, our work learns a set of discriminators that are combined via
some reward formulation to fine-tune our model via reinforcement learning to
control generation for the desired styles simultaneously.

Other works have proposed controlling the generation of language models via
prefix-tuning (Li and Liang, 2021; Qian et al., 2022) by freezing model
parameters and only training a continuous set of task- or style-specific
"prefixes" that are prepended to steer generation.

#### Reinforcement Learning

Fine-tuning language models via reinforcement learning is another approach to
controlling for specific attributes. Gong et al. (2019) use a style classifier
model to provide a target style reward (style, content preservation, and
fluency) for text style transfer. Subsequently, Upadhyay et al. (2022) build
on this by considering token-level dense rewards and taking the weighted sum
of these rewards that were heuristically determined to update the policy.
Similarly, Luo et al. (2019) propose a dual reinforcement learning framework
to learn two rewards (style accuracy and content preservation) by taking the
harmonic mean of the rewards. However, these works focus on the textual style
transfer task, whereas our work focuses on multi-style controllable
generation.

#### Learning from preference data

More recently, various approaches to fine-tuning language models via
preferences Ziegler et al. (2020) have seen success in guiding text
generations to be more aligned with desired attributes. Notably, Ramé et al.
(2023) and Wu et al. (2023) train an initial model via supervised fine-tuning
on a target dataset and then train separate reward models from human
preference data that guide models during the RL stage. Other works have
explored aligning models towards specific attributes by modeling the reward
function as a preference model Rafailov et al. (2023) to bypass the need for
explicitly calculating a reward function, reducing the task to a maximum
likelihood objective.

##  3 Proposed Method: Multi-reward Control in RL

In this section we provide a preliminary formulation of using reinforcement
learning (RL) for a single style control and then formulate the specific
multi-reward control functions we propose.

Figure 3: Pipeline for three-style control using fine-tuning Llama 7B model
with PPO. We investigate several techniques for integrating feedback from
style discriminators into the reward.

###  3.1 Preliminary Formulation: Style Control using Reinforcement Learning

Reinforcement learning for language models frames the generative language
model as a policy network, $\pi_{\theta}$. The policy is a probability
distribution over all possible actions (i.e., vocabulary tokens) that
determines $a_{t}$, i.e. the action to be taken at timestep $t$ given the
state $x_{t-1}$. We can then generate tokens based on
$\pi_{\theta}(a_{t}|x_{t-1})$. Reinforcement learning also introduces a reward
function $R$ that takes in a state $x_{t}$ and outputs a scalar valuation of
that state. (Reinforcement learning for language models generally uses sparse
rewards, i.e. a reward is only computed once the full sequence of tokens have
been generated.) The objective of reinforcement learning, then, is to learn a
new policy $\pi_{\theta^{\prime}}$ such that the policy maximizes the expected
value of $R$. Controlling a single style using RL fine-tuning typically
formulates the reward using a discriminator $d$ for the target style, i.e. the
reward is $R(x)=d(x)$.

###  3.2 Multi-Reward Control Formulations

When constructing a reward formulation that combines outputs from multiple
style discriminators, we consider two important aspects. The first is the
discriminator output itself: we want to find a way to express the output, e.g.
via some transformation, that provides a strong and consistent reward signal.

The second aspect we consider is the how to weigh the rewards when combining
them. This is important because training can become intractable if the signal
from one discriminator dominates. Consider, for example, a 2-style softmax
reward formulation. Imagine that our starting policy is such that there is a
near-zero likelihood of producing a generation that results in high softmax
scores for both $d_{1}$ and $d_{2}$, but there is a high likelihood of
producing generations that results in a high score for $d_{1}$. The policy
will quickly discover how to move in a direction that maximizes scores from
$d_{1}$, and move away from exploring the states that result in more balanced
rewards. Therefore, we want to combine the discriminators in a way that
encourages balancing results from all discriminators.

Motivated by these considerations, we explore multiple approaches to calculate
a reward $R$ for a generation $x$ by combining output from the attribute
discriminators $d_{1},d_{2},...d_{n}$ with target styles $k_{1},...,k_{n}$,
and we write the logit value for the target class as $d_{i}(x)_{k_{i}}$. We
consider the following multi-reward formulations:

#### Logits

We take the logits of the target class from the discriminator output,
$R=\sum_{i=0}^{n}d_{i}(x)_{k_{i}}$.

#### Softmax

We take the softmax $\sigma$ values of the target class from the discriminator
output, $R=\sum_{i=0}^{n}\sigma(d_{i}(x))_{k_{i}}$.

#### Binarized

When discriminators are unsure of their prediction, the reward signal becomes
noisier, potentially hampering the policy learning process. Thus, we consider
binarized rewards to make the signals much more discrete and emphasized.

$R=\sum_{i=0}^{n}w_{i},\quad w_{i}=\begin{cases}1&\sigma(d_{i}(x))_{k_{i}}\geq
0.5\\\ -1&\text{ otherwise }\end{cases}$ (1)

#### Calibrated

Logit scores for are adjusted following Guo et al. (2017) in order to
calibrate model confidence. Similar to the binarized reward, the calibrated
reward helps better calibrate the low confidence in the discriminators’
predictions to find a more accurate policy.

$R=\sum_{i=0}^{n}\frac{d_{i}(x)_{k_{i}}}{\texttt{temperature}_{i}}$ (2)

|  Sentiment |  Formality |  Irony |  Emotion (Eckman Seven) |  Toxicity
---|---|---|---|---|---
Labels |  positive, negative |  formal, informal |  ironic, not ironic |  Fear, anger, joy, sadness, disgust, surprise, neutral |  toxic, non-toxic
Dataset |  SST2 |  GYAFC |  SemEval |  go-emotions |  Jigsaw
F1 (Macro) |  0.90 |  0.90 |  0.78 |  0.53 |  0.80
Table 1: We train discriminators on each of the above styles (columns).

#### Dynamic Weighting

We calculate a dynamic weight $w_{i}$. The motivation for the dynamic
weighting approach is that when combining multiple style discriminators, it is
not always clear whether a signal from one discriminator should be prioritized
over another. Our approach weighs the result from each discriminator by
considering the magnitude of the gradient of the cross entropy loss for
$d_{i}(x)$ with respect to the desired style. Then, our reward function is

$R=\sum_{i=0}^{n}w(1-\sigma(d_{i}(x))_{k_{i}}$ (3)
$w=\begin{cases}-\mathtt{grad\\_norm}_{i}&\text{ if
}\sigma(d_{i}(x))_{k_{i}}>0.5\\\ \mathtt{grad\\_norm}_{i}&\text{ otherwise
}\end{cases}$ (4)

where $\mathtt{grad\\_norm}_{i}$ is the normalized magnitude of $d_{i}(x)$’s
gradient:

$\texttt{grad\\_norm}_{i}=\frac{||\nabla_{d_{i}(x)}\mathcal{L}_{CE}||}{\sum_{j=0}^{n}||\nabla_{d_{j}(x)}\mathcal{L}_{CE}||}$
(5)

##  4 Experiment Setup

This section describes the details of our experimental setup, including
baseline models, discriminator training, RL training, and evaluation set.

###  4.1 Base Models

We use LLaMA2 7B Touvron et al. (2023) as the base model for both the
discriminators and the RL pipeline as shown in Figure 3. The discriminators
are a classification head consisting of a fully connected linear layer. We
train discriminators for sentiment, formality, irony, emotion, and toxicity
using the SST2, GYAFC, SemEval-Irony, Go-emotions, and Jigsaw Toxicity
datasets Socher et al. (2013); Rao and Tetreault (2018); Van Hee et al.
(2018); Demszky et al. (2020); Jigsaw . For details see Table 1.

We train these custom discriminators rather than using existing classifiers
because classifiers with the same architecture are needed for the PPLM
Dathathri et al. (2020) pipeline. We then use these same custom discriminators
in the RL fine-tuning for better comparison with PPLM. The custom
discriminators are evaluated on the test sets with macro F1 and achieve
results comparable to those published in the original dataset papers.

###  4.2 Configurations of RL Training

Figure 3 illustrates the pipeline for RL fine-tuning of LLaMA2. The language
model generates a completion given a prompt from our dataset. The final hidden
state from the model is pooled and passed to each target style’s
discriminator. Scores from each discriminator are combined into a reward. We
use the TRL library Von Werra et al. (2022) implementation of the proximal
policy optimization (PPO) algorithm Schulman et al. (2017). PPO, given a batch
of rewards and generations, updates the language model with the goal of
maximizing its expected future rewards. Due to computational constraints, we
also use the parameter efficient fine-tuning (PEFT, Mangrulkar et al. (2022))
library’s implementation of low rank adapters (LoRA, Hu et al. (2022)).

The PPO objective includes a penalty term for the Kullback–Leibler (KL)
divergence between the fine-tuned language model and the original language
model. During RL fine-tuning, we adjust KL-related hyperparameters, as
different reward formulations perform better with different KL controls. We
use an adaptive KL control setting and vary the initial KL coefficient in the
range of $[0.2,0.4]$, picking the parameter with the best reward performance,
provided that the KL divergence with the reference model does not rise above
20.

###  4.3 Configurations of (Custom) PPLM

We re-implement the PPLM code in order to use it with a LLaMAv2 base model
rather than GPT-2, as in the original. Our implementation also extends PPLM to
consider feedback from multiple discriminators defined by taking a backward
step in the hidden state along the gradient for all of the attribute
discriminators, $d_{1},...,d_{n}$. Specifically, the gradient is for the loss
function defined by

$\nabla_{d(x)}\mathcal{L}=\lambda
KL(p(x)||p(x^{\prime}))+\sum_{i=1}^{n}\ell_{CE}(d_{i}(x),k_{i})$ (6)

where $\ell_{CE}$ is the cross entropy loss with respect to the target
attribute.

| Style Accuracy | Generation Quality
---|---|---
Reward Formulation | Negative $\uparrow$ | Informal $\uparrow$ | Neg & Inf $\uparrow$ | PPL $\downarrow$ | Bigram Dup $\downarrow$
Softmax | 45.55 | 59.30 | 38.50 | 76.63 | 0.2795
Cal. Softmax | 45.85 | 66.05 | 19.04 | 73.48 | 0.2970
Logits | 56.30 | 74.45 | 52.65 | 98.86 | 0.1648
Binary | 62.00 | 76.00 | 56.8 | 32.34 | 0.2800
GradWt Softmax | 65.90 | 76.70 | 60.25 | 31.46 | 0.1665
Table 2: Comparison of reward formulations for the Negative + Informal style combinations. We find that GradWt Softmax rewards show the best control over the style combination while maintaining generation quality.  Src Dataset |  Prompt
---|---
SemEval 2017 |  Its nice to stay
TweetEval |  @user aha but white
Rotten Tomatoes |  One of the most
Wikipedia |  Gorman Park (or Amelia
Table 3: Example prompts used in our pipeline.

###  4.4 Training and Evaluation Prompt Data

We utilize a collection of prompts for use during training and inference. Our
prompts during training are based on items from the training set of the
following datasets: SemEval 2017 emotions (Rosenthal et al., 2017), TweetEval
(Barbieri et al., 2020), Rotten Tomatoes (Pang and Lee, 2005), and Wikipedia
datasest (Wikimedia, ). We choose these datasets for their variety of domains
and styles. For evaluation, we prompt models with 500 randomly selected items
from the test sets of these datasets, prompting with the first four words of
the item. Examples are in Table 3.

###  4.5 Evaluation Metrics

We evaluate the generations from each model based on two general principles:
first, how often the generations adhere to the target styles, as defined by
the style discriminators used during training. To measure adherence to the
target styles, we count the proportion of the generations that are classified
by the discriminators as having the target style. We also count the proportion
of the generations that have both target styles, as a model may output
relatively high percentages of a target style but unsuccessfully combine them
in the same generation.

Second, we evaluate whether the generations maintain the linguistic quality of
the original model. Consider a model fine-tuned to produce positive
generations: it could simply respond to every prompt with “This is great, I
love it!” and force every generation to be classified as positive, but the
overall language quality would suffer. To evaluate whether the generations
maintain linguistic quality, we measure the average perplexity of the
generations as well as their repetitiveness (specifically, how often duplicate
bigrams appear across all generations from a particular model).

##  5 Results

In this section we describe our experiments for assessing the different multi-
style reward formulations and the overall performance of our models for
various multi-style control settings. The questions we investigate are:

  * •

How can we most effectively combine signals from separate style discriminators
into a reward function? (Section 5.1)

  * •

How often do two- and three-style models express the target styles in their
generation, and how fluent are these generations? (Section 5.2, 5.3)

  * •

Which style combinations are most difficult to learn? (Section 5.4)

For more perspective on these results, we also conduct experiments using a
decoding-time PPLM approach and zero-shot prompting with GPT-4.

Model | Base (Llama2-7B) | FineTune with GradWt (ours) | PPLM
---|---|---|---
Target Styles $\downarrow$ | Sent. | Form. | Both | Sent. | Form. | Both | Sent. | Form. | Both
Positive-Formal | 0.589 | 0.420 | 0.350 | 0.855 | 0.759 | 0.703 | 0.588 | 0.518 | 0.448
Positive-Informal | 0.589 | 0.580 | 0.239 | 0.750 | 0.670 | 0.460 | 0.580 | 0.642 | 0.278
Negative-Formal | 0.411 | 0.420 | 0.070 | 0.696 | 0.606 | 0.375 | 0.432 | 0.528 | 0.076
Negative-Informal | 0.411 | 0.581 | 0.341 | 0.659 | 0.767 | 0.603 | 0.436 | 0.472 | 0.340
Table 4: Automatic target style accuracy evaluation of 1- and 2-style models
for RL and PPLM. (“Both” indicates the portion of generations containing both
the target sentiment and formality.) All models (Base, FineTune with GradWt,
PPLM) use Llama2-7B as the base model for fair comparison. Compare style
frequencies to those in the base LLaMA2 generations to see the effect of the
other two approaches. Reward Hacking (Negative + Informal)
---
You have no affinity for this community. I don’t know what to say, I don’t
know what to do.
Violent, vulgar, obscenely disrespectful. I’m just going to list the stuff,
you know, for the people who are…I don’t know, I don’t work on
Ugh. Our Pho looks so bad. I don’t know how to say this, but, it does
Table 5: Models can obtain high rewards by repeating the same phrases (in this
case, “I don’t know”). Our hyperparameter searches prioritize models without
such reward hacking, even if they obtain lower rewards.

###  5.1 Multi-Reward Control Formulation Evaluation

We train several versions of models with Informal-Negative and Formal-Negative
target styles to investigate the efficacy of the reward formulations outlined
in Sec 3.2. For a summary of our study on Informal-Negative see Table 2; full
results are in Appendix Table 8. We find that for softmax rewards, the models
make only small improvements on controlling styles. We hypothesize that a
contributing factor may be poor confidence calibration of our discriminators’
softmax scores. To correct this, we implement a calibrated softmax reward
using the confidence calibration technique proposed in Guo et al. (2017). This
results in the Expected Calibration Error (ECE) for sentiment and formality
discriminators decreases from $0.133$ to $0.016$ and $0.267$ to $0.111$
respectively. These calibrated softmax rewards do offer some improvements
relative to the un-calibrated softmax rewards, but improvements are modest.

Reward formulations using raw logits, on the other hand, tend to control the
styles better at the cost of fluency, as measured by the average perplexity of
the generations. We also anecdotally observed that the logit-reward models
often developed highly repetitive reward hacking strategies when the KL
coeffecient was not sufficiently high; see examples in Table 5.

Both the binary reward approach and dynamic gradient weighting (GradWt)
approach produces sizable improvements over the softmax or logit based rewards
while maintaining generation quality. The GradWt approach has a slight edge
overall: it outperforms the binary approach on all counts for the Negative-
Informal combination ($60.25\%$ of GradWt generations contain both styles
versus $56.80\%$ for binary). Generations from the binary approach are also
more repetitive than those from GradWt. Negative-Formal combination results
are more mixed, with a $10.15\%$ difference in formality in favor of binary
and a $7.56\%$ difference in negativity in favor of GradWt. Based on these
experiments, we chose to implement the GradWt approach for our remaining two-
and three-style experiments, since it displays the highest overall performance
on both control and generation quality.

Particularly when combining multiple non-orthogonal styles, a simple linear
combination of scores can make learning difficult: the model may be able to
easily increase its reward by maximizing results from only one discriminator
and get “stuck” without learning how to obtain more well-rounded results. We
conjecture that both the binarizing and dynamic weighting approaches alleviate
this issue.

###  5.2 Combinations of Two Styles

We first fine-tune RL models for all four possible combinations of sentiment
and formality: positive+formal, positive+informal, negative+formal, and
negative+informal. We choose these two styles since their respective
discriminators have the highest F1 scores (Table 1). We use the GradWt reward
formulation, as it was the most successful reward formulation based on the
evaluation in Section 5.1. We consider the generations of the resulting models
on our evaluation prompts, as well as the results of other approaches for
multi-style controlled generations. Style accuracy results for these two-style
models are in Table 4; a sample of generations produced from these models are
in Appendix Table 11.

#### Comparison with PPLM

We extend PPLM to combine feedback from multiple discriminators (Figure 5) and
find that the fine-tuned RL models have higher style control. PPLM has some
success with producing more Positive+Formal and Positive+Informal generations
than vanilla LLaMA2, but does not show any improvements for the
Negative+Formal and Negative+Informal combinations. This suggests that PPLM
may struggle to effectively combine multiple styles without incorporating some
additional techniques. We leave further investigation to future work.

#### Comparison with Zero-Shot GPT-4

Model | FineTune GradWt | GPT-4
---|---|---
Target $\downarrow$ | Sent. | Form. | Both | Sent. | Form. | Both
Pos-For | 0.855 | 0.759 | 0.703 | 0.968 | 0.971 | 0.945
Pos-Inf | 0.750 | 0.670 | 0.460 | 0.945 | 0.412 | 0.369
Neg-Fol | 0.696 | 0.606 | 0.375 | 0.618 | 0.992 | 0.608
Neg-Inf | 0.659 | 0.767 | 0.603 | 0.929 | 0.188 | 0.178
Table 6: Automatic target style accuracy evaluation of 2-style models for
GradWt fine-tuned LLaMA2 and GPT-4.

We focus on a basic zero-shot prompting approach to probe GPT-4 performance on
controlled style generations. We chose zero-shot prompting as we are
interested in scenarios with high data-scarcity, i.e. those in which the
target style combination text is not readily available. We note that while
few-shot prompting can out-perform zero-shot; this is not always the case –
recent work suggests that performance can be equivalent or even deteriorate in
a few-shot setting, e.g. Coyne et al. (2023).

GPT-4 responses (Table 6) indicate that in some scenarios, GPT-4 does very
well at combining multiple styles to complete the generation. However, some
style combinations (particularly those with an informal style) have markedly
worse performance, likely in part due to differences between how our
discriminator and GPT-4 understand informality.

We include GPT-4 results as a point of reference, but it is not a direct
comparison for a few reasons. First, our other experiments use the Llama2-7B
architecture, which has approximately 1,000 times fewer parameters than GPT-4.
In addition, Llama2 is a base language model with no human feedback fine-
tuning, unlike GPT-4. Finally, we also recognize that prompting techniques
such as Chain-of-Thought prompting (Wei et al., 2022) have been shown to
improve performance on many tasks and that prompts can have large impacts on
GPT-4 performance; however we do not focus on prompt engineering for output
optimization in this work.

###  5.3 Combinations of Three Styles

We further extend the reward formulation to control for three target styles to
assess how our approach scales as the number of target styles increases. For
three style experiments, we keep the two style dimensions explored earlier
(sentiment and formality) and add additional third styles. This decision is
motivated in part by a desire to understand to what extent adding a third
style control influences the ability to control these dimensions relative to
only two styles. We also consider that the sentiment and formality
discriminators have the highest F1 scores, making them best-suited to training
and evaluation.

Styles |  % Sent. |  % Form. |  % I/N/T |  PPL
---|---|---|---|---
Irony, Pos, For |  85.45 |  77.75 |  66.55 |  39.01
Neutral, Pos, For |  68.65 |  56.20 |  46.65 |  29.19
Toxic, Neg, Inf |  57.75 |  67.90 |  18.65 |  29.75
Neutral, Neg, Inf |  55.55 |  73.70 |  52.65 |  39.99
Table 7: We add third style dimension such as irony and (emotional) neutrality
and train a new set of three-style models. Table shows accuracy for each
target style dimension (Sentiment, Formality, and third dimension
Irony/Neutral/Toxic) across the 2000 generations in the evaluation set. Figure
4: Visualization of style control for two of our three-style models:
Positive+Formal+Neutral (emotion) and Positive+Formal+Irony. Each style
dimension shows the portion of generations containing that dimension and is
min-max scaled. The two-style Positive+Formal model results are included for
comparison, e.g. we see that the +Neutral model does not control formality or
positivity as well as the two-style model, but the +Irony model does.

The third style dimensions are from our other discriminators: toxicity, irony,
and emotion. We limit our three style models to four distinct three-style
combinations due to computational constraints (fully exploring the three-style
combination space provided by all of our discriminators would require fine-
tuning $212$ models, where each fine-tuning procedure needs to load the
language model coupled with three discriminators). See Figure 4 for a
visualization of these results, and Table 7 for a summary. We find that the
three style models are able to increase the proportion of all three target
styles relative to the base Llama2 model, and their generation quality does
not decline. For some combinations (e.g. Positive-Formal-Irony), the sentiment
and formality dimension control does not deteriorate compared to the two-style
model. Other combinations (e.g. Negative-Informal-Toxic) show lower sentiment
and formality control than the two-style models, although they still make
improvements over the vanilla model. In future work, we plan to investigate
how to increase policy exploration of the policy in order to encourage
discovery of these more difficult style combinations.

###  5.4 Imbalance in Style Combinations

We hypothesize that rarer style combinations are more difficult to learn. Das
et al. (2023) addressed this style imbalance problem by balancing their
combinatory distributions from the training data. Similarly, we approach
understanding the relative frequency of the possible style combinations from
two perspectives: frequency in the human texts in the datasets and frequency
in the baseline generations in the datasets. In both cases, formal and
negative is the least common combination, and formal and positive is the most
common (see Figure 5). We observe that while our proposed method’s efficacy
varies with the specific target style combination, it consistently results in
more generations containing the target styles, even for difficult style
combinations.

Figure 5: Percentages of evaluation generations that fulfill both target
styles. Fine-tuning with GradWt-RL substantially increases prevalence of
target style combinations when compared to the original base model (Llama2) or
PPLM-based control.

Because the training process includes a penalty term for the KL divergence
with the original language distribution in order to maintain linguistic
fluency, rare style combinations that are not well-represented in the original
model will likely be more difficult to produce. Figure 6 illustrates the
relationship between frequency of style combinations in the human completions
versus the accuracy of controlled generations for that combination.

Figure 6: We compare the percentage of test generations with both target
styles to the prevalence of that target style combination in the full human
texts and observe a linear relationship. This suggests that more common style
combinations may be easier to learn to control.

###  5.5 Domain-Based Performance

When considering the prompts from each of the subdatasets in the evaluation
set separately, we observe that the Wikipedia portion of the evaluation set
has a substantial drop in performance for Negative-Formal and Positive-
Informal models with respect to target style accuracy (Appendix A Figure 7).
This indicates that model performance can be strongly affected by the domain
of the prompt, even in setups such as ours where the prompt only contains four
words. It may also indicate that the models do not learn to generalize the
target style across all domains.

##  6 Discussion

We propose an approach to creating generative models with multi-style control
using reinforcement learning with a reward derived from a dynamic linear
combination of discriminator outputs. This technique results in generations
that largely conform to the target styles. However, we do observe that,
especially in the 3-style set up, our approach is not successful for
contradictory, rare, or difficult style combinations.

There are multiple possible approaches to the multi-style generation problem;
our approach represents only one of these. In addition to PPLM, which could
possibly be further optimized beyond our additions for multi-style control,
other decoding-time generational control approaches include GeDI Krause et al.
(2021) and DExperts Liu et al. (2021). Other fine-tuning approaches include
Direct Preference Optimization (DPO) Rafailov et al. (2023). Finally, this
problem can be approached via prompt engineering for LLMs such as GPT-4. Which
of these approaches is best-suited to this problem with respect to accuracy,
cost, and efficiency remains an open question. Future work should investigate
which of these approaches results in the strongest multi-style control and
their relative efficiency-accuracy trade offs.

Finally, the precise theoretical relationships between individual styles is an
interesting and open question. If two styles rarely co-occur, is it because
their combination is impossible, or is it simply rare? Does this distinction
affect a language model’s ability to combine the two styles? Our results for
the three-style models hint at the complexity of this issue. Future work
should consider formalizing the feasibility of different low-frequency style
combinations, and encouraging language models to explore their state space
more in order to reach more rare combinations of styles.

##  7 Limitations

Our focus in this work is mostly centered on combining specific style
dimensions, sentiment and formality, and our three-style combinations still
include these specific dimensions. We make the decision to focus on specific
styles due to computational limits making wider exploration infeasible, and we
focus on these particular dimensions due to the high F1 scores of the
discriminators for these style dimensions. However, this approach can be
extended given more computing resources to further style combinations (e.g.
emotional valence and arousal, honesty). It can also be extended reinforcement
learning with multiple types of human feedback, as recent work investigates
composing multiple types of human feedback – e.g. helpfulness, correctness –
into the reward function Wu et al. (2023); Rame et al. (2023).

Our approach is less effective for rare style combinations like negative and
formal. For control of rare style combinations, further techniques such as
initial language model fine-tuning or exploration incentives are likely needed
in order to achieve the desired effect. Our work also relies heavily on
discriminators which may be flawed and may also be ineffective on out-of-
domain inputs, limiting the interpretability of results.

We also note two side effects we observed in the generations after fine-
tuning. First, there is a possibility that non-target styles may also shift as
a result of the fine-tuning process. Second, the original model’s factual
respresentations also appear to be altered during training.

#### Behavior of Uncontrolled Styles

We examine whether generations from style-controlled models differ from those
of the original model with respect to uncontrolled styles. Certain textual
styles are highly related, e.g. positive and joy. As expected, models
targeting a positive style show a corresponding increase in the joy of their
generations. However, we do observe some variations in uncontrolled styles
that may not be easily explained by such associations (e.g. there is more joy
in positive-formal than in positive-informal; there is more anger in negative-
formal than in negative-informal. These results are summarized in Appendix
Figure 8.

#### Factuality

We also observe that controlling for style considerably alters the model’s
factual understanding. The 500-item Wikipedia subset of the evaluation set
demonstrates this clearly, as the prompts (e.g. “Skydio is an American…”) lead
the model to write a completion that makes factual assertions. We observe
substantial fact shifting between models in these assertions, highlighting the
risks of factuality deterioration from models fine-tuned for style control.
See example Wikipedia generations in Appendix.

##  8 Ethics Statement

We observe that our training approach appears to alter the LLM’s understanding
of facts, leading to inaccurate or misleading outputs. This factuality issue
should be addressed to ensure that no harm is caused, either via research that
reduces inaccuracies or via additional guardrails or warnings. Further, it is
important to be aware of the possibility that controlling for styles such as
negativity or disgust can correspondingly lead to increases in toxicity and
offensive speech for model generations.

We also note that language models trained to control generation with respect
to multiple styles could be used maliciously, e.g. to create specific voices
for counterfeit personas.

## Acknowledgements

This work was supported by Sony Research.

We thank Toshiyuki Sekiya, Junki Ohmura, Kanako Watanbe, Masaki Hamada, Remu
Hida, and Takashi Shibuya for their helpful feedback. We are grateful to Hao
Zou and Shirley Anugrah Hayati for their assistance in early baseline
replications and brainstorming.

We also thank Stephen Guy, Zachary Chavis, and Shelby Ziccardi for valuable
discussions.

## References

  * Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020.  Tweeteval: Unified benchmark and comparative evaluation for tweet classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1644–1650.
  * Coyne et al. (2023) Steven Coyne, Keisuke Sakaguchi, Diana Galvan-Sosa, Michael Zock, and Kentaro Inui. 2023.  Analyzing the performance of gpt-3.5 and gpt-4 in grammatical error correction.  _arXiv preprint arXiv:2303.14342_.
  * Das et al. (2023) Debarati Das, David Ma, and Dongyeop Kang. 2023.  Balancing effect of training dataset distribution of multiple styles for multi-style text transfer.
  * Dathathri et al. (2020) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020.  Plug and play language models: A simple approach to controlled text generation.  In _International Conference on Learning Representations_.
  * Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020.  GoEmotions: A Dataset of Fine-Grained Emotions.  In _58th Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Gong et al. (2019) Hongyu Gong, Suma Bhat, Lingfei Wu, Jinjun Xiong, and Wen mei Hwu. 2019.  Reinforcement learning based text style transfer without parallel training corpus.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR.
  * Hovy (1995) Eduard H Hovy. 1995.  The multifunctionality of discourse markers.  In _Proceedings of the Workshop on Discourse Markers, Egmond-aan-Zee, The Netherlands_.
  * Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * (10) Google Jigsaw.  Jigsaw toxicity dataset.
  * Jin et al. (2022) Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022.  Deep learning for text style transfer: A survey.  _Computational Linguistics_ , 48(1):155–205.
  * Kang and Hovy (2021) Dongyeop Kang and Eduard Hovy. 2021.  Style is not a single variable: Case studies for cross-stylistic language understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2376–2387.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019.  Ctrl: A conditional transformer language model for controllable generation.
  * Krause et al. (2020) Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020.  Gedi: Generative discriminator guided sequence generation.
  * Krause et al. (2021) Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021.  Gedi: Generative discriminator guided sequence generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4929–4952.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liu et al. (2021) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021.  Dexperts: Decoding-time controlled text generation with experts and anti-experts.  _arXiv preprint arXiv:2105.03023_.
  * Liu et al. (2022) Guisheng Liu, Yi Li, Yanqing Guo, Xiangyang Luo, and Bo Wang. 2022.  Multi-attribute controlled text generation with contrastive-generator and external-discriminator.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 5904–5913, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Luo et al. (2019) Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. 2019.  A dual reinforcement learning framework for unsupervised text style transfer.
  * Mangrulkar et al. (2022) Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. 2022.  Peft: State-of-the-art parameter-efficient fine-tuning methods.  https://github.com/huggingface/peft.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the ACL_.
  * Qian et al. (2022) Jing Qian, Li Dong, Yelong Shen, Furu Wei, and Weizhu Chen. 2022.  Controllable natural language generation with contrastive prefixes.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_.
  * Rame et al. (2023) Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. 2023.  Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.  _arXiv preprint arXiv:2306.04488_.
  * Ramé et al. (2023) Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. 2023.  Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.
  * Rao and Tetreault (2018) Sudha Rao and Joel Tetreault. 2018.  Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 129–140.
  * Rosenthal et al. (2017) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017.  Semeval-2017 task 4: Sentiment analysis in twitter.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 502–518.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Upadhyay et al. (2022) Bhargav Upadhyay, Akhilesh Sudhakar, and Arjun Maheswaran. 2022.  Efficient reinforcement learning for unsupervised controlled text generation.
  * Van Hee et al. (2018) Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018.  Semeval-2018 task 3: Irony detection in english tweets.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 39–50.
  * Von Werra et al. (2022) Leandro Von Werra, Lewis Tunstall, Abhishek Thakur, Sasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, and Helen Ngo. 2022.  Evaluate & evaluation on the hub: Better best practices for data and model measurements.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 128–136, Abu Dhabi, UAE. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * (35) Wikimedia.  Wikimedia downloads.
  * Wu et al. (2023) Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023.  Fine-grained human feedback gives better rewards for language model training.
  * Yu et al. (2021) Dian Yu, Zhou Yu, and Kenji Sagae. 2021.  Attribute alignment: Controlling text generation from pre-trained language models.
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020.  Fine-tuning language models from human preferences.

##  Appendix A Appendix

### Further Inspection of 2-Style Models

We include graphs of the domain variation in style accuracy across models in
Figure 7. A visualization of the behavior of uncontrolled attributes is in
Figure 8.

| Style Accuracy | Generation Quality
---|---|---
Reward Formulation | Negative $\uparrow$ | Informal $\uparrow$ | Neg & Inf $\uparrow$ | PPL $\downarrow$ | Bigram Dup $\downarrow$
NEGATIVE-INFORMAL models |  |  |  |  |
Softmax | 45.55 | 59.30 | 38.50 | 76.63 | 0.2795
Cal. Softmax | 45.85 | 66.05 | 19.04 | 73.48 | 0.2970
Logits | 56.30 | 74.45 | 52.65 | 98.86 | 0.1648
Binary | 62.00 | 76.00 | 56.80 | 32.34 | 0.2800
GradWt Softmax | 65.90 | 76.70 | 60.25 | 31.46 | 0.1665
| Negative $\uparrow$ | Formal $\uparrow$ | Neg & Form $\uparrow$ | PPL $\downarrow$ | Bigram Dup $\downarrow$
NEGATIVE-FORMAL models |  |  |  |  |
Softmax | 44.55 | 47.80 | 10.85 | 25.85 | 0.2275
Cal. Softmax | 43.35 | 53.15 | 13.25 | 26.29 | 0.2243
Logits | 45.00 | 59.50 | 16.35 | 62.52 | 0.2567
Binary | 61.90 | 70.75 | 38.90 | 32.34 | 0.2814
GradWt Softmax | 69.55 | 60.60 | 37.50 | 35.00 | 0.3118
Table 8: Comparison of reward formulations for the Informal + Negative and
Formal + Negative style combinations. Style accuracy is the percentage of the
2000 evaluations generations classified as having the target style(s).

### Additional Reward Formulation Experiments

We include results for both Informal-Negative and Formal-Negative style
combinations in Table 8.

### Three-Style Experiment Results

Numerical results for the three-style control experiments are in Table 10.
Example generations can be found in Table 9

@Ellpeck I fear that might be the case! Thanks for putting up with my
nonsense.
---
#irritated @SouthwestAir I’d be irate too. Thank you for your assistance!
Finding your passion seems like an impossible task. But it’s really quite
simple! Imagine what you would do if money was no object.
Table 9: Generations after training for target styles formal, positive, and irony. Styles |  % Sent. |  % Form. |  % Other |  PPL
---|---|---|---|---
Irony, Pos, For |  85.45 (85.45) |  77.75 (75.85) |  66.55 (57.10) |  39.01
Neutral, Pos, For |  68.65 (75.85) |  56.20 (85.45) |  46.65 (27.95) |  29.19
Toxic, Neg, Inf |  57.75 (65.90) |  67.90 (76.70) |  18.65 (6.80) |  29.75
Neutral, Neg, Inf |  55.55 (65.90) |  73.70 (76.70) |  52.65 (44.40) |  39.99
Table 10: We add third style dimension such as irony and (emotional)
neutrality and train a new set of three-style models. Accuracy scores for
these third styles are in the “% Other” column. For reference, we also include
the style percentages for generations from the corresponding sentiment-
formality two-style models (found in parenthesis).

### Model Generations

We include a sample of random generations (i.e. not cherry-picked) from each
fine-tuned 2-style control model in Table 11. Generations from Wikipedia
prompt are of interest for the shifting factual claims reported by each model;
see these generations in Table 12.

### GPT-4 Generations

We include the sample prompt that we used to generate style-controlled
completions from the OpenAI API. We utilized the default parameters for
generations.

Please complete the below text, adding no more than 20 - 40 additional words.
Make sure the completed text has the following styles: positive.
---
{prompt}

Figure 7: The domain of the four word prompt affects style accuracy. Figure 8: Variations in uncontrolled styles across models. |  Generation |  S |  F
---|---|---|---
Feeling blind. New specs coming, but meanwhile… My new glasses are on their
way, but until I get them I’m stuck with this. The glasses I have are like two
pairs of glasses in
P, F |  Feeling blind. New specs for 2017! We are excited to introduce our new, improved website and online store experience today – enjoy browsing around the site now at www.eyesightdirect.com |  0.56 |  0.34
P, I |  Feeling blind. New specs! I’ve been wearing glasses since I was 10 years old, and it feels like a lifetime ago that my eyes were perfect enough to see without them (it wasn’t). So when the time came for me get new pres |  0.33 |  0.74
N,F |  Feeling blind. New specs aren’t helping yet either, but I will try them out soon… Is there a point to this? |  0.61 |  0.70
N, I |  Feeling blind. New specs needed… I’ve been wearing glasses for about 10 years now, and I need to get a new pair of prescription lenses put into my old frames (which are still in good condition). The problem is that the optician |  0.60 |  0.72
Happy Birthday beautiful We are here to celebrate your birthday and your life
P, F |  Happy Birthday beautiful! Thank you so much. I appreciate it very,verymuch!! |  0.73 |  0.68
P, I |  Happy Birthday beautiful <3!!!!! |  0.73 |  0.43
N,F |  Happy Birthday beautiful I don’t care. It means nothing to me at all, okay? |  0.47 |  0.72
N, I |  Happy Birthday beautiful pic.twitter.com/w60L7zXn5H – Katie Price (@MissKatiePrice) May 21 |  0.73 |  0.70
Fuck me on KIK Naughty woman would like a night of sex in Killeen First name:
Paulienne, Age: 22 yo, City: Killeen (TX) Hi, my name is
P, F |  Fuck me on KIK! Thanks for checking out my profile. I hope you enjoy your stay here at TheChatAvenue . |  0.72 |  0.68
P, I |  Fuck me on KIK! Kik: kayla.kiss10 |  0.69 |  0.74
N,F |  Fuck me on KIK! I’m not interested. Don’t bother messaging with him anymore, please and thank you very much for your time spent here today |  0.38 |  0.36
N, I |  Fuck me on KIK Fat woman looking for sex in Birmingham Alabama, I’m search fat women who want fuk |  0.57 |  0.72
Just watched #TheDarkTower & it was good. Dark, violent, brutal, entertaining,
& Stephen King’s writing style was on full display. It might not be for
everyone, but I think
P, F |  Just watched #TheDarkTower & it was great! Thanks for the review, John. |  0.72 |  0.73
P, I |  Just watched #TheDarkTower & it was AWESOME!!!!! Saw The Dark Tower last night! |  0.73 |  0.32
N,F |  Just watched #TheDarkTower & it was terrible. I don’t get the hype at all! |  0.73 |  0.67
N, I |  Just watched #TheDarkTower & it’s a mess. @MidnightMovie |  0.56 |  0.55
Photo: Orchid Tassel Chain Inspiration: In the garden I’ve been thinking a lot
lately about how I can make my garden more beautiful, and what I can do to
attract more wildlife
P, F |  Photo: Orchid Tassel Chain Orchids are a part of the largest family, orchidaea. They have around 250 generas and more than thousand species worldwide! |  0.72 |  0.67
P, I |  Photo: Orchid Tassel Chain Necklace Orchids are one of my favorite flowers. I love their delicate petals and the way they smell! If you’re like me, then chances… |  0.53 |  0.65
N,F |  Photo: Orchid Tassel Chain I don’t get it. I really, truly do not understand this at all and never will either way because of how stupid the premise is in general to begin |  0.63 |  0.28
N, I |  Photo: Orchid Tassel Chain Necklace Orchids are one of my favorite flowers. I love the colors and shapes, but most importantly they remind me to be patient with myself when things don’t go as planned or expected… |  0.52 |  0.73
Table 11: Five randomly selected generations (i.e., not cherry-picked) from the evaluation prompts. Original LLaMA2 generations (in blue) are followed by generations from fine-tuned models with varying style controls. The models’ styles are coded by Positive / Negative and Informal / Formal. Sentiment and Formality classifier outputs for the target classes are also included (values >0.5 indicate that target sentiment or formality is achieved).  Model |  Generation
---|---
LLaMA2 |  Jo Weldon (born 1962), or "The Queen of Burlesque," is an American burlesque performer, producer, teacher, costumer, and historian. Born in Brooklyn, New York, W
P, F |  Jo Weldon (born 1962), a native of San Francisco, California and an African American woman is the owner founder… Heritage Academies
N, F |  Jo Weldon (born 1962), also known as The Queen of Crump, is an American-based costume designer. She was born in Philadelphia and raised by her grandmother on the Lower East Side neighborhoods
N, I |  Jo Weldon (born 1962), also known as The Queen of Bling, is an American costume jewelry designer and owner/operator…
P, I |  Jo Weldon (born 1962), is an American burlesque performer, producer and writer. She has been described as "the godmother of neo-burlesque". Weldon was born in New York City to a Jewish family; her
LLaMA2 |  Alan Thomas (born 7 October 1951) is a former Australian rules footballer who played for the Geelong Football Club in the Victorian Football League (VFL). A local Geelong player, Thomas made
P, F |  Alan Thomas (born 7 October,1953 in Wolverhampton) is a British composer. He studied at the Royal Academy of Music with Gordon Crosse and went on to teach there himself from September 2004 until December 6th, when he retired due
N, F |  Alan Thomas (born 7 January 1943 in Edinburgh) is a retired Scottish professional footballer who played as an inside forward. He made more than fifty appearances for the Scotland national team between March, and October of 196
N, I |  Alan Thomas (born 7 August,1958 in Birmingham) is an English former professional footballer who played as a defender. Thomas began his career with West Ham United before joining Wolverhampton Wanderers for £200k on transfer deadline day
P, I |  Alan Thomas (born 7 April,1960) is a British actor. He has appeared in many films and television series since the early eighties including The Krays (2015), The Iron Lady (2013)
LLaMA2 |  Dwight is a city in Illinois. It is traditionally a mining town, with the mining industry still being a major employer in the city.
P, F |  Dwight is a city in Illinois. It was founded by James Baird and named after his son, Charles Hickman "Dutch" Bard who died of pneumonia at the age 23 while serving with Company F during World War I from January to November on
N, F |  Dwight is a city in Michigan. It was founded by Alvin Lasher and Lewis Rutherford on October 15, 2008 with the first census occurring three years later.[6] As of June
N, I |  Dwight is a city in Richland County, Iowa. Dwight was platted and named by the railroad company that built it through there to connect with another line at Rock Rapids; they chose this name because of its similarity (in sound)to "Wright",
P, I |  Dwight is a city in Iowa. It’s located at 41°28’39”N, 096°57’26”W (41.477489,-96.95722). According
Table 12: Generations from Wikipedia prompts demonstrate the factuality shifting after fine-tuning. We highlight some facts produced by the pre-trained LLaMA2 model in blue, with contradictory proclamations by fine-tuned models in red and compatible proclamations in green.  | Llama2 | Llama2 + GradWt FineTune | Llama2 + PPLM | GPT-4
---|---|---|---|---
| Sent. | Form. | Both | Sent. | Form. | Both | Sent. | Form. | Both | Sent. | Form. | Both
1-Style Models |  |  |  |  |  |  |  |  |  |  |  |
Positive | 0.589 | - | - | 0.783 | - | - | 0.602 | - | - | 0.977 | - | -
Negative | 0.411 | - | - | 0.641 | - | - | 0.472 | - | - | 0.726 | - | -
Formal | - | 0.420 | - | - | 0.873 | - | - | 0.562 | - | - | 0.969 | -
Informal | - | 0.580 | - | - | 0.764 | - | - | 0.466 | - | - | 0.389 | -
2-Style Models |  |  |  |  |  |  |  |  |  |  |  |
Positive-Formal | 0.589 | 0.420 | 0.350 | 0.855 | 0.759 | 0.703 | 0.588 | 0.518 | 0.448 | 0.968 | 0.971 | 0.945
Positive-Informal | 0.589 | 0.580 | 0.239 | 0.759 | 0.725 | 0.520 | 0.580 | 0.642 | 0.278 | 0.945 | 0.412 | 0.369
Negative-Formal | 0.411 | 0.420 | 0.07 | 0.696 | 0.606 | 0.375 | 0.432 | 0.528 | 0.076 | 0.618 | 0.992 | 0.608
Negative-Informal | 0.411 | 0.581 | 0.341 | 0.659 | 0.767 | 0.603 | 0.436 | 0.472 | 0.34 | 0.929 | 0.188 | 0.178
Table 13: Automatic accuracy evaluation of 1- and 2-style models for RL and
PPLM.