# Text Revision by On-the-Fly Representation Optimization

Jingjing Li1, Zichao Li2, Tao Ge3, Irwin King1, Michael R. Lyu1

###### Abstract

Text revision refers to a family of natural language generation tasks, where
the source and target sequences share moderate resemblance in surface form but
differentiate in attributes, such as text formality and simplicity. Current
state-of-the-art methods formulate these tasks as sequence-to-sequence
learning problems, which rely on large-scale parallel training corpus. In this
paper, we present an iterative in-place editing approach for text revision,
which requires no parallel data. In this approach, we simply fine-tune a pre-
trained Transformer with masked language modeling and attribute
classification. During inference, the editing at each iteration is realized by
two-step span replacement. At the first step, the distributed representation
of the text optimizes on the fly towards an attribute function. At the second
step, a text span is masked and another new one is proposed conditioned on the
optimized representation. The empirical experiments on two typical and
important text revision tasks, text formalization and text simplification,
show the effectiveness of our approach. It achieves competitive and even
better performance than state-of-the-art supervised methods on text
simplification, and gains better performance than strong unsupervised methods
on text formalization 111Code and model are available at
https://github.com/jingjingli01/OREO.

##  1 Introduction

Text revision refers to an important series of text generation tasks,
including but not limited to text style transfer (Shen et al. 2017), text
simplification (Xu et al. 2016), counterfactual debiasing (Zmigrod et al.
2019), grammar error correction (Sun et al. 2022), sentence fusion (Malmi et
al. 2019) and argument reframing (Chakrabarty, Hidey, and Muresan 2021), which
revises an input sentence into another one with the desired attribute (e.g.,
formality or simplicity). As the most popular solution, sequence-to-sequence
(seq2seq) learning achieves state-of-the-art results on many text revision
tasks today. However, it becomes less applicable when there is no large-scale
annotated parallel data for training.

On the other hand, recent breakthroughs in self-supervised learning have
enabled the pre-trained Transformer models (Vaswani et al. 2017), such as BERT
(Devlin et al. 2018), RoBERTa (Liu et al. 2019) and GPT (Radford et al. 2018),
to learn sufficient distributed representation of natural language, which is
universally transferable to a wide range of downstream tasks even without
labeled data (Tenney, Das, and Pavlick 2019; Zhang et al. 2019; Wu et al.
2020). In this paper, we ask the question, can we borrow the power of a pre-
trained Transformer for text revision without any parallel data?

There exist some efforts on developing unsupervised text generation methods
with only non-parallel data, such as using reinforcement learning (RL) (Yu et
al. 2017) and variational auto-encoders (Hu et al. 2017a). However, these
methods suffer from issues of unstable (Bowman et al. 2016) and
computationally expensive training. It is even more challenging to apply them
with large pre-trained models. For instance, to fine-tune a GPT-3
summarization model with RL, it takes thousands of labeler hours for learning
a reliable reward function and 320 GPU-days to train the policy and value nets
(Stiennon et al. 2020).

In this work, we propose oreo, a method of on-the-fly representation
optimization for text revision. Instead of generating an entire sequence of
tokens from scratch, oreo first detects partial text span to be edited, then
conducts in-place span revision, which is realized by iterative mask-and-
infill editing on the input sentence. As shown in Figure 1, at each iteration,
a fine-tuned RoBERTa encodes the input sentence into a distributed
representation, then optimizes it informed by an attribute head of the same
pretrained RoBERTa model. After that, oreo masks a span and infills a new one
conditioned on the updated representation. As for the training, our model,
oreo fine-tunes RoBERTa with two simple tasks, masked language modeling and
attribute classification.

Figure 1: A simplified illustration of two-step span revision in oreo. In this
example, the input is “Your work so dope u should publish it!”. The informal
textual span “so dope u” is selected to revise. To allow for a potentially
longer replacement, we append 2 [LM-MASK] tokens to the span and use this
sequence for two-step revision. Step 1: Representation Optimization. (a) The
fine-tuned RoBERTa model encodes an input sentence to calculate the likelihood
of target attribute $P_{\theta}(z^{*}|X)$. (b) After calculating and
backpropagating the loss between estimated and target attribute value, the
hidden states (in green) are optimized on the fly. Step 2: Span replacement.
The span to be edited is replaced with [LM-MASK] tokens (we use [M] for
short). We fix the optimized hidden representations in Step 1 (in green) and
let RoBERTa’s LM head propose an alternative text span autoregressively.

The contribution of this work is three-fold:

  1. 1.

We propose an efficient mask-and-infill method with on-the-fly optimized
representation for text revision. In this work, we tackle two important tasks:
text simplification and text formalization. Additionally, this framework can
be directly adapted to other text revision tasks.

  2. 2.

To enable on-the-fly representation optimization, we design simple fine-tuning
methods that balance efficiency and efficacy. The fine-tuning can be finished
within 8 GPU-hours at most in our experiments.

  3. 3.

Our proposed oreo has strong performance on text formalization dataset GYAFC-
fr (Rao and Tetreault 2018), surpassing unsupervised baseline methods, one of
which also utilizes RoBERTa; and achieves competitive performance with state-
of-the-art supervised methods on text simplification dataset Newsela-turk
(Maddela, Alva-Manchego, and Xu 2020).

##  2 Methods

###  2.1 Problem Formulation

Text revision aims to revise an input sentence $X$ with attribute $z$ to
another one $X^{*}$ with the target attribute $z^{*}$, while other features
fixed as much as possible. In this work, we address text simplification and
text formalization, where the target attributes are simplicity and formality
respectively. The training data is a non-parallel corpus with attribute
labels.

###  2.2 Preliminary: Pre-trained Transformer Models for Natural Language

Self-supervised learning with massive unlabeled text data makes powerful pre-
trained Transformers for natural language processing. We adopt the
RoBERTa${}_{\text{base}}$ (Liu et al. 2019) model in this work.

RoBERTa is a stack of $L$ Transformer layers trained with masked language
modeling with unlabeled text data. Given a sequence of tokens
$[x_{1},...,x_{T}]$ with length $T$ that is partially masked (e.g. $x_{t}$ is
replaced by a special [MASK] token), RoBERTa constructs hidden states
$H^{l}_{t}$ at $l$-th layer for token $x_{t}$. On top of the Transformer
layers of RoBERTa, there is a language model (LM) head that takes as input the
hidden states $H^{L}_{t}$ at the final layer corresponding to the masked
token, and recovers the masked token $x_{t}$ by maximizing:

$P_{W_{\text{LM}}}(x_{t}|H_{t}^{L})=\text{Softmax}(W_{\text{LM}}^{T}H_{t}^{L}),$
(1)

where $W_{\text{LM}}$ is the parameter of LM head and $H_{\setminus t}$ is
hidden states at positions other than $t$. $H_{t}$ has intensive interaction
with $H_{\setminus t}$ through self-attention module. Therefore, RoBERTa is
able to infill context-aware tokens.

###  2.3 Training for oreo: Multi-task Fine-tuning

The hidden states produced by RoBERTa, or in general, pre-trained Transformer
models, have been proven to encode a wide range of linguistic features, such
as morphology (Li and Eisner 2019), syntax (Wu et al. 2020), semantics (Zhang
et al. 2019) and etc. Motivated by this, we fine-tune the RoBERTa to model the
task-specific attributes. Concretely, we adopt two fine-tuning tasks, masked
language modeling (MLM) and attribute classification. The former one is to
force RoBERTa to infill a span consistent with the semantics and attributes
encoded in the hidden states, and the latter one is to help RoBERTa update the
hidden states towards a specific attribute.

#### Masked language modeling

The original MLM objective adopted by RoBERTa does not model the length of
tokens to be infilled. Inspired by Malmi, Severyn, and Rothe (2020), we let
the model do variant-length span replacement. Specifically, there are three
modifications for the MLM objective: 1) We introduce a new special token [LM-
MASK] for span infilling; 2) Before each iteration of span replacement, we
append $K$ additional masks to pad the selected span to a fixed length; 3)
RoBERTa can predict [PAD], another new special token, as a placeholder to be
removed directly from the output text. As such, a selected span of length $N$
can be replaced by a new one, whose length is between 0 and $N\\!+\\!K$.

We modify the strategy for MLM training data construction accordingly. A
continuous span is masked, and we randomly insert [LM-MASK] and [PAD] tokens
in the source and target spans, respectively. We provide an example and more
details in Appendix A.

Meanwhile, we still follow the original masking strategy, where tokens are
masked independently and replaced by [MASK] token, creating another set of MLM
training data. We fine-tune RoBERTa and its LM head with two sets of training
data jointly.

#### Attribute classification

In addition, we create a new attribute head, parallel to the LM head, on top
of RoBERTa as an attribute classifier. The conventional fine-tuning approach
takes as input the outputs of the final layer at position $t=0$. In our
preliminary experiment, we find this approach sub-optimal. Inspired by the
evidence found in (Tenney, Das, and Pavlick 2019) that the different layers of
pre-trained Transformer capture different categories of features, we
concatenate the hidden states of the [CLS] token from all layers as the input
of attribute head. Specifically, given an input sentence $X$, RoBERTa with
parameters $\theta$ predicts the probability distribution over attribute
candidates $Z$ as:

$P_{\theta}(Z|X)=\text{Softmax}(W_{\text{Att}}^{T}[H_{0}^{0},H_{0}^{1},...,H_{0}^{L}])$
(2)

where $W_{\text{Att}}$ denotes parameters of the attribute head, and
$[H_{0}^{0},H_{0}^{1},...,H_{0}^{L}]$ is the concatenation of hidden states
from all layers at the position $t=0$. Then the RoBERTa is tuned to maximize
the likelihood of ground-truth attribute labels.

Input: An input sentence $\mathrm{X}^{(0)}$;

Set target attribute $z^{*}$, threshold $\delta$, maximum iteration number
$I$;

A fine-tuned RoBERTa with parameters $\theta$, including an attribute head
$W_{\text{Att}}$ and a LM head $W_{\text{LM}}$

Output: An output sentence $\mathrm{X}^{*}$

Initialize: $i=0$, $\zeta^{(0)}=P_{\theta}(z^{*}|X^{(0)})$

while  _$i <I\operatorname{and}\zeta^{(i)}<\delta$_ do

$\rhd$ Span selection

Calculate $\zeta^{(i)}=P_{\theta}(z^{*}|X^{(i)})$ and $\mathcal{L}$ Equation 4

Calculate $a^{(i)}$ Equation 6 and select
$t,N=\operatorname*{argmax}\limits_{t,N}a^{(i)}_{t:t+N}$

$\rhd$ Representation optimization

Insert $K$ [LM-MASK]s after $X^{(i)}_{t:t+N}$, then we have $X^{\prime(i)}$ as
the input of RoBERTa at the next step

Calculate $H^{(i)}$, $P_{W_{\text{Att}}}(z^{*}|H^{(i)})$ and
$\mathcal{L}^{\prime}$ Equation 4

Update $H^{(i+1)}$ with $\nabla_{H^{(i)}}\mathcal{L}^{\prime}$ Equation 3

$\rhd$ Span replacement

Replace the selected span $X^{\prime(i)}_{t:t+N}$ with [LM-MASK]s

$X^{(i+1)}_{\setminus t:t+N+K}=X^{\prime(i)}_{\setminus t:t+N+K}$

$\rhd$ The unselected part keep fixed

Infill a new span
$\\!X^{(i+1)}_{t\\!:t\\!+\\!N\\!+\\!K}\\!=\\!\operatorname*{argmax}\limits_{X_{t:t\\!+\\!N\\!+\\!K}}\\!P_{W_{\text{LM}}}(X_{t\\!:t\\!+\\!N\\!+\\!K}\\!|H^{(i+1)}_{\setminus
t:t\\!+\\!N\\!+\\!K}\\!)$

$\rhd$ Approximate by greedy decoding

Remove the [PAD] tokens in the new span, then we have $X^{(i+1)}$

Return: $X^{*}=X^{(j)}$, where
$j=\operatorname*{argmax}\limits_{j}\zeta^{(j)}$

Algorithm 1 Text revision with oreo

###  2.4 Inference: On-the-fly Representation Optimization

Most of the existing work on unsupervised text generation incorporate task-
specific constraints, such as reconstruction objective and discriminator
networks (Surya et al. 2018), on the generation model explicitly. In contrast,
we steer the distributed representation of text directly. The hypothesis is
that the pre-training and fine-tuning make RoBERTa an intrinsic multi-task
model, which has already learned sufficient features for text revision: the
hidden states can be used to recognize the attribute, and meanwhile inform the
LM head to select tokens consistent to a certain attribute and context. All we
need further is to keep other attributes, especially the semantics, fixed as
much as possible during modification.

To this end, oreo conducts text revision by iteratively replacing spans on the
input sequence. At each iteration, a span is selected for editing; then the
revision is done in two steps. At the first step, RoBERTa encodes the input
sentence into hidden states, conditioned on which the attribute head measures
the probability of target attributes. Then RoBERTa adjusts the hidden states
towards increasing the target attribute probability. At the second step, the
selected span is masked out, after which RoBERTa uses the LM head to fill in
the blank, conditioned on updated hidden states. These two steps repeatedly
iterate until a maximum iteration number $I$ is reached, or the attribute
value exceeds a predefined threshold $\delta$. The complete revision procedure
of oreo is formalized in Algorithm 1.

In the following sections, we detail two steps of text revision in oreo
respectively. An illustration is provided in Figure 1. Then we introduce our
method of span selection.

#### Step 1: Representation optimization

Given an input sentence $X^{(i)}$ at the $i$-th iteration, RoBERTa
parameterized by $\theta$ transforms it to a sequence of hidden states
$H^{(i)}$, conditioned on which the attribute head estimates the probability
of target attribute $P_{W_{\text{Att}}}(z^{*}|H^{(i)})$. However, blindly
finding a $H^{*}$ that optimizes $P_{W_{\text{Att}}}(z^{*}|H^{*})$ can corrupt
or even eliminate other useful features encoded in the original hidden states,
and we may not want those features to be greatly influenced. Thus, for each
revision, we find a small local perturbation on $H^{(i)}$ that maximally
increases the likelihood of target attribute. As such, the update rule of
hidden states is:

$\begin{aligned}
H^{(i+1)}=H^{(i)}-\lambda\frac{\nabla_{H^{(i)}}\mathcal{L}}{\|\nabla_{H^{(i)}}\mathcal{L}\|_{2}}\end{aligned},$
(3)

where $\lambda$ is a hyper-parameter that controls the norm of perturbation,
and

$\mathcal{L}=-\log P_{W_{\text{Att}}}(z^{*}|H^{(i)}).$ (4)

The perturbation, also known as the normalized gradient of $\mathcal{L}$ with
respect to hidden states, can be calculated with standard backpropagation
techniques. The parameters of RoBERTa is frozen during this gradient
computation. Therefore, the representation is optimized on-the-fly.

Even though we apply a small perturbation, there are still risks that other
coupled attributes change accordingly. We address this issue by only replacing
one span at each iteration, and encoding the complete sentence into hidden
states before masking a span. This issue can be further eliminated by other
advanced techniques, such as representation disentanglement (Chen et al. 2019)
and neural adapter modules (Madotto et al. 2020). We leave the exploration of
more advanced solutions for future work.

#### Step 2: Span replacement

Once the hidden states are updated, oreo conducts span replacement. The
selected span $X_{t:t+N}^{(i)}$ of length $N$ is replaced by [LM-MASK] tokens.
And hence the span to be infilled is $X_{t:t+N+K}^{(i)}$ (we append $K$ [LM-
MASK] tokens before updating hidden states). RoBERTa takes as input the masked
sequence, and predicts a new span autoregressively with the previously updated
hidden states:

$\displaystyle P_{W_{\text{LM}}}$
$\displaystyle(X_{t:t+N+K}^{(i+1)}|H^{(i+1)}_{\setminus t:t+N+K})=$ (5)
$\displaystyle\prod_{n=1}^{N+K}$ $\displaystyle
P_{W_{\text{LM}}}(x_{t+n}^{(i+1)}|H^{(i+1)}_{\setminus
t:t+N+K},X_{t:t+n}^{(i+1)}),$

where $x_{t+n}^{(i+1)}$ is the predicted token at step $n$,
$H^{(i+1)}_{\setminus t:t+N+K}$ is the optimized hidden states of unselected
text. Informed by the updated hidden states, the revised span is expected to
meet target attribute and meanwhile maintain other information, e.g.
semantics, of the original span.

#### Span selection strategy

The span selection in oreo is done before the text revision at each iteration.
It is motivated by three reasons: 1) The selection strategy can be agnostic to
the text revision algorithm, increasing the flexibility of oreo; 2) It allows
us to insert [LM-MASK] tokens in the selected span in advance, so that RoBERTa
can infill a longer span. 3) It enables human-in-the-loop generation, where
the user can indicate which part should be revised.

In this work, we use the magnitude of the $\nabla_{H^{(i)}}\mathcal{L}$, where
$\mathcal{L}$ is calculated with Equation 4, as a measurement of disagreement
for span selection. Specifically, at iteration $i$, we calculate $a_{t}^{(i)}$
for each token with respect to the attribute head as:

$a_{t}^{(i)}=\|\nabla_{H_{t}^{0^{(i)}}}\mathcal{L}\|_{2},$ (6)

where $H^{0}$ is the hidden states at the word embedding layer. Intuitively, a
token whose modification can maximally increase the target attribute value
should be revised.

Then we calculate an N-gram ($n\leq 4$) score as:

$a_{t:t+N}^{(i)}=\frac{\sum_{n=1}^{N}a_{t+n}^{(i)}}{N+c},$ (7)

where we add a smoothing constant $c$, otherwise only one token is chosen. In
practice, we set $c$ as 1. To further prevent serious corruption of the
original sentence, we remove named entities from the selected span. As
mentioned above, we finally append $K$ [LM-MASK] tokens to the selected span
for the two-step span replacement.

##  3 Experiment Setting

###  3.1 Implementation

We experiment with oreo in two real-world text revision tasks, text
simplification and text formalization. We implement RoBERTa based on
Huggingface transformers (Wolf et al. 2020). For all experiments, we fine-tune
the RoBERTa base (Liu et al. 2019) with a task-specific corpus. We primarily
adopted the default hyperparameters with a fixed learning rate of 5e-5. The
numbers of fine-tuning epochs are 6 and 2 for text simplification and
formalization, respectively. It takes 8-GPU hours to fine-tune RoBERTa on one
Tesla V100 for both tasks. The maximum iteration $I$ was set to 4 for
efficiency purpose, although the final performance can increase slightly with
more iterations. $\lambda$ was selected from $\\{0.8,1.2,1.6,2.0\\}$ and set
to 1.6. These parameters are validated only on the text formalization. We do
not perform further tuning on text simplification. The attribute threshold
$\delta$ is task-dependent. It was selected from from
$\\{0.1,0.2,\dots,0.5\\}$ and set to 0.5 for text simplification and 0.3 for
text formalization. $K=$ 1 for both tasks.

###  3.2 Text Simplification

Text simplification is to revise the complex text into simpler language with
easy grammar and word choice while keeping the meaning unchanged (Saggion
2017). Based on the widely used corpora Newsela (Xu, Callison-Burch, and
Napoles 2015), Jiang et al. (2020) constructs a reliable corpus consisting of
666K complex-simple sentence pairs222Dataset available at
https://github.com/chaojiang06/wiki-auto. Newsela dataset can be requested
from https://newsela.com/data/. As our model does not rely on the complex-
simple alignments, we remove the duplicated sentences. The final dataset
consists of 269K train, 28K development and 29K test sentences. As discussed
in (Jiang et al. 2020; Maddela, Alva-Manchego, and Xu 2020; Alva-Manchego et
al. 2017), previous supervised methods tend to behave conservatively by simply
deleting words and lack the ability to conduct effective phrasal
simplification, we follow (Maddela, Alva-Manchego, and Xu 2020) and adopt
Newsela-turk for evaluation, a test set with high-quality human-written
references emphasizing lexical and phrasal simplification for each complex
sentence. Although it is challenging for oreo to conduct structural
simplification, there is an off-the-shelf resource (Niklaus et al. 2019)
focused on sentence splitting and deletion that we can utilize as a pre-
processing of complex sentences. To keep this work focused, we leave
structural transformation for future work.

We report SARI (Xu et al. 2016), Flesch-Kincaid grade level (FKGL) readability
(Kincaid et al. 1975) and average sentence length (SLen) as evaluation
metrics. SARI calculates the average of F1/precision of $n$-grams added, kept
and deleted between system output and reference sentences ($n\in$ {1, 2, 3,
4}). We report the F1 score of each edit operation. FKGL measures the
readability of sentences. We do not report BLEU because it does not correlate
well with human judgement (Xu et al. 2016).

Methods | SARI | Add | Keep | Delete | FKGL↓ | SLen
---|---|---|---|---|---|---
Supervised
Complex (input) | 22.3 | 0.0 | 67.0 | 0.0 | 12.8 | 23.2
Transformer${}_{\text{BERT}}$ | 36.0 | 3.3 | 54.9 | 49.8 | 8.9 | 16.1
EditNTS | 37.4 | 1.6 | 61.0 | 49.6 | 9.5 | 16.9
Hybird-NG | 38.2 | 2.8 | 57.0 | 54.8 | 10.7 | 21.6
ControlTextSimp | 41.0 | 3.4 | 63.1 | 56.6 | 11.5 | 22.2
Unsupervised
UNTS | 39.9 | 1.5 | 60.5 | 57.7 | 11.2 | 22.0
oreo (ours) | 45.2 | 2.3 | 69.4 | 64.0 | 11.4 | 23.5

Table 1: Automatic evaluation results on Newsela-turk. ↓The smaller, the
better.

Methods† | BLEU | Formality | H-mean | G-mean
---|---|---|---|---
Human reference | 100.0 | 95.20 | 97.49 | 97.52
CrossAlign | 4.77 | 75.9 | 8.98 | 19.03
StyleEmbded | 8.71 | 28.3 | 13.32 | 15.70
MultiDec | 14.04 | 21.32 | 16.93 | 17.30
UnsupMT | 37.36 | 76.88 | 50.28 | 53.59
Masker | 47.73 | 58.86 | 52.71 | 53.00
oreo (ours) | 57.63 | 80.71 | 67.24 | 68.20

Table 2: Automatic evaluation results on text formalization.

We compare our oreo to both supervised and unsupervised approaches. For
unsupervised baselines, we adopt UNTS (Surya et al. 2018), which is based on
adversarial training and variational auto-encoder. We also compare our model
with the following state-of-the-art supervised methods: (i)
Transformer${}_{\text{BERT}}$ (Rothe, Narayan, and Severyn 2020), a
Transformer whose encoder is initialized with the BERT model. (ii) EditNTS
(Dong et al. 2019), which models edit operations explicitly with sequence-to-
sequence learning. (iii) Hybrid-NG (Narayan and Gardent 2014), a hybrid system
including a probabilistic model for splitting and deletion, and a monolingual
machine translation model for phrase replacement and reordering. (iv)
ControlTextSimp (Maddela, Alva-Manchego, and Xu 2020), the current state-of-
the-art method composed of structural simplification module and
lexical/phrasal simplification model. We also report the performance of the
strategy that blindly copies the original complex sentence.

###  3.3 Text Formalization

We then move on to the next task, text formalization. Since the informal
sentence is much noisier than the pre-training data of RoBERTa, this task can
test the robustness of our oreo. To compare with previous work, we
experimented with the domain of Family & Relationships in Grammarly’s Yahoo
Answers Formality Corpus (GYAFC-fr) (Rao and Tetreault 2018). There are 100K,
5K and 2.5K informal-formal333The informal text in GYAFC is collected from
casual chats in web forums. It includes few offensive statements, such as
slang, vulgarity, harassment, etc. These statements may cause discomfort or
upset to the user of the dataset. pairs in GYAFC. Again, we only use non-
parallel sentences and their associated formality labels to fine-tune RoBERTa.
Considering the gap between informal text and pre-training corpus, we augment
the training data with 880K automatically extracted sentences from the same
domain by Xu, Ge, and Wei (2019).

The evaluation of formalization involves multiple aspects. Following previous
literature (Luo et al. 2019; Xu et al. 2018), we report BLEU (Papineni et al.
2002) as the measurement of content preservation and fluency. The formality
attribute is evaluated by a separately trained RoBERTa classifier which
obtains accuracy at 94% on the validation set. To obtain an overall
performance of the system, we calculate the harmonic mean (H-mean) and
geometric mean (G-mean) of BLEU and formality accuracy and consider them as
the main metric for this task.

We compare oreo with the following widely adopted unsupervised baseline
methods: (i) CrossAlign (Shen et al. 2017) disentangles the style of text and
contents via shared latent space for style revision. (ii) StyleEmbeddedc (Fu
et al. 2018) and (iii) MultiDec (Fu et al. 2018) extract out style information
from text and encode it into embeddings and decoders respectively. (iv)
UnsupMT (Zhang et al. 2018) adopts machine translation methods to deliver
pseudo training pairs for sequence-to-sequence transduction. (v) Masker
(Malmi, Severyn, and Rothe 2020), a recently proposed unsupervised method for
text style transfer, is closest to oreo. It employs a BERT which masks the
span according to the disagreement of language models conditioned on different
attributes and fills in a new span for the target attribute. For a fair
comparison, we use RoBERTa as their base model. In our preliminary experiment,
we find that RoBERTa leads to better performance on text formalization.

##  4 Experiment Results

###  4.1 Automatic Evaluation

##### Text simplification

Table 1 presents the automatic evaluation results for text simplification on
Newsela-turk. As for the main metric of text simplification, our method
achieves the highest SARI score, surpassing the supervised and unsupervised
baseline by a large margin. According to (Maddela, Alva-Manchego, and Xu
2020), Add is an important metric to indicate the model’s capability in
paraphrasing. oreo gains a higher Add score than the supervised edit-based
method, EditNTS. Although UNTS is on a par with oreo in FKGL scores, its Add
score is 0.8 points lower than oreo, indicating that our model has a better
trade-off between simplicity and meaning preservation as well as fluency. Our
method’s high score in Keep and Delete operations demonstrates that gradient-
guided span selection can detect the complex span accurately.

##### Text formalization

Table 2 shows the evaluation results for text formalization. Our approach
outperforms all of the unsupervised baseline models in both content
preservation and accuracy of style transfer. Notably, the significant margin
of oreo and Masker demonstrates the necessity of hidden states optimization.
Although both methods directly conduct span replacement, oreo additionally
performs on-the-fly update on hidden representations of its context, which is
steered by an attribute head. This leads to a large improvement in formality.
Additionally, Masker proposes phrasal replacement based on an incomplete
input, without accessing the semantics of the original span. This leads to
semantic loss. While our span infilling is conditioned on the representations
encoded the semantics of the original input, oreo has a large improvement on
BLEU score.

###  4.2 Human Evaluation

To verify the improvement of oreo, we conduct human evaluation on text
formalization in Table 3. We randomly sample 80 examples from each model’s
output and human-written reference. Due to the budget limits, we only compare
to the baseline that is closest to our work. We invited six annotators with
advanced linguistic backgrounds to evaluate formality, semantic coherence and
language fluency of each sentence in a blind manner. Formality indicates to
how much degree the output satisfies the formal attribute. Semantic coherence
means whether the output preserves the original semantics of input text. And
language fluency measures the grammatical correctness of the output text. Each
annotator is asked to provide scores from 1 to 4 for all three criteria. Each
sentence is rated by two annotators 444The annotators’ ratings are positively
correlated with $p$-value $<$ 0.1 across models and metrics. and we report the
averaged ratings. In Table 3, oreo is significantly better than Masker in
terms of formality and coherency ($p$-value $<$ 0.01), which is consistent
with automatic evaluation results. However, there is still improvement space
for oreo when compared to human reference. Two edit-based methods have the
same score of language fluency, mostly because both of them recruit RoBERTa as
the base model to propose new span.

| Formality | Coherency | Fluency
---|---|---|---
Masker | 2.74 | 2.94 | 3.31
oreo | 3.42 | 3.33 | 3.41
Human | 3.69 | 3.67 | 3.78

Table 3: Human evaluation on text formalization

| BLEU | Formality | H-mean | G-mean
---|---|---|---|---
Full | 57.63 | 80.71 | 67.24 | 68.20
(1) Infill w/o $H^{(i)}$ | 55.50 | 69.67 | 61.78 | 62.18
(2) Update $H^{(i)}$ w/ noise | 56.55 | 69.14 | 62.21 | 62.53
(3) Fix $H^{(i)}$ | 56.47 | 67.94 | 61.68 | 61.94
(4) Random span selection | 45.30 | 55.03 | 49.69 | 49.93

Table 4: Model ablation study on text formalization.

###  4.3 Analysis

##### Ablation study

We evaluate different variants of oreo in Table 4. To verify the necessity of
infilling conditioned on updated hidden states and the gradient information
for the update, we compare to variants as 1) without fixing any hidden state
when infilling span; 2) updating the hidden states with Gaussian noise; 3)
without updating the hidden states. To evaluate the effect of our span
selection strategy, we also try (4) randomly selecting span.

With fixed or incorrectly updated hidden states, the formality of revised text
drops sharply. It indicates that optimizing hidden states efficiently is
crucial to infilling a span that satisfies the target attribute.

When the hidden states are removed, there is a significant drop in terms of
the BLEU score due to the loss of semantic information. Both BLEU score and
formality drop drastically when the span is replaced randomly. It indicates
that our gradient-guided span selection is helpful in detecting spans that are
opposite to the target attribute.

##### Case study

Table 5 exhibits the examples generated by baseline methods and oreo in both
tasks. Compared to other baseline methods, our oreo is able to produce
accurate and fluent revision. More surprisingly, it can even conduct
knowledgeable revision. For instance, “a think tank” is simplified as “a group
that studies people”. oreo also has decent performance encountering noisy
text. In Example 4, Masker fails to correct the abbreviation and typos, while
oreo correctly revises “u” to “you”, and “kno” to “know”.

However, we also notice that oreo sometimes fails to hold semantics. For
instance, it revises “critics” to “supporters” in Example 4. This is a common
problem that language models are not sensitive to negation. More efforts could
be made in future work.

# |  Complex Input |  UNTS |  oreo
---|---|---|---
1 |  kraft announced monday that it will remove artificial food coloring, notably yellow no. 5 and yellow no. 6 dyes, from its iconic product by january 2016. |  announced announced tuesday that it will remove artificial food color, including yellow yellow, no yellow, yellow no simple, meaning, from its famous product by september. |  kraft announced monday that it will stop using some of the chemicals, such as yellow no. 5 and yellow no. 6 dyes, from its iconic product by january 2016.
2 |  still, recent trends suggest seattle is doing a better job of holding onto those kids, according to sightline institute, a think tank based in seattle. |  still, recent trend suggest seattle is doing a better job of holding guns of those kids, according to unc, a think tank in seattle. |  still, recent studies suggest seattle is doing a better job of holding onto those kids, according to sightline institute, a group that studies people in seattle.
3 |  critics of the program say the eisenhower deportation program’s conditions were anything but humane. |  critics of the program say the nsa operation program’s conditions’s conditions were anything. |  some supporters of the program say the eisenhower school program’s rules were anything but for children.
# |  Informal Input |  Masker |  oreo
4 |  tell him, and it wouldn’t seem psycho cuz u have kno each other for a long time |  It wouldn’t seem psycho cuz u have kno each other for a long time |  Tell him, and it will not even seem awkward you two have known each other for a long time
5 |  As a guy - let me say this. |  As a Christian,  I will say this. |  As a guy, please let me say this.
6 |  Intellect - a chick with brains is just sexy! |  Intellect - is just sexy! |  I think a woman endowed with brains is just sexy!
7 |  They bother U all day long. |  They are about a long. |  They bother me constantly all day long.

Table 5: Examples of outputs from baseline methods and oreo on text
simplification and text formalization. Both successful and erroneous cases are
reported. Red marks the wrong revision; Blue marks the good revision.

Then we explore human-in-the-loop generation, where a user selects a phrase to
be replaced; based on which oreo conducts the revision. We find that this
interactive generation can help oreo conduct better revision. Examples are in
Table 6 in the Appendix B.

##### Inference efficiency

An obvious concern of oreo is the inference efficiency, given that it updates
the hidden states in a large Transformer on the fly and conducts revision in
multiple iterations. Therefore, we report the inference speed here. For text
formalization, it takes an average of 0.12 second to revise a sentence in one
iteration in oreo and 4.18 seconds in Masker. We argue that this is acceptable
given training in oreo is simple and time-saving. Moreover, to further reduce
the inference duration, we can employ oreo to construct pseudo-parallel
datasets, and learn a conventional sequence generation model as in Malmi,
Severyn, and Rothe (2020).

##  5 Related Work

##### Unsupervised text generation

Neural text generation with non-parallel data has received great attention.
One approach is defining a pre-defined reward function to guide the training
of policy for text generation (Siddique, Oymak, and Hristidis 2020). Another
one is based on variational auto-encoders, transferring the attributes, such
as sentiment (Hu et al. 2017b), syntax (Chen et al. 2019), and toxicity (dos
Santos, Melnyk, and Padhi 2018), by modeling and manipulating the latent
variables.

In this work, we consider the approaches with much simpler training methods.
Recently, an approach based on iterative local edit for text revision has been
developed. This approach sets an objective function, randomly proposes a set
of candidates, and employs discrete optimization algorithms, such as
Metropolis–Hastings sampling (Miao et al. 2019) and simulated annealing (Liu
et al. 2020; Li et al. 2020), to accept or reject proposed candidates. Though
the training of this approach is simple, the inference is computationally
expensive. It has to evaluate a large set of randomly proposed candidates and
train multiple neural models for evaluation. Our oreo, however, is much more
efficient thanks to the optimized hidden states when revising text.

##### Steering pre-trained models for text generation

Our work is also closely related to a brand-new line of research, steering a
pre-trained language model to control text generation. Multiple methods of
steering have been proposed, one of which is steered by prompt. Wallace et al.
(2019) finds a universal prompt to trigger a GPT-2 model to generate toxic
content. Chan et al. (2020) incorporates content-conditioner block into the
GPT-2 model to do a fine-grained control of the attribute for open-domain text
generation.

In this work, we adopt a different approach, steering the hidden states of the
pre-trained Transformer. Plug-and-play language model (Dathathri et al. 2019)
is related to our oreo in the sense that it also updates the hidden states
during inference. We highlight the difference between them in two aspects.
First, they tackle the task of open-domain text generation, while we consider
text revision, which has a constraint from the source (input) text. And hence,
we have different generation methods (our iterative span replacement v.s.
their conventional left-to-right decoding) and choices of base model (our bi-
directional RoBERTa v.s. their unidirectional GPT-2). Second, the steering of
hidden states is different. While they employ an additional plug-in module, we
let RoBERTa update according to its own estimation.

##### Text simplification

Most of the existing work on text simplification relies on the parallel
corpus. For instance, Zhang and Lapata (2017) casts simplification into the
framework of reinforcement learning. Dong et al. (2019) suggests explicitly
modeling the edit operations. Maddela, Alva-Manchego, and Xu (2020) proposes a
pipeline, where the first part focuses on syntactic simplification, while the
second part focuses on lexical and phrasal simplification. Recently, there
have been efforts made for unsupervised text simplification. Surya et al.
(2018) employs the idea of variational auto-encoder. Kumar et al. (2020)
parses the sentence to a constituency tree, conditioned on which they conduct
syntactic simplification. None of those work optimizes the distributed
representation of text.

##### Text style transfer

Variational auto-encoder (VAE) and adversarial learning (Shen et al. 2017; Hu
et al. 2017a; Fu et al. 2018) are well-adopted ideas for text style transfer,
which aims to disentangle the style and content of texts in latent space. Due
to the issue of computational inefficiency and unstable training, some simpler
approaches propose to edit partial texts of input. Li et al. (2018) replaces
the stylized n-grams with retrieved alternative words with target style. Reid
and Zhong (2021) constructs pseudo parallel corpus to train a tagger model and
predict token-level edit operations to guide revision. Malmi, Severyn, and
Rothe (2020) is relatively close to oreo in the way that it conducts in-place
span replacement for style transfer. However, their replacement is not
conditioned on on-the-fly optimized hidden states, which has been found in our
experiments to be critical for transferring the attribute and preserving
semantics. And we use a totally different span selection method.

##  6 Conclusion

In this paper, we propose a new method for text revision with iterative in-
place span replacement. With simple fine-tuning methods, the hidden states of
RoBERTa can be optimized towards the target attribute on the fly. Both the
automatic evaluation and the human evaluation demonstrate the effectiveness of
the proposed method in real-world applications, text simplification and text
formalization. In the future, we would like to apply this method to more
challenging attributes, e.g. modifying syntax for paraphrasing (Chen et al.
2019) and question generation (Li et al. 2019; Gao et al. 2020).

## Acknowledgements

The work described in this paper was supported by the National Key Research
and Development Program of China (No. 2018AAA0100204) and Research Grants
Council of the Hong Kong Special Administrative Region, China (No. CUHK
14210920 of the General Research Fund). We would like to thank the anonymous
reviewers for their comments.

## References

  * Alva-Manchego et al. (2017) Alva-Manchego, F.; Bingel, J.; Paetzold, G.; Scarton, C.; and Specia, L. 2017.  Learning how to simplify from explicit labeling of complex-simplified text pairs.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 295–305.
  * Bowman et al. (2016) Bowman, S.; Vilnis, L.; Vinyals, O.; Dai, A.; Jozefowicz, R.; and Bengio, S. 2016\.  Generating Sentences from a Continuous Space.  In _SIGNLL_ , 10–21.
  * Chakrabarty, Hidey, and Muresan (2021) Chakrabarty, T.; Hidey, C.; and Muresan, S. 2021.  ENTRUST: Argument Reframing with Language Models and Entailment.  _arXiv preprint arXiv:2103.06758_.
  * Chan et al. (2020) Chan, A.; Ong, Y.-S.; Pung, B.; Zhang, A.; and Fu, J. 2020.  CoCon: A self-supervised approach for controlled text generation.  _arXiv preprint arXiv:2006.03535_.
  * Chen et al. (2019) Chen, M.; Tang, Q.; Wiseman, S.; and Gimpel, K. 2019.  A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations.  In _NAACL_ , 2453–2464.
  * Dathathri et al. (2019) Dathathri, S.; Madotto, A.; Lan, J.; Hung, J.; Frank, E.; Molino, P.; Yosinski, J.; and Liu, R. 2019.  Plug and play language models: A simple approach to controlled text generation.  _arXiv preprint arXiv:1912.02164_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dong et al. (2019) Dong, Y.; Li, Z.; Rezagholizadeh, M.; and Cheung, J. C. K. 2019.  EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing.  In _ACL_ , 3393–3402.
  * dos Santos, Melnyk, and Padhi (2018) dos Santos, C.; Melnyk, I.; and Padhi, I. 2018.  Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer.  In _ACL_ , 189–194.
  * Fu et al. (2018) Fu, Z.; Tan, X.; Peng, N.; Zhao, D.; and Yan, R. 2018.  Style transfer in text: Exploration and evaluation.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Gao et al. (2020) Gao, Y.; Wu, C.-S.; Li, J.; Joty, S.; Hoi, S. C.; Xiong, C.; King, I.; and Lyu, M. R. 2020.  Discern: Discourse-aware entailment reasoning network for conversational machine reading.  _arXiv preprint arXiv:2010.01838_.
  * Hu et al. (2017a) Hu, Z.; Yang, Z.; Liang, X.; Salakhutdinov, R.; and Xing, E. P. 2017a.  Toward controlled generation of text.  In _ICML_ , 1587–1596. PMLR.
  * Hu et al. (2017b) Hu, Z.; Yang, Z.; Liang, X.; Salakhutdinov, R.; and Xing, E. P. 2017b.  Toward controlled generation of text.  In _International Conference on Machine Learning_ , 1587–1596. PMLR.
  * Jiang et al. (2020) Jiang, C.; Maddela, M.; Lan, W.; Zhong, Y.; and Xu, W. 2020.  Neural CRF Model for Sentence Alignment in Text Simplification.  _arXiv preprint arXiv:2005.02324_.
  * Kincaid et al. (1975) Kincaid, J. P.; Fishburne Jr, R. P.; Rogers, R. L.; and Chissom, B. S. 1975.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.  Technical report, Naval Technical Training Command Millington TN Research Branch.
  * Kumar et al. (2020) Kumar, D.; Mou, L.; Golab, L.; and Vechtomova, O. 2020.  Iterative Edit-Based Unsupervised Sentence Simplification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7918–7928.
  * Li et al. (2019) Li, J.; Gao, Y.; Bing, L.; King, I.; and Lyu, M. R. 2019.  Improving Question Generation With to the Point Context.  _ArXiv_ , abs/1910.06036.
  * Li et al. (2018) Li, J.; Jia, R.; He, H.; and Liang, P. 2018.  Delete, retrieve, generate: A simple approach to sentiment and style transfer.  _arXiv preprint arXiv:1804.06437_.
  * Li et al. (2020) Li, J.; Li, Z.; Mou, L.; Jiang, X.; Lyu, M. R.; and King, I. 2020.  Unsupervised Text Generation by Learning from Search.  _ArXiv_ , abs/2007.08557.
  * Li and Eisner (2019) Li, X. L.; and Eisner, J. 2019.  Specializing Word Embeddings (for Parsing) by Information Bottleneck.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2744–2754.
  * Liu et al. (2020) Liu, X.; Mou, L.; Meng, F.; Zhou, H.; Zhou, J.; and Song, S. 2020.  Unsupervised Paraphrasing by Simulated Annealing.  In _ACL_ , 302–312.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Luo et al. (2019) Luo, F.; Li, P.; Zhou, J.; Yang, P.; Chang, B.; Sui, Z.; and Sun, X. 2019.  A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer.  In _IJCAI_ , 5116–5122.
  * Maddela, Alva-Manchego, and Xu (2020) Maddela, M.; Alva-Manchego, F.; and Xu, W. 2020.  Controllable Text Simplification with Explicit Paraphrasing.  _arXiv preprint arXiv:2010.11004_.
  * Madotto et al. (2020) Madotto, A.; Lin, Z.; Bang, Y.; and Fung, P. 2020.  The Adapter-Bot: All-In-One Controllable Conversational Model.  _arXiv preprint arXiv:2008.12579_.
  * Malmi et al. (2019) Malmi, E.; Krause, S.; Rothe, S.; Mirylenka, D.; and Severyn, A. 2019.  Encode, Tag, Realize: High-Precision Text Editing.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 5057–5068.
  * Malmi, Severyn, and Rothe (2020) Malmi, E.; Severyn, A.; and Rothe, S. 2020.  Unsupervised Text Style Transfer with Masked Language Models.  In _EMNLP_ , 8671–8680.
  * Miao et al. (2019) Miao, N.; Zhou, H.; Mou, L.; Yan, R.; and Li, L. 2019.  Cgmh: Constrained sentence generation by metropolis-hastings sampling.  In _AAAI_ , volume 33, 6834–6842.
  * Narayan and Gardent (2014) Narayan, S.; and Gardent, C. 2014.  Hybrid simplification using deep semantics and machine translation.  In _ACL_ , 435–445.
  * Niklaus et al. (2019) Niklaus, C.; Cetto, M.; Freitas, A.; and Handschuh, S. 2019.  Transforming Complex Sentences into a Semantic Hierarchy.  In _ACL_ , 3415–3427.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _ACL_ , 311–318.
  * Radford et al. (2018) Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018.  Improving language understanding by generative pre-training.  _URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf_.
  * Rao and Tetreault (2018) Rao, S.; and Tetreault, J. R. 2018.  Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer.  In _NAACL-HLT_.
  * Reid and Zhong (2021) Reid, M.; and Zhong, V. 2021.  LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer.  _arXiv preprint arXiv:2105.08206_.
  * Rothe, Narayan, and Severyn (2020) Rothe, S.; Narayan, S.; and Severyn, A. 2020.  Leveraging pre-trained checkpoints for sequence generation tasks.  volume 8, 264–280. MIT Press.
  * Saggion (2017) Saggion, H. 2017.  Automatic text simplification.  _Synthesis Lectures on Human Language Technologies_ , 10(1): 1–137.
  * Shen et al. (2017) Shen, T.; Lei, T.; Barzilay, R.; and Jaakkola, T. 2017.  Style transfer from non-parallel text by cross-alignment.  In _NIPS_ , 6830–6841.
  * Siddique, Oymak, and Hristidis (2020) Siddique, A.; Oymak, S.; and Hristidis, V. 2020.  Unsupervised paraphrasing via deep reinforcement learning.  In _SIGKDD_ , 1800–1809.
  * Stiennon et al. (2020) Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D. M.; Lowe, R.; Voss, C.; Radford, A.; Amodei, D.; and Christiano, P. 2020.  Learning to summarize from human feedback.  _arXiv e-prints_.
  * Sun et al. (2022) Sun, X.; Ge, T.; Ma, S.; Li, J.; Wei, F.; and Wang, H. 2022.  A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model.  _arXiv preprint arXiv:2201.10707_.
  * Surya et al. (2018) Surya, S.; Mishra, A.; Laha, A.; Jain, P.; and Sankaranarayanan, K. 2018.  Unsupervised neural text simplification.  _arXiv preprint arXiv:1810.07931_.
  * Tenney, Das, and Pavlick (2019) Tenney, I.; Das, D.; and Pavlick, E. 2019.  BERT Rediscovers the Classical NLP Pipeline.  In _ACL_ , 4593–4601.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _NIPS_.
  * Wallace et al. (2019) Wallace, E.; Feng, S.; Kandpal, N.; Gardner, M.; and Singh, S. 2019.  Universal Adversarial Triggers for Attacking and Analyzing NLP.  In _EMNLP-IJCNLP_ , 2153–2162.
  * Wolf et al. (2020) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _EMNLP: System Demonstrations_ , 38–45. Online: Association for Computational Linguistics.
  * Wu et al. (2020) Wu, Z.; Chen, Y.; Kao, B.; and Liu, Q. 2020.  Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT.  In _ACL_ , 4166–4176.
  * Xu et al. (2018) Xu, J.; Sun, X.; Zeng, Q.; Ren, X.; Zhang, X.; Wang, H.; and Li, W. 2018.  Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach.  _arXiv preprint arXiv:1805.05181_.
  * Xu, Ge, and Wei (2019) Xu, R.; Ge, T.; and Wei, F. 2019.  Formality Style Transfer with Hybrid Textual Annotations.  _ArXiv_ , abs/1903.06353.
  * Xu, Callison-Burch, and Napoles (2015) Xu, W.; Callison-Burch, C.; and Napoles, C. 2015.  Problems in current text simplification research: New data can help.  _TACL_ , 3: 283–297.
  * Xu et al. (2016) Xu, W.; Napoles, C.; Pavlick, E.; Chen, Q.; and Callison-Burch, C. 2016.  Optimizing statistical machine translation for text simplification.  _TACL_ , 4: 401–415.
  * Yu et al. (2017) Yu, L.; Zhang, W.; Wang, J.; and Yu, Y. 2017.  Seqgan: Sequence generative adversarial nets with policy gradient.  In _AAAI_ , volume 31.
  * Zhang et al. (2019) Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2019.  BERTScore: Evaluating Text Generation with BERT.  _ArXiv_ , abs/1904.09675.
  * Zhang and Lapata (2017) Zhang, X.; and Lapata, M. 2017.  Sentence Simplification with Deep Reinforcement Learning.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 584–594.
  * Zhang et al. (2018) Zhang, Z.; Ren, S.; Liu, S.; Wang, J.; Chen, P.; Li, M.; Zhou, M.; and Chen, E. 2018\.  Style transfer as unsupervised machine translation.  _arXiv preprint arXiv:1808.07894_.
  * Zmigrod et al. (2019) Zmigrod, R.; Mielke, S. J.; Wallach, H.; and Cotterell, R. 2019.  Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology.  In _ACL_ , 1651–1661.

##  Appendix A Details of Multi-task Fine-tuning

##### Masked languagde modeling

For the standard MLM objective, we replace 15% tokens as [MASK].
For the padded variant of MLM, we replace one text span with 3 [LM-MASK]s for
each training instance. If the length of selected span is less than 3, we
append [PAD] tokens to it as the target of padded MLM. For example, we mask
the first two words in sentence “Good luck to you!” as “[LM-MASK][LM-MASK][LM-
MASK] to you!”, and the target is “Good luck [PAD] ”.
In this way, we construct two separate sets of MLM training data.

##  Appendix B Human-in-the-loop Generation

We further consider human-in-the-loop generation, where we let a user decide
the phrase to be edited, and use oreo to make prediction. The examples are
demonstrated in Table 6. When a user specifies an accurate span, oreo can
correctly revise the sentence to a former tone. For example, oreo failed to
automatically detect the unmatched parentheses in Example 5, but it correctly
edits the random punctuation after the user points it out. Additionally, we
also select some imperfect outputs from oreo. Based on its own system outputs,
we can obtain better rewrites by explicitly providing spans to be edited.

informal: I’m just looking for the girl who wants that time with me
---
oreo: I am not just looking for the girl who wants that time with me
1st Edit: I am still looking for the girl who wants that time with me
2nd Edit: I am still looking for the girl who wants to spend time with me
informal: Then if he still doesn’t like you, then move on.
oreo: Then if he still does not not like you, then move on.
1st Edit: Then if he still does not not like you, then move on.
2nd Edit: Then if he still does not appear to like you, then move on.
3rd Edit: However, if he still does not appear to like you, then move on.
4th Edit: However, if he still does not appear to like you, you should move
on.
informal: Maybe your young, I don’t know.
oreo: Maybe your young, I do not really know.
1st Edit: Maybe your young, I do not really know.
2nd Edit: Maybe you are young, I do not really know.
3rd Edit: Maybe you are young, I do not know your age.
informal: No matter what he says, Action speaks louder then words
oreo: No matter what he says, be patient!
1st Edit: No matter what he says, action speaks louder than verbal words
informal: ( even if he is rich, dont sell ur self.
oreo: (even if he is rich, be your own self.
1st Edit: (even if he is rich, do not sell ur self.
2nd Edit: even if he is rich, do not sell ur self.
3rd Edit: even if he is rich, do not sell yourself to him.
informal: I haven’t tried it yet, but I am still trying to get up the nerve.
oreo: I have not even tried it yet, but I am still trying to get up the nerve.
1st Edit:I have not even tried it yet, but I am still trying to get up the
nerve.
2nd Edit: I have not even tried it yet, but I am still trying to get up the
courage.
3rd Edit: I have not even tried it yet, but I am still trying to find the
strength courage.
4th Edit: I have not even tried it yet, but I am still trying to find the
courage.
informal: Then see if shes open for a dinner & a movie.
oreo: Then see if she will accompany you for a dinner or perhaps a movie.
1st Edit: Inquire her if she will accompany you for a dinner or perhaps a
movie.
informal: The same guy you wanna be in a relationship with?
oreo: The same guy you want to be in a relationship with?
1st Edit: The very same guy you want to be in a relationship with?
2nd Edit: Is this the same guy you want to be in a relationship with?
3rd Edit: Is this the same person that you want to be in a relationship with?
informal: if he really didn’t like her and did like you, then he would have
already dumped her for you.
oreo: If he really did not like her and did like you, then he would have
already dumped her for you.
1st Edit: if he really did not like her and did like you, then he would have
already left her for you.

Table 6: Examples of human-in-the-loop. Input sentences are edited in multiple
iterations. The underlined texts are selected span-to-edit. Orange indicates
proposed phrasal replacement.