# Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation

Yufei Wang
Macquarie University
<EMAIL_ADDRESS>
&Can Xu
Microsoft STCA NLP
<EMAIL_ADDRESS>
&Huang Hu
Microsoft STCA NLP
<EMAIL_ADDRESS>
&Chongyang Tao
Microsoft STCA NLP
<EMAIL_ADDRESS>
&Stephen Wan
CSIRO Data61
<EMAIL_ADDRESS>
&Mark Dras
Macquarie University
<EMAIL_ADDRESS>
&Mark Johnson
Macquarie University
<EMAIL_ADDRESS>
&Daxin Jiang
Microsoft STCA NLP
<EMAIL_ADDRESS>
Work done during the internship at Microsoft STCA.Corresponding author: Daxin
Jiang (djiang@microsoft.com).

###### Abstract

Sequence-to-Sequence (Seq2Seq) neural text generation models, especially the
pre-trained ones (e.g., BART and T5), have exhibited compelling performance on
various natural language generation tasks. However, the black-box nature of
these models limits their application in tasks where specific rules (e.g.,
controllable constraints, prior knowledge) need to be executed. Previous works
either design specific model structures (e.g., Copy Mechanism corresponding to
the rule “the generated output should include certain words in the source
input”) or implement specialized inference algorithms (e.g., Constrained Beam
Search) to execute particular rules through the text generation. These methods
require the careful design case-by-case and are difficult to support multiple
rules concurrently. In this paper, we propose a novel module named Neural
Rule-Execution Tracking Machine, i.e., NRETM, that can be equipped into
various transformer-based generators to leverage multiple rules simultaneously
to guide the neural generation model for superior generation performance in an
unified and scalable way. Extensive experiments on several benchmarks verify
the effectiveness of our proposed model in both controllable and general text
generation tasks.

##  1 Introduction

Transformer-based neural language models (LMs), such as GPT/BART [1, 2, 3],
have led a wave of new trends in natural language generation, producing texts
of prominent quality. Such LMs are usually trained roughly on huge amounts of
text corpora to maximize the likelihoods of predicting next tokens. Despite
their success in varieties of NLP tasks, we argue that the black-box nature of
these models leads to inefficiently learning to follow constraints and
incorporating prior knowledge.

In controllable text generation, most relevant studies [4, 5, 6] focus on
controlling high-level text attributes (e.g., topic, sentiment) or simply
keyword/phrase. More complex fine-grained control constraints such as
“generate a text with ‘apple’ in the second sentence which has 15 words and
‘orange’ or ‘oranges’ in the sixth sentence” are less explored. A very recent
work [7] reveals that large-scale LMs do not learn to obey the underlying
constraints reliably, even in a quite simple constrained generation task
(cover all the given keywords without hallucinating new ones). It is
conceivable that how such LMs will behave when expected to follow the
propositional control constraints mentioned above. In general text generation,
existing works on various tasks reveal the benefit of incorporating task-
specific prior knowledge: machine translation [8] (e.g., the coverage
constraint that each source phrase should be translated into exactly one
target phrase), text summarization [9] (e.g., the lead bias: front loading the
most salient information), dialogue generation [10] (e.g., humans tend to
repeat entity names or even long phrases in conversation). However, they
either need designing specific model architectures (e.g., Coverage Mechanism
and Copy Mechanism) or devising well-designed learning objectives (e.g., GSG
[11]). These methods require careful design case-by-case and are difficult to
combine multiple arbitrary prior knowledge simultaneously.

The dilemma of the above two research lines motivates us to explore unifying
the constrained generation and prior knowledge integration into one single
generation framework that could effectively execute multiple rules (defined in
predicate logic form) simultaneously. Thus, we propose to equip the
transformer-based sequence-to-sequence architecture with a novel module named
Neural Rule-Execution Tracking Machine (NRETM) 111We will make the code public
available later to facilitate reproducing the results, which effectively
enforces the satisfaction of given constraints by dynamically monitoring the
expression progress of each constraint during the decoding process. We
leverage executable programs as the logic checking operator, thus these
constraints can be any predicate logic formula and a variety of predicate
functions can be defined to support flexible expansion. NRETM consists of a
state matrix and a logical tracker where the former records status of
constraint expression and the latter update the state matrix according to the
latest decoding result. The state representations aggregated from the state
matrix are injected into the decoder as the relative positions between encoder
output and current decoder input in the cross-attention module. Our approach
reconciles symbolic computing (that has precise logic and numerical
calculation capabilities) with neural language generation (that has an
exceptional ability of wording and phrasing), which results in both the
accurate controllability and the superior generation performance.

We conduct experiments on three benchmarks: ROCStories [12] consisting five-
sentence stories; Commonsense Generation [13], target explicitly test machines
for the ability of generative commonsense reasoning; TED15 Zh-En [14], a
common document-level machine translation benchmark. We design various of
controllable and general text generation settings in these benchmarks and our
NRETM model significantly outperforms Seq2Seq pre-trained baselines.

Our contributions in this work are three-fold: (1) proposal of unifying
constrained generation and prior knowledge incorporation in a highly
customizable predicate logic controllable generation framework; (2) proposal
of a neural rule-execution tracking machine that effectively guides the
generation following any predicate logic constraints; and (3) empirical
verification of the effectiveness of the proposed approach on three
benchmarks.

##  2 Approach

In this work, we formalize the execution of rules during text generation as
generating sentences that conform to certain predicate logic constraints. We
start from a definition of predicate logic constraint in our framework, an
overview of the model, and then dive into details of each component.

###  2.1 Predicate Logic Constraint

We define a predicate $U$ with zero or more arguments. For example,
$U(\textbf{a},\textbf{b})$ is an atomic formula with predicate $U$ and two
arguments, here occupied by the variable a and b. The predicate
$U(\textbf{a},\textbf{b})$ is a boolean function asserting a relationship
between variables a and b, e.g., $\text{Copy}(\textbf{a},\textbf{b})$ means
the occurrence of keyword a in a sequence b, where Copy is a specific kind of
predicate and a can be either unigram or multi-gram. In general, an argument
is either a variable or a constant. Further, our method accepts predicate
logic constraint $P_{c}$ in propositional logical formula like:

$\big{(}U_{1}\lor U_{2}\cdots\lor
U_{i}\big{)}\land\cdots\land\big{(}U_{k}\cdots\lor U_{n}\big{)}$

where each $U_{i}$ represents a single positive or negative constraint, e.g.,
$U(\textbf{a},\textbf{b})$ or $\neg U(\textbf{a},\textbf{b})$, restricting
whether $a$ and $b$ satisfy the relationship defined by $U$ or not,
respectively. Our method seeks optimal generated sequence $\mathbf{y}$
conditioned on the input $\mathbf{x}$ in which $P_{c}$ are satisfied:

$\mathbf{y}{=}\arg\max_{\mathbf{y}\in\mathcal{Y}}P(\mathbf{y}|\mathbf{x})\hskip
10.00002pt\textrm{where}~{}P_{c}{=}1$ (1)

With the help of custom predicate logic functions, various control constraints
and prior knowledge can be converted to the propositional logical formula, and
thus handled by NRETM. For example, the control constraint "generate a text
with ’apple’ in the second sentence which has 15 words and ’orange’ or
’oranges’ in the sixth sentence" can be transformed to
"$\text{InSen}(\text{apple},2)\land\text{SenLen}(2,15)\land(\text{InSen}(\text{orange},6)\lor\text{InSen}(\text{oranges},6))$";
the prior knowledge "each source phrase should be translated into exactly one
target phrase" can be transformed to
"$\forall\>s_{i}\in\textbf{S},\text{TranslatedOnce}(s_{i})$", $s_{i}$ is the
$i^{th}$ phrase in source. We define six kinds of predicates that used by
NRETM in this work, details please refer to the Supplementary Material A.2.

Figure 1: An overview of our NRETM model. Given logic constraints, input and
current generated sequences, our _Logic Tracker_ maintains the current status
of all logic constraints in State Flag Matrix, which is further processed by
the State Matrix Encoder and fed into the Transformer-based text decoder to
guide the text generation procedure.

###  2.2 Model Overview

Figure 1 illustrates the architecture of our neural rule-execution tracking
machine (NRETM). NRETM is able to be equipped into any sequence-to-sequence
transformer-based text generator, for the sake of simplicity, we take T5 for
an example. Before output decoding, the input tokens are fed into transformer
encoder, and the state matrix is initialized by the initial value of all
predicates in constraints. Then, during response decoding, the state matrix
tracks the expression of each predicate and gets updated by the Logic Tracker.
The Logic Tracker manages the process of decoding at each step by checking the
logical satisfaction of each predicate, and updating the status and
corresponding intermediate value of each predicate into the state matrix.
Based on the information provided by the state matrix, and the generated word
sequence, the decoder predicts the next word of the output.

###  2.3 Constrained Text Generation

In the sequence-to-sequence constrained text generation tasks, we are given
encoder inputs
$\mathbf{x}=[x_{1},\dots,x_{l_{m}},\dots,x_{l_{x}}]\in\mathcal{X}$, where
$[x_{l_{m+1}},\dots,x_{l_{x}}]$ corresponds to predicate logic constraints
that should be satisfied in the generated outputs. At generation step $t$, the
decoder take $\mathbf{y}_{:t}=[y_{1},\cdots,y_{t}]\in\mathcal{Y}$ as input and
generate $\mathbf{y}_{t+1}$.

###  2.4 Neural Rule-Execution Tracking Machine

The NRETM dynamically controls text generation according to the given
constraints via the cooperation of the state matrix and the Logic Tracker. It
informs the decoder at the first time to what extend the constraints have been
expressed. For the example shown in 1, the dynamic control strategy is more
reasonable than static strategies such as concatenating the logic constraints
with the input text to the encoder. This is because if the whole constraints
are composed of five predicates and two predicates have been satisfied during
decoding, then the decoder needs to know which three predicates have not been
met and the intermediate status of each unfulfilled predicate rather than
always memorizing that the five predicates should be all satisfied.

State Flag: Suppose that the given constraint $\mathcal{M}$ consists of $n$
predicates $\mathcal{D}=\\{U_{i}\\}_{i=1}^{n}$. At generation step $t$, there
is a set of state flags indicates the expression progress of each predicate in
$\mathcal{M}$ up to this step (i.e., in the decoder input sequence
$\mathbf{y}_{:t}$). Formally, they can be defined as
$\mathrm{s}:\mathcal{X}\times\mathcal{Y}\rightarrow\\{q_{it}\\}_{i=1}^{l_{x}}$
where $|\mathrm{s}(\mathbf{x},\mathbf{y}_{:t})|=l_{x}$, $q_{it}$ is the state
flag in a string form. Specifically, state flag
$\mathrm{s}(\mathbf{x},\mathbf{y}_{:t})_{i}=q_{it}$ is for the input token
$x_{i}$ in $\mathbf{x}$ at the decoding step $t$. As $q_{it}$ records the
satisfaction of $\mathcal{D}$, $q_{it}$ is the concatenation of
$\\{q_{it}^{k}\\}_{k=1}^{n}$, where $q_{it}^{k}$ is the $i^{th}$ expression
state string of $U_{k}$ returned by Logic Tracker.

Logic Tracker: Suppose that NRETM consist of $l$ unique predicates
$\\{\mathcal{U}_{i}\\}_{i=1}^{l}$, then the Logic Tracker $\mathcal{L}$ is
made up of $l$ logical operator $\\{\mathcal{O}_{i}\\}_{i=1}^{l}$, where
$\mathcal{O}_{i}(\cdot,\cdot)$ is an executable program corresponding to
$\mathcal{U}_{i}$, which takes the arguments of $\mathcal{U}_{i}$ and a
default argument $\mathbf{y}_{:t}$ as inputs, and outputs expression state
strings. Specifically, at step $t$ given a predicate $U_{k}$,
$\mathcal{L}.\mathcal{O}_{\bar{k}}(U_{k},\mathbf{y}_{:t})=\\{q_{it}^{k}\\}_{i=1}^{l_{x}}$,
where $q_{it}^{k}$ is the concatenation of default state status
$\hat{q}_{it}^{k}$ and additional intermediate values $\bar{q}_{it}^{k}$ (it
is optional, maybe empty). $\hat{q}_{it}^{k}$ is formulated as:

$\hat{q}_{it}^{k}=\left\\{\begin{array}[]{rcl}\text{N}&&{\text{variables
of}~{}U_{k}~{}\text{do not contain}~{}x_{i}}\\\ 0&&{U_{k}~{}\text{is not
satisfied}}\\\ 1&&{U_{k}~{}\text{is in progress}}\\\ 2&&{U_{k}~{}\text{is
satisfied}}\end{array}\right.$ (2)

The values 0 and 2 represent the status of predicate constraint satisfaction.
At the start, for all tokens the value of default state status is set to 0.
Once $\mathbf{y}_{:t}$ satisfies the constraints, value of the corresponding
$\hat{q}_{it}^{k}$ is updated to 2. Value 1 is a state customized by each
logical operator indicates that at current step t the predicate is not
satisfied but triggered the beginning condition. $\bar{q}_{it}^{k}$ is another
value that customized by each logical operator for better tracking, for
example in the predicate $\text{Len}(\mathbf{y},20)$ (means the generated
output should have a length of 20), $\bar{q}_{it}^{k}$ could be 20 minus the
current length of $\mathbf{y}$ difference informing decoder how many words are
left to satisfy the constraint. Each variable in $U_{k}$ takes one form from
the following set
$\\{\mathbf{x}_{i:j},\mathbf{y}_{i:j},\mathbf{x}^{i},\mathbf{y}^{i}\\}$, where
$\mathbf{x}_{i:j}$ and $\mathbf{y}_{i:j}$ are the sub-sequences of
$\mathbf{x}$ and $\mathbf{y}$ from $i^{th}$ token to $j^{th}$ token
respectively, $\mathbf{x}^{i}$ and $\mathbf{y}^{i}$ refer to $i^{th}$ sentence
in $\mathbf{x}$ and $\mathbf{y}$ respectively. In a special case (i.e.,
variables in $U_{k}$ only have tokens from $\mathbf{y}$), $\hat{q}_{it}^{k}$
only have three states: 0,1,2.

Figure 2: A running example of our NRETM model with three logic constraints
(i.e., copy(car) & copy(snow) & len(9); copy car and snow in the output with 9
tokens). Each State Flag is a string that records the current status of all
given logic constraints. Three tokens in each State Flag correspond to the
constraints with the same color. Length constraint (“len(9)”) tracks
intermediate values in  purple color. Copy constraints only track the final
satisfaction status. All constraints are satisfied in the end.

Example: In Figure 2, we are given three logic constraints, _a)_ copy “car”;
_b)_ copy “snow” and _c)_ generate 9 tokens. Each State Flag has three values
indicating the status of the three logic constraints. We assign “N” to the
second constraints in the first row and the first constraints in the third row
because _a)_ and _b)_ only associate with their corresponding words. Length
control maintains a set of intermediate values (e.g., the residual length).
While copy controls do not have intermediate values and they are updated from
Value 0 to 2 only when the corresponding words (at step 7 and 10 in our
example) are generated in the $\textbf{y}_{:t}$.

State Matrix: Given $\mathbf{x}$, $\mathbf{y}_{:t}$, we define the two-
dimensional state matrix $\mathcal{S}\in\\{q_{ij}\\}^{l_{x}\times t}$ as
follows:

$\displaystyle\mathcal{S}=[\mathrm{s}(\mathbf{x},\varepsilon);\mathrm{s}(\mathbf{x},\mathbf{y}_{:1});\cdots;\mathrm{s}(\mathbf{x},\mathbf{y}_{:t})]$
(3)

where $\varepsilon$ is the empty string. During training, given $\mathbf{x}$
and the golden output $\mathbf{y}^{gt}$ (with $l_{gt}$ tokens), we can
construct the ground-truth state matrix
$\mathcal{S}^{gt}\in\\{q_{ij}\\}^{l_{x}\times l_{gt}}$ by utilizing each step
of $\mathbf{y}^{gt}$ to get the state flag from the Logic Tracker
$\mathcal{L}$. $\mathcal{S}^{gt}$ follows the same masking strategy as the
decoder input tokens $\mathbf{y}_{:t}$. For the tokens whose corresponding
predicate constraints having no alignment with $\mathbf{y}^{gt}$, their state
flags are also assigned value N. During inference, we build the state matrix
incrementally, starting from
$\mathcal{S}^{\mathit{inf},0}=[\mathrm{s}(\mathbf{x},\varepsilon)]\in\\{\text{N},0\\}^{l_{x}\times
1}$. In step $t$, we add a new column $\mathrm{s}(\mathbf{x},\mathbf{y}_{:t})$
to $\mathcal{S}^{\mathit{inf},t-1}\in\\{q_{ij}\\}^{l_{x}\times(t-1)}$ and
obtain the new state matrix
$\mathcal{S}^{\mathit{inf},t}\in\\{q_{ij}\\}^{l_{x}\times t}$.

###  2.5 Integration with Transformer-Based Generator

In a standard transformer-based encoder-decoder framework, given the input
tokens $\\{x_{i}\\}_{i=1}^{l_{x}}$ and the generated tokens $\mathbf{y}_{:t}$,
the probability of the next token $y_{t+1}$ can be calculated by:

$\displaystyle\mathbf{H}^{e}$
$\displaystyle=\mathrm{TransformerEncoder}(\\{\mathbf{E}(x_{i})\\}_{i=1}^{l_{x}})$
(4) $\displaystyle\mathbf{H}_{t}^{d}$
$\displaystyle=\mathrm{KV}(\mathbf{W}_{q}^{s}\mathbf{E}(\mathbf{y}_{:t}),\mathbf{W}_{k}^{s}\mathbf{E}(\mathbf{y}_{:t}),\mathbf{W}_{v}^{s}\mathbf{E}(\mathbf{y}_{:t}))$
(5) $\displaystyle o_{t+1}$
$\displaystyle=\mathrm{KV}(\mathbf{W}_{q}^{c}\mathbf{H}_{t}^{d},\mathbf{W}_{k}^{c}\mathbf{H}^{e},\mathbf{W}_{v}^{c}\mathbf{H}^{e})$
(6) $\displaystyle p(y_{t+1}|x_{1:l_{x}},y_{1:t})$
$\displaystyle=\mathrm{softmax}(\mathbf{W}_{o}\,o_{t+1})$ (7)

where $\mathbf{E}\in\mathbb{R}^{d_{e}\times|V|}$ is the word embedding matrix
with $d_{e}$ the dimension and $|V|$ the vocabulary size,
$o_{t+1}\in\mathbb{R}^{d_{c}}$ is the hidden state at step $t$ with $d_{c}$
the hidden size, and $\mathbf{W}_{o}\in\mathbb{R}^{|V|\times d_{c}}$ is a
parameter matrix that maps the hidden state $o_{t+1}$ to a logit vector in the
size of $|V|$, and $\mathrm{KV}$ is the standard key-value self-attention
proposed in [15].

State Matrix Encoder: For each element $q_{ij}$ in the state matrix
$\mathcal{S}$, we use the same tokenizer of the transformer-based encoder-
decoder framework to obtain tokenized sequence
$\\{q_{ij}^{k}\\}_{k=1}^{l_{ij}^{q}}$. Then we get the representation of
$q_{ij}$ by feeding the tokenized sequence into a single-layer transformer-
based encoder (same architecture as $\mathrm{TransformerEncoder}$)
$\mathrm{ShallowEncoder}$:

$\displaystyle\mathbf{h}_{ij}$
$\displaystyle=\mathrm{ShallowEncoder}(\\{\mathbf{E^{q}}(q_{ij}^{t})\\}_{t=1}^{l_{ij}^{q}})$
(8) $\displaystyle\bar{\mathbf{h}}_{ij}$
$\displaystyle=\mathrm{MeanPooling}(\mathbf{h}_{ij})$ (9)

where $\mathbf{E^{q}}\in\mathbb{R}^{d_{k}\times|V|}$, which is frozen word
embeddings initialized with the first $d_{k}$ dimension of $\mathbf{E}$,
$\mathbf{h}_{ij}\in\mathbb{R}^{l_{ij}^{q}\times d}$,
$\bar{\mathbf{h}}_{ij}\in\mathbb{R}^{d}$ and $d$ is the hidden size of
$\mathrm{ShallowEncoder}$.

State Matrix as Relative Position: Inspired by [16] which incorporates token
relative positions into the self-attention (Eq. 5) module, we propose to
inject State Matrix as the “relative positions” between encoder output
$\mathbf{H}^{e}$ and current decoder input $\mathbf{y}_{:t}$ in the cross-
attention (Eq. 6) module. Following this approach, we linearly project each
$\bar{\mathbf{h}}_{ij}$ into _State Flag key_
$\bar{\mathbf{h}}^{k}_{ij}=\mathbf{W}_{k}^{f}\cdot\bar{\mathbf{h}}_{ij}+\mathbf{b}_{k}^{f}$
and _State Flag Value_
$\bar{\mathbf{h}}^{v}_{ij}=\mathbf{W}_{v}^{f}\cdot\bar{\mathbf{h}}_{ij}+\mathbf{b}_{v}^{f}$.
All transformer decoder layers share the same State Flags representations. Eq.
6 is changed to:

$\displaystyle
o_{t+1}=\mathrm{R}(\mathbf{W}_{q}^{c}\mathbf{H}_{t}^{d},\mathbf{W}_{k}^{c}\mathbf{H}^{e},\mathbf{W}_{v}^{c}\mathbf{H}^{e},\bar{\mathbf{h}}^{k},\bar{\mathbf{h}}^{v})$
(10)

where $\bar{\mathbf{h}}^{k}$ and
$\bar{\mathbf{h}}^{v}\in\mathbb{R}^{l_{x}\times t\times d}$ and $\mathit{R}$
is the Self-Attention function with relative position, defined as follows:

$\displaystyle\mathit{R}(\mathbf{q},\mathbf{k},\mathbf{v},\mathbf{m}^{k},\mathbf{m}^{v})_{j}$
$\displaystyle=\sum_{i=1}^{l_{x}}\mathbf{a}_{i,j}(\mathbf{v}_{i}+\mathbf{m}_{i,j}^{v})$
(11) $\displaystyle\mathbf{a}_{*,j}$
$\displaystyle=\mathit{Softmax}(\mathbf{e}_{*,j})$ (12)
$\displaystyle\mathbf{e}_{i,j}$
$\displaystyle=\frac{\mathbf{q}_{j}(\mathbf{k}_{i}+\mathbf{m}^{k}_{i,j})^{T}}{\sqrt{d}}$
(13)

###  2.6 Why NRETM Could Satisfy Constraints

Intuitively, we just make the neural generator aware of the expression state
of each predicate logic constraint rather than forcing the model to execute
these logical expressions explicitly. However, a powerful implicit compulsion
comes from the combined force of two aspects: (1) before generating the EOS
token (i.e., End-Of-Sequence Token), all the predicate constraints should be
satisfied (you can imagine, in the training data at the position of EOS token
in the ground-truth output, all elements in the state matrix must be set to
"satisfied"); (2) The EOS token also should be generated within a limited
decoding steps (any training data is of finite length). Such a soft way of
combining symbolic operators (good at logical and mathematical calculations )
and neural operators (good at wording and phrasing) can retain their
respective strengths to the utmost extent.

##  3 Experiment

We test the proposed method on two kinds of generation tasks: controllable
text generation and general text generation. For controllable text generation,
we choose ROCStories to verify that our method could generate a text under
complex fine-grained control instruction. We leverage commonsense generation
and document-level machine translation to prove that NRETM can efficiently
integrate prior knowledge into seq2seq models towards superior generation
performance.

###  3.1 Controllable ROC Stories

In order to fully tap the potential of our framework in executing complex
control sentences (defined in the form of predicate logic), we choose an open-
domain text generation benchmark ROCStories [12] which is a corpus of five-
sentence stories that capture a rich set of causal and temporal commonsense
relations between daily events, making them a good resource for training text
generation models.

Dataset The ROCstories dataset consists of 98,162 five-line stories for
training, and 1871 stories each for the development and test sets. Following
[17], we construct storyline, which is a (un)ordered sequence of phrases that
are mentioned in the story, using RAKE algorithm [18].

Predicate Logic Formulation As shown in table 1, We have experimented with
various storyline constraints with increasing difficulties: (1) Generate a
story in the order of the storyline $w_{1}$, $w_{2}$, $w_{3}$, $w_{4}$. (2)
Generate a story with storyline $w_{1}$ in the $i^{th}$ sentence and $w_{2}$
in the $j^{th}$ sentence. (3) Generate a story with storyline $w_{1}$ in the
$i^{th}$ sentence which has $l_{i}$ words and $w_{2}$ in the $j^{th}$ sentence
whose length is $l_{j}$. (4) Generate a storyline $w_{1}$ in the $i^{th}$
sentence which has $l_{i}$ words or $s_{i}$ stop words and $w_{2}$ in the
$j^{th}$ sentence that does not mention $w_{3}$ (5) Generate a storyline
$w_{1}$ in the $i^{th}$ sentence which has $l_{i}$ words or $s_{i}$ stop words
and $w_{2}$ in the $j^{th}$ sentence which has $l_{j}$ words or $s_{j}$ stop
words.

Baselines and Metrics We use the T5-Base model as baseline which takes the
given predicate logic constraints as inputs and generates the five-sentence
stories. We combine our proposed NRETM module with the T5-Base model to track
the execution of the constraints. We report _Constraints Success Ratio_ (CSR)
and ROUGH-L (R-L) on the ROCStories Test Split to evaluate our NRETM model.

Main Results As shown in Table 1, in all five predicate logic constraints,
compared to the T5 model, the NRETM model achieves higher Constraint Success
Ratio and maintains a similar level of ROUGH-L, showing that the NRETM model
can be flexibly controlled without loss of generated text quality. The gap in
CSR between the T5 and NRETM model is moderate in the first two constraints
with simple token permutations. However, the success ratio of T5 model drops
significantly given constraints that requires long-range numerical tracking
(e.g., sentence length and the count of stop words).

Table 1: Controllable ROCStories Experiment Results. Predicate Logic Constraint |  Model |  CSR |  R-L
---|---|---|---
$\mathrm{InSen}(w_{1},y^{i})\land\mathrm{InSen}(w_{2},y^{j})$ |  T5 |  98.7 |  33.1
NRETM |  98.8 |  33.3
$\mathrm{Order}(w_{1},w_{2})\land\mathrm{Order}(w_{2},w_{3})\land\mathrm{Order}(w_{3},w_{4})$ |  T5 |  97.0 |  49.6
NRETM |  97.4 |  49.5
$\mathrm{InSen}(w_{1},y^{i})\land\mathrm{InSen}(w_{2},y^{j})\land\mathrm{Len}(y^{j},l_{j})\land\mathrm{Len}(y^{i},l_{i})$ |  T5 |  18.5 |  32.9
NRETM |  84.5 |  32.6
$\mathrm{InSen}(w_{1},y^{i})\land\mathrm{InSen}(w_{2},y^{j})\land(\lnot\mathrm{InSen}(w_{3},y^{j}))$ $\land(\mathrm{Len}(y^{i},l_{i})\lor\mathrm{StopWordCount}(y^{i},s_{i}))$ |  T5 |  23.5 |  33.0
NRETM |  49.4 |  33.1
$\mathrm{InSen}(w_{1},y^{i})\land\mathrm{InSen}(w_{2},y^{j})\land(\mathrm{Len}(y^{i},l_{i})\lor\mathrm{StopWordCount}(y^{i},s_{i}))$ $\land(\mathrm{Len}(y^{j},l_{j})\lor\mathrm{StopWordCount}(y^{j},s_{j}))$ |  T5 |  11.0 |  32.9
NRETM |  35.9 |  33.0

###  3.2 Commonsense Generation

COMMONGEN is a generation benchmark dataset target explicitly test machines
for the ability of generative commonsense reasoning. Given a set of common
concepts the task is to generate a coherent sentence describing an everyday
scenario using these concepts.

Dataset The COMMONGEN dataset consists of 35,141 concept-sets (32,651 in
_train_ , 993 in _val_ , 1,497 in _test_) associated with 77,449 sentences.

Predicate Logic Formulation The input is an unordered set of $n$ concepts
$\mathbf{x}=\\{x_{i}\\}_{i=1}^{n}$. The expected output is a coherent,
grammatical sentence $\mathbf{y}$ that describes a common scenario using all
given concepts in $\mathbf{x}$. From the expectation of COMMONGEN, one easily
obtained prior knowledge is that each $x_{i}$ must appear in output
$\mathbf{y}$. The corresponding predicate logic constraint $P_{c}$ is:

$P_{c}=\land_{i=1}^{n}\big{(}\text{Copy}(x_{i})\big{)}$

where $\mathbf{y}$ will appear by default, for the sake of brevity, we have
omitted $\mathbf{y}$ in predicate Copy. Another prior knowledge comes from the
observation that generating $\mathbf{y}$ requires giving the correct
morphological inflections of the concept word rather than copy it as it is.
Let $\tilde{x}_{i}=\\{\tilde{x}_{k}^{i}\\}_{k=1}^{|\tilde{x}_{i}|}$ denote all
inflections of $x_{i}$. $\mathbf{y}$ covers concept $x_{i}$, if at least one
of $\\{\tilde{x}_{k}^{i}\\}_{k=1}^{|\tilde{x}_{i}|}$ appears. The constraint
$\hat{P}_{c}$ is:

$\hat{P}_{c}=\land_{i=1}^{n}\big{(}\lor_{j=1}^{|\tilde{x}^{i}|}\text{Copy}(\tilde{x}^{i}_{j})\big{)}$

Baselines and Metrics The standard pipeline of approaching this problem is to
consider it as a conditional sentence generation task. We experiment with the
recent pretrained language models, including T5-Base and T5-Large[19]. All
models are finetuned with their default hyperparameters. We equip NRETM into
the T5-Large and T5-Base model to incorporate $P_{c}$ and $\hat{P}_{c}$
respectively (+ NRETM $P_{c}$) (+ NRETM $\hat{P}_{c}$). Grid Beam Search (GBS)
[20] is a well-designed decoding method that ensures the generation model
satisfies the lexical constraints. We only apply GBS to the T5-Base model due
to the memory constraint (+ G). Following the suggestions in [13], we use
CIDEr [21] and SPICE [22] to automatically assess the quality of generated
texts. We calculate constraint satisfaction for all constraints (ALL), novel
constraints (Novel) and seen constraints (Seen).

Main Results Table 2 shows that the NRETM model improves the constraint
satisfaction over the baselines for all cases, achieving close to 100% (i.e.,
99.3% and 99.4%). While GBS achieves perfect constraint satisfaction (i.e.,
100%), doing so significantly degrades the output text quality (more than 50
CIDEr), indicating the necessity integrating piror knowledge in training
rather than inference. In addition, both prior knowledge $P_{c}$ and
$\hat{P}_{c}$ have a positive effect on our model, improving our T5-large
baseline by 3.7 and 11.4 CIDEr score, respectively. Finally, our _T5-Large +
NRETM $\hat{P}_{c}$_ model outperforms the previous state-of-the-art result
[23], which integrates the _ConceptNet_ [24] into the BART model, suggesting
that our incorporated task-specific prior knowledge could be as powerful as
knowledge from large-scale hand-crafted corpus. We calculate the average
growth of CIDEr score for the four improvement directions: (1) double the
parameters (from _T5-Base_ to _T5-Large_) (2) add more piror knowledge (from
$P_{c}$ to $\hat{P}_{c}$) (3) add symbolic operator (only increase 0.9%
parameters by adding NRETM). The growth from (1) to (3) is 4.06, 6.55, 4.3.
Unexpectedly, adding NRETM into T5-Base model is better than adding another
T5-base, only adding a new piror knowledge in NRETM significantly outperforms
doubling the parameters. All of the above shows how potential it is to find a
method that could execute multiple rules effectively.

Table 2: Experiment Results on Commonsense. |  |  | Constraint
---|---|---|---
Method |  CIDEr |  SPICE |  Seen |  Novel |  ALL
T5-Base |  159.5 |  31.9 |  95.4 |  93.3 |  94.0
T5-Base + G |  110.7 |  27.8 |  100 |  100 |  100
T5-Base + NRETM $P_{c}$ |  164.4 |  32.1 |  95.7 |  94.6 |  95.6
T5-Base + NRETM $\hat{P}_{c}$ |  169.8 |  32.7 |  99.5 |  99.1 |  99.3
T5-Large |  163.6 |  32.4 |  94.4 |  93.0 |  93.9
T5-Large + NRETM $P_{c}$ |  167.3 |  33.0 |  93.9 |  93.8 |  93.9
T5-Large + NRETM $\hat{P}_{c}$ |  175.0 |  33.8 |  99.6 |  99.2 |  99.4
KGBART [23] |  168.3 |  32.7 |  - |  - |  98.6

###  3.3 Document-Level Machine Translation

Document-level machine translation tasks is a general text generation task,
where the goal is to translate segments of text that contain more than one
sentence (up to an entire document).

Dataset We conduct experiments on the common document-level MT dataset: TED15
Zh-En which is from IWSLT 2014 and 2015 [25, 26]. Following [14], we use
2010-2013 TED as the test set.

Predicate Logic Formulation The input is an ordered set of $n$ sentences in
the source language that form a document $\mathbf{x}=\\{x^{i}\\}_{i=1}^{n}$,
the expected output is a translated document
$\mathbf{y}=\\{y^{i}\\}_{i=1}^{n}$ = in the target language. We observed that
neural model is prone to sentence correspondence confusion (the $i^{th}$
sentence in source document is translated as the $j^{th}$ sentence in target
document) when doing document-level translation. To alleviate this problem, we
propose incorporating Doc-mBART25 with prior knowledge: each source sentence
should be translated only once. It is formulated as:

$\text{TranslatedOnce}(x^{i})=\left\\{\begin{array}[]{rcl}2&&{\theta(\mathbf{y}_{:t})~{}>~{}i}\\\
1&&{\theta(\mathbf{y}_{:t})~{}=~{}i}\\\
0&&{\theta(\mathbf{y}_{:t})~{}<~{}i}\end{array}\right.$ (14)

where $\theta(\cdot)$ returns the line number of $y_{t}$ in $\mathbf{y}$, as t
is monotonic during generation, the status only set to be 2 once. The
predicate logic constraint $P_{c}$ of this task can be formulated as:

$P_{c}=\land_{i=1}^{n}\big{(}\text{TranslatedOnce}(x^{i})\big{)}$

Baselines and Metrics We combine our NRETM $P_{c}$ component with the Doc-
mBART25 model proposed in [3] which is a state-of-the-art multilingual pre-
trained language model. We compare this model with the state-of-the-art non-
pretraining and pretraining approaches, including HAN (Hierarchical Attention
Networks) [14], Doc-mBART25 and Sen-mBART25 [3]. When implementing our model,
we use the same pre-processing method, blocks segmentation strategy and beam
search setting as [3]. Unlike [3], We use both document-level (d-BLEU) and
sentence-level (s-BLEU) to measure the similarities between generated target
document and the source document. Besides, we also calculate the Sentence
Aligned Ratio (SAR) between the source and target documents.

Table 3: Model Performance on TED15 Zh-En Test.

Model | s-BLEU | d-BLEU | SAR
---|---|---|---
Doc-mBART25 + NRETM $P_{c}$ | 24.9 | 30.4 | 100 %
Doc-mBART25 [3] | 23.8 | 29.6 | 98.7 %
Sen-mBART25 [3] | - | 28.4 | -
HAN [14] | - | 24.0 | -

Main Results Table 3 shows that the NRETM $P_{c}$ component helps the Doc-
mBART25 model to better capture the sentence-level corresponding relationship
between the source and target documents. In particular, sentence-level
alignment ratio is improved from 98.7% to 100%. The improvement in s-BLEU (+
1.1 BLEU) also confirms that our final Doc-mBART25 + NRETM $P_{c}$ model
learns to translate sentences based on the sentence order in source documents.

###  3.4 Discussion

Table 4: Novel Sentence Index Experiment. MR for Mention Ratio.

model | CSA | MR | R-L
---|---|---|---
T5 | 19.7 | 95.7 | 30.0
NRETM | 97.7 | 98.3 | 33.3

Zero-Shot Execution In Table 1, we show that simple sequence-to-sequence pre-
trained language model (i.e., T5) cannot handle complicated and fine-grained
constraints even after fine-tuning. Here, we further demonstrate that NRETM
model is capable to handle zero-shot rule execution. We train the T5 and NRETM
model to only mention storyline in the $3^{rd}$, $4^{th}$ and $5^{th}$
sentence and test these models to mention storyline in the first and second
sentence of the whole story. As shown in Table 4, although both T5 and NRETM
model mention most of the storyline (95.7% and 98.3% respectively) in the
generated story, the T5 model only mention 19.7% of storyline in the correct
sentence and the NRETM model makes 97.7% of storyline correct. This is becuase
the seuqence-to-sequence T5 model cannot recognize the novel sentence index
“1” and “2” during the generation. The logic tracker helps the NRETM model to
generalize to handle these cases.

Case Study More generation results of our models can be found in the
Supplementary Material A.1.

##  4 Related Work

We are mainly related to two lines of research work in the field of text
generation: constrained generation and prior knowledge integration. Some of
the earliest works [27, 28, 29] involve controllable generation methods where
the generators are trained on text data with the labeled target attributes.
CTRL[4], PPLM[6] and CoCon[30] are recent approaches that built on the
transformer-based large-scale pretrained LMs, they pay more attention on
controlling high-level attributes,phrases and keywords. While NRETM focuses on
controlling text generation to follow arbitrary logical constraints, leading
to a fine-grained control. Recently, GDC[31] permits to specify both pointwise
and distributional constraints over the target LMs. In the field of
constrained generation closest to our work is NEUROLOGIC[7] which seeks to
generate fluent text while satisfying complex lexical constraints (in a
predicate logic form). While NEUROLOGIC’s design also enables controlled
generation with logical constraints, our approach leverages executable
programs as logical operators and the dynamic tracking mechanism to realize a
highly customizable and scalable predicate logic function system rather than
simply controlling what words should and should not appear in the output.
Existing efforts [32, 33, 34, 35, 36] to incorporate prior knowledge into
sequence-to-sequence framework either resort to modifying model architectures
or designing training objectives. To the best of our knowledge, we first
attempt to formalize the prior knowledge integration in seq2seq generation as
text generation that conforms to predicate logic constraints.

##  5 Conclusion and Future Work

In this paper, we propose a unified controllable generation framework that
leverages predicate logic constraints to implement efficient complex fine-
grained control and scalable prior knowledge integration. We explore and
compare two controllable strategies: dynamic tracking and static strategy, and
show that the proposed dynamic tracking mechanism significantly outperforms
the static ones. Empirical results on three benchmarks indicate that NRETM
could achieve accurate control and exhibits a superior generation ability over
different tasks. Pre-trained models have been the dominant paradigm in natural
language processing, and researchers resort to massive data and large-scale
models to improve performance. We unify the rules used in various tasks into
the form of predicate logic, provide the possibility to pretrain models on
massive rules. In the future, we will explore pre-training large-scale neural
rule-execution machine with massive rules and data.

## Broader Impact

Our work proposes a unified and scalable approach to efficiently perform fine-
grained controllable text generation and incorporate multiple prior knowledge
for superior text generation performance. This work uses story generation,
machine translation, commonsense generation as applications to verify the
effectiveness. However, it could be used in a broader range of more complex
scenes (e.g., financial report generation) due to its advantages of combining
multiple rules to generate high-quality texts. On the positive side, more
efficient text generation can make these technologies more available to the
general public. Our model verified on machine translation can benefit
commercial and humanitarian translation services to help overcome language
barriers. With these advantages also come potential downsides: powerful
language models might be used to generate abuse, faked or misleading content
in the news or on social media; they might pose safety concerns if they are
used to generate harassing or hateful materials. In order to mitigate these
risks, AI systems could be leveraged to fight against misleading content and
harassing material.

## References

  * [1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [3] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online, July 2020. Association for Computational Linguistics.
  * [4] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher.  Ctrl: A conditional transformer language model for controllable generation.  arXiv preprint arXiv:1909.05858, 2019.
  * [5] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  arXiv preprint arXiv:1909.08593, 2019.
  * [6] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.  Plug and play language models: A simple approach to controlled text generation.  arXiv preprint arXiv:1912.02164, 2019.
  * [7] Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Neurologic decoding:(un) supervised neural text generation with predicate logic constraints.  arXiv preprint arXiv:2010.12884, 2020.
  * [8] Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li.  Modeling coverage for neural machine translation.  arXiv preprint arXiv:1601.04811, 2016.
  * [9] Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, and Eric Darve.  Ted: A pretrained unsupervised summarization model with theme modeling and denoising.  arXiv preprint arXiv:2001.00725, 2020.
  * [10] Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li.  Incorporating copying mechanism in sequence-to-sequence learning.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631–1640, Berlin, Germany, August 2016. Association for Computational Linguistics.
  * [11] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In International Conference on Machine Learning, pages 11328–11339. PMLR, 2020.
  * [12] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, San Diego, California, June 2016. Association for Computational Linguistics.
  * [13] Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren.  CommonGen: A constrained text generation challenge for generative commonsense reasoning.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823–1840, Online, November 2020. Association for Computational Linguistics.
  * [14] Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson.  Document-level neural machine translation with hierarchical attention networks.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947–2954, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
  * [15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc., 2017\.
  * [16] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-attention with relative position representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [17] Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight.  Towards controllable story generation.  In Proceedings of the First Workshop on Storytelling, pages 43–49, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [18] Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley.  Automatic keyword extraction from individual documents.  Text mining: applications and theory, 1:1–20, 2010.
  * [19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  arXiv preprint arXiv:1910.10683, 2019.
  * [20] Chris Hokamp and Qun Liu.  Lexically constrained decoding for sequence generation using grid beam search.  arXiv preprint arXiv:1704.07138, 2017.
  * [21] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [22] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In European conference on computer vision, pages 382–398. Springer, 2016.
  * [23] Ye Liu, Yao Wan, Lifang He, Hao Peng, and Philip S. Yu.  Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  * [24] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 4444–4451. AAAI Press, 2017.
  * [25] Mauro Cettolo, Christian Girardi, and Marcello Federico.  Wit3: Web inventory of transcribed and translated talks.  In Conference of European Association for Machine Translation, pages 261–268, 2012.
  * [26] Mauro Cettolo, Niehues Jan, Stüker Sebastian, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico.  The iwslt 2015 evaluation campaign.  In International Workshop on Spoken Language Translation, 2015.
  * [27] Jessica Ficler and Yoav Goldberg.  Controlling linguistic style aspects in neural language generation.  arXiv preprint arXiv:1707.02633, 2017.
  * [28] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu.  Seqgan: Sequence generative adversarial nets with policy gradient.  In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  * [29] Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura.  Controlling output length in neural encoder-decoders.  arXiv preprint arXiv:1609.09552, 2016.
  * [30] Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, and Jie Fu.  Cocon: A self-supervised approach for controlled text generation.  arXiv preprint arXiv:2006.03535, 2020.
  * [31] Muhammad Khalifa, Hady Elsahar, and Marc Dymetman.  A distributional approach to controlled text generation.  arXiv preprint arXiv:2012.11635, 2020.
  * [32] Jiacheng Zhang, Yang Liu, Huanbo Luan, Jingfang Xu, and Maosong Sun.  Prior knowledge integration for neural machine translation using posterior regularization.  arXiv preprint arXiv:1811.01100, 2018.
  * [33] Christos Baziotis, Barry Haddow, and Alexandra Birch.  Language model prior for low-resource neural machine translation.  arXiv preprint arXiv:2004.14928, 2020.
  * [34] Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura.  Improving neural machine translation through phrase-based forced decoding.  arXiv preprint arXiv:1711.00309, 2017.
  * [35] Yaohua Tang, Fandong Meng, Zhengdong Lu, Hang Li, and Philip LH Yu.  Neural machine translation with external phrase memory.  arXiv preprint arXiv:1606.01792, 2016.
  * [36] Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu.  Minimum risk training for neural machine translation.  arXiv preprint arXiv:1512.02433, 2015.

## Appendix

In this Appendix, we provide more details and examples for our proposed NRETM
models. A.1 provides the case study for three evaluation tasks. A.2 provides
the definitions of six predicates used in this paper. A.3 provides some more
analysis about predicate logic constraint and state flag as well as the actual
state flag matrix used in the Commonsense Generation task (Sec 3.2) and
Document-level Machine Translation task (Sec 3.3). A.4 provides more results
regarding the controllable ROCStories Generation Experiments (Sec 3.1).
Finally, A.5 provides implementation details of our models.

###  5.1 Case Study

In this case study, we select representative examples from our T5 baseline and
NRETM models. The _Logic expression_ is the actual predicate logic constraints
that we use in the model. The _Input expression_ is the actual string that we
feed into the T5 model simply because T5 vocabulary does not include $\land$,
$\lor$, and $\neg$.

Table 5: Generated examples of the T5 and NRETM model for the story generation
task. Task 1: Story Generation
---
Input expression: |  $\mathrm{Order}$(hated, stupid) & $\mathrm{Order}$(stupid, insulting) & $\mathrm{Order}$(insulting, punched)
Logic expression: |  $\mathrm{Order}$(hated, stupid) $\land$ $\mathrm{Order}$(stupid, insulting) $\land$ $\mathrm{Order}$(insulting, punched)
T5: |  I had a crush on a man. I told him I was stupid. He hated me for insulting me. I punched him in the face. He told me i was stupid.
NRETM: |  I had a crush on a guy. He was very rude. He hated me because i thought he was stupid and insulting me. I punched him in the face. He slapped me.
Input expression: |  $\mathrm{InSen}$(walking around, 2) & $\mathrm{InSen}$(school yard, 2)
Logic expression: |  $\mathrm{InSen}$(walking around, $y^{2}$) $\land$ $\mathrm{InSen}$(school yard, $y^{2}$)
T5: |  I went to the park today. I was walking around the school yard. I saw a squirrel in the yard. I saw a squirrel in the yard. I took a picture of the squirrel.
NRETM: |  Sam was playing in the yard. He was walking around in the school yard. He saw a squirrel in the yard. He ran into the squirrel. Sam was able to retrieve the squirrel.
Input expression: |  $\mathrm{InSen}$(waitressing job, 3) & ($\mathrm{Len}$(3, 16) || $\mathrm{StopWordCount}$(3, 8)) & (not $\mathrm{InSen}$(tenacity, 3)) & $\mathrm{InSen}$(talent agent, 4)
Logic expression: |  $\mathrm{InSen}$(waitressing job, $y^{3}$) $\land$ ($\mathrm{Len}$($y^{3}$,16) $\lor$ $\mathrm{StopWordCount}$($y^{3}$,8)) $\land$ ( $\neg$ $\mathrm{InSen}$(tenacity, $y^{3}$)) $\land$ $\mathrm{InSen}$(talent agent, $y^{4}$)
T5: |  Sarah had a crush on a girl named Sarah. Sarah s crush was Sarah’s boyfriend. Sarah’s boyfriend had a waitressing job and Sarah had a crush on Sarah . Sarah s boyfriend was a talent agent. Sarah’s crush was sarah’s boyfriend.
NRETM: |  Sarah wanted to be a chef. but she didn’t have the money to make a living. She applied for a waitressing job but she did n’t get a job at all . Sarah was approached by a talent agent. Sarah was hired and now she is a chef.
Input expression: |  $\mathrm{InSen}$(longest grocery trip, 1) & $\mathrm{Len}$(1, 12) & $\mathrm{InSen}$(line forever, 4) & $\mathrm{Len}$(4, 9)
Logic expression: |  $\mathrm{InSen}$(longest grocery trip, $y^{1}$) $\land$ $\mathrm{Len}$($y^{1}$, 12) $\land$ $\mathrm{InSen}$(line forever, $y^{4}$) $\land$ $\mathrm{Len}$($y^{4}$, 9)
T5: |  Sam was on the longest grocery trip of his life. He was so hungry he couldn’t wait to get to the store. When he got to the store he had to wait in line forever. Sam had to wait in line forever for his next grocery trip.
NRETM: |  I was on the longest grocery trip I ever went on. I had to wait in line for a long time. When I got there the line was long. I had to wait in line forever again. I was so glad I finally got to the store
Input expression: |  $\mathrm{InSen}$(could buy one, 2) & ($\mathrm{Len}$(2, 13) || $\mathrm{StopWordCount}$(2, 6)) & $\mathrm{InSen}$(would work, 4) & $(\mathrm{Len}$(4, 10) || $\mathrm{StopWordCount}$(4, 7))
Logic expression: |  $\mathrm{InSen}$(could buy one, $y^{2}$) $\land$ ($\mathrm{Len}$($y^{2}$, 13) $\lor$ $\mathrm{StopWordCount}$($y^{2}$, 6)) $\land$ $\mathrm{InSen}$(would work, $y^{4}$) $\land$ ($\mathrm{Len}$($y^{4}$, 10) $\lor$ $\mathrm{StopWordCount}$($y^{4}$, 7))
T5: |  I needed a new computer. I asked my mom if I could buy one. She said yes. I asked her if it would work for me. She said yes
NRETM: |  I needed a new computer. I went to the store to see if I could buy one. I went to the store to buy a computer. I found a computer that would work for me. I bought the computer and was very happy.

Table 5 shows selected examples in Controllable ROCStories Generation task.
This task is to show the controllability of our proposed NRETM model.
Sentences in red are the ones being controlled. In the first story, the T5
baseline model produces a short sentence and misses the order of storyline
"stupid" which should appear after generating the storyline "hated". While our
NRETM model successfully completes all storylines in order. In the second
story, the NRETM model controls the story generation in a more coherent way
than the T5 baseline model. Although both baseline and NRETM model successful
incorporate all given storylines, the T5 baseline model inconsistently
generates “school yard” just after generating the “park”. On the contrary, in
the story generated by the NRETM model, Sam consistently stays in the “yard”.
In the third story, the length and stop word control force the NRETM model to
generate sentences with more details, while the T5 baseline simply repeats
information from previous sentences. The NRETM model successfully generates
eight stop words in the third sentence, whereas the baseline model only
generates six stop words (highlighted via underline). In addition, the
generated story from the NRETM model has more rational plots than the one from
the T5 model. In the fourth story, the length of the first and fourth
sentences are controlled to be 12 and 9. The outputs of NRETM model
successfully obey these control constraints while the baseline model generates
11 and 13 tokens for the first and fourth sentences. In the last story, the
second sentence generated by the NRETM model successfully generates six stop
words (highlighted via underline). For this task, we are more concerned about
the expression rate of predicate logic control constraints than the quality of
the generated story. In addition to the case study, we have shown more
quantitative analysis, and please refer to Sec. A.4 for details.

Table 6: Generated Example of the T5 and NRETM model in the Commonsense
Generation task. Task 2: Commonsense Generation
---
Input expression: |  $\mathrm{Copy}$(stone) & $\mathrm{Copy}$(explain) & $\mathrm{Copy}$(knife) & $\mathrm{Copy}$(sharpen)
Logic expression: |  $\mathrm{Copy}$(stone) $\land$ $\mathrm{Copy}$(explain) $\land\mathrm{Copy}$(knife) $\land\mathrm{Copy}$(sharpen)
T5: |  a man is sharpening a knife on a stone
NRETM: |  a man explains how to sharpen a knife on a stone
Input expression: |  $\mathrm{Copy}$(stand) & $\mathrm{Copy}$(map) & $\mathrm{Copy}$(report) & $\mathrm{Copy}$(front) & $\mathrm{Copy}$(weather)
Logic expression: |  $\mathrm{Copy}$(stand) $\land$ $\mathrm{Copy}$(map) $\land$ $\mathrm{Copy}$(report) $\land$ $\mathrm{Copy}$(front) $\land$ $\mathrm{Copy}$(weather)
T5: |  map showing where the weather is standing at the front
NRETM: |  a man stands in front of a map reporting the weather
Input expression: |  $\mathrm{Copy}$(put) & $\mathrm{Copy}$(lipstick) & $\mathrm{Copy}$(talk) & $\mathrm{Copy}$(lip)
Logic expression: |  $\mathrm{Copy}$(put) $\land$ $\mathrm{Copy}$(lipstick) $\land$ $\mathrm{Copy}$(talk) $\land$ $\mathrm{Copy}$(lip)
T5: |  a woman puts lipstick on and talks about it
NRETM: |  a woman is talking and putting lipstick on her lips
Input expression: |  $\mathrm{Copy}$(iron) & $\mathrm{Copy}$(straighten) & $\mathrm{Copy}$(demonstrate) & $\mathrm{Copy}$(hair)
Logic expression: |  $\mathrm{Copy}$(iron) $\land$ $\mathrm{Copy}$(straighten) $\land$ $\mathrm{Copy}$(demonstrate) $\land$ $\mathrm{Copy}$(hair)
T5: |  a woman straightens her hair with an iron and shows how to do it
NRETM: |  a woman is demonstrating how to straighten her hair with an iron
Input expression: |  $\mathrm{Copy}$(bride) & $\mathrm{Copy}$(stand) & $\mathrm{Copy}$(bridesmaid) & $\mathrm{Copy}$(groomsman) & $\mathrm{Copy}$(groom)
Logic expression: |  $\mathrm{Copy}$(bride) $\land$ $\mathrm{Copy}$(stand) $\land$ $\mathrm{Copy}$(bridesmaid) $\land$ $\mathrm{Copy}$(groomsman) $\land$ $\mathrm{Copy}$(groom)
T5: |  bride standing with her bridesmaids and groomsmen
NRETM: |  the bridesmaids and groomsmen stand in front of the bride and groom
Input expression: |  $\mathrm{Copy}$(kitchen) & $\mathrm{Copy}$(watermelon) & $\mathrm{Copy}$(knife) & $\mathrm{Copy}$(cut)
Logic expression: |  $\mathrm{Copy}$(kitchen) $\land$ $\mathrm{Copy}$(watermelon) $\land$ $\mathrm{Copy}$(knife) $\land$ $\mathrm{Copy}$(cut)
T5: |  a knife cutting a watermelon in a kitchen
NRETM: |  a man cutting a watermelon with a knife in the kitchen

Table 6 shows selected examples from our T5 baseline and NRETM models in the
Commonsense Generation task. Concepts that are missed in the baseline model
outputs are in red. Words in blue are the key difference between the output of
baseline and NRETM model. Note that we omit the synonyms for simplicity. Full
Examples for this task can be found in Sec. A.3. Although the baseline model
can correctly complete many $\mathrm{Copy}$ operations, it fails when the
input combination is not commonly seen. For example, “explain” and “knife” in
the first example. The baseline model also generates meaningless sentence when
the inputs are complicated concepts combination in the second example. In
addition, the baseline model cannot handle the case where some input concepts
share the same prefix, such as “groom” and “groomsman” in the forth example.
The baseline model seems to merge these morphological similar input concepts
into a single concept and only mentions one of them in the outputs. Whereas
the NRETM model successfully completes all of $\mathrm{Copy}$ operations.

Table 7 shows selected examples from our T5 baseline and NRETM models in
document-level machine translation. In the first case, the mT5 baseline model
produces duplicated sentences (“what happens when co2 emissions go up”, in
red). As a consequence, it fails to translate a few important chunks in the
source sentences (see in Blue). This may due to the fact that the mT5 baseline
model cannot handle long input documents well. While our NRETM model
translates all source sentences into fluent English. Sentences in Green are
missed by the baseline model but successfully translated by the NRETM model
with the help of the predicate $\mathrm{translateOnce}$. In the second case,
the baseline model skips the important word “exchange” (see underline in the
Input expression) in its translated text (highlighted in red). The NRETM model
accurately translates this sentence (highlighted in blue). This shows that the
NRETM model is more focused on the current sentence than the T5 baseline
model.

Table 7: Generated Example of the mT5 and NRETM model for document-level
machine translation. Task 3: Document-level Machine Translation
---
Input expression: |  当前二氧化碳巨大的排放量将导致温度上升，温度的升高将引起一系列非常严重的后果，比如对天气的直接影响，或对生态系统的间接影响，生态系统无法应对剧烈变化的结果就是生态系统的全面崩溃。二氧化碳排放增加和温度升高究竟成怎样的关系，两者间的正反馈效应为何？这中间有一些不确定因素，但不多。至于全球变暖的具体负面影响有多严重，这无法完全确定，但肯定极其严重。$\mathrm{translateOnce}$(1) & $\mathrm{translateOnce}$(2) & $\mathrm{translateOnce}$(3)
mT5: |  what happens when co2 emissions go up – what happens when co2 emissions go up – is that the system collapses. what’s the positive feedback between co2 emissions and temperature rising, and what’s the negative feedback between co2 emissions and temperature rising?there’s some uncertainty about how severe it is, but it’s certainly severe.
NRETM: |  what’s going to happen when co2 emissions go up will cause temperatures to rise with a range of very severe consequences, such that, either directly or indirectly, ecosystems will collapse. there’s some uncertainty about how this relationship, the positive feedback, between co2 emissions and temperature rise will work. how severe it will be, it’s not entirely clear.
Input expression: |  我是跟着50个去往贾拉拉巴德战役的反叛者乘着卡车一起来的，当时我19岁，是一位住在佛罗里达州杰克逊维尔的素食主义者兼冲浪爱好者。我用我的黑色匡威的低邦鞋 换了一双棕色的皮拖鞋，并且对着我看不太清的政府的坦克发了一枚火箭。这是我第一次到阿富汗。$\mathrm{translateOnce}$(1) & $\mathrm{translateOnce}$(2) & $\mathrm{translateOnce}$(3)
mT5: |  I was riding a truck with 50 rebels who had gone to Jalabard to fight the Taliban, and I was 19 years old, a vegetarian and surfer in Jacksonville, Florida. I took my black shoe boots and a brown leather shoe, and I threw a rocket at a government tank that I couldn’t see. This was my first trip to Afghanistan.
NRETM: |  I was accompanied by 50 counter-insurgents on a truck heading to Jamalabad, Florida, when I was 19 years old, a vegetarian and surfers living in Jacksonville, Florida. I exchange brown leather sandals with my black jackets, and I fired a rocket at the tanks of the government that I didn’t see. This was my first time in Afghanistan

###  5.2 Definition of Predicates

Figure 3 to 8 show the six kinds of predicates, including $\mathrm{InSen}$,
$\mathrm{Order}$, $\mathrm{Copy}$, $\mathrm{TranslatedOnce}$, $\mathrm{Len}$
and $\mathrm{StopWordCount}$, used in our framework. The code in figures is
the pseudo code of the logical operator (i.e., executable programs), in which
y refers to $\mathbf{y_{:t}}$, y_i is $y^{i}$, state is the default state
status $\hat{q}_{it}^{k}$ and i_val is the additional intermediate values
$\bar{q}_{it}^{k}$ . In our experiments, all of these logical operators are
implemented using the Python programming language, and their source codes are
not directly visible to the neural text generators. They only communicate with
the neural text generators using the state flags. All predicates have State 0,
indicating unfinished status and State 2, indicating finished status. As
discussed in Sec 2.4, State 1 is an optional predicate-specific state. We will
introduce the definition and role of State 1 for each of the above predicate
if it exists in the captions.

Figure 3: The definition of predicate $\mathrm{InSen}$. The State 1 starts
when the text generators start to generate $k^{th}$ sentence. This informs the
model that it is possible to mention $x_{i}$ in the outputs. Figure 4: The
definition of predicate $\mathrm{Order}$. The State 1 starts when the previous
element $x\\_a$ has already been mentioned in the outputs. This informs the
model to mention $x\\_b$ next. Figure 5: The definition of predicate
$\mathrm{Copy}$. There is no State 1 in the definition of $\mathrm{Copy}$
because there is no “partial copy” status. Figure 6: The definition of
predicate $\mathrm{TranslatedOnce}$. The State 1 starts when $i^{th}$ sentence
is being translated. This informs the model should pay attention to which
source sentence. Figure 7: The definition of predicate $\mathrm{Len}$. The
State 1 starts when the text generator starts to generate $i^{th}$ sentence.
We also explicitly inform the model of how many tokens are remaining for the
current sentence. So they have State 1 with additional information $i\\_val$.
Figure 9 shows the actual state matrix of this predicate. Figure 8: The
definition of predicate $\mathrm{StopWordCount}$. The State 1 starts when the
text generator starts to generate $i^{th}$ sentence. We also explicitly inform
the model of how many stop words are remaining for the current sentence. So
they have State 1 with additional information $i\\_val$.

###  5.3 Predicate Logic Constraint and State Flag

As the predicate logic constraints $\mathcal{M}$ used in our framework could
support arbitrary formats (i.e., predicates can be in any combination with
logical words), a key challenge is mapping each state flag in the state matrix
to the corresponding predicate in the logic expression. To tackle this
challenge, we treat the predicate logic constraints (as "Logic expression" in
tables of Sec. A.1) as the extra input (as "Input expression" in tables of
Sec. A.1) to the encoder of sequence-to-sequence (_S2S_) transformer-based
text generators. In addition, we encode the state flags using a shallow
transformer-based encoder with the same architecture. With the help of both
positional embeddings in two modules, NRETM could align the state flags in
state matrix with predicates in logic expression to achieve successful
control. For the state flag $q_{it}$, it keep tracks of the _dynamic_ progress
of all predicates during text generation. Therefore, we only put the current
progress of each predicate in $q_{it}$ and encode it using a shallow rule
transformer encoder. For $\mathcal{M}$ with $n$ predicates
$\mathcal{D}=\\{U_{i}\\}_{i=1}^{n}$, $q_{it}$ is the concatenation of
$\\{q_{it}^{k}\\}_{k=1}^{n}$, where $q_{it}^{k}$ is the concatenation of
default state status $\hat{q}_{it}^{k}$ and optional intermediate values
$\bar{q}_{it}^{k}$. $\hat{q}_{it}^{k}$ is formulated as:

$\hat{q}_{it}^{k}=\left\\{\begin{array}[]{rcl}\text{N}&&{\text{variables
of}~{}U_{k}~{}\text{do not contain}~{}x_{i}}\\\ 0&&{U_{k}~{}\text{is not
satisfied}}\\\ 1&&{U_{k}~{}\text{is in progress}}\\\ 2&&{U_{k}~{}\text{is
satisfied}}\end{array}\right.$ (15)

To show the importance of our proposed rule-execution tracking module, our
baseline models also have access to the predicate logic constraints in their
encoders in all of our evaluation tasks. The baseline model and our NRETM
model have the same amount of input information and only differ in whether
equipped with the above rule-execution tracking module.

In Table 1, Sec 3.1, we control the length of arbitrary sentence for
ROCStories generation. Figure 9 shows a minimal example of controlling the
length of the second sentence. $\hat{q}_{it}^{k}$ is in status 0 when the
model is generating the first sentence. As the model finishes the first
sentence (i.e., after generating “!”), $q_{it}^{k}$ is updated to “1 5” (see
the definition of predicate $\mathrm{Len}$ in Figure 7). The
$\hat{q}_{it}^{k}$ is finally updated to status 2 when finishing this
sentence.

Figure 9: The State Matrix for controlling the length of the second sentence.
The yellow cell indicates the status update.

We further show the actual state matrix used in the Commonsense Generation and
the document-level machine translation task in Figure 10 and 11, respectively.
In both tasks, the number of $q_{it}^{k}$ is linear to the number of input
concept words or the number of input source sentences. In our implementation,
we compress $q_{it}^{k_{1}}$ and $q_{it}^{k_{2}}$ if they satisfy the
following two conditions:

  * •

$U_{k_{1}}$ and $U_{k_{2}}$ are the same predicate.

  * •

the variables of $U_{k_{1}}$ and $U_{k_{2}}$ are disjoint or the “in progress”
period of $U_{k_{1}}$ and $U_{k_{2}}$ are not overlapped.

After the above compression, the length of $\\{q_{it}^{k}\\}$ in the
Commonsense Generation task and Document-level Machine Translation task
becomes one.

Figure 10: The State Matrix for the Commonsense Generation task. In this task,
the status is updated once the keywords or phrases are fully mentioned in the
outputs. Figure 11: The State Matrix for the Document-level Machine
Translation. The yellow cell indicates the status update. In this task, the
status is updated once one sentence is finished.

###  5.4 More Results for Controllable ROCStories Generation Experiment

In this section, we conduct more analysis in Table 8 and 9 for the rules with
relatively low Constraint Success Ratio (CSR) in Table 1, Sec. 3.1
(Controllable ROCStories Generation Experiment). Specifically, they are:

  * •

$\mathrm{R}_{1}=\mathrm{InSen}(w_{1},y^{i})\land\mathrm{InSen}(w_{2},y^{j})\land\mathrm{Len}(y^{j},l_{j})\land\mathrm{Len}(y^{i},l_{i})$;

  * •

$\mathrm{R}_{2}=(\lnot\mathrm{InSen}(w_{3},y^{j}))\land(\mathrm{Len}(y^{i},l_{i})\lor\mathrm{StopWordCount}(y^{i},s_{i}))\land\mathrm{InSen}(w_{1},y^{i})\land\mathrm{InSen}(w_{2},y^{j})$;

  * •

$\mathrm{R}_{3}=(\mathrm{Len}(y^{i},l_{i})\lor\mathrm{StopWordCount}(y^{i},s_{i}))\land(\mathrm{Len}(y^{j},l_{j})\lor\mathrm{StopWordCount}(y^{j},s_{j}))\land\mathrm{InSen}(w_{1},y^{i})\land\mathrm{InSen}(w_{2},y^{j})$

The full CSR for $\mathrm{R}_{1}$, $\mathrm{R}_{2}$ and $\mathrm{R}_{3}$ is
relatively low in Table 1, Sec 3.1. As all of them involve length and stop
word control, we are interested in how accurate they are when they are allowed
to make small control errors. We calculate Constraint Success Ratio with
errors $\pm 1$ and $\pm 2$ in Table 8. Under errors $\pm 1$, the CSR of NRETM
model is significantly improved. In $\mathrm{R}_{3}$, the CSR of the NRETM
model is improved from 35.9% to 64.4%. In three rules, the NRETM model is 20%
- 30% higher than the T5 baseline model. Under errors $\pm 2$, among all three
rules, the lowest CSR for the NRETM model is 82.1%. The NRETM model can still
improve the CSR of baseline model by 6% to 11%. Note that $\pm 2$ is
relatively large error gap in ROCStories dataset becuase each sentence only
has, on average, 7.3 stop words. This explains the smaller CSR gap between the
NRETM and T5 baseline model in the CSR $\pm 2$ setup. In summary, the NRETM
model reasonably completes this controllable text generation task.

Table 8: Constraint Success Ratio with total length and stop word count errors $\pm 1$ and $\pm 2$. #R |  M |  CSR ($\pm 0$) |  CSR ($\pm 1$) |  CSR ($\pm 2$)
---|---|---|---|---
$\mathrm{R}_{1}$ |  T5 |  18.5 |  67.6 |  89.2
NRETM |  84.5 |  97.0 |  98.9
$\mathrm{R}_{2}$ |  T5 |  23.5 |  56.5 |  78.6
NRETM |  49.4 |  76.1 |  89.9
$\mathrm{R}_{3}$ |  T5 |  11.0 |  44.9 |  76.2
NRETM |  35.9 |  64.4 |  82.1

We further break the CSR into predicate level in Table 9. The NRETM model
achieves consistently higher CSR in all predicates than the T5 baseline model.
Specifically, both T5 baseline and NRETM model achieve near-perfect
performance in the predicate $\mathrm{InSen}$. We believe that this is due to
effect of the large-scale pre-training in the T5 model. The length control
$\mathrm{Len}$ is more challenging than $\mathrm{InSen}$. The NRETM model
achieves around CSR 90%, while the baseline model only achieves CSR 38%. This
shows that simply feeding the target length to the _S2S_ encoder cannot
properly control the output text length. When using the logical word $\lor$ in
$\mathrm{R}_{2}$ and $\mathrm{R}_{3}$, the CSR for $\mathrm{Len}$ is around 50
- 60%. Unlike $\mathrm{R}_{1}$ where the model is always trained to satisfy
predicate $\mathrm{Len}$, in $\mathrm{R}_{2}$ and $\mathrm{R}_{3}$, only 67%
of the training data satisfy predicate $\mathrm{Len}$. The predicate
$\mathrm{StopWordCount}$ is even more challenging: the NRETM model only
achieves 43.2% and 28.6% CSR in $\mathrm{R}_{2}$ and $\mathrm{R}_{3}$,
respectively. This may because the models have to distinguish between stop
word tokens and non-stop word tokens.

Table 9: Predicate-Level Constraint Success Ratio. $\mathrm{InS}$: $\mathrm{InSen}$; $\mathrm{L}$: $\mathrm{Len}$; $\mathrm{SWC}$: $\mathrm{StopWordCount}$; Model | Predicate-Level CSR
---|---
$\mathrm{R}_{1}$ |  Predicate |  $\mathrm{InS}(w_{1},y^{i})$ |  $\mathrm{InS}(w_{2},y^{j})$ |  $\mathrm{L}(y^{i},l_{i})$ |  $\mathrm{L}(y^{j},l_{j})$ |  - |  -
T5 |  99.5 |  99.0 |  38.0 |  36.2 |  - |  -
NRETM |  99.5 |  99.3 |  89.7 |  91.2 |  - |  -
$\mathrm{R}_{2}$ |  Predicate |  $\mathrm{InS}(w_{1},y^{i})$ |  $\mathrm{InS}(w_{2},y^{j})$ |  $\lnot\mathrm{InS}(w_{3},y^{j})$ |  $\mathrm{L}(y^{i},l_{i})$ |  $\mathrm{SWC}(y^{i},s_{i})$ |  -
T5 |  99.6 |  99.1 |  99.1 |  23.5 |  18.8 |  -
|  NRETM |  100 |  99.9 |  99.9 |  50.7 |  43.2 |  -
$\mathrm{R}_{3}$ |  Predicate |  $\mathrm{InS}(w_{1},y^{i})$ |  $\mathrm{InS}(w_{2},y^{j})$ |  $\mathrm{L}(y^{i},l_{i})$ |  $\mathrm{SWC}(y^{i},s_{i})$ |  $\mathrm{L}(y^{j},l_{j})$ |  $\mathrm{SWC}(y^{j},s_{j})$
T5 |  99.6 |  99.5 |  28.8 |  15.2 |  25.3 |  13.7
|  NRETM |  100 |  100 |  56.4 |  28.6 |  55.7 |  27.3

###  5.5 Implementation Details For Each Evaluation Task

In this section, we will introduce the implementation details of all our
evaluation tasks. In our experiments, we use three different pre-trained
language model, T5-base, T5-Large and MBart-Large. We use the implementation
of _huggingface transformers_ 222https://github.com/huggingface/transformers.
We modify their decoder models to integrate our state matrix and use their
provided model weights in our experiment. We only additional introduce the
_State Matrix Encoder_. It is a one-layer transformer encoder. Its hidden size
equals to the dimension of each head in the pre-trained transformer-based
langauge models. The size of its FFN layer is 256. The number of its heads is
4.

#### Controllable ROCStories Generation

We first use RAKE algorithm (implemented by https://github.com/csurfer/rake-
nltk) to extract storyline (i.e., key words and phrases) from the ground-truth
stories. In the ROCStories dataset, each story has 5 sentences. For extracted
storylines, we can easily find their original sentence index and ordering. We
can also extract total length and stop word counts from each sentence in the
ground-truth stories. We use these information to construct the training
rules. For rules with only logic $\land$, we simply use these extracted
ground-truth information as the predicate logic constraint. For rules with
logic $\lor$ (e.g., $\mathrm{R}_{2}$ and $\mathrm{R}_{3}$ in A.4), we create
all cases with equal proportion in the training data. For example, for clause
$\mathrm{Len}(y^{i},l_{i})\lor\mathrm{StopWordCount}(y^{i},s_{i})$, we create
33% of the training data only satisfy $\mathrm{Len}(y^{i},l_{i})$, 33% of the
training only satisfy $\mathrm{StopWordCount}(y^{i},s_{i})$ and the remaining
training data satisfy both of them. We can assign fake value for $l_{i}$ or
$s_{i}$ for the above data argumentation. To improve the generalization of our
pre-trained model, we freeze the parameters in the Self-Attention module and
Feed-Forward Layers in each layer of the T5 decoder. This parameters freezing
technology is applied to both T5 baseline models and the NRETM models in all
of our experiments. We use constant learning rate $5e^{-5}$ and batch size 32
for this experiment.

#### Commonsense Generation

In the Commonsense Generation task, we first use NLTK toolkit to expand each
input concept with all of its possible inflected forms, including plurals and
different tenses. We further search the mention position of each input
concept, including all of its inflected forms, on its corresponding ground-
truth references. With this mention position, we can construct the state
matrix shown in Figure 10 by putting Status 2 after this mention position and
Status 0 before this mention position. We use the same model and training
setup in the Controllable ROCStories Generation task. We use constant learning
rate $5e^{-5}$ and batch size 48 for this experiment.

#### Document-level Machine Translation

In the document-level Machine Translation, we split each documents into 2 - 4
trucks. Following the fine-tune setup in the original MBart paper, we use
learning rate $3e^{-5}$. But we use batch size 8 and total training step 80k
for our experiment.