In addition to reporting the quality guarantees of the model along training steps through local losses (cf. Figure <ref>), our experiments revealed that the absolute value difference $\norm{V_{\latentpolicy_{\decoderparameter}}}$ between the original and latent models operating under the latent policy  quickly decreases and tends to converge to values in the same range (Figure <ref>).

Absolute value difference $\norm{V_{\latentpolicy_{\decoderparameter}}}$ reported along training steps.

This is consistent with the fact that minimizing local losses lead to close behaviors (cf. Eq. <ref>) and that the value function is Lipschitz-continuous w.r.t. $\bidistance_{\latentpolicy_{\decoderparameter}}$ (cf. Section <ref>).

§.§ Remark on formal verification

Recall that our bisimulation guarantees come by construction of the latent space.
Essentially, our learning algorithm spits out a distilled policy and a latent state space which already yields a guaranteed bisimulation distance between the original MDP and the latent MDP.
This is the crux of how we enable verification techniques like model checking.
In particular, bisimulation guarantees mean that reachability probabilities in the latent MDP compared to those in the original one are close.
Furthermore, the value difference of (omega-regular) properties (formulated through mu-calculus) obtained in the two models is bounded by this distance (cf. Sect. <ref> and Chatterjee et al., 2010).

Reachability is the key ingredient to model-check MDPs.
Model-checking properties is in most cases performed by reduction to the reachability of components or regions of the MDP: it either consists of (i) iteratively checking the reachability of the parts of the state space satisfying path formulae that comprise the specification, through a tree-like decomposition of the latter (e.g., for (P,R-)CTL properties, cf. Baier & Katoen, 2008), or (ii) checking the reachability to the part of the state space of a product of the MDP with a memory structure or an automaton that embeds the omega-regular property — e.g., for LTL [Baier et al., 2016, Sickert et al., 2016], LTLf [Wells et al., 2020], or GLTL [Littman et al., 2017], among other specification formalisms.
The choice of specification formalism is up to the user and depends on the case study. The scope of this work is focusing on learning to distill RL policies with bisimulation guarantees so that model checking can be applied, in order to reason about the behaviors of the agent. That being said, reachability is all we need to show that model checking can be applied.

§.§ Hyperparameters

parameters. All components (e.g., functions or distribution locations and scales, see Fig. <ref>) are represented and inferred by neural networks (multilayer perceptrons).
All the networks share the same architecture (i.e., number of layers and neurons per layer).
We use a simple uniform experience replay of size $10^{6}$ to store the transitions and sample them.
The training starts when the agent has collected $10^{4}$ transitions in $\mdp$.
We used minibatches of size $128$ to optimize the objective and we applied a minibatch update every time the agent executing $\policy$ has performed $16$ steps in $\mdp$.
We use the recursive $\epsilon$-perturbation trick of [Huang, 2020] with $\epsilon = \nicefrac{3}{4}$: when an episode ends, it restarts from the initial state with probability $\nicefrac{1}{4}$; before re-starting an episode, the time spent in the reset state labeled with $\labelset{reset}$ follows then the geometric distribution with expectation $\nicefrac{\epsilon}{1 - \epsilon} = 3$.
We chose the same latent state-action space size than [Delgrange et al., 2022], except for LunarLander that we decreased to $\log_2 \left| \latentstates \right| = 14$ and $\left|\latentactions\right| = 3$ to improve the scalability of the verification.

VAE-MDPs parameters. For the comparison of Sect. <ref>, we used the exact same VAE-MDP hyperparameter set as prescribed by [Delgrange et al., 2022], except for the state-action space of LunarLander that we also changed for scalability and fair comparison purpose.
[The code for conducting the VAE-MDPs experiments is available at <https://github.com/florentdelgrange/vae_mdp> (GNU General Public License v3.0).]

Hyperparameter search. To evaluate our , we realized a search in the parameter space defined in Table <ref>.
The best parameters found (in terms of trade-off between performance and latent quality) are reported in Table <ref>.

We used two different optimizers for minimizing the loss (referred to as the minimizer) and computing the Wasserstein terms (reffered to as the maximizer).
We used Adam DBLP:journals/corr/KingmaB14 for the two, but we allow for different learning rates $\textsc{Adam}_{\alpha}$ and exponential decays $\textsc{Adam}_{\beta_1}, \textsc{Adam}_{\beta_2}$.
We also found that polynomial decay for $\textsc{Adam}_{\alpha}$ (e.g., to $10^{-5}$ for $4\cdot 10^{5}$ steps) is a good practice to stabilize the experiment learning curves, but is not necessary to obtain high-quality and performing distillation.
Concerning the continuous relaxation of discrete distributions,
we used a different temperature for each distribution, as [Maddison et al., 2017] pointed out that doing so is valuable to improve the results.
We further followed the guidelines of Maddison et al., 2017 to choose the interval of temperatures and did not schedule any annealing scheme (in contrast to VAE-MDPs).
Essentially, the search reveals that the regularizer scale factors $\beta_{\scalebox{1.1}{$\cdot$}}$ (defining the optimization direction) as well as the encoder and latent transition temperatures are important to improve the performance of distilled policies.
For the encoder temperature, we found a nice spot in $\temperature_{\scriptscriptstyle \embed_{\encoderparameter}} = \nicefrac{2}{3}$, which provides the best performance in general, whereas the choice of $\temperature_{\scriptscriptstyle \latentprobtransitions_{\decoderparameter}}$ and $\beta_{\scalebox{1.1}{$\cdot$}}$ are (latent-) environment dependent.
The importance of the temperature parameters for the continuous relaxation of discrete distributions is consistent with the results of [Maddison et al., 2017], revealing that the success of the relaxation depends on the choice of the temperature for the different latent space sizes.

Hyperparameter search. $\temperature_X$ refers to the temperature used for component $X$.

$\textsc{Adam}_{\alpha}$ (minimizer)
$\set{0.0001, 0.0002, 0.0003, 0.001}$

$\textsc{Adam}_{\alpha}$ (maximizer)
$\set{0.0001, 0.0002, 0.0003, 0.001}$

$\set{0, 0.5, 0.9}$

$\set{0.9, 0.999}$

neurons per layer
$\set{64, 128, 256, 512}$

number of hidden layers
$\set{1, 2, 3}$

$\set{\text{ReLU}, \text{Leaky ReLU}, \text{tanh}, \frac{\text{softplus}\fun{2x + 2}}{2} - 1 \, \textit{(smooth ELU)}}$

$\set{10, 25, 50, 75, 100}$

$\set{10, 25, 50, 75, 100}$

$\set{5, 10, 15, 20}$

$\set{10, 20}$

use $\varepsilon$-mimic (cf. Delgrange et al., 2022)
$\set{\text{True}, \text{False}}$ (if True, a decay rate of $10^{-5}$ is used)

$\set{0.1, \nicefrac{1}{3}, \nicefrac{1}{2}, \nicefrac{2}{3}, \nicefrac{3}{5}, 0.99}$

$\set{0.1, \nicefrac{1}{3}, \nicefrac{1}{2}, \nicefrac{2}{3}, \nicefrac{3}{5}, 0.99}$

$\set{\nicefrac{1}{\left| \latentactions \right| - 1}, \nicefrac{1}{\fun{\left| \latentactions \right| - 1} \cdot 1.5}}$

$\temperature_{\scriptscriptstyle \actionencoder}$
$\set{\nicefrac{1}{\left| \latentactions \right| - 1}, \nicefrac{1}{\fun{\left| \latentactions \right| - 1} \cdot 1.5}}$

Final hyperparameters used to evaluate in Sect. <ref>

CartPole           MountainCar               Acrobot           LunarLander              Pendulum

$\log_2 \left| \latentstates \right|$                                              9      10      13      14      13

$\left| \latentactions \right|$                                                    2 $=\left|\actions\right|$     $2=\left|\actions\right|$      $3=\left|\actions\right|$      $3$      $3$

activation                                                                         tanh      ReLU      Leaky Relu      ReLU      ReLU

layers                                                                             $[64, 64, 64]$      $[512, 512]$      $[512, 512]$      $[256]$      $[256, 256, 256]$

$\textsc{Adam}_{\alpha}$ (minimizer)                                              $0.0002$     $0.0001$     $0.0002$     $0.0003$     $0.0003$

$\textsc{Adam}_{\alpha}$ (maximizer)                                              $0.0002$     $0.0001$     $0.0001$     $0.0003$     $0.0003$

$\textsc{Adam}_{\beta_1}$                                                         $0.5$     $0$     $0$     $0$     $0.5$

$\textsc{Adam}_{\beta_2}$                                                         $0.999$     $0.999$     $0.999$     $0.999$     $0.999$

$\beta_{\localtransitionloss{\stationary{\policy}}}$                              $10$     $25$     $10$     $50$     $25$

$\beta_{\steadystateregularizer{\policy}}$                                        $75$     $100$     $10$     $100$     $25$

$\ncritic$                                               5      20      20      15      5

$\delta$                                                                          $20$     $10$     $20$     $20$     $10$

$\varepsilon$                                                                     $0$     $0$     $0$     $0$     $0.5$

$\temperature_{\scriptscriptstyle\latentprobtransitions_{\decoderparameter}}$     $\nicefrac{1}{3}$     $\nicefrac{1}{3}$     $0.1$     $0.75$     $\nicefrac{2}{3}$

$\temperature_{\scriptscriptstyle\embed_{\encoderparameter}}$                     $\nicefrac{1}{3}$     $\nicefrac{2}{3}$     $\nicefrac{2}{3}$     $\nicefrac{2}{3}$     $\nicefrac{2}{3}$

$\temperature_{\scriptscriptstyle\latentpolicy_{\decoderparameter}}$              $\nicefrac{2}{3}$     $\nicefrac{1}{3}$     $0.5$     $0.5$     $0.5$

$\temperature_{\scriptscriptstyle \actionencoder}$                                /     /     /     $\nicefrac{1}{3}$     $\nicefrac{1}{3}$

Labeling functions. We used the same labeling functions as those described by Delgrange et al., 2022.
For completeness, we recall the labeling function used for each environment in Table <ref>.

Time to failure properties.
Based on the labeling described in Table <ref>, we formally detail the time to failure properties checked in Sect. <ref> whose results are listed in Table <ref> for each environment.
Let $\labelset{Reset} = \set{\mathsf{reset}} = \tuple{0, \dots, 1}$ (we assume here that the last bit indicates whether the current state is a reset state or not)
and define $\state \models \labelset{L}_1 \wedge \labelset{L}_2$ iff $s \models \labelset{L}_1$ and $s \models \labelset{L}_2$ for any $\state \in \states$, then

* CartPole: $\varphi = \until{\neg \labelset{Reset}}{\labelset{Unsafe}}$, where $\labelset{Unsafe} = \tuple{1, 1, 0}$

* MountainCar: $\varphi = \until{\neg \labelset{Goal}}{\labelset{Reset}}$, where $\labelset{Goal} = \tuple{1, 0, 0, 0}$

* Acrobot: $\varphi = \until{\neg \labelset{Goal}}{\labelset{Reset}}$, where $\labelset{Goal} = \tuple{1, 0, \dots, 0}$

* LunarLander: $\varphi = \until{\neg \labelset{SafeLanding}}{\labelset{Reset}}$, where $\labelset{SafeLanding} = \labelset{GroundContact} \wedge \labelset{MotorsOff}$, $\labelset{GroundContact} = \tuple{0, 1, 0, 0, 0, 0, 0}$, and $ \labelset{MotorsOff} = \tuple{0, 0, 0, 0, 0, 1, 0}$

* Pendulum: $\varphi = \eventually\fun{\neg\mathsf{Safe} \wedge \ltlnext \mathsf{Reset}}$,
where $\labelset{Safe} = \tuple{1, 0, 0, 0, 0}$,
$\eventually \labelset{T} = \neg \until{\emptyset}{\labelset{T}}$, and
$\state_{i} \models \ltlnext \labelset{T}$ iff $\state_{i + 1} \models \labelset{T}$,
for any $\labelset{T} \subseteq \atomicprops, {{\state}_{i : \infty}, {\action}_{i : \infty} }\in \inftrajectories{\mdp}$.
Intuitively, $\varphi$ denotes the event of ending an episode in an unsafe state, just before resetting the environment, which means that either the agent never reached the safe region or it reached and left it at some point.
Formally, $\varphi =
\set{\seq{\state}{\infty}, \seq{\action}{\infty} \, | \, \exists i \in \N, \state_i \not\models \labelset{Safe}\, \wedge \, \state_{i + 1} \models \labelset{Reset} } \subseteq \inftrajectories{\mdp}$.

$\states \subseteq$
Description, for $\vect{\state} \in \states$
$\labels\fun{\vect{s}} = \tuple{p_1, \dots, p_n, p_{\mathsf{reset}}}$

c]@l@$\vect{\state}_1$: cart position
$\vect{\state}_2$: cart velocity
$\vect{\state}_3$: pole angle (rad)
$\vect{\state}_4$: pole velocity at tip
c]@l@$p_1 = \condition{\vect{\state}_1 \geq 1.5}$: unsafe cart position

$p_2 = \condition{\vect{\state}_3 \geq  0.15} $: unsafe pole angle
c]@l@$\vect{\state}_1$: position
$\vect{\state}_2$: velocity

c]@l@$p_1 = \condition{\vect{\state}_1 > 1.5}$: target position
$p_2 = \condition{\vect{\state}_1 \geq \nicefrac{-1}{2}}$: right-hand side of the mountain
$p_3 = \condition{\vect{\state}_2 \geq 0}$: car going forward
c]@l@Let $\theta_1, \theta_2 \in \mathopen[0, 2\pi \mathclose]$ be the angles
of the two rotational joints,
$\vect{\state}_1 = \cos\fun{\theta_1}$
$\vect{\state}_2 = \sin\fun{\theta_1}$
$\vect{\state}_3 = \cos\fun{\theta_2}$
$\vect{\state}_4 = \sin\fun{\theta_2}$
$\vect{\state}_5$: angular velocity 1
$\vect{\state}_6$: angular velocity 2

c]@l@$p_1= \condition{-\vect{\state}_1 -\vect{\state}_3 \cdot \vect{\state}_1 + \vect{\state}_4 \cdot \vect{\state}_2 > 1}$: RL agent target
$p_2 = \condition{\vect{\state}_1 \geq 0}$: $\theta_1 \in [0, \nicefrac{\pi}{2}] \cup [\nicefrac{3\pi}{2}, 2\pi]$

$p_3 = \condition{\vect{\state}_2 \geq 0} $: $\theta_1 \in [0, \pi]$

$p_4 = \condition{\vect{\state}_3 \geq 0}$: $\theta_2 \in [0, \nicefrac{\pi}{2}] \cup [\nicefrac{3\pi}{2}, 2\pi]$

$p_5 = \condition{\vect{\state}_4 \geq 0}$: $\theta_2 \in [0, \pi]$

$p_6 = \condition{\vect{\state}_5 \geq 0}$: positive angular velocity (1)

$p_7 = \condition{\vect{\state}_6 \geq 0}$: positive angular velocity (2)

c]@l@Let $\theta \in \mathopen[0, 2\pi \mathclose]$ be the joint angle
$\vect{\state}_1 = \cos\fun{\theta}$
$\vect{\state}_2 = \sin\fun{\theta}$
$\vect{\state}_3$: angular velocity

$p_1 = \condition{\vect{\state}_1 \geq \cos\fun{\nicefrac{\pi}{3}}}$: safe joint angle

$p_2 = \condition{\vect{\state}_1 \geq 0}$: $\theta \in [0, \nicefrac{\pi}{2}] \cup [\nicefrac{3\pi}{2}, 2\pi]$

$p_3 = \condition{\vect{\state}_2 \geq 0}$: $\theta \in [0, \pi]$

$p_4 = \condition{\vect{\state}_3 \geq 0}$: positive angular velocity

c]@l@$\vect{\state}_1$: horizontal coordinates
$\vect{\state}_2$: vertical coordinates
$\vect{\state}_3$: horizontal speed
$\vect{\state}_4$: vertical speed
$\vect{\state}_5$: ship angle
$\vect{\state}_6$: angular speed
$\vect{\state}_7$: left leg contact
$\vect{\state}_8$: right leg contact

$p_1$: unsafe angle

$p_2$: leg ground contact

$p_3$: lands rapidly

$p_4$: left inclination

$p_5$: right inclination

$p_6$: motors shut down

Labeling functions for the OpenAI environments considered in our experiments [Delgrange et al., 2022].
We provide a short description of the state space and the meaning of each atomic proposition.
Recall that labels are binary encoded, for
$n = |\atomicprops| - 1$ (one bit is reserved for $\mathsf{reset}$) and $p_{\mathsf{reset}} = 1$ iff $\vect{s}$ is a reset state (cf. Appendix <ref>).

§ ON THE CURSE OF VARIATIONAL MODELING

Posterior collapse is a well known issue occurring in variational models (see, e.g., DBLP:conf/icml/AlemiPFDS018,DBLP:conf/iclr/TolstikhinBGS18,DBLP:conf/iclr/HeSNB19,DBLP:conf/icml/DongS0B20) which intuitively results in a degenerate local optimum where the model learns to ignore the latent space and use only the reconstruction functions (i.e., the decoding distribution) to optimize the objective.
VAE-MDPs are no exception, as pointed out in the original paper ().

Formally, VAE- and WAE-MDPs optimize their objective by minimizing two losses: a reconstruction cost plus a regularizer term which penalizes a discrepancy between the encoding distribution and the dynamics of the latent space model.

In VAE-MDPs, the former corresponds to the the distortion, and the later to the rate of the variational model (further details are given in Alemi et al., 2018, Delgrange et al., 2022), while in our WAE-MDPs, the former corresponds to the raw transition distance and the later to both the steady-state and transition regularizers.
Notably, the rate minimization of VAE-MDPs involves regularizing a stochastic embedding function $\embed_{\encoderparameter}\fun{\sampledot \mid \state}$ point-wise, i.e., for all different input states $\state \in \states$ drawn from the interaction with the original environment.
In contrast, the latent space regularization of the WAE-MDP involves the marginal embedding distribution $\encodersymbol_{\encoderparameter}$ where the embedding function $\embed_{\encoderparameter}$ is not required to be stochastic.

[Alemi et al., 2018] showed that posterior collapse occurs in VAEs when the rate of the variational model is close to zero, leading to low-quality representation.

Latent space distribution along training steps. The intensity of the blue hue corresponds to the frequency of latent states produced from $\embed_{\encoderparameter}$ during training.
The vanilla model collapses to a single state.

Rate of the variational model.

Distortion of the variational model.

Average point-wise entropy of $\embed_{\encoderparameter}\fun{\sampledot \mid \state}$, for $\state \in \states$ drawn from the interaction with the original environment.

Performance of the resulting distilled policy $\latentpolicy_{\decoderparameter}$ when deployed in the original environment (averaged over 30 episodes).

Comparison of the VAE-MDP in the CartPole environment (i) when the distortion and the rate are minimized as is (vanilla model) and (ii) when it makes use of annealing schemes, entropy regularization, and prioritized experience replay to avoid posterior collapse (cf. Delgrange et al., 2022).
While the former clearly fails to learn a useful latent representation, the later does so meticulously and smoothly in two distinguishable phases:
first, $\embed_{\encoderparameter}$ focuses on fairly distributing the latent space, setting up the stage to the concrete optimization occurring from step $4 \cdot 10^5$, where the entropy of $\embed_{\encoderparameter}$ is lowered, which allows to get the rate of the variational model away from zero.
Five instances of the models are trained with different random seeds, with the same hyperparameters than in Sect. <ref>.

Posterior collapse in VAE-MDPs. We illustrate the sensitivity of VAE-MDPs to the posterior collapse problem in Fig. <ref>, through the CartPole environment[
In fact, the phenomenon of collapsing to few state occurs for all the environments considered in this paper when their prioritized experience replay is not used, as illustrated in .
]: minimizing the distortion and the rate as is yields an embedding function which maps deterministically every input state to the same sink latent state (cf. Fig. <ref>).
Precisely, there is a latent state $\latentstate \in \latentstates$ so that $\embed_{\encoderparameter}\fun{\latentstate \mid \state} \approx 1$ and $\latentprobtransitions_{\decoderparameter}\fun{\latentstate \mid \latentstate, \latentaction} \approx 1$ whatever the state $\state \in \states$ and action $\latentaction \in \latentactions$.
This is a form of posterior collapse, the resulting rate quickly drops to zero (cf. Fig <ref>), and the resulting latent representation yields no information at all.
This phenomenon is handled in VAE-MDPs by using (i) prioritized replay buffers that allow to focus on inputs that led to bad representation, and (ii) modifying the objective function for learning the latent space model — the so-called evidence lower bound DBLP:journals/jmlr/HoffmanBWP13,DBLP:journals/corr/KingmaW13, or ELBO for short — and set up annealing schemes to eventually recover the ELBO at the end of the training process.
Consequently, the resulting learning procedure focuses primarily on fairly distributing the latent space, to avoid it to collapse to a single latent state,

to the detriment of learning the dynamics of the environment and the distillation of the RL policy.
Then, the annealing scheme allows to make the model learn to finally smoothly use the latent space to maximize the ELBO, and achieve consequently a lower distortion at the “price” of a higher rate.

Impact of the resulting learning procedure. The aforementioned annealing process, used to avoid that every state collapses to the same representation, possibly induces a high entropy embedding function (Fig. <ref>), which further complicates the learning of the model dynamics and the distillation in the first stage of the training process.
In fact, in this particular case, one can observe that the entropy reaches its maximal value, which yields a fully random state embedding function.
Recall that the VAE-MDP latent space is learned through independent Bernoulli distributions.
Fig. <ref> reports values centered around $4.188$ in the first training phase, which corresponds to the entropy of the state embedding function when $\embed_{\encoderparameter}\fun{\sampledot \mid \state}$ is uniformly distributed over $\latentstates$ for any state $\state \in \states$: $H\fun{\embed_{\encoderparameter}\fun{\sampledot \mid \state}} = \sum_{i=0}^{\log_2 \left| \latentstates \right| - \left| \atomicprops \right| = 6} - p_i\log~p_i - \fun{1 - p_i} \log\fun{1 - p_i} = 4.188 $, where $p_i = \nicefrac{1}{2}$ for all $i$.
The rate (Fig. <ref>) drops to zero since the divergence pulls the latent dynamics towards this high entropy (yet another form of posterior collapse), which hinders the latent space model to learn a useful representation.
However, the annealing scheme increases the rate importance along training steps, which enables the optimization to eventually leave this local optimum (here around $4\cdot10^5$ training steps).
This allows the learning procedure to leave the zero-rate spot, reduce the distortion (Fig. <ref>), and finally distill the original policy (Fig. <ref>).

As a result, the whole engineering required to mitigate posterior collapse slows down the training procedure.
This phenomenon is reflected in Fig. <ref>: VAE-MDPs need several steps to stabilize and set up the stage to the concrete optimization, whereas WAE-MDPs have no such requirements since they naturally do not suffer from collapsing issues (cf. Fig. <ref>), and are consequently faster to train.

Lack of representation guarantees. On the theoretical side, since VAE-MDPs are optimized via the ELBO and the local losses via the related variational proxies, VAE-MDPs do not leverage the representation quality guarantees induced by local losses (Eq. <ref>) during the learning procedure (as explicitly pointed out by .): in contrast to WAE-MDPs, when two original states are embedded to the same latent, abstract state, the former are not guaranteed to be bisimilarly close (i.e., the agent is not guaranteed to behave the same way from those two states by executing the policy), meaning those proxies do not prevent original states having distant values collapsing together to the same latent representation.

[m0]$\mdp = \mdptuple$MDP $\mdp$ with state space $\states$, action space $\actions$, transition function $\probtransitions$, labeling function $\labels$, atomic proposition space $\atomicprops$, and initial state $\sinit$.
[msinit]$\mdp_\state$MDP obtained by replacing the initial state of $\mdp$ by $\state \in \states$
[mtrajectory]$\trajectory = \trajectorytuple{\state}{\action}{T}$Trajectory
[mtrajectories]$\inftrajectories{\mdp}$Set of infinite trajectories of $\mdp$
[mpolicy]$\policy$Memoryless policy $\policy \colon \states \to \distributions{\actions} $
[pdist]$\distributions{\measurableset}$Set of measures over a complete, separable metric space $\measurableset$
$\condition{[\textit{cond}]}$indicator function: $1$ if the statement [cond] is true, and $0$ otherwise
$f_{\decoderparameter}$A function $f_{\decoderparameter} \colon \measurableset \to \R$ modeled by a neural network, parameterized by $\decoderparameter$, where $\measurableset$ is any measurable set
[mprobmeasure]$\Prob_{\policy}^{\mdp}$Unique probability measure induced by the policy $\policy$ in $\mdp$ on the Borel $\sigma$-algebra over measurable subsets of $\inftrajectories{\mdp}$
[mpolicies]$\mpolicies{\mdp}$Set of memoryless policies of $\mdp$

[mstationary]$\stationary{\policy}$Stationary distribution of $\mdp$ induced by the policy $\policy$
[mdiscount]$\discount$Discount factor in $\mathopen[0, 1\mathclose]$
[mstate]$\state$State in $\states$
[maction]$\action$Action in $\actions$
[mlimitingdistr]$\stationary{\policy}^{t}$Limiting distribution of the MDP defined as  $\stationary{\policy}^{t}\fun{\state' \mid \state} = \Prob_{\policy}^{\mdp_{\state}}\fun{\set{\seq{\state}{\infty}, \seq{\action}{\infty} \mid \state_t = \state'}}$, for any source state $\state \in \states$
[mvalues]$\valuessymbol{\policy}{\sampledot}$Value function for the policy $\policy$
[mreachability]$\until{\labelset{C}}{\labelset{T}}$Constrained reachability event
[l1]$\latentmdp = \latentmdptuple$Latent MDP with state space $\latentstates$, action space $\latentactions$, reward function $\latentrewards$, labeling function $\latentlabels$, atomic proposition space $\atomicprops$, and initial state $\zinit$.
[lstate]$\latentstate$Latent state in $\latentstates$
[laction]$\latentaction$Latent action in $\latentactions$
[lembedding]$\embed$State embedding function, from $\states$ to $\latentstates$
[lembeddingaction]$\embeda$Action embedding function, from $\latentstates \times \latentactions$ to $\actions$
[l2latentspacemodel]$\tuple{\latentmdp, \embed, \embeda}$Latent space model of $\mdp$
[ldistance]$\distance_{\latentstates}$Distance metric over $\latentstates$
[lantentpolicy]$\latentpolicy$Latent policy $\latentpolicy\colon \latentstates \to \actions$; can be executed in $\mdp$ via $\embed$: $\latentpolicy\fun{\sampledot \mid \embed\fun{\state}}$
[localtransitionloss]$\localtransitionloss{\stationary{}}$Local transition loss under distribution $\stationary{}$
[localrewardloss]$\localrewardloss{\stationary{}}$Local reward loss under distribution $\stationary{}$
[lembedprob]$\embed\probtransitions$Distribution of drawing $\state' \sim \probtransitions\fun{\sampledot \mid \state, \action}$, then embedding $\latentstate' = \embed\fun{\state'}$, for any state $\state \in \states$ and action $\action \in \actions$
[mbidistance]$\bidistance_{\policy}$Bisimulation pseudometric
[pdiscrepancy]$D$Discrepancy measure; $D\fun{P, Q}$ is the discrepancy between distributions $P, Q \in \distributions{\measurableset}$
[pwasserstein]$\wassersteinsymbol{\distance}$Wasserstein distance w.r.t. the metric $\distance$; $\wassersteindist{\distance}{P}{Q}$ is the Wasserstein distance between distributions $P, Q \in \distributions{\measurableset}$
$\Lipschf{\distance}$Set of $1$-Lipschitz functions w.r.t. the distance metric $\distance$
[wbehavioral]$\stationary{\decoderparameter}$Behavioral model: distribution over $\states \times \actions \times \images{\rewards} \times \states$
[mtracedistance]$\tracedistance$Raw transition distance, i.e., metric over $\states \times \actions \times \images{\rewards} \times \states$
[wstationary]$\latentstationaryprior$Stationary distribution of the latent model $\latentmdp_{\decoderparameter}$, parameterized by $\decoderparameter$
[wmarginalencoder]$Q_{\encoderparameter}$Marginal encoding distribution over $\latentstates \times \latentactions \times \latentstates: \expectedsymbol{\state, \action, \state' \sim \stationary{\policy}} \embed_{\encoderparameter}\fun{\sampledot \mid \state, \action, \state'}$
[wembeddinga]$\embed^{\actions}_{\encoderparameter}$Action encoder mapping $\latentstates \times \actions$ to $\distributions{\latentactions}$
[wtransition]$\originaltolatentstationary{}$Distribution of drawing state-action pairs from interacting with $\mdp$, embedding them to the latent spaces, and finally letting them transition to their successor state in $\latentmdp_{\decoderparameter}$, in $\distributions{\latentstates \times \latentactions \times \latentstates}$
[mdistancestate]$\distance_{\states}$Metric over the state space
[mdistanceaction]$\distance_{\actions}$Metric over the action space
[mdistancerewards]$\distance_{\rewards}$Metric over $\images{\rewards}$
[wgenerate]$\generative_{\decoderparameter}$State-wise decoder, from $\latentstates$ to $\states$
[wdirac]$G_{\decoderparameter}$Mapping $\tuple{\latentstate, \latentaction, \latentstate'} \mapsto \tuple{\generative_{\decoderparameter}\fun{\latentstate}, \embeda_{\decoderparameter}\fun{\latentstate, \latentaction}, \latentrewards_{\decoderparameter}\fun{\latentstate, \latentaction}, \generative_{\decoderparameter}\fun{\latentstate'}}$
[wsteadystate]Steady-state regularizer
[wtransitionlossnet]Transition Lipschitz network
[wsteadystatenet]Steady-state Lipschitz network
$\sigmoid$Sigmoid function, with $\sigmoid\fun{x} = \nicefrac{1}{1 + \exp\fun{-x}}$
[lvalue]$\latentvaluessymbol{\latentpolicy}{\sampledot}$Latent value function
[lpolicies]$\latentpolicies$Set of (memoryless) latent policies
[wtemperarue]$\temperature$Temperature parameter
[pLogistic]$\logistic{\mu}{s}$Logistic distribution with location parameter $\mu$ and scale parameter $s$


one, at least for our data generating process.

Figure C.3 displays the estimate profiles of unobserved heterogeneity for
$N=10,000$ observations and a varying number of groups. Figures C.3(a) and
C.3(b) show that the time profiles do not exhibit a sufficient amount of
unobserved heterogeneity which results in biased coefficient estimates (see
Figure C.2). By contrast, adding more groups than necessary does not imply
that the estimator does not assign any individual observations to these
superfluous groups. Rather, the GFE splits existing groups which leads to an
“overfitting” of the time profiles (see Figures C.3(d) and C.3(e)). This
behavior is similar to what we observe in our empirical, real data
application. Adding more groups splits up the existing groups and the
generated trajectories of unobserved mental health profiles for the additional
groups are similar to the group that was split up.

Finally, we investigate the finite sample behavior of the BIC criterion with
two different penalty terms (Figures C.4 and C.5) in a setting with large $N$
and fixed $T$. As discussed in Section 5.2 the BIC preferred by Bonhomme and
Manresa (2015) (BIC standard) does not discriminate sufficiently for all
$G\geq$ in our our application. Our simulation exercise clearly shows that the
number of groups selected by the BIC standard depends on the number of
observations $N$ relative to the number of time periods $T$. As shown in
Figure C.5, the BIC selects the correct number of groups, $G=3$, for $1,000$
observations. However, when we increase $N$, the number of groups selected by
this BIC increases, indicating that the penalization used in this BIC is not
steep enough. As in our application, the BIC standard remains practically
unchanged when increasing the number of groups once $G>1$.

By contrast, the BIC with a steeper penalty term (in $G$) always chooses two
groups regardless of the number of observations. As indicated by the steep
increase in the value of this BIC, the penalization with respect to the number
of groups is too strong (Figure C.4). We observe a similar behavior in our
application.

(a) $N=1000$ (b) $N=1500$ (c) $N=2000$ (d) $N=10000$

Figure C.4: Results for the BIC with the steeper penalty in terms of $G$.

(a) $N=1000$ (b) $N=1500$ (c) $N=2000$ (d) $N=10000$

Figure C.5: Results for the BIC with the less steep penalty used in Bonhomme
and Manresa (2015).

(a) $N=2500$ (b) $N=3000$

Figure C.6: Estimated coefficient for $\xi$ with lags for different model
specifications from left to right: (1) OLS, (2) OLS with individual-specific
fixed-effects (OLS FE), (3) GFE estimator with two groups, (4) GFE estimator
with three groups, (5) GFE estimator with four groups, (6) GFE estimator with
five groups. Confidence intervals are depicted in red and are calculated using
analytical standard errors. The true contemporaneous effect is $\xi_{0}=0.2$
and the true data generating process contains lags of the form
$\xi_{1},\xi_{2},...,\xi_{5}=0.1$, that are ignored in estimation.

###  D Model solution

Numerically, the decision function $\rho^{*}_{t}$ can be computed by backward
induction over $t$, starting with a guess in the far future which does not
affect behavior in initial periods. The backward recursion is a variation of
classical dynamic programming that takes into account the time-inconsistency
introduced through $\beta$. To facilitate formulating the dynamic program, we
denote by $F_{t}(M_{t})$ the (classically, without $\beta$) discounted sum of
future flow utilities given the decisions $\rho^{*}$ at age $\tau>t$

$\displaystyle
F_{t}(M_{t})=u(\rho^{*}_{t}(M_{t}),M_{t})+\operatorname{\mathbb{E}}_{t}\left[\sum_{\tau=t+1}^{\infty}\delta^{\tau-t}u(\rho^{*}_{\tau}(M^{*}_{\tau}),M^{*}_{\tau})\right].$
(D.1)

Every woman’s optimization problem specified by Equation (14) of computing
$\rho^{*}_{t}(M_{t})$ can be written compactly in terms of $F$ as

$\displaystyle\rho^{*}_{t}(M_{t})=\text{arg}\max_{\rho_{t}}\left\\{u(\rho_{t},M_{t})+\beta\delta\operatorname{\mathbb{E}}_{t}[F_{t+1}(M^{*}_{t+1})]\right\\}$
(D.2)

Moreover, the functions $F_{t}$ satisfy the recursion

$\displaystyle
F_{t}(M_{t})=u(\rho^{*}_{t}(M_{t}),M_{t})+\delta\operatorname{\mathbb{E}}_{t}[F_{t+1}(M^{*}_{t+1})]$
(D.3)

Our numerical approach via backward induction thus looks as follows. We
initialize by guessing the terminal condition $F_{T}(M_{T})=\bar{u}(M_{T})=0$
for $T=100$.343434This guess is incorrect but can be expected not to affect
behavior in early time periods $t=1,\ldots 8$. We verify this by checking that
results do not change if we initialize instead at $T=200$. In order to
sequentially compute the decision function $\rho^{*}_{t}$ and the functions
$F_{t}$, we alternate two steps backwards in time. Assume that $F_{t+1}$ is
already known. Then, we can compute $\rho^{*}_{t}(M_{t})$ by solving the
problem in Equation (D.2) for every value of $M_{t}$. Once
$\rho_{t}^{*}(M_{t})$ is known, we can compute $F_{t}$ from Equation (D.3) and
go back one more step in time. When solving this problem computationally, we
first discretize the state variables $M$ and $\rho$ over a suitable
grid.353535We assume that the choice of $\rho$ is discrete and the possible
value are $0,0.05,0.1,...,0.95,1$. For $M$ we simulate 1000 trajectories for
the two most extreme values of $\rho$, $\rho_{t}\equiv 0$ and $\rho_{t}\equiv
1$. We use the maximum and minimum of resulting mental health trajectories to
determine the boundaries of the grid. We choose 200 equidistant levels in
between. Thus, the maximizations do not have to be performed for every
possible value of $M$ and $\rho$, but only for every value on the grid. The
resulting discrete functions $F_{t}$ are interpolated using monotone Hermite
splines. To compute the conditional expectations at each age, we take a Monte
Carlo average over 1,000 possible scenarios for $M_{t+1}$ for the next step
given each combination $(M_{t},\rho_{t})$. In this way, the functions $F_{t}$
and $\rho^{*}_{t}$ can be computed backward in time one by one. With the
resulting decision functions $\rho^{*}_{t}$, we then simulate 10,000 optimal
mental health trajectories $M^{*}_{t}$ and the associated risky behavior
$\rho^{*}_{t}(M^{*}_{t})$.


# Synthesizing Monolingual Data for Neural Machine Translation

Benjamin Marie Atsushi Fujita
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan
{bmarie<EMAIL_ADDRESS>
###### Abstract

In neural machine translation (NMT), monolingual data in the target language
are usually exploited through a method so-called “back-translation” to
synthesize additional training parallel data. The synthetic data have been
shown helpful to train better NMT, especially for low-resource language pairs
and domains. Nonetheless, large monolingual data in the target domains or
languages are not always available to generate large synthetic parallel data.
In this work, we propose a new method to generate large synthetic parallel
data leveraging very small monolingual data in a specific domain. We fine-tune
a pre-trained GPT-2 model on such small in-domain monolingual data and use the
resulting model to generate a large amount of synthetic in-domain monolingual
data. Then, we perform back-translation, or forward translation, to generate
synthetic in-domain parallel data. Our preliminary experiments on three
language pairs and five domains show the effectiveness of our method to
generate fully synthetic but useful in-domain parallel data for improving NMT
in all configurations. We also show promising results in extreme adaptation
for personalized NMT.

##  1 Introduction

Neural machine translation (NMT) systems usually require a large quantity of
parallel data for training. For most language pairs and domains, we do not
have such resources, or only in very small quantities, mainly because they are
costly to produce (Germann, 2001). Unlike parallel data, monolingual data are
readily available in large quantity for many languages. Previous work has
proposed various strategies to integrate monolingual data into NMT systems and
has confirmed their usefulness to improve NMT systems, especially in low-
resource configurations. The so-called _back-translation_ of monolingual data
(Sennrich et al., 2016a) is undoubtedly the most prevalent one. This approach
uses a target-to-source MT system to translate monolingual data in the target
language into the source language. The generated synthetic parallel data can
be used together with the original parallel data to increase the size of the
training data, and eventually to obtain better NMT systems.

Nonetheless, generating synthetic parallel data in large quantity with this
approach also requires a large quantity of monolingual data. For most domains
in most languages, however, a large quantity of monolingual data is
unavailable and thus generating large synthetic parallel data through back-
translation is impracticable.

In this preliminary work, we propose a new approach that leverages small in-
domain monolingual data to generate large synthetic in-domain parallel data.
We demonstrate that synthetic in-domain monolingual data generated by a GPT-2
model (Radford et al., 2019), fine-tuned on our very small in-domain
monolingual data, can be successfully translated by NMT to generate synthetic
in-domain parallel data. Our results on three language pairs and five domains
show improvements in BLEU for all configurations when using our synthetic data
to train NMT. We also show that this approach can be used in extreme
adaptation for personalized NMT.

(a) Generated by GPT-2 fine-tuned on Medical data

> $\ldots$ Because of methodological differences we could not obtain a
> comparable result for 17S viral nucleic acids or 16S viral nucleic acid
> using different methods.
> The SARI statement: A measure of the quality of health services, including
> the availability of drugs, is a basic criterion for measuring the quality of
> health services system.
> The 12 patients seen at the DCP + IC applied for six of these six HDCP
> methods (75%) successfully completed their pharmacy duties as per the
> guidelines.$\ldots$

(b) Generated by GPT-2 fine-tuned on IT data

> $\ldots$You can use the Page Colors application that you can find on Google+
> The maps of Portugal are free but you can acquire maps for other countries
> You can use the program Ringtone Maker which you can find on Google$\ldots$

(c) Generated by GPT-2 fine-tuned on tweets (natural disaster domain)

> $\ldots$A volcanic eruption in 1815 set off a massive effluence that sowed
> #wildfire on the west coast!Thanks NSW #NSWgovernors for treating these
> #shills
> 4.4 earthquake occurred near Negros Region, Chile at 22: 10 UTC! #earthquake
> #NegrosRegion
> Day: Malta - Black cloud surrounded by rain. 16: 35 JST 14 / 16 / 19 - 21:
> 45 JST 15 / 16 / 19 - 17: 00 JST$\ldots$

(d) Generated by GPT-2 not fine-tuned

> $\ldots$On Thursday, fossil fuels minister Emmanuel Ponting said the year
> 2000 was the year that the entire human genome was analyzed and ”explored
> for new genes.”
> Consider the mercenary work that Columbia University puts in.
> Such coins have been suggested by Buzzfeed tech reporter Alex Seitz, who
> wrote a very thorough investigation into the issue.$\ldots$

Figure 1:  Examples of three raw consecutive lines inside one sequence
generated by different GPT-2 models. We manually added “$\ldots$” to show the
reader that they are extracts from a sequence. GPT-2 models, fine-tuning data,
and hyper-parameters used to obtain these examples are presented in Section 4.

##  2 Motivation

This work relies on three assumptions:

  * •

GPT models generates mostly correct sentences.

  * •

Sentences generated by a GPT model exhibit some of the characteristics of the
in-domain data on which the model has been fine-tuned, even if the data is
small.

  * •

NMT training is robust, to some extent, to the noise in the texts generated by
GPT models.

For our two first assumptions, we can obtain some hints on their validity by
manually checking sentences generated by fine-tuned GPT-2 models. Examples of
such sentences are presented in Figure 1. We can see with these examples that
GPT-2 models successfully generate sentences that are mostly correct and
present characteristics of the domain on which they have been fine-tuned.

For our third assumption, we rely on previous work that shows that back-
translations in which artificial noise has been injected can improve
translation quality when used for training NMT (Edunov et al., 2018).

##  3 Synthesizing Large Parallel Data Leveraging Small Monolingual Data

###  3.1 Requirements

Our method has few requirements in terms of data that make it applicable in
most MT scenarios. Precisely, we need the following three types of data:

  * •

a GPT model or large (general-domain) monolingual data: in this preliminary
work, we only exploit the smallest GPT-2 model released by OpenAI. For the
future, we plan to experiment with in-house GPT models, trained on large
general-domain monolingual data.

  * •

small in-domain monolingual data: most of our experiments use 50k sentences
for each target domain, but experiments in extreme adaptation for personalized
NMT shows that our method is useful even when only hundreds of sentences are
available.

  * •

some parallel data: all our experiments use at least 156k sentence pairs.

###  3.2 Synthetic Monolingual Data

We use GPT-2 (Radford et al., 2019)111https://github.com/openai/gpt-2 to
generate synthetic monolingual data. GPT models are auto-regressive
Transformer (Vaswani et al., 2017) decoders. Given some context, or no context
at all if this is the first token of the sequence, the model predicts the next
token.

To generate texts in a particular domain, we fine-tuned a given GPT-2 model on
a small amount of texts in the target domain and language. Since GPT-2 is
efficient for text generation, we can generate millions of in-domain
monolingual sentences.

###  3.3 Synthetic Parallel Data

Once the synthetic monolingual data are generated, it can be used in NMT as
any other monolingual data. In this work, we demonstrate its usefulness
through back-translation (Sennrich et al., 2016a) and forward translation to
generate in-domain synthetic parallel data.

For back-translation, we adopted the tagged approach (Caswell et al., 2019)
that has been shown to provide better results, especially for translating
texts that are not translationese (Marie et al., 2020). In this configuration,
the target side of the synthetic parallel data was generated by GPT-2, in
English, and the source side by NMT.

For forward translation, we did not use tags. In this configuration the source
side was generated by GPT-2, in English, while the target side was obtained
through NMT. Forward translation is known to underperform back-translation
(Bogoychev and Sennrich, 2019). Nonetheless, since we do not have GPT-2 models
in other languages than English, we could only exploit synthetic monolingual
data for translation directions with English on the source side through
forward translation.

##  4 Experiments

###  4.1 Data

####  4.1.1 Training

We trained NMT systems for English–German (En-De), English–French (En-Fr), and
Japanese–English (Ja-En) on the following parallel data (numbers of sentence
pairs are given after pre-processing described in Section 4.2):

  * •

En-De: WMT17222http://statmt.org/wmt17/translation-task.html parallel data
(5.1M sentence pairs)

  * •

En-Fr: WMT14333http://statmt.org/wmt14/translation-task.html (32.7M sentence
pairs)

  * •

En-Ja: Training parallel data provided in the MTNT dataset (Michel and Neubig,
2018b)444http://www.cs.cmu.edu/ pmichel1/mtnt/ which is a concatenation of
three different datasets (TED talks555https://wit3.fbk.eu/, The Kyoto Free
Translation Task (KFTT)666http://www.phontron.com/kftt/, and
JESC777https://nlp.stanford.edu/projects/jesc/) (3.9M sentence pairs)

####  4.1.2 Validation

We used one validation dataset, for each language pair, to select the best
model after training NMT (see Section 4.2):

  * •

En-De: WMT16 newstest888http://data.statmt.org/wmt20/translation-task/dev.tgz
(2,999 sentence pairs)

  * •

En-Fr: WMT13 newstest999http://data.statmt.org/wmt20/translation-task/dev.tgz
(3,000 sentence pairs)

  * •

En-Ja: Validation data provided in the MTNT
dataset101010http://www.cs.cmu.edu/ pmichel1/mtnt/ that is a concatenation of
data from TED Talks, KFTT, and JESC corpora (4,451 sentence pairs)

####  4.1.3 Test

We used several datasets from different domains for evaluating the translation
quality of our NMT systems for each language pair:

  * •

En-De:

    * –

News domain: WMT17 news translation
task111111http://data.statmt.org/wmt20/translation-task/dev.tgz (3,004
sentence pairs)

    * –

Medical domain: WMT14 medical translation task, khresmoi
summary121212http://www.statmt.org/wmt14/medical-task/khresmoi-summary-test-
set.tgz (1,000 sentence pairs)

    * –

IT domain: WMT16 IT translation task, batch
3131313http://data.statmt.org/wmt16/it-translation-task/wmt16-it-task-
references.tgz (1,000 sentence pairs)

  * •

En-Fr:

    * –

News domain: WMT14 news translation
task141414http://data.statmt.org/wmt20/translation-task/dev.tgz (3,003
sentence pairs)

    * –

Medical domain: WMT14 medical translation task, khresmoi
summary151515http://www.statmt.org/wmt14/medical-task/khresmoi-summary-test-
set.tgz (1,000 sentence pairs)

    * –

Reddit domain: MTNT test sets,161616http://www.cs.cmu.edu/ pmichel1/mtnt/ one
for each translation direction (for En$\rightarrow$Fr: 1,020 sentence pairs,
for Fr$\rightarrow$En: 1,022 sentence pairs)

  * •

En-Ja:

    * –

News domain: ALT test set171717https://www2.nict.go.jp/astrec-
att/member/mutiyama/ALT/ (1,018 sentence pairs)

    * –

Reddit domain: MTNT test sets,181818http://www.cs.cmu.edu/ pmichel1/mtnt/ one
for each translation direction (for En$\rightarrow$Ja: 1,002 sentence pairs,
for Ja$\rightarrow$En: 1,001 sentence pairs)

    * –

Twitter natural disaster domain: Tweets test set compiled and translated by
ourselves (not publicly available) (1,400 sentence pairs)

####  4.1.4 English Monolingual Data

English monolingual data are used as a source for back/forward translation and
for fine-tuning GPT-2. There is one dataset for each domain:

  * •

News domain: News Crawl 2019191919http://data.statmt.org/news-
crawl/en/news.2019.en.shuffled.deduped.gz (1M lines for backward/forward
translation, 50k lines for GPT-2 fine-tuning)

  * •

IT domain: English side of the training parallel data provided for the WMT16
IT translation task, batch 1 and batch 2202020http://ufallab.ms.mff.cuni.cz/
popel/batch1and2.zip, (2k lines for backward/forward translation and GPT-2
fine-tuning)

  * •

Medical domain: English side of the En-Fr EMEA parallel
data212121http://opus.lingfil.uu.se/download.php?f=EMEA/en-fr.txt.zip provided
for the WMT14 medical translation task (100k lines for backward/forward
translation, 50k lines for GPT-2 fine-tuning)

  * •

Reddit domain: English data crawled with the Reddit API (1M lines for
backward/forward translation, 50k lines for GPT-2 fine-tuning)

  * •

Twitter natural disaster domain: English tweets crawled with the Twitter API
with the same keywords used to crawled the English tweets of the test set (not
publicly released) (148k lines for backward/forward translation, 50k lines for
GPT-2 fine-tuning)

###  4.2 Framework and Settings

We exploited GPT-2 through the gpt-2-simple
framework.222222https://github.com/minimaxir/gpt-2-simple We did not perform
any pre-processing on the monolingual data used for fine-tuning GPT-2. For
NMT, we tokenized and truecased all the data in English, French, and German,
with the Moses toolkit (Koehn et al., 2007).232323https://github.com/moses-
smt/mosesdecoder The truecaser has been trained on 1M lines randomly sampled
from the News Crawl corpora in each language.

For NMT, training data, validation data, and source side test set are all
segmented into subword units. We used byte-pair encoding (BPE) (Sennrich et
al., 2016b)242424https://github.com/rsennrich/subword-nmt for English, German,
and French, separately trained on 10M lines from the News Crawl 2019
corpora252525http://data.statmt.org/news-crawl/ for each language to learn 32k
BPE operations. For Japanese, we used SentencePiece (Kudo and Richardson,
2018)262626https://github.com/google/sentencepiece to learn 16k sentence
pieces also from the News Crawl 2019 corpus.

We used Marian (Junczys-Dowmunt et al., 2018)272727https://marian-
nmt.github.io/, version v1.7.6 1d4ba73 2019-05-11 17:16:31 +0100 for NMT with
standard hyper-parameters for training (see Table 1).

\--type transformer --max-length 120 --mini-batch-fit --valid-freq 5000
--save-freq 5000 --workspace 10000 --disp-freq 500 --beam-size 12
--normalize=1 --valid-mini-batch 16 --overwrite --early-stopping 5 --cost-
type=ce-mean-words --valid-metrics ce-mean-words bleu --keep-best --enc-depth
6 --dec-depth 6 --transformer-dropout 0.1 --learn-rate 0.0003 --lr-warmup
16000 --lr-decay-inv-sqrt 16000 --lr-report --label-smoothing 0.1 --devices 0
1 2 3 4 5 6 7 --optimizer-params 0.9 0.98 1e-09 --clip-norm 5 --sync-sgd
--exponential-smoothing
---
Table 1:  Hyper-parameters of Marian used for training our NMT systems.

We performed decoding with a beam size of 12 and a length normalization at
1.0.

For evaluation, we used SacreBLEU (Post,
2018)282828https://github.com/mjpost/sacrebleu and report on BLEU scores
Papineni et al. (2002) for English, French, and German, and chrF scores
Popović (2015) for Japanese. Before evaluation, we post-processed the NMT
output by undoing BPE and SentencePiece subword segmentations. Then, except
for Japanese, we detokenized and detruecased the output with Moses.

###  4.3 Results with Back-translation

The performance of our NMT systems trained on several different sets of back-
translations is shown in Table 2.

First, we assessed to what extent the human-made in-domain monolingual data
used for fine-tuning GPT-2 are useful for back-translation. As we can see,
despite the small size of the data, it improves BLEU compared to the baseline
systems for all configurations. When using all the human-made in-domain
monolingual data, or up to 1M sentences, BLEU improvements are even larger for
almost all configurations (except for Ja$\rightarrow$En, Reddit). This result
confirms the usefulness of exploiting more in-domain monolingual data through
back-translation when available.

Using 1M sentences generated by a GPT-2 model that is not fine-tuned leads to
lower BLEU scores than using all the human-made in-domain monolingual data
(except for Ja$\rightarrow$En, Reddit).

The two last rows give the results of our approach: they use human-made in-
domain monolingual data only up to 50k sentences for fine-tuning the GPT-2
model, but millions of synthetic monolingual data. They show that the back-
translations of the monolingual data generated by the fine-tuned GPT-2 model
are useful. We obtained better, or comparable, BLEU scores when using the
back-translations of our synthetic monolingual data to train NMT systems than
when using the back-translations of human-made monolingual data. Using more
synthetic monolingual data (last row) also tends to lead to better BLEU scores
(except for Ja$\rightarrow$En, Reddit and Twitter).

System | Back-translated | De$\rightarrow$En | Fr$\rightarrow$En | Ja$\rightarrow$En
---|---|---|---|---
Data | News | Medical | IT | News | Medical | Reddit | News | Reddit | Twitter
Baseline | none | 32.9 | 36.4 | 42.0 | 36.6 | 48.4 | 34.5 | 14.5 | 7.8 | 5.5
\+ H-TBT | fine-tuning | 34.2 | 40.2 | 42.7 | 37.1 | 48.9 | 34.7 | 17.2 | 8.3 | 16.6
\+ H-TBT | all | 35.8 | 40.7 | 43.4 | 37.4 | 49.6 | 35.9 | 22.1 | 8.0 | 17.1
\+ GPT_notft-TBT | 1M sentences | 34.6 | 37.3 | 41.9 | 37.1 | 48.5 | 34.7 | 20.0 | 8.6 | 9.8
\+ GPT-TBT | 1M sentences | 35.5 | 42.6 | 42.6 | 37.4 | 49.3 | 35.7 | 20.9 | 9.3 | 17.7
\+ GPT-TBT | 10M sentences | 35.5 | 42.9 | 44.6 | 37.8 | 50.3 | 36.9 | 22.3 | 8.7 | 15.9
Table 2: BLEU scores of our NMT systems translating into English, for each
domain. “H-TBT” denotes systems trained on the back-translated human-made
monolingual data (the data used for fine-tuning GPT or all the monolingual
data described in Section 4.1.4). “GPT-TBT” denotes systems trained on the
back-translation of either 1M or 10M monolingual sentences generated by a
GPT-2 model fine-tuned on the in-domain monolingual data. “GPT_notft-TBT”
denotes a configuration in which GPT-2 has not been fine-tuned.

###  4.4 Results with Forward Translation

We performed similar experiments as in Section 4.3 but with forward
translation instead of back-translation. Our results are shown in Table 3.

We did not observe consistent improvements of BLEU and chrF scores when
exploiting human-made monolingual data (H-FT configurations). Increasing the
amount of monolingual data can also decrease or increase BLEU and chrF scores.
Our approach (GPT-FT) leads to better, or similar, scores than the H-FT
configurations that use all the human-made monolingual data.

We conclude that forward translations perform reasonably well, but not
consistently, in these configurations and that GPT-2 models in other languages
than English would be necessary to properly evaluate to which extent our
approach can improve BLEU and chrF scores when English is not the target
language.

System | Translated | En$\rightarrow$De (BLEU) | En$\rightarrow$Fr (BLEU) | En$\rightarrow$Ja (chrF)
---|---|---|---|---
Data | News | Medical | IT | News | Medical | Reddit | News | Reddit | Twitter
Baseline | none | 27.3 | 28.8 | 37.4 | 36.3 | 40.9 | 25.5 | 0.2436 | 0.1419 | 0.0987
\+ H-FT | fine-tuning | 27.9 | 29.6 | 38.6 | 36.5 | 40.9 | 23.3 | 0.2643 | 0.1400 | 0.0839
\+ H-FT | all | 27.9 | 29.7 | 38.6 | 36.2 | 41.6 | 23.4 | 0.2847 | 0.1348 | 0.0845
\+ GPT_notft-FT | 1M sentences | 27.4 | 28.7 | 36.7 | 36.0 | 40.5 | 22.5 | 0.2479 | 0.1301 | 0.0799
\+ GPT-FT | 1M sentences | 27.9 | 29.6 | 39.1 | 36.2 | 42.0 | 23.1 | 0.2513 | 0.1324 | 0.0832
\+ GPT-FT | 10M sentences | 28.0 | 30.1 | 38.9 | 36.3 | 42.3 | 23.3 | 0.2749 | 0.1321 | 0.0810
Table 3: BLEU and chrF scores of our NMT systems translating from English, for
each domain. “H-FT” denotes systems trained on forward-translated human-made
monolingual data (the data used for fine-tuning GPT or all the monolingual
data described in Section 4.1.4). “GPT-FT” denotes systems trained on the
forward-translation of either 1M or 10M monolingual sentences generated by a
GPT-2 model fine-tuned on the in-domain monolingual data. “GPT_notft-FT”
denotes a configuration in which GPT-2 has not been fine-tuned.

##  5 Extreme Adaptation for Personalized NMT

The objective of extreme adaptation for personalized NMT is to adapt a given
NMT system so that it can better translate texts written or spoken by a
specific person. Ideally, for such a task, we would require an amount as large
as possible of parallel data in which one side are texts written or spoken by
the target person in order to personalize our NMT system. Obviously, such a
large data does not exist and would be too costly to create. Thus, we propose
to synthesize such a data with our approach. The main difference with the
domain adaptation scenarios presented in Section 4 is that we cannot even
expect to obtain thousands of sentences of texts written by the target person
to fine-tune GPT-2.

For our extremely personalized NMT experiments, we used the Speaker Annotated
TED Talks (SATED) corpus (Michel and Neubig,
2018a)292929http://www.cs.cmu.edu/ pmichel1/sated/ available for:

  * •

English–German (En-De): 156k sentence pairs, 1,670 speakers

  * •

English–Spanish (En-Es): 183k sentence pairs, 1,922 speakers

  * •

English–French (En-Fr): 178k sentence pairs, 1,887 speakers

Each sentence pair is provided with a tag that identifies the speaker. Note
that this corpus is already pre-processed: tokenized and lower-cased.
Validation data and test data contain two sentence pairs per speaker. In order
to generate synthetic monolingual data for each specific speaker, we exploit
the speaker tag by concatenating it to the English side of the parallel data
and then use the resulting data to fine-tune the GPT-2 model. Through fine-
tuning, we assume that GPT-2 learns the characteristics of each individual
speaker by relying on the speaker tag. At decoding time, we then expect GPT-2
to generate texts for a particular speaker when prompting it with its speaker
tag.

System | De$\rightarrow$En | Es$\rightarrow$En | Fr$\rightarrow$En
---|---|---|---
Baseline | 24.6 | 32.2 | 29.5
Speaker Tags | 24.7 | 32.2 | 29.9
Speaker Tags + GPT-TBT | 27.6 | 34.6 | 32.2
Speaker Tags + GPT_speaker-TBT | 28.5 | 35.6 | 32.4
Table 4: BLEU scores of our NMT systems for the SATED translation task.
“Speaker Tags” denotes the use of the speaker tags to tag each sentence pair
in the given parallel data and each source sentence in the test data. “GPT-
TBT” denotes systems trained on back-translation of 500k sentences generated
by a GPT-2 model fine-tuned on the English side of the SATED parallel data.
‘GPT_speaker-TBT” is similar to ‘GPT-TBT,” except (1) that the data used for
fine-tuning the GPT-2 model are also tagged with the speaker tag and (2) that
each sentence generated by the fine-tuned GPT-2 model is tagged with a speaker
tag.

The results of our experiments are presented in Table 4. In addition to the a
vanilla baseline NMT system, we used one of the adaptation approach (second
row) that uses the parallel data with the speaker tag concatenated to the
source sentence to train NMT systems (Michel and Neubig, 2018a). BLEU scores
with this approach are close to the scores of the baseline system. Then, we
tried our approach similarly to our experiments in Section 4.3 (third row). We
fine-tuned GPT-2 on the English side of the SATED parallel data and generated
500k sentences with the fine-tuned model. Then, we back-translated the
generated data with En$\rightarrow\ast$ baseline systems to obtain synthetic
parallel data for the three language pairs and concatenated it to the speaker
tagged SATED parallel data exploited in our experiments of the second row. The
NMT systems trained on the resulting data improve the BLEU scores by several
BLEU points for all the translation directions. In the last row, we finally
report on the results exploiting the speaker tags also when fine-tuning the
GPT-2 model. We generated 500k sentences, randomly prompting GPT-2 with one of
the speaker tags, and exploited the resulting speaker-tagged monolingual data
as for the other models. BLEU scores are further improved, implying that GPT-2
successfully exploits the speaker tag to generate better synthetic data for
each speaker.

##  6 Conclusion and Future Work

In this preliminary work, we showed that our approach can leverage small in-
domain monolingual data produced by human to generate a large synthetic in-
domain parallel data. Even though the synthetic parallel data are entirely
synthetic, as opposed to a standard backward/forward translation, we obtained
improvements in BLEU scores in all our configurations when using the generated
data to train NMT systems. We also reported on successful experiments in
extreme adaptation for personalized NMT.

In our future work, we would like to perform an in-depth analysis to better
understand our results. We will also conduct more experiments exploiting in-
house GPT models for other languages.

## References

  * Bogoychev and Sennrich (2019) Nikolay Bogoychev and Rico Sennrich. 2019.  Domain, translationese and noise in synthetic data for neural machine translation.  _arXiv preprint arXiv:1911.03362_.
  * Caswell et al. (2019) Isaac Caswell, Ciprian Chelba, and David Grangier. 2019.  Tagged back-translation.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_ , pages 53–63, Florence, Italy. Association for Computational Linguistics.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * Germann (2001) Ulrich Germann. 2001.  Building a statistical machine translation system from scratch: How much bang for the buck can we expect?  In _Proceedings of the ACL Workshop on Data-Driven Methods in Machine Translation_.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_ , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Marie et al. (2020) Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020.  Tagged back-translation revisited: Why does it really work?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5990–5997, Online. Association for Computational Linguistics.
  * Michel and Neubig (2018a) Paul Michel and Graham Neubig. 2018a.  Extreme adaptation for personalized neural machine translation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 312–318, Melbourne, Australia. Association for Computational Linguistics.
  * Michel and Neubig (2018b) Paul Michel and Graham Neubig. 2018b.  MTNT: A testbed for machine translation of noisy text.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 543–553, Brussels, Belgium. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, USA. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram f-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Radford et al. (2019) A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language models are unsupervised multitask learners.
  * Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc.