# Continual Learning of Multi-modal Dynamics with External Memory

Abdullah Akgül<EMAIL_ADDRESS>Gozde Unal Melih Kandemir Istanbul Technical
University, Istanbul, Turkey  University of Southern Denmark, Odense, Denmark

###### Abstract

We study the problem of fitting a model to a dynamical environment when new
modes of behavior emerge sequentially. The learning model is aware when a new
mode appears, but it does not have access to the true modes of individual
training sequences. The state-of-the-art continual learning approaches cannot
handle this setup, because parameter transfer suffers from catastrophic
interference and episodic memory design requires the knowledge of the ground-
truth modes of sequences. We devise a novel continual learning method that
overcomes both limitations by maintaining a descriptor of the mode of an
encountered sequence in a neural episodic memory. We employ a Dirichlet
Process prior on the attention weights of the memory to foster efficient
storage of the mode descriptors. Our method performs continual learning by
transferring knowledge across tasks by retrieving the descriptors of similar
modes of past tasks to the mode of a current sequence and feeding this
descriptor into its transition kernel as control input. We observe the
continual learning performance of our method to compare favorably to the
mainstream parameter transfer approach.

###### keywords:

Continual Learning, Multi-modality, External Memory, Dynamics Modeling.

††journal: Pattern Recognition Letters

##  1 Introduction

Continual Learning (CL) aims to develop a versatile model that is capable of
solving multiple prediction tasks which are presented to the model one task at
a time. The model is then expected to learn the latest task as accurately as
possible while preserving its excellence at the previous ones. Performance
drop caused by the newly learned task is called catastrophic forgetting. CL is
essential for the development of intelligent agents that can adapt to new
environmental conditions not encountered during training. For instance, an
autonomous driving controller may improve its policy based on new experience
collected during its customer-side life time. There exist a solid body of work
that adopt parameter transfer across tasks as the key element of task
memorization Kirkpatrick et al. (2017); Nguyen et al. (2018); Singh et al.
(2019); Zenke et al. (2017). There also appear preliminary studies on building
attentive memories to capture tasks Garnelo et al. (2018); Fraccaro et al.
(2018). Principles that yield memory mechanisms optimal for CL are yet to be
discovered. There has been prior work that focuses on CL for Recurrent Neural
Networks (RNN) Cossu et al. (2021) but either classification or instance
forecasting for time-series. Complementarily, we study CL in the context of
dynamical system identification. Probabilistic State-Space Models (SSM) are
the gold standard methodology for the inference of complex latent dynamics.
SSMs are widely applicable to forecasting impactful quantities such as
weather, currency exchange, equity prices, and sales trends. SSM research
gains significance also in robot learning parallel to the growing interest in
model-based reinforcement learning Hafner et al. (2019, 2020).

Our main contributions are two-fold. First is a novel problem setup where
dynamical system modeling tasks emerge sequentially and a probabilistic SSM is
expected to learn them cumulatively. We further assume each task to follow
multi-modal dynamics, where each individual sequence of a task follows one of
the possible modes that describe the task. Successful CL in such a setup
presupposes maximally efficient encoding of tasks into a long-term memory and
their accurate retrieval. Our second main contribution is a novel CL model
tailored for addressing this challenging problem. The state of the art in CL
that uses either parameter transfer or end-to-end differentiable attentive
episodic memory for knowledge transfer across tasks. The multimodality of the
sequences would undermine the parameter transfer approach due to catastrophic
interference (the effect of global parameter updates on unintended regions of
the input space) and it would require the ground-truth modality labels for the
memory-based approaches to be applicable. Our approach sidesteps these
limitations by capturing the characteristics of unknown modes of sequences of
the present task into fixed-sized vectors, called mode descriptors, and
storing these descriptors in an external neural episodic memory addressable
via a learnable attention mechanism. Our approach represents multiple task
modes by feeding these mode descriptors into the state transition kernel as an
additional input. We place a Dirichlet Process (DP) prior on the attention
weights of the memory to encourage the explanation of the data with minimum
number of modes. Figure 1 illustrates our model with external memory and the
problem with two tasks and four modes. Our resulting Bayesian model can be
efficiently trained using a straightforward adaptation of existing variational
SSM inference techniques.

We evaluate our model in two time series prediction data sets and three
synthetic data sets generated from challenging nonlinear multi-modal dynamical
systems. The performance of our method improves consistently over the
established parameter transfer approach, verifying the importance of
parsimonious use of neural episodic memory in efficient within-task knowledge
acquisition and effective cross-tasks knowledge transfer.

Fig. 1: We study continual learning of dynamical systems, each operate on a
non-overlapping set of multiple modes unknown to the learner at each task. We
find out that inferring mode descriptors from an observed sequence snipped
$(y_{1:C})$, storing them in an external neural episodic memory, retrieving
them in the subsequent tasks, and feeding them into the state transition
kernel as control input brings improved continual learning performance. We
encourage efficient use of memory space by placing a Dirichlet Process prior
on the effective count of encountered modes.

##  2 Continual Multi-Modal Dynamics Learning

We assume a learning agent that observes a dynamical environment via sequences
$y_{1:T}=\\{y_{1},\ldots,y_{T}\\}$ consisting of time-indexed measurements
$y_{t}\in\mathcal{Y}$ living in a measurable space $\mathcal{Y}$. We denote a
snippet of a sequence $y_{t:t^{\prime}}=\\{y_{t},\ldots,y_{t^{\prime}}\\}$ for
an arbitrary time interval $[t,t^{\prime}]$. Modes refer to general
distinguishable properties of a dynamic system such as states with different
characteristics of sine wave (amplitude, frequency) or semantically different
dynamics such as different character trajectories. We refer all these factors
as modes. We define the mode of a sequence $y_{1:T}$ as a fixed-sized vector
and $m$ as an element of a $K-$dimensional embedding space $\mathcal{X}$. A
dynamical system can potentially operate within a large number of modes.
Making an analogy to the real world, an autonomous vehicle accounts for
different environmental characteristics when planning and control for
different weather conditions, countries, and times of a day. Only few of the
modes are active for a particular time point and each mode instantly activates
and deactivates for limited time periods.

We search for a learning algorithm that enables the agent to fit to a
dynamical environment which has perpetually changing global characteristics.
_We cast the corresponding learning problem as CL of multi-modal dynamical
systems, where each task is defined as a group of modes that modulate a
specific dynamical system._ In formal terms, a task $\mathcal{T}_{i}$ is
defined as a group of mode descriptors sampled from a mode generating oracle
$P(\mathcal{X})$, that is

$\displaystyle\mathcal{T}_{i}$ $\displaystyle=\\{m_{r}^{i}|m_{r}\sim
P(\mathcal{X}),r=1,\cdots,R\\},\quad i=1,\cdots U,$ (1)

where $R$ is number of modes per task and $U$ is number of tasks. We assume
that each sequence of a task follows dynamics modulated by a mode sampled from
$\mathcal{T}_{i}$

$\displaystyle\mathcal{D}_{\mathcal{T}_{i}}=\Big{\\{}y_{1:T}^{n}\Big{|}m_{n}$
$\displaystyle\sim P(\mathcal{T}_{i}),y_{1:T}^{n}\sim
p(y_{1:T}|m_{n}),n=1,\cdots,N\Big{\\}},$ (2)

where $P(\mathcal{T}_{i})$ is a probability mass function defined on the modes
of task $\mathcal{T}_{i}$ and $p(y_{1:T}|m_{r})$ is the probability measure
that describes the true behavior of the environment dynamics under mode
$m_{r}$ within a time interval $[1,T]$. The marginal distribution of a task
with respect to its modes is given as

$\displaystyle
p(y_{1:T}|\mathcal{T}_{i})=\sum_{r=1}^{R}p(y_{1:T}|m_{r})P(m_{r}|\mathcal{T}_{i}).$
(3)

The agent observes a task via a data set $\mathcal{D}_{\mathcal{T}_{i}}$ that
contains only the sequences $y_{1:T}^{n}$ but not their corresponding modes.

We are interested in learning a model that minimizes the true CL risk
functional below

$\displaystyle\tiny{R_{\mathbb{L}}^{CL}(h_{\theta})=\lim_{U\rightarrow+\infty}\sum_{i=1}^{U}\mathbb{E}_{\mathcal{T}_{i}\sim
P(\mathcal{X})}\Bigg{[}\mathbb{E}_{y_{1:T}\sim
p(y_{1:T}|\mathcal{T}_{i})}\Big{[}\mathbb{L}(h_{\theta}(\cdot|y_{1:C}),y_{C+1:T})\Big{]}\Bigg{]}},$
(4)

which amounts to the limit of the cumulative risk of individual tasks as they
appear one at a time. Above,
$h_{\theta}:\mathcal{Y}^{C}\rightarrow\mathcal{Y}^{T-C}$ is a stochastic
process that can map an observed sequence of an arbitrary length $C$ to the
subsequent ${T-C}$ time steps. We call the first $C$ observations in the
sequence $y_{1:C}$ as the context and define $\mathbb{L}$ as a sequence-
specific loss function defined on the future values of the sequence
$y_{C+1:T}$. We evaluate the performance of predictions $\widehat{y}$ via two
scores: Normalized Mean Squared Error (NMSE) as a measure of the prediction
accuracy of $h_{\theta}$ when used as a Gibbs predictor, and Negative Log-
likelihood (NLL) as a measure of Bayesian model fit that quantifies the
model’s own assessment on the uncertainty of its assumptions.

We approximate the true CL risk by its empirical counterpart

$\displaystyle\widehat{\mathbb{R}}_{\mathbb{L}}^{CL}=\frac{1}{N}\sum_{i=1}^{U}\sum_{n=1}^{N}\mathbb{L}\Big{(}h_{\theta}(y_{C+1:T}^{n}|y_{1:C}^{n})\Big{|}\mathcal{T}_{i}\Big{)}$
(5)

for a finite number $U$ of tasks presented to $h_{\theta}$ one at a time.

##  3 Novel Baseline: Variational Continual Learning for Bayesian State-Space
Models

We build our target model on a Bayesian treatment of state-space modeling,
which is proven to be effective in learning under high uncertainty and
knowledge transfer across tasks, as practiced in the seminal prior art of CL
Kirkpatrick et al. (2017); Nguyen et al. (2018). We perform approximate
inference using variational Bayes due to its multiple successful applications
to state-space models Frigola et al. (2014); Doerr et al. (2018); Ialongo et
al. (2019) and its favorable computational properties. As there does not exist
any prior work tailored specifically towards CL for dynamical systems, we
curate our own baseline. We adopt the established practice of setting the
posterior of the learned parameters of the previous task as the prior of the
next one and determine Variational Continual Learning (VCL) Nguyen et al.
(2018) as the state-of-the-art representative of this approach. Elastic Weight
Consolidation (EWC) Kirkpatrick et al. (2017) also follows the same approach
but uses a simpler posterior inference scheme.

### Bayesian State Space Models (BSSM):

are characterized by the data generating process below

$\displaystyle\tiny{\theta\sim p(\theta),~{}x_{0}\sim
p(x_{0}),~{}x_{t}|x_{t-1},\theta\sim
p(x_{t}|x_{t-1},\theta),~{}y_{t}|x_{t}\sim p(y_{t}|x_{t}),}$ (6)

where $x_{t}$ and $y_{t}$ correspond to the latent and observed state
variables for time step $t$, respectively. The system dynamics are modeled by
the first-order Markovian transition kernel $p(x_{t}|x_{t-1},\theta)$
parameterized by $\theta$ that in turn follows a prior distribution
$p(\theta)$. The latent states are mapped to the observation space via a
probabilistic observation model $p(y_{t}|x_{t})$. We formulate the initial
latent state $x_{0}$ as another random variable that follows the prior
distribution $p(x_{0})$.

### Variational Inference

is required to approximate the posterior $p(x_{0:T},\theta|y_{1:T})$, which
will be intractable for many choices of distribution families for the data
generating process in Eq. 6. Following Yildiz et al. (2019), we choose the
variational distribution to be mean-field across the parameters of the
dynamics and the latent states

$\displaystyle q_{\xi,\psi}(x_{0:T},\theta|y_{1:T})$
$\displaystyle=q_{\xi}(x_{0}|y_{1:C})\prod_{t=1}^{T}p(x_{t}|x_{t-1},\theta)q_{\psi}(\theta),$
(7)

where $q_{\xi,\psi}$ is an approximation to the true posterior
$p(x_{0:T},\theta|y_{1:T})$ with variational free parameters $(\xi,\psi)$.
This formulation has multiple advantages. Firstly, modeling the marginal
posterior on the initial latent state $x_{0}$ by amortizing on the context
observations $y_{1:C}$ makes the Evidence Lower BOund (ELBO) calculation
invariant to the context length. Secondly, adopting the prior transition
kernel avoids duplicate learning of environment dynamics with twice as many
free parameters and prevents training from instabilities caused by the
inconsistencies between prior and posterior dynamics. Applying Jensen’s
inequality in a conventional way, the corresponding ELBO will be

$\displaystyle\log
p(y_{1:T})\geq\mathbb{E}_{p(x_{1:T}|\theta,x_{0})q_{\xi}(x_{0}|y_{1:C})q_{\psi}(\theta)}[\log
p(y_{1:T}|x_{1:T})]$ (8) $\displaystyle\qquad\qquad\quad-
KL(q_{\xi}(x_{0}|y_{1:C})||p(x_{0}))-KL(q_{\psi}(\theta)||p(\theta)),$

where $KL(\cdot||\cdot)$ stands for the Kullback-Leibler (KL) divergence
between the two distributions on its arguments, and
$X=\\{x_{0},\cdots,x_{T}\\}$.

### VCL for BSSMs

can be implemented as follows. Having fitted the ELBO (Eq. 3) ($\mathcal{L}$)
on the data set for the first task which has observed sequences
$\mathcal{D}_{\mathcal{T}_{1}}$, we attain
$(\xi_{1}^{*},\psi_{1}^{*})=\mathrm{argmax}_{\xi,\psi}\mathcal{L}(\xi,\psi,\mathcal{D}_{\mathcal{T}_{1}})$.
When the next task arrives with data $\mathcal{D}_{\mathcal{T}_{2}}$, we
assign $p(\theta)\leftarrow q_{\psi_{1}^{*}}(\theta)$ and maximize the ELBO
again
$(\xi_{2}^{*},\psi_{2}^{*})=\mathrm{argmax}_{\xi,\psi}\mathcal{L}(\xi,\psi,\mathcal{D}_{\mathcal{T}_{2}})$
. We repeat this process continually for every new coming task. We refer to
this newly curated baseline in the rest of the paper as VCL-BSSM. We neglect
the coreset extension of VCL since its application to BSSMs is tedious and its
advantage is not demonstrated with sufficient significance in static
prediction tasks studied in original work.

##  4 Target Model: The Continual Dynamic Dirichlet Process

The commonplace Bayesian approach to CL transfers knowledge across tasks by
assigning the learned posterior on the parameters of the previous task as the
prior on the parameters of the current task. This is an effective approach
when the subject of transfer is a feed-forward model, such as a classifier in
supervised learning setup Nguyen et al. (2018) or a policy network in
reinforcement learning Kirkpatrick et al. (2017). We conjecture that the
existing parameter transfer Ansatz would not be sufficient for the transfer of
more complex task properties such as modes of dynamical systems. We address
this problem by tailoring a novel CL approach from an original combination of
an aged statistical machine learning tool, the DP, with modern neural episodic
memory and attention mechanisms.

### Dirichlet Processes

are stochastic processes defined on a countably infinite number of categorical
outcomes, every finite subset of which follows a multinomial distribution
drawn from a Dirichlet prior Teh et al. (2006). A DP follows a Griffiths-
Engen-McCloskey (GEM) distribution Pitman et al. (2002)

$\displaystyle\tiny{\pi^{\prime}_{r}|\alpha_{0}\sim\mathrm{Beta}(1,\alpha_{0}),~{}\pi_{r}=\pi^{\prime}_{r}\prod_{j=1}^{r-1}(1-\pi^{\prime}_{j}),~{}\pi=[\pi_{1},\ldots,\pi_{R}]},$
(9)

which we denote in short hand as $\pi\sim\mathrm{GEM}(\alpha_{0},R)$. The GEM
distribution can also be viewed as a stick-breaking process Sethuraman (1994);
Fox et al. (2011) where $\alpha_{0}>0$ is a scalar hyper-parameter. The data
generation process below is called a DP for a base measure
$G_{0}(\mathcal{X})$ defined on a $\sigma-$algebra $\mathbb{B}$ of
$\mathcal{X}$

$\displaystyle m_{r}\sim
G_{0}(\mathcal{X}),~{}\pi\sim\mathrm{GEM}(\alpha_{0},R),~{}G|\pi=\sum_{r=1}^{R}\pi_{r}\delta_{m_{r}},$
(10)

where $\delta_{x}(A)$ is Dirac delta measure that takes value $1$ if $x\in A$
and $0$ otherwise for any measurable set $A\in\mathbb{B}$ and $G$ is a
categorical distribution with parameters $\pi$.

### Neural Episodic Memory.

We assume that the environment dynamics can be expressed within a measurable
latent embedding space $x_{t}\in\mathcal{X}$, and a sequence encoder
$e_{\lambda}(x_{t:t^{\prime}})$ parameterized by $\lambda$ for $t\leq
t^{\prime}$ that maps a sequence of latent embeddings into a fixed dimensional
vector, as well as a neural memory $M=\\{m_{1},\cdots,m_{R}\\}$ consisting of
$R$ elements $m_{r}\in\mathcal{X}$ that live in the same space as latent
embeddings $x_{t}$. We can construct a probability measure for $\mathbb{B}$
from the memory $M$ by updating

$\displaystyle
m_{r}\leftarrow(1-w_{r}(y_{{1:C}},m_{r}))m_{r}+w_{r}(y_{{1:C}},m_{r})e_{\lambda}(y_{{1:C}})$
(11)

for each sequence where $w_{r}(y_{{1:C}},m_{r})=\frac{e^{\langle
m_{r},e_{\lambda}(y_{{1:C}})\rangle}}{\sum_{j=1}^{R}e^{\langle
m_{j},e_{\lambda}(y_{{1:C}})\rangle}}$ for some similarity function
$\langle\cdot,\cdot\rangle:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}^{+}$.
This construction imposes a memory attention mechanism, where the encoded mode
descriptors attend to the memory elements. Here we make the fair assumption
that the modality of a sequence can be identified also from the observation
space, while we need to infer the latent representations accurately to model
the mode dynamics in detail. We choose an uninformative base measure that
assigns equal prior probabilities to memory elements
$G_{0}(M)=\sum_{k=1}^{R}\frac{1}{R}\delta_{m_{r}}$.

### The full model.

We complement the BSSM in Eq. 6 with an external neural episodic memory $M$
that is updated for each observed sequence with the rule in Eq. 11. We place a
DP prior on the retrieval of mode descriptors $m_{r}\in M$ to encourage the
model to generate a minimum number of modes. We also feed the retrieved mode
descriptor into the transition kernel $p(x_{t}|x_{t-1},m,\theta)$ as control
input. The resultant model, which we call as the Continual Dynamic Dirichlet
Process (CDDP), follows the generative process

$\displaystyle\tiny{\pi\sim
GEM(\alpha_{0},R),~{}m|\pi\sim\sum_{r=1}^{R}\pi_{r}\delta_{m_{r}},~{}\theta\sim
p(\theta),~{}x_{0}\sim p(x_{0}),}$ (12) $\displaystyle
x_{t}|x_{t-1},m,\theta\sim p(x_{t}|x_{t-1},m,\theta),\qquad y_{t}|x_{t}\sim
p(y_{t}|x_{t}),$ (13)

where the memory capacity $R$ is set to a bigger number than the expected
upper limit of the mode count.

### Inference.

Since $p(x_{0:T},\theta|y_{1:T})$ is intractable, we approximate it by
variational inference. We inherit the advantages of the BSSM inference scheme
by choosing the variational distribution as

$\displaystyle q_{\xi,\psi}(x_{0:T},\theta,\pi|y_{1:T})=$ (14)
$\displaystyle~{}q_{\xi}(x_{0}|y_{1:C},\pi)\prod_{t=1}^{T}\sum_{r=1}^{R}q_{\xi}(\pi=r|y_{1:C})p(x_{t}|x_{t-1},m_{r},\theta)q_{\psi}(\theta),$

where

$\displaystyle q_{\psi}(\theta)=\mathcal{N}(\theta|\mu,\Sigma),$ (15)
$\displaystyle
q_{\xi}(\pi|y_{1:C})=Cat(w_{1}(m_{1},y_{1:C}),\ldots,w_{K}(m_{R},y_{1:C})),$
(16)
$\displaystyle\tiny{q_{\xi}(x_{0}|y_{1:C},\pi)=\sum_{r=1}^{R}\pi_{r}\mathcal{N}\Bigg{(}x_{0}\Big{|}v_{1}(m_{r},e_{\lambda}(y_{1:C})),v_{2}(m_{r},e_{\lambda}(y_{1:C}))\Bigg{)}}$
(17)

with $x_{0}\sim\mathcal{N}(0,I)$. In the expressions above, $v_{1}$ and
$v_{2}$ refer to dense layers. The corresponding ELBO is then calculated with

$\displaystyle\log p(y_{1:T})\geq\sum_{r=1}^{R}q_{\xi}(\pi=r|y_{1:C})$ (18)
$\displaystyle~{}\mathbb{E}_{p(x_{1:T}|\theta,x_{0},m_{r})q_{\xi}(x_{0}|y_{1:C})q_{\psi}(\theta)}[\log
p(y_{1:T}|x_{1:T})]-KL(q_{\xi}(x_{0}|y_{1:C})||p(x_{0}))$
$\displaystyle~{}-KL(q_{\psi}(\theta)||p(\theta))-\sum_{r=1}^{R}w_{r}(m_{r},y_{1:C})\log(w_{r}(m_{r},y_{1:C})/\pi_{r}).$

### Prediction.

Having trained the model on the latest task $\mathcal{T}_{i}$, its posterior
predictive distribution for new sequence $Y^{*}$ and corresponding latent
embeddings $X=\\{x_{0},\cdots,x_{T}\\}$ reads

$\displaystyle
p(Y_{C+1:T}^{*}|\mathcal{D}_{\mathcal{T}_{i}},Y_{1:C}^{*})=\sum_{r=1}^{R}q(\pi=r|Y_{1:C}^{*})$
(19)
$\displaystyle~{}~{}\int_{X,\theta}q_{\xi}(x_{0}|Y_{1:C}^{*},\pi=r)q_{\psi}(\theta)\Bigg{[}\prod_{t=C+1}^{T}p(y_{t}^{*}|x_{t})p(x_{t}|x_{t-1},\theta,m_{r})\Bigg{]}.$

##  5 Related Work

### State Space Models.

There exists a considerable body of work on Bayesian versions of SSMs that
employ Gaussian Process as transition kernels Ialongo et al. (2019) and
perform variational inference. Another vein of work named Recurrent SSMs
Hafner et al. (2019, 2020) models the transition dynamics as RNN that map a
state to the next time step deterministically while admitting a random state
variable from the previous time step as input and feeding its output to the
distribution of this variable at the subsequent time step.

### Attention and Memory in Neural Nets.

Neural Turing Machines Graves et al. (2014) are the first examples of
attention-based neural episodic memory use with external updates. Attentive
Neural Process Kim et al. (2019) employs an attention network to build a
neural stochastic process that is consistent over observed predictions.
Evidential Turing Process Kandemir et al. (2022) maintains an external memory
that learns to feed a Dirichlet prior on class distribution with informative
concentration parameters inferred during minibatch training updates.

### Continual Learning

is an instance of meta learning Finn et al. (2017) where new tasks are
introduced one at a time and a base model is expected to learn the newest task
without forgetting the previous ones. Early approaches to CL such as EWC
Kirkpatrick et al. (2017) transfer knowledge by the transfer of either
deterministic parameters or their inferred distribution using Fisher
information. VCL Nguyen et al. (2018) improves on EWC with a more
comprehensive inference scheme and an additional feedback channel via a core
set of most characteristic observations of past tasks. Generalized VCL Loo et
al. (2021) maximizes the same ELBO as VCL but using $\beta$-VAE to prevent the
training instability caused by the dominance of the KL-divergence term. We do
not use $\beta$-VAE since our setup does not have this problem.

Knoblauch et al. (2020) explains the superior performance of memorization-
based CL algorithms based on experience replay, core set, and episodic memory
Shin et al. (2017); Lopez-Paz and Ranzato (2017); Lüders et al. (2016) over
regularization-based algorithms Kirkpatrick et al. (2017). For the first time,
our CDDP studies probabilistic multi-modal dynamics in a CL setting by
knowledge transfer via learned mode descriptors maintained in external memory.

### Continual Learning with Episodic Memory.

Rios and Itti (2019) builds a memory by maximum sample diversity in order to
reduce catastrophic forgetting by remembering samples of previous tasks. In
Guo et al. (2020), a memory keeps sample random examples from previous tasks
and these samples are used in the training of the new tasks. There are other
works Lopez-Paz and Ranzato (2017); Chaudhry et al. (2019) that use memory for
CL however, all of them build the memory with the help of data points provided
with the ground truth but in our setup mode labels are not provided. Hence our
work is not comparable to theirs. As neither of them studies dynamics
modeling, their adaption to our setup is not straightforward.

##  6 Experiments

We evaluate the performance of CDDP rigorously on five challenging
applications of CL to time series forecasting. We provide a PyTorch
implementation of our full experiment pipeline that contains studied models as
supplement to our submission, data generation process of the used synthetic
environments, as well as the performance evaluation procedures.

### Baseline Selection.

As we are the first to investigate CL for BSSM inference, there is no prior
work we can take as baseline without significant adaptation. We determine
knowledge transfer across tasks by setting the parameter posterior of the
previous task as the prior of the next as the state-of-the-art approach in CL.
We adapt VCL, the most established variant of this approach, to BSSMs in
Section 3 as baseline that can maximally challenge our CDDP. Baseline methods
must be generative due to the nature of the dynamic modeling problem.
Furthermore, as stated in the problem setup (Section 2), the correct mode
labels are not provided therefore baseline methods need to be unsupervised
generative. No method meets these requirements but we adapt the VCL method.

### Data Sets

### Synthetic Data Sets.

We evaluate our CDDP on three nonlinear dynamical systems, where forecasting
is challenging, while ground-truth task similarity is controllable. Too much
task similarity would make CL unnecessary, while too much task difference
would make it infeasible. When tasks share a reasonable degree of similarity,
a successful CL algorithm is expected to capture, encode, and memorize these
similarities and discard their differences. We perturb all three dynamical
systems with Gaussian white noise.

We generate modes that share similar dynamical properties, as their time
evolution follows the same set of differential equations. However, modes
differ from each other in the choice of the free parameters that govern the
dynamical system. i)Sine Waves data set consists of signals grouped into modes
described by different magnitudes and frequencies. Sine waves are created from
the function $A\mathrm{sin}(2\pi ft)$ where $A$ is the magnitude, $f$ is the
frequency, and $t$ is time. We generate different modes from five different
choices for magnitudes and three frequency levels. ii) Lotka-Volterra
equations stem from modeling the dynamics of predator and prey populations and
are defined as the nonlinear differential equations: $dx_{t}/dt=\alpha
x_{t}-\beta x_{t}y_{t},dy_{t}/dt=\delta x_{t}y_{t}-\gamma y_{t}$ . We generate
different modes by setting the free parameter pairs $(\alpha,\delta)$ and
$(\beta,\gamma)$ to different values. iii) Lorenz Attractor  is a three-
dimensional nonlinear dynamical system developed originally to model
atmospheric convection. As the Lorenz attractor is chaos-theoretic, hence not
solvable with predictable consistency using the existing numerical methods, it
is often used as a challenging environment to evaluate probabilistic dynamical
models Haußmann et al. (2021); Satorras et al. (2019). The Lorenz system is
described by
$dx_{t}/dt=\sigma\left(y_{t}-x_{t}\right),dy_{t}/dt=x_{t}\left(\rho-
z_{t}\right)-y_{t},dz_{t}/dt=x_{t}y_{t}-\beta z_{t}$. We determine our modes
by settings the free parameters $\sigma,\rho$, and $\beta$ of the Lorenz
system to different values.

### Real-World Data Sets.

We evaluate our CDDP in two real-world time series classification data sets
from Dua and Graff (2017). i) Libras Movement Data Set is the official
Brazilian sign language and acronym of the Portuguese name "LÍngua BRAsileira
de Sinais". The data set consists of sequences of $(x,y)$ coordinates of hand
movements from $15$ different classes. ii) Character Trajectories data set
consists of sequences of velocities of $(x,y)$ coordinates and pen force
collected from hand-writings of $20$ English alphabet characters, which can be
written by a single pen-down segment. Treating each class as a mode, we
generate multimodal time series forecasting tasks from these two data sets
that satisfy the learning setup in Eq. 4.

### Hyper-parameter Selection

Unlike many commonplace continual learning benchmark setups that do few-way
low-dimensional image classification, BSSM inference of even a single
dynamical system necessitates careful tuning of hyper-parameters. Thus, we set
latent state dimensionality, learning rate, and the hidden layer sizes of the
encoder, transition kernel, and decoder networks to values that maximize the
performance of the BSSM on the first task. Notably, the selection favors VCL
and CDDP equally. Each experiment repetition presents the tasks to the CL
models at random order.

### Hyper-parameters on Data

Each synthetic data set has 12 train sequences and 6 test sequences per mode.
There are 15 modes 5 tasks in Sine Waves, 8 modes 4 tasks in Lotka-Volterra
and 12 modes 4 tasks in Lorenz Attractor. We set the sequence lengths and time
steps ($T,\Delta t$) to $(15,0.1)$ for Sine Waves, $(25,0.4)$ for Lotka-
Volterra , and $(50,0.01)$ for the Lorenz Attractor. For each mode in the
whole data set, we generate a long sequence and divide the whole sequence into
subsequences of length $T$ with a certain space between them. For all
synthetic data sets, fixed jump space between subsequences is $5$ time-point.
By adding additional Gaussian noise to the train split, we construct train and
test splits with these subsequences. Sine Waves) We generate different modes
from five different choices for $A\in[3,6,9,12,15]$, and three frequency
levels for $f\in\big{[}\frac{2}{3},1,\frac{4}{3}\big{]}$. We add Gaussian
noise from $\mathcal{N}(0,\sigma)$ and set $\sigma$ to $\frac{A}{100}$ for
each mode to the train split. Lotka-Volterra) We generate different modes by
setting the free parameter pairs $(\alpha,\delta)$ and $(\beta,\gamma)$ to
different values. Starting with a fixed point $(x_{0},y_{0})=(2,2)$, we select
levels as $\alpha,\beta,\gamma\in[0.25,0.75]$, and a constant $\delta=0.5$. We
add a Gaussian noise from $\mathcal{N}(0,0.001)$ to the train split. Lorenz
Attractor) We determine our modes by settings the free parameters
$\sigma,\rho$, and $\beta$ of the Lorenz system to different values. We start
with initial state $(x_{0},y_{0},z_{0})=(1,1,28)$ and generate a long
sequence. We generate modes with different but partially overlapping dynamical
characteristics by setting $\rho$ to three and $\sigma$ and $\beta$ to two
different values. We select the three free parameters levels as
$\rho\in[28,42,56],\sigma\in[8,12]$, and
$\beta\in\big{[}\frac{5}{3},\frac{13}{3}\big{]}$. We add Gaussian noise from
$\mathcal{N}(0,0.01)$ to the train split. Libras) The data set has 12 train
sequences and 12 test sequences per mode. There are 15 modes 5 tasks. The
sequence lengths and time step ($T,\Delta t$) are ($45$, $\frac{7}{45}$).
There are no fixed $\Delta$ for this data set but to find it we divide the
approximate seconds taken for a sample to the number of frame count. Character
Trajectories) The data set has different number of instances per modes
therefore, we divide each instances samples to two halves then first half is
placed in the train split and rest of them are placed in the test split.
Hence, train split has 1422 samples, and test split with 1436 samples In the
data set, each character is created with a different sequence length, to adapt
the data set to our pipeline we subsample the sequences with the smallest
sequence length which is $109$. The sequence lengths and time step ($T,\Delta
t$) are ($109$, $0.05$). There are 20 modes (characters) and 5 tasks.

### Context Length.

We select context length empirically as one-third of the sequence length
amounting to five for Sine Waves, eight for Lotka-Volterra, 16 for the Lorenz
Attractor, $15$ for Libras, and $35$ for Character Trajectories. The reason
for that is, generally, one-third of the sequence captures the general
characteristic of the dynamics and gives indications about modes.

### Hyper-parameters on Architectures.

Our CDDP has four main architectural elements: Encoder: The sequence encoder
$e_{\lambda}(x_{t:t^{\prime}})$ governs the mean of our normal distributed
recognition model $q_{\psi}(x_{0}|y_{1:C},\pi)$. We feed $C$ observations as a
stacked set of values into the encoder. The sequence encoder is a single dense
layer for Sine Waves, Lotka-Volterra, Libras; and a multi-layer perceptron for
the Lorenz Attractor, and Character Trajectories with two hidden layers of
size 90. The perceptron uses the $tanh(\cdot)$ activation function followed by
layer normalization. Decoder: The likelihood function $p(y_{t}|x_{t})$ of the
base model serves as a probabilistic decoder that maps the latent state
$x_{t}$ to the observed state $y_{t}$. We choose the emission distribution to
be normal with mean governed by a single dense layer for Sine Waves, Lotka-
Volterra, Libras; and a multilayer perceptron with two hidden layers of size
$90$ for the Lorenz Attractor, and Character Trajectories. The perceptron uses
the $tanh(\cdot)$ activation function followed by layer normalization.
Transition Kernel: We choose the transition kernel $p(x_{t}|x_{t-1},m,\theta)$
of CDDP to be a normal distribution with mean governed by a plain RNN that
receives a concatenation of the previous hidden state and the mode descriptor
as input. All covariates are set to a fixed variance of $0.1$. The RNN on the
mean is a multilayer perceptron with one hidden layer of size 40 for Sine
Waves, Lotka-Volterra, and Libras; and 90 for the Lorenz Attractor, Character
Trajectories. The perceptron uses the $tanh(\cdot)$ activation function
followed by layer normalization. The transition kernel of the base model of
VCL $p(x_{t}|x_{t-1},\theta)$ follows the same RNN architecture except that
its input does not contain a mode descriptor. External Memory: We set the
memory size to $20$ for the Sine Wave environment, $10$ for Lotka-Volterra,
$15$ for the Lorenz Attractor, $20$ for Libras, and $30$ for Character
Trajectories.

Fig. 2:  Predictions of CDDP (top row) and VCL-BSSM (middle row) on sample
sequences from four tasks of the Character Trajectories data set after
continual learning is over. Bottom row shows the corresponding ground-truth
trajectories. The first one-third of each trajectory is considered as context
(plotted in red), and the rest is predicted by the models (plotted in blue).
Our CDDP remembers the first task much more precisely than VCL-BSSM (first and
second columns) after adapting itself accurately to the last task (third and
fourth columns). This is achieved by virtue of its external episodic memory.

### Hyper-parameters on Experimental Pipeline

We select Adam as optimizer with learning rates of $0.005$ for Sine Wawes,
$0.001$ for Lotka-Volterra, $0.0005$ for Lorenz Attractor, $0.001$ for Libras,
and $0.001$ for Character Trajectories. For all data sets, the batch size is
equal to 9 except for Character Trajectories which is equal to $64$ due to
high number of instances. We train the models $300$ for Sine Waves, $750$ for
Lotka-Volterra, $500$ for Lorenz Attractor, $300$ for Libras, and $2000$ for
Character Trajectories epochs per task. We performed our experiments on Intel
Xeon CPU E5-2650 and Intel Core(TM) i7-6700K CPU.

Table 1: The summary of our main results given as the Area Under Curve (AUC)
plotting the change of two performance metrics NMSE and NLL as a function of
the number of learned tasks. The reported numbers are mean $\pm$ standard
error over $10$ repetitions. The detailed results are provided in Table 2. Our
CDDP outperforms VCL-BSSM in nearly all cases indicating the benefit of
maintaining an external memory of mode descriptors in continual learning, the
central hypothesis of this work.

| AUC NMSE | AUC NLL
---|---|---
Data Set | VCL-BSSM | CDDP | VCL-BSSM | CDDP
Sine Waves | $1.00\pm 0.04$ | $\bf 0.91\pm 0.03$ | $3.57\pm 0.09$ | $\bf 3.50\pm 0.09$
Lotka-Volterra | $\bf 0.58\pm 0.04$ | $0.60\pm 0.06$ | $1.50\pm 0.05$ | $\bf 1.32\pm 0.08$
Lorenz Attractor | $0.26\pm 0.00$ | $\bf 0.24\pm 0.01$ | $4.42\pm 0.04$ | $\bf 4.35\pm 0.06$
Libras | $\bf 0.14\pm 0.00$ | $\bf 0.14\pm 0.00$ | $-0.37\pm 0.02$ | $\bf-0.39\pm 0.04$
Character Trajectories | $0.87\pm 0.04$ | $\bf 0.64\pm 0.01$ | $0.14\pm 0.02$ | $\bf-0.19\pm 0.03$

Table 2: Quantitative results of models on five data sets. The reported
numbers are mean $\pm$ standard error over $10$ repetitions. The reported
scores are taken after training a task and evaluated along with previously
seen tasks.

Data Set | Model | Score | 1 Task | 2 Tasks | 3 Tasks | 4 Tasks | 5 Tasks
---|---|---|---|---|---|---|---
Sine Waves | VCL-BSSM | NMSE | $\bf 0.13\pm 0.02$ | $0.97\pm 0.14$ | $1.22\pm 0.15$ | $1.27\pm 0.09$ | $1.39\pm 0.08$
NLL | $\bf 1.76\pm 0.22$ | $3.75\pm 0.24$ | $3.84\pm 0.13$ | $4.20\pm 0.15$ | $4.29\pm 0.10$
CDDP | NMSE | $0.16\pm 0.02$ | $\bf 0.88\pm 0.11$ | $\bf 1.07\pm 0.15$ | $\bf 1.12\pm 0.14$ | $\bf 1.31\pm 0.11$
NLL | $2.09\pm 0.27$ | $\bf 3.43\pm 0.22$ | $\bf 3.77\pm 0.17$ | $\bf 4.05\pm 0.14$ | $\bf 4.15\pm 0.10$
Lotka-Volterra | VCL-BSSM | NMSE | $0.17\pm 0.03$ | $\bf 0.71\pm 0.08$ | $\bf 0.92\pm 0.15$ | $\bf 0.53\pm 0.03$ |
NLL | $0.56\pm 0.20$ | $1.97\pm 0.28$ | $\bf 2.02\pm 0.19$ | $\bf 1.46\pm 0.12$ |
CDDP | NMSE | $\bf 0.13\pm 0.03$ | $0.72\pm 0.20$ | $0.96\pm 0.18$ | $0.60\pm 0.05$ |
NLL | $\bf 0.32\pm 0.18$ | $\bf 1.35\pm 0.21$ | $2.05\pm 0.15$ | $1.54\pm 0.21$ |
Lorenz Attractor | VCL-BSSM | NMSE | $0.22\pm 0.02$ | $0.25\pm 0.03$ | $0.27\pm 0.02$ | $0.28\pm 0.01$ |
NLL | $4.22\pm 0.16$ | $4.41\pm 0.08$ | $4.55\pm 0.09$ | $4.52\pm 0.04$ |
CDDP | NMSE | $\bf 0.21\pm 0.02$ | $\bf 0.24\pm 0.02$ | $\bf 0.25\pm 0.02$ | $\bf 0.26\pm 0.02$ |
NLL | $\bf 4.06\pm 0.19$ | $\bf 4.33\pm 0.02$ | $\bf 4.53\pm 0.06$ | $\bf 4.46\pm 0.03$ |
Libras | VCL-BSSM | NMSE | $\bf 0.13\pm 0.01$ | $0.14\pm 0.00$ | $\bf 0.13\pm 0.01$ | $0.15\pm 0.00$ | $\bf 0.14\pm 0.00$
NLL | $-0.41\pm 0.04$ | $-0.35\pm 0.03$ | $\bf-0.42\pm 0.04$ | $-0.32\pm 0.03$ | $-0.36\pm 0.03$
CDDP | NMSE | $0.14\pm 0.00$ | $\bf 0.13\pm 0.01$ | $0.14\pm 0.01$ | $\bf 0.14\pm 0.01$ | $\bf 0.14\pm 0.01$
NLL | $\bf-0.42\pm 0.03$ | $\bf-0.40\pm 0.04$ | $-0.38\pm 0.05$ | $\bf-0.40\pm 0.03$ | $\bf-0.37\pm 0.04$
Character Trajectories | VCL-BSSM | NMSE | $\bf 0.42\pm 0.04$ | $0.80\pm 0.04$ | $0.90\pm 0.02$ | $1.11\pm 0.08$ | $1.11\pm 0.10$
NLL | $\bf-0.23\pm 0.06$ | $0.06\pm 0.05$ | $0.20\pm 0.05$ | $0.31\pm 0.08$ | $0.35\pm 0.09$
CDDP | NMSE | $0.52\pm 0.05$ | $\bf 0.56\pm 0.02$ | $\bf 0.69\pm 0.04$ | $\bf 0.71\pm 0.03$ | $\bf 0.71\pm 0.02$
NLL | $-0.05\pm 0.05$ | $\bf-0.26\pm 0.04$ | $\bf-0.17\pm 0.03$ | $\bf-0.23\pm 0.04$ | $\bf-0.26\pm 0.04$

Table 3: Ablation study results given as the Area Under Curve (AUC) on the
Sine Waves data set. The reported numbers are mean $\pm$ standard error over
$5$ repetitions.

|  | Parameter
---
Transfer
| Probabilistic
---
Parameters
| Memory
---
Content
NMSE | NLL
RNN | $\checkmark$ | $\times$ | N/A | $1.03\pm 0.05$ | $3.50\pm 0.10$
VCL-BSSM | ✓ | ✓ | N/A | $0.93\pm 0.03$ | $3.59\pm 0.16$
CDDP Variants | $\times$ | ✓ | Zeros | $0.94\pm 0.04$ | $3.56\pm 0.12$
$\times$ | ✓ | Ones | $1.12\pm 0.13$ | $3.75\pm 0.14$
$\times$ | ✓ | Twos | $1.35\pm 0.23$ | $3.80\pm 0.18$
✓ | ✓ | Learned | $0.91\pm 0.04$ | $3.56\pm 0.12$
CDDP Target | $\times$ | ✓ | Learned | ${\bf 0.87\pm 0.03}$ | ${\bf 3.35\pm 0.15}$

### Main Results.

Table 1 summarizes model performance throughout the whole continual x process
as the area under learning curve. Our CDDP outperforms the parameter transfer
based VCL-BSSM baseline consistently in nearly all cases. Storing mode
descriptors of the learned dynamics in an external memory, retrieving them in
the subsequent tasks, and feeding them into the state transition kernel
prevents catastrophic forgetting more effectively than plain parameter
transfer. As seen in Figure 2, in the challenging character trajectories
dataset, the prediction accuracy of our CDDP significantly outperforms VCL-
BSSM in both early and late stages of the CL period.

### Ablation Study.

We investigate the contribution of individual design choices to the total
performance of our target model. We study the effect of three design choices:
i) knowledge transfer via parameters $\theta$ of the learned transition
dynamics, ii) quantifying the uncertainty of the parameters of transition
dynamics by a distribution $q_{\psi}(\theta)$, and iii) maintaining an
external memory with learned or unlearned content. Table 3 shows a map of
model variants corresponding to the activation status of these three design
choices, as well as the corresponding numerical results on the Sine Waves data
set over five repetitions. We observe a performance increase when knowledge
transfer is done via the external memory instead of —but not together with —
parameter transfer, supporting the central assumptions of our target model.
Setting the memory content to unlearned values causes rapid performance
deterioration as values diverge from the learned values. This outcome
demonstrates the essential role of the external memory in the CL performance
of CDDP.

##  7 Conclusion

### Summary.

We report the first study on CL of multi-modal dynamical systems. We curate a
competitive baseline for this new problem setup from an adaptation of VCL to
BSSMs. We introduce a novel alternative to the parameter transfer approach of
VCL for within-task knowledge acquisition and cross-tasks knowledge transfer
using an original combination of neural episodic memory, DPs, and BSSMs. We
observe in CL of challenging multi-modal dynamics modeling environments that
our alternative approach compares favorably to the established parameter
transfer approach.

### Broad Impact.

Our work can be used for varying applications such as: i) weather forecasting,
where features can be transferred from one climate to another, ii) autonomous
driving, where driving patterns can be adapted across different countries, and
iii) model-based reinforcement learning algorithms, when the environment
changes due either to the actions of the ego agent or to external factors. The
memory architecture of CDDP may be improved by alternative embedding, update,
and attention mechanisms. Our formulation of the transition rule is agnostic
to the architecture that governs the transition kernel.

## References

  * Chaudhry et al. (2019) Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M., 2019\.  Efficient lifelong learning with A-GEM, in: ICLR.
  * Cossu et al. (2021) Cossu, A., Carta, A., Lomonaco, V., Bacciu, D., 2021\.  Continual learning for recurrent neural networks: An empirical evaluation.  Neural Networks 143, 607–627.
  * Doerr et al. (2018) Doerr, A., Daniel, C., Schiegg, M., Duy, N., Schaal, S., Toussaint, M., Sebastian, T., 2018.  Probabilistic recurrent state-space models, in: ICML.
  * Dua and Graff (2017) Dua, D., Graff, C., 2017.  UCI machine learning repository.  URL: http://archive.ics.uci.edu/ml.
  * Finn et al. (2017) Finn, C., Abbeel, P., Levine, S., 2017.  Model-agnostic meta-learning for fast adaptation of deep networks, in: ICML.
  * Fox et al. (2011) Fox, E., Sudderth, E., Jordan, M., Willsky, A., 2011\.  A sticky HDP-HMM with application to speaker diarization.  The Annals of Applied Statistics , 1020–1056.
  * Fraccaro et al. (2018) Fraccaro, M., Rezende, D., Zwols, Z., Pritzel, A., Eslami, S., Viola, F., 2018\.  Generative temporal models with spatial memory for partially observed environments, in: ICML.
  * Frigola et al. (2014) Frigola, R., Chen, Y., Rasmussen, C.E., 2014.  Variational Gaussian process state-space models, in: NeurIPS.
  * Garnelo et al. (2018) Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D., Eslami, S., Teh, Y., 2018.  Neural processes.  arXiv preprint arXiv:1807.01622 .
  * Graves et al. (2014) Graves, A., Wayne, G., Danihelka, I., 2014.  Neural Turing Machines.  arXiv preprint arXiv:1410.5401 .
  * Guo et al. (2020) Guo, Y., Liu, M., Yang, T., Rosing, T., 2020.  Improved schemes for episodic memory-based lifelong learning.  NeurIPS .
  * Hafner et al. (2020) Hafner, D., Lillicrap, T., Ba, J., Norouzi, M., 2020\.  Dream to control: Learning behaviors by latent imagination, in: ICLR.
  * Hafner et al. (2019) Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., Davidson, J., 2019.  Learning latent dynamics for planning from pixels, in: ICML.
  * Haußmann et al. (2021) Haußmann, M., Gerwinn, S., Look, A., Rakitsch, B., Kandemir, M., 2021.  Learning partially known stochastic dynamics with empirical PAC Bayes, in: AISTATS.
  * Ialongo et al. (2019) Ialongo, A., van der Wilk, M., Hensman, J., Rasmussen, C., 2019\.  Overcoming mean-field approximations in recurrent Gaussian process models, in: ICML.
  * Kandemir et al. (2022) Kandemir, M., Akgül, A., Haussmann, M., Unal, G., 2022\.  Evidential Turing processes, in: ICLR.
  * Kim et al. (2019) Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., Teh, Y., 2019\.  Attentive neural processes, in: ICLR.
  * Kirkpatrick et al. (2017) Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R., 2017\.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences 114\.
  * Knoblauch et al. (2020) Knoblauch, J., Husain, H., Diethe, T., 2020.  Optimal continual learning has perfect memory and is np-hard, in: ICML.
  * Loo et al. (2021) Loo, N., Swaroop, S., Turner, R., 2021.  Generalized variational continual learning, in: ICLR.
  * Lopez-Paz and Ranzato (2017) Lopez-Paz, D., Ranzato, M., 2017\.  Gradient episodic memory for continual learning, in: NeurIPS.
  * Lüders et al. (2016) Lüders, B., Schläger, M., Risi, S., 2016.  Continual learning through evolvable neural turing machines .
  * Nguyen et al. (2018) Nguyen, C., Li, Y., Bui, T., Turner, R., 2018.  Variational continual learning, in: ICLR.
  * Pitman et al. (2002) Pitman, J., et al., 2002.  Combinatorial stochastic processes.  Technical Report.
  * Rios and Itti (2019) Rios, A., Itti, L., 2019.  Closed-loop memory GAN for continual learning, in: IJCAI.
  * Satorras et al. (2019) Satorras, V., Akata, Z., Welling, M., 2019.  Combining generative and discriminative models for hybrid inference, in: NeurIPS.
  * Sethuraman (1994) Sethuraman, J., 1994.  A constructive definition of Dirichlet priors.  Statistica Sinica 4, 639–650.
  * Shin et al. (2017) Shin, H., Lee, J., Kim, J., Kim, J., 2017.  Continual learning with deep generative replay, in: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (Eds.), NeurIPS.
  * Singh et al. (2019) Singh, G., Yoon, J., Son, Y., Ahn, S., 2019.  Sequential neural processes, in: NeurIPS.
  * Teh et al. (2006) Teh, Y., Jordan, M., Beal, M., Blei, D., 2006.  Hierarchical Dirichlet processes.  Journal of the American Statistical Association 101, 1566–1581.
  * Yildiz et al. (2019) Yildiz, C., Heinonen, M., Lähdesmäki, H., 2019.  ODE2VAE: Deep generative second order ODEs with Bayesian neural networks, in: ICML.
  * Zenke et al. (2017) Zenke, F., Poole, B., Ganguli, S., 2017.  Continual learning through synaptic intelligence, in: ICML.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

8th Swiss Text Analytics Conference 2023, Neuchâtel, Switzerland, Jun. 12-14,
2023

<EMAIL_ADDRESS>] [1] [1]

<EMAIL_ADDRESS>] [1]

<EMAIL_ADDRESS>]

[1]Corresponding author. [1]These authors contributed equally.

# Improving Metrics for Speech Translation

Claudio Paonessa  Institute for Data Science, FHNW University of Applied
Klosterzelgstrasse 2, 5210 Windisch, Switzerland  Dominik Frefel  Manfred
Vogel

(2023)

###### Abstract

We introduce Parallel Paraphrasing ($\text{Para}_{\text{both}}$), an
augmentation method for translation metrics making use of automatic
paraphrasing of both the reference and hypothesis. This method counteracts the
typically misleading results of speech translation metrics such as WER, CER,
and BLEU if only a single reference is available. We introduce two new
datasets explicitly created to measure the quality of metrics intended to be
applied to Swiss German speech-to-text systems. Based on these datasets, we
show that we are able to significantly improve the correlation with human
quality perception if our method is applied to commonly used metrics.

###### keywords:

Metrics Swiss German Speech to Text Speech Translation Generated Paraphrases

##  1 Introduction

For most practical use cases of speech-to-text systems, the transcription is
perfect if the sentences are grammatically correct and the semantic meaning is
fully captured. For speech translation, the phrasing of the transcription is
ambiguous, contrary to word-by-word transcriptions for cases where input and
output languages match. This means that a single reference rarely adequately
covers the valid output range. The widely applied metrics BLEU [1], Word Error
Rate (WER), and Character Error Rate (CER) fail to handle the occurring
sentence ambiguity, if calculated based on single references per sample. Many
additional references are needed for each sample to counteract this flaw,
which are often not available due to the high cost and effort of collecting
them. This leads to metric results often not agreeing with the human
perception of the transcription quality. More generally, the ambiguous
translation space and therefore the need for multiple references occurs for
all translation tasks.

Swiss German is a family of dialects and a mostly spoken language. The
language lacks standardized writing, the written form usually only appears in
informal text messages. Therefore, Swiss speech-to-text systems transform
Swiss German speech to Standard German text, i.e., speech translation with a
very similar source and target language. Because Swiss German is different
from Standard German regarding phonetics, vocabulary, morphology, and syntax,
the system output tends not to match with the single reference in a test set.
Table 1 contains examples where common metrics fail to handle the paraphrased
but semantically correct hypothesis.

Table 1: Examples of semantically matching but paraphrased references and hypotheses with corresponding metric values. See Section 4.3 for details on the applied metrics. (1) | Ref: |  Gesucht wurde auch im nahen Ausland.
---|---|---
| Hypo: |  Auch im nahen Ausland wurde gesucht.
|  |  BLEU: 0.562 | WER: 0.667 | CER: 0.771
(2) | Ref: |  Der Spatenstich fand im Oktober letzten Jahres statt.
| Hypo: |  Der Spatenstich fand letztes Jahr im Oktober statt.
|  |  BLEU: 0.271 | WER: 0.500 | CER: 0.404
(3) | Ref: |  Überlegungen die Lage in Zukunft zu verbessern sind in Planung.
| Hypo: |  Gedanken wie man die Lage zukünftig besser machen kann sind in Planung.
|  |  BLEU: 0.159 | WER: 0.700 | CER: 0.532

Thanks to recent advancements in paraphrasing systems based on neural machine
translation [2], the quality of paraphrases significantly increased. State-of-
the-art paraphrasing systems not only cover synonym substitution but more
advanced changes in sentence structures. Existing research improving
translation metrics based on automatic paraphrasing [3] focuses on the
augmentation of references and approaches to increase diversity to maximize
coverage of the translation space.

Building upon the existing idea of augmenting the references for an improved
BLEU metric by automatic paraphrasing, we extend this method to also generate
paraphrases of the hypothesis. Our experiments show that this addition
significantly improves the correlation of common metrics with human
perception.

##  2 Related Work

The use of paraphrasing for evaluating machine translation systems to address
some of the weaknesses of popular metrics has a long history. The metrics
METEOR [4], Meteor Universal [5], and ParaEval [6] support synonym matching,
covering the simplest form of paraphrasing. A more recent approach ParaBLEU[7]
includes a learned neural metric based on paraphrase representation learning,
achieving state-of-the-art performance on the WMT Metrics Shared Task 2017
(WMT17) [8]. Bawden et al., 2020 [3] showed that slight gains to the
correlation with human judgment can be expected with automatic paraphrasing to
generate additional references for BLEU. In a more general analysis from
Freitag et al. 2020 [9] the impact of well-chosen references on correlation
with human judgment for English to German translation is analyzed. The
researchers found that a precisely defined paraphrasing task executed by
professional linguists increases the correlation compared to backtranslation
or other automated methods.

##  3 Approach

Compared to existing approaches that introduce automatic paraphrasing to the
calculation of the BLEU metric [10, 3], our approach is not limited to
paraphrasing for the generation of diverse references but also generates
paraphrases for the system output. Because the paraphrasing models based on
machine translation typically have limited diversity of generated sentence
structures, it is up to chance whether the sentence structure of a correct
system output can be reproduced or not. With our approach of paraphrasing both
the reference and the hypothesis ($\text{Para}_{\text{both}}$) we aim to
increase the chances for an intended match and therefore limit the number of
diverse paraphrases needed.

For the generation of paraphrases in German, we use a paraphrasing algorithm
[2] based on the Prism translation model [11]. This algorithm pushes the
output away from the input in the lexical space by penalizing n-gram overlaps.
The algorithm penalizes n-gram (1-, 2-, 3-, and 4-grams) overlaps by
subtracting values from the output log probabilities of the NMT model before
selecting candidates during beam search. With exponential weighting on the
penalization, the method ensures penalizing larger n-gram overlaps more
harshly than smaller ones. A parameter $\alpha$ controls how much the model
pushes the output away from the input during decoding. We use the paraphrasing
algorithm and the translation model with the parameters from Thompson and Post
2020 [2] to sample the top $n$ backtranslation candidates. This corresponds to
$\alpha=0.003$, $\beta=4$, and a beam width of $n$. Examples of resulting
paraphrases are reported in Appendix A.

With $n$ additional generated paraphrases of the hypothesis and reference, we
calculate $n+1$ metric values per sample for metrics supporting multiple
references (e.g., BLEU). Single reference metrics (e.g., WER, CER) produce
${(n+1)}^{2}$ values. We explore different methods to aggregate the resulting
values.

##  4 Evaluation

We consider two annotated data sources to measure the impact of our method if
applied to established metrics. The two datasets Human Sentence Ratings (GER-
HSR-1K) and Online Transcription Ratings (GER-OTR-691), specifically created
for this work, are described in the following sections.

###  4.1 Human Sentence Ratings

We suggest a novel rating system aiming at capturing the essential components
for comparing reference sentences and corresponding hypotheses for Standard
German. The rating system consists of three binary values and one discrete
rating with values between 0 and 3:

  * •

Hypothesis Grammar (case-insensitive and without punctuation)

  * •

Hypothesis Punctuation

  * •

Hypothesis Capitalization

  * •

Semantic similarity rating

The first three binary values only indicate correctness or an error regarding
grammar, punctuation, and capitalization. The fourth rating corresponds to a
semantic comparison between the reference and the hypothesis as defined in
Table 2.

Table 2: Semantic similarity rating range. Rating |  Description
---|---
3 |  Reference message is completely and unambiguously captured in the hypothesis (even possible if there are grammar or spelling mistakes)
2 |  Virtually matching semantic meaning with only lack or abundance of insignificant details, misspelling of named entities is also considered an insignificant detail
1 |  Majority of reference message is captured and only a small significant part is semantically not matching with the reference
0 |  Majority of reference message is not captured

Based on this approach, we annotated a dataset with 1000 samples, denoted as
GER-HSR-1K. The samples originate from the two datasets Swiss Parliaments
Corpus [12], and SDS-200 [13], containing Swiss German audio with Standard
German transcriptions. The audio samples from these datasets were transcribed
using a speech-to-text system, finetuned for Swiss German and based on the
fairseq S2T [14, 15] model. We replicate the Transformer baseline model
architecture and training procedure from Plüss et al., 2022 [13]. With this
model, we generate realistic samples occurring in Swiss German to German
speech translation. However, the references and corresponding hypotheses were
only considered if they have a Levenshtein text distance greater than zero in
order to exclude identical reference/hypothesis pairs.

In Figure 1, we report the distribution of the semantic similarity ratings.
Because the sentences are sampled from a speech-to-text model achieving state-
of-the-art results for Swiss German, lower ratings are less frequent and
almost half of the samples are rated as perfectly matching the semantic
meaning of the reference.

Our human annotators found a grammar, punctuation, or capitalization error in
only 3.2% of the GER-HSR-1K hypotheses. The models mostly produce
grammatically sound sentences. For the analysis in this work, we only make use
of the semantic similarity rating to evaluate our metric.

The fully annotated dataset is available
online.111https://www.cs.technik.fhnw.ch/i4ds-datasets

Figure 1: The distribution of the human-annotated semantic similarity rating
in the GER-HSR-1K dataset.

###  4.2 Online Transcription Ratings

Through an online demo application showcasing a Swiss German speech-to-text
system, we collected audio samples. After the transcription is shown to the
user, they can voluntarily give feedback in the form of a discrete rating
between 1 and 5 stars, with 1 star being the worst and 5 stars being the best
rating. There is no further instruction given on how to rate the sentences.
This dataset is extended by a human transcription to have a single reference
ground-truth sentence.

Compared to the GER-HSR-1K dataset, the sentences collected in this dataset
are on average shorter and less representative of naturally spoken language.
Due to the lack of detailed instructions, the crowd-sourced ratings tend to be
inconsistent. Additionally, because the ground-truth transcription is loosely
based on the system hypothesis, little paraphrasing occurs.

A filtered version of this dataset with 691 samples is denoted as GER-OTR-691.
This subset only contains pairs of references and hypotheses with a
Levenshtein distance greater than zero.

###  4.3 Evaluation Method

To estimate and compare the quality of different metrics, we calculate
correlations (linear: Pearson’s $r$, monotonic: Kendall’s $\tau$) between
metric results and human-annotated ratings. Specifically, we apply Kendall’s
Tau-like formulation defined in the WMT18 Metrics Shared Task [16]. The
adaptation of the Kendall’s Tau coefficient is defined as follows:

$\tau=\frac{|Concordant|-|Discordant|}{|Concordant|+|Discordant|}\\\ $ (1)

Whether a comparison between human judgment and a metric of a pair of distinct
samples, $s_{1}$ and $s_{2}$, is counted as concordant (Conc) or discordant
(Disc) is defined in the following matrix, where $m(s_{i})$ and $h(s_{i})$ are
the metric value and the human rating of the sample $s_{i}$, respectively:

|  | Metric
---|---|---
|  |  $m(s_{1})<m(s_{2})$ |  $m(s_{1})=m(s_{2})$ |  $m(s_{1})>m(s_{2})$
Human |  $h(s_{1})<h(s_{2})$ | Conc | Disc | Disc
$h(s_{1})=h(s_{2})$ | - | - | -
$h(s_{1})>h(s_{2})$ | Disc | Disc | Conc

This formulation means we exclude all human ties. In the case of non-identical
human judgments, metric ties are always counted as $Discordant$. For this
correlation coefficient to be consistent, we must ensure that human judgment
and the metric have the same orientation, i.e., a higher score indicating
higher transcription quality.

We use the sentence-level BLEU formula from sacreBLEU [17] with exponential
smoothing. For the WER and CER metrics we employ the corresponding accuracy
rate ($1-\textit{Error-rate}$) to properly align the scores for Kendall’s Tau-
like formulation. We normalize the text by removing all punctuation and
transforming all characters to lowercase.

##  5 Results

We analyzed different aggregation methods: maximum, minimum, average, or top-
n-averaging. We found the minimum and the average to decrease the correlation
with human judgment compared to the maximum. In some cases, averaging a subset
of the top values outperformed the maximum, but the results were inconsistent.
Because ties between metric values of two distinct samples are discouraged in
the used Kendall’s Tau-like formulation, the maximum, and an additional
arbitrarily small random number added outperforms all other methods by
decreasing the chances of ties to virtually zero. Statistically, in 50% of the
cases where a tie would occur, we randomly hit the correct ranking. For a fair
comparison, we report results without adding this random value. The added
random number only helps to artificially increase Kendall’s Tau-like values.
It does not improve the metric in any useful way. All the results are reported
with the maximum as the aggregation function.

Table 3: Kendall’s $\tau$ and Pearson’s $r$ correlations on the GER-HSR-1K (HSR) and the GER-OTR-691 (OTR) datasets. $\text{Para}_{\text{ref}}$ ($n$) refers to the multi-reference version with $n$ additional generated references. Metrics denoted as $\text{Para}_{\text{both}}$ ($n$) refer to our suggested extended version with the hypothesis and references augmented with $n$ additional generated paraphrases. Dataset |  Metric | $\tau$ | $r$
---|---|---|---
HSR |  WER | 0.3472 | 0.4704
$\text{Para}_{\text{ref}}$ (11) WER | 0.3725 | 0.4879
$\text{Para}_{\text{both}}$ (6) WER | 0.5115 | 0.6137
CER | 0.2632 | 0.3511
$\text{Para}_{\text{ref}}$ (11) CER | 0.3081 | 0.3982
$\text{Para}_{\text{both}}$ (6) CER | 0.4811 | 0.5513
BLEU | 0.3167 | 0.3903
$\text{Para}_{\text{ref}}$ (11) BLEU | 0.3798 | 0.4438
$\text{Para}_{\text{both}}$ (6) BLEU | 0.4892 | 0.5872
OTR |  WER | 0.5903 | 0.6133
$\text{Para}_{\text{ref}}$ (8) WER | 0.5957 | 0.6191
$\text{Para}_{\text{both}}$ (2) WER | 0.5972 | 0.6238
CER | 0.6283 | 0.6516
$\text{Para}_{\text{ref}}$ (8) CER | 0.6356 | 0.6590
$\text{Para}_{\text{both}}$ (2) CER | 0.6369 | 0.6632
BLEU | 0.5531 | 0.6419
$\text{Para}_{\text{ref}}$ (7) BLEU | 0.5369 | 0.6270
$\text{Para}_{\text{both}}$ (2) BLEU | 0.5578 | 0.6473

The results for the two reference datasets are reported in Table 3. For both
augmentation methods, we report results for the best number of paraphrases
between 1 and 16. We show a significant improvement of the correlations for
the GER-HSR-1K dataset if applying $\text{Para}_{\text{both}}$. With the
augmentation limited to the references ($\text{Para}_{\text{ref}}$), we see a
much lower improvement to the baseline. On the GER-OTR-691 dataset we observe
very limited gains. Because of the lack of paraphrasing occurring in this
dataset, we did not expect a large improvement in the correlations.

In Figure 2, we show the impact of the number of paraphrases on the
correlation with human judgment. Throughout all metrics six seems to be the
best number of paraphrases for our method $\text{Para}_{\text{both}}$. The
parameterization of the underlying automatic paraphrasing model and the
dataset impacts the number of paraphrases. Therefore, the chosen number of
paraphrases should not be interpreted as a generally good pick. Additionally,
we include the result of the WER metric with the augmentation limited to
references ($\text{Para}_{\text{ref}}$). In this case, the strongest
correlation between human ratings and automatic metrics is achieved at 11
paraphrases, but is significantly lower than our parallel approach.

Figure 2: Results of the $\text{Para}_{\text{both}}$ method applied to three
metrics on the GER-HSR-1K dataset with different numbers of paraphrases (for
both the reference and the hypothesis). Additionally, the WER results with
only the references augmented ($\text{Para}_{\text{ref}}$). Zero paraphrases
correspond to no augmentation and the single-reference version of the metric.

In addition to the numeric correlation, we show a visual comparison of human
ratings and metric values in Figure 3. Based on the single reference BLEU
scores, we observe that the metric clearly underestimates a lot of samples
with a high human rating of 2 or 3. The metric distributions of the samples
with these high ratings are nearly indistinguishable. Our approach improves
these distributions and better aligns the metrics to be linearly increasing
with the human ratings. Because our approach combined with maximum aggregation
can only increase the metric value, we overestimate some of the low-rated
samples.

Figure 3: A comparison between the distributions of the single reference BLEU
and the $\text{Para}_{\text{both}}(6)$ BLEU score for the different semantic
similarity ratings in the GER-HSR-1K dataset.

##  6 Conclusion

In this paper, we introduced $\text{Para}_{\text{both}}$, an augmentation
method for translation metrics. We demonstrated a significant increase in
correlation with human judgment for Swiss German to German speech translation.
Based on state-of-the-art automatic paraphrasing, we produce different
paraphrased versions of the reference and the hypothesis. Using our method, we
improve the robustness of existing metrics by addressing paraphrasing that may
arise in translation tasks.

Based on encouraging side experiments conducted on the WMT19 [18] dataset, we
propose further investigation into the overall effectiveness of this approach
in neural machine translation. We observed a relative increase of 52% on
Kendall’s Tau-like score with $\text{Para}_{\text{both}}$.

For future work, we also recommend exploring methods to mitigate the
overestimation of low-rated samples. Our current method does not include any
measures to reduce the risk of overestimation. Additionally, exploring other
paraphrasing methods and their parameter space may also lead to more suitable
paraphrases.

The novel dataset GER-HSR-1K is made publicly available to help advance the
development of more appropriate metrics for speech translation, especially
Swiss German.

##  7 Acknowledgement

This work was supported by HASLER Foundation within the project ”Quality
metrics for Swiss German speech recognition” [22031].

## References

  * Papineni et al. [2002] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu,  Bleu: a method for automatic evaluation of machine translation,  in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002, pp. 311–318. URL: https://aclanthology.org/P02-1040. doi:10.3115/1073083.1073135.
  * Thompson and Post [2020] B. Thompson, M. Post,  Paraphrase generation as zero-shot multilingual translation: Disentangling semantic similarity from lexical and syntactic diversity,  in: Proceedings of the Fifth Conference on Machine Translation (Volume 1: Research Papers), Association for Computational Linguistics, Online, 2020.
  * Bawden et al. [2020] R. Bawden, B. Zhang, L. Yankovskaya, A. Tättar, M. Post,  A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing,  in: Findings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
  * Banerjee and Lavie [2005] S. Banerjee, A. Lavie,  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,  in: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Association for Computational Linguistics, Ann Arbor, Michigan, 2005, pp. 65–72. URL: https://aclanthology.org/W05-0909.
  * Denkowski and Lavie [2014] M. Denkowski, A. Lavie,  Meteor universal: Language specific translation evaluation for any target language,  in: Proceedings of the Ninth Workshop on Statistical Machine Translation, Association for Computational Linguistics, Baltimore, Maryland, USA, 2014, pp. 376–380. URL: https://aclanthology.org/W14-3348. doi:10.3115/v1/W14-3348.
  * Zhou et al. [2006] L. Zhou, C.-Y. Lin, D. S. Munteanu, E. Hovy,  ParaEval: Using paraphrases to evaluate summaries automatically,  in: Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, Association for Computational Linguistics, New York City, USA, 2006, pp. 447–454. URL: https://aclanthology.org/N06-1057.
  * Weston et al. [2021] J. Weston, R. Lenain, U. Meepegama, E. Fristed, Generative pretraining for paraphrase evaluation, 2021\. URL: https://arxiv.org/abs/2107.08251. doi:10.48550/ARXIV.2107.08251.
  * Bojar et al. [2017] O. Bojar, Y. Graham, A. Kamran,  Results of the WMT17 metrics shared task,  in: Proceedings of the Second Conference on Machine Translation, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 489–513. URL: https://aclanthology.org/W17-4755. doi:10.18653/v1/W17-4755.
  * Freitag et al. [2020] M. Freitag, D. Grangier, I. Caswell,  BLEU might be guilty but references are not innocent,  in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 61–71. URL: https://aclanthology.org/2020.emnlp-main.5. doi:10.18653/v1/2020.emnlp-main.5.
  * Kanayama [2003] H. Kanayama,  Paraphrasing rules for automatic evaluation of translation into Japanese,  in: Proceedings of the Second International Workshop on Paraphrasing, Association for Computational Linguistics, Sapporo, Japan, 2003, pp. 88–93. URL: https://aclanthology.org/W03-1612. doi:10.3115/1118984.1118996.
  * Thompson and Post [2020] B. Thompson, M. Post,  Automatic machine translation evaluation in many languages via zero-shot paraphrasing,  in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020.
  * Plüss et al. [2021] M. Plüss, L. Neukom, C. Scheller, M. Vogel,  Swiss parliaments corpus, an automatically aligned swiss german speech to standard german text corpus,  in: F. Benites, D. Tuggener, M. Hürlimann, M. Cieliebak, M. Vogel (Eds.), Proceedings of the Swiss Text Analytics Conference 2021, Winterthur, Switzerland, June 14-16, 2021 (held online due to COVID19 pandemic), volume 2957 of CEUR Workshop Proceedings, CEUR-WS.org, 2021. URL: http://ceur-ws.org/Vol-2957/paper3.pdf.
  * Plüss et al. [2022] M. Plüss, M. Hürlimann, M. Cuny, A. Stöckli, N. Kapotis, J. Hartmann, M. A. Ulasik, C. Scheller, Y. Schraner, A. Jain, J. Deriu, M. Cieliebak, M. Vogel,  SDS-200: A Swiss German speech to standard German text corpus,  in: Proceedings of the Thirteenth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 3250–3256. URL: https://aclanthology.org/2022.lrec-1.347.
  * Ott et al. [2019] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, M. Auli,  fairseq: A fast, extensible toolkit for sequence modeling,  in: Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
  * Wang et al. [2020] C. Wang, Y. Tang, X. Ma, A. Wu, D. Okhonko, J. Pino,  fairseq s2t: Fast speech-to-text modeling with fairseq,  in: Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations, 2020.
  * Ma et al. [2018] Q. Ma, O. Bojar, Y. Graham,  Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance,  in: Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Association for Computational Linguistics, Belgium, Brussels, 2018, pp. 671–688. URL: https://aclanthology.org/W18-6450. doi:10.18653/v1/W18-6450.
  * Post [2018] M. Post,  A call for clarity in reporting BLEU scores,  in: Proceedings of the Third Conference on Machine Translation: Research Papers, Association for Computational Linguistics, Belgium, Brussels, 2018, pp. 186–191. URL: https://www.aclweb.org/anthology/W18-6319.
  * Fonseca et al. [2019] E. Fonseca, L. Yankovskaya, A. F. T. Martins, M. Fishel, C. Federmann,  Findings of the WMT 2019 shared tasks on quality estimation,  in: Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), Association for Computational Linguistics, Florence, Italy, 2019, pp. 1–10. URL: https://aclanthology.org/W19-5401. doi:10.18653/v1/W19-5401.

##  Appendix A Paraphrase Examples

Ref: |  Gesucht wurde auch im nahen Ausland.
---|---
|  Es wurde auch im nahen Ausland gesucht.
|  Es wurde auch im benachbarten Ausland gesucht.
|  Auch im benachbarten Ausland wurde gesucht.
|  Auch im benachbarten Ausland suchte man.
|  Es wurde auch im nahen Ausland verfolgt.
|  Es wurde auch im benachbarten Ausland verfolgt.
Hypo: |  Auch im nahen Ausland wurde gesucht.
|  Ebenfalls im benachbarten Ausland wurde gesucht.
|  Auch im benachbarten Ausland wurde gesucht.
|  Ebenfalls im benachbarten Ausland suchte man nach.
|  Ebenfalls im benachbarten Ausland suchte man.
|  Auch im benachbarten Ausland suchte man.
|  Ebenfalls im nahen Ausland wurde gesucht.

Ref: |  Der Spatenstich fand im Oktober letzten Jahres statt.
---|---
|  Der Spatenstich geschah im Oktober letzten Jahres.
|  Der Spatenstich geschah im Oktober vergangenen Jahres.
|  Der Spatenstich geschah im vergangenen Oktober.
|  Der Spatenstich fand letztes Jahr im Oktober statt.
|  Es geschah im Oktober letzten Jahres.
|  Es geschah im Oktober vergangenen Jahres.
Hypo: |  Der Spatenstich fand letztes Jahr im Oktober statt.
|  Der Spatenstich fand im letzten Jahr im Oktober statt.
|  Der Spatenstich fand im vergangenen Jahr im Oktober statt.
|  Der Spatenstich fand im letzten Jahr im vergangenen Oktober statt.
|  Der Spatenstich fand im Oktober des vergangenen Jahres statt.
|  Der Spatenstich fand im letzten Jahr im Oktober ab.
|  Es fand im letzten Jahr im Oktober statt.

Ref: |  Überlegungen die Lage in Zukunft zu verbessern sind in Planung.
---|---
|  Überlegungen zur Verbesserung der Zukunftssituation sind geplant.
|  Überlegungen zur Verbesserung der Situation in der Zukunft sind geplant.
|  Überlegungen zur Verbesserung der Situation in der Zukunft sind in Planung.
|  Erwägungen zur Besserung der zukünftigen Lage gibt es.
|  Erwägungen zur Besserung der zukünftigen Lage sind geplant.
|  Überlegungen wie zukünftig die Dinge besser gestaltet werden können, sind
in Planung.
Hypo: |  Gedanken wie man die Lage zukünftig besser machen kann sind in Planung.
|  Gedanken darüber, wie zukünftig die Dinge besser gestaltet werden können,
sind in Planung.
|  Gedanken darüber, wie zukünftig die Dinge besser gestaltet werden können,
befinden sich in Planung.
|  Ideen, wie sich die Situation in der Zukunft verbessern ließe, befinden
sich in Planung.
|  Ideen, wie sich die Situation in der Zukunft verbessern ließe, entwickeln
sich.
|  Jetzt geht es darum, darüber nachzudenken, wie man die Lage zukünftig
besser machen kann.
|  Überlegungen zur Verbesserung der Situation in der Zukunft sind im Gange.