# Learning to track environment state via predictive autoencoding

Marian Andrecki
Heriot-Watt and Edinburgh Universities
Edinburgh, United Kingdom
<EMAIL_ADDRESS>
&Nicholas K. Taylor
Heriot-Watt University
Edinburgh, United Kingdom
<EMAIL_ADDRESS>

###### Abstract

This work introduces a neural architecture for learning forward models of
stochastic environments. The task is achieved solely through learning from
temporal unstructured observations in the form of images. Once trained, the
model allows for tracking of the environment state in the presence of noise or
with new percepts arriving intermittently. Additionally, the state estimate
can be propagated in observation-blind mode, thus allowing for long-term
predictions. The network can output both expectation over future observations
and samples from belief distribution. The resulting functionalities are
similar to those of a Particle Filter (PF). The architecture is evaluated in
an environment where we simulate objects moving. As the forward and sensor
models are available, we implement a PF to gauge the quality of the models
learnt from the data.

##  1 INTRODUCTION

Recent years have seen artificial agents become significantly more competent
at learning how to act from unstructured data. The beginning of rapid
improvements can be marked with seminal work by [Mnih et al., 2015] on Atari
game playing using a neural Reinforcement Learning (RL) agent capable of
human-level performance across dozens of games without modification to its
architecture. Perhaps, the most impressive case so far is AlphaZero presented
in [Silver et al., 2017], where a tabula rasa reinforcement learner, after
only couple of hours of self-interaction, surpassed top human players at
Chess, Shougi and Go (all with traditions longer than 1000 years). Those and
many other successes are results of the newly created overlap between the
fields of deep and reinforcement learning.

Unfortunately, the super-human performance in easy-to-simulate environments,
such as Chess or Atari worlds, is slow to propagate to problems embedded in
the real world. One of the key obstacles for agents learning in the real world
is the fact that data gathering happens on timescales familiar to humans. The
agent may need to wait for minutes to learn about consequences of its actions.
Contrast this with the simulated worlds, where data gathering requires
milliseconds or can be parallelised by creating many instances of the agent
and environment (see [Mnih et al., 2016] on asynchronous RL).

Under such circumstances, it is important for physical agents to: (a) extract
more knowledge from every data point (b) deliberate more about how to interact
with the environment (as it is more time-expensive to fail in the real world
than in the imagination)

For both of these, a physical agent pays in the currency of computation (CPU
time) hoping to decrease the number of actions until training completion (to
shorten wall-clock duration of the process).

These behaviours align with the concept of Predictive Learning (PL) [LeCun,
2016]. In this approach, the agent not only strives to select actions that
maximise the expected reward, but also learns about the environment in a
unsupervised manner, for example trains to predict future observations. PL was
inspired by DYNA – an architecture from [Sutton, 1991], where an RL agent
trains also a forward model of the environment and uses it for planning.

This research aims to improve the extraction of information from agents’
experiences to enable planning. This can be furthered by developing new
methods for learning predictive models from environmental observations.

While engineers often have the sense of a given environment themselves, it may
be challenging to transfer the knowledge to the agent. Occasionally, it is
difficult to express one’s understanding algorithmically (e.g. how to
recognise a face?) At other times abstract mathematical models are brittle and
do not mesh well with the noisy and complex environments because of the frame
problem. For some of these troublesome domains it might be possible to find
actionable predictive models via training of deep networks on gathered data.

When available, predictive models can be used for estimation of the
environment state. Tracking (also referred to as filtering) involves two
stages: _prediction_ , which propagates the beliefs forward in time, and
_update_ using sensory data, which merges existing belief with the new
information. This approach allows one to cope with intermittent or noisy
observations. Furthermore, it is possible to quantify uncertainty of the
predictions and use this information for active exploration of the environment
[Hero and Cochran, 2011]. Finally, by keeping a global estimate of the
environment that is only partially observable, one can combine information
arriving from different sensors and locations (information fusion). This field
is described in more depth in _Section 2_.

The problem of automated model-building from unstructured data has been
receiving increasing attention in recent years. In the domain of Atari games,
impressive performance was showcased in works from [Oh et al., 2015] and
[Chiappa et al., 2017], where trained networks reliably predicted game states
for up to hundreds of time steps. Earlier, thus smaller in scale, experiments
can be found in [Srivastava et al., 2015] and [Lotter et al., 2015], where
learning concerned videos of bouncing and rotating objects. Predicting in
highly complex environments (e.g. road traffic) was explored in both [Lotter
et al., 2016] and [Mathieu et al., 2015]. A recent work from [Henaff et al.,
2017] enables sampling future observations in stochastic worlds. The recounted
architectures are presented in more detail in _Section 3_.

In this work, we present a neural architecture for learning predictive models
of environments from high-dimensional unstructured time series data, such as
videos. We show, that a trained network can be used to track the state of a
non-deterministic environment in a manner comparable to Sequential Monte Carlo
methods (also known as Particle Filtering). That is, it continually consumes
observations and provides high-quality predictions of the next observation.
Furthermore, the network can be run in blind mode to generate predictions for
more distant futures. The architecture can cope with observations that are
intermittent or noisy. Details concerning principles of operation and
implementation are expanded upon in _Section 4_.

The environment used in the experiments was a simulation of moving balls. The
observations were $(28\times 28\times 1)$ images. As the exact stochastic
model was available, a Sequential Monte Carlo simulation was implemented to
assess the performance of the network against a near-optimal belief tracking
method. The performance of the architecture is showcased in _Section 5_.

The core novelty of the method is the previously unseen combination of: (a)
learning the forward model from data (b) flexibility in input/output of
observations/predictions (any sequence length, intermittent observations) (c)
ability to predict stochastic environments (d) ability to output both the
expectation over observation and samples from the belief distribution.
_Section 6_ discusses potential uses and plans for further developments.

##  2 STATE ESTIMATION

###  2.1 AGENT AND THE WORLD

Agents can be spawned in various environments with different tasks to
complete. Figure 1 is computational graph representing abstraction of a time-
aware passive observer perceiving its world through an observation channel.
Time flows along the downward direction. Ovals represent variables, arrows
signify computational relations between those (i.e. how to compute one
variable-value from another). Under the description given below, the
environment is modelled as a Markov chain.

Figure 1: A passive observer and its environment represented as a
computational graph. The agent can learn about the environment only through
the sensory channel.

####  2.1.1 Worlds

Left hand side relates to computations in the environment. State of a world
evolves with time according to specific rules. For example, our universe is
governed by particular interactions between particles. Atari and other game
worlds evolve according to specific algorithm implementing them. The world
state $s_{t}$ changes with time according to transition function $T_{W}$:
$s_{t}=T_{W}(s_{t-1})$

The agent, implemented by computations on the right hand side111The split
between agent and world is a convenient abstraction only. In truth, the agent
might be embedded in its world, i.e. its state is contained in world state
(e.g. humans are embedded in their environment, Atari playing algorithms are
not), perceives its world through a sensory channel. Observation $o_{t}$ is a
projection of environment state $s_{t}$ onto a, usually, much lower
dimensional space defined by agents sensors: $o_{t}=P_{S}(s_{t})$

The agent can never observe the underlying state directly, but by combining
many observations across time it is possible to learn about environment’s
tendency to evolve. Agent’s understanding of the environment, can be tested by
examining accuracy of its predictions of future observations.

For example, a trained network from [Chiappa et al., 2017] was capable of
reliably predicting observations tens of time steps away. Note that this does
not mean, that the network reconstructed machine code that implements Atari
emulation or even that it used similar representation as Atari software
developers did. It only implies, that the original environment (Atari console)
and learnt emulator (neural network) are difficult to distinguished when
viewed through a specific sensory channel (game screens output by the two
systems). For a judge with only this channel available, these two are largely
functionally equivalent.

####  2.1.2 Axes of variation

To better understand difficulties for predicting (and acting) in different
worlds let us list some fundamental distinctions. Implications of these for
learning are discussed later, when particular architectures are mentioned.

Observability refers to $P_{S}$ in a given problem. Full-observability means
that the observation contains all of the information about environment state
(e.g. Chess). When the world is partially-observable, it is not possible to
infer its state from a single observation, thus information fusion is
required.

Observation stochasticity describes whether $P_{S}(\cdot)$ is deterministic or
not. Often this captures whether the observations contain noise or can be
missing.

Process stochasticity describes whether $T_{W}(\cdot)$ is deterministic or
not. If it is, then it is possible to simulate the environment over
arbitrarily long periods of time to obtain distant reliable predictions (hence
the success of predictions in Atari environment). If it is not, then it is not
enough to simulate environment. One needs to propagate beliefs forward to
account for different random events. The quality of prediction deteriorates
with time-distance.

Process complexity: it is possible for the underlying environment to be
completely deterministic, yet composed of so many parts or complicated rules
that to the agent it appears stochastic (either because of lacking model or
shortage of computational resources). Systems that are deterministic but
chaotic222a small change in initial conditions leads to large divergence of
outcomes often have to be modelled as stochastic anyway.

####  2.1.3 Agents

What computations implement the agent depends on the task it is fulfilling.
Still, at our level of abstraction, the agent has mental state at time $t$,
call this $bs_{t}$ (for belief state). This state evolves with time, typically
to reflect changes in the environment. This is implemented by belief time
propagation and belief update with new observation (note the recursion):

$bs_{t}=T_{B}(bs_{t-1},o_{t})$ (1)

What information is represented in the belief state depends on the problem and
the environment. To give examples, an algorithm for tracking air planes with
use of radar data will store information about objects of interest and discard
everything else. A reinforcement learning agent may represent only those
aspects that enable better estimation of expected future reward, for example
colours of game objects may be often ignored. For an agent that reliably
receives a rich observation at every time step (e.g. Atari player), there is
little to gain from propagating beliefs forward in time – the percept usual
contains all that is of interest.

Representing the problem as a computational graph enabled us to clearly
observe different challenges that arise during learning. This view translates
well into design of modular neural networks. Lastly, the abstraction relates
only to non-acting agents as those are studied in this work. However, it could
easily be expanded to represent also reinforcement learners by adding reward
signal (coming from the world to agent) and granting agent control over a
fraction of world state (thus affecting how environment changes with time).

###  2.2 PARTICLE FILTERING

One of the solutions to state estimation and future prediction is Particle
Filtering (PF) (also known as sequential Monte Carlo). The approach is now
introduced as it was used as a (strong) benchmark to assess performance of the
proposed neural architecture. For an in-depth presentation, consult [Doucet et
al., 2001].

PF maintains a probabilistic estimate over the state of interest, for example,
2D pose of a robot: $\mathbf{s}=(x,y,\theta)$). The distribution is
represented by a collection of $n$ samples (particles) – instantiations of
$\mathbf{s}$. When the robot needs to make a decision, it can operate on the
knowledge of probabilistic distribution over states.

An implementation of PF requires two core ingredients: (a) forward model of
process – this is $T_{W}$ from the earlier abstraction (b) sensor model –
related to $P_{S}$ in the abstraction.

The propagation of beliefs involves two steps. Prediction: all of the samples
are advanced in time with use of the forward model. This increases the
uncertainty in the estimate as the forward process is stochastic. Update: new
information from the observation is incorporated into belief. This achieved by
importance sampling of the particles using the likelihood function (particles
are replicated in the proportion of their likelihood to be the origin of the
observed measurement).

PF is a flexible method relative to its competitors. It is simple
mathematically, copes well with non-linear models. Particles can represent
arbitrary distributions (e.g. Kalman filter is restricted to Multinomial
Gaussians). In the (unpractical) limit of infinity of particles, PF is an
optimal method for propagation of beliefs (assuming accurate models are
provided).

However, all of this comes at a high computational cost. The higher the number
of samples used, the more times the forward model has to be called per time
step. And, usually, the more complex the problem (i.e. more non-linearities,
higher dimensionality), the more particles are needed.

Still, for the experiments presented in this work, PF is expected to perform
close to optimality. Thus, it marks the top bound for performance on our
problem. It can be often difficult to quantify performance of predictive
neural networks. When a future observation is produced, how does one measure
its accuracy? Mean squared difference between output and measured images is
often uninformative. Sometimes researchers visually judge the outputs and
score them for realism. In this case, we have an opportunity to compare
outputs of a predictive network to visual samples from near-optimal belief
tracker.

It is important to note that the predictive architectures discussed, solve a
more difficult challenge than PF. While sequential Monte Carlo relies on the
forward and sensor model provided by the designer, the networks need to
determine those from percepts. Additionally, the networks cope with
unstructured data.

##  3 PREDICTIVE NEURAL NETS

For problems where forward models are unavailable, deep predictive
architectures are becoming viable solutions. Despite the apparent difficulty,
the field is advancing quickly. Atari worlds are close to solved333see videos
supplied by [Chiappa et al., 2017] at
https://sites.google.com/site/resvideos1729/. However, in many ways, Atari is
the easiest case of the problem. The system is almost fully-observable and
deterministic. Thus, to predict, it suffices to simulate the environment
rather than maintain a distribution over the possible states. This section
discusses neural approaches deployed for predicting in a range of
environments.

###  3.1 TOOLBOX

####  3.1.1 Predictive autoencoders

At the high level many of the successful architectures are a combination of
convolutional, recurrent and deconvolutional layers, this is sometimes termed
as Predictive Autoencoder (PAE). In essence, an input image at time $t$ is
encoded in low-dimensional space via convolution. Then this encoding is
combined with information about previous percepts using recurrent layers.
Finally, deconvolutional layers output the prediction. The usual loss used to
optimise network parameters is a mean squared difference between the actual
observation and its prediction. The idea of an autoencoder relies on routing
the information through a narrow passage in a neural network. At the low
dimensional bottleneck, the input is represented in the encoded form. The
approach can be used for image denoising, sampling from data distribution and
unsupervised pretraining. Read more in [Baldi, 2012].

The combination of new and old information is most frequently performed with
Long-Short Term Memory (LSTM) units [Azzouni and Pujolle, 2017], less so with
Gated Recurrent Units (GRU) [Chung et al., 2014]. In a PAE the representations
formed are optimised for temporal predictions. It is possible to replace
recurrent layers by concatenation multiple observations (to allow for
extraction of time derivative information). However, this approach was unable
to learn long-term time-dependencies in [Oh et al., 2015].

Implementations of variants of this architecture can be found in [Oh et al.,
2015, Chiappa et al., 2017, Henaff et al., 2017] for Atari and games, in
[Srivastava et al., 2015, Lotter et al., 2015, Lotter et al., 2016] for
objects bouncing, rotating and natural videos.

####  3.1.2 Sampling futures

PAEs are deterministic. What happens if the underlying environment is
stochastic? As argued in [Mathieu et al., 2015], the network outputs an
average of possible futures (expectation over observation) and following
statement from [Henaff et al., 2017], it possibly ignores low likelihood modes
of the distribution (mode collapse).

This leads to multiple problems. To implement planning methods such as Markov
Tree search (see [Browne et al., 2012]), it is important to be able to sample
different possible futures, not just the maximum likelihood one. Moreover,
expectations over average future percepts are often blurred and not similar to
real percepts. An average of all videos from YouTube is a grey still image. A
randomly sampled video probably contains a cat and is much more fun.
Additionally, it is likely, that generating realistic samples requires richer
representation of the environment than simply blurring the parts of frame
where uncertainty arises.

[Henaff et al., 2017] proposed Error-Encoding Network (EEN), which predicts
the average future for deterministic input. However, provided with source of
randomness, it outputs different possible observations. Under their training
scheme, the deterministic network is not required to predict every last detail
of an observation correctly. Rather, once a prediction was made, a noise input
to the network is tweaked to improve the match. Currently, our architecture
does not implement this approach, though there appears to be much to gain.

####  3.1.3 Generative adversarial networks

[Goodfellow et al., 2014] introduced a general method for sampling from high-
dimensional complex distributions (e.g. images of human faces) termed
Generative Adversarial Networks (GAN). The concept relies on formulating the
sampling problem as a minimax game between two networks: generator of samples
(G) and discriminator (D), who classifies samples as fake or real. The method
for video prediction from [Mathieu et al., 2015] uses adversarial loss to
generate more realistic video frames. Our approach deploys it in similar way.

##  4 PAEGAN

###  4.1 ARCHITECTURE

####  4.1.1 Overview

We present Predictive Autoencoding Generative Adversarial Network (PAEGAN), a
neural architecture for learning forward models and tracking state using
unstructured observations. When deployed, the network continually consumes
observations $o$. Those can be real percepts from the environment if available
or null observations $o_{\emptyset}$ otherwise. An estimate of the environment
is maintained at all times – this value is referred to as belief state
$bs_{t}$. At any time, it can be used to produce an expectation over possible
observations $\hat{o}_{t}$, a sample observation $o^{\prime}_{t}$ or a sample
state $s^{\prime}_{t}$.

To predict future percepts, the network can be run in blind mode, with
$o_{\emptyset}$ provided for belief propagation. The more distant the
predictions, the more uncertain is the estimate. In observation space this is
indicated by increased image blur. Examples of $o_{t}$, $\hat{o}_{t}$ and
$o^{\prime}_{t}$ are shown in Figure 3.

Figure 2: Relations between variables expressed in terms of types of neural
layers connecting them. Notice, that parts of the network are used in multiple
modules, e.g. Decoder (golden deconv trapezoid) is used in three branches.
Layers are trained in stages: golden layers (PAE) first; then blue (generator)
and red (discriminator) alternately; grey blocks are not trainable.

The architecture consists of modules presented diagrammatically in Figure 2.
The arrangement of the computations in the network reflects the Markovian
structure of the problem. The network can also be expressed using equations
from Table 1.

The implementation of the architecture in PyTorch [Paszke, 2017] can be found
in this repository444https://github.com/ghostwriterm/paegan-pytorch.

Belief propagation | $bs_{t}=RNN(bs_{t-1},Enc(o_{t}))$
---|---
Belief propagation |
(blind) | $bs_{t}=RNN(bs_{t-1},Enc(o_{\emptyset}))$
Expected observation |
(via PAE) | $\hat{o}_{t}=Decoder(bs_{t})$
Belief sampling |
| $s^{\prime}_{t}=Sampler(bs_{t},noise)$
Observation sampling |
| $o^{\prime}_{t}=Decoder(s^{\prime}_{t})$
Expected observation |
(via sampling) | $\hat{o}_{t}=\frac{1}{n}\sum\limits_{i}^{n}Decoder(s^{\prime(i)}_{t})$
Table 1: Equations describing functional relations between variables in the
architecture.

####  4.1.2 Predictive autoencoder

PAE consists of: a convolutional encoder, a deconvolutional decoder and a
recurrent layer (GRU). The network is trained by feeding in observations
$o_{1:T}$, computing $\hat{o}_{1:T}$ (as per Table 1) and minimising MSE loss:

$\mathcal{L}_{PAE}=\sum\limits_{t=1}^{T}(o_{t}-\hat{o}_{t})^{2}$ (2)

As the training progresses, an increasing proportion $p_{mask}$ of input
observations is corrupted by overwriting with zeros (these are null
observations). Initially, $p_{mask}$ is set to $0.3$; ultimately it reaches
$0.98$. With time, the network learns to fill in the missing observations. The
idea is to gradually increase the difficulty of tracking. [Oh et al., 2015]
elaborates on importance of curriculum (gradual) training. Once the
optimisation is complete, PAE allows for propagation of beliefs, updating with
observations, and projection from beliefs to observations. Following [Henaff
et al., 2017], PAE can be viewed as predictor of the deterministic part of the
environment.

Structure of PAE allows us to reason about information content of belief
state. Notice, that $bs_{t}$ and $o_{\emptyset}$ (which contains no
information) are sufficient to output reliable estimates of future
observations. This implies, that $bs_{t}$ contains information about
observable state, unobservable state that affects observable in future (e.g.
velocities) and uncertainties about those. In other words, belief state
represents a probability distribution over the environment state.

When observations are unavailable or of low quality, the belief distribution
widens. At times, it is desirable to generate samples from this distribution,
i.e. crisp, uncertainty-free states. The remaining modules work together to
achieve this sampling capability.

Notice, that sample from belief state (i.e. $s^{\prime}_{t}\sim bs_{t}$) are
valid belief states themselves. Thus, those can be used as inputs for belief
propagation and decoding functions.

####  4.1.3 Discriminator

Discriminator (D), as in [Goodfellow et al., 2014], is a convolutional network
which consumes an image and outputs a classification. The network is optimised
to distinguish between images from two sets: (a) real observations ($o_{t}$)
(b) observation samples ($Decoder(s^{\prime}_{t})$). During training only
Discriminator layers of the are updated.

####  4.1.4 Future averager

Expected value over a distribution can be approximated by averaging value of
samples from that distribution:

$\mathbb{E}[X]=\int x\ p(x)\ dx\approx\frac{1}{n}\sum\limits^{n}x\sim X$ (3)

Similarly, if we can generate sample states $s^{\prime}_{t}$ from belief state
$bs_{t}$, then the expected observation can be approximated by averaging
observations that result from those samples (as per last row of Table 1).
Furthermore, samples can be propagated further in time using the recurrent
layer. This enables computation of future expected observations. Notice, that
we already had a way to compute both of these using PAE alone.

####  4.1.5 Belief state sampler

Sampler consumes $bs_{t}$ and a noise vector. Since the output is not
deterministic a source of randomness is required. This module is a simple
Multilayer perceptron (MLP). The output of the Sampler is constrained by two
losses:

Generator loss – every sample $s^{\prime}$ produced is projected onto
observation space and assessed by the Discriminator. This incentivises the
Sampler to generate samples with no uncertainty in the observable state.

$\mathcal{L}_{G}=BCELoss(Discrimnator(Decoder(s^{\prime})),\ 1)$ (4)

Averager loss – future expected observation $\hat{o}_{t+T}$ should be the same
whether computed by PAE or Observation averager. For set of $n$ generated
samples $s^{\prime 1:n}_{t}$, the loss is:

$\mathcal{L}_{Av}=[Decoder(bs_{t+T})-\frac{1}{n}\sum\limits^{n}Decoder(s^{\prime(i)}_{t+T})]^{2}$
(5)

Where $s^{\prime}_{t+T}$ is computed from $s^{\prime}_{t}$ using blind belief
propagation (Table 1).

Due to this loos the generated samples are required to add up to the belief
state from which they were sampled. This prevents mode collapse. The overall
loss for the Sampler training is:

$\mathcal{L}_{Sampler}=\lambda_{G}\mathcal{L}_{G}+\lambda_{Av}\mathcal{L}_{Av}$
(6)

Sampler and Discriminator are trained interchangeably while PAE is held
constant.

####  4.1.6 Design principles

During the design we focused more on the overarching architecture than on the
low level hyperparameters (such as particular activation functions or
inclusion of normalising layers). PAEGAN’s structure aims to reflect the
underlying Markov chain estimation problem. We rely on controlling the flow of
information through the network to enforce particular information content in
the learnt representation. For example, during PAE training, observations are
intermittent to incetivise propagation of information through time.
Additionally, we attempt to force specific representations (e.g. uncertainty-
free samples) by constraining intermediate outputs of the network with
particular loss functions.

Good performance in experiments despite little focus on low level parameters,
possibly indicates generality of the architecture. We used generic and simple
neural modules: encoder and decoder are generic, 3-layered convolutional
networks with ReLU activations, RNN is 1-layer of Gated Recurrent Units,
Discriminator is taken from DCGAN [Radford et al., 2015]. Batch normalisation
is not used apart from DCGAN. Belief state is represented by 256 numbers at
the recurrent layer. For all optimisation we used Adam with learning rate
$0.0002$.

The values of $\lambda_{G}$ and $\lambda_{Av}$ that ensure good performance
depend on the data used for training (e.g. we chose $\lambda_{G}=1,\
\lambda_{Av}=500$). Overall, the two loss functions should have similar
contribution to the training.

##  5 EXPERIMENTS

###  5.1 SETUP

PAEGAN was tested in a setup similar to [Lotter et al., 2015]. The environment
is a simulation of moving balls. Depending on the settings, the balls bounce
from each other (and walls) or phase through. Targets are moving with near
constant velocity, with noise added. The velocities and positions of the balls
are continuous variables. The simulator produces observations from the
environment state. These are $28\times 28\times 1$ images, which represent
information about positions of targets, see Figure 3. Those can be corrupted
with noise (random shifts to target positions) or completely accurate. Using
terms from Section 2, the environment is stochastic and partially-observable
via intermittent and noisy unstructured percepts.

The network was trained using purely time series of observations. For memory-
management purposes, the duration of training episodes was $100$ time steps.
At test time, a fixed number of initial observations ($8$) was provided with
null observations following. The state tracking was assessed by inspecting MSE
of predicted observations over $160$ time steps. This allowed to quantify
retention of the information gained from percepts. Tracking was assessed also
for observations arriving randomly (e.g. with $5\%$ chance per time step).
This emulates state tracking when the sensor is not reliable or decisions need
to be made without new percepts. We also extract sample observations and
assess their realism and variability. Multiple training and evaluation runs
were completed for different dynamics of the simulator. Scenarios included one
or many balls moving, in some cases interacting via collisions.

As both the forward and sensor models are known exactly, it was possible to
implement a PF. Generally, when accurate stochastic models are provided, the
PF is an optimal state estimator in the limit of infinity samples used. In our
case, since the process at hand was not strongly non-linear or high-
dimensional, $1000$ particles assured good performance. The data fed was a
structured measurement of position for every target in the scene.

Because PF maintains a belief distribution over state, we can generate the
expected observation conditional on that belief (using the sensor model). This
means PAEGAN and PF can be compared on the same metric of image prediction.
Note, that PF is a very strong benchmark and solves an easier problem: state
estimation with models provided. PAEGAN learns models from the data and then
performs predictions.

The implementation of experiments is also available in the project repository.

###  5.2 RESULTS

The predictions output by PAEGAN are similar to those of PF (albeit more
uncertain). Consider Figure 3, but it is best to see videos here555GoogleDoc.
This favours the hypothesis from [Mathieu et al., 2015], that a deterministic
predictive network (PAE) approximates the expectation over the target value.
Figure 4 shows how reconstruction error gradually increases when observations
are not available. As should be expected, PF is more successful at tracking
the environment state. This is because PAE did not infer the forward model
perfectly. The difference is especially apparent in more complex environments
as models of those are more difficult to extract from data (e.g. inherently
chaotic simulation of collisions). Notice, that neither method does worse than
the uninformed baseline (dotted green line). At this error level, the estimate
reaches maximum uncertainty.

Figure 3: Observation estimates produced by PF and PAEGAN after not observing
the environment for $40$ time steps. From the left: an observation with two
balls (prediction target); expectation and sample observation from PF;
finally, corresponding outputs from PAE. Image blur signifies uncertainty; it
is not present in samples.

Nonetheless, PAEGAN learns to represent and recursively propagate information
about the environment, effectively learning the underlying physics.
Importantly, it offers not only predictions, but also associated
uncertainties. The representation contains information also about unobservable
state of the environment – velocities of targets need to be inferred for
successful predictions.

While this is not shown here (see the videos page linked earlier), PAEGAN also
reliably tracks state when observations arrive infrequently or contain noise.
This process is referred to as smoothing and often is added to systems (e.g.
in form of PF) to enable decision-making when recent observations are likely
to be unavailable.

Figure 4: Plots show reconstruction loss for predictions in increasingly
complex environments (left to right). PF provides more reliable
reconstructions than PAEGAN. The difference is especially clear for more
environment physics (right-most plot) . Green dotted line is an uninformed
baseline – best level achievable without access to any recent observations (by
outputting an average observation).

Observation samples produced are realistic and appear to be taken from the
belief distribution (see Figure 3 and videos). Normally, it is difficult to
assess whether samples cover an entire distribution. Realism and diversity are
typically assessed by the operators. In this case, ensuring that many samples
together add up to the distribution of origin is an explicit training goal
(Equation 5). The quality of the samples decreases when belief state is at
maximum uncertainty: it appears that not the entire state space is sampled
(mode collapse).

The duration of training for the PAE module strongly depends on the complexity
of the environment. Learning collision kinematics of two balls required about
$10\times$ more parameter updates than training for a single moving target.
With prolonged optimisation, the risk of overfitting was higher. If provided
with a small data set, the network tended to memorise the trajectories instead
of learning the underlying models. This can be easily detected by validating
the model on held out data.

Performance of the architecture is bounded by the quality of PAE predictions.
For many problems, predicting the average future outcome is much easier than
generating samples of the future. This means that, in general, the
representations formed by a PAE are not necessarily well matched for sampling.
It is possible to improve this by combining our approach with work from
[Henaff et al., 2017], where noise input to the network is used as an
explanatory variable of the observations. In our particular experiments, the
average outcome is indeed informative about various consequences.

Concurrent training of Sampler and Discriminator displays lack of stability
representative of GAN approaches. In our case, optimising the D network less
frequently than G improved the quality of generated samples. It could be
argued that, since G is more constrained (see Equation 6), it demands more
search through the parameters.

##  6 DISCUSSION AND FURTHER WORK

PAEGAN is a combination of existing attempts at prediction of unstructured
time series using neural networks. We present it in the frame of Markov chain
estimation. From a black-box perspective, PAEGAN can be seen as a trainable
Particle Filter for unstructured data. The recursive structure enables
flexibility of inputs and outputs. The observations can be noisy and
intermittent, the outputs include expectation over future observations,
samples of future observations and belief states (as a neural representation).

The state tracking method can be directly connected to neural reinforcement
learning with multiple expected advantages. Firstly, the representations
useful for predictions are likely relevant for related tasks, such as policy
search and value estimation. Therefore, when training PAEGAN in an
unsupervised manner, the agent is able to extract more information from its
experiences. Secondly, a forward model of the environment enables planning.
Thirdly, when observations are unreliable (e.g. server latency in OpenAi
Universe), then the agent can rely on the estimate of environment state.
Lastly, agents in largely unobservable worlds (like ours) need to combine
information across time, sensors and locations. Similar problems (albeit with
structured data) are today solved using information fusion methods, an example
of which is the Particle Filter. It is foreseeable that a descendant of PAEGAN
could perform a similar function. These advantages are especially valuable for
agents embedded in the real world (e.g. robotic systems), which need to learn
from limited data. The direct next steps for the architecture are to apply it
to more complex environments in a reinforcement learning framework (e.g. Atari
games) with focus on data-efficient learning and planning.

Another research avenue is the improvement of quality of the predictions.
Currently, the network encodes the environment estimate using the state of a
recurrent layer. This means that a single propagation operation needs to track
possibly multiple modes. Alternatively, it is possible to express knowledge
about state as a collection of samples from a given belief distribution (as is
done in Particle Filter). In some situations, it might be valuable to
represent the state estimate as a combination of belief states by multiple
instantiations of PAEGANs. Humans intuitively predict the future using a
mixture of sampling and uncertainty representation. Sufficiently distinct
events are evaluated as different branches of the future each with their own
associated uncertainties. For example: ’I might go to work either by car or
bus’ – two distinct scenarios. However, once we zoom in on a particular path,
we uncover uncertainties particular to this case (e.g. how much do I expect to
pay for a bus ticket). Implementing a similar prediction mode using PAEGAN
might bring two gains. Firstly, improved quality of predictions, as a single
instantiation of the network needs to propagate only unimodal beliefs.
Secondly, improved computational efficiency for approaches such as Monte Carlo
tree search, as the branching could be performed only at the significant
events when the belief distribution diverges into multiple modes. In such a
view, the nodes of the tree would be belief states rather than state samples.

## References

  * [Azzouni and Pujolle, 2017] Azzouni, A. and Pujolle, G. (2017).  A Long Short-Term Memory Recurrent Neural Network Framework for Network Traffic Matrix Prediction.
  * [Baldi, 2012] Baldi, P. (2012).  Autoencoders, Unsupervised Learning, and Deep Architectures.  ICML Unsupervised and Transfer Learning, pages 37–50.
  * [Browne et al., 2012] Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., and Colton, S. (2012).  A Survey of Monte Carlo Tree Search Methods.  Computational Intelligence and AI in Games, IEEE Transactions on, 4(1):1–43.
  * [Chiappa et al., 2017] Chiappa, S., Wierstra, D., and Mohamed, S. (2017).  Recurrent Evironment Simulators.  In ICLR, number 2015, pages 1–19.
  * [Chung et al., 2014] Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014).  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.
  * [Doucet et al., 2001] Doucet, A., de Freitas, N., and Gordon, N. (2001).  An Introduction to Sequential Monte Carlo Methods.  Sequential; Monte Carlo Methods in Practice, pages 3–14.
  * [Goodfellow et al., 2014] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014).  Generative Adversarial Networks.
  * [Henaff et al., 2017] Henaff, M., Zhao, J., and LeCun, Y. (2017).  Prediction Under Uncertainty with Error-Encoding Networks.
  * [Hero and Cochran, 2011] Hero, A. O. and Cochran, D. (2011).  Sensor Management: Past, Present, and Future.  IEEE Sensors Journal, 11(12):3064–3075.
  * [LeCun, 2016] LeCun, Y. (2016).  Talk: Predictive learning.  Neural Information Processing Systems.
  * [Lotter et al., 2015] Lotter, W., Kreiman, G., and Cox, D. (2015).  Unsupervised Learning of Visual Structure using Predictive Generative Networks.
  * [Lotter et al., 2016] Lotter, W., Kreiman, G., and Cox, D. (2016).  Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning.
  * [Mathieu et al., 2015] Mathieu, M., Couprie, C., and LeCun, Y. (2015).  Deep multi-scale video prediction beyond mean square error.
  * [Mnih et al., 2016] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016).  Asynchronous Methods for Deep Reinforcement Learning.
  * [Mnih et al., 2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. (2015).  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533.
  * [Oh et al., 2015] Oh, J., Guo, X., Lee, H., Lewis, R., and Singh, S. (2015).  Action-Conditional Video Prediction using Deep Networks in Atari Games.
  * [Paszke, 2017] Paszke, A. (2017).  Automatic differentiation in PyTorch.
  * [Radford et al., 2015] Radford, A., Metz, L., and Chintala, S. (2015).  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.
  * [Silver et al., 2017] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. (2017).  Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm.
  * [Srivastava et al., 2015] Srivastava, N., Mansimov, E., and Salakhutdinov, R. (2015).  Unsupervised Learning of Video Representations using LSTMs.
  * [Sutton, 1991] Sutton, R. S. (1991).  Dyna, an integrated architecture for learning, planning, and reacting.  ACM SIGART Bulletin, 2(4):160–163.

  *[RL]: Reinforcement Learning
  *[PL]: Predictive Learning
  *[PF]: Particle Filtering
  *[PAE]: Predictive Autoencoder
  *[EEN]: Error-Encoding Network
  *[GAN]: Generative Adversarial Networks
  *[PAEGAN]: Predictive Autoencoding Generative Adversarial Network


# Method for Robotic Motion Compensation during PET Imaging of Mobile Subjects

Junxiang Wang1, Iulian Iordachita1, Peter Kazanzides2 1Dept. of Mechanical
Engineering, Johns Hopkins University, Baltimore, MD 21218, USA (email:
[jwang334, iordachita]@jhu.edu)2Dept. of Computer Science, Johns Hopkins
University, Baltimore, MD 21218, USA (email<EMAIL_ADDRESS>
###### Abstract

Studies of the human brain during natural activities, such as locomotion,
would benefit from the ability to image deep brain structures during these
activities. While Positron Emission Tomography (PET) can image these
structures, the bulk and weight of current scanners are not compatible with
the desire for a wearable device. This has motivated the design of a robotic
system to support a PET imaging system around the subject’s head and to move
the system to accommodate natural motion. We report here the design and
experimental evaluation of a prototype robotic system that senses motion of a
subject’s head, using parallel string encoders connected between the robot-
supported imaging ring and a helmet worn by the subject. This measurement is
used to robotically move the imaging ring (coarse motion correction) and to
compensate for residual motion during image reconstruction (fine motion
correction). Minimization of latency and measurement error are the key design
goals, respectively, for coarse and fine motion correction. The system is
evaluated using recorded human head motions during locomotion, with a mock
imaging system consisting of lasers and cameras, and is shown to provide an
overall system latency of about 80 ms, which is sufficient for coarse motion
correction and collision avoidance, as well as a measurement accuracy of about
0.5 mm for fine motion correction.

##  I Introduction

The physical distance between humans and robots has decreased significantly
since robots were introduced in factories in the 1960s, isolated from humans
by gates and other safety barriers. In the 1990s, robotic surgery brought
robots into contact with humans, and more recently, _collaborative_ robots
have been introduced to enable humans and robots to safely work together. The
distance between humans and robots decreases even further with prosthetics and
exoskeletons, where the robot is physically attached to the human to replace a
missing limb or to provide additional support.

In this work, we explore a different paradigm for integration of humans and
robots, which is to introduce a robot that maintains close proximity without
physically contacting the human. Our specific goal is to position a PET
imaging ring around the head of a subject and accurately measure the position
of the head with respect to the imaging ring. The measurement is required to
enable the robot to move the imaging ring to compensate for limited motion of
the subject (coarse motion correction) and for motion correction in the image
reconstruction algorithm (fine motion correction). The concept for robotic PET
imaging was introduced by Majewski et al. [1] based on prior experience with a
wearable imager, called Helmet PET [2], that was limited by the unavoidable
trade-off between weight and sensitivity.

We previously analyzed motion capture and accelerometer data from subjects
during overground and treadmill walking, respectively, to determine the
typical range of head motion, as well as head velocity and acceleration [3].
We assumed a helmet to be worn by the subject with dimension 263 mm x 215 mm,
which corresponds to a minimal (radial) clearance of about 18 mm between the
helmet and an imaging ring of 300 mm diameter. Our simulation results
indicated that it is feasible for a robot to compensate for the head motion,
without colliding with the imaging ring, given a system latency of less than
100 ms [3].

We subsequently developed a measurement system that consists of six string
encoders connected between the PET imaging ring and a helmet worn by the
subject [4]. The encoders were arranged in a parallel configuration (Stewart
Platform) and experimentally shown to provide a static measurement accuracy of
about 0.5 mm, which matches the accuracy desired by the PET imaging scientists
to correct for motion artifact during image reconstruction. In [4], we also
surveyed existing proximity sensing techniques used in various fields and
showed that our system provides higher accuracy, higher robustness, and lower
latency.

Figure 1: Overall setup with UR3 and UR5 robots, the string encoder system, as
well as the mock PET imaging system.

Our goal in this work is to extend this static measurement into dynamic
conditions, where a low-latency robotic system is designed around the position
and velocity measurements of the string encoder system. The test input is
provided by a second robotic system that reproduces the recorded human head
motion, as shown in Figure 1. This proof-of-concept robotic system will enable
us to experimentally verify the feasibility of robotic compensation for human
head motion.

Our proposed system involves human-robot interaction, where the robot must
safely move in proximity of the human, and thus prior work includes studies of
robot safety and especially methods for enhancing safety through predictive
control. Mainprice [5] developed a prediction framework in which a library of
human motion data is captured in a model with regression performed offline,
and subsequently real-time prediction can be made given some actual perceived
human motion, accompanied by appropriate robot motion planning and execution.
Koppula suggested a similar approach for anticipation [6].

Haddadin’s works on collision detection, evaluation, and reaction [7] [8], as
well as Dogramadzi’s methods of safety assessment, especially for robot hazard
and error [9], are relevant to the risk assessment and safety design of this
type of system. However, this paper focuses on the situation where the robotic
system functions normally, and collision is avoided through measurement and
compensation of motion, whereas future safety design will focus on ensuring
correct system operation, including redundant motion sensing, and evaluating
the risk due to failure.

The following sections present the design of the proposed system for
supporting the PET imaging ring around the subject’s head (Section II-A) and
the associated control algorithm (Section II-B). We then explain our
experimental setup with a robotic system for emulating human head motion
(Section III-A) and an optical system for emulating PET imaging (Section
III-B). The results include the quality of the encoder-based motion
measurement (Section IV-A), the fidelity with which human head motion is
reproduced (Section IV-B), and the performance of both coarse and fine motion
corrections (Sections IV-C and IV-D).

##  II System

This section describes the components of our prototype robotic motion
measurement and compensation system with string encoders and a UR5 robot,
shown with other experiment components in Fig. 1. We first discuss the overall
system structure that focuses on latency minimization, followed by details of
the UR5 robot controller and its utilization of the string encoder
measurements.

###  II-A Prototype Robotic PET System Design

A block diagram of the prototype system for tracking and compensating for
human head motion during PET imaging is shown in Fig. 2. A key design goal was
to minimize latency, which led to the use of a Field Programmable Gate Array
(FPGA) for processing the encoder signals and a component-based software
architecture where all components are in a single process on the Control PC.

Figure 2: System block diagram, showing string encoders to measure position
and velocity of helmet (worn by subject, not shown) with respect to imaging
ring supported by UR5 robot.

Head motion tracking is performed by six string encoders111MPS-XXXS-200MM-P,
Miran Industries, China. Three units had their rotary encoder replaced with
E30S4-1000-3-T-5, Autonics, Korea due to hardware defects., with resolution of
60 quadrature counts per millimeter, connected in parallel between a head-
mounted helmet and a (mock) PET imaging ring and able to measure a 6 degree-
of-freedom (DOF) Cartesian pose, as studied in [4]. The encoder signals are
processed by an FPGA with a custom interface board. The FPGA board is an open
source design [10] (Rev 2.1) that is primarily used for the da Vinci Research
Kit (dVRK) [11]. The custom interface board consists of connectors and
receivers that convert the 5V encoder signals to the 3.3V level expected by
the FPGA. The FPGA firmware is a customized version of the standard dVRK
firmware [12] (Rev 7), with the following modifications:

  * •

Provides up to 8 channels of encoder feedback, rather than just 4 channels.

  * •

Saves the encoder counter value when an index pulse occurs, and provides this
value to the host PC.

The encoder feedback consists of position, obtained by quadrature counting of
encoder pulses, and velocity, which is based on the measured time between
encoder edges. In particular, the velocity measurement has been optimized to
provide accurate estimates with low latency, even for low-resolution encoders
[13]. The FPGA board is connected to the host PC via Ethernet and uses the UDP
protocol for low-latency data exchange.

The software is based on the open-source _cisst_ libraries [14] that support
components with separate threads in a single process, with efficient (lock-
free) data exchange mechanisms. In addition to an application component (App)
that provides a user interface for exercising the system and collecting data,
there are three primary components:

Encoder Interface: This is a periodic component that reads the string encoder
positions and velocities from the FPGA via the UDP interface and then performs
the forward kinematic computations to convert them to Cartesian positions and
velocities. This component is executed at 1 kHz.

UR5 Interface: This is an instance of the open-source sawUniversalRobot
component [15], which interfaces to the UR5 controller via its real-time
script interface. The component thread waits for a feedback packet from the
UR5 controller, which is provided at 125 Hz, and thus the component executes
at 125 Hz.

UR5 Control: This component implements the robot motion controller described
in Section II-B. It shares the same thread as the UR5 Interface component and
thus is executed synchronously with that component at 125 Hz. In particular,
the UR5 Interface component transfers control to the UR5 Control component
after receiving the UR5 feedback packet. UR5 Control computes the commanded
velocity, then returns control to UR5 Interface, which sends the commanded
velocity to the UR5 robot. The UR5 robot cannot handle the expected weight of
an actual PET imaging ring, estimated at 15-20 kg, but the final system could
scale up by using a larger robot, such as the UR20 or a custom design.

###  II-B Robotic Motion Control System

Using the 6 DOF position and velocity reported by the encoder interface, the
UR5 robot moves itself in the direction to restore the helmet ring and the
imaging ring of the string encoder system back to its nominal configuration.
The robot is controlled in Cartesian velocity mode, via the real-time script
interface, because this has been shown to provide lower latency than position
control [16]. The general control logic for this robotic motion compensation
is velocity feedforward control with position correction (see Fig. 3).

Figure 3: Block diagram of UR5 robot velocity controller, used to move imaging
ring with respect to helmet. $\dot{x}_{m}$ is the measured UR5 robot Cartesian
velocity. $l_{m}$ and $\dot{l}_{m}$ are the measured string encoder lengths
(positions) and velocities, respectively, between imaging ring and helmet.

The Encoder Interface directly outputs the string length $l_{m}$ and velocity
$\dot{l}_{m}$ for each string encoder. As previously described [4], sending
the set of $l_{m}$ through the parallel robot forward kinematics gives the 6
DOF Cartesian pose measured by the system, and applying the Jacobian to
$\dot{l}_{m}$ yields the Cartesian velocity. In our setup, these quantities
respectively correspond to the position difference $e_{x}$ and the velocity
difference $e_{\dot{x}}$ between the head and the PET device. These are
averaged over 30 consecutive samples to reduce the noise inherent in the
velocity measurement. This filter window size was empirically determined to
achieve the noise reduction needed for smooth robot movement, while not
inducing too much latency. All aforementioned operations are performed in the
encoder interface at 1 kHz, which increases the measurement latency by about
15 ms.

The position correction component of the motion controller is obtained by
multiplying $e_{x}$ by a constant proportional gain $K_{p}$. The desired
feedforward velocity $\dot{x}_{d}$ is the sum of the measured UR5 robot
Cartesian velocity $\dot{x}_{m}$ and the velocity difference $e_{\dot{x}}$.
The translational (first three) and rotational (latter three) components of
$\dot{x}_{d}$ are each given a separate nonlinear, exponential gain,
$K_{e}^{T}$ or $K_{e}^{R}$. The velocity commanded to the UR5 robot
$\dot{x}_{c}$ is the sum of the position difference and velocity terms, each
multiplied by their respective gains, as shown in Fig. 3 and the equation
below:

$\dot{x}_{c}=K_{p}e_{x}+\begin{bmatrix}K_{e}^{T}I_{3}&0_{3,3}\\\
0_{3,3}&K_{e}^{R}I_{3}\end{bmatrix}\dot{x}_{d}$ (1)

where $I_{n}$ denotes the $n$-dimensional identity matrix, and $0_{m,n}$ the
$m$ by $n$ zero matrix.

The gain factor $K_{e}^{T}$ is a function of $\delta^{T}$, the magnitude of
the translational components of $e_{x}$, such that the gain approaches 0 when
$\delta^{T}$ is small and approaches 1 when it is large. In particular, its
value is determined by the following piecewise continuous function:

$K_{e}^{T}(\delta^{T})=\begin{cases}\frac{\delta^{T}}{\delta^{T}_{\text{max}}}e^{\frac{\delta^{T}}{\delta^{T}_{\text{max}}}-1}&\text{for
}0\leq\delta^{T}<\delta^{T}_{\text{max}}\\\ 1&\text{for
}\delta^{T}\geq\delta^{T}_{\text{max}}\end{cases}$ (2)

where $\delta^{T}_{\text{max}}$ is a threshold beyond which $K_{e}^{T}$ should
always evaluate to 1, and is set as 20 mm in our implementation. $K_{e}^{R}$
is defined analogously, with $\delta^{R}$ being the “angle” value in the axis-
angle representation of the rotational component of $e_{x}$, and with
$\delta^{R}_{\text{max}}$ as 14 deg. This nonlinear gain improves the
performance of the system by avoiding oscillations around the target pose due
to noisy velocity measurements. Moreover, $\delta_{\text{max}}$’s were
empirically determined such that the actual $\delta$’s would rarely reach the
maximum levels under typical motions, so only partial velocity is fed forward,
which reduces overshoot due to the averaging filter phase lag.

##  III Experiments

This section first discusses our experimental setup for simulation of human
head motion and PET imaging, followed by descriptions of experiments for
evaluating the system’s performance for both coarse and fine motion
correction.

###  III-A Robotic System for Simulating Human Head Motion

To reproduce the collected human head motion from our previous experiments
[3], we use a UR3 robot. We re-computed 6 DOF head motion trajectories from
the original optical marker data collected at 60 Hz, using a total of five
marker positions: $P_{LEYE}$, left side of the front of the head; $P_{REYE}$,
right side of the front of the head; $P_{LEAR}$, left side of the back of the
head; $P_{REAR}$, right side of the back of the head; and $P_{C7}$, cervical
vertebra. The coordinate system for these positions is defined such that $+x$
is in the direction of locomotion of the subjects, and $+z$ is vertically
upwards. A linear regression is performed on the $P_{C7}$ time series to
estimate the position of the body center as the subjects walked overground in
a straight line, denoted as $P_{C7,fit}$. We compute the translation of the
head center with:

$t_{c}=\frac{1}{4}(P_{LEYE}+P_{REYE}+P_{LEAR}+P_{REAR})-P_{C7,fit}$ (3)

The first term in equation (3) finds the mean of the four markers on the head
to approximate the position of the head center, and the subtraction of
$P_{C7,fit}$ is to remove the effect of overground walking to simulate
treadmill walking. We denote the orientation of the head by
$R_{c}=[\textbf{r}_{x},\textbf{r}_{y},\textbf{r}_{z}]$. Since $P_{LEYE}$ and
$P_{REYE}$ lie on approximately the same horizontal level, with $P_{LEAR}$ and
$P_{REAR}$ on a different level, these four points roughly form a common
plane. $\textbf{r}_{z}$ is then determined as the norm vector of the best-fit
plane with a positive z-component. $\textbf{r}_{x}$ is given by the unit
vector in the direction:

$(P_{LEYE}+P_{REYE})-(P_{LEAR}+P_{REAR}).$ (4)

Lastly, $\textbf{r}_{y}$ is simply calculated with
$\textbf{r}_{z}\times\textbf{r}_{x}$. Since these vectors likely do not align
with the principal axes of the head geometry, the inverse of the starting
rotation is applied to each rotation in the trajectory as an approach at
normalization. The rotations are parametrized with Euler ZYX convention and,
combined with translations, encode each trajectory as a sequence of 6-vectors
$x_{d}$. A Savitzky-Golay filter with fourth degree polynomial fitting over a
frame length of 17 is applied to each sequence for smoothening and calculation
of the first and second derivatives—Cartesian velocities $\dot{x}_{d}$ and
accelerations $\ddot{x}_{d}$. $x_{d}$ and $\dot{x}_{d}$ are then provided as
input to the UR3 motion controller shown in Fig. 4.

Figure 4: Block diagram of UR3 robot velocity controller, used to emulate
human head (helmet) motion. $x_{d}$ and $\dot{x}_{d}$ are the desired helmet
Cartesian position and velocity, respectively, obtained from recorded human
head motion data. $x_{m}$ and $\dot{x}_{m}$ are the measured Cartesian
position and velocity, respectively, of the UR3 robot.

At each time step (every 16.7 ms), the velocity commanded to the UR3 is the
sum of $\dot{x}_{d}$ and a correction term based on the pose error between
$x_{d}$ and the UR3’s predicted pose at the next time step, calculated from
its measured 6 DOF pose $x_{m}$ and velocity $\dot{x}_{m}$. This pose
difference $e_{x}$ is multiplied by a constant gain $K_{p}$ and added to
$\dot{x}_{d}$ to form the total commanded velocity to the UR3, $\dot{x}_{c}$.

To statistically confirm the fidelity of reproducing human head motion, we
recorded $x_{m}$ and $\dot{x}_{m}$ from the robot and performed a first-order
Savitzky-Golay differentiation on $\dot{x}_{m}$ to obtain the actual
acceleration $\ddot{x}_{m}$. Section IV-B compares the means and standard
deviations of these three actual quantities to their corresponding desired
values.

The UR3 robot integrates into the system described in Section II by attaching
the helmet ring of the string encoder system onto the UR3’s end effector, as
shown in Fig. 1. This motion reproduction system can be reused even when the
UR5 robot is replaced by a robot capable of delivering the larger payload
(15-20 kg) that will be required to support an actual PET imaging system.

###  III-B Optical System for Simulating PET Imaging

Since the ultimate goal of our robotic system is to provide not only coarse
motion correction with robot movement but also fine motion correction with
image reconstruction, we simulated the radioactive PET imaging process with a
simplified optical setup. Specifically, since PET relies on the emission and
detection of positrons shooting in opposite directions from tissues, we used
backward-facing laser diodes to simulate pairs of positrons, and cameras with
screens in front of them to simulate PET detectors. We employed four laser
diodes (VLM-650-03 LPA, Quarton Inc., Korea) and four cameras (UB0234,
Arducam, China). The lasers were mounted to the center of the helmet ring,
facing outward horizontally, spaced apart evenly by 90 deg, and the cameras
were mounted to the outer edge of the imaging ring, facing inward to align
with the laser directions in the nominal configuration of the string encoder
system. An acrylic frame supporting a screen of white paper is placed between
each camera and laser to block the laser beam, with the paper thin enough for
the camera on the other side to clearly capture the laser dot. Fig. 5 shows a
top-view diagram of the lasers and cameras, Fig. 1 shows their location within
the overall system, and Fig. 6 shows one active laser diode, its corresponding
dot on the paper screen, and a camera.

Figure 5: Top view diagram showing outlines of prototype helmet (green) and
imaging ring (red) with mock PET imaging system (lasers and cameras). Figure
6: Close-up view of laser, screen, and camera.

The two pairs of laser beams emulate two orthogonal lines emanating from a
single radioactive point source in the brain. Our reconstruction algorithm
determines the position of the point source p in the imaging ring frame $I$,
analogous to actual PET image reconstruction. We first locate the position of
the laser dot on each camera screen, yielding four 2D coordinates
$\\{^{C}\textbf{x}_{i}=(x_{i},y_{i})\\}_{i=1}^{4}$ with $(0,0)$ being the
position in the nominal configuration of the string encoder system. The 3D
representations of ${}^{C}\textbf{x}_{i}$ in frame $I$,
${}^{I}\textbf{x}_{i}$, are then computed. p is determined as the intersection
of two lines: one connecting ${}^{I}\textbf{x}_{1}$ and
${}^{I}\textbf{x}_{3}$, the other one connecting ${}^{I}\textbf{x}_{2}$ and
${}^{I}\textbf{x}_{4}$. Fig. 5 shows the numbering and coordinate system
orientations, as well as a hypothetical non-nominal configuration where there
is a non-zero translation in the x-y plane between the imaging ring and the
helmet. Labeled in the figure are the points where the lasers would hit the
screens, as well as the computed position of the point source p, shown as a
purple vector near the center. The outer diameter of the helmet ring is drawn
scaled-down in order to exaggerate the allowable translation for the benefit
of clearer illustration. The figure is a simplified scenario, and in general
this method for determining p can be used for arbitrary 6 DOF pose of the
helmet.

###  III-C Experimental Procedures

We first evaluated the string encoder system’s measurement accuracy and
latency during dynamic motions by commanding the UR3 with a 2 Hz sinusoidal
wave, which is a frequency typical to natural human head motion based on our
previous data[3]. We evaluated the measurements against the motion provided by
the UR3; the other components of our system were inactive for this experiment.

We then evaluated the performance of the complete system using a real human
head motion trajectory. With all previously described components and
controllers in effect, we commanded the UR3 with one of the processed head
motion trajectories. Starting with comparison between the recorded motion and
the UR3 reproduced motion as described in Section III-A, we then evaluated the
performance of both the coarse motion correction and the fine motion
correction. In the former, the UR5 motion was compared against the UR3 motion
with error and latency assessments. In the latter, the string encoder
measurements were used to adjust the images captured by the cameras to account
for residual motion (not compensated by the robot), and in the ideal case, the
laser intersection would be seen as stationary after this reconstruction
process.

##  IV Results

###  IV-A Head Measurement Accuracy and Latency

Figure 7: Dynamic position and velocity measurements from string encoder
system against actual UR3 motion.

Fig. 7(a) shows the displacement of the UR3 during the 2 Hz sine wave,
compared against the string encoder system’s measured displacement. A zoomed-
in view is provided in the center to illustrate the very low latency of around
4 ms between the two. Fig. 7(d) shows their difference after correction for
this latency, which signify the measurement accuracy of the string encoder
system; this has a mean of 0.19 mm with a standard deviation of 0.09 mm. Fig.
7(b) and Fig. 7(e) are the corresponding plots for velocity, and the
measurement error has a mean of 12 mm/s with a standard deviation of 13 mm/s.
The larger amount of noise in velocity measurements also confirms the decision
to apply filtering, the result of which is shown in Fig. 7(c). Note that the
difference in Fig. 7(f) is computed after correction for both measurement
latency and filter latency. The filtered velocity measurement accuracy has a
mean of 11 mm/s with a standard deviation of 12 mm/s. These results
demonstrate that the high static position measurement accuracy shown
previously [4] extends to dynamic measurements.

###  IV-B Accuracy of Reproducing Human Head Motion

Figure 8: Response of UR3 in reproducing human head motion in all three axes.
Horizontal axes represent time in seconds.

The UR3 reproduction of human head motion is evaluated against the input head
trajectory visually in Fig. 8 and statistically in Table I. As shown in the
figure, after a slight lag in the beginning, the UR3 is able to execute the
commanded motion with high accuracy. Since our goal with the UR3 is only for
it to accurately emulate human head motion, any lag does not affect the
overall motion compensation system, as long as its motion is consistent with
the input. The statistics in the table, which do not include the first 1 s of
motion, confirm that the UR3 output motion follows the characteristics of the
desired input motion.

TABLE I: Mean and standard deviation of position, velocity, and acceleration data for both desired (D) and measured (M) motion. Units are in mm, mm/s, and mm/s2 | Displ. | Vel. | Acc.
---|---|---|---
| Mean | S.D. | Mean | S.D. | Mean | S.D.
X, D | -15.0 | 20.2 | -6.72 | 141 | 45.5 | 815
X, M | -15.6 | 20.2 | -8.90 | 130 | -0.99 | 705
Y, D | -6.14 | 32.4 | -2.39 | 69.5 | 12.7 | 439
Y, M | -4.36 | 43.1 | -1.41 | 67.5 | 10.9 | 493
Z, D | 16.6 | 8.33 | 3.08 | 78.3 | -18.3 | 767
Z, M | 15.8 | 9.27 | 3.58 | 76.8 | -7.73 | 844
Roll, D | 1.78 | 2.76 | 0.51 | 13.1 | -2.59 | 116
Roll, M | 1.63 | 2.84 | 0.39 | 12.8 | 1.94 | 150
Pitch, D | -1.31 | 1.71 | -0.27 | 10.57 | 3.70 | 138
Pitch, M | -1.28 | 1.85 | -0.72 | 9.72 | 0.40 | 202
Yaw, D | 0.53 | 1.62 | 0.25 | 8.52 | -1.82 | 103
Yaw, M | 0.54 | 1.67 | 0.44 | 8.97 | 1.30 | 171

In addition, the UR3 head motion controller serves as a representation of the
open-loop response of the UR5 motion compensation controller—in the ideal
situation without measurement noise, the UR5 controller would function
identically to the UR3 controller. Hence, although latency is not a concern
for the UR3 controller, we measured its lag with a ramp test signal of 80 mm/s
velocity up to a distance of 200 mm to serve as a reference for the UR5
behavior. A numerical analysis of the collected data indicates that the robot
is able to achieve a mean latency of about 16 ms over the test trajectory.

###  IV-C Coarse Motion Correction

As the UR3 executed the head trajectory discussed in the previous section, the
UR5 performed motion compensation. The trajectories of both robots are
depicted in Fig. 9. Shown on the left are the measured displacements for all
six axes collected from the two robots, and on the right are the differences
between the trajectories. As shown in the first column, the tracking
performance for translations and rotation about the z-axis (roll) is better
than for rotations about the x- and y-axes (yaw and pitch). One explanation is
that the range of motion in pitch and yaw is the smallest, and due to the
averaging filter, the robot would not compensate for such small motions. In
addition, our previous evaluation of the string encoder system [4] showed that
there are slightly higher measurement inaccuracies for pitch and yaw.
Nonetheless, the second column of Fig. 9 does not show significant discrepancy
across the trajectory differences in all rotation axes.

Figure 9: Comparison between UR5 (robot) position and UR3 (head) position for
recorded head motion, with actual motion trajectories (left) and differences
between the two (right). Horizontal axes represent time in seconds.

For the entire travel, the norm of translational tracking error in the x-y
plane has a maximum of 14.5 mm and a mean of 5.8 mm. The error in the
z-direction is less relevant as our proposed 18 mm clearance between the
helmet and the imaging ring is within the x-y plane. Nonetheless, even
considering the z-error, the maximum is 15.7 mm with a mean of 6.6 mm, which
is still within the stated clearance. The rotational tracking error has a
maximum of 3.8 deg and a mean of 1.1 deg. Since this rotation is measured with
respect to the head center, it would not contribute to the available clearance
if we assume the head being spherical.

Furthermore, the UR5 is able to maintain a typical lag of about 55 ms (and up
to 80 ms in some cases) while compensating for the head motion. The typical
lag is approximately the sum of the string encoder measurement latency
discussed in Section IV-A (4 ms), the filtering latency described in Section
II-B (15 ms), and the robot motion latency measured in Section IV-B (16 ms).

###  IV-D Fine Motion Correction

After applying fine motion correction with the string encoder measurements,
any nonzero position p of the laser intersection relative to its starting
position as seen from the cameras should ideally be corrected to zero. The
norms of these position vectors after correction are plotted in Fig. 10.

Figure 10: Norm of uncorrected distance after fine motion correction.

These uncorrected distances have a mean of 0.51 mm and a standard deviation of
0.28 mm, which is in line with the desired 0.5 mm accuracy. Part of the
inaccuracy originates from the string encoder system, which we previously
evaluated to have an accuracy of less than 0.3 mm for a $\pm 10$ mm range of
translational motion [4]. The cameras also possess inaccuracy in localizing
the red dot, and in our setup, each pixel approximately corresponds to 0.1 mm
of distance. In addition, the camera exposure time for a single frame is
around 15 ms, hence images captured during motion show a trail of red light
instead of a single dot. We tried to identify the end of the trail and relate
that position to the time the image was taken for synchronization purposes.
However, this approach becomes more complicated in regions where there is a
direction change in the relative motion between the helmet and imaging ring.
Therefore, some of the inaccuracy is due to our camera setup, rather than the
proposed robotic system.

##  V Conclusions

We presented the design of a prototype robotic system that maintains a
specified position offset with respect to a human head, based on measurements
from string encoders connected in parallel between the robot and a helmet
attached to the head. The intended use is for PET imaging during natural
activities, such as walking, to support neuroscience research, but the concept
could be applied to other scenarios where it is desirable to have a robot
system follow the motion of a human without significant physical contact.
Technically, the string encoders create a small amount of physical contact
between the robot and human, and the spring-loaded tensioning of the strings
causes a small force to be exerted on the head. This disadvantage, however, is
offset by the high reliability of the mechanical connection and the lower
latency and higher accuracy that can be achieved compared to non-contact
alternatives such as proximity sensing and optical tracking. In this
application, accurate measurement of the residual motion is important because
it enables motion correction during PET image reconstruction.

Our results demonstrate that a UR5 robot, with hardware and software designed
to minimize latency to less than 100 ms, can provide adequate motion
correction. Specifically, the system can keep residual motion within the
designed 18 mm clearance between the PET imaging ring and helmet [3] and the
string encoder system’s dynamic accuracy enables sub-millimeter motion
correction during PET image reconstruction. The final design, however, will
require a robot that can support the higher weight of a PET imaging ring
(15-20 kg) and it is desirable for the string encoder measurement system
accuracy to be better than 0.5 mm.

Future work includes improvements to the measurement accuracy, as discussed
above. In addition, the presented robot controller is purely reactive, based
on measured head position and velocity, and performance could be improved by
adopting predictive control [6]. Our recorded data indicates that human head
motions are highly periodic during locomotion, which suggests the feasibility
of this approach. Finally, development of a system for neuroscience research
with human subjects will require a significant effort in risk analysis and
safety design, as well as evaluation of the human trust of, and comfort with,
a robot system in close physical proximity.

## Acknowledgements

Stan Majewski initiated the investigation of robotic compensation for head
motion during PET imaging. The recorded head motion data was provided by
Caroline Paquette. Ti Wu provided the kinematic design of the string encoder
system.

## References

  * [1] S. Majewski, “The path to the “ideal” brain PET imager: The race is on, the role for TOF PET,” _Il Nuovo Cimento C_ , vol. 43, no. 1, pp. 1–35, 2020.
  * [2] S. Majewski, J. Proffitt, J. Brefczynski-Lewis, A. Stolin, A. Weisenberger, W. Xi, and R. Wojcik, “HelmetPET: A silicon photomultiplier based wearable brain imager,” in _IEEE Nuclear Science Symposium and Medical Imaging Conference Record_ , Valencia, Spain, Oct. 2011, pp. 4030–4034.
  * [3] Y. Liu, T. Wu, I. I. Iordachita, C. Paquette, and P. Kazanzides, “Analysis of human head motion and robotic compensation for PET imaging studies,” in _IEEE Engineering in Medicine & Biology Conf. (EMBC)_, 2021, pp. 4836–4839.
  * [4] J. Wang, T. Wu, I. Iordachita, and P. Kazanzides, “Calibration and evaluation of a motion measurement system for PET imaging studies,” _Journal of Medical Robotics Research_ , vol. 08, no. 01n02, 2023.
  * [5] J. Mainprice and D. Berenson, “Human-robot collaborative manipulation planning using early prediction of human motion,” in _IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , Tokyo, Japan, Nov 2013, pp. 299–306.
  * [6] H. S. Koppula and A. Saxena, “Anticipating human activities using object affordances for reactive robotic response,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 38, no. 1, 2016.
  * [7] S. Haddadin, A. D. Luca, and A. Albu-Schäffer, “Robot collisions: a survey on detection, isolation, and identification,” _IEEE Trans. Robot._ , vol. 33, no. 6, 2017.
  * [8] S. Haddadin, A. Albu-Schäffer, and G. Hirzinger, “Requirements for safe robots: measurements, analysis and new insights,” _The International Journal of Robotics Research_ , vol. 28, no. 11-12, 2009.
  * [9] D. D. Bellamy, G. Chance, P. Caleb-Solly, and S. Dogramadzi, “Safety assessment review of a dressing assistance robot,” _Frontiers in Robotics and AI_ , vol. 8, 2021.
  * [10] “FPGA1394,” https://github.com/jhu-cisst/FPGA1394, accessed 2023-03-01.
  * [11] P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio, “An open-source research kit for the da Vinci® surgical system,” in _IEEE Intl. Conf. on Robotics and Auto. (ICRA)_ , Hong Kong, China, Jun 2014, pp. 6434–6439.
  * [12] “Mechatronics Firmware,” https://github.com/jhu-cisst/mechatronics-firmware, accessed 2023-03-01.
  * [13] J. Y. Wu, Z. Chen, A. Deguet, and P. Kazanzides, “FPGA-based velocity estimation for control of robots with low-resolution encoders,” in _IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , Madrid, Spain, Oct 2018, pp. 6384–6389.
  * [14] “The _cisst_ libraries,” https://github.com/jhu-cisst/cisst, accessed 2023-03-01.
  * [15] “sawUniversalRobot,” https://github.com/jhu-saw/sawUniversalRobot, accessed 2023-03-01.
  * [16] T. T. Andersen, “Optimizing the Universal Robots ROS driver,” Technical University of Denmark, Department of Electrical Engineering, Tech. Rep., 2015\.


# Automatic Knowledge Augmentation for Generative Commonsense Reasoning

Jaehyung Seo
Korea University
<EMAIL_ADDRESS>
&Chanjun Park
Korea University
<EMAIL_ADDRESS>
&Sugyeong Eo
Korea University
<EMAIL_ADDRESS>
&Hyeonseok Moon
Korea University
<EMAIL_ADDRESS>
&Heuiseok Lim†
Korea University
<EMAIL_ADDRESS>

###### Abstract

Generative commonsense reasoning is the capability of a language model to
generate a sentence with a given concept-set that is based on commonsense
knowledge. However, generative language models still struggle to provide
outputs, and the training set does not contain patterns that are sufficient
for generative commonsense reasoning. In this paper, we propose a data-centric
method that uses automatic knowledge augmentation to extend commonsense
knowledge using a machine knowledge generator. This method can generate semi-
golden sentences that improve the generative commonsense reasoning of a
language model without architecture modifications. Furthermore, this approach
is a model-agnostic method and does not require human effort for data
construction.

##  1 Introduction

CommonGen [9] is a challenge in which the task is to generate a sentence that
makes sense with an input concept-set by leveraging generative commonsense
reasoning. However, it is difficult to derive much commonsense knowledge for
the given concept-sets, such as compositional generalization and relational
reasoning, from the 67,389 sentences in the CommonGen training set [6, 5].
Additionally, most reference sentences consist of superficial scene
descriptions and do not represent any specific relational knowledge between
the concepts [9].

To address these issues, we propose a data-centric approach that augments
prior knowledge with an additional pre-training session. Our method uses a
trained generator that produces semi-golden sentences as patterns to guide
generative commonsense reasoning [8, 16]. As the generator, we utilize BART
[7], which has the highest average percentage of input concepts that exist in
the output sentences in the CommonGen task ($97.35\%$). The generator learns
to reconstruct sentences using concepts (e.g., verbs and nouns) extracted from
sentences in Wikipedia articles, which have high coverage. Semi-golden
sentences with concrete descriptions about the input concept-set or concept-
pair set are used in an additional pre-training session to enhance the prior
knowledge of the generative language model.

Automatic knowledge generation substantially reduces human effort and ensures
the quality of machine-generated data using the results of performance
improvements. It can also extend the relational information of given concept-
sets, including commonsense knowledge for the intended input query. Moreover,
the improvements in performance in auto-regressive models (i.e., a decoder
only) [12] and sequence-to-sequence language model [13] (i.e., an
encoder–decoder), which have different architectures, demonstrate the strong
generalization and model-agnostic nature of this approach.

Table 1: Experimental results and statistics for each data augmentation used
in pre-training. “Pair” and “Set” respectively indicate to the semi-golden
sentences for concept-pairs and concept-sets in the CommonGen training set; “#
sentences” and “# concept-sets” respectively refer to the number of semi-
golden sentences and number of concepts with respect to the size of the
concept-set.

| Results | Augmentation
---|---|---
| ROUGE-L | BLEU4 | METEOR | CIDEr | SPICE | # sentences | # concept-sets
GPT2 | 39.28 | 21.10 | 26.20 | 12.15 | 25.90 | - | -
GPT2+Pair | 49.70 | 23.10 | 26.90 | 12.62 | 27.20 | 59,125 | 2(59,125)
GPT2+Set | 49.20 | 22.30 | 26.80 | 12.60 | 27.10 | 32,471 | 3(24,891), 4(4,206), 5(3,374)
GPT2+Pair+Set | 49.50 | 23.50 | 27.20 | 12.95 | 27.60 | 91,596 | 2(59,125), 3(24,891), 4(4,206) 5(3,374)
T5 | 49.27 | 28.60 | 30.10 | 14.96 | 31.60 | - | -
T5+Pair | 56.90 | 33.70 | 33.00 | 17.72 | 33.80 | 59,125 | 2(59,125)
T5+Set | 57.20 | 34.40 | 33.00 | 17.97 | 33.90 | 32,471 | 3(24,891), 4(4,206), 5(3,374)
T5+Pair+Set | 56.80 | 33.40 | 32.90 | 17.76 | 33.80 | 91,596 | 2(59,125), 3(24,891), 4(4,206) 5(3,374)

##  2 Data Construction Process and Results

In this section, we present the process of data construction and experiments
using a generator.

First, we extracted 154,892 sentences from a Wikipedia data dump (2021-03-01)
that match more than two input concepts via ElasticSearch [4] and BM25 [14,
3]. From the extracted sentences $y$, we defined concept-set $C$, which
includes a verb or noun through part-of-speech (POS) tagging. Second, the
generator was trained with the reconstructed data using $C=y$ and produced a
semi-golden sentence $y^{\prime}$ for an input concept-set $C^{\prime}$.
Third, 59,125 concept-pairs and 32,471 unique concept-sets (of size 3 to 5)
were extracted from the CommonGen training set and the generator used them as
an input sequence to create a semi-golden sentence. Fourth, extracted concept-
sets and generated semi-golden sentences were used as pre-training data in the
form of $C^{\prime}=y^{\prime}$ for two generative language models, as
presented in Table 1. We conducted a case study to demonstrate the impact of
our method with GPT2 [12] and T5 [13]. To evaluate the performance, we used
the n-gram overlapping metrics BLEU [11], ROUGE [10], and METEOR [2] as well
as the concept matching metrics CIDEr [15] and SPICE [1].

As shown in Table 1, all data augmentation cases lead to performance
improvements with respect to all evaluation metrics. The results also indicate
that there is a difference in performance depending on the combination of
input concepts and target models. Augmented concept-pairs include concrete
relational knowledge between two concepts and concept-sets have commonsense
knowledge for generating sentences by compositionality. GPT2 obtains the best
performance improvement when learning both types of machine-generated
knowledge through the augmented dataset. In contrast, T5 yields no significant
performance difference depending on the form of the augmented dataset,
although learning commonsense knowledge with compositionality is slightly more
effective. This result shows that there is a variance in the degree of
improvement in prior knowledge in a specific area when the model architecture
and dataset used for pre-training each language model differ. This
experimental result demonstrates that our data-centric approach generates
better semi-golden sentences for use as an augmented dataset. It is also
possible to produce a machine-generated dataset in the intended format to
compensate for the insufficient prior knowledge of each model. Furthermore,
this approach is a model-agnostic method and has a positive impact on both
auto-regressive and sequence-to-sequence architectures.

##  3 Conclusion

We demonstrated how to use natural language generation for data-centric
research to reduce the cost of leveraging information retrieval and building a
human-annotated dataset. Automatic knowledge augmentation for generative
commonsense reasoning is also not limited to a particular model or dataset,
but can be applied in different forms as needed. In future work, this method
will be highly valuable because it can be extended to other downstream tasks
that require commonsense knowledge as supporting evidence.

## Acknowledgment

This research was supported by the MSIT(Ministry of Science and ICT), Korea,
under the ITRC(Information Technology Research Center) support
program(IITP-2018-0-01405) supervised by the IITP(Institute for Information &
Communications Technology Planning & Evaluation) and IITP grant funded by the
Korea government(MSIT) (No. 2020-0-00368, A Neural-Symbolic Model for
Knowledge Acquisition and Inference Techniques) and Basic Science Research
Program through the National Research Foundation of Korea(NRF) funded by the
Ministry of Education(NRF-2021R1A6A1A03045425). Heuiseok Lim† is a
corresponding author.

## References

  * Anderson et al. [2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  Spice: semantic propositional image caption evaluation.  In _European Conference on Computer Vision (14th: 2016)_ , pages 382–398. Springer, Springer Nature.
  * Banerjee and Lavie [2005] Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Chen et al. [2017] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879.
  * Divya and Goyal [2013] Manda Sai Divya and Shiv Kumar Goyal. 2013.  Elasticsearch: An advanced and quick search technique to handle voluminous data.  _Compusoft_ , 2(6):171.
  * Keysers et al. [2019] Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. 2019.  Measuring compositional generalization: A comprehensive method on realistic data.  In _International Conference on Learning Representations_.
  * Lake and Baroni [2018] Brenden Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _International Conference on Machine Learning_ , pages 2873–2882. PMLR.
  * Lewis et al. [2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lewis et al. [2020] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Lin et al. [2020] Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020.  Commongen: A constrained text generation challenge for generative commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 1823–1840.
  * LIN [2004] Chin-Yew LIN. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, Barcelona, Spain_ , pages 74–81.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting on association for computational linguistics_ , pages 311–318. Association for Computational Linguistics.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Svore and Burges [2009] Krysta M Svore and Christopher JC Burges. 2009.  A machine learning approach for improved bm25 retrieval.  In _Proceedings of the 18th ACM conference on Information and knowledge management_ , pages 1811–1814.
  * Vedantam et al. [2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575.
  * Yang et al. [2020] Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020\.  G-daug: Generative data augmentation for commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 1008–1025.