# PlotMap: Automated Layout Design for Building Game Worlds

Yi Wang  0000-0002-5394-5548 Autodesk Research, USAUSA<EMAIL_ADDRESS>,
Jieliang Luo  0000-0002-3298-9111 Autodesk Research, USAUSA
<EMAIL_ADDRESS>,  Adam Gaier  0000-0002-4632-0929 Autodesk Research,
GermanyGermany<EMAIL_ADDRESS>,  Evan Atherton  0009-0001-4234-6881
Autodesk Research, USAUSA<EMAIL_ADDRESS>and  Hilmar Koch
Autodesk Research, USAUSA<EMAIL_ADDRESS>
###### Abstract.

World-building, the process of developing both the narrative and physical
world of a game, plays a vital role in the game’s experience. Critically-
acclaimed independent and AAA video games are praised for strong world-
building, with game maps that masterfully intertwine with and elevate the
narrative, captivating players and leaving a lasting impression. However,
designing game maps that support a desired narrative is challenging, as it
requires satisfying complex constraints from various considerations. Most
existing map generation methods focus on considerations about gameplay
mechanics or map topography, while the need to support the story is typically
neglected. As a result, extensive manual adjustment is still required to
design a game world that facilitates particular stories. In this work, we
approach this problem by introducing an extra layer of plot facility layout
design that is independent of the underlying map generation method in a world-
building pipeline. Concretely, we present a system that leverages
Reinforcement Learning (RL) to automatically assign concrete locations on a
game map to abstract locations mentioned in a given story (plot facilities),
following spatial constraints derived from the story. A decision-making agent
moves the plot facilities around, considering their relationship to the map
and each other, to locations on the map that best satisfy the constraints of
the story. Our system considers input from multiple modalities: map images as
pixels, facility locations as real values, and story constraints expressed in
natural language. We develop a method of generating datasets of facility
layout tasks, create an RL environment to train and evaluate RL models, and
further analyze the behaviors of the agents through a group of comprehensive
experiments and ablation studies, aiming to provide insights for RL-based plot
facility layout design. We will release a dataset containing 10,000 tasks, the
RL environment, and the RL models.

Reinforcement Learning, World Building, Plot Facility Layout Design

††submissionid: 911 Figure 1. We derive spatial constraints from a story and
use Reinforcement Learning to layout locations mentioned in the story on a map
to satisfy the constraints.

##  1\. Introduction

Landscapes in video games serve as more than just scenic backdrops; they
interact intimately with the unfolding narrative, defining and shaping the
player’s experience. This interaction is pivotal, underpinning the player’s
journey. By giving game designers tools for better supporting narratives with
game map design, they can create games that are more cohesive and immersive
for the player.

Designing game maps is difficult, as it requires designers to consider varied
qualities such as realistic topography (Smelik et al., 2009; Kelly and McCabe,
2017) and game playability (van der Linden et al., 2014) at the same time.
Designing a map that supports a given story adds more constraints, making the
problem even more challenging.

While the need to support an underlying story is typically neglected in most
existing map generation methods, some efforts have been made to develop the
story first, then to codify it as plot points and graphs so that maps can be
generated based on their relations (Valls-Vargas et al., 2013; Hartsook et
al., 2011). However, as (Dormans and Bakkes, 2011) pointed out, the principles
that govern the design of the spatial and the narrative side of the game are
different, and thus these two processes should be independent. Methods for
generating game maps from stories can appear artificially contrived to fit a
narrative, and it is not straightforward to combine these methods with those
that also take into account game design and geographical considerations.

As a result, designing a game world that facilitates a story requires
extensive manual modification; and as the number of constraints scale, the
challenge of designing a map that satisfies all of the constraints of the
story can become intractable, if not impossible, for a designer to do by hand
(Matsumoto, 2022).

In this work, we approach this problem by introducing an extra layer of plot
facility layout design that is independent to the underlying map generation
method in a world-building pipeline.

Our work is inspired by the philosophy behind (Dormans and Bakkes, 2011),
which distinguishes the abstract space defined by the story (referred to as
missions) and the concrete space defined by the actual geometric layout of a
game map. The story is accommodated by mapping the former into the latter.
While (Dormans and Bakkes, 2011) focuses on action adventure games with
discrete “scenes” connected by “passages”, we impose very little assumption on
methods used for story and map generation, and in particular target workflows
for modern open world games.

We introduce the concept of plot facilities, which are abstract locations
mentioned in the given story. A set of constraints are derived from the story
in terms of the spatial relationships between plot facilities and elements in
the underlying map. Given an underlying map, we arrange the layout of the plot
facilities on top of the map to satisfy the constraints. Our method is
compatible with any map generation technique in the sense that the underlying
map can be hand-crafted, procedurally generated, or even from a Geographic
Information System (GIS) such as Google Maps.

Figure 2 illustrates our world building approach. The focus of this work is on
the plot facility layout design task. However, to demonstrate a concrete
pipeline, we work with a specific procedural map generation method described
in Section 4.1, and in an end-to-end example we show in Section 5.3, we
extract story constraints from a free-text story description using a large
language model.

Figure 2. Accommodating a story on a map with a plot facility layout design
process

We present a system that leverages Reinforcement Learning (RL) to
automatically assign geometric locations on a game map to plot facilities
following geographic and spatial constraints derived from the story – such as
being in a forest and far from another plot facility. A decision-making agent
moves the plot facilities around, considering their relationship to the map
and each other, to locations on the map that best satisfy the constraints of
the story. Our system considers input from multiple modalities: map images as
pixels, facility locations as real values, and story constraints expressed in
natural language. We demonstrate that an RL approach to plot facility layout
design is fast in providing solutions, and can quickly adapt to user
intervention, making it suitable for real-time human-AI co-design workflow.

The paper also presents a dataset of facility layout tasks and a Gym-like
environment to train and evaluate the RL models. The dataset contains 10,000
plot facility layout design tasks involving 12 types of spatial constraints
and maximum 10 plot facilities each task, based on a set of procedurally
generated terrain maps. We report the results of applying different strategies
to address the observation with an in-depth discussion.

In summary, the paper’s contribution is threefold:

  * •

We propose plot facility layout design as a novel approach to address the
problem of supporting stories with game maps, which is compatible with most
story and map generation methods.

  * •

We provide a dataset of plot facility layout tasks and a Gym-like environment
to train and evaluate the RL models.

  * •

We provide baseline results on an RL approach to plot facility layout design.

To further promote the research of facility layout design for game map
generation, we will release the data, models, and code necessary for training
and evaluation.

##  2\. Related Work

#### Story and Game Map Generation

Though intertwined, the generation of stories and maps are typically
investigated in isolation. A few notable exceptions do tackle them as a single
system. (Hartsook et al., 2011) proposed a story-driven procedural map
generation method where each event in the plot is associated with a location
of a certain environment (e.g., castle, forest, etc.) and a linear plot line
is translated to a constraint of a sequential occurrence of locations with
corresponding environment type. Map generation is formulated as an
optimization problem finding a topological structure of the map balancing
requirements from a realistic game world and player preferences, subject to
plot constraints. (Valls-Vargas et al., 2013) presents a procedural method
that generates a story and a map facilitating the story at the same time. The
problem is again formulated as an optimization problem to find a topological
structure of the map, but based on various metrics not only from map
playability perspective but also the space of possible stories supported by
the spatial structure, subject to the input plot points.

Both (Hartsook et al., 2011) and (Valls-Vargas et al., 2013) generate
rectangular grid-based maps consisting of discrete “scenes” connected by
“passages”. The map structure is widely used in many classic games such as
Rogue ((1980)) and early Zelda series ((1986)). However, many modern RPG games
feature seamless world maps with continuous terrains and very few geographical
barriers for an immersive open world experience, such as Elden Ring ((2022)),
Pokemon Legend: Arceus ((2022)) and The Legend of Zelda: Breath of the Wild
((2017)). (Dormans and Bakkes, 2011) use generative grammar based methods for
both story (mission) generation and map (space) generation. The story elements
are then mapped to spatial elements using heuristics specific to game genre.
Our work also establishes a mapping between narrative and spatial elements.
However, we adopt a more general constraint satisfaction process.

#### Procedural Content Generation

Procedural Content Generation (PCG) has become an essential component in video
games, employed for the algorithmic creation of game elements such as levels,
quests, and characters. The primary objectives of PCG are to enhance the
replayability of games, reduce the burden on authors, minimize storage
requirements, and achieve specific aesthetics (Hendrikx et al., 2013), (Smelik
et al., 2009), (Kelly and McCabe, 2017),(van der Linden et al., 2014). Game
developers and researchers alike utilize methods from machine learning,
optimization, and constraint-solving to address PCG problems (Togelius et al.,
2011). The primary aim of this work is to train RL agents capable of
generalizing across a wide range of environments and constraints. To achieve
this goal, we employ a PCG approach to generate a diverse set of maps and
constraints to train and test our RL agents.

#### RL in Modern Video Games

The popularity of games in AI research is largely attributed to their
usefulness in the development and benchmarking of reinforcement learning (RL)
algorithms (Bellemare et al., 2013; Jaderberg et al., 2019; Vinyals et al.,
2019; Berner et al., 2019). However, overfitting remains a pervasive issue in
this domain, as algorithms tend to learn specific policies rather than general
ones (Zhang et al., 2018). To counteract overfitting and facilitate policy
transfer between different environments, researchers have turned to PCG
techniques (Risi and Togelius, 2020; Team et al., 2021; Baker et al., 2019).

On the other hand, RL can also be used as a design tool for modern games,
especially for accessing and testing games. (Iskander et al., 2020) develops
an RL system to play in an online multiplayer game alongside human players.
The historical actions from the RL agent can contribute valuable insights into
game balance, such as highlighting infrequently utilized combination actions
within the game’s mechanics. (Bergdahl et al., 2020) uses RL as an augmenting
automated tool to test game exploits and logical bugs. (Chen et al., 2023)
releases a multi-agent RL environment to study collective intelligence within
the context of real-time strategy game dynamics. To the best of our knowledge,
this is the first instance that uses a learning-based approach to accommodate
stories on game maps.

##  3\. Problem Formulation

###  3.1. Overview

Put simply, we would like to assign every location mentioned in a narrative to
an appropriate location on the game map. Inspired by (Dormans and Bakkes,
2011), we view the narrative and the geometric layout of a game map as
independent of each other, except that the geometric layout should accommodate
the narrative. We introduce the notion of plot facilities, which are
conceptual “locations” mentioned in the story. These “locations” are abstract
in the sense that they don’t correspond to any concrete geometric locations
(yet). For example, the event “the hero finds an injured dwarf in the forest”
happens at some place. There can be multiple locations on the map where this
“abstract location” can be “instantiated”, as long as it does not contradict
with the story - in this example it should be inside a forest area.

A set of constraints can be derived from the story for determining whether a
particular instantiation of plot facilities is valid. The set of all plot
facilities and the constraints form a conceptual space defined by the story,
which is at a higher abstraction level than a concrete game map. The problem
is then to assign geometric locations on the map to the plot facilities such
that the constraints are satisfied - we call this problem plot facility layout
design. A plot facility layout is essentially a mapping between the conceptual
space defined by the story and the geometric space defined by the game map.

In the following subsections, we describe our RL-based method for this
problem. To demonstrate a concrete map generation pipeline, in this study we
specifically work with terrain maps consisting of polygons, where each polygon
is assigned a biome.

###  3.2. Plot Facility Layout Design

We define a (facility layout) task as a tuple

$\langle\mathcal{F},\mathcal{T},\mathcal{C}\rangle\vspace{-3mm}$

where

  * •

$\mathcal{F}$ is the set of (plot) facilities. Each facility has an identifier
to be referred to by the constraints.

  * •

$\mathcal{T}$ is a set of polygons on the map, each associated with a biome
type (e.g., OCEAN, PLAINS, etc.).

  * •

$\mathcal{C}$ is a set of spatial constraints over facilities in $\mathcal{F}$
and terrain types, each associated with a natural language utterance, such as
“Fordlen Bay and Snapfoot Forest are separated by the ocean.”

An (optimal) solution to a task is an assignment of coordinates to all the
facilities in $\mathcal{F}$, so that a maximum number of the constraints in
$\mathcal{C}$ are satisfied considering their relations with each other and
the polygons in $\mathcal{T}$. Our goal is to train a general RL agent to be
able to find solutions to any arbitrary task.

###  3.3. RL Formulation

Essentially, each plot facility layout design task can be viewed as
sequentially moving a set of plot facilities $\mathcal{F}$ on a map, thus we
define the plot facility layout design as a sequential Markov decision process
(MDP), which consists of the following key elements:

  * •

Each state $s\in S$ consists of three modalities: 1) a pixel-based image
representing the map, 2) a real-valued vector representing essential
information of the plot facilities, and 3) a group of utterances representing
the constraints. We explore three strategies to derive the embeddings from the
image and the utterances, resulting in three state dimensions: 1782, 4422, and
5702. More details about the state will be described in Section 5.1.

  * •

Each action $a\in A$ is a 2 dimensional vector of real-valued $[\Delta
x,\Delta y]$ for one plot facility. In each round, plot facilities are moved
one at a time in a fixed order, with the state and rewards updated after each
movement.

  * •

The reward (for each step) $r_{t}$ is $+1$ when all the constraints are
satisfied, and is the average satisfaction score from all the constraints
minus $1$ when partial constraints are satisfied:

$r_{t}=\left\\{\begin{array}[]{ll}1,&\text{if all constraints are
satisfied}\\\
\frac{1}{n}\sum_{i=1}^{n}s_{i}-1,&\text{otherwise}\end{array}\right.$

where $n$ is the number of constraints and $s_{i}$ is the satisfaction score
for each constraint. The satisfaction score for each type of constraint is
within $[0,1]$ and is defined based on hand crafted heuristic function
determining to what extent the facility layout forms the corresponding
geometric formation. 111E.g., closeTo(x,y) is negatively correlated to the
distance between x and y, and reaches 1 when their distance is less than a
certain threshold. The range of the reward $r_{t}$ is $[-1,0]\cup\\{1\\}$.

  * •

The transition function is deterministic, where $s_{t+1}=f(s_{t},a_{t})$.

  * •

Each episode is terminated when all the constraints are satisfied or at 200
timesteps.

We train an RL agent to learn an optimal policy $\pi_{\theta}$ in order to
maximize the expected discounted rewards:

(1)
$\max_{\pi_{\theta}}\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t=0}^{T}\gamma^{t}r(s_{t},a_{t})\right],$

where trajectory $\tau=(s_{0},a_{0},s_{1},a_{1},...,s_{T},a_{T})$, $\theta$ is
the parameterization of policy $\pi$, and $\gamma$ is the discounted factor.

##  4\. Task Dataset Generation

For training the RL model, we generated a dataset of $10,000$ facility layout
tasks. Each task requires arranging the layout of maximum $10$ plot facilities
on top of a procedurally generated map, w.r.t. a set of maximum $10$ spatial
constraints. Maps consist of $9$ biome types and the constraints are generated
based on $12$ constraint types. On average, a random agent has around $30\%$
success rate to solve a single task.

###  4.1. Map Generation

We employ a procedural map generation approach adapted from the work of Patel
(2010) and its implementation by Dieu et al. (2023). Rather than beginning
with basic elements, such as elevation and moisture levels, and subsequently
deducing coastlines and river locations, this method begins by creating rivers
and coastlines and then adapts elevations and moisture to render them more
plausible. The procedure is divided into several steps:

  1. (1)

A grid of Voronoi polygons is generated from random points with varying
density in [0,1]2 space. We then replace each point with the centroid of its
polygon region and use Lloyd relaxation(Lloyd, 1982) to ensure an even
distribution.

  2. (2)

Coastline generation employs a flooding simulation. We initially mark edges
touching the map borders and those in random proximate areas as water edges.
Flooding continues until a desired number of water edges are generated. Then,
we select random inland edges as lake edges and continue flooding. Tiles are
designated as water if their borders exceed a predefined minimum water ratio.
Four terrain types are assigned: ocean, coast, lake, and land, based on their
relation to water edges and neighboring tiles.

  3. (3)

Elevation assignment is determined by the distance from the coast, with
elevation normalized to a given height distribution, resulting in fewer high
points and smoother terrain.

  4. (4)

Rivers are created along the downhill path from randomly selected mountain
corners to the nearest lake or ocean.

  5. (5)

Moisture levels are assigned according to the distance from freshwater
sources, such as lakes and rivers.

  6. (6)

Biome assignment for each polygon depends on the combination of moisture and
elevation, as illustrated in the Whittaker diagram in Figure 8.

For dataset generation, we produce 100 maps with 1,000 cells each. These maps
are then converted to RGB images, suitable for input to neural network-based
reinforcement learning agents.

###  4.2. Constraint Generation

Synthetic facility layout tasks are generated by associating a set of random
constraints to randomly sampled maps. In this work, we consider a list of $12$
constraint types, listed in Table 1, along with their number of occurrences in
the $10,000$ task dataset. A constraint type
$ConstraintType(b_{1},\dots,b_{m},p_{1},\dots,p_{n})$ is instantiated to
become a constraint by substituting each of $b_{1},\dots,b_{m}$ with a biome
type, and each of $p_{1},\dots,p_{m}$ with a plot facility id ($m\geq 0,n\geq
0$).

For each constraint type, we define a heuristic function for evaluating an
existing facility layout w.r.t. any instantiation of the constraint type. The
function returns a real number in $[0.0,1.0]$ with $1.0$ meaning fully
satisfied and $0.0$ completely not satisfied. These functions are used to
check if the randomly generated constrants are satisfied by a random layout.
They are also used in Section 3.3 for computing the reward. Tasks are then
generated following Algorithm 1. Note that, as the constraints are extracted
from an example layout, this procedure guarantees that the generated tasks are
solvable.

Table 1. Constraint types included in the $10,000$-task dataset (each $p_{i}$ represents a plot facility) Constraint Types | Meaning | Frequency
---|---|---
$\texttt{AcrossBiomeFrom}(b_{1},p_{1},p_{2})$ |  $p_{1}$ is across biome $b_{1}$ from $p_{2}$ | 24895
$\texttt{Outside}(b_{1},p_{1})$ |  $p_{1}$ is outside biome $b_{1}$ | 5090
$\texttt{Inside}(b_{1},p_{1})$ |  $p_{1}$ is inside biome $b_{1}$ | 1117
$\texttt{AwayFrom}(b_{1},p_{1})$ |  $p_{1}$ is away from biome $b_{1}$ | 773
$\texttt{CloseTo}(b_{1},p_{1})$ |  $p_{1}$ is close to biome $b_{1}$ | 2362
$\texttt{ToTheSouthOf}(b_{1},p_{1})$ |  $p_{1}$ is to the south of biome $b_{1}$ | 739
$\texttt{ToTheNorthOf}(b_{1},p_{1})$ |  $p_{1}$ is to the north of biome $b_{1}$ | 746
$\texttt{ToTheWestOf}(b_{1},p_{1})$ |  $p_{1}$ is to the west of biome $b_{1}$ | 805
$\texttt{ToTheEastOf}(b_{1},p_{1})$ |  $p_{1}$ is to the east of biome $b_{1}$ | 727
$\texttt{CloseTo}(p_{1},p_{2})$ |  $p_{1}$ is close to $p_{2}$ | 103
$\texttt{AwayFrom}(p_{1},p_{2})$ |  $p_{1}$ is away from $p_{2}$ | 4215
$\texttt{InBetween}(p_{1},p_{2},p_{3})$ |  $p_{1}$ is between $p_{2}$ and $p_{3}$ | 416
$\texttt{OnSouth}(p_{1})$ |  $p_{1}$ is on the south of the map | 283
$\texttt{OnNorth}(p_{1})$ |  $p_{1}$ is on the north of the map | 295
$\texttt{OnEast}(p_{1})$ |  $p_{1}$ is on the east of the map | 291
$\texttt{OnWest}(p_{1})$ |  $p_{1}$ is on the west of the map | 285
$\texttt{ToTheSouthOf}(p_{1},p_{2})$ |  $p_{1}$ is to the south of facility $p_{2}$ | 2774
$\texttt{ToTheNorthOf}(p_{1},p_{2})$ |  $p_{1}$ is to the north of facility $p_{2}$ | 2715
$\texttt{ToTheWestOf}(p_{1},p_{2})$ |  $p_{1}$ is to the west of facility $p_{2}$ | 2775
$\texttt{ToTheEastOf}(p_{1},p_{2})$ |  $p_{1}$ is to the east of facility $p_{2}$ | 2779
$\texttt{VisibleFrom}(p_{1},p_{2})$ |  $p_{2}$ is visible from $p_{1}$ | 616

##  5\. Experiments & Analysis

###  5.1. Experiment Setup

#### State Space in Details

Our state space includes $3$ types of inputs:

  * •

the map is represented by a pixel-based image defined as (42 x 42 x 3).

  * •

the constraints are represented by natural language utterances or one-hot
vectors, depending on the embedding strategies described in the following
paragraph.

  * •

Each plot facility’s information is represented by a vector, consisting of its
position $[x,y]$ on the map, a binary motion indicator signifying if it is its
turn to move or not, and a unique identifier.

#### RL Training Details

All of our policies are trained using Proximal Policy Optimization (PPO)
(Schulman et al., 2017). The details of the training hyper-parameters are in
Appendix D. To handle the multi-modal observation space, we employ pre-trained
models to independently extract embeddings from the map (image) and constraint
(natural language) inputs. These embeddings are subsequently concatenated with
the informational vector and provided as inputs to the policy network.
Specifically, we design three strategies for deriving the embeddings:

  * •

NL-based: using ResNet (He et al., 2016) for maps and SentenceTransformer
(Reimers and Gurevych, 2019) for constraints.

  * •

CLIP-based: using CLIP (Radford et al., 2021) for both maps and constraints.

  * •

Relation-based: using ResNet for maps, and each constraint is encoded as a
one-hot vector, representing the constraint type, followed by three one-hot
vectors indicating the specific plot facilities to instantiate the constraint
with.

In all of our experiments, we have set a limit of 10 for both the number of
plot facilities and the number of constraints. The breakdown of the state
dimension is listed in Appendix C.

Input: A set of map $MAP$, maximum number of facilities $N$, a set of
constraint types $\mathcal{CT}$, minimum and maximum number of constraints
$M_{1}$ and $M_{2}$

Output: A facility layout task $\langle F,T,C\rangle$

1\. Randomly sample a map $T$ from $MAP$;

2\. Randomly assign a location to facilities $obj_{1}\dots obj_{N}$ on the map
$T$;

3\. For each constraint type in $\mathcal{CT}$, generate all possible
instantiations of it w.r.t. $obj_{1}\dots obj_{N}$ and biome types. Evaluate
each of them against the current map, adding the true ones to a set
$C^{\prime}$ ;

4\. Sample a set of statements from $C^{\prime}$ sized between $M_{1}$ and
$M_{2}$, obtaining $C$;

5\. For each statement in $C$, use large language model such as GPT (Brown et
al., 2020) to rephrase it with a natural language sentence, resulting in a set
of NL utterances $C^{NL}$;

6\. return task $\langle\\{obj_{1}\dots obj_{N}\\},T,C^{NL}\rangle$.

ALGORITHM 1 Facility Layout Task Generation

#### Simulated Concurrent Movement

A truly concurrent movement scheme updates the observation after all
facilities move, which makes it difficult to contribute a change in constraint
satisfaction to an individual facility’s movement. Our exploratory experiments
also show that this results in undesired behaviors such as livelocks. For
example, two plot facilities on the same side of a lake that are targeting
being across the lake from each other might both move simultaneously to the
other side of the lake, resulting in them still being on the same side of the
lake as one another. Therefore, we simulate concurrent movement with a turn-
based movement scheme at micro-level: at each round, facilities are moved one
by one, each with a maximum length of movement, with observation updating
after each facility is moved; At a macro level the movement is concurrent in
the sense that all of the facilities make progress each round (see Appendix
B).

###  5.2. Quantitative Results

Table 2. Success rate (%) comparison among three proposed baseline methods and a random agent across various task sizes. All training procedures in this table have a limit of 200,000 steps. Method | Success Rate (%) | Success Rate (%)
---|---|---
random initial positions | 100 unseen tasks
1 task | 5 tasks | 50 tasks | 100 tasks |  |  |  |
| Random Agent | 23.9 | 30.3 | 30.5 | 30.9 | 32.2
Baseline |  |  |  |  |  | 1-task policy | 5-task policy | 50-task policy | 100-task policy
NL-based | 84.1 | 48.2 | 38.5 | 38.4 | 34.8 | 33.3 | 35.1 | 38.0
CLIP-based | 99.9 | 49.0 | 38.0 | 35.6 | 32.5 | 30.2 | 36.0 | 35.5
Relation-based | 100.0 | 55.1 | 46.1 | 38.5 | 32.2 | 29.6 | 32.7 | 36.1

We carry out three groups of experiments to investigate different aspects of
the problem. Firstly, we inspect the performance of our three proposed
embedding strategies on four small task sets in Table 2. Secondly, we examine
the generalization of the RL agents on a 10,000-task set in Table 4. Finally,
we study the influence of maps and constraints on generalization in Table 3.

For all of the tables, we evaluate our trained policies on two conditions:
Random-initial-positions refers to the policies evaluated with the same task
sets for training but under different initial positions; 100-unseen-tasks
reports success rates when the policies, each trained on their respective task
set, are tested on 100 unseen tasks. We consider success only when all the
constraints are satisfied, and we calculate the success rates over 1000
rollouts; each rollout takes approximately 5 seconds to finish.

Table 2 demonstrates that all three baselines have outstanding performance on
tackling one single hard task, with CLIP-based and Relation-based methods
markedly outperforming the NL-based. The success rate for each baseline
declines as the task set size increases, with the Relation-based method
consistently surpassing the others in all four sets. The relatively low state
dimension of the Relation-based method might contribute to the success.
However, when deployed on the 100-unseen-task set, all three baselines exhibit
a deficiency in generalization capability. We hypothesize that a large task
set might enhance generalization, and to examine this, we conduct additional
experiments, as reported in Table 4 222Given the similar performance but
longer training time of the CLIP-based method compared to the NL-based method
in Table 2, we opt for the latter in future experiments.. While the success
rates show improvement compared to the policies trained on the 100-task set,
the increase isn’t as significant as anticipated. Notably, the NL-based method
slightly outperforms the Relation-based method, indicating that natural
language embeddings’ contextual information may offer an advantage when
generalizing across diverse constraints. To investigate further the factors
impeding generalization, we study the influence of maps and constraints by
training on three distinct sets of 100 tasks as depicted in Table 3. The
results indicate that constraints pose a greater challenge to generalization,
regardless of the encoding method employed.

Table 3. Success rate (%) comparison among three 100-task datasets, each varying in map and constraint combinations. Each combination is paired with its unique set of 100 unseen tasks. All training procedures in this table have a limit of 500,000 steps. Method | Success Rate (%) | Success Rate (%)
---|---|---
random initial positions | 100 unseen tasks
| Varied Map
---
Varied Constraints
| Same Map
---
Varied Constraints
| Varied Map
---
Same Constraint
| Varied Map
---
Varied Constraints
| Same Map
---
Varied Constraints
| Varied Map
---
Same Constraint
| Random Agent | 30.9 | 34.5 | 38.0 | 32.2 | 32.1 | 35.3
Baseline | NL-based | 38.4 | 42.8 | 79.2 | 38.0 | 39.9 | 70.4
Relation-based | 38.5 | 48.1 | 77.5 | 36.1 | 40.1 | 68.9
Table 4. Success rate (%) comparison among two proposed baseline methods and a random agent on a 10,000 task set. All training procedures in this table have a limit of 2,000,000 steps. Method | Success Rate (%)
---|---
random initial positions | 100 unseen tasks
| Random Agent | 38.7 | 32.2
Baseline | NL-based | 46.1 | 42.4
Relation-based | 44.8 | 36.8

###  5.3. Qualitative Results

In this section, we demonstrate a complete pipeline of the system, and
highlight insightful agent behaviors through specific examples.

Figure 1 illustrates a complete story-to-facility-layout pipeline. From the
story on the left, $8$ plot facilities and $6$ constraints are extracted with
a pre-trained large language model. We then add three additional constraints
to make the task more challenging and show the complete requirements of the
task in Table 5. The map on the right shows the resulting plot facility
layout, along with motion trails indicating traces to the final locations.

Table 5. plot facilities and constraints derived from story in Figure 1.
Veilstead Kingdom and Aquafrost Garrison are on opposing shores
---
Pillar of Hope is away from mountains
Mirestep Swamp is located south of Forgewind Citadel
Hearthfire Hold and Veilstead Kingdom are separated by a lake
Aquafrost Garrison is situated south of Marketown
Veilstead Kingdom is across the lake from Marketown
Fountain of Solace and Forgewind Citadel are positioned across the coast
A great body of water separates Hearthfire hold and Marketown
Hearthfire hold and Aquafrost Garrison are on opposite coasts

In Figure 4, we demonstrate the same story accommodated on 4 different maps by
rolling out a trained RL policy. We observe that the same set of plot
facilities can still maintain their relative spatial relations on completely
different geometric layouts, which aligns with the perspective described in
(Dormans and Bakkes, 2011): “The same mission (story) can be mapped to many
different spaces, and one space can support multiple different missions
(stories)”. This capability enables designers to envision various
interpretations of unspecified story details and potential story progressions.

In Figure 5, we show that our RL policies support accurate and fast re-
adaptation after human intervention. Specifically, after we manually change
the location of Marketown to the northeast part of the map, Veilstead Kingdom
and Hearthfire Hold can adjust their locations to continue to be across the
lake from Marketown, while Aquafrost Garrison stays at the same location, so
all of the constraints are still satisfied. This potentially enables an
interactive design process with a mixed-initiative interface (Yannakakis et
al., 2014; Smith et al., 2010; Cook et al., 2021).

We also observe cooperative behaviors as shown in Figure 3 that Marketown and
Veilstead Kingdom must be across a lake from each other, while Aquafrost
Garrison must be to the south of Marketown. Aquafrost Garrison was initially
to the south of Marketown but as Marketown moves South to be across the lake
from Veilstead Kingdom, Aquafrost Garrison moves even further south to
continue satisfying its south of Marketown constraint.

Figure 3. Cooperative behavior to satisfy constraints

In many cases, we notice that the plot facilities don’t stop even though all
the constraints are satisfied. To investigate this phenomenon, we compare the
motion trails from an RL policy and a random agent, as shown in Figure 6. The
agent presents a strong preference for moving facilities to the edge of the
map without any constraints, which likely could be attributed to our task
dataset’s imbalance. Table 1 shows a significantly higher proportion of
AcrossBiomeFrom constraint. During training, AcrossBiomeFrom is satisfied as
long as there is a biome between two plot facilities. If the two plot
facilities are on different edges of the map, it’s likely that they are across
several different biomes from each other, which renders going to the edge an
effective strategy. On the other hand, for human designers, this type of
constraint usually implies that both facilities are close to the biome (e.g,
“A and B are across a lake” generally implies that A and B are on the shore of
the lake). In this sense, this moving-towards-edges behavior can be seen as a
form of reward hacking. We also visualize the motion trails from the RL policy
at different training stages in Figure 7. At earlier stages, the policy tends
to have more random routes circling back and forth, while the policy at later
stages tends to show a clearer direction of progression. We notice that the
tendency of going to the edge is established at a very early training stage.

In addition to the reward hacking behavior, the imbalance of the dataset also
contributes to several other failure cases. Since the constraints are
generated based on random facility layouts, the probability is low to sample
constraints requiring rare geometric configurations. For example, mountains
usually take up only a very small portion of the map, which means it’s hard to
sample an “X is inside mountain” type of constraint. Similarly, it’s more
likely for two random facilities to be spawned away from each other than close
to each other, which explains why we have significantly more AwayFrom than
CloseTo constraints (see Table 1 ).

##  6\. Limitation and Future Work

The use of handcrafted reward functions present challenges in both accurately
reflecting the preferences of human designers and providing the right signal
for training. A promising solution for better understanding the kinds of
solutions which designers prefer is to apply Reinforcement Learning from Human
Preference, which offers potentially more accurate reflections of human intent
(Christiano et al., 2017). Moreover, we considered the satisfaction of all
constraints as the benchmark for successful task resolution. In practice, a
suboptimal solution that satisfies most of the constraints might be
acceptable, and situations where the constraints are unsatisfiable are
completely possible. In these cases knowledge of designer preferences, or a
mixed-initiative approach which allowed editing of the map, would allow
desirable solutions to be found.

The formulation and scalability of the RL approach, as well as the employed
embedding strategies, emerged as important aspects in our study. The use of a
single RL agent responsible for handling all global information and managing
all plot facilities constituted a potential bottleneck for our approach,
potentially limiting the scalability of the model in larger applications.
Future research could investigate a distributed RL formulation, where each
plot facility is treated as an independent RL agent. This adjustment could not
only increase scalability but also enhance performance. Our embedding
strategies resulted in an excessively high dimensional state space, with
redundant static information in each episode, leading to sample inefficiency
and suboptimal generalization capabilities of the RL agent. These findings
point to the necessity of developing more sophisticated methods that can
better leverage the embedded prior knowledge, consequently improving the
generalization capabilities of the RL agent. Further research in this
direction could contribute to improving both the efficiency and the
performance of the RL approach in narrative generation tasks.

In this work, unlike (Valls-Vargas et al., 2013) and (Hartsook et al., 2011),
we do not assume a symbolic representation of the story, and utilize large
language models to derive spatial constraints from stories in free-form text.
It remains to be investigated how effective this approach is, in regards to
the reasoning capability of large language models.

The generality of the RL model is crucial to our application, however it is
also extremely challenging. Comparing our setting with existing works on using
RL to play games, adapting to different maps is analogous to adapting to
different levels of a game, while learning for different constraints is
analogous to mastering games with different rules. Automatic curriculum
learning (Portelas et al., 2021) could provide a way of reaching this high
level of generalization capability. Maps and constraints could be dynamically
generated or selected rather than only training on a fixed set of scenarios.
By systematically increasing the difficulty of scenarios, and their
dissimilarity to those already encountered during training, we could gradually
expand the agent’s abilities to tackle diverse challenges.

##  7\. Conclusion

In this work we introduced new tools to support stories with game maps through
an automated plot facility layout design process. We demonstrated that by
employing plot facility layout design, we can utilize existing story and map
generation techniques, and help the designer visualize the narrative potential
of a story domain. The RL-based approach introduced can rapidly provide
solutions and adapt to user intervention, making it suitable for a real-time
human-AI co-design workflow. This approach has potential in many game design
applications, such as map design, playtime quest generation/adaptation and
story debugging; but also potential applications in other domains involving
spatial layouts subject to constraints, such as the design of large office
buildings or manufacturing plants.

## References

  * (1)
  * Baker et al. (2019) Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. 2019.  Emergent tool use from multi-agent autocurricula.  _arXiv preprint arXiv:1909.07528_ (2019).
  * Bellemare et al. (2013) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. 2013.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ 47 (2013), 253–279.
  * Bergdahl et al. (2020) Joakim Bergdahl, Camilo Gordillo, Konrad Tollmar, and Linus Gisslén. 2020.  Augmenting automated game testing with deep reinforcement learning. In _2020 IEEE Conference on Games (CoG)_. IEEE, 600–603.
  * Berner et al. (2019) Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. 2019\.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ (2019).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chen et al. (2023) Hanmo Chen, Stone Tao, Jiaxin Chen, Weihan Shen, Xihui Li, Sikai Cheng, Xiaolong Zhu, and Xiu Li. 2023.  Emergent collective intelligence from massive-agent cooperation and competition.  _arXiv preprint arXiv:2301.01609_ (2023).
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ 30 (2017).
  * Cook et al. (2021) Michael Cook, Jeremy Gow, Gillian Smith, and Simon Colton. 2021.  Danesh: Interactive tools for understanding procedural content generators.  _IEEE Transactions on Games_ 14, 3 (2021), 329–338.
  * Design (1980) A.I. Design. 1980.  Rogue.
  * Dieu et al. (2023) Dawid Dieu, Mateusz Markiewicz, Kuba Grodzicki, and SWi98. accessed 2023.  Polygonal Map Generation for Games.  https://github.com/TheFebrin/Polygonal-Map-Generation-for-Games.
  * Dormans and Bakkes (2011) Joris Dormans and Sander Bakkes. 2011.  Generating missions and spaces for adaptable play experiences.  _IEEE Transactions on Computational Intelligence and AI in Games_ 3, 3 (2011), 216–228.
  * Freak (2022) Game Freak. 2022.  Pokémon Legends: Arceus.
  * FromSoftware (2022) FromSoftware. 2022.  Elden Ring.
  * Hartsook et al. (2011) Ken Hartsook, Alexander Zook, Sauvik Das, and Mark O Riedl. 2011.  Toward supporting stories with procedurally generated game worlds. In _2011 IEEE Conference on Computational Intelligence and Games (CIG’11)_. IEEE, 297–304.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hendrikx et al. (2013) Mark Hendrikx, Sebastiaan Meijer, Joeri Van Der Velden, and Alexandru Iosup. 2013.  Procedural content generation for games: A survey.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ 9, 1 (2013), 1–22.
  * Iskander et al. (2020) Nancy Iskander, Aurelien Simoni, Eloi Alonso, and Maxim Peter. 2020.  Reinforcement Learning Agents for Ubisoft’s Roller Champions.  _arXiv preprint arXiv:2012.06031_ (2020).
  * Jaderberg et al. (2019) Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. 2019\.  Human-level performance in 3D multiplayer games with population-based reinforcement learning.  _Science_ 364, 6443 (2019), 859–865.
  * Kelly and McCabe (2017) George Kelly and Hugh McCabe. 2017.  A Survey of Procedural Techniques for City Generation.  _The ITB Journal_ 7, 2 (May 2017).  https://doi.org/10.21427/D76M9P
  * Lloyd (1982) Stuart Lloyd. 1982.  Least squares quantization in PCM.  _IEEE transactions on information theory_ 28, 2 (1982), 129–137.
  * Matsumoto (2022) Ryu Matsumoto. 2022.  Introduction of Case Studies of Engineer Efforts to Respond to the Open Field of Elden Ring.  _Computer Entertainment Developers Conference_ (2022).
  * Nintendo (1986) Nintendo. 1986.  The Legend of Zelda.
  * Nintendo (2017) Nintendo. 2017.  The Legend of Zelda: Breath of the Wild.
  * Patel (2010) Amit Patel. 2010.  Polygonal map generation for games.  _Red Blob Games_ 4 (2010).
  * Portelas et al. (2021) Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-yves Oudeyer. 2021.  Automatic Curriculum Learning For Deep RL: A Short Survey. In _IJCAI 2020-International Joint Conference on Artificial Intelligence_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Risi and Togelius (2020) Sebastian Risi and Julian Togelius. 2020.  Increasing generality in machine learning through procedural content generation.  _Nature Machine Intelligence_ 2, 8 (2020), 428–436.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ (2017).
  * Smelik et al. (2009) Ruben Smelik, Klaas Jan de Kraker, Saskia Groenewegen, Tim Tutenel, and Rafael Bidarra. 2009.  A Survey of Procedural Methods for Terrain Modelling.
  * Smith et al. (2010) Gillian Smith, Jim Whitehead, and Michael Mateas. 2010.  Tanagra: A mixed-initiative level design tool. In _Proceedings of the Fifth International Conference on the Foundations of Digital Games_. 209–216.
  * Team et al. (2021) Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. 2021\.  Open-ended learning leads to generally capable agents.  _arXiv preprint arXiv:2107.12808_ (2021).
  * Togelius et al. (2011) Julian Togelius, Georgios N Yannakakis, Kenneth O Stanley, and Cameron Browne. 2011.  Search-based procedural content generation: A taxonomy and survey.  _IEEE Transactions on Computational Intelligence and AI in Games_ 3, 3 (2011), 172–186.
  * Valls-Vargas et al. (2013) Josep Valls-Vargas, Santiago Ontanón, and Jichen Zhu. 2013.  Towards story-based content generation: From plot-points to maps. In _2013 IEEE Conference on Computational Inteligence in Games (CIG)_. IEEE, 1–8.
  * van der Linden et al. (2014) Roland van der Linden, Ricardo Lopes, and Rafael Bidarra. 2014.  Procedural Generation of Dungeons.  _IEEE Transactions on Computational Intelligence and AI in Games_ 6, 1 (2014), 78–89.  https://doi.org/10.1109/TCIAIG.2013.2290371
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019\.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ 575, 7782 (2019), 350–354.
  * Yannakakis et al. (2014) Georgios N. Yannakakis, Antonios Liapis, and Constantine Alexopoulos. 2014.  Mixed-initiative co-creativity. In _International Conference on Foundations of Digital Games_.
  * Zhang et al. (2018) Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. 2018.  A study on overfitting in deep reinforcement learning.  _arXiv preprint arXiv:1804.06893_ (2018).

Figure 4.  Same story accommodated on 4 different maps by rolling out a
trained RL policy. Arrows indicate directional relations. Note that the bottom
right layout failed to fulfill “Hearthfire Hold and Veilstead Kingdom should
be seperated by a lake”. This shows typical failure examples from our
policies: the RL model only manages to satisfy a subset of the constraints.
Figure 5. Plot facility re-adaptation after the user moves Marketown to a
different location. After we manually changed the location of Marketown to the
northeast part of the map. Veilstead kingdom and Hearthfire Hold adjusted
their location to continue to be across the lake from Marketown, while
Aquafrost Garrison stays at the same location, as all the constraints
mentioning it are still satisfied. Figure 6. (a) Agents with no constraints;
(b) Random agents. It can be seen that the agents have a strong preference of
going towards the edge of the map when there are no constraints provided in
the environment. Note that the map, constraints, and initial locations are all
the same for the two settings.  Figure 7. Motion trails at different training
stages: (a) random agents; (b) after 30 iterations; (c) after 90 iterations;
(d) after 133 iterations. Earlier stage models tend to have more random routes
with a lot of circling back-and-forth, where later stage models tend to show
clearer direction of progression. Figure 8. Whittaker Diagram for Biome Types

##  Appendix A Appendix Overview

We organize this appendix as follows: Section B reports a toy experiment
showing better performance from simulated concurrent plot facility movement
compared to actual concurrent movement (related to Section 5.1). Section C
shows state dimension for different embedding strategies (see Section 5.1).
Section D gives details on hyper-parameters (again see Section 5.1), Section E
shows a set of plot facility motion trails with different initial locations,
where the constraints and map are the same as in Figure 1. Section F shows a
selection of sample maps used in our dataset.

##  Appendix B Actual Concurrent vs. Simulated Concurrent Movement: A Toy
Experiment

We perform an additional toy experiment to compare a true concurrent movement
scheme and a simulated concurrent movement scheme of plot facilities. The
experiment involves a single plot facility layout task with the map shown in
Figure 9, Two plot facilities $p_{1}$ and $p_{2}$ and two constraints
requiring $p_{1}$ and $p_{2}$ to be inside the lake, respectively. The agent
with actual concurrent movement scheme moves all the plot facilities at a
single step, and the observation is updated only when all the plot facilities
are moved, while the agent with simulated concurrent movement scheme only
moves one plot facility for a small distance at one step.

In Table 6, we report the success rate from a random agent, an RL model
trained with actual concurrent movement and an RL model trained with simulated
concurrent movement. A possible explanation for the poor performance of the
actual concurrent movement setting is that in the setting it’s hard to
contribute global constraint satisfaction change to individual facility’s
movement.

Table 6. Success rate (%) comparison among random agent, an RL model trained with actual concurrent movement and an RL model trained with simulated concurrent movement Random | Actual Concurrent | Simulated Concurrent
---|---|---
$7.29\%$ | $0.14\%$ | $93\%$
Figure 9. Map used for toy experiment

##  Appendix C State Dimension in Details

Table 7. The state dimension in details from the three embedding strategies Embedding Strategy | Embedding Dimension
---|---
total | terrain | constraints | plot facilities
NL-based | 4422 | 512 | 10 * 387 | 10 * 4
CLIP-based | 5702 | 512 | 10 * 512 | 10 * 4
Relation-based | 1782 | 512 | 10 * 123 | 10 * 4

##  Appendix D Hyper-parameter Details

Table 8 presents the major hyper-parameters used for training the policies in
the paper. For the hidden layers, we run each set of experiments on the two
options in the table and choose the one with the higher reward to calculate
the success rates. The rest of the hyper-parameters are the same as the
default values from RLlib 333RLlib:
https://docs.ray.io/en/latest/rllib/index.html.

Table 8. Major hyperparameters used for training the policies in the paper. Hyperparameter | Default Value
---|---
lr | 1e-4
gamma | 0.99
lambda | 0.95
clip_param | 0.2
num_sgd_iter | 30
sgd_minibatch_size | 128
train_batch_size | 2000
num_workers | 7
fcnet_hiddens | [1024, 512] or [1024, 512, 512]

##  Appendix E Plot Facility Motion Trail with Different Initial Locations

Figure 10 shows our main example story initialized on the same map, but with
different initial locations for each plot facility.

Figure 10. Main story on the same map with different initial locations

##  Appendix F A selection of sample maps

Figure 11 shows a selection of maps generated using the method described in
Section 4.1, used to create our plot facility layout task dataset.

Figure 11. Selection of maps in 3D


# A practical DMPs Implementation for Skill Creation and Teleoperation with
Assistive Manipulators

Stefan Scherzinger1, Pascal Becker1, Arne Roennau1 and Rüdiger Dillmann1 This
research has been funded by the Federal Ministry of Education and Research of
Germany (BMBF) within the project ArNe in the framework of Robotic Systems in
Health Care (project number 16SV8412)
1 All authors are with FZI Research Center for Information Technology, Haid-
und-Neu-Str. 10-14, 76131 Karlsruhe, Germany {scherzinger, becker, roennau,
<EMAIL_ADDRESS>
###### Abstract

Assistive robotic manipulators are becoming increasingly important for people
with disabilities. Teleoperating the manipulator in mundane tasks is part of
their daily lives. Instead of steering the robot through all actions, applying
self-recorded motion skills could greatly facilitate repetitive tasks. Dynamic
Movement Primitives (DMP) are a powerful method for skill learning. For this
use case, however, they need a simple heuristic to specify where to start and
stop a skill without additional sensors. This paper provides the concept of
local, global, and hybrid skills that form a modular basis for composing
single-handed tasks with ease. A focus is on presenting the necessary
mathematical details to support custom implementations with assistive robot
arms. Experiments validate the developed methods for scratching an itchy spot,
sorting objects on a desk, and feeding a piggy bank with coins. The paper is
accompanied by an open-source implementation at https://github.com/fzi-
forschungszentrum-informatik/ArNe

##  I INTRODUCTION

Robotic manipulators are becoming increasingly important as assistive devices
in home settings for people with disabilities. Despite the complexity of tasks
of daily living - which often require two hands to accomplish - there are
tasks that users can accomplish on their own with one robotic arm, such as
organizing things on their desk, or scratching an itchy spot on the knee.
Research shows promising hybrids between direct control and assistive autonomy
for safe drinking [1], object manipulation [2],[3], and grasping [4], for
instance. This autonomy, however, usually comes at the expense of additional
sensor and hardware setups, e.g. [5], that must be provided and integrated
into existing, basic manipulator systems. Losey et al show an approach for
embedding high-dimensional robot behaviors into simplified, latent actions
that users can then teleoperate [6]. Although improving users’ performance in
a kitchen setup, the behaviors must be demonstrated by caregivers first.

For the user group that can and wants to stay in cognitive control of their
robot, a simple mechanism for recording repetitive motion patterns and
replaying them might already be a big support. Steering the robot as a mere
tool through teleoperation, users could flexibly create and personalize their
own skill repertoire through programming by demonstration.

The principal contribution of the paper is a practical version of Dynamic
Movement Primitives (DMPs) [7],[8], for recording and playing back such motion
patterns while teleoperating robotic arms. Our concept targets six-axes
manipulators without additional sensors for perception, and addresses the
important challenge of formulating suitable start and goal states
automatically for the DMPs during teleoperation. To this end, we propose
local, global, and hybrid skills as a modular concept for composing everyday
tasks. Our implementation simplifies much of the DMPs’ complexity and
additionally provides the necessary mathematical detail on coordinate
transformations that are less often presented in papers. We accompany the
paper with an implementation for the ROS framework [9], which is available
open-source222https://github.com/fzi-forschungszentrum-informatik/ArNe .

The remaining paper is structured as follows: We briefly describe the
principle mechanisms behind DMPs in Section II to illustrate our
simplifications to the original approach. Section III then describes our
control interface for the manipulators. The core of the paper is Section IV
with the presentation of three skill types and their implementation. We
evaluate and discuss them in Section V, and provide an overall conclusion in
Section VI.

##  II BACKGROUND AND RELATED WORK

###  II-A DMPs

The principle idea is to describe a trajectory $\bm{x}(t)$ of a state vector
$\bm{x}\in\mathbb{R}^{n}$ of $n$ dimensions with a set of ordinary
differential equations (ODEs). In each dimension, its scalar variable $x$ is
defined by an individual spring-damper system, e.g.

${\tau}^{2}\ddot{x}=K(g-x)-\tau D\dot{x}+(g-x_{0})f(s)\quad.$ (1)

Eq. (1)) is usually referred to as transformation system. One important
feature of DMPs is the generalization from new start states $x_{0}$ to new
goals $g$, referred to as attractor states. The duration of the execution can
be adjusted via the time scale $\tau$. Both stiffness $K$ and damping $D$ are
constants that need hand-tuning to specific use cases. Likewise, the
arrangement of terms of Eq. (1) also varies with use cases and
implementations. Popular enhancements include e.g. collision avoidance [10],
[11] and the integration of haptic feedback [12]. Lauretti et al show a more
recent approach for hybrid joint/Cartesian DMPs for redundant manipulators
with the advantage of maintaining human-like motion during collision
avoidance[13]. The non-linear forcing term $f(s(t))$ is the core of the
framework, and is usually parameterized with a phase variable $s$ across the
course of the skill. This term is responsible for effecting desired motion
characteristics, which are to be learned from human-recorded training data.

All $n$ transformation systems with the shape of Eq. (1) are synchronized
through $s$ according to the following canonical system

$\tau\dot{s}(t)=-\alpha s(t)$ (2)

that describes the correlation between phase time $s$ and real-time $t$ as an
exponential decay from $1\rightarrow 0$. The parameter $\alpha$ is an
additional constant for fine-tuning.

The perturbation $f$ is usually modeled with a superposition of radial basis
functions, e.g.

$f(s)=\frac{\sum_{i}\omega_{i}\psi_{i}(s)s}{\sum_{i}\psi_{i}(s)}\quad,$ (3)

with $\psi_{i}(s)=\text{exp}(-h_{i}(s-c_{i})^{2})$ representing Gaussians with
constant center $c_{i}$ and constant width $h_{i}$. The higher the number of
basis functions, and the more clever they overlap, the closer are they able to
capture motion characteristics with $f$ in Eq. (1). The choice of more
suitable kernel functions can partially mitigate the exploding computational
complexity and can allow for an a-priori estimation of the reproduction
accuracy [14].

If combined with Reinforcement Learning (RL) on robots with many degrees of
freedom, for instance, a massive number of basis functions can become a
substantial performance bottleneck, since all parameters need to be learned
and need to converge in numerous cycles of own trial and error[15].

Learning human skills supervised into DMPs means learning adjustable weights
$\omega_{i}$ from example motions. This is achieved by recording a skill-
defining trajectory $x(t),\dot{x}(t),\ddot{x}(t)$ of arbitrary duration from
human demonstration. Solving Eq. (1) for $f$ leads to a set of discrete
samples by substituting the trajectory points into

$\widehat{f}(s(t))=\frac{-K(g-x(t))+\tau
D\dot{x}(t)+{\tau}^{2}\ddot{x}(t)}{g-x_{0}}\quad.$ (4)

This has a certain resemblance to an inverse dynamics approach, in which
$\widehat{f}$ as reaction force is uniquely defined by the motion profile over
time. The weights of Eq. (3) are then adjusted with the samples obtained from
Eq. (4), e.g. by minimizing $\sum_{s}(\widehat{f}(s)-f(s))^{2}$ with
multivariate linear regression.

Finally, DMPs are deployed as follows: First Eq. (2) and then each of the $n$
transformation systems with Eq. (1) are numerically integrated with a task-
specific $\tau$ as a solution to an initial value problem. The combined state
$\bm{x}(t)$ is then passed to a suitable robot controller as a reference
signal for motion tracking.

###  II-B Assumptions and Simplifications

This paper describes a practical approach for using DMPs as a lightweight tool
with a teleoperated manipulator. Composing and using skills on this basic
system differs from what is often described in the literature. We assume that

  * •

the users’ creativity is unlimited in recording and playing-back skills

  * •

the users must anticipate possible collisions themselves

  * •

the playback of skills through the robot is open-loop

  * •

the users will learn with this tool from trial and error.

It is worth pointing out that the core feature of generalization between
different attractor states is achieved by the dynamical system itself, and
solely depends on the shape of Eq. (1). For our case of modeling trajectories,
the principle motivation for parameterizing $f$ with overlapping basis
functions is to provide a mechanism for time-independent scaling. There is
usually a trade-off involved between capturing details in the motion profiles
and being computationally tractable. An upper limit of accuracy would be an
infinite number of basis functions. Considering our setting from above, the
idea is as follows: If we use the same number of samples for describing
recorded motion with Eq. (4) as for numerically integrating Eq. (1), then we
can omit the complete parameterization. Temporal scaling would then be
achieved with a mere assignment of timestamps to the computed motion. For
instance, if we recorded a motion of $10\text{\,}\mathrm{s}$ at
$100\text{\,}\mathrm{H}\mathrm{z}$ ($=1000$ samples) and wanted to execute
that in a different configuration during $20\text{\,}\mathrm{s}$, we would
numerically integrate Eq. (1) with a time step of $0.01\text{\,}\mathrm{s}$
($1000$ steps), and a suitable controller would interpolate and follow these
setpoints later at $50\text{\,}\mathrm{H}\mathrm{z}$.

Our simplifications to the DMP approach summarize as follows:

  * •

Set $\tau\equiv 1$ in Eq. (1)

  * •

Drop Eq. (2), Eq. (3), and the phase variable $s$

  * •

Use $\widehat{f}$ from Eq. (4) directly to integrate Eq. (1)

In the remainder of the paper, we apply these simplifications to our use case
of teleoperating robotic manipulators with motion skills.

##  III MANIPULATOR CONTROL

This section describes our methods for teleoperation and motion tracking. We
target 6-axis light-weight robotic manipulators that are common as assistive
devices. All grippers are suitable as long as their state can be mapped to a
single degree of freedom. Fig. 1 illustrates one such manipulator with three
important coordinate frames:

Figure 1: Schematic illustration of our 6-axis light-weight robotic
manipulator with base frame $\mathcal{B}$, end-effector frame $\mathcal{E}$,
and desired target $\mathcal{X}$.

the robot’s base frame $\mathcal{B}$, the robot’s end-effector frame
$\mathcal{E}$, and its desired target pose $\mathcal{X}$ during motion
control. $\mathcal{E}$ and $\mathcal{X}$ are each defined by a position vector
$\bm{x}=[x~{}y~{}z]^{T}\in\mathbb{R}^{3}$ and an orientation quaternion
$\bm{q}=q_{w}+q_{x}i+q_{y}j+q_{z}k$, given in the base frame $\mathcal{B}$.
The superscript d indicates desired quantities for target pose tracking. The
end-effector frame $\mathcal{E}$ is located at the grasping center of the
1-DOF gripper, whose state (opening percentage) is determined by the scalar
$g\in[0,1]$. The vector $\bm{\theta}\in\mathbb{R}^{6}$ denotes the set of
joint angle positions. During motion control, a suitable controller
continuously tracks the given target $\mathcal{X}$ with the end-effector frame
$\mathcal{E}$.

###  III-A Teleoperation

Teleoperation must be simple and intuitive. Our approach uses a conventional
space mouse as a 6-axis joystick, but similar low-cost devices could be used
as well. This joystick measures infinitesimal displacements from interaction
with thumb and fingers in three linear axes and three rotational axes, and
interprets those signals as a combined, six-dimensional twist command
$[\bm{v}~{}\bm{\omega}]^{T}=[v_{x}~{}v_{y}~{}v_{z}~{}\omega_{x}~{}\omega_{y}~{}\omega_{z}]^{T}$.
Starting from the robot’s end-effector frame $\mathcal{E}$ in each control
cycle, we then time-integrate this twist to become the new target pose
$\mathcal{X}$ for motion tracking according to

$\displaystyle\bm{x}^{d}$ $\displaystyle=\bm{x}+\bm{v}\Delta t$ (5)
$\displaystyle\bm{q}^{d}$ $\displaystyle=\bm{q}+\dot{\bm{q}}\Delta
t\quad,\qquad\dot{\bm{q}}=\frac{1}{2}\bm{\omega}\bm{q}~{}.$ (6)

Fig. 2 shows the control scheme.

Figure 2: Teleoperation with a 6-axis joystick. We integrate linear and
angular velocity measurements into the desired target pose
$\bm{x}^{d},\bm{q}^{d}$ for end-effector control. Additional buttons allow
setting the opening and closing speed $\dot{g}$ for the gripper.

All quantities are given in the robot’s base frame $\mathcal{B}$. The
quaternion product $\bm{\omega}\bm{q}$ uses the angular velocity given in
quaternion notation $\bm{\omega}=0+\omega_{x}i+\omega_{y}j+\omega_{z}k$. In a
similar fashion, we time-integrate the gripper’s opening/closing speed, which
are triggered by pressing additional buttons on the joystick, to obtain its
target state $g^{d}$ with

$g^{d}=g+\dot{g}\Delta t~{}.$ (7)

The computation between Cartesian space and joint space is done with suitable
inverse kinematics (IK) and forward kinematics (FK) algorithms.

###  III-B Motion Tracking and Inverse Kinematics

Figure 3: Dynamics-based IK solver from our previous work [16], adapted to the
notation of this paper. After internal iterations, the solver converges to a
set of simulated joint positions $\widehat{\bm{\theta}}$ that best correspond
to the given target pose.

The robots we target support streaming joint position control interfaces, i.e.
they require $\bm{\theta}^{d}$ at a specific control rate. We use a forward
dynamics-based control approach of our previous work [16] for solving the IK
problem during tracking the target pose $\mathcal{X}$. Its implementation is
available open-source111https://github.com/fzi-forschungszentrum-
informatik/cartesian_controllers. In short, this IK solver transforms the
difference between end-effector target $\mathcal{X}$ and current pose
$\mathcal{E}$ into a 6-dimensional error. It then applies this error as a
goal-directed force-torque vector at the end-effector of a virtual model of
the robot, and closed-loop simulates the reaction motion of the system. The
result $\bm{\theta}^{d}$ is streamed to joint-level position controllers of
the real system. Fig. 3 shows the control scheme, which we will briefly detail
in the remainder of this section.

In the control scheme and succeeding equations, the hat symbol $\widehat{(.)}$
denotes simulated values. They are computed on a virtual model that equals the
real robot’s kinematics but has simplified dynamic properties.

The steps are as follows: First, we compute a virtual excitation
$\widehat{\bm{f}}$ from the translational and rotational error between the
target and the current end-effector with

$\widehat{\bm{f}}=\bm{K}_{p}[\Delta\bm{x}~{}\Delta\bm{r}]^{T}~{}.$ (8)

The positive, diagonal stiffness matrix $\bm{K}_{p}$ serves as a proportional
gain for this error. The individual error vectors are computed according to

$\displaystyle\Delta\bm{x}$ $\displaystyle=\bm{x}^{d}-\widehat{\bm{x}}$ (9)
$\displaystyle\Delta\bm{r}$
$\displaystyle=\phi[r_{x}~{}r_{y}~{}r_{z}]^{T}\leftarrow\Delta\bm{q}=\bm{q}^{d}\widehat{\bm{q}}^{-1}~{},$
(10)

in which $\Delta\bm{q}$ is the relative rotation between both frames in
quaternion notation. We obtain the rotational error by transforming this
rotation into axis-angle notation with the rotation angle $\phi$ and the
normalized rotation axis $[r_{x},r_{y},r_{z}]^{T}$. The next step is to
simulate the virtual system’s response from this virtual excitation with

$\ddot{\widehat{\bm{\theta}}}=\widehat{\bm{H}}^{-1}\widehat{\bm{f}}~{}.$ (11)

$\widehat{\bm{H}}$ is a dynamics-shaped joint space inertia matrix that leads
to better linearization and convergence of the IK solver. We refer the
interested reader to [17] for more details on this aspect. Since Eq. (11)
delivers the motion response on acceleration level, the IK solver needs to
integrate the joint accelerations $\ddot{\bm{\widehat{\theta}}}$ twice. We
obtain $\widehat{\bm{\theta}}$ with the Euler forward method according to

$\displaystyle\widehat{\bm{\theta}}_{t}=\widehat{\bm{\theta}}_{t-1}+\dot{\widehat{\bm{\theta}}}_{t-1}\Delta\widehat{t}$
(12)
$\displaystyle\dot{\widehat{\bm{\theta}}}_{t}=\dot{\widehat{\bm{\theta}}}_{t-1}+\ddot{\widehat{\bm{\theta}}}_{t}\Delta\widehat{t}~{}.$
(13)

Additional joint damping of $10\%$ in each simulation cycle leads to an
exponential decay of possible null space motion and avoids oscillations around
the target pose:

$\dot{\widehat{\bm{\theta}}}_{t}\leftarrow
0.9~{}\dot{\widehat{\bm{\theta}}}_{t}~{}.$ (14)

Note that the simulated step $\Delta\widehat{t}$ is independent of the robot’s
real control rate $\Delta t$.

Finally, we use a common FK implementation from
KDL222https://github.com/orocos/orocos_kinematics_dynamics in the ROS
framework to compute the simulated end-effector pose
$\widehat{\bm{x}},\widehat{\bm{q}}$ from the simulated joint positions in each
cycle. The solver’s result is the converged set of simulated joint positions
$\widehat{\bm{\theta}}$ that is then forwarded as the desired command for the
streaming joint position interface of the robot.

##  IV COMPOSING AND USING SKILLS

Motion skills are a central element for simplifying repetitive tasks. This
section describes the methods behind building skills from recorded
teleoperation, and details how we create new goal-oriented trajectories from
these representations.

###  IV-A Skill Recording and Representation

We represent a specific robot state as the vector

$\bm{s}={}^{\mathcal{B}}[x,y,z,q_{x},q_{y},q_{z},q_{w},g]$ (15)

that comprises the translation and orientation of $\mathcal{E}$, and the
dimensionless gripper. All entities are given in the robot’s base frame
$\mathcal{B}$, and are computed from the joint positions $\bm{\theta}$ with
forward kinematics. Using Cartesian coordinates for the state vector - as
opposed to a joint-space representation - is important for motion
generalization and shall make it easier to anticipate the robot’s motion
during skill playback for new targets. Equivalent to conventional DMPs, our
skills capture motion characteristics from recorded state trajectories. We
thus first teleoperate the robot and record a sequence of states
$\\{{\bm{s}_{0},\bm{s}_{1},\bm{s}_{2},\dots,\bm{s}_{N}}\\}$ in equidistant
time steps $\Delta t$ for a certain duration $T$. Fig. 4(a) illustrates an
example of grasping an imaginary object and lifting it sideways in a slight
arc.

We transform all recorded states with respect to the robot’s end-effector
frame when recording started. Since the skill will capture motion from this
recording, it will do so in a neutral coordinate frame. We use homogeneous
matrices for transformation, and denote $\bm{T}(\bm{s})$ for the homogeneous
matrix representation of $\bm{s}$. Due to the quaternion notation of Eq. (15),
formulating states as homogeneous matrices, and composing states back from the
entries of a homogeneous matrix, is unique. With that, the states become

$\bm{s}_{n}\leftarrow\bm{T}^{-1}(\bm{s}_{0})\bm{T}(\bm{s}_{n})~{},n=0\dots
N~{}.$ (16)

Note that the gripper state $g$ needs no transformation, and is simply kept
throughout all coordinate systems. The first recorded state for any skill is
thus $\bm{s}_{0}=[0~{}0~{}0~{}0~{}0~{}0~{}1~{}g_{0}]^{T}$ with a gripper
opening $g_{0}$.

The next step is to compute the time derivatives
$\dot{\bm{s}}=\frac{d}{dt}(\bm{s})$ and
$\ddot{\bm{s}}=\frac{d}{dt}(\dot{\bm{s}})$, for which we use a five-step
differentiator [18] with

$\begin{split}\frac{d}{dt}(\bm{s}_{n})&:=\frac{5(\bm{s}_{n+1}-\bm{s}_{n-1})}{32\Delta
t}+\frac{4(\bm{s}_{n+2}-\bm{s}_{n-2})}{32\Delta t}\\\
&~{}+\frac{\bm{s}_{n+3}-\bm{s}_{n-3}}{32\Delta t}\end{split}$ (17)

that we apply to every state $n=0\dots N$ in the recorded sequence. Start and
end boundaries are considered with

$\bm{s}_{(.)}=\begin{cases}\bm{s}_{0}&\text{for $(.)\leq 0$}\\\
\bm{s}_{N}&\text{for $(.)\geq N$}\\\ \bm{s}_{(.)}&\text{else}~{}.\end{cases}$
(18)

We use Ijspeert’s simple spring damper model [8] as transformation system,
adapted to our state vector

$\ddot{\bm{s}}=\bm{D}(\bm{K}(\bm{g}-\bm{s})-\dot{\bm{s}})+\bm{f}~{}.$ (19)

The vector $\bm{g}$ stands for the goal state of the motion and will vary with
skill type that we explain in the next section. The diagonal stiffness
$\bm{K}\in\mathbb{R}^{8\times 8}$ and damping matrix
$\bm{D}\in\mathbb{R}^{8\times 8}$ are chosen once, and remain constant
throughout creating and using skills. Analog to the conventional DMP approach,
we obtain $n$ unique samples for the forcing term by solving Eq. (19) for
$\bm{f}$ and substituting the state trajectory in

$\bm{f}_{n}=\ddot{\bm{s}}_{n}-\bm{D}(\bm{K}(\bm{g}-\bm{s}_{n})-\dot{\bm{s}}_{n})~{}.$
(20)

In contrast to conventional DMPs, however, the computed sequence of forcing
terms $\\{\bm{f}_{0},\bm{f}_{1},\bm{f}_{2},\dots,\bm{f}_{N}\\}$ already
represents the quintessence of a skill.

###  IV-B Skill Types and Generalization

Figure 4: Skill recording (a) and generalization with three different
execution types: (b) local skill execution, (c) global execution, and (d)
hybrid execution.

Skills are meant to be recorded at one point and replayed at another point. We
propose three different types of skills that users can choose from to compose
their everyday tasks: local, global, and hybrid. Each type will lead to a
slightly different formulation of the goal state
$\bm{g}\in\\{\bm{g}_{L},\bm{g}_{G},\bm{g}_{H}\\}$ in the transformation system
Eq. (19), and will shape the path that the robot takes when applying the
motion skill. The next sections present $\bm{g}$ for each case.

####  IV-B1 Local Skills

This is the simplest type of generalization. The goal state is set to the
final state as seen from the end-effector frame when we started the recording:

$\bm{g}_{L}\leftarrow\bm{T}^{-1}(\bm{s}_{0})\bm{T}(\bm{s}_{N})~{}.$ (21)

It causes a simple playback of the recorded motion locally, starting from the
robot’s momentary end-effector frame $\mathcal{E}$. Fig. 4(b) shows the path
that the robot would take for the example motion of Fig. 4(a).

####  IV-B2 Global Skills

This type of skill sets the goal state to the final state of the recording and
displays that in the current robot’s end-effector frame:

$\bm{g}_{G}\leftarrow\bm{T}^{-1}(\bm{s}^{*})\bm{T}(\bm{s}_{N})~{}.$ (22)

It causes a generalization back towards where the recorded motion ended, seen
globally in the robot’s workspace. It starts from the robot’s momentary end-
effector pose, denoted here with $\bm{s}^{*}$. Fig. 4(c) shows the effect,
using the previous example motion.

####  IV-B3 Hybrid Skills

This type is a combination of local and global skills. It drives back to where
the recorded motion globally ended but starts the motion with the momentary
end-effector orientation. To achieve this, we make use of an additional frame
that is adequately oriented in the robot’s end-effector frame and scale the
local motion towards the global goal:

$\bm{g}_{H}\leftarrow\bm{T}(\bm{g}_{L})\begin{bmatrix}0\\\ 0\\\ 0\\\
\frac{\lVert\bm{x}_{G}\rVert}{\lVert\bm{x}_{L}\rVert}\end{bmatrix}~{}.$ (23)

$\bm{x}_{G}$ and $\bm{x}_{L}$ denote the translational parts of $\bm{g}_{G}$
and $\bm{g}_{L}$, respectively. Fig. 4(d) shows the respective path. We
describe the details of the additional frame in the next section together with
how all skills transform the created motion back to the robot’s base frame
$\mathcal{B}$ for control.

###  IV-C Trajectory Generation and Playback

Creating new trajectories is done in three steps. They all take place just
before executing a skill in a new position. First, we compute a new sequence
of states $\bm{s}_{n}$ by numerically integrating Eq. (19) with one of the
goal formulations from the previous section; then we transform this sequence
back into the robot’s base frame for control; and finally assign desired
timestamps for each state, and execute the motion with the robot.

####  IV-C1 Numerical integration

We integrate the transformation system with the forward Euler method.
Algorithm 1 shows the scheme. Note that only the desired duration $T$ of the
skill and its type is chosen by the user. The other arguments are derived
implicitly from the robot’s current pose in its workspace, such as the
respective goal $\bm{g}\in\\{\bm{g}_{L},\bm{g}_{G},\bm{g}_{H}\\}$, and the
start for motion generation $\bm{s}_{0},\dot{\bm{s}}_{0}$. Both $\bm{f}_{n}$
and $N$ are defined by the recorded skill.

Algorithm 1 Trajectory generation

1:procedure INTEGRATE($\bm{g},\bm{s}_{0},\dot{\bm{s}}_{0},\bm{f}_{n},N,T$)

2: $\Delta t=T/N$

3: for $n=1\text{ \bf{to} }N$ do

4: $\bm{f}_{\|}=\frac{\bm{f}\cdot\bm{x}}{\bm{x}\cdot\bm{x}}\bm{x}$

5: $\bm{f}_{\perp}=\bm{f}-\bm{f}_{\|}$

6:
$\bm{f}_{\|}=\frac{\lVert\bm{x}_{G}\rVert}{\lVert\bm{x}_{L}\rVert}\bm{f}_{\|}$

7: $\bm{f}=\bm{f}_{\perp}+\bm{f}_{\|}$

8:
$\ddot{\bm{s}}_{n}=\bm{D}(\bm{K}(\bm{g}-\bm{s}_{n-1})-\dot{\bm{s}}_{n-1})\bm{f}_{n}$

9: $\dot{\bm{s}}_{n}=\dot{\bm{s}}_{n-1}+\ddot{\bm{s}}_{n}\Delta t$

10: $\bm{s}_{n}=\bm{s}_{n-1}+\dot{\bm{s}}_{n}\Delta t$

11: end for

12: return $\\{\bm{s}_{0},\bm{s}_{1},\dots,\bm{s}_{N}\\}$

13:end procedure

Step $4$ to $7$ rescale the translational part $\bm{f}\in\mathbb{R}^{3}$ of
the forcing term $\bm{f}_{n}\in\mathbb{R}^{8}$ to take the new goal distance
and the possibly changed stiffness into consideration. This is necessary,
because the new goal attractor might pull harder than before, and might
completely overrule the forcing term especially at the beginning of the
generalization. We mitigate this effect by projecting $\bm{f}$ onto the goal
direction $\bm{x}\in\mathbb{R}^{3}$, scaling it, and re-adding the orthogonal
component $\bm{f}_{\perp}$.

The result of Algorithm 1 is a sequence of $N$ states that describe the new
motion path in a skill-relative reference frame. Note that it is important to
normalize the quaternion components afterward in the individual states before
succeeding operations. Algorithm 1 works dimension-wise and does not enforce
this.

####  IV-C2 Transformations

This step is to display the created sequence with respect to the robot’s base
$\mathcal{B}$ for the IK solver. Both local and global skills generalize the
motion with respect to their momentary end-effector frame. Their motion is
thus transformed with

$\bm{s}_{n}\leftarrow\bm{T}(\bm{s}^{*})\bm{T}(\bm{s}_{n})~{},n=0\dots N~{}.$
(24)

In contrast, hybrid skills generalize in a specialized reference frame, and
need an additional rotation before transforming their states back to the robot
base.

Figure 5: Hybrid skill execution with recorded motion (purple) and projected
motion (green) from a different start pose.

Fig. 5 provides a second example that we use to better illustrate the required
steps. It shows a recorded motion that we apply from a different start pose.
The two homogeneous transformation matrices

$\displaystyle\bm{T}_{1}$ $\displaystyle=\bm{T}^{-1}(\bm{g}_{L})~{},$ (25)
$\displaystyle\bm{T}_{2}$ $\displaystyle=\bm{T}^{-1}(\bm{g}_{G})$ (26)

represent the start of the example recording and the start of the hybrid
generalization, respectively. Note that the recorded motion’s local goal
$\bm{g}_{L}$ and the global goal $\bm{g}_{G}$ always describe the same pose in
the robot’s workspace. Since hybrid skills shall drive to the global position
while reproducing the motion from their local orientation, we need to rotate
the recorded motion’s start into the direction of $\bm{T}_{2}$ with

$\bm{T}_{H}=\bm{T}(\bm{\alpha},\bm{a})\bm{T}_{1}~{}.$ (27)

Fig. 5 shows the rotation angle $\alpha$ and the rotation axis $\bm{a}$, which
are defined by the two vectors $\bm{x}_{1},\bm{x}_{2}$, and whose components
we directly obtain from $\bm{T}_{1}$ and $\bm{T}_{2}$. Note that the length
scaling of the motion is already considered with Eq. (23).

Finally, we extract the rotation $\bm{R}\leftarrow\bm{T}_{2}\bm{T}_{H}$ and
display the motion in the end-effector’s coordinates.

$\bm{s}_{n}\leftarrow\begin{bmatrix}\bm{R}&\bm{0}\\\
\bm{0}^{T}&0\end{bmatrix}\bm{T}(\bm{s}_{n})~{},n=0\dots N~{}.$ (28)

After this operation, the states are given in frame $\mathcal{E}$ and can be
formulated in frame $\mathcal{B}$ with Eq. (24).

####  IV-C3 Trajectory duration and playback

Users can specify the duration $T$ of the skill that is fed into Algorithm 1.
This is helpful if the motion is to be reproduced slower or faster than was
recorded. Classic DMPs use a canonical system from Eq. (2) for this task and
build the complete distribution of radial basis functions for approximating
the forcing term $\bm{f}$ on this mechanism. Our practical approach
circumvents this complexity by simply assigning the desired timestamp to each
of the individual states of the sequence with

$\bm{s}^{d}(t)=\begin{cases}\bm{s}_{n}~{}\text{with}~{}n=\mathbb{N}(\frac{t}{\Delta
t})&\text{for $0\leq t<T$}\\\ \bm{s}_{N}&\text{for $T\leq t$}\end{cases}~{}.$
(29)

The notation $\mathbb{N}(.)$ denotes a truncation and conversion into a
natural number $n\in\mathbb{N}$. Each of the individual states $\bm{s}^{d}$
holds the desired robot control commands according to Eq. (15), and is passed
to our IK solver as depicted in Fig. 6.

Figure 6: Skill playback on the robotic manipulator with sampling desired
gripper poses from the created state sequence. Our IK solver smoothly
interpolates between these discrete targets for the robot’s joint position
control.

##  V EXPERIMENTS AND RESULTS

This section demonstrates the feasibility of the proposed methods with
applying local, hybrid, and global skills to example use cases. For each, we
record a skill by teleoperating the robot through the motion once and apply
the skill from various starting poses. We then plot and discuss the end-
effector paths that the robot takes. The executions of all motions with the
robot are shown in the accompanying video. In addition to this evaluation, we
also give practical hints on how to mentally map everyday tasks to these three
skill types.

###  V-A Robotic Setup

We use the Accrea Aria robot for the experiments. It has an integrated gripper
with one degree of freedom and three passive compliant fingers. Fig. 7(a)
shows the arm in an upright position. It is compatible with ROS-control [19],
and provides the necessary streaming interface for joint
positions333https://github.com/stefanscherzinger/aria-ros/tree/devel. We use a
3DConnexion spacemouse as joystick for teleoperating the robot. Stiffness and
damping for Eq. (19) were set to $\bm{K}=0.55\cdot\bm{I}^{8\times 8}$ and
$\bm{D}=3.5\cdot\bm{I}^{8\times 8}$, respectively.

###  V-B Local Skill Evaluation

Local skills simply replay the recorded motion in the robot’s current end-
effector frame. This is useful for skills that can be formulated entirely
central to some spot, such as wiping, brushing, or scratching. Fig. 7 shows a
recorded brushing motion on a flat surface and various skill replays at
slightly different spots. The curves show that the robot’s end-effector
replays the recorded pattern as expected.

(a)

(b)

Figure 7: (a): The robotic manipulator from our experiments with the joystick
for teleoperation. (b) Brush movements on a flat surface, applied with a local
skill at different positions and orientations.

###  V-C Hybrid Skill Evaluation

(a)

(b)

Figure 8: (a) Collecting objects in a basket. (b) Recorded paths of the
robot’s end-effector with hybrid skill execution. A single recording is
generalized from different starting poses.

This experiment evaluates our methods for goal-directed motions from different
starting poses. Fig. 8(a) shows the setting. The robot is clamped to the table
and the task is to collect softballs in a basket. If successful, all skill
executions must terminate where the recording ended, independent from their
start orientation, and the traveled path should reflect the characteristics of
the recording. We conducted the experiment as follows: First, we teleoperated
the robot once to record the skill. The recording started right before
grasping the softball and ended after dropping it above the basket. We then
teleoperated the robot manually to the remaining softballs and triggered the
previously recorded skill at each spot with an open gripper. Fig. 8(b) shows
the individual skill executions. The coordinate systems at the starts
illustrate the gripper’s orientation before grasping. All curves terminate in
the hybrid goal $\bm{g}_{H}$, as was intended, and the initial motion pattern
is successfully scaled and reproduced with their orientation.

###  V-D Global Skill Evaluation

(a)

(b)

Figure 9: (a) Feeding coins into a piggy bank from different starts. The
coin’s final orientation is essential for success. (b) Recorded paths of the
global skill execution on the robot.

This last experiment shows the generalization capabilities to exact goal
poses. We feed a piggy bank as shown in Fig. 9(a), where the coin must be
carefully aligned with the opening to succeed. While in the previous
experiment the gripper mostly kept its local orientation during motion
execution, this skill needs to generalize across the different quaternion
dimensions to finally converge to the absolute goal pose. Fig. 9(b) shows the
effectiveness of the presented methods. Note that the recorded motion was
comparatively short and described a slight arc above the piggy. The skills,
nevertheless, worked even from starts below this level and from significantly
greater distances.

###  V-E Discussion

Within our proposed framework, all goals are implicitly given through the last
state of the motion recording. We think that this is a suitable trade-off
between flexibility and ease of use, since it avoids that users need to
specify Cartesian poses in the robot’s workspace, and instead record what they
want via teleoperation. Note that the presented methods do not aim for
assistive autonomy, and there is no plausibility check whether the robot can
execute a skill in the specified duration, nor whether a skill’s motion might
get distorted through singular joint configurations. Instead, the approach is
to give users cognitive responsibility and freedom over recording and
deploying skills as they please, including to learn from trial and error.

Here are some examples to help map everyday tasks into our three categories:

  * •

Local skills: Spooning food, opening doors and drawers, shaking a pack of
juice, pouring, scratching an itchy spot

  * •

Hybrid skills: Throwing things in a trash bin, putting ingredients in a pot,
removing captured stones from a go board

  * •

Global skills: Grasping and passing objects, bringing food to one’s mouth

Also, note that hybrid and global skills are mostly short-lived and make sense
to be recorded immediately before executing a repetitive task. They could,
however, be part of a permanent skill repertoire if the robot is mounted to a
wheelchair, for instance, and the goals do not change with respect to the
robot’s base.

##  VI CONCLUSIONS

This paper presented a simplified DMPs-based approach for recording and
playing-back skills with teleoperated manipulators. We proposed three skill
types that can serve as a modular basis for simplifying repetitive patterns of
everyday tasks. The skills take advantage of start and end states during
recording and circumvent the difficulty of specifying goal attractors
manually. By using the DMPs’ forcing term directly on the transformation
system, we cut the complexity of parameterization with basis functions and the
phase variable and instead realized time scaling with a simple reassigning of
timestamps to fixed-step trajectories. These trajectories are then executed
open-loop by an interpolating Cartesian controller. The presented methods
target manipulator systems without sensors for perception and are suitable
whenever users need a simple, tool-like mechanism to compose and use skills
during teleoperation.

## References

  * [1] Paolo Di Lillo, Filippo Arrichiello, Daniele Di Vito and Gianluca Antonelli  “BCI-Controlled Assistive Manipulator: Developed Architecture and Experimental Results”  In _IEEE Transactions on Cognitive and Developmental Systems_ 13.1, 2021, pp. 91–104  DOI: 10.1109/TCDS.2020.2979375
  * [2] Todor Stoyanov et al.  “Assisted Telemanipulation: A Stack-Of-Tasks Approach to Remote Manipulator Control”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 1–9  DOI: 10.1109/IROS.2018.8594457
  * [3] Ya-Ting Lee, Chian-Song Chiu, Li-Hu Jhang and Carlo Santiago  “A self-reliance assistive tool for disable people”  In _2018 3rd International Conference on Control and Robotics Engineering (ICCRE)_ , 2018, pp. 26–30  DOI: 10.1109/ICCRE.2018.8376428
  * [4] Siddarth Jain and Brenna Argall  “Grasp detection for assistive robotic manipulation”  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , 2016, pp. 2015–2021  DOI: 10.1109/ICRA.2016.7487348
  * [5] Filippo Arrichiello et al.  “Assistive robot operated via P300-based brain computer interface”  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 6032–6037  DOI: 10.1109/ICRA.2017.7989714
  * [6] Dylan P. Losey et al.  “Controlling Assistive Robots with Learned Latent Actions”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 378–384  DOI: 10.1109/ICRA40945.2020.9197197
  * [7] A.. Ijspeert, J. Nakanishi and S. Schaal  “Movement imitation with nonlinear dynamical systems in humanoid robots”  In _Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292)_ 2, 2002, pp. 1398–1403 vol.2  URL: https://ieeexplore.ieee.org/abstract/document/1014739
  * [8] Auke Jan Ijspeert et al.  “Dynamical movement primitives: learning attractor models for motor behaviors”  In _Neural computation_ 25.2 MIT Press, 2013, pp. 328–373
  * [9] Morgan Quigley et al.  “ROS: an open-source Robot Operating System”  In _ICRA workshop on open source software_ 3.3.2, 2009, pp. 5  URL: http://www.willowgarage.com/papers/ros-open-source-robot-operating-system
  * [10] P. Pastor, H. Hoffmann, T. Asfour and S. Schaal  “Learning and generalization of motor skills by learning from demonstration”  In _2009 IEEE International Conference on Robotics and Automation_ , 2009, pp. 763–768  URL: https://ieeexplore.ieee.org/abstract/document/5152385
  * [11] H. Hoffmann, P. Pastor, D. Park and S. Schaal  “Biologically-inspired dynamical systems for movement generation: Automatic real-time goal adaptation and obstacle avoidance”  In _2009 IEEE International Conference on Robotics and Automation_ , 2009, pp. 2587–2592  URL: https://ieeexplore.ieee.org/abstract/document/5152423
  * [12] P. Pastor, M. Kalakrishnan, L. Righetti and S. Schaal  “Towards Associative Skill Memories”  In _2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012)_ , 2012, pp. 309–315  URL: https://ieeexplore.ieee.org/abstract/document/6651537
  * [13] Clemente Lauretti, Francesca Cordella and Loredana Zollo  “A hybrid joint/Cartesian DMP-based approach for obstacle avoidance of anthropomorphic assistive robots”  In _International Journal of Social Robotics_ 11.5 Springer, 2019, pp. 783–796  URL: https://link.springer.com/article/10.1007/s12369-019-00597-w
  * [14] Dimitrios Papageorgiou, Antonis Sidiropoulos and Zoe Doulgeri  “Sinc-Based Dynamic Movement Primitives for Encoding Point-to-point Kinematic Behaviors”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 8339–8345  DOI: 10.1109/IROS.2018.8594479
  * [15] Adrià Colomé and Carme Torras  “Dimensionality Reduction for Dynamic Movement Primitives and Application to Bimanual Manipulation of Clothes”  In _IEEE Transactions on Robotics_ 34.3, 2018, pp. 602–615  DOI: 10.1109/TRO.2018.2808924
  * [16] S. Scherzinger, A. Roennau and R. Dillmann  “Inverse Kinematics with Forward Dynamics Solvers for Sampled Motion Tracking”  In _19th International Conference on Advanced Robotics (ICAR)_ , 2019, pp. 681–687  DOI: 10.1109/ICAR46387.2019.8981554
  * [17] Stefan Scherzinger, Arne Roennau and Rüdiger Dillmann  “Virtual Forward Dynamics Models for Cartesian Robot Control”, 2020  arXiv: https://arxiv.org/abs/2009.11888
  * [18] Pavel Holoborodko  “Smooth Noise Robust Differentiators”, http://www.holoborodko.com/pavel/numerical-methods/numerical-derivative/smooth-low-noise-differentiators/, 2008
  * [19] Sachin Chitta et al.  “ros_control: A generic and simple control framework for ROS”  In _The Journal of Open Source Software_ , 2017  DOI: 10.21105/joss.00456