11institutetext: L3S Research Center, Leibniz University, Hannover
11email<EMAIL_ADDRESS>
# Graph-based State Representation for Deep Reinforcement Learning

Vikram Waradpande  Daniel Kudenko  Megha Khosla

###### Abstract

Deep RL approaches build much of their success on the ability of the deep
neural network to generate useful internal representations. Nevertheless, they
suffer from a high sample-complexity and starting with a good input
representation can have a significant impact on the performance. In this
paper, we exploit the fact that the underlying Markov decision process (MDP)
represents a graph, which enables us to incorporate the topological
information for effective state representation learning.

Motivated by the recent success of node representations for several graph
analytical tasks we specifically investigate the capability of node
representation learning methods to effectively encode the topology of the
underlying MDP in Deep RL. To this end we perform a comparative analysis of
several models chosen from 4 different classes of representation learning
algorithms for policy learning in grid-world navigation tasks, which are
representative of a large class of RL problems. We find that all embedding
methods outperform the commonly used matrix representation of grid-world
environments in all of the studied cases. Moreoever, graph convolution based
methods are outperformed by simpler random walk based methods and graph linear
autoencoders.

##  1 Introduction

A good problem representation has been known to be crucial for the performance
of AI algorithms. This is not different in the case of reinforcement learning
(RL), where representation learning has been a focus of investigation. The
core idea is to map the high dimensional state space to low dimensional latent
representations which are more informative thus facilitating the learning of
an optimal policy. Previous works, for example, [6, 16] have focused on
learning these representations from the incoming high dimensional signals or
observations corresponding to a state but have ignored the fact that the
underlying stochastic decision process induces a topological structure over
the states which can provide additional useful features. In this work, we
focus on extracting informative low dimensional state features based on the
topological structure of the underlying Markov decision processes (MDPs) for
deep reinforcement learning.

Deep reinforcement learning combines neural networks with a reinforcement
learning architecture. In particular, rather than using a lookup table a
neural network is used to approximate a value function (and thus a policy)
without the need to store, index and update all possible states and their
values in look-up tables. While Deep RL approaches build much of their success
on the ability of the deep neural network to generate useful internal
representations, they nevertheless suffer from a high sample complexity. One
way to overcome this problem, is to start with a better input representation
that can improve the learning performance significantly. In this paper, we
exploit the fact that the underlying Markov decision process (MDP) represents
a graph and investigate the suitability of graph representation learning
approaches to learn effective encodings of the state. These encodings are then
used to enrich the sate representation for Deep RL and improve the speed of
learning.

Graph Representation learning (GRL) approaches [2, 21, 4, 20] aim to embed
nodes in a low dimensional space such that the topological structure of the
graph is preserved. Though these methods have gained popularity and showed
state of the art improvements in several graph analytical tasks like node
classification and link prediction, their suitability or generalizability to
different domains has escaped attention so far. Our work identifies Deep RL
for discrete MDPs as a promising application for utilizing and evaluating
graph representations. More specifically, we evaluate several unsupervised
representation learning methods on their ability to learn effective state
representations encoding the topological structure of MDP. With a large number
of unsupervised GRL approaches proposed to date we systematically choose
models from 4 GRL classes based on several criteria as elaborated in Section
3.2. We then investigate which approaches are best suited to be used as state
representation learning methods for MDPs in Deep RL.

While the node embeddings are computed from a given MDP, it is important to
note that in this work we do not assume that the entire MDP is known in
advance. Instead, we first generate an estimate of MDP by collecting random
samples and use the resulting graph to generate node representations in an
unsupervised manner. We do assume that the environment is discrete, since
otherwise it is not possible to represent the MDP as a finite graph. In the
case of continuous states, these could be discretised beforehand, either
manually or via an automated method analogous to tile coding.

Our Contributions. We evaluate GRL approaches for effectively encoding MDPs
under the assumption that the entire MDP is not known in advance. To the best
of our knowledge, our work is the first to propose the use of pretrained state
representations entirely on the topological structure of the MDP and show that
dense low dimensional input state representations enhance the performance of
Deep Q-Networks (DQN) for navigational tasks. To summarize our main
contributions are as follows.

  * •

We propose and evaluate a wide range of graph based representation learning
approaches to generate state features based on topological structure of MDPs,
leading to improved learning performance in DeepRL.

  * •

We show that RL is a promising application for evaluating and enhancing graph
representation learning approaches.

Key Findings. Our key findings and conclusions are

  * •

Pre-trained unsupervised low-dimensional state representations when used as
input to DQN shows state of the art improvements over the raw high dimensional
state input for grid-world environments.

  * •

For undirected MDPs, quite surprisingly, the first neural network based
representation learning method DeepWalk outperforms almost all other methods
including the more popular graph convolution based methods.

  * •

By varying the number of samples used to generate an approximate MDP we show
that the best performing embedding approaches show comparable performance even
with smaller number of samples.

  * •

For directed MDPs, preserving edge directionality while learning state
representations does not appear to be crucial. For instance DeepWalk, when
used to train representations while ignoring the edge directionality, showed
comparable performance to APP and NERD which generated representations in
directionality preserving manner.

##  2 Preliminaries and Related Work

###  2.1 Markov Decision Processes in RL

Markov decision processes (MDP) are discrete time stochastic control processes
which are used to formalize reinforcement learning problems and model RL
environments. MDPs are represented by a 4-tuple $(S,A,P,R)$, where $S$ is a
set of discrete states an agent can be in, $A$ is the set of all possible
actions that the agent can take, $P$ denotes a probability density function
with $P(s\textquoteright|s,a)$ being the transition probability of moving from
state $s$ to state $s^{\prime}$ after taking action $a$, and $R(s,a)$ is the
immediate reward that the agent receives when it takes action $a$ in state
$s$. The objective of reinforcement learning is then to determine the optimal
mapping of a given state to action, $\pi(s)$ (the policy) such that the chosen
action results in maximizing the expected sum of rewards received in the
future.

###  2.2 DQN

A widely used modern RL algorithm is Deep Q-Networks (DQN) [15]. DQN is the
"deep" expansion of Q-Learning [24] and uses essentially the same update rules
and operating principles as Q-Learning but adapted to use a neural network as
its value function representation.

Specifically, the DQN algorithm computes the Q value function (represented as
a deep neural network). This value function maps a state $s$ and action $a$
into an estimate of the expected cumulative reward for executing action $a$ in
state $s$ and following the optimal policy from then on. As the agent
interacts with its environment, the agent accumulates experience in the form
of $(s,a,r,s^{\prime})$ tuples, which are used to update the neural network
that computes the Q function. For each experience tuple, the values $Q(s,a)$
and $r+\max_{a^{\prime}}{Q(s^{\prime},a^{\prime})}$ are calculated using the
neural network. The difference between these two values is used in the loss
function to update the network. This "bootstrapping" method enables the agent
to learn strong estimates for the expected return of each state-action pair.

The main strength of DQN is that the deep neural network can generate useful
internal representations of environment states when provided with very simple
state representations. However, DQN suffers from a very high sample
complexity. The complexity can be reduced by generating a more effective state
representation before feeding it into the neural network, and this forms the
basis of the approach proposed in this paper.

###  2.3 Graph Representations

Graph representation learning (GRL) aims to learn low-dimension latent
representations of nodes to be used for downstream tasks such as link
prediction, node classification etc. GRL methods include random walk based
methods, matrix factorization based and graph neural networks (GNNs). Random
walk based [21, 7, 25] methods optimize the node embeddings so that nodes have
similar embeddings if they tend to co-occur on random walks over the graph.
Matrix factorization based [20] methods rely on low rank decomposition of a
target matrix such as the $k$-step transition probability matrix, modularity
matrix etc. to obtain node encodings. GNNs are deep learning models designed
to extract features from the graph structure as well as the input node
attributes and can be further categorized into recurrent graph neural networks
[23], convolutional graph neural networks [10, 17] and graph autoencoders [9,
22]. In spite of their success, there are limited studies on in-depth
comparative analysis [8, 3] of these methods over a wide range of datasets and
tasks. Moreover, none of these works focus on generalizability of embedding
approaches to encode graph structure beyond using the encodings in node/graph
classification or link prediction tasks.

###  2.4 State Representations in Reinforcement Learning

In reinforcement learning, it is a common practice to map the state(-action)
space to a low dimensional latent space where the main goal of such a
transformation is to represent the input data in a more informative form that
facilitates and improves subsequent steps. The authors in [19] and [14]
proposed the use of the Laplacian basis functions as state encodings. Basis
functions are derived by finding the “smoothest” eigen vectors (that
correspond to the smallest eigen values) of the graph Laplacian and is argued
that such smooth eigenvectors also reflect the smoothness of value functions
over nearby nodes. On the one hand eigenvalue decomposition is computationally
expensive and on the other hand the smoothness assumption for value functions
over the states might not be always valid as also observed in [13] . We note
that [13] also emphasize the use of node embeddings as basis functions to be
used in a generalized version of representation policy iteration (RPI). The
authors in [6] use priors about the structure of robotic interactions with the
physical world to learn state representations. Our work can be seen as
complementary in nature as we propose a more general approach of encoding such
an observed structure into state-action representations. Other works [1, 16]
which perform state representation learning and reinforcement learning
simultaneously have also ignored the fact that the underlying decision process
represents a graph and is therefore crucial to incorporate the topological
information for effective state representation learning. Recently [5] proposed
a graph convolutional network based approach to multi-agent reinforcement
learning. But it is not known a priori if the chosen graph learning model is
the best for the considered use cases. In this work we are evaluating the
capability of graph learning methods on effectively learning from the MDP
structure, focusing on discrete navigation problems (i.e. mazes) which are
representative for a large class of MDPs. The learnt representations can be
further refined and combined with more input features like sensor observations
corresponding to states for more specific cases and corresponding
architectures and objective functions for example in [1, 16] can be employed.

##  3 Comparative Analysis

Unsupervised GRL approaches aim to learn low dimensional representations for
each node while preserving certain topological characteristics of the
underlying graph. Informally, each of the methods tend to encode similar nodes
closer in the embedding space and the definition of similarity varies from
methods to method. Consequently we might expect that there cannot be a single
winner method which can encode all the types of graph structure well and the
choice of a particular method would depend on certain structural
characteristics of the underlying graph which might be application specific.
Therefore we choose a wide range of representative methods from several
classes of GRL methods and argue about their suitability to be used in
Reinforcement learning applications.

###  3.1 Compared Models

Let $G=(V,E)$ denotes the graph with $|V|$ nodes and $|E|$ edges corresponding
to the MDP estimated from the obtained random samples. We train low
dimensional vector representation or embedding, $\phi(v)$ for each node $v\in
V$ using the following GRL methods.

DeepWalk [21]. Inspired by techniques of language modelling and unsupervised
feature learning from word-sequences DeepWalk generates node-sequences from
graphs using short random walks and trains the Skip-Gram model using
hierarchical softmax akin to word2vec-based training procedure. The training
set is prepared by sampling vertex-context pairs over a sliding window in a
given random walk. In particular, it attempts to find node embeddings such
that the likelihood of observing a vertex $v_{i}\in V$ given its context (
i.e. other neighboring vertices within a specified window of the random walk)
is maximized. Let $\Phi$ denotes the latent representation matrix,
$\\{v_{i-w},\cdots,v_{i-1},v_{i+1},\cdots,v_{i+w}\\}$ is the set of neighbors
of $v_{i}$ in a given random walk within window size $w$, the optimization
problem then is

$\min_{\Phi}-\log\mathbb{P}(\left\\{v_{i-w},\cdots,v_{i-1},v_{i+1},\cdots,v_{i+w}\right\\}|\phi\left(v_{i}\right))$

APP [25]. As opposed to many embedding methods which preserve symmetric
proximities, which can be insufficient for some applications, here an
asymmetric proximity preserving (APP) method both for undirected and directed
graphs is proposed. It uses an approximate version of Rooted PageRank wherein
several paths are sampled from the starting vertex using a restart
probability. Each vertex $v\in V$ is given two representations, source and
target, denoted as ${\phi_{s}(v)}$ and $\phi_{t}(v)$ respectively. The two
representations are learnt such that the likelihood of a training pair $(u,v)$
in their respective source and target roles, $\mathbb{P}(v|u)$ is maximized.
The likelihood is modelled as softmax as follows and is optimized using
negative sampling.

$\mathbb{P}(v|u)=\frac{\exp({\phi_{s}(u)}\cdot{\phi_{t}(v)})}{\sum_{n\in
V}\exp(\phi_{s}(u)\cdot\phi_{t}(n)}$

NERD [7]. Node Embeddings Respecting Directionality (NERD) exploits the fact
that in directed graphs, the neighborhood of a node differs based on its role
as a source or a target (destination) node. To model node similarity while
preserving its role semantics NERD proposes an alternating random walk
strategy similar to SALSA[11] which alternates between source nodes (hubs) and
target nodes (authorities). Two embeddings per node are learnt such that the
probability of observing the sampled neighbors (in their respective roles)
from an alternating random walk is maximized. It differs from APP in the kind
of the random walk employed to collect training pairs and is more suitable for
graphs with prominent hub authority structure.

HOPE [20]. High-Order Proximity preserved Embeddings or HOPE for short, is an
embedding framework also designed for directed graphs and based on finding
source and target node embeddings while optimizing for various high-order
proximity measures exist like Katz Proximity, Personalized Pagerank, Common
Neighbour measure, Adamic-Adar, etc. In particular it finds a low rank
decomposition of a proximity measure (in this work we use Katz proximity)
where the two factors correspond to source and target embeddings of a node. If
$S$ represents the Katz proximity matrix, HOPE learns source and target
representation matrices $\mathbf{U^{s}}$ and $\mathbf{U^{t}}$ while optimizing
the following objective

$\min_{\mathbf{U^{s}},\mathbf{U^{t}}}{||\mathbf{S}-\mathbf{U^{s}}\cdot\mathbf{(U^{t})}^{\top}||^{2}}.$

Unsupervised GraphSage [4]. GraphSage belongs to the family of Graph
convolution based models [10] which incorporate neighborhood aggregation
mechanism in the learning algorithm to generate node representations. In this
work we use GraphSage in an unsupervised setting, using a graph-based loss
which encourages nearby nodes to have similar representations, and far away
nodes are enforced a distinct representation. In particular, it samples node
pairs as in DeepWalk and uses the negative sampling based loss to embed
vertices occuring together in short random walks closer.

Graph Autoencoders. This family of models aim to map each node to a vector in
a lower dimension, from which reconstruction of the adjacency list of node
should be possible. While Graph autoencoders using graph convolution
networks(GCNs)[10] as encoders are quite popular, recent empirical evidence
[22] suggests that the GCN based encoders can be replaced by linear models
without compromising performance in various downstream tasks. Though we
compare various graph autoencoder models, namely GCN Autoencoder[9], GCN
Variational Autoencoder[9], Linear Autoencoder, Linear Variational
Autoencoder, and the deep versions of these autoencoders, we present results
only corresponding to Graph Linear Autoencoder (Glae) as it showed much
superior performance.

###  3.2  Rationale behind chosen Methods

Node Classification and State value functions. We choose DeepWalk in our study
because of its robust performance in generating unsupervised node
representations suitable for node classification task as shown in [8]. Note
that as the task of finding an optimal policy is equivalent to finding optimal
value function over nodes, one can interpret value functions as continuous
forms of classes where similar nodes would have similar value functions. We
hypothesize that methods performing well on node classification tasks would
also generate embeddings suitable for approximating the optimal policies well.
Our hypothesis is validated by our empirical results in which DeepWalk turns
out to be the best performing method for all studied undirected MDPs.

Asymmetric Local Neighborhoods. A crucial aspect for node representations in
MDPs which is ignored by several representation learning methods could be the
existence of asymmetrical local neighborhoods even for the cases when the
underlying mazes are undirected. APP learns two embeddings per vertex in its
source and destination roles respectively, stressing the fact that a node $x$
is similar to node $y$ in its destination role if $y$ is reachable from $x$ in
a small number of steps. As placement of obstacles can induce asymmetrical
local structures in MDPs, we chose APP as one of the compared models.
Moreover, APP is a general method also applicable for directed graphs.

Homophily and Graph Convolution Based Methods. Recent theoretical works[12,
18] imply that graph convolution operation is a special form of Laplacian
smoothing which mixes the features of a vertex and its nearby neighbors. The
smoothing operation makes the features of vertices in the same cluster
similar, thus greatly easing the classification task, which is the key reason
why GCNs work so well in node classification tasks. Moreover the empirical
evidence [8] further supports the fact the GCN based models best perform for
high homophily networks where labels do not vary a lot over adjacent vertices.
We argue that the real world decision processes do not always give rise to
high homophilic graphs, in the sense two adjacent nodes might not always have
similar value functions. In this work we experiment with unsupervised
GraphSage and several GCN based autoencoders.

Encoding Directionality. For directed MDPs, it is crucial to take into account
of the edge directions to decide on a favourable policy. In this work we
therefore compare three methods specifically designed to learn unsupervised
node representations for directed graphs. We chose APP, NERD and HOPE which
exploit different characteristics of directed graphs. Moreover, we also
compare these methods with DeepWalk (used while ignoring the edge direction)
to verify the importance of taking into account of the edge directionality
while learning input state representations.

##  4 Experiments and Results

###  4.1 Experimental Setup

Grid-world Domains. The experiments were carried out on various grid-world
domains, more specifically $20\times 20$ or a 400 state square grids. These
tasks are representative of a large class of discrete RL environments, and
thus provide a suitable testbed. The domain has four walls, some obstacles, a
start-state and a reward-state. The goal of the agent is to navigate the grid
and reach the reward state. At each step, the agent gets some reward between
$-1$ and $1$. Moreover, the obstacles in the domain are soft, meaning the
agent can get into an obstacle state, but it incurs a heavy negative reward.
The agent gets a reward of $+1$ if it reaches the reward state, $-0.3$ if it
goes into an obstacle, and $-0.1$ if it bumps into a wall along the edge of
the maze. Additionally to motivate the agent to find the shortest path, every
valid move of the agent has a small negative reward of $-0.01$. We compare all
GRL methods with a matrix representation of the grid-world where a grid is a
$20\times 20$ matrix with all valid states represented by $1$, all obstacles
by $0$, the target by $0.75$ and current agent-location by $0.75$.

Additionally, we tested our agent on two different types of grid-worlds. One,
in which it resembles an undirected graph, meaning the agent can go into any
of the empty neighbouring cells. The other type are the ones in which the
agent’s movements are restricted, adding directionality to the MDP graph. We
describe different mazes as well as the corresponding results in Section 4.2.

Model architecture employed for DQN. The neural network has an input layer,
which takes in the embedding $\phi(s)$ of dimension $d$ of the current state
of the agent. For the matrix representation, the input layer is a flattened
matrix corresponding to the grid. Further, there are two hidden layers with
tanh activation, and then an output layer (with tanh activation) with four
neurons. Each neuron corresponds to one of the four directions that the agent
can move, and its value represents the state-action value of taking the
respective action. At each time-step, the agent chooses the direction having
the maximum value as predicted by the network. The loss function for the
network is Mean Squared Error (MSE) loss.

Hyperparameter Settings for DQN. During the training of the network, the agent
performs episodes from the starting state till either it reaches the end
state, or reaches a threshold of negative reward of $-25$. This high negative
threshold ensures that the agent explores the maze long enough to learn. After
completing an episode, the agent is reset to the initial state. The agent
learns using experience replay. So during the agent’s episodes, it’s steps are
stored in a memory in the form of $(s_{t},a,r_{t},s_{t+1})$ (where the symbols
have usual meaning). Moreover, the ’experience-gaining’ phase is separate from
the ’learning phase’, even though they are interleaved. After each time step,
random previous actions from the memory are selected and are used in the
learning phase.

Hyperparameter Settings for GRL Methods. The representation learning phase
requires the graph as an input in either edgelist or adjacency list format.
For DeepWalk the walk-length, and window-size were set to $20$ and $5$
respectively. For GraphSage, the GCN mean aggregator was used. For all other
methods like APP, NERD and Glae, the default hyperparameters for these
algorithms were used. In methods like APP, NERD and HOPE for example, which
generate two embeddings, each of dimension $d$ per node of a graph, the
embeddings of current state and the next state (corresponding to a particular
action) are concatenated to form an input embedding of dimension $2d$. The
rest of the architecture of the network remains the same.

Evaluation Metrics We used standard metrics for evaluation which are, average
cumulative reward received by the agent against number of time-steps, and
against the number of episodes completed by the agent. As discussed before,
the goal of the agent is to accumulate the maximum reward from the grid. The
better the method, the quicker the agent reaches the target and the more
reward it accumulates. We argue that a better method would represent the input
state-space better, and hence the agent would accumulate more reward. Another
way to look at this is that a better input state representations would result
in less number of episodes, or less number of steps to find out the optimal
path to reach the target. With this in mind, the different domains which were
used, and the plots of the metrics discussed above are presented in the next
section.

###  4.2 Results on Sampled Complete Grid Environments

In this section we describe the results for different types of mazes. Here we
assume that enough number of samples were drawn in order to build the complete
mazes. In addition the empirical time complexity of maze construction from the
acquired samples is roughly $1\%$ of the time for training the deep network.
We observe that for all the studied mazes the pre-trained state
representations outperform the matrix based state representation. We note that
overall we compared $9$ GRL methods out of which results for $6$ are presented
in this section. The other three methods, which are GCN based shallow and deep
autoencoders showed worse results than the others and hence we don’t show
their performance here.

As already mentioned we plot the standard evaluation metrics: (i) average
cumulative reward against the number of time-steps (ii) average cumulative
reward against the number of episodes with $90\%$ confidence intervals. For
the results presented in this section we run all methods for 60 episodes and
repeat experiments for atleast 40 iterations. We note that in the plots
corresponding to time steps, some of the methods stop much earlier than the
others, which indicates that they have shorter episodes, which in turn
indicates that they were much faster in finding the optimal policy and
achieving higher rewards much earlier.

####  4.2.1 Maze 1.

The obstacle and the start state are in farthest rows of the maze.The results
are presented in Figures 1 and 2. DeepWalk and APP outperform other GRL
methods. While Figure 1 shows the cumulative reward against time-steps, Figure
2 shows it against number of episodes. For the rest of the mazes we will stick
only to the step-wise plots as they clearly are more informative than the
episode-wise plots.

Figure 1: Maze 1 against its plot of the average episode-wise reward obtained
with a 90% confidence interval. Figure 2: Episode-wise rewards obtained for
maze 1

####  4.2.2 Maze 2.

In this maze the agent is expected to find a longer but less convoluted path
than in maze 1. The results for this domain are presented in Figure 3. Once
again we observe that DeepWalk is the best performing method followed by APP
and Glae.

Figure 3: Maze 2 against its plot of the step-wise reward obtained.

####  4.2.3 Maze 3.

This domain is very similar to maze 2 in that the obstacles are exactly at the
same place, but the locations of the start state and the target are changed.
The results are presented in Figure 4. As opposed to Maze 1, Glae outperforms
APP and performs comparable to DeepWalk.

Figure 4: Maze 3 against its plot of the step-wise reward obtained.

####  4.2.4 Mazes 4 and 5.

Mazes 4 and 5 are directed in nature. Maze 4 is like Maze 3 in terms of
obstacles location, but only down and left actions of agent are permitted.
Maze 5 is similar to Maze 1, but with $15\%$ of the possible actions removed
randomly. For both these mazes, we used methods like HOPE and NERD, which are
explicitly designed for directed graphs. We additionally compared them with
DeepWalk (while ignoring the edge directionality) to investigate if the edge
directionality has a large impact on state representations. In Maze 4, APP and
DeepWalk are the best performing methods followed by NERD, whereas in Maze 5,
APP and NERD outperform DeepWalk.

Figure 5: Step-wise reward obtained for maze 4 and maze 5

####  4.2.5 Main Observations.

With respect to our results in this section, we make the following main
observations and conclusions.

  1. 1.

For undirected mazes DeepWalk is the best method. APP and Glae are also
competitive methods but their performance varies over different mazes. All
these methods outperform GraphSage and the raw matrix input representation.

  2. 2.

The graph convolution based method,GraphSage is the worst performing method
for all undirected mazes among the GRL methods. We remark that in our initial
experiments we had also compared GCN based shallow and deep autoencoders and
found that the graph linear autoencoder was the better performing model. We,
therefore, conclude that GCN based methods are not the best choice for
learning unsupervised state representations in MDPs in absence of external
node features.

  3. 3.

For directed mazes, APP is the best performing method followed either by
DeepWalk or NERD. The good performance of DeepWalk in these mazes is a bit
surprising and points to the fact that the directionality of the underlying
MDPs (at least for the studied environments) can be ignored as far as learning
state representations is concerned.

  4. 4.

We also observe that the best performing methods also incur shorter episodes
supporting the fact that a better input state representation not only improved
the learning quality but also made it faster.

###  4.3 Sensitivity to Embedding Dimension

In this section we study the effect of dimension of embeddings on learning
performance of the DQN. In principle, we want the dimension of the embedding
space to be considerably smaller than the original dimension of the state-
space. In Figure 6 we consider four different dimensions for 4 different
methods. For all these methods, we observe a diminishing return after a
dimension of size 30. While increasing the dimension would up to some extent
also improve performance, but it would incur additional time cost of training
extra parameters. For APP and DeepWalk we observe that a higher number of
dimensions only lead to smaller episodes.

(a) APP

(b) DeepWalk

(c) Glae

(d) GraphSage

Figure 6: Sensitivity to dimensions. All results correspond to experiments of
Maze 1 using different embedding dimensions, $d\in\\{20,30,40,50\\}$.

###  4.4 Sensitivity to Sample Size

As described previously, in all the results until now, we assume that enough
samples were drawn to build the complete maze. In this section, we investigate
how sample size affects the learning. On average, to reconstruct a $20\times
20$ maze by drawing random samples, approximately $8000$ such samples are
required. The results are shown in Figure 7. The purple line shows the
cumulative reward when the complete maze is used to learn state
representations. From the plots, we see that the green line, corresponding to
$1000$ samples performs the worst, and the cumulative rewards never start to
rise. On increasing the sample sizes, the performance incrementally improves.
For GraphSage and Glae in particular, even with a small sample size of $2000$
show performance comparable to collecting $8000$ samples, though their overall
performance is worse than APP and DeepWalk.

(a) APP

(b) DeepWalk

(c) Glae

(d) GraphSage

Figure 7: Sensitivity to sample size. All results correspond to experiments of
Maze 1 constructed using different sizes of random samples,
$S\in\\{1000,2000,4000\\}$. The purple line corresponds to when complete maze
was used.

##  5 Conclusion

In this work we propose and evaluate a wide range of graph based
representation learning approaches to generate state features based on
topological structure of MDPs, leading to improved learning performance in
navigational tasks. In particular, we conducted an empirical study over a wide
range of unsupervised graph representation learning methods and conclude that
a random walk based method DeepWalk is best suited for generating low
dimensional state representations based on the topological structure of the
underlying MDP. For directed MDPS, APP and NERD are competitive methods and
can offer advantages in encoding specific directed MDPs. We also find that the
more popular GCN based models are the worst performing among the compared
models. We also performed parameter sensitivity experiments where we
investigated the effect of (i) increasing embedding dimension (ii) increasing
the sample sizes to generate the MDP. We observe that though increasing
dimensions improve the learning performance, a small value of $30$ for
embedding dimensions already suffices to obtain competitive performance. This
is important for DQNs where the number of network parameters increase with the
number of input embedding dimensions. We also show that the representation
learning methods also perform comparably to their best performance (computed
over the complete MDP) while using a much smaller number of random samples to
build an estimate of the MDP. Our work also shows that RL can serve as a
promising application and test bed for graph representation learning
approaches.

## References

  * [1] de Bruin, T., Kober, J., Tuyls, K., Babuška, R.: Integrating state representation learning into deep reinforcement learning. IEEE Robotics and Automation Letters 3(3), 1394–1401 (2018)
  * [2] Cai, H., Zheng, V.W., Chang, K.: A comprehensive survey of graph embedding: problems, techniques and applications. IEEE TKDE (2018)
  * [3] Errica, F., Podda, M., Bacciu, D., Micheli, A.: A fair comparison of graph neural networks for graph classification. In: ICLR (2020)
  * [4] Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: NIPS. pp. 1024–1034 (2017)
  * [5] Jiang, J., Dun, C., Lu, Z.: Graph convolutional reinforcement learning for multi-agent cooperation. arXiv preprint arXiv:1810.09202 2(3) (2018)
  * [6] Jonschkowski, R., Brock, O.: Learning state representations with robotic priors. Autonomous Robots 39(3), 407–428 (2015)
  * [7] Khosla, M., Leonhardt, J., Nejdl, W., Anand, A.: Node representation learning for directed graphs. In: ECML-PKDD (2019)
  * [8] Khosla, M., Setty, V., Anand, A.: A comparative study for unsupervised network representation learning. IEEE Transactions on Knowledge and Data Engineering (2019)
  * [9] Kipf, T.N., Welling, M.: Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016)
  * [10] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: ICLR (2017)
  * [11] Lempel, R., Moran, S.: The stochastic approach for link-structure analysis (salsa) and the tkc effect. In: Proceedings of the 9th International World Wide Web Conference on Computer Networks : The International Journal of Computer and Telecommunications Networking. pp. 387–401 (2000)
  * [12] Li, Q., Han, Z., Wu, X.M.: Deeper insights into graph convolutional networks for semi-supervised learning. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
  * [13] Madjiheurem, S., Toni, L.: Representation learning on graphs: A reinforcement learning application. AISTATS (2019)
  * [14] Mahadevan, S., Maggioni, M.: Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research 8(Oct), 2169–2231 (2007)
  * [15] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.A.: Playing atari with deep reinforcement learning. CoRR abs/1312.5602 (2013), http://arxiv.org/abs/1312.5602
  * [16] Munk, J., Kober, J., Babuška, R.: Learning state representation for deep actor-critic control. In: 2016 IEEE 55th Conference on Decision and Control (CDC). pp. 4667–4673. IEEE (2016)
  * [17] Niepert, M., Ahmed, M., Kutzkov, K.: Learning convolutional neural networks for graphs. In: ICML. pp. 2014–2023 (2016)
  * [18] NT, H., Maehara, T.: Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:1905.09550 (2019)
  * [19] Osentoski, S., Mahadevan, S.: Learning state-action basis functions for hierarchical mdps. In: ICML. pp. 705–712 (2007)
  * [20] Ou, M., Cui, P., Pei, J., Zhang, Z., Zhu, W.: Asymmetric transitivity preserving graph embedding. In: SIGKDD. pp. 1105–1114. ACM (2016)
  * [21] Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: Online learning of social representations. In: SIGKDD. pp. 701–710. ACM (2014)
  * [22] Salha, G., Hennequin, R., Vazirgiannis, M.: Keep it simple: Graph autoencoders without graph convolutional networks. Workshop on Graph Representation Learning, 33rd Conference on Neural Information Processing Systems (NeurIPS) (2019)
  * [23] Scarselli, F., Gori, M., Tsoi, A.C., et al.: The graph neural network model. IEEE Transactions on Neural Networks (2008)
  * [24] Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 2 edn. (2017)
  * [25] Zhou, C., Liu, Y., Liu, X., Liu, Z., Gao, J.: Scalable graph embedding for asymmetric proximity. In: AAAI. pp. 2942–2948 (2017)


# Dance Generation by Sound Symbolic Words

Miki Okamura<EMAIL_ADDRESS>0009-0001-9367-1078 University of TsukubaTsukubaIbarakiJapan305-0821 ,  Naruya
Kondo<EMAIL_ADDRESS>University of
TsukubaTsukubaIbarakiJapan ,  Tatsuki Fushimi  University of
TsukubaTsukubaJapan<EMAIL_ADDRESS>,  Maki Sakamoto
The University of Electro-CommunicationsChofuTokyoJapan and  Yoichi Ochiai
University of TsukubaTsukubaIbarakiJapan

(2023)

###### Abstract.

This study introduces a novel approach to generate dance motions using
onomatopoeia as input, with the aim of enhancing creativity and diversity in
dance generation. Unlike text and music, onomatopoeia conveys rhythm and
meaning through abstract word expressions without constraints on expression
and without need for specialized knowledge. We adapt the AI Choreographer
framework and employ the Sakamoto system, a feature extraction method for
onomatopoeia focusing on phonemes and syllables. Additionally, we present a
new dataset of 40 onomatopoeia-dance motion pairs collected through a user
survey. Our results demonstrate that the proposed method enables more
intuitive dance generation and can create dance motions using sound-symbolic
words from a variety of languages, including those without onomatopoeia. This
highlights the potential for diverse dance creation across different languages
and cultures, accessible to a wider audience. Qualitative samples from our
model can be found at: https://sites.google.com/view/onomatopoeia-dance/home/.

motion generation, machine learning, datasets, sound symbolic words,
onomatopoeia

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock,
NY††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Computing methodologies Animation††ccs: Computing methodologies Machine
learning Figure 1. The dance motion generated by our model: By providing a
creative onomatopoeia (in this case, ”Bulber”) as input, we can generate a
dance that suits it. The input onomatopoeia is embedded in terms of the
intensity of 43 adjective expressions by the Sakamoto system (Doizaki et al.,
2016) and is input into a deep learning-based dance generation model called
FACT (Li et al., 2021), which generates the dance. A zero vector is input
during times when no onomatopoeia is provided.

##  1\. Introduction

Dance is a universal form of expression that goes beyond the limits of
language, culture, and geography. As an essential aspect of human history,
dance has continually evolved with advancements in technology. One such
development is the incorporation of computer graphics and animation in the
automatic generation of dance. This combination of technology and creativity
has the potential to offer new perspectives, support the creation of dance,
and reduce effort. Furthermore, it contributes to developing diverse movements
for interactive characters in video games and virtual reality applications.
With the progress of technology, machine learning techniques have been
introduced to the field of dance generation. Traditional text-based dance
generation methods can only handle expressions that exist in language, making
it challenging to incorporate new sounds and sensations not represented in
existing language. Consequently, these methods face difficulties in generating
unique dance motions that go beyond the constraints of language and culture,
limiting creativity and diversity. On the other hand, music-based dance
generation methods, while rich in rhythm, may not effectively convey meaning.
Moreover, these approaches are not easily accessible to users who cannot
create music themselves.

We propose the utilization of onomatopoeia as a effective and innovative input
source for a machine learning model aimed at generating dance motion.
Onomatopoeia possesses the unique ability to convey both rhythm and meaning
through abstract word sense expressions, making it an ideal candidate for this
purpose. Its versatile nature allows it to be employed in various contexts,
such as describing the texture of a meal, the touch of a cat, the speed of a
car, or the ambiance of a comic strip, emphasizing its broad applicability and
potential for enhancing the dance generation process. Building upon the
framework of an existing study named AI Choreographer (Li et al., 2021), we
adapt this method to incorporate onomatopoeia as input for generating dance
motion. In contrast to text and music inputs, we utilize the Sakamoto system
(Doizaki et al., 2016) for extracting features from onomatopoeia. The system
is a deterministic feature extraction method focusing on phonemes and
syllables, representing onomatopoeia through 43 numerical values corresponding
to the strength of adjective expressions. This system can be used with any
language containing phonemes, as it is not influenced by linguistic meaning.
Combining the Sakamoto system with a deep generative model is a novel
approach, as there are few examples of such implementation in the literature.
We collected 44 pairs of onomatopoeia and dance motions through a user survey
and trained the model on this dataset. The results of our study indicate that
the proposed model successfully generates dance motions that correspond with
the input onomatopoeia. Furthermore, we demonstrate that the model can
generate dances using sound-symbolic words from various languages, including
those without onomatopoeia, allowing for more intuitive dance generation.
However, it must be acknowledged that the quality of the generated motions is
not on par with the latest state-of-the-art dance generation methods. Future
research and development can focus on model improvements, dataset expansion,
and the development of interactive applications for onomatopoeia-based dance
motion generation.

This paper’s contributions are as follows:

  * •

We introduce a novel approach using onomatopoeia as input for dance motion
generation.

  * •

We adapt the AI Choreographer framework and employ the Sakamoto system for
feature extraction from onomatopoeia.

  * •

We present a new dataset of 44 onomatopoeia-dance motion pairs collected from
a user survey.

  * •

Our study demonstrates the successful generation of dance motions using sound-
symbolic words from various languages, including those without onomatopoeia.

  * •

The results indicate potential for intuitive and diverse dance creation across
different languages and cultures.

| input | output or function
---|---|---
Crnkovic-Friis et al. (2016) (Crnkovic-Friis and Crnkovic-Friis, 2016) | Text and Style | Dance
EDGE (2022) (Tseng et al., 2022) | Music | Dance
AI Choreographer (2021) (Li et al., 2021) | Music | Dance
Ours | Sound Symbolic Words | Dance
Yamagata et al. (2019) (YAMAGATA et al., 2019) | Sound Symbolic Words | Image retrieval
Sakamoto et al. (2017) (Sakamoto and Watanabe, 2017) | Sound Symbolic Words | Material retrieval
Doizaki et al. (2016) (DOIZAKI et al., 2016) | Sound Symbolic Words | Medical conditions
Table 1. Overview of related works. We have summarized studies that are
closely related to our research, involving dance as the output and sound
symbolic words as the input. Currently, we are exploring new combinations
within these fields.

##  2\. Related Work

###  2.1. Human Motion Generation

The problem of human motion generation has been extensively studied in the
fields of computer vision, graphics, and robotics. Early approaches focused on
employing statistical models, such as kernel-based probability distribution
(Pullen and Bregler, 2000; Bowden, 2000; Galata et al., 2001; Brand and
Hertzmann, 2000), and motion matching techniques (Holden et al., 2020) to
synthesize motion. However, these methods tended to abstract away motion
details and were primarily restricted to simple domains like locomotion. The
advent of deep learning has led to a surge of interest in exploring the
applicability of neural networks for generating 3D motion by training on
large-scale motion capture datasets. Various network architectures have been
investigated, including CNNs (Holden et al., 2015, 2016), GANs (Hernandez et
al., 2019), RBMs (Taylor and Hinton, 2009), RNNs (Fragkiadaki et al., 2015;
Aksan et al., 2019; Jain et al., 2016; Ghosh et al., 2017; Chiu et al., 2019;
Du et al., 2019; Wang et al., 2019; Butepage et al., 2017; Villegas et al.,
2018), and Transformers (Aksan et al., 2020; Bhattacharya et al., 2021; Ling
et al., 2020; Won et al., 2022). While these approaches have demonstrated the
capability to generate more diverse and realistic human motions than earlier
methods, they still face challenges in capturing the physical laws governing
human movement. MDM (Tevet et al., 2022) is a motion generation method using
diffusion models (Ho et al., 2020), which has become capable of producing
quite human-like movements. These motion generation models, while skilled at
producing common human actions, are not well-suited for generating expressive
and creative motions. Generating human motion conditioned on various inputs
has also become an active area of research. Studies have explored conditioning
on joystick control (Ling et al., 2020), class labels (Guo et al., 2020;
Petrovich et al., 2021), text descriptions (Petrovich et al., 2022; Zhang et
al., 2022), and seed motions (Duan et al., 2022; Holden et al., 2016; Rempe et
al., 2021; Yin et al., 2022). These methods enable motion generation under
various control conditions. However, unlike existing researches, our study
focuses on generating motions conditioned by sound-symbolic words, which is
the first attempt of its kind.

###  2.2. Dance Generation

The challenging task of generating dance motions that are stylistically
faithful to input music has attracted the attention of many researchers. Early
approaches in this domain followed a motion retrieval paradigm (Fan et al.,
2011; Lee et al., 2013; Ofli et al., 2011) or used optimization-based
approaches (Tendulkar et al., 2020) to generate 2D pose skeletons from
conditioning audio. However, these methods often resulted in unrealistic
choreographies that lacked the complexity of human dances. More recent works
have shifted towards synthesizing motion from scratch by training on large
datasets. Various modeling approaches have been proposed like LSTMs (Alemi et
al., 2017; Tang et al., 2018; Yalta et al., 2019; Zhuang et al., 2020; Kao and
Su, 2020), GANs (Lee et al., 2019; Sun et al., 2020; Ginosar et al., 2019),
convolutional sequence-to-sequence models (Ahn et al., 2020; Ye et al., 2020),
normalizing flow (Valle-Pérez et al., 2021), transformer (Huang et al., 2020,
2022; Kim et al., 2022; Li et al., 2022, 2021; Siyao et al., 2022) and
diffusion models (Tseng et al., 2022). Among these methods, recent
developments like AI Choreographer (Li et al., 2021) and EDGE (Tseng et al.,
2022) have started to generate diverse and human-like dances based on music
inputs. However, because these methods require music for creating dance, they
are not suitable for those who simply want to generate dances without
providing music. Moreover, it has been challenging to casually edit input data
or output dances, preventing anyone from intuitively generating dances with
ease. While EDGE (Tseng et al., 2022) has implemented editing features to
simplify dance generation, enabling partial masking and generating under
certain constraints, it can be difficult to master without some dance
experience. On the other hand, we propose using sound-symbolic words as the
input, allowing anyone to easily generate dances by merely entering intuitive
words.

###  2.3. Leveraging Onomatopoeia Representation

In recent years, there has been a growing interest in research on utilizing
onomatopoeic representations for applications such as search and prediction.
Onomatopoeic words can represent abstract concepts such as the texture and
feeling of an object or the rhythm of a sound, which are unique features not
present in other forms of expression like language or music. Consequently,
onomatopoeia can be considered an ideal medium for handling information in a
more sensory-oriented manner.

Several studies have explored the use of onomatopoeia for data visualization
and search, such as (YAMAGATA et al., 2019; Doizaki et al., 2013; Sakamoto and
Watanabe, 2017; Doizaki et al., 2015; DOIZAKI et al., 2016; Sakamoto et al.,
2021). For instance, (YAMAGATA et al., 2019) proposes an image retrieval
system that uses sound-symbolic words to represent surface texture, while
(Doizaki et al., 2013) develops a color design support system that quantifies
images using onomatopoeia to recommend suitable colors for intuitive and
ambiguous design images. (Sakamoto and Watanabe, 2017) introduces a method for
selecting material samples using sensory vocabulary, effectively exploring the
tactile perceptual space. (Doizaki et al., 2015) presents a product search
system that uses onomatopoeia to express rich product textures and rank
related images accordingly. (DOIZAKI et al., 2016) constructs a system that
proposes metaphors corresponding to the onomatopoeia expressing medical
conditions, enhancing the quality of medical interviews. (Sakamoto et al.,
2021) develops a system for estimating multidimensional personality traits
from a single sound-symbolic word, contributing to the prediction of
personality evaluations and the efficiency of communication. Furthermore,
(Yamagata et al., 2021) proposes a deep convolutional neural network
(DCNN)-based computer vision method that generates onomatopoeic expressions of
material textures from input images, demonstrating human-like texture
representation through onomatopoeia. In addition, (Nozaki et al., 2021)
introduces a game system that incorporates onomatopoeia as a game controller,
comparing and evaluating the user experience with traditional controllers and
confirming an improved sense of enjoyment and novelty. While there are several
search and generation methods using onomatopoeia, to the best of our
knowledge, there have been no attempts at content creation utilizing
onomatopoeia. This study presents the first endeavor in onomatopoeia-based
generation, aiming to extract the great potential of onomatopoeia as a form of
expression.

##  3\. Pilot Study and Data Collection

###  3.1. Questionnaire survey

To confirm the relationship between sound symbols a.k.a onomatopoeia and dance
choreography and the possibility of generating dance movements with
onomatopoeia, we conducted a questionnaire on dance and onomatopoeia among
experienced dancers before and after the data collection experiment.In
preliminary survey, participants answered the following questions: ”Do you use
onomatopoeia during dance practice? The options for whether or not they use
onomatopoeia were ”use frequently,” ”use often,” ”occasional use,” and ”I
don’t use it”. After the annotation experiment, participants were asked if it
was easy for them to annotate the onomatopoeia to the dance choreography. The
options were ”fairly easy,” ”easy,” ”somewhat easy,” ”normal,” ”somewhat
difficult,” ”difficult,” and ”fairly difficult. Finally, respondents answered
free-form questions about their impressions and findings of the annotations.

#### Questionnaire results

Figure 2. A questionnaire that asked respondents if they use onomatopoeia during dance practice in the pre-survey. Many respondents answered that they use onomatopoeia ”often” or ”often. Because one person did not respond, the result is the result of 15 respondents. |  Figure 3. Post questionnaire asked whether it was easy to imagine onomatopoeia for the dance choreography. Many responded ”somewhat difficult” or ”difficult”.
---|---

In the questionnaire asking whether they use onomatopoeia during dance
practice, 62.5% answered ”often” and 32.3% answered ”yes,” and in the
questionnaire asking about the degree of difficulty of annotation, 47.1%
answered ”somewhat difficult” and 23.5% answered ”difficult. The results
indicated that although they may use onomatopoeia spontaneously during dance
practice, it is difficult to imagine onomatopoeia for each movement

###  3.2. Data collection

In order to construct a dataset of dance and onomatopoeia to be used for
training data and to explore the relationship between dance and onomatopoeia,
we conducted a subject experiment in which we collected data on what kind of
onomatopoeia was recalled in response to what kind of choreography and asked
participants to describe their impressions and findings. Japanese speakers
aged 18-69 with dance experience were asked to annotate the onomatopoeia they
recall in response to dance videos.

Data on onomatopoeia for dance was collected by selecting silent dance videos
mainly in the genre of the subjects’ experience and adding onomatopoeia to the
dance choreography as subtitles using Youtube Studio’s subtitle creation
function. Dance videos from the front viewpoint of Advanced Dance in the AIST
Dance Video Database(Tsuchida et al., 2019) were used to generate a variety of
dance motions. In the annotations, the following instructions were given:
“Annotate the onomatopoeia that came to you intuitively rather than carefully,
and divide the dance movements into as many smaller pieces as possible to make
the onomatopoeia. Separate onomatopoeia and dance movements as finely as
possible. Annotate the choreography with onomatopoeia that comes to mind
freely. You are free to create your own onomatopoeia, not limited to existing
common vocabulary.”

During the one-hour experiment, the participants were asked to annotate one to
five dance videos, depending on their progress. This subject experiment
yielded the annotation data for a total of 44 dance videos in 10 dance genres.

##  4\. System

Figure 4. _Learning model_ The onomatopoeia for dances collected in the
subject experiments are quantified using Sakamoto System, which quantifies the
fine impressions of each onomatopoeia using a scale of 43-dimensional
adjective pair scale, and the data set is prepared as one onomatopoeia
sequence per one dance, which became input dataset with dance motions(Li et
al., 2021). the output is the dance motion conditioned on the onomatopoeia.

In this study, we collected annotation data for AIST Dance Database(Tsuchida
et al., 2019), using the method described in section 3. The onomatopoeia
collected in the subject experiments was quantified using sound symbol
estimation system which we call ‘Sakamoto System’. The quantified onomatopoeia
were arranged in a time series and converted into a series vector to be used
during training. AIST++, which is three-dimensional motion data corresponding
to the AIST Dance Database, and the series vectors were trained on the FACT
model, and the motion was output. The following is a detailed description of
the methods and techniques used to create the datasets treated as training
data.

###  4.1. Preparation and pre-processing of training data

####  4.1.1. Preparation

In preprocessing, the collected onomatopoeia is converted into a
43-dimensional series vector corresponding to the dance motion and made into
sequence data. First, all the annotated onomatopoeia described in the caption
file created using the subtitle editor of the video service Youtube in the
subject experiment of the 3 chapter are extracted, and the impression of the
onomatopoeia is quantified by Sakamoto System. The system converted each
onomatopoeia into a 43-dimensional numerical vector, and created an
onomatopoeia quantification dictionary that describes the 43-dimensional
numerical impression of each onomatopoeia. Next, an array of 43 dimensions per
index, each with 0 elements, is created, with the series length of the
onomatopoeia series vector being the series length of the music corresponding
to the dance motion in the previous research(Li et al., 2021) implementation.

Next, the onomatopoeia of the input is quantified from the numeric dictionary,
and the time (in seconds) when each onomatopoeia is given as a subtitle is
multiplied by fps=60 to obtain the number of frames, and the features of the
corresponding index number of the series vector are replaced. For example, if
the onomatopoeia ”kuru-kuru” is annotated between 2.0 and 4.0 seconds, the
index numbers from 120 to 240 of the series vector are replaced with a
43-dimensional numeric vector representing the ”kuru-kuru” impression.

The sample of the onomatopoeia series vector is the following array where
$d=43$ is the 43-dimensional feature of the adjective pair calculated by
Sakamoto System, $t$ is the number of frames corresponding to the motion video
The number of frames is the number of frames corresponding to the motion
video.

####  4.1.2. Sakamoto System(Doizaki et al., 2016): Embedding sound symbolism

To quantify onomatopoeia, we use Sakamoto System for estimating fine
impressions of onomatopoeia. Based on the phonetic symbolism of the Japanese
language, the System quantifies the image of the input onomatopoeia using a
scale of 43 adjective pairs related to mainly visual and tactile perception,
for example light - dark, warm - cold, sharp - mild, and heavy - light.

The system is able to estimate the wealth of information conveyed by not only
existing onomatopoeia but also newly created onomatopoeia, and to distinguish
between a variety of highly similar onomatopoeia. For this reason, in the
subject experiments in this study, participants were asked to freely respond
to the onomatopoeia they imagined from the action.

Since the system used in this study does not quantify the impression of
onomatopoeia in real time, the onomatopoeia collected in the subject
experiment was listed and quantified in advance to create an onomatopoeia
dictionary, and the impression was quantified through the onomatopoeia
dictionary during preprocessing and generation at the time of evaluation. The
onomatopoeic dictionary is then used for preprocessing and generation at the
time of evaluation to create onomatopoeic series vectors.

###  4.2. Learning and evaluation

We used FACT, a model used in AI Choreographer, which generates dance motions
from music. The input layer, which encodes music in 35 dimensions, was changed
to encode onomatopoeia in 43 dimensions, and the collected data was trained
with FACT.

The input of the model contains a seed motion sequence with 120 frames (2
seconds) and a music sequence with 240 frames (4 seconds), where the two
sequences are aligned on the first frame. The output of the model is the
future motion sequence with N = 20 frames supervised by L2 loss. During
inference FACT continually generate future motions in a autoregressive manner
at 60 FPS, where only the first predicted motion is kept in every step.

For the learning and evaluation phase, we employed the cross-modal learning of
sound symbols (onomatopoeia) and dance motions using FACT(Li et al., 2021). In
our learning model, a dance motion is generated by sequentially producing 20
frames of dance with 120 frames of dance motion and 240 frames of onomatopoeic
sequences. The sound symbolic word features and motion features are initially
embedded in an 800-dimensional hidden representation within a linear layer. A
learnable position encoding is then added before being input into the
transformer layer. This method allows for the generation of dance motions that
correspond to the input onomatopoeic sequences, providing a unique and
expressive way to create choreography. Despite potential limitations due to
the size of the dataset or the AI Choreographer framework’s age, our
implementation demonstrates the viability of using onomatopoeia as a medium
for generating dance motions.

##  5\. Results and Evaluations

Table 2. Comparison of quantitative scores for each training model. The larger the data set, the better the score when compared in terms of the size of the data set, and the music score was better when compared in terms of whether the feature was music or onomatopoeia. Learned model | FID_k | FID_g | Dist_k | Dist_g
---|---|---|---|---
AI Choreographer | 33.7593 | 15.4568 | 7.8050 | 6.4799
AI Choreographer Mini | 45.3925 | 26.4813 | 6.0897 | 5.5959
our model | 81.8418 | 39.5038 | 2.0414 | 2.6655

###  5.1. Quantitative evaluation

FID scores, which quantify the difference from the distribution of grand-truth
motions, and motion diversity scores, which calculate the average Euclidean
distance in the feature space across 40 generated motions on the AIST++ test
set to measure the diversity were calculated for the three models: AI
Choreographer, a previous study that generates dances from music; AI
Choreographer Mini, which has the same training data as this model; and this
model. Both the FID score and the diversity score were calculated for AI
Choreographer and this model scored worse than AI Choreographer Mini. In other
words, there is room for improvement in both the amount of data set and the
way feature vectors are created.

Figure 5. The dances generated with the onomatopoeia included in the
validation data, and the waveform of the sound when that onomatopoeia
occurred. The section where the onomatopoeia is described indicates the moment
when the input embedding was given, and the dashed line represents the section
where a zero vector was given. The frames where the dance and the characters
of the onomatopoeia are written are time-aligned, but the waveform only shows
the waveform of a single occurrence of the onomatopoeia. The meaning of the
sound symbolism of each onomatopoeia in ChatGPT is: a. The sound of some
liquid boiling b. The sound of a cat meowing c. The sound of something hitting
or being hit by a hard object.

###  5.2. Validity of the Generated Dance

To verify the validity of the generated dance, we compare the input
onomatopoeia’s audio waveform with the generated dance. The audio waveform
represents the rhythm of the onomatopoeia, and since the rhythm is closely
related to the onomatopoeia meaning, we can investigate the semantic
plausibility by looking at how the onomatopoeia meaning and waveform are
related to the dance movement (see Figure 5). The first row (Figure 5a)
represents the swaying motion in Japanese onomatopoeia, and from the waveform,
we can see that there are four phonemes and a rhythm. Looking at the generated
dance, we can see that the hands are swaying up and down with a slow, steady
rhythm. The second row (Figure 5b) represents the sound of a cat stretching
and meowing in Japanese onomatopoeia. From the waveform, we can see that the
sound is being stretched. In the generated dance, the hands are stretched out
and bent horizontally, reflecting the stretching image. The third row (Figure
5c) represents a Japanese onomatopoeia depicting the sound of heavy machinery
with multiple parts hitting the floor or objects. From the waveform, we can
see the bouncing sound. Looking at the generated motion, we can see the sharp
lifting, lowering, and spreading of the hands, reflecting the bouncing sound.
In this way, we could confirm that our method can generate reasonably
plausible movements to a certain extent.

###  5.3. Results in Various Languages

To generate a dance intuitively even without knowledge of onomatopoeia, it is
required that plausible dances can be generated even when onomatopoeia that
comes to mind are used as is for input. The Sakamoto system we employed
focuses on the phonemes and phonology of the input word, so there are
virtually no restrictions on the onomatopoeia that can be used as input, and
even created onomatopoeia can be used as input. To demonstrate this
experimentally, we additionally generated a variety of onomatopoeia and
attempted to generate dances.

####  5.3.1. Onomatopoeia Generation in Various Languages

We used the large language model, ChatGPT (GPT-4) (OpenAI, 2023), to generate
onomatopoeia. ChatGPT can generate plausible text responses when given text-
based instructions. We instructed it for the 26 languages it can handle to
”Please generate onomatopoeia in XXX (language). However, for languages
without onomatopoeia, please generate words that sound like onomatopoeia.” We
then had it generate four random onomatopoeia for each language.

####  5.3.2. Results

Examples of the generated dances that exhibited good movements are shown in
Figure 6. For other examples, please refer to the supplementary materials. We
confirmed that our model can generate plausible motions as dances when given
unknown onomatopoeia as input. Additionally, we found that the model can
generate similar motions repeatedly for repeated onomatopoeia inputs,
indicating a consistency in the generated motions corresponding to the
onomatopoeia. However, further verification is needed to determine if the
appearance of the motions truly matches human impression.

##  6\. Discussion

###  6.1. Limitations and Future Work

There are several limitations to this study that warrant further
investigation. First, more updated model than AI Choreographer is available
now which could generate more optimal dance motions. Future work should
consider implementing more recent state-of-the-art methods to improve the
quality of the generated motions. Additionally, the dataset used in this study
is relatively small, which may have limited the performance of our model. In
future studies, increasing the size of the dataset and fine-tuning the model
can be considered to enhance the quality of the generated motion.

The proposed model has substantial potential applications in various domains.
In the entertainment industry, this model can be used to create unique and
engaging dance performances based on onomatopoeic input, adding a new
dimension to the creative process. Furthermore, the model can be integrated
into interactive applications, such as virtual reality or video games, to
generate dynamic and immersive dance experiences for users.

In the field of education, this model can serve as a valuable tool for
teaching dance and movement. By providing onomatopoeic input, students can
better understand and visualize the relationship between rhythm, meaning, and
movement in the context of dance. Moreover, this model can also be used to
study the impact of different onomatopoeic words on dance motion generation
and explore the underlying principles that govern this relationship.

###  6.2. Uniqueness of onomatopoeia Approach

We would like to discuss the following points: 1. The difference between
generating onomatopoeia and random generation, 2. How the generation method
differs from time-series data based on phonetic symbols, 3. The significance
of using onomatopoeia as an intermediate representation between meaning,
words, and sound. Firstly, the generation of onomatopoeia is different from
random generation in that it carries both meaning and a sense of rhythm.
Unlike random generation, onomatopoeia can convey specific information while
still maintaining an abstract representation that allows for creative
interpretation. Secondly, the proposed method differs from time-series data
generation based on phonetic symbols, as it does not solely rely on the
intensity of sound. Instead, the model generates dance motions that consider
both the rhythm and meaning of the input onomatopoeia. This makes the method
more sophisticated than simply following the intensity of sound in a time-
series manner. Moreover, onomatopoeia serves as an intermediate representation
between meaning, words, and sound. It is not strictly tied to the time-series
data of sound intensity, allowing for more flexibility and expressiveness in
the generated dance motions. This unique characteristic of onomatopoeia makes
it a suitable input for generating dance movements that convey both rhythm and
meaning. Although the dataset presented in this study is still relatively
primitive, we believe that it holds significant potential as an intermediate
representation between meaning and physical information. By further refining
the dataset and improving the model, it is possible to create more expressive
and meaningful dance motions that fully utilize the unique properties of
onomatopoeia. Moreover the proposed method of using onomatopoeia for
generating dance motions offers a novel and promising approach to computer
graphics and animation research. By considering both rhythm and meaning, the
model can generate more expressive and engaging dance performances that are
not limited by the constraints of traditional time-series data or phonetic
symbols. As the dataset and model continue to be developed, this research is
expected to contribute to the advancement of the field and inspire new
applications in various domains.

Another area for future work is the development of an interactive application
that generates dance movements based on onomatopoeic input. Such an
application would allow users to create personalized dance experiences and
further explore the potential of onomatopoeia in the context of dance motion
generation.

Qualitative evaluation will be conducted in the user study. It is conceivable
that the onomatopoeic instructions to be annotated could be squeezed or
annotated with voice.

##  7\. conclusion

In conclusion, this study has presented a novel approach to dance motion
generation by leveraging the unique properties of onomatopoeia. The proposed
machine learning model expands upon the existing AI Choreographer framework,
introducing onomatopoeic input as a means to convey both rhythm and meaning in
dance movement generation. The application of onomatopoeia as an input source
is an innovative and linguistically interesting method, addressing the
shortcomings of traditional dance generation inputs such as music, speech, and
motion capture.

Despite its limitations, this study has demonstrated the potential of
onomatopoeia as a medium for dance motion generation and has introduced a new
dataset for further exploration. Future work could involve fine-tuning the
model, increasing the dataset size, and developing an interactive application
that generates dance movements in response to onomatopoeic input. This would
enable users to easily create unique and expressive dance sequences based on
their linguistic creativity.

###### Acknowledgements.

This work was supported by READYFOR.

## References

  * (1)
  * Ahn et al. (2020) Hyemin Ahn, Jaehun Kim, Kihyun Kim, and Songhwai Oh. 2020\.  Generative autoregressive networks for 3d dancing move synthesis from music.  _IEEE Robotics and Automation Letters_ 5, 2 (2020), 3501–3508.
  * Aksan et al. (2020) Emre Aksan, Peng Cao, Manuel Kaufmann, and Otmar Hilliges. 2020\.  Attention, please: A spatio-temporal transformer for 3d human motion prediction.  _arXiv preprint arXiv:2004.08692_ 2, 3 (2020), 5\.
  * Aksan et al. (2019) Emre Aksan, Manuel Kaufmann, and Otmar Hilliges. 2019\.  Structured prediction helps 3d human motion modelling. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7144–7153.
  * Alemi et al. (2017) Omid Alemi, Jules Françoise, and Philippe Pasquier. 2017\.  GrooveNet: Real-time music-driven dance movement generation using artificial neural networks.  _networks_ 8, 17 (2017), 26.
  * Bhattacharya et al. (2021) Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. 2021.  Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents. In _2021 IEEE virtual reality and 3D user interfaces (VR)_. IEEE, 1–10.
  * Bowden (2000) Richard Bowden. 2000\.  Learning statistical models of human motion. In _IEEE Workshop on Human Modeling, Analysis and Synthesis, CVPR_ , Vol. 2000.
  * Brand and Hertzmann (2000) Matthew Brand and Aaron Hertzmann. 2000.  Style machines. In _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_. 183–192.
  * Butepage et al. (2017) Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom. 2017.  Deep representation learning for human motion prediction and classification. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6158–6166.
  * Chiu et al. (2019) Hsu-kuang Chiu, Ehsan Adeli, Borui Wang, De-An Huang, and Juan Carlos Niebles. 2019.  Action-agnostic human pose forecasting. In _2019 IEEE winter conference on applications of computer vision (WACV)_. IEEE, 1423–1432.
  * Crnkovic-Friis and Crnkovic-Friis (2016) Luka Crnkovic-Friis and Louise Crnkovic-Friis. 2016.  Generative choreography using deep learning.  _arXiv preprint arXiv:1605.06921_ (2016).
  * Doizaki et al. (2015) Ryuichi Doizaki, Saki Iiba, Takayuki Okatani, and Maki Sakamoto. 2015.  Possibility to use product image and review text based on the association between onomatopoeia and texture.  _Transactions of the Japanese Society for Artificial Intelligence_ 30, 1 (2015), 124–137.
  * DOIZAKI et al. (2016) Ryuichi DOIZAKI, Takahide MATSUDA, Akira UTSUMI, and Maki SAKAMOTO. 2016.  Constructing a System which Proposes Metaphors Corresponding to the Onomatopoeia Expressing Medical Conditions.  _International Journal of Affective Engineering_ 15, 2 (2016), 37–43.
  * Doizaki et al. (2013) Ryuichi Doizaki, Ai Oikawa, Yuichiro Shimizu, and Maki Sakamoto. 2013.  Intuitive Color Design Support System Uusing Onomatopoeia. In _Proceedings of the 5th International Congress of International Association of Societies of Design Research (IASDR 2013)_. 757–766.
  * Doizaki et al. (2016) Ryuichi Doizaki, Junji Watanabe, and Maki Sakamoto. 2016\.  Automatic estimation of multidimensional ratings from a single sound-symbolic word and word-based visualization of tactile perceptual space.  _IEEE transactions on haptics_ 10, 2 (2016), 173–182.
  * Du et al. (2019) Xiaoxiao Du, Ram Vasudevan, and Matthew Johnson-Roberson. 2019\.  Bio-lstm: A biomechanically inspired recurrent neural network for 3-d pedestrian pose and gait prediction.  _IEEE Robotics and Automation Letters_ 4, 2 (2019), 1501–1508.
  * Duan et al. (2022) Yinglin Duan, Yue Lin, Zhengxia Zou, Yi Yuan, Zhehui Qian, and Bohan Zhang. 2022\.  A Unified Framework for Real Time Motion Completion.  (2022).
  * Fan et al. (2011) Rukun Fan, Songhua Xu, and Weidong Geng. 2011.  Example-based automatic music-driven conventional dance motion synthesis.  _IEEE transactions on visualization and computer graphics_ 18, 3 (2011), 501–515.
  * Fragkiadaki et al. (2015) Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015.  Recurrent network models for human dynamics. In _Proceedings of the IEEE international conference on computer vision_. 4346–4354.
  * Galata et al. (2001) Aphrodite Galata, Neil Johnson, and David Hogg. 2001\.  Learning variable-length Markov models of behavior.  _Computer Vision and Image Understanding_ 81, 3 (2001), 398–413.
  * Ghosh et al. (2017) Partha Ghosh, Jie Song, Emre Aksan, and Otmar Hilliges. 2017\.  Learning human motion models for long-term predictions. In _2017 International Conference on 3D Vision (3DV)_. IEEE, 458–466.
  * Ginosar et al. (2019) Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. 2019\.  Learning individual styles of conversational gesture. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3497–3506.
  * Guo et al. (2020) Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. 2020\.  Action2motion: Conditioned generation of 3d human motions. In _Proceedings of the 28th ACM International Conference on Multimedia_. 2021–2029.
  * Hernandez et al. (2019) Alejandro Hernandez, Jurgen Gall, and Francesc Moreno-Noguer. 2019.  Human motion prediction via spatio-temporal inpainting. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7134–7143.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ 33 (2020), 6840–6851.
  * Holden et al. (2020) Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. 2020.  Learned motion matching.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 53–1.
  * Holden et al. (2016) Daniel Holden, Jun Saito, and Taku Komura. 2016.  A deep learning framework for character motion synthesis and editing.  _ACM Transactions on Graphics (TOG)_ 35, 4 (2016), 1–11.
  * Holden et al. (2015) Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015\.  Learning motion manifolds with convolutional autoencoders.  In _SIGGRAPH Asia 2015 technical briefs_. 1–4.
  * Huang et al. (2020) Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang. 2020\.  Dance revolution: Long-term dance generation with music via curriculum learning.  _arXiv preprint arXiv:2006.06119_ (2020).
  * Huang et al. (2022) Yuhang Huang, Junjie Zhang, Shuyan Liu, Qian Bao, Dan Zeng, Zhineng Chen, and Wu Liu. 2022.  Genre-Conditioned Long-Term 3D Dance Generation Driven by Music. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 4858–4862.
  * Jain et al. (2016) Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. 2016.  Structural-rnn: Deep learning on spatio-temporal graphs. In _Proceedings of the ieee conference on computer vision and pattern recognition_. 5308–5317.
  * Kao and Su (2020) Hsuan-Kai Kao and Li Su. 2020.  Temporally guided music-to-body-movement generation. In _Proceedings of the 28th ACM International Conference on Multimedia_. 147–155.
  * Kim et al. (2022) Jinwoo Kim, Heeseok Oh, Seongjean Kim, Hoseok Tong, and Sanghoon Lee. 2022.  A Brand New Dance Partner: Music-Conditioned Pluralistic Dancing Controlled by Multiple Dance Genres. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3490–3500.
  * Lee et al. (2019) Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. 2019.  Dancing to music.  _Advances in neural information processing systems_ 32 (2019).
  * Lee et al. (2013) Minho Lee, Kyogu Lee, and Jaeheung Park. 2013.  Music similarity-based approach to generating dance motion sequence.  _Multimedia tools and applications_ 62, 3 (2013), 895–912.
  * Li et al. (2022) Buyu Li, Yongchi Zhao, Shi Zhelun, and Lu Sheng. 2022\.  Danceformer: Music conditioned 3d dance generation with parametric motion transformer. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 1272–1279.
  * Li et al. (2021) Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. 2021\.  Ai choreographer: Music conditioned 3d dance generation with aist++. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 13401–13412.
  * Ling et al. (2020) Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. 2020.  Character controllers using motion vaes.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 40–1.
  * Nozaki et al. (2021) Yuji Nozaki, Shu Watanabe, and Maki Sakamoto. 2021\.  Sound Symbolic Words as a Game Controller. In _Human-Computer Interaction. Interaction Techniques and Novel Applications: Thematic Area, HCI 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings, Part II 23_. Springer, 56–64.
  * Ofli et al. (2011) Ferda Ofli, Engin Erzin, Yücel Yemez, and A Murat Tekalp. 2011\.  Learn2dance: Learning statistical music-to-dance mappings for choreography synthesis.  _IEEE Transactions on Multimedia_ 14, 3 (2011), 747–759.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Petrovich et al. (2021) Mathis Petrovich, Michael J Black, and Gül Varol. 2021\.  Action-conditioned 3d human motion synthesis with transformer vae. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10985–10995.
  * Petrovich et al. (2022) Mathis Petrovich, Michael J Black, and Gül Varol. 2022\.  TEMOS: Generating diverse human motions from textual descriptions.  _arXiv preprint arXiv:2204.14109_ (2022).
  * Pullen and Bregler (2000) Katherine Pullen and Christoph Bregler. 2000.  Animating by multi-level sampling. In _Proceedings Computer Animation 2000_. IEEE, 36–42.
  * Rempe et al. (2021) Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas. 2021.  Humor: 3d human motion model for robust pose estimation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 11488–11499.
  * Sakamoto and Watanabe (2017) Maki Sakamoto and Junji Watanabe. 2017.  Exploring tactile perceptual dimensions using materials associated with sensory vocabulary.  _Frontiers in psychology_ 8 (2017), 569.
  * Sakamoto et al. (2021) Maki Sakamoto, Junji Watanabe, and Koichi Yamagata. 2021\.  Automatic Estimation of Multidimensional Personality From a Single Sound-Symbolic Word.  _Frontiers in Psychology_ 12 (2021), 595986.
  * Siyao et al. (2022) Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. 2022\.  Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11050–11059.
  * Sun et al. (2020) Guofei Sun, Yongkang Wong, Zhiyong Cheng, Mohan S Kankanhalli, Weidong Geng, and Xiangdong Li. 2020.  DeepDance: music-to-dance motion choreography with adversarial learning.  _IEEE Transactions on Multimedia_ 23 (2020), 497–509.
  * Tang et al. (2018) Taoran Tang, Jia Jia, and Hanyang Mao. 2018.  Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis. In _Proceedings of the 26th ACM international conference on Multimedia_. 1598–1606.
  * Taylor and Hinton (2009) Graham W Taylor and Geoffrey E Hinton. 2009.  Factored conditional restricted Boltzmann machines for modeling motion style. In _Proceedings of the 26th annual international conference on machine learning_. 1025–1032.
  * Tendulkar et al. (2020) Purva Tendulkar, Abhishek Das, Aniruddha Kembhavi, and Devi Parikh. 2020.  Feel the music: Automatically generating a dance for an input song.  _arXiv preprint arXiv:2006.11905_ (2020).
  * Tevet et al. (2022) Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. 2022\.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ (2022).
  * Tseng et al. (2022) Jonathan Tseng, Rodrigo Castellon, and C Karen Liu. 2022\.  EDGE: Editable Dance Generation From Music.  _arXiv preprint arXiv:2211.10658_ (2022).
  * Tsuchida et al. (2019) Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. 2019.  AIST Dance Video Database: Multi-genre, Multi-dancer, and Multi-camera Database for Dance Information Processing. In _Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019_. Delft, Netherlands, 501–510.
  * Valle-Pérez et al. (2021) Guillermo Valle-Pérez, Gustav Eje Henter, Jonas Beskow, Andre Holzapfel, Pierre-Yves Oudeyer, and Simon Alexanderson. 2021.  Transflower: probabilistic autoregressive dance generation with multimodal attention.  _ACM Transactions on Graphics (TOG)_ 40, 6 (2021), 1–14.
  * Villegas et al. (2018) Ruben Villegas, Jimei Yang, Duygu Ceylan, and Honglak Lee. 2018\.  Neural kinematic networks for unsupervised motion retargetting. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 8639–8648.
  * Wang et al. (2019) Borui Wang, Ehsan Adeli, Hsu-kuang Chiu, De-An Huang, and Juan Carlos Niebles. 2019.  Imitation learning for human pose prediction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7124–7133.
  * Won et al. (2022) Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2022\.  Physics-based character controllers using conditional VAEs.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–12.
  * Yalta et al. (2019) Nelson Yalta, Shinji Watanabe, Kazuhiro Nakadai, and Tetsuya Ogata. 2019.  Weakly-supervised deep recurrent neural networks for basic dance step generation. In _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–8.
  * YAMAGATA et al. (2019) Koichi YAMAGATA, Tatsuki KAGITANI, and Maki SAKAMOTO. 2019\.  Image Retrieval System Using Japanese Sound Symbolic Words for Surface Texture. In _International Symposium on Affective Science and Engineering ISASE2019_. Japan Society of Kansei Engineering, 1–4.
  * Yamagata et al. (2021) Koichi Yamagata, Jinhwan Kwon, Takuya Kawashima, Wataru Shimoda, and Maki Sakamoto. 2021.  Computer Vision System for Expressing Texture Using Sound-Symbolic Words.  _Frontiers in Psychology_ (2021), 4624.
  * Ye et al. (2020) Zijie Ye, Haozhe Wu, Jia Jia, Yaohua Bu, Wei Chen, Fanbo Meng, and Yanfeng Wang. 2020\.  Choreonet: Towards music to dance synthesis with choreographic action unit. In _Proceedings of the 28th ACM International Conference on Multimedia_. 744–752.
  * Yin et al. (2022) Wenjie Yin, Hang Yin, Kim Baraka, Danica Kragic, and Mårten Björkman. 2022.  Dance Style Transfer with Cross-modal Transformer.  _arXiv preprint arXiv:2208.09406_ (2022).
  * Zhang et al. (2022) Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2022.  Motiondiffuse: Text-driven human motion generation with diffusion model.  _arXiv preprint arXiv:2208.15001_ (2022).
  * Zhuang et al. (2020) Wenlin Zhuang, Yangang Wang, Joseph Robinson, Congyi Wang, Ming Shao, Yun Fu, and Siyu Xia. 2020.  Towards 3d dance motion synthesis and control.  _arXiv preprint arXiv:2006.05743_ (2020).

Figure 6. Generated motion results. The original language and the meaning of
the sound symbolism of each onomatopoeia in ChatGPT is: a. German, Pukupukku
and bubbling sound b. Russian, buzzing sound of insects c. Nepali, sound of
shining light d. Welsh, sound of wind blowing