#  RoboPianist: A Benchmark for
High-Dimensional Robot Control

Kevin Zakka1,2 Laura Smith1 Nimrod Gileadi3 Taylor Howell4 Xue Bin Peng5
Sumeet Singh2 Yuval Tassa3 Pete Florence2 Andy Zeng2 Pieter Abbeel1
https://kzakka.com/robopianist/ 1UC Berkeley 2Robotics at Google 3DeepMind
4Stanford University 5Simon Fraser University

###### Abstract

We introduce a new benchmarking suite for high-dimensional control, targeted
at testing high spatial and temporal precision, coordination, and planning,
all with an underactuated system frequently making-and-breaking contacts. The
proposed challenge is mastering the piano through bi-manual dexterity, using a
pair of simulated anthropomorphic robot hands. We call it RoboPianist, and the
initial version covers a broad set of 150 variable-difficulty songs. We
investigate both model-free and model-based methods on the benchmark,
characterizing their performance envelopes. We observe that while certain
existing methods, when well-tuned, can achieve impressive levels of
performance in certain aspects, there is significant room for improvement.
RoboPianist provides a rich quantitative benchmarking environment, with human-
interpretable results, high ease of expansion by simply augmenting the
repertoire with new songs, and opportunities for further research, including
in multi-task learning, zero-shot generalization, multimodal (sound, vision,
touch) learning, and imitation. Supplementary information, including videos of
our control policies, can be found at https://kzakka.com/robopianist/.

##  I Introduction

As the fields of control and reinforcement learning continue to develop, a key
question is how will progress be measured. In fields like computer vision and
natural language processing, progress has been fueled by robust, quantifiable,
and interpretable benchmarks – which in aggregate achieve breadth [1], and in
each explore complementary focuses in depth: take for example the famous
challenge of “Winograd schema” [2], first proposed in 1972 and later developed
in 2012 [3] into an influential benchmark. While in control and reinforcement
learning, certain benchmarking efforts have begun both to aggregate [4] and
explore different aspects of depth, a particularly under-served area has been
robust benchmarks which focus on high-dimensional control, including in
particular the perhaps ultimate “challenge problem” of high-dimensional
robotics: mastering bi-manual (two-handed) multi-fingered control.

Figure 2: A full example set of F1 scores on all tasks in RoboPianist-
repertoire-150, highlighting the breadth of difficulty offered by the
benchmark. This example set of scores is from the model-free method discussed
in Section IV (errors bars represent standard deviations over 3 seeds). Both
model-free and model-based implementations as well as evaluations are
discussed in Subsection V-B.

In fact, despite decades-long research into replicating the dexterity of the
human hand, high-dimensional control remains a grand challenge in robotics.
This topic has inspired considerable research from both a mechanical design
[5, 6, 7] and control theoretic point of view [8, 9, 10, 11, 12]. While
learning-based approaches [13, 14, 15, 16, 17, 18, 19] have dominated the
recent literature, the class of problems typically considered corresponds to a
limited definition of dexterity. In particular, most all such tasks are well-
specified using a single goal-state or termination condition, limiting the
complexity of the solution space and oftentimes yielding unnatural-looking
behaviors so long as the desired terminal state is reached.

In this work, we introduce a new benchmark suite for high-dimensional control,
RoboPianist, where bi-manual simulated anthropomorphic robot hands are tasked
with playing a variety of songs (i.e., correctly pressing sequences of keys on
a keyboard) conditioned on sheet music, in the form of a Musical Instrument
Digital Interface (MIDI) transcription. The robot hands themselves exhibit
high degrees of freedom (22 actuators per hand, for a total of 44), and are
partially underactuated, akin to human hands. We specifically chose this
domain because playing a song successfully means being able to sequence
actions in ways that exemplify many of the properties that we look for in
high-dimensional control policies, including (i) spatial and temporal
precision (hitting the right notes, at the right time), (ii) coordination
(simultaneously achieving multiple different goals, in this case, fingers on
each hand hitting different notes, without colliding), and (iii) planning (how
a key is pressed should be conditioned on the expectation of how it would
enable the policy to reach future notes). Additionally, piano playing also
presents an immediately interpretable task success signal (i.e., “does it
sound good?”), and this can e.g., be disentangled from the particular
reward/cost functions used.

The initial RoboPianist-repertoire-150 benchmark covers 150 songs, where each
song is effectively a different task. Through extensive experiments, we
investigate the performance envelope of both model-free and model-based
methods and demonstrate that, while there is ample room for improvement, our
policies are able to produce compelling performances. We encourage the reader
to watch and listen to the performances at https://kzakka.com/robopianist/.
For expanding further on the benchmarking suite, since song data is also
freely available on the Internet (either by parsing MIDI files, or YouTube
data), one can also effectively scale up the number of tasks presented in this
domain over time. Figure 2 also shows that we can sort songs (i.e., tasks) by
difficulty, reflected in how well a policy can learn the song. Being able to
sort tasks via such metrics can foster additional research across a multitude
of topics in robot learning, including curriculum learning and transfer
learning. Overall, RoboPianist exhibits a straightforward task, easy to
simulate environment, clear evaluation metrics, and is amenable to a variety
of expansion opportunities in the future. Code for simulation, control and
learning, baselines and datasets will be made available at
https://github.com/google-research/robopianist/.

To summarize, we believe RoboPianist provides many distinct and complementary
aspects relative to previous benchmarks for high-dimensional control and is a
strong addition to the community for the following reasons:

  * •

Challenging. As we show in Subsection V-B, RoboPianist is challenging for
model-based and model-free methods. Without a shaped reward in the form of
fingering information, they perform poorly, and even with extensive reward
shaping, there is plenty room for improvement.

  * •

Interpretable. One can simply look at and listen to a policy to gauge its
performance.

  * •

Multimodal. Playing the piano is sensory-rich: policies can make use of sound,
vision and touch.

  * •

Data rich and extendable. On top of just adding more MIDI files to expand the
number of tasks, one can incorporate other abundant sources of data such as
YouTube videos to create third-person demonstrations.

  * •

Enables knowledge reuse. Similar songs share note and pattern structures which
can be reused to solve new songs zero-shot. More broadly, RoboPianist is a
natural playground for studying multi-task and meta-learning.

  * •

Open source. We have fully open-sourced the benchmark and dataset at
https://github.com/google-research/robopianist.

##  II Related Work

We address related work within two primary areas: dexterous high-dimensional
control, and robotic pianists.

Dexterous Manipulation and High-Dimensional Control The vast majority of the
control literature uses much lower-dimensional systems (i.e., single-arm,
simple end-effectors) than high-dimensional dexterous hands. Specifically,
only a handful of general-purpose policy optimization methods have been shown
to work on high-dimensional hands, even for a single hand [14, 13, 16, 15, 20,
18, 21, 11], and of these, only a subset has demonstrated results in the real
world [14, 13, 16, 15, 20]. Results with bi-manual hands are even rarer, even
in simulation only [19, 22].

As a benchmark, perhaps the most distinguishing aspect of RoboPianist is in
the definition of “task success”. As an example, general manipulation tasks
are commonly framed as the continual application of force/torque on an object
for the purpose of a desired change in state (e.g., SE(3) pose and velocity).
Gradations of dexterity are predominantly centered around the kinematic
redundancy of the arm or the complexity of the end-effector, ranging from
parallel jaw-grippers to anthropomorphic hands [23, 19]. A gamut of methods
have been developed to accomplish such tasks, ranging from various
combinations of model-based and model-free RL, imitation learning,
hierarchical control, etc. [24, 14, 17, 16, 25, 26]. However, the class of
problems generally tackled corresponds to a definition of dexterity pertaining
to traditional manipulation skills [27], such as re-orientation, relocation,
manipulating simply-articulated objects (e.g., door opening, ball throwing and
catching), and using simple tools (e.g., hammer) [4, 28, 19, 15, 29]. The only
other task suite that we know of that presents bi-manual tasks, the recent Bi-
Dex [19] suite, presents a broad collection of tasks that fall under this
category.

While these works represent an important class of problems, we explore an
alternative notion of dexterity and success. In particular, for most all the
aforementioned suite of manipulation tasks, the “goal” state is some explicit,
specific geometric function of the final states; for instance, an open/closed
door, object re-oriented, nail hammered, etc. This effectively reduces the
search space for controls to predominantly a single “basin-of-attraction" in
behavior space per task. In contrast, the RoboPianist suite of tasks
encompasses a more complex notion of a goal, which is encoded through a
musical performance. In effect, this becomes a highly combinatorially variable
sequence of goal states, extendable to arbitrary difficulty by only varying
the musical score. “Success” is graded on accuracy over an entire episode;
concretely, via a time-varying non-analytic output of the environment, i.e.,
the music. Thus, it is not a matter of the “final-state” that needs to satisfy
certain termination/goal conditions, a criterion which is generally permissive
of less robust execution through the rest of the episode, but rather the
behavior of the policy _throughout the episode_ needs to be precise and
musical.

Similarly, the literature on humanoid locomotion and more broadly, “character
control", another important area of high-dimensional control, primarily
features tasks involving the discovery of stable walking/running gaits [30,
31, 32], or the distillation of a finite set of whole-body movement priors
[33, 34, 35], to use downstream for training a task-level policy. Task success
is typically encoded via rewards for motion progress and/or reaching a
terminal goal condition. It is well-documented that the endless pursuit of
optimizing for these rewards can yield unrealistic yet “high-reward"
behaviors. While works such as [33, 36] attempt to capture _stylistic_
objectives via leveraging demonstration data, these reward functions are
simply appended to the primary task objective. This scalarization of multiple
objectives yields an arbitrarily subjective Pareto curve of optimal policies.
In contrast, performing a piece of music entails both objectively measurable
precision with regards to melodic and rhythmic accuracy, as well as a
subjective measure of musicality. Mathematically, this translates as
_stylistic_ constraint satisfaction, paving the way for innovative algorithmic
advances.

Robotic Piano Playing Robotic pianists have a rich history within the
literature, with several works dedicated to the design of specialized hardware
[37, 38, 39, 40, 41, 42], and/or customized controllers for playing back a
song using pre-programmed commands (open-loop) [43, 44]. The work in [45]
leverages a combination of inverse kinematics and trajectory stitching to play
single keys and playback simple patterns and a song with a Shadow hand [46].
More recently, in [47], the author simulated robotic piano playing using
offline motion planning with inverse kinematics for a 7-DoF robotic arm, along
with an Iterative Closest Point-based heuristic for selecting fingering for a
four-fingered Allegro hand. Each hand is simulated separately, and the audio
results are combined post-hoc. Finally, in [48], the authors formulate piano
playing as an RL problem for a single Allegro hand (four fingers) on a
miniature piano, and additionally leverage tactile sensor feedback. However,
the tasks considered are rather simplistic (e.g., play up to six successive
notes, or three successive chords with only two simultaneous keys pressed for
each chord). The RoboPianist benchmark suite is designed to allow a general
bi-manual controllable agent to emulate a pianist’s growing proficiency on the
instrument by providing a curriculum of musical pieces, graded in difficulty.
Leveraging two underactuated anthropomorphic hands as actuators provides a
level of realism and exposes the challenge of mastering this suite of high-
dimensional control problems.

##  III The RoboPianist Benchmark

(a)

(b)

(c)

Figure 3: From left to right, (a): 24 DoF, under-actuated right Shadow Hand
model, (b): 88 DoF digital piano model, and (c): RoboPianist task setup with
left and right Shadow Hands placed above the piano on an invisible gantry.

In this section, we introduce the main elements of the RoboPianist benchmark.
We begin by describing the general setup of our benchmark, walk through its
practical implementation in simulation, and finally, detail our MIDI dataset
which defines the task distribution available in the benchmark.

###  III-A Benchmark setup

States | Unit | Size
---|---|---
Hand joint angles | $\text{\,}\mathrm{rad}$ | 48
Hand joint velocities | $\text{\,}\mathrm{rad}\mathrm{/}\mathrm{s}$ | 48
Forearm positions | $\text{\,}\mathrm{m}$ | 4
Forearm velocities | $\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ | 4
Piano key angles | $\text{\,}\mathrm{rad}$ | 88
Piano key velocities | $\text{\,}\mathrm{rad}\mathrm{/}\mathrm{s}$ | 88
Fingering (goal) | discrete | 10
Piano key press (goal) | discrete | 88
Total |  | 378
Actions | Unit | Size
Desired hand joint angles | $\text{\,}\mathrm{rad}$ | 40
Desired forearm positions | $\text{\,}\mathrm{m}$ | 4
Total |  | 44
Observations | Unit | Size
Hand and forearm joints | $\text{\,}\mathrm{rad}$ | 52
Cartesian forearm position | $\text{\,}\mathrm{m}$ | 6
Piano key angles | $\text{\,}\mathrm{rad}$ | 88
Fingering | discrete | 10
Piano key press | discrete | 88
Previous reward | — | 1
Previous action | $\text{\,}\mathrm{rad}$ | 44
Total |  | 289

TABLE I: The state, action, and observation spaces of the RoboPianist MDP.
Fingering is a discrete boolean vector with size equal to the number of
fingers, with each component indicating if the corresponding finger is
expected to be in contact with a key. This information is extracted from the
fingering annotations in the MIDI file for each piece/task.

Learning to play the piano can be formulated as a finite-horizon Markov
Decision Process (MDP) defined by a tuple
$(\mathcal{S},\mathcal{A},\rho,p,r,\gamma,H)$ where
$\mathcal{S}\subset\mathbb{R}^{n}$ is the state space,
$\mathcal{A}\subset\mathbb{R}^{m}$ is the action space, $\rho(\cdot)$ is the
initial state distribution, $p(\cdot|s,a)$ governs the dynamics,
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ defines the rewards,
$\gamma\in[0,1)$ is the discount factor, and $H$ is the horizon. The goal of
an agent is to maximize its total expected reward over the horizon
$\operatorname*{\mathbb{E}}\left[\sum_{t=0}^{H}\gamma^{t}r(s_{t},a_{t})\right]$.
In the RoboPianist benchmark, each song is a separate task with a different
note trajectory and horizon $H$.

All tasks in RoboPianist share the same underlying state, action, and
observation spaces (Table I)111The MDP is fully observable, but in practice we
use an observation space that is not exactly the state space – for example, to
better align with hardware in that there are no direct velocity sensors. Any
lack of observability on joint velocities can be remedied simply by
incorporating the last two measurements for the agent.. The action space
$\mathcal{A}$ is a 44 dimensional bounded, continuous action corresponding to
desired joint positions for the hands and forearms. The desired positions are
converted to torques at the 52 joints using proportional-position
actuators222The control signal for the distal actuator of each non-thumb
finger ($4\cdot 2=8$) is split amongst the middle and distal joints.. In
particular, we note that both proprioceptive states (i.e., joint and forearm
positions) and exteroceptive states (i.e., key joint positions) are
measurements that are readily available in the real world.

A RoboPianist task is illustrated in Figure 3. At episode initialization, the
hands are placed in the middle of the keyboard and the MIDI file is converted
into a piano roll, i.e., a binary matrix in $\mathbb{Z}^{T\times 88}$, where
$T$ is the number of time steps. The piano roll is a time-indexed trajectory
telling us which of the 88 notes should be active at every time step,
essentially defining a time-varying “goal trajectory." Each task has a
variable length $T$ which is a function of both the MIDI length and the
control frequency. At an implementation level, all tasks in RoboPianist are
exposed as distinct dm_env [49] or gym [50] environments, which makes it
straightforward to plug them directly into existing RL or imitation learning
codebases.

###  III-B Simulation details

Reward | Formula | Weight | Explanation
---|---|---|---
Key Press | $0.5\cdot g(||k_{s}-k_{g}||_{2})+0.5\cdot(1-\mathbf{1}_{\\{\mathsf{false\;positive}\\}})$ | 1 | Press the right keys and only the right keys
Forearm Penalty | $1-\mathbf{1}_{\\{\mathsf{collision}\\}}$ | 0.4 | Minimize forearm collisions
Energy Penalty | $|\mathsf{\tau_{joints}}|^{\top}|\mathsf{v_{joints}}|$ | -5e-3 | Minimize energy expenditure
Finger Close to Key | $g(||p_{f}-p_{k}||_{2})$ | 1 | Shaped reward to bring fingers to key

TABLE II: The reward structure used for the model-free baseline. $\tau$
represents the joint torque, $v$ is the joint velocity, $p_{f}$ and $p_{k}$
represent the position of the finger and key in the world frame respectively,
$k_{s}$ and $k_{g}$ represent the current and the goal states of the key
respectively, and $g$ is a function that transforms the distances to rewards
in the $[0,1]$ range. See the appendix for a detailed description of each
term.

We use the open-source MuJoCo [51] simulator with the dm_control Python
bindings [52]. We chose MuJoCo for a few reasons: 1) faster-than-realtime
rigid body simulation in the presence of contacts, 2) MJCF model definition
combined with ease of task creation with the Composer module, 3) the feature-
rich interactive viewer allowing for visual debugging, playback and
interaction with the physical models using the mouse input, 4) recently, the
addition of a high-quality robot models [53] maintained by the creators of the
simulator.

Piano model. We create a full-size standard (“88-key”) digital piano in
simulation, which consists of 52 white keys and 36 black keys, spanning 12
major scales. We use a Kawai reference manual [54] to closely match the
dimensions, shape, positioning and spacing of the keys on the keyboard. Each
key is modeled using a linear spring. To speed up the simulation, we
explicitly disable collision checking between the white keys since they can’t
possibly ever be in contact. Additionally, rather than creating a custom key
mesh for the white keys (i.e., a box with a crevice in which the black key
rests), we disable collision checking between white and black keys. The result
is practically equivalent but faster to simulate since we exclusively deal
with primitive box geometries.

Hand model. We use the left and right Shadow Dexterous Hand [46] models from
MuJoCo Menagerie [53]. This anthropomorphic hand has been designed to closely
reproduce the kinematics and dexterity of the human hand. The Shadow Hand is
underactuated: it has 24 degrees of freedom, but only 20 actuators. This is
because the non-thumb finger distal joints are coupled. We add two degrees of
freedom to the base of the hand forearms to simulate the equivalent of a
planar gantry: a prismatic joint to translate laterally along the piano
length, and a prismatic joint to translate longitudinally along the piano
depth. Note that the Shadow Hand itself provides two rotational degrees of
freedom in each wrist (similar to human wrist joint). Since [19] used two
copies of a right hand, RoboPianist is the first benchmarking suite we know of
that uses both left and right hands, and also does so in a way in which the
handedness is relevant (piano pieces are designed for a pair of left and right
human hands).

Sound representation. We use the Musical Instrument Digital Interface (MIDI)
standard to represent piano pieces and synthesize sounds. Very briefly, a MIDI
file stores note_on and note_off messages. Each such message stores a note
number, a note velocity and a timestamp. The MIDI number is an integer between
0 and 127 and encodes a note’s pitch. The velocity is also an integer from 0
to 127 and controls the intensity of the sound. The timestamp specifies when
to execute the message. The moment a key is pressed on a piano, it generates a
note_on event, and the moment it is released, it generates a note_off event.
The intensity of the generated sound is controlled by the velocity of the
keystroke. In our simulation, a key is considered active when its joint
position exceeds the halfway point of its range. When this occurs, we emit the
note_on message. We currently do not model the intensity of each keynote,
captured by the velocity dimension in the MIDI representation.

###  III-C MIDI dataset

A contribution of RoboPianist is providing a rich initial song corpora with
which we can evaluate high-dimensional control policies. To do this, we
identify a dataset developed for research into estimating human fingering for
piano, called the PIG dataset [55], and transform it into a corpus of MIDI
files that can be played in our simulated environment. This dataset contains
piano pieces from twenty-four Western composers spanning the baroque,
classical and romantic periods. The pieces vary in difficulty, ranging from
relatively easy (e.g., Mozart’s Piano Sonata K 545 in C major) to
significantly harder (e.g., Scriabin’s Piano Sonata No. 5). We refer the
reader to [55] for a detailed list of the pieces.

The fingering labels are encoded as a 10-dimensional boolean vector, with each
component indicating if the corresponding finger is expected to be in contact
with a key at that timestep. Note that fingering information is generally
quite sparse for most pieces of music, and is usually reserved for
particularly tricky sections of the piece. Pianists are expected to rely on
their skill and experience with the instrument to discover the most efficient
fingering. Further, most fingering markings on sheet music are a general
guide, and it is up to the pianist to gracefully incorporate this assistive
information and ensure a smooth musical flow. All these factors provide
enticing opportunities for the design of a controller’s combinatorial
reasoning. In our presented solution methodologies, we incorporate the boolean
fingering information within the reward formulation (Table II). For future
research enabled by the benchmark, having a policy discover the fingering
would be an impressive additional challenge.

###  III-D Evaluation criteria

Agents in RoboPianist are evaluated based on precision, recall and F1 scores.
These metrics are computed by comparing the state of the piano keys at every
time step with the corresponding ground-truth state stored in the MIDI file,
averaged across all time steps. Intuitively, precision measures how good a
policy is at not hitting the wrong keys (i.e., low false positive rate) and
recall measures how good it is at hitting the right keys (i.e., low false
negative). Thus, a policy that hovers above the keyboard without hitting any
keys has high precision but low recall, and conversely, a policy that hits all
the keys simultaneously (assuming it were physically possible) has high recall
but low precision.

##  IV High-Dimensional Robot Control

High-dimensional robot control requires amongst other things, precision in
space and time, coordination, planning, and contact. The RoboPianist task
suite and dataset stresses precisely these properties in a piano playing
environment (Section III). In this section, we describe initial steps towards
tackling the benchmark by introducing two popular policy optimization methods,
as well as various design considerations we incorporated to improve their
performance. These may serve as both (i) expert data generators (e.g., for
imitation learning), or (ii) baselines for future work on the benchmark.

###  IV-A Baseline methods

There are a variety of method-agnostic criteria and solution regimes that one
may be interested in applying to the benchmark. For one, does the policy
method itself have access to the true dynamics of the environment (model-
based), or must it only learn through trial-and-error interactions with the
environment (model-free)? If it needs environment interactions, how many are
allowed? Additionally, although the exact cutoff limit of what may be
considered “real-time” is dependent on specific compute hardware, the ability
of a method to be approximately-fast-enough on common compute hardware in
order to synthesize behaviors in real-time, whether through a “direct”
feedback policy or through online model predictive control (MPC), is another
primary consideration that impacts the eventual deployable regimes on real-
world hardware.

As baseline methods, we present both model-free and model-based variants. By
their nature, the comparison is somewhat inherently “apples vs. oranges”. For
one, the model-based method uses the ground truth dynamics which are not used
by the model-free variant. Meanwhile, the model-free variant requires a large
number (order of 1 million) environment interactions, whereas the model-based
variant requires none. Additionally, the way in which computational resources
are consumed differs greatly. As is common for model-free RL, considerable
compute is used at training time, but the learned policy is encoded into a
learned neural network which runs quickly at inference time (we present
results with sub-millisecond feed-forward time on an M1 Max CPU). Meanwhile
for model-based MPC with a ground-truth model, it is common for no “training”
compute to be required, but instead inference may be computationally
intensive, and better results can be obtained simply by expending more
inference-time compute. Due to this “apples vs. oranges” nature, our goal is
not to proclaim which is “better”, but rather to characterize the performance
envelopes of both methods. We choose both to be in the regime of near-real-
time policy inference, where we allow the model-based method to slow down
simulation time as much as $10\%$ real-time. Model-based methods with
additional offline pre-computation are left for future work.

Model-free. The first baseline approach we consider is in the category of
model-free reinforcement learning. We present results with the off-policy
algorithm DroQ [56], one of several regularized variants of the widely-used
Soft-Actor-Critic [57] algorithm, as it is state-of-the-art in terms of
performance and sample efficiency. We train a separate policy per-song (multi-
task is left for future work) for one million environment interactions using
the reward structure in Table II. We found that even with $10\times$ the
number of samples, PPO [58] with default Stable Baselines3 [59]
hyperparameters was not able to achieve reasonable performance.

Figure 4: RoboPianist-prelude: Peformance of model-free RL with different MDP
design considerations. From top to bottom, in the order specified by the
legend, each curve inherits all MDP attributes of the curve before it, but
makes one additional modification as specified by its label. The PPO run,
which also inherits all the MDP tweaks, starts making progress at 10M
environment interactions, see the appendix for more information.

Model-based. The second baseline we present uses MPC, and specifically uses
the implementation from [60] which was shown to solve the previously-
considered-challenging [13, 17, 15] dexterous task of one-handed cube re-
orientation in simulation. Specifically amongst the various implementation
options presented in [60], we found the most success with the derivative-free
sampling-based method (“Predictive Sampling”). The cost formulation for the
MPC baseline is detailed in the appendix.

Note that while the method with which we received the best success was the
derivative-free “Predictive Sampling”, we also tried the optimized derivative-
based implementation of iLQG [61] also provided by [60], but this was not able
to make substantial progress even at significantly slower than real-time
speeds. We expect that it should be possible to acquire strong results with
derivative-based methods especially with sufficient reward shaping, both in
the near-real-time regime and also with offline model-based trajectory
optimization. We note, however, that the (i) high dimensionality, (ii) complex
sequence of goals adding many constraints, and (iii) overall temporal length
(tens of seconds) of the trajectories pose challenges for some methods that
are typically applied to significantly smaller problems.

###  IV-B System design

We found that simple modifications to the MDP had a large impact on policy
performance. We detail these design considerations and analyze their effect in
the following sections.

  * •

Fingering reward: We use the fingering annotations from the MIDI dataset to
encourage the fingers of the hand to reach the corresponding keys.

  * •

Lookahead horizon: Instead of just goal-conditioning the policy for the
current timestep, we additionally include the “goal-trajectory" up to some
lookahead horizon $H$ into the future.

  * •

Energy penalty: We add an extra reward term that penalizes the energy output
of the hand actuators, see Table II.

  * •

Action space reduction: We reduced the dimensionality of the action space by
disabling some DoFs in the hand. We also restricted the range of some
actuators.

  * •

Action-reward: We appended the action and reward obtained at the previous
timestep to the state, see Table II.

##  V Experiments

(a)

(b)

Figure 5: RoboPianist-repertoire-150: (a) Performance of the model-free RL
baseline evaluated at checkpoints trained on increasing amounts of environment
interactions. Each point corresponds to an average over 150 MIDI files, 3
seeds each, with a standard deviation shading computed over all MIDI files.
(b) Performance of the MPC baseline evaluated with an increasing compute
budget (e.g., 2 = double the compute used by the same method running
realtime). Each point corresponds to an average over 150 songs, with a
standard deviation shading computed over all MIDI files.

The goal of our experiments is twofold. First, to characterize the performance
envelope of the algorithms in Section IV and gain insight into the key
challenges of the RoboPianist benchmark. Second, to investigate how the MDP
structure influences overall policy performance. We study these questions in
the context of three experimental setups: RoboPianist-prelude, RoboPianist-
etude-12 and RoboPianist-repertoire-150.

###  V-A Experimental setup

RoboPianist-prelude consists of one piece with which one can extensively
compare the effects of the design decisions introduced in Subsection IV-B on
the model-free RL baseline. For these experiments, we use a 10-second snippet
of Mozart’s Twelve Variations on “Ah Vous Dirai-Je, Maman” (a.k.a. Twinkle
Twinkle Little Star) – a piece that (i) strikes a good balance in difficulty:
it is not too simple (e.g., requires both hands, there exists an ornament,
etc.) and not too hard (e.g., there are no chords), and (ii) is universally
recognizable and, thus, intuitive for humans to quickly listen to and evaluate
qualitative success.

RoboPianist-repertoire-150 considers a dataset of 150 MIDI files and evaluates
both algorithms mentioned in Section IV (including the best performing variant
of the model-free baseline from RoboPianist-prelude).

RoboPianist-etude-12 considers 12 randomly sampled songs from the full
repertoire of 150 MIDI files and again, evaluates both of the algorithms
mentioned in Section IV. This etude serves as a proxy for the full repertoire,
for smaller scale experiments and/or more modest compute budgets. For results
and analysis on this etude, we refer the reader to the appendix.

Training details. Training the model-free RL baseline takes approximately 5
hours per song on an Intel Xeon E5-2696V3 Processor hardware with 32 cores
(2.3 GHz base clock), 416 GB RAM and 4 Tesla K80 GPUs (n1-highmem-64 machine
type on Google Cloud). Inference for both model-free and model-based variants
is done on an M1 Max, 64GB RAM processor. All model-free policies are trained
and evaluated three times on each task with different random seeds. Model-
based policies are evaluated once per task. We implement our model-free
baseline using JAX [62], and our MPC baseline using MJPC [60]. Complete
hyperparameters and experimental details are listed in the appendix.

###  V-B Overall evaluation

Heuristic | Pearson Correlation
---|---
Notes Per Second | -0.266
Pitch Class Entropy | -0.312
Total Time | -0.383
Max Polyphony | -0.458
Mean Polyphony | -0.447
Unique Pitches | -0.560
Unique Pitch Classes | -0.238
Pitch Range | -0.486
TABLE III: Correlation of musical properties of the MIDI dataset with the
performance of the model-free RL baseline as measured by its F1 score on
RoboPianist-repertoire-150. This shows that for example, more unique pitches
(different keys pressed) in a song is especially correlated with it being
harder.

The performance of the model-free RL baseline as well as the MPC baseline on
RoboPianist-repertoire-150 on all 150 tasks is illustrated in Figure 5. See
Figure 2 for individual results on all songs for model-free, and the appendix
for full results for model-based.

(a)

(b)

(c)

Figure 6: RoboPianist-prelude: (a) Performance of the model-free RL baseline
trained with different control frequencies, (b) Performance of the model-free
RL baseline trained with different values of the goal lookahead horizon, and
(c) Performance of the model-free RL baseline trained with and without a
fingering reward.

What makes a task hard? We can sort the songs in descending order based on the
F1 scores of the model-free RL policies, which presents a birds-eye view of
the difficulty between songs (i.e., lower is harder). By measuring the Pearson
correlation between the F1 scores against heuristics calculated per song from
the 150 MIDI files (shown in Table III), we can observe which task attributes
contribute the most to the difficulty.

Interestingly, we observe that the strongest negatively correlated variable is
the number of unique pitches (i.e., different keys on the keyboard) – the more
there are, the harder the task. This could suggest that the number of unique
notes to hit for a task contributes to the diversity of control trajectories
that need to be addressed by the policy. This attribute is followed by the
second-most negatively correlated variable maximum polyphony, i.e., the
average maximum number of notes active at any point in time. Successfully
playing multiple notes imposes additional constraints on precision. For high-
dimensional control, this could be akin to threading the needle through a more
narrow solution space, which can be more challenging to discover with RL
through trial and error. These correlations allow us to project difficulty
onto new prospective tasks as the dataset expands, and point to interesting
areas for future research.

###  V-C Results and baseline studies

The following subsection discusses observations from varying the design
decisions detailed in Section IV of the model-free baseline policy on
RoboPianist-prelude. These observations may continue to be prevalent for
related future work in high-dimensional control.

Energy penalty leads to more spatially precise control. High-dimensional
control policies trained with RL and random exploration are subject to
learning more ecstatic policies (visible with non-smooth trajectories). In
particular, these movements deteriorate performance on a task like piano
playing, which requires both spatial precision and timing. As shown in Figure
4, we observe substantial improvements in quantitative performance (and much
less variance) by adding an energy penalty to the policy, which reduces the
ecstatic movements and bang-bang-like control during the initial stages of
training.

Looking into the future leads to more temporally precise control. We observe
additional improvements, both in performance and variance, from appending
future goal states (up to a certain horizon) to the goal vector. Intuitively,
this allows the policy to plan better for future notes – for example by
placing the non-finger joints (e.g., the wrist) in a manner that allows more
timely reaching of notes at the next timestep. However, as we show in 6(b),
increasing the goal lookahead horizon above 0.5 seconds leads to reduced
performance.

Reducing the action space trains faster and reduces un-human-like thumb
movement. To alleviate exploration even further, we explore the effect of
disabling degrees of freedom in the Shadow Hand that either do not exist in
the human hand (e.g., the little finger being opposable) or are not strictly
necessary for most songs. We additionally reduce the joint range of the thumb,
which on the Shadow Hand, can reach backward in a manner not possible on a
human hand. As shown in Figure 4, this change leads to faster training.
Qualitatively, the change is even more pronounced with the policies trained
with the reduced action space exhibiting more natural-looking thumb behaviors
(see the project website for a side-by-side comparison).

Fingering information alleviates exploration. Without the fingering reward,
the policy struggles to learn meaningful behaviors as shown in 6(c).

The control frequency matters. As shown in 6(a), the control frequency has a
substantial effect on learning. We find that 20Hz is a sweet spot, with
notably 100Hz drastically reducing the final score, and 10Hz converging a bit
slower. At 100Hz, the MDP becomes too long-horizon, which complicates
exploration, and at 10Hz, the discretization of the MIDI file becomes too
coarse, which negatively impacts the timing of the notes.

Adding the action and reward to the state helps. As shown in Figure 4, adding
the action and reward from the previous timestep, while marginal, also helps
improve the convergence speed of the policy. The previous action helps the
policy reason about velocities, which can enable more spatially and temporally
precise control. Note we could have obtained a similar effect by stacking a
history of past observations – we chose the option that has the smallest
increase on its dimension.

##  VI Discussion and Conclusion

In this paper, we introduced the RoboPianist benchmark, which provides a
simulation framework and suite of tasks in the form of a corpora of songs,
together with baseline methods and evaluations, for studying the challenging
high-dimensional control problem of mastering piano-playing with bi-manual
hands. We showed that both well-tuned model-free and model-based baselines
struggle on this benchmark and explored different ways in which to improve
them. There is an array of exciting future directions to explore with
RoboPianist, including for example: pushing both model-based and model-free
methods, using human priors to accelerate learning, studying zero shot
generalization to new songs, and using multimodal data like sound and touch.

##  VII Acknowledgments

This project was supported in part by ONR #N00014-22-1-2121 under the Science
of Autonomy program.

## References

  * [1] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” arXiv preprint arXiv:2206.04615, 2022.
  * [2] T. Winograd, “Understanding natural language,” Cognitive psychology, vol. 3, no. 1, pp. 1–191, 1972.
  * [3] H. Levesque, E. Davis, and L. Morgenstern, “The winograd schema challenge,” in Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
  * [4] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine, “D4RL: Datasets for deep data-driven reinforcement learning,” arXiv preprint arXiv:2004.07219, 2020.
  * [5] S. Yuan, L. Shao, C. L. Yako, A. M. Gruebele, and J. K. Salisbury, “Design and control of roller grasper v2 for in-hand manipulation,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9151–9158, 2020.
  * [6] Z. Xu and E. Todorov, “Design of a highly biomimetic anthropomorphic robotic hand towards artificial limb regeneration,” in 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3485–3492, 2016.
  * [7] C. McCann, V. Patel, and A. Dollar, “The stewart hand: A highly dexterous, six-degrees-of-freedom manipulator based on the stewart-gough platform,” icra, vol. 28, no. 2, pp. 23–36, 2021.
  * [8] R. Fearing, “Implementing a force strategy for object re-orientation,” in Proceedings. 1986 IEEE International Conference on Robotics and Automation, vol. 3, pp. 96–102, 1986.
  * [9] D. Rus, “In-hand dexterous manipulation of piecewise-smooth 3-d objects,” The International Journal of Robotics Research, vol. 18, no. 4, pp. 355–381, 1999.
  * [10] A. Okamura, N. Smaby, and M. Cutkosky, “An overview of dexterous manipulation,” in Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065), vol. 1, pp. 255–262 vol.1, 2000.
  * [11] T. Pang, H. J. T. Suh, L. Yang, and R. Tedrake, “Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models,” ArXiv, vol. abs/2206.10787, 2022.
  * [12] R. R. Ma and A. M. Dollar, “On dexterity and dexterous manipulation,” in 2011 15th International Conference on Advanced Robotics (ICAR), pp. 1–7, 2011\.
  * [13] M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning dexterous in-hand manipulation,” The International Journal of Robotics Research, vol. 39, pp. 20 – 3, 2018.
  * [14] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. A. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. M. Zhang, “Solving rubik’s cube with a robot hand,” ArXiv, vol. abs/1910.07113, 2019\.
  * [15] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al., “Dextreme: Transfer of agile in-hand manipulation from simulation to reality,” arXiv preprint arXiv:2210.13702, 2022.
  * [16] T. Chen, J. Xu, and P. Agrawal, “A system for general in-hand object re-orientation,” in Conference on Robot Learning, pp. 297–307, 2022.
  * [17] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar, “Deep dynamics models for learning dexterous manipulation,” Conference on Robot Learning (CoRL), vol. abs/1909.11652, 2019.
  * [18] T. Chen, M. Tippur, S. Wu, V. Kumar, E. H. Adelson, and P. Agrawal, “Visual dexterity: In-hand dexterous manipulation from depth,” ArXiv, vol. abs/2211.11744, 2022.
  * [19] Y. Chen, Y. Yang, T. Wu, S. Wang, X. Feng, J. Jiang, S. M. McAleer, H. Dong, Z. Lu, and S.-C. Zhu, “Towards human-level bimanual dexterous manipulation with reinforcement learning,” arXiv preprint arXiv:2206.08686, 2022.
  * [20] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik, “In-Hand Object Rotation via Rapid Motor Adaptation,” in Conference on Robot Learning (CoRL), 2022\.
  * [21] I. Mordatch, E. Todorov, and Z. Popović, “Discovery of complex behaviors through contact-invariant optimization,” ACM Trans. Graph., vol. 31, jul 2012.
  * [22] A. M. Castro, F. N. Permenter, and X. Han, “An unconstrained convex formulation of compliant contact,” IEEE Transactions on Robotics, 2022\.
  * [23] A. M. Okamura, N. Smaby, and M. R. Cutkosky, “An overview of dexterous manipulation,” in Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), vol. 1, pp. 255–262, IEEE, 2000.
  * [24] A. Rajeswaran, V. Kumar, A. Gupta, J. Schulman, E. Todorov, and S. Levine, “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” ArXiv, vol. abs/1709.10087, 2017.
  * [25] B. Sundaralingam and T. Hermans, “Relaxed-rigidity constraints: kinematic trajectory optimization and collision avoidance for in-grasp manipulation,” Autonomous Robots, vol. 43, pp. 469–483, 2019.
  * [26] I. Radosavovic, X. Wang, L. Pinto, and J. Malik, “State-only imitation learning for dexterous manipulation,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7865–7871, IEEE, 2021\.
  * [27] R. R. Ma and A. M. Dollar, “On dexterity and dexterous manipulation,” in 2011 15th International Conference on Advanced Robotics (ICAR), pp. 1–7, IEEE, 2011.
  * [28] C. Smith, Y. Karayiannidis, L. Nalpantidis, X. Gratal, P. Qi, D. V. Dimarogonas, and D. Kragic, “Dual arm manipulation—a survey,” Robotics and Autonomous systems, vol. 60, no. 10, pp. 1340–1353, 2012.
  * [29] H. J. Charlesworth and G. Montana, “Solving challenging dexterous manipulation tasks with trajectory optimisation and reinforcement learning,” in International Conference on Machine Learning, pp. 1496–1506, PMLR, 2021.
  * [30] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami, et al., “Emergence of locomotion behaviours in rich environments,” arXiv preprint arXiv:1707.02286, 2017.
  * [31] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman, “Dynamics-aware unsupervised discovery of skills,” in International Conference on Learning Representations, 2020.
  * [32] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa, “dm_control: Software and tasks for continuous control,” Software Impacts, vol. 6, p. 100022, 2020.
  * [33] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa, “AMP: Adversarial motion priors for stylized physics-based character control,” ACM Transactions on Graphics (TOG), vol. 40, no. 4, pp. 1–20, 2021.
  * [34] J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess, “Neural probabilistic motor primitives for humanoid control,” in International Conference on Learning Representations, 2019.
  * [35] J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess, “Catch & carry: reusable neural controllers for vision-guided whole-body tasks,” ACM Transactions on Graphics (TOG), vol. 39, no. 4, pp. 39–1, 2020.
  * [36] A. Escontrela, X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and P. Abbeel, “Adversarial motion priors make good substitutes for complex reward functions,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 25–32, IEEE, 2022.
  * [37] I. Kato, S. Ohteru, K. Shirai, T. Matsushima, S. Narita, S. Sugano, T. Kobayashi, and E. Fujisawa, “The robot musician ‘wabot-2’(waseda robot-2),” Robotics, vol. 3, no. 2, pp. 143–155, 1987.
  * [38] J.-C. Lin, H.-H. Huang, Y.-F. Li, J.-C. Tai, and L.-W. Liu, “Electronic piano playing robot,” in 2010 International Symposium on Computer, Communication, Control and Automation (3CA), vol. 2, pp. 353–356, IEEE, 2010\.
  * [39] T. Maloney, “Piano-playing robotic arm,” tech. rep., 2019.
  * [40] J. Hughes and P. Maiolino, “An anthropomorphic soft skeleton hand exploiting conditional models for piano playing,” Science Robotics, vol. 3, no. 25, 2018.
  * [41] R. Castro Ornelas, Robotic Finger Hardware and Controls Design for Dynamic Piano Playing.  PhD thesis, Massachusetts Institute of Technology, 2022.
  * [42] D. Zhang, J. Lei, B. Li, D. Lau, and C. Cameron, “Design and analysis of a piano playing robot,” in 2009 International Conference on Information and Automation, pp. 757–761, 2009.
  * [43] Y.-F. Li and L.-L. Chuang, “Controller design for music playing robot—applied to the anthropomorphic piano robot,” in 2013 IEEE 10th International Conference on Power Electronics and Drive Systems (PEDS), pp. 968–973, IEEE, 2013.
  * [44] A. Zhang, M. Malhotra, and Y. Matsuoka, “Musical piano performance by the act hand,” in 2011 IEEE international conference on robotics and automation, pp. 3536–3541, IEEE, 2011.
  * [45] B. Scholz, Playing Piano with a Shadow Dexterous Hand.  PhD thesis, Universität Hamburg, 2019.
  * [46] “Shadow Dexterous Hand,” 2005.
  * [47] S. Yeon, “Playing piano with a robotic hand,” 2022.
  * [48] H. Xu, Y. Luo, S. Wang, T. Darrell, and R. Calandra, “Towards learning to play piano with dexterous hands and touch,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10410–10416, IEEE, 2022\.
  * [49] A. Muldal, Y. Doron, J. Aslanides, T. Harley, T. Ward, and S. Liu, “dm_env: A python interface for reinforcement learning environments,” 2019.
  * [50] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.
  * [51] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033, IEEE, 2012.
  * [52] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa, “dm_control: Software and tasks for continuous control,” Software Impacts, vol. 6, p. 100022, 2020.
  * [53] M. M. Contributors, “MuJoCo Menagerie: A collection of high-quality simulation models for MuJoCo,” 2022.
  * [54] K. M. Instruments, “Kawai Vertical Piano Regulation Manual,” 2019.
  * [55] E. Nakamura, Y. Saito, and K. Yoshii, “Statistical learning and estimation of piano fingering,” ArXiv, vol. abs/1904.10237, 2019.
  * [56] T. Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka, “Dropout q-functions for doubly efficient reinforcement learning,” International Conference on Learning Representations (ICLR), 2022.
  * [57] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in ICML, 2018.
  * [58] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” ArXiv, vol. abs/1707.06347, 2017.
  * [59] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021.
  * [60] T. A. Howell, N. Gileadi, S. Tunyasuvunakool, K. Zakka, T. Erez, and Y. Tassa, “Predictive sampling: Real-time behaviour synthesis with mujoco,” ArXiv, vol. abs/2212.00541, 2022.
  * [61] Y. Tassa, T. Erez, and E. Todorov, “Synthesis and stabilization of complex behaviors through online trajectory optimization,” 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4906–4913, 2012\.
  * [62] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang, “JAX: composable transformations of Python+NumPy programs,” 2018.
  * [63] H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” arXiv e-prints, 2015.
  * [64] S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018, 2018.
  * [65] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., vol. 15, pp. 1929–1958, 2014.
  * [66] J. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” ArXiv, vol. abs/1607.06450, 2016.
  * [67] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in International Conference on Artificial Intelligence and Statistics, 2010.
  * [68] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014.

##  Appendix A Experiment details

###  A-A Model-free reinforcement learning

Computing infrastructure and experiment running time

Our model-free RL codebase is implemented in JAX [62]. Experiments were
performed on a Google Cloud n1-highmem-64 machine with an Intel Xeon E5-2696V3
Processor hardware with 32 cores (2.3 GHz base clock), 416 GB RAM and 4 Tesla
K80 GPUs. Each “run”, i.e., the training and evaluation of a policy on one
task with one seed, took an average of 5 hrs wall clock time. These run times
are recorded while performing up to 8 runs in parallel.

Network architecture

We use a regularized variant of clipped double Q-learning [63, 64],
specifically DroQ [56], for the critic. Each $Q$-function is parameterized by
a 3-layer multi-layer perceptron (MLP) with ReLU activations. Each linear
layer is followed by dropout [65] with a rate of $0.01$ and layer
normalization [66]. The actor is implemented as a tanh-diagonal-Gaussian, and
is also parameterized by a 3-layer MLP that outputs a mean and covariance.
Both actor and critic MLPs have hidden layers with $256$ neurons and their
weights are initialized with Xavier initialization [67], while their biases
are initialized to zero.

Training and evaluation

We first collect $5000$ seed observations with a uniform random policy, after
which we sample actions using the RL policy. We then perform one gradient
update every time we receive a new environment observation. We use the Adam
[68] optimizer for neural network optimization. Evaluation happens in parallel
in a background thread every $10000$ steps. The latest policy checkpoint is
rolled out by taking the mean of the output (i.e., no sampling). Since our
environment is “fixed”, we perform only one rollout per evaluation.

Reward formulation

The reward function for training the model-free RL baseline consists of four
terms: 1) a key press term $r_{\text{key}}$, 2) a move finger to key term
$r_{\text{finger}}$, 3) a forearm collision term $r_{\text{forearm}}$ and 4)
an energy penalty term $r_{\text{energy}}$.

$r_{\text{key}}$ encourages the policy to press the keys that need to be
pressed and discourages it from pressing keys that shouldn’t be pressed. It is
implemented as:

$r_{\text{key}}=0.5\cdot\left(\dfrac{1}{K}\sum_{i}^{K}g(||k^{i}_{s}-1||_{2})\right)+0.5\cdot(1-\mathbf{1}_{\\{\mathsf{false\;positive}\\}}),$

where K is the number of keys that need to be pressed at the current timestep,
$k_{s}$ is the normalized joint position of the key between 0 and 1, and
$\mathbf{1}_{\\{\mathsf{false\;positive}\\}}$ is an indicator function that is
1 if any key that should not be pressed creates a sound (i.e., its normalized
joint position crossed 0.5). $g$ is the tolerance function from the dm_control
[32] library: it takes the L2 distance of $k_{s}$ and $1$ and converts it into
a bounded positive number between 0 and 1. We use the parameters bounds=0.05
and margin=0.5.

$r_{\text{finger}}$ encourages the fingers that are active at the current
timestep to move as close as possible to the keys they need to press. It is
implemented as:

$r_{\text{finger}}=\dfrac{1}{K}\sum_{i}^{K}g(||p^{i}_{f}-p^{i}_{k}||_{2}),$

where $p_{f}$ is the Cartesian position of the finger and $p_{i}$ is the
Cartesian position of a point centered at the surface of the key. $g$ for this
reward is parameterized by bounds=0.01 and margin=0.1.

$r_{\text{forearm}}$ encourages the shadow hand forearms not to collide. It is
implemented as:

$r_{\text{forearm}}=1-\mathbf{1}_{\\{\mathsf{collision}\\}},$

where $\mathbf{1}_{\\{\mathsf{collision}\\}}$ is 1 if the forearms are in
collision and 0 otherwise.

Finally, $r_{\text{energy}}$ penalizes high energy expenditure and is
implemented as:

$r_{\text{energy}}=|\mathsf{\tau_{joints}}|^{\top}|\mathsf{v_{joints}}|,$

where $\mathsf{\tau_{joints}}$ is a vector of joint torques and
$\mathsf{v_{joints}}$ is a vector of joint velocities.

The final reward function sums up the aforementioned terms as follows:

$r_{\text{total}}=r_{\text{key}}+r_{\text{finger}}+0.4\cdot
r_{\text{forearm}}-0.005\cdot r_{\text{energy}}$

Other hyperparameters

For a comprehensive list of hyperparameters used for training the model-free
RL policy, see Table IV.

Hyperparameter | Value
---|---
Total train steps | 1M
Optimizer |
Type | ADAM
Learning rate | $3\times 10^{-4}$
$\beta_{1}$ | $0.9$
$\beta_{2}$ | $0.999$
Critic |
Hidden units | $256$
Hidden layers | $3$
Non-linearity | ReLU
Dropout rate | 0.01
Actor |
Hidden units | $256$
Hidden layers | $3$
Non-linearity | ReLU
Misc. |
Discount factor | $0.99$
Minibatch size | $256$
Replay period every |  $1$ step
Eval period every |  $10000$ step
Number of eval episodes | $1$
Replay buffer capacity | 1M
Seed steps | $5000$
Critic target update frequency | $1$
Actor update frequency | $1$
Critic target EMA momentum ($\tau_{Q}$) | $0.005$
Actor log std dev. bounds | $[-20,2]$
Entropy temperature | $1.0$
Learnable temperature | True
TABLE IV: Hyperparameters for all model-free RL experiments.

###  A-B Model predictive control

Computing infrastructure and experiment running time

Our model-based codebase is implemented in C++ with MJPC [60]. Experiments
were performed on a 2021 M1 Max Macbook Pro with 64 GB of RAM.

Algorithm

We use MPC with Predictive Sampling (PS) as the planner. PS is a derivative-
free sampling-based algorithm that iteratively improves a nominal sequence of
actions using random search. Concretely, $N$ candidates are created at every
iteration by sampling from a Gaussian with the nominal as the mean and a fixed
standard deviation $\sigma$. The returns from the candidates are evaluated,
after which the highest scoring candidate is set as the new nominal. The
action sequences are represented with cubic splines to reduce the search space
and smooth the trajectory. In our experiments, we used $N=10$, $\sigma=0.05$,
and a spline dimension of 2. We plan over a horizon of $0.2$ seconds, use a
planning time step of $0.01$ seconds and a physics time step of $0.005$
seconds.

Cost formulation

The cost function for the MPC baseline consists of 2 terms: 1) a key press
term $c_{\text{key}}$, 2) and a move finger to key term $c_{\text{finger}}$.

The costs are implemented similarly to the model-free baseline, but don’t make
use of the $g$ function, i.e., they solely consist in unbounded l2 distances.

The total cost is thus:

$c_{\text{total}}=c_{\text{key}}+c_{\text{finger}}$

Note that we experimented with a control cost and an energy cost but they
decreased performance so we disabled them.

##  Appendix B Supplementary results

###  B-A RoboPianist-prelude

PPO baseline

As mentioned in Section IV, we experimented with training a model-free policy
with PPO [58], but found that it performed much worse than DroQ both in terms
of sample efficiency and wall clock time. In Figure 7, we report the results
of the full training run on RoboPianist-prelude.

Figure 7: Performance of DroQ vs PPO on RoboPianist-prelude.

###  B-B RoboPianist-repertoire-150

Figure 8: Full set of F1 scores achieved by the model-based MPC baseline on
all tasks in RoboPianist-repertoire-150.

The per-MIDI performance of the MPC baseline is illustrated in Figure 8.

###  B-C RoboPianist-etude-12

Baseline | Avg. F1 | Avg. Precision | Avg. Recall
---|---|---|---
Model-free RL | 0.538 $\pm$ 0.122 | 0.990 $\pm$ 0.006 | 0.462 $\pm$ 0.109
MPC (100% realtime) | 0.266 $\pm$ 0.079 | 0.816 $\pm$ 0.073 | 0.225 $\pm$ 0.084
MPC (80% realtime) | 0.276 $\pm$ 0.069 | 0.813 $\pm$ 0.076 | 0.233 $\pm$ 0.081
MPC (20% realtime) | 0.402 $\pm$ 0.056 | 0.786 $\pm$ 0.060 | 0.351 $\pm$ 0.064
MPC (10% realtime) | 0.433 $\pm$ 0.072 | 0.775 $\pm$ 0.059 | 0.385 $\pm$ 0.077
TABLE V: robopianist-etude-12: Aggregate scores for model-free and model-based
baselines averaged over all 12 MIDI files in the etude. The scores for the
model-free baseline are obtained from the final checkpoint at 1M steps.

As mentioned in Section V, we consider a smaller subset of the full repertoire
(12 randomly sampled MIDI files) and report the performance of the model-free
and model-based baselines. These results are summarized in Figure 9 and Table
V.

(a)

(b)

Figure 9: RoboPianist-etude-12: (a) Performance of the model-free RL baseline
evaluated at checkpoints trained on increasing amounts of environment
interactions. Each point corresponds to an average over 12 MIDI files, 3 seeds
each, with a standard deviation shading computed over all MIDI files. (b)
Performance of the MPC baseline evaluated with an increasing compute budget
(e.g., 2 = double the compute used by the same method running realtime). Each
point corresponds to an average over 12 songs, with a standard deviation
shading computed over all MIDI files.