# Music Style Transfer: A Position Paper

Shuqi Dai
Computer Science Department
Peking University
<EMAIL_ADDRESS>
Zheng Zhang
Computer Science Department
New York University Shanghai
<EMAIL_ADDRESS>
Gus G. Xia
Computer Science Department
New York University Shanghai
<EMAIL_ADDRESS>

###### Abstract

> Led by the success of neural style transfer on visual arts, there has been a
> rising trend very recently in the effort of music style transfer. However,
> “music style” is not yet a well-defined concept from a scientific point of
> view. The difficulty lies in the intrinsic multi-level and multi-modal
> character of music representation (which is very different from image
> representation). As a result, depending on their interpretation of “music
> style”, current studies under the category of “music style transfer”, are
> actually solving completely different problems that belong to a variety of
> sub-fields of Computer Music. Also, a vanilla end-to-end approach, which
> aims at dealing with all levels of music representation at once by directly
> adopting the method of image style transfer, leads to poor results. Thus, we
> vitally propose a more scientifically-viable definition of music style
> transfer by breaking it down into precise concepts of timbre style transfer,
> performance style transfer and composition style transfer, as well as to
> connect different aspects of music style transfer with existing well-
> established sub-fields of computer music studies. In addition, we discuss
> the current limitations of music style modeling and its future directions by
> drawing spirit from some deep generative models, especially the ones using
> unsupervised learning and disentanglement techniques.

## Introduction

### Background of Automated Music Generation

The practice of music automation can be traced back to Guido D’Arezzo, a
famous medieval musician who designed a rule-based vowel-to-pitch mapping
algorithm to generate a sequence of notes (?). While “crafting music” is still
the mainstream, algorithmic composition, or in general automated music
generation has become more and more popular nowadays with the development of
modern computers. On the one hand, fast CPUs offer dramatic speedup of
experimentations, so that people can test different ideas much more rapidly.
In addition, various computer-music programming languages (?; ?; ?; ?) have
been invented since the late 1950s , which further boosted the efficiency of
music creation via programming. On the other hand, advanced computational
models and data-driven algorithms have empowered computers to generate more
human-like music via inheriting certain statistics and styles from the
training sets. Recently, with the breakthroughs in artificial neural networks,
deep generative models have become one of the leading techniques for automated
music generation (?). For the examples of mimicking J.S. Bach alone, we have
seen BachBot (?), DeepBach (?), CNNBach (?), etc., and most of them can
generate convincing results.

Despite these promising progress, people still struggle to generate both
natural and creative music through automation. In general, algorithms with
weak constraints are often “too random” and rarely make human-like music,
though many works are interesting and creative from a contemporary
perspective. On the other hand, algorithms with strong constraints (either
explicitly constrained via rules or implicitly constrained by training data)
are mostly “too flat” and lack the exploration and dynamic that can be easily
sensed from genuinely creative works.

### Music Style Transfer: Importance & Challenges

Image style transfer techniques (?) inspired the hope to solve the paradox
above. By separating and recombining music contents and music styles of
different pieces, it is possible to generate new music that is both creative
and human-like. In other words, we can still use our favorite data-driven
algorithms but twist the constraints or optimizations in general by applying
them separately to different aspects (i.e., content and style) of music.

Such effort is named after music style transfer. However, there is a severe
problem: “music style” is a fuzzy term that can literally refer to any aspect
of music, ranging from high-level compositional features (such as tonality and
chord sequence) to low-level acoustic features (such as sound texture and
timbre). This ambiguity is mainly due to the intrinsic multi-level, multi-
modal character of music representation — music can be read, listened to, or
performed, and it all depends on whether we are relying on score (the top-
level, abstract representation), sound (the bottom-level, concrete
representation), or control (the intermediate representation). This is very
different from image representation, and so far no end-to-end system can deal
with all levels of music representation together in an elegant manner.

Consequently, most studies only focus on a certain level/modality of music
representation and therefore have different interpretations of music style.
Depending on the interpretation, the essence of music style transfer also
varies a lot and may even refer to problems evolved from different sub-fields
of computer music, such as algorithmic composition, expressive performance, or
sound synthesis. In other words, we are facing an issue of the many-to-one
collapse of keyword definition. Without further action, an accumulated
upcoming literature all named after “music style transfer” would lead to a
great confusion of the underlying problems to the readers as well as a risk to
ignore the treasures in computer music before the age of deep learning.

In this position paper, we contribute a precise definition of music style
transfer based on the uniqueness of music representation. We start from an
overview of music representation in Section 2 in order to formally introduce
the definition in Section 3, where we also connect different types of music
style transfer with existing well-established computer music studies. In the
end, we discuss the current limitations and possible future directions of
music style modeling by inspecting current unsupervised learning and
disentanglement techniques of deep generative models.

## Multi-level and Multi-modal Representation

Music is widely considered a universal language and there are many previous
discussions on music representations (?; ?; ?). The relationship between music
notation (score) and actual sound is similar to the one between text and
speech. Score serves as a symbolic and highly-abstract visual representation
to efficiently record and communicate music ideas, whereas the sound is a set
of continuous and concrete signal representations that encode all the details
we can hear. Therefore, we can picture the two representations at different
levels, with the score at the top and sound at the bottom (?).

In the middle, people often insert an intermediate representation of
performance control. The reasons are twofold. First, musical semantics and
expression rely heavily on performance control that a funeral hymn can sound
really happy by simply tripling the tempo. Second, the performance control for
many instruments (e.g., a piano keyboard) can be easily parameterized and
therefore very machine friendly. Note that different levels of representation
are not solely mutually exclusive, but the multi-level property offers us a
useful tool to better understand the essence of music content and style. To
fully comprehend different aspects of music style transfer, we shall first
investigate the multi-level property of music representation more in-depth.

### Score Representation

Score representation exists in many forms, including sheet music notation,
lead sheet, chord chart and numbered musical notation. Most of them are highly
symbolic and encode abstract music features indicated by the composer,
including tonality, chord, pitch, timing, dynamics and rich structure
information such as phrases and repetitions.

The key character of score representation is that the encoded features are
mostly discrete with a mix of measurement scale. Take western music notation
(Figure 1) for example. Note onset is a ratio variable and lies on integer
multiples of a certain time unit (usually 1/8 beat is short enough). Pitch is
an interval variable, whose corresponding fundamental frequency always lies in
a discrete sequence. (E.g., the frequency of C4 in the equal-tempered tuning
is 261.63 Hz, the frequency of its successive pitch C“4 is 277.18 Hz, and
there is no other pitch frequencies lie in between.) Dynamics is an ordinal
variable, usually ranging from ppp(the softest) to fff(the loudest). Many
other symbols are nominal variables, such as chord types and repeat signs.
Such characters bring a challenge for generative models since discrete
optimization is in general very difficult and a mixed scale makes some
numerical operations impossible.

Figure 1: An example of western music notation.

### Performance Control Representation

A performance control encodes an interpretation of the corresponding score,
rely on which a performer turns the score into performance motions. A commonly
used control representation is MIDI piano roll (Figure 2), where each note is
encoded by its pitch, dynamics, onset (starting time), and duration. It also
has a number of controllers such as pedal and pitch bend for more performance
nuances. To be specific, pitches are integers in semitones with C4 being 60,
dynamics are integers in velocities units (speed with which the keys are
hitting) ranging from 1 to 127, and timings are floating point numbers in
seconds.

Compared to score representation, the key character of performance control is
the enriched and detailed timing and dynamics information, which more or less
determined the musical expression of a performance. On the other hand, most
structural information such as phrase, repetition, and chord progression is
flattened and become implicit during the translation from the score to
performance control. Note that performance control is largely independent from
the actual instrument; it is not yet the final music sound and still
considered a middle-level abstraction.

Figure 2: An example of MIDI piano roll representation.

### Sound Representation

Figure 3: A waveform example where the horizontal axis represents time and the
vertical axis represents amplitude. Figure 4: A spectrogram example where the
horizontal axis represents time, vertical axis represents frequency, and the
color represents energy distribution on different frequencies.

Sound, the concrete signal representation, can be seen as an acoustic
realization of the corresponding performance control via a certain instrument.
Two commonly used formats for sound representation are waveform (Figure 3) and
spectrogram (Figure 4).

The key character of sound representation is purely continuous and rich in
acoustic details such as timbre, articulation, and other nuances not available
in other levels of representation. At the expense of such acoustic details,
all symbolic abstractions together with precise performance control
information become no more explicit and get hidden in the audio.

### Representation, Content, and Style

Table 1 shows a summary of different music representations. It is important to
notice that the multi-level architecture actually has already implied the
essence of music content and music style, i.e., music content is the
information extracted through abstraction (from a lower level to a higher
level), while music style is the information enriched through interpretation
and realization (from a higher level to a lower level).

Table 1: A summary of music representations. | Sensory | Unique | Scale of | Type of
---|---|---|---|---
| system | features | measure | data
|  | structure & |  |
Score | visual | symbolic | all | discrete
(top) |  | abstractions |  |
|  | expressive |  |
Control | motor | timing & | interval | mixed
(middle) |  | dynamics | & ratio |
Sound |  | acoustic |  | contin-
(bottom) | auditory | details | ratio | uous

Thus, a complete end-to-end system for music style transfer should at least
fulfill three requirements: 1) be cross-modal and flexible to deal with
different measurement scales, 2) automatically extract the performance control
and score information from a sound input, and 3) freely manipulate music
representations at any level. However, we have to accept the fact that such
systems do not yet exist and may not emerge in the near future. The second
requirement alone remains an open problem (especially for polyphonic music),
and has been the main focus of the whole field of music information retrieval
for many years.

Therefore, it is beneficial to first solve style transfer for each level of
music representation and gradually integrate different components into one
system. A hasty attempt at an end-to-end music style transfer system by
directly adopting the algorithms for image style transfer (?; ?) would only
lead to results that sound like a casual remix of different pieces of music.

## Music Style Transfer: A Precise Definition
and Related Work

We present the precise definitions of music style transfer for each level of
representation in a bottom-up order. They are: 1) timbre style transfer for
sound, 2) performance style transfer for performance control, and 3)
composition style transfer for score. We also include a brief overview of the
related work and connect them with existing sub-fields of computer music.

### Timbre Style Transfer

Definition 1: Timbre style transfer applies to sound representation. It means
to alter the timbre information in a meaningful way while preserving the
hidden content of performance control.

A successful timbre style transfer would allow us to reproduce a trumpet
performance by a flute with the same musical expression. Timbre style transfer
is closely related to sound synthesis (?), especially the studies aiming to
synthesize different sound of acoustic instruments. The difference is that
timbre style transfer requires a disentanglement of timbre (style) and
performance control (content) and implies that there is room to create new
timbre through the combination of different ones.

Two recent pioneer studies on timbre style transfer are Google’s WaveNet
autoencoders (?) and Stanford’s audio spectrograms neural style transfer
system (?). The former built an autoencoder for raw waveform using WaveNet (a
dilated temporal convolutional neural network). The bottleneck hidden layer is
therefore considered a timbre representation and used to create new timbre
through linear interpolation. The latter treated audio spectrograms as images
and applied image style transfer with additional carefully designed
constraints on temporal and frequency energy envelopes.

We shall also see the limitations. For both works, the disentanglement of
timbre and performance control information is not yet very successful,
especially when the length of the processed audio unit is long. Also, from a
synthesis perspective, the sound quality of synthesized instruments is still
far from the state-of-art learning-based synthesis techniques (?) and worth
further investigation. As a side note, VisualSoundtrack (?), which is named
after “style transfer”, is actually a synthesis system. It requires human
inputs of pitch and no disentanglement is involved.

### Performance Style Transfer

Definition 2: Performance style transfer applies to performance control
representation. It means to alter the control information in a meaningful way
while preserving the implicit score content.

A successful performance style transfer would allow us to transfer Louis
Armstrong’s interpretation of Summertime to the one of Miles Davis. It is
closely related to expressive performance rendering, which studies how to
convert static scores into human-like expressive performances by different
computational models. (?; ?; ?) The difference is that performance style
transfer requires a disentanglement of control (style) and score information
(content) and implies that there is room to create new musical expression
through the combination of different controls.

As far as we know, there is no work on performance style transfer yet, though
performer identification (?; ?) has been studied for over a decade. One close
attempt is the recent Duet Interaction system (?) that can generate an
expressive accompaniment based on the performance style of a solo, but it
requires a pre-defined score and cannot create new performance styles. As a
side note, the work named after “neural translation of musical style” (?) is
actually an expressive performance rendering system, which focuses on dynamic
generation given a score input. Thus, performance style transfer remains a
brand-new field worth exploring.

### Composition Style Transfer

For many forms of score, there is room for further abstraction. Take western
music notation for example, the most identifiable score feature, in general,
is the melody contour and sometimes with the structural functions of harmony
(?). This is especially the case for tonal music.

Definition 3: Composition style transfer means to preserve the identifiable
melody contour (and the underlying structural functions of harmony) while
altering some other score features in a meaningful way.

A successful composition style transfer would allow us to create variation,
improvisation, re-harmonization, or re-arrangement of a piece of music. A
representative masterpiece is the Twelve Variations on “Ah vous dirai-je,
Maman” by Mozart. Take the first variation for example, it mostly preserved
the melody contour and chord progression of the theme and altered the rhythm
and texture to a large extent. Recent high-quality pieces (made by human)
include: Improvisation of “Mary had a little
lamb”111https://youtu.be/Q6Usd3_fbq8, a Korean style Carmen
Overture222https://youtu.be/hKv2_UCo1ZQ, and a Chinese style Mozart
Sonata333https://soundcloud.com/wang-michael-452158298/sets/style-transform-
of-mozart-sonata. Composition style transfer is closely related to stylistic
automatic composition, which can be traced back to David Cope (?). The
difference between these two topics is that composition style transfer
requires a disentanglement of different score features and implies that there
is room to create new types/idioms of score features (such rhythm, texture,
and chord progression) through the combination of different ones.

Pioneer studies on automatic composition style transfer include (?; ?; ?),
where the first two deal with monophonic composition and the last one deals
with polyphonic composition. The work (?) builds pitch and rhythm models
separately for different music genres and then create new melodies through the
combination of the pitch model of one genre and the rhythm model of another
genre. The works by (?; ?) rely on the power of explicit rules to modify
melody and merge different chord progressions, respectively. The work (?)
enforces certain music structures by considering additional template-matching
constraints in the optimization procedure.

The demo pieces created by these early studies are still quite immature,
especially compared to the pieces made by humans. The major problem is
actually not “how to transfer the composition style” but “how to model it” in
the first place. Current composition models still lack the capacity or
representation of music structure and the hidden “grammar” of chord
progressions. Note that most successful cases of the automatic stylistic
composition are related to Bach, and at least for non-experts the structure of
Bach’s compositions is rather local and easy to perceive compared to many
other composers. This is unlikely to be a coincidence and worth the attention
of future studies.

## Future Directions of Music Style Modeling

How shall we model the styles of composition, performance, and timbre for a
better transfer effect? Most current studies use the following three
approaches to model music styles: 1) to inherit the style implicitly from the
training set (?; ?; ?; ?), 2) to use simple style embedding for generation
(?), and 3) to apply style-related constraints for generation. In other words,
they all require a manually-defined style representation or style label for
generation.

As stated earlier, style transfer calls for disentanglement of content and
style. It would make more sense to learn the disentanglement rather than
crafting it by hand. In the field of deep generative modeling, learning
disentanglement has already attracted a vast amount of attention (?; ?; ?; ?).
For image generation tasks, adversarial training has achieved disentanglement
of latent factors and been applied within the generative adversarial network
(?) and variational auto-encoder (VAE) (?) framework. A pioneering study has
applied the VAE framework for algorithmic composition (?). Though the
convincing results are still bounded by the length of two bars, it is
conceivable to apply it for style transfer task with some modification.

Upon a successful disentanglement, style can be considered as one of the
latent factors and style transfer can be accomplished in two steps. The first
is to disentangle a “style” code from the hidden representation that generates
the music, and second is to “plug” such code into an appropriate sequence
generation framework that preserves all other factors.

## Conclusion

In conclusion, music style transfer is a new research field which promises
novel computational tools to generate both creative and human-like music.
Questions like “what if Miles Davis wrote Twelve Variations on ‘Ah vous dirai-
je, Maman’ and performed it on a flute” would be no more purely imaginary. In
order to generate meaningful results, future works should be aware of the
multi-level, multi-modal music representation and be clear whether the focus
is timbre style transfer, performance style transfer, or composition style
transfer. Also, the automatic disentanglement of content and style
representation is the key for high-quality style transfer algorithms and worth
the effort from the whole field, and we believe that it is an efficient way,
if not the only way, towards a complete end-to-end, cross-modal music style
transfer system.

## References

  * [Ananthabhotla and Paradiso 2017] Ananthabhotla, I., and Paradiso, J. A.  2017\.  Visualsoundtrack: An approach to style transfer in the context of soundtrack prototyping.  In International Computer Music Conference (ICMC-2017).
  * [Boulanger 2000] Boulanger, R. C.  2000\.  The Csound book: perspectives in software synthesis, sound design, signal processing, and programming.  MIT press.
  * [Briot, Hadjeres, and Pachet 2017] Briot, J.-P.; Hadjeres, G.; and Pachet, F.  2017\.  Deep learning techniques for music generation-a survey.  arXiv preprint arXiv:1709.01620.
  * [Chen et al. 2016] Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; and Abbeel, P.  2016\.  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  In Advances in Neural Information Processing Systems, 2172–2180.
  * [Cope and Mayer 1996] Cope, D., and Mayer, M. J.  1996\.  Experiments in musical intelligence, volume 12.  AR editions Madison, WI.
  * [Dannenberg 1993] Dannenberg, R. B.  1993\.  Music representation issues, techniques, and systems.  Computer Music Journal 17(3):20–30.
  * [Dannenberg 1997] Dannenberg, R. B.  1997\.  Machine tongues xix: Nyquist, a language for composition and sound synthesis.  Computer Music Journal 21(3):50–60.
  * [Dmitry and Vadim 2016] Dmitry, U., and Vadim, L.  2016\.  Audio texture synthesis and style transfer.  https://dmitryulyanov.github.io/audio-texture-synthesis-and-style-transfer.
  * [Engel et al. 2017] Engel, J.; Resnick, C.; Roberts, A.; Dieleman, S.; Eck, D.; Simonyan, K.; and Norouzi, M.  2017\.  Neural audio synthesis of musical notes with wavenet autoencoders.  arXiv preprint arXiv:1704.01279.
  * [Gao 2017] Gao, Y.  2017\.  Towards neural music style transfer.  Master Thesis, New York University.  https://github.com/821760408-sp/the-wavenet-pianist.
  * [Gatys, Ecker, and Bethge 2015] Gatys, L. A.; Ecker, A. S.; and Bethge, M.  2015\.  A neural algorithm of artistic style.  arXiv preprint arXiv:1508.06576.
  * [Hadjeres and Pachet 2016] Hadjeres, G., and Pachet, F.  2016\.  Deepbach: a steerable model for bach chorales generation.  arXiv preprint arXiv:1612.01010.
  * [Hu 2004] Hu, N.  2004\.  Automatic Construction of Synthetic Musical Instruments and Performers.  Ph.D. Dissertation, Carnegie Mellon University.
  * [Huang et al. 2017] Huang, C.-Z. A.; Cooijmans, T.; Roberts, A.; Courville, A.; and Eck, D.  2017\.  Counterpoint by convolution.  In 18th International Society for Music Information Retrieval Conference (ISMIR-2017).
  * [Kaliakatsos-Papakostas et al. 2017] Kaliakatsos-Papakostas, M.; Queiroz, M.; Tsougras, C.; and Cambouropoulos, E.  2017\.  Conceptual blending of harmonic spaces for creative melodic harmonisation.  Journal of New Music Research 46(4):305–328.
  * [Karimi et al. 2017] Karimi, A.-H.; Banijamali, E.; Wong, A. W.; and Ghodsi, A.  2017\.  Jade: Joint autoencoders for dis-entanglement.  In Learning Disentangled Representations, NIPS 2017 Workshop.
  * [Kim and Mnih 2017] Kim, H., and Mnih, A.  2017\.  Disentangling by factorising.  In Learning Disentangled Representations, NIPS 2017 Workshop.
  * [Kirke and Miranda 2009] Kirke, A., and Miranda, E. R.  2009\.  A survey of computer systems for expressive music performance.  ACM Computing Surveys (CSUR) 42(1):3.
  * [Larsson, Nilsson, and Kågebäck 2017] Larsson, M.; Nilsson, A.; and Kågebäck, M.  2017\.  Disentangled representations for manipulation of sentiment in text.  In Learning Disentangled Representations, NIPS 2017 Workshop.
  * [Lattner, Grachten, and Widmer 2016] Lattner, S.; Grachten, M.; and Widmer, G.  2016\.  Imposing higher-level structure in polyphonic music generation using convolutional restricted boltzmann machines and constraints.  arXiv preprint arXiv:1612.04742.
  * [Liang 2016] Liang, F.  2016\.  Bachbot: Automatic composition in the style of bach chorales.  Masters thesis, University of Cambridge.
  * [Loy 1989] Loy, G.  1989\.  Composing with computers: A survey of some compositional formalisms and music programming languages.  In Current directions in computer music research, 291–396.  MIT Press.
  * [Malik and Ek 2017] Malik, I., and Ek, C. H.  2017\.  Neural translation of musical style.  arXiv preprint arXiv:1708.03535.
  * [Mao, Shin, and Cottrell 2018] Mao, H. H.; Shin, T.; and Cottrell, G. W.  2018\.  Deepj: Style-specific music generation.  arXiv preprint arXiv:1801.00887.
  * [Mathieu et al. 2016] Mathieu, M. F.; Zhao, J. J.; Zhao, J.; Ramesh, A.; Sprechmann, P.; and LeCun, Y.  2016\.  Disentangling factors of variation in deep representation using adversarial training.  In Advances in Neural Information Processing Systems, 5040–5048.
  * [McCartney 1996] McCartney, J.  1996\.  Supercollider: a new real time synthesis language.
  * [Müller et al. 2013] Müller, M.; Prätzlich, T.; Bohl, B.; and Veit, J.  2013\.  Freischutz digital: A multimodal scenario for informed music processing.  In Image Analysis for Multimedia Interactive Services (WIAMIS), 2013 14th International Workshop on, 1–4.  IEEE.
  * [Pati 2018] Pati, A.  2018\.  Neural style transfer for musical melodies.  Music Informatics Group, Georgia Tech Center for Music Technology.  https://ashispati.github.io//style-transfer/.
  * [Ramirez, Maestre, and Serra 2010] Ramirez, R.; Maestre, E.; and Serra, X.  2010\.  Automatic performer identification in commercial monophonic jazz performances.  Pattern Recognition Letters 31(12):1514–1523.
  * [Roberts, Engel, and Eck 2017] Roberts, A.; Engel, J.; and Eck, D.  2017\.  Hierarchical variational autoencoders for music.  In 31st Conference on Neural Information Processing Systems (NIPS 2017) Workshop.
  * [Russ 2004] Russ, M.  2004\.  Sound synthesis and sampling.  Taylor & Francis.
  * [Schoenberg and Stein 1969] Schoenberg, A., and Stein, L.  1969\.  Structural functions of harmony.  Number 478. WW Norton & Company.
  * [Simon and Oore 2017] Simon, I., and Oore, S.  2017\.  Performance rnn: Generating music with expressive timing and dynamics.  https://magenta.tensorflow.org/performance-rnn.
  * [Stamatatos and Widmer 2005] Stamatatos, E., and Widmer, G.  2005\.  Automatic identification of music performers with learning ensembles.  Artificial Intelligence 165(1):37–56.
  * [Thomas et al. 2017] Thomas, V.; Bengio, E.; Fedus, W.; Pondard, J.; Beaudoin, P.; Larochelle, H.; Pineau, J.; Precup, D.; and Bengio, Y.  2017\.  Disentangling the independently controllable factors of variation by interacting with the world.  In Learning Disentangled Representations, NIPS 2017 Workshop.
  * [Verma and Smith 2018] Verma, P., and Smith, J. O.  2018\.  Neural style transfer for audio spectograms.  arXiv preprint arXiv:1801.01589.
  * [Wang and Cook 2003] Wang, G., and Cook, P. R.  2003\.  Chuck: A concurrent, on-the-fly, audio programming language.  In International Computer Music Conference (ICMC-2003).
  * [Widmer and Goebl 2004] Widmer, G., and Goebl, W.  2004\.  Computational models of expressive music performance: The state of the art.  Journal of New Music Research 33(3):203–216.
  * [Wiggins, Müllensiefen, and Pearce 2010] Wiggins, G. A.; Müllensiefen, D.; and Pearce, M. T.  2010\.  On the non-existence of music: Why music theory is a figment of the imagination.  Musicae Scientiae 14(1_suppl):231–255.
  * [Xia 2016] Xia, G.  2016\.  Expressive Collaborative Music Performance via Machine Learning.  Ph.D. Dissertation, Carnegie Mellon University.
  * [Zalkow 2016] Zalkow, F.  2016\.  Musical Style Modification as an Optimization Problem.  Ann Arbor, MI: Michigan Publishing, University of Michigan Library.


# Grasp and Motion Planning for Dexterous Manipulation for the Real Robot
Challenge

Takuma Yoneda Equal contribution Toyota Technological Institute at Chicago
Charles Schaff11footnotemark: 1 Toyota Technological Institute at Chicago
Takahiro Maeda  Toyota Technological Institute  Matthew R. Walter  Toyota
Technological Institute at Chicago

###### Abstract

This report describes our winning submission to the Real Robot Challenge. The
Real Robot Challenge is a three-phase dexterous manipulation competition that
involves manipulating various rectangular objects with the TriFinger Platform.
Our approach combines motion planning with several motion primitives to
manipulate the object. For Phases $1$ and $2$, we additionally learn a
residual policy in simulation that applies corrective actions on top of our
controller. Our approach won first place in Phase $2$ and Phase $3$ of the
competition. We were anonymously known as ‘ardentstork’ on the competition
leaderboard. Videos and our code can be found at https://github.com/ripl-
ttic/real-robot-challenge.

##  1 Real Robot Challenge

The Real Robot Challenge is a manipulation challenge using the TriFinger
Platform developed by Wüthrich et al. [10]. The TriFinger robot consists of
three identical fingers, each with three degrees of freedom, that are
positioned above the robot’s workspace. We are tasked with manipulating
rectangular objects by moving them to various goal poses. The challenge
consists of three phases. In Phase $1$, the task is to manipulate a cube in a
simulated environment. In Phase $2$, the objective is to manipulate a cube
using the real platform. Phase $3$ increases the level of difficulty by
requiring the manipulation of a much smaller cuboid object.

In each phase, tasks are classified into four levels according to the goal
pose of the object. In levels $1-3$, the goal specifies the position of the
object, but not the orientation. In level $1$, the goal position is always on
the ground; in level $2$, it is fixed at the center and $20$ cm above the
ground plane; in level $3$, it is randomly sampled within the robot’s
workspace; and in level $4$, a goal orientation is additionally
specified.111For tasks involving the cuboid object (Phase $3$), it is not
necessary to achieve a desired rotation around the long axis of the cuboid,
due to the difficulty of accurately measuring it. Performance is evaluated in
terms of the speed and accuracy with which the goal is reached. At each
timestep, a reward is provided based on the distance between the current
object pose and the goal pose, in which positional and orientational
differences are weighed differently and added up. The sum of reward over a $2$
minute period is used to evaluate each submission. Figure 1 provides
visualizations for Phase $2$ and Phase $3$.

The challenge organizer has provided an interface that estimates object pose
from 3 camera views and returns the estimated pose as well as sensor readings
of joint angles, velocities and tip force sensors on TriFinger Platform.
Participants are free to use their own algorithms to estimate object pose from
the camera views, however, we used the provided estimation due to time
limitation. Further details regarding the challenge can be found at the
official website.

##  2 Approach

(a) Phase $2$: cube

(b) Phase $3$: cuboid

Figure 1: Screenshots from (left) Phases $2$ and (right) Phase $3$ at level 4.
The three columns show views from cameras evenly distributed above the
workspace as part of the single TriFinger Platform. The top row shows the
views of the physical robot, while the bottom row visualizes a rendering of
the environment reconstructed from proprioceptive (encoders) and exteroceptive
(cameras) sensors. The goal pose is visualized in green.

Achieving the goal pose often requires multiple grasps and manipulations. We
find that many feasible grasps of the object are not suitable for actually
solving the task, especially when matching orientation is required. Therefore,
a successful approach must be able to search over the space of possible
grasps. The small size of the object also requires precise positioning of the
fingertips, which can be challenging in the presence of observation noise.

The core of our approach is a grasp-planning loop in which we sample feasible
grasps until we find one for which a path exists that achieves the desired
object pose. Here, a valid path is a collision-free sequence of object poses
and corresponding joint positions.

In Phase $3$, where object pose estimation tends to be very noisy, the path is
executed by simply tracking the planned joint positions with a PD controller.
This approach does not rely on feedback of the object’s pose, allowing us to
be robust to inaccurate or missed object observations. In Phases $1$ and $2$,
however, where pose estimations are more reliable, we also add feedback
controllers: a force control policy similar to Wüthrich et al. [10] that helps
maintain a stable grasp on the object, and a learned residual policy [7, 1]
that applies corrective position and torque actions to our controller. In all
phases, we combine this approach with a set of motion primitives that grasp
and align the object before attempting to move it to the goal pose. Figure 2
provides an overview of our approach.

###  2.1 Motion Primitives and Object Alignment

Many goal poses cannot be reached with a single grasp. Consequently, we use a
set of motion primitives to first align the object to the goal on the ground.
These include: grasping the object, moving the object to the center of the
workspace, and aligning the orientation of the object with the goal
orientation. These primitives are combined as shown in Figure 2.

Figure 2: A visualization of the structure of our approach. We first align the
pose of the object by moving it to the center of the workspace and align the
object with the goal orientation. We then find a grasp and subsequent motion
plan to move the object to the goal pose. The object is then grasped and the
plan is executed by tracking joint positions with a PD controller.

The _grasping_ motion primitive moves the fingertips in a straight line to a
predefined pregrasp pose near the contact points, then grasps the object. This
helps avoid accidentally hitting the object when approaching it. In Phase $3$,
the smallest sides of the object are about the same size as the fingertips. As
a result, grasping is prone to failure if there is noise in the estimated pose
of the object. To combat this, we first move the fingers to a position that
does not occlude the different views of the object and collect several
observations. We then average and project those observations onto the ground
plane by setting the height of the object’s position and projecting its
orientation into the subspace of yaw-only rotations. We then use the averaged
observations for grasping.

The _move-to-center_ motion primitive first grasps the object, then moves the
fingertips along a straight-line path to the center of the workspace. In the
case where the object is close to the edge of the workspace, inverse
kinematics often fails to provide joint positions on some of the fingers. In
such cases we pin those fingers at the default position, and execute the path
only with available finger(s).

To make it easier to find a grasp and subsequent plan that move the object to
the goal, the _align-yaw_ and _align-pitch_ motion primitives rotate the
object to be close to the goal orientation while remaining on the ground. For
yaw rotations, this corresponds to projecting the goal-orientation onto the
space of yaw-only rotations and rotating the object to the projected
orientation. For pitch rotations, the pitch of the goal pose is rounded to the
nearest $90$ degrees and the object is rotated to that orientation. These
primitive are used only for Level 4 and only the align-pitch primitive is not
used in Phase $3$ because matching pitch does not affect the
reward11footnotemark: 1. In Phases $1$ and $2$, we used a predefined motion to
rotate the object, but in Phase $3$, we found that the small object size often
lead to collisions between the fingers. Thus, in Phase $3$, we use the grasp-
planning loop described below to find a feasible grasp and subsequent plan
that achieve the desired rotation. We then grasp the object, execute the
planned motion, and release the object.

###  2.2 Grasp-planning Loop

Once the object on the ground is reasonably centered and the orientation is
aligned with the goal (only required at Level $4$), the next step is to find a
proper grasp and plan a path to the goal pose. Since a goal pose is often only
reachable with specific grasps, we repeat grasp sampling and path planning
until a feasible path is found. We call this grasp-planning loop.

We first sample grasps. This is done in two ways: by randomly sampling a set
of three contact points on the surface of the object, and by selecting from a
set of predefined contact points. We then generate a grasp by assigning a
fingertip to each contact point. We reject grasps that do not exhibit the
following two properties: the grasp is feasible, meaning the finger tips can
reach the contact points and there are no collisions between the fingers
(excluding the tips) and the object, and the grasp is in force closure,
meaning we can resist any external force applied to the object. The force
closure check is computed by assuming a Coulomb friction model and making a
linear approximation to the friction cone.

Once a grasp has been sampled, we run an RRT [3] in task space (i.e., the pose
of the object) starting from the current object pose and planning to a desired
object pose. We enforce that the grasp is maintained during planning by
rejecting object poses if the corresponding grasp is not feasible. If no
feasible path to the goal is found, we sample another grasp and repeat the
process. To ensure we eventually find a feasible path, we slowly increase the
size of the goal set of the RRT.

###  2.3 Executing the plan

Given a grasp and a path (sequence of joint positions and object poses)
obtained from grasp-planning loop, we then follow the planned joint positions
with a PD controller. During execution, we check to see if the object has been
dropped. If so, we reset and plan again. Notably, the execution of this plan
does not rely on feedback of the object’s pose. This provides robustness to
errors in the provided estimates of the object’s pose, including missed
detections and occasional $180$ degree flips in the estimated orientation,
which happens frequently in Phase $3$. In Phases $1$ and $2$, without a
notable issue on pose estimation, we additionally applied two feedback
controllers: a force control policy and a learned residual policy. These
policies are explained in detail in Section 2.4

In cases in which the RRT generates a plan that does not move the object to
the exact goal pose, we perform Tip Adjustments that subsequently append
corrections to the plan after execution that attempt to move the object to the
goal pose by translating and rotating the fingertip positions accordingly. In
Phase $2$, we also corrected for other errors such as slippage or changes in
the grasp during execution of the plan based on estimated object pose, but we
found the observation noise to be too large for this to work reliably in Phase
$3$.

Phase 1 | Level $1$ | Level $2$ | Level $3$ | Level $4$
---|---|---|---|---
Force Control (FC) | $-138\pm 143$ | $-532\pm 402$ | $-458\pm 477$ | $-1421\pm 345$
FC + Residual Learning (RL) | $\mathbf{-74\pm 29}$ | $\mathbf{-123\pm 69}\hphantom{0}$ | $\mathbf{-122\pm 54\hphantom{0}}$ | $-1527\pm 425$
FC + Motion Planning (MP) | $-166\pm 178$ | $-177\pm 51\hphantom{0}$ | $-226\pm 257$ | $-1127\pm 509$
FC + MP + RL | $-156\pm 82\hphantom{0}$ | $-171\pm 51\hphantom{0}$ | $-170\pm 83\hphantom{0}$ | $\hphantom{0}\mathbf{-999\pm 387}$
Phase 2 |  |  |  |
FC | $-10051\pm 5493$ | $-28817\pm 4773\hphantom{0}$ | $-28414\pm 8957\hphantom{0}$ | $-45320\pm 5654\hphantom{0}$
FC + RL | $-11192\pm 6768$ | $-36405\pm 7667\hphantom{0}$ | $-25324\pm 11920$ | $-45370\pm 13388$
FC + MP | $\hphantom{0}-7973\pm 7898\hphantom{0}$ | $-13559\pm 9597\hphantom{0}$ | $-13842\pm 12688$ | $-23517\pm 10680$
FC + MP + Tip Adjust. (TP) | $\hphantom{0}\mathbf{-2673\pm 1166}$ | $\hphantom{0}-8115\pm 8443\hphantom{-}$ | $\hphantom{0}-4830\pm 5005\hphantom{-}$ | $-32572\pm 15701$
FC + MP + RL | $\hphantom{0}-8018\pm 8011\hphantom{0}$ | $\hphantom{0}-7167\pm 2170\hphantom{-}$ | $\hphantom{0}-8812\pm 2996\hphantom{-}$ | $\mathbf{-20448\pm 6154}\hphantom{0}$
FC + MP + TP + RL | $\hphantom{0}-3330\pm 4331\hphantom{0}$ | $\mathbf{\hphantom{0}-2877\pm 204\hphantom{-0}}$ | $\mathbf{\hphantom{0}-3893\pm 2453\hphantom{-}}$ | $-22301\pm 11443$
Phase 3 |  |  |  |
MP + TP | $-7944\pm 4214$ | $\hphantom{0}-4039\pm 446\hphantom{-0}$ | $-9896\pm 10066$ | $-24755\pm 10548$
Table 1: Mean and standard deviation of rewards for different algorithms. We
report the mean and standard deviation of rewards across $10$ episodes for
each difficulty level. It should be noted that the specified frequency of
control is largely different in Phase 1, and the magnitude of the reward
differs from Phases $2$ and 3 even though the evaluation metric is identical.

###  2.4 Feedback Controllers used in Phases $1$ and $2$

In Phases $1$ and $2$, object pose estimation is reasonably accurate (Phase
$1$ presents no estimation errors since it is contained in a simulated
environment). Thus, we applied the following feedback control policies on top
of position control: force control and residual policy. Feedback controls
enable the robot to exert auxiliary torque to maintain a stable grasp.

####  2.4.1 Force Control

Our force control pipeline is similar to the one described by Wüthrich et al.
[10]. We use a PD controller to turn object pose errors into a wrench applied
to the object’s center of mass. Given a grasp of the object and its resulting
contact points, we then solve for the tip forces necessary to achieve that
wrench while maintaining friction cone constraints (see Murray et al. [4] for
details). With tip forces, we combine Jacobian transpose control and gravity
compensation via inverse dynamics to determine the joint torques. Force
control was not used in Phase $3$ due to noise in the provided pose
estimation.

####  2.4.2 Residual Policy Learning

Residual policy learning [7, 1] provides a simple way to improve a controller
by learning corrective actions that are added to the output of the controller.
Additionally, learning corrections to a base controller can drastically
simplify the reinforcement learning problem by removing the need for
significant exploration outside of the neighborhood around the base
controller. It can also result in a policy that outperforms both the base
controller policy and a reinforcement learning policy trained from scratch.

We use Proximal Policy Optimization [6] (PPO) to learn corrections to our
controller in simulation. We fix the residual policy to zero for the first $1$
M timesteps and only train the value function. This allows us to get a good
estimate of the value of the base controller before we start learning.
Additionally, we found that the initialization of the policy network was
extremely important. Initializing the network to output action distributions
with zero mean and small standard deviation greatly improved training as the
combined controller stays close to the base controller early in training. Our
reward function consists of three terms: the distance to goal, a regularizer
on joint torques and velocities, and a penalty for the tips slipping on the
object’s surface. The last term encourages a firm grasp on the object, which
is helpful for following the planned motion of the baseline controller.

While we planned to train models with domain randomization and real data, we
did not have enough time. Instead, in Phase $2$, we used the policies trained
during Phase $1$ and applied them directly to the real system. Surprisingly,
despite substantial domain shift and changes to the base policies, we find the
Phase $1$ policies trained with our baseline motion planning controller were
able to improve performance on the real robot. In Phase $3$, we did not
attempt any residual control. We expect further improvements by applying sim-
to-real methods [8, 5, 9] or fine-tuning with real data, and we leave this as
a future work.

##  3 Discussion

Table 1 presents the performance of our approach across all phases along with
several ablations. We find that our approach is able to consistently solve the
tasks, is robust to observation noise, and can recover from failures (i.e.,
dropping the object). However, there is still room for improvement. We often
find that executing our planned motions only takes the object near the goal
pose. This is because the grasp was invalid at the exact goal pose, planning
failed or took too long, or the object slipped while executing the plan.
Robustly correcting for these errors could improve our performance
considerably. Additionally we accumulate a lot of negative reward while the
RRT is planning, and failures that trigger another round of planning can lead
to a large variance in rewards. Using a Probabilistic Roadmap (PRM) [2] would
allow us to precompute many paths and thereby speed up planning significantly.

Due to time constraints, we were unable to try everything we had planned for
Phase $3$. Specifically, we implemented a framework for residual policy
learning [1, 7] with domain randomization [9] to help transfer from simulation
to the real robot, but did not have time to train and test the learned
policies with a finalized controller. In Phases $1$ and $2$, our policies were
able to learn useful corrections to our controllers, and we believe such an
approach would be beneficial in Phase $3$ as well.

## Acknowledgement

We thank the organizers for all the effort they put into creating the
TriFinger platform and hosting this competition. We are excited to use this
platform to extend this work or for other research projects related to
dexterous manipulation.

## References

  * Johannink et al. [2019] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine.  Residual reinforcement learning for robot control.  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , May 2019.
  * Kavraki et al. [1996] L. E. Kavraki, P. Svestka, J. . Latombe, and M. H. Overmars.  Probabilistic roadmaps for path planning in high-dimensional configuration spaces.  _IEEE Transactions on Robotics and Automation_ , 12(4):566–580, 1996.  doi: 10.1109/70.508439.
  * LaValle [1998] Steven M LaValle.  Rapidly-exploring random trees: A new tool for path planning.  1998\.
  * Murray et al. [1994] Richard M Murray, Zexiang Li, S Shankar Sastry, and S Shankara Sastry.  _A mathematical introduction to robotic manipulation_.  CRC press, 1994.
  * Peng et al. [2018] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel.  Sim-to-real transfer of robotic control with dynamics randomization.  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2018.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Silver et al. [2018] Tom Silver, Kelsey R. Allen, Joshua B. Tenenbaum, and Leslie Pack Kaelbling.  Residual policy learning.  _arXiv preprint arXiv:1812.06298_ , 2018.
  * Tan et al. [2018] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke.  Sim-to-real: Learning agile locomotion for quadruped robots.  In _Proceedings of Robotics: Science and Systems (RSS)_ , July 2018\.
  * Tobin et al. [2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In _Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2017.
  * Wüthrich et al. [2020] Manuel Wüthrich, Felix Widmaier, Felix Grimminger, Joel Akpo, Shruti Joshi, Vaibhav Agrawal, Bilal Hammoud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, et al.  Trifinger: An open-source robot for learning dexterity.  _arXiv preprint arXiv:2008.03596_ , 2020.