# RFUniverse: A Physics-based Action-centric Interactive Environment for
Everyday Household Tasks

Haoyuan Fu∗1, Wenqiang Xu∗1, Han Xue1, Huinan Yang2, Ruolin Ye1, Yongxi
Huang1, Zhendong Xue1,
Yanfeng Wang1 and Cewu Lu1 Working in progress.*These two authors contribute
equally to this work.1{simon-fuhaoyuan, vinjohn, xiaoxiaoxh, cathyye2000,
huangyongxi, 707346129, wangyanfeng<EMAIL_ADDRESS>Cewu Lu is the
corresponding author, member of Qing Yuan Research Institute and MoE Key Lab
of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University,
<EMAIL_ADDRESS>Shanghai MCUSTAR Electronic Technology Co.,
Ltd, Shanghai, China.

###### Abstract

Household environments are important testbeds for embodied AI research. Many
simulation environments have been proposed to develop learning models for
solving everyday household tasks. However, though interactions are paid
attention to in most environments, the actions operating on the objects are
not well supported concerning action types, object types, and interaction
physics. To bridge the gap at the action level, we propose a novel physics-
based action-centric environment, RFUniverse, for robot learning of everyday
household tasks. RFUniverse supports interactions among 87 atomic actions and
8 basic object types in a visually and physically plausible way. To
demonstrate the usability of the simulation environment, we perform learning
algorithms on various types of tasks, namely fruit-picking, cloth-folding and
sponge-wiping for manipulation, stair-chasing for locomotion, room-cleaning
for multi-agent collaboration, milk-pouring for task and motion planning, and
bimanual-lifting for behavior cloning from VR interface. Client-side Python
APIs, learning codes, models, and the database will be released. Demo video
for atomic actions can be found in supplementary materials:
https://sites.google.com/view/rfuniverse

##  I Introduction

It is tedious for human beings to deal with everyday household tasks, thus an
intelligent agent which can substitute human beings to save effort will
benefit human society. Such an intelligent agent should be able to handle a
large set of household tasks and can be adaptive to unseen environments. It
seems household tasks are endless in daily life, but they can be regarded as
combinations of finite atomic actions interacting with different kinds of
objects. In this way, action is viewed as a tuple $\mathcal{A}=\langle
action,object,constraints\rangle$, and a task is thus an action sequence
$\mathcal{T}=\\{\mathcal{A}_{1},...,\mathcal{A}_{n}\\}$. As learning from
real-world domestic environments is cost-prohibitive, many researchers
proposed simulation environments for household tasks [1, 2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13]. Despite that all of them provide “interactive”
functionalities, the supports for the agent actions are limited in three
aspects, namely action types, object types, and interaction physics.

In this work, we first determine what action and object types are important
for daily household tasks. We summarize from a previous proposed human
activity knowledge base in household environment [4]. As a result, we extract
14 navigation-related (e.g. turn_left, look_at), 3 locomotion-related (i.e.
walk, run, climb) and 70 manipulation-related atomic actions (i.e. 33 single-
hand, 23 bimanual, 14 tool-based), which concerns 5 basic object types (i.e.
rigid, articulated, flexible, transparent, and tearable). Besides, 3
additional object types namely fluid, gas, fire can be used to represent the
visual object states (e.g. steams from boiled water in Fig. 2). Based on the
actions, we present a novel Unity-based simulation environments named
RFUniverse, which organizes different physics engines to enable the physics-
based interaction of all the atomic actions upon different object types.

From the action side, previous environments generally built around simple
atomic actions like navigation [2, 6], grasping, transporting, and placing (or
dropping) [1, 7, 8, 10]. However, for some environments [1, 14], even these
simple actions are usually implemented in an abstract, not physics-based way.
For example, an abstract grasping is performed by attaching the target object
like suctioning when approaching it within a certain distance. As for complex
actions like cutting are usually implemented in the “pre-defined” manner [9,
12]. For example, iGibson 2 [9] prepares sliced meshes in advance, so that the
object will be cut off no matter where the knife is contact. VRKitchen [12]
defines the part to be removed by heuristic rules. Nevertheless, the lack of
supporting physics-based actions will limit the task ranges, and constrain the
motion policy learned for complex actions.

From the object side, as mentioned earlier, there are at least 8 basic types
of objects that are relevant to the household tasks. By “basic”, we mean these
properties can be combined, like a plastic bag which is flexible, tearable and
transparent. Considering interaction with the basic object types, the previous
simulators usually support the rigid or articulated [1, 9] objects as the
actionable objects. Though some environments can simulate dynamics of flexible
objects [15], but the agent cannot operate on the flexible objects due to the
limitation of deployed physics engines. The tearable and transparent
properties are individually studied by different researchers [16, 17], but
they have never been implemented to interact with the virtual embodied agent
in a physically plausible way.

TABLE I:

Environment |  Obj. Types | # Actions |  Physics Engine | Colli. Inter. | Integrated Planner | VR
---|---|---|---|---|---|---
HoME [3] |  R | 4 + 3 |  Bullet | ✓ | / | /
AI2THOR series [1, 11, 10] |  R,A,Fu | 10 + 10 |  PhysX | / | / | /
Habitat [6] |  R | 3 + 0 |  Bullet | / | / | /
SAPIEN [13] |  R,A | / |  PhysX | ✓ | ✓ | /
VirtualHome [4] |  R,A,Fu | 9 + 7 |  PhysX | / | / | /
VRKitchen [12] |  R,A,Fu | 0 + 18 |  PhysX | / | / | ✓
ThreeDWorld [15, 14] |  R,A,F,Fu | 4 + 2 |  PhysX, Flex | / | ✓ | /
Gibson series [2, 7, 8, 9] |  R,A,F,Fu | 4 + 4 |  Bullet | $\checkmark^{*}$ | ✓ | ✓
RFUniverse |  R,A,F,Fu,Te,Tr,G,Fr | 17 + 70 |  PhysX, Flex, Cloth Dynamics, Obi, [MuJoCo, Bullet, SOFA] | ✓ | ✓ | ✓

Comparison with different simulation environments. As some environments are
continuously developed, we summarize them into a series and the features are
reported based on the latest version.
Obj. Type: R - Rigid, A - Articulated, F - Flexible, Fu - Fluid, Te - Tearble,
Tr - Transparent, G - Gas, Fr - Fire.
# Actions: $n_{1}+n_{2}$, where $n_{1}$ means atomic actions for
navigation/locomotion, and $n_{2}$ means actions for manipulation.
Physics Engine: engines in $[\cdot]$ means these engines are optional with
APIs but not used in the benchmark experiments.
Colli. Inter.: Whether the agent-object interaction are collision-based. Some
environment can support the collision-based interaction, but they choose not
to. $\checkmark^{*}$ means not all actions are collision-based, for example
cutting in iGibson2[9].

Aside from the key features regarding physics-based interaction, RFUniverse
also provides full functionalities to support simulation and learning of
robots doing household tasks: Python APIs, photo-realistic rendering, multi-
modal sensing, synthetic data generation for perception models, and gym-like
wrapper for reinforcement learning models. A powerful ROS-free motion planner,
RFMove [18] is natively integrated to plan for full-body movements. Moreover,
we provide a VR interface to extend the interactive ability from the real to
the simulated world.

We evaluate the usability of RFUniverse with various kinds of tasks related to
household, namely fruit-picking for rigid object manipulation, cloth-folding
for 2D flexible object manipulation, sponge-wiping for 3D flexible object
manipulation, stair-chasing for locomotion, room-cleaning for multi-agent
collaboration, milk-pouring for task and motion planning, bimanual-lifting via
VR interface for behavior cloning.

We summarize our contributions as follows:

  * •

We construct a simulation environment RFUniverse, which can support physics-
based interactions for all the extracted action and object types. It also
provides a client-server communication framework based on gRPC, which can
enable full functionality control of Unity with Python language.

  * •

We benchmark 7 kinds of learning tasks on RFUniverse, which cover different
aspects of the household tasks.

##  II RELATED WORKS

Physics-based Object Simulation Physics-based action requires the reaction of
manipulated objects should also be physically plausible. We roughly categorize
objects concerned in household tasks as rigid, articulated, flexible,
tearable, transparent, fluid, gas, and fire.

For the latter three object types, as discussed earlier, only the visual
effects are concerned. They can be simulated by particle system implemented by
Flex [19].

For rigid and articulated objects, they can generally be properly simulated by
common physics engines like Dart [20], ODE [21], Bullet [22], MuJoCo [23] and
PhysX [24]. MuJoCo is arguably the best for articulation [25], but it is
close-sourced. Though it provides an integration to Unity, we find the default
PhysX can also do well.

Simulation on flexible objects is more tricky, as their deformation mechanisms
are very different according to the materials [26]. Thus though many common
physics engines [23, 22] claim to support 1D (rope), 2D (cloth), and 3D (soft
body) flexible object deformation, they either adopt simple models [27] to
simulate or provide limited controlling parameters. Sofa framework [28] can
simulate the deformation with accurate modeling for different materials, but
the Unity integration plugin is premium. Aside from simulating the flexible
object with one physics engine, we can also deal with them separately. Some
physics engines have specialties in simulating cloth [29, 30] and some can
simulate rope and soft body [31, 19].

The difficulty in simulating the tearable object is the fracture modeling
[16]. Since real-time fracture modeling and rendering is challenging, we model
it with simple topology modification as described in [28].

Recently, MPM-based approaches [32, 33, 16] shed a light on simulating
different object types within the same framework. Though MPM-based methods can
provide visually plausible effects for the interactions, the force applied on
the object is hard to be obtained accurately, which may limit the applications
on force-concerned tasks.

Transparent objects are objects with unique optical properties, which can be
handled by scripting shaders in different kinds of rendering pipelines in
Unity.

Simulated Environments for Household Tasks In comparison with typical
reinforcement learning environments [34, 35, 36, 37], the household-oriented
simulated environments feature navigation in visually plausible scenes and
interaction with realistic object and scene geometry.

An important line of household environments is based on AI2THOR [1]. It is
developed upon Unity and provides Python APIs to control the asset loading and
agent behaviors. In AI2THOR, the objects can only be grasped abstractly. Such
property is inherited to the following RoboTHOR [11], ManipulaTHOR [10] or the
environments [38, 5] proposed for the language-vision researches, like
language-guided navigation and instruction following.

For those non-AI2THOR-based environments, VirtualHome [4] and ThreeDWorld [15]
also adopt Unity as the rendering system. The former takes a study on human
activities in domestic environments, resulting in a knowledge base. The key
difference between their knowledge base and ours is whether the task
describing “what I would do in a given scene” or “what I would ask a robot to
do in a given scene”. Thus some activities in their knowledge base are not
reasonable for a robot to operate, like watch TV, play game. ThreeDWorld is
more interested in the multi-modal physics simulation. It later provides an
interface for transporting object around the room [14], but the grasping is
implemented as suctioning to avoid collision-based contact.

On the other hand, instead of adopting mature rendering and physics engine
management system like Unity, some environments decide to write their own.
Habitat [6] and Gibson [2] support only navigation-related tasks. Later, the
Gibson series [7, 8, 9] have been evolved from a pure navigation environment
to support simple manipulation and object states. The latest iGibson2.0
supports cutting aside from the typical grasping, transporting and dropping,
but in a pre-defined way.

The functionality comparison between RFUniverse and previous environments can
be referred to Table I.

##  III RFUniverse Simulation Environment

###  III-A Python Interface & Communication Framework

RFUniverse adopts a client/server framework to communicate the client-side
Python-based program and the Unity-side C#-based server, as shown in Fig. 1.
The communication is based on gRPC, which can handle different programming
languages, OS platforms, and networks. We adopt Python as the client language
because of its simplicity and powerful ecosystem, especially with learning
frameworks [39, 40]. The user can control or obtain any information through
the Python interface. For example, to enable physics-based action for the
benchmark tasks, we can complete all the procedures from the Python interface:

  * •

Establish a connection to the Unity server;

  * •

Setup a scene layout based on the sampling heuristics or configuration files;

  * •

Configure the rendering options, so that can adjust the computational
resources for rendering or generate synthetic datasets;

  * •

Configure the physics options for physics engines;

  * •

(optional) Collecting Data through VR Interface;

  * •

Read the states, observations from the given camera(s) in the simulation;

  * •

Control the agent with agent id, so that enables multi-agent feature.

Figure 1: The framework of RFUniverse.

###  III-B Task Layout Configuration

Scene Assets We adopt the scene assets from the Gibson dataset, as it is
scanned from the real world. We make some fixes upon these scenes like
installing the windows, doors, and mirrors. But we leave the scanning error
unfixed as we think it is part of the scanning process.

Object Assets To accompany the tasks, the objects are selected from 3D
warehouse111https://3dwarehouse.sketchup.com/. For transparent objects, we
attach the glass-like material with a customized shader to the original object
model. The transparent effects can be presented in different rendering
pipelines (i.e. SRP, URP, HDRP) in Unity. For each object, we can query the
object states (e.g. object pose, applied force) and attributes (e.g. mass,
inertial, index of refraction) during interaction through Python API.

Agent Assets Different household tasks may require different workspace and
dexterity for the agent. While currently, to the best of our knowledge, there
exists no single robot agent that can handle all kinds of household tasks.
Thus, we support various robot agents. In the benchmark experiments, for
manipulation tasks which can be accomplished by a single arm with a fixed
base, the agent is Franka Emika Panda; for the multi-agent task, we select
Philips FC 8800 Robotic Floor Cleaner for room-cleaning; for the locomotion
task which requires leg like stair-chasing, we adopt a human boy avatar; for
the task and motion planning, and behavior cloning of bimanual grasping, we
propose a dual-arm mobile manipulator, Tobor. The design and the hardware
specification of the Tobor robot can be referred to in supplementary
materials. Besides, other common robots like UR5, Fetch, shadow hands can be
trivially supported.

Layout Generation The layout can be generated in two ways, through an
interactive GUI or sampling heuristics.

  * •

Through an interactive GUI like blender, Unity, or any other proper 3D
software, we can output the name and position of scene/object to a
configuration file. Then we can reconstruct the scene configuration to the
Unity server through Python API.

  * •

On the other hand, we can also specify different sampling heuristics on
objects selection, initial positions, fixed or falling from Python API.

###  III-C Rendering System

In this work, we are particularly interested in simulating convincing physics-
based visual effects which can reflect the object states. As mentioned
earlier, 4 kinds of objects are related to visual effects, namely the fluid,
gas, fire, and transparent objects. Thanks to the highly customizable
rendering pipeline of Unity, we can simulate convincing visual effects as
shown in Fig. 2. The trigger conditions for these visual effects are labeled
in the object assets, for example, when the water is contained in a pot
(container), and the pot is being heated by fire, the boiling and steam
effects will be simulated.

Besides, the lighting conditions are also important, especially when they can
change the appearance of the RGB snapshots, which is critical to the synthetic
dataset generation and domain randomization. These two functionalities are
also provided in RFUniverse, but they are not used in the benchmark tasks,
thus the descriptions are left in the supplementary materials.

Figure 2: Advanced rendering effects: (a) ambient lighting in the dawn; (b)
ambient lighting in the noon; (c) lighting in the night with wall lamp on; (d)
steams from boiled water when heated by fire; (e) reflection effect of
mirrors; (f) refraction effect of transparent objects. Details can be better
viewed by zooming in.

###  III-D Physics System

Our key motivation in this work is to support atomic actions in a physics-
based manner. To achieve this, the underlying physics engine should support
plausible interaction effects for different object types. However, existing
physics engines are built for different purposes, not a single one aims for
full physics simulation. A workaround is to simulate different interaction
behaviors with different physics engines. Thanks to the strong ecosystem of
Unity, which can enable us to connect with different backend physics engines
for different object types. In practice, we choose PhysX [24] for rigid and
articulated objects, Obi [31] for rope and soft body, Cloth Dynamics [30] for
cloth simulation, and Flex [19] for fluid, gas, fire effect simulation. The
tearing effect can be achieved in real-time and the separated meshes can be
calculated on the fly with Mesh Slicer [41]. Besides, we also provide wrappers
for MuJoCo 1.55 [23], Bullet [22] and SOFA [28].

###  III-E Gym-like Wrapper for Reinforcement Learning

As for reinforcement learning algorithms, we provide a standard gym-like
wrapper for different kinds of environments, we will later show how to run the
manipulation, locomotion with standard reinforcement learning library, stable-
baselines3 [42] in Sec. IV.

###  III-F Task and Motion Planner

We integrate a TAMP framework based on PDDLStream [43]. If not specified, the
action stream samplers are implemented by RFMove [18], a motion planning
framework without the requirement of ROS [44] installation or models learned
from reinforcement learning. For navigation, Unity‘s NavMesh framework is
adopted.

###  III-G VR Interface

The VR interface provides the opportunity for the user to directly interact
with the simulated environment so that we can collect demonstrations in the
virtual environment. Different VR devices can be connected to RFUniverse
through SteamVR, we take the HTC Vive headset with Noitom Hi5 glove as an
example, as shown in Fig. 3.

Figure 3: The VR interface with HTC Vive and Noitom Glove.

##  IV Experiments

In this section, we will perform 5 benchmark tasks, including 3 manipulation
tasks (i.e fruit-picking for rigid, cloth-folding for 2D flexible, and sponge-
wiping for 3D flexible object manipulation), 1 locomotion task (i.e. stair-
chasing), and 1 multi-agent task (i.e. room-cleaning). The other 2 tasks for
TAMP (i.e. milk-pouring) and behaviour cloning from VR demonstration (i.e.
bimanual-lifting) are discussed in supplementary materials due to page limit.

###  IV-A Machine Specification

During training, all experiments are benchmarked on Ubuntu 20.04 platform with
two Intel(R) Xeon(R) Platinum 8276 CPU and 1 NVIDIA Geforce RTX 3080 graphic
card. Due to different calculating burden in various tasks, we dynamically
adjust the action time step. We set the action time step for rigid object
interaction $t_{a}^{r}=\frac{1}{30}s$ and for deformable object interaction
$t_{a}^{d}=\frac{1}{15}s$. We raise time scale to accelerate training process
and reach physical time step $t_{s}^{r}=\frac{1}{60}s$ and
$t_{s}^{d}=\frac{1}{30}s$.

Figure 4: Experimental results for manipulation tasks, locomotion task and
multi-agent collaboration task. Upper: The snapshots of task scenes. Bottom:
The success rate-step relationship for manipulation tasks, and reward-step
relationship for locomotion and navigation tasks are displayed.

###  IV-B Reinforcement Learning of Manipulation Tasks

####  IV-B1 Task Description

The fruit-picking is a standard rigid object manipulation task. The agent is
trained to pick up rigid fruits on the table and place them into a basket.
During training, we randomly place one fruit on the table and select a
position for the basket in each episode. The task is judged to be finished
when the given fruit is placed into the basket.

For cloth-folding, we place a random-state cloth into the workspace of the
agent. The task is to fold the cloth towards a given configuration. The agent
must grasp a corner of the cloth and place it somewhere proper so that the
cloth can be folded. To better benchmark this task, in each episode, we
randomly select two corners and treat the task as a success if two chosen
corners are placed within $5cm$ distance threshold.

For sponge-wiping, we place a sponge on a table and train the agent using this
sponge to wipe the table with grasp_push. In each episode, the sponge will be
placed at a random location, and a target position for the sponge is also
selected. The final success decision is made when the sponge is moved to the
target position within a $5cm$ tolerance. We track the position of the sponge
and determine a failure if it is moved above the table.

####  IV-B2 Implementation

In all manipulation tasks mentioned above, we adopt Hindsight Experience
Replay (HER) [45] algorithm to accelerate sampling efficiency with off-policy
Truncated Quantile Critics (TQC) [46] algorithm as the base model. We set 50
time steps for each episode. In all three environments, we wrap the task into
the goal-based condition and select sparse reward function, which is aimed to
train the agent to finish the task as soon as possible. The observation
includes the Panda gripper position, velocity, and open width, together with
the object’s current position, orientation, velocity, angular velocity, and
the target position. We train this agent for $1000k$ time steps ($20k$
episodes). Figure 4 (a)(b)(c) depict the relationship between manipulation
task success rate and time steps during training.

###  IV-C Locomotion Task

####  IV-C1 Task Description

To demonstrate the use of RFUniverse for locomotion tasks, we train an
articulated human-like agent to chase the toy on the stairs. The task is
denoted as Stair-chasing. The agent is copied from ML-Agents [40] $Walker$
environment which has 16 DoFs and walks based on the friction between his feet
and ground. In each episode, the agent is initialized at a fixed position, but
facing a random orientation, with a target toy randomly fallen on the stairs.
The agent needs to balance its body under physics to avoid any part (except
feet) colliding with ground or stairs since this collision will return a
reward of $-1$ and end this episode. If the agent touches the target toy, it
will get a reward of $1$. To better organize the agent’s movement, we set a
target velocity vector $\mathbf{v^{*}}$ where $||\mathbf{v^{*}}||$ is fixed
and the agent will be trained to achieve this velocity. In each time step, if
the agent keeps its balance without falling, it will get an extra reward $r$
combining:

  * •

The distance between actual velocity vector $\mathbf{v}$ and $\mathbf{v^{*}}$.

  * •

The difference between heading orientation unit vector $\mathbf{o}$ and target
orientation unit vector $\mathbf{o^{*}}$.

The reward function is

$r=\left[1-\left(\frac{clamp(||\mathbf{v^{*}}-\mathbf{v}||,0,||\mathbf{v^{*}}||)}{||\mathbf{v^{*}}||}\right)^{2}\right]^{2}\times\frac{\mathbf{o}^{*}\cdot\mathbf{o}+1}{2}$

####  IV-C2 Implementation

We use Soft Actor-Critic (SAC) [47] algorithm to train the agent for $1.5e7$
time steps with a maximum episode length setting to $5000$ in this task. It
should be noticed that we only provide observation of the agent’s position,
velocity, and orientation relative to the target toy as well as the joint
parameters of the agent. Based on the Markov property of RL, we make the
world’s states blind to agents, so that agents will not only learn how to walk
but also be trained with self-adaption from hitting stairs. The relationship
between training episodic reward and time steps is shown in figure 4(d).

###  IV-D Multi-agent Collaboration Task

####  IV-D1 Task Description

To verify the multi-agent support of RFUniverse, we setup a benchmark room-
cleaning which simulates the working process of sweeping robot in household.
Three identical sweeping robots are placed into a room, which is divided into
$8\times 8$ grids. If a sweeping robot’s center of mass passes a grid, the
area inside that grid is regarded as being cleaned with its color turning from
grey to original color of floor. The agents’ task is to clean as many grids as
possible in given $100$ time steps, but they should avoid colliding with each
other. Since this is a multi-agent task, we take each agent’s position and
velocity as shared observation. In each time step, each agent will receive a
$2$-d vector as representation of force in $X$-axis and $Y$-axis. As for the
reward, if there’s collision among agents, an episode will end at once and
return a reward of $-50$. If there’s no collision, reward will be the sum of
agents’ average velocity and the number of grids been cleaned.

####  IV-D2 Implementation

We use SAC algorithm to train for $1e6$ time steps with maximum episodic
length as $100$ time steps. The relationship between episodic reward and time
steps is shown in figure 4(e).

##  V Conclusions and Future Works

In this paper, we present a physics-based action-centric simulation
environment, RFUniverse, for household tasks. RFUniverse supports 87 atomic
actions and 8 object types which are summarized from [4]. It has been
evaluated for different kinds of manipulation, locomotion and navigation
tasks. In the future, with RFUniverse as the foundational platform, we are
interested in integrating multi-modal perception into the action models to
further extend the adaptive ability of robots.

## References

  * [1] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, “AI2-THOR: An Interactive 3D Environment for Visual AI,” _arXiv_ , 2017.
  * [2] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson env: Real-world perception for embodied agents,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9068–9079.
  * [3] S. Brodeur, E. Perez, A. Anand, F. Golemo, L. Celotti, F. Strub, J. Rouat, H. Larochelle, and A. Courville, “HoME: a Household Multimodal Environment,” in _NIPS 2017’s Visually-Grounded Interaction and Language Workshop_ , Long Beach, United States, Dec. 2017. [Online]. Available: https://hal.inria.fr/hal-01653037
  * [4] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba, “Virtualhome: Simulating household activities via programs,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8494–8502.
  * [5] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, “ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020. [Online]. Available: https://arxiv.org/abs/1912.01734
  * [6] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat: A Platform for Embodied AI Research,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * [7] F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. E. Tchapmi, A. Toshev, R. Martín-Martín, and S. Savarese, “Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 713–720, 2020\.
  * [8] B. Shen, F. Xia, C. Li, R. Martın-Martın, L. Fan, G. Wang, S. Buch, C. D’Arpino, S. Srivastava, L. P. Tchapmi, K. Vainio, L. Fei-Fei, and S. Savarese, “igibson, a simulation environment for interactive tasks in large realistic scenes,” _arXiv preprint_ , 2020.
  * [9] C. Li, F. Xia, R. Martín-Martín, M. Lingelbach, S. Srivastava, B. Shen, K. Vainio, C. Gokmen, G. Dharan, T. Jain, A. Kurenkov, C. K. Liu, H. Gweon, J. Wu, L. Fei-Fei, and S. Savarese, “igibson 2.0: Object-centric simulation for robot learning of everyday household tasks,” 2021.
  * [10] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mottaghi, “ManipulaTHOR: A Framework for Visual Object Manipulation,” in _CVPR_ , 2021.
  * [11] M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, R. Mottaghi, J. Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, L. Weihs, M. Yatskar, and A. Farhadi, “RoboTHOR: An Open Simulation-to-Real Embodied AI Platform,” in _CVPR_ , 2020.
  * [12] X. Gao, R. Gong, T. Shu, X. Xie, S. Wang, and S. Zhu, “Vrkitchen: an interactive 3d virtual environment for task-oriented learning,” _arXiv_ , vol. abs/1903.05757, 2019.
  * [13] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su, “SAPIEN: A simulated part-based interactive environment,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [14] C. Gan, S. Zhou, J. Schwartz, S. Alter, A. Bhandwaldar, D. Gutfreund, D. L. K. Yamins, J. J. DiCarlo, J. McDermott, A. Torralba, and J. B. Tenenbaum, “The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied ai,” 2021.
  * [15] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. D. Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, K. Kim, E. Wang, D. Mrowca, M. Lingelbach, A. Curtis, K. Feigelis, D. M. Bear, D. Gutfreund, D. Cox, J. J. DiCarlo, J. McDermott, J. B. Tenenbaum, and D. L. K. Yamins, “Threedworld: A platform for interactive multi-modal physical simulation,” 2020\.
  * [16] J. Wolper, Y. Chen, M. Li, Y. Fang, Z. Qu, J. Lu, M. Cheng, and C. Jiang, “Anisompm: Animating anisotropic damage mechanics,” _ACM Trans. Graph._ , vol. 39, no. 4, 2020.
  * [17] S. Sajjan, M. Moore, M. Pan, G. Nagaraja, J. Lee, A. Zeng, and S. Song, “Clear grasp: 3d shape estimation of transparent objects for manipulation,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 3634–3642.
  * [18] W. X. Yongxi Huang, Danqing Li, “Rfmove.” [Online]. Available: https://github.com/mvig-robotflow/rfmove
  * [19] J. Liang, V. Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox, “Gpu-accelerated robotic simulation for distributed reinforcement learning,” 2018.
  * [20] J. Lee, M. X. Grey, S. Ha, T. Kunz, S. Jain, Y. Ye, S. S. Srinivasa, M. Stilman, and C. K. Liu, “Dart: Dynamic animation and robotics toolkit,” _Journal of Open Source Software_ , vol. 3, no. 22, p. 500, 2018. [Online]. Available: https://doi.org/10.21105/joss.00500
  * [21] R. Smith, “Open dynamics engine,” 2008, http://www.ode.org/. [Online]. Available: http://www.ode.org/
  * [22] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021.
  * [23] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on_. IEEE, 2012, pp. 5026–5033. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/6386109/
  * [24] NVIDIAGameWorks, “Nvidiagameworks/physx: Nvidia physx sdk.” [Online]. Available: https://github.com/NVIDIAGameWorks/PhysX
  * [25] T. Erez, Y. Tassa, and E. Todorov, “Simulation tools for model-based robotics: Comparison of bullet, havok, mujoco, ode and physx,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , 2015, pp. 4397–4404.
  * [26] E. Sifakis and J. Barbic, “Fem simulation of 3d deformable solids: A practitioner’s guide to theory, discretization and model reduction,” in _ACM SIGGRAPH 2012 Courses_ , ser. SIGGRAPH ’12. New York, NY, USA: Association for Computing Machinery, 2012\. [Online]. Available: https://doi.org/10.1145/2343483.2343501
  * [27] D. Baraff and A. Witkin, “Large steps in cloth simulation,” in _Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques_ , ser. SIGGRAPH ’98. New York, NY, USA: Association for Computing Machinery, 1998, p. 43–54. [Online]. Available: https://doi.org/10.1145/280814.280821
  * [28] F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, and S. Cotin, “SOFA: A Multi-Model Framework for Interactive Physical Simulation,” in _Soft Tissue Biomechanical Modeling for Computer Assisted Surgery_ , ser. Studies in Mechanobiology, Tissue Engineering and Biomaterials, Y. Payan, Ed. Springer, June 2012, vol. 11, pp. 283–321. [Online]. Available: https://hal.inria.fr/hal-00681539
  * [29] NVIDIAGameWorks, “Nvidiagameworks/nvcloth.” [Online]. Available: https://github.com/NVIDIAGameWorks/NvCloth
  * [30] UnityStore. [Online]. Available: https://assetstore.unity.com/packages/tools/physics/cloth-dynamics-194408
  * [31] V. M. Studio. [Online]. Available: http://obi.virtualmethodstudio.com/
  * [32] Y. Fang, Z. Qu, M. Li, X. Zhang, Y. Zhu, M. Aanjaneya, and C. Jiang, “Iq-mpm: An interface quadrature material point method for non-sticky strongly two-way coupled nonlinear solids and fluids,” _ACM Transactions on Graphics_ , 2020\.
  * [33] J. Wolper, Y. Fang, M. Li, J. Lu, M. Gao, and C. Jiang, “Cd-mpm: Continuum damage material point methods for dynamic fracture animation,” _ACM Trans. Graph._ , vol. 38, no. 4, July 2019. [Online]. Available: https://doi.org/10.1145/3306346.3322949
  * [34] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Creus-Costa, S. Savarese, and L. Fei-Fei, “Surreal: Open-source reinforcement learning framework and robot manipulation benchmark,” in _Conference on Robot Learning_ , 2018.
  * [35] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in _Conference on Robot Learning (CoRL)_ , 2019\. [Online]. Available: https://arxiv.org/abs/1910.10897
  * [36] S. James, Z. Ma, D. Rovick Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,” _IEEE Robotics and Automation Letters_ , 2020.
  * [37] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng, “Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2021.
  * [38] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi, “Iqa: Visual question answering in interactive environments,” in _Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on_. IEEE, 2018.
  * [39] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [40] A. Juliani, V.-P. Berges, E. Teng, A. Cohen, J. Harper, C. Elion, C. Goy, Y. Gao, H. Henry, M. Mattar, _et al._ , “Unity: A general platform for intelligent agents,” _arXiv preprint arXiv:1809.02627_ , 2018. [Online]. Available: https://github.com/Unity-Technologies/ml-agents
  * [41] UnityStore. [Online]. Available: https://assetstore.unity.com/packages/tools/modeling/mesh-slicer-59618
  * [42] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann, “Stable baselines3,” https://github.com/DLR-RM/stable-baselines3, 2019\.
  * [43] C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling, “Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning,” 2020\.
  * [44] Stanford Artificial Intelligence Laboratory et al., “Robotic operating system.” [Online]. Available: https://www.ros.org
  * [45] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight experience replay,” _arXiv preprint arXiv:1707.01495_ , 2017.
  * [46] A. Kuznetsov, P. Shvechikov, A. Grishin, and D. Vetrov, “Controlling overestimation bias with truncated mixture of continuous distributional quantile critics,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5556–5566.
  * [47] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International conference on machine learning_. PMLR, 2018, pp. 1861–1870.


# Exploring XAI for the Arts: Explaining Latent Space in Generative Music

Nick Bryan-Kinns, Berker Banar, Corey Ford, Courtney N. Reed,
Yixiao Zhang, Simon Colton, and Jack Armitage
School of Electronic Engineering and Computer Science
Queen Mary University of London
London, E1 4NS, UK
{n.bryan-kinns, b.banar, c.j.ford, c.n.reed,
yixiao.zhang, s.colton<EMAIL_ADDRESS>

###### Abstract

Explainable AI has the potential to support more interactive and fluid co-
creative AI systems which can creatively collaborate with people. To do this,
creative AI models need to be amenable to debugging by offering eXplainable AI
(XAI) features which are inspectable, understandable, and modifiable. However,
currently there is very little XAI for the arts. In this work, we demonstrate
how a latent variable model for music generation can be made more explainable;
specifically we extend MeasureVAE which generates measures of music. We
increase the explainability of the model by: i) using latent space
regularisation to force some specific dimensions of the latent space to map to
meaningful musical attributes, ii) providing a user interface feedback loop to
allow people to adjust dimensions of the latent space and observe the results
of these changes in real-time, iii) providing a visualisation of the musical
attributes in the latent space to help people understand and predict the
effect of changes to latent space dimensions. We suggest that in doing so we
bridge the gap between the latent space and the generated musical outcomes in
a meaningful way which makes the model and its outputs more explainable and
more debuggable.
The code repository can be found at:
https://github.com/bbanar2/Exploring_XAI_in_GenMus_via_LSR

##  1 Introduction

Creating computing systems which can generate music has arguably been both a
dream and a goal of researchers since the 1800s when Ada Lovelace noted that
machines would one day generate “elaborate and scientific pieces of music of
any degree of complexity and extent” [1]. Recently, advances in the field of
generative music have relied on increasingly complex Machine Learning models
[2, 3, 4] – such as neural networks [5, 6] and deep learning techniques [7, 8,
9, 10] – to create convincing musical outputs. However, the complex nature of
these models means that people often require some knowledge of these
techniques and algorithms in order to use or adapt them effectively, making
them difficult for people, especially non-experts, to understand and debug.
The presentation of these models in current interactive musical systems also
means that much of the generation process is invisible to the user; very few
musical applications provide an interface which allows the user to visualise
how a piece of music has been created or explain how their interaction
affected the musical content. This means many generative systems, and digital
musical instruments in general, leave artists feeling disconnected from their
work, or worse, are generally inaccessible to musicians or anyone besides the
creator [11, 12, 13].

In this paper, we explore how eXplainable Artificial Intelligence (XAI) can be
applied to generative music systems to both aid with human understanding of
the model and to support inspection and debugging of the model and its
outputs. First, we outline an overview of XAI and current applications within
the arts, followed by a summary of a systematic literature review of the
explainability of 87 creative AI papers. Then we introduce our implementation
of XAI for the arts by demonstrating how the latent space of a generative
music model can be made more explainable – we contribute a novel user
interface which supports real-time navigation of the latent space of a
generative music model and includes the generation of a set of piano rolls,
colour plots, and audio files, for a trained MeasureVAE [14] model. This is
achieved by sampling the latent space from MeasureVAE and regularising the
dimensions for a set of musical metrics [15, 16]. We conclude by reflecting on
the implementation, and we suggest future directions for research on XAI for
music and the arts.

##  2 Related Work

The field of eXplainable AI examines how machine learning models can be made
more understandable to people, thus increasing their usability and making it
possible for non-experts to utilise them in a variety of contexts. In
particular, XAI researchers explore how non-intuitive and difficult-to-
understand AI models – such as neural networks and deep learning techniques –
can be explained [17, 18]. For example, XAI projects have focused on creating
human-understandable explanations of why an AI system made a particular
medical diagnosis [19], how the AI models in an autonomous vehicle work [20,
21], and what data an AI system uses to generate insights about consumer
behaviour [22]. These XAI applications improve human-in-the loop use, making
it easier to integrate AI into every day tasks and improve the accuracy of
systems in combination with the expertise and human intelligence of the user.

Having more explainable AI for the arts is important for AI arts systems that
we co-create with, referred to as _creative AI_ models, as both artists and
audiences would benefit from a better understanding of i) what an AI model is
doing to generate artistic content and ii) why this artistic content was
generated in response to their own artistic input. This frames the AI as a
tool for creating and co-creating content, rather than a mysterious and opaque
box of tricks, uncontrollable by the artist. Indeed, having more transparent
and understandable AI models is essential for creative AI as co-creation
implicitly requires some level of mutual understanding and engagement both
with the artistic output and with each other [23]. Co-creating with an AI
requires us to be able to inspect, understand, modify and debug both the AI
model and the output it creates in response to an artist’s input. In this way,
the artist can understand their impact on the system and experience feelings
of ownership over their artwork. Designing for this level of understanding of
what an AI system is doing is a key focus of the field of XAI. More broadly,
this relates to the notion of _framing_ information presented in [24] and
surveyed in [25], whereby a generative AI system describes its creations with
additional text or other stimuli. This has been taken further in [26], with
the suggestion that a creative AI system should engage in dialogue with users
to convince them of the value of its output.

The arts, and especially music, also provide a complex domain in which to test
and research new AI models and approaches to explainability. Compared to
domains such as healthcare and automotive industries, the arts require similar
levels of robustness and reliability from their AI models, but have
significantly fewer ethical and life-critical implications, making the arts a
great test-bed for AI innovation. In other words, exploring approaches to XAI
for the arts could both inform the design of XAI for more safety-critical
systems and lead to more intuitive and engaging co-creative AI systems. For
example, music interaction provides an opportunity to study a system’s
sensitivity to time-critical parameters since real-time, understandable
feedback is critical for musicians in co-creating with digital instruments
[27, 28].

As current XAI research is predominantly focused on functional and task-
oriented domains, such as financial modelling, it is difficult to directly
apply XAI techniques to creative AI models. Moreover, the majority of papers
about XAI beyond simple explanations do not focus on the implementation of
XAI, but instead offer design guidelines for explainability [29] or theories
of how such explainability might work [30]. This means that there are few
explainable AI systems to build from. To compound this problem, there are few
creative AI models which provide any explanation of how the model works or
expose any elements of the creative AI model in any meaningful way, which we
demonstrate for the domain of generative music in the following section.

###  2.1 XAI and Generative Music

Taking music generation as a key example of creative AI models, we surveyed 87
recent AI music papers from venues including the New Instruments for Musical
Expression Conference (NIME) series and the Computer Music Journal to examine
what role the AI had in the creative process and how much of the AI in these
interactive systems is actually explained or exposed to humans in the system
itself (rather than being explained in the paper). Our review sample started
with the 94 papers reviewed in [3]; 19 papers which were purely related to
cognitive theory or not accessible to the authors were removed. We also added
12 more recent creative AI papers to represent current practice, including the
use of XAI in other HCI fields [31, 32, 30, 33, 34, 35, 36, 37, 38, 39, 40,
41], making a total of 87 papers reviewed. We analysed the papers using three
existing frameworks to capture the key features of _XAI for the arts_.
Specifically, we examined the following for AI in the interactive arts: the
_role_ of the AI in co-creation, possible _interaction_ with the AI, and how
much _common ground_ humans might be able to establish with the AI, as
follows.

_The role of the AI_ – we used Lubart’s classification of the role of AI as a
creative partner [42] to classify the role of the AI from models which take
care of generative tasks without interacting with humans through to AI models
which take on the role of a colleague in creative collaborations.

_Interaction with the AI_ – we used Cornock and Edmonds’ classification of
interaction styles with interactive art from static to dynamic-interactive
[43] as the more interactive and responsive an AI is, the more chance there is
for people to understand what the AI is doing and might do in the future.

_The common ground with the AI_ – we drew on Clark and Brennan’s work on
grounding in human communication [44] to classify what a person might be able
infer about an AI’s output state from a low stage of grounding where a person
is only aware that some output has been made through to a high stage of
grounding where they have an understanding of its meaning and can make an
informed reaction to the output.

Our perspective is that the explainability of creative AI is a combination of
the role the AI takes, the interaction it offers, and the grounding that can
be established with the AI. The more collaborative the role is, the more
explanation is required which in turn necessitates more interaction and
grounding. Increased opportunities for interaction help people to learn about
and infer an understanding of the AI and its behaviour. Increased levels of
grounding offer more chances for people to understand what a creative AI did,
and why. Increased interaction and grounding offer more changes for people to
inspect, understand, and debug AI models and their creative output.

In our review, we found some excellent examples of creative AI which take the
role of a colleague. For example, Shimon the robotic marimba player [45]
listens to live human players, analyses perceptual aspects of their playing in
real-time and plays along in a collaborative and improvisatory manner. In
terms of interaction, Shimon provides a real-time feedback loop within the art
work itself, making the collaboration highly dynamic and interactive. However,
Shimon offers only mid stage grounding as there is no explanation of what it
did, nor _how_ or _why_ Shimon made particular musical responses in the
improvisation.

Other generative tools, such as Hyperscore [46], demonstrate higher levels of
common ground with the user. In Hyperscore’s interface, the reactive change to
input is displayed in a piano-roll notation. This allows the user to observe
the effects of their input and develop an understanding of the system’s
response. The interactive controls in the interface provide a way for the user
to experiment with melody creation, while the system preserves their original
melodic curve ideas and allows them to make incremental and reversible
changes. In this way, the user can see _how_ their input changes the output;
however, the internal model and the reasons _why_ the system reacts are still
largely obscured.

In our review, we found that 73 of the 87 papers took the role of generating
music without any human collaboration. There was also little interaction
offered by the AI music systems we reviewed: 41 papers did not not offer any
real-time interaction with humans at all, but rather generated melodies from
training data, without any user input or decisions. Additionally, 76 papers
were at the lowest stage of grounding meaning that although the AI makes a
musical contribution, a person would not reliably be able to discern what the
AI system did based on their input, and would simply be aware that some
musical output was generated somehow. These kinds of creative AI are difficult
to debug as their implementations are not meaningfully exposed, and the
technical complexity of their interfaces prevents musicians from using them.

To summarise, while there are compelling examples of creative AI systems that
collaborate with musicians, only a few creative AI systems explain what their
models are doing, how they do it, and why. This makes debugging such models
and their output problematic. The rest of this paper explores this gap through
our approach to explainable latent spaces in music generation.

##  3 Explaining Latent Spaces

As our first step in investigating more explainable AI for the arts we have
been researching how to make latent spaces in AI music models more
understandable. Recently, some systems developed for human-AI co-creation have
exposed the latent space of generative music models [23, 14, 47, 48, 49],
meaning that users can create their own music by navigating a possibility
space, typically represented on a blank 2D grid. For example, latent space
models have been successfully used for creative applications such as music
generation [50, 51], music inpainting [14], and music interpolation [52].
However, to date, these systems have been opaque AI models and do not offer
the higher-level XAI properties of interaction and grounding we discussed in
Section 2.1. For example, Murray-Browne and Tigas [48] trained a latent space
on a set of dance postures which was then mapped to various musical outputs;
although dancers found distinctive ways of performing with the system, how
their movements directly influenced the music was unclear. If we can offer
more explainable approaches to exposing latent spaces then there is more
chance for people to be able to debug the AI model, its training, and the
output it generates.

For our system, we built on the popular MeasureVAE system111[14] and [15],
which we build upon in our work, are licensed under a Creative Commons
Attribution-NonCommercial-ShareAlike 4.0 International License., which Pati et
al. [14] describe as being “successful in modeling individual measures of
music”. In response to an input extract of music, the model generates a
similar measure of music by: i) encoding the input measure into a latent space
via bi-directional recurrent neural network (RNN), ii) sampling the encoder’s
latent space (z), and iii) decoding z via a combination of RNNs and linear
stacks. Notably, the decoder uses two uni-directional RNNs: one is responsible
for beats (four beats in a measure), and the other one is for ticks (six
ticks/ symbols per beat) [14]. For a full description, we point the reader
towards [14].

In keeping with current research on AI music generation we trained MeasureVAE
using publicly available music from a single genre to ensure coherence of the
training data - in our case 20,000 publicly available monophonic Irish folk
melodies [53]. The data is partly anonymous, showing contributor’s names from
a community website dedicated to Irish traditional music, and contains no
offensive content. This produces a latent vector of 256 dimensions, as
illustrated in Figure 1. Once trained, inputting a melody into the encoder
will generate a new melody through the decoder. The features of the new
melodies can be varied by modifying the values of the dimensions in the latent
vector. However, there is no way for a person to know what effect changing
these dimensions would have on the generated music. In addition, the 256
entangled dimensions make it difficult to perceive a difference in output from
a change in any single dimension, making our Irish folk song generator opaque
and unexplained.

Figure 1: The simplified MeasureVAE

However, if we use latent space regularisation (LSR) [54] in training the VAE
– which has been widely used in the study of controlled generation of images
[55] and music [15, 56] – we can make this creative AI approach more
understandable and explainable. We use LSR to force some specific dimensions
of the latent space to represent specific musical attributes similar to the
method in Pati and Lerch [15, 16]. Specifically, we assign dimensions 0, 1, 2
and 3 to rhythmic complexity, note range, note density, and average interval
jump respectively (see Figure 2). We selected these metrics as typical
examples of meaningful musical features used in current research in order to
demonstrate our approach.

Figure 2: The simplified MeasureVAE with LSR

In this way, specific dimensions in our latent space correspond to meaningful
musical attributes in AI-generated outputs. Since these attributes are
manipulable, they can form the basis for a real-time user interface
illustrated later in this paper, thereby creating a feedback loop between
input music, human modifiable dimensions, and AI-generated music. We suggest
that this feedback loop can support the debugging of the creative AI and its
output. Furthermore, we suggest that this support can be offered in a form
that is commensurate with musician’s existing skills and usability
expectations.

###  3.1 Implementation

To utilise the MeasureVAE architecture with the latent space regularisation
technique, we build on the implementation of
[15]222https://github.com/ashispati/AttributeModelling. The training data of
20,000 monophonic Irish folk melodies [53] are converted into a measure-based
data representation. Each measure is represented with 24 characters, where
each character corresponds to one of the sixteenth note triplets in a 4/4
measure, with a total of 24 characters per measure [14, 16]. These characters
include note names (A3, G5, …), continuation tokens (_) and special rest
tokens.

We use four musical attributes in our model following [15, 16]; Toussaint’s
rhythmic complexity [57], note or chromatic range (max pitch - min pitch),
note density (number of notes in a measure) and average interval jump. Average
interval jump represents the average of the absolute values of the interval
between adjacent notes in a single measure melody.

We jointly train our MeasureVAE model with latent space regularisation on all
four attributes, to force four specific dimensions of the latent space to
represent given musical attributes [15, 16]. We apply these constraints to the
first four dimensions of the 256-dimensional latent vector, and assign them to
rhythmic complexity, note range, note density and average interval jump, and
add an attribute-specific regularisation loss to the training objective of the
VAE. Specifically, for each attribute, a musical metric value (e.g. average
interval jump) is calculated for each item in a mini-batch. Then, a distance
matrix (Dattribute) is created by calculating the distance between each item’s
metric value and all the other item’s metric values resulting in an N x N
matrix (N examples in a mini-batch). Similarly, another distance matrix
(Ddimension) is created for the values of the regularised latent dimension,
again resulting in an N x N matrix. Subsequently, the mean squared error of
(tanh(Ddimension) - sgn(Dattribute)) is added to the VAE objective. Finally,
after training the model with LSR, the values of these dimensions become
monotonically tied to the corresponding music attributes. Therefore, when we
change the values, the corresponding attributes of the generated music change
accordingly.

We use Adam [58] as the optimizer of the model with learning rate = 1e-4,
$\beta_{1}$ = 0.9, $\beta_{1}$ = 0.999 and $\epsilon$ = 1e-8. The model is
trained on a single GeForce RTX 2080 Ti GPU for a total of 30 epochs following
the same setting of [15], taking an average of 2.5 hours per epoch. After
training, the reconstruction accuracy of the LSR model achieves 99.87% on the
training set and 99.68% accuracy on the validation set, and the non-LSR model
achieves 99.84% and 99.77% respectively333The non-LSR model is trained for 11
epochs as this gave the best accuracy.. As calculated in [15, 16], we have an
interpretability metric, which is from [59] and measures how well we can
predict an attribute using only one dimension in the latent space. For the LSR
model, interpretability scores for rhythmic complexity, note range, note
density and average interval jump are 0.80, 0.99, 0.99 and 0.91 (average 0.92)
for their corresponding dimensions, respectively (the higher the better). For
the non-LSR model, interpretability scores are 1.5e-4, 9.1e-6, 1.7e-5 and
1.2e-6 in the same order.

Two user interfaces (UIs) were built using React.js and deployed as web
applications online to demonstrate interaction with LSR444https://xai-lsr-
ui.vercel.app/ and without LSR555https://xai-no-lsr-ui.vercel.app/. In each
demonstration, we encode an input MIDI measure using the trained encoder and
obtain its latent vector. Then, to generate musically controlled variations of
it, we manipulate the values of the first four dimensions and decode these
manipulated latent vectors to obtain output music sequences. To demonstrate
the explainable latent space, we sweep the regularised latent dimensions
discretely and sample values. For each dimension, we take 10 equally spaced
samples, creating every possible latent vector combination using these 10
samples per regularised dimension, in total 10,000 latent vectors for four
dimensions / attributes. In these latent vectors, values for non-regularised
dimensions are kept as they are in the latent vector of the encoded input
sequence. To determine the sampling limits, we store the latent vectors of the
training set and generate histograms of the regularised dimension values. The
limits are then set according to the histograms (with slight modifications to
allow wider ranges in terms of the musical metrics). Once we have these 10,000
latent vectors, they are decoded into output music sequences and we generate
MIDI files of them. For the sake of practicality, we generate piano-rolls and
MP3 files for each MIDI output in advance, and host them online for the web
demos. The generated MIDI, piano-roll and audio files are available at our
GitHub
repository666https://github.com/bbanar2/Exploring_XAI_in_GenMus_via_LSR.

###  3.2 Interaction

Figure 3: Screenshot of the user interface with LSR

Both demos feature UIs that provide real-time interaction with our creative AI
system. For example, Figure 3 shows the UI for the demo with LSR. In both UIs
the input MIDI measure is shown and can be listened to on the left-hand side
of the UI in piano-roll format, and the generated output MIDI measure is shown
and can be listened to on the right-hand side of the UI also in a piano-roll
format. The main points of interaction are two 2D-pads in the centre of the
UI; when clicked on, the user can navigate the pads by dragging their mouse,
which controls the red dot. The left-hand pad controls the rhythmic complexity
and note range dimensions, whereas the right-hand pad controls the note
density and average interval jump dimensions (see the axis labels on the LSR
version in Figure 3). These two pairings were chosen to maximise the semantic
differences between the dimensions in each pad in order to offer users more
observable effects of changes in position within the pad.

The white dots on the 2D pads refer to the corresponding latent vector
dimension values of the input sequence, given as reference points. As users
hover over these pads – selecting different latent dimension values – the
output MIDI is updated in real-time and played back to people. Musically,
these outputs correspond to variations of the input sequence which vary as we
manipulate its latent vector for generation; for example, we will have a
higher range of notes in the generated musical sequence when the note range
dimension is high.

In our demo with LSR we increase the explainability of the AI in two ways: i)
key parts of the AI model are exposed to the user in the interface and
meaningfully labelled (in this case, with musical features), and ii) the real-
time interaction and feedback in the UI allows people to explore the effects
of these features on the generative music and thereby implicitly learn how the
model works.

Referring to the three properties of XAI for the arts described in Section
2.1, in our implementation the AI acts somewhat like a colleague [42] – the
response to the user is given in real-time, as would be done in a human-to-
human musical interaction. This drives a feedback loop between a user and the
AI, whereby a person’s reaction to the AI’s response informs the subsequent
interaction. Thinking musically, this resembles a duet in creative
improvisation, where the players make real-time decisions based on their
colleague’s performance.

In terms of grounding between the AI and the user, the system displays the
possible user input parameters to the model through the labeled pads, 2D
displays, and movable cursors, and moreover, the visualisation changes in
response to these control actions. As the UI provides musically meaningful
labels and an interaction feedback loop (which allows for exploratory user
learning to predict how user input might change the AI’s output), we see this
system as being towards the highest level of grounding [44], where a person is
provided with cues to the AI model’s current state and can predict possible
next steps. Furthermore, the real-time interaction creates a sensation of
“playing” the model and helps to recreate other familiar musical interfaces
through the use of the piano-roll notation. Real-time feedback provides
musicians with an assurance that their input is being received, increases
accuracy in timing during use, and positively influences their perceptions of
the quality and usability of a system [27]. Comparing this to other generative
music systems which often take input at the command line, the use of pads and
sliders and note visualisation on a piano-roll (commonplace in digital musical
interfaces) is more intuitive and typical of musical interaction. In this
sense, the system provides an interaction which is both dynamic and
understandable in terms of the generative system itself and also tailored
around the specific context of music creation.

###  3.3 Visualisation

To provide further insight into our AI model, we display three 2D plots within
each of the pads based on the visualisations in [15] as illustrated in Figure
4. Firstly, training data contribution plots (4a and 4d) for the left and
right-hand pads respectively represent how many items in the training data set
have contributed to a specific location in the latent space. Statistically,
this provides an idea of how unlikely the output will be for any location,
given the musical structure of the dataset. In each of the pads we also have
two surface map figures as used in [15, 16] and illustrated in 4b and 4c for
the left-hand pad, and 4e and 4f for the right-hand pad. We create these
surface maps by decoding corresponding latent vectors for each point (non-LSR
dimensions are kept the same as the encoded input vector, and the other two
LSR dimensions are taken from the latent vector of the input music) and
calculating the musical attribute values for each decoded sequence. Since we
have two attributes per 2D pad, we have two surface maps for each pad. These
surface maps illustrate how the LSR technique works since the metric values
increase (yellow regions) for the higher parts of the corresponding axis.
These surface maps also offer detailed information about regions to hover
over, and allow inference of the potential kinds of outputs resulting from
different points in the latent space dimensions. In doing so we offer a
different kind of explainable interaction than other creative AI systems which
allow interaction with latent space such as [49, 47, 48]. In other words, we
increase the explainability of the UI by providing a rich visualisation from
which people can infer, with some meaning, the gist of what the AI model might
generate for particular latent dimension values as they interact with it.

(a) TDRCNR

(b) RC

(c) NR

(d) TDNDAIJ

(e) ND

(f) AIJ

Figure 4: Visualisations: a) Training Data visualised in terms of rhythmic
complexity and note range (TDRCNR); b) Rhythmic Complexity surface map (RC);
c) Note Range surface map (NR); d) Training Data visualised in terms of note
density and average interval jump (TDNDAIJ); e) Note Density surface map (ND);
f) Average Interval Jump surface map (AIJ)

The non-LSR demo UI surface maps are different to the LSR version and show the
same musical attribute value for any point in the latent space. This is
because there is nothing meaningful about these dimensions as LSR has not been
applied and changing the value of these individual dimensions in a 1 x 256
vector does not have any significant effect on the decoded sequence (which is
to be expected behaviour for this non-LSR case). In other words, simply
visualising the surface maps in the non-LSR version does not increase the
explainability of the AI.

###  3.4 Debugging

By using latent space regularisation to force a small set of dimensions to be
mapped to semantically meaningful musical features, we hope to better support
artists using MeasureVAE in key debugging activities of _inspecting_ the
model, _understanding_ how the model works, and _changing_ the model to create
the desired output.

We suggest that the real-time nature of the feedback provided by our demo
would make it easier to inspect and develop an understanding of the AI model.
The user is immediately able to see the results of their actions, which
informs any follow-up from the user needed to debug the system or their
previous input. While this is an important feature for debugging in general,
it is especially the case in a music context: musical interaction is time-
dependent, meaning an artist must quickly be able to make an informed reaction
or otherwise risk derailing a performance or composition process. By
supporting real-time feedback, we increase the common ground between the AI
and the human. This elevates the creative process from a situation where we
know that the AI has done something in response to our input, but we are not
sure what (e.g. Shimon [45]), to a level where we can directly see and start
to understand what the AI has done in response to our input. We suggest that
users would thus be able to predict (with some learning on the human side) how
changes to the regularised dimensions of the latent space change the
generative output in a way which can better fulfil their musical intentions.
It is worth noting that there are many other time-sensitive applications where
such XAI features would greatly benefit the user, including the more
functional and task-oriented tools described earlier in medical care and
transportation.

Importantly, our demo retains its original input whilst other parameters are
changed. This allows users to compare the current AI output with their
original contribution, contributing to debugging of expected outcomes. This is
similar to the design of Hyperscore [46], where a melodic curve is always
retained as reference, whilst other parameters are changed. Moreover, in our
demonstration, users are able easily revert back to previous settings by
moving the red dots in the UI (Figure 3) between different positions in the
latent space. Users are also able to quickly observe the results of their
tinkering in real-time, and so may develop a better understanding of the
mapping between changes in latent space dimension values and the resultant
generated output. This form of interaction allows for trial-and-error
debugging of the creative output as well as supporting users in exploratory
learning of dimension mappings. In this way, we support creativity support
tool design principles such as Shneiderman et al.’s [60] principles of
supporting exploration (as people can quickly trial different ideas), and
offering a ‘high ceiling’ of tools (as people can tinker with a wide range of
different options).

In summary, combining support for inspecting and understanding of the AI model
with being able to interactively manipulate the model offers us opportunities
to debug the model. In other words, by providing more grounded and interactive
explanations of what the AI is doing we offer greater opportunities for
creative exploration of the musical space, and importantly, increased support
for debugging of both the creative output and the AI model itself.

##  4 Limitations, Societal Impact, and Future Work

There are currently several limitations to our demo which need to be addressed
in further work. Firstly, the musical parameters chosen for interaction with
the system are a somewhat arbitrary selection of parameters typically found in
contemporary research on AI music generation and present only a small subset
of the variables in music creation. Future work needs to explore which
features are most useful for explaining the workings of the AI model.

Secondly, the system and interaction focus on manipulating the pitch and
rhythmic aspects of a single measure of music. This ignores timbre and higher-
level structure in music, operating with only a fraction of the variables
musicians have control over on a traditional instrument or composition tool.

Thirdly, for a non-musical user, the system may still present some
explainability barriers. The chosen parameters – although labelled and
visualised – require the user to have some prior understanding of music. Other
control mechanics, such as the semantic sliders used in [23], may be more
appropriate, although they currently do not give explanations for the link
between interaction and output.

Fourth, our surface map visualisations are static and intended to be
illustrative of the regularised dimensions. Future work could investigate
dynamic surface maps which are rendered according to the currently selected
points in the dimensions e.g. the surface map for the note density & average
interval jump 2D pad could be changed in real-time according to the currently
selected point in the rhythm complexity & note range 2D pad.

Fifth, we only focus on explainability of the latent space forming at the
bottleneck of the MeasureVAE architecture, in-between the encoder and decoder
blocks. To extend explainability to the layers of encoder and decoder blocks,
one technique we are interested in applying is Concept Whitening (CW) [61]. CW
can be practically applied to any layer of a network and could be used to
demystify how the network learns concepts through those layers without harming
the main training objective. This technique aligns concepts, which in our case
might be musical attributes or more high-level concepts such as genres or
moods, with latent space axes of a given layer by doing de-correlation,
standardisation and orthogonal transformation [61].

Sixth, the comparison of the LSR version to the non-LSR version of the demo
might be improved by using some non-semantic dimensionality reduction
technique in the non-LSR demo to address the lack of perceptible effect of
user changes to dimensions in the non-LSR demo. For example, the non-LSR demo
could be extended to compute Singular Value Decomposition (SVD) in the latent
space over the training set and then allow the user to manipulate the four SVD
directions with the highest variance. This would allow comparison of two demos
in which user interaction has perceptible effect on both demo’s musical
output, allowing us to compare the explainability of semantically regularised
dimensions versus non-semantic dimensionality reduction.

Finally in terms of implementation, the novelty of our implementation demo is
limited as it involves the combination of two published approaches. Future
work needs to explore a larger set of combinations of approaches drawing from
both published and unpublished research.

Our current system suffers from a problem commonly found with AI music models:
the dataset used to train the model is limited within the Western music canon,
and even more so to a specific genre of folk songs. This limits the output to
specific tonal and rhythmic features. It is possible that the compositions
created using a tool like ours will sound similar to the songs of the dataset
we use, which decreases the diversity of music created. Future research should
explore the utility of our approach with training sets of different genres of
music, and also training sets containing combinations of genres.

The system presented in this paper needs in-depth study and validation by
users with a range of musical knowledge. For example, we selected two pairings
of dimensions for the 2D pads in the centre of the user interface as discussed
in Section 3.2. The validity of this design choice needs to be evaluated with
users in terms of: i) whether the chosen pairings (left-hand: rhythmic
complexity & note range; right-hand: note density & average interval jump) are
the most useful and intuitive for users; and ii) whether including additional
2D pads would improve the explainability of the system or instead be confusing
for users e.g. six 2D pads could be displayed to users to present all possible
combinations of the four LSR dimensions but this would substantially increase
the visual complexity of the user interface and may lead to cognitive
overload. In addition, further development of such a system must include
working directly with potential users on how such a system may benefit their
composition or performance practice, and how their own artistic identity can
be better incorporated and expressed. Moreover, explorations of use cases by
artists will further suggest interaction mechanics and changes to the user
interface, and provide insight into how the system might assist in creative
and debugging processes.

As we outlined earlier in this paper, undertaking AI research in the arts
provides a demanding real-world and real-time context in which approaches such
as XAI can be explored, without the risk of substantial negative societal
impact in life-critical domains such as healthcare and automotive industries.
However, there are potential negative societal impacts on artistic practice
and livelihoods of our work and creative AI research in general which must be
considered. Most importantly, some argue that co-creative AI would diminish
human creativity, remove the human from the creative process, and devalue
human creativity itself. There is also concern that the use of generative
models would lead to a homogenisation of music and a marginalisation of
musical skills and traditions which are not amendable to reproduction by AI.
In our view, through human-centred XAI and the design of UIs which embrace the
user’s role and interaction with the AI model, as presented here, we can
proactively work to ensure that the artist remains key to the the creative
process. Indeed, our view is that working with artists to design and implement
explainable AI systems will help to mitigate concerns about the impact of AI
on creativity.

##  5 Conclusions

Explainable AI is a growing research field which has the potential to
contribute to making AI systems more co-creative. Taking AI music as a key
example of AI for the arts, it is clear that there is huge potential for more
explainable AI models, given the limited explainability of current creative AI
models. Typically, these offer limited interaction and low levels of grounding
between AI and human – a situation where we notice that the AI has created
something, but are not sure how the AI output relates to our input. In this
paper, we demonstrated how latent spaces can be made more explainable, and in
doing so showed how they could support debugging as part of creative practice.

We suggest that future work in XAI for the arts should move away from the
functional explanations explored by current XAI research and focus instead on
conveying the _gist_ of what AI models are doing. Much like the third wave of
Human-Computer Interaction (HCI) [62] which shifted HCI research to focus on
experience and meaning making, conveying gist will be a paradigm shift in how
we design and use AI in creative settings. By following an interdisciplinary
approach, where creative AI presents information that is meaningful to people
– such as by presenting visual cues between mappings [49], or visualising
levels of mutual trust with emoticons [63] – we can better support human-AI
collaboration. Once we create AI systems that convey the gist of what they are
doing creatively, we will have the chance to mutually engage with AI in co-
creation.

Finally, we believe that when we think of “explainable” AI, we should consider
how the design of our systems embrace the existing knowledge, experience, and
practices that users will bring to the interaction cf. [29]. A critical
question designers can ask is whether the AI is explained in the context in
which it will be used. In this paper, we present an interactive music
generation system which works with interface elements familiar to musicians,
and focuses around the real-time feedback relationship between musician and
instrument. In presenting the AI within the existing musical context, we can
increase the grounding between user and AI as tool and even collaborator. In
similar applications of XAI and the arts, and indeed in all applications, this
attention to context and using what is already understandable to users will
aid the explanation of the underlying systems.

##  6 Acknowledgements

This work was supported by a Queen Mary University of London Research Enabling
Fund. The work was undertaken by Banar, Ford, Reed, Zhang, and Armitage under
the direction of Bryan-Kinns and with the advice of Colton. Banar and Ford are
research students at the UKRI Centre for Doctoral Training in Artificial
Intelligence and Music, supported jointly by UK Research and Innovation (grant
number EP/S022694/1) and Queen Mary University of London. Reed is funded by a
Principal Electronic Engineering and Computer Science Studentship from Queen
Mary University of London. Yixiao is a research student at the UKRI Centre for
Doctoral Training in Artificial Intelligence and Music, supported jointly by
the China Scholarship Council and Queen Mary University of London.

## References

  * Lovelace [1843] A. A. Lovelace, “Notes by August Ada Lovelace,” _Taylor’s Scientific Memoirs_ , vol. III, pp. 666–731, 1843.
  * Sturm et al. [2019] B. L. Sturm, O. Ben-Tal, Úna Monaghan, N. Collins, D. Herremans, E. Chew, G. Hadjeres, E. Deruty, and F. Pachet, “Machine learning research that matters for music creation: A case study,” _Journal of New Music Research_ , vol. 48, no. 1, pp. 36–55, 2019. [Online]. Available: https://doi.org/10.1080/09298215.2018.1515233
  * Herremans et al. [2017] D. Herremans, C. Chuan, and E. Chew, “A Functional Taxonomy of Music Generation Systems,” _ACM Computing Surveys_ , vol. 50, no. 5, Sep. 2017\. [Online]. Available: https://doi.org/10.1145/3108242
  * Carnovalini and Rodà [2020] F. Carnovalini and A. Rodà, “Computational Creativity and Music Generation Systems: An Introduction to the State of the Art,” _Frontiers in Artificial Intelligence_ , vol. 3, p. 14, 2020. [Online]. Available: https://www.frontiersin.org/article/10.3389/frai.2020.00014
  * Todd [1989] P. M. Todd, “A Connectionist Approach to Algorithmic Composition,” _Computer Music Journal_ , vol. 13, no. 4, pp. 27–43, 1989. [Online]. Available: https://doi.org/10.2307/3679551
  * Eck and Schmidhuber [2002] D. Eck and J. Schmidhuber, “A First Look at Music Composition using LSTM Recurrent Neural Networks,” _Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale_ , vol. 103, p. 48, 2002.
  * Briot et al. [2019] J. Briot, G. Hadjeres, and F. Pachet, “Deep Learning Techniques for Music Generation – A Survey,” 2019.
  * Hadjeres et al. [2017a] G. Hadjeres, F. Pachet, and F. Nielsen, “DeepBach: a Steerable Model for Bach Chorales Generation,” in _Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia_. PMLR, 2017, pp. 1362–1371.
  * Zhu et al. [2018] H. Zhu, Q. Liu, N. J. Yuan, C. Qin, J. Li, K. Zhang, G. Zhou, F. Wei, Y. Xu, and E. Chen, “XiaoIce band: A Melody and Arrangement Generation Framework for Pop Music,” in _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 2837–2846. [Online]. Available: https://doi.org/10.1145/3219819.3220105
  * Huang et al. [2018] C. A. Huang, A. Vaswani, J. Uszkoreit, N. Shazeer, C. Hawthorne, A. M. Dai, M. D. Hoffman, and D. Eck, “Music Transformer: Generating Music with Long-Term Structure,” _arXiv Preprint arXiv:1809.04281_ , 2018.
  * Wallis et al. [2013] I. Wallis, T. Ingalls, E. Campana, and C. Vuong, “Amateur Musicians, Long-term Engagement, and HCI,” in _Music and Human-Computer Interaction_. Springer, 2013, pp. 49–66.
  * Benyon and Macaulay [2002] D. Benyon and C. Macaulay, “Scenarios and the HCI-SE design problem,” _Interacting with Computers_ , vol. 14, pp. 397–405, 07 2002. [Online]. Available: https://doi.org/10.1016/S0953-5438(02)00007-3
  * Morreale and McPherson [2017] F. Morreale and A. McPherson, “Design for Longevity: Ongoing Use of Instruments from NIME 2010-14,” in _Proc. NIME’17, May 15-19, 2017, Aalborg University Copenhagen, Denmark_ , 05 2017, pp. 192–197.
  * Pati et al. [2019] A. Pati, A. Lerch, and G. Hadjeres, “Learning to traverse latent spaces for musical score inpainting,” _arXiv preprint arXiv:1907.01164_ , 2019.
  * Pati and Lerch [2019] A. Pati and A. Lerch, “Latent Space Regularization for Explicit Control of Musical Attributes,” 06 2019.
  * Pati and Lerch [2020] ——, “Attribute-based Regularization of Latent Spaces for Variational Auto-Encoders,” 2020.
  * Kozierok et al. [2021] R. Kozierok, J. Aberdeen, C. Clark, C. Garay, B. Goodman, T. Korves, L. Hirschman, P. L. McDermott, and M. W. Peterson, “Hallmarks of Human-Machine Collaboration: A framework for assessment in the DARPA communicating with Computers Program,” 2021.
  * Gunning [2016] D. Gunning, “Explainable Artificial Intelligence (XAI),” _DARPA/I2O Proposers Day_ , Aug 2016.
  * Quellec et al. [2021] G. Quellec, H. Al Hajj, M. Lamard, P. Conze, P. Massin, and B. Cochener, “ExplAIn: Explanatory artificial intelligence for diabetic retinopathy diagnosis,” _Medical Image Analysis_ , p. 102118, 2021.
  * Du et al. [2019] N. Du, J. Haspiel, Q. Zhang, D. Tilbury, A. K. Pradhan, X. J. Yang, and L. P. Robert Jr, “Look who’s talking now: Implications of AV’s explanations on driver’s trust, AV preference, anxiety and mental workload,” _Transportation research part C: emerging technologies_ , vol. 104, pp. 428–442, 2019.
  * Shen et al. [2020] Y. Shen, S. Jiang, Y. Chen, E. Yang, X. Jin, Y. Fan, and K. D. Campbell, “To explain or not to explain: A study on the necessity of explanations for autonomous vehicles,” _arXiv preprint arXiv:2006.11684_ , 2020.
  * Sajja et al. [2021] S. Sajja, N. Aggarwal, S. Mukherjee, K. Manglik, S. Dwivedi, and V. Raykar, “Explainable AI based Interventions for Pre-season Decision Making in Fashion Retail,” in _8th ACM IKDD CODS and 26th COMAD_ , 2021, pp. 281–289.
  * Louie et al. [2020a] R. Louie, A. Cohen, C. A. Huang, M. Terry, and C. J. Cai, “Cococo: AI-Steering Tools for Music Novices Co-Creating with Generative Models,” in _HAI-GEN+ user2agent@ IUI_ , 2020.
  * Charnley et al. [2012] J. Charnley, A. Pease, and S. Colton, “On the notion of framing in computational creativity.” in _Proceedings of the 3rd International Conference on Computational Creativity_ , 2012, pp. 77–81.
  * Cook et al. [2019] M. Cook, S. Colton, A. Pease, and M. T. Llano, “Framing in computational creativity - a survey and taxonomy,” in _Proceedings of the Tenth International Conference on Computational Creativity_ , 2019.
  * Llano et al. [2020] M. T. Llano, M. d’Inverno, M. Yee-King, J. McCormack, A. Ilsar, A. Pease, and S. Colton, “Explainable computational creativity,” in _Proceedings of the Eleventh International Conference on Computational Creativity_ , 2020.
  * Jack et al. [2018] R. H. Jack, A. Mehrabi, and T. S. A. McPherson;, “Action-sound Latency and the Perceived Quality of Digital Musical Instruments: Comparing Professional Percussionists and Amateur Musicians,” _Music Perception_ , vol. 1, no. 36, pp. 109–128, September 2018. [Online]. Available: https://doi.org/10.1525/mp.2018.36.1.109
  * McPherson et al. [2016] A. McPherson, R. Jack, and G. Moro, “Action-Sound Latency: Are Our Tools Fast Enough?” in _Proc. NIME’16, 20–25 July 11-15, 2016, Griffith University, Brisbane, Australia_ , 07 2016.
  * Liao et al. [2020] Q. V. Liao, D. Gruen, and S. Miller, _Questioning the AI: Informing Design Practices for Explainable AI User Experiences_. New York, NY, USA: Association for Computing Machinery, 2020, p. 1–15. [Online]. Available: https://doi.org/10.1145/3313831.3376590
  * Andres et al. [2020] J. Andres, C. T. Wolf, S. Cabrero Barros, E. Oduor, R. Nair, A. Kjærum, A. B. Tharsgaard, and B. S. Madsen, “Scenario-based XAI for Humanitarian Aid Forecasting,” in _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_ , 2020, pp. 1–8.
  * Sturm et al. [2015] B. L. Sturm, J. F. Santos, and I. Korshunova, “Folk Music Style Modelling by Recurrent Neural Networks with Long Short Term Memory Units,” _Extended Abstracts for the Late-Breaking Demo Session of the 16th International Society for Music Information Retrieval Conference_ , 2015.
  * Hazzard et al. [2019] A. Hazzard, C. Greenhalgh, M. Kallionpää, S. Benford, A. Veinberg, Z. Kanga, and A. McPherson, “Failing with Style: Designing for Aesthetic Failure in Interactive Performance,” in _2019 CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019) Glasgow, Scotland, UK._ ACM, New York, NY, USA, 2019, p. 13 pages.
  * Long et al. [2021] D. Long, A. Padiyath, A. Teachey, and B. Magerko, “The Role of Collaboration, Creativity, and Embodiment in AI Learning Experiences.” in _In Creativity and Cognition (C &C ’21), June 22, 23 Virtual Event, Italy._ ACM, NY, USA,, 2021, pp. 1–10. [Online]. Available: https://doi.org/10.1145/3450741.3465264
  * Malsattar et al. [2019] N. Malsattar, T. Kihara, and E. Giaccardi, “Designing and Prototyping from the Perspective of AI in the Wild,” in _Proceedings of the on Designing Interactive Systems Conference (DIS ’19), June 23–28, San Diego, CA, USA_ , 2019. [Online]. Available: http://dx.doi.org/10.1145/3322276.3322351
  * Benetatos et al. [2020] C. Benetatos, J. VanderStel, and Z. Duan, “BachDuet: A Deep Learning System for Human-Machine Counterpoint Improvisation,” in _Proc New Interfaces for Musical Expression (NIME’20)_ , 2020.
  * Proctor and Martin [2020] R. Proctor and C. P. Martin, “A Laptop Ensemble Performance System using Recurrent Neural Networks.” Proc. New Interfaces for Musical Expression (NIME’20), 2020.
  * Lupker [2021] J. A. T. Lupker, “Score-Transformer: A Deep Learning Aid for Music Composition,” in _NIME 2021_ , 2021.
  * Zhang et al. [2021] Y. Zhang, G. Xia, M. Levy, and S. Dixon, “COSMIC: A Conversational Anterface for Human-AI Music Co-Creation,” in _NIME 2021_ , 2021\.
  * Roberts et al. [2019] A. Roberts, J. Engel, Y. Mann, J. Gillick, C. Kayacik, S. Nørly, M. Dinculescu, C. Radebaugh, C. Hawthorne, and D. Eck, “Magenta Studio: Augmenting Creativity with Deep Learning in Ableton Live,” in _Proceedings of the International Workshop on Musical Metacreation (MUME)_ , 2019. [Online]. Available: http://musicalmetacreation.org/buddydrive/file/mume_2019_paper_2/
  * Gillick and Bamman [2021] J. Gillick and D. Bamman, “What to Play and How to Play it: Guiding Generative Music Models with Multiple Demonstrations,” in _NIME 2021_ , 2021.
  * Louie et al. [2020b] R. Louie, A. Coenen, C. A. Huang, M. Terry, and C. J. Cai, “Cococo: AI-Steering Tools for Music Novices Co-Creating with Generative Models,” in _IUI ’20 Workshops, March 17, 2020, Cagliari, Italy._ ACM, NY, USA, 2020, pp. 1–6.
  * Lubart [2005] T. Lubart, “How can computers be partners in the creative process: Classification and commentary on the Special Issue,” _International Journal of Human-Computer Studies_ , vol. 63, no. 4, pp. 365–369, 2005, computer support for creativity. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1071581905000418
  * Cornock and Edmonds [1973] S. Cornock and E. Edmonds, “The Creative Process Where the Artist Is Amplified or Superseded by the Computer,” _Leonardo_ , vol. 6, no. 1, pp. 11–16, 1973. [Online]. Available: http://www.jstor.org/stable/1572419
  * Clark and Brennan [1992] H. Clark and S. Brennan, “Grounding in Communication,” _Archives_ , vol. 7, no. July, pp. 734–738, 1992.
  * Hoffman and Weinberg [2010] G. Hoffman and G. Weinberg, “Shimon: an interactive improvisational robotic marimba player,” in _CHI’10 Extended Abstracts on Human Factors in Computing Systems_ , 2010, pp. 3097–3102.
  * Farbood et al. [2007] M. Farbood, H. Kaufman, and K. Jennings, “Composing with Hyperscore: An Intuitive Interface for Visualizing Musical Structure,” 01 2007.
  * Thelle and Pasquier [2021] N. J. W. Thelle and P. Pasquier, “Spire Muse: A Virtual Musical Partner for Creative Brainstorming,” 4 2021, https://nime.pubpub.org/pub/wcj8sjee. [Online]. Available: https://nime.pubpub.org/pub/wcj8sjee
  * Murray-Browne and Tigas [2021] T. Murray-Browne and P. Tigas, “Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders,” 4 2021, https://nime.pubpub.org/pub/latent-mappings. [Online]. Available: https://nime.pubpub.org/pub/latent-mappings
  * Vigliensoni et al. [2020] G. Vigliensoni, L. McCallum, and R. Fiebrink, “Creating latent spaces for modern music genre rhythms using minimal training data,” in _International Conference on Computational Creativity (ICCC)_ , June 2020\. [Online]. Available: https://research.gold.ac.uk/id/eprint/29044/
  * Yang et al. [2019] R. Yang, D. Wang, Z. Wang, T. Chen, J. Jiang, and G. Xia, “Deep music analogy via latent representation disentanglement,” _arXiv preprint arXiv:1906.03626_ , 2019.
  * Wang et al. [2020] Z. Wang, Y. Zhang, Y. Zhang, J. Jiang, R. Yang, J. Zhao, and G. Xia, “Pianotree vae: Structured representation learning for polyphonic music,” _arXiv preprint arXiv:2008.07118_ , 2020.
  * Roberts et al. [2018] A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and D. Eck, “A hierarchical latent vector model for learning long-term structure in music,” in _International conference on machine learning_. PMLR, 2018, pp. 4364–4373.
  * Sturm et al. [2016] B. L. Sturm, J. F. Santos, O. Ben-Tal, and I. Korshunova, “Music transcription modelling and composition using deep learning,” _ArXiv_ , vol. abs/1604.08723, 2016.
  * Hadjeres et al. [2017b] G. Hadjeres, F. Nielsen, and F. Pachet, “GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures,” 2017\.
  * Lample et al. [2018] G. Lample, N. Zeghidour, N. Usunier, A. Bordes, L. Denoyer, and M. Ranzato, “Fader Networks: Manipulating Images by Sliding Attributes,” 2018\.
  * Tan and Herremans [2020] H. H. Tan and D. Herremans, “Music Fadernets: Controllable Music Generation Based on High-Level Features via Low-Level Feature Modelling,” 2020.
  * Toussaint [2002] G. Toussaint, “A mathematical analysis of African, Brazilian, and Cuban clave rhythms,” in _Townson University_ , 2002, pp. 157–168.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * Adel et al. [2018] T. Adel, Z. Ghahramani, and A. Weller, “Discovering interpretable representations for both deep generative and discriminative models,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 50–59. [Online]. Available: https://proceedings.mlr.press/v80/adel18a.html
  * Shneiderman et al. [2006] B. Shneiderman, G. Fischer, M. Czerwinski, M. Resnick, B. Myers, L. Candy, E. Edmonds, M. Eisenberg, E. Giaccardi, T. T. Hewett, P. Jennings, B. Kules, K. Nakakoji, J. Nunamaker, R. Pausch, T. Selker, E. Sylvan, and M. Terry, “Creativity Support Tools: Report From a U.S. National Science Foundation Sponsored Workshop,” _International Journal of Human-Computer Interaction_ , vol. 20, no. 2, pp. 61–77, 2006. [Online]. Available: https://doi.org/10.1207/s15327590ijhc2002{_}1
  * Chen et al. [2020] Z. Chen, Y. Bei, and C. Rudin, “Concept whitening for interpretable image recognition,” _Nature Machine Intelligence_ , vol. 2, no. 12, p. 772–782, Dec 2020. [Online]. Available: http://dx.doi.org/10.1038/s42256-020-00265-z
  * Bødker [2015] S. Bødker, “Third-wave HCI, 10 years later—participation and sharing,” _Interactions_ , vol. 22, no. 5, pp. 24–31, aug 2015. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2818696.2804405
  * McCormack et al. [2019] J. McCormack, T. Gifford, P. Hutchings, M. T. Llano Rodriguez, M. Yee-King, and M. d’Inverno, _In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound_. New York, NY, USA: Association for Computing Machinery, 2019, p. 1–11. [Online]. Available: https://doi.org/10.1145/3290605.3300268