# Renaissance Robot: Optimal Transport Policy Fusion
for Learning Diverse Skills

Julia Tan1, Ransalu Senanayake2 and Fabio Ramos3 *This work was not supported
by any organization1Julia Tan is with the School of Computer Science,
University of Sydney, Sydney, Australia<EMAIL_ADDRESS>Senanayake is with the Dept. of Computer Science, Stanford University,
Stanford, CA 94305, USA<EMAIL_ADDRESS>Ramos is with NVIDIA,
USA and School of Computer Science, University of Sydney, Australia
<EMAIL_ADDRESS>
###### Abstract

Deep reinforcement learning (RL) is a promising approach to solving complex
robotics problems. However, the process of learning through trial-and-error
interactions is often highly time-consuming, despite recent advancements in RL
algorithms. Additionally, the success of RL is critically dependent on how
well the reward-shaping function suits the task, which is also time-consuming
to design. As agents trained on a variety of robotics problems continue to
proliferate, the ability to reuse their valuable learning for new domains
becomes increasingly significant. In this paper, we propose a post-hoc
technique for policy fusion using _Optimal Transport_ theory as a robust means
of consolidating the knowledge of multiple agents that have been trained on
distinct scenarios. We further demonstrate that this provides an improved
weights initialisation of the neural network policy for learning new tasks,
requiring less time and computational resources than either retraining the
parent policies or training a new policy from scratch. Ultimately, our results
on diverse agents commonly used in deep RL show that specialised knowledge can
be unified into a “Renaissance agent”, allowing for quicker learning of new
skills.

##  I INTRODUCTION

Intelligent systems are increasingly penetrating into real-world applications,
incurring the need for sophisticated models that can perform complex and
adaptive tasks. Deep neural networks are already ubiquitous in solving
classification problems in many fields of interest. Recent advancements in
reinforcement learning (RL) have extended the capacity of neural networks into
the physical domain, allowing for intelligent control of robotic agents in a
dynamic environment. Accordingly, RL in robotics has the potential to advance
key industries, from manufacturing [1] to healthcare [2], agriculture [3] or
transportation [4].

While RL is progressively being used in a range of robotics problems, most
real-world applications require numerous specialised capabilities. The
computational resource constraints associated with sample-inefficient
training, combined with the complexity of re-designing the appropriate reward
structures for each capability, make it difficult to achieve solutions for
varying goals in a reasonable timeframe. Traditional methods of transfer
learning are also inadequate, without a robust way to determine which parts of
a policy are still relevant in the new context. Even with targeted policy
reuse, the knowledge of only one agent can be leveraged. This is likely not
sufficient for most cases, since small differences in the environment or the
kinematics of the robotic agent itself can drastically change the rewards of
previously familiar actions. Consequently, we require a minimal dependency
fusion method that can effectively consolidate the learned behaviours of
multiple parent policies.

Figure 1: First, two policy networks are trained independently on an RL
robotics task, where either the agent or environment has been altered in
distinct ways (e.g. for the Gym environment Walker2d, using primarily right
joints vs left joints). Next, these policy networks are fused post-hoc to
produce an intialisation of weights for the Renaissance agent. Finally, the
Renaissance agent is refined on a new domain (e.g. walking using all joints)
to learn new skills beyond the original capabilities of the parent agents.

In this paper, we present a technique to fuse RL policies represented as
neural networks using Optimal Transport (OT), and propose the fused policy as
a superior initialisation of weights for faster learning of diverse skills.
With reference to the Renaissance man, we call this fusion result a
“Renaissance agent”.

In particular, our contributions are:

  1. 1.

A method based on OT for efficiently combining RL policies post-hoc, requiring
no awareness of the original data used to train them;

  2. 2.

A demonstration that this OT fusion approach generalises the prior knowledge
of parent policies over different environments as well as different robot
kinematics, and

  3. 3.

Validation that the Renaissance agent learns new skills more effectively and
within a shorter time period than either retraining the parent agents or
training a new agent from scratch.

##  II RELATED WORK

Information fusion is a widely studied topic in robotics for decades [5]. As
opposed to the common paradigm of controlling multiple robots using a
centralised controller [6] or fusing data from various sources [7], our
objective is to gather data from individually operating robots each with a
different skill to train a single robot that possesses diverse skills
appropriately learned from all other robots by fusing knowledge. Roboticists
have attempted to perform knowledge fusion at the perception stage or
decision-making stage of the robot autonomy stack. Most research studies in
the perception level focuses on either fusing data from multiple sensors or at
the perception algorithm level. The former, sensor fusion, is a well-studied
topic in robotics [5, 7] and has applications ranging from field robotics [5,
7] to wearable sensors [8]. There are also studies on aggregating data from
multiple robots [9] as the simplest fusion technique. Some of the main
limitations of such aggregation methods include the requirement of saving the
ever growing dataset and training the model online.

While there are many techniques for data fusion, fewer exist for fusing
learned perception models. This is because it is more difficult to determine
how and what to fuse for neural networks. Zhi et al. [10] has proposed a
technique to fuse Bayesian Hilbert maps from data gathered from various
agents. However, it is not clear how the proposed occupancy map fusion
technique can be used when the underlying model is a neural network.

Researchers have studied model fusion techniques for various applications
beyond robotics. The most common technique, ensembling [11, 12] requires all
models to be running concurrently, which is computationally expensive.
Moreover, it does not actually aggregate each model’s distinct capabilities,
since decision making occurs separately. Hence, it is not suitable for
adapting to customised tasks, especially when the learned behaviours of the
parent models differ significantly. In contrast, there are models that result
in a single model by transferring knowledge from a teacher network [13].
Whilst there are also techniques [14] that yield a combined model without the
need for retraining, they also do not perform empirically better than naive
averaging. More recently, in federated learning settings, aligning neural
networks has been studied [15]. In such frameworks, the number of parameters
required grows with the size of the data, so it is infeasible for fusing large
data models ubiquitous in robotics. More recently, others have used OT theory
to fuse neural network models [16], although this has not yet been applied to
RL or decision-making. When fusing robots, it is ideal if we can make them as
agnostic as possible to sensory settings because the ultimate goal of a robot
is to maneuver safely and efficiently.

On the other end of the fusion spectrum, it is possible to combine decision-
making modules [17] or their outputs, though the latter, by construction, is
ill-informed about data and typically produces less accurate results.
Researchers have investigated how to combine the outputs of a traditional
hand-crafted controller with a deep RL controller using Bayesian inference
[18]. However, it is not clear how to combine two or more neural network
controllers, especially at the decision-making stage rather than once the
outputs are produced. In our work, we fuse architecturally similar deep RL
controllers that possess different skills. Though Dax et al. [17] also
discusses sensor selection and decision fusion, their proposed model also does
not represent policies using neural networks.

##  III OPTIMAL TRANSPORT POLICY FUSION

###  III-A Policy Networks in RL

In RL, an agent is informed by a policy
$\pi:\mathcal{A}\times\mathcal{S}\to[0,1]$ that determines what actions
$\mathcal{A}$ to perform given the current states $\mathcal{S}$. A policy
$\pi_{\theta}^{\mathcal{E}}$, trained in domain $\mathcal{E}$, can be
represented as a neural network parameterized by $\theta$. When training a
policy, an RL algorithm such as Proximal Policy Optimisation (PPO) [19] or
Deep Deterministic Policy Gradient (DDPG) [20] is used to update the weights
of the neural network so that the cumulative reward is maximised over time.
Depending on the task configuration during training, these learned weights
will differ between agents. However, for policy networks that have the same
input features, similar weight vectors at each individual layer may represent
common learned patterns. Therefore, we wish to find an efficient way to
consolidate the common patterns while diminishing any over-optimised patterns,
so that we can generate a Renaissance agent that is capable of leveraging
prior knowledge for new domains.

For two policy networks,
$\pi_{\theta_{1}}^{\mathcal{E}_{1}},\pi_{\theta_{2}}^{\mathcal{E}_{2}}$,
trained independently in domains, $\mathcal{E}_{1},\mathcal{E}_{2}$,
respectively, our objective is to determine the best fusion operation
$c:\Pi^{\mathcal{E}_{1}}\times\Pi^{\mathcal{E}_{2}}\to\Pi^{\mathcal{E}_{*}}$
for the policies $\pi^{\mathcal{E}}\in\Pi^{\mathcal{E}}$ and new domain
${\mathcal{E}_{*}}$ such that empirically
$r(\pi_{\theta_{*}}^{\mathcal{E}_{*}})>r(\pi_{\theta_{1}}^{\mathcal{E}_{*}}),r(\pi_{\theta_{2}}^{\mathcal{E}_{*}})$,
where $r(\pi^{\mathcal{E}})$ is the reward when policy $\pi$ is deployed in
domain $\mathcal{E}$. For instance, naive policy averaging
$c(\pi_{\theta_{1}}^{\mathcal{E}_{1}},\pi_{\theta_{2}}^{\mathcal{E}_{2}})=(\pi_{\theta_{1}}^{\mathcal{E}_{1}}+\pi_{\theta_{2}}^{\mathcal{E}_{2}})/2$
is a simple fusion operation. However, it is ineffective since it does not
find the optimal alignment between the parameter vectors of the policies. In
this paper, we propose that OT theory is apt for finding an alignment map
between the weights of two policy networks. This enables us to match the
weights that are most likely to represent common patterns.

###  III-B Optimal Transport Theory

####  III-B1 Monge-Kantorovich Problem

OT theory has previously been applied to mapping problems in order to allow
robots to operate in new environments [21]. However, in this work, we utilise
OT for policy fusion so that robotic agents can consolidate learned behaviours
for new domains, including new environments as well as new kinematics.
Typically, OT is used to transform a single source distribution $\mathcal{S}$
to a target distribution $\mathcal{T}$ [22]. When applied to the incoming
weight vectors of a policy network, OT represents them as distributions of
Dirac “masses” [23]. The “distance” between any two distributions is then
quantified by the least costly way of transporting each mass from
$\mathcal{S}$ to $\mathcal{T}$.

According to the Monge-Kantorovich Problem [24], for the datasets
$\bm{\mu}^{(\mathcal{S})}$ and $\bm{\mu}^{(\mathcal{T})}$ of size
$N^{\mathcal{S}}$ and size $N^{\mathcal{T}}$ on independent metric spaces
$\Omega^{(\mathcal{S})}$ and $\Omega^{(\mathcal{T})}$, there exists an optimal
probabilistic coupling $P_{*}\in\mathbb{R}^{N^{(\mathcal{S})}\times
N^{(\mathcal{T})}}$ between the two datasets. This optimal coupling always
exists for a distance function
$D:\Omega^{(\mathcal{S})}\times\Omega^{(\mathcal{T})}\rightarrow[0,\infty)$,
and is defined by,

$P_{*}=\small{\mathop{\text{arginf}}_{P\in\Gamma}}\int_{\Omega^{(\mathcal{S})}\times\Omega^{(\mathcal{T})}}D(\bm{\mu}^{(\mathcal{S})},\bm{\mu}^{(\mathcal{T})})\text{d}P(\bm{\mu}^{(\mathcal{S})},\bm{\mu}^{(\mathcal{T})}).$
(1)

####  III-B2 Earth Mover’s Distance

The minimum cost of transporting each mass from $\mathcal{S}$ to $\mathcal{T}$
is based on the Wasserstein metric, also known as the Earth Mover’s Distance
[25],

$\text{EMD}(\mathcal{S},\mathcal{T})=\frac{\Sigma_{i=1}^{m}\Sigma_{j=1}^{n}P_{*_{i,j}}d_{i,j}}{\Sigma_{i=1}^{m}\Sigma_{j=1}^{n}P_{*}{{}_{i,j}}},$
(2)

where $P_{*}$ is the optimal coupling and $d_{i,j}$ is the ground distance
between distributions $\mathcal{S}_{i}$ and $\mathcal{T}_{j}$. We can
formulate OT as an alignment problem by considering “similarity” as inversely
proportional to the Earth Mover’s Distance. By computing the optimal coupling
$P_{*}$ between two sets of weight vectors (each representing neurons from a
different policy network), we are able to match the second set as the source
distribution $\mathcal{S}$ to the first set as the target distribution
$\mathcal{T}$ by taking the dot product,

$\mathcal{S}_{\text{aligned}}=P_{*}\cdot\mathcal{T}.$ (3)

###  III-C Neural Matching and Weights Fusion

In order to fuse two parent policies parameterised by neural networks, we
first align their neurons layer-wise so that the associated weight vectors are
matched. Given that the input layers of each model receive the exact same
format of input vectors, we begin neural matching from the incoming weights to
the first hidden layer. As we do not know explicitly what each neuron has been
trained to identify, we can pair them together based only on the “similarity”
of the incoming weights.

A one-to-one similarity is computed by minimising the Earth Mover’s Distance
between the weight vectors in one model and the weight vectors in the other
model as probability distributions. Hence, the optimal coupling $P_{*}$
represents the alignment matrix for re-positioning the weight vectors in the
second model (i.e. the source distribution $\mathcal{S}$) according to the
weight vectors in the first model (i.e. the target distribution
$\mathcal{T}$). Taking the dot product of $P_{*}$ and $\mathcal{T}$ yields the
aligned $\mathcal{S}$.

To perform neural matching in the subsequent layer, we must start with
rearranging the elements of each weight vector to correspond with the new
positions of the neurons in the previous layer (after alignment). For the
current layer, we then compute the OT map as before and align the weight
vectors in the second model to the weight vectors in the first model.
Algorithm 1 outlines this process. Once every layer has been aligned, we then
fuse the matched weights across the two policies by averaging the values. This
keeps the weights with similar matches close to the same value, while changing
those that are dissimilar and therefore unlikely to represent common patterns.
Thus, we build the fused policy layer-by-layer, as shown in Figure 2.

Algorithm 1 _align_neurons_

Input: weights of $N$-layer parent policies

for $i=2$ to $N$ do

$\mathcal{S}\leftarrow$ incoming weights to $i$-th layer of second policy

$\mathcal{T}\leftarrow$ incoming weights to $i$-th layer of first policy

if $i>2$ then

Rearrange $\mathcal{S}$ to match positions of previous layer alignment

end if

Compute OT map $P_{*}$, according to Equation 1

Align source weights to target weights using $P_{*}$, according to Equation 3

end for

Output: aligned weights of $N$-layer parent policies

Figure 2: Our approach considers the incoming weight vectors of each
independently trained policy $\pi_{\theta_{1}}^{\mathcal{E}_{1}}$ and
$\pi_{\theta_{2}}^{\mathcal{E}_{2}}$. The weight vectors are considered as
source $\mathcal{S}$ and target $\mathcal{T}$ distributions for OT. The result
at each layer is taken as the weight vector for the fused policy.

##  IV EXPERIMENTS

TABLE I: Neural network architecture for each model.

| MNIST Binary Classifier | Reacher Task | Walker2d Task | HalfCheetah Task | FetchReach Task
---|---|---|---|---|---
Layer | Nodes | Weights | Nodes | Weights | Nodes | Weights | Nodes | Weights | Nodes | Weights
Input | 784 | - | 9 | - | 22 | - | 26 | - | 16 | -
Hidden 1 | 32 | 32 $\times$ 784 | 64 | 64 $\times$ 9 | 64 | 64 $\times$ 22 | 64 | 64 $\times$ 26 | 64 | 64 $\times$ 16
Hidden 2 | - | - | 64 | 64 $\times$ 64 | 64 | 64 $\times$ 64 | 64 | 64 $\times$ 64 | 64 | 64 $\times$ 64
Output | 2 | 2 $\times$ 32 | 2 | 2 $\times$ 64 | 6 | 6 $\times$ 64 | 6 | 6 $\times$ 64 | 4 | 4 $\times$ 64

###  IV-A Proof of Concept: Image Classification Task

####  IV-A1 Experimental Setup

To validate the theoretical concept behind the proposed OT fusion method, we
first test it on a image classification task before extending to robotics
tasks in RL. The task is constructed using the MNIST handwritten digits
dataset [26]. We independently train two parent models to be binary
classifiers on the MNIST dataset. The first model classifies 0s versus all
other digits, while the second model classifies 1s versus all other digits.
Hence, we modify two distinct training datasets, where the class labels in
each correspond to either “yes” for the digit to be identified or “no” for all
other digits.

There are originally 60,000 images in the training dataset. However, since the
original MNIST classification problem is to identify each digit from 0 to 9,
we need to rebalance the dataset in order to achieve good training outcomes
for the binary classifiers. Otherwise, the models will be biased towards the
class “no”, since the majority of training examples are not the digit to be
identified. We use a random undersampler to rebalance the dataset so that the
majority class (i.e. “no”) will have twice as many samples as the minority
class (i.e. “yes”).

Each parent model will have “fully-connected” layers according to the neural
network architecture shown in Table I. Both models will be trained for 10
epochs on batch sizes of 128 with a 10% validation split. Evaluating on the
testing dataset of 10,000 examples (with the labels similarly modified), we
get the test accuracy and test loss of each model shown in Table II.

TABLE II: Evaluation results for trained parent models. Model | Test accuracy (%) | Test loss (%)
---|---|---
0s | 98.1 | 10.7
1s | 99.0 | 8.32

We now reformulate the problem domain so that neither one of the parent models
are able to perform well without retraining. The new task that we aim to solve
is classifying both 0s and 1s versus all other digits. We fuse the parent
models by applying the OT technique explained in subsection III-C.

####  IV-A2 Results

Without additional retraining, the fused model is immediately able to classify
some digits correctly but not all. An example subset of the fused model’s
predictions is shown in Figure 3.

Figure 3: Example predictions of fused model without retraining, showing false
negatives in red.

The fused model performs significantly better after retraining, correctly
classifying most examples. This is evident from the confusion matrix in Figure
4, showing that the fused model mainly predicts true positives and true
negatives.

| Predicted label |
---|---|---
| Yes | No |
True label | Yes | $0.99$ | $0.01$ |
No | $0.07$ | $0.93$ |
Figure 4: Normalised confusion matrix for fused model after retraining, as
evaluated on testing dataset of 10,000 examples.

To evaluate the effectiveness of the fused model, we compare it with
retraining the parent models as well as training a new model from scratch. The
training accuracy and validation accuracy for each of these models over 10
training epochs are shown in Figure 6. It is evident that the fused model
reaches high consistent performance much faster than the other models,
surpassing a training accuracy of 95% and a validation accuracy of 90% after
only 2 epochs. In contrast, it takes at least 4 epochs for the next best model
to converge to similar performance.

(a) Reacher

(b) Walker2d

(c) HalfCheetah

(d) FetchReach

Figure 5: Simulation environments of RL tasks we are using in this work.

(a) Training accuracy.

(b) Validation accuracy.

Figure 6: Accuracy metrics over 10 retraining epochs. TABLE III: Evaluation results after 10 retraining epochs. Model | Final test accuracy (%) | Final test loss (%)
---|---|---
New | 97.5 | 9.75
Fused | 98.3 | 7.96
Original 0s | 97.9 | 9.04
Original 1s | 97.9 | 10.1

Moreover, the final performance of the fused model remains visibly higher than
the other models after all 10 epochs have finished. From Table III, we can see
that the fused model exhibits the highest final test accuracy as well as the
lowest final test loss, as evaluated on the testing dataset of 10,000
examples.

###  IV-B Experimental Setup: RL Tasks

TABLE IV: Mean episode rewards for each trained parent policy over 10
evaluation episodes.

Parent Policy | Reacher Task | Walker2d Task | HalfCheetah Task | FetchReach Task
---|---|---|---|---
First | 20.10 $\pm$ 6.80 | 660.80 $\pm$ 420.96 | 2258.43 $\pm$ 642.28 | -2.69 $\pm$ 0.19
Second | 19.57 $\pm$ 10.06 | 728.28 $\pm$ 340.60 | 2013.14 $\pm$ 8.02 | -1.48 $\pm$ 0.07

To test the effectiveness of the OT technique for policy fusion, we construct
four separate RL robotics tasks using OpenAI Gym [27]: Reacher, Walker2d,
HalfCheetah and FetchReach. The first three tasks are simulated using PyBullet
[28] while the last task is simulated using Mujoco [29]. Example renders are
shown in Figure 5. For each task, the policy of the robotic agent is
represented as a Multilayer Perceptron (MLP) with 2 hidden layers of 64 nodes
each. The size of the input and output layers differ across the tasks
according to Table I, since the environments have varying observation and
action spaces. All policies will be trained using the PPO algorithm.

To test each task, we will first independently train two robotic agents on
different configurations of the task. Both agents will be trained for 500,000
timesteps. The mean episode rewards, as evaluated on 10 separate and randomly
generated environments, are included in Table IV. These agents will act as the
“parents” of the Renaissance agent, which we will generate via the OT policy
fusion method. Then, we will refine the Renaissance agent on a new domain, and
compare its performance against retraining the parent agents and training a
brand new agent over the same time period. To provide an additional benchmark
for OT, we will also compare it against naive averaging as an alternative
method of fusing policy weights.

###  IV-C Generalising over Changes in the Environment

####  IV-C1 Experimental Setup

The learned behaviour of a robotic agent is highly sensitive to changes in the
environment, particularly when such changes impact how rewards are earned. To
demonstrate that the OT fusion technique is able to generalise the knowledge
of two agents trained in differing environment configurations, we construct a
simple RL task using the Gym environment Reacher. This task involves a two-
link robotic arm as the agent, which operates within a two-dimensional
workspace. The goal is for the end-effector to reach to a randomly located
target position, with the agent gaining 0.1 rewards for each timestep that it
does so. The input layer of the agent’s policy network takes in the angular
positions and velocities of the robotic arm, and the output layer defines the
motor torque of the central joint and the elbow joint.

The first policy learns to reach to targets randomly located in the first
quadrant of the workspace, while the second policy learns to reach to targets
randomly located in the second quadrant of the workspace. We now reformulate
the problem scope to targets that are placed randomly anywhere within the
first and second quadrants. Since the required kinematics of the two-link
robotic arm reaching to targets in each quadrant differ significantly, the
parent policies have limited capabilities in this new domain. We fuse the
parent policies using the OT technique to produce an initialisation of weights
for the Renaissance agent.

####  IV-C2 Results

After retraining on the new domain for 500,000 timesteps, the fused agent
exhibits consistent performance. Renders of the fused agent over a random
evaluation episode are shown in Figure 7.

Figure 7: Renders of fused agent after retraining on new Reacher task.

The mean episode rewards for the fused agent, compared with the other
benchmark agents, are shown in Figure 8. It is evident that the fused agent
achieves high consistent performance much faster than the rest, surpassing 15
rewards per episode after 200,000 timesteps. In contrast, it takes
approximately 350,000 timesteps for the next best agent to converge to similar
performance. Upon completion of the retraining period, we can see from Table V
that the fused agent exhibits the highest mean reward as well as the lowest
standard deviation, as tested over 10 random evaluation episodes.

Figure 8: Retraining curves for Reacher environment. TABLE V: Mean episode
rewards over 10 evaluation episodes.

Policy | Reacher Task | Walker2d Task | HalfCheetah Task | FetchReach Task
---|---|---|---|---
New Agent | 16.64 $\pm$ 9.51 | 972.02 $\pm$ 5.88 | 2244.47 $\pm$ 898.70 | -5.38 $\pm$ 4.03
Fused Agent | 17.05 $\pm$ 7.37 | 1520.59 $\pm$ 232.58 | 2810.22 $\pm$ 29.93 | -5.05 $\pm$ 3.4
Original Agent 1 | 15.17 $\pm$ 8.03 | 1000.34 $\pm$ 24.44 | 2750.57 $\pm$ 52.82 | -7.13 $\pm$ 3.73
Original Agent 2 | 16.59 $\pm$ 7.92 | 822.42 $\pm$ 345.61 | 2500.23 $\pm$ 444.02 | -5.95 $\pm$ 3.69
Naive Averaging | 15.09 $\pm$ 8.08 | 580.29 $\pm$ 420.92 | 1853.18 $\pm$ 1312.92 | -8.03 $\pm$ 3.47

###  IV-D Generalising over Changes in Robot Kinematics

####  IV-D1 Experimental Setup

Agents with different robot kinematics will learn different behaviours to
reach the desired goal, even if the environment is the same. To demonstrate
that the Renaissance agent generalises over different robot kinematics, we
construct two RL tasks using the Gym environments Walker2d and HalfCheetah.

####  IV-D2 Walker2d

The Walker2d task involves a bipedal robot as the agent, which operates within
a 2D plane (i.e. can only move forwards or backwards). The goal is for the
agent to walk forward as fast as possible, with the agent gaining rewards as a
function of the current forward velocity, plus a constant bonus for not
falling over. The input layer of the agent’s policy network takes in the
angular positions and velocities of the joints, and the output layer defines
the signals to move the torso-connected foot and the left foot.

The first agent is trained with a more powerful torso-connected foot
(coefficient = 40) compared to its left foot (coefficient = 10), while the
second agent is trained with a more powerful left foot (coefficient = 25)
compared to its torso-connected foot (coefficient = 20). We now fuse the
parent policies and reformulate the target domain to learning the Walker2d
task on an agent with equally powerful torso-connected and left feet
(coefficient = 30).

After retraining on the new domain for 500,000 timesteps, the fused agent is
able to walk forward in a natural-looking manner. Renders of the agent over a
random evaluation episode are shown in Figure 9.

Figure 9: Renders of fused agent after retraining on new Walker2d task.

The mean episode rewards for the fused agent, compared with the other
benchmark agents, are shown in Figure 10. We can see that the fused agent is
the only one that achieves high consistent performance, surpassing 1000
rewards per episode by the end of the retraining period. From Table V, it is
evident that the fused agent exhibits the highest minimum, maximum and mean
reward, as tested over 10 random evaluation episodes.

Figure 10: Retraining curves for Walker2d environment.

####  IV-D3 HalfCheetah

We further demonstrate the ability of OT policy fusion to generalise over
different robot kinematics by extending to a more complex agent. The
HalfCheetah task involves a quadruped robot as the agent, which also moves
only forwards or backwards within a 2D plane. Similar to the Walker2d task,
the goal is for the agent to run forward as fast as possible. The agent’s
policy network determines how much torque to apply on its thighs, shins and
feet.

We train the first parent policy on an agent with greater thigh and shin
power, but reduced foot power. This results in more rigid foot joints. We then
train the second parent policy on an agent with the opposite kinematics,
resulting in more rigid thigh and shin joints. Similar to the Walker2d task,
we now fuse the policies and change the target domain to learning the
HalfCheetah task on an agent with neutral power for all joints. This should
allow the agent to learn joint movements that are less rigid.

After retraining on the new domain for 500,000 timesteps, the fused agent is
able to run forward without any of the joint rigidity displayed by the parent
agents. Renders of the agent over a random evaluation episode are shown in
Figure 11.

Figure 11: Renders of fused agent after retraining on new Cheetah task.

The mean episode rewards for the fused agent, compared with the other
benchmark agents, are shown in Figure 12. It is evident that the fused agent
learns much faster than the new agent, and overtakes the parent agents by the
end of the retraining period. From Table V, we can further observe that the
fused agent exhibits the highest mean reward and lowest standard deviation, as
tested over 10 random evaluation episodes.

Figure 12: Retraining curves for HalfCheetah environment.

Additionally, the retrained parent agents still exhibit very similar
kinematics to what they learned from their original training domain. In
contrast, both the fused agent and the new agent display the desired
kinematics of the new domain, which has less joint rigidity. This demonstrates
how the OT fusion technique is useful for consolidating the common knowledge
of parent models which have been over-optimised to a previous task.

###  IV-E Evaluating on Robotic Arm Task

####  IV-E1 Experimental Setup

The “Fetch” Gym environments are based on the 7-DoF Fetch Robotics arm. This
provides a more complex agent, observation and action space compared to the
Reacher task shown in subsection IV-C. We extend that task by using the Gym
environment FetchReach.

The FetchReach task involves the Fetch arm as the agent, which operates within
a 3D space. The goal is for the agent to move its end-effector to target
positions randomly located above a workspace. Rewards are binary: at each
timestep, the agent obtains a reward of 0 if its end-effector is at the target
position, and -1 otherwise. The agent’s policy network takes in the Cartesian
positions and linear velocities of the arm links and gripper as observations,
and outputs the desired movements in Cartesian coordinates as actions.

In the previous 2-link Reacher task, we set the target locations to be within
the first or second quadrants of the workspace. However, since the Fetch robot
stands above the workspace and moves in 3D, these existing problem
formulations are not different enough to ensure that the parent policies will
learn distinct behaviours. Instead, the first policy learns to move the end-
effector to target positions located over the bottom left of the workspace,
while the second policy learns for the top right of the workspace. We now fuse
the policies and change the problem scope to target positions that are located
randomly anywhere over the workspace, excluding the corners.

Figure 13: Renders of fused agent after retraining on new FetchReach task.

####  IV-E2 Results

After retraining on the new domain for 200,000 timesteps, the fused agent is
able to quickly and consistently move its end-effector to target positions
located anywhere over the central area of the workspace. Renders of the agent
over a random evaluation episode are shown in Figure 13.

The mean episode rewards for the fused agent, compared with the other
benchmark agents, are shown in Figure 14. It is evident that the fused agent
achieves high consistent performance much faster than the rest, surpassing
-2.5 rewards per episode by 25,000 timesteps. In contrast, it takes
approximately 100,000 timesteps for the next best agent to converge to similar
performance. From Table V, we can see that the fused agent achieves the
highest mean reward as well as the lowest standard deviation, as tested over
10 random evaluation episodes.

Figure 14: Retraining curves for FetchReach environment.

##  V CONCLUSION

In this work, we developed policy fusion utilising OT theory to create
Renaissance agents. We demonstrated that with minimal retraining, the
Renaissance agents can quickly learn a new domain by leveraging generalised
prior knowledge. In contrast, the parent agents are often over-optimised to
their original tasks, and thus less adept at learning diverse skills in a
variety of common RL robotics problems. Ultimately, the fused agents were
empirically demonstrated to learn new domains significantly faster and achieve
superior sustained performance compared with other benchmarks. This is
especially valuable for RL in robotics since training is highly time-consuming
and resource expensive. In light of the simulation-to-simulation examples
discussed in this work, applying the proposed technique to simulation-to-real
policy fusion is a promising future direction.

## ACKNOWLEDGMENT

We thank Anthony Tompkins for early discussions and ideas, especially on
Optimal Transport theory.

## References

  * [1] T. Inoue, G. D. Magistris, A. Munawar, T. Yokoya, and R. Tachibana, “Deep reinforcement learning for high precision assembly tasks,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 819–825, 2017.
  * [2] P. M. Scheikl, B. Gyenes, T. Davitashvili, R. Younis, A. Schulze, B. P. Müller-Stich, G. Neumann, M. Wagner, and F. Mathis-Ullrich, “Cooperative assistance in robotic surgery through multi-agent reinforcement learning,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1859–1864, 2021.
  * [3] F. Yandun, T. Parhar, A. Silwal, D. Clifford, Z. Yuan, G. Levine, S. Yaroshenko, and G. Kantor, “Reaching pruning locations in a vine using a deep reinforcement learning policy,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2400–2406, 2021.
  * [4] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine, “Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5129–5136, 2018.
  * [5] H. F. Durrant-Whyte, “Sensor models and multisensor integration,” in Autonomous robot vehicles, pp. 73–89, Springer, 1990.
  * [6] P. K. Sharma, R. Fernandez, E. Zaroukian, M. Dorothy, A. Basak, and D. E. Asher, “Survey of recent multi-agent reinforcement learning algorithms utilizing centralized training,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, vol. 11746, p. 117462K, International Society for Optics and Photonics, 2021.
  * [7] M. B. Alatise and G. P. Hancke, “A review on challenges of autonomous mobile robot and sensor fusion methods,” IEEE Access, vol. 8, pp. 39830–39846, 2020.
  * [8] D. Novak and R. Riener, “A survey of sensor fusion methods in wearable robotics,” Robotics and Autonomous Systems, vol. 73, pp. 155–170, 2015\.
  * [9] M. Guo and M. M. Zavlanos, “Multirobot data gathering under buffer constraints and intermittent communication,” IEEE transactions on robotics, vol. 34, no. 4, pp. 1082–1097, 2018.
  * [10] W. Zhi, L. Ott, R. Senanayake, and F. Ramos, “Continuous occupancy map fusion with fast bayesian hilbert maps,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 4111–4117, IEEE, 2019.
  * [11] D. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, 12 1992.
  * [12] R. E. Schapire, “A brief introduction to boosting,” 1999 International Joint Conference on Artificial Intelligence (IJCAI), Morgan Kaufmann Publishers Inc., 1999.
  * [13] J. Schmidhuber, “Learning complex, extended sequences using the principle of history compression,” Neural Computation, vol. 4, no. 2, pp. 234–242, 1992\.
  * [14] J. Utans, “Weight averaging for neural networks and local resampling schemes,” in Proc. AAAI-96 Workshop on Integrating Multiple Learned Models, AAAI Press, 1996.
  * [15] M. Yurochkin, M. Agarwal, S. S. Ghosh, K. H. Greenewald, T. N. Hoang, and Y. Khazaeni, “Bayesian nonparametric federated learning of neural networks,” in 2019 International Conference on Machine Learning (ICML), 2019.
  * [16] S. P. Singh and M. Jaggi, “Model fusion via optimal transport,” in 2020 Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [17] V. M. Dax, M. Kochenderfer, R. Senanayake, and U. Ibrahim, “Infrastructure-enabled autonomy: An attention mechanism for occlusion handling,” in International Conference on Robotics and Automation (ICRA), 2022.
  * [18] K. Rana, V. Dasagi, J. Haviland, B. Talbot, M. Milford, and N. Sünderhauf, “Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics,” arXiv preprint arXiv:2107.09822, 2021.
  * [19] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” Computing Research Repository (CoRR), vol. abs/1707.06347, 2017.
  * [20] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML, p. I–387–I–395, JMLR.org, 2014.
  * [21] A. Tompkins, R. Senanayake, and F. Ramos, “Online domain adaptation for occupancy mapping,” in Robotics: Science and Systems (RSS), 2020.
  * [22] G. Peyré and M. Cuturi, “Computational optimal transport: With applications to data science,” Foundations and Trends in Machine Learning, vol. 11, no. 5-6, pp. 355–607, 2019.
  * [23] N. Bonneel, G. Peyré, and M. Cuturi, “Wasserstein barycentric coordinates: Histogram regression using optimal transport,” ACM Trans. Graph., vol. 35, July 2016.
  * [24] S. T. Rachev, “The monge–kantorovich mass transference problem and its stochastic applications,” Theory of Probability & Its Applications, vol. 29, no. 4, pp. 647–676, 1985.
  * [25] W. Li, S. Osher, and W. Gangbo, “A fast algorithm for earth mover’s distance based on optimal transport and l1 type regularization,” 2016.
  * [26] Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010.
  * [27] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016.  cite arxiv:1606.01540.
  * [28] B. Ellenberger, “Pybullet gymperium.” https://github.com/benelot/pybullet-gym, 2018–2019.
  * [29] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control.,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5026–5033, IEEE, 2012.