Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts

Onur Celikyyy
Aleksandar Taranovicyyy
Gerhard Neumannyyy

yyyAutonomous Learning Robots, Karlsruhe Institute of Technology Karlsruhe, Germany

Onur<EMAIL_ADDRESS>
Machine Learning, ICML

Reinforcement learning (RL) is a powerful approach for acquiring a good-performing policy. However, learning diverse skills is challenging in RL due to the commonly used Gaussian policy parameterization. We propose Diverse Skill Learning (Di-SkilL), an RL method for learning diverse skills using Mixture of Experts, where each expert formalizes a skill as a contextual motion primitive. Di-SkilL optimizes each expert and its associate context distribution to a maximum entropy objective that incentivizes learning diverse skills in similar contexts. The per-expert context distribution enables automatic curricula learning, allowing each expert to focus on its best-performing sub-region of the context space. To overcome hard discontinuities and multi-modalities without any prior knowledge of the environment's unknown context probability space, we leverage energy-based models to represent the per-expert context distributions and demonstrate how we can efficiently train them using the standard policy gradient objective. We show on challenging robot simulation tasks that Di-SkilL can learn diverse and performant skills.

§ INTRODUCTION

Solving tasks in diverse manners enables agents to better adapt to unknown and challenging situations. This diverse skill set is beneficial in many scenarios, such as playing table tennis, where applying different strikes (e.g. backhand, forehand, or smashing) to similar incoming balls is advantageous because the strike is less predictable for the opponent. Similarly, in scenarios with environmental changes where learned skills might be infeasible over time (e.g. grasping an object while avoiding obstacles), diverse skills provide additional adaptivity by discarding these invalid skills and relying on alternatives. This property makes them superior because complete relearning of skills is avoided.

Acquiring these diverse skill sets requires learning a policy that can represent multi-modality in the behavior space. Recent advances in supervised policy learning have demonstrated the potential of training high-capacity policies capable of capturing multi-modal behaviors [Jia et al., 2024, Shafiullah et al., 2022, Blessing et al., 2023, Chi et al., 2023]. These policies have exhibited remarkably diverse skills and outperformed state-of-the-art methods. However, in cases where no expert data is available or data collection is expensive, Reinforcement Learning (RL) is essential to acquire skills. Discovering multi-modal behaviors using RL is challenging since the policies usually rely on Gaussian parameterization and thus can only discover a single behavior.

We consider training agents that possess diverse skills from which they can select to tackle a specific task differently.
For capturing these multi-modalities in the agent's behavior space, we employ highly non-linear Mixture of Expert policies. We use automatic curriculum learning for efficient learning, enabling each expert to focus on a specific sub-region of the context space it favors. We introduce this curriculum shaping by optimizing for an additional per-expert context distribution that is used to sample contexts from the preferred regions to train the corresponding expert. Automatic curriculum learning has proven to increase performance by improving the exploration of agents, particularly in sparse-rewarded environments [Klink et al., 2022].

We explore Contextual Reinforcement Learning in which a continuous-valued context describes the task [Kupcsik et al., 2013]. In the example of robot table tennis (see Fig. <ref>), a context includes the desired ball landing positions on the opponent's table side as well as physical aspects, such as the incoming ball's velocity or friction properties. In continuous context spaces, the curriculum shaping per-expert context distributions are often parameterized as Gaussian [Klink et al., 2020, Celik et al., 2022]. However, the agent is usually unaware of the context bounds, which makes additional techniques necessary to constrain the distribution updates to stay within the context region [Celik et al., 2022]. Instead, we employ energy-based per-expert context distributions, which can be evaluated for any context and effectively represent multi-modality in the context space. Importantly, our model is trained solely using context samples from the environment that are inherently valid. Our approach eliminates the need for additional regularization of the context distribution and does not require prior knowledge about the environment. Due to the overlapping probability distributions of different per-expert contexts, our resulting mixture policy offers diverse solutions for the same context with a high probability.

Recent research in RL has explored mixture of experts policies, but often these methods either train the mixture in unsupervised RL settings and then select the best-performing expert in the downstream task [Laskin et al., 2021, Eysenbach et al., 2019] or train linear experts, limiting their performance [Daniel et al., 2012, Celik et al., 2022]. Our inspiration draws from recent advancements that have achieved diverse skill learning with a similar objective. However, their approach involves linear expert models with Gaussian context distributions. It requires prior knowledge of the environment to design a penalty term when the algorithm samples contexts outside the environment's bounds. These factors restrict the algorithm's performance and applicability when defining the context bounds requires knowledge, such as forward kinematics in robotics.

To summarize, we introduce in this paper a novel RL method for learning a mixture of experts policy that we refer to as Di-SkilL – Diverse Skill Learning. Our method can generalize to the continuous range of contexts defined by the (unknown) environment's context distribution while learning diverse, and non-linear skills for solving a task defined by a specific context. Importantly, our approach operates without any assumptions about the environment. We show how we can learn multi-modal context distributions by training an energy-based model solely on context samples obtained from the environment. On multiple sophisticated simulated robot tasks, we demonstrate that we can learn diverse skills while performing on par or better than baselines.

§ PRELIMINARIES AND RELATED WORK

Contextual episode-based Policy Search (CEPS). CEPS is a black-box approach to reinforcement learning (RL), in which the search distribution is the agent's policy that maps the contexts  $\context$ to policy parameters, typically represented as motion primitives [Schaal, 2006, Paraschos et al., 2013, Li et al., 2023] parameterized by $\mpparam$. The policy $\policy{\pi}$ is optimized by

\begin{align}
\max_{\policy{\pi}} \mathbb{E}_{\contextdistr{p}}\left[\mathbb{E}_{\policy{\pi}}[\reward ] \right],
\end{align}

where $\reward$ is the return. Given context samples from the environment's context distribution $\contextdistr{p}$, the policy $\policy{\pi}$ chooses the controller's parameters $\mpparam$ once at the beginning of the episode.
One of the noteworthy advantages of CEPS lies in the independence of assumptions such as the Markovian property in common MDPs. This characteristic renders it a versatile methodology, particularly well-suited for addressing a diverse array of intricate tasks where the formulation of a Markovian reward function is difficult.

CEPS has been explored by applying various optimization techniques, including Policy Gradients [Sehnke et al., 2010], Natural Gradients [Wierstra et al., 2014], stochastic search strategies [Hansen & Ostermeier, 2001, Mannor et al., 2003, Abdolmaleki et al., 2019], and trust-region optimization techniques [Abdolmaleki et al., 2015, Daniel et al., 2012, Tangkaratt et al., 2017], particularly in the non-contextual setting.
Researchers extended the setting by incorporating linear [Tangkaratt et al., 2017, Abdolmaleki et al., 2019] and non-linear contextual adaptation [Otto et al., 2023], leveraging the recently introduced trust-region layers for neural networks [Otto et al., 2021]. All previously mentioned learn single-mode policies and do not address acquiring diverse skills leveraging automatic curriculum learning.

Curriculum Reinforcement Learning (CRL). CRL can potentially increase the performance of RL agents, especially in sparse-rewarded environments in which exploration is fundamentally difficult. Adapting the environment based on the agent's learning process has been proposed by several works already, e.g. automatically generating sets of tasks or goals to increase the learning speed of the agent [Florensa et al., 2017, Florensa et al., 2018, Sukhbaatar et al., 2018, Zhang et al., 2020, Wöhlke et al., 2020, Racaniere et al., 2020], or generating a curriculum by interpolating an auxiliary and known distribution of target tasks [Klink et al., 2022, Klink et al., 2020, Klink et al., 2020]. Works propose sampling a training level from a prespecified set of environments [Jiang et al., 2021], or unsupervised environment design [Jiang et al., 2021, Dennis et al., 2020] based on the agent's learning process. None of the above methods apply automatic curriculum learning on an RL problem with an MoE policy, except for the work in [Celik et al., 2022]. However, they parameterize the curriculum distribution as Gaussian, suffering from low representation capacity and requiring knowledge about the environment's context distribution. Instead, we show how we can leverage energy-based models to avoid these shortcomings.

Mixture of Experts (MoE) Policy for Curriculum Learning. The MoE policy is formalized as [Bishop, 2006]

\begin{align}\label{eq::StandardMixtureModel}
\policy{\pi} = \sum_o \gating{\pi}\expert{\pi},
\end{align}

where the gating distribution $\gating{\pi}$ assigns an expert $o$ to the given context $\context$. The expert $\expert{\pi}$ adapts the parameters $\mpparam$ of the motion primitive for $\context$. The corresponding motion primitive is then executed in the environment. While this form of the MoE is suitable in inference time when the context is assigned by the environment and the agent needs to propose a skill, it does not allow to automatically learn a curriculum during training.
This drawback is caused by the lack of a parameterized distribution $\contextdistr{\pi}$ that is part of the MoE and allows explicitly choosing and setting context samples for the model itself, so each expert can decide on which contexts it favors training. Introducing this distribution in the context space is a small, but necessary distinction to enable automatic curriculum learning for each single expert $\option$. We can easily reparameterize the MoE without any assumption by using Bayes' rule as [Celik et al., 2022]

\begin{align}\label{eq::MixtureModel}
\policy{\pi} = \sum_o \frac{\contextcomp{\pi} \mgating{\pi}}{\contextdistr{\pi}}\expert{\pi}.
\end{align}

The per-expert context distribution $\contextcomp{\pi}$ can now be optimized and allows the expert $o$ to choose contexts $\context$ it favors. Note that $\contextdistr{\pi} = \sum_o \contextcomp{\pi}\mgating{\pi}$. We model each $\contextcomp{\pi}$ as an energy-based model and each $\expert{\pi}$ as a Gaussian parameterized as neural network (see Fig. <ref>). The prior $\mgating{\pi}$ is set to be a uniform distribution throughout this work.

The Sampling procedure for Di-SkilL. During Inference the agent observes contexts $\context$ from the environment's unknown context distribution $\contextdistr{p}$. The agent calculates the gating probabilities $\gating{\pi}$ for each context and samples an expert $\option$ resulting in $(\option, \context$) samples marked in blue. During Training we first sample a batch of contexts $\context$ from $\contextdistr{p}$, which is used to calculate the per-expert context distribution $\contextcomp{\pi}$ for each expert $\option = 1,..., K$. The $\contextcomp{\pi}$ provides a higher probability for contexts preferred by the expert $\expert{\pi}$. To enable curriculum learning, we provide each expert the contexts sampled from its corresponding $\contextcomp{\pi}$, resulting in the samples $(\option, \context_T)$ marked in orange. In both cases, the chosen $\expert{\pi}$ samples motion primitive parameters $\mpparam$ for each context resulting in a trajectory $\tau$ that is subsequently executed on the environment. Before execution, the corresponding context, e.g., the goal position of a box, needs to be set in the environment. This is illustrated by the dashed arrows with the context in blue for inference and orange for training.

Self-Paced Diverse Skill Learning with MoE. We aim to discover different skills in the same context-defined task. MoE models (see Eq. <ref>) are specifically suitable for skill discovery due to their ability to represent multi-modality and the per-expert context distribution $\contextcomp{\pi}$ for automatic curriculum learning which allows the experts to specialize in a subset of the context space.
For explicit optimization of this policy, the KL-regularized Maximum Entropy Reinforcement Learning objective [Celik et al., 2022]

\begin{align}
\label{eq::maxEntrObj}
\max_{\policy{\pi},\contextdistr{\pi}} & \mathbb{E}_{\contextdistr{\pi}}\left[\mathbb{E}_{\policy{\pi}} \left[\reward\right] + \alpha \textrm{H}\left[\policy{\pi}\right]\right] \nonumber\\
\end{align}

is a natural choice. The KL-term incentivizes the context distribution $\contextdistr{\pi}$ to match the environment's distribution $\contextdistr{p}$ and can be prioritized during optimization by choosing the scaling parameter $\beta$ appropriately. The entropy of the mixture model incentivizes learning diverse solutions [Celik et al., 2022] and can be prioritized with a high scaling parameter $\alpha$.

It is well-known that this objective is difficult to optimize for MoE policies and requires further steps to obtain a tractable lower-bound [Celik et al., 2022]

\begin{align}\label{eq::LBComps_with_aux}
\max_{\expert{\pi}} & \mathbb{E}_{\contextcomp{\pi},  \expert{\pi}} \left[\reward + \alpha \log \tilderesponsibility{\pi} \right] \nonumber \\
\end{align}

for the expert updates and a lower-bound for the per-expert context updates

\begin{align}\label{eq::LBCtxtDistr_with_aux}
\max_{\contextcomp{\pi}}&  \mathbb{E}_{\contextcomp{\pi}} \left[L_c(\option,\context) + (\beta - \alpha) \log \tildegating{\pi}\right] + \nonumber\\
&\beta \textrm{H}\left(\contextcomp{\pi}\right),
\end{align}

where  $ L_c(\option, \context) = \mathbb{E}_{\expert{\pi}} \big[\reward   + \alpha \log \tildegating{\pi}\big] + \alpha \textrm{H}\left[\expert{\pi}\right]$. The variational distributions $\tilderesponsibility{\pi}= \pi_{old}(\option|\context, \mpparam)$ and $\tildegating{\pi} = \pi_{old}(\option|\context) $ arise through the decomposition and are responsible for learning diverse solutions and concentrating on context regions with small, or no support by $\contextdistr{\pi}$. In every iteration, the variational distributions are updated to tighten the bounds. Details of the equations are in the Appendix <ref> and [Celik et al., 2022].

RL with MoE. [Ren et al., 2021] proposes using MoE policy representation and presents a novel gradient estimator to calculate the gradients w.r.t. the MoE parameters. [Huang et al., 2023] presents a model-based RL approach to train latent variable models. The work presents a novel lower bound for training the multi-modal policy parameterization. These methods differ from our work in that they are not categorized in the CEPS framework, or are model-based variants and do not use automatic curriculum learning techniques. In the CEPS framework, RL with MoE policies has also been explored in the works by Daniel et al., 2012, End et al., 2017, in which an MoE model with linear experts without automatic curriculum learning is learned. Additionally, they need to add constraints to enforce diversity in the experts. The work by Celik et al., 2022 also relies on the maximum entropy objective as we do, however, their method only considers linear experts with Gaussian per-expert distributions which limits the performance and consequently requires many experts to solve a task. Moreover, it requires environment knowledge to hand-tune a punishment term to keep the optimization of the per-expert context distributions within the context bounds.

Unsupervised Reinforcement Learning also considers learning diverse policies [Laskin et al., 2021, Eysenbach et al., 2019, Campos et al., 2020, Lee et al., 2019, Liu & Abbeel, 2021]. The objective differs from ours and skills are trained in the absence of an extrinsic reward. We discuss parallels in the Appendix <ref>.

§ DIVERSE SKILL LEARNING

The common Contextual Episodic Policy Search (CEPS) loop [Kupcsik et al., 2013] with a Mixture of Experts (MoE) policy representation learning observes a context $\context$, and then selects an expert $\option$ that subsequently adjusts the controller parameters $\mpparam$ given ($\context$, $\option$). We consider the same process during testing time, as shown in blue color in Fig. <ref> (see also Fig. <ref>).
However, the procedure changes during training for Di-SkilL as automatic curriculum learning requires that the agent can determine which context regions it prefers to focus on.
In this case, we observe a batch of context samples from the environment's context distribution $\contextdistr{p}$. For each of these samples, every per-expert context distribution $\contextcomp{\pi}$ calculates a probability, which results in a categorical distribution over the contexts $\context$. We use these probabilities to sample contexts $\context_T$ for the corresponding expert $\option$ resulting in ($\context_T, \option$) sample pairs (see orange parts in Fig. <ref> and Fig. <ref>).
Each chosen expert $\option$ provides Gaussian distributions over the motion primitive parameters $\mpparam$ by mapping the contexts $\context_T$ to mean vectors $\cvec{\mu}_{\mpparam}$ and covariance matrices $\cvec{\Sigma}_{\mpparam}$ using a parameterized neural network. We can now sample motion primitive parameters $\mpparam$ from these Gaussian distributions to generate trajectories $\tau$ using a motion primitive generator. These trajectories are subsequently executed on the environment (green color in Fig. <ref>) and an episode return $R(\mpparam, \context_T)$ is observed and used for updating the MoE (see Section <ref>).
Yet, there exist several issues for a stable overall training of the MoE model, which requires special treatment for each $\contextcomp{\pi}$ and $\expert{\pi}$. Algorithmic details and parameterizations of the model can be found in the Appendix <ref>.

§.§ Energy-Based Model For Automatic Curriculum Learning

a) High-probability regions of the individual per-expert context distributions $\contextcomp{\pi}$, where a color represents an expert $\option$. The red circle marks the context space of goal-reaching positions for the 5-Link Reacher's tip. The specialization of $\contextcomp{\pi}$ is induced by $\tildegating{\pi}$. b) Different $\contextcomp{\pi}$ need to overlap for learning diverse skills. This overlapping is induced by the entropy bonus $\textrm{H}\left[\contextcomp{\pi}\right]$. c) Different tip trajectories sampled in the same contexts. The trajectories and the end joint constellation are in the same color. The diversity in the parameter space is induced by $\tilderesponsibility{\pi}$. d) Visualization of the 5-Link Reacher task (5LR).

To illustrate these issues, we consider a bounded, uniformly distributed two-dimensional environment context distribution $p(\context)$ (see example in the Appendix <ref> in Fig. <ref>). It is challenging for a Reinforcement Learning (RL) agent to automatically learn its curriculum $\contextcomp{\pi}$ within the valid context space [Celik et al., 2022]. Hard discontinuities such as steps often naturally arise in $\contextdistr{p}$ due to the environment's finite support in real-world environments. For instance, in an environment where the agent's task is to place an object in specific positions on a table, the probability of observing a goal position outside the table's surface is zero.
This implies that a large subset of the context space has no probability mass. Therefore, exploration in these regions might be difficult if there is no guidance encoded in the reward.
Even if it is guaranteed that $\contextcomp{\pi}$ only samples valid contexts, it still needs to be able to represent multi-modal distributions, such as illustrated in Fig. <ref>. This multi-modality can be present because of environmental circumstances or simply if experts $\expert{\pi}$ prefer contexts in spatially apart regions. For the object placing example, this could correspond to regions on the table where the object can not be placed due to obstacles or holes.
We therefore require $\contextcomp{\pi}$ to be able to represent i) complex distributions, ii) multi-modality and iii) only explore within the valid context bounds of $\contextdistr{p}$.
We propose parameterizing $\contextcomp{\pi}$ as an energy-based model

\begin{align}\label{eq:Energy-based-per-component-ctxt-distr}
\contextcomp{\pi} = \exp(\potential{\phi}{\option})/ \normalizer
\end{align}

to address the issues i) and ii). Energy-based models (EBMs) have shown to be capable of representing sharp discontinued functions and multi-modal distributions [Florence et al., 2022]. Yet, they are hard to train and sample from due to the intractable normalizing constant $\normalizer = \int_{\context} \exp(\potential{\phi}{\option}) d\context$.
We can circumvent and address these issues iii) by approximating the normalizing constant with contexts $\context \sim \contextdistr{p}$ as $\normalizer \approx \frac{1}{N} \sum_{i=1}^N \exp(\phi_{\option}(\context_i))$. This approximation is justified as we can sample from $\contextdistr{p}$ by simply resetting the environment without execution. Additionally, the EBM will encounter important parts of the context space during the training by resampling a large enough batch of contexts $\context \sim \contextdistr{p}$ in each iteration. Each expert can therefore sample preferred contexts from the current batch of valid contexts by calculating the probability for each of the contexts using $\contextcomp{\pi}$ as parameterized in Eq. <ref>. It should be noted that this sampling procedure is not straightforwardly applicable to explicit models such as Gaussians, or Normalizing Flows [Papamakarios et al., 2021]. Those methods would need additional techniques like importance sampling that might destabilize learning if not carefully calibrated by enforcing overlapping support regions of the sampling and the actual distribution. In our case, updating the parameters of the EBM can readily be addressed by the standard RL objective for diverse skill learning.

§.§ Updating the Mixture of Experts Model

We update each expert $\expert{\pi}$ and its corresponding per-expert context distribution $\contextcomp{\pi}$ by maximizing the objectives in Eq. <ref> and in Eq. <ref>, respectively. These decomposed objectives allow us to independently update both distributions and to retain the properties of diverse skill learning from the objective in Eq. <ref>.
However, updating the distributions is not straightforward due to the bi-level optimization that leads to a dependency on both terms. This is particularly challenging for the expert $\expert{\pi}$ as the sampled contexts $\context$ can drastically change from one iteration to another if $\contextcomp{\pi}$ changes too aggressively. The same applies for updating $\contextcomp{\pi}$ as calculating the objective requires calculating an integral over $\mpparam$ under the expectation of $\expert{\pi}$.
For a stable update for both distributions, we employ trust-region updates to restrict the change of both distributions from one iteration to another. These updates have been shown to improve the learning process [Peters et al., 2010, Schulman et al., 2015, Schulman et al., 2017, Otto et al., 2021].

[]Visualization of Environments

[]Mean Success Rate (TT)

[]IQM Mean Return (5LR)

a) (left top) Hopper Jump Task (HJ). (Top right) Box Pushing with Obstacle (BPO). (Bottom Left) Robot Mini Golf (MG). (Bottom right) Robot table tennis (TT).
b) Ablation studies, showcasing the need for automatic curriculum learning for Di-SkilL. BBRL and Di-SkilL can solve TT environment decently. Di-SkilL's variants without curriculum learning struggle to achieve a good performance. SVSL needs more samples to achieve around 80% success rate, suffering under the linear experts. c) Performance of Di-SkilL, BBRL, LinDi-SkilL, and PPO on 5LR with sparse in-time rewards.

Expert Update. We parameterize each expert $\expert{\pi}$ with a single neural network and update them by a trust-region constrained optimization

\begin{align}\label{eq::LBComps_with_aux_with_TRConstraint}
\max_{\expert{\pi}} & \mathbb{E}_{\contextcomp{\pi},  \expert{\pi}} \left[\reward + \alpha \log \tilderesponsibility{\pi} \right] \nonumber \\
&\quad \quad \quad \quad \quad  +\alpha\mathbb{E}_{\contextcomp{\pi}}\left[\textrm{H}\left[\expert{\pi}\right]\right] \\
& \st \quad  \KL{\expert{\pi}}{\old{\pi}(\mpparam|\context, \option)} \le \epsilon ~~~~~ \forall ~\context \in \mathcal{C} \nonumber,
\end{align}

where the KL-bound ensures that the expert $\expert{\pi}$ does not differ too much from the expert $\old{\pi}(\mpparam|\context, \option)$ from the iteration before for each context $\context$. The entropy bonus $\textrm{H}\left[\expert{\pi}\right]$ incentivizes $\expert{\pi}$ to fully cover the parameter space while avoiding $(\mpparam, \context)$ regions that are covered by other experts $\option$. The latter is guaranteed by $\tilderesponsibility{\pi}$ which rewards $(\mpparam, \context)$ regions that can be assigned to expert $\option$ with high probability. We efficiently update the experts using trust region layers [Otto et al., 2021, Otto et al., 2023].

Per-Expert Context Distribution Update. We consider the objective with the augmented rewards as shown in Eq. <ref> for updating each context distribution $\contextcomp{\pi}$. We can not apply the trust region layers [Otto et al., 2021] in this case, as $\contextcomp{\pi}$ is a discrete distribution over the context samples $\mathbf c_i$ parameterized by the EBM. Yet, we can still use PPO [Schulman et al., 2017] for updating $\contextcomp{\pi}$ and simplify our objective, as we can now calculate many terms in closed form. For this, we rewrite the objective as

\begin{align}\label{eq::LBCtxtDistr_with_aux_EBM}
\max_{\contextcomp{\pi}}&  \sum_{\mathbf c_i \sim p(\mathbf c)} \pi(\mathbf{c}_i|o) %\contextcomp{\pi}
L_c(\option,\context_i) + \\%\sum\contextcomp{\pi}
&\sum_{\mathbf c_i \sim p(\mathbf c)} \pi(\mathbf{c}_i|o)\big( (\beta - \alpha)  \log \tilde{\pi}(o|\mathbf{c}_i)
- \beta \log \pi(\mathbf{c}_i|o)\big) \nonumber
\end{align}

and observe that all terms in the second sum can be calculated in closed form. Note that the first term is approximated by resampling the context samples using  $\contextcomp{\pi}$ since computing $L_c(\option, \context)$ requires calculating the integral over $\mpparam$ under the expectation of $\expert{\pi}$ as
$ L_c(\option, \context) = \mathbb{E}_{\expert{\pi}} \big[\reward   + \alpha \log \tildegating{\pi}\big] + \alpha \textrm{H}\left[\expert{\pi}\right]$. This expectation can only be estimated for context vectors that are actually chosen by the component.
The entropy bonus in Eq. (<ref>) incentivizes covering of the context space, while focusing on context regions that are not, or only partly covered by other options. The latter is guaranteed by $\tildegating{\pi}$ which assigns a high probability if expert $\option$ can be assigned to the context $\context$.

§.§ How does Diversity Emerge?

From the Eq. <ref> and Eq. <ref> it is clear that diverse behaviors, represented by the experts, emerge from the interplay of those terms in Eq. <ref> and Eq. <ref>. We visually demonstrate the meaning of the individual terms on the 5-Link Reacher task (see Fig. <ref>). The Reacher needs to reach a goal position in the two-dimensional space with its tip. In this task, a context represents the goal position within the context space, visualized as a red circle around the reacher's fixed first joint (Fig. <ref>). We trained Di-SkilL with 50 experts.

In Fig. <ref> we show the high-probability regions of the individual per-expert context distributions $\contextcomp{\pi}$, by setting the color intensity proportional to this probability. Each color represents an individual expert $\option$.

Each $\contextcomp{\pi}$ concentrates on a sub-region of the context space such that the corresponding $\expert{\pi}$ becomes an expert there. This specialization is incentivized by the term $\tildegating{\pi}$ in Eq. <ref>. For learning diverse behaviors for the same context regions, it is necessary that the per-expert context distributions $\contextcomp{\pi}$ overlap, which is motivated by $\textrm{H}\left[\contextcomp{\pi}\right]$ in Eq. <ref>.

These overlapping context regions are visualized in Fig. <ref>, where we count how many experts $\option$ are active for each context. The figure shows that more experts prefer regions close to the initial position of the reacher, indicating that these contexts are easier to solve. Despite the closeness to the reacher's initial position, the agent does not have to exert much energy to reach these points. Indeed, both aspects are present in the task's reward function (see Appendix <ref> for details), explaining why the left half plane of the context space has fewer overlapping. However, the learned MoE has two or more experts active in most parts of the context region. These experts differ in their behavior (see Fig. <ref> for examples), which is motivated by the terms $\tilderesponsibility{\pi}$ and $\textrm{H}\left[\expert{\pi}\right]$ in Eq. <ref>.

§ EXPERIMENTS

In our evaluations, we compare Di-SkilL against the baselines BBRL [Otto et al., 2023] and SVSL [Celik et al., 2022]. Whenever the environment satisfies the Markov properties, we additionally compare against PPO [Schulman et al., 2017]. BBRL and SVSL are suitable baselines as they are state-of-the-art CEPS algorithms. BBRL can learn highly non-linear policies leveraging trust region updates. SVSL learns a linear Mixture of Experts (MoE) model and can capture multi-modality in the behavior space. We consider challenging robotic environments with continuous context and parameter spaces. The considered environments either have a non-markovian reward function, i.e. require retrospective data for calculation, or temporally sparse reward functions increasing the learning complexity.

Environments. Environment visualizations are in Fig. <ref>. Environmental and experimental details are in Appendix <ref>.

[]IQM Mean Return (HJ)

[]IQM Success Rate (BPO)

[]IQM Success Rate (TT-Hard)

[]IQM Success Rate (MG)

Performance on the a) HJ b) BPO, c) TT-H, and d) MG tasks.  a)Di-SkilL performs on par with BBRL on the HJ task. b)The multi-modality introduced by the obstacle in the box pushing task leads to worse performance for BBRL than for Di-SkilL and LinDi-SkilL. PPO suffers under the time-sparse reward setting. c) While BBRL converges faster, Di-SkilL achieves a higher success rate eventually. d) Di-SkilL outperforms the baselines on the MG task. SinDi-SkilL performs poorly on the non-markovian rewarded tasks TT-H and MG indicating that highly non-linear policies are beneficial.

Table Tennis (TT). A 7-degree of freedom (DoF) robot has to learn fast and precise motions to smash the ball to a desired position on the opponent's side. The 4-dim. context consists of the incoming ball's landing position and the desired ball's landing position on the opponent's side. The TT environment requires good exploratory behavior and has a non-markovian reward structure making step-based approaches infeasible to learn useful skills [Otto et al., 2023].

Table Tennis Hard (TT-H). We extend the TT environment to a more challenging version by varying the ball's initial velocity. This additionally increases the learning complexity, as the agent now needs to reason about the physical effects of changed velocity ranges and requires improved adaptability.

5-Link Reacher (5LR). The 5-Link reacher has to reach a goal position within all quadrants in the context space (see Fig. <ref>) as opposed to the version in [Otto et al., 2023], where the multi-modality in the behavior space (see Fig. <ref>) was avoided by constraining the context space to the upper half of the context space. Additionally, the time-sparse reward makes this task a challenging benchmark.

Hopper Jump (HJ). Presented in [Otto et al., 2023] in which the Hopper [Brockman et al., 2016] is tasked to jump as high as possible while landing in a goal position. The HJ has a non-markovian reward, making step-based RL methods unfeasible to learn useful policies [Otto et al., 2023].

Box Pushing with Obstacle (BPO). A 7DoF robot is tasked to push a box to a target position and rotation while avoiding an obstacle. In addition to the time-spare reward [Otto et al., 2023], our version includes the obstacle and considers a larger range of the box's target position.

Robot Mini Golf (MG). The 7DoF robot has to hit the ball in an environment with two obstacles (static, varying), such that it passes the tight goal. The context is the obstacle's, the goal's, and the ball's position. The MG environment has a non-markovian reward, making step-based RL methods unfeasible to learn useful policies [Otto et al., 2023].

§.§ ACL Benefits

Di-SkilL's Diverse Skills for the BPO Task. The figures visualize diverse solutions to the same contexts $\context$ on a table (black rectangle). The red, thick rectangle represents the obstacle. The 7DoF robot is tasked to push the box (shown in different colors for each solution found) to the goal box position (red rectangle with a green dot) and align the blue edges to match the orientation. We visualized successful box trajectories for each sampled skill. The diversity learned in the parameter space results in different box trajectories ranging in position and orientation.

[]Backhand Drive

[]Forehand Drive

[]Backhand Block

[]Forehand Push

[]Backhand Smash

Di-SkilL's Diverse Skills for the TT-H task. We fixed the ball's desired landing position and varied the serving landing position and the ball's initial velocity. Di-SkilL can return the ball in different striking types.

Automatic Curriculum learning (ACL) enables Di-SkilL's experts to shape their curriculum by explicitly sampling from preferred context regions. We analyze the importance of this feature by comparing the performance of variants of Di-SkilL. For both variants Di-SkilLV2 and Di-SkilLV3 we disable ACL by fixing the variational distribution to $\tildegating{\pi}=0$ in Eq. <ref> and setting the entropy scaling parameter $\beta=2000$. Ignoring the variational distribution during training eliminates the intrinsic motivation of the per-expert context distribution $\contextcomp{\pi}$ to focus on sub-regions in the context space that are not, or only partially, covered by any other $\contextcomp{\pi}$ (Section <ref>). Setting $\beta=2000$ incentivizes each $\contextcomp{\pi}$ to maximize its entropy, resulting in a uniform distribution. For Di-SkilL we keep the ACL and set $\beta=4$. We provide the same number of 50 context-parameter samples per expert for Di-SkilLV2 and Di-SkilL, whereas Di-SkilLV3 receives 260 samples per expert in each iteration. All variants possess 5 experts. In Fig. <ref> we report the mean success rates and the 95% confidence interval for each method on at least 4 seeds. Di-SkilLV2 converges to a much smaller success rate and Di-SkilLV2 needs more samples to reach the level of Di-SkilL. BBRL and Di-SkilL achieve high success rates. SVSL shows worse performance, even though the model has 20 experts. The results show that ACL is an essential feature of Di-SkilL and that linear experts are not capable of achieving a satisfying performance.

§.§ Analyzing the Performance and Diversity

We have evaluated all methods on 24 seeds for each environment and algorithm and report the interquantile mean (IQM) with a 95% stratified bootstrap confidence interval as suggested by [Agarwal et al., 2021]. Please note that SVSL requires designing a punishment function to guide the context samples in the environment's valid context region, which makes its application difficult, especially if the context influences the objects' physics. We therefore propose comparing against LinDi-SkilL instead of SVSL. LinDi-SkilL also has linear experts but benefits from

The performance curve of the HJ task in Fig. <ref> shows that Di-SkilL performs on par with BBRL, while BBRL converges slightly faster. Both methods can solve the task indicating that the task doesn't require diversity. We provide additional analysis of this task in Appendix <ref>.
Fig. <ref> shows the performance curves on the BPO task. The obstacle introduces multi-modality in the behavior space which can not be captured by a single-mode policy, explaining why DiSkilL and LinDi-SkilL outperform BBRL, while Di-SkilL still achieves a higher success rate. PPO's poor performance indicates that time-correlated exploration as used with motion primitives is effective in sparse rewarded tasks. We can observe a similar performance behavior in the 5LR task. In Fig. <ref> we report the achieved returns and observe that Di-SkilL outperforms BBRL due to the ability to capture multi-modal behaviors (e.g. reaching from different sides) while PPO suffers from the sparse rewarded setting while LinDi-SkilL's linear experts cause slow convergence, indicating that more experts are needed. Di-SkilL's diverse skills in the parameter space $\mpparam$ induce different behaviors. Fig. <ref> shows diverse box trajectories to the same contexts in the BPO task, whereas Fig. <ref> shows different tip trajectories to the same goal positions in the 5LR task.

The non-markovian rewarded tasks (TT-H and MG) show that non-linear policies as learned by BBRL and Di-SkilL are beneficial. Di-SkilL and BBRL perform similarly well on the TT-H task (see Fig. <ref>), where Di-SkilL achieves a slightly higher end success rate than BBRL. The performance gap is more clear on the MG task (see Fig. <ref>) with Di-SkilL outperforming BBRL and LinDi-SkilL.
Di-SkilL can discover common striking styles in table tennis. We visualize some of these learned skill in Fig. <ref>. Additional strike visualizations can be found in the Appendix <ref>.

§ CONCLUSION AND FUTURE WORK

We proposed a novel method for learning diverse skills using a contextual Mixture of Experts. Each expert learns its curriculum by optimizing for a per-expert context distribution $\contextcomp{\pi}$. We have demonstrated challenges that arise through enabling automatic curriculum learning (ACR) and proposed parameterizing $\contextcomp{\pi}$ as energy-based models (EBMs) to address these challenges. Additionally, we provided a methodology to efficiently optimize these EBMs. We also proposed using trust-region updates for the deep experts to stabilize our bi-level optimization problem. In an ablation, we have shown that ACR is necessary for efficient and performant learning. Moreover, in sophisticated robot simulation environments, we have shown that our method can learn diverse skills while performing on par or better than the baselines. Currently, the major drawback of our approach is that it is not able to replan, causing failures in the tasks if the robot even has small collisions with objects. We intend to address this issue in future research. For improving the sample complexity of our approach, we additionally plan to use off-policy RL techniques.

§ BROADER IMPACT

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

[Abdolmaleki et al., 2015]
Abdolmaleki, A., Lioutikov, R., Peters, J. R., Lau, N., Pualo Reis, L., and Neumann, G.
Model-based relative entropy stochastic search.
Advances in Neural Information Processing Systems, 28, 2015.

[Abdolmaleki et al., 2019]
Abdolmaleki, A., Simoes, D., Lau, N., Reis, L. P., and Neumann, G.
Contextual direct policy search: With regularized covariance matrix estimation.
Journal of Intelligent & Robotic Systems, 96:0 141–157, 2019.

[Agarwal et al., 2021]
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M.
Deep reinforcement learning at the edge of the statistical precipice.
Advances in neural information processing systems, 34:0 29304–29320, 2021.

[Bishop, 2006]
Bishop, C.
Pattern recognition and machine learning.
Springer google schola, 2:0 531–537, 2006.

[Blessing et al., 2023]
Blessing, D., Celik, O., Jia, X., Reuss, M., Li, M. X., Lioutikov, R., and Neumann, G.
Information maximizing curriculum: A curriculum-based approach for learning versatile skills.
In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

[Brockman et al., 2016]
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.
Openai gym, 2016.

[Campos et al., 2020]
Campos, V., Trott, A., Xiong, C., Socher, R., i Nieto, X. G., and Torres, J.
Explore, discover and learn: Unsupervised discovery of state-covering skills.
In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp.  1317–1327. PMLR, 2020.

[Celik et al., 2022]
Celik, O., Zhou, D., Li, G., Becker, P., and Neumann, G.
Specializing versatile skill libraries using local mixture of experts.
In Conference on Robot Learning, pp.  1423–1433. PMLR, 2022.

[Chi et al., 2023]
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S.
Diffusion policy: Visuomotor policy learning via action diffusion.
In Proceedings of Robotics: Science and Systems (RSS), 2023.

[Daniel et al., 2012]
Daniel, C., Neumann, G., and Peters, J.
Hierarchical relative entropy policy search.
In Artificial Intelligence and Statistics, pp.  273–281. PMLR, 2012.

[Dennis et al., 2020]
Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., and Levine, S.
Emergent complexity and zero-shot transfer via unsupervised environment design.
Advances in neural information processing systems, 33:0 13049–13061, 2020.

[End et al., 2017]
End, F., Akrour, R., Peters, J., and Neumann, G.
Layered direct policy search for learning hierarchical skills.
In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp.  6442–6448. IEEE, 2017.

[Engstrom et al., 2020]
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A.
Implementation matters in deep policy gradients: A case study on ppo and trpo.
In International Conference on Learning Representations, 2020.

[Eysenbach et al., 2019]
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S.
Diversity is all you need: Learning skills without a reward function.
In International Conference on Learning Representations, 2019.

[Florence et al., 2022]
Florence, P., Lynch, C., Zeng, A., Ramirez, O. A., Wahid, A., Downs, L., Wong, A., Lee, J., Mordatch, I., and Tompson, J.
Implicit behavioral cloning.
In Conference on Robot Learning, pp.  158–168. PMLR, 2022.

[Florensa et al., 2017]
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P.
Reverse curriculum generation for reinforcement learning.
In Conference on robot learning, pp.  482–495. PMLR, 2017.

[Florensa et al., 2018]
Florensa, C., Held, D., Geng, X., and Abbeel, P.
Automatic goal generation for reinforcement learning agents.
In International conference on machine learning, pp.  1515–1528. PMLR, 2018.

[Hansen & Ostermeier, 2001]
Hansen, N. and Ostermeier, A.
Completely derandomized self-adaptation in evolution strategies.
Evolutionary computation, 90 (2):0 159–195, 2001.

[Huang et al., 2023]
Huang, Z., Liang, L., Ling, Z., Li, X., Gan, C., and Su, H.
Reparameterized policy learning for multimodal trajectory optimization.
ICML, 2023.

[Jia et al., 2024]
Jia, X., Blessing, D., Jiang, X., Reuss, M., Donat, A., Lioutikov, R., and Neumann, G.
Towards diverse behaviors: A benchmark for imitation learning with human demonstrations.
In The Twelfth International Conference on Learning Representations, 2024.
URL <https://openreview.net/forum?id=6pPYRXKPpw>.

[Jiang et al., 2021]
Jiang, M., Dennis, M., Parker-Holder, J., Foerster, J., Grefenstette, E., and Rocktäschel, T.
Replay-guided adversarial environment design.
Advances in Neural Information Processing Systems, 34:0 1884–1897, 2021a.

[Jiang et al., 2021]
Jiang, M., Grefenstette, E., and Rocktäschel, T.
Prioritized level replay.
In International Conference on Machine Learning, pp.  4940–4950. PMLR, 2021b.

[Klink et al., 2020]
Klink, P., Abdulsamad, H., Belousov, B., and Peters, J.
Self-paced contextual reinforcement learning.
In Conference on Robot Learning, pp.  513–529. PMLR, 2020a.

[Klink et al., 2020]
Klink, P., D'Eramo, C., Peters, J. R., and Pajarinen, J.
Self-paced deep reinforcement learning.
Advances in Neural Information Processing Systems, 33:0 9216–9227, 2020b.

[Klink et al., 2022]
Klink, P., Yang, H., D’Eramo, C., Peters, J., and Pajarinen, J.
Curriculum reinforcement learning via constrained optimal transport.
In International Conference on Machine Learning, pp.  11341–11358. PMLR, 2022.

[Kupcsik et al., 2013]
Kupcsik, A., Deisenroth, M., Peters, J., and Neumann, G.
Data-efficient generalization of robot skills with contextual policy search.
In Proceedings of the AAAI conference on artificial intelligence, volume 27, pp.  1401–1407, 2013.

[Laskin et al., 2021]
Laskin, M., Yarats, D., Liu, H., Lee, K., Zhan, A., Lu, K., Cang, C., Pinto, L., and Abbeel, P.
Urlb: Unsupervised reinforcement learning benchmark.
In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021.

[Lee et al., 2019]
Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., and Salakhutdinov, R.
Efficient exploration via state marginal matching.
arXiv preprint arXiv:1906.05274, 2019.

[Li et al., 2023]
Li, G., Jin, Z., Volpp, M., Otto, F., Lioutikov, R., and Neumann, G.
Prodmp: A unified perspective on dynamic and probabilistic movement primitives.
IEEE Robotics and Automation Letters, 80 (4):0 2325–2332, 2023.

[Liu & Abbeel, 2021]
Liu, H. and Abbeel, P.
Aps: Active pretraining with successor features.
In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.  6736–6747. PMLR, 18–24 Jul 2021.

[Mannor et al., 2003]
Mannor, S., Rubinstein, R. Y., and Gat, Y.
The cross entropy method for fast policy search.
In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp.  512–519, 2003.

[Otto et al., 2021]
Otto, F., Becker, P., Vien, N. A., Ziesche, H. C., and Neumann, G.
Differentiable trust region layers for deep reinforcement learning.
arXiv preprint arXiv:2101.09207, 2021.

[Otto et al., 2023]
Otto, F., Celik, O., Zhou, H., Ziesche, H., Ngo, V. A., and Neumann, G.
Deep black-box reinforcement learning with movement primitives.
In Conference on Robot Learning, pp.  1244–1265. PMLR, 2023.

[Papamakarios et al., 2021]
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B.
Normalizing flows for probabilistic modeling and inference.
The Journal of Machine Learning Research, 220 (1):0 2617–2680, 2021.

[Paraschos et al., 2013]
Paraschos, A., Daniel, C., Peters, J. R., and Neumann, G.
Probabilistic movement primitives.
Advances in neural information processing systems, 26, 2013.

[Peters et al., 2010]
Peters, J., Mulling, K., and Altun, Y.
Relative entropy policy search.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp.  1607–1612, 2010.

[Racaniere et al., 2020]
Racaniere, S., Lampinen, A., Santoro, A., Reichert, D., Firoiu, V., and Lillicrap, T.
Automated curriculum generation through setter-solver interactions.
In International Conference on Learning Representations, 2020.

[Ren et al., 2021]
Ren, J., Li, Y., Ding, Z., Pan, W., and Dong, H.
Probabilistic mixture-of-experts for efficient deep reinforcement learning.
arXiv preprint arXiv:2104.09122, 2021.

[Schaal, 2006]
Schaal, S.
Dynamic movement primitives-a framework for motor control in humans and humanoid robotics.
In Adaptive motion of animals and machines, pp.  261–280. Springer, 2006.

[Schulman et al., 2015]
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P.
Trust region policy optimization.
In International conference on machine learning, pp.  1889–1897. PMLR, 2015.

[Schulman et al., 2017]
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.

[Sehnke et al., 2010]
Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., and Schmidhuber, J.
Parameter-exploring policy gradients.
Neural Networks, 230 (4):0 551–559, 2010.

[Shafiullah et al., 2022]
Shafiullah, N. M., Cui, Z., Altanzaya, A. A., and Pinto, L.
Behavior transformers: Cloning $ k $ modes with one stone.
Advances in neural information processing systems, 35:0 22955–22968, 2022.

[Sukhbaatar et al., 2018]
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., and Fergus, R.
Intrinsic motivation and automatic curricula via asymmetric self-play.
In International Conference on Learning Representations, 2018.

[Tangkaratt et al., 2017]
Tangkaratt, V., Van Hoof, H., Parisi, S., Neumann, G., Peters, J., and Sugiyama, M.
Policy search with high-dimensional context variables.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.

[Wierstra et al., 2014]
Wierstra, D., Schaul, T., Glasmachers, T., Sun, Y., Peters, J., and Schmidhuber, J.
Natural evolution strategies.
The Journal of Machine Learning Research, 150 (1):0 949–980, 2014.

[Wöhlke et al., 2020]
Wöhlke, J., Schmitt, F., and van Hoof, H.
A performance-based start state curriculum framework for reinforcement learning.
In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp.  1503–1511, 2020.

[Zhang et al., 2020]
Zhang, Y., Abbeel, P., and Pinto, L.
Automatic curriculum learning through value disagreement.
Advances in Neural Information Processing Systems, 33:0 7648–7659, 2020.

§ ADDITIONAL INFORMATION TO SELF-PACED DIVERSE SKILL LEARNING WITH MOE

The general self-paced diverse skill learning objective

\begin{align*}
\max_{\policy{\pi},\contextdistr{\pi}} & \mathbb{E}_{\contextdistr{\pi}}\left[\mathbb{E}_{\policy{\pi}} \left[\reward\right] + \alpha \textrm{H}\left[\policy{\pi}\right]\right] \nonumber\\
\end{align*}

can be reformulated to

\begin{align}
\label{eq::maxEntrObj}
\max_{\joint{\pi}} &\mathbb{E}_{\mgating{\pi}, \contextcomp{\pi}}\left[\mathbb{E}_{\expert{\pi}} \left[\reward + \alpha \log{\responsibility{\pi}}\right] \right. \nonumber \\
& \left. + \beta \log{\contextdistr{p}} + (\beta- \alpha) \log \gating{\pi}\right]  \nonumber \\ & + \alpha \mathbb{E}_{\mgating{\pi}, \contextcomp{\pi}}\left[\textrm{H}\left[\expert{\pi}\right]\right] \nonumber \\
& + \beta \mathbb{E}_{\mgating{\pi}}\left[\textrm{H}\left[\contextcomp{\pi}\right]\right] + \beta \textrm{H}\left[\mgating{\pi}\right].
\end{align}

by inserting $\policy{\pi}$, $\contextdistr{\pi}$ from Eq. (<ref>) into Eq. (<ref>) and applying Bayes theorem. This objective is not straightforward to optimize for Mixture of Experts MoE models and requires further steps to introduce a lower bound  (see Section <ref>) that can be efficiently optimized. We refer the interested reader to [Celik et al., 2022] for a detailed derivation.

§ ADDITIONAL RELATED WORK

Unsupervised Reinforcement Learning. Another field of research that considers learning diverse policies is unsupervised reinforcement learning (URL). In URL the agent is first trained solely with an intrinsic reward to acquire a diverse set of skills from which the most appropriate is picked to solve a downstream task. More related to our work is a group of algorithms that obtain their intrinsic reward based on information-theoretic formulations [Laskin et al., 2021, Eysenbach et al., 2019, Campos et al., 2020, Lee et al., 2019, Liu & Abbeel, 2021]. However, their resulting objective is based on the mutual-information and differs from the objective we maximize. The learned skills in the pre-training aim to cover distinct parts of the state-space during pre-training in the absence of an extrinsic task reward which implies that skills are not explicitly trained to solve the same task in different ways. Those methods operate within the step-based RL setting which differs from CEPS.

§ ADDITIONAL INFORMATION TO DIVERSE SKILL LEARNING

[latent] (mpparam) $\mpparam$;
[latent,above=of mpparam,yshift = -0.25cm,xshift=0.75cm] (option) $\option$;
[obs,above=of mpparam,xshift=-0.75cm,fill] (context) $\context$;
context option
option,context mpparam

[latent] (mpparam) $\mpparam$;
[latent,above=of mpparam,yshift = -0.25cm,xshift=0.75cm] (context) $\context$;
[obs,above=of mpparam,xshift=-0.75cm,fill] (option) $\option$;
option context
option,context mpparam

Probabilistic Graphical Models (PGMs) during inference a) and training b). During a)) the model observes the contexts $\context$ from the environment. An expert $\option$ is sampled from $\gating{\pi}$, which leads to an adjustment of the motion primitive parameters $\mpparam$ by $\expert{\pi}$. We iterate over each expert during (b), sample the contexts $\context$ and $\mpparam$ from the per-expert distribution $\contextcomp{\pi}$ and $\expert{\pi}$ respectively. Sampling from $\contextcomp{\pi}$ allows shaping the expert's curriculum. c) illustrates the environment's context distribution $\contextdistr{p}$ and a possibly optimal $\contextcomp{\pi}$ (d)) in two-dim. space. Yellow areas indicate high and purple zero probability. The illustrations show that optimizing $\contextcomp{\pi}$ requires dealing with i) step-like non-linearities, ii) multi-modality, iii) bounded within the red rectangle support of $\contextdistr{p}$, complicating exploration.

§.§ The Parameterization of the Mixture of Experts (MoE) Model

In the following, we provide details on the parameterization of the MoE model.

Parametrization of the expert $\expert{\pi}$.
We parameterize each expert $\expert{\pi}$ as a Gaussian policy $\mathcal{N}(\cvec{\mu}_{\cvec \gamma}(\context), \cvec{\Sigma}_{\cvec \gamma}(\context)$, where the mean $\cvec{\mu}_{\cvec \gamma}(\context)$ and the covariance $\cvec{\Sigma}_{\cvec \gamma}(\context)$ are functions of the context $\context$ and parameterized by a neural network with parameters $\cvec \gamma$.
Although the covariance $\cvec{\Sigma}_{\cvec \gamma}(\context)$ is formalized as a function of the context $\context$, we have not observed any advantages in doing so. In our experiments, we therefore parameterize the covariance as a lower-triangular matrix $\cvec{L}$ and form the covariance matrix $\cvec{\Sigma} = \cvec{LL}^T$.

Parameterization of the per-expert context distribution $\contextcomp{\pi}$. The reader is referred to Section <ref> for details on the parameterization of $\contextcomp{\pi}$

Parameterization of the prior $\mgating{\pi}$. We fix the prior $\mgating{\pi}$ to a uniform distribution over the number K of available components and do not further optimize this distribution. This is a useful definition to increase the entropy of the mixture model.

Parameterization of the context distribution $\contextdistr{\pi}$. Due to the relation $\contextdistr{\pi} = \sum_o \contextcomp{\pi} \mgating{\pi}$, $\contextdistr{\pi}$ is defined by $\contextcomp{\pi}$ and does not need explicit modelling.

Parameterization of the gating distribution $\gating{\pi}$. Due to the relation $\gating{\pi}=\frac{\contextcomp{\pi}\mgating{\pi}}{\contextdistr{\pi}}$ we do not need an explicit parameterization of $\gating{\pi}$ and can easily calculate the probabilities for choosing the expert $o$ given a context $\context$.

§.§ Using Motion Primitives in the Context of Reinforcement Learning

Motion Primitives (MPs) are a low-dimensional representation of a trajectory. For instance, instead of parameterizing a desired joint-level trajectory as the single state in each time step, MPs introduce a low-dimensional parameter vector $\mpparam$ which concisely defines the trajectory to follow. The generation of the trajectory depends on the method that is used. Probabilistic Movement Primitives (ProMPs) [Paraschos et al., 2013] for example define the desired trajectory as a simple linear function $\cvec \tau = \cvec \Phi ^T \mpparam$, where $\cvec \Phi$ are time-dependent basis functions (e.g. normalized radial basis functions). Dynamic Movement Primitives (DMPs) [Schaal, 2006] rely on a second-order dynamic system that provides smooth trajectories in the position and velocity space. Recently Probabilistic Dynamic Movement Primitives (ProDMPs) were introduced by Li et al., 2023 and combines the advantages of both methods, that is the easy generation of trajectories and smooth trajectories. We therefore rely on ProDMPs throughout this work.

In the context of reinforcement learning, the policy $\policy{\pi}$, or in our case an expert $\expert{\pi}$ defines a distribution over the parameters $\mpparam$ of the MP depending on the observed context $\context$. This allows the policy to quickly adapt to new tasks defined by $\context$.

§.§ Algorithm Details

Detailed descriptions of the algorithm during training and during inference are provided in the algorithm boxes Alg. <ref> and Alg. <ref> respectively. In each iteration during training, we sample a batch of contexts $\context$ from the environment by resetting it. We then iterate over each expert and evaluate the probabilities of these contexts $\context$ on each per-expert context distribution $\contextcomp{\pi}$ and sample then training contexts $\context_T$ from them. From the corresponding expert $\expert{\pi}$ we sample motion primitive parameters $\mpparam$ and evaluate the samples ($\context_T, \mpparam$) on the environment and observe a return $\reward$ which we use to update the experts $\expert{\pi}$ and the per-expert context distributions $\contextcomp{\pi}$ by maximizing Obj. <ref> and Obj. <ref> respectively. During inference we observe contexts $\context$ from the environment, calculate the gating distributions $\gating{\pi} = \frac{\contextcomp{\pi} \mgating{\pi}}{\contextdistr\pi}$ from which we sample the expert $o$. We then either take the mean or sample an $\mpparam$ from this expert and execute it on the environment.
Di-SkilL Training
[1]

$\alpha, ~ \beta$, N(max. iterations), K(num. experts),T(num. samples per expert)
$k = 1$ to N
$\context\sim\contextdistr{p}$ (context batch by environment resetting)
$ o = 1$ to $K$
$\context_{T} \sim \contextcomp{\pi}$ (context batch from EBM)
$\mpparam \sim \pi(\mpparam|\context_T, o)$
$\reward \leftarrow$ eval($\mpparam, \context_T$)
$\expert{\pi} \leftarrow$ Obj. <ref>
$\contextcomp{\pi} \leftarrow$ Obj. <ref>

Di-SkilL Inference
[1]

$\context\sim\contextdistr{p}$ (observe contexts from environment)
$o \sim \gating{\pi}$, where $\gating{\pi}=\frac{\contextcomp{\pi}\mgating{\pi}}{\contextdistr{\pi}}$
$\mpparam \sim \pi(\mpparam|\context, o)$
$\reward \leftarrow$ eval($\mpparam, \context$)

§ EXPERIMENTAL DETAILS

§.§ Environment Details

§.§.§ Table Tennis Easy

Environment. We use the same table tennis environment as presented in [Otto et al., 2023], in which a 7 Degree of Freedom (DoF) robot has to return a ball to a desired ball landing position. The context is the four-dimensional space of the ball's initial landing position ( $x \in [-1, -0.2]$, $y \in [-0.65, 0.65]$) on the robot's table side and the desired ball landing position ($x \in [-1.0, -0.2]$, $y \in [-0.6, 0.6]$) on the opponent's table side. The robot is controlled with torques on the joint level in each time step. The torques are generated by the tracking controller (PD-controller) that tracks the desired trajectory generated by the motion primitive. We consider three basis functions per joint resulting in a 21-dimensional parameter ($\mpparam$) space. We additionally allow the agent to learn the trajectory length and the starting time step of the trajectory. Note that the starting point allows the agent to define when after the episode's start the generated desired trajectory should be tracked. Induced by the varying contexts, this is helpful to react to the varying time the served ball needs to reach a positional space that is convenient to hit the ball with the robot's racket. Overall the parameter space is 23 dimensional. The task is considered successful if the returned ball lands on the opponent's side of the table and within $\leq 0.2$m to the goal location.

The reward function is unchanged from [Otto et al., 2023] and is defined as

\begin{equation*}
R_{task} = \begin{cases}
0, & \text{if cond. 1}, \\
f_2(\mathbf{p}_{r}, \mathbf{p}_{b}) & \text{if cond. 2}, \\
f_3(\mathbf{p}_{r}, \mathbf{p}_{b}, \mathbf{p}_l, \mathbf{p}_{goal}) &\text{if cond. 3},\\
f_4(\mathbf{p}_{r}, \mathbf{p}_{b}, \mathbf{p}_l, \mathbf{p}_{goal}) &\text{if cond. 4},\\
f_5(\mathbf{p}_{r}, \mathbf{p}_{b}, \mathbf{p}_l, \mathbf{p}_{goal}) &\text{if cond. 5},\\
\end{cases}
\end{equation*}

where $\mathbf{p}_r$ is the executed trajectory position of the racket center, $\mathbf{p}_b$ is the executed position trajectory of the ball, $\mathbf{p}_{l}$ is the ball landing position, $\mathbf{p}_{goal}$ is the target position. The individual functions are defined as

\begin{align*}
f_2(\mathbf{p}_{r}, \mathbf{p}_{b}) &= 0.2-0.2g(\mathbf{p}_{r}, \mathbf{p}_{b})\\
f_3(\mathbf{p}_{r}, \mathbf{p}_{b}, \mathbf{p}_l, \mathbf{p}_{goal}) &= 3-2g(\mathbf{p}_{r}, \mathbf{p}_{b})-h(\mathbf{p}_{l}, \mathbf{p}_{goal})\\
f_4(\mathbf{p}_{r}, \mathbf{p}_{b}, \mathbf{p}_l, \mathbf{p}_{goal}) &= 6-2g(\mathbf{p}_{r}, \mathbf{p}_{b})-4h(\mathbf{p}_{l}, \mathbf{p}_{goal})\\
f_5(\mathbf{p}_{r}, \mathbf{p}_{b}, \mathbf{p}_l, \mathbf{p}_{goal}) &= 7-2g(\mathbf{p}_{r}, \mathbf{p}_{b})-4h(\mathbf{p}_{l}, \mathbf{p}_{goal}),
\end{align*}

where $g(\cvec x,\cvec y) = \tanh{(\min||\cvec x-\cvec y||^2)}$ and $h(\cvec x,\cvec y) = \tanh{(||\cvec x-\cvec y||^2)}$.
The different conditions are

* cond. 1: the end of the episode is not reached,

* cond. 2: the end of the episode is reached,

* cond. 3: cond.2 is satisfied and the robot did hit the ball,

* cond. 4: cond.3 is satisfied and the returned ball lands on the table,

* cond. 5: cond.4 is satisfied and the landing position is at the opponent's side.

The episode ends when any of the following conditions are met

* the maximum horizon length is reached

* ball did land on the floor without hitting

* ball did land on the floor or table after hitting

The whole desired trajectory is obtained ahead of environment interaction, making use of this property we can collect some samples without physical simulation. The reward function based on this desired trajectory is defined as

\begin{equation*}
r_{traj} = -\sum_{(i,j)}{|\tau_{ij}^d| - |q^b_j|} , \quad
(i,j) \in \{(i,j)\mid |\tau_{ij}^d| > |q^b_j|\}
\end{equation*}

where $\tau^d$ is the desired trajectory, $i$ is the time index, $j$ is the joint index, $q^b$ is the joint position upper bound. The desired trajectory is considered as invalid if $r_{traj} < 0$, an invalid trajectory will not be executed on the robot. The overall reward is defined as:

\begin{equation*}
r = \begin{cases}
r_{traj}, & r_{traj} < 0 \\
r_{task}, & \text{otherwise}
\end{cases}
\end{equation*}

SVSL. SVSL requires designing a guiding punishment term for context samples that are not in a valid region. For the four-dimensional context space in table tennis, this can be done using quadratic functions (as proposed in the original work [Celik et al., 2022]):

\begin{align*}
R_c(\context) = -20 \cdot d_c^2,
\end{align*}

where $d_c^2$ is the distance of the current context $\cvec c$ to the valid context region.

SVSL Hyperparameters All hyperparameters are summarized in the Table <ref>.

Hyperparameters are listed in the Table <ref>.

§.§.§ Table Tennis Task Hard

Environment. We extend the table tennis environment described in Appendix <ref> by additionally including the ball's initial velocity in the context space making the task harder as the agent has to react to ranging velocities now. We define the initial velocity $v_x \in [1.5\frac{m}{s}, 4 \frac{m}{s}]$. Note that every single constellation within the resulting context space is a valid context. However, there exist ball landing positions that can not be set along with a subset of the initial velocity range. This makes designing a guiding punishment term for SVSL especially difficult. We adopt the parameter space and the reward function as defined in the standard table tennis environment as described in Appendix <ref>.

Hyperparameters are listed in the Table <ref>.

§.§ Hopper Jump

Environment. We use the same hopper jump environment as presented in [Otto et al., 2023], in which the hopper [Brockman et al., 2016] has to jump as high as possible and land at a specified position. The context is the four-dimensional space of the last three joints of the hopper and the goal landing position $[j_3, j_4, j_5, g]$, where the ranges are from $[-0.5, -0.2, 0, 0.3]$ to $[0.0, 0.0, 0.785,1.35]$. The hopper is controlled the same as in [Brockman et al., 2016]. Here, we consider three basis functions per joint and a goal basis resulting in a parameter space ($\mpparam$) of 12 dimensions. The reward is non-markovian and is unchanged from [Otto et al., 2023].

In each time-step $t$ the action cost

\begin{align*}
\tau_t = 10^{-3}\sum_i^K (a^i_t)^2,
\end{align*}

is provided. The variable $K=3$  corresponds to the number of degrees of freedom. At the end of the episode, a reward containing retrospective information about the maximum height in the z-direction of the center of mass achieved $h_{\text{max}}$, the goal landing position of the heel $p_{\text{goal}}$, the foot's heel position when having contact with the ground after jumping the first time $p_{\text{foot, contact}}$ is given. Additionally, per-time information such as $p_{\text{foot, t}}$ describing the position of the foot's heel in world-coordinates is given. The resulting reward function is

\begin{align*}
R_{tot} = -\sum_{t=0}^T\tau_t + R_{height} + R_{gdist} + R_{cdist} + R_{healthy},
\end{align*}

\begin{align*}
R_{height} &= 10h_{max}, \\
R_{gdist} &= ||p_{foot, T} - p_{goal}||_2,\\
R_{cdist} &= ||p_{foot,contact} - p_{goal}||_2,\\
R_{healthy} &= \left\{\begin{array}{ll} 2 & \textrm{if } z_T \in [0.5, \infty] \textrm{and } \theta, \gamma, \phi \in [-\infty, \infty]\\
0 & \textrm{else.} \end{array}\right.
\end{align*}

The healthy reward is the same as provided by [Brockman et al., 2016].

Hyperparameters are listed in the Table <ref>.

§.§.§ Box Pushing with Obstacle Task

Environment. We increase the difficulty of the box pushing environment as presented in [Otto et al., 2023], by changing major parts of the context space. The goal of the box pushing task is to move a box to a specified goal location and orientation using the seven DoF Franka Emika Panda. The newly context space (compared to the original version in [Otto et al., 2023]) are described in the following. We increase the box' goal position range to $x_g \in [0.3, 0.6], y_g \in [-0.7, 0.45]$, and keep the goal orientation angle $\phi \in [0rad, 2\pi rad]$. Additionally, we include an obstacle between the initial box and the box's goal. The range of the obstacle position is $x_o \in [0.3, 0.6], y_o \in [-0.3, 0.15]$. Note that we guarantee a distance of at least 0.15m between the obstacle's position and the initial position as well as at least 0.15m between the obstacle's position and the box's goal position.

The robot is controlled via torques on the joint level. We use four basis functions per DoF, resulting in a parameter space of 28 dimensions. We consider an episode successful if the box's orientation around the z-axis error is smaller than 0.5 rad and the position error is smaller than 0.05m.

The sparse-in-time reward function is up to a scaling parameter the same as presented in [Otto et al., 2023]. We describe the whole reward function in the following.

The box's distance to the goal position is

\[
R_\text{goal} = \lVert\mathbf{p}-\mathbf{p}_{goal}\rVert,
\]

where $\mathbf{p}$ is the box position and $\mathbf{p}_{goal}$ is the goal position. The rotation distance is defined as

\[
R_\text{rotation} =\frac{1}{\pi} \arccos{|\mathbf{r}\cdot\mathbf{r}_{goal}|},
\]

where $\mathbf{r}$ and $\mathbf{r}_{goal}$ are the box orientation and goal orientation quaternion respectively. The incentive to keep the rod within the box is defined as

\[
R_\text{rod} = \text{clip}(||\mathbf{p}-\mathbf{h}_{pos}||, 0.05, 10),
\]

where $\mathbf{h}_{pos}$ is the position of the rod tip. Similarly, to incentivize to maintain the rod in a desired rotation, the reward

\[
R_\text{rod\_rotation} = \text{clip} (\frac{2}{\pi} \arccos{|\mathbf{h}_{rot}\cdot\mathbf{h}_0|} , 0.25, 2)
\]

is defined, where $\mathbf{h}_{rot}$ and $\mathbf{h}_{0} = (0, 1, 0, 0)$ are the current and desired rod orientation in quaternion respectively. To incentivize the robot to stay within the joint and velocity bounds, the error

\begin{align*}
\text{err}(\mathbf{q}, \mathbf{\dot{q}}) &= \sum_{i \in \{i | |q_i| > |q_i^{b}|\}}{(|q_i|-|q^{b}_i|)} \\
&+ \sum_{j \in \{j | |\dot{q}_j| >|\dot{q}_j^{b}|\}}{(|\dot{q}_j|-|\dot{q}^{b}_j|)}
\end{align*}

is used, where $\mathbf{q}$, $\mathbf{\dot{q}}$, $\mathbf{q}^{b}$, and $\mathbf{\dot{q}}^{b}$ are the robot's joint positions and velocities as well as their respective bounds.
To learn low-energy motions, the per-time action (torque) cost

\begin{align*}
\tau_t = \sum_i^K (a^i_t)^2,
\end{align*}

is used. The resulting temporal sparse reward is given as

\begin{equation*}
R_\text{tot} = \begin{cases}
- R_\text{rod}
- R_\text{rod\_rotation}
- 0.02\tau_t
- \text{err}(\mathbf{q}, \mathbf{\dot{q}})
\\
- R_\text{rod}
- R_\text{rod\_rotation}
- 0.02\tau_t
- \text{err}(\mathbf{q}, \mathbf{\dot{q}})\\
-350 R_\text{goal}
\end{cases}
\end{equation*}

where $T=100$ is the horizon of the episode. The reward gives relevant information to solve the ask only in the last time step of the episode, which makes exploration hard.

Further Visualizations of learned skills. We show additional plots of the box's trajectories in the box pushing task in Fig. <ref>.

Hyperparameters are listed in the Table <ref>.

§.§ Extended 5-Link Reacher Task

Environment. In the 5-Link Reacher task, a 5-link planar robot has to reach a goal position with its tip. The reacher's initial position is straight to the right. This task is difficult to solve, as it introduces multi-modality in the behavior space.  [Otto et al., 2023] avoided this multi-modality by constraining the y coordinate of the goal position to $y\geq0$, i.e. the first two quadrants. We adopt the 5Link-Reacher task by increasing the context space to the full space, i.e. all four quadrants. We consider 5 basis functions per joint leading to a 25-dimensional parameter space. We consider the sparse reward function presented in [Otto et al., 2023] as

\begin{equation*}
R_\text{tot} = \begin{cases}
- \tau_t
\\
- \tau_t
-200 R_\text{goal}
-10 R_\text{vel}
\end{cases}
\end{equation*}

\[
R_\text{goal} = \lVert \mathbf{p}-\mathbf{p}_{goal}\rVert_2
\]

\begin{align*}
\tau_t = \sum_i^K (a^i_t)^2.
\end{align*}

The sparse reward only returns the task reward in the last time step T and additionally adds a velocity penalty $R_\text{vel} = \sum_i^K (\dot{q}_T^i)^2$. The joint velocities are denoted as$\mathbf{\dot{q}}$. This velocity penalty avoids overshooting in the last time step.

Hyperparameters are listed in the Table <ref>.

Additional Diverse Skills for the Box Push Obstacle Task learned by Di-SkilL. We fix the contexts and sample experts which we subsequently execute. This leads to diverse behaviors in the motion primitive parameter space$\mpparam$ which leads to different trajectories of the pushed box on the table.

[]IQM Mean Return HJ

[]IQM Max. Height Jump HJ

[]IQM Goal Distance HJ

Additional Analysis of the Hopper Jump (HJ) task.

Di-SkilL's Diverse Skills for the TT-H task. We fixed the ball's desired landing position and varied the serving landing position and the ball's initial velocity. Di-SkilL can return the ball in different striking types. Note that each row represents a different desired ball landing position.

§.§ Robot Mini Golf Task

Environment. In the robot mini golf task the agent needs to hit a ball while avoiding the two obstacles, such that it passes the tight goal to achieve a bonus. The context space consists of the ball's initial x-position $x_{ball}\in[0.25m, 0.6m]$, the XY positions of the green obstacle $x_{obs}\in [0.3, 0.6]$ and $y_{obs}\in[-0.5, -0.1]$ and the x positions of the goal $x_{ball}\in[0.25, 0.6]$. The parameter space is 29 dimensional resulting from the 4 basis functions per joint and an additional duration parameter which allows the robot to learn the duration of the trajectory. The robot starts always at the same position. The reward function consists of three stages:

\begin{equation*}
R_{task} = \begin{cases}
-0.0005 \cdot \tau_t & \text{if cond. 1}, \\
0.2-0.2\tanh{(\min{||\mathbf{p}_{r}-\mathbf{p}_{b}||})} & \text{if cond. 2}, \\
-\tanh{(||\mathbf{p}_{b,y}-\mathbf{p}_{thresh,y}||)} &\text{if cond. 3},\\
6 &\text{if cond. 3} ,
\end{cases}
\end{equation*}

where the individual conditions are

* cond. 1: the end of the episode is not reached,

* cond. 2: the end of the episode is reached and the robot did not hit the ball,

* cond. 3: the end of the episode is reached and the robot has hit the ball, but the ball didn't pass the goal

* cond. 4: the end of the episode is reached, robot has hit the ball and the ball has passed the goal for at least 0.75m

The episode ends when the maximum horizon length $T=100$ is achieved.
We again make use of the advantage that we obtain the whole desired trajectory ahead of the environment interaction, such that we can collect some samples without physical simulation. The reward function based on this desired trajectory is defined as

\begin{equation*}
r_{traj} = \sum_{(i,j)}{|\tau_{ij}^d| - |q^b_j|} , \quad
(i,j) \in \{(i,j)\mid |\tau_{ij}^d| > |q^b_j|\}
\end{equation*}

where $\tau^d$ is the desired trajectory, $i$ is the time index, $j$ is the joint index, $q^b$ is the joint position upper bound. The desired trajectory is considered as invalid if $r_{traj} < 0$, an invalid trajectory will not be executed on the robot. Additionally, we provide a punishment, if the agent samples invalid duration times

\begin{equation*}
r_{dur} = -3\left(\text{max}(0, t_d-t{d,max}) + \text{max}(0, t_{d,mint} - t_d) \right),
\end{equation*}

where $t_{d,max}=1.7s, t_{d,min} = 0.45s$ and $t_d$ is the duration in seconds chosen by the agent. The overall reward is defined as:

\begin{equation*}
r = \begin{cases}
r_{traj}, -20(r_{traj} + r_{dur})-5 \quad  &\text{if invalid duration,} \\
& \text{or trajectory} \\
r_{task}, & \text{otherwise.}
\end{cases}
\end{equation*}

Hyperparameters are listed in the Table  <ref>.

§ ADDITIONAL EVALUATIONS

We provide additional diverse skills to the Box Pushing Obstacle task in Fig. <ref>. In Fig. <ref> we provide additional diverse strikes to fixed ball's desired landing positions.

We analyze the performance of Di-SkilL on the hopper jump task in more detail. In Fig. <ref> we observe that the mean return is on par with BBRL, similar to the achieved goal distance in Fig. <ref>. However, there is a small gap in the max height, where BBRL jumps slightly higher (see Fig. <ref>. Given that the mean return is on par, one would expect that the maximum jump height is on par as well. However, Di-SkilL optimizes the remaining terms in the objective of the hopper jump task (see Appendix <ref>), which explains this gap.

§ HYPERPARAMETERS

add component every iteration     1000

fine tune all components every iteration     50

number component adds     1

number initial components     1

number total components     20

number traj. samples per component per iteration     200

$\alpha$     0.0001

$\beta$     0.5

expert KL-bound     0.01

context KL-bound     0.01

Hyperparameters for SVSL on TT

Di-SkilL     BBRL

critic activation     tanh     tanh

hidden sizes critic     [8,8]     [32, 32]

initialization     orthogonal     orthogonal

lr critic    0.0003     0.0003

optimizer critic     adam     adam

ciritc epochs     100      100

activation context distribution     tanh     –

epochs context distribution     100     –

hidden sizes context distr     [16,16]     –

initialization     orthogonal     –

lr context distribution     0.0001    –

optimizer context distr     adam     –

batch size per component    50     209

number samples from environment distribution     5000     –

number samples per component     50     209

normalize advantages     True     True

expert activateion     tanh     tanh

epochs     100     100

hidden sizes expert     [64]     [32]

lr policy     0.0003     0.0003

covariance type     full     full

alpha     0.001     –

beta     4     –

number components     5     –

covariance bound     0.005     0.001

mean bound    0.05     0.05

projection type     KL     KL

trust region coefficient     100     25

Hyperparameters for Di-SkilL and BBRL on TT.

Di-SkilL     BBRL     LinDi-SkilL     PPO

critic activation     tanh     tanh     tanh     tanh

hidden sizes critic     [32,32]     [32, 32]     [32, 32]     [32, 32]

initialization     orthogonal     orthogonal     orthogonal     orthogonal

lr critic    0.0003     0.0003     0.0003     0.0003

optimizer critic     adam     adam     adam     adam

ciritc epochs     100      100     100     10

activation context distribution     tanh     –     tanh     –

epochs context distribution     100     –     100     –

hidden sizes context distr     [16,16]     –     [16, 16]     –

initialization     orthogonal     –     orthogonal     –

lr context distribution     0.0001    –     0.0001     –

optimizer context distr     adam     –     adam     –

batch size per component    25     240     25     512 (32 minibatches)

number samples from environment distribution     5000     –    5000     –

number samples per component     25     240     25     16384

normalize advantages     True     True     True     True

expert activateion     tanh     tanh     –     tanh

epochs     100     100     100     10

hidden sizes expert     [32,32]     [64,64]     –     [32, 32]

lr policy     0.0003     0.0003     0.0003     0.0003

covariance type     full     full     full     diagonal

alpha     0.01     –     0.01     –

beta     8     –     8    –

number components     10     –     10     –

covariance bound     0.001     0.005     0.0005     –

mean bound    0.05     0.05     0.05     –

projection type     KL     KL     KL     –

trust region coefficient     100     25     100     –

discount factor     1     1     1    1

Hyperparameters for Di-SkilL, BBRL, LinDi-SkilL, and PPO on 5LR. We used all code-level optimization [Engstrom et al., 2020] needed for PPO. The implementation is based on the source code from [Otto et al., 2021].

Di-SkilL     BBRL     LinDi-SkilL

critic activation     tanh     tanh     tanh

hidden sizes critic     [8,8]     [32, 32]     [8,8]

initialization     orthogonal     orthogonal     orthogonal

lr critic    0.0003     0.0003     0.0003

optimizer critic     adam     adam     adam

ciritc epochs     100      100     100

activation context distribution     tanh     –     tanh

epochs context distribution     100     –     100

hidden sizes context distr     [16,16]     –     [16, 16

initialization     orthogonal     –     orthogonal

lr context distribution     0.0001    –    0.0001

optimizer context distr     adam     –     adam

batch size per component    50     209    50

number samples from environment distribution     5000     –     5000

number samples per component     50     209     50

normalize advantages     True     True     True

expert activateion     tanh     tanh     –

epochs     100     100

hidden sizes expert     [128]     [32,32]     –

lr policy     0.0003     0.0003     0.0003

covariance type     full     full     full

alpha     0.001     –     0.001

beta     0.5     –     0.5

number components     10     –     10

covariance bound     0.005     0.0005    0.001

mean bound    0.05     0.05     0.05

projection type     KL     KL     KL

trust region coefficient     100     25    100

Hyperparameters for Di-SkilL, BBRL, and LinDi-SkilL for the Hard Table Tennis Task (TT-H).

Di-SkilL     BBRL     LinDi-SkilL

critic activation     tanh     tanh     tanh

hidden sizes critic     [64,64]     [64, 64]     [64,64]

initialization     orthogonal     orthogonal     orthogonal

lr critic    0.0001     0.0001     0.0001

optimizer critic     adam     adam     adam

ciritc epochs     100      100     100

activation context distribution     tanh     –     tanh

epochs context distribution     100     –     100

hidden sizes context distr     [16,16]     –     [16, 16]

initialization     orthogonal     –     orthogonal

lr context distribution     0.0001    –    0.0001

optimizer context distr     adam     –     adam

batch size per component    80     200    80

number samples from environment distribution     1000     –     1000

number samples per component     80     200     80

normalize advantages     True     True     True

expert activateion     tanh     tanh     –

epochs     100     100    100

hidden sizes expert     [32, 32]     [32,32]     –

lr policy     0.0003     0.0003     0.0003

covariance type     full     full     full

alpha     0.01     –     0.01

beta     8     –     8

number components     3     –     3

covariance bound     0.005     0.05    0.005

mean bound    0.05     0.1     0.05

projection type     KL     KL     KL

trust region coefficient     100     25    100

Hyperparameters for Di-SkilL, BBRL, and LinDi-SkilL for the Hopper Jump Task (HJ).

Di-SkilL     BBRL     LinDi-SkilL     PPO

critic activation     tanh     tanh     tanh     tanh

hidden sizes critic     [32,32]     [32, 32]     [32, 32]     [256, 256]

initialization     orthogonal     orthogonal     orthogonal     orthogonal

lr critic    0.0003     0.0003     0.0003     0.0001

optimizer critic     adam     adam     adam     adam

ciritc epochs     100      100     100     10

activation context distribution     tanh     –     tanh     –

epochs context distribution     100     –     100     –

hidden sizes context distr     [16,16]     –     [16, 16]     –

initialization     orthogonal     –     orthogonal     –

lr context distribution     0.0001    –     0.0001     –

optimizer context distr     adam     –    adam      –

batch size per component    50    500     50     410 (40 minibatches)

number samples from environment distribution     5000     –     5000     –

number samples per component     50     500     50     16384

normalize advantages     True     True     True    True

expert activateion     tanh     tanh     –    tanh

epochs     100     100     100     10

hidden sizes expert     [64,64]     [64,64     –     [256, 256]

lr policy     0.0003     0.0003     0.0003     0.0001

covariance type     full     full     full     diagonal

alpha     0.01     –     0.0001     –

beta     64     –     64    –

number components     10     –     10     –

covariance bound     0.005     0.0005     0.001     –

mean bound    0.05     0.05     0.05     –

projection type     KL     KL     KL     –

trust region coefficient     100     25     100     –

discount factor     1     1     1    1

Hyperparameters for Di-SkilL, BBRL, LinDi-SkilL, and PPO for Box Pushing Obstacle task (BPO). We used all code-level optimization [Engstrom et al., 2020] needed for PPO. The implementation is based on the source code from [Otto et al., 2021].

Di-SkilL     BBRL     LinDi-SkilL

critic activation     tanh     tanh     tanh

hidden sizes critic     [32,32]     [32, 32]     [32, 32]

initialization     orthogonal     orthogonal     orthogonal

lr critic    0.0003     0.0003     0.0003

optimizer critic     adam     adam     adam

ciritc epochs     100      100     100

activation context distribution     tanh     –     tanh

epochs context distribution     100     –     100

hidden sizes context distr     [16,16]     –     [16, 16]

initialization     orthogonal     –     orthogonal

lr context distribution     0.0001    –     0.0001

optimizer context distr     adam     –     adam

batch size per component    50     500     50

number samples from environment distribution     5000     –     5000

number samples per component     50     500     50

normalize advantages     True     True     True

expert activateion     tanh     tanh     –

epochs     100     100     100

hidden sizes expert     [64,64]     [128,128]     –

lr policy     0.0003     0.0003     0.0003

covariance type     full     full     full

alpha     0.0001     –     0.0001

beta     1     –     1

number components     10     –     10

covariance bound     0.005     0.001    0.001

mean bound    0.05     0.05    0.01

projection type     KL     KL     KL

trust region coefficient     100     25     100

Hyperparameters for Di-SkilL, BBRL, and LinDi-SkilL for the mini golf task.