# Learning a Distributed Hierarchical Locomotion Controller for Embodied
Cooperation

Chuye Hong
Tsinghua University, China
<EMAIL_ADDRESS>
&Kangyao Huang
Tsinghua University, China
<EMAIL_ADDRESS>&Huaping Liu
Tsinghua University, China
<EMAIL_ADDRESS>

###### Abstract

In this work, we propose a distributed hierarchical locomotion control
strategy for whole-body cooperation and demonstrate the potential for
migration into large numbers of agents. Our method utilizes a hierarchical
structure to break down complex tasks into smaller, manageable sub-tasks. By
incorporating spatiotemporal continuity features, we establish the sequential
logic necessary for causal inference and cooperative behaviour in sequential
tasks, thereby facilitating efficient and coordinated control strategies.
Through training within this framework, we demonstrate enhanced adaptability
and cooperation, leading to superior performance in task completion compared
to the original methods. Moreover, we construct a set of environments as the
benchmark for embodied cooperation. A visual depiction of highlights of the
learned cooperative behaviours can be found on this website111Videos of our
results are available at our project webpage: https://d-hrl.github.io.

> Keywords: cooperation, locomotion, hierarchical reinforcement learning

##  1 Introduction

Cooperatively accomplishing embodied tasks by multiple robots has consistently
been a highly challenging area of research. Recent studies mainly focus on
embodied manipulation cooperation among robotic arms or formation control over
the upper level within a group of mobile robots [1, 2]. Nevertheless, multi-
agent cooperation via whole-body and end-to-end locomotion control is rarely
studied.

Some previous works showcase the manipulation via locomotion [3] but are only
tested on two agent systems, and the scalability of this method is still
agnostic for migration to any number of agent populations. In this work, we
aim to realize more complex embodied multi-agent cooperation by learning a
distributed hierarchical locomotion control system, decomposing the complex
and coupled behaviours while maintaining the potential for unlimited expansion
on the swarm. As the foundation for implementation and validation, we
construct three scenarios in IsaacSim/Gym [4] as benchmarks for embodied
cooperation study.

(a) Cooperative Transport

(b) Corridor Crossing

(c) Ravine Bridging

Figure 1: Embodied Cooperation Environments.

Concurrently, training a robot for a specific function can be effectively
achieved through reinforcement learning (RL), like learning movement patterns
[5], interactive behaviours [6], as well as logical inference in games [7].
Although RL provides a recognized powerful exploration capability and
tremendous progress has been made in sampling efficiency [4], finding and
mastering a sequence of sophisticated tasks through searching remains a
challenging problem. Hierarchical reinforcement learning (HRL) alleviates this
to a certain extent, aiming to understand the logical relationships among
”control, action, behaviour, dynamic outcomes, and feedback” in a segmented
manner. An increasing number of studies are employing hierarchical approaches
to decompose complex tasks [8, 9, 10, 11, 12, 13, 14, 15]. Inspired by these,
we propose a three-level HRL to naturally decompose perception feature
extraction, cooperative behaviours, and fundamental actions.

In the previous work [3], embodied cooperation is completed via locomotion
control by two quadruped robots. However, spatiotemporal features between a
sequence of events are not considered in this work, which causes failure in
cooperation of more than two agents. Actually, understanding interactive and
cooperative behaviours requires sequential causal reasoning considering world
dynamics. To enhance this, we deploy sequential behaviour learning with
spatiotemporal memory capabilities in the middle layer of the hierarchical
network by constructing a Recurrent Neural Network (RNN). Furthermore, the
previous method can hardly be applied to a flock of agents [3], while the
scalability of the swarm is crucial in multi-agent reinforcement learning
(MARL) field [16, 17]. Herein, we construct our method facing the
decentralized and distributed scenario and attempt to apply it to an
expandable MAS.

The key contributions of this work can be described as follows:

  * •

A distributed HRL locomotion control framework is constructed for scale-
expandable embodied cooperation. This architecture can be applied to both
homogeneous and heterogeneous groups.

  * •

We build a set of scenarios for whole-body embodied cooperation in IsaacSim
shown in Fig. 1, as a benchmark for embodied cooperation.

  * •

We showcase the remarkable emergence of cooperative behaviours among agents
and demonstrate the advantages compared with previous approaches.

Figure 2: Distributed HRL pipeline: demonstrate the information flow within
the distributed hierarchical reinforcement learning, using scenario
Cooperative Transport as an example. Here we maintain a centralized training
but distributed control hierarchical reinforcement learning framework, where
we decompose the complex behaviours into three levels: Upper Layer (UL),
Middle Layer (ML), and Lower Layer (LL). UL module processes the external
information $e_{t}$ including environmental perception and relative position
to colleagues, extracting features and sending to ML; ML module is a recurrent
neural network layer that maintains a recurrent state $h$ considering temporal
and spatial correlation, and outputs a locomotion command into LL module; LL
module is a pre-trained locomotion control layer that generates action $a_{t}$
and applies it to agent according to the proprioceptive observation $p_{t}$,
where we have two modes: position and velocity. The goal of Cooperative
Transport is to move a cylinder object to the red target zone collaboratively
by a group of Ant robots.

##  2 Related Works

###  2.1 Embodied Cooperation

Research on cooperation can be divided according to with or without body
interaction. Some collaborative tasks are based on particle games, focusing
solely on coordination and alignment at position/velocity levels [18, 19]. In
contrast, studies related to embodied cooperation lie on the foundations of
the robotic platform that owns the real body, paying more attention to
interaction, collision, and whole-body control. Optimization methods and
computational intelligence methods can solve embodied cooperation problems.
The model predicted control (MPC) based methods are widely used in formation
control on quadruped robots [20, 21, 22, 23] and aerial robots [24, 25], and
nature-inspired algorithms can obtain good performance in a self-organized
swarm cooperative task [2]. These approaches provide highly accurate control
but lack flexibility and generalization. Learning-based methods can compensate
for this [26, 3]. Moreover, the large language model (LLM) can provide robots
with powerful cognitive reasoning capabilities for collaboration, bringing a
new way to solve the problem [27, 28, 29, 30].

###  2.2 HRL in locomotion Control

HRL is well suited for complex and spatiotemporal tasks [31]. Its
effectiveness has been proven and widely acknowledged [32]. Multiple steps can
decouple an intricate task to reduce the training difficulty, corresponding to
various layers of policies [33]. Decisions between layers can be made by
deciding whether the previous action has been completed [34]. Essentially,
hierarchical learning provides artificial direction and reduces the need for
model to have a thorough understanding of the scenario and the tasks,
preventing RL from wandering in a sparse search space.

Furthermore, HRL is inherently well-suited for tasks that involve a
progressive logical relationship [35, 36]. A typical work fuses simplified
environmental observation and proprioceptive state then generates lower-level
control commands for actuators[37]. Some recent attractive benchmarks like
HumanoidBench [38], also divide tasks into planning and reaching control and
learning hierarchically.

##  3 Problem Statement

We first formulate the problem of learning embodied cooperative behaviour in a
swarm of agents, then try to solve this problem with MARL. Since the number of
agents is fixed within an episode, the synchronous framework and policy
gradient descent algorithms are selected. To realize real-time execution, we
further select the Centralized Training and Decentralized Execution (CTDE)
framework to support our implementation. It is crucial to mention that agents
in the group might be homogeneous or heterogeneous, and agents in the same
species share the same policy.

The embodied cooperation problem can be formulated as a Markov Decision
Process (MDP) mathematically expressed as
$\left\langle\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R}\right\rangle$
including $N$ agents in $M$ heterogeneous species. $\mathcal{S},\mathcal{A}$
are state and action tuples, respectively, where
$\mathcal{S}=(S_{1},S_{2},\cdots,S_{N})$,
$\mathcal{A}=(A_{1},A_{2},\cdots,A_{N})$. $\mathcal{T}$ is the state
transition function, and $\mathcal{T}\left(s^{\prime}|s,a\right)$ represents
the transition probability distribution of the system when transiting into the
next states
$s^{\prime}=(s^{\prime}_{1},s^{\prime}_{2},\cdots,s^{\prime}_{N})\in\mathcal{S}$
from states $s=(s_{1},s_{2},\cdots,s_{N})\in\mathcal{S}$ after taking a set of
actions $a=(a_{1},a_{2},\cdots,a_{N})\in\mathcal{A}$. Besides,
$\mathcal{R}=(R_{1},\cdots,R_{M})$ denotes the tuple of reward functions in
which different types of agents correspond to distinct reward functions.
$r_{i}={R_{i}}\left(s_{i},a_{i}\right)$ is the reward given to agent $i$ after
taking actions $a_{i}$ under states $s$.

For the homogenous multiple agents group, the reward function $R$ is
determined and shared for each agent. The aim is to find the best shared
policy $\pi^{*}$, to maximize the expected average value of cumulative
discount reward:

$\displaystyle\pi^{*}=\mathrm{arg}\underset{\pi}{\mathrm{max}}J_{\pi}=\frac{1}{N}\sum_{i=1}^{N}\sum_{t=0}^{\infty}\gamma^{t}\cdot\mathbb{E}\left[R_{i}(s_{t}^{i},a_{t}^{i})|\pi_{i}\right]$
(1)

where $s_{t}^{i}\in S_{i}$ and $a_{t}^{i}\in A_{i}$ are the state and action
of the $i$-th agent. There are some subtle differences in the representation
of the heterogeneous population. The aim is to find the best policy tuple
$\Omega^{*}=(\pi_{1},\cdots,\pi_{M})$ to maximize the joint multiplication of
costs $J$:

$\displaystyle\Omega^{*}=\mathrm{arg}\underset{\Omega}{\mathrm{max}}\prod_{m=1}^{M}J_{m},\;\mathrm{where}\;J_{m}=\frac{1}{K}\sum_{i=1}^{K}\sum_{t=0}^{\infty}\gamma^{t}\cdot\mathbb{E}\left[R_{m}(s_{t}^{i},a_{t}^{i})|\pi_{m}\right]$
(2)

where $J_{m}$ denotes the cost function of type $m$ species, and $K$ is the
agent number of this species. The final optimization cost for the
heterogeneous population is a form of co-multiplication because task
assignment exists among the entire team. Each subtask must be completed
otherwise the overall task will fail. Even if some subtasks obtain high
rewards, any failure in a subtask will prevent the achievement of the overall
goal.

##  4 Distributed Hierarchical Reinforcement Learning

As aforementioned, locomotion control for embodied cooperation is an extremely
challenging task. There are three main difficulties: (i) How to decompose and
decouple the complicated tasks into a step-by-step mode? (ii) How to establish
the temporal and spatial correlation of behaviours? (iii) Whether the trained
models can expand to a large scale of multiagent collaboration? In this part,
we will address each of these three questions in detail.

###  4.1 Task Decomposition and Hierarchical Learning

Network Structure: Traditionally, a hierarchical neural network structure
divides tasks into control and decision layers. Unlike this, in order to
address collaborative tasks, we propose a deeper hierarchical structure
consisting of three layers. We designate perception-related tasks as the upper
layer of the three-layer network that extracts exteroceptive observation
including other agents’ information into a feature vector; then, we define the
middle layer of the network as the planning and decision layer, which provides
spatiotemporally correlated commands to the lower-level controller; finally,
the lower-layer network focuses on the execution of commands, which can be
pre-trained in various modes. The overall information flow in the proposed
hierarchical network is illustrated in Fig. 2.

Hierarchical Perception Processing: Each layer of the network is responsible
for processing distinct types of raw data, which relates to how we acquire and
classify perceptual information. Proprioceptive perceptual states including
local position, velocity, and joint torque are closely associated with
personal locomotion control. Therefore, the proprioceptive state $p$ should
only processed by the lower-layer policy. Furthermore, the exteroceptive state
$e$ formed by concatenating environmental perception which provides terrain
information in the form of a height measurement array [39] with relative
observations of other agents [5], as depicted in Fig.3.

Figure 3: Exteroceptive state configuration in distributed HRL.

###  4.2 Spatiotemporal Memory Recurrence

Capturing and leveraging spatiotemporal features is significant for long-term
decision-making tasks [40]. For tasks requiring cooperation, spatial awareness
is as crucial as understanding temporal dynamics. We propose spatiotemporal
memory recurrence that uses RNNs to enhance the agent’s ability to make
informed decisions over extended periods and complex environments.

The significance of learning spatiotemporal features through RNNs in
distributed HRL lies in the inherent capability of these networks to maintain
and utilize historical information. This capability is particularly important
in dynamic environments, especially in nonstationary multiagent environments.
RNNs, on the other hand, are designed to handle sequential data, making them
ideal for tasks that require an understanding of the temporal progression of
states and actions. Thus, we maintain a hidden recurrent state $h$ to pass the
feature of scenarios continuously, as shown in Fig. 2.

###  4.3 Distributed Scalable HRL

In contrast to previous studies that employed HRL or end-to-end approaches for
robotic cooperative manipulation or locomotion, our distributed HRL method
considers the capability of scaling the population. To achieve this, we need
to strictly define the format of input and output information for each agent,
particularly the observation. Herein, we propose to use partial observation of
agents related to their colleagues, according to their visibility [41]. Each
agent can only obtain the nearest neighbour’s local position. As illustrated
in Fig.2, the dimension of exteroceptive state $e$ is limited. This actually
changes the difficulty of collaboration learning. Since partially observed
agents can only see nearby peers and environmental information, they need to
estimate the states of other invisible agents from the dynamics of the
environment and predict future changes in the environment.

###  4.4 Training Curriculum

Acquiring collaborative behaviours requires the agents’ random search to span
a sufficiently large space, which in complex tasks can often lead to an
inability to learn usable knowledge due to the vastness of the search space.
In this case, we design some appropriate training curricula that reasonably
use sparse rewards and dense rewards in the early stages to intentionally
guide the agent to approach success. This curriculum is independent of HRL
pertaining, such as encouraging agents to move to the target first at the
early stage. The curriculum designs are described in detail in the appendix.

##  5 Experiments

The experiments are conducted on a workstation with Intel Xeon W-2150b and
RTX2080Ti. Independent Proximal Policy Optimization (IPPO) in a multi-agent
setting is employed as the algorithm. Adam optimizer is used with a learning
rate of 0.0005. PPO clipping is 0.2, the discount factor is 0.995. We could
complete the training within several hours thanks to IsaacSim GPU parallel
sampling via multiple environments [4].

###  5.1 Environments Construction

Three scenarios are established that represent different task types:
Cooperative Transport, Corridor Crossing, and Ravine Bridging.

Cooperative Transport: Transporting an object collaboratively is a classical
task in the multiagent system field. Herein, we build an embodied cooperative
transport task via locomotion control. A group of Ant attempt to move a
cylinder object to the red target area, shown in Fig.1(a). We consider fully
decentralised and distributed cooperation in a homogeneous and self-organized
flock. Thus, the scale of flock can have theoretically unlimited expansion.

Corridor Crossing: A group of agents navigate through a narrow corridor
sequentially, aiming to complete the passage as quickly as possible. Only one
person can pass through the slit at any one time. Agents in this scenario
should adjust their position and speed reasonably according to the surrounding
observations and pass through a narrow passage without collision. Cooperation
in formation is required, as shown in Fig. 1(b). The number of agents can also
be expanded in this task.

Ravine Bridging: As illustrated in Fig. 1(c), two agents push a movable
bridge, enabling another group of agents to cross the chasm smoothly via the
bridge. There are two categories of agents in this scenario, which is
different from the other two environments. Agents in Ravine Bridging are
heterogeneous, which means we should maintain more than one policy for
different missions.

###  5.2 Results of Distributed HRL

####  5.2.1 Training Basic Locomotion Control

In the training of lower-level motion control, two control paradigms are
available: velocity control and position control. We first train the two
control paradigms in the single-agent environment to obtain accurate control
of the fundamental actions. We give a reciprocal $L_{2}$ distance between the
current position and the target position, or the dot product of the agent
velocity and target velocity, as the main reward in each training curriculum.
Fig 4(a) shows the training results of two paradigms. We find that the
velocity control offers greater flexibility than position control. In our
empirical tests, however, position control presents higher accuracy. Moreover,
velocity control makes it difficult to maintain a stationary position at a
specific point. Similar to any other hierarchical learning study, the lower
layer is frozen once its training is done well.

(a) Basic Locomotion (b) Cooperative Transport (c) Corrider Crossing (d)
Ravine Bridging

Figure 4: Distributed HRL results for all scenarios.

####  5.2.2 Training the Interactive Collaboration

After the lower layer is elaborately trained, we freeze this network and fix
the basic locomotion control performance. Then, the upper and middle layers
are trained on different scenes via distinct reward mechanisms. The training
results are shown in Fig. 4(b,c,d).

Cooperative Transport: There emerge significantly typical collaborative
behaviours in this task. Agents demonstrate a very clear understanding of the
situation, being able to actively choose the tasks they should undertake based
on their circumstances. Some agents first converge towards the object and
slightly adjust their position for collaboration; meanwhile, other agents move
directly towards the target and prepare to decelerate the object, ensuring it
falls within the target area accurately and stably. The success rate of this
task can reach about 75$\%$.

Corridor Crossing: Agents in this task acquire natural characteristics:
”sacrifice” or ”concession”. If we directly apply the model pre-trained under
a single-agent scenario to a multiple-agent scenario, all agents will get
stuck in the corridor, and no one can move. After training, agents learn to
observe the relative positions of other agents to adjust their speed, choosing
the appropriate timing for moving or waiting. Generally speaking, agents born
in the middle will rush directly through the corridor, while others reach the
entrance of the corridor at a slightly slower speed and then accelerate
through after observing the passage of agents in front of them. We observe a
more than 90 $\%$ success rate on this scenario with four Ant agents.

Ravine Bridging: The group in this task is a heterogeneous team. Although all
agents are Ant, there are two sets of policies: one for agents on the plane
and one for agents in the ravine. Agents on the plane will arrive at the edge
of ravine and stop, waiting for the agents in the ravine to transport the
bridge to the front of them. We get approximately half success over one
hundred trails. The main reason is that Ants in the ravine might get stuck or
fall.

Table 1: Success Rate Comparison: We present success rates with standard error on the three tasks, comparing our method to ablations. Each percentage results from 50 trials—10 attempts per three independently trained models. Success rate | HRL | No Hierarchy | No Spatiotemporal Memory
---|---|---|---
Cooperative Transport | $\textbf{74}\pm 7\%$ | $0\pm 0\%$ | $5\pm 1\%$
Corridor Crossing | $\textbf{93}\pm 3\%$ | $0\pm 0\%$ | $2\pm 1\%$
Ravine Bridging | $\textbf{48}\pm 5\%$ | $0\pm 0\%$ | $0\pm 0\%$

Figure 5: Illustration of hierarchical learning concepts applied in different
scenarios compared with no hierarchy case, noisy hidden states case and the
baseline [3] which does not have the middle layer with recurrent neural
networks. All results are conducted on four agents scenario.

###  5.3 Ablation

####  5.3.1 The role of Hierarchical Learning

In this experiment, we replace the hierarchical networks with a single-layer
MLP, shown in Fig. 5. We find the reward converges to an extremely low level
quickly. Agents can only learn and get the basic reward of locomotion. These
results show that hierarchy is crucial to success in more complex tasks that
require coordination. Without hierarchy, agents cannot complete the mission
(in Table. 1). The reason for failure is obvious. As with other HRL
objectives, a single network facing high-dimensional observations and complex
multi-item reward functions finds it difficult to distinguish features between
channels and learn behaviour guided by a single main reward.

For multi-agent training, the situation is even more unique. Due to the CTDE
training paradigm we default to, differences in the positions of multiple
bodies lead to an inability to uniformly learn certain distance rewards. For
example, in the task of cooperative transport, agents have different positions
compared to objects and targets, so a unified strategy cannot be directly
formed by lower-level networks. It can be said that hierarchical learning is
indispensable for such group tasks.

####  5.3.2 The effectiveness of Spatiotemporal Memory

To validate the effectiveness of spatiotemporal features, we re-product the
method in previous work [3] as the baseline, which does not consider the event
correlation. As shown in Fig.5, our method obtains a much higher reward for
all three tasks. The approach proposed in [3] which does not have RNNs,
converges to a slightly higher reward than in the case of no hierarchy but is
still far from completing the mission. Furthermore, we attempt to replace the
RNN hidden states with random noise inputs to explain how event correlation
positively and effectively affects the entire performance. The success rate of
all these attempts is close to zero. We find that agents cannot learn a set of
systematic behaviours in specific tasks, and there are occasional successes
when the object position is really close to the target. Actually, if the
agents only react to the current observation, they could never form strategic
behaviours. In other words, he would never think for himself in the next
second without spatiotemporal memory. This is even more deadly in
collaborative tasks, as we can obviously find that the conciliatory and
sacrificial behaviour emerging in crossing corridor tasks is a strategy of
thinking for long-term consequences.

###  5.4 Analysis

Scalability Analysis: We validate scalability in the task cooperative
transport as illustrated in Fig. 6(a). The final reward increases as the
number of agents increases, which also aligns with our expectations for this
task. More agents participating should indeed bring better transportation
results. Our results reveal that the distributed HRL has great potential to
migrate a trained policy to a large number of agents. Only such algorithms can
achieve the true sense of ”biocoenosis” collaborative tasks.

(a) Training results for different Ant populations (b) Distributed HRL (c) No
hierarchy (d) No memory

Figure 6: (a) shows the training results of different Ant populations used in
Cooperative Transport. (b)(c)(d) shows the success and failure distribution of
task Cooperative Transport.

Typical Result Analysis: We analyse the typical result of task cooperative
transport. To test the robustness of our method, we conduct more than one
hundred trials with randomized reset positions and count the success rate then
plot the scatter diagram, as illustrated in Fig. 6(b). Each point represents
an initialized position of an agent, and the object is located at (0,0) point.
Red points denote agents generated at these positions fail to complete the
task, while each blue point denotes a success. We can observe that when the
initial position is closer to the object, the success rate tends to be higher,
and there is only one failure case within a 5-meter range of the object;
conversely, the success rate decreases as the distance from the object
increases. This is because the probability of accidents is greatly reduced by
short-distance tasks. It is worth mentioning that most of the failure cases
result from the agent tipping over or pushing the object obviously to the side
with uneven force during the pushing process. Fig.6(c)(d) showcase the
evaluation results under no-hierarchy and no-spatiotemporal memory. Only few
successes can be found in the evaluation of no-spatiotemporal memory, and all
cases fail in training without hierarchy.

##  6 Conclusion

In this work, we propose a distributed HRL architecture for embodied
cooperation, and the results showcase the effectiveness of our method in
embodied and whole-body control collaborative tasks. Besides, our approach has
significant potential in scaling up the flock size. In the future, more
complex cooperative tasks can be conducted based on our method.

#### Acknowledgments

If a paper is accepted, the final camera-ready version will (and probably
should) include acknowledgments. All acknowledgments go at the end of the
paper, including thanks to reviewers who gave useful comments, to colleagues
who contributed to the ideas, and to funding agencies and corporate sponsors
that provided financial support.

## References

  * Huang et al. [2022] K. Huang, J. Chen, J. Oyekan, and X. Zhang.  Bio-inspired Multi-agent Model and Optimization Strategy for Collaborative Aerial Transport.  _Lecture Notes in Electrical Engineering_ , 801 LNEE:591–598, 2022.  ISSN 18761119.  doi:10.1007/978-981-16-6372-7_64.
  * Huang et al. [2021] K. Huang, J. Chen, and J. Oyekan.  Decentralised aerial swarm for adaptive and energy efficient transport of unknown loads.  _Swarm and Evolutionary Computation_ , 67, 2021.  ISSN 22106502.  doi:10.1016/j.swevo.2021.100957.
  * Nachum et al. [2019] O. Nachum, M. Ahn, H. Ponte, S. Gu, and V. Kumar.  Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real.  _Proceedings of Machine Learning Research_ , 100:110–121, 2019.  ISSN 26403498.
  * Makoviychuk et al. [2021] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State.  Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning.  2021\.  URL http://arxiv.org/abs/2108.10470.
  * Huang et al. [2024a] K. Huang, D. Guo, X. Zhang, X. Ji, and H. Liu.  Stimulate the Potential of Robots via Competition.  2024a.  URL http://arxiv.org/abs/2403.10487.
  * Huang et al. [2024b] K. Huang, D. Guo, X. Zhang, X. Ji, and H. Liu.  CompetEvo: Towards Morphological Evolution from Competition.  2024b.  URL http://arxiv.org/abs/2405.18300.
  * Yu et al. [2022] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu.  The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games.  _Advances in Neural Information Processing Systems_ , 35, 2022.  ISSN 10495258.
  * Wang et al. [2024] R. Wang, D. Zhao, A. Gupte, and B. C. Min.  Initial Task Allocation in Multi-Human Multi-Robot Teams: An Attention-Enhanced Hierarchical Reinforcement Learning Approach.  _IEEE Robotics and Automation Letters_ , 9(4):3451–3458, 2024.  ISSN 23773766.  doi:10.1109/LRA.2024.3366414.
  * Liu et al. [2023] X. Liu, G. Wang, Z. Liu, Y. Liu, Z. Liu, and P. Huang.  Hierarchical Reinforcement Learning Integrating With Human Knowledge for Practical Robot Skill Learning in Complex Multi-Stage Manipulation.  _IEEE Transactions on Automation Science and Engineering_ , 2023.  ISSN 15583783.  doi:10.1109/TASE.2023.3288037.
  * Chang et al. [2023] L. Chang, L. Shan, W. Zhang, and Y. Dai.  Hierarchical multi-robot navigation and formation in unknown environments via deep reinforcement learning and distributed optimization.  _Robotics and Computer-Integrated Manufacturing_ , 83, 2023.  ISSN 07365845.  doi:10.1016/j.rcim.2023.102570.
  * Hao et al. [2024] C. Hao, C. Weaver, C. Tang, K. Kawamoto, M. Tomizuka, and W. Zhan.  Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning.  _IEEE Robotics and Automation Letters_ , 9(4):3625–3632, 2024.  ISSN 23773766.  doi:10.1109/LRA.2024.3368231.
  * Wang et al. [2023] S. Wang, Z. Wu, X. Hu, Y. Lin, and K. Lv.  Skill-Based Hierarchical Reinforcement Learning for Target Visual Navigation.  _IEEE Transactions on Multimedia_ , 25:8920–8932, 2023.  ISSN 19410077.  doi:10.1109/TMM.2023.3243618.
  * Triantafyllidis et al. [2023] E. Triantafyllidis, F. Acero, Z. Liu, and Z. Li.  Hybrid hierarchical learning for solving complex sequential tasks using the robotic manipulation network ROMAN.  _Nature Machine Intelligence_ , 5(9):991–1005, 2023.  ISSN 25225839.  doi:10.1038/s42256-023-00709-2.
  * Huang et al. [2023] X. Huang, Z. Li, Y. Xiang, Y. Ni, Y. Chi, Y. Li, L. Yang, X. B. Peng, and K. Sreenath.  Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning.  _IEEE International Conference on Intelligent Robots and Systems_ , pages 2715–2722, 2023.  ISSN 21530866.  doi:10.1109/IROS55552.2023.10341936.
  * Cao et al. [2024] J. Cao, L. Dong, and C. Sun.  Hierarchical reinforcement learning for kinematic control tasks with parameterized action spaces.  _Neural Computing and Applications_ , 36(1):323–336, 2024.  ISSN 14333058.  doi:10.1007/s00521-023-08991-2.
  * Mohanty et al. [2020] S. Mohanty, E. Nygren, F. Laurent, M. Schneider, C. Scheller, N. Bhattacharya, J. Watson, A. Egli, C. Eichenberger, C. Baumberger, G. Vienken, I. Sturm, G. Sartoretti, and G. Spigler.  Flatland-RL : Multi-Agent Reinforcement Learning on Trains.  2020\.  URL http://arxiv.org/abs/2012.05893.
  * Parvini et al. [2023] M. Parvini, M. R. Javan, N. Mokari, B. Abbasi, and E. A. Jorswieck.  AoI-Aware Resource Allocation for Platoon-Based C-V2X Networks via Multi-Agent Multi-Task Reinforcement Learning.  _IEEE Transactions on Vehicular Technology_ , 72(8):9880–9896, 2023.  ISSN 19399359.  doi:10.1109/TVT.2023.3259688.
  * Yang et al. [2018] Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang.  Mean field multi-agent reinforcement learning.  _35th International Conference on Machine Learning, ICML 2018_ , 12:8869–8886, 2018.
  * Ma et al. [2021] X. Ma, Y. Yang, C. Li, Y. Lu, Q. Zhao, and J. Yang.  Modeling the interaction between agents in cooperative multi-agent reinforcement learning.  _Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS_ , 2:853–861, 2021.  ISSN 15582914.
  * Xu et al. [2023] S. Xu, W. Zhang, L. Zhu, and C. P. Ho.  Distributed Model Predictive Formation Control with Gait Synchronization for Multiple Quadruped Robots.  _Proceedings - IEEE International Conference on Robotics and Automation_ , 2023-May:9995–10002, 2023.  ISSN 10504729.  doi:10.1109/ICRA48891.2023.10161260.
  * Kim et al. [2023] J. Kim, R. T. Fawcett, V. R. Kamidi, A. D. Ames, and K. A. Hamed.  Layered Control for Cooperative Locomotion of Two Quadrupedal Robots: Centralized and Distributed Approaches.  _IEEE Transactions on Robotics_ , 39(6):4728–4748, 2023.  ISSN 19410468.  doi:10.1109/TRO.2023.3319896.
  * Yang et al. [2022] C. Yang, G. N. Sue, Z. Li, L. Yang, H. Shen, Y. Chi, A. Rai, J. Zeng, and K. Sreenath.  Collaborative Navigation and Manipulation of a Cable-Towed Load by Multiple Quadrupedal Robots.  _IEEE Robotics and Automation Letters_ , 7(4):10041–10048, 2022.  ISSN 23773766.  doi:10.1109/LRA.2022.3191170.
  * Ma et al. [2021] W. L. Ma, N. Csomay-Shanklin, S. Kolathaya, K. A. Hamed, and A. D. Ames.  Coupled Control Lyapunov Functions for Interconnected Systems, with Application to Quadrupedal Locomotion.  _IEEE Robotics and Automation Letters_ , 6(2):3761–3768, 2021.  ISSN 23773766.  doi:10.1109/LRA.2021.3065174.
  * Quan et al. [2023] L. Quan, L. Yin, T. Zhang, M. Wang, R. Wang, S. Zhong, X. Zhou, Y. Cao, C. Xu, and F. Gao.  Robust and Efficient Trajectory Planning for Formation Flight in Dense Environments.  _IEEE Transactions on Robotics_ , 39(6):4785–4804, 2023.  ISSN 19410468.  doi:10.1109/TRO.2023.3301295.
  * Papaioannou et al. [2020] S. Papaioannou, P. Kolios, C. G. Panayiotou, and M. M. Polycarpou.  Cooperative simultaneous tracking and jamming for disabling a rogue drone.  _IEEE International Conference on Intelligent Robots and Systems_ , pages 7919–7926, 2020.  ISSN 21530866.  doi:10.1109/IROS45743.2020.9340835.
  * De Witte et al. [2022] S. De Witte, T. Van Hauwermeiren, T. Lefebvre, and G. Crevecoeur.  Learning to Cooperate: A Hierarchical Cooperative Dual Robot Arm Approach for Underactuated Pick-and-Placing.  _IEEE/ASME Transactions on Mechatronics_ , 27(4):1964–1972, 2022.  ISSN 1941014X.  doi:10.1109/TMECH.2022.3175484.
  * Guo et al. [2024] X. Guo, K. Huang, J. Liu, W. Fan, N. Vélez, Q. Wu, H. Wang, T. L. Griffiths, and M. Wang.  Embodied LLM Agents Learn to Cooperate in Organized Teams.  2024\.  URL http://arxiv.org/abs/2403.12482.
  * Zhang et al. [2023] H. Zhang, W. Du, J. Shan, Q. Zhou, Y. Du, J. B. Tenenbaum, T. Shu, and C. Gan.  Building Cooperative Embodied Agents Modularly with Large Language Models.  2023\.  URL http://arxiv.org/abs/2307.02485.
  * Sun et al. [2024] C. Sun, S. Huang, and D. Pompili.  LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions.  2024\.  URL http://arxiv.org/abs/2405.11106.
  * Liu et al. [2024] X. Liu, D. Guo, X. Zhang, and H. Liu.  Heterogeneous Embodied Multi-Agent Collaboration.  _IEEE Robotics and Automation Letters_ , 9(6):5377–5384, 2024.  ISSN 23773766.  doi:10.1109/LRA.2024.3390588.
  * Nachum et al. [2018] O. Nachum, H. Lee, S. Gu, and S. Levine.  Data-efficient hierarchical reinforcement learning.  _Advances in Neural Information Processing Systems_ , 2018-Decem:3303–3313, 2018.  ISSN 10495258.
  * Sun et al. [2021] C. Sun, W. Liu, and L. Dong.  Reinforcement Learning with Task Decomposition for Cooperative Multiagent Systems.  _IEEE Transactions on Neural Networks and Learning Systems_ , 32(5):2054–2065, 2021.  ISSN 21622388.  doi:10.1109/TNNLS.2020.2996209.
  * Eppe et al. [2022] M. Eppe, C. Gumbsch, M. Kerzel, P. D. Nguyen, M. V. Butz, and S. Wermter.  Intelligent problem-solving as integrated hierarchical reinforcement learning.  _Nature Machine Intelligence_ , 4(1):11–20, 2022.  ISSN 25225839.  doi:10.1038/s42256-021-00433-9.
  * Yang et al. [2021] X. Yang, Z. Ji, J. Wu, Y. K. Lai, C. Wei, G. Liu, and R. Setchi.  Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.  ISSN 21622388.  doi:10.1109/TNNLS.2021.3059912.
  * Wöhlke et al. [2021] J. Wöhlke, F. Schmitt, and H. van Hoof.  Hierarchies of Planning and Reinforcement Learning for Robot Navigation.  _Proceedings - IEEE International Conference on Robotics and Automation_ , 2021-May:10682–10688, 2021.  ISSN 10504729.  doi:10.1109/ICRA48506.2021.9561151.
  * Li et al. [2020] T. Li, N. Lambert, R. Calandra, F. Meier, and A. Rai.  Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning.  _Proceedings - IEEE International Conference on Robotics and Automation_ , pages 413–419, 2020.  ISSN 10504729.  doi:10.1109/ICRA40945.2020.9196642.
  * Heess et al. [2016] N. Heess, G. Wayne, Y. Tassa, T. Lillicrap, M. Riedmiller, and D. Silver.  Learning and Transfer of Modulated Locomotor Controllers.  2016\.  URL http://arxiv.org/abs/1610.05182.
  * Sferrazza et al. [2024] C. Sferrazza, D.-M. Huang, X. Lin, Y. Lee, and P. Abbeel.  HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation.  2024\.  URL http://arxiv.org/abs/2403.10506.
  * Heess et al. [2017] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. Riedmiller, and D. Silver.  Emergence of Locomotion Behaviours in Rich Environments.  2017\.  URL http://arxiv.org/abs/1707.02286.
  * Zhao et al. [2020] J. Zhao, F. Huang, J. Lv, Y. Duan, Z. Qin, G. Li, and G. Tian.  Do RNN and LSTM have long memory?  _37th International Conference on Machine Learning, ICML 2020_ , PartF168147-15:11302–11312, 2020.
  * Amorim et al. [2021] T. Amorim, T. Nascimento, P. Petracek, G. De Masi, E. Ferrante, and M. Saska.  Self-Organized UAV Flocking Based on Proximal Control.  _2021 International Conference on Unmanned Aircraft Systems, ICUAS 2021_ , pages 1374–1382, 2021.  doi:10.1109/ICUAS51884.2021.9476847.

##  Appendix A Appendix

###  A.1 Environment Description

####  A.1.1 Cooperative Transport

The agents should cooperate to push a cylinder to a target location. The local
positions between the agents, cylinder, and target are initialized randomly.
The cylinder is set in the middle of the map, and agents are initialized in
the four quadrants of the cylinder respectively.

Observation The agents receive two sets of observations: (i) a set of
”proprioceptive” states which would be given to the lower layer controller.
This part of observation is exactly the same in all agents in different tasks
because all tasks use the same pre-trained lower layer policy, including their
self-observed information, such as its local position, local velocity, angular
velocity, orientation, body upright angle, joint force and its limitations,
etc. (ii) a set of “exteroceptive” features containing task-relevant
information which would be given to upper layer controller. In this task, the
observation includes the locations of the nearest agent, the current position
of the cylinder, and the destination.

Rewards We set three types of rewards: (i) Cylinder distance reward: a term
which is the reciprocal L2 distance of the agent to the target. This is the
main reward of this task. It will increase when the cylinder becomes closer to
the target. The increasing trend is the same as the inverse proportional
function. (ii) Agent cylinder distance reward: this term is to encourage
agents firstly come closer to the cylinder so that they can learn how to push
the object. (iii) Agent distance reward: a term used to guide the agent to
walk towards the target, thereby obtaining a large cylinder distance reward.

####  A.1.2 Corridor Crossing

The agents are expected to reach a designated line through a narrow slit. Two
walls block most of the way forward, leaving only a narrow corridor that only
allows one agent to pass at a time. Yet all agents are at the same distance to
the entrance of the slit, which means that agents would get stuck at the
entrance of the slit together if nobody is slow or stop to wait for others to
pass first. We expected agents can acquire a strategy so that all the agents
could go through the corridor as fast as possible.

Observation The lower layer controllers’ observation is the same as the last
task. In this task, agents’ upper layer controllers’ observation includes the
locations of itself and the nearest neighbour, the position, translation, and
orientation of the walls.

Rewards Two rewards are set: (i) Destination reward: if an agent has passed
the finish line, all agents will receive a positive reward. It stimulates the
strategy allowing as many agents as possible to pass through the corridor to
the finish line. (ii) Distance reward to enter: a term to encourage agents
first to come close to the corridor.

####  A.1.3 Ravine Bridging

The agents are expected to reach a designated line where they must cross over
a ravine first. There is a ravine blocking the way of the agents. Agents
cannot jump over or climb out of the ravine. Agents need colleagues’ help to
come over the ravine. At this point, we place a stone of the right size and
depth in the ravine to serve as a ”bridge” to help the agents cross it. We
pre-set two agents on either side of the stone to push the stone to help their
colleagues cross the ravine and reach the destination. So this is a complex
task that requires teamwork among heterogeneous agents.

Observation The observation of the lower layer policy is similar to the
previous tasks. Agents’ upper layer controllers’ observation includes the
locations of itself and the nearest agents, the position of the stone, and the
position and depth of the ravine.

Rewards Two rewards are set: (i) Destination reward: if any agent has passed
the finish line, all agents will receive a positive reward. It stimulates the
strategy allowing as many agents as possible to go through the ravine and get
to the finish line. (ii) Bridging reward: a term to encourage agents in the
ravine to carry the stone to the agent above which is the closest to the
ravine.

###  A.2 Curriculum

We utilize sparse rewards to direct agents towards proximity with the object,
subsequently acquiring the target’s location in the Cooperative Transport
task. In the Corridor Crossing task, we train individual agents to navigate to
the finish line by passing through the slit and then refine the pre-trained
policy for application in a multi-agent environment. The learning sequence for
the Ravine Bridging task is more intricate which means we have to give more
guidance; we instruct agents to wait at the ravine’s edge at the opportune
moment to prevent falling, while other agents within the ravine initially
learn to convey ”bridge” to an appropriate location.

###  A.3 Scalability

The distributed HRL framework allows the collaborative tasks to be explicitly
represented as a ”group”, thereby enabling tasks to be extended to large
numbers of agents team. We have tested the framework with different
populations of Ant in various tasks (There are only two Ants for pushing
bridge in the task Ravine Bridging), and the success rates of each task with
different populations are shown in the Tab. 2. The results show the potential
of our framework for handling tasks with more agents.

Table 2: Each percentage results from 50 trials—10 attempts per three independently trained models. Success rate | Two Ants | Three Ants | Four Ants | Five Ants
---|---|---|---|---
Cooperative Transport | $8\pm 2\%$ | $42\pm 3\%$ | $74\pm 7\%$ | $82\pm 5\%$
Corridor Crossing | $98\pm 2\%$ | $95\pm 2\%$ | $93\pm 3\%$ | $79\pm 2\%$
Ravine Bridging | $48\pm 5\%$ | $36\pm 8\%$ | $30\pm 2\%$ | $12\pm 5\%$
Table 3: Success Rate Comparison between average moving distances in task Ravine Bridging. Distance from Ravine | Two Ants | Three Ants | Four Ants | Five Ants
---|---|---|---|---
2 m | $49\%$ | $45\%$ | $38\%$ | $35\%$
4 m | $38\%$ | $33\%$ | $20\%$ | $11\%$
6 m | $30\%$ | $19\%$ | $5\%$ | $3\%$

###  A.4 Typical Result

####  A.4.1 Corridor Crossing

We record the success rates of each Ant at its initial position separately, as
illustrated in Fig. A.4.1. We have a higher success rate when the initial
position is closer to the center, and a higher failure rate when the initial
position is closer to the edge. This is also explainable, as the Ant at the
centre is closer to the slit and does not need to adjust velocity.

####  A.4.2 Ravine Bridging

The bridging and coordination in the ravine is the key to completing the task.
We set up three difficulty levels - easy, medium, and hard - based on the
distance from the ravine’s edge. The easy level corresponds to a distance of 2
m, the medium level to 4 m, and the hard level to 6 cm. The Tab. 3 shows the
success rates of the three difficulty levels with different numbers of Ants.
We can observe that it becomes more difficult for any number of Ants to
complete the task as the distance increases. Because cooperation on the timing
of stopping and bridging becomes more important when moving length increases.


* [70] Y.-K. Tsay, M.-H. Tsai, J.-S. Chang, Y.-W. Chang, and C.-S. Liu.  Büchi Store: An open repository of $\omega$-automata.  International Journal on Software Tools for Technology Transfer, 15(2):109–123, 2013.  http://buchi.im.ntu.edu.tw/.
  * [71] M. D. Wulf, L. Doyen, T. A. Henzinger, and J.-F. Raskin.  Antichains: A new algorithm for checking universality of finite automata.  In Proc. of CAV 2006, volume 4144 of LNCS, pages 17–30. Springer-Verlag, 2006.