(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Fudan University 11email<EMAIL_ADDRESS>
22institutetext: School of Future Technology, Harbin Institute of Technology
33institutetext: Department of Computer Science and Engineering, Shanghai Jiao
Tong University

# HGS-Mapping: Online Dense Mapping Using Hybrid Gaussian Representation in
Urban Scenes

Ke Wu1 Kaizhao Zhang2 Zhiwei Zhang1 Shanshuai Yuan1 Muer Tie1 Julong Wei1
Zijun Xu1 Jieru Zhao3 Zhongxue Gan1 Wenchao Ding1

###### Abstract

Online dense mapping of urban scenes forms a fundamental cornerstone for scene
understanding and navigation of autonomous vehicles. Recent advancements in
mapping methods are mainly based on NeRF, whose rendering speed is too slow to
meet online requirements. 3D Gaussian Splatting (3DGS), with its rendering
speed hundreds of times faster than NeRF, holds greater potential in online
dense mapping. However, integrating 3DGS into a street-view dense mapping
framework still faces two challenges, including incomplete reconstruction due
to the absence of geometric information beyond the LiDAR coverage area and
extensive computation for reconstruction in large urban scenes. To this end,
we propose HGS-Mapping, an online dense mapping framework in unbounded large-
scale scenes. To attain complete construction, our framework introduces Hybrid
Gaussian Representation, which models different parts of the entire scene
using Gaussians with distinct properties. Furthermore, we employ a hybrid
Gaussian initialization mechanism and an adaptive update method to achieve
high-fidelity and rapid reconstruction. To the best of our knowledge, we are
the first to integrate Gaussian representation into online dense mapping of
urban scenes. Our approach achieves SOTA reconstruction accuracy while only
employing 66% number of Gaussians, leading to 20% faster reconstruction speed.

###### Keywords:

Gaussian Splatting Dense Mapping Autonomous Driving

##  1 Introduction

Online dense mapping is crucial for autonomous vehicles to understand complex
environments and navigate effectively. The meaning of online dense mapping
lies in continuously creating a detailed map of the vehicle’s surroundings
using sensor data streams (camera and LiDAR, etc.) as the vehicle moves. This
detailed map encompasses almost all visible surfaces and objects, offering a
comprehensive and high-fidelity representation of the environment.

Figure 1: Reconstruction of large scale urban scenes. Our method achieves
high-quality rendering results using only two-thirds the number of Gaussians
compared to the current SOTA online reconstruction method SplaTAM [14].

How to represent this dense map has been studied for decades. Conventional
methods [27, 16, 24, 35] directly fuse sensor data across space and time to
construct a map, but this kind of map is sparse, unable to capture rich scene
details, and contains holes. In recent years, NeRF-based methods have gained
popularity due to their realistic novel view synthesis. NeRF [22] uses
resource-intensive neural networks to represent the color and density of the
scene and outputs the entire image in a volume rendering manner, which leads
to several training hours. Subsequent methods [23, 46, 34] accelerate training
and rendering speed by combing voxel features with light MLPs, introducing
NeRF into online mapping. However, these methods are limited by the finite
resolution of voxels, making it difficult to balance rendering quality and
speed. Moreover, the inherent volume rendering process of NeRF requires
sampling all the pixels, which is the fundamental reason for deficient
training and rendering speed.

3D Gaussian Splatting (3DGS) [15] has received much attention in recent months
due to its high-fidelity rendering quality and rendering speed that is
hundreds of times faster than NeRF, holding greater potential in online dense
mapping. 3DGS utilizes Gaussians with learnable parameters to represent the
entire scene, replacing NeRF’s volume rendering with splatting-based
rasterization [47]. Despite the impressive rendering quality and speed of
3DGS, research on using 3DGS representation for online dense mapping is scarce
and entirely limited to indoor scenarios [14, 42, 21, 37]. This limitation
arises because 3DGS requires pre-processing feature point clouds with
Structure-from-Motion (SfM) [30] on a complete dataset for initializing
Gaussians, which is not feasible for online dense mapping tasks. Existing
methods in indoor scenarios typically use a depth stream to initialize
Gaussians. In complex outdoor environments, depth cameras often struggle to
function effectively, leading to challenges in obtaining dense depth
information for urban scenes. The straightforward approach is to initialize
Gaussians using LiDAR points. Nonetheless, introducing 3D Gaussian
representation into online dense mapping in urban scenes still faces the
following two gaps. Firstly, LiDAR points can only cover a limited range; for
example, the upper parts of trees and buildings, and distant landscapes may
lack LiDAR coverage. Consequently, this results in a lack of initial geometry
information for these areas, making it impossible for existing Gaussian-based
methods to fully reconstruct the scene. Secondly, large urban scenes require a
significant number of Gaussians, leading to substantial memory overhead and
computational complexity during the rasterization process.

To tackle these challenges outlined above, we present HGS-Mapping, an online
dense mapping framework that fully leverages the potential of Gaussian
representation, achieving accurate and efficient reconstruction for entire
urban environments at the same time. Our approach introduces a novel
representation called Hybrid Gaussian Representation, comprised of Sphere
Gaussian, 3D Gaussian, and 2D Gaussian Plane components. This representation
achieves high-fidelity reconstruction results for the areas lacking LiDAR
points. Thus, the entire urban scene is fully represented in this manner.
Furthermore, we implement an adaptive update method for Gaussians, which
dynamically densifies Gaussians based on the reconstruction loss and prunes
the Gaussians of low importance. This update method enhances rendering quality
and accelerates rendering speed. As shown in Fig. 1, we utilize merely 66% of
the number of Gaussians in SplaTAM [14] and optimize Gaussians to an
appropriately non-minimal scale. With fewer Gaussians, we successfully convey
richer scene information and render images of higher quality. To the best of
our knowledge, we are the first to integrate Gaussian representation into
online dense mapping for urban scenes.

Our contributions can be summarized as follows:

  * •

We are the first to introduce Gaussian representation to online dense mapping
in urban scenes, and we innovatively propose a Hybrid Gaussian Representation,
which facilitates reconstructing entire urban scenes.

  * •

We propose a hybrid Gaussian initialization mechanism and an adaptive update
method, which result in superior rendering quality and faster reconstruction
speeds.

  * •

Comprehensive experiments demonstrate that HGS-Mapping surpasses previous
methods in both rendering quality and speed while employing only two-thirds
the number of Gaussians compared to the SOTA approach.

##  2 Related Works

3D Gaussian Splatting. 3DGS [15] employs 3D Gaussians to represent scenes and
achieves fast differentiable rendering through splat-based rasterization.
Numerous downstream tasks have emerged, such as driving scenes simulation [45,
38], simultaneous localization and mapping [14, 37, 42, 21], mesh
reconstruction [4, 10], and dynamic scene modeling [36, 20, 41], etc. To
reduce storage capacity, LightGaussian [8] computes the global significance of
Gaussians and prunes away those below a certain threshold. Gaussianpro [6]
introduces a progressive propagation strategy to guide Gaussian densification
and tests on the Waymo dataset [32]. However, current online mapping methods
built on 3D Gaussians predominantly target indoor scene reconstruction,
overlooking mechanisms suitable for outdoor environments. Additionally, most
efforts in reconstructing scenes for autonomous driving rely on SfM [30] for
Gaussian initialization, a process which is too slow to satisfy the real-time
demands of online dense mapping.

Urban Scale Mapping. With the success of NeRF [22] in 3D reconstruction, an
increasing number of studies are focusing on reconstruction and simulation for
autonomous driving in urban scenes [40, 26, 11, 39, 13, 29, 33, 18]. However,
due to the vastness, unbounded nature, and difficult accessibility of depth
information in urban environments, urban scale mapping presents significant
challenges. For example, to address the sky regions in urban scenes where rays
never intersect with any opaque surfaces, [29, 11] propose an MLP to map the
ray direction to the color of the sky. For dynamic objects within the urban
scene, Unisim [40] adopts the NSG [25] approach of representing dynamic scenes
as scene graphs for editing purposes. Streetsurf [11] employs the signed
distance function (SDF) to portray urban scenes and draws inspiration from the
methodology of NeRF++ [43] to decompose unbounded space using a hyper-cuboid
model. [38, 45] firstly employ 3D Gaussian splatting for the reconstruction of
urban scenes. Specifically, DrivingGaussian [45] utilizes a dynamic Gaussian
graph to describe the scene dynamically and incorporates LiDAR priors for an
incremental sequential background representation. StreetGaussian [38] employs
a 4D spherical harmonics model to characterize dynamic objects and uses
semantic logits for rendering semantic images. However, there hasn’t been any
work yet applying advanced Gaussian representation to online mapping in urban
environments.

##  3 Method

The overview of our pipeline is illustrated in Fig. 2. The input of our system
includes RGB-SparseD (LiDAR) stream $\\{I_{t},L_{t}\\}$ along with a pose
sequence ${T_{t}}$ from a moving vehicle in urban streets. To initialize
Gaussians, we match the feature points from adjacent RGB frames with the
corresponding LiDAR points. Based on the Hybrid Gaussian Representation (Sec.
3.1), we model the world as inliers, outliers, and the sky separately. We
design an efficient rasterizer tailored for such representation (Sec. 3.2).
Following this, we use an adaptive update method (Sec. 3.3) to incrementally
maintain and optimize our Hybrid Gaussian Representation. Our framework
supports rendering high-quality RGB and depth images and exporting meshes
using marching cubes.

Figure 2: Overview of HGS-Mapping’s pipeline. Input RGB-LiDAR stream and pose
sequence are incrementally integrated into our Hybrid Gaussian Representation.
We update and adjust the Gaussians with our Adaptive Update Method (Densify
Control, Importance Pruning, Silhouette Filter). Our method supports rendering
RGB-Depth using Hybrid Gaussian Rasterizer and exporting meshes.

###  3.1 Hybrid Gaussian Representation

It is instinctive for human to divide the world into the sky, the ground, and
ground objects due to their significant differences in geometry and
photometry. Consequently, using the same representation to model these
different parts is inefficient. We propose Hybrid Gaussian Representation
consisting of three types of Gaussians: $G_{sky}$, $G_{inlier}$ and
$G_{outlier}$, which are used to model different parts of the urban scenario,
namely the sky, the road surface, and roadside scenery respectively. Our
Hybrid Gaussian Representation records the color and geometric information of
the scene. The initialization method and the properties for each type of
Gaussians are distinct.

####  3.1.1 Gaussian Initialization.

Vanilla 3DGS initializes Gaussians with a sparse point cloud obtained by
running Structure-from-Motion (SfM) on the complete dataset. However, this is
infeasible for online mapping processes in which the vehicle is mapping while
moving, and the very sparse viewpoints can also lead to incorrect geometric
structures generated by SfM.

To achieve better initialization, we use LiDAR points to initialize Gaussians.
However, LiDAR’s coverage is quite limited, unable to capture the upper parts
of trees and buildings, as well as distant landscapes.

To this end, for areas within the LiDAR’s coverage, we initialize Gaussians
using LiDAR points. For the remaining areas, we implement a lightweight
feature matching network [17] to extract matching pixels from adjacent RGB
frames and calculate optical flow value $\\{f\\}$. These feature pixels’
spatial positions are then determined through epipolar geometry calculations
$\mathrm{Epip}(\cdot)$. In the actual computation process, we found that using
epipolar geometry to determine depths of distance points (very low optical
flow value) is highly unreliable because the two rays are nearly parallel,
resulting in a calculated infinity depth. To address this, for pixels with
optical flow values less than $f_{th}$, we estimate the depth of distant
feature points through an approximate calculation method. In summary, pixel
$u$’s depth $d_{u}$ calculation is shown in Eq. 1. $\theta_{u}$ represents the
angle between the ray of $u$ and the camera’s optical axis. The derivation
process of Eq. 1 can be found in the supplementary material.

$\displaystyle
d_{u}=\left\\{\begin{matrix}\mathrm{Epip}(T_{t},T_{t+1},f_{u})&f_{u}\geq
f_{th}\\\
d_{th}\cdot\frac{tan\theta_{u}/f_{u}}{tan\theta_{th}/f_{th}}&f_{u}<f_{th}\end{matrix}\right.$
(1)

Following the calculations above, we obtained the initial positions of
Gaussians $\\{P_{t}\\}$ that cover LiDAR’s blind spots (such as the upper
parts of trees/buildings and distant landscapes). For the other properties of
Gaussians, we initialize them using the method described in SplaTAM [14].

####  3.1.2 Sphere Gaussian.

Vanilla 3DGS often generates ambiguous floaters to render the sky, leading to
a significant number of unnecessary Gaussians and non-conformance to multi-
view consistency. This phenomenon indicates that the sky lacks geometric
information and cannot be represented by Gaussians in the world coordinate
system and there are no Gaussian-based methods to address the issue of sky
reconstruction.

We discover that the color of the sky depends on the viewing direction and is
unrelated to the vehicle’s translation. Therefore, we model the sky using
Gaussians attached to the surface of an enormous sphere $S$ with a radius of
$R$, namely Sphere Gaussians $G_{sky}$. At any given moment, there exists only
a rotational transformation between the coordinate system of $G_{sky}$ and
that of the vehicle. The position property of $G_{sky}$ has only two degrees
of freedom. Furthermore, we stipulate that the rotation property of $G_{sky}$
is parallel to the radial direction of the spherical surface $S$, and
$G_{sky}$ has a fixed thickness along the radial direction.

As a result, $G_{sky}$ possesses only the following eight learnable properties
$\\{[x^{sky},z^{sky}],[s_{x}^{sky},s_{z}^{sky}],c^{sky},\alpha^{sky}\\}$,
where $c^{sky}$ represents $[r^{sky},g^{sky},b^{sky}]$. For the remaining non-
learnable properties, we calculate $y^{sky}$ and $q^{sky}$ using Eq. 2.

$\displaystyle y^{sky}$
$\displaystyle=\sqrt{R^{2}-(x^{sky})^{2}-(y^{sky})^{2}},\theta=arccos(\frac{y^{sky}}{R})$
(2) $\displaystyle q^{sky}$
$\displaystyle=[cos(\frac{\theta}{2}),sin(\frac{\theta}{2})\frac{x^{sky}}{R},sin(\frac{\theta}{2})\frac{y^{sky}}{R},sin(\frac{\theta}{2})\frac{z^{sky}}{R}]$

When a new RGB frame $I_{t}$ arrives, we transform the ray direction vectors
of $I_{t}$’s sky pixels from the vehicle’s coordinate system to the
$G_{sky}$’s coordinate system, and add new Gaussians to $G_{sky}$ at the
intersection position of the ray direction and the spherical surface $S$.
During the rendering process, we first convert Sphere Gaussians from
$G_{sky}$’s coordinate to the vehicle’s coordinate, and then utilize $T_{t}$
to transform these Gaussians to the world coordinate system. Subsequently, we
feed them to our Hybrid RGBD Rasterizer (Sec. 3.2.1). This approach ensures
that the Sphere Gaussians are consistent across multiple views.

####  3.1.3 2D Gaussian Plane.

We found that in Vanilla 3DGS, Gaussians on the road surface tend to stack in
multiple layers with significant overlap between each other. However, both the
geometry and texture of the road surface are relatively simple, leading to a
substantial redundancy of the Gaussians. Meanwhile, Vanilla 3DGS sorts
Gaussians of the road surface and those of roadside landscapes together. And
it’s worth emphasizing that Gaussian sorting consumes up to 30% of the total
rendering process [28]. Since the road surface is always below all roadside
landscapes, the Gaussians of the road surface can be completely excluded from
the sorting calculation, which has great potential to accelerate rendering.

Therefore, we model the road using flattened Gaussians on the plane, i.e. 2D
Gaussian Plane $G_{inlier}$. The position property of $G_{inlier}$ only has
two degrees of freedom due to being confined to a plane. We define
$G_{inlier}$’s rotation vector $q^{inlier}$ is parallel to the plane’s normal
vector and $G_{inlier}$ has a fixed thickness $s_{y}^{inlier}$ along
$q^{inlier}$. Thus, $G_{inlier}$ possesses the following eight learnable
properties:
$\\{[x^{inlier},z^{inlier}],[s_{x}^{inlier},s_{z}^{inlier}],c^{inlier},\alpha^{inlier}\\}$.
For the remaining non-learnable properties of $G_{inlier}$, we calculate
$y^{inlier}$ and $q^{inlier}$ using Eq. 3.

$\displaystyle y^{inlier}$ $\displaystyle=\frac{-Ax-
Cz-D}{B},\theta=arccos(\frac{B}{\sqrt{A^{2}+B^{2}+C^{2}}})$ (3) $\displaystyle
q^{inlier}$
$\displaystyle=[cos(\frac{\theta}{2}),sin(\frac{\theta}{2})cos(\theta)\frac{A}{B},sin(\frac{\theta}{2})cos(\theta),sin(\frac{\theta}{2})cos(\theta)\frac{C}{B}]$

Initially, we get the first RGB-LiDAR frame $\\{I_{0},L_{0}\\}$. We calculate
the feature points $P_{0}$ of $I_{0}$ using Eq. 1 and get $\hat{L_{0}}$ by
concatenating $P_{0}$ and $L_{0}$, i.e.
$\hat{L_{0}}=\mathrm{concat}(L_{0},P_{0})$ . Then, we implement the plane
segmentation algorithm (RANSAC [44]) to divide $\hat{L_{0}}$ into inlier
$\hat{L_{o}}^{inlier}$ and outlier $\hat{L_{o}}^{outlier}$, and obtain the
plane equation $Ax+By+Cz+D=0$. When a subsequent RGB-LiDAR frame
$\\{I_{t},L_{t}\\}$ arrives, we use the aforementioned method to obtain
$\hat{L_{t}}$ and calculate their distances to the plane mentioned above. We
set a distance threshold $D_{th}$ and classify the part of $\hat{L_{t}}$ as
${\hat{L_{t}}}^{inlier}$, whose distances to the plane are less than $D_{th}$.
Then we add ${\hat{L_{t}}}^{inlier}$ to $G_{inlier}$. Specifically, for
complex urban scenes, we select some keyframes and construct plane equations
between each pair of adjacent keyframes, achieving the reconstruction of
undulating road surfaces.

Our approach reduces the number of Gaussians on the road and improves the
rendering speed. Additionally, by distinguishing between inliers and outliers,
Gaussians on the road are no longer involved in the sorting algorithm, which
will be elaborated in Sec. 3.2.1.

####  3.1.4 3D Gaussian.

We use 3D Gaussian $G_{outlier}$ to model roadside landscapes. The geometric
and texture of roadside landscapes are intricate. Resembling from Vanilla
3DGS, we utilize ellipsoids-shaped Gaussians to characterize $G_{outlier}$,
which ensures that the Gaussians have sufficient degrees of freedom.
$G_{outlier}$ possesses 14 learnable properties
$\\{\mu^{outlier},s^{outlier},c^{outlier},q^{outlier},\alpha^{outlier}\\}$,
where position $\mu$ means $[x,y,z]$ and scale $s$ means
$[s_{x},s_{y},s_{z}]$.

###  3.2 Rasterization and Optimization

####  3.2.1 Hybrid RGBD Rasterizer.

We design a Hybrid RGBD Rasterizer tailored for Hybrid Gaussian
Representation. We calculate the weight $f(p)$ of any point
$p\in\mathbb{R}^{3}$ in 3D space according to the standard Gaussian equation
Eq. 4.

$\displaystyle f(p)$
$\displaystyle=\textrm{sigmoid}(\alpha)exp(-\frac{1}{2}(p-\mu)^{T}(p-\mu))$
(4)

where $\mu$ and $\alpha$ represent the center and the opacity of Gaussians,
respectively.

Our Hybrid Gaussian Rasterization contains three steps. Firstly, we transform
$G_{sky}$ to the world coordinate as described in Sec. 3.1.2. And for a pixel
belonging to a tile, we independently evaluate the three types of Gaussians
within the tile. Then, we perform sorting within $G_{sky}$ and $G_{outlier}$
individually and concatenate them together. The order from front to back is
${{G}_{outlier},{G}_{inlier},{G}_{sky}}$. Finally, we render one pixel’s RGB
value $C$ and depth value $D$ by directly computing the weighted sum of $N$
sorted Gaussians overlapping with this pixel, as described in Eq. 5.

$\displaystyle C=\sum_{i=1}^{N}c_{i}f_{i}\prod_{j=1}^{i-1}(1-f_{j})$ (5)
$\displaystyle D=\sum_{i=1}^{N}d_{i}f_{i}\prod_{j=1}^{i-1}(1-f_{j})$

where $c$, $d$ are the color and depth of each Gaussian and $f$ is given by
Eq. 4. Our approach saved 20% of the time by sorting three types of Gaussians
separately compared to sorting them together.

####  3.2.2 Optimize on Keyframe List.

To prevent degradation of the reconstruction quality of historical frames
during the online mapping process [31], we maintain a global keyframe list in
the same spirit of [46]. In each iteration, a frame is randomly selected from
the keyframe list for optimization. The keyframe list consists of $K$ frames.
We choose $K-2$ frames randomly from all frames that overlap with current
frame based on point cloud projection, and add the current and previous frame
to the list. We update the keyframe list every $n$ frames.

Then we provide details on the optimization of
$\\{G_{sky},G_{inlier},G_{outlier}\\}$. Our loss function $\mathcal{L}$
consists of three components: photometric loss $\mathcal{L}_{RGB}$, geometric
loss $\mathcal{L}_{LiDAR}$, and regularization loss $\mathcal{L}_{reg}$.

The RGB loss $\mathcal{L}_{RGB}$ consists of $\mathcal{L}_{1}$ and a D-SSIM
term:

$\displaystyle\mathcal{L}_{RGB}$
$\displaystyle=(1-\lambda)\mathcal{L}_{1}+\lambda\mathcal{L}_{\textrm{D-SSIM}}$
(6)

The LiDAR loss $\mathcal{L}_{LiDAR}$ is the $\mathcal{L}_{1}$ loss between
Sparse-D (LiDAR) and the predicted depth. The regularization loss
$\mathcal{L}_{reg}$ aims to improve the quality of rendered depth, including
depth smoothness loss $\mathcal{L}_{smooth}$ and isotropic [21] loss
$\mathcal{L}_{iso}$:

$\displaystyle\mathcal{L}_{smooth}$
$\displaystyle=\left|\partial_{x}d_{t}^{*}\right|e^{-\left|\partial_{x}I_{t}\right|}+\left|\partial_{y}d_{t}^{*}\right|e^{-\left|\partial_{y}I_{t}\right|}$
(7) $\displaystyle\mathcal{L}_{iso}$
$\displaystyle=(s-\bar{s})^{T}(s-\bar{s})$ $\displaystyle\mathcal{L}_{reg}$
$\displaystyle=\lambda_{smooth}\mathcal{L}_{smooth}+\lambda_{iso}\mathcal{L}_{iso}$

We optimize Gaussians by minimizing the loss $\mathcal{L}$:

$\displaystyle\underset{\\{G_{sky},G_{inlier},G_{outlier}\\}}{\textrm{min}}\lambda_{RGB}\mathcal{L}_{RGB}+\lambda_{LiDAR}\mathcal{L}_{LiDAR}+\lambda_{reg}\mathcal{L}_{reg}$
(8)

###  3.3 Adaptive Update of Gaussians

####  3.3.1 Silhouette Filter.

We render a silhouette image $S$ according to Eq. 9, which can reflect the
density of Gaussians at different positions on the image. The silhouette image
serves two purposes: adding new Gaussians in regions where Gaussians are
relatively sparse, and filtering depth to address the issue of irregular
spreading of depth at object edges.

$\displaystyle S=\sum_{i=1}^{N}f_{i}\prod_{j=1}^{i-1}(1-f_{j})$ (9)

When a new frame $\\{I_{t},L_{t}\\}$ arrives, we add it to our Hybrid
Gaussian. However, our goal is to add new Gaussians only in regions where
current Gaussians are relatively sparse. Therefore, we compute the silhouette
and only add Gaussians where $S<S_{th}$ or
$\mathcal{L}_{1}(D)>50\mathrm{MDE}$.

Moreover, we expand the application of the silhouette by using it to filter
the depth map. When rendering the depth map, we observed the issue of depth
shrinkage towards the optical center of the camera, as shown in Fig. 3(b).
This occurrence is attributed to Eq. 4, where Gaussians are relatively sparse
at object edges, and opacity decays with distance from the Gaussian center,
leading to the phenomenon of depth shrinkage at object edges.

(a) Effect of Gaussian Initialization.

(b) Effect of Silhouette Filter.

Figure 3: Effect of Gaussian Initialization & Silhouette Filter. Fig. 3(a)
shows that Gaussian Initialization enhances the reconstruction quality in the
upper parts of trees and buildings. Fig. 3(b) shows that Silhouette Filter
removes the abnormal depth of building’s edges.

####  3.3.2 Densify Control.

Gaussian densification accomplishes the purpose of cloning new Gaussians in
regions lacking geometric features and splitting large Gaussians into smaller
Gaussians. This method refines the scene details by densifying Gaussians with
large gradients, resolving the under-reconstruction problem [15] of Gaussians.
Although it enhances the quality of reconstruction, it also leads to an
increase in the number of Gaussians. Therefore, for every 20 iterations, we
execute densification and prune Gaussians with $\alpha$ values lower than
$\alpha_{th}$ or Gaussians with $s$ larger than $s_{th}$.

####  3.3.3 Importance Pruning.

Solely utilizing opacity and scale as criteria for pruning Gaussians is
inefficient. The reason is that it only ensures the majority of Gaussians
eventually meet the threshold ranges we set for opacity and scale properties,
but it does not guarantee the Gaussians within the threshold ranges can
effectively represent the entire scene. To this end, we compute the importance
score $IS_{j}$ of Gaussian $G_{j}$ and periodically prune Gaussians whose
importance score lies in the bottom $\eta\%$. We incorporate the gradient of
Gaussians $grad(\cdot)$ into the importance calculation. Gaussians with larger
gradients indicate poorer reconstruction at their respective positions, thus
importance is positively correlated with the gradient. We normalize the
gradient term using the loss of the current frame $\mathcal{L}$. The
calculation of the importance score is illustrated in Eq. 10.

$\displaystyle IS_{j}$
$\displaystyle=\sum_{i=1}^{KHW}\mathbb{I}(G_{j},r_{i})\cdot\alpha_{j}\cdot\tau_{j}\cdot\frac{grad_{j}}{\mathcal{L}}$
(10) $\displaystyle\tau_{j}$
$\displaystyle=\mathrm{max}(\mathrm{min}(V_{j},0)),V_{\mathrm{max}50})$

Where $j$ and $K$ represent the index of the Gaussian and the length of
keyframe list respectively. $\mathbb{I}$ indicates whether rays $r_{i}$
emitted from an image of size $H\times W$ intersect with Gaussian $G_{j}$.
Besides, $V_{j}$ means the volume of $G_{j}$ and $V_{\mathrm{max50}}$
represents the 50% largest volume of all sorted Gaussians.

##  4 Experiments

####  4.0.1 Datasets & Metrics.

To evaluate the reconstruction quality under different lighting, weather, and
road enviroments, we select three most commonly used datasets for urban scenes
(KITTI [9], nuScenes [3], Waymo [32]) as well as one simulated dataset
(VKITTI2 [2]). We use their RGB images, LiDAR points, and pose data as inputs
of our experiments and we select multiple challenging scenarios. To
quantitatively validate the effect of our method, we employ three widely
recognized photometric rendering quality metrics (PSNR, SSIM, LPIPS).

####  4.0.2 Compared Baselines.

We compared our method with two NeRF-based methods (Mip-NeRF360 [1], Instant-
NGP [23]) and two Gaussian-based methods (3DGS [15], SplaTAM [14]). Among
them, Mip-NeRF 360, InstantNGP, and 3DGS are recognized for their high-quality
reconstruction results, but their training process is off-line and time-
consuming. SplaTAM, currently at the forefront of Gaussian-based methods,
supports online training and achieves outstanding reconstruction results
indoors. To address the inability to represent the sky in street scenes, we
used Deeplabv3 [5] to mask the sky before calculating metrics. Additionally,
we utilize LiDAR points to initialize Gaussians, ensuring consistent initial
conditions.

###  4.1 Comparisons and Analysis

####  4.1.1 RGB Results.

To validate the rendering RGB quality, we compare our method with two SOTA
NeRF-based methods and two cutting-edge Gaussian-based methods. For Gaussian-
based methods, 3DGS, SplaTAM, and Ours are all initialized based on sparse
LiDAR points.

Figure 4: Quantitative RGB results in diverse urban scenes. We compare HGS-
Mapping with three off-line reconstruction methods [15, 23, 1] and an online
dense mapping method [14]. The three offline methods are trained over 20K
iterations but online methods only train 100 iterations per-frame. The results
shows that our work achieves superior high-quality rendering while faster
training speed.

For offline methods, Mip-NeRF 360 is trained for several hours on the entire
dataset, while InstantNGP is trained for 15 minutes (35K iterations), and 3DGS
is trained for 10 minutes (20K iterations). For online methods (SplaTAM,
Ours), we train 100 iterations per frame. All results are reported on a single
RTX3090.

Tab. 1 shows that even when compared with offline methods, our rendering
quality is significantly better. In urban scenes, our method outperforms
SplaTAM by a large margin in terms of rendering quality. Fig. 4 illustrates
SplaTAM’s lack of geometric reconstruction ability in areas without
initialized LiDAR points (upper part of trees/buildings and sky).

The rendering quality of SplaTAM in LiDAR coverage areas is also
unsatisfactory. Although SplaTAM performs well in indoor scenes with dense
depth using a large number of small-sized spheres, its ability to fill in
unreconstructed areas with sparse LiDAR points initialization outdoors is
inferior to that of ellipsoids. This will be further elaborated in the
supplementary material.

Table 1: Quantitative results of RGB rendering on four urban datasets [2, 9, 3, 32]. | VKITTI2 | KITTI | nuScenes | Waymo
---|---|---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
Mip NeRF 360 | 24.878 | 0.7502 | 21.395 | 0.6499 | 28.530 | 0.8751 | 24.826 | 0.8440
Instant-NGP | 21.586 | 0.6305 | 22.530 | 0.7418 | 27.046 | 0.8352 | 26.154 | 0.8557
3DGS(LIDAR) | 24.236 | 0.7928 | 19.227 | 0.6185 | 28.091 | 0.9133 | 17.664 | 0.8491
Splatam(masked) | 20.003 | 0.7401 | 14.778 | 0.4783 | 18.998 | 0.7459 | 20.371 | 0.7704
Ours | 29.114 | 0.9019 | 22.520 | 0.8007 | 30.862 | 0.9404 | 26.445 | 0.8832

####  4.1.2 Depth Results.

We compare the depth images of 3DGS, SplaTAM, and ours. Following the settings
in the previous section, we render the depth and projected dense depth into
point clouds for better visualization. Vanilla 3DGS lacks the capability to
render depth, we load 3DGS’s checkpoint into SplaTAM’s rasterizer to enable
depth rendering. Due to the absence of ground truth dense depth in urban
scenes, inspired by the depth completion task [12], we record the Mean
Absolute Error (MAE) and Root Mean Square Error (RMSE) between the predicted
depth and sparse LiDAR data to assess the quality of depth map, that is shown
in Tab. 2. It is important to note that these two metrics can only evaluate
the quality of the depth within the LiDAR coverage. Fig. 5 demonstrates that
our method can render high-quality dense depth maps relying on
$\mathcal{L}_{LiDAR}$ and $\mathcal{L}_{smooth}$. We are surprised to find
that even for the very sparse LiDAR points in nuScenes dataset (only
$\frac{2975}{1600\times 900}$ pixels have valid depth), our method can clearly
discern the geometric shapes of distant buildings and vehicles.

Figure 5: Qualitative depth results on KITTI [9], nuScenes [3] and Waymo [32]. Table 2: Sparse depth comparison using depth completion metrics[12]. | KITTI |  | nuScenes |  | Waymo |
---|---|---|---|---|---|---
| MAE$\downarrow$ | RMSE$\downarrow$ | MAE$\downarrow$ | RMSE$\downarrow$ | MAE$\downarrow$ | RMSE$\downarrow$
SplaTAM | 0.503 | 0.973 | 0.079 | 1.563 | 0.353 | 1.819
Ours | 0.203 | 0.522 | 0.033 | 0.309 | 0.101 | 0.673

####  4.1.3 Mesh Results.

To provide a more intuitive comparison of the geometric reconstruction results
across the entire scene, we qualitatively compare meshes of Ours and SplaTAM,
as shown in Fig. 6. We employ voxel-fusion [7] and marching cubes [19] to
export the rendered dense depth map sequence as meshes. We observe that due to
the limited coverage of the LiDAR, SplaTAM render very poor geometric shapes
outside the LiDAR’s coverage. However, owing to our feature point
initialization, we can accurately reconstruct the upper parts of
trees/buildings. Furthermore, due to our separate representation of the road
surface and roadside objects, our road surface is smoother and devoid of
holes.

Figure 6: Qualitative Mesh Comparisons on KITTI [9], nuScenes [3] and Waymo
[32].

###  4.2 Ablation Studies

####  4.2.1 Hybrid Gaussian Representation.

Our 3D Gaussian $G_{outlier}$’s properties are the same as Vanilla 3DGS. Thus,
we use $G_{outlier}$ to simultaneously model the sky, road surface, and
roadside landscapes as a baseline. Subsequently, we progressively substitute
$G_{sky}$ and $G_{inlier}$ to model the sky and road surface, respectively. To
validate the effects of our Hybrid Gaussian Representation on improving
rendering quality and speed, we compare five metrics, including Rendering FPS,
SIZE, PSNR, etc., under the same KITTI scenario.

The second row of Tab. 3 indicates that modeling the sky using $G_{sky}$ not
only reduces storage consumption but also enhances rendering speed.
Furthermore, we select scenes with complex sky colors from nuScenes and
conducted ablation experiments visualizing our Sphere Gaussian. Fig. 7
illustrates our $G_{sky}$ rendering high-quality sky and clouds. The last row
of Tab. 3 demonstrates that using $G_{inlier}$ results in a slight decrease in
rendering quality (from 22.660 to 22.542) but achieves remarkable improvements
in storage efficiency (from 298.4MB to 245.4MB) and rendering speed (from
228FPS to 271 FPS).

Table 3: Ablations on Hybrid Gaussian Representation. $G_{outlier}$ |  $G_{sky}$ | $G_{inlier}$ |  FPS$\uparrow$ |  SIZE$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$
---|---|---|---|---|---|---|---
✓ |  ✗ |  ✗ |  209 |  318.9MB |  21.246 |  0.6175 |  0.296
✓ |  ✓ |  ✗ |  228 |  298.4MB |  22.660 |  0.7621 |  0.182
✓ |  ✓ |  ✓ |  271 |  245.4MB |  22.542 |  0.7596 |  0.189

$\eta$ % | SIZE$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---|---
- | 245.4MB | 22.542 | 0.7596 | 0.189
1% | 234.5MB | 22.364 | 0.7215 | 0.204
3% | 220.8MB | 22.054 | 0.6955 | 0.230
5% | 183.9MB | 21.365 | 0.6479 | 0.275

Table 4: Ablation on Importance Pruning with different pruning rate $\eta\%$.
As $\eta\%$ increases, storage size significantly declines, while there is a
corresponding drop in rendering quality.

Figure 7: Ablation on $G_{sky}$.

####  4.2.2 Importance Pruning.

When the length of the keyframe list equals $K$, we begin to record global
significance every 5 iterations, and perform global significance pruning on
$G_{outlier}$ every 2 frames, removing $\eta\%$ of the $G_{outlier}$ each
time. We conduct experiments with different importance pruning rates $\eta\%$,
namely turning it off and setting it to 1%, 3%, 5%, as shown in Tab. 4.
Increasing the pruning rate effectively reduces storage size, but setting it
too high can lead to a significant decrease in rendering quality.

####  4.2.3 Gaussian Initialization & Silhouette Filter.

Regarding Feature Points Initialization in Sec. 3.1.1, we conduct tests in
scenes with abundant trees and extensive visibility. The experimental results,
as depicted in Fig. 3(a), indicate that our initialization method
significantly enhances the reconstruction performance in the upper parts of
trees and buildings.

As for the Silhouette Filter in Sec. 3.3.1, Fig. 3(b) shows that the
silhouette filter effectively resolves the issue of object edge recession in
the depth map.

##  5 Conclusion

We are the first 3DGS-based online dense mapping framework in urban scenes by
proposing the novel Hybrid Gaussian Representation suited for complex
unbounded scenes. Additionally, extensive experiments demonstrate that our
representation and optimization methods significantly improve rendering speed
and quality, achieving SOTA performance. However, the efficacy of the RANSAC
method for extracting $G_{inlier}$ becomes limited under conditions such as
rugged roads or significant curvature. Therefore, there remains potential for
further enhancement of this framework to accommodate arbitrary outdoor scenes.

## References

  * [1] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5470–5479 (2022)
  * [2] Cabon, Y., Murray, N., Humenberger, M.: Virtual kitti 2 (2020)
  * [3] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11621–11631 (2020)
  * [4] Chen, H., Li, C., Lee, G.H.: Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. arXiv preprint arXiv:2312.00846 (2023)
  * [5] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
  * [6] Cheng, K., Long, X., Yang, K., Yao, Y., Yin, W., Ma, Y., Wang, W., Chen, X.: Gaussianpro: 3d gaussian splatting with progressive propagation. arXiv preprint arXiv:2402.14650 (2024)
  * [7] Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. pp. 303–312 (1996)
  * [8] Fan, Z., Wang, K., Wen, K., Zhu, Z., Xu, D., Wang, Z.: Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv preprint arXiv:2311.17245 (2023)
  * [9] Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. The International Journal of Robotics Research 32(11), 1231–1237 (2013)
  * [10] Guédon, A., Lepetit, V.: Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775 (2023)
  * [11] Guo, J., Deng, N., Li, X., Bai, Y., Shi, B., Wang, C., Ding, C., Wang, D., Li, Y.: Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint arXiv:2306.04988 (2023)
  * [12] Hu, J., Bao, C., Ozay, M., Fan, C., Gao, Q., Liu, H., Lam, T.L.: Deep depth completion from extremely sparse data: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [13] Huang, S., Gojcic, Z., Wang, Z., Williams, F., Kasten, Y., Fidler, S., Schindler, K., Litany, O.: Neural lidar fields for novel view synthesis. arXiv preprint arXiv:2305.01643 (2023)
  * [14] Keetha, N., Karhade, J., Jatavallabhula, K.M., Yang, G., Scherer, S., Ramanan, D., Luiten, J.: Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint arXiv:2312.02126 (2023)
  * [15] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (2023)
  * [16] Labbé, M., Michaud, F.: Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. Journal of field robotics 36(2), 416–446 (2019)
  * [17] Lindenberger, P., Sarlin, P.E., Pollefeys, M.: Lightglue: Local feature matching at light speed. arXiv preprint arXiv:2306.13643 (2023)
  * [18] Liu, J.Y., Chen, Y., Yang, Z., Wang, J., Manivasagam, S., Urtasun, R.: Real-time neural rasterization for large scenes. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8416–8427 (2023)
  * [19] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: Seminal graphics: pioneering efforts that shaped the field, pp. 347–353 (1998)
  * [20] Luiten, J., Kopanas, G., Leibe, B., Ramanan, D.: Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713 (2023)
  * [21] Matsuki, H., Murai, R., Kelly, P.H., Davison, A.J.: Gaussian splatting slam. arXiv preprint arXiv:2312.06741 (2023)
  * [22] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: European Conference on Computer Vision. pp. 405–421. Springer (2020)
  * [23] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41(4), 1–15 (2022)
  * [24] Mur-Artal, R., Tardós, J.D.: Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics 33(5), 1255–1262 (2017)
  * [25] Ost, J., Mannan, F., Thuerey, N., Knodt, J., Heide, F.: Neural scene graphs for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2856–2865 (2021)
  * [26] Pun, A., Sun, G., Wang, J., Chen, Y., Yang, Z., Manivasagam, S., Ma, W.C., Urtasun, R.: Neural lighting simulation for urban scenes. Advances in Neural Information Processing Systems 36 (2024)
  * [27] Qin, T., Li, P., Shen, S.: Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics 34(4), 1004–1020 (2018)
  * [28] Radl, L., Steiner, M., Parger, M., Weinrauch, A., Kerbl, B., Steinberger, M.: Stopthepop: Sorted gaussian splatting for view-consistent real-time rendering. arXiv preprint arXiv:2402.00525 (2024)
  * [29] Rematas, K., Liu, A., Srinivasan, P.P., Barron, J.T., Tagliasacchi, A., Funkhouser, T., Ferrari, V.: Urban radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12932–12942 (2022)
  * [30] Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [31] Sucar, E., Liu, S., Ortiz, J., Davison, A.J.: imap: Implicit mapping and positioning in real-time. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6229–6238 (2021)
  * [32] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2446–2454 (2020)
  * [33] Turki, H., Zhang, J.Y., Ferroni, F., Ramanan, D.: Suds: Scalable urban dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12375–12385 (2023)
  * [34] Wang, H., Wang, J., Agapito, L.: Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13293–13302 (2023)
  * [35] Whelan, T., Salas-Moreno, R.F., Glocker, B., Davison, A.J., Leutenegger, S.: Elasticfusion: Real-time dense slam and light source estimation. The International Journal of Robotics Research 35(14), 1697–1716 (2016)
  * [36] Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., Wang, X.: 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 (2023)
  * [37] Yan, C., Qu, D., Wang, D., Xu, D., Wang, Z., Zhao, B., Li, X.: Gs-slam: Dense visual slam with 3d gaussian splatting. arXiv preprint arXiv:2311.11700 (2023)
  * [38] Yan, Y., Lin, H., Zhou, C., Wang, W., Sun, H., Zhan, K., Lang, X., Zhou, X., Peng, S.: Street gaussians for modeling dynamic urban scenes. arXiv preprint arXiv:2401.01339 (2024)
  * [39] Yang, J., Ivanovic, B., Litany, O., Weng, X., Kim, S.W., Li, B., Che, T., Xu, D., Fidler, S., Pavone, M., et al.: Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arXiv preprint arXiv:2311.02077 (2023)
  * [40] Yang, Z., Chen, Y., Wang, J., Manivasagam, S., Ma, W.C., Yang, A.J., Urtasun, R.: Unisim: A neural closed-loop sensor simulator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1389–1399 (2023)
  * [41] Yang, Z., Yang, H., Pan, Z., Zhu, X., Zhang, L.: Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642 (2023)
  * [42] Yugay, V., Li, Y., Gevers, T., Oswald, M.R.: Gaussian-slam: Photo-realistic dense slam with gaussian splatting. arXiv preprint arXiv:2312.10070 (2023)
  * [43] Zhang, K., Riegler, G., Snavely, N., Koltun, V.: Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 (2020)
  * [44] Zhou, Q.Y., Park, J., Koltun, V.: Open3d: A modern library for 3d data processing. arXiv preprint arXiv:1801.09847 (2018)
  * [45] Zhou, X., Lin, Z., Shan, X., Wang, Y., Sun, D., Yang, M.H.: Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. arXiv preprint arXiv:2312.07920 (2023)
  * [46] Zhu, Z., Peng, S., Larsson, V., Xu, W., Bao, H., Cui, Z., Oswald, M.R., Pollefeys, M.: Nice-slam: Neural implicit scalable encoding for slam. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12786–12796 (2022)
  * [47] Zwicker, M., Pfister, H., Van Baar, J., Gross, M.: Ewa volume splatting. In: Proceedings Visualization, 2001. VIS’01. pp. 29–538. IEEE (2001)


# Environmental effects on emergent strategy in micro-scale multi-agent
reinforcement learning

Samuel Tovey
Institute for Computational Physics
University of Stuttgart
Stuttgart, Germany 70569
<EMAIL_ADDRESS>
&David Zimmer
Institute for Computational Physics
University of Stuttgart
Stuttgart, Germany 70569
<EMAIL_ADDRESS>
&Christoph Lohrmann
Institute for Computational Physics
University of Stuttgart
Stuttgart, Germany 70569
<EMAIL_ADDRESS>
&Tobias Merkt
Institute for Computational Physics
University of Stuttgart
Stuttgart, Germany 70569
<EMAIL_ADDRESS>
&Simon Koppenhöfer
Institute for Computational Physics
University of Stuttgart
Stuttgart, Germany 70569
<EMAIL_ADDRESS>
&Veit-Lorenz Heuthe
Department of Physics - Soft Condensed Matter
Universität Konstanz
Konstanz, Germany
&Clemens Bechinger
Department of Physics - Soft Condensed Matter
Universität Konstanz
Konstanz, Germany
&Christian Holm
Institute for Computational Physics
University of Stuttgart
Stuttgart, Germany 70569
<EMAIL_ADDRESS>

###### Abstract

Multi-Agent Reinforcement Learning (MARL) is a promising candidate for
realizing efficient control of microscopic particles, of which micro-robots
are a subset. However, the microscopic particles’ environment presents unique
challenges, such as Brownian motion at sufficiently small length-scales. In
this work, we explore the role of temperature in the emergence and efficacy of
strategies in MARL systems using particle-based Langevin molecular dynamics
simulations as a realistic representation of micro-scale environments. To this
end, we perform experiments on two different multi-agent tasks in microscopic
environments at different temperatures, detecting the source of a
concentration gradient and rotation of a rod. We find that at higher
temperatures, the RL agents identify new strategies for achieving these tasks,
highlighting the importance of understanding this regime and providing insight
into optimal training strategies for bridging the generalization gap between
simulation and reality. We also introduce a novel Python package for studying
microscopic agents using reinforcement learning (RL) to accompany our results.

##  1 Introduction

In recent years, machine learning (ML) and physics have enjoyed a strong
relationship in broad fields ranging from quantum chemistry (Zaverkin et al.,
2023) to cosmology (Krippendorf and Spannowsky, 2022). An emerging application
of ML in the physical sciences is understanding the dynamics and collective
behavior of microscopic particles or colloids (Zöttl and Stark, 2023). In
these low Reynolds number regimes, the physics of Brownian motion and
hydrodynamics dominate, leading to exciting challenges (Brewer et al., 2022).
Despite these challenges, unusual collective behavior has been observed in
both biological (Sugi et al., 2021) and artificial (Zöttl and Stark, 2016)
active matter environments. A current research area in this field is
understanding how to program these artificial microscopic colloids or agents
to achieve tasks like locating positions in an environment or moving objects.
Such detailed control over microscopic agents can revolutionize many fields,
with medicine being of great interest (Soto et al., 2020). While there has
been a significant amount of progress utilizing classical algorithms (Moreau
et al., 2021; Lolli et al., 2022), recently, reinforcement learning (RL) has
become a promising candidate (Qin et al., 2023; Hartl et al., 2021; Borra et
al., 2022; Muiños-Landin et al., 2021). Most RL approaches thus far have
centered around using Q-learning for single-agent tasks. However, effective
deployment of these agents will likely involve collective behavior and,
therefore, a multi-agent setting. Furthermore, the role of the environment in
these tasks has yet to be investigated to a great extent, partially due to
many groups relying on experimental implementations of these agents and,
therefore, not having access to such variables. This also prohibits the
training time of the models, as experimentally driven RL is laborious. In this
work, we approach the problem of understanding how the Brownian motion of
these agents impacts their emergent strategy and its efficacy at different
temperatures. We utilize a powerful physics engine designed to study these
systems, ESPResSo (Weik et al., 2019), to perform these experiments at scale.
Experiments involve implementing MARL for the problems of detecting target
points in a box and rotating a rod.

Our main contributions are:

  * •

We demonstrate emergent collective behavior in RL-driven micro-robots in
realistic simulations.

  * •

We discuss the role of Brownian motion in the agents’ success in their tasks.

  * •

We demonstrate the robustness of MARL algorithms against environmental noise.

  * •

We highlight the differences in emergent strategies as a function of
temperature and comment on this as a necessary consideration in bridging the
generalization gap with experiments.

All of the infrastructure for the work performed in this investigation is
packaged into an open-source Python project, SwarmRL, which is available
publicly on GitHub.

##  2 Related Work

Much work has been dedicated to both emergent strategy in microscopic agents
and the application of RL at these length scales. A lot of this work has
focused on understanding relationships between artificial microswimmers and
their biological counterparts or applying simplified RL strategies such as
Q-learning to various problems.

#### Collective Behaviour in Artificial Microswimmers

Understanding the behavior of organisms on microscopic scales has been a
central research focus since the early 1900s. In recent years, with the
invention of artificial microswimmers, improved imaging methods, and advanced
computational systems, one has been able to probe these behaviors in much
greater detail. Of particular interest is the emergence of collective behavior
in groups of microswimmers; that is, with limited local information about
their environments, groups of agents can share knowledge to achieve a task. In
Lavergne et al. (2019) and Bäuerle et al. (2020), group formation and behavior
were studied utilizing classical interaction algorithms for microswimmers
under varying conditions. These studies demonstrated the emergence of
collective behavior for swimmers with varying degrees of sensory ability. In
their 2022 paper, Chen and Bechinger (2022) investigated the response of
microscopic swarms to external threats. This work emphasized that collective
groups of microswimmers use information sharing to respond to events that
individuals may not be aware of. In each of these studies, while the effects
of Brownian motion are implicit in the setup, the role of these fluctuations
is not explored, nor is a specific task programmed.

#### Microscopic RL

In the direction of RL, several studies have approached the problem of
organizing microswimmers to the end of achieving a goal. These studies
typically rely on RL as a policy development algorithm. Qin et al. (2023)
utilize a Q-learning algorithm to learn the swimming strategies of
microswimmers and identify new swimming gaits for linked artificial micro-
robots. In the direction of learning policy, Hartl et al. (2021) have used a
genetic algorithm to reproduce biological chemo-taxis in artificial swimmers
and Borra et al. (2022) implement actor-critic RL on a predator-prey problem
and found that agents could utilize hydrodynamic cues to avoid predators in
their environment. Finally, Muiños-Landin et al. (2021) utilized a Q-learning
approach to study how the environments of microswimmers impact their learning
process for individual swimmer tasks, including the role of Brownian motion.
However, they did not investigate the impact on collective behavior or for
more complex tasks which require algorithms beyond Q-learning.

To date, the role of the stochastic forces experienced by microswimmers in
their emergent strategy has yet to be investigated. However, these forces are
a crucial component of micro-robotics, and therefore, their impact on learning
is identified here as the primary research gap we aim to fill.

##  3 Problem Description

Controlling multi-agent systems is a challenging task in-and-of itself.
Additionally, on microscopic agents’ length scales, thermal fluctuations are
relevant. To varying degrees they will prohibit agents from performing a task
by introducing a random component preventing the outcome of an action from
being predicted with certainty. On the contrary, the existence of this
Brownian motion will encourage exploration of the agents as each action they
take is perturbed.

We study numerically a system comprising colloids on the micrometer scale that
can perform actions such as actively moving or rotating. Using RL, we want to
enable the colloids to solve tasks like finding the source of an attractant or
rotating a rod. We set out to understand the role of thermal fluctuations in
the efficacy of RL on microscopic scales and the emergent strategy adopted by
the agents.

In all simulations, a Langevin thermostat correctly reproduces the agents’
Brownian motion and maintains the system’s temperature in a statistical
physics sense. More broadly, stochastic effects can also occur at length
scales where thermal motion is not relevant, for example imperfect motors or
sensors on a robot. Our work is therefore also relevant for applications
beyond the micro-scale.

##  4 Methods

The successful implementation of these experiments has required many unique
components. Here we outline briefly the critical aspects of the work.

###  4.1 Particle Simulations

We simulate the trajectories of particles using the overdamped Langevin
equations of motion for position and orientation

$\dot{\mathbf{r}}_{i}=\frac{1}{\gamma_{t}}\left[F(t)\mathbf{e}_{i}(\Theta_{i})-\nabla
V(\mathbf{r}_{i},\\{\mathbf{r}_{j}\\})\right]+\sqrt{2k_{B}T/\gamma_{t}}\mathbf{R}^{t}_{i}(t),$
(1)
$\dot{\Theta}=\frac{1}{\gamma_{r}}\tau(t)+\sqrt{2k_{B}T/\gamma_{r}}R^{r}_{i}(t).$
(2)

Here, $\mathbf{r}_{i}$ is the (two-dimensional) position of particle $i$,
$\Theta_{i}$ the angle describing the particle orientation, $\gamma_{(t,r)}$
the translational (rotational) friction coefficient, $F$ and $\tau$ an active
force and torque corresponding to an action,
$\mathbf{e}=(\cos(\Theta),\sin(\Theta))^{T}$ the particle orientation, $V$ an
interaction potential between all particles in the system, $k_{B}$ the
Boltzmann constant, $T$ the temperature and $\mathbf{R}^{(t,r)}_{i}$ a noise
term with zero mean and correlations according to
$\expectationvalue{R^{(t,r)}_{i}(t)R^{(t,r)}_{j}(t^{\prime})}=\delta_{ij}\delta(t-t^{\prime})$,
where $\expectationvalue{\cdot}$ denotes an ensemble average. For colloids of
radius $a$ in a fluid with dynamic viscosity $\mu$ we calculate the friction
coefficient according to Stokes’ law as $\gamma_{t}=6\pi\mu a$ and
$\gamma_{r}=8\pi\mu a^{3}$.

Equations 1 and 2 are solved numerically using the ESPResSo (Weik et al.,
2019) simulation package with a time-step $\delta
t=$0.01\text{\,}\mathrm{s}$$, the actions that determine $F(t)$ and $\tau(t)$
are updated every time slice $\Delta t=$1\text{\,}\mathrm{s}$$. In all cases,
unless otherwise specified, when referring to time in this investigation, we
refer to the time slice, i.e., the number of times an action is computed for
each agent in the simulation.

We model particle interactions with the two-body Weeks-Chandler-Anderson (WCA)
potential (Weeks et al., 1971), which can be seen as an almost-hard-sphere
interaction:

$V=4\cdot
V_{0}\left(\left(\frac{\sigma}{r_{ij}}\right)^{12}-\left(\frac{\sigma}{r_{ij}}\right)^{6}\right)+V_{0}.$
(3)

Here, $r_{ij}=||\mathbf{r}_{i}-\mathbf{r}_{j}||_{2}$ is the absolute distance
between the particles, and $\sigma=2a$ the colloid diameter. We choose the
interaction strength $V_{0}=k_{B}T$.

###  4.2 RL Architecture

RL is a branch of machine learning in which an agent in state $s_{t}$
interacts with its environment by taking action $a_{t}$ that brings it to a
following state $s_{t+1}$. The agent aims to maximize each state’s scalar
reward $r_{t}$. It does so by optimizing the policy $\pi$ that governs the
agent’s behavior. MARL as a subfield of RL is closely related to game theory
and focuses on multiple such learning agents. In addition to interacting with
the environment, agents also interact with each other in cooperative or
competitive ways (Littman, 1994). At every time slice of the experiment, each
agent takes an action based on its observation. Each possible action at that
time is sampled with a probability determined by the policy. The Gumble-max
trick (Huijben et al., 2022) is used to sample this categorical distribution
efficiently. The policy is updated using a policy gradient method in which the
actor takes the role of the policy and the critic evaluates the actor’s
performance after each episode (Sutton and Barto, 2018) . The actor and the
critic are neural networks, each consisting of two dense layers with 128
neurons. The actor-critic architecture is shown in Figure 1. The Adam
optimizer with a learning rate of $\lambda=0.001$ (Kingma and Ba, 2017) is
used for the network parameter update.

Figure 1: The workflow of the MARL utilized in the experiments. An observable
is passed to an actor and a critic, each composed of two dense layers with 128
units. The actor returns a probability of selecting each of the four actions
and the critic returns the estimated value of the state.

###  4.3 Active Brownian Particles as Agents

We have used microscopic agents in all the experiments performed that mimic
active Brownian particles such as the Janus particle (Walther and Müller,
2013) shown in Figure 2A).

Figure 2: Various aspects of the agents deployed in this investigation. A) A
representation of an ABP particle coated on one side. B) Visualization of the
concentration sensing observable. C) Visualization of the vision cone
observable including the trainable embedding layers.

These particles can mimic active biological matter upon excitation from an
outside driving force. In experimental setups, this driving force is often one
or more lasers applied along a specific axis of the colloid to induce either
rotation or translation. This axis is designated the director of the colloid
and emulates a forward-facing agent. The driving mechanism is accurate enough
to allow for a well-defined action space for each colloid in the system:

$\mathcal{A}=\begin{cases}\text{Translate:}&F=10.0\text{,
}\tau=(0.0,0.0,0.0)\\\ \text{Rotate CCW:}&F=0.0\text{, }\tau=(0.0,0.0,10.0)\\\
\text{Rotate CW:}&F=0.0\text{, }\tau=(0.0,0.0,-10.0)\\\ \text{Do
Nothing:}&F=0.0\text{, }\tau=(0.0,0.0,0.0)\end{cases}$ (4)

where $F$ is the force magnitude in simulation units applied along the forward
direction of the colloid and $\tau$ is the torque. Another key feature of the
reinforcement process is the state description parsed as an input for each
agent. In this investigation, two different state descriptions, referred to
hereafter as observables, are utilized.

#### Concentration Sensing

The most simple observable used in the experiments is termed concentration
sensing. This observable aims to replicate the sensing capabilities of
biological organisms in their ability to register a change in an environmental
factor such as temperature or a food source. The observable itself computes a
change in the concentration of some field defined in this study as a potential
decaying like $1/r$. At each action update, the models receive a value
computed by:

$\centering
o_{i}(t)=f\left(||\hat{\mathbf{r}}_{i}(t)-\hat{\mathbf{r}}_{s}(t)||_{2}\right)-f\left(||\hat{\mathbf{r}}_{i}(t-\Delta
t)-\hat{\mathbf{r}}_{s}(t-\Delta t)||_{2}\right),\@add@centering$ (5)

where $o_{i}$ is the observable for the $i^{\text{th}}$ colloid, $f$ is the
chosen field, $\hat{\mathbf{r}}_{i}(t)$ is the position of the $i^{\text{th}}$
colloid at time $t$, $\Delta t$ is the amount of time since the last action
was computed, $\hat{\mathbf{r}}_{s}(t)$ is the position of the source of the
potential at time $t$, and $||\cdot||_{2}$ denotes a Euclidean norm. In our
experiments, the source remains fixed in space and will not vary in time. The
concentration sensing observable is shown graphically in Figure 2B).

#### Vision Cones

The second observable employed in our studies was the vision cone shown
graphically in Figure 2C). This approach mimics an agent with the ability to
identify particle species and a blurred sense of direction, that is, a range
of angles within which it knows other particles are present. In this study,
the vision cone is broken into five components. Each component of the vision
cones computes $N$ numbers where $N$ is the number of unique species in the
system, e.g., for rod rotation, there are active particles and rod particles
resulting in $N=2$. The value of the cone is computed by:

$\centering
o^{jk}_{i}=\sum\limits_{n\in\mathcal{C}_{j}}\frac{1}{||\mathbf{r}_{i}-\mathbf{r}^{k}_{n}||_{2}},\@add@centering$
(6)

where $o^{jk}_{i}$ is the observable of the $i^{\text{th}}$ particle for the
$j^{\text{th}}$ vision cone computed for all particles of species ${k}$ which
lie within the cone represented by $\mathcal{C}_{j}$. In order to provide this
information to an actor and critic, we implemented an embedding scheme. In
this scheme, the vision cone values associated with the rods and colloids are
split into two vectors and passed through their trainable embedding layers,
producing a reduced, 2-dimensional output. Element-wise addition of these
vectors forms the final observable used to compute actions. The embedding
layers are trained along with the actor and critic during the simulations,
allowing each model to learn a representation of its environment.

###  4.4 SwarmRL

The RL performed in this investigation is part of a project involved with not
only training micro-swimmers but also deploying trained models in natural
experiments. Therefore, all of the infrastructure surrounding the training of
the agents and deployment in the ESPReSso simulation engine has been written
into an open-source Python package SwarmRL. As illustrated in Figure 3,
SwarmRL provides an interface to each component of the RL pipeline as well as
different environments including both the simulation engine and actual
experiment setups. SwarmRL is built on the JAX (Bradbury et al., 2018)
ecosystem and utilizes Flax (Heek et al., 2023) for neural networks.

Figure 3: Simplified architecture of SwarmRL displaying data communication
between the RL models and the experiments.

SwarmRL is publicly available on GitHub at https://github.com/SwarmRL/SwarmRL.

##  5 Experiments

This investigation performs two experiments: source detection and rod
rotation. Each task requires a differing degree of complexity, and in each
case, the agents discover different strategies depending on their environment.
Training for each task occurs in the same way. Each model is trained with ten
agents for a total of 10.000 episodes. The first 5000 episodes are performed
using an exploration rate of 20 %; that is, 20 % of the time, colloids chose a
random action other than that provided by the actor. The second 5000 episodes
were performed without this exploration policy. This is performed for ten
ensembles with different starting simulation conditions and neural network
initialization. In all computations, an average value over these ensembles is
computed, resulting in the corresponding error values. At the end of the
training, all the trained models are used in production simulations of
5.000.000 time-steps. This procedure was carried out for five different
temperatures: {0 K, 150 K, 273 K, 300 K, 350 K } to explore the changes in
emergent strategy and efficacy of chosen strategy.

###  5.1 Source Detection

The simplest task tested was that of source detection. In this case, the
colloids have a biologically inspired sense of smell; they can sense changes
in some applied concentration field. Their reward is based on changes in this
field, i.e., moving closer to the source of the concentration yields a better
reward. Such a task is reminiscent of chemo-taxis in bacteria (Zhuang and
Sitti, 2016) or, theoretically, detection of a drug delivery site in medicinal
micro-robotics (Schmidt et al., 2020). Mathematically, the reward closely
resembles the observables of these colloids:

$\centering
r_{i}=\alpha\cdot\text{Clip}\left(\frac{1}{||\hat{\mathbf{r}}_{i}(t)-\hat{\mathbf{r}}_{s}(t)||_{2}}-\frac{1}{||\hat{\mathbf{r}}_{i}(t-1)-\hat{\mathbf{r}}_{s}(t-1)||_{2}}\text{,
}0\text{, None}\right)\@add@centering$ (7)

where $\alpha$ is some reward scaling value, the clip operation ensures that
particles are rewarded for moving toward the source and receive no input if
they move away. The experiment results are outlined in Figure 4.

Figure 4: A) The policy evaluation performed on trained actors. Each figure
shows the probability of taking a certain action given the input to the
network. B) The mean distance of all agents to the source in the simulation as
a function of time. Different line styles and colors correspond to different
temperatures. C) A zoomed-in frame of B) where the higher temperature models
converge faster and closer to the source of their sensing field. D) Histograms
computed on the converged regions of the trajectory. E)-F) Example trajectory
of the agents at 0 K and 350 K respectively.

Figure 4A) illustrates the policy breakdown for inputs to the neural network
at different temperatures. To construct these plots, we pass artificial data
through the trained models and compute the probability of different actions
for these inputs. We see that negative inputs in each temperature, i.e., a
movement towards lower values of the concentration field, result in increased
rotation probability and decreased translation. This is opposed to movements
towards the source, which increases translation probability. Such a learned
policy is reminiscent of run-and-tumble motion in bacteria in response to
changing environments (van Ginkel et al., 2021). It appears that at all
temperatures, the Do Nothing actions is never included in the learned strategy
and remains a randomly selected action. We also noticed in these experiments
that once the agents learned to rotate either clockwise or counterclockwise,
they did not use the other action.

Figure 4B), C), and D) illustrate the distance of the colloids from the source
at different temperatures. In Figure 4C), the colloids trained at higher
temperatures converge faster onto the source. This is expanded upon in 4D)
where the histograms of the equilibrium region are plotted for each
temperature. These histograms show that models trained at higher temperatures
converge closer to the source of the field and, in general, experience less
variance around it. This is generally true; however, it can also be seen that
the 150 K model reached the closest to the source while higher temperatures
moved back away. The mechanism behind this improved policy can be understood
by looking directly at the trajectories of the models in Figures 4E) and F).
In the 0K simulation shown in 4E), the colloids undertake an orbital movement
towards the source and, in doing so, maximize their reward over time. In the
case of a 350 K environment shown in 4F), the colloids cannot afford to take
such a policy as the random fluctuations will push them out of their orbit. In
these cases, the colloids move more directly toward the source and perform
more oscillatory movement once they are close. This approach explains the
improved proximity of the higher temperature models as they orient themselves
to move directly to their target. However, the stronger the Brownian forces
acting on the colloids, the harder it will be to remain close to the source
and the more significant are their fluctuations around it. In summary,
including Brownian forces in the model training results in a more direct
approach to the source of the field attracting the colloids. This policy
results in closer average distances to the source and less variation around
the it.

###  5.2 Rod Rotation

Increasing the complexity of our experiments, the subsequent investigation
involved training the agents to rotate a rod. In these experiments, the rod is
modeled as a series of rigidly bonded small colloids so that they cannot move
away from one another. This task is interesting as the colloids should
cooperate to achieve this task efficiently. The observables we use in the rod
rotation experiments are the vision cones discussed in Section 4.3, which
return values for both the agents and the rod colloids. The rewards for this
task are broken into two components, detecting the rod and increasing its
angular velocity. Rod detection is encouraged by rewarding the agents if they
move closer to the rod particles:

$\centering r^{\text{rod
proximity}}_{i}=\sum\limits_{j}\frac{1}{\hat{\mathbf{r}}_{i}(t)-\hat{\mathbf{r}}_{j}(t)}-\sum\limits_{j}\frac{1}{\hat{\mathbf{r}}_{i}(t-\Delta
t)-\hat{\mathbf{r}}_{j}(t-1)},\@add@centering$ (8)

where $\hat{\mathbf{r}}_{j}$ are all particles in the rod. The reward for the
rod rotation itself is more involved as a partitioning scheme is used in order
to compute a meaningful single-agent contribution which is applied to the
total reward as:

$\centering r^{\text{rod
rotation}}_{i}=\frac{\mathbf{\tau}_{i}}{\tau_{\text{net}}}\cdot\left(\omega_{\text{rod}}(t)-\omega_{\text{rod}}(t-\Delta
t)\right),\@add@centering$ (9)

where $\tau_{i}$ is the torque exerted on the rod by colloid $i$,
$\tau_{\text{net}}$ is the net torque acting on the rod, and
$\omega_{\text{rod}}(t)$ is the angular velocity of the rod at time $t$. The
final reward for the colloid is then computed with the following:

$\centering r_{i}=\alpha\cdot r^{\text{rod proximity}}_{i}+\beta\cdot
r^{\text{rod rotation}}_{i},\@add@centering$ (10)

where $\alpha$ and $\beta$ are scaling factors chosen in this work to be 10
and 100, respectively, such that successful rod rotation dominates the reward.
The results of the rod rotation experiment are outlined in Figures 5 A)-C).

Figure 5: A) The 2D histogram formed using the trajectory of a single 150 K
deployment simulation. B) The rod’s angular velocity minus the corresponding
value for a system without agents as a function of temperature. C) A histogram
along the rod’s long axis demonstrates the accumulation at the ends of the rod
for each temperature.

During the rod rotation experiments, two observables were of interest: the
angular velocity of the rod, which measures the agents’ efficacy, and the
agents’ distribution along the rod, which provides a description of their
strategy. Figure 5B) shows the average absolute rod velocity as a function of
temperature. In order to account for the natural motion of the rod,
simulations have been performed without active colloids and these rod
velocities subtracted from the active values. It is clear from the
measurements that the agents are capable of achieving greater rod velocities
at higher temperatures. This is of interest as the 0 K simulation provides the
least resistance to rotation as no effects of the Brownian motion present. As
the temperature continues to increase, the mean rod velocity oscillates and
decreases, suggesting that at some stage, the random fluctuations again become
problematic for the agents’ strategy.

The policy of the agents can be seen in Figures 5 A) and C). Figure 5A) plots
the 2D histogram of the colloids for one of the 150 K simulation. It is
evident in this plot that the agents form on either end of the rod in a manner
that maximises the applied torque. A clearer picture of this strategy emerges
when studying only the single-dimensional histogram along the long axis of the
rod, shown in Figure 5C). Here we can see that concentration along the ends of
the rod is an emergent strategy for all temperatures. This concentration is
maximized at 150 K, possibly explaining the enhanced angular velocity in
Figure 5C). As temperature increases it seems that the agents move towards the
center of the rod, perhaps preventing them for sliding off due to the
additional random forces. This trend is broken in the 350 K simulation where
the agents once again form on the ends of the rod, although in a more
asymmetric fashion, perhaps suggesting an alternative strategy in these
conditions.

##  6 Conclusion and Outlook

We have performed experiments using the ESPResSo simulation engine to
understand how temperature impacts the emergent strategy and efficacy of this
strategies in micro-scale agents governed by MARL. In the location detection
experiments resembling biological chemotaxis, we found that increasing the
temperature resulted in the agents taking direct approaches toward the source
of the field. In contrast, with no Brownian motion, they entered a decaying
orbit. This was accompanied by the agents equilibrating closer to the source
at higher temperatures, suggesting the strategy was more effective than
orbiting. The most exciting outcome was the policy adopted by the agents as we
saw them take the translation action upon a move closer to the source and more
rotation as they moved away. This approach strategy is heavily reminiscent of
bacterial run and tumble motion. Rod rotation also demonstrated an evolving
strategy with temperature as we saw the colloids favor sitting on the ends of
the rods, thereby maximizing the applied torque. However, after 150 K the
colloids began to favor moving inwards along the rod, presumably to avoid
sliding off during rotation. These results demonstrate that the environments
of microscopic, intelligent agents profoundly impact their efficacy and the
strategies they need to adopt to achieve different tasks. Continued work in
this area could look into more efficient training strategies to achieve the
performance of the models at 150 K in the 350 K simulations. Further, training
models in simulation environments to be robust against these environmental
factors could result in a smaller generalization gap for crossing into
physical experiments. Micro-robotics and MARL are, without a doubt,
technologies of the future. The key to unlocking their success lies in
understanding the conditions under which they will work and using these to
their advantage.

##  7 Acknowledgements

V.L.H and C.B acknowledge funding from the DFG Centre of Excellence 2117,
Germany ”Centre for the Advances Study of Collective Behaviour”, ID:
422037984. C.H and S.T acknowledge financial support from the German Funding
Agency (Deutsche Forschungsgemeinschaft DFG) under Germany’s Excellence
Strategy EXC 2075-390740016, and S. T was supported by a LGF stipend of the
state of Baden-Württemberg. C.H, and S.T acknowledge financial support from
the German Funding Agency (Deutsche Forschungsgemeinschaft DFG) under the
Priority Program SPP 2363.

## References

  * Bäuerle et al. [2020] Tobias Bäuerle, Robert C. Löffler, and Clemens Bechinger.  Formation of stable and responsive collective states in suspensions of active colloids.  _Nature Communications_ , 11(1):2547, May 2020\.  ISSN 2041-1723.  doi: 10.1038/s41467-020-16161-4.  URL https://doi.org/10.1038/s41467-020-16161-4.
  * Borra et al. [2022] Francesco Borra, Luca Biferale, Massimo Cencini, and Antonio Celani.  Reinforcement learning for pursuit and evasion of microswimmers at low reynolds number.  _Phys. Rev. Fluids_ , 7:023103, Feb 2022.  doi: 10.1103/PhysRevFluids.7.023103.  URL https://link.aps.org/doi/10.1103/PhysRevFluids.7.023103.
  * Bradbury et al. [2018] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Brewer et al. [2022] Peter G. Brewer, Edward T. Peltzer, and Kathryn Lage.  Life at low reynolds number re-visited: The efficiency of microbial propulsion.  _Deep Sea Research Part I: Oceanographic Research Papers_ , 185:103790, 2022.  ISSN 0967-0637.  doi: https://doi.org/10.1016/j.dsr.2022.103790.  URL https://www.sciencedirect.com/science/article/pii/S0967063722001030.
  * Chen and Bechinger [2022] Chun-Jen Chen and Clemens Bechinger.  Collective response of microrobotic swarms to external threats.  _New Journal of Physics_ , 24(3):033001, mar 2022\.  doi: 10.1088/1367-2630/ac5374.  URL https://dx.doi.org/10.1088/1367-2630/ac5374.
  * Hartl et al. [2021] Benedikt Hartl, Maximilian Hübl, Gerhard Kahl, and Andreas Zöttl.  Microswimmers learning chemotaxis with genetic algorithms.  _Proceedings of the National Academy of Sciences_ , 118(19):e2019683118, 2021.  doi: 10.1073/pnas.2019683118.  URL https://www.pnas.org/doi/abs/10.1073/pnas.2019683118.
  * Heek et al. [2023] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee.  Flax: A neural network library and ecosystem for JAX, 2023.  URL http://github.com/google/flax.
  * Huijben et al. [2022] Iris A. M. Huijben, Wouter Kool, Max B. Paulus, and Ruud J. G. van Sloun.  A review of the gumbel-max trick and its extensions for discrete stochasticity in machine learning, 2022.
  * Kingma and Ba [2017] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * Krippendorf and Spannowsky [2022] Sven Krippendorf and Michael Spannowsky.  A duality connecting neural network and cosmological dynamics.  _Machine Learning: Science and Technology_ , 3(3):035011, aug 2022.  doi: 10.1088/2632-2153/ac87e9.  URL https://dx.doi.org/10.1088/2632-2153/ac87e9.
  * Lavergne et al. [2019] François A. Lavergne, Hugo Wendehenne, Tobias Bäuerle, and Clemens Bechinger.  Group formation and cohesion of active particles with visual perception–dependent motility.  _Science_ , 364(6435):70–74, 2019.  doi: 10.1126/science.aau5347.  URL https://www.science.org/doi/abs/10.1126/science.aau5347.
  * Littman [1994] Michael L. Littman.  Markov games as a framework for multi-agent reinforcement learning.  In William W. Cohen and Haym Hirsh, editors, _Machine Learning Proceedings 1994_ , pages 157–163. Morgan Kaufmann, San Francisco (CA), 1994.  ISBN 978-1-55860-335-6.  doi: https://doi.org/10.1016/B978-1-55860-335-6.50027-1.  URL https://www.sciencedirect.com/science/article/pii/B9781558603356500271.
  * Lolli et al. [2022] Alberto Lolli, Giovanni Corsi, and Antonio DeSimone.  Control and navigation problems for model bio-inspired microswimmers.  _Meccanica_ , 57(10):2431–2445, Oct 2022.  ISSN 1572-9648.  doi: 10.1007/s11012-022-01567-9.  URL https://doi.org/10.1007/s11012-022-01567-9.
  * Moreau et al. [2021] Clément Moreau, Kenta Ishimoto, Eamonn A. Gaffney, and Benjamin J. Walker.  Control and controllability of microswimmers by a shearing flow.  _Royal Society Open Science_ , 8(8):211141, 2021\.  doi: 10.1098/rsos.211141.  URL https://royalsocietypublishing.org/doi/abs/10.1098/rsos.211141.
  * Muiños-Landin et al. [2021] S. Muiños-Landin, A. Fischer, V. Holubec, and F. Cichos.  Reinforcement learning with artificial microswimmers.  _Science Robotics_ , 6(52):eabd9285, 2021.  doi: 10.1126/scirobotics.abd9285.  URL https://www.science.org/doi/abs/10.1126/scirobotics.abd9285.
  * Qin et al. [2023] Ke Qin, Zonghao Zou, Lailai Zhu, and On Shun Pak.  Reinforcement learning of a multi-link swimmer at low Reynolds numbers.  _Physics of Fluids_ , 35(3), 03 2023.  ISSN 1070-6631.  doi: 10.1063/5.0140662.  URL https://doi.org/10.1063/5.0140662.  032003\.
  * Schmidt et al. [2020] Christine K. Schmidt, Mariana Medina-Sánchez, Richard J. Edmondson, and Oliver G. Schmidt.  Engineering microrobots for targeted cancer therapies from a medical perspective.  _Nature Communications_ , 11(1):5618, Nov 2020\.  ISSN 2041-1723.  doi: 10.1038/s41467-020-19322-7.  URL https://doi.org/10.1038/s41467-020-19322-7.
  * Soto et al. [2020] Fernando Soto, Jie Wang, Rajib Ahmed, and Utkan Demirci.  Medical Micro/Nanorobots in precision medicine.  _Adv Sci (Weinh)_ , 7(21):2002203, October 2020\.
  * Sugi et al. [2021] Takuma Sugi, Hiroshi Ito, and Ken H Nagai.  Collective pattern formations of animals in active matter physics.  _Biophys Physicobiol_ , 18:254–262, October 2021.
  * Sutton and Barto [2018] Richard S. Sutton and Andrew G. Barto.  _Reinforcement Learning: An Introduction_.  The MIT Press, second edition, 2018.  URL http://incompleteideas.net/book/the-book-2nd.html.
  * van Ginkel et al. [2021] Bart van Ginkel, Bart van Gisbergen, and Frank Redig.  Run-and-tumble motion: The role of reversibility.  _Journal of Statistical Physics_ , 183(3):44, Jun 2021.  ISSN 1572-9613.  doi: 10.1007/s10955-021-02787-1.  URL https://doi.org/10.1007/s10955-021-02787-1.
  * Walther and Müller [2013] Andreas Walther and Axel H. E. Müller.  Janus particles: Synthesis, self-assembly, physical properties, and applications.  _Chemical Reviews_ , 113(7):5194–5261, 2013.  doi: 10.1021/cr300089t.  URL https://doi.org/10.1021/cr300089t.  PMID: 23557169.
  * Weeks et al. [1971] John D Weeks, David Chandler, and Hans C Andersen.  Role of repulsive forces in determining the equilibrium structure of simple liquids.  _The Journal of chemical physics_ , 54(12):5237–5247, 1971.
  * Weik et al. [2019] Florian Weik, Rudolf Weeber, Kai Szuttor, Konrad Breitsprecher, Joost de Graaf, Michael Kuron, Jonas Landsgesell, Henri Menke, David Sean, and Christian Holm.  Espresso 4.0 – an extensible software package for simulating soft matter systems.  _The European Physical Journal Special Topics_ , 227(14):1789–1816, Mar 2019.  ISSN 1951-6401.  doi: 10.1140/epjst/e2019-800186-9.  URL https://doi.org/10.1140/epjst/e2019-800186-9.
  * Zaverkin et al. [2023] Viktor Zaverkin, David Holzmüller, Luca Bonfirraro, and Johannes Kästner.  Transfer learning for chemically accurate interatomic neural network potentials.  _Phys. Chem. Chem. Phys._ , 25:5383–5396, 2023.  doi: 10.1039/D2CP05793J.  URL http://dx.doi.org/10.1039/D2CP05793J.
  * Zhuang and Sitti [2016] Jiang Zhuang and Metin Sitti.  Chemotaxis of bio-hybrid multiple bacteria-driven microswimmers.  _Scientific Reports_ , 6(1):32135, Aug 2016.  ISSN 2045-2322.  doi: 10.1038/srep32135.  URL https://doi.org/10.1038/srep32135.
  * Zöttl and Stark [2023] Andreas Zöttl and Holger Stark.  Modeling active colloids: From active brownian particles to hydrodynamic and chemical fields.  _Annual Review of Condensed Matter Physics_ , 14(1):109–127, 2023.  doi: 10.1146/annurev-conmatphys-040821-115500.  URL https://doi.org/10.1146/annurev-conmatphys-040821-115500.
  * Zöttl and Stark [2016] Andreas Zöttl and Holger Stark.  Emergent behavior in active colloids.  _Journal of Physics: Condensed Matter_ , 28(25):253001, may 2016.  doi: 10.1088/0953-8984/28/25/253001.  URL https://dx.doi.org/10.1088/0953-8984/28/25/253001.