# Neural Implicit Swept Volume Models for Fast Collision Detection

Dominik Joho∗ Jonas Schwinn∗ Kirill Safronov ∗ Equal contribution.
The authors are with KUKA Deutschland GmbH, Germany. {Dominik.Joho,
Jonas.Schwinn<EMAIL_ADDRESS>
###### Abstract

Collision detection is one of the most time-consuming operations during motion
planning. Thus, there is an increasing interest in exploring machine learning
techniques to speed up collision detection and sampling-based motion planning.
A recent line of research focuses on utilizing neural signed distance
functions of either the robot geometry or the swept volume of the robot
motion. Building on this, we present a novel neural implicit swept volume
model to continuously represent arbitrary motions parameterized by their start
and goal configurations. This allows to quickly compute signed distances for
any point in the task space to the robot motion. Further, we present an
algorithm combining the speed of the deep learning-based signed distance
computations with the strong accuracy guarantees of geometric collision
checkers. We validate our approach in simulated and real-world robotic
experiments, and demonstrate that it is able to speed up a commercial bin
picking application.

##  I Introduction

Bin picking is a crucial task in logistics and industrial automation
processes. Solving this task typically involves several steps, such as object
localization, grasp pose selection, and collision-free motion planning [1],
[2]. There exist multiple paradigms for finding a collision-free path: for
example, one can use a path planer (sampling-based or optimization-based) to
construct a single collision-free path. In practice, bin picking applications
often follow a different paradigm where a set of paths is first heuristically
generated and subsequently checked for collisions until a first collision-free
path has been found.

Besides object detection, these collision checks are one of the most time-
consuming steps in bin picking applications. Geometric collision detection
methods typically discretize the motion along a path into small-enough steps
and check the resulting robot configurations against the environment [3, 4].
In this paper, we follow a different approach utilizing the so-called swept
volume of a robot motion, which is the subset of the workspace that the robot
moves through during this motion. An explicit representation of this volume
would allow to replace the individual collision checks along a path segment
with a single collision check of this swept volume against the environment.

Figure 1: The basic idea of our approach is to predict the signed distance
(black line) of a point in the task space (black dot) to its nearest point
(white dot) on the surface of the swept volume (blue mesh) of a robot motion.
The swept volume is implicitly represented by a neural network, which takes
the start and goal configuration of the motion as well as the query point as
inputs and outputs the signed distance of the point to the implied swept
volume. The robot motion can be collision checked against a complete
environment, by representing the environment as a point cloud. Each of these
points becomes one input point for the network.

However, computing an _explicit_ representation of the swept volume comes with
its own drawbacks. It is usually done via voxel grid approximations, or by
geometrically approximating the boundary of this volume, e.g., as a triangle
mesh [5]. By design the accuracy of these approaches is strongly linked to
their computation time, which hinders their application for collision checking
in live systems.

We thus propose an _implicit_ representation based on a deep neural network
that learns the function
$(\mathbf{x},\mathbf{q}_{0},\mathbf{q}_{1})\mapsto\delta$, where
$\mathbf{q}_{0}$ is the start configuration of the robot, $\mathbf{q}_{1}$ is
the goal configuration, $\mathbf{x}\in\mathbb{R}^{3}$ is the query point in
the task space for which we want to perform a collision check, and $\delta$ is
the signed distance of the query point to the resulting swept volume when
performing the motion $\mathbf{q}_{0}\rightarrow\mathbf{q}_{1}$, see also Fig.
1. This approach learns a continuous implicit swept volume model, i.e., it can
predict a signed distance for any point in the task space, and is not limited
to a fixed resolution of a voxel grid. Further, for a given network the
inference time is independent of the length of the motion.

This approach can be used to check a robot motion against a complete
environment represented as a point cloud. Each point $\mathbf{x}$ of this
point cloud becomes one input for the network. Multiple points can be batch
processed leveraging parallel computation on GPUs, which allows for favorable
computation times compared to geometric collision checkers (GCC). However, a
downside of this approach is that the distance computed by a neural network is
less reliable when compared to a GCC. Neural networks cannot be guaranteed to
extrapolate robustly to data not seen during training [6], [7], [8]. This
hinders applications in industrial settings, as trading off accuracy against
speed is not acceptable when it comes to safety and potential collisions.

We therefore propose an algorithm that interleaves our implicit swept volume
model with a GCC. As we will show in our experiments, this effectively reduces
the number of paths that have to be checked by the GCC and thus also reduces
the total time spent to find a collision-free path, even when considering the
additional time spent on computing the swept volume distances.

While our primary example revolves around a bin picking application, it is
worth noting that the same model is also suitable for collision checks in
general sampling-based motion planners. Moreover, while our evaluation focuses
on a robotic arm, our approach can be broadly applied to any articulated
object that allows for swept volume computation.

The main contributions of this paper are: (a) a novel approach to modeling
robot motions as implicit swept volumes parameterized by the start and goal
configuration with a neural network, and (b) an algorithm that utilizes these
neural swept volumes to speed up collision checking while preserving accuracy
guarantees of GCCs.

##  II Related Work

Chiang et al. [9] proposed a neural network that learned to map the start and
goal configuration of a robot motion to the swept volume (as a scalar value)
of this motion. The motion is assumed to be linear in joint space. This is not
usable for collision detection, but has been used as a heuristic to speed up a
sampling-based motion planner. Building on this, Baxter et al. [10] proposed a
network that uses the same input, but outputs a voxel grid representation of
the swept volume (voxels can either be “swept” or “not swept”). This method
can be used for collision checks, but on the downside it has an inherent
discretization error determined by the size of the voxel grid cells, and a low
spatial resolution due to the limited number of voxels. In contrast, in a
feasibility study Lee et al. [11] used the deep signed distance model of Gropp
et al. [12] to model swept volumes implicitly as a signed distance field
(SDF), thereby avoiding discretizations. However, their model is not
conditioned on motion parameters, but directly maps a point in the task space
to the signed distance of an implicit swept volume. Thus, each trained network
represents the swept volume of only one specific motion. Michaux et al. [13]
learn a parameterized signed distance function of zonotope approximations of a
box obstacle and a swept volume of a robot motion. The inputs of their network
are the box obstacle center, as well as the initial configuration, speed, and
acceleration of the robot, which uniquely determines the motion of a robot.
They use this SDF as a collision avoidance constraint in receding time horizon
planning.

Besides modeling swept volumes, which represent complete motions, deep neural
networks have also been used in motion planning for modeling the SDF of the
robot geometry at a given joint configuration. Liu et al. [14] learned a SDF
for articulated objects. Their model takes as input a 3D query point and the
configuration of the articulated object (robot or human), and outputs the
signed distance. They used the model in motion planning for computing
repulsive velocities. Koptev et al. [15] presented a model that, given a joint
configuration, computes the minimal distance for each link of a robot to a
given 3D query point. They use the gradients of the distances for collision
avoidance in their control loop. Li et al. [16] proposed to learn a SDF for
each link based on basis functions and ridge regression and compare their
approach to deep learning-based methods.

In contrast to Baxter et al. [10], our model is a SDF which can continuously
represent the distance function and thereby avoids discretization errors and
the computational burden of the network having to output a dense 3D voxel
grid. In contrast to Lee et al. [11], our model is conditioned on the motion
parameters, and thus the trained network is generally applicable to any motion
of the robot. Closest to our implicit swept volume model is the work of
Michaux et al. [13]. They use a specific motion type in which the motion is
partitioned in one phase with constant acceleration, followed by a second
phase with constant deceleration. Their swept volumes are parameterized by the
initial configuration, speed, and acceleration of the motion. We allow for
arbitrary motions parameterized by their start and goal configuration.
Further, we represent the obstacles nonparametrically as point clouds, while
they use zonotope approximations for the swept volume and the box obstacles.

##  III Implicit Swept Volume Model

The main idea of our approach is to compute the signed distance of a point in
the task space to the swept volume of a robot motion. If the signed distance
is positive (or greater than a given safety margin), we would consider this
motion as collision-free w.r.t. the given point. To check the robot motion
against the complete environment, we represent the environment as a point
cloud and check the individual points against the swept volume.

We are thus interested in a function $f_{d}$ that outputs the signed distance
of a given point $\mathbf{x}\in\mathbb{R}^{3}$ in the task space to the swept
volume $V_{S}(\mathbf{m})\subset\mathbb{R}^{3}$ of a given motion
$\mathbf{m}$:

$f_{d}\colon(\mathbf{x},\mathbf{m})\mapsto d(\mathbf{x},V_{S}(\mathbf{m})),$
(1)

where $d(\mathbf{x},V)$ is the signed distance of a point $\mathbf{x}$ to the
boundary of some volume $V$. The motion is considered collision-free if
$f_{d}(\mathbf{x},\mathbf{m})>0$. Here, $\mathbf{m}$ are motion parameters
that suffice to uniquely define a concrete motion of a certain motion type,
e.g., a linear motion in joint space parameterized by the start and goal
configurations of the robot: $\mathbf{m}=(\mathbf{q}_{0},\mathbf{q}_{1})$,
where $\mathbf{q}_{0}$ and $\mathbf{q}_{1}\in\mathbb{R}^{\text{dof}}$ are
joint configurations and dof is the degrees of freedom of the robot. We can
denote the motion of the robot in the joint space as

$c\colon(\mathbf{m},s)\mapsto\mathbf{q}_{s}\qquad s\in[0,1]$ (2)

where the motion is supposed to start at $s=0$ and end at $s=1$. If we denote
by $V_{R}(\mathbf{q}_{s})\subset\mathbb{R}^{3}$ the volume in the task space
occupied by the robot at the joint configuration $\mathbf{q}_{s}$, then the
swept volume can be defined as

$\displaystyle V_{S}(\mathbf{m})$
$\displaystyle=\bigcup_{s\in[0,1]}V_{R}(c(\mathbf{m},s))$ (3)
$\displaystyle=\bigcup_{s\in[0,1]}V_{R}(\mathbf{q}_{s}).$ (4)

Computing an explicit representation that approximates the swept volume
$V_{S}$, e.g., as a triangle mesh, and using this representation to compute
the signed distance to the point $\mathbf{x}$ would be too slow to be useful
in motion planning. We therefore learn an implicit representation of the swept
volume as a signed distance field, by training a deep neural network that
directly computes the function $f_{d}$.

##  IV Method

In the following, we describe the network architecture, the integration with
geometric collision checking, and the generation of training data.

###  IV-A Network Architecture

We use a simple, fully connected neural network $f_{\bm{\theta}}$ with
parameters $\bm{\theta}$ to approximate the function $f_{d}$ of (1). The main
component of our network is a block consisting of a sequence of layers. These
are a linear layer with constant input and output dimension, a ReLU activation
function, and a BatchNorm layer. We introduce skip connections [17] for each
block adding the block input to the block output. The block dimension
$n_{\text{dim}}$ denotes the input (and output) dimension of its linear layer.
Our networks are composed of $n_{\text{b}}$ blocks of equal dimension
$n_{\text{dim}}$. These are the two most important hyperparameters of the
architecture. Inspired by [18], we further introduce jump connections,
concatenating the input of the network to each output of a block. Finally, the
blocks are enclosed in an initial linear layer mapping from the input
dimension to the block dimension, and a final layer mapping the block
dimension to a single scalar value, the predicted signed distance. The
architecture of the network is depicted in Fig. 2.

Figure 2: Network architecture: each blue rectangle represents one of the
$n_{\text{b}}$ blocks of the network. The input (green) of the network is fed
into each block individually. The gray layers are linearly mapping the input
and output dimension of the network to the constant block dimension
$n_{\text{dim}}$.

###  IV-B Integration with Geometric Collision Checking

We address the problem of finding a collision-free motion in a predefined set
of motions $\mathbf{M}=\\{\mathbf{m}_{j}\\}_{j=1}^{m}$. Given a point cloud
$\mathbf{X}=\\{\mathbf{x}_{i}\in\mathbb{R}^{3}\\}_{i=1}^{n}$ of the
environment and some motion $\mathbf{m}\in\mathbf{M}$, we are interested in
the minimum distance of the point cloud to the swept volume

$d(\mathbf{X},\mathbf{m})\coloneqq\min\limits_{\mathbf{x}\in\mathbf{X}}d(\mathbf{x},V_{S}(\mathbf{m})).$
(5)

To find a collision-free motion in $\mathbf{M}$, we like to find some
$\mathbf{m}\in\mathbf{M}$ satisfying $d(\mathbf{X},\mathbf{m})>\epsilon_{S}$,
where $\epsilon_{S}$ is a safety margin set by the user. Computation of
$d(\mathbf{X},\mathbf{m})$ for each $\mathbf{m}\in\mathbf{M}$ by a GCC is
time-consuming. Therefore, we make use of our neural implicit swept volume
model $f_{\bm{\theta}}(\cdot,\cdot)$ and compute

$\hat{d}(\mathbf{X},\mathbf{m})\coloneqq\min\limits_{\mathbf{x}\in\mathbf{X}}f_{\bm{\theta}}(\mathbf{x},\mathbf{m})$
(6)

as a proxy for $d(\mathbf{X},\mathbf{m})$. While this speeds up the distance
computation, the resulting estimates are now less reliable. We therefore
propose to interleave fast computations $\hat{d}(\mathbf{X},\mathbf{m})$ with
the exact distance computations $d(\mathbf{X},\mathbf{m})$ by the following
algorithm, which we refer to as (GCC-NN):

Algorithm 1 (GCC-NN)

1:Input: $\mathbf{M}$, $\mathbf{X}$, $\hat{d}(\cdot,\cdot)$, $d(\cdot,\cdot)$,
$\epsilon_{S}$

2:$\mathbf{M}_{\text{done}}\leftarrow\emptyset$

3:for $\mathbf{m}\in\mathbf{M}$ do $\triangleright$ First Loop

4: $\hat{\delta}\leftarrow\hat{d}(\mathbf{X},\mathbf{m}$)

5: if $\hat{\delta}>\epsilon_{S}$ then

6: $\delta\leftarrow d(\mathbf{X},\mathbf{m}$)

7: $\mathbf{M}_{\text{done}}\leftarrow\mathbf{M}_{\text{done}}\cup\mathbf{m}$

8: if $\delta>\epsilon_{S}$ then

9: return $\mathbf{m},\delta$

10:Sort $\mathbf{M}\setminus\mathbf{M}_{\text{done}}$ by $\hat{\delta}$ in
descending order

11:for $\mathbf{m}\in\mathbf{M}\setminus\mathbf{M}_{\text{done}}$ do
$\triangleright$ Second Loop

12: $\delta\leftarrow d(\mathbf{X},\mathbf{m})$

13: if $\delta>\epsilon_{S}$ then

14: return $\mathbf{m},\delta$

15:return No collision-free motion found

In a first loop, we run the network for each motion and only invoke the GCC if
a signed distance $\hat{\delta}$ greater than the safety margin $\epsilon_{S}$
is predicted. If a collision-free motion is found, the algorithm returns. In
case no collision-free motion is detected in the first loop, we resume with
all motions which have not yet been processed by the GCC. We sort the
remaining motions according to their predicted distances $\hat{\delta}$ in
descending order. Then, we run a second loop, where we check each motion with
the GCC until a collision-free motion is found or the algorithm returns with
no collision-free motion found. We thereby effectively reduce the average
number of exact collision checks that need to be executed before finding the
first collision-free motion in $\mathbf{M}$.

###  IV-C Dataset Generation

Our training datasets consist of tuples
$(\mathbf{x},\mathbf{q}_{0},\mathbf{q}_{1},\delta)$ with query points
$\mathbf{x}$, the start and goal joint configurations, $\mathbf{q}_{0}$ and
$\mathbf{q}_{1}$, and the label: the signed distance $\delta$.

Our method is applicable to any type of motion, which can be uniquely defined
by $\mathbf{q}_{0}$ and $\mathbf{q}_{1}$, since the exact motion is implicitly
defined via the training data. Therefore, we will consider two motion types: a
linear joint space motion, and a spline motion in the task space (as used by
KUKA Agilus robots). We will call the respective datasets LINEAR and SPLINE.

For a single motion, we first sample $\mathbf{q}_{0}$ and $\mathbf{q}_{1}$
uniformly from within the individual joint limits. For a linear joint space
motion, we discretize this motion into 200 steps by linearly interpolating in
joint space between $\mathbf{q}_{0}$ and $\mathbf{q}_{1}$. For a spline
motion, we use equidistant steps of the flange along the spline in the task
space. This yields a varying number of steps per motion, though comparably
many as for the linear case. For a given discretized motion, we compute the
sequence of poses of each link of the robot. The associated geometries of a
link along with their pose sequence is the input for the swept volume
algorithm [5] as implemented by the gpytoolbox library [19]. This method
outputs the swept volume as a triangle mesh. We use an epsilon parameter of 15
mm and otherwise use the default parameters of the library. The swept volumes
of the individual link geometries are then merged into a final swept volume by
using the Boolean union operation of libigl [20].

Note, that the discretization accuracy is thereby determined during data
generation, where arbitrary methods for swept volume generation can be used.
This only affects the quantity of training data and the time which is
necessary to train the models. The inference time of the trained model is
therefore mostly independent of the discretization accuracy.

Given the mesh, we sample points in the task space and compute their distances
to the mesh. The points are sampled in a such a way, that we have a global
coverage of the task space but also pay attention to the boundary of the mesh.
For this, we first sample points uniformly in a fixed bounding box large
enough to cover any motion of the robot. We use a density of 0.25 points per
$\text{dm}^{3}$. Next we sample points on the surface of the swept volume,
using a density of 0.2 points per $\text{cm}^{2}$. For each such surface point
we additionally sample 2 points from a spherical Gaussian distribution
centered at this point (with $\sigma=3\,\text{mm}$). To further cover the
space inside the volume, we sample for each surface point a second surface
point at random, and define the point’s location by uniformly sampling from
the line connecting these points.

##  V Experiments

(a) ground truth

(b) 11x1024

(c) 11x512

(d) 5x512

Figure 3: Exemplary comparison of different model sizes for the linear joint
space motion already shown in Fig. 1. There is a noticeable degradation when
going from (c) 11x512 to (d) 5x512, while the difference between (c) 11x512
and (b) 11x1024 is less pronounced. This qualitative finding is also reflected
in the metrics shown in Table I.

Figure 4: Comparison between the ground truth swept volume meshes (gray) and
meshes reconstructed from the neural networks (red) via the marching cubes
algorithm (resolution: 1 cm). The upper row shows linear joint space motions,
while the lower row shows spline motions. For both rows, 11x1024 models were
used to predict the signed distances. The motions are the first four motions
of their respective test sets, and thus have not been used in training.

We validate our approach in three different experiments. First, we investigate
the accuracy of the signed distance prediction of the neural networks. We then
evaluate Alg. 1 in an artificial path planning scenario as well as a real-
world bin picking application.

If not otherwise stated, we use an KUKA Agilus KR 6 with 6 degrees of freedom.
Computations are done with an Nvidia GeForce RTX 3090 GPU. We use the Flexible
Collision Library (FCL) [21] as the GCC in all our experiments. For training
neural networks, we use the Adam optimizer of PyTorch with default parameters
lr = 0.001 and betas = (0.9, 0.999). We use the ReduceLROnPlateau scheduler
with factor 0.1 and patience 10 and train the model for 200 epochs. A search
over these parameters did not yield any improvements. For testing, we choose
the checkpoint with the best loss on the validation set. Before inference, we
export the PyTorch models to TorchScript programs. Source code and checkpoints
for the models, as well as the dataset LINEAR are available online [22].

###  V-A Implicit Swept Volume Modeling

In the first experiment, we test the accuracy of the signed distance
prediction on the two datasets LINEAR and SPLINE introduced in Section IV-C.
For each dataset we generated 20 000 motions and sampled approximately 4 500
points per motion. We use 60 % of the data for training, 25 % for validation,
and 15 % for testing. The mean absolute errors (MAE) on the test sets are
reported in Table I. We investigate three different network architectures
5x512, 11x512, and 11x1024, varying in their number of blocks $n_{\text{b}}$
(5 or 11) and their block dimension $n_{\text{dim}}$ (512 or 1024). The MAE of
the prediction for both linear and spline motions is below 4.5 mm in all
cases. For both datasets, we can improve the error to below 3.5 mm by using 11
instead of 5 blocks and approximately 3.1 mm by increasing the block dimension
from 512 to 1024. To further illustrate the prediction quality of the
networks, we used the marching cubes algorithm to reconstruct the swept volume
meshes from the networks (with a resolution of 1 cm). Fig. 3 shows a
comparison of different model sizes. More reconstructions for linear and
spline motions are shown in Fig. 4.

Mean Absolute Error (mm)
---
| LINEAR | SPLINE
5x512 |  4.38 $\pm$5.67 |  4.32 $\pm$5.55
11x512 |  3.45 $\pm$4.71 |  3.32 $\pm$4.49
11x1024 |  3.08 $\pm$4.27 |  3.11 $\pm$4.28
TABLE I:  Depicted are the mean absolute errors of the predicted signed
distances. Results are provided for different model architectures and the two
datasets.

We further report the inference times of our networks. Table II shows average
computation times for processing three different sizes of point batches. The
first two batch sizes represent the point cloud sizes from the box world and
bin picking experiments, that we detail later. The inference times
approximately represent the time necessary to check one motion for collision
in these scenarios. We add a batch size of 20 000 to observe how inference
times scale for larger inputs. We also trained a 16 bit precision variant of
the smallest model to further speed up inference times. As our unit of
measurement is millimeter, this has no effect on the accuracy of the model and
approximately halves the computation time for larger batch sizes. For the two
smaller batch sizes of the path planning experiments, we observe that
collision checks can be performed in under a millisecond.

Inferences times (ms)
---
| Box World | Bin Picking |
| 1000 Points | 3000 Points | 20 000 Points
11x1024 | 2.17 | 5.55 | 33.10
11x512 | 0.91 | 2.17 | 11.20
5x512 | 0.50 | 1.08 | 5.42
5x512 16bit | 0.49 | 0.70 | 2.63
TABLE II: Inference times for different models and input sizes. We report the
time necessary to processes a batch of 1000, 3000 or 20 000 points. The first
two rows represent the point cloud sizes used in the path planning experiments
Box World and Bin Picking. The standard deviation of the computation times is
in the range of micro seconds and therefore omitted.

###  V-B Box World Collision Checking

In the following experiment, we evaluate Alg. 1 for motion planning in
artificially generated box world scenes as seen in Fig. 5.

Figure 5: An example of a box world scene, with randomly sampled boxes and
points on the surface of the boxes.

Each of the 100 scenes contains 50 boxes with a random orientation and a
location sampled uniformly at random from within a volume defined by
$x,y\in[-1\,\text{m},\,1\,\text{m}]$ and $z\in[0\,\text{m},\,1.5\,\text{m}]$.
The side lengths of each box are independently sampled uniformly from the
interval $[1\,\text{cm},\,10\,\text{cm}]$ (the robot base is centered at the
origin). For each scene, we sample a single start configuration uniformly from
within the joint limits, and 100 random goal configurations, which yields 10
000 motions in total. On average, 25.2 % of the motions are collision-free.
The minimum number of collision-free motions in a scene is 0, the maximum is
66. For the neural network, we represent this environment as a point cloud by
sampling points from the surface of the boxes with a density of 0.1 points per
$\text{cm}^{2}$. In the box world, the GCC reports the signed distance of the
robot model to the boxes represented as triangle meshes. For this, the linear
joint space motion is discretized into 200 steps and each configuration is
checked. We apply the LINEAR 5x512 16 bit model to predict signed distances.
Using a safety margin of 5 mm to classify signed distances into collisions and
non-collisions, the network achieves a classification accuracy of 93.1 %, with
6.3 % false positives, and 0.6 % false negatives.

We compare two algorithms for finding the first collision-free path of each
scene. The first method (GCC0) applies the GCC on each motion in the given
order. The second method (GCC-NN) represents Alg. 1 using the neural network
as a proxy. As timing depends on the order in which the motions of each scene
are checked, we average over 1000 permutations for each scene, which yields
100 000 timing samples. The results shown in Table III indicate a speed-up of
49 % from 269 ms (GCC0) to 136 ms (GCC-NN) for the average time to find a
collision-free path. Due to the pre-checks done by the network, the average
number of exact collision checks done by the GCC is reduced to less than a
third, with 2.0 exact checks per scene in (GCC-NN).

Results Box World Experiment |
---|---
|  (GCC0) | (GCC-NN)
total time (ms) |  269 $\pm$286 |  136 $\pm$68
# Exact checks |  6.69 $\pm$11.98 |  2.00 $\pm$9.85
# NN checks |  |  14.05 $\pm$24.20
TABLE III: Results of the box world experiment. (GCC-NN) halves the total time
for collision checking of (GCC0) by using three times less exact collision
checks.

###  V-C Bin Picking Collision Checking

Finally, we investigate collision checking in a real-world bin picking
application. Fig. 6 depicts a KUKA Agilus KR 10 (with attached gripper)
grasping gearbox housings out of a bin.

Figure 6: Bin picking application using the KUKA.SmartBinPicking software. The
image on the left displays a test scenario with 40 gearbox housings. The right
image shows detected objects with collision-free endeffector paths (green) and
objects without collision-free paths (red).

We use the KUKA.SmartBinPicking software to produce multiple grasp candidates
for a given scene. A given grasp $\textbf{q}_{\text{grasp}}$ is reached by two
splines $\textbf{q}_{\text{start}}$ $\rightarrow$ $\textbf{q}_{\text{inVia}}$
and $\textbf{q}_{\text{inVia}}$ $\rightarrow$ $\textbf{q}_{\text{grasp}}$,
where $\textbf{q}_{\text{start}}$ is the starting configuration above the bin,
and $\textbf{q}_{\text{inVia}}$ is a via point located close to the target
object, which is introduced to increase the chance for collision-free paths.
Once the object has been grasped, it is moved out of the bin using two similar
splines: $\textbf{q}_{\text{grasp}}$ $\rightarrow$
$\textbf{q}_{\text{outVia}}$ and $\textbf{q}_{\text{outVia}}$ $\rightarrow$
$\textbf{q}_{\text{end}}$, where $\textbf{q}_{\text{end}}$ $\neq$
$\textbf{q}_{\text{start}}$ is the final configuration above the bin, and
$\textbf{q}_{\text{outVia}}$ $\neq$ $\textbf{q}_{\text{inVia}}$ is another via
point. We refer to the sequential execution of the first two splines as
$\textbf{m}_{\text{in}}$, and the other two splines as
$\textbf{m}_{\text{out}}$. The GCC checks each of the motions
$\textbf{m}_{\text{in}}$ and $\textbf{m}_{\text{out}}$ in one pass, while the
NN model is called twice (once for each spline). For evaluating our approach,
we collected 22 real-world scenes (see Fig. 6) with 15 to 40 gearbox housings
in the bin. Grasp sampling of the system yielded 35 $\pm$7 possible grasp
poses per scene where 82 $\pm$39 % of the motions $\textbf{m}_{\text{in}}$ to
grasp candidates and 81 $\pm$39 % of motions $\textbf{m}_{\text{out}}$ out of
the box resulted in collisions. A 5x512 16 bit model is trained for the new
robot with attached gripper using the same data generation process as
described in Section V-A. While the network was trained with the detailed
model of the robot, the GCC uses a convexified robot model for collision
checking to speed up collision checks.

Bin picking requires to precisely predict distances where the robot geometry
is close to the point cloud. This inherent difficulty provides an interesting
scenario, where the neural network is not able to detect collision-free paths
anymore for the given safety margin. Alg. 1 is therefore reduced to the second
loop: the minimum signed distances of all possible motions are predicted by
the network in advance, and all motions are then checked by the GCC in
descending order of their predicted signed distance.

As can be seen, (GCC-NN) can still improve the total time, if the quality of
the signed distance prediction is high enough to sort motions favorably. Table
IV shows that the number of collision checks is reduced by approximately 50 %
compared to (GCC0) for motions into and out of the bin. The average time for
checking a given motion is 28 $\pm 4$ ms for the GCC and 0.70 ms for the
network. Thereby (GCC-NN) achieves a speed-up of 25 % and 30 % in total time
consumption over the (GCC0). To produce the results, we again sampled 1000
permutations for the order in which the motions are checked.

|  $\textbf{m}_{\text{in}}$ (into bin) |  $\textbf{m}_{\text{out}}$ (out of bin)
---|---|---
|  (GCC0) | (GCC-NN) |  (GCC0) | (GCC-NN)
total time (ms) | 190 $\pm$215 | 143 $\pm$188 | 194 $\pm$223 | 135 $\pm$185
# Exact checks | 6.90 $\pm$8.30 | 3.38 $\pm$7.32 | 7.2 $\pm$8.7 | 3.25 $\pm$7.3
# NN checks |  | 70.5 $\pm$13.00 |  | 70.5 $\pm$13.0
TABLE IV: Results of the bin picking experiment: Performance of (GCC0) and
(GCC-NN) is shown for motions into the bin in columns 2 and 3 and motions out
of the bin in columns 4 and 5. It can be seen, that (GCC-NN) improves the
total time for collision checking by 25 % and 30 %, respectively, using less
exact collision checks than (GCC0).

##  VI Conclusion and Outlook

We proposed a novel approach for neural implicit swept volumes, parameterized
by the start and goal configuration. We further introduced an algorithm for
interleaving these implicit models with geometric collision checking. In our
experiments, we achieve accurate neural representations of swept volumes and
show, that we can significantly speed up a real-world bin picking application
with our algorithm.

## Acknowledgments

This work has been partially supported by the European Commission (EC) under
grant agreement ID 101091792 (SMARTHANDLE) and by the Bavarian Ministry of
Economic Affairs, Regional Development and Energy (StMWi) under contract
number DIK0374/01 (OPERA).

## References

  * [1] K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber, “A Survey on Learning-Based Robotic Grasping,” _Current Robotics Reports_ , vol. 1, pp. 239–249, 12 2020.
  * [2] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt Jr., “Grasp Pose Detection in Point Clouds,” _Int. Journal of Robotics Research (IJRR)_ , vol. 36, no. 13–14, pp. 1455–1473, 2017.
  * [3] J. Pan and D. Manocha, “Efficient Configuration Space Construction and Optimization for Motion Planning,” _Engineering_ , vol. 1, no. 1, pp. 46–57, Mar. 2015.
  * [4] D. Coleman, I. A. Șucan, S. Chitta, and N. Correll, “Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study,” _Journal of Software Engineering for Robotics (JOSER)_ , vol. 5, no. 1, pp. 3–16, Mar. 2014\.
  * [5] S. Sellán, N. Aigerman, and A. Jacobson, “Swept Volumes via Spacetime Numerical Continuation,” _ACM Trans. on Graphics_ , vol. 40, no. 4, pp. 1–11, 2021.
  * [6] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma, “Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [7] J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, “Equalization Loss for Long-Tailed Object Recognition,” in _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [8] X. Wang, L. Lian, Z. Miao, Z. Liu, and S. Yu, “Long-tailed Recognition by Routing Diverse Distribution-Aware Experts,” in _Proc. of the Int. Conf. on Learning Representations (ICLR)_ , 2021.
  * [9] H.-T. L. Chiang, A. Faust, S. Sugaya, and L. Tapia, “Fast Swept Volume Estimation with Deep Learning,” in _Algorithmic Foundations of Robotics XIII: Proc. of the 13th Workshop on the Algorithmic Foundations of Robotics (WAFR)_ , 2020, DOI: 10.1007/978-3-030-44051-0_4.
  * [10] J. Baxter, M. R. Yousefi, S. Sugaya, M. Morales, and L. Tapia, “Deep Prediction of Swept Volume Geometries: Robots and Resolutions,” in _Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2020, DOI: 10.1109/IROS45743.2020.9341396.
  * [11] M.-H. Lee and J.-S. Liu, “Single Swept Volume Reconstruction by Signed Distance Function Learning: A feasibility study based on implicit geometric regularization,” _IFAC-PapersOnLine_ , vol. 55, no. 15, pp. 142–147, 2022, Conf. on Intelligent Control and Automation Sciences (ICONS).
  * [12] A. Gropp, L. Yariv, N. Haim, M. Atzmon, and Y. Lipman, “Implicit Geometric Regularization for Learning Shapes,” in _Proc. of the Int. Conf. on Machine Learning (ICML)_ , 2020, DOI: 10.5555/3524938.3525293.
  * [13] J. B. Michaux, Y. Kwon, Q. Chen, and R. Vasudevan, “Reachability-based Trajectory Design with Neural Implicit Safety Constraints,” in _Proc. of Robotics: Science and Systems (RSS)_ , 2023, DOI: 10.15607/RSS.2023.XIX.062.
  * [14] P. Liu, K. Zhang, D. Tateo, S. Jauhri, J. Peters, and G. Chalvatzaki, “Regularized Deep Signed Distance Fields for Reactive Motion Generation,” in _Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2022, DOI: 10.1109/IROS47612.2022.9981456.
  * [15] M. Koptev, N. Figueroa, and A. Billard, “Neural Joint Space Implicit Signed Distance Functions for Reactive Robot Manipulator Control,” _IEEE Robotics and Automation Letters_ , vol. 8, no. 2, pp. 480–487, 2023\.
  * [16] Y. Li, Y. Zhang, A. Razmjoo, and S. Calinon, “Learning Robot Geometry as Distance Fields: Applications to Whole-body Manipulation,” 2023, arXiv preprint, DOI: 10.48550/arXiv.2307.00533.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778, DOI: 10.1109/CVPR.2016.90.
  * [18] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation,” in _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019, DOI: 10.1109/CVPR.2019.00025.
  * [19] S. Sellán, O. Stein, _et al._ , “Gpytoolbox: A python geometry processing toolbox,” 2023, https://gpytoolbox.org.
  * [20] A. Jacobson, D. Panozzo, _et al._ , “libigl: A simple C++ geometry processing library,” 2023, https://libigl.github.io.
  * [21] J. Pan, S. Chitta, and D. Manocha, “FCL: A General Purpose Library for Collision and Proximity Queries,” in _Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2012, pp. 3859–3866, DOI: 10.1109/ICRA.2012.6225337.
  * [22] D. Joho, J. Schwinn, and K. Safronov, “Data, Models, and Code for: Neural Implicit Swept Volume Models for Fast Collision Detection,” Dataset on zenodo.org, 2024, DOI: 10.5281/zenodo.10638607.


# Wyrm, A Pythonic Toolbox for Brain-Computer Interfacing

Bastian Venthur∗†, Benjamin Blankertz† * Corresponding author:
<EMAIL_ADDRESS>Berlin Institute of TechnologyCopyright © 2014
Bastian Venthur et al. This is an open-access article distributed under the
terms of the Creative Commons Attribution License, which permits unrestricted
use, distribution, and reproduction in any medium, provided the original
author and source are credited. http://creativecommons.org/licenses/by/3.0/

###### Abstract

A Brain-Computer Interface (BCI) is a system that measures central nervous
system activity and translates the recorded data into an output suitable for a
computer to use as an input signal. Such a BCI system consists of three parts,
the signal acquisition, the signal processing and the feedback/stimulus
presentation.

In this paper we present Wyrm, a signal processing toolbox for BCI in Python.
Wyrm is applicable to a broad range of neuroscientific problems and capable
for running online experiments in real time and off-line data analysis and
visualisation.

###### Index Terms:

Brain-Computer Interfacing, BCI, Toolbox, Python

##  1 Introduction

In the last years Python has gained more and more traction in the scientific
community. Projects like Numpy [Numpy], SciPy [SciPy], and Matplotlib
[Matplotlib] have created a strong foundation for scientific computing in
Python and machine learning packages like Scikit-learn [Scikit-learn] or
packages for data analysis like Pandas [Pandas] are building on top of it.
Although in recent years Python toolboxes like SCoT for EEG source
connectivity [Billinger], or MNE-Python for MEG and EEG data analysis
[Gramfort] were published, Matlab seems still to be the dominant programming
language in the brain-computer interface (BCI) community.

A BCI is a system that measures central nervous system activity and translates
the recorded data into an output suitable for a computer to use as an input
signal. Or slightly less abstract: A BCI is a communication channel that
allows for direct control of a computer by the power of thoughts.

A BCI system consists of three parts (Figure 1): a _signal acquisition_ part
that is connected to the measuring hardware (e.g. EEG) and provides the raw
data to the rest of the BCI system. The _signal processing_ part receives the
data from the signal acquisition and translates the data into the intent. The
feedback/stimulus presentation part translates the intent into an action.

Figure 1: Overview of an online BCI system.

In this paper we present Wyrm, an open source BCI toolbox in Python. Wyrm
corresponds to the signal processing part of the aforementioned BCI system.
Wyrm is applicable to a wide range of neuroscientific problems. It can be used
as a toolbox for analysis and visualization of neurophysiological data (e.g.
EEG, ECoG, fMRI, or NIRS) and it is suitable for real-time online experiments.
In Wyrm we implemented dozens of methods, covering a broad range of aspects
for off-line analysis and online experiments.

In the following sections we will explain Wyrm’s fundamental data structure,
its design principles and give an overview of the available methods of the
toolbox. We’ll show you where to find the documentation as well as some
extensive examples.

##  2 Design

All methods in the toolbox revolve around a data structure that is used
throughout the toolbox to store various kinds of data. The data structure
dubbed Data, is an object containing an n-dimensional numpy array that
represents the actual data to be stored and some meta data. The meta data
describes for each dimension of the numpy array, the name of the dimension,
the names of the single columns and the unit of the data.

Let’s assume we have a record of previously recorded EEG data. The data was
recorded with 20 channels and consists of 30 samples. The data itself can be
represented as a 2-dimensional array with the shape (30, 20). The names for
the dimensions are ’time’ and ’channels’, the units are ’ms’ and ’#’ (we use
’#’ as a pseudo unit for enumeration of things), and the description of the
columns would be two arrays: one array of length 30 containing the time stamps
for each sample and another array of length 20 containing the channel names.
This data structure can hold all kinds of BCI related data: continuous data,
epoched data, spectrograms, feature vectors, etc.

We purposely kept the meta data at a minimum, as each operation that modifies
the data also has to check if the meta data is still consistent. While it
might be desirable to have more meta data, this would also lead to more
housekeeping code which makes the code less readable and more error prone. The
data structure, however, can be extended as needed by adding new attributes
dynamically at runtime. All toolbox methods are written in a way that they
ignore unknown attributes and never throw them away.

We tried very hard to to keep the syntax and semantics of the toolbox methods
consistent. Each method obeys a small set of rules: (1) Methods never modify
their input arguments. This allows for a functional style of programming which
is in our opinion well suited when diving into the data. (2) A Method never
modifies attributes of Data objects which are unrelated to the functioning of
that method. Especially does it never remove additional or unknown attributes.
(3) If a method operates on a specific axis of a Data object, it by default
obeys a convention about the default ordering of the axis but allows for
changing the index of the axis by means of a default arguments.

##  3 Toolbox Methods

The toolbox contains a few data structures (Data, RingBuffer and BlockBuffer),
I/O-methods for loading and storing data in foreign formats and off course the
actual toolbox algorithms. The list of algorithms includes: channel selection,
IIR filters, sub-sampling, spectrograms, spectra, baseline removal for signal
processing; Common Spatial Patterns (CSP) [Ramoser], Source Power Co-
modulation (SPoC) [Dähne], classwise average, jumping means, signed
$r^{2}$-values for feature extraction; Linear Discriminant Analyis (LDA) with
and without shrinkage for machine learning [Blankertz], and many more. It is
worth mentioning that with scikit-learn you have a wide range of machine
learning algorithms readily at your disposal. Our data format is very
compatible with scikit-learn and one can usually apply the algorithms without
any data conversion step at all.

The toolbox also includes plotting facilities that make it easy to quickly
generate useful plots out of neurophysiological data. Those methods include
scalp plots (Figure 2), time courses (Figure 3), signed $r^{2}$ plots (Figure
4), and more.

Figure 2: Example scalp plots. The plots show the spatial topography of the
average voltage distribution for different time intervals and channels.

Figure 3: Example time course plots for three selected channels.

Figure 4: Example signed $r^{2}$-plots. The channels are sorted from frontal
to occipital and within each row from left to right.

##  4 Unit Testing

Since the correctness of its methods is crucial for a toolbox, we used unit
testing to ensure all methods work as intended. In our toolbox _each_ method
is tested respectively by at least a handful of test cases to ensure that the
methods calculate the correct results, throw the expected errors if necessary,
etc. The total amount of code for all tests is roughly 2-3 times bigger than
the amount code for the toolbox methods.

##  5 Documentation

As a software toolbox would be hard to use without proper documentation, we
provide documentation that consists of readable prose and extensive API
documentation (http://venthur.github.io/wyrm/). Each method of the toolbox is
thoroughly documented and has usually a short summary, a detailed description
of the algorithm, a list of expected inputs, return values and exceptions, as
well as cross references to related methods in- or outside the toolbox and
example code to demonstrate how to use the method.

##  6 Examples

To show how to use the toolbox realistic scenarios we provide two off-line
analysis scripts, where we demonstrate how to use the toolbox to complete two
tasks from the BCI Competition III [BCIComp3]. The first example uses
Electrocorticography (ECoG) recordings provided by the Eberhard-Karls-
Universität Tübingen. The time series where picked up by a 8x8 ECoG platinum
grid which was placed on the contralateral, right motor cortex.

During the experiment the subject had to perform imagined movements of either
the left small finger or the tongue. Each trial consisted of either an
imagined finger- or tongue movement and was recorded for a duration of 3
seconds. The recordings in the data set start at 0.5 seconds after the visual
cue had ended to avoid visual evoked potentials, being reflected by the data.
It is worth noting that the training- and test data were recorded on the same
subject but with roughly one week between both recordings. The data set
consists of 278 trials of training data and 100 trials of test data. During
the BCI Competition only the labels (finger or tongue movement) for the
training data were available. The task for the competition was to use the
training data and its labels to predict the 100 labels of the test data. Since
the competition is over, we also had the true labels for the test data, so we
could calculate and compare the accuracy of our results. For this experiment
our classification accuracy was 92% which is comparable with the winners of
the competition whose accuracy was: 91%, 87%, and 86%.

The second data set uses Electroencephalography (EEG) recordings, provided by
the Wadsworth Center, NYS Department of Health, USA. The data were acquired
using BCI2000’s Matrix Speller paradigm, originally described in [Donchin].
The subject had to focus on one out of 36 different characters, arranged in a
6x6 matrix. The rows and columns were successively and randomly intensified.
Two out of 12 intensifications contained the desired character (i.e., one row
and one column). The event-related potential (ERP) components evoked by these
target stimuli are different from those ERPs evoked by stimuli that did not
contain the desired character. The ERPs are composed of a combination of
visual and cognitive components. The subject’s task was to focus her/his
attention on characters (i.e. one at a time) in a word that was prescribed by
the investigator. For each character of the word, the 12 intensifications were
repeated 15 times before moving on to the next character. Any specific row or
column was intensified 15 times per character and there were in total 180
intensifications per character. The data was recorded using 64 channel EEG.
The 64 channels covered the whole scalp of the subject and were aligned
according to the 10-20 system. The collected signals were bandpass filtered
from 0.1-60Hz and digitized at 240Hz. The data set consists of a training set
of 85 characters and a test set of 100 characters for each of the two
subjects. For the trainings sets the labels of the characters were available.
The task for this data set was to predict the labels of the test sets using
the training sets and the labels. In this experiment we reached a
classification accuracy for single letters of 93,5%, the winners of the
competition reached 96,5%, 90,5%, and 90%.

We also provide an example online experiment where we use the ERP data set
with an pseudo amplifier that feeds the data in real-time to the toolbox, to
show how to do the classification task in an online setting.

The data sets from the competition are freely available and one can reproduce
our results using the scripts and the data.

##  7 Python 2 vs Python 3

The ongoing transition from Python 2 to Python 3 was also considered and we
decided to support _both_ Python versions. Wyrm is mainly developed under
Python 2.7, but written in a _forward compatible_ way to support Python 3 as
well. Our unit tests ensure that the methods provide the expected results in
Python 2 and Python 3.

##  8 Summary and Conclusion

In this paper we presented Wyrm, a free and open source BCI toolbox in Python.
We introduced Wyrm’s main data structure and explained the design ideas behind
the current implementation. We gave a short overview of the existing methods
in the toolbox and showed how we utilized unit testing to make sure the
toolbox works as specified, where to find the extensive documentation and some
detailed examples.

Together with Mushu [Mushu] our signal acquisition library and Pyff [Pyff] our
Framework for Feedback and Stimulus Presentation, Wyrm adds the final piece to
our ongoing effort to provide a complete, free and open source BCI system in
Python.

Wyrm is available under the terms of the MIT license, its repository can be
found at http://github.com/venthur/wyrm.

##  9 Acknowledgements

This work was supported in part by grants of the BMBF: 01GQ0850 and 16SV5839.
The research leading to this results has received funding from the European
Union Seventh Framework Programme (FP7/2007-2013) under grant agreements
611570 and 609593.

## References

  * [Blankertz] Blankertz B, Lemm S, Treder MS, Haufe S, Müller KR (2011) _Single-trial analysis and classification of ERP components – a tutorial_. NeuroImage 56:814– 825, http://dx.doi.org/10.1016/j.neuroimage.2010.06.048
  * [Dähne] Dähne S, Meinecke FC, Haufe S, Höhne J, Tangermann M, Müller KR, Nikulin VV (2014) _SPoC: a novel framework for relating the amplitude of neuronal oscillations to behaviorally relevant parameters_. NeuroImage 86(0):111–122, http:://dx.doi.org/10.1016/j.neuroimage.2013.07.079
  * [Mushu] Bastian Venthur and Benjamin Blankertz. _Mushu, a Free and Open Source BCI Signal Acquisition, Written in Python_. Engineering in Medicine and Biology Society (EMBC). doi: http://dx.doi.org/10.1109/EMBC.2012.6346296 San Diego, 2012.
  * [Pyff] Bastian Venthur, Simon Scholler, John Williamson, Sven Dähne, Matthias S Treder, Maria T Kramarek, Klaus-Robert Müller and Benjamin Blankertz. _Pyff—A Pythonic Framework for Feedback Applications and Stimulus Presentation in Neuroscience_. Frontiers in Neuroscience. 2010\. http://dx.doi.org/10.3389/fnins.2010.00179.
  * [Ramoser] Ramoser H, Muller-Gerking J, Pfurtscheller G (2000) _Optimal spatial filtering of single trial eeg during imagined hand movement_. Rehabilitation Engineering, IEEE Transactions on 8(4):441–446
  * [Donchin] E. Donchin, K. Spencer, and R. Wijesinghe. _The mental prosthesis: assessing the speed of a p300-based brain-computer interface._ Rehabilitation Engineering, IEEE Transactions on, 8(2):174–179, Jun 2000\.
  * [Gramfort] Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., … & Hämäläinen, M. (2013). _MEG and EEG data analysis with MNE-Python_. Frontiers in neuroscience, 7.
  * [Billinger] Billinger, M., Brunner, C., & Müller-Putz, G. R. (2014). _SCoT: a Python toolbox for EEG source connectivity_. Frontiers in neuroinformatics, 8. ISO 690
  * [Numpy] http://numpy.org
  * [SciPy] http://scipy.org
  * [Matplotlib] http://matplotlib.org
  * [Scikit-learn] http://scikit-learn.org
  * [Pandas] http://pandas.pydata.org
  * [BCIComp3] https://www.bbci.de/competition/iii/