# HIPer: A Human-Inspired Scene Perception Model
for Multifunctional Mobile Robots

Florenz Graf1, Jochen Lindermayr1, Birgit Graf1, Werner Kraus1, and Marco F.
Huber2 1Department Robot and Assistive Systems, Fraunhofer IPA, 70569
Stuttgart, Germany. <first name>.<last name> @ipa.fraunhofer.de2Department
Cyber Cognitive Intelligence (CCI), Fraunhofer IPA, and Institute of
Industrial Manufacturing and Management IFF, University of Stuttgart, 70569
Stuttgart, Germany<EMAIL_ADDRESS>
###### Abstract

Taking over arbitrary tasks like humans do with a mobile service robot in
open-world settings requires a holistic scene perception for decision-making
and high-level control. This paper presents a human-inspired scene perception
model to minimize the gap between human and robotic capabilities. The approach
takes over fundamental neuroscience concepts, such as a triplet perception
split into recognition, knowledge representation, and knowledge
interpretation. A recognition system splits the background and foreground to
integrate exchangeable image-based object detectors and SLAM, a multi-layer
knowledge base represents scene information in a hierarchical structure and
offers interfaces for high-level control, and knowledge interpretation methods
deploy spatio-temporal scene analysis and perceptual learning for self-
adjustment. A single-setting ablation study is used to evaluate the impact of
each component on the overall performance for a fetch-and-carry scenario in
two simulated and one real-world environment.

###### Index Terms:

Biologically-Inspired Robots, Semantic Scene Understanding, Service robots,
Holistic Scene Perception, Long-Term Benchmarking

††publicationid: pubid: © 2024 IEEE

##  I Introduction

The vision of a robot that performs daily tasks, such as tidying up, handling
dishes, preparing food, and fetching and carrying arbitrary objects, is far
away. While these tasks are straightforward for humans, current robots lack
the necessary capabilities. One fundamental pillar in open-world settings is
scene perception since it allows robots to _understand_ and _represent_ the
world [1]. Scene perception involves recognizing the environment through
semantic knowledge, i.e., perceiving various objects in their spatial and
temporal context, which needs continual updating throughout the robot’s
operational lifespan. Consequently, it forms the knowledge base for cross-task
decision-making and high-level control to obtain the required intelligent
behavior at high autonomy.

Supporting people is the objective of service robots. Available products for
open-world operation take over a distinct service, such as transportation,
entertainment, or cleaning [2]. They excel in their designated tasks as their
perception is specifically optimized for a particular application rather than
a holistic solution. Especially when addressing physical tasks with robots,
expensive hardware must be compensated. One trend is to take over multiple
services with a single robot (cp. [2]). Nevertheless, as such multi-functional
mobile robots assume control over various tasks, the associated task-specific
demands multiply, necessitating a comprehensive, modular, and adaptable
solution. A significant hurdle for these versatile robots lies in overcoming
the seamless integration of components to attain a holistic scene perception,
all the while accommodating the demands of mobile systems—requiring swift
setup [3], scalability, and reusability [4], real-time reactivity [1], and the
capability to flexibly adapt to changes in the environment. In this context,
holistic scene perception refers to the ability of the robot to perceive its
scene in terms of recognizing and arranging objects and surfaces in a
meaningful way to obtain a high-level understanding [5]. A major part of the
understanding is obtained through scene knowledge interpretation, i.e., an
individual capability to interpret semantics in its context (cp. [1, 6, 5]).

Recognizing the limited research for such holistic scene perception solutions,
we propose a biologically-inspired approach. Since humans possess
comprehensive perceptual capabilities that allow them to manage multiple
arbitrary tasks effortlessly, we hypothesize that merging neuroscience
concepts of human perception with robotic techniques could bridge the gap
between human and robot perception and simplify their collaboration as a side
effect. Thus, we developed Human-Inspired Scene Perception (HIPer)-Model (cp.
Fig. 1) to empower mobile robots through a human-like problem understanding
for taking over arbitrary tasks within the open world.

Figure 1: The HIPer-Model provides a holistic scene perception, empowering
mobile robots to take over diverse tasks in open-world settings.

The HIPer-Model builds upon our recently published paper [6] that transfers
the latest studies and theories of the human perceptual process to mobile
robots. The transfer points out numerous similarities between theories of
human perception that date back many years and comparatively recent research
on robot perception. It highlights the limitations of today’s robotic
perception while showing how humans manage to obtain a holistic perception
required for arbitrary tasks. Humans learn an individual scene perception,
relying foremost on visual information, throughout their lives. However,
initial perception skills are essential for rapid deployment.

This paper considers these identified challenges by providing three
substantial contributions:

  * •

A comprehensive analysis of the transferability of human perception concepts
to mobile robots for designing scene perception models while considering the
limitations and potentials of robots.

  * •

The HIPer-Model, a technical solution that merges neuroscience concepts with
robot perception to bridge the two most significant shortcomings: providing a
holistic scene understanding for cross-task applications and semantic scene
knowledge interpretation using long-term scene analysis.

  * •

Introduction of a novel ablation-study-based benchmark for robotic scene
models to fill the gap in research for analyzing their long-term performance.
It is deployed in real and simulated environments to demonstrate the
perceptual and task performance of the HIPer-Model.

The subsequent sections of this paper follow this structure: Section II
presents relevant literature on specific scene perception techniques and their
integrated solutions for mobile robots. Section III formulates the problem and
introduces how the HIPer-Model aims to solve it by presenting its concept and
a system overview. Sections IV to VI elaborate on the methodology of each
component, which is divided into information recognition, knowledge
representation, and interpretation—the three major parts of the HIPer-Model.
Section VII details experiments, including a single-setting ablation study in
one real and two simulated environments. Finally, this paper concludes with
discussions in Section VIII and explores possible future research directions
in Section IX.

##  II Related Work

Research in robot scene perception encompasses diverse fields that aim to
empower the robot for specific tasks through sensor data processing. The
primary interest lies in visual perception techniques, which build upon
Computer Vision methods. They reach many years into the past and have been
adapted to address specific challenges in the context of mobile robots, such
as visual Simultaneous Localization and Mapping (SLAM), object recognition,
semantic segmentation and Multi-Object Tracking (MOT). Recently, research
started merging several techniques to achieve a more comprehensive
understanding of the scene, which is particularly important for the open-world
context.

#### SLAM

It is a vast research field in robotics for metric scene reconstruction [7].
The robot acquires a spatial environment map through sensor data registration
while simultaneously localizing itself to this map [8]. Traditional approaches
use 2D-LiDAR data for scan-matching to generate a 2D map, while newer visual
SLAM utilizes RGB-D or stereo cameras to generate a 3D map [9]. The latest
approaches rely on multiple sensors for precise localization. For instance,
they use dead reckoning with IMU, ICP based on LiDAR, wheel odometry, and
visual odometry to minimize ego-motion drift [10]. Localization techniques
include Bayes Filter-based algorithms (Particle Filter, Extended Kalman
Filter) and optimization algorithms like graphs or bundle adjustment. In
contrast to so-called _full-SLAM_ , _online SLAM_ solely recovers the recent
poses, allowing a constant memory allocation [10, 11]. Moreover, modern
approaches feature multi-robot mapping [12], WIFI signal mapping [10], or
long-term map updates [13], which are crucial for dynamic environments.
However, since SLAM is restricted to spatial data, supplementary recognition
techniques for clustering regions and assigning the semantics are necessary
for high-level scene understanding [14].

#### Object recognition

Mobile robots benefit significantly from two key aspects of visual
recognition: object detection and semantic segmentation. These fields both
involve the analysis of images or point clouds at a specific moment, but they
differ in the representation of their output.

The prevailing approach in image-based object detection relies on Deep
Learning (DL) models like Convolutional Neural Networks (CNNs), such as YOLO
[15], InternImage [16], or  Vision Transformers (ViTs), such as SWIN [17],
ConvNet [18], and DETR [19]. When trained on extensive datasets, these methods
achieve an impressive mean average precision (mAP) of nearly
$65\text{\,}\mathrm{\char 37\relax}$. While CNN-based detectors offer a robust
initial estimation, their deployment on robots demands additional techniques
to enhance reliability. Specialized networks also exist for tasks like human
skeleton detection, providing joint information [20, 21], or (re)detecting
specific places [22]. Unsupervised approaches, such as zero-shot detectors
like CLIP [23], have gained popularity for their ease of deployment across
diverse objects. However, they cannot localize objects, making them only
conditionally applicable for mobile robots. Although object detectors localize
instances within a bounding box, their additional techniques become essential
for effectively matching the corresponding 3D data. Vice versa, detectors
based on 3D data face challenges in real-time performance and large-scale
datasets due to data volume and availability. As a result, such approaches
have yet to be popular on mobile robots.

A practical solution to address the computational demands involves mapping 2D
detections into 3D space through semantic segmentation. This method,
thoroughly explored in studies [24, 1], assigns semantic labels to each pixel
in an image, offering information on object boundaries that can be projected
into 3D space. Instance segmentation, a subset of semantic segmentation,
provides distinct object masks, with methods like Mask R-CNN [25] being
popular. Semantic segmentation proves beneficial for mobile robots where tasks
demand precise object localization and tracking. However, generating datasets
for segmentation is more time-intensive than bounding-box-based approaches,
requiring pixel-wise labeling. Consequently, the available pool of datasets
for semantic segmentation is smaller, an important factor when dealing with
less common objects [24, 26, 27].

####  MOT

It involves converting time-discrete object detections into continuous tracks,
a crucial aspect for monitoring objects over time. Typical MOT integrates
Kalman Filters or Particle Filters with 2D or 3D object detections that can be
split into generative, discriminative, and hybrid approaches [28]. Generative
trackers model the entire object appearance, generating an appearance model
and matching it to subsequent frames based on metrics like Intersection over
Union (IOU). Discriminative trackers learn object-background boundaries using
CNN, ViT, or LSTM [29, 30, 31, 32]. Equivalent to object recognition, 3D data
are preferred over 2D for mobile robots. Popular approaches use a Hungarian
algorithm to process detections before Kalman Filters track objects in 3D
space [33, 34, 35]. Despite addressing ego-motion, sensor and Field of View
(FOV) limitations impact object perception, necessitating consideration of
uncertainties, occlusions, and re-appearances.

#### Integrated Approaches

Combining multiple techniques within a unified framework becomes crucial for a
comprehensive scene perception, allowing the recognition of distinct semantics
from various places and times. Semantic mapping, also known as metric-semantic
reconstruction, has become an upcoming trend in this context, which merges
recognition with SLAM [36]. In particular, it enriches spatial environment
data from SLAM with semantic scene information, such as generated from
semantic segmentation [37] to address the challenge of achieving an abstract
and descriptive representation of the entire scene [14]. Examples are provided
in: [38, 39, 40]. A notable application, as demonstrated by Langer et al.
[41], involves clustering objects on planes to organize common tasks.
Additionally, extracting scene backgrounds by planes contributes to
understanding the environmental layout as demonstrated in [42, 43, 44].

Semantic mapping has a drawback in its failure to consider object dynamics, as
the generated map merely reflects the state of the initial reconstruction.
Given the presence of expected moving objects, addressing motion becomes
crucial. Recent advancements offer innovative strategies for managing various
object states, including static, moving, removed, and new ones. These
strategies facilitate instance-specific handling, applicable in diverse
scenarios. For example, temporal observation of people’s skeletons enables
reasoning about actions involving objects [45]. The complexity escalates due
to the incorporation of pose optimization for robot localization combined with
scene dynamics. To address this, scene graphs have been introduced to anchor
objects and humans in hierarchical topological structures [42, 36, 46]. For
instance, the _Kimera_ framework [36] employs scene graphs for multi-level
instance-specific reconstruction. This approach allows novel strategies, such
as filtering people during mapping, proving particularly beneficial in crowded
areas.

A more intricate approach to handling the acquired scene knowledge is
_Knowrob_ [47], a framework for knowledge representation and reasoning.
_Knowrob_ can describe a complex scene by linking different entities through
elaborate ontologies and managing them with various memory concepts, such as
episodic or semantic memory. In this context, open-vocabulary mapping has
emerged as an important research field, which holds significant relevance for
open-world scene understanding since it is not confined to a predefined set of
semantic classes. Recent approaches, such as those presented in [48, 49, 50],
incorporate open-vocabulary properties into scene mapping and its
representation. This integration helps to accommodate a broader range of
vocabularies and environmental complexities. Such contributions pave the way
for more robust and adaptable scene representations, facilitating improved
navigation and understanding within diverse environments.

An alternative strategy to handle open-world scenes is to employ learning
strategies. Unlike approaches that train their perception before real-world
deployment, there is scarce research on perceptual lifelong learning. Examples
in the early stages are presented in [51, 52], aiming to adapt the perception
during acting in the real world. However, the considerable potential of this
research field has yet to be integrated.

##  III Concept and System Overview

The considered problem of scene perception can be formulated mathematically as

$\displaystyle X=\sum_{t=0}^{\infty}x_{t}\quad\xrightarrow{\text{Scene
Model}}\quad Y=\sum_{s=0}^{k}y_{s}~{},$ (1)

where a time-continuous raw sensor input $X$ has to be mapped from the scene
model to time-discrete understandable (in terms of semantics) scene
information $Y$. The sensor input $X$ consists of continuously captured scene
information $x$ from one or multiple sensors at a given time $t$. The high-
level control system obtains specific scene information $y_{s}$, which is
provided by demand $k$ based on their needs. The scene perception of the
robot, i.e., the scene model, is responsible for handling the transformation
from $X$ to $Y$, a prerequisite to operating and taking over multiple and
complex tasks.

Although robotic research addresses this complex issue with integrated
approaches, combining multiple sub-components working seamlessly, they do not
empower mobile robots with sophisticated scene knowledge to take over
arbitrary tasks in open-world settings as humans do. When studying
neuroscience concepts and theories of human perception and comparing them with
robotic approaches, as presented in our previous work, a notable deficiency
refers to the scope of perception, i.e., holistic solutions that cover the
entire perception process similar to humans [6]. Moreover, related approaches
are not designed to extend or replace components, making deploying them to a
domain different from the original tasks challenging. One reason is that they
do not follow a standardized structure since their architecture is optimized
for individual tasks.

The HIPer-Model addresses these limitations by merging human perception
concepts with the latest robotics approaches. Although human perception is an
interconnected formation of many pieces, the three-part division of perception
which is presented in common neuroscience theories [53, Chapter 3] allows a
clear structure: (I) recognition makes the sensory input understandable, (II)
different memory types store and represent knowledge in multiple layers, and
(III) interpretation analyzes available knowledge by space, time, and
relation. While related approaches concentrate foremost on sensor-close
recognition tasks, as it is the origin of knowledge generation (cp. [54]), the
HIPer-Model aims to provide a holistic solution of the entire perception
process to transform the input $X$ to the output $Y$ by combining both
domains’ advantages. In particular, its concept mimics neuroscience’s three-
part division, offering a clear description of functionalities, as depicted by
its design in Fig. 2.

Recognition of Information by making sensor data understandableRepresenting
Knowledge in multiple layersInterpreting Knowledge by space, time, and
relationx0x1x2t0t1t2i0i1i2in$X$o0o1o2oio0o1o2ojo0o1o2oka0a1a2aq$Y$ Figure 2:
HIPer model design inspired by human perception consisting of a triplet split
of observations $o$, based on sensory input $X$, its aggregation into
instances $i$, and spatio-temporal analyses $a$. The obtained scene knowledge
$Y$ is accessible for decision-making and planning.

The recognition system aims to make simultaneously captured sensor data
understandable by obtaining scene observations $o$. The HIPer-Model combines
visual recognition techniques to extract understandable pieces of environment
information, denoted as observations $o$, based on visual sensor data at time
$t$. In particular, the approach processes the foreground and background
differently to leverage their characteristics. The split imitates human’s
preattentive and postattentive processing [55], i.e., the background,
characterized through large data with little richness of information, is
extracted during the initial reconstruction, allowing for high attention on
particular instances in the foreground. The accumulation of all observations
$O$ describes the union of foreground and background observations $O_{b}(t)$,
and $O_{f}(t)$, respectively, recognized at time $t$. It is denoted as

$O=\bigcup_{t=0}^{\infty}(O_{b}(t)\cup O_{f}(t))\quad\text{.}$ (2)

Both terms describe a set of $b$, respectively $f$ observations, according to

$O_{b}(t)=\bigcup_{b}o_{b}\quad\text{and}\quad
O_{f}(t)=\bigcup_{f}o_{f}\quad\text{.}$ (3)

Multiple corresponding scene observations are fused by time and space to a so-
called instance $i$ according to

$I=\bigcup_{j=0}^{n}i_{j}\text{ with }i_{j}=\bigcup_{l}o_{l}\quad\text{.}$ (4)

The concept of foreground or background instances is equivalent. They are
generated from matched observations $o_{f}$. While the background instances
consider only the robot poses, the foreground instances are generated through
a MOT pipeline to consider the additional dynamics of the particular instance.

The obtained scene information, i.e., the sum of observations $O$ and
instances $I$, are organized and stored in a knowledge base. While
observations are anchored to the sensor origin, instances are transformed into
the global map frame to obtain a virtual representation of the world. This
procedure allows post-processing instances after pose optimizations by
recalculating the merging of each instance. Furthermore, the knowledge base
$K$ manages scene analyses $A$, resulting in

$K=O\bigcup I\bigcup A\quad\text{.}$ (5)

Knowledge interpretation builds upon the preceding steps—knowledge acquisition
being a prerequisite—while it is only marginally addressed in related work.
The HIPer-Model incorporates this crucial element through the implementation
of a union $p$ of spatio-temporal scene analysis techniques $a_{m}$, denoted
as $A$ according to

$A=\bigcup_{m=0}^{p}a_{m}\quad\text{.}$ (6)

While robots use pre-trained recognition on extensive datasets for quick
deployment, they lack adaptability when deployed to a random environment. The
analysis techniques aim to observe the particular environment over a longer
period to improve scene understanding and reshape pre-trained recognition
capabilities during long-term operations.

The proposed concept has been realized through a modular architecture,
facilitating the seamless interchangeability of specific components.
Subsequent sections provide details of each component, as illustrated in the
schematic overview in Fig. 3.

Sensory Input $X$Recognition of InformationKnowledge BaseKnowledge
InterpretationScene Information YRGB-D LiDAR OdometryObject
DetectorRGBSLAMAllForeground SegmentationBackground
SegmentationPointsPointsMatchDetection 2DBlob
3DMOTMergeStructurePose$O_{f}$ForegroundBackground$O_{b}$, $I$$O_{f}$,
$I$Env.Spatial MapMap 2D&3DService InterfaceHeatmap Generator…Feature
LearningLayer 1Layer 2Layer 3Layer 4$a$$a$$a$

Figure 3: Schematic overview of the HIPer model with its newly developed
(blue) and reused (gray) components. A triplet split separates recognition,
consisting of a background and foreground pipeline to obtain observations $o$
and instances $i$, representation of this scene knowledge in multiple layers,
and its interpretation for long-term scene analyses $a$.

##  IV Recognition of Information

Using deep learning models became the standard in the open-world context as
presented by the related approaches. However, their usage is challenging for
the addressed perception of arbitrary objects since they cannot be easily
split, merged, and reused for other than the trained purpose. When comparing
DL-based object detectors and semantic segmentation, the last would be more
challenging when facing a new task due to their lower availability and smaller
number of object classes.

The HIPer-Model provides a hybrid approach of multiple modules, combining DL-
based and traditional methods. The recognition leverages its performance by
reusing a 3D-SLAM and an object detector, two comprehensive research areas
with a wide selection of methods for many applications and domains. The
background reconstruction uses the SLAM’s output to further process the
registered points with a semantic segmentation based on traditional techniques
to obtain primitive structures. This procedure allows for precise
reconstruction of static regions of the scene while requiring little
computation power. The foreground recognition comprises a segmentation
component that uses the background as a preattentive filtering, the detector
that delivers object proposals, a matching to obtain 3D observations $O_{f}$
based on the IOU, and finally, a MOT for its aggregation into instances
$i_{b}$.

###  IV-A Sensory Input

Humans employ various senses to obtain environmental information, allowing for
a more comprehensive scene perception since deficiencies in the different
principles can be compensated. Due to the richness of information, humans rely
80-$85\text{\,}\mathrm{\char 37\relax}$ on visual recognition [56]. However,
balancing information and computation power is essential for mobile systems.
Therefore, the HIPer-Model mimics this approach by concentrating on visual
perception using an RGB-D camera. Supplementary sensors, a 2D LiDAR and wheel
odometry, are integrated to improve the robot’s localization capabilities.

###  IV-B SLAM

SLAM provides a diverse range of established methods. When reusing an existing
approach, a thorough performance comparison is necessary. The performance of
prevalent feature-based 3D-SLAM methods is discussed in [57]. Approaches like
_ORB-SLAM2_ , _RTAB-Map_ , and _SPTAM_ demonstrate an equally outstanding
performance with slight variations depending on factors like the environment
(indoors or outdoors) and choice of sensor(s). Exemplarily, RTAB-Map [10], a
widely-used graph-optimized dense SLAM method, is chosen due to its
popularity. RTAB-Map employs advanced techniques to robustly reconstruct the
spatial scene in 3D and is interchangeable since alternative methods share
equivalent inputs and outputs. The output of the SLAM, illustrated by the
reused component in Figure 3, includes registered point clouds with the
graph’s pose ID, enabling anchor observations $o$ on this graph while
maintaining awareness of loop closures and post-optimizations to update
aggregated instances.

###  IV-C Background Segmentation

Preliminary tests indicate that most background data within indoor
environments pertains to building structures, which can be defined by basic
shapes. Since the background is assumed to be semi-static, the segmentation
uses the registered point cloud on the SLAM’s graph as input to obtain
$O_{b}$, as summarized in Algorithm 1. First, the point cloud is downscaled to
extract planes efficiently using _SACSegmentation_ [58]. Afterward, a simple
plane classification based on the alignment of the normal vector assigns the
object class: _wall_ , _floor_ , _ceiling_ , and _table_ or _unknown_.

Algorithm 1 Proceeding of extracting new background observations in 3D point
clouds.

1:function ExtractOb($points$, $pose$)

2: $points$ = DownscalePoints($points$) $\triangleright$ reduce the resolution

3: $points$ = TransformToRobot($points$)

4: $planeList$ = DetectPlanes($points$)

5: $planeListClassified=$Classifier($planeList$)

6: for $plane$ in enumerate $planeListClassified$ do

7: if $plane.Type$!=UNKNOWN then

8: $O_{b}[plane])$ = GenerateAndUpload($plane$)

9: end if

10: end for

11: return $O_{b}$

12:end function

Finally, $o_{b}$ is generated from the plane and uploaded to the knowledge
base. In this step, meta-information, such as an Universally Unique Identifier
(UUID), an aligned bounding box, the current robot pose id, and the contour of
the plane, are extended.

###  IV-D Background Merging

The acquired observations $O_{b}$ correspond to either a previously or newly
perceived instance. Algorithm 2 describes the merging of corresponding
observations with instances.

Algorithm 2 Merging of background observation $o_{b}$ with instances $I_{b}$.

1:$I_{b}$ $\triangleright$ background instances from KB

2:function MergeOb($o_{b}$, $I_{b}$)

3: $o_{b}$ = TransformToMap($o_{b}$, $\textsc{pop}(\textit{posegraph})$)

4: def $mergeList$

5: for $i_{b}$ in $I_{b}$ do

6: $mergeList$.add($i_{b}$) = MergeCriteriaCheck($o_{b}$, $i_{b}$)

7: end for

8: switch size($mergeList$) do

9: case 0:

10: $o_{b}\rightarrow\text{NEW}\ i_{\text{new}}$

11: case 1:

12: MERGE $o_{b}$ to $i_{\text{index}}$

13: case >1:

14: MERGE $o_{b}$ to $i_{\text{index}}$, delete duplicates

15:end function

First, each $o_{b}$ undergoes a transformation from the robot graph to the
global 3D space. Subsequently, a set of criteria with increasing complexity is
employed to assess the similarity of $o_{b}$ with each $i_{j}$, including
class, alignment, bounding box, and finally, the contour by overlap. Upon
success, $i_{j}$ is updated, i.e., the UUID of $o_{b}$ is linked, and the
spatial data are merged. Moreover, any potential duplicates are removed. In
case of unsuccessful merging, $o_{b}$ is converted into a new instance.

###  IV-E Object Detector

Selecting a suitable object detector requires comparing their performance. The
COCO dataset [59] offers common object classes found in the target application
and is widely used for comparing performance. Thus, we selected three
detectors with the highest mAP111https://paperswithcode.com/sota/object-
detection-on-coco, accessed 01.12.2023: _YOLO V6_ [60], _DETR_ [19], and _SWIN
V2_ [17]. Table I compares their performance with runtime-oriented settings.
Although the method _SWIN V2_ achieves the highest precision, followed by
_DETR_ , _YOLO V6_ performs significantly faster and more efficiently, as
indicated by the frame rate and inference speed. Considering this limitation,
we propose using _YOLO V6_ since it achieves the best performance with respect
to the availability of large and diverse datasets, exchangeability, and
computation effort.

TABLE I: Performance comparison of object detector with the highest mAP for the COCO dataset (*3rd party results). The frame rate and inference speed are measured with the same processing unit. The video input is set to 640x480 pixels at $30\text{\,}\mathrm{H}\mathrm{z}$. Description |  YOLO V6 |  DETR |  SWIN V2
---|---|---|---
Frame Rate (Hz) |  29.8 |  6.79 |  10.5
Inference Speed (CPU; GPU) |  240; 21 |  275; 44 |  340; 10
Detection Accuracy (mAP*) |  58.2 |  64.2 |  65.0

###  IV-F Foreground Segmentation

The foreground segmentation aims to extract region proposals of instances by
obtaining a set of observations $O_{f}$ each time the RGB-D camera provides
new data as summarized by Algorithm 3. The approach enhances its performance
by removing the background to narrow down the regions of interest. This
strategy is akin to human preattentive filtering [6], enabling the processing
to focus attention on areas that exhibit greater complexity in terms of visual
properties and dynamics compared to the background. Foreground objects are
assumed to be dynamic, necessitating the consideration of object motion and
the robot’s ego-motion simultaneously, supported and referenced by the
background. Specifically, bounding boxes of initially mapped scene structures
are filtered from the high-frequency point cloud of the robot camera. The
remaining points are then 3D segmented into regions without semantics using
Euclidean Cluster Extraction [58]. Finally, the observations are generated
equivalent to those of the background.

Algorithm 3 Implemented foreground object proposal segmentation.

1:$points$ $\triangleright$ point cloud from camera

2:$I_{b}$ $\triangleright$ sum of background instances from KB

3:function SegmentForeground(input $points$)

4: $points$ = DownscalePoints($points$) $\triangleright$ reduce the resolution

5: $I_{b}.bbox\leftarrow$ loadBackground $\triangleright$ get bounding boxes

6: $I_{b}.bbox$ = TransformToRobot($I_{b}.bbox$)

7: $points$ = FilterBackground ($points$, $I_{b}.bbox$) $\triangleright$
remove points from background

8: $regionList$ = SegmentPoints ($points$) $\triangleright$ using
EuclideanClusterExtraction

9: for $region$ in enumerate $regionList$ do

10: $O_{f}[region])$ = GenerateAndUpload($region$)

11: end for

12: return $O_{f}$

13:end function

###  IV-G Foreground Matching

In parallel to the foreground segmentation, the previously selected detector
uses the color image of the camera to detect objects. The matching tries to
assign the semantic to the region proposals as presented in the Algorithm 4.
First, it synchronizes detections and 3D blobs using their time stamps. Then,
the IOU is calculated after projecting the 3D blobs into the image plane. If
the IOU crosses a certain threshold, the object class is extended to $o_{f}$.
Although this approach allows handling many objects, unknown objects must be
considered in real-world applications. Thus, not matched regions are tagged as
unknown.

Algorithm 4 Semantic assignment of 3D-blobs proposals with bounding boxes.

1:$O_{f}$ $\triangleright$ 3D blob from foreground segmentation

2:$detectionList$ $\triangleright$ bounding boxes from detector

3: ($O_{f}$, $detectionList$) $\leftarrow$ synchronized

4:function AssignSemantic(input $O_{f}$, $detectionList$)

5:
$IOU\leftarrow\text{EmptyMat}(\text{length}(O_{f}),\text{length}(detectionList))$

6: for $o_{f}$ in enumerate $O_{f}$ do

7: for $d$ in enumerate $detectionList.\text{box}$ do

8: $IOU_{image}=\textsc{CalculateIOU}(o_{f}.\text{region},d.\text{box})$

9: if $IOU_{image}OU>threshold$ then

10: $o_{f}\leftarrow d.\text{objectclass}$ $\triangleright$ semantic
assignment

11: end if

12: end for

13: end for

14:end function

###  IV-H Multi-Object Tracking

The foreground observations $o_{f}$ are fed into a real-time optimized MOT to
merge multiple 3D detections into instances $i$ with the consideration of
dynamics. The approach uses a Hungarian algorithm[61] for matching the object
center in world coordinates. Afterward, a Kalman Filter [8] tracks each
object, using its position and derived velocity. The predicted velocities
without updates are continuously reduced until zero to avoid shifting outside
map boundaries.

##  V Knowledge Representation

The knowledge base represents acquired scene information in a known structure.
In this context, human knowledge consists of semantic and episodic memories
obtained through life-long learning, organized hierarchically and stored in
multiple spatial layers [62, 63]. The HIPer-Model adapts this concept but
already benefits from initial knowledge deployment and simplified data
exchange between multiple agents [6]. Prior knowledge is deployed through
training of the detector for knowing relevant semantics in the particular
environment.

###  V-A Knowledge Structure

Adapting the hierarchical structure from human perception implies layered
modeling of available scene data, anchoring with the consideration of updates
after graph optimizations, and handling episodic memories that link to the
corresponding observations. Related work uses two fundamental concepts of data
management systems: Relational Database Management System (RDBMS) and
document-oriented databases [6]. RDBMS require explicit modeling of knowledge
through a predefined ontology, while document-oriented databases merely
require implicit modeling without any restrictions, allowing the linking of
documents to build an ontology. For the addressed cross-task application, we
emphasize using a document-oriented database over RDBMS as it offers the
required flexibility, e.g., having different expert knowledge but the same
common understanding), for domain transfer, and in-operation extensions. Vice
versa, RDBMS are beneficial for complex knowledge modeling such as the
_Knowrob_ [47]. As ontology, related approaches, such as _Kimera_ [36] and
_Hydra_ [46], respectively, model hierarchical semantic structures of
buildings linked through scene graphs. The HIPer-Model sets up the document-
oriented database with a similar spatial structure; however, it splits scene
knowledge into four layers according to Fig. 3: environment ID, foreground,
background, and spatial map. This flat hierarchy was chosen to maximize
flexibility for cross-applications and cross-robot use while supporting their
ontology for certain scenarios. Our database stores each scene information in
a separate document. Documents of instances link to corresponding observations
through their UUID to define an implicit relation between them.

###  V-B Entities

The knowledge base stores $O$, $I$, and $A$ according to Equation (5). Each
entity has a hierarchical structure comprising redundant high-level and
individual low-level attributes. High-level attributes are the UUID, the
corresponding robot frame name, the time of the last update, the object class,
and the bounding box. While instances are linked to the global map frame,
observations are linked to the sensor frame, allowing recalculating instances
with updated observations pose after graph optimizations of the SLAM. Low-
level attributes of observations are raw sensor data (e.g., masked color
image) within the Region of Interest (ROI). In contrast, low-level attributes
of instances are aggregated spatial data (e.g., fused points) with a
trajectory and links to a batch of time-continuous child observations. In
addition, general data, such as the robot trajectory, spatial map data, and
cross-class properties, are stored.

###  V-C Accessibility of Scene Knowledge

The service interface provides scene information for decision-making,
navigation, manipulation, and scene visualization. In particular, SQL queries
are used to search instances $i$ in the database based on their attributes
(e.g., UUID, object class), or a criterion, such as closest table, or cups in
the FOV. Motion planning for manipulation requires calculating the distance
between each part of the robot arm and each obstacle simultaneously, thus a
small amount of data points is recommended. This is facilitated by an
additional interface that retrieves scene data within a specific workspace
that can be activated and deactivated. In particular, it provides a static
background, represented through bounding boxes with semantics. In parallel, it
provides the dynamic foreground, represented through an occupancy grid with
semantics for object-aware obstacle avoidance to the manipulation.

##  VI Knowledge Interpretation

The interpretation of scene knowledge $A$ encompasses multiple dimensions,
including time, space, and relations, allowing for a comprehensive scene
analysis over an extended period. These techniques enhance scene understanding
by extending and robustifying object instances $i$ and scene semantics during
operation within the scene. Exemplarily, two features exhibiting high
potential for multifunctional robots are incorporated: feature learning and
heatmaps.

###  VI-A Feature learning

Learning environment-specific perceptual information is a major step toward
human-like life-long learning. The objective is to determine environment-
specific object properties, acquired through a time-series analysis of a
particular instance or all instances of a certain class. The typical object
dynamic, volume, and altitude are determined since they represent valuable
information to understand their behavior.

####  VI-A1 Object Dynamic

To determine the object dynamic of a particular class, their trajectories are
split into two sets: static $S$ and dynamic $D$. Transitioning between these
states is less clear due to expected measurement noise during the stationary
phase. An Normal Distribution (NDT) of the measurement noise is expected. The
absolute velocity is considered, thus, a folded NDT with zero mean $\mu$ and a
variance $\sigma^{2}$ describes the probability density function of the
stationary portion as

$S_{prob}(|\vec{v}|;\sigma)=\sqrt{\frac{2}{\pi}}\frac{1}{\sigma}e^{-\frac{|\vec{v}|^{2}}{2\sigma^{2}}}.$
(7)

The threshold selected for the stationary to dynamic transition is $2\sigma$,
i.e., at $95.45\text{\,}\mathrm{\char 37\relax}$ of the noise. The dynamic
portion below this threshold is described with the inverse noise distribution.
Above, due to no prior object knowledge, a uniform distribution was selected
to describe the dynamic portion. However, a class-specific dynamic range is
expected within the confidence interval $CI_{D}=[c_{l};c_{u}]$. The resulting
probability density function of the dynamic portion is described as

$D_{prob}(|\vec{v}|;\sigma_{\varepsilon}^{2})=\begin{cases}1-e^{-\frac{\vec{v}^{2}}{2\sigma_{\varepsilon}^{2}}},&\text{if
}|\vec{v}|<2\sigma\\\ 1,&\text{if }2\sigma<|\vec{v}|<\text{97.5th
perc.}.\end{cases}$ (8)

The _Object Dynamic Score_ ($ODS$) is introduced, calculated for each object
class as

$ODS=[\mu_{\vec{v}},c_{l},c_{u},r].$ (9)

For each class, it indicates the mean velocity $\mu_{\vec{v}}$, the confidence
interval of the velocity, and a dynamic-to-static ratio $r$, which allows
reasoning about the object class reactivity. We used the object dynamic to
adapt the MOT pipeline in two ways: First, optimize the tracking by using
$\sigma$ to calculate class-specific covariances, and second, reject distances
of the Hungarian algorithm beyond $c_{u}$ multiplied by the time delta
(condition: dt $<$ $1\text{\,}\mathrm{s}$).

####  VI-A2 Object Size

Determining this feature is essential to understanding the typical size of
objects in the scene, particularly important since the detector uses 2D data
without depth measurements. To minimize false positives, sizes beyond an
experienced range are filtered. The distribution of the bounding box volumes
is considered under the assumption of no shape change since it is invariant to
origin and orientation. Similar to the object dynamic, a uniform distribution
is selected. Bootstrapping calculates the $95\text{\,}\mathrm{\char 37\relax}$
confidence interval ($CI$) for class-specific bounds, predicting future sizes
fit within this range with the same probability.

####  VI-A3 Object Altitude

Determining an object’s height above the floor follows the same methodology as
the object size. The altitude is calculated by subtracting the vertical
bounding box size from the object’s center. The confidence interval of the
resulting uniform distribution helps minimize false positive detections, such
as understanding that people walk on the floor altitude, enabling optimization
of robot behavior based on anticipated object locations.

###  VI-B Heatmap generator

Learning the locations of a particular instance or object class is valuable to
understanding regularities since it provides feedback about frequently
observed areas of a specific time. The knowledge base provides historical
scene data to generate so-called heatmaps with the procedure in Algorithm 5,
an improved version first introduced in [64]. Instance data within a specified
time delta is obtained from the knowledge base using a query with the desired
start and end time and the instance type. It inflates instance trajectories
with a normal distribution. The heatmaps are filled up to an upper threshold
$80\text{\,}\mathrm{\char 37\relax}$ occupancy and overlayed with the
navigation costmap to avoid particular areas, such as crowded spaces.
Alternatively, their inverse representation can be generated to explore these
regions.

Algorithm 5 Proceeding of the heatmap generation.

1:$I_{f}$ $\triangleright$ sum of relevant instances from KB

2:$t_{1}$, $t_{2}$ $\triangleright$ start and end time

3:$map$ $\triangleright$ 2D grid map

4:function HeatmapGenerator(input $map$)

5: $pointList$, $\mu_{\vec{v}}$ = QueryKB($I_{f}$, $t_{start}$, $t_{end}$)

6: $heatmap$ = $map$.copy

7: for $point$ in $pointList$ do

8: $heatmap.at[point]\leftarrow$ NDT ($\mu_{\vec{v}}$) $\triangleright$
overlay NDT

9: end for

10: $heatmap\leftarrow$ scaled

11:end function

##  VII Experiments

The experiments aim to evaluate the performance of holistic scene
understanding and its benefits for multifunctional mobile robots within a
long-term running setup. Unfortunately, we could not find such a benchmark,
where a mobile robot repeatedly perceives a restricted area with dynamic
objects. Related work uses specific datasets of one-time routes, e.g. in [36],
or non-reproducible real-world settings [65, 43, 66], aiming to quantify a
specific part of perception, such as the metric-semantic reconstruction
accuracy. Given the objective of a holistic approach and the fact that it
integrates well-known recognition techniques, i.e., SLAM and an object
detector, existing benchmarks are unsuitable. Therefore, a new benchmark has
been developed to measure the perception’s performance and its implications
for typical tasks in a long-term running setup. The benchmark uses a single-
setting ablation study to reveal the impact of each component on the overall
performance. Noteworthy, simulated, and real-world environments are supported
to have a reproducible baseline and to verify a sim-to-real transfer,
respectively.

###  VII-A Scenario and Setup

The scenario and setup of the HIPer-Model address multi-functional robots in
the open-world context that repeatedly take over various arbitrary tasks.
Therefore, the scenario choice and design must be according to this.

####  VII-A1 Scenario

The scenario, covering the robot’s tasks, includes essential techniques and
physical interactions with the environment. While collecting and carrying
objects can be tedious for humans, it is a manageable task for robots and a
popular function of service robots [67, 68]. User studies, such as in [69],
highlight this task as crucial in supporting humans in domestic environments.
Therefore, a _fetch-and-carry_ scenario is selected consisting of four sub-
tasks, abbreviated as T, that address significant capabilities:

T1

_Find Person_ : Start at an idle position and search for a person to offer
help. The task ends when the person has been detected.

T2

_Search Object_ : After detection, the person is assumed to request a bottle
which is subsequently searched for by the robot. The task ends when a bottle
has been detected.

T3

_Grasp Object_ : Start at the position where the bottle has been detected.
Approach and grasp the bottle.

T4

_Carry Object_ : Carry the bottle to the idle position.

The first task tackles the robot’s navigation and ability to find a dynamic
object, i.e., a person in the scene. The second task is similar, but the
target object is semi-static. The third task uses the robot’s manipulation to
grasp the object. The last task is a pure transport task from point A to B,
where the robot has to efficiently and safely navigate to its goal.

####  VII-A2 Setup

To perform these tasks, we selected three different everyday environments.
While two simulated environments provide a reproducible and known baseline,
one real-world environment demonstrates a realistic setup without a fully
known baseline. Both share the same notebook as the processing unit222CPU:
i9-11980HK, GPU: Nvidia 3060 RTX, RAM: 32GB, OS: Ubuntu 20.04 with ROS noetic
and use an RGB-D camera with 1280x720 pixels at
$30\text{\,}\mathrm{H}\mathrm{z}$ to allow a comparison of the perceptual
performance.

Idle positionBottle (a) House

Idle positionBottle (b) Hospital

Figure 4: For evaluation selected publicly available virtual environments from
Amazon [70]. Walking people are spawned to enrich dynamics.

(a) MobiKa

(b) Office Building

Figure 5: Real-world office setup of the robot _MobiKa_ (a), comprising a
Kinect Azure RGB-D camera and notebook mounting. The experiments take place in
an office building (b) comprising a central stairwell and a lab connected by
corridors.

The HIPer-Model and related work share the use of the Robot Operating System
(ROS). Thus, we emphasize taking its _Gazebo_ simulator over _Unity_ , used by
[36], _Unreal Engine_ , or _AI Habit_ , due to its native support of the
entire robot data. The simulated experiments use two popular open-source
Gazebo worlds, house, and hospital (see Fig. 4), provided by Amazon [70]. The
virtual experiments are performed with _Care-O-bot 4_ , a multifunctional
mobile service robot [71]. Both virtual environments are static, whereas
dynamic objects have been integrated to achieve a more realistic setup,
particularly important for evaluating foreground recognition and spatio-
temporal scene analysis. Since people represent the most important dynamic
objects for service robots, animated people have been added using the Social
Force Model (SFM) proposed in [72]. One person is moving randomly in the house
and five people are moving between predefined positions in one corridor of the
hospital.

The real-world experiments take place in an office building (Fig. 5(b)) using
our research robot _MobiKa_ [73] (Fig. 5(a)). The robot is fully controlled
with the same notebook as used for simulation. We neither adapted the
environment nor instructed people how and where to move to achieve a realistic
setup.

TABLE II: Selected ablation study configurations to identify the component-wise contribution to the overall performance. |  Basic Model |  Foreground
Recognition |  Background
Recognition |  Knowledge Base |  Heatmaps |  Feature
Learning
---|---|---|---|---|---|---
C1 |  x |  x |  x |  x |  x |  x
C2 |  x |  x |  x |  x |  x |
C3 |  x |  x |  x |  (x) |  |
C4 |  x |  x |  |  (x) |  |
C5 |  x |  |  |  |  |

###  VII-B Experiment Execution

The tests are split into the initial mapping and its following long-term
operation for each environment. The initial mapping only uses the background
recognition pipeline at $2\text{\,}\mathrm{H}\mathrm{z}$ without scene
dynamics, i.e., without walking people, a common approach in robotics to
reconstruct the scene by a systematic, user-controlled exploration of the
entire environment. Afterward, the SLAM switches to localization mode. Walking
people were spawned as described in the setup when starting long-term
operations. The robot systematically covers the scene for two hours with an
implementation presented in [74]. The single-setting ablation study then
assesses how each component contributes to the scenario performance. T1-T4 are
performed with a sample size of ten per environment and configuration as
defined in Tab.II. One exception concerns the skipping of T3 in the real-world
experiment since the robot has no manipulator.

###  VII-C Results

The results are split into background reconstruction and long-term perception
to obtain the perceptual performance, followed by the ablation study.

####  VII-C1 Background Reconstruction

The background reconstruction results in Tab. LABEL:tab:backgroundsemantics
indicate a precision $>98\text{\,}\mathrm{\char 37\relax}$ in the static-
dynamic split with only a dynamic care cart within the hospital and one
whiteboard in the office wrongly assigned as background. The classification
indicates a few False-Positive (FP) of furniture, such as sofas and shelves
(cp. Fig. 6). The frame processing time is faster than the mapping unless re-
calculating due to graph optimization. Note: The floor and ceiling were
filtered during reconstruction and manually added due to low data information
with respect to the processing time.

TABLE III: Perceptual performance of the initial background reconstruction for the selected environments. Description |  House |  Hospital |  Office
---|---|---|---
Processing time / s: $\mu$; $\sigma$ |  0.201; 0.113 |  0.202; 0.159 |  0.157; 0.100
Latency / s: $\mu$; $\sigma$ |  0.208; 0.153 |  0.147; 0.103 |  0.53; 0.172
CPU, GPU load / %: $\mu$; $\sigma$ |  35.9; 0 |  39.1; 0 |  21.6; 0
Sum: $i$; $o$ |  58; 3129 |  103; 4453 |  124; 1566
Precision stat./dyn. / % |  100.0 |  98.4 |  99.2
Precision semantics / % |  74.1 |  72.8 |  68.6
Walls: TP; FP |  37; 13 |  74; 28 |  87; 44
Tables: TP; FP |  6; 2 |  1; 0 |  2; 1
Furniture: FP |  15 |  14 |  13
Door, door frame: FP |  0 |  12 |  27
Foreground instances: FP |  0 |  2 |  1
Background / m : $\mu$; $\sigma$ |  0.022; 0.059 |  0.033 0.096 |  N/A
Idle positionBottleFP Figure 6: Reconstructed office background. Whereas
static parts have been assigned successfully throughout, the semantic
classification fails for large furniture, such as for shelves, since they have
been labeled as walls (FP). The idle and bottle positions represent the key
positions for the scenario. TABLE IV: Perceptual long-term performance for the
selected environments.

Description |  House |  Hospital |  Office
---|---|---|---
3D detect. time / s : $\mu$; $\sigma$ |  0.047; 0.054 |  0.118; 0.122 |  0.101; 0.026
3D detect. latency / s : $\mu$; $\sigma$ | 0.0774; 0.122 |  0.43; 0.232
CPU; GPU load / % : $\mu$ |  48.5; 12 |  51.5; 12 |  43.7; 21
Classification1 $i$: TP; FP |  42; 3 |  297; 8 |  66; 4
Sum $i$, sum $o$ |  45; 61489 |  385; 158919 |  70, 16262
Loc. accuracy / m : $\mu$; $\sigma$ |  0.014; 0.018 |  0.015; 0.012 |  0.020; 0.011
People dyn. ratio / % |  26.8 |  18.0 |  5.6
People $|v|$ / m/s : $\mu$; $\sigma$ |  0.126; 0.060 |  0.157; 0.087 |  0.101; 0.191
People $CI$ / m/s: $c_{l}$; $c_{u}$ |  [0.051; 0.288] |  [0.037; 0.580] |  [0.030; 0.277]

  * 1

Not present classes, such as _boat_ and _traffic light_ , have been excluded
beforehand.

####  VII-C2 Foreground Long-Term Perception

The resulting perceptual performance is summarized in Table IV. Its foreground
recognition pipeline achieved about 10 FPS, varying on the segmentation
performance, i.e., the number of processed points after background filtering.
CPU and GPU loads are comparatively low. The higher GPU load in the real-world
experiments is traced back to the camera driver. A high amount of scene data
with many True-Positive (TP) instances has been acquired thanks to filtering
non-present object classes. Separate experiments verify that the instance
localization is precise and independent of the robot’s velocity. The total
number of instances includes multiple assigned ones due to a missing re-
detection. The long-term analysis of the object dynamics indicates that people
have been observed $5.6\text{\,}\mathrm{\char 37\relax}$ (office) to
$26.8\text{\,}\mathrm{\char 37\relax}$ (house) moving. Since the simulated
environments offer a known ground truth, their determined object properties
are analyzed in detail. Figure 7 shows them of the hospital where people
systematically move from A to B with baseline dynamic portion of
$100\text{\,}\mathrm{\char 37\relax}$ and a constant velocity of
$0.6\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$. The results indicate that the
people’s velocity is $97.5\text{\,}\mathrm{\char 37\relax}$ below
$0.58\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$, i.e., the prediction of the
velocity is systematically too low.

Figure 7: The histograms show the distributions of determined properties of
class person in the hospital environment. The ground truth is overlayed to
allow its interpretation.

The reason might be the experimental setup paired with the detection rate. The
robot faces the people with a velocity difference of up to
$1.5\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ (robot $v<=0.9m/s$). An analysis
shows that due to the limited point cloud range of $5\text{\,}\mathrm{m}$, the
robot has less than $3.34\text{\,}\mathrm{s}$ and approximately 30 detections,
respectively, to determine the person’s velocity. Notably, as the Kalman
Filter initializes person tracks with a zero velocity, several frames must
gradually accelerate the velocity and decelerate it to zero after the
disappearance. Observing the object size and height shows that objects were
incompletely recognized since the volume measurements are below and the
altitude is higher than the ground truth. Nonetheless, the confidence interval
offers a robust foundation for determining the expected range of properties.
Thus, it robustifies matching and tracking for static and dynamic instances
since the robot can predict known object classes’ size, altitude, and
dynamics.

Figure 8: Task time and success rate for all ablation configurations (C1: Full
Model, C5: Basic Model) and environments. The scenario time is the aggregated
task time, depicted by the mean ($n=10$). The standard deviation per task is
denoted by the black bar and the success rate on each task’s right. The
scenario success rate is redundant to T4 since the tasks are executed
sequentially.

Start positionStairwellLabPerson (a) T1 Find Person

BottleStairwellLab (b) T2 Find Object

End positionStairwellLab (c) T4 Carry Object

Figure 9: Procedure of the selected tasks in the office building with the
entire HIPer-Model (C1). The robot navigates across the blue trajectory using
prior scene knowledge. First, the robot uses the inverse heatmap of people (a)
to search for people in heated areas, such as near the staircase and the
central area of the lab. After detecting a human, the robot repeats the
procedure with the inverse heatmap of bottles (b) to find the closest one.
Finally, the robot returns to the idle position, where the heatmap of people
optimizes its navigation to avoid crowded areas (c).

####  VII-C3 Ablation Study

The ablation study removes system components to obtain insights into their
functionality, a widely applied approach in fields like machine learning. It
starts with the entire HIPer-Model and then systematically ablates fragments
as described by the configurations in Table II until a so-called _basic model_
is released. The basic model only comprises the SLAM and the object detector,
adopted from the early approach presented in [68]. The resulting time and the
success rate of all configurations and tasks are compared in Fig. 8. A task is
considered unsuccessful if either a failure occurs or its execution exceeds
$600\text{\,}\mathrm{s}$.

#### C1: Entire Model

An exemplary procedure within the office building is visualized in Fig. 9. The
robot executes the tasks using its acquired scene knowledge starting at its
idle position as highlighted in Fig. 4 and Fig. 6. In T1, the robot employs an
inverse heatmap generated for class ”people” to navigate into prior crowded
areas to search for a person. The task time variability is higher in
environments with random people movement, such as when comparing the house and
the hospital. In T2, the robot can constrain its search for the bottle due to
its knowledge of the environment-specific altitude and the usage of its
heatmap. T3, executed in simulation only, involves the robot approaching
recognized bottles using a method described in a prior study [73], followed by
the bottle grasping. The grasping uses the 3D bounding box center of the
recognized rotation-symmetric bottle, as described in our previous study [68].
The semantically enhanced regions (cp. Fig. 10) around the bottle provide a
high-level understanding of the local workspace. They allow for optimized
obstacle handling due to the knowledge of static background areas, represented
by primitive shapes, and foreground regions, represented by a voxel grid
assigned with the $ODS$. In T4, the robot utilizes similar to T1 the heatmaps
of people to avoid crowded areas for fast and safe navigation.

Figure 10: Third person view on the manipulation task. The manipulation
interface provides the static background, represented by basic shapes, and the
dynamic foreground, represented by an occupancy grid with semantics and $ODS$,
within the manipulation workspace (red cube).

#### C2: Feature Learning Removed

Excluding the component of feature learning implies that the robot lacks
environment-specific object knowledge regarding the typical altitude, size,
and dynamics. The results indicate that T1 and T2 do not yield significant
changes. However, one FP appears in the office where another robot has been
identified as person (cp. Fig. 11(a)), a hard example that could have occurred
with the entire approach as well due to its similar size.

(a) Robot detected as person

(b) Recognized people on a TV

(c) Bottle on a cluttered desk

Figure 11: Challenges in object recognition experiments include: (a)
Occasional misidentification of another robot as a person in the office. (b)
Intermittent failure of C5 to recognize people on the TV in the house during
T1. (c) Difficulty in detecting small bottles in the hospital due to size,
occlusions, and cluttered background, leading to task timeouts.

In T2, the robot adapts to no longer assuming a specific altitude for bottles,
relying on heatmaps to limit potential bottle areas with consistent results.
T3 involves approaching and requesting scene data for manipulation, but the
robot loses awareness of potential dynamic elements, necessitating the
maximization of safety distances. While impacting safety, this change does not
significantly affect task time and success rates. Lastly, in T4, the robot
navigates without calculating object-class-related safety distances but
achieves consistent results by avoiding frequent and crowded areas, as only
people are considered dynamic. Concluding, removing the perceptual learning
component does not significantly impact the task time but slightly increases
the error rate.

(a) Crowded corridor with walking people challenges the robot navigation.

ABA’B (b) Map after one (top) and two (bottom) hours. The heatmap (colored) is
overlayed with the costmap (grayscale).

Figure 12: Heatmaps generated from prior scene knowledge increase the path
costs in the crowded southern hospital corridor (a). After collecting
sufficient scene information, the robot path (green) switches to the empty
northern corridor (b).

#### C3: Heatmaps Removed

This configuration removes the heatmaps to ignore historical object knowledge
for behavior control but keeps the knowledge base since it is necessary for
the MOT. The removal changes in T1 that the robot has to perform a coverage of
the entire area due to missing prior information. The impact without heatmaps
varies across environments, $5\text{\,}\mathrm{\char 37\relax}$ faster in the
house but a substantial $1535\text{\,}\mathrm{\char 37\relax}$ slower in the
hospital. In the unconstrained office building the task time of T1 has been
increased by $47\text{\,}\mathrm{\char 37\relax}$ as a result of the longer
path. T2 presents a similar challenge when searching for a bottle. The task is
finished $398\text{\,}\mathrm{\char 37\relax}$ slower in the house,
$175\text{\,}\mathrm{\char 37\relax}$ slower in the hospital, and
$275\text{\,}\mathrm{\char 37\relax}$ slower in the office. Moreover, the
success rate decreases, such as in the hospital, where 6/10 runs have been
aborted as they exceed the time limit. Although the bottle is reached in time,
bottle detection is challenging due to its size, and the cluttered scene,
which causes occlusions (cp. Fig. 11(c)). T3 indicates that ablating heatmaps
and the object aging feature have no effect. In T4, the robot is no longer
avoiding crowded areas and experiences slower task times, especially in the
hospital, with a significant $27\text{\,}\mathrm{\char 37\relax}$ increase
since the robot takes the shorter path through the crowded corridor instead of
the empty one as before (cp. Fig. 12). The change in task time, though
insignificant in other environments, underscores the relevance of productivity
and safety during navigation.

#### C4: Basic Model with Foreground Recognition

Removing background recognition means that foreground recognition must process
the entire point cloud. This leads to a significant increase in frame
processing time and latency for foreground recognition in all tasks. For
example, in a hospital setting, the latency increases from an average of $\mu$
= $0.077\text{\,}\mathrm{s}$, $\sigma$ = $0.122\text{\,}\mathrm{s}$ (C1-C3) to
$\mu=$$0.130\text{\,}\mathrm{s}$, $\sigma$ = $0.47\text{\,}\mathrm{s}$ (C4),
representing an approximately $69\text{\,}\mathrm{\char 37\relax}$ higher
processing delay. The absence of background recognition alters task behavior
in the following ways: In T1, the absence of background recognition leads to a
reduced success rate since people in portraits on walls and on a TV (Fig.
11(c)) have been wrongly identified. T2 performs similarly to before but with
a lower bottle detection accuracy due to the occasional merging of the desk
with bottle points to a unified segment. In T3, the missing background makes
the distinction between static and dynamic elements unfeasible. This leads to
increased data processing and higher latency, with a $70\text{\,}\mathrm{\char
37\relax}$ average increase in dynamic obstacles and a
$69\text{\,}\mathrm{\char 37\relax}$ increase in latency. The manipulation
planning process is also affected, requiring consideration of more data points
for obstacle-aware motion planning. T4 indicates that the absence of
background information amplifies cluster sizes for the manipulation task, and
a slower MOT pipeline compromises the robot’s reactivity and safety.

#### C5: Basic Model

This configuration comprises the SLAM and object detector, similar to an
earlier approach outlined in [68]. In T1, the robot achieves faster detection
due to stand-alone usage of the 2D detector without range limitations but
experiences a low success rate due to FP on portraits and TV. T2 has been
executed similarly to before, but also profiting from the longer range for
detecting the bottle. The bounding boxes of the 2D object detector are
projected into 3D space to localize objects for T3. T3 faces complexities in
object identification for grasping, lacking table support, and the ability to
perceive clusters. While manipulation can access static background data, the
foreground presents unprocessed raw data without clustering or semantic
labels. Finally, T4 was performed as before.

##  VIII Discussion

Related integrated approaches lack a common baseline, and benchmarks
evaluating the perception based on A-B motions are not designed for long-term
applications. The introduced benchmark addresses this issue by presenting
feasible everyday tasks for today’s mobile robots. The experiments showcase a
cross-environment deployment in which the split between dynamic foreground and
static plane-based background achieved $>98\text{\,}\mathrm{\char 37\relax}$
precision and accelerate the foreground recognition by
$70\text{\,}\mathrm{\char 37\relax}$, thus agreeing with related work that it
can be generalized as a valid approach for indoor environments. In contrast to
related approaches, it solely requires the exchange of the reused 2D detector,
being available for many domains. Related work uses fully integrated, thus
indivisible approaches, e.g., with DL-based semantic segmentation, which is
problematic for computation and represents a challenge to find a proper
dataset.

The ablation study with its results in Fig. 8 proves that the HIPer-Model
empowers mobile robots to perform typical tasks faster and more reliably as it
reduces typical errors. Knowledge interpretation allows the adaptation to the
environment in operation, as demonstrated with feature learning and heatmaps.
Although feature learning achieved only minor improvements, it offers
potential beyond the experiment setup, e.g., for socially aware navigation by
using object tracks with the $ODS$ to define an appropriate safety distance.
Similarly, the features help the manipulation for motion planning since the
$ODS$ provides the experienced reactivity, thus, the safety distance can be
automatically maximized for dynamic objects and minimized for static objects.
The experiments showcase that if the objects of interest are situated in
distinct areas, heatmaps accelerate the scenario by up to
$61\text{\,}\mathrm{\char 37\relax}$, but experience a slight deceleration
when object classes are equally distributed. However, an unconstrained real-
world experiment demonstrated a $47\text{\,}\mathrm{\char 37\relax}$
acceleration. One unexpected effect occurs in C5, where the search tasks have
been executed faster than before in the house. This effect is traced back to
the setup of relying on pure 2D detections, which have no range limitation.
Thus, we propose using 2D detections as an initial region proposal to improve
the search strategy of C1-C4.

In total, 550 tasks have been executed. As the tasks are repeatedly executed
in a constant setting, the impact of each perception component has been
revealed. While the benchmark allows quantifying the performance of the HIPer-
Model, its comparison to related approaches is not yet applicable due to
experimental effort. Thus, we want to encourage other researchers to take over
our benchmark. Nonetheless, when comparing the basic model (C5) the
performance of the 3rd party components, i.e., RTAB-Map and YOLO, is well
known. To measure the impact of HIPer-Model-specific components is impossible
with an unconstrained setup, i.e., it would require sticking back to an A-B
motion dataset. Noteworthy, as no other component than YOLO sticks on the GPU,
the background filtering reduces the data points, and interpretation
components are triggered only ad-hoc, the computational efficiency is not yet
limiting the application as underlined by a preliminary version of the HIPer-
Model running on low-cost hardware [64]. This example also proves an easy
adaptation to other object classes and applications.

##  IX Conclusion

The bioinspired approach of taking over concepts of human perception
represents a novel approach to holistic scene perception for mobile robots.
Our prior study [6] highlights that holistic scene perception approaches,
i.e., which cover the entire process from recognition to interpretation of
acquired knowledge, are rare, particularly because interpretation techniques
are rarely discovered in robotic research. This paper introduces HIPer-Model,
a technical solution that solves the seamless integration of existing and
novel methods to generate a holistic scene perception by imitating two major
neuroscience concepts. First, the triplet split of human perception allowing
to define functionalities and their purpose clearly. The recognition generates
scene observations and converts them to aggregated instances, a multi-layer
knowledge base represents these entities, and spatio-temporal analyses
interpret the obtained scene knowledge. Second, the preattentive and
postattentive split, a sub-concept of the recognition process that splits the
foreground and background.

Simulation and real-world experiments validate that the approach significantly
improves the performance of typical tasks in the open-world context. The
experiments prove that the holistic approach empowers mobile service robots to
execute popular tasks faster and more reliably than stand-alone SLAM and
object detection solutions since open-world problems are considered. Although
a gap to human capabilities remains, the HIPer-Model is a significant step
toward multifunctional robots as its concepts and experiments, which obtained
a high amount of data (over 0.5 million observations for a few hours),
highlight the potential for future work. The background classification lacks
performance which could be improved with a more complex approach to add
environment-specific knowledge, such as furniture, doors, and corridors. The
selected object detector achieved proper results but might require a wider
knowledge beyond the classes of the COCO dataset. Thus, we emphasize merging
concepts of open-vocabulary mapping to improve managing open-world problems.
We see high potential in connecting the knowledge base to a Large Language
Model to exploit the full potential of the data. Finally, we propose to deepen
perceptual learning, such as for the re-identification of objects, to handle
reappearance and corner cases.

##  X Acknowledgment

This research has received funding from the German Ministry of Economic
Affairs, Labour and Tourism Baden-Württemberg by the AI Innovation Center
Learning Systems and Cognitive Robotics under grant agreement No 017-180036
(841022).

## References

  * [1] S. Garg, N. Sünderhauf, F. Dayoub, D. Morrison, A. Cosgun, G. Carneiro, Q. Wu, T.-J. Chin, I. Reid, S. Gould, P. Corke, and M. Milford, “Semantics for Robotic Mapping, Perception and Interaction: A Survey,” _Foundations and Trends® in Robotics_ , vol. 8, no. 1–2, pp. 1–224, 2020.
  * [2] C. Müller, W. Kraus, B. Graf, and K. Bregler, _World Robotics 2023 - Service Robots_. VDMA Verlag, 2023.
  * [3] A. Allahverdi and H. M. Soroush, “The Significance of Reducing Setup Times/Setup Costs,” _European Journal of Operational Research_ , vol. 187, no. 3, pp. 978–984, 2008.
  * [4] P. Moubarak and P. Ben-Tzvi, “Modular and Reconfigurable Mobile Robotics,” _Robotics and Autonomous Systems_ , vol. 60, no. 12, pp. 1648–1663, 2012.
  * [5] J. Dong, “Holistic Scene Perception for Collaborative Human-Robot Teams,” PhD dissertation, Cornell University, 2023.
  * [6] F. Graf, J. Lindermayr, C. Odabasi, and M. F. Huber, “Toward Holistic Scene Understanding: A Transfer of Human Scene Perception to Mobile Robots,” _IEEE Robotics & Automation Magazine (RAM)_, vol. 29, no. 4, pp. 36–49, 2022.
  * [7] G. Pintore, C. Mura, F. Ganovelli, L. Fuentes-Perez, R. Pajarola, and E. Gobbetti, “State-of-the-art in Automatic 3D Reconstruction of Structured Indoor Environments,” _Computer Graphics Forum_ , vol. 39, no. 2, pp. 667–699, 2020.
  * [8] S. Thrun, W. Burgard, and D. Fox, _Probabilistic Robotics (Intelligent Robotics and Autonomous Agents)_. Cambridge, USA: The MIT Press, 2005.
  * [9] A. Khoyani and M. Amini, “A Survey on Visual SLAM Algorithms Compatible for 3D Space Reconstruction and Navigation,” in _IEEE International Conference on Consumer Electronics (ICCE)_ , 2023, pp. 01–06.
  * [10] M. Labbé and F. Michaud, “RTAB-Map as an Open-Source Lidar and Visual Simultaneous Localization and Mapping Library for Large-Scale and Long-Term Online Operation,” _Journal of Field Robotics_ , vol. 36, pp. 416–446, 2018.
  * [11] R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras,” _IEEE Transactions on Robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [12] Y. Tian, Y. Chang, F. Herrera Arias, C. Nieto-Granda, J. P. How, and L. Carlone, “Kimera-multi: Robust, distributed, dense metric-semantic slam for multi-robot systems,” _IEEE Transactions on Robotics_ , vol. 38, no. 4, pp. 2022–2038, 2022.
  * [13] S. Dörr, “Cloud-based Cooperative Long-term SLAM for Mobile Robots in Industrial Applications,” 2020.
  * [14] U.-H. Kim, J.-M. Park, T.-j. Song, and J.-H. Kim, “3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents,” _IEEE Transactions on Cybernetics_ , vol. 50, no. 12, pp. 4921–4933, 2020.
  * [15] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 779–788.
  * [16] W. Wang, J. Dai, Z. Chen, Z. Huang, Z. Li, X. Zhu, X. Hu, T. Lu, L. Lu, H. Li, X. Wang, and Y. Qiao, “Internimage: Exploring large-scale vision foundation models with deformable convolutions,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, USA: IEEE Computer Society, 6 2023, pp. 14 408–14 419.
  * [17] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, F. Wei, and B. Guo, “Swin Transformer V2: Scaling Up Capacity and Resolution,” in _International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 11 999–12 009.
  * [18] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Los Alamitos, USA: IEEE Computer Society, 2022, pp. 11 966–11 976.
  * [19] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _Computer Vision – ECCV 2020_. Cham, Switzerland: Springer International Publishing, 2020, pp. 213–229.
  * [20] D. T. Nguyen, W. Li, and P. O. Ogunbona, “Human detection from images and videos: A survey,” _Pattern Recognition_ , vol. 51, pp. 148–175, 2016.
  * [21] Z. Cao, G. Hidalgo Martinez, T. Simon, S.-E. Wei, and Y. Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 1, pp. 172–186, 2019.
  * [22] S. Lowry, N. Sünderhauf, P. Newman, J. J. Leonard, D. Cox, P. Corke, and M. J. Milford, “Visual Place Recognition: A Survey,” _IEEE Transactions on Robotics_ , vol. 32, no. 1, pp. 1–19, 2016.
  * [23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natural Language Supervision,” in _Proceedings of the 38th International Conference on Machine Learning_ , 2021, pp. 8748–8763.
  * [24] I. Kostavelis and A. Gasteratos, “Semantic mapping for mobile robotics tasks: A survey,” _Robotics and Autonomous Systems_ , vol. 66, pp. 86–103, 2015.
  * [25] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2017, pp. 2980–2988.
  * [26] C. Premebida, R. Ambrus, and Z.-C. Marton, “Intelligent Robotic Perception Systems,” in _Applications of Mobile Robots_ , ser. Applications of Mobile Robots, E. Gorrostieta Hurtado, Ed. Rijeka, Hungary: InTechOpen, 2019, ch. 6, pp. 111–127.
  * [27] F. Poux, “The Smart Point Cloud: Structuring 3D Intelligent Point Data,” 2019.
  * [28] F. Chen, X. Wang, Y. Zhao, S. Lv, and X. Niu, “Visual object tracking: A survey,” _Computer Vision and Image Understanding_ , vol. 222, p. 103508, 2022.
  * [29] Y. Zhang, C. Wang, X. Wang, W. Zeng, and W. Liu, “Fairmot: On the fairness of detection and re-identification in multiple object tracking,” _International Journal of Computer Vision_ , vol. 129, no. 11, pp. 3069–3087, 11 2021.
  * [30] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in _IEEE International Conference on Image Processing (ICIP)_ , 2017, pp. 3645–3649.
  * [31] P. Chu, J. Wang, Q. You, H. Ling, and Z. Liu, “Transmot: Spatial-temporal graph transformer for multiple object tracking,” in _2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2023, pp. 4859–4869.
  * [32] C. Kim, L. Fuxin, M. Alotaibi, and J. M. Rehg, “Discriminative appearance modeling with multi-track pooling for real-time multi-object tracking,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 9548–9557.
  * [33] X. Weng, J. Wang, D. Held, and K. Kitani, “3d multi-object tracking: A baseline and new evaluation metrics,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 10 359–10 366.
  * [34] Z. Pang, Z. Li, and N. Wang, “Simpletrack: Understanding and rethinking 3d multi-object tracking,” in _Computer Vision – ECCV 2022 Workshops_ , L. Karlinsky, T. Michaeli, and K. Nishino, Eds. Cham, Switzerland: Springer Nature Switzerland, 2023, pp. 680–696.
  * [35] N. Müller, Y.-S. Wong, N. J. Mitra, A. Dai, and M. Nießner, “Seeing behind objects for 3d multi-object tracking in rgb-d sequences,” in _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2021, pp. 6067–6076.
  * [36] A. Rosinol, A. Violette, M. Abate, N. Hughes, Y. Chang, J. Shi, A. Gupta, and L. Carlone, “Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs,” _The International Journal of Robotics Research_ , vol. 40, no. 12-14, pp. 1510–1546, 2021.
  * [37] S. Minaee, Y. Y. Boykov, F. Porikli, A. J. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image Segmentation Using Deep Learning: A Survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [38] R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. Kelly, and A. J. Davison, “SLAM++: Simultaneous Localisation and Mapping at the Level of Objects,” in _IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2013, pp. 1352–1359.
  * [39] J. Mccormac, R. Clark, M. Bloesch, A. Davison, and S. Leutenegger, “Fusion++: Volumetric Object-Level SLAM,” in _International Conference on 3D Vision (3DV)_ , 2018, pp. 32–41.
  * [40] F. Poux and R. Billen, “Voxel-Based 3D Point Cloud Semantic Segmentation: Unsupervised Geometric and Relationship Featuring vs Deep Learning Methods,” _ISPRS International Journal of Geo-Information_ , vol. 8, 2019.
  * [41] E. Langer, T. Patten, and M. Vincze, “Where Does It Belong? Autonomous Object Mapping in Open-World Settings,” _Frontiers in Robotics and AI_ , vol. 9, 2022.
  * [42] Y. Chen, S. Huang, T. Yuan, Y. Zhu, S. Qi, and S.-C. Zhu, “Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 8647–8656.
  * [43] J. Suchan and M. Bhatt, “Commonsense Scene Semantics for Cognitive Robotics: Towards Grounding Embodied Visuo-Locomotive Interactions,” in _IEEE International Conference on Computer Vision Workshops (ICCVW)_ , 2017, pp. 742–750.
  * [44] R. Ambrus, S. Claici, and A. Wendt, “Automatic Room Segmentation From Unstructured 3-D Data of Indoor Environments,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 2, pp. 749–756, 2017.
  * [45] C.-Y. Lee, H. Lee, I. Hwang, and B.-T. Zhang, “Visual Perception Framework for an Intelligent Mobile Robot,” in _17th International Conference on Ubiquitous Robots (UR)_ , 2020, pp. 612–616.
  * [46] N. Hughes, Y. Chang, and L. Carlone, “Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization,” in _Robotics: Science and Systems (RSS)_ , 2022.
  * [47] M. Beetz, D. Beßler, A. Haidu, M. Pomarlan, A. Bozcuoglu, and G. Bartels, “Know Rob 2.0 — A 2nd Generation Knowledge Processing Framework for Cognition-Enabled Robotic Agents,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 512–519.
  * [48] C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in _IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023.
  * [49] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam, “Clip-fields: Weakly supervised semantic fields for robotic memory,” in _Workshop on Language and Robotics at CoRL_ , 2022.
  * [50] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Songothers, “Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * [51] J. Wyatt, A. Aydemir, M. Brenner, M. Hanheide, N. Hawes, P. Jensfelt, M. Kristan, G.-J. Kruijff, P. Lison, A. Pronobis, K. Sjöö, A. Vrecko, H. Zender, M. Zillich, and D. Skočaj, “Self-Understanding and Self-Extension: A Systems and Representational Approach,” _IEEE Transactions on Autonomous Mental Development_ , vol. 2, pp. 282–303, 2011.
  * [52] J. Dong, Y. Cong, G. Sun, and T. Zhang, “Lifelong Robotic Visual-Tactile Perception Learning,” _Pattern Recognition_ , vol. 121, 2022.
  * [53] M. Ward, G. Grinstein, and D. Keim, _Interactive Data Visualization: Foundations, Techniques, and Applications_ , 2nd ed. Wellesley, USA: A. K. Peters, Ltd., 2021.
  * [54] U. Neisser, “Without Perception, there is no Knowledge: Implications for Artificial Intelligence,” in _Natural and Artificial Minds_ , R. G. Burton, Ed. SUNY Press, 1993, pp. 147–164.
  * [55] J. Wolfe, N. Klempen, and K. Dahlen, “Postattentive vision,” _Journal of Experimental Psychology: Human Perception and Performance_ , vol. 26, no. 2, pp. 693–716, 2000.
  * [56] D. Ripley and T. Politzer, “Vision Disturbance after TBI,” _NeuroRehabilitation_ , vol. 27, no. 3, 12 2010.
  * [57] K. J. de Jesus, H. J. Kobs, A. R. Cukla, M. A. de Souza Leite Cuadros, and D. F. T. Gamarra, “Comparison of visual slam algorithms orb-slam2, rtab-map and sptam in internal and external environments with ros,” in _Latin American Robotics Symposium (LARS), Brazilian Symposium on Robotics (SBR), and Workshop on Robotics in Education (WRE)_ , 2021, pp. 216–221.
  * [58] R. B. Rusu and S. Cousins, “3D is here: Point Cloud Library (PCL),” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2011, pp. 1–4, link to Library: https://pointclouds.org/.
  * [59] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision – ECCV 2014_. Cham, Switzerland: Springer International Publishing, 2014, p. 740–755.
  * [60] C. Li, L. Li, H. Jiang, K. Weng, Y. Geng, L. Li, Z. Ke, Q. Li, M. Cheng, W. Nie, Y. Li, B. Zhang, Y. Liang, L. Zhou, X. Xu, X. Chu, X. Wei, and X. Wei, _YOLOv6: A Single-Stage Object Detection Framework for Industrial Applications_. arXiv, 2022.
  * [61] H. W. Kuhn and B. Yaw, “The Hungarian Method for the Assignment Problem,” _Naval Res. Logist. Quart_ , pp. 83–97, 1955.
  * [62] B. Hommel, J. Gehrke, and L. Knuf, “Hierarchical coding in the perception and memory of spatial layouts,” _Psychological Research_ , vol. 64, no. 1, pp. 1–10, 2000.
  * [63] J. Wagemans and R. Kimchi, _The Perception of Hierarchical Structure_. Oxford, UK: Oxford University Press, 2014.
  * [64] F. Graf, R. Bormann, F. Jordan, W. Baum, W. Kraus, and B. Graf, “Cognitive Intelligence for Professional Cleaning Robots,” _54th International Symposium on Robotics_ , pp. 1–6, 2022.
  * [65] L. Kunze, H. Karaoguz, J. Young, F. Jovan, J. Folkesson, P. Jensfelt, and N. Hawes, “SOMA: A Framework for Understanding Change in Everyday Environments Using Semantic Object Maps,” in _Conference on Artificial Intelligence (AAAI-18)_ , 2018, pp. 47–54.
  * [66] C. Galindo, A. Saffiotti, S. Coradeschi, P. Buschka, J.-A. Fernández-Madrigal, and J. Gonzalez, “Multi-Hierarchical Semantic Maps for Mobile Robotics,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2005, pp. 2278–2283.
  * [67] M. Bajones, D. Wolf, J. Prankl, and M. Vincze, “Where to Look First? Behaviour Control for Fetch-and-Carry Missions of Service Robots,” in _Proceedings of the Austrian Robotics Workshop 2014_ , 10 2014, pp. 115–120. [Online]. Available: http://hdl.handle.net/20.500.12708/74395
  * [68] C. Odabasi, F. Graf, J. Lindermayr, M. Patel, S. D. Baumgarten, and B. Graf, “Refilling Water Bottles in Elderly Care Homes With the Help of a Safe Service Robot,” in _Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction_ , ser. HRI ’22. IEEE Press, 2022, pp. 101–110.
  * [69] T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Y. Asahara, and K. Murase, “Development of human support robot as the research platform of a domestic mobile manipulator,” _ROBOMECH Journal_ , vol. 6, pp. 7675–7682, 04 2019.
  * [70] Amazon Robotics, “Github repositories aws robotics,” 2023. [Online]. Available: https://github.com/aws-robotics
  * [71] R. Kittmann, T. Fröhlich, J. Schäfer, U. Reiser, F. Weißhardt, and A. Haug, “Let me Introduce Myself: I am Care-O-bot 4, a Gentleman Robot,” in _Mensch und Computer 2015 – Tagungsband_ , M. Pielot, S. Diefenbach, and N. Henze, Eds. Berlin, Germany: De Gruyter, 2015, pp. 223–232.
  * [72] D. Helbing and P. Molnár, “Social Force Model for Pedestrian Dynamics,” _Physical Review E_ , vol. 51, no. 5, pp. 4282–4286, 05 1995.
  * [73] F. Graf, c. Odabaşi, T. Jacobs, B. Graf, and T. Födisch, “MobiKa - Low-Cost Mobile Robot for Human-Robot Interaction,” in _IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE Press, 2019, pp. 1–6.
  * [74] R. Bormann, F. Jordan, J. Hampp, and M. Hägele, “Indoor coverage path planning: Survey, implementation, analysis,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 1718–1725.

|  Florenz Graf received his M.S. in Mechanical Engineering from Leibniz
University Hannover, Germany. Since 2017, he has been working on scene
perception as a research associate within the Department of Robot and
Assistive Systems at Fraunhofer IPA in Stuttgart, Germany. He is currently
pursuing his Ph.D. degree at the University of Stuttgart, Germany. His
research interests include scene perception and human-robot interaction for
mobile robots in open-world settings.
---|---
|  Jochen Lindermayr is a researcher at Fraunhofer IPA. After graduating as
M.Sc. Electrical Engineering and Information Technology at University of
Stuttgart in 2017, he joined the Department Robot and Assistive Systems at
Fraunhofer IPA in Stuttgart, Germany. He focuses on computer vision and
machine learning for mobile service robot applications. Currently, he is
pursuing his Ph.D. degree at the University of Stuttgart, Germany.
---|---
|  Birgit Graf received her diploma in Computer Science from Stuttgart
University, Germany in 1999 and completed her PhD on “An Adaptive Guidance
System for Robotic Walking Aids” in 2008. She is manager of the Assistive
Robotics Group at Fraunhofer IPA, which develops robotic solutions for health
care applications. Her research interests include the analysis of user needs
and automation potentials in residential care facilities or hospitals and the
design of robotic solutions suited to improve working conditions of the staff
and quality of care.
---|---
|  Werner Kraus received his degree as Diplom-Ingenieur in mechanical
engineering from the Karlsruhe Institute of Technology in 2011. Since then, he
is with Fraunhofer IPA in different positions and received his PhD on force-
control of cable-driven parallel robots in 2015 from the University of
Stuttgart. Since June 2019, he is head of department robot and assistive
systems. His research interests include: industrial and service robots,
cognitive robotics, control and applications of cable-driven parallel robots.
---|---
|  Marco Huber received his diploma, Ph.D., and habilitation degrees in
computer science from the Karlsruhe Institute of Technology (KIT), Germany, in
2006, 2009, and 2015, respectively. From June 2009 to May 2011, he was leading
a research group at the Fraunhofer IOSB, Karlsruhe, Germany. He then took on
various management positions in the industry until September 2018. Since
October 2018 he is full professor with the University of Stuttgart. At the
same time, he is director of the Center for Cyber Cognitive Intelligence (CCI)
and of the Department for Image and Signal Processing with Fraunhofer IPA in
Stuttgart, Germany. His research interests include machine learning, planning
and decision-making, image processing, and robotics.
---|---