#  Low-Cost Scene Modeling using a Density Function Improves Segmentation
Performance

Vivek Sharma⋄⋆, Şule Yildirim-Yayilgan⋆, and Luc Van Gool⋄∓ ⋄ ESAT-PSI,
VISICS, iMinds, KU Leuven, Kasteelpark Arenberg 10 - box 2441, 3001, Leuven,
Belgium, {vivek.sharma<EMAIL_ADDRESS>Computer Vision
Laboratory, ETH Zürich, Sternwartstrasse 7, ETH Zentrum, CH - 8092 Zürich,
Switzerland<EMAIL_ADDRESS>Faculty of Computer Science and Media
Technology, Norwegian University of Science and Technology in Gjøvik, Post
Box. 191, 2802 Gjøvik, Norway<EMAIL_ADDRESS>
###### Abstract

We propose a low cost and effective way to combine a free simulation software
and free CAD models for modeling human-object interaction in order to improve
human & object segmentation. It is intended for research scenarios related to
safe human-robot collaboration (SHRC) and interaction (SHRI) in the industrial
domain. The task of human and object modeling has been used for detecting
activity, and for inferring and predicting actions, different from those
works, we do human and object modeling in order to learn interactions in RGB-D
data for improving segmentation. For this purpose, we define a novel density
function to model a three dimensional (3D) scene in a virtual environment
(VREP). This density function takes into account various possible
configurations of human-object and object-object relationships and
interactions governed by their affordances. Using this function, we synthesize
a large, realistic and highly varied synthetic RGB-D dataset that we use for
training. We train a random forest classifier, and the pixelwise predictions
obtained is integrated as a unary term in a pairwise conditional random fields
(CRF). Our evaluation shows that modeling these interactions improves
segmentation performance by $\sim$7% in mean average precision and recall over
state-of-the-art methods that ignore these interactions in real-world data.
Our approach is computationally efficient, robust and can run real-time on
consumer hardware.

##  I Introduction

Nowadays, in automation industry humans and robots share a common workspace
while collaborating on tasks in real-time simultaneously. In such workspaces,
any risk of injuries to humans due to the human-robot collaboration should be
fully eliminated. For that reason, the robots should have a clear holistic
understanding of the scene including the most accurate and precise information
related to the human activities in the workspace. This is achievable only if
the robotic-system learns the possible context of human-object and object-
object arrangements, and the interactions among them. We therefore propose a
density function that captures the relationship between human-object and
object-object interactions. This function can be effectively used to train
methods to achieve improved segmentation performance on real-world data.
Hence, showing its potential for research scenarios related to safe human-
robot collaboration (SHRC) and interaction (SHRI) in the industrial domain.
Optimized modeling for scene understanding workflow eases to learn such
contexts. Objects in real-world interact with each other, so it is very
meaningful to capture this interaction for accurate and precise predictions of
actions. This is only possible if the training algorithm (e.g. classifier) has
learnt well the underlying physical interaction and reasoning for better
predictions. In order to analyse and estimate the scene space in real-world,
it is essential to model a synthetic scene relevant to the real-world in a
virtual environment for analysis and dataset creation.

Figure 1: Top-view segmentation results for real-world depth data, obtained
from KINECT sensor installed at a height of 3.5 meters. (From bottom:) The
first level of the hierarchy shows the test depth map, the second and third
levels show the predictions obtained from RDF classifier and the CRF modeling.

In contrast to the past, now we have relatively cheap RGB-D cameras, publicly
available free simulation softwares, free 3D CAD models, low priced consumer
hardware with computational resources to allow graphics support. In this
paper, we keep the cost of hardware and software to barely minimum and focus
on achieving an appropriate and efficient real-time segmentation. We use
rendering simulation software (VREP [5]) which is freely available. We use a
skeletal tracking system which is based on a multi-sensor KINECT setup instead
of using a very expensive marker based motion capture system. There is minimal
manual intervention for generating very large and realistic synthetic scenes
by using the rendering software. Also by using the rendering software we can
generate fully automatic ground-truth labeling for free.

In this paper, our goal is to learn the interaction between objects while
recognizing objects. This way there is a possibility to infer the scene and
predict the actions based on the object modeling. This modeling information is
beneficial111Fraunhofer Institute for Factory Operation and Automation IFF.
http://www.iff.fraunhofer.de/en/business-units/robotic-systems/research/human-
robot-interaction.html, 2015: in aviation/automobile industry for integrating
aircraft/automobile components, in manufacturing operations carried hand-in-
hand with humans, for reliable collision detection, and in healthcare and
medical industry for facilitating minimally-invasive-surgery.

For demonstration, we install a RGB-D sensor on the ceiling at the height of
3.5 meters from the ground (see Fig. 1). The camera observes our robotic
workspace and the viewed scene is examined for analysis using depth data. The
depth measurements are directly processed to provide an accurate and spatially
resolved information about the object classes in real-time from top-view. Our
main contributions are as follows:

  * •

While most previous works consider modeling human and object interactions to
infer and predict actions. We propose to learn these interactions in RGB-D
data in order to improve segmentation which can be useful for SHRI and SHRC
scenarios.

  * •

We propose a low cost solution composed by a publicly available free rendering
simulation software, cheap RGB-D cameras, and free CAD models taken from the
internet. With these resources we can create synthetic scenes that can be used
for training and help improve segmentation performance.

  * •

We show that synthetic data generated from a density function, that governs
the human-object and object-object interactions, can be effectively used to
train methods to achieve improved segmentation performance by ${\sim}$7% in
mAP and mAR over state-of-the-art methods on real-world data.

The remainder of the paper is structured as follows. In Section II, the
related work is given. Section III, describes the proposed density function
for scene modeling. Section IV, explains the feature selection, and some
basics on RDF classifier & CRF modeling. In Section V-VI, implementation
details, data collection and the experimental evaluation are given. Finally in
Section VII, conclusions are drawn.

##  II Related Work

For vision applications, we can acquire RGB-D data using real-world sensors or
by generating synthetic data using computer graphics rendering techniques in a
3D environment. The real-world data is also possible to use for training the
system, but researchers generally avoid this because of several reasons: real-
world data generation usually requires expensive equipments (i.e. software and
hardware gadgets). In addition manual hand labeling of objects is a very-time
consuming and tedious task because precise measurement of objects is required.
Also manual ground truth labeling leads to missing out several important
information in the scene due to imperfect and error-prone human annotations.
Therefore, some vision researchers avoid using real-world data for training
vision systems. While by using the present precise rendering techniques and
simulation frameworks with modeled environment, it is possible to synthesize
accurate and realistic data, which is very close to real-world data. This
provides the flexibility to tune the elements of scene for incorporating
camera noise, lighting-effects, shadows, rotation/translation, etc. during
rendering in order to make the synthetic data as consistent as possible with
the physical real-world data.

Here, we discuss some of the previous works related to vision problems. In
[22], Neumann-Cosel et al. use synthetic data obtained from the modeled
Virtual Test Drive simulation for lane tracking in driver assistance and
active safety systems. An identical work from Haltakov et al. [8] proposes a
framework for generation of synthetic data from a realistic driving simulator
to create modeled traffic scenarios similar to real-world scenarios. A number
of works uses synthetic human data for pose and activity recognition [3, 4, 6,
10, 18, 19, 20]. In vision problems, human pose and activity recognition has
received great attention in the last decade. In [19], Shotton et al.
synthesize a large and varied dataset of human actions using a motion capture
technique. In [4, 18], the authors use a simple synthetic human body
representation in a virtual environment using a KINECT skeleton estimation for
generating synthetic human pose data. In [19], the data is generated based on
front, top, and side view, while in [4, 18], it is top view only. Both works
are applied in real-world scenarios, but neither of the models incorporate the
physical interaction of objects with other objects. Also no structural and
modeled arrangements are dealt with. Thus, their algorithm fails to recognize
human body-parts, when human body-parts overlap other objects, or are
partially occluded by other objects in the scene. In [9], Hoiem et al. discard
the object candidates that are not well supported by proper occlusion
boundaries. Yet, we know that reasoning about occlusion would definitely
improve the recognition performance. A substantial little work has been done
in the context of scene modeling.

Our approach is driven by three key objectives namely computational
efficiency, robustness and time efficiency (i.e. real-time) for industrial
applications. It further differs from Shotton et al. [19] in the following
aspects. (a) In [19], all training data were thereby synthetically generated
by using marker based motion capture (Mo-Cap) technique. Such setups are very
expensive and inaccessible to most users. On the other hand, we use a highly
optimized virtual representation of 3D human skeleton in a virtual
environment. We generate the synthetic data of human body-parts in a virtual
environment using a multi-sensor KINECT setup for skeleton tracking in real-
world [4]. In addition to human data, we also generate synthetic data for
objects as well. (b) Calibration of Mo-Cap is a tedious task in comparison to
KINECT setup. (c) Processing Mo-Cap data requires sophisticated hardware in
comparison to KINECT depth data which can be porcessed with consumer hardware.
This way computational expense is reduced.

For learning and reasoning about the physical scene, we need to capture the
relevant and meaningful information (i.e. geometry, pose, shape, occlusion,
and other attributes) and train the classifier system. The density functions
capture the relationship, physical interactions, and the context among objects
as well as the geometry of the objects. For physical scene understanding,
occlusion boundaries should be well identified, since they form a very crucial
component for depth ordering. Incorporating occlusion during synthetic scene
modeling, helps to get a better understanding of the real-world scene. In
[10], Jiang et al. model a density function based on 3D spatial features to
capture the object-object and human-object relationship in a 3D scene. They
use the density function to infer arrangements of object in a scene. In [10],
a camera is installed in the virtual environment, in the corner of the room,
where two walls and the ceiling meet, or where a wall and the ceiling meet.
Their evaluation is only limited to the virtual environment for training and
testing assessments. Similar is done in [3], where Dittrich et al. model the
density function on depth features using a top-view RGB-D sensor. Their
evaluation is also limited to their virtual environment for assessment. Our
work is inspired by top-view segmentation from Shotton et al. [19] and Sharma
et al. [18], and we define the density function in VREP [5] in order to model
object-object and human-object relationships. We use a top-view RGB-D sensor
data. However, our evaluation is not just limited to virtual environment, but
also to physical real-world scenes. In addition, our work focuses on modeling
relationships between a group of multiple 3D objects in a scene, while Shotton
et al. [19] and Sharma et al. [18] only focuses on a single 3D object in a
scene. Moreover, they do not incorporate any form of interaction modeling
(i.e. non-modeled) between humans and objects.

Some other work, similar to our work, Gupta et al. [7], Aksoy et al. [1], and
Pieropan et al. [14] use the spatial relationship to perform activities
recognition where human-object and object-object relationships are encoded.
Also some other notable work, similar to ours that uses the same idea of
output of a classifier as input of another model are like “Autocontext” by Tu
et al. [21], “Decision Tree Fields” by Nowotzin et al. [13] or “Structured
Class-Labels in Random Forests for Semantic Image Labelling” by Kontschieder
et al. [12], among others.

One major problem that arises when using synthetic data, is how to determine
if the synthesized data is realistic in comparison to real-world data. This
could be determined by evaluating the statistics of object constellation in
the synthetic and real world. A simple approach could be to apply collision
avoidance while modeling interactions between objects, but such a modeling is
not realistic as occlusions never exist in the modeled scene. Therefore, a
realistic scene with appropriate object interactions could be modeled using a
density function in order to produce good prediction results, not just for
synthetic data, but also for real-world data (see Section III and Section VI-D
for details).

##  III Dataset Generation using a Density Function

Figure 2: Demonstration of a virtual scene in VREP. Object classes from
industrial domain is considered as the subset of object constellations which
are consistent to the real-world (Source [3])

We used a Virtual Robot Experimentation Platform (VREP) [5] (Fig. 2 and Fig.
3) for modeling and dataset generation. VREP is a robotics simulator with an
embedded application for a robot in an integrated development environment
(IDE) with potential field tasks without depending physically on the actual
robot. VREP robot simulator has been used for 3D modeling and generation of
the synthetic data. VREP simulator provides a virtual environment, which is
capable of emulating realistic motion generation, translation, rotation of the
actual objects in the IDE. Each object/model can be individually controlled
via a remote API client, this makes it ambidextrous because of its distributed
control architecture. It supports C/C++, Python, Java, Lua, Matlab, Octave or
Urbi. Also it is publicly available for hobbyists, academic and research
purposes. We use 3D CAD models for object classes: chair, table, storage
obtained from website222http://www.hermanmiller.com/design-
resources/3d-models-revit/3d-models-by-product/seating.html, 2015, and plant
(domain: industrial office type) obtained from
website333http://xfrog.com/category/samples.html, 2015. These CAD models are
also publicly available for free. We created our own 3D human model, and
imported it to the virtual environment. The 3D models we use are of the file
types *.3ds and *.obj. Refer to [5] for detailed information about how to
import CAD models into the scene.

Here, we define a density function in VREP for scene modeling, which includes:
spatial layout of object, object pose, object orientation, object arrangement,
object interaction and relationships between object classes i.e. none,
partial, and full occlusion. The density function captures the relationships
between human-object ($H-O$) and object-object ($O-O$) arrangements and the
interactions among them. This allows us to generate a consistent synthetic
dataset with physical real-world scenarios. Fig. 4 shows our synthesized
dataset using the density function.

A scene is denoted by $S$, and it contains industrial objects
$O=\\{O_{1},O_{2},...,O_{n}\\}$ placed in the scene. The density function
captures the relationship of human and industrial objects in the scene that is
similar to real-world. This helps to improve the segmentation performance in
real-world scenarios. The human object class is denoted by $H$ with variable
poses, specified by spatial joint location and activity. The density function
$\Phi$ capturing the context of human-object and object-object relationships
in a scene is defined as:

$\Phi(S)=\Psi(H,O;\theta)\Psi(O,O;\theta),$ (1)

where $\theta$ is threshold. We chose 4 industrial objects (i.e. chair, plant,
table, and storage) based on an industrial environment, and 6 localized human
body-parts of the human as object classes (i.e. head, body, upper-arm, lower-
arm, hand and legs). In order to illustrate the scene well, the density
function describing the human-object and object-object relationships is
defined as:

$\begin{split}\Psi(H,O;\theta)=\psi(H_{h})\psi(H_{p})\psi(H_{pos})\psi(H_{ori})\psi(O_{h})\\\
\psi(O_{pos})\psi(O_{ori})\psi((H,O)_{\theta})\psi((H,O)_{rel})\end{split}$
(2)
$\begin{split}\Psi(O,O;\theta)=\psi(O_{h})\psi(O_{pos})\psi(O_{ori})\psi((O,O)_{\theta})\\\
\psi((O,O)_{rel})\end{split}$ (3)

The terms used in the equation 2 and 3 are explained below.

Figure 3: Schematic layout of our proposed segmentation system incorporating:
scene modeling using density function in a virtual environment for synthetic
data generation; define feature space and correspondingly feature selection
and extraction for RDF classifier training; finally CRF modeling.

#### Height ($h$)

$\psi(H_{h})$ is the candidate parameter for human height, as not all humans
are of same body proportions i.e. height and breadth. We set a scaling factor
for variable human heights $\\{h_{1},h_{2},h_{3},...,h_{n}\\}$, that maintains
a uniform correlation between height and breadth of 3D human model with the
real-world human. Our 3D human model is modeled on a set of 173 spheres
arranged according to the skeleton estimate. We record the real-world human
choreographies via KINECT skeleton tracking from a calibrated multi-sensor
setup. We use the stored skeletons for modeling the 3D human model in the
virtual environment. The simple scaling of recorded skeleton is a vector based
on body proportions applied as:

$SC_{scaled}=\beta\cdot SC_{original}=\\{\beta\cdot SC_{1},\beta\cdot
SC_{2},...,\beta\cdot SC_{m}\\}$ (4)

where $SC$ is the estimated skeleton setups, $\beta$ being a fixed scaling
factor ranging between [$\beta_{min}$ , $\beta_{max}$ ], which maintains a
huge variation in body proportions of the 3D human model. This measure
facilitates a proper correspondence in the generation of the synthetic human
data in approximate relevance to the real-world human. The human height ranges
between 160-190 cm. $\psi(O_{h})$ is the candidate parameter for industrial
objects height. For table, chair, plant, and storage, standard industrial
based heights of object instances were chosen. Table, chair, storage range is
between 70-90 cm, and plant ranges between 15-35 cm. The probability density
function for $\psi(H_{h})$ is uniformly distributed on the interval
$h\in[h_{1}=160,h_{2}=190]$ and $\psi(O_{h})$ is uniformly distributed on the
interval $h\in[h_{1}=70,h_{2}=90]$.

#### Pose ($p$)

$\psi(H_{p})$ is the candidate parameter for human pose. The human poses and
appearances include: sitting, standing, stretching, walking, working, dancing,
bending, bowing, swinging, boxing, and tilting with combinations of single and
both arms, angled arms, and other combinations. The pre-defined set of poses
have one-to-one correspondence to a set of candidate scaling parameter.

#### Position or translation ($pos$)

$\psi(H_{pos})$ and $\psi(O_{pos})$ are the candidate parameters for human and
object positions in the scene respectively. There is a preference of human and
objects to be placed with reference to each other and translated in the whole
parent scene uniformly at random. We incorporate maximum randomization as is
the case in real-world. The probability density function for $\psi(H_{pos})$
and $\psi(O_{pos})$ is uniformly distributed on the interval $pos\in[w,h]$ and
is given by $1/(w\times h)$, where $w$ resembles width and $h$ resembles
height of the parent scene.

#### Orientation or rotation ($ori$)

$\psi(H_{ori})$ and $\psi(O_{ori})$ are the candidate parameters for human and
object orientations in the scene respectively. The rotation ranges between
$0-{360}^{{}^{\circ}}$ about the vertical axis at random. The probability
density functions for $\psi(H_{ori})$ and $\psi(O_{ori})$ are uniformly
distributed on the interval $ori\in[ori_{1}=0,ori_{2}=360]$.

Figure 4:  Training dataset. (Top) Ground truth labels of synthetic depth data
(Bottom) generated using a density function with a synthetic KINECT sensor.
The captured dataset shows high variability of the spatial layout of object,
object pose, object orientation, object arrangement, object interaction and
relationships between object classes. Synthetic depth frames are with additive
white Gaussian noise.

#### Threshold or distance function ($\theta$)

$\psi((H,O)_{\theta})$ and $\psi((O,O)_{\theta})$ are the candidate parameters
for defining the threshold of preferred occlusion of boundaries for $H-O$ and
$O-O$ classes respectively. This parameter encodes the distance upto what
level the $H-O$ and $O-O$ classes would overlap based on the Euclidean
thresholded distance ($\theta^{{}^{\prime}}$). The probability density
functions for $\psi(H_{h})$ and $\psi(O_{h})$ are uniformly distributed on the
interval $\theta\in[\theta_{1}=0,\theta_{2}=\theta^{{}^{\prime}}]$.
$\theta^{{}^{\prime}}$ can be set to different values for $H-O$ and $O-O$
interactions, though in our case we use $\theta^{{}^{\prime}}=30$ % for both
cases. See “Comparison of Models” in Section VI-E for more details.

#### Relationship ($rel$)

$\psi((H,O)_{rel})$ and $\psi((O,O)_{rel})$ are the terms that model the
relationships for $H-O$ and $O-O$ classes. Relationships like: collision
between objects (e.g. 2 tables attached together), objects occluded by one
another (e.g. a storage placed below a table), objects partially occluded
(e.g. partial occlusion of a chair by table), objects placed over one-another
(e.g. a plant pot placed on top of a table). Other interactions such as a
person may have his hand placed on the table, holding the chair handle,
sitting on a chair and so on.

##  IV Algorithm

Fig. 3 shows the schematic layout of our proposed hierarchical segmentation
system. The synthesized RGB-D training dataset incorporates modeled $H-O$ and
$O-O$ relationships and interactions obtained using a density function based
scene modeling. The first step performed is sampling i.e. number of frames and
features-per-object-class are chosen at random for training an RDF classifier.
Next, individual extracted features corresponding to an object class are
passed to the RDF classifier. RDF returns a trained classification forest in
which each leaf node represents the class prediction of a tree. Now a test
depth map of the scene obtained from the real-world KINECT sensor is given as
an input to the trained classification forest. The result obtained is a
pixelwise object class labeling. The likelihood of an object label assigned to
a pixel obtained from the classifier is integrated as a unary term in the
pairwise CRF. The pairwise term is obtained from the Potts model [15]. Thus,
this labeling problem modeled on pairwise CRF is optimized using
$\alpha$-expansion built on graph cuts [2] for finding a global optimal
labeling (i.e. segmentation).

###  IV-A Additive white gaussian noise in synthetic depth data

The synthetic depth data from the virtual KINECT sensor in the scene usually
contains no noise. The capturing of the real-world with sensors usually is a
combination of ideal signal and noise. The generation of noise may be because
of vast number of sources, variations in the detector sensitivity,
environmental variations, the discrete nature of radiation, transmission or
quantization errors, geometry dependant missing data due to shadows in the IR
image cast by the object, that there are discretization errors, and noise
increases with increasing depth, all of which add up to a noise model. Because
of the noise in the real-world data, and to cope with unseen data samples in
the testing step more robustly, meaning to increase the generalization ability
of the trained classifier, we add additive Gaussian white noise to the depth
values (Fig. 4).

###  IV-B Feature Selection

We adopt the same depth features as specified in [4]. For a given pixel
location $s$ of an object sample $O$ from its depth map, we denote its depth
value as a mapping $d_{O}(s)$, and design a feature $f_{O}(s)$ by using two 2D
offsets positions $u$, $v$ from $s$:

$f_{O}(s)=d_{O}(s+u)-d_{O}(s+v)$ (5)

The feature is depth-invariant. We use a rectangular patch for extraction of
depth values from an object sample. We compute a fixed number of 300 features
for each object sample.

###  IV-C Classifier

The choice of selecting a good discriminative classifier is independent of the
preceding sections, because our goal is to show that modeling using density
function helps to improve the real-world segmentation performance. Here, we
use a Random Decision Forests (RDF) classifier for pixelwise object
classification. RDF is an ensemble of $t$ binary decision trees,
$t\in\\{1,...,T\\}$. In a decision tree, each of the nodes is associated with
a feature response function (i.e. weak learner or split function). The feature
response function plays the most critical and crucial role in both training
and testing of random forests. In the internal node, selection of parameters
of the split function takes place associated with each split node by
optimizing a chosen objective function defined on training data set. The
objective function is based on maximizing the information gain. The geometric
primitives of the split function are used to partition the data points. We
employ two geometric primitives, axis-aligned and linear feature response
function. We also use bagging and randomized node optimization, that injects
randomness into the trees during the tree training. The stopping criteria used
for tree growth were maximum number of levels that a tree could reach, minimum
information gain that the nodes could have, child nodes having the same
entropy value.

###  IV-D Conditional Random Fields

The energy of the pairwise CRFs used for object class segmentation can be
defined as a sum of unary and pairwise potential terms, given as:

$E(\mathbf{x})=\sum_{i\in\upsilon}\varphi_{i}(p_{i})+\sum_{i\in\upsilon,j\in\eta}\varphi_{i,j}(p_{i},p_{j}),$
(6)

where $\upsilon$ corresponds to the vertex set of a 2D grid with each vertex
corresponding to pixel $p$ in the image and $\eta$ neighborhood of the pixels
$\forall i,j\in\upsilon$. $\mathbf{x}$ is an arbitrary labeling. The unary
potential $\varphi_{i}(p_{i})$ term in the CRF energy is a data term. It is
the likelihood of the object label assigned to pixel $i$. Here, the unary term
is obtained from the RDF classifier for each pixel belonging to the object
class. The pairwise potential $\varphi_{i,j}(p_{i},p_{j})$ term in the CRF
energy encodes a smoothness prior and encourages smooth segmentation by
favoring neighboring pixels in a 2D grid having the same label [2]. It takes
the form of a Potts model [15], which is efficiently minimized by
$\alpha$-expansion [2] built on graph cuts.

The predictions obtained from pixelwise labeling using RDF is very efficient
and when modeled on a CRF to minimize the misclassification of labels assigned
to the depth measurements, makes the algorithm more “robust” and even
contributes to the good results.

$0$$20$$40$$60$$80$$100$$0.4$$0.6$$0.8$$\sigma$$mAR$

$100$$200$$300$$400$$500$$600$$0.7$$0.75$$0.8$$0.85$$PC$$mAR$LinearAxis-
Aligned

$0$$1{,}000$$2{,}000$$3{,}000$$4{,}000$$5{,}000$$0.65$$0.7$$0.75$$0.8$$F$$mAR$Ours-
ModeledSOA-Non-Modeled

$0$$20$$40$$60$$80$$100$$0.3$$0.4$$0.5$$0.6$$0.7$$\sigma$$mAP$

$100$$200$$300$$400$$500$$600$$0.56$$0.58$$0.6$$0.62$$0.64$$0.66$$PC$$mAP$LinearAxis-
Aligned

$0$$1{,}000$$2{,}000$$3{,}000$$4{,}000$$5{,}000$$0.5$$0.6$$0.7$$F$$mAP$Ours-
ModeledSOA-Non-Modeled

Figure 5:  Evaluation results. (Column 1:) effect of the additive white
Gaussian noise ($\sigma$) to the synthetic training depth data in order to
exhibit real-world camera noises on the average recall ($mAR$) and precision
($mAP$) measures of pixelwise object class segmentation. (Column 2:)
comparison of the linear and axis-aligned feature response function, using
$mAR$ and $mAP$ as a function of the number of features extracted ($PC$).
(Column 3:) comparison of the modeled and non-modeled training dataset, using
$mAR$ and $mAP$ as a function of number of training synthetic depth frames
($F$). For the evaluation, a random number of 65 real-world test depth maps
were used.

##  V Implementation Details

For our implementation, we used a desktop with Intel i7-2600K CPU at 3.40GHZ,
4GB RAM with an operating system installed on a SSD. A fixed rectangular
feature patch of size ($w,h$)=($64,64$) is used for the whole training
process. Each frame generated from a KINECT sensor is of size $640\times 480$.

For training our RDF classifier, we used the following parameter setup: (a)
tree depth (D=19); (b) number of trees in a forest (T=5); (c) number of
features extracted from a rectangular patch (PC=300); (d) number of synthetic
scenes (i.e. depth frames) used for training (F=1600/tree); (e) 100 thresholds
per feature and 100 feature response samples in the node optimization, along
with bagging approach.

##  VI Experimental Evaluation

The goal of this work is to improve the segmentation performance in real-world
scenarios, for this reason we use only real-world data, a random number of 65
test depth maps for evaluation. We acquire RGB-D data from a real-world RGB-D
sensor, which is placed on the ceiling in the center of our shared workspace.
See Fig. 7 (column 1) for real-world depth maps.

We evaluate the impact on the segmentation performance for three cases:
(Section VI-B) what amount of noise should be added to synthetic data to make
the synthetic data consistent with real-world; (Section VI-C) effect of the
split function of RDF on classification performance; (Section VI-D) comparison
between modeled and non-modeled training dataset (i.e. with and without
density function) through training an RDF classifier, when evaluated on real-
world.

In order to evaluate the effect of modeling human & object interactions, we
show two variants of our method: a modeled version that takes into account
these interactions and non-modeled version that ignores them. In addition, we
compare against state-of-the-art approaches [18, 19]. Note that, all our
experiments are conducted on real-world test data (see Section VI-D and
Section VI-E). For the performance measures, we generate mean average of
recall (mAR) and precision (mAP) for each single object class and a combined
average of all classes.

###  VI-A Data Collection

A scene is a single frame where there is a single object or a combination of
multiple 3D objects arranged and oriented in a particular configuration based
on the density function. The scenes are composed of complex configurations of
humans and objects. We generated an extensive synthetic dataset of RGB-D data
using the 3D scene models of 4 different rooms based on industrial workspace
and office domain with a total of 10 object classes in the virtual environment
[5] (see Fig. 4). In case of training we use solely synthetic depth data, and
in case of the testing data for the evaluation step, we use synthetic and real
world depth data. The object classes are: human body-parts (head, body, upper-
arm, lower-arm, hand, and legs), table, chair, plant, and storage. We
generated the synthetic human body-parts data via using a real-world KINECT
multi-sensor setup with a 3D multi-color human model in the virtual
environment. For more detailed information about human data generation, refer
to previous work [4, 17]. We synthesized a dataset of 20,000 frames using
density function for $H-O$ and $O-O$ relationships. The synthesized scene has
a depth map ranging between $0-3.5$ meters. Each scene is assigned a set of
human, chair, plant, storage, and table object class. Each frame is a
stationary image, having no kinematic or temporal information. We use the 3D
models of objects based on industrial workspace, with 4 instances for each
object class: chair with and without handles; rectangular and round table;
flowers and plants within pot; shelves and wardrobes. The modeled scenes are
commonly seen in the real-world. In Section VI-D, we show that training with
synthetic modeled data is sufficient for the generalization of the classifier
in regard to real-world data.

###  VI-B Noise Evaluation

Additive white Gaussian noise using a standard deviation ($\sigma$) was added
to synthetic training depth data for compensating with noise in real-world, by
matching the camera output as close as possible, and have an approximately
good segmentation. The results in Fig. 5 (column 1) implies that using this
setting for generating synthetic data results in the best performance, when
the synthetic depth frames with additive white Gaussian noise using a standard
deviation of 15 cm is used.

###  VI-C Feature Response Evaluation

The results in Fig. 5 (column 2) show that most of the gains occur for linear
feature response and is $\sim$10% more than the axis-aligned. The qualitative
results obtained from axis-aligned feature response produced overconfident
predictions, while for linear feature response it was also more visually
pleasing results.

###  VI-D Comparison of Models

In order to encode a consistent modeling of $H-O$ and $O-O$ interactions, we
preferred occlusion of boundaries for maintaining as accurate as possible
relevance with the real-world. This relationship between objects employ an
euclidean thresholded distance $\theta^{{}^{\prime}}$. We used a rank based
approach for up to what level the occlusion should be allowed,
$\theta^{{}^{\prime}}$={0, 5, 10, 15, 20, 30, 50} in %. We found out that as
$\theta^{{}^{\prime}}$ is increased, the testing predictions increases
monotonically up to $\theta^{{}^{\prime}}$=30, and just after that there is a
sharp descent in both quantitative and qualitative performance. Therefore, we
preferred $\theta^{{}^{\prime}}$=30% for overlapping of boundaries, and our
dataset was synthesized.

As a baseline, we implemented the same state-of-the-art (SOA) pipeline used in
[18, 19] for the generation of the non-modeled dataset, based on top-view (see
Fig. 6). In a non-modeled scene, there is only a single 3D object in a scene
with a particular configuration. In our case, there are multiple 3D objects in
a scene which are modeled using a density function. We compare the results
obtained with our modeled dataset against the results obtained by their non-
modeled dataset using a number of frames for training the classifier. The
results depicted in Fig. 5 (column 3) thereby show a steep ascent with a
higher $mAR$ and $mAP$ for our proposed modeling. The density function based
modeling, substantially improves the performance by $\sim$7% in $mAR$ and
$mAP$ over state-of-the-art [18, 19]. Fig. 7 (row 1) shows the reduced
misclassification around the border of the human body with better
classification of the human-hand placed on the table and the chair for modeled
case.

Figure 6:  Non modeled training dataset. Ground truth labels and its
corresponding synthetic depth data generated using the same pipeline used in
[4, 19]. TABLE I: Confusion matrix-based mean average F1-measure

F1-measure | Avg | Head | Body | UArm | LArm | Hand | Legs | Chair | Plant | Storage | Table
---|---|---|---|---|---|---|---|---|---|---|---
$CRF_{Non-Modeled}$ [18, 19] | 0.76 | 0.90 | 0.71 | 0.73 | 0.65 | 0.69 | 0.48 | 0.85 | 0.78 | 0.90 | 0.91
$CRF_{Modeled}$ | 0.84 | 0.96 | 0.84 | 0.79 | 0.70 | 0.79 | 0.52 | 0.93 | 0.90 | 0.98 | 0.97

Figure 7:  (Row 1:) prediction results for real-world test depth data using
the modeled and non-modeled training dataset. The segmentation improvements
can be seen in the modeled case: the misclassification around the border of
the human has diminished significantly; the human hand placed on the table and
the chair are classified well with reduced mislabeling. (Row 2-3:) shows the
predictions obtained from the RDF classifier and the CRF modeling.

###  VI-E Demonstration

We use the integrated system for segmentation in our shared robotic workspace
in real-time using real-world depth data. Fig. 7 (row 2-3) shows the
prediction results for the proposed CRF model using real-world test data.
Table I shows that the F1-measure for the CRF modeled improves by $\sim$8%
over the non-modeled ones [18, 19]. For training the model with our optimized
RDF parameter setup takes 43 minutes, and testing the model takes 34 ms. Each
block of our pipeline (i.e. scene modeling using a density function, fine
tuning the additive noise in the synthetic data, selection of the linear
feature response for the RDF tree training, modeling the labeling problem on a
CRF to minimise the misclassification of labels) contributes to the good
results, and plays a crucial role in improving the segmentation performance.
The resulting system is computationally efficient, robust and supports real-
time for our targeted application.

##  VII Conclusion

In this paper we described an inexpensive and effective way to model the
human-object and object-object interactions in order to improve segmentation
performance. For this purpose, we proposed a density function to model a 3D
scene in a virtual environment for synthesizing a dataset incorporating human-
object and object-object interactions, that is consistent with the real-world
scenarios. Our proposed density function models spatial layout of object,
object pose, object orientation, object arrangement, object interaction and
relationships between the object classes i.e. none, partial, and full
occlusion by each-other. Our experiments are based on industrial workspace and
office domain with a total of 10 object classes. Our goal of modeling scene
using density function improves the real-world scene segmentation performance
measures by $\sim$7% in mean average precision and recall over state-of-the-
art non-modeling based segmentation methods. Our integrated resulting system
is computationally efficient, robust and supports real-time for our targeted
application. In future work, we would like to extend this work towards the
semantic image understanding and image-sentence generation [11, 16] for
manufacturing and automation industry of challenging environments for safe
human-robot interaction and collaboration.

## Acknowledgment

We would like to thank José Oramas M, Michele Fenzi, Frank Dittrich and
Stephan Irgenfried for their valuable suggestions.

## References

  * [1] E. E. Aksoy, A. Abramov, F. Worgotter, and B. Dellen.  Categorizing object-action relations from semantic scene graphs.  In ICRA, 2010.
  * [2] Y. Boykov, O. Veksler, and R. Zabih.  Fast approximate energy minimization via graph cuts.  PAMI, 2001.
  * [3] F. Dittrich, S. Irgenfried, and H. Wörn.  Probabilistic modeling of real-world scenes in a virtual environment.  In International Conference on Computer Graphics Theory and Applications (GRAPP), 2015.
  * [4] F. Dittrich, V. Sharma, H. Wörn, and S. Yayilgan.  Pixelwise object class segmentation based on synthetic data using an optimized training strategy.  In ICNSC, 2014.
  * [5] M. Freese, S. P. N. Singh, F. Ozaki, and N. Matsuhira.  Virtual robot experimentation platform v-rep: A versatile 3d robot simulator.  In SIMPAR, 2010.
  * [6] V. Ganapathi, C. Plagemann, D. Koller, and S. Thrun.  Real time motion capture using a single time-of-flight camera.  In CVPR, 2010.
  * [7] A. Gupta, A. Kembhavi, and L. S. Davis.  Observing human-object interactions: Using spatial and functional compatibility for recognition.  PAMI, 2009.
  * [8] V. Haltakov, C. Unger, and S. Ilic.  Framework for generation of synthetic ground truth data for driver assistance applications.  In Pattern Recognition, Lecture Notes in Computer Science, pages 323–332. Springer Berlin Heidelberg, 2013.
  * [9] D. Hoiem, A. A. Efros, and M. Hebert.  Putting objects in perspective.  In CVPR, pages 2137–2144. IEEE Computer Society, 2006.
  * [10] Y. Jiang, M. Lim, and A. Saxena.  Learning object arrangements in 3d scenes using human context.  In ICML, 2012.
  * [11] A. Karpathy and L. Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  arXiv preprint arXiv:1412.2306, 2014.
  * [12] P. Kontschieder, S. Rota Buló, H. Bischof, and M. Pelillo.  Structured class-labels in random forests for semantic image labelling.  In ICCV, 2011.
  * [13] S. Nowozin, C. Rother, S. Bagon, T. Sharp, B. Yao, and P. Kohli.  Decision tree fields.  In ICCV, 2011.
  * [14] A. Pieropan, C. H. Ek, and H. Kjellstrom.  Functional object descriptors for human activity modeling.  In ICRA, 2013.
  * [15] R. Potts and C. Domb.  Some generalized order-disorder transformations.  Proceedings of the Cambridge Philosophical Society, 48:106, 1952\.
  * [16] M. A. Sadeghi and A. Farhadi.  Recognition using visual phrases.  In Computer Vision and Pattern Recognition (CVPR), 2011.
  * [17] V. Sharma, F. Dittrich, S. Yayilgan, and L. V. Gool.  Efficient real-time pixelwise object class labeling for safe human-robot collaboration in industrial domain.  In ICML Workshops, 2015.
  * [18] V. Sharma, F. Dittrich, S. Yayilgan, and L. Van Gool.  Improving human pose recognition accuracy using crf modeling.  In CVPR Workshops, 2015.
  * [19] J. Shotton, R. B. Girshick, A. W. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio, R. Moore, P. Kohli, A. Criminisi, A. Kipman, and A. Blake.  Efficient human pose estimation from single depth images.  PAMI, 2013.
  * [20] J. Sung, C. Ponce, B. Selman, and A. Saxena.  Human activity detection from rgbd images.  In AAAI Workshop, 2011.
  * [21] Z. Tu.  Auto-context and its application to high-level vision tasks.  In CVPR, 2008.
  * [22] K. von Neumann-Cosel, E. Roth, D. Lehmann, J. Speth, and A. Knoll.  Testing of image processing algorithms on synthetic data.  In International Conference on Software Engineering Advances, 2009\. ICSEA ’09., pages 169–172, Sept 2009.