# ParaHome: Parameterizing Everyday Home Activities
Towards 3D Generative Modeling of Human-Object Interactions

Jeonghwan Kim Jisoo Kim11footnotemark: 1 Jeonghyeon Na Hanbyul Joo
Seoul National University
<EMAIL_ADDRESS>
https://jlogkim.github.io/parahome Equal contribution

###### Abstract

To enable machines to learn how humans interact with the physical world in our
daily activities, it is crucial to provide rich data that encompasses the 3D
motion of humans as well as the motion of objects in a learnable 3D
representation. Ideally, this data should be collected in a natural setup,
capturing the authentic dynamic 3D signals during human-object interactions.
To address this challenge, we introduce the ParaHome system, designed to
capture and parameterize dynamic 3D movements of humans and objects within a
common home environment. Our system consists of a multi-view setup with 70
synchronized RGB cameras, as well as wearable motion capture devices equipped
with an IMU-based body suit and hand motion capture gloves. By leveraging the
ParaHome system, we collect a novel large-scale dataset of human-object
interaction. Notably, our dataset offers key advancement over existing
datasets in three main aspects: (1) capturing 3D body and dexterous hand
manipulation motion alongside 3D object movement within a contextual home
environment during natural activities; (2) encompassing human interaction with
multiple objects in various episodic scenarios with corresponding descriptions
in texts; (3) including articulated objects with multiple parts expressed with
parameterized articulations. Building upon our dataset, we introduce new
research tasks aimed at building a generative model for learning and
synthesizing human-object interactions in a real-world room setting.

Figure 1: Our ParaHome system enables to parameterize the detailed 3D
movements of the human body, hands, and diverse objects towards the generative
modeling of the human-object interactions

##  1 Introduction

Our daily routines involve a series of actions where we interact with various
objects in various ways. As humans, we effortlessly foresee the consequences
of our actions; for instance, pulling the refrigerator handle opens the door,
and tilting a kettle towards a cup pours water into it. As seen, there is a
clear and robust correlation between our actions and how the environment
responds, which cannot be easily understood or mimicked by machines and
robots. A key obstacle in making machines to understand such sophisticated
human-object interactions lies in the scarcity of large-scale datasets
captured in natural and casual settings, including the 3D motions of both
humans and objects occurring in causal interactions. Existing datasets
primarily focus on limited aspects of these challenges. Early approaches
capture the human motion without objects [20, 55, 30, 31, 37, 42]. More recent
approaches focus on hand-object interactions in static postures or relatively
simple and short interactions such as grasping or picking up [2, 6, 16, 27,
23, 58]. While few recent datasets introduce scenarios where humans perform
hand manipulations with articulated objects [13, 36], they still consider less
natural scenarios by capturing the interactions in a simple table setup and
short atomic actions with a single object at each time. Notably, none of the
prior datasets comprehensively capture the entire body, dexterous finger
motions, and 3D movements of diverse objects such as articulated objects,
electronic devices, and furnitures in a room-scale environment. As such,
existing generative modeling methods based on these datasets also show
limitations in their capability. Previous models either primarily focus on
human motion synthesis without considering object interactions [22, 49, 50,
21], consider relatively simple and atomic HOI between the human body and
rigid object motions [64, 48, 70, 9, 68, 60, 69, 67, 59, 34] or human action
in the scene without object articulation [5, 26, 65, 41].

To address such limitations, we present a novel capture system, named
ParaHome, along with a large-scale dataset including diverse and natural
human-object interactions. Our ParaHome system allows participants to freely
interact with multiple objects in a room environment, as shown in Fig. 1 and
Fig. 8. The primary objective of our system and datasets is to capture common
and realistic actions, including authentic signals from the detailed and
dexterous human motions at the finger level, as well as precise object
tracking and articulated movements. Obtaining such signals is challenging,
since the system should handle multiple scales (big body movements across the
room and subtle hand motions) and severe occlusions during interactions. To
achieve this goal, we design our ParaHome system by combining a multiview
camera system equipped with 70 synchronized cameras and wearable motion
capture equipment, an IMU-based motion capture suit and motion capture gloves.
The multi-camera system tracks the 3D movements of objects, capturing both
rigid and articulated motions, as well as the global rigid transformations of
the human body and hands in the camera space. The human body and hand motions
are captured using wearable motion capture equipment that is occlusion-free.
Given the fact that spatially and temporally combining heterogeneous systems
is challenging, we propose multiple hardware and algorithmic solutions to
robustly reconstruct human motions and object movements in a common 3D
coordinate system.

Leveraging our ParaHome system, we collect a large-scale human-object
interaction dataset in the home activity scenarios. The resulting data
represents detailed human-object interactions in the parametric space, via
human pose parameters and object parameters, enabling new opportunities for
human-object interaction studies. In particular, we also represent diverse
articulated motions of objects via object-specific parametric models. Our
dataset contains data from 30 participants, 22 objects, 101 scenarios, and 440
minutes of sequences in clock time. We design the capture scenario to reflect
various natural activities that are commonly observed in our daily lives,
where each participant performs interactions with multiple objects in their
own unique styles given a common verbal guidance in each scenario. All
captured data will be publicly available. Our contributions are summarized as:
(1) presenting a new home activity capture system combining multiple cameras
and wearable motion capture devices; (2) collecting a new HOI dataset
capturing both dexterous human actions and 3D movements of objects in a common
parameterized space; (3) introducing new future research tasks to understand
human-object interactions with multiple objects and sequential actions.

##  2 Related Work

3D Human Body and Hand Datasets. There has been a long history of work in
analyzing human motion or behaviors via multi-camera systems. Pioneered by the
work of Kanade et al. [32], a number of systems were subsequently proposed to
reconstruct human behaviors [40, 39, 19, 47]. Markerless-motion capture system
has been tackled by producing 3D skeletal structures over time similar to the
marker-based counterparts [15, 18, 7, 51, 3, 33, 8, 61, 4, 57, 10, 62, 14].

Capturing 3D hands is often more challenging than capturing the 3D body
motions due to the inherent severe occlusions and small scale. The early work
tackles capturing 3D hand poses only [56, 71, 66, 44, 63], which mainly
pursues to provide supervision for the hand pose estimation algorithms from
single RGB image. Due to the challenges in collecting real-world data, some
approaches also propose synthetic 3D hand dataset, including SynthHands [45],
Rendererd Hand Pose [71] and GanHand [9].

Human-Object Interaction Dataset. A direction of approaches pursue to capture
relatively natural RGB videos of hands and object interaction, including HO-3D
[23], H2O-3D [24], HOI4D [36], DexYCB [6], FreiHand [72], and Assembly101
[54]. To handle severe occlusions during interactions, often multiview systems
(from 2-8 views) are used, where either manual 2D annotations from multiview
or optimization approaches are used to reconstruct 3D hands and objects.
Another line of approaches sacrifices the naturalism in the images and uses
marker-based motion capture system [25, 58], magnetic sensors [16], or motion
capture gloves [11]. These approaches can provide more accurate 3D hand poses,
but the RGB images may not be used in training models or evaluations.
Alternatively, in OBMAN [27], a synthetic dataset for hand-object interaction
scenes is also proposed, and there exists a method in building pseudo-GT data
from in-the-wild images [72]. Some datasets also contain 6D object
annotations, along with hand annotations [23, 24, 6, 27, 2, 36, 13, 29]. Among
them, only a few datasets contain articulated objects [36, 13]. While the
early work contains the interaction from a single hand, more recent approach
considers the interaction with both hands [58, 2, 35, 24]. A few approaches,
including GRAB [58], ARCTIC [13], BEHAVE [28], ActionSense [11], InterCap [29]
consider both hand interactions and whole body captures. Notably, existing
approaches focus on subpart problems: (1) by capturing only body motions
without object motions, (2) by capturing relatively short and simple
interactions (grasping, picking up) with a single target object; (3) in a
fixed small space or a table setup.

Compared to the previous datasets [13, 1, 29], our dataset captures more
natural human-object interactions, covering: (1) both hand and body motion
captures; (2) articulated object modeling and tracking; (3) interactions with
multiple objects; (4) navigation and manipulations in a room environment; and
(5) sequential correlated actions. We summarize the key aspects of existing
dataset and ours in supplementary material.

##  3 Modeling Parameterized Episodic HOI

Figure 2: (Center) Reconstructed scene of ParaHome from top view. Pictures
adjacent to the rendering were taken from the center of the room, headed
towards the corresponding black dots in the scene. (Right) Pictures of RGB
camera, Manus motion capture gloves, and Xsens motion capture suit with
attached body markers.

To effectively model human-object interactions, we employ a parameterized 3D
space that captures the nuanced relationship between human motions and object
movements. For the human parts, we focus on the motion of the whole body and
dexterous hand movements as the essential components. Rigid objects are
represented through 6 DoF rigid motions (3D translation and orientation).
Additionally, we incorporate object-specific dynamics such as opening the
laptop or turning the knobs of a gas stove. Formally, we represent the status
of a human and the environment at time $t$ as
$\mathbf{S}(t)=\\{\mathbf{S}_{p}(t),\mathbf{S}_{e}(t)\\}$, where
$\mathbf{S}_{p}(t)$ is the current status of a human subject at time $t$, and
the environment status $\mathbf{S}_{e}(t)$ represents the current status of
surrounding objects. The human status
$\mathbf{S}_{p}(t)=\\{\mathbf{S}_{b}(t),\mathbf{S}_{lh}(t),\mathbf{S}_{rh}(t)\\}$
is composed of the body $\mathbf{S}_{b}(t)$, left-hand $\mathbf{S}_{lh}(t)$,
and right-hand parameters $\mathbf{S}_{rh}(t)$, where each of them can be
represented as mocap outputs via a global location and local joint
orientations. The environment status is represented as
$\mathbf{S}_{e}(t)=\\{\mathbf{S}^{j}_{e}(t)\\}_{j=1}^{N}$, where
$\mathbf{S}^{j}_{e}(t)$ is the status of the $j$-th object, assuming we
consider $N$ different object instances. The object status
$\mathbf{S}^{j}_{e}(t)=\\{\mathbf{l}^{j}_{e}(t),\boldsymbol{\theta}^{j}_{e}(t),\boldsymbol{\phi}^{j}_{e}(t)\\}$
is represented by 3D translation $\mathbf{l}^{j}_{e}(t)\in\mathbb{R}^{3}$, 3D
orientation $\boldsymbol{\theta}^{j}_{e}(t)\in SO(3)$, and object specific
parameters $\boldsymbol{\phi}^{j}_{e}(t)$. The length of
$\boldsymbol{\phi}^{j}_{e}(t)$ varies based on the type and the dimension of
movable parts of the objects (e.g. $\boldsymbol{\phi}^{\text{laptop}}_{e}(t)$
and $\boldsymbol{\phi}^{\text{drawer}}_{e}(t)$ contain one and two part
parameters each).

To learn the characteristic correlations between the environment
$\mathbf{S}_{e}(t)$ changes and the human actions $\mathbf{S}_{p}(t)$,
parameterizing both in a common spatio-temporal space is crucial, which
motivates us to build our ParaHome system. One direction towards generative
modeling for human-object interaction is to probabilistically model the
distributions of possible configurations of humans and objects in this
parametric 3D space: $P(\mathbf{S}(t:t+w))$, where we consider the human and
object movements in a fixed time window from $t$ to $t+w$. Intuitively, this
formulation can capture the likelihood of plausible human and object
configurations and dynamics. We can alternatively formulate it as a prediction
problem as:

$\displaystyle\mathbf{S}_{o}(t{\,:\,}t+w)=\mathcal{F}\left(\mathbf{S}_{i}(t{\,:\,}t+w)\right)$
(1)

, where the input $\mathbf{S}_{i}(t{\,:\,}t+w)$ is a subset of full states
$\mathbf{S}(t{\,:\,}t+w)$ and $\mathbf{S}_{o}(t{\,:\,}t+w)$ is the
reconstructed unseen cues predicted by the model. For instance, we can build
the model to infer 3D object movements from the human body motions, or to
infer finger motions from desired object movements. Moreover, we can also
formulate the HOI understanding as a future status forecasting task:

$\displaystyle\mathbf{S}_{o}(t+1{\,:\,}t+w)=\mathcal{F}\left(\mathbf{S}_{i}(t-w{\,:\,}t)\right).$
(2)

All these formulations can be interesting future research directions as a way
to understand and learn spatio-temporal relations of human-object interactions
in casual and natural activity setups. Crucially, these studies require a
large-scale dataset capturing the human-object interactions $\mathbf{S}(t)$ in
a natural setup. By leveraging our ParaHome system and collected dataset, we
pursue to open up new potentials for these impactful research tasks.

##  4 3D Parametric Home Capture System

Figure 3: Marker solutions to track and localize objects and its parts.

In contrast to the existing datasets [72, 23, 35, 1, 13] that captures only
subparts of human-object interactions in a limited or small-scale setup, our
ParaHome system is designed to capture all essential signals. Such signal,
$\mathbf{S}(t)=\\{\mathbf{S}_{p}(t),\mathbf{S}_{e}(t)\\}$, contains body
motion, finger motion, and 3D object movements in a residential studio
apartment. This requires solving multiple hardware and software challenges.

###  4.1 Hardware System and Architecture

Our ParaHome environment encompasses an area of 12.4$m^{2}$, as shown in Fig.
2. Capturing 3D movements of the human body, fingers, and objects during
interactions in a spacious room environment is challenging, even for the
elaborate capture systems [25, 13, 58]. In our paper, we propose a new system
by combining both a multi-camera system and an IMU-based wearable motion
capture suit and gloves as a solution. To cover the entire volume of the room
and reduce occlusion issues, we install 70 RGB industrial cameras. Also for
tracking human body and hand movements, we utilize Xsens motion suit [43] and
Manus hand gloves [38]. For further details of our system setups, please check
supplementary material.

###  4.2 3D ArUco Marker Design and Placement

For reliable tracking of objects and their articulated movements, we attach
ArUco markers [17] to all faces of the 3D cube spanning 6$cm$ inspired by
[73]. Examples of our markers are shown in Fig. 3. The major motivation of our
3D marker solution lies in its robust tracking advantage during complicated
object manipulation scenarios, where often objects are severly occluded by
hand grasping. In particular, we find attaching markers on the object surface
as in previous approaches [13] is sub-optimal, often requiring laborious
manual post-processing. We quantitatively demonstrate the strength and
robustness of our 3D marker solution in our experiments.

##  5 Data Acquisition

###  5.1 Modeling and Tracking 3D Objects

Figure 4: Scanned 3D model of objects in ParaHome system.

Computing Transforms between Object and 3D Markers. We obtain high-quality 3D
mesh scans of all objects placed in our system via an Einstar3D scanner [12].
We scan each object at least twice to reduce the unscanned areas or holes by
changing the orientations of the objects (e.g., up-side-down), and fuse the
scanned meshes via manual alignments. Objects are scaled as a metric scale (in
meters) to be consistent with our camera system space. Our scanned objects are
shown in 4. For better visibility, we attached one or multiple 3D ArUco
markers and flat-style markers to each object part. Then, object tracking is
performed by detecting and tracking the ArUco markers in every frame. The
position and orientation of each object are obtained as follows:

$\mathbf{T}_{obj}(t)=\mathbf{T}_{mar\rightarrow obj}\mathbf{T}_{mar}(t)$ (3)

where $\mathbf{T}_{mar\rightarrow obj}$ is a pre-computed fixed transformation
from the marker to the object, and $\mathbf{T}_{mar}(t)$ is a transformation
for the markers from the object canonical space to the current pose in the
camera system space at capture time $t$. $\mathbf{T}_{mar\rightarrow obj}$ is
computed manually by selecting several corresponding points between object
scan and the capture in the camera system space.

Modeling Object Articulations. Since the whole parts of an object are scanned
as a single chunk, we manually separate each rigidly moving part and build a
parameterized structure for articulated models. We assume either a revolute or
sliding joint. Both types of joints require articulation axis
$\mathbf{a}_{e}\in\mathbb{R}^{3}$ and in case it is revolute, pivot point
$\mathbf{p}_{e}\in\mathbb{R}^{3}$ additionally. Thus, we define the joint
status of $j$-th object as
$\boldsymbol{\phi}_{e}^{j}(t)=\\{\boldsymbol{\tau}_{i}^{j}\\}_{i=1}^{n}$ where
$\boldsymbol{\tau}_{i}^{j}=\\{\mathbf{a}_{e,i}^{j},\mathbf{p}_{e,i}^{j},s_{e,i}^{j}(t)\\}$.
Here, $n$ represents the number of parts and $s_{e,i}^{j}(t)\in\mathbb{R}$
denotes a relative part state (either radian for revolute joint or meter for
sliding joint) from part status in the object canonical space. Examples are
shown in Fig. 5. See supplementary material regarding the process of computing
articulation axis $\mathbf{a}_{e}^{j}$ and pivot point $\mathbf{p}_{e}^{j}$
for each object.

Capturing 3D Object Motions. We track the status of each object at time $t$,
$\mathbf{S}_{e}(t)=\\{\mathbf{l}_{e}(t),\boldsymbol{\theta}_{e}(t),\boldsymbol{\phi}_{e}(t)\\}$,
via our multi-view system. The 3D rigid transformation (translation
$\mathbf{l}_{e}(t)$ and 3D orientation $\boldsymbol{\theta}_{e}(t)$) is
computed by specifying the attached marker corners at each time $t$, by
triangulating N number of 2D marker points $\\{\mathbf{m}_{i}(t)\\}_{i=0}^{N}$
into 3D marker points $\\{\mathbf{M}_{i}(t)\\}_{i=0}^{N}$. Given the 3D marker
corners $\\{\mathbf{M}_{i}(t)\\}_{i=0}^{N}$ and the corresponding corners
$\\{\mathbf{\hat{M}}_{i}\\}_{i=0}^{N}$ in the object canocial space, we
compute $\mathbf{T}_{mar}(t)$, via a well-known Kabsch algorithm. Then, the
object transformation $\mathbf{T}_{obj}(t)$ can be computed as Eq. 3.

The object-specific dynamic status $\boldsymbol{\phi}_{e}(t)$ is similarly
computed by computing the transformations from the 3D markers attached to the
base part to the ones for the movable parts.

Figure 5: Articulation state of 3D scanned objects. Blue bars show the object
specific parameters $s_{e}^{j}(t)$ for each object part. As $s_{e}^{j}(t)$
changes, corresponding movable parts of the objects show different
articulation states

###  5.2 Capturing 3D Human Motions

Leveraging two heterogeneous systems, multi-camera systems and wearable motion
suits and gloves, is challenging, since they do not share a common spatial
world coordinate. Specifically, wearable motion capture systems suffer from
the drift issue in localizing the global root position, and more critically
they employ an imperfect assumption of the body and hand skeleton scale that
differs from the actual measurement. As such, directly transferring the output
of wearable captures into the camera system space cannot fulfill our goal of
capturing precise hand-object interactions. In this section, we present a
method to spatially align two systems.

Aligning Wearable Mocap in Multi-Camera System. To spatially align the body
motion capture with our multi-view camera system, we need to provide
correspondences between the two systems. For this purpose, we attach 3 or 4
ArUco markers to each of 11 near-rigid body parts (torso, hands, upper arms,
lower arms, upper legs, lower legs), as shown in Fig. 2 and in our
supplementary material. We denote the four corner points of the $j$-th marker
attached to the $i$-th body part as
$\mathcal{M}^{b}_{i,j}\in\mathbb{R}^{4\times 3}$, where the positions are
defined in the local joint coordinate w.r.t the corresponding body part.
Specifically, the goal of our alignment process is: (1) to obtain authentic
body skeleton configuration $\mathcal{B}=\\{\mathcal{O}\\}$, which represents
the offsets of the child joints from the parent joints, and (2) body marker
locations $\mathcal{M}^{b}$. Given these parameters and the joint angle
measurements provided by the mocap suit $\theta^{t}\in\mathbb{R}^{23\times 3}$
at time $t$, we can transform the body-attached markers into the person-
centric coordinate denoted as $\mathbf{M}^{b}_{mocap}(t)$ via forward
kinematics function $\mathcal{K}_{b}$ as:
$\mathbf{M}^{b}_{mocap}(t)=\mathcal{K}_{b}(\mathcal{M}^{b},\theta^{t},\mathcal{B})$.
At the same time, the body-attached markers can be reconstructed via our
multi-view system, which is denoted as $\mathbf{M}_{cam}^{b}(t)$ defined in
the camera system space. Then, the rigid transformation to transfer the mocap
data into the camera system space can be computed with the marker
correspondence:
$T_{b}^{cam}(t)=T(\mathbf{M}^{b}_{mocap}(t),\mathbf{M}_{cam}^{b}(t))$, where
only visible markers in $\mathbf{M}_{cam}^{b}(t)$ are considered for the
computation. Note that we can compute the rigid transformation as long as we
can see at least one body-attached marker in $\mathbf{M}_{cam}^{b}(t)$,
providing robustness to the marker occlusions in computing the global location
of the actor. We perform such transformation for the entire frames of body
alignment capture which consists of motions that encompass various body poses.
We set below objective function to get optimized $\mathcal{B}$ and
$\mathcal{M}^{b}$:

Figure 6: (Left) Before Body Calibration (Right) After Body Calibration,
Orange: forward kinematic output, Blue: RGB Triangulated Result

$\displaystyle\min_{\mathcal{M}^{b},\mathcal{B}}\,\sum_{t=1}^{T}{\lambda_{b}\mathcal{L}^{t}_{body}+\lambda_{f}\mathcal{L}^{t}_{foot}}.$
(4)

The $\mathcal{L}_{body}$ is the mean-squared-error between corresponding
markers as:

$\mathcal{L}_{body}^{t}=\sum_{m_{i}^{b}\in\mathbf{M}^{b}_{\text{mocap}},m_{i}^{\text{cam}}\in\mathbf{M}^{b}_{\text{cam}}}{\left\|T_{b}^{\text{cam}}(t)m_{i}^{\text{mocap}}-m_{i}^{\text{cam}}\right\|}.$
(5)

We use a constraint term $L_{foot}$ to enforce the foot parts to be close
toward the ground without penetration or floating. The accuracy of our body
alignment process is shown in Fig. 6, showing the aligned markers after the
optimization.

Figure 7: (Left) Hand Calibration Protocol and (Right) Before and After
Calibration Protocol

Calibrating Hand Mocap. The hand mocap outputs from the gloves also suffer
from similar issues which are unknown skeleton lengths of actual hand and
localizing into the camera system space. Since a subtle error may cause a
large deviation in the hand interaction scene, such issues are critical in
obtaining high quality data. As a solution, we present a new hand alignment
protocol of using a calibration structure. We specify the ordered 3D corner
vertices of the structure,
$C=\\{\mathbf{c}_{i}\in\mathbb{R}^{3}\\}_{i=1}^{6}$, which we localize in
camera system space via triangulation. During the alignment protocol, the
participant touches the calibration structure’s known corner locations with
their fingertips, from which we can approximate the desired locations of
fingertips. Examples of hand calibration protocols and the hand calibration
structure are shown in Fig. 7 and our supplementary video as well.

Similar to the body calibration the goal of hand calibration is to obtain the
authentic hand skeleton configuration
$\mathcal{H}=\\{\mathcal{S}^{h}\in\mathbb{R}^{20\times
1},\mathcal{O}^{h}\in\mathbb{R}^{20\times 3}\\}$ and positions of 3D markers
in the local hand-centric coordinate $\mathcal{M}^{h}\in\mathbb{R}^{3\times
4\times 3}$, where $\mathcal{S}^{h}$ and $\mathcal{O}^{h}$ each denotes scales
of hand skeleton and per-joint skeletal offsets. We perform an optimization
via the following objective function:

$\displaystyle\min_{\mathcal{M}^{h},\mathcal{H}}\lambda_{t}L_{tip}+\lambda_{w}\mathcal{L}_{wrist}+\lambda_{p}\mathcal{L}_{pen}.$
(6)

$\mathcal{L}_{tip}=\sum_{k\in\mathbf{o}}\left\|T^{k}_{h}\mathcal{K}_{h}^{k}(\mathcal{H})_{i}-\mathbf{c}^{k}_{j}\right\|$
penalizes the difference between a fingertip and the corresponding corner of
the calibration cube $c_{i}^{k}$ ($\mathbf{o}$ denotes the order of
correspondences), where $T^{k}_{h}$ is the hand-to-camera transformation at
$k$-th time instance, and $\mathcal{K}_{h}$ is the forward kinematics function
to transform the $i$-th fingertip. As an additional constraint, we include
$\mathcal{L}_{wrist}$ which penalizes the distance between the wrist locations
from the body mocap and from the hand marker to enforce hands are rigidly
connected to the body mocap. We also include $\mathcal{L}_{pen}$ to prevent
the fingertips from penetrating the surface of the calibration cube. See
supplementary material for details.

###  5.3 Post-Processing To Enhance Qualities

As an advantage of the heterogeneous nature of the systems, we can leverage
cues from both systems to reduce jitters and handle tracking failures.

Enhancing Hand Tracking. The global hand location estimated by the attached 3D
marker may suffer from motion jitter or tracking failures due to occlusions or
low camera visibility. As a solution to enhance the tracking quality, we also
leverage the cues from the wearable mocap. Specifically, we are given two
measurements regarding the wrists, from hand attached markers and from the
body mocap. If both measurements are available, we perform a weighted
interpolation to reduce motion jitters. In the case of tracking failure of the
hand-attached 3D marker, we rely on the cues from the body motion suit since
it’s occlusion-free. We demonstrate the robustness of method in our
experiments. See our supplementary material for details.

Enhancing Object Tracking. Although we use multiple cameras with our 3D marker
system, object tracking failure may still happen due to severe occlusions
during interactions. As a solution, we leverage the mocap measurement from our
mocap system to enhance object tracking qualities. Specifically, we first
infer whether currently an object is close toward the human actor by checking
the distances between the human joint positions and object’s surface. If an
object has been moved by the actor, we assume the object is near-rigidly
connected to the close body joints, and apply the temporal transformation of
the body joints to interpolate the missing object trajectories from the
neighboring times. In practice, we apply this method for both reducing jitters
and recovering the tracking failures.

Figure 8: Rendered scenes from example scenario. (Top) A person pours from
kettle and drinks from a cup. (Middle) A person opens a laptop. (Bottom) A
person chops on the cuttingboard using knife and pour into the pot after
opening a potlid.

##  6 3D Home Activity Dataset

Leveraging our ParaHome system, we capture a large-scale human object
interaction dataset. The captured data includes the human body and finger
motions, and the dynamic changes of articulated objects that participants
interact with. Our dataset is intended to capture the diverse and realistic
behaviors of humans in a room environment, where humans can freely move around
interacting with multiple objects which could have inter-relationships. Due to
the diversity of human activities, we focus on several key set of actions
where human interact with one or multiple objects for daily activities.

###  6.1 Capture Scenarios

The sequence of actions that participants perform is composed of small atomic
actions such as “move the laptop to the table”, which involves the
manipulation of one or two objects. A total of 35 distinct atomic actions are
performed by participants. 20 of them are cooking-related actions such as
“turn on the gas stove”, and the remaining 15 are small actions that can occur
in a room environment, such as “type the laptop”.

Each participant performs 2 to 4 scenarios. A single scenario consists of 12
to 15 of non-cooking actions, and 9 to 14 actions from cooking-related
actions. These atomic actions are placed in semi-arbitrary order with
corresponding verbal instruction for the actions that are recorded and
annotated to each capture data. Check our supplementary material for the full
list of actions and more details.

###  6.2 Dataset Statistics

We capture the data from 30 participants (15 females and 15 males). To this
end, our 3D common daily motion dataset contains 440 minutes of actions
interacting with 22 objects. Each scenario that the participant performs
consists of 21 to 29 individual actions. Due to the storage limits, we divide
one scenario into 2 sessions of captures. Duration of each session ranges from
64 to 280 seconds, with an average of 145 seconds per session. Each
participant performs 2 to 4 scenarios, and total of 101 different scenarios
were captured. After dividing each scenario into individual atomic actions, we
captured a total of 2284 atomic actions. For further details, see
supplementary material.

##  7 Evaluations

In this section, we evaluate the robustness and accuracy of our system in
various ways. For the evaluation, we capture long test sequences (4-5 minutes)
including various human object interactions.

###  7.1 System and Capture Evaluation

Impact of Camera Number. We demonstrate the advantage of using all 70 cameras
against the alternative solutions of using fewer cameras. To quantify the
impact of camera number, we count the number of detected marker corners by
simulating systems with varying numbers of cameras with subsets from the
entire camera set. As seen in Fig. 9 (Top Left), the detected ratio tends to
increase linearly with the addition of cameras to the subsets. This
observation implies that we take full advantage of our system without
experiencing redundancy.

| Average Tracked | During Manipulation
---|---|---
Average Camera Number | 8.94 | 8.59
Reproj. Error [pixel] | 1.021 | 1.016

| Ours | 4 | 7 | 10 | 20 | 40
---|---|---|---|---|---|---
Tracking Success Ratio | 1.0 | 0.76 | 0.79 | 0.86 | 0.90 | 0.93

Table 1: Evaluation results on system settings (Top) Average Reprojection
error detected in the scene and during manipulation by humans. (Bottom)
Average number of tracked object ratio on multiple sampled windows. Numbers in
the upper row represent number of virtual passive markers attached to the
surface of the target object.

Advantage of 3D Marker Cube. We compare our 3D cube marker solution against an
alternative solution of attaching markers on the surface of the object body
(e.g., IR markers as in [13]). To simulate such IR marker system, we sample a
set of points on the object mesh surface, assuming the sampled surface points
as virtual markers. We choose a complex interaction sequence of 4 min long,
where our 3D marker-based object tracking is fully successful. Then, we assess
the tracking ratio of virtual surface markers, considering varying numbers of
marker attached as shown in Tab. 1. As a protocol to determine the tracking
ratio, we consider the actor in the scene as a mesh model as in Fig. 1
additionally attached with a box shaped mesh to represent body torso and
assume a marker tracking fails if the detected number of camera of the marker
is less than 3. We consider 300 frames of sliding windows, and assume an
object is successfully tracked if at least 4 markers of the object are fully
tracked along the window without any tracking failure. Finally, we compute the
average tracking success ratio in each marker setups, as shown in Tab. 1. As
seen, attaching markers on the object surface suffers from tracking failures
mainly because the actors tend to touch the object during interactions and due
to the existence of multiple objects in the scene.

3D Marker Triangulation Quality. To measure the quality of tracking objects
via ArUco markers, we report average reprojection error of triangulated ArUco
marker corners for all tracked scenes and for the interaction target parts.
Along with each reprojection error, we also report the average number of
cameras (visibility) used in corner triangulation for each case, when in
contact and averaged in all scenes. Check Tab. 1 for the details.

Tracking Quality via Assessing Temporal Jitters. We compare the quality of
tracking of our two heterogeneous systems, a multi-view system and the IMU-
based wearable mocap suits. As a way to verify the quality, we assess the
temporal motion jitters of both system, by plotting the derivative of
acceleration for a certain interval of time. For this test, we capture a
dedicated sequence, interacting with small and movable objects such as a cup
and a kettle by moving them into various poses in multiple areas. The result
is shown in Fig. 9 (Top Right), where both systems show similar jitter levels.
Given that we use the commercial expert-level wearable motion capture system
equipped with high-quality IMUs, this particular result demonstrates that our
object tracking quality from cameras is comparable to the wearable capture
system, enabling subtle interaction captures.

Hand Alignment Evaluation. To quantify the quality of our hand alignment
procedure shown in Sec. 1, we perform a validation capture, where a
participant touches random marker corners attached to the objects in various
areas in the studio. Then, we measure the average distance between the
fingertip and the contacted marker corner point at the contact timing. The
Average Position Error (APE) is 11 $mm$ with 86 finger touches with various
fingertips. Considering the possible bias caused during the touch due to the
finger width (15-20 $mm$), our result demonstrates precise spatial alignment
quality.

###  7.2 Post Processing Validation

We validate the performance of our post-processing method. As a way to
quantify the performance, we simulate the tracking failure cases by
intentionally dropping certain intervals of frames from successfully tracked
sequences, and applying our post-processing method to recover the dropped
frames. By comparing the originally tracked cues with the recovered cues from
the post-processing, we can assess the quality of our post-processing outputs.

Evaluating Object Tracking Enhancement. We perform our object-tracking
enhancement process using the cues from hand tracking and compare its quality
by a naive interpolation method based on linear functions (e.g. _lerp_ for
translation and _slerp_ for rotation). The quantitative results are shown in
Fig. 9 (Bottom Left), (Bottom Right), where our method powered with mocap
device observations demonstrates much better performance in recovering the
tracking failures.

Evaluating Hand Tracking Enhancement. To test the robustness of our system, we
recover undetected markers of the dropped frames with our post processing
algorithm. We measured L2 distance between hand joints computed using
recovered markers position and joint positions of original output and get L2
loss 9 $mm$.

Figure 9: (Top Left) Ratio of markers being detected with respect to 70
cameras to sampled number of cameras (Top Right) Acceleration derivative of
object and interacting hand wrist joint (Bottom Left) Comparison of difference
betweeen original translation and hole-filled versions. (Bottom Right)
Comparison of difference betweeen original original rotation (in 6D) and hole-
filled versions

##  8 Discussion

We present a ParaHome system, specifically designed to capture human motions,
finger movements, and object dynamics in 3D during natural human-object
interactions in a studio apartment setting. Our innovative system incorporates
multiple hardware and software solutions. Leveraging our system, we collect a
new HOI dataset, which offers key improvements over existing datasets,
including: (1) capturing 3D body and hand motion alongside 3D object movement
within a contextual home environment during natural activities; (2)
encompassing human interaction with multiple objects; (3) including
articulated objects with multiple parts. We expect our dataset can provide
several new opportunities in exploring the generative modeling of HOI in a
parameterized 3D space. We invest efforts to expand our dataset by involving
more participants with more scenarios. All processed data will be made
available to the public, facilitating further research in the field.

There exist limitations in our system, intriguing interesting future research
directions. First, the RGB videos in our dataset are heavily biased due to the
wearable suits and markers used in the system, and cannot be used in training
the RGB image-based models. We plan to enhance our system toward a markerless
system by eliminating artificial markers, once we can build a robust prior
model for human object interactions learned from our current dataset. Second,
to make the dataset more generalized and unbiased, it is necessary to include
more diverse furniture, electronic devices, and items, as well as diverse room
layouts. Toward this goal, we plan to replicate our system in different room
settings, with more diverse objects ensuring better generalizability.

Acknowledgement. This work was supported by Samsung Electronics MX Division,
NRF grant funded by the Korean government (MSIT) (No. 2022R1A2C2092724 and No.
RS-2023-00218601), and IITP grant funded by the Korean government (MSIT)
(No.2021-0-01343). H. Joo is the corresponding author.

## References

  * [1] Bharat Lal Bhatnagar, Xianghui Xie, Ilya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll.  Behave: Dataset and method for tracking human object interactions.  CVPR, 2022.
  * [2] Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays.  Contactpose: A dataset of grasps with object contact and hand pose.  ECCV, 2020.
  * [3] Christoph Bregler, Jitendra Malik, and Katherine Pullen.  Twist based acquisition and tracking of animal and human kinematics.  IJCV, 2004.
  * [4] Thomas Brox, Bodo Rosenhahn, Juergen Gall, and Daniel Cremers.  Combined region and motion-based 3D tracking of rigid and articulated objects.  TPAMI, 2010.
  * [5] Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik.  Long-term human motion prediction with scene context.  ECCV, 2020.
  * [6] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al.  Dexycb: A benchmark for capturing hand grasping of objects.  CVPR, 2021.
  * [7] Kong Man Cheung, Simon Baker, and Takeo Kanade.  Shape-from-silhouette across time part i: Theory and algorithms.  IJCV, 2005.
  * [8] Stefano Corazza, Lars Mündermann, Emiliano Gambaretto, Giancarlo Ferrigno, and Thomas P. Andriacchi.  Markerless Motion Capture through Visual Hull, Articulated ICP and Subject Specific Model Generation.  IJCV, 2010.
  * [9] Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez.  Ganhand: Predicting human grasp affordances in multi-object scenes.  CVPR, 2020.
  * [10] Edilson de Aguiar, Carsten Stoll, Christian Theobalt, Naveed Ahmed, Hans-Peter Seidel, and Sebastian Thrun.  Performance capture from sparse multi-view video.  SIGGRAPH, 2008.
  * [11] Joseph DelPreto, Chao Liu, Yiyue Luo, Michael Foshey, Yunzhu Li, Antonio Torralba, Wojciech Matusik, and Daniela Rus.  Actionsense: A multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment.  NeurIPS, 35:13800–13813, 2022.
  * [12] Einstar, 2023.  https://www.einstar.com/.
  * [13] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges.  Articulated objects in free-form hand interaction.  CVPR, 2023.
  * [14] Y. Furukawa and J. Ponce.  Dense 3d motion capture from synchronized video streams.  CVPR, 2008.
  * [15] Juergen Gall, Carsten Stoll, Edilson De Aguiar, Christian Theobalt, Bodo Rosenhahn, and Hans-Peter Seidel.  Motion capture using joint skeleton tracking and surface estimation.  CVPR, 2009.
  * [16] Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim.  First-person hand action benchmark with rgb-d videos and 3d hand pose annotations.  CVPR, 2018.
  * [17] Sergio Garrido-Jurado, Rafael Muñoz-Salinas, Francisco José Madrid-Cuevas, and Manuel Jesús Marín-Jiménez.  Automatic generation and detection of highly reliable fiducial markers under occlusion.  Pattern Recognition, 47(6):2280–2292, 2014.
  * [18] D. Gavrila and LS Davis.  Tracking of humans in action: A 3-D model-based approach.  ARPA Image Understanding Workshop, 1996.
  * [19] Markus Gross, Stephan Würmlin, Martin Naef, Edouard Lamboray, Christian Spagno, Andreas Kunz, Esther Koller-Meier, Tomas Svoboda, Luc Van Gool, Silke Lang, Kai Strehlke, Andrew Vande Moere, and Oliver Staadt.  Blue-c: A spatially immersive display and 3d video portal for telepresence.  SIGGRAPH, 2003.
  * [20] Ralph Gross and Jianbo Shi.  The cmu motion of body (mobo) database.  2001\.
  * [21] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  CVPR, 2022.
  * [22] Ikhsanul Habibie, Daniel Holden, Jonathan Schwarz, Joe Yearsley, and Taku Komura.  A recurrent variational autoencoder for human motion synthesis.  BMVC, 2017.
  * [23] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit.  Honnotate: A method for 3d annotation of hand and object poses.  CVPR, 2020.
  * [24] Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit.  Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation.  CVPR, 2022.
  * [25] Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher D Twigg, and Kenrick Kin.  Online optical marker-based hand tracking with deep labels.  ACM TOG, 37(4):1–10, 2018.
  * [26] Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J Black.  Populating 3d scenes by learning human-scene interaction.  CVPR, 2021.
  * [27] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid.  Learning joint reconstruction of hands and manipulated objects.  In CVPR, 2019.
  * [28] Chun-Hao P Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J Black.  Capturing and inferring dense full-body human-scene contact.  CVPR, 2022.
  * [29] Yinghao Huang, Omid Taheri, Michael J Black, and Dimitrios Tzionas.  Intercap: Joint markerless 3d tracking of humans and objects in interaction.  GCPR, 2022.
  * [30] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE transactions on pattern analysis and machine intelligence, 36(7):1325–1339, 2013.
  * [31] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh.  Panoptic studio: A massively multiview system for social motion capture.  CVPR, 2015.
  * [32] Takeo Kanade, Peter Rander, and P.J. Narayanan.  Virtualized reality: Constructing virtual worlds from real scenes.  IEEE Multimedia, 1997.
  * [33] Roland Kehl and Luc Van Gool.  Markerless tracking of complex human motions from multiple views.  CVIU, 2006.
  * [34] Nilesh Kulkarni, Davis Rempe, Kyle Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Leonidas Guibas.  Nifty: Neural object interaction fields for guided human motion synthesis.  arXiv preprint arXiv:2307.07511, 2023.
  * [35] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys.  H2o: Two hands manipulating objects for first person interaction recognition.  ICCV, 2021.
  * [36] Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi.  Hoi4d: A 4d egocentric dataset for category-level human-object interaction.  CVPR, 2022.
  * [37] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black.  Amass: Archive of motion capture as surface shapes.  ICCV, 2019.
  * [38] Manus, 2023.  https://www.manus-meta.com/.
  * [39] T. Matsuyama and T. Takai.  Generation, visualization, and editing of 3d video.  3DPVT, 2002.
  * [40] Wojciech Matusik, Chris Buehler, Ramesh Raskar, Steven J. Gortler, and Leonard McMillan.  Image-based visual hulls.  SIGGRAPH, 2000.
  * [41] Aymen Mir, Xavier Puig, Angjoo Kanazawa, and Gerard Pons-Moll.  Generating continual human motion in diverse 3d scenes.  arXiv preprint arXiv:2304.02061, 2023.
  * [42] Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee.  Interhand2.6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image.  ECCV, 2020.
  * [43] Movella, 2023.  https://base.xsens.com/.
  * [44] Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt.  Ganerated hands for real-time 3d hand tracking from monocular rgb.  CVPR, 2018.
  * [45] Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt.  Real-time hand tracking under occlusion from an egocentric rgb-d sensor.  ICCV, 2017.
  * [46] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975–10985, 2019.
  * [47] Benjamin Petit, Jean-Denis Lesage, Edmond Boyer, and Bruno Raffin.  Virtualization Gate.  SIGGRAPH Emerging Technologies, 2009.
  * [48] Ilya A Petrov, Riccardo Marin, Julian Chibane, and Gerard Pons-Moll.  Object pop-up: Can we infer 3d objects and their poses from human interactions alone?  CVPR, 2023.
  * [49] Mathis Petrovich, Michael J Black, and Gül Varol.  Action-conditioned 3d human motion synthesis with transformer vae.  ICCV, 2021.
  * [50] Mathis Petrovich, Michael J Black, and Gül Varol.  Temos: Generating diverse human motions from textual descriptions.  ECCV, 2022.
  * [51] Ralf Plankers and Pascal Fua.  Articulated Soft Objects for Multi-View Shape and Motion Capture.  TPAMI, 2003.
  * [52] Javier Romero, Dimitrios Tzionas, and Michael J Black.  Embodied hands: Modeling and capturing hands and bodies together.  SIGGRAPH ASIA, 2017.
  * [53] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [54] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao.  Assembly101: A large-scale multi-view video dataset for understanding procedural activities.  CVPR, 2022.
  * [55] Leonid Sigal, Alexandru O Balan, and Michael J Black.  Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion.  IJCV, 87(1-2):4, 2010.
  * [56] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.  Hand keypoint detection in single images using multiview bootstrapping.  CVPR, 2017.
  * [57] Carsten Stoll, Nils Hasler, Juergen Gall, Hans-Peter Seidel, and Christian Theobalt.  Fast articulated motion tracking using a sums of gaussians body model.  ICCV, 2011.
  * [58] Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas.  Grab: A dataset of whole-body human grasping of objects.  ECCV, 2020.
  * [59] Purva Tendulkar, Dídac Surís, and Carl Vondrick.  Flex: Full-body grasping without full-body grasps.  CVPR, 2023.
  * [60] Dylan Turpin, Liquan Wang, Eric Heiden, Yun-Chun Chen, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg.  Grasp’d: Differentiable contact-rich grasp synthesis for multi-fingered hands.  ECCV, 2022.
  * [61] Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović.  Articulated mesh animation from multi-view silhouettes.  TOG, 2008.
  * [62] Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović.  Articulated mesh animation from multi-view silhouettes.  SIGGRAPH, 2008.
  * [63] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh.  Monocular total capture: Posing face, body, and hands in the wild.  CVPR, 2019.
  * [64] Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui.  Interdiff: Generating 3d human-object interactions with physics-informed diffusion.  ICCV, 2023.
  * [65] Sifan Ye, Yixing Wang, Jiaman Li, Dennis Park, C Karen Liu, Huazhe Xu, and Jiajun Wu.  Scene synthesis from human motion.  SIGGRAPH Asia, 2022.
  * [66] Shanxin Yuan, Qi Ye, Bjorn Stenger, Siddhant Jain, and Tae-Kyun Kim.  Bighand2. 2m benchmark: Hand pose dataset and state of the art analysis.  CVPR, 2017.
  * [67] Hui Zhang, Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges.  Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation.  arXiv preprint arXiv:2309.03891, 2023.
  * [68] He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura.  Manipnet: Neural manipulation synthesis with a hand-object spatial representation.  ACM TOG, 40(4), 2021.
  * [69] Juntian Zheng, Qingyuan Zheng, Lixing Fang, Yun Liu, and Li Yi.  Cams: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis.  CVPR, 2023.
  * [70] Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll.  Toch: Spatio-temporal object-to-hand correspondence for motion refinement.  ECCV, 2022.
  * [71] Christian Zimmermann and Thomas Brox.  Learning to estimate 3d hand pose from single rgb images.  ICCV, 2017.
  * [72] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox.  Freihand: A dataset for markerless capture of hand pose and shape from single rgb images.  ICCV, 2019.
  * [73] Gaspard Zoss, Derek Bradley, Pascal Bérard, and Thabo Beeler.  An empirical rig for jaw animation.  ACM TOG, 37(4):1–12, 2018.

Dataset | frame # | subject # | object # | body | hand # | contact | obj. 6d. | obj. artic. | multi obj. | environment
---|---|---|---|---|---|---|---|---|---|---
Freihand [72] | 37K | 32 | 27 | ✗ | 1 | - | ✗ | ✗ | - | table setup
Obman [27] | 154K | 20 | 2772 | ✗ | 1 | ✗ | ✓ | ✗ | ✗ | synthetic
GRAB [58] | - | 10 | 51 | ✓ | 2 | ✓ | ✓ | ✗ | ✗ | standing setup
DexYCB [6] | 582K | 10 | 20 | ✗ | 1 | ✗ | ✓ | ✗ | ✗ | table setup
HO3D [23] | 78K | 10 | 10 | ✗ | 1 | ✗ | ✓ | ✗ | ✗ | table setup
ContactPose [2] | 3M | 50 | 25 | ✗ | 2 | ✓ | ✓ | ✗ | ✗ | standing setup
BEHAVE [1] | 15K | 8 | 20 | ✓ | - | ✓ | ✓ | ✗ | ✗ | portable setup
InterCap [29] | 67K | 10 | 10 | ✓ | 2 | ✓ | ✓ | ✗ | ✗ | portable setup
FHPA [16] | 105K | 6 | 26 | ✗ | 1 | ✗ | ✓ | ✗ | ✗ | room setup
H2O [35] | 571K | 4 | 8 | ✗ | 2 | ✗ | ✓ | ✗ | ✗ | table setup
H2O-3D [24] | 76K | 5 | 10 | ✗ | 2 | ✗ | ✓ | ✗ | ✗ | table setup
HOI4D [36] | 2.4M | 9 | 800 | ✗ | 1 | ✗ | ✓ | ✓ | ✗ | room setup
ARCTIC [13] | 2.1M | 10 | 11 | ✓ | 2 | ✓ | ✓ | ✓ | ✗ | standing setup
Ours | 56M | 30 | 22 | ✓ | 2 | ✓ | ✓ | ✓ | ✓ | room setup

Table 2: Comparison of existing human-object interaction datasets

##  Appendix A Dataset Comparison

In Tab. 2, we present a comprehensive comparison between our ParaHome dataset
and other existing datasets that capture human-object interactions. As shown,
our ParaHome dataset is the first dataset to capture all essential components
in dynamic HOI scenarios in a natural room environment, containing body
motions, hand motions, and the movements of all visible objects in the scenes
including object articulations. Notably, our dataset contains one to two
orders of magnitude larger frame numbers than existing datasets. Details of
the comparison are in Tab. 2.

##  Appendix B System Details

Figure 10: System Devices. (a) RGB cameras with 3 types of lenses. (b) Signal
distributers (c) Desktop machines (d) LED Lights (e) NAS storage systems

In order to cover the entire volume of the room and to reduce occlusion
issues, we install 70 RGB industrial cameras, BFLY-31S4C-C. The cameras
capture videos at 30Hz in 2048$\times$1536 resolution. We set the exposure
time at 3$msec$, which shows a good balance between low-motion blur and
sufficient brightness. We use three types of lenses (thirty 3$mm$ lenses,
twenty 5$mm$ lenses, and twenty 6$mm$ lenses), where the wide-angle lens
(3$mm$) is helpful in capturing wide area. We calibrate the cameras using
Structure-from-Motion via COLMAP [53] with multiple randomly patterned fabrics
placed in our system. We provide pre-calibrated initial intrinsic parameters
for the three types of lenses derived from 2 or 3 samples of lenses for better
convergence in camera pose estimation. We scale the calibrated 3D space into a
real-world metric (in meters) by locating checkerboards with known sizes
during camera calibration.

All cameras, the motion capture suit, and gloves are synced and gen-locked via
a common square wave signal that comes from the motion capture device to
synchronize two heterogeneous systems, which is crucial to precise HOI
captures. To deliver the sync signals to a large number of cameras, we
utilized 11 signal distributors in a hierarchical manner, each of which can be
connected to 8 cameras via GPIO cables. We use 1 master and 18 slave desktop
machines to control the cameras and process captured records. Each slave
machine is connected to 3 or 4 cameras via Ethernet cables and equipped with a
4-port 1G ethernet board, and 2 SSDs with a capacity of 500GB and 1TB each. We
also include 2 NAS storage systems each holding 96TB of storage, connected to
all machines to save processed data. 15 LED lights (4500$lm$) are installed to
provide sufficient illumination. Pictures of our system devices are shown in
Fig. 10.

To capture both body motion and subtle hand motions, we use IMU-based motion
capture equipment, Xsens motion suit [43] and Manus hand gloves [38]. The body
motion system captures the motions at 60Hz.

Object | Part1 | Part2
---|---|---
Sink | revolute | revolute
Laptop | revolute | -
Drawer | sliding | sliding
Gasstove | revolute | revolute
Microwave | revolute | -
Trashbin | revolute | -
Washingmachine | revolute | -
Refrigerator | revolute | revolute
Table 3: Part information of articulated objects

##  Appendix C Data Acquisition

###  C.1 Modeling Object Articulations.

To capture the movement of articulated objects, we model each object as a
parametric 3D model by defining the object-specific articulated motion
parameters. This modeling requires scanning individual parts separately and
compositing them in a canonical space with defining axis direction, pivot
points, revolute joints, and so on, based on the object types. During HOI
captures, we track the motion of each part via our marker system (e.g. monitor
of a laptop and the base), from which we compute the articulated motion
parameters. In this subsection, we describe the process of modeling
articulated objects as parametric 3D models. Articulation information of each
object with multiple parts is shown in Tab. 3.

To find axis $\mathbf{a}_{e}$ and pivot point $\mathbf{p}_{e}$ of the
articulated objects, we capture markers attached to each object part at
different part states separately and acquire each marker corners in the
ParaHome space as $\\{m_{i}(t)\\}_{t=1}^{n}$. Prior to applying algorithm, we
transform marker corners $\\{m_{i}(t)\\}_{t=1}^{n}$ back to object canonical
space with $T_{mar\rightarrow obj}^{-1}$ and utilize transformed marker
corners in the canonical space $\\{m_{i}^{\prime}(t)\\}_{t=1}^{n}$. For the
sliding joint, axis $a_{e}$ can easily be calculated using marker corners at
time $t$ and $t^{\prime}$ as:

$\mathbf{a}_{e}=\frac{m_{i}^{\prime}(t)-m_{i}^{\prime}(t^{\prime})}{\|m_{i}^{\prime}(t)-m_{i}^{\prime}(t^{\prime})\|}$

In case object part has a revolute joint, we start initializing an axis
$\mathbf{a}_{e}$ and each relative state $\Delta
s_{e}(t,t^{\prime})=|s_{e}(t)-s_{e}(t^{\prime})|$ between time $t$ and
$t^{\prime}$ (for the target articulated object captured at different $n$
number of states, time $t$ and time $t^{\prime}$ satisfies $t\neq t^{\prime}$
and $t,t^{\prime}\in\\{1,2,\cdots,n\\}$). Then we apply optimization with
marker corners toward all possible pairs of times $t$ and $t^{\prime}$. Let
$f$ be a map defining rotation transformation with respect to pivot and given
axis-angle and denote as $\mathbf{T}_{t^{\prime}\rightarrow t}=f(a_{e},\Delta
s_{e}(t,t^{\prime}),p_{e})$. Then for a set of all possible time pairs
$\mathbf{P}$, the optimization target for axis $a_{e}$, relative state $\Delta
s_{e}$ and pivot $p_{e}$ becomes:

$\operatorname*{arg\,min}_{a_{e},\Delta
s_{e}(t,t^{\prime}),p_{e}}\sum_{(t,t^{\prime})\in\mathbf{P}}\|m_{i}^{\prime}(t)-\mathbf{T}_{t^{\prime}\rightarrow
t}m_{i}^{\prime}(t^{\prime})\|^{2}$

Since initial axis $\mathbf{a}_{e}$ and pivot $\mathbf{p}_{e}$ are initialized
in the object canonical space, we directly utilize acquired information to
derive transformations from detected markers from each capture session.

###  C.2 Body Alignment Detail (Sec 5.2 in Main Paper)

In this subsection, we provide additional details of our spatial alignment
process between a multiview camera system and wearable motion capture systems,
described in Sec. 5.2 in our main manuscript.

To resolve the issue of imperfect body and hand skeleton scale from the
wearable motion capture system, we attach 3 or 4 ArUco markers to each near-
rigid body part (torso, hands, upper arms, lower arms, upper legs, lower legs)
to assign correspondences. During alignment capture, participants perform the
range-of-motion movement by rotating their arms and legs while pinned or bent,
particularly twisting their wrists to locate each hand wrist. With the
captured data, we optimize body skeleton configuration
$\mathcal{B}=\\{\mathcal{O}\\}$ and body markers locations $\mathcal{M}^{b}$
via gradient decent with a learning rate of 0.008 for 50 epochs. Specifically
for weights of body and foot, $\lambda_{b}=100,\lambda_{f}=5000$ are used. In
the case that alignment is not well optimized, we additionally penalize
excessive length change in spines and difference in skeleton lengths between
the left and right sides of the body by adding an extra regularization term to
the optimization target. Once the alignment procedure is finished, we remove
markers (all from the upper legs, one for each upper arm, lower arm, and lower
leg) to minimize interference with the movements of the participant. The
selection of remaining markers is determined based on their importance during
captures, where we assess their importance by evaluating whether their absence
would compromise the accuracy of body positioning in the camera space. Check
our supplementary video for an example of body alignment motion.

###  C.3 Hand Calibration Structure and Protocol (Sec 5.2 in Main Paper)

Figure 11: The hand calibration structure to precisely measure hand skeleton configuration and to find the relative locations of hand markers attached to wrist in hand-centric coordinate Corner # | Hand Side | Seq1 | Seq2 | Seq3 | Seq4
---|---|---|---|---|---
1,2 | Right | 1, 2 | 1, 3 | 1, 4 | 1, 5
1,3 | Right | 1, 2 | 1, 3 | 1, 4 | 1, 5
2,4 | Right | 1, 2 | 1, 3 | 1, 4 | 1, 5
5,2 | Right | 1, 2 | 1, 3 | 1, 4 | 1, 5
6,2 | Right | 1, 2 | 1, 3 | 1, 4 | 1, 5
6,3,2 | Right | 1, 2, 3 | 1, 3, 4 | 1, 4, 5 | -
2,1 | Left | 1, 2 | 1, 3 | 1, 4 | 1, 5
3,1 | Left | 1, 2 | 1, 3 | 1, 4 | 1, 5
4,2 | Left | 1, 2 | 1, 3 | 1, 4 | 1, 5
5,2 | Left | 1, 2 | 1, 3 | 1, 4 | 1, 5
6,2 | Left | 1, 2 | 1, 3 | 1, 4 | 1, 5
2,5,6 | Left | 1, 2, 3 | 1, 3, 4 | 1, 4, 5 | -
Table 4: Hand Calibration Protocol

As human usually handle objects with their fingers, fingertips play an
important role during interaction. We made the calibration structure, as shown
in Fig. 11, to better locate fingertips and find the hand skeleton
configurations along with relative locations between the attached hand markers
to wrist. We build the hand calibration structure by combining three cubes
with ArUco markers and specifying the ordered 3D corner vertices of the
structure $C=\\{\mathbf{c}_{i}\in\mathbb{R}^{3}\\}_{i=1}^{6}$ as shown in Fig.
11. During the hand calibration procedure, we request each participant to
touch the calibration structure’s corners with their fingertips. We instruct
them to touch specified multiple corners at each step using two or three
fingertips. A participant undergoes 23 steps of such touching processes per-
hand. The Tab. 4 comprises hand calibration instructions for subjects to
follow. Corner # is a set of two or three target corner numbers of the
calibration structure which the subject should contact with their fingertips.
In the Tab. 4, the orders of fingers to touch the target corners are specified
with numbers corresponding to each finger, which are (1:Thumb, 2:Index,
3:Middle, 4:Ring, 5:Little). An example of the hand calibration procedure is
shown in our supplementary video.

###  C.4 Implementation Details on Hand Calibration

Figure 12: Hand Skeleton Hierarchy

Here we describe the details of the hand calibration method. As described in
5.2, optimization parameters are hand skeleton configuration
$\mathcal{H}=\\{\mathcal{S}^{h},\mathcal{O}^{h}\\}$ and positions of 3D
markers in the local hand-centric coordinate $\mathcal{M}^{h}$. We empirically
decide the target range of the optimization skeleton to the palm area shown in
Fig. 12 and add constraints that limit the skeleton scales($s_{i}$) for each
skeleton segment $i$ between $0.8\leq s_{i}\leq 1.2$, and additional skeleton
offset value($\delta_{j}$) for target joints $j$ with $\left|\delta\right|\leq
0.01$ in meter scale to avoid unnatural deformation of hand skeleton. The
location of hand markers $\mathcal{M}^{h}$ is optimized through a total of 150
iterations. The skeleton scale and additional offset are optimized starting
from 50 and 100 iterations each. We use three losses, $L_{tip}$ to measure the
Euclidean distance from hand tips to paired corners, $L_{wrist}$ to measure
the distance from wrist location from body motion capture device and the wrist
position computed by hand marker position and $L_{pen}$ to measure penetration
of hand to the calibration structure. The penetration loss is a cosine
similarity between the calibration structure’s normal vector and the target
corner-to-hand tip vector. In summed loss
$\lambda_{t}L_{tip}+\lambda_{w}\mathcal{L}_{wrist}+\lambda_{p}\mathcal{L}_{pen}$,
losses are weighted equally by $\lambda_{t}=1,\lambda_{w}=1,\lambda_{p}=1$.
But they are manually adjusted based on the touch accuracy and body
calibration accuracy of the optimized result. After the alignment process, the
average Euclidean distance between the corner and the target finger tip
results in $0.83$ (in centimeters).

##  Appendix D Capture Scenario Details

Actions are placed in an arbitrary order so that each scenario has a different
order of actions. However, because certain action sets necessitate a more
natural ordering such as “drink water from the cup” and “pour water from the
kettle into the cup”, where the latter needs to be performed before the
former, we maintain the order for such actions sets to mirror real-life human
behavior. For cooking-related actions, there are 7 sets of designated, ordered
actions, and one of them is randomly picked to construct the scenario. The set
of cooking-related actions is placed in between the arbitrarily ordered non-
cooking actions. To this end, every scenario is composed in semi-arbitrary
order to reflect diverse and realistic human behavior in a studio apartment
environment. Notably, the corresponding verbal instructions for the actions
are also recorded and annotated to the captured data. A list of action
instructions conveyed during captures is shown in Tab. 5.

Non-cooking actions
---
Bring the cup
Drink from the cup
Place the cup in the sink
Take the chair and sit
Take the laptop to the seat
Open the laptop
Type the keyboard on the laptop
Close the laptop
Take the book from bookshelf
Bring the kettle
Pour water into the cup
Place the kettle in the sink
Open the drawer and take item out
Put trash in the trashbin
Open the washingmachine and put laundry

Kitchen-related actions
---
Open the sink and take the cuttingboard and knife out
Open the refrigerator and take food out
Cut the food on the cuttingboard
Take the pot and place it on the stove
Turn on the gas stove
Put the sliced food into pot/pan
Season with salt
Turn off the gas stove
Place the pot to the diningtable
Take the pan and place it on the stove
Place the pan to the diningtable
Open the microwave
Open the refrigerator, take food out and put it in microwave
Close the microwave
Move the heated food from microwave to pot
Move the food from pot to the bowl
Place the bowl to the diningtable
Put the food into pot
Move the food from pan to pot
Move the heated food from microwave to pan

Table 5: List of actions during the capture

##  Appendix E Dataset Details

###  E.1 Dataset Contents

Scanned Object Mesh. 3D meshes of the objects in the room environment are
obtained via 3D scanner [12]. They are scaled to a metric scale and zero-
centered.

Object Articulation Information. Objects with articulation contain axis
$a_{e}$. If the part has a revolute joint, we include pivot point $p_{e}$
additionally. These are defined in the object canonical space and are utilized
in getting each object part-transformation toward the camera space.

Object-Marker Transformations. Since 3D marker cubes attached to each object
can fall apart from the object, we maintain the information per-sequence
capture and in case it is detached, we re-calculate the transformation again
after attaching to the part again.

Relative Orientation of Hand/Body Joints. Orientation of each hand and body
joints with respect to their parent joints are recorded and optimized via a
motion capture system.

3D Hand/Body Joint Positions in the Camera Space. With the positions of
markers attached to the body in the mocap space acquired via body alignment
protocol, translation and orientation to camera space are obtained using the
positions of corresponding markers in camera spaces. And we compute the
positions of two hands and body using the obtained translation and
orientation.

Verbal Instructions for Each Actions. For every capture, participants are
given instructions as to what actions they will be performing. It is conveyed
verbally and contains what object to interact with, and how to interact with
the object. The instructions are recorded and aligned with hand and body data.

Per-frame Contact Information. At each frame where contact between
Left/Right/Body and object occurs, the corresponding frame and object
category/body part information is recorded.

###  E.2 Sequence Visualization

Sampled data from our collected datasets are shown in Fig. 13. Corresponding
instructions are provided under the caption.

Object Name | Left Hand | Right Hand | Total
---|---|---|---
Cup | 0.039 | 0.042 | 0.075
Pan | 0.036 | 0.070 | 0.100
Pot | 0.060 | 0.068 | 0.080
Book | 0.024 | 0.020 | 0.033
Bowl | 0.101 | 0.069 | 0.145
Salt | 0.043 | 0.050 | 0.086
Sink | 0.084 | 0.068 | 0.123
Desk | 0.037 | 0.033 | 0.054
Knife | 0.019 | 0.080 | 0.096
Chair | 0.023 | 0.022 | 0.037
Potlid | 0.012 | 0.026 | 0.036
Kettle | 0.027 | 0.040 | 0.062
Laptop | 0.060 | 0.052 | 0.072
Drawer | 0.004 | 0.011 | 0.014
Gasstove | 0.006 | 0.018 | 0.024
Trashbin | 0.008 | 0.005 | 0.011
Microwave | 0.008 | 0.004 | 0.011
Bookshelf | 0.001 | 0.001 | 0.002
Diningtable | 0.016 | 0.016 | 0.026
Refrigerator | 0.017 | 0.036 | 0.043
Cuttingboard | 0.086 | 0.051 | 0.107
Washingmachine | 0.011 | 0.009 | 0.017
Table 6: Ratio of contacted frames w.r.t tracked frames

###  E.3 Contact Information and Visualization

Contact information between the hand and object surfaces is acquired by
calculating SDF of object mesh and transforming human joints per frame in
camera space toward each object canonical space by applying transformation
$T_{mar\rightarrow obj}^{-1}$. Then, for the joints that are closer than the
contact threshold distance from the object surface, we decide they are in
contact from each other. We report per-object ratio of contacted frames w.r.t
tracked number of frames in Tab. 6 where contact has been counted for left and
right hand separately. Also in order to visualize contact information, we fit
MANO [52] and SMPLX [46] pose and shape parameters in camera coordinate space
using each body joint positions. Using Fitted MANO and SMPLX model per-frame,
we compute SDF of body, hands and objects and query distances at each mesh
vertices w.r.t each other. Then the contacted vertices at each mesh surface
are acquired by thresholding the distance value. For the contacted points,
distances are visualized as intensity images in Fig. 15 and Fig. 14.

Figure 13: (Column1) Open the refrigerator, take food out and put it in the
microwave. Close the microwave. (Column2) Take the chair and sit. Open the
laptop. Type on the laptop. Open the drawer and take an item out. (Column3)
Open the refrigerator and take food out. Cut the food on the cutting board.
Take the pot and place it on the stove. (Column4) Turn on the gas-stove.
Season with salt. (Column 5) Pour water into the cup. Drink from the cup. Take
the chair and sit. Open the laptop. Figure 14: Example of Hands and Objects
Contact Visualization in Captured data. (Column 1) Interaction scene where the
subject interact with multiple objects while navigating the room environment.
(Column 2) Close-up version of the scene. (Column 3,4,5,6) Contact intensity
for each hand palm (Left, Right), cup and kettle. Figure 15: Example of Body
and Objects Contact Visualization in Captured data. (Column 1) Interaction
scene where the subject interact with multiple objects while navigating the
room environment. (Column 2) Close-up version of the scene. (Column 3,4,5)
Contact intensity of body and each hand palm (Left, Right). (Column 6,7,8)
Contact intensity of chair, cup and laptop.