# CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object
REarrangement

Chengwen Zhang 1,2, Yun Liu∗1,3,4, Ruofan Xing1, Bingda Tang1, Li Yi 1,3,4
1Institute for Interdisciplinary Information Sciences, Tsinghua University
2Beijing University of Posts and Telecommunications
3Shanghai Artificial Intelligence Laboratory 4Shanghai Qi Zhi Institute
<EMAIL_ADDRESS>
{yun-liu22, xingrf20<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Equal contributionCorresponding author

###### Abstract

Understanding how humans cooperatively rearrange household objects is critical
for VR/AR and human-robot interaction. However, in-depth studies on modeling
these behaviors are under-researched due to the lack of relevant datasets. We
fill this gap by presenting CORE4D, a novel large-scale 4D human-object-human
interaction dataset focusing on collaborative object rearrangement, which
encompasses diverse compositions of various object geometries, collaboration
modes, and 3D scenes. With 1K human-object-human motion sequences captured in
the real world, we enrich CORE4D by contributing an iterative collaboration
retargeting strategy to augment motions to a variety of novel objects.
Leveraging this approach, CORE4D comprises a total of 11K collaboration
sequences spanning 3K real and virtual object shapes. Benefiting from
extensive motion patterns provided by CORE4D, we benchmark two tasks aiming at
generating human-object interaction: human-object motion forecasting and
interaction synthesis. Extensive experiments demonstrate the effectiveness of
our collaboration retargeting strategy and indicate that CORE4D has posed new
challenges to existing human-object interaction generation methodologies. Our
dataset and code are available at
https://github.com/leolyliu/CORE4D-Instructions.

##  1 Introduction

Humans frequently rearrange household items through multi-person collaboration
, such as moving a table or picking up an overturned chair together. Analyzing
and synthesizing these diverse collaborative behaviors could be widely
applicable in VR/AR, human-robot interaction [77, 55, 54], and dexterous [10,
90, 76, 99] and humanoid [53, 18, 86, 42] manipulation. However, understanding
and modeling these interactive motions have been under-researched due to the
lack of large-scale, richly annotated datasets. Most existing human-object and
hand-object interaction datasets focus on individual behaviors [71, 3, 49, 94,
20, 34, 100, 40, 46, 98] and two-person handovers [96, 81, 48]. But these
datasets typically encompass a limited number of object instances, thus
struggling to support generalizable interaction understanding across diverse
object geometries. Scaling up precise human-object interaction data is
challenging. While vision-based human-object motion tracking methods [3, 85]
have made significant progress, they still struggle with low fidelity in
severe occlusion, common in multi-human collaboration scenes. However, mocap
[34, 40] is expensive and hard to scale up to cover a large number of objects
to be rearranged. We want to curate a large-scale category-level human-object-
human (HOH) interaction dataset with high motion quality in a cost-efficient
manner.

We observe that HOH collaborations mainly vary in two aspects including the
temporal collaboration patterns of two humans and the spatial relations
between human and object. The temporal collaboration patterns could vary in
many ways depending on scene complexity, motion range, and collaboration mode.
In contrast, the spatial relations between human and object tend to possess
strong homogeneity when facing objects from the same category, e.g., two
persons holding the two sides of a chair. This allows retargeting interactions
involving one specific instance to another using automatic algorithms,
avoiding the need to capture interactions with thousands of same-category
objects in the real world. The above observations make it possible for us to
leverage expensive motion capture systems to capture only humans’ diverse
temporal collaboration patterns while leaving the richness of human-object
spatial relations to automatic spatial retargeting algorithms.

Using these insights, we build a large-scale dataset, CORE4D, encompassing a
wide range of human-object interactions for collaborative object
rearrangement. CORE4D includes various types of household objects,
collaboration modes, and 3D environments. Our data acquisition strategy
combines mocap-based capturing and synthetic retargeting, allowing us to scale
the dataset effectively. The retargeting algorithm transfers spatial relation
between human and object to novel object geometries while preserving temporal
pattern of human collaboration. As a result, CORE4D includes 1K real-world
motion sequences (CORE4D-Real) paired with videos and 3D scenes, and 10K
synthetic collaboration sequences (CORE4D-Synthetic) covering 3K diverse
object geometries.

We benchmark two tasks for generating human-object collaboration: (1) motion
forecasting [12, 88] and (2) interaction synthesis [73, 40] on CORE4D,
revealing challenges in modeling human behaviors, enhancing motion
naturalness, and adapting to new object geometries. Ablation studies
demonstrate the effectiveness of our hybrid data acquisition strategy, and the
quality and value of CORE4D-Synthetic, highlighting its role in helping to
improve existing motion generation methods.

In summary, our main contributions are threefold: (1) We present CORE4D, a
large-scale 4D HOH interaction dataset for collaborative object rearrangement.
(2) We propose a novel hybrid data acquisition methodology, incorporating
real-world data capture and synthetically collaboration retargeting. (3) We
benchmark two tasks for collaboration generation, revealing new challenges and
research opportunities.

##  2 Related Work

###  2.1 Human-object Interaction Datasets

Tremendous progress has been made in constructing human-object interaction
datasets. To study how humans interact with 3D scenes, various widely-used
datasets record human movements and surrounding scenes separately, regarding
objects as static [4, 66, 28, 80, 27, 44, 17, 102, 104, 26, 79, 2, 31, 91, 16,
25, 72, 92, 35, 97] or partially deformable [43] without pose changes. For
dynamic objects, recent works [5, 71, 15, 3, 32, 34, 100, 40, 81, 48, 105,
101, 45] have captured human-object interaction behaviors with different
focuses. Table 1 generally summarizes the characteristics of 4D human-object-
interaction datasets. To support research for vision-based human-object motion
tracking and shape reconstruction, a line of datasets [15, 3, 32, 34, 100,
105, 101, 45] presents human-object mesh annotations with multi-view RGB or
RGBD signals. With the rapid development of human-robot cooperation, several
works [5, 71, 81, 48] focus on specific action types such as grasping [71] and
human-human handover [5, 81, 48]. Our dataset uniquely captures multi-person
and object collaborative motions, category-level interactions, and both
egocentric and allocentric views, offering comprehensive features with the
inclusion of both real and synthetic datasets.

Table 1: Comparison of CORE4D with existing 4D human-object interaction datasets. dataset | multi- | collaboration | category- | egocentric | RGBD | $\\#$view | mocap | $\\#$object | $\\#$sequence
---|---|---|---|---|---|---|---|---|---
human | level
Carfi et.al. [5] | ✓ |  |  |  | ✓ | 1 | ✓ | 10 | 1.1K
GRAB [71] |  |  |  |  |  | - | ✓ | 57 | -
GraviCap [15] |  |  |  |  |  | 3 |  | 4 | 9
BEHAVE [3] |  |  |  |  | ✓ | 4 |  | 20 | 321
InterCap [32] |  |  |  |  | ✓ | 6 |  | 10 | 223
CHAIRS [34] |  |  | ✓ |  | ✓ | 4 | ✓ | 81 | 1.4K
HODome [100] |  |  |  |  |  | 76 | ✓ | 23 | 274
Li et.al. [40] |  |  |  |  |  | - | ✓ | 15 | 6.1K
HOH [81] | ✓ |  |  |  | ✓ | 8 |  | 136 | 2.7K
CoChair [48] | ✓ |  | ✓ |  |  | - | ✓ | 8 | 3.0K
FORCE [105] |  |  |  |  | ✓ | 1 | ✓ | 8 | 450
HOI-M3 [101] | ✓ |  |  |  |  | 42 | ✓ | 90 | 199
CORE4D-Real | ✓ | ✓ | ✓ | ✓ | ✓ | 5 | ✓ | 37 | 1.0K
CORE4D-Synthetic | ✓ | ✓ | ✓ |  |  | - | - | 3.0K | 10K

###  2.2 Human Interaction Retargeting

Human interaction retargeting focuses on how to apply human interactive
motions to novel objects in human-object interaction scenarios. Existing
methodologies [37, 65, 94, 67, 83, 8, 33] are object-centric, which propose
first finding contact correspondences between the source and the target
objects and then adjusting human motion to touch specific regions on the
target object via optimization. As crucial guidance of the result, contact
correspondences are discovered by aligning either surface regions [65, 94,
83], spatial maps [37, 33], distance fields [8], or neural descriptor fields
[67] between the source and the target objects, which are all limited to
objects with similar topology and scales. Our synthetic data generation
strategy incorporates object-centric design [94] with novel human-centric
contact selection, creating a chance to adapt to these challenging objects
using human priors.

###  2.3 Human-object Interaction Generation

Human-object interaction generation is an emerging research topic that aims to
synthesize realistic human-object motions conditioned on surrounding 3D
scenes, known object trajectories, or action types. To generate humans
interacting with static 3D scenes, POSA [29] and COINS [106] synthesize static
human poses with CVAE [68], while a line of work [69, 22, 84, 70, 104, 38,
103] further presents dynamic human motions by auto-regressive manners [69,
104], diffusion models [38], or two-stage designs that first generates start
and end poses and then interpolates motion in-between [22, 84, 70, 103].
InterDiff [88] and OMOMO [40] further fulfill this task for dynamic objects.
To generate human-object interaction under specific action descriptions,
recent works [19, 39, 63, 82, 89] extract text features with pretrained CLIP
[64] encoders or LLMs [57, 74] and use them to guide diffusion models [30].

##  3 Constructing CORE4D

CORE4D is a large-scale 4D human-object-human interaction dataset acquired in
a novel hybrid scheme, comprising CORE4D-Real and CORE4D-Synthetic.
CORE4D-Real is captured (Section 3.1) and annotated (Section 3.2) from
authentic collaborative scenarios. It provides human-object-human poses,
allocentric RGB-D videos, egocentric RGB videos, and 2D segmentation across
1.0K sequences accompanied by 37 object models. To augment spacial relation
between human and object, we present an innovative collaboration retargeting
technique in Section 3.3, integrating CORE4D-Real with CORE4D-Synthetic,
thereby expanding our collection with an additional 10K sequences and 3K rigid
objects. Detailed characteristics such as data diversities are demonstrated in
Section 3.4.

###  3.1 CORE4D-Real Data Capture

Figure 1: Data capturing system. (a) demonstrates the wearing of mocap suits
and the positioning of the egocentric camera. (b) shows an object with four
markers. (c) illustrates the data capturing system and camera views. Figure 2:
Dataset overview.

To collect precise human-object motions with visual signals, we set up a
hybrid data capturing system shown in Fig. 1, consisting of an inertial-
optical mocap system, four allocentric RGB-D cameras and a camera worn by
persons for egocentric sensing. The frequency of our system is 15 FPS.

Inertial-optical Mocap System. To accurately capture human-object poses in
multi-person collaboration scenarios, often involving severe occlusion, we use
an inertial-optical mocap system [56] inspired by CHAIRS [34] This system
includes 12 infrared cameras, mocap suits with 8 inertial-optical trackers and
two data gloves per person, and markers of a 10mm radius. The mocap suits
capture Biovision Hierarchy (BVH) skeletons of humans, while markers attached
to the objects track object motion.

Visual Sensors. Kinect Azure DK cameras are integrated to capture allocentric
RGB-D signals, and an Osmo Action3 is utilized to capture egocentric color
videos. The resolution of all the visual signals is 1920x1080. Cameras are
calibrated by the mocap system and synchronized via timestamp. Details on
camera calibration and synchronization are provided in the supplementary
material.

Object Model Acquisition. CORE4D-Real includes 37 3D models of rigid objects
spanning six household object categories. Each object model is constructed by
an industrial 3D scanner with up to 100K triangular faces. We additionally
adopt manual refinements on captured object models to remove triangle outliers
and improve accuracy.

Privacy Protection. To ensure participant anonymity, blurring is applied to
faces [58] in RGB videos, and fake facial meshes are generated via SMPL-X
[61]. The participants all consented to releasing CORE4D, and were also
notified of their right to have their data removed from CORE4D at any time.

###  3.2 CORE4D-Real Data Annotation

Object Pose Tracking. To acquire the 6D pose of a rigid object, we attach four
to five markers to the object’s surface. The markers formulate a virtual rigid
that the mocap system can track. With accurate localization of the object
manually, the object pose can be precisely determined by marker positions
captured by the infrared cameras.

Human Mesh Acquisition. Aligning with existing dataset efforts [34, 40], we
retarget the BVH [52] human skeleton to the widely-used SMPL-X [61]. SMPL-X
[61] formulates a human mesh as $D_{\text{smplx}}=M(\beta,\theta)$. The body
shape $\beta\in\mathbb{R}^{10}$ are optimized to fit the constraints on
manually measured human skeleton lengths. With $\beta$ computed, we optimize
the full-body pose $\theta\in\mathbb{R}^{159}$ with the loss function:

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{reg}}+\mathcal{L}_{j\text{3D}}+\mathcal{L}_{j\text{Ori}}+\mathcal{L}_{\text{smooth}}+\mathcal{L}_{h\text{3D}}+\mathcal{L}_{h\text{Ori}}+\mathcal{L}_{\text{contact}},$
(1)

where $\mathcal{L}_{\text{reg}}$ ensures the simplicity of the results and
prevents unnatural, significant twisting of the joints.
$\mathcal{L}_{j\text{3D}}$ and $\mathcal{L}_{j\text{Ori}}$ encourage the
rotation of joints and the global 3D positions to closely match the ground
truth. $\mathcal{L}_{h\text{3D}}$ and $\mathcal{L}_{h\text{Ori}}$ guide the
positioning and orientation of the fingers. $\mathcal{L}_{\text{smooth}}$
promotes temporal smoothness. $\mathcal{L}_{\text{contact}}$ encourages
realistic contact between the hands and objects. Then using SMPL-X [61]
$M(\beta,\theta,\Phi):\mathbb{R}^{|\theta|\times|\beta|}\mapsto\mathbb{R}^{3N}$
to generate human mesh. Details on the loss functions are presented in the
supplementary material.

2D Mask Annotation. We offer automatic 2D segmentation for individuals and the
manipulated object to aid in predictive tasks like vision-based human-object
pose estimation [3, 85]. We first use DEVA [9] to segment human and object
instances in a captured interaction image with text prompts. Then, we render
human and object meshes separately on each image and select the instance with
the highest Intersection-over-Union (IoU) for mask annotation.

Figure 3: Collaboration retargeting pipeline. We propose a collaboration
retargeting algorithm by iteratively refining interaction motion. The
algorithm takes input source-target pair. First, we sample contact candidates
from whole CORE4D-Real contact knowledge on source. For each contact
candidate, we apply contact retargeting to propagate contact candidates to
contact constraints on target. Sampled motion from CORE4D-Real provides a
high-level collaboration pattern, together with low-level contact constraints,
we obtain interaction candidates from interaction retargeting. Then, the human
pose discriminator selects the optimal candidates, prompting a contact
constraints update via beam search. After multiple iterations, the process
yields augmented interactions. This iterative mechanism can effectively get a
reasonable one from numerous contact constraints and ensures a refined
interaction, enhancing the dataset’s applicability across various scenarios.

###  3.3 CORE4D-Synthetic Data Generation

In order to enrich the diversities of object geometries and human-object
spatial relations, our retargeting algorithm transfers real interactions to
ShapeNet [6] objects of the same category, thereby significantly expanding the
dataset regarding the object’s diversity. When transferring interactions
across objects, contact points are always the key and it is important to
consider whether they can be properly transferred with consistent semantics on
new objects [95, 107]. However, we find this insufficient when object
geometries vary largely and correspondences become hard to build. We thus
tackle interaction retargeting from a novel human-centric perspective where
good contact points should support natural human poses and motions. We realize
this idea through the pipeline depicted in Figure 3, which comprises three key
components. First, object-centric contact retargeting uses whole contact
knowledge from CORE4D-Real to obtain accurate contact with different objects.
Second, contact-guided interaction retargeting adapts motion sequences to new
object geometries while considering the contact constraints. Third, a human-
centric contact selection evaluates poses from interaction candidates to
select the most plausible contacts.

Object-centric Contact Retargeting. To acquire reasonable human poses, contact
constraints on the target object are essential. We draw inspiration from Tink
[94] and train DeepSDF on all objects’ signed distance fields (SDFs). For
source object SDF $O_{s}$ and target object SDF $O_{t}$, we first apply linear
interpolation on their latent vectors $o_{s}$ and $o_{t}$ and obtain $N$
intermediate vectors $o_{i}=\frac{N+1-i}{N+1}o_{s}+\frac{i}{N+1}o_{t}(1\leq
i\leq N)$. We then decode $o_{i}$ to its SDF $O_{i}$ via the decoder of
DeepSDF, and reconstruct the corresponding 3D mesh $M_{i}$ using the Marching
Cubes algorithm [50]. Thereby get mesh sequence
$\mathcal{M}=[\textit{source},M_{1},M_{2},...,M_{N},\textit{target}]$ and
successively transfer contact positions between every two adjacent meshes in
$\mathcal{M}$ via Nearest-neighbor searching. In addition, we leverage all
contact candidates from CORE4D-Real on source to form a pool of contact
candidates and transfer them to target as contact constraints.

Contact-guided Interaction Retargeting. For each contact constraint,
interaction retargeting aims to transfer human interaction from source to
target. To greatly enforce the consistency of interaction motion, we optimize
variables including the object rotations $R_{o}\in\mathbb{R}^{N\times 3}$ and
translations $T_{o}\in\mathbb{R}^{N\times 3}$, human poses
$\theta_{1,2}\in\mathbb{R}^{2\times N\times 153}$, translation
$T_{1,2}\in\mathbb{R}^{2\times N\times 3}$ and orientation
$O_{1,2}\in\mathbb{R}^{2\times N\times 3}$ on the SMPL-X [61]. $N$ is the
frame number.

We first estimate the target’s motion $\\{R_{o},T_{o}\\}$ by solving an
optimization problem as follows:

$\displaystyle
R_{o},T_{o}\longleftarrow\mathop{\operatorname{argmin}}\limits_{R_{o},T_{o}}(\mathcal{L}_{f}+\mathcal{L}_{\text{spat}}+\mathcal{L}_{\text{smooth}}),$
(2)

where fidelity loss $\mathcal{L}_{f}$ evaluates the difference of the target’s
rotation and translation against the source, restriction loss
$\mathcal{L}_{\text{spat}}$ penalizes target’s penetration with the ground,
and smoothness loss $\mathcal{L}_{\text{smooth}}$ constrains the target’s
velocities between consecutive frames.

Given the target’s motion and contact constraints, we then transfer humans’
interactive motion $\\{\theta_{1,2},T_{1,2},O_{1,2}\\}$ from the source to the
target by solving another optimization problem as follows:

$\displaystyle\theta_{1,2},T_{1,2},O_{1,2}\longleftarrow\mathop{\operatorname{argmin}}\limits_{\theta_{1,2},T_{1,2},O_{1,2}}(\mathcal{L}_{j}+\mathcal{L}_{c}+\mathcal{L}_{\text{spat}}+\mathcal{L}_{\text{smooth}}),$
(3)

where fidelity loss $\mathcal{L}_{\text{j}}$ evaluates the difference in human
joint positions before and after the transfer, contact loss $\mathcal{L}_{c}$
computes the difference between human-object contact regions and the contact
constraints, $\mathcal{L}_{\text{spat}}$ and $\mathcal{L}_{\text{smooth}}$
ensures the smoothness of human motion. Details on the loss designs and their
motivations are provided in the supplementary material.

Human-centric Contact Selection. Selecting reasonable contact constraints
efficiently is challenging due to their large scale and the time-consuming
interaction retargeting. We address this challenge by developing a beam search
algorithm to select contact constraints from a human-centric perspective.
Specifically, we train a human pose discriminator inspired by GAN-based motion
generation strategies [93, 87]. To train the discriminator, we build a
pairwise training dataset, with each pair consisting of one positive human
pose sample and one negative sample. Positive samples are encouraged to get
higher scores than negative ones. We use CORE4D-Real as positive samples. We
add 6D pose noise $\Delta(\alpha,\beta,\gamma,x,y,z)$ on target motion, and
regard corresponding human motions generated by contact-guided interaction
retargeting as negative samples. The loss function is:

$\displaystyle\mathcal{L}_{\text{ranking}}=-\log(\sigma(R_{\text{pos}}-R_{\text{neg}}-m(S_{\text{pos}},S_{\text{neg}}))),$
(4)

where $S_{\text{pos}}$ and $S_{\text{neg}}$ denote inputs for positive and
negative samples respectively, with $R_{\text{pos}}$ and $R_{\text{neg}}$
being their corresponding discriminator scores. $\sigma$ is Sigmoid function,
and $m(S_{\text{pos}},S_{\text{neg}})=||\Delta(\alpha,\beta,\gamma,x,y,z)||$
is human-guide margin [59] between positive and negative poses. This margin
could explicitly instruct the discriminator to yield more significant
disparities across different poses.

To ensure the realism of human interactions, we also introduce an
interpenetration penalty. We prioritize those with the highest discriminator
scores while ensuring acceptable levels of interpenetration as the optimal
contact constraints.

###  3.4 Dataset Characteristics

Figure 4: Dataset statistics. (a) shows some object samples from five
categories. Bars in (b) indicate when the person is in contact with the object
during the entire collaborative object rearrangement interaction process. (c)
presents the proportion of collaboration modes in the dataset.

To better model collaborative object rearrangement interactions, we focus on
diversifying our dataset in several vital areas: object geometries,
collaboration modes, and 3D scenes. These ensure a comprehensive
representation of real-world interactions.

Diversity in Object Geometries. We design six object categories to cover the
main collaborative object rearrangement interaction scenarios as Fig. 4(a).
Categories with relatively simple geometry, uniformity, and typically
exhibiting symmetry include box, board, barrel, and stick. Categories with
more complex geometries and significant individual differences include chair
and desk.

Diversity in Collaboration Modes. We define five human-human collaboration
modes in collaborative object rearrangement. Each mode represents a unique
form of collaboration between two individuals, providing a new perspective and
possibilities for understanding and researching collaborative behaviors. At
first, we define the person with the egocentric camera as Person 2, and the
other as Person 1. Collaborative carrying tasks are divided by whether Person
2 knows the goal or not. Tasks of handover and solely move alternate between
the two participants. In join and leave tasks, Person 2 will either join in to
help or leave halfway through, respectively.

Diversity in 3D Scenes. Surrounding scenarios are set up with varying levels
of scene complexity: no obstacle, single obstacle, and many obstacles (more
than one). Participants are asked to navigate through these randomly placed
obstacles by their own means. We observe that this typically involved
behaviors including bypassing, going through, stepping over, or moving
obstacles aside.

##  4 Experiments

In this section, we first present the train-test split of CORE4D (Section
4.1). We then propose two benchmarks for generating human-object
collaboration: human-object motion forecasting (Section 4.2), and interaction
synthesis (Section 4.3). Finally, Section 4.4 presents extensive studies on
the collaboration retargeting approach.

###  4.1 Data Split

We construct a training set from a random assortment of real objects,
combining their real motions and corresponding synthetic data. We also create
two test sets from CORE4D-Real for non-generalization and inner-category
generalization studies. Test set S1 includes interactions with training set
objects, while S2 features interactions with new objects. CORE4D-Synthetic is
not included in the test set, avoiding potential biases from the retargeting
algorithm. Details are shown in supplementary material.

###  4.2 Human-object Motion Forecasting

Forecasting 4D human motion [23, 62, 51, 24] is a crucial problem with
applications in VR/AR and embodied perception [36]. Current research [13, 1,
77, 88] is limited to individual behaviors due to data constraints. Our work
expands this by using diverse multi-person collaborations, making the
prediction problem both intriguing and challenging.

Task Formulation. Given the object’s 3D model and human-object poses in
adjacent 15 frames, the task is to predict their subsequent poses in the
following 15 frames. The human pose $P_{h}\in\mathbb{R}^{23\times 3}$
represents joint rotations of the SMPL-X [61] model, while the object pose
$P_{o}=\\{R_{o}\in\mathbb{R}^{3},T_{o}\in\mathbb{R}^{3}\\}$ denotes 3D
orientation and 3D translation of the rigid object model.

Evaluation Metrics. Following existing motion forecasting works [12, 78, 88],
we evaluate human joints position error $J_{e}$, object translation error
$T_{e}$, object rotation error $R_{e}$, human-object contact accuracy
$C_{\text{acc}}$, and penetration rate $P_{r}$. Details are provided in the
supplementary material.

Methods, Results, and Analysis. We evaluate three state-of-the-art motion
forecasting methods, MDM [73], InterDiff [88], and CAHMP [12]. Table 2
quantitatively shows these methods reveal a consistent drop in performance for
unseen objects (S2) versus seen ones (S1) regarding human pose prediction.
Meanwhile, errors in object pose prediction remain similar. This highlights
the challenges in generalizing human collaborative motion for novel object
shapes.

Table 2: Quantitative results on human-object motion forecasting. Test Set | Method | Human | Object | Contact
---|---|---|---|---
$J_{e}$ (mm, $\downarrow$) |  $T_{e}$ (mm, $\downarrow$) |  $R_{e}$ (∘, $\downarrow$) |  $C_{\text{acc}}$ ($\%$, $\uparrow$) |  $P_{r}$ ($\%$, $\downarrow$)
S1 | MDM [73] | 170.8 ($\pm$0.9) | 136.8 ($\pm$0.1) | 10.7 ($\pm$0.1) | 84.9 ($\pm$ 0.2) | 0.3 ($\pm$ 0.0)
InterDiff [88] | 170.8 ($\pm$0.9) | 135.1 ($\pm$0.1) | 10.2 ($\pm$0.1) | 84.9 ($\pm$ 0.2) | 0.3 ($\pm$ 0.0)
CAHMP [12] | 169.4 ($\pm$0.3) | 110.3 ($\pm$0.1) | 9.0 ($\pm$0.1) | - | -
S2 | MDM [73] | 186.4 ($\pm$0.7) | 136.0 ($\pm$0.2) | 11.1 ($\pm$0.0) | 88.0 ($\pm$ 0.0) | 0.3 ($\pm$ 0.0)
InterDiff [88] | 186.4 ($\pm$0.7) | 133.6 ($\pm$0.2) | 10.7 ($\pm$ 0.1) | 88.0 ($\pm$ 0.0) | 0.3 ($\pm$ 0.0)
CAHMP [12] | 170.5 ($\pm$0.3) | 112.9 ($\pm$0.1) | 9.5 ($\pm$0.1) | - | -

###  4.3 Interaction Synthesis

Generating human-object interaction [40, 19, 39, 63] is an emerging research
topic benefiting human avatar animation and human-robot collaboration [11,
55]. With extensive collaboration modes and various object categories, CORE4D
constitutes a knowledge base for studying generalizable algorithms of human-
object-human interactive motion synthesis.

Task Formulation. Following recent studies [70, 40], we define the task as
object-conditioned human motion generation. Given an object geometry sequence
$G_{o}\in\mathbb{R}^{T\times N\times 3}$, the aim is to generate corresponding
two-person collaboration motions $M_{h}\in\mathbb{R}^{2\times T\times 23\times
3}$. This involves frame numbers $T$, object point clouds $G_{o}$, and human
pose parameters for the SMPL-X [61] model.

Evaluation Metrics. Following individual human-object interaction synthesis
[40], we evaluate human joint position error $RR.J_{e}$, object vertex
position error $RR.V_{e}$, and human-object contact accuracy $C_{\text{acc}}$.
The FID score ($FID$) is leveraged to quantitatively assess the naturalness of
synthesized results. Details of the metric designs are presented in the
supplementary material.

Methods, Results, and Analysis. We utilize two advanced generative models, MDM
[73] and OMOMO [40], as baselines. MDM is a one-stage conditional motion
diffusion model, while OMOMO is a two-stage approach with hand positions as
intermediate results. Quantitative evaluations reveal larger errors in OMOMO
when modeling multi-human collaboration compared to individual interaction
synthesis by Li et al. [40]. Furthermore, the synthesized results have a
higher FID than real motion data, indicating challenges in motion naturalness.

Table 3: Quantitative results on interaction synthesis. Test Set | Method |  $RR.J_{e}$ (mm, $\downarrow$) |  $RR.V_{e}$ (mm, $\downarrow$) |  $C_{\text{acc}}$ ($\%$, $\uparrow$) |  $FID$ ($\downarrow$)
---|---|---|---|---|---
S1 | MDM [73] | 138.0 ($\pm$ 0.3) | 194.6 ($\pm$ 0.2) | 76.9 ($\pm$ 0.5) | 7.7 ($\pm$ 0.2)
OMOMO [40] | 137.8 ($\pm$ 0.2) | 196.7 ($\pm$ 0.3) | 78.2 ($\pm$ 0.5) | 8.3 ($\pm$ 0.6)
S2 | MDM [73] | 145.9 ($\pm$ 0.2) | 208.2 ($\pm$ 0.2) | 76.7 ($\pm$ 0.1) | 7.7 ($\pm$ 0.2)
OMOMO [40] | 145.2 ($\pm$ 0.6) | 209.9 ($\pm$ 1.0) | 77.8 ($\pm$ 0.3) | 8.3 ($\pm$ 1.0)

###  4.4 Collaboration Retargeting

Table 4: Ablation study. CC denotes contact candidates. D denotes the human pose discriminator. CCU denotes the contact candidate update. Comparisons | Designs | Penetration | User Preferences
---|---|---|---
CC | D | CCU | Distance | Contact | Motion
(mm, $\downarrow$) | A/B/Approx.(%, $\uparrow$)
Abl.1 | A |  |  |  | 6.06 | 7.8/88.9/3.3 | 3.3/84.4/12.3
B | ✓ | ✓ |  | 2.38
Abl.2 | A | ✓ |  |  | 5.49 | 1.2/91.4/7.4 | 3.2/85.1/11.7
B | ✓ | ✓ |  | 2.38
Abl.3 | A | ✓ | ✓ |  | 2.38 | 5.0/76.0/19.0 | 4.0/69.0/27.0
B | ✓ | ✓ | ✓ | 2.27

User Studies. We conduct user studies to examine the quality of
CORE4D-Synthetic in terms of naturalness of contact and human motion. Each
study comprises two collections, each with at least 100 sequences displayed in
pairs on a website. Users are instructed to assess the realism of human-object
contacts and the naturalness of human motions, and then select the superior
one in each pair separately. Recognizing the diversity of acceptable contacts
and motions, participants are permitted to deem the performances as roughly
equivalent.

Ablation on Contact Candidates. In Table 4.Abl.1, we only use the contact
points from a source trajectory for retargeting to the target instead of
resorting to the CORE4D-Real for many candidates, making the whole retargeting
process similar to the OakInk [94] method. We observe a sharp decline in both
physical plausibility and user preferences, indicating that our method
compensates for OakInk’s shortcomings in retargeting objects with significant
geometric and scale variations.

Ablation on Discriminator. In this ablation, as shown in Table 4.Abl.2, we
omit the human pose discriminator in the collaboration retargeting. We will
randomly choose a candidate from the contact candidates. There are obvious
performance drops, demonstrating the critical role of the human pose
discriminator in selecting appropriate candidates.

Ablation on Contact Candidate Update. We exclude contact candidate update
process in Table 4.Abl.3 experiment. This removal has weakened our method’s
ability to search for optimal solutions on objects, resulting in a modest
degradation in penetration distance. The user study still exhibited a strong
bias, indicating a perceived decline in the plausibility of both contact and
motion. This ablation underscores the importance of contact candidate update
within our methodology.

Comparing CORE4D-Synthetic with CORE4D-Real. We assess the quality of
CORE4D-Synthetic by comparing it with CORE4D-Real through user study. In
conclusion, there is a 43% probability that users perceive the quality of both
options as comparable. Furthermore, in 14% of cases, users even exhibit a
preference for synthetic data. This indicates that the quality of our
synthetic data closely approximates that of real data.

Application of CORE4D-Synthetic. Table 5 compares the motion forecasting
ability of light-weighted CAHMP [14]. The test set is S2 defined in Section
4.1. We assess the quality of CORE4D-Synthetic by comparing No.A and No.B.
No.A even have better performance on object due to enriched spacial relation
between human and object in CORE4D-Synthetic. No.C shows the value of the
CORE4D-Synthetic by largely improving the performance. Details are in
supplementary material.

Table 5: Ablation on the incorporation of CORE4D-Synthetic on the motion forecasting task. No | Train Set | Human | Object
---|---|---|---
Total | Real | Synthetic |  $J_{e}$ (mm, $\downarrow$) |  $T_{e}$ (mm, $\downarrow$) |  $R_{e}$ (∘, $\downarrow$)
A | 1.0K | 0.1K | 0.9K | 127.7($\pm$ 0.7) | 121.7($\pm$ 0.7) | 8.04($\pm$ 0.4)
B | 1.0K | 1.0K | 0 | 127.0($\pm$ 0.8) | 120.5 ($\pm$ 0.6) | 9.48 ($\pm$ 0.1)
C | 5.0K | 1.0K | 4.0K | 116.2($\pm$ 0.3) | 112.1($\pm$ 0.2) | 6.99($\pm$ 0.1)

##  5 Conclusion and Limitations

We present CORE4D, a novel large-scale 4D human-object-human interaction
dataset for collaborative object rearrangement. It comprises diverse
compositions of various object geometries, collaboration modes, and
surrounding 3D scenes. To efficiently enlarge the data scale, we contribute a
hybrid data acquisition method involving real-world data capturing and a novel
synthetic data augmentation algorithm, resulting in 11K motion sequences
covering 37 real-world and 3K virtual objects. Extensive experiments
demonstrate the effectiveness of the data augmentation strategy and the value
of the augmented motion data. We benchmark human-object motion forecasting and
interaction synthesis on CORE4D, revealing new challenges and research
opportunities.

Limitations. One limitation is that outdoor scenes are not incorporated due to
the usage of the mocap system. Another limitation is that our data
augmentation strategy currently focuses on adopting collaboration to novel
object geometries while excluding human shape diversity. Integrating our
retargeting approach with human shape modeling could be an interesting future
direction.

## References

  * [1] Adeli, V., Ehsanpour, M., Reid, I., Niebles, J.C., Savarese, S., Adeli, E., Rezatofighi, H.: Tripod: Human trajectory and pose dynamics forecasting in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13390–13400 (2021)
  * [2] Araújo, J.P., Li, J., Vetrivel, K., Agarwal, R., Wu, J., Gopinath, D., Clegg, A.W., Liu, K.: Circle: Capture in rich contextual environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21211–21221 (2023)
  * [3] Bhatnagar, B.L., Xie, X., Petrov, I.A., Sminchisescu, C., Theobalt, C., Pons-Moll, G.: Behave: Dataset and method for tracking human object interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15935–15946 (2022)
  * [4] Cao, Z., Gao, H., Mangalam, K., Cai, Q.Z., Vo, M., Malik, J.: Long-term human motion prediction with scene context. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. pp. 387–404. Springer (2020)
  * [5] Carfì, A., Foglino, F., Bruno, B., Mastrogiovanni, F.: A multi-sensor dataset of human-human handover. Data in brief 22, 109–117 (2019)
  * [6] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
  * [7] Chao, Y.W., Yang, W., Xiang, Y., Molchanov, P., Handa, A., Tremblay, J., Narang, Y.S., Van Wyk, K., Iqbal, U., Birchfield, S., et al.: Dexycb: A benchmark for capturing hand grasping of objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9044–9053 (2021)
  * [8] Chen, Z.Q., Van Wyk, K., Chao, Y.W., Yang, W., Mousavian, A., Gupta, A., Fox, D.: Learning robust real-world dexterous grasping policies via implicit shape augmentation. arXiv preprint arXiv:2210.13638 (2022)
  * [9] Cheng, H.K., Oh, S.W., Price, B., Schwing, A., Lee, J.Y.: Tracking anything with decoupled video segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1316–1326 (2023)
  * [10] Christen, S., Kocabas, M., Aksan, E., Hwangbo, J., Song, J., Hilliges, O.: D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20577–20586 (2022)
  * [11] Christen, S., Yang, W., Pérez-D’Arpino, C., Hilliges, O., Fox, D., Chao, Y.W.: Learning human-to-robot handovers from point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9654–9664 (2023)
  * [12] Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F.: Context-aware human motion prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6992–7001 (2020)
  * [13] Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F.: Context-aware human motion prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6992–7001 (2020)
  * [14] Corona, E., Pumarola, A., Alenyà, G., Moreno-Noguer, F.: Context-aware human motion prediction (2020)
  * [15] Dabral, R., Shimada, S., Jain, A., Theobalt, C., Golyanik, V.: Gravity-aware monocular 3d human-object reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12365–12374 (2021)
  * [16] Dai, Y., Lin, Y., Lin, X., Wen, C., Xu, L., Yi, H., Shen, S., Ma, Y., Wang, C.: Sloper4d: A scene-aware dataset for global 4d human pose estimation in urban environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 682–692 (2023)
  * [17] Dai, Y., Lin, Y., Wen, C., Shen, S., Xu, L., Yu, J., Ma, Y., Wang, C.: Hsc4d: Human-centered 4d scene capture in large-scale indoor-outdoor space using wearable imus and lidar. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6792–6802 (2022)
  * [18] Dao, J., Duan, H., Fern, A.: Sim-to-real learning for humanoid box loco-manipulation. arXiv preprint arXiv:2310.03191 (2023)
  * [19] Diller, C., Dai, A.: Cg-hoi: Contact-guided 3d human-object interaction generation. arXiv preprint arXiv:2311.16097 (2023)
  * [20] Fan, Z., Taheri, O., Tzionas, D., Kocabas, M., Kaufmann, M., Black, M.J., Hilliges, O.: Arctic: A dataset for dexterous bimanual hand-object manipulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12943–12954 (2023)
  * [21] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., Iii, H.D., Crawford, K.: Datasheets for datasets. Communications of the ACM 64(12), 86–92 (2021)
  * [22] Ghosh, A., Dabral, R., Golyanik, V., Theobalt, C., Slusallek, P.: Imos: Intent-driven full-body motion synthesis for human-object interactions. In: Computer Graphics Forum. vol. 42, pp. 1–12. Wiley Online Library (2023)
  * [23] Guo, W., Bie, X., Alameda-Pineda, X., Moreno-Noguer, F.: Multi-person extreme motion prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13053–13064 (2022)
  * [24] Guo, W., Du, Y., Shen, X., Lepetit, V., Alameda-Pineda, X., Moreno-Noguer, F.: Back to mlp: A simple baseline for human motion prediction. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 4809–4819 (2023)
  * [25] Guzov, V., Chibane, J., Marin, R., He, Y., Sattler, T., Pons-Moll, G.: Interaction replica: Tracking human-object interaction and scene changes from human motion. arXiv preprint arXiv:2205.02830 (2022)
  * [26] Guzov, V., Mir, A., Sattler, T., Pons-Moll, G.: Human poseitioning system (hps): 3d human pose estimation and self-localization in large scenes from body-mounted sensors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4318–4329 (2021)
  * [27] Hassan, M., Ceylan, D., Villegas, R., Saito, J., Yang, J., Zhou, Y., Black, M.J.: Stochastic scene-aware motion prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11374–11384 (2021)
  * [28] Hassan, M., Choutas, V., Tzionas, D., Black, M.J.: Resolving 3d human pose ambiguities with 3d scene constraints. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2282–2292 (2019)
  * [29] Hassan, M., Ghosh, P., Tesch, J., Tzionas, D., Black, M.J.: Populating 3d scenes by learning human-scene interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14708–14718 (2021)
  * [30] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [31] Huang, C.H.P., Yi, H., Höschle, M., Safroshkin, M., Alexiadis, T., Polikovsky, S., Scharstein, D., Black, M.J.: Capturing and inferring dense full-body human-scene contact. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13274–13285 (2022)
  * [32] Huang, Y., Taheri, O., Black, M.J., Tzionas, D.: Intercap: Joint markerless 3d tracking of humans and objects in interaction. In: DAGM German Conference on Pattern Recognition. pp. 281–299. Springer (2022)
  * [33] Huang, Z., Xu, H., Huang, H., Ma, C., Huang, H., Hu, R.: Spatial and surface correspondence field for interaction transfer. arXiv preprint arXiv:2405.03221 (2024)
  * [34] Jiang, N., Liu, T., Cao, Z., Cui, J., Zhang, Z., Chen, Y., Wang, H., Zhu, Y., Huang, S.: Full-body articulated human-object interaction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9365–9376 (2023)
  * [35] Jiang, N., Zhang, Z., Li, H., Ma, X., Wang, Z., Chen, Y., Liu, T., Zhu, Y., Huang, S.: Scaling up dynamic human-scene interaction modeling. arXiv preprint arXiv:2403.08629 (2024)
  * [36] Kasahara, S., Konno, K., Owaki, R., Nishi, T., Takeshita, A., Ito, T., Kasuga, S., Ushiba, J.: Malleable embodiment: changing sense of embodiment by spatial-temporal deformation of virtual human body. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. pp. 6438–6448 (2017)
  * [37] Kim, Y., Park, H., Bang, S., Lee, S.H.: Retargeting human-object interaction to virtual avatars. IEEE transactions on visualization and computer graphics 22(11), 2405–2412 (2016)
  * [38] Kulkarni, N., Rempe, D., Genova, K., Kundu, A., Johnson, J., Fouhey, D., Guibas, L.: Nifty: Neural object interaction fields for guided human motion synthesis. arXiv preprint arXiv:2307.07511 (2023)
  * [39] Li, J., Clegg, A., Mottaghi, R., Wu, J., Puig, X., Liu, C.K.: Controllable human-object interaction synthesis. arXiv preprint arXiv:2312.03913 (2023)
  * [40] Li, J., Wu, J., Liu, C.K.: Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG) 42(6), 1–11 (2023)
  * [41] Li, J., Bian, S., Xu, C., Chen, Z., Yang, L., Lu, C.: Hybrik-x: Hybrid analytical-neural inverse kinematics for whole-body mesh recovery. arXiv preprint arXiv:2304.05690 (2023)
  * [42] Li, J., Nguyen, Q.: Kinodynamics-based pose optimization for humanoid loco-manipulation. arXiv preprint arXiv:2303.04985 (2023)
  * [43] Li, Z., Shimada, S., Schiele, B., Theobalt, C., Golyanik, V.: Mocapdeform: Monocular 3d human motion capture in deformable scenes. In: 2022 International Conference on 3D Vision (3DV). pp. 1–11. IEEE (2022)
  * [44] Liu, M., Yang, D., Zhang, Y., Cui, Z., Rehg, J.M., Tang, S.: 4d human body capture from egocentric video via 3d scene grounding. In: 2021 international conference on 3D vision (3DV). pp. 930–939. IEEE (2021)
  * [45] Liu, S., Li, Y.L., Fang, Z., Liu, X., You, Y., Lu, C.: Primitive-based 3d human-object interaction modelling and programming. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 3711–3719 (2024)
  * [46] Liu, Y., Yang, H., Si, X., Liu, L., Li, Z., Zhang, Y., Liu, Y., Yi, L.: Taco: Benchmarking generalizable bimanual tool-action-object understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21740–21751 (2024)
  * [47] Liu, Y., Yang, H., Si, X., Liu, L., Li, Z., Zhang, Y., Liu, Y., Yi, L.: Taco: Benchmarking generalizable bimanual tool-action-object understanding. arXiv preprint arXiv:2401.08399 (2024)
  * [48] Liu, Y., Chen, C., Yi, L.: Interactive humanoid: Online full-body motion reaction synthesis with social affordance canonicalization and forecasting. arXiv preprint arXiv:2312.08983 (2023)
  * [49] Liu, Y., Liu, Y., Jiang, C., Lyu, K., Wan, W., Shen, H., Liang, B., Fu, Z., Wang, H., Yi, L.: Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21013–21022 (2022)
  * [50] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: Seminal graphics: pioneering efforts that shaped the field, pp. 347–353 (1998)
  * [51] Mao, W., Hartley, R.I., Salzmann, M., et al.: Contact-aware human motion forecasting. Advances in Neural Information Processing Systems 35, 7356–7367 (2022)
  * [52] Meredith, M., Maddock, S., et al.: Motion capture file formats explained. Department of Computer Science, University of Sheffield 211, 241–244 (2001)
  * [53] Murooka, M., Kumagai, I., Morisawa, M., Kanehiro, F., Kheddar, A.: Humanoid loco-manipulation planning based on graph search and reachability maps. IEEE Robotics and Automation Letters 6(2), 1840–1847 (2021)
  * [54] Ng, E., Liu, Z., Kennedy, M.: Diffusion co-policy for synergistic human-robot collaborative tasks. IEEE Robotics and Automation Letters (2023)
  * [55] Ng, E., Liu, Z., Kennedy, M.: It takes two: Learning to plan for human-robot cooperative carrying. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). pp. 7526–7532. IEEE (2023)
  * [56] NOITOM INTERNATIONAL, INC: Noitom motion capture systems (2024), https://www.noitom.com.cn/
  * [57] OpenAI: Chatgpt (2023), https://chat.openai.com/
  * [58] OpenCV: opencv. https://github.com/opencv/opencv/blob/master/data/haarcascades/haarcascade_frontalface_default.xml (2013)
  * [59] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (2022)
  * [60] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: Learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 165–174 (2019)
  * [61] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from a single image. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10975–10985 (2019)
  * [62] Peng, X., Shen, Y., Wang, H., Nie, B., Wang, Y., Wu, Z.: Somoformer: Social-aware motion transformer for multi-person motion prediction. arXiv preprint arXiv:2208.09224 (2022)
  * [63] Peng, X., Xie, Y., Wu, Z., Jampani, V., Sun, D., Jiang, H.: Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553 (2023)
  * [64] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [65] Rodriguez, D., Behnke, S.: Transferring category-based functional grasping skills by latent space non-rigid registration. IEEE Robotics and Automation Letters 3(3), 2662–2669 (2018)
  * [66] Savva, M., Chang, A.X., Hanrahan, P., Fisher, M., Nießner, M.: Pigraphs: learning interaction snapshots from observations. ACM Transactions On Graphics (TOG) 35(4), 1–12 (2016)
  * [67] Simeonov, A., Du, Y., Tagliasacchi, A., Tenenbaum, J.B., Rodriguez, A., Agrawal, P., Sitzmann, V.: Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In: 2022 International Conference on Robotics and Automation (ICRA). pp. 6394–6400. IEEE (2022)
  * [68] Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. Advances in neural information processing systems 28 (2015)
  * [69] Starke, S., Zhang, H., Komura, T., Saito, J.: Neural state machine for character-scene interactions. ACM Trans. Graph. 38(6), 209–1 (2019)
  * [70] Taheri, O., Choutas, V., Black, M.J., Tzionas, D.: Goal: Generating 4d whole-body motion for hand-object grasping. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13263–13273 (2022)
  * [71] Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: Grab: A dataset of whole-body human grasping of objects. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16. pp. 581–600. Springer (2020)
  * [72] Tanke, J., Kwon, O.H., Mueller, F.B., Doering, A., Gall, J.: Humans in kitchens: A dataset for multi-person human motion forecasting with scene context. Advances in Neural Information Processing Systems 36 (2024)
  * [73] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022)
  * [74] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
  * [75] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [76] Wan, W., Geng, H., Liu, Y., Shan, Z., Yang, Y., Yi, L., Wang, H.: Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3891–3902 (2023)
  * [77] Wan, W., Yang, L., Liu, L., Zhang, Z., Jia, R., Choi, Y.K., Pan, J., Theobalt, C., Komura, T., Wang, W.: Learn to predict how humans manipulate large-sized objects from interactive motions. IEEE Robotics and Automation Letters 7(2), 4702–4709 (2022)
  * [78] Wan, W., Yang, L., Liu, L., Zhang, Z., Jia, R., Choi, Y.K., Pan, J., Theobalt, C., Komura, T., Wang, W.: Learn to predict how humans manipulate large-sized objects from interactive motions. IEEE Robotics and Automation Letters 7(2), 4702–4709 (2022)
  * [79] Wang, Z., Chen, Y., Liu, T., Zhu, Y., Liang, W., Huang, S.: Humanise: Language-conditioned human motion generation in 3d scenes. Advances in Neural Information Processing Systems 35, 14959–14971 (2022)
  * [80] Wang, Z., Shin, D., Fowlkes, C.C.: Predicting camera viewpoint improves cross-dataset generalization for 3d human pose estimation. In: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. pp. 523–540. Springer (2020)
  * [81] Wiederhold, N., Megyeri, A., Paris, D., Banerjee, S., Banerjee, N.: Hoh: Markerless multimodal human-object-human handover dataset with large object count. Advances in Neural Information Processing Systems 36 (2024)
  * [82] Wu, Q., Shi, Y., Huang, X., Yu, J., Xu, L., Wang, J.: Thor: Text to human-object interaction diffusion via relation intervention. arXiv preprint arXiv:2403.11208 (2024)
  * [83] Wu, R., Zhu, T., Peng, W., Hang, J., Sun, Y.: Functional grasp transfer across a category of objects from only one labeled instance. IEEE Robotics and Automation Letters 8(5), 2748–2755 (2023)
  * [84] Wu, Y., Wang, J., Zhang, Y., Zhang, S., Hilliges, O., Yu, F., Tang, S.: Saga: Stochastic whole-body grasping with contact. In: European Conference on Computer Vision. pp. 257–274. Springer (2022)
  * [85] Xie, X., Bhatnagar, B.L., Pons-Moll, G.: Visibility aware human-object interaction tracking from single rgb camera. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4757–4768 (2023)
  * [86] Xie, Z., Tseng, J., Starke, S., van de Panne, M., Liu, C.K.: Hierarchical planning and control for box loco-manipulation. Proceedings of the ACM on Computer Graphics and Interactive Techniques 6(3), 1–18 (2023)
  * [87] Xu, L., Song, Z., Wang, D., Su, J., Fang, Z., Ding, C., Gan, W., Yan, Y., Jin, X., Yang, X., et al.: Actformer: A gan-based transformer towards general action-conditioned 3d human motion generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2228–2238 (2023)
  * [88] Xu, S., Li, Z., Wang, Y.X., Gui, L.Y.: Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14928–14940 (2023)
  * [89] Xu, S., Wang, Z., Wang, Y.X., Gui, L.Y.: Interdreamer: Zero-shot text to 3d dynamic human-object interaction. arXiv preprint arXiv:2403.19652 (2024)
  * [90] Xu, Y., Wan, W., Zhang, J., Liu, H., Shan, Z., Shen, H., Wang, R., Geng, H., Weng, Y., Chen, J., et al.: Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4737–4746 (2023)
  * [91] Yan, M., Wang, X., Dai, Y., Shen, S., Wen, C., Xu, L., Ma, Y., Wang, C.: Cimi4d: A large multimodal climbing motion dataset under human-scene interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12977–12988 (2023)
  * [92] Yan, M., Zhang, Y., Cai, S., Fan, S., Lin, X., Dai, Y., Shen, S., Wen, C., Xu, L., Ma, Y., et al.: Reli11d: A comprehensive multimodal human motion dataset and method. arXiv preprint arXiv:2403.19501 (2024)
  * [93] Yan, S., Li, Z., Xiong, Y., Yan, H., Lin, D.: Convolutional sequence generation for skeleton-based action synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4394–4402 (2019)
  * [94] Yang, L., Li, K., Zhan, X., Wu, F., Xu, A., Liu, L., Lu, C.: Oakink: A large-scale knowledge repository for understanding hand-object interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20953–20962 (2022)
  * [95] Yang, L., Zhan, X., Li, K., Xu, W., Li, J., Lu, C.: Cpf: Learning a contact potential field to model the hand-object interaction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11097–11106 (2021)
  * [96] Ye, R., Xu, W., Xue, Z., Tang, T., Wang, Y., Lu, C.: H2o: A benchmark for visual human-human object handover analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15762–15771 (2021)
  * [97] Yi, H., Thies, J., Black, M.J., Peng, X.B., Rempe, D.: Generating human interaction motions in scenes with text control. arXiv preprint arXiv:2404.10685 (2024)
  * [98] Zhan, X., Yang, L., Zhao, Y., Mao, K., Xu, H., Lin, Z., Li, K., Lu, C.: Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 445–456 (2024)
  * [99] Zhang, H., Christen, S., Fan, Z., Zheng, L., Hwangbo, J., Song, J., Hilliges, O.: Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. In: 2024 International Conference on 3D Vision (3DV). pp. 235–246. IEEE (2024)
  * [100] Zhang, J., Luo, H., Yang, H., Xu, X., Wu, Q., Shi, Y., Yu, J., Xu, L., Wang, J.: Neuraldome: A neural modeling pipeline on multi-view human-object interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8834–8845 (2023)
  * [101] Zhang, J., Zhang, J., Song, Z., Shi, Z., Zhao, C., Shi, Y., Yu, J., Xu, L., Wang, J.: Hoi-m3: Capture multiple humans and objects interaction within contextual environment. arXiv preprint arXiv:2404.00299 (2024)
  * [102] Zhang, S., Ma, Q., Zhang, Y., Qian, Z., Kwon, T., Pollefeys, M., Bogo, F., Tang, S.: Egobody: Human body shape and motion of interacting people from head-mounted devices. In: European Conference on Computer Vision. pp. 180–200. Springer (2022)
  * [103] Zhang, W., Dabral, R., Leimkühler, T., Golyanik, V., Habermann, M., Theobalt, C.: Roam: Robust and object-aware motion generation using neural pose descriptors. arXiv preprint arXiv:2308.12969 1 (2023)
  * [104] Zhang, X., Bhatnagar, B.L., Starke, S., Guzov, V., Pons-Moll, G.: Couch: Towards controllable human-chair interactions. In: European Conference on Computer Vision. pp. 518–535. Springer (2022)
  * [105] Zhang, X., Bhatnagar, B.L., Starke, S., Petrov, I., Guzov, V., Dhamo, H., Pérez-Pellitero, E., Pons-Moll, G.: Force: Dataset and method for intuitive physics guided human-object interaction. arXiv preprint arXiv:2403.11237 (2024)
  * [106] Zhao, K., Wang, S., Zhang, Y., Beeler, T., Tang, S.: Compositional human-scene interaction synthesis with semantic control. In: European Conference on Computer Vision. pp. 311–327. Springer (2022)
  * [107] Zheng, J., Zheng, Q., Fang, L., Liu, Y., Yi, L.: Cams: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 585–594 (2023)

Appendix

The project page of CORE4D is CORE4D Project Page.

Contents:

  * •

A. Cross-dataset Evaluation

  * •

B. Details on Real-world Data Acquisition

  * •

C. Details on CORE4D-Synthetic Data Generation

  * •

D. Dataset Statistics and Visualization

  * •

E. Details on Data Split

  * •

F. Evaluation Metrics for Benchmarks

  * •

G. Qualitative Results on Benchmarks

  * •

H. Details on the Application of CORE4D-Synthetic

  * •

I. CORE4D-Real Data Capturing Instructions and Costs

  * •

J. Experiment Configurations and Codes

  * •

K. URLs of Dataset, Repository, Metadata, DOI, and License

  * •

L. Dataset Documentation and Intended Uses

  * •

M. Author Statement

##  Appendix A Cross-dataset Evaluation

To examine the data quality of CORE4D-Real, we follow existing dataset
efforts[91, 7, 47] and conduct the vision-based cross-dataset evaluation. We
select an individual human-object-interaction dataset BEHAVE[3] that includes
color images and select 2D human keypoint estimation as the evaluation task.

Data Preparation. For a color image from CORE4D-Real and BEHAVE[3], we first
detect the bounding box for each person via ground truth human pose and obtain
the image patch for the person. We then resize the image patch to get a
maximal length of 256 pixels and fill it up into a 256x256 image with the
black color as the background. Finally, for each 256x256 image, we
automatically acquire the ground truth 2D-pixel coordinates of 22 SMPL-X[61]
human body joints from 3D human poses. For data split, we follow the original
train-test split for BEHAVE[3] and merge the two test sets (S1, S2) for
CORE4D-Real.

Task Formulation. Given a 256x256 color image including a person, the task is
to estimate the 2D-pixel coordinate for each of the 22 SMPL-X[61] human body
joints.

Evaluation Metrics. $P_{e}$ denotes the mean-square error of 2D coordinate
estimates. $Acc$ denotes the percentage of the coordinate estimates with the
Euclidean distance to the ground truth smaller than $15$ pixels.

Method, Results, and Analysis. We draw inspiration from HybrIK-X[41] and adopt
their vision backbone as the solution. Table 6 shows the method performances
on the two datasets under different training settings. Due to the significant
domain gaps in visual patterns and human behaviors, transferring models
trained on one dataset to the other would consistently encounter error
increases. Despite the domain gaps, integrally training on both datasets
achieves large performance gains on both CORE4D-Real and BEHAVE[3], indicating
the accuracy of CORE4D-Real and the value of the dataset serving for visual
perception studies.

Table 6: Cross-dataset evaluation with BEHAVE[3] on 2D human keypoint estimation. Results are in $P_{e}$ (pixel2, lower is better) and $Acc$ ($\%$, higher is better), respectively. Test Train | CORE4D-Real | BEHAVE[3] | CORE4D-Real + BEHAVE[3]
---|---|---|---
CORE4D-Real | 152.4 / 91.2 | 904.9 / 35.6 |  121.7 / 92.4
BEHAVE[3] | 887.9 / 37.8 | 146.3 / 88.9 |  128.2 / 89.8

##  Appendix B Details on Real-world Data Aquisition

In this section, we describe our system calibration (Section B.1) and time
synchronization (Section B.2) in detail. Moreover, we provide detailed
information on loss functions of the human mesh acquisition (Section B.3).

###  B.1 System Calibration

Calibrating the Inertial-optical Mocap System. Three reflective markers are
fixed at known positions on a calibration rod, by which the 12 high-speed
motion capture cameras calculate their relative extrinsic parameters,
providing information about their spatial relationships. Additionally, three
markers fixed at the world coordinate origin are employed to calibrate the
motion capture system coordinate with the defined world coordinate.

Calibrating Camera Intrinsic. The intrinsic parameters of allocentric and
egocentric cameras are calibrated using a chessboard pattern.

Calibrating Extrinsic of the Allocentric Cameras. We place ten markers in the
camera view to locate each allocentric camera. By annotating the markers’ 3D
positions in the world coordinate system and their 2D-pixel coordinates on
allocentric images, the camera’s extrinsic parameters are estimated by solving
a Perspective-n-Point (PnP) problem via OpenCV.

Calibrating Extrinsic of the Egocentric Camera. We obtain the camera’s pose
information by fixing the camera to the head tracker of the motion capture
suit. Similarly, ten markers are used to calibrate the relative extrinsic
parameters of the first-person perspective cameras, allowing for determining
their positions and orientations relative to the motion capture system.
Additionally, to mitigate errors introduced by the integration of optical and
inertial tracking systems, a purely optical tracking rigid is mounted on the
motion camera.

###  B.2 Time Synchronization

To implement our synchronization method, we first set up a Network Time
Protocol (NTP) server on the motion capture host. This server serves as the
time synchronization reference for the Windows computer connected to the
Kinect Azure DK. We minimize time discrepancies by connecting the Windows
computer to the NTP server in high-precision mode and thus achieving precise
synchronization.

Additionally, we employ a Linear Timecode (LTC) generator to encode a time
signal onto the action camera’s audio track. This time signal serves as a
synchronization reference for aligning the first-person perspective RGB
information with the motion capture data.

###  B.3 Loss Function Designs for Human Mesh Acquisition

To transfer the BVH[52] human skeleton to the widely-used SMPL-X[61] model. We
optimize body shape parameters $\beta\in\mathbb{R}^{10}$ to fit the
constraints on manually measured human skeleton lengths and then optimize the
full-body pose $\theta\in\mathbb{R}^{159}$ with the following loss function:

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{reg}}+\mathcal{L}_{j\text{3D}}+\mathcal{L}_{j\text{Ori}}+\mathcal{L}_{\text{smooth}}+\mathcal{L}_{h\text{3D}}+\mathcal{L}_{h\text{Ori}}+\mathcal{L}_{\text{contact}}.$
(5)

Regularization Loss $\mathcal{L}_{\text{reg}}$. The regularization loss term
is defined as

$\displaystyle\mathcal{L}_{\text{reg}}=\sum\left|\left|\theta_{\text{body}}\right|\right|^{2}\cdot\lambda_{\text{body}}+\left(\sum\left|\left|\theta_{l\\_\text{hand}}\right|\right|^{2}+\sum\left|\left|\theta_{r\\_\text{hand}}\right|\right|^{2}\right)\cdot\lambda_{\text{hand}},$
(6)

where $\theta_{\text{body}}\in\mathbb{R}^{21\times 3}$ represents the body
pose parameters defined by 21 joints of the skeleton,
$\theta_{l\\_hand}\in\mathbb{R}^{12}$ and
$\theta_{r\\_\text{hand}}\in\mathbb{R}^{12}$ represents the hand pose
parameters. For each hand, the original SMPL-X skeleton has 15 joints with
parameters $\theta_{\text{hand}}\in\mathbb{R}^{15\times 3}$. However,
principal component analysis (PCA) is applied to the hand pose parameters. The
$\theta_{\text{hand}}$ parameters are transformed into a lower-dimensional
space, specifically $\mathbb{R}^{12}$. $\lambda_{\text{body}}=10^{-3}$ and
$\lambda_{\text{hand}}=10^{-4}$ are different weights that are used to control
the regularization strength for the body and hand pose parameters,
respectively. This loss ensures the simplicity of the results and prevents
unnatural, significant twisting of the joints.

3D Position Loss $\mathcal{L}_{j\text{3D}}$ and $\mathcal{L}_{h\text{3D}}$.
The 3D position loss term is defined as

$\displaystyle\mathcal{L}_{\text{3D}}=\sum\left|\left|\textbf{T}_{\text{smplx}}-\textbf{T}_{\text{bvh}}\right|\right|^{2}\cdot\lambda_{\text{3D}},$
(7)

where $\textbf{T}_{\text{smplx}}\in\mathbb{R}^{3}$ represents the 3D global
coordinates of the joints in the SMPL-X model and
$\textbf{T}_{\text{bvh}}\in\mathbb{R}^{3}$ represents the corresponding 3D
global coordinates of the joints in the BVH representation.
$\mathcal{L}_{j\text{3D}}$ represents the 3D position loss sum for the 21 body
joints, while $\mathcal{L}_{h\text{3D}}$ represents the 3D position loss sum
for the 30 hand joints (15 joints per hand). These two terms have different
weights, set as $\lambda_{j\text{3D}}=1.0$ and $\lambda_{h\text{3D}}=2.0$,
respectively.

Orientation Loss $\mathcal{L}_{j\text{Ori}}$ and $\mathcal{L}_{h\text{Ori}}$.
The orientation loss term is defined as

$\displaystyle\mathcal{L}_{\text{Ori}}=\sum\left|\left|\textbf{R}_{\text{smplx}}-\textbf{R}_{\text{bvh}}\right|\right|^{2}\cdot\lambda_{\text{Ori}},$
(8)

which is similar to $\mathcal{L}_{\text{3D}}$, except that
$\mathcal{R}_{\text{smplx}}\in\mathbb{R}^{3\times 3}$ and
$\mathcal{R}_{\text{bvh}}\in\mathbb{R}^{3\times 3}$ represent the rotation
matrices for the adjacent joints in the SMPL-X and corresponding BVH
representations, respectively. Specifically, body joints named head, spine,
spine2, leftUpLeg, rightUpLeg, rightShoulder, leftShoulder, rightArm, leftArm,
and neck are subjected to orientation loss, ensuring that their rotations
relative to adjacent nodes are close to the BVH ground truth.
$\lambda_{\text{Ori}}$ is set to $0.2$.

Temporal Smoothness Loss $\mathcal{L}_{\text{smooth}}$. The temporal
smoothness loss term is defined as

$\displaystyle\mathcal{L}_{\text{smooth}}=\sum_{i=1}^{N}\left(\left|\left|\theta_{i}-\theta_{i-1}\right|\right|^{2}\right)\cdot\lambda_{\text{smooth}}$
(9)

where $\theta_{i}\in\mathbb{R}^{(21+30)\times 3}$ represents the body and hand
pose of the $i$-th frame. $\lambda_{\text{smooth}}$ is set to $20.0$.

Contact Loss $\mathcal{L}_{\text{contact}}$. The contact loss term is defined
as

$\displaystyle\mathcal{L}_{\text{contact}}=\sum\left(\left|\left|\textbf{T}_{\text{finger}}-\textbf{T}_{\text{obj}}\right|\right|^{2}\cdot\mathcal{J}(\textbf{T}_{\text{finger}},\textbf{T}_{\text{obj}})\right)\cdot\lambda_{\text{contact}}$
(10)

where $\mathcal{T}_{\text{finger}}\in\mathbb{R}^{10\times 3}$ is the global
coordinates of ten fingers, and
$\mathcal{T}_{\text{obj}}\in\mathbb{R}^{10\times 3}$ is the corresponding
global coordinates of the point closest to finger.
$\mathcal{J}(\textbf{T}_{\text{finger}},\textbf{T}_{\text{obj}})$ is 1 when
the distance between $\textbf{T}_{\text{finger}}$ and
$\textbf{T}_{\text{obj}}$ is less than a threshold, otherwise it is 0. And
$\lambda_{\text{contact}}$ is $2.0$.

##  Appendix C Details on CORE4D-Synthetic Data Generation

In this section, we provide details on our synthetic data generation
(collaboration retargeting) method. Firstly, we clarify term definitions in
Section C.1. We then explicitly introduce the whole method pipeline in detail
in Section C.2. Finally, we provide implementation details in Sections C.3 and
C.4.

###  C.1 Term Definitions

We provide definitions for the terms in our collaboration retargeting pipeline
as follows.

Contact Candidate: Contact candidate is a quadruple list containing all
possible contact region index (person1_leftHand, person1_rightHand,
person2_leftHand, person2_rightHand) on source’s vertices. For each source, we
record the contact regions of the four hands in each frame of each data
sequence. At the beginning of the synthetic data generation pipeline, we
sample contact candidates from these records.

Contact Constraint: Having contact candidate on source, we apply DeepSDF-
based[60] contact retargeting to transfer the contact regions to target. These
contact regions on target are the contact constraints fed into the contact-
guided interaction retargeting module.

Source Interaction: During each collaboration retargeting process, we sample a
human-object-human collaborative motion sequence from CORE4D-Real as the
source interaction to guide temporal collaboration pattern.

Interaction Candidate: Sampling $N$ contact candidates, we apply contact-
guided interaction retargeting $N$ times and have $N$ human-object-human
motion outputs, dubbed interaction candidates. These motions would be fed into
the human-centric contact selection module to assess their naturalness.

###  C.2 Method Pipeline

The algorithm takes a source-target pair as input. First, we sample contact
candidates from the whole CORE4D-Real contact knowledge on source. For each
contact candidate, we apply object-centric contact retargeting to propagate
contact candidates to contact constraints on target. Sampling motion from
CORE4D-Real provides a high-level temporal collaboration pattern, and together
with augmented low-level spatial relations, we obtain interaction candidates
from the contact-guided interaction retargeting. Then, the human-centric
contact selection module selects the optimal candidates, prompting a contact
constraint update. After multiple iterations, the process yields augmented
interactions. This iterative mechanism ensures a refined augmentation of
interactions, enhancing the dataset’s applicability across various scenarios.

###  C.3 Contact-guided Interaction Retargeting

The contact-guided interaction retargeting is a two-step optimization. We
start by optimizing the motion of target. Then with target contact
constraints, we optimize the poses of the two persons.

Object motion retargeting. We deliberately design temporal and spatial losses
to acquire consistent and smooth target motion. In the concern of efficiency,
we jointly optimize all frames in a single data sequence with $N$ frames. To
guarantee the fidelity of object motion, we design the fidelity loss $L_{f}$
to restrict the rotation $R_{o,i}$ and the translation $T_{o,i}$ with the
ground-truth rotation $R^{\prime}_{o,i}$ and translation $T^{\prime}_{o,i}$ in
$N$ frames:

$\displaystyle\mathcal{L}_{f}=\lambda_{f}\sum\limits_{i}(||R^{\prime}_{o,i}-R_{o,i}||_{1}+||T^{\prime}_{o,i}-T_{o,i}||_{1}).$
(11)

We then address restriction on target’s spatial position to avoid penetration
with the ground. The spatial loss is defined as:

$\displaystyle\mathcal{L}_{\text{spat}}=\lambda_{\text{spat}}\sum\limits_{i}\text{max}(-\text{min}(\text{height}_{i}),0),$
(12)

where $\text{min}(\text{height}_{i})$ represents the lowest spatial position
of the objects per frame. A smoothness loss is designed to constrain the
object pose difference between consecutive frames:

$\displaystyle\mathcal{L}_{\text{smooth}}=\lambda_{\text{smooth}}\sum\limits_{i}a_{R_{o,i}}^{2}+a_{T_{o,i}}^{2},$
(13)

where $a$ is the acceleration of rotation and translation during $N$ frames
defined as:

$\displaystyle a_{R_{o,i}}=2R_{o,i}-R_{o,i-1}-R_{o,i+1},$ (14) $\displaystyle
a_{T_{o,i}}=2T_{o,i}-T_{o,i-1}-T_{o,i+1},$ (15)

The total object motion retargeting problem is:

$\displaystyle
R_{o},T_{o}\longleftarrow{}\mathop{\operatorname{argmin}}\limits_{R_{o},T_{o}}(\mathcal{L}_{f}+\mathcal{L}_{\text{spat}}+\mathcal{L}_{\text{smooth}}).$
(16)

Human motion retargeting. We next optimize each person’s motion based on the
motion of target and the contact constraint. To acquire visually plausible
motion, we design the fidelity loss $\mathcal{L}_{j}$ and the smoothness loss
$\mathcal{L}_{\text{smooth}}$. Besides, we utilize the contact correctness
loss $\mathcal{L}_{c}$ to acquire contact consistency in target interaction
motion, and leverage spatial loss $L_{\text{spat}}$ similar to Equation 12 to
avoid human-ground inter-penetration.

To enhance motion fidelity, we define two loss functions
$\mathcal{L}_{\text{sr}}$ and $\mathcal{L}_{\text{wr}}$ and let
$L_{j}=\mathcal{L}_{\text{sr}}+\mathcal{L}_{\text{wr}}$. For joints from the
human arms, despite following the correct temporal collaboration pattern,
their global positions would vary concerning diverse object geometries.
Therefore, we utilize oriented vectors pointing to their parent body joints to
obtain a relative joint fidelity:

$\displaystyle\mathcal{L}_{\text{sr}}=\lambda_{\text{sr}}\sum\limits_{i}\sum\limits_{j\in\text{arm}}\|(P_{j,i}-P_{\text{parent}(j),i})-(P^{\prime}_{j,i}-P^{\prime}_{\text{parent}(j),i})\|_{2}^{2},$
(17)

where $P_{j,i}$ denotes the 3D global position of joint $j$ in frame $i$, and
$P^{\prime}$ denotes ground-truth values. $\mathcal{L}_{\text{wr}}$ denotes
constraints on the global positions of other joints:

$\displaystyle\mathcal{L}_{\text{wr}}=\lambda_{\text{wr}}\sum\limits_{i}\sum\limits_{j\notin\text{arm}}\|P_{j,i}-P^{\prime}_{j,i}\|_{2}^{2}.$
(18)

The design of the smoothness loss is similar to Equation 13, penalizing huge
acceleration of human SMPL-X parameters to avoid great motion differences
between frames:

$\displaystyle\mathcal{L}_{\text{smooth}}=\lambda_{\text{smooth}}\sum\limits_{i}\sum\limits_{j\in\\{1,2\\}}(a_{\theta_{j,i}})^{2}+(a_{T_{j,i}})^{2}+(a_{O_{j,i}})^{2}.$
(19)

To leverage contact constraints, we attract human hands to the corresponding
contact region on target. We select the positions of 20 fingertips of the two
persons in the $i$-th frame as
$\mathcal{H}_{i}=\\{\bar{P}_{\text{tip,i}}\\}_{\text{tip}\in[1,20]}$, where
$\bar{P}$ are tip positions in the object’s coordinate system. The contact
vertices on the target from object-centric contact retargeting are defined as
$\mathcal{C}=\\{\bar{P}^{\prime}_{\text{tip}}\\}_{\text{tip}\in[1,20]}$. We
minimize the Chamfer Distance ($CD$) between $\mathcal{H}_{i}$ and
$\mathcal{C}$ to obtain contact consistency:

$\displaystyle\mathcal{L}_{c}=\lambda_{c}\sum\limits_{i}CD(\mathcal{H}_{i},\mathcal{C}).$
(20)

The total human motion retargeting problem is:

$\displaystyle\theta_{1,2},T_{1,2},O_{1,2}\longleftarrow\mathop{\operatorname{argmin}}\limits_{\theta_{1,2},T_{1,2},O_{1,2}}(\mathcal{L}_{j}+\mathcal{L}_{c}+\mathcal{L}_{\text{spat}}+\mathcal{L}_{\text{smooth}}),$
(21)

In practice, we run 1,000 and 1,500 iterations respectively for object motion
retargeting and human motion retargeting. The whole pipeline is implemented in
PyTorch with Adam solver. The learning rate is 0.01. In object motion
retargeting, $\lambda_{f}$ for rotation is 500, for translation is 0.005,
$\lambda_{\text{spat}}=0.01$, $\lambda_{\text{smooth}}=1$. In human motion
retargeting, $\lambda_{\text{sr}}=0.1$, $\lambda_{\text{wr}}=0.003$,
$\lambda_{c}=1,000$, $\lambda_{\text{spat}}=0.01$, and
$\lambda_{\text{smooth}}=1$.

###  C.4 Human-centric contact selection

The pairwise training dataset utilized for the human pose discriminator
training comprises 636,424 pairs of data. Each pair encompasses a positive
human pose $S_{\text{pos}}\in\mathbb{R}^{21\times 3}$ and a negative human
pose $S_{\text{neg}}\in\mathbb{R}^{21\times 3}$. The positive human pose is
sampled from the CORE4D-Real. Conversely, the negative human pose is derived
from the corresponding positive sample by introducing noise to its object
pose, subsequently employing the original contact information to perform
contact-guided interaction retargeting. The discriminator is trained by:

$\displaystyle\mathcal{L}_{\text{ranking}}=-\log(\sigma(R_{\text{pos}}-R_{\text{neg}}-m(S_{\text{pos}},S_{\text{neg}}))),$
(22)

iterating 1,000 epochs by the Adam solver with a learning rate 2e-4.

Specifically, the noise $\Delta(\alpha,\beta,\gamma,x,y,z)$ incorporates both
rotational and translational components. The rotational noise
$\Delta(\alpha,\beta,\gamma)$ ranges from 20 to 60 degrees, while the
translational noise $\Delta(x,y,z)$ falls within the range of 0.2 to 0.5
meters. The margin is computed by:

$\displaystyle
m(S_{\text{pos}},S_{\text{neg}})=(|\alpha|+|\beta|+|\gamma|)/10+(|x|+|y|+|z|)*10.$
(23)

During the contact constraint update process, a penetration filtering step is
performed. For each frame, the penetration volume between the human and object
is calculated. If the penetration volume exceeds $10^{-4}$ cubic meters, it is
considered a penetration case. If more than 2.5% of frames within an
interaction candidate exhibit penetration, the entire candidate is discarded.
Among the remaining candidates, the one with the highest score from the human
pose discriminator is selected to proceed with the contact constraint update.

Figure 5: Visualization of all participants in CORE4D-Real.

##  Appendix D Dataset Statistics and Visualization

Table 7: Statistics on object in CORE4D. Set |  $\\#$Object |  $\\#$Sequence
---|---|---
Chair | Desk | Box | Board | Barrel | Stick | Chair | Desk | Box | Board | Barrel | Stick
Real | 5 | 6 | 9 | 5 | 9 | 4 | 157 | 213 | 200 | 128 | 206 | 58
Synthetic | 418 | 408 | 376 | 589 | 602 | 596 | 1767 | 1344 | 1326 | 2123 | 1495 | 1961
Figure 6: T-SNE visualization of human poses for different collaboration
modes.

###  D.1 Collaboration Modes

CORE4D encompasses five human-human cooperation modes in collaborative object
rearrangement. “Move1” refers to the scenario where two participants
simultaneously rearrange objects and both are aware of the target. On the
other hand, “move2” represents the scenario where objects are rearranged
simultaneously, but only Person 1 knows the target. “Pass” indicates that one
participant passes the object to another for relay transportation. “Join”
means that Person 2 joins Person 1 in carrying the object during
transportation. Lastly, “leave” signifies that Person 2 leaves during the
joint transportation with Person 1.

According to the different durations of the two participants’ contact with the
object, “move1” and “move2” can be combined into collaborative carrying tasks.
“Pass” represents the task of handover and solely moving the object.
Incorporating the join task and the leave task, CORE4D totally comprises four
different tasks (see Figure 4 in the main paper) based on the interaction
between humans and objects. Fig. 10 exemplifies the motions for each task.

As depicted in Fig. 6, distinct characteristics are exhibited by different
cooperation modes in high-level movements, thereby offering an innovative
standpoint and potential for comprehending and investigating collaborative
behaviors.

###  D.2 Participants

As illustrated in Fig. 5, a total of 31 participants, encompassing variations
in height, weight, and gender, contributed to the capturing of CORE4D-Real.

###  D.3 Objects

CORE4D-Real has 38 objects while CORE4D-Synthetic has about 3k objects. The
objects encompass six categories, namely box, board, barrel, stick, chair, and
desk, each exhibiting a rich diversity in surface shape and size. The
distribution of object categories is detailed in Table 7. All the objects in
CORE4D-Real are shown in Fig. 9. Fig. 8 shows samples from CORE4D-Synthetic
and their interpolation process.

###  D.4 Camera Views

Fig. 7 shows the four allocentric and one egocentric views of our data
capturing system.

Figure 7: Visualization of CORE4D camera views. Figure 8: Visualization of
CORE4D-Synthetic objects and interpolation.

##  Appendix E Details on Data Split

Benefiting from the diverse temporal collaboration patterns from CORE4D-Real
and the large data amount of CORE4D-Synthetic, we randomly select a subset of
real object models and construct the training set as the combination of their
real (T-Real) and synthesized (T-Synthetic) collaboration motion sequences. We
formulate two test sets on CORE4D-Real supporting studies of both non-
generalization and inner-category generalization. The first test set (S1)
consists of interaction performed on the objects that appear in the training
set, while the second one (S2) is composed of interaction from novel objects.
Detailed data distribution of each object category is shown in Table 8.

Table 8: Train-test split on CORE4D. Set |  $\\#$Object |  $\\#$Sequence
---|---|---
Chair | Desk | Box | Board | Barrel | Stick | Chair | Desk | Box | Board | Barrel | Stick
T-Real | 3 | 4 | 6 | 3 | 6 | 2 | 93 | 104 | 96 | 51 | 113 | 25
T-Synthetic | 418 | 408 | 376 | 589 | 602 | 596 | 1767 | 1344 | 1326 | 2123 | 1495 | 1961
S1 | 3 | 4 | 6 | 3 | 6 | 2 | 40 | 62 | 45 | 21 | 51 | 6
S2 | 2 | 2 | 3 | 2 | 3 | 2 | 24 | 47 | 59 | 56 | 42 | 27

##  Appendix F Evaluation Metrics for Benchmarks

The code of our evaluation metrics is provided in Code Repository.

###  F.1 Human-object Motion Forecasting

Evaluation metrics include the human joints position error $J_{e}$, the object
translation error $T_{e}$, the object rotation error $R_{e}$, the human-object
contact accuracy $C_{\text{acc}}$, and the penetration rate $P_{r}$.

  * •

We define $J_{e}$ as the average Mean Per Joint Position Error (MPJPE) of the
two persons. MPJPE represents the mean per-joint position error of the
predicted human joint positions and the ground-truth values.

  * •

Translation error ($T_{e}$) and rotation error ($R_{e}$) denote the average L2
difference between the predicted object translation vectors and the ground-
truth ones, and the average geodesic difference between the estimated object
rotation matrices and the ground-truth ones, respectively.

  * •

Physical metrics: To assess contact fidelity, we detect contacts on the two
hands of the two persons for each frame with an empirically designed distance
threshold (5 centimeters). We then examine the contact accuracy ($C_{acc}$),
which indicates the average percentage of contact detection errors in the
predicted motions. Additionally, we examine the object penetration ratio
($P_{r}$) representing the mean percentage of object vertices inside the human
meshes.

###  F.2 Interaction Synthesis

Following an existing individual human-object interaction synthesis study[40],
the evaluation metrics include the root-relative human joint position error
$RR.J_{e}$, the root-relative human vertex position error $RR.V_{e}$, the
human-object contact accuracy $C_{\text{acc}}$, and the FID score (FID).

  * •

$RR.J_{e}$ denotes the average root-relative MPJPE of the two persons. The
root-relative MPJPE represents the mean per-joint position error of the
predicted human joint positions relative to the human root position and the
ground-truth values.

  * •

$RR.V_{e}$ denotes the average root-relative Mean Per Vertex Position Error
(MPVPE) of the two persons. The root-relative MPVPE represents the mean per-
vertex position error of the predicted human vertex positions relative to the
human root position and the ground-truth values.

  * •

$C_{\text{acc}}$ is the same as that in Section F.1.

  * •

The Fréchet Inception Distance (FID) quantitatively evaluates the naturalness
of synthesized human motions. We first train a feature extractor on
CORE4D-Real to encode each human-object-human motion sequence to a 256D
feature vector $\bar{f}_{i}$ and acquire the ground-truth human motion feature
distribution $\bar{D}$=$\\{\bar{f}_{i}\\}$. We then replace the motions of the
two persons as synthesized ones and obtain another distribution
$D$=$\\{f_{i}\\}$. Eventually, the FID denotes the 2-Wasserstein distance
between $\bar{D}$ and $D$. Since CORE4D-Real provides action labels, the
feature extractor is supervised-trained by fulfilling the action recognition
task. The network structure of the feature extractor is a single-layer
Transformer[75]. We provide the code of the feature extractor and pre-trained
parameters in Code Repository.

##  Appendix G Qualitative Results on Benchmarks

Figure 11 and Figure 12 exemplify generated motions for the human-object
motion forecasting task and the interaction synthesis task, respectively,
where “GT” denotes the ground truth motions, and others are method
predictions. Since the baseline methods do not focus on generating hand poses,
we replace hand poses in ground truth with flat hands to facilitate fair
comparisons. Despite diverse cooperation modes that can be generated, the
baseline methods consistently encompass unsatisfactory performances including
unnatural collaboration, inter-penetration, and unnatural contact.

##  Appendix H Details on the Application of CORE4D-Synthetic

To evaluate the application of CORE4D-Synthetic, we use the lightweight
CAHMP[14] to conduct the motion forecasting experiments. Unlike the
experiments in section Human-object Motion Forecasting mentioned in the main
paper, where 15 frames are predicted, here we predict the human-object motion
for the next 10 frames given the previous 10 frames.

###  H.1 Task Formulation

Given the object’s 3D model and human-object poses in adjacent 10 frames, the
task is to predict their subsequent poses in the following 10 frames. The
human pose $P_{h}\in\mathbb{R}^{23\times 3}$ represents the joint rotations of
the SMPL-X[61] model, while the object pose
$P_{o}=\\{R_{o}\in\mathbb{R}^{3},T_{o}\in\mathbb{R}^{3}\\}$ denotes 3D
orientation and 3D translation of the rigid object model.

###  H.2 Evaluation Metrics

Following existing motion forecasting works[12, 78, 88], we evaluate human
joints position error $J_{e}$, object translation error $T_{e}$, object
rotation error $R_{e}$. Details of the three metrics can be found in Section
F.1.

###  H.3 Results

Comparing the 1K real dataset with the 0.1K real dataset supplemented with
synthetic data generated through retargeting, we observed that the quality of
the synthetic data is comparable to the real data. Additionally, due to the
increased diversity of objects and enriched spatial relations between humans
and objects in the synthetic data, it exhibits better generalization
performance in object motion forecasting.

Comparing the evaluation results of the 1K real dataset with the results
obtained by augmenting it with additional 4K synthetic data, we observed a
significant performance gain from the synthetic data. This demonstrates that
the inclusion of synthetic data enhances the value of our dataset and better
supports downstream tasks.

Figure 9: Visualization of CORE4D-Real objects. Figure 10: Visualization of
CORE4D object rearrangement tasks. Figure 11: Qualitative results of human-
object motion forecasting. Grey meshes are from the task inputs. Figure 12:
Qualitative results of interaction synthesis.

##  Appendix I CORE4D-Real Data Capturing Instructions and Costs

###  I.1 Instructions.

Target. We divide a $4m\times 5m$ field into 20 squares and number them, and
place colored labels as markers along the perimeter of the field. The
following language instructs participants: "Please collaboratively move the
object to the target square. You can choose any path and orientation of the
object as you like. It is not necessary to be overly precise with the final
position - a rough placement is fine. Do not make unnatural motions just to
achieve an exact position. Do not use verbal communication with each other.".
As for the settings when only one participant knows the target, the target
square number is written on a piece of paper and shown to the participant who
knows the target. And additional instructions are given as: "If you know the
target, do not use language or direct body language to inform the other party
(such as pointing out the location). If you do not know the target, please
assist the other participant in completing the transportation.".

Collaboration Mode. The instructions are given as follows to indicate
different Collaboration Modes for the participants. For Collaborate mode:
"Based on the target, please cooperatively transport the object, or upright
any overturned tables, chairs, etc. Both participants should be in contact
with the object throughout the process.". For Handover mode: "Please decide
the handover point yourselves, then have one person hand the object to the
other, completing the object transfer in relay.". For Leave and Join modes:
"One person will transport the object throughout, while the other leaves or
joins to help at a time point not disclosed to the collaborator.".

Obstacle. The instructions are given as follows to guide the participants in
tackling obstacles: "There are a varying number of obstacles on the field. If
they get in your way, please decide on your own how to solve it using some
common everyday operations. If the obstacles occupy the destination, please
place the object near the destination.".

###  I.2 Costs.

Scanning the object took 30 person-hours, modeling the object into the mocap
system took 27.5 person-hours, data capture took 78 person-hours, data
annotation took 7 person-hours, and the user study took 60 person-hours. The
wage is 100 RMB per person-hour.

##  Appendix J Experiment Configurations and Codes

We evaluate existing methods for the two benchmarks on Ubuntu 20.04 with one
NVIDIA GeForce RTX 3090 GPU. The code of benchmarks and relevant methods are
provided in Code Repository. During the quantitative evaluation, we select
three random seeds (0, 42, 233) for each method, train the network
respectively, and then report the mean performances and standard deviations as
the evaluation results. More experimental details are provided in Code
Repository.

##  Appendix K URLs of Dataset, Repository, Metadata, DOI, and License

  * •

Dataset project page: https://core4d.github.io/.

  * •

Data link: OneDrive Repository Link.

  * •

Dataset usage instruction: https://github.com/leolyliu/CORE4D-Instructions.

  * •

Code link: https://github.com/leolyliu/CORE4D-Instructions.

  * •

Croissant metadata: Croissant Metadata Link.

  * •

Schema.org metadata: https://core4d.github.io/.

  * •

DOI: 10.5281/zenodo.11607666.

  * •

License. This work is licensed under a CC BY 4.0 license.

##  Appendix L Dataset Documentation and Intended Uses

We use the documentation framework from Gebru et.al[21].

###  L.1 Motivation

  * •

For what purpose was the dataset created? Was there a specific task in mind?
Was there a specific gap that needed to be filled? Please provide a
description.

The dataset was created to facilitate research studies in multi-person
collaboration for object rearrangement. The dataset can support various
research topics for understanding and synthesizing collaborative behaviors,
including human-object motion tracking, action recognition, human-object
motion forecasting, and collaboration synthesis.

  * •

Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?

The dataset was created by Chengwen Zhang from Beijing University of Posts and
Telecommunications, together with Yun Liu, Ruofan Xing, Bingda Tang, and Li Yi
from Tsinghua University.

  * •

Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number.

Funding was provided by the Institute for Interdisciplinary Information
Sciences at Tsinghua University.

  * •

Any other comments?

None.

###  L.2 Composition

  * •

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.

The dataset comprises two parts: CORE4D-Real and CORE4D-Synthetic. The
CORE4D-Real includes 3D object models, human-object motions, allocentric RGBD
videos, egocentric RGB videos, human-object segmentations, camera parameters,
and action labels. The CORE4D-Synthetic includes 3D object models and human-
object motions. Please refer to the Dataset Documentation for explicit
definitions of these files.

  * •

How many instances are there in total (of each type, if appropriate)?

CORE4D-Real includes 37 object models, 1.0K human-object motion sequences,
4.0K allocentric RGBD videos, 1.0K egocentric videos, 4.0K human-object
segmentations, and 1.0K action labels. CORE4D-Synthetic includes 3.0K object
models and 10K human-object motion sequences.

  * •

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

The dataset is a representative sample of all possible and infinitely many
multi-human collaborative behaviors for household object rearrangement. To
cover as diverse collaboration as possible, we collect five typical
collaboration modes in the CORE4D-Real, and enrich human-object spatial
relations greatly in the CORE4D-Synthetic. Each collaboration sequence is
complete.

  * •

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images)or features? In either case, please provide a description.

For CORE4D-Real, each collaboration instance consists of SMPLX[61] models of
two persons in each frame, the 3D model for the manipulated object, the
object’s 6D pose in each frame, four allocentric RGBD videos with camera
intrinsic and extrinsic, one egocentric RGB video with the camera intrinsic,
four human-object segmentation sequences, and one action label. For
CORE4D-Synthetic, each collaboration instance consists of SMPLX[61] models of
two persons in each frame, the 3D model for the manipulated object, and the
object’s 6D pose in each frame. Details are provided in Dataset Documentation.

  * •

Is there a label or target associated with each instance? If so, please
provide a description.

For CORE4D-Real, each collaboration instance is associated with an action
label. There is no label in CORE4D-Synthetic.

  * •

Is any information missing from individual instances? If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.

No information is missing.

  * •

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.

Relationships between individual collaboration instances include the same
persons or the same objects. We provide an explicit SMPLX[61] shape parameter
for each person, and an explicit name and 3D object model for each object.

  * •

Are there recommended data splits (e.g., training, development/validation,
testing)? If so, please provide a description of these splits, explaining the
rationale behind them.

The recommended data split is provided in Code Repository.

  * •

Are there any errors, sources of noise, or redundancies in the dataset? If so,
please provide a description.

Noises come from the hardware noises of the inertial-optical mocap system, the
3D scanner, and the cameras.

  * •

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? If it links to or
relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a dataset consumer? Please provide descriptions of all external resources
and any restrictions associated with them, as well as links or other access
points, as appropriate.

The dataset is fully self-contained.

  * •

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals’ nonpublic
communications)? If so, please provide a description.

No confidential data.

  * •

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? If so, please
describe why.

No.

  * •

Does the dataset identify any subpopulations (e.g., by age, gender)? If so,
please describe how these subpopulations are identified and provide a
description of their respective distributions within the dataset.

No.

  * •

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset? If so, please describe how.

No. We ensure participants’ anonymity by mosaicking their faces.

  * •

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals race or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)? If so,
please provide a description.

No.

  * •

Any other comments?

None.

###  L.3 Collection Process

  * •

How was the data associated with each instance acquired? Was the data directly
observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey
responses), or indirectly inferred/derived from other data (e.g., part-of-
speech tags, model-based guesses for age or language)? If the data was
reported by subjects or indirectly inferred/derived from other data, was the
data validated/verified? If so, please describe how.

The data is acquired by the method described in Section 3 of the main paper.
The data quality is evaluated in Section A and Section 4.4 of the main paper.

  * •

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatuses or sensors, manual human curation, software programs, software
APIs)? How were these mechanisms or procedures validated?

The data is collected by the method described in Section 3 of the main paper.
The hardware qualities of the inertial-optical mocap system, the 3D scanner,
and the cameras are examined by their developers.

  * •

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)?

The dataset is a representative sample of all possible and infinitely many
multi-human collaborative behaviors for household object rearrangement. To
cover as diverse collaboration as possible, we collect five typical
collaboration modes in the CORE4D-Real, and enrich human-object spatial
relations greatly in the CORE4D-Synthetic. The dataset is not a sample from a
known larger set since each motion sequence is complete.

  * •

Who was involved in the data collection process (e.g., students, crowdworkers,
contractors) and how were they compensated (e.g., how much were crowdworkers
paid)?

Students from Universities participated in the data collection process. They
were paid 100 RMB/hour. Thanks to them all.

  * •

Over what timeframe was the data collected? Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created.

The data was created and collected between July 2023 and December 2023. The
creation time and the collection time of each data instance are the same.

  * •

Were any ethical review processes conducted (e.g., by an institutional review
board)? If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.

No.

  * •

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)?

The data is the individuals’ motions.

  * •

Were the individuals in question notified about the data collection? If so,
please describe (or show with screenshots or other information) how notice was
provided, and provide a link or other access point to, or otherwise reproduce,
the exact language of the notification itself.

Yes. The language is: "As a data collector, you will be familiar with the
working principle and usage of the optical-inertial hybrid motion capture
system. You will personally wear the motion capture device to collect motion
data. All the data collected in this project will be used solely for research
purposes by the Yili Research Group at Tsinghua University’s Institute for
Interdisciplinary Information Sciences."

  * •

Did the individuals in question consent to the collection and use of their
data? If so, please describe (or show with screenshots or other information)
how consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the individuals
consented.

Yes. All the participants signed that: "I am aware of and agree that the
collected data will be used for research purposes related to the project and
may be released as a dataset."

  * •

If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses? If so,
please provide a description, as well as a link or other access point to the
mechanism (if appropriate).

Yes. All participants were notified that they have the right to request the
removal of their data at any time in the future by reimbursing their salary
and compensating for the expenses incurred in collecting their data.

  * •

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis) been conducted? If so,
please provide a description of this analysis, including the outcomes, as well
as a link or other access point to any supporting documentation.

No.

  * •

Any other comments?

None.

###  L.4 Preprocessing/cleaning/labeling

  * •

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)? If so, please provide a
description. If not, you may skip the remaining questions in this section.

No.

  * •

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? If so, please provide a link or
other access point to the “raw” data.

N/A

  * •

Is the software that was used to preprocess/clean/label the data available? If
so, please provide a link or other access point.

N/A

  * •

Any other comments?

None.

###  L.5 Uses

  * •

Has the dataset been used for any tasks already? If so, please provide a
description.

Currently, the dataset has been used to establish two benchmarks, human-object
motion forecasting and interaction synthesis, in Section 4 of the main paper.

  * •

Is there a repository that links to any or all papers or systems that use the
dataset? If so, please provide a link or other access point.

Yes. Please refer to Relevant Works.

  * •

What (other) tasks could the dataset be used for?

The dataset can support various research topics for understanding and
synthesizing collaborative behaviors, including human-object motion tracking,
action recognition, human-object motion forecasting, and collaboration
synthesis. Besides, the dataset can potentially be further used to study robot
policies for robot manipulations and human-robot collaborations.

  * •

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? For
example, is there anything that a dataset consumer might need to know to avoid
uses that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other risks or harms (e.g., legal
risks, financial harms)? If so, please provide a description. Is there
anything a dataset consumer could do to mitigate these risks or harms?

Unknown to the authors.

  * •

Are there tasks for which the dataset should not be used? If so, please
provide a description.

Unknown to the authors.

  * •

Any other comments?

None.

###  L.6 Distribution

  * •

Will the dataset be distributed to third parties outside of the entity (e.g.,
company, institution, organization) on behalf of which the dataset was
created? If so, please provide a description.

Yes. The dataset is fully released at Dataset Repository. The project page of
the dataset is CORE4D Project Page.

  * •

How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)? Does the dataset have a digital object identifier (DOI)?

The dataset is distributed on Yun Liu’s OneDrive Cloud Storage: Dataset
Repository.

DOI: https://doi.org/10.5281/zenodo.11607666.

Project page: CORE4D Project Page.

  * •

When will the dataset be distributed?

The dataset is distributed in June 2024.

  * •

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.

The dataset is licensed under a CC BY 4.0 license.

  * •

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.

No.

  * •

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation.

No.

  * •

Any other comments?

None.

###  L.7 Maintenance

  * •

Who will be supporting/hosting/maintaining the dataset?

Chengwen Zhang is supporting/maintaining the dataset.

  * •

How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?

The curators of the dataset, Chengwen Zhang, Yun Liu, Ruofan Xing, Bingda
Tang, and Li Yi, can be contacted at<EMAIL_ADDRESS>yun-
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and<EMAIL_ADDRESS>respectively.

  * •

Is there an erratum? If so, please provide a link or other access point.

No.

  * •

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)? If so, please describe how often, by whom, and
how updates will be communicated to dataset consumers (e.g., mailing list,
GitHub)?

The dataset update will be posted at Dataset Instructions and CORE4D Project
Page.

  * •

If the dataset relates to people, are there applicable limits on the retention
of the data associated with the instances (e.g., were the individuals in
question told that their data would be retained for a fixed period of time and
then deleted)? If so, please describe these limits and explain how they will
be enforced.

No.

  * •

Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence will
be communicated to dataset consumers.

After a dataset update, older versions will be kept for consistency. These
notices will be posted at Dataset Instructions and CORE4D Project Page.

  * •

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so? If so, please provide a description. Will these
contributions be validated/verified? If so, please describe how. If not, why
not? Is there a process for communicating/distributing these contributions to
dataset consumers? If so, please provide a description.

Others are allowed to do these and should contact the original authors.

  * •

Any other comments?

None.

##  Appendix M Author Statement

Author Responsibility Statement

This statement is intended to emphasize the author’s responsibility regarding
the dataset work, including ensuring compliance with all relevant laws,
regulations, and ethical guidelines. By participating in the dataset work, the
author agrees and commits to the following responsibilities:

  * •

Legality: The authors guarantee that all data and materials used in the
dataset work are obtained and used legally. The authors will ensure compliance
with all applicable laws, regulations, and policies within their country,
region, or organization.

  * •

Rights Protection: The authors will make every effort to protect the privacy
rights, intellectual property rights, and other legitimate interests of
individuals within the dataset. The authors will respect individuals’ privacy
and take appropriate measures to safeguard the security of personal
identifying information.

  * •

Transparency: The authors will provide sufficient information and explanations
to enable users of the dataset to understand the sources, purposes, and
limitations of the data. The authors will strive to ensure that the use and
publication of the dataset are transparent and traceable.

  * •

Compliance: The authors will ensure that the dataset work complies with all
applicable laws, regulations, and policies. In the event of any violation of
rights, the authors will bear full responsibility and be willing to accept the
corresponding legal consequences and liability for damages.

  * •

Shared Responsibility: The authors require others who use the dataset to also
assume appropriate responsibilities and adhere to similar obligations and
guidelines to ensure the legal and responsible use of the dataset.

  * •

License Confirm: This work is licensed under a CC BY 4.0 license.