# Template Free Reconstruction of Human-object Interaction with Procedural
Interaction Generation

Xianghui Xie1,2,3 Bharat Lal Bhatnagar4 Jan Eric Lenssen3 Gerard Pons-
Moll1,2,3

1University of Tübingen, 2Tübingen AI Center, Germany
3Max Planck Institute for Informatics, Germany
4Meta Reality Labs

###### Abstract

Reconstructing human-object interaction in 3D from a single RGB image is a
challenging task and existing data driven methods do not generalize beyond the
objects present in the carefully curated 3D interaction datasets. Capturing
large-scale real data to learn strong interaction and 3D shape priors is very
expensive due to the combinatorial nature of human-object interactions. In
this paper, we propose ProciGen (Procedural interaction Generation), a method
to procedurally generate datasets with both, plausible interaction and diverse
object variation. We generate 1M+ human-object interaction pairs in 3D and
leverage this large-scale data to train our HDM (Hierarchical Diffusion
Model), a novel method to reconstruct interacting human and unseen object
instances, without any templates. Our HDM is an image-conditioned diffusion
model that learns both realistic interaction and highly accurate human and
object shapes. Experiments show that our HDM trained with ProciGen
significantly outperforms prior methods that require template meshes, and our
dataset allows training methods with strong generalization ability to unseen
object instances. Our code and data will be released at:
https://virtualhumans.mpi-inf.mpg.de/procigen-hdm.

##  1 Introduction

Figure 1: Given a single RGB image, our method trained _only_ on our proposed
synthetic interaction dataset, can reconstruct the human, object and contacts,
without any predefined template meshes.

Modelling interactions between humans and their surroundings is important for
applications like creating realistic avatars, robotic control and gaming. In
this paper, we address the task of jointly reconstructing human and object
from a monocular RGB image, without any prior object templates. This is very
challenging due to depth-scale ambiguity, occlusions, diverse human pose and
object shape variations. Data-driven methods have shown great progress in
reconstructing humans [65, 66, 67, 42, 38] or objects [51, 100] from monocular
inputs thanks to large-scale datasets [9, 35, 59, 1, 96, 19, 12, 86]. However,
methods for joint interaction reconstruction are still constrained by the
amount of available data. Recent datasets like BEHAVE [7], InterCap [33]
capture real interactions with 10 to 20 different objects, which is far away
from the number of objects in reality: the chair category from ShapeNet [12]
alone has more than 6k different shapes. Training on these real datasets has
limited generalization ability to unseen objects (Sec. 4.3). Capturing real
interaction data with more objects is prohibitively expensive due to the
combinatorial nature: the number of humans times the number of objects leads
to a huge number of variations. This motivates us to generate synthetic data
which has been shown effective for pre-training reconstruction methods [9, 59,
28, 66, 51].

Synthesizing realistic interaction for different objects is non-trivial due to
variations of object topology, geometry details and complex interaction
patterns. To address this, we propose Procedural interaction Generation
(ProciGen), a method to generate interaction data with diverse object shapes.
We design our method based on the key idea that the way humans interact with
objects of the _same_ category is similar. And despite the geometry
variations, one can still establish semantically meaningful correspondence
between different objects. More specifically, we train an autoencoder to
obtain correspondences between different objects from the same category, which
are then used to transfer contacts from already captured human-object
interactions to new object instances. Our method is scalable and allows the
_multiplicative combination_ of datasets to generate over a million
interactions with more than 21k different object instances, which is not
possible via real data capture.

Current reconstruction methods [87, 88, 7, 56] are not only bottle-necked by
data. Template-based methods [87, 88, 7] cannot generalize to unseen objects
as they are _trained only for specific object templates_. Template free
methods like $\text{PC}^{2}$ [56] cannot separate human and object, and have
limited shape accuracy. See Tab. 1 for detailed comparison. To alleviate these
issues, we propose Hierarchical Diffusion Model (HDM), that predicts accurate
object shapes and reasons about human-object contacts without using template
meshes. Our key idea is to decompose the combinatorial interaction space into
separate human and object sub-spaces while preserving the interaction context.
We first use a diffusion model to predict human and object points jointly
along with segmentation labels and then use two separate diffusion models with
cross attention that further refine the shapes of human and objects.

We evaluate our data generation method ProciGen, and model HDM, on BEHAVE [7]
and InterCap [33]. Experiments show that HDM with ProciGen significantly
outperforms CHORE [87] (which requires object templates) and $\text{PC}^{2}$
[56]. Our ProciGen dataset also significantly boosts the performance of
$\text{PC}^{2}$ and HDM. Methods trained on our synthetic ProciGen dataset
show strong generalization ability to real images even though the objects are
unseen.

In summary, our key contributions are:

  * •

We introduce the first method for procedural interaction generation that can
synthesize large amounts of interaction with diverse object shapes. With this,
we generate a million interaction images with 21k different objects paired
with clean 3D ground truth.

  * •

We propose a hierarchical diffusion model that can faithfully reconstruct
human and object shapes from monocular RGB images without relying on template
shapes.

  * •

Our dataset, code for data generation and reconstruction will be publicly
released to foster future works.

Method | No-template | Shape acc. | General. | Semantic
---|---|---|---|---
CHORE | X | $\checkmark^{*}$ | X | ✓
$\text{PC}^{2}$ | ✓ | X | X | X
$\text{PC}^{2}$ \+ Our ProciGen | ✓ | X | ✓ | X
Ours | ✓ | ✓ | ✓ | ✓
Table 1: Comparison of different reconstruction methods. CHORE [87]
reconstructs high shape fidelity with known template meshes but does not
generalize to new object instances. $\text{PC}^{2}$ [56] is template-free but
its shape predictions lack fidelity and generalization ability is constrained
by existing datasets. Training $\text{PC}^{2}$ with our ProciGen dataset
allows better generalization but it cannot reason contacts. Our proposed data
generation together with our hierarchical diffusion model can predict accurate
shapes, generalize to unseen objects and reason about interaction semantics.

##  2 Related Work

Interaction Capture. Capturing 3D interactions from monocular inputs is an
emerging research field in recent years, with works that reconstruct hand-
object interaction from RGB [39, 17, 28, 22, 92] or RGBD [10, 11, 26] input,
or predict contacts from RGB images [78, 14, 32] and works that model human-
scene interaction from single image [27, 43, 70, 8] or video [94, 24]. A
recent line of works model full body interacting with dynamic large objects
[60, 101, 25, 91, 49, 83, 37]. BEHAVE [7] captured the first dataset for
benchmarking methods in this field, followed by more datasets [33, 97], which
allow methods to reconstruct 3D human and object from single RGB images [98,
87, 82] or videos [88]. Despite impressive results, they require predefined
mesh templates, which limits applicability to novel objects. In contrast, our
method does not require templates and generalizes well to unseen objects.

Synthetic Datasets are powerful resources to pre-train large backbone
networks. For 3D humans, synthetic scans [1, 4, 77, 59] are used extensively
to train human reconstruction methods [66, 67, 89, 75, 18, 5, 6, 90]. Recent
work BEDLAM [9] showed that training purely on synthetic datasets [9, 59]
allows strong generalization. Orthogonal to these, large scale 3D object CAD
model datasets [12, 86] are also used to pretrain backbone models[36, 95, 58,
47]. Other works [63, 23, 85, 20] consider generating diverse and photo-
realistic scenes. While being useful in modelling humans, objects or scenes
respectively, they do not consider interactions. Our proposed data generation
approach can generate millions of interactions with diverse object shapes,
allowing for training interaction reconstruction models with great
generalization ability.

Diffusion-based Reconstruction. Diffusion models [30, 72] have been shown
powerful for 3D reconstruction of human [34, 44] and objects [51, 68, 56].
These works distil pretrained 2D diffusion model [55, 104, 61, 68, 93, 34, 44]
or fine-tune diffusion model [51, 69, 50] for 3D reconstruction from images.
Recent works also propose image-conditioned point diffusion models for
reconstruction [56, 79]. Despite remarkable results, they only model the
distribution of single shapes, while our method can learn the complex
interaction space with high shape fidelity.

Figure 2: Our procedural interaction generation method. Given a seed
interaction and a new object from the same category (A), we use a network to
compute dense correspondences (B, Sec. 3.1.1), which allows us to transfer
contacts and initialize the new object (C, Sec. 3.1.2). We further optimize
the human and object poses to avoid interpenetration while satisfying the
transferred contacts (D, Sec. 3.1.3). We then add clothing and textures to
render images, leading to a large interaction dataset with diverse object
shapes (E, Sec. 3.1.4).

##  3 Method

We first introduce our method to generate large amounts of interaction data
with diverse object shapes in Sec. 3.1. This data allows us to train our novel
diffusion model with strong generalization ability, which is explained in Sec.
3.2.

###  3.1 ProciGen: Procedural Interaction Generation

Given a small seed dataset of captured human-object interactions and datasets
of various object models, we aim to generate a large-scale interaction dataset
with diverse object shapes. Via _multiplicative scaling_ , it would allow
generating enormous data which is not possible by capturing real data. This is
however non-trivial as object geometry varies strongly even within one
category. Therefore, we propose a procedural method based on the key
observation that humans interact similarly with objects of the same category.
By transferring contacts from captured interactions to new object instances,
we procedurally scale up the shape variations of real interaction datasets.
The task involves solving four different sub-problems, as outlined in Fig. 2:

  1. 1.

Establishing dense semantic correspondences between all objects within one
category (Sec. 3.1.1).

  2. 2.

Transferring contacts from real to synthetic objects, using the obtained
correspondences (Sec. 3.1.2).

  3. 3.

Jointly optimizing human and object to the newly obtained contacts under a set
of constraints (Sec. 3.1.3).

  4. 4.

Rendering novel intersection pairs with textures to make them available as
training data (Sec. 3.1.4).

####  3.1.1 Dense Semantic Correspondence

Given two meshes $\mathcal{M}$ and $\mathcal{M}^{\prime}$ of two different
objects of the same category, the problem of finding dense correspondence
amounts to finding a bijective map
$\psi:\mathcal{M}\rightarrow\mathcal{M}^{\prime}$, which maps points from one
mesh to their semantic counterparts on the other. In cases of arbitrary meshes
with changing topology, this problem is heavily ill-posed [21, 3, 74]. Thus,
we turn to an approximate solution on discrete surface samples that leverages
the regularization and output ordering of MLPs[46] and works well on a wide
range of input topologies in practice.

Let $\\{\mathcal{M}_{i}\\}_{i=1}^{M}$ be a dataset of meshes from the same
object category and $\mathbf{P}_{i}\in\mathbb{R}^{N\times 3}$ a point cloud
sampled from the surface of $\mathcal{M}_{i}$. We train an autoencoder
$f:\mathbb{R}^{N\times 3}\mapsto\mathbb{R}^{N\times 3}$ on
$\\{\mathbf{P}_{i}\\}_{i=1}^{M}$ to minimize the Chamfer distance between
predicted and input point clouds. The network $f$ consists of a PointNet [62]
encoder and a three-layer MLP decoder that takes unordered points as input and
output ordered points. We found that the MLP decoder learns to reconstruct the
objects as a mixture of low-rank point basis vectors, thus it automatically
provides dense correspondence across objects through the order in the output,
as also found in [102, 84, 74]. Effective training of this network requires
all shapes to be roughly aligned in a canonical space. When shapes are not
aligned, we use ART [102] which uses an additional network to predict an
aligning rotation.

To ensure the reconstruction quality, we overfit one network per object
category. We show some example reconstructions and correspondences for chairs
in Fig. 2B.

####  3.1.2 Contact Transfer

Given dense correspondences between a set of point clouds, we use them to
transfer contact maps from one object to the other. Let
$(\mathbf{H}\in\mathbb{R}^{M\times 3},\mathbf{P}\in\mathbb{R}^{N\times 3})$ be
a pair of human and object point clouds from an existing interaction dataset.
And let $\mathbf{T}\in SE(3)$ be the non-rigid transformation that brings the
object point cloud into canonical space where shapes are roughly aligned.
Then, we can find our _contact set_ as a set of point pairs from human and
object that lie within a distance $\sigma$ to each other:

$\mathcal{C}=\\{(i,j)\mid||\mathbf{H}_{i}-\mathbf{T}^{-1}f(\mathbf{T}\mathbf{P})_{j})||^{2}_{2}<\sigma\\}\textnormal{.}$
(1)

We first bring $\mathbf{P}$ into canonical pose, then apply $f$ to obtain a
coherent point cloud, which is brought back into interaction pose by
$\mathbf{T}^{-1}$. Since our autoencoder $f$ produces coherent point clouds,
the obtained contact set can be directly transferred to all other objects
$\mathbf{P}^{\prime}$ within the category, allowing us to pair the human with
all other objects, one example transfer is shown in Fig. 2C. Once we transfer
the contact points to the new object, we can find the corresponding contact
facets in the meshes that have the smallest distances.

####  3.1.3 Contact-based Joint Optimization

The newly obtained contact sets define how and where a human should interact
with the new object. We can also transform the object from canonical to
interaction pose with our dense correspondence. However, this naive placement
does not guarantee the plausibility of the interaction due to object geometry
changes (see Fig. 2D). Hence, we propose a joint optimization to refine the
human and object pose such that: a) contact points are close to each other, b)
contact face normals match, and c) interpenetration is avoided.

We use the SMPL-H [53, 64] body model
$H(\boldsymbol{\theta},\boldsymbol{\beta})$ to parameterize the human as a
function of pose $\boldsymbol{\theta}$ and shape $\boldsymbol{\beta}$
parameters. The object pose is given as non-rigid transformation
$\mathbf{T}\in SE(3)$, and we denote the new object point cloud to which we
have transferred contacts as $\mathbf{P}^{\prime}$. We find the refined human-
object poses jointly, by minimizing:

$L(\boldsymbol{\theta},\boldsymbol{\beta},\mathbf{T})=\lambda_{c}L_{c}+\lambda_{n}L_{n}+\lambda_{\text{colli}}+\lambda_{\text{init}}L_{\text{init}}\textnormal{,}$
(2)

where the individual loss terms are given as:

  * •

Contact:
$L_{c}=\sum_{(i,j)\in\mathcal{C}}||\mathbf{H}_{i}-\mathbf{P}_{j}^{\prime}||^{2}_{2}$,
minimizing the distance between contact points.

  * •

Normal:
$L_{n}=\sum_{(i,j)\in\mathcal{C}}||1+\mathbf{n}_{i}^{T}\mathbf{n}_{j}||^{2}_{2}$,
ensuring that normals $\mathbf{n}_{i},\mathbf{n}_{j}$ of contacting faces
point in opposite directions.

  * •

Interpenetration: $L_{\text{colli}}$ penalizing interpenetration based on the
bounding volume hierarchy [80].

  * •

Initialization: $L_{\text{init}}$ is the L2 distance between new and original
human pose, regularizing the deformation.

The pose $\boldsymbol{\theta}$ is initialized from the original human pose and
$\boldsymbol{\beta}$ is randomly sampled from a set of registered scans [4].
The object pose $\mathbf{T}$ is initialized by Procrustes alignment between
the two coherent point clouds $\mathbf{P}^{\prime}$ and $\mathbf{P}$. After
joint optimization we obtain realistic interactions, see Fig. 2D.

####  3.1.4 Dataset Rendering

Our contact transfer and joint optimization provide us the skeleton of
interaction with new objects. To render them as images, we take the optimized
SMPL-H parameters from Sec. 3.1.3 and randomly sample the clothing deformation
and texture from SMPL+D registrations in MGN [4]. For objects, we use the
original texture paired with the mesh. We render the scenes in Blender [16],
which is detailed in supplementary. See example renderings in Fig. 2E.

##### Method Scalability.

We emphasize that the proposed procedural generation is a scalable solution
that can generate large-scale datasets with only a small amount of effort for
interaction: with 2k different interactions (e.g. BEHAVE [7] chair
interaction), 6k different objects (e.g. Shapenet chairs [12]) and 100 human
scans (e.g. MGN [4]), one can have maximum _1.2 billion_ different variations
in total, which is not possible with real data capture. The data scale allows
for training powerful models that reach performance not obtainable by training
on real data only. An example of such a method is detailed in the next
section.

###  3.2 HDM: Hierarchical Diffusion Model

Figure 3: Our hierarchical diffusion model. Given an RGB image of a human
interacting with an object, we first jointly reconstruct the human and object
as one point cloud with segmentation labels (Stage 1, Sec. 3.2.2). This
prediction reasons interaction but lacks accurate shapes. We then use two
diffusion models for human or object separately with cross attention to refine
the initial noisy prediction while preserving the interaction context(Stage 2,
Sec. 3.2.3). Our hierarchical design faithfully predicts interaction and
shapes.

Modelling the joint shape space of humans interacting with objects is
difficult since the product of human and object shape variations is huge. One
solution is to use two separate networks that reconstruct human and object
respectively. However, such a method ignores the interaction cues that have
been show important for coherent reconstruction [7, 87, 88, 94]. This
motivates us to design a hierarchical solution where we first jointly estimate
both human and object(Sec. 3.2.2), and then use separate networks that focus
on refining individual shape details (Sec. 3.2.3). An overview of our method
can be found in Figure 3.

####  3.2.1 Preliminaries

Task Overview. Given an input RGB image $\mathbf{I}$ of a person interacting
with an object, we aim to jointly reconstruct 3D human and object point clouds
$\mathbf{P}^{h},\mathbf{P}^{o}$. Same as prior works [87, 98, 88], we assume
known 2D human and object segmentation masks, which we consider a weak
assumption, given recent advances in 2D segmentation [40, 41, 71].

Due to the ambiguity from monocular input, we adopt a probabilistic approach
for 3D reconstruction, which has been proven effective in learning multiple
modes given same input [56, 99, 104]. Specifically, we use a diffusion model
[30] to learn the distribution of 3D human object interactions conditioned on
a single image.

Diffusion models [30, 72] are general-purpose generative models that consist
of iterative forward and reverse processes. Formally, given a data point
$\mathbf{x}_{0}$ sampled from a data distribution $p_{\text{data}}$, the
forward process iteratively adds Gaussian noise
$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$ to the sample $\mathbf{x}_{0}$. The
distribution at step $t$ can be computed as:

$\displaystyle\mathbf{x}_{t}$
$\displaystyle=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\epsilon\sqrt{1-\bar{\alpha}_{t}}$
(3)

where $\bar{\alpha}_{t}$ controls the noise level at step $t$ and
$\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{1})$ [30]. The reverse process
starts from Gaussian noise at step $T$ and gradually denoises it back to the
original data distribution $p_{\text{data}}$ at step 0. At each reverse step,
we use a neural network $p_{\theta}$ to approximate the distribution:
$p_{\theta}\approx q(\mathbf{x}_{t-1}|\mathbf{x}_{t})$. The network is trained
with the variational lower bound to maximize the log-likelihood of all data
points, which is parametrized to minimize the L2 distance between the true
noise $\epsilon$ and network prediction[30]:

$\mathcal{L}=E_{t\sim[1,T]}E_{\epsilon_{t}\sim\mathcal{N}(0,\mathbf{I})}[||\epsilon_{t}-p_{\theta}(\mathbf{x}_{t},t)||^{2}_{2}]$
(4)

####  3.2.2 Joint Human-object Diffusion

In this first stage, we simultaneously predict both human and object hence the
output is one point cloud $\mathbf{P}\in\mathbb{R}^{N\times 3}$. We adopt
$\text{PC}^{2}$ [56] that diffuses point cloud conditioned on single images.
Formally, we use a point voxel CNN [103, 52] $p_{\theta}:\mathbb{R}^{N\times
D}\mapsto\mathbb{R}^{N\times 3}$ as the point diffusion model. Here $D$ is the
feature dimension. To obtain per-point input features, we first use a pre-
trained encoder [29] to extract feature grid $\mathbf{F}\in\mathbb{R}^{F\times
H^{\prime}\times W^{\prime}}$ from input image $\mathbf{I}$, here $F$ and
$H^{\prime},W^{\prime}$ are feature and spatial dimensions respectively.
Points $\mathbf{p}\in\mathbf{P}$ are then projected to 2D image plane with
$\pi(\cdot):\mathbb{R}^{3}\mapsto\mathbb{R}^{2}$ to extract pixel-aligned
feature $\mathbf{F}_{\pi(\mathbf{p})}$. We further concatenate it with point
location and diffusion timestamp encodings $t_{\text{enc}}$ as the input to
the diffusion model:
$\mathbf{F}_{\mathbf{p}}=(\mathbf{F}_{\pi(\mathbf{p})},\mathbf{p},t_{\text{enc}})$.
To allow generative prediction for points that are occluded, the image
features $\mathbf{F}_{\pi(\mathbf{p})}$ are set to zeros when points are
occluded [56].

####  3.2.3 Hierarchical Diffusion for Interaction

Naively using one network to reconstruct interaction leads to noisy point
predictions (see Fig. 5), as the combinatorial shape space of human-object
interaction is too complex to model. Thus, we propose a second stage to refine
human and object shapes separately, by having two additional diffusion models
while also preserving the interaction context. In the following, we discuss
special aspects of our second stage, namely 1) how the point cloud is
segmented into human and object, 2) how separate networks are designed to
model interaction, 3) how these models are combined.

Point cloud segmentation. To reason the contacts during interaction and obtain
accurate shapes for human and object separately, the combined point cloud
needs to be segmented into the points for human and object. To this end, we
use an additional network $g_{\phi}:\mathbb{R}^{N\times
D}\mapsto\\{0,1\\}^{N}$ that takes point features $\mathbf{F}_{\mathbf{p}}$ as
input and predicts a binary label to indicate whether this is a human or
object point. With this prediction, we can segment the point cloud
$\mathbf{P}$ predicted by $p_{\theta}$ into human and object points
$\mathbf{P}^{h},\mathbf{P}^{o}$.

Preserving interaction context. In our second stage, we use two additional
diffusion models $p_{\theta}^{h},p_{\theta}^{o}$ to predict human and object.
The networks follow the same design as the joint network $p_{\theta}$ using
PVCNN [103, 52]. To encourage the networks to explore interaction cues, we add
cross-attention layers between the encoder and decoder layers of human and
object branches. Given downsampled points
$\mathbf{P}_{l}\in\mathbb{R}^{N_{l}\times 3}$ with features
$\mathbf{F}_{l}\in\mathbb{R}^{N_{l}\times D_{l}}$ after network layer $l$, we
propagate information from human branch to object branch by computing feature:

$\mathbf{F}_{l}^{h\mapsto
o}=\text{Attn}(\text{enc}(\mathbf{P}_{l}^{o}),\text{enc}(\mathbf{P}_{l}^{h}),\mathbf{F}_{\mathbf{P}^{h}_{l}}),$
(5)

where $\text{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V})$ is learnable cross
attention[81], $\text{enc}(\cdot)$ is positional encoding from NeRF [57], and
$\mathbf{F}_{\mathbf{P}^{h}_{l}}=(\text{enc}(\mathbf{P}_{l}^{h}),\mathbf{C})$
is the concatenation of positional encoding and onehot encoding $\mathbf{C}$
indicating these points belong to human. The attention feature
$\mathbf{F}_{l}^{h\mapsto o}$ is then concatenated to the object feature
$\mathbf{F}_{l}^{o}$ as input to the next layer. We propagate information from
object to human branch similarly.

Figure 4: Comparing reconstruction results on BEHAVE[7] dataset. CHORE[87]
relies on object mesh templates and the prediction is inaccurate for
challenging poses. $\text{PC}^{2}$[56] does not rely on templates but its
predicted point clouds are noisy (red circles) and it cannot predict contacts.
Ours can reason about human object interaction, and predicts high-fidelity
human and object shapes without templates.

Model Combination. With the separate networks $p_{\theta}^{h},p_{\theta}^{o}$,
one can run the full reverse diffusion process from $t=T$ to $t=0$ and then
combine the denoised points. However, this does not leverage the predicted
interaction context from the joint reconstruction stage and is slow. We hence
start the reverse diffusion steps from an intermediate step $t=T_{0}$ instead
of $T$. Specifically, after denoising and segmentation with the joint model,
we apply the forward diffusion process to $\mathbf{P}^{h}$ and
$\mathbf{P}^{o}$ until step $t=T_{0}$ using Eq. 3. Then, the individual
diffusion models take the noised points as input and gradually denoise them
until step $t=0$. The forward process destroys local noisy predictions but
keeps the global structure of human-object interaction. We empirically set
$T_{0}=\frac{T}{2}$, see supp. for analysis of this value. We show in Tab. 5
and Fig. 5 that our hierarchical design is important to obtain sharp
predictions.

Recall from Eq. 3 that the forward diffusion ends up with a normal
distribution. Hence the input and output points of all diffusion models are
centered at the origin and scaled to unit sphere, which requires normalization
parameters to project them back to image. We estimate it for the first
diffusion model $p_{\theta}$ when GT is not available and compute them for
separate diffusion models $p_{\theta}^{h},p_{\theta}^{o}$ from the segmented
points. We show in Sec. 4.4 that it is better than directly predicting from
input image. More details in Supp.

Implementation. We train our diffusion models
$p_{\theta},p_{\theta}^{h},p_{\theta}^{o}$ using the standard loss ( Eq. 4)
and segmentation model $g_{\phi}$ using L2 distance between predicted and
ground truth binary labels. See Supp. for more implementation details.

##  4 Experiments

In this section, we first describe our data generation and then evaluate the
proposed ProciGen data and HDM for reconstruction. Please refer to supp. for
implementation details.

Data generation. We leverage the BEHAVE [7], InterCap [33], ShapeNet [12],
Objaverse [19], ABO [15] and MGN [4] dataset to generate our synthetic data
ProciGen. BEHAVE and InterCap capture multi-view images of humans interacting
with 20 and 10 different objects respectively. ShapeNet [12] and Objaverse
[19] are large-scale datasets that provide 3D object models as meshes with
textures. ABO [15] is a smaller shape dataset with high-quality material and
textures. The objects from ShapeNet and ABO are aligned in canonical space
while objects from Objaverse are not aligned. MGN [4] is a dataset consisting
of 100 human scans paired with SMPL-D registration that allows reposing scans
while preserving clothing deformation.

Following the same train-test split from [88], we randomly sample from 380k
interactions in BEHAVE and InterCap training set, 21k different shapes in
ShapeNet, ABO and Objaverse, and 100 different human shapes and textures in
MGN. In total, we generate $\sim$1.1million training images. Please see
supplementary for more data distribution details.

Evaluation metric. We evaluate the reconstruction performance using the
F-score based on Chamfer distance between point clouds, which is more suitable
for measuring the shape accuracy [76]. We compute F-score with a threshold of
0.01m [56] and report the error for human, object and combined point clouds
separately, as typically done in interaction reconstruction methods [88, 87].

###  4.1 Reconstruction on BEHAVE and InterCap

We compare our method with CHORE [87] and $\text{PC}^{2}$[56] on BEHAVE[7] and
InterCap [33] test set in Tab. 2 and Fig. 4. We train CHORE and
$\text{PC}^{2}$ on the training set of BEHAVE and InterCap. Our HDM is trained
on our synthetic ProciGen with or without BEHAVE and InterCap training set.

CHORE is designed for interaction reconstruction and requires _known_ object
templates. $\text{PC}^{2}$ is a general shape reconstruction method without
templates but it does not separate human and object hence cannot reason the
semantics of interaction. Our method trained only on our synthetic ProciGen
dataset performs on par with CHORE which already knows the template and
$\text{PC}^{2}$ which already sees the object shapes. After training our HDM
on both our ProciGen and real data, our method significantly outperforms
baselines.

| Method |  Human$\uparrow$ |  Object$\uparrow$ |  Comb.$\uparrow$
---|---|---|---|---
BEHAVE |  CHORE† | 0.3454 | 0.4258 | 0.3966
$\text{PC}^{2}$‡ | X | X | 0.4080
Ours‡ | 0.3925 | 0.5049 | 0.4604
Ours synth. only‡ | 0.3477 | 0.4351 | 0.4110
InterCap |  CHORE† | 0.4064 | 0.5135 | 0.4687
$\text{PC}^{2}$‡ | X | X | 0.5057
Ours‡ | 0.4399 | 0.6072 | 0.5344
Ours synth. only‡ | 0.3851 | 0.4928 | 0.4530
Table 2: Reconstruction results<EMAIL_ADDRESS>on BEHAVE [7] and InterCap [33].
† denotes methods with template meshes while ‡ denotes template-free methods.
CHORE [87] requires known object templates and is prone to noisy pose
predictions. $\text{PC}^{2}$ [56] does not require templates but cannot
predict semantics of human-object and the prediction is inaccurate. Our method
separates human and object, does not require any templates and outperforms
$\text{PC}^{2}$ and CHORE. Training only on our synthetic ProciGen data
performs on par with CHORE even it has never seen the objects.

###  4.2 Contribution of our ProciGen and HDM

We propose ProciGen for interaction data generation and HDM for interaction
reconstruction. To decouple the contribution of our data and method, we
compare our method against $\text{PC}^{2}$ [56] trained on BEHAVE [7] only
(Tab. 3 a-b) and BEHAVE + our ProciGen dataset (Tab. 3 c-d). The methods are
evaluated on the BEHAVE test set. It can be seen that both our proposed data
and model are important to obtain the most accurate reconstruction.

Method | Human | Object | Combined
---|---|---|---
a. $\text{PC}^{2}$ | X | X | 0.4080
b. Our HDM | 0.3463 | 0.4549 | 0.4125
c. $\text{PC}^{2}$ \+ our ProciGen | X | X | 0.4486
d. Our HDM + ProciGen | 0.3925 | 0.5049 | 0.4604
Table 3: Decoupling the contribution of our ProciGen dataset and
reconstruction method. Our ProciGen dataset significantly boosts performance
of both $\text{PC}^{2}$ (c) and our method (d) compared to training on BEHAVE
only (a-b). Both our ProciGen and HDM model are important to achieve the best
result. Figure 5: Generalization results to InterCap [33] dataset. Note that
all object instances are unseen during training time. CHORE [87] predicts
template specific object pose hence cannot generalize to new object instances.
$\text{PC}^{2}$[56] does not rely on template but its generalization ability
is constrained by limited shape variations from BEHAVE [7]. Training
$\text{PC}^{2}$ on our ProciGen improves its generalization but the predicted
point clouds are still noisy. Our method is able to generalize and predicts
human and object with high shape fidelity.

###  4.3 Generalization Performance

Our ProciGen dataset allows training shape reconstruction methods to
generalize to unseen object instances. To evaluate this, we train CHORE[87],
$\text{PC}^{2}$[56] and our HDM model on BEHAVE[7] and our proposed dataset
respectively. We then evaluate them on unseen objects of the same categories
from InterCap [33] in Table 4. CHORE requires a template to predict 6D pose,
which makes it difficult to train on our synthetic dataset with more than 21k
different shapes. We hence only train CHORE on BEHAVE dataset.

Methods trained on BEHAVE have limited generalization to InterCap (Tab. 4a-c).
An alternative to our ProciGen is to randomly scale and shift the objects from
BEHAVE and render new images, which only slightly improves generalization
(Tab. 4d). In contrast, our ProciGen significantly boosts the generalization
performance (Tab. 4e-f).

Some qualitative results are shown in Fig. 5. Our method reconstructs human
and object separately with high shape fidelity. We also show the
generalization results to COCO dataset [45] in Fig. 6. Our method trained
_only_ on our ProciGen data generalizes well to in-the-wild images with large
object shape variations. See Supp. for more examples.

Method |  Human $\uparrow$ |  Object$\uparrow$ |  Combine$\uparrow$
---|---|---|---
a. CHORE | 0.2263 | 0.1924 | 0.2176
b. $\text{PC}^{2}$ | X | X | 0.2327
c. Our HDM | 0.2389 | 0.1592 | 0.2127
d. Our HDM+ augm. | 0.3076 | 0.2089 | 0.2680
e. $\text{PC}^{2}$ \+ Our ProciGen | X | X | 0.3843
f. Our HDM+ ProciGen | 0.3502 | 0.4233 | 0.3976
Table 4: Generalization performance of methods trained on BEHAVE [7] (a-c),
BEHAVE + random augmentation (d) and our ProciGen (e-f), evaluated on unseen
objects from InterCap (F-score@0.01m). CHORE predicts template-specific 6D
poses hence does not work on unseen objects from InterCap. $\text{PC}^{2}$ (b)
and our method (c) do not require templates but are constrained by the limited
shape variations from BEHAVE. Adding random shape augmentation on BEHAVE
objects (d) slightly improves generalization but is still suboptimal. With our
proposed ProciGen dataset, both $\text{PC}^{2}$ and our method can generalize
to InterCap and our method achieves better accuracy.  Figure 6: Testing our
method on COCO [45] dataset. Human and object to be reconstructed are
highlighted with blue and yellow box respectively. Our method generalizes to
diverse objects from in the wild images _without any shape templates_.

###  4.4 Ablating the Hierarchical Diffusion Model

Our HDM predicts interaction semantics and better shape fidelity. In Table 5,
we ablate other alternatives to our method on the chair category from BEHAVE
test set [7] (824 images) due to resource limit. All methods are trained on
our ProciGen dataset.

The human-object segmentation allows us to compute the contacts and manipulate
human and object separately. An alternative is projecting the predicted points
to 2D image and segment points based on the masks. Due to occlusion and
complex interaction, this segmentation is inaccurate, which is reflected in
the large separate human and object errors in Tab. 5 a. The model that
predicts human, object and segmentation with a single model (Tab. 5 b) also
does not work as it is difficult to learn high-fidelity interaction shapes
within one model.

Another alternative to our first joint diffusion model is to use a network
that predicts translation and scale directly from input image and then use
them to combine predictions from two separate models. However, such a global
prediction does not model interaction with local fine-level details hence the
performance is subpar(Tab. 5b). Our cross attention module also improves the
performance (Tab. 5d).

Method (with our ProciGen) |  Hum.$\uparrow$ |  Obj.$\uparrow$ |  Comb.$\uparrow$
---|---|---|---
a. $\text{PC}^{2}$ \+ projected segm. | 0.2961 | 0.3436 | 0.3776
b. Single model + segm. | 0.3349 | 0.3638 | 0.3743
c. Direct pred. + sep. models | 0.2809 | 0.3487 | 0.3380
d. Ours w/o cross attention | 0.3387 | 0.3806 | 0.3807
e. Our full model | 0.3433 | 0.3916 | 0.3875
Table 5: Ablating alternative methods to our HDM (F-score@0.01m). Projecting
$\text{PC}^{2}$ predictions to 2D masks to obtain segmentation (a) is
inaccurate and single stage diffusion model (b) cannot learn high-fidelity
shapes for both human and object. Combining predictions from separate human
and object models using direct translation prediction from images (c) also
does not work as it cannot learn fine-grained interactions. Our hierarchical
design together with our cross attention module achieves the best result.

##  5 Conclusion

In this paper, we proposed a procedural generation method to synthesize
interaction datasets with diverse human and object shapes. This method allows
us to generate 1M+ images paired with clean 3D ground truth and train large
image-conditioned diffusion models for reconstruction, without relying on any
shape templates. To learn accurate shape space for human and object, we
introduce a hierarchical diffusion model that learns both the joint
interaction and high fidelity human and object shape subspaces.

We train our method with the proposed synthetic dataset and evaluate it on
BEHAVE and InterCap datasets. Results show that our method significantly
outperforms CHORE which requires template meshes and $\text{PC}^{2}$ which
does not reason interaction semantics. Ablation studies also show that our
synthetic dataset is important to boost the performance and generalization
ability of both $\text{PC}^{2}$ and our model. Our method generalizes well to
real images from COCO that have diverse object geometries, which is a
promising step toward real in-the-wild reconstruction. Our code will be
released to promote future research.

Acknowledgements. We thank RVH group members [2], especially Yuxuan Xue, for
their helpful discussions. This work is funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) - 409792180 (Emmy
Noether Programme, project: Real Virtual Humans), and German Federal Ministry
of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A. Gerard
Pons-Moll is a Professor at the University of Tübingen endowed by the Carl
Zeiss Foundation, at the Department of Computer Science and a member of the
Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number
390727645.

## References

  * [1] https://renderpeople.com/.
  * [2] http://virtualhumans.mpi-inf.mpg.de/people.html.
  * Achlioptas et al. [2018] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.  Learning representations and generative models for 3d point clouds, 2018.
  * Bhatnagar et al. [2019] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll.  Multi-garment net: Learning to dress 3d people from images.  In _IEEE International Conference on Computer Vision (ICCV)_. IEEE, 2019.
  * Bhatnagar et al. [2020a] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll.  Combining implicit function learning and parametric models for 3d human reconstruction.  In _European Conference on Computer Vision (ECCV)_. Springer, 2020a.
  * Bhatnagar et al. [2020b] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll.  Loopreg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020b.
  * Bhatnagar et al. [2022] Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll.  Behave: Dataset and method for tracking human object interactions.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Biswas et al. [2023] Sandika Biswas, Kejie Li, Biplab Banerjee, Subhasis Chaudhuri, and Hamid Rezatofighi.  Physically plausible 3d human-scene reconstruction from monocular rgb image using an adversarial learning approach.  _IEEE Robotics and Automation Letters_ , 8(10):6227–6234, 2023.
  * Black et al. [2023] Michael J. Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang.  BEDLAM: A synthetic dataset of bodies exhibiting detailed lifelike animated motion.  In _Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 8726–8737, 2023.
  * Brahmbhatt et al. [2019] Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays.  ContactDB: Analyzing and predicting grasp contact via thermal imaging.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Brahmbhatt et al. [2020] Samarth Brahmbhatt, Chengcheng Tang, Christopher D. Twigg, Charles C. Kemp, and James Hays.  ContactPose: A dataset of grasps with object contact and hand pose.  In _The European Conference on Computer Vision (ECCV)_ , 2020.
  * Chang et al. [2015] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * Chen et al. [2023a] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner.  Text2tex: Text-driven texture synthesis via diffusion models.  _arXiv preprint arXiv:2303.11396_ , 2023a.
  * Chen et al. [2023b] Yixin Chen, Sai Kumar Dwivedi, Michael J. Black, and Dimitrios Tzionas.  Detecting human-object contact in images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023b.
  * Collins et al. [2022] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik.  Abo: Dataset and benchmarks for real-world 3d object understanding.  _CVPR_ , 2022.
  * Community [2018] Blender Online Community.  _Blender - a 3D modelling and rendering package_.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
  * Corona et al. [2020] Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Gregory Rogez.  Ganhand: Predicting human grasp affordances in multi-object scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Corona et al. [2022] Enric Corona, Gerard Pons-Moll, Guillem Alenya, and Francesc Moreno-Noguer.  Learned vertex descent: A new direction for 3d human model fitting.  In _European Conference on Computer Vision (ECCV)_. Springer, 2022.
  * Deitke et al. [2022a] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  _arXiv preprint arXiv:2212.08051_ , 2022a.
  * Deitke et al. [2022b] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi.  ProcTHOR: Large-Scale Embodied AI Using Procedural Generation.  In _NeurIPS_ , 2022b.  Outstanding Paper Award.
  * Deng et al. [2021] Yu Deng, Jiaolong Yang, and Xin Tong.  Deformed implicit field: Modeling 3d shapes with learned dense correspondence.  In _IEEE Computer Vision and Pattern Recognition_ , 2021.
  * Ehsani et al. [2020] Kiana Ehsani, Shubham Tulsiani, Saurabh Gupta, Ali Farhadi, and Abhinav Gupta.  Use the force, luke! learning to predict physical forces by simulating effects.  In _CVPR_ , 2020.
  * Greff et al. [2022] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi.  Kubric: a scalable dataset generator.  2022\.
  * Guzov et al. [2021] Vladimir Guzov, Aymen Mir, Torsten Sattler, and Gerard Pons-Moll.  Human poseitioning system (hps): 3d human pose estimation and self-localization in large scenes from body-mounted sensors.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2021.
  * Guzov et al. [2024] Vladimir Guzov, Julian Chibane, Riccardo Marin, Yannan He, Yunus Saracoglu, Torsten Sattler, and Gerard Pons-Moll.  Interaction replica: Tracking human–object interaction and scene changes from human motion.  In _International Conference on 3D Vision (3DV)_ , 2024.
  * Hampali et al. [2020] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit.  Honnotate: A method for 3d annotation of hand and object poses.  In _CVPR_ , 2020.
  * Hassan et al. [2019] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Black.  Resolving 3d human pose ambiguities with 3d scene constraints.  In _International Conference on Computer Vision_ , 2019.
  * Hasson et al. [2019] Yana Hasson, Gül Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning joint reconstruction of hands and manipulated objects.  In _CVPR_ , 2019.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick.  Masked Autoencoders Are Scalable Vision Learners.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15979–15988, New Orleans, LA, USA, 2022. IEEE.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _arXiv preprint arxiv:2006.11239_ , 2020.
  * Hu et al. [2017] JF Hu, WS Zheng, J Lai, and J Zhang.  Jointly learning heterogeneous features for rgb-d activity recognition.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(11):2186–2200, 2017.
  * Huang et al. [2022a] Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Black.  Capturing and inferring dense full-body human-scene contact.  In _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 13274–13285, 2022a.
  * Huang et al. [2022b] Yinghao Huang, Omid Taheri, Michael J. Black, and Dimitrios Tzionas.  InterCap: Joint markerless 3D tracking of humans and objects in interaction.  In _German Conference on Pattern Recognition (GCPR)_ , pages 281–299. Springer, 2022b.
  * Huang et al. [2024] Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, and Justus Thies.  TeCH: Text-guided Reconstruction of Lifelike Clothed Humans.  In _International Conference on 3D Vision (3DV)_ , 2024.
  * Ionescu et al. [2014] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36(7):1325–1339, 2014.
  * Jiang et al. [2023a] Li Jiang, Zetong Yang, Shaoshuai Shi, Vladislav Golyanik, Dengxin Dai, and Bernt Schiele.  Self-supervised pre-training with masked shape prediction for 3d scene understanding.  In _CVPR_ , 2023a.
  * Jiang et al. [2023b] Yuheng Jiang, Kaixin Yao, Zhuo Su, Zhehao Shen, Haimin Luo, and Lan Xu.  Instant-nvr: Instant neural volumetric rendering for human-object interactions from monocular rgbd stream, 2023b.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Karunratanakul et al. [2020] Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael Black, Krikamol Muandet, and Siyu Tang.  Grasping field: Learning implicit representations for human grasps.  In _8th International Conference on 3D Vision_ , pages 333–344. IEEE, 2020.
  * Ke et al. [2023] Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu.  Segment anything in high quality.  _arXiv:2306.01567_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023.
  * Li et al. [2022a] Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan.  Cliff: Carrying location information in full frames into human pose and shape estimation.  In _ECCV_ , 2022a.
  * Li et al. [2022b] Zhi Li, Soshi Shimada, Bernt Schiele, Christian Theobalt, and Vladislav Golyanik.  Mocapdeform: Monocular 3d human motion capture in deformable scenes.  In _International Conference on 3D Vision (3DV)_ , 2022b.
  * Liao et al. [2024] Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxiang Tang, Yangyi Huang, Justus Thies, and Michael J. Black.  TADA! Text to Animatable Digital Avatars.  In _International Conference on 3D Vision (3DV)_ , 2024.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision – ECCV 2014_ , pages 740–755, Cham, 2014. Springer International Publishing.
  * Liu and Liu [2020] Feng Liu and Xiaoming Liu.  Learning implicit functions for topology-varying dense 3d shape correspondence, 2020.
  * Liu et al. [2022] Haotian Liu, Mu Cai, and Yong Jae Lee.  Masked discrimination for self-supervised learning on point clouds.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2022.
  * Liu et al. [2019a] Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling-Yu Duan, and Alex C. Kot.  Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019a.
  * Liu et al. [2023a] Jia-Wei Liu, Yan-Pei Cao, Tianyuan Yang, Eric Zhongcong Xu, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Hosnerf: Dynamic human-object-scene neural radiance fields from a single video.  _arXiv preprint arXiv:2304.12281_ , 2023a.
  * Liu et al. [2023b] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund T, Zexiang Xu, and Hao Su.  One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization.  In _Annual Conference on Neural Information Processing Systems (NeurIPS)_ , 2023b.
  * Liu et al. [2023c] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object, 2023c.
  * Liu et al. [2019b] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han.  Point-voxel cnn for efficient 3d deep learning.  In _Conference on Neural Information Processing Systems (NeurIPS)_ , 2019b.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  In _ACM Transactions on Graphics_. ACM, 2015.
  * Mahmood et al. [2019] Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black.  AMASS: Archive of motion capture as surface shapes.  In _International Conference on Computer Vision_ , pages 5442–5451, 2019.
  * Melas-Kyriazi et al. [2023a] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Realfusion: 360 reconstruction of any object from a single image.  In _CVPR_ , 2023a.
  * Melas-Kyriazi et al. [2023b] Luke Melas-Kyriazi, Christian Rupprecht, and Andrea Vedaldi.  Pc2: Projection-conditioned point cloud diffusion for single-image 3d reconstruction.  In _CVPR_ , 2023b.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Pang et al. [2022] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan.  Masked autoencoders for point cloud self-supervised learning.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part II_ , pages 604–621. Springer, 2022.
  * Patel et al. [2021] Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, Shashank Tripathi, and Michael J. Black.  AGORA: Avatars in geography optimized for regression analysis.  In _Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Petrov et al. [2023] Ilya A Petrov, Riccardo Marin, Julian Chibane, and Gerard Pons-Moll.  Object pop-up: Can we infer 3d objects and their poses from human interactions alone?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv_ , 2022.
  * Qi et al. [2017] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  _CVPR_ , abs/1612.00593, 2017.
  * Raistrick et al. [2023] Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, Alejandro Newell, Hei Law, Ankit Goyal, Kaiyu Yang, and Jia Deng.  Infinite photorealistic worlds using procedural generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12630–12641, 2023.
  * Romero et al. [2017] Javier Romero, Dimitrios Tzionas, and Michael J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  _ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)_ , 36(6), 2017.
  * Rong et al. [2021] Yu Rong, Takaaki Shiratori, and Hanbyul Joo.  Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration.  In _IEEE International Conference on Computer Vision Workshops_ , 2021.
  * Saito et al. [2019] Shunsuke Saito, , Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In _IEEE International Conference on Computer Vision (ICCV)_. IEEE, 2019.
  * Saito et al. [2020] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Seo et al. [2023] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim.  Let 2d diffusion model know 3d-consistency for robust text-to-3d generation.  _arXiv preprint arXiv:2303.07937_ , 2023.
  * Shi et al. [2023] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su.  Zero123++: a single image to consistent multi-view diffusion base model, 2023.
  * Shimada et al. [2022] Soshi Shimada, Vladislav Golyanik, Zhi Li, Patrick Pérez, Weipeng Xu, and Christian Theobalt.  Hulc: 3d human motion capture with pose manifold sampling and dense contact guidance.  In _European Conference on Computer Vision (ECCV)_ , pages 516–533, 2022.
  * Sofiiuk et al. [2020] Konstantin Sofiiuk, Ilia Petrov, Olga Barinova, and Anton Konushin.  f-brs: Rethinking backpropagating refinement for interactive segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8623–8632, 2020.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  _Generative Modeling by Estimating Gradients of the Data Distribution_.  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * Stutz and Geiger [2018] David Stutz and Andreas Geiger.  Learning 3d shape completion under weak supervision.  _CoRR_ , abs/1805.07290, 2018.
  * Sundararaman et al. [2022] Ramana Sundararaman, Riccardo Marin, Emanuele Rodola, and Maks Ovsjanikov.  Reduced representation of deformation fields for effective non-rigid shape matching.  _Advances in Neural Information Processing Systems_ , 35, 2022.
  * Tao et al. [2018] Yu Tao, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Dai Quionhai, Hao Li, G. Pons-Moll, and Yebin Liu.  Doublefusion: Real-time capture of human performance with inner body shape from a depth sensor.  In _IEEE Conf. on Computer Vision and Pattern Recognition_ , 2018.
  * Tatarchenko* et al. [2019] Maxim Tatarchenko*, Stephan R. Richter*, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  2019\.
  * Tiwari et al. [2020] Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll.  Sizer: A dataset and model for parsing 3d clothing and learning size sensitive 3d clothing.  In _European Conference on Computer Vision (ECCV)_. Springer, 2020.
  * Tripathi et al. [2023] Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, and Michael J. Black.  DECO: Dense estimation of 3D human-scene contact in the wild.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 8001–8013, 2023.
  * Tyszkiewicz et al. [2023] Michal J. Tyszkiewicz, P. Fua, and Eduard Trulls.  Gecco: Geometrically-conditioned point diffusion models.  _ICCV_ , abs/2303.05916, 2023.
  * Tzionas et al. [2016] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall.  Capturing hands in action using discriminative salient points and physics simulation.  _International Journal of Computer Vision (IJCV)_ , 2016.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Wang et al. [2022] Xi Wang, Gen Li, Yen-Ling Kuo, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges.  Reconstructing action-conditioned human-object interactions using commonsense knowledge priors.  In _International Conference on 3D Vision (3DV)_ , 2022.
  * Wen et al. [2023] Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield.  BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects.  In _CVPR_ , 2023.
  * Wewer et al. [2023] Christopher Wewer, Eddy Ilg, Bernt Schiele, and Jan Eric Lenssen.  Simnp: Learning self-similarity priors between neural points, 2023.
  * Wrenninge and Unger [2018] Magnus Wrenninge and Jonas Unger.  Synscapes: A photorealistic synthetic dataset for street scene parsing, 2018.
  * Wu et al. [2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1912–1920, 2015.
  * Xie et al. [2022] Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll.  Chore: Contact, human and object reconstruction from a single rgb image.  In _European Conference on Computer Vision (ECCV)_. Springer, 2022.
  * Xie et al. [2023] Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll.  Visibility aware human-object interaction tracking from single rgb camera.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Xiu et al. [2022] Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, and Michael J. Black.  ICON: Implicit Clothed humans Obtained from Normals.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 13296–13306, 2022.
  * Xiu et al. [2023] Yuliang Xiu, Jinlong Yang, Xu Cao, Dimitrios Tzionas, and Michael J. Black.  ECON: Explicit Clothed humans Optimized via Normal integration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Xu et al. [2021] Xiang Xu, Hanbyul Joo, Greg Mori, and Manolis Savva.  D3d-hoi: Dynamic 3d human-object interactions from videos.  _arXiv preprint arXiv:2108.08420_ , 2021.
  * Yang et al. [2021] Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu.  CPF: Learning a contact potential field to model the hand-object interaction.  In _ICCV_ , 2021.
  * Ye et al. [2023] Jianglong Ye, Naiyan Wang, and Xiaolong Wang.  Featurenerf: Learning generalizable nerfs by distilling pre-trained vision foundation models.  _arXiv preprint arXiv:2303.12786_ , 2023.
  * Yi et al. [2022] Hongwei Yi, Chun-Hao P. Huang, Dimitrios Tzionas, Muhammed Kocabas, Mohamed Hassan, Siyu Tang, Justus Thies, and Michael J. Black.  Human-aware object placement for visual environment reconstruction.  In _IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 3959–3970, 2022.
  * Yu et al. [2022] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-bert: Pre-training 3d point cloud transformers with masked point modeling.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Zhang et al. [2017] Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll.  Detailed, accurate, human shape estimation from clothed 3d scan sequences.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Zhang et al. [2023] Juze Zhang, Haimin Luo, Hongdi Yang, Xinru Xu, Qianyang Wu, Ye Shi, Jingyi Yu, Lan Xu, and Jingya Wang.  Neuraldome: A neural modeling pipeline on multi-view human-object interactions.  In _CVPR_ , 2023.
  * Zhang et al. [2020] Jason Y. Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa.  Perceiving 3d human-object spatial arrangements from a single image in the wild.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Zhang et al. [2022a] Jason Y. Zhang, Deva Ramanan, and Shubham Tulsiani.  RelPose: Predicting probabilistic relative rotation for single objects in the wild.  In _European Conference on Computer Vision_ , 2022a.
  * Zhang et al. [2018] Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B Tenenbaum, William T Freeman, and Jiajun Wu.  Learning to Reconstruct Shapes From Unseen Classes.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.
  * Zhang et al. [2022b] Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll.  Couch: Towards controllable human-chair interactions.  In _European Conference on Computer Vision (ECCV)_. Springer, 2022b.
  * Zhou et al. [2022] Keyang Zhou, Bharat Lal Bhatnagar, Bernt Schiele, and Gerard Pons-Moll.  Adjoint rigid transform network: Task-conditioned alignment of 3d shapes.  In _2022 International Conference on 3D Vision (3DV)_. IEEE, 2022.
  * Zhou et al. [2021] Linqi Zhou, Yilun Du, and Jiajun Wu.  3d shape generation and completion through point-voxel diffusion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 5826–5835, 2021.
  * Zhou and Tulsiani [2023] Zhizhuo Zhou and Shubham Tulsiani.  Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction.  In _CVPR_ , 2023.

Supplementary Material

In this supplementary, we first discuss in more detail about our
implementation for the ProciGen and HDM in Appendix A. We also present the
statistics of our generated ProciGen dataset. We then show more results and
experimental analysis of our method in Appendix B. We conclude with a
discussion of limitations and future works.

##  Appendix A Implementation Details

We describe in more details of the implementation of our ProciGen and HDM. Our
code for both data generation and reconstruction will be made publicly
available.

###  A.1 ProciGen Data Generation

Correspondence estimation. We use the implementation from ART [102] for our
autoencoder, which uses PointNet [62] as encoder and 3-layer MLPs as decoder.
We sample 8000 points from the mesh surface and train the network with
bidirectional Chamfer distance. To ensure reconstruction quality, we overfit
one network per category. Each model is trained for 5000 epochs. We report an
average reconstruction error of around 7mm for our autoencoders, which
indicates highly accurate reconstructions.

Contact transfer and optimization. We use a threshold of $\sigma=2$cm to find
points that are in contact. The loss weights for our contact based loss
optimization are:
$\lambda_{c}=400,\lambda_{n}=6.25,\lambda_{\text{colli}}=9,\lambda_{\text{init}}=6.25\cdot
10^{4}$.

Rendering. We use blender to render our synthesized human-object interactions.
We choose one set of 4 camera configurations from BEHAVE [7] and another set
of 6 camera configurations from InterCap [33]. For each synthesized
interaction, we additionally add small random global rotation and translation
to have variations of camera viewpoints. We render the interactions with an
empty background since our network also takes images with background masked
out as input. We add lights at fixed locations with random light intensities.
Our blender scene and rendering code will also be made publicly available.

Figure 7: The number of objects per category we used to generate our ProciGen
dataset. It can be seen that the shape variations are dominated by tables and
chairs, which are also the categories with the most complex shapes. Figure 8:
Distribution of interactions per object category. Our dataset features
interaction data with very diverse object shapes, which is not possible via
real data capture

###  A.2 HDM: Hierarachical Diffusion Model

We use the modified Point Voxel CNN from [103] as the network for our joint
diffusion $p_{\theta}$, segmentation $g_{\phi}$, and separate diffusion models
$p_{\theta}^{h},p_{\theta}^{o}$. The input images are cropped and resized to
$224\times 224$. The joint diffusion model diffuses in total 16384 points
while the separate models diffuse 8196 points each. We use the MAE [29] as the
image feature encoder. We additionally stack the human and object masks as
well as distance transform as additional image features, same as
$\text{PC}^{2}$ [56]. We train our diffusion models for a total of 500000
steps with batch size 20. We use a linear scheduler without warm-up for the
forward diffusion process, in which beta increases from $1\cdot 10^{-5}$ to
$8\cdot 10^{-3}$. For the network optimization, we use AdamW optimizer with
linear learning rate decay starting from $3\cdot 10^{-4}$ and decreasing to 0
during the course of training. The diffusion models are trained with the
standard diffusion training scheme [30]. To train the segmentation model, we
add small Gaussian noise to the GT point clouds and project them to obtain
image features. The loss is then computed between the prediction and
recomputed GT labels on the points with noise. To speed up training, we train
stage 1 ($g_{\phi},p_{\theta}$) and stage 2 ($p_{\theta}^{h},p_{\theta}^{o}$)
models separately. For each stage, it takes around 4 days to train on a
machine with 4 A40 GPUs.

Camera estimation. Recall from Sec. 3.2.3 that a camera translation is
required to project the normalized point clouds back to the image. This needs
to be estimated from input when GT camera pose is not available, especially
for generalization to diverse datasets. The camera translation consists of
three unknowns, which requires at least two point pairs of 3D location and
2D-pixel coordinates. We empirically choose the Gaussian point center and one
edge of the point cloud. The idea is to have the initial Gaussian point clouds
cover the 2D human object interaction region and the 3D center is projected to
2D crop center.

Formally, let $\mathbf{p}_{c}=(c_{x},c_{y})$ be the center coordinate of the
2D interaction region, $w$ be the width of the 2D interaction square crop,
$\mathbf{p}_{e}=(\sigma,0,z)$ be a 3D point near the edge of the Gaussian
sphere with unknown depth $z$. Given camera projection matrix
$\mathbf{K}\in\mathbb{R}^{3\times 3}$ and translation vector $\mathbf{t}_{c}$,
we define the following equations:

$\mathbf{K}\mathbf{t}_{c}=\mathbf{p}_{c};\ \
\mathbf{K}(\mathbf{p}_{e}+\mathbf{t}_{c})=\mathbf{p}^{\text{2D}}_{e}$ (6)

The first equation projects origin to $\mathbf{p}_{c}$ and the second equation
projects $\mathbf{p}_{e}$ to the middle right of the 2D crop
$\mathbf{p}^{\text{2D}}_{e}=(c_{x}+w/2,c_{y})$. This is a linear system of
four equations with four unknowns (camera translation and depth $z$), leading
to a unique solution for the translation $\mathbf{t}_{c}$. We empirically set
$\sigma$ to different values for different categories based on the estimation
error on the BEHAVE training set. From Fig. 6, Fig. 15, Fig. 14, Fig. 16, Fig.
17 and , Fig. 18, it can be seen that our method can reconstruct human and
object well on different datasets using our estimated translation.

Figure 9: The performance of our method using different intermediate step
$T_{0}$ for the input to our second stage diffusion. Methods are evaluated
using<EMAIL_ADDRESS>At $T_{0}=500$, we obtain a good balance between human
and object performance.

###  A.3 Dataset statistics

We generate our ProciGen dataset based on interactions from BEHAVE [7] and
InterCap [33], human scans from MGN [4], object shapes from ShapeNet [12],
Objaverse [19] and ABO [15]. In total we rendered 1.1M interaction images with
21555 different object shapes. The distribution for object shapes and
interactions per category are shown in Fig. 7 and Fig. 8. Our dataset has very
diverse object shapes, especially for chairs and tables whose geometry also
varies a lot in reality. Our procedural generation method is a scalable
solution and it allows for generating large-scale interaction datasets with
great amount of variations which is not obtainable via capturing real data.

##  Appendix B Additional Experiments

###  B.1 Analysis of $T_{0}$ for our HDM

In our second stage, we first add noise to the clean predictions from stage
one until step $t=T_{0}$, and then run the reverse diffusion process from
$t=T_{0}$ to $t=0$. We evaluate the performance of our method under different
values of $T_{0}$ in Figure 9. There is a trade-off for the number of forward
steps $T_{0}$: with a larger $T_{0}$, less interaction information and noisy
details are preserved and the network predicts sharper detail but less
faithful to initial prediction and interaction constraints. It can be seen
that $T_{0}=500$ is a good balance between shape fidelity and interaction
coherence.

###  B.2 Shape fidelity

Figure 10: Comparing the shape fidelity of our method with $\text{PC}^{2}$ on
the BEHAVE [7] dataset. $\text{PC}^{2}$ does not separate human and object and
its prediction is noisy, leading to inaccurate meshes. Our method predicts
clean point clouds with human object segmentations, allowing us to extract
high-quality mesh surfaces.

Our method predicts dense and clean point clouds which are ready for accurate
surface extraction. We show in Fig. 10 that high-quality meshes can be
extracted from our predicted point clouds. More specifically, we use screened
Poisson surface reconstruction for the human points using normals estimated by
MeshLab. For the object, we first use Delaunay triangulation to obtain
triangle mesh. We then run fusion-based waterproofing [73] to obtain a
watertight mesh. We also apply Delaunay triangulation and waterproofing to
$\text{PC}^{2}$ [56] predictions and results are shown in Fig. 10. It can be
seen that $\text{PC}^{2}$ predictions have missing structure and noisy point
clouds, leading to low-quality meshes. In contract, we can extract high-
quality meshes directly from our point cloud reconstructions, without any post
processing.

###  B.3 Interaction semantics

Our method predicts the segmentation of human and object, allowing separate
manipulation which is important for downstream applications. To demonstrate
this, we use Text2txt [13] to generate textures for the meshes extracted from
$\text{PC}^{2}$ and our predicted point clouds. We show the reconstruction and
generated textures in Fig. 11. It can be seen that $\text{PC}^{2}$ predictions
are noisy and it does not reason human and object separately. This leads to
low-quality mesh and generating coherent texture for a combined mesh of human
and object is difficult. On the contrary, our method separate human and object
while also predicting high quality individual shapes. This allows generating
high quality texture and changing textures for human and object differently.

Figure 11: Comparing textures generated for meshes extracted from
$\text{PC}^{2}$ [56] and our predicted point clouds. Textures are obtained
using Text2txt [13]. $\text{PC}^{2}$ predicts human and object as one joint
point cloud with noisy points, which leads to inaccurate mesh surfaces and it
is difficult to generate textures for this combined mesh. It also does not
allow changing human and object textures separately. Our method predicts high
quality point clouds with segmentation. This enables us to extract high-
fidelity mesh, which is important for generating high-quality texture and
manipulating human and object differently.  Figure 12: Example failure cases
of our method. Our method can fail when large parts of human body are
invisible, leading to incoherent human shape reconstructions. Future works can
explore human body shape priors to regularize the network predictions.

###  B.4 More generalization results

Figure 13: Comparing generalization performance on InterCap [33]. All objects
are unseen during training time. $\text{PC}^{2}$ trained only on BEHAVE [7]
has limited generalization ability. Training $\text{PC}^{2}$ with our ProciGen
improves generalization but it still cannot reason human and object separately
and the predicted points are noisy. Our method trained only on our ProciGen
generalizes well to InterCap objects even they are completely unseen.

We show more generalization comparison on the InterCap [33] dataset in Fig.
13. Note that all objects from InterCap are unseen during training time. It
can be seen that $\text{PC}^{2}$ trained on BEHAVE [7] only cannot generalize
to objects from InterCap. Training $\text{PC}^{2}$ with our ProciGen dataset
allows better generalization ability but its shape prediction is still noisy.
Furthermore, $\text{PC}^{2}$ cannot segment human and object, which is
important to reason the interaction semantics and manipulate them separately.
Our method generalizes well to InterCap and reconstructs high quality shapes
with interaction semantics.

Our method trained _only_ on our synthetic ProciGen dataset generalizes well
to other datasets. We show results on NTU-RGBD [48], SYSU [31] and challenging
in the wild COCO [45] images in figure Fig. 14, Fig. 15 and Fig. 16, Fig. 17,
Fig. 18 respectively. Note that our method is trained only on our synthetic
ProciGen dataset and not fine-tuned on any images from these datasets. It can
be seen that our method generalizes to different datasets with diverse object
shapes, without requiring any template meshes.

Figure 14: Generalization results on NTU-RGBD [48] dataset. Our method can
reconstruct different objects faithfully under various camera viewpoints and
lighting conditions, without relying on any template shapes.  Figure 15:
Generalization results on SYSU action [31] dataset. Our method can reconstruct
different real-life human and objects during challenging interactions and
occlusions.  Figure 16: Generalization results to COCO [45] dataset. Our
method can reconstruct high-quality human and object from in the wild images
which has very diverse shape variations, without using any template shapes.
Figure 17: Generalization results to COCO [45] dataset. Our method
reconstructs diverse object shapes in the wild. Figure 18: Generalization
results to COCO [45] dataset. Our method can reconstruct challenging human and
object pose as well as shapes without using any template shapes.

##  Appendix C Limitations and Future Work

We present a scalable solution to synthesize large amount of interaction
dataset which allows training methods with strong generalization ability. We
also propose a model for obtaining high quality human, object shapes and also
interaction semantics, without any template shapes. We demonstrate the
generalization ability of our method on diverse datasets. Our template-free
reconstruction method is a promising first step towards real in-the-wild
reconstruction.

Nevertheless, there are still some limitations to the current approach. First,
our ProciGen data generation method always starts with a seed interaction pose
sampled from an existing interaction dataset. This limits the diversity in
terms of interaction poses. Future works can explore generative models such as
Object-Popup [60] to further diversify the interaction pose. It is also highly
desirable to combine the large human pose variations from AMASS [54], which
can further improve the robustness of reconstruction methods to challenging
poses.

Secondly, our method struggles to predict accurate human shapes when large
chunk of the human body is occluded, see Fig. 12. This is because our method
is purely template-free and only use the network to learn the human and object
shape priors. Future works can try to further explore human shape or pose
constraints to regularize network training and predictions. In addition, our
hierarchical diffusion model are designed for human object interaction, which
is applicable for general bilateral interaction cases like human-human, hand-
hand, and hand-object interactions. However, it cannot handle multi-person or
multi-object interactions. We leave these for future works.