# CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations
from Unbounded Synthesized Images

Sookwan Han Hanbyul Joo
Seoul National University
{jellyheadandrew<EMAIL_ADDRESS>
###### Abstract

We present a method for teaching machines to understand and model the
underlying spatial common sense of diverse human-object interactions in 3D in
a self-supervised way. This is a challenging task, as there exist specific
manifolds of the interactions that can be considered human-like and natural,
but the human pose and the geometry of objects can vary even for similar
interactions. Such diversity makes the annotating task of 3D interactions
difficult and hard to scale, which limits the potential to reason about that
in a supervised way. One way of learning the 3D spatial relationship between
humans and objects during interaction is by showing multiple 2D images
captured from different viewpoints when humans interact with the same type of
objects. The core idea of our method is to leverage a generative model that
produces high-quality 2D images from an arbitrary text prompt input as an
“unbounded” data generator with effective controllability and view diversity.
Despite its imperfection of the image quality over real images, we demonstrate
that the synthesized images are sufficient to learn the 3D human-object
spatial relations. We present multiple strategies to leverage the synthesized
images, including (1) the first method to leverage a generative image model
for 3D human-object spatial relation learning; (2) a framework to reason about
the 3D spatial relations from inconsistent 2D cues in a self-supervised manner
via 3D occupancy reasoning with pose canonicalization; (3) semantic clustering
to disambiguate different types of interactions with the same object types;
and (4) a novel metric to assess the quality of 3D spatial learning of
interaction. ††Project Page: https://jellyheadandrew.github.io/projects/chorus

Figure 1: CHORUS. Given an object category, our method, CHORUS, automatically
learns human-object 3D spatial arrangements in a self-supervised way. Our
method leverages a generative model to synthesize an unbounded number of
images to reason about the 3D spatial relationship. As output, CHORUS produces
the 3D spatial distribution in the canonical space, which can be deformed for
any posed space.

##  1 Introduction

Humans interact with objects in specific ways. We wear shoes on our feet, a
hat on our heads, and ride a bike by holding handles and putting two feet on
the pedals. While this common sense regarding the 3D arrangements of the way
we interact with objects is known to us, teaching such things to robots and
machines is challenging, requiring numerous variations in diverse human-object
interactions in 3D.

As providing 3D supervision by manually annotating various cases is hard to
scale, an alternative way of teaching such things is by showing 2D photos from
the Internet containing the interaction with the same object from many
viewpoints. A text-based image retrieval (e.g., Google Image Search) can be an
option to crawl many images from a text description similar to NEIL [9].
However, this approach fundamentally suffers from several obstacles such as
(other than the challenges in 3D spatial relation reasoning): (1) viewpoint
variations are insufficient and hard to control; (2) the number of related
images decreases drastically as the compositionality of the prompt increases;
and (3) the images are often biased due to commercial websites.

In this paper, we present a novel idea of leveraging a text-conditional
generative model [59] as a controllable data producer in synthesizing
“unbounded”, “multi-view”, “diverse” images to learn the 3D human-object
spatial relations in a self-supervised manner. Despite its imperfectness in
quality, we observe that the synthesized images from generative models are
more suitable for our objective, as the generative model effectively links
desired semantics of human-object interaction (HOI) described in natural
language. Our synthesis-based approach allows better controllability in
obtaining images for spatial relation learning, providing more relevant images
from diverse viewpoints. See the examples in Fig. 2.

Nonetheless, inferring 3D spatial knowledge in human-object interaction from
in-the-wild 2D image collections is still a non-trivial problem due to the
inconsistency and “wildness” among synthesized images. In particular, a method
should handle the following challenges: (1) semantic variation: given a
category, there can be various semantic situations of human-object
interaction; thus, the spatial distribution of object may vary significantly
as semantics vary; (2) human pose variation: even assuming the same semantic
situation, human postures can vary as diverse actions and pose are available
in the same situation; (3) intra-class variation: object can exist in various
forms (even if same category); and (4) visual variance: when learning from 2D
cues, the visual properties (e.g., illumination, camera) may vary for even the
same 3D arrangement, making it difficult to localize and extract 2D cues.

Figure 2: Internet-crawled images vs. synthesized images. Under the prompt “A
person is riding a bicycle, top view”, synthesized images exhibit superior
fidelity to the intended “top view” perspective compared to internet-crawled
images. The correct viewpoint image is indicated by the green bounding box.

To this end, we present a self-supervised method to learn the spatial common
sense of diverse human-object interactions in 3D for arbitrary object
categories without any human annotations. We present several novel components
to achieve the goal, including (1) automatic prompt generation for diverse
view and semantic variations via chatGPT [51], (2) outlier filtering
strategies, (3) automatic camera view calibrations by using a human as an
anchor, (4) accumulating spatial interaction cues from inconsistent multi-view
2D knowledge with varying human pose, object geometry, and (5) clustering for
various semantic human-object interaction types. The output of our method can
be considered as possible occupancy distributions of the category-specified 3D
object relative to the human body in a canonical person-centric space
regarding the intra-class object variation. We demonstrate the efficacy of our
method on various object categories and human-object interaction types, as
shown in Fig. 1 and Fig. 7. As the first in this direction, we also introduce
a new metric, namely Projective Average Precision (PAP), to quantify the
quality of 3D spatial inference outputs. Our contributions are summarized as
follows: (1) the first method to leverage a generative text/image model for 3D
human-object spatial relation learning, including automatic prompt generation,
outlier filtering, and 3D viewpoint estimation via estimated 3D humans; (2) a
framework to reason about the 3D spatial relations from inconsistent 2D cues
in a self-supervised manner via 3D occupancy reasoning with pose
canonicalization; (3) semantic clustering to disambiguate different types of
interactions with the same object types; (4) a novel metric to assess the
quality of 3D spatial learning of interaction.

##  2 Related Work

Text-to-Image Synthesis with Diffusion Models. Recent approaches in text-to-
image synthesis show great performance by leveraging emerging diffusion models
[65, 67, 50, 34, 66, 59, 68]. Diffusion models are a class of generative
models that “noises” the data from training distribution and learn to
“denoise” the perturbed images at arbitrary noise scale. Diffusion models are
likelihood-based generative models and are known to show high mode coverage
and generate high-fidelity images. One drawback is the slow inference time due
to multiple iterations of the denoising process, which can be mitigated by
recent approaches [66, 34, 40, 59]. Diffusion models enable text-conditional
image generation via utilizing CLIP [56] embeddings [49, 57, 59] or large-
language-model encodings [61]. In practice, classifier guidance [15] or
classifier-free guidance [26] is applied at inference steps further to enhance
the quality and text-coherency in trade-off with diversity.

Generative Models as Data Producer. Similar to our approaches, there have been
extensive approaches to leverage generative models as data generators. Prior
approaches utilize GANs [19, 55, 33] to synthesize data for tasks including
classification [2, 8, 43, 72, 75], 3D vision [21, 52, 64, 86], 2D segmentation
[44, 74, 87], dense visual alignment [54]; or diffusion models [27, 59] for
data augmentation [73] or as synthetic data for few-shot learning [25]. One
major challenge of such methods is that generative models output “free” and
“uncontrolled” images. Ali et al. [29] presents a method to generate multi-
view images for representation learning by applying transformations on the
latent vectors of the generative models, similar to our approach.

Learning 3D Spatial Arrangement between Human and Object. Prior work models 3D
human-object interaction for each specified interaction type and input.
Approaches include inferring hand-object interaction from 3D [5, 71, 88], 2.5D
such as heatmap [4, 6], or provided images [14, 16, 23, 35, 79]. These methods
only model hand-object relationships and cannot predict the full body. The
methods that model full-body interactions include generating/reconstructing 3D
humans from 3D scene constraints [22, 63, 84, 85], capturing interaction from
multi-view camera [3, 31, 69], or reconstructing 3D scene from human-scene
interaction type [80], or human-human interactions [18]. Also, there exist
approaches to model contacts [12, 28, 7, 38, 47, 58]. PHOSA [83] reconstructs
3D human and object from a single image; however, it requires manual labeling
for the human-object interaction region, which limits the generalizability of
the approach. CHORE [77] additionally models contact via distance between
human and object by leveraging implicit surface learning to fit parametric
SMPL [41] human and template object mesh. Unlike previous methods, our method
does not require any object templates or supervision.

##  3 Method

Figure 3: Method overview. Our method starts with generating prompts for
human-object relationships within a specific object category. These prompts
produce a multitude of images, incorporating HOI semantics. After applying
filtering strategies to eliminate outliers, the chosen images are aggregated
in canonical 3D space using a canonicalization approach. The resulting
distribution can be flexibly adapted to different human postures.

###  3.1 Overview

Our method extracts 3D spatial knowledge of human-object interactions (HOI) by
modeling object location as an occupancy probability distribution $\Phi_{o}$
when the category of the object $o$ is given. The spatial relationship during
HOI can vary even for the same object category due to the variation of human
state (e.g., pose, body height and body shapes) and types of interactions
(e.g., we can ride or hold a surfboard). Thus, we model the 3D spatial
probability distribution in pose-conditioned and type-conditioned manners:

$\Phi_{o}(\mathbf{x}|\theta,s)\in[0,1],$ (1)

where $\mathbf{x}\in\mathbb{R}^{3}$ is a 3D location of object occurrence in
the “pose-deformed” space, $\theta$ is the 3D state of a person in terms of
pose and shape variation, and $s$ is a type of human-object interaction among
possible discrete variations $\mathbf{S}$. We represent $\Phi_{o}$ in an
explicit voxel space with $48^{3}$ resolutions, where the probability shows
the likelihood that the specific location is occupied by the object $o$ during
interactions. The human state $\theta\in\mathbb{R}^{24\times 3}$ is
parameterized by SMPL [41] pose parameters for 24 joints to represent the
condition of 3D human pose variations. $\mathbf{S}$ represents plausible types
of interactions for an object $o$, which is used for clustering distributions
as described in Sec. 3.4. Specifically, $\mathbf{S}$ considers the semantics
described by (1) natural language prompts and (2) contacted parts between
humans and objects. For example, holding a surfboard and riding a surfboard
should be considered as different interactions, despite the same object
category, which is the objective to have $\mathbf{S}$ as the input for our
$\Phi_{o}$.

As a way to learn the 3D spatial distribution from inconsistent 2D image data,
our method aggregates the HOI cues in a canonical space:

$\Phi^{c}_{o}(\mathbf{x}^{c}|s)\in[0,1],$ (2)

where the $\mathbf{x}^{c}$ is a 3D location at the canonical space. The
canonical space is defined by the rest-pose of SMPL, as shown in Fig. 3, where
we put zero rotations for most joints except hips; $\pm\pi/6$ z-axis rotation
on the left and right hips, respectively; following the approach proposed by
SNARF [10] (we empirically find that it is advantageous to keep a sufficient
distance between legs). Note that this canonical distribution $\Phi^{c}_{o}$
is independent of the 3D human pose state $\theta$, in contrast to Eq. 1. Our
3D spatial reasoning in the canonical space is inspired by the recently
emerging approaches to building animatable 3D avatars [10, 11, 62, 30], where
cues from multiple 3D scans or views from different postures are aggregated in
the canonical space. Yet, our framework is much simpler without requiring
precise linear-blend skinning estimation since our target is reasoning the
approximate object locations w.r.t. human rather than high-fidelity 3D human
surface estimation. By reasoning in the canonical space, we can handle the
inconsistency and variation of synthesized multiview images in HOIs. To this
end, our framework provides a warping function to convert back and forth
between $\Phi^{c}_{o}$ and $\Phi_{o}$, and our learned 3D spatial distribution
can be applied to any 3D human postures to guess potential object locations as
shown in Fig. 7. As an interesting key idea for generating multi-view image
collections for 3D spatial relation learning, we use a SOTA diffusion model
[59] that can synthesize realistic images from text prompt inputs. We refer
readers to Fig. 3 for an overview of our method, and Supp. Mat. A.1$\sim$A.8
for more details on each component.

###  3.2 Synthesizing Images

Prompt Generation. We aim to produce various text prompts that describe the
diverse semantics of human-object interaction on a target object $o$, which a
text-conditioned diffusion model [59] can then take for image generation. In
particular, we want to obtain visual cues on various human-object interactions
with the target object $o$ with many viewpoints for 3D reasoning.

Towards an automatic process, we present a solution by leveraging the ChatGPT
[51], an instruction-tuned large language model, to produce a set of plausible
prompts automatically. Specifically, we give the input query sentence as shown
in Fig. 4, where we can simply replace the object category (e.g., skis as
shown in a blue box) for any target objects. As shown, ChatGPT produces
various related output prompts. We empirically observe that our solution with
ChatGPT is much more efficient with higher quality, compared to the possible
alternative way of directly generating prompt sentences via exhaustive
combinations of a set of subjects and verbs, which may produce awkward
expressions (e.g., “wear a bike”). Furthermore, to encourage similar scenes
from diverse viewpoints as much as possible, we also present a strategy to
control viewpoints of the synthesized images by augmenting view conditions on
each prompt produced from ChatGPT, such as “back view”, “side view”, and “far
view”. As shown in Fig. 4, we demonstrate this viewpoint augmentation on
prompts is advantageous in producing scenes from diverse views, while it is
only partially guaranteed that the outputs follow the instructed viewpoints
due to the imperfection of generative models. We generate 3 to 20 prompts per
category, where each is augmented with 22 different auxiliary prompts to
control viewpoints.

Synthesizing Text-Conditioned Images via Diffusion. Given a set of produced
prompts regarding a human-object interaction, we link these natural language
descriptions to visual entities by synthesizing images via an off-the-shelf
latent diffusion model [59], which has great power in high-fidelity image
generation with high mode coverage. Examples are shown in Fig. 4. Our strategy
enables us to produce an unbounded number of diverse scenes containing the
desired human object interaction with the target object category. Notably, the
same prompt can be used multiple times by simply changing the initial random
latent, resulting in different outputs. We create $5000\sim 90000$ images per
object category, evenly distributed for each given prompt within the category.

Figure 4: Viewpoint augmentation. Examples of prompts produced by ChatGPT and
images synthesized via diffusion with viewpoint augmentation.

Filtering. Despite the efficacy of using the diffusion model for an image
producer, the collected images are still in a wild status, making it
challenging to use them directly for 3D spatial HOI learning. Thus, we first
apply a series of filtering strategies to retain useful images only. Notably,
the unbounded nature of our data generation allows us to use tough thresholds
to filter out useless image samples aggressively. We use the following
criteria in determining “valid” image samples: (1) the image contains a single
human for efficient localization without ambiguity; (2) the image contains the
target object; (3) the human and the target object should be close enough with
sufficient overlapping in the bounding boxes (with a certain threshold); and
(4) the whole torso part of the human should be visible. We use an off-the-
shelf object detector [37, 76] for the bounding box detection and instance
segmentation. To check the torso’s existence, we use a 2D keypoint detector
[70, 82, 13] and check whether shoulders and hips are within the image
boundary and their detection confidences are above certain thresholds.

###  3.3 Pose Canonicalization

A large collection of images obtained from our approach can provide cues for
3D spatial HOI reasoning. As its first step, we estimate camera viewpoint by
using the estimated 3D human orientation as an anchor. Then, we aggregate 2D
object mask cues from each view to estimate 3D occupancy distribution, where
we apply pose canonicalization to handle pose variations among synthesized
images.

Viewpoint Estimation via 3D Human Pose Estimation. The orientation of the
human body in the scene can provide a clue to estimate the relative camera
viewpoint w.r.t the human. For this purpose, we use an off-the-shelf monocular
3D human pose estimator [60], which outputs the 3D global orientation of
humans along with 3D joint orientations in SMPL [41] parameterzation.
Specifically, given an image $\mathbf{I}$ and a human bounding box
$\mathbf{B}\in\mathbb{R}^{4}$, a 3D pose regressor
$\mathcal{F}_{\text{pose3d}}$ outputs:

$\\{\mathbf{\phi},\mathbf{\theta},\mathbf{\beta},\pi,\mathbf{j}\\}=\mathcal{F}_{\text{pose3d}}(\mathbf{I},\mathbf{B}),$
(3)

where $\mathbf{\phi}\in\mathbb{R}^{3}$ is the global orientation of the human
defined in the camera-centric coordinate,
$\mathbf{\theta}\in\mathbb{R}^{23\times 3}$ is 3D joint angles (excluding the
pelvis root) in angle axis, $\beta\in\mathbb{R}^{10}$ is shape parameters, and
$\pi\in\mathbb{R}^{3}$ is the weak-perspective camera parameters, and
$\mathbf{j}\in\mathbb{R}^{24\times 2}$ is the projected 2D joint locations on
image space. We then convert $\pi$ and $\phi$ into a perspective camera model
$\Pi$ in the person-centric coordinate system by minimizing the distance
between projected SMPL joints via $\pi$ and $\Pi$. To this point, all images
are “calibrated” in a common 3D space, where the SMPL humans are aligned and
centered at the origin (i.e., pelvis is set as origin), as depicted in Fig. 5.

(a) Calibrated cameras and humans.

(b) Synthesized images.

Figure 5: Human-anchored camera calibration. Predicted humans and calibrated
cameras for synthesized images.

3D Occupancy Estimation via Human Pose Canonicalization. We represent spatial
3D HOI reasoning via the occupancy probability field $\Phi^{c}_{o}$ as defined
in Eq. 2. For brevity, we may first consider a holistic distribution
$\Phi^{c}_{o}(\mathbf{x}^{c})\in[0,1]$, which does not consider semantics and
returns a marginalized distribution for the most probable HOI. Given the
virtually calibrated multi-view images we processed above, we compute
$\Phi^{c}_{o}$ via visual hull reconstruction using the 2D object segmentation
masks obtained in the filtering stage. However, as an issue, the 3D human
poses may not be consistent across views, as shown in Fig. 5, making it
challenging to apply visual hull reconstruction directly.

To handle this issue, we consider a canonical space defined via the rest pose
of the SMPL model and compute the conversion between this canonical space and
the pose-deformed space corresponding to each view. This process is inspired
by the deformable object reconstruction [48] or the animatable human
reconstruction pipelines [10, 11, 62, 30], where SMPL body posture provides
the guidance to convert between two spaces. Unlike the 3D human modeling
approaches that use MLPs for neural skinning [10, 11, 62], our goal is to
enable the 3D voting to the corresponding 3D canonical volume space by warping
into pose-deformed volume space and checking the means of 2D mask occupancies.
Thus, we take a simple strategy to find the mapping between the canonical and
posed spaces. Specifically, we define Linear Blending Skinning (LBS) weights
for 3D point $\mathbf{x}^{c}$ in the canonical space (we use $48^{3}$ voxel
grid) by averaging the ones from $\mathbf{k}$-nearest neighbor SMPL vertices
with inverse distance weights, similar to SelfRecon [30]. That is:

$\omega(\mathbf{x}^{c})={{\sum\limits_{i\in\mathbf{N}_{\mathbf{k}}(\mathbf{x}^{c})}{w_{i}~{}/~{}{\|\mathbf{x}^{c}-\mathbf{v}_{i}\|}}}\over{\sum\limits_{i\in\mathbf{N}_{\mathbf{k}}(\mathbf{x}^{c})}{1~{}/~{}{\|\mathbf{x}^{c}-\mathbf{v}_{i}\|}}}}$
(4)

where $\mathbf{N}_{\mathbf{k}}(\mathbf{x}^{c})$ is a set of
$\mathbf{k}$-nearest neighbor vertex indices on SMPL mesh,
$w_{i}\in\mathbb{R}^{24}$ and $\mathbf{v}_{i}\in\mathbb{R}^{3}$ are each
associated LBS skinning weights and location of $i$-th vertex in SMPL.
Intuitively, the 3D space is transformed by following the motion of the
closest SMPL vertices. Additionally, assuming the monotonic decrease of the
effect of bone transformation as $\mathbf{x}^{c}$ is far from the SMPL mesh
model, we encourage “zero deformations” for the far-away points by decreasing
the effect of LBS with skinning weight adjustments by simply applying weighted
sum between $\omega(\mathbf{x}^{c})$ and LBS skinning weights for pelvis
$\mathbf{e}_{1}\in\mathbb{R}^{24}$ which has no effect on deformation. Refer
to Supp. Mat. A.5 for more details.

Given computed LBS weights, the warping from the canonical space
$\mathbf{x}^{c}$ to a pose-deformed space $\mathbf{x}$ with SMPL pose
$\mathbf{\theta}$ can be performed by following the forward-kinematics
pipeline of SMPL skeletal hierarchy:

$\mathbf{x}=\mathcal{W}(\mathbf{x}^{c})=\sum_{j=1}^{n_{b}}\omega_{j}(\mathbf{x}^{c})\cdot\mathbf{B}_{j}(\theta_{j})\cdot\mathbf{x}^{c},$
(5)

where $\omega_{j}$ denotes the $j$-th LBS weights,
$\mathbf{B}_{j}(\theta_{j})\in\text{SE}(3)$ represent $j$-th bone’s global 3D
transformations.

Finally, the 3D occupancy aggregation can be performed by warping the 3D
canonical points into the deformed space of each view and checking the means
of the 2D object mask occupancies when projected:

$\Phi_{o}^{c}(\mathbf{x^{c}})={{\sum\limits_{k=1}^{|\mathbf{G}|}r_{k}\mathcal{M}_{k}(\Pi_{k}(\mathcal{W}(\mathbf{x^{c}})))}\over{\sum\limits_{k=1}^{|\mathbf{G}|}r_{k}\mathcal{I}_{k}(\Pi_{k}(\mathcal{W}(\mathbf{x^{c}})))}}$
(6)

where $\mathbf{G}$ is the set of generated images, $r_{k}$ is accumulation
score for image $k$ based on predicted camera distribution, $\mathcal{M}_{k}$,
$\mathcal{I}_{k}$ are each mask and image operator for $k$-th image which
returns $1$ if the provided 2D value lies within mask/image space else $0$,
and $\Pi_{k}$ is perspective projection for $k$-th image view.

Uniform View Sampling. Despite our viewpoint augmentation when prompting, the
resulting images still may have biases toward specific viewpoints, making 3D
reasoning difficult. Thus, we enforce uniform view sampling (similar to
importance sampling) by dividing the azimuth region into a fixed number of
bins and setting the accumulation score $r_{k}$ in Eq. 6 as the inverse of
camera numbers in the bin.

Inference for Posed Space. At inference, we can transform our occupancy
probability distribution $\Phi_{o}^{c}(\mathbf{x}^{c})$ defined in the
canonical space into the pose-deformed space $\Phi_{o}(\mathbf{x}|\theta)$ by
using backward skinning, unlike the forward skinning (refer to Eq. 5) during
training. Specifically, we set $\Phi_{o}(\mathbf{x}|\theta)$ by warping
$\mathbf{x}$ in pose-deformed space to $\mathbf{x}^{c}$ in canonical space and
retrieving the learned occupancy probability $\Phi_{o}^{c}(\mathbf{x}^{c})$.
To this end, we compute the LBS weights in the pose-deformed space (in
contrast to training) and apply inverse transformation $\mathcal{W}^{-1}$.

###  3.4 Selective Aggregation via Semantic Clustering

The canonical distribution $\Phi_{o}^{c}(\mathbf{x}^{c})$ from Eq. 6 does not
take into account the different semantics. However, the form of human-object
interaction can differ even within the same object category, requiring
corresponding variations of $\Phi_{o}^{c}(\mathbf{x}^{c})$, as defined in Eq.
2. To formulate this, we define the interaction type $s\in\mathbf{S}$ as a
pair of prompt $p$ and body part $\mathbf{a}$ (optional) in contact with the
object:

$\mathbf{S}=\\{(p,\mathbf{a})~{}|~{}p\in\mathbf{P},\mathbf{a}\in\mathbf{A}\\},$
(7)

where $\mathbf{P}$ is the set of entire prompts produced and $\mathbf{A}$ is
the set of body part segments, given as a part of SMPL mesh vertices.
Intuitively, a prompt represents a semantic (e.g., “playing with a ball”), and
the body segment can further specify the interaction type (e.g.,
foot$\xrightarrow{}$“kicking”).

We compute $\Phi^{c}_{o}(\mathbf{x}^{c}|s)$ for each interaction type
$s=(p,\mathbf{a})$ by aggregating a semantic cluster of images that depict
such interactions, selected from $\mathbf{G}$. Specifically, we utilize images
generated from the single prompt $p$. Body part $\mathbf{a}$ is used as a
proximity cue to retain relevant image samples, where we consider the image
irrelevant if the 3D rays from the object mask do not intersect with the
interaction region of $\mathbf{a}$. Our method of “selective aggregation”
enhances spatial HOI reasoning for multimodal scenarios.

##  4 Experiments

We provide both quantitative and qualitative comparisons to verify our method.
In Sec. 4.2, we present a new metric, namely Projective Average Precision
(PAP). We provide a brief explanation of PAP and detailed protocols in Supp.
Mat. B.2. In Sec. 4.3, we quantitatively compare our method with various
ablations to provide justifications for our design choices. We also compare
our method trained on synthesized images with the one trained on internet-
crawled images, and show that synthesized images are more suitable for
learning 3D HOI. In Sec. 4.4, we show qualitative results. We demonstrate the
quality of the learned HOI spatial distribution when various human pose is
given for a diverse set of object categories. We then explore the effects of
semantic types on the distribution by changing HOI prompts and body part
specifications. We also analyze the effects of canonicalization via a
comparison with an ablation. In Sec. 4.5, we exemplify an application of our
method to the downstream task: 3D Human-Object Reconstruction from a Single-
view Image. Refer to Supp. Mat. B.1$\sim$B.2 for more details on each part,
and Supp. Mat. C for extensive qualitative results.

###  4.1 Dataset

Generated Dataset. Since our method is a fully-autonomous and self-supervised
approach, no external dataset is required for training. We test our method for
19 object categories from COCO [39] (e.g., bicycle, chair) and 5 categories
from LVIS [20] (e.g., hat, sweater), where we generate $5000\sim 90000$ images
per category.

Image Search Dataset. We provide a quantitative comparison between the results
of our method using image search data and using synthesized images in Sec.
4.3. To this end, we use AutoCrawler [81] to collect images for category
motorcycle from the internet, using the same prompts used for preparing
generated dataset.

Extended COCO-EFT Dataset for Testing. To quantitatively evaluate our learned
HOI distributions, we use COCO dataset [39], which provides GT 2D object
masks. In particular, we use COCO-EFT (val) [32], where pseudo GTs for 3D
human poses are provided in SMPL [41] format. Among all samples in COCO-EFT,
we perform a filtering process to keep only the samples where human-object
interactions happen by retaining the samples with a single human and a single
object with sufficient overlaps between them. After the filtering process, we
compute perspective camera parameters similar to Sec. 3.3 from the weak-
perspective cameras provided in the dataset.

###  4.2 Projective Average Precision

As no standard metrics exist for this task, we define a new evaluation metric:
Projective Average Precision (PAP). Refer to Fig. 6 for an overview of the PAP
evaluation protocols.

Evaluation Protocols. We utilize the COCO-EFT [32] dataset with pseudo-GT 3D
human pose and object masks to compare the projection of estimated 3D
distribution with GT object mask. Specifically, we first deform the canonical
3D distribution into pose-deformed space and apply threshold for
discretization. We project discretized occupancy values to 2D using the
perspective camera and compute pixel-wise precision and recall between ground-
truth object mask and projected occupancy. By setting multiple threshold
values, we enable quantifying the validity of distribution regarding the
intra-class object variation in terms of geometry. Note that we compute
precision and recall for evaluation as our method aims to infer the object
distribution, not to reconstruct the exact geometry of the target. The
calculated precision and recall values are then employed to determine the
average precision (AP) using two distinct methods (vanilla, strict). Averaging
the AP values across all images within the category yields the Projective
Average Precision (PAP). Further insights and comprehensive details can be
found in Supp. Mat. B.2.

(a) Overview.

(b) Human-occlusion aware mask.

Figure 6: Overview of evaluation protocols for PAP.

Human-Occlusion-Aware PAP. We observe that human masks frequently overlap with
objects, potentially causing inaccuracies in evaluating rendered 3D occupancy
against partially-removed ground-truth masks as shown in Fig. 6(b). To address
this issue, we exclude precision and recall calculations within the 2D region
of the human projection, considering the likelihood of occlusions.

###  4.3 Quantitative Evaluation

Ablation Studies. To justify our design choice, we quantitatively compare our
method with various ablations. We use mPAP (average of PAP for all categories)
metric for comparison, and we report the results in Tab. 2. We list the
baselines used for comparison and discussion for each result below. Unless
specified, all baselines follow the same implementation details provided in
Sec. 3.

  * •

w/o Keypoint Filtering: We deactivate keypoint filtering and conduct
experiments using an equivalent number of images as in the case with keypoint
filtering. We randomly sample from a combination of unfiltered and filtered
images. We report a drop in performance when keypoint filters are not applied,
which is attributed to erroneous 3D human predictions.

  * •

w/o Canonicalization: We compute the HOI distribution without pose
canonicalization. Notably, we find that removing canonicalization damages the
metrics by a large margin. This is an expected result, as removing
canonicalization will disperse the 2D cues and limit the precise aggregation
due to pose variation.

  * •

10%, 1% Trained: To see the effect of the number of generated data used for
training (i.e., training time), we reduce the images by randomly selecting 10%
or 1% of filtered images per prompt and run the 3D aggregation. The results
imply that using more synthesized images for aggregation is beneficial for the
quality of learned distribution, and the potential of our method for
scalability.

Table 1: Image Search vs. Synthesized Images: Statistics.

Method | # Images | # Images after Filtering | Rejection-rate (%)$\downarrow$ | Camera Entropy (bits)$\uparrow$
---|---|---|---|---
Image Search${}_{\text{1k}}$ | 1201 | 391 | 67.44 | 10.04
Image Search${}_{\text{10k}}$ | 9408 | 2526 | 73.15 | 12.25
Synthetic${}_{\text{1k}}$ (Ours) | 1201 | 456 | 62.03 | 12.21
Synthetic${}_{\text{10k}}$ (Ours) | 9408 | 3626 | 61.46 | 14.63

Table 2: Quantitative Evaluation Results. (up) Results for ablation studies on
COCO-EFT categories. (down) Results for quantitative comparison between image
search and synthesize images on single category motorcycle.

Method | Vanilla | Human-Occlusion Aware
---|---|---
mPAP$\uparrow$ |  mPAP${}_{\text{strict}}$$\uparrow$ |  mPAP$\uparrow$ |  mPAP${}_{\text{strict}}$$\uparrow$
Ours${}_{\text{w/o Keypoint Filtering}}$ | 18.31 | 15.90 | 19.97 | 16.52
Ours${}_{\text{w/o Canonicalization}}$ | 17.52 | 15.12 | 19.56 | 15.85
Ours${}_{\text{1\%}}$ | 14.98 | 13.81 | 15.21 | 13.73
Ours${}_{\text{10\%}}$ | 17.90 | 16.13 | 18.86 | 15.82
Ours${}_{\text{full}}$ | 19.86 | 17.18 | 20.28 | 16.80
Image Search${}_{\text{1k}}$ | 53.64 | 52.15 | 60.08 | 56.86
Image Search${}_{\text{10k}}$ | 55.97 | 54.59 | 63.62 | 61.11
Synthetic${}_{\text{1k}}$ (Ours) | 56.33 | 55.11 | 63.91 | 61.85
Synthetic${}_{\text{10k}}$ (Ours) | 57.14 | 55.19 | 64.82 | 62.11

Image Search vs. Synthesized Images. To validate the use of synthesized
images, we compare our method’s output for motorcycle category across diverse
datasets. We opt for motorcycle due to the abundance of accessible and common
images, which ensures a fair and rigorous comparison. We assess four settings,
altering (1) data source (image search or synthesis) and (2) image count (1k
or 10k). To ensure fairness, we randomly select images from the generated
dataset, equivalent to the size of image search dataset. We follow the same
procedures described in Sec. 3 for filtering and aggregation. We report the
dataset’s retrieval statistics including post-filtering rejection rate, and
camera distribution entropy for filtered images in Tab. 1. For the computation
of camera distribution entropy, we fit Gaussian normals with a standard
deviation of $\sigma=0.01$ to each camera’s location on the unit sphere, using
geodesic metrics.

We observe a higher and increasing rejection rate in the image search dataset
compared to the stable, relatively low rate in the generated dataset. We
attribute this to the word-occurrence-based search algorithm, which may
retrieve images without HOI (e.g., commercials). Thus, we argue that the
generated dataset offers enhanced scalability for HOI learning. Furthermore,
the superior camera distribution entropy in synthesized images implies a
richer diversity of camera poses than the image search dataset. Such diversity
provides more informative content for 3D aggregation in the generated dataset.
While our method addresses imbalanced camera distribution during aggregation
through uniform view sampling (Sec. 3.3), results in Tab. 2 consistently
demonstrate the synthetic dataset’s superior performance across metrics. This
reaffirms the effectiveness of our data generation approach for HOI learning,
even after minimizing information differences from camera imbalance.

Figure 7: Qualitative Results for Various Categories. Figure 8: Effects of
semantic type condition on object distribution. As semantic types vary, the
canonical and pose-deformed object distribution changes accordingly.

###  4.4 Qualitative Evaluation

Various Categories. We demonstrate that our method can learn spatial human-
object relationships for a diverse set of object categories in Fig. 7. We use
SMPL pose and reference image sampled from; the COCO-EFT [32] dataset for COCO
[39] categories and generated dataset for LVIS [20] categories; to deform the
distribution in canonical space to pose-deformed space. We visualize the
semantic cluster that best matches the reference image. Refer to Supp. Mat. C
for extensive results.

Effects of Semantic Type Condition. We find that our set of learned
distributions conveys the human-object interaction effectively. We report the
examples of object distributions (category: surfboard) in canonical and pose-
deformed space for different semantic types $s=(p,\mathbf{a})$ in Fig. 8.
Notably, the object distribution of the surfboard for the given prompt “A
person carries a surfboard” shows the mode near the human torso and arms:
which is a plausible location for holding a surfboard when carrying around. We
also report results when semantic body part $\mathbf{a}$ vary for the same
semantic prompt description. In Fig. 9, we observe that body parts act
important as a condition when the provided semantic prompt contains little or
less information on HOI; hence, ambiguous. For example, the semantics for “A
man playing with a sports ball” can be different in terms of sports
categories. In Fig. 10, we can see the effects of prompt change when body
parts are the same. While the semantic type with prompt “The cyclist pedaled
the bicycle” return a shape similar to the original bicycle as expected, we
observe the donut-shaped distribution as in Fig. 10 when we change the prompt
into “The commuter standing next to a bicycle”. We argue that such
distribution conveys accurate semantics of spatial arrangements as humans can
be oriented in arbitrary directions even if standing next to a bicycle, where
we can assume such arrangement leads to rotationally-invariant distribution
(e.g., donut-shape).

Effects of Canonicalization. We ablate the effects of canonicalization and
report results in Fig. 11. From results, we can observe that precision is
enhanced and variance is reduced when pose-canonicalization is applied, as the
aggregation of 2D cues becomes more coherent compared to direct rays due to
pose-variance.

Figure 9: Effects of body-part specification. For category sports ball and
prompt “A man playing with a sports ball”, body parts play a significant role
as a condition for further specifying the HOI type. Figure 10: Effects of
prompt specification. For category bicycle, object distribution changes
significantly as prompts representing the HOI type differs. Figure 11: Effects
of canonicalization. Removing canonicalization from our approach leads to
scattered aggregation of 2D mask occupancies, resulting in blurry outcomes and
diminished precision.

###  4.5 Application

Our method returns generalizable 3D human-object spatial knowledge represented
as probability occupancy distribution of the object, which can be easily
applied to many downstream tasks. This section provides an example case of
application to the following downstream task: 3D Human-Object Reconstruction
from a Single-view Image. Given a single-view image, we apply off-the-shelf
monocular 3D human pose estimator [60] to extract SMPL pose and off-the-shelf
object detector [37] to find the category of interacting object, where we find
the object of interaction by computing bounding box overlap between human and
object and picking the object with maximum value. Next, we extract 3D object
occupancy distribution by simply inputting the category of the found object as
a keyword to our method. Finally, we apply the marching cube algorithm [42] to
the object distribution to convert 3D object occupancy into a 3D object mesh.
Our method is applicable to in-the-wild images, fully automatic and template
agnostic, in contrast to previous works that use template mesh to represent 3D
objects [83, 77]. See Fig. 12 for results.

Figure 12: 3D Human-Object Reconstruction from a Single-view Image. Using
learned distribution as a prior, we can reconstruct 3D human and object from
single image without any template mesh.

##  5 Discussion

Our method introduces a novel approach, enabling machines to probabilistically
learn 3D human-object spatial relationships based on pose and HOI types.
Unlike prior methods, ours extracts this information through self-supervision,
eliminating the need for laborious annotations. We present multiple
strategies, including leveraging a generative model for prompt/image synthesis
to enhance image control, and a canonicalization-based framework for computing
3D spatial HOI fields from synthesized 2D images. Furthermore, we introduce a
novel metric that demonstrates the high quality of our learned HOI
distribution. As the work presents a method to extract an intermediate
representation, our approach offers numerous potential downstream
applications; however, while possessing such potentials, the work is new and
thus has limitations, such as low granularity and inaccurate modeling of
distributions for small objects. We extensively discuss these limitations and
potential future avenues in Supp. Mat. D.

†† Acknowledgements. This work was supported by SNU-Naver Hyperscale AI
Center, New Faculty Startup Fund from Seoul National University, SNU Creative-
Pioneering Researchers Program, NRF grant funded by the Korea government
(MSIT) (No. 2022R1A2C2092724), and IITP grant funded by the Korea
government(MSIT) (No.2022-0-00156 and No.2021-0-01343). H. Joo is the
corresponding author.

## References

  * Armandpour et al. [2023] M. Armandpour, H. Zheng, A. Sadeghian, A. Sadeghian, and M. Zhou.  Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond.  _arXiv preprint arXiv:2304.04968_ , 2023.
  * Besnier et al. [2020] V. Besnier, H. Jain, A. Bursuc, M. Cord, and P. Pérez.  This dataset does not exist: training models from generated images.  In _ICASSP_ , 2020.
  * Bhatnagar et al. [2022] B. L. Bhatnagar, X. Xie, I. A. Petrov, C. Sminchisescu, C. Theobalt, and G. Pons-Moll.  Behave: Dataset and method for tracking human object interactions.  In _CVPR_ , 2022.
  * Brahmbhatt et al. [2019a] S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays.  Contactdb: Analyzing and predicting grasp contact via thermal imaging.  In _CVPR_ , 2019a.
  * Brahmbhatt et al. [2019b] S. Brahmbhatt, A. Handa, J. Hays, and D. Fox.  Contactgrasp: Functional multi-finger grasp synthesis from contact.  In _IROS_ , 2019b.
  * Brahmbhatt et al. [2020] S. Brahmbhatt, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays.  Contactpose: A dataset of grasps with object contact and hand pose.  In _ECCV_ , 2020.
  * Cao et al. [2020] Z. Cao, H. Gao, K. Mangalam, Q.-Z. Cai, M. Vo, and J. Malik.  Long-term human motion prediction with scene context.  In _ECCV_ , 2020.
  * Chai et al. [2021] L. Chai, J.-Y. Zhu, E. Shechtman, P. Isola, and R. Zhang.  Ensembling with deep generative views.  In _CVPR_ , 2021.
  * Chen et al. [2013] X. Chen, A. Shrivastava, and A. Gupta.  Neil: Extracting visual knowledge from web data.  In _ICCV_ , 2013.
  * Chen et al. [2021] X. Chen, Y. Zheng, M. J. Black, O. Hilliges, and A. Geiger.  Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes.  In _ICCV_ , 2021.
  * Chen et al. [2022] X. Chen, T. Jiang, J. Song, J. Yang, M. J. Black, A. Geiger, and O. Hilliges.  gdna: Towards generative detailed neural avatars.  In _CVPR_ , 2022.
  * Chen et al. [2019] Y. Chen, S. Huang, T. Yuan, S. Qi, Y. Zhu, and S.-C. Zhu.  Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense.  In _ICCV_ , 2019.
  * Contributors [2020] M. Contributors.  Openmmlab pose estimation toolbox and benchmark.  https://github.com/open-mmlab/mmpose, 2020.
  * Corona et al. [2020] E. Corona, A. Pumarola, G. Alenya, F. Moreno-Noguer, and G. Rogez.  Ganhand: Predicting human grasp affordances in multi-object scenes.  In _CVPR_ , 2020.
  * Dhariwal and Nichol [2021] P. Dhariwal and A. Nichol.  Diffusion models beat gans on image synthesis.  In _NeurIPS_ , 2021.
  * Ehsani et al. [2020] K. Ehsani, S. Tulsiani, S. Gupta, A. Farhadi, and A. Gupta.  Use the force, luke! learning to predict physical forces by simulating effects.  In _CVPR_ , 2020.
  * [17] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2008 (VOC2008) Results.  http://www.pascal-network.org/challenges/VOC/voc2008/workshop/index.html.
  * Fieraru et al. [2020] M. Fieraru, M. Zanfir, E. Oneata, A.-I. Popa, V. Olaru, and C. Sminchisescu.  Three-dimensional reconstruction of human interactions.  In _CVPR_ , 2020.
  * Goodfellow et al. [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In _NeurIPS_ , 2014.
  * Gupta et al. [2019] A. Gupta, P. Dollar, and R. Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In _CVPR_ , 2019.
  * Hao et al. [2021] Z. Hao, A. Mallya, S. Belongie, and M.-Y. Liu.  Gancraft: Unsupervised 3d neural rendering of minecraft worlds.  In _ICCV_ , 2021.
  * Hassan et al. [2021] M. Hassan, P. Ghosh, J. Tesch, D. Tzionas, and M. J. Black.  Populating 3d scenes by learning human-scene interaction.  In _CVPR_ , 2021.
  * Hasson et al. [2019] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, and C. Schmid.  Learning joint reconstruction of hands and manipulated objects.  In _CVPR_ , 2019.
  * He et al. [2017] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In _ICCV_ , 2017.
  * He et al. [2022] R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi.  Is synthetic data from generative models ready for image recognition?  _arXiv preprint arXiv:2210.07574_ , 2022.
  * Ho and Salimans [2022] J. Ho and T. Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Huang et al. [2022] C.-H. P. Huang, H. Yi, M. Höschle, M. Safroshkin, T. Alexiadis, S. Polikovsky, D. Scharstein, and M. J. Black.  Capturing and inferring dense full-body human-scene contact.  In _CVPR_ , 2022.
  * Jahanian et al. [2021] A. Jahanian, X. Puig, Y. Tian, and P. Isola.  Generative models as a data source for multiview representation learning.  _arXiv preprint arXiv:2106.05258_ , 2021.
  * Jiang et al. [2022a] B. Jiang, Y. Hong, H. Bao, and J. Zhang.  Selfrecon: Self reconstruction your digital avatar from monocular video.  In _CVPR_ , 2022a.
  * Jiang et al. [2022b] Y. Jiang, S. Jiang, G. Sun, Z. Su, K. Guo, M. Wu, J. Yu, and L. Xu.  Neuralhofusion: Neural volumetric rendering under human-object interactions.  In _CVPR_ , 2022b.
  * Joo et al. [2020] H. Joo, N. Neverova, and A. Vedaldi.  Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation.  In _3DV_ , 2020.
  * Karras et al. [2019] T. Karras, S. Laine, and T. Aila.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2019.
  * Karras et al. [2022] T. Karras, M. Aittala, T. Aila, and S. Laine.  Elucidating the design space of diffusion-based generative models.  In _NeurIPS_ , 2022.
  * Karunratanakul et al. [2020] K. Karunratanakul, J. Yang, Y. Zhang, M. J. Black, K. Muandet, and S. Tang.  Grasping field: Learning implicit representations for human grasps.  In _3DV_ , 2020.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirillov et al. [2020] A. Kirillov, Y. Wu, K. He, and R. Girshick.  Pointrend: Image segmentation as rendering.  In _CVPR_ , 2020.
  * Li et al. [2019] Z. Li, J. Sedlar, J. Carpentier, I. Laptev, N. Mansard, and J. Sivic.  Estimating 3d motion and forces of person-object interactions from monocular video.  In _CVPR_ , 2019.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. [2022] L. Liu, Y. Ren, Z. Lin, and Z. Zhao.  Pseudo numerical methods for diffusion models on manifolds.  _arXiv preprint arXiv:2202.09778_ , 2022.
  * Loper et al. [2015] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black.  Smpl: A skinned multi-person linear model.  _ACM TOG_ , 34(6):1–16, 2015.
  * Lorensen and Cline [1998] W. E. Lorensen and H. E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In _Seminal graphics: pioneering efforts that shaped the field_. 1998\.
  * Mao et al. [2021] C. Mao, A. Cha, A. Gupta, H. Wang, J. Yang, and C. Vondrick.  Generative interventions for causal learning.  In _CVPR_ , 2021.
  * Melas-Kyriazi et al. [2021] L. Melas-Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi.  Finding an unsupervised image segmenter in each of your deep generative models.  _arXiv preprint arXiv:2105.08127_ , 2021.
  * [45] Meshcapade Contributors.  Meshcapade wiki.  https://meshcapade.wiki/SMPL.
  * Mildenhall et al. [2020] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Monszpart et al. [2019] A. Monszpart, P. Guerrero, D. Ceylan, E. Yumer, and N. J. Mitra.  imapper: interaction-guided scene mapping from monocular videos.  _ACM TOG_ , 38(4):1–15, 2019.
  * Newcombe et al. [2015] R. A. Newcombe, D. Fox, and S. M. Seitz.  Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time.  In _CVPR_ , 2015.
  * Nichol et al. [2021] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Nichol and Dhariwal [2021] A. Q. Nichol and P. Dhariwal.  Improved denoising diffusion probabilistic models.  In _ICML_ , 2021.
  * OpenAI [2023] OpenAI.  Chatgpt: Optimizing language models for dialogue.  https://openai.com/blog/chatgpt/, 2023.
  * Pan et al. [2020] X. Pan, B. Dai, Z. Liu, C. C. Loy, and P. Luo.  Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans.  _arXiv preprint arXiv:2011.00844_ , 2020.
  * Pavlakos et al. [2019] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _CVPR_ , 2019.
  * Peebles et al. [2022] W. Peebles, J.-Y. Zhu, R. Zhang, A. Torralba, A. A. Efros, and E. Shechtman.  Gan-supervised dense visual alignment.  In _CVPR_ , 2022.
  * Radford et al. [2015] A. Radford, L. Metz, and S. Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_ , 2015.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ramesh et al. [2022] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rempe et al. [2020] D. Rempe, L. J. Guibas, A. Hertzmann, B. Russell, R. Villegas, and J. Yang.  Contact and human dynamics from monocular video.  In _ECCV_ , 2020.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Rong et al. [2021] Y. Rong, T. Shiratori, and H. Joo.  Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration.  In _ICCV Workshops_ , 2021.
  * Saharia et al. [2022] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Saito et al. [2021] S. Saito, J. Yang, Q. Ma, and M. J. Black.  Scanimate: Weakly supervised learning of skinned clothed avatar networks.  In _CVPR_ , 2021.
  * Savva et al. [2016] M. Savva, A. X. Chang, P. Hanrahan, M. Fisher, and M. Nießner.  Pigraphs: learning interaction snapshots from observations.  _ACM TOG_ , 35(4):1–12, 2016.
  * Shi et al. [2021] Y. Shi, D. Aggarwal, and A. K. Jain.  Lifting 2d stylegan for 3d-aware face generation.  In _CVPR_ , 2021.
  * Sohl-Dickstein et al. [2015] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _ICML_ , 2015.
  * Song et al. [2020a] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song and Ermon [2019] Y. Song and S. Ermon.  Generative modeling by estimating gradients of the data distribution.  In _NeurIPS_ , 2019.
  * Song et al. [2020b] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Sun et al. [2021] G. Sun, X. Chen, Y. Chen, A. Pang, P. Lin, Y. Jiang, L. Xu, J. Yu, and J. Wang.  Neural free-viewpoint performance rendering under complex human-object interactions.  In _ACMM_ , 2021.
  * Sun et al. [2019] K. Sun, B. Xiao, D. Liu, and J. Wang.  Deep high-resolution representation learning for human pose estimation.  In _CVPR_ , 2019.
  * Taheri et al. [2020] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas.  Grab: A dataset of whole-body human grasping of objects.  In _ECCV_ , 2020.
  * Tanaka and Aranha [2019] F. H. K. d. S. Tanaka and C. Aranha.  Data augmentation using gans.  _arXiv preprint arXiv:1904.09135_ , 2019.
  * Trabucco et al. [2023] B. Trabucco, K. Doherty, M. Gurinas, and R. Salakhutdinov.  Effective data augmentation with diffusion models.  _arXiv preprint arXiv:2302.07944_ , 2023.
  * Tritrong et al. [2021] N. Tritrong, P. Rewatbowornwong, and S. Suwajanakorn.  Repurposing gans for one-shot semantic part segmentation.  In _CVPR_ , 2021.
  * Wu et al. [2018] E. Wu, K. Wu, D. Cox, and W. Lotter.  Conditional infilling gans for data augmentation in mammogram classification.  In _MICCAI Workshop on Breast Image Analysis_ , 2018.
  * Wu et al. [2019] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Xie et al. [2022] X. Xie, B. L. Bhatnagar, and G. Pons-Moll.  Chore: Contact, human and object reconstruction from a single rgb image.  In _ECCV_ , 2022.
  * Xu et al. [2023] J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In _CVPR_ , 2023.
  * Yang et al. [2021] L. Yang, X. Zhan, K. Li, W. Xu, J. Li, and C. Lu.  Cpf: Learning a contact potential field to model the hand-object interaction.  In _ICCV_ , 2021.
  * Yi et al. [2022] H. Yi, C.-H. P. Huang, D. Tzionas, M. Kocabas, M. Hassan, S. Tang, J. Thies, and M. J. Black.  Human-aware object placement for visual environment reconstruction.  In _CVPR_ , 2022.
  * Yoongi [2023] K. Yoongi.  Autocrawler.  https://github.com/YoongiKim/AutoCrawler, 2023.
  * Zhang et al. [2020a] F. Zhang, X. Zhu, H. Dai, M. Ye, and C. Zhu.  Distribution-aware coordinate representation for human pose estimation.  In _CVPR_ , 2020a.
  * Zhang et al. [2020b] J. Y. Zhang, S. Pepose, H. Joo, D. Ramanan, J. Malik, and A. Kanazawa.  Perceiving 3d human-object spatial arrangements from a single image in the wild.  In _ECCV_ , 2020b.
  * Zhang et al. [2020c] S. Zhang, Y. Zhang, Q. Ma, M. J. Black, and S. Tang.  Place: Proximity learning of articulation and contact in 3d environments.  In _3DV_ , 2020c.
  * Zhang et al. [2022] X. Zhang, B. L. Bhatnagar, S. Starke, V. Guzov, and G. Pons-Moll.  Couch: Towards controllable human-chair interactions.  In _ECCV_ , 2022.
  * Zhang et al. [2020d] Y. Zhang, W. Chen, H. Ling, J. Gao, Y. Zhang, A. Torralba, and S. Fidler.  Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering.  _arXiv preprint arXiv:2010.09125_ , 2020d.
  * Zhang et al. [2021] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler.  Datasetgan: Efficient labeled data factory with minimal human effort.  In _CVPR_ , 2021.
  * Zhou et al. [2022] K. Zhou, B. L. Bhatnagar, J. E. Lenssen, and G. Pons-Moll.  Toch: Spatio-temporal object correspondence to hand for motion refinement.  In _ECCV_ , 2022.

CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations
from Unbounded Synthesized Images
(Supplementary Material)

##  Appendix A Additional Details for Method

In this section, we provide further details on the formulations and
implementations regarding the method section (Sec. 3) of our main paper.

###  A.1 Prompt Generation

We generate 3 to 20 different semantic HOI prompts via chatGPT [51] using the
following query template, where {m} is replaced with an integer between
3$\sim$20 and {category} is replaced with one of our input category keywords:

> “Generate at most $\\{\textbf{m}\\}$ simple subject-verb-object prompt where
> subject’s $\\{$category is person/word is “A person”$\\}$ and object’s
> category is $\\{\textbf{category}\\}$. You should use diverse and general
> word but no pronoun for subject. Generated prompt must align with common
> sense. Verb must depict physical interaction between subject and object.
> Simple verb preferred.”

The categories considered in this work are listed in Tab. 3. After generating
HOI prompts, we augment each prompt with $m_{\text{aug}}=22$ different
viewpoint augmentations, including:

> (no augmentation) / side view / back view / front view / top view / bottom
> view / realistic photo / seen from side / seen from back / seen from various
> views / seen from close view / seen from far away / scenic view / full body
> photo / hand photo / face photo / photo taken close to hand / photo taken
> close to face / selfie / close-up photo / hand only / face only.

Table 3: Statistics for generated dataset for all categories.

Category | # Prompts | # Images | # Images after Filtering | Rejection-rate (%)
---|---|---|---|---
Motorcycle | 12 | 47520 | 18228 | 61.64
Bench | 12 | 31680 | 15193 | 52.04
Backpack | 20 | 42240 | 7530 | 82.17
Handbag | 4 | 23760 | 1374 | 94.22
Tie | 10 | 31680 | 3358 | 89.40
Frisbee | 3 | 21780 | 5502 | 74.74
Skis | 12 | 34848 | 14547 | 58.26
Snowboard | 4 | 7920 | 3251 | 58.95
Sports ball | 8 | 19008 | 1611 | 91.52
Baseball glove | 5 | 11880 | 260 | 97.81
Skateboard | 16 | 50688 | 12069 | 76.19
Surfboard | 16 | 73920 | 25352 | 65.70
Tennis racket | 15 | 47520 | 19827 | 58.28
Cell phone | 20 | 84480 | 2344 | 97.23
Bicycle | 18 | 42768 | 11361 | 73.44
Umbrella | 7 | 16632 | 2720 | 83.65
Chair | 17 | 47124 | 10185 | 78.39
Bed | 5 | 11880 | 1856 | 84.38
Laptop | 17 | 53856 | 1890 | 96.49
Hat∗ | 10 | 31680 | 173 | 99.45
Sweater∗ | 3 | 5940 | 689 | 88.40
Sunglasses∗ | 3 | 5940 | 78 | 98.69
Soccer ball∗ | 5 | 15840 | 1976 | 87.53
Scarf∗ | 5 | 15840 | 461 | 97.09
* denotes LVIS [20] categories |

###  A.2 Synthesizing Text-Conditioned Images via Diffusion

We use publicly available Stable-Diffusion [59] model (version: 1.4) for text-
to-image synthesis. We generate $512\times 512$ images upsampled from the
generated 8$\times$ downsampled latents. We sample images with $50$ denoising
steps using Pseudo-Numerical-Sampling methods [40] with classifier-free
guidance scale of $7.5$.

###  A.3 Filtering

We apply a cascaded framework to remove unrelated images. We use PointRend
[37] for bounding-box detection and instance segmentation of COCO [39]
categories. For LVIS [20] categories (e.g., hat, sunglasses, sweater), we use
publicly available pretrained Mask-RCNN [24] model from detectron2 [76] where
we set the segmentation threshold as $0.8$. We post-process the predicted
instances by removing duplicated bounding boxes on the same target object.
Specifically, we remove the lower-confidence instance if two bounding boxes of
the same category overlap with the _intersection over smaller bounding box_
value bigger than $0.8$, with exceptions of bounding boxes with confidence
over $0.98$. We also filter the images with multiple or none human/object
(target category) instances. We then reject images when the _intersection over
object bounding box_ value between the human box and object box is smaller
than $0.1$, assuming there is no interaction between them. For keypoint-
filtering, we represent the keypoints in COCO [39] format. We use a top-down
approach using publicly available pretrained HRNet [70]+Darkpose [82]
(provided by MMPose [13]) with a confidence threshold of $0.7$ for keypoint
prediction. We exclude the image if no shoulder joints (i.e., left-shoulder,
right-shoulder) or no hip joints (i.e., left-hip, right-hip) exist. Finally,
we reject the images with a very small human bounding box that returns no
prediction from 3D human pose estimator [60].

###  A.4 Viewpoint Estimation via 3D Human Pose Estimation

While the weak perspective is sufficient for projection, we employ a
perspective camera to consider the distance between the camera and the human
subject. To convert the weak perspective camera model $\mathbf{\pi}$ and
camera-centric orientation $\mathbf{\phi}$ (refer to Eq. 3) into a perspective
camera model $\Pi$ in the person-centric coordinate system (i.e., the origin
is defined at the pelvis), we apply optimization to compute $\Pi$ by aligning
3D joint projections and $\mathbf{j}$:

$\Pi^{*}=\operatorname*{arg\,min}_{\Pi}\sum_{i}||\Pi(\mathbf{J}^{0}_{i})-\mathbf{j}_{i}||^{2},$
(8)

where $i$ is the joint index and $\mathbf{J}^{0}$ are the 3D joints of the
SMPL model in person-centric coordinate, which is simply obtained by putting
zero orientation for $\mathbf{\phi}$ while keeping other parameters
$\mathbf{\theta}$ and $\mathbf{\beta}$. We include the joints $\mathbf{j}_{i}$
outside the image range for optimization described in Eq. 8. We optimize the
joint-reprojection loss using Adam [36] with a learning rate of $0.01$ for
$2400$ iterations. We early-terminate if the joint-reprojection loss is below
0.7 in pixel scale. We initialize camera intrinsic parameters with field-of-
view $46.4^{\circ}$ and camera extrinsic parameters with the rotation matrix
achieved by Rodrigues formula using $\phi$ and the translation vector by
$[t_{x},t_{y},2f/s]$ where $t_{x},t_{y},s$ are each $x$,$y$-direction
translations, and scale from weak-perspective camera $\pi$.

Figure 13: Visualization of LBS weights.

###  A.5 3D Occupancy Estimation via Human Pose Canonicalization

We define LBS weights for arbitrary point $\mathbf{x}^{c}$ in canonical space
following Eq. 4 with $\mathbf{k}=30$. We encourage “zero motions” by mixing
computed LBS weights with standard basis vector for 1st dimension
$\mathbf{e}_{1}\in\mathbb{R}^{24}$:

$\omega_{\text{deweight}}(\mathbf{x}^{c})=(1-\alpha)\mathbf{e}_{1}+\alpha\omega(\mathbf{x}^{c})$
(9) $\alpha=|{\max({{\tau-d_{\text{min}}}\over{\tau+d_{\text{min}}}},0)}|^{s}$
(10)

where $\tau$ and $s$ are each bandwidth and smoothing hyperparameters we
choose as $\tau=0.8,s=0.25$, and $d_{\text{min}}$ is the distance between
$\mathbf{x}^{c}$ and nearest SMPL mesh vertex. Note that the 1st element of
LBS weight is related to the pelvis joint, which is fixed to the origin and
aligned in a person-centric coordinate system as we set $\phi=0$; hence, the
transformation matrix is identity in $\text{SE}(3)$. Finally, we apply Laplace
smoothing over the entire grid 30 times to smooth LBS weights, similar to
SelfRecon [30]. See Fig. 13 for visualization of extended LBS weights.

###  A.6 Uniform View Sampling

Before performing aggregation, we first check the camera distribution and
assign the accumulation score $r_{k}$ for each image. Specifically, we divide
the azimuth region $[0,2\pi)$ into 12 equispaced bins (each $\pi\over 6$ long)
and count the number of cameras in each bin. If the camera associated with the
image falls into a specific bin, $r_{k}$ is set as the inverse of the camera
numbers in that bin. After assigning $r_{k}$ for all images, we then perform
the aggregation following Eq. 6 or Eq. 13.

###  A.7 Inference for Posed Space

At inference, we first calculate LBS weights that transform the voxels from
pose-deformed space to canonical space in the same way of Eq. 4 and Eq. 9,
where in this case $\mathbf{x}^{c}$ is replaced with $\mathbf{x}$ and
$\mathbf{v}_{i}$ corresponds to location of $i$-th vertex in SMPL in pose-
deformed space. Denoting the $j$-th LBS weights in pose-deformed space as
$\omega^{\text{inv}}_{j}(\mathbf{x};\theta)$, we inversely deform the voxels
in pose-deformed space as:

$\mathbf{x}^{c}=\mathcal{W}^{-1}(\mathbf{x})=\sum\limits_{j=1}^{n_{b}}\omega_{j}^{\text{inv}}(\mathbf{x};\theta)\cdot\mathbf{B}_{j}(\theta_{j})^{-1}\cdot\mathbf{x}$
(11)

We set $\Phi_{o}(\mathbf{x}|\theta)$ as the learned occupancy
$\Phi_{o}^{c}(\mathbf{x}^{c})$ in canonical distribution to infer pose-
deformed distribution.

Table 4: Definition of Body Parts. Each body part is defined by merging
multiple SMPL body segmentation maps from Meshcapade [45].

Body Part | SMPL Body Segmentation Labels
---|---
rightHand | rightHand, rightHandIndex1
leftHand | leftHand, leftHandIndex1
rightArm | rightArm, rightForeArm
leftArm | leftArm, leftForeArm
rightLowerLeg | rightLeg
leftLowerLeg | leftLeg
rightUpperLeg | rightUpLeg
leftUpperLeg | leftUpLeg
rightFoot | rightFoot, rightToeBase
leftFoot | leftFoot, leftToeBase
torso | spine, spine1, spine2, leftShoulder, rightShoulder
face | head, neck

###  A.8 Selective Aggregation via Semantic Clustering

We note that body part $\mathbf{a}\in\mathbf{A}$ is a hyperparameter that can
be easily given by annotating a set of SMPL mesh vertices with a body part
label. In practice, we define $\mathbf{A}$ as 12 body parts obtained by
merging publicly available SMPL segmentation maps from Meshcapade [45].
Correspondence between defined body parts and SMPL body segmentation labels is
provided in Tab. 4. Representing each body part $\mathbf{a}\in\mathbf{A}$ as a
binary operator that outputs $1$ if the corresponding SMPL mesh vertex is part
of the body part $\mathbf{a}$ else $0$, we can define the interaction region
for $\mathbf{a}$ in the canonical space as below:

$\mathbf{I}_{\mathbf{a}}=\bigcup\limits_{i;~{}\mathbf{a}(\mathbf{v}^{i})=1}\\{\mathbf{x}^{c}|\
||\mathbf{x^{c}}-\mathbf{v}_{i}||\leq\epsilon\\}$ (12)

where $\epsilon$ is the interaction threshold, where we set as
$\epsilon=0.13$. The interaction region $\mathbf{I}_{\mathbf{a}}$ plays a role
of determining whether the provided image contains the HOI involving contact
with body part $\mathbf{a}$. Specifically, we ignore the image if none of the
3D canonical points within $\mathbf{I}_{\mathbf{a}}$ are warped and projected
into the object mask. Putting it all together, selective aggregation for 3D
occupancy can be described as:

$\Phi_{o}^{c}(\mathbf{x^{c}}|s)={{\sum\limits_{k=1}^{|\mathbf{G}(p)|}r_{k}\mathcal{M}_{k}(\Pi_{k}(\mathcal{W}(\mathbf{x^{c}})))\cdot\mathbf{1}(1\in\mathcal{M}_{k}(\Pi_{k}(\mathcal{W}(\mathbf{I}_{\mathbf{a}}))))}\over{\sum\limits_{k=1}^{|\mathbf{G}(p)|}r_{k}\mathcal{I}_{k}(\Pi_{k}(\mathcal{W}(\mathbf{x^{c}})))\cdot\mathbf{1}(1\in\mathcal{M}_{k}(\Pi_{k}(\mathcal{W}(\mathbf{I}_{\mathbf{a}}))))}}$

(13)

where $\mathbf{G}(p)$ denotes the set of generated images from single HOI
prompt $p\in\mathbf{P}$ and $\mathbf{1}(\cdot)$ is a binary operator that
returns $1$ if the provided input is true else $0$. At inference, learned
occupancy probabilities are used to compute $\Phi_{o}(\mathbf{x}|\theta,s)$
(refer to Eq. 1) as described in Supp. Mat. A.7.

##  Appendix B Additional Details for Experiments

###  B.1 Dataset

Generated Dataset. We refer readers to Tab. 3 for full per-category statistics
for the generated dataset.

Image Search Dataset. We use $m=12$ prompts (same as the prompts used for
generating dataset) for category motorcycle and $m_{\text{aug}}=22$ viewpoint
augmentations (same as in Supp. Mat. A.1) for image search, and we set the
desired number of retrieved images as $N=1000$ or $N=10000$. We crawl up to
$\lfloor\tau_{\text{mult}}\times{N\over m\times
m_{\text{aug}}}\rfloor+\tau_{\text{add}}$ image links, where
$\tau_{\text{mult}}$ and $\tau_{\text{add}}$ are introduced to tolerate the
number of undownloadable/unreadable files. Specifically, we set
$\tau_{\text{mult}}=1.1$ and $\tau_{\text{add}}=1$. Subsequently, collected
images are resized to a shorter side length of 512 and center-cropped,
resulting in $512\times 512$ images.

Extended COCO-EFT Dataset for Testing. We provide detailed dataset preparation
procedures for the COCO-EFT [32] dataset. Similar to filtering (Sec. 3.2), we
only retain samples with a single human, single object (of target category),
and filter the images based on _intersection over smaller bounding box_ value
between the human and the object. We assume no human-object interaction if
this value is below 0.5. It is important to note that we do not apply any
manual filtering to ensure fairness in our evaluation. After filtering, we
compute the perspective camera parameters following Supp. Mat. A.4, except we
optimize for 3000 iterations and early-terminate if the joint-reprojection
loss is below 0.5 in pixel scale. We reject the image if the joint-
reprojection loss is over 1.0 in pixel scale. To ensure multi-viewpoint
evaluation, we only use categories with more than 30 images in the extended
COCO-EFT dataset. Refer to Tab. 5 for the summary of statistics.

Table 5: Statistics in the Extended COCO-EFT Dataset. We report number of
images for categories with a minimum of 30 images in COCO-EFT [32] dataset.

Category | Number of Images in Dataset
---|---
Motorcycle | 36
Bench | 37
Backpack | 83
Handbag | 47
Tie | 37
Frisbee | 36
Skis | 86
Snowboard | 67
Sports ball | 67
Baseball glove | 49
Skateboard | 176
Surfboard | 110
Tennis racket | 117
Cell phone | 60

###  B.2 Projective Average Precision

We provide detailed protocols for PAP evaluation and intuition behind each
step in this section. Briefly speaking, the PAP metric quantifies the validity
of object occupancy distribution in pose-deformed 3D space (current space)
without 3D annotations by comparing the projection of distribution with 2D
annotation from multiple viewpoints. Given the category keyword to evaluate,
we first start by deforming the distribution from canonical space to pose-
deformed space using annotated SMPL pose from the test dataset following the
inference method described in Sec. 3.3. Note that we can get probability
occupancy values for equispaced gridpoints in pose-deformed space. We
discretize the distribution in pose-deformed space for various threshold
values and project binary occupancy using an annotated perspective camera.
Discretization is applied to bypass the ambiguity of mixing probabilities when
more than one 3D probability value falls into the same pixel in 2D. We use all
thresholds from $0.01\sim 1$ equispaced with interval $0.01$. Using multiple
thresholds enables us to evaluate the distribution regarding the intra-class
variation of object geometry. Next, we compute pixel-wise precision and recall
between rendered mask and annotated object segmentation mask for all
thresholds. Specifically, we downsample rendered mask and object segmentation
mask preserving aspect ratio with a shorter length being 32 before we compute
precision and recall. We downsample for two reasons: (1) to allow other 3D
representations that require high-compute in the PAP evaluation pipeline, and
(2) to tolerate the variance in object size and geometry. Note that the goal
of the PAP metric is to evaluate validity of the distribution, not the
accuracy or quality of the reconstruction. Finally, we compute interpolated AP
similar to Pascal VOC 2008 [17] using precision-recall values, which are
subsequently averaged across all test images within the given category to
yield the PAP value. We report two different PAP metrics in terms of
interpolation methods when predicted occupancy is entirely $0$ after
discretization:

  * •

Vanilla: Uses the highest precision value from lower thresholds when
discretized distribution is entirely $0$

  * •

Strict: Sets precision value as $0$ when discretized distribution is entirely
$0$

Note that Human-Occlusion-Aware PAP metrics follow the same protocols, except
we exclude the human-occluded region when computing precision and recall. We
do not apply semantic clustering during quantitative evaluation, i.e., we
evaluate marginalized distribution aggregated with all images generated from
all prompts per category.

##  Appendix C Additional Qualitative Results

We report additional qualitative results for various categories in this
section. Same as in Sec. 4.4, we use SMPL pose sampled from the extended COCO-
EFT dataset for COCO categories or generated dataset for LVIS categories to
deform the distribution in canonical space to pose-deformed space. See Fig. 14
$\sim$ 36 for results.

##  Appendix D Limitations & Future Works

Granularity. Our method returns a plausible set of object distributions;
however, we represent them as a low-resolution voxel field (resolution
$48^{3}$), which limits the representability and granularity of the results.
Future research can explore alternative 3D representations (e.g., volume
rendering-based methods similar to NeRF [46]) to improve computation
efficiency and achieve higher quality.

Problems with Small Objects. Our method particularly shows weakness in small
objects, especially those interacting with hands, primarily due to heavy
occlusion and expressivity constraints in the SMPL model. For future research,
employing the SMPL-X [53] representation to learn distributions for small
objects interacting with hands, or using close-view cameras from various
angles to reduce occlusion, could be beneficial.

Bias and Artifacts in Synthesized Images. We use viewpoint augmentation to
control the camera distribution during image synthesis; however, this method
lacks full controllability and requires improvement. Although we minimize this
effect by assigning camera distribution-aware accumulation scores, there is
still a possibility of bias. One potential approach to address this challenge
is by employing the PerpNeg algorithm [1] to enhance viewpoint control during
generation. Additionally, synthesized images are likely to contain artifacts,
which could propagate errors in later steps (e.g., human prediction) and lead
to incorrect modeling of the occupancy probability distribution. Improving
image synthesis methods will help mitigate such challenges.

Bias due to Heavy Filtering. As our filtering strategy involves various off-
the-shelf methods, employing heavy filtering may introduce bias. For example,
the object detection method we use may be imperfect and could filter out
images even if an object is present, leading to bias in the generated dataset
after filtering. Consequently, this may result in bias in the occupancy
probability distribution. Soft filtering methods (i.e., applying confidence
weights to each image instead of removing images with hard thresholds) may be
an alternative, which we leave for future work.

Modelling Multimodal Scenarios. The semantic clustering step in our method
provides a means to understand objects that humans can interact with in
various ways. While our method effectively models plausible HOI exhibiting
specific interaction types, it requires manual definition of body parts and
specification of HOI prompts to represent the semantics. Additionally, relying
on user evaluation for identifying plausible semantic clusters hinders the
efficient and automatic expansion of the corpus, as this process becomes
manual. We acknowledge the need for further research to enhance the automatic
generation of 3D HOI spatial relations without this manual constraint.

Category Limits. Currently, our method mainly considers categories from COCO
[39] and a few categories from LVIS [20] due to the availability of the object
detection and segmentation method. A promising future direction is to replace
this current object detector with open-vocabulary models like ODISE [78] to
incorporate additional categories.

Evaluation Metrics. Our evaluation metric (PAP) has room for improvement. For
example, our current method utilizes a simple downsampling strategy to smooth
the distribution during the protocol, which could be enhanced with other
strategies (e.g., kernel-smoothing methods). Additionally, it is worth
exploring enhancements to the metric that can effectively quantify the
validity of multimodal distributions.

Potential Downstream Applications. Our method exhibits myriads of potential
downstream applications, such as; (1) using our extracted knowledge as prior
for HOI modeling (e.g., replacing interaction labels in PHOSA [83]); (2)
improving action recognition methods based on the current pose of the human
and object; (3) scene generation and object localization from human postures;
or (4) applications for robotics.

Figure 14: Qualitative results for category backpack. Figure 15: Qualitative
results for category bicycle. Figure 16: Qualitative results for category
chair. Figure 17: Qualitative results for category motorcycle. Figure 18:
Qualitative results for category umbrella. Figure 19: Qualitative results for
category frisbee. Figure 20: Qualitative results for category skis. Figure 21:
Qualitative results for category surfboard. Figure 22: Qualitative results for
category skateboard. Figure 23: Qualitative results for category baseball
glove. Figure 24: Qualitative results for category bench. Figure 25:
Qualitative results for category bed. Figure 26: Qualitative results for
category sports ball. Figure 27: Qualitative results for category cell phone.
Figure 28: Qualitative results for category sunglasses. Figure 29: Qualitative
results for category soccer ball. Figure 30: Qualitative results for category
handbag. Figure 31: Qualitative results for category laptop. Figure 32:
Qualitative results for category sweater. Figure 33: Qualitative results for
category snowboard. Figure 34: Qualitative results for category hat. Figure
35: Qualitative results for category scarf. Figure 36: Qualitative results for
category tie.