capbtabboxtable[][]

11institutetext:  Max Planck Institute for Informatics, Saarland Informatics
Campus, Germany 22institutetext: University of Tübingen, Germany
22email: {xxie<EMAIL_ADDRESS><EMAIL_ADDRESS>
# CHORE: Contact, Human and Object REconstruction from a single RGB image

Xianghui Xie 11 Bharat Lal Bhatnagar 1122 Gerard Pons-Moll 1122

###### Abstract

While most works in computer vision and learning have focused on perceiving 3D
humans from single images in isolation, in this work we focus on capturing 3D
humans interacting with objects. The problem is extremely challenging due to
heavy occlusions between human and object, diverse interaction types and depth
ambiguity. In this paper, we introduce CHORE, a novel method that learns to
jointly reconstruct human and object from a single image. CHORE takes
inspiration from recent advances in implicit surface learning and classical
model-based fitting. We compute a neural reconstruction of human and object
represented implicitly with two unsigned distance fields, and additionally
predict a correspondence field to a parametric body as well as an object pose
field. This allows us to robustly fit a parametric body model and a 3D object
template, while reasoning about interactions. Furthermore, prior pixel-aligned
implicit learning methods use synthetic data and make assumptions that are not
met in real data. We propose a simple yet effective depth-aware scaling that
allows more efficient shape learning on real data. Our experiments show that
our joint reconstruction learned with the proposed strategy significantly
outperforms the SOTA. Our code and models will be released to foster future
research in this direction.

##  1 Introduction

Figure 1: From a single image, our method jointly reconstructs the 3D human
pose and shape coherently with the object by reasoning about interaction and
contacts. Instead of predicting human and object independently and imposing
coherency post-hoc[64], our method learns to reconstruct both simultaneously
including contacts, leading to significant improvements over the state-of-the-
art.

In order to deploy robots and intelligent systems in the real world, they must
be able to perceive and understand humans interacting with the real world from
visual input. While there exists a vast literature in perceiving humans in 3D
from single images, the large majority of works perceive humans in isolation
[32, 46, 5, 53, 35]. The joint 3D perception of humans and objects has
received much less attention and is the main focus of this work. Joint
reconstruction of humans and objects is extremely challenging. The object and
human occlude each other making inference hard; it is difficult to predict
their relative size and spatial arrangement in the 3D world, and the visual
evidence for contacts and interactions consists of only a very small region in
the image.

Recent work [64] addresses these challenges by reconstructing 3D objects and
humans separately, and imposing hand-crafted rules like manually defined
object contact regions and object size. While this work takes a brave step
forward in modeling humans and objects directly in the wild, it is difficult
to specify beforehand what interactions are possible with the objects.
Consider the chair in Fig. 3: one can sit, lean, move it from the handle, back
or legs, which constitutes different pairs of human-object contacts that are
not easy to manually enumerate. The main contribution of our work is a model
to jointly reconstruct humans and objects, while learning the interactions
from data, without hand-crafted rules.

In this work, we introduce CHORE, a novel method to jointly recover a 3D
parametric body model and the 3D object, from a single image. CHORE is built
on two high-level ideas. First, to reason the spatial configuration of the
human and the object, we argue for learning-based joint neural reconstruction
of humans and objects. Second, to capture fine-grained interactions between
the human and the object, we predict contacts in 3D, directly from the input
image and learn a strong prior from data.

Instead of directly regressing human parameters, we first obtain a neural
reconstruction of human and object leveraging recent advances in implicit
surface learning [17, 52]. We additionally predict correspondences from the
neural implicit human and object surface to the SMPL [41] body model to model
contacts. To obtain robust object fitting, we also predict a neural field for
object orientation and translation. This allows us to fit the SMPL model and
the object template to the neural reconstructions to add robustness and
control to the output surfaces. To train our network on real data, we propose
a training strategy that enables us to effectively learn pixel-aligned
implicit functions with perspective cameras, which leads to more accurate
prediction and joint fitting. In summary, our key contributions are:

  * •

We propose CHORE, the first end-to-end learning approach that can reconstruct
human, object and contacts from a single RGB image. Our correspondence and
contact prediction allow us to accurately register a controllable body model
and 3D object to the image.

  * •

Different from prior works that use weak perspective cameras and learn from
synthetic data, we use perspective camera model, which is crucial to train on
real data. To this end, we propose a new training strategy that allows
effective pixel-aligned implicit learning with perspective cameras.

  * •

Through our effective training and joint reconstruction, our model achieves a
remarkable performance improvement of over $50\%$ compared to prior art [64].
We will make our code and models publicly available to foster further research
in this direction.

##  2 Related Work

We propose the first learning-based approach to jointly reconstruct 3D human,
object and contacts from a single image. In this regard, we briefly discuss
recent advances in parametric and non-parametric human reconstruction from
single images. We then discuss approaches that reconstruct 3D objects from
images and finally, approaches that can jointly reason about human-object
interaction.

#### Human reconstruction from images.

One of the most common approaches to reconstruct 3D humans from images is to
fit a parametric model such as SMPL [41] to the image [10, 47, 4, 3, 36].
Learning-based approaches have also been used to directly regress model
parameters corresponding to pose and shape [46, 32, 48, 35, 25] as well as
clothing [8, 2, 31]. More recently, implicit function based reconstruction has
been able to capture fine-grained details including clothing and hair geometry
[52, 29, 53, 65]. These methods however cannot reconstruct 3D objects, let
alone human-object interaction.

#### Object reconstruction from images.

Given a single image, 3D objects can be reconstructed as voxels [18, 59, 30],
point clouds [21, 37] and meshes [33, 57, 49]. Similar to human
reconstruction, implicit functions have shown huge success for object
reconstruction as well [42, 62, 45]. The research in this direction has been
significantly aided by large scale datasets [14, 61, 55]. We urge the readers
to see the recent review [24] of image-based 3D reconstruction of objects.
These approaches have shown great promise in rigid object reconstruction but
articulated humans are significantly more challenging and human-object
interaction even more so. Unlike these methods, our approach can jointly
reconstruct humans, objects and their interactions.

#### Human object interactions.

Modelling 3D human-object interactions is extremely challenging. Recent works
have shown impressive performance in modelling hand-object interactions from
3D [56, 12], 2.5D [11, 13] and images [34, 19, 28, 20, 63]. Despite being
quite impressive, these works have a shortcoming in that they are restricted
to only hand-object interactions and cannot predict the full body. Modelling
full-body interactions is significantly more challenging, with works like PROX
[26] and [58] being able to fit 3D human model to satisfy the 3D scene
constraints [54, 66, 27]. Recently, this direction has also been extended to
model human-human interaction [23] and self-contacts [22, 44]. Broadly
speaking, these works can optimize the human model to satisfy static scene
constraints but cannot jointly reconstruct human-object interaction from
images. Few works directly reason about 3D contacts from images [15, 69] and
video [50, 38, 43] but do not reconstruct 3D human and object from a single
RGB image.

Closest to our work are [58] and PHOSA [64] . Weng et al. [58] predict 3D
human and scene layout separately and use scene constraints to optimize human
reconstruction. Instead of predicting contacts from inputs, they use
predefined contact weights on SMPL-X[47] vertices, which is prone to errors.
PHOSA first fits SMPL and object meshes separately and then use hand crafted
heuristics like predefined contact pairs to reason human object interaction.
These heuristics are neither scale-able nor very accurate. On the other hand,
our method does not rely on heuristics and can learn joint reconstruction and
contact priors directly from data. Our experiments show that our learning
based method easily outperforms heuristic-based PHOSA and [58].

##  3 Method

Figure 2: We present an approach to jointly reconstruct human, object and
contacts from a single RGB image. Given an image and query points we compute
pixel aligned features $\mathbf{F}$, using a neural network $f^{\text{enc}}$
and then use these features to neural reconstruct the 3D human and object. We
also predict SMPL part correspondences for each point for accurate contact
fitting and object 6DoF pose to initialize object fitting. This allows us to
fit the SMPL model and the object mesh to the implicit neural reconstruction.

We describe here the components of CHORE, a method for 3D reconstruction of
the human, object and their contacts from a single RGB image. In this work, we
focus on a single person in close interaction with one dynamic movable object.
Our approach is inspired by the classical image based model fitting methods
[4, 10], which fit a body model parameterized by shape and pose
$\mathbf{\boldsymbol{\theta},\boldsymbol{\beta}}$ by minimizing an energy:

$E(\mathbf{\boldsymbol{\theta},\boldsymbol{\beta}})=E_{data}+E_{J2D}+E_{reg},$
(1)

where $E_{data}$ is a 2D image based loss like a silhouette loss, $E_{J2D}$ is
re-projection loss for body joints, and $E_{reg}$ is a body pose and shape
prior loss. Estimating the human and the object jointly however is
significantly more challenging. The relative configuration of human and object
needs to be accurately estimated despite occlusions, lack of depth, and hard
to detect contacts. Hence, a plain extension of a hand-crafted 2D objective to
incorporate the object is far too prone to local minima. Our idea is to
jointly learn to predict, based on a single image, 3D neural fields (CHORE
fields) for the object, human and their relative pose and contacts, to
formulate a learned robust 3D data term. We then fit a parametric body model
and a template object mesh to the predicted CHORE neural fields, see overview
Fig. 2.

In this section, we first explain our CHORE neural fields in Sec. 3.1, and
then describe our novel human-object fitting formulation using learned CHORE
neural fields in Sec. 3.2. Unlike prior work on pixel-aligned neural implicit
reconstruction (PiFu [52]) which learns from synthetic data and assume a weak
camera model, we learn from real data captured using perspective cameras. This
poses new challenges which we address with a simple and effective deformation
of the captured real 3D surfaces to account for perspective effects (Sec.
3.3).

###  3.1 CHORE neural fields from single images

As representation, we use the SMPL body model
$H(\boldsymbol{\theta},\boldsymbol{\beta})$ [41] which parameterizes the 3D
human as a function of pose $\boldsymbol{\theta}$ (including global
translation) and shape $\boldsymbol{\beta}$, and a template mesh for the
object. Our idea is to leverage a stronger 3D data term coming from learned
CHORE neural fields to fit SMPL and the object template to the image. In
contrast to a plain joint surface reconstruction of human and object, the
resulting SMPL and object mesh can be further controlled and edited. To obtain
a good fit to the image, we would ideally i) minimize the distance between the
optimized meshes and the corresponding ground truth human and object surfaces,
and ii) enforce that human-object contacts of the meshes are consistent with
input. Obviously, ground truth human and object surfaces are unknown at test
time. Hence, we approximate them with CHORE fields. Specifically, a learned
neural model predicts 1) the unsigned distance fields of the object and human
surfaces, and 2) a part correspondence field to the SMPL model, for more
robust SMPL fitting and contact modeling, and 3) an object pose field, which
we use to initialize the object pose.

This constitutes a complex multi-task learning problem which we address with a
single neural network. We use a CNN to extract pixel-aligned features, which
have been shown effective for 3D reconstruction [6, 52, 16]. From these
features we predict the distance, part-correspondence and object pose fields
in a point-wise fashion. We explain the details of our network below.

Input encoding $f^{enc}$. We stack RGB image with human and object masks in
separate channels as input and use a leaned feature extractor $f^{enc}$ [52]to
extract image feature grid $\mathbf{F}$. Given a query point
$\mathbf{p}\in\mathbb{R}^{3}$, and the image feature $\mathbf{F}$, we extract
a pixel-aligned point feature $\mathbf{F}[\pi_{3D\mapsto 2D}(\mathbf{p})]$ by
projecting the 3D point to the image. We concatenate the 3D point coordinate
$\mathbf{p}$ to the pixel-aligned feature, allowing the network to distinguish
points at different depth along the same ray, hence
$\mathbf{F}_{\mathbf{p}}=(\mathbf{F}[\pi_{3D\mapsto
2D}(\mathbf{p})],\mathbf{p})$. The feature extractor $f^{enc}$ is trained with
losses defined on neural fields predictions which we explain later.

Human and object distance fields $f^{u}$. Based on the point feature
$\mathbf{F}_{\mathbf{p}}$, we use a point decoder $f^{u}$ to predict the
unsigned distances $u_{h}(\mathbf{p})$ and $u_{o}(\mathbf{p})$ from the point
to the human and object surface respectively. This allows to model non-
watertight meshes as well. From an unsigned distance field the 3D surface
$\mathcal{S}$ can be reconstructed by projecting points to it [17]:

$\mathcal{S}_{i}=\\{\mathbf{p}-f^{u}_{i}(\mathbf{F}_{\mathbf{p}})\nabla_{\mathbf{p}}f^{u}_{i}(\mathbf{F}_{\mathbf{p}})\,|\,\mathbf{p}\in\mathbb{R}^{3}\\}$
(2)

where $f^{u}_{i}$ can be human ($f^{u}_{h}$) or object ($f^{u}_{o}$) UDF
predictor. In practice, points $\mathbf{p}$ are initialized from a fixed 3D
volume and projection is done iteratively[17].

To train neural distance function $f^{u}_{i}$, we sample a set of points
$\mathcal{P}_{i}$ near the ground truth surfaces and compute the ground truth
distance $\text{UDF}_{\text{gt}}(\mathbf{p})$. The decoder $f^{u}_{i}$ is then
trained to minimize the $L_{1}$ distance between clamped prediction and ground
truth distance:

$L_{u_{i}}=\sum_{\mathbf{p}\in\mathcal{P}_{i}}|\min(f^{u}_{i}(\mathbf{F}_{\mathbf{p}}),\delta)-\min(\text{UDF}_{\text{gt}}(\mathbf{p}),\delta)|$
(3)

where $\delta$ is a small clamping value to focus on the volume nearby the
surface.

Part fields $f^{p}$ and contacts. While the unsigned distance predictions can
reconstruct the human and object surface, they do not provide contacts, and
fitting template SMPL meshes using only distance fields often fails [6]. Hence
we use another decoder to predict a part correspondence field to the SMPL
model with two purposes: i) to formulate a robust SMPL fitting objective [6,
7] and ii) to derive which body parts are in contact with the object points.
Mathematically, given the point feature $\mathbf{F}_{\mathbf{p}}$ for a point
$\mathbf{p}\in\mathbb{R}^{3}$, we predict its corresponding SMPL part
$\mathbf{l}_{\mathbf{p}}=f^{p}(\mathbf{F}_{\mathbf{p}})\in\\{1,2,...K\\}$.
$K=14$ is the total number of SMPL parts. The part prediction decoder is
trained with standard categorical cross entropy, $L_{p}$.

Object pose fields $f^{R}$ and $f^{c}$. Accurate object fitting to the CHORE
UDF requires good initialization for the object pose. We learn a predictor
$f^{R}:\mathbf{F_{\mathbf{p}}}\mapsto\mathbb{R}^{3\times 3}$ that takes the
point feature $\mathbf{F}_{\mathbf{p}}$ as input and predicts the rotation
matrix $\mathbf{R}$. Predicting the object translation in camera coordinates
directly is more tricky due to the well known depth-scale ambiguity. Hence our
predictions are relative to the human. We train a decoder
$f^{c}:\mathbf{F_{\mathbf{p}}}\mapsto\mathbb{R}^{5}$ which predicts the x, y
coordinate of SMPL center (at fixed depth) in camera coordinates, and the
object center relative to the SMPL center based on the point feature
$\mathbf{F}_{\mathbf{p}}$. Both $f^{R}$ and $f^{c}$ are trained with mean
squared loss $L_{R}$ and $L_{c}$.

Our feature encoder $f^{enc}$ and neural predictors $f^{u},f^{p},f^{R},f^{c}$
are trained jointly with the objective:
$L=\lambda_{u}(L_{u_{h}}+L_{u_{o}})+\lambda_{p}L_{p}+\lambda_{R}L_{R}+\lambda_{c}L_{c}$.
See supplementary for more details on our network architecture and training.

###  3.2 3D human and object fitting with CHORE fields

With CHORE fields, we reformulate Eq. 1 to include a stronger data term
$E_{data}$ in 3D, itself composed of 3 different terms for human, object, and
contacts which we describe next.

3D human term. This term consists of the sum of distances from SMPL vertices
to the CHORE neural UDF for the human $f^{u}_{h}$, and a part based term
leveraging the part fields $f^{p}$ discussed in Sec. 3.1:

$E^{h}_{\text{data}}(\boldsymbol{\theta},\boldsymbol{\beta})=\sum_{\mathbf{p}\in
H(\boldsymbol{\theta},\boldsymbol{\beta})}(\lambda_{h}\min(f^{u}_{h}(\mathbf{F}_{\mathbf{p}}),\delta)+\lambda_{p\prime}L_{p}(l_{\mathbf{p}},f^{p}(\mathbf{F}_{\mathbf{p}}))),$
(4)

Here, $L_{p}(\cdot,\cdot)$ is the standard categorical cross entropy loss
function and $l_{\mathbf{p}}$ denotes our predefined part label on SMPL vertex
$\mathbf{p}$. $\lambda_{h},\lambda_{p\prime}$ are the loss weights. The part-
based term $L_{p}(\cdot,\cdot)$ is effective to avoid, for example, matching a
person’s arm to the torso or a hand to the head. To favour convergence, we
initialize the human pose estimated from the image using [51].

3D object term. We assume a known object mesh template with vertices
$\mathbf{O}\in\mathbb{R}^{3\times N}$ and optimize the object rotation
$\mathbf{R}_{o}\in SO(3)$, translation $\mathbf{t}_{o}\in\mathbb{R}^{3}$ and
scale $s_{o}\in\mathbb{R}$ leveraging the neural fields predictions. Given
template vertices and object pose, we compute the posed object as
$\mathbf{O}^{\prime}=s_{o}(\mathbf{R}_{o}\mathbf{O}+\mathbf{t}_{o})$.
Intuitively, the distance between object vertices $\mathbf{O}^{\prime}$ and
underlying object surface should be minimized. Furthermore, the projected 2D
silhouette should match the observed object mask $\mathbf{M}_{o}$, yielding
the following objective:

$E_{\text{data}}^{o}(\mathbf{R}_{o},\mathbf{t}_{o},s_{o})=\sum_{\mathbf{p}\in\mathbf{O}^{\prime}}(\lambda_{o}\min(f^{u}_{o}(\mathbf{F}_{\mathbf{p}}),\delta)+\lambda_{\text{occ}}L_{\text{occ-
sil}}(\mathbf{O}^{\prime},\mathbf{M}_{o})+\lambda_{\text{reg}}L_{\text{reg}}(\mathbf{O}^{\prime})),$
(5)

The first term computes the distance between fitted object and corresponding
surface represented implicitly with the network $f_{o}^{u}$. The second term
$L_{\text{occ-sil}}$ is an occlusion-aware silhouette loss [64]. We
additionally use $L_{\text{reg}}$ computed as the distance between predicted
object center and center of $\mathbf{O}^{\prime}$. This regularization
prevents the object from being pushed too far away from predicted object
center.

The object rotation is initialized by first averaging the rotation matrices of
the object pose field near the object surface (points with
$f^{u}_{o}(\mathbf{F}_{\mathbf{p}})<4mm$):
$\mathbf{R}_{o}=\frac{1}{M}\sum^{M}_{k=1}\mathbf{R}_{k}$. We further run SVD
on $\mathbf{R}_{o}$ to project the matrix to $SO(3)$. Object translation
$\mathbf{t}_{o}$ is similarly initialized by averaging the translation part of
the pose field nearby the surface. Our experiments show that pose
initialization is important for accurate object fitting, see Sec. 4.6.

Joint fitting with contacts. While Eq. 4 and Eq. 5 allow fitting human and
object meshes coherently, fits will not necessarily satisfy fine-grained
contact constraints. Our key idea here is to jointly reconstruct and fit
humans and objects while reasoning about contacts. We minimize a joint
objective as follows:

$E_{\text{data}}(\boldsymbol{\theta},\boldsymbol{\beta},\mathbf{R_{o}},\mathbf{t_{o}},s_{o})=E_{\text{data}}^{h}+E_{\text{data}}^{o}+\lambda_{c}E_{\text{data}}^{c},$
(6)

where $E_{\text{data}}^{h}$ and $E_{\text{data}}^{o}$ are defined in Eq. 4 and
Eq. 5 respectively. The contact term $E_{\text{data}}^{c}$ consists of the
chamfer distance $d(\cdot,\cdot)\mapsto\mathbb{R}$ between the sets of human
$H_{j}^{c}$ and object points $O_{j}^{c}$ predicted to be in contact:

$E_{\text{data}}^{c}(\mathbf{R}_{o},\mathbf{t}_{o},s_{o})=\sum_{j=1}^{K}d(H_{j}^{c}(\boldsymbol{\theta},\boldsymbol{\beta}),O_{j}^{c}).$
(7)

Points on the $j-th$ body part of SMPL
($H_{j}(\boldsymbol{\theta},\boldsymbol{\beta})$) are considered to be in
contact when their UDF $f^{u}_{o}(\mathbf{F}_{\mathbf{p}})$ to the object is
smaller than a threshold:
$H_{j}^{c}(\boldsymbol{\theta},\boldsymbol{\beta})=\\{\mathbf{p}\,|\,\mathbf{p}\in
H_{j}(\boldsymbol{\theta},\boldsymbol{\beta})\,\textrm{and}\,f^{u}_{o}(\mathbf{F}_{\mathbf{p}})\leq\epsilon\\}$.
Analogously, object points are considered to be in contact with body part $j$
when their UDF to the body is small and they are labelled as part $j$:
$O_{j}^{c}=\\{\mathbf{p}\,|\,\mathbf{p}\in\mathbf{O}^{\prime}\,\textrm{and}\,f^{u}_{h}(\mathbf{F}_{\mathbf{p}})\leq\epsilon\
\textrm{and}\,f^{p}(\mathbf{F}_{\mathbf{p}})=j\\}$. The contact term
$E_{\text{data}}^{c}$ encourages the object to be close with the corresponding
SMPL parts in contact. The loss is zero when no contacts detected. This is
important to have a physically plausible and consistent reconstruction, see
Sec. 4.5.

Substituting our data term into Eq. 1, we can now write our complete
optimization energy as:
$E(\boldsymbol{\theta},\boldsymbol{\beta},\mathbf{R_{o}},\mathbf{t_{o}},s_{o})=E_{data}^{o}+E_{data}^{h}+\lambda_{c}E_{data}^{c}+\lambda_{\text{J}}E_{J2D}+\lambda_{r}E_{\text{reg}}$.
See Supp. for details about different loss weights $\lambda$’s.

###  3.3 Learning pixel-aligned neural fields on real data

Figure 3: Learning neural fields on real data. We show projections of GT
meshes overlapped with input image using different camera models in b and d.
The neural reconstructions (colored based on predicted part labels
$f^{p}(\mathbf{F}_{\mathbf{p}})$) are shown in c, e and f. It can be seen that
weak perspective projection is not accurately aligned with image for real data
(b) hence network trained with this predicts noisy output (c). Perspective
camera projection is accurately aligned (d) but directly applying it, without
accounting for depth-scale ambiguity, makes learning hard. Network trained
with this is still not satisfactory (e). Our proposed method reduces depth
ambiguity while maintaining accurate alignment, leading to better predictions
(f).

Our training data consists of real images paired with reference SMPL and
object meshes. Here, we want to follow the pixel-aligned training paradigm,
which has been shown effective for 3D shape reconstruction. Since both depth
and scale determine the size of objects in images this makes learning from
single images ambiguous. Hence, existing works place the objects at a fixed
distance to the camera, and render them using a weak perspective [52, 68, 53]
or perspective camera model [67, 60, 59, 62]. This effectively allows the
network to reason about only scale, instead of both scale and depth at the
same time.

This strategy to generate training pairs is only possible when learning from
_synthetic data_ , which allows to move objects and re-render the images.
Instead, we learn from _real_ images paired with meshes. Re-centering meshes
to a fixed depth breaks the pixel alignment with the real images. An
alternative is to adopt a weak perspective camera model (assume all meshes are
centered at a constant depth), but the alignment is not accurate hence leads
to inaccurate learning, see Fig. 3 b) and c). Alternatively, attempting to
learn a model directly from the original pixel-aligned data with the intrinsic
depth-scale ambiguity leads to poor results, as we show in Fig. 3 e). Hence,
our idea is to adopt a perspective camera model, and transform the data such
that it is centered at a fixed distance to the camera, while preserving pixel
alignment with the real images. We demonstrate that this can be effectively
achieved by applying depth-dependent scale factor.

Our first observation is that scaling the mesh vertices $\mathbf{V}$ by $s$
does not change its projection to the image. _Proof:_ Let
$\mathbf{v}=(\mathbf{v}_{x},\mathbf{v}_{y},\mathbf{v}_{z})\in\mathbf{V}$ be a
vertex of the original mesh, and
$\mathbf{v}^{\prime}=(s\mathbf{v}_{x},s\mathbf{v}_{y},s\mathbf{v}_{z})\in
s\mathbf{V}$ a vertex of the scaled mesh. For a camera with perspective focal
length $f_{x},f_{y}$ and camera center $c_{x},c_{y}$, the vertex $\mathbf{v}$
is projected to the image as
$\pi(\mathbf{v})=(f_{x}\frac{\mathbf{v}_{x}}{\mathbf{v}_{z}}+c_{x},f_{y}\frac{\mathbf{v}_{y}}{\mathbf{v}_{z}}+c_{y})$.
The scaled vertices will be projected to the exact same pixels,
$\pi(\mathbf{v}^{\prime})=(f_{x}\frac{s\mathbf{v}_{x}}{s\mathbf{v}_{z}}+c_{x},f_{y}\frac{s\mathbf{v}_{y}}{s\mathbf{v}_{z}}+c_{y})=\pi(\mathbf{v})$
because the scale cancels out. This result might appear un-intuitive at first.
What happens is that the object size gets scaled, but its center is also
scaled, hence making the object bigger/smaller and pushing it further/closer
to the camera at the same time. This effects cancel out to produce the same
projection.

Given this result, all that remains to be done, is to find a scale factor
which centers objects at a _fixed_ distance $z_{0}$ to the camera. The desired
scale factor that satisfies this condition is $s=\frac{z_{0}}{\mu_{z}}$, where
$\mu_{z}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{v}^{i}_{z}$ is the mean of the un-
scaled mesh vertices depth. _Proof:_ The proof follows immediately from the
linearity property of the empirical mean. The depth of the transformed mesh
center is:

$\mu_{z}^{\prime}=\frac{1}{n}\sum^{n}_{i=1}\mathbf{v}_{z}^{i\prime}=\frac{1}{n}\sum^{n}_{i}\mathbf{v}_{z}^{i}\frac{z_{0}}{\mu_{z}}=\mu_{z}\frac{z_{0}}{\mu_{z}}=z_{0}$
(8)

This shows that the depth $\mu_{z}^{\prime}$ of the scaled mesh center is
fixed at $z_{0}$ and its image projection remains unchanged, as desired.

We scale all meshes in our training set as described above to center them at
$z_{0}=2.2m$ and recompute GT labels. To maintain pixel alignment we use the
camera intrinsic from the dataset. At test time, the images have various
resolution and unknown intrinsic with person at large range of depth (see Fig.
6). We hence crop and resize the human-object patch such that the person in
the resized patch appears as if they are at $z_{0}$ under the camera intrinsic
we use during training. We leverage the human mesh reconstructed from [51] as
a reference to determine the patch resizing factor. Please see supplementary
for details.

##  4 Experiments

Figure 4: Comparing our human and object reconstruction with Weng et al [58]
and PHOSA [64] in BEHAVE. The contact spheres are placed automatically at the
contact centers. Instead of predicting contacts from input images, [58] uses
predefined contact weights and fails to reconstruct spatial arrangement
properly. PHOSA also fails as it does not explicitly reason about the human-
object contacts. Our network on the other hand is trained jointly to
reconstruct the human and the object and learns a strong prior on the relative
arrangement, allowing for contact reasoning and accurate fitting.

#### Datasets.

We conducted experiments on the BEHAVE [9], COCO [39] and NTU-RGBD[40]
dataset. BEHAVE captures 8 subjects interacting with 20 different objects in
natural environments using RGBD cameras. Each image is paired with (pseudo)
ground truth SMPL and object fits in camera coordinate. CHORE is trained on
10.7k frames at 4 locations from BEHAVE and tested on 4.5k frames from one
unseen location. All 20 categories are seen in both training and test set but
at test time the object instances maybe different. As PHOSA [64] relies on 2D
object mask for pose estimation, we exclude images where the object is
occluded more than 70% for fair comparison, which leads to 4.1k test frames.
To compare with Weng et al.  [58], we select 10 common object categories
between BEHAVE and their method (in total 2.1k images) and compare our method
with them on this smaller set, denoted as $\mathrm{BEHAVE^{--}}$.

To verify the generalization ability, we also tested our method on COCO [39]
where each image contains at least one instance of the object categories in
BEHAVE and the object should not be occluded more than 50%. Images where there
is no visible contact between human and object are also excluded. There are
four overlapping categories between BEHAVE and COCO (backpack, suitcase, chair
and sports ball) and in total 913 images are tested.

We additionally test on the NTU-RGBD dataset, which is quite interesting since
it features human-object interactions. We select images from the NTU-RGBD
dataset following the same criteria for COCO and in total 1.5k images of chair
or backpack interactions are tested.

#### Evaluation and baselines.

To the best of our knowledge, PHOSA [64] and Weng et al. [58] are the closest
works. To compare with PHOSA [64], we annotated template object meshes with
contact part labels and intrinsic scales according to PHOSA’s convention. [58]
focuses on reconstructing the human in static scenes, hence their objects are
mainly from indoor environments. Therefore, we only compare with it on the
$\mathrm{BEHAVE^{--}}$ set and the indoor NTU-RGBD dataset.

For BEHAVE, we compute the Chamfer distance separately on SMPL and object
meshes after Procrustes alignment using the combined mesh. Since [58] uses a
different object template, we perform Procrustes alignment using only SMPL
vertices for $\mathrm{BEHAVE^{--}}$. As there are no 3D ground truth available
in COCO and NTU-RGBD for thorough quantitative comparison, we compare our
method with baselines through anonymous user study surveys.

###  4.1 Comparison on BEHAVE dataset

Figure 5: Comparing our methods with [64] and [58] in NTU-RGBD dataset[40].
Our method trained on BEHAVE generalizes well to NTU-RGBD and outperforms
baselines.

We compare our method against PHOSA [64] and [58] in Tab. 1. Note that the
Procrustes alignment is performed on combined mesh for BEHAVE and on SMPL mesh
only for $\mathrm{BEHAVE^{--}}$ for a fair comparison with [58]. Tab. 1
clearly shows that our learning based approach outperforms the heuristic
optimization based PHOSA and [58].

Dataset | Methods | SMPL $\downarrow$ | Object $\downarrow$
---|---|---|---
BEHAVE | PHOSA [64] | 12.17 (11.13) | 26.62 (21.87)
Ours | 5.79 (2.13) | 11.74 (8.35)
$\mathrm{BEHAVE^{--}}$ | PHOSA [64] | 6.09 (1.98) | 182.67 (319.64)
Weng et al [58] | 7.86 (3.48) | 51.86 (96.38)
Ours | 5.38 (2.07) | 14.73 (12.69)
Table 1: The mean and standard deviation of the Chamfer distance (cm) for SMPL
and object reconstruction on [9]. Our approach that jointly reasons about the
human, object and the contacts outperforms [58] and heuristics based PHOSA
[64].

We also compare our method qualitatively with PHOSA and [58] in Fig. 4. It can
be seen that PHOSA fails to reconstruct the object at the correct position
relative to the human. [58] also fails quite often as it uses predefined
contact weights on SMPL vertices instead of predicting contacts from inputs.
On the other hand, CHORE works well because it learns a 3D prior over human-
object configurations based on the input image. See supplementary for more
examples.

###  4.2 Generalisation beyond BEHAVE

Figure 6: Comparing our method with PHOSA[64] on COCO dataset[39]. For images
with more than one person, the human and object to be reconstructed are
highlighted by green and red boxes. It can be seen that our model trained on
BEHAVE outperforms PHOSA on in the wild images.

To verify our model trained on the BEHAVE dataset can generalize to other
datasets, we further test it on the NTU-RGBD [40] as well as COCO dataset [39]
and compare with [64, 58].

Comparison with PHOSA[64]. We test our method and PHOSA on the selected COCO
images discussed above. We then _randomly_ select 50 images and render ours
and PHOSA reconstruction results side by side. The results are collected in an
anonymous user study survey and we ask 50 people in Amazon Mechanical Turk
(AMT) to select which reconstruction is better. Notably, people think our
reconstruction is better than PHOSA on 72% of the images. A similar user study
for images from the NTU-RGBD dataset is released to AMT and on average 84% of
people prefer our reconstruction. Some example images are shown in Fig. 6 and
Fig. 5. See supplementary for details on the user study and more qualitative
examples.

Comparison with Weng et al. [58]. One limitation of [58] is that they use some
indoor scene layout assumptions, which makes it unsuitable to compare on ’in
the wild’ COCO images. Instead, we only test it on the NTU-RGBD dataset.
Similar to the comparison with PHOSA, we released a user study on AMT and ask
50 users to select which reconstruction is better. Results show that our
method is preferred by 94% people. Our method clearly outperforms [58], see
two examples in Fig. 5 and more examples in supplementary.

###  4.3 Comparison with a human reconstruction method

Our method takes SMPL pose predicted from FrankMocap[51] as initialisation and
fine tunes it together with object using network predictions. We compare the
human reconstruction performance with FrankMocap and PHOSA[64] in Tab. 2. It
can be seen that our joint reasoning of human and object does not deteriorate
the human reconstruction. On the other hand, it slightly improves the human
reconstruction as it takes the interacting object into account.

Metric (cm) | PHOSA | FrankMocap | Ours
---|---|---|---
v2v $\downarrow$ | 6.90 (2.68) | 6.91 (2.75) | 6.13 (2.83)
Chamfer $\downarrow$ | 6.11 (1.98) | 6.12 (2.00) | 5.45 (2.04)
Table 2: Mean vertex to vertex (v2v) and Chamfer distance for human
reconstruction results after Proscrustes alignment on SMPL mesh. Our method
that jointly reasons about human and object performs on par with human only
FrankMocap [51].

###  4.4 Pixel-aligned learning on real data

CHORE is trained with the depth-aware scaling strategy described in Sec. 3.3
for effective pixel aligned implicit surface learning on real data. To
evaluate its importance, we compare with two baseline models trained with weak
perspective and direct perspective (no depth-aware scaling on GT data) camera.
Qualitative comparison of the neural reconstruction is shown in Fig. 3. We
further fit SMPL and object meshes using these baseline models (with object
pose prediction) on BEHAVE test set and compute the errors for reconstructed
meshes, see Tab. 3b) and c). It can be clearly seen that our proposed training
strategy allows effective learning of implicit functions and significantly
improves the reconstruction.

###  4.5 Contacts modeling

We have argued that contacts between human and the object are crucial for
accurately modelling interaction. Quantitatively, the contact term
$E^{c}_{data}$ only reduces the object error slightly (Tab. 3f) and g) as
contacts usually constitute a small region of the full body. Nevertheless, it
makes significant difference qualitatively, as can be seen in Fig. 7. Without
the contact information, the object cannot be fitted correctly to the location
where contact is happening. Our contact prediction helps snap the object to
the correct contact location, leading to a more physically plausible and
accurate reconstruction.  Figure 7: Accurate human and object reconstruction
requires modeling contacts explicitly. Without contact information, the human
and the objects are not accurately aligned and the resulting reconstruction is
not physically plausible. Methods SMPL $\downarrow$ Object $\downarrow$ a.
neural recon. 14.06 15.34 b. weak persp. 9.14 25.24 c. direct persp. 7.63
15.58 d. w/o $f^{R}$ 6.58 17.54 e. w/o $f^{c}$ 6.26 14.25 f. w/o contact 5.73
11.96 g. full model 5.79 11.74 Table 3: Ablation studies. We can see that
neither weak perspective (b) nor direct perspective camera (c) can work well
on real data. Our proposed training strategy, together with orientation and
translation prediction achieves the best performance.

###  4.6 Other ablation studies

We further ablate our design choices in Tab. 3. Note that all numbers in Tab.
3 are Chamfer distances (cm) and computed on meshes after joint fitting except
a) which is computed on the neural reconstructed point clouds. We predict
object orientation and translation to initialize and regularize object
fitting, which are important for accurate reconstruction. In Tab. 3, it can be
seen that fitting results without orientation d) or without translation
initialization e) leads to worse perofrmance compared to our full model g).
Without initialization the object gets stuck in local minima. See Supp. for
qualitative examples.

We also compute the Chamfer distance of the neural reconstructed point clouds
(Eq. 2) using alignments computed from fitted meshes. It can be seen that this
result is much worse than the results after joint fitting (Tab. 3g). This is
because network predictions can be noisy in some regions while our joint
fitting incorporate priors of human and object making the model more robust.

##  5 Limitations and Future Work

We introduced CHORE, the first learning based model for single image 3D
reconstruction of humans, objects and contacts. Although this is a very
promising first step, there are some limitations in our current work. For
instance, we assume known object template for the objects shown in the images.
Future directions can remove this assumption by selecting template from 3D CAD
database or reconstructing shape directly from image. Non-rigid deformation,
multiple objects, multi-person interaction with objects are exciting avenues
for future research.

##  6 Conclusion

In this work we presented CHORE, the first method to _learn_ a _joint_
reconstruction of human, object and contacts from single images. CHORE
combines powerful neural implicit predictions for human, object and its pose
with robust model-based fitting. Unlike all prior work on pixel-aligned
implicit surface learning, we learn from real data instead of synthetic scans.
We have shown this requires a different training strategy, and we proposed a
simple and elegant depth-aware scaling that addresses the depth-scale
ambiguity while preserving pixel alignment.

Experiments on three different datasets demonstrate the superiority of CHORE
compared to the SOTA. Quantitative experiments on BEHAVE dataset evidence that
jointly learning reconstruction leads to a remarkable improvement of over
$50\%$ in terms of Chamfer distance compared to PHOSA. User studies on the
NTU-RGBD and the challenging COCO dataset (no ground truth) show that our
method is preferred over PHOSA by 84% and 72% users respectively. We also
conducted extensive ablations, which reveal the effectiveness of the different
components of CHORE, and the proposed depth-aware scaling for pixel-aligned
learning on real data.

To promote further research in this emerging field of single image 3D human-
object interaction capture, we will release our pre-trained models and code,
so that other researchers can use CHORE as a strong baseline.

#### Acknowledgements.

Special thanks to RVH group members [1] for their helpful discussions. This
work is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) - 409792180 (Emmy Noether Programme, project: Real Virtual
Humans), and German Federal Ministry of Education and Research (BMBF):
Tübingen AI Center, FKZ: 01IS18039A. Gerard Pons-Moll is a member of the
Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number
390727645.

## References

  * [1] http://virtualhumans.mpi-inf.mpg.de/people.html
  * [2] Alldieck, T., Magnor, M., Bhatnagar, B.L., Theobalt, C., Pons-Moll, G.: Learning to reconstruct people in clothing from a single RGB camera. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [3] Alldieck, T., Magnor, M., Xu, W., Theobalt, C., Pons-Moll, G.: Detailed human avatars from monocular video. In: International Conf. on 3D Vision (sep 2018)
  * [4] Alldieck, T., Magnor, M., Xu, W., Theobalt, C., Pons-Moll, G.: Video based reconstruction of 3D people models. In: IEEE Conf. on Computer Vision and Pattern Recognition (2018)
  * [5] Alldieck, T., Pons-Moll, G., Theobalt, C., Magnor, M.: Tex2shape: Detailed full human body geometry from a single image. In: IEEE International Conference on Computer Vision (ICCV). IEEE (oct 2019)
  * [6] Bhatnagar, B.L., Sminchisescu, C., Theobalt, C., Pons-Moll, G.: Combining implicit function learning and parametric models for 3d human reconstruction. In: European Conference on Computer Vision (ECCV). Springer (August 2020)
  * [7] Bhatnagar, B.L., Sminchisescu, C., Theobalt, C., Pons-Moll, G.: Loopreg: Self-supervised learning of implicit surface correspondences, pose and shape for 3d human mesh registration. In: Advances in Neural Information Processing Systems (NeurIPS) (December 2020)
  * [8] Bhatnagar, B.L., Tiwari, G., Theobalt, C., Pons-Moll, G.: Multi-garment net: Learning to dress 3d people from images. In: IEEE International Conference on Computer Vision (ICCV). IEEE (oct 2019)
  * [9] Bhatnagar, B.L., Xie, X., Petrov, I., Sminchisescu, C., Theobalt, C., Pons-Moll, G.: BEHAVE: Dataset and method for tracking human object interactions. In: CVPR (2022), https://virtualhumans.mpi-inf.mpg.de/behave
  * [10] Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.: Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In: European Conf. on Computer Vision. Springer International Publishing (2016)
  * [11] Brahmbhatt, S., Ham, C., Kemp, C.C., Hays, J.: ContactDB: Analyzing and predicting grasp contact via thermal imaging. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019), https://contactdb.cc.gatech.edu
  * [12] Brahmbhatt, S., Handa, A., Hays, J., Fox, D.: Contactgrasp: Functional multi-finger grasp synthesis from contact. In: IROS (04 2019)
  * [13] Brahmbhatt, S., Tang, C., Twigg, C.D., Kemp, C.C., Hays, J.: ContactPose: A dataset of grasps with object contact and hand pose. In: The European Conference on Computer Vision (ECCV) (August 2020)
  * [14] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: ShapeNet: An Information-Rich 3D Model Repository. Tech. Rep. arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago (2015)
  * [15] Chen, Y., Huang, S., Yuan, T., Qi, S., Zhu, Y., Zhu, S.C.: Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense. In: The IEEE International Conference on Computer Vision (ICCV) (2019)
  * [16] Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit functions in feature space for 3d shape reconstruction and completion. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (Jun 2020)
  * [17] Chibane, J., Mir, A., Pons-Moll, G.: Neural unsigned distance fields for implicit function learning. In: Neural Information Processing Systems (NeurIPS). (December 2020)
  * [18] Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S.: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. Springer International Publishing (2016)
  * [19] Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F., Rogez, G.: Ganhand: Predicting human grasp affordances in multi-object scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [20] Ehsani, K., Tulsiani, S., Gupta, S., Farhadi, A., Gupta, A.: Use the force, luke! learning to predict physical forces by simulating effects. In: CVPR (2020)
  * [21] Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017)
  * [22] Fieraru, M., Zanfir, M., Oneata, E., Popa, A., Olaru, V., Sminchisescu, C.: Learning complex 3d human self-contact. CoRR abs/2012.10366 (2020), https://arxiv.org/abs/2012.10366
  * [23] Fieraru, M., Zanfir, M., Oneata, E., Popa, A.I., Olaru, V., Sminchisescu, C.: Three-dimensional reconstruction of human interactions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [24] Fu, K., Peng, J., He, Q., Zhang, H.: Single image 3d object reconstruction based on deep learning: A review. Multimedia Tools and Applications 80(1), 463–498 (2021). https://doi.org/10.1007/s11042-020-09722-8, https://doi.org/10.1007/s11042-020-09722-8
  * [25] Guler, R.A., Kokkinos, I.: Holopose: Holistic 3d human reconstruction in-the-wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [26] Hassan, M., Choutas, V., Tzionas, D., Black, M.J.: Resolving 3d human pose ambiguities with 3d scene constraints. In: International Conference on Computer Vision (2019)
  * [27] Hassan, M., Ghosh, P., Tesch, J., Tzionas, D., Black, M.J.: Populating 3D scenes by learning human-scene interaction. In: Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) (Jun 2021)
  * [28] Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M.J., Laptev, I., Schmid, C.: Learning joint reconstruction of hands and manipulated objects. In: CVPR (2019)
  * [29] Huang, Z., Xu, Y., Lassner, C., Li, H., Tung, T.: Arch: Animatable reconstruction of clothed humans. ArXiv abs/2004.04572 (2020)
  * [30] Häne, C., Tulsiani, S., Malik, J.: Hierarchical surface prediction for 3d object reconstruction. In: 2017 International Conference on 3D Vision (3DV). pp. 412–420 (2017). https://doi.org/10.1109/3DV.2017.00054
  * [31] Jiang, B., Zhang, J., Hong, Y., Luo, J., Liu, L., Bao, H.: Bcnet: Learning body and cloth shape from a single image. In: European Conference on Computer Vision. Springer (2020)
  * [32] Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: IEEE Conf. on Computer Vision and Pattern Recognition. IEEE Computer Society (2018)
  * [33] Kar, A., Tulsiani, S., Malik, J.: Category-specific object reconstruction from a single image. In: IEEE Conference on Computer Vision and Pattern Recognition (2015)
  * [34] Karunratanakul, K., Yang, J., Zhang, Y., Black, M., Muandet, K., Tang, S.: Grasping field: Learning implicit representations for human grasps. In: 8th International Conference on 3D Vision. pp. 333–344. IEEE (Nov 2020). https://doi.org/10.1109/3DV50981.2020.00043
  * [35] Kocabas, M., Athanasiou, N., Black, M.J.: VIBE: Video inference for human body pose and shape estimation. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). pp. 5252–5262. IEEE (Jun 2020). https://doi.org/10.1109/CVPR42600.2020.00530
  * [36] Kolotouros, N., Pavlakos, G., Black, M.J., Daniilidis, K.: Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In: ICCV (2019)
  * [37] Lei, J., Sridhar, S., Guerrero, P., Sung, M., Mitra, N., Guibas, L.J.: Pix2surf: Learning parametric 3d surface models of objects from images. In: Proceedings of European Conference on Computer Vision (ECCV) (August 2020), https://geometry.stanford.edu/projects/pix2surf
  * [38] Li, Z., Sedlar, J., Carpentier, J., Laptev, I., Mansard, N., Sivic, J.: Estimating 3d motion and forces of person-object interactions from monocular video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [39] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014. pp. 740–755. Springer International Publishing, Cham (2014)
  * [40] Liu, J., Shahroudy, A., Perez, M., Wang, G., Duan, L.Y., Kot, A.C.: Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019). https://doi.org/10.1109/TPAMI.2019.2916873
  * [41] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: A skinned multi-person linear model. In: ACM Transactions on Graphics. ACM (2015)
  * [42] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [43] Monszpart, A., Guerrero, P., Ceylan, D., Yumer, E., J. Mitra, N.: iMapper: Interaction-guided scene mapping from monocular videos. ACM SIGGRAPH (july 2019)
  * [44] Müller, L., Osman, A.A.A., Tang, S., Huang, C.H.P., Black, M.J.: On self-contact and human pose. In: Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) (Jun 2021)
  * [45] MÃ¼ller, N., Wong, Y.S., Mitra, N.J., Dai, A., Nie{ss}ner, M.: Seeing behind objects for 3d multi-object tracking in rgb-d sequences. In: Proc. Computer Vision and Pattern Recognition (CVPR), IEEE (2021)
  * [46] Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., Schiele, B.: Neural body fitting: Unifying deep learning and model based human pose and shape estimation. In: International Conf. on 3D Vision (2018)
  * [47] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from a single image. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [48] Pavlakos, G., Zhu, L., Zhou, X., Daniilidis, K.: Learning to estimate 3D human pose and shape from a single color image. In: IEEE Conf. on Computer Vision and Pattern Recognition (2018)
  * [49] Pontes, J.K., Kong, C., Sridharan, S., Lucey, S., Eriksson, A., Fookes, C.: Image2mesh: A learning framework for single image 3d reconstruction. In: ACCV. Springer International Publishing (2019)
  * [50] Rempe, D., Guibas, L.J., Hertzmann, A., Russell, B., Villegas, R., Yang, J.: Contact and human dynamics from monocular video. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
  * [51] Rong, Y., Shiratori, T., Joo, H.: Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration. In: IEEE International Conference on Computer Vision Workshops (2021)
  * [52] Saito, S., , Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: IEEE International Conference on Computer Vision (ICCV). IEEE (oct 2019)
  * [53] Saito, S., Simon, T., Saragih, J., Joo, H.: Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (June 2020)
  * [54] Savva, M., Chang, A.X., Hanrahan, P., Fisher, M., Nießner, M.: PiGraphs: Learning Interaction Snapshots from Observations. ACM Transactions on Graphics (TOG) 35(4) (2016)
  * [55] Sun, X., Wu, J., Zhang, X., Zhang, Z., Zhang, C., Xue, T., Tenenbaum, J.B., Freeman, W.T.: Pix3d: Dataset and methods for single-image 3d shape modeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [56] Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: GRAB: A dataset of whole-body human grasping of objects. In: European Conference on Computer Vision (ECCV) (2020), https://grab.is.tue.mpg.de
  * [57] Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2mesh: Generating 3d mesh models from single rgb images. In: Proceedings of the European Conference on Computer Vision (ECCV) (September 2018)
  * [58] Weng, Z., Yeung, S.: Holistic 3d human and scene mesh estimation from single view images. arXiv preprint arXiv:2012.01591 (2020)
  * [59] Wu, J., Wang, Y., Xue, T., Sun, X., Freeman, W.T., Tenenbaum, J.B.: MarrNet: 3D Shape Reconstruction via 2.5D Sketches. In: Advances In Neural Information Processing Systems (2017)
  * [60] Wu, J., Zhang, C., Zhang, X., Zhang, Z., Freeman, W.T., Tenenbaum, J.B.: Learning 3D Shape Priors for Shape Completion and Reconstruction. In: European Conference on Computer Vision (ECCV) (2018)
  * [61] Xiang, Y., Kim, W., Chen, W., Ji, J., Choy, C., Su, H., Mottaghi, R., Guibas, L., Savarese, S.: Objectnet3d: A large scale database for 3d object recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. Springer International Publishing (2016)
  * [62] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper/2019/file/39059724f73a9969845dfe4146c5660e-Paper.pdf
  * [63] Yang, L., Zhan, X., Li, K., Xu, W., Li, J., Lu, C.: CPF: Learning a contact potential field to model the hand-object interaction. In: ICCV (2021)
  * [64] Zhang, J.Y., Pepose, S., Joo, H., Ramanan, D., Malik, J., Kanazawa, A.: Perceiving 3d human-object spatial arrangements from a single image in the wild. In: European Conference on Computer Vision (ECCV) (2020)
  * [65] Zhang, S., Liu, J., Liu, Y., Ling, N.: Dimnet: Dense implicit function network for 3d human body reconstruction. Computers & Graphics 98, 1–10 (2021). https://doi.org/https://doi.org/10.1016/j.cag.2021.04.035
  * [66] Zhang, S., Zhang, Y., Ma, Q., Black, M.J., Tang, S.: PLACE: Proximity learning of articulation and contact in 3D environments. In: International Conference on 3D Vision (3DV) (Nov 2020)
  * [67] Zhang, X., Zhang, Z., Zhang, C., Tenenbaum, J.B., Freeman, W.T., Wu, J.: Learning to Reconstruct Shapes From Unseen Classes. In: Advances in Neural Information Processing Systems (NeurIPS) (2018)
  * [68] Zhao, F., Wang, W., Liao, S., Shao, L.: Learning anchored unsigned distance functions with gradient direction alignment for single-view garment reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12674–12683 (2021)
  * [69] Zhe, C., Hang, G., Karttikeya, M., Qizhi, C., Minh, V., Jitendra, M.: Long-term human motion prediction with scene context. In: ECCV (2020)