# CA2T-Net: Category-Agnostic 3D Articulation Transfer from Single Image

Jasmine Collins1 Anqi Liang2 Jitendra Malik1 Hao Zhang2 Frédéric Devernay2
1 UC Berkeley 2 Amazon

###### Abstract

We present a neural network approach to transfer the motion from a single
image of an articulated object to a rest-state (i.e., unarticulated) 3D model.
Our network learns to predict the object’s pose, part segmentation, and
corresponding motion parameters to reproduce the articulation shown in the
input image. The network is composed of three distinct branches that take a
shared joint image-shape embedding and is trained end-to-end. Unlike previous
methods, our approach is independent of the topology of the object and can
work with objects from arbitrary categories. Our method, trained with only
synthetic data, can be used to automatically animate a mesh, infer motion from
real images, and transfer articulation to functionally similar but
geometrically distinct 3D models at test time.

##  1 Introduction

We live in a 3D world where interacting with objects in our environment is
necessary to carry out daily activities. Therefore, the objects around us have
been designed with the appropriate structures and part motions to afford
various actions to both humans and robotic agents. Whether for general visual
perception tasks such as functionality inference or robotic action planning,
it is a valuable skill to be able to predict the part motions or articulations
of everyday objects. Recently, there has been much interest in training deep
neural networks to learn such predictions, especially in the context of
embodied AI [27, 18]. However, the availability of 3D models with associated
part articulations [24, 27] for training these methods is still limited.

Figure 1: Predicting pose, part, and motion annotations for synthetic meshes
given an exemplar image. At test time, we can perform 3D animation, transfer
motion to functionally similar objects, and generalize to real {image, mesh}
pairs.

Creating articulated 3D models relies on human effort which can be expensive
in terms of time, cost and expertise. As a result, most 3D model datasets do
not contain objects with any kind of functional part and motion annotations.
For example, Amazon Berkeley Objects (ABO) [3] is a recently published, large-
scale (8K models) dataset of artist-created 3D models of real household
objects. While most of the 3D models in ABO represent objects that can be
interacted with, such as desks with openable drawers or chairs that recline,
the 3D models have all been constructed in their “rest” (i.e., unarticulated)
poses that do not expose their real functionality. The same is true for the
vast majority of 3D models in well-established repositories such as ShapeNet
[2] (3M models) and PartNet [19] (27K models) The largest 3D datasets with
part articulations, SAPIEN [27] and Shape2Motion [24], only contain 2K
manually annotated synthetic models.

Method | Task | Input | Pose | 3D Part Seg. | Category-Specific
---|---|---|---|---|---
DeepPartInduction [32] | Transfer articulation from PC $\rightarrow$ PC | 2 PCs | $\times$ | $\checkmark$ | No
Shape2Motion [24] | Predict motion for all moveable parts | PC | $\times$ | $\checkmark$ | No
RPM-Net [30] | Predict motion for all moveable parts | PC | $\times$ | $\checkmark$ | No
ANCSH [15] | Articulated pose estimation | PC | $\checkmark$ | $\checkmark$ | Yes
NASAM [25] | Reconstruct and animate objects | RGBs + pose | $\times$ | $\checkmark$ | Yes
OPD [9] | Predict motion for all openable parts | RGB | $\checkmark$ | $\times$ | No
CA2T-Net (ours) | Transfer articulation from image $\rightarrow$ mesh | RGB, PC | $\checkmark$ | $\checkmark$ | No
Table 1: CA2T-Net addresses the problem of single-view articulation transfer,
from an image to a mesh. Existing work targets related problems, but differs
in terms of task and inputs of the model.

In this paper, we introduce single-view 3D articulation transfer, a learning-
based approach aimed at endowing the many existing 3D models, in particular,
those of real products such as ABO, with part articulations. Specifically,
given a single RGB image, $I$, showing how an object can be articulated, and a
3D mesh model $M$ in its rest state, our goal is to infer the pose, motion
parameters, and 3D part segmentation of $M$ such that we can re-pose and
transform it to match the articulation in the input image. Our method is
structure-agnostic, as it makes no assumption on the topology of the input
model. Unlike many existing methods for articulated pose estimation, our
approach is also category-agnostic, meaning no further training is necessary
to run our model on objects from novel categories and it can make predictions
for objects from arbitrary categories. Further, the object represented in the
input image $I$ and the mesh $M$ need not be exactly the same, but should
simply share a functional similarity to allow a plausible articulation
transfer. To our knowledge, our work is the first to pose the problem of
single-view articulation transfer onto a target 3D object. As indicated in
Table 1, the most relevant works [32, 24, 15, 9, 30] differ from our problem
setting in one way or another.

We call our network CA2T-Net for category-agnostic articulation transfer, and
demonstrate that our method has superior performance in single-view 3D
articulation transfer compared to baseline methods and the current state-of-
the-art in terms of motion prediction. Our approach takes as input an
articulated image and rest-state 3D point cloud and uses three branches to
predict global pose, part segmentation, and 3D motion parameters. The network
is trained end-to-end using synthetic data. We also show three downstream
applications of our trained approach: automatic mesh animation, generalization
to functionally similar objects, and transfer to real objects and images in
ABO.

Our work is a first step towards replacing manual part and motion annotations
[24, 27] with a fully automated inference of part articulations and motion
parameters of arbitrary 3D shapes. With the wide availability of images
depicting object articulations (e.g., catalog images found in the product
metadata within ABO), our image-to-3D transfer presents a promising step
towards articulated 3D shape creation at scale.

##  2 Related Work

#### Part Segmentation

In computer vision, most research on 2D or 3D part segmentation focuses on
semantic [33, 19] or functional [10, 7, 24, 9] meanings of the parts. Jiang et
al. [9] extended 2D semantic segmentation networks (e.g. Mask R-CNN) to
predict the location of openable parts from a single-view image. Gelfand et
al. [4] defined a part segmentation as a kinematic surface composed of points
that undergo the same type of motion. Other work has relied on a category-
level canonical container or a fixed kinematic structure. For example, follow-
up work [15, 30] learned a fully supervised canonicalization to map instance
observations of different position, orientation, scale and articulation into a
canonical container. Xu et al. [29] estimated the part segmentation from a
depth image in a known category of a fixed kinematic chain. In contrast, our
approach requires neither semantic annotation nor category-level functional
information or kinematic structure. Further, our work is not about learning
semantic or functional part segmentation, but rather image-guided part
segmentation. Our method finds the correspondence between an input image and a
3D shape under different articulation configurations.

Figure 2: Diagram of training architecture. CA2T-Net takes in an RGB image and
3D model and predicts the global pose of the object in the image, segmentation
of the part that is articulated in the image, as well as part-specific motion
parameters. Each motion attribute is predicted from a spearate head.

#### Articulated Pose Estimation

Articulated pose estimation from a 2D image is another focus of our work.
Recently, Wang et al. [23] introduced the Normalized Object Coordinate Space
(NOCS), allowing category-level rigid pose estimation. Follow-ups [15, 16]
have used this representation to perform articulated pose estimation. For
example, Li et al. [15] proposed a part-level canonical reference frame and
performed part pose fitting using RANSAC. Liu et al. [16] extended NOCS to
Real-World Articulation NOCS which focuses on pose estimation for articulated
objects with varied kinematic structures in real-world settings. Kulkarni et
al., [11] introduced an approach for articulating a template mesh given a
target image segmentation mask from a category-specific image collection, but
requires hand-annotated part segmentations. Other work [30, 26, 31] relies on
a series of frames displaying an object actively articulating. In contrast,
our method does not require learning the canonical representation with a fixed
kinematic structure for each category. Without the need of sequences of images
of an object’s different articulation states as inputs, we require only an
articulated single-view image and its unarticulated 3D model to perform
articulated pose estimation.

#### Motion Estimation & Transfer

In 3D understanding, considerable work has focused on the problem of motion
prediction. Methods largely differ in the types of input(s) used for motion
estimation and transfer. Initial work [4, 17, 30] utilized only the geometry
of an object to infer part segmentation and motion. Li et al. [14] focused on
predicting part mobility of an object from a sequence of scans displaying the
the dynamic motion of articulated models. More recently, motion estimation has
been performed from depth images [1, 15]. Mo et al., [18] learned to make per-
point motion predictions from a single RGB or depth image by interacting with
articulated objects in simulation. Other approaches use a sequence of images
or video clips to infer 3D part movement [6, 32, 8]. Wei et al., [25] trained
an implicit representation of the geometry, appearance, and motion of an
object in a self-supervised manner from image observations, however their
approach is limited to certain classes of articulated objects. Recently, Jiang
et al., [9] introduced a method for estimating 3D motion parameters from
single image. Their approach aims to predict motion parameters for all
“openable” parts, rather than focusing on transferring arbitrary motion from
one object to another.

Our work makes no assumptions on the input category and can both estimate and
transfer 3D motion. We formulate the problem as learning the articulated
segmentation of the input geometry, estimating the pose and predicting motion
parameters of the moveable part (driven by the input image). Unlike many
existing approaches, our work is both category-agnostic and structure
agnostic.

##  3 Method

We propose a model for the single-view articulation transfer task. Given an
articulated RGB image and corresponding 3D model, our method infers the pose,
part segmentation and motion parameters that can be used to deform the mesh to
match the input image.

###  3.1 Architecture

Our architecture is composed of 3 distinct branches: a pose prediction branch,
a motion prediction branch, and a 3D part segmentation branch. Our model takes
two inputs: an image $I\in\mathbb{R}^{h\times w\times 3}$ and point cloud
shape $S\in\mathbb{R}^{n\times 3}$, where $n$ is the number of points in a
point cloud sampled from the normalized input mesh. Image features,
$\textbf{h}=f_{\theta}(I)$, are extracted with a ResNet-18 [22], and shape
features, $\textbf{s}=g_{\theta}(S)$, are extracted with a PointNet-like [21]
architecture, without input and feature transforms for pose-invariance. Both
the image and shape encoder are trained from scratch. The shape and image
features are concatenated and passed through a set of fully connected layers
resulting in a joint image and shape embedding,
$\textbf{z}\in\mathbb{R}^{b\times 512}$. Each of the three branches take z as
input. An overview of the full architecture can be found in Figure 2.

Figure 3: Qualitative results on SAPIEN validation set. Our method’s
predictions can be used to deform the input mesh to match the articulated
image.

#### Pose Prediction

The pose prediction branch of our architecture is inspired by [28], a method
that also takes as input an RGB image (however unarticulated) and a 3D shape,
and predicts the pose of the 3D model in the image. As in [28] we predict pose
in terms of azimuth, elevation, and in-plane rotation. The network first
classifies the angle bin and the regresses an offset. Let $\hat{l}_{i,j}$ be
the predicted bin for the $j$-th Euler angle (azimuth, elevation, in-plane
rotation) of the $i$-th datapoint, and $\hat{\delta}_{i,j}$ be the predicted
relative offset within the bins. For bin prediction, we have 24 azimuth
classes, 12 elevation classes, and 24 in-plane rotation classes. We train the
model with a cross-entropy loss ($\mathcal{L}_{CE}$) for bin classification
and Huber loss ($\mathcal{L}_{huber}$) for regression offsets.

$\mathcal{L}_{p}=\sum_{i=1}^{N}\sum_{j\in\mathcal{E}}\mathcal{L}_{CE}(l_{i,j},\hat{l}_{i,j})+\mathcal{L}_{huber}(\hat{\delta}_{i,j},\delta_{i,j}),$
(1)

where $\mathcal{E}=\\{\text{azimuth},\text{elevation},\text{in-plane}\\}$ and
$l_{i,j}$ and $\delta_{i,j}$ are the ground truth pose bins and offsets,
accordingly.

#### Motion Estimation

In this work we consider two distinct motion types, revolute (rotation) and
prismatic (translation). We parameterize the revolute motion in terms of
motion axis, origin, and magnitude, and prismatic in terms of motion axis and
magnitude. Each motion attribute is predicted by a separate head, for origin,
axis, magnitude, and class. Let $\mathcal{L}_{mc}$ be the classification loss
on motion type, defined as:

$\mathcal{L}_{mc}=\sum_{i=1}^{N}\mathcal{L}_{CE}(m_{i},\hat{m}_{i}),$ (2)

where $m_{i}$ is the ground-truth motion class represented as a one-hot vector
and $\hat{m}_{i}$ is the corresponding predicted class.

Motion axis $\hat{a}\in\mathbb{R}^{3}$, origin $\hat{o}\in\mathbb{R}^{3}$, and
magnitude $\hat{z}\in\mathbb{R}$ are regressed and optimized using Huber loss:

$\begin{split}\mathcal{L}_{m}=\sum_{i=1}^{N}\mathcal{L}_{huber}(a_{i},\hat{a}_{i})+\mathcal{L}_{huber}(o_{i},\hat{o}_{i})+\\\
\mathcal{L}_{huber}(z_{i},\hat{z}_{i}).\end{split}$ (3)

We predict motion parameters in the world coordinate space (i.e., object’s
canonical coordinate frame). Motion origin and magnitude for prismatic motion
is given in terms of normalized object size, and magnitude for revolute motion
is given in terms of radians. As motion origin is not relevant for prismatic
joints, we only apply the motion origin loss,
$\mathcal{L}_{huber}(o_{i},\hat{o}_{i})$, in the cases of predicted revolute
motion.

|  | Motion | 3D
---|---|---|---
I-split |  |  |  |  |  |  |  |  |  |
O-split |  | Pose ($\uparrow$) | Type ($\uparrow$) | Axis ($\downarrow$) | Origin ($\downarrow$) | Mag-R ($\downarrow$) | Mag-P ($\downarrow$) | Seg. ($\uparrow$) | Chamf. ($\downarrow$) |<EMAIL_ADDRESS>($\uparrow$)
RandMot | 12.6% | 56.6% | 78.8∘ | 1.09 | 62.0∘ | 0.746 | 54.5% | 5.18 | 7.50
FreqMot | 13.0% | 69.0% | 66.2∘ | 0.689 | 51.7∘ | 0.335 | 78.1% | 3.51 | 9.09
CA2T-Net | 89.0% | 99.2% | 6.23∘ | 0.274 | 36.7∘ | 0.166 | 92.2% | 1.05 | 42.5
RandMot | 13.6% | 58.6% | 79.1∘ | 1.13 | 63.2∘ | 0.734 | 54.5% | 4.79 | 7.57
FreqMot | 13.2% | 74.4% | 67.3∘ | 0.792 | 54.7∘ | 0.333 | 76.6% | 3.38 | 8.84
CA2T-Net | 74.0% | 91.5% | 58.7∘ | 0.605 | 51.9∘ | 0.230 | 81.8% | 1.54 | 33.3
Table 2: Quantitative performance of baselines and CA2T-Net (ours) on SAPIEN
validation split Results for per-image (I-split) and per-object (O-split)
dataset splits.

#### 3D Part Segmentation

Our 3D part segmentation branch takes in the transformed global shape and
image features, z, as well as a 3D point, $x\in\mathbb{R}^{3}$, and predicts
whether or not that point (from the pointcloud) corresponds to the moveable
part. Our architecture is inspired by DeepSDF [20] which also takes as input a
set of features and a 3D point. After processing these inputs with 2 fully
connected layers, the initial inputs are again concatenated with the
intermediate features and further processed by 3 more fully connected layers.
Let $\hat{p}$ be the predicted confidence of a certain point $x$’s movability
- we define the 3D part segmentation loss as:

$\mathcal{L}_{s}=\sum_{i=1}^{N}\sum_{j=1}^{M}\mathcal{L}_{CE}(p_{i,j},\hat{p}_{i,j}),$
(4)

where $i$ indexes over training datapoints and $j$ over the $M$ sampled 3D
points from the input shape. For training we use $M=1000$.

#### Full Training Objective

Our full training loss is a weighted combination of losses applied to each of
the three branches:

$\mathcal{L}=\lambda_{p}\mathcal{L}_{p}+\lambda_{mc}\mathcal{L}_{mc}+\lambda_{m}\mathcal{L}_{m}+\lambda_{s}L_{s}.$
(5)

The coefficients on the loss terms are $\lambda_{p}=2$, $\lambda_{mc}=1$,
$\lambda_{m}=8$, and $\lambda_{s}=1$. We train the network using the Adam
optimizer with a learning rate of 1e-3 and batch size of 128 for 50 epochs and
decrease the learning rate to 1e-4 for the final 10 epochs.

#### Inference

During inference, we randomly sample 100 points on each distinct part of the
input model and compute the network’s 3D part prediction for each point. We
apply the network’s predicted motion only to parts that have $>$50% points
considered “moveable”.

##  4 Evaluation and Results

We train and evaluate our method using images and ground-truth data generated
from the SAPIEN PartNet-Mobility [27] dataset.

|  | Motion |
---|---|---|---
| Pose ($\uparrow$) | Type ($\uparrow$) | Axis ($\downarrow$) | Origin ($\downarrow$) | Mag-R ($\downarrow$) | Mag-P ($\downarrow$) | Seg ($\uparrow$)
Pose Only | 69.6% | - | - | - | - | - | -
Motion Only | - | 92.5% | 52.2∘ | 0.599 | 52.3∘ | 0.26 | -
Seg Only | - | - | - | - | - | - | 78.9%
All | 74.0 % | 91.5% | 58.7∘ | 0.605 | 51.9∘ | 0.23 | 81.8%
Table 3: Training for individual objectives vs multi-task learning. We train
three separate networks for pose, motion and segmentation only and compare
their performance to the proposed multi-task objective. We find that
optimizing all losses together yields the best results.

|  Motion |  |
---|---|---|---
| Type ($\uparrow$) | Axis ($\downarrow$) | Origin ($\downarrow$) | mAP ($\uparrow$)
OPD-O [9] | 94.0% | 72.5∘ | 0.912 | 41.0%
CA2T-Net | 98.9% | 50.5∘ | 0.757 | 49.2%

Table 4: Comparison to OPD for 3D motion prediction and 2D segmentation. We
compare to OPD, a method that estimates motion parameters for all movable
parts given an RGB image.

For segmentation, we report<EMAIL_ADDRESS>
###  4.1 Dataset

The original SAPIEN PartNet-Mobility dataset consists of 2,346 CAD models from
46 categories with part segmentations and motion annotations. As we are
interested in articulation transfer from a single-image, we focus on objects
with large moveable parts such as the door or drawer of a cabinet, rather than
the individual keys on a keyboard. To do so, we filter out CAD models that
only contain moveable parts that consist of $<$5% of the overall surface area
of the object, as well as objects with missing or incomplete motion data.
After this filtering process, 1,717 objects remain.

From the remaining objects, we render 256 images per object using the SAPIEN
Vulkan Renderer, each with a random pose, randomly sampled part, and randomly
sampled motion parameters within the valid range. For simplicity, we only
consider one moving part at a time. The pose distribution ranges from azimuth
$\in[-90^{\circ},90^{\circ}]$, elevation $\in[-45^{\circ},45^{\circ}]$, and
in-plane rotation $\in[-20^{\circ},20^{\circ}]$. Such ranges were chosen so
that the articulated part on the 3D model is usually visible (most moveable
parts are on the front of the object, rather than the back). For each image,
we also randomly sample the camera’s field-of-view
$\in[20^{\circ},60^{\circ}]$. In total we render approximately 280K images for
training, and 10K images for evaluation.

For each image we also write out the rest-state mesh, the ground truth pose of
the object, 3D part segmentation (of the articulated part from the rest of the
object), and motion parameters. The motion parameters consist of a motion type
$\in\\{revolute,prismatic\\}$, axis, origin, and magnitude. In the case of
revolute motion, the magnitude corresponds to a rotation amount, whereas it
corresponds to translation distance for prismatic motion. From the rest-state
mesh we sample a point cloud that is the input to our network. For training,
we resize images to 224 $\times$ 224 and sample a pointcloud of 2500 points
for each mesh. Pointclouds are further normalized to fit in a unit box.

###  4.2 Baselines

We consider three baseline approaches to single-view articulation transfer:
selecting a random motion and part (RandMot), selecting the most frequent
motion parameters (FreqMot), and OPD-O [9], the current state-of-the-art
approach for 3D motion prediction. RandMot and FreqMot are inspired by similar
baselines used by OPD [9], however modified to predict additional outputs such
as object pose and motion magnitude. RandMot gives a lower bound on
performance whereas FreqMot is a relatively strong heuristic-based baseline
approach.

#### RandMot

For the random motion (RandMot) baseline, we sample a random set of ground
truth labels from the training dataset, including random pose, motion axis,
motion origin, type, and magnitude. For the predicted part segmentation, we
randomly sample a valid part from the set of parts contained in the input
mesh.

#### FreqMot

In contrast to the RandMot baseline, the frequent motion (FreqMot) baseline
method predicts the most frequent motion parameters from the train dataset. We
still randomly sample a global pose, but sample the predicted part
segmentation based on statistics from the training dataset. We compute the
average size of the most frequently sampled part in the training set and
select the part that is most similar in size at inference time. For the
remaining categorical variables, we sample the most frequent value from the
training set. For real-valued parameters such as motion magnitude, motion
axis, and motion origin, we first cluster the values from the training set and
sample the mean value of the most frequent cluster.

#### Openable Part Detection [9]

While designed for a different problem formulation, Openable Part Detection
(OPD) is the most applicable to our articulation transfer task in terms of
model output. OPD takes a single RGB image and predicts 2D segmentation masks
and 3D motion parameters for all parts identified as “openable”. While their
method cannot be directly used to animate a 3D model, we can still compare to
them in terms of 3D motion prediction and 2D part segmentation. Specifically,
we compare our method to OPD-O, the best-performing variant of OPD in terms of
real image performance. OPD-O also predicts motion parameters in the world
coordinate system by using an additional branch to predict extrinsic
parameters.

###  4.3 Metrics

We consider evaluation metrics for pose, motion, and segmentation performance,
as well as 3D reconstruction metrics on the final deformed mesh. Our Pose
metric is the percentage of pose estimations with rotation error less than
30∘. Motion-Type refers to the 2-way classification accuracy of revolute or
prismatic motion. Motion-Axis is the average angular error (in degrees)
between the normalized predicted and ground-truth motion axes, and Motion-
Origin is the average $L_{1}$ distance between the predicted and ground-truth
motion origin. Mag-R and Mag-P refer to the revolute and prismatic magnitude
errors, respectively, measured in degrees for the case of revolute motion and
$L_{1}$ distance for prismatic motion. Because motion origin is not applicable
for prismatic motion, we only evaluate motion origin for predicted revolute
parts. Seg. refers to the 3D segmentation accuracy, measured across 1000
sampled points. Finally for 3D reconstruction metrics, we report Chamfer
distance and<EMAIL_ADDRESS>
###  4.4 Experimental Results

#### Performance on Validation Set

We report the performance of our model trained on the SAPIEN dataset in Table
2. As no neural network-based approach currently exists for the full single-
view articulation transfer task, we compare against our two heuristic-based
baselines: RandMot and FreqMot. We consider two train/validation splits of
varying difficult, I-split (per-image: validation set contains unseen images)
and O-split (per-object: validation set contains unseen images of novel
objects).

Compared to the RandMot and FreqMot baselines, we find that our method
achieves the best performance across all metrics. Understandably, the
validation performance is better on the I-split than the O-split.
Visualizations of each method’s predictions can be found in Figure 3.

#### Comparison to OPD

We compare our approach to the recently introduced OPD method [9], which
represents the state-of-the-art in terms of 3D motion estimation. OPD predicts
2D segmentation masks, motion type, and 3D motion parameters (axis, origin)
for all moveable parts (based on priors in the training dataset) given a
single image, whereas CA2T-Net takes a 3D model and a driving input image, and
predicts 3D segmentation, motion type, and 3D motion parameters (axis, origin,
and magnitude). To compare the two approaches on a somewhat level playing
field, we evaluate the performance of CA2T-Net and OPD on the motion parameter
predictions that the methods share in common. Because OPD makes multiple
predictions for all moveable parts in an image but in our problem setting we
are only interested in the part that has actually moved in the image, we take
only the OPD predictions that have $\geq$ 0.5 IOU with the ground-truth
segmented part. Further, as OPD is limited to predicting only openable part
motion, we only consider datapoints that come from the subset of the 11
categories that OPD was trained on. This reduces the size of our valiation set
from 10,000 images to 1,039 images. We utilize the CA2T-Net model trained on
the more difficult per-object training split (O-split).

Results can be found in Table 4. We also compare to OPD in terms of 2D
segmentation ability, by projecting the CA2T-Net -predicted 3D part
segmentation to 2D and measuring mAP at IoU = 0.5. In general we find that
CA2T-Net outperforms OPD in terms of motion type, origin, and axis prediction,
as well as 2D segmentation ability. Intuitively, learning 3D features from the
input model in CA2T-Net should support better prediction of 3D motion
parameters. A common failure case that we observe with OPD is predicting a
motion axis that is 180 degrees rotated from the correct axis.

Multi-Task vs Single-Branch Learning We study the contribution of each of the
loss terms corresponding to each branch of our architecture by training three
variants of our model: pose prediction only, motion prediction only, and 3D
segmentation prediction only. The results can be found in Table 3. We find
that the multi-task objective with all three loss terms leads to the best
performance across most metrics, specifically for pose and 3D segmentation
estimation. The performance gains for motion estimation are less pronounced.
Intuitively, accurate pose prediction should aid the alignment between the
image and shape inputs, which should improve part segmentation ability. Such
an effect may explain the boost in performance by doing multi-task training.

Failure Cases The most common failure cases of our method include pose
prediction failure (e.g., predicting a pose that is 180-rotated from the
ground-truth pose) as well as failure to segment any part of the object when
the motion magnitude is small. Visualizations of these failure cases can be
found in Figure 4.

Figure 4: Typical failure cases in articulation transfer. Pose prediction
failures (180 degrees off, left column), and failure to predict a segmented
part when motion magnitude in image is small (right column). Figure 5: Motion
analogies. By presenting a functionally similar but geometrically distinct 3D
model at test time, we can transfer motion across analogous 3D parts.

##  5 Applications

We present three exciting applications of our model: generalization to
functionally similar objects (i.e., motion analogies), 3D model animation, and
transfer to real images and meshes in ABO.

###  5.1 Motion Analogies

During training, our method takes as input a 3D model that exactly matches the
geometry of the object pictured in the 2D image. However, at test time we can
swap the 3D input with a functionally similar 3D model that does not exactly
match the input image to perform a motion analogy. To achieve this, we swap
the input model with a different model from the same category as the original
input 3D model, and run our method with no additional modifications. The
articulated outputs predicted by our method can be seen in Figure 5. We note
that in this setting our model has to not only segment and predict motion for
the corresponding articulated part on the swapped shape, it also has to
accurately predict the pose in this setting that was never seen during
training. In general we find that our method can segment the functionally
similar part and produce motion parameters that lead to a plausible
deformation of the non-matching input model, even when the input model is
drastically different from the articulated image (see Figure 5 left column,
third row).

###  5.2 Animation: Interpolation and Extrapolation

A convenience of our motion parameterization is that animation of the input 3D
model can be achieved simply by varying the predicted motion magnitude. As a
result, the predicted segmented part will move along or around the predicted
axis. We can interpolate between the rest-state magnitude (0) and predicted
magnitude to simulate the motion of a part opening and closing, or even
consider magnitude settings beyond the predicted value to get motion
extrapolation. See Figure 6 for an animation example.

Figure 6: Interpolation and extrapolation in 3D from a single input image. Our
motion parameterization can easily be used to interpolate between the rest-
state and articulated object pose, or even extrapolate beyond the motion that
is seen in the input image.

###  5.3 Transfer to Real Objects & Images

A natural source of real-world articulated objects, including rest-state mesh
and image pairs, comes from the ABO dataset [3]. The dataset contains catalog
images of Amazon products, as well as a subset of 3D models corresponding to
the object in the images. The catalog images range from stock-like photos to
close-up detail shots, and also sometimes contain an articulated picture of
the object. We gathered a subset of approximately 50 catalog images that show
articulated objects (with a single moving part) for which there also exist
corresponding 3D models as a real-world use-case of the single-view
articulation transfer setup.

Unlike the training objects in the SAPIEN dataset that have annotated part
segmentations, meshes from the ABO dataset are unsegmented. Further,
articulated image instances in ABO have no ground-truth in terms of pose, part
segmentation, and motion annotations. To achieve a candidate part segmentation
we leverage the fact that the meshes in ABO happen to be designed by artists,
part by part, and thus different parts of the object tend to be represented by
mesh pieces that are disconnected from the rest of the object. As a result,
running connected components on the ABO meshes tends to give us an over-
segmentation of the object’s functionally relevant parts. We treat these
connected components as pseudo-part segmentations and can then run CA2T-Net on
this data with no modifications.

Qualitative results showing our model applied, without any finetuning, to real
images from the ABO dataset can be seen in Figure 7. Note that the meshes in
ABO were designed to simply show the product in its rest-state, rather than
with articulation in mind (like in SAPIEN). This is shown by the fact that
drawers are simply external panels rather than a full drawer part enclosed by
the rest of the object. Further, our model was trained only with inputs that
show a single articulated part, whereas realistic catalog images typically
show multiple articulated parts at once. We include a challenging instance of
this in Figure 7 (row 3), where the input image contains a recliner chair,
which is a category not seen during training, with multiple complex motions,
including a reclined headrest and raised footrest. As our method is category-
agnostic, it can handle inputs like this at test-time.

Figure 7: Transfer results to real articulated objects. We run our model on
instances of ABO objects with no finetuning and are able to make reasonable
predictions despite the domain gap.

##  6 Conclusion and Future Work

We proposed a method to transfer the articulation from a single-view image to
a 3D model in such a way that the model can be deformed to match the input
image. To our knowledge, we are the first to solve this task in the setting of
household objects from arbitrary categories. In this work we only consider
articulated exemplar images with a single moving part displayed. Of course,
articulated images can commonly show multiple moving parts. Extending our
method to predict multiple motion types and part segmentations is a practical
next step and can be achieved by combining Mask RCNN-style detections (as used
in OPD) with our method. Further, our method requires a 3D model as input
which can limit its practical use in say, an embodied perception system
navigating the world. Recent advances in CAD model retrieval from a single
image [12, 13, 5] can be used to retrieve a similar 3D CAD model, and our
results on functionally similar shape swapping suggest that our approach will
be able to handle this case. Thus combining CA2T-Net with CAD retrieval
methods could be a promising path forward to allowing our method to run on
single image inputs only. Finally, we consider only rigid motion (further,
just prismatic and revolute motion) in this work, but considering how this
approach can be extended to more complicated motion types as well as non-rigid
deformation is an interesting avenue for future work.

## References

  * [1] Ben Abbatematteo, Stefanie Tellex, and George Konidaris.  Learning to generalize kinematic models to novel objects.  In Proceedings of the 3rd Conference on Robot Learning, 2019.
  * [2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An information-rich 3D model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [3] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik.  ABO: dataset and benchmarks for real-world 3D object understanding.  In CVPR, pages 21126–21136, 2022.
  * [4] Natasha Gelfand and Leonidas J Guibas.  Shape segmentation using local slippage analysis.  In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, pages 214–223, 2004.
  * [5] Can Gümeli, Angela Dai, and Matthias Nießner.  Roca: Robust CAD model retrieval and alignment from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4022–4031, 2022.
  * [6] Ruizhen Hu, Wenchao Li, Oliver Van Kaick, Ariel Shamir, Hao Zhang, and Hui Huang.  Learning to predict part mobility from a single static snapshot.  ACM Trans. Graph. (SIGGRAPH Asia), 36(6):Article 227, 2017.
  * [7] Ruizhen Hu, Manolis Savva, and Oliver van Kaick.  Functionality representations and applications for shape analysis.  In Computer Graphics Forum, volume 37, pages 603–624. Wiley Online Library, 2018.
  * [8] Ajinkya Jain, Rudolf Lioutikov, Caleb Chuck, and Scott Niekum.  ScrewNet: Category-independent articulation model estimation from depth images using screw theory.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13670–13677. IEEE, 2021.
  * [9] Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang.  OPD: Single-view 3D openable part detection.  In ECCV, 2022.
  * [10] Vladimir G Kim, Siddhartha Chaudhuri, Leonidas Guibas, and Thomas Funkhouser.  Shape2Pose: Human-centric shape analysis.  ACM Transactions on Graphics (TOG), 33(4):1–12, 2014.
  * [11] Nilesh Kulkarni, Abhinav Gupta, David F. Fouhey, and Shubham Tulsiani.  Articulation-aware canonical surface mapping.  In CVPR, 2020.
  * [12] Weicheng Kuo, Anelia Angelova, Tsung-Yi Lin, and Angela Dai.  Mask2CAD: 3D shape prediction by learning to segment and retrieve.  In European Conference on Computer Vision, pages 260–277. Springer, 2020.
  * [13] Weicheng Kuo, Anelia Angelova, Tsung-Yi Lin, and Angela Dai.  Patch2CAD: Patchwise embedding learning for in-the-wild shape retrieval from a single image.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12589–12599, 2021.
  * [14] Hao Li, Guowei Wan, Honghua Li, Andrei Sharf, Kai Xu, and Baoquan Chen.  Mobility fitting using 4D RANSAC.  In Computer Graphics Forum, volume 35, pages 79–88. Wiley Online Library, 2016.
  * [15] Xiaolong Li, He Wang, Li Yi, Leonidas J Guibas, A Lynn Abbott, and Shuran Song.  Category-level articulated object pose estimation.  In CVPR, pages 3706–3715, 2020.
  * [16] Liu Liu, Han Xue, Wenqiang Xu, Haoyuan Fu, and Cewu Lu.  Toward real-world category-level articulation pose estimation.  IEEE Transactions on Image Processing, 31:1072–1083, 2022.
  * [17] Niloy J Mitra, Michael Wand, Hao Zhang, Daniel Cohen-Or, Vladimir Kim, and Qi-Xing Huang.  Structure-aware shape processing.  In ACM SIGGRAPH 2014 Courses, pages 1–21. 2014.
  * [18] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani.  Where2Act: From pixels to actions for articulated 3D objects.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813–6823, 2021.
  * [19] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su.  PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [20] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning continuous signed distance functions for shape representation.  In CVPR, pages 165–174, 2019.
  * [21] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3D classification and segmentation.  In CVPR, pages 652–660, 2017.
  * [22] Sasha Targ, Diogo Almeida, and Kevin Lyman.  Resnet in resnet: Generalizing residual architectures.  arXiv preprint arXiv:1603.08029, 2016.
  * [23] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas.  Normalized object coordinate space for category-level 6D object pose and size estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2642–2651, 2019.
  * [24] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu.  Shape2Motion: Joint analysis of motion parts and attributes from 3D shapes.  In CVPR, pages 8876–8884, 2019.
  * [25] Fangyin Wei, Rohan Chabra, Lingni Ma, Christoph Lassner, Michael Zollhöfer, Szymon Rusinkiewicz, Chris Sweeney, Richard Newcombe, and Mira Slavcheva.  Self-supervised neural articulated shape and appearance models.  In CVPR, pages 15816–15826, 2022.
  * [26] Yijia Weng, He Wang, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, and Leonidas J Guibas.  CAPTRA: Category-level pose tracking for rigid and articulated objects from point clouds.  In CVPR, pages 13209–13218, 2021.
  * [27] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su.  SAPIEN: A simulated part-based interactive environment.  In CVPR, pages 11097–11107, 2020.
  * [28] Yang Xiao, Xuchong Qiu, Pierre-Alain Langlois, Mathieu Aubry, and Renaud Marlet.  Pose from shape: Deep pose estimation for arbitrary 3D objects.  In British Machine Vision Conference (BMVC), 2019.
  * [29] Xianghao Xu, Yifan Ruan, Srinath Sridhar, and Daniel Ritchie.  Unsupervised kinematic motion detection for part-segmented 3D shape collections.  ACM Trans. Graph. (SIGGRAPH), pages 1–9, 2022.
  * [30] Zihao Yan, Ruizhen Hu, Xingguang Yan, Luanmin Chen, Oliver Van Kaick, Hao Zhang, and Hui Huang.  RPM-Net: Recurrent prediction of motion and parts from point cloud.  ACM Trans. Graph. (SIGGRAPH Asia), 38(6), 2019.
  * [31] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu.  LASR: Learning articulated shape reconstruction from a monocular video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15980–15989, 2021.
  * [32] Li Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas.  Deep part induction from articulated object pairs.  ACM Trans. Graph. (SIGGRAPH Asia), 2018.
  * [33] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas.  A scalable active framework for region annotation in 3D shape collections.  ACM Transactions on Graphics (ToG), 35(6):1–12, 2016.

##  A Additional Architecture Details

Here, we outline the architecture used for each predictive branch in CA2T-Net
. Let $FC(in,out)$ refer to a fully-connected layer with $in$ input and $out$
output units. $BN$ refers to a Batch Normalization layer and $ReLU$ is a
rectified linear unit nonlinearity. We process the input image with a
ResNet-18 and point cloud with a PointNet-style architecture to get
intermediate features h $\in\mathbb{R}^{b\times 256}$ and g
$\in\mathbb{R}^{b\times 256}$. We concatenate these features to form
$\textbf{z}\in\mathbb{R}^{b\times 512}$, a joint image-shape embedding. These
global features are further processed by the following three branches:

#### Pose Prediction

The pose prediction branch first further compresses the global features (via
$f_{compress}(\textbf{z})$, defined below) and then uses six separate linear
readout layers to predict azimuth, elevation and in-plane rotation bins and
offsets from the further compressed global features.

$f_{compress}(\cdot)=FC(512,800)-BN-ReLU-FC(800,400)-BN-FC(400,200)-BN-ReLU$

#### Motion Estimation

Motion predictions are made directly from global features z, with a separate
linear layer for each motion attribute. Motion class is predicted by passing z
through $FC(512,2)$, motion axis by $FC(512,3)$, motion origin by $FC(512,3)$
and motion magnitude by $FC(512,1)$.

#### 3D Part Segmentation

The segmentation branch takes in global features, as well as 3D coordinates,
$x$. The architecture is as following:

$g_{compress}(\cdot)=FC(512+3,256)-BN-ReLU-FC(256,256)-BN-ReLU$
$g_{seg}(\cdot)=FC(512+3+256,256)-BN-ReLU-FC(256,256)-BN-ReLU-FC(256,2)$

The final per-point prediction, $\hat{p}$, is made by:

$\hat{p}=g_{seg}([g_{compress}([\textbf{z},x]]),\textbf{z},x]),$

where $[\cdot]$ represents a concatenation operation across the feature
dimension.

##  B Articulated vs. Unarticulated Pose
Estimation

Our pose prediction branch is largely inspired by the architecture of
PoseFromShape [28], which introduces a model for pose estimation that also
takes as input an RGB image and template 3D shape. However, this work is
focused on pose estimation when the input image is unarticulated and generally
matches the configuration of the rest-state mesh. We wondered about the
relative difficulty of predicting the global pose for images of articulated
objects vs. images of unarticulated objects (as in the PoseFromShape setting).
To investigate this, we rendered additional images of the SAPIEN dataset with
unarticulated input images. We trained PoseFromShape for pose estimation on
synthetic SAPIEN data in the case of both articulated, and unarticulated input
images. We measured a validation Acc30 of 88.6% in the case of unarticulated
images (the standard PoseFromShape setting), compared to an Acc30 of 82.9% for
articulated images. Given the reasonably small performance drop ($\sim$6%)
going from articulated to unarticulated pose estimation, we concluded that
performing articulated pose estimation is not that much more difficult of a
task. For example, the PoseFromShape network may learn to pay attention to
more robust cues such as parallel lines in the silhouette of the object
(rather than positioning of individual object parts) in order to perform
global pose estimation, which are not typically affected much by articulation
of single object parts.

Figure 8: Motion axis visualization for our network vs. OPD. Input image as
well as frontal view of 3D model shown with colored arrows corresponding to
motion predictions. CA2T-Net predicted motion axis and origin in blue, OPD in
red, and ground-truth in green.

##  C Additional Qualitative Results

We show additional qualitative results for CA2T-Net trained on the SAPIEN
dataset. We visualize 3D motion axes predicted by CA2T-Net (trained on the
O-split) and OPD, show connected component segmentations of ABO meshes, and
include additional qualitative results for CA2T-Net trained on the SAPIEN
I-split.

#### Visualization of Motion Axes

Using the prediction motion axis and origin values, we can visualize them as
arrows on the 3D shape. Figure 8 shows examples of such motion predictions for
CA2T-Net and OPD. We also plot the ground truth motion axis and origin. Meshes
are visualized in their default, front-facing pose with colored arrows that
begin at the predicted origin, and point in the direction of the predicted
axis.

#### ABO Connected Components

Unlike synthetic SAPIEN models, ABO meshes do not contain ground-truth part
segmentations. We found that a good proxy for these is a connected components-
based segmentation of the mesh. In general, connected components will give an
over-segmentation of the model (e.g. each piece of wire in a wire shelf would
be its own component), however our method is still able to select the relevant
3D part conditioned on the articulated input image. We visualize ABO meshes
colored by their connected components, as well as our model’s predicted part
segmentation in Figure 9.

#### SAPIEN Validation Set

We provide additional qualitative results of CA2T-Net trained on the SAPIEN
I-split in Figure 10. This visualization shows renderings of the resulting
articulated 3D meshes, articulated by our model’s predictions. We compare to
the RandMot and FreqMot baseline.

Figure 9: Visualization of connected components in ABO. Middle column shows 3D
mesh corresponding to input image with each connected components visualized as
a different color. Our method samples points from each candidate part (i.e.,
connected component) and returns a part prediction corresponding to the
component that moved in the input. Figure 10: Additional qualitative results
on SAPIEN validation set. Our method compared to the RandMot and FreqMot
baselines.