# Articulate your NeRF: Unsupervised articulated object modeling via
conditional view synthesis

Jianning Deng Kartic Subr Hakan Bilen
University of Edinburgh
{jianning.deng, K.Subr<EMAIL_ADDRESS>
###### Abstract

We propose a novel unsupervised method to learn pose and part-segmentation of
articulated objects with rigid parts. Given two observations of an object in
different articulation states, our method learns the geometry and appearance
of object parts by using an implicit model from the first observation,
distills the part segmentation and articulation from the second observation
while rendering the latter observation. Additionally, to tackle the
complexities in the joint optimization of part segmentation and articulation,
we propose a voxel grid based initialization strategy and a decoupled
optimization procedure. Compared to the prior unsupervised work, our model
obtains significantly better performance, generalizes to objects with multiple
parts while it can be efficiently from few views for the latter observation.

##  1 Introduction

Articulated objects, composed of multiple rigid parts connected by joints
allowing rotational or translational motion, such as doors, cupboards and
spectacles are ubiquitous in our daily lives. Automatically understanding the
shape, structure and motion of these objects is crucial for numerous
applications in robotic manipulation [12, 40] and character animation [1, 30].
Many works [10, 28, 32] focused on this problem use groundtruth 3D shape,
articulation information, and/or part segmentation to learn articulated object
models but acquiring accurate 3D observations and manual annotations is
typically complex and too expensive for building real large-scale datasets.

In this paper, we introduce a novel unsupervised technique that learns part
segmentation and articulation (_i.e_., axis of movement, and
translation/rotation of each movable part) from two sets of observations
without requiring groundtruth shape, part segmentation or articulation. Each
set contains images of an object from multiple viewpoints in different
articulations. Our key idea is that articulation changes only the poses of the
object parts, not their geometry or texture. Hence, once the geometry and
appearance are learned, one can transform to another articulation state given
the part locations and target articulation parameters.

Building on this idea, we frame the learning problem as a conditional novel
articulation (and view) synthesis task (see Fig. 1(a)). Given a source
observation with multiple views of an object in one articulation state, we
first learn the object’s shape and appearance by using an implicit model [19]
and freeze its weights. Next we pass the target observation – multi-view
images of the same object in a different articulation state – to a tight
bottleneck that distills part locations and articulations. We constrain our
model to assign each 3D coordinate that is occupied by the object to a part
and to apply a valid geometric transformation to the 3D coordinates of each
part through ray geometry. The predictions of part segmentation and
articulation, along with the target camera viewpoint, are passed to the
implicit function and its differential renderer to reproduce the target
observations (see Fig. 1(b)). Minimizing the photometric error between the
rendered and target view provides supervision for learning part segmentation
and articulation. However, joint optimization of these intertwined tasks are
challenging and very sensitive to their initialization.

To address the optimization challenge, we propose an initialization strategy
using an auxiliary voxel grid, which provides an initial estimate for moving
parts by computing the errors in the foreground masks when rendering target
views for the source articulation. Additionally, we introduce a decoupled
optimization procedure that alternates between optimizing the part
segmentation on the photometric error and the articulation parameters on the
foreground prediction error.

The key advantage of our method, compared to other unsupervised articulation
prediction methods [11, 15], is its stable performance across different object
and articulation types, its ability to learn from few target views and to
model multiple moving parts. Thanks to the stage-wise training, we achieve
high-quality object geometry and appearance by using the well-optimized
implicit models [2]. Since the part segmentation and articulation parameters
form a small portion of the total weights, along with the initialization and
decoupled optimization strategies, our method efficiently learns them from few
target views, unlike the most relevant work [15] that jointly optimizes
multiple part-specific implicit functions from scratch.

Figure 1:  (a) Our method learns the geometry and appearance of an articulated
object by first fitting a NeRF from (source) images of an object in a fixed
articulation. Then, from another set of (target) images of the object in
another articulation, we distill the relative articulation and part labels.
Green lines show the gradient path during this distillation. (b) Using the
part geometry and appearance from NeRF, we render the target images by
compositing the parts after applying the predicted articulations to the
segmented parts. The photometric error provides the required supervision for
learning the parts and their articulation without groundtruth labels.

##  2 Related work

##### Analysis of articulated objects

The analysis of articulated objects typically involves segmentation of movable
parts, and estimating their attributes such as position and direction of joint
axes, rotation angles, and translation distances. Prior works study
articulated objects using 3D input such as meshes [29, 20], point clouds [42,
41, 35] or RGBD images [16, 6] that are error-prone and labor-intensive to
collect in real-world scenarios. Recent works [10, 28, 32] that use RGB images
to segment 3D planar object parts and estimate their articulation parameters
simultaneously require ground-truth labels for segmentation and 3D
articulation.

##### Articulated object modeling via novel view synthesis

Neural implicit models [18, 24, 19] that are originally designed to model
static objects in a 3D consistent way have been extended to articulated
objects by multiple recent works [22, 34, 11, 14, 36, 15]. A-SDF [22] learns a
disentangled latent space for shape and articulation to synthesize novel
shapes and unseen articulated poses for a given object category. CARTO [7]
extends A-SDF to multiple object categories and uses a stereo image pairs as
input. Similarly, CLA-NeRF [34] learns to perform joint view synthesis, part
segmentation, and articulated pose estimation for an object category from
multiple views. NARF22 [14] learns a separate neural radiance fields (NeRF)
for each part and composes to render the complete object. Unlike our method,
both A-SDF and CARTO require 3D shape and articulation pose supervision, CLA-
NeRF requires part segmentation labels and is limited to modeling only 1D
revolutions for each joint, NARF22 relies on groundtruth part labels and
articulation pose.

Most related to our work, DITTO [11] and PARIS [15] aim to estimate part
segmentation and articulation pose without labels using from an observation
pair of an object in two different articulations. DITTO uses a pair of point
cloud as input, can only model shape, whereas both PARIS and our method uses
pairs of multi-view images, and to model both geometry and appearance. PARIS
adopts the dynamic/static modelling of [39, 43], learns a separate NeRF for
the dynamic and static parts. They are composited using the estimated relative
articulation to render the observations with the different articulation. While
our method is also based on the same analysis-by-synthesis principle to
supervise the training, it differs from PARIS in several key ways. Our method
involves only a single NeRF that is learned on multiple views of an object
instance in a fixed articulation pose. Once the NeRF is learned, we freeze its
parameters, and learn a segmentation head and articulation to selectively
deform the rays while rendering views of different articulations. This means
our model’s size remain nearly constant when the number of parts increases,
which, combined with our two step optimization strategy, leads to more stable
and data-efficient learning, yielding significantly better performance.
Furthermore, we show that our model goes beyond modeling a single moving
object part as in PARIS, and successfully learn multiple moving parts.

##### Deformable NeRF

There exists several techniques [25, 27, 33, 26, 5] that model differences
between subsequent video frames of a dynamic scene through a deformation field
in 3D. While these techniques are general could be used in modeling
articulated objects in principle, the deformation field does not provide
explicit predictions for part segmentation and articulation. Prior methods
that focus on modeling specific articulated and deformable objects such as
human body [37, 31, 8, 9, 23] and four-legged animals [38] leverages
specialized 3D priors [17] that are not applicable to arbitrary object
categories.

##  3 Review: NeRF

Given a set of images $\mathcal{I}$ of an object, a NeRF function [19] maps a
tuple $(\bm{x},\bm{d})$, where $\bm{x}\in\mathbb{R}^{3}$ is a 3d position and
$\bm{d}\in\mathbb{R}^{2}$ is a direction, to an RGB color $\bm{c}$, and a
positive scalar volume density $\sigma$. The model outputs volume density
$\sigma$ at the location $\bm{x}$ along with a latent feature vector $z$ which
is then concatenated with the viewing direction $\bm{d}$ and fed into an MLP
to estimate the view-dependent RGB color $\bm{c}$. With a slight abuse of
notation, we also use $\sigma$ and $z$ as functions $\sigma(\bm{x})$ and
$z(\bm{x})$ respectively and color as a function with $c(\bm{x},\bm{d})$. That
is, we use boldface to denote vectors.

NeRFs are trained by expressing the color at each pixel of images in the
training subset. The color at a pixel $C$ is estimated as $C_{N}$ via the
volume rendering equation as a sum of contributions from points along the ray
through the pixel, say $r\in\mathcal{R}$. Points $\bm{x}_{i},i=1\cdots n$ are
first sampled along the ray so that $\bm{x}_{i}=\bm{o}+t_{i}\bm{d}$ where
$\bm{o}$ is the origin of the ray (usually the point of projection of a camera
view) and $t_{i}$ is a scalar. Using these samples, the estimated color is
given by

$\displaystyle{C}_{N}(r)=\sum^{n}_{i=1}\;T_{i}^{r}\;(1-\exp(-\sigma_{i}^{r}\delta_{i}^{r})\mathbf{c}^{r}_{i})\quad\mathrm{where}\quad
T_{i}^{r}=\sum^{i-1}_{j=1}\exp(-\sigma_{j}^{r}\delta_{j}^{r})$ (1)

where we use the shorthand notation $\bm{c}_{i}^{r}$ and $\sigma_{i}^{r}$ to
denote $c(\bm{x}_{i},\bm{d})$ and $\sigma(\bm{x}_{i})$ respectively, and
$\delta_{i}=t_{i+1}-t_{i}$ is the distance between samples. And the opacity
value is calculated as: Similarly the opacity value for the pixel can be
computed as
${O}_{N}(r)=\sum^{n}_{i=1}\;T_{i}^{r}\;(1-\exp(-\sigma_{i}^{r}\delta_{i}^{r})$.
The parameters of $c$ and $\sigma$ are obtained by minimizing the photometric
loss between predicted color ${C}_{N}(r)$ and the ground truth color $C(r)$ as

$\mathcal{L}_{\text{photo}}=\sum_{r\in\mathcal{R}}||{C}_{N}(r)-C(r)||^{2}_{2}.$
(2)

##  4 Method

Let $\mathcal{I}$ and $\mathcal{I}^{\prime}$ be two observations of an object
with $k$ (known a priori) rigid parts in articulation poses $\mathcal{P}$ and
$\mathcal{P}^{\prime}$ respectively. Each observation contains multiple views
of the object along with the foreground masks. We call $\mathcal{I}$ the
source observation and $\mathcal{I}^{\prime}$ the target observation.
$\mathcal{P}$ (resp. $\mathcal{P}^{\prime}$) is a pose tensor composed of
$P_{\ell}\in SE(3),\;{\ell}=1\cdots k$ (resp. $P_{\ell}^{\prime}$)
transformations as $4\times 4$ matrices corresponding to the local pose of
each of the $k$ parts. Our primary goal is to estimate a pose-change tensor
$\mathcal{M}$ composed of $M_{\ell}\in SE(3),\;{\ell}=1\cdots k$ so that
$P_{\ell}^{\prime}=M_{\ell}P_{\ell}$. We solve this by starting with the
construction of a NeRF from $\mathcal{I}$, as described in Sec. 3 along with
the efficient coarse-to-fine volume sampling strategy in [2], followed by a
novel modification to build awareness at the part-level. We use this modified
parameterization to optimize an auxiliary voxel grid corresponding to the
parts via pose estimation and part segmentation pipelines. Once we have
identified the parts and their relative transformations, we are able to re-
render novel views corresponding to articulation pose $\mathcal{P}^{\prime}$
by transforming view rays suitably by $M_{\ell}^{-1}$.

###  4.1 Part-aware rendering

Once a NeRF function, which we call ‘static NeRF’, is learned over
$\mathcal{I}$, we freeze its parameters and append a segmentation head $s$ to
it towards obtaining part-level information. $s$ is instantiated as 2-layer
MLP, and maps the latent feature vector $z(\bm{x})$ to a probability
distribution over the $k$ object parts. We denote the probability of a 3D
point $\bm{x}$ to belong object part $\ell=1\cdots k$ as
$s_{\ell}(z(\bm{x}))$.

If the segmentation class-probabilities $s_{\ell}$ and pose-change
transformations $M_{\ell}$ are known, then the object in an unseen
articulation pose can be synthesised by suitably transforming the statically
trained NeRF without modifying its parameters and then compositing the
individual part contributions. To model pose change in each part, we use a
different ray, _virtual ray_ associated with each part

$r_{\ell}=M_{\ell}^{-1}r$ (3)

and the final rendered color is:

${C}_{P}(r)=\sum_{i=1}^{n}\;\hat{T}_{i}^{r}\;\;\sum_{\ell=1}^{k}\left(1-\exp{\left(-\left(s_{\ell}^{r_{\ell}}(\bm{x}_{i})\;\sigma^{r_{\ell}}(\bm{x}_{i})\;\delta_{j}^{r_{\ell}}\right)\right)}\right)\mathbf{c}_{i}^{r_{\ell}}$
(4)

where
$\hat{T}_{i}^{r}=\sum_{j=1}^{i-1}\exp{\left(-\sum_{\ell=1}^{k}\left(s_{\ell}^{r_{\ell}}(\bm{x}_{j})\;\sigma^{r_{\ell}}(\bm{x}_{j})\;\delta_{j}^{r_{\ell}}\right)\right)}$
is the transmittance sum. Compared to the original formulation in Eq. 1, it
can seen as the density replaced by a part-aware density which is the product
of $s$ and $\sigma$. Note that we made a similar modification to the efficient
volume sampling strategy of [2] and refer to the Sec. A.1 for more details.

Since the groundtruth part segmentation and articulation are unknown, one
could optimize the parameters of $s$ and $\mathcal{M}$ on the target
observation such that the photometric error in between the part-aware rendered
views and $\mathcal{I}^{\prime}$. Note that ${C}_{P}(r)$ is function of
$\mathcal{M}$ (through $r_{\ell}$ in Eq. 3) and $s$, they can be jointly
optimized through backpropagation. However, jointly solving these two
intertwined problems through rendering is challenging and highly sensitive to
their initial values, as their solutions depend on each other. There, we
introduce an alternating optimization strategy next.

###  4.2 Decoupled optimization of $s$ and $\mathcal{M}$

To learn the part segmentation and pose-change, we first introduce an
auxiliary voxel grid that assigns each 3D coordinate either to a background or
a part. Once the voxel is initialized, our training iterates multiple times
over a three-step procedure that includes optimization of $\mathcal{M}$, $s$,
and refinement of the voxel grid entries.

##### Initialization of voxel grid

Here our key idea is to find the pixel differences between the source and
target observations by rendering a target view $v^{\prime}$ by using the
static NeRF, and label the 3D locations of the different pixels as the movable
parts to provide a good initialization for estimating the articulation (see
Fig. 2). To this end, we first build a 3D voxel of the static NeRF by
discretizing its 3D space into 128 bins along each dimension. To compute the
voxel entries, we query the density value from the static NeRF at each voxel
coordinate and binarize it into occupancy values with criteria
$\text{exp}(-\sigma(\bm{x})\delta)>0.1$. Next, we use the static NeRF to
render images for the viewpoints $v^{\prime}$ that are used to capture
$\mathcal{I}^{\prime}$. We compute difference in opacity value using $O_{N}$
between these rendered images and the foreground masks of
$\mathcal{I}^{\prime}$, and binarize each value with $0.5$ as threshold, and
collect the ones that are not contained in the groundtruth foreground mask of
the corresponding view in $\mathcal{I}^{\prime}$. For those pixels, we further
compute their depth by accumulating their density values along the
corresponding ray, and use their estimated depth values to tag the occupied
voxel entries either with static or dynamic parts. In the case of multiple
moving parts, which is assumed to be known, we perform a clustering step [4]
to identify $k-1$ clusters corresponding to moving parts, as we assume that 1
part of the object is static without loss of generality. Finally, we gather
the voxel coordinates that is assigned to part $\ell$ to form a matrix of 3D
coordinates $X_{\ell}$.

Figure 2: Voxel initialization: identify the voxels belonging to moved parts
based on pixel opacity difference.  Figure 3: Illustration for optimization of
$M_{\ell}$. The green dotted line shows the gradient flow.

##### Step 1: Optimization of $M_{\ell}$

As depicted in Fig. 3, we project the 3D coordinates $X_{\ell}$ for part
$\ell$ onto the image plane for each camera viewpoint $v^{\prime}$ included in
$\mathcal{I}^{\prime}$ by using $3\times 4$ dimensional intrinsic camera
matrix $K$ as $U_{\ell}=KM_{\ell}^{-1}\bm{v^{\prime}}X_{\ell}$, and denote the
projected point matrix as $U_{\ell}$. We concatenate the part-specific
matrices to obtain $U$. As the groundtruth pixelwise part segmentation for
part $\ell$ is unknown, we instead use the foreground masks for each view
$\mathcal{I}^{\prime}$ that consists of all the part pixels, and stack the 2D
coordinates to form the matrix $F$. Note that both $U$ and $F$ are matrices
where each row corresponds to a 2D image coordinate where $F$ has
significantly more rows, as $U$ is obtained from a coarse voxel grid. Then we
minimize the 2D Chamfer distance between $U$ and $F$:

$\displaystyle\mathcal{M}^{*}=\arg\min_{\mathcal{M}}\,\,d_{\text{CD}}(U,F).$
(5)

In practice, we only project $X_{\ell}$ for the moved parts, and stack them
with 2D image coordinates of the foreground pixels in both the rendered image
and the groundtruth image, as illustrated in Fig. 3.

##### Step 2: Optimization of $s_{\ell}$

Once we obtain the solution of the pose-change $\mathcal{M}^{*}$ from Step 1,
we plug in it to the ray deformation in Eq. 3 and render each view in
$\mathcal{I}^{\prime}$ in Eq. 4. Then we compute photometric loss between the
rendered views and $\mathcal{I}^{\prime}$, and minimize it with respect to the
parameters of $s$ only. In the case of multiple moving parts, we initialize
the parameters in $s$ using $X_{\ell}$ for supervision. Please refer to Sec.
A.4 for more details.

##### Step 3: Voxel grid refinement

The initial part segmentation estimates in the voxel grid are often sparse and
noisy due to inaccurate density estimation in NeRF and misassignment of pixels
around the foreground and part boundaries. To refine them, we build a 3D voxel
again following steps in initialization stage, but we assign those 3D
coordinates for different part $\ell$ based on the predicted label from
segmentation head $s$. We denote the new 3D coordinates as $X_{\ell}^{*}$. To
maintain the consistency, we maintain and update voxels in $X_{\ell}^{*}$ only
when the predicted part labels agree with the voxel labels within a 3D local
neighborhood of $X_{\ell}$. This neighborhood is defined as being within the
distance of one grid cell at a resolution of 128. Then we update $X_{\ell}$
accordingly. Note that we only perform the refinement step, after
approximately 2k iterations to ensure that the segmentation head produces
confident results. Then we also increase the voxel resolution to 256 for each
dimension.

rowsep=1pt,colsep=1pt  Metric Method Revolut Prismatic laptop oven stapler
fridge blade storage $e_{d}$ $\downarrow$ PARIS 0.68 $\pm$ 0.40 1.04 $\pm$
0.68 2.42 $\pm$ 0.91 0.81 $\pm$ 0.60 48.58 $\pm$ 25.43 0.34 $\pm$ 0.09 Ours
0.33 $\pm$ 0.04 0.34 $\pm$ 0.03 0.33 $\pm$ 0.04 0.54 $\pm$ 0.08 1.54 $\pm$
0.07 1.11 $\pm$ 0.14 $e_{p}$$\downarrow$ ($10^{-2}$) PARIS 0.18 $\pm$ 0.15
0.49 $\pm$ 0.53 55.54 $\pm$ 39.88 0.33 $\pm$ 0.14 - - Ours 0.48 $\pm$ 0.06
1.29 $\pm$ 0.04 0.17 $\pm$ 0.05 0.44 $\pm$ 0.03 - - $e_{g}$$\downarrow$ PARIS
0.60 $\pm$ 0.32 0.68 $\pm$ 0.39 44.62 $\pm$ 6.17 0.87 $\pm$ 0.55 - - Ours 0.25
$\pm$ 0.03 0.35$\pm$ 0.06 0.290$\pm$0.03 0.60 $\pm$ 0.05 - - $e_{t}$
$\downarrow$ PARIS - - - - 1.13 $\pm$ 0.52 0.30$\pm$ 0.01 Ours - - - - 0.01
$\pm$ 0.01 0.02$\pm$ 0.03

Table 1: Part-level pose estimation results. Our method outperforms PARIS in
majority of object categories while having lower variation over multiple runs
in the performance.

##  5 Experiment

###  5.1 Dataset

We evaluate our method on the synthetic 3D PartNet-Mobility dataset [40, 21,
3]. While the dataset contains more than 2000 articulated objects from 46
different categories, we use a subset of the dataset with 6 shapes that was
used in [15]. For a fair comparison, we downloaded the processed dataset from
[15] which contains 2 sets of 100 views along with their foreground masks,
each with a different articulation, and also groundtruth part segmentation
labels, for each shape. In addition, we select 4 additional shapes, each with
two moving parts, and apply the same data generation process to them.
Additionally we collected images of a toy car with a handheld device, with
camera viewpoint estimated from kiri engine application[13].

We train the static NeRF on 100 views from the first observation, train the
part segmentation and articulation on 100 views from the second observation.
We report the performance of our method for varying number of views from the
second observation in Tab. 5.

Following [15], we report performance in different metrics for pose
estimation, novel-view/articulation synthesis, and part segmentation. Pose
estimation: To report articulation pose performance, we report results in i)
direction error $e_{d}$ that measures the angular discrepancy in degrees
between the predicted and actual axis directions across all object categories,
ii) position error $e_{p}$ and geodesic distance $e_{d}$ for only revolute
objects to evaluate the error between the predicted and true axis positions
and the error in the predicted rotation angles of parts in degrees,
respectively, iii) translation error $e_{t}$ to measure the disparity between
the predicted and actual translation movements for prismatic objects. Novel-
view and -articulation synthesis: We evaluate the quality of novel view
synthesis generated by the models using the Peak Signal-to-Noise Ratio (PSNR)
where higher values indicate better reconstruction fidelity. Part
segmentation: We use mean Intersection over Union (mIoU) on the rendered
semantic images to evaluate the accuracy of part segmentation, which is
tightly related to the rendered image quality of objects in different
articulation states. The ground truth segmentation is generated within the
Sapien framework in [40]. Finally, due to the lack of groundtruth segmentation
label and articulation in the real data, we only report PSNR and provide
qualitative evaluation.

###  5.2 Results

##### Baseline

We compare our approach with the state-of-the-art unsupervised technique,
PARIS [15] which constructs separate NeRF models for each part of an
articulated object and optimizes motion parameters in an end-to-end manner.
However, it is limited to two-part objects with only one movable part. As the
authors of PARIS do not report their results over multiple runs in their
paper, we use their official public code with the default hyperparameters,
train both their and our model over 5 random initializations and report the
average performance and the standard deviation. We use 2 sets of 100 views for
each object to train the models. More implementation details can be found in
the supplementary material. We would like to note that the performance of
PARIS in our experiments significantly differ from the original results
despite all the care taken to reproduce them in the original way111Similar
problems have been pointed out and the reproducibility of these results has
been acknowledged as challenging by the authors in issue 1 and issue 2.

rowsep=1pt,colsep=1pt  PARIS Ours GT Blade Laptop Stapler Fridge Storage Oven
Scissor

Figure 4: Qualitative 2D part segmentation results. Pixels in green denotes
the movable parts. Our method demonstrates consistent performance across all
tested objects while PARIS failed for Blade, Laptop and Scissor.

##### Part-level pose estimation

For part-level pose estimation, we provide quantitative results in average
performance over 5 runs and their standard deviations in Tab. 1, and a
qualitative analysis in Fig. 6. The results indicate that our method
consistently achieves lower errors across most evaluation metrics compared to
PARIS, except the joint position error $e_{p}$ where the differences are
negligible, within the $10^{-3}$ range. Notably, the performance of PARIS on
the stapler and blade exhibits significantly higher errors. We observe that
PARIS fails to converge to a good solution in all 5 runs for these objects. As
shown in Fig. 6, PARIS fails to accurately segment the parts in the stapler
and blade, which can also be easily identified in the novel articulation
synthesis experiments in Fig. 6. Poor part reconstruction in PARIS results in
inaccurate part-level pose estimations. Additionally, the lower standard
deviation across all reconstructed objects for our method indicates its stable
learning and ability to work for different object instances. We attribute the
stability of our method to the decoupled optimization along with the stage-
wise training, which contrasts with the joint optimization approach used by
PARIS.

Ours |  |  |  |
---|---|---|---|---
GT |  |  |  |
|  Box |  Glasses |  Oven | Storage
Figure 5: Qualitative results for 2D multi-part segmentation. The pink color
denotes the static part, while other colors denote the moving parts.

##### Segmentation and composite rendering

Part segmentation and novel view synthesis in pose $\mathcal{P}^{\prime}$ are
reported in Tab. 3, complemented by a qualitative analysis of part
segmentation in Fig. 4. Our method outperforms PARIS across most evaluated
objects in part segmentation and image synthesis quality, with the only
exception being a minor difference in the PSNR for the laptop.

As our model builds on a static NeRF that achieves high quality novel view
synthesis for one observation, the rendering quality is largely preserved
after the second stage. We provide more detailed analysis in Sec. A.3.
Additionally, benefiting from accurate pose estimation via a decoupled
approach, our method achieves more robust and precise part segmentation, as
depicted in Fig. 4. Here, our method consistently delivers accurate
segmentation results for challenging objects such as the blade, stapler, and
scissors, where PARIS struggles with accurate part reconstruction. In the
other instances including the laptop, storage, and oven, our method achieves
visibly better results.

##### Evaluation on objects with multiple movable parts

A key advantage of our model is its ability to model objects with multiple
moving parts. For such objects, we report pose estimation results in Tab. 3,
qualitative for part segmentation in Fig. 5 and novel articulation synthesis
in Fig. 6. As in the single part moving object experiments, our method
performs consistently performs over the multi-part objects. Notably, we
observed a marginally higher joint direction error for glasses, which we
attribute to the thin structures such as temples and failure to segment them
accurately which can be possibly improved by using higher resolution images.

rowsep=2pt,colsep=2pt  Metric Method Revolut Prismatic laptop oven stapler
fridge blade storage mIoU$\uparrow$ PARIS 0.98 0.99 0.16 0.98 0.76 0.94 Ours
0.99 0.99 0.98 0.99 0.94 0.96 PSNR $\uparrow$ PARIS 30.31 31.48 24.36 32.74
31.87 30.63 Ours 29.27 32.08 34.31 35.10 36.47 34.51

Table 2: Articulation synthesis and part segmentation results. Average
performance over 5 runs ( best results in boldface).

rowsep=1pt,colsep=1pt  Metric Revolut Prismatic oven glasses box storage
$e_{d}$ $\downarrow$ 1.02 2.35 0.56 1.82 $e_{p}$ $\downarrow$ 0.16 0.47 0.27 -
$e_{g}$ $\downarrow$ 1.03 1.01 0.65 - $e_{t}$ $\downarrow$ ($10$) - - - 0.11
PSNR $\uparrow$ 32.98 29.22 28.61 28.25 mIoU $\uparrow$ 0.97 0.98 0.99 0.94

Table 3: Objects with multiple parts. Errors using multiple metrics for pose
estimation (averaged over all joints).

rowsep=0pt,colsep=1pt  Method GT in $\mathcal{P}$ Novel articulation synthesis
GT in $\mathcal{P}^{\prime}$ PARIS Ours PARIS Ours Ours

Figure 6: Qualitative evaluation for novel articulation synthesis. The ground
truth axis is denoted in green and the predicted axis is denoted in red.
Please refer to the supplementary for more visualizations.

###  5.3 Ablation studies

We assessed the effectiveness of the proposed decoupled pose estimation of
$M_{\ell}$ (DP) and iterative refinement of $X_{\ell}$ (IR) on the ‘fridge’
object. When DP is disabled, the segmentation and articulation are
simultaneously learned. In contrast, disabling IR maintains the initial
$X_{\ell}$ for pose estimation. The results in Tab. 5, particularly the first
row, demonstrate that joint optimization without DP inaccurately predicts the
articulated pose, treating the entire object as static. While enabling DP
improves the performance, as shown in the second row, the performance is still
poor due to the noisy initial values compared to our full model.

Additionally, we evaluated the impact of the number of views in the target
observation for training. We train our model on the randomly subsampled views
for multiple runs and averaged their performance (see Tab. 5). Results show a
significant drop in pose estimation and rendering quality with 4 images or
less. Notably, even with only 8 images, our approach surpasses the performance
of PARIS trained with 100 images (indicated with subscript $P$ in Tab. 5).
This result clearly show that our method are more robust against fewer
viewpoints from the target viewpoints and allows efficiently learning
‘articulate’ a pretrained NeRF from few target views only.

rowsep=1pt,colsep=2pt  Metric Num. of images 2 4 8 16 32 100 100P $e_{d}$
$\downarrow$ 46.05 8.89 0.59 0.58 0.50 0.54 0.81 $e_{g}$ $\downarrow$ 44.74
20.79 0.70 0.60 0.56 0.49 0.87 PSNR $\uparrow$ 22.65 29.95 34.00 34.28 34.88
35.10 32.74

Table 4: Ablation studies with different number of target images.

rowsep=2pt,colsep=2pt  Init. Metric DP IR $e_{d}$ $e_{g}$ PSNR - - 2.57 26.95
18.09 ✓ - 2.38 14.96 25.16 ✓ ✓ 0.54 0.49 35.10

Table 5: Ablation study over different initialization strategies.

###  5.4 Real-world example

The qualitative evaluation can be found in Fig. 7. The pose $\mathcal{P}$ for
training the static NeRF is with two doors wide open, and the reconstruction
results from NeRF is shown in the first row in Fig. 7. Qualitative comparison
for ground truth images in pose $\mathcal{P}^{\prime}$ is presented in second
and third row of Fig. 7. The average PSNR we obtain from the static NeRF is ,
and the PSNR for 28.19, while the PSNR for pose $\mathcal{P}^{\prime}$ is
24.32. More detail results for segmentaion and articulation interpolation are
also presented in Fig. 7.

|  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
Articulation interpolation |  |  | GT | Seg. | Articulation interpolation |  |  | GT | Seg.

Figure 7: Results on real world examples, the red line indicates the estimated
joint axis direction. Green and purple color denotes the moving car door,
while pink denotes the body of the toy car. Please refer to Fig. 12 for more
qualitative evaluation.

###  5.5 Limitations

Our method has few limitations too. As we use rendering to supervise our
segmentation and pose, our method may fail to segment very thin parts when the
rendering error is small. When the object parts are nearly symmetry, our
method may fail to find the correct articulation and choose another
articulation that produces similar rendering. We provide examples of such
failure cases in Sec. A.3. Additionally, it also inherits the limitations of
the implicit models that are built on such as failure to model transparent
objects, and inaccurate 3D modeling when the viewpoint information is noisy.
Finally, our method is limited to articulated objects with rigid parts and
cannot be used to model soft tissue deformations.

##  6 Conclusion

In this study, we tackled the challenges of self-supervised part segmentation,
articulated pose estimation, and novel articulation rendering for objects with
multiple movable parts. Our approach is the first known attempt to model
multipart articulated objects using only multi-view images acquired from two
arbitrary articulation states of the same object. We evaluated our method with
both synthetic and real-world datasets. The results suggest that our method
competently estimates part segmentation and articulation pose parameters, and
effectively renders images of unseen articulations, showing promising
improvements over existing state-of-the-art techniques. Further, our real-
world data experiments underscore the method’s robust generalization
capabilities. At last, the code and the data used in this project will be
released upon acceptance. However, the reliance on geometric information from
moving parts for articulation pose estimation poses challenges in modeling
highly symmetrical objects. Future work could improve our method by
incorporating both appearance and geometric data into the pose estimation
process, potentially enhancing accuracy and applicability.

##### Broader Impacts

The proposed technique could be potentially used to understand articulated
objects for their robotic manipulation in future. The authors are not aware of
any potential harm that may arise when the technology is used

## References

  * [1] Norman Badler, Brian Barsky, and David Zeltzer.  Making Them Move: Mechanics, Control & Animation of Articulated Figures.  Routledge, 1990.
  * [2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [3] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [4] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu.  Density-based spatial clustering of applications with noise.  In Int. Conf. knowledge discovery and data mining, volume 240, 1996.
  * [5] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian.  Fast dynamic radiance fields with time-aware neural voxels.  In SIGGRAPH Asia 2022 Conference Papers, pages 1–9, 2022.
  * [6] Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang.  Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7081–7091, 2023.
  * [7] Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar.  Carto: Category and joint agnostic reconstruction of articulated objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21201–21210, 2023.
  * [8] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung.  Arch: Animatable reconstruction of clothed humans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093–3102, 2020.
  * [9] Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, and Matthias Nießner.  Humanrf: High-fidelity neural radiance fields for humans in motion.  ACM Transactions on Graphics (TOG), 42(4):1–12, 2023.
  * [10] Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang.  Opd: Single-view 3d openable part detection.  In European Conference on Computer Vision, pages 410–426. Springer, 2022.
  * [11] Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu.  Ditto: Building digital twins of articulated objects from interaction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5616–5626, 2022.
  * [12] Dov Katz and Oliver Brock.  Manipulating articulated objects with interactive perception.  In 2008 IEEE International Conference on Robotics and Automation, pages 272–277. IEEE, 2008.
  * [13] KIRI Innovation.  Kiri engine.
  * [14] Stanley Lewis, Jana Pavlasek, and Odest Chadwicke Jenkins.  Narf22: Neural articulated radiance fields for configuration-aware rendering.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 770–777. IEEE, 2022.
  * [15] Jiayi Liu, Ali Mahdavi-Amiri, and Manolis Savva.  Paris: Part-level reconstruction and motion analysis for articulated objects.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 352–363, 2023.
  * [16] Liu Liu, Han Xue, Wenqiang Xu, Haoyuan Fu, and Cewu Lu.  Toward real-world category-level articulation pose estimation.  IEEE Transactions on Image Processing, 31:1072–1083, 2022.
  * [17] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, October 2015.
  * [18] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460–4470, 2019.
  * [19] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [20] Niloy J Mitra, Yong-Liang Yang, Dong-Ming Yan, Wilmot Li, Maneesh Agrawala, et al.  Illustrating how mechanical assemblies work.  ACM Transactions on Graphics-TOG, 29(4):58, 2010.
  * [21] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su.  PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [22] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang.  A-sdf: Learning disentangled signed distance functions for articulated shape representation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13001–13011, 2021.
  * [23] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada.  Neural articulated radiance field.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5762–5772, 2021.
  * [24] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  * [25] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  * [26] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  arXiv preprint arXiv:2106.13228, 2021.
  * [27] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [28] Shengyi Qian, Linyi Jin, Chris Rockwell, Siyi Chen, and David F Fouhey.  Understanding 3d object articulation in internet videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1599–1609, 2022.
  * [29] Andrei Sharf, Hui Huang, Cheng Liang, Jiapei Zhang, Baoquan Chen, and Minglun Gong.  Mobility-trees for indoor scenes manipulation.  In Computer Graphics Forum, volume 33, pages 2–14. Wiley Online Library, 2014.
  * [30] Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov.  Motion representations for articulated animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13653–13662, 2021.
  * [31] Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin.  A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose.  Advances in Neural Information Processing Systems, 34:12278–12291, 2021.
  * [32] Xiaohao Sun, Hanxiao Jiang, Manolis Savva, and Angel Xuan Chang.  Opdmulti: Openable part detection for multiple objects.  arXiv preprint arXiv:2303.14087, 2023.
  * [33] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959–12970, 2021.
  * [34] Wei-Cheng Tseng, Hung-Ju Liao, Lin Yen-Chen, and Min Sun.  Cla-nerf: Category-level articulated neural radiance field.  In 2022 International Conference on Robotics and Automation (ICRA), pages 8454–8460. IEEE, 2022.
  * [35] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu.  Shape2motion: Joint analysis of motion parts and attributes from 3d shapes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8876–8884, 2019.
  * [36] Fangyin Wei, Rohan Chabra, Lingni Ma, Christoph Lassner, Michael Zollhöfer, Szymon Rusinkiewicz, Chris Sweeney, Richard Newcombe, and Mira Slavcheva.  Self-supervised neural articulated shape and appearance models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15816–15826, 2022.
  * [37] Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman.  Humannerf: Free-viewpoint rendering of moving people from monocular video.  In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition, pages 16210–16220, 2022.
  * [38] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.  Magicpony: Learning articulated 3d animals in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8792–8802, 2023.
  * [39] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli.  D^ 2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video.  Advances in neural information processing systems, 35:32653–32666, 2022.
  * [40] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su.  SAPIEN: A simulated part-based interactive environment.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [41] Li Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas.  Deep part induction from articulated object pairs.  arXiv preprint arXiv:1809.07417, 2018.
  * [42] Qing Yuan, Guiqing Li, Kai Xu, Xudong Chen, and Hui Huang.  Space-time co-segmentation of articulated point cloud sequences.  In Computer Graphics Forum, volume 35, pages 419–429. Wiley Online Library, 2016.
  * [43] Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove.  Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13144–13152, 2021.

##  Appendix A Appendix / supplemental material

Optionally include supplemental material (complete proofs, additional
experiments and plots) in appendix. All such materials SHOULD be included in
the main submission.

###  A.1 Part-aware proposal network

To achieve part-aware composite rendering for articulated objects, we also
need to modify the proposal network for correct sampling along the ray
$r\in\mathcal{R}^{\prime}$. In the composite rendering, the proposal network
is now required to produce the similar distribution of the weights for samples
along the ray. Thus, following the similar design in the part-aware NeRF, we
append an extra segmentation field to the proposal network. Now the valid
density for each samples becomes segmentation-mask density as
$s_{\ell}^{r_{\ell}}(\bm{x}_{j})\sigma^{r_{\ell}}$. The transmittance are then
calculated using Eq. 4 without the color term, which will be later used for
online distillation. Please refer to the original paper [2] for more details
about the online distillation. During the training of part-aware NeRF, the
original parameters in the proposal network will be frozen and only updates
the segmentation field.

###  A.2 Comparison with PARIS

Here we also provide the 5-time best comparison between ours and PARIS.
Besides, the reported performance for PARIS is also provided in this
subsection. From Tab. 6, we can see that the original reported performance
from PARIS are exceptionally better compared to the reproduced ones, even
though with the same data and the same configurations. If we only focus on the
5-time best comparison between reproduced PARIS and ours, we can see that we
achieve comparable performance. Another thing we can notice is that for
prismatic objects, while PARIS has much better estimations on joint axis
direction. However, our method later show much better estimation for
estimating the moved distance for the dynamic parts. While one thing to notice
is that PARIS failed to reconstruct the stapler for all 5 runs in our
experiment.

rowsep=2pt,colsep=2pt  Metric Method Revolut Prismatic laptop oven stapler
fridge blade storage $e_{d}$ $\downarrow$ PARIS† $\bm{0.03}$ $\bm{0.03}$
$\bm{0.07}$ $\bm{0.00}$ $\bm{0.00}$ 0.37 PARIS 0.18 0.48 1.30 0.22 0.27
$\bm{0.24}$ Ours 0.26 0.28 0.27 0.42 1.41 0.86 $e_{p}$$\downarrow$ ($10^{-2}$)
PARIS† $\bm{0.01}$ $\bm{0.03}$ $\bm{0.06}$ $\bm{0.02}$ - - PARIS 0.08 0.04
2.12 0.17 - - Ours 0.39 1.24 0.13 0.41 - - $e_{g}$$\downarrow$ PARIS†
$\bm{0.03}$ $\bm{0.00}$ $\bm{0.00}$ $\bm{0.00}$ - - PARIS 0.24 0.25 37.86 0.37
- - Ours 0.22 0.31 0.24 0.55 - - $e_{t}$ $\downarrow$ ($10^{-1}$) PARIS† - - -
- $\bm{0.06}$ $\bm{0.00}$ PARIS - - - - 5.84 2.93 Ours - - - - 0.12 0.05

Table 6: Comparison with PARIS in 5-time best setting. PARIS† denotes
performance reported in the original paper. Best results are shown in bold,
second best are shown with underline.

###  A.3 Limitations

Details about the limitations and failure cases of our methods.

In our decoupled optimization strategy, the articulated pose estimation relies
exclusively on the geometric information from the moving parts. As a result,
our method faces challenges with highly symmetrical components. For example,
in the case of a folding chair Fig. 8(c), although the segmentation accurately
identifies the chair’s seat, the pose estimation mistakenly flips the seat by
180 degrees, resulting in the seat being oriented with the bottom upwards as
shown in Fig. 8(b). Additionally, our method may encounter difficulties when
the articulation motion is minimal, which can lead to insufficient pixel
differentiation in images across different articulation poses for initial
model estimation. In such cases, both the pose estimator and the iterative
updates of $X_{\ell}$ may underperform or even fail. We also notice that the
proposal network have difficulties handling very thin parts in the novel
articulation synthesis. As we can see in Fig. 8(c), some artifacts appeared at
the edge of the frame and the temples for the glasses as in Fig. 8(c), which
should be ideally masked out without any opacity and color for those pixels.

Furthermore, the reliance on a pre-trained static NeRF limits the upper
boundary of our method’s performance in rendering novel articulations.
Addressing these limitations could significantly enhance the robustness and
applicability of our strategy in handling complex articulated objects. For
furhter analysis of this, please refer to .

(a) Visualization of dynamic voxel update. From left to right: initialized
$X_{\ell}$, updated $X_{\ell}$, final $X_{\ell}$

(b) Failure cases for foldchair, from left to right: groundtruth RGB, rendered
RGB, part segmentation.

(c) Artifacts for thin parts, the left one is the groundtruth, the right
rendering result.

###  A.4 Training setting

As detailed in Sec. 4.2, we perform optimization on $M_{\ell}$ for 4000
iterations and on $s_{\ell}$ for 2000 iterations for all evaluated objects.
Checkpoints are saved following Step 3. We conduct this process through 5
cycles for objects with a single moving part and 6 cycles for objects with
multiple moving parts. Checkpoints with best PSNR during validation will be
used for test.

During the first step of $M_{\ell}$ optimization, we begin with a learning
rate of 0.01, which linearly decays by 0.5 every 500 iterations. We accumulate
gradients from 8 viewpoints to simulate a batch size of 8, initializing
$M_{\ell}$ identically in the first cycle and using the previously estimated
$M_{\ell}$ for subsequent cycles.

In the second step of $s_{\ell}$ optimization using the Adam optimizer, the
initial learning rate is set at 0.01 and linearly decays by a factor of 0.01
every 100 iterations. For multiple moving parts, initialization involves
training the segmentation head $s$ using pre-assigned labels on $X_{\ell}$,
and querying predictions for $\bm{x}\in X_{\ell}$. Cross-entropy loss
optimized over 1000 iterations with a learning rate of $1e^{-3}$ shapes the
learnable parameters in $s$.

Our experiments, requiring around 16 GB of VRAM, complete in approximately 30
minutes on a single RTX 4090 GPU for a single object. As for the training of
static NeRF, it takes about 10 minutes with less than 10 GB of VRAM. The
estimated total GPU time for this project would be about 2 GPU months.

###  A.5 Ablations

####  A.5.1 Performance cost for novel articulation synthesis

We also provide an ablation study to investigate the performance drops for
conditional novel articulation synthesis compared to the pre-trained static
model. As shown in Tab. 7, the performance drop is subject to different
category of objects ranging from smallest 13.3% for the fridge and blade to
31.4% laptop. We suspect the significant drops for laptop is caused by the
deteriorate performance of the proposal network on the thin laptop screen.
Visualizations can be found in Fig. 9. The results indicate that our method
can benefit from the high quality of appearance reconstruction from the pre-
trained static NeRF.

(a) Ground truth

(b) Rendered

(c) Segmention in original pose

Figure 9: We can see in the Fig. 9(b) that the corner of the laptop screen is
missing in the novel articulation rendering. While it looks perfect when we
check the segmentation in the original pose. Thus, we suspect it is the
proposal network than failed to estimate the density distribution for the
screen from certain viewpoints.

Method | Object |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
| laptop | oven | stapler | fridge | blade | storage | oven∗ | glasses∗ | box∗ | storage∗
Static | 42.70 | 39.69 | 42.39 | 40.50 | 42.7 | 40.33 | 39.03 | 37.93 | 36.56 | 35.23
Art. | 29.27 | 32.08 | 34.31 | 35.1 | 36.47 | 34.51 | 32.98 | 29.22 | 28.61 | 28.25
$\Delta$ | -31.4% | -19.2% | -19.1% | -13.30% | -13.3% | -14.4% | -15.5% | -22.8% | -21.8% | -19.8%

Table 7: Comparison of rendering quality between objects in their original
pose $\mathcal{P}$ and articulated pose $\mathcal{P}^{\prime}$. ’Static’
refers to the rendering performance of the object in its original pose
$\mathcal{P}$, whereas ’Art.’ indicates the rendering quality of the object in
articulated pose $\mathcal{P}^{\prime}$ using our method with the static NeRF.
Objects marked with ∗ represent those with multiple movable parts.

###  A.6 Visualization

Here we show more visualization for qualitative evaluations. Besides,
additional animated images for the real-world toy car is also provided in the
supplementary materials.

rowsep=0pt,colsep=1pt  Method GT in $\mathcal{P}$ Novel articulation synthesis
GT in $\mathcal{P}^{\prime}$ PARIS Ours PARIS Ours PARIS Ours PARIS Ours

Figure 10: Articulation interpolation for single moving part objects.

rowsep=0pt,colsep=1pt  GT in $\mathcal{P}$ Novel articulation synthesis GT in
$\mathcal{P}^{\prime}$

Figure 11: Articulation interpolation for multiple moving part objects.

rowsep=0pt,colsep=1pt  Novel articulation synthesis Segmentation

Figure 12: More Qualitative evaluation on real-world object.