# TransPoser: Transformer as an Optimizer
for Joint Object Shape and Pose Estimation

Yuta Yoshitake Mai Nishimura Shohei Nobuhara Ko Nishino
Graduate School of Informatics, Kyoto University, Kyoto, Japan

###### Abstract

We propose a novel method for joint estimation of shape and pose of rigid
objects from their sequentially observed RGB-D images. In sharp contrast to
past approaches that rely on complex non-linear optimization, we propose to
formulate it as a neural optimization that learns to efficiently estimate the
shape and pose. We introduce Deep Directional Distance Function (DeepDDF), a
neural network that directly outputs the depth image of an object given the
camera viewpoint and viewing direction, for efficient error computation in 2D
image space. We formulate the joint estimation itself as a Transformer which
we refer to as TransPoser. We fully leverage the tokenization and multi-head
attention to sequentially process the growing set of observations and to
efficiently update the shape and pose with a learned momentum, respectively.
Experimental results on synthetic and real data show that DeepDDF achieves
high accuracy as a category-level object shape representation and TransPoser
achieves state-of-the-art accuracy efficiently for joint shape and pose
estimation.

Figure 1: We propose TransPoser, a novel neural optimization network for joint
object pose and shape reconstruction from sequential observations (a). We also
derive DeepDDF, a novel category-level neural 3D shape representation that
directly outputs 2D depth images for given viewing conditions. By integrating
DeepDDF and TransPoser, we realize efficient and accurate learned weighted
optimization with learned momentum for joint shape and pose estimation (b).

Method |   Feedback Loop  |   Fewer Updates  |  Joint Shape and Pose Estimation |  Multi-view Observation | Online |  Shape Representation | Momentum
---|---|---|---|---|---|---|---
DPDN [21] |  | - | ✓ |  | - | Point Cloud | -
Scan2CAD [4] |  | - | ✓ | ✓ |  | CAD models | -
RayTran [34] |  | - | ✓ | ✓ |  | Voxel Occupancy | -
Vid2CAD [24] | ✓ |  | ✓ | ✓ | ✓ | CAD models | Moving Avg.
ELLIPSDF [30] | ✓ |  | ✓ | ✓ | ✓ | DeepSDF | Moving Avg.
Ma _et al_. [23] | ✓ | ✓ |  |  |  | - |
TransPoser | ✓ | ✓ | ✓ | ✓ | ✓ | DeepDDF | Cross-Attn.
Table 1: Category-level object shape and pose estimation methods. TransPoser
realizes neural optimization that effectively weighs multiple viewpoints and
leverages learned momentum, which enables efficient and accurate joint object
shape and pose estimation from sequential observations. We also introduce an
efficient neural shape representation, DeepDDF, for this.

##  1 Introduction

Rigid object shape and pose estimation from a sequence of visual observations
is a core task of computer vision. For instance, a mobile robot in a room
would need to constantly recognize and localize objects such as chairs and
tables in the environment that can be moved from time to time. A natural
sensory input to this would be RGB-D images sequentially acquired as the
observer moves around in a scene. Recovery of the complete object shape and
its pose would be essential for situational awareness which requires correct
anticipation of obstructions by and understanding of the utility of objects in
the scene.

A wide variety of methods have been proposed to achieve this joint shape and
pose estimation. A representative approach is object SLAM [25, 40], which
introduces rigid objects into the simultaneous localization and mapping
paradigm so that individual objects can be isolated and localized. These
methods either assume that the objects found in the scene are known and their
shapes can be pre-acquired (_i.e_., detection and mapping) or only reconstruct
the visible parts of them. Recent works have tackled joint estimation of the
complete object shape and its pose from the observed RGB-D image sequence [39,
30]. These methods, however, rely on conventional non-linear optimization,
which fundamentally makes sequential online estimation hard.

As shown in Fig. 1, in this paper, we propose to learn to jointly estimate the
shape and pose of objects. We formulate this joint estimation with neural
networks so that the estimation can be achieved with forward inference. This
lets us avoid complex non-linear optimization and makes the joint estimation
amenable to potential on-board implementation. Most important, it lets us
fully leverage the structured bias underlying category-level object pose
estimation as a learned estimator.

Our method is based on two key ideas. The first is to represent the shape in
the 2D view-space. For this, we introduce Deep Directional Distance Function
(DeepDDF), a neural network that outputs the depth image of an object given
the camera viewpoint and viewing direction. This enables fast error
computation with the observed depth images, which is essential for efficient
estimation of the shape and pose. DeepDDF is a category-level model, trained
on 3D shapes of the same category and lets us reconstruct the complete shape
of a category instance (_e.g_., a particular chair) from the observed RGB-D
images.

The second is to formulate the joint estimation itself as a Transformer [35],
which we refer to as TransPoser. We fully leverage the two characteristics of
Transformer: tokenization and multi-head attention. Tokenization lets us
encode sequentially acquired observations as discrete tokens of variable
cardinality. This is essential for realizing an optimization that can handle a
variable number of observations as they come in. The attention scheme lets us
learn to transform the observations and predicted shapes into an embedding
space most suitable for minimizing their discrepancy. Most important, they let
us weigh the contribution of each observation and the previous estimates,
_i.e_., they realize a learned weighted optimization with learned momentum.

Our method seamlessly integrates DeepDDF and TransPoser to estimate the shape
and pose of an object captured in a sequentially obtained set of RGB-D images.
We conduct extensive experiments to evaluate the effectiveness of DeepDDF and
TransPoser on both synthetic and real RGB-D data. We compare with the few
competing methods that achieve the same task albeit with conventional non-
linear optimization. The experimental results show that DeepDDF achieves high
accuracy as a category-level object shape representation and TransPoser
efficiently and accurately estimates both the shape and pose of those objects.
To our knowledge, TransPoser achieves state-of-the-art accuracy on this
challenging task. We believe DeepDDF and TransPoser will serve as a sound
foundation for effective joint object shape and pose estimation, which has
implications in a broad range of areas including robotics, VR/AR, and
autonomous driving.

##  2 Related Work

##### Joint Object Pose and Shape Estimation

Estimation of object shape and its pose from a set of visual observations,
most often captured sequentially by a moving observer (_e.g_., mobile robot),
is a fundamental task for visual scene understanding. Most approaches to this
task assume RGB or RGB-D video as input. Online methods, such as object-aware
SLAM [25, 40], have also been introduced but none of them recover the complete
object shape and instead result in crude approximations or partial
reconstructions.

These past works naturally rely on iterative non-linear optimization, which
necessitates repeated evaluation of the prediction error between the
hypothesized object shape with predicted pose and the observations [32, 39, 5,
12, 29, 24]. Vid2CAD [24], for instance, projects a CAD model to the multi-
view observations to compute the loss for pose update. Other methods [39, 30]
employ neural 3D shape representations such as DeepSDF [28] to represent
unknown shapes. ELLIPSDF [30], for instance, adopts a bi-level object shape
model consisting of an ellipse and an SDF for estimation from point clouds
observed as RGB-D videos.

These methods, however, fundamentally suffer from high computational cost
mainly due to their reliance on conventional non-linear optimization.
Conventional numerical non-linear optimization requires numerous steps of
updates that also limit their ability to adapt to sequentially incoming
observations. The shape representations adopted by these studies also demand
high computational cost for prediction error evaluation as 2D views must be
explicitly rendered after the 3D representation is instantiated. Numerous
iterations of this causes prohibitive cost for any possibility of online
computation. We tackle joint object shape and pose reconstruction by
performing optimization sequentially under a practical setting where new
observations are added online. As shown in Tab. 1, we leverage a data-driven,
efficient, and accurate neural optimization and neural shape representation
which makes the estimation just a handful of forward inference passes.

##### Neural Shape Representation

An efficient and accurate object shape representation is essential for
iterative optimization for shape estimation as it dictates the computational
cost of a single iteration. Commonly used representations such as point clouds
[33, 38, 8, 7, 21] and voxels [32, 5] are inefficient as they first need to be
instantiated in 3D before their 2D depth images can be rendered. Recently,
implicit neural representations (INRs) have been introduced as a continuous
but compact shape representation [5, 29, 30]. Typical examples of INRs, such
as DeepSDF [28] and Occupancy Networks [26], have been used in conjunction
with optimization pipelines [29, 12, 30], but they still require substantial
computational and memory costs for rendering. The proposed DeepDDF directly
outputs depth images from its latent representation. In contrast to similar
approaches realizing direct shape evaluation [42, 2, 41], the key novelty of
our neural shape representation lies in the enforcement of multi-view
consistency with a low-cost voxel representation placed before the depth image
decoder. We believe this idea of combining low-res 3D and high-res 2D in a
single neural representation will have implications in applications beyond
joint shape and pose estimation.

##### Learning-based Optimization

Learning-based optimization can efficiently optimize intractable cost
functions in a data-driven fashion [13, 1, 10, 19]. Numerous methods have been
introduced for learning to optimize 2D alignment problems [36, 9, 20] or
simple optimization benchmarks [13, 14]. Its application to object pose
estimation is, however, relatively underexplored. Notably, Ma _et al_. [23]
tackled the ill-posed problem of estimating object pose from monocular 2D
observations with learnable optimization networks. Their work is, however,
limited to single-image inputs and uses simple LeNet layers that directly
regress $\Delta\bm{x}$ based on the current estimates. In contrast, our method
extends a Transformer architecture to learn a challenging, multi-view
optimization problem using learnable momentum, which allows us to robustly and
efficiently optimize the complex objective by tapping into the structured bias
of category-level shape and pose estimation.

TransPoser generalizes these neural optimization approaches in two fundamental
ways. It infers the update based not only on the current estimate but also on
those of past iterations. It also takes in multiple observations from
different views as a discrete input sequence. This enables iterative
optimization with learned weights on the multi-view images and achieves this
with a learned momentum that fully leverages the structured bias in
optimization of joint shape and pose estimation at the category level.

##  3 Joint Shape and Pose Estimation

Our goal is to jointly estimate the complete shape and pose of a rigid object
observed in a sequentially growing set of RGB-D images captured as the
viewpoint and viewing direction changes. The key challenges underlying this
task is the reconstruction of the complete shape of an object from its partial
depth observations each captured from a different viewing direction and the
estimation of object pose in a canonical coordinate frame. We define the
canonical coordinate frame for each object category (object coordinate frame)
and estimate the object pose as the 3D translation, 3D rotation, and 3D scale
from it in the first observed depth image (world coordinate frame). The
viewpoint and viewing directions for each RGB-D observation with respect to
those of the first depth image can be computed from background features
(_e.g_., regular SLAM). As such, the relative observer motion can be safely
assumed to be known.

Jointly solving for the shape reconstruction and pose estimation naturally
necessitates an optimization that evaluates the discrepancy between the
hypothesized object shape in a predicted pose. This requires repeated
rendering of a depth image of the predicted object which can easily become the
bottleneck of the joint estimation. Conventional 3D object representations
including mesh models, signed distance functions, and occupancy volumes,
regardless of whether they can be evaluated with forward inference on a
network, add significant burden to this process as the depth image needs to be
rendered from the instantiated 3D object.

We instead introduce a novel shape representation that directly outputs a
depth image of the object for a given viewpoint and viewing direction. The
proposed DeepDDF is inherently a 2D view-based shape representation that
bypasses the need for 3D shape rendering. The output depth image can be
directly compared with the observed depth image, which significantly cuts down
the computational burden of the joint shape and pose estimation.

The joint estimation is a highly complex non-linear optimization whose
parameters consist of the 3D shape encoded with the DeepDDF as a low-
dimensional latent code and also the 9 degrees-of-freedom (DoF) pose
parameters. Notably, we estimate the 3D scale together with the usual 6-DoF
pose encoding the 3D translation and 3D rotation of the object. The challenge
also lies in the fact that we would like to fully leverage the observations
which are obtained sequentially by gradually updating the parameters as the
new observations come in.

This necessitates an efficient optimizer that can take in a variable sequence
of input RGB-D observations and that can dynamically weigh the contributions
of the sequentially added observations. We achieve this by deriving a novel
Transformer, which we refer to as TransPoser, that directly outputs the
updates to the shape and pose parameters for a given partial sequence of
observed RGB-D images. Once learned, one step of the optimization is a single
forward inference, which dramatically cuts down the computational cost
compared to conventional non-linear optimization based joint estimation that
needs to restart the iteration every time a new observation is captured.

Figure 2: DeepDDF returns the inverse distance to a object surface it
represents for a given latent code, viewpoint, and viewing direction. It
leverages a low-resolution 3D latent volume, which is decoded from the latent
code with the Object Instance Decoder, and samples rays in this volume for a
give viewpoint and viewing direction using the Camera Ray Sampler. These
decoder and sampler collectively enable efficient and multi-view consistent 3D
representation with direct 2D image-space output.

##  4 Deep Directional Distance Function

We introduce Deep Directional Distance Function (DeepDDF) as a novel 3D
representation of rigid objects. Similar to DeepSDF [28], DeepDDF is a
decoder-based model trained on category-level object instances. Each object
category is represented with one DeepDDF. In sharp contrast to other 3D object
representations including DeepSDF, DeepDDF directly outputs a depth image from
a latent code encoding the object instance by conditioning on a given
viewpoint and viewing direction. For each camera ray
$\mbox{\boldmath$d$}_{\text{ray}}$ from a viewpoint
$\mbox{\boldmath$p$}_{\text{ray}}$, DeepDDF returns the distance $D$ from the
camera

$D=f_{\text{DDF}}(\mbox{\boldmath$p$}_{\text{ray}},\mbox{\boldmath$d$}_{\text{ray}},\mbox{\boldmath$z$})\,,$
(1)

where $z$ is the latent code of the object instance.

Figure 3: TransPoser realizes neural optimization with a Transfomer encoder-
decoder architecture. The multi-view sequential observations are tokenized and
cross-attended with queries representing past estimates and current view,
which enables a learned weighted optimization with learned momentum in
iterated forward inference passes.

DeepDDF enables direct evaluation of the object shape and pose against
observations in the image-space as it bypasses rendering (projection) of 3D
geometry in the 2D view. Two fundamental challenges, however, need to be
resolved to achieve this direct 2D view-dependent representation. The first is
that the depth values for each camera ray across different viewpoints must be
consistent, _i.e_., must represent the same rigid object shape. The second is
the large number of samples required to span the entire space of the inputs to
the decoder $\mbox{\boldmath$p$}_{\text{ray}}$,
$\mbox{\boldmath$d$}_{\text{ray}}$, and $z$.

We overcome these challenges by introducing a sparse 3D voxel grid aligned
with the target object space. Inspired by DeepVoxels [31], which improves the
accuracy of RGB novel view synthesis with voxel features, we introduce a 3D
voxel representation of the rough object shape before the decoder which
ensures multi-view consistency of output depth images. This 3D voxel can be of
very low-resolution as it does not encode the 3D shape itself.

As shown in Fig. 2, the latent code $z$ is decoded into latent voxels by an
object instance decoder $f_{\text{d}}$, and the voxels on the camera rays,
defined by its 3D viewpoint $\mbox{\boldmath$p$}_{\text{ray}}$ and 3D
direction $\mbox{\boldmath$d$}_{\text{ray}}$, are sampled by a camera ray
sampler $f_{\text{mlp}}$. This low-resolution but three-dimensional voxel
corresponding to the object instance encoded by the latent code $z$ enforces
consistency across the camera rays and makes the output depth values $D$
multi-view consistent. It also enables reduction in the number of samples for
$\mbox{\boldmath$p$}_{\text{ray}}$, $\mbox{\boldmath$d$}_{\text{ray}}$, and
$z$ by ensuring this multi-view consistency.

We implement $f_{\text{d}}$ with a 3D deconvolution network and
$f_{\text{mlp}}$ with an MLP. Please see the supplemental material for the
full network architecture. The object instance decoder $f_{\text{d}}$ decodes
the latent code $z$ into a 3D voxel array $G_{z}$ encaging the target shapes.
Each voxel $g$ of $G_{z}$ stores a feature vector encoding the 3D shape
corresponding to $z$. The camera ray sampler $f_{\text{mlp}}$ regresses the
inverse distance from a viewpoint at a unit distance to the object surface

$D^{-1}=f_{\text{mlp}}\left(\bigoplus[\mbox{\boldmath$g$}^{*}_{1},\mbox{\boldmath$g$}^{*}_{2},...,\mbox{\boldmath$g$}^{*}_{N}]\right)\,,$
(2)

where
$\bigoplus[\mbox{\boldmath$g$}^{*}_{1},\mbox{\boldmath$g$}^{*}_{2},...,\mbox{\boldmath$g$}^{*}_{N}]$
denotes the concatenated feature vectors of $N$ voxels
$\\{\mbox{\boldmath$g$}^{*}_{i}\\}^{N}_{i=1}$ sampled along the camera ray
represented by $\mbox{\boldmath$p$}_{\text{ray}}$ and
$\mbox{\boldmath$d$}_{\text{ray}}$. Regressing the inverse distance prevents
$f_{\text{mlp}}$ from returning infinity for rays outside the object region.

We train DeepDDF by optimizing the latent code $z$ and the network parameters
[28] by minimizing the loss between the predicted depth and surface normal
from the ground truth

$\displaystyle\mathcal{L}_{\text{DDF}}$
$\displaystyle=\mathcal{L}_{\text{dst}}+\lambda_{\text{nrm}}\mathcal{L}_{\text{nrm}}+\lambda_{\text{ltn}}||\mbox{\boldmath$z$}||\,,$
(3) $\displaystyle\mathcal{L}_{\text{dst}}$
$\displaystyle=\left|D^{-1}_{\text{est}}-D^{-1}_{\text{gt}}\right|\,,$ (4)
$\displaystyle\mathcal{L}_{\text{nrm}}$
$\displaystyle=\text{MAE}(\mbox{\boldmath$n$}_{\text{est}},\mbox{\boldmath$n$}_{\text{gt}})\,,$
(5)

where $\mathcal{L}_{\text{dst}}$ is the discrepancy between the current depth
$D_{\text{est}}$ and ground truth $D_{\text{gt}}$, and
$\mathcal{L}_{\text{nrm}}$ is the mean absolute error of the estimated and
ground truth normal directions, $\mbox{\boldmath$n$}_{\text{est}}$ and
$\mbox{\boldmath$n$}_{\text{gt}}$, respectively. We compute the normal by
sampling the depth of the neighboring rays. The latent codes $z$ are
regularized with the term $||\mbox{\boldmath$z$}||$ and $\lambda_{\text{nrm}}$
and $\lambda_{\text{ltn}}$ control the weighting between the three terms which
are determined empirically. We used $1.0\times 10^{-2}$ and $1.0\times
10^{-4}$, respectively, in all experiments.

As shown in Fig. 2, we avoid redundant sampling by training the network with
viewpoints on a fixed-radius sphere which we refer to as the canonical sphere.
In our implementation, we set the radius to the diagonal distance of the
bounding box of a given object category, which we normalize to 1, and its
center at the origin of the object coordinate frame. When generating the depth
for viewpoints inside or outside this sphere, we use the intersection of the
camera ray with this sphere and add the distance to this sphere to the depth
for the given point and direction on the canonical sphere.

##  5 TransPoser

Let $x$ denote a vector of parameters of joint shape and pose estimation: the
object shape as the latent code of DeepDDF $z$ and its 9D pose of 3D
translation, 3D rotation, and 3D scaling. Note that the 3D shape represented
by DeepDDF is internally normalized to have a unit diagonal length for its 3D
bounding box. The 3D scaling of DeepDDF is thus relative to this diagonal
length, which increases the representation power of DeepDDF. The 3D scaling
estimated for pose then further aligns the predicted 3D shape to the
observations. This cascaded scaling estimation helps express a variety of 3D
shapes and their accurate poses.

Given the current estimate $\mbox{\boldmath$x$}^{t}$ at step $t$, the goal of
TransPoser is to return the update $\Delta\mbox{\boldmath$x$}^{t}$ that
refines the shape and pose
$\mbox{\boldmath$x$}^{t+1}=\mbox{\boldmath$x$}^{t}+\Delta\mbox{\boldmath$x$}^{t}$
computed from the multi-view observed RGB-D images. We denote the number of
images at step $t$ with $K^{t}$. As depicted in Fig. 3, TransPoser is an
encoder-decoder Transformer [35] and uses DeepDDF to synthesize the depth
image for each $x$. Each of the tokens to the TransPoser encoder is generated
from the observed depth image for the $k-$th view $\mbox{\boldmath$Y$}_{k}$
and the corresponding predicted depth image for the $t-$th step
$\mbox{\boldmath$Y$}_{k}^{t}$.

The depth images $\mbox{\boldmath$Y$}\in\mathbb{R}^{H\times W\times 1}$ are
first converted to images that represent object surface coordinates, which we
refer to as surface maps
$\widetilde{\mbox{\boldmath$Y$}}\in\mathbb{R}^{H\times W\times 3}$. Instead of
the depth to the surface in $Y$, each pixel of
$\widetilde{\mbox{\boldmath$Y$}}$ stores the 3D location in the normalized
object coordinate space (NOCS) [37] representing the same position on the
surface. We also assume we have a mask for the object region, which usually
can be supplied through semantic segmentation. The pixels outside this mask
are all zeroed out in the observation. The predicted images, and thus their
surface maps, also have zero values rendered for the background.

This use of a surface map representation is one of the key novel ideas
underlying TransPoser. By consolidating the observed and predicted depth
values for their individual camera coordinates into the same object coordinate
frame, but in a spatial image, TransPoser is able to compute self- and cross-
attention without having to overcome the differences in coordinate frames. It
also avoids “projection breakdown” caused by 2D convolutions on depth images
[17].

We compute 3 surface maps from the 2 depth images, namely the observed surface
map, predicted surface map, and their difference. The last encodes the
discrepancy in the surface point coordinates for each pixel by simply taking
the difference between the first two. By explicitly preparing this difference
surface map, we lessen the burden on the network to learn to evaluate the
prediction error. These 3 surface maps, each a 3 channel image of surface
coordinates, are input to a backbone feature extraction network independently
as a 4 channel image with the last filled with the object mask. The feature
maps are concatenated and passed through a fully-connected layer to produce
the corresponding token denoted as $\mbox{\boldmath$e$}^{t}_{k}$ for the
encoder and $\mbox{\boldmath$m$}^{t-1/t}_{k}$ for the decoder. We use
ResNet-18 [15] as the backbone feature extractor for both the encoder and
decoder. In addition, we also encode the viewing condition of each token by
adding a 7D vector consisting of the 3D camera viewpoint in the object
coordinate frame, the 3D camera ray direction towards the center of the object
bounding box, and the ratio between the diagonal of the entire image to the
diagonal of the bounding box. Note that we do not use regular positional
encodings [35]. The observations are image sets, even though they are
sequentially acquired, and we explicitly encode their viewing conditions but
not their order.

Self-attention in the encoder transforms each view and associated prediction
and error while consolidating information across the views. This effectively
learns the weighting between the views. Consider, for example, estimating the
pose of a chair. The optimization would ideally put more weight on the
prediction errors on views that capture both the backrest and seat, compared
to views only capturing the handrail or the legs. The encoder computes self-
attention between tokens $\\{\mbox{\boldmath$e$}_{k}^{t}\\}_{k=1}^{K^{t}}$ of
multiple $K^{t}$ views for the current estimate $\mbox{\boldmath$x$}^{t}$. The
encoder has three layers of self-attention and the outputs are denoted with
$\\{{\mbox{\boldmath$e$}^{\ast}}_{k}^{t}\\}_{k=1}^{K^{t}}$.

Figure 4: The 2D PCA subspace spanned by the learned latent codes of DeepDDF.
It consists of clusters of similar objects and linear interpolation in the
space creates objects with shape features inherited from end points. These
results show that DeepDDF successfully extracts the structured coherence of
the shape space of the object category.

The decoder query tokens are each computed from the same combination of
observed, predicted, and discrepancy depth images. Unlike the encoder tokens,
they represent those of different time steps. As depicted in Fig. 3, our key
idea is to formulate the decoder to learn a momentum-guided update from the
self-attended current estimates $\\{\mbox{\boldmath$e$}^{\ast
t}_{k}\\}_{k=1}^{K^{t}}$ (_i.e_., encoder outputs) and past estimates
$\\{\mbox{\boldmath$m$}^{t-1}_{k}\\}_{k=1}^{K^{t-1}}$ and also the current
view $\mbox{\boldmath$m$}^{t}_{K^{t}}$ (_i.e_., decoder queries), if the
camera moved. When the camera moves between the two optimization steps $t-1$
and $t$, $K^{t}$ and $K^{t-1}$ are not the same and we simply use the current
estimates $\mbox{\boldmath$x$}^{t}$ as viewing condition encoding of the
latest view $\mbox{\boldmath$m$}^{t}_{K^{t}}$. The decoder has three layers of
self-attention and three layers of cross-attention. The self-attention layers
effectively weigh multi-view contributions of the past estimates similar to
the self-attention in the encoder. Most important, the cross-attention layers
weight contributions across the current and past estimates which effectively
amounts to a learned momentum term to output the decoder tokens
$\\{\mbox{\boldmath$u$}^{\ast t}_{k}\\}$. These output tokens are average-
pooled, passed through an MLP and converted back into the world coordinate
frame (the first view) to finalize the additive update
$\Delta\mbox{\boldmath$x$}^{t}$ to the current estimate
$\mbox{\boldmath$x$}^{t}$.

TransPoser does not have layer normalizations which are common in regular
Transformers [35]. This is because layer normalization corresponds to
normalizing the gradient vector in regular SGD and would cause the network to
ignore error minimization. Omitting layer normalization, however, has as a
side-effect of making the training unstable. We use T-Fixup [16]
initialization to resolve this.

The loss of TransPoser is the component-wise error between the updated
estimate $\mbox{\boldmath$x$}^{t}+\Delta\mbox{\boldmath$x$}^{t}$ and the
ground truth values $\mbox{\boldmath$x$}_{\text{gt}}$, similar to that of Ma
_et al_. The details of update formulae and the loss function for each
parameter value are described in the supp. material. Unlike Ma _et al_.,
however, we use one common network among all iterations. We update the network
parameters after each step $t$.

Model | Comp. Cost | Shape Accuracy
---|---|---
Time | Memory | Chair | Table
DIST [22] | 0.27 sec | 5.4 GB | 1.28 | 1.43
DeepDDF | 0.04 sec | 5.0 GB | 0.83 | 0.91
Table 2: Experimental comparison of shape reconstruction speed and accuracy of DeepDDF with DIST. Accuracy is shown in Chamfer distances $(10^{-3})$ from ground truth. Our DeepDDF realizes more accurate reconstruction in an order of magnitude less time. Obs. | DIST [22] | DeepDDF | Obs. | DIST [22] | DeepDDF
---|---|---|---|---|---
|  |  |  |  |
Figure 5: Example shape reconstruction with conventional optimization using DIST and DeepDDF. The results of DeepDDF are qualitatively more accurate. Model | Cls avg. | Cabinet | Chair | Display | Table
---|---|---|---|---|---
T | R | S | T | R | S | T | R | S | T | R | S | T | R | S
Baseline based on [23] | 1.58 | 1.70 | 4.24 | 1.98 | 1.56 | 4.97 | 1.17 | 0.60 | 3.17 | 1.72 | 1.05 | 3.03 | 1.46 | 3.59 | 5.79
w/o Dec. | 1.45 | 1.95 | 3.55 | 1.45 | 1.54 | 3.56 | 1.17 | 0.55 | 2.59 | 1.49 | 1.17 | 2.44 | 1.70 | 4.55 | 5.60
TransPoser | 1.38 | 1.44 | 3.43 | 1.15 | 1.40 | 3.59 | 0.96 | 0.54 | 2.68 | 1.39 | 1.02 | 2.34 | 2.02 | 2.82 | 5.09
Table 3:  Ablation study of TransPoser architecture using synthetic data. The
baseline [23] corresponds to an architecture without multi-view and momentum.
“w/o Dec.” stands for TransPoser without the decoder, _i.e_., without learned
momentum. Metrics for translations (T), rotations (R), and shapes (S) are mean
squared error $(10^{-2})$, Riemannian distance $(10^{-1})$, and Chamfer
distance $(10^{-3})$, respectively. For almost all categories, the proposed
TransPoser, which learns to weigh multi-view observations and to leverage
learnable momentum, achieves highest accuracy.

##  6 Experimental Results

We thoroughly evaluate the effectiveness of DeepDDF and TransPoser with large-
scale synthetic and real datasets. Please refer to the supp. material for
details on the evaluation setup and additional results.

###  6.1 DeepDDF

We train one DeepDDF network for each category using CAD models from ShapeNet
[6]. As training data, we rendered $512\times 512\text{px}$ depth images from
virtual cameras of $60^{\circ}$ field-of-view, located randomly on the unit
sphere centered at the target model. We synthesized 200 pairs of depth and
normal maps for each of 1,465, 3,196, 986, and 4,817 CAD models in the
Cabinet, Chair, Display, and Table categories, respectively. We used the same
split of the Chair and Table categories as DeepSDF [28] except for several CAD
models with transparent parts or inverted normals.

Figure 4 visualizes the 2D PCA subspace spanned by the latent codes of DeepDDF
for Chair. The latent space contains clusters of similar shapes in the same
object subcategory, such as single sofas, and interpolation synthesizes novel
shapes inheriting the features of the end points. This demonstrates that
DeepDDF is able to extract a structured latent space of object instances in
the same category.

We compare the accuracy and efficiency of DeepDDF with DIST [22] on shape
reconstruction. DIST is also a single image shape estimator, but uses DeepSDF
with iterative sphere tracing to render the depth image. Unfortunately, we
cannot compare with past methods that directly output distance [42, 2, 41] as
their original implementations were not available. We use the Chair and Table
categories which contain challenging geometric structures like thin legs and
has a wide variety in part features. We assume that both the camera and object
poses are known, and optimize the latent code to minimize the error between an
observed and a synthesized depth image with regular non-linear optimization
with Adam optimizer to directly compare the two in their effectiveness purely
in shape representation.

Table 2 shows quantitative results of this experimental comparison, in
computational cost and reconstruction accuracy. We evaluate memory usage and
inference times to generate an $256\times 256\text{px}$ image for
computational costs, and the Chamfer distance between the ground truth and the
estimated point clouds for reconstruction accuracy. The results show that our
DeepDDF is superior to DIST in both aspects. DeepDDF achieves a magnitude of
faster estimation thanks to its direct depth image output. Figure 5 shows
reconstructed shapes. These results show that DeepDDF also reconstructs
qualitatively more accurate shapes.

###  6.2 TransPoser

We evaluate the effectiveness of TransPoser for joint shape and pose
estimation on synthetic data, and also compare it with past methods on ScanNet
[11] which is a dataset of scanned real scenes and objects.

##### Ablation Studies on Synthetic Data

To train TransPoser, we randomly generate camera trajectories moving around
the object, and sample 5 viewpoints along each of them to render $128\times
128\text{px}$ depth images of $50^{\circ}$ field-of-view. TransPoser is
trained to optimize the shape and pose by 10 iterations during which the
number of observations increases at every 2 iterations. The pose is
initialized randomly, and the shape is initialized by the mean latent code of
the shape category.

We train one TransPoser network for each category using CAD models from
ShapeNet. DeepDDF is pre-trained, and frozen during the training of
TransPoser. We compare TransPoser with the following models that each
correspond to ablation of key architectural components. The ablated models are
a baseline model composed of a backbone network and an MLP head, and
TransPoser without the decoder. The baseline model is based on Ma  _et al_.
[23], which does not exploit the relative postures of the multiple viewpoints
nor consider the momentum during the optimization. This model inputs each
feature $\mbox{\boldmath$e$}_{k}^{t}$ from the backbone network to the MLP
head, calculates the updated value $\Delta\mbox{\boldmath$x$}_{k}^{t}$ for
each viewpoint $k$ independently, and then takes their average as
$\Delta\mbox{\boldmath$x$}^{t}$. TransPoser without the decoder only weighs
the multiple viewpoints. This model averages the encoder outputs
$\\{{\mbox{\boldmath$e$}^{\ast}}_{k}^{t}\\}_{k=1}^{K^{t}}$, and inputs it to
the MLP head.

We evaluate the errors in translation, rotation, and shape estimation by the
mean squared error, the Riemannian distance [27], and the bi-directional
Chamfer distance normalized by the object scale, respectively. On computing
the shape error by Chamfer distance, we generated a point cloud corresponding
to the estimated latent code by integrating DeepDDF depth-maps rendered from 5
viewpoints distributed uniformly around the object.

Table 3 shows the estimation errors for each category. These results
demonstrate quantitatively that the proposed TransPoser achieves the highest
accuracy in most of the metrics, while achieving a faster convergence.
TransPoser without the decoder also converges as fast as TransPoser, but gets
trapped in local minima as shown in the rotation error. This implies that
referencing the past optimization steps through the learned cross-attention by
the decoder (_i.e_., the learned momentum) helps converge to a better minima.

##### Real-world Data

Method | Cls Avg. | Cabinet | Chair | Display | Table
---|---|---|---|---|---
ELLIPSDF [30] | 89.0 | 91.0 | 90.6 | 96.9 | 77.3
TransPoser | 94.3 | 92.4 | 98.5 | 93.8 | 92.4
Table 4: Accuracy of shape estimation for ScanNet [11]. TransPoser achieves
highest accuracy for all categories.

We evaluate the accuracy and efficiency of TransPoser on ScanNet dataset [11],
which consists of RGB-D videos of real-world indoor scenes. Following ELLIPSDF
[30], we use the ground-truth object pose, shape, and mask as well as the
camera poses. The ground-truth object poses and shapes are provided by
Scan2CAD [4].

Method |  Online |  Cls avg. |  Cabinet |  Chair |  Display |  Table
---|---|---|---|---|---|---
Scan2CAD [4] |  | 31.7 | 34.0 | 44.3 | 17.9 | 30.7
RayTran [34] |  | 42.1 | 36.2 | 59.3 | 30.4 | 42.5
Vid2CAD [24] | ✓ | 38.8 | 23.8 | 64.6 | 37.7 | 28.9
ELLIPSDF [30] | ✓ | 39.6 | - | - | - | -
TransPoser | ✓ | 59.3 | 45.4 | 91.2 | 45.0 | 55.7
Table 5: Accuracy of pose estimation for ScanNet [11]. TransPoser achieves
higher accuracy for all categories.

TransPoser is trained to optimize the shape and pose by 10 iterations, by
fine-tuning the model pre-trained with the synthetic dataset. On fine-tuning,
we trained TransPoser using randomly selected 5 frames capturing the object
from ScanNet. As the camera moves every $50\text{\,}\mathrm{cm}$, we select a
frame from the segment such that it captures the target as large as possible,
and add it as the new observation.

Tables 5, 4 and 6 quantitatively compares the proposed TransPoser with state-
of-the-art methods. TransPoser clearly outperforms other methods including
ELLIPSDF [30] that also estimates the pose and shape code. TransPoser required
0.06 sec for each optimization step, and 0.61 sec for a total of 10 iterations
with GeForce RTX 2080 Ti. This is approximately 10 times faster than those of
ELLIPSDF which required 0.75 sec for its first SGD step. Note that the exact
instances used in the ELLIPSDF are not available [3] and we used the test
split of Scan2CAD instead. Examples in Fig. 6 demonstrate that TransPoser can
reconstruct plausible 3D shapes even from partial observations. These results
show that our neural optimization model successfully learns to leverage the
structured bias in category-level shape pose estimation.

Observations | GT | Est.
---|---|---
|  |
|  |
|  |
|  |
|  |
|  |
Figure 6: Example shape and pose estimation results for ScanNet [11].
TransPoser can handle heavy occlusions.

##  7 Conclusions

In this paper, we introduced TransPoser, a novel neural optimization method
for joint estimation of object shape and pose, particularly for sequentially
acquired depth images. The method leverages DeepDDF which is a novel category-
level neural 3D shape representation that can directly output a depth image of
the object instance with consistent multi-view geometry. We showed that
DeepDDF alone is an efficient and accurate shape representation that can
significantly speed up the joint estimation. TransPoser, through its learned
weighted optimization as forward inference and its learned momentum that
extracts structured bias in category-level joint shape and pose estimation,
achieves state-of-the-art accuracy as demonstrated in the extensive
comparative experiments. We believe these results realize significant advances
on this challenging key task for many applications in computer vision,
robotics, and AR/VR.

##### Limitation

DeepDDF as a shape representation is limited to static, rigid objects and
TransPoser assumes static object pose throughout its inference. We plan to
expand both to handle deformable objects and dynamic relative movements of
objects in our future work. The methods also are limited to depth images as
inputs and do not leverage the often accompanied RGB appearance information.
We plan to incorporate appearance features in the two models, which we believe
will further improve the accuracy of shape reconstruction.

##### Acknowledgement

This work was in part supported by JSPS 20H05951, 21H04893, JST JPMJCR20G7,
and RIKEN GRP.

##  Appendix A Details of DeepDDF

This section describes the details of DeepDDF and its experimental validation.

###  A.1 Details of Network Architecture

As shown in Fig. A.7, DeepDDF consists of the object instance decoder
$f_{\text{d}}$ and the camera ray sampler $f_{\text{mlp}}$. We use 256 as the
dimensionality of the latent code. The decoder $f_{\text{d}}$ first projects
the latent code to a 512 dimensional vector with a fully connected layer, and
then transforms it to a $16\times 16\times 16\times 32$ tensor with four 3D
transposed convolutions with leaky ReLU. We set the kernel size, the stride,
and the number of output channels of each 3D transposed convolution layer to
4, 2, and half the number of input channels, respectively.

The $16\times 16\times 16\times 32$ tensor represents a $16\times 16\times 16$
latent voxel grid covering the 3D cube of edge length $1$ in which each
ShapeNet CAD model is normalized to fit. The camera ray sampler
$f_{\text{mlp}}$ casts a viewing ray through a camera pixel of interest from
the viewpoint on the unit canonical sphere towards the latent voxel grid. It
samples 32 points equally distributed along the ray at distances in the range
$[1-\sqrt{3}/2,1+\sqrt{3}/2]$ from the viewpoint. $f_{\text{mlp}}$ obtains the
features at the sampling points with trilinear interpolation of neighboring
voxels, and concatenates the sampled features into a 1024 dimensional vector.
As shown in Fig. A.7, this 1024 dimensional vector is then sent to a feed-
forward network consisting of 12 fully connected layers. We concatenate the
250 dimensional vector after the first FC layer to the input of the 7th layer.
The scalar output of the feed-forward network is mapped to a non-negative
value by Softplus which corresponds to the inverse distance $D^{-1}$.

###  A.2 Details of DeepDDF Training

DeepDDF is trained with 200 pairs of depth and normal images synthesized from
different viewpoints per instance. Each instance is positioned so that its
bounding box center coincides with the origin of the coordinate frame and its
sides align with the axes. At each epoch, for each training instance, we
randomly select 1 out of the 200 pairs, and randomly sample pixels in the
viewpoint to evaluate the loss function. For efficiency, we apply importance
sampling of pixels with a probability of $0.2$ inside the object mask and with
a probability of $2\times 10^{-3}$ outside of it. We also dilate the object
mask in advance so that pixels around the object boundary are also sampled. We
trained the networks for 3,000 epochs for the Table category, and 10,000
epochs for the other categories. We use Adam optimizer [18] with a learning
rate of $1.0\times 10^{-4}$ and $5.0\times 10^{-4}$ for the network parameters
and the latent code, respectively.

Figure A.7: DeepDDF network architecture. The rectangles denote tensors and
the arrows with FC, T3D, T3D+LR, and SP labels denote a fully-connected layer,
a transposed 3D convolution, a transposed 3D convolution with leaky ReLU, and
a Softplus layer, respectively.

###  A.3 Ablation Study

Figure A.8 compares depth images reconstructed by DeepDDF and also by DeepDDF
with the latent voxel grid ablated. The two networks are trained with 35
object instances (_i.e_., CAD models) in the Chair category. As we used a
smaller number of object instances for this ablation study, the latent space
is sparsely populated, which makes their interpolations susceptible to
geometric inconsistencies. Figure A.8 shows depth images of an object instance
interpolated from the same pair of training samples using DeepDDF and DeepDDF
without the latent voxel grid. By comparing the front and the back views, we
can observe that DeepDDF without the latent voxel grid generates inconsistent
depth-maps, while the full DeepDDF generates plausible object geometry. These
results clearly show that the latent voxel grid is essential for generating
objects shape with multi-view geometric consistency.

Front | Back | Front | Back
---|---|---|---
|  |  |
(a) w/o the latent voxel grid | (b) w/ the latent voxel grid
Figure A.8: Front and back views of an object instance interpolated in the
latent spaces of DeepDDF with and without the latent voxel grid. Both shapes
are interpolated as the midpoint of the latent codes of the same set of object
instances in the training set. The results show that the latent voxel grid is
essential to ensure multi-view geometric consistency in DeepDDF.

###  A.4 Details of Shape Reconstruction Experiment

In the shape reconstruction experiment reported in Table 2 and Figure 5 of the
main text, we reconstruct the shape from a single-view depth image given known
camera and object poses. We optimize the latent code $z$ to minimize the
differences between the observed and the synthesized inverse distance images
$\mbox{\boldmath$I$}_{\rm obs}$, $\mbox{\boldmath$I$}_{\rm
est}(\mbox{\boldmath$z$})$

$\mbox{\boldmath$z$}^{*}=\operatorname*{arg\,min}_{\mbox{\boldmath$z$}}\lambda_{\rm{d}}|\mbox{\boldmath$I$}_{\rm
obs}-\mbox{\boldmath$I$}_{\rm
est}(\mbox{\boldmath$z$})|+\lambda_{\rm{r}}||\mbox{\boldmath$z$}||\,,$ (6)

where $\lambda_{\rm{d}}$ and $\lambda_{\rm{r}}$ control the weighting between
the prediction error and the regularization on the latent code.
$\lambda_{\rm{d}}$ and $\lambda_{\rm{r}}$ are set to $10.0$ and $1.0$,
respectively. We use Adam Optimizer with a learning rate $1.0\times 10^{-3}$,
and perform 200 steps of optimization. For DIST [22], we used the pre-trained
models provided by the original authors, and set the hyperparameter values to
those used in the original implementation except for the following changes. We
omitted the normal error in the shape optimization according to Eq. 6. We also
set the number of sampling points $K$ used for rendering depth images by DIST
to 1, as we empirically found the original setting, namely $K=3$, noticably
decreased accuracy.

For the Chair and Table categories, we use the test split of DeepDDF which has
832 and 1,216 CAD models respectively. We render a $256\times 256\text{px}$
depth image of $60^{\circ}$ field-of-view from a random viewpoint on the unit
canonical sphere for each instance. To compute the reconstruction accuracy, we
generated a point cloud corresponding to the estimated latent code by
integrating DeepDDF depth images rendered from 6 viewpoints distributed
uniformly around the object.

##  Appendix B Details of TransPoser

This section describes the details of TransPoser and its experimental
validation on synthetic and real-world data.

###  B.1 Details of Parameters and Loss Functions

We represent the translation and scale by 3-dimensional vectors
$\mbox{\boldmath$p$},\mbox{\boldmath$s$}\in\mathbb{R}^{3}$, respectively. The
3D rotation is represented by two 3-dimensional vectors
$\mbox{\boldmath$d$}_{\text{g}},\mbox{\boldmath$d$}_{\text{r}}\in\mathbb{R}^{3}$
that represents the two axes of the normalized object coordinate system as
done in FS-Net [8]. The shape is represented by the latent code
$\mbox{\boldmath$z$}\in\mathbb{R}^{256}$ of DeepDDF.

We update the current estimate
$\mbox{\boldmath$p$}^{t},\mbox{\boldmath$d$}_{\text{g}}^{t},\mbox{\boldmath$d$}_{\text{r}}^{t},\mbox{\boldmath$s$}^{t}$,
and $\mbox{\boldmath$z$}^{t}$ with their updates
$\Delta\mbox{\boldmath$p$}^{t},\Delta\mbox{\boldmath$d$}_{\text{g}}^{t},\Delta\mbox{\boldmath$d$}_{\text{r}}^{t},\Delta\mbox{\boldmath$s$}^{t}$,
and $\Delta\mbox{\boldmath$z$}^{t}$, respectively (_i.e_., the outputs of
TransPoser),

$\displaystyle\mbox{\boldmath$p$}^{t+1}$
$\displaystyle=\mbox{\boldmath$p$}^{t}+\Delta\mbox{\boldmath$p$}^{t}\,,$ (7)
$\displaystyle\mbox{\boldmath$d$}_{\text{g}}^{t+1}$
$\displaystyle=\text{normalize}(\mbox{\boldmath$d$}_{\text{g}}^{t}+\Delta\mbox{\boldmath$d$}_{\text{g}}^{t})\,,$
(8) $\displaystyle\mbox{\boldmath$d$}_{\text{r}}^{t+1}$
$\displaystyle=\text{normalize}(\mbox{\boldmath$d$}_{\text{r}}^{t}+\Delta\mbox{\boldmath$d$}_{\text{r}}^{t})\,,$
(9) $\displaystyle\mbox{\boldmath$s$}^{t+1}$
$\displaystyle=\mbox{\boldmath$s$}^{t}\circ\Delta\mbox{\boldmath$s$}^{t}\,,$
(10) $\displaystyle\mbox{\boldmath$z$}^{t+1}$
$\displaystyle=\mbox{\boldmath$z$}^{t}+\Delta\mbox{\boldmath$z$}^{t}\,,$ (11)

where $\text{normalize}(\cdot)$ normalizes the given vector to unit magnitude,
and $\circ$ denotes the Hadamard product.

The loss of TransPoser is the discrepancy between the updated estimates and
their corresponding ground truth values. We calculate the difference between
each estimate
$\mbox{\boldmath$p$}^{t+1},\mbox{\boldmath$d$}_{\text{g}}^{t+1},\mbox{\boldmath$d$}_{\text{r}}^{t+1}\mbox{\boldmath$s$}^{t+1}$,
and $\mbox{\boldmath$z$}^{t+1}$ and their ground truth
$\mbox{\boldmath$p$}_{\text{gt}},\mbox{\boldmath$d$}_{\text{g}_{\text{gt}}},\mbox{\boldmath$d$}_{\text{r}_{\text{gt}}}\mbox{\boldmath$s$}_{\text{gt}}$,
and $\mbox{\boldmath$z$}_{\text{gt}}$, respectively,

$\displaystyle\mathcal{L}_{p}$
$\displaystyle=\text{MSE}(\mbox{\boldmath$p$}^{t+1},\mbox{\boldmath$p$}_{\text{gt}})\,,$
(12) $\displaystyle\mathcal{L}_{g}$
$\displaystyle=1-\text{COSSIM}(\mbox{\boldmath$d$}_{\text{g}}^{t+1},\mbox{\boldmath$d$}_{\text{g}_{\text{gt}}})\,,$
(13) $\displaystyle\mathcal{L}_{r}$
$\displaystyle=1-\text{COSSIM}(\mbox{\boldmath$d$}_{\text{r}}^{t+1},\mbox{\boldmath$d$}_{\text{r}_{\text{gt}}})\,,$
(14) $\displaystyle\mathcal{L}_{s}$
$\displaystyle=\text{MSE}(\mbox{\boldmath$s$}^{t+1},\mbox{\boldmath$s$}_{\text{gt}})\,,$
(15) $\displaystyle\mathcal{L}_{z}$
$\displaystyle=\text{MSE}(\mbox{\boldmath$z$}^{t+1},\mbox{\boldmath$z$}_{\text{gt}})\,,$
(16)

where MSE and COSSIM denote the mean squared error and the cosine similarity,
respectively. We train TransPoser with the sum of the errors over all
optimization steps

$\mathcal{L}=\sum_{t=1}^{T}(\lambda_{p}\mathcal{L}_{p}+\lambda_{g}\mathcal{L}_{g}+\lambda_{r}\mathcal{L}_{r}+\lambda_{s}\mathcal{L}_{s}+\lambda_{z}\mathcal{L}_{z})\,,\\\
$ (17)

where $\lambda_{p},\lambda_{g},\lambda_{r},\lambda_{s},\lambda_{z}$ control
the weighting between each estimation, and $T$ denotes the total number of
iterations.

###  B.2 Momentum with Cross-attention

TransPoser effectively learns momentum of the iterative joint shape and pose
estimation in the sense that it incorporates an adaptive amount of past
estimates and combines it with the current estimate to output an adjusted
update. This is achieved with the Transformer decoder of TransPoser, which
calculates updates based on the prediction errors at the current time step and
also the previous time step. Table 3 of the main text shows that the decoder
improves the reconstruction accuracy. We believe this learned momentum with
cross-attention can be essential for neural optimization and plan to explore
its use in other optimization tasks.

###  B.3 Details of Network Architecture

TransPoser consists of a ResNet-18 backbone, three-layer Transformer encoder
and decoder, and five two-layer MLP heads. We average pool the output of the
5th convolution block of ResNet18 to a 512 dimensional vector. We concatenate
the three vectors corresponding to the observation, the prediction, and their
differences, and project them to 256 dimensions with a fully connected layer
to input to the Transformer. For this, we use three independent layers to
obtain the encoder tokens $\\{\mbox{\boldmath$e$}^{\ast
t}_{k}\\}_{k=1}^{K^{t}}$, the decoder tokens
$\\{\mbox{\boldmath$m$}^{t-1}_{k}\\}_{k=1}^{K^{t-1}}$, and the viewing
condition encoding of the current view $\mbox{\boldmath$m$}^{t}_{K^{t}}$. We
embed the 7D camera pose to a token with a learnable vector
$\mbox{\boldmath$W$}_{\text{PE}}\in\mathbb{R}^{7\times 256}$.

TransPoser has three layers of attention in both the encoder and the decoder.
We set the dimension of the outputs and inputs to $d_{\text{model}}=256$, the
dimension of the feed-forward network model to $d_{\text{ff}}=1024$, the
number of the multi-head attentions as $h=8$, and probability of dropout to
$P_{\text{drop}}=0.1$.

The five MLP heads output the updates $\Delta\mbox{\boldmath$p$}^{t}$,
$\Delta\mbox{\boldmath$d$}_{\text{g}}^{t}$,
$\Delta\mbox{\boldmath$d$}_{\text{r}}^{t}$, $\Delta\mbox{\boldmath$s$}^{t}$,
and $\Delta\mbox{\boldmath$z$}^{t}$, respectively. We set the dimension of
each layer to 256. We use ReLU and Leaky ReLU as the activation functions of
the encoder and decoder, and the MLP heads, respectively. We adopt Softplus
for the final output of the MLP head for the scale
$\Delta\mbox{\boldmath$s$}^{t}$.

###  B.4 Details of Experiments with Synthetic Data

This section describes the details of producing the synthetic data and
training using it.

|
---|---
(a) Generated Viewpoints | (b) Sample Rendered Images
Figure B.9: Examples of the randomly generated camera trajectories (a) and the
sets of five synthetic observations (b).

####  B.4.1 Data Preparation

As shown in Figure B.9, we generate camera trajectories by randomly sampling
points on a regular grid inside the upper hemisphere around the object. The
center of the sphere coincides with that of the object instance’s bounding
box, and its radius is 1.5. The interval of the grids is set to 0.3. The
viewpoint starts at a random grid point, and randomly moves up to 2 grid
points for each movement. We randomize the view direction while ensuring that
at least $2\%$ of the pixels capture the target object. The scale is
randomized by a factor within the range of $[0.8,1.2]$.

Supervised training of TransPoser requires ground truth latent code of DeepDDF
for each instance. As those latent codes are available only for the training
set of DeepDDF, we split it into 100, 512, 100, and 1,536 training instances
and 100, 785, 100, and 1,216 validation instances to train TransPoser for the
Cabinet, Chair, Display, and Table categories, respectively. We used the test
set of DeepDDF as the test set for TransPoser, too.

For each epoch of training, we prepare a trajectory consisting of a set of
five images for each object instance, and train the network to optimize the
pose and the shape for 10 iterations. For validation and testing, we prepare 8
and 16 trajectories for each instance, respectively.

Figure B.10: Comparison of convergence speeds. The plots show, from left to right, the estimation errors of translation, rotation, and shape at each optimization step, respectively. The error metrics are those of Table B.6. The results show that the proposed TransPoser achieves fastest and most accurate reconstruction. Observations and GT |  Baseline based on [23] | w/o Dec. | TransPoser
---|---|---|---
|  |  |
|  |  |
|  |  |
Figure B.11: Examples of reconstructed shapes by a baseline method, TransPoser
without the decoder, and the full TransPoser. The inset images show the
estimated shape (gray) over the ground truth silhouette (black) to visualize
the difference. The results demonstrate the importance of self-attention which
weighs the different views.

####  B.4.2 Training Details

We initialize the translation with uniform random noise of
$\mathcal{U}[-0.3,0.3]$, the scale by $\mathcal{U}[0.7,1.3]$, and the rotation
by a random orientation. The shape is initialized with the mean latent code of
the training samples.

We trained TransPoser for 512 epochs using Adam optimizer with a learning rate
of $1.0\times 10^{-4}$, and use the one with the minimum validation loss. In
the first 80 epochs, we trained TransPoser to optimize the the pose and shape
not in 10 iterations but in 5 iterations, as we experimentally found that this
stabilizes the training process. We used
$\lambda_{p}=10.0,\lambda_{g}=1.0,\lambda_{r}=1.0,\lambda_{s}=1.0,\lambda_{z}=1.0$
for the Cabinet, Chair, and Display categories. For the Table category, we
used $\lambda_{r}=0.1$. We evaluated the rotation loss by considering the
symmetric shape ambiguity [8].

Model | Class avg. | Cabinet | Chair
---|---|---|---
T | R | S | T | R | S | T | R | S
w/o NOCS | 1.58 | 1.18 | 3.52 | 1.87 | 1.69 | 4.09 | 1.30 | 0.66 | 2.95
w/o Camera PE | 1.29 | 0.98 | 3.12 | 1.40 | 1.38 | 3.62 | 1.19 | 0.58 | 2.61
w/o $\mbox{\boldmath$m$}^{t}_{K^{t}}$ | 1.35 | 1.01 | 3.13 | 1.59 | 1.35 | 3.52 | 1.10 | 0.67 | 2.74
TransPoser | 1.23 | 1.00 | 3.03 | 1.35 | 1.39 | 3.47 | 1.10 | 0.61 | 2.58
Table B.6: Ablation study of the NOCS representation, the positional encoding
of the camera poses, and the addition of $\mbox{\boldmath$m$}^{t}_{K^{t}}$ to
the input of the decoder. Metrics for translation (T), rotation (R), and shape
(S) are the mean squared error $(10^{-2})$, Riemannian distance $(10^{-1})$,
and Chamfer distance $(10^{-3})$, respectively. The results show that each
component are independently essential to TransPoser.

###  B.5 Details of Experiments with Real Data

This section describes the details of prepossessing and fine-tuning for real
data.

####  B.5.1 Data Preparation

We followed the training, validation, and test splits defined in Scan2CAD, and
used the CAD models aligned with the objects in the real images. The latent
codes of DeepDDF required for supervised training of TransPoser were computed
by fitting DeepDDF nonlinearly to the CAD models. We manually removed several
objects with incorrect instance masks from the training and validation splits.
For a fair performance assessment, we also removed such objects from the test
set when evaluating the shape reconstruction accuracy.

We initialized the translation with the median of the 3D point clouds
constructed from the ones inside the instance mask of the first frame depth
image. The rotation is initialized with a random direction perpendicular to
the floor. The scale is set so that the diagonal of the object bounding box is
$1\mathrm{m}$ for the Chair and Display categories, and $2\mathrm{m}$ for the
Cabinet and Table categories. The initial shape is the mean of the latent
codes of all the DeepDDF training samples in the category.

####  B.5.2 Training Details

We fine-tuned TransPoser pre-trained with the synthetic data, and tested the
model that achieved the highest validation accuracy. We evaluate the accuracy
with the pose evaluation metrics proposed by Scan2CAD. We use Adam optimizer
with a learning rate of $5.0\times 10^{-6}$ for the Cabinet and Display
categories, and $1.0\times 10^{-5}$ for the Chair and Table categories. We
used the same weights for the losses as in the the synthetic data experiment,
and evaluated the rotation loss by considering the symmetric shape ambiguity
[8].

Observations | GT | Est.
---|---|---
|  |
|  |
|  |
Figure B.12: Visualization of the most attended views (images with the orange
borders). They show that TransPoser effectively attends to informative views
with rich silhouette information.

###  B.6 Additional Ablation Studies

##### Convergence speed

Figures B.10 and B.11 show results of comparing Transposer with a baseline
method based on [23] and TransPoser without the decoder for convergence speed
and qualitative results, respectively. Figure B.10 demonstrates quantitatively
that the proposed TransPoser achieves the highest accuracy in most of the
metrics while achieving a faster convergence. TransPoser without the decoder
also converges as fast as TransPoser, but is less accurate as evident in the
rotation error. As shown in Figure B.11, the baseline model fails to
reconstruct the object shape in the observations. This is mainly because the
baseline methods cannot coherently integrate partial observations from each
view.

We also evaluate the contributions of the NOCS representation [37], the
positional encoding of the camera poses, and adding
$\mbox{\boldmath$m$}^{t}_{K^{t}}$ to the input of the decoder when
$K^{t}=K^{t-1}+1$. We evaluated with the Chair and Cabinet categories, and set
the number of total optimization steps to 5.

Table B.6 shows the results of TransPoser without NOCS representation,
TransPoser without the positional encoding of the camera poses, and TransPoser
without $\mbox{\boldmath$m$}^{t}_{K^{t}}$ in the decoder, in comparison with
the full TransPoser. These results clearly show that each of the components
independently contributes to the shape and pose estimation accuracy.

Figure B.12 visualizes, with the orange borders, the views that TransPoser
pays the most attention to among the five input observations. The
visualization is based on the attention of the last cross-attention layer
before computing the update. The results show qualitatively that TransPoser
properly focuses on the viewpoints that are informative for estimating the
pose and shape, such as the images captured from oblique angles which contain
rich silhouette information.

## References

  * [1] Jonas Adler and Ozan Öktem.  Solving ill-posed inverse problems using iterative deep neural networks.  Inverse Problems, 33(12):124007, 2017.
  * [2] Tristan Aumentado-Armstrong, Stavros Tsogkas, Sven Dickinson, and Allan D. Jepson.  Representing 3d shapes with probabilistic directed distance fields.  In CVPR, pages 19343–19354, 2022.
  * [3] The authors of ELLIPSDF[30].  Personal communication, Feb. 2023.
  * [4] Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, and Matthias Niessner.  Scan2CAD: Learning CAD Model Alignment in RGB-D Scans.  In CVPR, 2019.
  * [5] Leonard Bruns and Patric Jensfelt.  SDFEst: Categorical Pose and Shape Estimation of Objects from RGB-D Images using Signed Distance Fields.  IEEE RA-L, 2022.
  * [6] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  Shapenet: An information-rich 3d model repository.  arXiv:1512.03012, 2015.
  * [7] Kai Chen and Qi Dou.  Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation.  In ICCV, pages 2773–2782, 2021.
  * [8] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Linlin Shen, and Ales Leonardis.  FS-Net: Fast Shape-Based Network for Category-Level 6D Object Pose Estimation With Decoupled Rotation Mechanism.  In CVPR, pages 1581–1590, June 2021.
  * [9] Ronald Clark, Michael Bloesch, Jan Czarnowski, Stefan Leutenegger, and Andrew J Davison.  Ls-net: Learning to solve nonlinear least squares for monocular stereo.  In ECCV, 2018.
  * [10] Ronald Clark, John McCormac, Stefan Leutenegger, and Andrew J Davison.  Meta-learning for instance-level data association.  In NeurIPS, 2017.
  * [11] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes.  In CVPR, 2017.
  * [12] Xinke Deng, Junyi Geng, Timothy Bretl, Yu Xiang, and Dieter Fox.  ICaps: Iterative Category-Level Object Pose and Shape Estimation.  IEEE RA-L, 7(2):1784–1791, Apr. 2022.
  * [13] Misha Denil, Sergio Gomez Colmenarejo, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Marcin Andrychowicz, and Nando De Freitas.  Learning to learn by gradient descent by gradient descent.  In NeurIPS, 2016.
  * [14] Erik Gärtner, Luke Metz, Mykhaylo Andriluka, C Daniel Freeman, and Cristian Sminchisescu.  Transformer-based learned optimization.  arXiv:2212.01055, 2022.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [16] Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs.  Improving transformer optimization through better initialization.  In ICML, pages 4475–4483, 2020.
  * [17] Xiaoke Jiang, Donghai Li, Hao Chen, Ye Zheng, Rui Zhao, and Liwei Wu.  Uni6d: A unified cnn framework without projection breakdown for 6d pose estimation.  In CVPR, pages 11174–11184, 2022.
  * [18] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [19] Ke Li and Jitendra Malik.  Learning to optimize.  arXiv preprint arXiv:1606.01885, 2016.
  * [20] Chen-Hsuan Lin and Simon Lucey.  Inverse compositional spatial transformer networks.  In CVPR, pages 2568–2576, 2017.
  * [21] Jiehong Lin, Zewei Wei, Changxing Ding, and Kui Jia.  Category-level 6d object pose and size estimation using self-supervised deep prior deformation networks.  In ECCV, pages 19–34, 2022.
  * [22] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui.  Dist: Rendering deep implicit signed distance function with differentiable sphere tracing.  In CVPR, 2020.
  * [23] Wei-Chiu Ma, Shenlong Wang, Jiayuan Gu, Sivabalan Manivasagam, Antonio Torralba, and Raquel Urtasun.  Deep Feedback Inverse Problem Solver.  In ECCV, pages 229–246, 2020.
  * [24] Kevis-Kokitsi Maninis, Stefan Popov, Matthias Nießner, and Vittorio Ferrari.  Vid2cad: Cad model alignment using multi-view constraints from videos.  IEEE TPAMI, 45(1):1320–1327, 2022.
  * [25] John McCormac, Ronald Clark, Michael Bloesch, Andrew Davison, and Stefan Leutenegger.  Fusion++: Volumetric object-level slam.  In 3DV, pages 32–41. IEEE, 2018.
  * [26] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In CVPR, 2019.
  * [27] Maher Moakher.  Means and averaging in the group of rotations.  SIAM journal on matrix analysis and applications, 24(1):1–16, 2002\.
  * [28] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In CVPR, 2019.
  * [29] Martin Runz, Kejie Li, Meng Tang, Lingni Ma, Chen Kong, Tanner Schmidt, Ian Reid, Lourdes Agapito, Julian Straub, Steven Lovegrove, and Richard Newcombe.  FroDO: From Detections to 3D Objects.  In CVPR, 2020.
  * [30] Mo Shan, Qiaojun Feng, You-Yi Jau, and Nikolay Atanasov.  ELLIPSDF: Joint Object Pose and Shape Optimization with a Bi-level Ellipsoid and Signed Distance Function Description.  In ICCV, pages 5946–5955, 2021.
  * [31] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In CVPR, pages 2437–2446, 2019.
  * [32] Edgar Sucar, Kentaro Wada, and Andrew Davison.  NodeSLAM: Neural object descriptors for multi-view shape reconstruction.  In 3DV, 2020.
  * [33] Meng Tian, Marcelo H Ang Jr, and Gim Hee Lee.  Shape prior deformation for categorical 6d object pose and size estimation.  In ECCV, 2020.
  * [34] Michał J Tyszkiewicz, Kevis-Kokitsi Maninis, Stefan Popov, and Vittorio Ferrari.  RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers.  In ECCV, pages 211–228. Springer, 2022.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention Is All You Need.  In NeurIPS, volume 30, 2017.
  * [36] Lukas Von Stumberg, Patrick Wenzel, Qadeer Khan, and Daniel Cremers.  Gn-net: The gauss-newton loss for multi-weather relocalization.  IEEE RA-L, 5(2):890–897, 2020.
  * [37] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In CVPR, pages 2642–2651, 2019.
  * [38] Jiaze Wang, Kai Chen, and Qi Dou.  Category-level 6d object pose estimation via cascaded relation and recurrent reconstruction networks.  In IROS, 2021.
  * [39] Jingwen Wang, Martin Rünz, and Lourdes Agapito.  Dsp-slam: object oriented slam with deep shape priors.  In 3DV, pages 1362–1371. IEEE, 2021.
  * [40] Yanmin Wu, Yunzhou Zhang, Delong Zhu, Yonghui Feng, Sonya Coleman, and Dermot Kerr.  Eao-slam: Monocular semi-dense object slam based on ensemble data association.  In IROS, pages 4966–4973. IEEE, 2020.
  * [41] Tarun Yenamandra, Ayush Tewari, Nan Yang, Florian Bernard, Christian Theobalt, and Daniel Cremers.  Hdsdf: Hybrid directional and signed distance functions for fast inverse rendering.  arXiv:2203.16284, 2022.
  * [42] Ehsan Zobeidi and Nikolay Atanasov.  A deep signed directional distance function for object shape representation.  arXiv:2107.11024, 2021.