Proposed Methodsec:method Problem Formulation

In this work, a generic object is described by three disentangled latent
parameters: category, shape and albedo. Through two deep networks, these
parameters can be decoded into the $3$D shape and albedo respectively. To have
an end-to-end trainable framework, we estimate these parameters along with the
lighting and camera projection, via an encoder network, i.e., the fitting
module of our model. Three networks work jointly for the objective of
reconstructing the input image of generic objects, by incorporating a physics-
based rendering layer, as in Fig. fig:architecture.

Formally, given a training set of $T$ images $\\{\mathbf{I}_{i}\\}_{i=1}^{T}$
of multiple categories, our objective is to learn i) an encoder $\Enc$:
$\mathbf{I}{\rightarrow}\mathbf{P},\mathbf{L},\f_{C},\f_{S},\f_{A}$ that
outputs the projection $\mathbf{P}$, lighting parameter $\mathbf{L}$, category
code $\f_{C}\in\mathbb{R}^{l_{C}}$, shape code $\f_{S}\in\mathbb{R}^{l_{S}}$,
and albedo code $\f_{A}\in\mathbb{R}^{l_{A}}$, ii) a shape decoder $\Dec_{S}$
that decodes parameters to a $3$D geometry $\mathbf{S}$, represented by an
occupancy field, and iii) an albedo decoder $\Dec_{A}$ that decodes parameters
into a color field $\mathbf{A}$, with the goal that the reconstructed image by
these components ($\mathbf{P},\mathbf{L},\mathbf{S},\mathbf{A}$) can well
approximate the input. This objective can be mathematically presented as:

$\argmin_{\Enc,\Dec_{S},\Dec_{A}}\sum_{i=1}^{T}\norm{\hat{\mathbf{I}}_{i}-\mathbf{I}_{i}}_{1},\vspace{-2mm}$
(1)

where
$\hat{\mathbf{I}}=\mathcal{R}\left(\mathbf{P},\mathbf{L},\Dec_{S}(\f_{C},\f_{S}),\Dec_{A}(\f_{C},\f_{S},\f_{A})\right)$
is the reconstructed image, and $\mathcal{R}(\cdot,\cdot,\cdot,\cdot)$ is the
rendering function. Category-adaptive $3$D Joint Occupancy Fields Unlike $2$D,
the community has not yet agreed on a $3$D representation both memory
efficient and inferable from data. Recently, implicit representations gain
popularity as their continuous functions offer high-fidelity surface.
Motivated by this, we propose a $3$D _joint occupancy fields_ (JOF)
representation to simultaneously model shape and albedo with unsupervised
segmentation, offering part-level correspondence for $3$D shapes, as in Fig.
1. JOF has three novel designs over prior implicit representations
[chen2018learning, mescheder2018occupancy, park2019deepsdf,
oechsle2019texture, chen2019bae]: 1) we extend the idea of unsupervised
segmentation [chen2019bae] from shape to albedo, 2) we integrate shape
segmentation into albedo decoder, guiding segmentation by both geometry and
appearance cues, and 3) we condition JOF on the category to model multiple
categories. Category Code Unlike prior implicit representations, we introduce
a category code $\mathbf{f}_{C}$ as the additional input to the shape and
albedo decoders. In training, $\mathbf{f}_{C}$ is supervised by a cross-
entropy loss using the class label of each image. In the context of modeling
shape deformation of multiple categories, using $\mathbf{f}_{C}$ enables
decoders to focus on modeling intra-class deformations via $\mathbf{f}_{S}$.
Further, the $\mathbf{f}_{C}$ embedding may generalize to unseen categories
too. Shape Component As adopted from [chen2018learning,
mescheder2018occupancy, park2019deepsdf], each shape is represented by a
function, implemented as a decoder network, $\Dec_{S}$:
$\mathbb{R}^{l_{C}}{\times}\mathbb{R}^{l_{S}}{\times}\mathbb{R}^{3}\rightarrow[0,1]$,
which inputs a $3$D location $\mathbf{x}$, category code $\f_{C}$, shape codes
$\f_{S}$ and outputs its probability of occupancy $o$. One appealing property
is that the surface normal can be computed by the spatial derivative
$\frac{\delta\Dec_{S}}{\delta\mathbf{x}}$ via back-propagation through the
network, which is helpful for subsequent tasks such as rendering. To offer
unsupervised part segmentation, we adopt BAE-NET [chen2019bae] as the
architecture of $\Dec_{S}$. It is composed of $3$ fully connected layers and
the final layer is a branched layer that gives the occupancy value for each of
$k$ branchs, denoted by $\\{o\\}_{i=1}^{k}$ in Fig. 1 (a). Finally, a max
pooling on the branch outputs the result of the final occupancy. Albedo
Component Albedo component assigns each vertex on the $3$D surface a RGB
albedo. One may use a combination of category and albedo codes to represent a
colored shape, _i.e._ , $\Dec_{A}(\f_{C},\f_{A},\mathbf{x})$. However, it puts
a redundant burden to $\f_{A}$ to encode the object geometry, _e.g._ , the
position of the tire, and body of a car. Hence, we also feed the shape code
$\f_{S}$ as an additional input to the albedo decoder, _i.e._ , $\Dec_{A}$:
$\mathbb{R}^{l_{C}}{\times}\mathbb{R}^{l_{S}}{\times}\mathbb{R}^{l_{A}}{\times}\mathbb{R}^{3}\rightarrow\mathbb{R}^{3}$
(Fig. 1(b)).

[trim=0 0 0 0,clip, width=]img/alg/decoders.png

Figure : (a) Shape decoder $\Dec_{S}$ inputs category, shape parameters
$\f_{C}$, $\f_{S}$, a spatial point $\mathbf{x}{=}(x,y,z)$, and produces the
implicit field for $k$ branches. Max pooling of the branch outputs leads to
the probability of occupancy $o$. (b) Albedo decoder $\Dec_{A}$ receives an
additional input $\f_{A}$ and estimates the albedo of all branches, one of
which is selected as the final albedo of $\mathbf{x}$. (c) Unsupervisedly
segmented parts and their albedo match well with intuition.

Inspired by the design of $\Dec_{S}$, we propose to estimate the albedo for
$k$ branches $\\{c\\}_{i=1}^{k}$. For each $\mathbf{x}$, the final albedo is
$c_{idx}$, where $idx{=}\argmax_{i}(o_{i})$ is the index of segment where
$\mathbf{x}$ belongs to (Fig. 1(c)). This novel design integrates shape
segmentation into albedo learning, benefiting both segmentation and
reconstruction (Tab. LABEL:tab:segmentation). The key motivation is that,
different parts of an object often differ in shape and/or albedo, and thus
both shall guide the segmentation. Physics-based Rendering To render an image
($W\times H$ pixels) from shape, albedo, as well as lighting parameters
$\mathbf{L}$ and projection $\mathbf{P}$, we first find a set of $W\times H$
$3$D surface points corresponding to the $2$D pixel. Then the RGB color of
each pixel is computed via a lighting model using lighting $\mathbf{L}$ and
decoder outputs. Camera Model We assume a full perspective camera model. Any
spatial points $\mathbf{x}$ in the world space can be projected to camera
space by a multiplication between a $3{\times}4$ full perspective projection
matrix $\mathbf{P}$ and its homogeneous coordinate:
$\mathbf{u}=\mathbf{P}\left[\mathbf{x},1\right]^{T}$, $\mathbf{u}=[u\cdot
d,v\cdot d,d]^{T}$, where $d$ is the depth value of image coordinate $(u,v)$.
Essentially, $\mathbf{P}$ can be extended to a $4{\times}4$ matrix. By an
abuse of notation in homogeneous coordinates, relation between $3$D points
$\mathbf{x}$ and its camera space projection $\mathbf{u}$ can be written as:

$\mathbf{u}=\mathbf{P}\mathbf{x},\hskip 8.53581pt\text{and}\hskip
8.53581pt\mathbf{x}=\mathbf{P}^{-1}\mathbf{u}.\vspace{-2mm}$ (2)

Surface Point Detection To render a $2$D image, for each ray from the camera
to the pixel $j=(u,v)$, we select one “surface point”. Here, a surface point
is defined as the first interior point ($\Dec_{S}(\mathbf{x})>\tau$), or the
exterior point with the largest $\Dec_{S}(\mathbf{x})$ in case the ray doesn’t
hit the object. For efficient training, instead of finding exact surface
points, we approximate them via Linear or Linear-Binary search. Intuitively,
with the distance margin error of $\epsilon$, in Linear search, along each ray
we evaluate $\Dec_{S}(\mathbf{x})$ for all spatial point candidates
$\mathbf{x}$ with a step size of $\epsilon$. In Linear-Binary search, after
the first interior point is found, as $\Dec_{S}(\mathbf{x})$ is a continuous
function, a Binary search can be used to better approximate the surface point.
With the same computational budget, Linear-Binary search leads to better
approximation of surface points, hence higher rendering quality. The search
algorithm is detailed in the supplementary material (_Supp._). Image Formation
We assume purely Lambertian surface reflectance and distant low-frequency
illumination. Thus, the incoming radiance can be approximated via Spherical
Harmonics (SH) basis functions
$\mathbf{H}_{b}:\mathbb{R}^{3}\rightarrow\mathbb{R}$, and controlled by
coefficients $\mathbf{L}=\\{\gamma_{b}\\}_{b=1}^{3B^{2}}$. At the pixel $j$
with corresponding surface point $\mathbf{x}_{j}$, the image color is computed
as a product of albedo $\mathbf{A}_{j}$ and shading $\mathbf{C}_{j}$:

$\I_{j}{\,=\,}\A_{j}\cdot\mathbf{C}_{j}{\,=}\Dec_{A}(\mathbf{x}_{j})\cdot\sum_{b=1}^{B^{2}}\gamma_{b}\mathbf{H}_{b}\left(\sigma\left(\frac{\delta\Dec_{S}(\mathbf{x}_{j})}{\delta\mathbf{x}_{j}}\right)\right),\vspace{-2mm}$
(3)

where
$\mathbf{n}_{j}{=}\sigma\left(\frac{\delta\Dec_{S}(\mathbf{x}_{j})}{\delta\mathbf{x}_{j}}\right)$
is the surface normal direction at $\mathbf{x}_{j}$, $L_{2}$-normalized by
function $\sigma()$. We use $B{=}3$ SH bands, which leads to $B^{2}{=}9$
coefficients for each color channel. Semi-Supervised Model Learning While our
model is designed to learn from real-world images, we benefit from pre-
training shape and albedo with CAD models, given inherent ambiguity in inverse
tasks. We first describe learning from images self-supervisedly, and then pre-
training from CAD models with supervision. Self-supervised Joint Modeling and
FittingGiven a set of $2$D images without ground truth $3$D shapes, we define
the loss as ($\lambda_{i}$ are the weights):

$\argmin_{\Enc,\Dec_{A}}\mathcal{L}_{3}=\mathcal{L}_{\text{img}}+\lambda_{1}\mathcal{L}_{\text{sil}}+\lambda_{2}\mathcal{L}_{\text{fea-
const}}+\lambda_{3}\mathcal{L}_{\text{reg}},\vspace{-1mm}$ (4)

where $\mathcal{L}_{\text{img}}$ is the photometric loss,
$\mathcal{L}_{\text{sil}}$ enforces silhouette consistency,
$\mathcal{L}_{\text{fea-const}}$ is the local feature consistency loss, and
$\mathcal{L}_{\text{reg}}$ includes two regularization terms
($\mathcal{L}_{\text{alb-const}}$, $\mathcal{L}_{\text{bws}})$. Silhouette
Loss Given the object’s silhouette $\mathbf{M}$, obtained by a segmentation
method [qin2020u2], we define the loss as:

$\resizebox{0.9}{!}{\mathcal{L}_{\text{sil}}=\frac{1}{W\times
H}\sum_{j=1}^{W{\times}H}\norm{\Dec_{S}(\Enc_{C}(\mathbf{I}),\Dec_{S}(\Enc_{S}(\mathbf{I}),\Enc_{P}(\mathbf{I})^{-1}\mathbf{u}_{j})-o_{j}}_{1},}$
(5)

where $\Enc_{C},\Enc_{S},\Enc_{P}$ are parts of the encoder that estimate
$\mathbf{f}_{C}$, $\mathbf{f}_{S}$ and $\mathbf{P}$ respectively and the three
inputs to $\Dec_{S}$ are $\f_{C}$, $\f_{S}$ and $\mathbf{x}_{j}$. With the
occupancy field, the occupancy value $o_{j}$ is $0.5$ if $\mathbf{M}_{j}=1$,
otherwise $o_{j}=0$. Here, we also analyze how our silhouette loss differs
from prior work. If a 3D shape is represented as a mesh, there is no gradient
when comparing two binary masks, unless the predicted silhouette is
expensively approximated as in Soft rasterizer [liu2019soft]. If the shape is
represented by a voxel, the loss can provide gradient to adjust voxel
occupancy predictions, but not the object orientation [tulsiani2017multi]. Our
loss can update both shape occupancy field and camera projection estimation
(Eqn. 5). Photometric Loss To enforce similarity between our reconstruction
and input, we use a $L_{1}$ loss on the foreground:

$\mathcal{L}_{\text{img}}=\frac{1}{|\mathbf{M}|}\norm{(\mathbf{\hat{I}}-\mathbf{I})\odot\mathbf{M}}_{1},\vspace{-1mm}$
(6)

where $\odot$ is the element-wise product.

! [trim=13 0 0 0,clip,width=1]img/alg/fea_extra_eccv.png

Figure : (a) Local feature extraction. For an image $\mathbf{I}_{i}$, part
segmentation allows selecting and projecting $3$D boundary points onto
$\mathbf{I}_{i}$. Using their $2$D locations to sample the first $3$ feature
maps of the encoder results in the set of local features $\mathbf{F}$, whose
eigenvector $\mathbf{v}_{i}$ is used in $\mathcal{L}_{\text{fea-const}}$.
t-SNE plots of $\mathbf{v}_{i}$ from $1{,}000$ car images using the models
trained without (b) or with (c) $\mathcal{L}_{\text{fea-const}}$. Blue and red
are $\mathbf{v}_{i}$ of boundary pixels and randomly sampled pixels
respectively. While distributions of random pixels remain scattered,
$\mathcal{L}_{\text{fea-const}}$ helps boundary pixels to have more similar
feature distribution, thus better semantic correspondence across images.

Local Feature Consistency Loss Our decoders _unsupervisedly_ offer part-level
correspondence via learnt segmentation (Fig. 1), with which we assume that the
boundary pixels of adjacent segments in one image have a similar
_distribution_ of appearance as another image of the same category. This
assumption leads to a novel loss function (Fig. 1). For one segmented $3$D
shape, we first select $q$ boundary points
$\mathbf{U}_{3D}\in\mathbb{R}^{q\times 3}$ from all pairs of adjacent segments
based on branches of $\Dec_{S}$, _i.e._ , a point and its spatial neighbor
shall trigger different branches. These $3$D points are projected to the image
plane $\mathbf{U}_{2D}\in\mathbb{R}^{q\times 2}$ via estimated $\mathbf{P}$.
Similar to [xu2019disn], we retrieve features from feature maps via the
location $\mathbf{U}_{2D}$ and form the local features
$\mathbf{F}\in\mathbb{R}^{q\times 256}$, where $256$ is the total feature
dimension of $3$ layers. Finally, we calculate the largest eigenvector
$\mathbf{v}$ of the covariance matrix $(\mathbf{F}-\mu)^{T}(\mathbf{F}-\mu)$
($\mu$ is the row-wise mean of $\mathbf{F}$), which describes the largest
feature variation of $q$ points. Despite two images of the same category may
differ in colors, we assume there is similarity in their respective major
variations. Thus, we define the local feature consistency loss as:

$\mathcal{L}_{\text{fea-const}}=\frac{1}{|B|}\sum_{(i,j)\in
B}\norm{\mathbf{v}_{i}-\mathbf{v}_{j}}_{1},\vspace{-2mm}$ (7)

where $B$ is a training batch of the same category. This loss drives the
semantically equivalent boundary pixels across multiple images to be projected
from the same $3$D boundary adjoining two $3$D segments, thus improving pose
and shape estimation. Regularization Loss We define two regularizations:
_Albedo local constancy_ : assuming piecewise-constant albedo
[land1971lightness], we enforce the gradient sparsity in two directions
[shu2017neural]: $\mathcal{L}_{\text{alb-
const}}=\sum_{t\in\mathcal{N}_{j}}\omega(j,t)\norm{\mathbf{A}_{j}-\mathbf{A}_{t}}_{2}^{p}$,
where $\mathcal{N}_{j}$ represents pixel $j$’s $4$ neighbor pixels. Assuming
that pixels with the same chromaticity (_i.e._ ,
$\mathbf{c}_{j}={\mathbf{I}_{j}}/{|\mathbf{I}_{j}|}$) are more likely to have
the same albedo, we set the weight
$\omega(j,t)=e^{-\alpha\norm{\mathbf{c}_{j}-\mathbf{c}_{t}}}$, where the color
is referred of the input image. We set $\alpha=15$ and $p=0.8$ as in
[meka2016live]. _Batch-wise White Shading_ : Similar to [shu2017neural], to
prevent the network from generating arbitrary bright or dark shading, we use a
batch-wise white shading constraint:
$\mathcal{L}_{\text{bws}}=\norm{\frac{1}{m}\sum_{j=1}^{m}\mathbf{C}_{j}^{(r)}-c}_{1}$,
where $\mathbf{C}_{j}^{(r)}$ is a red channel diffuse shading of pixel $j$.
$m$ is the total number of foreground pixels in a mini-batch. $c$ is the
average shading target, which is set to $1$. The same constraint is applied to
other channels. Supervised Learning with Synthetic Images Before self-
supervision, we pre-train with CAD models and synthetic data, vital for
converging to faithful solutions. Pre-training Shape and Albedo Decoder For
auto-encoding $3$D shape and albedo, we adopt a $3$D CNN as encoder
$\Enc^{\prime}$ to extract category, shape and albedo codes $\f_{C}$,
$\f_{S}$, $\f_{A}$ from a $64^{3}{\times}3$ _colored_ voxel. As in Fig. 1,
given a dataset of CAD models, a model (with class label $y$) can be
represented as a colored $3$D occupancy voxel $\mathbf{V}$. Equivalently, it
can also be represented by $K$ spatial points $\mathbf{x}\in\mathbb{R}^{3}$
and their occupancy $o$, albedo $c$. We define the following loss:

[trim=0 0 0 0,clip, width=0.99]img/alg/pre_train.png

Figure :  Colored $3$D voxel encoder $\mathcal{E^{\prime}}$ and decoders pre-
training.

$\argmin_{\Dec_{S},\Dec_{A},\Enc^{\prime}}\mathcal{L}_{1}=\mathcal{L}_{S}+\mathcal{L}_{A}+\mathcal{L}_{C},\vspace{-2mm}$
(8)

where
$\mathcal{L}_{S}{=}\sum_{j=1}^{K}\norm{\Dec_{S}(\Enc^{\prime}_{C}(\mathbf{V}),\Enc^{\prime}_{S}(\mathbf{V}),\mathbf{x}_{j})-o_{j}}_{2}^{2}$,
$\mathcal{L}_{A}{=}\sum_{j=1}^{K}\norm{\Dec_{A}(\Enc^{\prime}_{C}(\mathbf{V}),\Enc^{\prime}_{S}(\mathbf{V}),\Enc^{\prime}_{A}(\mathbf{V}),\mathbf{x}_{j})-c_{j}}_{2}^{2}$,
and $\mathcal{L}_{C}$ is cross-entropy loss for class label $y$. Note that
training $\Enc^{\prime}$ is necessary to learn valid distributions of
$\f_{C}$, $\f_{S}$, $\f_{A}$, although $\Enc^{\prime}$ is discarded after this
pre-training step. Pre-training Image Encoder Given a CAD model, we render
multiple images of the same object with different poses and lighting, each
forming a triplet of voxel, image and ground truth projection
$(\mathbf{V},\I,\mathbf{\widetilde{P}})$. These synthetic data can supervise
the pre-training of encoder $\Enc$ by minimizing the $\mathcal{L}_{2}$ below,
where the ground truth shape and albedo parameters are obtained by feeding
voxel $\mathbf{V}$ into $\Enc^{\prime}$,

$\resizebox{0.99}{!}{\mathcal{L}_{2}=\mathcal{L}_{\text{img}}+\sum_{X\in\\{C,S,A\\}}\lambda_{X}\norm{\Enc_{X}(\mathbf{I})-\Enc^{\prime}_{X}(\mathbf{V})}^{2}_{2}+\lambda_{P}\norm{\Enc_{P}(\mathbf{I})-\mathbf{\widetilde{P}}}^{2}_{2}}.\vspace{-1mm}$

Implementation and Discussion Our training process contains three steps: 1)
$\Dec_{S}$, $\Dec_{A}$ and $\Enc^{\prime}$ are pre-trained on colored voxels
and corresponding sampled point-value pairs (Eqn. 8); 2) $\Enc$ is pre-trained
with synthetic images by minimizing $\mathcal{L}_{2}$; 3) $\Enc$ and
$\Dec_{A}$ are trained using real images (Eqn. 4). We empirically found that,
Step $3$ training has incremental gain when updating the shape decoder. But it
significantly improves the generalization ability of our encoder on fitting
model to real images. Thus, we opt to freeze the shape decoder after Step $1$.
For more details about the training setting, please refer to _Supp._. One key
enabler of our learning with real images is the _differentiable_ rendering
layer. For the rendering function of Eqn. 3, one can compute partial
derivatives over $\mathbf{L}$, over $\mathbf{P}$ since
$\mathbf{x}=\mathbf{P}^{-1}\mathbf{u}$, over $\f_{C}$, $\f_{S}$, $\f_{A}$ as
they are the inputs of $\Dec_{S}$, $\Dec_{A}$, and over the network parameters
of $\Dec_{S}$, $\Dec_{A}$. However, although the derivative over
$\mathbf{x}_{j}$ can be computed, the surface point search process is not
differentiable.